The content for this site available on GitHub. If you want to launch the notebooks interactively click on the binder stamp below. 
< AutoEDAs using ydataprofiling | Contents | EDA Abalone >
Auto-ML using FLAML¶
Like ydata automates a lot of the EDA process, FLAML automates some of the ML process. Given a dataset, task type (ie classification, regression, etc), and a time budget, FLAML will automatically search for the best model and hyperparameters.
Like ydata, its a useful tool to quickly get a sense of the data and what models might work well but I've found it can be misleading and still isn't as good as a human in terms of understanding the data and what models might work best. It works best when guided with some domain knowledge and understanding of the data.
In [1]:
%reload_ext autoreload
%autoreload 2
In [2]:
from ucimlrepo import fetch_ucirepo
from sklearn.datasets import load_digits, load_breast_cancer
from ydata_profiling import ProfileReport
from flaml import AutoML, automl
from flaml.automl.data import get_output_from_log
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, r2_score
from helpers.base_imports import *
Upgrade to ydata-sdk
Improve your data and profiling with ydata-sdk, featuring data quality scoring, redundancy detection, outlier identification, text validation, and synthetic data generation.
Gather datasets¶
In [3]:
# Dictionary of datasets: name -> (features, target)
datasets = {}
# UCI datasets
for name, id in [("abalone", 1), ("iris", 53), ("wine", 109)]:
data = fetch_ucirepo(id=id).data
df = pd.concat([data.features, data.targets], axis=1)
datasets[name] = df
# sklearn.datasets: Digits
digits = load_digits(as_frame=True)
df_digits = digits.frame # Already includes 'target' column
datasets["digits"] = df_digits
# sklearn.datasets: Breast Cancer
bc = load_breast_cancer(as_frame=True)
df_bc = bc.frame # Already includes 'target' column
datasets["breast_cancer"] = df_bc
Setup FLAML run¶
Configurable parameters are task (classification/regression), metric (accuracy, f1, etc.), and time limit/budget (in seconds), estimator_list (rf, xgboost, etc), log_file (to save the results), and n_splits (number of folds for cross-validation). The default is 5-fold cross-validation.
In [8]:
def run_flaml_automl(
df, target_column, task="classification", time_budget=15, dataset_name="dataset"
):
# 1. Split into features and target
X = df.drop(columns=[target_column])
y = df[target_column]
# 2. Train/test split (80/20)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
# 3. Create FLAML AutoML instance
automl = AutoML()
log_path = f"data/{dataset_name}_flaml.log"
# 4. Fit AutoML model with configuration
automl.fit(
X_train=X_train,
y_train=y_train,
task=task,
time_budget=time_budget,
log_file_name=log_path, # Enable logging so we can extract val_loss
)
# 5. Evaluate on test set
y_pred = automl.predict(X_test)
if task == "classification":
score = accuracy_score(y_test, y_pred)
print(f"[{dataset_name}] Accuracy: {score:.4f}")
elif task == "regression":
score = r2_score(y_test, y_pred)
print(f"[{dataset_name}] R² Score: {score:.4f}")
print(f"[{dataset_name}] Best estimator: {automl.best_estimator}")
print(f"[{dataset_name}] Best config: {automl.best_config}")
print(f"[{dataset_name}] Best validation loss: {automl.best_loss:.4f}")
rows = []
with open(log_path, "r") as f:
for line in f:
rec = json.loads(line)
rows.append(
{
"iteration": rec.get("record_id"),
"val_loss": rec.get("validation_loss"),
"learner": rec.get("learner", "unknown"),
"runtime": rec.get("trial_time"), # or .get("wall_clock_time")
"sample_size": rec.get("sample_size"),
"hyperparams": rec.get("config", {}),
}
)
df_results = (
pd.DataFrame(rows)
.sort_values("val_loss")
.loc[
:,
[
"iteration",
"val_loss",
"learner",
"runtime",
"sample_size",
"hyperparams",
],
]
)
# 7. Save
df_results.to_csv(f"data/automl_{dataset_name}.csv", index=False)
df_results.to_html(f"data/automl_{dataset_name}.html", index=False)
print(
f"[{dataset_name}] Trials with val_loss, learner, runtime, sample_size and hyperparams saved."
)
return df_results
Run on datasets¶
In [9]:
# Run FLAML on abalone dataset
# based on the eda the abalone dataset is best suited for regression since the target column is continuous
abalone_res = run_flaml_automl(
df=datasets["abalone"],
target_column="Rings",
task="regression",
time_budget=15,
dataset_name="abalone",
)
abalone_res
[flaml.automl.logger: 05-07 17:17:22] {1728} INFO - task = regression
[flaml.automl.logger: 05-07 17:17:22] {1739} INFO - Evaluation method: cv
[flaml.automl.logger: 05-07 17:17:22] {1838} INFO - Minimizing error metric: 1-r2
[flaml.automl.logger: 05-07 17:17:22] {1955} INFO - List of ML learners in AutoML Run: ['lgbm', 'rf', 'xgboost', 'extra_tree', 'xgb_limitdepth', 'sgd']
[flaml.automl.logger: 05-07 17:17:22] {2258} INFO - iteration 0, current learner lgbm
[flaml.automl.logger: 05-07 17:17:22] {2393} INFO - Estimated sufficient time budget=416s. Estimated necessary time budget=3s.
[flaml.automl.logger: 05-07 17:17:22] {2442} INFO - at 0.1s, estimator lgbm's best error=0.7882, best estimator lgbm's best error=0.7882
[flaml.automl.logger: 05-07 17:17:22] {2258} INFO - iteration 1, current learner lgbm
[flaml.automl.logger: 05-07 17:17:22] {2442} INFO - at 0.1s, estimator lgbm's best error=0.7882, best estimator lgbm's best error=0.7882
[flaml.automl.logger: 05-07 17:17:22] {2258} INFO - iteration 2, current learner sgd
[flaml.automl.logger: 05-07 17:17:22] {2442} INFO - at 0.4s, estimator sgd's best error=0.7069, best estimator sgd's best error=0.7069
[flaml.automl.logger: 05-07 17:17:22] {2258} INFO - iteration 3, current learner xgboost
[flaml.automl.logger: 05-07 17:17:22] {2442} INFO - at 0.4s, estimator xgboost's best error=0.7878, best estimator sgd's best error=0.7069
[flaml.automl.logger: 05-07 17:17:22] {2258} INFO - iteration 4, current learner xgboost
[flaml.automl.logger: 05-07 17:17:22] {2442} INFO - at 0.5s, estimator xgboost's best error=0.7878, best estimator sgd's best error=0.7069
[flaml.automl.logger: 05-07 17:17:22] {2258} INFO - iteration 5, current learner lgbm
[flaml.automl.logger: 05-07 17:17:22] {2442} INFO - at 0.5s, estimator lgbm's best error=0.6291, best estimator lgbm's best error=0.6291
[flaml.automl.logger: 05-07 17:17:22] {2258} INFO - iteration 6, current learner xgboost
[flaml.automl.logger: 05-07 17:17:22] {2442} INFO - at 0.6s, estimator xgboost's best error=0.6302, best estimator lgbm's best error=0.6291
[flaml.automl.logger: 05-07 17:17:22] {2258} INFO - iteration 7, current learner extra_tree
[flaml.automl.logger: 05-07 17:17:23] {2442} INFO - at 0.8s, estimator extra_tree's best error=0.6472, best estimator lgbm's best error=0.6291
[flaml.automl.logger: 05-07 17:17:23] {2258} INFO - iteration 8, current learner rf
[flaml.automl.logger: 05-07 17:17:23] {2442} INFO - at 0.9s, estimator rf's best error=0.6186, best estimator rf's best error=0.6186
[flaml.automl.logger: 05-07 17:17:23] {2258} INFO - iteration 9, current learner lgbm
[flaml.automl.logger: 05-07 17:17:23] {2442} INFO - at 1.0s, estimator lgbm's best error=0.4949, best estimator lgbm's best error=0.4949
[flaml.automl.logger: 05-07 17:17:23] {2258} INFO - iteration 10, current learner lgbm
[flaml.automl.logger: 05-07 17:17:23] {2442} INFO - at 1.1s, estimator lgbm's best error=0.4949, best estimator lgbm's best error=0.4949
[flaml.automl.logger: 05-07 17:17:23] {2258} INFO - iteration 11, current learner lgbm
[flaml.automl.logger: 05-07 17:17:23] {2442} INFO - at 1.1s, estimator lgbm's best error=0.4949, best estimator lgbm's best error=0.4949
[flaml.automl.logger: 05-07 17:17:23] {2258} INFO - iteration 12, current learner rf
[flaml.automl.logger: 05-07 17:17:23] {2442} INFO - at 1.3s, estimator rf's best error=0.5196, best estimator lgbm's best error=0.4949
[flaml.automl.logger: 05-07 17:17:23] {2258} INFO - iteration 13, current learner rf
[flaml.automl.logger: 05-07 17:17:23] {2442} INFO - at 1.5s, estimator rf's best error=0.5196, best estimator lgbm's best error=0.4949
[flaml.automl.logger: 05-07 17:17:23] {2258} INFO - iteration 14, current learner xgboost
[flaml.automl.logger: 05-07 17:17:23] {2442} INFO - at 1.6s, estimator xgboost's best error=0.5914, best estimator lgbm's best error=0.4949
[flaml.automl.logger: 05-07 17:17:23] {2258} INFO - iteration 15, current learner rf
[flaml.automl.logger: 05-07 17:17:24] {2442} INFO - at 1.8s, estimator rf's best error=0.4841, best estimator rf's best error=0.4841
[flaml.automl.logger: 05-07 17:17:24] {2258} INFO - iteration 16, current learner extra_tree
[flaml.automl.logger: 05-07 17:17:24] {2442} INFO - at 2.0s, estimator extra_tree's best error=0.5738, best estimator rf's best error=0.4841
[flaml.automl.logger: 05-07 17:17:24] {2258} INFO - iteration 17, current learner lgbm
[flaml.automl.logger: 05-07 17:17:24] {2442} INFO - at 2.0s, estimator lgbm's best error=0.4906, best estimator rf's best error=0.4841
[flaml.automl.logger: 05-07 17:17:24] {2258} INFO - iteration 18, current learner sgd
[flaml.automl.logger: 05-07 17:17:24] {2442} INFO - at 2.1s, estimator sgd's best error=0.7069, best estimator rf's best error=0.4841
[flaml.automl.logger: 05-07 17:17:24] {2258} INFO - iteration 19, current learner lgbm
[flaml.automl.logger: 05-07 17:17:24] {2442} INFO - at 2.1s, estimator lgbm's best error=0.4906, best estimator rf's best error=0.4841
[flaml.automl.logger: 05-07 17:17:24] {2258} INFO - iteration 20, current learner extra_tree
[flaml.automl.logger: 05-07 17:17:24] {2442} INFO - at 2.3s, estimator extra_tree's best error=0.5738, best estimator rf's best error=0.4841
[flaml.automl.logger: 05-07 17:17:24] {2258} INFO - iteration 21, current learner xgboost
[flaml.automl.logger: 05-07 17:17:24] {2442} INFO - at 2.4s, estimator xgboost's best error=0.5832, best estimator rf's best error=0.4841
[flaml.automl.logger: 05-07 17:17:24] {2258} INFO - iteration 22, current learner extra_tree
[flaml.automl.logger: 05-07 17:17:24] {2442} INFO - at 2.6s, estimator extra_tree's best error=0.5193, best estimator rf's best error=0.4841
[flaml.automl.logger: 05-07 17:17:24] {2258} INFO - iteration 23, current learner rf
[flaml.automl.logger: 05-07 17:17:25] {2442} INFO - at 2.8s, estimator rf's best error=0.4702, best estimator rf's best error=0.4702
[flaml.automl.logger: 05-07 17:17:25] {2258} INFO - iteration 24, current learner rf
[flaml.automl.logger: 05-07 17:17:25] {2442} INFO - at 3.0s, estimator rf's best error=0.4702, best estimator rf's best error=0.4702
[flaml.automl.logger: 05-07 17:17:25] {2258} INFO - iteration 25, current learner sgd
[flaml.automl.logger: 05-07 17:17:25] {2442} INFO - at 3.0s, estimator sgd's best error=0.5408, best estimator rf's best error=0.4702
[flaml.automl.logger: 05-07 17:17:25] {2258} INFO - iteration 26, current learner sgd
[flaml.automl.logger: 05-07 17:17:25] {2442} INFO - at 3.4s, estimator sgd's best error=0.5408, best estimator rf's best error=0.4702
[flaml.automl.logger: 05-07 17:17:25] {2258} INFO - iteration 27, current learner rf
[flaml.automl.logger: 05-07 17:17:25] {2442} INFO - at 3.6s, estimator rf's best error=0.4702, best estimator rf's best error=0.4702
[flaml.automl.logger: 05-07 17:17:25] {2258} INFO - iteration 28, current learner sgd
[flaml.automl.logger: 05-07 17:17:26] {2442} INFO - at 3.6s, estimator sgd's best error=0.5408, best estimator rf's best error=0.4702
[flaml.automl.logger: 05-07 17:17:26] {2258} INFO - iteration 29, current learner sgd
[flaml.automl.logger: 05-07 17:17:26] {2442} INFO - at 3.9s, estimator sgd's best error=0.5408, best estimator rf's best error=0.4702
[flaml.automl.logger: 05-07 17:17:26] {2258} INFO - iteration 30, current learner sgd
[flaml.automl.logger: 05-07 17:17:26] {2442} INFO - at 3.9s, estimator sgd's best error=0.5071, best estimator rf's best error=0.4702
[flaml.automl.logger: 05-07 17:17:26] {2258} INFO - iteration 31, current learner rf
[flaml.automl.logger: 05-07 17:17:26] {2442} INFO - at 4.1s, estimator rf's best error=0.4702, best estimator rf's best error=0.4702
[flaml.automl.logger: 05-07 17:17:26] {2258} INFO - iteration 32, current learner lgbm
[flaml.automl.logger: 05-07 17:17:26] {2442} INFO - at 4.3s, estimator lgbm's best error=0.4551, best estimator lgbm's best error=0.4551
[flaml.automl.logger: 05-07 17:17:26] {2258} INFO - iteration 33, current learner lgbm
[flaml.automl.logger: 05-07 17:17:26] {2442} INFO - at 4.3s, estimator lgbm's best error=0.4551, best estimator lgbm's best error=0.4551
[flaml.automl.logger: 05-07 17:17:26] {2258} INFO - iteration 34, current learner extra_tree
[flaml.automl.logger: 05-07 17:17:26] {2442} INFO - at 4.5s, estimator extra_tree's best error=0.4825, best estimator lgbm's best error=0.4551
[flaml.automl.logger: 05-07 17:17:26] {2258} INFO - iteration 35, current learner lgbm
[flaml.automl.logger: 05-07 17:17:27] {2442} INFO - at 5.2s, estimator lgbm's best error=0.4512, best estimator lgbm's best error=0.4512
[flaml.automl.logger: 05-07 17:17:27] {2258} INFO - iteration 36, current learner extra_tree
[flaml.automl.logger: 05-07 17:17:27] {2442} INFO - at 5.4s, estimator extra_tree's best error=0.4825, best estimator lgbm's best error=0.4512
[flaml.automl.logger: 05-07 17:17:27] {2258} INFO - iteration 37, current learner sgd
[flaml.automl.logger: 05-07 17:17:27] {2442} INFO - at 5.4s, estimator sgd's best error=0.5071, best estimator lgbm's best error=0.4512
[flaml.automl.logger: 05-07 17:17:27] {2258} INFO - iteration 38, current learner xgboost
[flaml.automl.logger: 05-07 17:17:27] {2442} INFO - at 5.5s, estimator xgboost's best error=0.5832, best estimator lgbm's best error=0.4512
[flaml.automl.logger: 05-07 17:17:27] {2258} INFO - iteration 39, current learner extra_tree
[flaml.automl.logger: 05-07 17:17:28] {2442} INFO - at 5.7s, estimator extra_tree's best error=0.4713, best estimator lgbm's best error=0.4512
[flaml.automl.logger: 05-07 17:17:28] {2258} INFO - iteration 40, current learner lgbm
[flaml.automl.logger: 05-07 17:17:29] {2442} INFO - at 6.6s, estimator lgbm's best error=0.4512, best estimator lgbm's best error=0.4512
[flaml.automl.logger: 05-07 17:17:29] {2258} INFO - iteration 41, current learner extra_tree
[flaml.automl.logger: 05-07 17:17:29] {2442} INFO - at 6.8s, estimator extra_tree's best error=0.4713, best estimator lgbm's best error=0.4512
[flaml.automl.logger: 05-07 17:17:29] {2258} INFO - iteration 42, current learner extra_tree
[flaml.automl.logger: 05-07 17:17:29] {2442} INFO - at 7.0s, estimator extra_tree's best error=0.4713, best estimator lgbm's best error=0.4512
[flaml.automl.logger: 05-07 17:17:29] {2258} INFO - iteration 43, current learner lgbm
[flaml.automl.logger: 05-07 17:17:29] {2442} INFO - at 7.5s, estimator lgbm's best error=0.4512, best estimator lgbm's best error=0.4512
[flaml.automl.logger: 05-07 17:17:29] {2258} INFO - iteration 44, current learner extra_tree
[flaml.automl.logger: 05-07 17:17:30] {2442} INFO - at 7.7s, estimator extra_tree's best error=0.4713, best estimator lgbm's best error=0.4512
[flaml.automl.logger: 05-07 17:17:30] {2258} INFO - iteration 45, current learner sgd
[flaml.automl.logger: 05-07 17:17:30] {2442} INFO - at 8.1s, estimator sgd's best error=0.5071, best estimator lgbm's best error=0.4512
[flaml.automl.logger: 05-07 17:17:30] {2258} INFO - iteration 46, current learner extra_tree
[flaml.automl.logger: 05-07 17:17:30] {2442} INFO - at 8.3s, estimator extra_tree's best error=0.4713, best estimator lgbm's best error=0.4512
[flaml.automl.logger: 05-07 17:17:30] {2258} INFO - iteration 47, current learner lgbm
[flaml.automl.logger: 05-07 17:17:31] {2442} INFO - at 8.6s, estimator lgbm's best error=0.4500, best estimator lgbm's best error=0.4500
[flaml.automl.logger: 05-07 17:17:31] {2258} INFO - iteration 48, current learner extra_tree
[flaml.automl.logger: 05-07 17:17:31] {2442} INFO - at 8.8s, estimator extra_tree's best error=0.4713, best estimator lgbm's best error=0.4500
[flaml.automl.logger: 05-07 17:17:31] {2258} INFO - iteration 49, current learner lgbm
[flaml.automl.logger: 05-07 17:17:31] {2442} INFO - at 9.5s, estimator lgbm's best error=0.4500, best estimator lgbm's best error=0.4500
[flaml.automl.logger: 05-07 17:17:31] {2258} INFO - iteration 50, current learner rf
[flaml.automl.logger: 05-07 17:17:32] {2442} INFO - at 9.8s, estimator rf's best error=0.4592, best estimator lgbm's best error=0.4500
[flaml.automl.logger: 05-07 17:17:32] {2258} INFO - iteration 51, current learner sgd
[flaml.automl.logger: 05-07 17:17:32] {2442} INFO - at 9.8s, estimator sgd's best error=0.5071, best estimator lgbm's best error=0.4500
[flaml.automl.logger: 05-07 17:17:32] {2258} INFO - iteration 52, current learner rf
[flaml.automl.logger: 05-07 17:17:32] {2442} INFO - at 10.0s, estimator rf's best error=0.4592, best estimator lgbm's best error=0.4500
[flaml.automl.logger: 05-07 17:17:32] {2258} INFO - iteration 53, current learner lgbm
[flaml.automl.logger: 05-07 17:17:33] {2442} INFO - at 10.8s, estimator lgbm's best error=0.4500, best estimator lgbm's best error=0.4500
[flaml.automl.logger: 05-07 17:17:33] {2258} INFO - iteration 54, current learner lgbm
[flaml.automl.logger: 05-07 17:17:33] {2442} INFO - at 11.1s, estimator lgbm's best error=0.4500, best estimator lgbm's best error=0.4500
[flaml.automl.logger: 05-07 17:17:33] {2258} INFO - iteration 55, current learner lgbm
[flaml.automl.logger: 05-07 17:17:34] {2442} INFO - at 11.8s, estimator lgbm's best error=0.4500, best estimator lgbm's best error=0.4500
[flaml.automl.logger: 05-07 17:17:34] {2258} INFO - iteration 56, current learner xgboost
[flaml.automl.logger: 05-07 17:17:34] {2442} INFO - at 11.9s, estimator xgboost's best error=0.4723, best estimator lgbm's best error=0.4500
[flaml.automl.logger: 05-07 17:17:34] {2258} INFO - iteration 57, current learner xgboost
[flaml.automl.logger: 05-07 17:17:34] {2442} INFO - at 12.0s, estimator xgboost's best error=0.4723, best estimator lgbm's best error=0.4500
[flaml.automl.logger: 05-07 17:17:34] {2258} INFO - iteration 58, current learner xgboost
[flaml.automl.logger: 05-07 17:17:34] {2442} INFO - at 12.1s, estimator xgboost's best error=0.4723, best estimator lgbm's best error=0.4500
[flaml.automl.logger: 05-07 17:17:34] {2258} INFO - iteration 59, current learner xgboost
[flaml.automl.logger: 05-07 17:17:34] {2442} INFO - at 12.3s, estimator xgboost's best error=0.4723, best estimator lgbm's best error=0.4500
[flaml.automl.logger: 05-07 17:17:34] {2258} INFO - iteration 60, current learner lgbm
[flaml.automl.logger: 05-07 17:17:35] {2442} INFO - at 12.8s, estimator lgbm's best error=0.4500, best estimator lgbm's best error=0.4500
[flaml.automl.logger: 05-07 17:17:35] {2258} INFO - iteration 61, current learner xgboost
[flaml.automl.logger: 05-07 17:17:35] {2442} INFO - at 12.9s, estimator xgboost's best error=0.4723, best estimator lgbm's best error=0.4500
[flaml.automl.logger: 05-07 17:17:35] {2258} INFO - iteration 62, current learner rf
[flaml.automl.logger: 05-07 17:17:35] {2442} INFO - at 13.1s, estimator rf's best error=0.4592, best estimator lgbm's best error=0.4500
[flaml.automl.logger: 05-07 17:17:35] {2258} INFO - iteration 63, current learner xgboost
[flaml.automl.logger: 05-07 17:17:35] {2442} INFO - at 13.2s, estimator xgboost's best error=0.4723, best estimator lgbm's best error=0.4500
[flaml.automl.logger: 05-07 17:17:35] {2258} INFO - iteration 64, current learner xgboost
[flaml.automl.logger: 05-07 17:17:35] {2442} INFO - at 13.4s, estimator xgboost's best error=0.4689, best estimator lgbm's best error=0.4500
[flaml.automl.logger: 05-07 17:17:35] {2258} INFO - iteration 65, current learner lgbm
[flaml.automl.logger: 05-07 17:17:36] {2442} INFO - at 13.6s, estimator lgbm's best error=0.4500, best estimator lgbm's best error=0.4500
[flaml.automl.logger: 05-07 17:17:36] {2258} INFO - iteration 66, current learner rf
[flaml.automl.logger: 05-07 17:17:36] {2442} INFO - at 13.8s, estimator rf's best error=0.4592, best estimator lgbm's best error=0.4500
[flaml.automl.logger: 05-07 17:17:36] {2258} INFO - iteration 67, current learner extra_tree
[flaml.automl.logger: 05-07 17:17:36] {2442} INFO - at 14.0s, estimator extra_tree's best error=0.4713, best estimator lgbm's best error=0.4500
[flaml.automl.logger: 05-07 17:17:36] {2258} INFO - iteration 68, current learner rf
[flaml.automl.logger: 05-07 17:17:36] {2442} INFO - at 14.2s, estimator rf's best error=0.4592, best estimator lgbm's best error=0.4500
[flaml.automl.logger: 05-07 17:17:36] {2258} INFO - iteration 69, current learner xgb_limitdepth
[flaml.automl.logger: 05-07 17:17:36] {2442} INFO - at 14.4s, estimator xgb_limitdepth's best error=0.4629, best estimator lgbm's best error=0.4500
[flaml.automl.logger: 05-07 17:17:36] {2258} INFO - iteration 70, current learner rf
[flaml.automl.logger: 05-07 17:17:37] {2442} INFO - at 14.6s, estimator rf's best error=0.4460, best estimator rf's best error=0.4460
[flaml.automl.logger: 05-07 17:17:37] {2258} INFO - iteration 71, current learner xgb_limitdepth
[flaml.automl.logger: 05-07 17:17:37] {2442} INFO - at 14.8s, estimator xgb_limitdepth's best error=0.4629, best estimator rf's best error=0.4460
[flaml.automl.logger: 05-07 17:17:37] {2258} INFO - iteration 72, current learner xgb_limitdepth
[flaml.automl.logger: 05-07 17:17:37] {2442} INFO - at 14.9s, estimator xgb_limitdepth's best error=0.4629, best estimator rf's best error=0.4460
[flaml.automl.logger: 05-07 17:17:37] {2258} INFO - iteration 73, current learner sgd
[flaml.automl.logger: 05-07 17:17:37] {2442} INFO - at 15.3s, estimator sgd's best error=0.5071, best estimator rf's best error=0.4460
[flaml.automl.logger: 05-07 17:17:37] {2685} INFO - retrain rf for 0.0s
[flaml.automl.logger: 05-07 17:17:37] {2688} INFO - retrained model: RandomForestRegressor(max_features=0.6599465711768816, max_leaf_nodes=184,
n_estimators=34, n_jobs=-1, random_state=12032022)
[flaml.automl.logger: 05-07 17:17:37] {1985} INFO - fit succeeded
[flaml.automl.logger: 05-07 17:17:37] {1986} INFO - Time taken to find the best model: 14.640347957611084
[abalone] R² Score: 0.5436
[abalone] Best estimator: rf
[abalone] Best config: {'n_estimators': 34, 'max_features': 0.6599465711768816, 'max_leaves': 184}
[abalone] Best validation loss: 0.4460
[abalone] Trials with val_loss, learner, runtime, sample_size and hyperparams saved.
Out[9]:
| iteration | val_loss | learner | runtime | sample_size | hyperparams | |
|---|---|---|---|---|---|---|
| 10 | 10.0 | 0.445999 | rf | 0.281269 | 3132.0 | {'n_estimators': 34, 'max_features': 0.6599465... |
| 9 | 9.0 | 0.450047 | lgbm | 0.342309 | 3132.0 | {'n_estimators': 79, 'num_leaves': 4, 'min_chi... |
| 8 | 8.0 | 0.451225 | lgbm | 0.654422 | 3132.0 | {'n_estimators': 54, 'num_leaves': 13, 'min_ch... |
| 7 | 7.0 | 0.455054 | lgbm | 0.113279 | 3132.0 | {'n_estimators': 21, 'num_leaves': 4, 'min_chi... |
| 6 | 6.0 | 0.470249 | rf | 0.196618 | 3132.0 | {'n_estimators': 6, 'max_features': 0.77544484... |
| 5 | 5.0 | 0.484117 | rf | 0.189885 | 3132.0 | {'n_estimators': 7, 'max_features': 0.83996488... |
| 4 | 4.0 | 0.494909 | lgbm | 0.069021 | 3132.0 | {'n_estimators': 9, 'num_leaves': 4, 'min_chil... |
| 3 | 3.0 | 0.618633 | rf | 0.174744 | 3132.0 | {'n_estimators': 4, 'max_features': 1.0, 'max_... |
| 2 | 2.0 | 0.629099 | lgbm | 0.039539 | 3132.0 | {'n_estimators': 4, 'num_leaves': 4, 'min_chil... |
| 1 | 1.0 | 0.706939 | sgd | 0.244366 | 3132.0 | {'penalty': 'l2', 'alpha': 0.0001, 'l1_ratio':... |
| 0 | 0.0 | 0.788170 | lgbm | 0.041078 | 3132.0 | {'n_estimators': 4, 'num_leaves': 4, 'min_chil... |
| 11 | NaN | NaN | unknown | NaN | NaN | {} |
In [10]:
# Run FLAML on breast cancer dataset
# based on the eda the breast cancer dataset is best suited for classification since the target column is categorical
bc_results = run_flaml_automl(
df=datasets["breast_cancer"],
target_column="target",
task="classification",
time_budget=15,
dataset_name="breast_cancer",
)
bc_results
[flaml.automl.logger: 05-07 17:18:28] {1728} INFO - task = classification
[flaml.automl.logger: 05-07 17:18:28] {1739} INFO - Evaluation method: cv
[flaml.automl.logger: 05-07 17:18:28] {1838} INFO - Minimizing error metric: 1-roc_auc
[flaml.automl.logger: 05-07 17:18:28] {1955} INFO - List of ML learners in AutoML Run: ['lgbm', 'rf', 'xgboost', 'extra_tree', 'xgb_limitdepth', 'sgd', 'lrl1']
[flaml.automl.logger: 05-07 17:18:28] {2258} INFO - iteration 0, current learner lgbm
[flaml.automl.logger: 05-07 17:18:28] {2393} INFO - Estimated sufficient time budget=399s. Estimated necessary time budget=9s.
[flaml.automl.logger: 05-07 17:18:28] {2442} INFO - at 0.1s, estimator lgbm's best error=0.0361, best estimator lgbm's best error=0.0361
[flaml.automl.logger: 05-07 17:18:28] {2258} INFO - iteration 1, current learner lgbm
[flaml.automl.logger: 05-07 17:18:28] {2442} INFO - at 0.1s, estimator lgbm's best error=0.0262, best estimator lgbm's best error=0.0262
[flaml.automl.logger: 05-07 17:18:28] {2258} INFO - iteration 2, current learner lgbm
[flaml.automl.logger: 05-07 17:18:28] {2442} INFO - at 0.1s, estimator lgbm's best error=0.0262, best estimator lgbm's best error=0.0262
[flaml.automl.logger: 05-07 17:18:28] {2258} INFO - iteration 3, current learner lgbm
[flaml.automl.logger: 05-07 17:18:28] {2442} INFO - at 0.2s, estimator lgbm's best error=0.0192, best estimator lgbm's best error=0.0192
[flaml.automl.logger: 05-07 17:18:28] {2258} INFO - iteration 4, current learner sgd
[flaml.automl.logger: 05-07 17:18:28] {2442} INFO - at 0.2s, estimator sgd's best error=0.0636, best estimator lgbm's best error=0.0192
[flaml.automl.logger: 05-07 17:18:28] {2258} INFO - iteration 5, current learner lgbm
[flaml.automl.logger: 05-07 17:18:28] {2442} INFO - at 0.3s, estimator lgbm's best error=0.0192, best estimator lgbm's best error=0.0192
[flaml.automl.logger: 05-07 17:18:28] {2258} INFO - iteration 6, current learner lgbm
[flaml.automl.logger: 05-07 17:18:28] {2442} INFO - at 0.3s, estimator lgbm's best error=0.0192, best estimator lgbm's best error=0.0192
[flaml.automl.logger: 05-07 17:18:28] {2258} INFO - iteration 7, current learner lgbm
[flaml.automl.logger: 05-07 17:18:28] {2442} INFO - at 0.4s, estimator lgbm's best error=0.0192, best estimator lgbm's best error=0.0192
[flaml.automl.logger: 05-07 17:18:28] {2258} INFO - iteration 8, current learner lgbm
[flaml.automl.logger: 05-07 17:18:28] {2442} INFO - at 0.4s, estimator lgbm's best error=0.0192, best estimator lgbm's best error=0.0192
[flaml.automl.logger: 05-07 17:18:28] {2258} INFO - iteration 9, current learner sgd
[flaml.automl.logger: 05-07 17:18:29] {2442} INFO - at 0.6s, estimator sgd's best error=0.0595, best estimator lgbm's best error=0.0192
[flaml.automl.logger: 05-07 17:18:29] {2258} INFO - iteration 10, current learner xgboost
[flaml.automl.logger: 05-07 17:18:29] {2442} INFO - at 0.7s, estimator xgboost's best error=0.0430, best estimator lgbm's best error=0.0192
[flaml.automl.logger: 05-07 17:18:29] {2258} INFO - iteration 11, current learner extra_tree
[flaml.automl.logger: 05-07 17:18:29] {2442} INFO - at 0.8s, estimator extra_tree's best error=0.0367, best estimator lgbm's best error=0.0192
[flaml.automl.logger: 05-07 17:18:29] {2258} INFO - iteration 12, current learner rf
[flaml.automl.logger: 05-07 17:18:29] {2442} INFO - at 1.0s, estimator rf's best error=0.0301, best estimator lgbm's best error=0.0192
[flaml.automl.logger: 05-07 17:18:29] {2258} INFO - iteration 13, current learner xgboost
[flaml.automl.logger: 05-07 17:18:29] {2442} INFO - at 1.1s, estimator xgboost's best error=0.0430, best estimator lgbm's best error=0.0192
[flaml.automl.logger: 05-07 17:18:29] {2258} INFO - iteration 14, current learner xgboost
[flaml.automl.logger: 05-07 17:18:29] {2442} INFO - at 1.2s, estimator xgboost's best error=0.0319, best estimator lgbm's best error=0.0192
[flaml.automl.logger: 05-07 17:18:29] {2258} INFO - iteration 15, current learner rf
[flaml.automl.logger: 05-07 17:18:29] {2442} INFO - at 1.4s, estimator rf's best error=0.0301, best estimator lgbm's best error=0.0192
[flaml.automl.logger: 05-07 17:18:29] {2258} INFO - iteration 16, current learner extra_tree
[flaml.automl.logger: 05-07 17:18:30] {2442} INFO - at 1.5s, estimator extra_tree's best error=0.0259, best estimator lgbm's best error=0.0192
[flaml.automl.logger: 05-07 17:18:30] {2258} INFO - iteration 17, current learner lgbm
[flaml.automl.logger: 05-07 17:18:30] {2442} INFO - at 1.6s, estimator lgbm's best error=0.0192, best estimator lgbm's best error=0.0192
[flaml.automl.logger: 05-07 17:18:30] {2258} INFO - iteration 18, current learner extra_tree
[flaml.automl.logger: 05-07 17:18:30] {2442} INFO - at 1.8s, estimator extra_tree's best error=0.0259, best estimator lgbm's best error=0.0192
[flaml.automl.logger: 05-07 17:18:30] {2258} INFO - iteration 19, current learner lgbm
[flaml.automl.logger: 05-07 17:18:30] {2442} INFO - at 1.8s, estimator lgbm's best error=0.0191, best estimator lgbm's best error=0.0191
[flaml.automl.logger: 05-07 17:18:30] {2258} INFO - iteration 20, current learner xgboost
[flaml.automl.logger: 05-07 17:18:30] {2442} INFO - at 1.9s, estimator xgboost's best error=0.0138, best estimator xgboost's best error=0.0138
[flaml.automl.logger: 05-07 17:18:30] {2258} INFO - iteration 21, current learner xgboost
[flaml.automl.logger: 05-07 17:18:30] {2442} INFO - at 2.0s, estimator xgboost's best error=0.0138, best estimator xgboost's best error=0.0138
[flaml.automl.logger: 05-07 17:18:30] {2258} INFO - iteration 22, current learner xgboost
[flaml.automl.logger: 05-07 17:18:30] {2442} INFO - at 2.1s, estimator xgboost's best error=0.0138, best estimator xgboost's best error=0.0138
[flaml.automl.logger: 05-07 17:18:30] {2258} INFO - iteration 23, current learner rf
[flaml.automl.logger: 05-07 17:18:30] {2442} INFO - at 2.2s, estimator rf's best error=0.0301, best estimator xgboost's best error=0.0138
[flaml.automl.logger: 05-07 17:18:30] {2258} INFO - iteration 24, current learner xgboost
[flaml.automl.logger: 05-07 17:18:30] {2442} INFO - at 2.3s, estimator xgboost's best error=0.0138, best estimator xgboost's best error=0.0138
[flaml.automl.logger: 05-07 17:18:30] {2258} INFO - iteration 25, current learner extra_tree
[flaml.automl.logger: 05-07 17:18:31] {2442} INFO - at 2.5s, estimator extra_tree's best error=0.0219, best estimator xgboost's best error=0.0138
[flaml.automl.logger: 05-07 17:18:31] {2258} INFO - iteration 26, current learner sgd
[flaml.automl.logger: 05-07 17:18:31] {2442} INFO - at 2.6s, estimator sgd's best error=0.0571, best estimator xgboost's best error=0.0138
[flaml.automl.logger: 05-07 17:18:31] {2258} INFO - iteration 27, current learner xgboost
[flaml.automl.logger: 05-07 17:18:31] {2442} INFO - at 2.6s, estimator xgboost's best error=0.0138, best estimator xgboost's best error=0.0138
[flaml.automl.logger: 05-07 17:18:31] {2258} INFO - iteration 28, current learner xgboost
[flaml.automl.logger: 05-07 17:18:31] {2442} INFO - at 2.7s, estimator xgboost's best error=0.0138, best estimator xgboost's best error=0.0138
[flaml.automl.logger: 05-07 17:18:31] {2258} INFO - iteration 29, current learner sgd
[flaml.automl.logger: 05-07 17:18:31] {2442} INFO - at 2.8s, estimator sgd's best error=0.0571, best estimator xgboost's best error=0.0138
[flaml.automl.logger: 05-07 17:18:31] {2258} INFO - iteration 30, current learner extra_tree
[flaml.automl.logger: 05-07 17:18:31] {2442} INFO - at 2.9s, estimator extra_tree's best error=0.0219, best estimator xgboost's best error=0.0138
[flaml.automl.logger: 05-07 17:18:31] {2258} INFO - iteration 31, current learner rf
[flaml.automl.logger: 05-07 17:18:31] {2442} INFO - at 3.1s, estimator rf's best error=0.0269, best estimator xgboost's best error=0.0138
[flaml.automl.logger: 05-07 17:18:31] {2258} INFO - iteration 32, current learner rf
[flaml.automl.logger: 05-07 17:18:31] {2442} INFO - at 3.3s, estimator rf's best error=0.0269, best estimator xgboost's best error=0.0138
[flaml.automl.logger: 05-07 17:18:31] {2258} INFO - iteration 33, current learner xgboost
[flaml.automl.logger: 05-07 17:18:31] {2442} INFO - at 3.4s, estimator xgboost's best error=0.0138, best estimator xgboost's best error=0.0138
[flaml.automl.logger: 05-07 17:18:31] {2258} INFO - iteration 34, current learner sgd
[flaml.automl.logger: 05-07 17:18:32] {2442} INFO - at 3.5s, estimator sgd's best error=0.0571, best estimator xgboost's best error=0.0138
[flaml.automl.logger: 05-07 17:18:32] {2258} INFO - iteration 35, current learner xgboost
[flaml.automl.logger: 05-07 17:18:32] {2442} INFO - at 3.5s, estimator xgboost's best error=0.0138, best estimator xgboost's best error=0.0138
[flaml.automl.logger: 05-07 17:18:32] {2258} INFO - iteration 36, current learner xgboost
[flaml.automl.logger: 05-07 17:18:32] {2442} INFO - at 3.6s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:32] {2258} INFO - iteration 37, current learner sgd
[flaml.automl.logger: 05-07 17:18:32] {2442} INFO - at 3.6s, estimator sgd's best error=0.0571, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:32] {2258} INFO - iteration 38, current learner xgboost
[flaml.automl.logger: 05-07 17:18:32] {2442} INFO - at 3.7s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:32] {2258} INFO - iteration 39, current learner xgboost
[flaml.automl.logger: 05-07 17:18:32] {2442} INFO - at 3.8s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:32] {2258} INFO - iteration 40, current learner xgboost
[flaml.automl.logger: 05-07 17:18:32] {2442} INFO - at 3.9s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:32] {2258} INFO - iteration 41, current learner xgboost
[flaml.automl.logger: 05-07 17:18:32] {2442} INFO - at 4.0s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:32] {2258} INFO - iteration 42, current learner extra_tree
[flaml.automl.logger: 05-07 17:18:32] {2442} INFO - at 4.2s, estimator extra_tree's best error=0.0219, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:32] {2258} INFO - iteration 43, current learner rf
[flaml.automl.logger: 05-07 17:18:32] {2442} INFO - at 4.4s, estimator rf's best error=0.0202, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:32] {2258} INFO - iteration 44, current learner xgboost
[flaml.automl.logger: 05-07 17:18:33] {2442} INFO - at 4.5s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:33] {2258} INFO - iteration 45, current learner sgd
[flaml.automl.logger: 05-07 17:18:33] {2442} INFO - at 4.5s, estimator sgd's best error=0.0571, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:33] {2258} INFO - iteration 46, current learner xgboost
[flaml.automl.logger: 05-07 17:18:33] {2442} INFO - at 4.6s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:33] {2258} INFO - iteration 47, current learner rf
[flaml.automl.logger: 05-07 17:18:33] {2442} INFO - at 4.8s, estimator rf's best error=0.0202, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:33] {2258} INFO - iteration 48, current learner xgboost
[flaml.automl.logger: 05-07 17:18:33] {2442} INFO - at 4.9s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:33] {2258} INFO - iteration 49, current learner rf
[flaml.automl.logger: 05-07 17:18:33] {2442} INFO - at 5.1s, estimator rf's best error=0.0157, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:33] {2258} INFO - iteration 50, current learner xgboost
[flaml.automl.logger: 05-07 17:18:33] {2442} INFO - at 5.2s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:33] {2258} INFO - iteration 51, current learner extra_tree
[flaml.automl.logger: 05-07 17:18:33] {2442} INFO - at 5.4s, estimator extra_tree's best error=0.0219, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:33] {2258} INFO - iteration 52, current learner rf
[flaml.automl.logger: 05-07 17:18:34] {2442} INFO - at 5.6s, estimator rf's best error=0.0157, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:34] {2258} INFO - iteration 53, current learner xgboost
[flaml.automl.logger: 05-07 17:18:34] {2442} INFO - at 5.7s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:34] {2258} INFO - iteration 54, current learner rf
[flaml.automl.logger: 05-07 17:18:34] {2442} INFO - at 6.0s, estimator rf's best error=0.0157, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:34] {2258} INFO - iteration 55, current learner rf
[flaml.automl.logger: 05-07 17:18:34] {2442} INFO - at 6.2s, estimator rf's best error=0.0157, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:34] {2258} INFO - iteration 56, current learner xgboost
[flaml.automl.logger: 05-07 17:18:34] {2442} INFO - at 6.3s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:34] {2258} INFO - iteration 57, current learner rf
[flaml.automl.logger: 05-07 17:18:35] {2442} INFO - at 6.5s, estimator rf's best error=0.0157, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:35] {2258} INFO - iteration 58, current learner xgboost
[flaml.automl.logger: 05-07 17:18:35] {2442} INFO - at 6.6s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:35] {2258} INFO - iteration 59, current learner xgboost
[flaml.automl.logger: 05-07 17:18:35] {2442} INFO - at 6.7s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:35] {2258} INFO - iteration 60, current learner rf
[flaml.automl.logger: 05-07 17:18:35] {2442} INFO - at 7.0s, estimator rf's best error=0.0139, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:35] {2258} INFO - iteration 61, current learner xgboost
[flaml.automl.logger: 05-07 17:18:35] {2442} INFO - at 7.1s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:35] {2258} INFO - iteration 62, current learner rf
[flaml.automl.logger: 05-07 17:18:35] {2442} INFO - at 7.3s, estimator rf's best error=0.0139, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:35] {2258} INFO - iteration 63, current learner xgboost
[flaml.automl.logger: 05-07 17:18:35] {2442} INFO - at 7.4s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:35] {2258} INFO - iteration 64, current learner xgboost
[flaml.automl.logger: 05-07 17:18:36] {2442} INFO - at 7.5s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:36] {2258} INFO - iteration 65, current learner rf
[flaml.automl.logger: 05-07 17:18:36] {2442} INFO - at 7.9s, estimator rf's best error=0.0139, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:36] {2258} INFO - iteration 66, current learner xgboost
[flaml.automl.logger: 05-07 17:18:36] {2442} INFO - at 7.9s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:36] {2258} INFO - iteration 67, current learner extra_tree
[flaml.automl.logger: 05-07 17:18:36] {2442} INFO - at 8.1s, estimator extra_tree's best error=0.0195, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:36] {2258} INFO - iteration 68, current learner xgboost
[flaml.automl.logger: 05-07 17:18:36] {2442} INFO - at 8.2s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:36] {2258} INFO - iteration 69, current learner rf
[flaml.automl.logger: 05-07 17:18:37] {2442} INFO - at 8.5s, estimator rf's best error=0.0139, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:37] {2258} INFO - iteration 70, current learner xgboost
[flaml.automl.logger: 05-07 17:18:37] {2442} INFO - at 8.6s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:37] {2258} INFO - iteration 71, current learner xgboost
[flaml.automl.logger: 05-07 17:18:37] {2442} INFO - at 8.7s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:37] {2258} INFO - iteration 72, current learner xgboost
[flaml.automl.logger: 05-07 17:18:37] {2442} INFO - at 8.8s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:37] {2258} INFO - iteration 73, current learner sgd
[flaml.automl.logger: 05-07 17:18:37] {2442} INFO - at 8.8s, estimator sgd's best error=0.0571, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:37] {2258} INFO - iteration 74, current learner xgboost
[flaml.automl.logger: 05-07 17:18:37] {2442} INFO - at 8.9s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:37] {2258} INFO - iteration 75, current learner sgd
[flaml.automl.logger: 05-07 17:18:37] {2442} INFO - at 9.0s, estimator sgd's best error=0.0571, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:37] {2258} INFO - iteration 76, current learner rf
[flaml.automl.logger: 05-07 17:18:37] {2442} INFO - at 9.3s, estimator rf's best error=0.0139, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:37] {2258} INFO - iteration 77, current learner xgboost
[flaml.automl.logger: 05-07 17:18:37] {2442} INFO - at 9.4s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:37] {2258} INFO - iteration 78, current learner extra_tree
[flaml.automl.logger: 05-07 17:18:38] {2442} INFO - at 9.6s, estimator extra_tree's best error=0.0195, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:38] {2258} INFO - iteration 79, current learner xgboost
[flaml.automl.logger: 05-07 17:18:38] {2442} INFO - at 9.6s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:38] {2258} INFO - iteration 80, current learner xgboost
[flaml.automl.logger: 05-07 17:18:38] {2442} INFO - at 9.8s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:38] {2258} INFO - iteration 81, current learner sgd
[flaml.automl.logger: 05-07 17:18:38] {2442} INFO - at 9.8s, estimator sgd's best error=0.0527, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:38] {2258} INFO - iteration 82, current learner xgboost
[flaml.automl.logger: 05-07 17:18:38] {2442} INFO - at 9.9s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:38] {2258} INFO - iteration 83, current learner sgd
[flaml.automl.logger: 05-07 17:18:38] {2442} INFO - at 9.9s, estimator sgd's best error=0.0527, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:38] {2258} INFO - iteration 84, current learner extra_tree
[flaml.automl.logger: 05-07 17:18:38] {2442} INFO - at 10.1s, estimator extra_tree's best error=0.0195, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:38] {2258} INFO - iteration 85, current learner sgd
[flaml.automl.logger: 05-07 17:18:38] {2442} INFO - at 10.2s, estimator sgd's best error=0.0527, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:38] {2258} INFO - iteration 86, current learner xgboost
[flaml.automl.logger: 05-07 17:18:38] {2442} INFO - at 10.3s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:38] {2258} INFO - iteration 87, current learner extra_tree
[flaml.automl.logger: 05-07 17:18:39] {2442} INFO - at 10.5s, estimator extra_tree's best error=0.0178, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:39] {2258} INFO - iteration 88, current learner xgboost
[flaml.automl.logger: 05-07 17:18:39] {2442} INFO - at 10.5s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:39] {2258} INFO - iteration 89, current learner xgboost
[flaml.automl.logger: 05-07 17:18:39] {2442} INFO - at 10.6s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:39] {2258} INFO - iteration 90, current learner xgb_limitdepth
[flaml.automl.logger: 05-07 17:18:39] {2442} INFO - at 10.8s, estimator xgb_limitdepth's best error=0.0143, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:39] {2258} INFO - iteration 91, current learner xgb_limitdepth
[flaml.automl.logger: 05-07 17:18:39] {2442} INFO - at 10.9s, estimator xgb_limitdepth's best error=0.0143, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:39] {2258} INFO - iteration 92, current learner xgb_limitdepth
[flaml.automl.logger: 05-07 17:18:39] {2442} INFO - at 11.0s, estimator xgb_limitdepth's best error=0.0128, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:39] {2258} INFO - iteration 93, current learner sgd
[flaml.automl.logger: 05-07 17:18:39] {2442} INFO - at 11.1s, estimator sgd's best error=0.0527, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:39] {2258} INFO - iteration 94, current learner xgb_limitdepth
[flaml.automl.logger: 05-07 17:18:39] {2442} INFO - at 11.2s, estimator xgb_limitdepth's best error=0.0128, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:39] {2258} INFO - iteration 95, current learner xgb_limitdepth
[flaml.automl.logger: 05-07 17:18:39] {2442} INFO - at 11.4s, estimator xgb_limitdepth's best error=0.0128, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:39] {2258} INFO - iteration 96, current learner xgboost
[flaml.automl.logger: 05-07 17:18:40] {2442} INFO - at 11.4s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:40] {2258} INFO - iteration 97, current learner sgd
[flaml.automl.logger: 05-07 17:18:40] {2442} INFO - at 11.5s, estimator sgd's best error=0.0524, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:40] {2258} INFO - iteration 98, current learner extra_tree
[flaml.automl.logger: 05-07 17:18:40] {2442} INFO - at 11.7s, estimator extra_tree's best error=0.0172, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:40] {2258} INFO - iteration 99, current learner xgb_limitdepth
[flaml.automl.logger: 05-07 17:18:40] {2442} INFO - at 11.8s, estimator xgb_limitdepth's best error=0.0128, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:40] {2258} INFO - iteration 100, current learner xgboost
[flaml.automl.logger: 05-07 17:18:40] {2442} INFO - at 11.9s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:40] {2258} INFO - iteration 101, current learner xgboost
[flaml.automl.logger: 05-07 17:18:40] {2442} INFO - at 12.1s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:40] {2258} INFO - iteration 102, current learner xgb_limitdepth
[flaml.automl.logger: 05-07 17:18:40] {2442} INFO - at 12.2s, estimator xgb_limitdepth's best error=0.0109, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:40] {2258} INFO - iteration 103, current learner xgb_limitdepth
[flaml.automl.logger: 05-07 17:18:40] {2442} INFO - at 12.4s, estimator xgb_limitdepth's best error=0.0109, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:40] {2258} INFO - iteration 104, current learner lgbm
[flaml.automl.logger: 05-07 17:18:40] {2442} INFO - at 12.4s, estimator lgbm's best error=0.0178, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:40] {2258} INFO - iteration 105, current learner xgb_limitdepth
[flaml.automl.logger: 05-07 17:18:41] {2442} INFO - at 12.6s, estimator xgb_limitdepth's best error=0.0109, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:41] {2258} INFO - iteration 106, current learner lgbm
[flaml.automl.logger: 05-07 17:18:41] {2442} INFO - at 12.7s, estimator lgbm's best error=0.0161, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:41] {2258} INFO - iteration 107, current learner xgboost
[flaml.automl.logger: 05-07 17:18:41] {2442} INFO - at 12.8s, estimator xgboost's best error=0.0105, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:41] {2258} INFO - iteration 108, current learner xgb_limitdepth
[flaml.automl.logger: 05-07 17:18:41] {2442} INFO - at 12.9s, estimator xgb_limitdepth's best error=0.0109, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:41] {2258} INFO - iteration 109, current learner lgbm
[flaml.automl.logger: 05-07 17:18:41] {2442} INFO - at 13.0s, estimator lgbm's best error=0.0161, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:41] {2258} INFO - iteration 110, current learner xgb_limitdepth
[flaml.automl.logger: 05-07 17:18:41] {2442} INFO - at 13.1s, estimator xgb_limitdepth's best error=0.0109, best estimator xgboost's best error=0.0105
[flaml.automl.logger: 05-07 17:18:41] {2258} INFO - iteration 111, current learner xgb_limitdepth
[flaml.automl.logger: 05-07 17:18:41] {2442} INFO - at 13.3s, estimator xgb_limitdepth's best error=0.0078, best estimator xgb_limitdepth's best error=0.0078
[flaml.automl.logger: 05-07 17:18:41] {2258} INFO - iteration 112, current learner lgbm
[flaml.automl.logger: 05-07 17:18:41] {2442} INFO - at 13.3s, estimator lgbm's best error=0.0095, best estimator xgb_limitdepth's best error=0.0078
[flaml.automl.logger: 05-07 17:18:41] {2258} INFO - iteration 113, current learner xgb_limitdepth
[flaml.automl.logger: 05-07 17:18:42] {2442} INFO - at 13.5s, estimator xgb_limitdepth's best error=0.0078, best estimator xgb_limitdepth's best error=0.0078
[flaml.automl.logger: 05-07 17:18:42] {2258} INFO - iteration 114, current learner lgbm
[flaml.automl.logger: 05-07 17:18:42] {2442} INFO - at 13.6s, estimator lgbm's best error=0.0095, best estimator xgb_limitdepth's best error=0.0078
[flaml.automl.logger: 05-07 17:18:42] {2258} INFO - iteration 115, current learner sgd
[flaml.automl.logger: 05-07 17:18:42] {2442} INFO - at 13.6s, estimator sgd's best error=0.0524, best estimator xgb_limitdepth's best error=0.0078
[flaml.automl.logger: 05-07 17:18:42] {2258} INFO - iteration 116, current learner lgbm
[flaml.automl.logger: 05-07 17:18:42] {2442} INFO - at 13.7s, estimator lgbm's best error=0.0068, best estimator lgbm's best error=0.0068
[flaml.automl.logger: 05-07 17:18:42] {2258} INFO - iteration 117, current learner lgbm
[flaml.automl.logger: 05-07 17:18:42] {2442} INFO - at 13.8s, estimator lgbm's best error=0.0068, best estimator lgbm's best error=0.0068
[flaml.automl.logger: 05-07 17:18:42] {2258} INFO - iteration 118, current learner xgb_limitdepth
[flaml.automl.logger: 05-07 17:18:42] {2442} INFO - at 13.9s, estimator xgb_limitdepth's best error=0.0078, best estimator lgbm's best error=0.0068
[flaml.automl.logger: 05-07 17:18:42] {2258} INFO - iteration 119, current learner lgbm
[flaml.automl.logger: 05-07 17:18:42] {2442} INFO - at 14.0s, estimator lgbm's best error=0.0068, best estimator lgbm's best error=0.0068
[flaml.automl.logger: 05-07 17:18:42] {2258} INFO - iteration 120, current learner lgbm
[flaml.automl.logger: 05-07 17:18:42] {2442} INFO - at 14.1s, estimator lgbm's best error=0.0068, best estimator lgbm's best error=0.0068
[flaml.automl.logger: 05-07 17:18:42] {2258} INFO - iteration 121, current learner extra_tree
[flaml.automl.logger: 05-07 17:18:42] {2442} INFO - at 14.3s, estimator extra_tree's best error=0.0169, best estimator lgbm's best error=0.0068
[flaml.automl.logger: 05-07 17:18:42] {2258} INFO - iteration 122, current learner extra_tree
[flaml.automl.logger: 05-07 17:18:43] {2442} INFO - at 14.5s, estimator extra_tree's best error=0.0169, best estimator lgbm's best error=0.0068
[flaml.automl.logger: 05-07 17:18:43] {2258} INFO - iteration 123, current learner xgb_limitdepth
[flaml.automl.logger: 05-07 17:18:43] {2442} INFO - at 14.7s, estimator xgb_limitdepth's best error=0.0078, best estimator lgbm's best error=0.0068
[flaml.automl.logger: 05-07 17:18:43] {2258} INFO - iteration 124, current learner lgbm
[flaml.automl.logger: 05-07 17:18:43] {2442} INFO - at 14.8s, estimator lgbm's best error=0.0068, best estimator lgbm's best error=0.0068
[flaml.automl.logger: 05-07 17:18:43] {2258} INFO - iteration 125, current learner lgbm
[flaml.automl.logger: 05-07 17:18:43] {2442} INFO - at 14.9s, estimator lgbm's best error=0.0068, best estimator lgbm's best error=0.0068
[flaml.automl.logger: 05-07 17:18:43] {2258} INFO - iteration 126, current learner lrl1
[flaml.automl.logger: 05-07 17:18:43] {2442} INFO - at 15.0s, estimator lrl1's best error=0.0824, best estimator lgbm's best error=0.0068
[flaml.automl.logger: 05-07 17:18:43] {2685} INFO - retrain lgbm for 0.0s
[flaml.automl.logger: 05-07 17:18:43] {2688} INFO - retrained model: LGBMClassifier(colsample_bytree=0.82914060366862, learning_rate=1.0,
max_bin=1023, min_child_samples=15, n_estimators=11, n_jobs=-1,
num_leaves=8, reg_alpha=0.0009765625,
reg_lambda=0.12572859056792066, verbose=-1)
[flaml.automl.logger: 05-07 17:18:43] {1985} INFO - fit succeeded
[flaml.automl.logger: 05-07 17:18:43] {1986} INFO - Time taken to find the best model: 13.727378129959106
[breast_cancer] Accuracy: 0.9580
[breast_cancer] Best estimator: lgbm
[breast_cancer] Best config: {'n_estimators': 11, 'num_leaves': 8, 'min_child_samples': 15, 'learning_rate': 1.0, 'log_max_bin': 10, 'colsample_bytree': 0.82914060366862, 'reg_alpha': 0.0009765625, 'reg_lambda': 0.12572859056792066}
[breast_cancer] Best validation loss: 0.0068
[breast_cancer] Trials with val_loss, learner, runtime, sample_size and hyperparams saved.
Out[10]:
| iteration | val_loss | learner | runtime | sample_size | hyperparams | |
|---|---|---|---|---|---|---|
| 7 | 7.0 | 0.006834 | lgbm | 0.103078 | 426.0 | {'n_estimators': 11, 'num_leaves': 8, 'min_chi... |
| 6 | 6.0 | 0.007835 | xgb_limitdepth | 0.148194 | 426.0 | {'n_estimators': 12, 'max_depth': 6, 'min_chil... |
| 5 | 5.0 | 0.010476 | xgboost | 0.075668 | 426.0 | {'n_estimators': 4, 'max_leaves': 4, 'min_chil... |
| 4 | 4.0 | 0.013770 | xgboost | 0.075475 | 426.0 | {'n_estimators': 4, 'max_leaves': 4, 'min_chil... |
| 3 | 3.0 | 0.019136 | lgbm | 0.041527 | 426.0 | {'n_estimators': 4, 'num_leaves': 4, 'min_chil... |
| 2 | 2.0 | 0.019180 | lgbm | 0.045569 | 426.0 | {'n_estimators': 7, 'num_leaves': 4, 'min_chil... |
| 1 | 1.0 | 0.026202 | lgbm | 0.036277 | 426.0 | {'n_estimators': 4, 'num_leaves': 4, 'min_chil... |
| 0 | 0.0 | 0.036109 | lgbm | 0.039420 | 426.0 | {'n_estimators': 4, 'num_leaves': 4, 'min_chil... |
| 8 | NaN | NaN | unknown | NaN | NaN | {} |