The content for this site available on GitHub. If you want to launch the notebooks interactively click on the binder stamp below. 
< EDA Wine Quality Prediction continuous | Contents | Experiment Decision Tree Classifier on Iris dataset with postpruning >
Experiment: Adaboost on iris dataset¶
🔍 Quick Summary: AdaBoost (Adaptive Boosting)¶
Goal: Combine many weak learners (e.g., decision stumps) into a strong classifier.
Note: DTs try to solve the entire problem in one shot while adaboost builds a team of simple trees each one improving the last where the final prediciton is a weighted vote.
🧠 Core Idea:¶
- Train weak learners sequentially, each focusing more on examples the previous ones got wrong.
- Weight the predictions of each learner based on its accuracy—better learners get more say in the final vote.
🧮 Algorithm Steps (with Tunable Hyperparameters):¶
- Initialize weights on all training examples equally.
- Repeat for T rounds (T =
n_estimators):- Train a weak learner (usually
DecisionTreeClassifier(max_depth=1)— can tunebase_estimator, e.g., tree depth, or use a different model). - Compute the learner’s error on weighted data.
- Calculate its weight in the final ensemble.
- Update the weights on training examples (affected by
learning_rate).
- Train a weak learner (usually
- Final prediction = weighted majority vote of all learners.
🔧 Key Hyperparameters to Tune:¶
n_estimators— Number of boosting rounds (T)learning_rate— Shrinks contribution of each learner (default = 1.0)base_estimator— The weak learner model (commonly a decision stump)- If using a tree: tune
max_depth,min_samples_split, etc.
- If using a tree: tune
🧪 Expected Performance on the Iris Dataset¶
| Aspect | Expectation | Notes |
|---|---|---|
| Accuracy | High (~95–100%) | Clean and well-separated dataset. |
| Overfitting | Unlikely | Noisy data is minimal, and the feature space is simple. |
| Training Time | Very Fast | Small dataset (150 samples); training takes milliseconds. |
| Inference Time | Instantaneous | Just a few decision stumps to evaluate. |
| Best Base Learner | Decision stump (max_depth=1) |
Simple learners work well for AdaBoost on small datasets. |
🔑 Characteristics:¶
- Fast and simple.
- Works well on tabular data.
- Can overfit on noisy datasets.
- Sensitive to outliers because it keeps boosting hard examples.
In [1]:
%reload_ext autoreload
%autoreload 2
In [9]:
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier, export_text, plot_tree
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
# experiment helper imports
from sklearn.base import clone
from helpers.base_imports import *
Setup experiment with data and model¶
In [3]:
exp = Experiment(
type="c", # classification
name="adaboost-iris",
dataset="iris-20test-shuffled-v1",
)
exp
Out[3]:
We'll start with the regular unpruned DT then check the values of ccp_alpha with a validation curve to see if we can find a good value for pruning.
In [ ]:
RANDOM_SEED = 42
CACHE_DIR = None # or use a path if you want caching
# Define weak learner (decision stump)
base_stump = DecisionTreeClassifier(
criterion="entropy",
splitter="best",
max_depth=1, # shallow tree = decision stump
min_samples_split=2,
min_samples_leaf=1,
min_weight_fraction_leaf=0.0,
max_features=None,
random_state=RANDOM_SEED,
max_leaf_nodes=None,
min_impurity_decrease=0.0,
class_weight=None,
ccp_alpha=0.0,
)
# Build pipeline
steps = [
# NOTE: Scaling not strictly needed for trees, but good for consistency across models
("scaler", StandardScaler()),
(
"classifier",
AdaBoostClassifier(
estimator=base_stump,
n_estimators=50, # heuristic: start with 50, increase to 100-200 if underfitting
learning_rate=1.0, # typical range is .01 to 1.0 (heuristic: start with 1, try smaller values with more estimators since lower learning rates need higher estimators to compensate)
random_state=RANDOM_SEED,
),
),
]
exp.estimator = Pipeline(
steps=steps,
memory=CACHE_DIR,
)
In [6]:
exp.estimator.get_params()
Out[6]:
Get dataset by name (eda already done in another notebook and train/test split saved so we will be working with the same data)
In [7]:
notes, X_train, X_test, y_train, y_test, target_names = get_dataset(exp.dataset)
print(notes)
print(target_names)
X_train.shape, X_test.shape, y_train.shape, y_test.shape
Out[7]:
In [8]:
exp.update_param("n_train_samples", X_train.shape[0])
exp.update_param("n_test_samples", X_test.shape[0])
exp.summary_df
Out[8]:
Find optimal hyperparameters¶
What cv should we use?
In [11]:
cv_values = [3, 5]
for cv in cv_values:
print(f"\nRunning GridSearchCV with cv={cv}")
grid_search = GridSearchCV(
estimator=exp.estimator,
param_grid={
"classifier__n_estimators": [50, 100, 200],
"classifier__learning_rate": [1.0, 0.5, 0.1],
},
cv=cv,
scoring="accuracy",
n_jobs=-1,
)
grid_search.fit(X_train, y_train)
print("Best params:", grid_search.best_params_)
print("Best score:", grid_search.best_score_)
cv=3 does better so we'll stick with that and .1 learning rate with 100 estimators.
Train¶
In [12]:
cv = 3
my_params = {
"classifier__n_estimators": 100,
"classifier__learning_rate": 0.1,
}
exp.estimator.set_params(**my_params)
exp.estimator.get_params()
Out[12]:
In [14]:
# fit on training data
start_time = pd.Timestamp.now()
exp.estimator.fit(X=X_train, y=y_train)
train_time = pd.Timestamp.now() - start_time
In [15]:
exp.update_param("train_time", train_time)
exp.update_param(
"mean_accuracy",
exp.estimator.score(X_test, y_test),
# add_column=True
)
exp.summary_df
Out[15]:
Take a look at the trained model¶
In [19]:
exp.estimator.named_steps["classifier"]
Out[19]:
In [17]:
# convert target names series to list
target_names_list = target_names["target_names"].tolist()
target_names_list
Out[17]:
In [21]:
ada_model = exp.estimator.named_steps["classifier"]
n_trees = len(ada_model.estimators_)
print(f"AdaBoost used {n_trees} trees")
Take a look at each tree and how it was weighted in the final model.
In [22]:
output_lines = [f"AdaBoost used {n_trees} trees\n"]
for i, stump in enumerate(ada_model.estimators_):
tree_header = (
f"\n=== Tree {i + 1} (Weight = {ada_model.estimator_weights_[i]:.3f}) ==="
)
tree_text = export_text(stump, feature_names=X_train.columns)
print(tree_header)
print(tree_text)
output_lines.append(tree_header)
output_lines.append(tree_text + "\n")
# Save to file
with open(f"{RES_DIR}/{exp.name}-adaboost_trees.txt", "w") as f:
f.write("\n".join(output_lines))
In [20]:
for name, importance in zip(
X_train.columns, exp.estimator.named_steps["classifier"].feature_importances_
):
print(f"{name}: {importance:.3f}")
In [ ]:
exp.save(overwrite_existing=True)
< EDA Wine Quality Prediction continuous | Contents | Experiment Decision Tree Classifier on Iris dataset with postpruning >