Learn AI Series (#129) - AutoML and Neural Architecture Search

avatar

Learn AI Series (#129) - AutoML and Neural Architecture Search

variant-b-03-red.png

What will I learn

  • You will learn hyperparameter optimization properly -- grid search, random search, and why random search quietly beats the grid in almost every real setting;
  • Bayesian optimization, the trick of building a little probabilistic model of your own score landscape so you stop guessing and start aiming (this is what Optuna does under the hood);
  • Neural Architecture Search (NAS) -- the genuinely wild idea of letting the machine design the model itself, not just tune its knobs;
  • DARTS, the differentiable-architecture-search insight that dragged NAS down from 800 GPUs to a single card by making architecture choices something you can do gradient descent on;
  • the practical AutoML frameworks (FLAML, Auto-sklearn, Ray Tune) that, for tabular data, are frankly all most people ever need;
  • and the honest part -- when AutoML earns its compute and when it is an expensive way to reinvent XGBoost.

Requirements

  • A working modern computer running macOS, Windows or Ubuntu;
  • An installed Python 3(.10+) distribution with scikit-learn and PyTorch (pip install scikit-learn torch), plus pip install optuna flaml if you want to run the practical frameworks for real -- but as always you do NOT need a GPU to understand any of this, every idea here fits on a laptop;
  • You've been through episode #16 (Scikit-Learn), episode #44 (PyTorch nn.Module, where we first built real networks by hand), and especially episode #119 (experiment tracking) -- today is the episode where we stop tuning by hand and hand the tuning itself to the machine.

Difficulty

  • Beginner

Curriculum (of the Learn AI Series):

Learn AI Series (#129) - AutoML and Neural Architecture Search

I ended episode #128 with a slightly mischievous promise. We had just spent a whole episode teaching humans to do the hard design work by hand -- clipping gradients, budgeting epsilon, deciding which privacy tool fit which threat -- and I asked the obvious next question: what if we handed the design work itself to the machine? For the entire series so far, every architectural decision has come out of a human head. How many layers. Which activation. What learning rate. How much dropout. Today we automate the meta-decisions -- the decisions about the model rather than inside it. That is what AutoML means, and its most ambitious wing, Neural Architecture Search, does not just tune the knobs, it designs the machine that has the knobs.

Here is the framing I want you to carry through. AutoML lives on a ladder. At the bottom rung it is hyperparameter tuning -- pick better numbers for a model you already chose. One rung up it does model selection -- pick the model too. At the very top, NAS designs the architecture from primitive parts, layer by layer. The promise at every rung is the same: less human sweat, better models. The reality, which I will not sugarcoat, is that it works beautifully for some problems and is an extravagant way to burn GPU-hours for others. By the end you will know which is which -- and that judgement, ironically, is the one thing AutoML cannot automate ;-)

Solutions to episode #128's exercises

As promised, let's clear last time's homework before we build anything new.

Exercise 1 -- feel the privacy-utility trade-off. The task was to privately release the mean of 1000 values in [0, 100], sweep epsilon, and watch the error move. The shape you are meant to feel is the inverse relationship: as epsilon shrinks, noise (and error) blows up.

import numpy as np

def gaussian_mechanism(true_value, sensitivity, epsilon, delta=1e-5):
    sigma = sensitivity * np.sqrt(2 * np.log(1.25 / delta)) / epsilon
    return true_value + np.random.normal(0, sigma)

data = np.random.uniform(0, 100, size=1000)
true_mean = data.mean()
sensitivity = 100 / len(data)          # one capped value shifts the mean by <= 100/n

for eps in [0.1, 0.5, 1.0, 5.0, 10.0]:
    errors = [abs(gaussian_mechanism(true_mean, sensitivity, eps) - true_mean)
              for _ in range(200)]
    print(f"epsilon={eps:>4}: mean abs error = {np.mean(errors):.4f}")

You should see the error fall roughly like 1/epsilon -- moving from epsilon=0.1 to epsilon=10 cuts the average error by about a factor of a hundred. The one-sentence description I was fishing for: the curve is a steep hyperbola, punishing you brutally for the strongest privacy and flattening out once epsilon is large enough that the noise barely matters. That steepness is exactly why choosing epsilon is a real decision and not a formality.

Exercise 2 -- train something real with Opacus. Train a small MNIST classifier twice, once plain and once wrapped with make_private. The point was to see the two numbers that matter: test accuracy and the epsilon you spent.

from opacus import PrivacyEngine

# ... build model, optimizer, data_loader as usual (see episode #43) ...

privacy_engine = PrivacyEngine()
model, optimizer, data_loader = privacy_engine.make_private(
    module=model, optimizer=optimizer, data_loader=data_loader,
    noise_multiplier=1.1, max_grad_norm=1.0,
)
# ... your ORDINARY training loop, unchanged ...
print(f"spent epsilon = {privacy_engine.get_epsilon(delta=1e-5):.2f}")

The lesson lands in the numbers. Plain training lands somewhere around 99% on MNIST; the private run at noise_multiplier=1.1 gives up a few points and reports an epsilon in the single digits. Crank the noise multiplier up and two things move together in opposite directions -- accuracy drops further, and epsilon drops too (more noise, stronger privacy, worse model). That coupled see-saw IS the whole subject of episode #128 in one experiment.

Exercise 3 -- reason about the right tool. No single correct mapping, only defensible reasoning. (a) Three banks, a shared fraud model, no data sharing and no trusted server -> secure multi-party computation, because the whole constraint is "trust nobody, not even the aggregator". (b) A research team publishing aggregate survey statistics -> differential privacy, because you are releasing numbers to the public and DP is the only tool that bounds what any single respondent leaks. (c) A cloud provider running inference for a hospital that must never see the scans -> homomorphic encryption, because the data must stay encrypted while being computed on by an untrusted party. If you justified why each tool matched that constraint, you nailed it.

Hyperparameter optimization: the foundation

Everything above the bottom rung is just a fancier version of the bottom rung, so let's start there. A hyperparameter is any setting you fix before training -- learning rate, tree depth, number of estimators, regularisation strength -- as opposed to the weights the model learns during training. The whole game of hyperparameter optimization (HPO) is finding the combination that scores best on held-out data. The two oldest strategies are grid search and random search:

import numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import GradientBoostingClassifier

def grid_search(param_grid, X, y):
    """Exhaustive search over all combinations. Simple, thorough, expensive."""
    best_score, best_params = -np.inf, None
    keys = list(param_grid.keys())
    grid = np.array(np.meshgrid(*param_grid.values())).T.reshape(-1, len(keys))

    for row in grid:
        params = {k: type(param_grid[k][0])(v) for k, v in zip(keys, row)}
        score = cross_val_score(GradientBoostingClassifier(**params), X, y, cv=5).mean()
        if score > best_score:
            best_score, best_params = score, params
    return best_params, best_score

def random_search(param_distributions, X, y, n_iter=50):
    """Random sampling -- usually smarter than the grid, for a reason worth knowing."""
    best_score, best_params = -np.inf, None
    for _ in range(n_iter):
        params = {k: np.random.choice(v) if isinstance(v, list) else v.rvs()
                  for k, v in param_distributions.items()}
        score = cross_val_score(GradientBoostingClassifier(**params), X, y, cv=5).mean()
        if score > best_score:
            best_score, best_params = score, params
    return best_params, best_score

Now here is the counter-intuitive bit that trips people up. Random search beats grid search in most realistic settings -- this is not folklore, it is the Bergstra and Bengio result from 2012. Why on earth would random beat systematic? Because in any real problem the hyperparameters are NOT equally important. Maybe learning rate matters enormously and the second momentum term barely moves the needle. Grid search, being systematic, spends the same number of trials probing the useless dimension as the crucial one -- it wastes evaluations laying down a neat lattice. Random search, by sampling all dimensions at once, effectively tries far more distinct values of the parameter that actually matters, for the same compute budget. It is one of those results that feels wrong for about ten seconds and then feels obvious forever.

Bayesian optimization: stop guessing, start aiming

Random search is smart, but it has amnesia -- it never learns from the trials it already ran. Each guess is independent of the last. Bayesian optimization fixes exactly that. It builds a little probabilistic model (usually a Gaussian Process, which we met in spirit back in episode #32 on Bayesian methods) of how score depends on hyperparameters, and uses that model to decide where to look next. It aims.

import numpy as np

class SimpleBayesianOptimizer:
    """Bayesian optimization with a Gaussian-Process surrogate (sketch)."""
    def __init__(self, param_bounds, n_initial=5):
        self.bounds = param_bounds
        self.X_observed, self.y_observed = [], []
        self.n_initial = n_initial

    def suggest(self):
        # Random exploration until we have enough points to fit a surrogate.
        if len(self.X_observed) < self.n_initial:
            return {k: np.random.uniform(v[0], v[1]) for k, v in self.bounds.items()}

        # Otherwise: fit the GP to observed (params, score) pairs, then pick the
        # candidate that MAXIMISES the acquisition function (Expected Improvement).
        best_y = max(self.y_observed)
        best_acq, best_x = -np.inf, None
        for _ in range(1000):
            cand = {k: np.random.uniform(v[0], v[1]) for k, v in self.bounds.items()}
            acq = self._expected_improvement(cand, best_y)   # EI under GP posterior
            if acq > best_acq:
                best_acq, best_x = acq, cand
        return best_x

    def observe(self, params, score):
        self.X_observed.append(params)
        self.y_observed.append(score)

    def _expected_improvement(self, candidate, best_y):
        return np.random.random()   # placeholder -- real EI uses the GP mean and std

The engine here is the acquisition function. Expected Improvement asks, for each candidate point, "how much do I expect to beat my current best if I evaluate here?" and it naturally balances two urges: exploiting regions the surrogate thinks are good, and exploring regions it is uncertain about (because uncertainty is where surprises live). In practice you never hand-roll the Gaussian Process -- you reach for Optuna or Ray Tune, which do the surrogate, the acquisition, the pruning of hopeless trials, and the bookkeeping for you:

# import optuna
# def objective(trial):
#     lr      = trial.suggest_float('lr', 1e-5, 1e-1, log=True)
#     hidden  = trial.suggest_int('hidden', 64, 512)
#     dropout = trial.suggest_float('dropout', 0.0, 0.5)
#     model = build_model(lr, hidden, dropout)
#     return evaluate(model)        # return the validation metric to maximise
#
# study = optuna.create_study(direction='maximize')
# study.optimize(objective, n_trials=100)
# print(study.best_params)

The pay-off is real: Bayesian optimization typically finds strong hyperparameters in 20 to 50 evaluations where random search might need 100+. When each evaluation means training a network for an hour, that difference is the difference between an afternoon and a week. Nota bene -- log the whole study to your experiment tracker (episode #119). A hyperparameter search you cannot reproduce is a rumour, not a result.

Neural Architecture Search: designing the model itself

Everything so far tunes a model you chose. Neural Architecture Search climbs to the top of the ladder and designs the architecture as well -- how the layers connect, which operation sits at each position, where the skip connections go. The first move is to define a search space: the menu of operations the search is allowed to pick from at each spot in the network.

import torch
import torch.nn as nn

class NASSearchSpace:
    """The menu of candidate operations at each position in the network."""
    operations = {
        'conv3x3':     lambda c: nn.Conv2d(c, c, 3, padding=1),
        'conv5x5':     lambda c: nn.Conv2d(c, c, 5, padding=2),
        'sep_conv3x3': lambda c: nn.Sequential(
            nn.Conv2d(c, c, 3, padding=1, groups=c),   # depthwise
            nn.Conv2d(c, c, 1),                         # pointwise
        ),
        'max_pool': lambda c: nn.MaxPool2d(3, stride=1, padding=1),
        'avg_pool': lambda c: nn.AvgPool2d(3, stride=1, padding=1),
        'skip':     lambda c: nn.Identity(),
        'none':     lambda c: Zero(c),                 # deliberately no connection
    }

class Zero(nn.Module):
    """The 'no connection' operation -- literally multiply by zero."""
    def __init__(self, channels):
        super().__init__()
        self.channels = channels
    def forward(self, x):
        return torch.zeros_like(x)

The none operation is the clever inclusion, by the way -- letting the search choose to have no connection at a spot is how it learns to prune the graph, not just fill it. Now, the original NAS papers (Zoph and Le, 2017) treated this as a reinforcement-learning problem: a controller network samples a full architecture, you train that architecture to completion to see how good it is, and you use the final accuracy as the reward signal to update the controller. It worked -- and it cost 800 GPUs running for weeks. Every candidate architecture meant a full training run from scratch. That is not a technique, that is a research budget. Which is exactly the wall the next idea knocked down.

DARTS: making architecture differentiable

The breakthrough in DARTS (Differentiable ARchiTecture Search, Liu et al. 2018) is almost cheeky in its simplicity. Discrete choice -- "pick conv3x3 OR max_pool at this spot" -- is not differentiable, so you cannot use gradient descent on it. So DARTS refuses to choose discretely during the search. Instead it puts all candidate operations at each spot, mixes their outputs with learnable weights (a softmax over the choices), and optimises those weights with plain old gradient descent alongside the network weights.

class DARTSCell(nn.Module):
    """A cell where the CHOICE of operation is itself learnable."""
    def __init__(self, channels, n_ops=6):
        super().__init__()
        self.ops = nn.ModuleList([
            nn.Conv2d(channels, channels, 3, padding=1),
            nn.Conv2d(channels, channels, 5, padding=2),
            nn.Sequential(
                nn.Conv2d(channels, channels, 3, padding=1, groups=channels),
                nn.Conv2d(channels, channels, 1),
            ),
            nn.MaxPool2d(3, stride=1, padding=1),
            nn.AvgPool2d(3, stride=1, padding=1),
            nn.Identity(),
        ])
        # Architecture parameters -- one weight per candidate operation.
        self.alphas = nn.Parameter(torch.randn(n_ops) * 0.01)

    def forward(self, x):
        # Soft mixture: run EVERY op, weight each by softmax(alpha), sum them.
        weights = torch.softmax(self.alphas, dim=0)
        return sum(w * op(x) for w, op in zip(weights, self.ops))

# During search: optimise model weights AND alphas together (a bilevel problem).
# After search: DISCRETIZE -- keep only the op with the largest alpha at each spot.
# Then retrain that discrete architecture from scratch for real.

Read the forward method slowly, because it is the entire trick. During the search, the cell is a blend of all operations at once -- a conv3x3 that is 40% present, a max_pool that is 25% present, and so on. That blend is fully differentiable, so a single training run nudges the alphas toward whichever operations actually help. When the search finishes you discretize: at each position you keep the one operation with the highest alpha and throw the rest away, then retrain that clean architecture from scratch. The head-line number is the reason DARTS mattered so much -- it searches on a single GPU in one to four days instead of 800 GPUs for weeks. Having said that, the soft mixture is memory-hungry (you hold every operation in memory at once), and DARTS has a reputation for occasionally collapsing toward too many skip-connections -- there is a small cottage industry of follow-up papers (PC-DARTS, FairDARTS) patching exactly that. It is brilliant, not magic.

Practical AutoML frameworks

Now let's come back down to earth, because I do not want you leaving with the impression that you need NAS to benefit from AutoML. For the overwhelming majority of real problems -- anything that looks like a table of rows and columns -- you never touch architecture search at all. You reach for a framework that automates model selection and hyperparameter tuning end to end:

# FLAML -- Fast Lightweight AutoML (Microsoft). Give it a time budget, walk away.
# from flaml import AutoML
# automl = AutoML()
# automl.fit(X_train, y_train, task="classification",
#            time_budget=300,                      # 5 minutes, then stop
#            estimator_list=["lgbm", "xgboost", "rf"])
# print(automl.best_estimator)                     # nearly always lgbm or xgboost

# Ray Tune -- when you want DISTRIBUTED hyperparameter search across many machines.
# from ray import tune
# search_space = {
#     "lr":         tune.loguniform(1e-5, 1e-1),
#     "hidden":     tune.choice([128, 256, 512]),
#     "batch_size": tune.choice([32, 64, 128]),
# }
# analysis = tune.run(train_fn, config=search_space, num_samples=50,
#                     resources_per_trial={"cpu": 2, "gpu": 0.5})

There is a pattern here that took me embarrassingly long to internalise, so let me hand it to you for free. On tabular data, AutoML almost always converges to gradient boosting -- XGBoost or LightGBM (remember episode #19?) with well-tuned hyperparameters. Not a neural network. Not something exotic. The same boosted trees that have won Kaggle for a decade, tuned better than you would tune them by hand. For deep learning tasks the story is different: NAS-derived architectures like EfficientNet (episode #46) and NASNet genuinely rival hand-designed models -- but the search cost stays eye-watering, which is why almost nobody runs the search themselves. They just download the architecture somebody else already searched for. Which is a perfectly respectable form of "transfer" -- reusing architecture knowledge across tasks instead of re-searching from zero every time.

When AutoML helps, and when it doesn't

Right, the honest section. I would be doing you a disservice if I sold AutoML as a universal solvent. Here is my actual field guide:

automl_field_guide = {
    'reach_for_it': [
        'Tabular classification/regression -- AutoML reliably matches expert tuning',
        'Standard image classification -- borrow a NAS-derived backbone, done',
        'Tuning a FIXED architecture you already chose -- basically always worth it',
        'A quick strong baseline -- 30 minutes of AutoML beats a week of hand-tuning',
    ],
    'do_it_yourself': [
        'Genuinely novel problems -- AutoML only searches KNOWN spaces',
        'A tight compute budget -- full NAS is measured in GPU-years, not hours',
        'Hard constraints -- medical wants interpretability, edge wants tiny models',
        'When you must understand WHY it works -- AutoML hands you a black box',
    ],
}

The blunt assessment, from someone who has spent real money learning it: for most practitioners, Bayesian hyperparameter optimization (Optuna) on a manually designed architecture delivers about 90% of AutoML's benefit at maybe 10% of the complexity and cost. Full NAS is a tool for teams whose compute budget is measured in GPU-years and whose problem is important enough to justify it. Everyone else should tune well, borrow architectures liberally, and spend the saved GPU-hours on data quality (episode #14), which is where the real gains hide anyway. AutoML is a fantastic servant and a very expensive master -- know which one you are hiring ;-)

TL;DR - handing the design work to the machine

  • Hyperparameter optimization is the foundation of all AutoML -- and random search beats grid search in practice because hyperparameters differ wildly in importance and random sampling covers the important ones more broadly;
  • Bayesian optimization (Optuna, Ray Tune) builds a probabilistic model of your score landscape and aims its next guess via an acquisition function, finding good settings in 20-50 evaluations instead of 100+;
  • Neural Architecture Search climbs to the top rung and designs the model itself from a search space of candidate operations -- early NAS cost 800 GPUs for weeks because it trained every candidate to completion;
  • DARTS dragged that down to a single GPU in days by making the architecture choice differentiable -- blend all operations with learnable softmax weights, optimise with gradient descent, then discretize by keeping the strongest op at each spot;
  • practical AutoML frameworks (FLAML, Auto-sklearn) shine on tabular data, where they almost always converge to well-tuned gradient boosting rather than anything exotic;
  • for most people, Bayesian HPO on a hand-designed architecture buys ~90% of the benefit at a fraction of the cost -- reserve full NAS for problems that genuinely justify GPU-years.

Exercises

Three to chew on before next time -- and as always I'll walk through full solutions in the following episode.

  1. Race random search against Bayesian. Take any scikit-learn dataset and a GradientBoostingClassifier with three hyperparameters (learning rate, max depth, n_estimators). Tune it two ways: 40 iterations of random_search above, and 40 trials of an Optuna study over the same ranges. Record the best cross-validation score each method reaches after 10, 20, 30, 40 trials. Which one is ahead early, and which one wins by the end? Write one sentence explaining what you saw.

  2. Watch the alphas move. Instantiate the DARTSCell above on a small random input tensor, then run a tiny optimisation loop that trains the cell to imitate a fixed target output (mean-squared error is fine). Print torch.softmax(cell.alphas, dim=0) before training and after 200 steps. Does one operation clearly dominate? Describe what "discretizing" this trained cell would produce.

  3. Decide: AutoML or not? For each scenario, decide whether you would (a) run full AutoML, (b) run Bayesian HPO on a hand-picked model, or (c) design by hand -- and justify in two sentences: (i) a startup needs a strong churn-prediction baseline on tabular customer data by Friday; (ii) a research lab is inventing a model for a brand-new sensor modality nobody has published on; (iii) an embedded team needs the smallest possible image classifier that fits in 2MB of flash. There is no single right answer -- the reasoning is the whole point.

We have now taught the machine to tune its own knobs and even to draw its own blueprints. But notice what every technique in this episode quietly assumes: that correlation in the data is enough. AutoML searches for whatever pattern predicts the target best -- and it does not care, even slightly, whether that pattern reflects a real cause or a lucky coincidence that will evaporate the moment the world shifts. That gap between "predicts well" and "actually explains why" is one of the deepest cracks in modern ML, and it is exactly where we are headed next ;-)

Thanks for your time -- and happy tuning!

@scipio

stem stemsocial steemstem python programming
0
0
0.000
0 comments