Learn AI Series (#132) - AI for Structured Data

avatar

Learn AI Series (#132) - AI for Structured Data

variant-a-07-purple.png

What will I learn

  • You will learn why deep learning keeps losing to gradient boosting on tabular data -- the single most common data format in the entire industry, and deep learning's most stubborn blind spot;
  • TabNet -- how sparse, sequential attention can be pointed at columns in stead of pixels, and why the result is actually interpretable;
  • SAINT, where transformer-style self-attention runs in two directions at once -- across features, and across the other rows in your batch;
  • the honest, benchmark-backed answer to "when does a tree still win, and when does a neural net finally earn its keep?";
  • the old feature-engineering-versus-learned-features debate, and why the pragmatic answer annoys purists on both sides;
  • and a practical mental model you can actually reach for the next time somebody hands you a spreadsheet and says "predict this" ;-)

Requirements

  • A working modern computer running macOS, Windows or Ubuntu;
  • An installed Python 3(.10+) distribution with PyTorch and scikit-learn (pip install torch scikit-learn xgboost) -- everything here runs comfortably on a laptop, no GPU required today;
  • You've been through episode #19 (Gradient Boosting, the hero of today's story) and episodes #51-#53 (attention and the Transformer), because both TabNet and SAINT lean on that machinery directly. A quick glance back at episode #15 (feature engineering) will not hurt either.

Difficulty

  • Beginner

Curriculum (of the Learn AI Series):

Learn AI Series (#132) - AI for Structured Data

I ended episode #131 with a deliberately uncomfortable admission, and I promised we would come back to it. We spent that whole episode celebrating data that is proudly, irreducibly graph-shaped -- molecules, social networks, fraud rings -- and rightly so. But then I pointed out the twist that ought to nag at anybody who has built a hundred neural networks in a row: the overwhelming majority of the data actually sitting in the world's databases is not a molecule or a graph at all. It is boring, flat, tabular stuff. Rows and columns. The spreadsheets and SQL tables that quietly run every bank, shop and hospital on the planet. And on that kind of data -- the most common kind there is -- the fancy deep networks we have been lovingly hand-building very often lose to a humble gradient-boosted tree from episode #19.

So today we stare that embarrassment straight in the face. Why does deep learning struggle exactly where the data is most abundant? What have people tried to do about it? And -- the part you actually care about -- what should you reach for when the problem is a plain old table? Here we go.

Solutions to episode #131's exercises

As always, let's clear last time's graph homework first -- full code, no hand-waving.

Exercise 1 -- watch over-smoothing happen. The task was to rebuild the Cora classifier with 2, then 4, then 8 GCN layers, train each for 200 epochs, and then, for the 8-layer model, measure the average pairwise cosine similarity of the final node embeddings. The whole point is to see the accuracy collapse and connect it to the embeddings all drifting toward one vector.

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch_geometric.nn import GCNConv
from torch_geometric.datasets import Planetoid

dataset = Planetoid(root='/tmp/Cora', name='Cora')
data = dataset[0]

class DeepGCN(nn.Module):
    def __init__(self, in_dim, hidden, out_dim, n_layers):
        super().__init__()
        dims = [in_dim] + [hidden] * (n_layers - 1) + [out_dim]
        self.convs = nn.ModuleList(
            [GCNConv(dims[i], dims[i + 1]) for i in range(n_layers)]
        )

    def forward(self, data, return_embed=False):
        x, edge_index = data.x, data.edge_index
        for i, conv in enumerate(self.convs):
            x = conv(x, edge_index)
            if i < len(self.convs) - 1:
                x = F.relu(x)
        return x

def train_eval(n_layers):
    torch.manual_seed(0)
    model = DeepGCN(dataset.num_node_features, 64, dataset.num_classes, n_layers)
    opt = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4)
    for _ in range(200):
        model.train(); opt.zero_grad()
        out = model(data)
        F.cross_entropy(out[data.train_mask], data.y[data.train_mask]).backward()
        opt.step()
    model.eval()
    pred = model(data).argmax(dim=1)
    acc = (pred[data.test_mask] == data.y[data.test_mask]).float().mean().item()
    return acc

for n in (2, 4, 8):
    print(f"{n} layers -> test acc {train_eval(n):.3f}")

Run it and you will watch the accuracy slide from a healthy ~0.81 at 2 layers down toward coin-flip territory by 8. Now the diagnosis -- the cosine similarity of the over-smoothed embeddings:

def mean_pairwise_cosine(embeds):
    e = F.normalize(embeds, dim=1)
    sim = e @ e.t()                       # all pairwise cosines
    n = sim.shape[0]
    off_diag = (sim.sum() - n) / (n * n - n)   # exclude the diagonal (self-sim = 1)
    return off_diag.item()

torch.manual_seed(0)
deep = DeepGCN(dataset.num_node_features, 64, dataset.num_classes, 8)
opt = torch.optim.Adam(deep.parameters(), lr=0.01, weight_decay=5e-4)
for _ in range(200):
    deep.train(); opt.zero_grad()
    out = deep(data)
    F.cross_entropy(out[data.train_mask], data.y[data.train_mask]).backward()
    opt.step()
deep.eval()
print(f"mean pairwise cosine (8 layers): {mean_pairwise_cosine(deep(data)):.3f}")

The two-sentence answer I was fishing for: the deeper the stack, the more rounds of neighbour-averaging you apply, and averaging is a smoothing operation -- so node representations converge toward a common vector and the mean pairwise cosine creeps up toward 1. Once every node looks like every other node, the classifier has nothing left to separate, and the accuracy falls off a cliff. Over-smoothing is not a training bug you can tune away; it is baked into what stacking message-passing layers does.

Exercise 2 -- GCN versus GAT, head to head. Train both variants on Cora five times each with different seeds, then report mean and standard deviation. Do the error bars overlap?

import numpy as np
from torch_geometric.nn import GATConv

class GNN(nn.Module):
    def __init__(self, in_dim, hidden, out_dim, use_gat):
        super().__init__()
        if use_gat:
            self.c1 = GATConv(in_dim, hidden, heads=4, concat=True)
            self.c2 = GATConv(hidden * 4, out_dim, heads=1, concat=False)
        else:
            self.c1 = GCNConv(in_dim, hidden)
            self.c2 = GCNConv(hidden, out_dim)
        self.drop = nn.Dropout(0.5)

    def forward(self, data):
        x, ei = data.x, data.edge_index
        x = self.drop(F.relu(self.c1(x, ei)))
        return self.c2(x, ei)

def run(use_gat, seed):
    torch.manual_seed(seed)
    model = GNN(dataset.num_node_features, 64, dataset.num_classes, use_gat)
    opt = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4)
    for _ in range(200):
        model.train(); opt.zero_grad()
        out = model(data)
        F.cross_entropy(out[data.train_mask], data.y[data.train_mask]).backward()
        opt.step()
    model.eval()
    pred = model(data).argmax(dim=1)
    return (pred[data.test_mask] == data.y[data.test_mask]).float().mean().item()

for name, flag in [("GCN", False), ("GAT", True)]:
    accs = [run(flag, s) for s in range(5)]
    print(f"{name}: {np.mean(accs):.3f} +/- {np.std(accs):.3f}")

On Cora you will typically see GAT land a point or two above GCN, but with standard deviations around 0.005-0.01 the error bars very often overlap. The honest one-liner: on this particular small, homophilous citation graph, attention's extra compute buys you a marginal, not-always-significant gain -- GAT earns its keep on messier, heterophilous graphs, not on the friendly benchmark. This is exactly the "measure, don't assume" discipline from episode #13, and it is the same lesson waiting for us today with tables.

Exercise 3 -- build a graph by hand and reason about hops. Construct a 6-node friend graph, run simple_message_pass twice, and confirm two-hop influence. Then: for a node whose farthest neighbour is 4 hops away, how many rounds until that signal can reach it?

def simple_message_pass(x, edge_index):
    num_nodes = x.shape[0]
    src, dst = edge_index
    agg = torch.zeros_like(x)
    agg.index_add_(0, dst, x[src])
    deg = torch.zeros(num_nodes)
    deg.index_add_(0, dst, torch.ones(src.shape[0]))
    return agg / deg.clamp(min=1).unsqueeze(1)

# A line-ish graph: 0-1-2-3-4-5 (undirected), so node 0's farthest neighbour is 5.
edge_index = torch.tensor([
    [0, 1, 1, 2, 2, 3, 3, 4, 4, 5],
    [1, 0, 2, 1, 3, 2, 4, 3, 5, 4],
], dtype=torch.long)

x = torch.eye(6)            # one-hot features so we can literally trace influence
h1 = simple_message_pass(x, edge_index)
h2 = simple_message_pass(h1, edge_index)
print("Node 0 after 2 rounds, has it 'seen' node 2?",
      h2[0, 2].item() > 0)   # True: node 2 is exactly two hops from node 0

The answer to the closing question: message passing moves information exactly one hop per round, so a signal four hops away needs a minimum of four rounds (four layers) before it can possibly reach the target node. That, incidentally, is the exact same reasoning that makes deep GNNs tempting -- and, as exercise 1 just showed, over-smoothing is the tax you pay for reaching too far. Right, graphs put to bed. On to the spreadsheets.

Why deep learning struggles on tables

To understand the problem, think about why deep learning is so devastating on images and text in the first place. Images have spatial locality -- nearby pixels belong together, which is exactly the prior a convolution (episode #45) bakes in. Text has sequential structure -- word order carries meaning, which is what attention (episode #51) exploits. Both domains hand you a homogeneous input (all pixels, or all tokens) and, usually, oceans of data to pre-train on.

Tabular data has none of that. Not one of those properties. Let me be specific, because the specifics are the whole story:

Heterogeneous features. Column 1 is a person's age (continuous). Column 2 is their country (categorical, 195 possible values). Column 3 is their last purchase amount (continuous, heavy-tailed). Column 4 is a yes/no subscription flag (binary). Every column has a different scale, a different distribution, a different meaning. There is no sensible "convolution" you can slide across columns, because column 3 has nothing spatially in common with column 4.

No spatial or sequential structure. Shuffle the column order and you have lost exactly nothing -- the information content is identical. There is no "nearby" relationship between features for a convolutional prior to grab hold of.

Feature interactions are irregular. In an image, local pixel patterns matter everywhere in the frame. In a table, the interaction between "age" and "income" might be the crux of the whole problem, while "age" and "shoe size" is pure noise. Which interactions matter is dataset-specific, and the network has to discover that from scratch.

Smaller datasets. Most tabular problems have thousands to a few hundred thousand rows. Deep learning is happiest with millions. Trees, by contrast, are sample-efficient by design -- they greedily hunt for the single best split, and a good split needs surprisingly few examples to find.

Let's watch a plain MLP -- the naive deep-learning reflex -- have a go at a classic tabular dataset.

import torch
import torch.nn as nn
import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error

data = fetch_california_housing()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42
)
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

class SimpleMLP(nn.Module):
    def __init__(self, in_dim, hidden=128):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(in_dim, hidden), nn.ReLU(), nn.Dropout(0.1),
            nn.Linear(hidden, hidden), nn.ReLU(), nn.Dropout(0.1),
            nn.Linear(hidden, 1),
        )
    def forward(self, x):
        return self.net(x).squeeze(-1)

model = SimpleMLP(X_train.shape[1])
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
X_tr = torch.tensor(X_train, dtype=torch.float32)
y_tr = torch.tensor(y_train, dtype=torch.float32)

for epoch in range(100):
    model.train()
    loss = nn.MSELoss()(model(X_tr), y_tr)
    optimizer.zero_grad(); loss.backward(); optimizer.step()

model.eval()
with torch.no_grad():
    pred = model(torch.tensor(X_test, dtype=torch.float32)).numpy()
print(f"MLP RMSE: {mean_squared_error(y_test, pred, squared=False):.4f}")

Note that even to get this far the MLP needed the StandardScaler -- feed it raw features and it barely trains at all. Hold that thought, because a tree would not have cared one bit about scaling. Now run the tree and compare.

What gradient boosting does right

Understanding why trees win teaches you more than any of the fancy alternatives. And this is not a fringe opinion, mind you -- multiple large-scale benchmarks have confirmed it. Grinsztajn and colleagues (2022) tested across 45 tabular datasets and found tree-based models beating deep learning on medium-sized data, with less tuning. The "deep learning conquers everything" narrative has a glaring exception, and it happens to be the most common data format in industry. Having said that, the reasons are wonderfully concrete:

Axis-aligned splits are natural for tables. A decision tree asks "is income > 50k?" -- a threshold on a single feature. A staggering number of real-world decision boundaries are approximately axis-aligned like that. A neural network has to approximate the same boundary with curved surfaces, which is overkill and wastes capacity.

Trees handle heterogeneous features natively. Categorical, continuous, missing -- a tree splits on any of them without a preprocessing pipeline. A neural network needs encoding, normalisation, and imputation before it can even start.

Trees are invariant to monotone transformations. Whether a feature ranges 0-1 or 0-1,000,000 does not change a single split. Neural nets are painfully sensitive to scale -- which is exactly why our MLP needed that scaler.

Boosting corrects its own mistakes. Each new tree in a gradient-boosting ensemble specifically targets the samples the previous trees got wrong (this is the residual-fitting idea from episode #19). It is a built-in form of hard-example mining that an MLP simply does not do on its own.

from sklearn.ensemble import GradientBoostingRegressor

gbm = GradientBoostingRegressor(n_estimators=200, max_depth=5, random_state=42)
gbm.fit(X_train, y_train)   # note: NO scaling needed, the tree does not care
pred_gbm = gbm.predict(X_test)
print(f"GBM RMSE: {mean_squared_error(y_test, pred_gbm, squared=False):.4f}")

Run both and the tree almost certainly wins -- and it did so with default parameters and zero scaling. Not because the MLP is fundamentally incapable, but because it walked into the room with the wrong inductive biases for this kind of data. Keep that framing: it is not a story about intelligence, it is a story about priors.

TabNet: attention for feature selection

So can we build a network with better priors? Google's TabNet (2019) was one of the first serious attempts to beat trees on their own turf. Its key insight is lovely: make feature selection step-wise, the way a decision tree does. At each decision step, an attention mechanism picks which features to look at, processes them, and hands off to the next step -- loosely echoing how a tree selects one feature to split on at each node.

Two components carry the whole design:

Sparse attention for feature selection. At each step a learnable mask selects a subset of features. Crucially the mask uses sparsemax, not softmax, so it produces genuine zeros -- features are truly ignored, not merely down-weighted. That is what makes TabNet interpretable: you can literally read off which features it used at each step.

Sequential multi-step processing. In stead of one forward pass, TabNet processes the data across several "steps", each seeing the original features through a different mask, and sums the outputs. Think of it as the network making a little sequence of decisions, each focused on a different corner of the feature space.

import torch.nn.functional as F

class TabNetStep(nn.Module):
    """A simplified single TabNet decision step."""
    def __init__(self, in_dim, hidden_dim, out_dim):
        super().__init__()
        self.fc_attn = nn.Sequential(
            nn.Linear(in_dim, in_dim), nn.BatchNorm1d(in_dim),
        )
        self.fc_process = nn.Sequential(
            nn.Linear(in_dim, hidden_dim), nn.BatchNorm1d(hidden_dim), nn.ReLU(),
            nn.Linear(hidden_dim, out_dim),
        )

    def forward(self, x, prior_scales):
        # Which features matter for THIS step?
        attn = torch.softmax(self.fc_attn(x) * prior_scales, dim=-1)
        masked = attn * x
        out = self.fc_process(masked)
        # Features used a lot now get de-prioritised for later steps.
        prior_scales = prior_scales * (1.0 - attn)
        return out, attn, prior_scales

class MiniTabNet(nn.Module):
    def __init__(self, in_dim, hidden=64, out_dim=1, n_steps=3):
        super().__init__()
        self.steps = nn.ModuleList(
            [TabNetStep(in_dim, hidden, hidden) for _ in range(n_steps)]
        )
        self.head = nn.Linear(hidden, out_dim)

    def forward(self, x):
        prior = torch.ones_like(x)                 # all features start available
        agg = torch.zeros(x.shape[0], 64, device=x.device)
        for step in self.steps:
            h, attn, prior = step(x, prior)
            agg = agg + F.relu(h)
        return self.head(agg).squeeze(-1)

TabNet's pitch is that it is interpretable and needs no feature engineering. In practice the results are, let me be honest, mixed. On some datasets it matches or beats XGBoost; on others it quietly underperforms, especially when you have not spent a fortune on hyperparameter tuning. It is a genuinely clever design that did not turn out to be a free lunch -- which is a recurring theme in this whole area.

SAINT: self-attention meets intersample attention

SAINT (Self-Attention and Intersample Attention Transformer, 2021) is bolder. It applies transformer-style self-attention in two directions: across the features within a row (which columns relate to each other?), and across the rows within a batch (which other samples inform this one?).

The feature-wise half is the familiar Transformer of episode #52, just pointed sideways. Each feature becomes a "token": its value is embedded and added to a learned per-feature embedding, then multi-head self-attention lets the columns talk to each other -- "age" can attend to "income" to capture their joint effect without you ever hand-crafting an age_times_income column.

class FeatureSelfAttention(nn.Module):
    """Self-attention across features (columns) for each sample."""
    def __init__(self, n_features, embed_dim, n_heads=4):
        super().__init__()
        self.feature_embeddings = nn.Embedding(n_features, embed_dim)
        self.value_proj = nn.Linear(1, embed_dim)
        self.attn = nn.MultiheadAttention(embed_dim, n_heads, batch_first=True)
        self.norm = nn.LayerNorm(embed_dim)

    def forward(self, x):
        # x: (batch, n_features)
        _, n_feat = x.shape
        feat_ids = torch.arange(n_feat, device=x.device)
        feat_emb = self.feature_embeddings(feat_ids)     # (n_feat, embed_dim)
        val_emb = self.value_proj(x.unsqueeze(-1))        # (batch, n_feat, embed_dim)
        tokens = val_emb + feat_emb.unsqueeze(0)          # value + column identity
        attended, _ = self.attn(tokens, tokens, tokens)
        return self.norm(tokens + attended)               # residual, as always

The intersample half is the strange, interesting bit. For each feature, the model also attends across all the rows in the mini-batch, letting other training examples inform the current prediction -- a learned, soft cousin of k-nearest-neighbours baked right into the forward pass. The philosophical question SAINT poses is a good one: does the attention machinery that reshaped sequences and images actually transfer to tables? The answer, as far as the benchmarks go, is "partially". SAINT is competitive on many of them -- but it wants careful tuning and a large-enough dataset to justify all those parameters.

The honest benchmark: when does what win?

Let me be completely straight about what the evidence says, as of 2025-2026, because vendor blog posts will not:

Gradient boosting (XGBoost, LightGBM, CatBoost) wins on:

  • Small to medium datasets (roughly under 50k rows);
  • data with lots of categorical features;
  • problems where the real decision boundaries are close to axis-aligned;
  • situations with a tight training-time and tuning budget;
  • most Kaggle tabular competitions -- still, in 2026.

Deep learning approaches win on:

  • Very large datasets (hundreds of thousands to millions of rows);
  • problems needing one end-to-end differentiable pipeline (e.g. tabular columns plus an image or a text blob);
  • semi-supervised settings where unlabelled data is abundant;
  • problems where you want built-in uncertainty estimates from a neural ensemble.

Here is the decision logic I actually carry around in my head, written as throwaway pseudo-code -- not a model, just a way of thinking:

# This is a decision aid, NOT something you fit. Read it, don't train it.
def choose_model(dataset_size, has_mixed_modalities, tuning_budget):
    if has_mixed_modalities:
        return "deep_learning"       # you need one differentiable pipeline
    if dataset_size < 10_000:
        return "gradient_boosting"   # trees own the small-data regime
    if dataset_size < 50_000:
        return "try_both"            # genuinely competitive, so measure
    if tuning_budget == "low":
        return "gradient_boosting"   # far less sensitive to hyperparameters
    return "try_both"                # at scale, deep learning starts to compete

Nota bene: notice how often the honest answer is simply "try both and measure". That is not a cop-out, it is the discipline -- the same one from episode #13, where we learned that a number on a held-out set beats an opinion every single time.

Feature engineering vs. learned features

One of the loudest arguments for deep learning on tables is that it should abolish manual feature engineering. In stead of you hand-crafting age_times_income or log_purchase_amount, the network learns those transformations for you.

In theory, sure -- a wide-and-deep enough MLP can represent any feature interaction. In practice, hand-crafted features keep helping, even alongside deep models. Why? Because domain knowledge is free information that shrinks what the model has to discover from scratch. If you happen to know that "income relative to the regional median" matters more than raw income, encoding that directly spares the network from having to rediscover it from a limited pile of rows.

So the pragmatic move -- the one that quietly annoys purists on both sides -- is to do sensible feature engineering and use a model that can learn extra interactions on top. That is not cheating; it is efficiency. Even TabNet and SAINT are happier with a bit of thoughtful preprocessing under them.

The one place deep learning genuinely does dissolve feature engineering is when tabular data is fused with unstructured data -- images, text, audio. You cannot hand-engineer features out of raw pixels, but you can push them through a CNN (episode #45) or a Transformer and concatenate the result with your columns. When a prediction depends on both structured fields and a photo -- think property pricing from house features plus listing images -- an end-to-end neural network stops being a luxury and becomes the only sane architecture. That, right there, is the real frontier for deep learning on tables.

What the field looks like right now

The tabular-deep-learning space is moving fast, so let me not pretend it is settled. FT-Transformer (2021) made a quietly important point: a straightforward Transformer over feature tokens -- without TabNet's step machinery or SAINT's intersample gymnastics -- is competitive with far simpler code.

# The spine of FT-Transformer: tokenise features, then a standard Transformer.
class FTTransformerSpine(nn.Module):
    def __init__(self, n_features, embed_dim=32, depth=3, heads=8):
        super().__init__()
        self.tokenizer = FeatureSelfAttention(n_features, embed_dim, heads)
        layer = nn.TransformerEncoderLayer(embed_dim, heads, batch_first=True)
        self.encoder = nn.TransformerEncoder(layer, num_layers=depth)
        self.cls = nn.Parameter(torch.randn(1, 1, embed_dim))   # a [CLS]-style token
        self.head = nn.Linear(embed_dim, 1)

    def forward(self, x):
        tokens = self.tokenizer(x)                              # (batch, n_feat, embed)
        cls = self.cls.expand(x.shape[0], -1, -1)
        tokens = torch.cat([cls, tokens], dim=1)                # prepend the CLS token
        encoded = self.encoder(tokens)
        return self.head(encoded[:, 0]).squeeze(-1)             # read out from CLS

If that [CLS]-token readout looks familiar, it should -- it is exactly the trick BERT used back in episode #59, now pointed at columns in stead of words. More recent work goes further still: pre-training on large tabular corpora (the same recipe that made BERT and GPT what they are), and transfer learning across tabular datasets. It is early days, but the direction is unmistakable.

Still, the honest bottom line has not moved: if you have a bog-standard tabular problem, start with XGBoost or LightGBM. Train it in five minutes. Lock in your baseline. Then ask whether the marginal gain from a deep model justifies the extra complexity, training time, and tuning pain. For most tabular problems the answer is a cheerful "no". For the ones where it is "yes" -- multi-modal data, very large data, semi-supervised settings -- the architectures in this episode are your starting line.

Round-up

  • Deep learning struggles on tabular data because tables lack the spatial and sequential structure that CNNs and RNNs are built to exploit -- and because tabular datasets are usually small;
  • gradient boosting wins for concrete reasons: axis-aligned splits, native handling of heterogeneous and missing values, invariance to feature scale, and built-in hard-example mining;
  • TabNet brings sparse, sequential attention to feature selection -- interpretable, engineering-free in principle, but empirically hit-or-miss;
  • SAINT runs self-attention across features and across samples, a genuinely creative transfer of the Transformer -- competitive, but hungry for tuning and data;
  • the evidence is clear: trees dominate small-to-medium data; deep learning competes at scale and shines on multi-modal problems;
  • feature engineering still helps even deep models, because domain knowledge is free information;
  • the practical rule: start with XGBoost, lock in a baseline, and only go deeper when you have a concrete reason to.

Exercises

Three to chew on before next time -- and, as always, I'll walk through full solutions at the top of the following episode.

  1. Baseline the gap yourself. Take the California Housing setup from this episode and add an XGBoost regressor (xgboost.XGBRegressor, default params) alongside the MLP and the GradientBoostingRegressor. Report all three RMSEs. Then re-run the MLP without the StandardScaler and record what happens. Write two sentences: one on the tree-versus-MLP gap, one on why scaling mattered so much for only one of the three.

  2. Make the MLP fight back. The plain MLP started with lousy priors -- so give it better ones. Add batch normalisation between layers, widen it, and train for 300 epochs with an LR scheduler. How close can you push the MLP's RMSE to the GBM's? Note the smallest change that helped the most -- that tells you which missing prior hurt worst.

  3. Read TabNet's mind. Instantiate MiniTabNet on a small classification table of your choice (the sklearn breast-cancer dataset is fine), train it briefly, and for a handful of test rows print the attn mask from each of the three steps. Which features does it consistently select? Compare that shortlist against a GradientBoostingClassifier's feature_importances_ and write one sentence on whether the two methods agree about what matters.

We have spent this whole episode defending the honour of the humble tree, and rightly so. But notice the thread that kept resurfacing: deep learning's real advantage on tables shows up when data is plentiful -- and the cruel irony is that the tabular problems where we most want a good model are often the ones starved of rows. So here is the question that ought to itch: what if you could manufacture more rows? Not copy them, but generate believable, statistically faithful new examples to feed a data-hungry model -- and while you are at it, share a dataset without leaking a single real person's private record. That tension between "I need more data" and "I cannot expose the data I have" is exactly where we head next ;-)

Thanks for sticking with me this far -- now go and beat a tree!

@scipio

stem stemsocial steemstem python programming
0
0
0.000
0 comments