Learn AI Series (#154) - AI Research: How to Read Papers

avatar

Learn AI Series (#154) - AI Research: How to Read Papers

variant-b-09-blue.png

What will I learn?

  • You will learn the anatomy of a machine learning paper and what each section is actually for;
  • reading strategies: how to extract value from a paper without reading every word;
  • how to reproduce results and why reproduction is the real test of understanding;
  • navigating arXiv, Semantic Scholar, and Papers With Code effectively;
  • building a research reading habit that sticks without consuming your life;
  • when and how to go from reading papers to contributing your own.

Requirements

  • A working modern computer running macOS, Windows or Ubuntu;
  • Python 3.10+ installed;
  • Comfortable with the mathematical notation from episodes #8-9.

Difficulty

  • Advanced

Curriculum (of the Learn AI Series):

Learn AI Series (#154) - AI Research: How to Read Papers

Every single technique in this series came from a paper. The transformer we built in episode #52 and #53 came from "Attention Is All You Need" (Vaswani et al., 2017). The attention mechanism from #51 has its own lineage. PPO, which we leaned on back in the reinforcement learning stretch around #109, came from a paper by Schulman et al. (2017). Diffusion models from #84 and #85 trace back to Ho et al. (2020). At some point the tutorials run out -- yes, even THIS one, we are nearly at the end -- and you have to go to the source.

Here is the problem though: ML papers are intimidating. Dense notation, assumed background knowledge, a terse writing style that reads like the authors were being charged by the word, and that ever-present feeling that everyone else in the room understands this and you are the only one who doesn't. (You are not. Most people find them hard. The ones who claim they don't are either lying, or they have read several hundred of them and forgot what the first twenty felt like.)

I have been reading these things for quit some years now, and I can tell you the skill is learnable -- it is not a talent you are born with. This episode is about developing that skill: reading papers efficiently, extracting what you actually need, and -- eventually -- contributing your own. Let's dive right in!

Anatomy of an ML paper

Almost every ML paper follows the same skeleton. Once you know what each section is for, you stop reading front-to-back like a novel and start reading like an engineer who knows where the important bolt is. Here is the map:

paper_structure = {
    "abstract": {
        "purpose": "TL;DR of the entire paper in one paragraph",
        "what_to_extract": "The claim: what they did, why it matters, key result",
        "reading_time": "1 minute",
        "priority": "ALWAYS read this first",
    },
    "introduction": {
        "purpose": "Why this problem matters, what's been tried, what's new",
        "what_to_extract": "The gap they're filling - what couldn't be done before",
        "reading_time": "3-5 minutes",
        "priority": "Read fully for unfamiliar topics, skim for familiar ones",
    },
    "related_work": {
        "purpose": "Context: what other approaches exist and why they fall short",
        "what_to_extract": "Names of competing methods for comparison",
        "reading_time": "Skip on first pass, return if you need context",
        "priority": "Low - often written to satisfy reviewers, not to inform you",
    },
    "method": {
        "purpose": "The actual contribution - the new technique or architecture",
        "what_to_extract": "How it works. Equations, architecture, training procedure",
        "reading_time": "15-30 minutes (the core of the paper)",
        "priority": "HIGH - this is what you came for",
    },
    "experiments": {
        "purpose": "Evidence that the method works",
        "what_to_extract": "Datasets used, baselines compared, key metrics",
        "reading_time": "5-10 minutes",
        "priority": "HIGH - skeptically verify whether the claims hold up",
    },
    "ablation_study": {
        "purpose": "Which components of the method actually matter",
        "what_to_extract": "What happens when you remove each piece",
        "reading_time": "5 minutes",
        "priority": "VERY HIGH - often more informative than the main results",
    },
    "conclusion": {
        "purpose": "Summary + limitations + future work",
        "what_to_extract": "Honest limitations (if they list any)",
        "reading_time": "2 minutes",
        "priority": "Medium - useful for understanding scope",
    },
    "appendix": {
        "purpose": "Implementation details, extra experiments, hyperparameters",
        "what_to_extract": "The specifics you need to reproduce the work",
        "reading_time": "Only when reproducing",
        "priority": "Low initially, CRITICAL for reproduction",
    },
}

Notice which section I flagged as VERY HIGH: the ablation study. That is the one nobody tells you to read, and it is often the most honest part of the whole paper. The main results table is where authors put their best foot forward. The ablation is where they are forced to admit "actually, when we removed component X, most of our gain disappeared". That is gold. That tells you what the paper is really about, as opposed to what the title wants you to believe.

The three-pass reading strategy

Do NOT read papers linearly. I said it above and I will say it again because it is the single biggest mistake beginners make. Linear reading works for fiction -- it does not work for research. Use three passes:

Pass 1: Survey (5-10 minutes) -- Read the abstract, look at every figure and table WITH the captions, read the conclusion. After this pass you should know three things: what the paper claims, roughly what evidence they provide, and whether it is worth more of your time. Most papers -- I would say 90% of what crosses your desk -- stop right here. And that is fine. Deciding not to read something is a skill too.

Pass 2: Understanding (30-60 minutes) -- Read the introduction, method, and experiments carefully. Skip the proofs on the first read; chase the intuition in stead. When you hit notation you don't understand, write it down but don't get stuck on it. After this pass you should be able to explain the paper's main idea to another human in your own words. That "explain it to someone" test is the real bar, not "I read every line".

Pass 3: Reproduction (hours to days) -- Only for papers you actually intend to use. Read the method and appendix line by line, check every equation, look at the code if it exists, and try to implement the key pieces yourself. This is where understanding becomes real.

To make this stick, I keep a reading log. It sounds bureaucratic (I resisted it for years, being a bit allergic to process), but it forces me to extract the important bits while they are still fresh in my head:

class PaperReadingLog:
    """Track your paper reading for retention and reference."""

    def log_paper(self, paper):
        """Fill this out after each paper you read past Pass 1."""
        template = {
            # Identification
            "title": paper.get("title"),
            "authors": paper.get("authors"),
            "year": paper.get("year"),
            "url": paper.get("url"),

            # Pass 1 output
            "one_sentence_summary": "",   # What does this paper do?
            "key_claim": "",              # What do they claim?
            "worth_deep_read": False,     # Should I do Pass 2?

            # Pass 2 output
            "main_idea_in_my_words": "",  # Explain it like you're teaching
            "key_equation_or_algorithm": "",
            "datasets_used": [],
            "baselines_compared": [],
            "main_result": "",            # e.g. "3.2% improvement over X on Y"
            "limitations_they_admit": "",
            "limitations_they_dont_admit": "",  # your critical reading

            # Relevance
            "relevant_to_my_work": "",    # How could I use this?
            "related_papers_to_read": [],
            "tags": [],                   # e.g. ["transformers", "efficiency"]
        }
        return template

That limitations_they_dont_admit field is the most important one in the whole template, and it is the one that separates a paper reader from a paper consumer. Every paper has weaknesses the authors gently walk past. Trained only on English data. Evaluated only on clean, curated benchmarks. Using a compute budget that no normal practitioner will ever have (eight A100s for three weeks -- cute). Learning to spot those quietly-omitted gaps is the entire game. Having said that, don't turn into a cynic who dismisses everything; the goal is calibrated skepticism, the same mindset we built around evaluation back in episode #13.

Reading math in ML papers

The notation in ML papers follows conventions, but -- and this is maddening -- those conventions are rarely spelled out. The authors assume you already speak the language. So here is a cheat sheet for the symbols you will bump into most often:

notation_guide = {
    # Scalars (lowercase, italic)
    "x, y, z": "individual data points or values",
    "w, b": "weight and bias (model parameters)",
    "alpha, beta, gamma": "hyperparameters (learning rate, etc.)",
    "epsilon": "small number (noise, numerical stability)",
    "theta": "all model parameters collectively",
    "eta": "learning rate (sometimes alpha instead)",

    # Vectors (lowercase, bold)
    "x (bold)": "feature vector for one data point",
    "h (bold)": "hidden state vector",

    # Matrices (uppercase, bold)
    "X (bold)": "data matrix (rows=samples, cols=features)",
    "W (bold)": "weight matrix",
    "Q, K, V": "query, key, value matrices (attention, see #51)",
    "I": "identity matrix",

    # Operations
    "x^T": "transpose",
    "||x||": "norm (usually L2/Euclidean)",
    "x . y or x^T y": "dot product",
    "sum_i": "sum over index i",
    "argmin_theta": "the theta that minimizes",
    "E[x]": "expected value (average over a distribution)",
    "P(x|y)": "probability of x given y (conditional)",
    "nabla": "gradient / partial derivative (calculus, see #9)",

    # Common ML-specific
    "L or J": "loss function",
    "D_KL": "KL divergence (distance between distributions)",
    "softmax(z)_i": "exp(z_i) / sum(exp(z_j))",
    "sigma(z)": "sigmoid: 1/(1+exp(-z))",
    "hat{y}": "predicted value (y-hat)",
    "x ~ P": "x is sampled from distribution P",
    "N(mu, sigma^2)": "normal distribution, mean mu, variance sigma^2",
}

When you hit unfamiliar notation, work through it in this order. First, check if the paper defines it (often buried in a paragraph right before the equation -- authors love hiding definitions in prose). If not, check the related work or the paper they are building on. Still stuck? The symbol is probably a field convention, so go look up the foundational paper for that technique. Half of "reading papers" is really "reading the paper this paper stands on".

Now here is my single favourite trick, and it has saved me more times than I can count: when an equation scares you, plug in tiny concrete numbers. Notation is abstract on purpose, but the mechanics are always concrete. Take the softmax-flavoured attention score from episode #51 -- instead of staring at the symbols, compute it by hand:

import math

# A toy "attention" step: one query, three keys, three values.
# This is exactly the machinery from episodes #51-#53, shrunk to fit in your head.
query = [1.0, 0.0]
keys = [[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]
values = [10.0, 20.0, 30.0]

# 1. Score = dot product of query with each key (how "relevant" is each key?)
scores = [sum(q * k for q, k in zip(query, key)) for key in keys]
print("raw scores:", scores)          # [1.0, 0.0, 1.0]

# 2. Softmax turns scores into weights that sum to 1
exps = [math.exp(s) for s in scores]
total = sum(exps)
weights = [e / total for e in exps]
print("weights:", [round(w, 3) for w in weights])  # [0.422, 0.155, 0.422]

# 3. Output = weighted sum of the values
output = sum(w * v for w, v in zip(weights, values))
print("attention output:", round(output, 3))       # ~20.0

Run that. Watch a single number come out. Suddenly "scaled dot-product attention" is not a mystical phrase from a famous paper -- it is three dot products, a softmax, and a weighted average, exactly like we built it earlier in this series. THAT is what plugging in numbers does. It drags the abstraction down to earth where you can kick it.

Where to find papers

You cannot read papers if you cannot find the good ones. Here are the sources I actually use, warts and all:

paper_sources = {
    "arxiv.org": {
        "what": "Preprint server - papers before peer review",
        "good_for": "Latest research, fast access, free",
        "watch_out": "No peer review - quality varies WILDLY",
        "tip": "Browse cs.LG (ML), cs.CL (NLP), cs.CV (vision)",
    },
    "paperswithcode.com": {
        "what": "Papers indexed by task, dataset, and method - with code",
        "good_for": "Finding SOTA for a task, getting implementations",
        "watch_out": "Leaderboard chasing can be misleading",
        "tip": "The 'Methods' pages map out technique lineages",
    },
    "semanticscholar.org": {
        "what": "Academic search with a citation graph + AI summaries",
        "good_for": "Finding related work, seeing who cites whom",
        "watch_out": "The AI summaries can be plain wrong",
        "tip": "Use 'Highly Influential Citations' to find the real follow-ups",
    },
    "conference_proceedings": {
        "what": "NeurIPS, ICML, ICLR, CVPR, ACL, EMNLP",
        "good_for": "Peer-reviewed, generally higher quality than random arXiv",
        "watch_out": "6-12 month delay from submission to publication",
        "tip": "Accepted-paper lists live on the conference sites, usually free",
    },
}

My honest recommendation for getting started (and I wish someone had told me this at the beginning): go to Papers With Code, find a task you genuinely care about -- object detection, speech recognition, whatever we covered that hooked you -- look at the top methods on that task, and read those papers. Starting from a task you already understand gives you context, and context is what makes a paper readable. Picking a random arXiv preprint on your first day is like opening a novel at chapter 30 and wondering why nothing makes sense.

Once you have a candidate, you still have to decide whether it earns a Pass 2. I use a dead-simple triage score so I am not agonising over every abstract:

def triage_score(paper):
    """Cheap heuristic: is this paper worth a deep read RIGHT NOW?
    Not gospel - just a nudge to stop me hoarding 400 open tabs."""
    score = 0
    if paper["relevant_to_my_current_work"]:
        score += 3          # relevance beats everything
    if paper["has_code"]:
        score += 2          # reproducible = trustworthy + useful
    if paper["from_top_venue"]:
        score += 1          # peer review is a weak but real signal
    if paper["citations_per_year"] > 50:
        score += 1          # the field voted with its feet
    if paper["you_understand_the_problem"]:
        score += 2          # context makes it readable

    if score >= 6:
        return "PASS 2 now - this is worth your afternoon"
    if score >= 3:
        return "PASS 1 only - skim, log, revisit if it keeps coming up"
    return "SKIP - bookmark it and move on, life is short"

print(triage_score({
    "relevant_to_my_current_work": True,
    "has_code": True,
    "from_top_venue": False,
    "citations_per_year": 12,
    "you_understand_the_problem": True,
}))  # -> "PASS 2 now - this is worth your afternoon"

Is this scoring scheme scientific? Absolutely not ;-) It is a crutch, and a good one. The point is to make the read/skip decision fast so you spend your energy reading, not dithering over which of forty tabs deserves attention.

Reproducing results: the real test

You do not truly understand a paper until you can reproduce its results. And I do not mean the full state-of-the-art number on ImageNet with a rack of GPUs -- I mean a simplified version that captures the core idea. Here is the systematic approach:

def reproduce_paper_checklist(paper_info):
    """Systematic approach to paper reproduction."""
    steps = {
        "step_1_find_code": {
            "action": "Check if the authors released code",
            "where": [
                "GitHub link in the paper or abstract",
                "Papers With Code linked implementations",
                "Author's personal website / lab page",
            ],
            "if_no_code": "You'll learn MORE implementing from scratch anyway",
        },
        "step_2_minimal_dataset": {
            "action": "Find or make a tiny dataset for testing",
            "why": "Full-scale reproduction is slow and expensive",
            "example": "CIFAR-10 instead of ImageNet, WikiText-2 instead of C4",
        },
        "step_3_implement_core": {
            "action": "Implement the KEY contribution only",
            "why": "Most papers are one core idea plus a lot of engineering",
            "example": "For the attention paper: build attention, skip the "
                       "full seq2seq system around it",
        },
        "step_4_verify_components": {
            "action": "Test each piece before wiring end-to-end",
            "checks": [
                "Does the forward pass produce the right output shape?",
                "Does the loss drop on a tiny batch (the overfit test)?",
                "Do gradients flow (no NaN, no silent vanishing)?",
            ],
        },
        "step_5_compare_trends": {
            "action": "Reproduce the qualitative TREND, not the exact number",
            "why": "Exact numbers need every hidden detail; the trend proves "
                   "you understood the idea",
            "example": "Does method A beat method B on your data, in the "
                       "same direction the paper claims?",
        },
    }
    return steps

That step 4 "overfit test" deserves a special mention, because it is the fastest sanity check in all of ML and almost nobody does it first. Before you worry about generalisation, prove your model can memorise a single tiny batch. If the loss will not go to near-zero on ten examples, your implementation is broken -- full stop, no point training on the real dataset yet. Here it is as a concrete pattern you can drop into any reproduction:

def overfit_single_batch(model, batch, loss_fn, optimizer, steps=200):
    """The #1 debugging test: can the model memorize ONE batch?
    If loss doesn't crater, your code is wrong - fix that before scaling."""
    x, y = batch
    for step in range(steps):
        optimizer.zero_grad()
        pred = model(x)
        loss = loss_fn(pred, y)
        loss.backward()
        optimizer.step()
        if step % 50 == 0:
            print(f"step {step:4d}  loss {loss.item():.6f}")
    # A correct model on a tiny batch should reach ~0 loss.
    # If it plateaus high: check shapes, learning rate, and label alignment.
    return loss.item()

A realistic expectation for a from-scratch reproduction: you will land maybe 80-90% of the paper's reported numbers, and that is completely fine. The missing 10-20% usually comes from hyperparameter tuning that cost the authors months, or from implementation details that simply never made it into the PDF. The point was never to match their digits -- it was to verify the idea works and that YOU understand why it works.

Building a reading habit

Reading papers is like going to the gym. Easy to start, hard to sustain, and all the benefit lives in the consistency, not in any single heroic session. Here is what has actually worked for me over the years:

Fixed schedule. One paper per week, same day, every week. I do Sundays. Put it in your calendar and treat it like a meeting you are not allowed to skip -- because the moment it becomes "when I have time", you will never have time.

Reading group. Even two people is enough. You each read the paper, then you argue about what you understood and, more importantly, what you didn't. Explaining a thing out loud to another person is the single fastest route to understanding it -- if you cannot find someone in person, online ML reading groups exist for basically every level.

Active reading. Don't read passively, ever. Take notes, draw the architecture, implement the core equation (that softmax-by-hand thing from earlier). After each section, ask yourself: "could I explain this to somebody right now?" If the answer is no, re-read that section before you move on. No skipping ahead and hoping it clicks later.

The 100-paper milestone. I will be honest with you: the first twenty papers are miserable. Papers twenty to fifty start getting easier because you begin recognising the recurring patterns. Fifty to a hundred, you develop opinions (a dangerous and wonderful stage). After a hundred, you can skim most papers in ten minutes because you have already seen the ingredients before, just recombined. There is no shortcut around this. It is reading hours, plain and simple.

reading_habit = {
    "schedule": "1 paper per week, same day every week",
    "selection": [
        "Alternate: foundational classic, recent breakthrough, niche curiosity",
        "Prefer papers relevant to your current work (built-in motivation)",
        "Follow citation chains (X cites Y cites Z - go read Z)",
    ],
    "active_reading": [
        "Write a 3-sentence summary after Pass 1",
        "Explain the method in your own words after Pass 2",
        "Implement the core idea (even partially) once a month",
    ],
    "milestones": {
        10:  "You can identify a paper's type and contribution quickly",
        25:  "You recognize common notation and structure without a reference",
        50:  "You start having opinions about methodology choices",
        100: "You can read most papers at Pass-1 level in 10 minutes",
    },
}

From reader to contributor

At some point -- and it sneaks up on you -- you will have an idea that is not in any paper you have read. Maybe a novel combination of two techniques. Maybe a method applied to a domain nobody tried it on. Maybe just a cleaner way to do something you have watched people do inefficiently a dozen times. That is the moment you shift from reading papers to writing one.

The beautiful part: the structure is the same one we dissected at the top of this episode. You already know the skeleton -- now you are the one filling it in:

  • Abstract -- what you did, why it matters, the key result. Write this LAST, once you actually know what you found.
  • Introduction -- the problem, why existing methods fall short, your contribution.
  • Method -- your technique, explained clearly enough that a stranger could implement it from your words alone. This is the section that earns respect.
  • Experiments -- a FAIR comparison against strong baselines on standard benchmarks. Weak baselines fool nobody who reads the ablation.
  • Conclusion -- summary plus limitations. Be honest about the weaknesses; the reviewers will find them regardless, and admitting them first reads as confidence, not weakness.

You do not need a PhD to publish. arXiv is open to everyone. Workshop papers at the big conferences have a distinctly lower bar than the main track and are a genuinely great place to plant your first flag. The whole thing hinges on one requirement: a clear contribution. A thing you can point at and say "this did not exist before, and here is evidence that it works". That is it. That is a paper.

And this connects to where we are headed. You have spent this series building -- from linear regression in #10 all the way up through transformers, diffusion, RL, and shipping real products. The next step is learning to feed yourself from the research firehose without drowning in it, so all those pieces you have built click into one coherent picture, and so you know where to point everything next. We are on the last stretch of the road now ;-)

What to remember from all this

  • Papers follow a predictable skeleton -- abstract, intro, related work, method, experiments, ablation, conclusion -- and knowing the skeleton tells you exactly where to look for what you need;
  • use the three-pass strategy: survey (5-10 min, decides if it is worth your time), understanding (30-60 min, extract the core idea), reproduction (hours-days, only for papers you will actually use);
  • the ablation study is the most honest section in most papers -- read it before you trust the headline results;
  • when the math scares you, substitute tiny concrete numbers into the equation (like the by-hand attention example) -- notation is just a language, and it gets easier with exposure;
  • start from Papers With Code for task-specific work, use Semantic Scholar for the citation graph, and lean on conference proceedings for peer-reviewed quality;
  • reproduction means matching the qualitative TREND, not the exact digits -- and always run the overfit-one-batch test before you scale anything;
  • build a weekly habit and grind through the first fifty papers; after that the notation turns familiar, the structures turn recognisable, and papers go from intimidating to genuinely useful.

Bedankt for reading all the way to the bottom -- now go pick ONE paper behind a technique from this series, run it through Pass 1, and log your three sentences. Tot de volgende keer! ;-)

@scipio

stem stemsocial steemstem python programming
0
0
0.000
1 comments
avatar

Congratulations @scipio! You have completed the following achievement on the Hive blockchain And have been rewarded with New badge(s)

You have been a buzzy bee and published a post every day of the week.

You can view your badges on your board and compare yourself to others in the Ranking
If you no longer want to receive notifications, reply to this comment with the word STOP

0
0
0.000
Reply