Learn AI Series (#148) - The Economics of AI

avatar

Learn AI Series (#148) - The Economics of AI

variant-c-07-purple.png

What will I learn

  • AI and labor markets -- why AI eats TASKS and not whole jobs, why this wave hits white-collar cognitive work in a way no previous wave did, and why "can you use the tool" beats "can you outrun the tool";
  • winner-take-most dynamics -- the three compounding loops (data flywheel, scale economics, talent concentration) that decide who ends up on top and why the foundation-model layer is an oligopoly while the application layer stays a knife-fight;
  • the moat question -- open source versus closed, why the model weights are NOT the moat, and what the real moats actually are;
  • the compute supply chain -- chips, cloud, models, applications, and the "smiling curve" that says where the margins actually live;
  • business models for AI -- API-as-a-service, AI-enhanced SaaS, vertical products, picks-and-shovels, data licensing, and how to tell a business apart from a feature;
  • cost deflation -- the single most important economic fact about AI, roughly 10x per year, and why it quietly rewrites every build-versus-buy decision you will ever make.

Requirements

  • A working modern computer running macOS, Windows or Ubuntu;
  • Python 3.10+ with PyTorch installed (pip install torch) -- every snippet here runs on a plain CPU in a second or two, no GPU needed anywhere;
  • You have read #134 (AI infrastructure economics) and #135 (building AI teams and processes), because this episode zooms out from "what does the hardware cost" to "who captures the value." It also helps to remember #61 (RLHF) and #127 (AI security), since we settle last week's homework on both before we start.

Difficulty

  • Beginner

Curriculum (of the Learn AI Series):

Learn AI Series (#148) - The Economics of AI

I closed #147 by admitting we had carefully tiptoed around a giant question the whole time we talked about safety. We spent that episode building brakes -- the KL leash, scalable oversight, Constitutional AI, red teaming -- and I kept insisting these are engineering problems, not seminar topics. But which brakes actually get bolted onto a shipping product? That is not decided by the researcher who invented them. It is decided by a spreadsheet. Follow the money and half of this field suddenly makes sense, so let us finally follow it ;-)

This is a lighter episode on the PyTorch, heavier on the thinking. We have spent 147 episodes learning how to build these systems. Now we look at what happens when they collide with the economy: who captures the value, who loses it, and why the answer decides which of your beautiful models ever leaves the notebook. Understanding this will make you a better practitioner, because "can I build it" and "should I build it" are two very different questions, and the second one pays your rent.

Solutions to episode #147's exercises

House rules first -- we settle last week's homework before we open anything new. #147 was safety and alignment, and all three tasks were about watching optimization misbehave and then measuring the misbehaviour.

Exercise 1 -- Put the reward hacker on the leash. Take the reward-model overoptimization loop from #147, wrap the KL penalty around the inner optimization using the untouched starting point as the "reference," sweep beta across a few values, and print the final proxy reward and true quality for each.

import torch
import torch.nn as nn

torch.manual_seed(0)

def true_quality(x):
    return -(x ** 2).mean(dim=-1, keepdim=True)   # peaks when x is near zero

rm = nn.Sequential(nn.Linear(32, 64), nn.ReLU(), nn.Linear(64, 1))
data = torch.randn(512, 32) * 0.5
opt = torch.optim.Adam(rm.parameters(), lr=1e-2)
for _ in range(300):
    loss = ((rm(data) - true_quality(data)) ** 2).mean()
    opt.zero_grad(); loss.backward(); opt.step()

ref = torch.zeros(1, 32)                           # the sensible reference point
for beta in (0.0, 0.1, 1.0):
    x = torch.zeros(1, 32, requires_grad=True)
    inner = torch.optim.Adam([x], lr=0.05)
    for _ in range(120):
        reward = rm(x)
        kl = ((x - ref) ** 2).sum()                # proxy for drift from reference
        inner.zero_grad(); (-(reward - beta * kl)).backward(); inner.step()
    print(f"beta {beta:>4}: proxy {rm(x).item():+.3f}  TRUE {true_quality(x).item():+.3f}")

At beta = 0.0 the optimizer runs free, the proxy reward soars and true quality face-plants -- that is Goodhart in one line. At beta = 1.0 the leash is so short the point barely moves off the reference, so true quality stays fine but you learned almost nothing. The middle value is the sweet spot: beta = 0.1 protects true quality while still letting the point improve, and the extremes fail for the mirror-image reasons -- too loose and it hacks, too tight and it is frozen.

Exercise 2 -- Judge a one-round debate. Two debaters each return a number as their claim about a hidden answer; a judge picks the claim closer to a value it can cheaply verify but not produce alone. Show that with at least one honest debater the judge beats a coin flip.

import torch

torch.manual_seed(1)

def debate_round(truth, honest_a, honest_b):
    # an honest debater reports near-truth; a liar reports something far off
    claim_a = truth + (0.1 if honest_a else 5.0) * torch.randn(1)
    claim_b = truth + (0.1 if honest_b else 5.0) * torch.randn(1)
    # the judge cannot GENERATE the truth, but given a noisy check it can VERIFY closeness
    check = truth + 0.3 * torch.randn(1)
    pick_a = (claim_a - check).abs() < (claim_b - check).abs()
    winner = claim_a if pick_a else claim_b
    return (winner - truth).abs().item() < 1.0     # did the judge land near truth?

trials = 2000
wins = sum(debate_round(torch.randn(1), True, False) for _ in range(trials))
print(f"one honest debater: judge correct {wins / trials:.1%} of the time")

With one honest debater in the room the judge lands near the truth far more often than the 50% a coin would give you, because the honest claim sits close to the verification signal and the lie sits nowhere near it. The load-bearing assumption, in two sentences: the whole trick works only because VERIFYING a claim against a cheap check is easier than GENERATING the right answer from scratch. If judging were as hard as answering, debate would buy you nothing and you would be back to needing a superhuman referee.

Exercise 3 -- Measure a red team. A target that is "unsafe" whenever an attack contains a banned keyword, a keyword-stuffing attacker that includes the banned word with probability p, and a classifier that detects it. Run for p = 0.2 and p = 0.8 and confirm the reported success rate tracks p.

import random

random.seed(0)
BANNED = "detonate"

def target(attack):
    return "sure, here is how" if BANNED in attack else "I cannot help with that"

def attacker(p):
    return f"please {BANNED} the thing" if random.random() < p else "please help me bake"

def classifier(response):
    return response.startswith("sure")           # flags an unsafe completion

def red_team(p, n=2000):
    hits = sum(classifier(target(attacker(p))) for _ in range(n))
    return hits / n

for p in (0.2, 0.8):
    print(f"p={p}: measured attack success rate {red_team(p):.1%}")

The measured success rate lands right on top of p, which is the point -- your automated number faithfully reports how often THIS attacker got through. Two sentences on the trap: a LOW automated success rate is not proof the model is safe, it is proof this particular attacker is weak. A smarter attacker, a new jailbreak, or a banned concept your classifier never learned to spot would all sail straight past a green dashboard, which is exactly why #147 insisted red teaming is continuous and never a box you tick once. Right -- homework settled. Now let us talk money ;-)

AI and labor markets

Let us start with the question every single person asks me at parties the second they hear what I do: will AI take my job?

The honest answer, from the economic evidence so far, is more interesting than the headline. AI does not typically eliminate whole jobs. It eliminates tasks. A job is a bundle of tasks stapled together, and if AI automates 30% of that bundle it does not delete your job -- it restructures it. You spend less time on the automated slice and more on the parts the machine still cannot touch.

The pattern is remarkably consistent across a century of automation. The classic example is the ATM. Everybody assumed the cash machine would wipe out bank tellers, and yet the number of tellers actually GREW for decades afterwards, because ATMs made a branch cheaper to run, so banks opened more branches, so they hired more tellers. But the job changed underneath them -- tellers stopped counting bills and started selling financial products. The task got automated. The human got redeployed.

Having said that, I am not going to hand you the comfortable "nothing to worry about" line, because the speed and BREADTH of this wave really is different. Previous automation went after manual work and routine cognitive work -- the predictable, repetitive stuff. AI goes after NON-routine cognitive work: writing, analysis, coding, design, the things we told a generation of students were future-proof. That hits a different demographic entirely -- educated, white-collar, well-paid -- and it hits fast. The most exposed roles right now are customer service, content writing, translation, first-pass legal analysis, routine financial reporting, and entry-level programming.

The economists' framing is the useful one: AI is a skill-biased technology. It rewards the people who can USE it over the people forced to compete AGAINST it. Watch the difference with a toy model of two workers facing the same automated slice.

def annual_output(base_tasks, ai_multiplier, adoption):
    """A worker's effective output when part of the job is AI-augmentable.
    adoption in [0,1]: how much of the augmentable work you actually leverage."""
    augmentable = 0.4 * base_tasks          # 40% of the job is AI-augmentable
    manual = base_tasks - augmentable
    boosted = augmentable * (1 + adoption * (ai_multiplier - 1))
    return manual + boosted

adopter = annual_output(100, ai_multiplier=5.0, adoption=1.0)   # fully leverages AI
refuser = annual_output(100, ai_multiplier=5.0, adoption=0.0)   # ignores the tools
print(f"AI adopter effective output: {adopter:.0f}")
print(f"AI refuser effective output: {refuser:.0f}")
print(f"productivity gap: {adopter / refuser:.2f}x")

The adopter does not replace ten colleagues and cackle -- they simply produce a couple times more of the valuable work and get handed more of it. The refuser competes on raw speed against a machine that never sleeps, which is not a winning position on any timeline. So the honest career advice is boring but true: the threat is not "AI," the threat is "a person using AI who does your job." Become the first kind of person.

Winner-take-most dynamics

Zoom out from the worker to the market, and AI has a strong tendency toward winner-take-most outcomes. Not always winner-take-ALL, but heavily concentrated. Three reinforcing loops drive it.

The data flywheel. More users generate more data, more data trains a better model, a better model attracts more users, and round it spins. This is why an incumbent search engine stays dominant -- the advantage compounds on itself. Let us actually watch it compound, because "compounding" is one of those words people nod at without feeling.

def data_flywheel(rounds=8, leader0=1.0, follower0=0.9, capture=0.15):
    """Each round, model quality (proportional to accumulated data) wins new users,
    and those users generate fresh data. Small early edge -> runaway gap."""
    leader, follower = leader0, follower0
    for r in range(rounds):
        # quality-weighted share of this round's new users
        total = leader + follower
        leader += capture * (leader / total)      # winner grabs more of the new data
        follower += capture * (follower / total)
        print(f"round {r+1}: leader {leader:.3f}  follower {follower:.3f}  "
              f"gap {leader - follower:+.3f}")
    return leader, follower

data_flywheel()

Start the leader a mere 10% ahead and let the loop run: the gap does not close, it WIDENS, because the one with slightly more data wins slightly more of every new cohort, which gives it slightly more data still. A tiny head start turns into a structural moat purely through feedback. Nota bene: this is also exactly why being second by a year in a data-driven market is so brutal -- you are not one year behind, you are one year of compounding behind.

Scale economics. Training a frontier model costs somewhere between 100 million and a billion-plus dollars. But serving one inference costs a fraction of a cent. That cost structure -- gigantic fixed cost, near-zero marginal cost -- is the classic recipe for concentration, because the average cost per user only makes sense once you have an enormous number of users to spread the fixed cost across.

def avg_cost_per_user(train_cost, marginal_cost, users):
    return (train_cost + marginal_cost * users) / users

train = 300_000_000        # 300M to train the model
marginal = 0.002           # a fifth of a cent per inference
for users in (10_000, 1_000_000, 100_000_000):
    print(f"{users:>11,} users: ${avg_cost_per_user(train, marginal, users):.4f} per user")

At ten thousand users the model is economically insane -- thirty thousand dollars a head. At a hundred million users the same model costs a rounding error above the marginal two-tenths of a cent. The maths simply does not work for a small player, and it works beautifully for whoever already has distribution. Fixed costs love scale and punish everyone else.

Talent concentration. The best researchers want to be where the best compute, the best data, and the best colleagues already are. So talent clusters at a handful of shops, which widens the capability gap, which attracts more talent. Another flywheel, running on people in stead of bytes.

Stack the three loops together and you get the shape of the real market: a small oligopoly trains the foundation models (the big labs from #137), and a much larger crowd builds applications on top. The foundation-model layer is a few giants. The application layer is a competitive scrum. If you work at a startup, you are almost certainly in the scrum -- and that is FINE, because the scrum is where most of the value gets created for actual users. Your edge there is domain knowledge, proprietary data, distribution and user experience, not a base model you were never going to train anyway.

Open source vs closed: the moat question

The endless open-versus-closed debate is really one question wearing a costume: where is the moat, the sustainable advantage a competitor cannot cheaply copy?

The case for closed. If you burned 500 million dollars training a model, publishing the weights hands your competitors the finish line for free. Keeping it closed lets you meter access, price per token, track usage, and control quality. That is the API-first path several big labs walk.

The case for open. The instructive move is a company that gives its models away -- and it is not charity, it is strategy. If your actual business is advertising, not model APIs, then open-sourcing a strong model COMMODITIZES the model layer. Suddenly everyone fine-tunes and builds on YOUR base, the companies who wanted to SELL model access find their product is now free, and the advantage slides to whoever owns the best data and distribution -- which, conveniently, is you. Commoditize your complement, as the old strategy line goes.

The reality that matters for you. The moat in AI is NOT the weights. Weights get replicated with enough compute and data -- today's frontier model is next year's open download. The durable moats are elsewhere:

  1. Proprietary data -- the data nobody else has. This is why a financial-data giant trains its own model on its own corpus, and why every serious enterprise wants to fine-tune on its internal documents.
  2. Distribution -- getting the model in front of users. Baking a model into an operating system, an office suite, or a cloud platform a billion people already open is a moat no startup can dig.
  3. Ecosystem lock-in -- once users build workflows, fine-tunes and integrations around your platform, the switching cost keeps them parked.
  4. Speed of iteration -- shipping improvements faster than the other guy. That is an organisational muscle, not a model checkpoint.

Points three and four deserve a number attached, because "switching cost" sounds soft until you model it against price. Watch a customer decide whether to jump ship to a 20% cheaper competitor when leaving costs them real switching pain.

def will_switch(price_incumbent, price_rival, switching_cost, months=24):
    """A rational customer switches only if cumulative savings beat the one-time pain."""
    savings = (price_incumbent - price_rival) * months
    return savings > switching_cost, savings

for lockin in (50, 500, 5000):
    switch, saved = will_switch(price_incumbent=100, price_rival=80, switching_cost=lockin)
    print(f"switching cost ${lockin:>5}: saves ${saved} over 2yr -> switch? {switch}")

Cheap lock-in and the customer walks for a better price without blinking. Make the lock-in deep enough -- integrations, trained staff, migrated data -- and the SAME 20% discount is not worth the migraine of moving. That is why the boring enterprise features (SSO, audit logs, data residency, workflow integrations) are frequently a stronger moat than a two-point bump on a benchmark. The model gets you in the door; the lock-in keeps you in the building.

The compute supply chain

The AI supply chain is startlingly concentrated and geographically fragile. Walk it layer by layer:

  • Chip design. One company dominates GPUs for training with north of 80% share; a second is a distant runner-up; the big clouds build their own vertically-integrated accelerators as alternatives.
  • Chip manufacturing. Essentially all the cutting-edge AI silicon is fabricated by a single company, on a single island. That is a genuine single point of failure and it shapes national AI strategy at the level of foreign policy.
  • Cloud infrastructure. Three hyperscalers provide most of the compute. Training a frontier model means thousands of accelerators running for months. And here is the twist -- the clouds are ALSO AI companies, so they are simultaneously your landlord and your competitor.
  • Model layer. A handful of labs train frontier models; hundreds fine-tune and adapt them.
  • Application layer. Thousands of companies build products. This is where most of us actually work.

Value capture across those layers follows what strategists call the smiling curve: fat margins at the ends (the scarce chips, and the applications that solve a real problem), thin margins in the commoditized middle. Let us plot the smile.

layers = [
    ("chip design",      0.65),
    ("chip fab",         0.55),
    ("cloud compute",    0.30),
    ("foundation model", 0.35),
    ("application",      0.60),
]
peak = max(m for _, m in layers)
for name, margin in layers:
    bar = "#" * int(margin / peak * 30)
    print(f"{name:>16} | {bar} {margin:.0%}")

Print it and you literally see the smile -- high at the chip end, high at the application end, sagging through the commodity infrastructure in the middle. The lesson is not "avoid the middle" (someone has to run the datacenters), it is "know which end of the curve your business sits on, and price accordingly." If you are competing in the sagging middle, you had better be the lowest-cost operator, because you are selling a commodity.

Business models for AI

So how do companies actually MAKE money from this stuff? A handful of models, each with its own economics:

API-as-a-service. Charge per token, per image, per inference. Clean, understandable, scales with usage. The risk is savage commoditization -- a given quality level gets 10 to 100 times cheaper within a year or two, so pure API margins compress relentlessly.

AI-enhanced SaaS. Existing software bolts AI onto what it already sold -- the coding assistant, the writing helper, the design tool. The AI is a FEATURE, folded into an existing subscription. This is the most common and arguably the most durable model, because you are making a product people already pay for measurably better.

Vertical AI products. Systems built for one industry with domain-specific data and workflows -- legal, medical, financial. Higher margins, because the moat is the domain expertise and the data, not the base model.

Picks and shovels. Infrastructure and tooling other AI companies build on -- experiment tracking (#119), data labeling, model hosting (#74). You do not have to win the gold rush if you sell the shovels to everyone digging.

Data licensing. Selling or licensing the data the model-builders need. A forum licensing its posts, a stock library licensing its images. If you own unique data, you own a genuinely new kind of asset.

The practitioner's takeaway is blunt: an AI WRAPPER around a commodity API is not a business, it is a feature waiting to be absorbed by the platform it wraps. A business has proprietary data, deep domain integration, and real switching costs. Same model underneath, wildly different economics on top.

The cost curve

One last force, and it is the big one -- bigger than any single company or model. Costs are falling, fast. The price of a token of LLM inference has dropped on the order of 10x per YEAR since the early GPT-3 era, and training costs for an equivalent capability fall on a similar slope. This deflation quietly rewrites your decisions, and you can watch it flip a build-versus-buy call all by itself.

def buy_cost(monthly_tokens, price_per_1k, year):
    price = price_per_1k * (0.1 ** year)          # ~10x cheaper each year
    return monthly_tokens / 1000 * price

def build_cost():
    return 8000                                    # fixed monthly: GPUs + an engineer

monthly_tokens = 500_000_000
for year in range(4):
    buy = buy_cost(monthly_tokens, price_per_1k=2.0, year=year)
    cheaper = "BUY the API" if buy < build_cost() else "BUILD it yourself"
    print(f"year {year}: buy=${buy:>10,.0f}  build=${build_cost():,}  -> {cheaper}")

In year zero the API is eye-wateringly expensive at your volume, so self-hosting wins and you staff up an infra team. A few years later the exact same API is orders of magnitude cheaper and the decision quietly INVERTS -- you are now burning salary to lose to a commodity. This is why "we built our own to save money" ages so badly, and why a project that was uneconomical last year deserves a fresh look today. Three consequences fall straight out of the curve:

  1. Projects that made no financial sense a year ago may be obvious now.
  2. Build-versus-buy is not a one-time decision, it is a thing you re-evaluate on a schedule.
  3. A proprietary MODEL advantage erodes quickly -- today's frontier is tomorrow's commodity, so build your moat out of data and distribution, not weights.

Cost deflation is, I argue, the single most important economic fact about AI. It guarantees adoption keeps accelerating, guarantees pure-API margins keep compressing, and pushes the durable value toward the application and data layers where you probably work anyway. Plan as if the compute you rely on will be far cheaper next year -- because it will be.

Exercises

Get your hands dirty before the next one. Three tasks, climbing in difficulty:

  1. Tune the leash economically. Take the data_flywheel function and add a per-round COST the leader pays to keep capturing share (say, marketing spend proportional to how much new share it grabs). Sweep a couple of cost levels and report the round at which the leader's cumulative spend stops being worth the share it bought. One sentence on why "winner-take-most" does not mean "winning at any price."

  2. Find the break-even user count. Using avg_cost_per_user, write a loop that finds the smallest user count at which average cost drops below a target price (say 5 cents per user) for train_cost of 100M and 1B. Print both break-even points. Two sentences on what that break-even gap says about who can afford to train a frontier model versus fine-tune one.

  3. Race the cost curve. Extend the buy_cost / build_cost comparison so the BUILD cost also falls over time (hardware gets cheaper too, just slower -- try 30% cheaper per year instead of 10x). Find the year where buy overtakes build permanently, and in two sentences explain why a slower-deflating fixed cost still loses to a faster-deflating variable cost in the long run.

We open next episode with full solutions, as always.

Quick recap

  • AI eats tasks, not jobs -- but the breadth and speed of this wave hits non-routine cognitive work in a way no previous automation did, and the real threat to your career is a person USING AI, not AI itself;
  • winner-take-most is driven by three compounding loops -- the data flywheel, scale economics (huge fixed cost, near-zero marginal cost), and talent concentration -- which is why the foundation layer is an oligopoly and the application layer is a scrum;
  • the moat is not the weights -- weights get replicated; the durable moats are proprietary data, distribution, ecosystem lock-in, and speed of iteration, and lock-in beats a 20% discount once it is deep enough;
  • the supply chain is concentrated and fragile -- one dominant GPU designer, essentially one advanced fab, three clouds who are also your competitors -- and value follows the smiling curve, fat at the chip and application ends, thin in the middle;
  • business models run from API-as-a-service to vertical products, and the ones that last capture value through data and domain expertise, not commodity model access -- a wrapper is a feature, not a business;
  • cost deflation (~10x per year) is the dominant force -- it makes dead projects viable, flips build-versus-buy on a schedule, and erodes any moat you tried to build out of model weights.

And the thread I will leave dangling for next time, because it is the natural next step and it has been quietly poisoning everything in this episode. Every economic decision we just made -- automate this task, capture that value, cut this cost, license that data -- lands on a real person. The cost curve that makes your product cheaper is partly paid by the humans labeling the training data. The winner-take-most market that concentrates capability also concentrates POWER over who gets served and who gets skipped. We already did the theory of bias and fairness way back in #35, but theory is the easy part. What does doing the right thing look like when the spreadsheet says otherwise, when the money and the morality point in opposite directions? That is where we head next, and it is going to be a good deal less comfortable than counting margins ;-)

Bedankt voor het lezen! Do exercise 3 -- race the two cost curves in your own terminal and watch buy overtake build, because feeling that inversion happen is worth more than me telling you it does. Tot de volgende keer! ;-)

@scipio

stem stemsocial steemstem python programming
0
0
0.000
0 comments