Anatomy of Newer LLMs: What Has Changed Over Time

There’s this weird thing happening in AI right now. If you look at the architecture diagrams of GPT-3 from 2020 and DeepSeek-V3 from 2024, they look… almost the same. Both are transformers. Both use attention. Both predict the next token.

And yet, DeepSeek-V3 with 671 billion parameters costs about $6 million to train, runs on one-tenth the compute during inference, and outperforms models that cost 100x more to build. Meanwhile, OpenAI’s o3 solves math problems that would stump most PhD students by “thinking” for extended periods before answering. Claude 4 can read your entire codebase, all 75,000 lines, in a single pass and still remember what function you wrote on line 347.

Something changed. Actually, a lot of things changed.

The transformer architecture hasn’t been replaced; it’s been refined, optimized, and reimagined in ways that would make its 2017 creators do a double-take. We’re not in the era of “make it bigger” anymore. We’re in the era of “make it smarter.”

Let me show you what’s really going on under the hood.

The Core That Stayed (And Why That Matters)

The Transformer, introduced in the 2017 paper “Attention Is All You Need,” remains the foundation. Self-attention. Feed-forward networks. Positional encodings. The basic recipe hasn’t changed because, frankly, it works too well to abandon.

But here’s the thing: between 2022 and 2025, we’ve seen the equivalent of going from a Model T to a Tesla while technically both are still “cars with four wheels.” The chassis is the same; everything else got a serious upgrade.

Why keep the transformer? Because it’s embarrassingly parallel (great for GPUs), it scales predictably, and most importantly, we understand its failure modes. Starting from scratch means relearning all those hard lessons. Better to optimize what works than gamble on something untested at billion-dollar scales.

From Brute Force to Surgical Precision: Mixture of Experts

Let’s talk about one of the biggest architectural shifts: Mixture of Experts, or MoE.

The problem with dense models is simple: you’re activating every single parameter for every single token. That’s like turning on every light in your house just to read a book in one room. It works, but it’s wasteful.

MoE says: What if we only activate the parts of the model we actually need?

DeepSeek-V3 is the poster child for this approach. It has 671 billion total parameters, but only 37 billion are active for any given token. Think about that: you get the knowledge capacity of a massive model with the computational cost of a much smaller one. They trained it for just $6 million. For context, GPT-4 reportedly cost over $100 million.

Here’s how it works: the model has multiple “expert” networks, and a router (called a gating mechanism) decides which experts to activate based on the input. For DeepSeek-V3, they use 256 experts per layer, activating only 8 at a time, plus 1 shared expert that’s always on. It’s like having a team of specialists where each token gets routed to the doctors it actually needs to see.

Mixtral 8x7B does something similar — 8 experts, activate 2 per token. Total parameters: 47B. Active per token: 13B. The result? Performance comparable to much larger models, but you can actually run it on consumer hardware.

Llama 4 (released April 2025) made the jump to MoE architecture too. Meta’s Scout model has 109B total parameters with 16 experts, but only 17B active. The Maverick variant cranks it up to 400B total with 128 experts.

The challenge isn’t just building these expert systems: it’s load balancing. Early MoE models would sometimes route everything to just a few “favorite” experts while others sat idle. This routing collapse defeats the whole purpose.

Traditional solutions used auxiliary loss functions to encourage balanced routing, but these hurt model quality. DeepSeek-V3 pioneered an auxiliary-loss-free approach using bias terms that manually adjust when experts become overloaded. It’s messier from a load balancing perspective, but the model performs better overall. Sometimes the messy solution is the right solution.

The real impact? About 70% compute reduction compared to dense models of similar capability. That’s not incremental: that’s transformative.

Attention Mechanisms: The Memory Efficiency Revolution

Attention is the heart of transformers, but it’s also the memory bottleneck. The KV (key-value) cache, where the model stores past tokens for context, grows linearly with sequence length. Long conversations? Long documents? Your memory usage explodes.

This is where we’ve seen some of the cleverest optimizations.

Multi-Query Attention (MQA) was the first big breakthrough. Instead of having separate key and value heads for every query head, MQA shares a single K-V head across all queries. This reduces the KV cache by 10–100x and speeds up inference by up to 12x. The trade-off? Slight quality degradation because you’re compressing information.

Grouped-Query Attention (GQA) found the sweet spot. It groups multiple query heads to share K-V heads. It’s the Goldilocks solution: better quality than MQA, more efficient than full multi-head attention. Llama 2, Mistral 7B, and Llama 3 all adopted GQA. You get 2–4x cache reduction without major accuracy loss. Meta explicitly called out GQA as key to maintaining efficiency in Llama 3 despite adding parameters.

Multi-Head Latent Attention (MLA) is DeepSeek’s innovation. Instead of storing full K-V representations, MLA compresses them into a low-rank latent space. You store the compressed version in cache, then project back to full size when needed. It’s like zipping files — you save memory without losing the information when you unzip.

Cross-Layer Attention (CLA) takes it further: share K-V cache vertically across layers. Why recompute similar representations at every level? This can halve your cache usage.

The cumulative effect of these techniques is staggering. A model with MLA can handle much longer contexts in the same memory footprint, which is why DeepSeek-V3 and Claude 4 can process such massive context windows without melting your GPU.

Positional Encoding: Why RoPE Won

Early transformers used absolute positional encodings: basically telling the model “this is token 5, this is token 1000.” It worked, but it didn’t generalize well beyond the training length.

Enter RoPE (Rotary Position Embedding). Instead of adding position information, RoPE rotates the query and key vectors in a way that naturally encodes relative positions. The math is elegant (it uses complex-valued rotations), but the practical benefit is straightforward: models can extrapolate to longer sequences than they were trained on.

Nearly every modern LLM uses RoPE now: Llama, Mistral, GPT-NeoX, DeepSeek. It’s become the de facto standard because it just works. When Meta says Llama 3 can handle 8K tokens during training but extrapolates to longer sequences at inference, RoPE is why.

There are variations (ALiBi, xPos, YaRN) that tweak the approach, but RoPE’s core insight (encode position through rotation) became the foundation everyone builds on.

Activation Functions: The Silent Revolutionaries

This one’s less sexy but more important than you’d think.

We went from ReLU (max(0, x)) to GELU (smooth approximation of ReLU) to SwiGLU, and each step brought measurable improvements.

SwiGLU combines Swish activation (a smooth, non-monotonic function) with a gating mechanism. The “GLU” part means Gated Linear Unit: one part of the network decides what information to let through, another processes it. It’s like having a bouncer and a bartender instead of just a bartender.

Why does this matter? Models using SwiGLU consistently show 3–7 point improvements on benchmarks. Llama, Mistral, PaLM, Apple Intelligence Foundation models: they all use it now.

The trade-off: SwiGLU requires about 50% more parameters in the feed-forward network. But the performance gain is worth the cost. That said, it causes headaches in low-precision training (FP8) because it can create activation spikes that are hard to quantize accurately.

The Training Revolution: Beyond Just Scaling

Here’s where things get really interesting. The biggest shift isn’t just in architecture — it’s in how we train these models.

RLVR: A New Way to Think

Reinforcement Learning with Verifiable Rewards (RLVR) is changing the game for reasoning models.

Traditional RLHF (Reinforcement Learning from Human Feedback) used human preferences: “This answer is better than that answer.” It’s subjective. It’s expensive. It’s slow.

RLVR says: for tasks with verifiable outcomes (math, coding, logic) use binary rewards. Right or wrong. Compile or error. Proof valid or invalid.

DeepSeek-R1 was trained entirely with RLVR using their Group Relative Policy Optimization (GRPO) algorithm. No supervised fine-tuning datasets. Just raw RLVR. The model learned to generate extended reasoning traces by being rewarded for correct final answers.

The mechanism is clever: during training, the model generates long chains of thought. If the final answer is correct, the entire sequence gets a positive reward. The model learns that detailed reasoning usually leads to correct answers, so it starts “thinking” more carefully.

OpenAI’s o1 and o3 use similar approaches. They generate internal chains of thought (hidden from users) before producing final answers. On the ARC-AGI benchmark (a visual reasoning test designed to be resistant to memorization) o3 scored 75.7% in low-compute mode. For context, the previous state-of-the-art was around 50%, and human performance is 85%. We’re entering territory where AI genuinely reasons through novel problems.

But there’s debate about what’s actually happening. The optimistic view: RLVR expands reasoning capabilities by teaching models to break down problems systematically. The skeptical view: it optimizes sampling strategies without adding new capabilities. When you give base models enough attempts (high pass@k), they often outperform their RLVR-trained counterparts.

The truth is probably somewhere in between. RLVR makes models more reliably produce correct reasoning on the first try, which is what matters in practice.

Test-Time Compute Scaling: A New Frontier

Here’s a fundamental shift: we’re moving from training compute to inference compute.

For years, the equation was simple: bigger model + more training = better performance. But there’s a ceiling to how big you can make models before economics and physics intervene.

Test-time compute scaling says: let the model think longer during inference.

Chain-of-Thought (CoT) was the first step: explicitly prompting models to reason step-by-step. It worked shockingly well. Just saying “Let’s think step by step” improved accuracy on math problems by 20–30%.

Tree-of-Thought (ToT) extends this: explore multiple reasoning paths simultaneously, like a chess engine evaluating different move sequences.

Forest-of-Thought (FoT) goes further: multiple trees make collective decisions, like a committee of experts voting.

The o3 model demonstrates this dramatically. It can be configured for different levels of compute at test time: low, medium, or high. More compute means longer thinking, which means better accuracy on hard problems. On the hardest competition math problems, this approach lets smaller models outperform ones 14x larger.

The controversial part: are we discovering new capabilities or just brute-forcing through trial and error? Early evidence suggests that AI often finds correct answers through shorter reasoning chains than humans expect. The “underthinking” phenomenon (where the model gets it right without elaborate reasoning) suggests something interesting is happening beyond just exhaustive search.

Memory and Efficiency: The Pragmatic Innovations

All these fancy techniques are great, but you still need to fit models in actual hardware.

KV Cache Optimization

The cache problem never goes away: it just gets worse with longer contexts. Engineers have thrown everything at it:

• Quantization: Store values in lower precision. FP16 → INT8 → INT4. Each step cuts memory in half but loses some accuracy.
• Selective retention: Not all tokens are equally important. Keep the critical ones, discard or compress the rest.
• Low-rank decomposition: Use matrix factorization to compress cache representations.

Real deployments often stack these approaches. DeepSeek-V3 uses MLA (built-in compression) plus quantization, achieving cache sizes that would have been impossible just two years ago.

Context Window Extensions

Remember when 2K tokens was considered long context? Now we’re at 200K (Claude), 128K (Llama 3.1), and experiments pushing to 1 million tokens.

How did we get here without retraining models from scratch? Clever tricks:

• RoPE scaling: Interpolate or extend the rotation frequencies
• ALiBi, xPos, YaRN: Alternative position encoding schemes designed for long contexts
• Sparse attention patterns: Don’t attend to every token; use structured patterns to approximate full attention

Claude Sonnet 4’s 1 million token window means you can feed it entire software repositories and it won’t forget the function you defined 50,000 tokens ago. The “needle in a haystack” test (finding a specific sentence buried in massive text) shows 100% accuracy across the full context.

That’s not just bigger numbers. That’s a qualitative shift in what’s possible.

The Era Breakdown: 2022–2026

Let me give you the timeline, because understanding when things changed matters as much as what changed.

Era of Intelligent Chat (2022–2023): ChatGPT drops in November 2022 and the world loses its mind. Everyone races to build their own. Llama 1 and 2 emerge as open-source alternatives. The focus: make models that can actually talk to humans naturally. Context windows are 2K-4K tokens. Models are dense. Bigger is better.

Era of Multimodality (2023–2024): GPT-4 Vision arrives. Claude 3 adds images. Context windows explode: Claude hits 200K tokens. MoE architectures mature with Mixtral, showing you don’t need dense models. The focus shifts: it’s not just about size anymore, it’s about efficiency and capability breadth. Models start seeing, not just reading.

Era of Autonomy (2025–2026): This is where we are now. Models become agents. Claude Code lets AI delegate coding tasks via the command line. o3 solves competition-level math and coding problems. Llama 4 goes multimodal with an early fusion architecture. DeepSeek-V3 proves you can train frontier models for pocket change. The paradigm: models that think, reason, and act across extended timeframes.

What Hasn’t Worked (The Honest Section)

Let’s talk about the graveyard of ideas.

Pure parameter scaling has hit diminishing returns. Going from 100B to 1T parameters doesn’t give you 10x better performance anymore. The curve flattened. That’s why MoE and efficiency became critical — you need smarter architectures, not just bigger ones.

Data quantity over quality was another dead end. Turns out feeding your model the entire internet doesn’t help if half of it is garbage. Modern training focuses obsessively on data curation. DeepSeek-V3 was trained on 14.8 trillion tokens, but they were carefully filtered and of high quality. Quality > quantity, every time.

Benchmark gaming is a real problem. Models are overtrained on popular benchmarks, so scores inflate while real-world performance doesn’t improve commensurately. We’re seeing contamination issues where training data includes test set examples. Some benchmarks are becoming useless as evaluation tools.

Reward hacking in RLVR systems happens when models learn to exploit the reward signal instead of actually improving. They find loopholes. They take shortcuts. Claude 4 reportedly reduced these behaviors by 65% through better training procedures, but it’s an ongoing battle.

Hallucination reduction remains unsolved. Models still confidently make up facts. The best we’ve done is reduce frequency — Claude 2.1 cut false statements by about half compared to Claude 2.0 — but elimination? Not even close. Every new technique (RAG, chain-of-thought, RLVR) helps marginally, but we don’t have a silver bullet.

Looking Forward: The Next 12–24 Months

Where is this all heading?

Hybrid dense-sparse architectures will become standard. Pure MoE or pure dense is suboptimal. Future models will combine approaches, utilizing dense layers when full context is required and sparse layers when specialization is beneficial.

Test-time compute will scale aggressively. o3 is just the beginning. We’ll see models that can “think” for minutes or hours on hard problems, adaptively allocating compute where it matters most. The trade-off between speed and accuracy becomes user-controlled.

Edge deployment is coming. Llama 4 Scout at 17B active parameters is designed to run on smaller hardware. We’re heading toward powerful models on your phone, not just in the cloud. Privacy, latency, and cost all push in this direction.

Multimodal fusion will mature beyond simple add-ons. Llama 4’s “early fusion” approach — training text and vision together from scratch rather than bolting them together later — is the template. Native multimodality beats retrofit multimodality.

Cross-domain reward models will emerge. Right now, we have separate approaches for math (verifiable), coding (compiles or doesn’t), and open-ended tasks (human feedback). Future systems will unify these under coherent training frameworks.

Smaller, better models will eat market share from larger ones. We’re already seeing it: Llama 3.3’s 70B variant matches Llama 3.1’s 405B performance. DeepSeek proves efficiency beats brute force. The trend: capability per dollar and per watt matters more than raw capability.

The Philosophical Shift

The deeper change isn’t technical — it’s philosophical.

From 2020–2023, we believed in the scaling hypothesis: make it bigger, train it longer, and intelligence emerges. It worked! GPT-3 to GPT-4 felt like magic. But we hit limits — economic, physical, practical.

Now we’re in a different game. It’s not “bigger is better,” it’s “smarter is better.” It’s not “train a massive model once,” it’s “train efficient models that think deeply when needed.” It’s not “imitate all of human knowledge,” it’s “learn to reason through problems you’ve never seen.”

The transformer is still here, but it’s been refined into something its creators probably wouldn’t fully recognize. We’re using the same basic engine to do fundamentally different things.

Andrej Karpathy has this great line: “We’re not growing animals; we’re summoning ghosts.” These models don’t follow the learning curves we expected. They develop capabilities that surprise us. They fail in ways that confound our predictions. We’re doing engineering and science simultaneously, figuring out the rules as we go.

Conclusion: The Architecture as Living Document

If you told me in 2022 that we’d have models with 671 billion parameters that cost $6 million to train, that can process entire codebases in one shot, that solve PhD-level science problems, and that think for extended periods before answering… I would have asked how much compute that requires.

The answer, it turns out, is “less than you’d think — if you’re clever about it.”

The anatomy of modern LLMs is a case study in optimization. Every component — attention, experts, activations, positional encodings, training procedures — has been refined. The transformer didn’t change; we just learned to use it properly.

2024–2025 will be remembered as the year inference caught up to training. The year we stopped asking “how big can we make it” and started asking “how smart can we make it.” The year the architecture became a living document, constantly updated with new insights.

The models we’re building now aren’t just bigger versions of what came before. They’re different beasts entirely — smarter, more efficient, more capable, and honestly, more interesting.

And we’re just getting started.

Want to dive deeper? Check out the technical papers for DeepSeek-V3, o3, Llama 4, and Claude 4. The real innovations are in the details.