From Neurons to Networks: What We Left Behind
I keep this image on my phone. A biological neuron transforming into an artificial neural network.
It reminds me: we started by copying biology, then we diverged. That divergence enabled progress. But now we're facing problems where biological solutions might matter again.
The Neuron
A biological neuron is simple in concept. Dendrites receive signals. The cell body processes them. The axon sends signals forward.
But it operates under real constraints. Your brain runs on about 20 watts: less power than a dim lightbulb for 86 billion neurons. Neurons fire sparsely, staying silent most of the time. They adapt continuously, growing new connections and pruning unused ones. They respond to timing: when signals arrive matters, not just whether they arrive.
The neuron exists in an environment. It's embodied in a system that moves through the world, receives continuous feedback, operates with goals shaped by survival.
The Network
We built artificial neurons. Connected them in layers. Made signals flow forward. Each node sums inputs, applies a function, passes results to the next layer.
We kept the metaphor: connected units processing information, and simplified everything else. We removed spike timing and used continuous activations. We fixed architectures during training instead of growing them. We trained on static datasets instead of continuous experience.
This was the right choice. The simplification made these systems implementable. We could use backpropagation. We could scale to billions of parameters. We could train on datasets no human could process.
It worked. These systems now recognize images, translate languages, generate coherent text. Sometimes better than humans on specific benchmarks.
What Changed
But we optimized for different objectives than biology. Biology optimized for survival under energy constraints. We optimized for performance on static datasets with abundant compute.
The results reflect those objectives.
Modern networks achieve statistical sparsity through activation functions—many ReLU outputs are zero. But this isn't the same as biological sparsity, which is structural and adaptive. Mixture-of-Experts and sparse attention are steps in this direction, but they're recent and not foundational.
Transformers encode sequence through positional embeddings and attention. This captures some temporal structure, but it's coarse compared to spike timing. We represent "when" through position, not through the timing of discrete events.
Networks can adapt after training through fine-tuning, LoRA, and continual learning frameworks. But they don't self-adapt during deployment the way biological systems do. Adaptation requires human intervention and additional training.
Architecture isn't completely fixed—we have neural architecture search and dynamic routing in MoE models. But the search happens before deployment, not continuously during operation.
Where This Matters
I work on multilingual AI. The dominant approach is to scale. More data, bigger models, more parameters.
This works for English. There are billions of tokens available. The long tail of linguistic structures is well-represented. Scaling laws hold.
For most of the world's languages, scaling breaks down. When token frequencies collapse, you can't brute-force your way to performance. A model trained on 10 million Swahili tokens doesn't just perform worse than one trained on 10 billion English tokens: it fails to capture linguistic structures that only appear in the long tail.
You need different mechanisms. Better transfer learning. Stronger structural priors. More efficient use of limited data. Basically, approaches closer to how humans learn language: from thousands of hours of exposure, not billions of scraped tokens.
The same pattern appears in deployment. We train in controlled conditions on curated data. We deploy in dynamic environments where the distribution shifts continuously.
Models handle this poorly. Not because the architecture is wrong, but because they optimize for likelihood on training data. They have no calibrated uncertainty. No causal models. No sense of when they're operating outside their training regime.
They fail confidently, which is worse than failing cautiously.
What Biology Suggests
Biology didn't have infinite compute or curated datasets. It had real constraints that forced efficient solutions.
Energy efficiency matters now. Training frontier models costs millions in compute. Inference costs scale with deployment. These aren't academic concerns—they're economic and environmental constraints.
Continuous adaptation matters. Systems need to adjust to distribution shifts without catastrophic forgetting or full retraining. Biology does this constantly through synaptic plasticity and structural changes.
Uncertainty quantification matters. A system should know when it doesn't know. Biology handles this through multiple mechanisms—hesitation, exploratory behavior, reliance on known patterns when uncertain.
Embodiment might matter for certain tasks. The smartest language models make basic mistakes about physical causality because they never interacted with a physical world.
These aren't arguments to copy biology directly. That's still intractable. But they suggest directions worth exploring when scaling hits limits.
The Limits We're Hitting
Scaling still works, but with diminishing returns. Each 10x increase in compute buys less improvement than the last. The economic limits are real even if theoretical limits aren't.
Data availability is a hard constraint for most languages and domains. You can't scale when the data doesn't exist.
Deployment robustness is a practical necessity. Models that work in labs but break in production aren't useful, regardless of their benchmark scores.
These problems won't be solved by making models bigger. They require different approaches—better sample efficiency, continuous adaptation, calibrated uncertainty, structural priors that transfer across contexts.
This is where biological principles might help. Not as a blueprint, but as existence proofs that these problems have solutions.
What I'm Watching
The field is moving in some of these directions. Mixture-of-Experts for efficiency and specialization. Retrieval-augmented generation for grounding. Ensembles and multi-sample decoding for uncertainty. Continual learning frameworks for adaptation.
These are still early. Most deployed systems use standard architectures trained on static data with no post-deployment learning.
But the direction suggests people are recognizing the limits. You can't scale your way out of every problem. Sometimes you need smarter mechanisms, not more parameters.
For multilingual AI specifically, this means taking low-resource languages seriously as research problems, not just data collection problems. The constraints force you to think about learning efficiency, transfer, and structural priors.
The same constraints biology faced.
The Arrow
The arrow in the image goes from neuron to network. We took biological inspiration and built something different.
That was necessary. The simplification enabled progress we couldn't have made otherwise.
But the conversation isn't one-way. We're learning where our systems fail. Those failures point back to principles biology uses that we discarded.
Not because biology is better. Because biology solved problems under constraints we're now facing again.
Energy efficiency. Learning from limited data. Continuous adaptation. Operating under uncertainty.
We don't need to copy biology. We need to learn from the problems it solved.
The arrow goes both ways.
