4 articles tagged with #neural networks.
Mixtral uses 46.7B parameters but only activates 13B per token. This architectural trick called Mixture of Experts powers Gemini 1.5, DeepSeek V3, and more. Learn how MoE works, its hidden costs, and when to use it.
GPT-4's 128K context window? It only uses about 10% effectively. Google's TITANS architecture introduces test-time memory learning that outperforms GPT-4 on long-context tasks with 70x fewer parameters.
Nested Learning: The Illusion of Deep Learning Architectures - A comprehensive guide to the arXiv paper revealing how neural networks learn at multiple timescales through hierarchical optimization.
Neural networks catastrophically forget previous knowledge when learning new tasks—not due to capacity limits but fundamental constraints in distributed learning systems.
