As of: 2026-10-12 07:22 CEST.
Researched and written by AI agents (Claude Opus 5.5 in Claude Code). Human editor: Adam Bicz.
Canonical page: https://postcutoff.com/v/caleb-writes-code-kimi-k3-explained/

# Kimi K3 explained in 13min..

Caleb Writes Code, 21 July 2026, YouTube. 230,430 views as of 12 October 2026.
Kind: Review.
Watch: https://www.youtube.com/watch?v=rD20wJkPUB4

## Why it is here

Architecture explainer for Kimi K3 (description: 2.8T-parameter open model, Kimi Delta Attention, Stable LatentMoE); ~230k views.

## Description (written by Gemini from the video)

**Summary**  
This video is a whiteboard-style technical architecture breakdown of Moonshot AI's Kimi K3 model, presented by Caleb from the channel "Caleb Writes Code." The host analyzes how Kimi K3 achieves high inference and training efficiency through an ultra-sparse Mixture of Experts (MoE), Stable Latent MoE, Kimi Delta Attention (KDA), and Block Attention Residuals (AttnRes).

**What is shown**  
- [00:00] Timeline graphic contrasting the January 2025 "DeepSeek moment" with the release of Moonshot AI's Kimi K3.
- [01:08] Animated diagrams introducing sparse Mixture of Experts (MoE) architectures and comparing active parameter percentages across frontier models.
- [02:17] Visual breakdown comparing Kimi K2 (1T parameters, 384 experts, 8 active) and Kimi K3 (2.8T parameters, 896 experts, 16 active).
- [03:04] Diagram of cluster deployment across NVIDIA DGX B300, 64-GPU SuperNodes, and NVL72 racks, illustrating inter-GPU communication latency and token movement.
- [04:33] Flowchart of the Kimi K3 architecture diagram, detailing the Stable Latent MoE router, down-projection, shared experts, and up-projection.
- [06:21] Charts and benchmark plots comparing Decoding Acceleration vs. Performance and Time Per Output Token (TPOT) across decoding lengths up to 1M tokens.
- [08:14] Mathematical equations comparing memory decay rules across standard recurrent decay gates, Gated DeltaNet (GDN), and Kimi Delta Attention (KDA).
- [09:28] Table 1 from the paper detailing ablation studies on hybrid ratios of KDA to MLA attention layers (3:1 selected).
- [10:00] Excerpts of the paper's hardware-efficient chunkwise parallel training formulation and UT transform equations.
- [10:48] Comparative architectural schematics illustrating Standard Residuals, Full Attention Residuals, and Block Attention Residuals.
- [12:21] Table 3 showing benchmark comparisons of the baseline architecture versus Attention Residuals (AttnRes).
- [12:32] Discussion on potential US regulatory bans versus open-cloud deployment economics and API pricing.

**Claims & numbers**  
- **Model Size & Sparsity**: The presenter states Kimi K3 has 2.8 trillion total parameters with 896 experts, activating only 16 experts per token (~1.8% activation ratio), compared to Kimi K2's 1 trillion parameters, 384 experts, and 2% activation ratio (8 active).
- **Comparison to other MoE models**: The presenter claims MiniMax M3 activates ~3.1% of experts, Thinking Machine's Inkling activates ~3.1%, and NVIDIA Nemotron 3 Ultra activates ~4.3%.
- **Throughput & Speed**: The presenter claims KDA enables Kimi K3 to maintain a 6.3x decoding throughput improvement at 1M sequence length compared to standard Multi-Head Latent Attention (MLA).
- **Ablation Performance**: The presenter highlights Table 1 showing a 3:1 hybrid ratio of KDA to Gated MLA achieved the lowest validation perplexity (5.65) and training perplexity (9.23).
- **Pricing & Economics**: The presenter claims Kimi K3's reported API pricing is $3 per million input tokens and $15 per million output tokens.

**Notable quotes**  
- [01:04] *"And at a high level, Kimi K3 is all about efficiency and intelligence."*
- [02:42] *"...which goes to show how they scaled their model in size without sacrificing efficiency."*
- [12:38] *"There's already been talks of our government potentially banning Kimi K3, but I think that would be a huge mistake..."*

**Assessment**  
This is an independent technical review and explainer summarizing the technical design choices behind Moonshot AI's Kimi K3 release paper. The video relies on whiteboard animations, paper tables, and architectural diagrams to explain the system's mathematics and engineering tradeoffs rather than running live benchmark tests on camera.

_Described by gemini-3.8-flash on 2026-10-12 from the video's audio and frames._

## Related

- 2026-07-16: [Moonshot AI releases Kimi K3, a 2.8T-parameter open-weights multimodal model](https://postcutoff.com/e/2026-07-16-moonshot-kimi-k3/)