“Reflection Beam checkpoint is finally here and hence details of the architecture…”

Reflection Beam checkpoint is finally here and hence details of the architecture

Several important observations

• 48 q heads, 8 kv heads, SWA-512, swa theta 10k (conceptually same as laguna :D )
• pretty high aspect ratio 98.5 (not enough depth, harms reasoning)
• 1:5 SWA:GA ratio! so swa goes first here and quite aggressive
• 3 dense layers for stability
• depth based residual add
• some modification of yarn for long context, not sure about it yet

  Scalable-softmax query scale, (1 + position / 2048) ** 0.1, for layers that run without RoPE.