Elie Bakouch: Reflection Beam architecture details (sliding window 512, 5:1, gated attention, no shared experts)
elie @eliebakouchX
Why it matters
Most-viewed technical read (~30k views) of Beam’s released weights.
Summary
Posted Fri Oct 16 15:30:23 +0000 2026 by elie (@eliebakouch); 30,036 views, 303 likes, 21 reposts when read via api.fxtwitter.com on 2026-10-17. Sliding-window attention with window 512 and a 5:1 local:global pattern (same as Thinking Machines’ Inkling); global-attention scaling like Llama 4; residual depth scaling; gated attention, QK-norm, sandwich norm, logit softcapping; no shared experts. Links huggingface.co/reflection/Beam-1.0-501B-A23B.
Archived text
reflection beam oss architecture details
IT’S SLIDING WINDOW AGAIN, same window size and pattern as inkling (window 512 and 5:1, meta is 2k and 3:1). they use a similar global attention scaling as llama4 (https://arxiv.org/abs/2501.19399) which still makes it position dependent even if they don’t use RoPE
they use a residual depth scaling, quite similar to muse open but “fixed”, tho it’s not really fixed because you have a learned norm nearby. you can actually plot both the effective residual contributions and you see that it’s not super different from muse
they use gated attention (with a per head bias for the full attention layer), QK Norm, Sandwich norm, similar softcapping before the LM head (like gemma and also muse), no shared experts
the shape is pretty similar to other american oss open weight models (but tbf, mimo v2 flash and stepfun 3.5 and 5 also have this shape, it’s not only us labs 👀)
model weight: https://huggingface.co/reflection/Beam-1.0-501B-A23B
Quoting @reflection_ai: Beam is now available across platforms, with open weights under the Apache 2.0 license - in collaboration with an ecosystem of 40+ partners.
We’re releasing:
- Weights, tokenizer, and reference implementation
- BF16, FP8 and NVFP4 weights and a DFlash2 drafter
- The open source harness we built and trained Beam on: Mirror CLI
Weights and model card on Hugging Face: https://huggingface.co/reflection
