Quick Note

Core Methodology?

1. Hybrid Attention (Compressed Sparse Attention + Heavily Compressed Attention)
2. Manifold-Constrained Hyper-Connections (mHC)
3. Muon Optimizer

• Pretrain

  • 32T

• Post-Train
  • Domain-specific experts then on-policy distillation (wherein the unified model acts as the student learning to optimize the reverse KL loss with teacher models)

Motivation

Strenghts

Core Architecture

Related Works

• <Main baseline>
• Other methods to solve the same problems

Relation to My Research

Main research Gap