Million-Token Training Run Reaches 64 B200 GPUs

The research-stage result points toward long-context models for genomes and documents, with no release date announced.

Updated on Nov. 7, 2026 in Life Sciences

Isometric editorial illustration of stacked matte server blocks in oxblood, teal, and mustard connected by a single cable, suggesting large-scale AI computing hardware.
Lila Sciences and Anyscale trained a GLM-5.3-Flash model on million-token sequences using 64 NVIDIA B200 GPUs, a research-stage result with no release date announced. AI Illustration. Upload story photo >

Lila Sciences and Anyscale trained GLM-5.3-Flash on sequences of up to 1 million tokens using 64 NVIDIA B200 GPUs and the open-source SkyRL training stack. The training run is reported as a research-stage result, not a shipped product.

Why it matters

Training on million-token sequences is a step toward models that can read entire genomes or very long documents in one pass, rather than in fragments. The run also reflects a 64-fold increase over the team's initial training capacity, suggesting that long-context training is becoming accessible on modest GPU clusters.

The reported benchmarks are internal training throughput and memory figures from a single run on 64 NVIDIA B200 GPUs, with throughput of 13,700 tokens per second at 1 million tokens and 24,900 tokens per second at 256,000 tokens. Offloading reduced peak memory at 256,000 tokens from 142 GiB to 98 GiB, though no external comparison baseline is given.

The players

Lila Sciences

A research organization that conducted the GLM-5.3-Flash training and the human-genome fine-tuning test.

Anyscale

A company that worked with Lila Sciences on the long-sequence training run, using its distributed computing infrastructure.

The details

The model combines 34 layers of Kimi Delta Attention, a linear-cost attention method, with 11 layers of DeepSeek Sparse Attention, which attends only to a selected subset of tokens. To run the sparse layers efficiently, the team changed tensor dimensions so that specialized sparse-attention kernels could be used, halving memory use for that component. Training relied on activation recomputation, CPU offloading and context parallelism to manage memory and processing demands at long sequence lengths.

Timeline

  1. Article reporting the training results was published on November 7, 2026.

The Tech Race

Long-context training has been a competitive focus for model developers seeking to read whole genomes and long documents. This run places a 1-million-token training result on a 64-GPU cluster, a scale that is modest relative to the largest frontier training runs.

The work is a training research result and no release date, pricing or availability has been announced. Researchers working with long genomic or document sequences may eventually see these long-context methods in open-source training stacks, but no timeline for that is given.

The takeaway

Long-context training is becoming feasible on clusters of tens of GPUs, not only the largest facilities. Watch for an independent reproduction of the 4.8% held-out loss result and any downstream evaluation of the human-genome model.

Further reading

For the biology applications behind this work, see our Life Sciences coverage.

Source note: This article includes information reported by TokenPost.