Running GLM-5.2 5x faster at 500tps with limitation

- GLM-5.2 is a Chinese open-source large language model (from Zhipu AI, also called "ChatGLM") that competes with models like LLaMA and GPT. This blog shows a technique to run it at ~500 tokens per second on 8× NVIDIA B200 GPUs (the high-end "Blackwell" datacenter GPU) — roughly 5× faster than typical inference speeds for this model size. - "Tilert" appears to be the author's custom inference optimization (likely combining tensor parallelism, batched processing, and kernel-level tuning) to maximize GPU utilization. - The "limitation" in the title refers to the practical bottlenecks: memory bandwidth, batch size constraints, and the model's architecture not being perfectly suited for extreme batching, meaning even faster speeds would require fundamental model redesign. - Why this matters: High throughput inference reduces the cost and latency of running frontier AI models in production. 500 tps at this model scale is notable because it approaches real-time interactive use for large models, which are typically much slower without specialized hardware or software tricks.