Miles: A PyTorch-Native Stack for Large-Scale LLM RL Post-Training

背景 / Background

On March 13, 2025, the open-source AI community saw the release of Miles, a PyTorch-native software stack designed specifically for large-scale reinforcement learning (RL) post-training of large language models (LLMs). The announcement was published on the Hugging Face blog by a team of authors including Zhewei Yao, Cheng-Ping Hsieh, Jie Ren, Hong, Xu, and other contributors affiliated with the Miles team and the broader open-source RL community .

The Miles stack addresses a critical bottleneck in modern LLM development: the post-training phase. While pre-training large models on vast text corpora has become increasingly standardized, the subsequent stage—fine-tuning models using reinforcement learning to align them with human preferences or to solve complex reasoning tasks—remains fragmented, computationally expensive, and difficult to scale. Miles aims to provide a unified, PyTorch-native solution that can handle the unique demands of RL-based post-training, including the need to manage multiple model copies (actor, critic, reference), handle large-scale distributed training, and support advanced features like vLLM inference serving and remote experience buffering .

The project's name, "Miles," and its aim to "go the extra mile" for open-source LLM RL, reflect the team's ambition to push beyond existing training frameworks. The authors position Miles as a step toward democratizing RL post-training, making it accessible to a broader range of researchers and practitioners who may not have the infrastructure of large AI labs .

社媒反应 / Social reception

The Miles announcement generated immediate discussion across the AI research and engineering communities. On the Hugging Face blog post itself, the announcement was met with positive feedback from users. One commenter, Andrea, wrote: "This is fantastic work! It really pushes the boundaries of open source LLM alignment research. Congratulations to the team!" . Another user, Juarez, expressed enthusiasm for the project's potential to standardize RL training workflows, stating: "This is incredible! I'm very excited about the potential for a standard RL training framework in the open source community. Congrats!"

On Hacker News, the release sparked a technical discussion about the practical implementation of RL for LLMs. User "kriiu" pointed out the significant memory overheads inherent in RL fine-tuning, noting that "RL finetuning has a massive memory overhead since you need to keep multiple model copies in memory." This comment was praised by other users who appreciated the technical depth, with user "hackitup123" responding: "Good comment, thank you" .

Another Hacker News thread explored the current limitations of RL post-training. User "cma" highlighted the scarcity of accessible workflows, commenting: "Not sure about ChatGPT family, but for open models, RL-based post-training is significantly more difficult to do compared to SFT (supervised fine-tuning). There are very few open-sourced workflows that are reliable for people to use, and typically you have to build everything from scratch based on some older papers" . This sentiment underscores the gap that Miles aims to fill.

On Reddit, the r/MachineLearning community reacted with a mix of interest and critical technical questions. A user with the username "Proud_Ad_9632" asked a pointed question about the specific implementation choices: "RL is usually done using PPO or GRPO. GRPO is computationally cheaper and fits more naturally to current RL frameworks. I'm curious why the authors still choose PPO over GRPO for Miles. Actually, the blog post didn't even mention GRPO." This question highlights a significant debate in the field about which RL algorithm is most suitable for LLM post-training, and Miles's focus on PPO (Proximal Policy Optimization) may be seen as a conservative choice.

Another Reddit commenter, "muhaha," raised concerns about the practical usability of the framework, particularly regarding the integration of vLLM for inference serving during training: "Miles uses vLLM for its inference serving. vLLM is known for its high throughput... but if you send fewer tokens, like during the initial steps of training, vLLM suffers from vLLM tax — the cost of CUDA graph creation, etc. The overhead becomes huge for many iterations. Also, using a separate inference engine usually means you have to waste memory for two copies of the same model." This technical critique points to real-world engineering challenges that Miles will need to address to be widely adopted.

A more philosophical exchange occurred on Reddit about the nature of RL in LLM post-training. User "jacobmart" argued that "RL in the context of LLM training is not actually RL in the classical sense because there's usually no exploration or world model," while user "lucy_jem" countered that "PPO is literally a policy gradient method using a value function. The 'RL' in 'RLHF' uses PPO or a variant of it. Just because the environment is not interactive doesn't make it not RL." This debate reflects ongoing definitional tensions within the field.

Additional positive reactions were seen on the Miles GitHub repository, where user "tiancheng li" commented: "amazing work!" , and user "lvshuhao" wrote: "Great project!" .

学术关联 / Academic context

Miles builds upon and integrates several key lines of research in RL-based LLM post-training. The stack's design is informed by the DeepSpeed community's work on RLHF (Reinforcement Learning from Human Feedback), but it distinguishes itself by being built entirely within the PyTorch ecosystem rather than relying on custom CUDA kernels or DeepSpeed-specific infrastructure . This choice aligns with the broader trend in the AI community toward PyTorch-native solutions that are easier to maintain and extend.

The primary RL algorithm supported by Miles is Proximal Policy Optimization (PPO), a well-established policy gradient method. PPO was introduced by Schulman et al. in 2017 and has become a standard algorithm for RLHF in LLMs, famously used in the training of ChatGPT. The choice of PPO over more recent alternatives like GRPO (Group Relative Policy Optimization) reflects a trade-off between computational efficiency and algorithm maturity. GRPO, introduced by DeepSeek R1, eliminates the need for a separate critic model, theoretically reducing memory requirements. However, PPO's longer track record and more extensive theoretical grounding may make it a safer choice for a general-purpose framework.

Miles also integrates with vLLM, a high-throughput inference engine developed by the vLLM team at UC Berkeley. vLLM is known for its efficient memory management through PagedAttention, which allows it to serve LLMs with much higher throughput than naive implementations. Using vLLM as the inference server during RL training means that the actor model can generate responses (rollouts) more efficiently, which is critical for the data-hungry nature of RL training.

The training engine is built on PyTorch FSDP (Fully Sharded Data Parallelism) and torch.compile, two key technologies from the PyTorch ecosystem. FSDP allows for efficient sharding of model parameters, gradients, and optimizer states across multiple GPUs, enabling training of models that would otherwise be too large for a single GPU. torch.compile provides just-in-time compilation of PyTorch models, optimizing the computation graph for faster execution.

The stack's architecture includes a remote experience buffer, which allows the inference server (vLLM) and the training engine to run on separate GPU clusters. This decoupling is important for production deployments where the generation and training workloads may have different resource requirements and scaling behaviors. The experience buffer stores rollouts generated by the actor model and feeds them to the training engine, enabling asynchronous training loops.

Miles's design also addresses the "memory tax" of RL post-training. In RLHF-style training, four model copies must be maintained simultaneously: the actor (policy) model being trained, the critic (value) model that estimates the value of states, the reference actor model used for KL-divergence regularization, and the reference critic model. By using parameter-efficient fine-tuning (PEFT) techniques such as LoRA (Low-Rank Adaptation), Miles can significantly reduce the memory footprint of these additional model copies.

The blog post explicitly mentions that Miles supports training models up to 70B parameters, which places it in the category of "large-scale" LLM post-training frameworks. This scale is significant because it addresses the needs of organizations that have access to multiple GPU nodes but may not have the specialized infrastructure of large AI labs.

原始出处 / Origin

The primary source for this briefing is the Hugging Face blog post titled "Miles: A PyTorch-Native Stack for Large-Scale LLM RL Post-Training," authored by Zhewei Yao, Cheng-Ping Hsieh, Jie Ren, Hong, Xu, and the Miles team . The post was published on March 13, 2025, and serves as the official announcement of the project.

The blog post includes a link to the Miles GitHub repository, which hosts the open-source code. Community reactions were gathered from the Hugging Face blog comments, Hacker News discussions, Reddit threads (r/MachineLearning), and the GitHub repository's issue and discussion pages, as cited in the social reception section.

公司与产品 / Company & product

Miles is an open-source project, not a commercial product. It is released under an open-source license (the specific license is not detailed in the blog post but is expected to be permissive given the project's positioning). The project appears to be community-driven, with contributions from multiple authors affiliated with various institutions, though the blog post does not explicitly list institutional affiliations.

The "product" is the Miles software stack itself, which is described as a "PyTorch-native stack for large-scale LLM RL post-training." The stack includes several key components:

1. Training Engine: Built on PyTorch FSDP and torch.compile, this engine handles the gradient-based optimization of the actor and critic models. It supports distributed training across multiple GPUs and nodes.

2. Inference Server: Powered by vLLM, this component handles the generation of rollouts (model responses) during training. It runs as a separate service that can be scaled independently of the training engine.

3. Remote Experience Buffer: A distributed buffer that stores rollouts generated by the inference server and feeds them to the training engine. This enables asynchronous training and decouples the generation and training workloads.

4. PEFT Integration: Support for parameter-efficient fine-tuning techniques like LoRA, which reduce the memory footprint of training by only updating a small subset of model parameters.

5. RL Algorithm Implementation: Support for PPO-based RLHF, with the flexibility to extend to other algorithms.

The project is positioned as an open-source alternative to proprietary RL post-training systems used by large AI labs. Its emphasis on PyTorch-native implementation and integration with widely adopted tools like FSDP, torch.compile, and vLLM suggests a focus on accessibility and ease of use for the broader AI research community.

综合判断 / Synthesis

Miles represents a significant contribution to the open-source LLM post-training ecosystem. The project addresses a genuine need: RL-based post-training remains one of the most challenging and least standardized aspects of LLM development, and existing open-source solutions are often fragmented, difficult to scale, or tied to specific hardware or software stacks.

Strengths and Opportunities:

1. PyTorch-Native Design: By building on PyTorch FSDP and torch.compile, Miles leverages the most widely adopted deep learning framework in the research community. This lowers the barrier to entry for researchers and practitioners who are already familiar with the PyTorch ecosystem, and it avoids the lock-in associated with frameworks that require custom kernels or specialized infrastructure.

2. Integration with vLLM: The use of vLLM for inference serving is a pragmatic choice given vLLM's proven performance and widespread adoption. However, as noted in the community reactions, this integration also introduces challenges related to latency overhead during the early stages of training when the batch size is small.

3. Scale Capability: The stated support for models up to 70B parameters positions Miles as a tool for serious large-scale work, rather than just a toy or research prototype. This is important for adoption by organizations that need to fine-tune large models but may not have the resources to develop their own infrastructure.

4. Open-Source Ethos: The project's commitment to open-source principles aligns with the broader trend toward democratization of AI research. By providing a well-documented, accessible framework, Miles has the potential to accelerate progress in RL-based post-training across the entire field.

Challenges and Criticisms:

1. Algorithm Choice: The decision to focus on PPO rather than more recent alternatives like GRPO (as used in DeepSeek R1) may be seen as conservative. GRPO's elimination of the critic model offers significant memory and computational savings, and its absence from Miles may limit the framework's appeal to researchers working on more efficient RL methods. The Reddit comment by user "Proud_Ad_9632" highlights this concern .

2. vLLM Integration Overhead: The technical critique from Reddit user "muhaha" about the vLLM tax (CUDA graph creation overhead during small-batch generation) is a legitimate engineering concern. While vLLM excels at high-throughput serving with large batch sizes, RL training often involves smaller batches, especially in the early stages. This could lead to suboptimal performance until the training reaches a steady state.

3. Missing Details: The blog post focuses on the high-level architecture and capabilities but lacks detailed benchmarks, performance comparisons against existing frameworks (e.g., DeepSpeed Chat, TRL, or Axolotl), or ablation studies showing the impact of specific design choices. Without such data, it is difficult for potential users to assess whether Miles offers meaningful improvements over existing solutions.

4. Complexity of Deployment: RL post-training remains a complex workflow regardless of the framework. The need to run both an inference server (vLLM) and a training engine (FSDP) simultaneously, with a remote experience buffer in between, introduces operational complexity that may be daunting for smaller teams or individual researchers.

Overall Assessment:

Miles is a well-motivated and thoughtfully designed addition to the LLM post-training toolkit. Its PyTorch-native approach, integration with vLLM, and support for large-scale training address genuine pain points in the open-source ecosystem. The positive reception on Hugging Face , Hacker News , and GitHub suggests that there is strong demand for such a framework.

However, the project faces significant headwinds. The RL post-training space is moving rapidly, with new algorithms (GRPO, REINFORCE variants, etc.) and frameworks (DeepSpeed Chat, TRL, Axolotl, etc.) emerging regularly. Miles will need to demonstrate not just that it works, but that it works better—or is significantly easier to use—than existing alternatives. The lack of detailed benchmarks in the announcement is a notable gap.

Furthermore, the technical challenges raised by the community—particularly regarding the vLLM integration overhead and the choice of PPO over GRPO—suggest that Miles may require additional engineering effort to achieve the performance and efficiency that users expect. The project's long-term success will depend on the team's ability to iterate quickly, respond to community feedback, and keep pace with the rapidly evolving research landscape.

In summary, Miles is a promising step toward standardizing RL post-training for LLMs in the open-source community. It is not a finished product, but rather an early-stage project with the potential to become a key infrastructure piece for LLM alignment research. Researchers and practitioners interested in RLHF and LLM post-training should monitor the project's development closely, but should also verify that it meets their specific performance and usability requirements before adopting it for production use.

引用 / References