Miles, Slime, and Ray: Orchestrating the Post-Training Loop

Miles, Slime, and Ray: Orchestrating the Post-Training Loop

In large-language-model post-training systems, Miles and Slime are often mentioned alongside Ray, SGLang, and Megatron-LM. The right way to understand them is to separate responsibilities: Miles/Slime define the post-training loop, SGLang performs rollout inference, Megatron-LM or FSDP trains the policy, and Ray orchestrates distributed resources, processes, and lifecycles.

Slime is an extensible RL post-training framework built around rollout, reward or verification, data buffering, training, and weight synchronization. Miles was forked from Slime and has co-evolved with it, extending the same design toward production-scale workloads with stronger support for large MoE models, fully asynchronous RL, resilience, low-precision training, LoRA, and P2P/RDMA weight transfer.

The loop is:

Prompts → SGLang rollout → Reward/Verifier → Data Buffer
        → Megatron/FSDP training → Weight Sync → New rollout

Reward/Verifier is the step that defines what the model should learn. Weight Sync is the step that reconnects the updated policy to the online rollout fleet and therefore completes the operational loop.

Ray should be understood as the control plane, not as a universal tensor transport. It registers resources across machines, creates actors, manages placement groups, submits jobs, performs health checks, and invokes lifecycle RPCs such as pause, resume, and load weights. Training collectives and large data movement usually use specialized paths: NCCL or PyTorch Distributed for training communication, NCCL broadcast or RDMA/Mooncake for weight transfer, and shared storage or dedicated queues for large trajectories.

Engineering Weight Sync is a pipeline:

1. Pause generation through rollout actors.

2. Flush caches associated with the old policy.

3. Gather Tensor Parallel and, for MoE models, Expert Parallel shards.

4. Convert Megatron parameter names and layouts into the format expected by SGLang.

5. Pack tensors into fixed-size buckets.

6. Transfer buckets using NCCL broadcast, P2P/RDMA, or versioned shared-disk reload.

7. Wait for all target ranks to load the new version, then resume generation.

P2P transfer builds a training-to-rollout transfer plan, discovers registered remote memory, and can reuse SGLang’s weight-loader logic through a CPU model replica. This avoids redundant copies and ensures that the re-sharded tensors match the inference layout.

The most useful mental model is:

Miles/Slime = post-training loop
Ray = distributed control and orchestration
Megatron/FSDP = training
SGLang = rollout inference
NCCL/RDMA/storage = high-volume data movement

The central insight is that post-training is not just an optimizer step. It is a closed loop in which the model acts, receives verifiable feedback, updates its policy, and returns to the environment. Reward defines the direction, training performs the update, Weight Sync closes the loop, and Ray makes the distributed system operable.

References: Slime, Miles, Miles Quick Start, Launch Script, and P2P Weight Transfer.