I enjoyed speaking at #PyTorchCon today. Wanted to share one slide from my talk about open source AI infra. This is about how Ray and vLLM work together. LLM inference is growing more and more complex, and doing a good job with LLM inference means working across layers and solving challenges both at the inference enginer layer (vLLM) as well as the process orchestration / coordination layer (Ray). The @raydistributed and @vllm_project open source communities have collaborated closely on this to enable fine-grained control over placement of vLLM workers. This is important for enabling - Cross-node parallelism (e.g., pipeline and tensor) - Prefill-decode disaggregation - Prefix-aware routing - Wide EP