AWS Machine Learning Blog· Sunita Nadampalli·· 3 小时前AI 评分31
vLLM 推出 push-based NIXL connector，将 KV 传输移出 TTFT 关键路径
Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
AI 导读
vLLM 引入 push-based NIXL connector，把 KV cache 传输的协调与建连步骤从 decode 端的关键路径上移走，由 prefill 端配合后台线程完成 NIXL WRITE，从而在 disaggregated prefill/decode 部署中缩短 TTFT。
来源：AWS Machine Learning Blog · aws.amazon.com