[LG] Vidur: A Large-Scale Simulation Framework For LLM Inference arxiv.org/abs/2405.05465 - This paper presents Vidur, a high fidelity and easily extensible simulator for large language model (LLM) inference, along with a benchmark and search suite. - Vidur models the performance of LLM operators using a combination of experimental profiling and predictive modeling, and evaluates end-to-end inference performance for different workloads. - It estimates metrics like latency, throughput, model FLOPs utilization, memory utilization, etc. with high accuracy. - Vidur addresses challenges unique to simulating LLM inference like finer time granularity, varying iteration times, and cascading errors. - It uses insights like architectural uniformity of LLMs, operator triaging, and automated profiling for parallelism strategies to achieve fidelity. - Vidur-Search uses Vidur to automatically identify optimal cost-effective deployment configurations meeting performance constraints.