arxiv.org/abs/2309.06180
Preview meta tags from the arxiv.org website.
Linked Hostnames
27- 29 links toarxiv.org
- 13 links toinfo.arxiv.org
- 2 links tosubscribe.sorryapp.com
- 1 link toapi.semanticscholar.org
- 1 link tocore.ac.uk
- 1 link tocreativecommons.org
- 1 link todagshub.com
- 1 link todoi.org
Thumbnail
Search Engine Appearance
Efficient Memory Management for Large Language Model Serving with PagedAttention
High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems. On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within and across requests to further reduce memory usage. Our evaluations show that vLLM improves the throughput of popular LLMs by 2-4$\times$ with the same level of latency compared to the state-of-the-art systems, such as FasterTransformer and Orca. The improvement is more pronounced with longer sequences, larger models, and more complex decoding algorithms. vLLM's source code is publicly available at https://github.com/vllm-project/vllm
Bing
Efficient Memory Management for Large Language Model Serving with PagedAttention
High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems. On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within and across requests to further reduce memory usage. Our evaluations show that vLLM improves the throughput of popular LLMs by 2-4$\times$ with the same level of latency compared to the state-of-the-art systems, such as FasterTransformer and Orca. The improvement is more pronounced with longer sequences, larger models, and more complex decoding algorithms. vLLM's source code is publicly available at https://github.com/vllm-project/vllm
DuckDuckGo
Efficient Memory Management for Large Language Model Serving with PagedAttention
High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems. On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within and across requests to further reduce memory usage. Our evaluations show that vLLM improves the throughput of popular LLMs by 2-4$\times$ with the same level of latency compared to the state-of-the-art systems, such as FasterTransformer and Orca. The improvement is more pronounced with longer sequences, larger models, and more complex decoding algorithms. vLLM's source code is publicly available at https://github.com/vllm-project/vllm
General Meta Tags
23- title[2309.06180] Efficient Memory Management for Large Language Model Serving with PagedAttention
- titleopen search
- titleopen navigation menu
- titlecontact arXiv
- titlesubscribe to arXiv mailings
Open Graph Meta Tags
10- og:typewebsite
- og:site_namearXiv.org
- og:titleEfficient Memory Management for Large Language Model Serving with PagedAttention
- og:urlhttps://arxiv.org/abs/2309.06180v1
- og:image/static/browse/0.3.4/images/arxiv-logo-fb.png
Twitter Meta Tags
6- twitter:site@arxiv
- twitter:cardsummary
- twitter:titleEfficient Memory Management for Large Language Model Serving with...
- twitter:descriptionHigh throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for...
- twitter:imagehttps://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png
Link Tags
12- apple-touch-icon/static/browse/0.3.4/images/icons/apple-touch-icon.png
- canonical/abs/2309.06180
- icon/static/browse/0.3.4/images/icons/favicon-32x32.png
- icon/static/browse/0.3.4/images/icons/favicon-16x16.png
- manifest/static/browse/0.3.4/images/icons/site.webmanifest
Links
68- http://creativecommons.org/licenses/by/4.0
- http://gotit.pub/faq
- http://www.bibsonomy.org/BibtexHandler?requTask=upload&url=https://arxiv.org/abs/2309.06180&description=Efficient Memory Management for Large Language Model Serving with PagedAttention
- https://api.semanticscholar.org/arXiv:2309.06180
- https://arxiv.org