The Local LLM Index / Quantization & Formats / #225
psmarter/mini-infer
by psmarter · Quantization & Formats · updated 6mo ago
LLM inference engine from scratch — paged KV cache, continuous batching, chunked prefill, prefix caching, speculative decoding, CUDA graph, tensor parallelism, OpenAI-compatible serving
27
momentum
235
stars
12
forks
#225
rank
continuous-batchingcudainferenceinference-enginekv-cachelanguage-modelllmmachine-learningmoepagedattentionpytorchquantization
View on GitHub →