The Local LLM Index / Quantization & Formats / #68
swellweb/reame
by swellweb · Quantization & Formats · updated 1d ago
CPU-first LLM inference server on llama.cpp. Runs useful models on free-tier ARM boxes; rewriting the input made it ~6x faster and more accurate than tuning the engine. MIT, benchmarks and failures included.
65
momentum
101
stars
10
forks
#68
rank
armcpuedge-aiggufinferencekv-cachellama-cppllmlocal-llmopenai-apiprompt-compressionself-hosted
View on GitHub →