@LiorOnAI ยท
You can now run 70B LLMs on a 4GB GPU. AirLLM just made massive models usable on low-memory hardware. ๐ช๐ต๐ฎ๐ ๐ท๐๐๐ ๐ต๐ฎ๐ฝ๐ฝ๐ฒ๐ป๐ฒ๐ฑ AirLLM released memory-optimized inference for large language models. It runs 70B models on 4GB VRAM. It can even run 405B Llama 3.1 on 8GB VRAM. ๐๐ผ๐ ๐ถ๐ ๐๐ผ๐ฟ๐ธ๐ AirLLM loads models one layer at a time. Instead of loading everything: โ Load a layer โ Run computation โ Free memory โ Load the next layer This keeps GPU memory usage extremely low. ๐๐ฒ๐ ๐ฑ๐ฒ๐๐ฎ๐ถ๐น๐ โข No quantization required by default โข Optional 4-bit or 8-bit weight compression โข Same API as Hugging Face Transformers โข Supports CPU and GPU inference โข Works on Linux and macOS Apple Silicon ๐ช๐ต๐ฎ๐ ๐๐ผ๐ ๐ฐ๐ฎ๐ป ๐ฑ๐ผ โข Run Llama, Qwen, Mistral, Mixtral locally โข Test large models without cloud GPUs โข Prototype agents on cheap hardware
