Running a 17.6 GB AI Model on a 16 GB GPU: Layer Splitting vs. Smaller Quantization

I tested what to do when an AI model barely does not fit in a GPU's VRAM. Using an RTX 5060 Ti with 16 GB and a 17.6 GB model, I compared two workarounds: splitting the model's layers between VRAM and system memory, and switching to a smaller quantization of the same model.

■ Layer Splitting
- Measuring tokens per second at 0, 48, and 56 layers offloaded
- Hitting a sharp slowdown once the split crosses a threshold
■ Smaller Quantization
- Dropping one quantization level to bring the model fully into VRAM
- Comparing prompt processing and token generation before and after
- Which approach delivered the larger gain

If you are considering running large language models locally, this gives you a practical reference for choosing between partial offloading and a lighter quantization.