Testing an 8-GPU, 768 GB VRAM Liquid-Cooled Workstation for Local LLM Inference

A hands-on evaluation of the Camino Grando, a 4U liquid-cooled workstation that houses eight NVIDIA RTX Pro 6000 Blackwell Server Edition GPUs with 768 GB of combined VRAM, alongside an AMD EPYC 9474F CPU and 512 GB of DDR5 memory. It examines how such a box performs when serving large language models locally, from single-user inference to heavy multi-agent workloads.

■ Hardware and Setup
- 4U chassis layout, custom copper water blocks, shared CPU/GPU cooling loop, and the dripless quick-disconnect fittings
- Four hot-swappable PSUs, real-world power wiring and circuit requirements, PCIe lane allocation across the GPUs, and M.2 storage trade-offs
- Noise behavior under load and measured GPU/CPU temperatures
■ LLM Benchmarking
- Five models served with vLLM using tensor parallelism, covering generation speed, prompt processing, and time-to-first-token at short and long contexts
- Prefix caching behavior, a 300W versus 600W power cap comparison, and the effects of multi-GPU scaling over PCIe
■ Agent and Multi-User Load
- Concurrent coding agents launched from multiple machines, with GPU utilization and VRAM monitoring at large agent counts
- Throughput and latency scaling from 1 up to 64 simultaneous users
■ Fit and Ecosystem
- Typical use cases, idle power draw, vLLM recipe configuration, and current driver/model compatibility status

開発者や中小企業で、ローカルLLMの推論環境を検討している方、大規模VRAMを活用したい方に向いた内容です。視聴後は、マルチGPU構成で何を確認すべきか、電源・設置環境・並列実行の設計といった判断材料が得られ、実際にvLLMレシピを設定する際の着眼点も把握できます。