{"id":10174,"date":"2024-08-30T11:13:31","date_gmt":"2024-08-30T19:13:31","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=10174"},"modified":"2025-06-18T10:51:45","modified_gmt":"2025-06-18T10:51:45","slug":"build-local-llm-server","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/build-local-llm-server\/","title":{"rendered":"Building a Low-Cost Local LLM Server to Run 70 Billion Parameter Models"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

A guest post from Fabr\u00edcio Ceolin, DevOps Engineer at Comet. Inspired by the growing demand for large-scale language models, Fabr\u00edcio engineered a cost-effective local LLM server capable of running models with up to 70 billion parameters. In this guide, you’ll explore how to build a powerful and scalable local LLM environment, enabling you to harness the full potential of these advanced models.<\/em><\/p>\n\n\n\n

\n\n\n\n

Introduction<\/h2>\n\n\n\n

As the demand for large-scale language models (LLMs) continues to grow, particularly for running local AI agents, developers and researchers face significant challenges in managing the computational requirements of these powerful models. Running LLMs locally presents several hurdles, including the need for substantial hardware resources, high operational costs, and complex software configurations. These challenges are often a barrier for those who wish to experiment with, debug, and optimize LLM code without relying on expensive cloud-based solutions.<\/p>\n\n\n\n

This article addresses these challenges by providing a comprehensive guide to building a low-cost local LLM server capable of running models with up to 70 billion parameters. The proposed solution leverages affordable and repurposed hardware, initially intended for Ethereum mining, combined with advanced software tools like Kubernetes and OLLAMA, to create a scalable and efficient environment for LLM development.<\/p>\n\n\n\n

Following this guide, readers will learn how to assemble the necessary hardware, configure the software environment, deploy LLMs locally and run basic LLM queries. This approach reduces the cost associated with LLM experimentation and provides greater control over the development process, making it accessible to a broader audience of developers and researchers.<\/p>\n\n\n\n

Audience<\/h2>\n\n\n\n

This guide is primarily intended for developers and researchers with some familiarity with hardware setups and software configurations, particularly around GPUs, Docker, and Kubernetes. If you are less familiar with these technologies, additional resources and explanations are provided via links to help you follow along.<\/p>\n\n\n\n

Hardware Selection<\/h2>\n\n\n\n

The first step in building a local LLM server is selecting the proper hardware. Depending on the response speed you require, you can opt for a CPU, GPU, or even a MacBook<\/a>. For this project, I repurposed components originally intended for Ethereum mining to get a reasonable speed to run LLM agents. This approach provided both relative affordability and the computing power needed to run LLMs around the clock. I combined six GPUs to achieve a total of 96 GB of VRAM, essential for running LLMs with 70 billion parameters, which requires 84 GB of VRAM using 8-bit quantization<\/a>. Here\u2019s the hardware I used:<\/p>\n\n\n\n