What it takes to run a language model locally

If you want customer emails, price lists and contracts kept in-house, keeping AI on your own company server sounds sensible: the software sits in your office and the data stays with it. Before you run a language model locally, though, it helps to see the full list of what that involves. We discuss access, data handling and any third-party services with you before we build anything.¹

A language model is software that reads a question written in ordinary words and writes a reply. Language models are one of the tools of natural language processing, the field that teaches computers to read and write everyday text. Running one yourself means choosing hardware, installing and updating software, checking that the answers are good enough in Latvian and your other languages, and naming the person who looks after it.

This article goes through each of those points in order. It ends with an honest look at when a hosted tool, one that runs on someone else's servers, is the simpler choice.

What it means to run a language model locally on your own server

Running a local model requires its files and software to be installed.² A program such as Ollama runs the model.³ You download the model file and keep it on your own machine.³

Once the model file is on your machine, it can work completely offline.³ That holds as long as the setup does not depend on a remote service for any part of the work.²

Other software talks to the model through an API, which is the doorway other software uses to ask it questions. Ollama's API uses the same format as OpenAI's cloud service, so many existing tools can connect to it without changes.³ In practice, this gives your team a natural language interface: your inbox tool sends a customer email to the model, and a draft reply comes back.

Be realistic about that connection. Plan for setup work before linking the model to the places your team receives messages. In practice, a small connector program or automation tool picks up each new message, sends it to the model and puts the draft where staff will see it, such as a sheet row. Without anyone technical, start by pasting messages in by hand and copying drafts back, and automate only once the answers have proved good enough. The model does not reach your messages on its own.

A downloadable model brings setup and maintenance with it.² If your real goal is answers drawn from company documents, read a plain guide to building a company knowledge base first. The model is only one piece of that.

Memory decides which model you can run

Parameters are the internal settings a model learned during training. Model sizes are described by parameter counts, such as 7B or 70B.⁴ "7B" means seven billion. At full FP16 precision, a higher parameter count means more memory is needed.⁴ At full, uncompressed precision, each billion parameters needs about 2GB of memory, so a 7B model needs about 14GB and a 70B model about 140GB.⁴

The memory that matters most is VRAM, the memory on the graphics card. VRAM is the main hardware limit on which model sizes you can run without heavy compression.⁵

Quantization is the usual answer, and it simply means using a compressed model, much like a smaller photo file that looks almost the same. Q4_K_M quantization cuts memory roughly fourfold compared with FP16 while preserving about 95% of benchmark quality on most models.⁴ Full models often need 20–40GB of VRAM, while compressed versions can fit on a card with 8GB.⁶ With the right settings, ordinary 8GB graphics cards can run capable models.⁶ At the larger end, two RTX 4090 cards together give 48GB, enough to run Llama 3.1 70B at Q4_K_M.⁴

Speed is measured in tokens per second, which roughly means how fast the words appear on screen. The whole model has to fit in the graphics card's memory. A 7B model running at 45 tokens per second on a GPU can drop to 8 if even 10% of its layers hit system RAM.⁴ On a CPU-only machine, Ollama may produce about 3–6 words per second instead of 30 or more.³

For drafting a reply that a person reads later, slow can be acceptable. For staff waiting at a screen, it usually is not.

The rest of the machine: power, processor, storage and system

Consider the graphics card alongside the rest of the machine. Strong cards such as the 4090, and setups with several cards, call for a power supply rated at 1000–1200W or more.⁵ A capable processor keeps the machine responsive while it prepares text for the model.⁵ Fast storage shortens the time it takes to load a model.⁵ For small-model inference and lightweight local experiments, 32GB of system memory is enough.⁵

Ask a local supplier for a current quote before deciding. Then compare it, with power and staff hours, against a year of hosted subscriptions for the same users. Check the machine's power draw against your own electricity tariff too.

For AI workloads, Linux offers strong graphics driver support and fewer compatibility problems.⁵ If nobody in the office knows the operating system, either have an outside party set up and look after the server, or treat it as a reason to choose a hosted tool.

Then plan for the bad day. Hardware fails, and graphics cards wear out over time, which can stop work.⁷ Decide in advance what the team does when the server is down. Usually that means writing replies by hand, as before, until it is fixed.

Updates and the settings someone has to look after

Local models are popular, and choice is wide. Llama 3.1 has 115.9 million pulls in the Ollama model library.⁴ In our experience, new model versions and runner updates arrive often. Someone has to decide which ones to apply, apply them, and check that nothing has broken.

Some default settings also need changing. The context window means how much text the model reads at once, measured in tokens, which are small pieces of words. Ollama's default is 4,096 tokens, and it can be raised with one setting.⁴ A long email thread with a quote attached may not fit within the default, and the model will then miss part of it.

Ollama also unloads the model from memory after a few minutes of inactivity.³ After a quiet spell, check whether the first answer takes longer. Consider how much memory you can spare between requests.

How to check the answers are good enough

Quantization reduces the storage needed for model weights, but it can lower answer quality.² The roughly 95% figure describes benchmark quality on most models.⁴ A benchmark is a standard test, and it does not show how well the model will handle your company's documents and customer questions. It does not know your price list, your opening hours or how your customers phrase a complaint.

So build your own test. Collect real questions from an ordinary week: an enquiry about opening hours, a supplier asking about a late order, a customer who wants a quote changed. Write them as your customers actually do, in Latvian, Lithuanian, Estonian, Russian or English, with their spelling mistakes included. For each one, note what a good answer must contain.

Run the set through the model and have someone who knows the work mark each answer. Then run the same set again after every change: a new model version, a different compression level, a new setting. If the scores drop, do not apply the change.

No general list of "good Latvian models" replaces this test. Results vary with the model, the compression level and the way your own customers write, so try candidates on the same questions and keep the one that scores best.

Here is why a person reviews every draft.

Who looks after it, and what they do each month

A local model needs one named owner, plus a backup for holidays and sick days. Without a name, updates get skipped, quality slowly drops, and nobody notices until a customer does. Hardware can fail and graphics cards wear out,⁷ so the owner also watches the machine itself.

Be honest about the hours. Allow time for routine upkeep and for problems when they arise. Those hours come from the owner's usual work, so plan for that.

Plan for upkeep before anything is built. See a plan for the day after launch.

When a hosted tool is the simpler choice

Local models have real advantages. Developers choose them for faster responses, lower running costs and better data privacy than cloud services offer.⁶ They keep working without an internet connection once installed.³

Hosted tools have their own advantages. Premium access to a cloud AI service typically costs about $20 a month.³ When comparing hosted tools, consider who handles hardware, updates and repairs.

Starting with a hosted tool is mostly a matter of choosing one, creating an account for the people who will use it, and agreeing in writing which data may and may not be pasted in. For customer messages, ask a hosted provider where data is stored, whether it is used for training, and for its data processing agreement; on a local server, decide who has access and when old messages are deleted.

A fair rule: go local when a contract, a regulation or a firm company policy says the data must stay on your own hardware, or when your volume of use is very high. Otherwise, start with a hosted tool and look again later.

Either way, the useful part is the same. Drafts and answers from your approved documents, each with its source shown, can be built on a local model or on a hosted one. The choice of where the model runs comes second to choosing the task.

Frequently Asked Questions

How can I run large language models on a graphics card with little memory?

Use a compressed, or quantized, version of the model. With quantization and proper optimization, capable language models can run on consumer GPUs with 8GB of VRAM.⁶ Q4_K_M quantization cuts memory roughly 4x compared with FP16 while preserving roughly 95% of benchmark quality on most models.⁴ Make sure the whole model fits in the graphics card's memory, because speed drops sharply if part of it spills into system memory.⁴

How do I run a model locally on a company server?

Install a local runner and download a model. Ollama's local API uses an OpenAI-compatible format, so most existing tools work with it.³ Linux offers strong GPU driver support and fewer compatibility issues when working with popular deep learning frameworks.⁵ Plan for setup and ongoing maintenance, not just the install day.²

Does a local language model work without internet?

Yes, once the model and software are installed and the setup does not rely on a remote service.²,³ You will still need a connection to download updates and new model versions when you choose to apply them.

Will a local model answer well in Latvian?

It depends on the model and on how heavily it is compressed, so test it rather than assume. Run your own set of real customer questions in Latvian and the other languages you use, and repeat the test after every change. A person still checks the drafts before they are sent.

Is a local model cheaper than a cloud AI subscription?

Hosted services typically cost about $20 a month for premium access.³ Running a downloadable model brings setup and ongoing maintenance.² We choose the model per task: a commercial model from a provider such as OpenAI, Anthropic or Google, or an open model where your data rules call for it.¹