A new colleague asks how returns work for products with opened packaging. Someone stops their own work, searches three folders and finds an old note that may or may not still be correct. In a business of 15 or 30 people, repeated questions can be difficult to answer when information is scattered across files and people's heads.

You may want that internal information to be searchable without sending customer records and price lists to an outside service. That leads to a practical question: is a small local language model, running on a computer in your own office, good enough for the job?

The answer depends on the task. This article compares two everyday jobs: searching approved company documents, and drafting replies to customers. It also covers how to test a model before you trust it, and who stays responsible for the answers.

What a small local language model is

A language model is software that reads text and writes text. It is the part of natural language processing that most people now meet through chat tools. If you have ever searched "what is a large language model", the short answer is the same kind of software, built at a larger size and usually run on a provider's servers.

A local model runs on a computer you own. Its size is measured in parameters, the internal settings it learned during training. More parameters usually means better writing, but also more memory. Microsoft's Phi-4-mini has 3.8 billion parameters and runs on a machine with 8GB of memory.¹ Meta's Llama 3.3 8B is a widely recommended starting model that handles summarising and question answering on 8GB hardware.²

Consider testing a quantized model to see whether its memory use and answer quality suit your work. For most models, a common compression level keeps scores on a general knowledge test within one to three points of the full version.¹ Because the model runs on your device, sensitive business data stays there instead of going to external servers.³

This matters most when company knowledge lives in people's heads and you would rather not copy it to someone else's servers.

Searching company documents: when a small model is enough

The first job is narrow. A colleague asks a question, and the answer already exists somewhere in your approved documents: the returns policy, the price list, the service checklist. The system uses retrieval, which means it first finds the few passages that match the question and then writes an answer only from those passages.

A narrow job like this suits a small model. It does not need to know the world. It needs to read a few paragraphs and summarise them clearly. A well-chosen local model can handle everyday work such as summarising documents and answering questions.²

The context window is how much text the model can read at once. A context window of 128K tokens (pieces of words) is large enough to take in whole documents and long conversation histories.² With retrieval, the model usually reads much less than that, which keeps it fast. The data stays on the device the whole time.¹

Workflow · Staff question to checked, sourced answer
  1. Trigger

    A colleague asks a question

    For example: what are our rules for returning products when the packaging has been opened? It is typed into the shared chat or a search box.

  2. Prepared by the system

    Relevant passages found

    The system searches only the approved documents and picks out the few passages that match.

  3. Prepared by the system

    Short answer prepared with sources

    The small local model writes a short answer from those passages only and links each source.

  4. Checked by a person

    Person checks the source

    The colleague opens the linked document before acting. If the documents have no answer, the question goes to the named owner of that topic.

  5. Result

    Answer used or document corrected

    The answer is used. Any gap or outdated document is passed to the person who keeps the documents up to date.

The limit is plain: the answer can only be as good as the documents. If the returns policy changed in spring and nobody updated the document, the system will repeat the old rule. That is why the source link matters. This is the kind of work behind answers from your approved documents.

Customer replies: when to choose a larger model on a provider's servers

The second job is harder. A WhatsApp enquiry arrives in Russian asking whether you have a free slot on Saturday and what it costs. A complaint email in Latvian needs an apology, the correct price and a fair offer. The draft has to sound natural and include accurate prices. Someone also has to judge what solution to offer the customer.

A hosted model runs on a provider's servers, and you send it the text over the internet. From here on we call it the provider's model. Provider's models are usually larger. For most practical tasks, the best local 14B models reach roughly 80–90% of ChatGPT GPT-5.2 quality.² When staff are looking for information in company documents, that gap may not matter. For a reply to an upset customer in their own language, it might.

Compression also costs more on some tasks than on others.¹ Some smaller models are built for speed: Mistral 7B uses less memory than Llama 3.3 and gives answers faster, but it puts speed ahead of quality.² That is a good trade for a quick internal answer, and a weaker one for a careful customer letter.

Whichever model drafts the reply, a person reads it and approves it before it is sent. More on how human review keeps AI work reliable.

What the computer needs to run it

A model must fit into the computer's working memory, the fast memory it uses while running. Small models fit easily. Phi-4-mini uses roughly 3.5GB at a common compression level and runs on 8GB machines.¹ A 27-billion-parameter model at exactly four bits per parameter needs 13.5GB in decimal units for its language weights alone.⁴ OpenAI states that its compressed MXFP4 version runs within 16GB of memory.⁴

On a Mac with Apple Silicon, unified memory is shared between the GPU, CPU, operating system and active applications.⁴ On Apple Silicon, a laptop with 16GB of unified memory does not have all 16GB available to the model while the operating system and other applications are running.⁴ Apple unified memory can hold models larger than a single discrete GPU's VRAM, and llama.cpp supports Metal and lets the model run on both the main processor (CPU) and the graphics processor (GPU).⁴

Longer documents also cost memory. Raising the amount of text the model reads at once increases total memory use.⁴

Speed is measured in tokens per second, where a token is a piece of a word. Mistral Small 3 7B produces about 50 tokens per second on mid-range hardware with 16GB of RAM at Q4_K_M quantization.¹ That is fast enough for a short staff answer. Try it on the computer you already have before buying anything.

Test it on your own questions before you trust it

Published scores are a starting point, not proof. When Microsoft researchers inspected three widely used public test sets in their field, they found that 30 to 50 percent of the original test data was flawed.³ Qwen 3 7B scores 76.0 on HumanEval, against 72.6 for Llama 3.3.¹ Llama 3.3 8B scores 73.0 on MMLU at Q4_K_M quantization.¹ None of those numbers tell you whether it can explain your return policy.

Do not assume the model will answer well in Latvian, Lithuanian and Estonian; check the quality of the answers yourself. A simple test looks like this.

Collect about twenty real questions

Take them from the last month of staff chat, email and WhatsApp. Include easy ones, awkward ones and one or two that your documents cannot answer.

Include your customers' languages

Write some questions in Latvian, and in Russian, English, Lithuanian or Estonian if your customers use them.

Run the local model and the provider's model

Give both models the same approved documents and the same questions, so the comparison is fair.

Check each answer against its source

Open the linked document for every answer. Mark it correct, partly correct or wrong, and note where a model said it did not know.

Decide per task

The local model may be good enough to help staff find information and too weak for customer replies. Choose a model for each task, not one for everything.

Who is responsible for the result, and what does maintaining the system cost?

Running a model locally means no API costs and no rate limits.² The data stays on your device.¹ That is a real benefit, but it does not make the work free.

Warning

No per-use fee does not mean no cost. Someone has to gather the approved documents, remove old versions, set up the system with you, and keep the price list and policies current every time a price or rule changes. Staff time will also be needed to maintain the system after it is launched.

Name one person for each topic, such as returns, prices or delivery. That person keeps the documents correct and answers the questions the system cannot. If you want to plan this part, read a plain guide to building a company knowledge base.

Ask Lindevo for a quote for your engagement.

Frequently Asked Questions

Can a small local language model answer questions in Latvian?

Try the model with questions in Latvian and judge how accurate and natural the answers are. Include Latvian, and Russian, Lithuanian or Estonian if your customers use them, in your twenty test questions. If the answers in one language are weak, use a larger provider's model for that task.

Does running a model locally keep our customer data private?

Running the model locally keeps the data on your own device instead of sending it to outside servers.¹,³ You still decide who has access and which documents are approved. Answers with sources also make it easy to check where each fact came from.

What computer do we need for a small local model?

Phi-4-mini runs on hardware with 8GB of RAM and uses roughly 3.5GB at Q4_K_M quantization.¹ A 27-billion-parameter model at exactly four bits per parameter needs 13.5GB in decimal units for its language weights alone, and increasing the configured context length raises overall memory use.⁴ Run a test on the machine you already have before buying anything.

Is a small local model good enough for customer replies?

For simple drafts it can be, because a well-chosen local model can handle everyday email drafting.² For most practical tasks, the best local 14B models reach roughly 80–90% of ChatGPT GPT-5.2 quality.² Either way, a person approves every reply before it is sent.

Sources (4)

Have a process you'd like to talk through?

Book a call (opens in a new tab)