An owner may first notice an AI system through its outputs: a draft supplier reply, a quote follow-up or an order-sheet update. Behind each draft is a request to a language model. A gateway can record each model request and its cost, while keeping provider keys in the gateway.¹

When a business has three or four tools that use language models to prepare text, the access keys end up stored in different places and the costs are recorded separately. One key sits in the email automation, another in the report script, a third in a test someone forgot to switch off. Without a shared view, it can be hard to tell what the tools cost or where to look when a draft fails to appear.

One approach is to send model requests from agents and automations through an AI model gateway. This article explains what the gateway does, where the keys live, who sets spending limits, what happens when a model fails, and what a named person checks in the log.

What an AI model gateway does, in plain words

Take one ordinary task. A supplier emails to say a delivery will be two days late, and your system prepares a reply for your team to check. To write that reply, the system sends the email and some instructions to a model provider, which is the company whose model actually writes the text.

To reach the provider, a program needs an API key. That is a long password that one program uses to talk to another, and it is also how the provider knows whom to bill.

A gateway is one checkpoint that sits between your tools and the model providers.³ They send their requests to the gateway, and the gateway passes them on, deals with the API keys and handles the cases where a provider fails.³ A gateway of this kind can send requests to more than one provider, so you are not tied to a single company.⁴

LiteLLM, Cloudflare AI Gateway and TrueFoundry are examples of gateway products.¹,⁵,⁶ LiteLLM says its open-source gateway can be deployed;² ask whether the option proposed for you runs on your own server or is a paid online service.

Decide where the gateway will run and how it will connect to your spreadsheet and email system. It runs on a small server or as an online service that whoever builds your system sets up and looks after, while your sheet and inbox stay as they are. Lindevo connects the tools you already use, such as Gmail, Microsoft 365, WhatsApp and Google Sheets, as part of the project.⁷ Before launch, Lindevo agrees with your team who is responsible for the system and how support will be provided.⁷

Ask where customer emails are processed before using a gateway, and check what obligations apply. Get the answer in writing before go-live.

If you use only one ready-made tool, ask its supplier how it handles API keys and spending; you may not need a gateway of your own. If a single sheet script is your only tool calling a model, a gateway is probably more than you need; revisit the question when a second tool arrives. Whoever you hire to build it, local or not, should answer these questions about keys, limits and logs before starting.

This is also one of the reasons an AI system is more than a chatbot.

Why the keys stay in one place

Keep provider API keys in the gateway rather than inside each tool. The gateway records every request and what it cost, and the provider keys never leave it.¹ The keys are stored once, and access to them is controlled from one central set of rules.³

Scattered API keys are a real risk for a small business. A key copied into a spreadsheet script or a shared document can be found and used by someone else, and the bill goes to you.

Each app or person can receive a virtual key with a budget and rate limit.¹ In plain words, the provider API key gives access to your account with the model provider, and a virtual key is a separate pass for one tool whose access can be withdrawn without changing the provider API key. The email draft tool has one, the monthly report has another, and a developer testing something has a third. When a tool is retired or a contractor finishes, consider withdrawing only the key assigned to them.

Agent-to-agent calls using virtual keys appear in logs with their costs.¹ If it is a bought tool with its own model access, ask its supplier whether it can send requests through your gateway.

Spending limits: who sets them and what happens when one is hit

A rate limit is the number of requests allowed in a given time, such as per minute. LiteLLM lists budgets and rate limits alongside usage tracking by key, team and organisation.⁸

What happens at the limit depends on the rules. Usage above a set level can mean that the speed at which requests are processed is limited, that requests are moved to a cheaper model, or that they are blocked.³ With a hard cap, requests simply stop once the budget is used up.²

To see where costs come from, gateway analytics can show the number of requests, token usage and the total cost.⁵ Token usage is, roughly, how much text the models processed for you. The figures can be split by model, team, user or other attached labels.³

Who sets the limits? Agree on limits during the build based on the work you expect the system to handle. Have a named person set the limits and review the cost view; decide what figures your team needs to see.

Warning

A hard cap stops requests. Agree in advance what message will appear when a limit is reached. The office manager should see that draft preparation has paused, rather than having to guess from an empty list of drafts waiting for review.

When a request fails or no answer arrives in time

A timeout occurs when no answer is received within the set time.

A fallback chain specifies the models to try in order when an error or timeout occurs.³

Workflow · One draft request through the gateway
  1. Trigger

    Draft requested

    A supplier email arrives and the automation asks for a reply draft.

  2. Prepared by the system

    Key and budget checked

    The gateway checks the app's own key, budget and rate limit before sending anything.

  3. Prepared by the system

    First model asked

    The request goes to the first model in the chain, and its cost is recorded.

  4. Prepared by the system

    Backup model tried on error

    If the first model returns an error or times out, the next model in the fallback chain is tried.

  5. Checked by a person

    Exception reviewed

    If none of the models can complete the request, or a cap is hit, the named person reads the request log and decides what to do.

  6. Result

    Draft waits for approval

    The draft lands in the review queue; nothing is sent until a team member approves it.

Two rules matter here. First, a person checks a draft prepared by a backup model just like any other draft. Second, every model in the chain must handle Latvian, Lithuanian or Estonian well enough for your customers. A backup that writes good English but clumsy Latvian is not a real backup. For more on human review, see building reliable AI workflows with human review.

What a person checks when a request fails

Your system should have one named person who owns exceptions. In a small team, this can be someone at the company that builds your system, with one person on your side, often the office manager, who is told when drafts pause and approves any change. When a request fails, a cap is hit, or a draft looks odd, that person reads the log. Logs from a gateway like this should record the app or project, the model originally chosen and the model that actually answered, the provider, the status, the usage, the history of backup model attempts and the cost.⁴

There is also a view for single requests, where the person can open one call and see how routing and fallbacks were applied to it.³

Find the source app

Which tool sent the request: the reply drafts, the order entry, or the weekly report?

Compare the chosen and answering model

Check which model was originally chosen and which one actually answered. If a backup answered, the draft may need a closer look from your team.

Read the status

Note whether the request succeeded, timed out, returned a provider error or was stopped by a limit.

Read the backup attempt history

See which models were tried, in what order, and what the result of each attempt was.

Check the cost

Look at what the request and its retries cost.

Decide and tell you

The named person fixes the cause, raises a limit only with your agreement, or leaves it as it is. Then they tell you what happened in plain words.

Testing it before trusting it, and what it means for your build

A useful method is to cause a controlled failure and then check what triggered the fallback, the order of attempts, which model was served in the end, the status codes, the cost impact and whether the trace is visible.⁴

Run a quota test with a deliberately small limit.⁴ Use that test to check which error appears, whether the log names the quota source, and whether fallback is allowed after a rate limit.⁴

Check whether each answer is correct, its tone, and whether it matches your price list separately, because the gateway's checks do not cover any of these. That job stays with your sources and with the person who approves the draft.

Gateway monitoring data can show request counts, token usage and costs by model, and LiteLLM allows teams to swap models without changing code.²,³ Consider whether running AI models on a server you control is important before choosing them. One architect at a company using such a gateway described changing the model behind their tools as a simple settings change in the gateway, with no code changes needed.⁸

How much work this takes depends on how many of your tools call models and how often limits need to change. Ask where the gateway will run and how many requests it will handle before estimating hosting costs. If only one tool calls a model, compare using the provider directly with adding a gateway. If the gateway needs its own hosting, ask for that cost to be shown in the quote as well. Ask for the build and the expected model usage as two separate figures, so you know what is one-off work and what depends on volume each month. You can read more about how Lindevo works with a client.

Frequently Asked Questions

Does a small business need an AI model gateway?

Consider the gateway as part of how the system behind your drafts is built and run. A gateway can put an AI stack behind one key, show who is driving spend, and record requests and their costs.¹,² Ask whoever builds your system where the keys live and who sees the costs.

Does a gateway make replies slower?

Very little. LiteLLM reports that, in benchmarks measured with AI Gateway Bench, its Rust AI Gateway adds 0.66 milliseconds of latency at p99.² Caching can serve requests without asking the provider again, reducing waiting time and expense.⁵ Check whether routine questions can use cached responses, and review how unique customer emails are handled.

What happens to my drafts when the spending limit is reached?

Agree on the rules for handling a cap before launch. The speed at which requests are processed can be limited, or requests can be sent to a cheaper model or blocked.³ With a hard cap, requests stop when the cap is hit.² Decide who should be told when a cap is hit and who can approve raising it.

Does a fallback model handle Latvian as well as the main one?

Not always. Before adding a model to the fallback chain, test its Latvian, Lithuanian or Estonian on real customer enquiries. A person still checks the drafts it prepares.

Who holds the access keys to the AI models?

The provider keys are stored once in the gateway, not in each tool.¹,³ Each app or person gets its own virtual key with a budget and a rate limit.¹ Plan how to withdraw access when you retire a tool or end a contractor's access.

Sources (8)

Have a process you'd like to talk through?

Book a call (opens in a new tab)