Our AI systems run on our own server, not on rented black boxes: the agent team, the AI gateway, the lead radar's database, our memory and search services, and the monitoring that watches them all. The whole setup is written down as code, so it can be rebuilt on a fresh machine. It is watched around the clock, backed up every night to an encrypted copy in another data centre, and we test that the backups actually restore. We built it to run our own company, and we use the same practices when we host AI systems for clients.
What it does for us
Running AI systems yourself gives you control over data, costs and what runs where. It also means the boring parts are yours: updates, outages, backups, the night a disk fills up. This setup is how we make those parts routine:
- Nothing lives only in someone's head. Each service, its settings and its firewall rules are in a code repository. A change is reviewed like any other code change.
- Problems are noticed early. Disk space, memory, failed jobs, expiring certificates, AI spending and backup age all have alerts, and the alerts reach a person.
- Losing the server is survivable. The data that cannot be recreated is copied off-site every night, and we have restored from those copies.
- Our data stays where we put it. Business data and AI memory live on our own hardware, with encrypted backups in the EU.
How the infrastructure works
Click through the parts that keep it running. The screens are simplified illustrations with example data.
Described in code
Every service, from the AI gateway to the monitoring, is defined in configuration code with a simple on or off switch. One command builds the whole server from scratch, and the same code is how changes reach it.
Checked before it changes
Before a change goes out, automatic checks look for broken configuration, secrets that should not be in the code and settings that disagree with each other. Secrets themselves live in an encrypted vault, never in plain text.
Trigger
Watched around the clock
Metrics, logs and uptime checks cover every service and the hardware under them. Custom checks watch the things that matter here: backup age, disk health, AI spending and whether scheduled jobs actually ran.
Checked by a person
Alerts reach a person
When something crosses a line, an alert goes to a person with what is wrong, since when and a link to the right dashboard. A separate check outside the server expects a regular signal, so if the whole server goes silent, that is noticed too.
Result
Backed up every night
Every night the databases and settings that cannot be recreated are saved, encrypted and sent to storage in another data centre. The server can add new backups there but cannot delete old ones, so a break-in cannot take the history with it.
Checked by a person
Restores are tested
On a schedule, a person restores a service from backup into a separate, disconnected copy and checks it: the database opens, the encryption keys are there and the content is real, not an empty shell. Then the copy is deleted.
What we learned
Two lessons came from things that looked fine.
The first was a backup. One service kept an old, almost empty database at the usual location, while its real data lived somewhere else. A backup of the usual location would have run every night without an error and been useless. Only a restore test showed it. Now a backup counts only after it has been restored and its contents checked.
The second was the server itself. It had frozen a few times, and when we went through the records we found more freezes than we knew about, because some happened when nobody was looking. That is why a check outside the server now expects to hear from it regularly, and raises the alarm when it does not.
What it could do for your business
AI systems hosted with care. When we build an AI system for you, we can run it on infrastructure set up this way: described in code, monitored, backed up off-site and restore-tested. You get the control of self-hosting without inheriting the night shifts.
A health check for your own setup. If you already run servers or AI tools yourself, we can review how they are built, watched and backed up, and test a restore with you. Most setups have at least one backup that has never been restored.
What it does not do
- It does not make outages impossible. It makes them noticed sooner and recovered faster.
- It does not let anything change the live server outside the reviewed configuration. AI agents can propose changes; a person applies them.
- It does not treat a backup as safe until it has been restored.
Frequently asked questions
Why self-host instead of using cloud services?
For some work, control matters more than convenience: where business data and AI memory live, what each system can reach, and what it costs month to month. Self-hosting gives that control. We still use cloud services where they fit better, such as hosting this website.
Where are the backups kept?
In an EU data centre, separate from the server, encrypted before they leave it. The server can only add to that storage, not delete from it.
What happens if the server fails completely?
The configuration code rebuilds the services on new hardware, and the nightly backups restore the data. We have practised the restore part, which is the one that usually goes wrong.
Can you host our AI systems this way?
Yes. We can run the systems we build for you on infrastructure like this, or help your team set up the same practices on your own servers. Tell us what you need to run and we will suggest a first step.