AI Stack Library · Watch and back up
How to back up and monitor an AI server, and keep its disk from filling up
To back up and monitor an AI server, write each backup to a temporary file, check it, and only then swap it in; keep several generations, with one encrypted copy offsite; and restore one on purpose before you need it. Then watch the few things that stop work when they break: the machine is reachable, its services answer, scheduled jobs ran, last night's backup is good, the disk has room, and spending is inside its caps.
| If you are | Start with |
|---|---|
| An owner with business files and agent memory on the machine | A dedicated always-on machine, then a nightly backup of everything that cannot be rebuilt, plus one encrypted copy offsite |
| A developer with databases on the machine | A dedicated always-on machine, then database dumps written to a temporary file, checked, then swapped in, kept in dated generations |
| A business holding client data | A dedicated always-on machine, then encrypted offsite copies, a written rule for how long each is kept, and a restore you have tested |
| A creator with large video and audio files | A dedicated always-on machine, then masters on a separate drive or offsite, with caches and renders you can remake left out |
What to back up, and what to leave out
Back up what you could not make again. On an AI machine that is a short list, and it is the valuable part: the memory folder, the rules, the skills, your records of decisions, the databases and the client files.
Leave out what can be downloaded or rebuilt. Downloaded models, caches and build output are often the biggest files on the disk and the least worth copying.
A table of eight kinds of file on an AI machine, whether each goes in the backup, and why. Memory, rules, skills, decision records, databases, client files and media masters are backed up. Downloaded models, caches and build output are left out because they can be downloaded or rebuilt.
| Kind of file | Back it up? | Why |
|---|---|---|
| Memory folder and notes | Yes | Months of context about your business; nothing can regenerate it |
| Rules files and skills | Yes | Your procedures, written and corrected over time |
| Decision records | Yes | The reason behind each choice, which nobody remembers in a year |
| Databases | Yes, as a dump | A dump tool can copy a database while it is in use |
| Client files and settings | Yes | Unique, and often covered by an agreement |
| Video and audio masters | Yes, to a separate drive or offsite | Unique, but too large for the nightly copy |
| Downloaded models | No | Download them again |
| Caches and build output | No | Rebuilt automatically |
Write, check, then swap
The classic backup bug is writing the new copy straight over the last good one. If the run fails halfway, you are left with a file that has the right name, today's date and nothing useful inside. The most dangerous backup is the one that looks finished.
So write each new backup to a temporary file, check it, and only then swap it into place. When a check fails, yesterday's good copy stays exactly where it was, and an alert goes out.
Four steps. The new backup is written to a temporary file. It is checked: it opens, it decompresses, it is not suspiciously small and it holds what it should. Only then is it swapped into place. If any check fails, the last good backup stays and an alert goes out.
- Write to a temporary filenever over the last good copy
- Check it
- it opens
- it decompresses
- it is not suspiciously small
- it holds what it should
- Swap it into placeonly after every check passes
- If a check failskeep the last good copy and send an alert
What checking a backup means
Look at the size first. A nightly database dump that suddenly weighs less than a photo is not a backup, whatever its file name says.
Then let the backup tool read its own work. restic's check command verifies the structure of a backup store by default, and with --read-data it reads every stored piece back, which takes longer and can cost more bandwidth when the store is in the cloud. Borg's check command verifies a repository and its archives, and --verify-data checks the content itself, which its docs warn is very time consuming. Run the quick check every night and the full one less often.
For a database, back up a dump rather than the database's live files. PostgreSQL's pg_dump, for example, makes a consistent export while the database is being used. Its own docs add that, except in simple cases, pg_dump is generally not the right choice for regular backups of a production database, so a busy one needs one of the database's other backup methods.
Keep several generations, and one copy somewhere else
A backup paper by US-CERT, still hosted by CISA, sets out the 3-2-1 rule: keep three copies of any important file, on two different kinds of media, with one copy offsite, outside your home or business.
Keep several dated generations, too, so a problem you notice next week can still be rolled back. Never delete the newest, and never let a broken copy count toward the number you keep.
On a Mac, Time Machine is a sound first layer. Apple says it keeps hourly backups for the past 24 hours, daily backups for the past month and weekly backups for all previous months, and deletes the oldest when the backup disk is full. It is still one copy, in one building.
For the offsite copy, encrypt before anything leaves the machine, and keep the encryption key somewhere other than the machine itself. Borg supports authenticated encryption as an option, and restic's site says it backs up securely, using cryptography in every part of the process. rclone is a command-line program for managing files on cloud storage, and its check command compares the sizes and hashes of the files at both ends without changing either.
Two panels. On the machine: last night's backup, which is never deleted, several earlier nights, and a check after every run. Offsite: the same backups encrypted before upload, kept in another building on rented storage, and compared against the originals after each upload.
On the machine
- Last night's backup, never deleted
- Several earlier nights
- Checked after every run
Offsite
- The same backups, encrypted before they leave
- Kept in another building, on rented storage
- Compared with the originals after each upload
Restore one before you need it
Once a month, restore one backup into an empty folder and open what comes out: a memory note, a client file, the database loaded into a scratch copy. Write down the date and how long it took. That number is roughly how long you will be stuck on the bad day.
Watch six things first
Monitoring an AI machine starts small. Watch the few things that stop work when they break, and nothing else yet.
A six-row table. Reachable: the machine answers from outside, checked from outside. Services answer: each core service gives a real answer, checked on the machine. Jobs ran: every scheduled job checked in on time, checked from outside. Backups fresh: last night's backup exists, passed its check and is a normal size. Disk room: free space above the warning line and not falling fast. Spend inside caps: bills under their monthly limits.
| Check | Healthy looks like | Where the check runs |
|---|---|---|
| Reachable | The machine answers from outside | Outside the machine |
| Services answer | Each core service returns a real answer, not just a running process | On the machine |
| Jobs ran | Every scheduled job checked in on time | Outside, so silence gets noticed |
| Backups fresh | Last night's backup exists, passed its check and is a normal size | On the machine |
| Disk room | Free space above your warning line, and not falling fast | On the machine |
| Spend inside caps | Model and media bills under their monthly limits | On the machine, or the provider's dashboard |
Healthy means a real answer. A service that is running but answers nothing looks fine to a check that only asks whether it is running.
Uptime Kuma is a self-hosted monitoring tool that checks your services and lists a push monitor among its monitor types; it sends alerts through more than 90 notification services. Healthchecks.io stays silent while a job's pings arrive on time and raises an alert as soon as one does not. ntfy sends push notifications to your phone from a script with a plain web request, and you can run your own ntfy server.
Quiet when healthy, loud when stuck
An alert that fires every day gets ignored, and on the day it matters it is ignored too. Tune each check so it says nothing while things are fine, fixes what it safely can on its own, and tells a person only when something needs one.
Send that alert where you will see it: a push to your phone, not an email folder you open on Fridays.
Keep the disk from filling up
AI machines fill disks faster than office computers. Local models are large downloads, video work writes a large file for every clip, and logs grow every night. When the disk fills, jobs fail halfway and leave broken files behind them.
Watch two numbers: free space, and how fast it is falling. Free space can look fine at bedtime and be gone by morning, so warn on the rate as well as the level. On a Mac, the Storage section of System Settings shows how much space each category of file is using, which is the place to start looking.
Before any big job, such as a model download, a video render or a backup, check that there is room, and skip the run if there is not. A skipped run is healthy. A full disk is not.
Two questions asked in order. If the file can be downloaded or rebuilt, delete it. If there is a checked copy somewhere else, move it aside for two weeks and then delete it. Otherwise keep it and ask the owner.
- Can it be downloaded again or rebuilt?Yes Delete it
- Is there a checked copy somewhere else?Yes Move it aside for two weeks, then delete it
- None of the aboveKeep it, and ask the person who owns it
Never delete the only copy of anything, and never delete a backup to make room for the next backup.
Then watch the model calls
Once the six checks are quiet, add tracing for the agent's model calls: what each call cost, how long it took and how often it failed. It is useful. It is the last of these to set up, not the first.
What goes wrong
- A backup written straight over yesterday's, by a run that failed halfway.
- Backups that ran every night for months and were never once restored.
- One copy, on the same disk as the data it protects.
- A monitor whose worst warning sounds the same at half full and at empty.
- An alert that fires every day, and is ignored on the day it matters.
- A cleanup script that was written, tested and never scheduled.
Questions
How often should you back up an AI server?
Back up what you could not rebuild at least every night, and more often for anything that changes all day, such as a busy database. Keep several dated generations and one encrypted copy offsite.
Is Time Machine enough for an AI server?
It is a good local layer on a Mac: Apple says it keeps hourly, daily and weekly backups and deletes the oldest when the backup disk is full. It is one copy in one place, so add an encrypted offsite copy, and back up databases as dumps.
What should an AI server alert you about?
Six things to start: it cannot be reached, a core service stops answering, a scheduled job did not run, last night's backup is missing or failed its check, the disk is close to full, or spending passes its cap.
Terms on this page
Sources
- US-CERT: Data Backup Options (hosted by CISA), checked 2026-10-23
- Apple Support: Back up your Mac with Time Machine, checked 2026-10-23
- restic, checked 2026-10-23
- restic docs: Working with repositories (check), checked 2026-10-23
- BorgBackup docs, checked 2026-10-23
- BorgBackup docs: borg check, checked 2026-10-23
- rclone, checked 2026-10-23
- rclone docs: rclone check, checked 2026-10-23
- PostgreSQL docs: pg_dump, checked 2026-10-23
- Uptime Kuma, checked 2026-10-23
- Healthchecks.io docs, checked 2026-10-23
- ntfy docs, checked 2026-10-23
- Apple Support: Free up storage space on Mac, checked 2026-10-23