How to monitor database backups
Backups fail in boring ways. The disk fills up, a database password gets rotated, a server is rebuilt and the crontab doesn’t come along. Nothing complains, and the problem shows up on the day you need to restore. The fix is simple: make the backup report every run, and have something notice when the report doesn’t come.
What can go wrong
- The job doesn’t run at all. Cron is stopped, the server was replaced, the crontab line was lost in a migration.
- The dump fails but the script “succeeds”. In
pg_dump | gzip > filethe exit code is gzip’s, so a failed dump still exits with 0 unless the script usesset -o pipefail. - The dump is empty or cut short. Wrong database, missing permissions, full disk.
- The upload fails. Expired storage keys, a full bucket, a network problem.
- The job hangs. A lock or a slow network keeps it running forever, so it neither succeeds nor fails.
A backup script that reports
This Bash script for PostgreSQL covers all five cases. It tells Heartcron when it starts, reports any error with the line it failed on, checks the dump size and pings on success with the size in the body:
#!/usr/bin/env bash set -euo pipefail PING=https://ping.heartcron.com/<uuid> FILE=/var/backups/app-$(date +%F).sql.gz # "started": a run that never finishes is reported as hung curl -fsS -m 10 --retry 3 -o /dev/null "$PING/start" || true # any failing command reports the failure right away trap 'curl -fsS -m 10 --retry 3 -o /dev/null --data-raw "failed at line $LINENO" "$PING/fail"' ERR pg_dump "$DATABASE_URL" | gzip > "$FILE" # a dump under 1 MB means something is wrong; adjust for your database SIZE=$(stat -c %s "$FILE") if [ "$SIZE" -lt 1000000 ]; then echo "dump is only $SIZE bytes" >&2 false fi aws s3 cp "$FILE" "s3://my-backups/$(basename "$FILE")" curl -fsS -m 10 --retry 3 -o /dev/null --data-raw "size=$SIZE" "$PING"
A few details matter here:
set -euo pipefailmakes the script stop at the first failing command, including a failure inside a pipe.- The
ERRtrap sends/fail, so you get the alert within seconds instead of waiting for the next deadline. - The start ping ends with
|| true: if the monitoring service is unreachable, the backup still runs. - The text sent with
--data-rawis saved with the ping, so the dashboard shows the dump size for every night. stat -c %sis the Linux form; on macOS and BSD usestat -f %z.
Run it from cron at a quiet minute rather than exactly at midnight, and keep a log:
17 3 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1
Set up the check
- Create a check with the cron schedule
17 3 * * *and your server’s time zone. - Set the grace period longer than a normal run. If the backup takes 20 minutes, 45 to 60 minutes is reasonable. Because the script sends
/start, a run that started but didn’t finish within the grace period is reported as hung. - Pick where alerts go: Telegram, Slack, Discord, email or a webhook.
- Put the check’s URL into
PINGin the script.
MySQL and MariaDB
Only the dump line changes. --single-transaction takes a consistent snapshot of InnoDB tables without locking them, and --quick streams large tables instead of loading them into memory:
mysqldump --single-transaction --quick app | gzip > "$FILE"
Keep the password in ~/.my.cnf with chmod 600 rather than on the command line, where other users can see it in the process list.
Test the alerts once
- Run the script with a wrong
DATABASE_URL. A failure alert should arrive within seconds. - Comment out the crontab line for a night. An alert should arrive after the scheduled time plus the grace period.
Monitor restores too
A successful backup only means a file was written. Once a month, restore the latest dump into a scratch database and run a sanity query, such as counting rows in a table that is never empty. Make that a scheduled job with its own check, and you’ll know the backups actually work.