Monitoring systemd timers
systemd timers are a good replacement for cron: every run is logged in the journal, missed runs can be caught up after a reboot, and a job can’t start twice by accident. What they don’t do is tell anyone when a run fails. A failed service waits in systemctl --failed until somebody looks. Two small additions fix that: a ping after each successful run and a hook that reports failures.
Keep the ping URL in one place
Put each job’s check URL into a small environment file named after its service. Both the service and the failure hook below read it:
# /etc/heartcron/backup.service
PING_URL=https://ping.heartcron.com/<uuid>
The service
# /etc/systemd/system/backup.service
[Unit]
Description=Nightly database backup
OnFailure=heartcron-fail@%n.service
[Service]
Type=oneshot
EnvironmentFile=/etc/heartcron/backup.service
ExecStartPre=-/usr/bin/curl -fsS -m 10 --retry 3 -o /dev/null ${PING_URL}/start
ExecStart=/usr/local/bin/backup.sh
ExecStartPost=/usr/bin/curl -fsS -m 10 --retry 3 -o /dev/null ${PING_URL}
ExecStartPrereports the start. The leading-tells systemd to ignore a failure of this line, so an unreachable monitoring service never blocks the backup.- For a
Type=oneshotservice,ExecStartPostruns only afterExecStarthas exited successfully. The success ping can’t be sent for a failed run. OnFailurestarts another unit when this one fails.%npasses the full unit name,backup.service, to it.
The failure hook
One template unit handles failures of every job. It reads the URL from the file that matches the failed unit’s name and sends /fail, so the alert comes right away instead of after the grace period:
# /etc/systemd/system/heartcron-fail@.service
[Unit]
Description=Report a failure of %i to Heartcron
[Service]
Type=oneshot
EnvironmentFile=/etc/heartcron/%i
ExecStart=/usr/bin/curl -fsS -m 10 --retry 3 -o /dev/null ${PING_URL}/fail
The timer
# /etc/systemd/system/backup.timer
[Unit]
Description=Run the database backup every night
[Timer]
OnCalendar=*-*-* 03:17:00
Persistent=true
[Install]
WantedBy=timers.target
Persistent=true runs a missed backup as soon as the machine boots if it was off at 03:17. Enable it and check when it fires next:
systemctl daemon-reload systemctl enable --now backup.timer systemctl list-timers backup.timer systemd-analyze calendar "*-*-* 03:17:00"
The same schedule in Heartcron
Heartcron uses cron expressions, so translate OnCalendar:
| OnCalendar | Cron |
|---|---|
*-*-* 03:17:00 | 17 3 * * * |
Mon..Fri 09:00 | 0 9 * * 1-5 |
hourly | 0 * * * * |
*:0/15 | */15 * * * * |
monthly | 0 0 1 * * |
Timers follow the system time zone; use the same one in the check. Not sure what an expression means? Paste it into the cron expression explainer.
Test it
systemctl start backup.service # a manual run should send /start and the success ping
journalctl -u backup.service -n 50
Then make backup.sh exit with an error once: the failure hook should send an alert within seconds.