Make backup/replication services mutually exclusive #1

New issue

Open

opened 2026-07-28 11:53:40 +00:00 by mich · 0 comments

mich commented

2026-07-28 11:53:40 +00:00

Owner

rclomatic.service should not run while postgres-backup.service or replicate_d.service (zfs-replicate.sh) are still running.

Why: rclomatic's backup_databases target syncs /pool01/backup/databases, which postgres-backup.sh writes into — and its homes/gv0 targets (zfs: true) sync from whatever snapshot zfs-replicate.sh most recently replicated into pool01/replicas/{homes,gluster}. Right now the three timers are just spaced out by clock time (replicate_d.timer 00:15 UTC, postgres-backup.timer ~01:15-01:20 UTC, rclomatic.timer 03:05 UTC) with no actual dependency between them — if one runs long (e.g. a big pg_dump, a slow/retried zfs send, network issues), rclomatic could start syncing while the other is still mid-write, backing up a partial/inconsistent state.

Needs a real implementation decision, not done yet - options to consider:

systemd unit ordering/exclusion (After=/Requires= chaining the timers into one sequence, or Conflicts= so one won't start while another is active)
a simple lock/flag file each script checks before starting
collapsing all three into one timer that runs them in sequence

`rclomatic.service` should not run while `postgres-backup.service` or `replicate_d.service` (zfs-replicate.sh) are still running. Why: `rclomatic`'s `backup_databases` target syncs `/pool01/backup/databases`, which `postgres-backup.sh` writes into — and its `homes`/`gv0` targets (`zfs: true`) sync from whatever snapshot `zfs-replicate.sh` most recently replicated into `pool01/replicas/{homes,gluster}`. Right now the three timers are just spaced out by clock time (`replicate_d.timer` 00:15 UTC, `postgres-backup.timer` ~01:15-01:20 UTC, `rclomatic.timer` 03:05 UTC) with no actual dependency between them — if one runs long (e.g. a big pg_dump, a slow/retried zfs send, network issues), rclomatic could start syncing while the other is still mid-write, backing up a partial/inconsistent state. Needs a real implementation decision, not done yet - options to consider: - systemd unit ordering/exclusion (`After=`/`Requires=` chaining the timers into one sequence, or `Conflicts=` so one won't start while another is active) - a simple lock/flag file each script checks before starting - collapsing all three into one timer that runs them in sequence