fix: one deploy per host at a time, under a lock the host frees on its own
CI / check (pull_request) Successful in 17s

Several sessions deploy to the same test server, and two runs used to
overlap. On 2026-10-05 two `up -d --force-recreate` runs recreated each
other's containers and the API was down for a minute (#1). On 2026-10-06
two runs renamed each other's *.new compose files and one broke off; with
two different tags, one tag's compose files could have started the
other's images (#3).

- Each deploy first takes flock on ~/netork/.deploy.lock on the host. An
  ssh session holds it: the remote side takes the lock on fd 9, reports
  LOCKED and waits on its stdin, so ending the session frees it, whether
  the deploy finished, failed, was interrupted or lost its connection.
  Checked over real ssh on .50, including a client killed with -9.
- A second deploy prints who holds the lock, since when and with which
  tag, and waits up to DEPLOY_LOCK_WAIT seconds (default 900); then it
  gives up without touching the host.
- A host without flock is deployed without the lock, with a warning.
- deploy_server is now the lock around deploy_steps, the old body.

Closes #1
Closes #3
This commit is contained in:
Christian Manivong
2026-10-07 06:42:13 +02:00
parent 6444aa97f3
commit ec93619f8f
4 changed files with 270 additions and 3 deletions
+7
View File
@@ -41,6 +41,7 @@ elsewhere.
| `NETORK_VERSION` | Tag to deploy (default `latest`) |
| `REGISTRY_USER`, `REGISTRY_PASSWORD` | Registry login used on every server |
| `REGISTRY_USER_<server>`, `REGISTRY_PASSWORD_<server>`, `NETORK_VERSION_<server>` | Per-server overrides. `<server>` has its dots replaced by underscores, e.g. `_10_0_0_2` |
| `DEPLOY_LOCK_WAIT` | Seconds to wait for another deploy to the same host (default 900) |
## Usage
@@ -65,6 +66,12 @@ started earlier pulls whatever image the registry held before, which is stale.
## What a deploy does, per server
0. Takes the host's deploy lock, `flock` on `~/netork/.deploy.lock`, and holds it until
the deploy ends. A second deploy to the same host waits for it and says who holds
it, since when, and which tag they are deploying. It gives up after 15 minutes
(`DEPLOY_LOCK_WAIT`, in seconds) without touching the host. The lock belongs to an
ssh session, so a deploy that fails, is interrupted or loses its connection frees it
on its own. A host without `flock` is deployed without the lock, with a warning.
1. Logs in to the registry. The password travels over ssh's stdin, never on a command
line.
2. Pulls the engine image and copies `docker-compose.yml` and