fix: one deploy per host at a time, under a lock the host frees on its own
CI / check (pull_request) Successful in 17s
CI / check (pull_request) Successful in 17s
Several sessions deploy to the same test server, and two runs used to overlap. On 2026-10-05 two `up -d --force-recreate` runs recreated each other's containers and the API was down for a minute (#1). On 2026-10-06 two runs renamed each other's *.new compose files and one broke off; with two different tags, one tag's compose files could have started the other's images (#3). - Each deploy first takes flock on ~/netork/.deploy.lock on the host. An ssh session holds it: the remote side takes the lock on fd 9, reports LOCKED and waits on its stdin, so ending the session frees it, whether the deploy finished, failed, was interrupted or lost its connection. Checked over real ssh on .50, including a client killed with -9. - A second deploy prints who holds the lock, since when and with which tag, and waits up to DEPLOY_LOCK_WAIT seconds (default 900); then it gives up without touching the host. - A host without flock is deployed without the lock, with a warning. - deploy_server is now the lock around deploy_steps, the old body. Closes #1 Closes #3
This commit is contained in:
@@ -41,6 +41,7 @@ elsewhere.
|
||||
| `NETORK_VERSION` | Tag to deploy (default `latest`) |
|
||||
| `REGISTRY_USER`, `REGISTRY_PASSWORD` | Registry login used on every server |
|
||||
| `REGISTRY_USER_<server>`, `REGISTRY_PASSWORD_<server>`, `NETORK_VERSION_<server>` | Per-server overrides. `<server>` has its dots replaced by underscores, e.g. `_10_0_0_2` |
|
||||
| `DEPLOY_LOCK_WAIT` | Seconds to wait for another deploy to the same host (default 900) |
|
||||
|
||||
## Usage
|
||||
|
||||
@@ -65,6 +66,12 @@ started earlier pulls whatever image the registry held before, which is stale.
|
||||
|
||||
## What a deploy does, per server
|
||||
|
||||
0. Takes the host's deploy lock, `flock` on `~/netork/.deploy.lock`, and holds it until
|
||||
the deploy ends. A second deploy to the same host waits for it and says who holds
|
||||
it, since when, and which tag they are deploying. It gives up after 15 minutes
|
||||
(`DEPLOY_LOCK_WAIT`, in seconds) without touching the host. The lock belongs to an
|
||||
ssh session, so a deploy that fails, is interrupted or loses its connection frees it
|
||||
on its own. A host without `flock` is deployed without the lock, with a warning.
|
||||
1. Logs in to the registry. The password travels over ssh's stdin, never on a command
|
||||
line.
|
||||
2. Pulls the engine image and copies `docker-compose.yml` and
|
||||
|
||||
Reference in New Issue
Block a user