Christian Manivong
de46ab8a0f
CI / check (pull_request) Successful in 16s
netOrk's docker-compose.yml shipped a registry:2 for the satellite image, and this script started it on every deploy as infrastructure. Satellites pull from registry.netork.io; on 172.22.8.50 the registry held one old image, nothing had pulled from it in 30 days, and it accepted anonymous pushes on port 5000 of every instance (NetOrk/netork#763). - registry leaves INFRA_SERVICES. - A new step removes services netOrk no longer ships: the container netork-registry-1, then the volume netork_registry_data. `docker compose up` never removes a container whose service left the compose file, so without this each host would keep it until someone removed it by hand. The step is idempotent and works with the old compose file as well as the new one, so it can go out before netOrk drops the service.
125 lines
5.1 KiB
Markdown
125 lines
5.1 KiB
Markdown
# netOrk deploy
|
|
|
|
Deploys [netOrk](https://git.netork.io/NetOrk/netork) to one or more Docker hosts in
|
|
parallel, from pre-built images in a container registry.
|
|
|
|
The machine running the deploy needs this repository, `ssh`, and a `deploy.env`. It does
|
|
not need a netOrk checkout. Each server pulls the images itself. The compose files come
|
|
out of the engine image for the tag being deployed, so they always match the images they
|
|
start, and database migrations run from that same image.
|
|
|
|
## Requirements
|
|
|
|
**On the machine running the deploy:**
|
|
- bash
|
|
- `ssh` with key-based access to every target
|
|
|
|
**On every target server:**
|
|
- Docker with the compose plugin
|
|
- a directory `~/netork/` with netOrk's `.env`, which holds the database and application
|
|
settings the compose file reads (`env_file: .env`)
|
|
|
|
**Images:** `netork/engine` and `netork/ui` must be published in the registry under the
|
|
tag you deploy. The engine image must carry its compose files under `/app/deploy/`.
|
|
Tags built before that change are refused with a clear message.
|
|
|
|
## Setup
|
|
|
|
```bash
|
|
cp deploy.env.example deploy.env
|
|
$EDITOR deploy.env
|
|
```
|
|
|
|
`deploy.env` is gitignored. Point `DEPLOY_ENV_FILE` at another file to keep it
|
|
elsewhere.
|
|
|
|
| Variable | Meaning |
|
|
|---|---|
|
|
| `DEPLOY_SERVERS` | Space-separated default targets |
|
|
| `UI_SERVER` | Targets that also run `netork-ui` |
|
|
| `REGISTRY_HOST` | Registry to pull from; required |
|
|
| `NETORK_VERSION` | Tag to deploy (default `latest`) |
|
|
| `REGISTRY_USER`, `REGISTRY_PASSWORD` | Registry login used on every server |
|
|
| `REGISTRY_USER_<server>`, `REGISTRY_PASSWORD_<server>`, `NETORK_VERSION_<server>` | Per-server overrides. `<server>` has its dots replaced by underscores, e.g. `_10_0_0_2` |
|
|
| `DEPLOY_LOCK_WAIT` | Seconds to wait for another deploy to the same host (default 900) |
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
./deploy.sh # every server in DEPLOY_SERVERS
|
|
./deploy.sh 10.0.0.1 # one server
|
|
./deploy.sh 10.0.0.1 10.0.0.2 # several, in parallel
|
|
./deploy.sh --version=main-1a2b3c4 10.0.0.1 # one-off tag, this run only
|
|
```
|
|
|
|
Only tags netOrk's CI publishes are accepted:
|
|
- `latest`
|
|
- `latest-dev`
|
|
- `main-<sha>`
|
|
- `feature-<branch>`
|
|
- `X.Y.Z`
|
|
|
|
Anything else is refused before any server is touched.
|
|
|
|
If you deploy a branch build, wait until its CI run has published the images. A deploy
|
|
started earlier pulls whatever image the registry held before, which is stale.
|
|
|
|
## What a deploy does, per server
|
|
|
|
0. Takes the host's deploy lock, `flock` on `~/netork/.deploy.lock`, and holds it until
|
|
the deploy ends. A second deploy to the same host waits for it and says who holds
|
|
it, since when, and which tag they are deploying. It gives up after 15 minutes
|
|
(`DEPLOY_LOCK_WAIT`, in seconds) without touching the host. The lock belongs to an
|
|
ssh session, so a deploy that fails, is interrupted or loses its connection frees it
|
|
on its own. A host without `flock` is deployed without the lock, with a warning.
|
|
1. Logs in to the registry. The password travels over ssh's stdin, never on a command
|
|
line.
|
|
2. Pulls the engine image and copies `docker-compose.yml` and
|
|
`docker-compose.registry.yml` out of it into `~/netork/`.
|
|
3. Pulls the netOrk images, then force-recreates the API, the workers, `netork-beat`,
|
|
`flower` and, on UI servers, `netork-ui`. If the recreate fails, it is tried once
|
|
more after 5 s (`DEPLOY_RECREATE_RETRY_DELAY`). Compose's parallel recreate can lose
|
|
a container it just renamed and leave the rest stopped. If the second attempt fails
|
|
too, the container states are printed and the deploy fails.
|
|
4. Reconciles `apt-cacher-ng` and `signal-api`: each is recreated only if its
|
|
definition changed. It never touches `postgres` or `redis`.
|
|
It then removes services netOrk no longer ships, container and volume, where a host
|
|
still has them: today the bundled `registry` (NetOrk/netork#763).
|
|
5. Verifies that the containers run exactly the image that was pulled. If they don't, the
|
|
deploy fails.
|
|
6. Records `REGISTRY_HOST` and `NETORK_VERSION` in `~/netork/.env`, so that a
|
|
hand-typed `docker compose` on the host uses the same images.
|
|
7. Runs `alembic upgrade head` inside `netork-api`.
|
|
8. Prunes unused images.
|
|
|
|
A failing step stops that server's deploy with a non-zero exit, and the other servers
|
|
carry on. The script exits non-zero if any server failed and names those servers.
|
|
|
|
## Operating notes
|
|
|
|
- **Never delete these Docker volumes:**
|
|
- `celerybeat_schedule` holds the scheduler state.
|
|
- `signal_cli_data` holds netOrk's Signal device link. Losing it means pairing again
|
|
by QR code.
|
|
- `netork-beat` always rolls out together with the workers. The script does this for
|
|
you; keep it that way if you deploy by hand.
|
|
- Always deploy with this script. Do not point a local Docker client at a remote host
|
|
(`DOCKER_HOST=ssh://…`): it resolves volume paths locally and breaks the remote
|
|
containers.
|
|
|
|
## Development
|
|
|
|
```bash
|
|
pip install pytest ruff shellcheck-py
|
|
shellcheck deploy.sh
|
|
ruff format --check . && ruff check .
|
|
pytest
|
|
```
|
|
|
|
The tests are static and behavioural checks of `deploy.sh`. None of them reach a real
|
|
host.
|
|
|
|
## License
|
|
|
|
MIT, see [LICENSE](LICENSE).
|