Backup and restore

A recoverable Tunnex backup has two independent artifacts:

1. a PostgreSQL custom-format dump plus its manifest; and
2. the exact master key that sealed sensitive database values.

Store the master key somewhere different from the database backup. The manifest contains only a keyed fingerprint of the key, not the key itself.

There is no backup/restore page in the current console. Run this procedure on the deployment host or through the supported Kubernetes/database tooling.

Gateway private keys are generated and retained on gateways. Device private keys are delivered once and are not recoverable from the control plane.

Run this on the public-installer host from the tunnex/ directory. Use a root-only directory and an operator-selected label instead of putting secrets in filenames.

cd tunnex
umask 077
mkdir -p backups/manual

sudo docker compose -f tunnex.yml exec -T postgres sh -c \
  'pg_dump --format=custom --no-owner --username "$POSTGRES_USER" "$POSTGRES_DB"' \
  > backups/manual/tunnex.dump

sudo docker compose -f tunnex.yml exec -T postgres \
  pg_restore --list < backups/manual/tunnex.dump >/dev/null
DUMP_SHA256="$(sha256sum backups/manual/tunnex.dump | awk '{print $1}')"

sudo docker compose -f tunnex.yml exec -T \
  -e TUNNEX_BACKUP_DUMP_SHA256="$DUMP_SHA256" \
  api backupctl manifest "manual" \
  > backups/manual/tunnex.manifest.json

sudo docker compose -f tunnex.yml exec -T api \
  backupctl verify --dump-sha256 "$DUMP_SHA256" \
  < backups/manual/tunnex.manifest.json

Copy the master key once, then custody it separately:

(
  umask 077
  sudo docker compose -f tunnex.yml exec -T api \
    sh -c 'cat /var/lib/tunnex/secrets/master.key' > ./tunnex-master.key
)

Move tunnex-master.key to an offline password vault, HSM-backed secret store, or other controlled location not used for the dump. Do not leave the working copy beside backups/.

The public upgrade.sh --apply performs the dump, archive validation, digest-bound manifest creation, manifest verification, and preflight automatically under tunnex/backups/. Its backup still needs the separately custodied master key.

Dump the external PostgreSQL service using its provider-supported method. Bind the manifest to that dump’s SHA-256 and verify it against the mounted key:

DUMP_SHA256="$(sha256sum tunnex.dump | awk '{print $1}')"
kubectl -n tunnex exec deploy/tunnex-api -- \
  env TUNNEX_BACKUP_DUMP_SHA256="$DUMP_SHA256" \
  backupctl manifest "scheduled" > tunnex.manifest.json
kubectl -n tunnex exec -i deploy/tunnex-api -- \
  backupctl verify --dump-sha256 "$DUMP_SHA256" \
  < tunnex.manifest.json

The Helm chart requires you to create the master-key Secret; it never generates one. Back up the selected key value through your secret-management process. Do not paste it into shell history or store it in Helm values.

Practice this procedure on an isolated recovery host. The commands below erase and recreate the target database, so confirm the target and enter a maintenance window first. Restoring over a populated or newer schema is not supported: pg_restore --clean removes only objects represented in the dump and can leave newer target objects behind.

1. Put the matching tunnex-master.key, dump, and manifest on the recovery host.
2. Stop writers while leaving PostgreSQL available.
3. Install the master key in the tunnex_secrets volume.
4. Verify the manifest and dump digest before changing the database.
5. Drop and recreate the target database.
6. Restore PostgreSQL, then start and verify the stack.

cd tunnex
sudo docker compose -f tunnex.yml stop caddy nginx web api node-agent

sudo docker compose -f tunnex.yml run --rm --no-deps -T \
  --user 0:0 --entrypoint sh api -c \
  'umask 077; cat > /var/lib/tunnex/secrets/master.key; chown 10001:10001 /var/lib/tunnex/secrets/master.key; chmod 0600 /var/lib/tunnex/secrets/master.key' \
  < tunnex-master.key

DUMP_SHA256="$(sha256sum tunnex.dump | awk '{print $1}')"
sudo docker compose -f tunnex.yml run --rm --no-deps -T \
  --entrypoint backupctl api verify --dump-sha256 "$DUMP_SHA256" \
  < tunnex.manifest.json

sudo docker compose -f tunnex.yml exec -T postgres sh -c \
  'dropdb --if-exists --force --username "$POSTGRES_USER" "$POSTGRES_DB" &&
   createdb --username "$POSTGRES_USER" "$POSTGRES_DB"'

sudo docker compose -f tunnex.yml exec -T postgres \
  sh -c 'pg_restore --exit-on-error --no-owner --username "$POSTGRES_USER" --dbname "$POSTGRES_DB"' \
  < tunnex.dump

sudo docker compose -f tunnex.yml up -d
sudo docker compose -f tunnex.yml ps
sudo docker compose -f tunnex.yml exec -T api \
  wget -qO- http://127.0.0.1:8080/healthz
sudo docker compose -f tunnex.yml exec -T api \
  wget -qO- http://127.0.0.1:9090/readyz

A key mismatch exits nonzero—currently exit 2—and must stop the procedure. Restore with the same Tunnex release that created the dump. After the restored deployment passes the recovery checks, follow the supported forward upgrade path to reach a newer release. Never restore directly into an older or already-migrated newer database. The manifest records the schema version for operator evidence, but backupctl verify does not enforce migration compatibility.

After restore, verify:

• the expected organizations, policies, devices, and audit history exist;
• an existing gateway reconnects with the same identity, without enrollment;
• gateway desired/applied revisions converge;
• an existing tunnel exchanges traffic; and
• sign-in, SSO, and certificate issuance work as configured.

A successful page load alone does not prove that sealed data can be decrypted or that the fleet still trusts the restored agent CA.