⬆️ Lightflow Upgrade

Lightflow is a generic task runner: the upgrade logic lives entirely in the script it runs and the host-side gate it reaches over SSH. This tutorial configures an installed Lightflow instance to drive the service upgrades of the Podman services (calibre, immich, kouizine, n8n, planka, vaultwarden, …) through podmanctl --upgrade — one click in the UI instead of an SSH session, with the same security model as the Lightflow Backup pipeline: Lightflow never gets a shell on the host and can never aim podmanctl at anything else, the boundary lives in the upgrade-gate forced command on the host.

📋 Requirements

Info

Lightflow Upgrade requires the installation of:

• Lightflow
• podmanctl (performs the actual upgrade on the host)

---

🛤️ End-to-end path

How a run travels from the UI to the restarted stack — and who enforces what at each hop:

#	Hop	Enforced by
1	the task step runs python3 upgrade.py --upgrade <svc> in the Lightflow container	Lightflow (pool, step timeout, live logs)
2	ssh upgrade_ops@host.containers.internal "podmanctl --upgrade <svc> --timeout <n>"	key from the UPGRADE_SSH_KEY Variable; restrict = no shell, no forwarding
3	sshd ignores the requested command and runs the forced command sudo upgrade-gate	authorized_keys + sudoers (exactly one invocation, no wildcard)
4	the gate re-reads the request from SSH_ORIGINAL_COMMAND and accepts only podmanctl --upgrade <known-service> [--timeout N]	upgrade-gate (root)
5	timeout -k 10 N podmanctl --upgrade <svc>: stop → git pull → pull base + registry images → full rebuild → restart → prune	podmanctl (root); coreutils timeout = host-side deadline

The step's box is 🟩 when the whole chain exits 0 and 🟥 otherwise; podmanctl's live output (pulls, build, restart, final dashboard) streams into the step log while it runs.

---

🗂️ The one-step upgrade model

Where the backup pipeline needs four ordered steps (stop / sync / archive / start), an upgrade needs exactly one: podmanctl --upgrade already drives the full lifecycle on the host, so a single step — and a single box in the grid — carries the whole operation:

Step	Action	Box
upgrade	podmanctl --upgrade <svc> (stop → git pull → pull bases + registry images → rebuild → start → prune)	🟩 / 🟥

A failed upgrade leaves the service STOPPED — on purpose

podmanctl aborts on a failed git pull, and a failed build never starts broken code: the service is deliberately not restarted. Read the step log, fix the cause, then re-run the task or podmanctl --start <svc> on the host.

--upgrade never edits versions

Pinned tags (*_VERSION in the service env file, or the Image= tag in a Quadlet) must be bumped and pushed first — the upgrade then pulls the new tags and rebuilds on them. See each service runbook's Upgrading section.

---

🔐 Upgrade Gate

The host-side plumbing that the task calls through SSH:

• podmanctl (performs the upgrade — already installed)
• upgrade-gate (the validating forced command)
• upgrade_ops SSH account.

🚪 Install upgrade-gate

Install upgrade-gate to /usr/local/bin

# the forced command run for every upgrade_ops SSH connection: it re-reads
# the client's request from SSH_ORIGINAL_COMMAND and lets EXACTLY one shape
# through — `podmanctl --upgrade <service>` for a known service, with an
# optional `--timeout N` enforced host-side via coreutils timeout
sudo tee /usr/local/bin/upgrade-gate >/dev/null <<'EOF'
#!/bin/sh
# upgrade-gate — SSH dispatch for the upgrade_ops forced command.
# Runs as root via sudo (sudoers keeps SSH_ORIGINAL_COMMAND), so podmanctl's
# internal per-user sudo calls never prompt.
set -eu
set -f  # no glob expansion while word-splitting the client's command line

refuse() { echo "upgrade-gate: $1" >&2; exit 1; }

set -- ${SSH_ORIGINAL_COMMAND:-}
{ [ $# -eq 3 ] || [ $# -eq 5 ]; } && [ "$1" = "podmanctl" ] && [ "$2" = "--upgrade" ] \
  || refuse "refused: '${SSH_ORIGINAL_COMMAND:-}' (only 'podmanctl --upgrade <service> [--timeout N]' crosses)"
svc=$3
case "$svc" in *[!a-z0-9-]*|'') refuse "bad service name '$svc'" ;; esac
[ -d "/media/ssd/podman-users/$svc" ] || refuse "unknown service '$svc'"

# optional host-side deadline: upgrade.py derives it from the Lightflow step
# timeout; coreutils timeout TERMs podmanctl's process group at the deadline
# and KILLs 10s later — so a wedged upgrade dies on the host even if the SSH
# session never tears down
limit=""
if [ $# -eq 5 ]; then
  [ "$4" = "--timeout" ] || refuse "unexpected flag '$4' (only --timeout N)"
  case "$5" in ''|*[!0-9]*) refuse "bad timeout '$5' (expected integer seconds)" ;; esac
  limit=$5
fi

rc=0
if [ -n "$limit" ]; then
  timeout -k 10 "$limit" /usr/local/bin/podmanctl --upgrade "$svc" || rc=$?
else
  /usr/local/bin/podmanctl --upgrade "$svc" || rc=$?
fi

# safety net only: podmanctl re-owns the tree itself, between the pull and
# the builds — ownership MUST NOT change after a build, since Podman keys a
# COPY layer on the copied files' tar headers (uid/gid included), so a
# post-build chown makes identical sources hash differently next time and
# the next upgrade recompiles for nothing. This only ever fires for an
# upgrade that failed before podmanctl reached that step; data/ is skipped
# on purpose: those trees belong to the containers' mapped subuids.
SERVICE_DIR="/media/ssd/podman/$svc"
if [ -d "$SERVICE_DIR" ]; then
  find "$SERVICE_DIR" -path "$SERVICE_DIR/data" -prune \
    -o -user root -exec chown "${svc}_svc:${svc}_admins" {} +
fi
exit $rc
EOF

# executable for everyone, writable by root only
sudo chmod 0755 /usr/local/bin/upgrade-gate

Same validating-gate idea as backupctl --ssh-gate, shrunk to a single command shape. The trailing find/chown is a safety net, not the re-own that matters: podmanctl --upgrade already normalizes ownership between the pull and the builds, and that order is the whole point — re-owning files after a build changes their tar headers, which Podman's cache keys COPY layers on, so the next upgrade recompiles untouched code. The net only catches an upgrade that failed before podmanctl got that far.

👤 Create the upgrade_ops SSH account

Create the upgrade_ops user and sudoers rule

# dedicated account used ONLY by the Lightflow container to reach the gate.
# NOT added to the no-ssh group on purpose: SSH is its entire reason to exist,
# and the forced command below is the only thing it can ever execute.
sudo useradd --system --create-home --home-dir /home/upgrade_ops --shell /bin/bash upgrade_ops

# sudoers drop-in: keep the client's original command line visible to the
# gate (sshd exports it as SSH_ORIGINAL_COMMAND) and allow exactly ONE
# invocation as root — no wildcard
sudo tee /etc/sudoers.d/upgrade-gate >/dev/null <<'EOF'
Defaults!/usr/local/bin/upgrade-gate env_keep += "SSH_ORIGINAL_COMMAND"
upgrade_ops ALL=(root) NOPASSWD: /usr/local/bin/upgrade-gate
EOF
sudo chmod 0440 /etc/sudoers.d/upgrade-gate

# validate the sudoers syntax before it can lock you out
sudo visudo -cf /etc/sudoers.d/upgrade-gate

# smoke-test the gate from the admin account (env_keep applies here too):
# anything but `podmanctl --upgrade <service>` must be refused
SSH_ORIGINAL_COMMAND="podmanctl --stop vaultwarden" sudo upgrade-gate

🧰 Trust the service repositories for root

Let the root-run git pull work non-interactively

Through the gate, podmanctl's git pull runs as root with no TTY: the origin must be pullable without any prompt, and the repo must be trusted for root. Both fail closed — a refused pull aborts the upgrade and leaves the service stopped.

1. Audit the origins2. Rewire + trust for root3. Verify in the gate's context

# list every service clone's origin: a /media/ssd/git/... local path
# is fine, a https://git.fum-server.fr/... origin needs credentials
# and dies in the gate's no-TTY context with
#   "could not read Username for 'https://…': No such device or address"
for d in /media/ssd/podman/*/.git; do
  r=${d%/.git}
  printf '%-14s %s\n' "${r##*/}" "$(git -C "$r" remote get-url origin 2>/dev/null)"
done

# point each https clone at the co-located bare repo — nginx serves
# /media/ssd/git as git.fum-server.fr, so it is the SAME repository —
# and trust BOTH paths for root, or the pull dies with "detected
# dubious ownership"
for d in /media/ssd/podman/*/.git; do
  svc=$(basename "${d%/.git}")
  bare=/media/ssd/git/podman/$svc.git
  case "$(git -C "${d%/.git}" remote get-url origin 2>/dev/null)" in
    https://*)
      [ -d "$bare" ] || { echo "$svc: no $bare — skipped" >&2; continue; }
      git -C "${d%/.git}" remote set-url origin "$bare"
      echo "$svc: origin -> $bare" ;;
  esac
  sudo git config --system --add safe.directory "${d%/.git}"
  [ -d "$bare" ] && sudo git config --system --add safe.directory "$bare"
done

# setsid -w = no controlling TTY (the gate's context) while keeping the
# output and exit code — a plain `sudo git pull` would PROMPT for a
# username instead of reproducing the failure
sudo setsid -w git -C /media/ssd/podman/gokapi pull --rebase </dev/null
echo "rc=$?"

The push-deploy services (cardbox, claudenames, kouizine, lightflow, mediascope, pooping, tripwise) were already rewired by their runbooks' push-deploy step — the loop leaves them untouched. Service directories without a .git are simply skipped by podmanctl (➖).

🔑 Authorize Lightflow's SSH key

Lightflow doesn't keep a private key file on disk: the key material lives in the UPGRADE_SSH_KEY Variable (secret) and upgrade.py writes it to a private, short-lived temp file only while ssh runs. The same applies to the host-key line in UPGRADE_KNOWN_HOSTS — exactly the backup pipeline's model, with its own dedicated key pair and account.

Generate a key pair, authorize it, and capture the values to paste

# generate a throwaway key pair in a temp dir, the PRIVATE key will live only
# in the UPGRADE_SSH_KEY Variable, never as a file on the host or a mount
TMP="$(mktemp -d)"
ssh-keygen -t ed25519 -N "" -C "lightflow-upgrade" -f "${TMP}/id_ed25519"

# authorize the PUBLIC key for upgrade_ops with the SAME forced command
PUBKEY="$(cat ${TMP}/id_ed25519.pub)"
sudo mkdir -p /home/upgrade_ops/.ssh
sudo tee -a /home/upgrade_ops/.ssh/authorized_keys >/dev/null <<EOF
restrict,command="sudo /usr/local/bin/upgrade-gate" ${PUBKEY}
EOF
sudo chmod 0700 /home/upgrade_ops/.ssh
sudo chmod 0600 /home/upgrade_ops/.ssh/authorized_keys
sudo chown -R upgrade_ops:upgrade_ops /home/upgrade_ops/.ssh

# ---- value for UPGRADE_KNOWN_HOSTS ------------------------------------------
# the container reaches the host as host.containers.internal, so the known_hosts
# line must carry THAT name in front of the host's own ed25519 public host key
printf 'host.containers.internal %s\n' \
  "$(sudo awk '{print $1, $2}' /etc/ssh/ssh_host_ed25519_key.pub)"

# ---- value for UPGRADE_SSH_KEY ----------------------------------------------
# print the PRIVATE key to copy, then wipe the temp dir so nothing lingers
cat "${TMP}/id_ed25519"
rm -rf "${TMP}"

Paste the private key into the UPGRADE_SSH_KEY Variable (secret) and the host.containers.internal … line into UPGRADE_KNOWN_HOSTS. No private key file is kept on the host and no .ssh volume is mounted. The gate accepts only podmanctl --upgrade <service> [--timeout N] — nothing else crosses, in particular no stop/start and no shell.

---

⚙️ Configure Lightflow (in the UI)

1. Create the upgrade pool

Pools → Add pool

• Name: upgrade
• Slots: 1

A 1-slot pool serializes the upgrades: a full rebuild already saturates the ODROID's CPU, two at once would just thrash — and stretch both stacks' downtime.

2. Create the upgrade variables

Variables → Add variable (these become environment variables for the script):

Key	Value	Secret
UPGRADE_HOST	upgrade_ops@host.containers.internal	no
UPGRADE_SSH_KEY	(paste the SSH private key contents — see above)	yes
UPGRADE_KNOWN_HOSTS	(paste the known_hosts host-key line — see above)	no

There is no timeout Variable: the step timeout on each task is the single knob — Lightflow injects it as LIGHTFLOW_STEP_TIMEOUT and upgrade.py derives the host-side --timeout passed to the gate from it.

3. Create the upgrade script

upgrade.py

#!/usr/bin/env python3
"""
upgrade.py — sample Lightflow task script (podmanctl upgrade of a Podman service).

Lightflow itself is a generic runner: this script holds the upgrade logic and
reaches the host over the restricted `upgrade_ops` SSH forced command, which
runs `sudo upgrade-gate` on the host. The gate lets exactly ONE command shape
through — `podmanctl --upgrade <service> [--timeout N]` — so Lightflow never
gets a shell and can never aim podmanctl at anything else.

One Lightflow TASK per service, made of a single STEP — podmanctl drives the
whole lifecycle on the host (stop, git pull, pull base + registry images,
full rebuild, restart, prune), so one box in the grid carries the upgrade:

    Step 1  upgrade   python3 /scripts/upgrade.py --upgrade <service>   (on_success)

Configuration comes from Lightflow VARIABLES (injected as environment variables):

    UPGRADE_HOST         upgrade_ops@host.containers.internal   (required)
    UPGRADE_SSH_KEY      SSH private key — inline PEM value OR a file path
    UPGRADE_KNOWN_HOSTS  known_hosts entry — inline value OR a file path

The timeout knob is the per-service STEP timeout set in the UI — rebuild
times vary wildly between a registry-image stack and a source-built one,
so pick it per service and be generous. Lightflow enforces it in the
container (SIGTERM the step's process group, SIGKILL 10s later) and
injects it as LIGHTFLOW_STEP_TIMEOUT; this script derives a slightly
smaller host-side deadline from it and passes it to the gate, which runs
podmanctl under coreutils `timeout` — so a wedged upgrade is aborted ON
THE HOST even if the SSH session never tears down. A failed or timed-out
upgrade can leave the service stopped — on purpose (a half-upgraded
stack must not blind-restart).

Holding the SSH key (and known_hosts) inline in a secret Variable keeps the
script generic — any future device just needs its own key Variable, with no
host-side key files to provision. The key material is written to a private,
short-lived temp file only while `ssh` runs, then removed.
"""

import os
import re
import subprocess
import sys
import tempfile
import textwrap

ACTIONS = {"--upgrade"}


def env(key, default=None):
    return os.environ.get(key, default)


def materialize(value, kind):
    """Resolve a key / known_hosts VALUE to a file path that `ssh` can use.

    A Variable may hold either a filesystem PATH (legacy) or the INLINE value
    itself (preferred — generic, no host-side key files). Inline content is
    written to a private (0600) temp file; the caller removes it afterwards.
    A private key is recognised by its PEM header; a known_hosts entry by the
    whitespace separating its fields. A key whose line breaks were flattened
    to spaces by a single-line input field is re-wrapped into a valid PEM.
    Returns (path, is_temp).
    """
    # normalize line endings: a value pasted or stored with Windows CRLF
    # (or a stray \r) keeps \r inside the body, so force LF.
    text = value.replace("\r\n", "\n").replace("\r", "\n").strip()
    inline = "-----BEGIN" in text if kind == "key" else any(c.isspace() for c in text)
    if not inline:
        return value, False
    if kind == "key":
        # Some Variable fields flatten a multi-line PEM onto ONE line, turning
        # the newlines into spaces — ssh then fails with "error in libcrypto".
        # Rebuild a canonical PEM: keep the BEGIN/END markers, strip ALL
        # whitespace from the base64 body, then re-wrap at 70 columns. This is
        # a no-op on an already-correct key, so it is always safe to run.
        m = re.search(r"-----BEGIN ([A-Z0-9 ]+)-----(.*)-----END \1-----", text, re.S)
        if m:
            label = m.group(1)
            body = re.sub(r"\s+", "", m.group(2))
            text = "-----BEGIN {0}-----\n{1}\n-----END {0}-----".format(
                label, "\n".join(textwrap.wrap(body, 70))
            )
    fd, path = tempfile.mkstemp(prefix="lightflow-", suffix=f".{kind}")
    with os.fdopen(fd, "w") as f:
        f.write(text + "\n")
    os.chmod(path, 0o600)
    return path, True


def host_timeout():
    """Host-side deadline derived from the ONE timeout knob: the step
    timeout Lightflow injects as LIGHTFLOW_STEP_TIMEOUT. Slightly below
    it, so the gate's coreutils `timeout` aborts podmanctl cleanly before
    Lightflow tears down the process group; the 30s margin is capped at a
    quarter of the step timeout so short steps keep most of their budget."""
    step = int(env("LIGHTFLOW_STEP_TIMEOUT", "7200"))
    return max(10, step - min(30, step // 4))


def ssh_upgrade(args):
    """Run `podmanctl <args> --timeout <derived>` on the host via the SSH forced command."""
    host = env("UPGRADE_HOST")
    if not host:
        sys.exit("error: UPGRADE_HOST is not set (e.g. upgrade_ops@host.containers.internal)")
    key, key_tmp = materialize(env("UPGRADE_SSH_KEY", "/data/.ssh/id_ed25519"), "key")
    known, known_tmp = materialize(env("UPGRADE_KNOWN_HOSTS", "/data/.ssh/known_hosts"), "known_hosts")

    remote = "podmanctl " + " ".join(args) + f" --timeout {host_timeout()}"
    cmd = [
        "ssh",
        "-i", key,
        "-o", "BatchMode=yes",
        "-o", "StrictHostKeyChecking=accept-new",
        "-o", f"UserKnownHostsFile={known}",
        host,
        remote,
    ]
    print("+ " + " ".join(cmd), flush=True)
    try:
        return subprocess.run(cmd).returncode
    finally:
        for path, is_tmp in ((key, key_tmp), (known, known_tmp)):
            if is_tmp:
                try:
                    os.unlink(path)
                except OSError:
                    pass


def main():
    if len(sys.argv) < 3 or sys.argv[1] not in ACTIONS:
        sys.exit("usage: upgrade.py --upgrade <service>")
    action, service = sys.argv[1], sys.argv[2]
    if not service.strip():
        sys.exit("error: service name is empty")

    if action == "--upgrade":
        rc = ssh_upgrade(["--upgrade", service])
    else:
        sys.exit(f"error: unknown action {action!r} (--upgrade)")

    sys.exit(rc)


if __name__ == "__main__":
    main()

4. Create one task per service

Tasks → New task — example for vaultwarden:

• Name: upgrade_vaultwarden
• CRON: none needed — leave Auto off and trigger with ▶ Run now (an upgrade is a deliberate act; add a CRON only if you really want unattended upgrades)
• Pool: upgrade
• Steps (a single step running python3 upgrade.py --upgrade vaultwarden):

#	Name	Command	Arguments	Run rule	Timeout
1	upgrade	python3	--upgrade vaultwarden	on_success	600

Set the timeout per service — it must cover the whole stop → pull → rebuild → restart cycle: a registry-image stack (vaultwarden, planka, n8n) finishes in a few minutes, a source-built stack (kouizine, tripwise, mediascope, claudenames, cardbox, pooping) does a full rebuild after the base-image pull and can need 30–60 min on the ODROID. Duplicate the task for each service, changing the name, the service argument and the timeout.

Never upgrade lightflow through itself

podmanctl --upgrade lightflow stops the Lightflow stack — the very container running the task. The dying SSH client tears down the session and the host kills the half-done upgrade, leaving Lightflow stopped with nobody left to restart it. Upgrade Lightflow from the host instead — see the Lightflow runbook.

---

⬆️ Operations

Run and monitor the upgrades

• on demand: open a task → ▶ Run now (or ▶ Run on the Tasks page) — the normal flow after bumping a version in the service repo
• grid: 🟩 upgraded and restarted — 🟥 failed or timed out (the service may be left stopped: read the log, then re-run or podmanctl --start <svc>)
• logs: open a task → pick a run column → pick the upgrade step — the whole podmanctl output (pulls, build, restart, dashboard) streams live
• check: podmanctl --list --check on the host confirms every stack is up and the registry images are current

First run — smoke test

Trigger the smallest service first (upgrade_vaultwarden) and check:

1. the box goes green and the log shows the stop → pull → rebuild → start sequence
2. the service is back up: podmanctl --list (and the app answers in the browser)
3. the gate refuses anything else: SSH_ORIGINAL_COMMAND="podmanctl --stop vaultwarden" sudo upgrade-gate → refused

A timed-out upgrade is aborted on the host too

The gate runs podmanctl under coreutils timeout with a deadline derived from the step timeout (slightly smaller), so a wedged upgrade dies on the host even if the SSH session never tears down cleanly. Either way the interrupted upgrade — mid-pull or mid-build — leaves the service stopped. Be generous with the step timeouts and treat a red box as "go check the host", not "it probably finished anyway".