🪶 Installing Lightflow

Lightflow is a small, event-driven task scheduler with a web UI, a lightweight replacement for Apache Airflow on this server. It runs your scripts on a CRON schedule (or on demand), shows their history as a green / grey / red grid, and costs almost nothing when idle: no per-second polling, no always-on Python daemons.

This page covers installing and running the application as a self-contained web service. Lightflow is generic: what each task does lives entirely in the scripts it runs, so the application itself needs nothing beyond NGINX and Podman. For a worked end-to-end example — driving this server's nightly OneDrive backups — see the Lightflow Backup tutorial.

Features:

• 🦀 a single Rust binary (axum + tokio) serving its own React UI — a few MB of RAM, zero idle polling: the scheduler sleeps until the next CRON fire time and wakes exactly then
• 🗓️ per-task CRON schedule (1-minute granularity) configured entirely in the UI; manual triggers and edits reach the scheduler over an in-process channel and take effect instantly — no restart, no poll cycle, no gap between back-to-back runs
• 🧩 each task is an ordered list of steps; each runs as a subprocess and is one box in the 🟩🟥⬜ grid (exit 0 → success, 75 → skipped, anything else → failed), with a per-step timeout and an on_success / always run rule
• ⏱️ timeouts are enforced by the backend and kill the whole process group; pools cap concurrency and a task never overlaps itself
• 🐍 author scripts directly in an admin-only editor and install extra Python packages from the UI (into a venv on PYTHONPATH) — no image rebuild to change what runs
• 🔒 generic by design: what a task does lives entirely in the scripts it runs — the app ships no task-specific logic and holds no third-party credentials; a script reaches whatever it needs (a host gate, an API, a remote) entirely on its own
• 🔑 variables are injected as environment variables into every step; any variable can be marked secret and is masked in the UI
• 🗃️ SQLite metadata (one file, no second container); per-step logs are plain files, live-streamed to the browser over SSE
• 👤 user management with admin / viewer roles, argon2 passwords, email-validated accounts, optional TOTP two-factor auth, and opt-in failure alerts

Info

Lightflow is open-source and lives in this repository under src/ (Rust backend + React/TypeScript frontend), with the Podman packaging in build/, quadlet/, nginx/ and scripts/.

---

🧠 Architecture

┌───────────────────── lightflow (one container) ─────────────────────────┐
│  Rust binary (axum + tokio)                                             │
│   ├─ HTTP API + SSE           (REST + live run/log updates, no polling) │
│   ├─ static SPA               (React build served by the binary)        │
│   ├─ event-driven SCHEDULER   (sleeps until the next CRON; no polling)  │
│   └─ EXECUTOR                 (runs /scripts/* as subprocesses)         │
│  SQLite /data/lightflow.db    logs /data/logs/<task>/<run>/<step>.log   │
│  python3 + openssh-client     (to RUN the scripts, which SSH out)       │
└─────────────────────────────────────────────────────────────────────────┘

NGINX reverse-proxies lightflow.domain.fr → 127.0.0.1:HOST_PORT. Inside the container the backend serves both the API (/api/...) and the built UI; deep client routes return the SPA shell, so NGINX's 404 page is never triggered.

---

🔄 How it works

                  in-process channel (instant reload)
   Web UI  ──►  axum API  ──►  Scheduler  ──►  Executor  ──►  your scripts
     ▲             │          (sleeps to        (subprocess
     └──── SSE ────┘           next fire)         per step)

The scheduler keeps every enabled task's next-fire time in memory and blocks until the nearest one. Triggers and edits arrive over an in-process channel and wake it immediately. The executor runs a task's steps in order as subprocesses, applying pools, timeouts and run rules, and writes each step's stdout/stderr to a log file on disk — the database holds only small run/step metadata (status, timestamps, exit codes, a log path), so it stays tiny and logs stream cheaply.

---

📥 Installation

This section describes how to install and run Podman services using systemd Quadlet, enabling containers to restart automatically. It allows the service to run in its own isolated rootless environment with a dedicated Linux user.

📋 Requirements

Info

Lightflow requires the installation of

• NGINX
• Podman

---

⚙️ Configuration

📋 Service Setup

Info

Please follow the following installation steps:

• 👥 User and Group
• 🔁 Persistent Services
• 🔍 Detecting a container's runtime UID/GID

Define Service Variables

# define the service name
SERVICE_NAME=lightflow

Initialize Service Environment

# built-in bash safety and SERVICE_NAME validation
set -euo pipefail; [ -n "${SERVICE_NAME:-}" ] || { echo "SERVICE_NAME is empty"; exit 1; }

# automatically generate variables
SERVICE_HOME="/media/ssd/podman-users/${SERVICE_NAME}"
SERVICE_DIR="/media/ssd/podman/${SERVICE_NAME}"
SERVICE_USER="${SERVICE_NAME}_svc"
SERVICE_ADMIN="${SERVICE_NAME}_admins"

🔐 Service Isolation

Configure Rootless Podman UID/GID Namespace Mappings

# allocate rootless Podman UID/GID mappings if missing
SUBID_SIZE=65536
allocate_subids_if_missing() {
  local user="$1"
  local start
  if ! grep -q "^${user}:" /etc/subuid; then
    start="$(awk -F: 'BEGIN { max = 100000 } { end = $2 + $3; if (end > max) max = end } END { print max }' /etc/subuid)"
    sudo usermod --add-subuids "${start}-$((start + SUBID_SIZE - 1))" "$user"
  fi
  if ! grep -q "^${user}:" /etc/subgid; then
    start="$(awk -F: 'BEGIN { max = 100000 } { end = $2 + $3; if (end > max) max = end } END { print max }' /etc/subgid)"
    sudo usermod --add-subgids "${start}-$((start + SUBID_SIZE - 1))" "$user"
  fi
}
allocate_subids_if_missing "${SERVICE_USER}"

# verify rootless Podman UID/GID namespace mappings exist
grep -q "^${SERVICE_USER}:" /etc/subuid || { echo "ERROR: missing subuid mappings for ${SERVICE_USER}"; exit 1; }
grep -q "^${SERVICE_USER}:" /etc/subgid || { echo "ERROR: missing subgid mappings for ${SERVICE_USER}"; exit 1; }

# validate that all subordinate ID ranges in a file are non-overlapping
validate_subid_ranges() {
  local file="$1"
  awk -F: '
    {
      start = $2
      end   = $2 + $3 - 1
      for (i = 1; i <= count; i++) {
        if (start <= ends[i] && end >= starts[i]) {
          printf "ERROR: overlapping ranges in %s\n", FILENAME
          printf "  %s:%s:%s overlaps %s:%s:%s\n",
            $1, $2, $3,
            names[i], starts[i], sizes[i]
          exit 1
        }
      }
      count++
      names[count]  = $1
      starts[count] = start
      ends[count]   = end
      sizes[count]  = $3
    }
  ' "$file"
}
validate_subid_ranges /etc/subuid
validate_subid_ranges /etc/subgid

# configure the rootless storage driver BEFORE Podman first initializes its storage.
# fuse-overlayfs is a reliable rootless overlay backend and avoids kernel-version
# quirks of native rootless overlay on ARM/Armbian. It is optional with the default
# userns used here, but recommended; set it before the first storage init so no
# 'podman system reset' is needed later.
sudo -u "${SERVICE_USER}" mkdir -p "${SERVICE_HOME}/.config/containers"
sudo -u "${SERVICE_USER}" tee "${SERVICE_HOME}/.config/containers/storage.conf" >/dev/null <<'EOF'
[storage]
driver = "overlay"

[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"
EOF

# rebuild rootless Podman state to ensure existing containers/storage use current namespace mappings
# use a login shell (-i) so HOME points at the service user's home
# (storage now initializes with fuse-overlayfs; no 'podman system reset' needed)
sudo -iu "${SERVICE_USER}" podman system migrate

Configure Secure Service Directory Permissions

# Apply the admin/service ACL policy to one directory tree, deterministically.
#   $1 = tree to treat
#   $2 = (optional) a sub-path to skip entirely (e.g. a container data subtree)
#
# Why setfacl, not chmod: on an ACL'd tree, chmod edits only the mask, never group::,
# so stale execute bits survive. setfacl sets every entry explicitly (mask included),
# lowercase — the result is reproducible and no execute bit can resurface.
apply_acl_tree() {
    local tree="$1"
    local skip="${2:-}"
    local prune=()
    [ -n "$skip" ] && prune=( -path "$skip" -prune -o )                                         # skip this subtree if a 2nd arg is given

    sudo find "$tree" "${prune[@]}" -exec chown "${SERVICE_USER}:${SERVICE_ADMIN}" {} +         # service user owns; admin group is the owning group

    # directories — access ACL (perms on the existing dirs)
    sudo find "$tree" "${prune[@]}" -type d -exec setfacl    -m g:${SERVICE_ADMIN}:rwx {} +     # admin group: access existing dirs
    sudo find "$tree" "${prune[@]}" -type d -exec setfacl    -m u:${SERVICE_USER}:rwx {} +      # service user: access existing dirs
    sudo find "$tree" "${prune[@]}" -type d -exec setfacl    -m u::rwx,g::rwx,o::-,m::rwx {} +  # owner/group rwx, deny others, pin mask rwx

    # directories — default ACL (inherited by NEW files/dirs created later)
    sudo find "$tree" "${prune[@]}" -type d -exec setfacl -d -m g:${SERVICE_ADMIN}:rwx {} +     # admin group: inherit access on new items
    sudo find "$tree" "${prune[@]}" -type d -exec setfacl -d -m u:${SERVICE_USER}:rwx {} +      # service user: inherit access on new items
    sudo find "$tree" "${prune[@]}" -type d -exec setfacl -d -m u::rwx,g::rwx,o::- {} +         # default owner/group/other for new items

    # files — access ACL (rw only, never executable)
    sudo find "$tree" "${prune[@]}" -type f -exec setfacl    -m g:${SERVICE_ADMIN}:rw {} +      # admin group: access existing files
    sudo find "$tree" "${prune[@]}" -type f -exec setfacl    -m u:${SERVICE_USER}:rw {} +       # service user: access existing files
    sudo find "$tree" "${prune[@]}" -type f -exec setfacl    -m u::rw,g::rw,o::-,m::rw {} +     # owner/group rw, deny others, pin mask rw

    sudo find "$tree" "${prune[@]}" -type d -exec chmod g+s {} +                                # setgid: new items inherit the group (a MODE bit; ACLs can't set it)
}

# --- SERVICE_HOME: Podman's private runtime home ------------------------------
# Own the home + set setgid, then ACL ONLY the .config subtree (containers.conf,
# storage.conf, and the *.network / *.container quadlets you hand-edit) so admins
# can edit them sudo-less.
#
# Leave ~/.local (storage) and ~/.cache to Podman: ACLs on the overlay store break
# the runtime ("OCI permission denied" on the `merged` mount). Admin reaches them via sudo.
sudo chown "${SERVICE_USER}:${SERVICE_ADMIN}" "${SERVICE_HOME}"                       # service user owns; admin group is the owning group
sudo chmod u=rwx,g=rwx,o=,g+s "${SERVICE_HOME}"                                       # owner/admin access; deny others; inherit group on new items
apply_acl_tree "${SERVICE_HOME}/.config"                                              # .config subtree: full sudo-less admin ACL treatment

# --- bind-mount data dirs: created + owned BEFORE the SERVICE_DIR ACL pass ---------
# Each container writes as a mapped subuid, so its data dir must be owned by that subuid
# (chown as root; only root crosses the id range). We create + own them FIRST so they
# neither inherit SERVICE_DIR's default ACL (inheritance happens at creation) nor get
# walked by apply_acl_tree (which prunes data/).
#
# No ACLs here: Postgres refuses a group-accessible PGDATA. Admin reaches them via sudo;
# back Postgres up with pg_dumpall, not a raw file copy.
SUBUID_BASE="$(awk -F: -v u="${SERVICE_USER}" '$1==u {print $2}' /etc/subuid)"
SUBGID_BASE="$(awk -F: -v u="${SERVICE_USER}" '$1==u {print $2}' /etc/subgid)"

# data subdir : in-container uid/gid it must be owned by
data_dirs=(
    "sqlite-db:1000"        # /data         — lightflow sqlite-db directory (pinned User=1000:1000)
    "scripts:1000"          # /scripts      — lightflow scripts directory (pinned User=1000:1000)
    "python-libs:1000"      # /python-libs  — lightflow python library directory (pinned User=1000:1000)
)

mkdir -p "${SERVICE_DIR}/data"                                                        # the data/ parent: plain, no ACL
sudo chown "${SERVICE_USER}:${SERVICE_ADMIN}" "${SERVICE_DIR}/data"                   # owner/admin
sudo chmod 0750 "${SERVICE_DIR}/data"                                                 # children are set individually below
for entry in "${data_dirs[@]}"; do
    sub="${entry%%:*}"                                  # subdir name
    ids="${entry#*:}"                                   # "uid" or "uid:gid"
    uid="${ids%%:*}"                                    # uid
    gid="${ids#*:}"; [ "$gid" = "$ids" ] && gid="$uid"  # gid (defaults to uid)
    dir="${SERVICE_DIR}/data/${sub}"
    sudo mkdir -p "$dir"
    sudo chown -R "$((SUBUID_BASE + uid - 1)):$((SUBGID_BASE + gid - 1))" "$dir"   # map both to host subids
    sudo chmod 0700 "$dir"
done

# --- SERVICE_DIR: application data, env files, configuration -------------------
# Full treatment so the admin group gets sudo-less access AND the service user keeps
# access even on admin-created files. data/ is EXCLUDED: those dirs are owned by the
# containers' subuids and need engine-specific modes (Postgres rejects a group/other-
# accessible PGDATA), so they are handled separately just below.
apply_acl_tree "${SERVICE_DIR}" "${SERVICE_DIR}/data"                                 # treat everything EXCEPT data/

Configure Secure NGINX Static Assets Permissions

# Grant the NGINX worker user (www-data) read-only access to assets subtrees.
#   $1     = service directory → traverse-only (x)
#   $2..$n = assets subtrees served directly by NGINX (configs, icons, favicon, manifest, ...)
apply_acl_nginx() {
    local tree="$1"; shift
    local assets

    sudo setfacl -m u:www-data:x "$tree"                                        # traverse only: pass through, no listing, no reading

    for assets in "$@"; do
        # assets directories — enter + list, existing and future
        sudo find "$assets" -type d -exec setfacl    -m u:www-data:rx {} +      # www-data: read existing dirs
        sudo find "$assets" -type d -exec setfacl -d -m u:www-data:rx {} +      # www-data: inherit read on new items

        # assets files — www-data read-only, owner/admin rw, never executable.
        # Entries AND mask are pinned explicitly: a bare `setfacl -m u:www-data:r`
        # would recalculate the mask to the union of all entries, resurfacing the
        # latent rwx inherited from the default ACL (files would show group rwx).
        sudo find "$assets" -type f -exec setfacl \
          -m u:www-data:r,u:${SERVICE_USER}:rw,g:${SERVICE_ADMIN}:rw,u::rw,g::rw,o::-,m::rw {} +
    done
}

# --- SERVICE_DIR/nginx: served directly by NGINX -----------
# Created AFTER the apply_acl_tree pass so they inherit the default ACL + setgid
# group (service user/admin keep full access); www-data is then layered on top
# as a read-only named entry.
sudo mkdir -p "${SERVICE_DIR}/nginx"
sudo chown "${SERVICE_USER}:${SERVICE_ADMIN}" "${SERVICE_DIR}/nginx"
apply_acl_nginx "${SERVICE_DIR}" "${SERVICE_DIR}/nginx"

🛡️ Rootless Podman Defaults

Configure rootless Podman per-user configuration directory

# configure rootless Podman defaults for this service user
# - store per-user Podman configuration under ~/.config/containers
sudo -u "${SERVICE_USER}" mkdir -p "${SERVICE_HOME}/.config/containers/systemd"

Configure rootless Podman defaults

# configure rootless Podman defaults for this service user
# - use k8s-file logging for easier log rotation and inspection
sudo -u "${SERVICE_USER}" tee "${SERVICE_HOME}/.config/containers/containers.conf" >/dev/null <<EOF
[containers]
log_driver = "k8s-file"

[engine]
healthcheck_events=false
EOF

🔧 Service Configuration

Setup LightFlow Parameters

Before deploying, you need to define a few environment variables that will be used throughout the setup process.

• BASE_URL: public URL where the web service is accessible
• HOST_PORT: external port used by NGINX to route traffic to the service
• ADMIN_EMAIL: the default admin account (email only — there is no admin password in the env)
• SMTP settings: required so the admin and users receive their set-password and password-reset links

NGINX Proxy ConfigurationLightflow ConfigurationSMTP Configuration

###################################################################################
# NGINX Proxy Configuration
###################################################################################
HOST_PORT=10070

###################################################################################
# Lightflow Configuration
###################################################################################
BASE_URL=https://lightflow.domain.fr
ADMIN_EMAIL=admin@domain.fr

###################################################################################
# SMTP Configuration
###################################################################################
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_SECURITY=starttls
SMTP_USER=lightflow@domain.fr
SMTP_PASSWORD=your-smtp-password
SMTP_FROM=lightflow@domain.fr

Getting Your Gmail SMTP Password

To send emails via Gmail SMTP, you'll need to generate an App Password.
A special password used for third-party applications.
Visit myaccount.google.com/apppasswords to create one.

Use the generated password in the SMTP_PASSWORD field of your configuration.

Create the Environment file

lightflow.env

sudo -u "${SERVICE_USER}" tee "${SERVICE_DIR}/lightflow.env" >/dev/null <<EOF
###################################################################################
# NGINX Proxy Configuration
###################################################################################
HOST_PORT=${HOST_PORT}

###################################################################################
# Build Version
# Short git commit shown in the UI's About panel; read by the server at runtime
# (NOT baked into the image, so a commit never forces a backend rebuild). Left
# empty here and set by the bare repo's post-receive deploy hook on every push,
# via /usr/local/bin/lightflow-deploy (see "Updating Lightflow").
###################################################################################
GIT_COMMIT=

###################################################################################
# LightFlow Configuration
###################################################################################
LIGHTFLOW_BASE_URL=${BASE_URL}
LIGHTFLOW_TIMEZONE=Europe/Paris
LIGHTFLOW_GLOBAL_CONCURRENCY=4
LIGHTFLOW_LOG_RETENTION_DAYS=30
LIGHTFLOW_SESSION_TTL_DAYS=30
LIGHTFLOW_COOKIE_SECURE=true

###################################################################################
# Default admin (seeded 'pending' on first start)
###################################################################################
LIGHTFLOW_ADMIN_EMAIL=${ADMIN_EMAIL}

###################################################################################
# SMTP (required for set-password and password-reset links)
###################################################################################
SMTP_HOST=${SMTP_HOST}
SMTP_PORT=${SMTP_PORT}
SMTP_SECURITY=${SMTP_SECURITY}
SMTP_USER=${SMTP_USER}
SMTP_PASSWORD=${SMTP_PASSWORD}
SMTP_FROM=${SMTP_FROM}
EOF

Keep the .env files

All the secret informations will be stored in the .env files.

⚙️ Create LightFlow build

Create LightFlow build Dockerfile

build/Dockerfile

# setup of Dockerfile
mkdir -p "${SERVICE_DIR}/build"
tee "${SERVICE_DIR}/build/Dockerfile" > /dev/null <<'EOF'
# Single multi-stage Dockerfile, built as FOUR separately-tagged images by four
# Quadlet `.build` units that each select a stage with `Target=` + `ImageTag=`:
#
#   lightflow-planner.build   Target=planner  → localhost/lightflow-planner:latest
#   lightflow-frontend.build  Target=frontend → localhost/lightflow-frontend:latest
#   lightflow-backend.build   Target=backend  → localhost/lightflow-backend:latest
#   lightflow.build           Target=runtime  → localhost/lightflow:latest
#
# Every expensive cache lives inside a TAGGED image, so they all survive the
# `podman image prune` the runtime build runs: the backend stage KEEPS its Rust
# toolchain + target/ (cargo-chef's cooked deps), and the planner stage is tagged
# so its recipe layer isn't reclaimed as dangling (it used to be, which forced
# `cargo chef prepare` to re-run on every restart). The prune only reclaims old
# replaced image versions now.

#############################################
##### Shared Rust base with cargo-chef
#############################################
FROM docker.io/library/rust:1-slim AS chef
# build-essential: libsqlite3-sys (bundled SQLite) + ring (rustls) compile C.
RUN apt-get update \
    && apt-get install -y --no-install-recommends build-essential pkg-config \
    && rm -rf /var/lib/apt/lists/* \
    && cargo install cargo-chef --locked
WORKDIR /app

#############################################
##### Planner — capture the dependency recipe (cheap; no compilation).
##### Only the manifests are copied, so this layer + recipe.json are cached
##### unless Cargo.toml/Cargo.lock change — a source-only edit never re-runs it.
##### The recipe is byte-identical to a full-source prepare, so `cook` below
##### still cache-hits. Tagged as localhost/lightflow-planner:latest by
##### lightflow-planner.build so the runtime build's prune can't reclaim it.
#############################################
FROM chef AS planner
COPY src/backend/Cargo.toml src/backend/Cargo.lock ./
RUN cargo chef prepare --recipe-path recipe.json

#############################################
##### Backend — cook deps (cached), then build the app.
##### Tagged image; retains the toolchain + target/ so the cook cache persists.
#############################################
FROM chef AS backend
# Cooks ONLY the dependencies — cached unless Cargo.toml/Cargo.lock change,
# so a source-only edit recompiles just this crate, not the whole dep tree.
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --release --recipe-path recipe.json
COPY src/backend/ .
# build.rs bakes the build date here (and the commit in a dev tree with .git).
# The commit is deliberately NOT a build-arg: it is injected at runtime via the
# env file, so a commit that touches no backend code never busts this cache.
RUN cargo build --release

#############################################
##### Frontend — build the React SPA (npm deps cached separately from source)
#############################################
FROM docker.io/library/node:24-alpine AS frontend
WORKDIR /app
COPY src/frontend/package.json src/frontend/package-lock.json* ./
RUN npm ci || npm install
COPY src/frontend/ ./
RUN npm run build

#############################################
##### Runtime — slim final image
#############################################
FROM docker.io/library/debian:trixie-slim AS runtime
# python3 + openssh-client: run the user scripts (which SSH to the host gate);
# python3-pip + python3-venv: let admins install extra Python packages into a
# venv on the data volume (see packages.rs); ca-certificates: TLS roots for SMTP
# and PyPI; tini: PID 1 reaping; curl: healthcheck.
RUN apt-get update \
    && apt-get install -y --no-install-recommends \
      ca-certificates python3 python3-pip python3-venv openssh-client tini curl \
    && rm -rf /var/lib/apt/lists/*

# Runs as a fixed non-root uid so the host data dir can be pre-chowned to its subuid.
RUN useradd --system --uid 1000 --create-home --home-dir /home/lightflow lightflow

COPY --from=backend /app/target/release/lightflow /usr/local/bin/lightflow
# --chown: public/ assets keep the checkout's 0660 root:root mode, unreadable by
# the runtime uid 1000 — without this the backend 404s every favicon/manifest.
COPY --chown=lightflow:lightflow --from=frontend /app/dist /app/static

# UI-authored scripts live on their own bind-mounted volume at /scripts (kept out
# of the sqlite-db /data volume); the executor runs steps from there.
ENV LIGHTFLOW_STATIC_DIR=/app/static \
    LIGHTFLOW_DATA_DIR=/data \
    LIGHTFLOW_SCRIPTS_DIR=/scripts \
    LIGHTFLOW_PYTHON_LIBS_DIR=/python-libs \
    LIGHTFLOW_BIND=0.0.0.0:8080

EXPOSE 8080
USER lightflow
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["lightflow"]
EOF

🧩 Quadlet Service

Create lightflow-planner.build Podman Quadlet

lightflow-planner.build

# podman quadlet: create lightflow-planner.build
sudo -u "${SERVICE_USER}" tee "${SERVICE_HOME}/.config/containers/systemd/lightflow-planner.build" >/dev/null <<'EOF'
[Unit]
Description=Build lightflow planner image (cargo-chef dependency recipe)

[Build]
ImageTag=localhost/lightflow-planner:latest
# Build only the `planner` stage. Tagging it keeps its recipe layer out of the
# `dangling` set, so the runtime build's `podman image prune` can't reclaim it —
# otherwise `cargo chef prepare` re-runs on every restart. The stage copies only
# Cargo.toml/Cargo.lock, so this is a cache hit unless dependencies change.
# (If your podman is too old for Target=, use: PodmanArgs=--target planner)
Target=planner
File=/media/ssd/podman/lightflow/build/Dockerfile
SetWorkingDirectory=/media/ssd/podman/lightflow

[Service]
EnvironmentFile=/media/ssd/podman/lightflow/lightflow.env
EOF

Create lightflow-backend.build Podman Quadlet

lightflow-backend.build

# podman quadlet: create lightflow-backend.build
sudo -u "${SERVICE_USER}" tee "${SERVICE_HOME}/.config/containers/systemd/lightflow-backend.build" >/dev/null <<'EOF'
[Unit]
Description=Build lightflow backend image (Rust, cargo-chef cached)
# The planner image is built (and TAGGED) first so its recipe layer is a cache
# hit here and survives the runtime build's prune.
Requires=lightflow-planner-build.service
After=lightflow-planner-build.service

[Build]
ImageTag=localhost/lightflow-backend:latest
# Build only the `backend` stage. This tagged image keeps the toolchain +
# target/, so cargo-chef's cooked dependency cache survives `podman image prune`.
# (If your podman is too old for Target=, use: PodmanArgs=--target backend)
Target=backend
File=/media/ssd/podman/lightflow/build/Dockerfile
SetWorkingDirectory=/media/ssd/podman/lightflow

[Service]
EnvironmentFile=/media/ssd/podman/lightflow/lightflow.env
EOF

Create lightflow-frontend.build Podman Quadlet

lightflow-frontend.build

# podman quadlet: create lightflow-frontend.build
sudo -u "${SERVICE_USER}" tee "${SERVICE_HOME}/.config/containers/systemd/lightflow-frontend.build" >/dev/null <<'EOF'
[Unit]
Description=Build lightflow frontend image (React SPA)

[Build]
ImageTag=localhost/lightflow-frontend:latest
# Build only the `frontend` stage of the shared Dockerfile.
# (If your podman is too old for Target=, use: PodmanArgs=--target frontend)
Target=frontend
File=/media/ssd/podman/lightflow/build/Dockerfile
SetWorkingDirectory=/media/ssd/podman/lightflow

[Service]
EnvironmentFile=/media/ssd/podman/lightflow/lightflow.env
EOF

Create lightflow.build Podman Quadlet

lightflow.build

# podman quadlet: create lightflow.build
sudo -u "${SERVICE_USER}" tee "${SERVICE_HOME}/.config/containers/systemd/lightflow.build" >/dev/null <<'EOF'
[Unit]
Description=Build lightflow runtime image
# The phase images are built first so their layer caches are warm + tagged. The
# planner image is pulled in transitively (backend-build requires it).
Requires=lightflow-frontend-build.service lightflow-backend-build.service
After=lightflow-frontend-build.service lightflow-backend-build.service

[Build]
ImageTag=localhost/lightflow:latest
# Build the slim `runtime` stage; it COPYs the binary + SPA from the two phase
# stages (cache hits, since those were just built).
Target=runtime
File=/media/ssd/podman/lightflow/build/Dockerfile
SetWorkingDirectory=/media/ssd/podman/lightflow

[Service]
EnvironmentFile=/media/ssd/podman/lightflow/lightflow.env
# Safe to prune here: every expensive cache (cargo-chef recipe + cooked deps,
# npm) lives inside a TAGGED phase image (lightflow-planner / lightflow-backend /
# lightflow-frontend), so this only reclaims old replaced image versions.
ExecStartPost=-/usr/bin/podman image prune -f
EOF

Create Server Podman Quadlet

lightflow-server.container

# podman quadlet: create lightflow server
sudo -u "${SERVICE_USER}" tee "${SERVICE_HOME}/.config/containers/systemd/lightflow-server.container" >/dev/null <<'EOF'
[Unit]
Description=lightflow server
Requires=lightflow-build.service
After=lightflow-build.service

[Container]
EnvironmentFile=/media/ssd/podman/lightflow/lightflow.env
Image=lightflow.build
ContainerName=lightflow-server

# Attach the container's stdout/stderr straight to this systemd unit so the
# binary's `tracing` lines land in the unit journal (followed with the
# `journalctl _UID=…` command shown below). Without this, Podman's default
# journald driver logs them under CONTAINER_NAME=lightflow-server in a separate
# scope, so the unit view shows only systemd/podman lifecycle lines, never the
# app. Trade-off: with passthrough, `podman logs lightflow-server` no longer
# works — read via the journal.
LogDriver=passthrough

# No User= needed: the image declares USER lightflow (1000). The data dir is
# pre-chowned to 1000's mapped subuid, so a plain :rw mount works.
Volume=/media/ssd/podman/lightflow/data/sqlite-db:/data:rw
# UI-authored scripts live on their own volume (writable), mounted at /scripts.
Volume=/media/ssd/podman/lightflow/data/scripts:/scripts:rw
# Extra Python packages (a venv) live on their own volume, kept out of /data.
Volume=/media/ssd/podman/lightflow/data/python-libs:/python-libs:rw

PublishPort=127.0.0.1:${HOST_PORT}:8080

HealthCmd=curl -fsS -o /dev/null http://localhost:8080/api/alive
HealthStartPeriod=30s
HealthInterval=60s
HealthTimeout=5s
HealthRetries=3
HealthOnFailure=kill

[Service]
EnvironmentFile=/media/ssd/podman/lightflow/lightflow.env
UMask=0007
SuccessExitStatus=143

Restart=always
RestartSec=30
TimeoutStartSec=900

[Install]
WantedBy=default.target
EOF

UMask=0007: removes all permissions for others while preserving read/write access for the service owner and admin group on newly created files and directories SuccessExitStatus=143: treat a SIGTERM-style exit (143) as a clean stop, not a failure

Enable and Start Quadlet Services

# open an interactive shell as the service user
sudo -iu "${SERVICE_USER}"

# reload systemd user units
systemctl --user daemon-reload

# start Podman Quadlet services
systemctl --user start lightflow-server.service

# verify service status
systemctl --user status lightflow-server.service

To follow the service logs in real time, run sudo journalctl _UID=$(id -u ${SERVICE_USER}) -f from the Debian account.

---

🚀 Deploy Lightflow

Install NGINX

NGINX needs to be installed, follow the NGINX section.

Configure NGINX

NGINX needs to be configured using a file in /etc/nginx/sites-enabled directory.
This configuration file specify the documentation path:

/etc/nginx/sites-enabled/lightflow.domain.frActivate website

server {
  server_name lightflow.domain.fr;

  # setup 404 error_page
  error_page 404 /404.html;
  include snippets/error-404.conf;

  # show maintenance page when backend is down
  error_page 502 503 504 /maintenance.html;
  include snippets/error-maintenance.conf;

  # reverse proxy
  location / {
    proxy_pass http://127.0.0.1:10070;

    # keep it HTTP/1.1
    proxy_http_version 1.1;

    # forwarded headers
    include proxy_params;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Port $server_port;

    # server-sent events: live UI updates stream here
    proxy_buffering off;
    proxy_cache off;
    proxy_read_timeout 3600s;
    proxy_send_timeout 3600s;
    proxy_redirect off;
  }
}

# restart nginx
sudo nginx -t && sudo service nginx restart

Replace lightflow.domain.fr by the name of your website, and 10070 by HOST_PORT.

Activate HTTPS

To activate HTTPS protocol, follow the Let's Encrypt section.

First login — set the admin password

The admin account is created pending on first start (from LIGHTFLOW_ADMIN_EMAIL).
There is no admin password in lightflow.env. To activate it:

1. Open the site and, on the login page, click "First run? Send admin setup link".
2. Open the set-password link, from the e-mail (SMTP), or from the server log.
3. Choose a password then sign in.

This button is first-run only, the login page hides it as soon as an admin account is active.

Forgot a password later? The login page's "Forgot password?" link e-mails a reset link to any registered address. The confirmation is deliberately generic (it never reveals whether an e-mail exists) and rate-limited to one mail per address per minute; the link reuses the same set-password page and invalidates old sessions.

---

⬆️ Updating Lightflow

Lightflow deploys on push. The bare repo's post-receive hook calls the deploy wrapper, which stamps the pushed commit into lightflow.env, acknowledges the push right away, and then runs podmanctl --update lightflow (stop → git pull → rebuild → restart) detached in the background. The rebuild takes 10 min+ — far longer than the HTTP push stays open — so the heavy work is handed to a worker that outlives the push, and git push returns in seconds with a short status. podmanctl no longer stamps GIT_COMMIT itself — the wrapper owns that, so the About panel matches the deployed commit.

A flock in the worker serialises deploys: if you push again while a build is still running, the second deploy waits for the first to finish, then redeploys whatever master points at by then — concurrent git pull / image builds can never overlap, and the server always converges on the latest pushed commit. Follow a running deploy with sudo tail -f /var/log/lightflow-deploy.log.

One-time setup — deploy wrapper, hook and sudoers

1. Deploy wrapper + worker (root)2. post-receive hook (bare repo)3. sudoers rule4. Wire the clone for deploys

# the deploy wrapper bakes in the instantiated service dir — guard it exists first
[ -d "${SERVICE_DIR}" ] || { echo "missing ${SERVICE_DIR}"; exit 1; }

# (a) the wrapper — fast + synchronous: stamp the commit, ack the push, detach the build.
# ${SERVICE_DIR}/${SERVICE_NAME} expand now; the rest stays literal (\$…).
sudo tee /usr/local/bin/lightflow-deploy >/dev/null <<EOF
#!/bin/sh
set -eu
ENV_FILE=${SERVICE_DIR}/lightflow.env
LOG=/var/log/${SERVICE_NAME}-deploy.log

commit=\${1:-}
case "\$commit" in ''|*[!0-9a-f]*) echo "usage: lightflow-deploy <short-sha>" >&2; exit 2 ;; esac
[ -f "\$ENV_FILE" ] && grep -q '^GIT_COMMIT=' "\$ENV_FILE" || { echo "lightflow-deploy: no GIT_COMMIT= in \$ENV_FILE" >&2; exit 1; }

# fast + synchronous: stamp the commit and stream a short ack back over the push
sed -i "s/^GIT_COMMIT=.*/GIT_COMMIT=\$commit/" "\$ENV_FILE"
echo "lightflow: GIT_COMMIT -> \$commit"
echo "lightflow: starting 'podmanctl --update ${SERVICE_NAME}' in the background (~10 min)"

# slow + detached: a 10 min rebuild must NOT hold the HTTP push open (it trips the
# git/proxy timeout). Hand it to the worker in a NEW session with its std fds off
# the push connection, so 'git push' returns now.
setsid /usr/local/bin/${SERVICE_NAME}-deploy-run "\$commit" </dev/null >>"\$LOG" 2>&1 &

echo "lightflow: push accepted - deploy running in background"
echo "lightflow: follow it with ->  sudo tail -f \$LOG"
EOF
sudo chmod 0755 /usr/local/bin/lightflow-deploy

# (b) the worker — slow + detached: serialise deploys with flock, then rebuild.
sudo tee /usr/local/bin/${SERVICE_NAME}-deploy-run >/dev/null <<EOF
#!/bin/sh
set -eu
LOCK=/run/lock/${SERVICE_NAME}-deploy.lock
SERVICE_NAME=${SERVICE_NAME}

commit=\${1:-unknown}

# serialise deploys: fd 9 holds an exclusive lock for the whole worker. A second
# push that lands mid-build BLOCKS at flock, then runs once we finish — so it is
# never dropped; it redeploys whatever master points at by then.
exec 9>"\$LOCK"
echo "[\$(date -Is)] queued:  \$commit"
flock 9
echo "[\$(date -Is)] running: podmanctl --update \$SERVICE_NAME (commit \$commit)"
podmanctl --update "\$SERVICE_NAME"
echo "[\$(date -Is)] done:    \$commit"
EOF
sudo chmod 0755 /usr/local/bin/${SERVICE_NAME}-deploy-run

The hook's sudo runs the wrapper as root: it stamps GIT_COMMIT (the hex-only check keeps the wildcard sudoers rule safe — no sed injection), prints a short ack that streams back over the push, then setsid-detaches the worker so the long rebuild outlives the HTTP connection. The worker's flock serialises deploys; its output (plus podmanctl's) lands in /var/log/lightflow-deploy.log. No extra sudoers entry is needed — the wrapper is already root when it launches the worker.

# install into the bare repo's hooks/ — fires on every push
sudo tee /media/ssd/git/podman/lightflow.git/hooks/post-receive >/dev/null <<'EOF'
#!/bin/sh
set -eu
DEPLOY_BRANCH=refs/heads/master
while read -r oldrev newrev refname; do
  [ "$refname" = "$DEPLOY_BRANCH" ] || continue          # only the deploy branch
  sudo /usr/local/bin/lightflow-deploy "$(git rev-parse --short "$newrev")" </dev/null
done
EOF
# normalize the hook ownership
sudo chown www-data:debian /media/ssd/git/podman/lightflow.git/hooks/post-receive
sudo chmod 0750            /media/ssd/git/podman/lightflow.git/hooks/post-receive

The commit comes from the just-pushed ref (the bare repo already holds it), so it is stamped before the instantiated repo pulls — the env always matches the code the restart runs.

# let the git/web user run ONLY the deploy wrapper as root, no password
sudo tee /etc/sudoers.d/lightflow-deploy >/dev/null <<'EOF'
www-data ALL=(root) NOPASSWD: /usr/local/bin/lightflow-deploy *
EOF
sudo chmod 0440 /etc/sudoers.d/lightflow-deploy
sudo visudo -cf /etc/sudoers.d/lightflow-deploy

www-data is the user your HTTPS git backend runs the hook as — replace it if yours differs (e.g. if you push over SSH as debian).

# deploys pull non-interactively as root: pull from the co-located bare repo over a
# LOCAL path (the HTTPS origin needs a credential prompt + TTY, which a hook lacks)
git -C ${SERVICE_DIR} remote set-url origin /media/ssd/git/podman/lightflow.git

# trust both repos for root so the root-run `git pull` doesn't refuse with
# "detected dubious ownership"; --system applies whatever HOME the hook runs with
sudo git config --system --add safe.directory ${SERVICE_DIR}
sudo git config --system --add safe.directory /media/ssd/git/podman/lightflow.git

Without the local origin the pull dies with could not read Username for 'https://…'; without safe.directory it dies with detected dubious ownership. Either way podmanctl --update aborts and leaves the service stopped.

Deploy a new version

# from your working clone — the push triggers the whole deploy
git push origin master

The push returns in seconds with a short status (GIT_COMMIT -> …, deploy started); the stop → git pull → rebuild → restart then runs detached in the background. Watch it with sudo tail -f /var/log/lightflow-deploy.log; the About panel shows the new GIT_COMMIT once the restart completes.

A manual podmanctl --update lightflow still works but no longer refreshes GIT_COMMIT — run sudo lightflow-deploy <short-sha> (or just push) to update it.

Changing a task's CRON, steps, variables or pool in the UI takes effect immediately. The scheduler reloads in memory, no restart needed.