📦 Installing the Registry

The Registry is a private container image registry — the upstream distribution project, the same implementation Docker Hub is built on. It exists so this host stops compiling Rust.

An own project's image is built on a dev machine, pushed here, and podmanctl --update <app> pulls it instead of rebuilding — the ODROID is 4 cores / 4 GB, and a full dependency cook on it runs about 20 minutes per app. A Windows-on-ARM laptop builds the very same linux/arm64 image natively, and an x86 desktop cross-compiles it.

Features:

• 📦 Stores the fleet's prebuilt application images, tagged <app commit>.<platform commit>
• 🔒 Every request authenticated — this host serves production binaries for every other service
• 🧹 Weekly garbage collection reclaims blobs no manifest references any more
• 🪶 No database, no UI: a single Alpine container over a bind-mounted directory

Info

The project is open-source and can be downloaded here: https://github.com/distribution/distribution.

This service is infrastructure, not an app

Losing its storage costs nothing permanent — every image can be rebuilt and pushed again from a dev machine. But while it is down, every own project refuses to deploy: an app whose env file carries DEPLOY_REGISTRY will not fall back to building locally, by design.

---

📥 Installation

This section describes how to install and run Podman services using systemd Quadlet, enabling containers to restart automatically. It allows the service to run in its own isolated rootless environment with a dedicated Linux user.

📋 Requirements

Info

The Registry requires the installation of

• NGINX
• Podman

---

⚙️ Configuration

📋 Service Setup

Info

Please follow the following installation steps:

• 👥 User and Group
• 🔁 Persistent Services
• 🔍 Detecting a container's runtime UID/GID

Define Service Variables

# define the service name
SERVICE_NAME=registry

Initialize Service Environment

# built-in bash safety and SERVICE_NAME validation
set -euo pipefail; [ -n "${SERVICE_NAME:-}" ] || { echo "SERVICE_NAME is empty"; exit 1; }

# automatically generate variables
SERVICE_HOME="/media/ssd/podman-users/${SERVICE_NAME}"
SERVICE_DIR="/media/ssd/podman/${SERVICE_NAME}"
SERVICE_USER="${SERVICE_NAME}_svc"
SERVICE_ADMIN="${SERVICE_NAME}_admins"

🔐 Service Isolation

Configure Rootless Podman UID/GID Namespace Mappings

# allocate rootless Podman UID/GID mappings if missing
SUBID_SIZE=65536
allocate_subids_if_missing() {
  local user="$1"
  local start
  if ! grep -q "^${user}:" /etc/subuid; then
    start="$(awk -F: 'BEGIN { max = 100000 } { end = $2 + $3; if (end > max) max = end } END { print max }' /etc/subuid)"
    sudo usermod --add-subuids "${start}-$((start + SUBID_SIZE - 1))" "$user"
  fi
  if ! grep -q "^${user}:" /etc/subgid; then
    start="$(awk -F: 'BEGIN { max = 100000 } { end = $2 + $3; if (end > max) max = end } END { print max }' /etc/subgid)"
    sudo usermod --add-subgids "${start}-$((start + SUBID_SIZE - 1))" "$user"
  fi
}
allocate_subids_if_missing "${SERVICE_USER}"

# verify rootless Podman UID/GID namespace mappings exist
grep -q "^${SERVICE_USER}:" /etc/subuid || { echo "ERROR: missing subuid mappings for ${SERVICE_USER}"; exit 1; }
grep -q "^${SERVICE_USER}:" /etc/subgid || { echo "ERROR: missing subgid mappings for ${SERVICE_USER}"; exit 1; }

# validate that all subordinate ID ranges in a file are non-overlapping
validate_subid_ranges() {
  local file="$1"
  awk -F: '
    {
      start = $2
      end   = $2 + $3 - 1
      for (i = 1; i <= count; i++) {
        if (start <= ends[i] && end >= starts[i]) {
          printf "ERROR: overlapping ranges in %s\n", FILENAME
          printf "  %s:%s:%s overlaps %s:%s:%s\n",
            $1, $2, $3,
            names[i], starts[i], sizes[i]
          exit 1
        }
      }
      count++
      names[count]  = $1
      starts[count] = start
      ends[count]   = end
      sizes[count]  = $3
    }
  ' "$file"
}
validate_subid_ranges /etc/subuid
validate_subid_ranges /etc/subgid

# configure the rootless storage driver BEFORE Podman first initializes its storage.
# fuse-overlayfs is a reliable rootless overlay backend and avoids kernel-version
# quirks of native rootless overlay on ARM/Armbian. It is optional with the default
# userns used here, but recommended; set it before the first storage init so no
# 'podman system reset' is needed later.
sudo -u "${SERVICE_USER}" mkdir -p "${SERVICE_HOME}/.config/containers"
sudo -u "${SERVICE_USER}" tee "${SERVICE_HOME}/.config/containers/storage.conf" >/dev/null <<'EOF'
[storage]
driver = "overlay"

[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"
EOF

# rebuild rootless Podman state to ensure existing containers/storage use current namespace mappings
# use a login shell (-i) so HOME points at the service user's home
# (storage now initializes with fuse-overlayfs; no 'podman system reset' needed)
sudo -iu "${SERVICE_USER}" podman system migrate

Configure Secure Service Directory Permissions

# Apply the admin/service ACL policy to one directory tree, deterministically.
#   $1 = tree to treat
#   $2 = (optional) a sub-path to skip entirely (e.g. a container data subtree)
#
# Why setfacl, not chmod: on an ACL'd tree, chmod edits only the mask, never group::,
# so stale execute bits survive. setfacl sets every entry explicitly (mask included),
# lowercase — the result is reproducible and no execute bit can resurface.
apply_acl_tree() {
    local tree="$1"
    local skip="${2:-}"
    local prune=()
    [ -n "$skip" ] && prune=( -path "$skip" -prune -o )                                         # skip this subtree if a 2nd arg is given

    sudo find "$tree" "${prune[@]}" -exec chown "${SERVICE_USER}:${SERVICE_ADMIN}" {} +         # service user owns; admin group is the owning group

    # directories — access ACL (perms on the existing dirs)
    sudo find "$tree" "${prune[@]}" -type d -exec setfacl    -m g:${SERVICE_ADMIN}:rwx {} +     # admin group: access existing dirs
    sudo find "$tree" "${prune[@]}" -type d -exec setfacl    -m u:${SERVICE_USER}:rwx {} +      # service user: access existing dirs
    sudo find "$tree" "${prune[@]}" -type d -exec setfacl    -m u::rwx,g::rwx,o::-,m::rwx {} +  # owner/group rwx, deny others, pin mask rwx

    # directories — default ACL (inherited by NEW files/dirs created later)
    sudo find "$tree" "${prune[@]}" -type d -exec setfacl -d -m g:${SERVICE_ADMIN}:rwx {} +     # admin group: inherit access on new items
    sudo find "$tree" "${prune[@]}" -type d -exec setfacl -d -m u:${SERVICE_USER}:rwx {} +      # service user: inherit access on new items
    sudo find "$tree" "${prune[@]}" -type d -exec setfacl -d -m u::rwx,g::rwx,o::- {} +         # default owner/group/other for new items

    # files — access ACL (rw only, never executable)
    sudo find "$tree" "${prune[@]}" -type f -exec setfacl    -m g:${SERVICE_ADMIN}:rw {} +      # admin group: access existing files
    sudo find "$tree" "${prune[@]}" -type f -exec setfacl    -m u:${SERVICE_USER}:rw {} +       # service user: access existing files
    sudo find "$tree" "${prune[@]}" -type f -exec setfacl    -m u::rw,g::rw,o::-,m::rw {} +     # owner/group rw, deny others, pin mask rw

    sudo find "$tree" "${prune[@]}" -type d -exec chmod g+s {} +                                # setgid: new items inherit the group (a MODE bit; ACLs can't set it)
}

# --- SERVICE_HOME: Podman's private runtime home ------------------------------
# Own the home + set setgid, then ACL ONLY the .config subtree (containers.conf,
# storage.conf, and the *.network / *.container quadlets you hand-edit) so admins
# can edit them sudo-less.
#
# Leave ~/.local (storage) and ~/.cache to Podman: ACLs on the overlay store break
# the runtime ("OCI permission denied" on the `merged` mount). Admin reaches them via sudo.
sudo chown "${SERVICE_USER}:${SERVICE_ADMIN}" "${SERVICE_HOME}"                       # service user owns; admin group is the owning group
sudo chmod u=rwx,g=rwx,o=,g+s "${SERVICE_HOME}"                                       # owner/admin access; deny others; inherit group on new items
apply_acl_tree "${SERVICE_HOME}/.config"                                              # .config subtree: full sudo-less admin ACL treatment

# --- bind-mount data dirs: created + owned BEFORE the SERVICE_DIR ACL pass ---------
# Each container writes as a mapped subuid, so its data dir must be owned by that subuid
# (chown as root; only root crosses the id range). We create + own them FIRST so they
# neither inherit SERVICE_DIR's default ACL (inheritance happens at creation) nor get
# walked by apply_acl_tree (which prunes data/).
#
# No ACLs here: Postgres refuses a group-accessible PGDATA. Admin reaches them via sudo;
# back Postgres up with pg_dumpall, not a raw file copy.
SUBUID_BASE="$(awk -F: -v u="${SERVICE_USER}" '$1==u {print $2}' /etc/subuid)"
SUBGID_BASE="$(awk -F: -v u="${SERVICE_USER}" '$1==u {print $2}' /etc/subgid)"

# data subdir : in-container uid/gid it must be owned by
data_dirs=(
    "registry:1000"      # /var/lib/registry — matches User=1000 in the quadlet
)

mkdir -p "${SERVICE_DIR}/data"                                                        # the data/ parent: plain, no ACL
sudo chown "${SERVICE_USER}:${SERVICE_ADMIN}" "${SERVICE_DIR}/data"                   # owner/admin
sudo chmod 0750 "${SERVICE_DIR}/data"                                                 # children are set individually below
for entry in "${data_dirs[@]}"; do
    sub="${entry%%:*}"                                  # subdir name
    ids="${entry#*:}"                                   # "uid" or "uid:gid"
    uid="${ids%%:*}"                                    # uid
    gid="${ids#*:}"; [ "$gid" = "$ids" ] && gid="$uid"  # gid (defaults to uid)
    dir="${SERVICE_DIR}/data/${sub}"
    sudo mkdir -p "$dir"
    sudo chown -R "$((SUBUID_BASE + uid - 1)):$((SUBGID_BASE + gid - 1))" "$dir"   # map both to host subids
    sudo chmod 0700 "$dir"
done

# --- SERVICE_DIR: application data, env files, configuration -------------------
# Full treatment so the admin group gets sudo-less access AND the service user keeps
# access even on admin-created files. data/ is EXCLUDED: those dirs are owned by the
# containers' subuids and need engine-specific modes (Postgres rejects a group/other-
# accessible PGDATA), so they are handled separately just below.
apply_acl_tree "${SERVICE_DIR}" "${SERVICE_DIR}/data"                                 # treat everything EXCEPT data/

Configure Secure NGINX Static Assets Permissions

# Grant the NGINX worker user (www-data) read-only access to assets subtrees.
#   $1     = service directory → traverse-only (x)
#   $2..$n = assets subtrees served directly by NGINX (configs, icons, favicon, manifest, ...)
apply_acl_nginx() {
    local tree="$1"; shift
    local assets

    sudo setfacl -m u:www-data:x "$tree"                                        # traverse only: pass through, no listing, no reading

    for assets in "$@"; do
        # assets directories — enter + list, existing and future
        sudo find "$assets" -type d -exec setfacl    -m u:www-data:rx {} +      # www-data: read existing dirs
        sudo find "$assets" -type d -exec setfacl -d -m u:www-data:rx {} +      # www-data: inherit read on new items

        # assets files — www-data read-only, owner/admin rw, never executable.
        # Entries AND mask are pinned explicitly: a bare `setfacl -m u:www-data:r`
        # would recalculate the mask to the union of all entries, resurfacing the
        # latent rwx inherited from the default ACL (files would show group rwx).
        sudo find "$assets" -type f -exec setfacl \
          -m u:www-data:r,u:${SERVICE_USER}:rw,g:${SERVICE_ADMIN}:rw,u::rw,g::rw,o::-,m::rw {} +
    done
}

# --- SERVICE_DIR/nginx + SERVICE_DIR/auth: read directly by NGINX -------------
# Created AFTER the apply_acl_tree pass so they inherit the default ACL + setgid
# group (service user/admin keep full access); www-data is then layered on top
# as a read-only named entry.
#
# auth/ holds the htpasswd file the vhost checks on EVERY request. It is the one
# credential store on this host that NGINX itself reads, so it gets the same
# read-only treatment as a static asset — and nothing more: no listing for
# others, never executable.
sudo mkdir -p "${SERVICE_DIR}/nginx" "${SERVICE_DIR}/auth"
sudo chown "${SERVICE_USER}:${SERVICE_ADMIN}" "${SERVICE_DIR}/nginx" "${SERVICE_DIR}/auth"
apply_acl_nginx "${SERVICE_DIR}" "${SERVICE_DIR}/nginx" "${SERVICE_DIR}/auth"

🛡️ Rootless Podman Defaults

Configure rootless Podman per-user configuration directory

# configure rootless Podman defaults for this service user
# - store per-user Podman configuration under ~/.config/containers
sudo -u "${SERVICE_USER}" mkdir -p "${SERVICE_HOME}/.config/containers/systemd"

Configure rootless Podman defaults

# configure rootless Podman defaults for this service user
# - use k8s-file logging for easier log rotation and inspection
sudo -u "${SERVICE_USER}" tee "${SERVICE_HOME}/.config/containers/containers.conf" >/dev/null <<EOF
[containers]
log_driver = "k8s-file"

[engine]
healthcheck_events=false
EOF

🔧 Service Configuration

Setup Registry Parameters

Before deploying, you need to define a few environment variables that will be used throughout the setup process.

• HOST_PORT: external port used by NGINX to route traffic to the service

NGINX Proxy Configuration

###################################################################################
# NGINX Proxy Configuration
###################################################################################
HOST_PORT=10090

Create Environment files

registry.env

sudo -u "${SERVICE_USER}" tee "${SERVICE_DIR}/registry.env" >/dev/null <<EOF
###################################################################################
# NGINX Proxy Configuration
###################################################################################
HOST_PORT=${HOST_PORT}

###################################################################################
# Registry Configuration
###################################################################################
# distribution v3 (alpine, busybox wget, config at /etc/distribution/config.yml).
# Pin an exact version: a floating tag is resolved from THIS user's image store
# and only moves on \`podmanctl --upgrade\`, so it drifts silently otherwise.
REGISTRY_VERSION=3.1.1
# No TZ here, unlike the other services: nothing would consume it. The container
# gets no EnvironmentFile (see quadlet/registry-server.container), and the registry
# logs through the journal, which stamps its own local time.
EOF

Never hand this file to the container

The quadlet deliberately has no EnvironmentFile= in its [Container] section. Quadlet would turn it into podman run --env-file, and distribution reads every REGISTRY_* variable as a configuration override — REGISTRY_VERSION lands on the config's own top-level version: 0.1 field and the registry exits before it binds: parsing environment variable REGISTRY_VERSION: strconv.ParseUint: parsing "1.1": invalid syntax The [Service] EnvironmentFile= is a different mechanism (systemd expanding ${REGISTRY_VERSION} and ${HOST_PORT} in the generated ExecStart) and must stay.

Keep the .env files

All the secret informations will be stored in the .env files.

Create the push credentials

# htpasswd lives in apache2-utils; -B forces bcrypt (NGINX supports it, and the
# registry's own default of APR1 is MD5-based).
sudo apt install -y apache2-utils

# one account per person or machine that pushes images; -c creates the file,
# OMIT it when adding a second account or you truncate the first.
sudo -u "${SERVICE_USER}" htpasswd -Bc "${SERVICE_DIR}/auth/htpasswd" <your-username>

The service users never use these credentials: they pull from 127.0.0.1:10090, which does not pass through NGINX.

Let the service users pull over the loopback

# Rootless Podman refuses a plain-HTTP registry unless it is declared insecure.
# /etc/containers/registries.conf is the SYSTEM default every rootless user reads
# before its own config, so this one entry covers every service user at once.
sudo tee -a /etc/containers/registries.conf >/dev/null <<'EOF'

[[registry]]
location = "127.0.0.1:10090"
insecure = true
EOF

Plain HTTP is acceptable here and only here: the traffic never leaves the loopback interface. Anything arriving from outside the host goes through NGINX with TLS and authentication.

🧩 Quadlet Service

Install the Quadlet units

# the units live in the repository under quadlet/ — link them into the service
# user's systemd directory (the generator and podmanctl read them there).
# Re-runnable: every existing link into quadlet/ is dropped first — stale or
# dangling ones included — then the current set is linked, replacing a plain
# file of the same name if one is left over.
# find runs from / : GNU find returns to its starting directory when done, and a
# directory the service user cannot enter (the admin's home, mode 700) fails it — AFTER
# the delete, so a `set -e` shell would stop before relinking
(cd / && sudo -u "${SERVICE_USER}" find "${SERVICE_HOME}/.config/containers/systemd" -maxdepth 1 -type l -lname "${SERVICE_DIR}/quadlet/*" -delete)
sudo -u "${SERVICE_USER}" ln -sf "${SERVICE_DIR}"/quadlet/* "${SERVICE_HOME}/.config/containers/systemd/"

Symlinks, so a pull that edits a unit is live at the next systemctl --user daemon-reload (podmanctl --update reloads after its pull). A pull that adds, renames or removes a unit needs the two lines again, then the reload — quadlet changes are rare and deliberate.

Quadlet units (reference)

The files under quadlet/, exactly as the links above expose them to systemd:

registry.networkregistry-server.container

[Unit]
Description=registry network

[Network]
# A dedicated podman network, like every other service here. The registry talks
# to nothing else, so this exists for isolation rather than for DNS.
NetworkName=registry

[Install]
WantedBy=default.target

[Unit]
Description=registry server (fleet image registry)
Requires=registry-network.service
After=registry-network.service
# give up after 6 failed starts within 30 min (podmanctl --restart clears it)
StartLimitIntervalSec=1800
StartLimitBurst=6

[Container]
# NO EnvironmentFile= here, unlike every other service on this host. Quadlet turns
# it into `podman run --env-file`, which injects the WHOLE file into the container
# — and distribution reads every REGISTRY_* variable as a config override. Our own
# REGISTRY_VERSION then lands on the config's top-level `version: 0.1` field and
# the registry exits before it binds:
#   configuration error: parsing environment variable REGISTRY_VERSION:
#   strconv.ParseUint: parsing "1.1": invalid syntax
# ${REGISTRY_VERSION} and ${HOST_PORT} below are expanded by systemd from the
# [Service] EnvironmentFile=, which is a different mechanism and stays.
Image=docker.io/library/registry:${REGISTRY_VERSION}
ContainerName=registry-server
Network=registry.network

# The upstream image declares no USER and would run as root. Pin it to 1000 like
# every other cloned app; the data dir is pre-chowned to 1000's mapped subuid in
# the permissions step, so a plain :rw mount works.
User=1000:1000

# The shipped config.yml sets `log.level: debug` and `environment: development`
# — fine for a demo, far too chatty for a unit that journals every blob request.
Environment=REGISTRY_LOG_LEVEL=info
# Blob deletion must be on or `registry garbage-collect` marks blobs and reclaims
# nothing. It is already the default in the image's config.yml; stated here so a
# future image default cannot silently turn the GC unit into a no-op.
Environment=REGISTRY_STORAGE_DELETE_ENABLED=true

# Plain bind mount over the image's declared VOLUME /var/lib/registry, so podman
# does not spawn a stray anonymous volume. Unlike postgres there is no subdir to
# work around: the storage root IS the declared volume.
Volume=/media/ssd/podman/registry/data/registry:/var/lib/registry:rw

# Loopback only. Pushes from a dev machine arrive through the nginx vhost (TLS +
# basic auth); the service users pull straight from here, which is why their
# DEPLOY_REGISTRY names 127.0.0.1:${HOST_PORT} and not the public hostname.
PublishPort=127.0.0.1:${HOST_PORT}:5000

# The image is alpine-based with busybox wget and no curl. `/v2/` is the registry
# API root and answers 200 once storage is reachable.
HealthCmd=wget -q -O /dev/null http://127.0.0.1:5000/v2/
HealthStartPeriod=30s
HealthInterval=60s
HealthTimeout=3s
HealthRetries=3
HealthOnFailure=kill

[Service]
EnvironmentFile=/media/ssd/podman/registry/registry.env
UMask=0007
SuccessExitStatus=143

Restart=on-failure
RestartSec=10
RestartSteps=5
RestartMaxDelaySec=300
TimeoutStartSec=600

[Install]
WantedBy=default.target

Install the garbage collection units

# the gc units live in the repository under systemd/ — link them into the service
# user's PLAIN systemd directory, NOT the Quadlet one: podmanctl starts every
# *.container it finds there, so a run-to-completion unit beside the server would
# garbage-collect on every `podmanctl --start registry`.
# Re-runnable: every existing link into systemd/ is dropped first — stale or
# dangling ones included — then the current set is linked, replacing a plain
# file of the same name if one is left over (a copy from an earlier install).
# find runs from / for the same reason as in the Quadlet block above
sudo -u "${SERVICE_USER}" mkdir -p "${SERVICE_HOME}/.config/systemd/user"
(cd / && sudo -u "${SERVICE_USER}" find "${SERVICE_HOME}/.config/systemd/user" -maxdepth 1 -type l -lname "${SERVICE_DIR}/systemd/*" -delete)
sudo -u "${SERVICE_USER}" ln -sf "${SERVICE_DIR}"/systemd/* "${SERVICE_HOME}/.config/systemd/user/"

Symlinks, so a pull that edits a unit is live at the next systemctl --user daemon-reload (podmanctl --update reloads after its pull). A pull that adds, renames or removes a unit needs these lines again, then the reload. systemctl --user disable registry-gc.timer removes the link along with the enablement: run the block again before enabling it anew.

Garbage collection units (reference)

The files under systemd/, exactly as the links above expose them to systemd:

registry-gc.serviceregistry-gc.timer

[Unit]
# Garbage-collect blobs no manifest references any more.
#
# NOT a Quadlet unit, and deliberately NOT in ~/.config/containers/systemd/:
# podmanctl discovers every *.container there and starts it, so a
# run-to-completion unit living beside the server would garbage-collect on every
# `podmanctl --start registry`. This is a plain user unit in
# ~/.config/systemd/user/, which podmanctl never looks at.
#
# The registry is stopped for the sweep. GC marks, then deletes: a push that
# lands between the two phases can have its freshly-uploaded blobs deleted as
# unreferenced. Upstream's own guidance is to take writes offline for it.
Description=registry garbage collection
After=registry-server.service

[Service]
Type=oneshot
EnvironmentFile=/media/ssd/podman/registry/registry.env
ExecStartPre=/usr/bin/systemctl --user stop registry-server.service
# The config path moved in distribution v3: /etc/distribution/config.yml, not the
# v2 /etc/docker/registry/config.yml. --delete-untagged also drops manifests no
# tag points at: podmanctl removes the superseded tag from the SERVER's image
# store after each deploy, but the registry keeps every tag ever pushed until
# this runs.
ExecStart=/usr/bin/podman run --rm \
    --volume /media/ssd/podman/registry/data/registry:/var/lib/registry:rw \
    --user 1000:1000 \
    docker.io/library/registry:${REGISTRY_VERSION} \
    garbage-collect --delete-untagged /etc/distribution/config.yml
# Bring the registry back whether the sweep succeeded or not — a failed GC costs
# disk, a registry left down costs every deploy.
ExecStopPost=/usr/bin/systemctl --user start registry-server.service
TimeoutStartSec=1800

[Unit]
Description=weekly registry garbage collection

[Timer]
# Sunday 04:00, well clear of any deploy: the sweep stops the registry, and a
# push landing in that window fails. Persistent= so a sweep missed while the
# host was off runs at the next boot instead of being skipped for a week.
OnCalendar=Sun 04:00
Persistent=true
RandomizedDelaySec=900

[Install]
WantedBy=timers.target

---

▶️ Start the Registry

Enable and Start Quadlet Services

# open an interactive shell as the service user
sudo -iu "${SERVICE_USER}"

# reload systemd user units
systemctl --user daemon-reload

# start Podman Quadlet services
systemctl --user start registry-server.service

# verify service status
systemctl --user status registry-server.service

# the API root answers 200 once storage is reachable
curl -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:10090/v2/

To follow the service logs in real time, run sudo journalctl _UID=$(id -u ${SERVICE_USER}) -f from the Debian account.

Enable the weekly garbage collection

# as the service user - the units are linked by "Install the garbage collection units"
systemctl --user daemon-reload
systemctl --user enable --now registry-gc.timer
systemctl --user list-timers registry-gc.timer

Nothing else reclaims registry storage: podmanctl drops the superseded image from the server's image store after each deploy, but the registry keeps every tag ever pushed until this sweep runs.

---

🚀 Deploy the Registry

Install NGINX

NGINX needs to be installed, follow the NGINX section.

Configure NGINX

NGINX needs to be configured using a file in /etc/nginx/sites-enabled directory.
This configuration file specify the documentation path:

/etc/nginx/sites-enabled/registry.domain.frActivate website

# Per-client rate-limit bucket for the registry's auth probe. Must live in http{}
# context, so it sits outside the server{} block. $limit_key is the shared key
# from nginx.conf: the address for IPv4, the /64 prefix for IPv6 — keying on a
# full IPv6 address would let one subscriber prefix mint 2^64 empty buckets and
# bypass the limit entirely.
#
# Only `GET /v2/` is limited, not the whole API. That endpoint is where a client
# discovers auth and where a password guesser would hammer; every other request
# is part of a real push or pull, and a single push of a Rust image makes dozens
# of them. Rate-limiting the whole vhost would throttle deploys instead.
limit_req_zone $limit_key zone=registry_auth:10m rate=10r/m;

# The registry container, published on the loopback by its Quadlet unit. The
# keepalive pool matters less here than on the app sites — a push is a handful of
# long transfers, not many small requests — but a blob upload that has to open a
# fresh connection through rootless podman's userspace forwarder pays for it on
# every layer. Conservative pool and timeout: this upstream's idle behaviour is
# not ours to control, and a connection held past its close answers the request
# that races it with a 502.
upstream registry_app {
  server 127.0.0.1:10090;
  keepalive 16;
  keepalive_requests 1000;
  keepalive_timeout 15s;
}

server {
  server_name registry.domain.fr;

  # The security header quartet (HSTS, nosniff, X-Frame-Options, Referrer-Policy)
  # is inherited from the http{} block in nginx.conf — this file declares none.
  # WARNING: add_header does NOT accumulate. A single add_header added here, or
  # in any location{} below, replaces that whole inherited set for that scope —
  # silently, with no warning from `nginx -t`. If one is ever needed, restate all
  # four alongside it.

  # setup 404 error_page
  error_page 404 /404.html;
  include snippets/error-404.conf;

  # show maintenance page when backend is down
  error_page 502 503 504 /maintenance.html;
  include snippets/error-maintenance.conf;

  # Image layers are hundreds of MB and the registry streams them; 0 means "no
  # ceiling". This is the one vhost on the box where that is correct, and it is
  # only safe because auth below runs BEFORE any body is read.
  client_max_body_size 0;

  # Stream the body straight to the registry instead of spooling it to the
  # proxy's disk first. With request buffering on (the default), NGINX reads the
  # WHOLE layer into client_body_temp before opening the upstream connection —
  # gigabytes onto the SSD per push, and a full disk takes down every site on the
  # box, not just this one.
  proxy_request_buffering off;

  # The auth probe: what a client hits first, and what a password guesser would
  # hammer. Exact match, so it wins over `location /v2/`.
  location = /v2/ {
    limit_req zone=registry_auth burst=5 nodelay;
    limit_req_status 429;

    auth_basic "fleet registry";
    auth_basic_user_file /media/ssd/podman/registry/auth/htpasswd;

    proxy_pass http://registry_app;
    proxy_http_version 1.1;
    include proxy_params;
    # Empty Connection header: keeps the pooled upstream connection alive.
    proxy_set_header Connection "";
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Port $server_port;
  }

  # The registry API. Everything here is authenticated: this host publishes the
  # fleet's application images, so an anonymous reader could pull production
  # binaries and an anonymous writer could replace them. The service users do NOT
  # come through here — they pull from 127.0.0.1:10090 directly, which is why
  # their DEPLOY_REGISTRY is the loopback address and the dev machines' is this
  # hostname.
  location /v2/ {
    auth_basic "fleet registry";
    auth_basic_user_file /media/ssd/podman/registry/auth/htpasswd;

    proxy_pass http://registry_app;
    proxy_http_version 1.1;
    include proxy_params;
    proxy_set_header Connection "";
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Port $server_port;

    # A slow uplink pushing a 300 MB layer needs far longer than the default 60s,
    # and a stalled push that nginx gives up on leaves a partial upload behind.
    proxy_read_timeout 900s;
    proxy_send_timeout 900s;
    client_body_timeout 900s;

    # Layer transfers are single large streams; buffering them through nginx adds
    # latency and disk churn for nothing.
    proxy_buffering off;
    proxy_redirect off;
  }

  # Nothing else is served. The registry has no UI, and a bare GET / returning
  # the maintenance page is a clearer answer than proxying to a 404 from the API.
  location / {
    return 404;
  }

}

# restart nginx
sudo nginx -t && sudo service nginx restart

Replace registry.domain.fr by the name of your website.

Activate HTTPS

To activate HTTPS protocol, follow the Let's Encrypt section.

---

⚙️ Configure the Registry

Verify the vhost refuses anonymous access

This host stores production binaries for every other service. An anonymous reader could pull them; an anonymous writer could replace them. Before pushing anything, confirm from another machine that both are refused:

# must answer 401, NOT 200
curl -s -o /dev/null -w '%{http_code}\n' https://registry.domain.fr/v2/

# must answer 404: nothing outside the registry API is served
curl -s -o /dev/null -w '%{http_code}\n' https://registry.domain.fr/

A 200 on the first means auth_basic is missing from a location block — fix it before any image is pushed.

Smoke-test the round trip

Prove the registry works with a tiny image before spending a full application build on it.

From a dev machine (push)From the server (pull)

docker login registry.domain.fr

# --platform matters: the server is aarch64, and a dev machine's cached
# alpine:latest is whatever IT runs. Pushing an amd64 image makes the pull
# below fail on architecture, for a reason that looks nothing like the cause.
docker pull --platform linux/arm64 alpine
docker tag alpine registry.domain.fr/smoke:1
docker push registry.domain.fr/smoke:1

# A LOGIN shell (-i), not `sudo -u`: sudo keeps the caller's working directory,
# and a service user cannot chdir into another service's tree - rootless podman
# then fails with "cannot chdir to ...: Permission denied", which has nothing to
# do with the registry. -i also gives podman the right HOME for its image store.
sudo -iu claudenames_svc
podman pull 127.0.0.1:10090/smoke:1
podman rmi 127.0.0.1:10090/smoke:1
exit

A push that succeeds and a pull that follows it means TLS, authentication, storage and the loopback entry are all correct.

Migrate an app to published images

One app at a time, and the order matters — the server key goes last, because from the moment it is set a push with no matching image is refused rather than built.

# 1. dev machine: name the registry, then build and publish WITHOUT deploying
#    (DEPLOY_REGISTRY=registry.domain.fr in the app's own env file)
./deploy.ps1 -NoGitPush

# 2. server: point the app at the loopback address
echo 'DEPLOY_REGISTRY=127.0.0.1:10090' | sudo -u claudenames_svc tee -a \
    /media/ssd/podman/claudenames/claudenames.env

# 3. server: deploy by hand - it should pull the image and build nothing
podmanctl --update claudenames

To roll back, delete the DEPLOY_REGISTRY line from the server's env file: the next update rebuilds locally exactly as before.

Reclaiming space by hand

# as the service user - stops the registry for the sweep, then restarts it
systemctl --user start registry-gc.service
journalctl --user -u registry-gc.service -n 30

Garbage collection marks, then deletes. A push landing between the two phases can have its fresh blobs removed as unreferenced, which is why the sweep takes the registry offline rather than running alongside it.

---

⬆️ Updating the Registry

The Registry deploys by push: ./deploy.ps1, run from the repository's clone on a dev machine, pushes the commit — there is nothing to build there. The push fires the bare repo's hook, which calls podmanctl --deploy registry <short-sha>: it acknowledges the push, then deploys detached: a pull, then a restart on the stack's pinned upstream images (nothing is built), and an nginx reload once nginx -t accepts a changed nginx/ vhost. Every deploy ends with a receipt in the bare repo, refs/deploy/running or refs/deploy/failed, which is how deploy.ps1 learns the outcome. Follow a deploy with sudo tail -f /var/log/registry-deploy.log.

A deploy restarts the registry itself, and while it is down no own project can publish or deploy: run it when no deploy.ps1 is running.

🚀 Deploying a change

From the repository's clone on a dev machine:

git add -u && git commit -m "..."
./deploy.ps1

It refuses a dirty tree, or a branch that has diverged from the server, before pushing anything. After the push it waits until the receipt names the commit (-NoWait skips the wait): exit 0 once it runs, exit 4 when its deploy failed or is still queued, naming the log. With nothing left to push it reads the receipt instead: the server runs HEAD → exit 3 (skipped); no receipt → exit 3 marked unconfirmed, with the reason; HEAD failed or never deployed → exit 4, naming the podmanctl --deploy left to run there.

Every push restarts the stack — a commit that only touches CLAUDE.md included.

🪝 One-time setup — hook and clone wiring

The deploy entry point is central and installed once for the whole fleet — see Podman — Push deploys for podmanctl --deploy and its single sudoers rule. Only the two steps below are per-repository.

1. post-receive hook (bare repo)2. Wire the clone for deploys

# install into the bare repo's hooks/ — fires on every push
# ${SERVICE_NAME} expands now; the rest stays literal (\$…) for git to expand
sudo tee /media/ssd/git/podman/${SERVICE_NAME}.git/hooks/post-receive >/dev/null <<EOF
#!/bin/sh
set -eu
DEPLOY_BRANCH=refs/heads/master
while read -r oldrev newrev refname; do
  [ "\$refname" = "\$DEPLOY_BRANCH" ] || continue          # only the deploy branch
  sudo /usr/local/bin/podmanctl --deploy ${SERVICE_NAME} "\$(git rev-parse --short "\$newrev")" </dev/null
done
EOF
# normalize the hook ownership
sudo chown www-data:debian /media/ssd/git/podman/${SERVICE_NAME}.git/hooks/post-receive
sudo chmod 0750            /media/ssd/git/podman/${SERVICE_NAME}.git/hooks/post-receive

# git seeded hooks/ with executable *.sample files — inert, but strip the bit
sudo chmod -x /media/ssd/git/podman/${SERVICE_NAME}.git/hooks/*.sample

The commit comes from the just-pushed ref (the bare repo already holds it) and only names the deploy in its log. Nothing else is baked in: podmanctl derives the service directory, user, admin group, env file and log path from the name it is given.

# deploys pull non-interactively as root: pull from the co-located bare repo over a
# LOCAL path (the HTTPS origin needs a credential prompt + TTY, which a hook lacks)
git -C ${SERVICE_DIR} remote set-url origin /media/ssd/git/podman/${SERVICE_NAME}.git

# trust both repos for root so the root-run `git pull` doesn't refuse with
# "detected dubious ownership"; --system applies whatever HOME the hook runs with
sudo git config --system --add safe.directory ${SERVICE_DIR}
sudo git config --system --add safe.directory /media/ssd/git/podman/${SERVICE_NAME}.git

Without the local origin the pull dies with could not read Username for 'https://…'; without safe.directory it dies with detected dubious ownership. Either way the worker aborts before podmanctl --update and leaves the service running on its previous build.

🧱 Upgrading the Registry

The images are pinned in gitignored env files, edited on the server — a push never moves one: REGISTRY_VERSION in registry.env. Moving a pin is a deliberate step:

podmanctl --list --check            # UPDATE column shows when an upstream image moved
podmanctl --upgrade registry        # re-pull the images, rebuild, restart, prune the old ones

--update never re-pulls a tag it already holds, so a floating tag (latest, release) only moves with --upgrade; a bumped pin is pulled by either.