Files
ai-app/server/service
T
iris a8b6c13b01 Report a crashed service as failed, which OpenRC was never telling us
The OpenRC branch of this script had never run anywhere. A guest built to
reproduce the host says it was wrong in the way the `failed` state exists
to prevent: a service that fell over reported `stopped`, which reads as a
decision somebody made.

Two causes, both measured rather than reasoned about.

`rc-service status` prints `* status: crashed` to **stderr**. The check
was `status 2>/dev/null | grep -qw crashed`, which discards precisely the
word it is searching for, finds nothing, and falls through to `stopped`.
The old comment argued for reading the word rather than the exit code, and
that argument was sound except that the code turns out to be specific
rather than merely non-zero.

So it now reads the code, which says more than the text did: 0 started,
3 stopped, 32 crashed, and 1 for every way the question cannot be
answered -- an unknown service, XDG_RUNTIME_DIR unset, or a user softlevel
that was never initialised. That last group is a real state and not one of
the other three, so it exits non-zero and says so instead of guessing.

`set -e` is the second cause, found by running the first fix: every answer
except "running" is a non-zero exit, so a bare invocation killed the
script before the code could be looked at. It prints nothing and exits 3,
which is indistinguishable from a crash of this script itself.

Verified in the guest, all four states: not-installed, stopped, failed
after a real crash, and a non-zero exit with nothing on stdout when the
softlevel is removed. Before the fix the crashed case printed `stopped`.
2026-08-28 18:55:05 -04:00

269 lines
9.6 KiB
Bash
Executable File

#!/bin/sh
# Installs and controls ai-server as a user service, for Dev Updater to
# drive from the phone -- and for you to drive by hand, which is the same
# thing.
#
# Declared as the Server component of this checkout in `.dev-updater.ron`
# at the repository root, so the card drives it like any other project's.
# One button is worth knowing about before pressing it: on a phone that
# reaches ai-server through the tunnel rather than through Dev Updater,
# Stop leaves this server down until someone starts it again here.
#
# ./service install | uninstall | start | stop | restart | status | logs
#
# `status` prints exactly one of `running`, `stopped`, `failed` or
# `not-installed`
# and exits 0. Anything else it prints, or any non-zero exit, means it
# could not tell -- which the card shows as "couldn't check" rather than
# as a service that is down.
#
# Nothing here ever prompts. Dev Updater runs this with stdin closed and no
# terminal, so a `sudo` password prompt would not fail, it would hang until
# the timeout with the card stuck mid-action. Anything needing root exits
# with a message telling you to run it yourself once instead.
#
# User services on purpose: a system unit needs root to install, and a
# build machine's own account is where a dev server belongs. Note that a
# user service stops at logout unless lingering is enabled
# (`loginctl enable-linger $USER`), which you want for this one: the phone
# reaches this server whether or not anyone is logged in at the desk.
set -eu
SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd)
NAME=ai-server
BINARY="$SCRIPT_DIR/target/release/$NAME"
# Where this service's output goes, and the one generation kept behind it.
#
# Under $XDG_DATA_HOME rather than the checkout: a log is generated data,
# it outlives any one build, and the repository is shared with a machine
# that should not be able to read it. Rotated on start rather than by size
# or age, so what is kept is exactly "this run and the one before" -- which
# is the pair worth having after a crash and a restart, and is the reason
# the file is not simply appended to forever.
LOG_DIR="${XDG_DATA_HOME:-$HOME/.local/share}/$NAME"
LOG="$LOG_DIR/$NAME.log"
LOG_PREVIOUS="$LOG.1"
# Which init system is here, decided by asking rather than by looking for a
# binary: a machine can carry both, and an OpenRC older than 0.60 has
# rc-service but no --user at all.
detect() {
if command -v systemctl >/dev/null 2>&1 &&
systemctl --user show-environment >/dev/null 2>&1; then
echo systemd
elif command -v rc-service >/dev/null 2>&1 &&
rc-service --user --help >/dev/null 2>&1; then
echo openrc
else
echo none
fi
}
MANAGER=$(detect)
if [ "$MANAGER" = none ]; then
echo "No user-service manager here: this needs systemd with a user bus," >&2
echo "or OpenRC 0.60+ (older ones have no --user). Run $NAME by hand." >&2
exit 1
fi
# OpenRC keeps user-service state under XDG_RUNTIME_DIR and refuses without
# it. Worth saying plainly: from a server started outside a login session
# it can be unset, and the failure otherwise reads as a broken service
# rather than a missing variable.
if [ "$MANAGER" = openrc ] && [ -z "${XDG_RUNTIME_DIR:-}" ]; then
echo "XDG_RUNTIME_DIR is unset, and OpenRC stores user-service state in it." >&2
echo "Set it at login (elogind or pam_xdg) and try again." >&2
exit 1
fi
SYSTEMD_UNIT="${XDG_CONFIG_HOME:-$HOME/.config}/systemd/user/$NAME.service"
OPENRC_UNIT="${XDG_CONFIG_HOME:-$HOME/.config}/rc/init.d/$NAME"
installed() {
case "$MANAGER" in
systemd) [ -f "$SYSTEMD_UNIT" ] ;;
openrc) [ -f "$OPENRC_UNIT" ] ;;
esac
}
# What OpenRC's `status` exit code means. Measured on OpenRC 0.63.3 in a
# guest built for the purpose, not read from documentation:
#
# 0 started 3 stopped 32 crashed
# 1 could not find out
#
# 1 covers every way the question cannot be answered -- an unknown
# service, XDG_RUNTIME_DIR unset, or a user softlevel that was never
# initialised ("openrc did not boot this system"). Distinguishing it is
# the whole reason to read the code rather than the text.
#
# And the text must not be read. OpenRC prints `* status: crashed` to
# **stderr**, so the obvious `status 2>/dev/null | grep -qw crashed`
# throws away precisely the word it is searching for, finds nothing, and
# falls through to `stopped` -- reporting a service that fell over as one
# somebody chose to stop. That was this script's bug until it was
# measured, and the same shape is worth checking wherever a status is
# parsed rather than counted.
OPENRC_STARTED=0
OPENRC_STOPPED=3
OPENRC_CRASHED=32
require_binary() {
[ -x "$BINARY" ] && return 0
echo "No built server at $BINARY -- build it first." >&2
exit 1
}
do_install() {
require_binary
mkdir -p "$LOG_DIR"
case "$MANAGER" in
systemd)
mkdir -p "$(dirname "$SYSTEMD_UNIT")"
cat > "$SYSTEMD_UNIT" <<UNIT
[Unit]
Description=$NAME (installed by $SCRIPT_DIR/service)
[Service]
ExecStart=$BINARY
Restart=on-failure
WorkingDirectory=$SCRIPT_DIR
# Appended rather than truncated: the unit is not the thing that decides
# when a log starts over, since the start subcommand rotates. Restarting
# inside one run (Restart=on-failure) then keeps the whole story rather
# than erasing the reason. Needs systemd 240+.
#
# No backticks in this heredoc: the delimiter is unquoted so that \$LOG
# expands, which means a backtick would run as command substitution while
# the unit is being written.
StandardOutput=append:$LOG
StandardError=append:$LOG
[Install]
WantedBy=default.target
UNIT
systemctl --user daemon-reload
systemctl --user enable "$NAME" >/dev/null
;;
openrc)
mkdir -p "$(dirname "$OPENRC_UNIT")"
cat > "$OPENRC_UNIT" <<UNIT
#!/sbin/openrc-run
name="$NAME"
description="$NAME (installed by $SCRIPT_DIR/service)"
command="$BINARY"
command_background=true
directory="$SCRIPT_DIR"
pidfile="\${XDG_RUNTIME_DIR}/$NAME.pid"
# command_background discards output otherwise, which is why a crash left
# nothing to read.
output_log="$LOG"
error_log="$LOG"
UNIT
chmod +x "$OPENRC_UNIT"
rc-update --user add "$NAME" >/dev/null
;;
esac
}
do_uninstall() {
installed || return 0
case "$MANAGER" in
systemd)
systemctl --user disable --now "$NAME" >/dev/null 2>&1 || true
rm -f "$SYSTEMD_UNIT"
systemctl --user daemon-reload
;;
openrc)
rc-service --user "$NAME" stop >/dev/null 2>&1 || true
rc-update --user del "$NAME" >/dev/null 2>&1 || true
rm -f "$OPENRC_UNIT"
;;
esac
}
# Keeps the finished run and starts a fresh file for the next one.
rotate() {
mkdir -p "$LOG_DIR"
[ -f "$LOG" ] && mv -f "$LOG" "$LOG_PREVIOUS"
: > "$LOG"
}
control() {
installed || { echo "$NAME is not installed" >&2; exit 1; }
case "$MANAGER" in
systemd) systemctl --user "$1" "$NAME" ;;
openrc) rc-service --user "$NAME" "$1" ;;
esac
}
case "${1:-}" in
install) do_install ;;
uninstall) do_uninstall ;;
start | restart)
# Rotated before the manager is asked, so the file the service
# opens is the new one.
rotate
control "$1"
;;
stop) control stop ;;
logs)
# One path per line, newest first, and nothing else: the caller
# wants somewhere to read from, not a formatted report. Printing
# nothing at all is the answer for a service with no log yet,
# which reads as "not supported here" and costs nobody anything.
[ -f "$LOG" ] && echo "$LOG"
[ -f "$LOG_PREVIOUS" ] && echo "$LOG_PREVIOUS"
exit 0
;;
status)
if ! installed; then
echo not-installed
exit 0
fi
case "$MANAGER" in
systemd)
if systemctl --user --quiet is-active "$NAME"; then
echo running
# Before `stopped`, because a service that fell over
# satisfies neither of the other two and would otherwise
# be reported as a state somebody chose.
elif systemctl --user --quiet is-failed "$NAME"; then
echo failed
else
echo stopped
fi
;;
openrc)
# `|| code=$?` rather than a bare call: `set -e` is on, and
# every answer except "running" is a non-zero exit, so a
# plain invocation kills the script before it can say what
# the code meant.
code=0
rc-service --user "$NAME" status >/dev/null 2>&1 || code=$?
case $code in
"$OPENRC_STARTED") echo running ;;
"$OPENRC_CRASHED") echo failed ;;
"$OPENRC_STOPPED") echo stopped ;;
# Anything else is "could not find out", which is a
# real state and not one of the other three. Saying so
# costs a non-zero exit and buys the caller the truth;
# guessing `stopped` here is what the contract's
# `failed` state exists to stop.
*)
echo "could not ask OpenRC about $NAME -- is" \
"XDG_RUNTIME_DIR set and the user softlevel" \
"initialised?" >&2
exit 1
;;
esac
;;
esac
;;
*)
echo "usage: $0 install|uninstall|start|stop|restart|status|logs" >&2
exit 2
;;
esac