mptyl/ThothII
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix: harden Pi lifecycle recovery

Browse Source
This commit is contained in:
marcopan
2026-08-04 20:28:09 +02:00
parent 5b3ce93e31
commit 5ba2821a1b
24 changed files with 1764 additions and 384 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/contracts/thothctl-pi.md
+74 -35
View File
@@ -1,75 +1,114 @@
# `thothctl pi` lifecycle contract
`thothctl` is the only component that drives Docker lifecycle operations. The `core` container
does not mount a Docker socket and Pi is never updated in a running container.
does not mount a Docker socket, and Pi is never updated in a running container.
## Read-only operations
## Inspection and configuration
```text
thothctl --installation /absolute/path/thothii-installation.yaml pi status
thothctl --installation /absolute/path/thothii-installation.yaml pi doctor
thothctl --installation /absolute/path/thothii-installation.yaml pi test
thothctl --installation /absolute/path/thothii-installation.yaml pi logs [--follow]
thothctl --installation /absolute/path/thothii-installation.yaml pi logs
thothctl --installation /absolute/path/thothii-installation.yaml pi configure
```
`status` executes the image-bundled `pi --version`. `doctor` requires the rendered `core` image,
the external `THT_LLM_URL` contract, writable `/home/thoth/.pi`, the read-only Pi auth file, and
private `/health`. `test` additionally reads private `/models` and `/settings`; this temporary
composite smoke is replaced by the Pi Management API in Task 8. `logs` is core-only and uses the
same credential redaction as every other `thothctl` diagnostic.
`status` executes the image-bundled `pi --version`. `doctor` and `test` require a healthy core,
a successful Pi smoke, valid settings, and an exact selected provider/model pair from the
backend's available model entries. `pi check` remains an alias for `pi test`. Logs are always a
bounded, sanitized 200-line snapshot; there is no follow mode.
`pi check` is an alias for `pi test` for operational scripts.
On a TTY, `pi configure` presents numbered provider, model, and thinking choices. Providers and
models come from the backend's closed model list, and the model choices are restricted to the
selected provider. In non-interactive use, all choices must be explicit:
```text
thothctl --installation /absolute/path/thothii-installation.yaml pi configure \
--provider zai --model glm-5.2 --thinking medium
```
The helper snapshots the previous settings, applies the new values atomically, and verifies the
readback and rendered-configuration digest. A helper, readback, or digest failure triggers an
attempted restore followed by another readback. The command reports the actual host path from
`PI_AUTH_FILE`; credentials remain in that protected host file and must never be passed as flags.
## Updating Pi
An update always specifies a pinned Pi version and an explicit confirmation:
Every update requires a pinned version, an explicit source, and confirmation:
```text
thothctl --installation /absolute/path/thothii-installation.yaml pi update \
--version 0.81.0 --source build --yes
```
`--source build` rebuilds only `core` using `PI_VERSION=<version>`. A pulled source must be a
digest-pinned image; tags are rejected:
```text
thothctl --installation /absolute/path/thothii-installation.yaml pi update \
--version 0.81.0 --source pull \
--image registry.example.invalid/thothii-core@sha256:<64-lowercase-hex-digits> --yes
```
Before changing anything, the command validates the rendered Compose configuration, Pi auth/state
preconditions, health, current version, active sessions, the current image ID, named-volume set,
and a digest of the rendered non-secret configuration. It therefore keeps the exact installation
Compose files and environment, including the external `THT_LLM_URL` endpoint, when it recreates
only `core` with `--no-deps --force-recreate`. It never recreates `frontend` and never uses volume
replacement flags.
`--source build` rebuilds only `core` with `PI_VERSION=<version>`. `--source pull` requires an
immutable digest reference; mutable tags, URL forms, and credential-bearing references are
rejected. `--source` is never inferred.
Before inventory, update activates the durable maintenance gate. Activation writes
`/data/settings/maintenance.json` in the mounted settings volume, closes admission, and waits for
all leases. A recreated candidate reads that marker at startup and therefore starts gated. The
loopback-only control endpoints cannot be reached through the frontend proxy and do not depend on
the configured authentication principal mode. Lost activation/deactivation responses are resolved
by querying gate status.
Open, unarchived sessions stop an update. After an operator has completed or otherwise drained
their work, `--drain` makes the command re-check that the session list is empty before continuing.
their work, `--drain` makes the command poll the authenticated bare-array
`GET /sessions?scope=all` response until no active sessions remain.
## Recovery and rollback
The configured `core.image` is never retagged or mutated. Each installation transaction creates
unique candidate and previous tags, including when two installations share a configured tag or
the configured image is digest-pinned. A temporary lifecycle-only Compose override selects those
tags for build, recreate, and rollback. Terminal success removes the override.
Before a candidate is built or pulled, the command atomically writes:
Only `core` is recreated, with `--no-deps --force-recreate`; `frontend` is not recreated and no
volume-replacement flags are used. Verification checks health, exact requested Pi version, the
provider/model/settings smoke, unchanged non-secret rendered configuration, and the complete
persistence-mount fingerprint.
## Recovery, rollback, and maintenance cleanup
Recovery state and lock diagnostics live under:
```text
<projectDirectory>/.thothctl/update-state.json
<projectDirectory>/.thothctl/update-state.json.lock.owner.json
```
The file is mode `0600` and records only the previous/candidate image references and IDs, named
volume names, requested version/source, rendered-configuration digest, phase, and timestamp. It
never contains credentials, endpoint values, secret paths, Compose output, or logs.
The recovery file is mode `0600` and records transaction-scoped image identities, mount
fingerprints, target version/source, configuration digest, phase, and timestamp. It contains no
credentials, endpoint values, secret paths, Compose output, or logs. A cross-platform OS advisory
file lock serializes lifecycle operations; a crashed owner releases the lock automatically. Owner
metadata is diagnostic only and cannot wedge acquisition if empty, partial, or stale.
After recreate, the command checks core health, the requested `pi --version`, the Pi/core smoke,
unchanged configuration digest, and unchanged named-volume set. Any failure after recreation
automatically retags and recreates the recorded previous image. A failed or interrupted operation
leaves the same metadata for explicit operator recovery:
Any post-candidate failure explicitly confirms or reactivates maintenance and rescans sessions
before compensation. Automatic rollback selects the transaction's previous image through the
lifecycle override and clears maintenance only after the previous image, configuration, mounts,
health, Pi smoke, and terminal recovery write are verified. Ambiguous compensation remains gated.
For an interrupted transaction, first run:
```text
thothctl --installation /absolute/path/thothii-installation.yaml pi rollback --yes
```
Both update and rollback require `--yes`; without it they exit `2` before invoking Docker. Invalid
arguments, a pending recovery, and active sessions also exit `2`. Docker or verification failures
exit nonzero with concise, redacted guidance. The original Docker child exit code is preserved for
direct read-only/log command failures.
Inspect and clean a stale durable gate with:
```text
thothctl --installation /absolute/path/thothii-installation.yaml pi maintenance status
thothctl --installation /absolute/path/thothii-installation.yaml pi maintenance recover --yes
```
`maintenance recover` refuses a pending transaction. For terminal or absent recovery state, it
removes a stale lifecycle override, verifies the running installation when the gate is active, and
only then removes the durable marker and reopens admission. If rollback or recovery fails, leave
the marker in place, preserve `update-state.json`, repair the reported Docker/configuration issue,
and rerun rollback or maintenance recovery.
Missing confirmation, invalid arguments, active sessions, and an interrupted transaction exit
`2`. Docker and verification failures exit nonzero with concise, redacted guidance. Direct
read-only/log commands preserve the original Docker child exit code.