docs: converge operator guidance on tht
This commit is contained in:
+55
-54
@@ -3,9 +3,10 @@
|
||||
Server authentication uses generic OIDC with the reverse proxy preserving the configured public
|
||||
origin and callback path. Follow the [OIDC guide](authentication-oidc.md), [Authentik guide](authentik.md)
|
||||
when applicable, and the [authentication acceptance matrix](../testing/authentication-manual-acceptance.md).
|
||||
The host authentication CLI is `tht`: Workspace Validate is static, `tht auth check` is live and
|
||||
non-interactive, `--interactive` adds device-flow identity validation, and Workspace Test is the
|
||||
aggregate live gate.
|
||||
The host authentication CLI is `tht`. Use `tht --installation <descriptor> workspace inspect
|
||||
--workspace <id> --json` for the active workspace snapshot, `tht --installation <descriptor>
|
||||
auth check` for live non-interactive authentication diagnosis, `auth check --interactive` for
|
||||
device-flow identity validation, and `tht ... doctor --json` for the aggregate installation gate.
|
||||
|
||||
This guide is for an installer with basic Linux administration and very basic Docker knowledge.
|
||||
It deploys the same Compose distribution used on a local PC: the mandatory application is exactly
|
||||
@@ -30,7 +31,7 @@ storage overlay and migration procedure before exposing a production installatio
|
||||
value belongs in Git, images, browser storage, environment values, rendered Compose, or logs.
|
||||
- The Git-backed workspace registry is the source of truth. Installation-local bindings identify
|
||||
endpoints and secret-file paths; they do not replace the reviewed Git workspace descriptors.
|
||||
- `thothctl` is the operator CLI for start, stop, status, health, logs, Pi lifecycle, drain, and
|
||||
- `tht` is the operator CLI for start, stop, status, health, logs, Pi lifecycle, drain, and
|
||||
rollback. Raw Compose lifecycle commands bypass installation state and are unsupported.
|
||||
|
||||
Read [server workspace-registry installation](server-workspace-registry.md),
|
||||
@@ -51,7 +52,7 @@ sudo useradd --system --uid 10001 --user-group --home-dir /srv/thothii \
|
||||
```
|
||||
|
||||
Use a dedicated `thothii-ops` group for the small set of human operators. A human who runs
|
||||
`thothctl` must be in both `thothii-ops` (to traverse operator paths and read declared secret files
|
||||
`tht` must be in both `thothii-ops` (to traverse operator paths and read declared secret files
|
||||
for output redaction) and the host `docker` group (to invoke Docker). Docker-group membership is
|
||||
effectively host-root access; grant both memberships only to reviewed administrators. The
|
||||
non-login `thothii` account owns files and writable data but does not need Docker access.
|
||||
@@ -61,7 +62,7 @@ sudo groupadd --system thothii-ops
|
||||
sudo usermod --append --groups thothii-ops,docker "$USER"
|
||||
```
|
||||
|
||||
Log out and back in before continuing; `id` must show both groups. Do not run `thothctl` through
|
||||
Log out and back in before continuing; `id` must show both groups. Do not run `tht` through
|
||||
`sudo -u thothii`: that account deliberately lacks Docker access. Do not grant the human direct
|
||||
write access to runtime bind trees.
|
||||
|
||||
@@ -130,7 +131,7 @@ bridge gateway address or to a dedicated private host interface—never to `0.0.
|
||||
the check pass. A stable internal DNS record routed through an authenticated private listener is
|
||||
the preferred alternative.
|
||||
|
||||
After the first bounded start attempt, copy the exact core container name from `thothctl status`
|
||||
After the first bounded start attempt, copy the exact core container name from `tht status`
|
||||
into `CORE_NAME`, then derive—not guess—the network ID, Linux bridge interface, gateway, and
|
||||
subnet. Compose networks normally use `br-<first-12-network-id>`; an explicit
|
||||
`com.docker.network.bridge.name` option takes precedence:
|
||||
@@ -163,7 +164,7 @@ sudo iptables -I INPUT 1 -i "$BRIDGE" -s "$SUBNET" -d "$GATEWAY" -p tcp --dport
|
||||
sudo iptables -I INPUT 2 -d "$GATEWAY" -p tcp --dport "$EXTERNAL_PORT" -j REJECT
|
||||
```
|
||||
|
||||
Confirm reachability with `thothctl pi test` for the configured LLM/Pi path and with the
|
||||
Confirm reachability with `tht pi test` for the configured LLM/Pi path and with the
|
||||
authenticated Workspace Diagnostics action for DWH, vector collection/embedding pairing, and
|
||||
embedding endpoints. A timeout paired with `ss -lntp`, `ip address show dev "$BRIDGE"`, and the
|
||||
firewall counters distinguishes a loopback bind from a subnet/interface rule failure. Do not add
|
||||
@@ -200,7 +201,7 @@ the writable Pi-state root and then overlays protected `auth.json` plus tracked
|
||||
`settings.json` read-only below it. Docker requires those three hidden target files to exist under
|
||||
the host parent bind before startup. The initializer creates them atomically with UID/GID 10001,
|
||||
mode `0600`, rejects symlink roots or targets, and never overwrites existing contents. It is safe to rerun
|
||||
after restoring `pi-state`; run it before any `thothctl start`, Compose render/start, or Pi update.
|
||||
after restoring `pi-state`; run it before any `tht start`, Compose render/start, or Pi update.
|
||||
|
||||
Copy the path-only server environment and installation descriptor:
|
||||
|
||||
@@ -224,7 +225,7 @@ image overrides go after them.
|
||||
Create each installation credential (Pi/application, Git, and session storage) as an independent
|
||||
regular file in `/srv/thothii/secrets`, owned by
|
||||
UID 10001, group `thothii-ops`, and mode `0640`. Owner access lets the UID 10001 container read a
|
||||
file mounted under `/run/secrets`; group access lets the reviewed human run `thothctl`. The
|
||||
file mounted under `/run/secrets`; group access lets the reviewed human run `tht`. The
|
||||
operator environment records only absolute `*_FILE` or `*_SOURCE` paths for those installation
|
||||
credentials. DWH and Evidence values are entered later through Workspace management and persist
|
||||
as ciphertext under `/data/workspace-secrets`; the frontend receives no secret values. Do not
|
||||
@@ -253,7 +254,7 @@ sudo -u thothii cp /srv/thothii/operator/server.env deploy/env/local.env
|
||||
bash scripts/build-local.sh
|
||||
```
|
||||
|
||||
The printed local-profile start command is not the server start command; use `thothctl` below.
|
||||
The printed local-profile start command is not the server start command; use `tht` below.
|
||||
|
||||
Alternatively, create a reviewed untracked override with release images pinned by immutable
|
||||
digest. Mutable tags are not a production pin:
|
||||
@@ -274,39 +275,39 @@ services:
|
||||
Add that absolute file last in `overrides`. `core` and `session-migrate` must use the exact same
|
||||
core digest; neither may retain a local build or `:local` image. Frontend uses its own exact digest.
|
||||
Both images must come from one compatible release; the core image must retain the declared Pi
|
||||
version labels checked by `thothctl pi doctor`. Pull access
|
||||
version labels checked by `tht pi doctor`. Pull access
|
||||
belongs in the host Docker credential store, not in Compose or the installation descriptor.
|
||||
|
||||
## Install thothctl
|
||||
## Install tht
|
||||
|
||||
Build the operator binaries with Docker. No Go installation or Go knowledge is required:
|
||||
|
||||
```sh
|
||||
cd /srv/thothii/source/ThothII
|
||||
THT_THOTHCTL_OUTPUT_DIRECTORY=/srv/thothii/operator/build-output \
|
||||
bash scripts/build-thothctl.sh
|
||||
THT_THT_OUTPUT_DIRECTORY=/srv/thothii/operator/build-output \
|
||||
bash scripts/build-tht.sh
|
||||
sudo install -o root -g thothii-ops -m 0750 \
|
||||
/srv/thothii/operator/build-output/thothctl-linux-amd64 \
|
||||
/srv/thothii/operator/thothctl
|
||||
/srv/thothii/operator/build-output/tht-linux-amd64 \
|
||||
/srv/thothii/operator/tht
|
||||
```
|
||||
|
||||
The source checkout stays read-only to the human. The explicit output directory is the only build
|
||||
write boundary; the build script rejects relative or non-canonical output paths. After installation,
|
||||
remove or retain `build-output` according to the site's reviewed artifact policy.
|
||||
|
||||
Use `thothctl-linux-arm64` on an ARM64 server. Set these variables in the maintenance shell; do
|
||||
Use `tht-linux-arm64` on an ARM64 server. Set these variables in the maintenance shell; do
|
||||
not source `server.env` as shell code:
|
||||
|
||||
```sh
|
||||
THTCTL=/srv/thothii/operator/thothctl
|
||||
THT_BIN=/srv/thothii/operator/tht
|
||||
INSTALLATION=/srv/thothii/operator/thothii-installation.yaml
|
||||
"$THTCTL" --help
|
||||
"$THTCTL" --installation "$INSTALLATION" update --check-only
|
||||
"$THT_BIN" --help
|
||||
"$THT_BIN" --installation "$INSTALLATION" update --check-only
|
||||
```
|
||||
|
||||
Every operator command includes the descriptor explicitly. This preserves the installation's
|
||||
profile, overrides, project identity, and durable current-image selector. The general form is
|
||||
`thothctl --installation /absolute/path/thothii-installation.yaml <command>`.
|
||||
`tht --installation /absolute/path/thothii-installation.yaml <command>`.
|
||||
|
||||
## Start and verify readiness
|
||||
|
||||
@@ -317,8 +318,8 @@ selected core image after all installation overrides, so this procedure is ident
|
||||
and pinned modes. It exits nonzero unless both arrays are empty:
|
||||
|
||||
```sh
|
||||
"$THTCTL" --installation "$INSTALLATION" stop
|
||||
"$THTCTL" --installation "$INSTALLATION" sessions migrate --yes
|
||||
"$THT_BIN" --installation "$INSTALLATION" stop
|
||||
"$THT_BIN" --installation "$INSTALLATION" sessions migrate --yes
|
||||
```
|
||||
|
||||
Successful output has this shape (the `applied` list may contain versions on first use):
|
||||
@@ -330,12 +331,12 @@ Successful output has this shape (the `applied` list may contain versions on fir
|
||||
Only after seeing `"pending":[]` and `"drifted":[]`, start and verify:
|
||||
|
||||
```sh
|
||||
"$THTCTL" --installation "$INSTALLATION" start
|
||||
"$THTCTL" --installation "$INSTALLATION" status
|
||||
"$THTCTL" --installation "$INSTALLATION" doctor
|
||||
"$THT_BIN" --installation "$INSTALLATION" start
|
||||
"$THT_BIN" --installation "$INSTALLATION" status
|
||||
"$THT_BIN" --installation "$INSTALLATION" doctor
|
||||
curl --fail http://127.0.0.1:8080/health
|
||||
"$THTCTL" --installation "$INSTALLATION" pi doctor
|
||||
"$THTCTL" --installation "$INSTALLATION" pi test
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi doctor
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi test
|
||||
```
|
||||
|
||||
`/health` proves process liveness. Readiness additionally requires both healthy services, a valid
|
||||
@@ -365,10 +366,10 @@ trusted proxy, and never expose `core`.
|
||||
Configure only closed provider/model/reasoning choices. Credentials remain protected files:
|
||||
|
||||
```sh
|
||||
"$THTCTL" --installation "$INSTALLATION" pi status
|
||||
"$THTCTL" --installation "$INSTALLATION" pi configure
|
||||
"$THTCTL" --installation "$INSTALLATION" pi doctor
|
||||
"$THTCTL" --installation "$INSTALLATION" pi logs
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi status
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi configure
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi doctor
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi logs
|
||||
```
|
||||
|
||||
Before an update, announce maintenance and ask users to finish active work. `--drain` closes new
|
||||
@@ -376,9 +377,9 @@ admission and waits until no active sessions remain; it does not discard session
|
||||
and registry-source examples are:
|
||||
|
||||
```sh
|
||||
"$THTCTL" --installation "$INSTALLATION" pi update \
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi update \
|
||||
--version 0.81.0 --source build --yes --drain
|
||||
"$THTCTL" --installation "$INSTALLATION" pi update \
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi update \
|
||||
--version 0.81.0 --source pull \
|
||||
--image registry.example.com/thothii/core@sha256:<64-lowercase-hex-digits> \
|
||||
--yes --drain
|
||||
@@ -388,23 +389,23 @@ The transaction recreates only `core`, preserves volumes, verifies health/config
|
||||
automatically attempts rollback after a post-mutation failure. For interrupted or ambiguous state:
|
||||
|
||||
```sh
|
||||
"$THTCTL" --installation "$INSTALLATION" pi maintenance status
|
||||
"$THTCTL" --installation "$INSTALLATION" pi rollback --yes
|
||||
"$THTCTL" --installation "$INSTALLATION" pi maintenance recover --yes
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi maintenance status
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi rollback --yes
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi maintenance recover --yes
|
||||
```
|
||||
|
||||
Leave maintenance active if rollback cannot be verified. Preserve `.thothctl/<installation-id>/`
|
||||
Leave maintenance active if rollback cannot be verified. Preserve `.tht/<installation-id>/`
|
||||
recovery state, repair the reported host/configuration issue, and rerun rollback or maintenance
|
||||
recovery. Never delete or edit `current-image.yaml` or `update-state.json` to force progress.
|
||||
|
||||
## Back up and restore
|
||||
|
||||
Back up before source, workspace, session-schema, or Pi changes. Drain work, stop the installation,
|
||||
record `git rev-parse HEAD`, image digests, and `thothctl status`, then archive the three bind trees
|
||||
record `git rev-parse HEAD`, image digests, and `tht status`, then archive the three bind trees
|
||||
with numeric ownership. Do not include live secrets in this ordinary archive.
|
||||
|
||||
```sh
|
||||
"$THTCTL" --installation "$INSTALLATION" stop
|
||||
"$THT_BIN" --installation "$INSTALLATION" stop
|
||||
BACKUP=/srv/thothii-backups/2026-08-05
|
||||
sudo install -d -o root -g root -m 0700 "$BACKUP"
|
||||
sudo tar --numeric-owner --xattrs --acls -C /srv/thothii -czf "$BACKUP/runtime-data.tgz" \
|
||||
@@ -447,14 +448,14 @@ merge an archive into a non-empty tree.
|
||||
Begin with bounded, sanitized installation-aware commands:
|
||||
|
||||
```sh
|
||||
"$THTCTL" --installation "$INSTALLATION" status
|
||||
"$THTCTL" --installation "$INSTALLATION" doctor
|
||||
"$THTCTL" --installation "$INSTALLATION" logs
|
||||
"$THTCTL" --installation "$INSTALLATION" pi status
|
||||
"$THTCTL" --installation "$INSTALLATION" pi doctor
|
||||
"$THTCTL" --installation "$INSTALLATION" pi test
|
||||
"$THTCTL" --installation "$INSTALLATION" pi logs
|
||||
"$THTCTL" --installation "$INSTALLATION" pi maintenance status
|
||||
"$THT_BIN" --installation "$INSTALLATION" status
|
||||
"$THT_BIN" --installation "$INSTALLATION" doctor
|
||||
"$THT_BIN" --installation "$INSTALLATION" logs
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi status
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi doctor
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi test
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi logs
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi maintenance status
|
||||
```
|
||||
|
||||
Use the authenticated Workspace Management status and diagnostic actions for Git revision,
|
||||
@@ -468,7 +469,7 @@ external service identity; and the proxy/identity provider for login failures.
|
||||
|
||||
## Data-preserving uninstall
|
||||
|
||||
Drain and stop through `thothctl`, take and verify one final backup, and disable the TLS proxy
|
||||
Drain and stop through `tht`, take and verify one final backup, and disable the TLS proxy
|
||||
route. Set `THT_BACKUP_ROOT=/srv/thothii-backups` in `server.env`; the removal command verifies the
|
||||
filesystem identity of that backup root, all three bind trees, and every declared secret before
|
||||
and after removing anything.
|
||||
@@ -477,14 +478,14 @@ First run without confirmation. It displays the exact installation project, serv
|
||||
name, container ID, and stopped state, then exits without mutation. Check every target:
|
||||
|
||||
```sh
|
||||
"$THTCTL" --installation "$INSTALLATION" stop
|
||||
"$THTCTL" --installation "$INSTALLATION" remove
|
||||
"$THT_BIN" --installation "$INSTALLATION" stop
|
||||
"$THT_BIN" --installation "$INSTALLATION" remove
|
||||
```
|
||||
|
||||
If and only if both targets are the expected stopped `frontend` and `core` containers, confirm:
|
||||
|
||||
```sh
|
||||
"$THTCTL" --installation "$INSTALLATION" remove --yes exact-core-id exact-frontend-id
|
||||
"$THT_BIN" --installation "$INSTALLATION" remove --yes exact-core-id exact-frontend-id
|
||||
```
|
||||
|
||||
Replace both example IDs with the values from the immediately preceding dry-run. The command
|
||||
@@ -496,5 +497,5 @@ paths still identify the same filesystem objects. Keep `/srv/thothii/data`, `pi-
|
||||
descriptor if reinstallation is possible. Do not prune global Docker data.
|
||||
|
||||
Do **not** run `docker compose down --volumes`; it deletes persistent application data. Reusing the
|
||||
same protected descriptor path preserves the `thothctl` installation identity and allows a later
|
||||
same protected descriptor path preserves the `tht` installation identity and allows a later
|
||||
compatible source checkout to reconnect the retained state.
|
||||
|
||||
Reference in New Issue
Block a user