ThothII/docs/install/server.md

# Install ThothII on a Linux server

Server authentication uses generic OIDC with the reverse proxy preserving the configured public
origin and callback path. Follow the [OIDC guide](authentication-oidc.md), [Authentik guide](authentik.md)
when applicable, and the [authentication acceptance matrix](../testing/authentication-manual-acceptance.md).
The host authentication CLI is `tht`. Use `tht --installation <descriptor> workspace inspect
--workspace <id> --json` for the active workspace snapshot, `tht --installation <descriptor>
auth check` for live non-interactive authentication diagnosis, `auth check --interactive` for
device-flow identity validation, and `tht ... doctor --json` for the aggregate installation gate.

This guide is for an installer with basic Linux administration and very basic Docker knowledge.
It deploys the same Compose distribution used on a local PC: the mandatory application is exactly
`frontend` plus `core`, and pinned Pi is inside `core`. The server does not need host Pi, Node.js,
Python, Go, a browser shell, or a Docker socket inside either container.

Examples use `/srv/thothii` as an **example operator root**, `thoth.example.com` as a replaceable
DNS name, and systemd-based command names. Adapt them to local policy. Complete the server session
storage overlay and migration procedure before exposing a production installation.

## Deployment contract

- The generic Linux host and Docker Compose v2 are the deployment platform. No other
  application's Compose project, network, path, or runtime is required.
- `frontend` is the only published service and defaults to `127.0.0.1:8080`; `core` has no host
  port. A host Nginx or Caddy listener terminates TLS and sends all application traffic to
  `frontend`, never directly to `core`.
- DWH, vector database, embedding service, and LLM are external configurable endpoints. This
  remains true when they happen to run on the same physical server.
- Application, Git, connector, and session credentials are protected host files mounted read-only
  under `/run/secrets`. Pi's protected auth JSON uses its dedicated read-only Pi mount. No secret
  value belongs in Git, images, browser storage, environment values, rendered Compose, or logs.
- The Git-backed workspace registry is the source of truth. Installation-local bindings identify
  endpoints and secret-file paths; they do not replace the reviewed Git workspace descriptors.
- `tht` is the operator CLI for start, stop, status, health, logs, Pi lifecycle, drain, and
  rollback. Raw Compose lifecycle commands bypass installation state and are unsupported.

Read [server workspace-registry installation](server-workspace-registry.md),
[Pi management](pi-management.md), and the session-server comments in
`deploy/compose.session-server.yaml.example` before the first public start.

## Service account and directories

The container runtime identity is fixed at UID/GID 10001. Reserve the same host ID for a dedicated
non-login `thothii` account so bind-mounted ownership is obvious. Stop if either ID is already used
by another account; choose a reviewed host mapping instead of changing the image identity.

```sh
getent passwd 10001
getent group 10001
sudo useradd --system --uid 10001 --user-group --home-dir /srv/thothii \
  --create-home --shell /usr/sbin/nologin thothii
```

Use a dedicated `thothii-ops` group for the small set of human operators. A human who runs
`tht` must be in both `thothii-ops` (to traverse operator paths and read declared secret files
for output redaction) and the host `docker` group (to invoke Docker). Docker-group membership is
effectively host-root access; grant both memberships only to reviewed administrators. The
non-login `thothii` account owns files and writable data but does not need Docker access.

```sh
sudo groupadd --system thothii-ops
sudo usermod --append --groups thothii-ops,docker "$USER"
```

Log out and back in before continuing; `id` must show both groups. Do not run `tht` through
`sudo -u thothii`: that account deliberately lacks Docker access. Do not grant the human direct
write access to runtime bind trees.

Create explicit directories. `source` contains the clone; `operator` contains untracked path-only
configuration; the three writable trees are bind-mounted into `core`; `secrets` contains regular
files only. Backups are separate from live data. Reset the account home explicitly because
distribution `useradd` defaults may otherwise leave `/srv/thothii` non-traversable by
`thothii-ops`.

```sh
sudo install -d -o 10001 -g thothii-ops -m 2750 /srv/thothii
sudo install -d -o 10001 -g thothii-ops -m 2750 /srv/thothii/source
sudo install -d -o 10001 -g thothii-ops -m 2770 /srv/thothii/operator
sudo install -d -o 10001 -g thothii-ops -m 2750 /srv/thothii/secrets
sudo install -d -o 10001 -g 10001 -m 0750 /srv/thothii/data
sudo install -d -o 10001 -g 10001 -m 0750 /srv/thothii/pi-state
sudo install -d -o 10001 -g 10001 -m 0750 /srv/thothii/workspace-registry
sudo install -d -o root -g root -m 0700 /srv/thothii-backups
```

Verify `/srv/thothii` is owned by `10001:thothii-ops` with mode `2750`. The human operator can
traverse the parent but can write only `operator`; setgid keeps generated files in `thothii-ops`.
`source`, `secrets`, and all runtime bind trees remain non-group-writable. Do not make
`/srv/thothii` a shared application directory.

## Firewall and network boundaries

Set `THOTH_SERVER_BIND=127.0.0.1`. Permit inbound TCP 80/443 only to the TLS proxy; port 80 should
redirect to HTTPS. Do not open 8080 externally, and do not add a core port. If a separate proxy
host is used, replace loopback with a private, firewalled address and allow only that proxy source.

Allow outbound DNS and HTTPS to the source/Git registries, plus only the configured ports for the
Git remote, DWH, vector database, embedding service, LLM, session PostgreSQL, and any approved
bastion. Docker's private `thothii` network carries only `frontend`↔`core` traffic. Do not attach
the mandatory stack to another application's network.

After start, confirm the host listens as intended:

```sh
sudo ss -lntp
```

Expected public listeners are the proxy on 80/443 and the frontend on loopback 8080. There must be
no host listener for core port 8787.

## Address co-resident external services

Endpoint values are resolved inside `core`. Therefore container 127.0.0.1 means the container
itself, not the Linux host. Prefer real DNS names with TLS, authentication, and firewall policy,
even for services on this physical server.

When DNS is unavailable for a host-published service, create an untracked override such as
`/srv/thothii/operator/host-gateway.yaml` and add it to the installation descriptor:

```yaml
services:
  core:
    extra_hosts:
      - "host.docker.internal:host-gateway"
```

Use `host.docker.internal` in the endpoint binding. The `host-gateway` mapping supplies routing;
it does not bundle or trust the target service. A host service listening only on host
`127.0.0.1` is **not reachable** through this mapping. Bind that service to the ThothII Docker
bridge gateway address or to a dedicated private host interface—never to `0.0.0.0` merely to make
the check pass. A stable internal DNS record routed through an authenticated private listener is
the preferred alternative.

After the first bounded start attempt, copy the exact core container name from `tht status`
into `CORE_NAME`, then derive—not guess—the network ID, Linux bridge interface, gateway, and
subnet. Compose networks normally use `br-<first-12-network-id>`; an explicit
`com.docker.network.bridge.name` option takes precedence:

```sh
CORE_NAME=replace-with-exact-core-container-name
NETWORK_ID=$(docker inspect --format '{{range .NetworkSettings.Networks}}{{.NetworkID}}{{end}}' "$CORE_NAME")
NETWORK_NAME=$(docker network inspect --format '{{.Name}}' "$NETWORK_ID")
BRIDGE=$(docker network inspect --format '{{index .Options "com.docker.network.bridge.name"}}' "$NETWORK_ID")
test -n "$BRIDGE" || BRIDGE="br-${NETWORK_ID%${NETWORK_ID#????????????}}"
GATEWAY=$(docker network inspect --format '{{(index .IPAM.Config 0).Gateway}}' "$NETWORK_ID")
SUBNET=$(docker network inspect --format '{{(index .IPAM.Config 0).Subnet}}' "$NETWORK_ID")
printf 'network=%s bridge=%s gateway=%s subnet=%s\n' "$NETWORK_NAME" "$BRIDGE" "$GATEWAY" "$SUBNET"
ip address show dev "$BRIDGE"
```

Bind the co-resident service to `$GATEWAY`. In the host firewall `INPUT` chain, allow its exact
TCP port only when source is `$SUBNET`, input interface is `$BRIDGE`, and destination is
`$GATEWAY`; reject other sources to that listener and persist the rules using the distribution's
firewall manager. Docker's `DOCKER-USER` chain governs forwarded/published traffic and does not
replace this host-input rule. Ask the firewall administrator to implement the equivalent policy
with nftables when iptables is not the site's source of truth.

For an iptables-managed host, replace the port before applying these reviewed rules; the second
rule prevents any other interface/source from reaching that gateway listener:

```sh
EXTERNAL_PORT=replace-with-exact-service-port
sudo iptables -I INPUT 1 -i "$BRIDGE" -s "$SUBNET" -d "$GATEWAY" -p tcp --dport "$EXTERNAL_PORT" -j ACCEPT
sudo iptables -I INPUT 2 -d "$GATEWAY" -p tcp --dport "$EXTERNAL_PORT" -j REJECT
```

Confirm reachability with `tht pi test` for the configured LLM/Pi path and with the
authenticated Workspace Diagnostics action for DWH, vector collection/embedding pairing, and
embedding endpoints. A timeout paired with `ss -lntp`, `ip address show dev "$BRIDGE"`, and the
firewall counters distinguishes a loopback bind from a subnet/interface rule failure. Do not add
a shell to the browser or mount the Docker socket into core for this diagnostic.

Configure each boundary independently:

- DWH: read-only runtime identity, database/schema, verified TLS, and direct or REST endpoint.
- Vector database: endpoint plus exact database/schema, collection, distance metric, and writer
  policy declared by the reviewed workspace.
- Embedding service: endpoint and the collection/embedding pairing—model and dimensions must match
  the existing collection. Co-residence does not permit silently changing that pairing.
- LLM: authenticated endpoint selected through deployment and Pi configuration.

Never add those services to ThothII's mandatory Compose files. Follow
[the diagnostic protocol](../workspace-diagnostic-protocol.md) before enabling a workspace.

## Prepare operator files and secrets

Clone with LF line endings, then verify before every build:

```sh
sudo -u thothii git -c core.autocrlf=false clone \
  https://github.example.invalid/your-org/ThothII.git /srv/thothii/source/ThothII
cd /srv/thothii/source/ThothII
sudo -u thothii git config --local core.autocrlf false
bash scripts/verify-line-endings.sh
sudo /srv/thothii/source/ThothII/scripts/prepare-server-pi-state.sh \
  /srv/thothii/pi-state 10001 10001
```

The last command is a mandatory clean-install and restore preflight. The server profile bind-mounts
the writable Pi-state root and then overlays protected `auth.json` plus tracked `models.json` and
`settings.json` read-only below it. Docker requires those three hidden target files to exist under
the host parent bind before startup. The initializer creates them atomically with UID/GID 10001,
mode `0600`, rejects symlink roots or targets, and never overwrites existing contents. It is safe to rerun
after restoring `pi-state`; run it before any `tht start`, Compose render/start, or Pi update.

Copy the path-only server environment and installation descriptor:

```sh
sudo -u thothii cp deploy/env/server.env.example /srv/thothii/operator/server.env
sudo -u thothii cp docs/install/examples/thothii-installation.server.yaml \
  /srv/thothii/operator/thothii-installation.yaml
sudo chown 10001:thothii-ops /srv/thothii/operator/server.env \
  /srv/thothii/operator/thothii-installation.yaml
sudo chmod 0660 /srv/thothii/operator/server.env \
  /srv/thothii/operator/thothii-installation.yaml
```

The named human operator can now edit both placeholder files without `sudo`; use an editor that
preserves the group, or create replacements under `umask 0007` in the setgid operator directory.
Replace every placeholder with an absolute path. Use exactly one Git transport override. For
HTTPS, replace `deploy/compose.git-ssh.yaml` with `deploy/compose.git-https.yaml`. Keep the required
session-server overlay. Optional host-gateway or pinned
image overrides go after them.

Create each installation credential (Pi/application, Git, and session storage) as an independent
regular file in `/srv/thothii/secrets`, owned by
UID 10001, group `thothii-ops`, and mode `0640`. Owner access lets the UID 10001 container read a
file mounted under `/run/secrets`; group access lets the reviewed human run `tht`. The
operator environment records only absolute `*_FILE` or `*_SOURCE` paths for those installation
credentials. DWH and Evidence values are entered later through Workspace management and persist
as ciphertext under `/data/workspace-secrets`; the frontend receives no secret values. Do not
print file contents while testing permissions.

```sh
sudo find /srv/thothii/secrets -type f -exec chown 10001:thothii-ops {} +
sudo find /srv/thothii/secrets -type f -exec chmod 0640 {} +
sudo find /srv/thothii/secrets -type f \( ! -user thothii -o ! -group thothii-ops -o ! -perm 0640 \) -print
```

Configure the remote repository and exactly one read-only Git transport as described in
[server workspace repository installation](server-workspace-registry.md). After startup, complete
the selected workspace's DWH and Evidence credentials through Workspace management. Secret values
must never be pasted into `server.env`, the installation YAML, a URL, or a shell argument.

## Build locally or select pinned images

Choose one image source. For a source build, the repository's reproducible launcher builds the
same `core` and `frontend` images used by the local profile. It requires only Git, Docker, and
Compose; copy the reviewed path-only server environment to the launcher's untracked input first:

```sh
cd /srv/thothii/source/ThothII
sudo -u thothii cp /srv/thothii/operator/server.env deploy/env/local.env
bash scripts/build-local.sh
```

The printed local-profile start command is not the server start command; use `tht` below.

Alternatively, create a reviewed untracked override with release images pinned by immutable
digest. Mutable tags are not a production pin:

```yaml
services:
  core:
    build: !reset null
    image: registry.example.com/thothii/core@sha256:<64-lowercase-hex-digits>
  session-migrate:
    build: !reset null
    image: registry.example.com/thothii/core@sha256:<64-lowercase-hex-digits>
  frontend:
    build: !reset null
    image: registry.example.com/thothii/frontend@sha256:<64-lowercase-hex-digits>
```

Add that absolute file last in `overrides`. `core` and `session-migrate` must use the exact same
core digest; neither may retain a local build or `:local` image. Frontend uses its own exact digest.
Both images must come from one compatible release; the core image must retain the declared Pi
version labels checked by `tht pi doctor`. Pull access
belongs in the host Docker credential store, not in Compose or the installation descriptor.

## Install tht

Build the operator binaries with Docker. No Go installation or Go knowledge is required:

```sh
cd /srv/thothii/source/ThothII
THT_THT_OUTPUT_DIRECTORY=/srv/thothii/operator/build-output \
  bash scripts/build-tht.sh
sudo install -o root -g thothii-ops -m 0750 \
  /srv/thothii/operator/build-output/tht-linux-amd64 \
  /srv/thothii/operator/tht
```

The source checkout stays read-only to the human. The explicit output directory is the only build
write boundary; the build script rejects relative or non-canonical output paths. After installation,
remove or retain `build-output` according to the site's reviewed artifact policy.

Use `tht-linux-arm64` on an ARM64 server. Set these variables in the maintenance shell; do
not source `server.env` as shell code:

```sh
THT_BIN=/srv/thothii/operator/tht
INSTALLATION=/srv/thothii/operator/thothii-installation.yaml
"$THT_BIN" --help
"$THT_BIN" --installation "$INSTALLATION" update --check-only
```

Every operator command includes the descriptor explicitly. This preserves the installation's
profile, overrides, project identity, and durable current-image selector. The general form is
`tht --installation /absolute/path/thothii-installation.yaml <command>`.

## Start and verify readiness

Keep the TLS proxy stopped or firewalled during bootstrap. First stop the app, run the
installation-aware session migration, and inspect its pristine JSON. The command activates only
the `session-migrate` profile/service with `--no-deps --no-TTY`; it derives the migrator image from the
selected core image after all installation overrides, so this procedure is identical for source
and pinned modes. It exits nonzero unless both arrays are empty:

```sh
"$THT_BIN" --installation "$INSTALLATION" stop
"$THT_BIN" --installation "$INSTALLATION" sessions migrate --yes
```

Successful output has this shape (the `applied` list may contain versions on first use):

```json
{"applied":[],"drifted":[],"pending":[]}
```

Only after seeing `"pending":[]` and `"drifted":[]`, start and verify:

```sh
"$THT_BIN" --installation "$INSTALLATION" start
"$THT_BIN" --installation "$INSTALLATION" status
"$THT_BIN" --installation "$INSTALLATION" doctor
curl --fail http://127.0.0.1:8080/health
"$THT_BIN" --installation "$INSTALLATION" pi doctor
"$THT_BIN" --installation "$INSTALLATION" pi test
```

`/health` proves process liveness. Readiness additionally requires both healthy services, a valid
Pi provider/model smoke, a successful Git registry pull with an active validated snapshot, valid
workspace diagnostics, and ready session PostgreSQL. Use the authenticated Workspace Management
page to pull and diagnose the reviewed workspace. A liveness response alone is not release
approval.

After configuring the proxy, open <https://thoth.example.com> in a browser. Verify an unauthenticated
request is denied or redirected by the real identity provider, an authorized user can load the
same-origin UI and `/api`, an unauthorized user is denied, and an administrator alone can open Pi
Management. Keep port 8080 inaccessible from other hosts.

## Configure TLS and upstream authentication

Choose [Nginx](reverse-proxy-nginx.md) or [Caddy](reverse-proxy-caddy.md). Both examples terminate
TLS and proxy only to loopback `frontend`. They preserve SSE and clear client-supplied identity
headers before authentication.

The authentication gateway must validate a real login/session and return normalized issuer,
subject, display-name, and admin claims only after success. Merely forwarding those headers does
not authenticate anyone. Do not enable `AUTH_MODE=upstream` on a listener reachable around the
trusted proxy, and never expose `core`.

## Operate Pi, drain, and roll back

Configure only closed provider/model/reasoning choices. Credentials remain protected files:

```sh
"$THT_BIN" --installation "$INSTALLATION" pi status
"$THT_BIN" --installation "$INSTALLATION" pi configure
"$THT_BIN" --installation "$INSTALLATION" pi doctor
"$THT_BIN" --installation "$INSTALLATION" pi logs
```

Before an update, announce maintenance and ask users to finish active work. `--drain` closes new
admission and waits until no active sessions remain; it does not discard sessions. Build-source
and registry-source examples are:

```sh
"$THT_BIN" --installation "$INSTALLATION" pi update \
  --version 0.81.0 --source build --yes --drain
"$THT_BIN" --installation "$INSTALLATION" pi update \
  --version 0.81.0 --source pull \
  --image registry.example.com/thothii/core@sha256:<64-lowercase-hex-digits> \
  --yes --drain
```

The transaction recreates only `core`, preserves volumes, verifies health/configuration/Pi, and
automatically attempts rollback after a post-mutation failure. For interrupted or ambiguous state:

```sh
"$THT_BIN" --installation "$INSTALLATION" pi maintenance status
"$THT_BIN" --installation "$INSTALLATION" pi rollback --yes
"$THT_BIN" --installation "$INSTALLATION" pi maintenance recover --yes
```

Leave maintenance active if rollback cannot be verified. Preserve `.tht/<installation-id>/`
recovery state, repair the reported host/configuration issue, and rerun rollback or maintenance
recovery. Never delete or edit `current-image.yaml` or `update-state.json` to force progress.

## Back up and restore

Back up before source, workspace, session-schema, or Pi changes. Drain work, stop the installation,
record `git rev-parse HEAD`, image digests, and `tht status`, then archive the three bind trees
with numeric ownership. Do not include live secrets in this ordinary archive.

```sh
"$THT_BIN" --installation "$INSTALLATION" stop
BACKUP=/srv/thothii-backups/2026-08-05
sudo install -d -o root -g root -m 0700 "$BACKUP"
sudo tar --numeric-owner --xattrs --acls -C /srv/thothii -czf "$BACKUP/runtime-data.tgz" \
  data pi-state workspace-registry
sudo sh -ceu 'cd "$1"; sha256sum runtime-data.tgz > SHA256SUMS; sha256sum --check SHA256SUMS' sh "$BACKUP"
```

Back up the installation descriptor, path-only environment, generated overrides, source revision,
and secret files to separate encrypted access-controlled storage. Database-backed production
sessions require their own PostgreSQL-native consistent backup; the local bind tree is not a
substitute. Test both restore paths periodically.

Restore only while stopped. Verify the checksum, extract first into a new empty root, inspect
ownership and expected registry layout, then retain the old trees by renaming them before placing
the restored set. This keeps the previous state recoverable:

```sh
RESTORE=/srv/thothii-restore-2026-08-05
sudo install -d -o root -g root -m 0700 "$RESTORE"
sudo sh -ceu 'cd "$1"; sha256sum --check SHA256SUMS' sh /srv/thothii-backups/2026-08-05
sudo tar --numeric-owner --xattrs --acls -C "$RESTORE" \
  -xzf /srv/thothii-backups/2026-08-05/runtime-data.tgz
sudo test -d "$RESTORE/workspace-registry/repo"
sudo test -d "$RESTORE/workspace-registry/snapshots"
```

After placing the restored `pi-state` tree and before the first start, rerun
`sudo /srv/thothii/source/ThothII/scripts/prepare-server-pi-state.sh /srv/thothii/pi-state 10001 10001`.
It validates or recreates only the hidden regular mount targets; it does not alter restored Pi
state or any protected configuration source.

During the reviewed restore window, move each old tree to a timestamped sibling, move the matching
restored tree into `/srv/thothii`, restore the PostgreSQL session backup from the same recovery
point, and keep the proxy closed. Run `update --check-only`, `start`, `doctor`, `pi test`, registry
status, workspace diagnostics, and a known historical session before reopening traffic. Never
merge an archive into a non-empty tree.

## Diagnostics

Begin with bounded, sanitized installation-aware commands:

```sh
"$THT_BIN" --installation "$INSTALLATION" status
"$THT_BIN" --installation "$INSTALLATION" doctor
"$THT_BIN" --installation "$INSTALLATION" logs
"$THT_BIN" --installation "$INSTALLATION" pi status
"$THT_BIN" --installation "$INSTALLATION" pi doctor
"$THT_BIN" --installation "$INSTALLATION" pi test
"$THT_BIN" --installation "$INSTALLATION" pi logs
"$THT_BIN" --installation "$INSTALLATION" pi maintenance status
```

Use the authenticated Workspace Management status and diagnostic actions for Git revision,
degraded snapshot, bindings, DWH, vector, and embedding checks. Review proxy logs separately, but
configure both proxy and log shipping to exclude cookies, authorization data, identity payloads,
query strings, and secret values. Do not render Compose or print an environment as a diagnostic.

Typical boundaries are: `doctor` for Docker/Compose/LF/volume/service health; `pi doctor` for image
and provider/model integrity; registry status for Git/snapshot health; workspace diagnostics for
external service identity; and the proxy/identity provider for login failures.

## Data-preserving uninstall

Drain and stop through `tht`, take and verify one final backup, and disable the TLS proxy
route. Set `THT_BACKUP_ROOT=/srv/thothii-backups` in `server.env`; the removal command verifies the
filesystem identity of that backup root, all three bind trees, and every declared secret before
and after removing anything.

First run without confirmation. It displays the exact installation project, service, container
name, container ID, and stopped state, then exits without mutation. Check every target:

```sh
"$THT_BIN" --installation "$INSTALLATION" stop
"$THT_BIN" --installation "$INSTALLATION" remove
```

If and only if both targets are the expected stopped `frontend` and `core` containers, confirm:

```sh
"$THT_BIN" --installation "$INSTALLATION" remove --yes exact-core-id exact-frontend-id
```

Replace both example IDs with the values from the immediately preceding dry-run. The command
refuses confirmation if the current target set differs. The confirmed operation passes only those
previously displayed immutable container IDs to Docker,
uses no force or volume option, rejects running/replaced containers, and proves the preservation
paths still identify the same filesystem objects. Keep `/srv/thothii/data`, `pi-state`,
`workspace-registry`, `operator`, protected secrets, database backups, and the installation
descriptor if reinstallation is possible. Do not prune global Docker data.

Do **not** run `docker compose down --volumes`; it deletes persistent application data. Reusing the
same protected descriptor path preserves the `tht` installation identity and allows a later
compatible source checkout to reconnect the retained state.