docs: add autonomous server installation guide
This commit is contained in:
@@ -0,0 +1,9 @@
|
||||
# Copy this file to a protected operator path named exactly thothii-installation.yaml.
|
||||
# Replace every absolute placeholder. Select exactly one Git transport override.
|
||||
profile: server
|
||||
projectDirectory: "/absolute/path/to/ThothII"
|
||||
envFile: "/absolute/path/to/thothii-server-operator/server.env"
|
||||
overrides:
|
||||
- "/absolute/path/to/ThothII/deploy/compose.session-server.yaml.example"
|
||||
- "/absolute/path/to/ThothII/deploy/compose.git-ssh.yaml"
|
||||
- "/absolute/path/to/thothii-server-operator/connector-secrets.server.yaml"
|
||||
@@ -0,0 +1,97 @@
|
||||
# Put ThothII behind Caddy
|
||||
|
||||
This example assumes Caddy runs on the Linux host, ThothII `frontend` listens only on
|
||||
`127.0.0.1:8080`, public DNS points to the host, and a separate authentication gateway validates
|
||||
the user's real login/session. Replace the domain and auth-gateway address.
|
||||
|
||||
## Trust boundary
|
||||
|
||||
Caddy is the only public listener. It provides automatic HTTPS, performs `forward_auth`, and
|
||||
proxies only to `frontend`; frontend then sends same-origin `/api` requests to private `core`.
|
||||
Never send the public reverse proxy to core port 8787.
|
||||
|
||||
The authentication gateway must return 2xx only after validating a real credential or session.
|
||||
The ordered `route` below deletes every public/private Thoth identity request header before auth.
|
||||
Only on auth success does `copy_headers` rename normalized response claims into the private
|
||||
`X-Thoth-Trusted-*` headers consumed by frontend.
|
||||
Forwarding identity headers alone does not authenticate a user.
|
||||
|
||||
Do not replace the authentication gateway with static `header_up` values, a network allowlist, or
|
||||
browser-provided identity. The admin claim must come from reviewed identity-provider authorization.
|
||||
|
||||
## Example configuration
|
||||
|
||||
Save a reviewed site block in the Caddyfile. Caddy obtains and renews TLS certificates for the real
|
||||
DNS name; use the organization's approved ACME issuer or certificate policy.
|
||||
|
||||
```caddyfile
|
||||
thoth.example.com {
|
||||
route {
|
||||
# Remove untrusted browser claims before the auth subrequest.
|
||||
request_header -X-Authenticated-User
|
||||
request_header -X-Thoth-Principal-Issuer
|
||||
request_header -X-Thoth-Principal-Subject
|
||||
request_header -X-Thoth-Principal-Display-Name
|
||||
request_header -X-Thoth-Is-Admin
|
||||
request_header -X-Thoth-Trusted-Principal-Issuer
|
||||
request_header -X-Thoth-Trusted-Principal-Subject
|
||||
request_header -X-Thoth-Trusted-Principal-Display-Name
|
||||
request_header -X-Thoth-Trusted-Is-Admin
|
||||
|
||||
forward_auth auth-gateway:4180 {
|
||||
uri /verify
|
||||
copy_headers {
|
||||
X-Thoth-Principal-Issuer>X-Thoth-Trusted-Principal-Issuer
|
||||
X-Thoth-Principal-Subject>X-Thoth-Trusted-Principal-Subject
|
||||
X-Thoth-Principal-Display-Name>X-Thoth-Trusted-Principal-Display-Name
|
||||
X-Thoth-Is-Admin>X-Thoth-Trusted-Is-Admin
|
||||
}
|
||||
}
|
||||
|
||||
reverse_proxy 127.0.0.1:8080 {
|
||||
# Disable response batching so SSE reaches the browser immediately.
|
||||
flush_interval -1
|
||||
header_up Host {host}
|
||||
header_up X-Forwarded-Proto https
|
||||
}
|
||||
}
|
||||
|
||||
log {
|
||||
output file /var/log/caddy/thoth-access.log
|
||||
format json
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Configure log processing to remove cookies, authorization data, query strings, and identity
|
||||
headers. Keep Caddy's private keys and state outside the ThothII source/operator directories.
|
||||
|
||||
## Validate and reload
|
||||
|
||||
Keep the public firewall rule closed while validating. Confirm ThothII responds only on loopback,
|
||||
format a review copy if desired, validate the active file, then reload through the service manager:
|
||||
|
||||
```sh
|
||||
curl --fail http://127.0.0.1:8080/health
|
||||
caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
|
||||
sudo systemctl reload caddy
|
||||
```
|
||||
|
||||
Do not remove `forward_auth` if validation fails. Correct the Caddy version, adapter syntax,
|
||||
authentication upstream, DNS, or certificate policy instead.
|
||||
|
||||
## Test authentication and SSE
|
||||
|
||||
Open the firewall only after all of these pass:
|
||||
|
||||
1. An unauthenticated HTTPS request is redirected to login or returns 401/403.
|
||||
2. Supplying forged `X-Thoth-Principal-*`, `X-Thoth-Is-Admin`, or `X-Thoth-Trusted-*` request
|
||||
headers does not grant access.
|
||||
3. A real authenticated non-admin can use the application but cannot open Pi Management.
|
||||
4. A real authenticated admin can use Pi Management.
|
||||
5. A browser session receives live model updates without batching; a protected
|
||||
`curl --no-buffer` request using a real short-lived login cookie is also acceptable. Delete the
|
||||
cookie jar immediately afterward.
|
||||
|
||||
Re-run these checks after changing the identity provider, authentication gateway, Caddy, or
|
||||
ThothII release.
|
||||
@@ -0,0 +1,132 @@
|
||||
# Put ThothII behind Nginx
|
||||
|
||||
This example assumes Nginx runs on the Linux host, ThothII `frontend` listens only on
|
||||
`127.0.0.1:8080`, and a separate authentication gateway validates the user's real login/session.
|
||||
Replace the documentation domain, certificate paths, and auth-gateway address.
|
||||
|
||||
## Trust boundary
|
||||
|
||||
Nginx is the only public listener. It terminates TLS, performs an `auth_request`, and proxies only
|
||||
to `frontend`; frontend then uses its private same-origin `/api` route to reach `core`. Never proxy
|
||||
the public listener directly to core port 8787.
|
||||
|
||||
The authentication gateway must return 2xx only after validating a real credential or session. It
|
||||
returns normalized `X-Thoth-Principal-*` and `X-Thoth-Is-Admin` response headers. Nginx clears all
|
||||
client-supplied public and private-hop identity headers and copies only those successful auth
|
||||
response values into `X-Thoth-Trusted-*` on the private hop to frontend.
|
||||
Forwarding identity headers alone does not authenticate a user.
|
||||
|
||||
Do not substitute a static header, network allowlist, or client-provided header for the
|
||||
authentication gateway. The admin value must come from reviewed identity-provider authorization,
|
||||
not from a username supplied by the browser.
|
||||
|
||||
## Example configuration
|
||||
|
||||
Install an Nginx build that includes `ngx_http_auth_request_module`. Save a reviewed version of
|
||||
this server block under the host's Nginx configuration directory:
|
||||
|
||||
```nginx
|
||||
server {
|
||||
listen 80;
|
||||
server_name thoth.example.com;
|
||||
return 301 https://$host$request_uri;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name thoth.example.com;
|
||||
|
||||
ssl_certificate /etc/nginx/tls/thoth/fullchain.pem;
|
||||
ssl_certificate_key /etc/nginx/tls/thoth/privkey.pem;
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
|
||||
location = /_authenticate {
|
||||
internal;
|
||||
proxy_pass http://auth-gateway:4180/verify;
|
||||
proxy_pass_request_body off;
|
||||
proxy_set_header Content-Length "";
|
||||
proxy_set_header X-Original-URI $request_uri;
|
||||
proxy_set_header X-Original-Method $request_method;
|
||||
|
||||
# The auth service derives identity from the real login/session, never these headers.
|
||||
proxy_set_header X-Authenticated-User "";
|
||||
proxy_set_header X-Thoth-Principal-Issuer "";
|
||||
proxy_set_header X-Thoth-Principal-Subject "";
|
||||
proxy_set_header X-Thoth-Principal-Display-Name "";
|
||||
proxy_set_header X-Thoth-Is-Admin "";
|
||||
proxy_set_header X-Thoth-Trusted-Principal-Issuer "";
|
||||
proxy_set_header X-Thoth-Trusted-Principal-Subject "";
|
||||
proxy_set_header X-Thoth-Trusted-Principal-Display-Name "";
|
||||
proxy_set_header X-Thoth-Trusted-Is-Admin "";
|
||||
}
|
||||
|
||||
location / {
|
||||
auth_request /_authenticate;
|
||||
auth_request_set $thoth_principal_issuer
|
||||
$upstream_http_x_thoth_principal_issuer;
|
||||
auth_request_set $thoth_principal_subject
|
||||
$upstream_http_x_thoth_principal_subject;
|
||||
auth_request_set $thoth_principal_display_name
|
||||
$upstream_http_x_thoth_principal_display_name;
|
||||
auth_request_set $thoth_is_admin
|
||||
$upstream_http_x_thoth_is_admin;
|
||||
|
||||
# Discard browser claims. Carry only successful auth-subrequest values to frontend.
|
||||
proxy_set_header X-Authenticated-User "";
|
||||
proxy_set_header X-Thoth-Principal-Issuer "";
|
||||
proxy_set_header X-Thoth-Principal-Subject "";
|
||||
proxy_set_header X-Thoth-Principal-Display-Name "";
|
||||
proxy_set_header X-Thoth-Is-Admin "";
|
||||
proxy_set_header X-Thoth-Trusted-Principal-Issuer $thoth_principal_issuer;
|
||||
proxy_set_header X-Thoth-Trusted-Principal-Subject $thoth_principal_subject;
|
||||
proxy_set_header X-Thoth-Trusted-Principal-Display-Name $thoth_principal_display_name;
|
||||
proxy_set_header X-Thoth-Trusted-Is-Admin $thoth_is_admin;
|
||||
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-Proto https;
|
||||
proxy_set_header X-Forwarded-Host $host;
|
||||
proxy_set_header X-Forwarded-For $remote_addr;
|
||||
proxy_set_header Connection "";
|
||||
proxy_pass http://127.0.0.1:8080;
|
||||
proxy_http_version 1.1;
|
||||
|
||||
# SSE must reach the browser without response buffering or cache delay.
|
||||
proxy_buffering off;
|
||||
proxy_cache off;
|
||||
proxy_read_timeout 3600s;
|
||||
add_header X-Accel-Buffering no always;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Keep the certificate private key outside the ThothII tree. Do not log cookies, authorization
|
||||
headers, auth response bodies, or trusted identity headers.
|
||||
|
||||
## Validate and reload
|
||||
|
||||
First keep the public firewall rule closed. Confirm ThothII responds only on loopback, validate the
|
||||
full Nginx configuration, then reload without stopping existing connections:
|
||||
|
||||
```sh
|
||||
curl --fail http://127.0.0.1:8080/health
|
||||
sudo nginx -t
|
||||
sudo systemctl reload nginx
|
||||
```
|
||||
|
||||
If `nginx -t` reports that `auth_request` is unknown, install the distribution package/module that
|
||||
provides it. Do not remove authentication to make the syntax check pass.
|
||||
|
||||
## Test authentication and SSE
|
||||
|
||||
Open the firewall only after all of these pass:
|
||||
|
||||
1. An unauthenticated HTTPS request is redirected to login or returns 401/403.
|
||||
2. Supplying forged `X-Thoth-Principal-*`, `X-Thoth-Is-Admin`, or `X-Thoth-Trusted-*` request
|
||||
headers does not grant access.
|
||||
3. A real authenticated non-admin can use the application but cannot open Pi Management.
|
||||
4. A real authenticated admin can use Pi Management.
|
||||
5. A browser session receives live model updates for longer than the default proxy timeout without
|
||||
batching; a protected `curl --no-buffer` test using a real short-lived login cookie is also
|
||||
acceptable. Delete the cookie jar immediately afterward.
|
||||
|
||||
Re-run these checks after changing the identity provider, auth gateway, Nginx, or ThothII release.
|
||||
@@ -1,8 +1,9 @@
|
||||
# Server workspace-registry installation
|
||||
|
||||
This is the production operator guide. The application image is read-only, secrets are mounted
|
||||
read-only, and sessions use immutable Git-validated snapshots. Expose the application only behind
|
||||
an authenticated same-origin reverse proxy; never publish the core port directly.
|
||||
This is the production workspace-registry companion to [the autonomous Linux server guide](server.md).
|
||||
The application image is read-only, secrets are mounted read-only, and sessions use immutable
|
||||
Git-validated snapshots. Expose the application only behind an authenticated same-origin reverse
|
||||
proxy; never publish the core port directly.
|
||||
|
||||
## Service account, storage, and firewall
|
||||
|
||||
@@ -142,52 +143,50 @@ The Git registry itself may still use SSH normally.
|
||||
|
||||
Use the repository's canonical `compose.yaml` plus `deploy/compose.server.yaml`; they always
|
||||
start the mandatory `frontend` and `core` services. Do not copy or maintain a standalone
|
||||
application Compose file. Review `deploy/workspaces/server-sessions.yaml.example`, materialize it
|
||||
as a protected host file, and set its absolute `THT_SERVER_WORKSPACE_CONFIG` path. Copy the
|
||||
bindings env example into the operator directory, then set absolute `PI_AUTH_FILE`,
|
||||
`THT_SECRETS_FILE`, `THT_WORKSPACE_BINDINGS_ENV_FILE`, and connector `*_SOURCE` paths.
|
||||
The same operator env must set
|
||||
application Compose file. Copy `docs/install/examples/thothii-installation.server.yaml` to the
|
||||
protected operator directory and preserve its required session-server overlay, exactly one Git
|
||||
transport override, and generated connector-secret override.
|
||||
|
||||
Review `deploy/workspaces/server-sessions.yaml.example`, materialize it as a protected host file,
|
||||
and set its absolute `THT_SERVER_WORKSPACE_CONFIG` path. Copy the bindings env example into the
|
||||
operator directory, then set absolute `PI_AUTH_FILE`, `THT_SECRETS_FILE`,
|
||||
`THT_WORKSPACE_BINDINGS_ENV_FILE`, and connector `*_SOURCE` paths. The same operator env must set
|
||||
`THT_SESSION_DB_HOST`, `THT_SESSION_DB_NAME`, `THT_SESSION_RUNTIME_USER`,
|
||||
`THT_SESSION_RUNTIME_PASSWORD_SOURCE`, and `THT_SESSION_CA_SOURCE`;
|
||||
`deploy/compose.session-server.yaml.example` wires `postgres`, `verify-full`, and separate
|
||||
runtime/CA Docker secret mount paths. This is the public server profile,
|
||||
not a filesystem-session fallback. A Compose `.env` file is not a shell environment, so do not
|
||||
import it into the maintenance shell. Explicitly export the non-secret source and bindings paths
|
||||
before running the commands below.
|
||||
runtime/CA secret targets under `/run/secrets`. This public server profile never falls back to
|
||||
filesystem sessions. The path-only environment file is not shell code; do not source it.
|
||||
|
||||
Configure the portal proxy so the frontend and `/api` share one origin. It authenticates first and
|
||||
clears client identity headers, carries auth-request claims over the private hop as
|
||||
`X-Thoth-Trusted-*`, and lets the frontend proxy inject only the normalized
|
||||
`X-Thoth-Principal-Issuer`, `X-Thoth-Principal-Subject`, `X-Thoth-Principal-Display-Name`, and
|
||||
`X-Thoth-Is-Admin` claims expected by `AUTH_MODE=upstream`; it is the only public listener. Use
|
||||
`deploy/nginx-authenticated-proxy.conf.example` as the forwarding contract.
|
||||
From a trusted maintenance shell:
|
||||
Generate the connector override, then use the installation-aware operator CLI. Building
|
||||
`thothctl` requires only Docker and no Go knowledge. From a trusted maintenance shell:
|
||||
|
||||
```sh
|
||||
export THT_SOURCE_ROOT=/absolute/path/to/ThothII
|
||||
export THT_OPERATOR_ENV=/srv/thothii/operator/server.env
|
||||
export THT_WORKSPACE_BINDINGS_ENV_FILE=/srv/thothii/operator/workspace-bindings.env
|
||||
export THT_CONNECTOR_OVERRIDE=/srv/thothii/operator/connector-secrets.local.yaml
|
||||
"$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" --bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" --operator-env "$THT_OPERATOR_ENV" --output "$THT_CONNECTOR_OVERRIDE"
|
||||
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \
|
||||
-f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.server.yaml" \
|
||||
-f "$THT_SOURCE_ROOT/deploy/compose.session-server.yaml.example" \
|
||||
-f "$THT_SOURCE_ROOT/deploy/compose.git-ssh.yaml" -f "$THT_CONNECTOR_OVERRIDE" up --build -d
|
||||
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \
|
||||
-f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.server.yaml" \
|
||||
-f "$THT_SOURCE_ROOT/deploy/compose.session-server.yaml.example" \
|
||||
-f "$THT_SOURCE_ROOT/deploy/compose.git-ssh.yaml" -f "$THT_CONNECTOR_OVERRIDE" \
|
||||
exec -T core curl --fail --silent http://127.0.0.1:8787/health
|
||||
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \
|
||||
-f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.server.yaml" \
|
||||
-f "$THT_SOURCE_ROOT/deploy/compose.session-server.yaml.example" \
|
||||
-f "$THT_SOURCE_ROOT/deploy/compose.git-ssh.yaml" -f "$THT_CONNECTOR_OVERRIDE" \
|
||||
exec -T core curl --fail --silent http://127.0.0.1:8787/workspace-registry/status
|
||||
THT_SOURCE_ROOT=/srv/thothii/source/ThothII
|
||||
THT_OPERATOR_ENV=/srv/thothii/operator/server.env
|
||||
THT_WORKSPACE_BINDINGS_ENV_FILE=/srv/thothii/operator/workspace-bindings.env
|
||||
THT_CONNECTOR_OVERRIDE=/srv/thothii/operator/connector-secrets.server.yaml
|
||||
THTCTL=/srv/thothii/operator/thothctl
|
||||
INSTALLATION=/srv/thothii/operator/thothii-installation.yaml
|
||||
"$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" \
|
||||
--bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" \
|
||||
--operator-env "$THT_OPERATOR_ENV" --output "$THT_CONNECTOR_OVERRIDE"
|
||||
"$THTCTL" --installation "$INSTALLATION" update --check-only
|
||||
"$THTCTL" --installation "$INSTALLATION" start
|
||||
"$THTCTL" --installation "$INSTALLATION" status
|
||||
"$THTCTL" --installation "$INSTALLATION" doctor
|
||||
"$THTCTL" --installation "$INSTALLATION" pi doctor
|
||||
"$THTCTL" --installation "$INSTALLATION" pi test
|
||||
```
|
||||
|
||||
`/health` is liveness. Registry status verifies branch/head/degraded state and the active validated
|
||||
snapshot; authenticated `/workspaces` verifies application access. A server with no active snapshot
|
||||
is not ready for workspace sessions even if liveness succeeds.
|
||||
Configure [Nginx](reverse-proxy-nginx.md) or [Caddy](reverse-proxy-caddy.md) so frontend and `/api`
|
||||
share one TLS origin. The proxy authenticates first, clears client identity headers, and carries
|
||||
only successful authentication claims over the private `X-Thoth-Trusted-*` hop. Forwarding claims
|
||||
without authenticating the request is not an identity boundary.
|
||||
|
||||
`/health` is liveness. The authenticated Workspace Management page's registry status verifies
|
||||
branch/head/degraded state and the active validated snapshot; its workspace listing verifies
|
||||
application access. A server with no active snapshot is not ready for workspace sessions even if
|
||||
liveness succeeds.
|
||||
|
||||
## Pull, publish, upgrade, backup, and recovery
|
||||
|
||||
@@ -196,10 +195,12 @@ browser-local. Publish takes a canonical diff, validates before commit, and push
|
||||
lock. On `workspace_conflict`, pull, resolve the reviewed field-level draft, validate/test, and
|
||||
publish; never edit `repo/` inside a running volume.
|
||||
|
||||
For upgrades, record active status/head, drain active Pi work, stop `core`, and take a
|
||||
filesystem-consistent backup of `/srv/thothii/workspace-registry` plus `/srv/thothii/data`. Exclude
|
||||
`/srv/thothii/secrets`. Render Compose, deploy the compatible image, verify health/status, then
|
||||
resume proxy traffic.
|
||||
For upgrades, record active status/head, finish active work, use the documented `thothctl pi update
|
||||
--drain` transaction when Pi/core changes, and take a stopped, filesystem-consistent backup of
|
||||
`/srv/thothii/workspace-registry` plus `/srv/thothii/data` and Pi state. Exclude
|
||||
`/srv/thothii/secrets` from the ordinary archive. Validate the descriptor with `thothctl update
|
||||
--check-only`, deploy the compatible image through `thothctl`, verify health/status, then resume
|
||||
proxy traffic.
|
||||
|
||||
For legacy descriptor migration, use a temporary review clone and the legacy transformer with absolute paths.
|
||||
Its schema-v1 output is `migration_required`; explicitly supply vector database/schema, collection
|
||||
|
||||
@@ -0,0 +1,368 @@
|
||||
# Install ThothII on a Linux server
|
||||
|
||||
This guide is for an installer with basic Linux administration and very basic Docker knowledge.
|
||||
It deploys the same Compose distribution used on a local PC: the mandatory application is exactly
|
||||
`frontend` plus `core`, and pinned Pi is inside `core`. The server does not need host Pi, Node.js,
|
||||
Python, Go, a browser shell, or a Docker socket inside either container.
|
||||
|
||||
Examples use `/srv/thothii` as an **example operator root**, `thoth.example.com` as a replaceable
|
||||
DNS name, and systemd-based command names. Adapt them to local policy. Complete the server session
|
||||
storage overlay and migration procedure before exposing a production installation.
|
||||
|
||||
## Deployment contract
|
||||
|
||||
- The generic Linux host and Docker Compose v2 are the deployment platform. No other
|
||||
application's Compose project, network, path, or runtime is required.
|
||||
- `frontend` is the only published service and defaults to `127.0.0.1:8080`; `core` has no host
|
||||
port. A host Nginx or Caddy listener terminates TLS and sends all application traffic to
|
||||
`frontend`, never directly to `core`.
|
||||
- DWH, vector database, embedding service, and LLM are external configurable endpoints. This
|
||||
remains true when they happen to run on the same physical server.
|
||||
- Application, Git, connector, and session credentials are protected host files mounted read-only
|
||||
under `/run/secrets`. Pi's protected auth JSON uses its dedicated read-only Pi mount. No secret
|
||||
value belongs in Git, images, browser storage, environment values, rendered Compose, or logs.
|
||||
- The Git-backed workspace registry is the source of truth. Installation-local bindings identify
|
||||
endpoints and secret-file paths; they do not replace the reviewed Git workspace descriptors.
|
||||
- `thothctl` is the operator CLI for start, stop, status, health, logs, Pi lifecycle, drain, and
|
||||
rollback. Raw Compose lifecycle commands bypass installation state and are unsupported.
|
||||
|
||||
Read [server workspace-registry installation](server-workspace-registry.md),
|
||||
[Pi management](pi-management.md), and the session-server comments in
|
||||
`deploy/compose.session-server.yaml.example` before the first public start.
|
||||
|
||||
## Service account and directories
|
||||
|
||||
The container runtime identity is fixed at UID/GID 10001. Reserve the same host ID for a dedicated
|
||||
non-login `thothii` account so bind-mounted ownership is obvious. Stop if either ID is already used
|
||||
by another account; choose a reviewed host mapping instead of changing the image identity.
|
||||
|
||||
```sh
|
||||
getent passwd 10001
|
||||
getent group 10001
|
||||
sudo useradd --system --uid 10001 --user-group --home-dir /srv/thothii \
|
||||
--create-home --shell /usr/sbin/nologin thothii
|
||||
```
|
||||
|
||||
The human operator who runs `thothctl` needs Docker access. On many installations membership in
|
||||
the `docker` group is effectively host-root access; grant it only according to site policy. The
|
||||
non-login `thothii` account owns application data and secrets but does not itself need Docker
|
||||
access.
|
||||
|
||||
Create explicit directories. `source` contains the clone; `operator` contains untracked path-only
|
||||
configuration; the three writable trees are bind-mounted into `core`; `secrets` contains regular
|
||||
files only. Backups are separate from live data.
|
||||
|
||||
```sh
|
||||
sudo install -d -o 10001 -g 10001 -m 0750 /srv/thothii/source
|
||||
sudo install -d -o 10001 -g 10001 -m 0700 /srv/thothii/operator
|
||||
sudo install -d -o 10001 -g 10001 -m 0700 /srv/thothii/secrets
|
||||
sudo install -d -o 10001 -g 10001 -m 0750 /srv/thothii/data
|
||||
sudo install -d -o 10001 -g 10001 -m 0750 /srv/thothii/pi-state
|
||||
sudo install -d -o 10001 -g 10001 -m 0750 /srv/thothii/workspace-registry
|
||||
sudo install -d -o root -g root -m 0700 /srv/thothii-backups
|
||||
```
|
||||
|
||||
Do not make `/srv/thothii` a shared application directory. The source checkout may be read by the
|
||||
operator, while secret contents and writable data remain limited to reviewed administrators and
|
||||
UID 10001.
|
||||
|
||||
## Firewall and network boundaries
|
||||
|
||||
Set `THOTH_SERVER_BIND=127.0.0.1`. Permit inbound TCP 80/443 only to the TLS proxy; port 80 should
|
||||
redirect to HTTPS. Do not open 8080 externally, and do not add a core port. If a separate proxy
|
||||
host is used, replace loopback with a private, firewalled address and allow only that proxy source.
|
||||
|
||||
Allow outbound DNS and HTTPS to the source/Git registries, plus only the configured ports for the
|
||||
Git remote, DWH, vector database, embedding service, LLM, session PostgreSQL, and any approved
|
||||
bastion. Docker's private `thothii` network carries only `frontend`↔`core` traffic. Do not attach
|
||||
the mandatory stack to another application's network.
|
||||
|
||||
After start, confirm the host listens as intended:
|
||||
|
||||
```sh
|
||||
sudo ss -lntp
|
||||
```
|
||||
|
||||
Expected public listeners are the proxy on 80/443 and the frontend on loopback 8080. There must be
|
||||
no host listener for core port 8787.
|
||||
|
||||
## Address co-resident external services
|
||||
|
||||
Endpoint values are resolved inside `core`. Therefore container 127.0.0.1 means the container
|
||||
itself, not the Linux host. Prefer real DNS names with TLS, authentication, and firewall policy,
|
||||
even for services on this physical server.
|
||||
|
||||
When DNS is unavailable for a host-published service, create an untracked override such as
|
||||
`/srv/thothii/operator/host-gateway.yaml` and add it to the installation descriptor:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
core:
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
```
|
||||
|
||||
Use `host.docker.internal` in the endpoint binding. The `host-gateway` mapping supplies routing;
|
||||
it does not bundle or trust the target service. Keep the target port bound/firewalled for Docker
|
||||
host access only. A stable internal DNS record is the preferred alternative.
|
||||
|
||||
Configure each boundary independently:
|
||||
|
||||
- DWH: read-only runtime identity, database/schema, verified TLS, and direct or REST endpoint.
|
||||
- Vector database: endpoint plus exact database/schema, collection, distance metric, and writer
|
||||
policy declared by the reviewed workspace.
|
||||
- Embedding service: endpoint and the collection/embedding pairing—model and dimensions must match
|
||||
the existing collection. Co-residence does not permit silently changing that pairing.
|
||||
- LLM: authenticated endpoint selected through deployment and Pi configuration.
|
||||
|
||||
Never add those services to ThothII's mandatory Compose files. Follow
|
||||
[the diagnostic protocol](../workspace-diagnostic-protocol.md) before enabling a workspace.
|
||||
|
||||
## Prepare operator files and secrets
|
||||
|
||||
Clone with LF line endings, then verify before every build:
|
||||
|
||||
```sh
|
||||
sudo -u thothii git -c core.autocrlf=false clone \
|
||||
https://github.example.invalid/your-org/ThothII.git /srv/thothii/source/ThothII
|
||||
cd /srv/thothii/source/ThothII
|
||||
sudo -u thothii git config --local core.autocrlf false
|
||||
bash scripts/verify-line-endings.sh
|
||||
```
|
||||
|
||||
Copy the path-only server environment and installation descriptor:
|
||||
|
||||
```sh
|
||||
sudo -u thothii cp deploy/env/server.env.example /srv/thothii/operator/server.env
|
||||
sudo -u thothii cp docs/install/examples/thothii-installation.server.yaml \
|
||||
/srv/thothii/operator/thothii-installation.yaml
|
||||
sudo chmod 0600 /srv/thothii/operator/server.env \
|
||||
/srv/thothii/operator/thothii-installation.yaml
|
||||
```
|
||||
|
||||
Replace every placeholder with an absolute path. Use exactly one Git transport override. For
|
||||
HTTPS, replace `deploy/compose.git-ssh.yaml` with `deploy/compose.git-https.yaml`. Keep the required
|
||||
session-server overlay and generated connector-secret override. Optional host-gateway or pinned
|
||||
image overrides go after them.
|
||||
|
||||
Create each credential as an independent regular file in `/srv/thothii/secrets`, owned by
|
||||
UID/GID 10001 and mode `0600`. The operator environment records only absolute `*_FILE` or
|
||||
`*_SOURCE` paths. Compose mounts application and connector targets read-only under `/run/secrets`;
|
||||
the frontend receives none. Do not print file contents while testing permissions.
|
||||
|
||||
```sh
|
||||
sudo find /srv/thothii/secrets -type f -exec chown 10001:10001 {} +
|
||||
sudo find /srv/thothii/secrets -type f -exec chmod 0600 {} +
|
||||
sudo find /srv/thothii/secrets -type f ! -user thothii -print
|
||||
sudo find /srv/thothii/secrets -type f ! -perm 0600 -print
|
||||
```
|
||||
|
||||
Add `THT_WORKSPACE_BINDINGS_ENV_FILE=/srv/thothii/operator/workspace-bindings.env` and the matching
|
||||
connector `*_SOURCE` paths to `server.env`. Generate
|
||||
`/srv/thothii/operator/connector-secrets.server.yaml` as described in
|
||||
[server workspace-registry installation](server-workspace-registry.md). Secret values must never
|
||||
be pasted into `server.env`, the installation YAML, a URL, or a shell argument.
|
||||
|
||||
## Build locally or select pinned images
|
||||
|
||||
Choose one image source. For a source build, the repository's reproducible launcher builds the
|
||||
same `core` and `frontend` images used by the local profile. It requires only Git, Docker, and
|
||||
Compose; copy the reviewed path-only server environment to the launcher's untracked input first:
|
||||
|
||||
```sh
|
||||
cd /srv/thothii/source/ThothII
|
||||
sudo -u thothii cp /srv/thothii/operator/server.env deploy/env/local.env
|
||||
bash scripts/build-local.sh
|
||||
```
|
||||
|
||||
The printed local-profile start command is not the server start command; use `thothctl` below.
|
||||
|
||||
Alternatively, create a reviewed untracked override with release images pinned by immutable
|
||||
digest. Mutable tags are not a production pin:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
core:
|
||||
build: !reset null
|
||||
image: registry.example.com/thothii/core@sha256:<64-lowercase-hex-digits>
|
||||
frontend:
|
||||
build: !reset null
|
||||
image: registry.example.com/thothii/frontend@sha256:<64-lowercase-hex-digits>
|
||||
```
|
||||
|
||||
Add that absolute file last in `overrides`. Both images must come from one compatible release; the
|
||||
core image must retain the declared Pi version labels checked by `thothctl pi doctor`. Pull access
|
||||
belongs in the host Docker credential store, not in Compose or the installation descriptor.
|
||||
|
||||
## Install thothctl
|
||||
|
||||
Build the operator binaries with Docker. No Go installation or Go knowledge is required:
|
||||
|
||||
```sh
|
||||
cd /srv/thothii/source/ThothII
|
||||
bash scripts/build-thothctl.sh
|
||||
sudo install -o 10001 -g 10001 -m 0755 dist/thothctl/thothctl-linux-amd64 \
|
||||
/srv/thothii/operator/thothctl
|
||||
```
|
||||
|
||||
Use `thothctl-linux-arm64` on an ARM64 server. Set these variables in the maintenance shell; do
|
||||
not source `server.env` as shell code:
|
||||
|
||||
```sh
|
||||
THTCTL=/srv/thothii/operator/thothctl
|
||||
INSTALLATION=/srv/thothii/operator/thothii-installation.yaml
|
||||
"$THTCTL" --installation "$INSTALLATION" --help
|
||||
"$THTCTL" --installation "$INSTALLATION" update --check-only
|
||||
```
|
||||
|
||||
Every operator command includes the descriptor explicitly. This preserves the installation's
|
||||
profile, overrides, project identity, and durable current-image selector. The general form is
|
||||
`thothctl --installation /absolute/path/thothii-installation.yaml <command>`.
|
||||
|
||||
## Start and verify readiness
|
||||
|
||||
Keep the TLS proxy stopped or firewalled during bootstrap:
|
||||
|
||||
```sh
|
||||
"$THTCTL" --installation "$INSTALLATION" start
|
||||
"$THTCTL" --installation "$INSTALLATION" status
|
||||
"$THTCTL" --installation "$INSTALLATION" doctor
|
||||
curl --fail http://127.0.0.1:8080/health
|
||||
"$THTCTL" --installation "$INSTALLATION" pi doctor
|
||||
"$THTCTL" --installation "$INSTALLATION" pi test
|
||||
```
|
||||
|
||||
`/health` proves process liveness. Readiness additionally requires both healthy services, a valid
|
||||
Pi provider/model smoke, a successful Git registry pull with an active validated snapshot, valid
|
||||
workspace diagnostics, and ready session PostgreSQL. Use the authenticated Workspace Management
|
||||
page to pull and diagnose the reviewed workspace. A liveness response alone is not release
|
||||
approval.
|
||||
|
||||
After configuring the proxy, open <https://thoth.example.com> in a browser. Verify an unauthenticated
|
||||
request is denied or redirected by the real identity provider, an authorized user can load the
|
||||
same-origin UI and `/api`, an unauthorized user is denied, and an administrator alone can open Pi
|
||||
Management. Keep port 8080 inaccessible from other hosts.
|
||||
|
||||
## Configure TLS and upstream authentication
|
||||
|
||||
Choose [Nginx](reverse-proxy-nginx.md) or [Caddy](reverse-proxy-caddy.md). Both examples terminate
|
||||
TLS and proxy only to loopback `frontend`. They preserve SSE and clear client-supplied identity
|
||||
headers before authentication.
|
||||
|
||||
The authentication gateway must validate a real login/session and return normalized issuer,
|
||||
subject, display-name, and admin claims only after success. Merely forwarding those headers does
|
||||
not authenticate anyone. Do not enable `AUTH_MODE=upstream` on a listener reachable around the
|
||||
trusted proxy, and never expose `core`.
|
||||
|
||||
## Operate Pi, drain, and roll back
|
||||
|
||||
Configure only closed provider/model/reasoning choices. Credentials remain protected files:
|
||||
|
||||
```sh
|
||||
"$THTCTL" --installation "$INSTALLATION" pi status
|
||||
"$THTCTL" --installation "$INSTALLATION" pi configure
|
||||
"$THTCTL" --installation "$INSTALLATION" pi doctor
|
||||
"$THTCTL" --installation "$INSTALLATION" pi logs
|
||||
```
|
||||
|
||||
Before an update, announce maintenance and ask users to finish active work. `--drain` closes new
|
||||
admission and waits until no active sessions remain; it does not discard sessions. Build-source
|
||||
and registry-source examples are:
|
||||
|
||||
```sh
|
||||
"$THTCTL" --installation "$INSTALLATION" pi update \
|
||||
--version 0.81.0 --source build --yes --drain
|
||||
"$THTCTL" --installation "$INSTALLATION" pi update \
|
||||
--version 0.81.0 --source pull \
|
||||
--image registry.example.com/thothii/core@sha256:<64-lowercase-hex-digits> \
|
||||
--yes --drain
|
||||
```
|
||||
|
||||
The transaction recreates only `core`, preserves volumes, verifies health/configuration/Pi, and
|
||||
automatically attempts rollback after a post-mutation failure. For interrupted or ambiguous state:
|
||||
|
||||
```sh
|
||||
"$THTCTL" --installation "$INSTALLATION" pi maintenance status
|
||||
"$THTCTL" --installation "$INSTALLATION" pi rollback --yes
|
||||
"$THTCTL" --installation "$INSTALLATION" pi maintenance recover --yes
|
||||
```
|
||||
|
||||
Leave maintenance active if rollback cannot be verified. Preserve `.thothctl/<installation-id>/`
|
||||
recovery state, repair the reported host/configuration issue, and rerun rollback or maintenance
|
||||
recovery. Never delete or edit `current-image.yaml` or `update-state.json` to force progress.
|
||||
|
||||
## Back up and restore
|
||||
|
||||
Back up before source, workspace, session-schema, or Pi changes. Drain work, stop the installation,
|
||||
record `git rev-parse HEAD`, image digests, and `thothctl status`, then archive the three bind trees
|
||||
with numeric ownership. Do not include live secrets in this ordinary archive.
|
||||
|
||||
```sh
|
||||
"$THTCTL" --installation "$INSTALLATION" stop
|
||||
BACKUP=/srv/thothii-backups/2026-08-05
|
||||
sudo install -d -o root -g root -m 0700 "$BACKUP"
|
||||
sudo tar --numeric-owner --xattrs --acls -C /srv/thothii -czf "$BACKUP/runtime-data.tgz" \
|
||||
data pi-state workspace-registry
|
||||
sudo sha256sum "$BACKUP/runtime-data.tgz" >"$BACKUP/SHA256SUMS"
|
||||
```
|
||||
|
||||
Back up the installation descriptor, path-only environment, generated overrides, source revision,
|
||||
and secret files to separate encrypted access-controlled storage. Database-backed production
|
||||
sessions require their own PostgreSQL-native consistent backup; the local bind tree is not a
|
||||
substitute. Test both restore paths periodically.
|
||||
|
||||
Restore only while stopped. Verify the checksum, extract first into a new empty root, inspect
|
||||
ownership and expected registry layout, then retain the old trees by renaming them before placing
|
||||
the restored set. This keeps the previous state recoverable:
|
||||
|
||||
```sh
|
||||
RESTORE=/srv/thothii-restore-2026-08-05
|
||||
sudo install -d -o root -g root -m 0700 "$RESTORE"
|
||||
sudo sha256sum --check /srv/thothii-backups/2026-08-05/SHA256SUMS
|
||||
sudo tar --numeric-owner --xattrs --acls -C "$RESTORE" \
|
||||
-xzf /srv/thothii-backups/2026-08-05/runtime-data.tgz
|
||||
sudo test -d "$RESTORE/workspace-registry/repo"
|
||||
sudo test -d "$RESTORE/workspace-registry/snapshots"
|
||||
```
|
||||
|
||||
During the reviewed restore window, move each old tree to a timestamped sibling, move the matching
|
||||
restored tree into `/srv/thothii`, restore the PostgreSQL session backup from the same recovery
|
||||
point, and keep the proxy closed. Run `update --check-only`, `start`, `doctor`, `pi test`, registry
|
||||
status, workspace diagnostics, and a known historical session before reopening traffic. Never
|
||||
merge an archive into a non-empty tree.
|
||||
|
||||
## Diagnostics
|
||||
|
||||
Begin with bounded, sanitized installation-aware commands:
|
||||
|
||||
```sh
|
||||
"$THTCTL" --installation "$INSTALLATION" status
|
||||
"$THTCTL" --installation "$INSTALLATION" doctor
|
||||
"$THTCTL" --installation "$INSTALLATION" logs
|
||||
"$THTCTL" --installation "$INSTALLATION" pi status
|
||||
"$THTCTL" --installation "$INSTALLATION" pi doctor
|
||||
"$THTCTL" --installation "$INSTALLATION" pi test
|
||||
"$THTCTL" --installation "$INSTALLATION" pi logs
|
||||
"$THTCTL" --installation "$INSTALLATION" pi maintenance status
|
||||
```
|
||||
|
||||
Use the authenticated Workspace Management status and diagnostic actions for Git revision,
|
||||
degraded snapshot, bindings, DWH, vector, and embedding checks. Review proxy logs separately, but
|
||||
configure both proxy and log shipping to exclude cookies, authorization data, identity payloads,
|
||||
query strings, and secret values. Do not render Compose or print an environment as a diagnostic.
|
||||
|
||||
Typical boundaries are: `doctor` for Docker/Compose/LF/volume/service health; `pi doctor` for image
|
||||
and provider/model integrity; registry status for Git/snapshot health; workspace diagnostics for
|
||||
external service identity; and the proxy/identity provider for login failures.
|
||||
|
||||
## Data-preserving uninstall
|
||||
|
||||
Drain and stop through `thothctl`, take and verify one final backup, disable the TLS proxy route,
|
||||
and remove only this installation's stopped `frontend` and `core` containers and optional images
|
||||
by their exact Compose project labels. Keep `/srv/thothii/data`, `pi-state`,
|
||||
`workspace-registry`, `operator`, protected secrets, database backups, and the installation
|
||||
descriptor if reinstallation is possible. Do not prune global Docker data.
|
||||
|
||||
Do **not** run `docker compose down --volumes`; it deletes persistent application data. Reusing the
|
||||
same protected descriptor path preserves the `thothctl` installation identity and allows a later
|
||||
compatible source checkout to reconnect the retained state.
|
||||
Reference in New Issue
Block a user