mptyl/ThothII
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs: describe read-only workspace runtime configuration

Browse Source
This commit is contained in:
marcopan
2026-08-14 18:01:02 +02:00
parent 422f1d47b4
commit ab33e0ed0a
26 changed files with 511 additions and 911 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/install/server-workspace-registry.md
+109 -352
View File
@@ -1,371 +1,128 @@
# Server workspace-registry installation
# Server workspace repository installation
This is the production workspace-registry companion to [the autonomous Linux server guide](server.md).
The application image is read-only, secrets are mounted read-only, and sessions use immutable
Git-validated snapshots. Expose the application only behind an authenticated same-origin reverse
proxy; never publish the core port directly.
This manual supplements [server.md](server.md). A server installation reads one remote Git
repository hosted by GitHub, GitLab, Gitea, Bitbucket, or another Git server. ThothII fetches and
validates complete revisions but never edits, commits, pushes, or publishes workspace source.
## Architecture ownership contract
| Component | Ownership | Operator contract |
| --- | --- | --- |
| DWH | External | Approved installation/server endpoint; not part of the private semantic Compose stack. |
| LLM | External | Approved installation/server endpoint or provider policy outside the semantic stack. |
| Qdrant | Internal | Mandatory private Compose semantic service; persistent `qdrant-data` volume. |
| Ollama embedding | Internal | Mandatory private Compose semantic service for `qwen3-embedding:0.6b`. |
## Host preprocessing (P2)
The installed native `thothctl` is the only host entrypoint for workspace preprocessing
(introspection+LSH, FK review, schema indexing, HTTP Evidence). Use
`thothctl --installation <thothii-installation.yaml> workspace <command> --workspace <id> [--json]`
per `docs/contracts/workspace-preprocessing-cli.md` and the P2 walkthrough in
`docs/testing/p2-p6-manual-verification.md`. Preprocessing runs through the profile-gated
`workspace-maintenance` Compose service; it never starts a backend/Pi/frontend listener and never
attaches Git credentials.
## Effective configuration and `.tht-dwh` (P3)
Prepared DWH generations are reusable and safe: `thothctl` and the application derive the same
canonical effective configuration and logical identity, so prepared work is reused when nothing
relevant changed and refused when the database/endpoint/identity changed. See
`docs/contracts/tht-dwh.md` for generations, `OWNER.json`, `ACTIVE`, fingerprints, migration and
recovery. A content-only or Evidence-only change never forces a full re-introspection.
## Service account, storage, and firewall
Create a dedicated host service account and an operator root such as `/srv/thothii`. The core
container is non-root UID/GID `10001` (`thoth`), so give that identity read/write ownership before
first startup. Keep storage separated:
```text
/srv/thothii/data/ # settings, session data, Pi state as applicable
/srv/thothii/workspace-registry/ # repo/, snapshots/, state/, locks/
/srv/thothii/secrets/ # Git and connector secret files, setgid mode 2750
/srv/thothii/operator/ # untracked operator files, setgid mode 2770
```
After cloning the source and before the first render/start, initialize the empty Pi-state root with
the repository setup command:
```sh
sudo /srv/thothii/source/ThothII/scripts/prepare-server-pi-state.sh \
/srv/thothii/pi-state 10001 10001
```
The active server profile mounts that writable parent at `/home/thoth/.pi` and overlays three
read-only files beneath `agent/`. The setup command atomically creates the required hidden regular
targets with runtime ownership without copying secret or tracked file contents into writable
state. Rerun it after a restore and before Compose or `thothctl` startup; it is idempotent and does
not overwrite existing targets.
Permit outbound TCP only to approved Git/Gitea, DWH, LLM, and optional bastion endpoints.
Qdrant and Ollama run inside the Compose stack. Allow inbound traffic only from the reverse
proxy/Docker network. Do not give the runtime service
account Gitea administration, database-superuser rights, or a shell in the Git host.
## Gitea and remote Git setup
Create a private Gitea (or compatible Git) repository such as `platform/thoth-workspaces`. Protect
`main` according to the release policy and grant the ThothII publisher only the intended repository
scope. Commit the curator-owned root catalog `thoth-workspaces.yaml`, workspace descriptors under
`<id>/workspace.yaml`, curated embedded Evidence under `<id>/evidence/**`, and generated public
artifacts only at `workspace-docs/<id>/README.md` and
`workspace-docs/<id>/contract.env.example`; do not commit installation bindings or secret material.
`thoth-workspaces.yaml` uses the `schema_version` value `1` and the ordered `workspaces` list of
`{id, name, description?}` entries. It is authoritative for workspace ID, name, description, and
display order. `<id>/workspace.yaml` must match that metadata exactly, and `workspace-docs`
remains the reserved top-level generated-docs directory.
For SSH, create a least-privilege deploy key, record Gitea's host key in managed known-hosts, and
use `ssh://git@git.example.invalid/platform/thoth-workspaces.git`. For HTTPS, create a scoped
machine credential in the secret manager and mount the Gitea/private CA separately. Never use a
Gitea admin credential in the application.
Bootstrap an empty remote from a temporary review clone only after the catalog, descriptors, and
generated public artifacts have been reviewed; commit and push `main`. The running server is not a
descriptor authoring or conversion environment.
## Curator flow for shared-registry Evidence
Follow this order; the [canonical Evidence contract](../contracts/workspace-evidence-v3.md) defines
the source shapes and safety boundary.
1. Clone the one shared registry, or update the review clone with `git pull --ff-only`.
2. Keep `thoth-workspaces.yaml` curator-owned. It uses the `schema_version` value `1` and the ordered
`workspaces` list of `{id, name, description?}` entries; it is authoritative for workspace ID,
name, description, and display order.
3. For an existing workspace, edit `<id>/workspace.yaml` and any embedded `<id>/evidence/**`, then
commit and push.
4. For a new workspace, add the catalog slot first. If you want ThothII to bootstrap the
descriptor, leave `<id>/workspace.yaml` absent, commit and push, then pull that commit into the
installation; the slot appears as `configuration_required`.
5. The API may create `<id>/workspace.yaml` only when the catalog slot already exists and no Git
object exists at that path in the exact pulled base commit.
6. After bootstrap, existing descriptors change only through curator Git commit/push and
installation pull. The API never writes `thoth-workspaces.yaml` or `<id>/evidence/**`.
7. Inspect `workspace-docs/<id>/contract.env.example` and `workspace-docs/<id>/README.md`.
8. Provision only the selected Evidence `*_FILE` files outside Git and strictly below a root in
`THT_WORKSPACE_SECRET_ROOTS`; add matching host-only `*_SOURCE` paths for the generated
connector override.
9. Render or acquire the runtime config, then run `tht config check -c <path>`.
10. Stop: P2/P6 later performs preprocessing and materialization.
For example, separate signed-HTTP and static-S3 workspaces can use these host-only connector source
paths:
```dotenv
THT_WS_SIGNED_HTTP_EVIDENCE_SIGNED_URLS_SOURCE=/srv/thothii/secrets/signed-http-evidence-urls.json
THT_WS_STATIC_S3_EVIDENCE_ACCESS_KEY_SOURCE=/srv/thothii/secrets/static-s3-evidence-access-key
THT_WS_STATIC_S3_EVIDENCE_SECRET_KEY_SOURCE=/srv/thothii/secrets/static-s3-evidence-secret-key
THT_WS_STATIC_S3_EVIDENCE_SESSION_TOKEN_SOURCE=/srv/thothii/secrets/static-s3-evidence-session-token
```
The descriptor and declared filesystem root are validated at the same registry commit. The
browser shows a read-only Evidence summary, while exports omit Evidence bytes.
## Git credentials, CA, SSH key, and known-hosts mounts
Use the secret manager or a protected host-only procedure to create independent regular files under
`/srv/thothii/secrets`. As established in the server guide, use owner UID 10001, group
`thothii-ops`, file mode `0640`, and setgid directory mode `2750`. This lets the UID 10001
container and the reviewed Docker operator running `thothctl` read the files without making them
public. These path-only variables are mounted read-only by Compose:
```dotenv
THT_WORKSPACE_GIT_CREDENTIALS_FILE=/srv/thothii/secrets/git-credentials
THT_WORKSPACE_GIT_CA_FILE=/srv/thothii/secrets/git-ca.pem
THT_WORKSPACE_GIT_SSH_KEY_FILE=/srv/thothii/secrets/git-ssh-key
THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=/srv/thothii/secrets/git-known-hosts
PI_AUTH_FILE=/srv/thothii/secrets/pi-auth.json
THT_SECRETS_FILE=/srv/thothii/secrets/thothii.secrets
```
Use the credential file for HTTPS, or key and known-hosts for SSH. The base server Compose file
mounts neither transport; add exactly one `deploy/compose.git-https.yaml`
or `deploy/compose.git-ssh.yaml` override. Strict host-key checking stays enabled
and Git stderr is not exposed by the API. Rotate by atomically replacing the secret file,
restarting `core`, and performing pull/status; never put the material in an environment variable or
rendered Compose output.
## Shared Git values, local bindings, and secret files
Git describes workspace schema, immutable ID, DWH identity, semantic-index dimensions and
distance, internal embedding contract, and LLM policy. The installation supplies remote/branch/installation
ID and one absolute `THT_WORKSPACE_BINDINGS_ENV_FILE` containing only `THT_WS_*` transport,
endpoint, user, and `/run/secrets/...` path bindings. The base Compose loads that file only into
`core`. Secret contents are only in host files, never the values stored in Git or browser-local
drafts.
The runtime registry layout is persistent and must be backed up together:
```text
/data/workspace-registry/repo/
/data/workspace-registry/snapshots/
/data/workspace-registry/state/
/data/workspace-registry/locks/
```
Variable names derive from the immutable ID: `north-star-research` becomes `NORTH_STAR_RESEARCH`, producing
`THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE`. Keep the bindings file limited to DWH transport,
endpoint, user, and secret-path values; internal semantic services are supplied by Compose and do
not require workspace-local vector or embedding bindings.
Copy [the bindings env example](examples/workspace-bindings.env.example) to the protected operator
directory. Every path-valued `*_FILE` entry needs an absolute host-only `*_SOURCE` path. Generate
the untracked connector override from those files during bootstrap; do not copy or maintain a
workspace-specific Compose override. Operator-managed path-only files use owner UID 10001, group
`thothii-ops`, and mode `0660`; secret files remain `0640` and non-group-writable.
## Direct PostgreSQL, REST, and SSH tunnel bindings
Select only a transport allowed by canonical YAML; preserve database/schema/collection, model,
dimensions, and distance as Git-shared identity.
```dotenv
# Direct PostgreSQL with verified native TLS if a CA path is supplied.
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct
THT_WS_NORTH_STAR_RESEARCH_DWH_HOST=dwh.internal.example
THT_WS_NORTH_STAR_RESEARCH_DWH_PORT=5432
THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
```
```dotenv
# REST needs API-key file paths only when the descriptor declares authenticated diagnostics.
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=rest_api
THT_WS_NORTH_STAR_RESEARCH_DWH_BASE_URL=https://dwh.internal.example
THT_WS_NORTH_STAR_RESEARCH_DWH_API_KEY_FILE=/run/secrets/north-star-research-dwh-api-key
```
```dotenv
# SSH tunnel diagnostic only; runtime sessions are fail-closed in this release.
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=ssh_tunnel
THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_HOST=bastion.internal.example
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PORT=22
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_USER=thoth_tunnel
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PRIVATE_KEY_FILE=/run/secrets/north-star-research-dwh-tunnel-key
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_KNOWN_HOSTS_FILE=/run/secrets/north-star-research-dwh-known-hosts
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_HOST=dwh.internal.example
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_PORT=5432
```
REST diagnostics refuse private per-request CAs
rather than disable verification; use runtime-trusted HTTPS or verified direct/SSH native TLS. See
the [diagnostic protocol](../workspace-diagnostic-protocol.md) for its read-only checks and optional
reversible writer probe.
An SSH connector can be tested with strict host-key and target verification, but it intentionally
returns `workspace_not_activatable`; configure direct or REST transport before starting sessions.
The Git registry itself may still use SSH normally.
## Same-origin reverse proxy, bootstrap, and health
Use the repository's canonical `compose.yaml` plus `deploy/compose.server.yaml`; they always
start `frontend`, `core`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`.
Startup is CPU-first; use `THOTH_ENABLE_EMBEDDING_GPU=1` only when the server intentionally
exposes a supported GPU device to Docker. Qdrant is a derived but persistent index, and the
Ollama model cache persists the exact `qwen3-embedding:0.6b` model (`1024` dimensions, cosine
distance) for offline reuse. Do not copy or maintain a standalone
application Compose file. Copy `docs/install/examples/thothii-installation.server.yaml` to the
protected operator directory and preserve its required session-server overlay, exactly one Git
transport override, and generated connector-secret override.
Review `deploy/workspaces/server-sessions.yaml.example`, materialize it as a protected host file,
and set its absolute `THT_SERVER_WORKSPACE_CONFIG` path. Copy the bindings env example into the
operator directory, then set absolute `PI_AUTH_FILE`, `THT_SECRETS_FILE`,
`THT_WORKSPACE_BINDINGS_ENV_FILE`, and connector `*_SOURCE` paths. The same operator env must set
`THT_SESSION_DB_HOST`, `THT_SESSION_DB_NAME`, `THT_SESSION_RUNTIME_USER`,
`THT_SESSION_RUNTIME_PASSWORD_SOURCE`, and `THT_SESSION_CA_SOURCE`;
`deploy/compose.session-server.yaml.example` wires `postgres`, `verify-full`, and separate
runtime/CA secret targets under `/run/secrets`. This public server profile never falls back to
filesystem sessions. The path-only environment file is not shell code; do not source it.
Generate the connector override, then use the installation-aware operator CLI. Building
`thothctl` requires only Docker and no Go knowledge. From a trusted maintenance shell:
```sh
umask 0007
THT_SOURCE_ROOT=/srv/thothii/source/ThothII
THT_OPERATOR_ENV=/srv/thothii/operator/server.env
THT_WORKSPACE_BINDINGS_ENV_FILE=/srv/thothii/operator/workspace-bindings.env
THT_CONNECTOR_OVERRIDE=/srv/thothii/operator/connector-secrets.server.yaml
THTCTL=/srv/thothii/operator/thothctl
INSTALLATION=/srv/thothii/operator/thothii-installation.yaml
sudo "$THT_SOURCE_ROOT/scripts/prepare-server-pi-state.sh" /srv/thothii/pi-state 10001 10001
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \
-f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.server.yaml" \
-f "$THT_SOURCE_ROOT/deploy/compose.session-server.yaml.example" config --quiet
"$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" \
--bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" \
--operator-env "$THT_OPERATOR_ENV" --output "$THT_CONNECTOR_OVERRIDE"
"$THTCTL" --installation "$INSTALLATION" update --check-only
"$THTCTL" --installation "$INSTALLATION" start
"$THTCTL" --installation "$INSTALLATION" status
"$THTCTL" --installation "$INSTALLATION" doctor
"$THTCTL" --installation "$INSTALLATION" pi doctor
"$THTCTL" --installation "$INSTALLATION" pi test
```
Configure [Nginx](reverse-proxy-nginx.md) or [Caddy](reverse-proxy-caddy.md) so frontend and `/api`
share one TLS origin. The proxy authenticates first, clears client identity headers, and carries
only successful authentication claims over the private `X-Thoth-Trusted-*` hop. Forwarding claims
without authenticating the request is not an identity boundary.
`/health` is liveness. The authenticated Workspace Management page's registry status verifies
branch/head/degraded state and the active validated snapshot; its workspace listing verifies
application access. A catalog-only slot with no descriptor reports `configuration_required` and is
not ready for sessions until either the curator commits `<id>/workspace.yaml` or the one-time
bootstrap create flow writes it. A server with no active snapshot is not ready for workspace
sessions even if liveness succeeds.
## Pull, publish, upgrade, backup, and recovery
Use the authenticated Workspace Management UI or `POST /workspace-registry/pull` to fetch later
curator revisions. Existing curated workspaces are read-only in the browser. Use the UI to inspect
status, validate a workspace, test it on this installation, and optionally create one bootstrap
descriptor for a pulled `configuration_required` slot. After that first descriptor exists, change
it only through curator Git commit/push and installation pull; never edit `repo/` inside a running
volume.
For upgrades, record active status/head, finish active work, use the documented `thothctl pi update
--drain` transaction when Pi/core changes, and take a stopped, filesystem-consistent backup of
`/srv/thothii/workspace-registry` plus `/srv/thothii/data` and Pi state. Exclude
`/srv/thothii/secrets` from the ordinary archive. Validate the descriptor with `thothctl update
--check-only`, deploy the compatible image through `thothctl`, verify health/status, then resume
proxy traffic. If you are upgrading an older P1 registry, apply the reviewed migration in
[`docs/migrations/p1-to-p1-1-registry-layout.md`](../migrations/p1-to-p1-1-registry-layout.md)
and upgrade ThothII only after that commit is pushed.
<!-- workspace-descriptor-contract:start -->
Schema v3 is the only accepted workspace descriptor. Schema v1 and v2 workspace descriptors are
rejected before activation. Candidate snapshot validation makes initial activation or a pull fail
atomically and leaves the prior active snapshot unchanged. There is no in-product migrator or
automatic conversion. The repository must already contain reviewed v3 descriptors. One workspace
owns one Qdrant collection; schema, Evidence, and Memory records share it and stay separated by
payload `kind`.
<!-- workspace-descriptor-contract:end -->
If source material needs conversion, perform it outside ThothII in a separate reviewed process.
Commit only the resulting reviewed v3 descriptors. That external process must not import `${ENV}`
values, secret values, certificates, keys, or secret files into the repository.
| DWH | External | Configure the external endpoint and complete runtime credentials through the authenticated GUI. |
| LLM | External | Configure the external endpoint and model policy under installation control. |
| Qdrant | Internal | Compose runs private Qdrant and persists `qdrant-data`; include it in Qdrant backup/restore. |
| Ollama embedding | Internal | Compose runs private Ollama with `qwen3-embedding:0.6b`. |
## Semantic index ownership contract
| Scope | Ownership rule | Isolation rule |
| --- | --- | --- |
| Workspace semantic index | Each workspace reserves a single Qdrant collection. | Schema, Evidence, and Memory share that one collection and stay separated by payload `kind`. |
| Workspace semantic index | Each workspace keeps exactly one Qdrant collection reserved for itself. | Schema, Evidence, and memory records share that one collection and are separated by the `kind` payload. |
After valid bootstrap, Git outage retains the active snapshot with `degraded: true`. Repair
egress/DNS/CA/credentials, pull, and confirm healthy status. Roll back a bad descriptor through a
reviewed Git revert/release branch, advance the remote through normal policy, pull it, and confirm
the replacement snapshot. Restore a registry backup only while stopped and with a compatible image;
do not delete snapshots as a rollback shortcut.
The fixed semantic contract is 1024 dimensions and cosine distance. DWH and LLM remain external;
Qdrant, Ollama, and `embedding-model-init` remain private internal services.
## Troubleshooting and snapshot rollback
## Service account, storage, and firewall
| Stable code | Meaning and safe response |
| --- | --- |
| `workspace_invalid` | Invalid descriptor/path/snapshot; restore a reviewed canonical Git revision. |
| `binding_missing` | Missing/invalid local value or readable `*_FILE`; correct mount and permissions. |
| `workspace_not_activatable` | Bindings/diagnostics cannot activate; use sanitized fields to fix selected transport. |
| `workspace_stale` | Checkout changed/locked; stop concurrent registry work, never force Git in the volume. |
| `workspace_conflict` | Draft base stale; pull, resolve, validate, and retry after the curator pull/bootstrap flow. |
| `git_unavailable` | Storage/remote/DNS/firewall/lock failed; preserve degraded active state while repairing it. |
| `git_auth_failed` | SSH/HTTPS material rejected or unreadable; rotate/fix file without printing it. |
| `git_non_fast_forward` | Checkout diverged; reconcile through registry workflow and branch policy. |
| `git_push_rejected` | Gitea policy rejected the curator push or docs sync commit; review hooks/branch protection. |
| `connector_unavailable` | DNS/TLS/auth/resource identity failed; check egress and local bindings. |
| `semantic_index_incompatible` | Collection/model/dimensions/distance differs; perform explicit index migration. |
Run the application as the documented unprivileged service account. Keep the source checkout,
operator files, application data, and workspace authoring clone separate:
If the current snapshot is valid but Git remains down, continue only work safe on that pinned
revision and monitor status. If snapshots are missing or corrupt, stop the service, restore the
newest verified registry backup, start it privately, verify status, and then reopen proxy traffic.
A first-bootstrap failure has no fallback: repair remote trust rather than creating an unreviewed
runtime checkout.
## Qdrant backup/restore and cache recovery
Use the repository helpers for Qdrant backup/restore:
```sh
./scripts/vector-backup.sh --project-name thothii --output /secure/backups/thoth-qdrant-2026-08-08.tar
./scripts/vector-restore.sh --project-name thothii --input /secure/backups/thoth-qdrant-2026-08-08.tar --confirm-project thothii
```text
/srv/thothii/app/ # ThothII source release
/srv/thothii/operator/ # installation descriptor and protected Git files
/srv/thothii/data/ # application data, encrypted workspace vault, sessions
/srv/workspace-authoring/ # optional curator clone; never mounted into ThothII
```
Qdrant backup/restore targets exactly one labeled `qdrant-data` volume for the named Compose
project. Restore requires the exact repeated project confirmation, validates the archive before
stopping `qdrant`, stages rollback content, and restores semantic storage in place only for that
project-scoped volume. Before recovery, the registry must already contain a reviewed v3 descriptor
revision compatible with the restored collection. The helper does not restore descriptors, rename
collections, or resolve semantic-index incompatibilities.
Expose only the authenticated same-origin reverse proxy. Keep `core`, Qdrant, and Ollama private.
The Ollama model cache is a recoverable local cache, not the canonical semantic source of truth.
You may back up `embedding-models` for faster offline recovery, but a cache loss is recoverable by
re-pulling `qwen3-embedding:0.6b` through `embedding-model-init`.
## Prepare and publish a workspace source
Only the Git remote, DWH, LLM, and optional bastion endpoints stay external.
Create a local workspace in the external authoring repository, which contains
`thoth-workspaces.yaml`, one
`<workspace-id>/workspace.yaml` per catalog entry, optional repository-owned Evidence, and optional
curated schema annotations. It contains no credentials.
Publishing belongs to the curator workflow outside ThothII: validate, review, commit, and push the
source revision to the configured protected branch. Grant the ThothII service only read access.
<!-- workspace-descriptor-contract:start -->
Schema v3 is the only accepted workspace descriptor.
Schema v1 and v2 workspace descriptors are rejected before activation.
<!-- workspace-descriptor-contract:end -->
## Configure the remote Git repository
Copy `docs/install/examples/thothii-installation.server.yaml` to
`/srv/thothii/operator/thothii-installation.yaml`. Set `workspaceRepository.remote`, `.branch`, and
`.access`, then select exactly one Git transport override. The remote and credential are normally
repository-scoped read-only deploy credentials.
For SSH, mount a private key and pinned known-hosts file. For HTTPS, mount a Git credentials file
and the required CA chain. These installation credentials are not editable in Workspace
management and are never exposed by the API.
## Start and update the installation
Use the installation-aware controller described by `server.md`:
```bash
THTCTL=/srv/thothii/operator/thothctl
INSTALLATION=/srv/thothii/operator/thothii-installation.yaml
"$THTCTL" --installation "$INSTALLATION" start
"$THTCTL" --installation "$INSTALLATION" doctor
```
The descriptor composes `compose.yaml`, `deploy/compose.server.yaml`, the server session-storage
override, and one read-only Git transport override. **Update workspace repository** fetches a
candidate on the server; it does not transfer workspace files to the operator workstation.
## Complete runtime secrets in Workspace management
After repository activation, an authenticated user can:
1. Review the configured repository identity and update it without selecting a workspace.
2. Select a workspace to see the DWH/Evidence credential fields required by its connector modes.
3. Blind-save or rotate values; returned responses contain status only.
4. Run **Validate workspace source** and then test its configured connections.
5. Forget an obsolete value after dependent sessions and jobs have ended.
The backend encrypts values in `/data/workspace-secrets`, including the installation-specific
master key. The server profile persists that directory inside `THT_DATA_ROOT`; no workspace YAML
path depends on Linux, macOS, or Windows. Plaintext exists only in a restrictive temporary file
for the duration of a diagnostic, session, or maintenance lease.
Authorization is intentionally the current installation-wide authenticated-user policy. A future
role model or external secret manager can replace that policy without changing workspace source.
## Validation and activation behavior
Repository update is all-or-nothing: ThothII fetches the configured branch, validates catalog,
descriptors, Evidence paths, and cross-workspace invariants at one commit, then atomically activates
the complete candidate. A rejected candidate never replaces the previous active snapshot. The
application-owned checkout and snapshots are read-only runtime state.
Validation proves descriptor and repository structure. **Test connections** additionally
materializes the current runtime secrets and contacts only the selected workspace's configured
DWH/Evidence endpoints. Failure does not modify or publish workspace source.
## Backup, rotation, and recovery
Back up application data and Qdrant consistently. Qdrant backup/restore must cover `qdrant-data`;
application recovery must cover repository snapshots/state, sessions, settings, Pi state, and the
entire encrypted `/data/workspace-secrets` directory. Store backup encryption keys separately and
test restore procedures without production traffic.
Rotate DWH/Evidence credentials through Workspace management. Rotate Git access by atomically
replacing its protected installation file and restarting `core`. Recover a bad source revision by
reverting or correcting it in the external authoring repository and updating again.
## Troubleshooting
| Symptom | Meaning and action |
| --- | --- |
| Git authentication failed | Verify repository-scoped read permission, branch, key/token, CA, and host-key pinning. |
| Candidate validation failed | Correct the source repository; the prior active commit remains in service. |
| Runtime configuration required | Select the workspace and complete all required write-only fields. |
| Secret store unavailable | Stop writes, preserve `/data/workspace-secrets`, and restore vault plus master key together. |
| Connection test failed | Rotate the indicated runtime credential or correct the relevant non-secret endpoint. |