docs: document internal semantic infrastructure

This commit is contained in:
2026-08-08 20:52:33 +02:00
parent bff21507df
commit 22c3512ac8
12 changed files with 349 additions and 197 deletions
+9 -2
View File
@@ -130,8 +130,11 @@ before creating sessions. Git pull/push over SSH remains fully supported and is
## Bootstrap, first pull, and diagnostics
Use the repository's canonical `compose.yaml` plus `deploy/compose.local.yaml`; they always start
the mandatory `frontend` and `core` services. Do not copy or maintain a standalone application
Compose file. Copy [the bindings env example](examples/workspace-bindings.env.example) into an
`frontend`, `core`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`. This profile
is CPU-first. Add `THOTH_ENABLE_EMBEDDING_GPU=1` only on a Linux host that intentionally exposes a
supported GPU device to Docker. Qdrant is a derived but persistent index, while Ollama keeps a
local model cache for `qwen3-embedding:0.6b` (`1024` dimensions, cosine distance). Do not copy or
maintain a standalone application Compose file. Copy [the bindings env example](examples/workspace-bindings.env.example) into an
untracked operator directory and create a protected operator env file from
`deploy/env/local.env.example`. It must contain absolute `PI_AUTH_FILE`,
`THT_SECRETS_FILE`, `THT_WORKSPACE_BINDINGS_ENV_FILE`, and connector `*_SOURCE` paths.
@@ -172,6 +175,10 @@ Use `POST /workspace-registry/pull` to fetch later revisions. Run workspace diag
required DWH bindings are mounted. Schema-v3 diagnostics probe the internal Qdrant/Ollama
services through backend config; ordinary diagnostics are read-only.
Schema-v3 is the only operational descriptor format. Schema-v1/v2 descriptors remain
`migration_required` until an explicit reviewed migration writes schema version 3. One workspace owns one Qdrant collection; schema, Evidence, and Memory records share that collection and remain
isolated by payload `kind`.
To migrate an existing legacy descriptor, create/clone an empty private remote, set the absolute
`THT_SOURCE_ROOT`, transform with absolute paths, review the schema-v1 result, explicitly produce
the reviewed schema-v3 contract, then commit/push. The transformer never imports `${ENV}` values
+33 -3
View File
@@ -32,8 +32,9 @@ targets with runtime ownership without copying secret or tracked file contents i
state. Rerun it after a restore and before Compose or `thothctl` startup; it is idempotent and does
not overwrite existing targets.
Permit outbound TCP only to approved Git/Gitea, DWH, Qdrant, embedding, and bastion endpoints.
Allow inbound traffic only from the reverse proxy/Docker network. Do not give the runtime service
Permit outbound TCP only to approved Git/Gitea, DWH, LLM, and optional bastion endpoints.
Qdrant and Ollama run inside the Compose stack. Allow inbound traffic only from the reverse
proxy/Docker network. Do not give the runtime service
account Gitea administration, database-superuser rights, or a shell in the Git host.
## Gitea and remote Git setup
@@ -151,7 +152,11 @@ The Git registry itself may still use SSH normally.
## Same-origin reverse proxy, bootstrap, and health
Use the repository's canonical `compose.yaml` plus `deploy/compose.server.yaml`; they always
start the mandatory `frontend` and `core` services. Do not copy or maintain a standalone
start `frontend`, `core`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`.
Startup is CPU-first; use `THOTH_ENABLE_EMBEDDING_GPU=1` only when the server intentionally
exposes a supported GPU device to Docker. Qdrant is a derived but persistent index, and the
Ollama model cache persists the exact `qwen3-embedding:0.6b` model (`1024` dimensions, cosine
distance) for offline reuse. Do not copy or maintain a standalone
application Compose file. Copy `docs/install/examples/thothii-installation.server.yaml` to the
protected operator directory and preserve its required session-server overlay, exactly one Git
transport override, and generated connector-secret override.
@@ -221,6 +226,10 @@ Its schema-v1 output is `migration_required`; explicitly supply collection ident
and the reviewed v3 contract before commit. Never import `${ENV}` values or
copy secret files.
Schema-v3 is the only operational descriptor contract. Schema-v1/v2 descriptors remain
`migration_required` until an explicit reviewed migration writes version 3. One workspace owns one Qdrant collection; schema, Evidence, and Memory records share it and stay separated by payload
`kind`.
After valid bootstrap, Git outage retains the active snapshot with `degraded: true`. Repair
egress/DNS/CA/credentials, pull, and confirm healthy status. Roll back a bad descriptor through a
reviewed Git revert/release branch, advance the remote through normal policy, pull it, and confirm
@@ -248,3 +257,24 @@ revision and monitor status. If snapshots are missing or corrupt, stop the servi
newest verified registry backup, start it privately, verify status, and then reopen proxy traffic.
A first-bootstrap failure has no fallback: repair remote trust rather than creating an unreviewed
runtime checkout.
## Qdrant backup/restore and cache recovery
Use the repository helpers for Qdrant backup/restore:
```sh
./scripts/vector-backup.sh --project-name thothii --output /secure/backups/thoth-qdrant-2026-08-08.tar
./scripts/vector-restore.sh --project-name thothii --input /secure/backups/thoth-qdrant-2026-08-08.tar --confirm-project thothii
```
Qdrant backup/restore targets exactly one labeled `qdrant-data` volume for the named Compose
project. Restore requires the exact repeated project confirmation, validates the archive before
stopping `qdrant`, stages rollback content, and restores in place only for that project-scoped
volume. It does not migrate schema-v1/v2 workspaces, rename collections, or resolve semantic-index
incompatibilities.
The Ollama model cache is a recoverable local cache, not the canonical semantic source of truth.
You may back up `embedding-models` for faster offline recovery, but a cache loss is recoverable by
re-pulling `qwen3-embedding:0.6b` through `embedding-model-init`.
Only the Git remote, DWH, LLM, and optional bastion endpoints stay external.