docs: document internal semantic infrastructure

This commit is contained in:
2026-08-08 20:52:33 +02:00
parent bff21507df
commit 22c3512ac8
12 changed files with 349 additions and 197 deletions
+33 -3
View File
@@ -32,8 +32,9 @@ targets with runtime ownership without copying secret or tracked file contents i
state. Rerun it after a restore and before Compose or `thothctl` startup; it is idempotent and does
not overwrite existing targets.
Permit outbound TCP only to approved Git/Gitea, DWH, Qdrant, embedding, and bastion endpoints.
Allow inbound traffic only from the reverse proxy/Docker network. Do not give the runtime service
Permit outbound TCP only to approved Git/Gitea, DWH, LLM, and optional bastion endpoints.
Qdrant and Ollama run inside the Compose stack. Allow inbound traffic only from the reverse
proxy/Docker network. Do not give the runtime service
account Gitea administration, database-superuser rights, or a shell in the Git host.
## Gitea and remote Git setup
@@ -151,7 +152,11 @@ The Git registry itself may still use SSH normally.
## Same-origin reverse proxy, bootstrap, and health
Use the repository's canonical `compose.yaml` plus `deploy/compose.server.yaml`; they always
start the mandatory `frontend` and `core` services. Do not copy or maintain a standalone
start `frontend`, `core`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`.
Startup is CPU-first; use `THOTH_ENABLE_EMBEDDING_GPU=1` only when the server intentionally
exposes a supported GPU device to Docker. Qdrant is a derived but persistent index, and the
Ollama model cache persists the exact `qwen3-embedding:0.6b` model (`1024` dimensions, cosine
distance) for offline reuse. Do not copy or maintain a standalone
application Compose file. Copy `docs/install/examples/thothii-installation.server.yaml` to the
protected operator directory and preserve its required session-server overlay, exactly one Git
transport override, and generated connector-secret override.
@@ -221,6 +226,10 @@ Its schema-v1 output is `migration_required`; explicitly supply collection ident
and the reviewed v3 contract before commit. Never import `${ENV}` values or
copy secret files.
Schema-v3 is the only operational descriptor contract. Schema-v1/v2 descriptors remain
`migration_required` until an explicit reviewed migration writes version 3. One workspace owns one Qdrant collection; schema, Evidence, and Memory records share it and stay separated by payload
`kind`.
After valid bootstrap, Git outage retains the active snapshot with `degraded: true`. Repair
egress/DNS/CA/credentials, pull, and confirm healthy status. Roll back a bad descriptor through a
reviewed Git revert/release branch, advance the remote through normal policy, pull it, and confirm
@@ -248,3 +257,24 @@ revision and monitor status. If snapshots are missing or corrupt, stop the servi
newest verified registry backup, start it privately, verify status, and then reopen proxy traffic.
A first-bootstrap failure has no fallback: repair remote trust rather than creating an unreviewed
runtime checkout.
## Qdrant backup/restore and cache recovery
Use the repository helpers for Qdrant backup/restore:
```sh
./scripts/vector-backup.sh --project-name thothii --output /secure/backups/thoth-qdrant-2026-08-08.tar
./scripts/vector-restore.sh --project-name thothii --input /secure/backups/thoth-qdrant-2026-08-08.tar --confirm-project thothii
```
Qdrant backup/restore targets exactly one labeled `qdrant-data` volume for the named Compose
project. Restore requires the exact repeated project confirmation, validates the archive before
stopping `qdrant`, stages rollback content, and restores in place only for that project-scoped
volume. It does not migrate schema-v1/v2 workspaces, rename collections, or resolve semantic-index
incompatibilities.
The Ollama model cache is a recoverable local cache, not the canonical semantic source of truth.
You may back up `embedding-models` for faster offline recovery, but a cache loss is recoverable by
re-pulling `qwen3-embedding:0.6b` through `embedding-model-init`.
Only the Git remote, DWH, LLM, and optional bastion endpoints stay external.