docs: document internal semantic infrastructure
This commit is contained in:
@@ -130,8 +130,11 @@ before creating sessions. Git pull/push over SSH remains fully supported and is
|
||||
## Bootstrap, first pull, and diagnostics
|
||||
|
||||
Use the repository's canonical `compose.yaml` plus `deploy/compose.local.yaml`; they always start
|
||||
the mandatory `frontend` and `core` services. Do not copy or maintain a standalone application
|
||||
Compose file. Copy [the bindings env example](examples/workspace-bindings.env.example) into an
|
||||
`frontend`, `core`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`. This profile
|
||||
is CPU-first. Add `THOTH_ENABLE_EMBEDDING_GPU=1` only on a Linux host that intentionally exposes a
|
||||
supported GPU device to Docker. Qdrant is a derived but persistent index, while Ollama keeps a
|
||||
local model cache for `qwen3-embedding:0.6b` (`1024` dimensions, cosine distance). Do not copy or
|
||||
maintain a standalone application Compose file. Copy [the bindings env example](examples/workspace-bindings.env.example) into an
|
||||
untracked operator directory and create a protected operator env file from
|
||||
`deploy/env/local.env.example`. It must contain absolute `PI_AUTH_FILE`,
|
||||
`THT_SECRETS_FILE`, `THT_WORKSPACE_BINDINGS_ENV_FILE`, and connector `*_SOURCE` paths.
|
||||
@@ -172,6 +175,10 @@ Use `POST /workspace-registry/pull` to fetch later revisions. Run workspace diag
|
||||
required DWH bindings are mounted. Schema-v3 diagnostics probe the internal Qdrant/Ollama
|
||||
services through backend config; ordinary diagnostics are read-only.
|
||||
|
||||
Schema-v3 is the only operational descriptor format. Schema-v1/v2 descriptors remain
|
||||
`migration_required` until an explicit reviewed migration writes schema version 3. One workspace owns one Qdrant collection; schema, Evidence, and Memory records share that collection and remain
|
||||
isolated by payload `kind`.
|
||||
|
||||
To migrate an existing legacy descriptor, create/clone an empty private remote, set the absolute
|
||||
`THT_SOURCE_ROOT`, transform with absolute paths, review the schema-v1 result, explicitly produce
|
||||
the reviewed schema-v3 contract, then commit/push. The transformer never imports `${ENV}` values
|
||||
|
||||
@@ -32,8 +32,9 @@ targets with runtime ownership without copying secret or tracked file contents i
|
||||
state. Rerun it after a restore and before Compose or `thothctl` startup; it is idempotent and does
|
||||
not overwrite existing targets.
|
||||
|
||||
Permit outbound TCP only to approved Git/Gitea, DWH, Qdrant, embedding, and bastion endpoints.
|
||||
Allow inbound traffic only from the reverse proxy/Docker network. Do not give the runtime service
|
||||
Permit outbound TCP only to approved Git/Gitea, DWH, LLM, and optional bastion endpoints.
|
||||
Qdrant and Ollama run inside the Compose stack. Allow inbound traffic only from the reverse
|
||||
proxy/Docker network. Do not give the runtime service
|
||||
account Gitea administration, database-superuser rights, or a shell in the Git host.
|
||||
|
||||
## Gitea and remote Git setup
|
||||
@@ -151,7 +152,11 @@ The Git registry itself may still use SSH normally.
|
||||
## Same-origin reverse proxy, bootstrap, and health
|
||||
|
||||
Use the repository's canonical `compose.yaml` plus `deploy/compose.server.yaml`; they always
|
||||
start the mandatory `frontend` and `core` services. Do not copy or maintain a standalone
|
||||
start `frontend`, `core`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`.
|
||||
Startup is CPU-first; use `THOTH_ENABLE_EMBEDDING_GPU=1` only when the server intentionally
|
||||
exposes a supported GPU device to Docker. Qdrant is a derived but persistent index, and the
|
||||
Ollama model cache persists the exact `qwen3-embedding:0.6b` model (`1024` dimensions, cosine
|
||||
distance) for offline reuse. Do not copy or maintain a standalone
|
||||
application Compose file. Copy `docs/install/examples/thothii-installation.server.yaml` to the
|
||||
protected operator directory and preserve its required session-server overlay, exactly one Git
|
||||
transport override, and generated connector-secret override.
|
||||
@@ -221,6 +226,10 @@ Its schema-v1 output is `migration_required`; explicitly supply collection ident
|
||||
and the reviewed v3 contract before commit. Never import `${ENV}` values or
|
||||
copy secret files.
|
||||
|
||||
Schema-v3 is the only operational descriptor contract. Schema-v1/v2 descriptors remain
|
||||
`migration_required` until an explicit reviewed migration writes version 3. One workspace owns one Qdrant collection; schema, Evidence, and Memory records share it and stay separated by payload
|
||||
`kind`.
|
||||
|
||||
After valid bootstrap, Git outage retains the active snapshot with `degraded: true`. Repair
|
||||
egress/DNS/CA/credentials, pull, and confirm healthy status. Roll back a bad descriptor through a
|
||||
reviewed Git revert/release branch, advance the remote through normal policy, pull it, and confirm
|
||||
@@ -248,3 +257,24 @@ revision and monitor status. If snapshots are missing or corrupt, stop the servi
|
||||
newest verified registry backup, start it privately, verify status, and then reopen proxy traffic.
|
||||
A first-bootstrap failure has no fallback: repair remote trust rather than creating an unreviewed
|
||||
runtime checkout.
|
||||
|
||||
## Qdrant backup/restore and cache recovery
|
||||
|
||||
Use the repository helpers for Qdrant backup/restore:
|
||||
|
||||
```sh
|
||||
./scripts/vector-backup.sh --project-name thothii --output /secure/backups/thoth-qdrant-2026-08-08.tar
|
||||
./scripts/vector-restore.sh --project-name thothii --input /secure/backups/thoth-qdrant-2026-08-08.tar --confirm-project thothii
|
||||
```
|
||||
|
||||
Qdrant backup/restore targets exactly one labeled `qdrant-data` volume for the named Compose
|
||||
project. Restore requires the exact repeated project confirmation, validates the archive before
|
||||
stopping `qdrant`, stages rollback content, and restores in place only for that project-scoped
|
||||
volume. It does not migrate schema-v1/v2 workspaces, rename collections, or resolve semantic-index
|
||||
incompatibilities.
|
||||
|
||||
The Ollama model cache is a recoverable local cache, not the canonical semantic source of truth.
|
||||
You may back up `embedding-models` for faster offline recovery, but a cache loss is recoverable by
|
||||
re-pulling `qwen3-embedding:0.6b` through `embedding-model-init`.
|
||||
|
||||
Only the Git remote, DWH, LLM, and optional bastion endpoints stay external.
|
||||
|
||||
Reference in New Issue
Block a user