docs: document internal semantic infrastructure

This commit is contained in:
2026-08-08 20:52:33 +02:00
parent bff21507df
commit 22c3512ac8
12 changed files with 349 additions and 197 deletions
+30 -1
View File
@@ -1,8 +1,37 @@
# ThothII — Project State
> Starting-point snapshot for new sessions. Last updated: 2026-08-05 (Task 13 fix round 3/5).
> Starting-point snapshot for new sessions. Last updated: 2026-08-08 (Task 11 documentation and state update).
> Point a fresh session here ("read PROJECT_STATE.md") before substantial work.
## Internal Qdrant + Ollama semantic infrastructure — LIVE 2026-08-08
- **Compose topology.** The mandatory application stack is `frontend`, `core`, `qdrant`,
`embedding`, and the one-shot `embedding-model-init`. Startup is CPU-first by default; Linux
hosts may opt into GPU exposure with `THOTH_ENABLE_EMBEDDING_GPU=1`. Qdrant is private on the
Compose network and persists `/qdrant/storage` in `qdrant-data`. Ollama persists its local model
cache in `embedding-models`, and `embedding-model-init` blocks `core` until
`qwen3-embedding:0.6b` is present.
- **Semantic contract.** Internal semantic indexing is fixed to `qwen3-embedding:0.6b`,
`1024` dimensions, and cosine distance. Schema-v3 descriptors are operational; schema-v1/v2 descriptors remain `migration_required` until an explicit reviewed migration writes schema version 3. One workspace owns one Qdrant collection, and schema, Evidence, and Memory records
coexist inside that collection with payload `kind` separation.
- **Boundary and persistence.** Only DWH and LLM remain external runtime application endpoints.
There are no active external vector or embedding endpoint instructions, bindings, or secrets in
the supported operator manuals. Qdrant remains a derived but persistent semantic index: the
canonical sources of truth stay the workspace Git descriptors, phase artifacts, and memory
registry/ledger. The Ollama model cache is recoverable for offline startup but is not the
canonical source of semantic content.
- **Backup and recovery.** `./scripts/vector-backup.sh --project-name <name> --output <file>`
archives exactly one labeled `<project>_qdrant-data` volume and preserves the prior `qdrant`
running state. `./scripts/vector-restore.sh --project-name <name> --input <file>
--confirm-project <name>` requires the exact repeated project confirmation, validates manifest
and archive safety before stopping `qdrant`, stages rollback content, restores in place, and
restarts `qdrant` only if it was previously running. Restore does not migrate legacy workspace
descriptors, rename collections, or repair a semantic-index incompatibility.
- **Verification recorded for this docs/state update.** The installation-manual contract tests
now require the four-service-plus-init topology, the fixed internal model/dimensions, explicit
schema-v3 migration messaging, Qdrant collection ownership, Qdrant backup/restore safety, and
the absence of active external vector/embedding operator bindings from current manuals.
## Unified deployment release gate — Task 13 (2026-08-05)
- **Release coverage.** `scripts/unified-deployment-smoke.sh` gates the two-service render/build,