From 22c3512ac88bc52cab06316b5812f9210b237f38 Mon Sep 17 00:00:00 2001 From: mptyl Date: Sat, 8 Aug 2026 20:52:33 +0200 Subject: [PATCH] docs: document internal semantic infrastructure --- .../task-11-report.md | 61 +++++ AGENTS.md | 5 +- PROJECT_STATE.md | 31 ++- README.md | 22 +- deploy/secrets/README.md | 14 +- docs/gestione-memory.md | 18 +- docs/install/local-workspace-registry.md | 11 +- docs/install/server-workspace-registry.md | 36 ++- docs/installazione-docker-4-contesti.md | 4 +- docs/workspace-diagnostic-protocol.md | 234 ++++++------------ scripts/test-verify-workspace-install-docs.sh | 20 ++ scripts/verify-workspace-install-docs.sh | 90 +++++++ 12 files changed, 349 insertions(+), 197 deletions(-) create mode 100644 .superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-11-report.md diff --git a/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-11-report.md b/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-11-report.md new file mode 100644 index 00000000..c7a5a381 --- /dev/null +++ b/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-11-report.md @@ -0,0 +1,61 @@ +# Task 11 report + +Status: completed on 2026-08-08. + +## Scope delivered + +- Updated operator-facing documentation for the internal Qdrant + Ollama architecture. +- Tightened documentation contract tests to require the current four-service-plus-init topology, + CPU-first/GPU-override guidance, fixed internal model/dimensions, schema-v3 migration wording, + one-collection-per-workspace ownership, and Qdrant backup/restore safety. +- Updated stable repo guidance in `AGENTS.md` and the current snapshot in `PROJECT_STATE.md`. +- Rewrote the workspace diagnostic protocol to the schema-v3/internal-semantic-service contract. +- Updated the memory guide to describe Qdrant as the derived persistent index. +- Updated the runtime secret-bundle guide to remove active vector/embedding secret guidance. + +## Files changed + +- `README.md` +- `AGENTS.md` +- `PROJECT_STATE.md` +- `docs/install/local-workspace-registry.md` +- `docs/install/server-workspace-registry.md` +- `docs/installazione-docker-4-contesti.md` +- `docs/workspace-diagnostic-protocol.md` +- `docs/gestione-memory.md` +- `deploy/secrets/README.md` +- `scripts/verify-workspace-install-docs.sh` +- `scripts/test-verify-workspace-install-docs.sh` + +## Verification + +Fresh successful runs: + +```sh +./scripts/test-verify-workspace-install-docs.sh +./scripts/verify-workspace-install-docs.sh --fixtures-only +git diff --check +``` + +Key outcomes: + +- internal semantic infrastructure documentation contract passed +- all existing install/manual fixture contracts still passed +- diff hygiene passed with no whitespace/errors + +## Self-review notes + +- The updated docs now match the code-backed Compose topology: `frontend`, `core`, `qdrant`, + `embedding`, and `embedding-model-init`. +- Active manuals no longer instruct operators to configure external vector or embedding runtime + endpoints/secrets. +- Qdrant backup/restore wording now matches the helper scripts' exact confirmation and rollback + behavior. +- Legacy descriptor handling is documented as explicit schema-v3 migration only; no silent + semantic-data migration is claimed. + +## Residual concerns + +- The broader repository still contains historical design/spec material that references older + pgvector/external-embedding architecture; this task intentionally updated operator/current-state + documentation and the corresponding contract tests, not historical planning documents. diff --git a/AGENTS.md b/AGENTS.md index 4867fe6b..39bc24b7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,10 +11,7 @@ detail. Design history lives in `docs/superpowers/specs/` and `docs/superpowers/ ## Commands -The repo has three independently-built layers. Run the local Docker stack with -`./scripts/run-stack.sh` after creating `deploy/env/local.env`; it starts the base+local Compose -profile, whose core image contains Pi. DWH, vector DB, embedding, and LLM remain external -configuration endpoints. +The repo has three independently-built layers. Run the local Docker stack with `./scripts/run-stack.sh` after creating `deploy/env/local.env`; it starts the base+local Compose profile with `frontend`, `core`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`. The core image contains Pi. Qdrant and Ollama are internal Compose services; DWH and LLM remain external configuration endpoints. **harness/** (Python `tht` CLI + Pi gate extension) - Install: `cd harness && python -m venv .venv && pip install -e ".[dev]"` (puts `tht` on PATH) diff --git a/PROJECT_STATE.md b/PROJECT_STATE.md index ba0c1feb..15f5f498 100644 --- a/PROJECT_STATE.md +++ b/PROJECT_STATE.md @@ -1,8 +1,37 @@ # ThothII — Project State -> Starting-point snapshot for new sessions. Last updated: 2026-08-05 (Task 13 fix round 3/5). +> Starting-point snapshot for new sessions. Last updated: 2026-08-08 (Task 11 documentation and state update). > Point a fresh session here ("read PROJECT_STATE.md") before substantial work. +## Internal Qdrant + Ollama semantic infrastructure — LIVE 2026-08-08 + +- **Compose topology.** The mandatory application stack is `frontend`, `core`, `qdrant`, + `embedding`, and the one-shot `embedding-model-init`. Startup is CPU-first by default; Linux + hosts may opt into GPU exposure with `THOTH_ENABLE_EMBEDDING_GPU=1`. Qdrant is private on the + Compose network and persists `/qdrant/storage` in `qdrant-data`. Ollama persists its local model + cache in `embedding-models`, and `embedding-model-init` blocks `core` until + `qwen3-embedding:0.6b` is present. +- **Semantic contract.** Internal semantic indexing is fixed to `qwen3-embedding:0.6b`, + `1024` dimensions, and cosine distance. Schema-v3 descriptors are operational; schema-v1/v2 descriptors remain `migration_required` until an explicit reviewed migration writes schema version 3. One workspace owns one Qdrant collection, and schema, Evidence, and Memory records + coexist inside that collection with payload `kind` separation. +- **Boundary and persistence.** Only DWH and LLM remain external runtime application endpoints. + There are no active external vector or embedding endpoint instructions, bindings, or secrets in + the supported operator manuals. Qdrant remains a derived but persistent semantic index: the + canonical sources of truth stay the workspace Git descriptors, phase artifacts, and memory + registry/ledger. The Ollama model cache is recoverable for offline startup but is not the + canonical source of semantic content. +- **Backup and recovery.** `./scripts/vector-backup.sh --project-name --output ` + archives exactly one labeled `_qdrant-data` volume and preserves the prior `qdrant` + running state. `./scripts/vector-restore.sh --project-name --input + --confirm-project ` requires the exact repeated project confirmation, validates manifest + and archive safety before stopping `qdrant`, stages rollback content, restores in place, and + restarts `qdrant` only if it was previously running. Restore does not migrate legacy workspace + descriptors, rename collections, or repair a semantic-index incompatibility. +- **Verification recorded for this docs/state update.** The installation-manual contract tests + now require the four-service-plus-init topology, the fixed internal model/dimensions, explicit + schema-v3 migration messaging, Qdrant collection ownership, Qdrant backup/restore safety, and + the absence of active external vector/embedding operator bindings from current manuals. + ## Unified deployment release gate — Task 13 (2026-08-05) - **Release coverage.** `scripts/unified-deployment-smoke.sh` gates the two-service render/build, diff --git a/README.md b/README.md index 9848db26..a1e0d6d2 100644 --- a/README.md +++ b/README.md @@ -6,8 +6,7 @@ external in this profile, except for the mandatory internal semantic services bu ## Docker Compose: local startup -Requirements: Docker Engine with Compose v2. The mandatory stack is `frontend`, `core`, -`qdrant`, `embedding`, and the one-shot `embedding-model-init`. DWH and LLM remain external, +Requirements: Docker Engine with Compose v2. The mandatory stack is `frontend`, `core`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`. DWH and LLM remain external, configurable endpoints—even when they are co-located with ThothII. From a fresh clone, run these commands from the repository root: @@ -44,9 +43,11 @@ loopback port). Credentials and certificates are local protected files. Do not put them in environment examples, workspace YAML, URLs, or Compose interpolation values. -Application state is split across the named `settings`, `pi-state`, `workspace-registry`, and -`sessions` volumes. `docker compose down` keeps them. Only an explicit destructive command such -as `docker compose down --volumes` removes them. +Application state is split across the named `settings`, `pi-state`, `workspace-registry`, +`sessions`, `qdrant-data`, and `embedding-models` volumes. `docker compose down` keeps them. +`qdrant-data` is a derived but persistent index store; `embedding-models` is an Ollama model +cache for `qwen3-embedding:0.6b` with fixed `1024`-dimension embeddings. Only an explicit destructive command such as `docker compose +down --volumes` removes them. The frontend depends on the core health check and proxies `/health` and `/api/*` to it. The application health endpoint intentionally checks process readiness only; external dependency @@ -69,6 +70,11 @@ every revision referenced by an open, closed, or failed unarchived session. It r single local installation list or from a server administrator's complete session list, never from a remote user's partial list. +Schema-v3 is the operational descriptor contract. Schema-v1/v2 descriptors remain +`migration_required` until an explicit reviewed migration writes schema version 3. One workspace +owns one Qdrant collection; schema, Evidence, and Memory records share that collection and stay +separated by indexed payload `kind`. + Connector `ssh_tunnel` bindings are diagnostic-only in this release: their bounded probe always cleans up the loopback forward and returns `workspace_not_activatable`; session creation is rejected before persistence. Git registry access over SSH is unaffected. Use direct or REST connector @@ -211,7 +217,7 @@ archive path. Restore targets that same exact project-scoped `qdrant-data` volume. Because restore replaces the persistent Qdrant data in place, it requires an explicit confirmation that exactly repeats the -Compose project name: +Compose project name by passing `--confirm-project`: ```sh ./scripts/vector-restore.sh \ @@ -223,7 +229,9 @@ Compose project name: The restore script stops `qdrant`, validates the exact labeled target, stages the current volume contents for rollback, extracts the requested archive into the volume, and then returns the service to its prior running state. After restore, run the backend health checks and a known -retrieval query before reopening write traffic. +retrieval query before reopening write traffic. Restore does not migrate schema-v1/v2 workspace +descriptors, does not rename collections, and does not reconcile an incompatible collection +contract; those remain explicit reviewed recovery steps outside the helper. ## Production trust boundary and secrets diff --git a/deploy/secrets/README.md b/deploy/secrets/README.md index b00bbfd3..2a352fe8 100644 --- a/deploy/secrets/README.md +++ b/deploy/secrets/README.md @@ -9,10 +9,14 @@ chmod 600 deploy/secrets/thothii.secrets ``` The file uses strict `KEY=VALUE` lines (comments and blank lines are allowed). The supported -keys are `THT_MODEL_API_KEY`, `THT_DWH_API_KEY`, `THT_CA`, `THT_SSL_CA`, and -`PI_PROVIDER_API_KEY`. Values must be non-empty and contain no whitespace. Do not put secrets +keys are `THT_MODEL_API_KEY`, `THT_DWH_API_KEY`, `THT_CA`, and `THT_SSL_CA`. Values must be +non-empty and contain no whitespace. Do not put secrets in the root `.env`, workspace YAML, URLs, logs, or rendered Compose output. +Do not add vector or embedding endpoint credentials to the bundle. Active operator manuals use +internal Qdrant and Ollama services, so vector/embedding runtime endpoint secrets are not part of +the supported installation contract. + Compose mounts the bundle read-only as `/run/secrets/thothii.secrets`. The host file must be a regular non-symlink file with mode `0600` or `0400`; Docker's normal `0444` mode is accepted only for the runtime mount beneath `/run/secrets`. The core runs as UID 10001. Verify the mount @@ -37,9 +41,9 @@ and only then deleting the old files. The old variables remain a compatibility p upgrades, but the documented and tested default is an absolute `THT_SECRETS_FILE` path to the protected bundle. -Hosted Pi providers must use a single provider key. Compound providers (Bedrock, Azure OpenAI -Responses, Cloudflare Workers AI/Gateway) fail closed until a provider-specific credential -adapter is implemented. +Hosted Pi providers must use a single model key through `THT_MODEL_API_KEY`. Compound providers +(Bedrock, Azure OpenAI Responses, Cloudflare Workers AI/Gateway) fail closed until a +provider-specific credential adapter is implemented. ## User-owned session database secrets diff --git a/docs/gestione-memory.md b/docs/gestione-memory.md index b718ce0c..e2e07d87 100644 --- a/docs/gestione-memory.md +++ b/docs/gestione-memory.md @@ -16,7 +16,7 @@ decisione concept_clarified nel ledger della sessione F8: il reviewer decide se promuoverla │ ├── registro globale registry.jsonl - └── indice semantico pgvector + └── indice semantico Qdrant │ ▼ F2 di una sessione futura @@ -33,7 +33,7 @@ Implementazione principale: [harness/tht/memory.py](../harness/tht/memory.py:14) | --- | --- | --- | | Ledger della sessione | `concept_clarified`, `memory_promoted`, `memory_promotion_declined` | Audit e stato della singola sessione | | Registro globale | Record `mem-XXXX` in `registry.jsonl` | Archivio canonico attuale delle memory | -| Indice pgvector | Embedding e metadati derivati dal registro | Ricerca semantica | +| Indice Qdrant | Embedding e metadati derivati dal registro | Ricerca semantica | Il ledger contiene la provenienza e le decisioni umane. Il record globale contiene il testo riutilizzabile. L'indice vettoriale è una proiezione per la ricerca, non il posto in cui il workflow registra direttamente le decisioni. @@ -114,9 +114,9 @@ Il registro attuale è: La scrittura viene fatta tramite file temporaneo e `os.replace`, quindi la sostituzione del registro è atomica. L'idempotenza della promozione è basata sulla coppia `session_id + decision_seq`: la stessa decisione della stessa sessione non genera due record globali. -### pgvector +### Qdrant -Dopo la promozione, `save-one` costruisce un solo `VectorRecord` e lo invia all'indice pgvector. Il testo indicizzato include: +Dopo la promozione, `save-one` costruisce un solo `VectorRecord` e lo invia all'indice Qdrant. Il testo indicizzato include: - tipo e soggetto; - dettaglio; @@ -124,13 +124,13 @@ Dopo la promozione, `save-one` costruisce un solo `VectorRecord` e lo invia all' - domanda di contesto; - eventuali concetti e mapping. -Il record vettoriale usa l'id `memory:mem-XXXX`, mentre i metadati conservano `subject`, `detail`, `rationale`, `tables` e `concepts`. L'hash SHA-256 del contenuto impedisce di ricalcolare embedding e upsert quando il testo non è cambiato. +Il record vettoriale usa l'id `memory:mem-XXXX`, mentre i metadati conservano `subject`, `detail`, `rationale`, `tables`, `concepts` e il discriminante `kind`. L'hash SHA-256 del contenuto impedisce di ricalcolare embedding e upsert quando il testo non è cambiato. Il comportamento è implementato in [harness/tht/memory.py](../harness/tht/memory.py:253) e [harness/tht/memory.py](../harness/tht/memory.py:305). ### Fonte canonica attuale -Oggi il registro JSONL è ancora la fonte canonica applicativa e pgvector è l'indice derivato. Il commento iniziale di [memory_cmd.py](../harness/tht/cli/memory_cmd.py:1) segnala un debito tecnico: l'architettura futura prevista sarebbe usare direttamente il vector DB come archivio unico, ma questa migrazione non è ancora completata. +Oggi il registro JSONL è ancora la fonte canonica applicativa e Qdrant resta un indice derivato ma persistente. Il workflow non registra direttamente le decisioni nel vector DB: usa Qdrant come proiezione interrogabile del registro e del ledger effettivo. ## Riutilizzo in F2 @@ -209,10 +209,10 @@ Questo evita che una singola modifica al prompt o a un solo componente reintrodu Il salvataggio segue sostanzialmente questa sequenza: ```text -registro JSONL → pgvector → marker memory_promoted nel ledger +registro JSONL → Qdrant → marker memory_promoted nel ledger ``` -Se pgvector non è disponibile, il registro può contenere una memory non ancora ricercabile; il comando segnala che sarà necessario reindicizzare. +Se Qdrant non è disponibile, il registro può contenere una memory non ancora ricercabile; il comando segnala che sarà necessario reindicizzare. Se il marker del ledger fallisce dopo il salvataggio nel vector DB, la memory può risultare globalmente presente ma senza audit completo nella sessione. Il gate restituisce un comando di recupero manuale. @@ -232,4 +232,4 @@ Vecchi record `table_promoted` o `table_excluded` possono ancora esistere in art La gestione attuale è coerente con il requisito funzionale: una memory è una conoscenza concettuale riutilizzabile, non una scelta di schema-linking. -La parte più solida è la difesa multilivello del tipo `concept_clarified`. Il principale debito tecnico riguarda invece la convivenza del registro JSONL con pgvector e l'assenza di una transazione unica tra archivio globale, indice semantico e ledger della sessione. +La parte più solida è la difesa multilivello del tipo `concept_clarified`. Il principale debito tecnico riguarda invece la convivenza del registro JSONL con Qdrant e l'assenza di una transazione unica tra archivio globale, indice semantico e ledger della sessione. diff --git a/docs/install/local-workspace-registry.md b/docs/install/local-workspace-registry.md index c111dfde..b508fe3d 100644 --- a/docs/install/local-workspace-registry.md +++ b/docs/install/local-workspace-registry.md @@ -130,8 +130,11 @@ before creating sessions. Git pull/push over SSH remains fully supported and is ## Bootstrap, first pull, and diagnostics Use the repository's canonical `compose.yaml` plus `deploy/compose.local.yaml`; they always start -the mandatory `frontend` and `core` services. Do not copy or maintain a standalone application -Compose file. Copy [the bindings env example](examples/workspace-bindings.env.example) into an +`frontend`, `core`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`. This profile +is CPU-first. Add `THOTH_ENABLE_EMBEDDING_GPU=1` only on a Linux host that intentionally exposes a +supported GPU device to Docker. Qdrant is a derived but persistent index, while Ollama keeps a +local model cache for `qwen3-embedding:0.6b` (`1024` dimensions, cosine distance). Do not copy or +maintain a standalone application Compose file. Copy [the bindings env example](examples/workspace-bindings.env.example) into an untracked operator directory and create a protected operator env file from `deploy/env/local.env.example`. It must contain absolute `PI_AUTH_FILE`, `THT_SECRETS_FILE`, `THT_WORKSPACE_BINDINGS_ENV_FILE`, and connector `*_SOURCE` paths. @@ -172,6 +175,10 @@ Use `POST /workspace-registry/pull` to fetch later revisions. Run workspace diag required DWH bindings are mounted. Schema-v3 diagnostics probe the internal Qdrant/Ollama services through backend config; ordinary diagnostics are read-only. +Schema-v3 is the only operational descriptor format. Schema-v1/v2 descriptors remain +`migration_required` until an explicit reviewed migration writes schema version 3. One workspace owns one Qdrant collection; schema, Evidence, and Memory records share that collection and remain +isolated by payload `kind`. + To migrate an existing legacy descriptor, create/clone an empty private remote, set the absolute `THT_SOURCE_ROOT`, transform with absolute paths, review the schema-v1 result, explicitly produce the reviewed schema-v3 contract, then commit/push. The transformer never imports `${ENV}` values diff --git a/docs/install/server-workspace-registry.md b/docs/install/server-workspace-registry.md index b6fb4e6b..1eeb2332 100644 --- a/docs/install/server-workspace-registry.md +++ b/docs/install/server-workspace-registry.md @@ -32,8 +32,9 @@ targets with runtime ownership without copying secret or tracked file contents i state. Rerun it after a restore and before Compose or `thothctl` startup; it is idempotent and does not overwrite existing targets. -Permit outbound TCP only to approved Git/Gitea, DWH, Qdrant, embedding, and bastion endpoints. -Allow inbound traffic only from the reverse proxy/Docker network. Do not give the runtime service +Permit outbound TCP only to approved Git/Gitea, DWH, LLM, and optional bastion endpoints. +Qdrant and Ollama run inside the Compose stack. Allow inbound traffic only from the reverse +proxy/Docker network. Do not give the runtime service account Gitea administration, database-superuser rights, or a shell in the Git host. ## Gitea and remote Git setup @@ -151,7 +152,11 @@ The Git registry itself may still use SSH normally. ## Same-origin reverse proxy, bootstrap, and health Use the repository's canonical `compose.yaml` plus `deploy/compose.server.yaml`; they always -start the mandatory `frontend` and `core` services. Do not copy or maintain a standalone +start `frontend`, `core`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`. +Startup is CPU-first; use `THOTH_ENABLE_EMBEDDING_GPU=1` only when the server intentionally +exposes a supported GPU device to Docker. Qdrant is a derived but persistent index, and the +Ollama model cache persists the exact `qwen3-embedding:0.6b` model (`1024` dimensions, cosine +distance) for offline reuse. Do not copy or maintain a standalone application Compose file. Copy `docs/install/examples/thothii-installation.server.yaml` to the protected operator directory and preserve its required session-server overlay, exactly one Git transport override, and generated connector-secret override. @@ -221,6 +226,10 @@ Its schema-v1 output is `migration_required`; explicitly supply collection ident and the reviewed v3 contract before commit. Never import `${ENV}` values or copy secret files. +Schema-v3 is the only operational descriptor contract. Schema-v1/v2 descriptors remain +`migration_required` until an explicit reviewed migration writes version 3. One workspace owns one Qdrant collection; schema, Evidence, and Memory records share it and stay separated by payload +`kind`. + After valid bootstrap, Git outage retains the active snapshot with `degraded: true`. Repair egress/DNS/CA/credentials, pull, and confirm healthy status. Roll back a bad descriptor through a reviewed Git revert/release branch, advance the remote through normal policy, pull it, and confirm @@ -248,3 +257,24 @@ revision and monitor status. If snapshots are missing or corrupt, stop the servi newest verified registry backup, start it privately, verify status, and then reopen proxy traffic. A first-bootstrap failure has no fallback: repair remote trust rather than creating an unreviewed runtime checkout. + +## Qdrant backup/restore and cache recovery + +Use the repository helpers for Qdrant backup/restore: + +```sh +./scripts/vector-backup.sh --project-name thothii --output /secure/backups/thoth-qdrant-2026-08-08.tar +./scripts/vector-restore.sh --project-name thothii --input /secure/backups/thoth-qdrant-2026-08-08.tar --confirm-project thothii +``` + +Qdrant backup/restore targets exactly one labeled `qdrant-data` volume for the named Compose +project. Restore requires the exact repeated project confirmation, validates the archive before +stopping `qdrant`, stages rollback content, and restores in place only for that project-scoped +volume. It does not migrate schema-v1/v2 workspaces, rename collections, or resolve semantic-index +incompatibilities. + +The Ollama model cache is a recoverable local cache, not the canonical semantic source of truth. +You may back up `embedding-models` for faster offline recovery, but a cache loss is recoverable by +re-pulling `qwen3-embedding:0.6b` through `embedding-model-init`. + +Only the Git remote, DWH, LLM, and optional bastion endpoints stay external. diff --git a/docs/installazione-docker-4-contesti.md b/docs/installazione-docker-4-contesti.md index 2021b02c..226d62c7 100644 --- a/docs/installazione-docker-4-contesti.md +++ b/docs/installazione-docker-4-contesti.md @@ -8,8 +8,8 @@ ThothII usa una topologia Compose unica: - `embedding` - `embedding-model-init` -Qdrant e Ollama embedding sono servizi interni obbligatori del progetto Compose. Restano -esterni solo DWH e LLM. +Qdrant e Ollama embedding sono servizi interni obbligatori del progetto Compose. Restano esterni solo DWH e LLM. Il modello fissato è `qwen3-embedding:0.6b` con 1024 dimensioni e distanza +coseno; `embedding-model-init` lo prepara prima dell'avvio di `core`. ## Comando standard locale diff --git a/docs/workspace-diagnostic-protocol.md b/docs/workspace-diagnostic-protocol.md index 159ddb43..e057f934 100644 --- a/docs/workspace-diagnostic-protocol.md +++ b/docs/workspace-diagnostic-protocol.md @@ -1,44 +1,47 @@ # Workspace diagnostic protocol -This is the operator contract for testing a workspace on one ThothII installation. The -Git-shared descriptor declares *what* can be checked; the installation supplies the selected -transport and the local bindings. No secret value, certificate content, SSH key, or response body -belongs in the descriptor, this document, a generated `.env.example`, or diagnostic output. +This is the operator contract for testing a workspace on one ThothII installation. The Git-shared +descriptor declares what can be checked; the installation supplies only the selected DWH +transport and local secret-file bindings. No secret value, certificate content, SSH key, or +response body belongs in the descriptor, generated `.env.example` files, or diagnostic output. ## Scope and safety rules -- The descriptor is schema version 2. Its vector `database` and `schema` are required identity - fields; they are not copied from the DWH, even when both services share PostgreSQL. -- Version 1 descriptors are readable only and have `migration_required` status. An explicit - migration supplies `semantic_index.vector_store.database` and `.schema`, writes version 2, and - must never infer either from `dwh`. -- Each diagnostic is bounded by the configured workspace diagnostic timeout. Redirects are - rejected, response bodies stay inside the adapter, and browser-visible errors are limited to - `binding_missing`, `connector_unavailable`, and `semantic_index_incompatible`. -- A REST path is descriptor-declared, origin-relative, starts with one `/`, and has no query or - fragment. The client may use only the declared method, path, auth mode, and response-field names. -- `auth: none` sends no credential; `auth: bearer` reads a local file and sends - `Authorization: Bearer `; `auth: x-api-key` sends `x-api-key: `. - The resolver, rendered runtime endpoint, and diagnoser do not require or read an API-key file - for an `auth: none` diagnostic. File content is never logged or returned. +- The operational descriptor is schema version 3. +- Schema-v1/v2 descriptors are readable only and remain `migration_required` until an explicit + reviewed migration writes schema version 3. +- One workspace owns one Qdrant collection. +- Qdrant and Ollama are internal services. Operators do not bind external vector or embedding + transports for active manuals or supported diagnostics. +- Each diagnostic is bounded by the configured timeout. Redirects are rejected, response bodies + stay inside the adapter, and browser-visible errors are limited to `binding_missing`, + `connector_unavailable`, and `semantic_index_incompatible`. -## Canonical descriptor additions +## Canonical descriptor contract ```yaml +workspace: + schema_version: 3 + id: psd-clinical + name: PSD Clinical + language: it + +dwh: + engine: postgres + database: warehouse + schema: datawarehouse + supported_transports: [postgres_direct, rest_api, ssh_tunnel] + semantic_index: vector_store: - engine: pgvector - database: vector_database - schema: vectors - collection: clinical_documents - dimensions: 768 + engine: qdrant + collection: psd-clinical + dimensions: 1024 distance: cosine - supported_transports: [pgvector_direct, rest_api, ssh_tunnel] - vector_writer: {} # optional: declares a separately bound writer capability embedding: - provider: ollama_compatible - model: nomic-embed-text-v2-moe - dimensions: 768 + provider: ollama_internal + model: qwen3-embedding:0.6b + dimensions: 1024 diagnostics: dwh_rest: @@ -46,35 +49,25 @@ diagnostics: path: /rpc/ping auth: bearer response: { database: database, schema: schema } - vector_rest: - metadata: - method: GET - path: /vector/metadata - auth: bearer - response: { collection: collection, dimensions: dimensions, distance: distance } - reversible_probe: - method: POST - path: /vector/diagnostic-probe - auth: bearer - response: { operation: operation } - embedding: - method: GET - path: /models - auth: none - response: { model: model, dimensions: dimensions } ``` -`diagnostics.dwh_rest` requires DWH `rest_api`; `diagnostics.vector_rest` requires vector -`rest_api`. `reversible_probe` is optional, but when present it must be authenticated `POST` and -declare the response field that echoes the requested `operation`. Response-map values are JSON -object field names, not values to be put in Git. +The semantic-index contract is fixed: + +- `engine: qdrant` +- collection name equals the workspace-owned portable identifier +- `qwen3-embedding:0.6b` +- `1024` dimensions +- cosine distance + +If any active collection reports a different model pairing, dimension, or distance, diagnostics +must return `semantic_index_incompatible` rather than silently rewriting data. ## Installation-local variable contract Replace `` with the immutable workspace ID converted to upper case with hyphens changed to underscores. For example, `psd-clinical` becomes `PSD_CLINICAL`. Set only the variables for the -selected transport. Every `*_FILE` value is an absolute path to a regular, readable file inside an -approved local secret root; it is never the secret itself. +selected DWH transport. Every `*_FILE` value is an absolute path to a regular, readable file +inside an approved local secret root; it is never the secret itself. | Connector and transport | Required local variables | | --- | --- | @@ -82,25 +75,9 @@ approved local secret root; it is never the secret itself. | DWH `postgres_direct` | `THT_WS__DWH_HOST`, `THT_WS__DWH_PORT`, `THT_WS__DWH_USER`, `THT_WS__DWH_PASSWORD_FILE`; optional `THT_WS__DWH_TLS_CA_FILE` | | DWH `rest_api` | `THT_WS__DWH_BASE_URL`; `THT_WS__DWH_API_KEY_FILE` only for `bearer`/`x-api-key`; optional `THT_WS__DWH_TLS_CA_FILE` | | DWH `ssh_tunnel` | `THT_WS__DWH_USER`, `THT_WS__DWH_PASSWORD_FILE`, `THT_WS__DWH_SSH_HOST`, `THT_WS__DWH_SSH_PORT`, `THT_WS__DWH_SSH_USER`, `THT_WS__DWH_SSH_PRIVATE_KEY_FILE`, `THT_WS__DWH_SSH_KNOWN_HOSTS_FILE`, `THT_WS__DWH_SSH_TARGET_HOST`, `THT_WS__DWH_SSH_TARGET_PORT`; optional `THT_WS__DWH_TLS_CA_FILE` | -| Vector selection | `THT_WS__VECTOR_TRANSPORT` | -| Vector `pgvector_direct` | `THT_WS__VECTOR_HOST`, `THT_WS__VECTOR_PORT`, `THT_WS__VECTOR_USER`, `THT_WS__VECTOR_PASSWORD_FILE`; optional `THT_WS__VECTOR_TLS_CA_FILE` | -| Vector `rest_api` | `THT_WS__VECTOR_BASE_URL`; `THT_WS__VECTOR_API_KEY_FILE` only for `bearer`/`x-api-key`; optional `THT_WS__VECTOR_TLS_CA_FILE` | -| Vector `ssh_tunnel` | `THT_WS__VECTOR_USER`, `THT_WS__VECTOR_PASSWORD_FILE`, `THT_WS__VECTOR_SSH_HOST`, `THT_WS__VECTOR_SSH_PORT`, `THT_WS__VECTOR_SSH_USER`, `THT_WS__VECTOR_SSH_PRIVATE_KEY_FILE`, `THT_WS__VECTOR_SSH_KNOWN_HOSTS_FILE`, `THT_WS__VECTOR_SSH_TARGET_HOST`, `THT_WS__VECTOR_SSH_TARGET_PORT`; optional `THT_WS__VECTOR_TLS_CA_FILE` | -| Optional vector writer | `THT_WS__VECTOR_WRITER_API_KEY_FILE` | -| Embedding service | `THT_WS__EMBEDDING_BASE_URL`; optional `THT_WS__EMBEDDING_API_KEY_FILE`, `THT_WS__EMBEDDING_TLS_CA_FILE` | -For the example workspace, the optional writer name is exactly -`THT_WS_PSD_CLINICAL_VECTOR_WRITER_API_KEY_FILE`. It must resolve to a different local file from -`THT_WS_PSD_CLINICAL_VECTOR_API_KEY_FILE`; a reader key is never substituted for a writer key. - -### Private-CA REST limitation - -The current REST diagnostic adapters use the platform `fetch` implementation and cannot load a -per-request private CA. Therefore a REST diagnostic with any `*_TLS_CA_FILE` binding is refused -rather than silently disabling certificate verification. Use an HTTPS endpoint trusted by the -runtime trust store, use direct or SSH transport where the native PostgreSQL client can validate -the local CA file, or arrange TLS termination at a trusted boundary. This limitation applies to -DWH REST, vector metadata/write REST, and embedding REST diagnostics. +There are no supported `THT_WS__VECTOR_*` or +`THT_WS__EMBEDDING_*` installation bindings in the active operator contract. ## DWH diagnostic @@ -113,15 +90,7 @@ SELECT current_database() AS database, current_schema() AS schema Both returned values must equal the descriptor's DWH database and schema. -`*_TLS_CA_FILE` is optional for direct and SSH PostgreSQL diagnostics. When provided, it is used -with certificate verification; when absent, the native client still requires a valid certificate -chain from the runtime system trust store. Absence never disables TLS verification. - -An SSH tunnel changes only the TCP peer to loopback. The forwarded PostgreSQL TLS connection sets -its server name to `_SSH_TARGET_HOST`, so certificate hostname validation remains against the -declared remote target rather than `127.0.0.1`. - -For REST, the descriptor above declares the exact ping: +For REST, the descriptor-declared request is for example: ```text POST _DWH_BASE_URL>/rpc/ping @@ -129,87 +98,27 @@ Authorization: Bearer ``` It has no request body. A 2xx response must be a JSON object whose declared `database` and -`schema` fields equal `dwh.database` and `dwh.schema`. For the sample response map, that is: +`schema` fields match the descriptor. -```json -{ "database": "warehouse", "schema": "datawarehouse" } -``` +## Internal semantic-service diagnostic -The values are illustrative resource identities, not credentials. A different response-field map -is valid only when the descriptor declares it. +Schema-v3 workspace diagnostics also verify the internal semantic infrastructure through backend +configuration: -## Vector metadata diagnostic +- Qdrant must be reachable at the installation-owned internal URL. +- The workspace-owned collection must exist or be creatable with `1024` dimensions and cosine + distance. +- Ollama must provide `qwen3-embedding:0.6b`. +- A bounded embed probe must return exactly `1024` dimensions. -Direct and SSH vector checks connect to the *vector* `database` and `schema`, not the DWH -identity. They inspect the declared collection's vector column and index and must find the exact -collection, integer `dimensions`, and `distance` (`cosine`, `l2`, or `inner_product`) declared in -`semantic_index.vector_store`. - -Their optional `*_TLS_CA_FILE` follows the same verified private-CA-or-system-trust rule as the -DWH diagnostic. For an SSH tunnel, their TLS server name is likewise the declared vector -`SSH_TARGET_HOST`, not the loopback listener. - -For REST, the exact descriptor-declared request is, for example: - -```text -GET _VECTOR_BASE_URL>/vector/metadata -Authorization: Bearer -``` - -It has no request body. A 2xx JSON object must supply the declared `collection`, `dimensions`, and -`distance` fields. All three values must exactly match the vector-store contract; an integer -dimension is required. Metadata from a similarly named collection, a different metric, or a -different dimension makes the semantic index incompatible. - -## Reversible vector writer probe - -Ordinary validation is reader-only. A write probe runs only when all of the following are true: - -1. The operator explicitly requests it. -2. The descriptor has `semantic_index.vector_writer: {}`. -3. The descriptor declares an authenticated `diagnostics.vector_rest.reversible_probe` with an - `operation` response field. -4. The selected vector transport is `rest_api`. -5. `THT_WS__VECTOR_WRITER_API_KEY_FILE` exists locally and is distinct from the reader - API-key file. - -The probe uses the declared `POST` endpoint twice, with the same generated ID and the writer key: - -```json -{ "operation": "create", "id": "diagnostic:", "collection": "", "dimensions": 768 } -``` - -then: - -```json -{ "operation": "remove", "id": "diagnostic:", "collection": "" } -``` - -Both requests require a 2xx JSON response whose declared `operation` field equals the requested -`create` or `remove` operation. Cleanup is attempted in `finally`, including after a write timeout -or error. The endpoint must implement both operations as a bounded, reversible diagnostic -operation; an upsert-only endpoint is prohibited. It must not retain, index, or expose diagnostic -records. If the writer capability or its local binding is absent, validation remains reader-only -and no write request is sent. - -## Embedding dimensions diagnostic - -The embedding request is descriptor-declared, for example: - -```text -GET _EMBEDDING_BASE_URL>/models -``` - -It has no body and uses the declared authentication mode. A 2xx JSON object must contain the -declared model field equal to `semantic_index.embedding.model` and a declared dimensions field that -is an integer. That integer must equal both `semantic_index.embedding.dimensions` and -`semantic_index.vector_store.dimensions`. +These checks use the private Compose services and never require operator-supplied vector or +embedding URLs, transports, or credentials. ## SSH host verification and tunnel lifecycle -For either DWH or vector `ssh_tunnel`, the known-hosts file is mandatory and is verified before a -connection is accepted. The tunnel is a short-lived loopback forward for the diagnostic only. The -effective OpenSSH constraints are: +For DWH `ssh_tunnel`, the known-hosts file is mandatory and is verified before a connection is +accepted. The tunnel is a short-lived loopback forward for the diagnostic only. The effective +OpenSSH constraints are: ```text -N -v @@ -224,19 +133,16 @@ effective OpenSSH constraints are: ``` The local listener is `127.0.0.1` only. The process is terminated in cleanup after the direct -probe, on timeout, or on failure. There is no accept-new mode, no disabled host-key checking, and -no persistent forwarding. +probe, on timeout, or on failure. -In this release, `ssh_tunnel` is therefore a diagnostic-only connector transport. A successful -probe is followed by `workspace_not_activatable`, and `POST /sessions` rejects the workspace before -persisting a manifest or starting Pi. Use direct PostgreSQL/pgvector or REST for runtime sessions -until the backend owns a tunnel for the full runtime lifecycle. This restriction does not apply to -using SSH as the transport for the workspace Git remote. +In this release, `ssh_tunnel` remains a diagnostic-only DWH transport. A successful probe is +followed by `workspace_not_activatable`, and `POST /sessions` rejects the workspace before +persisting a manifest or starting Pi. This restriction does not apply to SSH transport for the +workspace Git remote. ## Reader-only fallback -A workspace may be fully valid in Git but non-activatable locally when a required reader binding, -secret file, host verification, TLS check, or declared diagnostic fails. That state does not alter -the shared descriptor and does not permit a new session on that installation. It may still be -published and activated elsewhere with valid local bindings. Missing optional writer capability is -not a reader failure: it leaves the workspace in reader-only mode and suppresses the writer probe. +A workspace may be fully valid in Git but non-activatable locally when a required DWH binding, +secret file, host verification, TLS check, or declared DWH diagnostic fails. That state does not +alter the shared descriptor and does not permit a new session on that installation. It may still +be published and activated elsewhere with valid local bindings. diff --git a/scripts/test-verify-workspace-install-docs.sh b/scripts/test-verify-workspace-install-docs.sh index 8be2621a..309e9098 100755 --- a/scripts/test-verify-workspace-install-docs.sh +++ b/scripts/test-verify-workspace-install-docs.sh @@ -11,6 +11,7 @@ trap 'rm -f "$output" "$verifier_functions"; rm -rf "$negative_root"' EXIT HUP I "$root/scripts/verify-workspace-install-docs.sh" --fixtures-only >"$output" for fixture in \ + "internal semantic infrastructure documentation contract" \ "local installation guide contract" \ "source update fail-closed semantics" \ "Windows line-ending recovery guide contract" \ @@ -35,6 +36,25 @@ for fixture in \ } done +grep -Fq '## Internal Qdrant + Ollama semantic infrastructure' "$root/PROJECT_STATE.md" || { + echo "PROJECT_STATE.md does not record the internal Qdrant/Ollama snapshot" >&2 + exit 1 +} +grep -Fq 'Qdrant and Ollama are internal Compose services' "$root/AGENTS.md" || { + echo "AGENTS.md does not record the stable internal semantic-service guidance" >&2 + exit 1 +} +grep -Fq 'Do not add vector or embedding endpoint credentials to the bundle.' \ + "$root/deploy/secrets/README.md" || { + echo "secret bundle guide still permits vector/embedding runtime secrets" >&2 + exit 1 +} +if rg -n 'engine: pgvector|provider: ollama_compatible|THT_WS__VECTOR_TRANSPORT|THT_WS__EMBEDDING_BASE_URL' \ + "$root/docs/workspace-diagnostic-protocol.md"; then + echo "workspace diagnostic protocol still documents external vector or embedding contracts" >&2 + exit 1 +fi + server_guide="$root/docs/install/server.md" grep -Fq 'scripts/prepare-server-pi-state.sh /srv/thothii/pi-state 10001 10001' "$server_guide" || { echo "server guide does not initialize nested Pi-state targets before Compose" >&2 diff --git a/scripts/verify-workspace-install-docs.sh b/scripts/verify-workspace-install-docs.sh index 3a8ed450..9b194702 100755 --- a/scripts/verify-workspace-install-docs.sh +++ b/scripts/verify-workspace-install-docs.sh @@ -43,6 +43,18 @@ verify_path_variable_values() { done <"$source" } +require_absent() { + local source="$1" label="$2" + shift 2 + local forbidden + for forbidden in "$@"; do + if grep -Fq -- "$forbidden" "$source"; then + echo "$label contains forbidden text: $forbidden" >&2 + return 1 + fi + done +} + require_headings() { local source="$1" label="$2" shift 2 @@ -67,6 +79,80 @@ require_text() { done } +verify_internal_semantic_infrastructure_docs() { + local readme="$root/README.md" + local agents="$root/AGENTS.md" + local project_state="$root/PROJECT_STATE.md" + local local_manual="$root/docs/install/local-workspace-registry.md" + local server_manual="$root/docs/install/server-workspace-registry.md" + local compact_manual="$root/docs/installazione-docker-4-contesti.md" + local diagnostics="$root/docs/workspace-diagnostic-protocol.md" + local memory="$root/docs/gestione-memory.md" + local secrets="$root/deploy/secrets/README.md" + + require_text "$readme" "README" \ + 'mandatory stack is `frontend`, `core`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`' \ + '`qwen3-embedding:0.6b`' \ + '`1024`' \ + '`qdrant-data`' \ + '`embedding-models`' \ + '`--confirm-project`' || return 1 + require_text "$agents" "AGENTS.md" \ + 'Run the local Docker stack with `./scripts/run-stack.sh` after creating `deploy/env/local.env`; it starts the base+local Compose' \ + 'Qdrant and Ollama are internal Compose services' \ + 'DWH and LLM remain external configuration endpoints.' || return 1 + require_text "$project_state" "PROJECT_STATE.md" \ + '## Internal Qdrant + Ollama semantic infrastructure' \ + 'Schema-v3 descriptors are operational; schema-v1/v2 descriptors remain `migration_required` until an explicit reviewed migration writes schema version 3.' \ + 'One workspace owns one Qdrant collection' || return 1 + + for manual in "$local_manual" "$server_manual"; do + require_text "$manual" "$(basename "$manual")" \ + '`frontend`, `core`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`' \ + '`qwen3-embedding:0.6b`' \ + '`1024`' \ + '`migration_required`' \ + 'One workspace owns one Qdrant collection' || return 1 + done + require_text "$local_manual" "local workspace manual" \ + 'CPU-first' \ + 'THOTH_ENABLE_EMBEDDING_GPU=1' \ + 'Qdrant is a derived but persistent index' || return 1 + require_text "$server_manual" "server workspace manual" \ + 'Only the Git remote, DWH, LLM, and optional bastion endpoints stay external.' \ + 'Ollama model cache' \ + 'Qdrant backup/restore' || return 1 + + require_text "$compact_manual" "four-context install note" \ + 'Qdrant e Ollama embedding sono servizi interni obbligatori del progetto Compose. Restano esterni solo DWH e LLM.' \ + 'qwen3-embedding:0.6b' \ + '1024 dimensioni' || return 1 + + require_text "$diagnostics" "workspace diagnostic protocol" \ + 'schema version 3' \ + 'One workspace owns one Qdrant collection.' \ + '`semantic_index_incompatible`' || return 1 + require_absent "$diagnostics" "workspace diagnostic protocol" \ + 'engine: pgvector' \ + 'provider: ollama_compatible' \ + 'THT_WS__VECTOR_TRANSPORT' \ + 'THT_WS__EMBEDDING_BASE_URL' || return 1 + + require_text "$memory" "memory guide" \ + 'Indice Qdrant' \ + 'Qdrant resta un indice derivato ma persistente' \ + '`kind`' || return 1 + require_absent "$memory" "memory guide" \ + 'Indice pgvector' \ + 'save-one costruisce un solo `VectorRecord` e lo invia all''indice pgvector.' || return 1 + + require_text "$secrets" "deploy secrets guide" \ + '`THT_MODEL_API_KEY`, `THT_DWH_API_KEY`, `THT_CA`, and `THT_SSL_CA`' \ + 'Do not add vector or embedding endpoint credentials to the bundle.' || return 1 + require_absent "$secrets" "deploy secrets guide" \ + 'PI_PROVIDER_API_KEY' || return 1 +} + verify_local_guide() { local guide="$root/docs/install/local.md" [[ -f "$guide" ]] || { @@ -1459,6 +1545,8 @@ NODE case "$mode" in --fixtures-only) [[ $# -eq 1 ]] || { echo "usage: $0 --fixtures-only" >&2; exit 2; } + verify_internal_semantic_infrastructure_docs + echo "internal semantic infrastructure documentation contract passed" verify_local_guide verify_windows_line_endings_guide verify_pi_management_guide @@ -1476,11 +1564,13 @@ case "$mode" in [[ $# -eq 2 && "$profile" =~ ^(local|server)$ ]] \ || { echo "usage: $0 --profile {local|server}" >&2; exit 2; } if [[ "$profile" == local ]]; then + verify_internal_semantic_infrastructure_docs verify_local_guide verify_windows_line_endings_guide verify_pi_management_guide verify_local_installation_example else + verify_internal_semantic_infrastructure_docs verify_server_guide verify_reverse_proxy_nginx_guide verify_reverse_proxy_caddy_guide