docs: harden internal semantic doc state

2026-08-08 21:00:50 +02:00
parent 22c3512ac8
commit aa502dcf7d
4 changed files with 275 additions and 59 deletions
@@ -59,3 +59,35 @@ Key outcomes:
 - The broader repository still contains historical design/spec material that references older
  pgvector/external-embedding architecture; this task intentionally updated operator/current-state
  documentation and the corresponding contract tests, not historical planning documents.
+
+## Fix round 1/5 — 2026-08-08
+
+Addressed reviewer findings:
+
+- Moved superseded rollout/state blocks in `PROJECT_STATE.md` behind an explicit
+  `## Historical snapshots and archived reference notes` boundary.
+- Renamed superseded snapshot headings so historical notes no longer present as active `LIVE`
+  state.
+- Added a current-state regression that rejects contradictory active blocks (for example:
+  schema-v2 operational, two-service active stack, or external vector/embedding runtime claims
+  before the historical boundary).
+- Refactored new internal-semantic doc checks away from exact-sentence coupling:
+  - parse `compose.yaml` structurally with YAML;
+  - parse workspace examples structurally with YAML;
+  - inspect backup/restore stable usage interface;
+  - keep targeted forbidden-term checks for active docs while allowing historical sections;
+  - use regex/concept checks for prose.
+
+Evidence:
+
+```sh
+./scripts/test-verify-workspace-install-docs.sh
+./scripts/verify-workspace-install-docs.sh --fixtures-only
+git diff --check
+```
+
+Observed RED before the fix:
+
+```text
+PROJECT_STATE.md: missing Historical snapshots boundary
+```
@@ -32,7 +32,9 @@
  schema-v3 migration messaging, Qdrant collection ownership, Qdrant backup/restore safety, and
  the absence of active external vector/embedding operator bindings from current manuals.

-## Unified deployment release gate — Task 13 (2026-08-05)
+## Historical snapshots and archived reference notes
+
+### Historical snapshot — Unified deployment release gate, Task 13 (2026-08-05)

 - **Release coverage.** `scripts/unified-deployment-smoke.sh` gates the two-service render/build,
  frontend-to-core routing, embedded pinned Pi, Git registry bootstrap, offline recreation, valid
@@ -80,7 +82,7 @@
  resource could be created, so authenticated workspace/fail-closed session behavior remains an
  explicit release gate. Native Windows PowerShell/Docker execution also remains pending.

-## Portable deployment decoupling — LIVE 2026-08-05
+### Historical snapshot — Portable deployment decoupling (superseded 2026-08-08)

 - **Mandatory stack.** The supported Compose stack is exactly `frontend` plus `core`; use the
  base file with `deploy/compose.local.yaml`, or with `deploy/compose.server.yaml` plus the
@@ -107,7 +109,7 @@
  and optional overrides, while the category-based coupling scan covers runtime, Docker smoke,
  install, operator, and positive deployment-test contracts and propagates scanner errors.

-## Portable Git workspace registry — source integration (2026-08-04)
+### Historical snapshot — Portable Git workspace registry, pre-schema-v3 (superseded 2026-08-08)

 - **Source of truth and scope.** The canonical workspace repository is a generic Git remote,
  configured only by `THT_WORKSPACE_GIT_REMOTE` and `THT_WORKSPACE_GIT_BRANCH` (there is no
@@ -140,7 +142,7 @@
  harness run remains a release command for the deployment environment; the earlier local
  long-running harness run was intentionally cancelled before it produced a final result.

-## Session summary redesign — LIVE 2026-07-23
+### Historical snapshot — Session summary redesign (2026-07-23)

 - Session documents are projected at read time in outcome-first order: original question,
  final SQL, persisted data preview, revised question, assumptions, one memory list, then
@@ -163,7 +165,7 @@
  `sha256:8311ca1308b459ece7236bf143da7b1a226ff4082fed924e1b5a207c24b6ca29`
  is running. Frontend and `/api/health` both returned HTTP 200.

-## Local Pi user auth + startup failure handling — LIVE 2026-07-21
+### Historical snapshot — Local Pi user auth + startup failure handling (2026-07-21)

 - The PSD Docker profile now bind-mounts the configurable host `PI_AUTH_FILE` read-only at
  `/home/thoth/.pi/agent/auth.json`; on this Mac it resolves to the real user profile
@@ -185,7 +187,7 @@
  `a390c8b8-0a91-4a37-967b-ce7ff9be9797`, `a2f974b2-4c48-4967-b4b6-afdbc2b2d541`, and
  `f66e1959-3c71-4b10-8aa1-606992046b7e` (API delete 204, subsequent lookup 404 for each).

-## User-owned sessions cutover — prepared, manual gate pending (2026-07-16)
+### Historical snapshot — User-owned sessions cutover (2026-07-16)

 - **Target contract:** the public server runs `AUTH_MODE=upstream` with Task 4 portal identity
  forwarding and Task 5 principal enforcement deployed together. Its session source of truth is
@@ -207,7 +209,7 @@
  release; never re-enable filesystem persistence, restore the archive into production, or
  dual-write during rollback.

-## Historical deployment — Docker locale (Profile A, co-located) — superseded 2026-08-05
+### Historical snapshot — Docker locale deployment, Profile A (superseded 2026-08-05)

 ThothII gira in Docker sul server co-locato, **embedded nel portale omics_portal** a `https://aritmolab.policlinicosandonato.it/datamart-builder` (backend invisibile, tutto same-origin via nginx del portale).

@@ -132,6 +132,72 @@ sed '/^case "\$mode" in/,$d' "$root/scripts/verify-workspace-install-docs.sh" >"
 # shellcheck source=/dev/null
 source "$verifier_functions"

+project_state_fixture="$negative_root/project-state.md"
+python3 - "$root/PROJECT_STATE.md" "$project_state_fixture" <<'PY'
+import pathlib, sys
+source = pathlib.Path(sys.argv[1]).read_text()
+target = pathlib.Path(sys.argv[2])
+marker = source.index("## Historical snapshots")
+contradiction = """
+## Contradictory release note — LIVE 2026-08-08
+
+- Schema-v2 descriptors are operational again.
+- The supported Compose stack is exactly `frontend` plus `core`.
+- DWH, vector DB, embedding, LLM, and reverse-proxy services are external configurable endpoints.
+
+"""
+target.write_text(source[:marker] + contradiction + source[marker:])
+PY
+project_state_output="$negative_root/project-state-output"
+set +e
+verify_project_state_current_contract "$project_state_fixture" contradictory-project-state >"$project_state_output" 2>&1
+project_state_status=$?
+set -e
+if [[ $project_state_status -eq 0 ]] || ! grep -Fq "contradictory active text" "$project_state_output"; then
+  echo "contradictory current-state fixture was not rejected correctly" >&2
+  cat "$project_state_output" >&2
+  exit 1
+fi
+
+workspace_fixture="$negative_root/workspace-invalid.yaml"
+python3 - "$root/deploy/workspaces/example.yaml" "$workspace_fixture" <<'PY'
+import pathlib, sys, yaml
+doc = yaml.safe_load(pathlib.Path(sys.argv[1]).read_text())
+doc["semantic_index"]["embedding"]["dimensions"] = 768
+pathlib.Path(sys.argv[2]).write_text(yaml.safe_dump(doc, sort_keys=False))
+PY
+workspace_output="$negative_root/workspace-output"
+set +e
+verify_workspace_descriptor_semantic_contract "$workspace_fixture" invalid-workspace >"$workspace_output" 2>&1
+workspace_status=$?
+set -e
+if [[ $workspace_status -eq 0 ]] || ! grep -Fq "embedding dimensions must be 1024" "$workspace_output"; then
+  echo "semantic workspace fixture was not rejected correctly" >&2
+  cat "$workspace_output" >&2
+  exit 1
+fi
+
+project_state_positive="$negative_root/project-state-positive.md"
+cat >"$project_state_positive" <<'EOF'
+# ThothII — Project State
+
+> Starting-point snapshot.
+
+## Internal Qdrant + Ollama semantic infrastructure — LIVE 2026-08-08
+
+- Schema-v3 descriptors are operational and v1/v2 remain `migration_required`.
+- One workspace owns one Qdrant collection.
+- Only DWH and LLM remain external runtime application endpoints.
+- The internal stack includes `qdrant`, `embedding`, and `embedding-model-init`.
+
+## Historical snapshots — superseded context
+
+### Historical snapshot — previous deployment
+
+- Older notes intentionally live only here.
+EOF
+verify_project_state_current_contract "$project_state_positive" positive-project-state >/dev/null
+
 adapted_reorder="$negative_root/caddy-adapted-reorder.json"
 adapted_bypass="$negative_root/caddy-adapted-bypass.json"
 node - "$adapted_reorder" "$adapted_bypass" <<'NODE'
@@ -55,6 +55,18 @@ require_absent() {
  done
 }

+require_pattern() {
+  local source="$1" label="$2" pattern="$3"
+  python3 - "$source" "$label" "$pattern" <<'PY'
+import pathlib, re, sys
+source = pathlib.Path(sys.argv[1]).read_text()
+label = sys.argv[2]
+pattern = sys.argv[3]
+if not re.search(pattern, source, re.MULTILINE | re.DOTALL):
+    raise SystemExit(f"{label} lacks required pattern: {pattern}")
+PY
+}
+
 require_headings() {
  local source="$1" label="$2"
  shift 2
@@ -79,10 +91,136 @@ require_text() {
  done
 }

+verify_compose_internal_semantic_contract() {
+  python3 - "$root/compose.yaml" <<'PY'
+import sys, yaml, pathlib
+doc = yaml.safe_load(pathlib.Path(sys.argv[1]).read_text())
+services = doc["services"]
+expected = {"core", "frontend", "qdrant", "embedding", "embedding-model-init"}
+if set(services) != expected:
+    raise SystemExit(f"compose.yaml services mismatch: {sorted(services)}")
+core = services["core"]
+env = core["environment"]
+for key, value in {
+    "THT_INTERNAL_QDRANT_URL": "http://qdrant:6333",
+    "THT_INTERNAL_EMBEDDING_URL": "http://embedding:11434",
+    "THT_INTERNAL_EMBEDDING_MODEL": "qwen3-embedding:0.6b",
+    "THT_INTERNAL_EMBEDDING_DIMENSIONS": "1024",
+}.items():
+    if env.get(key) != value:
+        raise SystemExit(f"core missing semantic env {key}={value}")
+for forbidden in ("THT_VEC_REST_URL", "THT_VEC_WRITE_REST_URL", "THT_OLLAMA_URL"):
+    if forbidden in env:
+        raise SystemExit(f"core still exposes deprecated env {forbidden}")
+if core["depends_on"]["qdrant"]["condition"] != "service_healthy":
+    raise SystemExit("core must wait for qdrant health")
+if core["depends_on"]["embedding-model-init"]["condition"] != "service_completed_successfully":
+    raise SystemExit("core must wait for model init success")
+for name, port in (("qdrant", "6333"), ("embedding", "11434")):
+    service = services[name]
+    if "ports" in service:
+        raise SystemExit(f"{name} must stay private")
+    if service.get("expose") != [port]:
+        raise SystemExit(f"{name} expose mismatch")
+if "devices" in str(services["embedding"]):
+    raise SystemExit("base embedding service must stay CPU-first")
+volumes = set(doc["volumes"])
+for required in ("qdrant-data", "embedding-models"):
+    if required not in volumes:
+        raise SystemExit(f"missing volume {required}")
+model_init = services["embedding-model-init"]
+if model_init["environment"].get("OLLAMA_MODEL") != "qwen3-embedding:0.6b":
+    raise SystemExit("model init must pin qwen3-embedding:0.6b")
+PY
+}
+
+verify_workspace_descriptor_semantic_contract() {
+  local source="${1:?source required}"
+  local label="${2:-$source}"
+  python3 - "$source" "$label" <<'PY'
+import pathlib, sys, yaml
+path = pathlib.Path(sys.argv[1])
+label = sys.argv[2]
+doc = yaml.safe_load(path.read_text())
+ws = doc["workspace"]
+semantic = doc["semantic_index"]
+vector = semantic["vector_store"]
+embedding = semantic["embedding"]
+if ws["schema_version"] != 3:
+    raise SystemExit(f"{label}: schema_version must be 3")
+if vector["engine"] != "qdrant":
+    raise SystemExit(f"{label}: vector store must be qdrant")
+if vector["collection"] != ws["id"]:
+    raise SystemExit(f"{label}: collection must equal workspace id")
+if vector["dimensions"] != 1024 or vector["distance"] != "cosine":
+    raise SystemExit(f"{label}: vector contract must be 1024/cosine")
+if embedding["provider"] != "ollama_internal":
+    raise SystemExit(f"{label}: embedding provider must be ollama_internal")
+if embedding["model"] != "qwen3-embedding:0.6b":
+    raise SystemExit(f"{label}: embedding model must be qwen3-embedding:0.6b")
+if embedding["dimensions"] != 1024:
+    raise SystemExit(f"{label}: embedding dimensions must be 1024")
+PY
+}
+
+verify_vector_helper_interfaces() {
+  local output status
+  output="$(mktemp "${TMPDIR:-/tmp}/thoth-vector-backup-help.XXXXXX")"
+  set +e
+  "$root/scripts/vector-backup.sh" >"$output" 2>&1
+  status=$?
+  set -e
+  [[ $status -eq 2 ]] || { cat "$output" >&2; rm -f "$output"; echo "vector-backup usage exit mismatch" >&2; return 1; }
+  grep -Eq 'usage: .*--project-name NAME --output FILE' "$output" || { cat "$output" >&2; rm -f "$output"; echo "vector-backup usage contract changed" >&2; return 1; }
+  set +e
+  "$root/scripts/vector-restore.sh" >"$output" 2>&1
+  status=$?
+  set -e
+  [[ $status -eq 2 ]] || { cat "$output" >&2; rm -f "$output"; echo "vector-restore usage exit mismatch" >&2; return 1; }
+  grep -Eq 'usage: .*--project-name NAME --input FILE --confirm-project NAME' "$output" || { cat "$output" >&2; rm -f "$output"; echo "vector-restore usage contract changed" >&2; return 1; }
+  rm -f "$output"
+}
+
+verify_project_state_current_contract() {
+  local source="${1:-$root/PROJECT_STATE.md}"
+  local label="${2:-PROJECT_STATE.md}"
+  python3 - "$source" "$label" <<'PY'
+import pathlib, re, sys
+text = pathlib.Path(sys.argv[1]).read_text()
+label = sys.argv[2]
+marker = re.search(r"^## Historical snapshots\b", text, re.MULTILINE)
+if not marker:
+    raise SystemExit(f"{label}: missing Historical snapshots boundary")
+current = text[:marker.start()]
+historical = text[marker.start():]
+required = [
+    r"Internal Qdrant \+ Ollama semantic infrastructure",
+    r"Schema-v3 descriptors are operational",
+    r"migration_required",
+    r"One workspace owns one Qdrant collection",
+    r"Only DWH and LLM remain external",
+    r"embedding-model-init",
+]
+for pattern in required:
+    if not re.search(pattern, current, re.MULTILINE):
+        raise SystemExit(f"{label}: current section missing {pattern}")
+forbidden = [
+    r"Schema-v2 descriptors are operational",
+    r"supported Compose stack is exactly `frontend` plus `core`",
+    r"DWH, vector DB, embedding, LLM",
+    r"vector DB, embedding, and LLM remain external",
+]
+for pattern in forbidden:
+    if re.search(pattern, current, re.MULTILINE):
+        raise SystemExit(f"{label}: current section still contains contradictory active text: {pattern}")
+if re.search(r"^## .*— LIVE", historical, re.MULTILINE):
+    raise SystemExit(f"{label}: historical section still contains LIVE headings")
+PY
+}
+
 verify_internal_semantic_infrastructure_docs() {
  local readme="$root/README.md"
  local agents="$root/AGENTS.md"
-  local project_state="$root/PROJECT_STATE.md"
  local local_manual="$root/docs/install/local-workspace-registry.md"
  local server_manual="$root/docs/install/server-workspace-registry.md"
  local compact_manual="$root/docs/installazione-docker-4-contesti.md"
@@ -90,67 +228,45 @@ verify_internal_semantic_infrastructure_docs() {
  local memory="$root/docs/gestione-memory.md"
  local secrets="$root/deploy/secrets/README.md"

-  require_text "$readme" "README" \
-    'mandatory stack is `frontend`, `core`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`' \
-    '`qwen3-embedding:0.6b`' \
-    '`1024`' \
-    '`qdrant-data`' \
-    '`embedding-models`' \
-    '`--confirm-project`' || return 1
-  require_text "$agents" "AGENTS.md" \
-    'Run the local Docker stack with `./scripts/run-stack.sh` after creating `deploy/env/local.env`; it starts the base+local Compose' \
-    'Qdrant and Ollama are internal Compose services' \
-    'DWH and LLM remain external configuration endpoints.' || return 1
-  require_text "$project_state" "PROJECT_STATE.md" \
-    '## Internal Qdrant + Ollama semantic infrastructure' \
-    'Schema-v3 descriptors are operational; schema-v1/v2 descriptors remain `migration_required` until an explicit reviewed migration writes schema version 3.' \
-    'One workspace owns one Qdrant collection' || return 1
+  verify_compose_internal_semantic_contract || return 1
+  verify_workspace_descriptor_semantic_contract "$root/deploy/workspaces/example.yaml" "example workspace" || return 1
+  verify_workspace_descriptor_semantic_contract "$root/deploy/workspaces/psd.yaml.example" "psd workspace example" || return 1
+  verify_vector_helper_interfaces || return 1
+  verify_project_state_current_contract "$root/PROJECT_STATE.md" "PROJECT_STATE.md" || return 1

+  require_pattern "$readme" "README" 'mandatory stack.+qdrant.+embedding.+embedding-model-init' || return 1
+  require_pattern "$readme" "README" 'qwen3-embedding:0\.6b' || return 1
+  require_pattern "$readme" "README" 'qdrant-data.+embedding-models' || return 1
+  require_pattern "$readme" "README" 'confirm-project' || return 1
+  require_pattern "$agents" "AGENTS.md" 'Qdrant and Ollama are internal Compose services' || return 1
+  require_pattern "$agents" "AGENTS.md" 'DWH and LLM remain external configuration endpoints' || return 1
  for manual in "$local_manual" "$server_manual"; do
-    require_text "$manual" "$(basename "$manual")" \
-      '`frontend`, `core`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`' \
-      '`qwen3-embedding:0.6b`' \
-      '`1024`' \
-      '`migration_required`' \
-      'One workspace owns one Qdrant collection' || return 1
+    require_pattern "$manual" "$(basename "$manual")" 'qwen3-embedding:0\.6b' || return 1
+    require_pattern "$manual" "$(basename "$manual")" 'migration_required' || return 1
+    require_pattern "$manual" "$(basename "$manual")" 'One workspace owns one Qdrant collection' || return 1
  done
-  require_text "$local_manual" "local workspace manual" \
-    'CPU-first' \
-    'THOTH_ENABLE_EMBEDDING_GPU=1' \
-    'Qdrant is a derived but persistent index' || return 1
-  require_text "$server_manual" "server workspace manual" \
-    'Only the Git remote, DWH, LLM, and optional bastion endpoints stay external.' \
-    'Ollama model cache' \
-    'Qdrant backup/restore' || return 1
-
-  require_text "$compact_manual" "four-context install note" \
-    'Qdrant e Ollama embedding sono servizi interni obbligatori del progetto Compose. Restano esterni solo DWH e LLM.' \
-    'qwen3-embedding:0.6b' \
-    '1024 dimensioni' || return 1
-
-  require_text "$diagnostics" "workspace diagnostic protocol" \
-    'schema version 3' \
-    'One workspace owns one Qdrant collection.' \
-    '`semantic_index_incompatible`' || return 1
+  require_pattern "$local_manual" "local workspace manual" 'CPU-first' || return 1
+  require_pattern "$local_manual" "local workspace manual" 'THOTH_ENABLE_EMBEDDING_GPU=1' || return 1
+  require_pattern "$server_manual" "server workspace manual" 'Qdrant backup/restore' || return 1
+  require_pattern "$server_manual" "server workspace manual" 'Only the Git remote, DWH, LLM, and optional bastion endpoints stay external' || return 1
+  require_pattern "$compact_manual" "four-context install note" 'Qdrant e Ollama embedding.+Restano esterni solo DWH e LLM' || return 1
+  require_pattern "$compact_manual" "four-context install note" '1024 dimensioni' || return 1
+  require_pattern "$diagnostics" "workspace diagnostic protocol" 'schema version 3' || return 1
+  require_pattern "$diagnostics" "workspace diagnostic protocol" 'semantic_index_incompatible' || return 1
  require_absent "$diagnostics" "workspace diagnostic protocol" \
    'engine: pgvector' \
    'provider: ollama_compatible' \
    'THT_WS_<NAMESPACE>_VECTOR_TRANSPORT' \
    'THT_WS_<NAMESPACE>_EMBEDDING_BASE_URL' || return 1
-
-  require_text "$memory" "memory guide" \
-    'Indice Qdrant' \
-    'Qdrant resta un indice derivato ma persistente' \
-    '`kind`' || return 1
+  require_pattern "$memory" "memory guide" 'Indice Qdrant' || return 1
+  require_pattern "$memory" "memory guide" 'indice derivato ma persistente' || return 1
+  require_pattern "$memory" "memory guide" '`kind`' || return 1
  require_absent "$memory" "memory guide" \
    'Indice pgvector' \
-    'save-one costruisce un solo `VectorRecord` e lo invia all''indice pgvector.' || return 1
-
-  require_text "$secrets" "deploy secrets guide" \
-    '`THT_MODEL_API_KEY`, `THT_DWH_API_KEY`, `THT_CA`, and `THT_SSL_CA`' \
-    'Do not add vector or embedding endpoint credentials to the bundle.' || return 1
-  require_absent "$secrets" "deploy secrets guide" \
-    'PI_PROVIDER_API_KEY' || return 1
+    "all'indice pgvector" || return 1
+  require_pattern "$secrets" "deploy secrets guide" 'THT_MODEL_API_KEY.+THT_DWH_API_KEY.+THT_CA.+THT_SSL_CA' || return 1
+  require_pattern "$secrets" "deploy secrets guide" 'Do not add vector or embedding endpoint credentials to the bundle' || return 1
+  require_absent "$secrets" "deploy secrets guide" 'PI_PROVIDER_API_KEY' || return 1
 }

 verify_local_guide() {