diff --git a/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-11-report.md b/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-11-report.md index 8e6c0b33..38ad3124 100644 --- a/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-11-report.md +++ b/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-11-report.md @@ -121,3 +121,30 @@ Observed RED during this round: PROJECT_STATE.md: historical section still contains active/live heading markers compact manual paraphrase lacks required pattern: (esterni solo|solo esterni|restano esterni) ``` + +## Fix round 3/5 — 2026-08-08 + +Addressed reviewer findings: + +- Added table-driven historical-heading fixtures for every Markdown heading level `#` through + `######`; all are rejected after the historical boundary when they contain `LIVE`/current-state + semantics. +- Added small structured ownership tables to the active local/server manuals and to the compact + Italian operator note. +- Added small structured semantic-index ownership tables to the active local/server manuals. +- Replaced the remaining scattered-token relationship checks with explicit structured-section + parsing: + - architecture ownership rows map DWH → external, LLM → external, Qdrant → internal, + Ollama embedding → internal; + - semantic-index ownership rows localize the one-workspace/one-collection contract and the + schema/Evidence/Memory isolation rule. +- Added adversarial fixtures that fail when the same tokens are merely scattered in free text. +- Added structured paraphrase fixtures that pass and omission/inversion fixtures that fail. + +Evidence: + +```sh +./scripts/test-verify-workspace-install-docs.sh +./scripts/verify-workspace-install-docs.sh --fixtures-only +git diff --check +``` diff --git a/docs/install/local-workspace-registry.md b/docs/install/local-workspace-registry.md index b508fe3d..11d217e3 100644 --- a/docs/install/local-workspace-registry.md +++ b/docs/install/local-workspace-registry.md @@ -10,6 +10,15 @@ bindings, credentials, and session data are local, while internal Qdrant/Ollama Compose stack. Never put credentials in workspace YAML, Git, browser drafts, diagnostics, or `.env.example`. +## Architecture ownership contract + +| Component | Ownership | Operator contract | +| --- | --- | --- | +| DWH | External | Installation-local endpoint/binding; never bundled into the Compose semantic stack. | +| LLM | External | Installation-local endpoint/policy choice outside the internal semantic services. | +| Qdrant | Internal | Mandatory private Compose semantic service; persistent `qdrant-data` volume. | +| Ollama embedding | Internal | Mandatory private Compose semantic service for `qwen3-embedding:0.6b`. | + ## Prerequisites - macOS: Docker Desktop, Git, and sufficient volume disk space. Git Credential Manager is useful @@ -179,6 +188,12 @@ Schema-v3 is the only operational descriptor format. Schema-v1/v2 descriptors re `migration_required` until an explicit reviewed migration writes schema version 3. One workspace owns one Qdrant collection; schema, Evidence, and Memory records share that collection and remain isolated by payload `kind`. +## Semantic index ownership contract + +| Scope | Ownership rule | Isolation rule | +| --- | --- | --- | +| Workspace semantic index | Each workspace reserves a single Qdrant collection. | Schema, Evidence, and Memory stay in that one collection and remain isolated by payload `kind`. | + To migrate an existing legacy descriptor, create/clone an empty private remote, set the absolute `THT_SOURCE_ROOT`, transform with absolute paths, review the schema-v1 result, explicitly produce the reviewed schema-v3 contract, then commit/push. The transformer never imports `${ENV}` values diff --git a/docs/install/server-workspace-registry.md b/docs/install/server-workspace-registry.md index 1eeb2332..841bb9a7 100644 --- a/docs/install/server-workspace-registry.md +++ b/docs/install/server-workspace-registry.md @@ -5,6 +5,15 @@ The application image is read-only, secrets are mounted read-only, and sessions Git-validated snapshots. Expose the application only behind an authenticated same-origin reverse proxy; never publish the core port directly. +## Architecture ownership contract + +| Component | Ownership | Operator contract | +| --- | --- | --- | +| DWH | External | Approved installation/server endpoint; not part of the private semantic Compose stack. | +| LLM | External | Approved installation/server endpoint or provider policy outside the semantic stack. | +| Qdrant | Internal | Mandatory private Compose semantic service; persistent `qdrant-data` volume. | +| Ollama embedding | Internal | Mandatory private Compose semantic service for `qwen3-embedding:0.6b`. | + ## Service account, storage, and firewall Create a dedicated host service account and an operator root such as `/srv/thothii`. The core @@ -230,6 +239,12 @@ Schema-v3 is the only operational descriptor contract. Schema-v1/v2 descriptors `migration_required` until an explicit reviewed migration writes version 3. One workspace owns one Qdrant collection; schema, Evidence, and Memory records share it and stay separated by payload `kind`. +## Semantic index ownership contract + +| Scope | Ownership rule | Isolation rule | +| --- | --- | --- | +| Workspace semantic index | Each workspace reserves a single Qdrant collection. | Schema, Evidence, and Memory share that one collection and stay separated by payload `kind`. | + After valid bootstrap, Git outage retains the active snapshot with `degraded: true`. Repair egress/DNS/CA/credentials, pull, and confirm healthy status. Roll back a bad descriptor through a reviewed Git revert/release branch, advance the remote through normal policy, pull it, and confirm diff --git a/docs/installazione-docker-4-contesti.md b/docs/installazione-docker-4-contesti.md index 226d62c7..08cd5db5 100644 --- a/docs/installazione-docker-4-contesti.md +++ b/docs/installazione-docker-4-contesti.md @@ -11,6 +11,15 @@ ThothII usa una topologia Compose unica: Qdrant e Ollama embedding sono servizi interni obbligatori del progetto Compose. Restano esterni solo DWH e LLM. Il modello fissato è `qwen3-embedding:0.6b` con 1024 dimensioni e distanza coseno; `embedding-model-init` lo prepara prima dell'avvio di `core`. +## Contratto sintetico di ownership + +| Componente | Ownership | Contratto operativo | +| --- | --- | --- | +| DWH | Esterno | Endpoint esterno configurato dall'installazione. | +| LLM | Esterno | Endpoint o policy esterna all'infrastruttura semantica interna. | +| Qdrant | Interno | Servizio Compose interno obbligatorio con volume persistente `qdrant-data`. | +| Ollama embedding | Interno | Servizio Compose interno obbligatorio per `qwen3-embedding:0.6b`. | + ## Comando standard locale ```sh diff --git a/scripts/test-verify-workspace-install-docs.sh b/scripts/test-verify-workspace-install-docs.sh index 3017057d..9430229e 100755 --- a/scripts/test-verify-workspace-install-docs.sh +++ b/scripts/test-verify-workspace-install-docs.sh @@ -159,25 +159,29 @@ if [[ $project_state_status -eq 0 ]] || ! grep -Fq "contradictory active text" " exit 1 fi -project_state_live_heading="$negative_root/project-state-live-heading.md" -python3 - "$root/PROJECT_STATE.md" "$project_state_live_heading" <<'PY' +for level in 1 2 3 4 5 6; do + project_state_live_heading="$negative_root/project-state-live-heading-h$level.md" + python3 - "$root/PROJECT_STATE.md" "$project_state_live_heading" "$level" <<'PY' import pathlib, sys source = pathlib.Path(sys.argv[1]).read_text() target = pathlib.Path(sys.argv[2]) +level = int(sys.argv[3]) marker = source.index("## Historical snapshots") historical = source[marker:] -historical = historical.replace("### Historical snapshot — Session summary redesign (2026-07-23)", "### Session summary redesign — LIVE 2026-07-23", 1) +replacement = "#" * level + " Session summary redesign — LIVE 2026-07-23" +historical = historical.replace("### Historical snapshot — Session summary redesign (2026-07-23)", replacement, 1) target.write_text(source[:marker] + historical) PY -set +e -verify_project_state_current_contract "$project_state_live_heading" historical-live-heading >"$project_state_output" 2>&1 -project_state_status=$? -set -e -if [[ $project_state_status -eq 0 ]] || ! grep -Fq "active/live heading markers" "$project_state_output"; then - echo "historical LIVE-heading fixture was not rejected correctly" >&2 - cat "$project_state_output" >&2 - exit 1 -fi + set +e + verify_project_state_current_contract "$project_state_live_heading" "historical-live-heading-h$level" >"$project_state_output" 2>&1 + project_state_status=$? + set -e + if [[ $project_state_status -eq 0 ]] || ! grep -Fq "active/live heading markers" "$project_state_output"; then + echo "historical LIVE-heading fixture was not rejected correctly for heading level $level" >&2 + cat "$project_state_output" >&2 + exit 1 + fi +done workspace_fixture="$negative_root/workspace-invalid.yaml" python3 - "$root/deploy/workspaces/example.yaml" "$workspace_fixture" <<'PY' @@ -225,13 +229,13 @@ import pathlib, sys path = pathlib.Path(sys.argv[1]) text = path.read_text() text = text.replace( - "One workspace owns one Qdrant collection; schema, Evidence, and Memory records share that collection and remain\nisolated by payload `kind`.", - "Each workspace reserves a single Qdrant collection. Schema, Evidence, and Memory stay inside that same collection and are separated by payload `kind`.", + "| Workspace semantic index | Each workspace reserves a single Qdrant collection. | Schema, Evidence, and Memory stay in that one collection and remain isolated by payload `kind`. |", + "| Workspace semantic index | A workspace keeps exactly one Qdrant collection reserved for itself. | Schema, Evidence, and Memory remain together in that collection and are still separated by payload `kind`. |", ) path.write_text(text) PY -require_concept_tokens "$local_manual_paraphrase" "local manual paraphrase" "workspace" "qdrant" "collection" >/dev/null -require_pattern "$local_manual_paraphrase" "local manual paraphrase" '(?is)(single|one|each).{0,120}(workspace|qdrant|collection).{0,120}(reserves|reserve|owns|single)' >/dev/null +semantic_index_spec='{"rows":[{"scope":"workspace semantic index","ownership rule":"(each|one|single|exactly one).*(workspace).*(single|one|exactly one).*(Qdrant).*(collection)|(workspace keeps exactly one qdrant collection reserved for itself)","isolation rule":"schema.*evidence.*memory.*(one|that).*(collection).*(kind|payload)|schema.*evidence.*memory.*together.*collection.*(kind|payload)"}]}' +verify_markdown_table_relationships "$local_manual_paraphrase" "local manual paraphrase" "Semantic index ownership contract" "$semantic_index_spec" >/dev/null local_manual_missing="$negative_root/local-manual-missing.md" cp "$root/docs/install/local-workspace-registry.md" "$local_manual_missing" @@ -240,13 +244,13 @@ import pathlib, sys path = pathlib.Path(sys.argv[1]) text = path.read_text() text = text.replace( - "Schema-v3 is the only operational descriptor format. Schema-v1/v2 descriptors remain\n`migration_required` until an explicit reviewed migration writes schema version 3. One workspace owns one Qdrant collection; schema, Evidence, and Memory records share that collection and remain\nisolated by payload `kind`.\n", - "Schema-v3 is the only operational descriptor format. Schema-v1/v2 descriptors remain `migration_required` until an explicit reviewed migration writes schema version 3.\n", + "| Workspace semantic index | Each workspace reserves a single Qdrant collection. | Schema, Evidence, and Memory stay in that one collection and remain isolated by payload `kind`. |\n", + "", ) path.write_text(text) PY set +e -require_pattern "$local_manual_missing" "local manual missing ownership" '(?is)(single|one|each).{0,120}(workspace|qdrant|collection).{0,120}(reserves|reserve|owns|single)' >"$workspace_output" 2>&1 +verify_markdown_table_relationships "$local_manual_missing" "local manual missing ownership" "Semantic index ownership contract" "$semantic_index_spec" >"$workspace_output" 2>&1 workspace_status=$? set -e if [[ $workspace_status -eq 0 ]]; then @@ -255,27 +259,48 @@ if [[ $workspace_status -eq 0 ]]; then exit 1 fi +local_manual_scattered="$negative_root/local-manual-scattered.md" +cp "$root/docs/install/local-workspace-registry.md" "$local_manual_scattered" +python3 - "$local_manual_scattered" <<'PY' +import pathlib, sys +path = pathlib.Path(sys.argv[1]) +text = path.read_text() +text = text.replace( + "| Workspace semantic index | Each workspace reserves a single Qdrant collection. | Schema, Evidence, and Memory stay in that one collection and remain isolated by payload `kind`. |\n", + "", +) +text += "\nWorkspace. Qdrant. Collection. Schema. Evidence. Memory. Payload kind.\n" +path.write_text(text) +PY +set +e +verify_markdown_table_relationships "$local_manual_scattered" "local manual scattered ownership" "Semantic index ownership contract" "$semantic_index_spec" >"$workspace_output" 2>&1 +workspace_status=$? +set -e +if [[ $workspace_status -eq 0 ]]; then + echo "scattered ownership tokens fixture was not rejected correctly" >&2 + cat "$workspace_output" >&2 + exit 1 +fi + compact_paraphrase="$negative_root/compact-paraphrase.md" cp "$root/docs/installazione-docker-4-contesti.md" "$compact_paraphrase" python3 - "$compact_paraphrase" <<'PY' import pathlib, sys path = pathlib.Path(sys.argv[1]) text = path.read_text() -text = text.replace( - "Qdrant e Ollama embedding sono servizi interni obbligatori del progetto Compose. Restano esterni solo DWH e LLM.", - "Nel Compose di ThothII Qdrant e l'embedding Ollama fanno parte dei servizi interni obbligatori; DWH e LLM restano invece gli unici servizi esterni.", -) +text = text.replace("| DWH | Esterno | Endpoint esterno configurato dall'installazione. |", "| DWH | Esterno | Endpoint esterno deciso dall'installazione. |") +text = text.replace("| LLM | Esterno | Endpoint o policy esterna all'infrastruttura semantica interna. |", "| LLM | Esterno | Endpoint o policy che resta esterna all'infrastruttura semantica interna. |") +text = text.replace("| Qdrant | Interno | Servizio Compose interno obbligatorio con volume persistente `qdrant-data`. |", "| Qdrant | Interno | Servizio Compose interno obbligatorio con il volume persistente `qdrant-data`. |") +text = text.replace("| Ollama embedding | Interno | Servizio Compose interno obbligatorio per `qwen3-embedding:0.6b`. |", "| Ollama embedding | Interno | Servizio Compose interno obbligatorio dedicato a `qwen3-embedding:0.6b`. |") path.write_text(text) PY -for pattern in \ - 'Qdrant' \ - 'Ollama' \ - '(interni obbligatori|servizi interni obbligatori|interni al progetto Compose)' \ - 'DWH' \ - 'LLM' \ - '(esterni solo|solo esterni|restano esterni|unici servizi esterni|unici esterni)'; do - require_pattern "$compact_paraphrase" "compact manual paraphrase" "$pattern" >/dev/null -done +compact_spec='{"rows":[ + {"componente":"^DWH$","ownership":"^Esterno$","contratto operativo":"endpoint.*estern"}, + {"componente":"^LLM$","ownership":"^Esterno$","contratto operativo":"esterna|esterno"}, + {"componente":"^Qdrant$","ownership":"^Interno$","contratto operativo":"interno.*obbligatorio.*qdrant-data"}, + {"componente":"^Ollama embedding$","ownership":"^Interno$","contratto operativo":"interno.*obbligatorio.*qwen3-embedding:0\\.6b"} +]}' +verify_markdown_table_relationships "$compact_paraphrase" "compact manual paraphrase" "Contratto sintetico di ownership" "$compact_spec" >/dev/null compact_inversion="$negative_root/compact-inversion.md" cp "$root/docs/installazione-docker-4-contesti.md" "$compact_inversion" @@ -283,14 +308,11 @@ python3 - "$compact_inversion" <<'PY' import pathlib, sys path = pathlib.Path(sys.argv[1]) text = path.read_text() -text = text.replace( - "Qdrant e Ollama embedding sono servizi interni obbligatori del progetto Compose. Restano esterni solo DWH e LLM.", - "Qdrant, Ollama, DWH e LLM restano tutti servizi esterni.", -) +text = text.replace("| Qdrant | Interno | Servizio Compose interno obbligatorio con volume persistente `qdrant-data`. |", "| Qdrant | Esterno | Servizio esterno condiviso. |") path.write_text(text) PY set +e -require_pattern "$compact_inversion" "compact inversion" '(interni obbligatori|servizi interni obbligatori|interni al progetto Compose)' >"$workspace_output" 2>&1 +verify_markdown_table_relationships "$compact_inversion" "compact inversion" "Contratto sintetico di ownership" "$compact_spec" >"$workspace_output" 2>&1 workspace_status=$? set -e if [[ $workspace_status -eq 0 ]]; then @@ -299,6 +321,27 @@ if [[ $workspace_status -eq 0 ]]; then exit 1 fi +compact_scattered="$negative_root/compact-scattered.md" +cp "$root/docs/installazione-docker-4-contesti.md" "$compact_scattered" +python3 - "$compact_scattered" <<'PY' +import pathlib, sys +path = pathlib.Path(sys.argv[1]) +text = path.read_text() +start = text.index("## Contratto sintetico di ownership") +end = text.index("## Comando standard locale") +text = text[:start] + "Qdrant Interno DWH Esterno LLM Esterno Ollama embedding Interno.\n\n" + text[end:] +path.write_text(text) +PY +set +e +verify_markdown_table_relationships "$compact_scattered" "compact scattered tokens" "Contratto sintetico di ownership" "$compact_spec" >"$workspace_output" 2>&1 +workspace_status=$? +set -e +if [[ $workspace_status -eq 0 ]]; then + echo "compact scattered-token fixture was not rejected correctly" >&2 + cat "$workspace_output" >&2 + exit 1 +fi + adapted_reorder="$negative_root/caddy-adapted-reorder.json" adapted_bypass="$negative_root/caddy-adapted-bypass.json" node - "$adapted_reorder" "$adapted_bypass" <<'NODE' diff --git a/scripts/verify-workspace-install-docs.sh b/scripts/verify-workspace-install-docs.sh index 87f71a92..6fc3e80a 100755 --- a/scripts/verify-workspace-install-docs.sh +++ b/scripts/verify-workspace-install-docs.sh @@ -105,6 +105,56 @@ for token in tokens: PY } +verify_markdown_table_relationships() { + local source="$1" label="$2" heading="$3" spec_json="$4" + python3 - "$source" "$label" "$heading" "$spec_json" <<'PY' +import json, pathlib, re, sys +path = pathlib.Path(sys.argv[1]) +label = sys.argv[2] +heading = sys.argv[3] +spec = json.loads(sys.argv[4]) +text = path.read_text() +match = re.search(rf"^##+\s+{re.escape(heading)}\s*$", text, re.MULTILINE) +if not match: + raise SystemExit(f"{label}: missing structured section '{heading}'") +lines = text[match.end():].splitlines() +table = [] +for line in lines: + if not line.strip(): + if table: + break + continue + if not line.lstrip().startswith("|"): + if table: + break + continue + table.append(line.rstrip()) +if len(table) < 3: + raise SystemExit(f"{label}: structured table '{heading}' is incomplete") +headers = [cell.strip().lower() for cell in table[0].strip().strip("|").split("|")] +rows = [] +for raw in table[2:]: + cells = [cell.strip() for cell in raw.strip().strip("|").split("|")] + if len(cells) != len(headers): + raise SystemExit(f"{label}: malformed row in '{heading}'") + rows.append(dict(zip(headers, cells))) +for row_spec in spec["rows"]: + found = False + for row in rows: + ok = True + for column, pattern in row_spec.items(): + value = row.get(column.lower(), "") + if not re.search(pattern, value, re.IGNORECASE | re.DOTALL): + ok = False + break + if ok: + found = True + break + if not found: + raise SystemExit(f"{label}: missing relationship in '{heading}': {row_spec}") +PY +} + verify_compose_internal_semantic_contract() { python3 - "$root/compose.yaml" <<'PY' import sys, yaml, pathlib @@ -251,6 +301,29 @@ verify_internal_semantic_infrastructure_docs() { verify_vector_helper_interfaces || return 1 verify_project_state_current_contract "$root/PROJECT_STATE.md" "PROJECT_STATE.md" || return 1 + local ownership_spec semantic_index_spec compact_spec + ownership_spec='{"rows":[ + {"component":"^DWH$","ownership":"^External$","operator contract":"external|endpoint|installation"}, + {"component":"^LLM$","ownership":"^External$","operator contract":"external|endpoint|policy"}, + {"component":"^Qdrant$","ownership":"^Internal$","operator contract":"internal|Compose|qdrant-data"}, + {"component":"^Ollama embedding$","ownership":"^Internal$","operator contract":"internal|Compose|qwen3-embedding:0\\.6b"} + ]}' + semantic_index_spec='{"rows":[ + {"scope":"workspace semantic index","ownership rule":"(each|one|single).*(workspace).*(single|one).*(Qdrant).*(collection)|(each workspace reserves a single qdrant collection)","isolation rule":"schema.*evidence.*memory.*(one|that).*(collection).*(kind|payload)"} + ]}' + compact_spec='{"rows":[ + {"componente":"^DWH$","ownership":"^Esterno$","contratto operativo":"endpoint.*estern"}, + {"componente":"^LLM$","ownership":"^Esterno$","contratto operativo":"esterna|esterno"}, + {"componente":"^Qdrant$","ownership":"^Interno$","contratto operativo":"interno.*obbligatorio.*qdrant-data"}, + {"componente":"^Ollama embedding$","ownership":"^Interno$","contratto operativo":"interno.*obbligatorio.*qwen3-embedding:0\\.6b"} + ]}' + + verify_markdown_table_relationships "$local_manual" "local workspace manual" "Architecture ownership contract" "$ownership_spec" || return 1 + verify_markdown_table_relationships "$server_manual" "server workspace manual" "Architecture ownership contract" "$ownership_spec" || return 1 + verify_markdown_table_relationships "$local_manual" "local workspace manual" "Semantic index ownership contract" "$semantic_index_spec" || return 1 + verify_markdown_table_relationships "$server_manual" "server workspace manual" "Semantic index ownership contract" "$semantic_index_spec" || return 1 + verify_markdown_table_relationships "$compact_manual" "four-context install note" "Contratto sintetico di ownership" "$compact_spec" || return 1 + require_pattern "$readme" "README" 'mandatory stack.+qdrant.+embedding.+embedding-model-init' || return 1 require_pattern "$readme" "README" 'qwen3-embedding:0\.6b' || return 1 require_pattern "$readme" "README" 'qdrant-data.+embedding-models' || return 1 @@ -260,19 +333,10 @@ verify_internal_semantic_infrastructure_docs() { for manual in "$local_manual" "$server_manual"; do require_pattern "$manual" "$(basename "$manual")" 'qwen3-embedding:0\.6b' || return 1 require_pattern "$manual" "$(basename "$manual")" 'migration_required' || return 1 - require_concept_tokens "$manual" "$(basename "$manual")" \ - "workspace" "qdrant" "collection" || return 1 - require_pattern "$manual" "$(basename "$manual")" '(?is)(one|single|each).{0,120}(workspace|qdrant|collection).{0,120}(owns|reserve|reserved|single)' || return 1 done require_pattern "$local_manual" "local workspace manual" 'CPU-first' || return 1 require_pattern "$local_manual" "local workspace manual" 'THOTH_ENABLE_EMBEDDING_GPU=1' || return 1 require_pattern "$server_manual" "server workspace manual" 'Qdrant backup/restore' || return 1 - require_pattern "$server_manual" "server workspace manual" '(Git remote|DWH|LLM|bastion)' || return 1 - require_pattern "$server_manual" "server workspace manual" '(stay external|remain external|sono esterni)' || return 1 - require_concept_tokens "$compact_manual" "four-context install note" \ - "qdrant" "ollama" "dwh" "llm" || return 1 - require_pattern "$compact_manual" "four-context install note" '(?is)(interni obbligatori|servizi interni obbligatori|interni al progetto compose)' || return 1 - require_pattern "$compact_manual" "four-context install note" '(?is)(esterni solo|solo esterni|restano esterni|unici servizi esterni|unici esterni)' || return 1 require_pattern "$compact_manual" "four-context install note" '1024 dimensioni' || return 1 require_pattern "$diagnostics" "workspace diagnostic protocol" 'schema version 3' || return 1 require_pattern "$diagnostics" "workspace diagnostic protocol" 'semantic_index_incompatible' || return 1