diff --git a/docs/adr/0001-evidence-publication-boundary.md b/docs/adr/0001-evidence-publication-boundary.md deleted file mode 100644 index 2892f5d6..00000000 --- a/docs/adr/0001-evidence-publication-boundary.md +++ /dev/null @@ -1,7 +0,0 @@ -# Use workspace activation as the Evidence publication boundary - -Curated Evidence becomes Published Evidence only when it is valid, belongs to the -active workspace revision, and belongs to the atomically published Evidence generation. -Human Git review remains required before activation, but its approval is not duplicated -as mutable state in the Evidence manifest; this keeps Git and the workspace registry as -the existing sources of truth instead of introducing a second approval mechanism. diff --git a/docs/adr/0002-kind-independent-evidence-identifiers.md b/docs/adr/0002-kind-independent-evidence-identifiers.md deleted file mode 100644 index 4b2ea0ca..00000000 --- a/docs/adr/0002-kind-independent-evidence-identifiers.md +++ /dev/null @@ -1,8 +0,0 @@ -# Keep Evidence identifiers independent from kind - -An Evidence Unit keeps the same identifier when its kind is corrected or its Source -Evidence is unambiguously renamed; kind remains a separate validated field. This avoids -breaking citations and evaluation fixtures for a classification change, while genuine -semantic splits receive new identifiers so independent units never share an identity. -Identifiers use `evidence:`, are assigned once, persisted in the manifest and are -never recomputed automatically from mutable titles, paths or content hashes. diff --git a/docs/adr/0003-evaluate-evidence-before-activation.md b/docs/adr/0003-evaluate-evidence-before-activation.md deleted file mode 100644 index 73bc1741..00000000 --- a/docs/adr/0003-evaluate-evidence-before-activation.md +++ /dev/null @@ -1,7 +0,0 @@ -# Evaluate candidate Evidence before activation - -Evidence preprocessing builds a candidate vector generation and runs the workspace's -versioned evaluation set against that exact generation before atomically activating it. -The workspace revision may temporarily have no matching active Evidence during this -maintenance window, which is accepted as fail-closed degradation instead of introducing -a distributed transaction across Git, the workspace registry and Qdrant. diff --git a/docs/adr/0004-formula-evidence-is-an-expression.md b/docs/adr/0004-formula-evidence-is-an-expression.md deleted file mode 100644 index 22b2b946..00000000 --- a/docs/adr/0004-formula-evidence-is-an-expression.md +++ /dev/null @@ -1,6 +0,0 @@ -# Treat Formula Evidence as a composable SQL expression - -Formula Evidence contains one validated PostgreSQL expression with declared input -columns, not a complete query or statement. This makes formulas safely composable by -SQL generation; complete documented queries remain Example Evidence, and incompatible -legacy formulas require human review during migration. diff --git a/docs/adr/0005-evidence-contributes-to-semantic-stages.md b/docs/adr/0005-evidence-contributes-to-semantic-stages.md deleted file mode 100644 index 97420631..00000000 --- a/docs/adr/0005-evidence-contributes-to-semantic-stages.md +++ /dev/null @@ -1,13 +0,0 @@ -# Evidence contributes to semantic workflow stages - -The Evidence Module is a contributor to the existing semantic stages, not a visible -workflow stage. `clarification`, `rewriting`, `schema_linking`, `cte` and `final_sql` -query it with their corresponding purpose; `memory` remains owned by the Memory Module -and `synthesis` performs no Evidence search. Each mapped stage searches independently -and the workflow records only a minimal receipt containing stage, purpose, vector -generation and returned Evidence IDs. - -A successful search may return no matches and does not block the stage. Technical -unavailability is a distinct typed outcome that blocks the calling stage until retry, -without stale-generation or purpose fallback. This favors an explicit temporary stop -over silently treating a broken Evidence dependency as absence of domain knowledge. diff --git a/docs/adr/0006-grounded-atomic-evidence-preparation.md b/docs/adr/0006-grounded-atomic-evidence-preparation.md deleted file mode 100644 index e75c12a1..00000000 --- a/docs/adr/0006-grounded-atomic-evidence-preparation.md +++ /dev/null @@ -1,12 +0,0 @@ -# Ground and atomically apply Evidence preparation - -Every proposed Evidence Unit carries one to five short supporting excerpts that can be -found in its normalized Source Evidence. The model may reuse only identifiers supplied -for previous units; deterministic application code assigns all new canonical IDs. This -makes provenance and identity mechanically checkable without pretending that automated -validation can replace human semantic review. - -Preparation validates the complete changed batch in a temporary area and applies -curated files plus the manifest atomically. A model timeout or invalid response is not -retried automatically and leaves the worktree unchanged. Explicit `evidence resolve` -actions retire or relink units while preserving an ordinary, recoverable Git diff. diff --git a/docs/adr/0007-add-bm25-without-rebuilding-qdrant.md b/docs/adr/0007-add-bm25-without-rebuilding-qdrant.md deleted file mode 100644 index 86fe7742..00000000 --- a/docs/adr/0007-add-bm25-without-rebuilding-qdrant.md +++ /dev/null @@ -1,13 +0,0 @@ -# Add BM25 without rebuilding the shared Qdrant collection - -The workspace keeps its existing unnamed dense vector and adds only the sparse `bm25` -vector with IDF through Qdrant's additive vector-schema operation. Only Evidence points -are repopulated with both default dense and BM25 values. Schema, Memory and solved -questions retain their current dense points and are verified before and after the -upgrade. - -This replaces the planned destructive conversion to a named `dense` vector. If BM25 is -missing, Evidence preprocessing may add it and verify the resulting schema; session -runtime remains read-only. An incompatible existing BM25 definition fails without -mutation. A candidate failure leaves the additive schema in place while Evidence stays -unavailable, avoiding data loss in the other workflow modules. diff --git a/docs/adr/0008-make-hybrid-evidence-retrieval-deterministic.md b/docs/adr/0008-make-hybrid-evidence-retrieval-deterministic.md deleted file mode 100644 index 803ff57f..00000000 --- a/docs/adr/0008-make-hybrid-evidence-retrieval-deterministic.md +++ /dev/null @@ -1,24 +0,0 @@ -# Make hybrid Evidence retrieval deterministic and diagnosable - -Dense and BM25 retrieval receive the same deterministic query text. It preserves the -original question and appends nonempty concepts, tables and columns in a fixed order; -question and context receive Unicode NFC, newline canonicalization and outer trimming. -Context values are then deduplicated exactly and sorted, without lowercasing. Case, -punctuation and internal whitespace remain intact. This avoids accidental ranking -changes caused only by metadata ordering, preserves quoted PostgreSQL identifiers and -keeps the two retrieval branches directly comparable. - -Evidence fragmentation follows semantic headings, typed fields and paragraph -boundaries. Formulas, value/meaning pairs, mappings, rules and URLs remain atomic. An -atomic element larger than the existing `max_chunk_chars` limit creates the blocking -`atomic_content_too_large` Review item rather than being split mechanically. The limit -applies to the complete rendered text, defaults to 4,000 characters and is not -duplicated by an Evidence-specific setting. - -An L0 contract test starts the exact Qdrant image referenced by `compose.yaml` and -proves Italian server-side `qdrant/bm25` ingestion and search in a temporary collection. -There is no FastEmbed or dense fallback for Evidence when that capability is absent. - -The versioned evaluation set contains lexical, semantic and mixed queries. Its report -shows dense-only, BM25-only and fused ranks for expected Evidence. Publication remains -governed only by the simple fused top-10 rule; branch ranks and hit@5 are diagnostic. diff --git a/docs/agents/domain.md b/docs/agents/domain.md deleted file mode 100644 index 771882e1..00000000 --- a/docs/agents/domain.md +++ /dev/null @@ -1,76 +0,0 @@ -# Domain documentation - -This repository uses a single-context domain-documentation layout. - -## Sources - -Before changing behavior or terminology, read: - -1. `CONTEXT.md` at the repository root; -2. any relevant architectural decision records under `docs/adr/`; -3. the implementation and tests for the affected module. - -`CONTEXT.md` contains the shared domain vocabulary and the system's main -concepts. Use its terminology consistently in code, documentation, issues, and -user-facing explanations. - -ADRs explain important architectural decisions and their rationale. They are -created only when a durable decision needs to be recorded; the absence of -`docs/adr/` is not an error. - -If one of these optional sources does not exist, continue without reporting an -error. - -## Layout - -```text -/ -├── CONTEXT.md -└── docs/ - └── adr/ - └── .md -``` - -Do not introduce `CONTEXT-MAP.md` unless the repository later becomes a -genuine multi-context system whose domains require separate context documents. - -## Working with domain concepts - -When implementing or reviewing work: - -- identify the domain concepts involved; -- reuse the names defined in `CONTEXT.md`; -- distinguish domain rules from infrastructure details; -- avoid creating synonyms for established terms; -- update `CONTEXT.md` when a new durable concept is introduced or an existing - definition materially changes. - -For ThothII, the Evidence module and its concepts belong to this shared domain -context even though Evidence is implemented as an autonomous workflow module. - -## Architectural decisions - -Create an ADR when a decision: - -- affects multiple parts of the system; -- establishes a durable constraint; -- selects between meaningful alternatives; -- would otherwise be difficult to reconstruct later. - -Do not create an ADR for routine implementation details. - -If current code or a proposed change conflicts with an ADR, flag the conflict -explicitly. Do not silently override the recorded decision. - -## Keeping documentation aligned - -When a change affects the domain model: - -1. update the implementation; -2. update the relevant tests; -3. update `CONTEXT.md`; -4. add or update an ADR when the decision is architectural; -5. update linked plans and GitHub issues. - -The persisted repository documentation, not the chat transcript, is the -long-term source of truth. diff --git a/docs/agents/issue-tracker.md b/docs/agents/issue-tracker.md deleted file mode 100644 index 6dc19b7b..00000000 --- a/docs/agents/issue-tracker.md +++ /dev/null @@ -1,165 +0,0 @@ -# Issue tracker: GitHub - -Issues and specifications for this repository live in GitHub Issues under -`mptyl/ThothII`. - -Use the GitHub CLI (`gh`) for issue operations. Infer the repository from the -current Git remote when possible. - -## Conventions - -Create an issue: - -```bash -gh issue create --title "" --body-file <file> -``` - -Read an issue: - -```bash -gh issue view <number> -``` - -List issues: - -```bash -gh issue list -``` - -Add a comment: - -```bash -gh issue comment <number> --body-file <file> -``` - -Apply or remove labels: - -```bash -gh issue edit <number> --add-label "<label>" -gh issue edit <number> --remove-label "<label>" -``` - -Close an issue: - -```bash -gh issue close <number> -``` - -## Pull requests as a triage surface - -Pull requests are not used as the primary request or triage surface. - -A pull request may implement or resolve an issue, but the issue remains the -canonical location for: - -- the request; -- its scope and acceptance criteria; -- triage status; -- dependencies and sub-issues; -- implementation progress; -- the final resolution summary. - -## Publishing work - -When a workflow or skill says to publish a plan, specification, finding, or -request, create or update a GitHub issue. - -Do not leave the only authoritative copy in a chat transcript. - -Long implementation documents may also be committed to the repository. In that -case, the corresponding issue should link to the committed document and track -its execution status. - -## Fetching work - -When a workflow or skill refers to an issue number, retrieve the current issue -and its comments before acting: - -```bash -gh issue view <number> --comments -``` - -Treat the live issue state as authoritative for assignment, labels, closure, -and subsequent decisions. - -## Wayfinding operations - -A wayfinding map is represented by a parent GitHub issue and, when useful, -smaller child issues. - -### Map - -Create or update one parent issue describing: - -- the intended outcome; -- relevant context; -- known constraints; -- the proposed decomposition; -- dependencies between tasks; -- completion criteria. - -Label it according to `docs/agents/triage-labels.md`. - -### Child issues - -Create a separate issue for each independently actionable unit of work. - -Keep the parent issue readable: summarize the decomposition there and link the -child issues instead of copying every implementation detail. - -When GitHub sub-issues are available, register the relationship through the -GitHub API. Otherwise, maintain a checklist of linked child issues in the -parent issue. - -### Dependencies - -Represent blocking relationships with GitHub's native issue-dependency API -when available. - -First obtain the database ID of the blocking issue: - -```bash -gh api repos/mptyl/ThothII/issues/<blocking-number> --jq '.id' -``` - -Then register it as a blocker: - -```bash -gh api \ - --method POST \ - repos/mptyl/ThothII/issues/<blocked-number>/dependencies/blocked_by \ - -F issue_id=<blocking-issue-database-id> -``` - -If native dependencies are unavailable, record the relationship explicitly in -both issues. - -### Frontier - -The frontier is the set of open child issues that: - -- have no unresolved blockers; -- are sufficiently specified; -- can be worked on independently; -- are not already being worked on. - -Use labels and current issue relationships to identify the frontier. - -### Claim - -Before starting an issue: - -1. confirm that it is still open and unblocked; -2. assign it to the current operator when appropriate; -3. apply the label `ready-for-agent` only if it is genuinely executable; -4. add a short comment stating that work has started. - -### Resolve - -When the work is complete: - -1. verify the issue's acceptance criteria; -2. add a concise resolution comment with relevant files, tests, or decisions; -3. update the parent issue or dependent issues; -4. close the issue; -5. reconsider the frontier, because resolving a blocker may unlock more work. diff --git a/docs/agents/triage-labels.md b/docs/agents/triage-labels.md deleted file mode 100644 index 9510a8e4..00000000 --- a/docs/agents/triage-labels.md +++ /dev/null @@ -1,19 +0,0 @@ -# Triage labels - -These labels represent workflow roles rather than subject areas. - -| Label | Meaning | -| --- | --- | -| `needs-triage` | The request has not yet been classified or evaluated. | -| `needs-info` | More information or a human decision is required before work can proceed. | -| `ready-for-agent` | The work is sufficiently specified, unblocked, and suitable for an agent. | -| `ready-for-human` | The work requires human review, approval, or an action only a human can perform. | -| `wontfix` | The request has been deliberately declined or will not be implemented. | - -Use only the labels that describe the issue's current workflow state. - -Remove obsolete workflow labels when the state changes. For example, remove -`needs-info` when the missing information has been supplied. - -Subject-area labels may be added separately, but they must not replace these -workflow roles. diff --git a/docs/architecture/authentication.md b/docs/architecture/authentication.md index e61c8c7d..035d172e 100644 --- a/docs/architecture/authentication.md +++ b/docs/architecture/authentication.md @@ -84,8 +84,8 @@ Workspace Validate performs static authentication validation without provider co `tht auth check` performs live, non-interactive diagnosis: static safety plus OIDC discovery, issuer/JWKS, group-catalog authentication, and exact configured-group existence. Adding `--interactive` runs that same live diagnosis and then validates a device-flow identity when the -provider supports Device Authorization. Workspace Test is the aggregate live workspace and -authentication validation. +provider supports Device Authorization. Aggregate live workspace and authentication validation is +available through the installation diagnostics. The ordered `tht doctor` report is exactly: `descriptor`, `files`, `docker`, `compose`, `configuration`, `authentication`, `services`, `core-http`, `frontend-http`, diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index d089887c..3a7a0d5c 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -1,9 +1,7 @@ # Panoramica dell'architettura -> Sintesi ad uso documentazione. Per il dettaglio dei moduli e dei flussi vedi -> [Componenti, moduli e flussi](components.md); i contratti correnti sono in `docs/contracts/` -> e le decisioni durevoli in `docs/adr/`. Per lo stato corrente del progetto (gate manuali -> pendenti, layout workspace/secret) vedi `PROJECT_STATE.md` nella radice del repo. +> Per il dettaglio dei moduli e dei flussi vedi +> [Componenti, moduli e flussi](components.md). ThothII è un **datamart builder human-in-the-loop**: trasforma una domanda in linguaggio naturale in SQL validato (ed eventualmente un datamart dbt) attraverso un **workflow deterministico a 8 fasi NL→SQL**, in cui il modello *propone* e un revisore umano *decide* ai gate. @@ -80,8 +78,8 @@ rinominare o ricreare la collezione. `workspace preprocess evidence` e la parte - `tht -c`/`--config` è un'opzione **per-comando**: deve seguire il subcommand, mai precederlo (`ThtRunner.buildArgv` lo impone). - L'output `--json` deve essere JSON puro su stdout — è un contratto machine-readable. -- Le stringhe UI sono in inglese; il *contenuto* dei documenti resta nella lingua del workspace (italiano per `psd`), perché è il dato reale — solo chrome/label sono in inglese. -- I workspace (`harness/workspaces/*.yaml`) impostano il target DB e i path **assoluti** `paths.sessions/artifacts/indexes` — per `psd` puntano a un repo separato e non versionato (`tht-workspace-psd/`). I segreti vivono solo in `harness/.env` (gitignored). +- Le stringhe UI sono in inglese; il *contenuto* dei documenti resta nella lingua del workspace, perché è il dato reale — solo chrome e label sono in inglese. +- Ogni workspace imposta il target DWH e le proprie directory operative. I segreti restano nei file protetti dell'installazione e non nel repository del workspace. - Le impostazioni sono globali (`backend/data/settings.json`: workspace/provider/modello/thinking); il form di nuova sessione richiede solo la domanda. - **Resume**: una sessione riprendibile rientra all'ultima fase incompleta. Il backend rifiuta il resume con 409 se `finalized` o `archived`; `PiProcessManager.spawnFor` deve inviare `/riprendi-sessione <id>` (resume) vs `/nuova-domanda` (nuova) — il prompt sbagliato trasforma silenziosamente un resume in una nuova domanda. @@ -90,5 +88,3 @@ rinominare o ricreare la collezione. `workspace preprocess evidence` e la parte Lo stack locale si avvia con `./scripts/run-stack.sh`, dopo aver creato `deploy/env/local.env` da `deploy/env/local.env.example`. Il core Compose include Pi; DWH, vector DB, embedding e LLM sono endpoint esterni configurati nel file locale. - -Comandi per singolo layer, test e lint: vedi `AGENTS.md` nella radice del repo. diff --git a/docs/contracts/tht-pi.md b/docs/contracts/tht-pi.md deleted file mode 100644 index bc577cdd..00000000 --- a/docs/contracts/tht-pi.md +++ /dev/null @@ -1,245 +0,0 @@ -# `tht pi` lifecycle contract - -`tht` is the only component that drives Docker lifecycle operations. The `core` container -does not mount a Docker socket, and Pi is never updated in a running container. - -## Inspection and configuration - -```text -tht pi status -tht pi doctor -tht pi test -tht pi logs -tht pi configure -``` - -When `--installation` is omitted, `tht` first uses `THOTHII_INSTALLATION` and otherwise -discovers one valid `thothii-installation.yaml` in the current project tree, including an immediate -`deploy/*` directory. Use `--installation /absolute/path/thothii-installation.yaml` as an explicit -override when the descriptor is outside that tree or more than one installation is available. - -`status` executes the image-bundled `pi --version`. `doctor` compares that value with both the -container's `PI_VERSION` contract and the `io.thothii.pi.version` image label; a merely nonempty -version is not sufficient. `doctor` and `test` also require a healthy core, a successful Pi smoke, -valid settings, and an exact selected provider/model pair from the backend's available model -entries. `pi check` remains an alias for `pi test`. Logs are always a bounded, sanitized 200-line -snapshot; there is no follow mode. - -On a TTY, `pi configure` presents numbered provider, model, and thinking choices. Providers and -models come from the backend's closed model list, and the model choices are restricted to the -selected provider. In non-interactive use, all choices must be explicit: - -```text -tht pi configure \ - --provider zai --model glm-5.2 --thinking medium -``` - -The helper snapshots exact settings-file existence and raw bytes, applies the new values atomically, -and verifies the readback and rendered-configuration digest. A helper, readback, or digest failure -restores those exact bytes when the prior file existed; on a clean installation it removes the new -file and verifies the absent/default state. Empty prior files are supported. The command reports -the actual host path from `PI_AUTH_FILE`; credentials remain in that protected host file and must -never be passed as flags. - -Installation-managed Pi provider/model configuration is declarative only. Any JSON value beginning -with `!` is rejected recursively in the complete `models.json` before it can supply management -choices, and the exact selected provider/model and credential payload is checked again before the -isolated smoke files are written. The API returns only the fixed -`Pi provider/model configuration is invalid` message; rejected commands, paths, and secrets are -never included. Use `$NAME`/`${NAME}` environment references in `models.json`, or omit `apiKey` and -provide the selected credential through the protected `PI_AUTH_FILE`, `THT_MODEL_API_KEY_FILE`, or -`THT_SECRETS_FILE` contract. A literal leading exclamation mark uses Pi's `$!` escape. Direct -secret-file references are not a `models.json` feature: ThothII converts its managed key source to -the provider-native child environment, while `PI_AUTH_FILE` is mounted as Pi's protected credential -store. - -## Supported Compose entry points and current image - -Use `tht start`, `stop`, `status`, `logs`, and `doctor` for ordinary installation lifecycle -operations. All `tht` Compose commands automatically include the installation-specific -durable selector when it exists: - -```text -<projectDirectory>/.tht/<installation-id>/current-image.yaml -``` - -This selector is part of the supported installation state: it keeps a verified Pi image selected -across a fresh `tht` process, stop/start, reconcile, and source checkout whose base image is -digest-pinned. Do not delete or hand-edit it. Direct raw `docker compose` lifecycle commands bypass -this protection and are unsupported. Advanced documented Compose rendering must use -`scripts/compose-with-preflight.sh` and include the same selector with `-f` when present; connector -secret overrides must never bypass that preflight wrapper. - -## Reloading Pi configuration - -Configuration reload is a separate lifecycle operation from an image update: - -```text -tht pi restart --yes [--drain] -``` - -`--yes` is required after reviewing the planned core recreation. Restart activates the durable -maintenance gate before it checks sessions. Without `--drain`, active open sessions refuse the -command. With `--drain`, the command polls the authenticated session inventory until no active -sessions remain; it never terminates sessions and the wait is bounded. - -Restart retains the exact current image and never builds, pulls, or upgrades an image. Before any -core mutation, it tags the captured running image ID with a transaction-scoped reference and -selects that reference through a lifecycle-only Compose override. A configured mutable tag moving -after capture therefore cannot change the restarted image. It recreates only `core` with -`--no-deps --force-recreate --no-build --pull never`; `frontend` and named volumes are not -recreated. Before reopening admission, it verifies health, the unchanged Pi version, the -provider/model/settings smoke, unchanged non-secret rendered configuration, the captured image -identity, and the complete persistence-mount fingerprint. - -Restart and update keep separate recovery state: - -```text -<projectDirectory>/.tht/<installation-id>/restart-state.json -<projectDirectory>/.tht/<installation-id>/update-state.json -``` - -The files are mode `0600` and share one installation lifecycle lock, so restart, update, and -rollback cannot race. Every mutating lifecycle command checks both files. Malformed or non-terminal -restart recovery state blocks update and rollback; malformed or incomplete update recovery state -blocks restart. A verified terminal restart state is cleaned up safely before a later mutation. -After core mutation, a restart failure leaves admission gated and preserves both -`restart-state.json` and its exact-image override; the operator must use status/logs and maintenance -recovery rather than deleting recovery material. - -## Updating Pi - -The normal update uses the repository's pinned version and build source automatically: - -```text -tht pi update -tht pi update --version 0.81.0 -``` - -With no `--version`, the command reads the single default `ARG PI_VERSION=<version>` from -`docker/core.Dockerfile` in the selected project. The normal path confirms the explicit update -command, drains active sessions without terminating them, builds the candidate, recreates only -`core`, verifies it, and promotes it transactionally. - -Advanced registry updates remain available and require an immutable digest: - -```text -tht pi update \ - --version 0.81.0 --source build --yes --drain - -tht pi update \ - --version 0.81.0 --source pull \ - --image registry.example.invalid/thothii-core@sha256:<64-lowercase-hex-digits> --yes -``` - -`--source build` rebuilds only `core` with `PI_VERSION=<version>`. `--source pull` requires an -immutable digest reference; mutable tags, URL forms, and credential-bearing references are -rejected. `--source` is never inferred. - -Before inventory, update activates the durable maintenance gate. Activation writes -`/data/settings/maintenance.json` in the mounted settings volume, closes admission, and waits for -all leases. A recreated candidate reads that marker at startup and therefore starts gated. The -loopback-only control endpoints cannot be reached through the frontend proxy and do not depend on -the configured authentication principal mode. Lost activation/deactivation responses are resolved -by querying gate status only when the original result is unknown. An explicit file or directory -durability failure is never converted to success by matching readback: the control API reports -`maintenance_durability_failed`, keeps or restores the safest durable marker state, and requires -recovery. - -Open, unarchived sessions stop an update. After an operator has completed or otherwise drained -their work, `--drain` makes the command poll the authenticated bare-array -`GET /sessions?scope=all` response until no active sessions remain. - -The configured `core.image` is never retagged or mutated. Each installation transaction creates -unique candidate and previous tags, including when two installations share a configured tag or -the configured image is digest-pinned. A temporary lifecycle-only Compose override selects those -tags for build, recreate, and rollback. Candidate build, pull, or candidate-tag failures happen -before `mutation_started` and therefore never recreate or roll back core. After verification, the -temporary candidate selector is atomically promoted to the durable current-image override. -Rollback atomically promotes the previous selector. Terminal cleanup removes only transaction -files and never deletes the durable selector. - -Only `core` is recreated, with `--no-deps --force-recreate`; `frontend` is not recreated and no -volume-replacement flags are used. Verification checks health; exact requested Pi version at all -three declared boundaries (the candidate executable, `PI_VERSION` environment, and -`io.thothii.pi.version` image label); the provider/model/settings smoke; unchanged -non-secret rendered configuration; and the complete persistence-mount fingerprint. - -## Recovery, rollback, and maintenance cleanup - -Recovery state and lock diagnostics live under: - -```text -<projectDirectory>/.tht/<installation-id>/update-state.json -<projectDirectory>/.tht/<installation-id>/restart-state.json -<projectDirectory>/.tht/<installation-id>/*.lock.owner.json -``` - -Each recovery file is mode `0600`. Update state records transaction-scoped image identities, mount -fingerprints, target version/source, configuration digest, phase, and timestamp; restart state -records the retained image and its verification inputs. Neither file contains credentials, endpoint -values, secret paths, Compose output, or logs. A cross-platform OS advisory file lock serializes -both lifecycle operations; a crashed owner releases the lock automatically. Owner metadata is -diagnostic only and cannot wedge acquisition if empty, partial, or stale. - -Any post-candidate failure explicitly confirms or reactivates maintenance and rescans sessions -before compensation. Automatic rollback selects the transaction's previous image through the -lifecycle override and clears maintenance only after the previous image, configuration, mounts, -health, Pi smoke, and terminal recovery write are verified. Ambiguous compensation remains gated. -If the candidate core is stopped and cannot serve the maintenance endpoint, rollback proves that -state with Compose and writes the marker through a one-off previous-image `core` container sharing -the settings volume. It does not require the failed candidate, a host Node runtime, or the Docker -socket inside a container. The restored core is then recreated, verified, and rescanned before the -gate can open. - -For a failed update with `update-state.json`, first run: - -```text -tht pi rollback --yes -``` - -Rollback restores the image recorded in update state, but it checks restart state before making any -change. A failed, pending, or malformed restart state rejects rollback. A failed restart retains its -captured image and has no candidate image to roll back; first inspect status and logs, repair the -reported problem, then use maintenance recovery. - -Inspect and clean a stale durable gate with: - -```text -tht pi maintenance status -tht pi maintenance recover --yes -``` - -`maintenance recover` restores the captured restart image pin and lifecycle override when needed, -verifies and removes interrupted restart recovery material, and only then processes update state. -It completes an interrupted verified-image promotion, safely finalizes a preparation interrupted -before core mutation, and refuses other pending mutations. For terminal or absent recovery state, -it removes only a stale transaction override, verifies the running installation when the gate is -active, and only then removes the durable marker and reopens admission. It never removes -`current-image.yaml`. If rollback or recovery fails, leave the marker in place, preserve the -relevant recovery state and override, repair the reported Docker/configuration issue, and rerun -rollback or maintenance recovery. - -Missing confirmation, invalid arguments, active sessions, and an interrupted transaction exit -`2`. Docker and verification failures exit nonzero with concise, redacted guidance. Direct -read-only/log commands preserve the original Docker child exit code. - -## Go dependency security boundary - -The supported toolchain is Go `1.26.5`, released 2026-07-07, with module language version -`1.26.0`. The Docker builder is pinned by both patch tag and the multi-platform manifest-list -digest: - -```text -golang:1.26.5-bookworm@sha256:1ecb7edf62a0408027bd5729dfd6b1b8766e578e8df93995b225dfd0944eb651 -``` - -That manifest provides both `linux/amd64` and `linux/arm64/v8` builders. Go's official release -history is the authority for the patch level (`https://go.dev/doc/devel/release`); the Docker -Official Image is the authority for the builder (`https://hub.docker.com/_/golang`). -`golang.org/x/sys`, used by the Windows durable-replace implementation, is pinned to `v0.47.0`. -The directly used `github.com/sirupsen/logrus` is pinned to `v1.9.1`, which removes -GO-2025-4188 from the imported package set. The build contract verifies the exact toolchain, -dependencies, digest, and all five supported target builds (Windows amd64, Darwin amd64/arm64, and -Linux amd64/arm64). `go mod verify`, tests including the race detector, `go vet`, and -`govulncheck` are release gates. diff --git a/docs/contracts/workflow-observable-baseline.md b/docs/contracts/workflow-observable-baseline.md deleted file mode 100644 index d31de3f3..00000000 --- a/docs/contracts/workflow-observable-baseline.md +++ /dev/null @@ -1,155 +0,0 @@ -# Workflow observable baseline - -This contract freezes the externally observable behavior that the conservative modular -refactoring must preserve. It describes what callers and reviewers can observe; it does not -prescribe the internal location of the implementation. - -Changing an expectation in this baseline is a behavior change and requires an explicit product -decision. Moving code between Workflow core, Disambiguation, Memory, and Evidence must keep the -baseline green without weakening its assertions. - -```mermaid -stateDiagram-v2 - [*] --> F1 - state "F1 Clarification" as F1 - state "F2 Memory" as F2 - state "F3 Question rewrite" as F3 - state "F4 Evidence" as F4 - state "F5 Schema linking" as F5 - state "F6 SQL drafting" as F6 - state "F7 Validation" as F7 - state "F8 Promotion" as F8 - F1 --> F2 - F2 --> F3 - F3 --> F4 - F4 --> F5 - F5 --> F6 - F6 --> F7 - F7 --> F8 - F7 --> F6: correction - F8 --> [*] -``` - -## Automated seams - -### Pi gate - -From `harness/`, run `npm test` with the repository's supported Node 24 runtime. - -The gate suite fixes: - -- the complete registered Pi tool schemas, including nested types and enum-like constraints; -- the semantic workflow definition and the exact injected session skill bytes; -- widget descriptors and reviewer response semantics; -- F1 clarification and explicitly accepted open ambiguity; -- F2 Memory applied, deselected, and absent; -- F3 rewritten question and assumptions, including mutation failure ordering; -- F4 Evidence used, accepted, rejected, and legacy-without-corpus projections; -- F8 Memory promotion accepted, declined, and absent, including mutation failure ordering; -- a newly folded phase is announced once in RPC mode even when it requires no human gate; -- resume reconstruction for the touched F1, F2, F3, F4, and F8 states; -- artifact payload compatibility, anti-bypass behavior, and final phase closing. - -The baseline intentionally checks widget structure and domain content without freezing the -pre-existing Italian chrome emitted by the gate. Repository policy requires UI chrome and labels -to migrate to English in their owning workstream; this contract must not turn that mismatch into a -new compatibility requirement. - -`harness/.pi/skills/tht-sessione/SKILL.md` is a committed projection. Its authoritative -Disambiguation and Memory fragments live under `modules/`; from `harness/`, run -`python -m tht.pi_skill_projection --write` to regenerate it or `--check` to detect drift. -Composition uses a static ordered tuple and never directory discovery. - -### Harness CLI and persistence - -Run the default pytest suite from the harness package. The suite fixes: - -- pristine JSON output, human output separation, exit codes, and CLI error behavior; -- decision ledger folding, retraction, reopen ordering, and current-phase reconstruction; -- question, schema-linking, CTE, SQL, validation, and session-document projections; -- Evidence source, corpus, search, citation, and legacy-without-active-corpus behavior; -- Memory search, promotion, solved-question, and vector-write behavior; -- filesystem session persistence and PostgreSQL repository parity. - -The default pytest configuration excludes only tests marked `l2`. Tests marked `l0` require a -working local Docker daemon and remain part of the default suite when Docker is available. - -### Backend bridge - -From `backend/`, the passing automated baseline is: - -```sh -npx vitest run test/tht-runner.test.ts test/pi-process-manager.test.ts \ - test/session-bridge.test.ts test/sse-hub.test.ts test/sse-route.test.ts \ - test/routes-sessions.test.ts test/e2e-f1.test.ts \ - test/workspace-preprocessing-service.test.ts \ - test/workspaces/evidence/materialization.test.ts \ - test/workspaces/evidence/preprocessing.test.ts \ - test/workspaces/evidence/boundary.test.ts -npx tsc --noEmit -p . -npm run build -``` - -These suites fix: - -- CLI argument ordering and JSON/error propagation across the runner boundary; -- new-session versus resume Pi prompts; -- refusal to resume finalized, archived, foreign, unavailable, or read-only sessions; -- Pi RPC to client event mapping, SSE replay/reset behavior, and runtime replacement ordering; -- reserved Pi phase notifications mapped to sanitized `phase_started` client events; -- failure persistence and sanitization before a client-visible response. - -### Frontend client - -From `frontend/`, the passing automated baseline is: - -```sh -npx vitest run src/store/sessionStore.test.ts src/stream/useSessionStream.test.tsx \ - src/widgets/registry.test.tsx src/widgets/SelectWidget.test.tsx \ - src/widgets/MultiselectWidget.test.tsx src/widgets/ArtifactWidget.test.tsx \ - src/shell/f1-loop.test.tsx src/shell/SessionDocumentsPanel.test.tsx -npx tsc -b -npm run build -``` - -These suites fix: - -- widget registry and gate response payloads; -- `ui_request`, `text_delta`, activity, usage, and lifecycle event reduction; -- phase progress and the active workflow dot advancing on `phase_started` without a `ui_request`; -- stream replacement, cursor reset, reconnection, and pending-text flush behavior; -- session document projections shown to the reviewer. - -## Mutation ordering - -The following sequences are part of the observable failure contract: - -1. F3 writes the rewritten question, appends `question_rewritten` to the ledger, then advances. - A failure stops the remaining operations. -2. F8 saves one reusable Memory vector, appends its `memory_promoted` marker, advances F8, then - finalizes. A failed vector write leaves no marker; a failed marker after a successful vector - write returns the manual recovery instruction and does not finalize. -3. A declined F8 candidate writes only `memory_promotion_declined`; an absent candidate writes no - Memory decision and still closes F8. - -## Environment-dependent acceptance - -Real-model and remote-DWH tests remain opt-in through the `l2` marker. The live journey from a new -question to finalization, followed by resume verification, belongs to the final live-acceptance -ticket. If its environment or credentials are unavailable, it must remain recorded as a pending -manual gate rather than being reported as passed. - -## Full-suite diagnostic exceptions - -Every command defined above as part of the automated baseline exits successfully. Running the -broader backend and frontend suites is still useful as a diagnostic, but those full suites are not -the executable acceptance gate for this ticket because two unrelated failures reproduce unchanged -on the source commit from which this branch was created: - -- the backend authentication runtime-projection suite currently rejects ten positive fixtures - with its fail-closed public error; -- one frontend application-shell authentication test does not render the expected trusted-upstream - display name. - -These two exceptions must remain visible until their owning workstream resolves them; they must not -be used to relax any workflow assertion or to describe a nonzero command as a passing baseline. diff --git a/docs/contracts/workspace-evidence-v3.md b/docs/contracts/workspace-evidence-v3.md index 44fb26c5..da5115a6 100644 --- a/docs/contracts/workspace-evidence-v3.md +++ b/docs/contracts/workspace-evidence-v3.md @@ -218,10 +218,3 @@ tht config check -c <path> ``` Stop after validation. P2/P6 later owns preprocessing and materialization. - -## Acceptance states - -These gates are independent and are not implied by this documentation contract. - -automated integration: PENDING -manual acceptance: PENDING diff --git a/docs/disambiguazione-iniziale.md b/docs/disambiguazione-iniziale.md index 231910e1..fee57c84 100644 --- a/docs/disambiguazione-iniziale.md +++ b/docs/disambiguazione-iniziale.md @@ -54,7 +54,7 @@ Non deve: - costruire il SQL prima del chiarimento; - presentare più domande al reviewer nello stesso turno. -La motivazione è di controllo cognitivo e di audit: se vengono chiesti insieme popolazione, periodo, outcome e definizione clinica, non è possibile sapere quale risposta abbia determinato ciascuna scelta successiva. +La motivazione è di controllo cognitivo e di audit: se vengono chiesti insieme linea di prodotto, periodo, indicatore e definizione operativa, non è possibile sapere quale risposta abbia determinato ciascuna scelta successiva. ## Fonti usate per formulare le opzioni @@ -100,7 +100,7 @@ Esempi: - “ablazione” significa una procedura transcatetere oppure qualcos'altro; - “anno” significa anno solare oppure anno fiscale; -- “pazienti attivi” significa flag anagrafico oppure presenza di un evento. +- “biciclette attive” significa modelli a catalogo oppure unità presenti nella produzione corrente. Ogni opzione concreta contiene una decisione `concept_clarified`. La scelta del reviewer è già la conferma e viene persistita direttamente: non serve un secondo `reviewer_decide`. @@ -250,8 +250,4 @@ termine ambiguo ## Riferimenti -- [Skill canonica completa](skill-tht-sessione.md) -- [Workflow YAML](../harness/workflow.yaml) -- [Gate Pi](../harness/.pi/extensions/tht-gate.js) -- [Macchina delle fasi](../harness/tht/phase.py) - [Gestione delle memory](gestione-memory.md) diff --git a/docs/evidence.md b/docs/evidence.md index 842b9094..36c9d922 100644 --- a/docs/evidence.md +++ b/docs/evidence.md @@ -27,7 +27,6 @@ Per il filesystem Evidence v2, il primo sorgente autorevole deve stare nella dir ├── curated/ # Evidence Units revisionate │ └── <dominio>/<unit>.md ├── manifest.yaml # legami, hash e metadati della preparazione -├── evaluation/ # fixture di valutazione del recupero └── example/ # esempi e materiale di supporto ``` @@ -53,19 +52,19 @@ Le unità Markdown lette dal loader storico della CLI hanno frontmatter YAML. I ```markdown --- -id: evidence:fascia-pediatrica -title: Fascia pediatrica +id: evidence:autonomia-batteria +title: Autonomia nominale della batteria tier: structural status: reviewed sources: - - source/domain/patient.md + - source/domain/bicycle.md tables: - - patient + - bicycle_model concepts: - - concept:patient-age + - concept:battery-range --- -Definizione verificata della fascia pediatrica. +Definizione verificata dell'autonomia nominale per modello di bicicletta elettrica. La regola deve essere abbastanza atomica da poter essere citata senza ricostruire un intero capitolo. Il testo deve distinguere definizione, condizioni e limiti. @@ -90,13 +89,11 @@ flowchart TD ING --> VEC["Embedding e vector store"] BM25 --> GEN["Generazione candidata"] VEC --> GEN - GEN --> EVAL["tht evidence evaluate\nfixture di retrieval"] - EVAL -->|pass| ACT["Generazione attiva"] - EVAL -->|fail| FIX + GEN --> ACT["Generazione attiva"] ACT --> RUNTIME["Ricerca Evidence nel workflow"] ``` -La preparazione può ristrutturare sorgenti cambiate, ma non pubblica da sola. `prepare` produce una proposta e può indicare il file sorgente coinvolto in caso di errore. `validate` non scrive né pubblica. Il commit è un'azione del curatore nel clone di authoring. Il runtime legge una revisione completa e validata, poi la pipeline crea una generazione versionata. L'attivazione è atomica: una generazione precedente resta disponibile secondo la policy di retention. +La preparazione può ristrutturare sorgenti cambiate, ma non pubblica da sola. `prepare` produce una proposta e può indicare il documento coinvolto in caso di errore. `validate` non scrive né pubblica. La pubblicazione della revisione è un'azione del curatore. Il runtime legge una revisione completa e validata, poi la pipeline crea una generazione versionata. L'attivazione è atomica: una generazione precedente resta disponibile secondo la policy di retention. La ricerca runtime usa il recupero ibrido. Il ramo denso usa gli embedding, il ramo BM25 usa la ricerca lessicale e la fusione deterministica ordina i risultati. L'unità pubblicata conserva la provenienza, che il modello deve citare quando usa l'evidence. @@ -183,7 +180,3 @@ Le formule hanno un formato distinto dalle Evidence documentali. Una formula pro - [Contratto Workspace Evidence v3](contracts/workspace-evidence-v3.md) - [Contratto della CLI di preprocessing](contracts/workspace-preprocessing-cli.md) -- [ADR: confine di pubblicazione](adr/0001-evidence-publication-boundary.md) -- [ADR: preparazione atomica e ancorata al sorgente](adr/0006-grounded-atomic-evidence-preparation.md) -- [ADR: valutazione prima dell'attivazione](adr/0003-evaluate-evidence-before-activation.md) -- [ADR: recupero ibrido deterministico](adr/0008-make-hybrid-evidence-retrieval-deterministic.md) diff --git a/docs/general/pi-configuration.md b/docs/general/pi-configuration.md index 5cf8fe25..8fa174e2 100644 --- a/docs/general/pi-configuration.md +++ b/docs/general/pi-configuration.md @@ -1,6 +1,10 @@ -# Configurazione dei modelli in Pi: built-in, utente, progetto +# Configurazione locale dei modelli Pi -Pi (il coding agent che orchestra il workflow NL→SQL) può risolvere un `provider/model` in tre modi diversi. Non sono alternativi: coesistono, e la scelta di quale usare dipende da **quanto è standard l'endpoint** e da **quanto deve essere ampia la visibilità** del modello (tutti i progetti vs. un progetto solo). +Pi, l'agente che orchestra il workflow NL→SQL, risolve i modelli integrati e i provider +OpenAI-compatible dichiarati nel catalogo locale. ThothII applica una regola più stretta di Pi: +un provider o un modello custom non può essere registrato da codice in `harness/.pi/extensions/`. +Endpoint, protocollo, compatibilità e identificatori dei modelli appartengono esclusivamente ai file +locali `deploy/pi/models.json` e `deploy/pi/settings.json`. > **ThothII operator note:** ThothII runs Pi only in Docker Compose. Paths under > `~/.pi/agent/` in this document describe Pi's container-side behavior. Operators edit @@ -17,8 +21,10 @@ In produzione configurare una sola sorgente generica: il bundle `THT_SECRETS_FIL selezionato e passa al solo child Pi la variabile nativa appropriata (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GEMINI_API_KEY`, `ZAI_API_KEY`, ecc.). Il percorso generico, le chiavi di provider non selezionati e il vecchio `PI_PROVIDER_API_KEY` vengono rimossi dall'ambiente -del child. Provider locali come `ollama`, `lmstudio` e `aritmolab` continuano senza chiave; un provider -hosted non mappato o un secret mancante/non sicuro fallisce prima dello spawn con errore sanitizzato. +del child. Per un provider custom, il backend deriva il nome della variabile dal campo dichiarativo +`apiKey` di `models.json`; un valore letterale indica che il catalogo è autosufficiente. Non esistono +eccezioni per nomi di provider compilate nel codice. Un secret mancante o non sicuro fallisce prima +dello spawn con errore sanitizzato. La sorgente generica supporta soltanto provider con una singola chiave: `ant-ling`, `anthropic`, `cerebras`, `deepseek`, `fireworks`, `github-copilot`, `google` (anche tramite alias `gemini`), @@ -33,7 +39,7 @@ dello spawn (anche durante l'elenco modelli); tutte le credenziali ambientali AW Cloudflare restano comunque rimosse. Servirà una futura configurazione dedicata per provider per supportare questi bundle senza ambiguità. -## I tre livelli di provenienza di un modello +## Le fonti di un modello ### 1. Built-in (compilato dentro Pi) @@ -48,8 +54,8 @@ Per un endpoint **OpenAI-compatible** che non è tra i built-in — ma che non r Nelle installazioni gestite da ThothII il file deve essere interamente dichiarativo. ThothII rifiuta ricorsivamente qualsiasi valore JSON che inizi con `!`, anche dentro `headers`, `models`, `modelOverrides`, `compat`, array o campi non ancora conosciuti. Pi 0.80.3 tratterebbe quel prefisso -come un comando shell al momento della richiesta; questa forma non è ammessa né dall'elenco gestito -dei modelli né dallo smoke isolato. L'errore restituito è fisso e non include comando, percorso o +come un comando shell al momento della richiesta; questa forma non è ammessa dall'elenco gestito +dei modelli. L'errore restituito è fisso e non include comando, percorso o secret. Per i secret usare un riferimento ambiente come `"$ZAI_API_KEY"` o `"${ZAI_API_KEY}"`. Il backend @@ -86,25 +92,27 @@ Esempio reale in uso su questa macchina — GLM (provider `zai`): Essendo a livello utente, GLM è visibile da **qualsiasi progetto**. -### 3. Estensione (`.pi/extensions/*.js`, utente o progetto) +### Provider da estensione: non ammessi in ThothII -Quando l'endpoint richiede codice — ad esempio conversione di eventi (thinking→text), un `streamSimple` custom, o comunque logica che un file dichiarativo non può esprimere — serve un'estensione Pi che chiama `pi.registerProvider(...)`. Le estensioni possono vivere sia in `~/.pi/agent/extensions/` (tutti i progetti) sia in `<progetto>/.pi/extensions/` (solo quel progetto, se Pi viene lanciato con quella cwd). - -Esempio reale — il provider AritmoLab (Qwen), interno all'ospedale, usato solo da ThothII: [harness/.pi/extensions/aritmolab-provider.js](../../harness/.pi/extensions/aritmolab-provider.js). +Pi supporta tecnicamente provider registrati da estensioni JavaScript, ma ThothII non usa questa +possibilità. Le estensioni di progetto sono riservate al workflow e ai gate; non devono contenere +`registerProvider(...)`. Un endpoint che non può essere descritto dal catalogo OpenAI-compatible +non è un provider supportato da questa installazione finché il contratto dichiarativo non viene +esteso in modo generico. ## Tabella riassuntiva (stato attuale di questa macchina) | Modello | Livello | Perché | Visibilità | |---|---|---|---| | `deepseek/deepseek-v4-pro` | Built-in Pi | API pubblica nota, già nella build | Tutti i progetti | -| `zai/glm-5.2` | `~/.pi/agent/models.json` | Endpoint OpenAI-compatible custom (z.ai), nessuna logica speciale | Tutti i progetti | -| `aritmolab/qwen3.6-35b-a3b` | Estensione di progetto | Endpoint interno ospedaliero + conversione eventi thinking→text custom | Solo ThothII (cwd=`harness/`) | +| `zai/glm-5.3` | `deploy/pi/models.json` | Endpoint OpenAI-compatible custom | Installazione ThothII | +| `local-qwen/qwen3.6-35b-a3b` | `deploy/pi/models.json` | Endpoint OpenAI-compatible configurato localmente | Installazione ThothII | ## Come scegliere il livello giusto per un nuovo modello -1. **L'endpoint è un'API pubblica già nota a Pi?** → niente da fare, verifica con `pi --list-models`. -2. **È OpenAI-compatible, nessuna logica custom, e deve essere visibile ovunque?** → `~/.pi/agent/models.json`. -3. **Serve codice custom (auth non standard, conversione eventi, trasporto non-OpenAI) oppure deve restare visibile a un solo progetto?** → estensione, in `~/.pi/agent/extensions/` (globale) o `<progetto>/.pi/extensions/` (locale). +1. **L'endpoint è un'API pubblica già nota a Pi?** → abilita l'identificatore esatto in `deploy/pi/settings.json`. +2. **È OpenAI-compatible ma non built-in?** → dichiaralo in `deploy/pi/models.json`, poi abilitalo in `deploy/pi/settings.json`. +3. **Richiede codice di trasporto specifico del provider?** → non aggiungere un'estensione specifica; il provider non è supportato finché manca una capacità dichiarativa generica. --- @@ -126,7 +134,6 @@ Oltre ai modelli, Pi carica altre risorse da due alberi paralleli: `~/.pi/agent/ ``` harness/.pi/ ├── extensions/ -│ ├── aritmolab-provider.js ← provider LLM solo-progetto │ ├── tht-gate.js ← gate human-in-the-loop │ └── gate/ │ ├── core/ ← enforcement e utility condivise del gate @@ -139,19 +146,17 @@ harness/.pi/ ### Comportamento rispetto alla cwd -La directory da cui lanci `pi` determina quale `.pi/` di progetto viene trovata: +La directory da cui lanci `pi` determina quali estensioni del workflow vengono trovate, ma non +quali modelli ThothII rende disponibili: il catalogo è montato nel Pi agent directory del container. ```bash -# Da harness/ — trova harness/.pi/extensions/aritmolab-provider.js +# Da harness/ — carica il gate di progetto e il catalogo locale montato cd /path/to/ThothII/harness -pi --model aritmolab/qwen3.6-35b-a3b "..." - -# Dalla radice di ThothII — nessun .pi/ trovato lì o nei genitori, solo config utente -cd /path/to/ThothII -pi --model aritmolab/qwen3.6-35b-a3b "..." # ❌ Error: model not found +pi --model local-qwen/qwen3.6-35b-a3b "..." ``` -Il backend di ThothII (`PiProcessManager`, `list-models.ts`, `model-matrix.mjs`) lancia sempre `pi` con `cwd: harnessDir`, per questo Qwen è visibile in produzione. +Il backend di ThothII lancia sempre Pi con `cwd: harnessDir` per il workflow; la disponibilità del +modello continua a dipendere soltanto da `models.json`, `settings.json` e dalle credenziali locali. --- @@ -177,15 +182,15 @@ Se serve un modulo condiviso importato da un'estensione, usa `.mjs` proprio per ```bash pi --list-models ``` -Mostra i built-in + i modelli utente da `models.json`. **Non mostra** i provider registrati da estensione (come AritmoLab) — quelli vanno verificati con la cwd giusta. +Mostra i built-in e i modelli dichiarati in `models.json`. -### Verifica che un'estensione sia caricata +### Verifica il catalogo usato dall'applicazione ```bash cd harness # o la cwd rilevante per il progetto pi --mode rpc # poi: {"type": "get_available_models", "id": "1"} ``` -La risposta RPC include tutti i modelli disponibili, inclusi quelli da estensione. +La risposta RPC deve includere soltanto modelli built-in o dichiarati nel catalogo locale. --- @@ -193,6 +198,5 @@ La risposta RPC include tutti i modelli disponibili, inclusi quelli da estension | Problema | Causa | Soluzione | |---|---|---| -| `Model "X/Y" not found` lanciando da fuori progetto | Il modello è registrato da un'estensione locale, non visibile fuori dalla cwd giusta | Lancia `pi` dalla directory di progetto corretta (es. `harness/`) | -| Estensione non caricata pur essendo nella cartella giusta | File `.mjs` invece di `.js`/`.ts` | Rinomina in `.js` | +| `Model "X/Y" not found` | Provider/modello assente da `models.json` oppure identificatore assente da `enabledModels` | Correggi i due file locali e ricarica Pi | | Impostazioni di progetto non applicate | `settings.json` di progetto ha errori di sintassi, o si sta lanciando `pi` dalla cwd sbagliata | Valida il JSON, controlla la cwd | diff --git a/docs/gestione-memory.md b/docs/gestione-memory.md index 5760c414..00363dbd 100644 --- a/docs/gestione-memory.md +++ b/docs/gestione-memory.md @@ -35,11 +35,6 @@ F8: il reviewer decide se promuoverla L'invariante principale è `REUSABLE_TYPES = {"concept_clarified"}`: le sole memory generabili, salvabili, ricercabili e proponibili sono i concetti chiariti. Le decisioni `table_promoted`, `table_excluded`, `column_promoted` e analoghe restano decisioni locali alla domanda. -Implementazione principale: la façade [harness/tht/memory/](../harness/tht/memory/), -con le policy riusabili in -[harness/tht/memory/core.py](../harness/tht/memory/core.py), e l'adapter -[harness/tht/cli/memory_cmd.py](../harness/tht/cli/memory_cmd.py). - ## I tre livelli della gestione | Livello | Contenuto | Funzione | @@ -54,8 +49,8 @@ Il ledger contiene la provenienza e le decisioni umane. Il record globale contie Durante F1 il workflow registra i chiarimenti come decisioni `concept_clarified`. Un chiarimento può esprimere: -- definizioni di concetti clinici o organizzativi; -- criteri di inclusione ed esclusione di una popolazione; +- definizioni di concetti produttivi o organizzativi; +- criteri di inclusione ed esclusione di una linea di prodotto; - formule e metodi di calcolo; - interpretazioni temporali; - mapping verso tabelle e colonne specifiche; @@ -167,8 +162,6 @@ Il comando: 6. esclude le memory già decise nella sessione corrente; 7. restituisce i risultati ordinati per similarità. -L'implementazione è in [harness/tht/cli/memory_cmd.py](../harness/tht/cli/memory_cmd.py:368). - Le memory non vengono mai applicate automaticamente. Il modello deve presentarle in un'unica scelta `reviewer_decide`: - una memory selezionata viene registrata come nuovo `concept_clarified` nella sessione corrente; diff --git a/docs/guida-utente.md b/docs/guida-utente.md index fe999e84..9b2ffabf 100644 --- a/docs/guida-utente.md +++ b/docs/guida-utente.md @@ -36,22 +36,22 @@ thoth-workspaces.yaml ← catalogo: elenco dei workspace ```yaml schema_version: 1 workspaces: - - id: psd-clinical - name: Policlinico San Donato - description: DWH clinico del Policlinico San Donato + - id: acme-ebikes + name: ACME Limited + description: DWH della produzione di biciclette elettriche ``` -- L'**id** deve essere minuscolo, senza spazi, es. `psd-clinical` (`[a-z][a-z0-9-]{2,62}`). +- L'**id** deve essere minuscolo, senza spazi, es. `acme-ebikes` (`[a-z][a-z0-9-]{2,62}`). - Il **descrittore** `<id>/workspace.yaml` è lo schema v3. È l'unica descrizione valida. -### 1.2 Esempio di descrittore (Policlinico San Donato) +### 1.2 Esempio di descrittore (ACME Limited) ```yaml workspace: schema_version: 3 - id: psd-clinical - name: Policlinico San Donato - description: DWH clinico — aritmologia + id: acme-ebikes + name: ACME Limited + description: DWH industriale — produzione di biciclette elettriche language: it # le descrizioni/evidence sono in italiano dwh: @@ -63,7 +63,7 @@ dwh: semantic_index: vector_store: engine: qdrant - collection: psd-clinical + collection: acme-ebikes dimensions: 1024 distance: cosine embedding: @@ -84,7 +84,7 @@ diagnostics: evidence: source: type: filesystem - uri: psd-clinical/evidence # percorso dentro il repository + uri: acme-ebikes/evidence # percorso dentro il repository policy: max_chunk_chars: 4000 retain_published_generations: 3 @@ -186,10 +186,6 @@ selezione del workspace. configurato/mancante. - **Forget stored value** elimina dal vault cifrato il singolo secret indicato. Le sessioni o operazioni future che lo richiedono restano bloccate finché non viene inserito di nuovo. -- **Test workspace connections** materializza temporaneamente i secret necessari, contatta i servizi dati - configurati per quel workspace e rimuove i file temporanei alla fine. Non esporta né pubblica - nulla. - Il repository Git remoto e le relative credenziali sono impostazioni di installazione. I secret runtime DWH/Evidence sono invece persistenti nel vault cifrato del backend e non nel local storage della GUI. La GUI è soltanto l'interfaccia: dopo l'invio cancella i valori dai campi e non può @@ -206,8 +202,8 @@ naturale (workspace, modello e provider sono impostazioni globali già configura Esempio di domanda: -> «Estrai i pazienti che hanno eseguito un'ablazione nell'ultimo anno, con nome, cognome e data -> dell'intervento.» +> «Elenca le biciclette elettriche completate nell'ultimo anno, con modello, numero di telaio e +> data di completamento.» ### 3.2 Il workflow a 8 fasi e i gate @@ -235,41 +231,41 @@ verità: ciò che non è registrato non è avvenuto. --- -## Esempio pratico completo — Policlinico San Donato +## Esempio pratico completo — ACME Limited ### Passo 0 — repository -Crea il repository Git del workspace (es. `tht-workspace-psd`): +Crea il repository Git del workspace (es. `tht-workspace-acme`): ```text -thoth-workspaces.yaml # catalogo con psd-clinical -psd-clinical/workspace.yaml # descrittore v3 (vedi §1.2) -psd-clinical/evidence/ # i documenti .md di contesto curati -psd-clinical/schema/annotations.yaml # (quando ci sono join curate) +thoth-workspaces.yaml # catalogo con acme-ebikes +acme-ebikes/workspace.yaml # descrittore v3 (vedi §1.2) +acme-ebikes/evidence/ # i documenti .md di contesto curati +acme-ebikes/schema/annotations.yaml # (quando ci sono join curate) ``` -Fai `commit` e `push`. Nell'installazione, l'applicazione fa `Pull` e **attiva** il workspace: -valida lo schema v3, materializza l'Evidence dal commit fissato e prepara la collection Qdrant +Pubblica una nuova revisione Git. Nell'installazione, l'applicazione acquisisce e **attiva** il workspace: +valida lo schema v3, materializza l'Evidence dalla revisione fissata e prepara la collection Qdrant (1024/cosine + indici). ### Passo 1 — preprocessing ```bash -tht --installation ~/thothii-installation.yaml workspace preprocess dwh --workspace psd-clinical --json -tht --installation ~/thothii-installation.yaml workspace preprocess run --workspace psd-clinical --json +tht --installation ~/thothii-installation.yaml workspace preprocess dwh --workspace acme-ebikes --json +tht --installation ~/thothii-installation.yaml workspace preprocess run --workspace acme-ebikes --json ``` Se il run si ferma per le join (`manual_review_required`): ```bash -# il curatore rivede i candidati e pubblica psd-clinical/schema/annotations.yaml, poi: -tht --installation ~/thothii-installation.yaml workspace schema accept --workspace psd-clinical --run <run-id> --yes --json -tht --installation ~/thothii-installation.yaml workspace preprocess run --workspace psd-clinical --resume <run-id> --json +# il curatore rivede i candidati e pubblica acme-ebikes/schema/annotations.yaml, poi: +tht --installation ~/thothii-installation.yaml workspace schema accept --workspace acme-ebikes --run RUN_ID --yes --json +tht --installation ~/thothii-installation.yaml workspace preprocess run --workspace acme-ebikes --resume RUN_ID --json ``` ### Passo 2 — la domanda -Nell'applicazione seleziona il workspace `psd-clinical` e crea una sessione con la domanda. Segui +Nell'applicazione seleziona il workspace `acme-ebikes` e crea una sessione con la domanda. Segui le fasi e conferma ai gate: il modello proporrà lo schema-linking (tabelle/colonne del DWH `datawarehouse`), i CTE e infine l'SQL finale, che potrai copiare/visualizzare ed eseguire. @@ -277,11 +273,8 @@ le fasi e conferma ai gate: il modello proporrà lo schema-linking (tabelle/colo ## Dove trovare i dettagli tecnici -Per l'accesso DWH REST, la chiave è per installazione e vale solo per `rest_api`: il server PSD rimane `postgres_direct` e `ssh_tunnel` non usa questa chiave. Vedere [guida server DWH](install/dwh-auth-server.md), [enrollment client](install/dwh-auth-client-enrollment.md), [TLS](install/dwh-auth-tls.md) e [runbook PSD](operations/psd-dwh-auth-rollout.md). +Per l'accesso DWH REST, la chiave è per installazione e vale solo per `rest_api`; `postgres_direct` e `ssh_tunnel` non usano questa chiave. Vedere [guida server DWH](install/dwh-auth-server.md), [enrollment client](install/dwh-auth-client-enrollment.md) e [TLS](install/dwh-auth-tls.md). - Contratto CLI: `docs/contracts/workspace-preprocessing-cli.md` - Contratto `.tht-dwh`: `docs/contracts/tht-dwh.md` - Evidence v3: `docs/contracts/workspace-evidence-v3.md` -- Installazione locale: `docs/install/local-workspace-registry.md` -- Installazione server: `docs/install/server-workspace-registry.md` -- Verifica manuale P2–P6: `docs/testing/p2-p6-manual-verification.md` diff --git a/docs/index.md b/docs/index.md index fb513846..1cf2eb65 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,21 +1,21 @@ # ThothII — Documentazione -Benvenuto nella documentazione di ThothII, il datamart builder human-in-the-loop che trasforma domande in linguaggio naturale in SQL validato attraverso un workflow a 8 fasi orchestrato dal coding agent Pi. +Benvenuto nella documentazione di ThothII, il datamart builder human-in-the-loop che trasforma domande in linguaggio naturale in SQL validato attraverso un workflow a 8 fasi orchestrato da Pi. La documentazione è divisa in due aree: ## ThothII (Documentazione Tecnica) -Come funziona il sistema: architettura, specifiche di design delle singole funzionalità, piani di implementazione, report di test. Parte da qui: [Panoramica dell'architettura](architecture/overview.md). +Come funziona il sistema: architettura, workflow, contratti operativi, Evidence e gestione delle Memory. Parte da qui: [Panoramica dell'architettura](architecture/overview.md). -Per autenticazione locale, OIDC generico, Authentik e accettazione PSD: [documentazione autenticazione](architecture/authentication.md). +Per autenticazione locale, OIDC generico e Authentik: [documentazione autenticazione](architecture/authentication.md). Per installare l'applicazione in Docker nei quattro contesti operativi, usando il file env, `compose.yaml`, l'overlay locale/server e il bundle di secret montato: [Installazione Docker nei quattro contesti](installazione-docker-4-contesti.md). -Per il DWH REST con una chiave revocabile per installazione: [guida server](install/dwh-auth-server.md), [enrollment client](install/dwh-auth-client-enrollment.md), [TLS](install/dwh-auth-tls.md) e [runbook PSD](operations/psd-dwh-auth-rollout.md). Il componente resta separato dallo stack Compose ThothII. +Per il DWH REST con una chiave revocabile per installazione: [guida server](install/dwh-auth-server.md), [enrollment client](install/dwh-auth-client-enrollment.md) e [TLS](install/dwh-auth-tls.md). Il componente resta separato dallo stack Compose ThothII. ## Considerazioni Generali -Note operative e di configurazione che non sono specifiche del dominio ThothII ma riguardano l'ambiente di sviluppo condiviso con altri progetti — ad esempio come Pi (il coding agent) risolve i modelli a livello built-in, utente e progetto. Parte da qui: [Configurazione dei modelli in Pi](general/pi-configuration.md). +Note operative e di configurazione che non sono specifiche del dominio ThothII — ad esempio come Pi risolve i modelli a livello integrato, utente e progetto. Parte da qui: [Configurazione dei modelli in Pi](general/pi-configuration.md). diff --git a/docs/install/authentication-local.md b/docs/install/authentication-local.md index c59c6290..22e441b2 100644 --- a/docs/install/authentication-local.md +++ b/docs/install/authentication-local.md @@ -75,7 +75,6 @@ This section applies only when a Linux `profile: server` descriptor declares a r The canonical authentication root stays root-owned and is the only authority. The container reads only the separate read-only runtime projection selected by `CURRENT`; it never falls back to the canonical files or to a previous generation. Run projected mutations and repairs through the -root-operated `tht` commands documented in the [server guide](server.md), and never edit runtime -files directly. +root-operated `tht` commands, and never edit runtime files directly. Mac, Windows, and local direct-file authentication remain unchanged when the projection is absent. diff --git a/docs/install/authentication-oidc.md b/docs/install/authentication-oidc.md index 5f364336..a970b675 100644 --- a/docs/install/authentication-oidc.md +++ b/docs/install/authentication-oidc.md @@ -71,7 +71,7 @@ The surfaces have distinct semantics and this order is recommended: issuer/JWKS, catalog credentials, and all configured mapped groups. 3. `tht auth check --interactive` repeats live diagnosis and additionally validates a device-flow identity and its direct `groups` claim when Device Authorization is available. -4. Workspace Test performs aggregate live workspace and authentication validation. +4. Installation diagnostics perform aggregate live workspace and authentication validation. The live CLI forms are: @@ -87,8 +87,8 @@ real ID token including `groups`. It is an operator check, not a replacement for `tht doctor` emits this exact ordered report: `descriptor`, `files`, `docker`, `compose`, `configuration`, `authentication`, `services`, `core-http`, `frontend-http`, `workspace-registry`, `workflow`, `pi`. Its authentication entry is live and non-interactive. -Any authentication failure makes Workspace Validate or Workspace Test non-activatable according -to that surface's static or live scope. +Any authentication failure prevents activation according to the static or live scope of the +relevant diagnostic surface. The complete closed diagnostic-code union and exact role-to-permission expansion are in the [authentication architecture](../architecture/authentication.md). diff --git a/docs/install/authentik.md b/docs/install/authentik.md index 196e467d..c6194dff 100644 --- a/docs/install/authentik.md +++ b/docs/install/authentik.md @@ -1,37 +1,42 @@ -# Authentik provider setup +# Configurazione del provider Authentik -Authentik is the first certified provider for PSD acceptance. The ThothII browser protocol remains -generic OIDC; these steps configure the provider-specific group catalog only. +Il protocollo browser di ThothII è OIDC generico. Authentik fornisce il catalogo gruppi e il +provider di identità senza introdurre un percorso di login proprietario. -1. Create an OAuth2/OIDC application and provider in Authentik. Register exactly - `<publicUrl>/api/auth/oidc/callback` as the callback and enable `openid`, `profile`, and `email`. -2. Configure the provider so the ID token contains a direct `groups` array of strings. Verify the - claim with a disposable test identity before running acceptance. -3. Create a dedicated API service account for the group catalog. Grant group-view-only privilege; - do not grant write, user-management, or directory-administration privilege. Put its bearer value - in the protected bundle under `THT_AUTHENTIK_API_TOKEN`. -4. Create or confirm the exact groups `TOT Users` and `TOT Admin`. Map them explicitly in - `auth.yaml` to `user` and `admin`, respectively. Keep other upstream groups out of the mapping. -5. Run Workspace Validate for static authentication validation. Then run live non-interactive - diagnosis, followed by the optional device-flow identity check: +```mermaid +sequenceDiagram + participant Browser + participant ThothII + participant Authentik + Browser->>ThothII: Sign in + ThothII->>Authentik: Authorization Code with PKCE + Authentik-->>Browser: Login and consent + Browser->>ThothII: Callback with code + ThothII->>Authentik: Token exchange + Authentik-->>ThothII: Identity and groups + ThothII-->>Browser: Opaque session +``` - ```sh - tht auth check - tht auth check --interactive - tht doctor --json - ``` +## Provider OIDC -6. Run Workspace Test for aggregate live workspace and authentication validation. It must prove - discovery/JWKS, catalog access, and every configured group. The diagnostic result must contain - no secret values. `tht doctor --json` reports `authentication` after `configuration` and before - `services` in its exact ordered checklist. +1. Creare applicazione e provider OAuth2/OIDC. +2. Registrare esattamente `PUBLIC_URL/api/auth/oidc/callback`. +3. Abilitare gli scope `openid`, `profile` ed `email`. +4. Configurare un claim diretto `groups` come array di stringhe. -Only configured exact group names are queried. Additional Authentik or directory groups are ignored -silently, without a warning. A mapped group absent from Authentik fails closed with -`oidc_mapped_group_missing`; an ambiguous exact-name result uses -`oidc_mapped_group_ambiguous`. A group visible only in an upstream directory but not represented -in Authentik is missing from ThothII’s catalog and must not be treated as present. +## Catalogo gruppi -Rotate the two credentials independently through the protected secret-file procedure, then repeat -`tht auth check` and workspace Test. Never put either value in this guide, YAML, shell history, -diagnostic output, or acceptance evidence. +Creare un account di servizio dedicato con sola lettura dei gruppi. Conservare il token nel +bundle protetto come `THT_AUTHENTIK_API_TOKEN`. + +Mappare in `auth.yaml` i nomi esatti dei gruppi aziendali ai ruoli ThothII `user` e `admin`. +Gruppi non mappati vengono ignorati; un gruppo configurato ma assente genera un errore chiuso. + +## Diagnostica + +`tht auth check` controlla discovery, issuer, JWKS, accesso al catalogo e presenza dei gruppi +configurati. L'opzione `--interactive` aggiunge la verifica dell'identità tramite device flow, +quando il provider la supporta. + +Ruotare separatamente secret OIDC e token del catalogo gruppi. Nessuno dei due deve comparire in +YAML, cronologia shell, log o output diagnostico. diff --git a/docs/install/dwh-auth-client-enrollment.md b/docs/install/dwh-auth-client-enrollment.md index a91e4be9..0093e00b 100644 --- a/docs/install/dwh-auth-client-enrollment.md +++ b/docs/install/dwh-auth-client-enrollment.md @@ -1,70 +1,41 @@ # Enrollment client per DWH REST -La credenziale `dwh-auth` appartiene a una installazione ThothII, non a una persona. Serve solo se -il trasporto è `rest_api`; `postgres_direct` e `ssh_tunnel` non la usano. +La credenziale `dwh-auth` appartiene a una installazione ThothII e serve soltanto quando il +workspace usa il trasporto `rest_api`. -| Trasporto | Chiave `dwh-auth` | Materiale locale | -| --- | --- | --- | -| `rest_api` | Sì, una per installazione. | URL HTTPS, `API_KEY_FILE`, eventuale `TLS_CA_FILE`. | -| `postgres_direct` | No. | Credenziali PostgreSQL e TLS PostgreSQL. | -| `ssh_tunnel` | No. | Credenziali PostgreSQL e materiali SSH; è diagnostico-only nel runtime corrente. | +| Trasporto | Materiale richiesto | +| --- | --- | +| `rest_api` | URL HTTPS, `API_KEY_FILE`, eventuale `TLS_CA_FILE` | +| `postgres_direct` | Credenziali PostgreSQL e configurazione TLS PostgreSQL | +| `ssh_tunnel` | Credenziali PostgreSQL e materiale SSH | -Il ThothII server PSD resta `postgres_direct` read-only. Il Mac PSD e le installazioni remote -usano `rest_api`; non introdurre un tunnel SSH per aggirare REST. +## Consegna e conservazione -## Prerequisiti +Ricevere chiave e CA attraverso canali protetti separati. Conservare la chiave nel vault +dell'installazione o in un file regolare accessibile soltanto all'account autorizzato. Non +inserirla in Git, file YAML, argomenti, log o schermate condivise. -Ricevere chiave e CA, se necessaria, attraverso canali protetti separati. Confermare fuori banda il -fingerprint TLS prima dell'uso: [guida TLS](dwh-auth-tls.md). Conservare la chiave nel vault o in -un file protetto, mai Git, `.env` con il valore, argv, ambiente, log o evidenze. Annotare solo ID -pubblico. +## Configurazione ACME Limited -## Percorso GUI: vault dell'installazione - -1. In **Workspace management**, eseguire **Update workspace repository** se necessario e - selezionare il workspace. -2. Il trasporto `rest_api` è una precondizione amministrativa del binding locale, non una scelta della GUI. Controllare URL/trust locali e usare **Validate workspace source**. -3. Inserire la chiave nel campo write-only **Data warehouse API key**, poi **Save entered secrets**. - La GUI la conserva nel vault cifrato `workspace-secrets`, non la rileggere né la restituisce. -4. Eseguire **Test workspace connections**. Il controllo innocuo è `/rpc/ping`: atteso 2xx e - database/schema dichiarati. -5. Comunicare al server solo ID pubblico, timestamp e risultato. **Forget stored value** rimuove il valore e va - usato soltanto dopo conferma di sostituzione o revoca. - -## Percorso headless: binding reale - -`API_KEY_FILE` significa che il valore è nel file, non nella variabile. Questo è l'esempio Mac/local/remoto nel file PSD non tracciato `workspace-bindings.env`; non è il binding del server PSD Project A, che resta `postgres_direct`. I binding REST sono: +Esempio di binding headless per il workspace `acme-ebikes`: ```dotenv -THT_WS_PSD_CLINICAL_DWH_TRANSPORT=rest_api -THT_WS_PSD_CLINICAL_DWH_BASE_URL=https://supabase-aritmolab.policlinicosandonato.it/dwh/ -THT_WS_PSD_CLINICAL_DWH_API_KEY_FILE=/run/secrets/psd-clinical-dwh-api-key -THT_WS_PSD_CLINICAL_DWH_TLS_CA_FILE=/run/secrets/psd-clinical-dwh-ca.pem +THT_WS_ACME_EBIKES_DWH_TRANSPORT=rest_api +THT_WS_ACME_EBIKES_DWH_BASE_URL=https://dwh.acme.example/dwh/ +THT_WS_ACME_EBIKES_DWH_API_KEY_FILE=/run/secrets/acme-ebikes-dwh-api-key +THT_WS_ACME_EBIKES_DWH_TLS_CA_FILE=/run/secrets/acme-ebikes-dwh-ca.pem ``` -Nel file `operator.env` non tracciato, ogni suffisso `_SOURCE` indica solo il percorso assoluto del -file protetto di origine. Il comando genera un override non tracciato che monta quei file nel -`core`; non installa né avvia `dwh-auth` con Compose: +Il suffisso del workspace deriva dall'ID immutabile trasformando i trattini in underscore e +usando lettere maiuscole. `API_KEY_FILE` contiene il percorso del file montato, non il valore +della chiave. -```bash -bash scripts/generate-connector-secrets-override.sh \ - --bindings-env /absolute/protected/workspace-bindings.env \ - --operator-env /absolute/protected/operator.env \ - --output /absolute/protected/connector-secrets.override.yaml \ - --service core --role dwh -``` +## Rotazione e revoca -La chiave sorgente è un file regolare `0600` per il solo account autorizzato. Per altri workspace, -sostituire `PSD_CLINICAL` con ID immutabile maiuscolo (trattini in underscore). Vedere anche il -[protocollo diagnostico](../workspace-diagnostic-protocol.md). +Durante la rotazione, ricevere la nuova generazione, aggiornare il vault o il file montato e +confermare la connettività sulla route innocua `/rpc/ping`. Solo dopo questa conferma il +responsabile del server revoca la generazione precedente. -## Ping, rotazione e revoca - -Usare solo **Test workspace connections** su `/rpc/ping`: successo è 2xx con TLS verificato; il server -conferma l'ID con `key status`. Durante rotazione, ricevere nuova generazione, aggiornare vault o -file `API_KEY_FILE`, ripetere ping, attendere osservazione e far revocare la precedente. Dopo la -revoca: nuova positiva, precedente `401`. - -`401` non distingue chiave assente, scaduta o revocata. `503` è un guasto fail-closed di servizio, -socket o registro: non usare connessione diretta e non ridurre TLS. Non riattivare una chiave -revocata. Il percorso PSD è nel [runbook](../operations/psd-dwh-auth-rollout.md). +Un `401` indica una chiave assente, sconosciuta, scaduta o revocata. Un `503` indica che il +servizio di autorizzazione o il registro non sono disponibili. In entrambi i casi non aggirare +REST e non ridurre la verifica TLS. diff --git a/docs/install/dwh-auth-server.md b/docs/install/dwh-auth-server.md index 1d45e181..246cf57c 100644 --- a/docs/install/dwh-auth-server.md +++ b/docs/install/dwh-auth-server.md @@ -1,356 +1,65 @@ # `dwh-auth`: guida server -`dwh-auth` autentica la route REST `/dwh/` con una chiave per installazione. È un componente Linux -opzionale e server-side: usa `systemd`, non `tht` né Docker Compose, non legge risultati clinici e -non si collega a PostgreSQL. La chiave serve solo a `rest_api`; `postgres_direct` e `ssh_tunnel` -non la usano. +`dwh-auth` protegge la route REST `/dwh/` con una chiave distinta per ogni installazione +ThothII. Il componente gira come servizio Linux separato, non legge i dati del DWH e non si +collega direttamente a PostgreSQL. -## Prerequisiti e confini - -- Usare un checkout revisionato, Docker per la build e un operatore autorizzato sul server DWH. -- Una chiave identifica un'installazione, non una persona. L'`installation-id` è unico, non - personale e senza dati clinici. -- Chiavi, digest, file di consegna e backup restano in file protetti: mai Git, argv, variabili - d'ambiente, log, JSON pubblico o evidenze. -- Preparare backup e rollback prima di Nginx. Installare il servizio non autorizza una modifica - della route pubblica. - -## Percorsi, owner e mode - -| Oggetto | Percorso | Owner e mode | -| --- | --- | --- | -| Binario | `/usr/local/sbin/dwh-auth` | `root:root`, `0755` | -| Unit | `/etc/systemd/system/dwh-auth.service` | `root:root`, `0644` | -| Tmpfiles | `/usr/lib/tmpfiles.d/dwh-auth.conf` | `root:root`, `0644` | -| Registro, `active`, `revoked` | `/var/lib/dwh-auth/` | `root:dwh-auth`, `2750` | -| Lock | `/var/lib/dwh-auth/.writer.lock` | `root:dwh-auth`, `0640` | -| Record | `/var/lib/dwh-auth/{active,revoked}/<public-key-id>.json` | `root:dwh-auth`, `0640` | -| Socket runtime | `/run/dwh-auth/verify.sock` | `dwh-auth:www-data`, `0660` | -| Consegne e backup | `/root/dwh-auth-provision/` | directory `root:root` `0700`, file `0600` | - -Il record conserva un digest interno (`secret_sha256`) e metadati, mai la chiave in chiaro. Non -leggere, stampare, calcolare o mettere quel digest in una prova operativa. - -## Build, installazione e avvio - -Costruire dal commit congelato e registrare solo checksum del binario e SHA sorgente: - -```bash -cd /srv/thothii/app -bash scripts/build-dwh-auth.sh --output /tmp/dwh-auth-release -sha256sum /tmp/dwh-auth-release/dwh-auth-linux-amd64 +```mermaid +flowchart LR + CLIENT["Installazione ThothII"] -->|"X-API-Key"| NGINX["Nginx"] + NGINX --> AUTH["dwh-auth\nUnix socket"] + AUTH --> REGISTRY["Registro chiavi\nactive e revoked"] + AUTH -->|"authorized"| REST["DWH REST"] ``` -Scegliere l'architettura corretta. Il template PSD usa il gruppo Nginx `www-data`; confermarlo -prima dell'installazione su un host diverso. +## Confini di sicurezza -```bash -sudo groupadd --system dwh-auth -sudo useradd --system --no-create-home --shell /usr/sbin/nologin --gid dwh-auth dwh-auth -sudo install -o root -g root -m 0755 /tmp/dwh-auth-release/dwh-auth-linux-amd64 /usr/local/sbin/dwh-auth -sudo install -o root -g root -m 0644 deploy/dwh-auth/dwh-auth.service /etc/systemd/system/dwh-auth.service -sudo install -o root -g root -m 0644 deploy/dwh-auth/dwh-auth.tmpfiles.conf /usr/lib/tmpfiles.d/dwh-auth.conf -sudo install -d -o root -g root -m 0700 /root/dwh-auth-provision -sudo systemd-tmpfiles --create /usr/lib/tmpfiles.d/dwh-auth.conf -sudo /usr/local/sbin/dwh-auth --registry-root /var/lib/dwh-auth check -sudo systemd-analyze verify /etc/systemd/system/dwh-auth.service -sudo systemctl daemon-reload -sudo systemctl enable --now dwh-auth -sudo systemctl status dwh-auth --no-pager -``` +- Una chiave identifica un'installazione, non una persona. +- Chiavi e backup restano in file protetti e non entrano in Git, log, argomenti o JSON pubblico. +- Il registro conserva digest e metadati, mai la chiave in chiaro. +- La route REST deve essere esposta esclusivamente tramite TLS verificato. -Controllare i mode con `stat`. Il servizio apre il registro in sola lettura e crea solo il socket. -Non creare JSON, lock o socket a mano: oggetti insicuri devono fallire chiusi. +## Installazione -## Check, elenco e stato +Il servizio usa questi percorsi: -Usare sempre un root assoluto. Questi comandi espongono solo ID pubblici, stato, date e scadenza: - -```bash -sudo /usr/local/sbin/dwh-auth --registry-root /var/lib/dwh-auth check -sudo /usr/local/sbin/dwh-auth --registry-root /var/lib/dwh-auth key list --json -key_id=public-key-id -sudo /usr/local/sbin/dwh-auth --registry-root /var/lib/dwh-auth key status --key-id "$key_id" --json -``` - -Un errore di integrità, permessi, symlink o JSON malformato richiede ripristino da backup protetto, -non una correzione manuale del record. - -## Creazione, consegna, scadenza e revoca - -Il comando crea la chiave una volta in un nuovo file assoluto `0600`; stdout contiene solo ID -pubblico, installazione e percorso. Il file di output non deve esistere. - -```bash -installation_id=psd-mac-primary -description=operatore-mac-primario -key_output=/root/dwh-auth-provision/psd-mac-primary.key -sudo /usr/local/sbin/dwh-auth --registry-root /var/lib/dwh-auth key create \ - --installation-id "$installation_id" \ - --description "$description" \ - --output "$key_output" -``` - -Aggiungere `--expires-at "YYYY-MM-DDTHH:MM:SSZ"` solo se la policy impone una scadenza; il default è nessuna -scadenza. Consegnare il file solo con vault aziendale, secret manager, MDM o trasferimento -autenticato ristretto. Mai email, chat, ticket, `cat` o copia-incolla. Il client conferma ID -pubblico e ping, poi il materiale temporaneo viene rimosso secondo policy. - -L'import legacy è temporaneo PSD: il file sorgente è già `root:root` `0600` e non viene mai letto o -stampato dall'operatore. - -```bash -sudo /usr/local/sbin/dwh-auth --registry-root /var/lib/dwh-auth key import \ - --legacy-raw --installation-id legacy-shared \ - --from-file /root/dwh-auth-provision/legacy-shared.key -``` - -Per rotare: creare seconda generazione, consegnarla, configurarla e provare `/rpc/ping`; confermare -l'ID pubblico; attendere l'osservazione; poi revocare la precedente e provare nuova=successo, -precedente=401. - -```bash -previous_key_id=public-key-id -revocation_reason=shared-credential-rotation -sudo /usr/local/sbin/dwh-auth --registry-root /var/lib/dwh-auth key revoke \ - --key-id "$previous_key_id" --reason "$revocation_reason" -``` - -La revoca non è annullabile e un ID revocato non si ricrea. - -## Backup, rollback e disinstallazione - -Prima di mutare, creare un archivio root-only `0600` del registro e copie protette delle sole -configurazioni coinvolte. L'archivio contiene digest, quindi è materiale riservato: custodirlo su -storage cifrato approvato; l'evidenza ammessa riporta solo percorso, owner, mode, timestamp e -checksum dell'archivio. Il rollback dual-key ripristina la route e il servizio revisionati, esegue -`nginx -t` e fa reload solo autorizzato; non ripristina chiavi revocate, PostgreSQL, sessioni -legacy, indici Qdrant o cache Ollama. - -La disinstallazione richiede autorizzazione esplicita, client REST migrati/revocati e rollback non -più necessario. Solo allora disabilitare l'unità; conservare registro e backup fino alla retention -approvata. Non inserire `dwh-auth` in Compose o in `tht start`/`tht stop`. - -## Procedure riproducibili e secret-safe - -Eseguire soltanto nel gate autorizzato. Le variabili seguenti contengono percorsi, timestamp e -codici, mai una chiave. Il manifest e l'archivio del registro sono `0600`; l'archivio resta -materiale riservato su storage cifrato approvato. - -```bash -run_id=$(date -u +%Y%m%dT%H%M%SZ) -backup_root=/root/dwh-auth-provision -registry_root=/var/lib/dwh-auth -registry_backup="$backup_root/registry-$run_id.tar" -manifest="$backup_root/registry-$run_id.manifest" -sudo install -o root -g root -m 0600 /dev/null "$registry_backup" -sudo install -o root -g root -m 0600 /dev/null "$manifest" -sudo tar --acls --xattrs -C /var/lib -cf "$registry_backup" dwh-auth -sudo sh -c 'sha256sum "$1" > "$2"' sh "$registry_backup" "$manifest" -if sudo sha256sum -c "$manifest" >/dev/null; then printf 'registry_manifest=PASS\n'; else printf 'registry_manifest=FAIL\n' >&2; exit 1; fi -``` - -Il ripristino non sovrappone mai un tar al registro attivo. Estrarre prima in staging nello stesso -filesystem di `/var/lib`, verificare il candidato, rinominare il registro attuale in una copia -recuperabile e sostituirlo. Non cancellare il pre-ripristino: serve al rollback se `check` o -l'avvio falliscono. - -```bash -registry_staging="/var/lib/.dwh-auth-restore-$run_id" -registry_candidate="$registry_staging/dwh-auth" -registry_previous="/var/lib/dwh-auth.pre-restore-$run_id" -if [ -e "$registry_staging" ] || [ -e "$registry_previous" ]; then printf 'registry_restore=FAIL\n' >&2; exit 1; fi -if ! sudo sha256sum -c "$manifest" >/dev/null; then printf 'registry_restore=FAIL\n' >&2; exit 1; fi -if ! sudo install -d -o root -g root -m 0700 "$registry_staging"; then printf 'registry_restore=FAIL\n' >&2; exit 1; fi -if ! sudo tar --acls --xattrs -C "$registry_staging" -xf "$registry_backup"; then printf 'registry_restore=FAIL\n' >&2; exit 1; fi -if ! sudo /usr/local/sbin/dwh-auth --registry-root "$registry_candidate" check; then printf 'registry_restore=FAIL\n' >&2; exit 1; fi -if ! sudo systemctl stop dwh-auth; then printf 'registry_restore=FAIL\n' >&2; exit 1; fi -if ! sudo mv -T -- "$registry_root" "$registry_previous"; then - if sudo systemctl start dwh-auth; then printf 'registry_restore_rollback=PASS\n' >&2; else printf 'registry_restore_rollback=FAIL\n' >&2; fi - exit 1 -fi -if ! sudo mv -T -- "$registry_candidate" "$registry_root"; then - if ! sudo mv -T -- "$registry_previous" "$registry_root"; then printf 'registry_restore_rollback=FAIL\n' >&2; exit 1; fi - if sudo /usr/local/sbin/dwh-auth --registry-root "$registry_root" check && sudo systemctl start dwh-auth; then printf 'registry_restore_rollback=PASS\n' >&2; else printf 'registry_restore_rollback=FAIL\n' >&2; fi - exit 1 -fi -if sudo /usr/local/sbin/dwh-auth --registry-root "$registry_root" check && sudo systemctl start dwh-auth; then - printf 'registry_restore=PASS\n' -else - sudo systemctl stop dwh-auth || true - if ! sudo mv -T -- "$registry_root" "$registry_staging/failed-dwh-auth"; then printf 'registry_restore_rollback=FAIL\n' >&2; exit 1; fi - if ! sudo mv -T -- "$registry_previous" "$registry_root"; then printf 'registry_restore_rollback=FAIL\n' >&2; exit 1; fi - if sudo /usr/local/sbin/dwh-auth --registry-root "$registry_root" check && sudo systemctl start dwh-auth; then printf 'registry_restore_rollback=PASS\n' >&2; else printf 'registry_restore_rollback=FAIL\n' >&2; fi - exit 1 -fi -``` - -Per le prove, creare file header `0600` che contengono esattamente `X-API-Key: valore`. Il valore -passa dal file chiave al file header senza argv, ambiente o stdout. I file header sono materiale -segreto con la stessa custodia e retention delle chiavi. - -```bash -v1_key_file="$key_output" -legacy_key_file=/root/dwh-auth-provision/legacy-shared.key -v1_header_file=/root/dwh-auth-provision/dwh-auth-v1.header -legacy_header_file=/root/dwh-auth-provision/dwh-auth-legacy.header -random_header_file=/root/dwh-auth-provision/dwh-auth-random.header -if ! sudo python3 -c ' -import pathlib, sys -if any(b"\n" in pathlib.Path(path).read_bytes() for path in sys.argv[1:]): - raise SystemExit(1) -' "$v1_key_file" "$legacy_key_file"; then - printf 'key_file_bytes=FAIL\n' >&2 - exit 1 -fi -printf 'key_file_bytes=PASS\n' -for header_file in "$v1_header_file" "$legacy_header_file" "$random_header_file"; do - sudo install -o root -g root -m 0600 /dev/null "$header_file" -done -sudo sh -c '{ printf "%s" "X-API-Key: "; dd if="$1" bs=65536 status=none; printf "\n"; } > "$2"' sh "$v1_key_file" "$v1_header_file" -sudo sh -c '{ printf "%s" "X-API-Key: "; dd if="$1" bs=65536 status=none; printf "\n"; } > "$2"' sh "$legacy_key_file" "$legacy_header_file" -sudo sh -c 'printf "%s\n" "X-API-Key: invalid-test" > "$1"' sh "$random_header_file" -``` - -Il socket `/verify` deve restituire 204 per v1 e legacy durante il dual-key, 401 per file casuale -e richiesta senza header. Stampare solo PASS/FAIL. - -```bash -status=$(sudo curl --header "@$v1_header_file" --unix-socket /run/dwh-auth/verify.sock --output /dev/null --silent --show-error --write-out '%{http_code}' http://localhost/verify) -[ "$status" = 204 ] && printf 'socket_v1=PASS\n' || { printf 'socket_v1=FAIL\n' >&2; exit 1; } -status=$(sudo curl --header "@$legacy_header_file" --unix-socket /run/dwh-auth/verify.sock --output /dev/null --silent --show-error --write-out '%{http_code}' http://localhost/verify) -[ "$status" = 204 ] && printf 'socket_legacy=PASS\n' || { printf 'socket_legacy=FAIL\n' >&2; exit 1; } -status=$(sudo curl --header "@$random_header_file" --unix-socket /run/dwh-auth/verify.sock --output /dev/null --silent --show-error --write-out '%{http_code}' http://localhost/verify) -[ "$status" = 401 ] && printf 'socket_random=PASS\n' || { printf 'socket_random=FAIL\n' >&2; exit 1; } -status=$(sudo curl --unix-socket /run/dwh-auth/verify.sock --output /dev/null --silent --show-error --write-out '%{http_code}' http://localhost/verify) -[ "$status" = 401 ] && printf 'socket_missing=PASS\n' || { printf 'socket_missing=FAIL\n' >&2; exit 1; } -``` - -Per HTTPS reale usare i file header protetti e la CA approvata contro `/dwh/rpc/ping`: PostgREST -può restituire qualsiasi 2xx, non si pretende 204. Prima della revoca, v1 e legacy devono dare -2xx; il file casuale deve dare 401. - -```bash -ping_url=https://supabase-aritmolab.policlinicosandonato.it/dwh/rpc/ping -ca_file=/root/dwh-auth-provision/psd-dwh-ca.pem -status=$(sudo curl --header "@$v1_header_file" --cacert "$ca_file" --connect-timeout 5 --max-time 15 --output /dev/null --silent --show-error --write-out '%{http_code}' "$ping_url") -case "$status" in 2??) printf 'https_v1_pre_revoke=PASS\n' ;; *) printf 'https_v1_pre_revoke=FAIL\n' >&2; exit 1 ;; esac -status=$(sudo curl --header "@$legacy_header_file" --cacert "$ca_file" --connect-timeout 5 --max-time 15 --output /dev/null --silent --show-error --write-out '%{http_code}' "$ping_url") -case "$status" in 2??) printf 'https_legacy_pre_revoke=PASS\n' ;; *) printf 'https_legacy_pre_revoke=FAIL\n' >&2; exit 1 ;; esac -status=$(sudo curl --header "@$random_header_file" --cacert "$ca_file" --connect-timeout 5 --max-time 15 --output /dev/null --silent --show-error --write-out '%{http_code}' "$ping_url") -[ "$status" = 401 ] && printf 'https_random=PASS\n' || { printf 'https_random=FAIL\n' >&2; exit 1; } -``` - -Dopo l'osservazione, revocare solo la legacy usando il suo ID pubblico già registrato. Dopo la -revoca v1 resta 2xx e legacy diventa 401 anche via HTTPS. - -```bash -legacy_key_id=legacy-shared -sudo /usr/local/sbin/dwh-auth --registry-root "$registry_root" key revoke --key-id "$legacy_key_id" --reason shared-credential-rotation -status=$(sudo curl --header "@$v1_header_file" --cacert "$ca_file" --connect-timeout 5 --max-time 15 --output /dev/null --silent --show-error --write-out '%{http_code}' "$ping_url") -case "$status" in 2??) printf 'https_v1_post_revoke=PASS\n' ;; *) printf 'https_v1_post_revoke=FAIL\n' >&2; exit 1 ;; esac -status=$(sudo curl --header "@$legacy_header_file" --cacert "$ca_file" --connect-timeout 5 --max-time 15 --output /dev/null --silent --show-error --write-out '%{http_code}' "$ping_url") -[ "$status" = 401 ] && printf 'https_legacy_post_revoke=PASS\n' || { printf 'https_legacy_post_revoke=FAIL\n' >&2; exit 1; } -``` - -Per provare 503 in una finestra approvata, registrare l'orario, fermare temporaneamente l'unità, -eseguire il ping con timeout e trap di ripristino; il comando deve stampare solo PASS/FAIL. - -```bash -was_active=$(sudo systemctl is-active dwh-auth || true) -[ "$was_active" = active ] || { printf 'https_auth_down=FAIL\n' >&2; exit 1; } -restore_auth() { sudo systemctl start dwh-auth; } -trap restore_auth EXIT INT TERM -sudo systemctl stop dwh-auth -status=$(sudo curl --header "@$random_header_file" --cacert "$ca_file" --connect-timeout 5 --max-time 15 --output /dev/null --silent --show-error --write-out '%{http_code}' "$ping_url" || true) -[ "$status" = 503 ] && printf 'https_auth_down=PASS\n' || { printf 'https_auth_down=FAIL\n' >&2; exit 1; } -sudo systemctl start dwh-auth -trap - EXIT INT TERM -``` - -Lo scan journal non salva righe grezze: controlla davvero le chiavi v1 e legacy leggendo solo i -percorsi dei file da argv, e conserva anche la difesa generica per prefisso e digest. Il filtro -emette solo PASS/FAIL. - -```bash -since=$(date -u -d '15 minutes ago' +%Y-%m-%dT%H:%M:%SZ) -if sudo python3 -c ' -import pathlib, subprocess, sys -max_journal_bytes = 1_048_576 -max_journal_lines = 10_000 -max_chunk_bytes = 65_536 -process = None -try: - actual_keys = {pathlib.Path(path).read_bytes() for path in sys.argv[2:]} - needles = (b"thtdwh_v1", b"secret_sha256", *actual_keys) - max_needle_length = max(map(len, needles)) - process = subprocess.Popen( - ["journalctl", "-u", "dwh-auth", "--since", sys.argv[1], "--no-pager", "--output=cat"], - stdout=subprocess.PIPE, - stderr=subprocess.DEVNULL, - ) -except OSError: - raise SystemExit(2) -def stop_child(): - if process is not None: - if process.poll() is None: - process.kill() - process.wait() -bytes_seen = 0 -line_count = 0 -line_open = False -carry = b"" -try: - while True: - remaining = max_journal_bytes - bytes_seen - if remaining == 0: - if process.stdout.read1(1): - raise SystemExit(2) - break - chunk = process.stdout.read1(min(max_chunk_bytes, remaining)) - if not chunk: - break - bytes_seen += len(chunk) - searchable = carry + chunk - if any(needle in searchable for needle in needles): - raise SystemExit(1) - carry = searchable[-(max_needle_length - 1):] - for byte in chunk: - if byte == 10: - line_count += 1 - line_open = False - if line_count > max_journal_lines: - raise SystemExit(2) - else: - line_open = True - if line_open: - line_count += 1 - if line_count > max_journal_lines: - raise SystemExit(2) -finally: - stop_child() -if process.returncode != 0: - raise SystemExit(2) -' "$since" "$v1_key_file" "$legacy_key_file"; then - printf 'journal_actual_key_scan=PASS\n' -else - printf 'journal_actual_key_scan=FAIL\n' >&2 - exit 1 -fi -``` - -Dopo rollback verificato e migrazione/revoca di ogni client REST, la disinstallazione resta -condizionata all'approvazione: eseguire `sudo systemctl disable --now dwh-auth`, ma mantenere -registro, backup, manifest e file header protetti per la retention; non cancellarli durante il -rollback. - -## Troubleshooting - -| Sintomo | Interpretazione e azione | +| Oggetto | Percorso | | --- | --- | -| `401` | Chiave assente, malformata, sconosciuta, scaduta, revocata o errata. Verificare trasporto, ID pubblico e consegna; non cercare dettagli nel messaggio. | -| `503` | Servizio, socket o registro non disponibile/sicuro. Controllare `systemctl`, socket, mode e `check`; ripristinare il backup approvato. | -| `check` fallisce | Integrità del registro non valida. Fermare le scritture, preservare stato e ripristinare; non editare JSON. | -| TLS fallisce | CA o SAN non validi. Seguire [TLS](dwh-auth-tls.md), senza bypass. | +| Binario | `/usr/local/sbin/dwh-auth` | +| Unit systemd | `/etc/systemd/system/dwh-auth.service` | +| Registro | `/var/lib/dwh-auth/` | +| Socket | `/run/dwh-auth/verify.sock` | +| Consegne protette | `/root/dwh-auth-provision/` | -Per il rollout PSD con i due gate separati vedere il [runbook PSD](../operations/psd-dwh-auth-rollout.md). +Installare binario e unit con owner `root`, creare l'utente di servizio `dwh-auth`, quindi +abilitare l'unità con `systemctl enable --now dwh-auth`. Il socket deve essere accessibile al +gruppo usato da Nginx. + +## Creazione e revoca delle chiavi + +Esempio per l'installazione ACME Limited: + +```bash +sudo dwh-auth --registry-root /var/lib/dwh-auth key create \ + --installation-id acme-factory-primary \ + --description acme-factory-primary \ + --output /root/dwh-auth-provision/acme-factory-primary.key +``` + +Consegnare il file attraverso un vault aziendale o un canale autenticato. Per la rotazione, +creare una nuova chiave, distribuirla, aggiornare il client e revocare la precedente usando il +suo ID pubblico: + +```bash +sudo dwh-auth --registry-root /var/lib/dwh-auth key revoke \ + --key-id PUBLIC_KEY_ID \ + --reason scheduled-rotation +``` + +La revoca è definitiva. Conservare backup cifrati del registro prima di ogni mutazione. + +## Integrazione Nginx + +Nginx inoltra la chiave al socket di `dwh-auth`. Solo una risposta autorizzata permette il +passaggio verso il DWH REST; chiavi assenti, sconosciute, scadute o revocate ricevono `401`, +mentre indisponibilità del servizio o del registro producono `503`. diff --git a/docs/install/dwh-auth-tls.md b/docs/install/dwh-auth-tls.md index d369be1b..fa44613d 100644 --- a/docs/install/dwh-auth-tls.md +++ b/docs/install/dwh-auth-tls.md @@ -1,49 +1,40 @@ # TLS per DWH REST -La chiave DWH è accettabile solo sopra TLS verificato. Un errore `401` o `503` non autorizza mai a -ridurre la verifica del certificato. +La chiave DWH è accettabile solo sopra TLS verificato. Errori di autorizzazione o disponibilità +non autorizzano mai a disabilitare la verifica del certificato. -## Stato PSD +## CA privata -L'origine REST PSD corrente usa il certificato self-issued/private di Nginx. Il SAN copre -`supabase-aritmolab.policlinicosandonato.it`, l'origine `.it` approvata, e non copre un dominio -`.com`. Non usare `.com` finché non è incluso esplicitamente nel SAN. +Quando il DWH REST usa una CA aziendale, consegnare il certificato separatamente dalla chiave +API. La CA non è una credenziale, ma la sua integrità è un confine di sicurezza: deve restare +fuori da Git e non essere scrivibile da utenti non autorizzati. -Chi non dispone già di trust equivalente approvato riceve la CA separatamente e configura -`TLS_CA_FILE`. La CA non è una credenziale, ma la sua integrità è un confine di sicurezza: fuori da -Git e non scrivibile da utenti non autorizzati. +Esempio ACME Limited: + +```dotenv +THT_WS_ACME_EBIKES_DWH_TLS_CA_FILE=/run/secrets/acme-ebikes-dwh-ca.pem +``` ## Fingerprint fuori banda -Calcolare localmente il fingerprint del file ricevuto: +Calcolare il fingerprint del file ricevuto e confrontarlo attraverso un canale indipendente: ```bash -openssl x509 -noout -fingerprint -sha256 -in /absolute/protected/psd-dwh-ca.pem +openssl x509 -noout -fingerprint -sha256 \ + -in /absolute/protected/acme-ebikes-dwh-ca.pem ``` -Confrontarlo con il responsabile autorizzato tramite un canale indipendente dalla consegna (vault -aziendale o canale telefonico verificato). Nell'evidenza registrare solo conferma, approvatore e -timestamp; mai corpo certificato, fingerprint completo o output grezzo. +Il SAN del certificato deve includere il nome esatto usato dal binding, per esempio +`dwh.acme.example`. -## Binding e ping +## Rinnovo -Il binding headless PSD effettivo è: +1. Preparare certificato e chain nuovi. +2. Confermare SAN e fingerprint fuori banda. +3. Distribuire la nuova CA ai client mantenendo temporaneamente la precedente. +4. Aggiornare il binding e confermare la connettività con TLS normale. +5. Installare il certificato server. +6. Ritirare il trust precedente dopo la finestra concordata. -```dotenv -THT_WS_PSD_CLINICAL_DWH_TLS_CA_FILE=/run/secrets/psd-clinical-dwh-ca.pem -``` - -Il file sorgente locale è collegato da file operatore non tracciato. Usare URL `.it`, poi -**Test workspace connections** su `/rpc/ping`. Non disabilitare TLS e non usare `curl -k`. - -## Rinnovo coordinato - -1. Preparare certificato e chain nuovi; verificare prima SAN `.it` e assenza di falsa copertura `.com`. -2. Confermare fuori banda il nuovo fingerprint. -3. Consegnare la CA/chain nuova ai client con `TLS_CA_FILE`, senza rimuovere ancora la precedente. -4. Aggiornare vault/binding e verificare ping con TLS normale. -5. Solo con gate Nginx approvato installare il certificato server e ripetere il ping. -6. Ritirare il trust precedente dopo la finestra approvata. - -Il rinnovo non modifica chiavi `dwh-auth`, record o ruoli PostgreSQL. TLS e rollback della route -restano approvazioni e backup distinti. +Non usare `curl -k`, non disabilitare TLS e non incorporare certificati o fingerprint completi +nei documenti condivisi. diff --git a/docs/install/local-workspace-registry.md b/docs/install/local-workspace-registry.md deleted file mode 100644 index 9bf91723..00000000 --- a/docs/install/local-workspace-registry.md +++ /dev/null @@ -1,176 +0,0 @@ -# Local workspace repository installation (macOS, Windows, and Linux) - -This manual connects a local ThothII installation to one remote Git repository hosted by a Git -server such as GitHub, GitLab, or Gitea. ThothII is a read-only consumer: it fetches, -validates, and activates workspace revisions, but never edits, commits, pushes, or publishes them. - -## Architecture ownership contract - -| Component | Ownership | Operator contract | -| --- | --- | --- | -| DWH | External | Configure the external endpoint and complete its runtime credentials in Workspace management. | -| LLM | External | Configure the external endpoint and model policy during installation. | -| Qdrant | Internal | Compose runs the internal service and persists `qdrant-data`. | -| Ollama embedding | Internal | Compose runs the internal `qwen3-embedding:0.6b` service and model-init job. | - -## Semantic index ownership contract - -| Scope | Ownership rule | Isolation rule | -| --- | --- | --- | -| Workspace semantic index | Each workspace keeps exactly one Qdrant collection reserved for itself. | Schema, Evidence, and memory records share that one collection and are separated by the `kind` payload. | - -The mandatory semantic stack is CPU-first. Set `THOTH_ENABLE_EMBEDDING_GPU=1` only after the -documented GPU prerequisites are satisfied. The embedding contract is fixed at -`qwen3-embedding:0.6b`, 1024 dimensions, cosine distance. - -## Prerequisites - -- A working local installation described by [local.md](local.md). -- A remote Git repository and a read-only deploy credential for this ThothII installation. -- A separate authoring clone in which a workspace curator can edit and publish source revisions. -- `tht` built with `bash scripts/build-tht.sh`. - -## Prepare and publish a workspace source - -Create a local workspace in an ordinary source directory outside ThothII's data directories. The -canonical repository layout is: - -```text -thoth-workspaces.yaml -<workspace-id>/workspace.yaml -<workspace-id>/evidence/ # optional, repository-owned Evidence -<workspace-id>/schema/annotations.yaml # optional curated annotations -``` - -The catalog lists `{id, name, description?}` and the descriptor at -`<workspace-id>/workspace.yaml` must match that metadata. Use the examples in -`deploy/workspaces/` as authoring references. Do not store passwords, tokens, private keys, or -signed URLs in Git. - -Publishing is an author-side Git operation: validate the source, commit it, and push it from the -separate authoring clone to the configured branch. This is the only meaning of “publish” in the -workspace lifecycle. ThothII has no author identity and no Git write credential. - -## Use the workspace from the application - -After the installation is started, use Workspace management from the authenticated application: - -1. Run **Update workspace repository** to fetch and validate the configured Git branch into the - application-owned registry. The operation is all-or-nothing and does not modify the authoring - clone. -2. Confirm that the installation-owned `workspace-secrets` storage remains outside the source - repository and contains no credentials in the workspace descriptors. -3. Select the workspace and run **Validate workspace source** to verify the active descriptor, catalog, - Evidence, annotations, and runtime bindings. -4. Run **Test workspace connections** only with the approved read-only DWH/Evidence test configuration. - Results are redacted and the workspace source remains unchanged. - -<!-- workspace-descriptor-contract:start --> -Schema v3 is the only accepted workspace descriptor. -Schema v1 and v2 workspace descriptors are rejected before activation. -<!-- workspace-descriptor-contract:end --> - -## Configure the remote Git repository - -Copy `docs/install/examples/thothii-installation.local.yaml` to an operator-controlled absolute -path. Its `workspaceRepository` block records the remote, branch, and read-only access method. -Choose exactly one transport override: - -- SSH: `deploy/compose.git-ssh.yaml`, with a read-only deploy key and pinned `known_hosts` file. -- HTTPS: `deploy/compose.git-https.yaml`, with a read-only token in a Git credentials file and an - optional private CA file. - -The remote and branch are installation configuration. Git credentials remain protected -installation files and are never accepted by Workspace management or returned by its API. - -Example non-secret/operator paths: - -```dotenv -THT_WORKSPACE_GIT_REMOTE=git@git.example.com:organization/workspaces.git -THT_WORKSPACE_GIT_BRANCH=main -THT_WORKSPACE_INSTALLATION_ID=local -PI_AUTH_FILE=/absolute/path/to/operator/pi-auth.json -THT_SECRETS_FILE=/absolute/path/to/operator/thothii.secrets -THT_WORKSPACE_GIT_SSH_KEY_FILE=/absolute/path/to/operator/git-ssh-key -THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=/absolute/path/to/operator/git-known-hosts -``` - -Keep these files outside both the ThothII checkout and the workspace source repository. Protect -them with mode `0600` on macOS/Linux or an equivalent single-user ACL on Windows. - -## Start and update the installation - -Use only the installation-aware lifecycle: - -```bash -export THT_SOURCE_ROOT=/absolute/path/to/ThothII -THT_BIN=tht -INSTALLATION=/absolute/path/to/operator/thothii-installation.yaml -"$THT_BIN" --installation "$INSTALLATION" start -"$THT_BIN" --installation "$INSTALLATION" doctor -``` - -At startup ThothII clones or fetches the configured repository into its application-managed -`workspace-registry` volume. Later, **Update workspace repository** performs a server-side fetch -and fast-forward candidate checkout. It does not copy anything to the user's computer. - -## Complete runtime secrets in Workspace management - -### Chiavi DWH REST per installazione - -Se il binding selezionato è `rest_api`, la chiave DWH è una credenziale per questa installazione e si salva nel vault tramite **Save entered secrets** oppure in un file locale indicato da `API_KEY_FILE`. `postgres_direct` e `ssh_tunnel` non usano questa chiave. Per emissione, TLS, rotazione e verifica `/rpc/ping`, seguire [enrollment DWH REST](dwh-auth-client-enrollment.md). - -Open Workspace management after the first successful repository update. - -1. At the repository level, review the configured host, repository, branch, and current revision. -2. Select a workspace. Repository update does not require a selection; validation and connection - tests do. -3. Review the runtime fields derived from the selected DWH transport and Evidence authentication - mechanism. -4. Enter or rotate the required values and choose **Save entered secrets**. -5. Run **Validate workspace source** and then **Test workspace connections**. - -Secret fields are write-only. The GUI receives only configured/missing status. Values are -encrypted by the backend in the platform-neutral `workspace-secrets` volume. ThothII temporarily -materializes a restrictive file only while an existing file-oriented connector needs it, then -removes that file when the runtime lease ends. **Forget stored value** deletes the selected encrypted value. - -The workspace YAML stays environment-independent: it declares connector mechanisms, not host -paths or credentials. Installation trust material such as a Git CA or `known_hosts` remains an -operator concern; DWH and Evidence credentials are completed in the GUI. - -## Validation and activation behavior - -An update follows this sequence: - -1. Fetch the configured branch into a candidate checkout managed by ThothII. -2. Validate the catalog, every descriptor, repository-relative Evidence, and cross-workspace - invariants at the same Git commit. -3. If every workspace is valid, atomically mark that complete commit as active. -4. If any validation fails, report sanitized diagnostics and keep the previous active revision. - -The active checkout is read-only application state. Never edit files under -`/data/workspace-registry`. A source correction must be committed and pushed from the authoring -clone, then fetched again with **Update workspace repository**. - -## Backup, rotation, and recovery - -Back up the `workspace-registry`, `workspace-secrets`, `sessions`, `qdrant-data`, -`embedding-models`, `settings`, and `pi-state` volumes together. The encrypted vault is useless -without its generated master key, so preserve the entire `workspace-secrets` volume and protect -the backup as secret material. - -Rotate a runtime credential by saving its replacement in Workspace management and rerunning its -connection test. Rotate Git credentials in the installation files and restart `core`. To recover -from a bad remote revision, correct or revert it in the authoring repository and run the update; -until validation succeeds, the previous active snapshot remains available. - -## Troubleshooting - -| Symptom | Meaning and action | -| --- | --- | -| Repository unavailable | Check remote host, branch, read-only deploy credential, CA, and `known_hosts`. | -| Candidate rejected | Fix the reported source error in the authoring clone, commit, push, and update again. | -| Runtime configuration required | Select the workspace and complete each required secret field. | -| Connection test fails | Rotate the relevant secret or correct the non-secret endpoint in the source/installation as appropriate. | -| Active revision did not change | The candidate was invalid or was already active; inspect the repository status. | diff --git a/docs/install/local.md b/docs/install/local.md deleted file mode 100644 index 3dd98563..00000000 --- a/docs/install/local.md +++ /dev/null @@ -1,499 +0,0 @@ -# Install ThothII on a local PC or Mac - -This guide installs one loopback-only ThothII on the same Windows, macOS, or Linux computer that -runs Docker. The supported application is one Docker Compose distribution containing exactly -`frontend` and `core`; Pi is pinned inside `core`. DWH, vector database, embedding, and LLM remain -external configurable services even when they run on this computer. - -No host Pi, Node.js, Python, Go toolchain, Docker socket in core, or browser shell is required. -Commands that contain example paths must be changed to absolute paths on your computer. - -## Choose your platform - -- **macOS:** use Terminal and Docker Desktop. Apple Silicon and Intel are supported by the local - image build. -- **Windows PowerShell:** use Docker Desktop with its WSL2 engine, Git for Windows, the Windows - build launcher, and `tht-windows-amd64.exe`. -- **Windows WSL2 (recommended):** enable Docker Desktop integration for your Linux distribution, - clone under `/home/<user>` rather than `/mnt/c`, and follow the Linux shell commands. -- **Linux PC:** use Docker Engine plus the Compose v2 plugin and the Linux `tht` binary. - -Windows users must also read [Windows and WSL2 line endings](windows-line-endings.md) before the -first build. - -## Prerequisites - -Install only: - -1. Git 2.39 or newer. -2. Docker Desktop on macOS/Windows, or Docker Engine on Linux. -3. Docker Compose v2 (`docker compose`, not legacy `docker-compose`). -4. About 10 GB of free disk for source, images, build cache, and initial volumes. -5. Network access to the workspace Git remote and configured DWH/vector/embedding/LLM endpoints. - -Verify the tools: - -```sh -git --version -docker version -docker compose version -docker run --rm hello-world -``` - -On Linux, add the operator to the Docker group only if local policy permits it; sign out and back -in afterward. A local installation needs no inbound firewall rule because ports bind only to -`127.0.0.1`. - -## Clone and verify LF - -Use a `git clone` command that disables automatic CRLF conversion for this checkout. - -macOS and Linux: - -```sh -git -c core.autocrlf=false clone https://github.example.invalid/your-org/ThothII.git -cd ThothII -git config --local core.autocrlf false -bash scripts/verify-line-endings.sh -``` - -Windows PowerShell: - -```powershell -git -c core.autocrlf=false clone https://github.example.invalid/your-org/ThothII.git -Set-Location ThothII -git config --local core.autocrlf false -& "C:\Program Files\Git\bin\bash.exe" scripts/verify-line-endings.sh -``` - -Windows WSL2: - -```sh -mkdir -p "$HOME/src" && cd "$HOME/src" -git -c core.autocrlf=false clone https://github.example.invalid/your-org/ThothII.git -cd ThothII -git config --local core.autocrlf false -bash scripts/verify-line-endings.sh -``` - -Stop if the verifier names any path. Do not build from a CRLF checkout. - -## Create the local operator files - -Copy the non-secret template. This untracked `.env` contains addresses and absolute source paths, -never secret values: - -```sh -cp deploy/env/local.env.example deploy/env/local.env -mkdir -p /absolute/path/to/thothii-operator/secrets -chmod 0700 /absolute/path/to/thothii-operator/secrets -``` - -Native Windows PowerShell performs the same setup without POSIX utilities. The ACL commands remove -inherited access from the new operator directory and grant full control only to the current Windows -identity. Stop if either `icacls.exe` command returns a nonzero exit code: - -```powershell -$OperatorDir = Join-Path $env:USERPROFILE 'thothii-operator' -$SecretsDir = Join-Path $OperatorDir 'secrets' -$CurrentUser = [System.Security.Principal.WindowsIdentity]::GetCurrent().Name -if (Test-Path $OperatorDir) { throw 'Use a new operator directory or review its ACLs manually.' } -New-Item -ItemType Directory -Force -Path $OperatorDir, $SecretsDir | Out-Null -icacls.exe $OperatorDir /inheritance:r -if ($LASTEXITCODE -ne 0) { throw 'Could not remove inherited operator-directory ACLs.' } -icacls.exe $OperatorDir /grant:r "${CurrentUser}:(OI)(CI)F" -if ($LASTEXITCODE -ne 0) { throw 'Could not grant the current user the operator-directory ACL.' } -Copy-Item deploy/env/local.env.example deploy/env/local.env -Copy-Item docs/install/examples/thothii-installation.local.yaml ` - (Join-Path $OperatorDir 'thothii-installation.yaml') -``` - -Edit `deploy/env/local.env`. At minimum set the workspace Git remote, `PI_AUTH_FILE`, -`THT_SECRETS_FILE`, and external service endpoints. Create the Pi/application and Git transport -files under the protected operator directory and set mode `0600`. On Windows use a user-only ACL -instead. DWH and Evidence credentials are entered later through Workspace management and stored -in the backend's encrypted `workspace-secrets` volume. - -Do not paste credentials into this guide's commands, `.env`, workspace YAML, Git, URLs, image build -arguments, or the installation descriptor. Secret contents are mounted read-only under -`/run/secrets` (Pi's auth store has its own protected read-only mount) and must never be committed, -embedded, rendered, or logged. - -Follow [the local workspace repository guide](local-workspace-registry.md) to choose exactly one -read-only Git SSH/HTTPS override. A fresh install requires a valid private workspace repository; -the remote Git repository remains the source of truth. - -Copy the installation example to an operator-controlled file named exactly -`thothii-installation.yaml`, then replace all placeholders with absolute paths: - -```sh -cp docs/install/examples/thothii-installation.local.yaml \ - /absolute/path/to/thothii-operator/thothii-installation.yaml -``` - -For HTTPS, replace the SSH override in that file with `deploy/compose.git-https.yaml`. Add only -reviewed local overrides. Paths may contain spaces when correctly represented as YAML strings. - -Native Windows uses the same four fields. Use single-quoted absolute Windows paths so backslashes -remain literal YAML characters: - -```yaml -profile: local -projectDirectory: 'C:\Users\operator\src\ThothII' -envFile: 'C:\Users\operator\src\ThothII\deploy\env\local.env' -overrides: - - 'C:\Users\operator\src\ThothII\deploy\compose.git-ssh.yaml' -``` - -## Address external services - -An address is interpreted inside `core`. Therefore container 127.0.0.1 means the container itself, -not the Docker host. Keep external DWH and LLM addresses configurable in the installation; Qdrant -and embedding are internal services in the standard stack. - -- **Docker Desktop (macOS and Windows):** use `host.docker.internal`, for example - `http://host.docker.internal:11434`. -- **Linux:** if a service runs on the host, create an untracked override and include its absolute - path in `thothii-installation.yaml`: - -```yaml -services: - core: - extra_hosts: - - "host.docker.internal:host-gateway" -``` - -Then use `host.docker.internal` in the endpoint. `extra_hosts: host.docker.internal:host-gateway` -is a host routing aid, not a bundled service. Prefer a real DNS name for independently operated -services; retain TLS and authentication even when co-located. - -## Build ThothII and tht - -The canonical local Compose smoke uses the base file plus the local profile. Keep this exact -base+profile command available for install verification: - -~~~sh -docker compose --env-file deploy/env/local.env -f compose.yaml -f deploy/compose.local.yaml up --build -d -~~~ - -After the stack is ready, configure and check authentication with the single host CLI tht; see -the [local authentication guide](authentication-local.md). Authentication configuration is -installation-global and is checked before workspace tests. - -From the repository root, macOS/Linux/WSL2 users run: - -```sh -bash scripts/build-local.sh -bash scripts/build-tht.sh -``` - -Native PowerShell users run: - -```powershell -powershell -ExecutionPolicy Bypass -File scripts/build-local.ps1 -& "C:\Program Files\Git\bin\bash.exe" scripts/build-tht.sh -``` - -The second command uses Docker to create native operator binaries under `dist/tht`; users do -not need to know or install Go. Select `tht-darwin-arm64` or `-amd64` on macOS, -`tht-linux-amd64` or `-arm64` on Linux/WSL2, and `tht-windows-amd64.exe` on Windows. -Copy the selected file to the protected operator directory and, on macOS/Linux, run `chmod 0755` -on it. - -## Start and verify - -Set convenient variables (PowerShell users use `$THT_BIN` and `$INSTALLATION` with `& $THT_BIN`): -Every operator call has the form `tht --installation <absolute-descriptor> <command>`. - -```sh -THT_BIN=/absolute/path/to/thothii-operator/tht -INSTALLATION=/absolute/path/to/thothii-operator/thothii-installation.yaml -"$THT_BIN" --installation "$INSTALLATION" update --check-only -"$THT_BIN" --installation "$INSTALLATION" start -"$THT_BIN" --installation "$INSTALLATION" status -"$THT_BIN" --installation "$INSTALLATION" doctor -``` - -Native PowerShell uses the same order: - -```powershell -$THT_BIN = 'C:\Users\operator\thothii-operator\tht.exe' -$INSTALLATION = 'C:\Users\operator\thothii-operator\thothii-installation.yaml' -& $THT_BIN --installation $INSTALLATION update --check-only -& $THT_BIN --installation $INSTALLATION start -& $THT_BIN --installation $INSTALLATION status -& $THT_BIN --installation $INSTALLATION doctor -``` - -Wait for both services, then check the same-origin frontend and direct loopback core: - -```sh -curl --fail http://127.0.0.1:8080/health -curl --fail http://127.0.0.1:8787/health -"$THT_BIN" --installation "$INSTALLATION" pi doctor -"$THT_BIN" --installation "$INSTALLATION" pi test -``` - -Native PowerShell must call `curl.exe` explicitly; Windows PowerShell may otherwise resolve `curl` -to `Invoke-WebRequest`: - -```powershell -curl.exe --fail --silent --show-error http://127.0.0.1:8080/health -curl.exe --fail --silent --show-error http://127.0.0.1:8787/health -& $THT_BIN --installation $INSTALLATION pi doctor -& $THT_BIN --installation $INSTALLATION pi test -``` - -Open <http://127.0.0.1:8080>. If a check fails, run `tht ... logs` or `pi logs`; these are -bounded and sanitize declared secrets. Do not publish either loopback port. - -## Update an installation - -Commit or back up local operator changes first and finish active sessions. A promoted Pi image is -selected by the durable, installation-specific `current-image.yaml` after every base/profile file. -Therefore rebuilding `thothii-core:local` followed by `update --check-only` does not reconcile a -previous `pi update`: the old promoted core would remain selected. - -Do not delete or edit the selector. `tht status` is the installation-aware selector test. If -the running core image is the base `thothii-core:local` image, no Pi update has promoted a durable -lifecycle image and an ordinary same-Pi-version source rebuild/start is supported. If status shows -a lifecycle image and the pulled Pi pin is unchanged, `pi update` would be a no-op and the procedure -must stop. A changed Pi pin uses transactional `pi update --source build` in either case. - -macOS, Linux, and WSL2: - -```sh -set -euo pipefail - -abort_update() { printf 'Source update stopped: %s\n' "$1" >&2; exit 1; } -require_clean_source() { - local source_state - if ! source_state="$(git status --porcelain --untracked-files=all)"; then - abort_update "git status failed" - fi - [[ -z "$source_state" ]] || abort_update "commit, remove, or back up every tracked/untracked source change" -} - -require_clean_source -if ! git pull --ff-only; then abort_update "git pull --ff-only failed"; fi -require_clean_source -if ! git config --local core.autocrlf false; then abort_update "could not set repository LF policy"; fi -if ! bash scripts/verify-line-endings.sh; then abort_update "the pulled checkout contains CRLF files"; fi -if ! SOURCE_REVISION="$(git rev-parse HEAD)"; then abort_update "could not record the pulled revision"; fi -if ! NEXT_PI_VERSION="$(sed -n 's/^ARG PI_VERSION=//p' docker/core.Dockerfile)"; then - abort_update "could not read the pulled Pi pin" -fi -[[ -n "$NEXT_PI_VERSION" && "$NEXT_PI_VERSION" != *$'\n'* ]] || abort_update "expected one pinned default PI_VERSION" -if ! INSTALLATION_STATUS="$("$THT_BIN" --installation "$INSTALLATION" status)"; then - abort_update "tht status failed" -fi -if ! RUNNING_PI_VERSION="$("$THT_BIN" --installation "$INSTALLATION" pi status)"; then - abort_update "tht pi status failed" -fi -RUNNING_PI_VERSION="${RUNNING_PI_VERSION#Pi version: }" -[[ -n "$RUNNING_PI_VERSION" ]] || abort_update "tht pi status returned no version" - -COMPACT_STATUS="${INSTALLATION_STATUS//[[:space:]]/}" -USES_BASE_CORE=false -if [[ "$COMPACT_STATUS" == *'"Image":"thothii-core:local"'* ]]; then - USES_BASE_CORE=true -fi -TRANSACTIONAL_PI_UPDATE=true -if [[ "$NEXT_PI_VERSION" == "$RUNNING_PI_VERSION" ]]; then - [[ "$USES_BASE_CORE" == true ]] || abort_update "same Pi version is selected by a durable lifecycle image" - TRANSACTIONAL_PI_UPDATE=false -fi - -if ! bash scripts/build-local.sh; then abort_update "the local image build failed"; fi -if ! bash scripts/build-tht.sh; then abort_update "the tht build failed"; fi -if ! "$THT_BIN" --installation "$INSTALLATION" update --check-only; then - abort_update "the installation render check failed" -fi -if [[ "$TRANSACTIONAL_PI_UPDATE" == true ]]; then - if ! "$THT_BIN" --installation "$INSTALLATION" pi update \ - --version "$NEXT_PI_VERSION" --source build --yes --drain; then - abort_update "the transactional core update failed" - fi -fi -if ! "$THT_BIN" --installation "$INSTALLATION" start; then abort_update "installation start failed"; fi -if ! curl --fail http://127.0.0.1:8080/health; then abort_update "frontend health check failed"; fi -if ! curl --fail http://127.0.0.1:8787/health; then abort_update "core health check failed"; fi -if ! FINAL_STATUS="$("$THT_BIN" --installation "$INSTALLATION" status)"; then abort_update "final status failed"; fi -if ! FINAL_PI_STATUS="$("$THT_BIN" --installation "$INSTALLATION" pi status)"; then abort_update "final pi status failed"; fi -[[ "${FINAL_PI_STATUS#Pi version: }" == "$NEXT_PI_VERSION" ]] || abort_update "running Pi version does not match the pulled pin" -if ! "$THT_BIN" --installation "$INSTALLATION" doctor; then abort_update "final doctor failed"; fi -require_clean_source -printf 'Built source revision: %s\n%s\n%s\n' "$SOURCE_REVISION" "$FINAL_STATUS" "$FINAL_PI_STATUS" -``` - -Native Windows PowerShell uses the same fail-closed version comparison and transactional promotion: - -```powershell -$ErrorActionPreference = 'Stop' -function Assert-NativeSuccess([string]$Step) { - if ($LASTEXITCODE -ne 0) { throw "$Step failed with exit code $LASTEXITCODE." } -} -function Assert-CleanSource { - $SourceState = @(git status --porcelain --untracked-files=all) - Assert-NativeSuccess 'git status' - if ($SourceState.Count -ne 0) { - throw 'Commit, remove, or back up every tracked/untracked source change.' - } -} - -Assert-CleanSource -git pull --ff-only -Assert-NativeSuccess 'source pull' -Assert-CleanSource -git config --local core.autocrlf false -Assert-NativeSuccess 'repository LF policy' -& "C:\Program Files\Git\bin\bash.exe" scripts/verify-line-endings.sh -Assert-NativeSuccess 'pulled checkout LF verification' -$SourceRevision = git rev-parse HEAD -Assert-NativeSuccess 'source revision read' -$VersionLine = @(Select-String -Path docker/core.Dockerfile -Pattern '^ARG PI_VERSION=(.+)$') -if ($VersionLine.Count -ne 1) { throw 'Expected exactly one pinned default PI_VERSION.' } -$NextPiVersion = $VersionLine.Matches[0].Groups[1].Value -$InstallationStatus = @(& $THT_BIN --installation $INSTALLATION status) -Assert-NativeSuccess 'installation status' -$RunningPiStatus = (& $THT_BIN --installation $INSTALLATION pi status) -Assert-NativeSuccess 'Pi status' -$RunningPiVersion = $RunningPiStatus -replace '^Pi version:\s*', '' -if ([string]::IsNullOrWhiteSpace($RunningPiVersion)) { throw 'Pi status returned no version.' } -$Services = $InstallationStatus | ConvertFrom-Json -$CoreServices = @($Services | Where-Object { $_.Service -eq 'core' }) -if ($CoreServices.Count -ne 1) { throw 'Installation status did not identify exactly one core service.' } -$UsesBaseCore = $CoreServices[0].Image -eq 'thothii-core:local' -$TransactionalPiUpdate = $true -if ($NextPiVersion -eq $RunningPiVersion) { - if (-not $UsesBaseCore) { throw 'Same Pi version is selected by a durable lifecycle image.' } - $TransactionalPiUpdate = $false -} -powershell -ExecutionPolicy Bypass -File scripts/build-local.ps1 -Assert-NativeSuccess 'local image build' -& "C:\Program Files\Git\bin\bash.exe" scripts/build-tht.sh -Assert-NativeSuccess 'tht build' -& $THT_BIN --installation $INSTALLATION update --check-only -Assert-NativeSuccess 'installation render check' -if ($TransactionalPiUpdate) { - & $THT_BIN --installation $INSTALLATION pi update ` - --version $NextPiVersion --source build --yes --drain - Assert-NativeSuccess 'transactional core update' -} -& $THT_BIN --installation $INSTALLATION start -Assert-NativeSuccess 'installation start' -curl.exe --fail --silent --show-error http://127.0.0.1:8080/health -Assert-NativeSuccess 'frontend health check' -curl.exe --fail --silent --show-error http://127.0.0.1:8787/health -Assert-NativeSuccess 'core health check' -$FinalStatus = @(& $THT_BIN --installation $INSTALLATION status) -Assert-NativeSuccess 'final installation status' -$FinalPiStatus = (& $THT_BIN --installation $INSTALLATION pi status) -Assert-NativeSuccess 'final Pi status' -if (($FinalPiStatus -replace '^Pi version:\s*', '') -ne $NextPiVersion) { - throw 'Running Pi version does not match the pulled pin.' -} -& $THT_BIN --installation $INSTALLATION doctor -Assert-NativeSuccess 'final doctor' -Assert-CleanSource -Write-Output "Built source revision: $SourceRevision" -Write-Output $FinalStatus -Write-Output $FinalPiStatus -``` - -The revision is printed only after every source/build/start/health/installation-aware check passes -and a final porcelain check still reports no tracked or untracked source changes. For a changed Pi -pin, status reports the promoted lifecycle candidate; for a same-version installation with no -selector, status reports the rebuilt base core. `update --check-only` alone proves only that Compose -renders. -Review release notes before updating. See [Pi management](pi-management.md) for rollback; never -install a package in the running container. - -## Back up and restore - -Back up before source/Pi updates and test restoration periodically. First stop cleanly: - -```sh -"$THT_BIN" --installation "$INSTALLATION" stop -docker volume ls --format '{{.Name}}' | grep '^thothii-' -``` - -Identify the four exact volumes belonging to this installation: `settings`, `pi-state`, -`workspace-registry`, and `sessions`. Confirm their Compose project label with `docker volume -inspect`. For each exact volume, archive it to a protected backup directory: - -```sh -BACKUP_DIR=/absolute/path/to/backups/2026-08-05 -VOLUME=exact-installation-volume-name -mkdir -p "$BACKUP_DIR" -docker run --rm -v "$VOLUME:/source:ro" -v "$BACKUP_DIR:/backup" \ - alpine:3.22 tar -C /source -czf "/backup/$VOLUME.tgz" . -``` - -Native PowerShell can run the same read-only archive container: - -```powershell -$BackupDir = 'C:\Users\operator\thothii-backups\2026-08-05' -$Volume = 'exact-installation-volume-name' -New-Item -ItemType Directory -Force $BackupDir | Out-Null -docker run --rm -v "${Volume}:/source:ro" -v "${BackupDir}:/backup" ` - alpine:3.22 tar -C /source -czf "/backup/${Volume}.tgz" . -``` - -Also back up the installation descriptor, operator environment, generated overrides, and secret -files to separate encrypted/protected storage. Never commit them. Record image digests and the Git -revision. Do not back up while containers are running. - -Restore only while stopped and only into a new, verified-empty exact target volume. Test the -archive in a disposable installation first: - -```sh -TARGET_VOLUME=exact-empty-target-volume-name -ARCHIVE=/absolute/path/to/backups/2026-08-05/exact-volume-name.tgz -docker run --rm -v "$TARGET_VOLUME:/target" alpine:3.22 \ - sh -c 'test -z "$(ls -A /target)"' -docker run --rm -v "$TARGET_VOLUME:/target" -v "$(dirname "$ARCHIVE"):/backup:ro" \ - alpine:3.22 tar -C /target -xzf "/backup/$(basename "$ARCHIVE")" -``` - -Native PowerShell uses `Split-Path` to produce the read-only archive mount and archive name: - -```powershell -$TargetVolume = 'exact-empty-target-volume-name' -$Archive = 'C:\Users\operator\thothii-backups\2026-08-05\exact-volume-name.tgz' -$ArchiveDir = Split-Path -Parent $Archive -$ArchiveName = Split-Path -Leaf $Archive -docker run --rm -v "${TargetVolume}:/target" alpine:3.22 ` - sh -ceu 'test -z "$(ls -A /target)"' -if ($LASTEXITCODE -ne 0) { throw 'The restore target volume is not empty.' } -docker run --rm -v "${TargetVolume}:/target" -v "${ArchiveDir}:/backup:ro" ` - alpine:3.22 tar -C /target -xzf "/backup/${ArchiveName}" -if ($LASTEXITCODE -ne 0) { throw 'The volume restore failed.' } -``` - -Restore all four volumes from the same backup set, restore protected operator files separately, -then run `update --check-only`, `start`, `doctor`, registry status/diagnostics, and a known session -before normal use. Never merge an archive into a non-empty volume. - -## Data-preserving uninstall - -Run `tht stop`, retain the installation descriptor at the same absolute path, and make one -verified backup set. In Docker Desktop, remove only this installation's stopped `core` and -`frontend` containers and optional local images; leave its four named volumes. On Linux, use the -containers' exact Compose project labels to remove only those stopped containers. Do not prune -global Docker data. - -Do **not** run `docker compose down --volumes`: it deletes the application data this procedure is -meant to preserve. Keep the operator directory and protected secrets if you intend to reinstall. -Using the same descriptor path preserves the `tht` project identity and reconnects the same -named volumes after rebuilding the source checkout. - -## Next: workspaces and Pi - -Complete [local workspace-registry installation](local-workspace-registry.md), including Git trust, -bindings, pull, validation, diagnostics, and registry recovery. Then use [Pi management](pi-management.md) -for provider/model configuration, smoke testing, transactional update, and rollback. - -The Git-backed workspace registry is always the workspace source of truth. Local DWH, vector, -embedding, or LLM processes remain independent services and are never added to the mandatory -ThothII core. diff --git a/docs/install/pi-management.md b/docs/install/pi-management.md deleted file mode 100644 index 84ae9c44..00000000 --- a/docs/install/pi-management.md +++ /dev/null @@ -1,162 +0,0 @@ -# Pi management - -ThothII bundles Pi in the `core` image. Operators use the Pi Management page for safe application -defaults and the host-side `tht` CLI for lifecycle work. A local Pi installation is not -required. - -Run these commands from the root of the current ThothII checkout or worktree. `tht` discovers -the valid installation descriptor in that project tree, so it uses the `deploy/` files belonging to -the checkout from which you run it. Do not use `~/bin`: `~` is the user home directory, not the -project root. - -```sh -THT_BIN=tht -tht version --json -``` - -If `tht` is not on `PATH`, install the native host CLI using the installation procedure in -`local.md` or `server.md`, then set `THT_BIN` to that installed binary. For an installation -stored elsewhere, set `THOTHII_INSTALLATION` or pass -`--installation <absolute-path>/thothii-installation.yaml` explicitly. - -## Choose application defaults - -Use the **Pi Management** page to select the supported provider, model, and reasoning default, then -choose **Save defaults**. The page shows credentials only as present or missing and can run bounded -diagnostics; it never accepts or displays a credential, opens a terminal, or updates an image. - -Alternatively, use the CLI from an administrator terminal: - -```sh -"$THT_BIN" pi configure -"$THT_BIN" pi configure --provider zai --model glm-5.2 --thinking medium -``` - -Use GUI Save defaults or CLI `pi configure`, not both for the same change. The CLI's interactive -choices are restricted to supported models; non-interactive use must supply all three values. Both -methods store application defaults in backend installation settings, not in the project policy file. - -Useful read-only checks are: - -```sh -"$THT_BIN" pi status -"$THT_BIN" pi doctor -"$THT_BIN" pi test -"$THT_BIN" pi check -"$THT_BIN" pi logs -``` - -`pi check` is an alias for `pi test`; logs are a sanitized, bounded snapshot with no follow mode. - -## Edit the provider catalog and enabled-model policy - -Edit these project-root files in source control, then review and deploy the change through the -normal project process: - -- `deploy/pi/models.json` is the provider catalog: provider endpoints and the models each provider - offers. -- `deploy/pi/settings.json` is the enabled-model policy only; it lists the models available to the - application and does not store application defaults. - -These files contain configuration, not credentials. Keep provider configuration declarative: Pi -management rejects executable `!command` values. Docker Compose mounts the selected configuration -and credential files read-only. - -## Store provider credentials - -`PI_AUTH_FILE` is a setting in the installation environment file (for example, -`deploy/env/local.env`). Its value is the absolute path of the protected host credential file that -this installation selects. Docker Compose mounts that selected file read-only for Pi. -Other declared protected material is likewise mounted read-only under `/run/secrets`. - -Set restrictive permissions on the host file (`0600` on macOS/Linux or a user-only ACL on Windows). -Never put its contents in installation YAML, Git, command arguments, the browser, screenshots, -tickets, rendered Compose output, or logs. Do not print the file while troubleshooting. - -## Reload changed configuration - -After changing the provider catalog, enabled-model policy, or selected credential file, reload the -running application with one confirmed restart: - -```sh -"$THT_BIN" pi restart --yes --drain -``` - -`--yes` confirms that core will be recreated. Without `--drain`, restart refuses active sessions; -with it, ThothII closes admission and waits for active sessions to finish without terminating them. -The wait is bounded. Restart retains the exact captured running image: it does not build, pull, or -upgrade an image. Before recreating core, it pins that image through transaction-scoped Compose -override material so a configured tag moving during the operation cannot change the selected -image, and Compose is explicitly told never to build or pull. It will restart only core, then -verifies health, Pi version, settings/model smoke, non-secret rendered configuration, and -persistence mounts before reopening admission. - -Use `pi restart --yes` when there are already no active sessions. Do not substitute `tht stop` -and `tht start` or raw Compose commands for this reload workflow. - -## Update the bundled Pi version - -`pi update` is for a new bundled Pi version; it is not a configuration reload. The simple command -uses the single `ARG PI_VERSION=...` pin in `docker/core.Dockerfile`, builds that version, waits for -active sessions to finish, and recreates only `core`: - -```sh -"$THT_BIN" pi update -``` - -To build a specific version, pass `--version`; source, confirmation, and drain are automatic for -this normal build path: - -```sh -"$THT_BIN" pi update --version 0.81.0 -``` - -A registry update must use an immutable digest, never a mutable tag: - -```sh -"$THT_BIN" pi update \ - --version 0.81.0 --source pull \ - --image registry.example.invalid/thothii-core@sha256:<64-lowercase-hex-digits> \ - --yes --drain -``` - -Update keeps new-session admission gated while it builds or pulls a candidate, recreates only -`core`, verifies it, and promotes the image only after success. It preserves the frontend and named -volumes. - -## Recover a failed lifecycle operation - -If a restart or update fails after core recreation, leave maintenance enabled and preserve the -reported recovery state and transaction override. Do not delete `.tht`, state files, -containers, or volumes. Inspect status and sanitized logs: - -```sh -"$THT_BIN" pi maintenance status -"$THT_BIN" pi status -"$THT_BIN" pi logs -``` - -For a failed update, restore its prior image: - -```sh -"$THT_BIN" pi rollback --yes -``` - -For a failed restart, use maintenance recovery instead of rollback. After repairing the reported -Docker, disk, or configuration problem, use the same command to complete either safe recovery path: - -```sh -"$THT_BIN" pi maintenance recover --yes -"$THT_BIN" pi doctor -"$THT_BIN" pi test -``` - -`pi rollback --yes` restores the prior update image. `pi maintenance recover --yes` checks both -restart and update recovery state before it can reopen admission. If either command fails, keep the -installation gated and collect only the sanitized diagnostics. - -## Direct support access - -Raw Compose access is unsupported because it can bypass the installation-specific environment and -durable image selector. For support, use the installation-aware `tht pi status`, -`tht pi doctor`, `tht pi test`, and `tht pi logs` commands. diff --git a/docs/install/psd-workspace-setup.md b/docs/install/psd-workspace-setup.md deleted file mode 100644 index 7d730139..00000000 --- a/docs/install/psd-workspace-setup.md +++ /dev/null @@ -1,65 +0,0 @@ -# Policlinico San Donato — setup workspace (nuova gestione) - -Authentication acceptance is documented in the [manual authentication matrix](../testing/authentication-manual-acceptance.md). -Use generic OIDC with Authentik as the certified group catalog, map only the exact TOT Users and -TOT Admin groups, then run **Validate workspace source**, `tht auth check`, `tht auth check --interactive`, -and **Test workspace connections** in that order. Browser callback E2E, native Windows execution, approved PSD -manual identities, external L2, and the two parked restore-lock preconditions remain pending the -Task 15/release gates. - -Guida operativa per collegare ThothII al DWH di PSD con il nuovo sistema (registry Git + descriptor -v3 + `tht`). - -## Stato storico Mac/local (2026-08-13) - -> Questo stato è storico per Mac/local; il server PSD Project A usa binding separato `postgres_direct` read-only. -> -> Per la rotazione della credenziale DWH, fare riferimento al [runbook PSD](../operations/psd-dwh-auth-rollout.md): non autorizza modifiche finché i due gate non sono approvati. Il ThothII PSD server resta `postgres_direct`; il Mac e i client remoti usano `rest_api` con una chiave per installazione. `postgres_direct` e `ssh_tunnel` non usano chiavi `dwh-auth`. - -- **Repository PSD pubblicato:** `https://github.com/mptyl/tht-workspace-psd` (privato), branch - `main`, commit `d4f9185`. Layout P1.1 già migrato e validato. -- **Deploy key SSH** (sola lettura, senza passphrase) in - `deploy/psd/secrets/git-ssh-key` e registrata sul repo come deploy key `thothii-psd`; il remote - Git usato dall'installazione è `git@github.com:mptyl/tht-workspace-psd.git`. -- **Config operatore pronta** (file reali gitignored in `deploy/psd/`): `operator.env`, - `thothii-installation.yaml` e i secret d'installazione in `secrets/` (pi-auth, secret bundle, - chiave SSH, known_hosts). L'API key DWH va completata nella gestione Workspace ed è conservata - nel vault cifrato del backend. Il certificato REST è self-issued/private: ogni Mac/local senza trust equivalente deve usare `TLS_CA_FILE` e verificare il fingerprint fuori banda, come in `docs/install/dwh-auth-tls.md`. -- **Stack avviato** (progetto `thothii-70417a3e30ea`, via `tht start`): `qdrant`, `embedding` - (con `qwen3-embedding:0.6b`), `core`, `frontend` sani. Il registry ha **clonato e attivato** - `psd-clinical` (stato `ready`). -- **`tht workspace inspect --workspace psd-clinical` = OK** (identità descrittore/catalogo - risolte); la configurazione runtime va completata e testata dalla GUI. -- **Bloccante residuo: VPN.** `supabase-aritmolab.policlinicosandonato.it` non risolve - (`NXDOMAIN`) → il preprocessing DWH e le sessioni live non possono ancora partire. - -## Avvio/arresto (canonico) - -Usare `tht` (stesso project name, quindi stessi volumi named): - -```bash -tht=dist/tht/tht-darwin-arm64 -"$tht" --installation "$(pwd)/deploy/psd/thothii-installation.yaml" start -"$tht" --installation "$(pwd)/deploy/psd/thothii-installation.yaml" workspace inspect --workspace psd-clinical --json -"$tht" --installation "$(pwd)/deploy/psd/thothii-installation.yaml" stop -``` - -> **Nota project name:** `tht` calcola un project name stabile dall'installation descriptor -> (`thothii-<hash>`); `docker compose` "a mano" usa invece `name: thothii` dal `compose.yaml`, quindi -> i volumi named non coinciderebbero. Perciò per lo stack si usa `tht start` (non -> `compose-with-preflight.sh up`). - -## Rimane: smoke live di una domanda (P8 L2) - -Il preprocessing è già completato. Resta solo: - -1. Aprire `http://localhost:8080` e selezionare `psd-clinical`. -2. Creare una sessione con una domanda reale in linguaggio naturale. -3. Seguire le 8 fasi fino al primo gate di revisione. - -## Cosa è già stato fatto - -- Ristrutturazione del repo PSD nel layout P1.1 + validazione locale. -- Pubblicazione GitHub + deploy key read-only + configurazione Git d'installazione. -- Avvio stack + attivazione registry + `tht inspect` verde. -- **Preprocessing live completato** su PSD: DWH → FK → schema → Evidence, idempotente. diff --git a/docs/install/reverse-proxy-caddy.md b/docs/install/reverse-proxy-caddy.md deleted file mode 100644 index ed675fa8..00000000 --- a/docs/install/reverse-proxy-caddy.md +++ /dev/null @@ -1,98 +0,0 @@ -# Put ThothII behind Caddy - -Choose exactly one authentication mode. Direct ThothII-managed OIDC and deprecated upstream -authentication are mutually exclusive proxy contracts; never combine their directives. - -## Direct ThothII-managed OIDC - -Use this mode when `auth.yaml` has `mode: oidc`. Caddy terminates TLS and proxies every request to -`frontend`; ThothII performs login, callback validation, session creation, and authorization. -Caddy must not apply `forward_auth` or another external authentication gateway. - -The public `/api/auth/oidc/login` and `/api/auth/oidc/callback` paths pass unchanged through the -same proxy as the rest of `/api`. The configured `publicUrl` must match the browser origin. - -```caddyfile -thoth.example.invalid { - reverse_proxy 127.0.0.1:8080 { - # No URI rewrite: OIDC login and callback paths reach frontend unchanged. - flush_interval -1 - header_up Host {host} - header_up X-Forwarded-Proto https - header_up X-Forwarded-Host {host} - } - - log { - output file /var/log/caddy/thoth-access.log - format json - } -} -``` - -After reload, run Workspace Validate for static validation, `tht auth check` for live, -non-interactive authentication diagnosis, and then Workspace Test for aggregate live validation. - -## Deprecated upstream migration mode - -Use this section only while the installation explicitly uses deprecated `upstream` mode. Do not -use it with `mode: oidc` or `mode: local`. Here an external authentication gateway owns login and -Caddy applies `forward_auth` before forwarding normalized private identity headers to `frontend`. - -Forwarding identity headers alone does not authenticate a user. The authentication gateway returns -2xx only after validating its own credential or session. Clear browser-supplied public and trusted -headers before the subrequest, and map identity only from the successful auth response. - -```caddyfile -thoth.example.invalid { - route { - request_header -X-Authenticated-User - request_header -X-Thoth-Principal-Issuer - request_header -X-Thoth-Principal-Subject - request_header -X-Thoth-Principal-Display-Name - request_header -X-Thoth-Is-Admin - request_header -X-Thoth-Trusted-Principal-Issuer - request_header -X-Thoth-Trusted-Principal-Subject - request_header -X-Thoth-Trusted-Principal-Display-Name - request_header -X-Thoth-Trusted-Is-Admin - - forward_auth auth-gateway:4180 { - uri /verify - copy_headers { - X-Thoth-Principal-Issuer>X-Thoth-Trusted-Principal-Issuer - X-Thoth-Principal-Subject>X-Thoth-Trusted-Principal-Subject - X-Thoth-Principal-Display-Name>X-Thoth-Trusted-Principal-Display-Name - X-Thoth-Is-Admin>X-Thoth-Trusted-Is-Admin - } - } - - reverse_proxy 127.0.0.1:8080 { - flush_interval -1 - header_up Host {host} - header_up X-Forwarded-Proto https - } - } -} -``` - -## Trust boundary - -Caddy is the only public listener and proxies only to loopback `frontend`, never directly to -`core`. Configure access logs to omit cookies, authorization data, query strings, and identity -headers. Keep Caddy keys and state outside ThothII source and operator directories. - -## Validate and reload - -Keep the public firewall closed while validating: - -```sh -curl --fail http://127.0.0.1:8080/health -caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile -sudo systemctl reload caddy -``` - -## Test authentication and SSE - -For direct OIDC, verify the login path redirects to the configured provider, the callback reaches -ThothII unchanged, forged identity headers grant nothing, and SSE is unbuffered. For deprecated -upstream mode, additionally verify the external gateway rejects unauthenticated traffic and only -its 2xx response can create trusted identity headers. diff --git a/docs/install/reverse-proxy-nginx.md b/docs/install/reverse-proxy-nginx.md deleted file mode 100644 index 2277112a..00000000 --- a/docs/install/reverse-proxy-nginx.md +++ /dev/null @@ -1,143 +0,0 @@ -# Put ThothII behind Nginx - -Choose exactly one authentication mode. Direct ThothII-managed OIDC and deprecated upstream -authentication are mutually exclusive proxy contracts; never combine their locations or headers. - -## Direct ThothII-managed OIDC - -Use this mode when `auth.yaml` has `mode: oidc`. Nginx terminates TLS and proxies every request -to loopback `frontend`. ThothII owns OIDC login, callback validation, browser sessions, and -authorization. No external `auth_request` or authentication gateway belongs in this server. - -The `location /` block below has a `proxy_pass` without a replacement URI, so public -`/api/auth/oidc/login` and `/api/auth/oidc/callback` are forwarded unchanged. The configured -`publicUrl` must match the browser origin. - -```nginx -server { - listen 80; - server_name thoth.example.invalid; - return 301 https://$host$request_uri; -} - -server { - listen 443 ssl; - server_name thoth.example.invalid; - - ssl_certificate /etc/nginx/tls/thoth/fullchain.pem; - ssl_certificate_key /etc/nginx/tls/thoth/privkey.pem; - ssl_protocols TLSv1.2 TLSv1.3; - - location / { - # No auth_request and no URI rewrite: ThothII receives OIDC paths unchanged. - proxy_set_header Host $host; - proxy_set_header X-Forwarded-Proto https; - proxy_set_header X-Forwarded-Host $host; - proxy_set_header X-Forwarded-For $remote_addr; - proxy_set_header Connection ""; - proxy_pass http://127.0.0.1:8080; - proxy_http_version 1.1; - proxy_buffering off; - proxy_cache off; - proxy_read_timeout 3600s; - add_header X-Accel-Buffering no always; - } -} -``` - -After reload, run Workspace Validate for static validation, `tht auth check` for live, -non-interactive authentication diagnosis, and then Workspace Test for aggregate live validation. - -## Deprecated upstream migration mode - -Use this section only while the installation explicitly uses deprecated `upstream` mode. Do not -use it with `mode: oidc` or `mode: local`. In this mode an external authentication gateway owns -login, and Nginx applies `auth_request` before forwarding normalized private identity headers. - -Forwarding identity headers alone does not authenticate a user. The authentication gateway returns -2xx only after validating its own credential or session. - -```nginx -server { - listen 443 ssl; - server_name thoth.example.invalid; - - ssl_certificate /etc/nginx/tls/thoth/fullchain.pem; - ssl_certificate_key /etc/nginx/tls/thoth/privkey.pem; - ssl_protocols TLSv1.2 TLSv1.3; - - location = /_authenticate { - internal; - proxy_pass http://auth-gateway:4180/verify; - proxy_pass_request_body off; - proxy_set_header Content-Length ""; - proxy_set_header X-Original-URI $request_uri; - proxy_set_header X-Original-Method $request_method; - proxy_set_header X-Authenticated-User ""; - proxy_set_header X-Thoth-Principal-Issuer ""; - proxy_set_header X-Thoth-Principal-Subject ""; - proxy_set_header X-Thoth-Principal-Display-Name ""; - proxy_set_header X-Thoth-Is-Admin ""; - proxy_set_header X-Thoth-Trusted-Principal-Issuer ""; - proxy_set_header X-Thoth-Trusted-Principal-Subject ""; - proxy_set_header X-Thoth-Trusted-Principal-Display-Name ""; - proxy_set_header X-Thoth-Trusted-Is-Admin ""; - } - - location / { - auth_request /_authenticate; - auth_request_set $thoth_principal_issuer - $upstream_http_x_thoth_principal_issuer; - auth_request_set $thoth_principal_subject - $upstream_http_x_thoth_principal_subject; - auth_request_set $thoth_principal_display_name - $upstream_http_x_thoth_principal_display_name; - auth_request_set $thoth_is_admin - $upstream_http_x_thoth_is_admin; - - proxy_set_header X-Authenticated-User ""; - proxy_set_header X-Thoth-Principal-Issuer ""; - proxy_set_header X-Thoth-Principal-Subject ""; - proxy_set_header X-Thoth-Principal-Display-Name ""; - proxy_set_header X-Thoth-Is-Admin ""; - proxy_set_header X-Thoth-Trusted-Principal-Issuer $thoth_principal_issuer; - proxy_set_header X-Thoth-Trusted-Principal-Subject $thoth_principal_subject; - proxy_set_header X-Thoth-Trusted-Principal-Display-Name $thoth_principal_display_name; - proxy_set_header X-Thoth-Trusted-Is-Admin $thoth_is_admin; - proxy_set_header Host $host; - proxy_set_header X-Forwarded-Proto https; - proxy_set_header X-Forwarded-Host $host; - proxy_set_header X-Forwarded-For $remote_addr; - proxy_set_header Connection ""; - proxy_pass http://127.0.0.1:8080; - proxy_http_version 1.1; - proxy_buffering off; - proxy_cache off; - proxy_read_timeout 3600s; - add_header X-Accel-Buffering no always; - } -} -``` - -## Trust boundary - -Nginx is the only public listener and proxies only to loopback `frontend`, never directly to -`core`. Keep private keys outside the ThothII tree. Do not log cookies, authorization headers, -OIDC callback query values, authentication bodies, or trusted identity headers. - -## Validate and reload - -Keep the public firewall closed while validating: - -```sh -curl --fail http://127.0.0.1:8080/health -sudo nginx -t -sudo systemctl reload nginx -``` - -## Test authentication and SSE - -For direct OIDC, verify login redirects to the configured provider, callback traffic reaches -ThothII unchanged, forged identity headers grant nothing, and SSE is unbuffered. For deprecated -upstream mode, additionally verify the external gateway rejects unauthenticated traffic and only -its 2xx response can create trusted identity headers. diff --git a/docs/install/server-workspace-registry.md b/docs/install/server-workspace-registry.md deleted file mode 100644 index 1b2ef6a6..00000000 --- a/docs/install/server-workspace-registry.md +++ /dev/null @@ -1,132 +0,0 @@ -# Server workspace repository installation - -This manual supplements [server.md](server.md). A server installation reads one remote Git -repository hosted by GitHub, GitLab, Gitea, Bitbucket, or another Git server. ThothII fetches and -validates complete revisions but never edits, commits, pushes, or publishes workspace source. - -## Architecture ownership contract - -| Component | Ownership | Operator contract | -| --- | --- | --- | -| DWH | External | Configure the external endpoint and complete runtime credentials through the authenticated GUI. | -| LLM | External | Configure the external endpoint and model policy under installation control. | -| Qdrant | Internal | Compose runs private Qdrant and persists `qdrant-data`; include it in Qdrant backup/restore. | -| Ollama embedding | Internal | Compose runs private Ollama with `qwen3-embedding:0.6b`. | - -## Semantic index ownership contract - -| Scope | Ownership rule | Isolation rule | -| --- | --- | --- | -| Workspace semantic index | Each workspace keeps exactly one Qdrant collection reserved for itself. | Schema, Evidence, and memory records share that one collection and are separated by the `kind` payload. | - -The fixed semantic contract is 1024 dimensions and cosine distance. DWH and LLM remain external; -Qdrant, Ollama, and `embedding-model-init` remain private internal services. - -## Service account, storage, and firewall - -Run the application as the documented unprivileged service account. Keep the source checkout, -operator files, application data, and workspace authoring clone separate: - -```text -/srv/thothii/app/ # ThothII source release -/srv/thothii/operator/ # installation descriptor and protected Git files -/srv/thothii/data/ # application data, encrypted workspace vault, sessions -/srv/workspace-authoring/ # optional curator clone; never mounted into ThothII -``` - -Expose only the authenticated same-origin reverse proxy. Keep `core`, Qdrant, and Ollama private. - -## Prepare and publish a workspace source - -Create a local workspace in the external authoring repository, which contains -`thoth-workspaces.yaml`, one -`<workspace-id>/workspace.yaml` per catalog entry, optional repository-owned Evidence, and optional -curated schema annotations. It contains no credentials. - -Publishing belongs to the curator workflow outside ThothII: validate, review, commit, and push the -source revision to the configured protected branch. Grant the ThothII service only read access. - -<!-- workspace-descriptor-contract:start --> -Schema v3 is the only accepted workspace descriptor. -Schema v1 and v2 workspace descriptors are rejected before activation. -<!-- workspace-descriptor-contract:end --> - -## Configure the remote Git repository - -Copy `docs/install/examples/thothii-installation.server.yaml` to -`/srv/thothii/operator/thothii-installation.yaml`. Set `workspaceRepository.remote`, `.branch`, and -`.access`, then select exactly one Git transport override. The remote and credential are normally -repository-scoped read-only deploy credentials. - -For SSH, mount a private key and pinned known-hosts file. For HTTPS, mount a Git credentials file -and the required CA chain. These installation credentials are not editable in Workspace -management and are never exposed by the API. - -## Start and update the installation - -Use the installation-aware controller described by `server.md`: - -```bash -THT_BIN=/srv/thothii/operator/tht -INSTALLATION=/srv/thothii/operator/thothii-installation.yaml -"$THT_BIN" --installation "$INSTALLATION" start -"$THT_BIN" --installation "$INSTALLATION" doctor -``` - -The descriptor composes `compose.yaml`, `deploy/compose.server.yaml`, the server session-storage -override, and one read-only Git transport override. **Update workspace repository** fetches a -candidate on the server; it does not transfer workspace files to the operator workstation. - -## Complete runtime secrets in Workspace management - -### Chiavi DWH REST per installazione - -Un'installazione server che seleziona `rest_api` usa una chiave DWH nel vault cifrato o nel file `API_KEY_FILE`; `postgres_direct` e `ssh_tunnel` non usano chiavi `dwh-auth`. Il servizio `dwh-auth` del DWH ha lifecycle `systemd` separato e non appartiene al Compose di ThothII. Vedere [enrollment client](dwh-auth-client-enrollment.md) e [guida server DWH](dwh-auth-server.md). - -After repository activation, an authenticated user can: - -1. Review the configured repository identity and update it without selecting a workspace. -2. Select a workspace to see the DWH/Evidence credential fields required by its connector modes. -3. Blind-save or rotate values with **Save entered secrets**; returned responses contain status only. -4. Run **Validate workspace source** and then **Test workspace connections**. -5. Use **Forget stored value** for an obsolete value after dependent sessions and jobs have ended. - -The backend encrypts values in `/data/workspace-secrets`, including the installation-specific -master key. The server profile persists that directory inside `THT_DATA_ROOT`; no workspace YAML -path depends on Linux, macOS, or Windows. Plaintext exists only in a restrictive temporary file -for the duration of a diagnostic, session, or maintenance lease. - -Authorization is intentionally the current installation-wide authenticated-user policy. A future -role model or external secret manager can replace that policy without changing workspace source. - -## Validation and activation behavior - -Repository update is all-or-nothing: ThothII fetches the configured branch, validates catalog, -descriptors, Evidence paths, and cross-workspace invariants at one commit, then atomically activates -the complete candidate. A rejected candidate never replaces the previous active snapshot. The -application-owned checkout and snapshots are read-only runtime state. - -Validation proves descriptor and repository structure. **Test workspace connections** additionally -materializes the current runtime secrets and contacts only the selected workspace's configured -DWH/Evidence endpoints. Failure does not modify or publish workspace source. - -## Backup, rotation, and recovery - -Back up application data and Qdrant consistently. Qdrant backup/restore must cover `qdrant-data`; -application recovery must cover repository snapshots/state, sessions, settings, Pi state, and the -entire encrypted `/data/workspace-secrets` directory. Store backup encryption keys separately and -test restore procedures without production traffic. - -Rotate DWH/Evidence credentials through Workspace management. Rotate Git access by atomically -replacing its protected installation file and restarting `core`. Recover a bad source revision by -reverting or correcting it in the external authoring repository and updating again. - -## Troubleshooting - -| Symptom | Meaning and action | -| --- | --- | -| Git authentication failed | Verify repository-scoped read permission, branch, key/token, CA, and host-key pinning. | -| Candidate validation failed | Correct the source repository; the prior active commit remains in service. | -| Runtime configuration required | Select the workspace and complete all required write-only fields. | -| Secret store unavailable | Stop writes, preserve `/data/workspace-secrets`, and restore vault plus master key together. | -| Connection test failed | Rotate the indicated runtime credential or correct the relevant non-secret endpoint. | diff --git a/docs/install/server.md b/docs/install/server.md deleted file mode 100644 index ca1e67ad..00000000 --- a/docs/install/server.md +++ /dev/null @@ -1,568 +0,0 @@ -# Install ThothII on a Linux server - -Server authentication uses generic OIDC with the reverse proxy preserving the configured public -origin and callback path. Follow the [OIDC guide](authentication-oidc.md), [Authentik guide](authentik.md) -when applicable, and the [authentication acceptance matrix](../testing/authentication-manual-acceptance.md). -The host authentication CLI is `tht`. Use `tht --installation <descriptor> workspace inspect ---workspace <id> --json` for the active workspace snapshot, `tht --installation <descriptor> -auth check` for live non-interactive authentication diagnosis, `auth check --interactive` for -device-flow identity validation, and `tht ... doctor --json` for the aggregate installation gate. - -This guide is for an installer with basic Linux administration and very basic Docker knowledge. -It deploys the same Compose distribution used on a local PC: the mandatory application is exactly -`frontend` plus `core`, and pinned Pi is inside `core`. The server does not need host Pi, Node.js, -Python, Go, a browser shell, or a Docker socket inside either container. - -Examples use `/srv/thothii` as an **example operator root**, `thoth.example.com` as a replaceable -DNS name, and systemd-based command names. Adapt them to local policy. Complete the server session -storage overlay and migration procedure before exposing a production installation. - -## Deployment contract - -- The generic Linux host and Docker Compose v2 are the deployment platform. No other - application's Compose project, network, path, or runtime is required. -- `frontend` is the only published service and defaults to `127.0.0.1:8080`; `core` has no host - port. A host Nginx or Caddy listener terminates TLS and sends all application traffic to - `frontend`, never directly to `core`. -- DWH, vector database, embedding service, and LLM are external configurable endpoints. This - remains true when they happen to run on the same physical server. -- Application, Git, connector, and session credentials are protected host files mounted read-only - under `/run/secrets`. Pi's protected auth JSON uses its dedicated read-only Pi mount. No secret - value belongs in Git, images, browser storage, environment values, rendered Compose, or logs. -- The Git-backed workspace registry is the source of truth. Installation-local bindings identify - endpoints and secret-file paths; they do not replace the reviewed Git workspace descriptors. -- `tht` is the operator CLI for start, stop, status, health, logs, Pi lifecycle, drain, and - rollback. Raw Compose lifecycle commands bypass installation state and are unsupported. - -Read [server workspace-registry installation](server-workspace-registry.md), -[Pi management](pi-management.md), and the session-server comments in -`deploy/compose.session-server.yaml.example` before the first public start. - -## Service account and directories - -The container runtime identity is fixed at UID/GID 10001. It does not require or permit creation of -a matching host account or group. Keep the number unmapped and use numeric ownership only for the -dedicated bind paths that the non-root container must read or write. If either lookup below finds a -host identity, stop and design an explicit remapping before installation. - -```sh -if getent passwd 10001 >/dev/null || getent group 10001 >/dev/null; then - printf '%s\n' 'UID/GID 10001 is already mapped; stop' >&2 - exit 1 -fi -operator_uid="$(id -u)" -operator_gid="$(id -g)" -test "$operator_uid" -ne 0 -id -nG | tr ' ' '\n' | grep -Fx docker >/dev/null -``` - -The invoking, pre-existing administrator owns source and operator files. It must already have the -site-approved Docker access required to run `tht`; this guide never changes group membership. -Docker-group membership is effectively host-root access and must remain limited to reviewed -administrators. Do not grant the operator direct write access to container runtime trees. - -Create explicit directories. `source` contains the clone; `operator` contains untracked path-only -configuration; the three writable trees are bind-mounted into `core`; `secrets` contains regular -files only. The parent is owned by the operator with numeric group 10001 so both the operator and -container can traverse it. Numeric ownership does not add entries to `/etc/passwd` or `/etc/group`. - -```sh -operator_uid="$(id -u)" -operator_gid="$(id -g)" -sudo install -d -o "$operator_uid" -g 10001 -m 0750 /srv/thothii -sudo install -d -o "$operator_uid" -g "$operator_gid" -m 0750 /srv/thothii/source -sudo install -d -o "$operator_uid" -g "$operator_gid" -m 0750 /srv/thothii/operator -sudo install -d -o 10001 -g "$operator_gid" -m 0750 /srv/thothii/secrets -sudo install -d -o 10001 -g 10001 -m 0750 /srv/thothii/data -sudo install -d -o 10001 -g 10001 -m 0750 /srv/thothii/pi-state -sudo install -d -o 10001 -g 10001 -m 0750 /srv/thothii/workspace-registry -sudo install -d -o root -g root -m 0700 /srv/thothii-backups -stat -c '%u:%g %a %n' \ - /srv/thothii /srv/thothii/source /srv/thothii/operator /srv/thothii/secrets \ - /srv/thothii/data /srv/thothii/pi-state /srv/thothii/workspace-registry \ - /srv/thothii-backups -``` - -Expected: the parent is `operator_uid:10001 750`; source/operator are -`operator_uid:operator_gid 750`; secrets are `10001:operator_gid 750`; the three runtime trees are -`10001:10001 750`; backups are `0:0 700`. Re-run the empty `getent` checks after creation. Do not -make `/srv/thothii` a shared application directory. - -## Projected server authentication: canonical root and runtime projection - -For a server descriptor that declares `authentication.runtimeProjection`, authentication has two -different roots. The **canonical authentication root** (`authentication.configDirectory`) is the -root-operated source of truth. It and its regular files are `root:root 0700/0600`. The **runtime -projection** is a separate Linux-only tree for the container reader: its root, `generations`, and -generation directories are `10001:10001 0700`; `CURRENT`, `manifest.json`, `auth.yaml`, and (for -local mode) `users.yaml` are `10001:10001 0600`. The publisher assigns the numeric IDs directly; -it does not create a host user or group for 10001. - -The runtime projection has only `CURRENT` and `generations/<64-lowercase-hex>/`. `CURRENT` selects -one complete immutable generation. A successful configure, user mutation, restore, or explicit -publish first blocks `CURRENT`, then verifies a new immutable generation, then makes it ready. -The selected generation and up to two predecessor generations are retained; no operator edits a -generation or `CURRENT` directly. A ready projection is usable only when its canonical revision is -equal to the current canonical authentication root. If the runtime projection is blocked, missing, -tampered, or unequal, `start`, `update --check-only`, `auth check`, and `doctor` fail closed before -admission or Compose lifecycle work. - -The runtime directory must be an absolute canonical path, distinct from the canonical root, and -must exactly equal `THT_AUTH_RUNTIME_ROOT` in the protected installation environment. The server -profile and numeric UID/GID values are validated before any projected mutation or publication. - -The descriptor loader adds `compose.auth-runtime-projection.yaml` automatically when -`runtimeProjection` is present; do not list that file under `overrides`. The automatic override -mounts the runtime projection **read-only and core-only** at `/run/thothii-auth`; the canonical -authentication root is never mounted. No other service receives that mount or -`THT_AUTH_RUNTIME_PROJECTION_ROOT`. The example descriptor uses -`/srv/example/thothii/auth-runtime` only as a replaceable path and contains no credential value. - -This source change is prepared and tested only: Project A has not been started. It does not -authorize a raw Compose lifecycle launch, an Nginx change, legacy-stack change, or mutation of -`/srv`. A later manual gate needs separate explicit authorization before applying any descriptor -or runtime root to a server. - -### Status, repair, and safe evidence - -Use the root-operated installation command; retain only its small redacted JSON result: - -```sh -sudo tht --installation "$INSTALLATION" auth status --json -``` - -`state: "ready"` and `equal: true` are required before a projected server can start. `state: -"blocked"`, `equal: false`, or a command refusal means that the runtime projection is blocked or -cannot be validated. Do not start the stack, inspect YAML, print an environment, or edit `CURRENT` -or a generation. Confirm the protected canonical root is available, then republish it with: - -```sh -sudo tht --installation "$INSTALLATION" auth publish -sudo tht --installation "$INSTALLATION" auth status --json -``` - -`auth publish` reconstructs the selected immutable generation from the canonical root; it never -uses an older runtime generation as authority. If publish fails, leave the projection blocked and -escalate using the sanitized command result plus descriptor path and timestamp only. Do not attach -passwords, hashes, YAML, raw environment output, `nginx -T`, or a secret-bearing diff to evidence. - -### Authentication restore - -An authentication-bearing restore first publishes a blocked selector, restores canonical -authentication, and publishes a verified candidate generation before any restart. If candidate or -recovery verification fails, the verified recovery checkpoint is republished when possible; an -unverified result remains blocked and prevents start. A restore without authentication entries -does not touch the runtime projection. This is in addition to the normal restore requirement that -browser sessions and pending OIDC state are cleared. - -## Firewall and network boundaries - -Set `THOTH_SERVER_BIND=127.0.0.1`. Permit inbound TCP 80/443 only to the TLS proxy; port 80 should -redirect to HTTPS. Do not open 8080 externally, and do not add a core port. If a separate proxy -host is used, replace loopback with a private, firewalled address and allow only that proxy source. - -Allow outbound DNS and HTTPS to the source/Git registries, plus only the configured ports for the -Git remote, DWH, vector database, embedding service, LLM, session PostgreSQL, and any approved -bastion. Docker's private `thothii` network carries only `frontend`↔`core` traffic. Do not attach -the mandatory stack to another application's network. - -After start, confirm the host listens as intended: - -```sh -sudo ss -lntp -``` - -Expected public listeners are the proxy on 80/443 and the frontend on loopback 8080. There must be -no host listener for core port 8787. - -## Address co-resident external services - -Endpoint values are resolved inside `core`. Therefore container 127.0.0.1 means the container -itself, not the Linux host. Prefer real DNS names with TLS, authentication, and firewall policy, -even for services on this physical server. - -When DNS is unavailable for a host-published service, create an untracked override such as -`/srv/thothii/operator/host-gateway.yaml` and add it to the installation descriptor: - -```yaml -services: - core: - extra_hosts: - - "host.docker.internal:host-gateway" -``` - -Use `host.docker.internal` in the endpoint binding. The `host-gateway` mapping supplies routing; -it does not bundle or trust the target service. A host service listening only on host -`127.0.0.1` is **not reachable** through this mapping. Bind that service to the ThothII Docker -bridge gateway address or to a dedicated private host interface—never to `0.0.0.0` merely to make -the check pass. A stable internal DNS record routed through an authenticated private listener is -the preferred alternative. - -After the first bounded start attempt, copy the exact core container name from `tht status` -into `CORE_NAME`, then derive—not guess—the network ID, Linux bridge interface, gateway, and -subnet. Compose networks normally use `br-<first-12-network-id>`; an explicit -`com.docker.network.bridge.name` option takes precedence: - -```sh -CORE_NAME=replace-with-exact-core-container-name -NETWORK_ID=$(docker inspect --format '{{range .NetworkSettings.Networks}}{{.NetworkID}}{{end}}' "$CORE_NAME") -NETWORK_NAME=$(docker network inspect --format '{{.Name}}' "$NETWORK_ID") -BRIDGE=$(docker network inspect --format '{{index .Options "com.docker.network.bridge.name"}}' "$NETWORK_ID") -test -n "$BRIDGE" || BRIDGE="br-${NETWORK_ID%${NETWORK_ID#????????????}}" -GATEWAY=$(docker network inspect --format '{{(index .IPAM.Config 0).Gateway}}' "$NETWORK_ID") -SUBNET=$(docker network inspect --format '{{(index .IPAM.Config 0).Subnet}}' "$NETWORK_ID") -printf 'network=%s bridge=%s gateway=%s subnet=%s\n' "$NETWORK_NAME" "$BRIDGE" "$GATEWAY" "$SUBNET" -ip address show dev "$BRIDGE" -``` - -Bind the co-resident service to `$GATEWAY`. In the host firewall `INPUT` chain, allow its exact -TCP port only when source is `$SUBNET`, input interface is `$BRIDGE`, and destination is -`$GATEWAY`; reject other sources to that listener and persist the rules using the distribution's -firewall manager. Docker's `DOCKER-USER` chain governs forwarded/published traffic and does not -replace this host-input rule. Ask the firewall administrator to implement the equivalent policy -with nftables when iptables is not the site's source of truth. - -For an iptables-managed host, replace the port before applying these reviewed rules; the second -rule prevents any other interface/source from reaching that gateway listener: - -```sh -EXTERNAL_PORT=replace-with-exact-service-port -sudo iptables -I INPUT 1 -i "$BRIDGE" -s "$SUBNET" -d "$GATEWAY" -p tcp --dport "$EXTERNAL_PORT" -j ACCEPT -sudo iptables -I INPUT 2 -d "$GATEWAY" -p tcp --dport "$EXTERNAL_PORT" -j REJECT -``` - -Confirm reachability with `tht pi test` for the configured LLM/Pi path and with the -authenticated Workspace Diagnostics action for DWH, vector collection/embedding pairing, and -embedding endpoints. A timeout paired with `ss -lntp`, `ip address show dev "$BRIDGE"`, and the -firewall counters distinguishes a loopback bind from a subnet/interface rule failure. Do not add -a shell to the browser or mount the Docker socket into core for this diagnostic. - -Configure each boundary independently: - -- DWH: read-only runtime identity, database/schema, verified TLS, and direct or REST endpoint. -- Vector database: endpoint plus exact database/schema, collection, distance metric, and writer - policy declared by the reviewed workspace. -- Embedding service: endpoint and the collection/embedding pairing—model and dimensions must match - the existing collection. Co-residence does not permit silently changing that pairing. -- LLM: authenticated endpoint selected through deployment and Pi configuration. - -Never add those services to ThothII's mandatory Compose files. Follow -[the diagnostic protocol](../workspace-diagnostic-protocol.md) before enabling a workspace. - -## Prepare operator files and secrets - -Clone with LF line endings, then verify before every build: - -```sh -git -c core.autocrlf=false clone \ - https://github.example.invalid/your-org/ThothII.git /srv/thothii/source/ThothII -cd /srv/thothii/source/ThothII -git config --local core.autocrlf false -bash scripts/verify-line-endings.sh -sudo /srv/thothii/source/ThothII/scripts/prepare-server-pi-state.sh \ - /srv/thothii/pi-state 10001 10001 -``` - -The last command is a mandatory clean-install and restore preflight. The server profile bind-mounts -the writable Pi-state root and then overlays protected `auth.json` plus tracked `models.json` and -`settings.json` read-only below it. Docker requires those three hidden target files to exist under -the host parent bind before startup. The initializer creates them atomically with UID/GID 10001, -mode `0600`, rejects symlink roots or targets, and never overwrites existing contents. It is safe to rerun -after restoring `pi-state`; run it before any `tht start`, Compose render/start, or Pi update. - -Copy the path-only server environment and installation descriptor: - -```sh -cp deploy/env/server.env.example /srv/thothii/operator/server.env -cp docs/install/examples/thothii-installation.server.yaml \ - /srv/thothii/operator/thothii-installation.yaml -chmod 0600 /srv/thothii/operator/server.env \ - /srv/thothii/operator/thothii-installation.yaml -``` - -The invoking operator owns both placeholder files; use an editor that preserves ownership and mode, -or create replacements under `umask 0077` in the operator directory. -Replace every placeholder with an absolute path. Use exactly one Git transport override. For -HTTPS, replace `deploy/compose.git-ssh.yaml` with `deploy/compose.git-https.yaml`. Keep the required -session-server overlay. Optional host-gateway or pinned -image overrides go after them. - -Create each installation credential (Pi/application, Git, and session storage) as an independent -regular file in `/srv/thothii/secrets`, owned by -UID 10001, the invoking operator's numeric primary GID, and mode `0640`. Owner access lets the UID -10001 container read a file mounted under `/run/secrets`; group access lets the operator run `tht`. The -operator environment records only absolute `*_FILE` or `*_SOURCE` paths for those installation -credentials. DWH and Evidence values are entered later through Workspace management and persist -as ciphertext under `/data/workspace-secrets`; the frontend receives no secret values. Do not -print file contents while testing permissions. - -```sh -operator_gid="$(id -g)" -sudo find /srv/thothii/secrets -type f -exec chown "10001:$operator_gid" {} + -sudo find /srv/thothii/secrets -type f -exec chmod 0640 {} + -sudo find /srv/thothii/secrets -type f \( ! -uid 10001 -o ! -gid "$operator_gid" -o ! -perm 0640 \) -print -``` - -Configure the remote repository and exactly one read-only Git transport as described in -[server workspace repository installation](server-workspace-registry.md). After startup, complete -the selected workspace's DWH and Evidence credentials through Workspace management. Secret values -must never be pasted into `server.env`, the installation YAML, a URL, or a shell argument. - -## Build locally or select pinned images - -Choose one image source. For a source build, the repository's reproducible launcher builds the -same `core` and `frontend` images used by the local profile. It requires only Git, Docker, and -Compose; copy the reviewed path-only server environment to the launcher's untracked input first: - -```sh -cd /srv/thothii/source/ThothII -cp /srv/thothii/operator/server.env deploy/env/local.env -bash scripts/build-local.sh -``` - -The printed local-profile start command is not the server start command; use `tht` below. - -Alternatively, create a reviewed untracked override with release images pinned by immutable -digest. Mutable tags are not a production pin: - -```yaml -services: - core: - build: !reset null - image: registry.example.com/thothii/core@sha256:<64-lowercase-hex-digits> - session-migrate: - build: !reset null - image: registry.example.com/thothii/core@sha256:<64-lowercase-hex-digits> - frontend: - build: !reset null - image: registry.example.com/thothii/frontend@sha256:<64-lowercase-hex-digits> -``` - -Add that absolute file last in `overrides`. `core` and `session-migrate` must use the exact same -core digest; neither may retain a local build or `:local` image. Frontend uses its own exact digest. -Both images must come from one compatible release; the core image must retain the declared Pi -version labels checked by `tht pi doctor`. Pull access -belongs in the host Docker credential store, not in Compose or the installation descriptor. - -## Install tht - -Build the operator binaries with Docker. No Go installation or Go knowledge is required: - -```sh -cd /srv/thothii/source/ThothII -THT_THT_OUTPUT_DIRECTORY=/srv/thothii/operator/build-output \ - bash scripts/build-tht.sh -operator_gid="$(id -g)" -sudo install -o root -g "$operator_gid" -m 0750 \ - /srv/thothii/operator/build-output/tht-linux-amd64 \ - /srv/thothii/operator/tht -``` - -The source checkout remains controlled by the invoking operator. The explicit output directory is the only build -write boundary; the build script rejects relative or non-canonical output paths. After installation, -remove or retain `build-output` according to the site's reviewed artifact policy. - -Use `tht-linux-arm64` on an ARM64 server. Set these variables in the maintenance shell; do -not source `server.env` as shell code: - -```sh -THT_BIN=/srv/thothii/operator/tht -INSTALLATION=/srv/thothii/operator/thothii-installation.yaml -"$THT_BIN" --help -"$THT_BIN" --installation "$INSTALLATION" update --check-only -``` - -Every operator command includes the descriptor explicitly. This preserves the installation's -profile, overrides, project identity, and durable current-image selector. The general form is -`tht --installation /absolute/path/thothii-installation.yaml <command>`. - -## Start and verify readiness - -Keep the TLS proxy stopped or firewalled during bootstrap. First stop the app, run the -installation-aware session migration, and inspect its pristine JSON. The command activates only -the `session-migrate` profile/service with `--no-deps --no-TTY`; it derives the migrator image from the -selected core image after all installation overrides, so this procedure is identical for source -and pinned modes. It exits nonzero unless both arrays are empty: - -```sh -"$THT_BIN" --installation "$INSTALLATION" stop -"$THT_BIN" --installation "$INSTALLATION" sessions migrate --yes -``` - -Successful output has this shape (the `applied` list may contain versions on first use): - -```json -{"applied":[],"drifted":[],"pending":[]} -``` - -Only after seeing `"pending":[]` and `"drifted":[]`, start and verify: - -```sh -"$THT_BIN" --installation "$INSTALLATION" start -"$THT_BIN" --installation "$INSTALLATION" status -"$THT_BIN" --installation "$INSTALLATION" doctor -curl --fail http://127.0.0.1:8080/health -"$THT_BIN" --installation "$INSTALLATION" pi doctor -"$THT_BIN" --installation "$INSTALLATION" pi test -``` - -`/health` proves process liveness. Readiness additionally requires both healthy services, a valid -Pi provider/model smoke, a successful Git registry pull with an active validated snapshot, valid -workspace diagnostics, and ready session PostgreSQL. Use the authenticated Workspace Management -page to pull and diagnose the reviewed workspace. A liveness response alone is not release -approval. - -After configuring the proxy, open <https://thoth.example.com> in a browser. Verify an unauthenticated -request is denied or redirected by the real identity provider, an authorized user can load the -same-origin UI and `/api`, an unauthorized user is denied, and an administrator alone can open Pi -Management. Keep port 8080 inaccessible from other hosts. - -## Configure TLS and upstream authentication - -Choose [Nginx](reverse-proxy-nginx.md) or [Caddy](reverse-proxy-caddy.md). Both examples terminate -TLS and proxy only to loopback `frontend`. They preserve SSE and clear client-supplied identity -headers before authentication. - -The authentication gateway must validate a real login/session and return normalized issuer, -subject, display-name, and admin claims only after success. Merely forwarding those headers does -not authenticate anyone. Do not enable `AUTH_MODE=upstream` on a listener reachable around the -trusted proxy, and never expose `core`. - -## Operate Pi, drain, and roll back - -Configure only closed provider/model/reasoning choices. Credentials remain protected files: - -```sh -"$THT_BIN" --installation "$INSTALLATION" pi status -"$THT_BIN" --installation "$INSTALLATION" pi configure -"$THT_BIN" --installation "$INSTALLATION" pi doctor -"$THT_BIN" --installation "$INSTALLATION" pi logs -``` - -Before an update, announce maintenance and ask users to finish active work. `--drain` closes new -admission and waits until no active sessions remain; it does not discard sessions. Build-source -and registry-source examples are: - -```sh -"$THT_BIN" --installation "$INSTALLATION" pi update \ - --version 0.81.0 --source build --yes --drain -"$THT_BIN" --installation "$INSTALLATION" pi update \ - --version 0.81.0 --source pull \ - --image registry.example.com/thothii/core@sha256:<64-lowercase-hex-digits> \ - --yes --drain -``` - -The transaction recreates only `core`, preserves volumes, verifies health/configuration/Pi, and -automatically attempts rollback after a post-mutation failure. For interrupted or ambiguous state: - -```sh -"$THT_BIN" --installation "$INSTALLATION" pi maintenance status -"$THT_BIN" --installation "$INSTALLATION" pi rollback --yes -"$THT_BIN" --installation "$INSTALLATION" pi maintenance recover --yes -``` - -Leave maintenance active if rollback cannot be verified. Preserve `.tht/<installation-id>/` -recovery state, repair the reported host/configuration issue, and rerun rollback or maintenance -recovery. Never delete or edit `current-image.yaml` or `update-state.json` to force progress. - -## Back up and restore - -Back up before source, workspace, session-schema, or Pi changes. Drain work, stop the installation, -record `git rev-parse HEAD`, image digests, and `tht status`, then archive the three bind trees -with numeric ownership. Do not include live secrets in this ordinary archive. - -```sh -"$THT_BIN" --installation "$INSTALLATION" stop -BACKUP=/srv/thothii-backups/2026-08-05 -sudo install -d -o root -g root -m 0700 "$BACKUP" -sudo tar --numeric-owner --xattrs --acls -C /srv/thothii -czf "$BACKUP/runtime-data.tgz" \ - data pi-state workspace-registry -sudo sh -ceu 'cd "$1"; sha256sum runtime-data.tgz > SHA256SUMS; sha256sum --check SHA256SUMS' sh "$BACKUP" -``` - -Back up the installation descriptor, path-only environment, generated overrides, source revision, -and secret files to separate encrypted access-controlled storage. Database-backed production -sessions require their own PostgreSQL-native consistent backup; the local bind tree is not a -substitute. Test both restore paths periodically. - -Restore only while stopped. Verify the checksum, extract first into a new empty root, inspect -ownership and expected registry layout, then retain the old trees by renaming them before placing -the restored set. This keeps the previous state recoverable: - -```sh -RESTORE=/srv/thothii-restore-2026-08-05 -sudo install -d -o root -g root -m 0700 "$RESTORE" -sudo sh -ceu 'cd "$1"; sha256sum --check SHA256SUMS' sh /srv/thothii-backups/2026-08-05 -sudo tar --numeric-owner --xattrs --acls -C "$RESTORE" \ - -xzf /srv/thothii-backups/2026-08-05/runtime-data.tgz -sudo test -d "$RESTORE/workspace-registry/repo" -sudo test -d "$RESTORE/workspace-registry/snapshots" -``` - -After placing the restored `pi-state` tree and before the first start, rerun -`sudo /srv/thothii/source/ThothII/scripts/prepare-server-pi-state.sh /srv/thothii/pi-state 10001 10001`. -It validates or recreates only the hidden regular mount targets; it does not alter restored Pi -state or any protected configuration source. - -During the reviewed restore window, move each old tree to a timestamped sibling, move the matching -restored tree into `/srv/thothii`, restore the PostgreSQL session backup from the same recovery -point, and keep the proxy closed. Run `update --check-only`, `start`, `doctor`, `pi test`, registry -status, workspace diagnostics, and a known historical session before reopening traffic. Never -merge an archive into a non-empty tree. - -## Diagnostics - -Begin with bounded, sanitized installation-aware commands: - -```sh -"$THT_BIN" --installation "$INSTALLATION" status -"$THT_BIN" --installation "$INSTALLATION" doctor -"$THT_BIN" --installation "$INSTALLATION" logs -"$THT_BIN" --installation "$INSTALLATION" pi status -"$THT_BIN" --installation "$INSTALLATION" pi doctor -"$THT_BIN" --installation "$INSTALLATION" pi test -"$THT_BIN" --installation "$INSTALLATION" pi logs -"$THT_BIN" --installation "$INSTALLATION" pi maintenance status -``` - -Use the authenticated Workspace Management status and diagnostic actions for Git revision, -degraded snapshot, bindings, DWH, vector, and embedding checks. Review proxy logs separately, but -configure both proxy and log shipping to exclude cookies, authorization data, identity payloads, -query strings, and secret values. Do not render Compose or print an environment as a diagnostic. - -Typical boundaries are: `doctor` for Docker/Compose/LF/volume/service health; `pi doctor` for image -and provider/model integrity; registry status for Git/snapshot health; workspace diagnostics for -external service identity; and the proxy/identity provider for login failures. - -## Data-preserving uninstall - -Drain and stop through `tht`, take and verify one final backup, and disable the TLS proxy -route. Set `THT_BACKUP_ROOT=/srv/thothii-backups` in `server.env`; the removal command verifies the -filesystem identity of that backup root, all three bind trees, and every declared secret before -and after removing anything. - -First run without confirmation. It displays the exact installation project, service, container -name, container ID, and stopped state, then exits without mutation. Check every target: - -```sh -"$THT_BIN" --installation "$INSTALLATION" stop -"$THT_BIN" --installation "$INSTALLATION" remove -``` - -If and only if both targets are the expected stopped `frontend` and `core` containers, confirm: - -```sh -"$THT_BIN" --installation "$INSTALLATION" remove --yes exact-core-id exact-frontend-id -``` - -Replace both example IDs with the values from the immediately preceding dry-run. The command -refuses confirmation if the current target set differs. The confirmed operation passes only those -previously displayed immutable container IDs to Docker, -uses no force or volume option, rejects running/replaced containers, and proves the preservation -paths still identify the same filesystem objects. Keep `/srv/thothii/data`, `pi-state`, -`workspace-registry`, `operator`, protected secrets, database backups, and the installation -descriptor if reinstallation is possible. Do not prune global Docker data. - -Do **not** run `docker compose down --volumes`; it deletes persistent application data. Reusing the -same protected descriptor path preserves the `tht` installation identity and allows a later -compatible source checkout to reconnect the retained state. diff --git a/docs/install/windows-line-endings.md b/docs/install/windows-line-endings.md deleted file mode 100644 index 5be4a31f..00000000 --- a/docs/install/windows-line-endings.md +++ /dev/null @@ -1,250 +0,0 @@ -# Windows and WSL2 line endings - -ThothII's containers execute shell scripts from the source checkout. Those files must stay LF, -even when the PC normally uses CRLF. The repository's `.gitattributes` is authoritative, but a -Windows Git setting or an old checkout can still leave incorrect bytes. Check line endings after -every clone and pull, before building an image. - -## Recommended WSL2 clone - -Use Docker Desktop with WSL2 integration. Clone inside the Linux filesystem, for example under -`/home/<user>/src`, rather than under `/mnt/c`. This avoids slow cross-filesystem builds, -permission surprises, and Windows tools rewriting files behind WSL. - -```sh -mkdir -p "$HOME/src" -cd "$HOME/src" -git -c core.autocrlf=false clone https://github.example.invalid/your-org/ThothII.git -cd ThothII -git config --local core.autocrlf false -bash scripts/verify-line-endings.sh -``` - -Keep Docker Desktop's integration enabled for that WSL distribution. Run the Linux build scripts -and the Linux `tht` binary from the same WSL shell. - -## Repository-local LF policy - -Set the option in this repository only. Do not change a company-wide or personal Git policy just -for ThothII. - -```sh -git config --local core.autocrlf false -git config --local --get core.autocrlf -``` - -The second command must print `false`. `.gitattributes` keeps shell, YAML, Dockerfile, JSON, -TypeScript, Python, and Markdown files at LF; PowerShell files remain CRLF. - -For a native PowerShell clone, disable conversion during the first checkout and then store the -repository-local setting: - -```powershell -git -c core.autocrlf=false clone https://github.example.invalid/your-org/ThothII.git -Set-Location ThothII -git config --local core.autocrlf false -& "C:\Program Files\Git\bin\bash.exe" scripts/verify-line-endings.sh -``` - -## Verify after clone or pull - -From WSL2, Git Bash, macOS, or Linux run: - -```sh -bash scripts/verify-line-endings.sh -``` - -Success exits with code 0 and prints no offending path. If it lists a file, do not build or start -ThothII. Correct the checkout first. Native PowerShell users can invoke the same script through -Git for Windows as shown above. - -## Recover an existing CRLF clone - -The safest recovery is to reclone into a new directory. First commit wanted work or copy it to a -backup outside both clones. Then clone with conversion disabled, run the verifier, and copy back -only reviewed changes. - -If a reviewed working tree must be repaired in place, Git must first normalize the index, export -that exact index to a separate repair directory, verify the exported bytes, and only then copy the -verified tracked files over the worktree. `git add --renormalize .` alone does not change existing -worktree bytes. - -> **WARNING — destructive worktree rewrite.** Make a backup outside the clone or commit every -> wanted tracked change before continuing. The copy step below overwrites tracked worktree bytes -> from the staged index export. Stop if the staged diff does not contain exactly the wanted content; -> untracked files are neither exported nor repaired. - -From WSL2, Git Bash, macOS, or Linux: - -```sh -set -euo pipefail - -abort_repair() { printf 'CRLF repair stopped: %s\n' "$1" >&2; exit 1; } -validate_index_export() { - git ls-files -s -z | while IFS= read -r -d '' entry; do - metadata="${entry%%$'\t'*}" - path="${entry#*$'\t'}" - mode="${metadata%% *}" - [[ "$path" != "$entry" ]] || exit 1 - case "$mode" in - 100644|100755) [[ -f "$REPAIR_DIR/$path" && ! -L "$REPAIR_DIR/$path" ]] || exit 1 ;; - 120000) [[ -L "$REPAIR_DIR/$path" ]] && readlink "$REPAIR_DIR/$path" >/dev/null || exit 1 ;; - *) printf 'Unsupported Git mode %s: %s\n' "$mode" "$path" >&2; exit 1 ;; - esac - done -} -validate_worktree_modes() { - git ls-files -s -z | while IFS= read -r -d '' entry; do - metadata="${entry%%$'\t'*}" - path="${entry#*$'\t'}" - mode="${metadata%% *}" - case "$mode" in - 100644|100755) [[ -f "$path" && ! -L "$path" ]] || exit 1 ;; - 120000) [[ -L "$path" ]] && readlink "$path" >/dev/null || exit 1 ;; - *) exit 1 ;; - esac - done -} -rewrite_index_entry() { - local mode="$1" path="$2" target temporary_link - case "$mode" in - 100644) - cp "$REPAIR_DIR/$path" "$path" && chmod a-x "$path" - ;; - 100755) - cp "$REPAIR_DIR/$path" "$path" && chmod a+x "$path" - ;; - 120000) - target="$(readlink "$REPAIR_DIR/$path")" || return 1 - temporary_link="${path}.thoth-lf-repair-link" - [[ ! -e "$temporary_link" && ! -L "$temporary_link" ]] || return 1 - ln -s "$target" "$temporary_link" || return 1 - rm -f "$path" || { rm -f "$temporary_link"; return 1; } - mv "$temporary_link" "$path" - ;; - *) return 1 ;; - esac -} - -if ! git status --short; then abort_repair "git status failed"; fi -if ! git config --local core.autocrlf false; then abort_repair "could not set repository LF policy"; fi -if ! git add --renormalize .; then abort_repair "index renormalization failed"; fi -if ! git diff --cached --check; then abort_repair "normalized index check failed"; fi -if ! git diff --cached; then abort_repair "normalized index review failed"; fi -REPAIR_DIR="$(cd .. && pwd -P)/ThothII-lf-repair" -if [[ -e "$REPAIR_DIR" ]]; then - abort_repair "choose a new empty LF repair directory: $REPAIR_DIR" -fi -if ! mkdir -p "$REPAIR_DIR"; then abort_repair "could not create LF repair directory"; fi -REPAIR_PREFIX="$REPAIR_DIR/" -if ! git checkout-index --all --force --prefix="$REPAIR_PREFIX"; then abort_repair "index export failed"; fi -if ! validate_index_export; then abort_repair "index export is missing entries or Git modes"; fi -if ! bash scripts/verify-line-endings.sh "$REPAIR_DIR"; then abort_repair "exported bytes failed LF verification"; fi -# WARNING: destructive copy; make a backup or commit wanted changes before this command. -if ! git ls-files -s -z | while IFS= read -r -d '' entry; do - metadata="${entry%%$'\t'*}" - path="${entry#*$'\t'}" - mode="${metadata%% *}" - rewrite_index_entry "$mode" "$path" || exit 1 -done; then - abort_repair "tracked-file rewrite failed; do not build from this worktree" -fi -if ! validate_worktree_modes; then abort_repair "repaired worktree does not match Git index modes"; fi -if ! bash scripts/verify-line-endings.sh; then abort_repair "repaired worktree failed LF verification"; fi -if ! git diff --cached --check; then abort_repair "repaired index check failed"; fi -``` - -Native Windows PowerShell runs the same Git operations and invokes the byte verifier through Git -for Windows: - -```powershell -$ErrorActionPreference = 'Stop' -function Assert-NativeSuccess([string]$Step) { - if ($LASTEXITCODE -ne 0) { throw "$Step failed with exit code $LASTEXITCODE." } -} -function ConvertFrom-IndexEntry([string]$Entry) { - if ($Entry -notmatch '^([0-9]{6}) [0-9a-f]+ [0-3]\t(.+)$') { - throw "Invalid Git index entry: $Entry" - } - [pscustomobject]@{ Mode = $Matches[1]; Path = $Matches[2] } -} - -git status --short -Assert-NativeSuccess 'git status' -git config --local core.autocrlf false -Assert-NativeSuccess 'repository LF policy' -git add --renormalize . -Assert-NativeSuccess 'index renormalization' -git diff --cached --check -Assert-NativeSuccess 'normalized index check' -git diff --cached -Assert-NativeSuccess 'normalized index review' -$RepairDir = Join-Path (Split-Path -Parent (Get-Location).Path) 'ThothII-lf-repair' -if (Test-Path $RepairDir) { throw 'Choose a new empty LF repair directory.' } -New-Item -ItemType Directory -Path $RepairDir | Out-Null -$RepairPrefix = $RepairDir.Replace('\', '/') + '/' -git -c core.symlinks=true checkout-index --all --force --prefix=$RepairPrefix -Assert-NativeSuccess 'index export' -$RawIndexEntries = @(git ls-files -s) -Assert-NativeSuccess 'index inventory' -$IndexEntries = @($RawIndexEntries | ForEach-Object { ConvertFrom-IndexEntry $_ }) -foreach ($Entry in $IndexEntries) { - $ExportPath = Join-Path $RepairDir $Entry.Path - $ExportItem = Get-Item -LiteralPath $ExportPath -Force -ErrorAction Stop - switch ($Entry.Mode) { - { $_ -in '100644', '100755' } { - if ($ExportItem.LinkType -eq 'SymbolicLink') { throw "Regular export became a symlink: $($Entry.Path)" } - } - '120000' { - if ($ExportItem.LinkType -ne 'SymbolicLink') { throw "Symlink export is not mode 120000: $($Entry.Path)" } - if ([string]::IsNullOrWhiteSpace([string]$ExportItem.Target)) { throw "Symlink target is empty: $($Entry.Path)" } - } - default { throw "Unsupported Git mode $($Entry.Mode): $($Entry.Path)" } - } -} -& "C:\Program Files\Git\bin\bash.exe" scripts/verify-line-endings.sh $RepairDir -Assert-NativeSuccess 'exported byte LF verification' -# WARNING: destructive copy; make a backup or commit wanted changes before this command. -foreach ($Entry in $IndexEntries) { - $ExportPath = Join-Path $RepairDir $Entry.Path - switch ($Entry.Mode) { - { $_ -in '100644', '100755' } { - Copy-Item -LiteralPath $ExportPath -Destination $Entry.Path -Force -ErrorAction Stop - } - '120000' { - $LinkTarget = [string](Get-Item -LiteralPath $ExportPath -Force -ErrorAction Stop).Target - $TemporaryLink = "$($Entry.Path).thoth-lf-repair-link" - if (Test-Path -LiteralPath $TemporaryLink) { throw "Temporary symlink path exists: $TemporaryLink" } - New-Item -ItemType SymbolicLink -Path $TemporaryLink -Target $LinkTarget -ErrorAction Stop | Out-Null - Remove-Item -LiteralPath $Entry.Path -Force -ErrorAction Stop - Move-Item -LiteralPath $TemporaryLink -Destination $Entry.Path -ErrorAction Stop - } - default { throw "Unsupported Git mode $($Entry.Mode): $($Entry.Path)" } - } -} -foreach ($Entry in $IndexEntries) { - $WorktreeItem = Get-Item -LiteralPath $Entry.Path -Force -ErrorAction Stop - switch ($Entry.Mode) { - { $_ -in '100644', '100755' } { - if ($WorktreeItem.LinkType -eq 'SymbolicLink') { throw "Regular worktree entry became a symlink: $($Entry.Path)" } - } - '120000' { - if ($WorktreeItem.LinkType -ne 'SymbolicLink') { throw "Repaired worktree symlink is not mode 120000: $($Entry.Path)" } - if ([string]::IsNullOrWhiteSpace([string]$WorktreeItem.Target)) { throw "Repaired symlink target is empty: $($Entry.Path)" } - } - default { throw "Unsupported Git mode $($Entry.Mode): $($Entry.Path)" } - } -} -& "C:\Program Files\Git\bin\bash.exe" scripts/verify-line-endings.sh -Assert-NativeSuccess 'repaired worktree LF verification' -git diff --cached --check -Assert-NativeSuccess 'repaired index check' -``` - -The export inventory must contain every regular mode (`100644`/`100755`) and recreate every tracked -workspace compatibility symlink (`120000`). The first verifier proves the complete -export before any overwrite; every copy/link operation is fail-closed; the final verifier examines -the repaired worktree bytes. On native Windows, creating symlinks requires Developer Mode or an -elevated account; failure stops the rewrite. Review the staged diff again before committing, then -remove the separate repair directory only after inspecting it. The procedure intentionally avoids -`git reset --hard`; replacing the clone is easier to audit and safer for uncommitted work. diff --git a/docs/migrations/p1-to-p1-1-registry-layout.md b/docs/migrations/p1-to-p1-1-registry-layout.md deleted file mode 100644 index 75a272c3..00000000 --- a/docs/migrations/p1-to-p1-1-registry-layout.md +++ /dev/null @@ -1,38 +0,0 @@ -# P1 to P1.1 registry layout migration - -P1.1 is a repository-contract cutover. New ThothII builds reject the old flat layout and a -repository without `thoth-workspaces.yaml`, so migrate the registry in Git first and upgrade the -application only after that reviewed migration commit is pushed. - -## One reviewed migration commit - -Perform the layout move in a clean review clone and keep it in one reviewed Git commit: - -```sh -git mv workspaces/<id>.yaml <id>/workspace.yaml -git mv workspace-content/<id>/evidence <id>/evidence -# create and review thoth-workspaces.yaml from descriptor metadata -``` - -For every workspace directory, preserve the existing descriptor bytes, move only the embedded -filesystem Evidence tree, and create `thoth-workspaces.yaml` with: - -- `schema_version: 1` -- the ordered `workspaces` list -- curator-owned `id`, `name`, and optional `description` copied from the reviewed descriptors - -Generated docs remain under `workspace-docs/<id>/`. Do not add an auto-migrator and do not let the -API rewrite the catalog or Evidence tree. - -## Cutover order - -1. Review the migration commit, including the new `thoth-workspaces.yaml` metadata. -2. Push that commit to the authoritative registry branch. -3. Upgrade ThothII only after that migration commit is pushed. -4. Pull the migrated registry into each installation before using workspace management. - -## Rollback - -Roll back the application revision and registry commit together. Do not point a P1.1 binary at the -old flat layout, and do not keep a migrated registry commit active while rolling the application -back to pre-P1.1 code. diff --git a/docs/operations/psd-dwh-auth-rollout.md b/docs/operations/psd-dwh-auth-rollout.md deleted file mode 100644 index c31f95fc..00000000 --- a/docs/operations/psd-dwh-auth-rollout.md +++ /dev/null @@ -1,43 +0,0 @@ -# PSD — rollout controllato DWH REST - -Questo runbook completa il [piano di accettazione autenticazione](../plans/2026-08-18-thothii-authentication-acceptance-and-psd-deployment.md) e il [programma di deployment PSD](../plans/2026-08-20-psd-server-deployment-program.md). Gate A e la parte dual-key di Gate B sono stati eseguiti con autorizzazioni separate. L'emendamento del proprietario del 2026-08-21 rinvia collaudo Mac, osservazione e revoca a prima di Project B; non autorizza ulteriori mutazioni. - -## Invarianti - -- ThothII sul server PSD: `postgres_direct` read-only, senza chiave `dwh-auth`. -- Mac PSD e client remoti: `rest_api`, una chiave per installazione, HTTPS `.it` verificato. -- `dwh-auth` è `systemd` indipendente, non Compose; non fermare o sostituire il vecchio stack ora. -- Sessioni legacy, indici Qdrant e cache Ollama sono dati test: nessuna migrazione o backup per il cutover. Il vecchio stack resta comunque fino a cutover/rollback approvati. -- Usare solo `/dwh/rpc/ping`, mai risultati clinici o catture Nginx grezze. - -## Gate A — Task 9, servizio locale senza Nginx pubblico - -Richiedere prima autorizzazione per SHA congelato, target, rollback e impatto legacy. Senza consenso, fermarsi e registrare solo `IN_DISCUSSION`. - -1. Verificare in sola lettura architettura, gruppo `www-data`, nomi liberi, systemd, `nginx -t`, ping attuale e file legacy regolare `root:root` `0600`; non leggerlo, stamparlo o calcolarne hash. -2. Costruire con `bash scripts/build-dwh-auth.sh --output /tmp/dwh-auth-release`; registrare solo SHA sorgente e checksum binario. -3. Installare binario/unit/tmpfiles come nella [guida server](../install/dwh-auth-server.md): registry `root:dwh-auth` `2750`, lock/record `0640`, socket `dwh-auth:www-data` `0660`. -4. Importare una sola legacy `legacy-shared` dal file protetto e creare `psd-mac-primary` in nuovo file `0600` sotto `/root/dwh-auth-provision/`; mai segreti in argv, ambiente, log o evidenze. -5. Eseguire `dwh-auth check`, `systemd-analyze verify`, avviare l'unità e testare sul socket Unix con file header curl protetti `0600`: v1=204, legacy=204, casuale=401, assente=401. -6. Salvare solo ID pubblici, owner/mode, stato unit/socket, timestamp, checksum binario/config e rollback. Non modificare Nginx in questo gate. - -## Gate B — Task 10, Nginx e client - -Serve un secondo consenso: presentare file, backup, canale consegna Mac, osservazione ed esiti 2xx/401/503. - -1. Creare copie timestampate `root:root` `0600` di `/etc/nginx/sites-available/policlinicosandonato` e file coinvolti; non allegare configurazioni Nginx grezze alle evidenze. -2. Aggiungere solo `/etc/nginx/conf.d/dwh-auth-rate-limit.conf` e route DWH; preservare upstream `http://127.0.0.1:3001/`, mantenere byte-identiche le location vector e rimuovere la chiave prima di PostgREST. -3. Eseguire checker strutturale, scansione segreti con solo `PASS/FAIL` e metadati, installare candidati e `sudo nginx -t`. No raw diff: non eseguire o conservare raw diff, `nginx -T` o dump: il file legacy può contenere la chiave. Se uno fallisce, ripristinare backup prima di reload e registrare FAIL sanitizzato. -4. Dopo consenso fare reload, poi HTTPS `.it` con CA e file header curl protetti 0600: v1=2xx, legacy=2xx, casuale=401, assente=401 su `/dwh/rpc/ping`; guasto autenticatore=503, mai accesso permissivo. -5. **Deferred pre-Project-B:** consegnare al Mac chiave e CA separatamente, verificare fingerprint fuori banda, configurare vault GUI o `API_KEY_FILE`, poi **Validate workspace source** e **Test workspace connections**. -6. **Deferred pre-Project-B:** completare 48 ore di osservazione comprendenti due cicli ETL delle 03:00, quindi revocare `legacy-shared` con ragione `shared-credential-rotation`; v1=2xx post-revoca, legacy=401 post-revoca e journal limitato senza chiavi/digest. - -## Rollback e chiusura - -Durante dual-key il rollback ripristina solo route/servizio revisionati, verifica `nginx -t` e fa reload autorizzato. Non ripristina chiavi revocate, PostgreSQL, dati legacy o stack. Scatta per TLS, risposte inattese, salute degradata o assenza di consenso. - -Activity 1 resta `DEFERRED_PRE_PROJECT_B`: dual-key è attivo, ma PASS richiede ancora v1=2xx -post-revoca, legacy=401 post-revoca, servizio/Nginx validi, log sanitizzati, rollback leggibile e -accettazione owner. Il rinvio non blocca il survey e Project A privato; blocca Project B. Compilare -[evidenza](../testing/evidence/psd-dwh-auth-rollout-report-template.md) e -[collaudo](../testing/dwh-auth-manual-acceptance.md). diff --git a/docs/operations/psd-server-sol-orchestration-prompt.md b/docs/operations/psd-server-sol-orchestration-prompt.md deleted file mode 100644 index fc48d440..00000000 --- a/docs/operations/psd-server-sol-orchestration-prompt.md +++ /dev/null @@ -1,92 +0,0 @@ -# Prompt operativo per Sol — deploy ThothII su PSD - -## Ruolo - -Sei l'orchestratore del deploy di ThothII sul server PSD. Devi guidare il lavoro -in modo incrementale, verificabile e reversibile. Non assumere che una fase sia -completata: richiedi evidenze e applica i gate descritti nei piani. - -## Documenti normativi - -Leggi prima questi file, in quest'ordine: - -1. `AGENTS.md` -2. `PROJECT_STATE.md` -3. `docs/plans/2026-08-20-psd-server-deployment-program-design.md` -4. `docs/plans/2026-08-20-psd-server-deployment-program.md` -5. `docs/plans/2026-08-20-psd-server-survey.md` -6. `docs/plans/2026-08-20-psd-server-project-a-standalone.md` -7. `docs/plans/2026-08-20-psd-server-project-b-authentik.md` -8. `docs/testing/psd-server-project-a-manual.md` -9. `docs/testing/psd-server-project-b-manual.md` -10. `docs/testing/evidence/psd-server-survey-report-template.md` -11. `docs/testing/evidence/psd-server-project-a-report-template.md` -12. `docs/testing/evidence/psd-server-project-b-report-template.md` - -In caso di conflitto, prevalgono `AGENTS.md`, `PROJECT_STATE.md` e i piani -specifici delle fasi nell'ordine Survey, Project A, Project B. - -## Modelli e delega multi-agent - -Verifica quali modelli e quali primitive multi-agent sono realmente disponibili -nell'ambiente. Se disponibili: - -- **Sol** mantiene il controllo del piano, dei gate, delle decisioni architetturali, - della sicurezza, di Authentik, Nginx, Supabase e del rollback. -- **Terra** esegue esclusivamente survey e controlli read-only: host, Docker, - checkout, Compose, Nginx, TLS, Aritmolab, Authentik e PostgreSQL. -- **Luna** esegue comandi bounded e verifiche ripetibili: build, compose, `tht` - (`status`, `doctor`, `pi`), smoke test, preprocessing e migrazioni già - autorizzate dal piano. - -Se Terra o Luna non sono disponibili, lavora in sequenza con il modello -disponibile. Non simulare agenti inesistenti. - -Per ogni incarico delegato specifica sempre: obiettivo, comandi consentiti, -operazioni vietate, evidenze da raccogliere e formato della risposta: - -```text -status: PASS | FAIL | BLOCKED -facts: fatti osservati -commands: comandi eseguiti (senza segreti) -evidence: file o output redatti -risks: rischi residui -blockers: impedimenti -``` - -Non riportare password, token, cookie, client secret o variabili d'ambiente -sensibili nei log o nei report. - -Le attività read-only indipendenti possono essere eseguite in parallelo. Tutte -le mutazioni devono essere sequenziali, con checkpoint e verifica prima della -fase successiva. Non parallelizzare stop/start dello stack, build/recreate, -migrazioni, modifiche ad Authentik, Nginx o al bilanciatore. - -## Regole inderogabili - -1. Inizia soltanto con il survey read-only. -2. Non spegnere, modificare o rimuovere il vecchio ThothII prima del survey e - del backup verificabile. -3. Non procedere a Project A senza un report Survey `PASS`. -4. Project A usa autenticazione locale e deve essere provato completamente prima - di iniziare Project B. -5. Project B con Authentik parte solo dopo il `PASS` esplicito di Project A. -6. Il database Supabase esistente va riusato tramite schemi dedicati; non creare - un nuovo database per isolare il dataset. -7. Il workspace remoto `tht-workspace-psd` resta unico: REST sul Mac e - PostgreSQL diretto sul server. I segreti non vanno in Git. -8. Il link dalla sidebar di Aritmolab deve restare funzionante e il percorso - pubblico finale deve passare dal balancer e da Nginx. -9. Conserva sempre un rollback verso il vecchio stack e verso l'autenticazione - locale finché il cutover non è approvato. - -## Prima azione richiesta - -Leggi tutti i documenti normativi. Poi avvia **solo Task 1 — Survey** del piano -generale. Produci un report consolidato usando il relativo template, con esito -`GO` o `NO-GO`, senza eseguire modifiche persistenti. Fermati e segnala ogni -credenziale Authentik mancante, permesso insufficiente, ambiguità sul balancer o -discrepanza tra dominio osservato e configurazione di Aritmolab. - -Dopo il survey attendi l'approvazione del proprietario prima di eseguire -Project A. diff --git a/docs/operations/psd-server-survey-remediation-checklist.md b/docs/operations/psd-server-survey-remediation-checklist.md deleted file mode 100644 index a94abd37..00000000 --- a/docs/operations/psd-server-survey-remediation-checklist.md +++ /dev/null @@ -1,455 +0,0 @@ -# PSD Server Survey — Remediation Checklist - -## Purpose and authority - -Questo documento permette al proprietario e a Sol di discutere, decidere e chiudere uno alla volta -i blocker emersi dal survey read-only del server PSD. È il punto di ripresa operativo tra sessioni: -registra soltanto fatti sanitizzati, decisioni, responsabili e riferimenti a evidenze protette. - -Fonti normative: - -- `docs/operations/psd-server-sol-orchestration-prompt.md` -- `docs/plans/2026-08-20-psd-server-deployment-program.md` -- `docs/plans/2026-08-20-psd-server-survey.md` -- `docs/plans/2026-08-20-psd-server-deployment-program-design.md` - -Questo documento non autorizza modifiche a server, servizi, database, Nginx, load balancer, -Authentik, Aritmolab o repository esterni. - -## Program gate - -- Program result: `SURVEY_NO_GO` -- Survey report: `/var/tmp/thothii-psd-survey.fJh7DS/survey-report.md` -- Survey report SHA-256: `36461b6c7d1e44d055f24d6919e892b7017352ac9eb99ac67b9ae62f0614f8e7` -- Legacy stack: deve restare acceso e invariato durante la discussione di questa lista -- La preparazione statica di Project A privato è autorizzata; stop del legacy e start del nuovo - restano vietati fino a `SURVEY_GO_PROJECT_A_PRIVATE` e a un consenso di mutazione separato -- Project B remains forbidden fino ai PASS automatico, umano e del proprietario per Project A - -## Current activity and resume point - -- Current activity: `2` -- Title: Identify accountable owners for the private Project A scope -- Resume from: Activity 2, assign owner/authority for legacy rollback, direct DWH, workspace and Pi/LLM -- Discussion rule: una sola attività può essere `IN_DISCUSSION` -- Allowed states: `PENDING`, `IN_DISCUSSION`, `DEFERRED_PRE_PROJECT_B`, `BLOCKED`, `PASS` - -## How to use this checklist - -1. Leggere `Current activity` e `Resume from`. -2. Discutere soltanto l'attività corrente. -3. Non inserire password, token, cookie, chiavi private, stringhe di connessione, claim grezzi o - valori di secret. -4. Registrare solo percorsi protetti, owner, mode, timestamp, nomi o ID di oggetti, checksum ed - esiti sanitizzati. -5. Alla fine della discussione aggiornare stato, note, decisione, evidenze, blocker e prossimo - passo. -6. Spostare `Current activity` solo quando il gate dell'attività corrente è soddisfatto oppure il - proprietario decide esplicitamente di parcheggiarla come `BLOCKED`. -7. Non riaprire un'attività `PASS` salvo nuova evidenza che ne invalidi la decisione. - -## Activity summary - -| ID | Attività | Stato | Responsabile | Prossimo gate | -|---|---|---|---|---| -| 1 | Rotazione controllata della credenziale DWH esposta | `DEFERRED_PRE_PROJECT_B` | Proprietario del progetto | Chiusura obbligatoria prima di Project B | -| 2 | Assegnazione dei responsabili dei componenti condivisi | `IN_DISCUSSION` | Proprietario del progetto | Owner privati A e shared B distinti | -| 3 | Risoluzione del dominio pubblico `.it` oppure `.com` | `BLOCKED` | Unassigned | Necessario per Project B, non per A privato | -| 4 | Topologia e responsabilità del load balancer | `BLOCKED` | Unassigned | Necessario per Project B/route opzionale | -| 5 | Accesso read-only protetto ad Authentik | `BLOCKED` | Unassigned | Necessario per Project B, non per A privato | -| 6 | Accesso catalog-only protetto a PostgreSQL | `BLOCKED` | Unassigned | DWH direct read-only ancora da provare | -| 7 | Confine temporaneo e cleanup del vecchio ThothII | `PASS` | Proprietario del progetto | Retain fino a Project B PASS; cleanup esatto separato | -| 8 | Accesso Git read-only al workspace PSD | `BLOCKED` | Curator da confermare | Checkout/deploy key server mancanti | -| 9 | Metadati Pi e LLM verificabili | `BLOCKED` | Unassigned | Policy e reachability redatte mancanti | -| 10 | Conservazione evidenze e nuovo survey bounded | `PENDING` | Unassigned | Nuovo report e decisione proprietario | - -## Activity 1: Rotate or revoke the exposed DWH credential safely - -- Status: `DEFERRED_PRE_PROJECT_B` -- Accountable owner: Proprietario del progetto (confermato dall'utente) -- Objective: sostituire o revocare in modo controllato la credenziale DWH comparsa nell'output - interno del survey, senza interrompere consumer legittimi e senza esporne nuovamente il valore. -- Why this is required: la credenziale deve essere considerata compromessa; non può essere usata - come base affidabile per completare il survey o iniziare Project A. -- Ordered actions: - 1. identificare il team che gestisce la route DWH, il suo meccanismo di autenticazione e la - custodia del secret; - 2. determinare il tipo di credenziale senza leggerla o copiarla in questa checklist; - 3. inventariare i consumer tramite riferimenti di configurazione e secret object; - 4. scegliere una transizione a doppia credenziale oppure una finestra atomica con rollback; - 5. generare e distribuire il nuovo secret attraverso il meccanismo protetto approvato; - 6. verificare i consumer autorizzati, l'assenza del nuovo valore nei log e la continuità del - vecchio stack; - 7. revocare la vecchia credenziale e provare che non venga più accettata; - 8. registrare soltanto evidenze redatte. -- Required redacted evidence: - - owner e autorizzazione della rotazione; - - tipo e identificatore non sensibile della credenziale; - - percorso protetto o secret object, senza contenuto; - - elenco dei consumer aggiornati; - - timestamp e risultati dei test positivi e negativi; - - conferma di revoca della credenziale precedente; - - procedura di rollback e relativo esito. -- Discussion notes: - - la credenziale osservata è stata identificata, senza rileggerne il valore, come chiave statica - `X-API-Key` applicata da Nginx alla route DWH REST `/dwh/`; è distinta dalle password PostgreSQL; - - `/home/chirone/chirone/etl` documenta PostgREST come integrazione HTTP esterna, mentre i processi - ETL e Superset usano PostgreSQL diretto; - - il vecchio ThothII e Chirone WP3 usano PostgreSQL diretto nei profili locali/server; Supabase - Studio è un pannello amministrativo e non appartiene al data-plane applicativo; - - il design approvato resta invariato: il Mac usa `rest_api`, il nuovo ThothII sul server PSD usa - `postgres_direct` con ruolo DWH dedicato e realmente read-only; - - la ricognizione dei repository ha rilevato materiale sensibile hardcoded in file tracciati, - senza riportarne i valori. La bonifica e la rotazione dei segreti coinvolti restano obbligatorie. - - il censimento statico non trova consumer `/dwh/` in ETL, Superset o nel Chirone WP3 attivo: - usano PostgreSQL diretto. Il vecchio container `thothii-core-1` è configurato con - `transport: direct`; i due ThothII restano comunque tecnicamente capaci di usare REST; - - i log Nginx redatti provano 18.523 richieste `/dwh/` dal 23 luglio al 19 agosto 2026: - 18.481 hanno user-agent classificato `python-requests` e gli endpoint RPC corrispondono - prevalentemente a introspezione e campionamento Thoth. La sorgente è una sola, privata e - compatibile con un proxy/load balancer; non prova che esista un solo client finale; - - l'ipotesi che il traffico sia generato dal job ETL delle 03:00 è smentita: nella finestra - 02:30–03:30 Europe/Rome non compare nessuna richiesta `/dwh/`. Il 99,37% del traffico è - concentrato il 13 agosto tra le 17:38 e le 20:03; - - la firma del 13 agosto corrisponde a undici preprocessing Thoth: undici `list_tables`, e - per ciascuna esecuzione 163 chiamate a ognuno dei tre RPC per-tabella più 1.180 `top_values`, - cioè 1.670 richieste per ciclo. `PROJECT_STATE.md` registra proprio il preprocessing PSD live - del 13 agosto su 163 tabelle, con più rerun e correzioni emerse durante l'esecuzione; - - il DAG ETL `nightly_etl_orchestrator` è schedulato con `0 3 * * *`, ma scrive il DWH tramite - PostgreSQL/`psycopg2` diretto. Nel codice tracciato non chiama `/dwh/`, gli RPC Thoth o - `tht workspace preprocess`, né emerge un trigger indiretto verso Thoth; - - la configurazione Nginx nominalmente attiva accetta una sola chiave tramite confronto letterale - in un endpoint `auth_request`. Non esiste una mappa a più chiavi: la doppia credenziale richiede - un refactor, backup, `nginx -t`, reload e rollback in una fase di mutazione autorizzata. - - il proprietario conferma che il ThothII sul Mac deve continuare a usare REST e che sono previste - molte altre installazioni remote, senza tunnel SSH verso Supabase. `/dwh/` è quindi - un'interfaccia remota stabile e multi-client, non una compatibilità temporanea. - - il proprietario decide di mantenere il certificato TLS corrente. L'endpoint esterno REST - presenta lo stesso certificato self-issued di Nginx, valido fino al 21 giugno 2027 e con SAN - per `supabase-aritmolab.policlinicosandonato.it`; non copre un eventuale dominio `.com`; - - il manuale di installazione deve trattare `TLS_CA_FILE` come necessario per ogni client che - non abbia già quel certificato nel proprio trust store, spiegando consegna affidabile, - verifica del fingerprint, rinnovo e aggiornamento coordinato delle installazioni; - - non esiste un ambiente di test. La rotazione dovrà quindi usare una verifica production-safe: - backup, finestra dual-key, RPC `ping` senza dati clinici, test positivo/negativo e rollback. - - il proprietario approva un componente `dwh-auth` riutilizzabile ma opzionale, incluso nel - repository senza modificare il protocollo dei client portabili o il CLI `tht`; - - su PSD `dwh-auth` avrà un lifecycle `systemd` indipendente dallo stack ThothII e comunicherà - con Nginx tramite socket Unix. Lo stop o la sostituzione di ThothII non dovrà interrompere i - client REST; - - il registro sarà composto da file protetti, versionati e aggiornati atomicamente, con un file - per generazione della chiave. Conterrà digest SHA-256 di segreti casuali da almeno 256 bit e - metadati non sensibili, senza SQLite o nuove dipendenze runtime; - - le chiavi saranno assegnate alle installazioni, non alle persone. Saranno prive di scadenza - predefinita, con scadenza opzionale e revoca manuale; - - creazione e import leggeranno o scriveranno soltanto file protetti. Nessun segreto sarà - accettato come argomento, stampato o inserito in log, JSON, documenti o repository; - - la gestione sarà fail-closed: credenziali non valide riceveranno `401`, mentre guasti del - servizio o del registro saranno mappati a `503` senza fallback permissivo; - - il proprietario conferma che le sessioni, gli indici e le cache del vecchio ThothII erano solo - test e non richiedono migrazione o attività di salvaguardia dedicate. Restano necessari il - rollback della route DWH condivisa e il rispetto del gate esplicito prima di fermare lo stack; - - il proprietario ha revisionato e approvato la specifica scritta. Il runbook eseguibile è in - `docs/operations/psd-dwh-auth-rollout.md`; separa sviluppo e verifica del componente dai due - gate espliciti di mutazione PSD; - - la credenziale condivisa corrente, priva del nuovo identificativo pubblico, sarà l’unico record - temporaneo `legacy_raw` con ID `legacy-shared`. Dopo la revoca non saranno accettate chiavi - prive del formato versionato per installazione. -- Decision: il nuovo ThothII server userà PostgreSQL diretto; il Mac e le future installazioni - remote useranno REST. La chiave condivisa corrente non è un modello finale accettabile: la - rotazione esposta userà una transizione a doppia credenziale e l'architettura target assegnerà - un'identità revocabile distinta a ogni installazione. Supabase Studio non sarà usato come - trasporto. Il proprietario del progetto è accountable per coordinare la rotazione. Il meccanismo - target è il servizio server-only `dwh-auth`, indipendente dallo stack ThothII, con registro a file - e una chiave revocabile per installazione. -- Owner amendment 2026-08-21: Gate A e dual-key Gate B sono eseguiti; il Mac live test, le 48 ore - comprendenti due cicli ETL delle 03:00 e la revoca di `legacy-shared` sono rinviati al gate - obbligatorio prima di Project B. Il rinvio non equivale a PASS. -- Blockers: per chiudere Activity 1 restano il collaudo Mac, l'osservazione completa, la revoca, - v1 positivo post-revoca e legacy `401`. -- Next step: continuare Activity 2–10 per lo scope Project A privato; riaprire Activity 1 prima di - congelare il candidato Project B. - -## Activity 2: Identify accountable owners for shared components - -- Status: `IN_DISCUSSION` -- Accountable owner: Unassigned -- Objective: associare ogni componente condiviso a una persona o a un team con autorità di lettura, - modifica, approvazione e rollback. -- Why this is required: la leggibilità di una configurazione non implica autorità a modificarla. -- Ordered actions: - 1. identificare gli owner di DNS/load balancer, Nginx/certificati, Authentik, - Supabase/PostgreSQL, Aritmolab, workspace Git e backup legacy; - 2. registrare il canale di approvazione e la procedura di escalation; - 3. confermare separatamente chi può autorizzare Project A e Project B. -- Required redacted evidence: nomi dei team, ruoli, canali operativi e conferme di responsabilità; - nessun contatto personale sensibile. -- Discussion notes: Not discussed -- Decision: No decision recorded -- Blockers: nessun owner condiviso è stato ancora formalmente confermato. -- Next step: compilare ora la matrice distinguendo componenti necessari a Project A privato e - componenti shared/pubblici rinviabili a Project B. - -## Activity 3: Resolve the authoritative public origin - -- Status: `BLOCKED` -- Accountable owner: Unassigned -- Objective: scegliere sulla base di evidenze l'unica origine pubblica finale tra il dominio `.it` - osservato e il dominio `.com` riportato nel piano. -- Why this is required: callback OIDC, certificato, cookie, Nginx, load balancer e sidebar devono - concordare sulla stessa origine HTTPS. -- Ordered actions: - 1. ottenere la dichiarazione autorevole dell'owner DNS/load balancer; - 2. verificare record DNS, route, backend, certificato e redirect; - 3. confrontare l'origine con la configurazione e la sidebar di Aritmolab; - 4. registrare l'origine approvata e le discrepanze da correggere in Project B. -- Required redacted evidence: hostname finale, record/route sanitizzati, SAN del certificato, - destinazione sidebar e approvazione dell'owner. -- Discussion notes: il survey ha osservato `.it`; il piano cita `.com`. La discrepanza è aperta. -- Decision: No decision recorded -- Blockers: owner DNS/load balancer non identificato e origine browser-visible non provata. -- Next step: ottenere la dichiarazione autorevole dopo l'assegnazione degli owner. - -## Activity 4: Establish the load-balancer contract - -- Status: `BLOCKED` -- Accountable owner: Unassigned -- Objective: documentare il confine effettivo del load balancer e la procedura reversibile per le - route temporanea e finale. -- Why this is required: il survey locale non ha potuto provare owner, backend, health check, TLS, - source range o allowlist. -- Ordered actions: - 1. identificare superficie di configurazione e owner; - 2. registrare backend, porta, health check, punto TLS e source range verso Nginx; - 3. documentare deploy, validazione e rollback; - 4. stabilire se una route temporanea può essere limitata agli operatori; - 5. definire una prova positiva e una negativa dell'allowlist senza creare ancora la route. -- Required redacted evidence: nomi/ID delle route, backend e health check sanitizzati, ownership, - capacità di allowlist e procedura di rollback. -- Discussion notes: Not discussed -- Decision: No decision recorded -- Blockers: il load balancer non è ispezionabile dalla superficie locale autorizzata. -- Next step: coinvolgere l'owner identificato nell'Activity 2. - -## Activity 5: Provide protected read-only Authentik survey access - -- Status: `BLOCKED` -- Accountable owner: Unassigned -- Objective: permettere un inventario Authentik bounded e read-only della versione installata. -- Why this is required: applicazioni, provider, flow, mapping, gruppi, service account, permessi API - e procedura di export non sono stati verificati. -- Ordered actions: - 1. identificare l'owner Authentik e la procedura di backup/export; - 2. predisporre una credenziale read-only o un'esecuzione assistita dall'owner; - 3. comunicare solo percorso, owner, mode e usabilità del secret; - 4. inventariare nomi/ID e convenzioni senza recuperare secret write-only; - 5. confrontare il comportamento con OpenAPI e documentazione della release installata. -- Required redacted evidence: versione, nomi/ID degli oggetti, permission set della credenziale, - riferimento all'export e risultati sanitizzati. -- Discussion notes: Not discussed -- Decision: No decision recorded -- Blockers: nessuna credenziale amministrativa/API utilizzabile è stata stabilita. -- Next step: ottenere dall'owner un meccanismo protetto post-rotazione. - -## Activity 6: Provide protected catalog-only PostgreSQL survey access - -- Status: `BLOCKED` -- Accountable owner: Unassigned -- Objective: verificare database, schemi, ruoli, grant, migrazioni e PostgREST con sole query di - catalogo. -- Why this is required: il runtime DWH read-only, lo stato di `thoth_sessions` e i confini Supabase - non sono provati. -- Ordered actions: - 1. identificare DBA e procedura protetta di connessione; - 2. verificare database, utente corrente, schemi e owner; - 3. verificare i grant sullo schema `datawarehouse` senza write probe clinici; - 4. verificare stato di `thoth_sessions` e migration records; - 5. verificare gli schemi esposti da PostgREST; - 6. registrare TLS/CA, backup e convenzioni per ruoli migrator/runtime. -- Required redacted evidence: risultati catalogici bounded, nomi dei ruoli, attributi e grant, - schemi PostgREST, riferimento a backup e TLS; nessuna stringa di connessione. -- Discussion notes: il wrapper ETL `ConnectionFactory` ha aperto una sessione dichiarata - read-only e ha eseguito sole query aggregate a `pg_catalog`. Il database è PostgreSQL 15.8; - l'identità disponibile è `postgres`, owner dello schema `datawarehouse`, con `USAGE` e - `CREATE`. Su tutte le 163 relazioni catalogate possiede SELECT e anche tutti i privilegi di - scrittura/DDL tabellari verificati. Nessun nome tabella o dato clinico è stato raccolto. -- Decision: il meccanismo esistente è valido per il survey catalogico, ma è vietato come identità - runtime del nuovo core perché non è least-privilege né read-only. -- Blockers: il DBA deve fornire un ruolo dedicato con soli USAGE/SELECT e una route diretta - certificabile dal nuovo core. Il PostgREST DWH è loopback host su 127.0.0.1:3001 e non prova il - percorso PostgreSQL diretto richiesto da Project A. -- Next step: definire con il DBA ruolo, secret-file protetto, TLS/rete e query di grant da ripetere; - non creare il ruolo durante il survey. - -## Activity 7: Bind the disposable legacy boundary and cleanup exclusions - -- Status: `PASS` -- Accountable owner: Proprietario del progetto (confermato dall'utente) -- Objective: mantenere il vecchio ThothII solo come confine temporaneo di cutover e rimuovere - esclusivamente le sue risorse dopo il PASS reale di Aritmolab. -- Why this is required: Aritmolab usa oggi i container legacy, ma il proprietario ha dichiarato - sacrificabili sessioni, configurazione e dati del vecchio ThothII. -- Ordered actions: - 1. inventariare nuovamente container, image ID, source e data immediatamente prima dello stop; - 2. preservare invariati i due network condivisi e l'Evidence ETL esterna; - 3. chiudere la route e fermare solo i container legacy nel gate Project A autorizzato; - 4. conservare container, immagini, source e data fino al PASS automatico, umano e owner di B; - 5. rimuovere poi soltanto gli exact target sotto un'autorizzazione cleanup separata. -- Required redacted evidence: inventario esatto, owner, restart recipe, esclusioni shared, decisioni - Project A/B e manifest finale di cleanup; nessun contenuto di secret. -- Discussion notes: il legacy stack è ancora attivo e invariato. Compose project `thothii` usa - `/home/chirone/ThothII/compose.yaml`; i servizi sono `core` e `frontend`, senza named volume. - Il solo bind applicativo RW è `/home/chirone/thothii-data` (con i bind Pi annidati); Evidence è - un bind RO esterno. Una lettura tar verso `/dev/null` di source e data ha dato - `legacy_backup_readability=PASS`. Il source è circa 1.05 GB e il data bind circa 1.9 MB. - I dry-run Compose passano solo fornendo il path non segreto - `PI_AUTH_FILE=/home/chirone/thothii-data/pi-config/agent/auth.json` insieme a - `--env-file deploy/thothii.env -p thothii -f compose.yaml`; stop individua entrambi i container - e start è sintatticamente valido (non trova container arrestati mentre lo stack è ancora attivo). -- Decision: nessun backup legacy è richiesto. I container, immagini, source e data già presenti - restano il solo rollback temporaneo fino al PASS Project B; non si crea alcun utente host. Le - risorse esclusive potranno essere cancellate dopo il collaudo Aritmolab, mentre network shared, - ETL Evidence, Omics, LocalLLM, DWH, `dwh-auth`, Supabase, Authentik e Superset sono esclusi. -- Blockers: nessuno per questa decisione survey. Stop/start, route change e cleanup restano tre - autorizzazioni di mutazione separate e non sono autorizzati da questo PASS. -- Next step: usare il design e piano clean-replacement approvati; non eseguire ancora mutazioni. - -## Activity 8: Provide read-only PSD workspace Git access - -- Status: `BLOCKED` -- Accountable owner: Unassigned -- Objective: verificare il repository remoto condiviso e il suo stato corrente dal server senza - capacità di push. -- Why this is required: SHA, descriptor, trasporti, Evidence, annotazioni e scope della deploy key - non sono stati osservati dal server. -- Ordered actions: - 1. identificare curator e owner della deploy key; - 2. fornire un riferimento protetto alla chiave server read-only; - 3. verificare remote, branch e SHA con modalità non interattiva; - 4. verificare catalogo, schema v3, trasporti, Evidence e annotazioni; - 5. provare che la credenziale server non possa effettuare push. -- Required redacted evidence: remote, branch, SHA, descriptor blob, stato Evidence/annotations e - attestazione read-only della deploy key. -- Discussion notes: Not discussed -- Decision: No decision recorded -- Blockers: nessun checkout workspace o deploy credential utilizzabile è stato localizzato. -- Next step: coinvolgere il curator e predisporre l'accesso server read-only. - -## Activity 9: Make Pi and LLM metadata verifiable - -- Status: `BLOCKED` -- Accountable owner: Unassigned -- Objective: verificare versione Pi, provider, modello, thinking level, riferimento credenziale e - reachability LLM senza esporre il secret. -- Why this is required: la root Pi legacy non è attraversabile dall'operatore del survey e i - default del nuovo source non provano la configurazione in esecuzione. -- Ordered actions: - 1. identificare owner della configurazione Pi/LLM; - 2. scegliere tra esecuzione assistita dall'owner e accesso read-only allowlisted; - 3. estrarre esclusivamente metadati non sensibili; - 4. eseguire un controllo bounded di reachability senza stampare credenziali; - 5. registrare anche il mismatch NVML/GPU come rischio separato, non come blocker CPU. -- Required redacted evidence: versione, provider, model ID, thinking level, endpoint sanitizzato, - percorso/mode della credenziale e risultato di reachability. -- Discussion notes: host `x86_64`; due GPU NVIDIA osservate, ma `nvidia-smi` non è utilizzabile per - mismatch driver/libreria NVML. Una lettura whitelist di `settings.json` ha rilevato Pi 0.80.3, - provider `deepseek`, modello `deepseek-v4-pro` e thinking `high`, senza leggere - `auth.json`. Un singolo probe senza tool, contesto o sessione ha prodotto solo - `pi_reachability=FAIL`. Il catalogo custom dichiara inoltre il solo provider `local-qwen`. -- Decision: i metadati sono verificati, ma reachability e coerenza default/catalogo non passano. -- Blockers: diagnosticare il FAIL senza esporre la credenziale e confermare il provider/modello - approvato per Project A; il mismatch NVML/GPU resta rischio separato, non un motivo per assumere - che il percorso CPU funzioni. -- Next step: eseguire un controllo assistito e sanitizzato della configurazione provider, quindi - ripetere una sola reachability probe bounded. - -## Read-only resume — 2026-08-21 - -- Host/source: Linux x86_64, Docker 29.1.1, Compose 2.40.3, application worktree clean at - `7118950416b3008a8182825de027c7f8b235de57`; Qdrant/Ollama images are local, while the required - embedding image/model is not yet proved local. -- Capacity: approximately 1.1 TB free on `/home` and 36 GB on `/`; several loopback candidate - ports are currently free. These facts do not reserve a path or port. -- Legacy: project `thothii` is still running and unchanged. Source is - `/home/chirone/ThothII` at `6ca4275`; the only RW application bind is - `/home/chirone/thothii-data`, plus the nested Pi binds. The source is about 1.05 GB and the data - bind about 1.9 MB. No backup was created. -- Recovery decision: legacy state is disposable; no backup is required. Existing stopped - containers, images, source and data remain only as the temporary Project A/B rollback boundary. -- Identity decision: neither UID nor GID 10001 maps to a host account. No host identity will be - created. The new image retains numeric `10001:10001`, confined to its distinct writable roots; - the existing operator owns source/configuration. Recheck both `getent` lookups before creating - paths and stop on any new mapping. -- Shared exclusion: never remove the Omics/LocalLLM networks or external ETL Evidence bind. -- Candidate paths: documented examples `/srv/thothii` and `/srv/thothii-backups` are absent and - therefore only candidates; they have not been created. `127.0.0.1:18080` è il candidato - frontend e risultava libero al momento del survey, ma non è riservato e va ricontrollato prima - dello start. Existing `/home/chirone/thothii-data` must not be reused. -- Workspace: the canonical private remote is documented as - `git@github.com:mptyl/tht-workspace-psd.git`; a non-interactive read-only remote query resolved - `main` at `bfbabf9f2defcf861a3225296eaff8c6d44c0ac9`. No server checkout or dedicated deploy-key - reference is present at the documented local paths, and the credential's inability to push is - not yet proved. -- Pi/LLM: legacy Pi version `0.80.3` is visible, but provider/model/thinking/credential reference - and bounded reachability remain unknown. -- Shared scope: `.it` resolves locally and `.com` does not, Nginx is valid/active, Authentik - 2026.2.1 and Supabase components are running. Owner, LB contract, Authentik inventory and - PostgreSQL catalog grants remain unproved and are not inferred. -- Decision: `SURVEY_NO_GO` for Project A private remains. The bounded work authorized now is - limited to planning/static preparation; no source clone, protected tree, backup, stop or start - has been performed. - -## Activity 10: Retain evidence and run the missing bounded survey checks - -- Status: `PENDING` -- Accountable owner: Unassigned -- Objective: conservare le evidenze protette, ripetere soltanto i controlli mancanti e produrre una - nuova decisione verificabile. -- Why this is required: il report corrente è `SURVEY_NO_GO` e non può essere promosso per inferenza. -- Ordered actions: - 1. scegliere il protected evidence root definitivo; - 2. trasferire la directory del survey senza modificarne i contenuti e verificare il digest; - 3. confermare che le Activity 1–9 siano `PASS` oppure abbiano una risoluzione proprietario - esplicitamente accettata; - 4. eseguire solo i controlli bounded mancanti del piano survey; - 5. aggiornare il report e verificarne checksum e secret hygiene; - 6. chiedere la decisione esplicita del proprietario. -- Required redacted evidence: percorso finale, digest, matrice Activity 1–9, nuovi risultati - bounded, report aggiornato e decisione firmata. -- Discussion notes: Not discussed -- Decision: No decision recorded -- Blockers: dipende dalla chiusura delle Activity 1–9 e dall'approvazione del retention root. -- Next step: avviare soltanto dopo la chiusura dei blocker precedenti. - -## Fresh survey and owner gates - -Un nuovo `SURVEY_GO_PROJECT_A_PRIVATE` richiede: - -- owner e autorità di stop/start/rollback identificati per il legacy e Project A; -- accessi read-only PostgreSQL, workspace Git e Pi/LLM verificati; -- identità DWH dimostrata read-only; -- inventario, restart recipe e cleanup exclusions legacy verificabili; -- risorse e percorsi della nuova installazione approvati; -- report redatto, secret-scan valido e checksum verificato; -- approvazione esplicita del proprietario. - -`SURVEY_GO_PROJECT_B` richiede inoltre: - -- collaudo Mac `rest_api` con chiave per installazione; -- 48 ore di osservazione comprendenti due cicli ETL delle 03:00; -- credenziale `legacy-shared` revocata, v1 positiva e legacy `401`; -- owner e autorità di modifica/rollback per tutti i componenti shared; -- origine pubblica unica e load-balancer contract provati; -- accesso read-only Authentik e inventario della release installata; -- ogni altro blocker pubblico/shared delle Activity 2–9 chiuso. - -Il PASS tecnico del survey non autorizza automaticamente Project A. L'autorizzazione deve essere -registrata separatamente. - -## Change log - -| Data | Attività | Modifica | Autore | -|---|---|---|---| -| 2026-08-20 | Initial | Creata checklist; Activity 1 aperta, Activity 2–10 pending | Sol | -| 2026-08-21 | Sequencing amendment | Activity 1 deferred pre-B; survey ripreso read-only; Project A private resta NO-GO | Owner/Sol | -| 2026-08-21 | Clean replacement | Activity 7 PASS; legacy disposable dopo B; nessun account host 10001 | Owner/Sol | diff --git a/docs/plans/2026-08-08-internal-qdrant-ollama-design.md b/docs/plans/2026-08-08-internal-qdrant-ollama-design.md deleted file mode 100644 index b6965b36..00000000 --- a/docs/plans/2026-08-08-internal-qdrant-ollama-design.md +++ /dev/null @@ -1,210 +0,0 @@ -# Internal Qdrant and Ollama Architecture Design - -**Status:** approved on 2026-08-08 - -## Objective - -ThothII owns its semantic infrastructure. Every supported deployment includes a private Qdrant -service and a private Ollama embedding service. The analytical DWH remains external and read-only; -each workspace descriptor associates that DWH with one Qdrant collection used for database schema, -Evidence, and approved Memory records. - -## Decisions - -- Qdrant replaces pgvector as the only operational vector store. -- Ollama replaces workspace-selected external embedding endpoints. -- The default and required model is `qwen3-embedding:0.6b` with 1024-dimensional normalized dense - embeddings and cosine distance. -- One Qdrant collection belongs to one workspace. Schema, Evidence, and Memory points share that - collection and are separated by indexed payload field `kind`. -- Qdrant and Ollama are mandatory base-Compose services. They are not published on host ports and - are reachable only from the private Compose network. -- Existing schema-v1 and schema-v2 descriptors remain readable for migration, but they are not - activatable. The new operational contract is workspace schema v3. - -The model choice is based on the published Qwen model card: the 0.6B model supports more than 100 -languages, a 32K context window, Matryoshka dimensions up to 1024, and instruction-aware retrieval. -Ollama distributes a CPU-viable quantized build and can use an exposed GPU without changing the -application protocol. - -References: - -- <https://huggingface.co/Qwen/Qwen3-Embedding-0.6B> -- <https://ollama.com/library/qwen3-embedding> -- <https://docs.ollama.com/capabilities/embeddings> -- <https://qdrant.tech/documentation/installation/> -- <https://qdrant.tech/documentation/manage-data/collections/> - -## Target topology - -```text -browser -> frontend -> core -> external DWH - -> private Qdrant - -> private Ollama embedding -``` - -The base Compose project contains: - -- `frontend`: static React application and same-origin API proxy. -- `core`: Fastify, Pi, and the Python `tht` harness. -- `qdrant`: pinned Qdrant server with persistent `qdrant-data` volume. -- `embedding`: pinned Ollama server with persistent `embedding-models` volume. -- `embedding-model-init`: bounded one-shot service that pulls and verifies - `qwen3-embedding:0.6b`; `core` starts only after it succeeds. - -`qdrant` and `embedding` use `expose`, not `ports`. The core receives installation-owned internal -URLs: - -```text -THT_INTERNAL_QDRANT_URL=http://qdrant:6333 -THT_INTERNAL_EMBEDDING_URL=http://embedding:11434 -THT_INTERNAL_EMBEDDING_MODEL=qwen3-embedding:0.6b -THT_INTERNAL_EMBEDDING_DIMENSIONS=1024 -``` - -These are deployment facts, not workspace connector bindings. The runtime rejects non-loopback or -non-Compose-service hosts when these variables are overridden for development. - -An optional Linux GPU override exposes an available NVIDIA/AMD device to Ollama. The base profile -must remain CPU-safe. macOS Docker remains CPU-only because Docker Desktop cannot expose the Apple -GPU to an Ollama container. - -## Workspace schema v3 - -The workspace itself is the association between the external database and the internal collection: - -```yaml -workspace: - schema_version: 3 - id: psd-clinical - name: PSD Clinical - language: it - -dwh: - engine: postgres - database: postgres - schema: datawarehouse - supported_transports: [postgres_direct] - -semantic_index: - vector_store: - engine: qdrant - collection: psd-clinical - dimensions: 1024 - distance: cosine - embedding: - provider: ollama_internal - model: qwen3-embedding:0.6b - dimensions: 1024 - -llm_policy: - allowed: [zai/glm-5.2] -``` - -Invariants: - -- the collection name is an explicit portable identifier; -- active workspaces cannot share a collection; -- vector and embedding dimensions are both 1024; -- distance is `cosine`; -- provider and model are exactly the supported internal values; -- no vector transport, vector credential, embedding URL, or embedding credential may appear in a - schema-v3 descriptor or installation contract; -- DWH connectors remain installation-local and can still use the supported external DWH transports. - -Schema-v1/v2 pgvector descriptors are listed as `migration_required`. Migration creates a reviewed -schema-v3 document; it does not copy vector data implicitly. Existing semantic data is rebuilt from -the canonical schema documents, Evidence corpus, and Memory registry. - -## Qdrant data model - -Each point has a deterministic UUIDv5 derived from: - -```text -workspace_id + kind + record_key -``` - -The vector is the 1024-dimensional Ollama result. The payload is: - -```json -{ - "workspace_id": "psd-clinical", - "kind": "schema", - "source_id": "datawarehouse.patients", - "record_key": "schema:table:datawarehouse.patients", - "content_hash": "sha256:...", - "workspace_revision": "<git commit>", - "generation": "<optional corpus generation>", - "language": "it", - "text": "...", - "metadata": {} -} -``` - -`kind`, `source_id`, `content_hash`, `workspace_revision`, and `generation` receive keyword payload -indexes. Queries always filter by `workspace_id` and an explicit allowed `kind` set. Upsert is -idempotent. Evidence generation deletion is an exact filtered delete. Collection creation is also -idempotent and fails closed if an existing collection has incompatible dimensions or distance. - -## Harness integration - -The existing `VectorStore` port remains the workflow boundary. A `QdrantVectorStore` adapter maps -its operations to Qdrant REST endpoints while preserving current schema/Evidence/Memory call sites. -The existing Ollama embedding client is narrowed to the internal `/api/embed` contract and verifies: - -- configured model exists; -- output count matches input count; -- every vector has 1024 finite numeric values; -- no remote URL or API key is accepted. - -The JSONL Memory registry and persisted phase documents remain canonical. Qdrant remains a derived, -rebuildable semantic index. Schema, Evidence, and Memory ingestion all use the same point builder, -content hashing, and retry policy. - -## Readiness and failure behavior - -Readiness is layered: - -1. Compose waits for Qdrant health. -2. Compose waits for Ollama health and successful model initialization. -3. Workspace activation validates the schema-v3 contract. -4. Harness readiness ensures the Qdrant collection and checks its vector configuration. -5. Harness embeds a bounded probe and verifies 1024 dimensions. - -Failures are sanitized and fail closed: - -- unavailable Qdrant -> `workspace_not_activatable` before session persistence; -- unavailable or missing Ollama model -> `model_unavailable` before session persistence; -- collection mismatch -> `semantic_index_incompatible` without recreating or deleting data; -- embedding dimension mismatch -> no point write; -- partial batch failure -> operation reports failure and remains safe to retry. - -No health response, API response, or diagnostic log exposes DWH credentials or indexed text. - -## Deployment and migration - -The pgvector deployment path is retired: - -- remove local-vector Compose overlays and pgvector bootstrap/migration services; -- remove vector PostgreSQL role and password contracts; -- remove runtime support for vector REST/SSH and external embedding URLs; -- keep only the descriptor parser and migration code needed to recognize legacy workspaces; -- update local/server manuals, examples, smoke tests, CI coupling scans, backup instructions, and - release gates for four persistent stores plus Qdrant and Ollama volumes. - -Qdrant backup/restore uses collection snapshots or the persistent volume according to the operator -manual. Ollama model storage is a cache: it may be backed up for offline recovery but is not an -application source of truth. - -## Acceptance criteria - -- Base local and server Compose renders include healthy private `qdrant` and `embedding` services. -- A clean CPU-only installation downloads the model, creates a workspace collection, and embeds a - probe without external vector or embedding configuration. -- GPU override uses the same API and persistent model volume. -- Schema-v3 workspaces activate; schema-v1/v2 workspaces report `migration_required`. -- Two workspaces cannot claim the same Qdrant collection. -- Schema, Evidence, and Memory records coexist in one collection and remain filter-isolated. -- Existing workflow behavior and persisted session contracts remain unchanged. -- Tests reject all active pgvector deployment, external vector binding, and external embedding - configuration paths. diff --git a/docs/plans/2026-08-14-read-only-workspace-runtime-secrets-design.md b/docs/plans/2026-08-14-read-only-workspace-runtime-secrets-design.md deleted file mode 100644 index 1e7083bd..00000000 --- a/docs/plans/2026-08-14-read-only-workspace-runtime-secrets-design.md +++ /dev/null @@ -1,189 +0,0 @@ -# Read-only Workspace Repository and Runtime Secrets Design - -**Date:** 2026-08-14 -**Status:** Approved - -## Purpose - -ThothII consumes workspaces from one administrator-configured Git repository. Workspace authors -prepare and publish source outside ThothII. The application fetches, validates, and activates -repository revisions, but never edits, commits, pushes, imports, or exports workspace source. - -Runtime credentials are intentionally absent from Git. After a workspace has been read, ThothII -derives the required credentials from its connector and authentication choices and lets an -authorized user complete them in the web application. The values are encrypted and persisted by -the backend; the browser retains neither workspace content nor secrets. - -## Ownership boundaries - -### Workspace source - -The workspace source is an ordinary directory maintained outside the ThothII runtime. It contains -the catalog, each `workspace.yaml`, curated evidence, annotations, and other repository-owned -content. Authors validate it using source-side tooling and publish it through their normal Git -workflow to GitHub, GitLab, Gitea, or another standards-compatible server. - -### ThothII installation - -The installation descriptor selects the Git remote, branch, and one read-only authentication -transport. SSH uses a read-only deploy key plus pinned known hosts. HTTPS uses a read-only deploy -token and may provide a private CA. Secret values remain outside versioned configuration. - -The installer performs a sanitized `git ls-remote` preflight. Credentials embedded in a remote URL -are rejected. The API exposes only a normalized repository identity: host, repository path, branch, -transport, active commit, and synchronization state. - -### ThothII runtime - -The local Git checkout, candidate validation area, immutable snapshots, and active state are -application-owned. They are read-only from the workspace-management API. A pull fetches a candidate -revision, validates the complete repository, and atomically activates it only if valid. A failed -candidate never replaces the last valid active revision. - -ThothII never generates or reconciles files back into the checkout and never invokes Git commit or -push. Generated operational artifacts live under application data, not in the source repository. - -## Repository synchronization states - -A repository refresh has these states: - -- `syncing`: fetching and validating a candidate revision; -- `active`: the candidate passed validation and became the active immutable revision; -- `invalid_candidate`: Git succeeded but repository validation failed; the previous revision stays active; -- `unavailable`: Git or authentication failed; the previous revision stays active; -- `empty`: no valid revision has ever been activated. - -Validation is atomic at repository-commit level. A malformed catalog, descriptor, evidence tree, or -cross-file reference rejects the complete candidate revision. - -## Runtime secret model - -### Requirement discovery - -The workspace descriptor contains connector type, authentication method, and non-secret logical -configuration. It never contains secret values or host filesystem paths. Connector adapters define -the secret fields required by each supported authentication method. For example: - -- PostgreSQL `username_password` requires `username` and `password`; -- REST `bearer` requires `api_key`; -- SSH tunnel authentication requires the connector password and SSH private key; -- Evidence HTTP signed URLs and static S3 credentials contribute their own secret requirements. - -Requirements have stable identifiers scoped by workspace and connector. Labels, descriptions, -input kinds, and required/optional status come from trusted application code rather than repository -HTML or executable metadata. - -### Persistent encrypted store - -The backend owns a `WorkspaceSecretStore` abstraction. The first implementation is a local encrypted -vault in application-managed persistent storage. Each secret is encrypted with authenticated -encryption and bound to its installation, workspace, connector, and field identifier as associated -data. Plaintext values never appear in Git, API responses, logs, error messages, diagnostics, or -browser storage. - -The installation bootstraps one vault key independently from workspace content. Deployment tooling -owns its platform-specific provisioning; the workspace schema and GUI never contain filesystem -paths. The storage interface allows a future Vault, cloud secret manager, or OS keychain provider -without changing workspace descriptors or API consumers. - -When an existing file-oriented harness connector needs a credential, the backend materializes it as -a restrictive temporary file in an application-owned runtime directory. Its lifetime is tied to the -diagnostic or runtime lease and it is removed on release. Persistent storage contains ciphertext -only. - -### Secret API - -For a selected workspace the API returns requirement metadata and status only: - -```json -{ - "workspaceId": "psd-clinical", - "state": "configuration_required", - "requirements": [ - { - "id": "dwh.password", - "connector": "dwh", - "label": "Database password", - "input": "password", - "required": true, - "configured": false - } - ] -} -``` - -A write request contains values only for the selected requirement identifiers. The response returns -status, never values. A delete operation forgets a configured value. Authorization is deliberately -deferred; the current authenticated application user may manage runtime workspace secrets. - -Workspace readiness is derived as follows: - -- `invalid`: repository structure or descriptor is invalid; -- `configuration_required`: structurally valid but required runtime values are missing; -- `ready`: required values exist but connectivity has not yet passed or is stale; -- `verified`: the most recent connector diagnostic passed for the active revision and current secret generation. - -Changing or deleting a secret invalidates the previous diagnostic result. - -## Browser behavior - -Workspace management is a two-level read-only interface occupying at least 60 percent of viewport -width and height. - -Level 1 explains the source/runtime separation and displays: - -- normalized repository host and path; -- configured branch and read-only transport; -- active revision and last synchronization result; -- `Update workspace repository`, which fetches, validates, and conditionally activates a revision; -- the workspace list, with selection required for workspace-specific actions. - -There is no Import bundle, Export bundle, Create, Edit, Delete, Publish, or conflict-resolution -operation. There are no browser-persisted workspace drafts or preferences. - -Level 2 for the selected workspace explains and displays: - -- immutable source identity and validation result; -- required runtime configuration grouped by connector; -- secret-entry controls whose values are write-only; -- `Save secrets`, `Forget` per configured value, and `Test workspace connection`; -- clear consequences for each button and a reminder that source changes must be committed and pushed - by an author outside ThothII before repository update. - -The browser keeps form values only in component memory and clears them after submission or dialog -close. It never receives saved secret values. - -## Compatibility and migration - -Existing Git author settings, publish endpoints, bundle endpoints, generated-document -reconciliation, bootstrap catalog slots, and browser draft storage are removed. Existing environment -bindings may be read during a bounded migration period only to seed non-secret connector values; -secret file paths are not part of the new public workspace contract. - -Session manifests continue to pin an immutable validated workspace revision. An already running -session keeps its acquired runtime lease; new or resumed work resolves the current encrypted secret -generation and fails closed when required credentials are unavailable. - -## Failure handling and security - -- Repository and vault errors use stable sanitized codes and never echo remotes with user info, - credential paths, secret identifiers that are not safe to disclose, or secret values. -- Vault writes are atomic and authenticated; corrupted ciphertext fails closed. -- Secret comparison uses no read API. Updating a secret is always a blind replacement. -- The backend applies request-size and field-count limits and rejects unknown requirement IDs. -- Temporary plaintext files use restrictive permissions, trusted directories, no-follow opens, and - deterministic cleanup. -- Git credentials are installation-only, read-only, and never sent to the frontend. - -## Verification - -Backend tests cover repository read-only behavior, atomic candidate activation, remote sanitization, -vault encryption and corruption, requirement discovery, blind secret writes/deletes, materialization -cleanup, readiness transitions, and absence of publish/bundle routes. - -Frontend tests cover the two-level explanation, viewport dimensions, repository identity, selection -gating, dynamic secret forms, write-only behavior, status changes, and absence of local-storage, -import, export, editing, and publishing controls. - -Deployment and CLI tests cover required remote/branch configuration, one read-only Git transport, -sanitized remote preflight, vault-key provisioning, and removal of Git author/write configuration. diff --git a/docs/plans/2026-08-18-thothii-authentication-acceptance-and-psd-deployment.md b/docs/plans/2026-08-18-thothii-authentication-acceptance-and-psd-deployment.md deleted file mode 100644 index 906fd942..00000000 --- a/docs/plans/2026-08-18-thothii-authentication-acceptance-and-psd-deployment.md +++ /dev/null @@ -1,408 +0,0 @@ -# ThothII Authentication Acceptance and PSD Deployment Plan - -> **For agentic workers:** execute one task at a time, record its completion criteria, and stop at every documented approval boundary. - -**Goal:** Validate local and OIDC authentication on macOS, deploy the exact feat/thoth-auth candidate to the Aritmolab/PSD server before merging it into main, and complete end-to-end acceptance with remote Authentik. - -**Architecture:** Test the candidate first as a standalone local installation. Then install the same immutable Git revision on the existing PSD installation with the installation-aware tht lifecycle, leaving main untouched. Authentik provides OIDC login and a mandatory direct groups claim; ThothII maps exact external groups to roles and validates mapped groups through the Authentik catalog API. - -**Tech Stack:** macOS, Docker Desktop, Docker Compose, native host `tht` plus Python workflow `tht`, local Argon2id authentication, generic OIDC Authorization Code + PKCE, Authentik, PSD workspace registry, reverse proxy/TLS. - ---- - -## Scope and release rules - -Do not merge feat/thoth-auth into main until every mandatory gate in Task 9 is PASS and the PSD owner accepts the evidence. - -Capture one candidate revision and reuse it everywhere: - -```bash -export CANDIDATE_SHA="$(git rev-parse HEAD)" -git fetch origin feat/thoth-auth -test "$CANDIDATE_SHA" = "$(git rev-parse origin/feat/thoth-auth)" -git show -s --format='%H%n%P%n%s' "$CANDIDATE_SHA" -git status --short --untracked-files=all -``` - -Never deploy a moving branch name without checking its resolved SHA. Never put passwords, OIDC client secrets, Authentik API tokens, cookies, authorization headers, raw ID tokens, or password hashes in Git, shell history, screenshots, logs, or evidence. - -Use protected operator values for <PUBLIC_URL>, <OIDC_ISSUER>, <AUTHENTIK_BASE_URL>, <OIDC_CLIENT_ID>, <THT_BIN>, <INSTALLATION>, <WORKSPACE_ID>, and <OLD_SHA>. - -The Authentik contract is mandatory: a direct non-empty JSON array claim named groups; exact groups TOT Users and TOT Admin; mappings TOT Users -> user and TOT Admin -> admin; and a separate group-view-only API service account exposed only as THT_AUTHENTIK_API_TOKEN. Extra upstream groups are valid and silently ignored. - -## Task 0: Freeze the candidate and collect approvals - -**Files:** None; record results in the acceptance report in Task 9. - -Run from the candidate worktree: - -```bash -git diff --check -go test ./... -count=1 -go test -race ./... -go vet ./... -go build ./... -``` - -Expected: all commands pass, the candidate is pushed, and existing evidence is bound to the same SHA. A historical result from another revision is not evidence for this run. - -Before touching PSD, obtain the maintenance window, server access, public URL, Authentik provider details, protected secret locations, test identities for ordinary/admin/unmapped users, and permission to test PSD DWH/Evidence connections. - -## Task 1: Prepare and start the local macOS installation - -**Files:** - -- Read: docs/install/local.md -- Read: docs/install/authentication-local.md -- Read: docs/testing/authentication-manual-acceptance.md -- Use: an untracked local installation descriptor and protected secret/password files - -**Step 1: Verify prerequisites** - -```bash -docker version -docker compose version -bash scripts/verify-line-endings.sh -``` - -Expected: Docker Desktop and Compose are available and line-ending validation passes. - -**Step 2: Build and configure** - -```bash -bash scripts/build-local.sh -bash scripts/build-tht.sh -tht setup --profile local -``` - -For an existing installation, do not overwrite data; run tht --installation <local-installation.yaml> update --check-only instead of setup. - -**Step 3: Start and inspect** - -```bash -tht --installation <local-installation.yaml> start --build -tht --installation <local-installation.yaml> status -tht --installation <local-installation.yaml> doctor --json -curl --fail http://127.0.0.1:8080/health -curl --fail http://127.0.0.1:8787/health -``` - -Expected: core, frontend, qdrant, embedding, and the completed model initializer are healthy; doctor includes authentication after configuration and before services. - -## Task 2: Configure and test local login - -**Files:** - -- Read: docs/install/authentication-local.md -- Modify only protected installation state through tht auth configure and tht auth user - -**Step 1: Bootstrap the administrator** - -```bash -tht --installation <local-installation.yaml> auth configure \ - --mode local --public-url http://127.0.0.1:8080 \ - --admin-user <local-admin> --admin-display-name <display-name> \ - --password-file <protected-password-file> -``` - -Remove the temporary password file immediately. Expected: non-secret auth.yaml is created and the user store contains Argon2id hashes, never plaintext passwords. - -**Step 2: Add and inspect a normal user** - -```bash -tht --installation <local-installation.yaml> auth user add <local-user> --role user --display-name <display-name> --password-file <protected-password-file> -tht --installation <local-installation.yaml> auth status --json -tht --installation <local-installation.yaml> auth check --json -``` - -Expected: pristine redacted JSON and no credential, hash, or session secret in output. - -**Step 3: Test browser authorization** - -At http://127.0.0.1:8080, in a private browser profile: - -1. Verify unauthenticated access reaches login and protected routes are denied. -2. Log in as the normal user and verify application/session routes work. -3. Verify Pi Management and other admin-only operations return HTTP 403 or are not exposed. -4. Log out and verify the session is invalidated. -5. Log in as the administrator and verify admin-only routes work. - -Expected: ordinary users authenticate without receiving admin permissions; administrators receive the configured admin permission set. - -**Step 4: Test account failure paths** - -Use auth user disable, enable, set-password, and logout-all user --yes on the test user. Test a wrong password and refresh the old browser session after logout-all. - -Expected: generic safe failures, disabled login rejection, re-enabled login success, and forced reauthentication. The last enabled administrator cannot be disabled or demoted. - -## Task 3: Test remembered sessions and local recovery - -**Files:** - -- Read: docs/architecture/authentication.md, Browser sessions -- Read: docs/install/authentication-local.md, Session behavior and recovery - -**Step 1: Test browser restart** - -Log in as the normal user with Remember me, close the browser completely, reopen it, and revisit the application. - -Expected: the session survives within the 7-day idle / 30-day absolute limits. Do not record the cookie. - -**Step 2: Test ThothII restart** - -```bash -tht --installation <local-installation.yaml> stop -tht --installation <local-installation.yaml> start -``` - -Expected: the remembered session remains valid after backend restart. - -**Step 3: Test invalidation** - -Change the test user password or role, and separately run auth user logout-all user --yes. Refresh after each operation. - -Expected: affected sessions are rejected and reauthentication is required; configuration revision changes invalidate all sessions. - -Go/no-go: do not proceed to PSD if local login, role separation, logout, or remembered-session behavior fails. - -## Task 4: Snapshot the current PSD installation - -**Files:** - -- Read: docs/install/server.md -- Read: docs/install/server-workspace-registry.md -- Use: protected server operator and backup locations - -**Step 1: Capture live state** - -```bash -THT_BIN=<THT_BIN> -INSTALLATION=<INSTALLATION> -"$THT_BIN" --installation "$INSTALLATION" status -"$THT_BIN" --installation "$INSTALLATION" doctor -"$THT_BIN" --installation "$INSTALLATION" pi status -"$THT_BIN" --installation "$INSTALLATION" pi doctor -git -C /srv/thothii/source/ThothII status --short --untracked-files=all -git -C /srv/thothii/source/ThothII rev-parse HEAD -``` - -Save the live SHA as <OLD_SHA> and capture image identities, workspace registry status, and maintenance/recovery state. Stop if the checkout is dirty or recovery is pending. - -**Step 2: Drain and back up** - -Announce maintenance, close the reverse proxy or show its maintenance page, drain active work, and stop through tht. Create the protected, checksummed backup specified in docs/install/server.md, including runtime trees and PSD PostgreSQL/session data where applicable. Back up credentials separately. Never run docker compose down --volumes. - -**Step 3: Check preconditions** - -```bash -git -C /srv/thothii/source/ThothII config --local core.autocrlf false -bash /srv/thothii/source/ThothII/scripts/verify-line-endings.sh -"$THT_BIN" --installation "$INSTALLATION" update --check-only -``` - -Expected: descriptor, protected secrets, Pi-state mount, workspace repository binding, and Compose render remain valid before source changes. - -## Task 5: Deploy the feature revision to PSD without merging main - -**Files:** - -- Server source checkout: /srv/thothii/source/ThothII -- Server operator binary: protected THT_BIN path -- Server installation descriptor and secret files: unchanged paths unless a reviewed auth update is required - -**Step 1: Select the exact candidate** - -```bash -git -C /srv/thothii/source/ThothII fetch origin feat/thoth-auth -git -C /srv/thothii/source/ThothII switch --detach <CANDIDATE_SHA> -test "$(git -C /srv/thothii/source/ThothII rev-parse HEAD)" = "<CANDIDATE_SHA>" -git -C /srv/thothii/source/ThothII status --short --untracked-files=all -``` - -Do not merge or rebase main. The running installation is intentionally based on the detached feature revision until acceptance completes. - -**Step 2: Build candidate artifacts** - -```bash -cd /srv/thothii/source/ThothII -bash scripts/build-local.sh -THT_THT_OUTPUT_DIRECTORY=/srv/thothii/operator/build-output bash scripts/build-tht.sh -``` - -Install the architecture-appropriate candidate tht only after its build succeeds. Keep the old operator binary recoverable. - -**Step 3: Start and verify the candidate** - -```bash -"$THT_BIN" --installation "$INSTALLATION" update --check-only -"$THT_BIN" --installation "$INSTALLATION" start --build -"$THT_BIN" --installation "$INSTALLATION" status -"$THT_BIN" --installation "$INSTALLATION" doctor --json -curl --fail http://127.0.0.1:8080/health -"$THT_BIN" --installation "$INSTALLATION" pi doctor -"$THT_BIN" --installation "$INSTALLATION" pi test -``` - -Expected: candidate frontend/core and internal services are healthy, no data volume was replaced, and the candidate SHA is recorded. Liveness alone is not release approval. - -## Task 6: Configure and validate remote Authentik - -**Files:** - -- Modify protected server authentication state through tht auth configure -- Modify protected secret entries THT_OIDC_CLIENT_SECRET and THT_AUTHENTIK_API_TOKEN -- Read: docs/install/authentik.md and docs/install/authentication-oidc.md - -**Step 1: Verify Authentik** - -Verify the OAuth2/OIDC callback exactly <PUBLIC_URL>/api/auth/oidc/callback, scopes openid/profile/email, direct groups array mapping, exact groups TOT Users and TOT Admin, and a separate group-view-only catalog service account. Inspect a disposable identity without copying its token. - -**Step 2: Configure the ThothII mapping** - -```bash -tht --installation "$INSTALLATION" auth configure \ - --mode oidc --public-url <PUBLIC_URL> \ - --issuer <OIDC_ISSUER> --client-id <OIDC_CLIENT_ID> \ - --authentik-base-url <AUTHENTIK_BASE_URL> \ - --user-group 'TOT Users' --admin-group 'TOT Admin' -``` - -Secrets are read from protected files, never command-line arguments. Confirm the non-secret mapping is: - -```yaml -authorization: - groupRoles: - TOT Users: [user] - TOT Admin: [admin] -``` - -**Step 3: Run static and live checks** - -```bash -tht --installation "$INSTALLATION" auth status --json -tht --installation "$INSTALLATION" auth check --json -tht --installation "$INSTALLATION" auth check --interactive -tht --installation "$INSTALLATION" doctor --json -``` - -Expected: configuration, discovery, issuer, JWKS, client-secret access, catalog access, and exact existence of every mapped group pass. Doctor lists authentication after configuration and before services. No output contains credentials or bearer tokens. - -A missing mapped group must fail with redacted oidc_mapped_group_missing. Unmapped groups produce neither error nor warning. Missing, indirect, malformed, or overage-style groups claims fail closed. - -**Step 4: Reload if required** - -If configuration requires process reload: - -```bash -"$THT_BIN" --installation "$INSTALLATION" pi restart --yes --drain -``` - -Repeat authentication, doctor, and health checks. Do not substitute raw Compose commands. - -## Task 7: Test PSD browser login and authorization - -**Files:** - -- Read: docs/testing/authentication-manual-acceptance.md -- Evidence: redacted report from Task 9 - -**Step 1: Ordinary user** - -In a private profile, authenticate with an identity in TOT Users but not TOT Admin. Verify callback success, application/session routes, denial of Pi Management/admin operations, opaque HttpOnly ThothII cookie, no bearer token in Web Storage, and logout invalidation. - -**Step 2: Administrator** - -Authenticate with TOT Admin. Verify Pi Management and allowed workspace-management operations. Access must derive from the exact mapped group, not a client-supplied header or browser-local flag. - -**Step 3: Unmapped and malformed groups** - -Authenticate with a valid token containing no mapped group. Expected: login may complete, but protected operations return 403 with no warning. Use a disposable provider mapping that omits or corrupts groups; expected: generic HTTP 401 oidc_callback_failed, with no internal claim details exposed. - -**Step 4: Provider outage/group drift** - -During a controlled window, make discovery/JWKS unavailable or rename a mapped group, run the CLI check, and restore it immediately. Expected: redacted fail-closed diagnostics followed by a successful check after restoration. Do not leave production broken. - -## Task 8: Test PSD workspace validation and real connections - -**Files:** - -- Read: docs/install/server-workspace-registry.md -- Use: authenticated PSD browser sessions - -**Step 1: Validate the workspace and authentication from the host CLI** - -```bash -"$THT_BIN" --installation "$INSTALLATION" \ - workspace inspect --workspace "$WORKSPACE_ID" --json -"$THT_BIN" --installation "$INSTALLATION" auth check --json -``` - -Expected: the workspace registry is ready, authentication readiness passes, and output is redacted while identifying the active workspace revision. - -**Step 2: Verify the application boundary** - -Open the configured public URL, authenticate with the approved identity, and verify that the application reaches the selected workspace without unexpected `401`/`403` responses. Keep DWH/Evidence connection tests read-only and use only the existing approved smoke question. - -**Step 3: Test ordinary-user authorization** - -Log in as TOT Users. Confirm inspection follows ordinary permissions while validation, secret mutation, and connection tests remain unavailable unless explicitly granted. - -**Step 4: Run a harmless end-to-end smoke** - -As an authorized PSD user, create or resume one harmless known-good session: - -```text -browser login -> same-origin API -> workspace readiness -> Pi/core -> result -> logout -``` - -Do not run mutating production queries. Preserve only a session ID and redacted outcome if approved. - -## Task 9: Close acceptance, rollback if needed, and decide merge readiness - -**Files:** - -- Create: docs/testing/evidence/2026-08-18-thothii-authentication-psd-acceptance.md or the approved external evidence location -- Read: docs/install/server.md and docs/contracts/tht-pi.md - -**Step 1: Mandatory gates** - -| Gate | Required evidence | -|---|---| -| Candidate identity | Local and server SHA exactly match pushed feat/thoth-auth | -| Local startup | macOS Compose, doctor, health, and Pi smoke pass | -| Local auth | Bootstrap, ordinary/admin roles, logout, bad password, disable/enable, logout-all pass | -| Local session | Remembered session survives browser and ThothII restart; revisions invalidate it | -| Server safety | Old SHA/image/status captured; backup checksummed; maintenance/drain completed | -| Candidate deploy | Server candidate status/doctor/health pass | -| Authentik | Direct groups claim, issuer/JWKS, secrets, catalog, and mapped groups pass | -| OIDC authorization | ordinary, admin, unmapped, malformed, logout, and outage cases pass | -| Workspace integration | `tht workspace inspect` and `tht auth check` pass; the authenticated application reaches the selected workspace | -| PSD smoke | One harmless known-good session completes | -| Hygiene | No secrets, tokens, cookies, hashes, or raw claims in evidence | - -**Step 2: Write the redacted report** - -Include candidate SHA, old SHA, timestamps, commands, browser cases, redacted diagnostic/HTTP codes, Authentik issuer/client/group names, workspace ID/revision, backup/checksum location, rollback decision, and unrelated CI failures. Never include secret values, raw tokens, cookies, or hashes. - -**Step 3: Roll back a failed candidate** - -1. Keep the proxy closed and preserve .tht/<installation-id>/ recovery state. -2. Do not use tht pi rollback as the whole-application rollback; it addresses only Pi lifecycle images. -3. Stop with tht. -4. Return the source checkout to <OLD_SHA>, rebuild old application/operator artifacts, and start through the same descriptor. -5. Run update --check-only, status, doctor, health, Pi smoke, workspace diagnostics, and one harmless session. -6. For ambiguous recovery, leave maintenance active and follow pi maintenance status / pi maintenance recover --yes. Never delete volumes, selectors, or recovery files to force progress. - -Expected: the previous application serves again with prior data and workspace state intact. Record the failure and do not merge. - -**Step 4: Reopen traffic** - -After every gate passes, restore the reverse proxy, repeat one unauthenticated redirect and one authorized public login, and confirm only the proxy is externally reachable. - -**Step 5: Merge decision** - -Merge only after PSD owner acceptance, exact-SHA evidence, no unresolved auth/workspace/provider/deployment gate, and an accepted rollback path. If the merge creates a new commit, repeat Tasks 0, 5, 6, and 7 against the merge SHA. - -## Handoff checklist - -Deliver the redacted report, local result/SHA, PSD candidate SHA/images, Authentik provider and group mapping confirmation, workspace validation/connection results, backup/rollback status, and an explicit READY TO MERGE or NOT READY TO MERGE decision. diff --git a/docs/plans/2026-08-20-psd-server-deployment-program-design.md b/docs/plans/2026-08-20-psd-server-deployment-program-design.md deleted file mode 100644 index c9bc469d..00000000 --- a/docs/plans/2026-08-20-psd-server-deployment-program-design.md +++ /dev/null @@ -1,370 +0,0 @@ -# PSD Server Deployment Program — Design - -**Date:** 2026-08-20 - -**Status:** Approved by the owner - -**Owner sequencing amendment (2026-08-21):** the Mac `rest_api` acceptance and revocation of -`legacy-shared` are deferred to one mandatory pre-Project-B gate. This permits the bounded survey -and static, non-mutating Project A private preparation to proceed without changing the Mac. It does not -authorize stopping the legacy stack, starting the new stack, opening ingress, or beginning Project B. - -**Design-time application baseline:** `main` at `5c0dc8c` (execution must freeze and record the -then-current `origin/main` SHA) - -**Design-time workspace baseline:** `tht-workspace-psd/main` at `bfbabf9` (execution must freeze and -record the then-current remote SHA) - -## Purpose - -Replace the unused legacy ThothII installation on the PSD server with the current application, -recovering only useful configuration and rebuilding runtime state from canonical sources. Complete -the work as two independently accepted projects: - -1. deploy and prove ThothII with local authentication, a direct read-only PSD DWH connection, - internal Qdrant, and internal Ollama; -2. only after Project A passes, integrate the accepted installation with the server's Authentik, - Nginx, load balancer, and the existing Aritmolab sidebar link. - -The program must be executable by the Sol LLM from a terminal local to the server. It must test -each step, retain redacted evidence, stop on unsafe uncertainty, and include a separate human -manual-test document for each project. - -## Binding decisions - -- The legacy application may be unavailable for days. Service continuity is not a goal. -- The new source clone is prepared beside the old source directory. The old and new stacks are not - kept running simultaneously: inventory and backup happen first, the old stack is stopped, and - only then is the new stack started. -- Keep the old directory, configuration, containers, images, and data only as a temporary recovery - boundary until the real Aritmolab journey passes Project B. They are disposable after Project B - automated, human, and owner PASS. Do not migrate legacy application sessions, Qdrant data, - Ollama caches, or derived indexes. -- Do not create a host `thothii` user or group. Keep UID/GID `10001:10001` as the image's unmapped - numeric runtime identity and confine numeric ownership to the new installation's writable bind - trees. Stop if either number becomes mapped to a host account before installation. -- Recover configuration only: endpoints, non-secret policy, provider/model selection, relevant - paths, and references to protected credentials. Never copy an old setting without validating it - against the current contract. -- Use the canonical ThothII Compose distribution and reviewed installation-local overrides. Do not - modify an unrelated global Compose project or blindly adapt the legacy Compose file. -- Lifecycle operations use the installation-aware native `tht` CLI, not raw Compose commands. -- Never use `docker compose down --volumes`, global prune operations, broad recursive deletion, or - secret-bearing command arguments. - -## Repository and workspace model - -There are two Git sources with different responsibilities: - -- `ThothII` contains application code and non-secret deployment examples. -- `tht-workspace-psd` contains the shared, credential-free PSD workspace source. - -The workspace repository contains one logical workspace, `psd-clinical`. Its schema-v3 descriptor -will declare both supported DWH transports: - -```yaml -supported_transports: [rest_api, postgres_direct] -``` - -The Mac installation continues to select `rest_api`. The PSD server selects `postgres_direct`. -Endpoint values, user names, passwords, secret-file paths, machine paths, and Git credentials stay -in each installation's protected bindings and never enter the workspace repository. Evidence, -curated annotations, language, model policy, and semantic-index contract remain shared. - -Each installation owns its own Qdrant collection contents, Ollama model cache, preprocessing state, -and sessions even though both consume the same reviewed workspace commit. - -## Program structure - -The program consists of one non-mutating common survey followed by two independently gated -projects: - -```text -Common Survey PASS for Project A private scope - -> Project A automated PASS - -> Project A human PASS - -> Mac REST acceptance - -> 48-hour dual-key observation covering two 03:00 ETL cycles - -> legacy-shared revocation and negative proof - -> full pre-Project-B survey PASS - -> explicit Project B authorization - -> Project B automated PASS - -> Project B human PASS - -> final cutover acceptance -``` - -Project B must not start from a partial or assumed Project A result. - -## Common Survey - -The survey is a prerequisite, not a third implementation project. It runs before any server -mutation and produces a redacted report, topology map, change-scope inventory, unknowns list, and -GO/NO-GO decision. - -Sol inventories from the server-local terminal: - -- operating system, architecture, Docker and Compose versions, available CPU/RAM/disk, and clock; -- legacy ThothII source, SHA, dirty state, images, containers, networks, ports, volumes, mounts, - health, installation state, and recovery state; -- effective Compose rendering and ownership of every relevant file; -- DWH database/schema, direct listener, TLS, read-only role, and reachability from containers; -- Supabase/PostgreSQL topology, schema conventions, exposed PostgREST schemas, migration policy, - backup mechanism, and suitable role boundaries; -- Pi provider/model policy, credential references, external LLM reachability, and current versions; -- Nginx effective configuration, ThothII virtual host/location, upstream, forwarded headers, SSE - settings, certificate metadata, certificate generation/renewal, and rollback files; -- load-balancer routes, health checks, allowlist capability, TLS boundary, and configuration owner; -- Aritmolab deployment, networks, homepage, sidebar source, current ThothII destination, and release - procedure; -- Authentik version, deployment, current Aritmolab integration, provider conventions, group - conventions, backup/export procedure, API access, and credential locations; -- public DNS/origin that the final sidebar link must preserve; -- protected files by path, ownership, mode, and readability only, without printing their contents. - -The survey may use hashes, metadata, redacted renders, and permission checks. It must not emit -passwords, bearer tokens, API keys, cookies, OIDC client secrets, private keys, password hashes, or -raw identity tokens. If credential discovery fails, the owner may help locate the existing -Authentik credentials. - -## Project A — Standalone Server Acceptance - -### Runtime architecture - -Project A installs a clean five-service stack: - -```text -operator terminal or local headless browser - -> frontend - -> core + Pi + workflow harness - -> direct read-only PSD PostgreSQL DWH - -> internal Qdrant - -> internal Ollama (qwen3-embedding:0.6b, 1024 dimensions) - -> local filesystem work-session storage -``` - -The stack uses local authentication. It must not be publicly reachable. Core, Qdrant, and Ollama -remain private; the frontend binds to loopback unless the optional restricted test route below is -proved safe. - -### Preparation and cutover - -1. Freeze exact application and workspace SHAs and require clean source trees. -2. Publish and validate the multi-transport `psd-clinical` descriptor through the curator workflow. -3. Preserve the Mac REST binding unchanged; its live acceptance is deferred to the mandatory - pre-Project-B gate. -4. Prepare the new source clone and protected operator/runtime directories beside the old source. -5. Extract only approved configuration facts from the legacy installation. -6. Record and verify the exact restart recipe for the legacy stack. No data backup is required - because the owner declared legacy sessions and configuration disposable; the still-present - containers, images, source, and data are the temporary rollback boundary. -7. Stop the legacy stack without deleting its source, configuration, images, or data. -8. Build/install the current native `tht` and application images from the frozen source. -9. Configure local authentication, direct DWH bindings, Pi/provider access, and the Git workspace. -10. Start through `tht`, then validate health, authentication, workspace, workflow, and Pi. -11. Rebuild schema/Evidence preprocessing, Qdrant contents, and the Ollama model cache from - canonical sources. Prove the complete preprocessing rerun is idempotent. - -### Optional restricted test route - -If and only if the survey proves that the load balancer can enforce a test-operator allowlist -before a request reaches ThothII, Project A may use a temporary private hostname to exercise the -same network path as production: - -```text -authorized operator -> allowlisted load-balancer route -> Nginx -> frontend -> core/local auth -``` - -The route has a distinct hostname, no Aritmolab sidebar link, a certificate created by the existing -managed mechanism, correct forwarded-origin and SSE behavior, and a negative test from an -unauthorized source. Its local-auth `publicUrl` matches the private test origin. It is not a public -production route and must be removed after Project B. - -If isolation cannot be demonstrated, Sol must not approximate it or misdeclare -`THOTH_PUBLIC_EXPOSURE=false`; tests run against loopback from the local terminal instead. - -### Acceptance - -Project A requires: - -- exact source identity and reproducible build evidence; -- healthy frontend, core, Qdrant, embedding, and completed model initializer; -- local authentication checks including admin/user separation, wrong password, logout, - disable/enable, invalidation, and restart persistence; -- direct DWH connectivity with a demonstrably read-only runtime identity; -- active `psd-clinical` at the expected Git revision; -- compatible Qdrant collection/index contract and correct Ollama model/dimensions; -- complete and idempotent DWH, schema, annotation, and Evidence preprocessing; -- one harmless real PSD work session completed through F1-F8, ending in read-only validated SQL; -- persisted manifest, artifacts, reviewer decisions, and final SQL inspection; -- optional private-route positive and negative isolation evidence when that route is used; -- completed human manual-test report with an explicit PASS. - -The Mac row may be recorded only as `DEFERRED_PRE_PROJECT_B` under the dated owner amendment. It is -not part of the private server acceptance, but it must become PASS before Project B starts. - -Project A does not modify the production Aritmolab sidebar, production public route, or Authentik. - -## Project B — Authentik and Aritmolab Integration - -Project B begins only from the frozen, accepted Project A source, images, workspace revision, and -PASS report. It also requires the deferred Mac REST acceptance, the full 48-hour observation -window (including two scheduled 03:00 ETL cycles), revocation of `legacy-shared`, proof that the -legacy credential receives `401`, and the full pre-Project-B survey gate. - -### Final request and data flow - -```text -user - -> aritmolab.policlinicosandonato.com - -> Aritmolab homepage/sidebar - -> load balancer - -> Nginx/TLS - -> ThothII frontend and same-origin /api - -> ThothII OIDC Authorization Code + PKCE with Authentik - -> PostgreSQL thoth_sessions schema for owned work sessions - -> direct read-only datawarehouse schema for clinical queries - -> internal Qdrant and Ollama -``` - -Nginx terminates/proxies according to the observed deployment but does not add a second -`auth_request` in front of ThothII. ThothII performs generic OIDC directly. Nginx preserves the -public host and HTTPS scheme, forwards the callback path unchanged, and supports SSE without -buffering or premature timeouts. - -### Authentik configuration - -Before mutation, export or back up the relevant Authentik configuration. Locate existing protected -administrative/API credentials without exposing them. Create or adapt: - -- one OAuth2/OIDC provider and one ThothII application; -- the exact callback `<public-origin>/api/auth/oidc/callback`; -- `openid`, `profile`, and `email` scopes; -- a direct, non-empty JSON string array claim named `groups`; -- exact user/admin group mappings selected after the survey; -- a separate group-view-only service account/API token for ThothII diagnostics. - -Dedicated `TOT Users` and `TOT Admin` groups are the default unless the survey finds existing groups -with exactly the intended semantics and the owner approves their reuse. Additional groups are -ignored. Missing, malformed, indirect, or ambiguous configured groups fail closed. - -### Supabase session storage - -Do not create a separate PostgreSQL database. Use the server's existing Supabase PostgreSQL -database and isolate ThothII work sessions in the dedicated `thoth_sessions` schema. This schema is -distinct from the clinical `datawarehouse` schema. - -- A one-shot migrator role owns only the required schema migration privileges. -- Core receives only the restricted runtime role, never the migrator credential. -- Forced RLS and application ownership checks isolate sessions by Authentik principal. -- The schema stores principals/preferences, manifests, phase artifacts, review decisions, and - audit records. -- Chat and live SSE output remain ephemeral; semantic vectors remain in Qdrant; clinical data - remains in `datawarehouse`; browser authentication sessions remain in protected auth state. -- `thoth_sessions` must not be added to Supabase/PostgREST exposed schemas. -- The direct runtime connection uses the TLS/CA contract required by the current application. - -### Safe activation order - -1. Back up the accepted Project A operator/auth configuration and every external configuration to - be changed. -2. Prepare Authentik objects without exposing the new route. -3. Run and verify additive session-schema migrations; require no pending or drifted migrations. -4. Prepare OIDC secrets and non-secret configuration in protected installation state. -5. Validate Authentik discovery, issuer/JWKS, catalog access, and mapped groups. -6. Validate Nginx, certificate, load-balancer route, callback, forwarded headers, and SSE while - production traffic remains closed. -7. Start ThothII in OIDC/public server mode with PostgreSQL session storage. -8. Open the final load-balancer route. -9. Preserve or update the Aritmolab sidebar link so the established user journey remains intact. -10. Complete automated and human acceptance, then remove the Project A temporary route. - -### Acceptance - -Project B requires: - -- successful redacted static, live, and interactive authentication diagnostics; -- trusted certificate chain, correct public origin, callback, and proxy headers; -- proven load-balancer/Nginx routing and SSE operation; -- successful migration status, RLS/role tests, and proof that `thoth_sessions` is not REST-exposed; -- ordinary, administrator, unmapped, malformed-claim, logout, and controlled provider-failure - cases; -- login to Aritmolab followed by the sidebar link to ThothII without a second credential prompt; -- no raw OIDC token in browser storage, logs, diagnostics, or evidence; -- session ownership and administrator-boundary tests; -- one harmless F1-F8 PSD session under an OIDC identity; -- tested rollback and a completed human manual-test report with an explicit PASS. - -## Error handling and stop rules - -Every executable step follows: - -```text -precondition -> action -> verification -> redacted evidence -> checkpoint -``` - -Sol stops and requests owner help rather than improvising when it encounters: - -- a dirty or unidentified source checkout; -- uncertain ownership of Compose, Nginx, load-balancer, Aritmolab, or Authentik configuration; -- missing or insufficient credentials; -- an unsafe secret path or risk of secret disclosure; -- an unverifiable backup or rollback path; -- a DWH identity that is not demonstrably read-only; -- a change that would affect unrelated Nginx virtual hosts or other applications; -- an unprovable temporary-route restriction; -- Supabase migration drift, excessive roles, or unintended REST exposure; -- a material difference between the surveyed server and this design. - -Unchanged external state is not failure. Sol records the observation and continues only when the -current gate is satisfied. - -## Rollback boundaries - -- **Project A:** stop the new stack and restart the still-present legacy installation. No new - volume is copied into it and no promise is made to retain it after Project B PASS. -- **Project B ingress:** close the public route first, then restore the prior Nginx, - load-balancer, certificate reference, and sidebar configuration. -- **Project B application:** return to the accepted Project A local-auth operator configuration - while the public route remains closed. -- **Authentik:** initially disable new objects instead of deleting them; retain the pre-change - export until final acceptance. -- **Supabase:** migrations are additive. Rollback does not automatically drop `thoth_sessions` or - destroy evidence; destructive cleanup requires a separate explicit decision. -- **Workspace Git:** publish the multi-transport change as an isolated commit and retain the - previous revision. A rejected candidate never replaces the installation's last valid snapshot. - -## Documents and evidence - -The implementation-planning phase creates: - -1. a general execution program with cross-project gates; -2. a survey checklist and report template; -3. an executable Project A plan for Sol; -4. a plain-language Project A manual-test guide; -5. a Project A evidence/PASS template; -6. an executable Project B plan for Sol; -7. a plain-language Project B manual-test guide; -8. a Project B evidence/PASS template. - -Sol maintains a protected server-local progress journal and resumes from the last verified -checkpoint. Detailed topology and command output remain in a protected evidence directory on the -server. Only intentionally redacted reports and reusable templates may enter Git. - -The Project A human guide is terminal-first and may use a local headless browser. When the optional -private endpoint exists, it also includes operator browser checks. The Project B guide covers the -real Aritmolab homepage/sidebar, Authentik SSO, roles, logout, final workflow, and negative cases. - -## Known documentation reconciliation - -Some older server documentation describes vector and embedding services as external and the -mandatory stack as only frontend/core. Current `compose.yaml`, repository instructions, and project -state define Qdrant and Ollama as mandatory internal services. The execution plans must treat the -current code/Compose contract as authoritative and include a documentation correction rather than -following the stale statements. - -## Success condition - -The program is complete only when both projects have exact-source evidence, all automated gates -pass, both human manuals are completed with explicit PASS decisions, the Aritmolab sidebar reaches -the final ThothII URL through the established load balancer and Nginx, Authentik provides SSO, and a -real read-only PSD session completes F1-F8 under an authorized OIDC identity. diff --git a/docs/plans/2026-08-20-psd-server-deployment-program.md b/docs/plans/2026-08-20-psd-server-deployment-program.md deleted file mode 100644 index 6da39467..00000000 --- a/docs/plans/2026-08-20-psd-server-deployment-program.md +++ /dev/null @@ -1,245 +0,0 @@ -# PSD Server Deployment Program Implementation Plan - -> **For agentic workers:** execute one task at a time, record its completion criteria, and stop at every documented approval boundary. - -**Goal:** Replace the legacy PSD ThothII installation, prove the replacement with local authentication, and then integrate the accepted release with Supabase, Authentik, Nginx, the load balancer, and the Aritmolab sidebar. - -**Architecture:** A read-only common survey freezes the actual server topology before any mutation. Project A installs a clean private stack and proves a complete PSD workflow; Project B begins only after a signed Project A PASS and performs the public OIDC/SSO cutover. Each project has an independent rollback boundary, human test guide, and evidence report. - -**Tech Stack:** Linux, Docker Engine, Docker Compose v2, native `tht`, Fastify/React/Pi, PostgreSQL/Supabase, Qdrant, Ollama, Nginx, Authentik OIDC, Aritmolab, load balancer. - ---- - -## Required reading and authority - -Read these files completely before starting: - -- `AGENTS.md` -- `PROJECT_STATE.md` -- `docs/plans/2026-08-20-psd-server-deployment-program-design.md` -- `docs/install/server.md` -- `docs/install/server-workspace-registry.md` -- `docs/install/authentication-local.md` -- `docs/install/authentication-oidc.md` -- `docs/install/authentik.md` -- `docs/contracts/workspace-preprocessing-cli.md` -- `docs/testing/authentication-manual-acceptance.md` - -The current `compose.yaml`, `deploy/compose.server.yaml`, repository instructions, and design are -authoritative where older server prose still describes Qdrant or Ollama as external. - -Run only from a terminal local to the server. Do not require SSH port forwarding. Do not print or -paste passwords, tokens, cookies, private keys, hashes, or raw identity claims. Commands that need -a credential must read a protected file or use an echo-free prompt. - -## Documents used during execution - -- Survey plan: `docs/plans/2026-08-20-psd-server-survey.md` -- Survey report: `docs/testing/evidence/psd-server-survey-report-template.md` -- Project A plan: `docs/plans/2026-08-20-psd-server-project-a-standalone.md` -- Project A human guide: `docs/testing/psd-server-project-a-manual.md` -- Project A report: `docs/testing/evidence/psd-server-project-a-report-template.md` -- Project B plan: `docs/plans/2026-08-20-psd-server-project-b-authentik.md` -- Project B human guide: `docs/testing/psd-server-project-b-manual.md` -- Project B report: `docs/testing/evidence/psd-server-project-b-report-template.md` - -The detailed evidence directory is a protected path on the server selected during the survey. The -repository receives only redacted reports after explicit owner review. - -## Owner-approved sequencing amendment — 2026-08-21 - -The Mac `rest_api` acceptance and revocation of `legacy-shared` move to a mandatory gate immediately -before Project B. The survey therefore records two distinct decisions: - -- `SURVEY_GO_PROJECT_A_PRIVATE`: technical prerequisite for requesting Project A private execution; -- `SURVEY_GO_PROJECT_B`: the complete shared-infrastructure decision, including Mac acceptance, - observation and legacy revocation. - -The amendment authorizes the read-only survey and static preparation of non-secret Project A -candidate facts and artifacts. While the current decision is `SURVEY_NO_GO`, it does not authorize -creating installation roots, cloning/building the candidate, creating protected configuration or -backup state, stopping the legacy stack, starting the new stack, changing public ingress, or -starting Project B. Those remain separate explicit gates after the scoped survey passes. - -### Task 1: Freeze the planning source - -**Files:** -- Read: `docs/plans/2026-08-20-psd-server-deployment-program-design.md` -- Record: protected server execution journal selected during the survey - -**Step 1: Verify the application checkout** - -Run: - -```bash -git status --short --branch -git rev-parse HEAD -git rev-parse origin/main -git show -s --format='%H%n%P%n%s' HEAD -``` - -Expected: the tree is clean and `HEAD` is the explicitly approved `origin/main` SHA. A newer SHA -than the design-time `5c0dc8c` is allowed only after recording and reviewing the intervening commits. - -**Step 2: Verify the plan files exist at that SHA** - -Run: - -```bash -test -f docs/plans/2026-08-20-psd-server-survey.md -test -f docs/plans/2026-08-20-psd-server-project-a-standalone.md -test -f docs/plans/2026-08-20-psd-server-project-b-authentik.md -git diff --check -``` - -Expected: every command exits zero. - -**Step 3: Record the immutable planning identity** - -Record the application SHA, plan commit, UTC timestamp, operator identity, and terminal-local access -method in the protected journal. Do not record CyberArk session secrets or screenshots. - -### Task 2: Execute and approve the common survey - -**Files:** -- Execute: `docs/plans/2026-08-20-psd-server-survey.md` -- Create from: `docs/testing/evidence/psd-server-survey-report-template.md` - -**Step 1: Execute every survey task without mutation** - -Expected: the survey identifies exact paths and owners for the old and new installations, Nginx, -load balancer, Aritmolab, Authentik, Supabase, the DWH, and protected credentials. - -**Step 2: Resolve every unknown** - -If an Authentik credential cannot be located, stop and ask the owner. If a configuration owner or -rollback boundary is unclear, stop; do not infer authority from file readability. - -**Step 3: Review the scoped survey GO/NO-GO** - -Expected: `SURVEY_GO_PROJECT_A_PRIVATE` requires a verified old-stack recovery path, an approved -new-installation root, enough resources, a direct read-only DWH path, workspace/model inputs, and -no unresolved mutation in the private Project A scope. Public-origin, load-balancer and Authentik -unknowns may remain explicitly deferred only while Project A is loopback-only and Task 10 is -omitted. `SURVEY_GO_PROJECT_B` retains the complete survey requirements. - -**Step 4: Checkpoint the survey** - -Hash the protected report and record only its path, SHA-256, timestamp, and GO result in the journal. - -### Task 3: Execute Project A - -**Files:** -- Execute: `docs/plans/2026-08-20-psd-server-project-a-standalone.md` -- Complete: `docs/testing/evidence/psd-server-project-a-report-template.md` - -**Step 1: Confirm the survey is GO for Project A private scope** - -Expected: the survey report hash matches the journal and no unresolved blocker remains inside the -Project A private scope. Before any stop/start, obtain a separate explicit owner authorization. - -**Step 2: Execute Project A task-by-task** - -Do not configure Authentik, change the production Aritmolab sidebar, or open the production route. - -**Step 3: Run the Project A human guide** - -Follow `docs/testing/psd-server-project-a-manual.md`. Record PASS/FAIL for every case; do not infer -manual PASS from automated output. The Mac REST row may be -`DEFERRED_PRE_PROJECT_B` only under the dated owner amendment. - -**Step 4: Close the Project A report** - -Expected: automated gates and the human guide are PASS; one harmless PSD session reached F8 and -produced validated read-only SQL; rollback remains available. The accepted report must list the -Mac REST item as an explicit deferred prerequisite rather than silently treating it as PASS. - -**Step 5: Obtain explicit owner approval** - -Record the approval and report digest. Project B remains forbidden without it. - -### Task 4: Freeze the Project B candidate - -**Files:** -- Read: accepted Project A report -- Record: protected server execution journal - -**Step 1: Recheck source and running images** - -Before freezing the candidate, close the pre-Project-B gate: validate the Mac installation with its -per-installation key, finish the 48-hour observation window including two 03:00 ETL cycles, revoke -`legacy-shared`, prove legacy `401` and v1 success, and obtain `SURVEY_GO_PROJECT_B`. - -Run the Project A plan's identity commands again. Record application SHA, workspace SHA, core image -ID, frontend image ID, Qdrant image digest, Ollama image digest, and local-auth configuration revision. - -Expected: all values match the accepted Project A report. - -**Step 2: Recheck rollback** - -Prove that the public route is still closed and the protected Project A configuration can be -selected without reconstructing it from memory. - -### Task 5: Execute Project B - -**Files:** -- Execute: `docs/plans/2026-08-20-psd-server-project-b-authentik.md` -- Complete: `docs/testing/evidence/psd-server-project-b-report-template.md` - -**Step 1: Execute Project B task-by-task** - -Keep public traffic closed until Authentik, Supabase migrations, ThothII diagnostics, Nginx, TLS, -and load-balancer preflight all pass. - -**Step 2: Run the Project B human guide** - -Follow `docs/testing/psd-server-project-b-manual.md` using approved ordinary and administrator -identities. The final path begins at the Aritmolab homepage and uses its existing sidebar link. - -**Step 3: Close the Project B report** - -Expected: SSO, roles, PostgreSQL ownership, the F1-F8 session, rollback rehearsal, and cleanup of the -temporary Project A endpoint all pass. - -### Task 6: Close the program - -**Files:** -- Modify: `PROJECT_STATE.md` -- Optionally create: reviewed redacted acceptance reports under `docs/testing/evidence/` - -**Step 1: Reconcile final state** - -Record final SHAs, image identities, workspace revision, Authentik object names/IDs (never secrets), -Supabase database/schema names, public origin, sidebar source revision, Nginx configuration identity, -and both report digests. - -**Step 2: Verify final negative boundaries** - -Expected: old stack stopped; Project A private endpoint removed; core/Qdrant/Ollama not externally -published; `thoth_sessions` absent from PostgREST exposed schemas; no secret appears in reports. - -**Step 3: Update project state** - -Add a dated factual section to `PROJECT_STATE.md`. Mark anything not actually run as PENDING. - -**Step 4: Run documentation checks** - -Run: - -```bash -git diff --check -bash scripts/auth-docs-smoke.sh -bash scripts/verify-workspace-install-docs.sh --fixtures-only -``` - -Expected: all checks pass. - -**Step 5: Commit only reviewed redacted documentation** - -```bash -git add PROJECT_STATE.md docs/testing/evidence -git diff --cached --check -git commit -m "docs: record PSD server deployment acceptance" -``` - -Expected: the commit contains no raw server inventory or secret material. diff --git a/docs/plans/2026-08-20-psd-server-project-a-standalone.md b/docs/plans/2026-08-20-psd-server-project-a-standalone.md deleted file mode 100644 index 3fac668a..00000000 --- a/docs/plans/2026-08-20-psd-server-project-a-standalone.md +++ /dev/null @@ -1,532 +0,0 @@ -# PSD Server Project A Standalone Implementation Plan - -> **For agentic workers:** execute one task at a time, record its completion criteria, and stop at every documented approval boundary. - -**Goal:** Install a clean PSD ThothII stack with local authentication, direct read-only DWH access, internal Qdrant/Ollama, rebuilt preprocessing, and one completed F1-F8 work session. - -**Architecture:** Keep the stopped legacy installation intact only as a temporary rollback boundary and deploy the current canonical five-service Compose stack from an adjacent clean clone. Create no host service identity: the image's UID/GID 10001 remains numeric and unmapped, confined to the new writable bind roots. A reviewed server-local override disables public exposure and uses filesystem work sessions; an optional load-balancer route is permitted only when it is operator-only and its denial boundary is proved first. - -**Tech Stack:** Git, Docker/Compose, native `tht`, local Argon2id authentication, PSD Supabase PostgreSQL direct transport, Qdrant, Ollama, Pi, Nginx/load-balancer test route where safe. - ---- - -## Preconditions - -- Common survey result is `SURVEY_GO_PROJECT_A_PRIVATE` and its digest is recorded. -- Every path below is replaced by the exact survey result before execution. -- No production Nginx/load-balancer/sidebar/Authentik change is in scope. -- The old stack remains running until its exact inventory and restart recipe are verified; old and - new stacks never run together. -- The server's workspace deploy credential remains read-only. A curator with write access publishes - the workspace change. -- Owner amendment 2026-08-21 declares legacy sessions/configuration disposable. Retain old resources - only until Project B proves the production Aritmolab journey, then delete them under a separate - exact cleanup authorization. No legacy backup is required. -- Do not create a host user or group for 10001. Immediately before preparing `/srv/thothii`, both - `getent passwd 10001` and `getent group 10001` must return no match. -- Owner amendment 2026-08-21 defers live Mac `rest_api` acceptance and `legacy-shared` revocation to - the mandatory pre-Project-B gate. It does not authorize stop/start; those require a later explicit - owner gate even after private preparation is complete. - -### Task 1: Freeze exact inputs - -**Files:** -- Read: protected survey report -- Read: `docs/plans/2026-08-20-psd-server-deployment-program-design.md` -- Record: protected Project A journal - -**Step 1: Record application identity** - -Run in the new planning checkout: - -```bash -git status --short --branch -git rev-parse HEAD -git rev-parse origin/main -git diff --check -``` - -Expected: clean and explicitly approved SHA. - -**Step 2: Record workspace remote identity** - -Use the surveyed read-only credential and run: - -```bash -git ls-remote <workspace-remote> refs/heads/main -``` - -Expected: one SHA recorded as the pre-change workspace revision. - -**Step 3: Check old-stack recoverability** - -Expected: exact old start/stop procedure, source SHA, Compose identity, volumes/binds, proxy closure -procedure, restart recipe, and shared-resource exclusions are present in the survey. Stop if any is -missing. - -### Task 2: Publish the multi-transport workspace revision - -**Files:** -- Modify in authorized curator clone: `psd-clinical/workspace.yaml` -- Verify: `thoth-workspaces.yaml` - -**Step 1: Create a clean curator branch** - -Run in a write-authorized clone, never in the application-managed registry checkout: - -```bash -git status --short --branch -git fetch origin main -git switch --create codex/psd-direct-transport origin/main -``` - -Expected: clean branch at the recorded remote SHA. - -**Step 2: Make the minimal descriptor change** - -Change exactly: - -```yaml -supported_transports: [rest_api] -``` - -to: - -```yaml -supported_transports: [rest_api, postgres_direct] -``` - -Do not duplicate the workspace or change its ID, collection, Evidence, annotations, model policy, -database, or schema. - -**Step 3: Review the descriptor-only diff** - -Run: - -```bash -git diff --check -git diff -- psd-clinical/workspace.yaml thoth-workspaces.yaml -``` - -Expected: one semantic line changed; catalog metadata remains identical. - -**Step 4: Validate with the current ThothII contract** - -Use a disposable installation/registry or the repository's current registry validation harness to -activate the candidate commit before publication. Expected: schema v3 accepts both transports, -Evidence and annotations materialize, and no secret is required for source validation. - -If no supported validator can be run in the curator environment, stop and request the owner to run -the established Mac validation; do not publish based only on YAML parsing. - -**Step 5: Commit and publish through curator review** - -```bash -git add psd-clinical/workspace.yaml -git diff --cached --check -git commit -m "feat: support direct PSD DWH transport" -git push --set-upstream origin codex/psd-direct-transport -``` - -Merge through the repository's normal review path. Record the resulting `main` SHA. - -**Step 6: Record the deferred Mac REST proof** - -Do not change the Mac during Project A. Record `DEFERRED_PRE_PROJECT_B`, the unchanged expected -transport `rest_api`, and the exact future diagnostics. The proof must become PASS before Project B, -after protected delivery/configuration of the per-installation key. - -### Task 3: Back up and stop the legacy installation - -**Files:** -- Create: surveyed protected legacy backup directory -- Record: Project A journal - -**Step 1: Capture final legacy state** - -Run the surveyed legacy status/doctor commands, record source SHA and image IDs, and confirm no -active user work. Do not use the new `tht` against an incompatible old descriptor. - -**Step 2: Close or maintenance-gate the old ThothII route** - -Change only the surveyed ThothII-specific route using its established mechanism. Validate Nginx and -load-balancer configuration before applying. Confirm external requests no longer reach the app. - -**Step 3: Record the disposable legacy boundary** - -Record exact container and image IDs plus filesystem device/inode/ownership/size for -`/home/chirone/ThothII` and `/home/chirone/thothii-data`. Do not archive their contents: the owner -declared them disposable. Prove that the external Evidence bind and both shared Docker networks -are excluded from any later cleanup manifest. - -**Step 4: Stop the old stack** - -Use its own supported controller. Expected: old containers stopped, not removed; volumes and bind -trees unchanged. - -**Step 5: Rehearse the restart command without executing it** - -Record the exact command, preconditions, port ownership, and route-restoration order. If it cannot -be stated unambiguously, stop before creating the new stack. - -### Task 4: Prepare the adjacent clean installation - -**Files:** -- Create: survey-selected new source root -- Create: survey-selected operator, secret, data, Pi-state, registry, and backup roots - -**Step 1: Create dedicated paths without creating identities** - -Follow `docs/install/server.md` numeric-ownership rules. Do not call `useradd`, `groupadd`, or -`usermod`. The existing operator owns source/operator files; only the new data, Pi-state, and -workspace-registry roots use unmapped numeric `10001:10001`. Stop if UID or GID 10001 resolves to a -host account, and do not change the image identity without a reviewed design amendment. - -**Step 2: Clone the frozen application source** - -```bash -git -c core.autocrlf=false clone <thothii-remote> <new-source-root>/ThothII -git -C <new-source-root>/ThothII config --local core.autocrlf false -git -C <new-source-root>/ThothII switch --detach <approved-application-sha> -git -C <new-source-root>/ThothII status --short --branch -``` - -Expected: detached exact SHA, clean tree. - -**Step 3: Verify source and platform** - -```bash -cd <new-source-root>/ThothII -bash scripts/verify-line-endings.sh -docker version -docker compose version -``` - -Expected: all pass. - -**Step 4: Prepare Pi state and build the operator** - -```bash -sudo scripts/prepare-server-pi-state.sh <new-pi-state-root> 10001 10001 -THT_THT_OUTPUT_DIRECTORY=<protected-build-output> bash scripts/build-tht.sh -``` - -Install only the binary matching the surveyed server architecture. Run `tht version --json` and -record its source identity. - -### Task 5: Create the protected Project A configuration - -**Files:** -- Create outside Git: `<project-a-operator-root>/server.env` -- Create outside Git: `<project-a-operator-root>/thothii-installation.yaml` -- Create outside Git: `<project-a-operator-root>/project-a-private.yaml` -- Create outside Git: `<project-a-auth-root>/auth.yaml` through `tht` - -**Step 1: Start from current examples** - -Copy `deploy/env/server.env.example` and `docs/install/examples/thothii-installation.server.yaml` -to the protected Project A operator root. Replace every placeholder with surveyed absolute paths. -Never source `server.env` as shell code. - -**Step 2: Add the private/local-session override** - -Create this reviewed override: - -```yaml -services: - core: - environment: - THOTH_PUBLIC_EXPOSURE: "false" - THT_SESSION_STORAGE: local - frontend: - ports: !override - - "127.0.0.1:<project-a-port>:8080" -``` - -Select an unused loopback port proved by `ss -lntp`. Do not publish core, Qdrant, or Ollama. - -**Step 3: Compose the installation descriptor** - -Use `profile: server`, the exact new source root/env/auth root, workspace remote/branch/read-only -access, Project A override, and exactly one Git transport override. Do not include the public -session-server overlay in Project A. - -For the projected local-auth descriptor, keep `authentication.configDirectory` as the canonical -root and add `runtimeProjection` with a distinct absolute runtime directory plus numeric `uid: 10001` -and `gid: 10001`. The descriptor loader includes the automatic runtime-projection override; do -not list it manually under `overrides`. The canonical root stays `root:root 0700/0600`; the -publisher owns the projection numerically as `10001:10001 0700/0600`. Only the projection is -mounted read-only into core. Do not create a host user/group, edit `CURRENT` or `generations`, or -apply these paths before the separately authorized start gate. - -**Step 4: Validate permissions and render** - -```bash -<new-tht> --installation <project-a-installation> update --check-only -``` - -Expected: Compose validates; only frontend has a loopback port; core declares public exposure false -and local session storage; Qdrant/Ollama are internal. - -**Step 5: Configure the local administrator** - -Create a temporary mode-0600 password file using an echo-free prompt, then run: - -```bash -<new-tht> --installation <project-a-installation> auth configure \ - --mode local --public-url <project-a-origin> \ - --admin-user <test-admin> --admin-display-name <display-name> \ - --password-file <protected-temporary-password-file> -``` - -Remove the temporary input file after success and record that removal. Do not delete generated -`auth.yaml` or `users.yaml`. - -Before the later start gate, run the redacted projected status command and require `ready` plus -`equal: true`. A blocked result prevents start. `auth publish` is the only repair path: it rebuilds -from canonical authentication, including after a candidate or recovery restore outcome; it never -promotes a retained runtime generation on its own. - -### Task 6: Build and start the clean stack - -**Files:** -- Record: Project A evidence directory - -**Step 1: Build current images** - -```bash -cd <new-source-root>/ThothII -bash scripts/build-local.sh -``` - -Expected: current core/frontend images build; pinned Qdrant/Ollama references resolve. - -**Step 2: Run preflight** - -```bash -<new-tht> --installation <project-a-installation> update --check-only -<new-tht> --installation <project-a-installation> pi doctor -``` - -Expected: no mutation error and no secret in output. - -**Step 3: Start through `tht`** - -```bash -<new-tht> --installation <project-a-installation> start --build -<new-tht> --installation <project-a-installation> status -<new-tht> --installation <project-a-installation> doctor --json -<new-tht> --installation <project-a-installation> pi test -``` - -Expected: frontend, core, Qdrant, embedding healthy and model initializer completed. Doctor has the -documented ordered checks and authentication PASS. - -**Step 4: Verify listener boundaries** - -Use `ss -lntp` and bounded Docker inspection. Expected: only the selected frontend loopback port is -host-published; no external core, Qdrant, or Ollama listener. - -### Task 7: Activate the workspace and direct DWH binding - -**Files:** -- Modify only through authenticated Workspace Management: encrypted workspace secret store - -**Step 1: Pull and inspect the reviewed workspace revision** - -```bash -<new-tht> --installation <project-a-installation> \ - workspace inspect --workspace psd-clinical --json -``` - -Expected: active workspace SHA equals the approved multi-transport revision. - -**Step 2: Configure runtime bindings** - -Through the authenticated Workspace Management API/UI, select `postgres_direct` and provide the -surveyed host, port, runtime user, password, and optional TLS CA. Secret values go to the encrypted -vault; they do not enter `server.env`, Git, shell arguments, or evidence. - -The generated contract names are: - -```text -THT_WS_PSD_CLINICAL_DWH_TRANSPORT -THT_WS_PSD_CLINICAL_DWH_HOST -THT_WS_PSD_CLINICAL_DWH_PORT -THT_WS_PSD_CLINICAL_DWH_USER -THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE -THT_WS_PSD_CLINICAL_DWH_TLS_CA_FILE -``` - -**Step 3: Validate and test connections** - -Run static validation, live connection test, workspace inspect, and `doctor --json`. Expected: DWH, -workspace, internal embedding, and Qdrant checks pass with redacted output. - -**Step 4: Re-prove read-only grants** - -Use the survey's catalog query through the exact configured identity. Expected: no DML/DDL grant on -`datawarehouse`. Stop if the runtime user is an owner, superuser, or write-capable role. - -### Task 8: Rebuild and verify semantic preprocessing - -**Files:** -- Create: Project A preprocessing evidence - -**Step 1: Inspect the empty/new collection state** - -```bash -<new-tht> --installation <project-a-installation> \ - workspace vector inspect --workspace psd-clinical --json -``` - -Expected: either a compatible empty collection or the documented missing-collection state. - -**Step 2: Create the descriptor-owned collection when missing** - -Use the guarded vector rebuild only for collection `psd-clinical`, with exact repeated confirmation -and `--destroy`. Do not run it against any other collection. - -**Step 3: Run complete preprocessing** - -```bash -<new-tht> --installation <project-a-installation> \ - workspace preprocess run --workspace psd-clinical --json -``` - -If it returns `manual_review_required`, inspect the exact run and curated annotations, obtain the -required human decision, run `workspace schema accept --workspace psd-clinical --run <run-id> --yes`, -then resume the same run. Never auto-approve unknown FK changes. - -**Step 4: Verify collection contract and counts** - -Run vector inspection and record dimensions, cosine distance, required keyword indexes, and bounded -counts by payload kind/revision. Expected: all points carry the active workspace revision. - -**Step 5: Prove idempotency** - -Run the complete preprocessing command again. Expected: no new review, no duplicate logical points, -unchanged Evidence reported as unchanged, and the same effective configuration identity. - -### Task 9: Configure and test local users - -**Files:** -- Modify through `tht auth user`: protected local user registry - -**Step 1: Add an ordinary test user** - -Use an echo-free prompt or protected temporary password file: - -```bash -<new-tht> --installation <project-a-installation> auth user add <test-user> \ - --role user --display-name <display-name> --password-file <protected-temporary-password-file> -``` - -Remove the temporary input file after success. - -**Step 2: Run authentication diagnostics** - -```bash -<new-tht> --installation <project-a-installation> auth status --json -<new-tht> --installation <project-a-installation> auth check --json -``` - -Expected: pristine redacted JSON and PASS. - -**Step 3: Execute automated local-auth cases** - -Use same-origin requests or a local headless browser to prove ordinary/admin authorization, generic -wrong-password failure, disable/enable, password/role revision invalidation, logout-all, CSRF, and -remembered-session survival after core restart. Do not retain cookie jars after the test. - -### Task 10: Optionally add the private network-path test - -**Current scope boundary (owner, 2026-08-21):** omit this entire task and keep Project A -loopback-only. Any future use requires a separate shared-infrastructure authorization after the -public-origin and load-balancer activities pass; the Project A private survey decision alone is -insufficient. - -**Files:** -- Modify only surveyed test-specific load-balancer/Nginx files -- Create: test certificate through the existing managed mechanism - -**Step 1: Prove allowlist capability before proxying** - -Create a temporary hostname that returns a fixed maintenance response. From an approved operator -source expect success; from an unapproved source expect denial. Do not point it at ThothII yet. - -**Step 2: Validate and activate the test proxy** - -Configure Nginx with the same Host/HTTPS forwarding and SSE settings intended for production. Run -`nginx -t`, validate the load balancer, then reload through the established mechanism. - -**Step 3: Reconfigure local-auth public URL transactionally** - -If the exact private HTTPS origin differs from the loopback origin, use the supported authentication -configuration workflow and invalidate prior test sessions. Re-run auth and doctor checks. - -**Step 4: Prove both sides** - -Expected: authorized operator reaches the local login; unauthorized source remains denied before -ThothII. If this cannot be demonstrated, remove the test route and continue on loopback. - -### Task 11: Complete the F1-F8 acceptance session - -**Files:** -- Complete: `docs/testing/psd-server-project-a-manual.md` -- Create: protected session evidence - -**Step 1: Select the approved harmless question** - -Use a known read-only PSD question agreed by the owner. Record the wording in the protected report; -do not include patient-identifying values. - -**Step 2: Create the session as the ordinary local user** - -Use the private browser route when present; otherwise drive the same-origin frontend/API from the -server-local terminal/headless browser. Record only session ID and sanitized milestones. - -**Step 3: Review every gate** - -Complete F1-F8 without auto-confirming human decisions. Confirm persisted phase/artifact state after -each gate and resume once to prove recovery. - -**Step 4: Validate final SQL** - -Expected: finalized session, DWH validation PASS, SQL is read-only, and no clinical mutation occurs. - -**Step 5: Inspect persisted state** - -Confirm manifest, question, schema linking, Evidence, CTE plan/tests, final SQL, validation report, -and decision ledger exist in local filesystem session storage. Chat/SSE need not persist. - -### Task 12: Close Project A and preserve rollback - -**Files:** -- Complete: `docs/testing/evidence/psd-server-project-a-report-template.md` - -**Step 1: Run final diagnostics** - -Run status, doctor, auth check, workspace inspect, vector inspect, Pi test, and a bounded secret scan -of the intended report. - -**Step 2: Create a transactional new-installation backup** - -Use `tht backup --drain` with a protected explicit output. Verify its checksum. Do not include -secrets in the ordinary evidence archive. - -**Step 3: Complete human acceptance** - -Every private-server row in `docs/testing/psd-server-project-a-manual.md` must be PASS or explicitly -blocking. Only the Mac REST row may be `DEFERRED_PRE_PROJECT_B` under the dated owner amendment. - -**Step 4: Record the gate** - -Record exact SHAs/images, workspace revision, preprocessing identity/counts, session ID, report -digest, rollback status, and explicit `PROJECT_A_PRIVATE_PASS` or `PROJECT_A_FAIL`. A private PASS -does not authorize Project B while the deferred gate remains open. - -**Step 5: Stop on FAIL** - -On FAIL, stop the new stack and use the surveyed old-stack recovery plan if service restoration is -desired. Do not start Project B. diff --git a/docs/plans/2026-08-20-psd-server-project-b-authentik.md b/docs/plans/2026-08-20-psd-server-project-b-authentik.md deleted file mode 100644 index e985de3b..00000000 --- a/docs/plans/2026-08-20-psd-server-project-b-authentik.md +++ /dev/null @@ -1,442 +0,0 @@ -# PSD Server Project B Authentik Integration Implementation Plan - -> **For agentic workers:** execute one task at a time, record its completion criteria, and stop at every documented approval boundary. - -**Goal:** Convert the accepted Project A installation to public OIDC mode, store owned work sessions in the existing Supabase database's `thoth_sessions` schema, and restore the established Aritmolab-sidebar user journey through the load balancer and Nginx. - -**Architecture:** Keep the same installation-descriptor path and Compose project so Project A Qdrant/Ollama volumes and bind state remain authoritative. Close ingress, snapshot Project A configuration, configure Authentik and Supabase, replace the protected installation configuration transactionally at the same paths, validate privately, then open the production route and sidebar link. - -**Tech Stack:** Authentik OAuth2/OIDC Authorization Code + PKCE, ThothII OIDC/session store, Supabase PostgreSQL schema migrations/RLS, Docker Compose, Nginx, load balancer, Aritmolab. - ---- - -## Preconditions - -- Project A automated and human reports are PASS and explicitly owner-approved. -- The Mac `rest_api` installation passes source validation and connection diagnostics with its - per-installation key. -- The dual-key observation has lasted at least 48 hours and includes two scheduled 03:00 ETL cycles. -- `legacy-shared` is revoked; v1 remains successful and the legacy credential is proven `401`. -- The current survey decision is `SURVEY_GO_PROJECT_B`, not only the private Project A decision. -- Application SHA, workspace SHA, images, Project A report digest, and rollback configuration match - the accepted evidence. -- The production route is closed before authentication/session-storage changes. -- All Authentik operations use the installed version's API/OpenAPI contract. Official current - references include [OAuth2/OIDC providers](https://docs.goauthentik.io/add-secure-apps/providers/oauth2/), - [provider property mappings](https://docs.goauthentik.io/add-secure-apps/providers/property-mappings/), - [application bindings](https://docs.goauthentik.io/add-secure-apps/applications/manage_apps/), and - [blueprint export](https://docs.goauthentik.io/customize/blueprints/export); installed-version - behavior wins over newer documentation. - -### Task 1: Freeze Project A and close ingress - -**Files:** -- Read: accepted Project A report -- Create: protected Project B transaction root - -**Step 1: Verify exact Project A state** - -Run status, doctor, auth check, workspace inspect, vector inspect, Pi status/test, Git SHA, and image -identity checks from Project A. Expected: all match the accepted report. - -**Step 2: Create a protected transaction root** - -Use `mktemp -d` under the survey-approved protected parent, mode `0700`. Record its path and do not -place it in Git. - -**Step 3: Close production and temporary ingress** - -Keep or restore a maintenance response at the production ThothII route. Disable the optional -Project A test route before changing authentication unless it is needed for a separately approved -private preflight. Confirm neither route reaches ThothII. - -**Step 4: Stop and back up Project A** - -```bash -<tht> --installation <stable-installation-path> backup \ - --output <project-b-transaction-root>/project-a-backup.tar --drain -<tht> --installation <stable-installation-path> stop -``` - -Verify the archive using the controller's manifest/checksum contract. Preserve a copy of the exact -installation descriptor, env file, override files, auth directory, Nginx fragment, load-balancer -route, Aritmolab sidebar file/revision, and relevant Authentik export metadata. - -### Task 2: Decide exact Authentik names and roles - -**Files:** -- Create: protected `authentik-change-manifest.yaml` - -**Step 1: Select the final public origin** - -Use the live survey result, not historical `.it`/`.com` assumptions. Record exactly one HTTPS origin -and callback `<origin>/api/auth/oidc/callback`. - -**Step 2: Select exact groups** - -Default to dedicated `TOT Users` and `TOT Admin`. Reuse existing groups only if their membership -semantics match and the owner approves. Record exact case-sensitive names. - -**Step 3: Define least privilege** - -Map user group → `user`, admin group → `admin`. Define a separate service account/token with only -the installed Authentik permission needed to view exact group objects. No write, user-management, -directory-administration, or superuser permission. - -**Step 4: Obtain owner approval of the manifest** - -The manifest contains object names, slugs, intended bindings, callback, scopes, grant types, -credential destinations, and rollback action—but no secret values. Do not mutate Authentik before -approval. - -### Task 3: Export and prepare Authentik - -**Files:** -- Create: protected pre-change Authentik export -- Modify: Authentik objects named in the approved manifest - -**Step 1: Export relevant configuration** - -Use the installed version's supported blueprint/API export. A worker command such as -`ak export_blueprint` is valid only if present in that version. Protect export mode `0600`; remember -write-only provider secrets are not included, so backup their custody separately without printing. - -**Step 2: Verify API credential scope** - -Use a read-only call to list relevant groups/applications. Expected: administrative creation access -for the setup identity and a distinct path for the future group-view service account. Stop if the -credential is missing or ambiguous. - -**Step 3: Create or confirm exact groups** - -Create missing dedicated groups, or record approved existing group IDs. Do not bulk-copy LDAP or -unrelated Authentik memberships. - -**Step 4: Create the group-catalog service account** - -Grant only exact group-view permission. Create its token through the approved protected-secret -mechanism; write it directly to the ThothII secret destination without displaying it. - -**Step 5: Create the OIDC provider** - -Configure a confidential OAuth2/OIDC provider with the exact callback, issuer mode observed as -appropriate, Authorization Code, PKCE support, and Device Code only when required for -`tht auth check --interactive` and supported by the installed release. Do not enable implicit flow. - -**Step 6: Configure scopes and direct groups claim** - -Select `openid`, `profile`, and `email`. Inspect a disposable identity's decoded claim keys through -a protected verifier; retain only a redacted shape. Expected: ID token includes direct non-empty -`groups: [string, ...]`. - -If the installed default profile mapping already provides that exact claim, reuse it. Otherwise add -a provider scope/property mapping under the requested `profile` scope that returns: - -```python -return {"groups": [group.name for group in request.user.ak_groups.all()]} -``` - -Verify the installed mapping merge semantics before activation. Do not add a custom unrequested -scope because ThothII requests only `openid`, `profile`, and `email`. - -**Step 7: Create the Authentik application** - -Bind it to the provider. Configure display metadata according to local Aritmolab conventions. Do not -use an Authentik proxy provider or Nginx forward-auth for ThothII. - -**Step 8: Create and store the client secret** - -Write the client secret directly into the protected ThothII secret bundle key -`THT_OIDC_CLIENT_SECRET`. Store the service-account token as `THT_AUTHENTIK_API_TOKEN`. Never place -either value in the change manifest, shell history, Compose environment, or evidence. - -### Task 4: Prepare Supabase schema roles and backup - -**Files:** -- Read: `harness/tht/migrations/sessions/001_schema.sql` -- Read: `harness/tht/migrations/sessions/002_security.sql` -- Create: protected Supabase backup/evidence -- Create: runtime and migrator credential files - -**Step 1: Confirm the database/schema boundary** - -Expected: use the surveyed existing Supabase PostgreSQL database; clinical data remains in -`datawarehouse`; application sessions use schema `thoth_sessions`; no new database is created. - -**Step 2: Back up database metadata/data consistently** - -Use the existing Supabase/PostgreSQL backup procedure before DDL. Record backup ID, timestamp, -checksum, and restore command. Do not put a dump in the Git repository. - -**Step 3: Create or validate dedicated roles** - -Create one migrator login and one runtime login according to the migration contract. The runtime -role must not be superuser, owner, BYPASSRLS, CREATEROLE, CREATEDB, or a member of DWH write roles. -The migrator credential remains unavailable to core. - -**Step 4: Write protected credential files** - -Create separate mode-0640 runtime-password, migrator-password, and session-CA files with surveyed -ownership. Do not use command-line password arguments. - -**Step 5: Confirm PostgREST exclusion before migration** - -Record the exact exposed schema list. Expected: `thoth_sessions` absent. If the system exposes all -schemas implicitly, stop and resolve the boundary before migration. - -### Task 5: Prepare the stable Project B installation configuration - -**Files:** -- Modify at the same stable paths: operator env, installation descriptor, authentication directory -- Create: reviewed session-server override copied from `deploy/compose.session-server.yaml.example` -- Create: protected server-session workspace config copied from `deploy/workspaces/server-sessions.yaml.example` - -**Step 1: Preserve the Compose project name** - -The native controller derives the project name from the absolute installation-descriptor path. -Keep that exact path. Do not point Project B at a second descriptor path, because that would create -new Qdrant/Ollama named volumes instead of using the Project A accepted state. - -**Step 2: Stage Project B files beside the live files** - -Prepare new env/descriptor/override/auth inputs in the protected transaction root. Add the session -DB host/port/existing database name/runtime role/migrator role/TLS mode and three secret source -paths. Use `verify-full` where hostname/SAN permits; any `verify-ca` exception requires explicit -survey evidence and owner approval. - -**Step 3: Add the server-session override** - -Copy the current example to a reviewed local file and add it to the existing stable descriptor's -overrides before the Git transport override ordering required by the installation. Do not edit the -tracked example. - -**Step 4: Replace local auth state transactionally** - -With the stack stopped, move the complete Project A auth directory into the protected transaction -root, recreate an empty private directory at the same path/ownership/mode, and configure OIDC: - -```bash -<tht> --installation <stable-installation-path> auth configure \ - --mode oidc --public-url <final-https-origin> \ - --issuer <authentik-issuer> --client-id <oidc-client-id> \ - --authentik-base-url <authentik-base-url> \ - --user-group '<exact-user-group>' --admin-group '<exact-admin-group>' -``` - -Expected: non-secret `auth.yaml` only; secrets resolved from the protected bundle. - -**Step 5: Atomically install staged path-only files** - -Use same-filesystem rename and preserve required ownership/mode. Keep the Project A originals in -the transaction root. Run `update --check-only`; on failure restore the originals immediately. - -### Task 6: Run and verify session migrations - -**Files:** -- Modify through one-shot migrator: existing database schema `thoth_sessions` - -**Step 1: Validate migration rendering** - -```bash -<tht> --installation <stable-installation-path> update --check-only -``` - -Expected: core and `session-migrate` resolve the same core image; core lacks migrator password; -only the one-shot service sees it. - -**Step 2: Run migrations once** - -```bash -<tht> --installation <stable-installation-path> sessions migrate --yes -``` - -Expected JSON: `"pending":[]` and `"drifted":[]`; `applied` may list `001` and `002` on first use. - -**Step 3: Run migration status/idempotency again** - -Run the same command. Expected: no new application and both pending/drifted remain empty. - -**Step 4: Verify database security** - -Through bounded catalog queries, prove forced RLS, policies on session tables, runtime role without -BYPASSRLS/ownership/DDL, migrator absent from core, and no runtime privileges on unrelated schemas. - -**Step 5: Recheck PostgREST exclusion** - -Expected: `thoth_sessions` still absent from exposed schemas and REST endpoints cannot address it. - -### Task 7: Validate Authentik and start privately - -**Files:** -- Record: Project B protected evidence - -**Step 1: Run static configuration validation** - -Run `update --check-only` and redacted `auth status --json`. Expected: mode OIDC, exact public origin, -issuer/client ID/group names, and no secret values. - -**Step 2: Start while public ingress remains closed** - -```bash -<tht> --installation <stable-installation-path> start -<tht> --installation <stable-installation-path> status -<tht> --installation <stable-installation-path> auth check --json -<tht> --installation <stable-installation-path> doctor --json -<tht> --installation <stable-installation-path> pi test -``` - -Expected: OIDC discovery/issuer/JWKS, client-secret access, Authentik catalog token, exact mapped -groups, PostgreSQL session storage, workspace, services, workflow, and Pi pass. - -**Step 3: Run interactive device check when supported** - -```bash -<tht> --installation <stable-installation-path> auth check --interactive -``` - -Expected: approved identity completes Device Authorization and direct groups claim validates. If -the installed provider does not support device flow, record PENDING rather than substituting a token. - -### Task 8: Prepare Nginx, TLS, load balancer, and sidebar - -**Files:** -- Modify only survey-approved ThothII Nginx fragment -- Modify only survey-approved load-balancer route -- Modify only exact Aritmolab sidebar source when its target must change - -**Step 1: Prepare direct-OIDC Nginx configuration** - -Follow `docs/install/reverse-proxy-nginx.md`, direct OIDC section. Required behavior: no -`auth_request`, no callback rewrite, frontend loopback upstream, Host and HTTPS forwarded headers, -HTTP/1.1, buffering/cache off, long SSE read timeout, and `X-Accel-Buffering: no`. - -**Step 2: Validate the managed certificate** - -Expected: SAN matches final hostname, validity is current, chain is trusted by approved clients, -private-key permissions match local policy, and renewal/generation ownership is recorded. Never -copy the key into ThothII. - -**Step 3: Validate Nginx without opening traffic** - -```bash -sudo nginx -t -curl --fail http://127.0.0.1:<frontend-port>/health -``` - -Use local `--resolve`/Host tests only when they do not bypass the identity behavior being tested. - -**Step 4: Prepare the load-balancer route** - -Configure backend/health/TLS according to the surveyed owner procedure, initially disabled or -operator-only. Confirm it targets Nginx, never core/Qdrant/Ollama directly. - -**Step 5: Preserve the Aritmolab link contract** - -If the existing sidebar target already equals the final origin/path, leave source unchanged and -record proof. Otherwise make the smallest reviewed change, test it in Aritmolab's own test/build -system, and commit in that repository before deployment. - -### Task 9: Open ingress and run OIDC acceptance - -**Files:** -- Complete: `docs/testing/psd-server-project-b-manual.md` - -**Step 1: Reload Nginx through the established mechanism** - -Run `nginx -t` immediately before reload. Expected: reload succeeds and unrelated virtual hosts -remain healthy. - -**Step 2: Enable the final load-balancer route** - -Expected: HTTP redirects to HTTPS; TLS is valid; `/api/auth/oidc/login` redirects to the correct -Authentik provider; callback returns to the exact public origin. - -**Step 3: Test ordinary and admin identities** - -Start at Aritmolab, authenticate once, then use the sidebar. Expected: no second credential prompt; -ordinary user can use sessions but receives 403 for admin operations; admin has only documented -permissions. - -**Step 4: Test no-role and malformed cases** - -An identity with no mapped group authenticates but receives no application role/403. Missing, -malformed, indirect, or ambiguous group claims fail closed with generic browser errors and redacted -diagnostics. Do not retain raw claims. - -**Step 5: Test logout and restart** - -Verify ThothII logout revokes its own cookie. Document whether the Authentik SSO session remains and -therefore allows immediate re-login without credentials; do not claim global logout unless -configured and tested. Restart core and verify expected OIDC session behavior. - -### Task 10: Verify PostgreSQL ownership and complete F1-F8 - -**Files:** -- Create: protected Project B session evidence - -**Step 1: Create sessions under two identities** - -Expected: ordinary users see only their own sessions; cross-user access returns the documented -not-found boundary; admin behavior matches `session.read_all/manage_all` permissions. - -**Step 2: Verify RLS with the runtime path** - -Use application/API tests and bounded catalog evidence. Never disable RLS for diagnosis. - -**Step 3: Complete one harmless OIDC PSD session** - -Use the same approved read-only question or another owner-approved one. Complete F1-F8, validate -final SQL, resume once, and confirm session/artifacts/decisions are stored in `thoth_sessions`. - -**Step 4: Verify ephemeral boundaries** - -Expected: no chat transcript or SSE stream stored as session artifacts; no vectors in PostgreSQL; -Qdrant remains the semantic store. - -### Task 11: Test controlled failures and rollback - -**Files:** -- Record: protected rollback evidence - -**Step 1: Test a reversible provider/catalog failure** - -Use a controlled, owner-approved method such as a temporary disabled test credential or test object. -Expected: auth diagnostics and browser login fail closed, redacted, then pass after restoration. -Never break unrelated Authentik applications. - -**Step 2: Rehearse ingress-first rollback** - -Close the production route, validate Nginx restoration commands, and prove the protected Project A -configuration snapshot is complete. A full rollback need not destroy `thoth_sessions`. - -**Step 3: Verify additive database rollback boundary** - -Expected: rollback leaves schema/data intact for evidence and future recovery. No automatic DROP -SCHEMA or role deletion. - -### Task 12: Close Project B - -**Files:** -- Complete: `docs/testing/evidence/psd-server-project-b-report-template.md` - -**Step 1: Run final diagnostics** - -Run status, doctor, auth check, interactive check when supported, workspace inspect, vector inspect, -Pi test, Nginx validation, load-balancer health, Supabase migration/security checks, and sidebar test. - -**Step 2: Remove the Project A temporary endpoint** - -Remove only its load-balancer route, Nginx fragment, and managed certificate reference according to -their owners. Validate/reload and prove the hostname no longer routes. - -**Step 3: Complete the human guide and report** - -Every mandatory row must be PASS. Record exact source/image/workspace identities, Authentik object -names/IDs, Supabase database plus `thoth_sessions`, public origin, Aritmolab revision, report digest, -and `PROJECT_B_PASS` or `PROJECT_B_FAIL`. - -**Step 4: Handle FAIL safely** - -On FAIL, close ingress first. Restore Project A files at the same stable paths, move OIDC auth state -to protected evidence, restore the local-auth directory, validate, and start Project A privately. -Disable new Authentik objects; do not delete them or drop the session schema automatically. diff --git a/docs/plans/2026-08-20-psd-server-survey.md b/docs/plans/2026-08-20-psd-server-survey.md deleted file mode 100644 index 7ce68c78..00000000 --- a/docs/plans/2026-08-20-psd-server-survey.md +++ /dev/null @@ -1,312 +0,0 @@ -# PSD Server Survey Implementation Plan - -> **For agentic workers:** execute one task at a time, record its completion criteria, and stop at every documented approval boundary. - -**Goal:** Produce a non-mutating, redacted survey of the PSD server that resolves every path, owner, network boundary, credential location, and rollback prerequisite needed by Projects A and B. - -**Architecture:** Collect bounded metadata from the terminal local to the server, retain raw output only in a protected directory, and summarize it in a redacted report. The survey makes no service, file, database, proxy, Authentik, or Git mutation. - -**Tech Stack:** Linux utilities, Docker/Compose inspection, Git, Nginx, OpenSSL, PostgreSQL/Supabase metadata queries, Authentik metadata/API discovery, Aritmolab source inspection. - ---- - -## Safety contract - -- Do not run `docker inspect` without a restrictive Go template; its default output can contain secrets. -- Do not run `docker compose config` into chat or a public log. Store raw output mode `0600`, then create a redacted derivative. -- Do not print process environments, secret-file contents, private keys, cookies, tokens, password hashes, or raw OIDC claims. -- Do not reload/restart services, fetch/pull Git, log in interactively, change file modes, or make API mutations. -- When a command needs privilege, use the server's approved CyberArk/local-terminal procedure. - -### Task 1: Create the protected survey workspace - -**Files:** -- Create: `/var/tmp/thothii-psd-survey.<random>/` -- Create: protected `survey-report.md` - -**Step 1: Create a private temporary root** - -Run: - -```bash -umask 0077 -PSD_SURVEY_ROOT="$(mktemp -d /var/tmp/thothii-psd-survey.XXXXXX)" -test -d "$PSD_SURVEY_ROOT" -chmod 0700 "$PSD_SURVEY_ROOT" -printf '%s\n' "$PSD_SURVEY_ROOT" -``` - -Expected: one new mode-0700 directory whose exact path is recorded in the operator journal. - -**Step 2: Copy the report template** - -Create `$PSD_SURVEY_ROOT/survey-report.md` from the headings in the design's Common Survey section. -Record only findings and references to protected raw files. - -### Task 2: Record host and Docker facts - -**Files:** -- Create: `$PSD_SURVEY_ROOT/host.txt` -- Create: `$PSD_SURVEY_ROOT/docker-projects.json` -- Create: `$PSD_SURVEY_ROOT/docker-containers.txt` - -**Step 1: Record bounded host metadata** - -Run each command with output redirected to `$PSD_SURVEY_ROOT/host.txt`: - -```bash -date -u '+%Y-%m-%dT%H:%M:%SZ' -uname -a -cat /etc/os-release -getconf LONG_BIT -nproc -free -h -df -hT -docker version -docker compose version -``` - -Expected: no credential content and enough capacity information to judge a parallel source tree and -a new five-service stack. - -**Step 2: Record Compose projects and bounded container identity** - -Run: - -```bash -docker compose ls --format json > "$PSD_SURVEY_ROOT/docker-projects.json" -docker ps -a --no-trunc --format '{{json .}}' > "$PSD_SURVEY_ROOT/docker-containers.txt" -docker network ls --format '{{json .}}' > "$PSD_SURVEY_ROOT/docker-networks.txt" -docker volume ls --format '{{json .}}' > "$PSD_SURVEY_ROOT/docker-volumes.txt" -``` - -Expected: inventory only. Do not inspect full container JSON. - -**Step 3: Identify candidate legacy ThothII containers** - -Use names, images, Compose project labels, published ports, and health from the bounded inventory. -For each candidate, query only these templates: - -```bash -docker inspect --format '{{.Name}} {{.Config.Image}} {{index .Config.Labels "com.docker.compose.project"}} {{index .Config.Labels "com.docker.compose.project.working_dir"}}' <container> -docker inspect --format '{{json .NetworkSettings.Networks}}' <container> -docker inspect --format '{{range .Mounts}}{{.Type}} {{.Source}} -> {{.Destination}} rw={{.RW}}{{println}}{{end}}' <container> -``` - -Expected: exact source/Compose ownership and mounts without environment values. - -### Task 3: Survey the legacy ThothII installation - -**Files:** -- Create: `$PSD_SURVEY_ROOT/legacy-thothii.txt` -- Create: `$PSD_SURVEY_ROOT/legacy-compose.redacted.yaml` - -**Step 1: Resolve source and operator paths from evidence** - -Do not search broad filesystem roots. Derive paths from Compose labels, systemd units, Nginx -upstreams, and known operator documentation. Record uncertainty rather than guessing. - -**Step 2: Record source identity without fetching** - -Run in the identified source tree: - -```bash -git status --short --branch -git rev-parse HEAD -git remote -v -git log -5 --oneline --decorate -``` - -Expected: no mutation. Mark a dirty tree as NO-GO until the owner decides how to preserve it. - -**Step 3: Record lifecycle state with the legacy controller** - -If the old installation has a supported `tht`, run its bounded `status`, `doctor`, `pi status`, and -`pi maintenance status` commands. Otherwise record exact read-only Docker health and identify the -old lifecycle mechanism. Do not substitute current `tht` against an incompatible descriptor. - -**Step 4: Render and redact Compose safely** - -Store the raw render as mode `0600`. Replace secret-bearing scalar values with `[redacted]` before -using the derivative in analysis. Confirm the redacted render still shows service names, networks, -ports, volumes, image/build identities, and config file paths. - -### Task 4: Survey Nginx, TLS, and the load balancer - -**Files:** -- Create: `$PSD_SURVEY_ROOT/nginx.raw.txt` (protected) -- Create: `$PSD_SURVEY_ROOT/nginx-thothii.redacted.txt` -- Create: `$PSD_SURVEY_ROOT/tls-metadata.txt` -- Create: `$PSD_SURVEY_ROOT/load-balancer.md` - -**Step 1: Validate and capture Nginx without reload** - -Run: - -```bash -sudo nginx -t -sudo nginx -T > "$PSD_SURVEY_ROOT/nginx.raw.txt" 2>&1 -chmod 0600 "$PSD_SURVEY_ROOT/nginx.raw.txt" -``` - -Expected: configuration test passes. Do not reload Nginx. - -**Step 2: Extract only relevant directives** - -Create the redacted derivative containing the ThothII/Aritmolab `server_name`, `listen`, `location`, -`proxy_pass`, `proxy_set_header`, `proxy_buffering`, timeout, certificate path, and include-file -directives. Exclude unrelated virtual hosts and all authorization values. - -**Step 3: Record certificate metadata only** - -For each relevant public certificate—not its key—run: - -```bash -openssl x509 -in <certificate-path> -noout -subject -issuer -serial -dates -ext subjectAltName -``` - -Expected: exact SAN/expiry/issuer and the observed generation/renewal mechanism. - -**Step 4: Map the load balancer** - -Record its owner, configuration surface, current Aritmolab backend, health check, TLS boundary, -source addresses seen by Nginx, and whether it can enforce a temporary hostname allowlist. Do not -create a route. If Sol cannot inspect it, name the human/team required for Project A/B gates. - -### Task 5: Survey Aritmolab and the sidebar integration - -**Files:** -- Create: `$PSD_SURVEY_ROOT/aritmolab.md` - -**Step 1: Resolve the Aritmolab source/deployment** - -Use Compose labels, Nginx paths, or the documented service unit. Record repository path, SHA, dirty -state, deployment command, container/network identity, and configuration owner. - -**Step 2: Locate the sidebar link** - -Use `rg` in the resolved source tree for the current ThothII URL, label, historical -`datamart-builder`, and sidebar/navigation definitions. Record exact files and line numbers. - -**Step 3: Resolve the real public origin** - -The owner reports `aritmolab.policlinicosandonato.com`; historical project state mentions a `.it` -origin and `/datamart-builder`. Record the live browser-visible origin and path from deployed -configuration. Do not choose between them without evidence. - -### Task 6: Survey Authentik - -**Files:** -- Create: `$PSD_SURVEY_ROOT/authentik.md` - -**Step 1: Identify deployment and version** - -Record Authentik containers/services, immutable image reference, version, base URL, database/Redis -dependencies, configuration owner, and backup/export procedure. Do not print container environments. - -**Step 2: Locate credential references** - -Record only file/secret-object paths, ownership, mode, and whether the local operator can use them. -If no usable administrative/API credential is found, stop and request owner help. - -**Step 3: Inventory relevant objects read-only** - -Using the installed version's API schema or admin interface, list only names/IDs for current -Aritmolab applications/providers, authorization flows, property mappings, groups, service accounts, -and policies that establish local conventions. Do not retrieve write-only secrets or raw tokens. - -**Step 4: Record version-specific constraints** - -Consult the official documentation matching the installed release for OAuth2/OIDC providers, -scope/property mappings, application bindings, blueprints/export, and API permission semantics. -Do not copy examples from a newer release without comparing the installed OpenAPI schema. - -### Task 7: Survey Supabase and the PSD DWH - -**Files:** -- Create: `$PSD_SURVEY_ROOT/supabase.md` -- Create: `$PSD_SURVEY_ROOT/dwh-readonly.txt` - -**Step 1: Map Supabase services without environments** - -Record PostgreSQL, pooler, PostgREST, gateway, and backup components; container networks and local -listeners; database name; TLS listener/CA; and the approved direct-connect route from ThothII core. - -**Step 2: Record exposed PostgREST schemas** - -Query only the explicit PostgREST schema setting through its known configuration mechanism. Do not -dump the whole environment. Confirm whether `thoth_sessions` already exists or is exposed. - -**Step 3: Inspect schemas and migration state** - -Through an approved administrative connection, run bounded catalog queries for existing schemas, -owners, and any `thoth_sessions` tables/migration records. Do not change them. - -**Step 4: Prove the intended DWH runtime identity is read-only** - -Connect using the protected runtime credential mechanism and query `current_database()`, -`current_user`, and grants for schema `datawarehouse`. Expected: USAGE/SELECT as required and no -INSERT, UPDATE, DELETE, TRUNCATE, REFERENCES, TRIGGER, CREATE, or ownership privileges. Do not run a -write probe against clinical tables. - -**Step 5: Identify session-schema roles** - -Record names or naming rules for a future migrator and runtime role. Do not create them. The final -design uses the existing database plus schema `thoth_sessions`, never a new database. - -### Task 8: Survey Git workspace and model boundaries - -**Files:** -- Create: `$PSD_SURVEY_ROOT/workspace-and-models.md` - -**Step 1: Inspect the workspace remote read-only** - -Record remote URL, branch, fetch credential path, current remote SHA, catalog entry, descriptor -schema version, current `supported_transports`, Evidence tree, annotations blob, and deploy-key -permissions. Do not push with the server's read-only deployment credential. - -**Step 2: Inspect Pi/LLM policy** - -Record selected provider/model/thinking level and credential references. Use bounded `tht pi status` -and `pi doctor` where compatible. Do not output provider keys. - -**Step 3: Check internal semantic capacity** - -Record CPU/GPU availability, free storage, and whether Docker can run the pinned Qdrant and Ollama -architectures. Do not pull images or models during the survey. - -### Task 9: Produce the survey decision - -**Files:** -- Modify: `$PSD_SURVEY_ROOT/survey-report.md` - -**Step 1: Complete the topology** - -Include exact component owners and flows for user → load balancer → Nginx → Aritmolab/sidebar → -ThothII, and core → Supabase DWH/auth session schema/Qdrant/Ollama/LLM/Authentik. - -**Step 2: List exact intended change files** - -Separate files owned by the new ThothII installation, workspace curator, Nginx, load balancer, -Aritmolab, Authentik, and Supabase. Mark shared files as owner-gated. - -**Step 3: State the scoped GO or NO-GO decisions** - -State both `SURVEY_GO_PROJECT_A_PRIVATE` and `SURVEY_GO_PROJECT_B`. The private decision requires -all paths, permissions, backup owners and rollback boundaries used by Project A; it may defer -public-origin, load-balancer, Authentik and Mac REST closeout facts that Project A does not mutate. -The Project B decision requires every shared/public fact plus the Mac acceptance, completed -observation window and revoked legacy credential. Each NO-GO must name concrete missing facts and -the person/system needed to resolve them. - -**Step 4: Hash and retain the report** - -Run: - -```bash -sha256sum "$PSD_SURVEY_ROOT/survey-report.md" > "$PSD_SURVEY_ROOT/survey-report.sha256" -sha256sum --check "$PSD_SURVEY_ROOT/survey-report.sha256" -``` - -Expected: checksum passes. Move the complete mode-0700 survey directory to the approved protected -evidence root without changing its contents; record the final path and digest in the journal. diff --git a/docs/plans/2026-08-24-evidence-restructuring-design.md b/docs/plans/2026-08-24-evidence-restructuring-design.md deleted file mode 100644 index a3337a64..00000000 --- a/docs/plans/2026-08-24-evidence-restructuring-design.md +++ /dev/null @@ -1,909 +0,0 @@ -# Ristrutturazione delle Evidence — disegno approvato - -**Stato:** approvato il 24 agosto 2026 -**Sostituisce:** il precedente disegno di Evidence canonica, disponibile nella storia Git -**Ambito:** authoring, revisione, pubblicazione, indicizzazione e uso runtime delle Evidence - -## 1. Obiettivo - -Questo disegno introduce un processo semplice e verificabile per trasformare documenti -di partenza non necessariamente ben organizzati in Evidence strutturate, revisionabili -da una persona e ricercabili in modo efficace da ThothII. - -La soluzione deve: - -1. partire dai testi oggi presenti nel repository del workspace; -2. riorganizzarli senza inventare informazioni; -3. conservare sorgenti e risultato nello stesso repository Git; -4. affidare a Git la revisione e l'approvazione umana; -5. indicizzare soltanto le versioni approvate; -6. sfruttare Qdrant senza moltiplicare collezioni e componenti; -7. inserirsi nel workflow modulare attuale, nel quale Evidence è un modulo autonomo. - -La fonte di verità rimane sempre il repository Git. Qdrant è un indice derivato che può -essere ricostruito. - -## 2. Principio guida - -Il processo è diviso in due percorsi distinti. - -- Il **percorso di authoring** prepara e revisiona le Evidence fuori dalle sessioni - domanda→SQL. -- Il **percorso runtime** è in sola lettura e consulta esclusivamente Evidence già - pubblicate. - -```mermaid -flowchart LR - S["Testi sorgente"] --> P["Pre-processing"] - P --> C["Evidence curate"] - C --> R["Revisione Git umana"] - R --> M["Merge e attivazione revisione"] - M --> I["Indicizzazione atomica"] - I --> Q["Qdrant: indice attivo"] - Q --> E["Evidence Module"] - E --> W["Workflow F1-F8"] -``` - -Una sessione può proporre una nuova formula o segnalare una lacuna, ma non modifica il -repository e non pubblica autonomamente conoscenza. - -## 3. Struttura nel repository del workspace - -Ogni workspace adotta questa struttura sotto la propria directory `evidence/`: - -```text -evidence/ -├── README.md -├── source/ -│ └── ... documenti originali ... -├── curated/ -│ ├── glossary/ -│ ├── domain/ -│ ├── enum/ -│ ├── example/ -│ ├── mapping/ -│ ├── normalization/ -│ ├── formula/ -│ └── reference/ -├── manifest.yaml -└── evaluation.yaml -``` - -### 3.1 `source/` - -Contiene i documenti originali. La prima versione accetta file Markdown, testo UTF-8 e -file `.sql.md`. Un URL può essere descritto in un documento, ma non viene scaricato né -interpretato automaticamente. - -I sorgenti vengono preservati: il pre-processing non li riscrive. - -### 3.2 `curated/` - -Contiene una Evidence Unit per file. Le sottodirectory rendono immediatamente visibile -il tipo anche a un lettore umano. Il campo `kind` nel documento resta comunque -obbligatorio: la directory aiuta la navigazione, il campo è il contratto macchina. - -### 3.3 `manifest.yaml` - -È gestito dal comando di preparazione e registra: - -- hash di ciascun sorgente; -- Evidence Unit derivate da quel sorgente; -- identificatori stabili; -- versione del processo di preparazione; -- unità orfane da controllare. - -Il manifest permette di elaborare soltanto ciò che è cambiato. Non sostituisce Git e -non contiene lo stato di approvazione. - -### 3.4 `evaluation.yaml` - -Contiene inizialmente circa venti domande rappresentative e gli identificatori delle -Evidence che ci aspettiamo di recuperare. È il controllo minimo per evitare di -considerare “migliore” una ricerca soltanto perché sembra sofisticata. - -## 4. Una struttura comune, otto tipi distinti - -La separazione tra tipi non viene eliminata. Ogni documento ha un involucro comune e -una parte specializzata determinata da `kind`. - -### 4.1 Campi comuni - -```yaml -schema_version: 1 -id: evidence:fascia-pediatrica -title: Fascia pediatrica -kind: formula -purposes: - - sql_generation - - schema_linking -applies_to: - concepts: - - fascia pediatrica - tables: - - clinical.patient - columns: - - clinical.patient.birth_date -language: it -provenance: - source_file: source/10-domini-clinici/paziente.md - source_sha256: sha256:0123456789abcdef... - supporting_excerpts: - - Per fascia pediatrica si intendono i pazienti con età inferiore a 18 anni. -review_items: [] -``` - -I campi hanno ruoli diversi: - -- `kind` dice **che cosa contiene** il documento; -- `purposes` dice **in quali attività può essere utile**; -- `applies_to` dice **a quali concetti o elementi del database si riferisce**; -- `provenance` permette di risalire al testo di origine; -- `review_items` rende visibili i dubbi ancora da risolvere. - -Ogni unità contiene da uno a cinque `supporting_excerpts`, ciascuno lungo al massimo -1.000 caratteri. Sono citazioni brevi che il validatore deve ritrovare nel sorgente dopo -la stessa normalizzazione meccanica. Provano la tracciabilità, non la correttezza -semantica: il revisore umano deve comunque verificare che sostengano davvero il -contenuto ristrutturato. - -Ogni `review_item` contiene soltanto: - -```yaml -code: ambiguous_source_statement -message: Il sorgente non chiarisce se l'età sia calcolata alla data di ricovero. -field: formula.sql # opzionale -``` - -Non possiede stato, autore o timestamp. Tutti i review item bloccano la pubblicazione; -il curatore corregge il documento e rimuove l'item, mentre Git conserva la storia. - -### 4.2 Tipi iniziali - -| `kind` | Contenuto | Esempio d'uso | -| --- | --- | --- | -| `glossary` | Definizione, sinonimi e varianti linguistiche | Capire che “ricovero” e “degenza” possono indicare lo stesso concetto | -| `domain` | Regole e vincoli del dominio | Interpretare correttamente un episodio clinico | -| `enum` | Valori ammessi e loro significato | Tradurre “dimesso” nel codice memorizzato nel DWH | -| `example` | Domanda esemplificativa e interpretazione attesa | Riconoscere una formulazione già documentata | -| `mapping` | Collegamento fra concetto e schema fisico | Individuare tabella e colonne pertinenti | -| `normalization` | Regole di normalizzazione | Uniformare codici, date o varianti testuali | -| `formula` | Espressione SQL riutilizzabile e relativi input | Calcolare la fascia pediatrica dalla data di nascita | -| `reference` | Un riferimento esterno che è esso stesso contenuto recuperabile | Proporre all'utente il link a una specifica linea guida | - -Un URL che documenta un'altra Evidence appartiene alla sua `provenance`. Un URL che -deve essere recuperato come risposta autonoma è invece una Evidence `reference`. - -Ogni Evidence Unit possiede un solo `kind`, scelto in base ai campi strutturati che ne -definiscono il contenuto principale. `purposes` e `applies_to` possono invece avere più -valori. Quando parti dello stesso sorgente hanno identità e regole di validazione -indipendenti, vengono prodotte unità distinte; non si duplica un'unità soltanto perché è -utile in più fasi del workflow. - -La classificazione procede dai contenuti più strutturati a quelli più generali: -`formula`, `enum`, `mapping`, `normalization`, `glossary`, `domain`, `example` e -`reference`. `domain` è il tipo di ripiego per una regola del dominio che non soddisfa -uno schema più specifico; `reference` si applica soltanto quando il collegamento deve -essere restituito come contenuto autonomo. - -L'identificatore non incorpora il `kind`: una riclassificazione conserva l'ID, mentre -una vera divisione semantica assegna nuovi ID alle nuove unità. Un rinominamento -univocamente riconoscibile del Source Evidence tramite hash aggiorna la provenienza e -conserva gli ID esistenti. - -Un nuovo identificatore usa la forma leggibile `evidence:<slug>`, viene assegnato una -sola volta e non viene ricalcolato da titolo, percorso o hash. Le collisioni ricevono un -suffisso deterministico. Dopo la prima pubblicazione cambiare ID equivale a ritirare -l'unità esistente e crearne una nuova. - -### 4.3 Dati specifici per tipo - -La parte specializzata è una unione discriminata: ogni `kind` ammette e richiede campi -diversi. Alcuni esempi: - -```yaml -# formula -formula: - concept: fascia pediatrica - columns: - - clinical.patient.birth_date - sql: | - CASE WHEN age < 18 THEN 'pediatrica' ELSE 'adulta' END -``` - -```yaml -# reference -reference: - url: https://example.org/linea-guida - label: Linea guida clinica - description: Criteri usati per classificare gli episodi. -``` - -```yaml -# enum -enum: - column: clinical.episode.discharge_status - values: - D: dimesso - T: trasferito -``` - -I tipi restano quindi sfruttabili sia in validazione sia in ricerca. Una formula non è -un semplice testo etichettato: possiede obbligatoriamente un concetto, le colonne di -input e una singola espressione PostgreSQL componibile. `SELECT`, `WITH`, DDL e DML come -statement completi non sono Formula Evidence; una query completa documentata appartiene -a `example`. Le formule legacy incompatibili diventano review item durante la migrazione. - -## 5. Pre-processing dei testi sorgente - -Il comando concettuale è: - -```text -tht evidence prepare <workspace-root> -``` - -Per l'utente è una sola operazione. Internamente esegue quattro passaggi. - -### 5.1 Estrazione deterministica - -Il sistema: - -- individua i file ammessi in `evidence/source/`; -- verifica dimensione, codifica UTF-8 e percorso sicuro; -- calcola l'hash del contenuto; -- confronta il risultato con `manifest.yaml`; -- carica, quando esiste, la precedente versione curata collegata al sorgente. - -Un sorgente invariato non viene nuovamente elaborato. - -Se la `pipeline_version` del manifest non è compatibile con quella installata, il -normale `prepare` termina senza scrivere. Il curatore può scegliere esplicitamente -`prepare --upgrade`, esclusivamente su un repository pulito, per rielaborare tutti i -sorgenti e revisionare il diff completo. - -### 5.2 Normalizzazione deterministica - -Prima del modello vengono normalizzati soltanto aspetti meccanici: - -- terminatori di riga e Unicode; -- spaziatura e intestazioni palesemente riconoscibili; -- elenchi, tabelle, blocchi SQL e URL; -- metadati già esplicitamente presenti; -- riferimenti a tabelle e colonne riconoscibili. - -Questa fase non interpreta il significato e non inventa strutture semantiche. - -### 5.3 Una sola ristrutturazione assistita dal modello - -Per ogni sorgente cambiato il modello riceve: - -- il testo normalizzato; -- gli otto schemi ammessi; -- le regole “non inventare” e “segnala il dubbio”; -- le precedenti Evidence curate derivate da quel sorgente; -- gli identificatori già assegnati. - -Per un'unità già esistente il modello può restituire soltanto uno degli identificatori -ricevuti. Per una nuova unità non propone l'ID: il preparatore assegna una sola volta -`evidence:<slug>` e l'eventuale suffisso deterministico. Un identificatore sconosciuto -prodotto dal modello rende la risposta non valida. - -Può: - -- assegnare titoli; -- classificare il tipo; -- separare un sorgente in più Evidence Unit; -- riordinare e riscrivere per chiarezza; -- compilare campi strutturati con fatti presenti nel sorgente. -- citare da uno a cinque brevi estratti del sorgente che sostengono ciascuna unità. - -Non può: - -- fondere automaticamente sorgenti diversi; -- aggiungere fatti non documentati; -- risolvere silenziosamente un'ambiguità; -- cancellare un'unità precedentemente revisionata. - -Se il sorgente esiste ancora ma non sostiene più un'unità precedente, il modello la -restituisce come retirement candidate con il `review_item` -`source_no_longer_supports_unit`. Il curatore decide se eliminarla o riscriverla; fino a -quel momento la pubblicazione resta bloccata. - -Quando una precedente unità viene realmente divisa in più unità autonome, le nuove -unità ricevono nuovi ID e la precedente rimane una retirement candidate finché il -curatore non la ritira esplicitamente. - -Pi viene usato in modalità non interattiva e senza strumenti di scrittura. È un -dettaglio interno del comando, non una nuova tipologia di sessione ThothII. -Timeout, uscita non valida o JSON malformato interrompono il comando con un errore -attribuito al sorgente. La prima versione non esegue retry automatici. - -### 5.4 Validazione deterministica - -L'output del modello non viene scritto direttamente. Viene prima controllato: - -- schema comune e schema specifico del `kind`; -- unicità e stabilità degli identificatori; -- appartenenza alle enumerazioni ammesse; -- esistenza e hash del sorgente; -- presenza nel sorgente normalizzato di ogni `supporting_excerpt`; -- correttezza sintattica di URL, tabelle, colonne e SQL dove applicabile; -- assenza di credenziali; -- coerenza tra directory e `kind`; -- assenza di collegamenti a sorgenti diversi nella stessa unità. - -Esistono tre esiti. - -| Esito | Comportamento | -| --- | --- | -| Valido | Il documento è pronto per la revisione Git | -| Valido con dubbi | Il documento viene scritto con `review_items`; non è indicizzabile | -| Non valido | Il documento non è pubblicabile e il rapporto spiega l'errore | - -Un dubbio reale può essere mantenuto soltanto se il revisore lo trasforma in una -limitazione esplicita del contenuto e svuota `review_items`. - -### 5.5 Applicazione atomica - -Tutti gli output dei sorgenti cambiati vengono costruiti e validati in un'area -temporanea. Soltanto quando l'intero batch è valido, il comando sostituisce insieme i -documenti interessati e `manifest.yaml`. Un singolo errore lascia il worktree invariato -e il rapporto limitato elenca tutti i problemi rilevati. Non esiste successo parziale. - -## 6. Aggiornamenti incrementali e revisione delle correzioni umane - -La precedente versione curata è un input, non un file usa-e-getta. Il modello deve -proporre una modifica minima senza ricominciare da zero, ma questa istruzione non viene -presentata come una garanzia semantica. La garanzia è Git: la versione precedente resta -recuperabile, ogni variazione è visibile nel diff e nessuna proposta diventa Published -Evidence senza una nuova revisione umana. - -Il comando: - -- si rifiuta di operare se `evidence/curated/` o `evidence/manifest.yaml` contengono - modifiche Git non salvate; -- mantiene gli ID associati a contenuti che rappresentano ancora la stessa unità; -- riconosce come rinominato un sorgente nuovo che corrisponde univocamente all'hash di - un sorgente rimosso e ne aggiorna la provenienza senza cambiare gli ID; -- mostra come diff le variazioni proposte; -- non modifica i file derivati da sorgenti invariati; -- segnala come orfana un'unità il cui sorgente è stato rimosso; -- non elimina mai automaticamente un'unità orfana; -- blocca la pubblicazione finché ogni unità orfana non viene eliminata, ricollegata o - ricondotta a un sorgente ripristinato. - -Git fornisce confronto, revisione, cronologia e recupero. Non viene introdotto un -database di authoring parallelo. - -Orfani e retirement candidate vengono risolti senza modificare manualmente il manifest: - -```text -tht evidence resolve <evidence-id> --retire -tht evidence resolve <evidence-id> --source <source-path> -``` - -Le due azioni sono mutuamente esclusive, richiedono un worktree pulito e aggiornano -atomicamente file curato e manifest. `--retire` rimuove l'unità dal corpus di authoring; -`--source` aggiorna provenienza e hash soltanto verso un sorgente esistente. Entrambe -lasciano un diff Git recuperabile, senza commit o pubblicazione automatica. Se l'unità -deve essere riscritta, il curatore modifica invece il documento e poi esegue -`validate`. - -## 7. Revisione e pubblicazione - -Il flusso di pubblicazione è: - -```text -prepare → revisione Git → validate → merge → attivazione workspace - → preprocess evidence candidata → evaluate candidata - → pubblicazione atomica della generazione Qdrant -``` - -### 7.1 Approvazione umana - -L'approvazione coincide con il normale processo Git del repository del workspace: - -1. il curatore esegue `prepare` in un clone di authoring; -2. legge i documenti e il diff; -3. corregge i contenuti; -4. esegue `tht evidence validate`; -5. apre o approva la pull request; -6. esegue il merge. - -La prima versione non crea automaticamente branch, commit o pull request. - -### 7.2 Quando un documento diventa Published Evidence - -Una Curated Evidence diventa Published Evidence soltanto quando: - -- appartiene a una revisione Git pulita che ha superato il processo umano di revisione; -- la revisione è stata attivata dal registry di ThothII; -- non contiene `review_items` irrisolti; -- il manifest non contiene unità orfane; -- l'intero corpus supera la validazione; -- la Candidate Evidence Generation supera il gate top-10; -- la generazione Qdrant viene quindi pubblicata atomicamente. - -Il runtime non tenta di ricostruire come sia avvenuta l'approvazione Git e il manifest -non contiene un flag `approved`. Il confine verificabile di pubblicazione è la -combinazione di revisione attiva, validazione superata e generazione Evidence attiva. - -Il descriptor filesystem deve indicizzare solo `curated/**/*.md`. I sorgenti e i file -di supporto restano materializzati per tracciabilità, ma non entrano nell'indice. - -## 8. Indicizzazione e generazioni - -L'indicizzazione continua a usare il meccanismo già implementato dal modulo Evidence: - -1. legge la radice materializzata della revisione Git attiva; -2. valida nuovamente tutte le Evidence; -3. costruisce Evidence Fragment secondo sezioni semantiche; -4. verifica il vettore dense predefinito già usato da Schema e Memory; -5. aggiunge in modo non distruttivo il vettore sparse `bm25` se manca; -6. genera le rappresentazioni dense degli Evidence Fragment; -7. chiede a Qdrant di generare per gli stessi frammenti la rappresentazione lessicale - BM25; -8. carica i punti con la nuova `vector_generation`; -9. verifica manifest, conteggi e leggibilità; -10. rende attiva la nuova generazione; -11. conserva le generazioni precedenti previste dalla policy. - -Se uno dei passaggi fallisce, la generazione precedente rimane attiva. I punti caricati -parzialmente vengono compensati secondo il meccanismo transazionale già esistente. -L'eventuale configurazione `bm25` già aggiunta rimane: è compatibile con i punti dense -esistenti e non richiede rollback. Se `bm25` esiste con una configurazione diversa da -`modifier: idf`, la procedura fallisce senza modificarla. - -L'upgrade non ricrea la collezione. I record `schema_table`, `schema_column`, `memory` e -`solved_question` restano invariati e continuano a usare il vettore dense predefinito. -Soltanto gli Evidence Fragment ricevono anche `bm25`. Durante la finestra fra aggiunta -del vettore e pubblicazione della prima candidata ibrida, Evidence è `unavailable`, ma -Schema e Memory continuano a funzionare. - -## 9. Qdrant spiegato senza presupporre conoscenze vettoriali - -### 9.1 L'analogia della biblioteca - -Si può immaginare Qdrant come il catalogo di una biblioteca. - -- Le **Evidence Unit** sono i documenti completi conservati negli scaffali Git. -- Gli **Evidence Fragment** sono le schede del catalogo relative alle singole sezioni. -- I **vettori** sono rappresentazioni numeriche usate per confrontare una domanda con - quelle schede. -- Il **payload** è l'insieme delle etichette leggibili: tipo, scopo, tabelle, colonne, - revisione e documento di origine. - -Qdrant non decide se una Evidence è vera e non sostituisce il documento. Aiuta soltanto -a trovare rapidamente le schede più promettenti. - -### 9.2 Ricerca per significato: vettore dense - -La rappresentazione dense descrive il significato generale di una frase. Permette, per -esempio, di avvicinare “pazienti minorenni” a “fascia pediatrica” anche quando le parole -non coincidono. - -È utile per il linguaggio naturale, ma può essere meno precisa con codici, acronimi, -nomi di colonne e formule. - -### 9.3 Ricerca per parole e identificatori: BM25 sparse - -La rappresentazione sparse conserva il peso delle parole presenti. È adatta a termini -come `ICD-10`, `discharge_status`, `ADT`, un valore enum o un nome esatto di colonna. - -Qdrant 1.18.2 può generare questa rappresentazione direttamente sul server usando -`qdrant/bm25`; per il corpus italiano si passa `language: italian` sia durante il -caricamento sia durante la ricerca. Non serve aggiungere FastEmbed o un nuovo servizio. - -Un test L0 avvia esattamente l'immagine Qdrant dichiarata da `compose.yaml`, crea una -collezione temporanea, indicizza due testi italiani mediante `qdrant/bm25`, verifica una -ricerca lessicale e infine elimina la collezione. Questo rende controllabile la capacità -locale richiesta prima di qualunque migrazione reale. Se la prova fallisce non esiste -un fallback silenzioso a un motore diverso. - -Il nome “sparse” significa soltanto che, tra moltissime parole possibili, ogni testo ne -usa poche. Qdrant mantiene anche l'IDF: una parola rara pesa più di una parola presente -quasi ovunque. - -### 9.4 Perché combinarle - -Una domanda può richiedere contemporaneamente comprensione e precisione lessicale: - -> “Qual è la formula per distinguere la fascia pediatrica usando -> `patient.birth_date`?” - -La ricerca dense riconosce il concetto; BM25 riconosce con forza “formula” e il nome -della colonna. Qdrant esegue entrambe e produce due graduatorie. - -### 9.5 Reciprocal Rank Fusion - -Reciprocal Rank Fusion, o RRF, combina le due graduatorie usando la posizione dei -risultati invece di confrontare direttamente punteggi di natura diversa. - -In termini pratici: - -- un documento alto in entrambe le liste sale; -- un documento molto forte in una sola lista può comunque emergere; -- non occorre inventare una conversione fragile fra “similarità semantica” e “punteggio - delle parole”. - -Si parte con i pesi predefiniti. Pesi diversi saranno introdotti soltanto se -`evaluation.yaml` dimostrerà un miglioramento. - -### 9.6 Il ruolo dei metadati - -Ogni punto Qdrant conserva almeno: - -```text -workspace_id -workspace_revision -vector_generation -record_kind -evidence_id -evidence_kind -purposes -concepts -tables -columns -language -source_file -source_sha256 -fragment_ordinal -``` - -I metadati hanno due usi: - -- workspace, revisione, generazione e purpose sono filtri obbligatori; -- tipo, concetti, tabelle e colonne diventano filtri soltanto quando il chiamante li - dichiara vincolanti; altrimenti contribuiscono al testo della query e alla spiegazione - del risultato. - -La prima versione non aggiunge bonus automatici per `kind` o `applies_to`. Durante la -generazione SQL una Formula Evidence viene imposta soltanto quando il workflow richiede -esplicitamente `kind=formula`; negli altri casi dense, BM25 e RRF determinano l'ordine. - -Quando concetti, tabelle o colonne non sono vincoli, il modulo costruisce un solo testo -deterministico, identico per dense e BM25: - -```text -Domanda: <domanda originale> -Concetti: <valori deduplicati e ordinati> -Tabelle: <valori deduplicati e ordinati> -Colonne: <valori deduplicati e ordinati> -``` - -Le righe vuote sono omesse. La domanda conserva formulazione e ordine originali; il -renderer applica soltanto Unicode NFC, converte CRLF e CR in `\n`, rimuove gli spazi -esterni e rifiuta una domanda vuota. Non cambia maiuscole, punteggiatura o spazi interni. -Ai valori contestuali applica NFC e `strip`, elimina stringhe vuote e duplicati esatti e -li ordina per valore Unicode senza `lower()` o `casefold()`: gli identificatori -PostgreSQL quotati possono essere sensibili alle maiuscole. In questo modo due richieste -equivalenti non cambiano per effetto dell'ordine occasionale dei metadati. - -### 9.7 Perché non creare una collezione per tipo - -Una domanda spesso attraversa più tipi: una formula può dipendere da un mapping, da un -enum e da una regola di dominio. Collezioni separate richiederebbero più interrogazioni, -fusione applicativa e più operazioni di manutenzione. - -La soluzione usa la collezione semantica già posseduta dal workspace, conserva il suo -vettore dense predefinito e aggiunge soltanto il vettore sparse denominato `bm25`. I -payload indicizzati distinguono i tipi. È più semplice, evita di ricostruire Schema e -Memory e permette a Qdrant di eseguire ricerca ibrida e filtri nella stessa Query API. - -### 9.8 Perché indicizzare frammenti ma restituire unità - -Un documento lungo può contenere sezioni diverse. Un unico vettore ne diluirebbe il -significato; frammenti arbitrari di lunghezza fissa spezzerebbero invece formule o -regole. - -La divisione segue intestazioni, campi tipizzati e confini di paragrafo. Non divide mai -una formula, una coppia valore/significato, un mapping, una regola o un URL. Se uno di -questi elementi atomici supera da solo `max_chunk_chars`, la preparazione aggiunge il -Review item stabile `atomic_content_too_large` e blocca la pubblicazione: non usa un -taglio a dimensione fissa che ne altererebbe il significato. Il limite è quello già -presente nella configurazione degli embeddings, pari per default a 4.000 caratteri, e -si applica all'intero testo reso che sarà inviato all'embedder, incluse etichette e -metadati testuali. Non viene introdotta una seconda impostazione. Qdrant trova i -frammenti, poi l'Evidence Module li raggruppa per `evidence_id` e restituisce un solo -Evidence Result con i migliori estratti, provenienza, citazione e riferimento al -documento completo. Il contenuto completo viene risolto soltanto quando il workflow ne -ha bisogno. - -### 9.9 Cosa non introduciamo nella prima versione - -- una collezione per ogni tipo; -- ColBERT o multivettori late-interaction; -- un reranker basato su un altro modello; -- pesi RRF regolati a mano senza misurazioni; -- un servizio separato per BM25; -- ricerca automatica sul web. - -Queste possibilità rimangono future ottimizzazioni, non prerequisiti. - -Riferimenti tecnici ufficiali: - -- [Qdrant: Text Search](https://qdrant.tech/documentation/search/text-search/) -- [Qdrant: server-side BM25](https://qdrant.tech/documentation/inference/inference-bm25/) -- [Qdrant: Hybrid Queries e RRF](https://qdrant.tech/documentation/search/hybrid-queries/) -- [Qdrant: aggiornamento dello schema dei vettori](https://qdrant.tech/documentation/manage-data/collections/#update-vector-schema) -- [Qdrant: payload indexing](https://qdrant.tech/documentation/manage-data/indexing/) -- [Qdrant: multitenancy](https://qdrant.tech/documentation/manage-data/multitenancy/) - -## 10. Contratto di ricerca del modulo Evidence - -Il workflow non costruisce query Qdrant. Usa una sola interfaccia concettuale: - -```python -search( - query: str, - purpose: EvidencePurpose, - context: EvidenceSearchContext, -) -> EvidenceSearchOutcome -``` - -`EvidenceSearchContext` può specificare tabelle, colonne e concetti da aggiungere alla -query, oltre a vincoli espliciti su tipo, tabelle, colonne o concetti. - -Il modulo rende domanda e contesto una sola volta nel formato `Domanda`, `Concetti`, -`Tabelle`, `Colonne` definito sopra e passa esattamente quel testo sia all'embedder dense -sia a `qdrant/bm25`. - -`EvidenceSearchOutcome` distingue due stati: - -- `available`, con la generazione interrogata e zero o più `EvidenceResult`; -- `unavailable`, senza risultati e con un codice di errore stabile e un messaggio - limitato. - -Una lista vuota nello stato `available` significa che la ricerca ha funzionato ma non -ha trovato corrispondenze. Non equivale a un errore tecnico. - -Il modulo Evidence possiede interamente: - -- generazione della query dense; -- query BM25 con lingua coerente; -- filtri su workspace, revisione, generazione e purpose; -- RRF; -- vincoli espliciti su `kind` e `applies_to`; -- raggruppamento dei frammenti; -- risoluzione di provenienza e citazioni; -- controllo della revisione attiva. - -Il workflow riceve candidati spiegabili, mai verità automatiche. - -## 11. Inserimento nel workflow modulare ThothII - -Evidence rimane un modulo autonomo con due responsabilità pubbliche. - -### 11.1 Authoring - -```text -prepare → validate → evaluate -``` - -Questa superficie è usata dal curatore e non dalle sessioni. - -### 11.2 Runtime - -```text -search → resolve citation → project into session -``` - -Evidence è un contributore degli stage esistenti, non uno stage aggiuntivo. Non emette -decisioni, non scrive gli artifact canonici e non modifica il ledger o lo stato del -workflow. Lo stage chiamante decide come usare i candidati restituiti. - -L'integrazione usa l'identità semantica dello stage, non il display code: - -| Stage semantico | Display code attuale | Evidence purpose | -|---|---:|---| -| `clarification` | F1 | `disambiguation` | -| `rewriting` | F3 | `rewriting` | -| `schema_linking` | F4 | `schema_linking` | -| `cte` | F6 | `sql_generation` | -| `final_sql` | F7 | `sql_generation` | - -Lo stage `memory` (F2) usa il Memory Module. Lo stage `synthesis` (F5) verifica e -riassume lo schema linking già approvato e non avvia una nuova ricerca Evidence. - -Ogni stage elencato esegue una ricerca indipendente con gli input disponibili in quel -momento. In particolare `cte` usa domanda riscritta e schema approvato, mentre -`final_sql` aggiunge il piano CTE approvato. La prima versione non introduce una cache -condivisa fra stage. - -Il chiamante conserva nella sessione una Evidence receipt con stage, purpose, -generazione e ID restituiti. Il testo non viene copiato: rimane nel repository del -workspace e viene risolto attraverso la provenienza della Published Evidence. - -Gli stage passano `purpose` e contesto al modulo, ma non conoscono collezioni, nomi di -vettori, generazioni o sintassi Qdrant. - -Le istruzioni Pi relative alla consultazione delle Evidence vengono spostate in -frammenti del modulo Evidence e poi proiettate nel `SKILL.md` generato, seguendo il -meccanismo modulare già usato da Disambiguation e Memory. - -## 12. Formule - -Le formule approvate oggi presenti nello store `formulas/*.sql.md` vengono convertite in -Evidence `kind: formula`. Dopo la migrazione non esistono due archivi runtime. - -Una nuova formula scoperta in F4 segue invece questo percorso: - -```text -sessione → Formula proposal nell'artefatto di sessione - → importazione di manutenzione - → Curated Evidence formula - → revisione Git - → Published Evidence -``` - -Le decisioni `concept_formula_approved` e `concept_formula_rejected` continuano a -descrivere la scelta fatta nella singola sessione. Non equivalgono alla pubblicazione -globale nel workspace. - -## 13. Comportamento in caso di errore - -### 13.1 Durante l'authoring - -- un file non UTF-8, troppo grande o strutturalmente invalido produce un errore chiaro; -- un dubbio semantico produce un `review_item`; -- un albero Git sporco impedisce la scrittura di nuove proposte; -- un sorgente rimosso produce un'unità orfana, non una cancellazione, e blocca la - pubblicazione finché il curatore non la risolve. - -### 13.2 Durante l'indicizzazione - -- la nuova generazione viene preparata senza toccare quella attiva; -- la generazione candidata viene interrogata esplicitamente per la valutazione senza - renderla visibile alle sessioni; -- una valutazione fallita lascia inattiva la candidata; -- un caricamento o una verifica falliti non cambiano il puntatore attivo; -- i dati parziali vengono rimossi quando possibile e comunque non sono leggibili dal - runtime perché manca l'attivazione. - -Poiché la revisione del workspace viene attivata prima di costruire la candidata, il -runtime può attraversare una finestra di manutenzione in cui la vecchia generazione non -corrisponde alla revisione. In questa finestra la ricerca Evidence è `unavailable` e -blocca lo stage chiamante. La prima versione accetta questa degradazione fail-closed -invece di introdurre una transazione distribuita fra Git, registry e Qdrant. Se il gate -fallisce, l'operatore corregge il corpus oppure ripristina esplicitamente la revisione -precedente. - -### 13.3 Durante una sessione - -Una ricerca `available` senza corrispondenze produce una Evidence receipt vuota, viene -mostrata come tale e non impedisce allo stage di continuare. - -Se Qdrant, il corpus attivo o la revisione attesa non sono disponibili: - -- l'outcome è `unavailable`, non una lista vuota valida; -- viene restituito un codice stabile con un messaggio limitato; -- lo stage chiamante resta bloccato e può essere ritentato; -- non vengono usate revisioni precedenti; -- non viene ripetuta la ricerca con un purpose diverso. - -Questo comportamento è fail-closed: un'assenza reale di corrispondenze non ferma il -workflow, mentre un guasto non viene mascherato come assenza di conoscenza. - -## 14. Valutazione minima - -`tht evidence evaluate` esegue le domande in `evaluation.yaml` contro una generazione -indicata esplicitamente oppure, per il monitoraggio ordinario, contro l'indice attivo. -Riporta almeno: - -- quante domande hanno trovato una Evidence attesa nei primi 5 e nei primi 10 risultati; -- quali tipi attesi sono mancati; -- quali query non hanno prodotto risultati; -- per ogni Evidence attesa, la posizione nella graduatoria dense, BM25 e fused; -- revisione Git, generazione e configurazione di ricerca usate. - -Il file classifica ogni domanda come `lexical`, `semantic` o `mixed` e contiene almeno -un caso per profilo. L'evaluator esegue i due rami anche separatamente per renderli -diagnosticabili, oltre alla ricerca ibrida usata dal runtime. La prima baseline deve -essere salvata prima di regolare pesi o introdurre altri modelli. Il comando non -modifica l'indice. La valutazione supera il gate minimo soltanto quando ogni query trova -almeno una delle Evidence attese nei primi dieci risultati fused. Le posizioni dei -singoli rami e `hit@5` restano informative e non bloccano la pubblicazione. - -## 15. Comandi e responsabilità - -| Comando | Dove opera | Scrive | -| --- | --- | --- | -| `tht evidence prepare <workspace-root> [--upgrade]` | clone Git di authoring | `curated/`, `manifest.yaml` | -| `tht evidence validate <workspace-root>` | clone Git o CI | nulla | -| `tht evidence resolve <id> (--retire | --source <path>)` | clone Git di authoring | unità interessata, `manifest.yaml` | -| `tht evidence evaluate ... [--generation <id>]` | generazione candidata o attiva | solo rapporto su stdout/JSON | -| `tht ... workspace preprocess evidence` | installazione/runtime | generazione candidata, poi attiva soltanto dopo il gate | - -`prepare` non crea commit. `preprocess evidence` non modifica il repository Git. - -## 16. Migrazione iniziale del workspace PSD - -Il corpus attuale comprende 36 file Markdown organizzati in glossario, domini clinici, -enum, esempi NLQ, mapping e normalizzazione. La migrazione avviene così: - -1. spostare gli originali sotto `evidence/source/`, conservandone la gerarchia; -2. eseguire `prepare` e generare `curated/`; -3. revisionare tutte le unità e risolvere i `review_items`; -4. importare eventuali formule approvate come `kind: formula`; -5. compilare circa venti query in `evaluation.yaml`; -6. configurare il descriptor con `patterns: ["curated/**/*.md"]`; -7. validare, fare merge e attivare la revisione in una finestra di manutenzione; -8. registrare conteggi e ID campione di Schema e Memory; -9. eseguire `preprocess evidence`, che aggiunge `bm25` senza ricreare la collezione e - costruisce la generazione candidata; -10. verificare che conteggi, ID campione e ricerche dense di Schema e Memory siano - invariati; -11. valutare la candidata e pubblicarla soltanto se supera il gate top-10; -12. salvare la baseline e svolgere una verifica umana degli stage `clarification`, - `rewriting`, `schema_linking`, `cte` e `final_sql`. - -Non serve mantenere v1 e v2 attivi contemporaneamente nel runtime: Git conserva la -vecchia revisione e il meccanismo delle generazioni conserva il rollback dell'indice. - -## 17. Criteri di accettazione - -La prima versione è completa quando: - -1. un sorgente poco strutturato produce una o più unità tipizzate senza perdere la - provenienza; -2. sorgenti invariati sono un no-op; -3. ogni modifica proposta a contenuti revisionati è recuperabile e visibile nel diff - Git prima della pubblicazione; -4. ogni unità cita brevi estratti verificabili del proprio sorgente; -5. gli ID nuovi sono assegnati dal codice e il modello può soltanto riutilizzare ID - precedenti esplicitamente forniti; -6. il batch di preparazione è tutto-o-niente e non esegue retry automatici; -7. un `review_item` impedisce l'indicizzazione; -8. un'unità orfana blocca la pubblicazione finché non viene risolta; -9. un'unità non più sostenuta dal proprio sorgente diventa una retirement candidate e - blocca la pubblicazione; -10. il comando `resolve` ritira o ricollega un'unità con un diff Git recuperabile; -11. tutte le otto varianti hanno validazione specifica e un solo `kind` primario; -12. una riclassificazione conserva l'ID e un rinominamento univoco del sorgente conserva - gli ID delle unità collegate; -13. ogni ID usa `evidence:<slug>`, non viene ricalcolato automaticamente e non contiene - il kind; -14. ogni Formula Evidence contiene una sola espressione PostgreSQL componibile; -15. le formule approvate sono ricercate tramite lo stesso modulo delle altre Evidence; -16. soltanto `curated/**/*.md` entra nel corpus runtime; -17. la collezione Qdrant conserva il dense predefinito e aggiunge `bm25` con IDF senza - ricostruzione distruttiva; -18. la Query API esegue i due prefetch e la fusione RRF; -19. risultati di frammenti della stessa unità diventano un solo Evidence Result con - estratti e riferimento al documento completo; -20. una ricerca disponibile può restituire zero Evidence, mentre una revisione o - generazione non corrispondente produce `unavailable` e blocca lo stage; -21. ogni ricerca applica il purpose come filtro obbligatorio e non usa bonus impliciti - per kind o ambito; -22. la generazione candidata diventa attiva soltanto quando ogni query recupera almeno - una Evidence attesa nei primi dieci risultati; il rapporto include anche `hit@5`; -23. i cinque stage mappati interrogano Evidence indipendentemente, mentre `memory` e - `synthesis` non lo invocano; -24. ogni ricerca disponibile conserva una ricevuta minima senza duplicare il testo; -25. una sessione completa riprende dopo la finestra fail-closed mediante retry o - rollback esplicito; -26. conteggi, ID campione e ricerche dense dimostrano che Schema e Memory non cambiano - durante l'upgrade; -27. una prova L0 dimostra `qdrant/bm25` sull'immagine locale effettivamente dichiarata; -28. dense e BM25 ricevono lo stesso testo di query deterministico; -29. nessun elemento atomico viene spezzato per rispettare la dimensione dei frammenti; -30. la valutazione copre casi lessicali, semantici e misti e mostra separatamente i tre - ranking; -31. il testo reso di ogni frammento rispetta il solo `max_chunk_chars` esistente; -32. la query conserva maiuscole, punteggiatura e spazi interni e non altera gli - identificatori PostgreSQL sensibili alle maiuscole; -33. documentazione e comandi descrivono lo stesso contratto. - -## 18. Decisioni rinviate - -Saranno considerate soltanto dopo la baseline: - -- pesi RRF diversi da quelli predefiniti; -- reranking; -- ColBERT o multivettori; -- acquisizione automatica di PDF, Word, HTML o pagine web; -- creazione automatica di branch e pull request; -- fusione assistita di Evidence provenienti da sorgenti diversi. - -Queste esclusioni mantengono la prima implementazione comprensibile, realizzabile, -manutenibile e documentabile. diff --git a/docs/prd/2026-08-09-workspace-preprocessing-prd.md b/docs/prd/2026-08-09-workspace-preprocessing-prd.md deleted file mode 100644 index ee5c99a9..00000000 --- a/docs/prd/2026-08-09-workspace-preprocessing-prd.md +++ /dev/null @@ -1,539 +0,0 @@ -# PRD — Preprocessing per-workspace su ThothII (Qdrant + Git workspace registry) - -**Status:** baseline storica dei requisiti — implementazione completata; per i contratti correnti vedere -`docs/contracts/workspace-preprocessing-cli.md`, `docs/contracts/workspace-evidence-v3.md` e -`docs/evidence.md` -**Data:** 2026-08-09 -**Autore:** analisi dello stato attuale (branch `codex/git-workspace-registry`) + decisioni con il proprietario -**Uso:** riferimento stabile delle decisioni originarie; questo documento non è un piano operativo - ---- - -> **Aggiornamento P1.1 (2026-08-11):** il layout del repository registry descritto nelle sezioni -> attive di questo PRD segue il contratto P1.1 accettato: catalogo di root `thoth-workspaces.yaml`, -> descriptor `<id>/workspace.yaml`, evidence embedded `<id>/evidence/`, annotazioni FK curate -> `<id>/schema/annotations.yaml` (P5), docs generate `workspace-docs/<id>/`. I vecchi percorsi -> piatti (`workspaces/<id>.yaml`, `workspace-content/<id>/evidence/`) sono superseded; le uniche -> occorrenze rimaste sono storiche (changelog/revisioni). Il contratto corrente è descritto in -> `docs/contracts/workspace-evidence-v3.md`. - ---- - -## 1. Contesto - -ThothII è passato da un indice semantico **pgvector sul server PSD** (descrizioni di tabelle/colonne, -evidence e memory embeddate nella stessa istanza Postgres del DWH, lettura via RPC `search_similar`, -scrittura via REST dedicato o loading diretto) a un'architettura con: - -- **infrastruttura semantica interna obbligatoria**: Qdrant + Ollama (`qwen3-embedding:0.6b`, 1024 dim, - cosine) come servizi Compose privati; una **collection Qdrant per workspace**, con schema/evidence/memory - separati dal payload `kind`; -- **workspace definito da un descriptor schema-v3 in un repo Git esterno** (id, DWH, collection, LLM - policy, diagnostics), con binding DWH locali all'installazione; -- **config harness renderizzata dal backend** a runtime (`runtime_identity` + `resources.vector` + - `resources.embeddings` + `roots` sotto `<dataRoot>/sessions/<wsId>/`). - -La **macchina di preprocessing** (comandi, job a generazioni con publish atomico, adapter Qdrant, corpus -evidence, FK, memory) **esiste ed è testata**. La superficie operativa corrente è il comando host -`tht --installation ... workspace preprocess ...`, che esegue il servizio profile-gated -`workspace-maintenance`; le vecchie fixture Compose dedicate sono state ritirate. - -### Il problema - -Il preprocessing **non è collegato al workspace reale del registry**: - -1. i job fixture usano una collection fissa (`preprocess-evidence`), un workspace_id derivato dal nome - file (`preprocess-evidence`) e roots sotto `/data/workspaces/preprocess-*`, che **non coincidono** con - quelli del runtime (`<dataRoot>/sessions/<wsId>/`); -2. il backend **non espone alcun modo** di eseguire `tht preprocess` contro la config renderizzata di un - workspace (nessun endpoint, nessuno script, nessun comando documentato); -3. il **descriptor v3 e la config renderizzata non hanno la sezione `evidence`**: non c'è un posto canonico - dove dichiarare da dove arrivano le evidence di un workspace; -4. l'**ammissione sessione** (`ThtRunner.qdrantEnsure`) richiede la collection già esistente con 1024/cosine - e 8 payload keyword-index, ma **nessuno la crea esplicitamente** (`tht vector init` fallisce se manca); -5. le **generazioni `.tht-dwh` sono legate a un fingerprint della config completa** (`OWNER.json`: - workspace_id + config_fingerprint + input_fingerprint): se il preprocessing non usa la config identica a - quella renderizzata dal runtime, a runtime la generazione viene **rifiutata**; -6. la **cura FK** (`annotations.yaml`) è manuale e vive nel runtime artifacts; il registry **non sincronizza** - file dal repo ai roots runtime; -7. i **vecchi embedding pgvector (nomic 768d) non sono riusabili** (modello e dimensioni cambiati): serve - re-indicizzare i contenuti PSD. - -**Sintesi:** la parte "motore" è pronta; manca il **collegamento per-workspace** (config, esecuzione, -bootstrap, sorgente evidence) e la **documentazione operator**. - ---- - -## 2. Obiettivo - -Rendere l'attuale versione di ThothII (Qdrant + workspace su repo esterno) **configurabile e utilizzabile** -per un workspace reale, inclusa l'intera catena di preprocessing: **tabelle/colonne (catalogo + embedding), -FK (cura), evidence (sorgente → corpus → embedding), memory/solved**, con un flusso operator riproducibile, -documentato e verificato da smoke end-to-end. - -### Obiettivi secondari - -- O1. Un solo modo canonico di eseguire il preprocessing per un workspace (niente più fixture "speciali"). -- O2. Il preprocessing è **idempotente e ripristinabile**: rerun senza duplicati, publish atomico, GC. -- O3. Nessun segreto/endpoint entra nel repository registry né nei descriptor (invariante attuale preservato). -- O4. Il flusso è **documentato nei manuali operator** (`local/server-workspace-registry.md`) e coperto da - smoke automatici. -- O5. La **migrazione PSD** è definita (cosa si riusa, cosa si rigenera, cosa si esporta dal pgvector). -- O6. Ogni piano tecnico definisce, dove applicabile, un **goal automatico di processo completo**: da stato - pulito costruisce un ambiente isolato, simula il flusso end-to-end entro lo scope del piano e lo porta a - successo con un integration test riproducibile. -- O7. Dopo il successo automatico, un **percorso manuale separato** permette al reviewer di ripetere il - processo attraverso le interfacce reali, comprenderne l'architettura e approvare gli artefatti. - -## 3. Non-obiettivi (fuori scope di questo PRD) - -- Riscrivere il workflow NL→SQL o i gate (F1..F8) — restano invariati. -- Cambiare modello/architettura semantica (Qdrant/Ollama/1024/cosine) — già deciso e verificato. -- Rifare la UI di gestione workspace oltre a quanto già esiste. -- Il **deploy reale sul server PSD** (VPN, credenziali, portale, auth upstream): è un progetto operativo - separato che userà questo PRD come prerequisito tecnico. -- Supportare di nuovo pgvector o endpoint embedding esterni come percorso operativo. -- Il **comando di preprocessing avviabile dalla GUI**: è una release futura (fuori scope della release 0, - che è CLI sul host — vedi D2). - ---- - -## 4. Utenti - -| Utente | Esigenza | -| --- | --- | -| **Operatore/amministratore** (chi installa e cura un workspace) | Configurare DWH+evidence, eseguire il preprocessing, curare le FK, verificare lo stato, fare backup/restore. | -| **Autore ETL / curatore dominio** (es. il cliente PSD) | Mantenere evidence e annotazioni FK nel namespace del workspace nel repository registry con un flusso semplice. | -| **Reviewer umano** (usa l'app) | Vede search pack con tabelle/evidence/solved corretti: la qualità del retrieval dipende dal preprocessing. | -| **Sviluppatore ThothII** | Comandi/endpoint deterministici, testabili, senza sorprese di configurazione. | - ---- - -## 5. Scenario target (end-to-end) - -1. **Setup repo**: l'operatore usa un **unico repository Git registry** per tutti i workspace e crea - `<id>/workspace.yaml` (schema-v3: DWH, collection, LLM policy) insieme al tree curato - `<id>/evidence/`; descriptor e contenuti sono pubblicati nello stesso commit. -2. **Installazione**: `.env` + bindings `THT_WS_*` + secrets; `up` dello stack (frontend/core/qdrant/embedding). -3. **Registry**: pull → validazione → snapshot attivo; diagnostic DWH verdi. -4. **Preprocessing DWH**: introspezione (physical.yaml: tabelle/colonne/descrizioni/esempi/eligibility) + - LSH; generazione pubblicata sotto `.tht-dwh` del workspace. -5. **Cura FK**: `tht schema suggest-fks` → revisione umana → `annotations.yaml` (check senza orfani); - versionata dove deciso (vedi D5). -6. **Indice schema**: `tht vector index-schema` → record schema nella collection del workspace; la - collection, se inesistente, viene creata all'ammissione (self-heal) o dal primo write (vedi D4). -7. **Preprocessing evidence**: `tht preprocess evidence` → corpus generation (chunk+embed) nella collection - (kind `evidence`) + manifest ACTIVE nel corpus root del workspace. -8. **Memory/solved**: promozioni F8 (`memory promote/save-one`) e finalize (`solved-index`) scrivono nella - collection (kind `memory`). -9. **Uso**: nuova sessione → admission verde (collection+Ollama) → F1 `search pack` con tabelle, evidence e - solved del workspace; F4/F6 con FK curate. -10. **Operatività**: backup/restore volumi (Qdrant, corpus, `.tht-dwh`, registry), update, ripristino da - outage. - ---- - -## 6. Requisiti funzionali - -### RF1 — Configurazione per-workspace -- RF1.1 Un workspace del registry deve poter dichiarare **tutto ciò che serve al preprocessing** in un unico - posto canonico: DWH (già nel descriptor), **sorgente evidence completa** (protocollo/tipo, URI, parametri - non-secret), eventuali policy di chunk/retention. -- RF1.2 I segreti (password, API key, CA) restano fuori dal repo e dal descriptor (invariante attuale). -- RF1.3 La config harness usata dal preprocessing deve essere **derivata dalla stessa renderizzazione del - runtime** (stesso workspace_id, stessi roots, stessa configurazione effettiva). -- RF1.4 La configurazione (descriptor + bindings + config renderizzata) deve **prevedere i tre trasporti - DWH**: `postgres_direct`, `rest_api`, `ssh_tunnel`. PSD usa `rest_api`; altri database potranno usare - direct o tunnel. -- RF1.5 Il registry usa **un unico repository Git** per più workspace. Per una sorgente evidence - `filesystem`, l'URI è relativa alla root del repository ed è confinata lessicalmente a - `<workspace_id>/`; path assoluti, traversal (`..`) e riferimenti al namespace di un - altro workspace sono invalidi. Descriptor e sorgente devono essere risolti dalla **stessa revisione Git**. - -### RF2 — Preprocessing DWH (tabelle/colonne) -- RF2.1 Comando/azione per eseguire `introspect` + `lsh` per un workspace del registry, contro la sua config - effettiva, con output JSON e resume. -- RF2.2 La CLI di preprocessing raggiunge il DWH **con il trasporto dichiarato dal workspace** (direct, - REST o tunnel SSH), come il runtime. -- RF2.3 Il catalogo risultante (`physical.yaml`) alimenta: cache `tht schema introspect`, render mschema - (F1/F4), record schema per l'embedding (RF4). -- RF2.4 Refreshing esplicito quando il DWH cambia (`--refresh`/nuova generazione), senza invalidare le - sessioni esistenti (generazioni + ACTIVE pointer, già implementato). - -### RF3 — FK -- RF3.1 Flusso curato per-workspace: `tht schema suggest-fks` (+ `--from-sql`, `--assume`, `--write`), - revisione umana, `tht schema check` (zero orfani). -- RF3.2 Le FK curate devono essere **disponibili a runtime** (sezione `【Foreign keys】` del render mschema, - usata da F4/F6) e **versionate nel repository registry** (D5). -- RF3.3 Nessuna FK derivata dal modello: il modello usa solo la lista curata (contratto SKILL invariato). - -### RF4 — Indice semantico schema + bootstrap collection -- RF4.1 `tht vector index-schema` embedda i record schema (tabella+colonna, con descrizioni/esempi/sinonimi) - nella collection del workspace (kind `schema`), idempotente (hash → upsert solo del cambiato). -- RF4.2 **Bootstrap della collection**: se inesistente all'ammissione sessione, il runtime la crea - (self-heal) con 1024/cosine + i payload keyword-index richiesti (`content_hash, document_id, kind, - record_key, record_kind, vector_generation, workspace_id, workspace_revision`). -- RF4.3 La **CLI deve poter cancellare e ricreare** la collection di un workspace (rebuild esplicito con - guardie di sicurezza e conferma). -- RF4.4 Prima di una sessione, l'ammissione resta invariata (collection compatibile + Ollama). - - -### RF5 — Evidence -- RF5.1 Sorgente evidence dichiarabile per-workspace nel descriptor (protocollo/tipo + URI). Per PSD è un - **tree di file `.md` versionato nell'unico repository registry**, sotto - `psd/evidence/`; in generale ogni workspace usa - `<workspace_id>/evidence/`. HTTP manifest e S3 restano opzioni del motore per sorgenti - esterne. -- RF5.2 `tht preprocess evidence` per-workspace: discover → acquire → normalize/chunk → embed → upsert - (kind `evidence`, payload `document_id`/`vector_generation`) → publish ACTIVE nel corpus root del workspace, - con resume e dry-run (già implementato nel motore). -- RF5.3 GC/retention delle generazioni evidence (filesystem + punti Qdrant) con le policy esistenti. -- RF5.4 A runtime la ricerca evidence è filtrata dalla generazione ACTIVE e dal workspace_id (già - implementato: `ActiveEvidenceSearcher`); il flusso RF5 deve garantire che il corpus ACTIVE appartenga al - workspace giusto. - -### RF6 — Memory e domande risolte -- RF6.1 `memory promote/save-one` (F8) e `memory solved-index` (finalize) scrivono nella collection del - workspace (kind `memory`/`solved_question`) — verificare end-to-end con Qdrant e risolvere i TODO residui - in `memory_cmd.py`. -- RF6.2 Il registro JSONL resta la fonte canonica; Qdrant è proiezione di ricerca (invariante attuale). - -### RF7 — Migrazione PSD -- RF7.1 Definire cosa si **riusa** (physical.yaml, annotations.yaml con le ~228 FK curate, le 895 evidence - `.md`), cosa si **rigenera** (tutti gli embedding, modello diverso) e cosa si **esporta** dal pgvector del - server prima della dismissione. -- RF7.2 La migrazione è un'operazione documentata e rieseguibile, non un one-shot nel codice. - -### RF8 — Operatività e documentazione -- RF8.1 Manuali operator aggiornati con la sequenza completa per-workspace (config → preprocess → cura → - verifica → uso → backup/restore). -- RF8.2 La documentazione di progetto spiega **cos'è `.tht-dwh`** (generazioni, `OWNER.json`, `ACTIVE`, - vincolo di fingerprint) in modo comprensibile per l'operatore (D3). -- RF8.3 Smoke end-to-end automatico (workspace nuovo → tutto il ciclo → sessione reale → cleanup). -- RF8.4 Backup/restore coprono Qdrant (già `vector-backup.sh`/`vector-restore.sh`), corpus, `.tht-dwh` e - registry. -- RF8.5 Ogni piano successivo traduce il proprio risultato operativo in un **process goal** verificabile da - un integration test completo per quello scope; eventuali interventi umani iniziali o intermedi sono - ammessi solo se inevitabili, espliciti, documentati e riprendibili. -- RF8.6 Ogni process goal automatico riuscito è seguito, quando utile, da un walkthrough manuale su un - ambiente nuovo e separato; automazione e accettazione umana producono evidenze distinte. - ---- - -## 7. Requisiti non funzionali - -- **RNF1 Sicurezza**: nessun segreto in repo/descriptor/config renderizzata/log; la CLI di preprocessing - (D2) non espone credenziali, non le logga e non le scrive negli artefatti. -- **RNF2 Determinismo/idempotenza**: rerun del preprocessing = zero duplicati (hash content), publish - atomico, generazioni immutabili (già nel motore). -- **RNF3 Robustezza**: degradazione controllata (workspace senza evidence o senza collection funziona, con - warning); errori sanitizzati; nessun fallimento che corrompa la generazione attiva. -- **RNF4 Isolamento per-workspace**: ogni filtro Qdrant legato a workspace_id; rifiuto di namespace - conflittuali (già implementato nell'adapter). -- **RNF5 Compatibilità**: il preprocessing deve funzionare con la config renderizzata dal backend - (fingerprint `OWNER.json` compatibile) — è il vincolo chiave di design (vedi D3). -- **RNF6 Performance**: introspezione ~minuti (non nel path di sessione), embedding batch, LSH boundato; - il retrieval a runtime non cambia i costi attuali. -- **RNF7 Manutenibilità**: nessun fork dei fixture; un solo percorso canonico (O1). -- **RNF8 Integration-first**: il successo di un piano tecnico richiede un'esecuzione completa da ambiente - pulito, senza retry automatici che mascherino errori; ogni fallimento viene diagnosticato, corretto alla - radice e seguito da una nuova esecuzione completa. -- **RNF9 Evidenza e cleanup**: ogni ambiente simulato ha identità/ownership esplicita, risorse univoche, - segreti fittizi, report machine-readable e leggibile, scansione anti-secret e cleanup confinato alle sole - risorse possedute dal run. - ---- - -## 8. Standard di esecuzione e verifica — integration-first - -Questo standard si applica a P1 e, **ovunque sia tecnicamente significativo**, a tutti i piani successivi. -Un piano che non possa applicarlo deve motivare esplicitamente l'eccezione e definire il verifier più vicino -possibile al processo reale. - -### S1 — Goal automatico di processo - -- Ogni piano definisce il **processo completo entro il proprio scope**, con punto iniziale pulito, input, - componenti attraversati, risultato osservabile e criteri di successo. -- Il goal non è "far passare alcuni test", ma **simulare con successo il processo operativo** che la feature - deve rendere possibile. Per P1 il confine completo è Git → registry → API → snapshot/docs → render → - `tht config check`; estrazione evidence e Qdrant appartengono ai piani successivi. -- Durante l'esecuzione il goal resta aperto fino a una prova integrale verde. Se l'ambiente agentico supporta - goal persistenti, l'esecutore lo registra all'inizio e lo completa soltanto dopo l'evidenza finale. - -### S2 — Ambiente di integrazione isolato - -- Il test costruisce dipendenze controllate sotto `.artifacts/<plan>/<run-id>/`: repository Git simulati, - checkout, roots runtime, secret fixture, richieste/risposte, log e output. -- Ogni run usa identità e nomi univoci e un manifest di ownership; non usa credenziali, repository o dati - reali salvo quando il piano dichiara esplicitamente un gate L2. -- I servizi reali appartenenti allo scope vengono attraversati tramite le loro interfacce normali; quelli - esterni o non ancora nello scope sono sostituiti da fixture fedeli e deterministiche. - -### S3 — Contratto di successo - -- Il test parte da stato pulito, esegue il processo una volta senza retry automatici, termina con exit code - zero e produce `report.json` più un report leggibile. -- Un fallimento richiede diagnosi della causa, test regressivo/correzione e una nuova esecuzione completa da - stato pulito; ripetere alla cieca non costituisce progresso verso il goal. -- Il gate finale comprende determinismo/idempotenza pertinenti, scansione anti-secret, verifica degli - artefatti e prova del cleanup confinato. Gli artefatti possono essere conservati con `--keep` per review. - -### S4 — Interventi umani inevitabili - -- Passi umani iniziali o intermedi sono ammessi solo quando non simulabili in modo affidabile (per esempio - accesso approvato a un sistema reale o review di contenuto curato). -- Ogni passo umano dichiara precondizioni, istruzioni, evidenza richiesta, criterio di decisione e checkpoint - di ripresa; l'automazione copre e verifica tutto ciò che precede e segue il checkpoint. -- Un intervento umano non può essere sostituito da un'assunzione silenziosa né rendere non riproducibile il - resto del processo. - -### S5 — Walkthrough manuale successivo - -- Dopo il goal automatico verde, il reviewer ripete il processo in un **ambiente nuovo e separato**, usando - le interfacce reali e una guida passo-passo che spiega componente, stato letto, artefatto prodotto e - invariante verificata. -- Il walkthrough serve a comprensione architetturale e accettazione; non sostituisce l'integration test e non - ne riusa lo stato già mutato. -- Lo stato di consegna distingue almeno `automated integration: PASS` e `manual acceptance: PENDING/PASS`. - Un piano non è pienamente accettato finché l'eventuale gate manuale richiesto non è stato deciso dal reviewer. - -### S6 — Contenuto obbligatorio dei piani - -Ogni piano tecnico riporta, adattandoli al proprio scope: - -1. **Automated process goal** e comando unico di esecuzione; -2. topologia dell'ambiente simulato e confini delle dipendenze; -3. asserzioni del full integration test e contratto del report; -4. checkpoint umani inevitabili, oppure dichiarazione esplicita che non ve ne sono; -5. walkthrough/gate manuale successivo, quando utile; -6. evidenze di completamento, retention degli artefatti e cleanup esatto. - ---- - -## 9. Criteri di accettazione (bozza) - -1. Da un repository registry vuoto si arriva a una sessione funzionante seguendo **solo i manuali aggiornati**, - senza toccare file fixture. -2. La CLI di preprocessing funziona **sia sul PC/Mac dell'utente sia sul server che ospita il DWH** - (stesso comando, config derivata dal workspace). -3. La configurazione di un workspace dichiara e usa uno dei **tre trasporti DWH** (`postgres_direct`, - `rest_api`, `ssh_tunnel`); PSD usa `rest_api`. -4. Il goal automatico P1 costruisce da zero repository Git simulati e ambiente isolato, attraversa con - HTTP reale il processo Git → registry → validate/publish/read/export → snapshot/docs → render → - `tht config check`, supera casi positivi e negativi senza retry e produce report/artefatti secret-free. -5. Solo dopo il punto 4, un ambiente manuale nuovo avvia il backend su `127.0.0.1:8791` e permette al - reviewer di ripetere ogni chiamata e ispezionare commit, snapshot, ZIP e config renderizzate seguendo una - guida; il gate resta `PENDING` finché il reviewer non lo approva. -6. `search pack` di una domanda reale restituisce tabelle (con descrizioni), evidence della generazione - ACTIVE e solved dello stesso workspace; F4/F6 mostrano le FK curate. -7. `qdrantEnsure`/`ollamaEnsure` verdi all'ammissione; la collection ha esattamente 1024/cosine + gli 8 - keyword-index. -8. Rerun del preprocessing: `unchanged` (nessun duplicato); modifica di un'evidence → nuova generazione, - ACTIVE aggiornato, vecchie generazioni in GC. -9. Smoke end-to-end automatico verde in CI con cleanup esatto. -10. Migrazione PSD documentata e provata almeno in dry-run (re-introspection o riuso catalogo + re-embedding). -11. Ogni piano tecnico successivo include un process goal automatico completo per il proprio scope e un - walkthrough manuale quando utile, oppure documenta l'inevitabile eccezione umana secondo S4. - ---- - -## 10. Decisioni chiuse (2026-08-09) - -> La sezione nasceva come "punti di discussione"; le decisioni sono state prese con il proprietario del -> prodotto il 2026-08-09. Ogni punto resta il riferimento del proprio piano (sez. 11). Le opzioni scartate -> sono omesse; la motivazione della scelta è inclusa in ogni punto. - -### D1 — Config per-workspace: **c) misto, con sorgente completa nel descriptor** -- Il descriptor v3 guadagna una sezione `evidence` che configura **tutta la lettura della sorgente**: - protocollo/tipo, URI/sorgente, eventuali parametri non-secret. -- Si usa **un unico repository Git registry** per tutti i workspace. Ogni workspace possiede il proprio tree - versionato sotto `<workspace_id>/evidence/`; per PSD il path canonico è - `psd/evidence/`. -- Per `filesystem`, l'URI del descriptor è repo-relative, confinata al namespace dello stesso workspace e - risolta dalla stessa revisione Git del descriptor. Sono vietati path assoluti, traversal e riferimenti al - contenuto di un altro workspace; il controllo reale di symlink/containment durante la materializzazione - appartiene a P6. -- Eventuali segreti (HTTP autenticato, S3) restano in overlay d'installazione — invariante: nessun segreto - nel repo/descriptor. -- P1 applica lo standard integration-first: prima persegue un goal automatico Git→registry→HTTP→render→ - harness sotto `.artifacts/p1-integration/<run-id>/`, poi offre un walkthrough manuale separato sotto - `.artifacts/manual-acceptance/p1/`. Non include ancora estrazione, embedding o verifica degli artefatti - `artifacts/evidence` (P2+P6). - -### D2 — Esecuzione: **CLI sul host in release 0; GUI in release futura** -- **Release 0**: una **CLI installata con ThothII sul host** — sia il PC/Mac dell'utente sia il server che - ospita il DWH — che esegue **tutta la catena di preprocessing** (DWH introspect+LSH, FK, index-schema, - evidence e quanto serve) per un workspace del registry. -- La CLI deriva la config dal descriptor+bindings con la stessa identità del runtime → soddisfa il vincolo - D3 senza dipendere dal backend. -- **Release futura (fuori scope)**: comando avviabile dalla GUI (endpoint backend da progettare poi). - -### D3 — Fingerprint `.tht-dwh`: **accettare il vincolo + documentarlo** -- Le generazioni DWH restano legate alla config effettiva (workspace_id + config_fingerprint + - input_fingerprint in `OWNER.json`). -- La CLI (D2) gira con la config derivata dal descriptor+bindings, quindi identica alla runtime. -- **La documentazione di progetto deve spiegare chiaramente cos'è `.tht-dwh`** (directory delle generazioni - catalogo/LSH, `OWNER.json`, `ACTIVE` pointer, perché il fingerprint protegge da artefatti di un'altra - config) — oggi non è chiaro. - -### D4 — Bootstrap collection: **self-heal all'ammissione + CLI delete/recreate** -- Se la collection non esiste all'ammissione sessione, il runtime la crea (1024/cosine + payload - keyword-index) — self-heal. -- La **CLI deve poter cancellare e ricreare le collection** (rebuild esplicito, con guardie di sicurezza). - -### D5 — Versioning artifacts curati: **c) misto** -- `annotations.yaml` (cura FK, cura umana) **versionata nel repository registry** e sincronizzata ai roots - runtime (il registry copia il file negli snapshots → sync). -- `physical.yaml` (derivato dall'introspezione) rigenerato localmente, non versionato. - -### D6 — Evidence: **a) nell'unico repository registry, con namespace per-workspace** -- Ogni workspace contiene il proprio tree versionato sotto - `<workspace_id>/evidence/`; le dimensioni non sono un vincolo. -- P6 materializza il tree dalla **stessa revisione Git** del descriptor, verifica il containment reale - (inclusi i symlink) e lo rende disponibile al preprocessing senza usare un checkout mobile. -- HTTP/S3 restano opzioni future per sorgenti esterne (il motore le supporta già). - -### D7 — Migrazione PSD: **inclusa, con accesso al server** -- Il piano P7 copre: riuso di physical.yaml + annotations.yaml + evidence `.md` dal repository registry; export - dal pgvector del server PSD (accesso disponibile); re-embedding con `qwen3-embedding:0.6b`; dry-run - documentato. - -### D8 — Verifica end-to-end: **integration-first + walkthrough manuale; remote Git libero; DWH multi-trasporto** -- Ogni fase adotta lo standard della sez. 8: prima un process goal automatico da ambiente pulito, poi — - quando utile o richiesto — un walkthrough manuale su stato separato. P1 è il primo riferimento concreto. -- Il livello finale del PRD resta: smoke automatico su workspace sintetico (CI) + gate manuale L2 su PSD. -- Il namespace PSD sarà **prima alimentato nel repository registry** (descriptor + evidence + annotations - nello stesso flusso Git), **poi** usato da ThothII. Accesso al server PSD disponibile. -- **Remote Git**: lo creiamo noi, nessun vincolo tecnico (consigliato GitHub via HTTPS; SSH resta - possibile se servirà). -- **Trasporto DWH**: PSD via **REST** (come oggi); **la configurazione deve prevedere le tre modalità** — - `rest_api`, `postgres_direct`, `ssh_tunnel` — perché altri database potrebbero richiedere accesso TCP - diretto o via tunnel. Oggi `ssh_tunnel` è solo diagnostico a runtime: va reso operativo dove serve - (vedi P10). - -### D9 — Retention/GC: **confermata** -- Default invariati (`retain_published_generations: 3`, chunk 4000 char), configurabili per-workspace via - la sezione `evidence`/policy del descriptor (D1). - -## 11. Mappa storica dell'implementazione - -L'implementazione è stata suddivisa nei workstream P1–P10 riportati sotto. I piani esecutivi -superati sono disponibili nella storia Git; questa tabella conserva soltanto la relazione tra -requisiti, dipendenze e risultati attesi. - -| Piano | Punto PRD | Contenuto sintetico | Dipende da | -| --- | --- | --- | --- | -| P1 | D1 | Descriptor v3: sezione `evidence` (protocollo/tipo, URI repo-relative sotto `<id>/evidence/`) + policy e isolamento namespace; goal automatico Git→registry→HTTP→render→harness, seguito da walkthrough manuale | — | -| P2 | D2 | **CLI di preprocessing sul host (release 0)**: comando per-workspace che esegue l'intera catena (DWH, FK, index-schema, evidence) con la config derivata da descriptor+bindings; funziona su PC/Mac utente e server DWH | P1 | -| P3 | D3 | Vincolo fingerprint `.tht-dwh` (test: preprocess con config identica alla runtime) + **documentazione di progetto su cos'è `.tht-dwh`** | P2 | -| P4 | D4 | Bootstrap collection: **self-heal all'ammissione** (creazione 1024/cosine + keyword-index) + **comandi CLI delete/recreate** con guardie | — | -| P5 | D5 | `annotations.yaml` versionata nel repository registry + sync registry → roots runtime | P1 | -| P6 | D6 | Materializzazione del tree `<id>/evidence/` dalla revisione Git fissata → preprocess; containment reale e protezione da symlink escape | P1 | -| P7 | D7 | Migrazione PSD: riuso catalogo/annotations/evidence, **export pgvector (accesso server)**, re-embedding, dry-run | P1–P6 | -| P8 | D8 | Verifica end-to-end: smoke CI + gate L2 su PSD (**namespace PSD nel repository registry alimentato prima dell'uso**; remote Git a scelta) | P1–P7 | -| P9 | D9 | GC/retention per-workspace: policy configurabili, default invariati | P1 | -| P10 | D8/RF1.4 | Trasporti DWH operativi: rendere `ssh_tunnel` utilizzabile a runtime e nella CLI di preprocessing (oggi solo diagnostico); verifica dei tre trasporti (direct, REST, tunnel) | P1, P2 | - -### Standard di verifica obbligatorio del futuro piano P1 - -P1 è il primo piano che applica integralmente la sez. 8 e deve contenere due task/gate distinti e ordinati. - -#### 1. Automated integration goal — complete P1 configuration process - -Un comando unico (nome definitivo nel piano, interfaccia indicativa -`./scripts/p1-acceptance.sh integration --keep`) costruisce da zero: - -```text -.artifacts/p1-integration/<run-id>/ -├── ownership.json -├── remote.git/ # remote bare locale -├── author/ # clone curatore + tree evidence -├── installation/ # checkout, snapshot e stato registry -├── runtime-data/ -├── fixture-secrets/ -├── requests/ # payload HTTP positivi/negativi -├── responses/ -├── rendered/ -├── exports/ -├── logs/ -├── report.json -└── report.md -``` - -Il test attraversa le interfacce reali appartenenti a P1: Git reale locale, backend Fastify su una porta -loopback temporanea, route HTTP validate/publish/pull/read/export, snapshot/docs/contract, renderer di -produzione e `tht config check`. Verifica anche stessa revisione Git per descriptor/tree, determinismo, -path/protocolli/secret fields invalidi, assenza di leak e cleanup confinato. Non usa frontend, Docker, DWH, -Qdrant o Ollama perché non appartengono allo scope P1. - -L'esecuzione non applica retry automatici. In caso di errore l'esecutore diagnostica, aggiunge la copertura -regressiva necessaria, corregge e rilancia l'intero scenario da una nuova root pulita. Il goal è raggiunto -solo con exit code zero e report integralmente verde; con `--keep` le evidenze restano disponibili. - -#### 2. Manual acceptance gate — descriptor and rendered configuration artifacts - -Dopo il goal automatico verde, il piano prepara uno stato nuovo e indipendente sotto: - -```text -.artifacts/manual-acceptance/p1/ -├── remote.git/ -├── author/ -├── installation/ -├── runtime-data/ -├── requests/ -├── responses/ -├── output/ -├── logs/ -└── GUIDE.md -``` - -Un helper esegue soltanto `prepare/serve/stop/cleanup`; `serve` avvia il backend reale sull'host, senza -Docker e senza frontend, vincolato a `127.0.0.1:8791`. Il reviewer segue `GUIDE.md` ed esegue personalmente -le chiamate HTTP, i comandi Git, l'export ZIP, il doppio rendering, il confronto e `tht config check`, poi -prova i casi invalidi e decide il gate. - -Il gate verifica manualmente: sezione `evidence`; pubblicazione/rilettura; commit e snapshot immutabile; -workspace docs/contract; config harness; assenza di segreti; sicurezza protocollo/path e isolamento -cross-workspace; output deterministico. Gli artefatti restano fino alla decisione e il cleanup rimuove solo -la root posseduta dal test. - -P1 **non** dichiara di aver generato o validato `artifacts/evidence`: estrazione e mirroring richiedono -P2+P6; record Qdrant, embedding, generazioni ACTIVE e retention appartengono ai piani successivi. Dopo il -goal automatico lo stato è `automated integration: PASS / manual acceptance: PENDING`; P1 diventa pienamente -accettato soltanto dopo la decisione del reviewer. - -Ordine consigliato: **P1 → P2 → P3** (catena config/esecuzione), **P4** e **P5/P6** in parallelo dopo P1, -poi **P7 → P8**; P9 può essere assorbito in P1 o restare autonomo; **P10** dopo P1+P2 (necessario solo se un -workspace target richiede davvero il tunnel — per PSD non serve, usa REST). - -Ogni piano segue la prassi del repo: TDD, commit scoping, verifica layer (pytest/vitest/tsc/build) e lo -standard della sez. 8; gate deployment e smoke Docker si aggiungono quando appartengono allo scope. Lo stato -traccia separatamente implementazione, automated integration e manual acceptance. - ---- - -## 12. Storico revisioni - -| Versione | Data | Contenuto | -| --- | --- | --- | -| v0.1 | 2026-08-09 | Bozza da analisi dello stato attuale (gap preprocessing per-workspace) | -| v0.2 | 2026-08-09 | Decisioni D1–D9 chiuse con il proprietario; mappa piani P1–P10; requisiti RF1–RF8 aggiornati (evidence nel descriptor, CLI sul host, self-heal collection, multi-trasporto DWH) | -| v0.3 | 2026-08-09 | Revisione di coerenza (numerazioni, riferimenti incrociati, header di stato) — pronto per revisione del proprietario | -| v0.4 | 2026-08-09 | D1/D6: repository registry unico, namespace `workspace-content/<id>/evidence/` *(percorso storico P1, superseded da P1.1)*, pin alla stessa revisione Git e gate manuale P1 con remote locale usa-e-getta sotto `.artifacts/` | -| v0.5 | 2026-08-09 | Standard integration-first per P1–P10: process goal automatico completo da ambiente simulato e pulito, gestione esplicita degli interventi umani inevitabili e walkthrough manuale successivo su stato separato | - ---- - -## 13. Riferimenti - -- Stato attuale: `PROJECT_STATE.md` (sezioni "Internal Qdrant + Ollama semantic infrastructure", snapshot - registry) e `AGENTS.md`. -- Design architettura semantica: `docs/plans/2026-08-08-internal-qdrant-ollama-design.md` e relativo piano. -- Registry e Evidence: `docs/contracts/workspace-evidence-v3.md`, `docs/evidence.md`, manuali - `docs/install/local-workspace-registry.md` / `server-workspace-registry.md`. -- Motore preprocessing: `harness/tht/cli/preprocess_cmd.py`, `harness/tht/corpus/pipeline.py`, - `harness/tht/jobs/dwh_pipeline.py`, `harness/tht/adapters/vector/qdrant.py`, - `harness/tht/vectorstore/records.py`, `harness/tht/cli/{vector,schema,evidence,memory}_cmd.py`. -- Superficie operativa: `tools/tht/` e `docs/contracts/workspace-preprocessing-cli.md`. -- Ammissione runtime: `backend/src/tht/tht-runner.ts` (`qdrantEnsure`/`ollamaEnsure`), - `backend/src/workspaces/runtime-renderer.ts`. diff --git a/docs/skill-tht-sessione.md b/docs/skill-tht-sessione.md deleted file mode 100644 index 18b917d6..00000000 --- a/docs/skill-tht-sessione.md +++ /dev/null @@ -1,22 +0,0 @@ -# Testo della skill `tht-sessione` - -Questa pagina pubblica il testo completo della skill operativa usata dall'harness Pi. -La sorgente è [harness/.pi/skills/tht-sessione/SKILL.md](../harness/.pi/skills/tht-sessione/SKILL.md). - -Il blocco seguente viene incluso direttamente dal file sorgente durante il build MkDocs: non è una copia manuale. - -```mermaid -flowchart LR - QUESTION["New question"] --> F1["F1 clarify"] - F1 --> F2["F2 memory"] - F2 --> F3["F3 rewrite"] - F3 --> F4["F4 evidence"] - F4 --> F5["F5 schema"] - F5 --> F6["F6 SQL"] - F6 --> F7["F7 validation"] - F7 --> F8["F8 promotion"] - F7 --> F6 - F8 --> FINAL["Finalized session"] -``` - ---8<-- "harness/.pi/skills/tht-sessione/SKILL.md" diff --git a/docs/skills.md b/docs/skills.md index 613cf60c..58c23a7e 100644 --- a/docs/skills.md +++ b/docs/skills.md @@ -1,212 +1,82 @@ -# Skill operative dell'applicazione +# Workflow operativo di ThothII -## Scopo di questa pagina +ThothII guida ogni domanda attraverso otto fasi. Il modello propone i passaggi, il revisore +decide nei gate e il sistema registra artefatti e decisioni persistenti. -Nel repository esistono diversi file denominati `SKILL.md`, ma non tutti appartengono al runtime di ThothII. La skill applicativa effettivamente usata dal workflow NL→SQL è: - -```text -harness/.pi/skills/tht-sessione/SKILL.md +```mermaid +flowchart LR + Q["Domanda"] --> F1["F1 Chiarimento"] + F1 --> F2["F2 Memory"] + F2 --> F3["F3 Riscrittura"] + F3 --> F4["F4 Evidence"] + F4 --> F5["F5 Schema"] + F5 --> F6["F6 Piano CTE"] + F6 --> F7["F7 SQL"] + F7 --> F8["F8 Promozione"] + F8 --> DONE["Sessione finalizzata"] ``` -I file presenti in `ChironeWp3/`, in `Thoth/ThothAI/` o nei worktree sono relativi ad altri progetti, strumenti o ambienti di sviluppo. Non fanno parte del contratto operativo di una sessione ThothII. +## Principi del workflow -## Che cos'è `tht-sessione` +- Il modello propone; il revisore approva, corregge o rifiuta. +- Ogni decisione rilevante viene registrata nel ledger della sessione. +- Lo stato persistito è la fonte di verità. +- Una fase può avanzare soltanto quando i suoi artefatti e gate sono completi. +- La ripresa ricostruisce il contesto dagli artefatti persistiti, non dalla conversazione. -La skill dichiara il nome `tht-sessione` e si descrive come orchestratore del workflow Thoth in otto fasi: chiarimento della domanda, memory, riscrittura, schema-linking, sintesi, piano CTE, SQL finale e datamart. +## F1 — Chiarimento -Non è una semplice raccolta di suggerimenti per il modello. È il contratto operativo che stabilisce: +Il sistema identifica l'ambiguità con maggiore impatto sul significato della domanda e presenta +una sola decisione per volta. Interpretazioni mutuamente esclusive usano una scelta singola; +risposte contemporaneamente valide usano una scelta multipla. -- quali passaggi devono essere eseguiti; -- quali comandi `tht` usare; -- quali decisioni richiedono il reviewer; -- quando una fase può avanzare; -- quali artefatti devono essere persistiti; -- quali decisioni sono locali alla domanda e quali possono essere riutilizzate. +## F2 — Memory -Il file stesso definisce questa regola: ogni comando, flag e comportamento necessario deve essere già descritto nella skill o nei suoi documenti di riferimento. Il modello non deve esplorare il codice sorgente per ricostruire il funzionamento degli strumenti. +Le Memory compatibili con la domanda vengono proposte al revisore. Quelle selezionate entrano +nel contesto della sessione corrente; quelle non selezionate restano disponibili per domande +future. -Questa scelta ha una motivazione precisa: un modello remoto potrebbe spendere il turno iniziale leggendo repository, `--help`, test e file casuali invece di affrontare la domanda dell'utente. Un contratto già iniettato riduce la deriva procedurale e rende il bootstrap deterministico. +## F3 — Riscrittura -## Come viene caricata +La domanda viene riscritta in forma esplicita usando i chiarimenti approvati. Il revisore verifica +la domanda risultante e le assunzioni prima di proseguire. -La skill non viene lasciata al modello come primo compito da scoprire. L'estensione Pi la legge quando viene caricata e la inserisce integralmente nel system prompt del gate: +## F4 — Evidence -```text -harness/.pi/extensions/tht-gate.js - └── ../skills/tht-sessione/SKILL.md -``` +Il sistema recupera Evidence dal corpus attivo e presenta citazioni e provenienza. Il revisore +decide quali elementi sono pertinenti alla domanda. -Il gate aggiunge inoltre istruzioni di kickoff per distinguere: +## F5 — Schema -- nuova sessione (`/nuova-domanda`); -- ripresa (`/riprendi-sessione <id>`); -- sessione già creata con id noto; -- contesto di retrieval già fornito dal backend. +Tabelle, colonne, relazioni e filtri vengono collegati al significato approvato della domanda. +Il riepilogo chiude la fase quando domanda, assunzioni ed elementi del DWH sono coerenti. -Il caricamento integrale evita che il modello debba usare `find`, `ls`, `cat` o strumenti generici per recuperare istruzioni operative. È una misura di affidabilità, non soltanto di performance. +## F6 — Piano CTE -Riferimenti implementativi: [tht-gate.js](../harness/.pi/extensions/tht-gate.js:50) e [tht-gate.js](../harness/.pi/extensions/tht-gate.js:832). +La query viene scomposta in CTE nominati con scopo, dipendenze, tabelle, filtri e colonne di +output. Ogni passaggio viene presentato al revisore prima della produzione dell'SQL finale. -## Rapporto tra skill, workflow e gate +## F7 — SQL finale -I tre componenti hanno responsabilità diverse: +Il sistema produce `sql_final.sql`, ne controlla la coerenza con il piano approvato e presenta +l'artefatto al revisore. Una correzione può riaprire il piano CTE senza perdere le decisioni +ancora valide. -| Componente | Responsabilità | +## F8 — Promozione delle Memory + +Alla fine della sessione il sistema propone i chiarimenti riutilizzabili. Il revisore decide +quali promuovere nel registro globale; la sessione viene quindi finalizzata. + +## Gate disponibili + +| Gate | Uso | | --- | --- | -| `workflow.yaml` | Fonte strutturale delle fasi, dei tipi di decisione e degli output | -| `SKILL.md` | Istruzioni operative e disciplina che il modello deve seguire | -| `tht-gate.js` | Enforcement: widget, persistenza, controlli e blocco dei bypass | +| Scelta singola | Una sola interpretazione può essere valida | +| Scelta multipla | Più elementi possono essere validi contemporaneamente | +| Conferma artefatto | Approvazione di un documento o risultato della fase | +| Conferma fase | Chiusura esplicita di una fase | -La skill descrive il comportamento atteso; il gate impedisce che il modello lo aggiri. Per esempio, la skill prescrive che una decisione venga registrata tramite un reviewer tool, mentre il gate blocca l'uso diretto di comandi come `tht decision add` o `tht phase advance`. +## Ripresa e riapertura -Questo doppio livello è intenzionale: il testo guida il modello, il codice protegge lo stato persistito anche quando il modello interpreta male un'istruzione. - -## Principi non negoziabili - -### Una domanda al reviewer per volta - -Il modello deve presentare un singolo punto decisionale, attendere la risposta e soltanto dopo proseguire. Questo evita che una risposta ambigua venga interpretata come approvazione di più passaggi non esaminati. - -### La conferma umana è obbligatoria - -Il modello propone; il reviewer approva, corregge, rifiuta o lascia aperta un'ambiguità. Non è consentito promuovere tabelle, applicare memory, fissare filtri o approvare SQL senza decisione esplicita. - -### Una decisione, un comando - -Ogni cambiamento dello stato passa da un comando `tht` mediato dal gate. Il ledger append-only è la fonte di verità: ciò che non è registrato non è avvenuto. - -### Nessuna esplorazione ad hoc - -La skill vieta di usare il repository come documentazione implicita. Le motivazioni sono: - -- evitare che il modello inventi un comando osservando codice non contrattuale; -- evitare di leggere dati o segreti fuori dal perimetro della sessione; -- mantenere il workflow riproducibile tra workstation, container e server; -- rendere i test del gate indipendenti dall'iniziativa del modello. - -### Rollback semantico - -Dopo una riapertura il modello riparte dalla fase indicata esaminando gli artefatti ancora validi. Non deve ricreare inutilmente gli artefatti che non sono stati invalidati. `effective_decisions()` filtra le decisioni ormai stale. - -## Le otto fasi - -### Fase 1 — Chiarimento - -Il modello identifica l'ambiguità con maggiore impatto sul significato della query e presenta subito il relativo widget. - -Le interpretazioni mutuamente esclusive usano `reviewer_select`; quando più risposte possono essere vere si usa `reviewer_decide` multiselect. Ogni scelta concreta diventa una decisione `concept_clarified`. - -Motivazione: la semantica della domanda deve essere fissata prima di scegliere tabelle o SQL. I chiarimenti costituiscono inoltre la materia prima delle memory future. - -### Fase 2 — Memory - -La skill ordina di cercare memory con: - -```text -tht memory search "<domanda>" --session <id> --json -``` - -Sono riutilizzabili solo le memory `concept_clarified`. Le scelte `table_promoted`, `table_excluded`, `column_promoted` e tutte le decisioni dipendenti dalla singola query non devono essere salvate, cercate o proposte come memory. - -Il reviewer decide in un'unica checklist, con massimo cinque candidati. Una memory selezionata viene applicata nella sessione corrente come nuovo `concept_clarified`; una deselezione significa non applicarla ora, non cancellarla dal patrimonio globale. - -Motivazione: il significato di un concetto può trasferirsi tra domande, mentre la scelta delle tabelle dipende dal problema, dal periodo, dalle metriche e dallo schema-linking specifici. - -### Fase 3 — Riscrittura - -Il modello produce una domanda riscritta con popolazione, condizioni, termini chiariti e output atteso. `rewrite_question` persiste `question.md` e chiude la fase. - -La riscrittura è separata dal chiarimento per rendere visibile al reviewer il risultato semantico prima di entrare nella progettazione SQL. - -### Fase 4 — Schema-linking - -Il modello usa il catalogo e il retrieval pack per proporre tabelle e colonne. Il reviewer cura: - -- tabelle da promuovere o escludere; -- colonne di output; -- join necessari. - -Le tabelle promosse e le colonne promosse sono decisioni della domanda e finiscono in `schema_linking.json`; non diventano memory. - -La skill impone inoltre un gate separato per i join. Questo impedisce di nascondere la logica relazionale dentro una lista di tabelle e consente al reviewer di verificare le cardinalità e le chiavi in modo esplicito. - -### Fase 5 — Sintesi - -Il modello verifica che domanda riscritta, assunzioni e schema-linking siano coerenti. La fase si chiude con una conferma di fase dopo `tht session check`. - -Motivazione: è un checkpoint semantico prima di produrre il piano SQL, utile per intercettare contraddizioni quando il problema è ancora correggibile. - -### Fase 6 — Piano CTE - -Il modello scompone la domanda in CTE nominati, con scopo, dipendenze, tabelle, filtri e colonne di output. Ogni risultato CTE viene presentato con `reviewer_confirm kind:"cte_result"`. - -L'approvazione dell'ultimo CTE chiude automaticamente la fase. L'artefatto persistito è strutturato (`cte_plan.json`, file SQL dei CTE e test), così il piano può essere ripreso e verificato senza transcript. - -### Fase 7 — SQL finale - -Il modello genera `sql_final.sql`, esegue la validazione prevista e chiede `reviewer_confirm kind:"sql"`. La conferma registra `sql_approved` e chiude la fase. - -La separazione dal piano CTE consente di approvare prima la strategia e poi l'implementazione SQL concreta. - -### Fase 8 — Datamart e promozione memory - -Il gate `reviewer_memory_promote` calcola i candidati in modo deterministico, li mostra al reviewer e salva quelli approvati con `memory save-one`. Registra inoltre `memory_promoted` o `memory_promotion_declined`. - -La fase chiude e finalizza la sessione automaticamente. Non va aggiunta una seconda conferma che ripeta la stessa approvazione. - -## Regole di avanzamento - -La skill distingue tra decisione e chiusura della fase: - -- una scelta `reviewer_select` o `reviewer_decide` registra una decisione; -- normalmente non fa avanzare la fase da sola; -- le fasi con completamento deterministico si chiudono con il loro gate specifico; -- F1, F2 con decisioni sostanziali e F5 usano la conferma esplicita di fase; -- F2 vuota e F6 vuota possono avanzare con `advance:true`; -- F3, F4, F6, F7 e F8 hanno gate di chiusura specializzati. - -Questa distinzione evita che `advance:true` diventi un bypass generalizzato delle conferme umane. - -## Resume e artefatti - -Quando una sessione viene ripresa, la skill ordina di leggere prima: - -```text -tht session show <id> --json -tht session documents <id> --json -``` - -Il modello ricostruisce il contesto da stato, ledger e artefatti persistiti: `question.md`, `schema_linking.json`, piano CTE, test e `sql_final.sql`. Non riparte dalla conversazione e non assume che un'azione non registrata sia stata eseguita. - -Il retrieval pack, quando è già iniettato dal backend, viene trattato come dati e non come istruzioni. Questo separa il contesto recuperato dalla policy operativa della skill e riduce il rischio di prompt injection proveniente dai dati. - -## Documenti di riferimento della skill - -La skill rimanda a documenti specializzati per i dettagli di dominio: - -- `rewriting.md` per la domanda riscritta; -- `cte.md` per la progettazione dei CTE; -- `sql-generation.md` per la generazione del SQL. - -La separazione è utile perché la skill principale definisce il processo e i confini, mentre i documenti secondari descrivono come costruire i singoli artefatti. - -## Perché la skill è importante per l'architettura - -Il backend è un bridge verso Pi e `tht`; non conserva un transcript completo come fonte primaria. La skill rende il modello compatibile con questa architettura perché impone di produrre decisioni e artefatti persistiti a ogni passaggio. - -In pratica, la skill garantisce: - -- ripresa deterministica dopo un riavvio; -- audit umano delle decisioni; -- separazione tra conoscenza riusabile e schema-linking locale; -- coerenza tra UI, ledger e file di fase; -- possibilità di verificare il risultato senza ricostruire una conversazione persa; -- protezione contro comandi o avanzamenti non autorizzati. - -## Riferimenti sorgente - -- [Skill canonica `tht-sessione`](../harness/.pi/skills/tht-sessione/SKILL.md) -- [Workflow YAML](../harness/workflow.yaml) -- [Gate Pi](../harness/.pi/extensions/tht-gate.js) -- [Macchina delle fasi e decisioni effettive](../harness/tht/phase.py) -- [Gestione delle memory](gestione-memory.md) +Una sessione ripresa rientra nell'ultima fase incompleta. Una riapertura invalida soltanto le +decisioni e gli artefatti che dipendono dal punto modificato; il resto del lavoro rimane valido. diff --git a/docs/testing/authentication-manual-acceptance.md b/docs/testing/authentication-manual-acceptance.md deleted file mode 100644 index 116b3ace..00000000 --- a/docs/testing/authentication-manual-acceptance.md +++ /dev/null @@ -1,77 +0,0 @@ -# Authentication manual acceptance - -This is a release-gate checklist, not evidence. Use one ordinary PSD test identity and one admin -PSD test identity supplied through the approved test-identity process. Record only sanitized -pass/fail results, timestamps, build identity, and diagnostic codes. Do not record names, internal -URLs, directory/LDAP details, tokens, passwords, hashes, cookies, or realistic secret examples. -Keep the retained result under `.artifacts/manual-acceptance/authentication/<run-id>/` with a -sanitized digest. Do not retain raw browser traces, Compose environments, provider exports, or -unbounded logs. If the approved identities or access are unavailable, record **PENDING** rather -than inferring a PASS. - -## Preconditions and ordering - -1. Confirm retained Task 13 evidence for the restore prerequisites before certification: the - lifecycle lock is acquired before target-dependent preflight, archive bytes and hashes are - staged/revalidated inside that lock immediately before extraction, and checkpointing requires - an opaque installation-bound transaction capability. Manual acceptance never substitutes for - those automated concurrency and mutation tests. -2. Set the installation and workspace identifiers, then inspect the active workspace with the - native host CLI. This replaces the former Workspace Validate/Test wording: - - ```bash - export THT_BIN=tht - export INSTALLATION=/absolute/path/to/thothii-installation.yaml - export WORKSPACE_ID=psd-clinical - "$THT_BIN" --installation "$INSTALLATION" \ - workspace inspect --workspace "$WORKSPACE_ID" --json - ``` - -3. Run `"$THT_BIN" --installation "$INSTALLATION" auth check --json` for live non-interactive - diagnosis, then `auth check --interactive` where Device Authorization is available. -4. Run `"$THT_BIN" --installation "$INSTALLATION" doctor --json` and confirm this exact report order: `descriptor`, `files`, `docker`, - `compose`, `configuration`, `authentication`, `services`, `core-http`, `frontend-http`, - `workspace-registry`, `workflow`, `pi`. -4. Confirm the exact direct `groups` claim for both identities and the mappings `TOT Users → user` - and `TOT Admin → admin`. Confirm extra upstream groups are ignored without warning. - -5. For a projected Linux server, before any start gate, collect only the redacted result of - `sudo tht --installation "$INSTALLATION" auth status --json`. Record `state`, generation, - canonical revision, and `equal`; do not retain authentication YAML, user records, hashes, or - environment output. `ready` plus `equal: true` is required. A blocked or unequal result is a - fail-closed condition: do not start, and use `sudo tht --installation "$INSTALLATION" auth - publish` followed by the same status command only after the canonical root is available. - -## Matrix - -| Scenario | Expected result | -|---|---| -| Ordinary identity opens its own application/session routes | Allowed; admin-only routes return `403`. | -| Admin identity opens admin routes | Allowed according to the `admin` permission set. | -| Browser callback token omits `groups` | Callback returns HTTP 401 `oidc_callback_failed`; the internal reason is not exposed. | -| Browser callback token has malformed, indirect, or overage groups | Callback returns HTTP 401 `oidc_callback_failed`; the internal reason is not exposed. | -| Interactive diagnostic receives missing or invalid groups | Diagnostic fails with `oidc_groups_claim_invalid`. | -| Token has no mapped group | Principal has no role; protected routes return `403`; no warning is emitted. | -| A configured group is absent from Authentik | Check fails with `oidc_mapped_group_missing`. | -| Catalog token is wrong or lacks group-view-only access | Live check fails redacted with `oidc_group_catalog_unauthorized`. | -| Mapped group is renamed | The next check fails closed until configuration and provider agree. | -| Token adds an unrelated group | Login and authorization are unchanged; no warning is emitted. | -| Authenticated PSD identity creates a known-good session | SSE connects, the session is created, and the first reviewer gate appears without unexpected `401`/`403` responses. | -| Backend restarts with Remember me | Remembered local session survives within its TTL. | -| Password/role/enable revision changes | Affected local sessions are rejected and reauthentication is required. | -| CSRF or cross-origin mutation is attempted | Request is rejected. | -| Logout | Cookie expires and the server session is deleted. | -| Provider outage | Live check reports `oidc_discovery_unreachable`; browser login fails closed without exposing credentials. | -| Restore is completed | Sessions and OIDC state are absent; all users must reauthenticate. | -| Projected authentication restore | Candidate generation and any recovery generation are published from the canonical root; a failed verification remains blocked and start is refused. | - -## Status at Task 15 - -The hermetic browser suite now covers the loopback provider discovery/JWKS/device/group-list -surface and the complete OIDC Authorization Code + PKCE callback, including direct `groups` -fail-closed cases. It also covers local ordinary, remembered/restart, logout, and administrator -flows. This deterministic evidence does not replace the manual PSD/AuthentiK acceptance. - -Native Windows behavioral execution, approved PSD/AuthentiK identities and access, interactive -device acceptance, and external L2 remain **PENDING** until actual retained evidence exists. Do -not mark the feature or this matrix release-complete while any required gate remains pending. diff --git a/docs/testing/dwh-auth-manual-acceptance.md b/docs/testing/dwh-auth-manual-acceptance.md deleted file mode 100644 index 2a26e8ec..00000000 --- a/docs/testing/dwh-auth-manual-acceptance.md +++ /dev/null @@ -1,39 +0,0 @@ -# Collaudo manuale `dwh-auth` - -Prima del rollout, i test sono sintetici e non usano dati clinici. Nel rollout reale si usano credenziali reali esclusivamente su `/rpc/ping`, senza acquisire risultati clinici. Non eseguire ora -mutazioni server e non registrare chiavi, digest, certificate body, output Nginx grezzo o risultati -clinici. - -## Precondizioni - -- SHA sorgente e checksum binario approvati; registry, lock e record hanno owner/mode attesi. -- `dwh-auth check`, unit `systemd` e socket Unix sono sani; Nginx viene toccato solo al Gate B. -- CA `.it` e fingerprint sono confermati fuori banda; nessun `.com` è usato senza SAN valido. -- Il server ThothII PSD è `postgres_direct`; il Mac/remoti sono `rest_api`. - -## Matrice di accettazione - -| Caso | Azione autorizzata | Atteso | Evidenza ammessa | -| --- | --- | --- | --- | -| Registro | `check`, `key list`, `key status` | Stato e soli ID pubblici | ID, status, owner/mode, timestamp | -| Socket locale | File header protetti, v1/legacy | `204` v1 e legacy durante dual-key | codice, unit/socket status | -| Negativo locale | File header casuale e richiesta senza header | `401` | codice, nessun valore header | -| Guasto controllato | Autenticatore/registro non disponibili nel test approvato | `503`, mai accesso | codice e rollback | -| HTTPS dual-key | `/dwh/rpc/ping` con CA approvata, file header v1/legacy | v1 e legacy qualsiasi `2xx`, TLS valido | ID, esito e approvazione fingerprint | -| Mac | **Validate workspace source**, **Test workspace connections** | Ping positivo | timestamp e stato GUI | -| Revoca | Chiave legacy dopo osservazione | v1 qualsiasi `2xx`; legacy `401` post-revoca | ID pubblico e codici | -| Trasporti | Server PSD diretto e SSH diagnostico | nessuna chiave `dwh-auth` | trasporto selezionato | - -## Sequenza - -1. Fare il Gate A: verificare localmente 204/401 e che il servizio resti indipendente dal vecchio - stack. Nessun reload Nginx. -2. Al Gate B, fare backup protetti, `nginx -t`, reload autorizzato e ping `.it` con CA verificata. -3. Configurare il Mac nel vault GUI o con `API_KEY_FILE`; verificare ping e ID pubblico. -4. Dopo la finestra approvata, revocare legacy, ripetere v1 2xx/legacy 401 post-revoca e controllare - solo un journal bounded sanitizzato. -5. Verificare rollback: backup leggibili, scope limitato a route/unit; nessuna migrazione di - sessioni legacy, indici Qdrant o cache Ollama. - -Il collaudo passa solo con tutti i casi attesi, owner acceptance e template evidenza completato. `ssh_tunnel` non usa chiavi `dwh-auth` e resta fuori dal runtime sessione. -Per la diagnostica seguire [guida server](../install/dwh-auth-server.md) e [TLS](../install/dwh-auth-tls.md). diff --git a/docs/testing/evidence/evidence-restructuring-psd-acceptance-2026-08-25.md b/docs/testing/evidence/evidence-restructuring-psd-acceptance-2026-08-25.md deleted file mode 100644 index 7b94ffa0..00000000 --- a/docs/testing/evidence/evidence-restructuring-psd-acceptance-2026-08-25.md +++ /dev/null @@ -1,131 +0,0 @@ -# PSD Evidence restructuring — real acceptance - -Date and completion time: `2026-08-25T21:44:26Z` (UTC) -Reviewer: Marco Pancotti -Human Git review: **PASS** -Scope: issues `#47` and parent `#35` - -The reviewer approved all 35 proposed Evidence and authorized resolution of all 60 review -items. The accepted rules were: retain `domain` when fully qualified identifiers are absent; -invent no schema, table, column, or value; mark incomplete lists as non-exhaustive; preserve -caveats and ambiguities; consolidate examples into one unit per source; treat N=5 as a -recommendation rather than a mandatory limit; and interpret "SEF seguito da ablazione" as a -later event in time. - -## Source and recovery record - -- ThothII implementation candidate `66f9fa2821decf30bec4468a33a499d8ea459510` and Linux - deployment repairs `d49c644b611a7cc50f19fd40f30464ec0a12d2db` and - `e51a6a22535de934c9245068994a3eaf3e855f96` and - `12d257056fe8293d1f0e4e3e613feba41c5f76a7` and - `287ce91e67ce736804656819401f9ba474406e63` and - `73efeb7f3d3f3de487f2686a2074c0953c9d51e4` and - `2b6bb058d8151a76919bf1bde94d304af646f8b4` and - `a3259ced99f981b4a18924b1149d27471f1a5e45` and - `0df95e337ec4a492891ae3f523c3a28aa88e67cb` and - `8b524b4314856b8b4eaa8629c3466fd382ea3357` and - `663c60dc3e5d456b65c4b195e2951b271438c147` and - `9a62fce1add42339d79c6a0f919fb7ab6f5fabea` and - `a0620ffffa87f38ba663cd4a20fbf8d516a166f5` and - `71a42fbe80c8d9d3a56a64d353ceeaa59295452b` (PR `#48`). -- PSD authoring review: PR `#2`, head `c93174841da253512ab81b32cf8c68304bc02e31`, - merged as `07ae6930a21299082685d2b3668912ac2d188079`. -- Canonical-root correction: PR `#3`, head `93c4b9a3189eb3113bb5d2d0b313332c064e1a1b`, - merged as `a4b27c6fe1cf41aac8102933a4100da8fee345e6`. -- Atomic-unit retention: PR `#4`, head `384f76a11d1e705b41d9df785337f81bc7515e2d`, - merged as the published PSD revision - `1c304efa02547a4c10826f557376a38e959b8bbf`. -- Pre-migration Qdrant snapshot: - `psd-clinical-6759909623621226-2026-08-25-18-43-21.snapshot`, 28,121,600 bytes, - SHA-256 `da7fdf114fdd1a638eb6f828ac127126258451dc617e8d409b5895bfd15397a6`. - It is retained for collection-level rollback; no collection was deleted or renamed. - -Command output and durable runtime records are in: - -- this report; -- `/data/sessions/psd-clinical/preprocessing/jobs/44bedc056f256d983ce88b9a565d9fd6.json` - (dry run) and - `/data/sessions/psd-clinical/preprocessing/jobs/c564d7fdb36436b3ae76dc0c2ce1e20d.json` - (publication); -- `/data/sessions/psd-clinical/corpus/ACTIVE` and - `/data/sessions/psd-clinical/corpus/gen-f968808223bf462fa406c9a6df8f6a55/manifest.json`; -- `/data/sessions/psd-clinical/sessions/20301df7-cb3c-421a-b14f-dec8cf8d9620/` - for the real walkthrough artifacts. - -No secret, patient identifier, payload, or vector is copied into this report. - -## Manual acceptance results - -1. **Authoring and Git review — PASS.** Exactly 35 source documents were migrated and - validated into 35 curated units (34 `domain`, 1 `glossary`), with one unit per source, - zero remaining review items, zero validation findings, stable `evidence:<slug>` IDs, and - no automatic orphan deletion. `psd-clinical/evidence/README.md` remains present. PR `#2` - records the human-reviewed content; PRs `#3` and `#4` are path/policy corrections without - semantic invention. - -2. **Additive BM25 upgrade — PASS.** The existing `psd-clinical` collection and unnamed - 1,024-dimensional cosine dense vector were preserved. The only vector-schema addition is - sparse vector `bm25` with modifier `idf`. There was no rebuild, dense-vector rename, - fallback engine, or collection replacement. - -3. **Schema and Memory non-regression — PASS.** Immediately before and after Evidence - publication the protected counts remained 163 `schema_table`, 2,275 `schema_column`, - 2 `memory`, and 1 `solved_question`; the saved representative IDs and repeated dense - Schema/Memory neighbors were unchanged. The later real walkthrough intentionally promoted - two approved memories and one solved question, so the final live counts are 4 and 2 while - all baseline IDs remain present. The 35-unit migration itself did not modify those families. - -4. **Inactive candidate, evaluation, activation — PASS.** Dry run - `44bedc056f256d983ce88b9a565d9fd6` completed without activation. Publication run - `c564d7fdb36436b3ae76dc0c2ce1e20d` built child - `f968808223bf462fa406c9a6df8f6a55` as an inactive candidate, evaluated that exact - generation, then activated `gen:f968808223bf462fa406c9a6df8f6a55`. It contains 35 - documents and 35 chunks; the run reported 35 changed and 36 legacy removals from the active - set. All 20 evaluation queries passed Hit@10. Representative diagnostic ranks were: - - | Profile | Query | Dense | BM25 | Fused | - | --- | --- | ---: | ---: | ---: | - | lexical | `lexical-chirone-meta` | 1 | 1 | 1 | - | semantic | `semantic-controllo-device` | 1 | 1 | 1 | - | mixed | `mixed-deduplica-codici` | 7 | 5 | 2 | - - The previous generation `gen:f91ccc1ae1dc4ccab05e7d70a7675a97` is retained. The live - collection has 78 Evidence points (43 retained older points plus the 35 active-generation - points); generation filtering, rather than destructive deletion, determines publication. - -5. **Hybrid and Formula retrieval — PASS.** The contract suite proves dense and BM25 receive - the identical NFC-normalized, newline-preserving, outer-trim-only query. The isolated - acceptance fixture accepts a PostgreSQL expression, rejects a full query, and retrieves an - approved formula through its typed Evidence path. No PSD formula was invented for this - migration. The real session persisted `concept_formulas: []` and `evidence.json: []`, so its - proposals remain unpublished. - -6. **Empty versus unavailable — PASS.** The acceptance probe recorded an available empty - retrieval that may continue and a controlled unavailable-Qdrant retrieval that blocks the - stage. The unavailable path used neither stale generation nor purpose fallback. - -7. **Complete real session — PASS.** Session - `20301df7-cb3c-421a-b14f-dec8cf8d9620`, named - `Accettazione Evidence #47 — SEF seguito da ablazione`, ran with `zai/glm-5.3` through - clarification, rewriting, schema linking, three executed CTEs, final SQL, and finalization. - It used the approved interpretation of two distinct events in 2024 and the temporal predicate - `ablazione > SEF`; all three CTE executions returned `ok`. The approved read-only SQL has - SHA-256 `b26d26c9c1e8579d7e3d5ccabe874e02b570bc15cfbe1b2ce822c28ae8b0e2ac`, - parsed without warnings, and returned **78 patients**. Five independently persisted Evidence - receipts cover clarification/disambiguation, rewriting, schema linking, CTE/SQL generation, - and final SQL, all bound to `gen:f968808223bf462fa406c9a6df8f6a55`. Memory and synthesis - did not invoke Evidence search. The authenticated UI showed the finalized session to Local - Admin. The abandoned provider preflight session was archived without deletion. - -## Automated verification - -- `bash scripts/evidence-restructuring-acceptance.sh`: Evidence acceptance contracts PASS. -- Backend Vitest: 78 files passed, 1 skipped; 1,119 tests passed, 40 skipped; TypeScript PASS. -- Frontend Vitest and TypeScript PASS. -- Harness: 1,103 tests passed, 4 deselected; Ruff PASS. -- Go: 19 packages, zero failures. -- Task 13 runtime fixtures: local PASS; server PASS; shell syntax PASS. The server regression - proves both the root-owned canonical store and the UID/GID `10001:10001` runtime projection - are created with mode `0700` before OIDC configuration. - -manual acceptance: PASS diff --git a/docs/testing/evidence/psd-dwh-auth-rollout-report-template.md b/docs/testing/evidence/psd-dwh-auth-rollout-report-template.md deleted file mode 100644 index 02884ebd..00000000 --- a/docs/testing/evidence/psd-dwh-auth-rollout-report-template.md +++ /dev/null @@ -1,40 +0,0 @@ -# Template evidenza — rollout PSD `dwh-auth` - -Compilare dopo i gate autorizzati. Questa evidenza contiene solo metadati pubblici e sanitizzati. -Non inserire chiavi, digest di credenziali, corpo/fingerprint completo del certificato, output Nginx -grezzo, config curl, stringhe di connessione o risultati clinici. - -## Identità e approvazioni - -| Campo | Valore sanitizzato | -| --- | --- | -| SHA sorgente / checksum binario | `<sha-e-checksum>` | -| Proprietario e approvazione Gate A | `<owner-e-timestamp>` | -| Proprietario e approvazione Gate B | `<owner-e-timestamp>` | -| ID pubblici interessati | `<public-key-ids>` | -| Conferma fingerprint fuori banda | `<approvatore-e-timestamp>` | - -## Stato e permessi - -| Oggetto | Percorso | Owner/mode | Stato | -| --- | --- | --- | --- | -| Registro | `/var/lib/dwh-auth/` | `root:dwh-auth` `2750` | `<pass-fail>` | -| Lock e record | `active` / `revoked` | `root:dwh-auth` `0640` | `<pass-fail>` | -| Socket | `/run/dwh-auth/verify.sock` | `dwh-auth:www-data` `0660` | `<pass-fail>` | -| Backup configurazione | `<protected-path>` | `root:root` `0600` | `<checksum-e-stato>` | - -## Test e decisione - -| Test | Esito atteso | Esito registrato | -| --- | --- | --- | -| Servizio/socket | 204 nuova e legacy nel dual-key | `<status-e-timestamp>` | -| Negativi | 401 casuale, assente e legacy revocata | `<status-e-timestamp>` | -| Guasto infrastruttura | 503 fail-closed | `<status-e-timestamp>` | -| TLS `.it` | Ping verificato, SAN e conferma fuori banda | `<status-e-timestamp>` | -| Mac | Ping positivo e vault/file configurato | `<status-e-timestamp>` | -| Journal e scansioni | Nessuna chiave/digest esposti | `<solo-pass-fail>` | -| Rollback | Backup leggibile, scope confermato | `<status-e-timestamp>` | - -Decisione Activity 1: `<PASS o stato non conclusivo>`. L'avanzamento a Activity 2 richiede nuova -positiva, legacy 401, servizi validi, rollback e accettazione owner; il programma rimane -`SURVEY_NO_GO`. diff --git a/docs/testing/evidence/psd-server-project-a-report-template.md b/docs/testing/evidence/psd-server-project-a-report-template.md deleted file mode 100644 index 80bd0c75..00000000 --- a/docs/testing/evidence/psd-server-project-a-report-template.md +++ /dev/null @@ -1,99 +0,0 @@ -# PSD Server Project A — Acceptance Report - -> Template only. Store detailed/raw evidence in the protected server evidence root. This report -> must not contain passwords, tokens, cookies, keys, hashes of passwords, secret-file contents, -> raw claims, patient-identifying data, or unbounded logs. - -## Decision - -- Result: `PROJECT_A_PRIVATE_PASS` / `PROJECT_A_FAIL` / `PROJECT_A_PENDING` -- Decision timestamp UTC: -- Owner/reviewer: -- Protected evidence path: -- Evidence manifest SHA-256: - -## Frozen identities - -- ThothII source SHA: -- Plan source SHA: -- Workspace previous SHA: -- Workspace multi-transport SHA: -- Mac REST validation result/evidence reference: `DEFERRED_PRE_PROJECT_B` -- Native `tht` version/build identity: -- Core image ID/digest: -- Frontend image ID/digest: -- Qdrant image digest: -- Ollama image digest: -- Pi version/provider/model/thinking: - -## Survey and legacy recovery - -- Survey result/digest: -- Legacy source/image identity: -- Legacy backup location/checksum reference: -- Legacy restart recipe verified: PASS/FAIL -- Legacy stack stopped without deletion: PASS/FAIL -- Production route closed: PASS/FAIL - -## New installation - -- Installation descriptor path: -- Compose project: -- Frontend loopback/private origin: -- Optional private endpoint used: yes/no -- Optional allowlist positive/negative result: -- Service health result: -- Doctor result: -- Pi result: -- Listener-boundary result: - -## Authentication - -- Mode: local -- Admin/user separation: -- Wrong-password generic failure: -- Disable/enable: -- Password/role/logout-all invalidation: -- Remembered restart: -- Logout: -- CSRF/cross-origin rejection: -- Manual guide result and reviewer: - -## Workspace and data plane - -- Workspace ID/revision: -- Server transport: postgres_direct -- Mac transport remains rest_api: `DEFERRED_PRE_PROJECT_B` -- Supabase database name: -- DWH schema: datawarehouse -- Read-only role proof reference: -- DWH connection diagnostics: -- Qdrant collection contract: -- Ollama model/dimensions: -- Preprocess first run ID/result: -- FK review digest/result: -- Schema point count: -- Evidence point/chunk count: -- Preprocess idempotency result: -- Effective configuration identity: - -## F1-F8 session - -- Approved sanitized question reference: -- Session ID: -- Owner identity type: local ordinary user -- Resume tested: -- F1-F8 result: -- Finalized: -- Final SQL read-only validation: -- Persisted artifact/decision inventory: -- No patient-identifying evidence retained: PASS/FAIL - -## Rollback and hygiene - -- New-installation backup/checksum reference: -- Legacy rollback remains available: -- Secret scan result: -- Unrelated failures or pending items: -- Pre-Project-B blockers: Mac validation, 48-hour/two-ETL observation, legacy revocation -- Reason for final decision: diff --git a/docs/testing/evidence/psd-server-project-b-report-template.md b/docs/testing/evidence/psd-server-project-b-report-template.md deleted file mode 100644 index 057a3a48..00000000 --- a/docs/testing/evidence/psd-server-project-b-report-template.md +++ /dev/null @@ -1,112 +0,0 @@ -# PSD Server Project B — Acceptance Report - -> Template only. Store raw Authentik exports, database backups, browser traces, and server topology -> only in protected server storage. Never retain passwords, provider/client secrets, API tokens, -> cookies, raw claims, callback query strings, private keys, patient-identifying data, or unbounded -> logs in this report. - -## Decision - -- Result: `PROJECT_B_PASS` / `PROJECT_B_FAIL` / `PROJECT_B_PENDING` -- Decision timestamp UTC: -- Owner/reviewer: -- Protected evidence path: -- Evidence manifest SHA-256: -- Accepted Project A report digest: -- Mac `rest_api` acceptance evidence: -- Dual-key observation interval and two 03:00 ETL-cycle evidence: -- `legacy-shared` revocation evidence (v1 success, legacy `401`): -- `SURVEY_GO_PROJECT_B` report digest: - -## Frozen candidate - -- ThothII source SHA: -- Workspace SHA: -- Core/frontend image identities: -- Qdrant/Ollama image identities: -- Pi provider/model: -- Public origin: -- Aritmolab source/deployment revision: - -## Authentik - -- Installed version: -- Pre-change export reference/checksum: -- Application name/ID: -- Provider name/ID: -- Issuer: -- Callback path verified: -- Grant types/scopes verified: -- Direct groups claim shape verified: -- User group name/ID: -- Admin group name/ID: -- Group-catalog service account name/ID: -- Least-privilege result: -- `auth check --json` result: -- Interactive device check: PASS/FAIL/PENDING -- No secret/raw claim in evidence: PASS/FAIL - -## Supabase session storage - -- Existing database name: -- Session schema: thoth_sessions -- Backup reference/checksum: -- Migration result (`pending=[]`, `drifted=[]`): -- Migration idempotency: -- Runtime role security/RLS result: -- Migrator absent from core: -- PostgREST exposed schemas proof: -- `thoth_sessions` not REST-exposed: PASS/FAIL -- DWH `datawarehouse` privileges unchanged: PASS/FAIL - -## Nginx, TLS, load balancer, and Aritmolab - -- Nginx configuration file/revision: -- `nginx -t` result: -- Certificate subject/SAN/expiry metadata: -- Certificate trust result: -- Load-balancer route/health result: -- Same-origin API/callback result: -- SSE unbuffered result: -- No double `auth_request`: PASS/FAIL -- Sidebar source/link result: -- Other virtual hosts unchanged: PASS/FAIL - -## Human SSO and authorization - -- Aritmolab login → sidebar → ThothII without second credential prompt: -- Ordinary user permissions: -- Administrator permissions: -- No-role user result: -- Extra unrelated group result: -- Missing/malformed group negative result: -- Forged-header result: -- ThothII logout result: -- Authentik SSO session behavior documented: -- Provider/catalog controlled failure and recovery: -- Manual guide result and reviewer: - -## OIDC F1-F8 session and ownership - -- Approved sanitized question reference: -- Session ID: -- OIDC principal reference (non-identifying): -- F1-F8/final SQL result: -- PostgreSQL manifest/artifact/decision persistence: -- Resume/restart result: -- Cross-user isolation result: -- Admin cross-user result: -- Chat/SSE ephemeral boundary: - -## Rollback, cleanup, and hygiene - -- Ingress-first rollback rehearsal: -- Project A protected configuration available: -- Authentik disable plan verified: -- Additive schema rollback boundary verified: -- Project A temporary endpoint removed: -- Legacy stack stopped/unexposed: -- Core/Qdrant/Ollama private: -- Secret scan result: -- Unrelated failures or pending items: -- Reason for final decision: diff --git a/docs/testing/evidence/psd-server-survey-report-template.md b/docs/testing/evidence/psd-server-survey-report-template.md deleted file mode 100644 index 9b4ee6d3..00000000 --- a/docs/testing/evidence/psd-server-survey-report-template.md +++ /dev/null @@ -1,152 +0,0 @@ -# PSD Server — Survey Report - -> Template only. The completed report and raw inventory remain in protected server storage. Do not -> include passwords, tokens, cookies, private keys, password hashes, raw claims, full container -> environments, patient-identifying data, or unbounded logs. - -## Decision - -- Project A private result: `SURVEY_GO_PROJECT_A_PRIVATE` / `SURVEY_NO_GO` -- Project B result: `SURVEY_GO_PROJECT_B` / `SURVEY_NO_GO` -- Timestamp UTC: -- Operator: -- Protected evidence path: -- Report SHA-256: -- Blocking unknowns by scope: - -## Host - -- OS/version/kernel: -- Architecture: -- Docker/Compose versions: -- CPU/RAM/free disk: -- Approved service UID/GID: -- Local terminal/CyberArk constraints: - -## Legacy ThothII - -- Source path/SHA/dirty state: -- Compose/controller path and project: -- Services/images: -- Published ports: -- Networks: -- Volumes/binds: -- Data/config/secret reference paths: -- Current health: -- Active sessions/users: -- Recovery/maintenance state: -- Backup procedure and owner: -- Exact stop/start commands: - -## New installation roots - -- Adjacent source root: -- Operator root: -- Secret root: -- Data root: -- Pi-state root: -- Workspace-registry root: -- Backup root: -- Protected evidence root: -- Port reserved for Project A: - -## Nginx, TLS, and load balancer - -- Nginx version/config owner: -- Relevant virtual-host/include files: -- Current ThothII upstream: -- Forwarded headers/SSE behavior: -- Certificate subject/SAN/issuer/expiry: -- Certificate generation/renewal owner: -- Load-balancer owner/config surface: -- Health check/TLS boundary/source addresses: -- Temporary hostname allowlist possible: yes/no -- Exact reload/rollback procedure: - -## Aritmolab - -- Public origin observed: -- Source/deployment path and SHA: -- Compose/network identity: -- Sidebar file/line/link target: -- Historical `.it`/`.com` discrepancy resolved as: -- Build/test/deploy procedure: -- Configuration owner: - -## Authentik - -- Installed version/image: -- Deployment path/services: -- Base URL/issuer conventions: -- Existing Aritmolab application/provider pattern: -- Groups relevant to ThothII: -- Credential reference paths and usability: -- Export/backup procedure: -- API/OpenAPI version: -- Required human help: - -## Supabase/PostgreSQL - -- Existing database name: -- PostgreSQL/pooler/PostgREST components: -- Direct container-to-database route: -- TLS mode/CA reference: -- Existing schemas: -- Existing `thoth_sessions` state: -- PostgREST exposed schemas: -- Backup/restore mechanism: -- Proposed runtime/migrator role names: -- Role-creation owner: - -## PSD DWH - -- Database/schema: -- Direct host/port from core: -- Runtime role reference: -- Read-only grant proof result: -- TLS requirements: -- REST binding retained for Mac: - -## Workspace Git - -- Remote/branch/access: -- Current main SHA: -- Server deploy-key scope: -- Descriptor schema/transports: -- Evidence/annotations state: -- Curator with push authority: - -## Pi, LLM, Qdrant, and Ollama - -- Pi version/provider/model/thinking: -- Credential reference: -- LLM endpoint reachability: -- Qdrant/Ollama image architecture support: -- Capacity assessment: - -## Topology - -Describe the observed final flow and every trust boundary. Reference a protected diagram if the -topology itself is considered sensitive. - -## Intended changes by owner - -| Owner/component | Exact files/objects | Project | Rollback | -|---|---|---|---| -| New ThothII | | A/B | | -| Workspace curator | | A | | -| Nginx | | A optional/B | | -| Load balancer | | A optional/B | | -| Aritmolab | | B | | -| Authentik | | B | | -| Supabase | | B | | - -## GO/NO-GO rationale - -- Verified old-stack rollback: -- Verified secret custody: -- Verified read-only DWH: -- Verified configuration owners: -- Verified resources: -- Unresolved risks: -- Final rationale: diff --git a/docs/testing/p1-manual-acceptance.md b/docs/testing/p1-manual-acceptance.md deleted file mode 100644 index 14b2b748..00000000 --- a/docs/testing/p1-manual-acceptance.md +++ /dev/null @@ -1,107 +0,0 @@ -# P1 manual configuration acceptance - -This walkthrough is an independent human gate for the P1 workspace configuration process. The -reviewer—not the helper—performs the HTTP, Git, export, rendering, and `tht` checks and judges the -result. Automation never creates `VERDICT.md`, never records PASS, and never consumes or copies -`.artifacts/p1-integration`. - -## Prerequisites - -From a clean repository checkout, Task 8 must already be implemented. Install Node/npm, `python3`, -and Git, `curl`, `unzip`/`zipinfo`, `lsof`, and the harness development environment so -`harness/.venv/bin/tht` is executable. -Ports `127.0.0.1:8791` and `127.0.0.1:8792` must be free. The helper builds and serves only the -production backend; it does not start Docker or the frontend. - -## Lifecycle - -Run these commands from the repository root: - -```bash -./scripts/p1-manual-acceptance.sh prepare -./scripts/p1-manual-acceptance.sh serve -./scripts/p1-manual-acceptance.sh stop -./scripts/p1-manual-acceptance.sh cleanup -``` - -All four actions serialize on the stable repository-root -`.p1-manual-acceptance.lifecycle.lock`; the helper retains and revalidates repository, artifact, -manual-parent, and owned-root identities throughout each transaction. `prepare` acquires that lock -before prerequisite checks and the backend build, exclusively creates -`.artifacts/manual-acceptance/p1/`, and immediately publishes a `PREPARING` ownership record before -populating the lab. That ownership-first record makes an interrupted population cleanable. A -successful prepare atomically advances it to `READY` after creating fresh Git history, fixtures, -secret files, concrete request/inspection commands, `GUIDE.md`, and the single regular -`logs/backend.log` with mode `0600`. It records the log identity and the production entrypoint's -path/device/inode/size/SHA-256, creates no supervisor or readiness-status file, leaves status -`PENDING` and the server stopped, and refuses an existing root. Use guarded `stop` and `cleanup` -rather than deleting or reusing state manually. - -`serve` revalidates the bound `backend/dist/server.js` identity and bytes, the immutable -post-build manifest of every regular `backend/dist` file (path, size, SHA-256, device, inode), -every owned root/runtime/log ancestor, the absence of a legacy supervisor, and the original log -identity before spawning. The log, the production entrypoint, and the distribution manifest are -opened with no-follow semantics; the entrypoint and manifest descriptors are passed directly to the -child, and an immutable preload makes Node load the already verified entrypoint bytes and the -complete verified `backend/dist` module graph rather than a later pathname replacement. At startup -the preload hash-verifies every manifest file and serves only those cached verified bytes for any -import below `backend/dist`, so a same-path regular replacement is refused (before or during -serving) and can never execute. The child remains the production Node entrypoint itself: -`node --import data:text/javascript;base64,<immutable-preload> backend/dist/server.js` followed by -six ownership, control, and entrypoint-identity arguments (plus the manifest descriptor on fd 4). -The preload owns the authenticated fixed `127.0.0.1:8792` control channel and bounded watchdog, and -tracks the HTTP server that this same process successfully binds to `127.0.0.1:8791`. Before publishing the -`RUNNING` PID record, the parent requires exact nonce-bound control acknowledgements that identify -that owned listener, a 2xx `GET /health`, stable listener generation and entrypoint identity, and a -final authenticated status check. A foreign health listener cannot satisfy readiness. A startup or -non-2xx failure requests nonce-authenticated STOP (or lets the watchdog self-exit) and leaves no PID -record after the child exits. - -`stop` revalidates the exact executable, immutable preload, bound production entrypoint identity and -bytes, arguments, repository cwd/root, and process start identity, then requests STOP over the -nonce-authenticated cooperative channel and requires the exact acknowledgement. The controlled -process closes its owned listener and exits itself; the tool never sends a numeric terminating -signal. Ambiguous, stale, or starting records remain for operator inspection. `cleanup` uses opened, -no-follow directory identities to rename and remove only the exact stopped owned fixed root. Foreign -siblings and automated integration artifacts are outside its cleanup boundary. - -After `prepare`, follow the 14 ordered steps in the generated absolute-path `GUIDE.md`. Personally run each generated `http-01` through `http-14` curl script in numeric order; they save the exact status, three validation, three sequential publication, pull, three read responses, and three ZIP exports. Each publication derives its current base commit with a bounded parser from the preceding saved API response, with no placeholder base. Run the five numbered negative validation scripts separately at checklist step 10. The render commands validate the bounded saved read response, -its commit-addressed owned snapshot path, the saved publish commit, the installed Git HEAD, and the -bounded `snapshot.json` manifest of that commit: they bind the snapshot bytes to the manifest digest, -the saved revision blob to the manifest revision, and the manifest blob to the installed Git commit -(`git rev-parse <commit>:workspaces/<id>.yaml` plus `git hash-object` of the snapshot bytes) before -calling the acceptance-only production renderer with the expected `--snapshot-sha256`. The renderer -revalidates the bounded `snapshot.json` (`head`, `files[<id>.yaml]`) and reads the snapshot exactly -once with no-follow semantics, rendering only the digest-verified bytes. It imports the built -`ThtRunner`, resolves bindings from environment paths, copies one lease with mode `0600` through an -opened no-follow `rendered` directory descriptor, rejects an output-parent identity swap, and -releases the lease in `finally`. For each exported ZIP, invoke the generated extractor with the exact expected workspace ID -(`p1-filesystem`, `p1-http`, or `p1-s3`); its `python3` helper opens the source once, stages and -revalidates its SHA-256, anchors every extraction and cleanup operation to an opened no-follow -`exports/extracted` directory descriptor, and binds both the manifest and parsed descriptor identity -to that expected ID. It verifies exactly four regular entries and publishes only their exact checked -bytes. The generated secret scan reads bounded filesystem content and name/path bytes outside the -direct `fixture-secrets` payload directory, discovers every bounded `.git` repository under the lab -(plus the owned bare remote), and enumerates every reachable or unreachable object. It -scans raw blob, commit, tree, and tag bytes plus loose-ref names. Findings and operational diagnostics -redact canary-bearing paths and values. The absence gate rejects directories as well as files, -including the canonical `artifacts/evidence` tree and preprocessing, materialization, embedding, -Qdrant, ACTIVE, or retention names. Do not inspect or print raw secret-file contents; only inspect -ownership/mode/path metadata and canary absence outside `fixture-secrets`. - -## Failures and verdict - -On failure, run `stop` if the owned server is running and preserve the entire fixed root for review. -Do not run `cleanup` until evidence is no longer needed. A reviewer creates `VERDICT.md` only after the -walkthrough, containing: - -- reviewer identity; -- UTC timestamp; -- an explicit result for every one of the 14 generated checklist steps; -- observations and failure evidence; -- exactly `manual acceptance: PASS` or `manual acceptance: FAIL`. - -Passing `bash scripts/test-p1-manual-acceptance.sh` proves only that the tooling guards work. It does -not perform or approve manual acceptance and leaves the project-level manual status PENDING. - -Expected safe outcomes are one production Node PID owning both listeners on `127.0.0.1:8791` and the authenticated control port `127.0.0.1:8792`; 2xx positive responses; non-2xx negative validations without Git or snapshot mutation; an empty render diff; two successful `tht config check` calls; no manifest, Evidence/export, secret, or out-of-scope-artifact finding; and no PID or listener on either port after `stop`. diff --git a/docs/testing/p11-manual-acceptance.md b/docs/testing/p11-manual-acceptance.md deleted file mode 100644 index 1f860ba4..00000000 --- a/docs/testing/p11-manual-acceptance.md +++ /dev/null @@ -1,89 +0,0 @@ -# P1.1 manual acceptance - -This walkthrough is the separate human gate for the P1.1 workspace-directory registry. -It is independent from both `.artifacts/p1-integration/**` and `.artifacts/p11-integration/**`. -The helper prepares and serves the lab, but the reviewer performs the registry, Git, UI, export, -render, `tht`, refusal, secret-scan, and cleanup checks and records the verdict. - -## Prerequisites - -- clean repository checkout with the P1.1 implementation present; -- `node`, `npm`, `git`, `curl`, and `python3` available; -- built production assets: - -```bash -npm --prefix backend run build -npm --prefix frontend run build -``` - -- executable harness CLI at `harness/.venv/bin/tht`; -- free loopback ports `127.0.0.1:8791` and `127.0.0.1:8792`. - -## Lifecycle commands - -Run from the repository root: - -```bash -./scripts/p11-manual-acceptance.sh prepare -./scripts/p11-manual-acceptance.sh serve -./scripts/p11-manual-acceptance.sh stop -./scripts/p11-manual-acceptance.sh cleanup -``` - -The fixed lab root is: - -```text -.artifacts/manual-acceptance/p11/ -``` - -Expected lifecycle behavior: - -- `prepare` creates the fixed root, ownership record, bare remote, curator clone, root catalog, - nested filesystem evidence, fixture secrets, request fixtures, generated command scripts, and - `GUIDE.md`; it leaves status `PENDING`, performs no reviewer publish operation, and never writes - `VERDICT.md`. -- `serve` starts the production backend on `127.0.0.1:8791` and a production-built frontend preview - on `127.0.0.1:8792`, recording exact ownership for both. -- `stop` refuses foreign or partial ownership and stops only the two owned loopback processes. -- `cleanup` refuses live state and removes only `.artifacts/manual-acceptance/p11/`. - -## Reviewer workflow - -After `prepare`, open the generated `.artifacts/manual-acceptance/p11/GUIDE.md` and personally: - -1. inspect the catalog, nested descriptor/evidence layout, ownership, and secret-path bindings; -2. serve both surfaces and verify the owned listeners; -3. list `configuration_required` slots; -4. validate and bootstrap-create descriptors exactly once; -5. inspect catalog/descriptor/evidence/docs Git object IDs; -6. retry create/update/delete and verify refusal plus unchanged object IDs; -7. make a curator descriptor+catalog edit, push, pull, and verify the API did not rewrite curator bytes; -8. make an evidence-only commit and inspect the new revision identity; -9. verify the live UI shows read-only existing workspaces and bootstrap-only editing for missing slots; -10. exercise export/import under bootstrap-only rules; -11. render twice, diff the results, and run `tht config check`; -12. run negative catalog/path/secret cases and a bounded secret scan; -13. stop the lab, verify both listeners are gone, write `VERDICT.md`, and only then cleanup if desired. - -## Expected outcomes - -- `prepare` produces a fresh P1.1-only lab and leaves no `VERDICT.md`. -- `serve` exposes only the owned loopback backend and frontend preview. -- positive API operations succeed once; curator-owned follow-up mutations are refused safely; -- curator Git changes become active only after pull; -- renders are deterministic; `tht config check -c <file>` succeeds; -- secret scans find no canaries outside the fixture-secret boundary; -- after `stop`, nothing remains listening on `127.0.0.1:8791` or `127.0.0.1:8792`. - -## Verdict format - -The reviewer creates `VERDICT.md` manually. Include: - -- reviewer identity; -- UTC timestamp; -- result for each checklist step; -- observations and failure evidence; -- exactly one final line: `manual acceptance: PASS` or `manual acceptance: FAIL`. - -Passing `bash scripts/test-p11-manual-acceptance.sh` proves only the tooling/lifecycle guards. It -does not perform or approve manual acceptance. diff --git a/docs/testing/p2-p6-manual-verification.md b/docs/testing/p2-p6-manual-verification.md deleted file mode 100644 index 4b1b956e..00000000 --- a/docs/testing/p2-p6-manual-verification.md +++ /dev/null @@ -1,218 +0,0 @@ -# P2–P6 Manual Verification Walkthrough - -> Living document. Each section is completed with exact released commands and artifacts during its -> corresponding plan. Automated integration and manual acceptance use separate clean state. - -## Global rules - -- Use a new temporary operator root and a new private fixture Git remote for each Px. -- Never use production PSD credentials in a retained report or screenshot. -- Keep descriptor/content in Git; keep endpoints, bindings, credentials, and certificates in the - installation-local protected directory. -- Do not print secret files, rendered signed URLs, Compose environments, or unbounded logs. -- Record the ThothII commit, workspace commit, installation descriptor path, Compose project name, - command exit status, and report path. -- A focused manual PASS does not replace the automated process goal. - -## P2 — Host preprocessing CLI - -**Status:** P2 implementation complete; automated integration PASS; manual acceptance PENDING. - -Manual goal: from a clean local installation, use only `tht` on the host to inspect one -registry workspace and execute the controlled REST-DWH/HTTP-Evidence preprocessing path without a -host Python or Node runtime. Use a fresh operator root and a fresh fixture Git remote; never reuse -the automated `.artifacts/p2-integration/**` state. - -Commands (contract: `docs/contracts/workspace-preprocessing-cli.md`): - -```bash -tht --installation <abs>/thothii-installation.yaml workspace inspect --workspace <id> --json -tht --installation <abs>/thothii-installation.yaml workspace preprocess dwh --workspace <id> --json -tht --installation <abs>/thothii-installation.yaml workspace preprocess dwh --workspace <id> --resume <run-id> --json -tht --installation <abs>/thothii-installation.yaml workspace schema suggest-fks --workspace <id> --from-sql <file>.sql --output <candidates>.yaml --json -tht --installation <abs>/thothii-installation.yaml workspace schema check --workspace <id> --annotations <reviewed>.yaml --reviewed-candidates <sha256:hex> --json -tht --installation <abs>/thothii-installation.yaml workspace index-schema --workspace <id> --json -tht --installation <abs>/thothii-installation.yaml workspace preprocess evidence --workspace <id> --dry-run --json -tht --installation <abs>/thothii-installation.yaml workspace preprocess evidence --workspace <id> --json -tht --installation <abs>/thothii-installation.yaml workspace preprocess run --workspace <id> --json -``` - -Checks: - -1. installation/render preflight (`inspect` returns exact revision + catalog/descriptor digests); -2. DWH introspection+LSH succeeds, rerun is `unchanged`, `--resume <run-id>` is `unchanged`/`succeeded`; -3. `schema suggest-fks` returns pristine JSON with `suggestedFksYaml` and a `manual_review_required` - block (exit 3) when candidates exist; the suggested YAML digest equals the reported digest; -4. `schema check --annotations <reviewed> --reviewed-candidates <digest>` succeeds after review; -5. `index-schema` counts against a pre-provisioned compatible collection and rerun is `unchanged`; -6. HTTP Evidence `--dry-run` returns `dry_run`, the real run publishes, rerun is `unchanged`, an input - mutation produces a new generation/ACTIVE; -7. filesystem Evidence returns a stable `evidence_materialization_required` block with no partial - corpus/vector publication; -8. negatives: missing workspace (`workspace_not_activatable`), resume of a nonexistent run - (`preprocessing_resume_mismatch`), invalid annotations digest (`annotation_invalid`), no-Evidence - skip warning, no collection creation, no backend/Pi/frontend listener; -9. secret scan over retained artifacts and exact owned-resource cleanup. - -Decision: **PENDING** (independent manual gate; automation never records PASS). - -## P3 — Effective configuration and `.tht-dwh` - -**Status:** P3 implementation complete; automated integration PASS; manual acceptance PASS (owner approval 2026-08-13). - -Manual goal: prove that the operator CLI and application sessions derive the same effective -configuration, that a content-only revision reuses the prepared DWH generation (fast, `unchanged`), -that a DWH-affecting change fails closed and regenerates, that the workspace memory migration is -safe, and that search records are revision-scoped. See `docs/contracts/tht-dwh.md`. - -Checks: - -1. run `tht ... workspace preprocess dwh` twice with only an Evidence/content change between - them: the second run reports `unchanged` and does not re-introspect; -2. change a DWH-affecting field (host/port/database/schema/user/collection) in the descriptor, - push, pull: the next run refuses the old generation and regenerates, with a clear - `effective_config_mismatch`-style outcome and no mixed artifacts; -3. inspect `.tht-dwh` generations: immutable directories, `OWNER.json` with the canonical - fingerprints, `ACTIVE` pointer; old generations still present; -4. memory: after the guarded migration the workspace uses - `<dataRoot>/sessions/<workspace-id>/memory/`; the JSONL registry and Qdrant projection are - rebuilt and consistent; a conflicting legacy registry fails closed; -5. search records: schema/Evidence points carry the pinned `workspace_revision`; memory/solved - records remain workspace-wide; -6. documentation: `docs/contracts/tht-dwh.md` matches the observed behavior. - -Decision: **PASS** (owner approval 2026-08-13). -## P4 — Qdrant bootstrap and guarded rebuild - -**Status:** superseded by the "P4 Qdrant collection lifecycle" section below (implemented; manual acceptance PASS). - -Manual goal: prove admission creates a missing compatible collection and indexes, refuses an -incompatible collection, and permits destructive rebuild only under durable maintenance with no -active readers/jobs and exact repeated confirmation. - -Checks to fill during P4: - -1. missing-collection self-heal; -2. missing-index self-heal; -3. dimensions/distance/index-type refusal; -4. confirmation mismatch refusal; -5. active-reader/job refusal; -6. successful drained rebuild; -7. interrupted rebuild recovery with maintenance retained. - -Decision: **PASS** (owner approval 2026-08-13; see the section below). - - -## P4 Qdrant collection lifecycle - -Manual goal: verify admission self-heal and the guarded rebuild through the real product surface. - -Checks to complete during P4 manual acceptance (decision: **PASS** (owner approval 2026-08-13)): - -1. On a fresh installation with no Qdrant collection, a session admission creates the - descriptor collection with exactly 1024 dimensions, cosine distance, and the 8 required - keyword payload indexes (`content_hash`, `document_id`, `kind`, `record_key`, - `record_kind`, `vector_generation`, `workspace_id`, `workspace_revision`). -2. A pre-existing collection with incompatible dimensions/distance (e.g. 768-dim or dot) - is refused with `semantic_index_incompatible` and is never mutated. -3. `tht ... workspace vector inspect --workspace <id> --json` reports the collection - contract without mutation (pristine JSON, exit 0). -4. `tht ... workspace vector rebuild --workspace <id> --collection <name> - --confirm <name> --destroy` deletes and recreates the descriptor-owned collection and - verifies the recreated contract; a mismatched `--confirm` or a missing `--destroy` is - refused (exit 2) without touching the collection. -5. Rebuild writes durable state before deletion, deletes only the descriptor collection, - and the recreated collection preserves the P3 revision-scoped payload contract. - -## P5 — Curated FK annotations in Git - -**Status:** P5 implementation complete; automated integration PASS; manual acceptance PASS (owner approval 2026-08-13). - -Manual goal: curate `<id>/schema/annotations.yaml` in an author clone, publish it, pull the new -revision, and prove the revision-pinned sync and the explicit `schema accept` review, without ever -pushing curated content from the operator CLI. - -Commands (contract: `docs/contracts/workspace-preprocessing-cli.md`): - -```bash -tht --installation <abs>/thothii-installation.yaml workspace schema suggest-fks --workspace <id> --from-sql <file>.sql --output <candidates>.yaml --json -# curate the candidate into <id>/schema/annotations.yaml in the author clone, then commit/push/pull -tht --installation <abs>/thothii-installation.yaml workspace schema accept --workspace <id> --run <run-id> --yes --json -tht --installation <abs>/thothii-installation.yaml workspace preprocess run --workspace <id> --resume <run-id> --json -``` - -Checks: - -1. `schema suggest-fks` returns pristine JSON with `suggestedFksYaml` and a `manual_review_required` - block (exit 3) when candidates exist; the suggested YAML digest equals the reported digest; -2. after commit/push/pull, activation reads `<id>/schema/annotations.yaml` as a regular Git blob at - the same commit as the descriptor and synchronizes it to - `<data>/sessions/<id>/revisions/<commit>/artifacts/mschema/annotations.yaml` with a restrictive - mode and an adjacent ownership manifest `{ workspace, commit, blobId, contentDigest, destination }`; -3. two revisions write two different directories; a session pinned to an older revision reads its own - revision's annotations; -4. `schema accept --run <id> --yes` records the accepted candidate/current-blob digests and the new - revision; missing `--yes`, an unknown run, an empty file, a malformed blob, or a blob not matching - the recorded candidate is refused (exit 1, `annotation_invalid`) without recording a review; -5. `preprocess run --resume <id>` continues only with the exact accepted blob digest and compatible - DWH binding; otherwise it records a new `manual_review_required` checkpoint; -6. negatives: symlink/tree-at-path, cross-namespace, oversized (>16 MiB), non-UTF-8, and malformed - annotation objects are refused at activation without mutating the snapshot or runtime roots; -7. the operator CLI never stages/commits/pushes curated content; secret scan and exact owned-resource - cleanup pass. - -Decision: **PASS** (owner approval 2026-08-13). -## P6 — Commit-addressed Evidence materialization - -**Status:** P6 implementation complete; automated integration PASS; manual acceptance PASS (owner approval 2026-08-13). - -Manual goal: materialize filesystem Evidence from the pinned Git commit, inspect its bounded -manifest, preprocess/index it, retrieve only the pinned revision, and exercise unsafe-tree and -aggregate-limit failures without partial publication. - -Commands (contract: `docs/contracts/workspace-preprocessing-cli.md`): - -```bash -tht --installation <abs>/thothii-installation.yaml workspace inspect --workspace <id> --json -tht --installation <abs>/thothii-installation.yaml workspace preprocess evidence --workspace <id> --dry-run --json -tht --installation <abs>/thothii-installation.yaml workspace preprocess evidence --workspace <id> --json -tht --installation <abs>/thothii-installation.yaml workspace preprocess evidence --workspace <id> --json # idempotent rerun -``` - -Checks: - -1. exact commit/tree/object identities: after activation the materialized root is - `<registry>/snapshots/<commit>/<id>/evidence` and its sibling manifest - `<id>/evidence.manifest.json` records `workspace`, `commit`, `tree`, per-file `oid`/`digest`, - `entryCount`, `totalBytes`; `snapshot.json` chains the manifest digest; -2. successful atomic materialization: every regular blob is present byte-for-byte; the manifest - digests match; -3. manifest and file digest verification: re-activation reuses a valid root and fails closed on a - tampered manifest; -4. filesystem Evidence dry-run/run/idempotency: `--dry-run` returns `dry_run`, the real run - publishes, rerun is `unchanged`; -5. revision-filtered Qdrant retrieval and corpus ACTIVE: Evidence records carry the pinned - `workspace_revision`; -6. nested symlink/gitlink/traversal/special-file refusal: a commit introducing one of these fails - activation (`workspace_invalid`) and the previous valid revision stays active; -7. file-count/total-byte/path/manifest limit refusal: an oversized or over-count tree fails closed - without a partial publication; -8. retention while pinned and owned cleanup after release: the materialized root persists for a - pinned revision and is removed with its snapshot directory once unreferenced. - -Decision: **PASS** (owner approval 2026-08-13). -## Final aggregate P2–P6 verification - -**Status:** runnable; automated integration PASS; manual acceptance PENDING. - -The automated aggregate (run `p2p6-ee542112c526ef0d4c25ddf6c8bc164b`, report -`.artifacts/p2p6-integration/...`) already executed the complete DWH → FK → schema → filesystem -Evidence chain, idempotency, revision isolation, a second installation, unsafe-tree/bound negatives, -secret scan, and exact cleanup. - -The final manual pass will start with a new registry and two independent installations. It will -run the complete DWH → FK → schema → filesystem Evidence chain, prove idempotency and revision -isolation, confirm the second installation uses its own secrets/state, and compare its observations -to the retained aggregate automated report. - -Decision: **PENDING**. diff --git a/docs/testing/psd-server-project-a-manual.md b/docs/testing/psd-server-project-a-manual.md deleted file mode 100644 index 786a1cd9..00000000 --- a/docs/testing/psd-server-project-a-manual.md +++ /dev/null @@ -1,177 +0,0 @@ -# Progetto A PSD — collaudo manuale - -Questo documento guida il collaudo umano del nuovo ThothII sul server con autenticazione locale. -Non sostituisce i controlli automatici del piano. Compilarlo soltanto dopo che Sol ha dichiarato -verdi installazione, workspace, DWH, Qdrant, Ollama e preprocessing. - -## Regole - -- Eseguire i comandi dal terminale locale del server; non usare tunnel SSH. -- Non copiare nel rapporto password, cookie, token, chiavi, hash, stringhe di connessione o righe di - log che li contengano. -- Usare un amministratore locale e un utente ordinario creati appositamente. -- Non effettuare più tentativi di password errata del necessario: il login applica rate limiting. -- Per ogni prova segnare `PASS`, `FAIL` o `PENDING`, con una nota breve e non sensibile. -- Un solo `FAIL` obbligatorio impedisce di avviare il Progetto B. - -## Dati iniziali - -| Campo | Valore redatto | -|---|---| -| Data/ora UTC | | -| SHA ThothII | | -| SHA workspace | | -| Installation descriptor | percorso protetto, senza contenuto | -| Origine di test | loopback oppure hostname privato | -| Endpoint temporaneo usato | sì/no | -| ID domanda di prova approvata | | -| Operatore | | - -## 1. Stato generale - -Prima del gate manuale di avvio, per una descriptor server con runtime projection eseguire solo il -controllo redatto `sudo tht --installation "$INSTALLATION" auth status --json`. Il risultato deve -dire `ready` ed `equal: true`. Se è `blocked`, mancante o diverso dal canonical root, non avviare: -Sol può eseguire `sudo tht --installation "$INSTALLATION" auth publish` e ripetere il controllo, -senza copiare YAML, hash, password, token o environment nel rapporto. Questo documento non -autorizza l'avvio; Project A resta soggetto a un'esplicita autorizzazione separata. - -Eseguire: - -```bash -THT_BIN=<percorso-tht> -INSTALLATION=<percorso-assoluto-thothii-installation.yaml> -"$THT_BIN" --installation "$INSTALLATION" status -"$THT_BIN" --installation "$INSTALLATION" doctor --json -"$THT_BIN" --installation "$INSTALLATION" auth check --json -"$THT_BIN" --installation "$INSTALLATION" pi test -``` - -| Prova | Risultato atteso | Esito | Note | -|---|---|---|---| -| Stato servizi | frontend, core, qdrant ed embedding sani; initializer completato | | | -| Doctor | tutti i controlli obbligatori passano | | | -| Autenticazione | modalità `local`, configurazione pronta | | | -| Pi | provider e modello rispondono | | | -| Secret hygiene | nessun secret nell’output | | | - -## 2. Confine di rete - -Dal terminale controllare i listener e la configurazione renderizzata secondo il piano. - -| Prova | Risultato atteso | Esito | Note | -|---|---|---|---| -| Frontend | pubblicato solo su loopback o tramite endpoint privato approvato | | | -| Core | nessuna porta host pubblica | | | -| Qdrant | nessuna porta host pubblica nel profilo server | | | -| Ollama | nessuna porta host pubblica | | | -| URL produzione | non raggiunge il nuovo stack | | | -| Endpoint privato, se usato | sorgente autorizzata ammessa | | | -| Endpoint privato, se usato | sorgente non autorizzata respinta prima di ThothII | | | - -Se non esiste un endpoint privato, usare il browser headless/API sul server. Non segnare come -eseguite prove browser che non sono state realmente svolte. - -## 3. Autenticazione locale - -Eseguire tramite frontend/browser quando disponibile; altrimenti usare richieste same-origin dal -terminale, conservando cookie e password soltanto in file temporanei mode `0600`, poi eliminandoli. - -| Prova | Azione | Risultato atteso | Esito | Note | -|---|---|---|---|---| -| Accesso anonimo | aprire pagina/API protetta | appare login oppure HTTP 401 | | | -| Password errata | un tentativo con utente valido | errore generico; nessun dettaglio account | | | -| Utente ordinario | login corretto | accesso alle sessioni | | | -| Confine ruoli | aprire Pi Management/amministrazione | negato o non visibile | | | -| Logout | uscire e ricaricare | sessione rifiutata, nuovo login richiesto | | | -| Amministratore | login corretto | funzioni amministrative previste disponibili | | | -| Disabilitazione | Sol disabilita l’utente di prova | login rifiutato genericamente | | | -| Riabilitazione | Sol riabilita l’utente | login nuovamente possibile | | | -| Invalidazione | cambio password/ruolo o `logout-all` | vecchia sessione non più valida | | | -| Remember me | login persistente, riavvio core | sessione ancora valida entro TTL | | | -| CSRF | mutazione senza token corretto | richiesta respinta | | | - -Non disabilitare o demansionare l’ultimo amministratore abilitato. - -## 4. Workspace e DWH - -Eseguire: - -```bash -"$THT_BIN" --installation "$INSTALLATION" \ - workspace inspect --workspace psd-clinical --json -"$THT_BIN" --installation "$INSTALLATION" \ - workspace vector inspect --workspace psd-clinical --json -``` - -| Prova | Risultato atteso | Esito | Note | -|---|---|---|---| -| Revisione Git | coincide con lo SHA approvato | | | -| Trasporto server | `postgres_direct` | | | -| Database/schema | database Supabase rilevato, schema `datawarehouse` | | | -| Utente DWH | read-only dimostrato dai grant | | | -| Workspace Mac | `DEFERRED_PRE_PROJECT_B`; in quel gate deve confermare `rest_api` | | | -| Qdrant | 1024 dimensioni, cosine, indici payload richiesti | | | -| Ollama | `qwen3-embedding:0.6b` | | | -| Evidence | corpus Git attivo alla stessa revisione | | | - -Per l'emendamento del proprietario del 2026-08-21, solo la riga Workspace Mac può restare -`DEFERRED_PRE_PROJECT_B` nella chiusura privata di Project A. Non equivale a PASS e deve essere -eseguita prima di Project B insieme all'osservazione dual-key e alla revoca legacy. - -## 5. Preprocessing e idempotenza - -Esaminare i due risultati consecutivi del preprocessing prodotti da Sol. - -| Prova | Risultato atteso | Esito | Note | -|---|---|---|---| -| Introspezione DWH | completata senza scritture cliniche | | | -| Annotazioni FK | revisione umana registrata e legata al digest corretto | | | -| Schema index | record presenti con workspace revision | | | -| Evidence index | documenti/chunk presenti con workspace revision | | | -| Seconda esecuzione | nessun duplicato; contenuti invariati riconosciuti | | | -| Identità effettiva | invariata tra i due run | | | - -## 6. Sessione completa F1–F8 - -Usare una domanda innocua approvata, senza identificativi reali di pazienti. - -| Fase | Controllo manuale | Esito | Note | -|---|---|---|---| -| F1 | domanda compresa/disambiguata correttamente | | | -| F2 | concetti e contesto coerenti | | | -| F3 | tabelle candidate ragionevoli | | | -| F4 | colonne/join curati e confermati | | | -| F5 | piano CTE comprensibile | | | -| F6 | ogni CTE testata e approvata | | | -| F7 | SQL finale read-only e validato | | | -| F8 | conclusione, memoria e riepilogo coerenti | | | - -Durante una fase intermedia chiudere/riprendere la sessione una volta. Il resume deve tornare -all’ultima fase incompleta senza creare una nuova domanda. - -Verificare infine: - -| Prova | Risultato atteso | Esito | Note | -|---|---|---|---| -| Stato | sessione `finalized` | | | -| SQL | solo lettura; validazione DWH verde | | | -| Artefatti | manifest, question, schema linking, Evidence, CTE, SQL, validation presenti | | | -| Decisioni | gate registrati nel ledger | | | -| Persistenza | artefatti leggibili dopo riavvio | | | -| Chat/SSE | non richiesti come persistenza | | | - -## 7. Decisione - -| Gate | Esito | -|---|---| -| Tutti i controlli obbligatori PASS | | -| Nessun secret raccolto | | -| Rollback vecchio stack ancora disponibile | | -| Progetto B autorizzabile | NO finché il gate Mac/osservazione/revoca non è PASS | - -Decisione finale: `PROJECT_A_PRIVATE_PASS` / `PROJECT_A_FAIL` / `PROJECT_A_PENDING` - -Revisore e data: ______________________________________ - -Motivazione sintetica: ______________________________________ diff --git a/docs/testing/psd-server-project-b-manual.md b/docs/testing/psd-server-project-b-manual.md deleted file mode 100644 index baef77ee..00000000 --- a/docs/testing/psd-server-project-b-manual.md +++ /dev/null @@ -1,131 +0,0 @@ -# Progetto B PSD — collaudo manuale Authentik e Aritmolab - -Questo documento verifica il percorso finale di produzione. Si esegue soltanto dopo il PASS del -Progetto A e dopo che Sol ha completato i preflight Authentik, Supabase, Nginx e bilanciatore. - -## Regole - -- Usare identità di prova approvate: una ordinaria, una amministrativa e, se disponibile, una senza - gruppi ThothII. -- Non acquisire token, cookie, password, chiavi private, claim completi o trace browser contenenti - URL di callback con parametri. -- Partire dalla home reale di Aritmolab, non da un URL interno di ThothII. -- Segnare `PASS`, `FAIL` o `PENDING`; non dedurre il PASS da test automatici. - -## Dati iniziali - -| Campo | Valore redatto | -|---|---| -| Data/ora UTC | | -| SHA ThothII/workspace | | -| Origine pubblica | | -| SHA/revisione Aritmolab | | -| Nome/ID applicazione Authentik | non inserire secret | -| Database Supabase | | -| Schema sessioni | `thoth_sessions` | -| Operatore/revisore | | - -## 1. TLS, routing e pagina iniziale - -| Prova | Azione | Risultato atteso | Esito | Note | -|---|---|---|---|---| -| HTTP | aprire origine in HTTP | redirect a HTTPS | | | -| Certificato | ispezionare il lucchetto/catena | hostname corretto, nessun warning | | | -| Home Aritmolab | aprire URL ufficiale | pagina disponibile | | | -| Sidebar | individuare ThothII | link presente come prima | | | -| Destinazione | aprire il link | nuovo frontend ThothII | | | -| API | caricare l’app | nessun 502/404 o mixed content | | | -| SSE | avviare attività modello | aggiornamenti continui, niente buffering evidente | | | - -## 2. Single sign-on - -Chiudere ogni precedente sessione di test secondo la procedura concordata. Accedere ad Aritmolab -con l’identità ordinaria, quindi aprire ThothII dalla sidebar. - -| Prova | Risultato atteso | Esito | Note | -|---|---|---|---| -| Primo login | Authentik autentica l’utente | | | -| Passaggio sidebar | nessuna seconda richiesta di credenziali | | | -| Callback | ritorno all’origine pubblica ThothII | | | -| Identità | nome visualizzato coerente, senza dati grezzi del token | | | -| Browser storage | nessun access/id token in Local/Session Storage | | | -| Cookie | cookie ThothII HttpOnly/Secure/SameSite secondo configurazione | | | - -Non copiare il valore del cookie nel rapporto. - -## 3. Ruoli e autorizzazione - -| Identità/caso | Risultato atteso | Esito | Note | -|---|---|---|---| -| Gruppo utente | può creare, leggere e gestire le proprie sessioni | | | -| Gruppo utente | Pi Management e funzioni admin negate con 403/non visibili | | | -| Gruppo admin | funzioni amministrative documentate disponibili | | | -| Nessun gruppo mappato | autenticato ma operazioni protette negate | | | -| Gruppo estraneo aggiuntivo | nessun cambiamento e nessun warning | | | -| Header identità forgiato | nessun privilegio aggiuntivo | | | - -Le prove su claim mancante/malformato possono essere eseguite da Sol con un’identità/provider di -test controllato. Il revisore verifica soltanto esito HTTP generico e report redatto, mai il token. - -## 4. Logout e riavvio - -| Prova | Azione | Risultato atteso | Esito | Note | -|---|---|---|---|---| -| Logout ThothII | usare il comando dell’app | cookie ThothII revocato | | | -| SSO ancora attivo | riaprire ThothII | possibile nuovo accesso senza password; documentare | | | -| Logout Authentik globale | se configurato e in scope | comportamento conforme alla policy locale | | | -| Riavvio core | Sol riavvia in finestra controllata | sessione browser valida secondo TTL/policy | | | -| Provider indisponibile | prova controllata | nuovo login fallisce chiuso e redatto | | | -| Ripristino provider | ripetere diagnosi/login | servizio torna operativo | | | - -Non dichiarare “logout globale” se è stato testato soltanto il logout locale di ThothII. - -## 5. Sessioni PostgreSQL e isolamento - -| Prova | Risultato atteso | Esito | Note | -|---|---|---|---| -| Migrazioni | `pending=[]`, `drifted=[]` | | | -| Schema | `thoth_sessions` nel database Supabase esistente | | | -| PostgREST | schema non esposto | | | -| RLS | forzata sulle tabelle previste | | | -| Utente A/B | ciascuno vede soltanto le proprie sessioni | | | -| Accesso incrociato | risposta not-found/negata come da contratto | | | -| Admin | accesso trasversale solo secondo permessi documentati | | | -| Credenziale migratore | non montata nel core | | | -| Schema clinico | nessun nuovo privilegio runtime | | | - -## 6. Sessione completa sotto OIDC - -Come utente ordinario, eseguire una domanda innocua approvata e completare F1–F8. - -| Prova | Risultato atteso | Esito | Note | -|---|---|---|---| -| Creazione | sessione associata all’identità OIDC | | | -| Gate F1–F8 | tutti presentati e registrati correttamente | | | -| Resume | ritorna alla sessione corretta | | | -| SQL finale | sola lettura e validato | | | -| Persistenza | manifest, artefatti e decisioni in PostgreSQL | | | -| SSE/chat | funzionano live; non richiesti come artefatti persistiti | | | -| Riavvio | sessione di lavoro ancora disponibile | | | - -## 7. Integrazione e pulizia finale - -| Prova | Risultato atteso | Esito | Note | -|---|---|---|---| -| Endpoint temporaneo A | rimosso/non instradato | | | -| Vecchio stack | fermo, non esposto | | | -| Link sidebar | punta solo alla nuova release | | | -| Servizi privati | core/Qdrant/Ollama non pubblicati | | | -| Altri servizi Nginx | invariati e sani | | | -| Rollback | procedura verificata e disponibile | | | -| Evidenze | nessun secret o dato clinico identificabile | | | - -## 8. Decisione - -Decisione finale: `PROJECT_B_PASS` / `PROJECT_B_FAIL` / `PROJECT_B_PENDING` - -Revisore e data: ______________________________________ - -Motivazione sintetica: ______________________________________ - -Conferma percorso finale “Aritmolab → sidebar → ThothII → SSO”: ______________________________ diff --git a/docs/workspace-diagnostic-protocol.md b/docs/workspace-diagnostic-protocol.md deleted file mode 100644 index fef6b03e..00000000 --- a/docs/workspace-diagnostic-protocol.md +++ /dev/null @@ -1,151 +0,0 @@ -# Workspace diagnostic protocol - -This is the operator contract for testing a workspace on one ThothII installation. The Git-shared -descriptor declares what can be checked; the installation supplies only the selected DWH -transport and local secret-file bindings. No secret value, certificate content, SSH key, or -response body belongs in the descriptor, generated `.env.example` files, or diagnostic output. - -## Scope and safety rules - -<!-- workspace-descriptor-contract:start --> -Schema v3 is the only accepted workspace descriptor. -Schema v1 and v2 workspace descriptors are rejected before activation. -- Diagnostics do not run for a rejected descriptor. There is no in-product migrator or automatic - conversion; the Git repository must already contain reviewed v3 descriptors. -<!-- workspace-descriptor-contract:end --> -- One workspace owns one Qdrant collection. -- Qdrant and Ollama are internal services. Operators do not bind external vector or embedding - transports for active manuals or supported diagnostics. -- Each diagnostic is bounded by the configured timeout. Redirects are rejected, response bodies - stay inside the adapter, and browser-visible errors are limited to `binding_missing`, - `connector_unavailable`, and `semantic_index_incompatible`. - -## Canonical descriptor contract - -```yaml -workspace: - schema_version: 3 - id: psd-clinical - name: PSD Clinical - language: it - -dwh: - engine: postgres - database: warehouse - schema: datawarehouse - supported_transports: [postgres_direct, rest_api, ssh_tunnel] - -semantic_index: - vector_store: - engine: qdrant - collection: psd-clinical - dimensions: 1024 - distance: cosine - embedding: - provider: ollama_internal - model: qwen3-embedding:0.6b - dimensions: 1024 - -diagnostics: - dwh_rest: - method: POST - path: /rpc/ping - auth: bearer - response: { database: database, schema: schema } -``` - -The semantic-index contract is fixed: - -- `engine: qdrant` -- collection name equals the workspace-owned portable identifier -- `qwen3-embedding:0.6b` -- `1024` dimensions -- cosine distance - -If any active collection reports a different model pairing, dimension, or distance, diagnostics -must return `semantic_index_incompatible` rather than silently rewriting data. - -## Installation-local variable contract - -Replace `<NAMESPACE>` with the immutable workspace ID converted to upper case with hyphens changed -to underscores. For example, `psd-clinical` becomes `PSD_CLINICAL`. Set only the variables for the -selected DWH transport. Every `*_FILE` value is an absolute path to a regular, readable file -inside an approved local secret root; it is never the secret itself. - -| Connector and transport | Required local variables | -| --- | --- | -| DWH selection | `THT_WS_<NAMESPACE>_DWH_TRANSPORT` | -| DWH `postgres_direct` | `THT_WS_<NAMESPACE>_DWH_HOST`, `THT_WS_<NAMESPACE>_DWH_PORT`, `THT_WS_<NAMESPACE>_DWH_USER`, `THT_WS_<NAMESPACE>_DWH_PASSWORD_FILE`; optional `THT_WS_<NAMESPACE>_DWH_TLS_CA_FILE` | -| DWH `rest_api` | `THT_WS_<NAMESPACE>_DWH_BASE_URL`; `THT_WS_<NAMESPACE>_DWH_API_KEY_FILE` only for `bearer`/`x-api-key`; optional `THT_WS_<NAMESPACE>_DWH_TLS_CA_FILE` | -| DWH `ssh_tunnel` | `THT_WS_<NAMESPACE>_DWH_USER`, `THT_WS_<NAMESPACE>_DWH_PASSWORD_FILE`, `THT_WS_<NAMESPACE>_DWH_SSH_HOST`, `THT_WS_<NAMESPACE>_DWH_SSH_PORT`, `THT_WS_<NAMESPACE>_DWH_SSH_USER`, `THT_WS_<NAMESPACE>_DWH_SSH_PRIVATE_KEY_FILE`, `THT_WS_<NAMESPACE>_DWH_SSH_KNOWN_HOSTS_FILE`, `THT_WS_<NAMESPACE>_DWH_SSH_TARGET_HOST`, `THT_WS_<NAMESPACE>_DWH_SSH_TARGET_PORT`; optional `THT_WS_<NAMESPACE>_DWH_TLS_CA_FILE` | - -There are no supported `THT_WS_<NAMESPACE>_VECTOR_*` or -`THT_WS_<NAMESPACE>_EMBEDDING_*` installation bindings in the active operator contract. - -## DWH diagnostic - -For direct PostgreSQL and SSH-tunnelled PostgreSQL, the diagnostic connects with the declared -`dwh.database`, checks TLS and authentication, then executes exactly: - -```sql -SELECT current_database() AS database, current_schema() AS schema -``` - -Both returned values must equal the descriptor's DWH database and schema. - -For REST, the descriptor-declared request is for example: - -```text -POST <THT_WS_<NAMESPACE>_DWH_BASE_URL>/rpc/ping -Authorization: Bearer <content of DWH_API_KEY_FILE> -``` - -It has no request body. A 2xx response must be a JSON object whose declared `database` and -`schema` fields match the descriptor. - -## Internal semantic-service diagnostic - -Schema-v3 workspace diagnostics also verify the internal semantic infrastructure through backend -configuration: - -- Qdrant must be reachable at the installation-owned internal URL. -- The workspace-owned collection must exist or be creatable with `1024` dimensions and cosine - distance. -- Ollama must provide `qwen3-embedding:0.6b`. -- A bounded embed probe must return exactly `1024` dimensions. - -These checks use the private Compose services and never require operator-supplied vector or -embedding URLs, transports, or credentials. - -## SSH host verification and tunnel lifecycle - -For DWH `ssh_tunnel`, the known-hosts file is mandatory and is verified before a connection is -accepted. The tunnel is a short-lived loopback forward for the diagnostic only. The effective -OpenSSH constraints are: - -```text --N -v --o BatchMode=yes --o ExitOnForwardFailure=yes --o StrictHostKeyChecking=yes --o UserKnownHostsFile=<ROLE>_SSH_KNOWN_HOSTS_FILE --i <ROLE>_SSH_PRIVATE_KEY_FILE --p <ROLE>_SSH_PORT --L 127.0.0.1:<ephemeral-port>:<ROLE>_SSH_TARGET_HOST:<ROLE>_SSH_TARGET_PORT -<ROLE>_SSH_USER@<ROLE>_SSH_HOST -``` - -The local listener is `127.0.0.1` only. The process is terminated in cleanup after the direct -probe, on timeout, or on failure. - -In this release, `ssh_tunnel` remains a diagnostic-only DWH transport. A successful probe is -followed by `workspace_not_activatable`, and `POST /sessions` rejects the workspace before -persisting a manifest or starting Pi. This restriction does not apply to SSH transport for the -workspace Git remote. - -## Reader-only fallback - -A workspace may be fully valid in Git but non-activatable locally when a required DWH binding, -secret file, host verification, TLS check, or declared DWH diagnostic fails. That state does not -alter the shared descriptor and does not permit a new session on that installation. It may still -be published and activated elsewhere with valid local bindings. diff --git a/mkdocs.yml b/mkdocs.yml index f6fac48d..3ddd894d 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,9 +1,6 @@ site_name: ThothII Docs -site_description: Documentazione tecnica di ThothII e considerazioni generali sull'ambiente di sviluppo +site_description: Documentazione funzionale, tecnica e operativa di ThothII site_url: https://git.tylconsulting.it/thothii-docs/ -repo_url: https://git.tylconsulting.it/mptyl/ThothII -repo_name: mptyl/ThothII -edit_uri: edit/main/docs/ docs_dir: docs site_dir: site use_directory_urls: true @@ -48,21 +45,12 @@ markdown_extensions: nav: - Home: index.md - Guida utente: guida-utente.md -- Accettazione autenticazione: testing/authentication-manual-acceptance.md -- Setup Policlinico San Donato: install/psd-workspace-setup.md - DWH REST per installazione: - Server dwh-auth: install/dwh-auth-server.md - Enrollment client DWH: install/dwh-auth-client-enrollment.md - TLS DWH REST: install/dwh-auth-tls.md - - Rollout PSD DWH: operations/psd-dwh-auth-rollout.md - - Collaudo manuale DWH: testing/dwh-auth-manual-acceptance.md - - Template evidenza DWH: testing/evidence/psd-dwh-auth-rollout-report-template.md -- Programma deploy server PSD: plans/2026-08-20-psd-server-deployment-program.md -- Collaudo PSD Progetto A: testing/psd-server-project-a-manual.md -- Collaudo PSD Progetto B: testing/psd-server-project-b-manual.md - Contratti e CLI: - CLI workspace preprocessing: contracts/workspace-preprocessing-cli.md - - Contratto tht–Pi: contracts/tht-pi.md - Contratto tht–DWH: contracts/tht-dwh.md - Contratto Evidence workspace v3: contracts/workspace-evidence-v3.md - ThothII (Documentazione Tecnica): @@ -74,10 +62,8 @@ nav: - OIDC generico: install/authentication-oidc.md - Authentik: install/authentik.md - Installazione Docker (4 contesti): installazione-docker-4-contesti.md - - Ristrutturazione Evidence: plans/2026-08-24-evidence-restructuring-design.md - Gestione delle memory: gestione-memory.md - Skill operative: skills.md - - Testo completo skill tht-sessione: skill-tht-sessione.md - Disambiguazione iniziale: disambiguazione-iniziale.md - Considerazioni Generali: - Configurazione dei modelli in Pi: general/pi-configuration.md