mptyl/ThothII
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs: focus public documentation on product usage

Browse Source
This commit is contained in:
marcopan
2026-08-26 10:15:07 +02:00
parent 23bc2f6555
commit a54d4769dd
67 changed files with 290 additions and 9963 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/adr/0001-evidence-publication-boundary.md
-7
View File
@@ -1,7 +0,0 @@
# Use workspace activation as the Evidence publication boundary
Curated Evidence becomes Published Evidence only when it is valid, belongs to the
active workspace revision, and belongs to the atomically published Evidence generation.
Human Git review remains required before activation, but its approval is not duplicated
as mutable state in the Evidence manifest; this keeps Git and the workspace registry as
the existing sources of truth instead of introducing a second approval mechanism.
docs/adr/0002-kind-independent-evidence-identifiers.md
-8
View File
@@ -1,8 +0,0 @@
# Keep Evidence identifiers independent from kind
An Evidence Unit keeps the same identifier when its kind is corrected or its Source
Evidence is unambiguously renamed; kind remains a separate validated field. This avoids
breaking citations and evaluation fixtures for a classification change, while genuine
semantic splits receive new identifiers so independent units never share an identity.
Identifiers use `evidence:<slug>`, are assigned once, persisted in the manifest and are
never recomputed automatically from mutable titles, paths or content hashes.
docs/adr/0003-evaluate-evidence-before-activation.md
-7
View File
@@ -1,7 +0,0 @@
# Evaluate candidate Evidence before activation
Evidence preprocessing builds a candidate vector generation and runs the workspace's
versioned evaluation set against that exact generation before atomically activating it.
The workspace revision may temporarily have no matching active Evidence during this
maintenance window, which is accepted as fail-closed degradation instead of introducing
a distributed transaction across Git, the workspace registry and Qdrant.
docs/adr/0004-formula-evidence-is-an-expression.md
-6
View File
@@ -1,6 +0,0 @@
# Treat Formula Evidence as a composable SQL expression
Formula Evidence contains one validated PostgreSQL expression with declared input
columns, not a complete query or statement. This makes formulas safely composable by
SQL generation; complete documented queries remain Example Evidence, and incompatible
legacy formulas require human review during migration.
docs/adr/0005-evidence-contributes-to-semantic-stages.md
-13
View File
@@ -1,13 +0,0 @@
# Evidence contributes to semantic workflow stages
The Evidence Module is a contributor to the existing semantic stages, not a visible
workflow stage. `clarification`, `rewriting`, `schema_linking`, `cte` and `final_sql`
query it with their corresponding purpose; `memory` remains owned by the Memory Module
and `synthesis` performs no Evidence search. Each mapped stage searches independently
and the workflow records only a minimal receipt containing stage, purpose, vector
generation and returned Evidence IDs.
A successful search may return no matches and does not block the stage. Technical
unavailability is a distinct typed outcome that blocks the calling stage until retry,
without stale-generation or purpose fallback. This favors an explicit temporary stop
over silently treating a broken Evidence dependency as absence of domain knowledge.
docs/adr/0006-grounded-atomic-evidence-preparation.md
-12
View File
@@ -1,12 +0,0 @@
# Ground and atomically apply Evidence preparation
Every proposed Evidence Unit carries one to five short supporting excerpts that can be
found in its normalized Source Evidence. The model may reuse only identifiers supplied
for previous units; deterministic application code assigns all new canonical IDs. This
makes provenance and identity mechanically checkable without pretending that automated
validation can replace human semantic review.
Preparation validates the complete changed batch in a temporary area and applies
curated files plus the manifest atomically. A model timeout or invalid response is not
retried automatically and leaves the worktree unchanged. Explicit `evidence resolve`
actions retire or relink units while preserving an ordinary, recoverable Git diff.
docs/adr/0007-add-bm25-without-rebuilding-qdrant.md
-13
View File
@@ -1,13 +0,0 @@
# Add BM25 without rebuilding the shared Qdrant collection
The workspace keeps its existing unnamed dense vector and adds only the sparse `bm25`
vector with IDF through Qdrant's additive vector-schema operation. Only Evidence points
are repopulated with both default dense and BM25 values. Schema, Memory and solved
questions retain their current dense points and are verified before and after the
upgrade.
This replaces the planned destructive conversion to a named `dense` vector. If BM25 is
missing, Evidence preprocessing may add it and verify the resulting schema; session
runtime remains read-only. An incompatible existing BM25 definition fails without
mutation. A candidate failure leaves the additive schema in place while Evidence stays
unavailable, avoiding data loss in the other workflow modules.
docs/adr/0008-make-hybrid-evidence-retrieval-deterministic.md
-24
View File
@@ -1,24 +0,0 @@
# Make hybrid Evidence retrieval deterministic and diagnosable
Dense and BM25 retrieval receive the same deterministic query text. It preserves the
original question and appends nonempty concepts, tables and columns in a fixed order;
question and context receive Unicode NFC, newline canonicalization and outer trimming.
Context values are then deduplicated exactly and sorted, without lowercasing. Case,
punctuation and internal whitespace remain intact. This avoids accidental ranking
changes caused only by metadata ordering, preserves quoted PostgreSQL identifiers and
keeps the two retrieval branches directly comparable.
Evidence fragmentation follows semantic headings, typed fields and paragraph
boundaries. Formulas, value/meaning pairs, mappings, rules and URLs remain atomic. An
atomic element larger than the existing `max_chunk_chars` limit creates the blocking
`atomic_content_too_large` Review item rather than being split mechanically. The limit
applies to the complete rendered text, defaults to 4,000 characters and is not
duplicated by an Evidence-specific setting.
An L0 contract test starts the exact Qdrant image referenced by `compose.yaml` and
proves Italian server-side `qdrant/bm25` ingestion and search in a temporary collection.
There is no FastEmbed or dense fallback for Evidence when that capability is absent.
The versioned evaluation set contains lexical, semantic and mixed queries. Its report
shows dense-only, BM25-only and fused ranks for expected Evidence. Publication remains
governed only by the simple fused top-10 rule; branch ranks and hit@5 are diagnostic.
docs/agents/domain.md
-76
View File
@@ -1,76 +0,0 @@
# Domain documentation
This repository uses a single-context domain-documentation layout.
## Sources
Before changing behavior or terminology, read:
1. `CONTEXT.md` at the repository root;
2. any relevant architectural decision records under `docs/adr/`;
3. the implementation and tests for the affected module.
`CONTEXT.md` contains the shared domain vocabulary and the system's main
concepts. Use its terminology consistently in code, documentation, issues, and
user-facing explanations.
ADRs explain important architectural decisions and their rationale. They are
created only when a durable decision needs to be recorded; the absence of
`docs/adr/` is not an error.
If one of these optional sources does not exist, continue without reporting an
error.
## Layout
```text
/
├── CONTEXT.md
└── docs/
└── adr/
└── <decision>.md
```
Do not introduce `CONTEXT-MAP.md` unless the repository later becomes a
genuine multi-context system whose domains require separate context documents.
## Working with domain concepts
When implementing or reviewing work:
- identify the domain concepts involved;
- reuse the names defined in `CONTEXT.md`;
- distinguish domain rules from infrastructure details;
- avoid creating synonyms for established terms;
- update `CONTEXT.md` when a new durable concept is introduced or an existing
definition materially changes.
For ThothII, the Evidence module and its concepts belong to this shared domain
context even though Evidence is implemented as an autonomous workflow module.
## Architectural decisions
Create an ADR when a decision:
- affects multiple parts of the system;
- establishes a durable constraint;
- selects between meaningful alternatives;
- would otherwise be difficult to reconstruct later.
Do not create an ADR for routine implementation details.
If current code or a proposed change conflicts with an ADR, flag the conflict
explicitly. Do not silently override the recorded decision.
## Keeping documentation aligned
When a change affects the domain model:
1. update the implementation;
2. update the relevant tests;
3. update `CONTEXT.md`;
4. add or update an ADR when the decision is architectural;
5. update linked plans and GitHub issues.
The persisted repository documentation, not the chat transcript, is the
long-term source of truth.
docs/agents/issue-tracker.md
-165
View File
@@ -1,165 +0,0 @@
# Issue tracker: GitHub
Issues and specifications for this repository live in GitHub Issues under
`mptyl/ThothII`.
Use the GitHub CLI (`gh`) for issue operations. Infer the repository from the
current Git remote when possible.
## Conventions
Create an issue:
```bash
gh issue create --title "<title>" --body-file <file>
```
Read an issue:
```bash
gh issue view <number>
```
List issues:
```bash
gh issue list
```
Add a comment:
```bash
gh issue comment <number> --body-file <file>
```
Apply or remove labels:
```bash
gh issue edit <number> --add-label "<label>"
gh issue edit <number> --remove-label "<label>"
```
Close an issue:
```bash
gh issue close <number>
```
## Pull requests as a triage surface
Pull requests are not used as the primary request or triage surface.
A pull request may implement or resolve an issue, but the issue remains the
canonical location for:
- the request;
- its scope and acceptance criteria;
- triage status;
- dependencies and sub-issues;
- implementation progress;
- the final resolution summary.
## Publishing work
When a workflow or skill says to publish a plan, specification, finding, or
request, create or update a GitHub issue.
Do not leave the only authoritative copy in a chat transcript.
Long implementation documents may also be committed to the repository. In that
case, the corresponding issue should link to the committed document and track
its execution status.
## Fetching work
When a workflow or skill refers to an issue number, retrieve the current issue
and its comments before acting:
```bash
gh issue view <number> --comments
```
Treat the live issue state as authoritative for assignment, labels, closure,
and subsequent decisions.
## Wayfinding operations
A wayfinding map is represented by a parent GitHub issue and, when useful,
smaller child issues.
### Map
Create or update one parent issue describing:
- the intended outcome;
- relevant context;
- known constraints;
- the proposed decomposition;
- dependencies between tasks;
- completion criteria.
Label it according to `docs/agents/triage-labels.md`.
### Child issues
Create a separate issue for each independently actionable unit of work.
Keep the parent issue readable: summarize the decomposition there and link the
child issues instead of copying every implementation detail.
When GitHub sub-issues are available, register the relationship through the
GitHub API. Otherwise, maintain a checklist of linked child issues in the
parent issue.
### Dependencies
Represent blocking relationships with GitHub's native issue-dependency API
when available.
First obtain the database ID of the blocking issue:
```bash
gh api repos/mptyl/ThothII/issues/<blocking-number> --jq '.id'
```
Then register it as a blocker:
```bash
gh api \
--method POST \
repos/mptyl/ThothII/issues/<blocked-number>/dependencies/blocked_by \
-F issue_id=<blocking-issue-database-id>
```
If native dependencies are unavailable, record the relationship explicitly in
both issues.
### Frontier
The frontier is the set of open child issues that:
- have no unresolved blockers;
- are sufficiently specified;
- can be worked on independently;
- are not already being worked on.
Use labels and current issue relationships to identify the frontier.
### Claim
Before starting an issue:
1. confirm that it is still open and unblocked;
2. assign it to the current operator when appropriate;
3. apply the label `ready-for-agent` only if it is genuinely executable;
4. add a short comment stating that work has started.
### Resolve
When the work is complete:
1. verify the issue's acceptance criteria;
2. add a concise resolution comment with relevant files, tests, or decisions;
3. update the parent issue or dependent issues;
4. close the issue;
5. reconsider the frontier, because resolving a blocker may unlock more work.
docs/agents/triage-labels.md
-19
View File
@@ -1,19 +0,0 @@
# Triage labels
These labels represent workflow roles rather than subject areas.
| Label | Meaning |
| --- | --- |
| `needs-triage` | The request has not yet been classified or evaluated. |
| `needs-info` | More information or a human decision is required before work can proceed. |
| `ready-for-agent` | The work is sufficiently specified, unblocked, and suitable for an agent. |
| `ready-for-human` | The work requires human review, approval, or an action only a human can perform. |
| `wontfix` | The request has been deliberately declined or will not be implemented. |
Use only the labels that describe the issue's current workflow state.
Remove obsolete workflow labels when the state changes. For example, remove
`needs-info` when the missing information has been supplied.
Subject-area labels may be added separately, but they must not replace these
workflow roles.
docs/architecture/authentication.md
+2 -2
View File
@@ -84,8 +84,8 @@ Workspace Validate performs static authentication validation without provider co
`tht auth check` performs live, non-interactive diagnosis: static safety plus OIDC discovery,
issuer/JWKS, group-catalog authentication, and exact configured-group existence. Adding
`--interactive` runs that same live diagnosis and then validates a device-flow identity when the
provider supports Device Authorization. Workspace Test is the aggregate live workspace and
authentication validation.
provider supports Device Authorization. Aggregate live workspace and authentication validation is
available through the installation diagnostics.
The ordered `tht doctor` report is exactly: `descriptor`, `files`, `docker`, `compose`,
`configuration`, `authentication`, `services`, `core-http`, `frontend-http`,
docs/architecture/overview.md
+4 -8
View File
@@ -1,9 +1,7 @@
# Panoramica dell'architettura
> Sintesi ad uso documentazione. Per il dettaglio dei moduli e dei flussi vedi
> [Componenti, moduli e flussi](components.md); i contratti correnti sono in `docs/contracts/`
> e le decisioni durevoli in `docs/adr/`. Per lo stato corrente del progetto (gate manuali
> pendenti, layout workspace/secret) vedi `PROJECT_STATE.md` nella radice del repo.
> Per il dettaglio dei moduli e dei flussi vedi
> [Componenti, moduli e flussi](components.md).
ThothII è un **datamart builder human-in-the-loop**: trasforma una domanda in linguaggio naturale in SQL validato (ed eventualmente un datamart dbt) attraverso un **workflow deterministico a 8 fasi NL→SQL**, in cui il modello *propone* e un revisore umano *decide* ai gate.
@@ -80,8 +78,8 @@ rinominare o ricreare la collezione. `workspace preprocess evidence` e la parte
- `tht -c`/`--config` è un'opzione **per-comando**: deve seguire il subcommand, mai precederlo (`ThtRunner.buildArgv` lo impone).
- L'output `--json` deve essere JSON puro su stdout — è un contratto machine-readable.
- Le stringhe UI sono in inglese; il *contenuto* dei documenti resta nella lingua del workspace (italiano per `psd`), perché è il dato reale — solo chrome/label sono in inglese.
- I workspace (`harness/workspaces/*.yaml`) impostano il target DB e i path **assoluti** `paths.sessions/artifacts/indexes` — per `psd` puntano a un repo separato e non versionato (`tht-workspace-psd/`). I segreti vivono solo in `harness/.env` (gitignored).
- Le stringhe UI sono in inglese; il *contenuto* dei documenti resta nella lingua del workspace, perché è il dato reale — solo chrome e label sono in inglese.
- Ogni workspace imposta il target DWH e le proprie directory operative. I segreti restano nei file protetti dell'installazione e non nel repository del workspace.
- Le impostazioni sono globali (`backend/data/settings.json`: workspace/provider/modello/thinking); il form di nuova sessione richiede solo la domanda.
- **Resume**: una sessione riprendibile rientra all'ultima fase incompleta. Il backend rifiuta il resume con 409 se `finalized` o `archived`; `PiProcessManager.spawnFor` deve inviare `/riprendi-sessione <id>` (resume) vs `/nuova-domanda` (nuova) — il prompt sbagliato trasforma silenziosamente un resume in una nuova domanda.
@@ -90,5 +88,3 @@ rinominare o ricreare la collezione. `workspace preprocess evidence` e la parte
Lo stack locale si avvia con `./scripts/run-stack.sh`, dopo aver creato
`deploy/env/local.env` da `deploy/env/local.env.example`. Il core Compose include Pi; DWH,
vector DB, embedding e LLM sono endpoint esterni configurati nel file locale.
Comandi per singolo layer, test e lint: vedi `AGENTS.md` nella radice del repo.
docs/contracts/tht-pi.md
-245
View File
@@ -1,245 +0,0 @@
# `tht pi` lifecycle contract
`tht` is the only component that drives Docker lifecycle operations. The `core` container
does not mount a Docker socket, and Pi is never updated in a running container.
## Inspection and configuration
```text
tht pi status
tht pi doctor
tht pi test
tht pi logs
tht pi configure
```
When `--installation` is omitted, `tht` first uses `THOTHII_INSTALLATION` and otherwise
discovers one valid `thothii-installation.yaml` in the current project tree, including an immediate
`deploy/*` directory. Use `--installation /absolute/path/thothii-installation.yaml` as an explicit
override when the descriptor is outside that tree or more than one installation is available.
`status` executes the image-bundled `pi --version`. `doctor` compares that value with both the
container's `PI_VERSION` contract and the `io.thothii.pi.version` image label; a merely nonempty
version is not sufficient. `doctor` and `test` also require a healthy core, a successful Pi smoke,
valid settings, and an exact selected provider/model pair from the backend's available model
entries. `pi check` remains an alias for `pi test`. Logs are always a bounded, sanitized 200-line
snapshot; there is no follow mode.
On a TTY, `pi configure` presents numbered provider, model, and thinking choices. Providers and
models come from the backend's closed model list, and the model choices are restricted to the
selected provider. In non-interactive use, all choices must be explicit:
```text
tht pi configure \
--provider zai --model glm-5.2 --thinking medium
```
The helper snapshots exact settings-file existence and raw bytes, applies the new values atomically,
and verifies the readback and rendered-configuration digest. A helper, readback, or digest failure
restores those exact bytes when the prior file existed; on a clean installation it removes the new
file and verifies the absent/default state. Empty prior files are supported. The command reports
the actual host path from `PI_AUTH_FILE`; credentials remain in that protected host file and must
never be passed as flags.
Installation-managed Pi provider/model configuration is declarative only. Any JSON value beginning
with `!` is rejected recursively in the complete `models.json` before it can supply management
choices, and the exact selected provider/model and credential payload is checked again before the
isolated smoke files are written. The API returns only the fixed
`Pi provider/model configuration is invalid` message; rejected commands, paths, and secrets are
never included. Use `$NAME`/`${NAME}` environment references in `models.json`, or omit `apiKey` and
provide the selected credential through the protected `PI_AUTH_FILE`, `THT_MODEL_API_KEY_FILE`, or
`THT_SECRETS_FILE` contract. A literal leading exclamation mark uses Pi's `$!` escape. Direct
secret-file references are not a `models.json` feature: ThothII converts its managed key source to
the provider-native child environment, while `PI_AUTH_FILE` is mounted as Pi's protected credential
store.
## Supported Compose entry points and current image
Use `tht start`, `stop`, `status`, `logs`, and `doctor` for ordinary installation lifecycle
operations. All `tht` Compose commands automatically include the installation-specific
durable selector when it exists:
```text
<projectDirectory>/.tht/<installation-id>/current-image.yaml
```
This selector is part of the supported installation state: it keeps a verified Pi image selected
across a fresh `tht` process, stop/start, reconcile, and source checkout whose base image is
digest-pinned. Do not delete or hand-edit it. Direct raw `docker compose` lifecycle commands bypass
this protection and are unsupported. Advanced documented Compose rendering must use
`scripts/compose-with-preflight.sh` and include the same selector with `-f` when present; connector
secret overrides must never bypass that preflight wrapper.
## Reloading Pi configuration
Configuration reload is a separate lifecycle operation from an image update:
```text
tht pi restart --yes [--drain]
```
`--yes` is required after reviewing the planned core recreation. Restart activates the durable
maintenance gate before it checks sessions. Without `--drain`, active open sessions refuse the
command. With `--drain`, the command polls the authenticated session inventory until no active
sessions remain; it never terminates sessions and the wait is bounded.
Restart retains the exact current image and never builds, pulls, or upgrades an image. Before any
core mutation, it tags the captured running image ID with a transaction-scoped reference and
selects that reference through a lifecycle-only Compose override. A configured mutable tag moving
after capture therefore cannot change the restarted image. It recreates only `core` with
`--no-deps --force-recreate --no-build --pull never`; `frontend` and named volumes are not
recreated. Before reopening admission, it verifies health, the unchanged Pi version, the
provider/model/settings smoke, unchanged non-secret rendered configuration, the captured image
identity, and the complete persistence-mount fingerprint.
Restart and update keep separate recovery state:
```text
<projectDirectory>/.tht/<installation-id>/restart-state.json
<projectDirectory>/.tht/<installation-id>/update-state.json
```
The files are mode `0600` and share one installation lifecycle lock, so restart, update, and
rollback cannot race. Every mutating lifecycle command checks both files. Malformed or non-terminal
restart recovery state blocks update and rollback; malformed or incomplete update recovery state
blocks restart. A verified terminal restart state is cleaned up safely before a later mutation.
After core mutation, a restart failure leaves admission gated and preserves both
`restart-state.json` and its exact-image override; the operator must use status/logs and maintenance
recovery rather than deleting recovery material.
## Updating Pi
The normal update uses the repository's pinned version and build source automatically:
```text
tht pi update
tht pi update --version 0.81.0
```
With no `--version`, the command reads the single default `ARG PI_VERSION=<version>` from
`docker/core.Dockerfile` in the selected project. The normal path confirms the explicit update
command, drains active sessions without terminating them, builds the candidate, recreates only
`core`, verifies it, and promotes it transactionally.
Advanced registry updates remain available and require an immutable digest:
```text
tht pi update \
--version 0.81.0 --source build --yes --drain
tht pi update \
--version 0.81.0 --source pull \
--image registry.example.invalid/thothii-core@sha256:<64-lowercase-hex-digits> --yes
```
`--source build` rebuilds only `core` with `PI_VERSION=<version>`. `--source pull` requires an
immutable digest reference; mutable tags, URL forms, and credential-bearing references are
rejected. `--source` is never inferred.
Before inventory, update activates the durable maintenance gate. Activation writes
`/data/settings/maintenance.json` in the mounted settings volume, closes admission, and waits for
all leases. A recreated candidate reads that marker at startup and therefore starts gated. The
loopback-only control endpoints cannot be reached through the frontend proxy and do not depend on
the configured authentication principal mode. Lost activation/deactivation responses are resolved
by querying gate status only when the original result is unknown. An explicit file or directory
durability failure is never converted to success by matching readback: the control API reports
`maintenance_durability_failed`, keeps or restores the safest durable marker state, and requires
recovery.
Open, unarchived sessions stop an update. After an operator has completed or otherwise drained
their work, `--drain` makes the command poll the authenticated bare-array
`GET /sessions?scope=all` response until no active sessions remain.
The configured `core.image` is never retagged or mutated. Each installation transaction creates
unique candidate and previous tags, including when two installations share a configured tag or
the configured image is digest-pinned. A temporary lifecycle-only Compose override selects those
tags for build, recreate, and rollback. Candidate build, pull, or candidate-tag failures happen
before `mutation_started` and therefore never recreate or roll back core. After verification, the
temporary candidate selector is atomically promoted to the durable current-image override.
Rollback atomically promotes the previous selector. Terminal cleanup removes only transaction
files and never deletes the durable selector.
Only `core` is recreated, with `--no-deps --force-recreate`; `frontend` is not recreated and no
volume-replacement flags are used. Verification checks health; exact requested Pi version at all
three declared boundaries (the candidate executable, `PI_VERSION` environment, and
`io.thothii.pi.version` image label); the provider/model/settings smoke; unchanged
non-secret rendered configuration; and the complete persistence-mount fingerprint.
## Recovery, rollback, and maintenance cleanup
Recovery state and lock diagnostics live under:
```text
<projectDirectory>/.tht/<installation-id>/update-state.json
<projectDirectory>/.tht/<installation-id>/restart-state.json
<projectDirectory>/.tht/<installation-id>/*.lock.owner.json
```
Each recovery file is mode `0600`. Update state records transaction-scoped image identities, mount
fingerprints, target version/source, configuration digest, phase, and timestamp; restart state
records the retained image and its verification inputs. Neither file contains credentials, endpoint
values, secret paths, Compose output, or logs. A cross-platform OS advisory file lock serializes
both lifecycle operations; a crashed owner releases the lock automatically. Owner metadata is
diagnostic only and cannot wedge acquisition if empty, partial, or stale.
Any post-candidate failure explicitly confirms or reactivates maintenance and rescans sessions
before compensation. Automatic rollback selects the transaction's previous image through the
lifecycle override and clears maintenance only after the previous image, configuration, mounts,
health, Pi smoke, and terminal recovery write are verified. Ambiguous compensation remains gated.
If the candidate core is stopped and cannot serve the maintenance endpoint, rollback proves that
state with Compose and writes the marker through a one-off previous-image `core` container sharing
the settings volume. It does not require the failed candidate, a host Node runtime, or the Docker
socket inside a container. The restored core is then recreated, verified, and rescanned before the
gate can open.
For a failed update with `update-state.json`, first run:
```text
tht pi rollback --yes
```
Rollback restores the image recorded in update state, but it checks restart state before making any
change. A failed, pending, or malformed restart state rejects rollback. A failed restart retains its
captured image and has no candidate image to roll back; first inspect status and logs, repair the
reported problem, then use maintenance recovery.
Inspect and clean a stale durable gate with:
```text
tht pi maintenance status
tht pi maintenance recover --yes
```
`maintenance recover` restores the captured restart image pin and lifecycle override when needed,
verifies and removes interrupted restart recovery material, and only then processes update state.
It completes an interrupted verified-image promotion, safely finalizes a preparation interrupted
before core mutation, and refuses other pending mutations. For terminal or absent recovery state,
it removes only a stale transaction override, verifies the running installation when the gate is
active, and only then removes the durable marker and reopens admission. It never removes
`current-image.yaml`. If rollback or recovery fails, leave the marker in place, preserve the
relevant recovery state and override, repair the reported Docker/configuration issue, and rerun
rollback or maintenance recovery.
Missing confirmation, invalid arguments, active sessions, and an interrupted transaction exit
`2`. Docker and verification failures exit nonzero with concise, redacted guidance. Direct
read-only/log commands preserve the original Docker child exit code.
## Go dependency security boundary
The supported toolchain is Go `1.26.5`, released 2026-07-07, with module language version
`1.26.0`. The Docker builder is pinned by both patch tag and the multi-platform manifest-list
digest:
```text
golang:1.26.5-bookworm@sha256:1ecb7edf62a0408027bd5729dfd6b1b8766e578e8df93995b225dfd0944eb651
```
That manifest provides both `linux/amd64` and `linux/arm64/v8` builders. Go's official release
history is the authority for the patch level (`https://go.dev/doc/devel/release`); the Docker
Official Image is the authority for the builder (`https://hub.docker.com/_/golang`).
`golang.org/x/sys`, used by the Windows durable-replace implementation, is pinned to `v0.47.0`.
The directly used `github.com/sirupsen/logrus` is pinned to `v1.9.1`, which removes
GO-2025-4188 from the imported package set. The build contract verifies the exact toolchain,
dependencies, digest, and all five supported target builds (Windows amd64, Darwin amd64/arm64, and
Linux amd64/arm64). `go mod verify`, tests including the race detector, `go vet`, and
`govulncheck` are release gates.
docs/contracts/workflow-observable-baseline.md
-155
View File
@@ -1,155 +0,0 @@
# Workflow observable baseline
This contract freezes the externally observable behavior that the conservative modular
refactoring must preserve. It describes what callers and reviewers can observe; it does not
prescribe the internal location of the implementation.
Changing an expectation in this baseline is a behavior change and requires an explicit product
decision. Moving code between Workflow core, Disambiguation, Memory, and Evidence must keep the
baseline green without weakening its assertions.
```mermaid
stateDiagram-v2
[*] --> F1
state "F1 Clarification" as F1
state "F2 Memory" as F2
state "F3 Question rewrite" as F3
state "F4 Evidence" as F4
state "F5 Schema linking" as F5
state "F6 SQL drafting" as F6
state "F7 Validation" as F7
state "F8 Promotion" as F8
F1 --> F2
F2 --> F3
F3 --> F4
F4 --> F5
F5 --> F6
F6 --> F7
F7 --> F8
F7 --> F6: correction
F8 --> [*]
```
## Automated seams
### Pi gate
From `harness/`, run `npm test` with the repository's supported Node 24 runtime.
The gate suite fixes:
- the complete registered Pi tool schemas, including nested types and enum-like constraints;
- the semantic workflow definition and the exact injected session skill bytes;
- widget descriptors and reviewer response semantics;
- F1 clarification and explicitly accepted open ambiguity;
- F2 Memory applied, deselected, and absent;
- F3 rewritten question and assumptions, including mutation failure ordering;
- F4 Evidence used, accepted, rejected, and legacy-without-corpus projections;
- F8 Memory promotion accepted, declined, and absent, including mutation failure ordering;
- a newly folded phase is announced once in RPC mode even when it requires no human gate;
- resume reconstruction for the touched F1, F2, F3, F4, and F8 states;
- artifact payload compatibility, anti-bypass behavior, and final phase closing.
The baseline intentionally checks widget structure and domain content without freezing the
pre-existing Italian chrome emitted by the gate. Repository policy requires UI chrome and labels
to migrate to English in their owning workstream; this contract must not turn that mismatch into a
new compatibility requirement.
`harness/.pi/skills/tht-sessione/SKILL.md` is a committed projection. Its authoritative
Disambiguation and Memory fragments live under `modules/`; from `harness/`, run
`python -m tht.pi_skill_projection --write` to regenerate it or `--check` to detect drift.
Composition uses a static ordered tuple and never directory discovery.
### Harness CLI and persistence
Run the default pytest suite from the harness package. The suite fixes:
- pristine JSON output, human output separation, exit codes, and CLI error behavior;
- decision ledger folding, retraction, reopen ordering, and current-phase reconstruction;
- question, schema-linking, CTE, SQL, validation, and session-document projections;
- Evidence source, corpus, search, citation, and legacy-without-active-corpus behavior;
- Memory search, promotion, solved-question, and vector-write behavior;
- filesystem session persistence and PostgreSQL repository parity.
The default pytest configuration excludes only tests marked `l2`. Tests marked `l0` require a
working local Docker daemon and remain part of the default suite when Docker is available.
### Backend bridge
From `backend/`, the passing automated baseline is:
```sh
npx vitest run test/tht-runner.test.ts test/pi-process-manager.test.ts \
test/session-bridge.test.ts test/sse-hub.test.ts test/sse-route.test.ts \
test/routes-sessions.test.ts test/e2e-f1.test.ts \
test/workspace-preprocessing-service.test.ts \
test/workspaces/evidence/materialization.test.ts \
test/workspaces/evidence/preprocessing.test.ts \
test/workspaces/evidence/boundary.test.ts
npx tsc --noEmit -p .
npm run build
```
These suites fix:
- CLI argument ordering and JSON/error propagation across the runner boundary;
- new-session versus resume Pi prompts;
- refusal to resume finalized, archived, foreign, unavailable, or read-only sessions;
- Pi RPC to client event mapping, SSE replay/reset behavior, and runtime replacement ordering;
- reserved Pi phase notifications mapped to sanitized `phase_started` client events;
- failure persistence and sanitization before a client-visible response.
### Frontend client
From `frontend/`, the passing automated baseline is:
```sh
npx vitest run src/store/sessionStore.test.ts src/stream/useSessionStream.test.tsx \
src/widgets/registry.test.tsx src/widgets/SelectWidget.test.tsx \
src/widgets/MultiselectWidget.test.tsx src/widgets/ArtifactWidget.test.tsx \
src/shell/f1-loop.test.tsx src/shell/SessionDocumentsPanel.test.tsx
npx tsc -b
npm run build
```
These suites fix:
- widget registry and gate response payloads;
- `ui_request`, `text_delta`, activity, usage, and lifecycle event reduction;
- phase progress and the active workflow dot advancing on `phase_started` without a `ui_request`;
- stream replacement, cursor reset, reconnection, and pending-text flush behavior;
- session document projections shown to the reviewer.
## Mutation ordering
The following sequences are part of the observable failure contract:
1. F3 writes the rewritten question, appends `question_rewritten` to the ledger, then advances.
A failure stops the remaining operations.
2. F8 saves one reusable Memory vector, appends its `memory_promoted` marker, advances F8, then
finalizes. A failed vector write leaves no marker; a failed marker after a successful vector
write returns the manual recovery instruction and does not finalize.
3. A declined F8 candidate writes only `memory_promotion_declined`; an absent candidate writes no
Memory decision and still closes F8.
## Environment-dependent acceptance
Real-model and remote-DWH tests remain opt-in through the `l2` marker. The live journey from a new
question to finalization, followed by resume verification, belongs to the final live-acceptance
ticket. If its environment or credentials are unavailable, it must remain recorded as a pending
manual gate rather than being reported as passed.
## Full-suite diagnostic exceptions
Every command defined above as part of the automated baseline exits successfully. Running the
broader backend and frontend suites is still useful as a diagnostic, but those full suites are not
the executable acceptance gate for this ticket because two unrelated failures reproduce unchanged
on the source commit from which this branch was created:
- the backend authentication runtime-projection suite currently rejects ten positive fixtures
with its fail-closed public error;
- one frontend application-shell authentication test does not render the expected trusted-upstream
display name.
These two exceptions must remain visible until their owning workstream resolves them; they must not
be used to relax any workflow assertion or to describe a nonzero command as a passing baseline.
docs/contracts/workspace-evidence-v3.md
-7
View File
@@ -218,10 +218,3 @@ tht config check -c <path>
```
Stop after validation. P2/P6 later owns preprocessing and materialization.
## Acceptance states
These gates are independent and are not implied by this documentation contract.
automated integration: PENDING
manual acceptance: PENDING
docs/disambiguazione-iniziale.md
+2 -6
View File
@@ -54,7 +54,7 @@ Non deve:
- costruire il SQL prima del chiarimento;
- presentare più domande al reviewer nello stesso turno.
La motivazione è di controllo cognitivo e di audit: se vengono chiesti insieme popolazione, periodo, outcome e definizione clinica, non è possibile sapere quale risposta abbia determinato ciascuna scelta successiva.
La motivazione è di controllo cognitivo e di audit: se vengono chiesti insieme linea di prodotto, periodo, indicatore e definizione operativa, non è possibile sapere quale risposta abbia determinato ciascuna scelta successiva.
## Fonti usate per formulare le opzioni
@@ -100,7 +100,7 @@ Esempi:
- “ablazione” significa una procedura transcatetere oppure qualcos'altro;
- “anno” significa anno solare oppure anno fiscale;
- “pazienti attivi” significa flag anagrafico oppure presenza di un evento.
- “biciclette attive” significa modelli a catalogo oppure unità presenti nella produzione corrente.
Ogni opzione concreta contiene una decisione `concept_clarified`. La scelta del reviewer è già la conferma e viene persistita direttamente: non serve un secondo `reviewer_decide`.
@@ -250,8 +250,4 @@ termine ambiguo
## Riferimenti
- [Skill canonica completa](skill-tht-sessione.md)
- [Workflow YAML](../harness/workflow.yaml)
- [Gate Pi](../harness/.pi/extensions/tht-gate.js)
- [Macchina delle fasi](../harness/tht/phase.py)
- [Gestione delle memory](gestione-memory.md)
docs/evidence.md
+8 -15
View File
@@ -27,7 +27,6 @@ Per il filesystem Evidence v2, il primo sorgente autorevole deve stare nella dir
├── curated/ # Evidence Units revisionate
│ └── <dominio>/<unit>.md
├── manifest.yaml # legami, hash e metadati della preparazione
├── evaluation/ # fixture di valutazione del recupero
└── example/ # esempi e materiale di supporto
```
@@ -53,19 +52,19 @@ Le unità Markdown lette dal loader storico della CLI hanno frontmatter YAML. I
```markdown
---
id: evidence:fascia-pediatrica
title: Fascia pediatrica
id: evidence:autonomia-batteria
title: Autonomia nominale della batteria
tier: structural
status: reviewed
sources:
- source/domain/patient.md
- source/domain/bicycle.md
tables:
- patient
- bicycle_model
concepts:
- concept:patient-age
- concept:battery-range
---
Definizione verificata della fascia pediatrica.
Definizione verificata dell'autonomia nominale per modello di bicicletta elettrica.
La regola deve essere abbastanza atomica da poter essere citata senza ricostruire
un intero capitolo. Il testo deve distinguere definizione, condizioni e limiti.
@@ -90,13 +89,11 @@ flowchart TD
ING --> VEC["Embedding e vector store"]
BM25 --> GEN["Generazione candidata"]
VEC --> GEN
GEN --> EVAL["tht evidence evaluate\nfixture di retrieval"]
EVAL -->|pass| ACT["Generazione attiva"]
EVAL -->|fail| FIX
GEN --> ACT["Generazione attiva"]
ACT --> RUNTIME["Ricerca Evidence nel workflow"]
```
La preparazione può ristrutturare sorgenti cambiate, ma non pubblica da sola. `prepare` produce una proposta e può indicare il file sorgente coinvolto in caso di errore. `validate` non scrive né pubblica. Il commit è un'azione del curatore nel clone di authoring. Il runtime legge una revisione completa e validata, poi la pipeline crea una generazione versionata. L'attivazione è atomica: una generazione precedente resta disponibile secondo la policy di retention.
La preparazione può ristrutturare sorgenti cambiate, ma non pubblica da sola. `prepare` produce una proposta e può indicare il documento coinvolto in caso di errore. `validate` non scrive né pubblica. La pubblicazione della revisione è un'azione del curatore. Il runtime legge una revisione completa e validata, poi la pipeline crea una generazione versionata. L'attivazione è atomica: una generazione precedente resta disponibile secondo la policy di retention.
La ricerca runtime usa il recupero ibrido. Il ramo denso usa gli embedding, il ramo BM25 usa la ricerca lessicale e la fusione deterministica ordina i risultati. L'unità pubblicata conserva la provenienza, che il modello deve citare quando usa l'evidence.
@@ -183,7 +180,3 @@ Le formule hanno un formato distinto dalle Evidence documentali. Una formula pro
- [Contratto Workspace Evidence v3](contracts/workspace-evidence-v3.md)
- [Contratto della CLI di preprocessing](contracts/workspace-preprocessing-cli.md)
- [ADR: confine di pubblicazione](adr/0001-evidence-publication-boundary.md)
- [ADR: preparazione atomica e ancorata al sorgente](adr/0006-grounded-atomic-evidence-preparation.md)
- [ADR: valutazione prima dell'attivazione](adr/0003-evaluate-evidence-before-activation.md)
- [ADR: recupero ibrido deterministico](adr/0008-make-hybrid-evidence-retrieval-deterministic.md)
docs/general/pi-configuration.md
+34 -30
View File
@@ -1,6 +1,10 @@
# Configurazione dei modelli in Pi: built-in, utente, progetto
# Configurazione locale dei modelli Pi
Pi (il coding agent che orchestra il workflow NL→SQL) può risolvere un `provider/model` in tre modi diversi. Non sono alternativi: coesistono, e la scelta di quale usare dipende da **quanto è standard l'endpoint** e da **quanto deve essere ampia la visibilità** del modello (tutti i progetti vs. un progetto solo).
Pi, l'agente che orchestra il workflow NL→SQL, risolve i modelli integrati e i provider
OpenAI-compatible dichiarati nel catalogo locale. ThothII applica una regola più stretta di Pi:
un provider o un modello custom non può essere registrato da codice in `harness/.pi/extensions/`.
Endpoint, protocollo, compatibilità e identificatori dei modelli appartengono esclusivamente ai file
locali `deploy/pi/models.json` e `deploy/pi/settings.json`.
> **ThothII operator note:** ThothII runs Pi only in Docker Compose. Paths under
> `~/.pi/agent/` in this document describe Pi's container-side behavior. Operators edit
@@ -17,8 +21,10 @@ In produzione configurare una sola sorgente generica: il bundle `THT_SECRETS_FIL
selezionato e passa al solo child Pi la variabile nativa appropriata
(`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GEMINI_API_KEY`, `ZAI_API_KEY`, ecc.). Il percorso generico,
le chiavi di provider non selezionati e il vecchio `PI_PROVIDER_API_KEY` vengono rimossi dall'ambiente
del child. Provider locali come `ollama`, `lmstudio` e `aritmolab` continuano senza chiave; un provider
hosted non mappato o un secret mancante/non sicuro fallisce prima dello spawn con errore sanitizzato.
del child. Per un provider custom, il backend deriva il nome della variabile dal campo dichiarativo
`apiKey` di `models.json`; un valore letterale indica che il catalogo è autosufficiente. Non esistono
eccezioni per nomi di provider compilate nel codice. Un secret mancante o non sicuro fallisce prima
dello spawn con errore sanitizzato.
La sorgente generica supporta soltanto provider con una singola chiave: `ant-ling`, `anthropic`,
`cerebras`, `deepseek`, `fireworks`, `github-copilot`, `google` (anche tramite alias `gemini`),
@@ -33,7 +39,7 @@ dello spawn (anche durante l'elenco modelli); tutte le credenziali ambientali AW
Cloudflare restano comunque rimosse. Servirà una futura configurazione dedicata per provider per
supportare questi bundle senza ambiguità.
## I tre livelli di provenienza di un modello
## Le fonti di un modello
### 1. Built-in (compilato dentro Pi)
@@ -48,8 +54,8 @@ Per un endpoint **OpenAI-compatible** che non è tra i built-in — ma che non r
Nelle installazioni gestite da ThothII il file deve essere interamente dichiarativo. ThothII
rifiuta ricorsivamente qualsiasi valore JSON che inizi con `!`, anche dentro `headers`, `models`,
`modelOverrides`, `compat`, array o campi non ancora conosciuti. Pi 0.80.3 tratterebbe quel prefisso
come un comando shell al momento della richiesta; questa forma non è ammessa né dall'elenco gestito
dei modelli né dallo smoke isolato. L'errore restituito è fisso e non include comando, percorso o
come un comando shell al momento della richiesta; questa forma non è ammessa dall'elenco gestito
dei modelli. L'errore restituito è fisso e non include comando, percorso o
secret.
Per i secret usare un riferimento ambiente come `"$ZAI_API_KEY"` o `"${ZAI_API_KEY}"`. Il backend
@@ -86,25 +92,27 @@ Esempio reale in uso su questa macchina — GLM (provider `zai`):
Essendo a livello utente, GLM è visibile da **qualsiasi progetto**.
### 3. Estensione (`.pi/extensions/*.js`, utente o progetto)
### Provider da estensione: non ammessi in ThothII
Quando l'endpoint richiede codice — ad esempio conversione di eventi (thinking→text), un `streamSimple` custom, o comunque logica che un file dichiarativo non può esprimere — serve un'estensione Pi che chiama `pi.registerProvider(...)`. Le estensioni possono vivere sia in `~/.pi/agent/extensions/` (tutti i progetti) sia in `<progetto>/.pi/extensions/` (solo quel progetto, se Pi viene lanciato con quella cwd).
Esempio reale — il provider AritmoLab (Qwen), interno all'ospedale, usato solo da ThothII: [harness/.pi/extensions/aritmolab-provider.js](../../harness/.pi/extensions/aritmolab-provider.js).
Pi supporta tecnicamente provider registrati da estensioni JavaScript, ma ThothII non usa questa
possibilità. Le estensioni di progetto sono riservate al workflow e ai gate; non devono contenere
`registerProvider(...)`. Un endpoint che non può essere descritto dal catalogo OpenAI-compatible
non è un provider supportato da questa installazione finché il contratto dichiarativo non viene
esteso in modo generico.
## Tabella riassuntiva (stato attuale di questa macchina)
| Modello | Livello | Perché | Visibilità |
|---|---|---|---|
| `deepseek/deepseek-v4-pro` | Built-in Pi | API pubblica nota, già nella build | Tutti i progetti |
| `zai/glm-5.2` | `~/.pi/agent/models.json` | Endpoint OpenAI-compatible custom (z.ai), nessuna logica speciale | Tutti i progetti |
| `aritmolab/qwen3.6-35b-a3b` | Estensione di progetto | Endpoint interno ospedaliero + conversione eventi thinking→text custom | Solo ThothII (cwd=`harness/`) |
| `zai/glm-5.3` | `deploy/pi/models.json` | Endpoint OpenAI-compatible custom | Installazione ThothII |
| `local-qwen/qwen3.6-35b-a3b` | `deploy/pi/models.json` | Endpoint OpenAI-compatible configurato localmente | Installazione ThothII |
## Come scegliere il livello giusto per un nuovo modello
1. **L'endpoint è un'API pubblica già nota a Pi?** → niente da fare, verifica con `pi --list-models`.
2. **È OpenAI-compatible, nessuna logica custom, e deve essere visibile ovunque?** → `~/.pi/agent/models.json`.
3. **Serve codice custom (auth non standard, conversione eventi, trasporto non-OpenAI) oppure deve restare visibile a un solo progetto?** → estensione, in `~/.pi/agent/extensions/` (globale) o `<progetto>/.pi/extensions/` (locale).
1. **L'endpoint è un'API pubblica già nota a Pi?** → abilita l'identificatore esatto in `deploy/pi/settings.json`.
2. **È OpenAI-compatible ma non built-in?** → dichiaralo in `deploy/pi/models.json`, poi abilitalo in `deploy/pi/settings.json`.
3. **Richiede codice di trasporto specifico del provider?** → non aggiungere un'estensione specifica; il provider non è supportato finché manca una capacità dichiarativa generica.
---
@@ -126,7 +134,6 @@ Oltre ai modelli, Pi carica altre risorse da due alberi paralleli: `~/.pi/agent/
```
harness/.pi/
├── extensions/
│ ├── aritmolab-provider.js ← provider LLM solo-progetto
│ ├── tht-gate.js ← gate human-in-the-loop
│ └── gate/
│ ├── core/ ← enforcement e utility condivise del gate
@@ -139,19 +146,17 @@ harness/.pi/
### Comportamento rispetto alla cwd
La directory da cui lanci `pi` determina quale `.pi/` di progetto viene trovata:
La directory da cui lanci `pi` determina quali estensioni del workflow vengono trovate, ma non
quali modelli ThothII rende disponibili: il catalogo è montato nel Pi agent directory del container.
```bash
# Da harness/ — trova harness/.pi/extensions/aritmolab-provider.js
# Da harness/ — carica il gate di progetto e il catalogo locale montato
cd /path/to/ThothII/harness
pi --model aritmolab/qwen3.6-35b-a3b "..."
# Dalla radice di ThothII — nessun .pi/ trovato lì o nei genitori, solo config utente
cd /path/to/ThothII
pi --model aritmolab/qwen3.6-35b-a3b "..." # ❌ Error: model not found
pi --model local-qwen/qwen3.6-35b-a3b "..."
```
Il backend di ThothII (`PiProcessManager`, `list-models.ts`, `model-matrix.mjs`) lancia sempre `pi` con `cwd: harnessDir`, per questo Qwen è visibile in produzione.
Il backend di ThothII lancia sempre Pi con `cwd: harnessDir` per il workflow; la disponibilità del
modello continua a dipendere soltanto da `models.json`, `settings.json` e dalle credenziali locali.
---
@@ -177,15 +182,15 @@ Se serve un modulo condiviso importato da un'estensione, usa `.mjs` proprio per
```bash
pi --list-models
```
Mostra i built-in + i modelli utente da `models.json`. **Non mostra** i provider registrati da estensione (come AritmoLab) — quelli vanno verificati con la cwd giusta.
Mostra i built-in e i modelli dichiarati in `models.json`.
### Verifica che un'estensione sia caricata
### Verifica il catalogo usato dall'applicazione
```bash
cd harness # o la cwd rilevante per il progetto
pi --mode rpc
# poi: {"type": "get_available_models", "id": "1"}
```
La risposta RPC include tutti i modelli disponibili, inclusi quelli da estensione.
La risposta RPC deve includere soltanto modelli built-in o dichiarati nel catalogo locale.
---
@@ -193,6 +198,5 @@ La risposta RPC include tutti i modelli disponibili, inclusi quelli da estension
| Problema | Causa | Soluzione |
|---|---|---|
| `Model "X/Y" not found` lanciando da fuori progetto | Il modello è registrato da un'estensione locale, non visibile fuori dalla cwd giusta | Lancia `pi` dalla directory di progetto corretta (es. `harness/`) |
| Estensione non caricata pur essendo nella cartella giusta | File `.mjs` invece di `.js`/`.ts` | Rinomina in `.js` |
| `Model "X/Y" not found` | Provider/modello assente da `models.json` oppure identificatore assente da `enabledModels` | Correggi i due file locali e ricarica Pi |
| Impostazioni di progetto non applicate | `settings.json` di progetto ha errori di sintassi, o si sta lanciando `pi` dalla cwd sbagliata | Valida il JSON, controlla la cwd |
docs/gestione-memory.md
+2 -9
View File
@@ -35,11 +35,6 @@ F8: il reviewer decide se promuoverla
L'invariante principale è `REUSABLE_TYPES = {"concept_clarified"}`: le sole memory generabili, salvabili, ricercabili e proponibili sono i concetti chiariti. Le decisioni `table_promoted`, `table_excluded`, `column_promoted` e analoghe restano decisioni locali alla domanda.
Implementazione principale: la façade [harness/tht/memory/](../harness/tht/memory/),
con le policy riusabili in
[harness/tht/memory/core.py](../harness/tht/memory/core.py), e l'adapter
[harness/tht/cli/memory_cmd.py](../harness/tht/cli/memory_cmd.py).
## I tre livelli della gestione
| Livello | Contenuto | Funzione |
@@ -54,8 +49,8 @@ Il ledger contiene la provenienza e le decisioni umane. Il record globale contie
Durante F1 il workflow registra i chiarimenti come decisioni `concept_clarified`. Un chiarimento può esprimere:
- definizioni di concetti clinici o organizzativi;
- criteri di inclusione ed esclusione di una popolazione;
- definizioni di concetti produttivi o organizzativi;
- criteri di inclusione ed esclusione di una linea di prodotto;
- formule e metodi di calcolo;
- interpretazioni temporali;
- mapping verso tabelle e colonne specifiche;
@@ -167,8 +162,6 @@ Il comando:
6. esclude le memory già decise nella sessione corrente;
7. restituisce i risultati ordinati per similarità.
L'implementazione è in [harness/tht/cli/memory_cmd.py](../harness/tht/cli/memory_cmd.py:368).
Le memory non vengono mai applicate automaticamente. Il modello deve presentarle in un'unica scelta `reviewer_decide`:
- una memory selezionata viene registrata come nuovo `concept_clarified` nella sessione corrente;
docs/guida-utente.md
+27 -34
View File
@@ -36,22 +36,22 @@ thoth-workspaces.yaml ← catalogo: elenco dei workspace
```yaml
schema_version: 1
workspaces:
- id: psd-clinical
name: Policlinico San Donato
description: DWH clinico del Policlinico San Donato
- id: acme-ebikes
name: ACME Limited
description: DWH della produzione di biciclette elettriche
```
- L'**id** deve essere minuscolo, senza spazi, es. `psd-clinical` (`[a-z][a-z0-9-]{2,62}`).
- L'**id** deve essere minuscolo, senza spazi, es. `acme-ebikes` (`[a-z][a-z0-9-]{2,62}`).
- Il **descrittore** `<id>/workspace.yaml` è lo schema v3. È l'unica descrizione valida.
### 1.2 Esempio di descrittore (Policlinico San Donato)
### 1.2 Esempio di descrittore (ACME Limited)
```yaml
workspace:
schema_version: 3
id: psd-clinical
name: Policlinico San Donato
description: DWH clinico — aritmologia
id: acme-ebikes
name: ACME Limited
description: DWH industriale — produzione di biciclette elettriche
language: it # le descrizioni/evidence sono in italiano
dwh:
@@ -63,7 +63,7 @@ dwh:
semantic_index:
vector_store:
engine: qdrant
collection: psd-clinical
collection: acme-ebikes
dimensions: 1024
distance: cosine
embedding:
@@ -84,7 +84,7 @@ diagnostics:
evidence:
source:
type: filesystem
uri: psd-clinical/evidence # percorso dentro il repository
uri: acme-ebikes/evidence # percorso dentro il repository
policy:
max_chunk_chars: 4000
retain_published_generations: 3
@@ -186,10 +186,6 @@ selezione del workspace.
configurato/mancante.
- **Forget stored value** elimina dal vault cifrato il singolo secret indicato. Le sessioni o operazioni future
che lo richiedono restano bloccate finché non viene inserito di nuovo.
- **Test workspace connections** materializza temporaneamente i secret necessari, contatta i servizi dati
configurati per quel workspace e rimuove i file temporanei alla fine. Non esporta né pubblica
nulla.
Il repository Git remoto e le relative credenziali sono impostazioni di installazione. I secret
runtime DWH/Evidence sono invece persistenti nel vault cifrato del backend e non nel local storage
della GUI. La GUI è soltanto l'interfaccia: dopo l'invio cancella i valori dai campi e non può
@@ -206,8 +202,8 @@ naturale (workspace, modello e provider sono impostazioni globali già configura
Esempio di domanda:
> «Estrai i pazienti che hanno eseguito un'ablazione nell'ultimo anno, con nome, cognome e data
> dell'intervento.»
> «Elenca le biciclette elettriche completate nell'ultimo anno, con modello, numero di telaio e
> data di completamento.»
### 3.2 Il workflow a 8 fasi e i gate
@@ -235,41 +231,41 @@ verità: ciò che non è registrato non è avvenuto.
---
## Esempio pratico completo — Policlinico San Donato
## Esempio pratico completo — ACME Limited
### Passo 0 — repository
Crea il repository Git del workspace (es. `tht-workspace-psd`):
Crea il repository Git del workspace (es. `tht-workspace-acme`):
```text
thoth-workspaces.yaml # catalogo con psd-clinical
psd-clinical/workspace.yaml # descrittore v3 (vedi §1.2)
psd-clinical/evidence/ # i documenti .md di contesto curati
psd-clinical/schema/annotations.yaml # (quando ci sono join curate)
thoth-workspaces.yaml # catalogo con acme-ebikes
acme-ebikes/workspace.yaml # descrittore v3 (vedi §1.2)
acme-ebikes/evidence/ # i documenti .md di contesto curati
acme-ebikes/schema/annotations.yaml # (quando ci sono join curate)
```
Fai `commit` e `push`. Nell'installazione, l'applicazione fa `Pull` e **attiva** il workspace:
valida lo schema v3, materializza l'Evidence dal commit fissato e prepara la collection Qdrant
Pubblica una nuova revisione Git. Nell'installazione, l'applicazione acquisisce e **attiva** il workspace:
valida lo schema v3, materializza l'Evidence dalla revisione fissata e prepara la collection Qdrant
(1024/cosine + indici).
### Passo 1 — preprocessing
```bash
tht --installation ~/thothii-installation.yaml workspace preprocess dwh --workspace psd-clinical --json
tht --installation ~/thothii-installation.yaml workspace preprocess run --workspace psd-clinical --json
tht --installation ~/thothii-installation.yaml workspace preprocess dwh --workspace acme-ebikes --json
tht --installation ~/thothii-installation.yaml workspace preprocess run --workspace acme-ebikes --json
```
Se il run si ferma per le join (`manual_review_required`):
```bash
# il curatore rivede i candidati e pubblica psd-clinical/schema/annotations.yaml, poi:
tht --installation ~/thothii-installation.yaml workspace schema accept --workspace psd-clinical --run <run-id> --yes --json
tht --installation ~/thothii-installation.yaml workspace preprocess run --workspace psd-clinical --resume <run-id> --json
# il curatore rivede i candidati e pubblica acme-ebikes/schema/annotations.yaml, poi:
tht --installation ~/thothii-installation.yaml workspace schema accept --workspace acme-ebikes --run RUN_ID --yes --json
tht --installation ~/thothii-installation.yaml workspace preprocess run --workspace acme-ebikes --resume RUN_ID --json
```
### Passo 2 — la domanda
Nell'applicazione seleziona il workspace `psd-clinical` e crea una sessione con la domanda. Segui
Nell'applicazione seleziona il workspace `acme-ebikes` e crea una sessione con la domanda. Segui
le fasi e conferma ai gate: il modello proporrà lo schema-linking (tabelle/colonne del DWH
`datawarehouse`), i CTE e infine l'SQL finale, che potrai copiare/visualizzare ed eseguire.
@@ -277,11 +273,8 @@ le fasi e conferma ai gate: il modello proporrà lo schema-linking (tabelle/colo
## Dove trovare i dettagli tecnici
Per l'accesso DWH REST, la chiave è per installazione e vale solo per `rest_api`: il server PSD rimane `postgres_direct` e `ssh_tunnel` non usa questa chiave. Vedere [guida server DWH](install/dwh-auth-server.md), [enrollment client](install/dwh-auth-client-enrollment.md), [TLS](install/dwh-auth-tls.md) e [runbook PSD](operations/psd-dwh-auth-rollout.md).
Per l'accesso DWH REST, la chiave è per installazione e vale solo per `rest_api`; `postgres_direct` e `ssh_tunnel` non usano questa chiave. Vedere [guida server DWH](install/dwh-auth-server.md), [enrollment client](install/dwh-auth-client-enrollment.md) e [TLS](install/dwh-auth-tls.md).
- Contratto CLI: `docs/contracts/workspace-preprocessing-cli.md`
- Contratto `.tht-dwh`: `docs/contracts/tht-dwh.md`
- Evidence v3: `docs/contracts/workspace-evidence-v3.md`
- Installazione locale: `docs/install/local-workspace-registry.md`
- Installazione server: `docs/install/server-workspace-registry.md`
- Verifica manuale P2–P6: `docs/testing/p2-p6-manual-verification.md`
docs/index.md
+5 -5
View File
@@ -1,21 +1,21 @@
# ThothII — Documentazione
Benvenuto nella documentazione di ThothII, il datamart builder human-in-the-loop che trasforma domande in linguaggio naturale in SQL validato attraverso un workflow a 8 fasi orchestrato dal coding agent Pi.
Benvenuto nella documentazione di ThothII, il datamart builder human-in-the-loop che trasforma domande in linguaggio naturale in SQL validato attraverso un workflow a 8 fasi orchestrato da Pi.
La documentazione è divisa in due aree:
## ThothII (Documentazione Tecnica)
Come funziona il sistema: architettura, specifiche di design delle singole funzionalità, piani di implementazione, report di test. Parte da qui: [Panoramica dell'architettura](architecture/overview.md).
Come funziona il sistema: architettura, workflow, contratti operativi, Evidence e gestione delle Memory. Parte da qui: [Panoramica dell'architettura](architecture/overview.md).
Per autenticazione locale, OIDC generico, Authentik e accettazione PSD: [documentazione autenticazione](architecture/authentication.md).
Per autenticazione locale, OIDC generico e Authentik: [documentazione autenticazione](architecture/authentication.md).
Per installare l'applicazione in Docker nei quattro contesti operativi, usando il file env,
`compose.yaml`, l'overlay locale/server e il bundle di secret montato:
[Installazione Docker nei quattro contesti](installazione-docker-4-contesti.md).
Per il DWH REST con una chiave revocabile per installazione: [guida server](install/dwh-auth-server.md), [enrollment client](install/dwh-auth-client-enrollment.md), [TLS](install/dwh-auth-tls.md) e [runbook PSD](operations/psd-dwh-auth-rollout.md). Il componente resta separato dallo stack Compose ThothII.
Per il DWH REST con una chiave revocabile per installazione: [guida server](install/dwh-auth-server.md), [enrollment client](install/dwh-auth-client-enrollment.md) e [TLS](install/dwh-auth-tls.md). Il componente resta separato dallo stack Compose ThothII.
## Considerazioni Generali
Note operative e di configurazione che non sono specifiche del dominio ThothII ma riguardano l'ambiente di sviluppo condiviso con altri progetti — ad esempio come Pi (il coding agent) risolve i modelli a livello built-in, utente e progetto. Parte da qui: [Configurazione dei modelli in Pi](general/pi-configuration.md).
Note operative e di configurazione che non sono specifiche del dominio ThothII — ad esempio come Pi risolve i modelli a livello integrato, utente e progetto. Parte da qui: [Configurazione dei modelli in Pi](general/pi-configuration.md).
docs/install/authentication-local.md
+1 -2
View File
@@ -75,7 +75,6 @@ This section applies only when a Linux `profile: server` descriptor declares a r
The canonical authentication root stays root-owned and is the only authority. The container reads
only the separate read-only runtime projection selected by `CURRENT`; it never falls back to the
canonical files or to a previous generation. Run projected mutations and repairs through the
root-operated `tht` commands documented in the [server guide](server.md), and never edit runtime
files directly.
root-operated `tht` commands, and never edit runtime files directly.
Mac, Windows, and local direct-file authentication remain unchanged when the projection is absent.
docs/install/authentication-oidc.md
+3 -3
View File
@@ -71,7 +71,7 @@ The surfaces have distinct semantics and this order is recommended:
issuer/JWKS, catalog credentials, and all configured mapped groups.
3. `tht auth check --interactive` repeats live diagnosis and additionally validates a device-flow
identity and its direct `groups` claim when Device Authorization is available.
4. Workspace Test performs aggregate live workspace and authentication validation.
4. Installation diagnostics perform aggregate live workspace and authentication validation.
The live CLI forms are:
@@ -87,8 +87,8 @@ real ID token including `groups`. It is an operator check, not a replacement for
`tht doctor` emits this exact ordered report: `descriptor`, `files`, `docker`, `compose`,
`configuration`, `authentication`, `services`, `core-http`, `frontend-http`,
`workspace-registry`, `workflow`, `pi`. Its authentication entry is live and non-interactive.
Any authentication failure makes Workspace Validate or Workspace Test non-activatable according
to that surface's static or live scope.
Any authentication failure prevents activation according to the static or live scope of the
relevant diagnostic surface.
The complete closed diagnostic-code union and exact role-to-permission expansion are in the
[authentication architecture](../architecture/authentication.md).
docs/install/authentik.md
+36 -31
View File
@@ -1,37 +1,42 @@
# Authentik provider setup
# Configurazione del provider Authentik
Authentik is the first certified provider for PSD acceptance. The ThothII browser protocol remains
generic OIDC; these steps configure the provider-specific group catalog only.
Il protocollo browser di ThothII è OIDC generico. Authentik fornisce il catalogo gruppi e il
provider di identità senza introdurre un percorso di login proprietario.
1. Create an OAuth2/OIDC application and provider in Authentik. Register exactly
`<publicUrl>/api/auth/oidc/callback` as the callback and enable `openid`, `profile`, and `email`.
2. Configure the provider so the ID token contains a direct `groups` array of strings. Verify the
claim with a disposable test identity before running acceptance.
3. Create a dedicated API service account for the group catalog. Grant group-view-only privilege;
do not grant write, user-management, or directory-administration privilege. Put its bearer value
in the protected bundle under `THT_AUTHENTIK_API_TOKEN`.
4. Create or confirm the exact groups `TOT Users` and `TOT Admin`. Map them explicitly in
`auth.yaml` to `user` and `admin`, respectively. Keep other upstream groups out of the mapping.
5. Run Workspace Validate for static authentication validation. Then run live non-interactive
diagnosis, followed by the optional device-flow identity check:
```sh
tht auth check
tht auth check --interactive
tht doctor --json
```mermaid
sequenceDiagram
participant Browser
participant ThothII
participant Authentik
Browser->>ThothII: Sign in
ThothII->>Authentik: Authorization Code with PKCE
Authentik-->>Browser: Login and consent
Browser->>ThothII: Callback with code
ThothII->>Authentik: Token exchange
Authentik-->>ThothII: Identity and groups
ThothII-->>Browser: Opaque session
```
6. Run Workspace Test for aggregate live workspace and authentication validation. It must prove
discovery/JWKS, catalog access, and every configured group. The diagnostic result must contain
no secret values. `tht doctor --json` reports `authentication` after `configuration` and before
`services` in its exact ordered checklist.
## Provider OIDC
Only configured exact group names are queried. Additional Authentik or directory groups are ignored
silently, without a warning. A mapped group absent from Authentik fails closed with
`oidc_mapped_group_missing`; an ambiguous exact-name result uses
`oidc_mapped_group_ambiguous`. A group visible only in an upstream directory but not represented
in Authentik is missing from ThothII’s catalog and must not be treated as present.
1. Creare applicazione e provider OAuth2/OIDC.
2. Registrare esattamente `PUBLIC_URL/api/auth/oidc/callback`.
3. Abilitare gli scope `openid`, `profile` ed `email`.
4. Configurare un claim diretto `groups` come array di stringhe.
Rotate the two credentials independently through the protected secret-file procedure, then repeat
`tht auth check` and workspace Test. Never put either value in this guide, YAML, shell history,
diagnostic output, or acceptance evidence.
## Catalogo gruppi
Creare un account di servizio dedicato con sola lettura dei gruppi. Conservare il token nel
bundle protetto come `THT_AUTHENTIK_API_TOKEN`.
Mappare in `auth.yaml` i nomi esatti dei gruppi aziendali ai ruoli ThothII `user` e `admin`.
Gruppi non mappati vengono ignorati; un gruppo configurato ma assente genera un errore chiuso.
## Diagnostica
`tht auth check` controlla discovery, issuer, JWKS, accesso al catalogo e presenza dei gruppi
configurati. L'opzione `--interactive` aggiunge la verifica dell'identità tramite device flow,
quando il provider la supporta.
Ruotare separatamente secret OIDC e token del catalogo gruppi. Nessuno dei due deve comparire in
YAML, cronologia shell, log o output diagnostico.
docs/install/dwh-auth-client-enrollment.md
+27 -56
View File
@@ -1,70 +1,41 @@
# Enrollment client per DWH REST
La credenziale `dwh-auth` appartiene a una installazione ThothII, non a una persona. Serve solo se
il trasporto è `rest_api`; `postgres_direct` e `ssh_tunnel` non la usano.
La credenziale `dwh-auth` appartiene a una installazione ThothII e serve soltanto quando il
workspace usa il trasporto `rest_api`.
| Trasporto | Chiave `dwh-auth` | Materiale locale |
| --- | --- | --- |
| `rest_api` | Sì, una per installazione. | URL HTTPS, `API_KEY_FILE`, eventuale `TLS_CA_FILE`. |
| `postgres_direct` | No. | Credenziali PostgreSQL e TLS PostgreSQL. |
| `ssh_tunnel` | No. | Credenziali PostgreSQL e materiali SSH; è diagnostico-only nel runtime corrente. |
| Trasporto | Materiale richiesto |
| --- | --- |
| `rest_api` | URL HTTPS, `API_KEY_FILE`, eventuale `TLS_CA_FILE` |
| `postgres_direct` | Credenziali PostgreSQL e configurazione TLS PostgreSQL |
| `ssh_tunnel` | Credenziali PostgreSQL e materiale SSH |
Il ThothII server PSD resta `postgres_direct` read-only. Il Mac PSD e le installazioni remote
usano `rest_api`; non introdurre un tunnel SSH per aggirare REST.
## Consegna e conservazione
## Prerequisiti
Ricevere chiave e CA attraverso canali protetti separati. Conservare la chiave nel vault
dell'installazione o in un file regolare accessibile soltanto all'account autorizzato. Non
inserirla in Git, file YAML, argomenti, log o schermate condivise.
Ricevere chiave e CA, se necessaria, attraverso canali protetti separati. Confermare fuori banda il
fingerprint TLS prima dell'uso: [guida TLS](dwh-auth-tls.md). Conservare la chiave nel vault o in
un file protetto, mai Git, `.env` con il valore, argv, ambiente, log o evidenze. Annotare solo ID
pubblico.
## Configurazione ACME Limited
## Percorso GUI: vault dell'installazione
1. In **Workspace management**, eseguire **Update workspace repository** se necessario e
selezionare il workspace.
2. Il trasporto `rest_api` è una precondizione amministrativa del binding locale, non una scelta della GUI. Controllare URL/trust locali e usare **Validate workspace source**.
3. Inserire la chiave nel campo write-only **Data warehouse API key**, poi **Save entered secrets**.
La GUI la conserva nel vault cifrato `workspace-secrets`, non la rileggere né la restituisce.
4. Eseguire **Test workspace connections**. Il controllo innocuo è `/rpc/ping`: atteso 2xx e
database/schema dichiarati.
5. Comunicare al server solo ID pubblico, timestamp e risultato. **Forget stored value** rimuove il valore e va
usato soltanto dopo conferma di sostituzione o revoca.
## Percorso headless: binding reale
`API_KEY_FILE` significa che il valore è nel file, non nella variabile. Questo è l'esempio Mac/local/remoto nel file PSD non tracciato `workspace-bindings.env`; non è il binding del server PSD Project A, che resta `postgres_direct`. I binding REST sono:
Esempio di binding headless per il workspace `acme-ebikes`:
```dotenv
THT_WS_PSD_CLINICAL_DWH_TRANSPORT=rest_api
THT_WS_PSD_CLINICAL_DWH_BASE_URL=https://supabase-aritmolab.policlinicosandonato.it/dwh/
THT_WS_PSD_CLINICAL_DWH_API_KEY_FILE=/run/secrets/psd-clinical-dwh-api-key
THT_WS_PSD_CLINICAL_DWH_TLS_CA_FILE=/run/secrets/psd-clinical-dwh-ca.pem
THT_WS_ACME_EBIKES_DWH_TRANSPORT=rest_api
THT_WS_ACME_EBIKES_DWH_BASE_URL=https://dwh.acme.example/dwh/
THT_WS_ACME_EBIKES_DWH_API_KEY_FILE=/run/secrets/acme-ebikes-dwh-api-key
THT_WS_ACME_EBIKES_DWH_TLS_CA_FILE=/run/secrets/acme-ebikes-dwh-ca.pem
```
Nel file `operator.env` non tracciato, ogni suffisso `_SOURCE` indica solo il percorso assoluto del
file protetto di origine. Il comando genera un override non tracciato che monta quei file nel
`core`; non installa né avvia `dwh-auth` con Compose:
Il suffisso del workspace deriva dall'ID immutabile trasformando i trattini in underscore e
usando lettere maiuscole. `API_KEY_FILE` contiene il percorso del file montato, non il valore
della chiave.
```bash
bash scripts/generate-connector-secrets-override.sh \
--bindings-env /absolute/protected/workspace-bindings.env \
--operator-env /absolute/protected/operator.env \
--output /absolute/protected/connector-secrets.override.yaml \
--service core --role dwh
```
## Rotazione e revoca
La chiave sorgente è un file regolare `0600` per il solo account autorizzato. Per altri workspace,
sostituire `PSD_CLINICAL` con ID immutabile maiuscolo (trattini in underscore). Vedere anche il
[protocollo diagnostico](../workspace-diagnostic-protocol.md).
Durante la rotazione, ricevere la nuova generazione, aggiornare il vault o il file montato e
confermare la connettività sulla route innocua `/rpc/ping`. Solo dopo questa conferma il
responsabile del server revoca la generazione precedente.
## Ping, rotazione e revoca
Usare solo **Test workspace connections** su `/rpc/ping`: successo è 2xx con TLS verificato; il server
conferma l'ID con `key status`. Durante rotazione, ricevere nuova generazione, aggiornare vault o
file `API_KEY_FILE`, ripetere ping, attendere osservazione e far revocare la precedente. Dopo la
revoca: nuova positiva, precedente `401`.
`401` non distingue chiave assente, scaduta o revocata. `503` è un guasto fail-closed di servizio,
socket o registro: non usare connessione diretta e non ridurre TLS. Non riattivare una chiave
revocata. Il percorso PSD è nel [runbook](../operations/psd-dwh-auth-rollout.md).
Un `401` indica una chiave assente, sconosciuta, scaduta o revocata. Un `503` indica che il
servizio di autorizzazione o il registro non sono disponibili. In entrambi i casi non aggirare
REST e non ridurre la verifica TLS.
docs/install/dwh-auth-server.md
+54 -345
View File
@@ -1,356 +1,65 @@
# `dwh-auth`: guida server
`dwh-auth` autentica la route REST `/dwh/` con una chiave per installazione. È un componente Linux
opzionale e server-side: usa `systemd`, non `tht` né Docker Compose, non legge risultati clinici e
non si collega a PostgreSQL. La chiave serve solo a `rest_api`; `postgres_direct` e `ssh_tunnel`
non la usano.
`dwh-auth` protegge la route REST `/dwh/` con una chiave distinta per ogni installazione
ThothII. Il componente gira come servizio Linux separato, non legge i dati del DWH e non si
collega direttamente a PostgreSQL.
## Prerequisiti e confini
- Usare un checkout revisionato, Docker per la build e un operatore autorizzato sul server DWH.
- Una chiave identifica un'installazione, non una persona. L'`installation-id` è unico, non
personale e senza dati clinici.
- Chiavi, digest, file di consegna e backup restano in file protetti: mai Git, argv, variabili
d'ambiente, log, JSON pubblico o evidenze.
- Preparare backup e rollback prima di Nginx. Installare il servizio non autorizza una modifica
della route pubblica.
## Percorsi, owner e mode
| Oggetto | Percorso | Owner e mode |
| --- | --- | --- |
| Binario | `/usr/local/sbin/dwh-auth` | `root:root`, `0755` |
| Unit | `/etc/systemd/system/dwh-auth.service` | `root:root`, `0644` |
| Tmpfiles | `/usr/lib/tmpfiles.d/dwh-auth.conf` | `root:root`, `0644` |
| Registro, `active`, `revoked` | `/var/lib/dwh-auth/` | `root:dwh-auth`, `2750` |
| Lock | `/var/lib/dwh-auth/.writer.lock` | `root:dwh-auth`, `0640` |
| Record | `/var/lib/dwh-auth/{active,revoked}/<public-key-id>.json` | `root:dwh-auth`, `0640` |
| Socket runtime | `/run/dwh-auth/verify.sock` | `dwh-auth:www-data`, `0660` |
| Consegne e backup | `/root/dwh-auth-provision/` | directory `root:root` `0700`, file `0600` |
Il record conserva un digest interno (`secret_sha256`) e metadati, mai la chiave in chiaro. Non
leggere, stampare, calcolare o mettere quel digest in una prova operativa.
## Build, installazione e avvio
Costruire dal commit congelato e registrare solo checksum del binario e SHA sorgente:
```bash
cd /srv/thothii/app
bash scripts/build-dwh-auth.sh --output /tmp/dwh-auth-release
sha256sum /tmp/dwh-auth-release/dwh-auth-linux-amd64
```mermaid
flowchart LR
CLIENT["Installazione ThothII"] -->|"X-API-Key"| NGINX["Nginx"]
NGINX --> AUTH["dwh-auth\nUnix socket"]
AUTH --> REGISTRY["Registro chiavi\nactive e revoked"]
AUTH -->|"authorized"| REST["DWH REST"]
```
Scegliere l'architettura corretta. Il template PSD usa il gruppo Nginx `www-data`; confermarlo
prima dell'installazione su un host diverso.
## Confini di sicurezza
```bash
sudo groupadd --system dwh-auth
sudo useradd --system --no-create-home --shell /usr/sbin/nologin --gid dwh-auth dwh-auth
sudo install -o root -g root -m 0755 /tmp/dwh-auth-release/dwh-auth-linux-amd64 /usr/local/sbin/dwh-auth
sudo install -o root -g root -m 0644 deploy/dwh-auth/dwh-auth.service /etc/systemd/system/dwh-auth.service
sudo install -o root -g root -m 0644 deploy/dwh-auth/dwh-auth.tmpfiles.conf /usr/lib/tmpfiles.d/dwh-auth.conf
sudo install -d -o root -g root -m 0700 /root/dwh-auth-provision
sudo systemd-tmpfiles --create /usr/lib/tmpfiles.d/dwh-auth.conf
sudo /usr/local/sbin/dwh-auth --registry-root /var/lib/dwh-auth check
sudo systemd-analyze verify /etc/systemd/system/dwh-auth.service
sudo systemctl daemon-reload
sudo systemctl enable --now dwh-auth
sudo systemctl status dwh-auth --no-pager
```
- Una chiave identifica un'installazione, non una persona.
- Chiavi e backup restano in file protetti e non entrano in Git, log, argomenti o JSON pubblico.
- Il registro conserva digest e metadati, mai la chiave in chiaro.
- La route REST deve essere esposta esclusivamente tramite TLS verificato.
Controllare i mode con `stat`. Il servizio apre il registro in sola lettura e crea solo il socket.
Non creare JSON, lock o socket a mano: oggetti insicuri devono fallire chiusi.
## Installazione
## Check, elenco e stato
Il servizio usa questi percorsi:
Usare sempre un root assoluto. Questi comandi espongono solo ID pubblici, stato, date e scadenza:
```bash
sudo /usr/local/sbin/dwh-auth --registry-root /var/lib/dwh-auth check
sudo /usr/local/sbin/dwh-auth --registry-root /var/lib/dwh-auth key list --json
key_id=public-key-id
sudo /usr/local/sbin/dwh-auth --registry-root /var/lib/dwh-auth key status --key-id "$key_id" --json
```
Un errore di integrità, permessi, symlink o JSON malformato richiede ripristino da backup protetto,
non una correzione manuale del record.
## Creazione, consegna, scadenza e revoca
Il comando crea la chiave una volta in un nuovo file assoluto `0600`; stdout contiene solo ID
pubblico, installazione e percorso. Il file di output non deve esistere.
```bash
installation_id=psd-mac-primary
description=operatore-mac-primario
key_output=/root/dwh-auth-provision/psd-mac-primary.key
sudo /usr/local/sbin/dwh-auth --registry-root /var/lib/dwh-auth key create \
--installation-id "$installation_id" \
--description "$description" \
--output "$key_output"
```
Aggiungere `--expires-at "YYYY-MM-DDTHH:MM:SSZ"` solo se la policy impone una scadenza; il default è nessuna
scadenza. Consegnare il file solo con vault aziendale, secret manager, MDM o trasferimento
autenticato ristretto. Mai email, chat, ticket, `cat` o copia-incolla. Il client conferma ID
pubblico e ping, poi il materiale temporaneo viene rimosso secondo policy.
L'import legacy è temporaneo PSD: il file sorgente è già `root:root` `0600` e non viene mai letto o
stampato dall'operatore.
```bash
sudo /usr/local/sbin/dwh-auth --registry-root /var/lib/dwh-auth key import \
--legacy-raw --installation-id legacy-shared \
--from-file /root/dwh-auth-provision/legacy-shared.key
```
Per rotare: creare seconda generazione, consegnarla, configurarla e provare `/rpc/ping`; confermare
l'ID pubblico; attendere l'osservazione; poi revocare la precedente e provare nuova=successo,
precedente=401.
```bash
previous_key_id=public-key-id
revocation_reason=shared-credential-rotation
sudo /usr/local/sbin/dwh-auth --registry-root /var/lib/dwh-auth key revoke \
--key-id "$previous_key_id" --reason "$revocation_reason"
```
La revoca non è annullabile e un ID revocato non si ricrea.
## Backup, rollback e disinstallazione
Prima di mutare, creare un archivio root-only `0600` del registro e copie protette delle sole
configurazioni coinvolte. L'archivio contiene digest, quindi è materiale riservato: custodirlo su
storage cifrato approvato; l'evidenza ammessa riporta solo percorso, owner, mode, timestamp e
checksum dell'archivio. Il rollback dual-key ripristina la route e il servizio revisionati, esegue
`nginx -t` e fa reload solo autorizzato; non ripristina chiavi revocate, PostgreSQL, sessioni
legacy, indici Qdrant o cache Ollama.
La disinstallazione richiede autorizzazione esplicita, client REST migrati/revocati e rollback non
più necessario. Solo allora disabilitare l'unità; conservare registro e backup fino alla retention
approvata. Non inserire `dwh-auth` in Compose o in `tht start`/`tht stop`.
## Procedure riproducibili e secret-safe
Eseguire soltanto nel gate autorizzato. Le variabili seguenti contengono percorsi, timestamp e
codici, mai una chiave. Il manifest e l'archivio del registro sono `0600`; l'archivio resta
materiale riservato su storage cifrato approvato.
```bash
run_id=$(date -u +%Y%m%dT%H%M%SZ)
backup_root=/root/dwh-auth-provision
registry_root=/var/lib/dwh-auth
registry_backup="$backup_root/registry-$run_id.tar"
manifest="$backup_root/registry-$run_id.manifest"
sudo install -o root -g root -m 0600 /dev/null "$registry_backup"
sudo install -o root -g root -m 0600 /dev/null "$manifest"
sudo tar --acls --xattrs -C /var/lib -cf "$registry_backup" dwh-auth
sudo sh -c 'sha256sum "$1" > "$2"' sh "$registry_backup" "$manifest"
if sudo sha256sum -c "$manifest" >/dev/null; then printf 'registry_manifest=PASS\n'; else printf 'registry_manifest=FAIL\n' >&2; exit 1; fi
```
Il ripristino non sovrappone mai un tar al registro attivo. Estrarre prima in staging nello stesso
filesystem di `/var/lib`, verificare il candidato, rinominare il registro attuale in una copia
recuperabile e sostituirlo. Non cancellare il pre-ripristino: serve al rollback se `check` o
l'avvio falliscono.
```bash
registry_staging="/var/lib/.dwh-auth-restore-$run_id"
registry_candidate="$registry_staging/dwh-auth"
registry_previous="/var/lib/dwh-auth.pre-restore-$run_id"
if [ -e "$registry_staging" ] || [ -e "$registry_previous" ]; then printf 'registry_restore=FAIL\n' >&2; exit 1; fi
if ! sudo sha256sum -c "$manifest" >/dev/null; then printf 'registry_restore=FAIL\n' >&2; exit 1; fi
if ! sudo install -d -o root -g root -m 0700 "$registry_staging"; then printf 'registry_restore=FAIL\n' >&2; exit 1; fi
if ! sudo tar --acls --xattrs -C "$registry_staging" -xf "$registry_backup"; then printf 'registry_restore=FAIL\n' >&2; exit 1; fi
if ! sudo /usr/local/sbin/dwh-auth --registry-root "$registry_candidate" check; then printf 'registry_restore=FAIL\n' >&2; exit 1; fi
if ! sudo systemctl stop dwh-auth; then printf 'registry_restore=FAIL\n' >&2; exit 1; fi
if ! sudo mv -T -- "$registry_root" "$registry_previous"; then
if sudo systemctl start dwh-auth; then printf 'registry_restore_rollback=PASS\n' >&2; else printf 'registry_restore_rollback=FAIL\n' >&2; fi
exit 1
fi
if ! sudo mv -T -- "$registry_candidate" "$registry_root"; then
if ! sudo mv -T -- "$registry_previous" "$registry_root"; then printf 'registry_restore_rollback=FAIL\n' >&2; exit 1; fi
if sudo /usr/local/sbin/dwh-auth --registry-root "$registry_root" check && sudo systemctl start dwh-auth; then printf 'registry_restore_rollback=PASS\n' >&2; else printf 'registry_restore_rollback=FAIL\n' >&2; fi
exit 1
fi
if sudo /usr/local/sbin/dwh-auth --registry-root "$registry_root" check && sudo systemctl start dwh-auth; then
printf 'registry_restore=PASS\n'
else
sudo systemctl stop dwh-auth || true
if ! sudo mv -T -- "$registry_root" "$registry_staging/failed-dwh-auth"; then printf 'registry_restore_rollback=FAIL\n' >&2; exit 1; fi
if ! sudo mv -T -- "$registry_previous" "$registry_root"; then printf 'registry_restore_rollback=FAIL\n' >&2; exit 1; fi
if sudo /usr/local/sbin/dwh-auth --registry-root "$registry_root" check && sudo systemctl start dwh-auth; then printf 'registry_restore_rollback=PASS\n' >&2; else printf 'registry_restore_rollback=FAIL\n' >&2; fi
exit 1
fi
```
Per le prove, creare file header `0600` che contengono esattamente `X-API-Key: valore`. Il valore
passa dal file chiave al file header senza argv, ambiente o stdout. I file header sono materiale
segreto con la stessa custodia e retention delle chiavi.
```bash
v1_key_file="$key_output"
legacy_key_file=/root/dwh-auth-provision/legacy-shared.key
v1_header_file=/root/dwh-auth-provision/dwh-auth-v1.header
legacy_header_file=/root/dwh-auth-provision/dwh-auth-legacy.header
random_header_file=/root/dwh-auth-provision/dwh-auth-random.header
if ! sudo python3 -c '
import pathlib, sys
if any(b"\n" in pathlib.Path(path).read_bytes() for path in sys.argv[1:]):
raise SystemExit(1)
' "$v1_key_file" "$legacy_key_file"; then
printf 'key_file_bytes=FAIL\n' >&2
exit 1
fi
printf 'key_file_bytes=PASS\n'
for header_file in "$v1_header_file" "$legacy_header_file" "$random_header_file"; do
sudo install -o root -g root -m 0600 /dev/null "$header_file"
done
sudo sh -c '{ printf "%s" "X-API-Key: "; dd if="$1" bs=65536 status=none; printf "\n"; } > "$2"' sh "$v1_key_file" "$v1_header_file"
sudo sh -c '{ printf "%s" "X-API-Key: "; dd if="$1" bs=65536 status=none; printf "\n"; } > "$2"' sh "$legacy_key_file" "$legacy_header_file"
sudo sh -c 'printf "%s\n" "X-API-Key: invalid-test" > "$1"' sh "$random_header_file"
```
Il socket `/verify` deve restituire 204 per v1 e legacy durante il dual-key, 401 per file casuale
e richiesta senza header. Stampare solo PASS/FAIL.
```bash
status=$(sudo curl --header "@$v1_header_file" --unix-socket /run/dwh-auth/verify.sock --output /dev/null --silent --show-error --write-out '%{http_code}' http://localhost/verify)
[ "$status" = 204 ] && printf 'socket_v1=PASS\n' || { printf 'socket_v1=FAIL\n' >&2; exit 1; }
status=$(sudo curl --header "@$legacy_header_file" --unix-socket /run/dwh-auth/verify.sock --output /dev/null --silent --show-error --write-out '%{http_code}' http://localhost/verify)
[ "$status" = 204 ] && printf 'socket_legacy=PASS\n' || { printf 'socket_legacy=FAIL\n' >&2; exit 1; }
status=$(sudo curl --header "@$random_header_file" --unix-socket /run/dwh-auth/verify.sock --output /dev/null --silent --show-error --write-out '%{http_code}' http://localhost/verify)
[ "$status" = 401 ] && printf 'socket_random=PASS\n' || { printf 'socket_random=FAIL\n' >&2; exit 1; }
status=$(sudo curl --unix-socket /run/dwh-auth/verify.sock --output /dev/null --silent --show-error --write-out '%{http_code}' http://localhost/verify)
[ "$status" = 401 ] && printf 'socket_missing=PASS\n' || { printf 'socket_missing=FAIL\n' >&2; exit 1; }
```
Per HTTPS reale usare i file header protetti e la CA approvata contro `/dwh/rpc/ping`: PostgREST
può restituire qualsiasi 2xx, non si pretende 204. Prima della revoca, v1 e legacy devono dare
2xx; il file casuale deve dare 401.
```bash
ping_url=https://supabase-aritmolab.policlinicosandonato.it/dwh/rpc/ping
ca_file=/root/dwh-auth-provision/psd-dwh-ca.pem
status=$(sudo curl --header "@$v1_header_file" --cacert "$ca_file" --connect-timeout 5 --max-time 15 --output /dev/null --silent --show-error --write-out '%{http_code}' "$ping_url")
case "$status" in 2??) printf 'https_v1_pre_revoke=PASS\n' ;; *) printf 'https_v1_pre_revoke=FAIL\n' >&2; exit 1 ;; esac
status=$(sudo curl --header "@$legacy_header_file" --cacert "$ca_file" --connect-timeout 5 --max-time 15 --output /dev/null --silent --show-error --write-out '%{http_code}' "$ping_url")
case "$status" in 2??) printf 'https_legacy_pre_revoke=PASS\n' ;; *) printf 'https_legacy_pre_revoke=FAIL\n' >&2; exit 1 ;; esac
status=$(sudo curl --header "@$random_header_file" --cacert "$ca_file" --connect-timeout 5 --max-time 15 --output /dev/null --silent --show-error --write-out '%{http_code}' "$ping_url")
[ "$status" = 401 ] && printf 'https_random=PASS\n' || { printf 'https_random=FAIL\n' >&2; exit 1; }
```
Dopo l'osservazione, revocare solo la legacy usando il suo ID pubblico già registrato. Dopo la
revoca v1 resta 2xx e legacy diventa 401 anche via HTTPS.
```bash
legacy_key_id=legacy-shared
sudo /usr/local/sbin/dwh-auth --registry-root "$registry_root" key revoke --key-id "$legacy_key_id" --reason shared-credential-rotation
status=$(sudo curl --header "@$v1_header_file" --cacert "$ca_file" --connect-timeout 5 --max-time 15 --output /dev/null --silent --show-error --write-out '%{http_code}' "$ping_url")
case "$status" in 2??) printf 'https_v1_post_revoke=PASS\n' ;; *) printf 'https_v1_post_revoke=FAIL\n' >&2; exit 1 ;; esac
status=$(sudo curl --header "@$legacy_header_file" --cacert "$ca_file" --connect-timeout 5 --max-time 15 --output /dev/null --silent --show-error --write-out '%{http_code}' "$ping_url")
[ "$status" = 401 ] && printf 'https_legacy_post_revoke=PASS\n' || { printf 'https_legacy_post_revoke=FAIL\n' >&2; exit 1; }
```
Per provare 503 in una finestra approvata, registrare l'orario, fermare temporaneamente l'unità,
eseguire il ping con timeout e trap di ripristino; il comando deve stampare solo PASS/FAIL.
```bash
was_active=$(sudo systemctl is-active dwh-auth || true)
[ "$was_active" = active ] || { printf 'https_auth_down=FAIL\n' >&2; exit 1; }
restore_auth() { sudo systemctl start dwh-auth; }
trap restore_auth EXIT INT TERM
sudo systemctl stop dwh-auth
status=$(sudo curl --header "@$random_header_file" --cacert "$ca_file" --connect-timeout 5 --max-time 15 --output /dev/null --silent --show-error --write-out '%{http_code}' "$ping_url" || true)
[ "$status" = 503 ] && printf 'https_auth_down=PASS\n' || { printf 'https_auth_down=FAIL\n' >&2; exit 1; }
sudo systemctl start dwh-auth
trap - EXIT INT TERM
```
Lo scan journal non salva righe grezze: controlla davvero le chiavi v1 e legacy leggendo solo i
percorsi dei file da argv, e conserva anche la difesa generica per prefisso e digest. Il filtro
emette solo PASS/FAIL.
```bash
since=$(date -u -d '15 minutes ago' +%Y-%m-%dT%H:%M:%SZ)
if sudo python3 -c '
import pathlib, subprocess, sys
max_journal_bytes = 1_048_576
max_journal_lines = 10_000
max_chunk_bytes = 65_536
process = None
try:
actual_keys = {pathlib.Path(path).read_bytes() for path in sys.argv[2:]}
needles = (b"thtdwh_v1", b"secret_sha256", *actual_keys)
max_needle_length = max(map(len, needles))
process = subprocess.Popen(
["journalctl", "-u", "dwh-auth", "--since", sys.argv[1], "--no-pager", "--output=cat"],
stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL,
)
except OSError:
raise SystemExit(2)
def stop_child():
if process is not None:
if process.poll() is None:
process.kill()
process.wait()
bytes_seen = 0
line_count = 0
line_open = False
carry = b""
try:
while True:
remaining = max_journal_bytes - bytes_seen
if remaining == 0:
if process.stdout.read1(1):
raise SystemExit(2)
break
chunk = process.stdout.read1(min(max_chunk_bytes, remaining))
if not chunk:
break
bytes_seen += len(chunk)
searchable = carry + chunk
if any(needle in searchable for needle in needles):
raise SystemExit(1)
carry = searchable[-(max_needle_length - 1):]
for byte in chunk:
if byte == 10:
line_count += 1
line_open = False
if line_count > max_journal_lines:
raise SystemExit(2)
else:
line_open = True
if line_open:
line_count += 1
if line_count > max_journal_lines:
raise SystemExit(2)
finally:
stop_child()
if process.returncode != 0:
raise SystemExit(2)
' "$since" "$v1_key_file" "$legacy_key_file"; then
printf 'journal_actual_key_scan=PASS\n'
else
printf 'journal_actual_key_scan=FAIL\n' >&2
exit 1
fi
```
Dopo rollback verificato e migrazione/revoca di ogni client REST, la disinstallazione resta
condizionata all'approvazione: eseguire `sudo systemctl disable --now dwh-auth`, ma mantenere
registro, backup, manifest e file header protetti per la retention; non cancellarli durante il
rollback.
## Troubleshooting
| Sintomo | Interpretazione e azione |
| Oggetto | Percorso |
| --- | --- |
| `401` | Chiave assente, malformata, sconosciuta, scaduta, revocata o errata. Verificare trasporto, ID pubblico e consegna; non cercare dettagli nel messaggio. |
| `503` | Servizio, socket o registro non disponibile/sicuro. Controllare `systemctl`, socket, mode e `check`; ripristinare il backup approvato. |
| `check` fallisce | Integrità del registro non valida. Fermare le scritture, preservare stato e ripristinare; non editare JSON. |
| TLS fallisce | CA o SAN non validi. Seguire [TLS](dwh-auth-tls.md), senza bypass. |
| Binario | `/usr/local/sbin/dwh-auth` |
| Unit systemd | `/etc/systemd/system/dwh-auth.service` |
| Registro | `/var/lib/dwh-auth/` |
| Socket | `/run/dwh-auth/verify.sock` |
| Consegne protette | `/root/dwh-auth-provision/` |
Per il rollout PSD con i due gate separati vedere il [runbook PSD](../operations/psd-dwh-auth-rollout.md).
Installare binario e unit con owner `root`, creare l'utente di servizio `dwh-auth`, quindi
abilitare l'unità con `systemctl enable --now dwh-auth`. Il socket deve essere accessibile al
gruppo usato da Nginx.
## Creazione e revoca delle chiavi
Esempio per l'installazione ACME Limited:
```bash
sudo dwh-auth --registry-root /var/lib/dwh-auth key create \
--installation-id acme-factory-primary \
--description acme-factory-primary \
--output /root/dwh-auth-provision/acme-factory-primary.key
```
Consegnare il file attraverso un vault aziendale o un canale autenticato. Per la rotazione,
creare una nuova chiave, distribuirla, aggiornare il client e revocare la precedente usando il
suo ID pubblico:
```bash
sudo dwh-auth --registry-root /var/lib/dwh-auth key revoke \
--key-id PUBLIC_KEY_ID \
--reason scheduled-rotation
```
La revoca è definitiva. Conservare backup cifrati del registro prima di ogni mutazione.
## Integrazione Nginx
Nginx inoltra la chiave al socket di `dwh-auth`. Solo una risposta autorizzata permette il
passaggio verso il DWH REST; chiavi assenti, sconosciute, scadute o revocate ricevono `401`,
mentre indisponibilità del servizio o del registro producono `503`.
docs/install/dwh-auth-tls.md
+25 -34
View File
@@ -1,49 +1,40 @@
# TLS per DWH REST
La chiave DWH è accettabile solo sopra TLS verificato. Un errore `401` o `503` non autorizza mai a
ridurre la verifica del certificato.
La chiave DWH è accettabile solo sopra TLS verificato. Errori di autorizzazione o disponibilità
non autorizzano mai a disabilitare la verifica del certificato.
## Stato PSD
## CA privata
L'origine REST PSD corrente usa il certificato self-issued/private di Nginx. Il SAN copre
`supabase-aritmolab.policlinicosandonato.it`, l'origine `.it` approvata, e non copre un dominio
`.com`. Non usare `.com` finché non è incluso esplicitamente nel SAN.
Quando il DWH REST usa una CA aziendale, consegnare il certificato separatamente dalla chiave
API. La CA non è una credenziale, ma la sua integrità è un confine di sicurezza: deve restare
fuori da Git e non essere scrivibile da utenti non autorizzati.
Chi non dispone già di trust equivalente approvato riceve la CA separatamente e configura
`TLS_CA_FILE`. La CA non è una credenziale, ma la sua integrità è un confine di sicurezza: fuori da
Git e non scrivibile da utenti non autorizzati.
Esempio ACME Limited:
```dotenv
THT_WS_ACME_EBIKES_DWH_TLS_CA_FILE=/run/secrets/acme-ebikes-dwh-ca.pem
```
## Fingerprint fuori banda
Calcolare localmente il fingerprint del file ricevuto:
Calcolare il fingerprint del file ricevuto e confrontarlo attraverso un canale indipendente:
```bash
openssl x509 -noout -fingerprint -sha256 -in /absolute/protected/psd-dwh-ca.pem
openssl x509 -noout -fingerprint -sha256 \
-in /absolute/protected/acme-ebikes-dwh-ca.pem
```
Confrontarlo con il responsabile autorizzato tramite un canale indipendente dalla consegna (vault
aziendale o canale telefonico verificato). Nell'evidenza registrare solo conferma, approvatore e
timestamp; mai corpo certificato, fingerprint completo o output grezzo.
Il SAN del certificato deve includere il nome esatto usato dal binding, per esempio
`dwh.acme.example`.
## Binding e ping
## Rinnovo
Il binding headless PSD effettivo è:
1. Preparare certificato e chain nuovi.
2. Confermare SAN e fingerprint fuori banda.
3. Distribuire la nuova CA ai client mantenendo temporaneamente la precedente.
4. Aggiornare il binding e confermare la connettività con TLS normale.
5. Installare il certificato server.
6. Ritirare il trust precedente dopo la finestra concordata.
```dotenv
THT_WS_PSD_CLINICAL_DWH_TLS_CA_FILE=/run/secrets/psd-clinical-dwh-ca.pem
```
Il file sorgente locale è collegato da file operatore non tracciato. Usare URL `.it`, poi
**Test workspace connections** su `/rpc/ping`. Non disabilitare TLS e non usare `curl -k`.
## Rinnovo coordinato
1. Preparare certificato e chain nuovi; verificare prima SAN `.it` e assenza di falsa copertura `.com`.
2. Confermare fuori banda il nuovo fingerprint.
3. Consegnare la CA/chain nuova ai client con `TLS_CA_FILE`, senza rimuovere ancora la precedente.
4. Aggiornare vault/binding e verificare ping con TLS normale.
5. Solo con gate Nginx approvato installare il certificato server e ripetere il ping.
6. Ritirare il trust precedente dopo la finestra approvata.
Il rinnovo non modifica chiavi `dwh-auth`, record o ruoli PostgreSQL. TLS e rollback della route
restano approvazioni e backup distinti.
Non usare `curl -k`, non disabilitare TLS e non incorporare certificati o fingerprint completi
nei documenti condivisi.
docs/install/local-workspace-registry.md
-176
View File
@@ -1,176 +0,0 @@
# Local workspace repository installation (macOS, Windows, and Linux)
This manual connects a local ThothII installation to one remote Git repository hosted by a Git
server such as GitHub, GitLab, or Gitea. ThothII is a read-only consumer: it fetches,
validates, and activates workspace revisions, but never edits, commits, pushes, or publishes them.
## Architecture ownership contract
| Component | Ownership | Operator contract |
| --- | --- | --- |
| DWH | External | Configure the external endpoint and complete its runtime credentials in Workspace management. |
| LLM | External | Configure the external endpoint and model policy during installation. |
| Qdrant | Internal | Compose runs the internal service and persists `qdrant-data`. |
| Ollama embedding | Internal | Compose runs the internal `qwen3-embedding:0.6b` service and model-init job. |
## Semantic index ownership contract
| Scope | Ownership rule | Isolation rule |
| --- | --- | --- |
| Workspace semantic index | Each workspace keeps exactly one Qdrant collection reserved for itself. | Schema, Evidence, and memory records share that one collection and are separated by the `kind` payload. |
The mandatory semantic stack is CPU-first. Set `THOTH_ENABLE_EMBEDDING_GPU=1` only after the
documented GPU prerequisites are satisfied. The embedding contract is fixed at
`qwen3-embedding:0.6b`, 1024 dimensions, cosine distance.
## Prerequisites
- A working local installation described by [local.md](local.md).
- A remote Git repository and a read-only deploy credential for this ThothII installation.
- A separate authoring clone in which a workspace curator can edit and publish source revisions.
- `tht` built with `bash scripts/build-tht.sh`.
## Prepare and publish a workspace source
Create a local workspace in an ordinary source directory outside ThothII's data directories. The
canonical repository layout is:
```text
thoth-workspaces.yaml
<workspace-id>/workspace.yaml
<workspace-id>/evidence/ # optional, repository-owned Evidence
<workspace-id>/schema/annotations.yaml # optional curated annotations
```
The catalog lists `{id, name, description?}` and the descriptor at
`<workspace-id>/workspace.yaml` must match that metadata. Use the examples in
`deploy/workspaces/` as authoring references. Do not store passwords, tokens, private keys, or
signed URLs in Git.
Publishing is an author-side Git operation: validate the source, commit it, and push it from the
separate authoring clone to the configured branch. This is the only meaning of “publish” in the
workspace lifecycle. ThothII has no author identity and no Git write credential.
## Use the workspace from the application
After the installation is started, use Workspace management from the authenticated application:
1. Run **Update workspace repository** to fetch and validate the configured Git branch into the
application-owned registry. The operation is all-or-nothing and does not modify the authoring
clone.
2. Confirm that the installation-owned `workspace-secrets` storage remains outside the source
repository and contains no credentials in the workspace descriptors.
3. Select the workspace and run **Validate workspace source** to verify the active descriptor, catalog,
Evidence, annotations, and runtime bindings.
4. Run **Test workspace connections** only with the approved read-only DWH/Evidence test configuration.
Results are redacted and the workspace source remains unchanged.
<!-- workspace-descriptor-contract:start -->
Schema v3 is the only accepted workspace descriptor.
Schema v1 and v2 workspace descriptors are rejected before activation.
<!-- workspace-descriptor-contract:end -->
## Configure the remote Git repository
Copy `docs/install/examples/thothii-installation.local.yaml` to an operator-controlled absolute
path. Its `workspaceRepository` block records the remote, branch, and read-only access method.
Choose exactly one transport override:
- SSH: `deploy/compose.git-ssh.yaml`, with a read-only deploy key and pinned `known_hosts` file.
- HTTPS: `deploy/compose.git-https.yaml`, with a read-only token in a Git credentials file and an
optional private CA file.
The remote and branch are installation configuration. Git credentials remain protected
installation files and are never accepted by Workspace management or returned by its API.
Example non-secret/operator paths:
```dotenv
THT_WORKSPACE_GIT_REMOTE=git@git.example.com:organization/workspaces.git
THT_WORKSPACE_GIT_BRANCH=main
THT_WORKSPACE_INSTALLATION_ID=local
PI_AUTH_FILE=/absolute/path/to/operator/pi-auth.json
THT_SECRETS_FILE=/absolute/path/to/operator/thothii.secrets
THT_WORKSPACE_GIT_SSH_KEY_FILE=/absolute/path/to/operator/git-ssh-key
THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=/absolute/path/to/operator/git-known-hosts
```
Keep these files outside both the ThothII checkout and the workspace source repository. Protect
them with mode `0600` on macOS/Linux or an equivalent single-user ACL on Windows.
## Start and update the installation
Use only the installation-aware lifecycle:
```bash
export THT_SOURCE_ROOT=/absolute/path/to/ThothII
THT_BIN=tht
INSTALLATION=/absolute/path/to/operator/thothii-installation.yaml
"$THT_BIN" --installation "$INSTALLATION" start
"$THT_BIN" --installation "$INSTALLATION" doctor
```
At startup ThothII clones or fetches the configured repository into its application-managed
`workspace-registry` volume. Later, **Update workspace repository** performs a server-side fetch
and fast-forward candidate checkout. It does not copy anything to the user's computer.
## Complete runtime secrets in Workspace management
### Chiavi DWH REST per installazione
Se il binding selezionato è `rest_api`, la chiave DWH è una credenziale per questa installazione e si salva nel vault tramite **Save entered secrets** oppure in un file locale indicato da `API_KEY_FILE`. `postgres_direct` e `ssh_tunnel` non usano questa chiave. Per emissione, TLS, rotazione e verifica `/rpc/ping`, seguire [enrollment DWH REST](dwh-auth-client-enrollment.md).
Open Workspace management after the first successful repository update.
1. At the repository level, review the configured host, repository, branch, and current revision.
2. Select a workspace. Repository update does not require a selection; validation and connection
tests do.
3. Review the runtime fields derived from the selected DWH transport and Evidence authentication
mechanism.
4. Enter or rotate the required values and choose **Save entered secrets**.
5. Run **Validate workspace source** and then **Test workspace connections**.
Secret fields are write-only. The GUI receives only configured/missing status. Values are
encrypted by the backend in the platform-neutral `workspace-secrets` volume. ThothII temporarily
materializes a restrictive file only while an existing file-oriented connector needs it, then
removes that file when the runtime lease ends. **Forget stored value** deletes the selected encrypted value.
The workspace YAML stays environment-independent: it declares connector mechanisms, not host
paths or credentials. Installation trust material such as a Git CA or `known_hosts` remains an
operator concern; DWH and Evidence credentials are completed in the GUI.
## Validation and activation behavior
An update follows this sequence:
1. Fetch the configured branch into a candidate checkout managed by ThothII.
2. Validate the catalog, every descriptor, repository-relative Evidence, and cross-workspace
invariants at the same Git commit.
3. If every workspace is valid, atomically mark that complete commit as active.
4. If any validation fails, report sanitized diagnostics and keep the previous active revision.
The active checkout is read-only application state. Never edit files under
`/data/workspace-registry`. A source correction must be committed and pushed from the authoring
clone, then fetched again with **Update workspace repository**.
## Backup, rotation, and recovery
Back up the `workspace-registry`, `workspace-secrets`, `sessions`, `qdrant-data`,
`embedding-models`, `settings`, and `pi-state` volumes together. The encrypted vault is useless
without its generated master key, so preserve the entire `workspace-secrets` volume and protect
the backup as secret material.
Rotate a runtime credential by saving its replacement in Workspace management and rerunning its
connection test. Rotate Git credentials in the installation files and restart `core`. To recover
from a bad remote revision, correct or revert it in the authoring repository and run the update;
until validation succeeds, the previous active snapshot remains available.
## Troubleshooting
| Symptom | Meaning and action |
| --- | --- |
| Repository unavailable | Check remote host, branch, read-only deploy credential, CA, and `known_hosts`. |
| Candidate rejected | Fix the reported source error in the authoring clone, commit, push, and update again. |
| Runtime configuration required | Select the workspace and complete each required secret field. |
| Connection test fails | Rotate the relevant secret or correct the non-secret endpoint in the source/installation as appropriate. |
| Active revision did not change | The candidate was invalid or was already active; inspect the repository status. |
docs/install/local.md
-499
View File
@@ -1,499 +0,0 @@
# Install ThothII on a local PC or Mac
This guide installs one loopback-only ThothII on the same Windows, macOS, or Linux computer that
runs Docker. The supported application is one Docker Compose distribution containing exactly
`frontend` and `core`; Pi is pinned inside `core`. DWH, vector database, embedding, and LLM remain
external configurable services even when they run on this computer.
No host Pi, Node.js, Python, Go toolchain, Docker socket in core, or browser shell is required.
Commands that contain example paths must be changed to absolute paths on your computer.
## Choose your platform
- **macOS:** use Terminal and Docker Desktop. Apple Silicon and Intel are supported by the local
image build.
- **Windows PowerShell:** use Docker Desktop with its WSL2 engine, Git for Windows, the Windows
build launcher, and `tht-windows-amd64.exe`.
- **Windows WSL2 (recommended):** enable Docker Desktop integration for your Linux distribution,
clone under `/home/<user>` rather than `/mnt/c`, and follow the Linux shell commands.
- **Linux PC:** use Docker Engine plus the Compose v2 plugin and the Linux `tht` binary.
Windows users must also read [Windows and WSL2 line endings](windows-line-endings.md) before the
first build.
## Prerequisites
Install only:
1. Git 2.39 or newer.
2. Docker Desktop on macOS/Windows, or Docker Engine on Linux.
3. Docker Compose v2 (`docker compose`, not legacy `docker-compose`).
4. About 10 GB of free disk for source, images, build cache, and initial volumes.
5. Network access to the workspace Git remote and configured DWH/vector/embedding/LLM endpoints.
Verify the tools:
```sh
git --version
docker version
docker compose version
docker run --rm hello-world
```
On Linux, add the operator to the Docker group only if local policy permits it; sign out and back
in afterward. A local installation needs no inbound firewall rule because ports bind only to
`127.0.0.1`.
## Clone and verify LF
Use a `git clone` command that disables automatic CRLF conversion for this checkout.
macOS and Linux:
```sh
git -c core.autocrlf=false clone https://github.example.invalid/your-org/ThothII.git
cd ThothII
git config --local core.autocrlf false
bash scripts/verify-line-endings.sh
```
Windows PowerShell:
```powershell
git -c core.autocrlf=false clone https://github.example.invalid/your-org/ThothII.git
Set-Location ThothII
git config --local core.autocrlf false
& "C:\Program Files\Git\bin\bash.exe" scripts/verify-line-endings.sh
```
Windows WSL2:
```sh
mkdir -p "$HOME/src" && cd "$HOME/src"
git -c core.autocrlf=false clone https://github.example.invalid/your-org/ThothII.git
cd ThothII
git config --local core.autocrlf false
bash scripts/verify-line-endings.sh
```
Stop if the verifier names any path. Do not build from a CRLF checkout.
## Create the local operator files
Copy the non-secret template. This untracked `.env` contains addresses and absolute source paths,
never secret values:
```sh
cp deploy/env/local.env.example deploy/env/local.env
mkdir -p /absolute/path/to/thothii-operator/secrets
chmod 0700 /absolute/path/to/thothii-operator/secrets
```
Native Windows PowerShell performs the same setup without POSIX utilities. The ACL commands remove
inherited access from the new operator directory and grant full control only to the current Windows
identity. Stop if either `icacls.exe` command returns a nonzero exit code:
```powershell
$OperatorDir = Join-Path $env:USERPROFILE 'thothii-operator'
$SecretsDir = Join-Path $OperatorDir 'secrets'
$CurrentUser = [System.Security.Principal.WindowsIdentity]::GetCurrent().Name
if (Test-Path $OperatorDir) { throw 'Use a new operator directory or review its ACLs manually.' }
New-Item -ItemType Directory -Force -Path $OperatorDir, $SecretsDir | Out-Null
icacls.exe $OperatorDir /inheritance:r
if ($LASTEXITCODE -ne 0) { throw 'Could not remove inherited operator-directory ACLs.' }
icacls.exe $OperatorDir /grant:r "${CurrentUser}:(OI)(CI)F"
if ($LASTEXITCODE -ne 0) { throw 'Could not grant the current user the operator-directory ACL.' }
Copy-Item deploy/env/local.env.example deploy/env/local.env
Copy-Item docs/install/examples/thothii-installation.local.yaml `
(Join-Path $OperatorDir 'thothii-installation.yaml')
```
Edit `deploy/env/local.env`. At minimum set the workspace Git remote, `PI_AUTH_FILE`,
`THT_SECRETS_FILE`, and external service endpoints. Create the Pi/application and Git transport
files under the protected operator directory and set mode `0600`. On Windows use a user-only ACL
instead. DWH and Evidence credentials are entered later through Workspace management and stored
in the backend's encrypted `workspace-secrets` volume.
Do not paste credentials into this guide's commands, `.env`, workspace YAML, Git, URLs, image build
arguments, or the installation descriptor. Secret contents are mounted read-only under
`/run/secrets` (Pi's auth store has its own protected read-only mount) and must never be committed,
embedded, rendered, or logged.
Follow [the local workspace repository guide](local-workspace-registry.md) to choose exactly one
read-only Git SSH/HTTPS override. A fresh install requires a valid private workspace repository;
the remote Git repository remains the source of truth.
Copy the installation example to an operator-controlled file named exactly
`thothii-installation.yaml`, then replace all placeholders with absolute paths:
```sh
cp docs/install/examples/thothii-installation.local.yaml \
/absolute/path/to/thothii-operator/thothii-installation.yaml
```
For HTTPS, replace the SSH override in that file with `deploy/compose.git-https.yaml`. Add only
reviewed local overrides. Paths may contain spaces when correctly represented as YAML strings.
Native Windows uses the same four fields. Use single-quoted absolute Windows paths so backslashes
remain literal YAML characters:
```yaml
profile: local
projectDirectory: 'C:\Users\operator\src\ThothII'
envFile: 'C:\Users\operator\src\ThothII\deploy\env\local.env'
overrides:
- 'C:\Users\operator\src\ThothII\deploy\compose.git-ssh.yaml'
```
## Address external services
An address is interpreted inside `core`. Therefore container 127.0.0.1 means the container itself,
not the Docker host. Keep external DWH and LLM addresses configurable in the installation; Qdrant
and embedding are internal services in the standard stack.
- **Docker Desktop (macOS and Windows):** use `host.docker.internal`, for example
`http://host.docker.internal:11434`.
- **Linux:** if a service runs on the host, create an untracked override and include its absolute
path in `thothii-installation.yaml`:
```yaml
services:
core:
extra_hosts:
- "host.docker.internal:host-gateway"
```
Then use `host.docker.internal` in the endpoint. `extra_hosts: host.docker.internal:host-gateway`
is a host routing aid, not a bundled service. Prefer a real DNS name for independently operated
services; retain TLS and authentication even when co-located.
## Build ThothII and tht
The canonical local Compose smoke uses the base file plus the local profile. Keep this exact
base+profile command available for install verification:
~~~sh
docker compose --env-file deploy/env/local.env -f compose.yaml -f deploy/compose.local.yaml up --build -d
~~~
After the stack is ready, configure and check authentication with the single host CLI tht; see
the [local authentication guide](authentication-local.md). Authentication configuration is
installation-global and is checked before workspace tests.
From the repository root, macOS/Linux/WSL2 users run:
```sh
bash scripts/build-local.sh
bash scripts/build-tht.sh
```
Native PowerShell users run:
```powershell
powershell -ExecutionPolicy Bypass -File scripts/build-local.ps1
& "C:\Program Files\Git\bin\bash.exe" scripts/build-tht.sh
```
The second command uses Docker to create native operator binaries under `dist/tht`; users do
not need to know or install Go. Select `tht-darwin-arm64` or `-amd64` on macOS,
`tht-linux-amd64` or `-arm64` on Linux/WSL2, and `tht-windows-amd64.exe` on Windows.
Copy the selected file to the protected operator directory and, on macOS/Linux, run `chmod 0755`
on it.
## Start and verify
Set convenient variables (PowerShell users use `$THT_BIN` and `$INSTALLATION` with `& $THT_BIN`):
Every operator call has the form `tht --installation <absolute-descriptor> <command>`.
```sh
THT_BIN=/absolute/path/to/thothii-operator/tht
INSTALLATION=/absolute/path/to/thothii-operator/thothii-installation.yaml
"$THT_BIN" --installation "$INSTALLATION" update --check-only
"$THT_BIN" --installation "$INSTALLATION" start
"$THT_BIN" --installation "$INSTALLATION" status
"$THT_BIN" --installation "$INSTALLATION" doctor
```
Native PowerShell uses the same order:
```powershell
$THT_BIN = 'C:\Users\operator\thothii-operator\tht.exe'
$INSTALLATION = 'C:\Users\operator\thothii-operator\thothii-installation.yaml'
& $THT_BIN --installation $INSTALLATION update --check-only
& $THT_BIN --installation $INSTALLATION start
& $THT_BIN --installation $INSTALLATION status
& $THT_BIN --installation $INSTALLATION doctor
```
Wait for both services, then check the same-origin frontend and direct loopback core:
```sh
curl --fail http://127.0.0.1:8080/health
curl --fail http://127.0.0.1:8787/health
"$THT_BIN" --installation "$INSTALLATION" pi doctor
"$THT_BIN" --installation "$INSTALLATION" pi test
```
Native PowerShell must call `curl.exe` explicitly; Windows PowerShell may otherwise resolve `curl`
to `Invoke-WebRequest`:
```powershell
curl.exe --fail --silent --show-error http://127.0.0.1:8080/health
curl.exe --fail --silent --show-error http://127.0.0.1:8787/health
& $THT_BIN --installation $INSTALLATION pi doctor
& $THT_BIN --installation $INSTALLATION pi test
```
Open <http://127.0.0.1:8080>. If a check fails, run `tht ... logs` or `pi logs`; these are
bounded and sanitize declared secrets. Do not publish either loopback port.
## Update an installation
Commit or back up local operator changes first and finish active sessions. A promoted Pi image is
selected by the durable, installation-specific `current-image.yaml` after every base/profile file.
Therefore rebuilding `thothii-core:local` followed by `update --check-only` does not reconcile a
previous `pi update`: the old promoted core would remain selected.
Do not delete or edit the selector. `tht status` is the installation-aware selector test. If
the running core image is the base `thothii-core:local` image, no Pi update has promoted a durable
lifecycle image and an ordinary same-Pi-version source rebuild/start is supported. If status shows
a lifecycle image and the pulled Pi pin is unchanged, `pi update` would be a no-op and the procedure
must stop. A changed Pi pin uses transactional `pi update --source build` in either case.
macOS, Linux, and WSL2:
```sh
set -euo pipefail
abort_update() { printf 'Source update stopped: %s\n' "$1" >&2; exit 1; }
require_clean_source() {
local source_state
if ! source_state="$(git status --porcelain --untracked-files=all)"; then
abort_update "git status failed"
fi
[[ -z "$source_state" ]] || abort_update "commit, remove, or back up every tracked/untracked source change"
}
require_clean_source
if ! git pull --ff-only; then abort_update "git pull --ff-only failed"; fi
require_clean_source
if ! git config --local core.autocrlf false; then abort_update "could not set repository LF policy"; fi
if ! bash scripts/verify-line-endings.sh; then abort_update "the pulled checkout contains CRLF files"; fi
if ! SOURCE_REVISION="$(git rev-parse HEAD)"; then abort_update "could not record the pulled revision"; fi
if ! NEXT_PI_VERSION="$(sed -n 's/^ARG PI_VERSION=//p' docker/core.Dockerfile)"; then
abort_update "could not read the pulled Pi pin"
fi
[[ -n "$NEXT_PI_VERSION" && "$NEXT_PI_VERSION" != *$'\n'* ]] || abort_update "expected one pinned default PI_VERSION"
if ! INSTALLATION_STATUS="$("$THT_BIN" --installation "$INSTALLATION" status)"; then
abort_update "tht status failed"
fi
if ! RUNNING_PI_VERSION="$("$THT_BIN" --installation "$INSTALLATION" pi status)"; then
abort_update "tht pi status failed"
fi
RUNNING_PI_VERSION="${RUNNING_PI_VERSION#Pi version: }"
[[ -n "$RUNNING_PI_VERSION" ]] || abort_update "tht pi status returned no version"
COMPACT_STATUS="${INSTALLATION_STATUS//[[:space:]]/}"
USES_BASE_CORE=false
if [[ "$COMPACT_STATUS" == *'"Image":"thothii-core:local"'* ]]; then
USES_BASE_CORE=true
fi
TRANSACTIONAL_PI_UPDATE=true
if [[ "$NEXT_PI_VERSION" == "$RUNNING_PI_VERSION" ]]; then
[[ "$USES_BASE_CORE" == true ]] || abort_update "same Pi version is selected by a durable lifecycle image"
TRANSACTIONAL_PI_UPDATE=false
fi
if ! bash scripts/build-local.sh; then abort_update "the local image build failed"; fi
if ! bash scripts/build-tht.sh; then abort_update "the tht build failed"; fi
if ! "$THT_BIN" --installation "$INSTALLATION" update --check-only; then
abort_update "the installation render check failed"
fi
if [[ "$TRANSACTIONAL_PI_UPDATE" == true ]]; then
if ! "$THT_BIN" --installation "$INSTALLATION" pi update \
--version "$NEXT_PI_VERSION" --source build --yes --drain; then
abort_update "the transactional core update failed"
fi
fi
if ! "$THT_BIN" --installation "$INSTALLATION" start; then abort_update "installation start failed"; fi
if ! curl --fail http://127.0.0.1:8080/health; then abort_update "frontend health check failed"; fi
if ! curl --fail http://127.0.0.1:8787/health; then abort_update "core health check failed"; fi
if ! FINAL_STATUS="$("$THT_BIN" --installation "$INSTALLATION" status)"; then abort_update "final status failed"; fi
if ! FINAL_PI_STATUS="$("$THT_BIN" --installation "$INSTALLATION" pi status)"; then abort_update "final pi status failed"; fi
[[ "${FINAL_PI_STATUS#Pi version: }" == "$NEXT_PI_VERSION" ]] || abort_update "running Pi version does not match the pulled pin"
if ! "$THT_BIN" --installation "$INSTALLATION" doctor; then abort_update "final doctor failed"; fi
require_clean_source
printf 'Built source revision: %s\n%s\n%s\n' "$SOURCE_REVISION" "$FINAL_STATUS" "$FINAL_PI_STATUS"
```
Native Windows PowerShell uses the same fail-closed version comparison and transactional promotion:
```powershell
$ErrorActionPreference = 'Stop'
function Assert-NativeSuccess([string]$Step) {
if ($LASTEXITCODE -ne 0) { throw "$Step failed with exit code $LASTEXITCODE." }
}
function Assert-CleanSource {
$SourceState = @(git status --porcelain --untracked-files=all)
Assert-NativeSuccess 'git status'
if ($SourceState.Count -ne 0) {
throw 'Commit, remove, or back up every tracked/untracked source change.'
}
}
Assert-CleanSource
git pull --ff-only
Assert-NativeSuccess 'source pull'
Assert-CleanSource
git config --local core.autocrlf false
Assert-NativeSuccess 'repository LF policy'
& "C:\Program Files\Git\bin\bash.exe" scripts/verify-line-endings.sh
Assert-NativeSuccess 'pulled checkout LF verification'
$SourceRevision = git rev-parse HEAD
Assert-NativeSuccess 'source revision read'
$VersionLine = @(Select-String -Path docker/core.Dockerfile -Pattern '^ARG PI_VERSION=(.+)$')
if ($VersionLine.Count -ne 1) { throw 'Expected exactly one pinned default PI_VERSION.' }
$NextPiVersion = $VersionLine.Matches[0].Groups[1].Value
$InstallationStatus = @(& $THT_BIN --installation $INSTALLATION status)
Assert-NativeSuccess 'installation status'
$RunningPiStatus = (& $THT_BIN --installation $INSTALLATION pi status)
Assert-NativeSuccess 'Pi status'
$RunningPiVersion = $RunningPiStatus -replace '^Pi version:\s*', ''
if ([string]::IsNullOrWhiteSpace($RunningPiVersion)) { throw 'Pi status returned no version.' }
$Services = $InstallationStatus | ConvertFrom-Json
$CoreServices = @($Services | Where-Object { $_.Service -eq 'core' })
if ($CoreServices.Count -ne 1) { throw 'Installation status did not identify exactly one core service.' }
$UsesBaseCore = $CoreServices[0].Image -eq 'thothii-core:local'
$TransactionalPiUpdate = $true
if ($NextPiVersion -eq $RunningPiVersion) {
if (-not $UsesBaseCore) { throw 'Same Pi version is selected by a durable lifecycle image.' }
$TransactionalPiUpdate = $false
}
powershell -ExecutionPolicy Bypass -File scripts/build-local.ps1
Assert-NativeSuccess 'local image build'
& "C:\Program Files\Git\bin\bash.exe" scripts/build-tht.sh
Assert-NativeSuccess 'tht build'
& $THT_BIN --installation $INSTALLATION update --check-only
Assert-NativeSuccess 'installation render check'
if ($TransactionalPiUpdate) {
& $THT_BIN --installation $INSTALLATION pi update `
--version $NextPiVersion --source build --yes --drain
Assert-NativeSuccess 'transactional core update'
}
& $THT_BIN --installation $INSTALLATION start
Assert-NativeSuccess 'installation start'
curl.exe --fail --silent --show-error http://127.0.0.1:8080/health
Assert-NativeSuccess 'frontend health check'
curl.exe --fail --silent --show-error http://127.0.0.1:8787/health
Assert-NativeSuccess 'core health check'
$FinalStatus = @(& $THT_BIN --installation $INSTALLATION status)
Assert-NativeSuccess 'final installation status'
$FinalPiStatus = (& $THT_BIN --installation $INSTALLATION pi status)
Assert-NativeSuccess 'final Pi status'
if (($FinalPiStatus -replace '^Pi version:\s*', '') -ne $NextPiVersion) {
throw 'Running Pi version does not match the pulled pin.'
}
& $THT_BIN --installation $INSTALLATION doctor
Assert-NativeSuccess 'final doctor'
Assert-CleanSource
Write-Output "Built source revision: $SourceRevision"
Write-Output $FinalStatus
Write-Output $FinalPiStatus
```
The revision is printed only after every source/build/start/health/installation-aware check passes
and a final porcelain check still reports no tracked or untracked source changes. For a changed Pi
pin, status reports the promoted lifecycle candidate; for a same-version installation with no
selector, status reports the rebuilt base core. `update --check-only` alone proves only that Compose
renders.
Review release notes before updating. See [Pi management](pi-management.md) for rollback; never
install a package in the running container.
## Back up and restore
Back up before source/Pi updates and test restoration periodically. First stop cleanly:
```sh
"$THT_BIN" --installation "$INSTALLATION" stop
docker volume ls --format '{{.Name}}' | grep '^thothii-'
```
Identify the four exact volumes belonging to this installation: `settings`, `pi-state`,
`workspace-registry`, and `sessions`. Confirm their Compose project label with `docker volume
inspect`. For each exact volume, archive it to a protected backup directory:
```sh
BACKUP_DIR=/absolute/path/to/backups/2026-08-05
VOLUME=exact-installation-volume-name
mkdir -p "$BACKUP_DIR"
docker run --rm -v "$VOLUME:/source:ro" -v "$BACKUP_DIR:/backup" \
alpine:3.22 tar -C /source -czf "/backup/$VOLUME.tgz" .
```
Native PowerShell can run the same read-only archive container:
```powershell
$BackupDir = 'C:\Users\operator\thothii-backups\2026-08-05'
$Volume = 'exact-installation-volume-name'
New-Item -ItemType Directory -Force $BackupDir | Out-Null
docker run --rm -v "${Volume}:/source:ro" -v "${BackupDir}:/backup" `
alpine:3.22 tar -C /source -czf "/backup/${Volume}.tgz" .
```
Also back up the installation descriptor, operator environment, generated overrides, and secret
files to separate encrypted/protected storage. Never commit them. Record image digests and the Git
revision. Do not back up while containers are running.
Restore only while stopped and only into a new, verified-empty exact target volume. Test the
archive in a disposable installation first:
```sh
TARGET_VOLUME=exact-empty-target-volume-name
ARCHIVE=/absolute/path/to/backups/2026-08-05/exact-volume-name.tgz
docker run --rm -v "$TARGET_VOLUME:/target" alpine:3.22 \
sh -c 'test -z "$(ls -A /target)"'
docker run --rm -v "$TARGET_VOLUME:/target" -v "$(dirname "$ARCHIVE"):/backup:ro" \
alpine:3.22 tar -C /target -xzf "/backup/$(basename "$ARCHIVE")"
```
Native PowerShell uses `Split-Path` to produce the read-only archive mount and archive name:
```powershell
$TargetVolume = 'exact-empty-target-volume-name'
$Archive = 'C:\Users\operator\thothii-backups\2026-08-05\exact-volume-name.tgz'
$ArchiveDir = Split-Path -Parent $Archive
$ArchiveName = Split-Path -Leaf $Archive
docker run --rm -v "${TargetVolume}:/target" alpine:3.22 `
sh -ceu 'test -z "$(ls -A /target)"'
if ($LASTEXITCODE -ne 0) { throw 'The restore target volume is not empty.' }
docker run --rm -v "${TargetVolume}:/target" -v "${ArchiveDir}:/backup:ro" `
alpine:3.22 tar -C /target -xzf "/backup/${ArchiveName}"
if ($LASTEXITCODE -ne 0) { throw 'The volume restore failed.' }
```
Restore all four volumes from the same backup set, restore protected operator files separately,
then run `update --check-only`, `start`, `doctor`, registry status/diagnostics, and a known session
before normal use. Never merge an archive into a non-empty volume.
## Data-preserving uninstall
Run `tht stop`, retain the installation descriptor at the same absolute path, and make one
verified backup set. In Docker Desktop, remove only this installation's stopped `core` and
`frontend` containers and optional local images; leave its four named volumes. On Linux, use the
containers' exact Compose project labels to remove only those stopped containers. Do not prune
global Docker data.
Do **not** run `docker compose down --volumes`: it deletes the application data this procedure is
meant to preserve. Keep the operator directory and protected secrets if you intend to reinstall.
Using the same descriptor path preserves the `tht` project identity and reconnects the same
named volumes after rebuilding the source checkout.
## Next: workspaces and Pi
Complete [local workspace-registry installation](local-workspace-registry.md), including Git trust,
bindings, pull, validation, diagnostics, and registry recovery. Then use [Pi management](pi-management.md)
for provider/model configuration, smoke testing, transactional update, and rollback.
The Git-backed workspace registry is always the workspace source of truth. Local DWH, vector,
embedding, or LLM processes remain independent services and are never added to the mandatory
ThothII core.
docs/install/pi-management.md
-162
View File
@@ -1,162 +0,0 @@
# Pi management
ThothII bundles Pi in the `core` image. Operators use the Pi Management page for safe application
defaults and the host-side `tht` CLI for lifecycle work. A local Pi installation is not
required.
Run these commands from the root of the current ThothII checkout or worktree. `tht` discovers
the valid installation descriptor in that project tree, so it uses the `deploy/` files belonging to
the checkout from which you run it. Do not use `~/bin`: `~` is the user home directory, not the
project root.
```sh
THT_BIN=tht
tht version --json
```
If `tht` is not on `PATH`, install the native host CLI using the installation procedure in
`local.md` or `server.md`, then set `THT_BIN` to that installed binary. For an installation
stored elsewhere, set `THOTHII_INSTALLATION` or pass
`--installation <absolute-path>/thothii-installation.yaml` explicitly.
## Choose application defaults
Use the **Pi Management** page to select the supported provider, model, and reasoning default, then
choose **Save defaults**. The page shows credentials only as present or missing and can run bounded
diagnostics; it never accepts or displays a credential, opens a terminal, or updates an image.
Alternatively, use the CLI from an administrator terminal:
```sh
"$THT_BIN" pi configure
"$THT_BIN" pi configure --provider zai --model glm-5.2 --thinking medium
```
Use GUI Save defaults or CLI `pi configure`, not both for the same change. The CLI's interactive
choices are restricted to supported models; non-interactive use must supply all three values. Both
methods store application defaults in backend installation settings, not in the project policy file.
Useful read-only checks are:
```sh
"$THT_BIN" pi status
"$THT_BIN" pi doctor
"$THT_BIN" pi test
"$THT_BIN" pi check
"$THT_BIN" pi logs
```
`pi check` is an alias for `pi test`; logs are a sanitized, bounded snapshot with no follow mode.
## Edit the provider catalog and enabled-model policy
Edit these project-root files in source control, then review and deploy the change through the
normal project process:
- `deploy/pi/models.json` is the provider catalog: provider endpoints and the models each provider
offers.
- `deploy/pi/settings.json` is the enabled-model policy only; it lists the models available to the
application and does not store application defaults.
These files contain configuration, not credentials. Keep provider configuration declarative: Pi
management rejects executable `!command` values. Docker Compose mounts the selected configuration
and credential files read-only.
## Store provider credentials
`PI_AUTH_FILE` is a setting in the installation environment file (for example,
`deploy/env/local.env`). Its value is the absolute path of the protected host credential file that
this installation selects. Docker Compose mounts that selected file read-only for Pi.
Other declared protected material is likewise mounted read-only under `/run/secrets`.
Set restrictive permissions on the host file (`0600` on macOS/Linux or a user-only ACL on Windows).
Never put its contents in installation YAML, Git, command arguments, the browser, screenshots,
tickets, rendered Compose output, or logs. Do not print the file while troubleshooting.
## Reload changed configuration
After changing the provider catalog, enabled-model policy, or selected credential file, reload the
running application with one confirmed restart:
```sh
"$THT_BIN" pi restart --yes --drain
```
`--yes` confirms that core will be recreated. Without `--drain`, restart refuses active sessions;
with it, ThothII closes admission and waits for active sessions to finish without terminating them.
The wait is bounded. Restart retains the exact captured running image: it does not build, pull, or
upgrade an image. Before recreating core, it pins that image through transaction-scoped Compose
override material so a configured tag moving during the operation cannot change the selected
image, and Compose is explicitly told never to build or pull. It will restart only core, then
verifies health, Pi version, settings/model smoke, non-secret rendered configuration, and
persistence mounts before reopening admission.
Use `pi restart --yes` when there are already no active sessions. Do not substitute `tht stop`
and `tht start` or raw Compose commands for this reload workflow.
## Update the bundled Pi version
`pi update` is for a new bundled Pi version; it is not a configuration reload. The simple command
uses the single `ARG PI_VERSION=...` pin in `docker/core.Dockerfile`, builds that version, waits for
active sessions to finish, and recreates only `core`:
```sh
"$THT_BIN" pi update
```
To build a specific version, pass `--version`; source, confirmation, and drain are automatic for
this normal build path:
```sh
"$THT_BIN" pi update --version 0.81.0
```
A registry update must use an immutable digest, never a mutable tag:
```sh
"$THT_BIN" pi update \
--version 0.81.0 --source pull \
--image registry.example.invalid/thothii-core@sha256:<64-lowercase-hex-digits> \
--yes --drain
```
Update keeps new-session admission gated while it builds or pulls a candidate, recreates only
`core`, verifies it, and promotes the image only after success. It preserves the frontend and named
volumes.
## Recover a failed lifecycle operation
If a restart or update fails after core recreation, leave maintenance enabled and preserve the
reported recovery state and transaction override. Do not delete `.tht`, state files,
containers, or volumes. Inspect status and sanitized logs:
```sh
"$THT_BIN" pi maintenance status
"$THT_BIN" pi status
"$THT_BIN" pi logs
```
For a failed update, restore its prior image:
```sh
"$THT_BIN" pi rollback --yes
```
For a failed restart, use maintenance recovery instead of rollback. After repairing the reported
Docker, disk, or configuration problem, use the same command to complete either safe recovery path:
```sh
"$THT_BIN" pi maintenance recover --yes
"$THT_BIN" pi doctor
"$THT_BIN" pi test
```
`pi rollback --yes` restores the prior update image. `pi maintenance recover --yes` checks both
restart and update recovery state before it can reopen admission. If either command fails, keep the
installation gated and collect only the sanitized diagnostics.
## Direct support access
Raw Compose access is unsupported because it can bypass the installation-specific environment and
durable image selector. For support, use the installation-aware `tht pi status`,
`tht pi doctor`, `tht pi test`, and `tht pi logs` commands.
docs/install/psd-workspace-setup.md
-65
View File
@@ -1,65 +0,0 @@
# Policlinico San Donato — setup workspace (nuova gestione)
Authentication acceptance is documented in the [manual authentication matrix](../testing/authentication-manual-acceptance.md).
Use generic OIDC with Authentik as the certified group catalog, map only the exact TOT Users and
TOT Admin groups, then run **Validate workspace source**, `tht auth check`, `tht auth check --interactive`,
and **Test workspace connections** in that order. Browser callback E2E, native Windows execution, approved PSD
manual identities, external L2, and the two parked restore-lock preconditions remain pending the
Task 15/release gates.
Guida operativa per collegare ThothII al DWH di PSD con il nuovo sistema (registry Git + descriptor
v3 + `tht`).
## Stato storico Mac/local (2026-08-13)
> Questo stato è storico per Mac/local; il server PSD Project A usa binding separato `postgres_direct` read-only.
>
> Per la rotazione della credenziale DWH, fare riferimento al [runbook PSD](../operations/psd-dwh-auth-rollout.md): non autorizza modifiche finché i due gate non sono approvati. Il ThothII PSD server resta `postgres_direct`; il Mac e i client remoti usano `rest_api` con una chiave per installazione. `postgres_direct` e `ssh_tunnel` non usano chiavi `dwh-auth`.
- **Repository PSD pubblicato:** `https://github.com/mptyl/tht-workspace-psd` (privato), branch
`main`, commit `d4f9185`. Layout P1.1 già migrato e validato.
- **Deploy key SSH** (sola lettura, senza passphrase) in
`deploy/psd/secrets/git-ssh-key` e registrata sul repo come deploy key `thothii-psd`; il remote
Git usato dall'installazione è `git@github.com:mptyl/tht-workspace-psd.git`.
- **Config operatore pronta** (file reali gitignored in `deploy/psd/`): `operator.env`,
`thothii-installation.yaml` e i secret d'installazione in `secrets/` (pi-auth, secret bundle,
chiave SSH, known_hosts). L'API key DWH va completata nella gestione Workspace ed è conservata
nel vault cifrato del backend. Il certificato REST è self-issued/private: ogni Mac/local senza trust equivalente deve usare `TLS_CA_FILE` e verificare il fingerprint fuori banda, come in `docs/install/dwh-auth-tls.md`.
- **Stack avviato** (progetto `thothii-70417a3e30ea`, via `tht start`): `qdrant`, `embedding`
(con `qwen3-embedding:0.6b`), `core`, `frontend` sani. Il registry ha **clonato e attivato**
`psd-clinical` (stato `ready`).
- **`tht workspace inspect --workspace psd-clinical` = OK** (identità descrittore/catalogo
risolte); la configurazione runtime va completata e testata dalla GUI.
- **Bloccante residuo: VPN.** `supabase-aritmolab.policlinicosandonato.it` non risolve
(`NXDOMAIN`) → il preprocessing DWH e le sessioni live non possono ancora partire.
## Avvio/arresto (canonico)
Usare `tht` (stesso project name, quindi stessi volumi named):
```bash
tht=dist/tht/tht-darwin-arm64
"$tht" --installation "$(pwd)/deploy/psd/thothii-installation.yaml" start
"$tht" --installation "$(pwd)/deploy/psd/thothii-installation.yaml" workspace inspect --workspace psd-clinical --json
"$tht" --installation "$(pwd)/deploy/psd/thothii-installation.yaml" stop
```
> **Nota project name:** `tht` calcola un project name stabile dall'installation descriptor
> (`thothii-<hash>`); `docker compose` "a mano" usa invece `name: thothii` dal `compose.yaml`, quindi
> i volumi named non coinciderebbero. Perciò per lo stack si usa `tht start` (non
> `compose-with-preflight.sh up`).
## Rimane: smoke live di una domanda (P8 L2)
Il preprocessing è già completato. Resta solo:
1. Aprire `http://localhost:8080` e selezionare `psd-clinical`.
2. Creare una sessione con una domanda reale in linguaggio naturale.
3. Seguire le 8 fasi fino al primo gate di revisione.
## Cosa è già stato fatto
- Ristrutturazione del repo PSD nel layout P1.1 + validazione locale.
- Pubblicazione GitHub + deploy key read-only + configurazione Git d'installazione.
- Avvio stack + attivazione registry + `tht inspect` verde.
- **Preprocessing live completato** su PSD: DWH → FK → schema → Evidence, idempotente.
docs/install/reverse-proxy-caddy.md
-98
View File
@@ -1,98 +0,0 @@
# Put ThothII behind Caddy
Choose exactly one authentication mode. Direct ThothII-managed OIDC and deprecated upstream
authentication are mutually exclusive proxy contracts; never combine their directives.
## Direct ThothII-managed OIDC
Use this mode when `auth.yaml` has `mode: oidc`. Caddy terminates TLS and proxies every request to
`frontend`; ThothII performs login, callback validation, session creation, and authorization.
Caddy must not apply `forward_auth` or another external authentication gateway.
The public `/api/auth/oidc/login` and `/api/auth/oidc/callback` paths pass unchanged through the
same proxy as the rest of `/api`. The configured `publicUrl` must match the browser origin.
```caddyfile
thoth.example.invalid {
reverse_proxy 127.0.0.1:8080 {
# No URI rewrite: OIDC login and callback paths reach frontend unchanged.
flush_interval -1
header_up Host {host}
header_up X-Forwarded-Proto https
header_up X-Forwarded-Host {host}
}
log {
output file /var/log/caddy/thoth-access.log
format json
}
}
```
After reload, run Workspace Validate for static validation, `tht auth check` for live,
non-interactive authentication diagnosis, and then Workspace Test for aggregate live validation.
## Deprecated upstream migration mode
Use this section only while the installation explicitly uses deprecated `upstream` mode. Do not
use it with `mode: oidc` or `mode: local`. Here an external authentication gateway owns login and
Caddy applies `forward_auth` before forwarding normalized private identity headers to `frontend`.
Forwarding identity headers alone does not authenticate a user. The authentication gateway returns
2xx only after validating its own credential or session. Clear browser-supplied public and trusted
headers before the subrequest, and map identity only from the successful auth response.
```caddyfile
thoth.example.invalid {
route {
request_header -X-Authenticated-User
request_header -X-Thoth-Principal-Issuer
request_header -X-Thoth-Principal-Subject
request_header -X-Thoth-Principal-Display-Name
request_header -X-Thoth-Is-Admin
request_header -X-Thoth-Trusted-Principal-Issuer
request_header -X-Thoth-Trusted-Principal-Subject
request_header -X-Thoth-Trusted-Principal-Display-Name
request_header -X-Thoth-Trusted-Is-Admin
forward_auth auth-gateway:4180 {
uri /verify
copy_headers {
X-Thoth-Principal-Issuer>X-Thoth-Trusted-Principal-Issuer
X-Thoth-Principal-Subject>X-Thoth-Trusted-Principal-Subject
X-Thoth-Principal-Display-Name>X-Thoth-Trusted-Principal-Display-Name
X-Thoth-Is-Admin>X-Thoth-Trusted-Is-Admin
}
}
reverse_proxy 127.0.0.1:8080 {
flush_interval -1
header_up Host {host}
header_up X-Forwarded-Proto https
}
}
}
```
## Trust boundary
Caddy is the only public listener and proxies only to loopback `frontend`, never directly to
`core`. Configure access logs to omit cookies, authorization data, query strings, and identity
headers. Keep Caddy keys and state outside ThothII source and operator directories.
## Validate and reload
Keep the public firewall closed while validating:
```sh
curl --fail http://127.0.0.1:8080/health
caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
sudo systemctl reload caddy
```
## Test authentication and SSE
For direct OIDC, verify the login path redirects to the configured provider, the callback reaches
ThothII unchanged, forged identity headers grant nothing, and SSE is unbuffered. For deprecated
upstream mode, additionally verify the external gateway rejects unauthenticated traffic and only
its 2xx response can create trusted identity headers.
docs/install/reverse-proxy-nginx.md
-143
View File
@@ -1,143 +0,0 @@
# Put ThothII behind Nginx
Choose exactly one authentication mode. Direct ThothII-managed OIDC and deprecated upstream
authentication are mutually exclusive proxy contracts; never combine their locations or headers.
## Direct ThothII-managed OIDC
Use this mode when `auth.yaml` has `mode: oidc`. Nginx terminates TLS and proxies every request
to loopback `frontend`. ThothII owns OIDC login, callback validation, browser sessions, and
authorization. No external `auth_request` or authentication gateway belongs in this server.
The `location /` block below has a `proxy_pass` without a replacement URI, so public
`/api/auth/oidc/login` and `/api/auth/oidc/callback` are forwarded unchanged. The configured
`publicUrl` must match the browser origin.
```nginx
server {
listen 80;
server_name thoth.example.invalid;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl;
server_name thoth.example.invalid;
ssl_certificate /etc/nginx/tls/thoth/fullchain.pem;
ssl_certificate_key /etc/nginx/tls/thoth/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
location / {
# No auth_request and no URI rewrite: ThothII receives OIDC paths unchanged.
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Connection "";
proxy_pass http://127.0.0.1:8080;
proxy_http_version 1.1;
proxy_buffering off;
proxy_cache off;
proxy_read_timeout 3600s;
add_header X-Accel-Buffering no always;
}
}
```
After reload, run Workspace Validate for static validation, `tht auth check` for live,
non-interactive authentication diagnosis, and then Workspace Test for aggregate live validation.
## Deprecated upstream migration mode
Use this section only while the installation explicitly uses deprecated `upstream` mode. Do not
use it with `mode: oidc` or `mode: local`. In this mode an external authentication gateway owns
login, and Nginx applies `auth_request` before forwarding normalized private identity headers.
Forwarding identity headers alone does not authenticate a user. The authentication gateway returns
2xx only after validating its own credential or session.
```nginx
server {
listen 443 ssl;
server_name thoth.example.invalid;
ssl_certificate /etc/nginx/tls/thoth/fullchain.pem;
ssl_certificate_key /etc/nginx/tls/thoth/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
location = /_authenticate {
internal;
proxy_pass http://auth-gateway:4180/verify;
proxy_pass_request_body off;
proxy_set_header Content-Length "";
proxy_set_header X-Original-URI $request_uri;
proxy_set_header X-Original-Method $request_method;
proxy_set_header X-Authenticated-User "";
proxy_set_header X-Thoth-Principal-Issuer "";
proxy_set_header X-Thoth-Principal-Subject "";
proxy_set_header X-Thoth-Principal-Display-Name "";
proxy_set_header X-Thoth-Is-Admin "";
proxy_set_header X-Thoth-Trusted-Principal-Issuer "";
proxy_set_header X-Thoth-Trusted-Principal-Subject "";
proxy_set_header X-Thoth-Trusted-Principal-Display-Name "";
proxy_set_header X-Thoth-Trusted-Is-Admin "";
}
location / {
auth_request /_authenticate;
auth_request_set $thoth_principal_issuer
$upstream_http_x_thoth_principal_issuer;
auth_request_set $thoth_principal_subject
$upstream_http_x_thoth_principal_subject;
auth_request_set $thoth_principal_display_name
$upstream_http_x_thoth_principal_display_name;
auth_request_set $thoth_is_admin
$upstream_http_x_thoth_is_admin;
proxy_set_header X-Authenticated-User "";
proxy_set_header X-Thoth-Principal-Issuer "";
proxy_set_header X-Thoth-Principal-Subject "";
proxy_set_header X-Thoth-Principal-Display-Name "";
proxy_set_header X-Thoth-Is-Admin "";
proxy_set_header X-Thoth-Trusted-Principal-Issuer $thoth_principal_issuer;
proxy_set_header X-Thoth-Trusted-Principal-Subject $thoth_principal_subject;
proxy_set_header X-Thoth-Trusted-Principal-Display-Name $thoth_principal_display_name;
proxy_set_header X-Thoth-Trusted-Is-Admin $thoth_is_admin;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Connection "";
proxy_pass http://127.0.0.1:8080;
proxy_http_version 1.1;
proxy_buffering off;
proxy_cache off;
proxy_read_timeout 3600s;
add_header X-Accel-Buffering no always;
}
}
```
## Trust boundary
Nginx is the only public listener and proxies only to loopback `frontend`, never directly to
`core`. Keep private keys outside the ThothII tree. Do not log cookies, authorization headers,
OIDC callback query values, authentication bodies, or trusted identity headers.
## Validate and reload
Keep the public firewall closed while validating:
```sh
curl --fail http://127.0.0.1:8080/health
sudo nginx -t
sudo systemctl reload nginx
```
## Test authentication and SSE
For direct OIDC, verify login redirects to the configured provider, callback traffic reaches
ThothII unchanged, forged identity headers grant nothing, and SSE is unbuffered. For deprecated
upstream mode, additionally verify the external gateway rejects unauthenticated traffic and only
its 2xx response can create trusted identity headers.
docs/install/server-workspace-registry.md
-132
View File
@@ -1,132 +0,0 @@
# Server workspace repository installation
This manual supplements [server.md](server.md). A server installation reads one remote Git
repository hosted by GitHub, GitLab, Gitea, Bitbucket, or another Git server. ThothII fetches and
validates complete revisions but never edits, commits, pushes, or publishes workspace source.
## Architecture ownership contract
| Component | Ownership | Operator contract |
| --- | --- | --- |
| DWH | External | Configure the external endpoint and complete runtime credentials through the authenticated GUI. |
| LLM | External | Configure the external endpoint and model policy under installation control. |
| Qdrant | Internal | Compose runs private Qdrant and persists `qdrant-data`; include it in Qdrant backup/restore. |
| Ollama embedding | Internal | Compose runs private Ollama with `qwen3-embedding:0.6b`. |
## Semantic index ownership contract
| Scope | Ownership rule | Isolation rule |
| --- | --- | --- |
| Workspace semantic index | Each workspace keeps exactly one Qdrant collection reserved for itself. | Schema, Evidence, and memory records share that one collection and are separated by the `kind` payload. |
The fixed semantic contract is 1024 dimensions and cosine distance. DWH and LLM remain external;
Qdrant, Ollama, and `embedding-model-init` remain private internal services.
## Service account, storage, and firewall
Run the application as the documented unprivileged service account. Keep the source checkout,
operator files, application data, and workspace authoring clone separate:
```text
/srv/thothii/app/ # ThothII source release
/srv/thothii/operator/ # installation descriptor and protected Git files
/srv/thothii/data/ # application data, encrypted workspace vault, sessions
/srv/workspace-authoring/ # optional curator clone; never mounted into ThothII
```
Expose only the authenticated same-origin reverse proxy. Keep `core`, Qdrant, and Ollama private.
## Prepare and publish a workspace source
Create a local workspace in the external authoring repository, which contains
`thoth-workspaces.yaml`, one
`<workspace-id>/workspace.yaml` per catalog entry, optional repository-owned Evidence, and optional
curated schema annotations. It contains no credentials.
Publishing belongs to the curator workflow outside ThothII: validate, review, commit, and push the
source revision to the configured protected branch. Grant the ThothII service only read access.
<!-- workspace-descriptor-contract:start -->
Schema v3 is the only accepted workspace descriptor.
Schema v1 and v2 workspace descriptors are rejected before activation.
<!-- workspace-descriptor-contract:end -->
## Configure the remote Git repository
Copy `docs/install/examples/thothii-installation.server.yaml` to
`/srv/thothii/operator/thothii-installation.yaml`. Set `workspaceRepository.remote`, `.branch`, and
`.access`, then select exactly one Git transport override. The remote and credential are normally
repository-scoped read-only deploy credentials.
For SSH, mount a private key and pinned known-hosts file. For HTTPS, mount a Git credentials file
and the required CA chain. These installation credentials are not editable in Workspace
management and are never exposed by the API.
## Start and update the installation
Use the installation-aware controller described by `server.md`:
```bash
THT_BIN=/srv/thothii/operator/tht
INSTALLATION=/srv/thothii/operator/thothii-installation.yaml
"$THT_BIN" --installation "$INSTALLATION" start
"$THT_BIN" --installation "$INSTALLATION" doctor
```
The descriptor composes `compose.yaml`, `deploy/compose.server.yaml`, the server session-storage
override, and one read-only Git transport override. **Update workspace repository** fetches a
candidate on the server; it does not transfer workspace files to the operator workstation.
## Complete runtime secrets in Workspace management
### Chiavi DWH REST per installazione
Un'installazione server che seleziona `rest_api` usa una chiave DWH nel vault cifrato o nel file `API_KEY_FILE`; `postgres_direct` e `ssh_tunnel` non usano chiavi `dwh-auth`. Il servizio `dwh-auth` del DWH ha lifecycle `systemd` separato e non appartiene al Compose di ThothII. Vedere [enrollment client](dwh-auth-client-enrollment.md) e [guida server DWH](dwh-auth-server.md).
After repository activation, an authenticated user can:
1. Review the configured repository identity and update it without selecting a workspace.
2. Select a workspace to see the DWH/Evidence credential fields required by its connector modes.
3. Blind-save or rotate values with **Save entered secrets**; returned responses contain status only.
4. Run **Validate workspace source** and then **Test workspace connections**.
5. Use **Forget stored value** for an obsolete value after dependent sessions and jobs have ended.
The backend encrypts values in `/data/workspace-secrets`, including the installation-specific
master key. The server profile persists that directory inside `THT_DATA_ROOT`; no workspace YAML
path depends on Linux, macOS, or Windows. Plaintext exists only in a restrictive temporary file
for the duration of a diagnostic, session, or maintenance lease.
Authorization is intentionally the current installation-wide authenticated-user policy. A future
role model or external secret manager can replace that policy without changing workspace source.
## Validation and activation behavior
Repository update is all-or-nothing: ThothII fetches the configured branch, validates catalog,
descriptors, Evidence paths, and cross-workspace invariants at one commit, then atomically activates
the complete candidate. A rejected candidate never replaces the previous active snapshot. The
application-owned checkout and snapshots are read-only runtime state.
Validation proves descriptor and repository structure. **Test workspace connections** additionally
materializes the current runtime secrets and contacts only the selected workspace's configured
DWH/Evidence endpoints. Failure does not modify or publish workspace source.
## Backup, rotation, and recovery
Back up application data and Qdrant consistently. Qdrant backup/restore must cover `qdrant-data`;
application recovery must cover repository snapshots/state, sessions, settings, Pi state, and the
entire encrypted `/data/workspace-secrets` directory. Store backup encryption keys separately and
test restore procedures without production traffic.
Rotate DWH/Evidence credentials through Workspace management. Rotate Git access by atomically
replacing its protected installation file and restarting `core`. Recover a bad source revision by
reverting or correcting it in the external authoring repository and updating again.
## Troubleshooting
| Symptom | Meaning and action |
| --- | --- |
| Git authentication failed | Verify repository-scoped read permission, branch, key/token, CA, and host-key pinning. |
| Candidate validation failed | Correct the source repository; the prior active commit remains in service. |
| Runtime configuration required | Select the workspace and complete all required write-only fields. |
| Secret store unavailable | Stop writes, preserve `/data/workspace-secrets`, and restore vault plus master key together. |
| Connection test failed | Rotate the indicated runtime credential or correct the relevant non-secret endpoint. |
docs/install/server.md
-568
View File
@@ -1,568 +0,0 @@
# Install ThothII on a Linux server
Server authentication uses generic OIDC with the reverse proxy preserving the configured public
origin and callback path. Follow the [OIDC guide](authentication-oidc.md), [Authentik guide](authentik.md)
when applicable, and the [authentication acceptance matrix](../testing/authentication-manual-acceptance.md).
The host authentication CLI is `tht`. Use `tht --installation <descriptor> workspace inspect
--workspace <id> --json` for the active workspace snapshot, `tht --installation <descriptor>
auth check` for live non-interactive authentication diagnosis, `auth check --interactive` for
device-flow identity validation, and `tht ... doctor --json` for the aggregate installation gate.
This guide is for an installer with basic Linux administration and very basic Docker knowledge.
It deploys the same Compose distribution used on a local PC: the mandatory application is exactly
`frontend` plus `core`, and pinned Pi is inside `core`. The server does not need host Pi, Node.js,
Python, Go, a browser shell, or a Docker socket inside either container.
Examples use `/srv/thothii` as an **example operator root**, `thoth.example.com` as a replaceable
DNS name, and systemd-based command names. Adapt them to local policy. Complete the server session
storage overlay and migration procedure before exposing a production installation.
## Deployment contract
- The generic Linux host and Docker Compose v2 are the deployment platform. No other
application's Compose project, network, path, or runtime is required.
- `frontend` is the only published service and defaults to `127.0.0.1:8080`; `core` has no host
port. A host Nginx or Caddy listener terminates TLS and sends all application traffic to
`frontend`, never directly to `core`.
- DWH, vector database, embedding service, and LLM are external configurable endpoints. This
remains true when they happen to run on the same physical server.
- Application, Git, connector, and session credentials are protected host files mounted read-only
under `/run/secrets`. Pi's protected auth JSON uses its dedicated read-only Pi mount. No secret
value belongs in Git, images, browser storage, environment values, rendered Compose, or logs.
- The Git-backed workspace registry is the source of truth. Installation-local bindings identify
endpoints and secret-file paths; they do not replace the reviewed Git workspace descriptors.
- `tht` is the operator CLI for start, stop, status, health, logs, Pi lifecycle, drain, and
rollback. Raw Compose lifecycle commands bypass installation state and are unsupported.
Read [server workspace-registry installation](server-workspace-registry.md),
[Pi management](pi-management.md), and the session-server comments in
`deploy/compose.session-server.yaml.example` before the first public start.
## Service account and directories
The container runtime identity is fixed at UID/GID 10001. It does not require or permit creation of
a matching host account or group. Keep the number unmapped and use numeric ownership only for the
dedicated bind paths that the non-root container must read or write. If either lookup below finds a
host identity, stop and design an explicit remapping before installation.
```sh
if getent passwd 10001 >/dev/null || getent group 10001 >/dev/null; then
printf '%s\n' 'UID/GID 10001 is already mapped; stop' >&2
exit 1
fi
operator_uid="$(id -u)"
operator_gid="$(id -g)"
test "$operator_uid" -ne 0
id -nG | tr ' ' '\n' | grep -Fx docker >/dev/null
```
The invoking, pre-existing administrator owns source and operator files. It must already have the
site-approved Docker access required to run `tht`; this guide never changes group membership.
Docker-group membership is effectively host-root access and must remain limited to reviewed
administrators. Do not grant the operator direct write access to container runtime trees.
Create explicit directories. `source` contains the clone; `operator` contains untracked path-only
configuration; the three writable trees are bind-mounted into `core`; `secrets` contains regular
files only. The parent is owned by the operator with numeric group 10001 so both the operator and
container can traverse it. Numeric ownership does not add entries to `/etc/passwd` or `/etc/group`.
```sh
operator_uid="$(id -u)"
operator_gid="$(id -g)"
sudo install -d -o "$operator_uid" -g 10001 -m 0750 /srv/thothii
sudo install -d -o "$operator_uid" -g "$operator_gid" -m 0750 /srv/thothii/source
sudo install -d -o "$operator_uid" -g "$operator_gid" -m 0750 /srv/thothii/operator
sudo install -d -o 10001 -g "$operator_gid" -m 0750 /srv/thothii/secrets
sudo install -d -o 10001 -g 10001 -m 0750 /srv/thothii/data
sudo install -d -o 10001 -g 10001 -m 0750 /srv/thothii/pi-state
sudo install -d -o 10001 -g 10001 -m 0750 /srv/thothii/workspace-registry
sudo install -d -o root -g root -m 0700 /srv/thothii-backups
stat -c '%u:%g %a %n' \
/srv/thothii /srv/thothii/source /srv/thothii/operator /srv/thothii/secrets \
/srv/thothii/data /srv/thothii/pi-state /srv/thothii/workspace-registry \
/srv/thothii-backups
```
Expected: the parent is `operator_uid:10001 750`; source/operator are
`operator_uid:operator_gid 750`; secrets are `10001:operator_gid 750`; the three runtime trees are
`10001:10001 750`; backups are `0:0 700`. Re-run the empty `getent` checks after creation. Do not
make `/srv/thothii` a shared application directory.
## Projected server authentication: canonical root and runtime projection
For a server descriptor that declares `authentication.runtimeProjection`, authentication has two
different roots. The **canonical authentication root** (`authentication.configDirectory`) is the
root-operated source of truth. It and its regular files are `root:root 0700/0600`. The **runtime
projection** is a separate Linux-only tree for the container reader: its root, `generations`, and
generation directories are `10001:10001 0700`; `CURRENT`, `manifest.json`, `auth.yaml`, and (for
local mode) `users.yaml` are `10001:10001 0600`. The publisher assigns the numeric IDs directly;
it does not create a host user or group for 10001.
The runtime projection has only `CURRENT` and `generations/<64-lowercase-hex>/`. `CURRENT` selects
one complete immutable generation. A successful configure, user mutation, restore, or explicit
publish first blocks `CURRENT`, then verifies a new immutable generation, then makes it ready.
The selected generation and up to two predecessor generations are retained; no operator edits a
generation or `CURRENT` directly. A ready projection is usable only when its canonical revision is
equal to the current canonical authentication root. If the runtime projection is blocked, missing,
tampered, or unequal, `start`, `update --check-only`, `auth check`, and `doctor` fail closed before
admission or Compose lifecycle work.
The runtime directory must be an absolute canonical path, distinct from the canonical root, and
must exactly equal `THT_AUTH_RUNTIME_ROOT` in the protected installation environment. The server
profile and numeric UID/GID values are validated before any projected mutation or publication.
The descriptor loader adds `compose.auth-runtime-projection.yaml` automatically when
`runtimeProjection` is present; do not list that file under `overrides`. The automatic override
mounts the runtime projection **read-only and core-only** at `/run/thothii-auth`; the canonical
authentication root is never mounted. No other service receives that mount or
`THT_AUTH_RUNTIME_PROJECTION_ROOT`. The example descriptor uses
`/srv/example/thothii/auth-runtime` only as a replaceable path and contains no credential value.
This source change is prepared and tested only: Project A has not been started. It does not
authorize a raw Compose lifecycle launch, an Nginx change, legacy-stack change, or mutation of
`/srv`. A later manual gate needs separate explicit authorization before applying any descriptor
or runtime root to a server.
### Status, repair, and safe evidence
Use the root-operated installation command; retain only its small redacted JSON result:
```sh
sudo tht --installation "$INSTALLATION" auth status --json
```
`state: "ready"` and `equal: true` are required before a projected server can start. `state:
"blocked"`, `equal: false`, or a command refusal means that the runtime projection is blocked or
cannot be validated. Do not start the stack, inspect YAML, print an environment, or edit `CURRENT`
or a generation. Confirm the protected canonical root is available, then republish it with:
```sh
sudo tht --installation "$INSTALLATION" auth publish
sudo tht --installation "$INSTALLATION" auth status --json
```
`auth publish` reconstructs the selected immutable generation from the canonical root; it never
uses an older runtime generation as authority. If publish fails, leave the projection blocked and
escalate using the sanitized command result plus descriptor path and timestamp only. Do not attach
passwords, hashes, YAML, raw environment output, `nginx -T`, or a secret-bearing diff to evidence.
### Authentication restore
An authentication-bearing restore first publishes a blocked selector, restores canonical
authentication, and publishes a verified candidate generation before any restart. If candidate or
recovery verification fails, the verified recovery checkpoint is republished when possible; an
unverified result remains blocked and prevents start. A restore without authentication entries
does not touch the runtime projection. This is in addition to the normal restore requirement that
browser sessions and pending OIDC state are cleared.
## Firewall and network boundaries
Set `THOTH_SERVER_BIND=127.0.0.1`. Permit inbound TCP 80/443 only to the TLS proxy; port 80 should
redirect to HTTPS. Do not open 8080 externally, and do not add a core port. If a separate proxy
host is used, replace loopback with a private, firewalled address and allow only that proxy source.
Allow outbound DNS and HTTPS to the source/Git registries, plus only the configured ports for the
Git remote, DWH, vector database, embedding service, LLM, session PostgreSQL, and any approved
bastion. Docker's private `thothii` network carries only `frontend`↔`core` traffic. Do not attach
the mandatory stack to another application's network.
After start, confirm the host listens as intended:
```sh
sudo ss -lntp
```
Expected public listeners are the proxy on 80/443 and the frontend on loopback 8080. There must be
no host listener for core port 8787.
## Address co-resident external services
Endpoint values are resolved inside `core`. Therefore container 127.0.0.1 means the container
itself, not the Linux host. Prefer real DNS names with TLS, authentication, and firewall policy,
even for services on this physical server.
When DNS is unavailable for a host-published service, create an untracked override such as
`/srv/thothii/operator/host-gateway.yaml` and add it to the installation descriptor:
```yaml
services:
core:
extra_hosts:
- "host.docker.internal:host-gateway"
```
Use `host.docker.internal` in the endpoint binding. The `host-gateway` mapping supplies routing;
it does not bundle or trust the target service. A host service listening only on host
`127.0.0.1` is **not reachable** through this mapping. Bind that service to the ThothII Docker
bridge gateway address or to a dedicated private host interface—never to `0.0.0.0` merely to make
the check pass. A stable internal DNS record routed through an authenticated private listener is
the preferred alternative.
After the first bounded start attempt, copy the exact core container name from `tht status`
into `CORE_NAME`, then derive—not guess—the network ID, Linux bridge interface, gateway, and
subnet. Compose networks normally use `br-<first-12-network-id>`; an explicit
`com.docker.network.bridge.name` option takes precedence:
```sh
CORE_NAME=replace-with-exact-core-container-name
NETWORK_ID=$(docker inspect --format '{{range .NetworkSettings.Networks}}{{.NetworkID}}{{end}}' "$CORE_NAME")
NETWORK_NAME=$(docker network inspect --format '{{.Name}}' "$NETWORK_ID")
BRIDGE=$(docker network inspect --format '{{index .Options "com.docker.network.bridge.name"}}' "$NETWORK_ID")
test -n "$BRIDGE" || BRIDGE="br-${NETWORK_ID%${NETWORK_ID#????????????}}"
GATEWAY=$(docker network inspect --format '{{(index .IPAM.Config 0).Gateway}}' "$NETWORK_ID")
SUBNET=$(docker network inspect --format '{{(index .IPAM.Config 0).Subnet}}' "$NETWORK_ID")
printf 'network=%s bridge=%s gateway=%s subnet=%s\n' "$NETWORK_NAME" "$BRIDGE" "$GATEWAY" "$SUBNET"
ip address show dev "$BRIDGE"
```
Bind the co-resident service to `$GATEWAY`. In the host firewall `INPUT` chain, allow its exact
TCP port only when source is `$SUBNET`, input interface is `$BRIDGE`, and destination is
`$GATEWAY`; reject other sources to that listener and persist the rules using the distribution's
firewall manager. Docker's `DOCKER-USER` chain governs forwarded/published traffic and does not
replace this host-input rule. Ask the firewall administrator to implement the equivalent policy
with nftables when iptables is not the site's source of truth.
For an iptables-managed host, replace the port before applying these reviewed rules; the second
rule prevents any other interface/source from reaching that gateway listener:
```sh
EXTERNAL_PORT=replace-with-exact-service-port
sudo iptables -I INPUT 1 -i "$BRIDGE" -s "$SUBNET" -d "$GATEWAY" -p tcp --dport "$EXTERNAL_PORT" -j ACCEPT
sudo iptables -I INPUT 2 -d "$GATEWAY" -p tcp --dport "$EXTERNAL_PORT" -j REJECT
```
Confirm reachability with `tht pi test` for the configured LLM/Pi path and with the
authenticated Workspace Diagnostics action for DWH, vector collection/embedding pairing, and
embedding endpoints. A timeout paired with `ss -lntp`, `ip address show dev "$BRIDGE"`, and the
firewall counters distinguishes a loopback bind from a subnet/interface rule failure. Do not add
a shell to the browser or mount the Docker socket into core for this diagnostic.
Configure each boundary independently:
- DWH: read-only runtime identity, database/schema, verified TLS, and direct or REST endpoint.
- Vector database: endpoint plus exact database/schema, collection, distance metric, and writer
policy declared by the reviewed workspace.
- Embedding service: endpoint and the collection/embedding pairing—model and dimensions must match
the existing collection. Co-residence does not permit silently changing that pairing.
- LLM: authenticated endpoint selected through deployment and Pi configuration.
Never add those services to ThothII's mandatory Compose files. Follow
[the diagnostic protocol](../workspace-diagnostic-protocol.md) before enabling a workspace.
## Prepare operator files and secrets
Clone with LF line endings, then verify before every build:
```sh
git -c core.autocrlf=false clone \
https://github.example.invalid/your-org/ThothII.git /srv/thothii/source/ThothII
cd /srv/thothii/source/ThothII
git config --local core.autocrlf false
bash scripts/verify-line-endings.sh
sudo /srv/thothii/source/ThothII/scripts/prepare-server-pi-state.sh \
/srv/thothii/pi-state 10001 10001
```
The last command is a mandatory clean-install and restore preflight. The server profile bind-mounts
the writable Pi-state root and then overlays protected `auth.json` plus tracked `models.json` and
`settings.json` read-only below it. Docker requires those three hidden target files to exist under
the host parent bind before startup. The initializer creates them atomically with UID/GID 10001,
mode `0600`, rejects symlink roots or targets, and never overwrites existing contents. It is safe to rerun
after restoring `pi-state`; run it before any `tht start`, Compose render/start, or Pi update.
Copy the path-only server environment and installation descriptor:
```sh
cp deploy/env/server.env.example /srv/thothii/operator/server.env
cp docs/install/examples/thothii-installation.server.yaml \
/srv/thothii/operator/thothii-installation.yaml
chmod 0600 /srv/thothii/operator/server.env \
/srv/thothii/operator/thothii-installation.yaml
```
The invoking operator owns both placeholder files; use an editor that preserves ownership and mode,
or create replacements under `umask 0077` in the operator directory.
Replace every placeholder with an absolute path. Use exactly one Git transport override. For
HTTPS, replace `deploy/compose.git-ssh.yaml` with `deploy/compose.git-https.yaml`. Keep the required
session-server overlay. Optional host-gateway or pinned
image overrides go after them.
Create each installation credential (Pi/application, Git, and session storage) as an independent
regular file in `/srv/thothii/secrets`, owned by
UID 10001, the invoking operator's numeric primary GID, and mode `0640`. Owner access lets the UID
10001 container read a file mounted under `/run/secrets`; group access lets the operator run `tht`. The
operator environment records only absolute `*_FILE` or `*_SOURCE` paths for those installation
credentials. DWH and Evidence values are entered later through Workspace management and persist
as ciphertext under `/data/workspace-secrets`; the frontend receives no secret values. Do not
print file contents while testing permissions.
```sh
operator_gid="$(id -g)"
sudo find /srv/thothii/secrets -type f -exec chown "10001:$operator_gid" {} +
sudo find /srv/thothii/secrets -type f -exec chmod 0640 {} +
sudo find /srv/thothii/secrets -type f \( ! -uid 10001 -o ! -gid "$operator_gid" -o ! -perm 0640 \) -print
```
Configure the remote repository and exactly one read-only Git transport as described in
[server workspace repository installation](server-workspace-registry.md). After startup, complete
the selected workspace's DWH and Evidence credentials through Workspace management. Secret values
must never be pasted into `server.env`, the installation YAML, a URL, or a shell argument.
## Build locally or select pinned images
Choose one image source. For a source build, the repository's reproducible launcher builds the
same `core` and `frontend` images used by the local profile. It requires only Git, Docker, and
Compose; copy the reviewed path-only server environment to the launcher's untracked input first:
```sh
cd /srv/thothii/source/ThothII
cp /srv/thothii/operator/server.env deploy/env/local.env
bash scripts/build-local.sh
```
The printed local-profile start command is not the server start command; use `tht` below.
Alternatively, create a reviewed untracked override with release images pinned by immutable
digest. Mutable tags are not a production pin:
```yaml
services:
core:
build: !reset null
image: registry.example.com/thothii/core@sha256:<64-lowercase-hex-digits>
session-migrate:
build: !reset null
image: registry.example.com/thothii/core@sha256:<64-lowercase-hex-digits>
frontend:
build: !reset null
image: registry.example.com/thothii/frontend@sha256:<64-lowercase-hex-digits>
```
Add that absolute file last in `overrides`. `core` and `session-migrate` must use the exact same
core digest; neither may retain a local build or `:local` image. Frontend uses its own exact digest.
Both images must come from one compatible release; the core image must retain the declared Pi
version labels checked by `tht pi doctor`. Pull access
belongs in the host Docker credential store, not in Compose or the installation descriptor.
## Install tht
Build the operator binaries with Docker. No Go installation or Go knowledge is required:
```sh
cd /srv/thothii/source/ThothII
THT_THT_OUTPUT_DIRECTORY=/srv/thothii/operator/build-output \
bash scripts/build-tht.sh
operator_gid="$(id -g)"
sudo install -o root -g "$operator_gid" -m 0750 \
/srv/thothii/operator/build-output/tht-linux-amd64 \
/srv/thothii/operator/tht
```
The source checkout remains controlled by the invoking operator. The explicit output directory is the only build
write boundary; the build script rejects relative or non-canonical output paths. After installation,
remove or retain `build-output` according to the site's reviewed artifact policy.
Use `tht-linux-arm64` on an ARM64 server. Set these variables in the maintenance shell; do
not source `server.env` as shell code:
```sh
THT_BIN=/srv/thothii/operator/tht
INSTALLATION=/srv/thothii/operator/thothii-installation.yaml
"$THT_BIN" --help
"$THT_BIN" --installation "$INSTALLATION" update --check-only
```
Every operator command includes the descriptor explicitly. This preserves the installation's
profile, overrides, project identity, and durable current-image selector. The general form is
`tht --installation /absolute/path/thothii-installation.yaml <command>`.
## Start and verify readiness
Keep the TLS proxy stopped or firewalled during bootstrap. First stop the app, run the
installation-aware session migration, and inspect its pristine JSON. The command activates only
the `session-migrate` profile/service with `--no-deps --no-TTY`; it derives the migrator image from the
selected core image after all installation overrides, so this procedure is identical for source
and pinned modes. It exits nonzero unless both arrays are empty:
```sh
"$THT_BIN" --installation "$INSTALLATION" stop
"$THT_BIN" --installation "$INSTALLATION" sessions migrate --yes
```
Successful output has this shape (the `applied` list may contain versions on first use):
```json
{"applied":[],"drifted":[],"pending":[]}
```
Only after seeing `"pending":[]` and `"drifted":[]`, start and verify:
```sh
"$THT_BIN" --installation "$INSTALLATION" start
"$THT_BIN" --installation "$INSTALLATION" status
"$THT_BIN" --installation "$INSTALLATION" doctor
curl --fail http://127.0.0.1:8080/health
"$THT_BIN" --installation "$INSTALLATION" pi doctor
"$THT_BIN" --installation "$INSTALLATION" pi test
```
`/health` proves process liveness. Readiness additionally requires both healthy services, a valid
Pi provider/model smoke, a successful Git registry pull with an active validated snapshot, valid
workspace diagnostics, and ready session PostgreSQL. Use the authenticated Workspace Management
page to pull and diagnose the reviewed workspace. A liveness response alone is not release
approval.
After configuring the proxy, open <https://thoth.example.com> in a browser. Verify an unauthenticated
request is denied or redirected by the real identity provider, an authorized user can load the
same-origin UI and `/api`, an unauthorized user is denied, and an administrator alone can open Pi
Management. Keep port 8080 inaccessible from other hosts.
## Configure TLS and upstream authentication
Choose [Nginx](reverse-proxy-nginx.md) or [Caddy](reverse-proxy-caddy.md). Both examples terminate
TLS and proxy only to loopback `frontend`. They preserve SSE and clear client-supplied identity
headers before authentication.
The authentication gateway must validate a real login/session and return normalized issuer,
subject, display-name, and admin claims only after success. Merely forwarding those headers does
not authenticate anyone. Do not enable `AUTH_MODE=upstream` on a listener reachable around the
trusted proxy, and never expose `core`.
## Operate Pi, drain, and roll back
Configure only closed provider/model/reasoning choices. Credentials remain protected files:
```sh
"$THT_BIN" --installation "$INSTALLATION" pi status
"$THT_BIN" --installation "$INSTALLATION" pi configure
"$THT_BIN" --installation "$INSTALLATION" pi doctor
"$THT_BIN" --installation "$INSTALLATION" pi logs
```
Before an update, announce maintenance and ask users to finish active work. `--drain` closes new
admission and waits until no active sessions remain; it does not discard sessions. Build-source
and registry-source examples are:
```sh
"$THT_BIN" --installation "$INSTALLATION" pi update \
--version 0.81.0 --source build --yes --drain
"$THT_BIN" --installation "$INSTALLATION" pi update \
--version 0.81.0 --source pull \
--image registry.example.com/thothii/core@sha256:<64-lowercase-hex-digits> \
--yes --drain
```
The transaction recreates only `core`, preserves volumes, verifies health/configuration/Pi, and
automatically attempts rollback after a post-mutation failure. For interrupted or ambiguous state:
```sh
"$THT_BIN" --installation "$INSTALLATION" pi maintenance status
"$THT_BIN" --installation "$INSTALLATION" pi rollback --yes
"$THT_BIN" --installation "$INSTALLATION" pi maintenance recover --yes
```
Leave maintenance active if rollback cannot be verified. Preserve `.tht/<installation-id>/`
recovery state, repair the reported host/configuration issue, and rerun rollback or maintenance
recovery. Never delete or edit `current-image.yaml` or `update-state.json` to force progress.
## Back up and restore
Back up before source, workspace, session-schema, or Pi changes. Drain work, stop the installation,
record `git rev-parse HEAD`, image digests, and `tht status`, then archive the three bind trees
with numeric ownership. Do not include live secrets in this ordinary archive.
```sh
"$THT_BIN" --installation "$INSTALLATION" stop
BACKUP=/srv/thothii-backups/2026-08-05
sudo install -d -o root -g root -m 0700 "$BACKUP"
sudo tar --numeric-owner --xattrs --acls -C /srv/thothii -czf "$BACKUP/runtime-data.tgz" \
data pi-state workspace-registry
sudo sh -ceu 'cd "$1"; sha256sum runtime-data.tgz > SHA256SUMS; sha256sum --check SHA256SUMS' sh "$BACKUP"
```
Back up the installation descriptor, path-only environment, generated overrides, source revision,
and secret files to separate encrypted access-controlled storage. Database-backed production
sessions require their own PostgreSQL-native consistent backup; the local bind tree is not a
substitute. Test both restore paths periodically.
Restore only while stopped. Verify the checksum, extract first into a new empty root, inspect
ownership and expected registry layout, then retain the old trees by renaming them before placing
the restored set. This keeps the previous state recoverable:
```sh
RESTORE=/srv/thothii-restore-2026-08-05
sudo install -d -o root -g root -m 0700 "$RESTORE"
sudo sh -ceu 'cd "$1"; sha256sum --check SHA256SUMS' sh /srv/thothii-backups/2026-08-05
sudo tar --numeric-owner --xattrs --acls -C "$RESTORE" \
-xzf /srv/thothii-backups/2026-08-05/runtime-data.tgz
sudo test -d "$RESTORE/workspace-registry/repo"
sudo test -d "$RESTORE/workspace-registry/snapshots"
```
After placing the restored `pi-state` tree and before the first start, rerun
`sudo /srv/thothii/source/ThothII/scripts/prepare-server-pi-state.sh /srv/thothii/pi-state 10001 10001`.
It validates or recreates only the hidden regular mount targets; it does not alter restored Pi
state or any protected configuration source.
During the reviewed restore window, move each old tree to a timestamped sibling, move the matching
restored tree into `/srv/thothii`, restore the PostgreSQL session backup from the same recovery
point, and keep the proxy closed. Run `update --check-only`, `start`, `doctor`, `pi test`, registry
status, workspace diagnostics, and a known historical session before reopening traffic. Never
merge an archive into a non-empty tree.
## Diagnostics
Begin with bounded, sanitized installation-aware commands:
```sh
"$THT_BIN" --installation "$INSTALLATION" status
"$THT_BIN" --installation "$INSTALLATION" doctor
"$THT_BIN" --installation "$INSTALLATION" logs
"$THT_BIN" --installation "$INSTALLATION" pi status
"$THT_BIN" --installation "$INSTALLATION" pi doctor
"$THT_BIN" --installation "$INSTALLATION" pi test
"$THT_BIN" --installation "$INSTALLATION" pi logs
"$THT_BIN" --installation "$INSTALLATION" pi maintenance status
```
Use the authenticated Workspace Management status and diagnostic actions for Git revision,
degraded snapshot, bindings, DWH, vector, and embedding checks. Review proxy logs separately, but
configure both proxy and log shipping to exclude cookies, authorization data, identity payloads,
query strings, and secret values. Do not render Compose or print an environment as a diagnostic.
Typical boundaries are: `doctor` for Docker/Compose/LF/volume/service health; `pi doctor` for image
and provider/model integrity; registry status for Git/snapshot health; workspace diagnostics for
external service identity; and the proxy/identity provider for login failures.
## Data-preserving uninstall
Drain and stop through `tht`, take and verify one final backup, and disable the TLS proxy
route. Set `THT_BACKUP_ROOT=/srv/thothii-backups` in `server.env`; the removal command verifies the
filesystem identity of that backup root, all three bind trees, and every declared secret before
and after removing anything.
First run without confirmation. It displays the exact installation project, service, container
name, container ID, and stopped state, then exits without mutation. Check every target:
```sh
"$THT_BIN" --installation "$INSTALLATION" stop
"$THT_BIN" --installation "$INSTALLATION" remove
```
If and only if both targets are the expected stopped `frontend` and `core` containers, confirm:
```sh
"$THT_BIN" --installation "$INSTALLATION" remove --yes exact-core-id exact-frontend-id
```
Replace both example IDs with the values from the immediately preceding dry-run. The command
refuses confirmation if the current target set differs. The confirmed operation passes only those
previously displayed immutable container IDs to Docker,
uses no force or volume option, rejects running/replaced containers, and proves the preservation
paths still identify the same filesystem objects. Keep `/srv/thothii/data`, `pi-state`,
`workspace-registry`, `operator`, protected secrets, database backups, and the installation
descriptor if reinstallation is possible. Do not prune global Docker data.
Do **not** run `docker compose down --volumes`; it deletes persistent application data. Reusing the
same protected descriptor path preserves the `tht` installation identity and allows a later
compatible source checkout to reconnect the retained state.
docs/install/windows-line-endings.md
-250
View File
@@ -1,250 +0,0 @@
# Windows and WSL2 line endings
ThothII's containers execute shell scripts from the source checkout. Those files must stay LF,
even when the PC normally uses CRLF. The repository's `.gitattributes` is authoritative, but a
Windows Git setting or an old checkout can still leave incorrect bytes. Check line endings after
every clone and pull, before building an image.
## Recommended WSL2 clone
Use Docker Desktop with WSL2 integration. Clone inside the Linux filesystem, for example under
`/home/<user>/src`, rather than under `/mnt/c`. This avoids slow cross-filesystem builds,
permission surprises, and Windows tools rewriting files behind WSL.
```sh
mkdir -p "$HOME/src"
cd "$HOME/src"
git -c core.autocrlf=false clone https://github.example.invalid/your-org/ThothII.git
cd ThothII
git config --local core.autocrlf false
bash scripts/verify-line-endings.sh
```
Keep Docker Desktop's integration enabled for that WSL distribution. Run the Linux build scripts
and the Linux `tht` binary from the same WSL shell.
## Repository-local LF policy
Set the option in this repository only. Do not change a company-wide or personal Git policy just
for ThothII.
```sh
git config --local core.autocrlf false
git config --local --get core.autocrlf
```
The second command must print `false`. `.gitattributes` keeps shell, YAML, Dockerfile, JSON,
TypeScript, Python, and Markdown files at LF; PowerShell files remain CRLF.
For a native PowerShell clone, disable conversion during the first checkout and then store the
repository-local setting:
```powershell
git -c core.autocrlf=false clone https://github.example.invalid/your-org/ThothII.git
Set-Location ThothII
git config --local core.autocrlf false
& "C:\Program Files\Git\bin\bash.exe" scripts/verify-line-endings.sh
```
## Verify after clone or pull
From WSL2, Git Bash, macOS, or Linux run:
```sh
bash scripts/verify-line-endings.sh
```
Success exits with code 0 and prints no offending path. If it lists a file, do not build or start
ThothII. Correct the checkout first. Native PowerShell users can invoke the same script through
Git for Windows as shown above.
## Recover an existing CRLF clone
The safest recovery is to reclone into a new directory. First commit wanted work or copy it to a
backup outside both clones. Then clone with conversion disabled, run the verifier, and copy back
only reviewed changes.
If a reviewed working tree must be repaired in place, Git must first normalize the index, export
that exact index to a separate repair directory, verify the exported bytes, and only then copy the
verified tracked files over the worktree. `git add --renormalize .` alone does not change existing
worktree bytes.
> **WARNING — destructive worktree rewrite.** Make a backup outside the clone or commit every
> wanted tracked change before continuing. The copy step below overwrites tracked worktree bytes
> from the staged index export. Stop if the staged diff does not contain exactly the wanted content;
> untracked files are neither exported nor repaired.
From WSL2, Git Bash, macOS, or Linux:
```sh
set -euo pipefail
abort_repair() { printf 'CRLF repair stopped: %s\n' "$1" >&2; exit 1; }
validate_index_export() {
git ls-files -s -z | while IFS= read -r -d '' entry; do
metadata="${entry%%$'\t'*}"
path="${entry#*$'\t'}"
mode="${metadata%% *}"
[[ "$path" != "$entry" ]] || exit 1
case "$mode" in
100644|100755) [[ -f "$REPAIR_DIR/$path" && ! -L "$REPAIR_DIR/$path" ]] || exit 1 ;;
120000) [[ -L "$REPAIR_DIR/$path" ]] && readlink "$REPAIR_DIR/$path" >/dev/null || exit 1 ;;
*) printf 'Unsupported Git mode %s: %s\n' "$mode" "$path" >&2; exit 1 ;;
esac
done
}
validate_worktree_modes() {
git ls-files -s -z | while IFS= read -r -d '' entry; do
metadata="${entry%%$'\t'*}"
path="${entry#*$'\t'}"
mode="${metadata%% *}"
case "$mode" in
100644|100755) [[ -f "$path" && ! -L "$path" ]] || exit 1 ;;
120000) [[ -L "$path" ]] && readlink "$path" >/dev/null || exit 1 ;;
*) exit 1 ;;
esac
done
}
rewrite_index_entry() {
local mode="$1" path="$2" target temporary_link
case "$mode" in
100644)
cp "$REPAIR_DIR/$path" "$path" && chmod a-x "$path"
;;
100755)
cp "$REPAIR_DIR/$path" "$path" && chmod a+x "$path"
;;
120000)
target="$(readlink "$REPAIR_DIR/$path")" || return 1
temporary_link="${path}.thoth-lf-repair-link"
[[ ! -e "$temporary_link" && ! -L "$temporary_link" ]] || return 1
ln -s "$target" "$temporary_link" || return 1
rm -f "$path" || { rm -f "$temporary_link"; return 1; }
mv "$temporary_link" "$path"
;;
*) return 1 ;;
esac
}
if ! git status --short; then abort_repair "git status failed"; fi
if ! git config --local core.autocrlf false; then abort_repair "could not set repository LF policy"; fi
if ! git add --renormalize .; then abort_repair "index renormalization failed"; fi
if ! git diff --cached --check; then abort_repair "normalized index check failed"; fi
if ! git diff --cached; then abort_repair "normalized index review failed"; fi
REPAIR_DIR="$(cd .. && pwd -P)/ThothII-lf-repair"
if [[ -e "$REPAIR_DIR" ]]; then
abort_repair "choose a new empty LF repair directory: $REPAIR_DIR"
fi
if ! mkdir -p "$REPAIR_DIR"; then abort_repair "could not create LF repair directory"; fi
REPAIR_PREFIX="$REPAIR_DIR/"
if ! git checkout-index --all --force --prefix="$REPAIR_PREFIX"; then abort_repair "index export failed"; fi
if ! validate_index_export; then abort_repair "index export is missing entries or Git modes"; fi
if ! bash scripts/verify-line-endings.sh "$REPAIR_DIR"; then abort_repair "exported bytes failed LF verification"; fi
# WARNING: destructive copy; make a backup or commit wanted changes before this command.
if ! git ls-files -s -z | while IFS= read -r -d '' entry; do
metadata="${entry%%$'\t'*}"
path="${entry#*$'\t'}"
mode="${metadata%% *}"
rewrite_index_entry "$mode" "$path" || exit 1
done; then
abort_repair "tracked-file rewrite failed; do not build from this worktree"
fi
if ! validate_worktree_modes; then abort_repair "repaired worktree does not match Git index modes"; fi
if ! bash scripts/verify-line-endings.sh; then abort_repair "repaired worktree failed LF verification"; fi
if ! git diff --cached --check; then abort_repair "repaired index check failed"; fi
```
Native Windows PowerShell runs the same Git operations and invokes the byte verifier through Git
for Windows:
```powershell
$ErrorActionPreference = 'Stop'
function Assert-NativeSuccess([string]$Step) {
if ($LASTEXITCODE -ne 0) { throw "$Step failed with exit code $LASTEXITCODE." }
}
function ConvertFrom-IndexEntry([string]$Entry) {
if ($Entry -notmatch '^([0-9]{6}) [0-9a-f]+ [0-3]\t(.+)$') {
throw "Invalid Git index entry: $Entry"
}
[pscustomobject]@{ Mode = $Matches[1]; Path = $Matches[2] }
}
git status --short
Assert-NativeSuccess 'git status'
git config --local core.autocrlf false
Assert-NativeSuccess 'repository LF policy'
git add --renormalize .
Assert-NativeSuccess 'index renormalization'
git diff --cached --check
Assert-NativeSuccess 'normalized index check'
git diff --cached
Assert-NativeSuccess 'normalized index review'
$RepairDir = Join-Path (Split-Path -Parent (Get-Location).Path) 'ThothII-lf-repair'
if (Test-Path $RepairDir) { throw 'Choose a new empty LF repair directory.' }
New-Item -ItemType Directory -Path $RepairDir | Out-Null
$RepairPrefix = $RepairDir.Replace('\', '/') + '/'
git -c core.symlinks=true checkout-index --all --force --prefix=$RepairPrefix
Assert-NativeSuccess 'index export'
$RawIndexEntries = @(git ls-files -s)
Assert-NativeSuccess 'index inventory'
$IndexEntries = @($RawIndexEntries | ForEach-Object { ConvertFrom-IndexEntry $_ })
foreach ($Entry in $IndexEntries) {
$ExportPath = Join-Path $RepairDir $Entry.Path
$ExportItem = Get-Item -LiteralPath $ExportPath -Force -ErrorAction Stop
switch ($Entry.Mode) {
{ $_ -in '100644', '100755' } {
if ($ExportItem.LinkType -eq 'SymbolicLink') { throw "Regular export became a symlink: $($Entry.Path)" }
}
'120000' {
if ($ExportItem.LinkType -ne 'SymbolicLink') { throw "Symlink export is not mode 120000: $($Entry.Path)" }
if ([string]::IsNullOrWhiteSpace([string]$ExportItem.Target)) { throw "Symlink target is empty: $($Entry.Path)" }
}
default { throw "Unsupported Git mode $($Entry.Mode): $($Entry.Path)" }
}
}
& "C:\Program Files\Git\bin\bash.exe" scripts/verify-line-endings.sh $RepairDir
Assert-NativeSuccess 'exported byte LF verification'
# WARNING: destructive copy; make a backup or commit wanted changes before this command.
foreach ($Entry in $IndexEntries) {
$ExportPath = Join-Path $RepairDir $Entry.Path
switch ($Entry.Mode) {
{ $_ -in '100644', '100755' } {
Copy-Item -LiteralPath $ExportPath -Destination $Entry.Path -Force -ErrorAction Stop
}
'120000' {
$LinkTarget = [string](Get-Item -LiteralPath $ExportPath -Force -ErrorAction Stop).Target
$TemporaryLink = "$($Entry.Path).thoth-lf-repair-link"
if (Test-Path -LiteralPath $TemporaryLink) { throw "Temporary symlink path exists: $TemporaryLink" }
New-Item -ItemType SymbolicLink -Path $TemporaryLink -Target $LinkTarget -ErrorAction Stop | Out-Null
Remove-Item -LiteralPath $Entry.Path -Force -ErrorAction Stop
Move-Item -LiteralPath $TemporaryLink -Destination $Entry.Path -ErrorAction Stop
}
default { throw "Unsupported Git mode $($Entry.Mode): $($Entry.Path)" }
}
}
foreach ($Entry in $IndexEntries) {
$WorktreeItem = Get-Item -LiteralPath $Entry.Path -Force -ErrorAction Stop
switch ($Entry.Mode) {
{ $_ -in '100644', '100755' } {
if ($WorktreeItem.LinkType -eq 'SymbolicLink') { throw "Regular worktree entry became a symlink: $($Entry.Path)" }
}
'120000' {
if ($WorktreeItem.LinkType -ne 'SymbolicLink') { throw "Repaired worktree symlink is not mode 120000: $($Entry.Path)" }
if ([string]::IsNullOrWhiteSpace([string]$WorktreeItem.Target)) { throw "Repaired symlink target is empty: $($Entry.Path)" }
}
default { throw "Unsupported Git mode $($Entry.Mode): $($Entry.Path)" }
}
}
& "C:\Program Files\Git\bin\bash.exe" scripts/verify-line-endings.sh
Assert-NativeSuccess 'repaired worktree LF verification'
git diff --cached --check
Assert-NativeSuccess 'repaired index check'
```
The export inventory must contain every regular mode (`100644`/`100755`) and recreate every tracked
workspace compatibility symlink (`120000`). The first verifier proves the complete
export before any overwrite; every copy/link operation is fail-closed; the final verifier examines
the repaired worktree bytes. On native Windows, creating symlinks requires Developer Mode or an
elevated account; failure stops the rewrite. Review the staged diff again before committing, then
remove the separate repair directory only after inspecting it. The procedure intentionally avoids
`git reset --hard`; replacing the clone is easier to audit and safer for uncommitted work.
docs/migrations/p1-to-p1-1-registry-layout.md
-38
View File
@@ -1,38 +0,0 @@
# P1 to P1.1 registry layout migration
P1.1 is a repository-contract cutover. New ThothII builds reject the old flat layout and a
repository without `thoth-workspaces.yaml`, so migrate the registry in Git first and upgrade the
application only after that reviewed migration commit is pushed.
## One reviewed migration commit
Perform the layout move in a clean review clone and keep it in one reviewed Git commit:
```sh
git mv workspaces/<id>.yaml <id>/workspace.yaml
git mv workspace-content/<id>/evidence <id>/evidence
# create and review thoth-workspaces.yaml from descriptor metadata
```
For every workspace directory, preserve the existing descriptor bytes, move only the embedded
filesystem Evidence tree, and create `thoth-workspaces.yaml` with:
- `schema_version: 1`
- the ordered `workspaces` list
- curator-owned `id`, `name`, and optional `description` copied from the reviewed descriptors
Generated docs remain under `workspace-docs/<id>/`. Do not add an auto-migrator and do not let the
API rewrite the catalog or Evidence tree.
## Cutover order
1. Review the migration commit, including the new `thoth-workspaces.yaml` metadata.
2. Push that commit to the authoritative registry branch.
3. Upgrade ThothII only after that migration commit is pushed.
4. Pull the migrated registry into each installation before using workspace management.
## Rollback
Roll back the application revision and registry commit together. Do not point a P1.1 binary at the
old flat layout, and do not keep a migrated registry commit active while rolling the application
back to pre-P1.1 code.
docs/operations/psd-dwh-auth-rollout.md
-43
View File
@@ -1,43 +0,0 @@
# PSD — rollout controllato DWH REST
Questo runbook completa il [piano di accettazione autenticazione](../plans/2026-08-18-thothii-authentication-acceptance-and-psd-deployment.md) e il [programma di deployment PSD](../plans/2026-08-20-psd-server-deployment-program.md). Gate A e la parte dual-key di Gate B sono stati eseguiti con autorizzazioni separate. L'emendamento del proprietario del 2026-08-21 rinvia collaudo Mac, osservazione e revoca a prima di Project B; non autorizza ulteriori mutazioni.
## Invarianti
- ThothII sul server PSD: `postgres_direct` read-only, senza chiave `dwh-auth`.
- Mac PSD e client remoti: `rest_api`, una chiave per installazione, HTTPS `.it` verificato.
- `dwh-auth` è `systemd` indipendente, non Compose; non fermare o sostituire il vecchio stack ora.
- Sessioni legacy, indici Qdrant e cache Ollama sono dati test: nessuna migrazione o backup per il cutover. Il vecchio stack resta comunque fino a cutover/rollback approvati.
- Usare solo `/dwh/rpc/ping`, mai risultati clinici o catture Nginx grezze.
## Gate A — Task 9, servizio locale senza Nginx pubblico
Richiedere prima autorizzazione per SHA congelato, target, rollback e impatto legacy. Senza consenso, fermarsi e registrare solo `IN_DISCUSSION`.
1. Verificare in sola lettura architettura, gruppo `www-data`, nomi liberi, systemd, `nginx -t`, ping attuale e file legacy regolare `root:root` `0600`; non leggerlo, stamparlo o calcolarne hash.
2. Costruire con `bash scripts/build-dwh-auth.sh --output /tmp/dwh-auth-release`; registrare solo SHA sorgente e checksum binario.
3. Installare binario/unit/tmpfiles come nella [guida server](../install/dwh-auth-server.md): registry `root:dwh-auth` `2750`, lock/record `0640`, socket `dwh-auth:www-data` `0660`.
4. Importare una sola legacy `legacy-shared` dal file protetto e creare `psd-mac-primary` in nuovo file `0600` sotto `/root/dwh-auth-provision/`; mai segreti in argv, ambiente, log o evidenze.
5. Eseguire `dwh-auth check`, `systemd-analyze verify`, avviare l'unità e testare sul socket Unix con file header curl protetti `0600`: v1=204, legacy=204, casuale=401, assente=401.
6. Salvare solo ID pubblici, owner/mode, stato unit/socket, timestamp, checksum binario/config e rollback. Non modificare Nginx in questo gate.
## Gate B — Task 10, Nginx e client
Serve un secondo consenso: presentare file, backup, canale consegna Mac, osservazione ed esiti 2xx/401/503.
1. Creare copie timestampate `root:root` `0600` di `/etc/nginx/sites-available/policlinicosandonato` e file coinvolti; non allegare configurazioni Nginx grezze alle evidenze.
2. Aggiungere solo `/etc/nginx/conf.d/dwh-auth-rate-limit.conf` e route DWH; preservare upstream `http://127.0.0.1:3001/`, mantenere byte-identiche le location vector e rimuovere la chiave prima di PostgREST.
3. Eseguire checker strutturale, scansione segreti con solo `PASS/FAIL` e metadati, installare candidati e `sudo nginx -t`. No raw diff: non eseguire o conservare raw diff, `nginx -T` o dump: il file legacy può contenere la chiave. Se uno fallisce, ripristinare backup prima di reload e registrare FAIL sanitizzato.
4. Dopo consenso fare reload, poi HTTPS `.it` con CA e file header curl protetti 0600: v1=2xx, legacy=2xx, casuale=401, assente=401 su `/dwh/rpc/ping`; guasto autenticatore=503, mai accesso permissivo.
5. **Deferred pre-Project-B:** consegnare al Mac chiave e CA separatamente, verificare fingerprint fuori banda, configurare vault GUI o `API_KEY_FILE`, poi **Validate workspace source** e **Test workspace connections**.
6. **Deferred pre-Project-B:** completare 48 ore di osservazione comprendenti due cicli ETL delle 03:00, quindi revocare `legacy-shared` con ragione `shared-credential-rotation`; v1=2xx post-revoca, legacy=401 post-revoca e journal limitato senza chiavi/digest.
## Rollback e chiusura
Durante dual-key il rollback ripristina solo route/servizio revisionati, verifica `nginx -t` e fa reload autorizzato. Non ripristina chiavi revocate, PostgreSQL, dati legacy o stack. Scatta per TLS, risposte inattese, salute degradata o assenza di consenso.
Activity 1 resta `DEFERRED_PRE_PROJECT_B`: dual-key è attivo, ma PASS richiede ancora v1=2xx
post-revoca, legacy=401 post-revoca, servizio/Nginx validi, log sanitizzati, rollback leggibile e
accettazione owner. Il rinvio non blocca il survey e Project A privato; blocca Project B. Compilare
[evidenza](../testing/evidence/psd-dwh-auth-rollout-report-template.md) e
[collaudo](../testing/dwh-auth-manual-acceptance.md).
docs/operations/psd-server-sol-orchestration-prompt.md
-92
View File
@@ -1,92 +0,0 @@
# Prompt operativo per Sol — deploy ThothII su PSD
## Ruolo
Sei l'orchestratore del deploy di ThothII sul server PSD. Devi guidare il lavoro
in modo incrementale, verificabile e reversibile. Non assumere che una fase sia
completata: richiedi evidenze e applica i gate descritti nei piani.
## Documenti normativi
Leggi prima questi file, in quest'ordine:
1. `AGENTS.md`
2. `PROJECT_STATE.md`
3. `docs/plans/2026-08-20-psd-server-deployment-program-design.md`
4. `docs/plans/2026-08-20-psd-server-deployment-program.md`
5. `docs/plans/2026-08-20-psd-server-survey.md`
6. `docs/plans/2026-08-20-psd-server-project-a-standalone.md`
7. `docs/plans/2026-08-20-psd-server-project-b-authentik.md`
8. `docs/testing/psd-server-project-a-manual.md`
9. `docs/testing/psd-server-project-b-manual.md`
10. `docs/testing/evidence/psd-server-survey-report-template.md`
11. `docs/testing/evidence/psd-server-project-a-report-template.md`
12. `docs/testing/evidence/psd-server-project-b-report-template.md`
In caso di conflitto, prevalgono `AGENTS.md`, `PROJECT_STATE.md` e i piani
specifici delle fasi nell'ordine Survey, Project A, Project B.
## Modelli e delega multi-agent
Verifica quali modelli e quali primitive multi-agent sono realmente disponibili
nell'ambiente. Se disponibili:
- **Sol** mantiene il controllo del piano, dei gate, delle decisioni architetturali,
della sicurezza, di Authentik, Nginx, Supabase e del rollback.
- **Terra** esegue esclusivamente survey e controlli read-only: host, Docker,
checkout, Compose, Nginx, TLS, Aritmolab, Authentik e PostgreSQL.
- **Luna** esegue comandi bounded e verifiche ripetibili: build, compose, `tht`
(`status`, `doctor`, `pi`), smoke test, preprocessing e migrazioni già
autorizzate dal piano.
Se Terra o Luna non sono disponibili, lavora in sequenza con il modello
disponibile. Non simulare agenti inesistenti.
Per ogni incarico delegato specifica sempre: obiettivo, comandi consentiti,
operazioni vietate, evidenze da raccogliere e formato della risposta:
```text
status: PASS | FAIL | BLOCKED
facts: fatti osservati
commands: comandi eseguiti (senza segreti)
evidence: file o output redatti
risks: rischi residui
blockers: impedimenti
```
Non riportare password, token, cookie, client secret o variabili d'ambiente
sensibili nei log o nei report.
Le attività read-only indipendenti possono essere eseguite in parallelo. Tutte
le mutazioni devono essere sequenziali, con checkpoint e verifica prima della
fase successiva. Non parallelizzare stop/start dello stack, build/recreate,
migrazioni, modifiche ad Authentik, Nginx o al bilanciatore.
## Regole inderogabili
1. Inizia soltanto con il survey read-only.
2. Non spegnere, modificare o rimuovere il vecchio ThothII prima del survey e
del backup verificabile.
3. Non procedere a Project A senza un report Survey `PASS`.
4. Project A usa autenticazione locale e deve essere provato completamente prima
di iniziare Project B.
5. Project B con Authentik parte solo dopo il `PASS` esplicito di Project A.
6. Il database Supabase esistente va riusato tramite schemi dedicati; non creare
un nuovo database per isolare il dataset.
7. Il workspace remoto `tht-workspace-psd` resta unico: REST sul Mac e
PostgreSQL diretto sul server. I segreti non vanno in Git.
8. Il link dalla sidebar di Aritmolab deve restare funzionante e il percorso
pubblico finale deve passare dal balancer e da Nginx.
9. Conserva sempre un rollback verso il vecchio stack e verso l'autenticazione
locale finché il cutover non è approvato.
## Prima azione richiesta
Leggi tutti i documenti normativi. Poi avvia **solo Task 1 — Survey** del piano
generale. Produci un report consolidato usando il relativo template, con esito
`GO` o `NO-GO`, senza eseguire modifiche persistenti. Fermati e segnala ogni
credenziale Authentik mancante, permesso insufficiente, ambiguità sul balancer o
discrepanza tra dominio osservato e configurazione di Aritmolab.
Dopo il survey attendi l'approvazione del proprietario prima di eseguire
Project A.
docs/operations/psd-server-survey-remediation-checklist.md
-455
View File
@@ -1,455 +0,0 @@
# PSD Server Survey — Remediation Checklist
## Purpose and authority
Questo documento permette al proprietario e a Sol di discutere, decidere e chiudere uno alla volta
i blocker emersi dal survey read-only del server PSD. È il punto di ripresa operativo tra sessioni:
registra soltanto fatti sanitizzati, decisioni, responsabili e riferimenti a evidenze protette.
Fonti normative:
- `docs/operations/psd-server-sol-orchestration-prompt.md`
- `docs/plans/2026-08-20-psd-server-deployment-program.md`
- `docs/plans/2026-08-20-psd-server-survey.md`
- `docs/plans/2026-08-20-psd-server-deployment-program-design.md`
Questo documento non autorizza modifiche a server, servizi, database, Nginx, load balancer,
Authentik, Aritmolab o repository esterni.
## Program gate
- Program result: `SURVEY_NO_GO`
- Survey report: `/var/tmp/thothii-psd-survey.fJh7DS/survey-report.md`
- Survey report SHA-256: `36461b6c7d1e44d055f24d6919e892b7017352ac9eb99ac67b9ae62f0614f8e7`
- Legacy stack: deve restare acceso e invariato durante la discussione di questa lista
- La preparazione statica di Project A privato è autorizzata; stop del legacy e start del nuovo
restano vietati fino a `SURVEY_GO_PROJECT_A_PRIVATE` e a un consenso di mutazione separato
- Project B remains forbidden fino ai PASS automatico, umano e del proprietario per Project A
## Current activity and resume point
- Current activity: `2`
- Title: Identify accountable owners for the private Project A scope
- Resume from: Activity 2, assign owner/authority for legacy rollback, direct DWH, workspace and Pi/LLM
- Discussion rule: una sola attività può essere `IN_DISCUSSION`
- Allowed states: `PENDING`, `IN_DISCUSSION`, `DEFERRED_PRE_PROJECT_B`, `BLOCKED`, `PASS`
## How to use this checklist
1. Leggere `Current activity` e `Resume from`.
2. Discutere soltanto l'attività corrente.
3. Non inserire password, token, cookie, chiavi private, stringhe di connessione, claim grezzi o
valori di secret.
4. Registrare solo percorsi protetti, owner, mode, timestamp, nomi o ID di oggetti, checksum ed
esiti sanitizzati.
5. Alla fine della discussione aggiornare stato, note, decisione, evidenze, blocker e prossimo
passo.
6. Spostare `Current activity` solo quando il gate dell'attività corrente è soddisfatto oppure il
proprietario decide esplicitamente di parcheggiarla come `BLOCKED`.
7. Non riaprire un'attività `PASS` salvo nuova evidenza che ne invalidi la decisione.
## Activity summary
| ID | Attività | Stato | Responsabile | Prossimo gate |
|---|---|---|---|---|
| 1 | Rotazione controllata della credenziale DWH esposta | `DEFERRED_PRE_PROJECT_B` | Proprietario del progetto | Chiusura obbligatoria prima di Project B |
| 2 | Assegnazione dei responsabili dei componenti condivisi | `IN_DISCUSSION` | Proprietario del progetto | Owner privati A e shared B distinti |
| 3 | Risoluzione del dominio pubblico `.it` oppure `.com` | `BLOCKED` | Unassigned | Necessario per Project B, non per A privato |
| 4 | Topologia e responsabilità del load balancer | `BLOCKED` | Unassigned | Necessario per Project B/route opzionale |
| 5 | Accesso read-only protetto ad Authentik | `BLOCKED` | Unassigned | Necessario per Project B, non per A privato |
| 6 | Accesso catalog-only protetto a PostgreSQL | `BLOCKED` | Unassigned | DWH direct read-only ancora da provare |
| 7 | Confine temporaneo e cleanup del vecchio ThothII | `PASS` | Proprietario del progetto | Retain fino a Project B PASS; cleanup esatto separato |
| 8 | Accesso Git read-only al workspace PSD | `BLOCKED` | Curator da confermare | Checkout/deploy key server mancanti |
| 9 | Metadati Pi e LLM verificabili | `BLOCKED` | Unassigned | Policy e reachability redatte mancanti |
| 10 | Conservazione evidenze e nuovo survey bounded | `PENDING` | Unassigned | Nuovo report e decisione proprietario |
## Activity 1: Rotate or revoke the exposed DWH credential safely
- Status: `DEFERRED_PRE_PROJECT_B`
- Accountable owner: Proprietario del progetto (confermato dall'utente)
- Objective: sostituire o revocare in modo controllato la credenziale DWH comparsa nell'output
interno del survey, senza interrompere consumer legittimi e senza esporne nuovamente il valore.
- Why this is required: la credenziale deve essere considerata compromessa; non può essere usata
come base affidabile per completare il survey o iniziare Project A.
- Ordered actions:
1. identificare il team che gestisce la route DWH, il suo meccanismo di autenticazione e la
custodia del secret;
2. determinare il tipo di credenziale senza leggerla o copiarla in questa checklist;
3. inventariare i consumer tramite riferimenti di configurazione e secret object;
4. scegliere una transizione a doppia credenziale oppure una finestra atomica con rollback;
5. generare e distribuire il nuovo secret attraverso il meccanismo protetto approvato;
6. verificare i consumer autorizzati, l'assenza del nuovo valore nei log e la continuità del
vecchio stack;
7. revocare la vecchia credenziale e provare che non venga più accettata;
8. registrare soltanto evidenze redatte.
- Required redacted evidence:
- owner e autorizzazione della rotazione;
- tipo e identificatore non sensibile della credenziale;
- percorso protetto o secret object, senza contenuto;
- elenco dei consumer aggiornati;
- timestamp e risultati dei test positivi e negativi;
- conferma di revoca della credenziale precedente;
- procedura di rollback e relativo esito.
- Discussion notes:
- la credenziale osservata è stata identificata, senza rileggerne il valore, come chiave statica
`X-API-Key` applicata da Nginx alla route DWH REST `/dwh/`; è distinta dalle password PostgreSQL;
- `/home/chirone/chirone/etl` documenta PostgREST come integrazione HTTP esterna, mentre i processi
ETL e Superset usano PostgreSQL diretto;
- il vecchio ThothII e Chirone WP3 usano PostgreSQL diretto nei profili locali/server; Supabase
Studio è un pannello amministrativo e non appartiene al data-plane applicativo;
- il design approvato resta invariato: il Mac usa `rest_api`, il nuovo ThothII sul server PSD usa
`postgres_direct` con ruolo DWH dedicato e realmente read-only;
- la ricognizione dei repository ha rilevato materiale sensibile hardcoded in file tracciati,
senza riportarne i valori. La bonifica e la rotazione dei segreti coinvolti restano obbligatorie.
- il censimento statico non trova consumer `/dwh/` in ETL, Superset o nel Chirone WP3 attivo:
usano PostgreSQL diretto. Il vecchio container `thothii-core-1` è configurato con
`transport: direct`; i due ThothII restano comunque tecnicamente capaci di usare REST;
- i log Nginx redatti provano 18.523 richieste `/dwh/` dal 23 luglio al 19 agosto 2026:
18.481 hanno user-agent classificato `python-requests` e gli endpoint RPC corrispondono
prevalentemente a introspezione e campionamento Thoth. La sorgente è una sola, privata e
compatibile con un proxy/load balancer; non prova che esista un solo client finale;
- l'ipotesi che il traffico sia generato dal job ETL delle 03:00 è smentita: nella finestra
02:30–03:30 Europe/Rome non compare nessuna richiesta `/dwh/`. Il 99,37% del traffico è
concentrato il 13 agosto tra le 17:38 e le 20:03;
- la firma del 13 agosto corrisponde a undici preprocessing Thoth: undici `list_tables`, e
per ciascuna esecuzione 163 chiamate a ognuno dei tre RPC per-tabella più 1.180 `top_values`,
cioè 1.670 richieste per ciclo. `PROJECT_STATE.md` registra proprio il preprocessing PSD live
del 13 agosto su 163 tabelle, con più rerun e correzioni emerse durante l'esecuzione;
- il DAG ETL `nightly_etl_orchestrator` è schedulato con `0 3 * * *`, ma scrive il DWH tramite
PostgreSQL/`psycopg2` diretto. Nel codice tracciato non chiama `/dwh/`, gli RPC Thoth o
`tht workspace preprocess`, né emerge un trigger indiretto verso Thoth;
- la configurazione Nginx nominalmente attiva accetta una sola chiave tramite confronto letterale
in un endpoint `auth_request`. Non esiste una mappa a più chiavi: la doppia credenziale richiede
un refactor, backup, `nginx -t`, reload e rollback in una fase di mutazione autorizzata.
- il proprietario conferma che il ThothII sul Mac deve continuare a usare REST e che sono previste
molte altre installazioni remote, senza tunnel SSH verso Supabase. `/dwh/` è quindi
un'interfaccia remota stabile e multi-client, non una compatibilità temporanea.
- il proprietario decide di mantenere il certificato TLS corrente. L'endpoint esterno REST
presenta lo stesso certificato self-issued di Nginx, valido fino al 21 giugno 2027 e con SAN
per `supabase-aritmolab.policlinicosandonato.it`; non copre un eventuale dominio `.com`;
- il manuale di installazione deve trattare `TLS_CA_FILE` come necessario per ogni client che
non abbia già quel certificato nel proprio trust store, spiegando consegna affidabile,
verifica del fingerprint, rinnovo e aggiornamento coordinato delle installazioni;
- non esiste un ambiente di test. La rotazione dovrà quindi usare una verifica production-safe:
backup, finestra dual-key, RPC `ping` senza dati clinici, test positivo/negativo e rollback.
- il proprietario approva un componente `dwh-auth` riutilizzabile ma opzionale, incluso nel
repository senza modificare il protocollo dei client portabili o il CLI `tht`;
- su PSD `dwh-auth` avrà un lifecycle `systemd` indipendente dallo stack ThothII e comunicherà
con Nginx tramite socket Unix. Lo stop o la sostituzione di ThothII non dovrà interrompere i
client REST;
- il registro sarà composto da file protetti, versionati e aggiornati atomicamente, con un file
per generazione della chiave. Conterrà digest SHA-256 di segreti casuali da almeno 256 bit e
metadati non sensibili, senza SQLite o nuove dipendenze runtime;
- le chiavi saranno assegnate alle installazioni, non alle persone. Saranno prive di scadenza
predefinita, con scadenza opzionale e revoca manuale;
- creazione e import leggeranno o scriveranno soltanto file protetti. Nessun segreto sarà
accettato come argomento, stampato o inserito in log, JSON, documenti o repository;
- la gestione sarà fail-closed: credenziali non valide riceveranno `401`, mentre guasti del
servizio o del registro saranno mappati a `503` senza fallback permissivo;
- il proprietario conferma che le sessioni, gli indici e le cache del vecchio ThothII erano solo
test e non richiedono migrazione o attività di salvaguardia dedicate. Restano necessari il
rollback della route DWH condivisa e il rispetto del gate esplicito prima di fermare lo stack;
- il proprietario ha revisionato e approvato la specifica scritta. Il runbook eseguibile è in
`docs/operations/psd-dwh-auth-rollout.md`; separa sviluppo e verifica del componente dai due
gate espliciti di mutazione PSD;
- la credenziale condivisa corrente, priva del nuovo identificativo pubblico, sarà l’unico record
temporaneo `legacy_raw` con ID `legacy-shared`. Dopo la revoca non saranno accettate chiavi
prive del formato versionato per installazione.
- Decision: il nuovo ThothII server userà PostgreSQL diretto; il Mac e le future installazioni
remote useranno REST. La chiave condivisa corrente non è un modello finale accettabile: la
rotazione esposta userà una transizione a doppia credenziale e l'architettura target assegnerà
un'identità revocabile distinta a ogni installazione. Supabase Studio non sarà usato come
trasporto. Il proprietario del progetto è accountable per coordinare la rotazione. Il meccanismo
target è il servizio server-only `dwh-auth`, indipendente dallo stack ThothII, con registro a file
e una chiave revocabile per installazione.
- Owner amendment 2026-08-21: Gate A e dual-key Gate B sono eseguiti; il Mac live test, le 48 ore
comprendenti due cicli ETL delle 03:00 e la revoca di `legacy-shared` sono rinviati al gate
obbligatorio prima di Project B. Il rinvio non equivale a PASS.
- Blockers: per chiudere Activity 1 restano il collaudo Mac, l'osservazione completa, la revoca,
v1 positivo post-revoca e legacy `401`.
- Next step: continuare Activity 2–10 per lo scope Project A privato; riaprire Activity 1 prima di
congelare il candidato Project B.
## Activity 2: Identify accountable owners for shared components
- Status: `IN_DISCUSSION`
- Accountable owner: Unassigned
- Objective: associare ogni componente condiviso a una persona o a un team con autorità di lettura,
modifica, approvazione e rollback.
- Why this is required: la leggibilità di una configurazione non implica autorità a modificarla.
- Ordered actions:
1. identificare gli owner di DNS/load balancer, Nginx/certificati, Authentik,
Supabase/PostgreSQL, Aritmolab, workspace Git e backup legacy;
2. registrare il canale di approvazione e la procedura di escalation;
3. confermare separatamente chi può autorizzare Project A e Project B.
- Required redacted evidence: nomi dei team, ruoli, canali operativi e conferme di responsabilità;
nessun contatto personale sensibile.
- Discussion notes: Not discussed
- Decision: No decision recorded
- Blockers: nessun owner condiviso è stato ancora formalmente confermato.
- Next step: compilare ora la matrice distinguendo componenti necessari a Project A privato e
componenti shared/pubblici rinviabili a Project B.
## Activity 3: Resolve the authoritative public origin
- Status: `BLOCKED`
- Accountable owner: Unassigned
- Objective: scegliere sulla base di evidenze l'unica origine pubblica finale tra il dominio `.it`
osservato e il dominio `.com` riportato nel piano.
- Why this is required: callback OIDC, certificato, cookie, Nginx, load balancer e sidebar devono
concordare sulla stessa origine HTTPS.
- Ordered actions:
1. ottenere la dichiarazione autorevole dell'owner DNS/load balancer;
2. verificare record DNS, route, backend, certificato e redirect;
3. confrontare l'origine con la configurazione e la sidebar di Aritmolab;
4. registrare l'origine approvata e le discrepanze da correggere in Project B.
- Required redacted evidence: hostname finale, record/route sanitizzati, SAN del certificato,
destinazione sidebar e approvazione dell'owner.
- Discussion notes: il survey ha osservato `.it`; il piano cita `.com`. La discrepanza è aperta.
- Decision: No decision recorded
- Blockers: owner DNS/load balancer non identificato e origine browser-visible non provata.
- Next step: ottenere la dichiarazione autorevole dopo l'assegnazione degli owner.
## Activity 4: Establish the load-balancer contract
- Status: `BLOCKED`
- Accountable owner: Unassigned
- Objective: documentare il confine effettivo del load balancer e la procedura reversibile per le
route temporanea e finale.
- Why this is required: il survey locale non ha potuto provare owner, backend, health check, TLS,
source range o allowlist.
- Ordered actions:
1. identificare superficie di configurazione e owner;
2. registrare backend, porta, health check, punto TLS e source range verso Nginx;
3. documentare deploy, validazione e rollback;
4. stabilire se una route temporanea può essere limitata agli operatori;
5. definire una prova positiva e una negativa dell'allowlist senza creare ancora la route.
- Required redacted evidence: nomi/ID delle route, backend e health check sanitizzati, ownership,
capacità di allowlist e procedura di rollback.
- Discussion notes: Not discussed
- Decision: No decision recorded
- Blockers: il load balancer non è ispezionabile dalla superficie locale autorizzata.
- Next step: coinvolgere l'owner identificato nell'Activity 2.
## Activity 5: Provide protected read-only Authentik survey access
- Status: `BLOCKED`
- Accountable owner: Unassigned
- Objective: permettere un inventario Authentik bounded e read-only della versione installata.
- Why this is required: applicazioni, provider, flow, mapping, gruppi, service account, permessi API
e procedura di export non sono stati verificati.
- Ordered actions:
1. identificare l'owner Authentik e la procedura di backup/export;
2. predisporre una credenziale read-only o un'esecuzione assistita dall'owner;
3. comunicare solo percorso, owner, mode e usabilità del secret;
4. inventariare nomi/ID e convenzioni senza recuperare secret write-only;
5. confrontare il comportamento con OpenAPI e documentazione della release installata.
- Required redacted evidence: versione, nomi/ID degli oggetti, permission set della credenziale,
riferimento all'export e risultati sanitizzati.
- Discussion notes: Not discussed
- Decision: No decision recorded
- Blockers: nessuna credenziale amministrativa/API utilizzabile è stata stabilita.
- Next step: ottenere dall'owner un meccanismo protetto post-rotazione.
## Activity 6: Provide protected catalog-only PostgreSQL survey access
- Status: `BLOCKED`
- Accountable owner: Unassigned
- Objective: verificare database, schemi, ruoli, grant, migrazioni e PostgREST con sole query di
catalogo.
- Why this is required: il runtime DWH read-only, lo stato di `thoth_sessions` e i confini Supabase
non sono provati.
- Ordered actions:
1. identificare DBA e procedura protetta di connessione;
2. verificare database, utente corrente, schemi e owner;
3. verificare i grant sullo schema `datawarehouse` senza write probe clinici;
4. verificare stato di `thoth_sessions` e migration records;
5. verificare gli schemi esposti da PostgREST;
6. registrare TLS/CA, backup e convenzioni per ruoli migrator/runtime.
- Required redacted evidence: risultati catalogici bounded, nomi dei ruoli, attributi e grant,
schemi PostgREST, riferimento a backup e TLS; nessuna stringa di connessione.
- Discussion notes: il wrapper ETL `ConnectionFactory` ha aperto una sessione dichiarata
read-only e ha eseguito sole query aggregate a `pg_catalog`. Il database è PostgreSQL 15.8;
l'identità disponibile è `postgres`, owner dello schema `datawarehouse`, con `USAGE` e
`CREATE`. Su tutte le 163 relazioni catalogate possiede SELECT e anche tutti i privilegi di
scrittura/DDL tabellari verificati. Nessun nome tabella o dato clinico è stato raccolto.
- Decision: il meccanismo esistente è valido per il survey catalogico, ma è vietato come identità
runtime del nuovo core perché non è least-privilege né read-only.
- Blockers: il DBA deve fornire un ruolo dedicato con soli USAGE/SELECT e una route diretta
certificabile dal nuovo core. Il PostgREST DWH è loopback host su 127.0.0.1:3001 e non prova il
percorso PostgreSQL diretto richiesto da Project A.
- Next step: definire con il DBA ruolo, secret-file protetto, TLS/rete e query di grant da ripetere;
non creare il ruolo durante il survey.
## Activity 7: Bind the disposable legacy boundary and cleanup exclusions
- Status: `PASS`
- Accountable owner: Proprietario del progetto (confermato dall'utente)
- Objective: mantenere il vecchio ThothII solo come confine temporaneo di cutover e rimuovere
esclusivamente le sue risorse dopo il PASS reale di Aritmolab.
- Why this is required: Aritmolab usa oggi i container legacy, ma il proprietario ha dichiarato
sacrificabili sessioni, configurazione e dati del vecchio ThothII.
- Ordered actions:
1. inventariare nuovamente container, image ID, source e data immediatamente prima dello stop;
2. preservare invariati i due network condivisi e l'Evidence ETL esterna;
3. chiudere la route e fermare solo i container legacy nel gate Project A autorizzato;
4. conservare container, immagini, source e data fino al PASS automatico, umano e owner di B;
5. rimuovere poi soltanto gli exact target sotto un'autorizzazione cleanup separata.
- Required redacted evidence: inventario esatto, owner, restart recipe, esclusioni shared, decisioni
Project A/B e manifest finale di cleanup; nessun contenuto di secret.
- Discussion notes: il legacy stack è ancora attivo e invariato. Compose project `thothii` usa
`/home/chirone/ThothII/compose.yaml`; i servizi sono `core` e `frontend`, senza named volume.
Il solo bind applicativo RW è `/home/chirone/thothii-data` (con i bind Pi annidati); Evidence è
un bind RO esterno. Una lettura tar verso `/dev/null` di source e data ha dato
`legacy_backup_readability=PASS`. Il source è circa 1.05 GB e il data bind circa 1.9 MB.
I dry-run Compose passano solo fornendo il path non segreto
`PI_AUTH_FILE=/home/chirone/thothii-data/pi-config/agent/auth.json` insieme a
`--env-file deploy/thothii.env -p thothii -f compose.yaml`; stop individua entrambi i container
e start è sintatticamente valido (non trova container arrestati mentre lo stack è ancora attivo).
- Decision: nessun backup legacy è richiesto. I container, immagini, source e data già presenti
restano il solo rollback temporaneo fino al PASS Project B; non si crea alcun utente host. Le
risorse esclusive potranno essere cancellate dopo il collaudo Aritmolab, mentre network shared,
ETL Evidence, Omics, LocalLLM, DWH, `dwh-auth`, Supabase, Authentik e Superset sono esclusi.
- Blockers: nessuno per questa decisione survey. Stop/start, route change e cleanup restano tre
autorizzazioni di mutazione separate e non sono autorizzati da questo PASS.
- Next step: usare il design e piano clean-replacement approvati; non eseguire ancora mutazioni.
## Activity 8: Provide read-only PSD workspace Git access
- Status: `BLOCKED`
- Accountable owner: Unassigned
- Objective: verificare il repository remoto condiviso e il suo stato corrente dal server senza
capacità di push.
- Why this is required: SHA, descriptor, trasporti, Evidence, annotazioni e scope della deploy key
non sono stati osservati dal server.
- Ordered actions:
1. identificare curator e owner della deploy key;
2. fornire un riferimento protetto alla chiave server read-only;
3. verificare remote, branch e SHA con modalità non interattiva;
4. verificare catalogo, schema v3, trasporti, Evidence e annotazioni;
5. provare che la credenziale server non possa effettuare push.
- Required redacted evidence: remote, branch, SHA, descriptor blob, stato Evidence/annotations e
attestazione read-only della deploy key.
- Discussion notes: Not discussed
- Decision: No decision recorded
- Blockers: nessun checkout workspace o deploy credential utilizzabile è stato localizzato.
- Next step: coinvolgere il curator e predisporre l'accesso server read-only.
## Activity 9: Make Pi and LLM metadata verifiable
- Status: `BLOCKED`
- Accountable owner: Unassigned
- Objective: verificare versione Pi, provider, modello, thinking level, riferimento credenziale e
reachability LLM senza esporre il secret.
- Why this is required: la root Pi legacy non è attraversabile dall'operatore del survey e i
default del nuovo source non provano la configurazione in esecuzione.
- Ordered actions:
1. identificare owner della configurazione Pi/LLM;
2. scegliere tra esecuzione assistita dall'owner e accesso read-only allowlisted;
3. estrarre esclusivamente metadati non sensibili;
4. eseguire un controllo bounded di reachability senza stampare credenziali;
5. registrare anche il mismatch NVML/GPU come rischio separato, non come blocker CPU.
- Required redacted evidence: versione, provider, model ID, thinking level, endpoint sanitizzato,
percorso/mode della credenziale e risultato di reachability.
- Discussion notes: host `x86_64`; due GPU NVIDIA osservate, ma `nvidia-smi` non è utilizzabile per
mismatch driver/libreria NVML. Una lettura whitelist di `settings.json` ha rilevato Pi 0.80.3,
provider `deepseek`, modello `deepseek-v4-pro` e thinking `high`, senza leggere
`auth.json`. Un singolo probe senza tool, contesto o sessione ha prodotto solo
`pi_reachability=FAIL`. Il catalogo custom dichiara inoltre il solo provider `local-qwen`.
- Decision: i metadati sono verificati, ma reachability e coerenza default/catalogo non passano.
- Blockers: diagnosticare il FAIL senza esporre la credenziale e confermare il provider/modello
approvato per Project A; il mismatch NVML/GPU resta rischio separato, non un motivo per assumere
che il percorso CPU funzioni.
- Next step: eseguire un controllo assistito e sanitizzato della configurazione provider, quindi
ripetere una sola reachability probe bounded.
## Read-only resume — 2026-08-21
- Host/source: Linux x86_64, Docker 29.1.1, Compose 2.40.3, application worktree clean at
`7118950416b3008a8182825de027c7f8b235de57`; Qdrant/Ollama images are local, while the required
embedding image/model is not yet proved local.
- Capacity: approximately 1.1 TB free on `/home` and 36 GB on `/`; several loopback candidate
ports are currently free. These facts do not reserve a path or port.
- Legacy: project `thothii` is still running and unchanged. Source is
`/home/chirone/ThothII` at `6ca4275`; the only RW application bind is
`/home/chirone/thothii-data`, plus the nested Pi binds. The source is about 1.05 GB and the data
bind about 1.9 MB. No backup was created.
- Recovery decision: legacy state is disposable; no backup is required. Existing stopped
containers, images, source and data remain only as the temporary Project A/B rollback boundary.
- Identity decision: neither UID nor GID 10001 maps to a host account. No host identity will be
created. The new image retains numeric `10001:10001`, confined to its distinct writable roots;
the existing operator owns source/configuration. Recheck both `getent` lookups before creating
paths and stop on any new mapping.
- Shared exclusion: never remove the Omics/LocalLLM networks or external ETL Evidence bind.
- Candidate paths: documented examples `/srv/thothii` and `/srv/thothii-backups` are absent and
therefore only candidates; they have not been created. `127.0.0.1:18080` è il candidato
frontend e risultava libero al momento del survey, ma non è riservato e va ricontrollato prima
dello start. Existing `/home/chirone/thothii-data` must not be reused.
- Workspace: the canonical private remote is documented as
`git@github.com:mptyl/tht-workspace-psd.git`; a non-interactive read-only remote query resolved
`main` at `bfbabf9f2defcf861a3225296eaff8c6d44c0ac9`. No server checkout or dedicated deploy-key
reference is present at the documented local paths, and the credential's inability to push is
not yet proved.
- Pi/LLM: legacy Pi version `0.80.3` is visible, but provider/model/thinking/credential reference
and bounded reachability remain unknown.
- Shared scope: `.it` resolves locally and `.com` does not, Nginx is valid/active, Authentik
2026.2.1 and Supabase components are running. Owner, LB contract, Authentik inventory and
PostgreSQL catalog grants remain unproved and are not inferred.
- Decision: `SURVEY_NO_GO` for Project A private remains. The bounded work authorized now is
limited to planning/static preparation; no source clone, protected tree, backup, stop or start
has been performed.
## Activity 10: Retain evidence and run the missing bounded survey checks
- Status: `PENDING`
- Accountable owner: Unassigned
- Objective: conservare le evidenze protette, ripetere soltanto i controlli mancanti e produrre una
nuova decisione verificabile.
- Why this is required: il report corrente è `SURVEY_NO_GO` e non può essere promosso per inferenza.
- Ordered actions:
1. scegliere il protected evidence root definitivo;
2. trasferire la directory del survey senza modificarne i contenuti e verificare il digest;
3. confermare che le Activity 1–9 siano `PASS` oppure abbiano una risoluzione proprietario
esplicitamente accettata;
4. eseguire solo i controlli bounded mancanti del piano survey;
5. aggiornare il report e verificarne checksum e secret hygiene;
6. chiedere la decisione esplicita del proprietario.
- Required redacted evidence: percorso finale, digest, matrice Activity 1–9, nuovi risultati
bounded, report aggiornato e decisione firmata.
- Discussion notes: Not discussed
- Decision: No decision recorded
- Blockers: dipende dalla chiusura delle Activity 1–9 e dall'approvazione del retention root.
- Next step: avviare soltanto dopo la chiusura dei blocker precedenti.
## Fresh survey and owner gates
Un nuovo `SURVEY_GO_PROJECT_A_PRIVATE` richiede:
- owner e autorità di stop/start/rollback identificati per il legacy e Project A;
- accessi read-only PostgreSQL, workspace Git e Pi/LLM verificati;
- identità DWH dimostrata read-only;
- inventario, restart recipe e cleanup exclusions legacy verificabili;
- risorse e percorsi della nuova installazione approvati;
- report redatto, secret-scan valido e checksum verificato;
- approvazione esplicita del proprietario.
`SURVEY_GO_PROJECT_B` richiede inoltre:
- collaudo Mac `rest_api` con chiave per installazione;
- 48 ore di osservazione comprendenti due cicli ETL delle 03:00;
- credenziale `legacy-shared` revocata, v1 positiva e legacy `401`;
- owner e autorità di modifica/rollback per tutti i componenti shared;
- origine pubblica unica e load-balancer contract provati;
- accesso read-only Authentik e inventario della release installata;
- ogni altro blocker pubblico/shared delle Activity 2–9 chiuso.
Il PASS tecnico del survey non autorizza automaticamente Project A. L'autorizzazione deve essere
registrata separatamente.
## Change log
| Data | Attività | Modifica | Autore |
|---|---|---|---|
| 2026-08-20 | Initial | Creata checklist; Activity 1 aperta, Activity 2–10 pending | Sol |
| 2026-08-21 | Sequencing amendment | Activity 1 deferred pre-B; survey ripreso read-only; Project A private resta NO-GO | Owner/Sol |
| 2026-08-21 | Clean replacement | Activity 7 PASS; legacy disposable dopo B; nessun account host 10001 | Owner/Sol |
docs/plans/2026-08-08-internal-qdrant-ollama-design.md
-210
View File
@@ -1,210 +0,0 @@
# Internal Qdrant and Ollama Architecture Design
**Status:** approved on 2026-08-08
## Objective
ThothII owns its semantic infrastructure. Every supported deployment includes a private Qdrant
service and a private Ollama embedding service. The analytical DWH remains external and read-only;
each workspace descriptor associates that DWH with one Qdrant collection used for database schema,
Evidence, and approved Memory records.
## Decisions
- Qdrant replaces pgvector as the only operational vector store.
- Ollama replaces workspace-selected external embedding endpoints.
- The default and required model is `qwen3-embedding:0.6b` with 1024-dimensional normalized dense
embeddings and cosine distance.
- One Qdrant collection belongs to one workspace. Schema, Evidence, and Memory points share that
collection and are separated by indexed payload field `kind`.
- Qdrant and Ollama are mandatory base-Compose services. They are not published on host ports and
are reachable only from the private Compose network.
- Existing schema-v1 and schema-v2 descriptors remain readable for migration, but they are not
activatable. The new operational contract is workspace schema v3.
The model choice is based on the published Qwen model card: the 0.6B model supports more than 100
languages, a 32K context window, Matryoshka dimensions up to 1024, and instruction-aware retrieval.
Ollama distributes a CPU-viable quantized build and can use an exposed GPU without changing the
application protocol.
References:
- <https://huggingface.co/Qwen/Qwen3-Embedding-0.6B>
- <https://ollama.com/library/qwen3-embedding>
- <https://docs.ollama.com/capabilities/embeddings>
- <https://qdrant.tech/documentation/installation/>
- <https://qdrant.tech/documentation/manage-data/collections/>
## Target topology
```text
browser -> frontend -> core -> external DWH
-> private Qdrant
-> private Ollama embedding
```
The base Compose project contains:
- `frontend`: static React application and same-origin API proxy.
- `core`: Fastify, Pi, and the Python `tht` harness.
- `qdrant`: pinned Qdrant server with persistent `qdrant-data` volume.
- `embedding`: pinned Ollama server with persistent `embedding-models` volume.
- `embedding-model-init`: bounded one-shot service that pulls and verifies
`qwen3-embedding:0.6b`; `core` starts only after it succeeds.
`qdrant` and `embedding` use `expose`, not `ports`. The core receives installation-owned internal
URLs:
```text
THT_INTERNAL_QDRANT_URL=http://qdrant:6333
THT_INTERNAL_EMBEDDING_URL=http://embedding:11434
THT_INTERNAL_EMBEDDING_MODEL=qwen3-embedding:0.6b
THT_INTERNAL_EMBEDDING_DIMENSIONS=1024
```
These are deployment facts, not workspace connector bindings. The runtime rejects non-loopback or
non-Compose-service hosts when these variables are overridden for development.
An optional Linux GPU override exposes an available NVIDIA/AMD device to Ollama. The base profile
must remain CPU-safe. macOS Docker remains CPU-only because Docker Desktop cannot expose the Apple
GPU to an Ollama container.
## Workspace schema v3
The workspace itself is the association between the external database and the internal collection:
```yaml
workspace:
schema_version: 3
id: psd-clinical
name: PSD Clinical
language: it
dwh:
engine: postgres
database: postgres
schema: datawarehouse
supported_transports: [postgres_direct]
semantic_index:
vector_store:
engine: qdrant
collection: psd-clinical
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
allowed: [zai/glm-5.2]
```
Invariants:
- the collection name is an explicit portable identifier;
- active workspaces cannot share a collection;
- vector and embedding dimensions are both 1024;
- distance is `cosine`;
- provider and model are exactly the supported internal values;
- no vector transport, vector credential, embedding URL, or embedding credential may appear in a
schema-v3 descriptor or installation contract;
- DWH connectors remain installation-local and can still use the supported external DWH transports.
Schema-v1/v2 pgvector descriptors are listed as `migration_required`. Migration creates a reviewed
schema-v3 document; it does not copy vector data implicitly. Existing semantic data is rebuilt from
the canonical schema documents, Evidence corpus, and Memory registry.
## Qdrant data model
Each point has a deterministic UUIDv5 derived from:
```text
workspace_id + kind + record_key
```
The vector is the 1024-dimensional Ollama result. The payload is:
```json
{
"workspace_id": "psd-clinical",
"kind": "schema",
"source_id": "datawarehouse.patients",
"record_key": "schema:table:datawarehouse.patients",
"content_hash": "sha256:...",
"workspace_revision": "<git commit>",
"generation": "<optional corpus generation>",
"language": "it",
"text": "...",
"metadata": {}
}
```
`kind`, `source_id`, `content_hash`, `workspace_revision`, and `generation` receive keyword payload
indexes. Queries always filter by `workspace_id` and an explicit allowed `kind` set. Upsert is
idempotent. Evidence generation deletion is an exact filtered delete. Collection creation is also
idempotent and fails closed if an existing collection has incompatible dimensions or distance.
## Harness integration
The existing `VectorStore` port remains the workflow boundary. A `QdrantVectorStore` adapter maps
its operations to Qdrant REST endpoints while preserving current schema/Evidence/Memory call sites.
The existing Ollama embedding client is narrowed to the internal `/api/embed` contract and verifies:
- configured model exists;
- output count matches input count;
- every vector has 1024 finite numeric values;
- no remote URL or API key is accepted.
The JSONL Memory registry and persisted phase documents remain canonical. Qdrant remains a derived,
rebuildable semantic index. Schema, Evidence, and Memory ingestion all use the same point builder,
content hashing, and retry policy.
## Readiness and failure behavior
Readiness is layered:
1. Compose waits for Qdrant health.
2. Compose waits for Ollama health and successful model initialization.
3. Workspace activation validates the schema-v3 contract.
4. Harness readiness ensures the Qdrant collection and checks its vector configuration.
5. Harness embeds a bounded probe and verifies 1024 dimensions.
Failures are sanitized and fail closed:
- unavailable Qdrant -> `workspace_not_activatable` before session persistence;
- unavailable or missing Ollama model -> `model_unavailable` before session persistence;
- collection mismatch -> `semantic_index_incompatible` without recreating or deleting data;
- embedding dimension mismatch -> no point write;
- partial batch failure -> operation reports failure and remains safe to retry.
No health response, API response, or diagnostic log exposes DWH credentials or indexed text.
## Deployment and migration
The pgvector deployment path is retired:
- remove local-vector Compose overlays and pgvector bootstrap/migration services;
- remove vector PostgreSQL role and password contracts;
- remove runtime support for vector REST/SSH and external embedding URLs;
- keep only the descriptor parser and migration code needed to recognize legacy workspaces;
- update local/server manuals, examples, smoke tests, CI coupling scans, backup instructions, and
release gates for four persistent stores plus Qdrant and Ollama volumes.
Qdrant backup/restore uses collection snapshots or the persistent volume according to the operator
manual. Ollama model storage is a cache: it may be backed up for offline recovery but is not an
application source of truth.
## Acceptance criteria
- Base local and server Compose renders include healthy private `qdrant` and `embedding` services.
- A clean CPU-only installation downloads the model, creates a workspace collection, and embeds a
probe without external vector or embedding configuration.
- GPU override uses the same API and persistent model volume.
- Schema-v3 workspaces activate; schema-v1/v2 workspaces report `migration_required`.
- Two workspaces cannot claim the same Qdrant collection.
- Schema, Evidence, and Memory records coexist in one collection and remain filter-isolated.
- Existing workflow behavior and persisted session contracts remain unchanged.
- Tests reject all active pgvector deployment, external vector binding, and external embedding
configuration paths.
docs/plans/2026-08-14-read-only-workspace-runtime-secrets-design.md
-189
View File
@@ -1,189 +0,0 @@
# Read-only Workspace Repository and Runtime Secrets Design
**Date:** 2026-08-14
**Status:** Approved
## Purpose
ThothII consumes workspaces from one administrator-configured Git repository. Workspace authors
prepare and publish source outside ThothII. The application fetches, validates, and activates
repository revisions, but never edits, commits, pushes, imports, or exports workspace source.
Runtime credentials are intentionally absent from Git. After a workspace has been read, ThothII
derives the required credentials from its connector and authentication choices and lets an
authorized user complete them in the web application. The values are encrypted and persisted by
the backend; the browser retains neither workspace content nor secrets.
## Ownership boundaries
### Workspace source
The workspace source is an ordinary directory maintained outside the ThothII runtime. It contains
the catalog, each `workspace.yaml`, curated evidence, annotations, and other repository-owned
content. Authors validate it using source-side tooling and publish it through their normal Git
workflow to GitHub, GitLab, Gitea, or another standards-compatible server.
### ThothII installation
The installation descriptor selects the Git remote, branch, and one read-only authentication
transport. SSH uses a read-only deploy key plus pinned known hosts. HTTPS uses a read-only deploy
token and may provide a private CA. Secret values remain outside versioned configuration.
The installer performs a sanitized `git ls-remote` preflight. Credentials embedded in a remote URL
are rejected. The API exposes only a normalized repository identity: host, repository path, branch,
transport, active commit, and synchronization state.
### ThothII runtime
The local Git checkout, candidate validation area, immutable snapshots, and active state are
application-owned. They are read-only from the workspace-management API. A pull fetches a candidate
revision, validates the complete repository, and atomically activates it only if valid. A failed
candidate never replaces the last valid active revision.
ThothII never generates or reconciles files back into the checkout and never invokes Git commit or
push. Generated operational artifacts live under application data, not in the source repository.
## Repository synchronization states
A repository refresh has these states:
- `syncing`: fetching and validating a candidate revision;
- `active`: the candidate passed validation and became the active immutable revision;
- `invalid_candidate`: Git succeeded but repository validation failed; the previous revision stays active;
- `unavailable`: Git or authentication failed; the previous revision stays active;
- `empty`: no valid revision has ever been activated.
Validation is atomic at repository-commit level. A malformed catalog, descriptor, evidence tree, or
cross-file reference rejects the complete candidate revision.
## Runtime secret model
### Requirement discovery
The workspace descriptor contains connector type, authentication method, and non-secret logical
configuration. It never contains secret values or host filesystem paths. Connector adapters define
the secret fields required by each supported authentication method. For example:
- PostgreSQL `username_password` requires `username` and `password`;
- REST `bearer` requires `api_key`;
- SSH tunnel authentication requires the connector password and SSH private key;
- Evidence HTTP signed URLs and static S3 credentials contribute their own secret requirements.
Requirements have stable identifiers scoped by workspace and connector. Labels, descriptions,
input kinds, and required/optional status come from trusted application code rather than repository
HTML or executable metadata.
### Persistent encrypted store
The backend owns a `WorkspaceSecretStore` abstraction. The first implementation is a local encrypted
vault in application-managed persistent storage. Each secret is encrypted with authenticated
encryption and bound to its installation, workspace, connector, and field identifier as associated
data. Plaintext values never appear in Git, API responses, logs, error messages, diagnostics, or
browser storage.
The installation bootstraps one vault key independently from workspace content. Deployment tooling
owns its platform-specific provisioning; the workspace schema and GUI never contain filesystem
paths. The storage interface allows a future Vault, cloud secret manager, or OS keychain provider
without changing workspace descriptors or API consumers.
When an existing file-oriented harness connector needs a credential, the backend materializes it as
a restrictive temporary file in an application-owned runtime directory. Its lifetime is tied to the
diagnostic or runtime lease and it is removed on release. Persistent storage contains ciphertext
only.
### Secret API
For a selected workspace the API returns requirement metadata and status only:
```json
{
"workspaceId": "psd-clinical",
"state": "configuration_required",
"requirements": [
{
"id": "dwh.password",
"connector": "dwh",
"label": "Database password",
"input": "password",
"required": true,
"configured": false
}
]
}
```
A write request contains values only for the selected requirement identifiers. The response returns
status, never values. A delete operation forgets a configured value. Authorization is deliberately
deferred; the current authenticated application user may manage runtime workspace secrets.
Workspace readiness is derived as follows:
- `invalid`: repository structure or descriptor is invalid;
- `configuration_required`: structurally valid but required runtime values are missing;
- `ready`: required values exist but connectivity has not yet passed or is stale;
- `verified`: the most recent connector diagnostic passed for the active revision and current secret generation.
Changing or deleting a secret invalidates the previous diagnostic result.
## Browser behavior
Workspace management is a two-level read-only interface occupying at least 60 percent of viewport
width and height.
Level 1 explains the source/runtime separation and displays:
- normalized repository host and path;
- configured branch and read-only transport;
- active revision and last synchronization result;
- `Update workspace repository`, which fetches, validates, and conditionally activates a revision;
- the workspace list, with selection required for workspace-specific actions.
There is no Import bundle, Export bundle, Create, Edit, Delete, Publish, or conflict-resolution
operation. There are no browser-persisted workspace drafts or preferences.
Level 2 for the selected workspace explains and displays:
- immutable source identity and validation result;
- required runtime configuration grouped by connector;
- secret-entry controls whose values are write-only;
- `Save secrets`, `Forget` per configured value, and `Test workspace connection`;
- clear consequences for each button and a reminder that source changes must be committed and pushed
by an author outside ThothII before repository update.
The browser keeps form values only in component memory and clears them after submission or dialog
close. It never receives saved secret values.
## Compatibility and migration
Existing Git author settings, publish endpoints, bundle endpoints, generated-document
reconciliation, bootstrap catalog slots, and browser draft storage are removed. Existing environment
bindings may be read during a bounded migration period only to seed non-secret connector values;
secret file paths are not part of the new public workspace contract.
Session manifests continue to pin an immutable validated workspace revision. An already running
session keeps its acquired runtime lease; new or resumed work resolves the current encrypted secret
generation and fails closed when required credentials are unavailable.
## Failure handling and security
- Repository and vault errors use stable sanitized codes and never echo remotes with user info,
credential paths, secret identifiers that are not safe to disclose, or secret values.
- Vault writes are atomic and authenticated; corrupted ciphertext fails closed.
- Secret comparison uses no read API. Updating a secret is always a blind replacement.
- The backend applies request-size and field-count limits and rejects unknown requirement IDs.
- Temporary plaintext files use restrictive permissions, trusted directories, no-follow opens, and
deterministic cleanup.
- Git credentials are installation-only, read-only, and never sent to the frontend.
## Verification
Backend tests cover repository read-only behavior, atomic candidate activation, remote sanitization,
vault encryption and corruption, requirement discovery, blind secret writes/deletes, materialization
cleanup, readiness transitions, and absence of publish/bundle routes.
Frontend tests cover the two-level explanation, viewport dimensions, repository identity, selection
gating, dynamic secret forms, write-only behavior, status changes, and absence of local-storage,
import, export, editing, and publishing controls.
Deployment and CLI tests cover required remote/branch configuration, one read-only Git transport,
sanitized remote preflight, vault-key provisioning, and removal of Git author/write configuration.
docs/plans/2026-08-18-thothii-authentication-acceptance-and-psd-deployment.md
-408
View File
@@ -1,408 +0,0 @@
# ThothII Authentication Acceptance and PSD Deployment Plan
> **For agentic workers:** execute one task at a time, record its completion criteria, and stop at every documented approval boundary.
**Goal:** Validate local and OIDC authentication on macOS, deploy the exact feat/thoth-auth candidate to the Aritmolab/PSD server before merging it into main, and complete end-to-end acceptance with remote Authentik.
**Architecture:** Test the candidate first as a standalone local installation. Then install the same immutable Git revision on the existing PSD installation with the installation-aware tht lifecycle, leaving main untouched. Authentik provides OIDC login and a mandatory direct groups claim; ThothII maps exact external groups to roles and validates mapped groups through the Authentik catalog API.
**Tech Stack:** macOS, Docker Desktop, Docker Compose, native host `tht` plus Python workflow `tht`, local Argon2id authentication, generic OIDC Authorization Code + PKCE, Authentik, PSD workspace registry, reverse proxy/TLS.
---
## Scope and release rules
Do not merge feat/thoth-auth into main until every mandatory gate in Task 9 is PASS and the PSD owner accepts the evidence.
Capture one candidate revision and reuse it everywhere:
```bash
export CANDIDATE_SHA="$(git rev-parse HEAD)"
git fetch origin feat/thoth-auth
test "$CANDIDATE_SHA" = "$(git rev-parse origin/feat/thoth-auth)"
git show -s --format='%H%n%P%n%s' "$CANDIDATE_SHA"
git status --short --untracked-files=all
```
Never deploy a moving branch name without checking its resolved SHA. Never put passwords, OIDC client secrets, Authentik API tokens, cookies, authorization headers, raw ID tokens, or password hashes in Git, shell history, screenshots, logs, or evidence.
Use protected operator values for <PUBLIC_URL>, <OIDC_ISSUER>, <AUTHENTIK_BASE_URL>, <OIDC_CLIENT_ID>, <THT_BIN>, <INSTALLATION>, <WORKSPACE_ID>, and <OLD_SHA>.
The Authentik contract is mandatory: a direct non-empty JSON array claim named groups; exact groups TOT Users and TOT Admin; mappings TOT Users -> user and TOT Admin -> admin; and a separate group-view-only API service account exposed only as THT_AUTHENTIK_API_TOKEN. Extra upstream groups are valid and silently ignored.
## Task 0: Freeze the candidate and collect approvals
**Files:** None; record results in the acceptance report in Task 9.
Run from the candidate worktree:
```bash
git diff --check
go test ./... -count=1
go test -race ./...
go vet ./...
go build ./...
```
Expected: all commands pass, the candidate is pushed, and existing evidence is bound to the same SHA. A historical result from another revision is not evidence for this run.
Before touching PSD, obtain the maintenance window, server access, public URL, Authentik provider details, protected secret locations, test identities for ordinary/admin/unmapped users, and permission to test PSD DWH/Evidence connections.
## Task 1: Prepare and start the local macOS installation
**Files:**
- Read: docs/install/local.md
- Read: docs/install/authentication-local.md
- Read: docs/testing/authentication-manual-acceptance.md
- Use: an untracked local installation descriptor and protected secret/password files
**Step 1: Verify prerequisites**
```bash
docker version
docker compose version
bash scripts/verify-line-endings.sh
```
Expected: Docker Desktop and Compose are available and line-ending validation passes.
**Step 2: Build and configure**
```bash
bash scripts/build-local.sh
bash scripts/build-tht.sh
tht setup --profile local
```
For an existing installation, do not overwrite data; run tht --installation <local-installation.yaml> update --check-only instead of setup.
**Step 3: Start and inspect**
```bash
tht --installation <local-installation.yaml> start --build
tht --installation <local-installation.yaml> status
tht --installation <local-installation.yaml> doctor --json
curl --fail http://127.0.0.1:8080/health
curl --fail http://127.0.0.1:8787/health
```
Expected: core, frontend, qdrant, embedding, and the completed model initializer are healthy; doctor includes authentication after configuration and before services.
## Task 2: Configure and test local login
**Files:**
- Read: docs/install/authentication-local.md
- Modify only protected installation state through tht auth configure and tht auth user
**Step 1: Bootstrap the administrator**
```bash
tht --installation <local-installation.yaml> auth configure \
--mode local --public-url http://127.0.0.1:8080 \
--admin-user <local-admin> --admin-display-name <display-name> \
--password-file <protected-password-file>
```
Remove the temporary password file immediately. Expected: non-secret auth.yaml is created and the user store contains Argon2id hashes, never plaintext passwords.
**Step 2: Add and inspect a normal user**
```bash
tht --installation <local-installation.yaml> auth user add <local-user> --role user --display-name <display-name> --password-file <protected-password-file>
tht --installation <local-installation.yaml> auth status --json
tht --installation <local-installation.yaml> auth check --json
```
Expected: pristine redacted JSON and no credential, hash, or session secret in output.
**Step 3: Test browser authorization**
At http://127.0.0.1:8080, in a private browser profile:
1. Verify unauthenticated access reaches login and protected routes are denied.
2. Log in as the normal user and verify application/session routes work.
3. Verify Pi Management and other admin-only operations return HTTP 403 or are not exposed.
4. Log out and verify the session is invalidated.
5. Log in as the administrator and verify admin-only routes work.
Expected: ordinary users authenticate without receiving admin permissions; administrators receive the configured admin permission set.
**Step 4: Test account failure paths**
Use auth user disable, enable, set-password, and logout-all user --yes on the test user. Test a wrong password and refresh the old browser session after logout-all.
Expected: generic safe failures, disabled login rejection, re-enabled login success, and forced reauthentication. The last enabled administrator cannot be disabled or demoted.
## Task 3: Test remembered sessions and local recovery
**Files:**
- Read: docs/architecture/authentication.md, Browser sessions
- Read: docs/install/authentication-local.md, Session behavior and recovery
**Step 1: Test browser restart**
Log in as the normal user with Remember me, close the browser completely, reopen it, and revisit the application.
Expected: the session survives within the 7-day idle / 30-day absolute limits. Do not record the cookie.
**Step 2: Test ThothII restart**
```bash
tht --installation <local-installation.yaml> stop
tht --installation <local-installation.yaml> start
```
Expected: the remembered session remains valid after backend restart.
**Step 3: Test invalidation**
Change the test user password or role, and separately run auth user logout-all user --yes. Refresh after each operation.
Expected: affected sessions are rejected and reauthentication is required; configuration revision changes invalidate all sessions.
Go/no-go: do not proceed to PSD if local login, role separation, logout, or remembered-session behavior fails.
## Task 4: Snapshot the current PSD installation
**Files:**
- Read: docs/install/server.md
- Read: docs/install/server-workspace-registry.md
- Use: protected server operator and backup locations
**Step 1: Capture live state**
```bash
THT_BIN=<THT_BIN>
INSTALLATION=<INSTALLATION>
"$THT_BIN" --installation "$INSTALLATION" status
"$THT_BIN" --installation "$INSTALLATION" doctor
"$THT_BIN" --installation "$INSTALLATION" pi status
"$THT_BIN" --installation "$INSTALLATION" pi doctor
git -C /srv/thothii/source/ThothII status --short --untracked-files=all
git -C /srv/thothii/source/ThothII rev-parse HEAD
```
Save the live SHA as <OLD_SHA> and capture image identities, workspace registry status, and maintenance/recovery state. Stop if the checkout is dirty or recovery is pending.
**Step 2: Drain and back up**
Announce maintenance, close the reverse proxy or show its maintenance page, drain active work, and stop through tht. Create the protected, checksummed backup specified in docs/install/server.md, including runtime trees and PSD PostgreSQL/session data where applicable. Back up credentials separately. Never run docker compose down --volumes.
**Step 3: Check preconditions**
```bash
git -C /srv/thothii/source/ThothII config --local core.autocrlf false
bash /srv/thothii/source/ThothII/scripts/verify-line-endings.sh
"$THT_BIN" --installation "$INSTALLATION" update --check-only
```
Expected: descriptor, protected secrets, Pi-state mount, workspace repository binding, and Compose render remain valid before source changes.
## Task 5: Deploy the feature revision to PSD without merging main
**Files:**
- Server source checkout: /srv/thothii/source/ThothII
- Server operator binary: protected THT_BIN path
- Server installation descriptor and secret files: unchanged paths unless a reviewed auth update is required
**Step 1: Select the exact candidate**
```bash
git -C /srv/thothii/source/ThothII fetch origin feat/thoth-auth
git -C /srv/thothii/source/ThothII switch --detach <CANDIDATE_SHA>
test "$(git -C /srv/thothii/source/ThothII rev-parse HEAD)" = "<CANDIDATE_SHA>"
git -C /srv/thothii/source/ThothII status --short --untracked-files=all
```
Do not merge or rebase main. The running installation is intentionally based on the detached feature revision until acceptance completes.
**Step 2: Build candidate artifacts**
```bash
cd /srv/thothii/source/ThothII
bash scripts/build-local.sh
THT_THT_OUTPUT_DIRECTORY=/srv/thothii/operator/build-output bash scripts/build-tht.sh
```
Install the architecture-appropriate candidate tht only after its build succeeds. Keep the old operator binary recoverable.
**Step 3: Start and verify the candidate**
```bash
"$THT_BIN" --installation "$INSTALLATION" update --check-only
"$THT_BIN" --installation "$INSTALLATION" start --build
"$THT_BIN" --installation "$INSTALLATION" status
"$THT_BIN" --installation "$INSTALLATION" doctor --json
curl --fail http://127.0.0.1:8080/health
"$THT_BIN" --installation "$INSTALLATION" pi doctor
"$THT_BIN" --installation "$INSTALLATION" pi test
```
Expected: candidate frontend/core and internal services are healthy, no data volume was replaced, and the candidate SHA is recorded. Liveness alone is not release approval.
## Task 6: Configure and validate remote Authentik
**Files:**
- Modify protected server authentication state through tht auth configure
- Modify protected secret entries THT_OIDC_CLIENT_SECRET and THT_AUTHENTIK_API_TOKEN
- Read: docs/install/authentik.md and docs/install/authentication-oidc.md
**Step 1: Verify Authentik**
Verify the OAuth2/OIDC callback exactly <PUBLIC_URL>/api/auth/oidc/callback, scopes openid/profile/email, direct groups array mapping, exact groups TOT Users and TOT Admin, and a separate group-view-only catalog service account. Inspect a disposable identity without copying its token.
**Step 2: Configure the ThothII mapping**
```bash
tht --installation "$INSTALLATION" auth configure \
--mode oidc --public-url <PUBLIC_URL> \
--issuer <OIDC_ISSUER> --client-id <OIDC_CLIENT_ID> \
--authentik-base-url <AUTHENTIK_BASE_URL> \
--user-group 'TOT Users' --admin-group 'TOT Admin'
```
Secrets are read from protected files, never command-line arguments. Confirm the non-secret mapping is:
```yaml
authorization:
groupRoles:
TOT Users: [user]
TOT Admin: [admin]
```
**Step 3: Run static and live checks**
```bash
tht --installation "$INSTALLATION" auth status --json
tht --installation "$INSTALLATION" auth check --json
tht --installation "$INSTALLATION" auth check --interactive
tht --installation "$INSTALLATION" doctor --json
```
Expected: configuration, discovery, issuer, JWKS, client-secret access, catalog access, and exact existence of every mapped group pass. Doctor lists authentication after configuration and before services. No output contains credentials or bearer tokens.
A missing mapped group must fail with redacted oidc_mapped_group_missing. Unmapped groups produce neither error nor warning. Missing, indirect, malformed, or overage-style groups claims fail closed.
**Step 4: Reload if required**
If configuration requires process reload:
```bash
"$THT_BIN" --installation "$INSTALLATION" pi restart --yes --drain
```
Repeat authentication, doctor, and health checks. Do not substitute raw Compose commands.
## Task 7: Test PSD browser login and authorization
**Files:**
- Read: docs/testing/authentication-manual-acceptance.md
- Evidence: redacted report from Task 9
**Step 1: Ordinary user**
In a private profile, authenticate with an identity in TOT Users but not TOT Admin. Verify callback success, application/session routes, denial of Pi Management/admin operations, opaque HttpOnly ThothII cookie, no bearer token in Web Storage, and logout invalidation.
**Step 2: Administrator**
Authenticate with TOT Admin. Verify Pi Management and allowed workspace-management operations. Access must derive from the exact mapped group, not a client-supplied header or browser-local flag.
**Step 3: Unmapped and malformed groups**
Authenticate with a valid token containing no mapped group. Expected: login may complete, but protected operations return 403 with no warning. Use a disposable provider mapping that omits or corrupts groups; expected: generic HTTP 401 oidc_callback_failed, with no internal claim details exposed.
**Step 4: Provider outage/group drift**
During a controlled window, make discovery/JWKS unavailable or rename a mapped group, run the CLI check, and restore it immediately. Expected: redacted fail-closed diagnostics followed by a successful check after restoration. Do not leave production broken.
## Task 8: Test PSD workspace validation and real connections
**Files:**
- Read: docs/install/server-workspace-registry.md
- Use: authenticated PSD browser sessions
**Step 1: Validate the workspace and authentication from the host CLI**
```bash
"$THT_BIN" --installation "$INSTALLATION" \
workspace inspect --workspace "$WORKSPACE_ID" --json
"$THT_BIN" --installation "$INSTALLATION" auth check --json
```
Expected: the workspace registry is ready, authentication readiness passes, and output is redacted while identifying the active workspace revision.
**Step 2: Verify the application boundary**
Open the configured public URL, authenticate with the approved identity, and verify that the application reaches the selected workspace without unexpected `401`/`403` responses. Keep DWH/Evidence connection tests read-only and use only the existing approved smoke question.
**Step 3: Test ordinary-user authorization**
Log in as TOT Users. Confirm inspection follows ordinary permissions while validation, secret mutation, and connection tests remain unavailable unless explicitly granted.
**Step 4: Run a harmless end-to-end smoke**
As an authorized PSD user, create or resume one harmless known-good session:
```text
browser login -> same-origin API -> workspace readiness -> Pi/core -> result -> logout
```
Do not run mutating production queries. Preserve only a session ID and redacted outcome if approved.
## Task 9: Close acceptance, rollback if needed, and decide merge readiness
**Files:**
- Create: docs/testing/evidence/2026-08-18-thothii-authentication-psd-acceptance.md or the approved external evidence location
- Read: docs/install/server.md and docs/contracts/tht-pi.md
**Step 1: Mandatory gates**
| Gate | Required evidence |
|---|---|
| Candidate identity | Local and server SHA exactly match pushed feat/thoth-auth |
| Local startup | macOS Compose, doctor, health, and Pi smoke pass |
| Local auth | Bootstrap, ordinary/admin roles, logout, bad password, disable/enable, logout-all pass |
| Local session | Remembered session survives browser and ThothII restart; revisions invalidate it |
| Server safety | Old SHA/image/status captured; backup checksummed; maintenance/drain completed |
| Candidate deploy | Server candidate status/doctor/health pass |
| Authentik | Direct groups claim, issuer/JWKS, secrets, catalog, and mapped groups pass |
| OIDC authorization | ordinary, admin, unmapped, malformed, logout, and outage cases pass |
| Workspace integration | `tht workspace inspect` and `tht auth check` pass; the authenticated application reaches the selected workspace |
| PSD smoke | One harmless known-good session completes |
| Hygiene | No secrets, tokens, cookies, hashes, or raw claims in evidence |
**Step 2: Write the redacted report**
Include candidate SHA, old SHA, timestamps, commands, browser cases, redacted diagnostic/HTTP codes, Authentik issuer/client/group names, workspace ID/revision, backup/checksum location, rollback decision, and unrelated CI failures. Never include secret values, raw tokens, cookies, or hashes.
**Step 3: Roll back a failed candidate**
1. Keep the proxy closed and preserve .tht/<installation-id>/ recovery state.
2. Do not use tht pi rollback as the whole-application rollback; it addresses only Pi lifecycle images.
3. Stop with tht.
4. Return the source checkout to <OLD_SHA>, rebuild old application/operator artifacts, and start through the same descriptor.
5. Run update --check-only, status, doctor, health, Pi smoke, workspace diagnostics, and one harmless session.
6. For ambiguous recovery, leave maintenance active and follow pi maintenance status / pi maintenance recover --yes. Never delete volumes, selectors, or recovery files to force progress.
Expected: the previous application serves again with prior data and workspace state intact. Record the failure and do not merge.
**Step 4: Reopen traffic**
After every gate passes, restore the reverse proxy, repeat one unauthenticated redirect and one authorized public login, and confirm only the proxy is externally reachable.
**Step 5: Merge decision**
Merge only after PSD owner acceptance, exact-SHA evidence, no unresolved auth/workspace/provider/deployment gate, and an accepted rollback path. If the merge creates a new commit, repeat Tasks 0, 5, 6, and 7 against the merge SHA.
## Handoff checklist
Deliver the redacted report, local result/SHA, PSD candidate SHA/images, Authentik provider and group mapping confirmation, workspace validation/connection results, backup/rollback status, and an explicit READY TO MERGE or NOT READY TO MERGE decision.
docs/plans/2026-08-20-psd-server-deployment-program-design.md
-370
View File
@@ -1,370 +0,0 @@
# PSD Server Deployment Program — Design
**Date:** 2026-08-20
**Status:** Approved by the owner
**Owner sequencing amendment (2026-08-21):** the Mac `rest_api` acceptance and revocation of
`legacy-shared` are deferred to one mandatory pre-Project-B gate. This permits the bounded survey
and static, non-mutating Project A private preparation to proceed without changing the Mac. It does not
authorize stopping the legacy stack, starting the new stack, opening ingress, or beginning Project B.
**Design-time application baseline:** `main` at `5c0dc8c` (execution must freeze and record the
then-current `origin/main` SHA)
**Design-time workspace baseline:** `tht-workspace-psd/main` at `bfbabf9` (execution must freeze and
record the then-current remote SHA)
## Purpose
Replace the unused legacy ThothII installation on the PSD server with the current application,
recovering only useful configuration and rebuilding runtime state from canonical sources. Complete
the work as two independently accepted projects:
1. deploy and prove ThothII with local authentication, a direct read-only PSD DWH connection,
internal Qdrant, and internal Ollama;
2. only after Project A passes, integrate the accepted installation with the server's Authentik,
Nginx, load balancer, and the existing Aritmolab sidebar link.
The program must be executable by the Sol LLM from a terminal local to the server. It must test
each step, retain redacted evidence, stop on unsafe uncertainty, and include a separate human
manual-test document for each project.
## Binding decisions
- The legacy application may be unavailable for days. Service continuity is not a goal.
- The new source clone is prepared beside the old source directory. The old and new stacks are not
kept running simultaneously: inventory and backup happen first, the old stack is stopped, and
only then is the new stack started.
- Keep the old directory, configuration, containers, images, and data only as a temporary recovery
boundary until the real Aritmolab journey passes Project B. They are disposable after Project B
automated, human, and owner PASS. Do not migrate legacy application sessions, Qdrant data,
Ollama caches, or derived indexes.
- Do not create a host `thothii` user or group. Keep UID/GID `10001:10001` as the image's unmapped
numeric runtime identity and confine numeric ownership to the new installation's writable bind
trees. Stop if either number becomes mapped to a host account before installation.
- Recover configuration only: endpoints, non-secret policy, provider/model selection, relevant
paths, and references to protected credentials. Never copy an old setting without validating it
against the current contract.
- Use the canonical ThothII Compose distribution and reviewed installation-local overrides. Do not
modify an unrelated global Compose project or blindly adapt the legacy Compose file.
- Lifecycle operations use the installation-aware native `tht` CLI, not raw Compose commands.
- Never use `docker compose down --volumes`, global prune operations, broad recursive deletion, or
secret-bearing command arguments.
## Repository and workspace model
There are two Git sources with different responsibilities:
- `ThothII` contains application code and non-secret deployment examples.
- `tht-workspace-psd` contains the shared, credential-free PSD workspace source.
The workspace repository contains one logical workspace, `psd-clinical`. Its schema-v3 descriptor
will declare both supported DWH transports:
```yaml
supported_transports: [rest_api, postgres_direct]
```
The Mac installation continues to select `rest_api`. The PSD server selects `postgres_direct`.
Endpoint values, user names, passwords, secret-file paths, machine paths, and Git credentials stay
in each installation's protected bindings and never enter the workspace repository. Evidence,
curated annotations, language, model policy, and semantic-index contract remain shared.
Each installation owns its own Qdrant collection contents, Ollama model cache, preprocessing state,
and sessions even though both consume the same reviewed workspace commit.
## Program structure
The program consists of one non-mutating common survey followed by two independently gated
projects:
```text
Common Survey PASS for Project A private scope
-> Project A automated PASS
-> Project A human PASS
-> Mac REST acceptance
-> 48-hour dual-key observation covering two 03:00 ETL cycles
-> legacy-shared revocation and negative proof
-> full pre-Project-B survey PASS
-> explicit Project B authorization
-> Project B automated PASS
-> Project B human PASS
-> final cutover acceptance
```
Project B must not start from a partial or assumed Project A result.
## Common Survey
The survey is a prerequisite, not a third implementation project. It runs before any server
mutation and produces a redacted report, topology map, change-scope inventory, unknowns list, and
GO/NO-GO decision.
Sol inventories from the server-local terminal:
- operating system, architecture, Docker and Compose versions, available CPU/RAM/disk, and clock;
- legacy ThothII source, SHA, dirty state, images, containers, networks, ports, volumes, mounts,
health, installation state, and recovery state;
- effective Compose rendering and ownership of every relevant file;
- DWH database/schema, direct listener, TLS, read-only role, and reachability from containers;
- Supabase/PostgreSQL topology, schema conventions, exposed PostgREST schemas, migration policy,
backup mechanism, and suitable role boundaries;
- Pi provider/model policy, credential references, external LLM reachability, and current versions;
- Nginx effective configuration, ThothII virtual host/location, upstream, forwarded headers, SSE
settings, certificate metadata, certificate generation/renewal, and rollback files;
- load-balancer routes, health checks, allowlist capability, TLS boundary, and configuration owner;
- Aritmolab deployment, networks, homepage, sidebar source, current ThothII destination, and release
procedure;
- Authentik version, deployment, current Aritmolab integration, provider conventions, group
conventions, backup/export procedure, API access, and credential locations;
- public DNS/origin that the final sidebar link must preserve;
- protected files by path, ownership, mode, and readability only, without printing their contents.
The survey may use hashes, metadata, redacted renders, and permission checks. It must not emit
passwords, bearer tokens, API keys, cookies, OIDC client secrets, private keys, password hashes, or
raw identity tokens. If credential discovery fails, the owner may help locate the existing
Authentik credentials.
## Project A — Standalone Server Acceptance
### Runtime architecture
Project A installs a clean five-service stack:
```text
operator terminal or local headless browser
-> frontend
-> core + Pi + workflow harness
-> direct read-only PSD PostgreSQL DWH
-> internal Qdrant
-> internal Ollama (qwen3-embedding:0.6b, 1024 dimensions)
-> local filesystem work-session storage
```
The stack uses local authentication. It must not be publicly reachable. Core, Qdrant, and Ollama
remain private; the frontend binds to loopback unless the optional restricted test route below is
proved safe.
### Preparation and cutover
1. Freeze exact application and workspace SHAs and require clean source trees.
2. Publish and validate the multi-transport `psd-clinical` descriptor through the curator workflow.
3. Preserve the Mac REST binding unchanged; its live acceptance is deferred to the mandatory
pre-Project-B gate.
4. Prepare the new source clone and protected operator/runtime directories beside the old source.
5. Extract only approved configuration facts from the legacy installation.
6. Record and verify the exact restart recipe for the legacy stack. No data backup is required
because the owner declared legacy sessions and configuration disposable; the still-present
containers, images, source, and data are the temporary rollback boundary.
7. Stop the legacy stack without deleting its source, configuration, images, or data.
8. Build/install the current native `tht` and application images from the frozen source.
9. Configure local authentication, direct DWH bindings, Pi/provider access, and the Git workspace.
10. Start through `tht`, then validate health, authentication, workspace, workflow, and Pi.
11. Rebuild schema/Evidence preprocessing, Qdrant contents, and the Ollama model cache from
canonical sources. Prove the complete preprocessing rerun is idempotent.
### Optional restricted test route
If and only if the survey proves that the load balancer can enforce a test-operator allowlist
before a request reaches ThothII, Project A may use a temporary private hostname to exercise the
same network path as production:
```text
authorized operator -> allowlisted load-balancer route -> Nginx -> frontend -> core/local auth
```
The route has a distinct hostname, no Aritmolab sidebar link, a certificate created by the existing
managed mechanism, correct forwarded-origin and SSE behavior, and a negative test from an
unauthorized source. Its local-auth `publicUrl` matches the private test origin. It is not a public
production route and must be removed after Project B.
If isolation cannot be demonstrated, Sol must not approximate it or misdeclare
`THOTH_PUBLIC_EXPOSURE=false`; tests run against loopback from the local terminal instead.
### Acceptance
Project A requires:
- exact source identity and reproducible build evidence;
- healthy frontend, core, Qdrant, embedding, and completed model initializer;
- local authentication checks including admin/user separation, wrong password, logout,
disable/enable, invalidation, and restart persistence;
- direct DWH connectivity with a demonstrably read-only runtime identity;
- active `psd-clinical` at the expected Git revision;
- compatible Qdrant collection/index contract and correct Ollama model/dimensions;
- complete and idempotent DWH, schema, annotation, and Evidence preprocessing;
- one harmless real PSD work session completed through F1-F8, ending in read-only validated SQL;
- persisted manifest, artifacts, reviewer decisions, and final SQL inspection;
- optional private-route positive and negative isolation evidence when that route is used;
- completed human manual-test report with an explicit PASS.
The Mac row may be recorded only as `DEFERRED_PRE_PROJECT_B` under the dated owner amendment. It is
not part of the private server acceptance, but it must become PASS before Project B starts.
Project A does not modify the production Aritmolab sidebar, production public route, or Authentik.
## Project B — Authentik and Aritmolab Integration
Project B begins only from the frozen, accepted Project A source, images, workspace revision, and
PASS report. It also requires the deferred Mac REST acceptance, the full 48-hour observation
window (including two scheduled 03:00 ETL cycles), revocation of `legacy-shared`, proof that the
legacy credential receives `401`, and the full pre-Project-B survey gate.
### Final request and data flow
```text
user
-> aritmolab.policlinicosandonato.com
-> Aritmolab homepage/sidebar
-> load balancer
-> Nginx/TLS
-> ThothII frontend and same-origin /api
-> ThothII OIDC Authorization Code + PKCE with Authentik
-> PostgreSQL thoth_sessions schema for owned work sessions
-> direct read-only datawarehouse schema for clinical queries
-> internal Qdrant and Ollama
```
Nginx terminates/proxies according to the observed deployment but does not add a second
`auth_request` in front of ThothII. ThothII performs generic OIDC directly. Nginx preserves the
public host and HTTPS scheme, forwards the callback path unchanged, and supports SSE without
buffering or premature timeouts.
### Authentik configuration
Before mutation, export or back up the relevant Authentik configuration. Locate existing protected
administrative/API credentials without exposing them. Create or adapt:
- one OAuth2/OIDC provider and one ThothII application;
- the exact callback `<public-origin>/api/auth/oidc/callback`;
- `openid`, `profile`, and `email` scopes;
- a direct, non-empty JSON string array claim named `groups`;
- exact user/admin group mappings selected after the survey;
- a separate group-view-only service account/API token for ThothII diagnostics.
Dedicated `TOT Users` and `TOT Admin` groups are the default unless the survey finds existing groups
with exactly the intended semantics and the owner approves their reuse. Additional groups are
ignored. Missing, malformed, indirect, or ambiguous configured groups fail closed.
### Supabase session storage
Do not create a separate PostgreSQL database. Use the server's existing Supabase PostgreSQL
database and isolate ThothII work sessions in the dedicated `thoth_sessions` schema. This schema is
distinct from the clinical `datawarehouse` schema.
- A one-shot migrator role owns only the required schema migration privileges.
- Core receives only the restricted runtime role, never the migrator credential.
- Forced RLS and application ownership checks isolate sessions by Authentik principal.
- The schema stores principals/preferences, manifests, phase artifacts, review decisions, and
audit records.
- Chat and live SSE output remain ephemeral; semantic vectors remain in Qdrant; clinical data
remains in `datawarehouse`; browser authentication sessions remain in protected auth state.
- `thoth_sessions` must not be added to Supabase/PostgREST exposed schemas.
- The direct runtime connection uses the TLS/CA contract required by the current application.
### Safe activation order
1. Back up the accepted Project A operator/auth configuration and every external configuration to
be changed.
2. Prepare Authentik objects without exposing the new route.
3. Run and verify additive session-schema migrations; require no pending or drifted migrations.
4. Prepare OIDC secrets and non-secret configuration in protected installation state.
5. Validate Authentik discovery, issuer/JWKS, catalog access, and mapped groups.
6. Validate Nginx, certificate, load-balancer route, callback, forwarded headers, and SSE while
production traffic remains closed.
7. Start ThothII in OIDC/public server mode with PostgreSQL session storage.
8. Open the final load-balancer route.
9. Preserve or update the Aritmolab sidebar link so the established user journey remains intact.
10. Complete automated and human acceptance, then remove the Project A temporary route.
### Acceptance
Project B requires:
- successful redacted static, live, and interactive authentication diagnostics;
- trusted certificate chain, correct public origin, callback, and proxy headers;
- proven load-balancer/Nginx routing and SSE operation;
- successful migration status, RLS/role tests, and proof that `thoth_sessions` is not REST-exposed;
- ordinary, administrator, unmapped, malformed-claim, logout, and controlled provider-failure
cases;
- login to Aritmolab followed by the sidebar link to ThothII without a second credential prompt;
- no raw OIDC token in browser storage, logs, diagnostics, or evidence;
- session ownership and administrator-boundary tests;
- one harmless F1-F8 PSD session under an OIDC identity;
- tested rollback and a completed human manual-test report with an explicit PASS.
## Error handling and stop rules
Every executable step follows:
```text
precondition -> action -> verification -> redacted evidence -> checkpoint
```
Sol stops and requests owner help rather than improvising when it encounters:
- a dirty or unidentified source checkout;
- uncertain ownership of Compose, Nginx, load-balancer, Aritmolab, or Authentik configuration;
- missing or insufficient credentials;
- an unsafe secret path or risk of secret disclosure;
- an unverifiable backup or rollback path;
- a DWH identity that is not demonstrably read-only;
- a change that would affect unrelated Nginx virtual hosts or other applications;
- an unprovable temporary-route restriction;
- Supabase migration drift, excessive roles, or unintended REST exposure;
- a material difference between the surveyed server and this design.
Unchanged external state is not failure. Sol records the observation and continues only when the
current gate is satisfied.
## Rollback boundaries
- **Project A:** stop the new stack and restart the still-present legacy installation. No new
volume is copied into it and no promise is made to retain it after Project B PASS.
- **Project B ingress:** close the public route first, then restore the prior Nginx,
load-balancer, certificate reference, and sidebar configuration.
- **Project B application:** return to the accepted Project A local-auth operator configuration
while the public route remains closed.
- **Authentik:** initially disable new objects instead of deleting them; retain the pre-change
export until final acceptance.
- **Supabase:** migrations are additive. Rollback does not automatically drop `thoth_sessions` or
destroy evidence; destructive cleanup requires a separate explicit decision.
- **Workspace Git:** publish the multi-transport change as an isolated commit and retain the
previous revision. A rejected candidate never replaces the installation's last valid snapshot.
## Documents and evidence
The implementation-planning phase creates:
1. a general execution program with cross-project gates;
2. a survey checklist and report template;
3. an executable Project A plan for Sol;
4. a plain-language Project A manual-test guide;
5. a Project A evidence/PASS template;
6. an executable Project B plan for Sol;
7. a plain-language Project B manual-test guide;
8. a Project B evidence/PASS template.
Sol maintains a protected server-local progress journal and resumes from the last verified
checkpoint. Detailed topology and command output remain in a protected evidence directory on the
server. Only intentionally redacted reports and reusable templates may enter Git.
The Project A human guide is terminal-first and may use a local headless browser. When the optional
private endpoint exists, it also includes operator browser checks. The Project B guide covers the
real Aritmolab homepage/sidebar, Authentik SSO, roles, logout, final workflow, and negative cases.
## Known documentation reconciliation
Some older server documentation describes vector and embedding services as external and the
mandatory stack as only frontend/core. Current `compose.yaml`, repository instructions, and project
state define Qdrant and Ollama as mandatory internal services. The execution plans must treat the
current code/Compose contract as authoritative and include a documentation correction rather than
following the stale statements.
## Success condition
The program is complete only when both projects have exact-source evidence, all automated gates
pass, both human manuals are completed with explicit PASS decisions, the Aritmolab sidebar reaches
the final ThothII URL through the established load balancer and Nginx, Authentik provides SSO, and a
real read-only PSD session completes F1-F8 under an authorized OIDC identity.
docs/plans/2026-08-20-psd-server-deployment-program.md
-245
View File
@@ -1,245 +0,0 @@
# PSD Server Deployment Program Implementation Plan
> **For agentic workers:** execute one task at a time, record its completion criteria, and stop at every documented approval boundary.
**Goal:** Replace the legacy PSD ThothII installation, prove the replacement with local authentication, and then integrate the accepted release with Supabase, Authentik, Nginx, the load balancer, and the Aritmolab sidebar.
**Architecture:** A read-only common survey freezes the actual server topology before any mutation. Project A installs a clean private stack and proves a complete PSD workflow; Project B begins only after a signed Project A PASS and performs the public OIDC/SSO cutover. Each project has an independent rollback boundary, human test guide, and evidence report.
**Tech Stack:** Linux, Docker Engine, Docker Compose v2, native `tht`, Fastify/React/Pi, PostgreSQL/Supabase, Qdrant, Ollama, Nginx, Authentik OIDC, Aritmolab, load balancer.
---
## Required reading and authority
Read these files completely before starting:
- `AGENTS.md`
- `PROJECT_STATE.md`
- `docs/plans/2026-08-20-psd-server-deployment-program-design.md`
- `docs/install/server.md`
- `docs/install/server-workspace-registry.md`
- `docs/install/authentication-local.md`
- `docs/install/authentication-oidc.md`
- `docs/install/authentik.md`
- `docs/contracts/workspace-preprocessing-cli.md`
- `docs/testing/authentication-manual-acceptance.md`
The current `compose.yaml`, `deploy/compose.server.yaml`, repository instructions, and design are
authoritative where older server prose still describes Qdrant or Ollama as external.
Run only from a terminal local to the server. Do not require SSH port forwarding. Do not print or
paste passwords, tokens, cookies, private keys, hashes, or raw identity claims. Commands that need
a credential must read a protected file or use an echo-free prompt.
## Documents used during execution
- Survey plan: `docs/plans/2026-08-20-psd-server-survey.md`
- Survey report: `docs/testing/evidence/psd-server-survey-report-template.md`
- Project A plan: `docs/plans/2026-08-20-psd-server-project-a-standalone.md`
- Project A human guide: `docs/testing/psd-server-project-a-manual.md`
- Project A report: `docs/testing/evidence/psd-server-project-a-report-template.md`
- Project B plan: `docs/plans/2026-08-20-psd-server-project-b-authentik.md`
- Project B human guide: `docs/testing/psd-server-project-b-manual.md`
- Project B report: `docs/testing/evidence/psd-server-project-b-report-template.md`
The detailed evidence directory is a protected path on the server selected during the survey. The
repository receives only redacted reports after explicit owner review.
## Owner-approved sequencing amendment — 2026-08-21
The Mac `rest_api` acceptance and revocation of `legacy-shared` move to a mandatory gate immediately
before Project B. The survey therefore records two distinct decisions:
- `SURVEY_GO_PROJECT_A_PRIVATE`: technical prerequisite for requesting Project A private execution;
- `SURVEY_GO_PROJECT_B`: the complete shared-infrastructure decision, including Mac acceptance,
observation and legacy revocation.
The amendment authorizes the read-only survey and static preparation of non-secret Project A
candidate facts and artifacts. While the current decision is `SURVEY_NO_GO`, it does not authorize
creating installation roots, cloning/building the candidate, creating protected configuration or
backup state, stopping the legacy stack, starting the new stack, changing public ingress, or
starting Project B. Those remain separate explicit gates after the scoped survey passes.
### Task 1: Freeze the planning source
**Files:**
- Read: `docs/plans/2026-08-20-psd-server-deployment-program-design.md`
- Record: protected server execution journal selected during the survey
**Step 1: Verify the application checkout**
Run:
```bash
git status --short --branch
git rev-parse HEAD
git rev-parse origin/main
git show -s --format='%H%n%P%n%s' HEAD
```
Expected: the tree is clean and `HEAD` is the explicitly approved `origin/main` SHA. A newer SHA
than the design-time `5c0dc8c` is allowed only after recording and reviewing the intervening commits.
**Step 2: Verify the plan files exist at that SHA**
Run:
```bash
test -f docs/plans/2026-08-20-psd-server-survey.md
test -f docs/plans/2026-08-20-psd-server-project-a-standalone.md
test -f docs/plans/2026-08-20-psd-server-project-b-authentik.md
git diff --check
```
Expected: every command exits zero.
**Step 3: Record the immutable planning identity**
Record the application SHA, plan commit, UTC timestamp, operator identity, and terminal-local access
method in the protected journal. Do not record CyberArk session secrets or screenshots.
### Task 2: Execute and approve the common survey
**Files:**
- Execute: `docs/plans/2026-08-20-psd-server-survey.md`
- Create from: `docs/testing/evidence/psd-server-survey-report-template.md`
**Step 1: Execute every survey task without mutation**
Expected: the survey identifies exact paths and owners for the old and new installations, Nginx,
load balancer, Aritmolab, Authentik, Supabase, the DWH, and protected credentials.
**Step 2: Resolve every unknown**
If an Authentik credential cannot be located, stop and ask the owner. If a configuration owner or
rollback boundary is unclear, stop; do not infer authority from file readability.
**Step 3: Review the scoped survey GO/NO-GO**
Expected: `SURVEY_GO_PROJECT_A_PRIVATE` requires a verified old-stack recovery path, an approved
new-installation root, enough resources, a direct read-only DWH path, workspace/model inputs, and
no unresolved mutation in the private Project A scope. Public-origin, load-balancer and Authentik
unknowns may remain explicitly deferred only while Project A is loopback-only and Task 10 is
omitted. `SURVEY_GO_PROJECT_B` retains the complete survey requirements.
**Step 4: Checkpoint the survey**
Hash the protected report and record only its path, SHA-256, timestamp, and GO result in the journal.
### Task 3: Execute Project A
**Files:**
- Execute: `docs/plans/2026-08-20-psd-server-project-a-standalone.md`
- Complete: `docs/testing/evidence/psd-server-project-a-report-template.md`
**Step 1: Confirm the survey is GO for Project A private scope**
Expected: the survey report hash matches the journal and no unresolved blocker remains inside the
Project A private scope. Before any stop/start, obtain a separate explicit owner authorization.
**Step 2: Execute Project A task-by-task**
Do not configure Authentik, change the production Aritmolab sidebar, or open the production route.
**Step 3: Run the Project A human guide**
Follow `docs/testing/psd-server-project-a-manual.md`. Record PASS/FAIL for every case; do not infer
manual PASS from automated output. The Mac REST row may be
`DEFERRED_PRE_PROJECT_B` only under the dated owner amendment.
**Step 4: Close the Project A report**
Expected: automated gates and the human guide are PASS; one harmless PSD session reached F8 and
produced validated read-only SQL; rollback remains available. The accepted report must list the
Mac REST item as an explicit deferred prerequisite rather than silently treating it as PASS.
**Step 5: Obtain explicit owner approval**
Record the approval and report digest. Project B remains forbidden without it.
### Task 4: Freeze the Project B candidate
**Files:**
- Read: accepted Project A report
- Record: protected server execution journal
**Step 1: Recheck source and running images**
Before freezing the candidate, close the pre-Project-B gate: validate the Mac installation with its
per-installation key, finish the 48-hour observation window including two 03:00 ETL cycles, revoke
`legacy-shared`, prove legacy `401` and v1 success, and obtain `SURVEY_GO_PROJECT_B`.
Run the Project A plan's identity commands again. Record application SHA, workspace SHA, core image
ID, frontend image ID, Qdrant image digest, Ollama image digest, and local-auth configuration revision.
Expected: all values match the accepted Project A report.
**Step 2: Recheck rollback**
Prove that the public route is still closed and the protected Project A configuration can be
selected without reconstructing it from memory.
### Task 5: Execute Project B
**Files:**
- Execute: `docs/plans/2026-08-20-psd-server-project-b-authentik.md`
- Complete: `docs/testing/evidence/psd-server-project-b-report-template.md`
**Step 1: Execute Project B task-by-task**
Keep public traffic closed until Authentik, Supabase migrations, ThothII diagnostics, Nginx, TLS,
and load-balancer preflight all pass.
**Step 2: Run the Project B human guide**
Follow `docs/testing/psd-server-project-b-manual.md` using approved ordinary and administrator
identities. The final path begins at the Aritmolab homepage and uses its existing sidebar link.
**Step 3: Close the Project B report**
Expected: SSO, roles, PostgreSQL ownership, the F1-F8 session, rollback rehearsal, and cleanup of the
temporary Project A endpoint all pass.
### Task 6: Close the program
**Files:**
- Modify: `PROJECT_STATE.md`
- Optionally create: reviewed redacted acceptance reports under `docs/testing/evidence/`
**Step 1: Reconcile final state**
Record final SHAs, image identities, workspace revision, Authentik object names/IDs (never secrets),
Supabase database/schema names, public origin, sidebar source revision, Nginx configuration identity,
and both report digests.
**Step 2: Verify final negative boundaries**
Expected: old stack stopped; Project A private endpoint removed; core/Qdrant/Ollama not externally
published; `thoth_sessions` absent from PostgREST exposed schemas; no secret appears in reports.
**Step 3: Update project state**
Add a dated factual section to `PROJECT_STATE.md`. Mark anything not actually run as PENDING.
**Step 4: Run documentation checks**
Run:
```bash
git diff --check
bash scripts/auth-docs-smoke.sh
bash scripts/verify-workspace-install-docs.sh --fixtures-only
```
Expected: all checks pass.
**Step 5: Commit only reviewed redacted documentation**
```bash
git add PROJECT_STATE.md docs/testing/evidence
git diff --cached --check
git commit -m "docs: record PSD server deployment acceptance"
```
Expected: the commit contains no raw server inventory or secret material.
docs/plans/2026-08-20-psd-server-project-a-standalone.md
-532
View File
@@ -1,532 +0,0 @@
# PSD Server Project A Standalone Implementation Plan
> **For agentic workers:** execute one task at a time, record its completion criteria, and stop at every documented approval boundary.
**Goal:** Install a clean PSD ThothII stack with local authentication, direct read-only DWH access, internal Qdrant/Ollama, rebuilt preprocessing, and one completed F1-F8 work session.
**Architecture:** Keep the stopped legacy installation intact only as a temporary rollback boundary and deploy the current canonical five-service Compose stack from an adjacent clean clone. Create no host service identity: the image's UID/GID 10001 remains numeric and unmapped, confined to the new writable bind roots. A reviewed server-local override disables public exposure and uses filesystem work sessions; an optional load-balancer route is permitted only when it is operator-only and its denial boundary is proved first.
**Tech Stack:** Git, Docker/Compose, native `tht`, local Argon2id authentication, PSD Supabase PostgreSQL direct transport, Qdrant, Ollama, Pi, Nginx/load-balancer test route where safe.
---
## Preconditions
- Common survey result is `SURVEY_GO_PROJECT_A_PRIVATE` and its digest is recorded.
- Every path below is replaced by the exact survey result before execution.
- No production Nginx/load-balancer/sidebar/Authentik change is in scope.
- The old stack remains running until its exact inventory and restart recipe are verified; old and
new stacks never run together.
- The server's workspace deploy credential remains read-only. A curator with write access publishes
the workspace change.
- Owner amendment 2026-08-21 declares legacy sessions/configuration disposable. Retain old resources
only until Project B proves the production Aritmolab journey, then delete them under a separate
exact cleanup authorization. No legacy backup is required.
- Do not create a host user or group for 10001. Immediately before preparing `/srv/thothii`, both
`getent passwd 10001` and `getent group 10001` must return no match.
- Owner amendment 2026-08-21 defers live Mac `rest_api` acceptance and `legacy-shared` revocation to
the mandatory pre-Project-B gate. It does not authorize stop/start; those require a later explicit
owner gate even after private preparation is complete.
### Task 1: Freeze exact inputs
**Files:**
- Read: protected survey report
- Read: `docs/plans/2026-08-20-psd-server-deployment-program-design.md`
- Record: protected Project A journal
**Step 1: Record application identity**
Run in the new planning checkout:
```bash
git status --short --branch
git rev-parse HEAD
git rev-parse origin/main
git diff --check
```
Expected: clean and explicitly approved SHA.
**Step 2: Record workspace remote identity**
Use the surveyed read-only credential and run:
```bash
git ls-remote <workspace-remote> refs/heads/main
```
Expected: one SHA recorded as the pre-change workspace revision.
**Step 3: Check old-stack recoverability**
Expected: exact old start/stop procedure, source SHA, Compose identity, volumes/binds, proxy closure
procedure, restart recipe, and shared-resource exclusions are present in the survey. Stop if any is
missing.
### Task 2: Publish the multi-transport workspace revision
**Files:**
- Modify in authorized curator clone: `psd-clinical/workspace.yaml`
- Verify: `thoth-workspaces.yaml`
**Step 1: Create a clean curator branch**
Run in a write-authorized clone, never in the application-managed registry checkout:
```bash
git status --short --branch
git fetch origin main
git switch --create codex/psd-direct-transport origin/main
```
Expected: clean branch at the recorded remote SHA.
**Step 2: Make the minimal descriptor change**
Change exactly:
```yaml
supported_transports: [rest_api]
```
to:
```yaml
supported_transports: [rest_api, postgres_direct]
```
Do not duplicate the workspace or change its ID, collection, Evidence, annotations, model policy,
database, or schema.
**Step 3: Review the descriptor-only diff**
Run:
```bash
git diff --check
git diff -- psd-clinical/workspace.yaml thoth-workspaces.yaml
```
Expected: one semantic line changed; catalog metadata remains identical.
**Step 4: Validate with the current ThothII contract**
Use a disposable installation/registry or the repository's current registry validation harness to
activate the candidate commit before publication. Expected: schema v3 accepts both transports,
Evidence and annotations materialize, and no secret is required for source validation.
If no supported validator can be run in the curator environment, stop and request the owner to run
the established Mac validation; do not publish based only on YAML parsing.
**Step 5: Commit and publish through curator review**
```bash
git add psd-clinical/workspace.yaml
git diff --cached --check
git commit -m "feat: support direct PSD DWH transport"
git push --set-upstream origin codex/psd-direct-transport
```
Merge through the repository's normal review path. Record the resulting `main` SHA.
**Step 6: Record the deferred Mac REST proof**
Do not change the Mac during Project A. Record `DEFERRED_PRE_PROJECT_B`, the unchanged expected
transport `rest_api`, and the exact future diagnostics. The proof must become PASS before Project B,
after protected delivery/configuration of the per-installation key.
### Task 3: Back up and stop the legacy installation
**Files:**
- Create: surveyed protected legacy backup directory
- Record: Project A journal
**Step 1: Capture final legacy state**
Run the surveyed legacy status/doctor commands, record source SHA and image IDs, and confirm no
active user work. Do not use the new `tht` against an incompatible old descriptor.
**Step 2: Close or maintenance-gate the old ThothII route**
Change only the surveyed ThothII-specific route using its established mechanism. Validate Nginx and
load-balancer configuration before applying. Confirm external requests no longer reach the app.
**Step 3: Record the disposable legacy boundary**
Record exact container and image IDs plus filesystem device/inode/ownership/size for
`/home/chirone/ThothII` and `/home/chirone/thothii-data`. Do not archive their contents: the owner
declared them disposable. Prove that the external Evidence bind and both shared Docker networks
are excluded from any later cleanup manifest.
**Step 4: Stop the old stack**
Use its own supported controller. Expected: old containers stopped, not removed; volumes and bind
trees unchanged.
**Step 5: Rehearse the restart command without executing it**
Record the exact command, preconditions, port ownership, and route-restoration order. If it cannot
be stated unambiguously, stop before creating the new stack.
### Task 4: Prepare the adjacent clean installation
**Files:**
- Create: survey-selected new source root
- Create: survey-selected operator, secret, data, Pi-state, registry, and backup roots
**Step 1: Create dedicated paths without creating identities**
Follow `docs/install/server.md` numeric-ownership rules. Do not call `useradd`, `groupadd`, or
`usermod`. The existing operator owns source/operator files; only the new data, Pi-state, and
workspace-registry roots use unmapped numeric `10001:10001`. Stop if UID or GID 10001 resolves to a
host account, and do not change the image identity without a reviewed design amendment.
**Step 2: Clone the frozen application source**
```bash
git -c core.autocrlf=false clone <thothii-remote> <new-source-root>/ThothII
git -C <new-source-root>/ThothII config --local core.autocrlf false
git -C <new-source-root>/ThothII switch --detach <approved-application-sha>
git -C <new-source-root>/ThothII status --short --branch
```
Expected: detached exact SHA, clean tree.
**Step 3: Verify source and platform**
```bash
cd <new-source-root>/ThothII
bash scripts/verify-line-endings.sh
docker version
docker compose version
```
Expected: all pass.
**Step 4: Prepare Pi state and build the operator**
```bash
sudo scripts/prepare-server-pi-state.sh <new-pi-state-root> 10001 10001
THT_THT_OUTPUT_DIRECTORY=<protected-build-output> bash scripts/build-tht.sh
```
Install only the binary matching the surveyed server architecture. Run `tht version --json` and
record its source identity.
### Task 5: Create the protected Project A configuration
**Files:**
- Create outside Git: `<project-a-operator-root>/server.env`
- Create outside Git: `<project-a-operator-root>/thothii-installation.yaml`
- Create outside Git: `<project-a-operator-root>/project-a-private.yaml`
- Create outside Git: `<project-a-auth-root>/auth.yaml` through `tht`
**Step 1: Start from current examples**
Copy `deploy/env/server.env.example` and `docs/install/examples/thothii-installation.server.yaml`
to the protected Project A operator root. Replace every placeholder with surveyed absolute paths.
Never source `server.env` as shell code.
**Step 2: Add the private/local-session override**
Create this reviewed override:
```yaml
services:
core:
environment:
THOTH_PUBLIC_EXPOSURE: "false"
THT_SESSION_STORAGE: local
frontend:
ports: !override
- "127.0.0.1:<project-a-port>:8080"
```
Select an unused loopback port proved by `ss -lntp`. Do not publish core, Qdrant, or Ollama.
**Step 3: Compose the installation descriptor**
Use `profile: server`, the exact new source root/env/auth root, workspace remote/branch/read-only
access, Project A override, and exactly one Git transport override. Do not include the public
session-server overlay in Project A.
For the projected local-auth descriptor, keep `authentication.configDirectory` as the canonical
root and add `runtimeProjection` with a distinct absolute runtime directory plus numeric `uid: 10001`
and `gid: 10001`. The descriptor loader includes the automatic runtime-projection override; do
not list it manually under `overrides`. The canonical root stays `root:root 0700/0600`; the
publisher owns the projection numerically as `10001:10001 0700/0600`. Only the projection is
mounted read-only into core. Do not create a host user/group, edit `CURRENT` or `generations`, or
apply these paths before the separately authorized start gate.
**Step 4: Validate permissions and render**
```bash
<new-tht> --installation <project-a-installation> update --check-only
```
Expected: Compose validates; only frontend has a loopback port; core declares public exposure false
and local session storage; Qdrant/Ollama are internal.
**Step 5: Configure the local administrator**
Create a temporary mode-0600 password file using an echo-free prompt, then run:
```bash
<new-tht> --installation <project-a-installation> auth configure \
--mode local --public-url <project-a-origin> \
--admin-user <test-admin> --admin-display-name <display-name> \
--password-file <protected-temporary-password-file>
```
Remove the temporary input file after success and record that removal. Do not delete generated
`auth.yaml` or `users.yaml`.
Before the later start gate, run the redacted projected status command and require `ready` plus
`equal: true`. A blocked result prevents start. `auth publish` is the only repair path: it rebuilds
from canonical authentication, including after a candidate or recovery restore outcome; it never
promotes a retained runtime generation on its own.
### Task 6: Build and start the clean stack
**Files:**
- Record: Project A evidence directory
**Step 1: Build current images**
```bash
cd <new-source-root>/ThothII
bash scripts/build-local.sh
```
Expected: current core/frontend images build; pinned Qdrant/Ollama references resolve.
**Step 2: Run preflight**
```bash
<new-tht> --installation <project-a-installation> update --check-only
<new-tht> --installation <project-a-installation> pi doctor
```
Expected: no mutation error and no secret in output.
**Step 3: Start through `tht`**
```bash
<new-tht> --installation <project-a-installation> start --build
<new-tht> --installation <project-a-installation> status
<new-tht> --installation <project-a-installation> doctor --json
<new-tht> --installation <project-a-installation> pi test
```
Expected: frontend, core, Qdrant, embedding healthy and model initializer completed. Doctor has the
documented ordered checks and authentication PASS.
**Step 4: Verify listener boundaries**
Use `ss -lntp` and bounded Docker inspection. Expected: only the selected frontend loopback port is
host-published; no external core, Qdrant, or Ollama listener.
### Task 7: Activate the workspace and direct DWH binding
**Files:**
- Modify only through authenticated Workspace Management: encrypted workspace secret store
**Step 1: Pull and inspect the reviewed workspace revision**
```bash
<new-tht> --installation <project-a-installation> \
workspace inspect --workspace psd-clinical --json
```
Expected: active workspace SHA equals the approved multi-transport revision.
**Step 2: Configure runtime bindings**
Through the authenticated Workspace Management API/UI, select `postgres_direct` and provide the
surveyed host, port, runtime user, password, and optional TLS CA. Secret values go to the encrypted
vault; they do not enter `server.env`, Git, shell arguments, or evidence.
The generated contract names are:
```text
THT_WS_PSD_CLINICAL_DWH_TRANSPORT
THT_WS_PSD_CLINICAL_DWH_HOST
THT_WS_PSD_CLINICAL_DWH_PORT
THT_WS_PSD_CLINICAL_DWH_USER
THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE
THT_WS_PSD_CLINICAL_DWH_TLS_CA_FILE
```
**Step 3: Validate and test connections**
Run static validation, live connection test, workspace inspect, and `doctor --json`. Expected: DWH,
workspace, internal embedding, and Qdrant checks pass with redacted output.
**Step 4: Re-prove read-only grants**
Use the survey's catalog query through the exact configured identity. Expected: no DML/DDL grant on
`datawarehouse`. Stop if the runtime user is an owner, superuser, or write-capable role.
### Task 8: Rebuild and verify semantic preprocessing
**Files:**
- Create: Project A preprocessing evidence
**Step 1: Inspect the empty/new collection state**
```bash
<new-tht> --installation <project-a-installation> \
workspace vector inspect --workspace psd-clinical --json
```
Expected: either a compatible empty collection or the documented missing-collection state.
**Step 2: Create the descriptor-owned collection when missing**
Use the guarded vector rebuild only for collection `psd-clinical`, with exact repeated confirmation
and `--destroy`. Do not run it against any other collection.
**Step 3: Run complete preprocessing**
```bash
<new-tht> --installation <project-a-installation> \
workspace preprocess run --workspace psd-clinical --json
```
If it returns `manual_review_required`, inspect the exact run and curated annotations, obtain the
required human decision, run `workspace schema accept --workspace psd-clinical --run <run-id> --yes`,
then resume the same run. Never auto-approve unknown FK changes.
**Step 4: Verify collection contract and counts**
Run vector inspection and record dimensions, cosine distance, required keyword indexes, and bounded
counts by payload kind/revision. Expected: all points carry the active workspace revision.
**Step 5: Prove idempotency**
Run the complete preprocessing command again. Expected: no new review, no duplicate logical points,
unchanged Evidence reported as unchanged, and the same effective configuration identity.
### Task 9: Configure and test local users
**Files:**
- Modify through `tht auth user`: protected local user registry
**Step 1: Add an ordinary test user**
Use an echo-free prompt or protected temporary password file:
```bash
<new-tht> --installation <project-a-installation> auth user add <test-user> \
--role user --display-name <display-name> --password-file <protected-temporary-password-file>
```
Remove the temporary input file after success.
**Step 2: Run authentication diagnostics**
```bash
<new-tht> --installation <project-a-installation> auth status --json
<new-tht> --installation <project-a-installation> auth check --json
```
Expected: pristine redacted JSON and PASS.
**Step 3: Execute automated local-auth cases**
Use same-origin requests or a local headless browser to prove ordinary/admin authorization, generic
wrong-password failure, disable/enable, password/role revision invalidation, logout-all, CSRF, and
remembered-session survival after core restart. Do not retain cookie jars after the test.
### Task 10: Optionally add the private network-path test
**Current scope boundary (owner, 2026-08-21):** omit this entire task and keep Project A
loopback-only. Any future use requires a separate shared-infrastructure authorization after the
public-origin and load-balancer activities pass; the Project A private survey decision alone is
insufficient.
**Files:**
- Modify only surveyed test-specific load-balancer/Nginx files
- Create: test certificate through the existing managed mechanism
**Step 1: Prove allowlist capability before proxying**
Create a temporary hostname that returns a fixed maintenance response. From an approved operator
source expect success; from an unapproved source expect denial. Do not point it at ThothII yet.
**Step 2: Validate and activate the test proxy**
Configure Nginx with the same Host/HTTPS forwarding and SSE settings intended for production. Run
`nginx -t`, validate the load balancer, then reload through the established mechanism.
**Step 3: Reconfigure local-auth public URL transactionally**
If the exact private HTTPS origin differs from the loopback origin, use the supported authentication
configuration workflow and invalidate prior test sessions. Re-run auth and doctor checks.
**Step 4: Prove both sides**
Expected: authorized operator reaches the local login; unauthorized source remains denied before
ThothII. If this cannot be demonstrated, remove the test route and continue on loopback.
### Task 11: Complete the F1-F8 acceptance session
**Files:**
- Complete: `docs/testing/psd-server-project-a-manual.md`
- Create: protected session evidence
**Step 1: Select the approved harmless question**
Use a known read-only PSD question agreed by the owner. Record the wording in the protected report;
do not include patient-identifying values.
**Step 2: Create the session as the ordinary local user**
Use the private browser route when present; otherwise drive the same-origin frontend/API from the
server-local terminal/headless browser. Record only session ID and sanitized milestones.
**Step 3: Review every gate**
Complete F1-F8 without auto-confirming human decisions. Confirm persisted phase/artifact state after
each gate and resume once to prove recovery.
**Step 4: Validate final SQL**
Expected: finalized session, DWH validation PASS, SQL is read-only, and no clinical mutation occurs.
**Step 5: Inspect persisted state**
Confirm manifest, question, schema linking, Evidence, CTE plan/tests, final SQL, validation report,
and decision ledger exist in local filesystem session storage. Chat/SSE need not persist.
### Task 12: Close Project A and preserve rollback
**Files:**
- Complete: `docs/testing/evidence/psd-server-project-a-report-template.md`
**Step 1: Run final diagnostics**
Run status, doctor, auth check, workspace inspect, vector inspect, Pi test, and a bounded secret scan
of the intended report.
**Step 2: Create a transactional new-installation backup**
Use `tht backup --drain` with a protected explicit output. Verify its checksum. Do not include
secrets in the ordinary evidence archive.
**Step 3: Complete human acceptance**
Every private-server row in `docs/testing/psd-server-project-a-manual.md` must be PASS or explicitly
blocking. Only the Mac REST row may be `DEFERRED_PRE_PROJECT_B` under the dated owner amendment.
**Step 4: Record the gate**
Record exact SHAs/images, workspace revision, preprocessing identity/counts, session ID, report
digest, rollback status, and explicit `PROJECT_A_PRIVATE_PASS` or `PROJECT_A_FAIL`. A private PASS
does not authorize Project B while the deferred gate remains open.
**Step 5: Stop on FAIL**
On FAIL, stop the new stack and use the surveyed old-stack recovery plan if service restoration is
desired. Do not start Project B.
docs/plans/2026-08-20-psd-server-project-b-authentik.md
-442
View File
@@ -1,442 +0,0 @@
# PSD Server Project B Authentik Integration Implementation Plan
> **For agentic workers:** execute one task at a time, record its completion criteria, and stop at every documented approval boundary.
**Goal:** Convert the accepted Project A installation to public OIDC mode, store owned work sessions in the existing Supabase database's `thoth_sessions` schema, and restore the established Aritmolab-sidebar user journey through the load balancer and Nginx.
**Architecture:** Keep the same installation-descriptor path and Compose project so Project A Qdrant/Ollama volumes and bind state remain authoritative. Close ingress, snapshot Project A configuration, configure Authentik and Supabase, replace the protected installation configuration transactionally at the same paths, validate privately, then open the production route and sidebar link.
**Tech Stack:** Authentik OAuth2/OIDC Authorization Code + PKCE, ThothII OIDC/session store, Supabase PostgreSQL schema migrations/RLS, Docker Compose, Nginx, load balancer, Aritmolab.
---
## Preconditions
- Project A automated and human reports are PASS and explicitly owner-approved.
- The Mac `rest_api` installation passes source validation and connection diagnostics with its
per-installation key.
- The dual-key observation has lasted at least 48 hours and includes two scheduled 03:00 ETL cycles.
- `legacy-shared` is revoked; v1 remains successful and the legacy credential is proven `401`.
- The current survey decision is `SURVEY_GO_PROJECT_B`, not only the private Project A decision.
- Application SHA, workspace SHA, images, Project A report digest, and rollback configuration match
the accepted evidence.
- The production route is closed before authentication/session-storage changes.
- All Authentik operations use the installed version's API/OpenAPI contract. Official current
references include [OAuth2/OIDC providers](https://docs.goauthentik.io/add-secure-apps/providers/oauth2/),
[provider property mappings](https://docs.goauthentik.io/add-secure-apps/providers/property-mappings/),
[application bindings](https://docs.goauthentik.io/add-secure-apps/applications/manage_apps/), and
[blueprint export](https://docs.goauthentik.io/customize/blueprints/export); installed-version
behavior wins over newer documentation.
### Task 1: Freeze Project A and close ingress
**Files:**
- Read: accepted Project A report
- Create: protected Project B transaction root
**Step 1: Verify exact Project A state**
Run status, doctor, auth check, workspace inspect, vector inspect, Pi status/test, Git SHA, and image
identity checks from Project A. Expected: all match the accepted report.
**Step 2: Create a protected transaction root**
Use `mktemp -d` under the survey-approved protected parent, mode `0700`. Record its path and do not
place it in Git.
**Step 3: Close production and temporary ingress**
Keep or restore a maintenance response at the production ThothII route. Disable the optional
Project A test route before changing authentication unless it is needed for a separately approved
private preflight. Confirm neither route reaches ThothII.
**Step 4: Stop and back up Project A**
```bash
<tht> --installation <stable-installation-path> backup \
--output <project-b-transaction-root>/project-a-backup.tar --drain
<tht> --installation <stable-installation-path> stop
```
Verify the archive using the controller's manifest/checksum contract. Preserve a copy of the exact
installation descriptor, env file, override files, auth directory, Nginx fragment, load-balancer
route, Aritmolab sidebar file/revision, and relevant Authentik export metadata.
### Task 2: Decide exact Authentik names and roles
**Files:**
- Create: protected `authentik-change-manifest.yaml`
**Step 1: Select the final public origin**
Use the live survey result, not historical `.it`/`.com` assumptions. Record exactly one HTTPS origin
and callback `<origin>/api/auth/oidc/callback`.
**Step 2: Select exact groups**
Default to dedicated `TOT Users` and `TOT Admin`. Reuse existing groups only if their membership
semantics match and the owner approves. Record exact case-sensitive names.
**Step 3: Define least privilege**
Map user group → `user`, admin group → `admin`. Define a separate service account/token with only
the installed Authentik permission needed to view exact group objects. No write, user-management,
directory-administration, or superuser permission.
**Step 4: Obtain owner approval of the manifest**
The manifest contains object names, slugs, intended bindings, callback, scopes, grant types,
credential destinations, and rollback action—but no secret values. Do not mutate Authentik before
approval.
### Task 3: Export and prepare Authentik
**Files:**
- Create: protected pre-change Authentik export
- Modify: Authentik objects named in the approved manifest
**Step 1: Export relevant configuration**
Use the installed version's supported blueprint/API export. A worker command such as
`ak export_blueprint` is valid only if present in that version. Protect export mode `0600`; remember
write-only provider secrets are not included, so backup their custody separately without printing.
**Step 2: Verify API credential scope**
Use a read-only call to list relevant groups/applications. Expected: administrative creation access
for the setup identity and a distinct path for the future group-view service account. Stop if the
credential is missing or ambiguous.
**Step 3: Create or confirm exact groups**
Create missing dedicated groups, or record approved existing group IDs. Do not bulk-copy LDAP or
unrelated Authentik memberships.
**Step 4: Create the group-catalog service account**
Grant only exact group-view permission. Create its token through the approved protected-secret
mechanism; write it directly to the ThothII secret destination without displaying it.
**Step 5: Create the OIDC provider**
Configure a confidential OAuth2/OIDC provider with the exact callback, issuer mode observed as
appropriate, Authorization Code, PKCE support, and Device Code only when required for
`tht auth check --interactive` and supported by the installed release. Do not enable implicit flow.
**Step 6: Configure scopes and direct groups claim**
Select `openid`, `profile`, and `email`. Inspect a disposable identity's decoded claim keys through
a protected verifier; retain only a redacted shape. Expected: ID token includes direct non-empty
`groups: [string, ...]`.
If the installed default profile mapping already provides that exact claim, reuse it. Otherwise add
a provider scope/property mapping under the requested `profile` scope that returns:
```python
return {"groups": [group.name for group in request.user.ak_groups.all()]}
```
Verify the installed mapping merge semantics before activation. Do not add a custom unrequested
scope because ThothII requests only `openid`, `profile`, and `email`.
**Step 7: Create the Authentik application**
Bind it to the provider. Configure display metadata according to local Aritmolab conventions. Do not
use an Authentik proxy provider or Nginx forward-auth for ThothII.
**Step 8: Create and store the client secret**
Write the client secret directly into the protected ThothII secret bundle key
`THT_OIDC_CLIENT_SECRET`. Store the service-account token as `THT_AUTHENTIK_API_TOKEN`. Never place
either value in the change manifest, shell history, Compose environment, or evidence.
### Task 4: Prepare Supabase schema roles and backup
**Files:**
- Read: `harness/tht/migrations/sessions/001_schema.sql`
- Read: `harness/tht/migrations/sessions/002_security.sql`
- Create: protected Supabase backup/evidence
- Create: runtime and migrator credential files
**Step 1: Confirm the database/schema boundary**
Expected: use the surveyed existing Supabase PostgreSQL database; clinical data remains in
`datawarehouse`; application sessions use schema `thoth_sessions`; no new database is created.
**Step 2: Back up database metadata/data consistently**
Use the existing Supabase/PostgreSQL backup procedure before DDL. Record backup ID, timestamp,
checksum, and restore command. Do not put a dump in the Git repository.
**Step 3: Create or validate dedicated roles**
Create one migrator login and one runtime login according to the migration contract. The runtime
role must not be superuser, owner, BYPASSRLS, CREATEROLE, CREATEDB, or a member of DWH write roles.
The migrator credential remains unavailable to core.
**Step 4: Write protected credential files**
Create separate mode-0640 runtime-password, migrator-password, and session-CA files with surveyed
ownership. Do not use command-line password arguments.
**Step 5: Confirm PostgREST exclusion before migration**
Record the exact exposed schema list. Expected: `thoth_sessions` absent. If the system exposes all
schemas implicitly, stop and resolve the boundary before migration.
### Task 5: Prepare the stable Project B installation configuration
**Files:**
- Modify at the same stable paths: operator env, installation descriptor, authentication directory
- Create: reviewed session-server override copied from `deploy/compose.session-server.yaml.example`
- Create: protected server-session workspace config copied from `deploy/workspaces/server-sessions.yaml.example`
**Step 1: Preserve the Compose project name**
The native controller derives the project name from the absolute installation-descriptor path.
Keep that exact path. Do not point Project B at a second descriptor path, because that would create
new Qdrant/Ollama named volumes instead of using the Project A accepted state.
**Step 2: Stage Project B files beside the live files**
Prepare new env/descriptor/override/auth inputs in the protected transaction root. Add the session
DB host/port/existing database name/runtime role/migrator role/TLS mode and three secret source
paths. Use `verify-full` where hostname/SAN permits; any `verify-ca` exception requires explicit
survey evidence and owner approval.
**Step 3: Add the server-session override**
Copy the current example to a reviewed local file and add it to the existing stable descriptor's
overrides before the Git transport override ordering required by the installation. Do not edit the
tracked example.
**Step 4: Replace local auth state transactionally**
With the stack stopped, move the complete Project A auth directory into the protected transaction
root, recreate an empty private directory at the same path/ownership/mode, and configure OIDC:
```bash
<tht> --installation <stable-installation-path> auth configure \
--mode oidc --public-url <final-https-origin> \
--issuer <authentik-issuer> --client-id <oidc-client-id> \
--authentik-base-url <authentik-base-url> \
--user-group '<exact-user-group>' --admin-group '<exact-admin-group>'
```
Expected: non-secret `auth.yaml` only; secrets resolved from the protected bundle.
**Step 5: Atomically install staged path-only files**
Use same-filesystem rename and preserve required ownership/mode. Keep the Project A originals in
the transaction root. Run `update --check-only`; on failure restore the originals immediately.
### Task 6: Run and verify session migrations
**Files:**
- Modify through one-shot migrator: existing database schema `thoth_sessions`
**Step 1: Validate migration rendering**
```bash
<tht> --installation <stable-installation-path> update --check-only
```
Expected: core and `session-migrate` resolve the same core image; core lacks migrator password;
only the one-shot service sees it.
**Step 2: Run migrations once**
```bash
<tht> --installation <stable-installation-path> sessions migrate --yes
```
Expected JSON: `"pending":[]` and `"drifted":[]`; `applied` may list `001` and `002` on first use.
**Step 3: Run migration status/idempotency again**
Run the same command. Expected: no new application and both pending/drifted remain empty.
**Step 4: Verify database security**
Through bounded catalog queries, prove forced RLS, policies on session tables, runtime role without
BYPASSRLS/ownership/DDL, migrator absent from core, and no runtime privileges on unrelated schemas.
**Step 5: Recheck PostgREST exclusion**
Expected: `thoth_sessions` still absent from exposed schemas and REST endpoints cannot address it.
### Task 7: Validate Authentik and start privately
**Files:**
- Record: Project B protected evidence
**Step 1: Run static configuration validation**
Run `update --check-only` and redacted `auth status --json`. Expected: mode OIDC, exact public origin,
issuer/client ID/group names, and no secret values.
**Step 2: Start while public ingress remains closed**
```bash
<tht> --installation <stable-installation-path> start
<tht> --installation <stable-installation-path> status
<tht> --installation <stable-installation-path> auth check --json
<tht> --installation <stable-installation-path> doctor --json
<tht> --installation <stable-installation-path> pi test
```
Expected: OIDC discovery/issuer/JWKS, client-secret access, Authentik catalog token, exact mapped
groups, PostgreSQL session storage, workspace, services, workflow, and Pi pass.
**Step 3: Run interactive device check when supported**
```bash
<tht> --installation <stable-installation-path> auth check --interactive
```
Expected: approved identity completes Device Authorization and direct groups claim validates. If
the installed provider does not support device flow, record PENDING rather than substituting a token.
### Task 8: Prepare Nginx, TLS, load balancer, and sidebar
**Files:**
- Modify only survey-approved ThothII Nginx fragment
- Modify only survey-approved load-balancer route
- Modify only exact Aritmolab sidebar source when its target must change
**Step 1: Prepare direct-OIDC Nginx configuration**
Follow `docs/install/reverse-proxy-nginx.md`, direct OIDC section. Required behavior: no
`auth_request`, no callback rewrite, frontend loopback upstream, Host and HTTPS forwarded headers,
HTTP/1.1, buffering/cache off, long SSE read timeout, and `X-Accel-Buffering: no`.
**Step 2: Validate the managed certificate**
Expected: SAN matches final hostname, validity is current, chain is trusted by approved clients,
private-key permissions match local policy, and renewal/generation ownership is recorded. Never
copy the key into ThothII.
**Step 3: Validate Nginx without opening traffic**
```bash
sudo nginx -t
curl --fail http://127.0.0.1:<frontend-port>/health
```
Use local `--resolve`/Host tests only when they do not bypass the identity behavior being tested.
**Step 4: Prepare the load-balancer route**
Configure backend/health/TLS according to the surveyed owner procedure, initially disabled or
operator-only. Confirm it targets Nginx, never core/Qdrant/Ollama directly.
**Step 5: Preserve the Aritmolab link contract**
If the existing sidebar target already equals the final origin/path, leave source unchanged and
record proof. Otherwise make the smallest reviewed change, test it in Aritmolab's own test/build
system, and commit in that repository before deployment.
### Task 9: Open ingress and run OIDC acceptance
**Files:**
- Complete: `docs/testing/psd-server-project-b-manual.md`
**Step 1: Reload Nginx through the established mechanism**
Run `nginx -t` immediately before reload. Expected: reload succeeds and unrelated virtual hosts
remain healthy.
**Step 2: Enable the final load-balancer route**
Expected: HTTP redirects to HTTPS; TLS is valid; `/api/auth/oidc/login` redirects to the correct
Authentik provider; callback returns to the exact public origin.
**Step 3: Test ordinary and admin identities**
Start at Aritmolab, authenticate once, then use the sidebar. Expected: no second credential prompt;
ordinary user can use sessions but receives 403 for admin operations; admin has only documented
permissions.
**Step 4: Test no-role and malformed cases**
An identity with no mapped group authenticates but receives no application role/403. Missing,
malformed, indirect, or ambiguous group claims fail closed with generic browser errors and redacted
diagnostics. Do not retain raw claims.
**Step 5: Test logout and restart**
Verify ThothII logout revokes its own cookie. Document whether the Authentik SSO session remains and
therefore allows immediate re-login without credentials; do not claim global logout unless
configured and tested. Restart core and verify expected OIDC session behavior.
### Task 10: Verify PostgreSQL ownership and complete F1-F8
**Files:**
- Create: protected Project B session evidence
**Step 1: Create sessions under two identities**
Expected: ordinary users see only their own sessions; cross-user access returns the documented
not-found boundary; admin behavior matches `session.read_all/manage_all` permissions.
**Step 2: Verify RLS with the runtime path**
Use application/API tests and bounded catalog evidence. Never disable RLS for diagnosis.
**Step 3: Complete one harmless OIDC PSD session**
Use the same approved read-only question or another owner-approved one. Complete F1-F8, validate
final SQL, resume once, and confirm session/artifacts/decisions are stored in `thoth_sessions`.
**Step 4: Verify ephemeral boundaries**
Expected: no chat transcript or SSE stream stored as session artifacts; no vectors in PostgreSQL;
Qdrant remains the semantic store.
### Task 11: Test controlled failures and rollback
**Files:**
- Record: protected rollback evidence
**Step 1: Test a reversible provider/catalog failure**
Use a controlled, owner-approved method such as a temporary disabled test credential or test object.
Expected: auth diagnostics and browser login fail closed, redacted, then pass after restoration.
Never break unrelated Authentik applications.
**Step 2: Rehearse ingress-first rollback**
Close the production route, validate Nginx restoration commands, and prove the protected Project A
configuration snapshot is complete. A full rollback need not destroy `thoth_sessions`.
**Step 3: Verify additive database rollback boundary**
Expected: rollback leaves schema/data intact for evidence and future recovery. No automatic DROP
SCHEMA or role deletion.
### Task 12: Close Project B
**Files:**
- Complete: `docs/testing/evidence/psd-server-project-b-report-template.md`
**Step 1: Run final diagnostics**
Run status, doctor, auth check, interactive check when supported, workspace inspect, vector inspect,
Pi test, Nginx validation, load-balancer health, Supabase migration/security checks, and sidebar test.
**Step 2: Remove the Project A temporary endpoint**
Remove only its load-balancer route, Nginx fragment, and managed certificate reference according to
their owners. Validate/reload and prove the hostname no longer routes.
**Step 3: Complete the human guide and report**
Every mandatory row must be PASS. Record exact source/image/workspace identities, Authentik object
names/IDs, Supabase database plus `thoth_sessions`, public origin, Aritmolab revision, report digest,
and `PROJECT_B_PASS` or `PROJECT_B_FAIL`.
**Step 4: Handle FAIL safely**
On FAIL, close ingress first. Restore Project A files at the same stable paths, move OIDC auth state
to protected evidence, restore the local-auth directory, validate, and start Project A privately.
Disable new Authentik objects; do not delete them or drop the session schema automatically.
docs/plans/2026-08-20-psd-server-survey.md
-312
View File
@@ -1,312 +0,0 @@
# PSD Server Survey Implementation Plan
> **For agentic workers:** execute one task at a time, record its completion criteria, and stop at every documented approval boundary.
**Goal:** Produce a non-mutating, redacted survey of the PSD server that resolves every path, owner, network boundary, credential location, and rollback prerequisite needed by Projects A and B.
**Architecture:** Collect bounded metadata from the terminal local to the server, retain raw output only in a protected directory, and summarize it in a redacted report. The survey makes no service, file, database, proxy, Authentik, or Git mutation.
**Tech Stack:** Linux utilities, Docker/Compose inspection, Git, Nginx, OpenSSL, PostgreSQL/Supabase metadata queries, Authentik metadata/API discovery, Aritmolab source inspection.
---
## Safety contract
- Do not run `docker inspect` without a restrictive Go template; its default output can contain secrets.
- Do not run `docker compose config` into chat or a public log. Store raw output mode `0600`, then create a redacted derivative.
- Do not print process environments, secret-file contents, private keys, cookies, tokens, password hashes, or raw OIDC claims.
- Do not reload/restart services, fetch/pull Git, log in interactively, change file modes, or make API mutations.
- When a command needs privilege, use the server's approved CyberArk/local-terminal procedure.
### Task 1: Create the protected survey workspace
**Files:**
- Create: `/var/tmp/thothii-psd-survey.<random>/`
- Create: protected `survey-report.md`
**Step 1: Create a private temporary root**
Run:
```bash
umask 0077
PSD_SURVEY_ROOT="$(mktemp -d /var/tmp/thothii-psd-survey.XXXXXX)"
test -d "$PSD_SURVEY_ROOT"
chmod 0700 "$PSD_SURVEY_ROOT"
printf '%s\n' "$PSD_SURVEY_ROOT"
```
Expected: one new mode-0700 directory whose exact path is recorded in the operator journal.
**Step 2: Copy the report template**
Create `$PSD_SURVEY_ROOT/survey-report.md` from the headings in the design's Common Survey section.
Record only findings and references to protected raw files.
### Task 2: Record host and Docker facts
**Files:**
- Create: `$PSD_SURVEY_ROOT/host.txt`
- Create: `$PSD_SURVEY_ROOT/docker-projects.json`
- Create: `$PSD_SURVEY_ROOT/docker-containers.txt`
**Step 1: Record bounded host metadata**
Run each command with output redirected to `$PSD_SURVEY_ROOT/host.txt`:
```bash
date -u '+%Y-%m-%dT%H:%M:%SZ'
uname -a
cat /etc/os-release
getconf LONG_BIT
nproc
free -h
df -hT
docker version
docker compose version
```
Expected: no credential content and enough capacity information to judge a parallel source tree and
a new five-service stack.
**Step 2: Record Compose projects and bounded container identity**
Run:
```bash
docker compose ls --format json > "$PSD_SURVEY_ROOT/docker-projects.json"
docker ps -a --no-trunc --format '{{json .}}' > "$PSD_SURVEY_ROOT/docker-containers.txt"
docker network ls --format '{{json .}}' > "$PSD_SURVEY_ROOT/docker-networks.txt"
docker volume ls --format '{{json .}}' > "$PSD_SURVEY_ROOT/docker-volumes.txt"
```
Expected: inventory only. Do not inspect full container JSON.
**Step 3: Identify candidate legacy ThothII containers**
Use names, images, Compose project labels, published ports, and health from the bounded inventory.
For each candidate, query only these templates:
```bash
docker inspect --format '{{.Name}} {{.Config.Image}} {{index .Config.Labels "com.docker.compose.project"}} {{index .Config.Labels "com.docker.compose.project.working_dir"}}' <container>
docker inspect --format '{{json .NetworkSettings.Networks}}' <container>
docker inspect --format '{{range .Mounts}}{{.Type}} {{.Source}} -> {{.Destination}} rw={{.RW}}{{println}}{{end}}' <container>
```
Expected: exact source/Compose ownership and mounts without environment values.
### Task 3: Survey the legacy ThothII installation
**Files:**
- Create: `$PSD_SURVEY_ROOT/legacy-thothii.txt`
- Create: `$PSD_SURVEY_ROOT/legacy-compose.redacted.yaml`
**Step 1: Resolve source and operator paths from evidence**
Do not search broad filesystem roots. Derive paths from Compose labels, systemd units, Nginx
upstreams, and known operator documentation. Record uncertainty rather than guessing.
**Step 2: Record source identity without fetching**
Run in the identified source tree:
```bash
git status --short --branch
git rev-parse HEAD
git remote -v
git log -5 --oneline --decorate
```
Expected: no mutation. Mark a dirty tree as NO-GO until the owner decides how to preserve it.
**Step 3: Record lifecycle state with the legacy controller**
If the old installation has a supported `tht`, run its bounded `status`, `doctor`, `pi status`, and
`pi maintenance status` commands. Otherwise record exact read-only Docker health and identify the
old lifecycle mechanism. Do not substitute current `tht` against an incompatible descriptor.
**Step 4: Render and redact Compose safely**
Store the raw render as mode `0600`. Replace secret-bearing scalar values with `[redacted]` before
using the derivative in analysis. Confirm the redacted render still shows service names, networks,
ports, volumes, image/build identities, and config file paths.
### Task 4: Survey Nginx, TLS, and the load balancer
**Files:**
- Create: `$PSD_SURVEY_ROOT/nginx.raw.txt` (protected)
- Create: `$PSD_SURVEY_ROOT/nginx-thothii.redacted.txt`
- Create: `$PSD_SURVEY_ROOT/tls-metadata.txt`
- Create: `$PSD_SURVEY_ROOT/load-balancer.md`
**Step 1: Validate and capture Nginx without reload**
Run:
```bash
sudo nginx -t
sudo nginx -T > "$PSD_SURVEY_ROOT/nginx.raw.txt" 2>&1
chmod 0600 "$PSD_SURVEY_ROOT/nginx.raw.txt"
```
Expected: configuration test passes. Do not reload Nginx.
**Step 2: Extract only relevant directives**
Create the redacted derivative containing the ThothII/Aritmolab `server_name`, `listen`, `location`,
`proxy_pass`, `proxy_set_header`, `proxy_buffering`, timeout, certificate path, and include-file
directives. Exclude unrelated virtual hosts and all authorization values.
**Step 3: Record certificate metadata only**
For each relevant public certificate—not its key—run:
```bash
openssl x509 -in <certificate-path> -noout -subject -issuer -serial -dates -ext subjectAltName
```
Expected: exact SAN/expiry/issuer and the observed generation/renewal mechanism.
**Step 4: Map the load balancer**
Record its owner, configuration surface, current Aritmolab backend, health check, TLS boundary,
source addresses seen by Nginx, and whether it can enforce a temporary hostname allowlist. Do not
create a route. If Sol cannot inspect it, name the human/team required for Project A/B gates.
### Task 5: Survey Aritmolab and the sidebar integration
**Files:**
- Create: `$PSD_SURVEY_ROOT/aritmolab.md`
**Step 1: Resolve the Aritmolab source/deployment**
Use Compose labels, Nginx paths, or the documented service unit. Record repository path, SHA, dirty
state, deployment command, container/network identity, and configuration owner.
**Step 2: Locate the sidebar link**
Use `rg` in the resolved source tree for the current ThothII URL, label, historical
`datamart-builder`, and sidebar/navigation definitions. Record exact files and line numbers.
**Step 3: Resolve the real public origin**
The owner reports `aritmolab.policlinicosandonato.com`; historical project state mentions a `.it`
origin and `/datamart-builder`. Record the live browser-visible origin and path from deployed
configuration. Do not choose between them without evidence.
### Task 6: Survey Authentik
**Files:**
- Create: `$PSD_SURVEY_ROOT/authentik.md`
**Step 1: Identify deployment and version**
Record Authentik containers/services, immutable image reference, version, base URL, database/Redis
dependencies, configuration owner, and backup/export procedure. Do not print container environments.
**Step 2: Locate credential references**
Record only file/secret-object paths, ownership, mode, and whether the local operator can use them.
If no usable administrative/API credential is found, stop and request owner help.
**Step 3: Inventory relevant objects read-only**
Using the installed version's API schema or admin interface, list only names/IDs for current
Aritmolab applications/providers, authorization flows, property mappings, groups, service accounts,
and policies that establish local conventions. Do not retrieve write-only secrets or raw tokens.
**Step 4: Record version-specific constraints**
Consult the official documentation matching the installed release for OAuth2/OIDC providers,
scope/property mappings, application bindings, blueprints/export, and API permission semantics.
Do not copy examples from a newer release without comparing the installed OpenAPI schema.
### Task 7: Survey Supabase and the PSD DWH
**Files:**
- Create: `$PSD_SURVEY_ROOT/supabase.md`
- Create: `$PSD_SURVEY_ROOT/dwh-readonly.txt`
**Step 1: Map Supabase services without environments**
Record PostgreSQL, pooler, PostgREST, gateway, and backup components; container networks and local
listeners; database name; TLS listener/CA; and the approved direct-connect route from ThothII core.
**Step 2: Record exposed PostgREST schemas**
Query only the explicit PostgREST schema setting through its known configuration mechanism. Do not
dump the whole environment. Confirm whether `thoth_sessions` already exists or is exposed.
**Step 3: Inspect schemas and migration state**
Through an approved administrative connection, run bounded catalog queries for existing schemas,
owners, and any `thoth_sessions` tables/migration records. Do not change them.
**Step 4: Prove the intended DWH runtime identity is read-only**
Connect using the protected runtime credential mechanism and query `current_database()`,
`current_user`, and grants for schema `datawarehouse`. Expected: USAGE/SELECT as required and no
INSERT, UPDATE, DELETE, TRUNCATE, REFERENCES, TRIGGER, CREATE, or ownership privileges. Do not run a
write probe against clinical tables.
**Step 5: Identify session-schema roles**
Record names or naming rules for a future migrator and runtime role. Do not create them. The final
design uses the existing database plus schema `thoth_sessions`, never a new database.
### Task 8: Survey Git workspace and model boundaries
**Files:**
- Create: `$PSD_SURVEY_ROOT/workspace-and-models.md`
**Step 1: Inspect the workspace remote read-only**
Record remote URL, branch, fetch credential path, current remote SHA, catalog entry, descriptor
schema version, current `supported_transports`, Evidence tree, annotations blob, and deploy-key
permissions. Do not push with the server's read-only deployment credential.
**Step 2: Inspect Pi/LLM policy**
Record selected provider/model/thinking level and credential references. Use bounded `tht pi status`
and `pi doctor` where compatible. Do not output provider keys.
**Step 3: Check internal semantic capacity**
Record CPU/GPU availability, free storage, and whether Docker can run the pinned Qdrant and Ollama
architectures. Do not pull images or models during the survey.
### Task 9: Produce the survey decision
**Files:**
- Modify: `$PSD_SURVEY_ROOT/survey-report.md`
**Step 1: Complete the topology**
Include exact component owners and flows for user → load balancer → Nginx → Aritmolab/sidebar →
ThothII, and core → Supabase DWH/auth session schema/Qdrant/Ollama/LLM/Authentik.
**Step 2: List exact intended change files**
Separate files owned by the new ThothII installation, workspace curator, Nginx, load balancer,
Aritmolab, Authentik, and Supabase. Mark shared files as owner-gated.
**Step 3: State the scoped GO or NO-GO decisions**
State both `SURVEY_GO_PROJECT_A_PRIVATE` and `SURVEY_GO_PROJECT_B`. The private decision requires
all paths, permissions, backup owners and rollback boundaries used by Project A; it may defer
public-origin, load-balancer, Authentik and Mac REST closeout facts that Project A does not mutate.
The Project B decision requires every shared/public fact plus the Mac acceptance, completed
observation window and revoked legacy credential. Each NO-GO must name concrete missing facts and
the person/system needed to resolve them.
**Step 4: Hash and retain the report**
Run:
```bash
sha256sum "$PSD_SURVEY_ROOT/survey-report.md" > "$PSD_SURVEY_ROOT/survey-report.sha256"
sha256sum --check "$PSD_SURVEY_ROOT/survey-report.sha256"
```
Expected: checksum passes. Move the complete mode-0700 survey directory to the approved protected
evidence root without changing its contents; record the final path and digest in the journal.
docs/plans/2026-08-24-evidence-restructuring-design.md
-909
View File
@@ -1,909 +0,0 @@
# Ristrutturazione delle Evidence — disegno approvato
**Stato:** approvato il 24 agosto 2026
**Sostituisce:** il precedente disegno di Evidence canonica, disponibile nella storia Git
**Ambito:** authoring, revisione, pubblicazione, indicizzazione e uso runtime delle Evidence
## 1. Obiettivo
Questo disegno introduce un processo semplice e verificabile per trasformare documenti
di partenza non necessariamente ben organizzati in Evidence strutturate, revisionabili
da una persona e ricercabili in modo efficace da ThothII.
La soluzione deve:
1. partire dai testi oggi presenti nel repository del workspace;
2. riorganizzarli senza inventare informazioni;
3. conservare sorgenti e risultato nello stesso repository Git;
4. affidare a Git la revisione e l'approvazione umana;
5. indicizzare soltanto le versioni approvate;
6. sfruttare Qdrant senza moltiplicare collezioni e componenti;
7. inserirsi nel workflow modulare attuale, nel quale Evidence è un modulo autonomo.
La fonte di verità rimane sempre il repository Git. Qdrant è un indice derivato che può
essere ricostruito.
## 2. Principio guida
Il processo è diviso in due percorsi distinti.
- Il **percorso di authoring** prepara e revisiona le Evidence fuori dalle sessioni
domanda→SQL.
- Il **percorso runtime** è in sola lettura e consulta esclusivamente Evidence già
pubblicate.
```mermaid
flowchart LR
S["Testi sorgente"] --> P["Pre-processing"]
P --> C["Evidence curate"]
C --> R["Revisione Git umana"]
R --> M["Merge e attivazione revisione"]
M --> I["Indicizzazione atomica"]
I --> Q["Qdrant: indice attivo"]
Q --> E["Evidence Module"]
E --> W["Workflow F1-F8"]
```
Una sessione può proporre una nuova formula o segnalare una lacuna, ma non modifica il
repository e non pubblica autonomamente conoscenza.
## 3. Struttura nel repository del workspace
Ogni workspace adotta questa struttura sotto la propria directory `evidence/`:
```text
evidence/
├── README.md
├── source/
│ └── ... documenti originali ...
├── curated/
│ ├── glossary/
│ ├── domain/
│ ├── enum/
│ ├── example/
│ ├── mapping/
│ ├── normalization/
│ ├── formula/
│ └── reference/
├── manifest.yaml
└── evaluation.yaml
```
### 3.1 `source/`
Contiene i documenti originali. La prima versione accetta file Markdown, testo UTF-8 e
file `.sql.md`. Un URL può essere descritto in un documento, ma non viene scaricato né
interpretato automaticamente.
I sorgenti vengono preservati: il pre-processing non li riscrive.
### 3.2 `curated/`
Contiene una Evidence Unit per file. Le sottodirectory rendono immediatamente visibile
il tipo anche a un lettore umano. Il campo `kind` nel documento resta comunque
obbligatorio: la directory aiuta la navigazione, il campo è il contratto macchina.
### 3.3 `manifest.yaml`
È gestito dal comando di preparazione e registra:
- hash di ciascun sorgente;
- Evidence Unit derivate da quel sorgente;
- identificatori stabili;
- versione del processo di preparazione;
- unità orfane da controllare.
Il manifest permette di elaborare soltanto ciò che è cambiato. Non sostituisce Git e
non contiene lo stato di approvazione.
### 3.4 `evaluation.yaml`
Contiene inizialmente circa venti domande rappresentative e gli identificatori delle
Evidence che ci aspettiamo di recuperare. È il controllo minimo per evitare di
considerare “migliore” una ricerca soltanto perché sembra sofisticata.
## 4. Una struttura comune, otto tipi distinti
La separazione tra tipi non viene eliminata. Ogni documento ha un involucro comune e
una parte specializzata determinata da `kind`.
### 4.1 Campi comuni
```yaml
schema_version: 1
id: evidence:fascia-pediatrica
title: Fascia pediatrica
kind: formula
purposes:
- sql_generation
- schema_linking
applies_to:
concepts:
- fascia pediatrica
tables:
- clinical.patient
columns:
- clinical.patient.birth_date
language: it
provenance:
source_file: source/10-domini-clinici/paziente.md
source_sha256: sha256:0123456789abcdef...
supporting_excerpts:
- Per fascia pediatrica si intendono i pazienti con età inferiore a 18 anni.
review_items: []
```
I campi hanno ruoli diversi:
- `kind` dice **che cosa contiene** il documento;
- `purposes` dice **in quali attività può essere utile**;
- `applies_to` dice **a quali concetti o elementi del database si riferisce**;
- `provenance` permette di risalire al testo di origine;
- `review_items` rende visibili i dubbi ancora da risolvere.
Ogni unità contiene da uno a cinque `supporting_excerpts`, ciascuno lungo al massimo
1.000 caratteri. Sono citazioni brevi che il validatore deve ritrovare nel sorgente dopo
la stessa normalizzazione meccanica. Provano la tracciabilità, non la correttezza
semantica: il revisore umano deve comunque verificare che sostengano davvero il
contenuto ristrutturato.
Ogni `review_item` contiene soltanto:
```yaml
code: ambiguous_source_statement
message: Il sorgente non chiarisce se l'età sia calcolata alla data di ricovero.
field: formula.sql # opzionale
```
Non possiede stato, autore o timestamp. Tutti i review item bloccano la pubblicazione;
il curatore corregge il documento e rimuove l'item, mentre Git conserva la storia.
### 4.2 Tipi iniziali
| `kind` | Contenuto | Esempio d'uso |
| --- | --- | --- |
| `glossary` | Definizione, sinonimi e varianti linguistiche | Capire che “ricovero” e “degenza” possono indicare lo stesso concetto |
| `domain` | Regole e vincoli del dominio | Interpretare correttamente un episodio clinico |
| `enum` | Valori ammessi e loro significato | Tradurre “dimesso” nel codice memorizzato nel DWH |
| `example` | Domanda esemplificativa e interpretazione attesa | Riconoscere una formulazione già documentata |
| `mapping` | Collegamento fra concetto e schema fisico | Individuare tabella e colonne pertinenti |
| `normalization` | Regole di normalizzazione | Uniformare codici, date o varianti testuali |
| `formula` | Espressione SQL riutilizzabile e relativi input | Calcolare la fascia pediatrica dalla data di nascita |
| `reference` | Un riferimento esterno che è esso stesso contenuto recuperabile | Proporre all'utente il link a una specifica linea guida |
Un URL che documenta un'altra Evidence appartiene alla sua `provenance`. Un URL che
deve essere recuperato come risposta autonoma è invece una Evidence `reference`.
Ogni Evidence Unit possiede un solo `kind`, scelto in base ai campi strutturati che ne
definiscono il contenuto principale. `purposes` e `applies_to` possono invece avere più
valori. Quando parti dello stesso sorgente hanno identità e regole di validazione
indipendenti, vengono prodotte unità distinte; non si duplica un'unità soltanto perché è
utile in più fasi del workflow.
La classificazione procede dai contenuti più strutturati a quelli più generali:
`formula`, `enum`, `mapping`, `normalization`, `glossary`, `domain`, `example` e
`reference`. `domain` è il tipo di ripiego per una regola del dominio che non soddisfa
uno schema più specifico; `reference` si applica soltanto quando il collegamento deve
essere restituito come contenuto autonomo.
L'identificatore non incorpora il `kind`: una riclassificazione conserva l'ID, mentre
una vera divisione semantica assegna nuovi ID alle nuove unità. Un rinominamento
univocamente riconoscibile del Source Evidence tramite hash aggiorna la provenienza e
conserva gli ID esistenti.
Un nuovo identificatore usa la forma leggibile `evidence:<slug>`, viene assegnato una
sola volta e non viene ricalcolato da titolo, percorso o hash. Le collisioni ricevono un
suffisso deterministico. Dopo la prima pubblicazione cambiare ID equivale a ritirare
l'unità esistente e crearne una nuova.
### 4.3 Dati specifici per tipo
La parte specializzata è una unione discriminata: ogni `kind` ammette e richiede campi
diversi. Alcuni esempi:
```yaml
# formula
formula:
concept: fascia pediatrica
columns:
- clinical.patient.birth_date
sql: |
CASE WHEN age < 18 THEN 'pediatrica' ELSE 'adulta' END
```
```yaml
# reference
reference:
url: https://example.org/linea-guida
label: Linea guida clinica
description: Criteri usati per classificare gli episodi.
```
```yaml
# enum
enum:
column: clinical.episode.discharge_status
values:
D: dimesso
T: trasferito
```
I tipi restano quindi sfruttabili sia in validazione sia in ricerca. Una formula non è
un semplice testo etichettato: possiede obbligatoriamente un concetto, le colonne di
input e una singola espressione PostgreSQL componibile. `SELECT`, `WITH`, DDL e DML come
statement completi non sono Formula Evidence; una query completa documentata appartiene
a `example`. Le formule legacy incompatibili diventano review item durante la migrazione.
## 5. Pre-processing dei testi sorgente
Il comando concettuale è:
```text
tht evidence prepare <workspace-root>
```
Per l'utente è una sola operazione. Internamente esegue quattro passaggi.
### 5.1 Estrazione deterministica
Il sistema:
- individua i file ammessi in `evidence/source/`;
- verifica dimensione, codifica UTF-8 e percorso sicuro;
- calcola l'hash del contenuto;
- confronta il risultato con `manifest.yaml`;
- carica, quando esiste, la precedente versione curata collegata al sorgente.
Un sorgente invariato non viene nuovamente elaborato.
Se la `pipeline_version` del manifest non è compatibile con quella installata, il
normale `prepare` termina senza scrivere. Il curatore può scegliere esplicitamente
`prepare --upgrade`, esclusivamente su un repository pulito, per rielaborare tutti i
sorgenti e revisionare il diff completo.
### 5.2 Normalizzazione deterministica
Prima del modello vengono normalizzati soltanto aspetti meccanici:
- terminatori di riga e Unicode;
- spaziatura e intestazioni palesemente riconoscibili;
- elenchi, tabelle, blocchi SQL e URL;
- metadati già esplicitamente presenti;
- riferimenti a tabelle e colonne riconoscibili.
Questa fase non interpreta il significato e non inventa strutture semantiche.
### 5.3 Una sola ristrutturazione assistita dal modello
Per ogni sorgente cambiato il modello riceve:
- il testo normalizzato;
- gli otto schemi ammessi;
- le regole “non inventare” e “segnala il dubbio”;
- le precedenti Evidence curate derivate da quel sorgente;
- gli identificatori già assegnati.
Per un'unità già esistente il modello può restituire soltanto uno degli identificatori
ricevuti. Per una nuova unità non propone l'ID: il preparatore assegna una sola volta
`evidence:<slug>` e l'eventuale suffisso deterministico. Un identificatore sconosciuto
prodotto dal modello rende la risposta non valida.
Può:
- assegnare titoli;
- classificare il tipo;
- separare un sorgente in più Evidence Unit;
- riordinare e riscrivere per chiarezza;
- compilare campi strutturati con fatti presenti nel sorgente.
- citare da uno a cinque brevi estratti del sorgente che sostengono ciascuna unità.
Non può:
- fondere automaticamente sorgenti diversi;
- aggiungere fatti non documentati;
- risolvere silenziosamente un'ambiguità;
- cancellare un'unità precedentemente revisionata.
Se il sorgente esiste ancora ma non sostiene più un'unità precedente, il modello la
restituisce come retirement candidate con il `review_item`
`source_no_longer_supports_unit`. Il curatore decide se eliminarla o riscriverla; fino a
quel momento la pubblicazione resta bloccata.
Quando una precedente unità viene realmente divisa in più unità autonome, le nuove
unità ricevono nuovi ID e la precedente rimane una retirement candidate finché il
curatore non la ritira esplicitamente.
Pi viene usato in modalità non interattiva e senza strumenti di scrittura. È un
dettaglio interno del comando, non una nuova tipologia di sessione ThothII.
Timeout, uscita non valida o JSON malformato interrompono il comando con un errore
attribuito al sorgente. La prima versione non esegue retry automatici.
### 5.4 Validazione deterministica
L'output del modello non viene scritto direttamente. Viene prima controllato:
- schema comune e schema specifico del `kind`;
- unicità e stabilità degli identificatori;
- appartenenza alle enumerazioni ammesse;
- esistenza e hash del sorgente;
- presenza nel sorgente normalizzato di ogni `supporting_excerpt`;
- correttezza sintattica di URL, tabelle, colonne e SQL dove applicabile;
- assenza di credenziali;
- coerenza tra directory e `kind`;
- assenza di collegamenti a sorgenti diversi nella stessa unità.
Esistono tre esiti.
| Esito | Comportamento |
| --- | --- |
| Valido | Il documento è pronto per la revisione Git |
| Valido con dubbi | Il documento viene scritto con `review_items`; non è indicizzabile |
| Non valido | Il documento non è pubblicabile e il rapporto spiega l'errore |
Un dubbio reale può essere mantenuto soltanto se il revisore lo trasforma in una
limitazione esplicita del contenuto e svuota `review_items`.
### 5.5 Applicazione atomica
Tutti gli output dei sorgenti cambiati vengono costruiti e validati in un'area
temporanea. Soltanto quando l'intero batch è valido, il comando sostituisce insieme i
documenti interessati e `manifest.yaml`. Un singolo errore lascia il worktree invariato
e il rapporto limitato elenca tutti i problemi rilevati. Non esiste successo parziale.
## 6. Aggiornamenti incrementali e revisione delle correzioni umane
La precedente versione curata è un input, non un file usa-e-getta. Il modello deve
proporre una modifica minima senza ricominciare da zero, ma questa istruzione non viene
presentata come una garanzia semantica. La garanzia è Git: la versione precedente resta
recuperabile, ogni variazione è visibile nel diff e nessuna proposta diventa Published
Evidence senza una nuova revisione umana.
Il comando:
- si rifiuta di operare se `evidence/curated/` o `evidence/manifest.yaml` contengono
modifiche Git non salvate;
- mantiene gli ID associati a contenuti che rappresentano ancora la stessa unità;
- riconosce come rinominato un sorgente nuovo che corrisponde univocamente all'hash di
un sorgente rimosso e ne aggiorna la provenienza senza cambiare gli ID;
- mostra come diff le variazioni proposte;
- non modifica i file derivati da sorgenti invariati;
- segnala come orfana un'unità il cui sorgente è stato rimosso;
- non elimina mai automaticamente un'unità orfana;
- blocca la pubblicazione finché ogni unità orfana non viene eliminata, ricollegata o
ricondotta a un sorgente ripristinato.
Git fornisce confronto, revisione, cronologia e recupero. Non viene introdotto un
database di authoring parallelo.
Orfani e retirement candidate vengono risolti senza modificare manualmente il manifest:
```text
tht evidence resolve <evidence-id> --retire
tht evidence resolve <evidence-id> --source <source-path>
```
Le due azioni sono mutuamente esclusive, richiedono un worktree pulito e aggiornano
atomicamente file curato e manifest. `--retire` rimuove l'unità dal corpus di authoring;
`--source` aggiorna provenienza e hash soltanto verso un sorgente esistente. Entrambe
lasciano un diff Git recuperabile, senza commit o pubblicazione automatica. Se l'unità
deve essere riscritta, il curatore modifica invece il documento e poi esegue
`validate`.
## 7. Revisione e pubblicazione
Il flusso di pubblicazione è:
```text
prepare → revisione Git → validate → merge → attivazione workspace
→ preprocess evidence candidata → evaluate candidata
→ pubblicazione atomica della generazione Qdrant
```
### 7.1 Approvazione umana
L'approvazione coincide con il normale processo Git del repository del workspace:
1. il curatore esegue `prepare` in un clone di authoring;
2. legge i documenti e il diff;
3. corregge i contenuti;
4. esegue `tht evidence validate`;
5. apre o approva la pull request;
6. esegue il merge.
La prima versione non crea automaticamente branch, commit o pull request.
### 7.2 Quando un documento diventa Published Evidence
Una Curated Evidence diventa Published Evidence soltanto quando:
- appartiene a una revisione Git pulita che ha superato il processo umano di revisione;
- la revisione è stata attivata dal registry di ThothII;
- non contiene `review_items` irrisolti;
- il manifest non contiene unità orfane;
- l'intero corpus supera la validazione;
- la Candidate Evidence Generation supera il gate top-10;
- la generazione Qdrant viene quindi pubblicata atomicamente.
Il runtime non tenta di ricostruire come sia avvenuta l'approvazione Git e il manifest
non contiene un flag `approved`. Il confine verificabile di pubblicazione è la
combinazione di revisione attiva, validazione superata e generazione Evidence attiva.
Il descriptor filesystem deve indicizzare solo `curated/**/*.md`. I sorgenti e i file
di supporto restano materializzati per tracciabilità, ma non entrano nell'indice.
## 8. Indicizzazione e generazioni
L'indicizzazione continua a usare il meccanismo già implementato dal modulo Evidence:
1. legge la radice materializzata della revisione Git attiva;
2. valida nuovamente tutte le Evidence;
3. costruisce Evidence Fragment secondo sezioni semantiche;
4. verifica il vettore dense predefinito già usato da Schema e Memory;
5. aggiunge in modo non distruttivo il vettore sparse `bm25` se manca;
6. genera le rappresentazioni dense degli Evidence Fragment;
7. chiede a Qdrant di generare per gli stessi frammenti la rappresentazione lessicale
BM25;
8. carica i punti con la nuova `vector_generation`;
9. verifica manifest, conteggi e leggibilità;
10. rende attiva la nuova generazione;
11. conserva le generazioni precedenti previste dalla policy.
Se uno dei passaggi fallisce, la generazione precedente rimane attiva. I punti caricati
parzialmente vengono compensati secondo il meccanismo transazionale già esistente.
L'eventuale configurazione `bm25` già aggiunta rimane: è compatibile con i punti dense
esistenti e non richiede rollback. Se `bm25` esiste con una configurazione diversa da
`modifier: idf`, la procedura fallisce senza modificarla.
L'upgrade non ricrea la collezione. I record `schema_table`, `schema_column`, `memory` e
`solved_question` restano invariati e continuano a usare il vettore dense predefinito.
Soltanto gli Evidence Fragment ricevono anche `bm25`. Durante la finestra fra aggiunta
del vettore e pubblicazione della prima candidata ibrida, Evidence è `unavailable`, ma
Schema e Memory continuano a funzionare.
## 9. Qdrant spiegato senza presupporre conoscenze vettoriali
### 9.1 L'analogia della biblioteca
Si può immaginare Qdrant come il catalogo di una biblioteca.
- Le **Evidence Unit** sono i documenti completi conservati negli scaffali Git.
- Gli **Evidence Fragment** sono le schede del catalogo relative alle singole sezioni.
- I **vettori** sono rappresentazioni numeriche usate per confrontare una domanda con
quelle schede.
- Il **payload** è l'insieme delle etichette leggibili: tipo, scopo, tabelle, colonne,
revisione e documento di origine.
Qdrant non decide se una Evidence è vera e non sostituisce il documento. Aiuta soltanto
a trovare rapidamente le schede più promettenti.
### 9.2 Ricerca per significato: vettore dense
La rappresentazione dense descrive il significato generale di una frase. Permette, per
esempio, di avvicinare “pazienti minorenni” a “fascia pediatrica” anche quando le parole
non coincidono.
È utile per il linguaggio naturale, ma può essere meno precisa con codici, acronimi,
nomi di colonne e formule.
### 9.3 Ricerca per parole e identificatori: BM25 sparse
La rappresentazione sparse conserva il peso delle parole presenti. È adatta a termini
come `ICD-10`, `discharge_status`, `ADT`, un valore enum o un nome esatto di colonna.
Qdrant 1.18.2 può generare questa rappresentazione direttamente sul server usando
`qdrant/bm25`; per il corpus italiano si passa `language: italian` sia durante il
caricamento sia durante la ricerca. Non serve aggiungere FastEmbed o un nuovo servizio.
Un test L0 avvia esattamente l'immagine Qdrant dichiarata da `compose.yaml`, crea una
collezione temporanea, indicizza due testi italiani mediante `qdrant/bm25`, verifica una
ricerca lessicale e infine elimina la collezione. Questo rende controllabile la capacità
locale richiesta prima di qualunque migrazione reale. Se la prova fallisce non esiste
un fallback silenzioso a un motore diverso.
Il nome “sparse” significa soltanto che, tra moltissime parole possibili, ogni testo ne
usa poche. Qdrant mantiene anche l'IDF: una parola rara pesa più di una parola presente
quasi ovunque.
### 9.4 Perché combinarle
Una domanda può richiedere contemporaneamente comprensione e precisione lessicale:
> “Qual è la formula per distinguere la fascia pediatrica usando
> `patient.birth_date`?”
La ricerca dense riconosce il concetto; BM25 riconosce con forza “formula” e il nome
della colonna. Qdrant esegue entrambe e produce due graduatorie.
### 9.5 Reciprocal Rank Fusion
Reciprocal Rank Fusion, o RRF, combina le due graduatorie usando la posizione dei
risultati invece di confrontare direttamente punteggi di natura diversa.
In termini pratici:
- un documento alto in entrambe le liste sale;
- un documento molto forte in una sola lista può comunque emergere;
- non occorre inventare una conversione fragile fra “similarità semantica” e “punteggio
delle parole”.
Si parte con i pesi predefiniti. Pesi diversi saranno introdotti soltanto se
`evaluation.yaml` dimostrerà un miglioramento.
### 9.6 Il ruolo dei metadati
Ogni punto Qdrant conserva almeno:
```text
workspace_id
workspace_revision
vector_generation
record_kind
evidence_id
evidence_kind
purposes
concepts
tables
columns
language
source_file
source_sha256
fragment_ordinal
```
I metadati hanno due usi:
- workspace, revisione, generazione e purpose sono filtri obbligatori;
- tipo, concetti, tabelle e colonne diventano filtri soltanto quando il chiamante li
dichiara vincolanti; altrimenti contribuiscono al testo della query e alla spiegazione
del risultato.
La prima versione non aggiunge bonus automatici per `kind` o `applies_to`. Durante la
generazione SQL una Formula Evidence viene imposta soltanto quando il workflow richiede
esplicitamente `kind=formula`; negli altri casi dense, BM25 e RRF determinano l'ordine.
Quando concetti, tabelle o colonne non sono vincoli, il modulo costruisce un solo testo
deterministico, identico per dense e BM25:
```text
Domanda: <domanda originale>
Concetti: <valori deduplicati e ordinati>
Tabelle: <valori deduplicati e ordinati>
Colonne: <valori deduplicati e ordinati>
```
Le righe vuote sono omesse. La domanda conserva formulazione e ordine originali; il
renderer applica soltanto Unicode NFC, converte CRLF e CR in `\n`, rimuove gli spazi
esterni e rifiuta una domanda vuota. Non cambia maiuscole, punteggiatura o spazi interni.
Ai valori contestuali applica NFC e `strip`, elimina stringhe vuote e duplicati esatti e
li ordina per valore Unicode senza `lower()` o `casefold()`: gli identificatori
PostgreSQL quotati possono essere sensibili alle maiuscole. In questo modo due richieste
equivalenti non cambiano per effetto dell'ordine occasionale dei metadati.
### 9.7 Perché non creare una collezione per tipo
Una domanda spesso attraversa più tipi: una formula può dipendere da un mapping, da un
enum e da una regola di dominio. Collezioni separate richiederebbero più interrogazioni,
fusione applicativa e più operazioni di manutenzione.
La soluzione usa la collezione semantica già posseduta dal workspace, conserva il suo
vettore dense predefinito e aggiunge soltanto il vettore sparse denominato `bm25`. I
payload indicizzati distinguono i tipi. È più semplice, evita di ricostruire Schema e
Memory e permette a Qdrant di eseguire ricerca ibrida e filtri nella stessa Query API.
### 9.8 Perché indicizzare frammenti ma restituire unità
Un documento lungo può contenere sezioni diverse. Un unico vettore ne diluirebbe il
significato; frammenti arbitrari di lunghezza fissa spezzerebbero invece formule o
regole.
La divisione segue intestazioni, campi tipizzati e confini di paragrafo. Non divide mai
una formula, una coppia valore/significato, un mapping, una regola o un URL. Se uno di
questi elementi atomici supera da solo `max_chunk_chars`, la preparazione aggiunge il
Review item stabile `atomic_content_too_large` e blocca la pubblicazione: non usa un
taglio a dimensione fissa che ne altererebbe il significato. Il limite è quello già
presente nella configurazione degli embeddings, pari per default a 4.000 caratteri, e
si applica all'intero testo reso che sarà inviato all'embedder, incluse etichette e
metadati testuali. Non viene introdotta una seconda impostazione. Qdrant trova i
frammenti, poi l'Evidence Module li raggruppa per `evidence_id` e restituisce un solo
Evidence Result con i migliori estratti, provenienza, citazione e riferimento al
documento completo. Il contenuto completo viene risolto soltanto quando il workflow ne
ha bisogno.
### 9.9 Cosa non introduciamo nella prima versione
- una collezione per ogni tipo;
- ColBERT o multivettori late-interaction;
- un reranker basato su un altro modello;
- pesi RRF regolati a mano senza misurazioni;
- un servizio separato per BM25;
- ricerca automatica sul web.
Queste possibilità rimangono future ottimizzazioni, non prerequisiti.
Riferimenti tecnici ufficiali:
- [Qdrant: Text Search](https://qdrant.tech/documentation/search/text-search/)
- [Qdrant: server-side BM25](https://qdrant.tech/documentation/inference/inference-bm25/)
- [Qdrant: Hybrid Queries e RRF](https://qdrant.tech/documentation/search/hybrid-queries/)
- [Qdrant: aggiornamento dello schema dei vettori](https://qdrant.tech/documentation/manage-data/collections/#update-vector-schema)
- [Qdrant: payload indexing](https://qdrant.tech/documentation/manage-data/indexing/)
- [Qdrant: multitenancy](https://qdrant.tech/documentation/manage-data/multitenancy/)
## 10. Contratto di ricerca del modulo Evidence
Il workflow non costruisce query Qdrant. Usa una sola interfaccia concettuale:
```python
search(
query: str,
purpose: EvidencePurpose,
context: EvidenceSearchContext,
) -> EvidenceSearchOutcome
```
`EvidenceSearchContext` può specificare tabelle, colonne e concetti da aggiungere alla
query, oltre a vincoli espliciti su tipo, tabelle, colonne o concetti.
Il modulo rende domanda e contesto una sola volta nel formato `Domanda`, `Concetti`,
`Tabelle`, `Colonne` definito sopra e passa esattamente quel testo sia all'embedder dense
sia a `qdrant/bm25`.
`EvidenceSearchOutcome` distingue due stati:
- `available`, con la generazione interrogata e zero o più `EvidenceResult`;
- `unavailable`, senza risultati e con un codice di errore stabile e un messaggio
limitato.
Una lista vuota nello stato `available` significa che la ricerca ha funzionato ma non
ha trovato corrispondenze. Non equivale a un errore tecnico.
Il modulo Evidence possiede interamente:
- generazione della query dense;
- query BM25 con lingua coerente;
- filtri su workspace, revisione, generazione e purpose;
- RRF;
- vincoli espliciti su `kind` e `applies_to`;
- raggruppamento dei frammenti;
- risoluzione di provenienza e citazioni;
- controllo della revisione attiva.
Il workflow riceve candidati spiegabili, mai verità automatiche.
## 11. Inserimento nel workflow modulare ThothII
Evidence rimane un modulo autonomo con due responsabilità pubbliche.
### 11.1 Authoring
```text
prepare → validate → evaluate
```
Questa superficie è usata dal curatore e non dalle sessioni.
### 11.2 Runtime
```text
search → resolve citation → project into session
```
Evidence è un contributore degli stage esistenti, non uno stage aggiuntivo. Non emette
decisioni, non scrive gli artifact canonici e non modifica il ledger o lo stato del
workflow. Lo stage chiamante decide come usare i candidati restituiti.
L'integrazione usa l'identità semantica dello stage, non il display code:
| Stage semantico | Display code attuale | Evidence purpose |
|---|---:|---|
| `clarification` | F1 | `disambiguation` |
| `rewriting` | F3 | `rewriting` |
| `schema_linking` | F4 | `schema_linking` |
| `cte` | F6 | `sql_generation` |
| `final_sql` | F7 | `sql_generation` |
Lo stage `memory` (F2) usa il Memory Module. Lo stage `synthesis` (F5) verifica e
riassume lo schema linking già approvato e non avvia una nuova ricerca Evidence.
Ogni stage elencato esegue una ricerca indipendente con gli input disponibili in quel
momento. In particolare `cte` usa domanda riscritta e schema approvato, mentre
`final_sql` aggiunge il piano CTE approvato. La prima versione non introduce una cache
condivisa fra stage.
Il chiamante conserva nella sessione una Evidence receipt con stage, purpose,
generazione e ID restituiti. Il testo non viene copiato: rimane nel repository del
workspace e viene risolto attraverso la provenienza della Published Evidence.
Gli stage passano `purpose` e contesto al modulo, ma non conoscono collezioni, nomi di
vettori, generazioni o sintassi Qdrant.
Le istruzioni Pi relative alla consultazione delle Evidence vengono spostate in
frammenti del modulo Evidence e poi proiettate nel `SKILL.md` generato, seguendo il
meccanismo modulare già usato da Disambiguation e Memory.
## 12. Formule
Le formule approvate oggi presenti nello store `formulas/*.sql.md` vengono convertite in
Evidence `kind: formula`. Dopo la migrazione non esistono due archivi runtime.
Una nuova formula scoperta in F4 segue invece questo percorso:
```text
sessione → Formula proposal nell'artefatto di sessione
→ importazione di manutenzione
→ Curated Evidence formula
→ revisione Git
→ Published Evidence
```
Le decisioni `concept_formula_approved` e `concept_formula_rejected` continuano a
descrivere la scelta fatta nella singola sessione. Non equivalgono alla pubblicazione
globale nel workspace.
## 13. Comportamento in caso di errore
### 13.1 Durante l'authoring
- un file non UTF-8, troppo grande o strutturalmente invalido produce un errore chiaro;
- un dubbio semantico produce un `review_item`;
- un albero Git sporco impedisce la scrittura di nuove proposte;
- un sorgente rimosso produce un'unità orfana, non una cancellazione, e blocca la
pubblicazione finché il curatore non la risolve.
### 13.2 Durante l'indicizzazione
- la nuova generazione viene preparata senza toccare quella attiva;
- la generazione candidata viene interrogata esplicitamente per la valutazione senza
renderla visibile alle sessioni;
- una valutazione fallita lascia inattiva la candidata;
- un caricamento o una verifica falliti non cambiano il puntatore attivo;
- i dati parziali vengono rimossi quando possibile e comunque non sono leggibili dal
runtime perché manca l'attivazione.
Poiché la revisione del workspace viene attivata prima di costruire la candidata, il
runtime può attraversare una finestra di manutenzione in cui la vecchia generazione non
corrisponde alla revisione. In questa finestra la ricerca Evidence è `unavailable` e
blocca lo stage chiamante. La prima versione accetta questa degradazione fail-closed
invece di introdurre una transazione distribuita fra Git, registry e Qdrant. Se il gate
fallisce, l'operatore corregge il corpus oppure ripristina esplicitamente la revisione
precedente.
### 13.3 Durante una sessione
Una ricerca `available` senza corrispondenze produce una Evidence receipt vuota, viene
mostrata come tale e non impedisce allo stage di continuare.
Se Qdrant, il corpus attivo o la revisione attesa non sono disponibili:
- l'outcome è `unavailable`, non una lista vuota valida;
- viene restituito un codice stabile con un messaggio limitato;
- lo stage chiamante resta bloccato e può essere ritentato;
- non vengono usate revisioni precedenti;
- non viene ripetuta la ricerca con un purpose diverso.
Questo comportamento è fail-closed: un'assenza reale di corrispondenze non ferma il
workflow, mentre un guasto non viene mascherato come assenza di conoscenza.
## 14. Valutazione minima
`tht evidence evaluate` esegue le domande in `evaluation.yaml` contro una generazione
indicata esplicitamente oppure, per il monitoraggio ordinario, contro l'indice attivo.
Riporta almeno:
- quante domande hanno trovato una Evidence attesa nei primi 5 e nei primi 10 risultati;
- quali tipi attesi sono mancati;
- quali query non hanno prodotto risultati;
- per ogni Evidence attesa, la posizione nella graduatoria dense, BM25 e fused;
- revisione Git, generazione e configurazione di ricerca usate.
Il file classifica ogni domanda come `lexical`, `semantic` o `mixed` e contiene almeno
un caso per profilo. L'evaluator esegue i due rami anche separatamente per renderli
diagnosticabili, oltre alla ricerca ibrida usata dal runtime. La prima baseline deve
essere salvata prima di regolare pesi o introdurre altri modelli. Il comando non
modifica l'indice. La valutazione supera il gate minimo soltanto quando ogni query trova
almeno una delle Evidence attese nei primi dieci risultati fused. Le posizioni dei
singoli rami e `hit@5` restano informative e non bloccano la pubblicazione.
## 15. Comandi e responsabilità
| Comando | Dove opera | Scrive |
| --- | --- | --- |
| `tht evidence prepare <workspace-root> [--upgrade]` | clone Git di authoring | `curated/`, `manifest.yaml` |
| `tht evidence validate <workspace-root>` | clone Git o CI | nulla |
| `tht evidence resolve <id> (--retire | --source <path>)` | clone Git di authoring | unità interessata, `manifest.yaml` |
| `tht evidence evaluate ... [--generation <id>]` | generazione candidata o attiva | solo rapporto su stdout/JSON |
| `tht ... workspace preprocess evidence` | installazione/runtime | generazione candidata, poi attiva soltanto dopo il gate |
`prepare` non crea commit. `preprocess evidence` non modifica il repository Git.
## 16. Migrazione iniziale del workspace PSD
Il corpus attuale comprende 36 file Markdown organizzati in glossario, domini clinici,
enum, esempi NLQ, mapping e normalizzazione. La migrazione avviene così:
1. spostare gli originali sotto `evidence/source/`, conservandone la gerarchia;
2. eseguire `prepare` e generare `curated/`;
3. revisionare tutte le unità e risolvere i `review_items`;
4. importare eventuali formule approvate come `kind: formula`;
5. compilare circa venti query in `evaluation.yaml`;
6. configurare il descriptor con `patterns: ["curated/**/*.md"]`;
7. validare, fare merge e attivare la revisione in una finestra di manutenzione;
8. registrare conteggi e ID campione di Schema e Memory;
9. eseguire `preprocess evidence`, che aggiunge `bm25` senza ricreare la collezione e
costruisce la generazione candidata;
10. verificare che conteggi, ID campione e ricerche dense di Schema e Memory siano
invariati;
11. valutare la candidata e pubblicarla soltanto se supera il gate top-10;
12. salvare la baseline e svolgere una verifica umana degli stage `clarification`,
`rewriting`, `schema_linking`, `cte` e `final_sql`.
Non serve mantenere v1 e v2 attivi contemporaneamente nel runtime: Git conserva la
vecchia revisione e il meccanismo delle generazioni conserva il rollback dell'indice.
## 17. Criteri di accettazione
La prima versione è completa quando:
1. un sorgente poco strutturato produce una o più unità tipizzate senza perdere la
provenienza;
2. sorgenti invariati sono un no-op;
3. ogni modifica proposta a contenuti revisionati è recuperabile e visibile nel diff
Git prima della pubblicazione;
4. ogni unità cita brevi estratti verificabili del proprio sorgente;
5. gli ID nuovi sono assegnati dal codice e il modello può soltanto riutilizzare ID
precedenti esplicitamente forniti;
6. il batch di preparazione è tutto-o-niente e non esegue retry automatici;
7. un `review_item` impedisce l'indicizzazione;
8. un'unità orfana blocca la pubblicazione finché non viene risolta;
9. un'unità non più sostenuta dal proprio sorgente diventa una retirement candidate e
blocca la pubblicazione;
10. il comando `resolve` ritira o ricollega un'unità con un diff Git recuperabile;
11. tutte le otto varianti hanno validazione specifica e un solo `kind` primario;
12. una riclassificazione conserva l'ID e un rinominamento univoco del sorgente conserva
gli ID delle unità collegate;
13. ogni ID usa `evidence:<slug>`, non viene ricalcolato automaticamente e non contiene
il kind;
14. ogni Formula Evidence contiene una sola espressione PostgreSQL componibile;
15. le formule approvate sono ricercate tramite lo stesso modulo delle altre Evidence;
16. soltanto `curated/**/*.md` entra nel corpus runtime;
17. la collezione Qdrant conserva il dense predefinito e aggiunge `bm25` con IDF senza
ricostruzione distruttiva;
18. la Query API esegue i due prefetch e la fusione RRF;
19. risultati di frammenti della stessa unità diventano un solo Evidence Result con
estratti e riferimento al documento completo;
20. una ricerca disponibile può restituire zero Evidence, mentre una revisione o
generazione non corrispondente produce `unavailable` e blocca lo stage;
21. ogni ricerca applica il purpose come filtro obbligatorio e non usa bonus impliciti
per kind o ambito;
22. la generazione candidata diventa attiva soltanto quando ogni query recupera almeno
una Evidence attesa nei primi dieci risultati; il rapporto include anche `hit@5`;
23. i cinque stage mappati interrogano Evidence indipendentemente, mentre `memory` e
`synthesis` non lo invocano;
24. ogni ricerca disponibile conserva una ricevuta minima senza duplicare il testo;
25. una sessione completa riprende dopo la finestra fail-closed mediante retry o
rollback esplicito;
26. conteggi, ID campione e ricerche dense dimostrano che Schema e Memory non cambiano
durante l'upgrade;
27. una prova L0 dimostra `qdrant/bm25` sull'immagine locale effettivamente dichiarata;
28. dense e BM25 ricevono lo stesso testo di query deterministico;
29. nessun elemento atomico viene spezzato per rispettare la dimensione dei frammenti;
30. la valutazione copre casi lessicali, semantici e misti e mostra separatamente i tre
ranking;
31. il testo reso di ogni frammento rispetta il solo `max_chunk_chars` esistente;
32. la query conserva maiuscole, punteggiatura e spazi interni e non altera gli
identificatori PostgreSQL sensibili alle maiuscole;
33. documentazione e comandi descrivono lo stesso contratto.
## 18. Decisioni rinviate
Saranno considerate soltanto dopo la baseline:
- pesi RRF diversi da quelli predefiniti;
- reranking;
- ColBERT o multivettori;
- acquisizione automatica di PDF, Word, HTML o pagine web;
- creazione automatica di branch e pull request;
- fusione assistita di Evidence provenienti da sorgenti diversi.
Queste esclusioni mantengono la prima implementazione comprensibile, realizzabile,
manutenibile e documentabile.
docs/prd/2026-08-09-workspace-preprocessing-prd.md
-539
View File
@@ -1,539 +0,0 @@
# PRD — Preprocessing per-workspace su ThothII (Qdrant + Git workspace registry)
**Status:** baseline storica dei requisiti — implementazione completata; per i contratti correnti vedere
`docs/contracts/workspace-preprocessing-cli.md`, `docs/contracts/workspace-evidence-v3.md` e
`docs/evidence.md`
**Data:** 2026-08-09
**Autore:** analisi dello stato attuale (branch `codex/git-workspace-registry`) + decisioni con il proprietario
**Uso:** riferimento stabile delle decisioni originarie; questo documento non è un piano operativo
---
> **Aggiornamento P1.1 (2026-08-11):** il layout del repository registry descritto nelle sezioni
> attive di questo PRD segue il contratto P1.1 accettato: catalogo di root `thoth-workspaces.yaml`,
> descriptor `<id>/workspace.yaml`, evidence embedded `<id>/evidence/`, annotazioni FK curate
> `<id>/schema/annotations.yaml` (P5), docs generate `workspace-docs/<id>/`. I vecchi percorsi
> piatti (`workspaces/<id>.yaml`, `workspace-content/<id>/evidence/`) sono superseded; le uniche
> occorrenze rimaste sono storiche (changelog/revisioni). Il contratto corrente è descritto in
> `docs/contracts/workspace-evidence-v3.md`.
---
## 1. Contesto
ThothII è passato da un indice semantico **pgvector sul server PSD** (descrizioni di tabelle/colonne,
evidence e memory embeddate nella stessa istanza Postgres del DWH, lettura via RPC `search_similar`,
scrittura via REST dedicato o loading diretto) a un'architettura con:
- **infrastruttura semantica interna obbligatoria**: Qdrant + Ollama (`qwen3-embedding:0.6b`, 1024 dim,
cosine) come servizi Compose privati; una **collection Qdrant per workspace**, con schema/evidence/memory
separati dal payload `kind`;
- **workspace definito da un descriptor schema-v3 in un repo Git esterno** (id, DWH, collection, LLM
policy, diagnostics), con binding DWH locali all'installazione;
- **config harness renderizzata dal backend** a runtime (`runtime_identity` + `resources.vector` +
`resources.embeddings` + `roots` sotto `<dataRoot>/sessions/<wsId>/`).
La **macchina di preprocessing** (comandi, job a generazioni con publish atomico, adapter Qdrant, corpus
evidence, FK, memory) **esiste ed è testata**. La superficie operativa corrente è il comando host
`tht --installation ... workspace preprocess ...`, che esegue il servizio profile-gated
`workspace-maintenance`; le vecchie fixture Compose dedicate sono state ritirate.
### Il problema
Il preprocessing **non è collegato al workspace reale del registry**:
1. i job fixture usano una collection fissa (`preprocess-evidence`), un workspace_id derivato dal nome
file (`preprocess-evidence`) e roots sotto `/data/workspaces/preprocess-*`, che **non coincidono** con
quelli del runtime (`<dataRoot>/sessions/<wsId>/`);
2. il backend **non espone alcun modo** di eseguire `tht preprocess` contro la config renderizzata di un
workspace (nessun endpoint, nessuno script, nessun comando documentato);
3. il **descriptor v3 e la config renderizzata non hanno la sezione `evidence`**: non c'è un posto canonico
dove dichiarare da dove arrivano le evidence di un workspace;
4. l'**ammissione sessione** (`ThtRunner.qdrantEnsure`) richiede la collection già esistente con 1024/cosine
e 8 payload keyword-index, ma **nessuno la crea esplicitamente** (`tht vector init` fallisce se manca);
5. le **generazioni `.tht-dwh` sono legate a un fingerprint della config completa** (`OWNER.json`:
workspace_id + config_fingerprint + input_fingerprint): se il preprocessing non usa la config identica a
quella renderizzata dal runtime, a runtime la generazione viene **rifiutata**;
6. la **cura FK** (`annotations.yaml`) è manuale e vive nel runtime artifacts; il registry **non sincronizza**
file dal repo ai roots runtime;
7. i **vecchi embedding pgvector (nomic 768d) non sono riusabili** (modello e dimensioni cambiati): serve
re-indicizzare i contenuti PSD.
**Sintesi:** la parte "motore" è pronta; manca il **collegamento per-workspace** (config, esecuzione,
bootstrap, sorgente evidence) e la **documentazione operator**.
---
## 2. Obiettivo
Rendere l'attuale versione di ThothII (Qdrant + workspace su repo esterno) **configurabile e utilizzabile**
per un workspace reale, inclusa l'intera catena di preprocessing: **tabelle/colonne (catalogo + embedding),
FK (cura), evidence (sorgente → corpus → embedding), memory/solved**, con un flusso operator riproducibile,
documentato e verificato da smoke end-to-end.
### Obiettivi secondari
- O1. Un solo modo canonico di eseguire il preprocessing per un workspace (niente più fixture "speciali").
- O2. Il preprocessing è **idempotente e ripristinabile**: rerun senza duplicati, publish atomico, GC.
- O3. Nessun segreto/endpoint entra nel repository registry né nei descriptor (invariante attuale preservato).
- O4. Il flusso è **documentato nei manuali operator** (`local/server-workspace-registry.md`) e coperto da
smoke automatici.
- O5. La **migrazione PSD** è definita (cosa si riusa, cosa si rigenera, cosa si esporta dal pgvector).
- O6. Ogni piano tecnico definisce, dove applicabile, un **goal automatico di processo completo**: da stato
pulito costruisce un ambiente isolato, simula il flusso end-to-end entro lo scope del piano e lo porta a
successo con un integration test riproducibile.
- O7. Dopo il successo automatico, un **percorso manuale separato** permette al reviewer di ripetere il
processo attraverso le interfacce reali, comprenderne l'architettura e approvare gli artefatti.
## 3. Non-obiettivi (fuori scope di questo PRD)
- Riscrivere il workflow NL→SQL o i gate (F1..F8) — restano invariati.
- Cambiare modello/architettura semantica (Qdrant/Ollama/1024/cosine) — già deciso e verificato.
- Rifare la UI di gestione workspace oltre a quanto già esiste.
- Il **deploy reale sul server PSD** (VPN, credenziali, portale, auth upstream): è un progetto operativo
separato che userà questo PRD come prerequisito tecnico.
- Supportare di nuovo pgvector o endpoint embedding esterni come percorso operativo.
- Il **comando di preprocessing avviabile dalla GUI**: è una release futura (fuori scope della release 0,
che è CLI sul host — vedi D2).
---
## 4. Utenti
| Utente | Esigenza |
| --- | --- |
| **Operatore/amministratore** (chi installa e cura un workspace) | Configurare DWH+evidence, eseguire il preprocessing, curare le FK, verificare lo stato, fare backup/restore. |
| **Autore ETL / curatore dominio** (es. il cliente PSD) | Mantenere evidence e annotazioni FK nel namespace del workspace nel repository registry con un flusso semplice. |
| **Reviewer umano** (usa l'app) | Vede search pack con tabelle/evidence/solved corretti: la qualità del retrieval dipende dal preprocessing. |
| **Sviluppatore ThothII** | Comandi/endpoint deterministici, testabili, senza sorprese di configurazione. |
---
## 5. Scenario target (end-to-end)
1. **Setup repo**: l'operatore usa un **unico repository Git registry** per tutti i workspace e crea
`<id>/workspace.yaml` (schema-v3: DWH, collection, LLM policy) insieme al tree curato
`<id>/evidence/`; descriptor e contenuti sono pubblicati nello stesso commit.
2. **Installazione**: `.env` + bindings `THT_WS_*` + secrets; `up` dello stack (frontend/core/qdrant/embedding).
3. **Registry**: pull → validazione → snapshot attivo; diagnostic DWH verdi.
4. **Preprocessing DWH**: introspezione (physical.yaml: tabelle/colonne/descrizioni/esempi/eligibility) +
LSH; generazione pubblicata sotto `.tht-dwh` del workspace.
5. **Cura FK**: `tht schema suggest-fks` → revisione umana → `annotations.yaml` (check senza orfani);
versionata dove deciso (vedi D5).
6. **Indice schema**: `tht vector index-schema` → record schema nella collection del workspace; la
collection, se inesistente, viene creata all'ammissione (self-heal) o dal primo write (vedi D4).
7. **Preprocessing evidence**: `tht preprocess evidence` → corpus generation (chunk+embed) nella collection
(kind `evidence`) + manifest ACTIVE nel corpus root del workspace.
8. **Memory/solved**: promozioni F8 (`memory promote/save-one`) e finalize (`solved-index`) scrivono nella
collection (kind `memory`).
9. **Uso**: nuova sessione → admission verde (collection+Ollama) → F1 `search pack` con tabelle, evidence e
solved del workspace; F4/F6 con FK curate.
10. **Operatività**: backup/restore volumi (Qdrant, corpus, `.tht-dwh`, registry), update, ripristino da
outage.
---
## 6. Requisiti funzionali
### RF1 — Configurazione per-workspace
- RF1.1 Un workspace del registry deve poter dichiarare **tutto ciò che serve al preprocessing** in un unico
posto canonico: DWH (già nel descriptor), **sorgente evidence completa** (protocollo/tipo, URI, parametri
non-secret), eventuali policy di chunk/retention.
- RF1.2 I segreti (password, API key, CA) restano fuori dal repo e dal descriptor (invariante attuale).
- RF1.3 La config harness usata dal preprocessing deve essere **derivata dalla stessa renderizzazione del
runtime** (stesso workspace_id, stessi roots, stessa configurazione effettiva).
- RF1.4 La configurazione (descriptor + bindings + config renderizzata) deve **prevedere i tre trasporti
DWH**: `postgres_direct`, `rest_api`, `ssh_tunnel`. PSD usa `rest_api`; altri database potranno usare
direct o tunnel.
- RF1.5 Il registry usa **un unico repository Git** per più workspace. Per una sorgente evidence
`filesystem`, l'URI è relativa alla root del repository ed è confinata lessicalmente a
`<workspace_id>/`; path assoluti, traversal (`..`) e riferimenti al namespace di un
altro workspace sono invalidi. Descriptor e sorgente devono essere risolti dalla **stessa revisione Git**.
### RF2 — Preprocessing DWH (tabelle/colonne)
- RF2.1 Comando/azione per eseguire `introspect` + `lsh` per un workspace del registry, contro la sua config
effettiva, con output JSON e resume.
- RF2.2 La CLI di preprocessing raggiunge il DWH **con il trasporto dichiarato dal workspace** (direct,
REST o tunnel SSH), come il runtime.
- RF2.3 Il catalogo risultante (`physical.yaml`) alimenta: cache `tht schema introspect`, render mschema
(F1/F4), record schema per l'embedding (RF4).
- RF2.4 Refreshing esplicito quando il DWH cambia (`--refresh`/nuova generazione), senza invalidare le
sessioni esistenti (generazioni + ACTIVE pointer, già implementato).
### RF3 — FK
- RF3.1 Flusso curato per-workspace: `tht schema suggest-fks` (+ `--from-sql`, `--assume`, `--write`),
revisione umana, `tht schema check` (zero orfani).
- RF3.2 Le FK curate devono essere **disponibili a runtime** (sezione `【Foreign keys】` del render mschema,
usata da F4/F6) e **versionate nel repository registry** (D5).
- RF3.3 Nessuna FK derivata dal modello: il modello usa solo la lista curata (contratto SKILL invariato).
### RF4 — Indice semantico schema + bootstrap collection
- RF4.1 `tht vector index-schema` embedda i record schema (tabella+colonna, con descrizioni/esempi/sinonimi)
nella collection del workspace (kind `schema`), idempotente (hash → upsert solo del cambiato).
- RF4.2 **Bootstrap della collection**: se inesistente all'ammissione sessione, il runtime la crea
(self-heal) con 1024/cosine + i payload keyword-index richiesti (`content_hash, document_id, kind,
record_key, record_kind, vector_generation, workspace_id, workspace_revision`).
- RF4.3 La **CLI deve poter cancellare e ricreare** la collection di un workspace (rebuild esplicito con
guardie di sicurezza e conferma).
- RF4.4 Prima di una sessione, l'ammissione resta invariata (collection compatibile + Ollama).
### RF5 — Evidence
- RF5.1 Sorgente evidence dichiarabile per-workspace nel descriptor (protocollo/tipo + URI). Per PSD è un
**tree di file `.md` versionato nell'unico repository registry**, sotto
`psd/evidence/`; in generale ogni workspace usa
`<workspace_id>/evidence/`. HTTP manifest e S3 restano opzioni del motore per sorgenti
esterne.
- RF5.2 `tht preprocess evidence` per-workspace: discover → acquire → normalize/chunk → embed → upsert
(kind `evidence`, payload `document_id`/`vector_generation`) → publish ACTIVE nel corpus root del workspace,
con resume e dry-run (già implementato nel motore).
- RF5.3 GC/retention delle generazioni evidence (filesystem + punti Qdrant) con le policy esistenti.
- RF5.4 A runtime la ricerca evidence è filtrata dalla generazione ACTIVE e dal workspace_id (già
implementato: `ActiveEvidenceSearcher`); il flusso RF5 deve garantire che il corpus ACTIVE appartenga al
workspace giusto.
### RF6 — Memory e domande risolte
- RF6.1 `memory promote/save-one` (F8) e `memory solved-index` (finalize) scrivono nella collection del
workspace (kind `memory`/`solved_question`) — verificare end-to-end con Qdrant e risolvere i TODO residui
in `memory_cmd.py`.
- RF6.2 Il registro JSONL resta la fonte canonica; Qdrant è proiezione di ricerca (invariante attuale).
### RF7 — Migrazione PSD
- RF7.1 Definire cosa si **riusa** (physical.yaml, annotations.yaml con le ~228 FK curate, le 895 evidence
`.md`), cosa si **rigenera** (tutti gli embedding, modello diverso) e cosa si **esporta** dal pgvector del
server prima della dismissione.
- RF7.2 La migrazione è un'operazione documentata e rieseguibile, non un one-shot nel codice.
### RF8 — Operatività e documentazione
- RF8.1 Manuali operator aggiornati con la sequenza completa per-workspace (config → preprocess → cura →
verifica → uso → backup/restore).
- RF8.2 La documentazione di progetto spiega **cos'è `.tht-dwh`** (generazioni, `OWNER.json`, `ACTIVE`,
vincolo di fingerprint) in modo comprensibile per l'operatore (D3).
- RF8.3 Smoke end-to-end automatico (workspace nuovo → tutto il ciclo → sessione reale → cleanup).
- RF8.4 Backup/restore coprono Qdrant (già `vector-backup.sh`/`vector-restore.sh`), corpus, `.tht-dwh` e
registry.
- RF8.5 Ogni piano successivo traduce il proprio risultato operativo in un **process goal** verificabile da
un integration test completo per quello scope; eventuali interventi umani iniziali o intermedi sono
ammessi solo se inevitabili, espliciti, documentati e riprendibili.
- RF8.6 Ogni process goal automatico riuscito è seguito, quando utile, da un walkthrough manuale su un
ambiente nuovo e separato; automazione e accettazione umana producono evidenze distinte.
---
## 7. Requisiti non funzionali
- **RNF1 Sicurezza**: nessun segreto in repo/descriptor/config renderizzata/log; la CLI di preprocessing
(D2) non espone credenziali, non le logga e non le scrive negli artefatti.
- **RNF2 Determinismo/idempotenza**: rerun del preprocessing = zero duplicati (hash content), publish
atomico, generazioni immutabili (già nel motore).
- **RNF3 Robustezza**: degradazione controllata (workspace senza evidence o senza collection funziona, con
warning); errori sanitizzati; nessun fallimento che corrompa la generazione attiva.
- **RNF4 Isolamento per-workspace**: ogni filtro Qdrant legato a workspace_id; rifiuto di namespace
conflittuali (già implementato nell'adapter).
- **RNF5 Compatibilità**: il preprocessing deve funzionare con la config renderizzata dal backend
(fingerprint `OWNER.json` compatibile) — è il vincolo chiave di design (vedi D3).
- **RNF6 Performance**: introspezione ~minuti (non nel path di sessione), embedding batch, LSH boundato;
il retrieval a runtime non cambia i costi attuali.
- **RNF7 Manutenibilità**: nessun fork dei fixture; un solo percorso canonico (O1).
- **RNF8 Integration-first**: il successo di un piano tecnico richiede un'esecuzione completa da ambiente
pulito, senza retry automatici che mascherino errori; ogni fallimento viene diagnosticato, corretto alla
radice e seguito da una nuova esecuzione completa.
- **RNF9 Evidenza e cleanup**: ogni ambiente simulato ha identità/ownership esplicita, risorse univoche,
segreti fittizi, report machine-readable e leggibile, scansione anti-secret e cleanup confinato alle sole
risorse possedute dal run.
---
## 8. Standard di esecuzione e verifica — integration-first
Questo standard si applica a P1 e, **ovunque sia tecnicamente significativo**, a tutti i piani successivi.
Un piano che non possa applicarlo deve motivare esplicitamente l'eccezione e definire il verifier più vicino
possibile al processo reale.
### S1 — Goal automatico di processo
- Ogni piano definisce il **processo completo entro il proprio scope**, con punto iniziale pulito, input,
componenti attraversati, risultato osservabile e criteri di successo.
- Il goal non è "far passare alcuni test", ma **simulare con successo il processo operativo** che la feature
deve rendere possibile. Per P1 il confine completo è Git → registry → API → snapshot/docs → render →
`tht config check`; estrazione evidence e Qdrant appartengono ai piani successivi.
- Durante l'esecuzione il goal resta aperto fino a una prova integrale verde. Se l'ambiente agentico supporta
goal persistenti, l'esecutore lo registra all'inizio e lo completa soltanto dopo l'evidenza finale.
### S2 — Ambiente di integrazione isolato
- Il test costruisce dipendenze controllate sotto `.artifacts/<plan>/<run-id>/`: repository Git simulati,
checkout, roots runtime, secret fixture, richieste/risposte, log e output.
- Ogni run usa identità e nomi univoci e un manifest di ownership; non usa credenziali, repository o dati
reali salvo quando il piano dichiara esplicitamente un gate L2.
- I servizi reali appartenenti allo scope vengono attraversati tramite le loro interfacce normali; quelli
esterni o non ancora nello scope sono sostituiti da fixture fedeli e deterministiche.
### S3 — Contratto di successo
- Il test parte da stato pulito, esegue il processo una volta senza retry automatici, termina con exit code
zero e produce `report.json` più un report leggibile.
- Un fallimento richiede diagnosi della causa, test regressivo/correzione e una nuova esecuzione completa da
stato pulito; ripetere alla cieca non costituisce progresso verso il goal.
- Il gate finale comprende determinismo/idempotenza pertinenti, scansione anti-secret, verifica degli
artefatti e prova del cleanup confinato. Gli artefatti possono essere conservati con `--keep` per review.
### S4 — Interventi umani inevitabili
- Passi umani iniziali o intermedi sono ammessi solo quando non simulabili in modo affidabile (per esempio
accesso approvato a un sistema reale o review di contenuto curato).
- Ogni passo umano dichiara precondizioni, istruzioni, evidenza richiesta, criterio di decisione e checkpoint
di ripresa; l'automazione copre e verifica tutto ciò che precede e segue il checkpoint.
- Un intervento umano non può essere sostituito da un'assunzione silenziosa né rendere non riproducibile il
resto del processo.
### S5 — Walkthrough manuale successivo
- Dopo il goal automatico verde, il reviewer ripete il processo in un **ambiente nuovo e separato**, usando
le interfacce reali e una guida passo-passo che spiega componente, stato letto, artefatto prodotto e
invariante verificata.
- Il walkthrough serve a comprensione architetturale e accettazione; non sostituisce l'integration test e non
ne riusa lo stato già mutato.
- Lo stato di consegna distingue almeno `automated integration: PASS` e `manual acceptance: PENDING/PASS`.
Un piano non è pienamente accettato finché l'eventuale gate manuale richiesto non è stato deciso dal reviewer.
### S6 — Contenuto obbligatorio dei piani
Ogni piano tecnico riporta, adattandoli al proprio scope:
1. **Automated process goal** e comando unico di esecuzione;
2. topologia dell'ambiente simulato e confini delle dipendenze;
3. asserzioni del full integration test e contratto del report;
4. checkpoint umani inevitabili, oppure dichiarazione esplicita che non ve ne sono;
5. walkthrough/gate manuale successivo, quando utile;
6. evidenze di completamento, retention degli artefatti e cleanup esatto.
---
## 9. Criteri di accettazione (bozza)
1. Da un repository registry vuoto si arriva a una sessione funzionante seguendo **solo i manuali aggiornati**,
senza toccare file fixture.
2. La CLI di preprocessing funziona **sia sul PC/Mac dell'utente sia sul server che ospita il DWH**
(stesso comando, config derivata dal workspace).
3. La configurazione di un workspace dichiara e usa uno dei **tre trasporti DWH** (`postgres_direct`,
`rest_api`, `ssh_tunnel`); PSD usa `rest_api`.
4. Il goal automatico P1 costruisce da zero repository Git simulati e ambiente isolato, attraversa con
HTTP reale il processo Git → registry → validate/publish/read/export → snapshot/docs → render →
`tht config check`, supera casi positivi e negativi senza retry e produce report/artefatti secret-free.
5. Solo dopo il punto 4, un ambiente manuale nuovo avvia il backend su `127.0.0.1:8791` e permette al
reviewer di ripetere ogni chiamata e ispezionare commit, snapshot, ZIP e config renderizzate seguendo una
guida; il gate resta `PENDING` finché il reviewer non lo approva.
6. `search pack` di una domanda reale restituisce tabelle (con descrizioni), evidence della generazione
ACTIVE e solved dello stesso workspace; F4/F6 mostrano le FK curate.
7. `qdrantEnsure`/`ollamaEnsure` verdi all'ammissione; la collection ha esattamente 1024/cosine + gli 8
keyword-index.
8. Rerun del preprocessing: `unchanged` (nessun duplicato); modifica di un'evidence → nuova generazione,
ACTIVE aggiornato, vecchie generazioni in GC.
9. Smoke end-to-end automatico verde in CI con cleanup esatto.
10. Migrazione PSD documentata e provata almeno in dry-run (re-introspection o riuso catalogo + re-embedding).
11. Ogni piano tecnico successivo include un process goal automatico completo per il proprio scope e un
walkthrough manuale quando utile, oppure documenta l'inevitabile eccezione umana secondo S4.
---
## 10. Decisioni chiuse (2026-08-09)
> La sezione nasceva come "punti di discussione"; le decisioni sono state prese con il proprietario del
> prodotto il 2026-08-09. Ogni punto resta il riferimento del proprio piano (sez. 11). Le opzioni scartate
> sono omesse; la motivazione della scelta è inclusa in ogni punto.
### D1 — Config per-workspace: **c) misto, con sorgente completa nel descriptor**
- Il descriptor v3 guadagna una sezione `evidence` che configura **tutta la lettura della sorgente**:
protocollo/tipo, URI/sorgente, eventuali parametri non-secret.
- Si usa **un unico repository Git registry** per tutti i workspace. Ogni workspace possiede il proprio tree
versionato sotto `<workspace_id>/evidence/`; per PSD il path canonico è
`psd/evidence/`.
- Per `filesystem`, l'URI del descriptor è repo-relative, confinata al namespace dello stesso workspace e
risolta dalla stessa revisione Git del descriptor. Sono vietati path assoluti, traversal e riferimenti al
contenuto di un altro workspace; il controllo reale di symlink/containment durante la materializzazione
appartiene a P6.
- Eventuali segreti (HTTP autenticato, S3) restano in overlay d'installazione — invariante: nessun segreto
nel repo/descriptor.
- P1 applica lo standard integration-first: prima persegue un goal automatico Git→registry→HTTP→render→
harness sotto `.artifacts/p1-integration/<run-id>/`, poi offre un walkthrough manuale separato sotto
`.artifacts/manual-acceptance/p1/`. Non include ancora estrazione, embedding o verifica degli artefatti
`artifacts/evidence` (P2+P6).
### D2 — Esecuzione: **CLI sul host in release 0; GUI in release futura**
- **Release 0**: una **CLI installata con ThothII sul host** — sia il PC/Mac dell'utente sia il server che
ospita il DWH — che esegue **tutta la catena di preprocessing** (DWH introspect+LSH, FK, index-schema,
evidence e quanto serve) per un workspace del registry.
- La CLI deriva la config dal descriptor+bindings con la stessa identità del runtime → soddisfa il vincolo
D3 senza dipendere dal backend.
- **Release futura (fuori scope)**: comando avviabile dalla GUI (endpoint backend da progettare poi).
### D3 — Fingerprint `.tht-dwh`: **accettare il vincolo + documentarlo**
- Le generazioni DWH restano legate alla config effettiva (workspace_id + config_fingerprint +
input_fingerprint in `OWNER.json`).
- La CLI (D2) gira con la config derivata dal descriptor+bindings, quindi identica alla runtime.
- **La documentazione di progetto deve spiegare chiaramente cos'è `.tht-dwh`** (directory delle generazioni
catalogo/LSH, `OWNER.json`, `ACTIVE` pointer, perché il fingerprint protegge da artefatti di un'altra
config) — oggi non è chiaro.
### D4 — Bootstrap collection: **self-heal all'ammissione + CLI delete/recreate**
- Se la collection non esiste all'ammissione sessione, il runtime la crea (1024/cosine + payload
keyword-index) — self-heal.
- La **CLI deve poter cancellare e ricreare le collection** (rebuild esplicito, con guardie di sicurezza).
### D5 — Versioning artifacts curati: **c) misto**
- `annotations.yaml` (cura FK, cura umana) **versionata nel repository registry** e sincronizzata ai roots
runtime (il registry copia il file negli snapshots → sync).
- `physical.yaml` (derivato dall'introspezione) rigenerato localmente, non versionato.
### D6 — Evidence: **a) nell'unico repository registry, con namespace per-workspace**
- Ogni workspace contiene il proprio tree versionato sotto
`<workspace_id>/evidence/`; le dimensioni non sono un vincolo.
- P6 materializza il tree dalla **stessa revisione Git** del descriptor, verifica il containment reale
(inclusi i symlink) e lo rende disponibile al preprocessing senza usare un checkout mobile.
- HTTP/S3 restano opzioni future per sorgenti esterne (il motore le supporta già).
### D7 — Migrazione PSD: **inclusa, con accesso al server**
- Il piano P7 copre: riuso di physical.yaml + annotations.yaml + evidence `.md` dal repository registry; export
dal pgvector del server PSD (accesso disponibile); re-embedding con `qwen3-embedding:0.6b`; dry-run
documentato.
### D8 — Verifica end-to-end: **integration-first + walkthrough manuale; remote Git libero; DWH multi-trasporto**
- Ogni fase adotta lo standard della sez. 8: prima un process goal automatico da ambiente pulito, poi —
quando utile o richiesto — un walkthrough manuale su stato separato. P1 è il primo riferimento concreto.
- Il livello finale del PRD resta: smoke automatico su workspace sintetico (CI) + gate manuale L2 su PSD.
- Il namespace PSD sarà **prima alimentato nel repository registry** (descriptor + evidence + annotations
nello stesso flusso Git), **poi** usato da ThothII. Accesso al server PSD disponibile.
- **Remote Git**: lo creiamo noi, nessun vincolo tecnico (consigliato GitHub via HTTPS; SSH resta
possibile se servirà).
- **Trasporto DWH**: PSD via **REST** (come oggi); **la configurazione deve prevedere le tre modalità** —
`rest_api`, `postgres_direct`, `ssh_tunnel` — perché altri database potrebbero richiedere accesso TCP
diretto o via tunnel. Oggi `ssh_tunnel` è solo diagnostico a runtime: va reso operativo dove serve
(vedi P10).
### D9 — Retention/GC: **confermata**
- Default invariati (`retain_published_generations: 3`, chunk 4000 char), configurabili per-workspace via
la sezione `evidence`/policy del descriptor (D1).
## 11. Mappa storica dell'implementazione
L'implementazione è stata suddivisa nei workstream P1–P10 riportati sotto. I piani esecutivi
superati sono disponibili nella storia Git; questa tabella conserva soltanto la relazione tra
requisiti, dipendenze e risultati attesi.
| Piano | Punto PRD | Contenuto sintetico | Dipende da |
| --- | --- | --- | --- |
| P1 | D1 | Descriptor v3: sezione `evidence` (protocollo/tipo, URI repo-relative sotto `<id>/evidence/`) + policy e isolamento namespace; goal automatico Git→registry→HTTP→render→harness, seguito da walkthrough manuale | — |
| P2 | D2 | **CLI di preprocessing sul host (release 0)**: comando per-workspace che esegue l'intera catena (DWH, FK, index-schema, evidence) con la config derivata da descriptor+bindings; funziona su PC/Mac utente e server DWH | P1 |
| P3 | D3 | Vincolo fingerprint `.tht-dwh` (test: preprocess con config identica alla runtime) + **documentazione di progetto su cos'è `.tht-dwh`** | P2 |
| P4 | D4 | Bootstrap collection: **self-heal all'ammissione** (creazione 1024/cosine + keyword-index) + **comandi CLI delete/recreate** con guardie | — |
| P5 | D5 | `annotations.yaml` versionata nel repository registry + sync registry → roots runtime | P1 |
| P6 | D6 | Materializzazione del tree `<id>/evidence/` dalla revisione Git fissata → preprocess; containment reale e protezione da symlink escape | P1 |
| P7 | D7 | Migrazione PSD: riuso catalogo/annotations/evidence, **export pgvector (accesso server)**, re-embedding, dry-run | P1–P6 |
| P8 | D8 | Verifica end-to-end: smoke CI + gate L2 su PSD (**namespace PSD nel repository registry alimentato prima dell'uso**; remote Git a scelta) | P1–P7 |
| P9 | D9 | GC/retention per-workspace: policy configurabili, default invariati | P1 |
| P10 | D8/RF1.4 | Trasporti DWH operativi: rendere `ssh_tunnel` utilizzabile a runtime e nella CLI di preprocessing (oggi solo diagnostico); verifica dei tre trasporti (direct, REST, tunnel) | P1, P2 |
### Standard di verifica obbligatorio del futuro piano P1
P1 è il primo piano che applica integralmente la sez. 8 e deve contenere due task/gate distinti e ordinati.
#### 1. Automated integration goal — complete P1 configuration process
Un comando unico (nome definitivo nel piano, interfaccia indicativa
`./scripts/p1-acceptance.sh integration --keep`) costruisce da zero:
```text
.artifacts/p1-integration/<run-id>/
├── ownership.json
├── remote.git/ # remote bare locale
├── author/ # clone curatore + tree evidence
├── installation/ # checkout, snapshot e stato registry
├── runtime-data/
├── fixture-secrets/
├── requests/ # payload HTTP positivi/negativi
├── responses/
├── rendered/
├── exports/
├── logs/
├── report.json
└── report.md
```
Il test attraversa le interfacce reali appartenenti a P1: Git reale locale, backend Fastify su una porta
loopback temporanea, route HTTP validate/publish/pull/read/export, snapshot/docs/contract, renderer di
produzione e `tht config check`. Verifica anche stessa revisione Git per descriptor/tree, determinismo,
path/protocolli/secret fields invalidi, assenza di leak e cleanup confinato. Non usa frontend, Docker, DWH,
Qdrant o Ollama perché non appartengono allo scope P1.
L'esecuzione non applica retry automatici. In caso di errore l'esecutore diagnostica, aggiunge la copertura
regressiva necessaria, corregge e rilancia l'intero scenario da una nuova root pulita. Il goal è raggiunto
solo con exit code zero e report integralmente verde; con `--keep` le evidenze restano disponibili.
#### 2. Manual acceptance gate — descriptor and rendered configuration artifacts
Dopo il goal automatico verde, il piano prepara uno stato nuovo e indipendente sotto:
```text
.artifacts/manual-acceptance/p1/
├── remote.git/
├── author/
├── installation/
├── runtime-data/
├── requests/
├── responses/
├── output/
├── logs/
└── GUIDE.md
```
Un helper esegue soltanto `prepare/serve/stop/cleanup`; `serve` avvia il backend reale sull'host, senza
Docker e senza frontend, vincolato a `127.0.0.1:8791`. Il reviewer segue `GUIDE.md` ed esegue personalmente
le chiamate HTTP, i comandi Git, l'export ZIP, il doppio rendering, il confronto e `tht config check`, poi
prova i casi invalidi e decide il gate.
Il gate verifica manualmente: sezione `evidence`; pubblicazione/rilettura; commit e snapshot immutabile;
workspace docs/contract; config harness; assenza di segreti; sicurezza protocollo/path e isolamento
cross-workspace; output deterministico. Gli artefatti restano fino alla decisione e il cleanup rimuove solo
la root posseduta dal test.
P1 **non** dichiara di aver generato o validato `artifacts/evidence`: estrazione e mirroring richiedono
P2+P6; record Qdrant, embedding, generazioni ACTIVE e retention appartengono ai piani successivi. Dopo il
goal automatico lo stato è `automated integration: PASS / manual acceptance: PENDING`; P1 diventa pienamente
accettato soltanto dopo la decisione del reviewer.
Ordine consigliato: **P1 → P2 → P3** (catena config/esecuzione), **P4** e **P5/P6** in parallelo dopo P1,
poi **P7 → P8**; P9 può essere assorbito in P1 o restare autonomo; **P10** dopo P1+P2 (necessario solo se un
workspace target richiede davvero il tunnel — per PSD non serve, usa REST).
Ogni piano segue la prassi del repo: TDD, commit scoping, verifica layer (pytest/vitest/tsc/build) e lo
standard della sez. 8; gate deployment e smoke Docker si aggiungono quando appartengono allo scope. Lo stato
traccia separatamente implementazione, automated integration e manual acceptance.
---
## 12. Storico revisioni
| Versione | Data | Contenuto |
| --- | --- | --- |
| v0.1 | 2026-08-09 | Bozza da analisi dello stato attuale (gap preprocessing per-workspace) |
| v0.2 | 2026-08-09 | Decisioni D1–D9 chiuse con il proprietario; mappa piani P1–P10; requisiti RF1–RF8 aggiornati (evidence nel descriptor, CLI sul host, self-heal collection, multi-trasporto DWH) |
| v0.3 | 2026-08-09 | Revisione di coerenza (numerazioni, riferimenti incrociati, header di stato) — pronto per revisione del proprietario |
| v0.4 | 2026-08-09 | D1/D6: repository registry unico, namespace `workspace-content/<id>/evidence/` *(percorso storico P1, superseded da P1.1)*, pin alla stessa revisione Git e gate manuale P1 con remote locale usa-e-getta sotto `.artifacts/` |
| v0.5 | 2026-08-09 | Standard integration-first per P1–P10: process goal automatico completo da ambiente simulato e pulito, gestione esplicita degli interventi umani inevitabili e walkthrough manuale successivo su stato separato |
---
## 13. Riferimenti
- Stato attuale: `PROJECT_STATE.md` (sezioni "Internal Qdrant + Ollama semantic infrastructure", snapshot
registry) e `AGENTS.md`.
- Design architettura semantica: `docs/plans/2026-08-08-internal-qdrant-ollama-design.md` e relativo piano.
- Registry e Evidence: `docs/contracts/workspace-evidence-v3.md`, `docs/evidence.md`, manuali
`docs/install/local-workspace-registry.md` / `server-workspace-registry.md`.
- Motore preprocessing: `harness/tht/cli/preprocess_cmd.py`, `harness/tht/corpus/pipeline.py`,
`harness/tht/jobs/dwh_pipeline.py`, `harness/tht/adapters/vector/qdrant.py`,
`harness/tht/vectorstore/records.py`, `harness/tht/cli/{vector,schema,evidence,memory}_cmd.py`.
- Superficie operativa: `tools/tht/` e `docs/contracts/workspace-preprocessing-cli.md`.
- Ammissione runtime: `backend/src/tht/tht-runner.ts` (`qdrantEnsure`/`ollamaEnsure`),
`backend/src/workspaces/runtime-renderer.ts`.
docs/skill-tht-sessione.md
-22
View File
@@ -1,22 +0,0 @@
# Testo della skill `tht-sessione`
Questa pagina pubblica il testo completo della skill operativa usata dall'harness Pi.
La sorgente è [harness/.pi/skills/tht-sessione/SKILL.md](../harness/.pi/skills/tht-sessione/SKILL.md).
Il blocco seguente viene incluso direttamente dal file sorgente durante il build MkDocs: non è una copia manuale.
```mermaid
flowchart LR
QUESTION["New question"] --> F1["F1 clarify"]
F1 --> F2["F2 memory"]
F2 --> F3["F3 rewrite"]
F3 --> F4["F4 evidence"]
F4 --> F5["F5 schema"]
F5 --> F6["F6 SQL"]
F6 --> F7["F7 validation"]
F7 --> F8["F8 promotion"]
F7 --> F6
F8 --> FINAL["Finalized session"]
```
--8<-- "harness/.pi/skills/tht-sessione/SKILL.md"
docs/skills.md
+59 -189
View File
@@ -1,212 +1,82 @@
# Skill operative dell'applicazione
# Workflow operativo di ThothII
## Scopo di questa pagina
ThothII guida ogni domanda attraverso otto fasi. Il modello propone i passaggi, il revisore
decide nei gate e il sistema registra artefatti e decisioni persistenti.
Nel repository esistono diversi file denominati `SKILL.md`, ma non tutti appartengono al runtime di ThothII. La skill applicativa effettivamente usata dal workflow NL→SQL è:
```text
harness/.pi/skills/tht-sessione/SKILL.md
```mermaid
flowchart LR
Q["Domanda"] --> F1["F1 Chiarimento"]
F1 --> F2["F2 Memory"]
F2 --> F3["F3 Riscrittura"]
F3 --> F4["F4 Evidence"]
F4 --> F5["F5 Schema"]
F5 --> F6["F6 Piano CTE"]
F6 --> F7["F7 SQL"]
F7 --> F8["F8 Promozione"]
F8 --> DONE["Sessione finalizzata"]
```
I file presenti in `ChironeWp3/`, in `Thoth/ThothAI/` o nei worktree sono relativi ad altri progetti, strumenti o ambienti di sviluppo. Non fanno parte del contratto operativo di una sessione ThothII.
## Principi del workflow
## Che cos'è `tht-sessione`
- Il modello propone; il revisore approva, corregge o rifiuta.
- Ogni decisione rilevante viene registrata nel ledger della sessione.
- Lo stato persistito è la fonte di verità.
- Una fase può avanzare soltanto quando i suoi artefatti e gate sono completi.
- La ripresa ricostruisce il contesto dagli artefatti persistiti, non dalla conversazione.
La skill dichiara il nome `tht-sessione` e si descrive come orchestratore del workflow Thoth in otto fasi: chiarimento della domanda, memory, riscrittura, schema-linking, sintesi, piano CTE, SQL finale e datamart.
## F1 — Chiarimento
Non è una semplice raccolta di suggerimenti per il modello. È il contratto operativo che stabilisce:
Il sistema identifica l'ambiguità con maggiore impatto sul significato della domanda e presenta
una sola decisione per volta. Interpretazioni mutuamente esclusive usano una scelta singola;
risposte contemporaneamente valide usano una scelta multipla.
- quali passaggi devono essere eseguiti;
- quali comandi `tht` usare;
- quali decisioni richiedono il reviewer;
- quando una fase può avanzare;
- quali artefatti devono essere persistiti;
- quali decisioni sono locali alla domanda e quali possono essere riutilizzate.
## F2 — Memory
Il file stesso definisce questa regola: ogni comando, flag e comportamento necessario deve essere già descritto nella skill o nei suoi documenti di riferimento. Il modello non deve esplorare il codice sorgente per ricostruire il funzionamento degli strumenti.
Le Memory compatibili con la domanda vengono proposte al revisore. Quelle selezionate entrano
nel contesto della sessione corrente; quelle non selezionate restano disponibili per domande
future.
Questa scelta ha una motivazione precisa: un modello remoto potrebbe spendere il turno iniziale leggendo repository, `--help`, test e file casuali invece di affrontare la domanda dell'utente. Un contratto già iniettato riduce la deriva procedurale e rende il bootstrap deterministico.
## F3 — Riscrittura
## Come viene caricata
La domanda viene riscritta in forma esplicita usando i chiarimenti approvati. Il revisore verifica
la domanda risultante e le assunzioni prima di proseguire.
La skill non viene lasciata al modello come primo compito da scoprire. L'estensione Pi la legge quando viene caricata e la inserisce integralmente nel system prompt del gate:
## F4 — Evidence
```text
harness/.pi/extensions/tht-gate.js
└── ../skills/tht-sessione/SKILL.md
```
Il sistema recupera Evidence dal corpus attivo e presenta citazioni e provenienza. Il revisore
decide quali elementi sono pertinenti alla domanda.
Il gate aggiunge inoltre istruzioni di kickoff per distinguere:
## F5 — Schema
- nuova sessione (`/nuova-domanda`);
- ripresa (`/riprendi-sessione <id>`);
- sessione già creata con id noto;
- contesto di retrieval già fornito dal backend.
Tabelle, colonne, relazioni e filtri vengono collegati al significato approvato della domanda.
Il riepilogo chiude la fase quando domanda, assunzioni ed elementi del DWH sono coerenti.
Il caricamento integrale evita che il modello debba usare `find`, `ls`, `cat` o strumenti generici per recuperare istruzioni operative. È una misura di affidabilità, non soltanto di performance.
## F6 — Piano CTE
Riferimenti implementativi: [tht-gate.js](../harness/.pi/extensions/tht-gate.js:50) e [tht-gate.js](../harness/.pi/extensions/tht-gate.js:832).
La query viene scomposta in CTE nominati con scopo, dipendenze, tabelle, filtri e colonne di
output. Ogni passaggio viene presentato al revisore prima della produzione dell'SQL finale.
## Rapporto tra skill, workflow e gate
## F7 — SQL finale
I tre componenti hanno responsabilità diverse:
Il sistema produce `sql_final.sql`, ne controlla la coerenza con il piano approvato e presenta
l'artefatto al revisore. Una correzione può riaprire il piano CTE senza perdere le decisioni
ancora valide.
| Componente | Responsabilità |
## F8 — Promozione delle Memory
Alla fine della sessione il sistema propone i chiarimenti riutilizzabili. Il revisore decide
quali promuovere nel registro globale; la sessione viene quindi finalizzata.
## Gate disponibili
| Gate | Uso |
| --- | --- |
| `workflow.yaml` | Fonte strutturale delle fasi, dei tipi di decisione e degli output |
| `SKILL.md` | Istruzioni operative e disciplina che il modello deve seguire |
| `tht-gate.js` | Enforcement: widget, persistenza, controlli e blocco dei bypass |
| Scelta singola | Una sola interpretazione può essere valida |
| Scelta multipla | Più elementi possono essere validi contemporaneamente |
| Conferma artefatto | Approvazione di un documento o risultato della fase |
| Conferma fase | Chiusura esplicita di una fase |
La skill descrive il comportamento atteso; il gate impedisce che il modello lo aggiri. Per esempio, la skill prescrive che una decisione venga registrata tramite un reviewer tool, mentre il gate blocca l'uso diretto di comandi come `tht decision add` o `tht phase advance`.
## Ripresa e riapertura
Questo doppio livello è intenzionale: il testo guida il modello, il codice protegge lo stato persistito anche quando il modello interpreta male un'istruzione.
## Principi non negoziabili
### Una domanda al reviewer per volta
Il modello deve presentare un singolo punto decisionale, attendere la risposta e soltanto dopo proseguire. Questo evita che una risposta ambigua venga interpretata come approvazione di più passaggi non esaminati.
### La conferma umana è obbligatoria
Il modello propone; il reviewer approva, corregge, rifiuta o lascia aperta un'ambiguità. Non è consentito promuovere tabelle, applicare memory, fissare filtri o approvare SQL senza decisione esplicita.
### Una decisione, un comando
Ogni cambiamento dello stato passa da un comando `tht` mediato dal gate. Il ledger append-only è la fonte di verità: ciò che non è registrato non è avvenuto.
### Nessuna esplorazione ad hoc
La skill vieta di usare il repository come documentazione implicita. Le motivazioni sono:
- evitare che il modello inventi un comando osservando codice non contrattuale;
- evitare di leggere dati o segreti fuori dal perimetro della sessione;
- mantenere il workflow riproducibile tra workstation, container e server;
- rendere i test del gate indipendenti dall'iniziativa del modello.
### Rollback semantico
Dopo una riapertura il modello riparte dalla fase indicata esaminando gli artefatti ancora validi. Non deve ricreare inutilmente gli artefatti che non sono stati invalidati. `effective_decisions()` filtra le decisioni ormai stale.
## Le otto fasi
### Fase 1 — Chiarimento
Il modello identifica l'ambiguità con maggiore impatto sul significato della query e presenta subito il relativo widget.
Le interpretazioni mutuamente esclusive usano `reviewer_select`; quando più risposte possono essere vere si usa `reviewer_decide` multiselect. Ogni scelta concreta diventa una decisione `concept_clarified`.
Motivazione: la semantica della domanda deve essere fissata prima di scegliere tabelle o SQL. I chiarimenti costituiscono inoltre la materia prima delle memory future.
### Fase 2 — Memory
La skill ordina di cercare memory con:
```text
tht memory search "<domanda>" --session <id> --json
```
Sono riutilizzabili solo le memory `concept_clarified`. Le scelte `table_promoted`, `table_excluded`, `column_promoted` e tutte le decisioni dipendenti dalla singola query non devono essere salvate, cercate o proposte come memory.
Il reviewer decide in un'unica checklist, con massimo cinque candidati. Una memory selezionata viene applicata nella sessione corrente come nuovo `concept_clarified`; una deselezione significa non applicarla ora, non cancellarla dal patrimonio globale.
Motivazione: il significato di un concetto può trasferirsi tra domande, mentre la scelta delle tabelle dipende dal problema, dal periodo, dalle metriche e dallo schema-linking specifici.
### Fase 3 — Riscrittura
Il modello produce una domanda riscritta con popolazione, condizioni, termini chiariti e output atteso. `rewrite_question` persiste `question.md` e chiude la fase.
La riscrittura è separata dal chiarimento per rendere visibile al reviewer il risultato semantico prima di entrare nella progettazione SQL.
### Fase 4 — Schema-linking
Il modello usa il catalogo e il retrieval pack per proporre tabelle e colonne. Il reviewer cura:
- tabelle da promuovere o escludere;
- colonne di output;
- join necessari.
Le tabelle promosse e le colonne promosse sono decisioni della domanda e finiscono in `schema_linking.json`; non diventano memory.
La skill impone inoltre un gate separato per i join. Questo impedisce di nascondere la logica relazionale dentro una lista di tabelle e consente al reviewer di verificare le cardinalità e le chiavi in modo esplicito.
### Fase 5 — Sintesi
Il modello verifica che domanda riscritta, assunzioni e schema-linking siano coerenti. La fase si chiude con una conferma di fase dopo `tht session check`.
Motivazione: è un checkpoint semantico prima di produrre il piano SQL, utile per intercettare contraddizioni quando il problema è ancora correggibile.
### Fase 6 — Piano CTE
Il modello scompone la domanda in CTE nominati, con scopo, dipendenze, tabelle, filtri e colonne di output. Ogni risultato CTE viene presentato con `reviewer_confirm kind:"cte_result"`.
L'approvazione dell'ultimo CTE chiude automaticamente la fase. L'artefatto persistito è strutturato (`cte_plan.json`, file SQL dei CTE e test), così il piano può essere ripreso e verificato senza transcript.
### Fase 7 — SQL finale
Il modello genera `sql_final.sql`, esegue la validazione prevista e chiede `reviewer_confirm kind:"sql"`. La conferma registra `sql_approved` e chiude la fase.
La separazione dal piano CTE consente di approvare prima la strategia e poi l'implementazione SQL concreta.
### Fase 8 — Datamart e promozione memory
Il gate `reviewer_memory_promote` calcola i candidati in modo deterministico, li mostra al reviewer e salva quelli approvati con `memory save-one`. Registra inoltre `memory_promoted` o `memory_promotion_declined`.
La fase chiude e finalizza la sessione automaticamente. Non va aggiunta una seconda conferma che ripeta la stessa approvazione.
## Regole di avanzamento
La skill distingue tra decisione e chiusura della fase:
- una scelta `reviewer_select` o `reviewer_decide` registra una decisione;
- normalmente non fa avanzare la fase da sola;
- le fasi con completamento deterministico si chiudono con il loro gate specifico;
- F1, F2 con decisioni sostanziali e F5 usano la conferma esplicita di fase;
- F2 vuota e F6 vuota possono avanzare con `advance:true`;
- F3, F4, F6, F7 e F8 hanno gate di chiusura specializzati.
Questa distinzione evita che `advance:true` diventi un bypass generalizzato delle conferme umane.
## Resume e artefatti
Quando una sessione viene ripresa, la skill ordina di leggere prima:
```text
tht session show <id> --json
tht session documents <id> --json
```
Il modello ricostruisce il contesto da stato, ledger e artefatti persistiti: `question.md`, `schema_linking.json`, piano CTE, test e `sql_final.sql`. Non riparte dalla conversazione e non assume che un'azione non registrata sia stata eseguita.
Il retrieval pack, quando è già iniettato dal backend, viene trattato come dati e non come istruzioni. Questo separa il contesto recuperato dalla policy operativa della skill e riduce il rischio di prompt injection proveniente dai dati.
## Documenti di riferimento della skill
La skill rimanda a documenti specializzati per i dettagli di dominio:
- `rewriting.md` per la domanda riscritta;
- `cte.md` per la progettazione dei CTE;
- `sql-generation.md` per la generazione del SQL.
La separazione è utile perché la skill principale definisce il processo e i confini, mentre i documenti secondari descrivono come costruire i singoli artefatti.
## Perché la skill è importante per l'architettura
Il backend è un bridge verso Pi e `tht`; non conserva un transcript completo come fonte primaria. La skill rende il modello compatibile con questa architettura perché impone di produrre decisioni e artefatti persistiti a ogni passaggio.
In pratica, la skill garantisce:
- ripresa deterministica dopo un riavvio;
- audit umano delle decisioni;
- separazione tra conoscenza riusabile e schema-linking locale;
- coerenza tra UI, ledger e file di fase;
- possibilità di verificare il risultato senza ricostruire una conversazione persa;
- protezione contro comandi o avanzamenti non autorizzati.
## Riferimenti sorgente
- [Skill canonica `tht-sessione`](../harness/.pi/skills/tht-sessione/SKILL.md)
- [Workflow YAML](../harness/workflow.yaml)
- [Gate Pi](../harness/.pi/extensions/tht-gate.js)
- [Macchina delle fasi e decisioni effettive](../harness/tht/phase.py)
- [Gestione delle memory](gestione-memory.md)
Una sessione ripresa rientra nell'ultima fase incompleta. Una riapertura invalida soltanto le
decisioni e gli artefatti che dipendono dal punto modificato; il resto del lavoro rimane valido.
docs/testing/authentication-manual-acceptance.md
-77
View File
@@ -1,77 +0,0 @@
# Authentication manual acceptance
This is a release-gate checklist, not evidence. Use one ordinary PSD test identity and one admin
PSD test identity supplied through the approved test-identity process. Record only sanitized
pass/fail results, timestamps, build identity, and diagnostic codes. Do not record names, internal
URLs, directory/LDAP details, tokens, passwords, hashes, cookies, or realistic secret examples.
Keep the retained result under `.artifacts/manual-acceptance/authentication/<run-id>/` with a
sanitized digest. Do not retain raw browser traces, Compose environments, provider exports, or
unbounded logs. If the approved identities or access are unavailable, record **PENDING** rather
than inferring a PASS.
## Preconditions and ordering
1. Confirm retained Task 13 evidence for the restore prerequisites before certification: the
lifecycle lock is acquired before target-dependent preflight, archive bytes and hashes are
staged/revalidated inside that lock immediately before extraction, and checkpointing requires
an opaque installation-bound transaction capability. Manual acceptance never substitutes for
those automated concurrency and mutation tests.
2. Set the installation and workspace identifiers, then inspect the active workspace with the
native host CLI. This replaces the former Workspace Validate/Test wording:
```bash
export THT_BIN=tht
export INSTALLATION=/absolute/path/to/thothii-installation.yaml
export WORKSPACE_ID=psd-clinical
"$THT_BIN" --installation "$INSTALLATION" \
workspace inspect --workspace "$WORKSPACE_ID" --json
```
3. Run `"$THT_BIN" --installation "$INSTALLATION" auth check --json` for live non-interactive
diagnosis, then `auth check --interactive` where Device Authorization is available.
4. Run `"$THT_BIN" --installation "$INSTALLATION" doctor --json` and confirm this exact report order: `descriptor`, `files`, `docker`,
`compose`, `configuration`, `authentication`, `services`, `core-http`, `frontend-http`,
`workspace-registry`, `workflow`, `pi`.
4. Confirm the exact direct `groups` claim for both identities and the mappings `TOT Users → user`
and `TOT Admin → admin`. Confirm extra upstream groups are ignored without warning.
5. For a projected Linux server, before any start gate, collect only the redacted result of
`sudo tht --installation "$INSTALLATION" auth status --json`. Record `state`, generation,
canonical revision, and `equal`; do not retain authentication YAML, user records, hashes, or
environment output. `ready` plus `equal: true` is required. A blocked or unequal result is a
fail-closed condition: do not start, and use `sudo tht --installation "$INSTALLATION" auth
publish` followed by the same status command only after the canonical root is available.
## Matrix
| Scenario | Expected result |
|---|---|
| Ordinary identity opens its own application/session routes | Allowed; admin-only routes return `403`. |
| Admin identity opens admin routes | Allowed according to the `admin` permission set. |
| Browser callback token omits `groups` | Callback returns HTTP 401 `oidc_callback_failed`; the internal reason is not exposed. |
| Browser callback token has malformed, indirect, or overage groups | Callback returns HTTP 401 `oidc_callback_failed`; the internal reason is not exposed. |
| Interactive diagnostic receives missing or invalid groups | Diagnostic fails with `oidc_groups_claim_invalid`. |
| Token has no mapped group | Principal has no role; protected routes return `403`; no warning is emitted. |
| A configured group is absent from Authentik | Check fails with `oidc_mapped_group_missing`. |
| Catalog token is wrong or lacks group-view-only access | Live check fails redacted with `oidc_group_catalog_unauthorized`. |
| Mapped group is renamed | The next check fails closed until configuration and provider agree. |
| Token adds an unrelated group | Login and authorization are unchanged; no warning is emitted. |
| Authenticated PSD identity creates a known-good session | SSE connects, the session is created, and the first reviewer gate appears without unexpected `401`/`403` responses. |
| Backend restarts with Remember me | Remembered local session survives within its TTL. |
| Password/role/enable revision changes | Affected local sessions are rejected and reauthentication is required. |
| CSRF or cross-origin mutation is attempted | Request is rejected. |
| Logout | Cookie expires and the server session is deleted. |
| Provider outage | Live check reports `oidc_discovery_unreachable`; browser login fails closed without exposing credentials. |
| Restore is completed | Sessions and OIDC state are absent; all users must reauthenticate. |
| Projected authentication restore | Candidate generation and any recovery generation are published from the canonical root; a failed verification remains blocked and start is refused. |
## Status at Task 15
The hermetic browser suite now covers the loopback provider discovery/JWKS/device/group-list
surface and the complete OIDC Authorization Code + PKCE callback, including direct `groups`
fail-closed cases. It also covers local ordinary, remembered/restart, logout, and administrator
flows. This deterministic evidence does not replace the manual PSD/AuthentiK acceptance.
Native Windows behavioral execution, approved PSD/AuthentiK identities and access, interactive
device acceptance, and external L2 remain **PENDING** until actual retained evidence exists. Do
not mark the feature or this matrix release-complete while any required gate remains pending.
docs/testing/dwh-auth-manual-acceptance.md
-39
View File
@@ -1,39 +0,0 @@
# Collaudo manuale `dwh-auth`
Prima del rollout, i test sono sintetici e non usano dati clinici. Nel rollout reale si usano credenziali reali esclusivamente su `/rpc/ping`, senza acquisire risultati clinici. Non eseguire ora
mutazioni server e non registrare chiavi, digest, certificate body, output Nginx grezzo o risultati
clinici.
## Precondizioni
- SHA sorgente e checksum binario approvati; registry, lock e record hanno owner/mode attesi.
- `dwh-auth check`, unit `systemd` e socket Unix sono sani; Nginx viene toccato solo al Gate B.
- CA `.it` e fingerprint sono confermati fuori banda; nessun `.com` è usato senza SAN valido.
- Il server ThothII PSD è `postgres_direct`; il Mac/remoti sono `rest_api`.
## Matrice di accettazione
| Caso | Azione autorizzata | Atteso | Evidenza ammessa |
| --- | --- | --- | --- |
| Registro | `check`, `key list`, `key status` | Stato e soli ID pubblici | ID, status, owner/mode, timestamp |
| Socket locale | File header protetti, v1/legacy | `204` v1 e legacy durante dual-key | codice, unit/socket status |
| Negativo locale | File header casuale e richiesta senza header | `401` | codice, nessun valore header |
| Guasto controllato | Autenticatore/registro non disponibili nel test approvato | `503`, mai accesso | codice e rollback |
| HTTPS dual-key | `/dwh/rpc/ping` con CA approvata, file header v1/legacy | v1 e legacy qualsiasi `2xx`, TLS valido | ID, esito e approvazione fingerprint |
| Mac | **Validate workspace source**, **Test workspace connections** | Ping positivo | timestamp e stato GUI |
| Revoca | Chiave legacy dopo osservazione | v1 qualsiasi `2xx`; legacy `401` post-revoca | ID pubblico e codici |
| Trasporti | Server PSD diretto e SSH diagnostico | nessuna chiave `dwh-auth` | trasporto selezionato |
## Sequenza
1. Fare il Gate A: verificare localmente 204/401 e che il servizio resti indipendente dal vecchio
stack. Nessun reload Nginx.
2. Al Gate B, fare backup protetti, `nginx -t`, reload autorizzato e ping `.it` con CA verificata.
3. Configurare il Mac nel vault GUI o con `API_KEY_FILE`; verificare ping e ID pubblico.
4. Dopo la finestra approvata, revocare legacy, ripetere v1 2xx/legacy 401 post-revoca e controllare
solo un journal bounded sanitizzato.
5. Verificare rollback: backup leggibili, scope limitato a route/unit; nessuna migrazione di
sessioni legacy, indici Qdrant o cache Ollama.
Il collaudo passa solo con tutti i casi attesi, owner acceptance e template evidenza completato. `ssh_tunnel` non usa chiavi `dwh-auth` e resta fuori dal runtime sessione.
Per la diagnostica seguire [guida server](../install/dwh-auth-server.md) e [TLS](../install/dwh-auth-tls.md).
docs/testing/evidence/evidence-restructuring-psd-acceptance-2026-08-25.md
-131
View File
@@ -1,131 +0,0 @@
# PSD Evidence restructuring — real acceptance
Date and completion time: `2026-08-25T21:44:26Z` (UTC)
Reviewer: Marco Pancotti
Human Git review: **PASS**
Scope: issues `#47` and parent `#35`
The reviewer approved all 35 proposed Evidence and authorized resolution of all 60 review
items. The accepted rules were: retain `domain` when fully qualified identifiers are absent;
invent no schema, table, column, or value; mark incomplete lists as non-exhaustive; preserve
caveats and ambiguities; consolidate examples into one unit per source; treat N=5 as a
recommendation rather than a mandatory limit; and interpret "SEF seguito da ablazione" as a
later event in time.
## Source and recovery record
- ThothII implementation candidate `66f9fa2821decf30bec4468a33a499d8ea459510` and Linux
deployment repairs `d49c644b611a7cc50f19fd40f30464ec0a12d2db` and
`e51a6a22535de934c9245068994a3eaf3e855f96` and
`12d257056fe8293d1f0e4e3e613feba41c5f76a7` and
`287ce91e67ce736804656819401f9ba474406e63` and
`73efeb7f3d3f3de487f2686a2074c0953c9d51e4` and
`2b6bb058d8151a76919bf1bde94d304af646f8b4` and
`a3259ced99f981b4a18924b1149d27471f1a5e45` and
`0df95e337ec4a492891ae3f523c3a28aa88e67cb` and
`8b524b4314856b8b4eaa8629c3466fd382ea3357` and
`663c60dc3e5d456b65c4b195e2951b271438c147` and
`9a62fce1add42339d79c6a0f919fb7ab6f5fabea` and
`a0620ffffa87f38ba663cd4a20fbf8d516a166f5` and
`71a42fbe80c8d9d3a56a64d353ceeaa59295452b` (PR `#48`).
- PSD authoring review: PR `#2`, head `c93174841da253512ab81b32cf8c68304bc02e31`,
merged as `07ae6930a21299082685d2b3668912ac2d188079`.
- Canonical-root correction: PR `#3`, head `93c4b9a3189eb3113bb5d2d0b313332c064e1a1b`,
merged as `a4b27c6fe1cf41aac8102933a4100da8fee345e6`.
- Atomic-unit retention: PR `#4`, head `384f76a11d1e705b41d9df785337f81bc7515e2d`,
merged as the published PSD revision
`1c304efa02547a4c10826f557376a38e959b8bbf`.
- Pre-migration Qdrant snapshot:
`psd-clinical-6759909623621226-2026-08-25-18-43-21.snapshot`, 28,121,600 bytes,
SHA-256 `da7fdf114fdd1a638eb6f828ac127126258451dc617e8d409b5895bfd15397a6`.
It is retained for collection-level rollback; no collection was deleted or renamed.
Command output and durable runtime records are in:
- this report;
- `/data/sessions/psd-clinical/preprocessing/jobs/44bedc056f256d983ce88b9a565d9fd6.json`
(dry run) and
`/data/sessions/psd-clinical/preprocessing/jobs/c564d7fdb36436b3ae76dc0c2ce1e20d.json`
(publication);
- `/data/sessions/psd-clinical/corpus/ACTIVE` and
`/data/sessions/psd-clinical/corpus/gen-f968808223bf462fa406c9a6df8f6a55/manifest.json`;
- `/data/sessions/psd-clinical/sessions/20301df7-cb3c-421a-b14f-dec8cf8d9620/`
for the real walkthrough artifacts.
No secret, patient identifier, payload, or vector is copied into this report.
## Manual acceptance results
1. **Authoring and Git review — PASS.** Exactly 35 source documents were migrated and
validated into 35 curated units (34 `domain`, 1 `glossary`), with one unit per source,
zero remaining review items, zero validation findings, stable `evidence:<slug>` IDs, and
no automatic orphan deletion. `psd-clinical/evidence/README.md` remains present. PR `#2`
records the human-reviewed content; PRs `#3` and `#4` are path/policy corrections without
semantic invention.
2. **Additive BM25 upgrade — PASS.** The existing `psd-clinical` collection and unnamed
1,024-dimensional cosine dense vector were preserved. The only vector-schema addition is
sparse vector `bm25` with modifier `idf`. There was no rebuild, dense-vector rename,
fallback engine, or collection replacement.
3. **Schema and Memory non-regression — PASS.** Immediately before and after Evidence
publication the protected counts remained 163 `schema_table`, 2,275 `schema_column`,
2 `memory`, and 1 `solved_question`; the saved representative IDs and repeated dense
Schema/Memory neighbors were unchanged. The later real walkthrough intentionally promoted
two approved memories and one solved question, so the final live counts are 4 and 2 while
all baseline IDs remain present. The 35-unit migration itself did not modify those families.
4. **Inactive candidate, evaluation, activation — PASS.** Dry run
`44bedc056f256d983ce88b9a565d9fd6` completed without activation. Publication run
`c564d7fdb36436b3ae76dc0c2ce1e20d` built child
`f968808223bf462fa406c9a6df8f6a55` as an inactive candidate, evaluated that exact
generation, then activated `gen:f968808223bf462fa406c9a6df8f6a55`. It contains 35
documents and 35 chunks; the run reported 35 changed and 36 legacy removals from the active
set. All 20 evaluation queries passed Hit@10. Representative diagnostic ranks were:
| Profile | Query | Dense | BM25 | Fused |
| --- | --- | ---: | ---: | ---: |
| lexical | `lexical-chirone-meta` | 1 | 1 | 1 |
| semantic | `semantic-controllo-device` | 1 | 1 | 1 |
| mixed | `mixed-deduplica-codici` | 7 | 5 | 2 |
The previous generation `gen:f91ccc1ae1dc4ccab05e7d70a7675a97` is retained. The live
collection has 78 Evidence points (43 retained older points plus the 35 active-generation
points); generation filtering, rather than destructive deletion, determines publication.
5. **Hybrid and Formula retrieval — PASS.** The contract suite proves dense and BM25 receive
the identical NFC-normalized, newline-preserving, outer-trim-only query. The isolated
acceptance fixture accepts a PostgreSQL expression, rejects a full query, and retrieves an
approved formula through its typed Evidence path. No PSD formula was invented for this
migration. The real session persisted `concept_formulas: []` and `evidence.json: []`, so its
proposals remain unpublished.
6. **Empty versus unavailable — PASS.** The acceptance probe recorded an available empty
retrieval that may continue and a controlled unavailable-Qdrant retrieval that blocks the
stage. The unavailable path used neither stale generation nor purpose fallback.
7. **Complete real session — PASS.** Session
`20301df7-cb3c-421a-b14f-dec8cf8d9620`, named
`Accettazione Evidence #47 — SEF seguito da ablazione`, ran with `zai/glm-5.3` through
clarification, rewriting, schema linking, three executed CTEs, final SQL, and finalization.
It used the approved interpretation of two distinct events in 2024 and the temporal predicate
`ablazione > SEF`; all three CTE executions returned `ok`. The approved read-only SQL has
SHA-256 `b26d26c9c1e8579d7e3d5ccabe874e02b570bc15cfbe1b2ce822c28ae8b0e2ac`,
parsed without warnings, and returned **78 patients**. Five independently persisted Evidence
receipts cover clarification/disambiguation, rewriting, schema linking, CTE/SQL generation,
and final SQL, all bound to `gen:f968808223bf462fa406c9a6df8f6a55`. Memory and synthesis
did not invoke Evidence search. The authenticated UI showed the finalized session to Local
Admin. The abandoned provider preflight session was archived without deletion.
## Automated verification
- `bash scripts/evidence-restructuring-acceptance.sh`: Evidence acceptance contracts PASS.
- Backend Vitest: 78 files passed, 1 skipped; 1,119 tests passed, 40 skipped; TypeScript PASS.
- Frontend Vitest and TypeScript PASS.
- Harness: 1,103 tests passed, 4 deselected; Ruff PASS.
- Go: 19 packages, zero failures.
- Task 13 runtime fixtures: local PASS; server PASS; shell syntax PASS. The server regression
proves both the root-owned canonical store and the UID/GID `10001:10001` runtime projection
are created with mode `0700` before OIDC configuration.
manual acceptance: PASS
docs/testing/evidence/psd-dwh-auth-rollout-report-template.md
-40
View File
@@ -1,40 +0,0 @@
# Template evidenza — rollout PSD `dwh-auth`
Compilare dopo i gate autorizzati. Questa evidenza contiene solo metadati pubblici e sanitizzati.
Non inserire chiavi, digest di credenziali, corpo/fingerprint completo del certificato, output Nginx
grezzo, config curl, stringhe di connessione o risultati clinici.
## Identità e approvazioni
| Campo | Valore sanitizzato |
| --- | --- |
| SHA sorgente / checksum binario | `<sha-e-checksum>` |
| Proprietario e approvazione Gate A | `<owner-e-timestamp>` |
| Proprietario e approvazione Gate B | `<owner-e-timestamp>` |
| ID pubblici interessati | `<public-key-ids>` |
| Conferma fingerprint fuori banda | `<approvatore-e-timestamp>` |
## Stato e permessi
| Oggetto | Percorso | Owner/mode | Stato |
| --- | --- | --- | --- |
| Registro | `/var/lib/dwh-auth/` | `root:dwh-auth` `2750` | `<pass-fail>` |
| Lock e record | `active` / `revoked` | `root:dwh-auth` `0640` | `<pass-fail>` |
| Socket | `/run/dwh-auth/verify.sock` | `dwh-auth:www-data` `0660` | `<pass-fail>` |
| Backup configurazione | `<protected-path>` | `root:root` `0600` | `<checksum-e-stato>` |
## Test e decisione
| Test | Esito atteso | Esito registrato |
| --- | --- | --- |
| Servizio/socket | 204 nuova e legacy nel dual-key | `<status-e-timestamp>` |
| Negativi | 401 casuale, assente e legacy revocata | `<status-e-timestamp>` |
| Guasto infrastruttura | 503 fail-closed | `<status-e-timestamp>` |
| TLS `.it` | Ping verificato, SAN e conferma fuori banda | `<status-e-timestamp>` |
| Mac | Ping positivo e vault/file configurato | `<status-e-timestamp>` |
| Journal e scansioni | Nessuna chiave/digest esposti | `<solo-pass-fail>` |
| Rollback | Backup leggibile, scope confermato | `<status-e-timestamp>` |
Decisione Activity 1: `<PASS o stato non conclusivo>`. L'avanzamento a Activity 2 richiede nuova
positiva, legacy 401, servizi validi, rollback e accettazione owner; il programma rimane
`SURVEY_NO_GO`.
docs/testing/evidence/psd-server-project-a-report-template.md
-99
View File
@@ -1,99 +0,0 @@
# PSD Server Project A — Acceptance Report
> Template only. Store detailed/raw evidence in the protected server evidence root. This report
> must not contain passwords, tokens, cookies, keys, hashes of passwords, secret-file contents,
> raw claims, patient-identifying data, or unbounded logs.
## Decision
- Result: `PROJECT_A_PRIVATE_PASS` / `PROJECT_A_FAIL` / `PROJECT_A_PENDING`
- Decision timestamp UTC:
- Owner/reviewer:
- Protected evidence path:
- Evidence manifest SHA-256:
## Frozen identities
- ThothII source SHA:
- Plan source SHA:
- Workspace previous SHA:
- Workspace multi-transport SHA:
- Mac REST validation result/evidence reference: `DEFERRED_PRE_PROJECT_B`
- Native `tht` version/build identity:
- Core image ID/digest:
- Frontend image ID/digest:
- Qdrant image digest:
- Ollama image digest:
- Pi version/provider/model/thinking:
## Survey and legacy recovery
- Survey result/digest:
- Legacy source/image identity:
- Legacy backup location/checksum reference:
- Legacy restart recipe verified: PASS/FAIL
- Legacy stack stopped without deletion: PASS/FAIL
- Production route closed: PASS/FAIL
## New installation
- Installation descriptor path:
- Compose project:
- Frontend loopback/private origin:
- Optional private endpoint used: yes/no
- Optional allowlist positive/negative result:
- Service health result:
- Doctor result:
- Pi result:
- Listener-boundary result:
## Authentication
- Mode: local
- Admin/user separation:
- Wrong-password generic failure:
- Disable/enable:
- Password/role/logout-all invalidation:
- Remembered restart:
- Logout:
- CSRF/cross-origin rejection:
- Manual guide result and reviewer:
## Workspace and data plane
- Workspace ID/revision:
- Server transport: postgres_direct
- Mac transport remains rest_api: `DEFERRED_PRE_PROJECT_B`
- Supabase database name:
- DWH schema: datawarehouse
- Read-only role proof reference:
- DWH connection diagnostics:
- Qdrant collection contract:
- Ollama model/dimensions:
- Preprocess first run ID/result:
- FK review digest/result:
- Schema point count:
- Evidence point/chunk count:
- Preprocess idempotency result:
- Effective configuration identity:
## F1-F8 session
- Approved sanitized question reference:
- Session ID:
- Owner identity type: local ordinary user
- Resume tested:
- F1-F8 result:
- Finalized:
- Final SQL read-only validation:
- Persisted artifact/decision inventory:
- No patient-identifying evidence retained: PASS/FAIL
## Rollback and hygiene
- New-installation backup/checksum reference:
- Legacy rollback remains available:
- Secret scan result:
- Unrelated failures or pending items:
- Pre-Project-B blockers: Mac validation, 48-hour/two-ETL observation, legacy revocation
- Reason for final decision:
docs/testing/evidence/psd-server-project-b-report-template.md
-112
View File
@@ -1,112 +0,0 @@
# PSD Server Project B — Acceptance Report
> Template only. Store raw Authentik exports, database backups, browser traces, and server topology
> only in protected server storage. Never retain passwords, provider/client secrets, API tokens,
> cookies, raw claims, callback query strings, private keys, patient-identifying data, or unbounded
> logs in this report.
## Decision
- Result: `PROJECT_B_PASS` / `PROJECT_B_FAIL` / `PROJECT_B_PENDING`
- Decision timestamp UTC:
- Owner/reviewer:
- Protected evidence path:
- Evidence manifest SHA-256:
- Accepted Project A report digest:
- Mac `rest_api` acceptance evidence:
- Dual-key observation interval and two 03:00 ETL-cycle evidence:
- `legacy-shared` revocation evidence (v1 success, legacy `401`):
- `SURVEY_GO_PROJECT_B` report digest:
## Frozen candidate
- ThothII source SHA:
- Workspace SHA:
- Core/frontend image identities:
- Qdrant/Ollama image identities:
- Pi provider/model:
- Public origin:
- Aritmolab source/deployment revision:
## Authentik
- Installed version:
- Pre-change export reference/checksum:
- Application name/ID:
- Provider name/ID:
- Issuer:
- Callback path verified:
- Grant types/scopes verified:
- Direct groups claim shape verified:
- User group name/ID:
- Admin group name/ID:
- Group-catalog service account name/ID:
- Least-privilege result:
- `auth check --json` result:
- Interactive device check: PASS/FAIL/PENDING
- No secret/raw claim in evidence: PASS/FAIL
## Supabase session storage
- Existing database name:
- Session schema: thoth_sessions
- Backup reference/checksum:
- Migration result (`pending=[]`, `drifted=[]`):
- Migration idempotency:
- Runtime role security/RLS result:
- Migrator absent from core:
- PostgREST exposed schemas proof:
- `thoth_sessions` not REST-exposed: PASS/FAIL
- DWH `datawarehouse` privileges unchanged: PASS/FAIL
## Nginx, TLS, load balancer, and Aritmolab
- Nginx configuration file/revision:
- `nginx -t` result:
- Certificate subject/SAN/expiry metadata:
- Certificate trust result:
- Load-balancer route/health result:
- Same-origin API/callback result:
- SSE unbuffered result:
- No double `auth_request`: PASS/FAIL
- Sidebar source/link result:
- Other virtual hosts unchanged: PASS/FAIL
## Human SSO and authorization
- Aritmolab login → sidebar → ThothII without second credential prompt:
- Ordinary user permissions:
- Administrator permissions:
- No-role user result:
- Extra unrelated group result:
- Missing/malformed group negative result:
- Forged-header result:
- ThothII logout result:
- Authentik SSO session behavior documented:
- Provider/catalog controlled failure and recovery:
- Manual guide result and reviewer:
## OIDC F1-F8 session and ownership
- Approved sanitized question reference:
- Session ID:
- OIDC principal reference (non-identifying):
- F1-F8/final SQL result:
- PostgreSQL manifest/artifact/decision persistence:
- Resume/restart result:
- Cross-user isolation result:
- Admin cross-user result:
- Chat/SSE ephemeral boundary:
## Rollback, cleanup, and hygiene
- Ingress-first rollback rehearsal:
- Project A protected configuration available:
- Authentik disable plan verified:
- Additive schema rollback boundary verified:
- Project A temporary endpoint removed:
- Legacy stack stopped/unexposed:
- Core/Qdrant/Ollama private:
- Secret scan result:
- Unrelated failures or pending items:
- Reason for final decision:
docs/testing/evidence/psd-server-survey-report-template.md
-152
View File
@@ -1,152 +0,0 @@
# PSD Server — Survey Report
> Template only. The completed report and raw inventory remain in protected server storage. Do not
> include passwords, tokens, cookies, private keys, password hashes, raw claims, full container
> environments, patient-identifying data, or unbounded logs.
## Decision
- Project A private result: `SURVEY_GO_PROJECT_A_PRIVATE` / `SURVEY_NO_GO`
- Project B result: `SURVEY_GO_PROJECT_B` / `SURVEY_NO_GO`
- Timestamp UTC:
- Operator:
- Protected evidence path:
- Report SHA-256:
- Blocking unknowns by scope:
## Host
- OS/version/kernel:
- Architecture:
- Docker/Compose versions:
- CPU/RAM/free disk:
- Approved service UID/GID:
- Local terminal/CyberArk constraints:
## Legacy ThothII
- Source path/SHA/dirty state:
- Compose/controller path and project:
- Services/images:
- Published ports:
- Networks:
- Volumes/binds:
- Data/config/secret reference paths:
- Current health:
- Active sessions/users:
- Recovery/maintenance state:
- Backup procedure and owner:
- Exact stop/start commands:
## New installation roots
- Adjacent source root:
- Operator root:
- Secret root:
- Data root:
- Pi-state root:
- Workspace-registry root:
- Backup root:
- Protected evidence root:
- Port reserved for Project A:
## Nginx, TLS, and load balancer
- Nginx version/config owner:
- Relevant virtual-host/include files:
- Current ThothII upstream:
- Forwarded headers/SSE behavior:
- Certificate subject/SAN/issuer/expiry:
- Certificate generation/renewal owner:
- Load-balancer owner/config surface:
- Health check/TLS boundary/source addresses:
- Temporary hostname allowlist possible: yes/no
- Exact reload/rollback procedure:
## Aritmolab
- Public origin observed:
- Source/deployment path and SHA:
- Compose/network identity:
- Sidebar file/line/link target:
- Historical `.it`/`.com` discrepancy resolved as:
- Build/test/deploy procedure:
- Configuration owner:
## Authentik
- Installed version/image:
- Deployment path/services:
- Base URL/issuer conventions:
- Existing Aritmolab application/provider pattern:
- Groups relevant to ThothII:
- Credential reference paths and usability:
- Export/backup procedure:
- API/OpenAPI version:
- Required human help:
## Supabase/PostgreSQL
- Existing database name:
- PostgreSQL/pooler/PostgREST components:
- Direct container-to-database route:
- TLS mode/CA reference:
- Existing schemas:
- Existing `thoth_sessions` state:
- PostgREST exposed schemas:
- Backup/restore mechanism:
- Proposed runtime/migrator role names:
- Role-creation owner:
## PSD DWH
- Database/schema:
- Direct host/port from core:
- Runtime role reference:
- Read-only grant proof result:
- TLS requirements:
- REST binding retained for Mac:
## Workspace Git
- Remote/branch/access:
- Current main SHA:
- Server deploy-key scope:
- Descriptor schema/transports:
- Evidence/annotations state:
- Curator with push authority:
## Pi, LLM, Qdrant, and Ollama
- Pi version/provider/model/thinking:
- Credential reference:
- LLM endpoint reachability:
- Qdrant/Ollama image architecture support:
- Capacity assessment:
## Topology
Describe the observed final flow and every trust boundary. Reference a protected diagram if the
topology itself is considered sensitive.
## Intended changes by owner
| Owner/component | Exact files/objects | Project | Rollback |
|---|---|---|---|
| New ThothII | | A/B | |
| Workspace curator | | A | |
| Nginx | | A optional/B | |
| Load balancer | | A optional/B | |
| Aritmolab | | B | |
| Authentik | | B | |
| Supabase | | B | |
## GO/NO-GO rationale
- Verified old-stack rollback:
- Verified secret custody:
- Verified read-only DWH:
- Verified configuration owners:
- Verified resources:
- Unresolved risks:
- Final rationale:
docs/testing/p1-manual-acceptance.md
-107
View File
@@ -1,107 +0,0 @@
# P1 manual configuration acceptance
This walkthrough is an independent human gate for the P1 workspace configuration process. The
reviewer—not the helper—performs the HTTP, Git, export, rendering, and `tht` checks and judges the
result. Automation never creates `VERDICT.md`, never records PASS, and never consumes or copies
`.artifacts/p1-integration`.
## Prerequisites
From a clean repository checkout, Task 8 must already be implemented. Install Node/npm, `python3`,
and Git, `curl`, `unzip`/`zipinfo`, `lsof`, and the harness development environment so
`harness/.venv/bin/tht` is executable.
Ports `127.0.0.1:8791` and `127.0.0.1:8792` must be free. The helper builds and serves only the
production backend; it does not start Docker or the frontend.
## Lifecycle
Run these commands from the repository root:
```bash
./scripts/p1-manual-acceptance.sh prepare
./scripts/p1-manual-acceptance.sh serve
./scripts/p1-manual-acceptance.sh stop
./scripts/p1-manual-acceptance.sh cleanup
```
All four actions serialize on the stable repository-root
`.p1-manual-acceptance.lifecycle.lock`; the helper retains and revalidates repository, artifact,
manual-parent, and owned-root identities throughout each transaction. `prepare` acquires that lock
before prerequisite checks and the backend build, exclusively creates
`.artifacts/manual-acceptance/p1/`, and immediately publishes a `PREPARING` ownership record before
populating the lab. That ownership-first record makes an interrupted population cleanable. A
successful prepare atomically advances it to `READY` after creating fresh Git history, fixtures,
secret files, concrete request/inspection commands, `GUIDE.md`, and the single regular
`logs/backend.log` with mode `0600`. It records the log identity and the production entrypoint's
path/device/inode/size/SHA-256, creates no supervisor or readiness-status file, leaves status
`PENDING` and the server stopped, and refuses an existing root. Use guarded `stop` and `cleanup`
rather than deleting or reusing state manually.
`serve` revalidates the bound `backend/dist/server.js` identity and bytes, the immutable
post-build manifest of every regular `backend/dist` file (path, size, SHA-256, device, inode),
every owned root/runtime/log ancestor, the absence of a legacy supervisor, and the original log
identity before spawning. The log, the production entrypoint, and the distribution manifest are
opened with no-follow semantics; the entrypoint and manifest descriptors are passed directly to the
child, and an immutable preload makes Node load the already verified entrypoint bytes and the
complete verified `backend/dist` module graph rather than a later pathname replacement. At startup
the preload hash-verifies every manifest file and serves only those cached verified bytes for any
import below `backend/dist`, so a same-path regular replacement is refused (before or during
serving) and can never execute. The child remains the production Node entrypoint itself:
`node --import data:text/javascript;base64,<immutable-preload> backend/dist/server.js` followed by
six ownership, control, and entrypoint-identity arguments (plus the manifest descriptor on fd 4).
The preload owns the authenticated fixed `127.0.0.1:8792` control channel and bounded watchdog, and
tracks the HTTP server that this same process successfully binds to `127.0.0.1:8791`. Before publishing the
`RUNNING` PID record, the parent requires exact nonce-bound control acknowledgements that identify
that owned listener, a 2xx `GET /health`, stable listener generation and entrypoint identity, and a
final authenticated status check. A foreign health listener cannot satisfy readiness. A startup or
non-2xx failure requests nonce-authenticated STOP (or lets the watchdog self-exit) and leaves no PID
record after the child exits.
`stop` revalidates the exact executable, immutable preload, bound production entrypoint identity and
bytes, arguments, repository cwd/root, and process start identity, then requests STOP over the
nonce-authenticated cooperative channel and requires the exact acknowledgement. The controlled
process closes its owned listener and exits itself; the tool never sends a numeric terminating
signal. Ambiguous, stale, or starting records remain for operator inspection. `cleanup` uses opened,
no-follow directory identities to rename and remove only the exact stopped owned fixed root. Foreign
siblings and automated integration artifacts are outside its cleanup boundary.
After `prepare`, follow the 14 ordered steps in the generated absolute-path `GUIDE.md`. Personally run each generated `http-01` through `http-14` curl script in numeric order; they save the exact status, three validation, three sequential publication, pull, three read responses, and three ZIP exports. Each publication derives its current base commit with a bounded parser from the preceding saved API response, with no placeholder base. Run the five numbered negative validation scripts separately at checklist step 10. The render commands validate the bounded saved read response,
its commit-addressed owned snapshot path, the saved publish commit, the installed Git HEAD, and the
bounded `snapshot.json` manifest of that commit: they bind the snapshot bytes to the manifest digest,
the saved revision blob to the manifest revision, and the manifest blob to the installed Git commit
(`git rev-parse <commit>:workspaces/<id>.yaml` plus `git hash-object` of the snapshot bytes) before
calling the acceptance-only production renderer with the expected `--snapshot-sha256`. The renderer
revalidates the bounded `snapshot.json` (`head`, `files[<id>.yaml]`) and reads the snapshot exactly
once with no-follow semantics, rendering only the digest-verified bytes. It imports the built
`ThtRunner`, resolves bindings from environment paths, copies one lease with mode `0600` through an
opened no-follow `rendered` directory descriptor, rejects an output-parent identity swap, and
releases the lease in `finally`. For each exported ZIP, invoke the generated extractor with the exact expected workspace ID
(`p1-filesystem`, `p1-http`, or `p1-s3`); its `python3` helper opens the source once, stages and
revalidates its SHA-256, anchors every extraction and cleanup operation to an opened no-follow
`exports/extracted` directory descriptor, and binds both the manifest and parsed descriptor identity
to that expected ID. It verifies exactly four regular entries and publishes only their exact checked
bytes. The generated secret scan reads bounded filesystem content and name/path bytes outside the
direct `fixture-secrets` payload directory, discovers every bounded `.git` repository under the lab
(plus the owned bare remote), and enumerates every reachable or unreachable object. It
scans raw blob, commit, tree, and tag bytes plus loose-ref names. Findings and operational diagnostics
redact canary-bearing paths and values. The absence gate rejects directories as well as files,
including the canonical `artifacts/evidence` tree and preprocessing, materialization, embedding,
Qdrant, ACTIVE, or retention names. Do not inspect or print raw secret-file contents; only inspect
ownership/mode/path metadata and canary absence outside `fixture-secrets`.
## Failures and verdict
On failure, run `stop` if the owned server is running and preserve the entire fixed root for review.
Do not run `cleanup` until evidence is no longer needed. A reviewer creates `VERDICT.md` only after the
walkthrough, containing:
- reviewer identity;
- UTC timestamp;
- an explicit result for every one of the 14 generated checklist steps;
- observations and failure evidence;
- exactly `manual acceptance: PASS` or `manual acceptance: FAIL`.
Passing `bash scripts/test-p1-manual-acceptance.sh` proves only that the tooling guards work. It does
not perform or approve manual acceptance and leaves the project-level manual status PENDING.
Expected safe outcomes are one production Node PID owning both listeners on `127.0.0.1:8791` and the authenticated control port `127.0.0.1:8792`; 2xx positive responses; non-2xx negative validations without Git or snapshot mutation; an empty render diff; two successful `tht config check` calls; no manifest, Evidence/export, secret, or out-of-scope-artifact finding; and no PID or listener on either port after `stop`.
docs/testing/p11-manual-acceptance.md
-89
View File
@@ -1,89 +0,0 @@
# P1.1 manual acceptance
This walkthrough is the separate human gate for the P1.1 workspace-directory registry.
It is independent from both `.artifacts/p1-integration/**` and `.artifacts/p11-integration/**`.
The helper prepares and serves the lab, but the reviewer performs the registry, Git, UI, export,
render, `tht`, refusal, secret-scan, and cleanup checks and records the verdict.
## Prerequisites
- clean repository checkout with the P1.1 implementation present;
- `node`, `npm`, `git`, `curl`, and `python3` available;
- built production assets:
```bash
npm --prefix backend run build
npm --prefix frontend run build
```
- executable harness CLI at `harness/.venv/bin/tht`;
- free loopback ports `127.0.0.1:8791` and `127.0.0.1:8792`.
## Lifecycle commands
Run from the repository root:
```bash
./scripts/p11-manual-acceptance.sh prepare
./scripts/p11-manual-acceptance.sh serve
./scripts/p11-manual-acceptance.sh stop
./scripts/p11-manual-acceptance.sh cleanup
```
The fixed lab root is:
```text
.artifacts/manual-acceptance/p11/
```
Expected lifecycle behavior:
- `prepare` creates the fixed root, ownership record, bare remote, curator clone, root catalog,
nested filesystem evidence, fixture secrets, request fixtures, generated command scripts, and
`GUIDE.md`; it leaves status `PENDING`, performs no reviewer publish operation, and never writes
`VERDICT.md`.
- `serve` starts the production backend on `127.0.0.1:8791` and a production-built frontend preview
on `127.0.0.1:8792`, recording exact ownership for both.
- `stop` refuses foreign or partial ownership and stops only the two owned loopback processes.
- `cleanup` refuses live state and removes only `.artifacts/manual-acceptance/p11/`.
## Reviewer workflow
After `prepare`, open the generated `.artifacts/manual-acceptance/p11/GUIDE.md` and personally:
1. inspect the catalog, nested descriptor/evidence layout, ownership, and secret-path bindings;
2. serve both surfaces and verify the owned listeners;
3. list `configuration_required` slots;
4. validate and bootstrap-create descriptors exactly once;
5. inspect catalog/descriptor/evidence/docs Git object IDs;
6. retry create/update/delete and verify refusal plus unchanged object IDs;
7. make a curator descriptor+catalog edit, push, pull, and verify the API did not rewrite curator bytes;
8. make an evidence-only commit and inspect the new revision identity;
9. verify the live UI shows read-only existing workspaces and bootstrap-only editing for missing slots;
10. exercise export/import under bootstrap-only rules;
11. render twice, diff the results, and run `tht config check`;
12. run negative catalog/path/secret cases and a bounded secret scan;
13. stop the lab, verify both listeners are gone, write `VERDICT.md`, and only then cleanup if desired.
## Expected outcomes
- `prepare` produces a fresh P1.1-only lab and leaves no `VERDICT.md`.
- `serve` exposes only the owned loopback backend and frontend preview.
- positive API operations succeed once; curator-owned follow-up mutations are refused safely;
- curator Git changes become active only after pull;
- renders are deterministic; `tht config check -c <file>` succeeds;
- secret scans find no canaries outside the fixture-secret boundary;
- after `stop`, nothing remains listening on `127.0.0.1:8791` or `127.0.0.1:8792`.
## Verdict format
The reviewer creates `VERDICT.md` manually. Include:
- reviewer identity;
- UTC timestamp;
- result for each checklist step;
- observations and failure evidence;
- exactly one final line: `manual acceptance: PASS` or `manual acceptance: FAIL`.
Passing `bash scripts/test-p11-manual-acceptance.sh` proves only the tooling/lifecycle guards. It
does not perform or approve manual acceptance.
docs/testing/p2-p6-manual-verification.md
-218
View File
@@ -1,218 +0,0 @@
# P2–P6 Manual Verification Walkthrough
> Living document. Each section is completed with exact released commands and artifacts during its
> corresponding plan. Automated integration and manual acceptance use separate clean state.
## Global rules
- Use a new temporary operator root and a new private fixture Git remote for each Px.
- Never use production PSD credentials in a retained report or screenshot.
- Keep descriptor/content in Git; keep endpoints, bindings, credentials, and certificates in the
installation-local protected directory.
- Do not print secret files, rendered signed URLs, Compose environments, or unbounded logs.
- Record the ThothII commit, workspace commit, installation descriptor path, Compose project name,
command exit status, and report path.
- A focused manual PASS does not replace the automated process goal.
## P2 — Host preprocessing CLI
**Status:** P2 implementation complete; automated integration PASS; manual acceptance PENDING.
Manual goal: from a clean local installation, use only `tht` on the host to inspect one
registry workspace and execute the controlled REST-DWH/HTTP-Evidence preprocessing path without a
host Python or Node runtime. Use a fresh operator root and a fresh fixture Git remote; never reuse
the automated `.artifacts/p2-integration/**` state.
Commands (contract: `docs/contracts/workspace-preprocessing-cli.md`):
```bash
tht --installation <abs>/thothii-installation.yaml workspace inspect --workspace <id> --json
tht --installation <abs>/thothii-installation.yaml workspace preprocess dwh --workspace <id> --json
tht --installation <abs>/thothii-installation.yaml workspace preprocess dwh --workspace <id> --resume <run-id> --json
tht --installation <abs>/thothii-installation.yaml workspace schema suggest-fks --workspace <id> --from-sql <file>.sql --output <candidates>.yaml --json
tht --installation <abs>/thothii-installation.yaml workspace schema check --workspace <id> --annotations <reviewed>.yaml --reviewed-candidates <sha256:hex> --json
tht --installation <abs>/thothii-installation.yaml workspace index-schema --workspace <id> --json
tht --installation <abs>/thothii-installation.yaml workspace preprocess evidence --workspace <id> --dry-run --json
tht --installation <abs>/thothii-installation.yaml workspace preprocess evidence --workspace <id> --json
tht --installation <abs>/thothii-installation.yaml workspace preprocess run --workspace <id> --json
```
Checks:
1. installation/render preflight (`inspect` returns exact revision + catalog/descriptor digests);
2. DWH introspection+LSH succeeds, rerun is `unchanged`, `--resume <run-id>` is `unchanged`/`succeeded`;
3. `schema suggest-fks` returns pristine JSON with `suggestedFksYaml` and a `manual_review_required`
block (exit 3) when candidates exist; the suggested YAML digest equals the reported digest;
4. `schema check --annotations <reviewed> --reviewed-candidates <digest>` succeeds after review;
5. `index-schema` counts against a pre-provisioned compatible collection and rerun is `unchanged`;
6. HTTP Evidence `--dry-run` returns `dry_run`, the real run publishes, rerun is `unchanged`, an input
mutation produces a new generation/ACTIVE;
7. filesystem Evidence returns a stable `evidence_materialization_required` block with no partial
corpus/vector publication;
8. negatives: missing workspace (`workspace_not_activatable`), resume of a nonexistent run
(`preprocessing_resume_mismatch`), invalid annotations digest (`annotation_invalid`), no-Evidence
skip warning, no collection creation, no backend/Pi/frontend listener;
9. secret scan over retained artifacts and exact owned-resource cleanup.
Decision: **PENDING** (independent manual gate; automation never records PASS).
## P3 — Effective configuration and `.tht-dwh`
**Status:** P3 implementation complete; automated integration PASS; manual acceptance PASS (owner approval 2026-08-13).
Manual goal: prove that the operator CLI and application sessions derive the same effective
configuration, that a content-only revision reuses the prepared DWH generation (fast, `unchanged`),
that a DWH-affecting change fails closed and regenerates, that the workspace memory migration is
safe, and that search records are revision-scoped. See `docs/contracts/tht-dwh.md`.
Checks:
1. run `tht ... workspace preprocess dwh` twice with only an Evidence/content change between
them: the second run reports `unchanged` and does not re-introspect;
2. change a DWH-affecting field (host/port/database/schema/user/collection) in the descriptor,
push, pull: the next run refuses the old generation and regenerates, with a clear
`effective_config_mismatch`-style outcome and no mixed artifacts;
3. inspect `.tht-dwh` generations: immutable directories, `OWNER.json` with the canonical
fingerprints, `ACTIVE` pointer; old generations still present;
4. memory: after the guarded migration the workspace uses
`<dataRoot>/sessions/<workspace-id>/memory/`; the JSONL registry and Qdrant projection are
rebuilt and consistent; a conflicting legacy registry fails closed;
5. search records: schema/Evidence points carry the pinned `workspace_revision`; memory/solved
records remain workspace-wide;
6. documentation: `docs/contracts/tht-dwh.md` matches the observed behavior.
Decision: **PASS** (owner approval 2026-08-13).
## P4 — Qdrant bootstrap and guarded rebuild
**Status:** superseded by the "P4 Qdrant collection lifecycle" section below (implemented; manual acceptance PASS).
Manual goal: prove admission creates a missing compatible collection and indexes, refuses an
incompatible collection, and permits destructive rebuild only under durable maintenance with no
active readers/jobs and exact repeated confirmation.
Checks to fill during P4:
1. missing-collection self-heal;
2. missing-index self-heal;
3. dimensions/distance/index-type refusal;
4. confirmation mismatch refusal;
5. active-reader/job refusal;
6. successful drained rebuild;
7. interrupted rebuild recovery with maintenance retained.
Decision: **PASS** (owner approval 2026-08-13; see the section below).
## P4 Qdrant collection lifecycle
Manual goal: verify admission self-heal and the guarded rebuild through the real product surface.
Checks to complete during P4 manual acceptance (decision: **PASS** (owner approval 2026-08-13)):
1. On a fresh installation with no Qdrant collection, a session admission creates the
descriptor collection with exactly 1024 dimensions, cosine distance, and the 8 required
keyword payload indexes (`content_hash`, `document_id`, `kind`, `record_key`,
`record_kind`, `vector_generation`, `workspace_id`, `workspace_revision`).
2. A pre-existing collection with incompatible dimensions/distance (e.g. 768-dim or dot)
is refused with `semantic_index_incompatible` and is never mutated.
3. `tht ... workspace vector inspect --workspace <id> --json` reports the collection
contract without mutation (pristine JSON, exit 0).
4. `tht ... workspace vector rebuild --workspace <id> --collection <name>
--confirm <name> --destroy` deletes and recreates the descriptor-owned collection and
verifies the recreated contract; a mismatched `--confirm` or a missing `--destroy` is
refused (exit 2) without touching the collection.
5. Rebuild writes durable state before deletion, deletes only the descriptor collection,
and the recreated collection preserves the P3 revision-scoped payload contract.
## P5 — Curated FK annotations in Git
**Status:** P5 implementation complete; automated integration PASS; manual acceptance PASS (owner approval 2026-08-13).
Manual goal: curate `<id>/schema/annotations.yaml` in an author clone, publish it, pull the new
revision, and prove the revision-pinned sync and the explicit `schema accept` review, without ever
pushing curated content from the operator CLI.
Commands (contract: `docs/contracts/workspace-preprocessing-cli.md`):
```bash
tht --installation <abs>/thothii-installation.yaml workspace schema suggest-fks --workspace <id> --from-sql <file>.sql --output <candidates>.yaml --json
# curate the candidate into <id>/schema/annotations.yaml in the author clone, then commit/push/pull
tht --installation <abs>/thothii-installation.yaml workspace schema accept --workspace <id> --run <run-id> --yes --json
tht --installation <abs>/thothii-installation.yaml workspace preprocess run --workspace <id> --resume <run-id> --json
```
Checks:
1. `schema suggest-fks` returns pristine JSON with `suggestedFksYaml` and a `manual_review_required`
block (exit 3) when candidates exist; the suggested YAML digest equals the reported digest;
2. after commit/push/pull, activation reads `<id>/schema/annotations.yaml` as a regular Git blob at
the same commit as the descriptor and synchronizes it to
`<data>/sessions/<id>/revisions/<commit>/artifacts/mschema/annotations.yaml` with a restrictive
mode and an adjacent ownership manifest `{ workspace, commit, blobId, contentDigest, destination }`;
3. two revisions write two different directories; a session pinned to an older revision reads its own
revision's annotations;
4. `schema accept --run <id> --yes` records the accepted candidate/current-blob digests and the new
revision; missing `--yes`, an unknown run, an empty file, a malformed blob, or a blob not matching
the recorded candidate is refused (exit 1, `annotation_invalid`) without recording a review;
5. `preprocess run --resume <id>` continues only with the exact accepted blob digest and compatible
DWH binding; otherwise it records a new `manual_review_required` checkpoint;
6. negatives: symlink/tree-at-path, cross-namespace, oversized (>16 MiB), non-UTF-8, and malformed
annotation objects are refused at activation without mutating the snapshot or runtime roots;
7. the operator CLI never stages/commits/pushes curated content; secret scan and exact owned-resource
cleanup pass.
Decision: **PASS** (owner approval 2026-08-13).
## P6 — Commit-addressed Evidence materialization
**Status:** P6 implementation complete; automated integration PASS; manual acceptance PASS (owner approval 2026-08-13).
Manual goal: materialize filesystem Evidence from the pinned Git commit, inspect its bounded
manifest, preprocess/index it, retrieve only the pinned revision, and exercise unsafe-tree and
aggregate-limit failures without partial publication.
Commands (contract: `docs/contracts/workspace-preprocessing-cli.md`):
```bash
tht --installation <abs>/thothii-installation.yaml workspace inspect --workspace <id> --json
tht --installation <abs>/thothii-installation.yaml workspace preprocess evidence --workspace <id> --dry-run --json
tht --installation <abs>/thothii-installation.yaml workspace preprocess evidence --workspace <id> --json
tht --installation <abs>/thothii-installation.yaml workspace preprocess evidence --workspace <id> --json # idempotent rerun
```
Checks:
1. exact commit/tree/object identities: after activation the materialized root is
`<registry>/snapshots/<commit>/<id>/evidence` and its sibling manifest
`<id>/evidence.manifest.json` records `workspace`, `commit`, `tree`, per-file `oid`/`digest`,
`entryCount`, `totalBytes`; `snapshot.json` chains the manifest digest;
2. successful atomic materialization: every regular blob is present byte-for-byte; the manifest
digests match;
3. manifest and file digest verification: re-activation reuses a valid root and fails closed on a
tampered manifest;
4. filesystem Evidence dry-run/run/idempotency: `--dry-run` returns `dry_run`, the real run
publishes, rerun is `unchanged`;
5. revision-filtered Qdrant retrieval and corpus ACTIVE: Evidence records carry the pinned
`workspace_revision`;
6. nested symlink/gitlink/traversal/special-file refusal: a commit introducing one of these fails
activation (`workspace_invalid`) and the previous valid revision stays active;
7. file-count/total-byte/path/manifest limit refusal: an oversized or over-count tree fails closed
without a partial publication;
8. retention while pinned and owned cleanup after release: the materialized root persists for a
pinned revision and is removed with its snapshot directory once unreferenced.
Decision: **PASS** (owner approval 2026-08-13).
## Final aggregate P2–P6 verification
**Status:** runnable; automated integration PASS; manual acceptance PENDING.
The automated aggregate (run `p2p6-ee542112c526ef0d4c25ddf6c8bc164b`, report
`.artifacts/p2p6-integration/...`) already executed the complete DWH → FK → schema → filesystem
Evidence chain, idempotency, revision isolation, a second installation, unsafe-tree/bound negatives,
secret scan, and exact cleanup.
The final manual pass will start with a new registry and two independent installations. It will
run the complete DWH → FK → schema → filesystem Evidence chain, prove idempotency and revision
isolation, confirm the second installation uses its own secrets/state, and compare its observations
to the retained aggregate automated report.
Decision: **PENDING**.
docs/testing/psd-server-project-a-manual.md
-177
View File
@@ -1,177 +0,0 @@
# Progetto A PSD — collaudo manuale
Questo documento guida il collaudo umano del nuovo ThothII sul server con autenticazione locale.
Non sostituisce i controlli automatici del piano. Compilarlo soltanto dopo che Sol ha dichiarato
verdi installazione, workspace, DWH, Qdrant, Ollama e preprocessing.
## Regole
- Eseguire i comandi dal terminale locale del server; non usare tunnel SSH.
- Non copiare nel rapporto password, cookie, token, chiavi, hash, stringhe di connessione o righe di
log che li contengano.
- Usare un amministratore locale e un utente ordinario creati appositamente.
- Non effettuare più tentativi di password errata del necessario: il login applica rate limiting.
- Per ogni prova segnare `PASS`, `FAIL` o `PENDING`, con una nota breve e non sensibile.
- Un solo `FAIL` obbligatorio impedisce di avviare il Progetto B.
## Dati iniziali
| Campo | Valore redatto |
|---|---|
| Data/ora UTC | |
| SHA ThothII | |
| SHA workspace | |
| Installation descriptor | percorso protetto, senza contenuto |
| Origine di test | loopback oppure hostname privato |
| Endpoint temporaneo usato | sì/no |
| ID domanda di prova approvata | |
| Operatore | |
## 1. Stato generale
Prima del gate manuale di avvio, per una descriptor server con runtime projection eseguire solo il
controllo redatto `sudo tht --installation "$INSTALLATION" auth status --json`. Il risultato deve
dire `ready` ed `equal: true`. Se è `blocked`, mancante o diverso dal canonical root, non avviare:
Sol può eseguire `sudo tht --installation "$INSTALLATION" auth publish` e ripetere il controllo,
senza copiare YAML, hash, password, token o environment nel rapporto. Questo documento non
autorizza l'avvio; Project A resta soggetto a un'esplicita autorizzazione separata.
Eseguire:
```bash
THT_BIN=<percorso-tht>
INSTALLATION=<percorso-assoluto-thothii-installation.yaml>
"$THT_BIN" --installation "$INSTALLATION" status
"$THT_BIN" --installation "$INSTALLATION" doctor --json
"$THT_BIN" --installation "$INSTALLATION" auth check --json
"$THT_BIN" --installation "$INSTALLATION" pi test
```
| Prova | Risultato atteso | Esito | Note |
|---|---|---|---|
| Stato servizi | frontend, core, qdrant ed embedding sani; initializer completato | | |
| Doctor | tutti i controlli obbligatori passano | | |
| Autenticazione | modalità `local`, configurazione pronta | | |
| Pi | provider e modello rispondono | | |
| Secret hygiene | nessun secret nell’output | | |
## 2. Confine di rete
Dal terminale controllare i listener e la configurazione renderizzata secondo il piano.
| Prova | Risultato atteso | Esito | Note |
|---|---|---|---|
| Frontend | pubblicato solo su loopback o tramite endpoint privato approvato | | |
| Core | nessuna porta host pubblica | | |
| Qdrant | nessuna porta host pubblica nel profilo server | | |
| Ollama | nessuna porta host pubblica | | |
| URL produzione | non raggiunge il nuovo stack | | |
| Endpoint privato, se usato | sorgente autorizzata ammessa | | |
| Endpoint privato, se usato | sorgente non autorizzata respinta prima di ThothII | | |
Se non esiste un endpoint privato, usare il browser headless/API sul server. Non segnare come
eseguite prove browser che non sono state realmente svolte.
## 3. Autenticazione locale
Eseguire tramite frontend/browser quando disponibile; altrimenti usare richieste same-origin dal
terminale, conservando cookie e password soltanto in file temporanei mode `0600`, poi eliminandoli.
| Prova | Azione | Risultato atteso | Esito | Note |
|---|---|---|---|---|
| Accesso anonimo | aprire pagina/API protetta | appare login oppure HTTP 401 | | |
| Password errata | un tentativo con utente valido | errore generico; nessun dettaglio account | | |
| Utente ordinario | login corretto | accesso alle sessioni | | |
| Confine ruoli | aprire Pi Management/amministrazione | negato o non visibile | | |
| Logout | uscire e ricaricare | sessione rifiutata, nuovo login richiesto | | |
| Amministratore | login corretto | funzioni amministrative previste disponibili | | |
| Disabilitazione | Sol disabilita l’utente di prova | login rifiutato genericamente | | |
| Riabilitazione | Sol riabilita l’utente | login nuovamente possibile | | |
| Invalidazione | cambio password/ruolo o `logout-all` | vecchia sessione non più valida | | |
| Remember me | login persistente, riavvio core | sessione ancora valida entro TTL | | |
| CSRF | mutazione senza token corretto | richiesta respinta | | |
Non disabilitare o demansionare l’ultimo amministratore abilitato.
## 4. Workspace e DWH
Eseguire:
```bash
"$THT_BIN" --installation "$INSTALLATION" \
workspace inspect --workspace psd-clinical --json
"$THT_BIN" --installation "$INSTALLATION" \
workspace vector inspect --workspace psd-clinical --json
```
| Prova | Risultato atteso | Esito | Note |
|---|---|---|---|
| Revisione Git | coincide con lo SHA approvato | | |
| Trasporto server | `postgres_direct` | | |
| Database/schema | database Supabase rilevato, schema `datawarehouse` | | |
| Utente DWH | read-only dimostrato dai grant | | |
| Workspace Mac | `DEFERRED_PRE_PROJECT_B`; in quel gate deve confermare `rest_api` | | |
| Qdrant | 1024 dimensioni, cosine, indici payload richiesti | | |
| Ollama | `qwen3-embedding:0.6b` | | |
| Evidence | corpus Git attivo alla stessa revisione | | |
Per l'emendamento del proprietario del 2026-08-21, solo la riga Workspace Mac può restare
`DEFERRED_PRE_PROJECT_B` nella chiusura privata di Project A. Non equivale a PASS e deve essere
eseguita prima di Project B insieme all'osservazione dual-key e alla revoca legacy.
## 5. Preprocessing e idempotenza
Esaminare i due risultati consecutivi del preprocessing prodotti da Sol.
| Prova | Risultato atteso | Esito | Note |
|---|---|---|---|
| Introspezione DWH | completata senza scritture cliniche | | |
| Annotazioni FK | revisione umana registrata e legata al digest corretto | | |
| Schema index | record presenti con workspace revision | | |
| Evidence index | documenti/chunk presenti con workspace revision | | |
| Seconda esecuzione | nessun duplicato; contenuti invariati riconosciuti | | |
| Identità effettiva | invariata tra i due run | | |
## 6. Sessione completa F1–F8
Usare una domanda innocua approvata, senza identificativi reali di pazienti.
| Fase | Controllo manuale | Esito | Note |
|---|---|---|---|
| F1 | domanda compresa/disambiguata correttamente | | |
| F2 | concetti e contesto coerenti | | |
| F3 | tabelle candidate ragionevoli | | |
| F4 | colonne/join curati e confermati | | |
| F5 | piano CTE comprensibile | | |
| F6 | ogni CTE testata e approvata | | |
| F7 | SQL finale read-only e validato | | |
| F8 | conclusione, memoria e riepilogo coerenti | | |
Durante una fase intermedia chiudere/riprendere la sessione una volta. Il resume deve tornare
all’ultima fase incompleta senza creare una nuova domanda.
Verificare infine:
| Prova | Risultato atteso | Esito | Note |
|---|---|---|---|
| Stato | sessione `finalized` | | |
| SQL | solo lettura; validazione DWH verde | | |
| Artefatti | manifest, question, schema linking, Evidence, CTE, SQL, validation presenti | | |
| Decisioni | gate registrati nel ledger | | |
| Persistenza | artefatti leggibili dopo riavvio | | |
| Chat/SSE | non richiesti come persistenza | | |
## 7. Decisione
| Gate | Esito |
|---|---|
| Tutti i controlli obbligatori PASS | |
| Nessun secret raccolto | |
| Rollback vecchio stack ancora disponibile | |
| Progetto B autorizzabile | NO finché il gate Mac/osservazione/revoca non è PASS |
Decisione finale: `PROJECT_A_PRIVATE_PASS` / `PROJECT_A_FAIL` / `PROJECT_A_PENDING`
Revisore e data: ______________________________________
Motivazione sintetica: ______________________________________
docs/testing/psd-server-project-b-manual.md
-131
View File
@@ -1,131 +0,0 @@
# Progetto B PSD — collaudo manuale Authentik e Aritmolab
Questo documento verifica il percorso finale di produzione. Si esegue soltanto dopo il PASS del
Progetto A e dopo che Sol ha completato i preflight Authentik, Supabase, Nginx e bilanciatore.
## Regole
- Usare identità di prova approvate: una ordinaria, una amministrativa e, se disponibile, una senza
gruppi ThothII.
- Non acquisire token, cookie, password, chiavi private, claim completi o trace browser contenenti
URL di callback con parametri.
- Partire dalla home reale di Aritmolab, non da un URL interno di ThothII.
- Segnare `PASS`, `FAIL` o `PENDING`; non dedurre il PASS da test automatici.
## Dati iniziali
| Campo | Valore redatto |
|---|---|
| Data/ora UTC | |
| SHA ThothII/workspace | |
| Origine pubblica | |
| SHA/revisione Aritmolab | |
| Nome/ID applicazione Authentik | non inserire secret |
| Database Supabase | |
| Schema sessioni | `thoth_sessions` |
| Operatore/revisore | |
## 1. TLS, routing e pagina iniziale
| Prova | Azione | Risultato atteso | Esito | Note |
|---|---|---|---|---|
| HTTP | aprire origine in HTTP | redirect a HTTPS | | |
| Certificato | ispezionare il lucchetto/catena | hostname corretto, nessun warning | | |
| Home Aritmolab | aprire URL ufficiale | pagina disponibile | | |
| Sidebar | individuare ThothII | link presente come prima | | |
| Destinazione | aprire il link | nuovo frontend ThothII | | |
| API | caricare l’app | nessun 502/404 o mixed content | | |
| SSE | avviare attività modello | aggiornamenti continui, niente buffering evidente | | |
## 2. Single sign-on
Chiudere ogni precedente sessione di test secondo la procedura concordata. Accedere ad Aritmolab
con l’identità ordinaria, quindi aprire ThothII dalla sidebar.
| Prova | Risultato atteso | Esito | Note |
|---|---|---|---|
| Primo login | Authentik autentica l’utente | | |
| Passaggio sidebar | nessuna seconda richiesta di credenziali | | |
| Callback | ritorno all’origine pubblica ThothII | | |
| Identità | nome visualizzato coerente, senza dati grezzi del token | | |
| Browser storage | nessun access/id token in Local/Session Storage | | |
| Cookie | cookie ThothII HttpOnly/Secure/SameSite secondo configurazione | | |
Non copiare il valore del cookie nel rapporto.
## 3. Ruoli e autorizzazione
| Identità/caso | Risultato atteso | Esito | Note |
|---|---|---|---|
| Gruppo utente | può creare, leggere e gestire le proprie sessioni | | |
| Gruppo utente | Pi Management e funzioni admin negate con 403/non visibili | | |
| Gruppo admin | funzioni amministrative documentate disponibili | | |
| Nessun gruppo mappato | autenticato ma operazioni protette negate | | |
| Gruppo estraneo aggiuntivo | nessun cambiamento e nessun warning | | |
| Header identità forgiato | nessun privilegio aggiuntivo | | |
Le prove su claim mancante/malformato possono essere eseguite da Sol con un’identità/provider di
test controllato. Il revisore verifica soltanto esito HTTP generico e report redatto, mai il token.
## 4. Logout e riavvio
| Prova | Azione | Risultato atteso | Esito | Note |
|---|---|---|---|---|
| Logout ThothII | usare il comando dell’app | cookie ThothII revocato | | |
| SSO ancora attivo | riaprire ThothII | possibile nuovo accesso senza password; documentare | | |
| Logout Authentik globale | se configurato e in scope | comportamento conforme alla policy locale | | |
| Riavvio core | Sol riavvia in finestra controllata | sessione browser valida secondo TTL/policy | | |
| Provider indisponibile | prova controllata | nuovo login fallisce chiuso e redatto | | |
| Ripristino provider | ripetere diagnosi/login | servizio torna operativo | | |
Non dichiarare “logout globale” se è stato testato soltanto il logout locale di ThothII.
## 5. Sessioni PostgreSQL e isolamento
| Prova | Risultato atteso | Esito | Note |
|---|---|---|---|
| Migrazioni | `pending=[]`, `drifted=[]` | | |
| Schema | `thoth_sessions` nel database Supabase esistente | | |
| PostgREST | schema non esposto | | |
| RLS | forzata sulle tabelle previste | | |
| Utente A/B | ciascuno vede soltanto le proprie sessioni | | |
| Accesso incrociato | risposta not-found/negata come da contratto | | |
| Admin | accesso trasversale solo secondo permessi documentati | | |
| Credenziale migratore | non montata nel core | | |
| Schema clinico | nessun nuovo privilegio runtime | | |
## 6. Sessione completa sotto OIDC
Come utente ordinario, eseguire una domanda innocua approvata e completare F1–F8.
| Prova | Risultato atteso | Esito | Note |
|---|---|---|---|
| Creazione | sessione associata all’identità OIDC | | |
| Gate F1–F8 | tutti presentati e registrati correttamente | | |
| Resume | ritorna alla sessione corretta | | |
| SQL finale | sola lettura e validato | | |
| Persistenza | manifest, artefatti e decisioni in PostgreSQL | | |
| SSE/chat | funzionano live; non richiesti come artefatti persistiti | | |
| Riavvio | sessione di lavoro ancora disponibile | | |
## 7. Integrazione e pulizia finale
| Prova | Risultato atteso | Esito | Note |
|---|---|---|---|
| Endpoint temporaneo A | rimosso/non instradato | | |
| Vecchio stack | fermo, non esposto | | |
| Link sidebar | punta solo alla nuova release | | |
| Servizi privati | core/Qdrant/Ollama non pubblicati | | |
| Altri servizi Nginx | invariati e sani | | |
| Rollback | procedura verificata e disponibile | | |
| Evidenze | nessun secret o dato clinico identificabile | | |
## 8. Decisione
Decisione finale: `PROJECT_B_PASS` / `PROJECT_B_FAIL` / `PROJECT_B_PENDING`
Revisore e data: ______________________________________
Motivazione sintetica: ______________________________________
Conferma percorso finale “Aritmolab → sidebar → ThothII → SSO”: ______________________________
docs/workspace-diagnostic-protocol.md
-151
View File
@@ -1,151 +0,0 @@
# Workspace diagnostic protocol
This is the operator contract for testing a workspace on one ThothII installation. The Git-shared
descriptor declares what can be checked; the installation supplies only the selected DWH
transport and local secret-file bindings. No secret value, certificate content, SSH key, or
response body belongs in the descriptor, generated `.env.example` files, or diagnostic output.
## Scope and safety rules
<!-- workspace-descriptor-contract:start -->
Schema v3 is the only accepted workspace descriptor.
Schema v1 and v2 workspace descriptors are rejected before activation.
- Diagnostics do not run for a rejected descriptor. There is no in-product migrator or automatic
conversion; the Git repository must already contain reviewed v3 descriptors.
<!-- workspace-descriptor-contract:end -->
- One workspace owns one Qdrant collection.
- Qdrant and Ollama are internal services. Operators do not bind external vector or embedding
transports for active manuals or supported diagnostics.
- Each diagnostic is bounded by the configured timeout. Redirects are rejected, response bodies
stay inside the adapter, and browser-visible errors are limited to `binding_missing`,
`connector_unavailable`, and `semantic_index_incompatible`.
## Canonical descriptor contract
```yaml
workspace:
schema_version: 3
id: psd-clinical
name: PSD Clinical
language: it
dwh:
engine: postgres
database: warehouse
schema: datawarehouse
supported_transports: [postgres_direct, rest_api, ssh_tunnel]
semantic_index:
vector_store:
engine: qdrant
collection: psd-clinical
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
diagnostics:
dwh_rest:
method: POST
path: /rpc/ping
auth: bearer
response: { database: database, schema: schema }
```
The semantic-index contract is fixed:
- `engine: qdrant`
- collection name equals the workspace-owned portable identifier
- `qwen3-embedding:0.6b`
- `1024` dimensions
- cosine distance
If any active collection reports a different model pairing, dimension, or distance, diagnostics
must return `semantic_index_incompatible` rather than silently rewriting data.
## Installation-local variable contract
Replace `<NAMESPACE>` with the immutable workspace ID converted to upper case with hyphens changed
to underscores. For example, `psd-clinical` becomes `PSD_CLINICAL`. Set only the variables for the
selected DWH transport. Every `*_FILE` value is an absolute path to a regular, readable file
inside an approved local secret root; it is never the secret itself.
| Connector and transport | Required local variables |
| --- | --- |
| DWH selection | `THT_WS_<NAMESPACE>_DWH_TRANSPORT` |
| DWH `postgres_direct` | `THT_WS_<NAMESPACE>_DWH_HOST`, `THT_WS_<NAMESPACE>_DWH_PORT`, `THT_WS_<NAMESPACE>_DWH_USER`, `THT_WS_<NAMESPACE>_DWH_PASSWORD_FILE`; optional `THT_WS_<NAMESPACE>_DWH_TLS_CA_FILE` |
| DWH `rest_api` | `THT_WS_<NAMESPACE>_DWH_BASE_URL`; `THT_WS_<NAMESPACE>_DWH_API_KEY_FILE` only for `bearer`/`x-api-key`; optional `THT_WS_<NAMESPACE>_DWH_TLS_CA_FILE` |
| DWH `ssh_tunnel` | `THT_WS_<NAMESPACE>_DWH_USER`, `THT_WS_<NAMESPACE>_DWH_PASSWORD_FILE`, `THT_WS_<NAMESPACE>_DWH_SSH_HOST`, `THT_WS_<NAMESPACE>_DWH_SSH_PORT`, `THT_WS_<NAMESPACE>_DWH_SSH_USER`, `THT_WS_<NAMESPACE>_DWH_SSH_PRIVATE_KEY_FILE`, `THT_WS_<NAMESPACE>_DWH_SSH_KNOWN_HOSTS_FILE`, `THT_WS_<NAMESPACE>_DWH_SSH_TARGET_HOST`, `THT_WS_<NAMESPACE>_DWH_SSH_TARGET_PORT`; optional `THT_WS_<NAMESPACE>_DWH_TLS_CA_FILE` |
There are no supported `THT_WS_<NAMESPACE>_VECTOR_*` or
`THT_WS_<NAMESPACE>_EMBEDDING_*` installation bindings in the active operator contract.
## DWH diagnostic
For direct PostgreSQL and SSH-tunnelled PostgreSQL, the diagnostic connects with the declared
`dwh.database`, checks TLS and authentication, then executes exactly:
```sql
SELECT current_database() AS database, current_schema() AS schema
```
Both returned values must equal the descriptor's DWH database and schema.
For REST, the descriptor-declared request is for example:
```text
POST <THT_WS_<NAMESPACE>_DWH_BASE_URL>/rpc/ping
Authorization: Bearer <content of DWH_API_KEY_FILE>
```
It has no request body. A 2xx response must be a JSON object whose declared `database` and
`schema` fields match the descriptor.
## Internal semantic-service diagnostic
Schema-v3 workspace diagnostics also verify the internal semantic infrastructure through backend
configuration:
- Qdrant must be reachable at the installation-owned internal URL.
- The workspace-owned collection must exist or be creatable with `1024` dimensions and cosine
distance.
- Ollama must provide `qwen3-embedding:0.6b`.
- A bounded embed probe must return exactly `1024` dimensions.
These checks use the private Compose services and never require operator-supplied vector or
embedding URLs, transports, or credentials.
## SSH host verification and tunnel lifecycle
For DWH `ssh_tunnel`, the known-hosts file is mandatory and is verified before a connection is
accepted. The tunnel is a short-lived loopback forward for the diagnostic only. The effective
OpenSSH constraints are:
```text
-N -v
-o BatchMode=yes
-o ExitOnForwardFailure=yes
-o StrictHostKeyChecking=yes
-o UserKnownHostsFile=<ROLE>_SSH_KNOWN_HOSTS_FILE
-i <ROLE>_SSH_PRIVATE_KEY_FILE
-p <ROLE>_SSH_PORT
-L 127.0.0.1:<ephemeral-port>:<ROLE>_SSH_TARGET_HOST:<ROLE>_SSH_TARGET_PORT
<ROLE>_SSH_USER@<ROLE>_SSH_HOST
```
The local listener is `127.0.0.1` only. The process is terminated in cleanup after the direct
probe, on timeout, or on failure.
In this release, `ssh_tunnel` remains a diagnostic-only DWH transport. A successful probe is
followed by `workspace_not_activatable`, and `POST /sessions` rejects the workspace before
persisting a manifest or starting Pi. This restriction does not apply to SSH transport for the
workspace Git remote.
## Reader-only fallback
A workspace may be fully valid in Git but non-activatable locally when a required DWH binding,
secret file, host verification, TLS check, or declared DWH diagnostic fails. That state does not
alter the shared descriptor and does not permit a new session on that installation. It may still
be published and activated elsewhere with valid local bindings.
mkdocs.yml
+1 -15
View File
@@ -1,9 +1,6 @@
site_name: ThothII Docs
site_description: Documentazione tecnica di ThothII e considerazioni generali sull'ambiente di sviluppo
site_description: Documentazione funzionale, tecnica e operativa di ThothII
site_url: https://git.tylconsulting.it/thothii-docs/
repo_url: https://git.tylconsulting.it/mptyl/ThothII
repo_name: mptyl/ThothII
edit_uri: edit/main/docs/
docs_dir: docs
site_dir: site
use_directory_urls: true
@@ -48,21 +45,12 @@ markdown_extensions:
nav:
- Home: index.md
- Guida utente: guida-utente.md
- Accettazione autenticazione: testing/authentication-manual-acceptance.md
- Setup Policlinico San Donato: install/psd-workspace-setup.md
- DWH REST per installazione:
- Server dwh-auth: install/dwh-auth-server.md
- Enrollment client DWH: install/dwh-auth-client-enrollment.md
- TLS DWH REST: install/dwh-auth-tls.md
- Rollout PSD DWH: operations/psd-dwh-auth-rollout.md
- Collaudo manuale DWH: testing/dwh-auth-manual-acceptance.md
- Template evidenza DWH: testing/evidence/psd-dwh-auth-rollout-report-template.md
- Programma deploy server PSD: plans/2026-08-20-psd-server-deployment-program.md
- Collaudo PSD Progetto A: testing/psd-server-project-a-manual.md
- Collaudo PSD Progetto B: testing/psd-server-project-b-manual.md
- Contratti e CLI:
- CLI workspace preprocessing: contracts/workspace-preprocessing-cli.md
- Contratto tht–Pi: contracts/tht-pi.md
- Contratto tht–DWH: contracts/tht-dwh.md
- Contratto Evidence workspace v3: contracts/workspace-evidence-v3.md
- ThothII (Documentazione Tecnica):
@@ -74,10 +62,8 @@ nav:
- OIDC generico: install/authentication-oidc.md
- Authentik: install/authentik.md
- Installazione Docker (4 contesti): installazione-docker-4-contesti.md
- Ristrutturazione Evidence: plans/2026-08-24-evidence-restructuring-design.md
- Gestione delle memory: gestione-memory.md
- Skill operative: skills.md
- Testo completo skill tht-sessione: skill-tht-sessione.md
- Disambiguazione iniziale: disambiguazione-iniziale.md
- Considerazioni Generali:
- Configurazione dei modelli in Pi: general/pi-configuration.md