docs: focus public documentation on product usage
This commit is contained in:
@@ -1,210 +0,0 @@
|
||||
# Internal Qdrant and Ollama Architecture Design
|
||||
|
||||
**Status:** approved on 2026-08-08
|
||||
|
||||
## Objective
|
||||
|
||||
ThothII owns its semantic infrastructure. Every supported deployment includes a private Qdrant
|
||||
service and a private Ollama embedding service. The analytical DWH remains external and read-only;
|
||||
each workspace descriptor associates that DWH with one Qdrant collection used for database schema,
|
||||
Evidence, and approved Memory records.
|
||||
|
||||
## Decisions
|
||||
|
||||
- Qdrant replaces pgvector as the only operational vector store.
|
||||
- Ollama replaces workspace-selected external embedding endpoints.
|
||||
- The default and required model is `qwen3-embedding:0.6b` with 1024-dimensional normalized dense
|
||||
embeddings and cosine distance.
|
||||
- One Qdrant collection belongs to one workspace. Schema, Evidence, and Memory points share that
|
||||
collection and are separated by indexed payload field `kind`.
|
||||
- Qdrant and Ollama are mandatory base-Compose services. They are not published on host ports and
|
||||
are reachable only from the private Compose network.
|
||||
- Existing schema-v1 and schema-v2 descriptors remain readable for migration, but they are not
|
||||
activatable. The new operational contract is workspace schema v3.
|
||||
|
||||
The model choice is based on the published Qwen model card: the 0.6B model supports more than 100
|
||||
languages, a 32K context window, Matryoshka dimensions up to 1024, and instruction-aware retrieval.
|
||||
Ollama distributes a CPU-viable quantized build and can use an exposed GPU without changing the
|
||||
application protocol.
|
||||
|
||||
References:
|
||||
|
||||
- <https://huggingface.co/Qwen/Qwen3-Embedding-0.6B>
|
||||
- <https://ollama.com/library/qwen3-embedding>
|
||||
- <https://docs.ollama.com/capabilities/embeddings>
|
||||
- <https://qdrant.tech/documentation/installation/>
|
||||
- <https://qdrant.tech/documentation/manage-data/collections/>
|
||||
|
||||
## Target topology
|
||||
|
||||
```text
|
||||
browser -> frontend -> core -> external DWH
|
||||
-> private Qdrant
|
||||
-> private Ollama embedding
|
||||
```
|
||||
|
||||
The base Compose project contains:
|
||||
|
||||
- `frontend`: static React application and same-origin API proxy.
|
||||
- `core`: Fastify, Pi, and the Python `tht` harness.
|
||||
- `qdrant`: pinned Qdrant server with persistent `qdrant-data` volume.
|
||||
- `embedding`: pinned Ollama server with persistent `embedding-models` volume.
|
||||
- `embedding-model-init`: bounded one-shot service that pulls and verifies
|
||||
`qwen3-embedding:0.6b`; `core` starts only after it succeeds.
|
||||
|
||||
`qdrant` and `embedding` use `expose`, not `ports`. The core receives installation-owned internal
|
||||
URLs:
|
||||
|
||||
```text
|
||||
THT_INTERNAL_QDRANT_URL=http://qdrant:6333
|
||||
THT_INTERNAL_EMBEDDING_URL=http://embedding:11434
|
||||
THT_INTERNAL_EMBEDDING_MODEL=qwen3-embedding:0.6b
|
||||
THT_INTERNAL_EMBEDDING_DIMENSIONS=1024
|
||||
```
|
||||
|
||||
These are deployment facts, not workspace connector bindings. The runtime rejects non-loopback or
|
||||
non-Compose-service hosts when these variables are overridden for development.
|
||||
|
||||
An optional Linux GPU override exposes an available NVIDIA/AMD device to Ollama. The base profile
|
||||
must remain CPU-safe. macOS Docker remains CPU-only because Docker Desktop cannot expose the Apple
|
||||
GPU to an Ollama container.
|
||||
|
||||
## Workspace schema v3
|
||||
|
||||
The workspace itself is the association between the external database and the internal collection:
|
||||
|
||||
```yaml
|
||||
workspace:
|
||||
schema_version: 3
|
||||
id: psd-clinical
|
||||
name: PSD Clinical
|
||||
language: it
|
||||
|
||||
dwh:
|
||||
engine: postgres
|
||||
database: postgres
|
||||
schema: datawarehouse
|
||||
supported_transports: [postgres_direct]
|
||||
|
||||
semantic_index:
|
||||
vector_store:
|
||||
engine: qdrant
|
||||
collection: psd-clinical
|
||||
dimensions: 1024
|
||||
distance: cosine
|
||||
embedding:
|
||||
provider: ollama_internal
|
||||
model: qwen3-embedding:0.6b
|
||||
dimensions: 1024
|
||||
|
||||
llm_policy:
|
||||
allowed: [zai/glm-5.2]
|
||||
```
|
||||
|
||||
Invariants:
|
||||
|
||||
- the collection name is an explicit portable identifier;
|
||||
- active workspaces cannot share a collection;
|
||||
- vector and embedding dimensions are both 1024;
|
||||
- distance is `cosine`;
|
||||
- provider and model are exactly the supported internal values;
|
||||
- no vector transport, vector credential, embedding URL, or embedding credential may appear in a
|
||||
schema-v3 descriptor or installation contract;
|
||||
- DWH connectors remain installation-local and can still use the supported external DWH transports.
|
||||
|
||||
Schema-v1/v2 pgvector descriptors are listed as `migration_required`. Migration creates a reviewed
|
||||
schema-v3 document; it does not copy vector data implicitly. Existing semantic data is rebuilt from
|
||||
the canonical schema documents, Evidence corpus, and Memory registry.
|
||||
|
||||
## Qdrant data model
|
||||
|
||||
Each point has a deterministic UUIDv5 derived from:
|
||||
|
||||
```text
|
||||
workspace_id + kind + record_key
|
||||
```
|
||||
|
||||
The vector is the 1024-dimensional Ollama result. The payload is:
|
||||
|
||||
```json
|
||||
{
|
||||
"workspace_id": "psd-clinical",
|
||||
"kind": "schema",
|
||||
"source_id": "datawarehouse.patients",
|
||||
"record_key": "schema:table:datawarehouse.patients",
|
||||
"content_hash": "sha256:...",
|
||||
"workspace_revision": "<git commit>",
|
||||
"generation": "<optional corpus generation>",
|
||||
"language": "it",
|
||||
"text": "...",
|
||||
"metadata": {}
|
||||
}
|
||||
```
|
||||
|
||||
`kind`, `source_id`, `content_hash`, `workspace_revision`, and `generation` receive keyword payload
|
||||
indexes. Queries always filter by `workspace_id` and an explicit allowed `kind` set. Upsert is
|
||||
idempotent. Evidence generation deletion is an exact filtered delete. Collection creation is also
|
||||
idempotent and fails closed if an existing collection has incompatible dimensions or distance.
|
||||
|
||||
## Harness integration
|
||||
|
||||
The existing `VectorStore` port remains the workflow boundary. A `QdrantVectorStore` adapter maps
|
||||
its operations to Qdrant REST endpoints while preserving current schema/Evidence/Memory call sites.
|
||||
The existing Ollama embedding client is narrowed to the internal `/api/embed` contract and verifies:
|
||||
|
||||
- configured model exists;
|
||||
- output count matches input count;
|
||||
- every vector has 1024 finite numeric values;
|
||||
- no remote URL or API key is accepted.
|
||||
|
||||
The JSONL Memory registry and persisted phase documents remain canonical. Qdrant remains a derived,
|
||||
rebuildable semantic index. Schema, Evidence, and Memory ingestion all use the same point builder,
|
||||
content hashing, and retry policy.
|
||||
|
||||
## Readiness and failure behavior
|
||||
|
||||
Readiness is layered:
|
||||
|
||||
1. Compose waits for Qdrant health.
|
||||
2. Compose waits for Ollama health and successful model initialization.
|
||||
3. Workspace activation validates the schema-v3 contract.
|
||||
4. Harness readiness ensures the Qdrant collection and checks its vector configuration.
|
||||
5. Harness embeds a bounded probe and verifies 1024 dimensions.
|
||||
|
||||
Failures are sanitized and fail closed:
|
||||
|
||||
- unavailable Qdrant -> `workspace_not_activatable` before session persistence;
|
||||
- unavailable or missing Ollama model -> `model_unavailable` before session persistence;
|
||||
- collection mismatch -> `semantic_index_incompatible` without recreating or deleting data;
|
||||
- embedding dimension mismatch -> no point write;
|
||||
- partial batch failure -> operation reports failure and remains safe to retry.
|
||||
|
||||
No health response, API response, or diagnostic log exposes DWH credentials or indexed text.
|
||||
|
||||
## Deployment and migration
|
||||
|
||||
The pgvector deployment path is retired:
|
||||
|
||||
- remove local-vector Compose overlays and pgvector bootstrap/migration services;
|
||||
- remove vector PostgreSQL role and password contracts;
|
||||
- remove runtime support for vector REST/SSH and external embedding URLs;
|
||||
- keep only the descriptor parser and migration code needed to recognize legacy workspaces;
|
||||
- update local/server manuals, examples, smoke tests, CI coupling scans, backup instructions, and
|
||||
release gates for four persistent stores plus Qdrant and Ollama volumes.
|
||||
|
||||
Qdrant backup/restore uses collection snapshots or the persistent volume according to the operator
|
||||
manual. Ollama model storage is a cache: it may be backed up for offline recovery but is not an
|
||||
application source of truth.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Base local and server Compose renders include healthy private `qdrant` and `embedding` services.
|
||||
- A clean CPU-only installation downloads the model, creates a workspace collection, and embeds a
|
||||
probe without external vector or embedding configuration.
|
||||
- GPU override uses the same API and persistent model volume.
|
||||
- Schema-v3 workspaces activate; schema-v1/v2 workspaces report `migration_required`.
|
||||
- Two workspaces cannot claim the same Qdrant collection.
|
||||
- Schema, Evidence, and Memory records coexist in one collection and remain filter-isolated.
|
||||
- Existing workflow behavior and persisted session contracts remain unchanged.
|
||||
- Tests reject all active pgvector deployment, external vector binding, and external embedding
|
||||
configuration paths.
|
||||
@@ -1,189 +0,0 @@
|
||||
# Read-only Workspace Repository and Runtime Secrets Design
|
||||
|
||||
**Date:** 2026-08-14
|
||||
**Status:** Approved
|
||||
|
||||
## Purpose
|
||||
|
||||
ThothII consumes workspaces from one administrator-configured Git repository. Workspace authors
|
||||
prepare and publish source outside ThothII. The application fetches, validates, and activates
|
||||
repository revisions, but never edits, commits, pushes, imports, or exports workspace source.
|
||||
|
||||
Runtime credentials are intentionally absent from Git. After a workspace has been read, ThothII
|
||||
derives the required credentials from its connector and authentication choices and lets an
|
||||
authorized user complete them in the web application. The values are encrypted and persisted by
|
||||
the backend; the browser retains neither workspace content nor secrets.
|
||||
|
||||
## Ownership boundaries
|
||||
|
||||
### Workspace source
|
||||
|
||||
The workspace source is an ordinary directory maintained outside the ThothII runtime. It contains
|
||||
the catalog, each `workspace.yaml`, curated evidence, annotations, and other repository-owned
|
||||
content. Authors validate it using source-side tooling and publish it through their normal Git
|
||||
workflow to GitHub, GitLab, Gitea, or another standards-compatible server.
|
||||
|
||||
### ThothII installation
|
||||
|
||||
The installation descriptor selects the Git remote, branch, and one read-only authentication
|
||||
transport. SSH uses a read-only deploy key plus pinned known hosts. HTTPS uses a read-only deploy
|
||||
token and may provide a private CA. Secret values remain outside versioned configuration.
|
||||
|
||||
The installer performs a sanitized `git ls-remote` preflight. Credentials embedded in a remote URL
|
||||
are rejected. The API exposes only a normalized repository identity: host, repository path, branch,
|
||||
transport, active commit, and synchronization state.
|
||||
|
||||
### ThothII runtime
|
||||
|
||||
The local Git checkout, candidate validation area, immutable snapshots, and active state are
|
||||
application-owned. They are read-only from the workspace-management API. A pull fetches a candidate
|
||||
revision, validates the complete repository, and atomically activates it only if valid. A failed
|
||||
candidate never replaces the last valid active revision.
|
||||
|
||||
ThothII never generates or reconciles files back into the checkout and never invokes Git commit or
|
||||
push. Generated operational artifacts live under application data, not in the source repository.
|
||||
|
||||
## Repository synchronization states
|
||||
|
||||
A repository refresh has these states:
|
||||
|
||||
- `syncing`: fetching and validating a candidate revision;
|
||||
- `active`: the candidate passed validation and became the active immutable revision;
|
||||
- `invalid_candidate`: Git succeeded but repository validation failed; the previous revision stays active;
|
||||
- `unavailable`: Git or authentication failed; the previous revision stays active;
|
||||
- `empty`: no valid revision has ever been activated.
|
||||
|
||||
Validation is atomic at repository-commit level. A malformed catalog, descriptor, evidence tree, or
|
||||
cross-file reference rejects the complete candidate revision.
|
||||
|
||||
## Runtime secret model
|
||||
|
||||
### Requirement discovery
|
||||
|
||||
The workspace descriptor contains connector type, authentication method, and non-secret logical
|
||||
configuration. It never contains secret values or host filesystem paths. Connector adapters define
|
||||
the secret fields required by each supported authentication method. For example:
|
||||
|
||||
- PostgreSQL `username_password` requires `username` and `password`;
|
||||
- REST `bearer` requires `api_key`;
|
||||
- SSH tunnel authentication requires the connector password and SSH private key;
|
||||
- Evidence HTTP signed URLs and static S3 credentials contribute their own secret requirements.
|
||||
|
||||
Requirements have stable identifiers scoped by workspace and connector. Labels, descriptions,
|
||||
input kinds, and required/optional status come from trusted application code rather than repository
|
||||
HTML or executable metadata.
|
||||
|
||||
### Persistent encrypted store
|
||||
|
||||
The backend owns a `WorkspaceSecretStore` abstraction. The first implementation is a local encrypted
|
||||
vault in application-managed persistent storage. Each secret is encrypted with authenticated
|
||||
encryption and bound to its installation, workspace, connector, and field identifier as associated
|
||||
data. Plaintext values never appear in Git, API responses, logs, error messages, diagnostics, or
|
||||
browser storage.
|
||||
|
||||
The installation bootstraps one vault key independently from workspace content. Deployment tooling
|
||||
owns its platform-specific provisioning; the workspace schema and GUI never contain filesystem
|
||||
paths. The storage interface allows a future Vault, cloud secret manager, or OS keychain provider
|
||||
without changing workspace descriptors or API consumers.
|
||||
|
||||
When an existing file-oriented harness connector needs a credential, the backend materializes it as
|
||||
a restrictive temporary file in an application-owned runtime directory. Its lifetime is tied to the
|
||||
diagnostic or runtime lease and it is removed on release. Persistent storage contains ciphertext
|
||||
only.
|
||||
|
||||
### Secret API
|
||||
|
||||
For a selected workspace the API returns requirement metadata and status only:
|
||||
|
||||
```json
|
||||
{
|
||||
"workspaceId": "psd-clinical",
|
||||
"state": "configuration_required",
|
||||
"requirements": [
|
||||
{
|
||||
"id": "dwh.password",
|
||||
"connector": "dwh",
|
||||
"label": "Database password",
|
||||
"input": "password",
|
||||
"required": true,
|
||||
"configured": false
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
A write request contains values only for the selected requirement identifiers. The response returns
|
||||
status, never values. A delete operation forgets a configured value. Authorization is deliberately
|
||||
deferred; the current authenticated application user may manage runtime workspace secrets.
|
||||
|
||||
Workspace readiness is derived as follows:
|
||||
|
||||
- `invalid`: repository structure or descriptor is invalid;
|
||||
- `configuration_required`: structurally valid but required runtime values are missing;
|
||||
- `ready`: required values exist but connectivity has not yet passed or is stale;
|
||||
- `verified`: the most recent connector diagnostic passed for the active revision and current secret generation.
|
||||
|
||||
Changing or deleting a secret invalidates the previous diagnostic result.
|
||||
|
||||
## Browser behavior
|
||||
|
||||
Workspace management is a two-level read-only interface occupying at least 60 percent of viewport
|
||||
width and height.
|
||||
|
||||
Level 1 explains the source/runtime separation and displays:
|
||||
|
||||
- normalized repository host and path;
|
||||
- configured branch and read-only transport;
|
||||
- active revision and last synchronization result;
|
||||
- `Update workspace repository`, which fetches, validates, and conditionally activates a revision;
|
||||
- the workspace list, with selection required for workspace-specific actions.
|
||||
|
||||
There is no Import bundle, Export bundle, Create, Edit, Delete, Publish, or conflict-resolution
|
||||
operation. There are no browser-persisted workspace drafts or preferences.
|
||||
|
||||
Level 2 for the selected workspace explains and displays:
|
||||
|
||||
- immutable source identity and validation result;
|
||||
- required runtime configuration grouped by connector;
|
||||
- secret-entry controls whose values are write-only;
|
||||
- `Save secrets`, `Forget` per configured value, and `Test workspace connection`;
|
||||
- clear consequences for each button and a reminder that source changes must be committed and pushed
|
||||
by an author outside ThothII before repository update.
|
||||
|
||||
The browser keeps form values only in component memory and clears them after submission or dialog
|
||||
close. It never receives saved secret values.
|
||||
|
||||
## Compatibility and migration
|
||||
|
||||
Existing Git author settings, publish endpoints, bundle endpoints, generated-document
|
||||
reconciliation, bootstrap catalog slots, and browser draft storage are removed. Existing environment
|
||||
bindings may be read during a bounded migration period only to seed non-secret connector values;
|
||||
secret file paths are not part of the new public workspace contract.
|
||||
|
||||
Session manifests continue to pin an immutable validated workspace revision. An already running
|
||||
session keeps its acquired runtime lease; new or resumed work resolves the current encrypted secret
|
||||
generation and fails closed when required credentials are unavailable.
|
||||
|
||||
## Failure handling and security
|
||||
|
||||
- Repository and vault errors use stable sanitized codes and never echo remotes with user info,
|
||||
credential paths, secret identifiers that are not safe to disclose, or secret values.
|
||||
- Vault writes are atomic and authenticated; corrupted ciphertext fails closed.
|
||||
- Secret comparison uses no read API. Updating a secret is always a blind replacement.
|
||||
- The backend applies request-size and field-count limits and rejects unknown requirement IDs.
|
||||
- Temporary plaintext files use restrictive permissions, trusted directories, no-follow opens, and
|
||||
deterministic cleanup.
|
||||
- Git credentials are installation-only, read-only, and never sent to the frontend.
|
||||
|
||||
## Verification
|
||||
|
||||
Backend tests cover repository read-only behavior, atomic candidate activation, remote sanitization,
|
||||
vault encryption and corruption, requirement discovery, blind secret writes/deletes, materialization
|
||||
cleanup, readiness transitions, and absence of publish/bundle routes.
|
||||
|
||||
Frontend tests cover the two-level explanation, viewport dimensions, repository identity, selection
|
||||
gating, dynamic secret forms, write-only behavior, status changes, and absence of local-storage,
|
||||
import, export, editing, and publishing controls.
|
||||
|
||||
Deployment and CLI tests cover required remote/branch configuration, one read-only Git transport,
|
||||
sanitized remote preflight, vault-key provisioning, and removal of Git author/write configuration.
|
||||
@@ -1,408 +0,0 @@
|
||||
# ThothII Authentication Acceptance and PSD Deployment Plan
|
||||
|
||||
> **For agentic workers:** execute one task at a time, record its completion criteria, and stop at every documented approval boundary.
|
||||
|
||||
**Goal:** Validate local and OIDC authentication on macOS, deploy the exact feat/thoth-auth candidate to the Aritmolab/PSD server before merging it into main, and complete end-to-end acceptance with remote Authentik.
|
||||
|
||||
**Architecture:** Test the candidate first as a standalone local installation. Then install the same immutable Git revision on the existing PSD installation with the installation-aware tht lifecycle, leaving main untouched. Authentik provides OIDC login and a mandatory direct groups claim; ThothII maps exact external groups to roles and validates mapped groups through the Authentik catalog API.
|
||||
|
||||
**Tech Stack:** macOS, Docker Desktop, Docker Compose, native host `tht` plus Python workflow `tht`, local Argon2id authentication, generic OIDC Authorization Code + PKCE, Authentik, PSD workspace registry, reverse proxy/TLS.
|
||||
|
||||
---
|
||||
|
||||
## Scope and release rules
|
||||
|
||||
Do not merge feat/thoth-auth into main until every mandatory gate in Task 9 is PASS and the PSD owner accepts the evidence.
|
||||
|
||||
Capture one candidate revision and reuse it everywhere:
|
||||
|
||||
```bash
|
||||
export CANDIDATE_SHA="$(git rev-parse HEAD)"
|
||||
git fetch origin feat/thoth-auth
|
||||
test "$CANDIDATE_SHA" = "$(git rev-parse origin/feat/thoth-auth)"
|
||||
git show -s --format='%H%n%P%n%s' "$CANDIDATE_SHA"
|
||||
git status --short --untracked-files=all
|
||||
```
|
||||
|
||||
Never deploy a moving branch name without checking its resolved SHA. Never put passwords, OIDC client secrets, Authentik API tokens, cookies, authorization headers, raw ID tokens, or password hashes in Git, shell history, screenshots, logs, or evidence.
|
||||
|
||||
Use protected operator values for <PUBLIC_URL>, <OIDC_ISSUER>, <AUTHENTIK_BASE_URL>, <OIDC_CLIENT_ID>, <THT_BIN>, <INSTALLATION>, <WORKSPACE_ID>, and <OLD_SHA>.
|
||||
|
||||
The Authentik contract is mandatory: a direct non-empty JSON array claim named groups; exact groups TOT Users and TOT Admin; mappings TOT Users -> user and TOT Admin -> admin; and a separate group-view-only API service account exposed only as THT_AUTHENTIK_API_TOKEN. Extra upstream groups are valid and silently ignored.
|
||||
|
||||
## Task 0: Freeze the candidate and collect approvals
|
||||
|
||||
**Files:** None; record results in the acceptance report in Task 9.
|
||||
|
||||
Run from the candidate worktree:
|
||||
|
||||
```bash
|
||||
git diff --check
|
||||
go test ./... -count=1
|
||||
go test -race ./...
|
||||
go vet ./...
|
||||
go build ./...
|
||||
```
|
||||
|
||||
Expected: all commands pass, the candidate is pushed, and existing evidence is bound to the same SHA. A historical result from another revision is not evidence for this run.
|
||||
|
||||
Before touching PSD, obtain the maintenance window, server access, public URL, Authentik provider details, protected secret locations, test identities for ordinary/admin/unmapped users, and permission to test PSD DWH/Evidence connections.
|
||||
|
||||
## Task 1: Prepare and start the local macOS installation
|
||||
|
||||
**Files:**
|
||||
|
||||
- Read: docs/install/local.md
|
||||
- Read: docs/install/authentication-local.md
|
||||
- Read: docs/testing/authentication-manual-acceptance.md
|
||||
- Use: an untracked local installation descriptor and protected secret/password files
|
||||
|
||||
**Step 1: Verify prerequisites**
|
||||
|
||||
```bash
|
||||
docker version
|
||||
docker compose version
|
||||
bash scripts/verify-line-endings.sh
|
||||
```
|
||||
|
||||
Expected: Docker Desktop and Compose are available and line-ending validation passes.
|
||||
|
||||
**Step 2: Build and configure**
|
||||
|
||||
```bash
|
||||
bash scripts/build-local.sh
|
||||
bash scripts/build-tht.sh
|
||||
tht setup --profile local
|
||||
```
|
||||
|
||||
For an existing installation, do not overwrite data; run tht --installation <local-installation.yaml> update --check-only instead of setup.
|
||||
|
||||
**Step 3: Start and inspect**
|
||||
|
||||
```bash
|
||||
tht --installation <local-installation.yaml> start --build
|
||||
tht --installation <local-installation.yaml> status
|
||||
tht --installation <local-installation.yaml> doctor --json
|
||||
curl --fail http://127.0.0.1:8080/health
|
||||
curl --fail http://127.0.0.1:8787/health
|
||||
```
|
||||
|
||||
Expected: core, frontend, qdrant, embedding, and the completed model initializer are healthy; doctor includes authentication after configuration and before services.
|
||||
|
||||
## Task 2: Configure and test local login
|
||||
|
||||
**Files:**
|
||||
|
||||
- Read: docs/install/authentication-local.md
|
||||
- Modify only protected installation state through tht auth configure and tht auth user
|
||||
|
||||
**Step 1: Bootstrap the administrator**
|
||||
|
||||
```bash
|
||||
tht --installation <local-installation.yaml> auth configure \
|
||||
--mode local --public-url http://127.0.0.1:8080 \
|
||||
--admin-user <local-admin> --admin-display-name <display-name> \
|
||||
--password-file <protected-password-file>
|
||||
```
|
||||
|
||||
Remove the temporary password file immediately. Expected: non-secret auth.yaml is created and the user store contains Argon2id hashes, never plaintext passwords.
|
||||
|
||||
**Step 2: Add and inspect a normal user**
|
||||
|
||||
```bash
|
||||
tht --installation <local-installation.yaml> auth user add <local-user> --role user --display-name <display-name> --password-file <protected-password-file>
|
||||
tht --installation <local-installation.yaml> auth status --json
|
||||
tht --installation <local-installation.yaml> auth check --json
|
||||
```
|
||||
|
||||
Expected: pristine redacted JSON and no credential, hash, or session secret in output.
|
||||
|
||||
**Step 3: Test browser authorization**
|
||||
|
||||
At http://127.0.0.1:8080, in a private browser profile:
|
||||
|
||||
1. Verify unauthenticated access reaches login and protected routes are denied.
|
||||
2. Log in as the normal user and verify application/session routes work.
|
||||
3. Verify Pi Management and other admin-only operations return HTTP 403 or are not exposed.
|
||||
4. Log out and verify the session is invalidated.
|
||||
5. Log in as the administrator and verify admin-only routes work.
|
||||
|
||||
Expected: ordinary users authenticate without receiving admin permissions; administrators receive the configured admin permission set.
|
||||
|
||||
**Step 4: Test account failure paths**
|
||||
|
||||
Use auth user disable, enable, set-password, and logout-all user --yes on the test user. Test a wrong password and refresh the old browser session after logout-all.
|
||||
|
||||
Expected: generic safe failures, disabled login rejection, re-enabled login success, and forced reauthentication. The last enabled administrator cannot be disabled or demoted.
|
||||
|
||||
## Task 3: Test remembered sessions and local recovery
|
||||
|
||||
**Files:**
|
||||
|
||||
- Read: docs/architecture/authentication.md, Browser sessions
|
||||
- Read: docs/install/authentication-local.md, Session behavior and recovery
|
||||
|
||||
**Step 1: Test browser restart**
|
||||
|
||||
Log in as the normal user with Remember me, close the browser completely, reopen it, and revisit the application.
|
||||
|
||||
Expected: the session survives within the 7-day idle / 30-day absolute limits. Do not record the cookie.
|
||||
|
||||
**Step 2: Test ThothII restart**
|
||||
|
||||
```bash
|
||||
tht --installation <local-installation.yaml> stop
|
||||
tht --installation <local-installation.yaml> start
|
||||
```
|
||||
|
||||
Expected: the remembered session remains valid after backend restart.
|
||||
|
||||
**Step 3: Test invalidation**
|
||||
|
||||
Change the test user password or role, and separately run auth user logout-all user --yes. Refresh after each operation.
|
||||
|
||||
Expected: affected sessions are rejected and reauthentication is required; configuration revision changes invalidate all sessions.
|
||||
|
||||
Go/no-go: do not proceed to PSD if local login, role separation, logout, or remembered-session behavior fails.
|
||||
|
||||
## Task 4: Snapshot the current PSD installation
|
||||
|
||||
**Files:**
|
||||
|
||||
- Read: docs/install/server.md
|
||||
- Read: docs/install/server-workspace-registry.md
|
||||
- Use: protected server operator and backup locations
|
||||
|
||||
**Step 1: Capture live state**
|
||||
|
||||
```bash
|
||||
THT_BIN=<THT_BIN>
|
||||
INSTALLATION=<INSTALLATION>
|
||||
"$THT_BIN" --installation "$INSTALLATION" status
|
||||
"$THT_BIN" --installation "$INSTALLATION" doctor
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi status
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi doctor
|
||||
git -C /srv/thothii/source/ThothII status --short --untracked-files=all
|
||||
git -C /srv/thothii/source/ThothII rev-parse HEAD
|
||||
```
|
||||
|
||||
Save the live SHA as <OLD_SHA> and capture image identities, workspace registry status, and maintenance/recovery state. Stop if the checkout is dirty or recovery is pending.
|
||||
|
||||
**Step 2: Drain and back up**
|
||||
|
||||
Announce maintenance, close the reverse proxy or show its maintenance page, drain active work, and stop through tht. Create the protected, checksummed backup specified in docs/install/server.md, including runtime trees and PSD PostgreSQL/session data where applicable. Back up credentials separately. Never run docker compose down --volumes.
|
||||
|
||||
**Step 3: Check preconditions**
|
||||
|
||||
```bash
|
||||
git -C /srv/thothii/source/ThothII config --local core.autocrlf false
|
||||
bash /srv/thothii/source/ThothII/scripts/verify-line-endings.sh
|
||||
"$THT_BIN" --installation "$INSTALLATION" update --check-only
|
||||
```
|
||||
|
||||
Expected: descriptor, protected secrets, Pi-state mount, workspace repository binding, and Compose render remain valid before source changes.
|
||||
|
||||
## Task 5: Deploy the feature revision to PSD without merging main
|
||||
|
||||
**Files:**
|
||||
|
||||
- Server source checkout: /srv/thothii/source/ThothII
|
||||
- Server operator binary: protected THT_BIN path
|
||||
- Server installation descriptor and secret files: unchanged paths unless a reviewed auth update is required
|
||||
|
||||
**Step 1: Select the exact candidate**
|
||||
|
||||
```bash
|
||||
git -C /srv/thothii/source/ThothII fetch origin feat/thoth-auth
|
||||
git -C /srv/thothii/source/ThothII switch --detach <CANDIDATE_SHA>
|
||||
test "$(git -C /srv/thothii/source/ThothII rev-parse HEAD)" = "<CANDIDATE_SHA>"
|
||||
git -C /srv/thothii/source/ThothII status --short --untracked-files=all
|
||||
```
|
||||
|
||||
Do not merge or rebase main. The running installation is intentionally based on the detached feature revision until acceptance completes.
|
||||
|
||||
**Step 2: Build candidate artifacts**
|
||||
|
||||
```bash
|
||||
cd /srv/thothii/source/ThothII
|
||||
bash scripts/build-local.sh
|
||||
THT_THT_OUTPUT_DIRECTORY=/srv/thothii/operator/build-output bash scripts/build-tht.sh
|
||||
```
|
||||
|
||||
Install the architecture-appropriate candidate tht only after its build succeeds. Keep the old operator binary recoverable.
|
||||
|
||||
**Step 3: Start and verify the candidate**
|
||||
|
||||
```bash
|
||||
"$THT_BIN" --installation "$INSTALLATION" update --check-only
|
||||
"$THT_BIN" --installation "$INSTALLATION" start --build
|
||||
"$THT_BIN" --installation "$INSTALLATION" status
|
||||
"$THT_BIN" --installation "$INSTALLATION" doctor --json
|
||||
curl --fail http://127.0.0.1:8080/health
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi doctor
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi test
|
||||
```
|
||||
|
||||
Expected: candidate frontend/core and internal services are healthy, no data volume was replaced, and the candidate SHA is recorded. Liveness alone is not release approval.
|
||||
|
||||
## Task 6: Configure and validate remote Authentik
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify protected server authentication state through tht auth configure
|
||||
- Modify protected secret entries THT_OIDC_CLIENT_SECRET and THT_AUTHENTIK_API_TOKEN
|
||||
- Read: docs/install/authentik.md and docs/install/authentication-oidc.md
|
||||
|
||||
**Step 1: Verify Authentik**
|
||||
|
||||
Verify the OAuth2/OIDC callback exactly <PUBLIC_URL>/api/auth/oidc/callback, scopes openid/profile/email, direct groups array mapping, exact groups TOT Users and TOT Admin, and a separate group-view-only catalog service account. Inspect a disposable identity without copying its token.
|
||||
|
||||
**Step 2: Configure the ThothII mapping**
|
||||
|
||||
```bash
|
||||
tht --installation "$INSTALLATION" auth configure \
|
||||
--mode oidc --public-url <PUBLIC_URL> \
|
||||
--issuer <OIDC_ISSUER> --client-id <OIDC_CLIENT_ID> \
|
||||
--authentik-base-url <AUTHENTIK_BASE_URL> \
|
||||
--user-group 'TOT Users' --admin-group 'TOT Admin'
|
||||
```
|
||||
|
||||
Secrets are read from protected files, never command-line arguments. Confirm the non-secret mapping is:
|
||||
|
||||
```yaml
|
||||
authorization:
|
||||
groupRoles:
|
||||
TOT Users: [user]
|
||||
TOT Admin: [admin]
|
||||
```
|
||||
|
||||
**Step 3: Run static and live checks**
|
||||
|
||||
```bash
|
||||
tht --installation "$INSTALLATION" auth status --json
|
||||
tht --installation "$INSTALLATION" auth check --json
|
||||
tht --installation "$INSTALLATION" auth check --interactive
|
||||
tht --installation "$INSTALLATION" doctor --json
|
||||
```
|
||||
|
||||
Expected: configuration, discovery, issuer, JWKS, client-secret access, catalog access, and exact existence of every mapped group pass. Doctor lists authentication after configuration and before services. No output contains credentials or bearer tokens.
|
||||
|
||||
A missing mapped group must fail with redacted oidc_mapped_group_missing. Unmapped groups produce neither error nor warning. Missing, indirect, malformed, or overage-style groups claims fail closed.
|
||||
|
||||
**Step 4: Reload if required**
|
||||
|
||||
If configuration requires process reload:
|
||||
|
||||
```bash
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi restart --yes --drain
|
||||
```
|
||||
|
||||
Repeat authentication, doctor, and health checks. Do not substitute raw Compose commands.
|
||||
|
||||
## Task 7: Test PSD browser login and authorization
|
||||
|
||||
**Files:**
|
||||
|
||||
- Read: docs/testing/authentication-manual-acceptance.md
|
||||
- Evidence: redacted report from Task 9
|
||||
|
||||
**Step 1: Ordinary user**
|
||||
|
||||
In a private profile, authenticate with an identity in TOT Users but not TOT Admin. Verify callback success, application/session routes, denial of Pi Management/admin operations, opaque HttpOnly ThothII cookie, no bearer token in Web Storage, and logout invalidation.
|
||||
|
||||
**Step 2: Administrator**
|
||||
|
||||
Authenticate with TOT Admin. Verify Pi Management and allowed workspace-management operations. Access must derive from the exact mapped group, not a client-supplied header or browser-local flag.
|
||||
|
||||
**Step 3: Unmapped and malformed groups**
|
||||
|
||||
Authenticate with a valid token containing no mapped group. Expected: login may complete, but protected operations return 403 with no warning. Use a disposable provider mapping that omits or corrupts groups; expected: generic HTTP 401 oidc_callback_failed, with no internal claim details exposed.
|
||||
|
||||
**Step 4: Provider outage/group drift**
|
||||
|
||||
During a controlled window, make discovery/JWKS unavailable or rename a mapped group, run the CLI check, and restore it immediately. Expected: redacted fail-closed diagnostics followed by a successful check after restoration. Do not leave production broken.
|
||||
|
||||
## Task 8: Test PSD workspace validation and real connections
|
||||
|
||||
**Files:**
|
||||
|
||||
- Read: docs/install/server-workspace-registry.md
|
||||
- Use: authenticated PSD browser sessions
|
||||
|
||||
**Step 1: Validate the workspace and authentication from the host CLI**
|
||||
|
||||
```bash
|
||||
"$THT_BIN" --installation "$INSTALLATION" \
|
||||
workspace inspect --workspace "$WORKSPACE_ID" --json
|
||||
"$THT_BIN" --installation "$INSTALLATION" auth check --json
|
||||
```
|
||||
|
||||
Expected: the workspace registry is ready, authentication readiness passes, and output is redacted while identifying the active workspace revision.
|
||||
|
||||
**Step 2: Verify the application boundary**
|
||||
|
||||
Open the configured public URL, authenticate with the approved identity, and verify that the application reaches the selected workspace without unexpected `401`/`403` responses. Keep DWH/Evidence connection tests read-only and use only the existing approved smoke question.
|
||||
|
||||
**Step 3: Test ordinary-user authorization**
|
||||
|
||||
Log in as TOT Users. Confirm inspection follows ordinary permissions while validation, secret mutation, and connection tests remain unavailable unless explicitly granted.
|
||||
|
||||
**Step 4: Run a harmless end-to-end smoke**
|
||||
|
||||
As an authorized PSD user, create or resume one harmless known-good session:
|
||||
|
||||
```text
|
||||
browser login -> same-origin API -> workspace readiness -> Pi/core -> result -> logout
|
||||
```
|
||||
|
||||
Do not run mutating production queries. Preserve only a session ID and redacted outcome if approved.
|
||||
|
||||
## Task 9: Close acceptance, rollback if needed, and decide merge readiness
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: docs/testing/evidence/2026-08-18-thothii-authentication-psd-acceptance.md or the approved external evidence location
|
||||
- Read: docs/install/server.md and docs/contracts/tht-pi.md
|
||||
|
||||
**Step 1: Mandatory gates**
|
||||
|
||||
| Gate | Required evidence |
|
||||
|---|---|
|
||||
| Candidate identity | Local and server SHA exactly match pushed feat/thoth-auth |
|
||||
| Local startup | macOS Compose, doctor, health, and Pi smoke pass |
|
||||
| Local auth | Bootstrap, ordinary/admin roles, logout, bad password, disable/enable, logout-all pass |
|
||||
| Local session | Remembered session survives browser and ThothII restart; revisions invalidate it |
|
||||
| Server safety | Old SHA/image/status captured; backup checksummed; maintenance/drain completed |
|
||||
| Candidate deploy | Server candidate status/doctor/health pass |
|
||||
| Authentik | Direct groups claim, issuer/JWKS, secrets, catalog, and mapped groups pass |
|
||||
| OIDC authorization | ordinary, admin, unmapped, malformed, logout, and outage cases pass |
|
||||
| Workspace integration | `tht workspace inspect` and `tht auth check` pass; the authenticated application reaches the selected workspace |
|
||||
| PSD smoke | One harmless known-good session completes |
|
||||
| Hygiene | No secrets, tokens, cookies, hashes, or raw claims in evidence |
|
||||
|
||||
**Step 2: Write the redacted report**
|
||||
|
||||
Include candidate SHA, old SHA, timestamps, commands, browser cases, redacted diagnostic/HTTP codes, Authentik issuer/client/group names, workspace ID/revision, backup/checksum location, rollback decision, and unrelated CI failures. Never include secret values, raw tokens, cookies, or hashes.
|
||||
|
||||
**Step 3: Roll back a failed candidate**
|
||||
|
||||
1. Keep the proxy closed and preserve .tht/<installation-id>/ recovery state.
|
||||
2. Do not use tht pi rollback as the whole-application rollback; it addresses only Pi lifecycle images.
|
||||
3. Stop with tht.
|
||||
4. Return the source checkout to <OLD_SHA>, rebuild old application/operator artifacts, and start through the same descriptor.
|
||||
5. Run update --check-only, status, doctor, health, Pi smoke, workspace diagnostics, and one harmless session.
|
||||
6. For ambiguous recovery, leave maintenance active and follow pi maintenance status / pi maintenance recover --yes. Never delete volumes, selectors, or recovery files to force progress.
|
||||
|
||||
Expected: the previous application serves again with prior data and workspace state intact. Record the failure and do not merge.
|
||||
|
||||
**Step 4: Reopen traffic**
|
||||
|
||||
After every gate passes, restore the reverse proxy, repeat one unauthenticated redirect and one authorized public login, and confirm only the proxy is externally reachable.
|
||||
|
||||
**Step 5: Merge decision**
|
||||
|
||||
Merge only after PSD owner acceptance, exact-SHA evidence, no unresolved auth/workspace/provider/deployment gate, and an accepted rollback path. If the merge creates a new commit, repeat Tasks 0, 5, 6, and 7 against the merge SHA.
|
||||
|
||||
## Handoff checklist
|
||||
|
||||
Deliver the redacted report, local result/SHA, PSD candidate SHA/images, Authentik provider and group mapping confirmation, workspace validation/connection results, backup/rollback status, and an explicit READY TO MERGE or NOT READY TO MERGE decision.
|
||||
@@ -1,370 +0,0 @@
|
||||
# PSD Server Deployment Program — Design
|
||||
|
||||
**Date:** 2026-08-20
|
||||
|
||||
**Status:** Approved by the owner
|
||||
|
||||
**Owner sequencing amendment (2026-08-21):** the Mac `rest_api` acceptance and revocation of
|
||||
`legacy-shared` are deferred to one mandatory pre-Project-B gate. This permits the bounded survey
|
||||
and static, non-mutating Project A private preparation to proceed without changing the Mac. It does not
|
||||
authorize stopping the legacy stack, starting the new stack, opening ingress, or beginning Project B.
|
||||
|
||||
**Design-time application baseline:** `main` at `5c0dc8c` (execution must freeze and record the
|
||||
then-current `origin/main` SHA)
|
||||
|
||||
**Design-time workspace baseline:** `tht-workspace-psd/main` at `bfbabf9` (execution must freeze and
|
||||
record the then-current remote SHA)
|
||||
|
||||
## Purpose
|
||||
|
||||
Replace the unused legacy ThothII installation on the PSD server with the current application,
|
||||
recovering only useful configuration and rebuilding runtime state from canonical sources. Complete
|
||||
the work as two independently accepted projects:
|
||||
|
||||
1. deploy and prove ThothII with local authentication, a direct read-only PSD DWH connection,
|
||||
internal Qdrant, and internal Ollama;
|
||||
2. only after Project A passes, integrate the accepted installation with the server's Authentik,
|
||||
Nginx, load balancer, and the existing Aritmolab sidebar link.
|
||||
|
||||
The program must be executable by the Sol LLM from a terminal local to the server. It must test
|
||||
each step, retain redacted evidence, stop on unsafe uncertainty, and include a separate human
|
||||
manual-test document for each project.
|
||||
|
||||
## Binding decisions
|
||||
|
||||
- The legacy application may be unavailable for days. Service continuity is not a goal.
|
||||
- The new source clone is prepared beside the old source directory. The old and new stacks are not
|
||||
kept running simultaneously: inventory and backup happen first, the old stack is stopped, and
|
||||
only then is the new stack started.
|
||||
- Keep the old directory, configuration, containers, images, and data only as a temporary recovery
|
||||
boundary until the real Aritmolab journey passes Project B. They are disposable after Project B
|
||||
automated, human, and owner PASS. Do not migrate legacy application sessions, Qdrant data,
|
||||
Ollama caches, or derived indexes.
|
||||
- Do not create a host `thothii` user or group. Keep UID/GID `10001:10001` as the image's unmapped
|
||||
numeric runtime identity and confine numeric ownership to the new installation's writable bind
|
||||
trees. Stop if either number becomes mapped to a host account before installation.
|
||||
- Recover configuration only: endpoints, non-secret policy, provider/model selection, relevant
|
||||
paths, and references to protected credentials. Never copy an old setting without validating it
|
||||
against the current contract.
|
||||
- Use the canonical ThothII Compose distribution and reviewed installation-local overrides. Do not
|
||||
modify an unrelated global Compose project or blindly adapt the legacy Compose file.
|
||||
- Lifecycle operations use the installation-aware native `tht` CLI, not raw Compose commands.
|
||||
- Never use `docker compose down --volumes`, global prune operations, broad recursive deletion, or
|
||||
secret-bearing command arguments.
|
||||
|
||||
## Repository and workspace model
|
||||
|
||||
There are two Git sources with different responsibilities:
|
||||
|
||||
- `ThothII` contains application code and non-secret deployment examples.
|
||||
- `tht-workspace-psd` contains the shared, credential-free PSD workspace source.
|
||||
|
||||
The workspace repository contains one logical workspace, `psd-clinical`. Its schema-v3 descriptor
|
||||
will declare both supported DWH transports:
|
||||
|
||||
```yaml
|
||||
supported_transports: [rest_api, postgres_direct]
|
||||
```
|
||||
|
||||
The Mac installation continues to select `rest_api`. The PSD server selects `postgres_direct`.
|
||||
Endpoint values, user names, passwords, secret-file paths, machine paths, and Git credentials stay
|
||||
in each installation's protected bindings and never enter the workspace repository. Evidence,
|
||||
curated annotations, language, model policy, and semantic-index contract remain shared.
|
||||
|
||||
Each installation owns its own Qdrant collection contents, Ollama model cache, preprocessing state,
|
||||
and sessions even though both consume the same reviewed workspace commit.
|
||||
|
||||
## Program structure
|
||||
|
||||
The program consists of one non-mutating common survey followed by two independently gated
|
||||
projects:
|
||||
|
||||
```text
|
||||
Common Survey PASS for Project A private scope
|
||||
-> Project A automated PASS
|
||||
-> Project A human PASS
|
||||
-> Mac REST acceptance
|
||||
-> 48-hour dual-key observation covering two 03:00 ETL cycles
|
||||
-> legacy-shared revocation and negative proof
|
||||
-> full pre-Project-B survey PASS
|
||||
-> explicit Project B authorization
|
||||
-> Project B automated PASS
|
||||
-> Project B human PASS
|
||||
-> final cutover acceptance
|
||||
```
|
||||
|
||||
Project B must not start from a partial or assumed Project A result.
|
||||
|
||||
## Common Survey
|
||||
|
||||
The survey is a prerequisite, not a third implementation project. It runs before any server
|
||||
mutation and produces a redacted report, topology map, change-scope inventory, unknowns list, and
|
||||
GO/NO-GO decision.
|
||||
|
||||
Sol inventories from the server-local terminal:
|
||||
|
||||
- operating system, architecture, Docker and Compose versions, available CPU/RAM/disk, and clock;
|
||||
- legacy ThothII source, SHA, dirty state, images, containers, networks, ports, volumes, mounts,
|
||||
health, installation state, and recovery state;
|
||||
- effective Compose rendering and ownership of every relevant file;
|
||||
- DWH database/schema, direct listener, TLS, read-only role, and reachability from containers;
|
||||
- Supabase/PostgreSQL topology, schema conventions, exposed PostgREST schemas, migration policy,
|
||||
backup mechanism, and suitable role boundaries;
|
||||
- Pi provider/model policy, credential references, external LLM reachability, and current versions;
|
||||
- Nginx effective configuration, ThothII virtual host/location, upstream, forwarded headers, SSE
|
||||
settings, certificate metadata, certificate generation/renewal, and rollback files;
|
||||
- load-balancer routes, health checks, allowlist capability, TLS boundary, and configuration owner;
|
||||
- Aritmolab deployment, networks, homepage, sidebar source, current ThothII destination, and release
|
||||
procedure;
|
||||
- Authentik version, deployment, current Aritmolab integration, provider conventions, group
|
||||
conventions, backup/export procedure, API access, and credential locations;
|
||||
- public DNS/origin that the final sidebar link must preserve;
|
||||
- protected files by path, ownership, mode, and readability only, without printing their contents.
|
||||
|
||||
The survey may use hashes, metadata, redacted renders, and permission checks. It must not emit
|
||||
passwords, bearer tokens, API keys, cookies, OIDC client secrets, private keys, password hashes, or
|
||||
raw identity tokens. If credential discovery fails, the owner may help locate the existing
|
||||
Authentik credentials.
|
||||
|
||||
## Project A — Standalone Server Acceptance
|
||||
|
||||
### Runtime architecture
|
||||
|
||||
Project A installs a clean five-service stack:
|
||||
|
||||
```text
|
||||
operator terminal or local headless browser
|
||||
-> frontend
|
||||
-> core + Pi + workflow harness
|
||||
-> direct read-only PSD PostgreSQL DWH
|
||||
-> internal Qdrant
|
||||
-> internal Ollama (qwen3-embedding:0.6b, 1024 dimensions)
|
||||
-> local filesystem work-session storage
|
||||
```
|
||||
|
||||
The stack uses local authentication. It must not be publicly reachable. Core, Qdrant, and Ollama
|
||||
remain private; the frontend binds to loopback unless the optional restricted test route below is
|
||||
proved safe.
|
||||
|
||||
### Preparation and cutover
|
||||
|
||||
1. Freeze exact application and workspace SHAs and require clean source trees.
|
||||
2. Publish and validate the multi-transport `psd-clinical` descriptor through the curator workflow.
|
||||
3. Preserve the Mac REST binding unchanged; its live acceptance is deferred to the mandatory
|
||||
pre-Project-B gate.
|
||||
4. Prepare the new source clone and protected operator/runtime directories beside the old source.
|
||||
5. Extract only approved configuration facts from the legacy installation.
|
||||
6. Record and verify the exact restart recipe for the legacy stack. No data backup is required
|
||||
because the owner declared legacy sessions and configuration disposable; the still-present
|
||||
containers, images, source, and data are the temporary rollback boundary.
|
||||
7. Stop the legacy stack without deleting its source, configuration, images, or data.
|
||||
8. Build/install the current native `tht` and application images from the frozen source.
|
||||
9. Configure local authentication, direct DWH bindings, Pi/provider access, and the Git workspace.
|
||||
10. Start through `tht`, then validate health, authentication, workspace, workflow, and Pi.
|
||||
11. Rebuild schema/Evidence preprocessing, Qdrant contents, and the Ollama model cache from
|
||||
canonical sources. Prove the complete preprocessing rerun is idempotent.
|
||||
|
||||
### Optional restricted test route
|
||||
|
||||
If and only if the survey proves that the load balancer can enforce a test-operator allowlist
|
||||
before a request reaches ThothII, Project A may use a temporary private hostname to exercise the
|
||||
same network path as production:
|
||||
|
||||
```text
|
||||
authorized operator -> allowlisted load-balancer route -> Nginx -> frontend -> core/local auth
|
||||
```
|
||||
|
||||
The route has a distinct hostname, no Aritmolab sidebar link, a certificate created by the existing
|
||||
managed mechanism, correct forwarded-origin and SSE behavior, and a negative test from an
|
||||
unauthorized source. Its local-auth `publicUrl` matches the private test origin. It is not a public
|
||||
production route and must be removed after Project B.
|
||||
|
||||
If isolation cannot be demonstrated, Sol must not approximate it or misdeclare
|
||||
`THOTH_PUBLIC_EXPOSURE=false`; tests run against loopback from the local terminal instead.
|
||||
|
||||
### Acceptance
|
||||
|
||||
Project A requires:
|
||||
|
||||
- exact source identity and reproducible build evidence;
|
||||
- healthy frontend, core, Qdrant, embedding, and completed model initializer;
|
||||
- local authentication checks including admin/user separation, wrong password, logout,
|
||||
disable/enable, invalidation, and restart persistence;
|
||||
- direct DWH connectivity with a demonstrably read-only runtime identity;
|
||||
- active `psd-clinical` at the expected Git revision;
|
||||
- compatible Qdrant collection/index contract and correct Ollama model/dimensions;
|
||||
- complete and idempotent DWH, schema, annotation, and Evidence preprocessing;
|
||||
- one harmless real PSD work session completed through F1-F8, ending in read-only validated SQL;
|
||||
- persisted manifest, artifacts, reviewer decisions, and final SQL inspection;
|
||||
- optional private-route positive and negative isolation evidence when that route is used;
|
||||
- completed human manual-test report with an explicit PASS.
|
||||
|
||||
The Mac row may be recorded only as `DEFERRED_PRE_PROJECT_B` under the dated owner amendment. It is
|
||||
not part of the private server acceptance, but it must become PASS before Project B starts.
|
||||
|
||||
Project A does not modify the production Aritmolab sidebar, production public route, or Authentik.
|
||||
|
||||
## Project B — Authentik and Aritmolab Integration
|
||||
|
||||
Project B begins only from the frozen, accepted Project A source, images, workspace revision, and
|
||||
PASS report. It also requires the deferred Mac REST acceptance, the full 48-hour observation
|
||||
window (including two scheduled 03:00 ETL cycles), revocation of `legacy-shared`, proof that the
|
||||
legacy credential receives `401`, and the full pre-Project-B survey gate.
|
||||
|
||||
### Final request and data flow
|
||||
|
||||
```text
|
||||
user
|
||||
-> aritmolab.policlinicosandonato.com
|
||||
-> Aritmolab homepage/sidebar
|
||||
-> load balancer
|
||||
-> Nginx/TLS
|
||||
-> ThothII frontend and same-origin /api
|
||||
-> ThothII OIDC Authorization Code + PKCE with Authentik
|
||||
-> PostgreSQL thoth_sessions schema for owned work sessions
|
||||
-> direct read-only datawarehouse schema for clinical queries
|
||||
-> internal Qdrant and Ollama
|
||||
```
|
||||
|
||||
Nginx terminates/proxies according to the observed deployment but does not add a second
|
||||
`auth_request` in front of ThothII. ThothII performs generic OIDC directly. Nginx preserves the
|
||||
public host and HTTPS scheme, forwards the callback path unchanged, and supports SSE without
|
||||
buffering or premature timeouts.
|
||||
|
||||
### Authentik configuration
|
||||
|
||||
Before mutation, export or back up the relevant Authentik configuration. Locate existing protected
|
||||
administrative/API credentials without exposing them. Create or adapt:
|
||||
|
||||
- one OAuth2/OIDC provider and one ThothII application;
|
||||
- the exact callback `<public-origin>/api/auth/oidc/callback`;
|
||||
- `openid`, `profile`, and `email` scopes;
|
||||
- a direct, non-empty JSON string array claim named `groups`;
|
||||
- exact user/admin group mappings selected after the survey;
|
||||
- a separate group-view-only service account/API token for ThothII diagnostics.
|
||||
|
||||
Dedicated `TOT Users` and `TOT Admin` groups are the default unless the survey finds existing groups
|
||||
with exactly the intended semantics and the owner approves their reuse. Additional groups are
|
||||
ignored. Missing, malformed, indirect, or ambiguous configured groups fail closed.
|
||||
|
||||
### Supabase session storage
|
||||
|
||||
Do not create a separate PostgreSQL database. Use the server's existing Supabase PostgreSQL
|
||||
database and isolate ThothII work sessions in the dedicated `thoth_sessions` schema. This schema is
|
||||
distinct from the clinical `datawarehouse` schema.
|
||||
|
||||
- A one-shot migrator role owns only the required schema migration privileges.
|
||||
- Core receives only the restricted runtime role, never the migrator credential.
|
||||
- Forced RLS and application ownership checks isolate sessions by Authentik principal.
|
||||
- The schema stores principals/preferences, manifests, phase artifacts, review decisions, and
|
||||
audit records.
|
||||
- Chat and live SSE output remain ephemeral; semantic vectors remain in Qdrant; clinical data
|
||||
remains in `datawarehouse`; browser authentication sessions remain in protected auth state.
|
||||
- `thoth_sessions` must not be added to Supabase/PostgREST exposed schemas.
|
||||
- The direct runtime connection uses the TLS/CA contract required by the current application.
|
||||
|
||||
### Safe activation order
|
||||
|
||||
1. Back up the accepted Project A operator/auth configuration and every external configuration to
|
||||
be changed.
|
||||
2. Prepare Authentik objects without exposing the new route.
|
||||
3. Run and verify additive session-schema migrations; require no pending or drifted migrations.
|
||||
4. Prepare OIDC secrets and non-secret configuration in protected installation state.
|
||||
5. Validate Authentik discovery, issuer/JWKS, catalog access, and mapped groups.
|
||||
6. Validate Nginx, certificate, load-balancer route, callback, forwarded headers, and SSE while
|
||||
production traffic remains closed.
|
||||
7. Start ThothII in OIDC/public server mode with PostgreSQL session storage.
|
||||
8. Open the final load-balancer route.
|
||||
9. Preserve or update the Aritmolab sidebar link so the established user journey remains intact.
|
||||
10. Complete automated and human acceptance, then remove the Project A temporary route.
|
||||
|
||||
### Acceptance
|
||||
|
||||
Project B requires:
|
||||
|
||||
- successful redacted static, live, and interactive authentication diagnostics;
|
||||
- trusted certificate chain, correct public origin, callback, and proxy headers;
|
||||
- proven load-balancer/Nginx routing and SSE operation;
|
||||
- successful migration status, RLS/role tests, and proof that `thoth_sessions` is not REST-exposed;
|
||||
- ordinary, administrator, unmapped, malformed-claim, logout, and controlled provider-failure
|
||||
cases;
|
||||
- login to Aritmolab followed by the sidebar link to ThothII without a second credential prompt;
|
||||
- no raw OIDC token in browser storage, logs, diagnostics, or evidence;
|
||||
- session ownership and administrator-boundary tests;
|
||||
- one harmless F1-F8 PSD session under an OIDC identity;
|
||||
- tested rollback and a completed human manual-test report with an explicit PASS.
|
||||
|
||||
## Error handling and stop rules
|
||||
|
||||
Every executable step follows:
|
||||
|
||||
```text
|
||||
precondition -> action -> verification -> redacted evidence -> checkpoint
|
||||
```
|
||||
|
||||
Sol stops and requests owner help rather than improvising when it encounters:
|
||||
|
||||
- a dirty or unidentified source checkout;
|
||||
- uncertain ownership of Compose, Nginx, load-balancer, Aritmolab, or Authentik configuration;
|
||||
- missing or insufficient credentials;
|
||||
- an unsafe secret path or risk of secret disclosure;
|
||||
- an unverifiable backup or rollback path;
|
||||
- a DWH identity that is not demonstrably read-only;
|
||||
- a change that would affect unrelated Nginx virtual hosts or other applications;
|
||||
- an unprovable temporary-route restriction;
|
||||
- Supabase migration drift, excessive roles, or unintended REST exposure;
|
||||
- a material difference between the surveyed server and this design.
|
||||
|
||||
Unchanged external state is not failure. Sol records the observation and continues only when the
|
||||
current gate is satisfied.
|
||||
|
||||
## Rollback boundaries
|
||||
|
||||
- **Project A:** stop the new stack and restart the still-present legacy installation. No new
|
||||
volume is copied into it and no promise is made to retain it after Project B PASS.
|
||||
- **Project B ingress:** close the public route first, then restore the prior Nginx,
|
||||
load-balancer, certificate reference, and sidebar configuration.
|
||||
- **Project B application:** return to the accepted Project A local-auth operator configuration
|
||||
while the public route remains closed.
|
||||
- **Authentik:** initially disable new objects instead of deleting them; retain the pre-change
|
||||
export until final acceptance.
|
||||
- **Supabase:** migrations are additive. Rollback does not automatically drop `thoth_sessions` or
|
||||
destroy evidence; destructive cleanup requires a separate explicit decision.
|
||||
- **Workspace Git:** publish the multi-transport change as an isolated commit and retain the
|
||||
previous revision. A rejected candidate never replaces the installation's last valid snapshot.
|
||||
|
||||
## Documents and evidence
|
||||
|
||||
The implementation-planning phase creates:
|
||||
|
||||
1. a general execution program with cross-project gates;
|
||||
2. a survey checklist and report template;
|
||||
3. an executable Project A plan for Sol;
|
||||
4. a plain-language Project A manual-test guide;
|
||||
5. a Project A evidence/PASS template;
|
||||
6. an executable Project B plan for Sol;
|
||||
7. a plain-language Project B manual-test guide;
|
||||
8. a Project B evidence/PASS template.
|
||||
|
||||
Sol maintains a protected server-local progress journal and resumes from the last verified
|
||||
checkpoint. Detailed topology and command output remain in a protected evidence directory on the
|
||||
server. Only intentionally redacted reports and reusable templates may enter Git.
|
||||
|
||||
The Project A human guide is terminal-first and may use a local headless browser. When the optional
|
||||
private endpoint exists, it also includes operator browser checks. The Project B guide covers the
|
||||
real Aritmolab homepage/sidebar, Authentik SSO, roles, logout, final workflow, and negative cases.
|
||||
|
||||
## Known documentation reconciliation
|
||||
|
||||
Some older server documentation describes vector and embedding services as external and the
|
||||
mandatory stack as only frontend/core. Current `compose.yaml`, repository instructions, and project
|
||||
state define Qdrant and Ollama as mandatory internal services. The execution plans must treat the
|
||||
current code/Compose contract as authoritative and include a documentation correction rather than
|
||||
following the stale statements.
|
||||
|
||||
## Success condition
|
||||
|
||||
The program is complete only when both projects have exact-source evidence, all automated gates
|
||||
pass, both human manuals are completed with explicit PASS decisions, the Aritmolab sidebar reaches
|
||||
the final ThothII URL through the established load balancer and Nginx, Authentik provides SSO, and a
|
||||
real read-only PSD session completes F1-F8 under an authorized OIDC identity.
|
||||
@@ -1,245 +0,0 @@
|
||||
# PSD Server Deployment Program Implementation Plan
|
||||
|
||||
> **For agentic workers:** execute one task at a time, record its completion criteria, and stop at every documented approval boundary.
|
||||
|
||||
**Goal:** Replace the legacy PSD ThothII installation, prove the replacement with local authentication, and then integrate the accepted release with Supabase, Authentik, Nginx, the load balancer, and the Aritmolab sidebar.
|
||||
|
||||
**Architecture:** A read-only common survey freezes the actual server topology before any mutation. Project A installs a clean private stack and proves a complete PSD workflow; Project B begins only after a signed Project A PASS and performs the public OIDC/SSO cutover. Each project has an independent rollback boundary, human test guide, and evidence report.
|
||||
|
||||
**Tech Stack:** Linux, Docker Engine, Docker Compose v2, native `tht`, Fastify/React/Pi, PostgreSQL/Supabase, Qdrant, Ollama, Nginx, Authentik OIDC, Aritmolab, load balancer.
|
||||
|
||||
---
|
||||
|
||||
## Required reading and authority
|
||||
|
||||
Read these files completely before starting:
|
||||
|
||||
- `AGENTS.md`
|
||||
- `PROJECT_STATE.md`
|
||||
- `docs/plans/2026-08-20-psd-server-deployment-program-design.md`
|
||||
- `docs/install/server.md`
|
||||
- `docs/install/server-workspace-registry.md`
|
||||
- `docs/install/authentication-local.md`
|
||||
- `docs/install/authentication-oidc.md`
|
||||
- `docs/install/authentik.md`
|
||||
- `docs/contracts/workspace-preprocessing-cli.md`
|
||||
- `docs/testing/authentication-manual-acceptance.md`
|
||||
|
||||
The current `compose.yaml`, `deploy/compose.server.yaml`, repository instructions, and design are
|
||||
authoritative where older server prose still describes Qdrant or Ollama as external.
|
||||
|
||||
Run only from a terminal local to the server. Do not require SSH port forwarding. Do not print or
|
||||
paste passwords, tokens, cookies, private keys, hashes, or raw identity claims. Commands that need
|
||||
a credential must read a protected file or use an echo-free prompt.
|
||||
|
||||
## Documents used during execution
|
||||
|
||||
- Survey plan: `docs/plans/2026-08-20-psd-server-survey.md`
|
||||
- Survey report: `docs/testing/evidence/psd-server-survey-report-template.md`
|
||||
- Project A plan: `docs/plans/2026-08-20-psd-server-project-a-standalone.md`
|
||||
- Project A human guide: `docs/testing/psd-server-project-a-manual.md`
|
||||
- Project A report: `docs/testing/evidence/psd-server-project-a-report-template.md`
|
||||
- Project B plan: `docs/plans/2026-08-20-psd-server-project-b-authentik.md`
|
||||
- Project B human guide: `docs/testing/psd-server-project-b-manual.md`
|
||||
- Project B report: `docs/testing/evidence/psd-server-project-b-report-template.md`
|
||||
|
||||
The detailed evidence directory is a protected path on the server selected during the survey. The
|
||||
repository receives only redacted reports after explicit owner review.
|
||||
|
||||
## Owner-approved sequencing amendment — 2026-08-21
|
||||
|
||||
The Mac `rest_api` acceptance and revocation of `legacy-shared` move to a mandatory gate immediately
|
||||
before Project B. The survey therefore records two distinct decisions:
|
||||
|
||||
- `SURVEY_GO_PROJECT_A_PRIVATE`: technical prerequisite for requesting Project A private execution;
|
||||
- `SURVEY_GO_PROJECT_B`: the complete shared-infrastructure decision, including Mac acceptance,
|
||||
observation and legacy revocation.
|
||||
|
||||
The amendment authorizes the read-only survey and static preparation of non-secret Project A
|
||||
candidate facts and artifacts. While the current decision is `SURVEY_NO_GO`, it does not authorize
|
||||
creating installation roots, cloning/building the candidate, creating protected configuration or
|
||||
backup state, stopping the legacy stack, starting the new stack, changing public ingress, or
|
||||
starting Project B. Those remain separate explicit gates after the scoped survey passes.
|
||||
|
||||
### Task 1: Freeze the planning source
|
||||
|
||||
**Files:**
|
||||
- Read: `docs/plans/2026-08-20-psd-server-deployment-program-design.md`
|
||||
- Record: protected server execution journal selected during the survey
|
||||
|
||||
**Step 1: Verify the application checkout**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
git status --short --branch
|
||||
git rev-parse HEAD
|
||||
git rev-parse origin/main
|
||||
git show -s --format='%H%n%P%n%s' HEAD
|
||||
```
|
||||
|
||||
Expected: the tree is clean and `HEAD` is the explicitly approved `origin/main` SHA. A newer SHA
|
||||
than the design-time `5c0dc8c` is allowed only after recording and reviewing the intervening commits.
|
||||
|
||||
**Step 2: Verify the plan files exist at that SHA**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
test -f docs/plans/2026-08-20-psd-server-survey.md
|
||||
test -f docs/plans/2026-08-20-psd-server-project-a-standalone.md
|
||||
test -f docs/plans/2026-08-20-psd-server-project-b-authentik.md
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: every command exits zero.
|
||||
|
||||
**Step 3: Record the immutable planning identity**
|
||||
|
||||
Record the application SHA, plan commit, UTC timestamp, operator identity, and terminal-local access
|
||||
method in the protected journal. Do not record CyberArk session secrets or screenshots.
|
||||
|
||||
### Task 2: Execute and approve the common survey
|
||||
|
||||
**Files:**
|
||||
- Execute: `docs/plans/2026-08-20-psd-server-survey.md`
|
||||
- Create from: `docs/testing/evidence/psd-server-survey-report-template.md`
|
||||
|
||||
**Step 1: Execute every survey task without mutation**
|
||||
|
||||
Expected: the survey identifies exact paths and owners for the old and new installations, Nginx,
|
||||
load balancer, Aritmolab, Authentik, Supabase, the DWH, and protected credentials.
|
||||
|
||||
**Step 2: Resolve every unknown**
|
||||
|
||||
If an Authentik credential cannot be located, stop and ask the owner. If a configuration owner or
|
||||
rollback boundary is unclear, stop; do not infer authority from file readability.
|
||||
|
||||
**Step 3: Review the scoped survey GO/NO-GO**
|
||||
|
||||
Expected: `SURVEY_GO_PROJECT_A_PRIVATE` requires a verified old-stack recovery path, an approved
|
||||
new-installation root, enough resources, a direct read-only DWH path, workspace/model inputs, and
|
||||
no unresolved mutation in the private Project A scope. Public-origin, load-balancer and Authentik
|
||||
unknowns may remain explicitly deferred only while Project A is loopback-only and Task 10 is
|
||||
omitted. `SURVEY_GO_PROJECT_B` retains the complete survey requirements.
|
||||
|
||||
**Step 4: Checkpoint the survey**
|
||||
|
||||
Hash the protected report and record only its path, SHA-256, timestamp, and GO result in the journal.
|
||||
|
||||
### Task 3: Execute Project A
|
||||
|
||||
**Files:**
|
||||
- Execute: `docs/plans/2026-08-20-psd-server-project-a-standalone.md`
|
||||
- Complete: `docs/testing/evidence/psd-server-project-a-report-template.md`
|
||||
|
||||
**Step 1: Confirm the survey is GO for Project A private scope**
|
||||
|
||||
Expected: the survey report hash matches the journal and no unresolved blocker remains inside the
|
||||
Project A private scope. Before any stop/start, obtain a separate explicit owner authorization.
|
||||
|
||||
**Step 2: Execute Project A task-by-task**
|
||||
|
||||
Do not configure Authentik, change the production Aritmolab sidebar, or open the production route.
|
||||
|
||||
**Step 3: Run the Project A human guide**
|
||||
|
||||
Follow `docs/testing/psd-server-project-a-manual.md`. Record PASS/FAIL for every case; do not infer
|
||||
manual PASS from automated output. The Mac REST row may be
|
||||
`DEFERRED_PRE_PROJECT_B` only under the dated owner amendment.
|
||||
|
||||
**Step 4: Close the Project A report**
|
||||
|
||||
Expected: automated gates and the human guide are PASS; one harmless PSD session reached F8 and
|
||||
produced validated read-only SQL; rollback remains available. The accepted report must list the
|
||||
Mac REST item as an explicit deferred prerequisite rather than silently treating it as PASS.
|
||||
|
||||
**Step 5: Obtain explicit owner approval**
|
||||
|
||||
Record the approval and report digest. Project B remains forbidden without it.
|
||||
|
||||
### Task 4: Freeze the Project B candidate
|
||||
|
||||
**Files:**
|
||||
- Read: accepted Project A report
|
||||
- Record: protected server execution journal
|
||||
|
||||
**Step 1: Recheck source and running images**
|
||||
|
||||
Before freezing the candidate, close the pre-Project-B gate: validate the Mac installation with its
|
||||
per-installation key, finish the 48-hour observation window including two 03:00 ETL cycles, revoke
|
||||
`legacy-shared`, prove legacy `401` and v1 success, and obtain `SURVEY_GO_PROJECT_B`.
|
||||
|
||||
Run the Project A plan's identity commands again. Record application SHA, workspace SHA, core image
|
||||
ID, frontend image ID, Qdrant image digest, Ollama image digest, and local-auth configuration revision.
|
||||
|
||||
Expected: all values match the accepted Project A report.
|
||||
|
||||
**Step 2: Recheck rollback**
|
||||
|
||||
Prove that the public route is still closed and the protected Project A configuration can be
|
||||
selected without reconstructing it from memory.
|
||||
|
||||
### Task 5: Execute Project B
|
||||
|
||||
**Files:**
|
||||
- Execute: `docs/plans/2026-08-20-psd-server-project-b-authentik.md`
|
||||
- Complete: `docs/testing/evidence/psd-server-project-b-report-template.md`
|
||||
|
||||
**Step 1: Execute Project B task-by-task**
|
||||
|
||||
Keep public traffic closed until Authentik, Supabase migrations, ThothII diagnostics, Nginx, TLS,
|
||||
and load-balancer preflight all pass.
|
||||
|
||||
**Step 2: Run the Project B human guide**
|
||||
|
||||
Follow `docs/testing/psd-server-project-b-manual.md` using approved ordinary and administrator
|
||||
identities. The final path begins at the Aritmolab homepage and uses its existing sidebar link.
|
||||
|
||||
**Step 3: Close the Project B report**
|
||||
|
||||
Expected: SSO, roles, PostgreSQL ownership, the F1-F8 session, rollback rehearsal, and cleanup of the
|
||||
temporary Project A endpoint all pass.
|
||||
|
||||
### Task 6: Close the program
|
||||
|
||||
**Files:**
|
||||
- Modify: `PROJECT_STATE.md`
|
||||
- Optionally create: reviewed redacted acceptance reports under `docs/testing/evidence/`
|
||||
|
||||
**Step 1: Reconcile final state**
|
||||
|
||||
Record final SHAs, image identities, workspace revision, Authentik object names/IDs (never secrets),
|
||||
Supabase database/schema names, public origin, sidebar source revision, Nginx configuration identity,
|
||||
and both report digests.
|
||||
|
||||
**Step 2: Verify final negative boundaries**
|
||||
|
||||
Expected: old stack stopped; Project A private endpoint removed; core/Qdrant/Ollama not externally
|
||||
published; `thoth_sessions` absent from PostgREST exposed schemas; no secret appears in reports.
|
||||
|
||||
**Step 3: Update project state**
|
||||
|
||||
Add a dated factual section to `PROJECT_STATE.md`. Mark anything not actually run as PENDING.
|
||||
|
||||
**Step 4: Run documentation checks**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
git diff --check
|
||||
bash scripts/auth-docs-smoke.sh
|
||||
bash scripts/verify-workspace-install-docs.sh --fixtures-only
|
||||
```
|
||||
|
||||
Expected: all checks pass.
|
||||
|
||||
**Step 5: Commit only reviewed redacted documentation**
|
||||
|
||||
```bash
|
||||
git add PROJECT_STATE.md docs/testing/evidence
|
||||
git diff --cached --check
|
||||
git commit -m "docs: record PSD server deployment acceptance"
|
||||
```
|
||||
|
||||
Expected: the commit contains no raw server inventory or secret material.
|
||||
@@ -1,532 +0,0 @@
|
||||
# PSD Server Project A Standalone Implementation Plan
|
||||
|
||||
> **For agentic workers:** execute one task at a time, record its completion criteria, and stop at every documented approval boundary.
|
||||
|
||||
**Goal:** Install a clean PSD ThothII stack with local authentication, direct read-only DWH access, internal Qdrant/Ollama, rebuilt preprocessing, and one completed F1-F8 work session.
|
||||
|
||||
**Architecture:** Keep the stopped legacy installation intact only as a temporary rollback boundary and deploy the current canonical five-service Compose stack from an adjacent clean clone. Create no host service identity: the image's UID/GID 10001 remains numeric and unmapped, confined to the new writable bind roots. A reviewed server-local override disables public exposure and uses filesystem work sessions; an optional load-balancer route is permitted only when it is operator-only and its denial boundary is proved first.
|
||||
|
||||
**Tech Stack:** Git, Docker/Compose, native `tht`, local Argon2id authentication, PSD Supabase PostgreSQL direct transport, Qdrant, Ollama, Pi, Nginx/load-balancer test route where safe.
|
||||
|
||||
---
|
||||
|
||||
## Preconditions
|
||||
|
||||
- Common survey result is `SURVEY_GO_PROJECT_A_PRIVATE` and its digest is recorded.
|
||||
- Every path below is replaced by the exact survey result before execution.
|
||||
- No production Nginx/load-balancer/sidebar/Authentik change is in scope.
|
||||
- The old stack remains running until its exact inventory and restart recipe are verified; old and
|
||||
new stacks never run together.
|
||||
- The server's workspace deploy credential remains read-only. A curator with write access publishes
|
||||
the workspace change.
|
||||
- Owner amendment 2026-08-21 declares legacy sessions/configuration disposable. Retain old resources
|
||||
only until Project B proves the production Aritmolab journey, then delete them under a separate
|
||||
exact cleanup authorization. No legacy backup is required.
|
||||
- Do not create a host user or group for 10001. Immediately before preparing `/srv/thothii`, both
|
||||
`getent passwd 10001` and `getent group 10001` must return no match.
|
||||
- Owner amendment 2026-08-21 defers live Mac `rest_api` acceptance and `legacy-shared` revocation to
|
||||
the mandatory pre-Project-B gate. It does not authorize stop/start; those require a later explicit
|
||||
owner gate even after private preparation is complete.
|
||||
|
||||
### Task 1: Freeze exact inputs
|
||||
|
||||
**Files:**
|
||||
- Read: protected survey report
|
||||
- Read: `docs/plans/2026-08-20-psd-server-deployment-program-design.md`
|
||||
- Record: protected Project A journal
|
||||
|
||||
**Step 1: Record application identity**
|
||||
|
||||
Run in the new planning checkout:
|
||||
|
||||
```bash
|
||||
git status --short --branch
|
||||
git rev-parse HEAD
|
||||
git rev-parse origin/main
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: clean and explicitly approved SHA.
|
||||
|
||||
**Step 2: Record workspace remote identity**
|
||||
|
||||
Use the surveyed read-only credential and run:
|
||||
|
||||
```bash
|
||||
git ls-remote <workspace-remote> refs/heads/main
|
||||
```
|
||||
|
||||
Expected: one SHA recorded as the pre-change workspace revision.
|
||||
|
||||
**Step 3: Check old-stack recoverability**
|
||||
|
||||
Expected: exact old start/stop procedure, source SHA, Compose identity, volumes/binds, proxy closure
|
||||
procedure, restart recipe, and shared-resource exclusions are present in the survey. Stop if any is
|
||||
missing.
|
||||
|
||||
### Task 2: Publish the multi-transport workspace revision
|
||||
|
||||
**Files:**
|
||||
- Modify in authorized curator clone: `psd-clinical/workspace.yaml`
|
||||
- Verify: `thoth-workspaces.yaml`
|
||||
|
||||
**Step 1: Create a clean curator branch**
|
||||
|
||||
Run in a write-authorized clone, never in the application-managed registry checkout:
|
||||
|
||||
```bash
|
||||
git status --short --branch
|
||||
git fetch origin main
|
||||
git switch --create codex/psd-direct-transport origin/main
|
||||
```
|
||||
|
||||
Expected: clean branch at the recorded remote SHA.
|
||||
|
||||
**Step 2: Make the minimal descriptor change**
|
||||
|
||||
Change exactly:
|
||||
|
||||
```yaml
|
||||
supported_transports: [rest_api]
|
||||
```
|
||||
|
||||
to:
|
||||
|
||||
```yaml
|
||||
supported_transports: [rest_api, postgres_direct]
|
||||
```
|
||||
|
||||
Do not duplicate the workspace or change its ID, collection, Evidence, annotations, model policy,
|
||||
database, or schema.
|
||||
|
||||
**Step 3: Review the descriptor-only diff**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
git diff --check
|
||||
git diff -- psd-clinical/workspace.yaml thoth-workspaces.yaml
|
||||
```
|
||||
|
||||
Expected: one semantic line changed; catalog metadata remains identical.
|
||||
|
||||
**Step 4: Validate with the current ThothII contract**
|
||||
|
||||
Use a disposable installation/registry or the repository's current registry validation harness to
|
||||
activate the candidate commit before publication. Expected: schema v3 accepts both transports,
|
||||
Evidence and annotations materialize, and no secret is required for source validation.
|
||||
|
||||
If no supported validator can be run in the curator environment, stop and request the owner to run
|
||||
the established Mac validation; do not publish based only on YAML parsing.
|
||||
|
||||
**Step 5: Commit and publish through curator review**
|
||||
|
||||
```bash
|
||||
git add psd-clinical/workspace.yaml
|
||||
git diff --cached --check
|
||||
git commit -m "feat: support direct PSD DWH transport"
|
||||
git push --set-upstream origin codex/psd-direct-transport
|
||||
```
|
||||
|
||||
Merge through the repository's normal review path. Record the resulting `main` SHA.
|
||||
|
||||
**Step 6: Record the deferred Mac REST proof**
|
||||
|
||||
Do not change the Mac during Project A. Record `DEFERRED_PRE_PROJECT_B`, the unchanged expected
|
||||
transport `rest_api`, and the exact future diagnostics. The proof must become PASS before Project B,
|
||||
after protected delivery/configuration of the per-installation key.
|
||||
|
||||
### Task 3: Back up and stop the legacy installation
|
||||
|
||||
**Files:**
|
||||
- Create: surveyed protected legacy backup directory
|
||||
- Record: Project A journal
|
||||
|
||||
**Step 1: Capture final legacy state**
|
||||
|
||||
Run the surveyed legacy status/doctor commands, record source SHA and image IDs, and confirm no
|
||||
active user work. Do not use the new `tht` against an incompatible old descriptor.
|
||||
|
||||
**Step 2: Close or maintenance-gate the old ThothII route**
|
||||
|
||||
Change only the surveyed ThothII-specific route using its established mechanism. Validate Nginx and
|
||||
load-balancer configuration before applying. Confirm external requests no longer reach the app.
|
||||
|
||||
**Step 3: Record the disposable legacy boundary**
|
||||
|
||||
Record exact container and image IDs plus filesystem device/inode/ownership/size for
|
||||
`/home/chirone/ThothII` and `/home/chirone/thothii-data`. Do not archive their contents: the owner
|
||||
declared them disposable. Prove that the external Evidence bind and both shared Docker networks
|
||||
are excluded from any later cleanup manifest.
|
||||
|
||||
**Step 4: Stop the old stack**
|
||||
|
||||
Use its own supported controller. Expected: old containers stopped, not removed; volumes and bind
|
||||
trees unchanged.
|
||||
|
||||
**Step 5: Rehearse the restart command without executing it**
|
||||
|
||||
Record the exact command, preconditions, port ownership, and route-restoration order. If it cannot
|
||||
be stated unambiguously, stop before creating the new stack.
|
||||
|
||||
### Task 4: Prepare the adjacent clean installation
|
||||
|
||||
**Files:**
|
||||
- Create: survey-selected new source root
|
||||
- Create: survey-selected operator, secret, data, Pi-state, registry, and backup roots
|
||||
|
||||
**Step 1: Create dedicated paths without creating identities**
|
||||
|
||||
Follow `docs/install/server.md` numeric-ownership rules. Do not call `useradd`, `groupadd`, or
|
||||
`usermod`. The existing operator owns source/operator files; only the new data, Pi-state, and
|
||||
workspace-registry roots use unmapped numeric `10001:10001`. Stop if UID or GID 10001 resolves to a
|
||||
host account, and do not change the image identity without a reviewed design amendment.
|
||||
|
||||
**Step 2: Clone the frozen application source**
|
||||
|
||||
```bash
|
||||
git -c core.autocrlf=false clone <thothii-remote> <new-source-root>/ThothII
|
||||
git -C <new-source-root>/ThothII config --local core.autocrlf false
|
||||
git -C <new-source-root>/ThothII switch --detach <approved-application-sha>
|
||||
git -C <new-source-root>/ThothII status --short --branch
|
||||
```
|
||||
|
||||
Expected: detached exact SHA, clean tree.
|
||||
|
||||
**Step 3: Verify source and platform**
|
||||
|
||||
```bash
|
||||
cd <new-source-root>/ThothII
|
||||
bash scripts/verify-line-endings.sh
|
||||
docker version
|
||||
docker compose version
|
||||
```
|
||||
|
||||
Expected: all pass.
|
||||
|
||||
**Step 4: Prepare Pi state and build the operator**
|
||||
|
||||
```bash
|
||||
sudo scripts/prepare-server-pi-state.sh <new-pi-state-root> 10001 10001
|
||||
THT_THT_OUTPUT_DIRECTORY=<protected-build-output> bash scripts/build-tht.sh
|
||||
```
|
||||
|
||||
Install only the binary matching the surveyed server architecture. Run `tht version --json` and
|
||||
record its source identity.
|
||||
|
||||
### Task 5: Create the protected Project A configuration
|
||||
|
||||
**Files:**
|
||||
- Create outside Git: `<project-a-operator-root>/server.env`
|
||||
- Create outside Git: `<project-a-operator-root>/thothii-installation.yaml`
|
||||
- Create outside Git: `<project-a-operator-root>/project-a-private.yaml`
|
||||
- Create outside Git: `<project-a-auth-root>/auth.yaml` through `tht`
|
||||
|
||||
**Step 1: Start from current examples**
|
||||
|
||||
Copy `deploy/env/server.env.example` and `docs/install/examples/thothii-installation.server.yaml`
|
||||
to the protected Project A operator root. Replace every placeholder with surveyed absolute paths.
|
||||
Never source `server.env` as shell code.
|
||||
|
||||
**Step 2: Add the private/local-session override**
|
||||
|
||||
Create this reviewed override:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
core:
|
||||
environment:
|
||||
THOTH_PUBLIC_EXPOSURE: "false"
|
||||
THT_SESSION_STORAGE: local
|
||||
frontend:
|
||||
ports: !override
|
||||
- "127.0.0.1:<project-a-port>:8080"
|
||||
```
|
||||
|
||||
Select an unused loopback port proved by `ss -lntp`. Do not publish core, Qdrant, or Ollama.
|
||||
|
||||
**Step 3: Compose the installation descriptor**
|
||||
|
||||
Use `profile: server`, the exact new source root/env/auth root, workspace remote/branch/read-only
|
||||
access, Project A override, and exactly one Git transport override. Do not include the public
|
||||
session-server overlay in Project A.
|
||||
|
||||
For the projected local-auth descriptor, keep `authentication.configDirectory` as the canonical
|
||||
root and add `runtimeProjection` with a distinct absolute runtime directory plus numeric `uid: 10001`
|
||||
and `gid: 10001`. The descriptor loader includes the automatic runtime-projection override; do
|
||||
not list it manually under `overrides`. The canonical root stays `root:root 0700/0600`; the
|
||||
publisher owns the projection numerically as `10001:10001 0700/0600`. Only the projection is
|
||||
mounted read-only into core. Do not create a host user/group, edit `CURRENT` or `generations`, or
|
||||
apply these paths before the separately authorized start gate.
|
||||
|
||||
**Step 4: Validate permissions and render**
|
||||
|
||||
```bash
|
||||
<new-tht> --installation <project-a-installation> update --check-only
|
||||
```
|
||||
|
||||
Expected: Compose validates; only frontend has a loopback port; core declares public exposure false
|
||||
and local session storage; Qdrant/Ollama are internal.
|
||||
|
||||
**Step 5: Configure the local administrator**
|
||||
|
||||
Create a temporary mode-0600 password file using an echo-free prompt, then run:
|
||||
|
||||
```bash
|
||||
<new-tht> --installation <project-a-installation> auth configure \
|
||||
--mode local --public-url <project-a-origin> \
|
||||
--admin-user <test-admin> --admin-display-name <display-name> \
|
||||
--password-file <protected-temporary-password-file>
|
||||
```
|
||||
|
||||
Remove the temporary input file after success and record that removal. Do not delete generated
|
||||
`auth.yaml` or `users.yaml`.
|
||||
|
||||
Before the later start gate, run the redacted projected status command and require `ready` plus
|
||||
`equal: true`. A blocked result prevents start. `auth publish` is the only repair path: it rebuilds
|
||||
from canonical authentication, including after a candidate or recovery restore outcome; it never
|
||||
promotes a retained runtime generation on its own.
|
||||
|
||||
### Task 6: Build and start the clean stack
|
||||
|
||||
**Files:**
|
||||
- Record: Project A evidence directory
|
||||
|
||||
**Step 1: Build current images**
|
||||
|
||||
```bash
|
||||
cd <new-source-root>/ThothII
|
||||
bash scripts/build-local.sh
|
||||
```
|
||||
|
||||
Expected: current core/frontend images build; pinned Qdrant/Ollama references resolve.
|
||||
|
||||
**Step 2: Run preflight**
|
||||
|
||||
```bash
|
||||
<new-tht> --installation <project-a-installation> update --check-only
|
||||
<new-tht> --installation <project-a-installation> pi doctor
|
||||
```
|
||||
|
||||
Expected: no mutation error and no secret in output.
|
||||
|
||||
**Step 3: Start through `tht`**
|
||||
|
||||
```bash
|
||||
<new-tht> --installation <project-a-installation> start --build
|
||||
<new-tht> --installation <project-a-installation> status
|
||||
<new-tht> --installation <project-a-installation> doctor --json
|
||||
<new-tht> --installation <project-a-installation> pi test
|
||||
```
|
||||
|
||||
Expected: frontend, core, Qdrant, embedding healthy and model initializer completed. Doctor has the
|
||||
documented ordered checks and authentication PASS.
|
||||
|
||||
**Step 4: Verify listener boundaries**
|
||||
|
||||
Use `ss -lntp` and bounded Docker inspection. Expected: only the selected frontend loopback port is
|
||||
host-published; no external core, Qdrant, or Ollama listener.
|
||||
|
||||
### Task 7: Activate the workspace and direct DWH binding
|
||||
|
||||
**Files:**
|
||||
- Modify only through authenticated Workspace Management: encrypted workspace secret store
|
||||
|
||||
**Step 1: Pull and inspect the reviewed workspace revision**
|
||||
|
||||
```bash
|
||||
<new-tht> --installation <project-a-installation> \
|
||||
workspace inspect --workspace psd-clinical --json
|
||||
```
|
||||
|
||||
Expected: active workspace SHA equals the approved multi-transport revision.
|
||||
|
||||
**Step 2: Configure runtime bindings**
|
||||
|
||||
Through the authenticated Workspace Management API/UI, select `postgres_direct` and provide the
|
||||
surveyed host, port, runtime user, password, and optional TLS CA. Secret values go to the encrypted
|
||||
vault; they do not enter `server.env`, Git, shell arguments, or evidence.
|
||||
|
||||
The generated contract names are:
|
||||
|
||||
```text
|
||||
THT_WS_PSD_CLINICAL_DWH_TRANSPORT
|
||||
THT_WS_PSD_CLINICAL_DWH_HOST
|
||||
THT_WS_PSD_CLINICAL_DWH_PORT
|
||||
THT_WS_PSD_CLINICAL_DWH_USER
|
||||
THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE
|
||||
THT_WS_PSD_CLINICAL_DWH_TLS_CA_FILE
|
||||
```
|
||||
|
||||
**Step 3: Validate and test connections**
|
||||
|
||||
Run static validation, live connection test, workspace inspect, and `doctor --json`. Expected: DWH,
|
||||
workspace, internal embedding, and Qdrant checks pass with redacted output.
|
||||
|
||||
**Step 4: Re-prove read-only grants**
|
||||
|
||||
Use the survey's catalog query through the exact configured identity. Expected: no DML/DDL grant on
|
||||
`datawarehouse`. Stop if the runtime user is an owner, superuser, or write-capable role.
|
||||
|
||||
### Task 8: Rebuild and verify semantic preprocessing
|
||||
|
||||
**Files:**
|
||||
- Create: Project A preprocessing evidence
|
||||
|
||||
**Step 1: Inspect the empty/new collection state**
|
||||
|
||||
```bash
|
||||
<new-tht> --installation <project-a-installation> \
|
||||
workspace vector inspect --workspace psd-clinical --json
|
||||
```
|
||||
|
||||
Expected: either a compatible empty collection or the documented missing-collection state.
|
||||
|
||||
**Step 2: Create the descriptor-owned collection when missing**
|
||||
|
||||
Use the guarded vector rebuild only for collection `psd-clinical`, with exact repeated confirmation
|
||||
and `--destroy`. Do not run it against any other collection.
|
||||
|
||||
**Step 3: Run complete preprocessing**
|
||||
|
||||
```bash
|
||||
<new-tht> --installation <project-a-installation> \
|
||||
workspace preprocess run --workspace psd-clinical --json
|
||||
```
|
||||
|
||||
If it returns `manual_review_required`, inspect the exact run and curated annotations, obtain the
|
||||
required human decision, run `workspace schema accept --workspace psd-clinical --run <run-id> --yes`,
|
||||
then resume the same run. Never auto-approve unknown FK changes.
|
||||
|
||||
**Step 4: Verify collection contract and counts**
|
||||
|
||||
Run vector inspection and record dimensions, cosine distance, required keyword indexes, and bounded
|
||||
counts by payload kind/revision. Expected: all points carry the active workspace revision.
|
||||
|
||||
**Step 5: Prove idempotency**
|
||||
|
||||
Run the complete preprocessing command again. Expected: no new review, no duplicate logical points,
|
||||
unchanged Evidence reported as unchanged, and the same effective configuration identity.
|
||||
|
||||
### Task 9: Configure and test local users
|
||||
|
||||
**Files:**
|
||||
- Modify through `tht auth user`: protected local user registry
|
||||
|
||||
**Step 1: Add an ordinary test user**
|
||||
|
||||
Use an echo-free prompt or protected temporary password file:
|
||||
|
||||
```bash
|
||||
<new-tht> --installation <project-a-installation> auth user add <test-user> \
|
||||
--role user --display-name <display-name> --password-file <protected-temporary-password-file>
|
||||
```
|
||||
|
||||
Remove the temporary input file after success.
|
||||
|
||||
**Step 2: Run authentication diagnostics**
|
||||
|
||||
```bash
|
||||
<new-tht> --installation <project-a-installation> auth status --json
|
||||
<new-tht> --installation <project-a-installation> auth check --json
|
||||
```
|
||||
|
||||
Expected: pristine redacted JSON and PASS.
|
||||
|
||||
**Step 3: Execute automated local-auth cases**
|
||||
|
||||
Use same-origin requests or a local headless browser to prove ordinary/admin authorization, generic
|
||||
wrong-password failure, disable/enable, password/role revision invalidation, logout-all, CSRF, and
|
||||
remembered-session survival after core restart. Do not retain cookie jars after the test.
|
||||
|
||||
### Task 10: Optionally add the private network-path test
|
||||
|
||||
**Current scope boundary (owner, 2026-08-21):** omit this entire task and keep Project A
|
||||
loopback-only. Any future use requires a separate shared-infrastructure authorization after the
|
||||
public-origin and load-balancer activities pass; the Project A private survey decision alone is
|
||||
insufficient.
|
||||
|
||||
**Files:**
|
||||
- Modify only surveyed test-specific load-balancer/Nginx files
|
||||
- Create: test certificate through the existing managed mechanism
|
||||
|
||||
**Step 1: Prove allowlist capability before proxying**
|
||||
|
||||
Create a temporary hostname that returns a fixed maintenance response. From an approved operator
|
||||
source expect success; from an unapproved source expect denial. Do not point it at ThothII yet.
|
||||
|
||||
**Step 2: Validate and activate the test proxy**
|
||||
|
||||
Configure Nginx with the same Host/HTTPS forwarding and SSE settings intended for production. Run
|
||||
`nginx -t`, validate the load balancer, then reload through the established mechanism.
|
||||
|
||||
**Step 3: Reconfigure local-auth public URL transactionally**
|
||||
|
||||
If the exact private HTTPS origin differs from the loopback origin, use the supported authentication
|
||||
configuration workflow and invalidate prior test sessions. Re-run auth and doctor checks.
|
||||
|
||||
**Step 4: Prove both sides**
|
||||
|
||||
Expected: authorized operator reaches the local login; unauthorized source remains denied before
|
||||
ThothII. If this cannot be demonstrated, remove the test route and continue on loopback.
|
||||
|
||||
### Task 11: Complete the F1-F8 acceptance session
|
||||
|
||||
**Files:**
|
||||
- Complete: `docs/testing/psd-server-project-a-manual.md`
|
||||
- Create: protected session evidence
|
||||
|
||||
**Step 1: Select the approved harmless question**
|
||||
|
||||
Use a known read-only PSD question agreed by the owner. Record the wording in the protected report;
|
||||
do not include patient-identifying values.
|
||||
|
||||
**Step 2: Create the session as the ordinary local user**
|
||||
|
||||
Use the private browser route when present; otherwise drive the same-origin frontend/API from the
|
||||
server-local terminal/headless browser. Record only session ID and sanitized milestones.
|
||||
|
||||
**Step 3: Review every gate**
|
||||
|
||||
Complete F1-F8 without auto-confirming human decisions. Confirm persisted phase/artifact state after
|
||||
each gate and resume once to prove recovery.
|
||||
|
||||
**Step 4: Validate final SQL**
|
||||
|
||||
Expected: finalized session, DWH validation PASS, SQL is read-only, and no clinical mutation occurs.
|
||||
|
||||
**Step 5: Inspect persisted state**
|
||||
|
||||
Confirm manifest, question, schema linking, Evidence, CTE plan/tests, final SQL, validation report,
|
||||
and decision ledger exist in local filesystem session storage. Chat/SSE need not persist.
|
||||
|
||||
### Task 12: Close Project A and preserve rollback
|
||||
|
||||
**Files:**
|
||||
- Complete: `docs/testing/evidence/psd-server-project-a-report-template.md`
|
||||
|
||||
**Step 1: Run final diagnostics**
|
||||
|
||||
Run status, doctor, auth check, workspace inspect, vector inspect, Pi test, and a bounded secret scan
|
||||
of the intended report.
|
||||
|
||||
**Step 2: Create a transactional new-installation backup**
|
||||
|
||||
Use `tht backup --drain` with a protected explicit output. Verify its checksum. Do not include
|
||||
secrets in the ordinary evidence archive.
|
||||
|
||||
**Step 3: Complete human acceptance**
|
||||
|
||||
Every private-server row in `docs/testing/psd-server-project-a-manual.md` must be PASS or explicitly
|
||||
blocking. Only the Mac REST row may be `DEFERRED_PRE_PROJECT_B` under the dated owner amendment.
|
||||
|
||||
**Step 4: Record the gate**
|
||||
|
||||
Record exact SHAs/images, workspace revision, preprocessing identity/counts, session ID, report
|
||||
digest, rollback status, and explicit `PROJECT_A_PRIVATE_PASS` or `PROJECT_A_FAIL`. A private PASS
|
||||
does not authorize Project B while the deferred gate remains open.
|
||||
|
||||
**Step 5: Stop on FAIL**
|
||||
|
||||
On FAIL, stop the new stack and use the surveyed old-stack recovery plan if service restoration is
|
||||
desired. Do not start Project B.
|
||||
@@ -1,442 +0,0 @@
|
||||
# PSD Server Project B Authentik Integration Implementation Plan
|
||||
|
||||
> **For agentic workers:** execute one task at a time, record its completion criteria, and stop at every documented approval boundary.
|
||||
|
||||
**Goal:** Convert the accepted Project A installation to public OIDC mode, store owned work sessions in the existing Supabase database's `thoth_sessions` schema, and restore the established Aritmolab-sidebar user journey through the load balancer and Nginx.
|
||||
|
||||
**Architecture:** Keep the same installation-descriptor path and Compose project so Project A Qdrant/Ollama volumes and bind state remain authoritative. Close ingress, snapshot Project A configuration, configure Authentik and Supabase, replace the protected installation configuration transactionally at the same paths, validate privately, then open the production route and sidebar link.
|
||||
|
||||
**Tech Stack:** Authentik OAuth2/OIDC Authorization Code + PKCE, ThothII OIDC/session store, Supabase PostgreSQL schema migrations/RLS, Docker Compose, Nginx, load balancer, Aritmolab.
|
||||
|
||||
---
|
||||
|
||||
## Preconditions
|
||||
|
||||
- Project A automated and human reports are PASS and explicitly owner-approved.
|
||||
- The Mac `rest_api` installation passes source validation and connection diagnostics with its
|
||||
per-installation key.
|
||||
- The dual-key observation has lasted at least 48 hours and includes two scheduled 03:00 ETL cycles.
|
||||
- `legacy-shared` is revoked; v1 remains successful and the legacy credential is proven `401`.
|
||||
- The current survey decision is `SURVEY_GO_PROJECT_B`, not only the private Project A decision.
|
||||
- Application SHA, workspace SHA, images, Project A report digest, and rollback configuration match
|
||||
the accepted evidence.
|
||||
- The production route is closed before authentication/session-storage changes.
|
||||
- All Authentik operations use the installed version's API/OpenAPI contract. Official current
|
||||
references include [OAuth2/OIDC providers](https://docs.goauthentik.io/add-secure-apps/providers/oauth2/),
|
||||
[provider property mappings](https://docs.goauthentik.io/add-secure-apps/providers/property-mappings/),
|
||||
[application bindings](https://docs.goauthentik.io/add-secure-apps/applications/manage_apps/), and
|
||||
[blueprint export](https://docs.goauthentik.io/customize/blueprints/export); installed-version
|
||||
behavior wins over newer documentation.
|
||||
|
||||
### Task 1: Freeze Project A and close ingress
|
||||
|
||||
**Files:**
|
||||
- Read: accepted Project A report
|
||||
- Create: protected Project B transaction root
|
||||
|
||||
**Step 1: Verify exact Project A state**
|
||||
|
||||
Run status, doctor, auth check, workspace inspect, vector inspect, Pi status/test, Git SHA, and image
|
||||
identity checks from Project A. Expected: all match the accepted report.
|
||||
|
||||
**Step 2: Create a protected transaction root**
|
||||
|
||||
Use `mktemp -d` under the survey-approved protected parent, mode `0700`. Record its path and do not
|
||||
place it in Git.
|
||||
|
||||
**Step 3: Close production and temporary ingress**
|
||||
|
||||
Keep or restore a maintenance response at the production ThothII route. Disable the optional
|
||||
Project A test route before changing authentication unless it is needed for a separately approved
|
||||
private preflight. Confirm neither route reaches ThothII.
|
||||
|
||||
**Step 4: Stop and back up Project A**
|
||||
|
||||
```bash
|
||||
<tht> --installation <stable-installation-path> backup \
|
||||
--output <project-b-transaction-root>/project-a-backup.tar --drain
|
||||
<tht> --installation <stable-installation-path> stop
|
||||
```
|
||||
|
||||
Verify the archive using the controller's manifest/checksum contract. Preserve a copy of the exact
|
||||
installation descriptor, env file, override files, auth directory, Nginx fragment, load-balancer
|
||||
route, Aritmolab sidebar file/revision, and relevant Authentik export metadata.
|
||||
|
||||
### Task 2: Decide exact Authentik names and roles
|
||||
|
||||
**Files:**
|
||||
- Create: protected `authentik-change-manifest.yaml`
|
||||
|
||||
**Step 1: Select the final public origin**
|
||||
|
||||
Use the live survey result, not historical `.it`/`.com` assumptions. Record exactly one HTTPS origin
|
||||
and callback `<origin>/api/auth/oidc/callback`.
|
||||
|
||||
**Step 2: Select exact groups**
|
||||
|
||||
Default to dedicated `TOT Users` and `TOT Admin`. Reuse existing groups only if their membership
|
||||
semantics match and the owner approves. Record exact case-sensitive names.
|
||||
|
||||
**Step 3: Define least privilege**
|
||||
|
||||
Map user group → `user`, admin group → `admin`. Define a separate service account/token with only
|
||||
the installed Authentik permission needed to view exact group objects. No write, user-management,
|
||||
directory-administration, or superuser permission.
|
||||
|
||||
**Step 4: Obtain owner approval of the manifest**
|
||||
|
||||
The manifest contains object names, slugs, intended bindings, callback, scopes, grant types,
|
||||
credential destinations, and rollback action—but no secret values. Do not mutate Authentik before
|
||||
approval.
|
||||
|
||||
### Task 3: Export and prepare Authentik
|
||||
|
||||
**Files:**
|
||||
- Create: protected pre-change Authentik export
|
||||
- Modify: Authentik objects named in the approved manifest
|
||||
|
||||
**Step 1: Export relevant configuration**
|
||||
|
||||
Use the installed version's supported blueprint/API export. A worker command such as
|
||||
`ak export_blueprint` is valid only if present in that version. Protect export mode `0600`; remember
|
||||
write-only provider secrets are not included, so backup their custody separately without printing.
|
||||
|
||||
**Step 2: Verify API credential scope**
|
||||
|
||||
Use a read-only call to list relevant groups/applications. Expected: administrative creation access
|
||||
for the setup identity and a distinct path for the future group-view service account. Stop if the
|
||||
credential is missing or ambiguous.
|
||||
|
||||
**Step 3: Create or confirm exact groups**
|
||||
|
||||
Create missing dedicated groups, or record approved existing group IDs. Do not bulk-copy LDAP or
|
||||
unrelated Authentik memberships.
|
||||
|
||||
**Step 4: Create the group-catalog service account**
|
||||
|
||||
Grant only exact group-view permission. Create its token through the approved protected-secret
|
||||
mechanism; write it directly to the ThothII secret destination without displaying it.
|
||||
|
||||
**Step 5: Create the OIDC provider**
|
||||
|
||||
Configure a confidential OAuth2/OIDC provider with the exact callback, issuer mode observed as
|
||||
appropriate, Authorization Code, PKCE support, and Device Code only when required for
|
||||
`tht auth check --interactive` and supported by the installed release. Do not enable implicit flow.
|
||||
|
||||
**Step 6: Configure scopes and direct groups claim**
|
||||
|
||||
Select `openid`, `profile`, and `email`. Inspect a disposable identity's decoded claim keys through
|
||||
a protected verifier; retain only a redacted shape. Expected: ID token includes direct non-empty
|
||||
`groups: [string, ...]`.
|
||||
|
||||
If the installed default profile mapping already provides that exact claim, reuse it. Otherwise add
|
||||
a provider scope/property mapping under the requested `profile` scope that returns:
|
||||
|
||||
```python
|
||||
return {"groups": [group.name for group in request.user.ak_groups.all()]}
|
||||
```
|
||||
|
||||
Verify the installed mapping merge semantics before activation. Do not add a custom unrequested
|
||||
scope because ThothII requests only `openid`, `profile`, and `email`.
|
||||
|
||||
**Step 7: Create the Authentik application**
|
||||
|
||||
Bind it to the provider. Configure display metadata according to local Aritmolab conventions. Do not
|
||||
use an Authentik proxy provider or Nginx forward-auth for ThothII.
|
||||
|
||||
**Step 8: Create and store the client secret**
|
||||
|
||||
Write the client secret directly into the protected ThothII secret bundle key
|
||||
`THT_OIDC_CLIENT_SECRET`. Store the service-account token as `THT_AUTHENTIK_API_TOKEN`. Never place
|
||||
either value in the change manifest, shell history, Compose environment, or evidence.
|
||||
|
||||
### Task 4: Prepare Supabase schema roles and backup
|
||||
|
||||
**Files:**
|
||||
- Read: `harness/tht/migrations/sessions/001_schema.sql`
|
||||
- Read: `harness/tht/migrations/sessions/002_security.sql`
|
||||
- Create: protected Supabase backup/evidence
|
||||
- Create: runtime and migrator credential files
|
||||
|
||||
**Step 1: Confirm the database/schema boundary**
|
||||
|
||||
Expected: use the surveyed existing Supabase PostgreSQL database; clinical data remains in
|
||||
`datawarehouse`; application sessions use schema `thoth_sessions`; no new database is created.
|
||||
|
||||
**Step 2: Back up database metadata/data consistently**
|
||||
|
||||
Use the existing Supabase/PostgreSQL backup procedure before DDL. Record backup ID, timestamp,
|
||||
checksum, and restore command. Do not put a dump in the Git repository.
|
||||
|
||||
**Step 3: Create or validate dedicated roles**
|
||||
|
||||
Create one migrator login and one runtime login according to the migration contract. The runtime
|
||||
role must not be superuser, owner, BYPASSRLS, CREATEROLE, CREATEDB, or a member of DWH write roles.
|
||||
The migrator credential remains unavailable to core.
|
||||
|
||||
**Step 4: Write protected credential files**
|
||||
|
||||
Create separate mode-0640 runtime-password, migrator-password, and session-CA files with surveyed
|
||||
ownership. Do not use command-line password arguments.
|
||||
|
||||
**Step 5: Confirm PostgREST exclusion before migration**
|
||||
|
||||
Record the exact exposed schema list. Expected: `thoth_sessions` absent. If the system exposes all
|
||||
schemas implicitly, stop and resolve the boundary before migration.
|
||||
|
||||
### Task 5: Prepare the stable Project B installation configuration
|
||||
|
||||
**Files:**
|
||||
- Modify at the same stable paths: operator env, installation descriptor, authentication directory
|
||||
- Create: reviewed session-server override copied from `deploy/compose.session-server.yaml.example`
|
||||
- Create: protected server-session workspace config copied from `deploy/workspaces/server-sessions.yaml.example`
|
||||
|
||||
**Step 1: Preserve the Compose project name**
|
||||
|
||||
The native controller derives the project name from the absolute installation-descriptor path.
|
||||
Keep that exact path. Do not point Project B at a second descriptor path, because that would create
|
||||
new Qdrant/Ollama named volumes instead of using the Project A accepted state.
|
||||
|
||||
**Step 2: Stage Project B files beside the live files**
|
||||
|
||||
Prepare new env/descriptor/override/auth inputs in the protected transaction root. Add the session
|
||||
DB host/port/existing database name/runtime role/migrator role/TLS mode and three secret source
|
||||
paths. Use `verify-full` where hostname/SAN permits; any `verify-ca` exception requires explicit
|
||||
survey evidence and owner approval.
|
||||
|
||||
**Step 3: Add the server-session override**
|
||||
|
||||
Copy the current example to a reviewed local file and add it to the existing stable descriptor's
|
||||
overrides before the Git transport override ordering required by the installation. Do not edit the
|
||||
tracked example.
|
||||
|
||||
**Step 4: Replace local auth state transactionally**
|
||||
|
||||
With the stack stopped, move the complete Project A auth directory into the protected transaction
|
||||
root, recreate an empty private directory at the same path/ownership/mode, and configure OIDC:
|
||||
|
||||
```bash
|
||||
<tht> --installation <stable-installation-path> auth configure \
|
||||
--mode oidc --public-url <final-https-origin> \
|
||||
--issuer <authentik-issuer> --client-id <oidc-client-id> \
|
||||
--authentik-base-url <authentik-base-url> \
|
||||
--user-group '<exact-user-group>' --admin-group '<exact-admin-group>'
|
||||
```
|
||||
|
||||
Expected: non-secret `auth.yaml` only; secrets resolved from the protected bundle.
|
||||
|
||||
**Step 5: Atomically install staged path-only files**
|
||||
|
||||
Use same-filesystem rename and preserve required ownership/mode. Keep the Project A originals in
|
||||
the transaction root. Run `update --check-only`; on failure restore the originals immediately.
|
||||
|
||||
### Task 6: Run and verify session migrations
|
||||
|
||||
**Files:**
|
||||
- Modify through one-shot migrator: existing database schema `thoth_sessions`
|
||||
|
||||
**Step 1: Validate migration rendering**
|
||||
|
||||
```bash
|
||||
<tht> --installation <stable-installation-path> update --check-only
|
||||
```
|
||||
|
||||
Expected: core and `session-migrate` resolve the same core image; core lacks migrator password;
|
||||
only the one-shot service sees it.
|
||||
|
||||
**Step 2: Run migrations once**
|
||||
|
||||
```bash
|
||||
<tht> --installation <stable-installation-path> sessions migrate --yes
|
||||
```
|
||||
|
||||
Expected JSON: `"pending":[]` and `"drifted":[]`; `applied` may list `001` and `002` on first use.
|
||||
|
||||
**Step 3: Run migration status/idempotency again**
|
||||
|
||||
Run the same command. Expected: no new application and both pending/drifted remain empty.
|
||||
|
||||
**Step 4: Verify database security**
|
||||
|
||||
Through bounded catalog queries, prove forced RLS, policies on session tables, runtime role without
|
||||
BYPASSRLS/ownership/DDL, migrator absent from core, and no runtime privileges on unrelated schemas.
|
||||
|
||||
**Step 5: Recheck PostgREST exclusion**
|
||||
|
||||
Expected: `thoth_sessions` still absent from exposed schemas and REST endpoints cannot address it.
|
||||
|
||||
### Task 7: Validate Authentik and start privately
|
||||
|
||||
**Files:**
|
||||
- Record: Project B protected evidence
|
||||
|
||||
**Step 1: Run static configuration validation**
|
||||
|
||||
Run `update --check-only` and redacted `auth status --json`. Expected: mode OIDC, exact public origin,
|
||||
issuer/client ID/group names, and no secret values.
|
||||
|
||||
**Step 2: Start while public ingress remains closed**
|
||||
|
||||
```bash
|
||||
<tht> --installation <stable-installation-path> start
|
||||
<tht> --installation <stable-installation-path> status
|
||||
<tht> --installation <stable-installation-path> auth check --json
|
||||
<tht> --installation <stable-installation-path> doctor --json
|
||||
<tht> --installation <stable-installation-path> pi test
|
||||
```
|
||||
|
||||
Expected: OIDC discovery/issuer/JWKS, client-secret access, Authentik catalog token, exact mapped
|
||||
groups, PostgreSQL session storage, workspace, services, workflow, and Pi pass.
|
||||
|
||||
**Step 3: Run interactive device check when supported**
|
||||
|
||||
```bash
|
||||
<tht> --installation <stable-installation-path> auth check --interactive
|
||||
```
|
||||
|
||||
Expected: approved identity completes Device Authorization and direct groups claim validates. If
|
||||
the installed provider does not support device flow, record PENDING rather than substituting a token.
|
||||
|
||||
### Task 8: Prepare Nginx, TLS, load balancer, and sidebar
|
||||
|
||||
**Files:**
|
||||
- Modify only survey-approved ThothII Nginx fragment
|
||||
- Modify only survey-approved load-balancer route
|
||||
- Modify only exact Aritmolab sidebar source when its target must change
|
||||
|
||||
**Step 1: Prepare direct-OIDC Nginx configuration**
|
||||
|
||||
Follow `docs/install/reverse-proxy-nginx.md`, direct OIDC section. Required behavior: no
|
||||
`auth_request`, no callback rewrite, frontend loopback upstream, Host and HTTPS forwarded headers,
|
||||
HTTP/1.1, buffering/cache off, long SSE read timeout, and `X-Accel-Buffering: no`.
|
||||
|
||||
**Step 2: Validate the managed certificate**
|
||||
|
||||
Expected: SAN matches final hostname, validity is current, chain is trusted by approved clients,
|
||||
private-key permissions match local policy, and renewal/generation ownership is recorded. Never
|
||||
copy the key into ThothII.
|
||||
|
||||
**Step 3: Validate Nginx without opening traffic**
|
||||
|
||||
```bash
|
||||
sudo nginx -t
|
||||
curl --fail http://127.0.0.1:<frontend-port>/health
|
||||
```
|
||||
|
||||
Use local `--resolve`/Host tests only when they do not bypass the identity behavior being tested.
|
||||
|
||||
**Step 4: Prepare the load-balancer route**
|
||||
|
||||
Configure backend/health/TLS according to the surveyed owner procedure, initially disabled or
|
||||
operator-only. Confirm it targets Nginx, never core/Qdrant/Ollama directly.
|
||||
|
||||
**Step 5: Preserve the Aritmolab link contract**
|
||||
|
||||
If the existing sidebar target already equals the final origin/path, leave source unchanged and
|
||||
record proof. Otherwise make the smallest reviewed change, test it in Aritmolab's own test/build
|
||||
system, and commit in that repository before deployment.
|
||||
|
||||
### Task 9: Open ingress and run OIDC acceptance
|
||||
|
||||
**Files:**
|
||||
- Complete: `docs/testing/psd-server-project-b-manual.md`
|
||||
|
||||
**Step 1: Reload Nginx through the established mechanism**
|
||||
|
||||
Run `nginx -t` immediately before reload. Expected: reload succeeds and unrelated virtual hosts
|
||||
remain healthy.
|
||||
|
||||
**Step 2: Enable the final load-balancer route**
|
||||
|
||||
Expected: HTTP redirects to HTTPS; TLS is valid; `/api/auth/oidc/login` redirects to the correct
|
||||
Authentik provider; callback returns to the exact public origin.
|
||||
|
||||
**Step 3: Test ordinary and admin identities**
|
||||
|
||||
Start at Aritmolab, authenticate once, then use the sidebar. Expected: no second credential prompt;
|
||||
ordinary user can use sessions but receives 403 for admin operations; admin has only documented
|
||||
permissions.
|
||||
|
||||
**Step 4: Test no-role and malformed cases**
|
||||
|
||||
An identity with no mapped group authenticates but receives no application role/403. Missing,
|
||||
malformed, indirect, or ambiguous group claims fail closed with generic browser errors and redacted
|
||||
diagnostics. Do not retain raw claims.
|
||||
|
||||
**Step 5: Test logout and restart**
|
||||
|
||||
Verify ThothII logout revokes its own cookie. Document whether the Authentik SSO session remains and
|
||||
therefore allows immediate re-login without credentials; do not claim global logout unless
|
||||
configured and tested. Restart core and verify expected OIDC session behavior.
|
||||
|
||||
### Task 10: Verify PostgreSQL ownership and complete F1-F8
|
||||
|
||||
**Files:**
|
||||
- Create: protected Project B session evidence
|
||||
|
||||
**Step 1: Create sessions under two identities**
|
||||
|
||||
Expected: ordinary users see only their own sessions; cross-user access returns the documented
|
||||
not-found boundary; admin behavior matches `session.read_all/manage_all` permissions.
|
||||
|
||||
**Step 2: Verify RLS with the runtime path**
|
||||
|
||||
Use application/API tests and bounded catalog evidence. Never disable RLS for diagnosis.
|
||||
|
||||
**Step 3: Complete one harmless OIDC PSD session**
|
||||
|
||||
Use the same approved read-only question or another owner-approved one. Complete F1-F8, validate
|
||||
final SQL, resume once, and confirm session/artifacts/decisions are stored in `thoth_sessions`.
|
||||
|
||||
**Step 4: Verify ephemeral boundaries**
|
||||
|
||||
Expected: no chat transcript or SSE stream stored as session artifacts; no vectors in PostgreSQL;
|
||||
Qdrant remains the semantic store.
|
||||
|
||||
### Task 11: Test controlled failures and rollback
|
||||
|
||||
**Files:**
|
||||
- Record: protected rollback evidence
|
||||
|
||||
**Step 1: Test a reversible provider/catalog failure**
|
||||
|
||||
Use a controlled, owner-approved method such as a temporary disabled test credential or test object.
|
||||
Expected: auth diagnostics and browser login fail closed, redacted, then pass after restoration.
|
||||
Never break unrelated Authentik applications.
|
||||
|
||||
**Step 2: Rehearse ingress-first rollback**
|
||||
|
||||
Close the production route, validate Nginx restoration commands, and prove the protected Project A
|
||||
configuration snapshot is complete. A full rollback need not destroy `thoth_sessions`.
|
||||
|
||||
**Step 3: Verify additive database rollback boundary**
|
||||
|
||||
Expected: rollback leaves schema/data intact for evidence and future recovery. No automatic DROP
|
||||
SCHEMA or role deletion.
|
||||
|
||||
### Task 12: Close Project B
|
||||
|
||||
**Files:**
|
||||
- Complete: `docs/testing/evidence/psd-server-project-b-report-template.md`
|
||||
|
||||
**Step 1: Run final diagnostics**
|
||||
|
||||
Run status, doctor, auth check, interactive check when supported, workspace inspect, vector inspect,
|
||||
Pi test, Nginx validation, load-balancer health, Supabase migration/security checks, and sidebar test.
|
||||
|
||||
**Step 2: Remove the Project A temporary endpoint**
|
||||
|
||||
Remove only its load-balancer route, Nginx fragment, and managed certificate reference according to
|
||||
their owners. Validate/reload and prove the hostname no longer routes.
|
||||
|
||||
**Step 3: Complete the human guide and report**
|
||||
|
||||
Every mandatory row must be PASS. Record exact source/image/workspace identities, Authentik object
|
||||
names/IDs, Supabase database plus `thoth_sessions`, public origin, Aritmolab revision, report digest,
|
||||
and `PROJECT_B_PASS` or `PROJECT_B_FAIL`.
|
||||
|
||||
**Step 4: Handle FAIL safely**
|
||||
|
||||
On FAIL, close ingress first. Restore Project A files at the same stable paths, move OIDC auth state
|
||||
to protected evidence, restore the local-auth directory, validate, and start Project A privately.
|
||||
Disable new Authentik objects; do not delete them or drop the session schema automatically.
|
||||
@@ -1,312 +0,0 @@
|
||||
# PSD Server Survey Implementation Plan
|
||||
|
||||
> **For agentic workers:** execute one task at a time, record its completion criteria, and stop at every documented approval boundary.
|
||||
|
||||
**Goal:** Produce a non-mutating, redacted survey of the PSD server that resolves every path, owner, network boundary, credential location, and rollback prerequisite needed by Projects A and B.
|
||||
|
||||
**Architecture:** Collect bounded metadata from the terminal local to the server, retain raw output only in a protected directory, and summarize it in a redacted report. The survey makes no service, file, database, proxy, Authentik, or Git mutation.
|
||||
|
||||
**Tech Stack:** Linux utilities, Docker/Compose inspection, Git, Nginx, OpenSSL, PostgreSQL/Supabase metadata queries, Authentik metadata/API discovery, Aritmolab source inspection.
|
||||
|
||||
---
|
||||
|
||||
## Safety contract
|
||||
|
||||
- Do not run `docker inspect` without a restrictive Go template; its default output can contain secrets.
|
||||
- Do not run `docker compose config` into chat or a public log. Store raw output mode `0600`, then create a redacted derivative.
|
||||
- Do not print process environments, secret-file contents, private keys, cookies, tokens, password hashes, or raw OIDC claims.
|
||||
- Do not reload/restart services, fetch/pull Git, log in interactively, change file modes, or make API mutations.
|
||||
- When a command needs privilege, use the server's approved CyberArk/local-terminal procedure.
|
||||
|
||||
### Task 1: Create the protected survey workspace
|
||||
|
||||
**Files:**
|
||||
- Create: `/var/tmp/thothii-psd-survey.<random>/`
|
||||
- Create: protected `survey-report.md`
|
||||
|
||||
**Step 1: Create a private temporary root**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
umask 0077
|
||||
PSD_SURVEY_ROOT="$(mktemp -d /var/tmp/thothii-psd-survey.XXXXXX)"
|
||||
test -d "$PSD_SURVEY_ROOT"
|
||||
chmod 0700 "$PSD_SURVEY_ROOT"
|
||||
printf '%s\n' "$PSD_SURVEY_ROOT"
|
||||
```
|
||||
|
||||
Expected: one new mode-0700 directory whose exact path is recorded in the operator journal.
|
||||
|
||||
**Step 2: Copy the report template**
|
||||
|
||||
Create `$PSD_SURVEY_ROOT/survey-report.md` from the headings in the design's Common Survey section.
|
||||
Record only findings and references to protected raw files.
|
||||
|
||||
### Task 2: Record host and Docker facts
|
||||
|
||||
**Files:**
|
||||
- Create: `$PSD_SURVEY_ROOT/host.txt`
|
||||
- Create: `$PSD_SURVEY_ROOT/docker-projects.json`
|
||||
- Create: `$PSD_SURVEY_ROOT/docker-containers.txt`
|
||||
|
||||
**Step 1: Record bounded host metadata**
|
||||
|
||||
Run each command with output redirected to `$PSD_SURVEY_ROOT/host.txt`:
|
||||
|
||||
```bash
|
||||
date -u '+%Y-%m-%dT%H:%M:%SZ'
|
||||
uname -a
|
||||
cat /etc/os-release
|
||||
getconf LONG_BIT
|
||||
nproc
|
||||
free -h
|
||||
df -hT
|
||||
docker version
|
||||
docker compose version
|
||||
```
|
||||
|
||||
Expected: no credential content and enough capacity information to judge a parallel source tree and
|
||||
a new five-service stack.
|
||||
|
||||
**Step 2: Record Compose projects and bounded container identity**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
docker compose ls --format json > "$PSD_SURVEY_ROOT/docker-projects.json"
|
||||
docker ps -a --no-trunc --format '{{json .}}' > "$PSD_SURVEY_ROOT/docker-containers.txt"
|
||||
docker network ls --format '{{json .}}' > "$PSD_SURVEY_ROOT/docker-networks.txt"
|
||||
docker volume ls --format '{{json .}}' > "$PSD_SURVEY_ROOT/docker-volumes.txt"
|
||||
```
|
||||
|
||||
Expected: inventory only. Do not inspect full container JSON.
|
||||
|
||||
**Step 3: Identify candidate legacy ThothII containers**
|
||||
|
||||
Use names, images, Compose project labels, published ports, and health from the bounded inventory.
|
||||
For each candidate, query only these templates:
|
||||
|
||||
```bash
|
||||
docker inspect --format '{{.Name}} {{.Config.Image}} {{index .Config.Labels "com.docker.compose.project"}} {{index .Config.Labels "com.docker.compose.project.working_dir"}}' <container>
|
||||
docker inspect --format '{{json .NetworkSettings.Networks}}' <container>
|
||||
docker inspect --format '{{range .Mounts}}{{.Type}} {{.Source}} -> {{.Destination}} rw={{.RW}}{{println}}{{end}}' <container>
|
||||
```
|
||||
|
||||
Expected: exact source/Compose ownership and mounts without environment values.
|
||||
|
||||
### Task 3: Survey the legacy ThothII installation
|
||||
|
||||
**Files:**
|
||||
- Create: `$PSD_SURVEY_ROOT/legacy-thothii.txt`
|
||||
- Create: `$PSD_SURVEY_ROOT/legacy-compose.redacted.yaml`
|
||||
|
||||
**Step 1: Resolve source and operator paths from evidence**
|
||||
|
||||
Do not search broad filesystem roots. Derive paths from Compose labels, systemd units, Nginx
|
||||
upstreams, and known operator documentation. Record uncertainty rather than guessing.
|
||||
|
||||
**Step 2: Record source identity without fetching**
|
||||
|
||||
Run in the identified source tree:
|
||||
|
||||
```bash
|
||||
git status --short --branch
|
||||
git rev-parse HEAD
|
||||
git remote -v
|
||||
git log -5 --oneline --decorate
|
||||
```
|
||||
|
||||
Expected: no mutation. Mark a dirty tree as NO-GO until the owner decides how to preserve it.
|
||||
|
||||
**Step 3: Record lifecycle state with the legacy controller**
|
||||
|
||||
If the old installation has a supported `tht`, run its bounded `status`, `doctor`, `pi status`, and
|
||||
`pi maintenance status` commands. Otherwise record exact read-only Docker health and identify the
|
||||
old lifecycle mechanism. Do not substitute current `tht` against an incompatible descriptor.
|
||||
|
||||
**Step 4: Render and redact Compose safely**
|
||||
|
||||
Store the raw render as mode `0600`. Replace secret-bearing scalar values with `[redacted]` before
|
||||
using the derivative in analysis. Confirm the redacted render still shows service names, networks,
|
||||
ports, volumes, image/build identities, and config file paths.
|
||||
|
||||
### Task 4: Survey Nginx, TLS, and the load balancer
|
||||
|
||||
**Files:**
|
||||
- Create: `$PSD_SURVEY_ROOT/nginx.raw.txt` (protected)
|
||||
- Create: `$PSD_SURVEY_ROOT/nginx-thothii.redacted.txt`
|
||||
- Create: `$PSD_SURVEY_ROOT/tls-metadata.txt`
|
||||
- Create: `$PSD_SURVEY_ROOT/load-balancer.md`
|
||||
|
||||
**Step 1: Validate and capture Nginx without reload**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
sudo nginx -t
|
||||
sudo nginx -T > "$PSD_SURVEY_ROOT/nginx.raw.txt" 2>&1
|
||||
chmod 0600 "$PSD_SURVEY_ROOT/nginx.raw.txt"
|
||||
```
|
||||
|
||||
Expected: configuration test passes. Do not reload Nginx.
|
||||
|
||||
**Step 2: Extract only relevant directives**
|
||||
|
||||
Create the redacted derivative containing the ThothII/Aritmolab `server_name`, `listen`, `location`,
|
||||
`proxy_pass`, `proxy_set_header`, `proxy_buffering`, timeout, certificate path, and include-file
|
||||
directives. Exclude unrelated virtual hosts and all authorization values.
|
||||
|
||||
**Step 3: Record certificate metadata only**
|
||||
|
||||
For each relevant public certificate—not its key—run:
|
||||
|
||||
```bash
|
||||
openssl x509 -in <certificate-path> -noout -subject -issuer -serial -dates -ext subjectAltName
|
||||
```
|
||||
|
||||
Expected: exact SAN/expiry/issuer and the observed generation/renewal mechanism.
|
||||
|
||||
**Step 4: Map the load balancer**
|
||||
|
||||
Record its owner, configuration surface, current Aritmolab backend, health check, TLS boundary,
|
||||
source addresses seen by Nginx, and whether it can enforce a temporary hostname allowlist. Do not
|
||||
create a route. If Sol cannot inspect it, name the human/team required for Project A/B gates.
|
||||
|
||||
### Task 5: Survey Aritmolab and the sidebar integration
|
||||
|
||||
**Files:**
|
||||
- Create: `$PSD_SURVEY_ROOT/aritmolab.md`
|
||||
|
||||
**Step 1: Resolve the Aritmolab source/deployment**
|
||||
|
||||
Use Compose labels, Nginx paths, or the documented service unit. Record repository path, SHA, dirty
|
||||
state, deployment command, container/network identity, and configuration owner.
|
||||
|
||||
**Step 2: Locate the sidebar link**
|
||||
|
||||
Use `rg` in the resolved source tree for the current ThothII URL, label, historical
|
||||
`datamart-builder`, and sidebar/navigation definitions. Record exact files and line numbers.
|
||||
|
||||
**Step 3: Resolve the real public origin**
|
||||
|
||||
The owner reports `aritmolab.policlinicosandonato.com`; historical project state mentions a `.it`
|
||||
origin and `/datamart-builder`. Record the live browser-visible origin and path from deployed
|
||||
configuration. Do not choose between them without evidence.
|
||||
|
||||
### Task 6: Survey Authentik
|
||||
|
||||
**Files:**
|
||||
- Create: `$PSD_SURVEY_ROOT/authentik.md`
|
||||
|
||||
**Step 1: Identify deployment and version**
|
||||
|
||||
Record Authentik containers/services, immutable image reference, version, base URL, database/Redis
|
||||
dependencies, configuration owner, and backup/export procedure. Do not print container environments.
|
||||
|
||||
**Step 2: Locate credential references**
|
||||
|
||||
Record only file/secret-object paths, ownership, mode, and whether the local operator can use them.
|
||||
If no usable administrative/API credential is found, stop and request owner help.
|
||||
|
||||
**Step 3: Inventory relevant objects read-only**
|
||||
|
||||
Using the installed version's API schema or admin interface, list only names/IDs for current
|
||||
Aritmolab applications/providers, authorization flows, property mappings, groups, service accounts,
|
||||
and policies that establish local conventions. Do not retrieve write-only secrets or raw tokens.
|
||||
|
||||
**Step 4: Record version-specific constraints**
|
||||
|
||||
Consult the official documentation matching the installed release for OAuth2/OIDC providers,
|
||||
scope/property mappings, application bindings, blueprints/export, and API permission semantics.
|
||||
Do not copy examples from a newer release without comparing the installed OpenAPI schema.
|
||||
|
||||
### Task 7: Survey Supabase and the PSD DWH
|
||||
|
||||
**Files:**
|
||||
- Create: `$PSD_SURVEY_ROOT/supabase.md`
|
||||
- Create: `$PSD_SURVEY_ROOT/dwh-readonly.txt`
|
||||
|
||||
**Step 1: Map Supabase services without environments**
|
||||
|
||||
Record PostgreSQL, pooler, PostgREST, gateway, and backup components; container networks and local
|
||||
listeners; database name; TLS listener/CA; and the approved direct-connect route from ThothII core.
|
||||
|
||||
**Step 2: Record exposed PostgREST schemas**
|
||||
|
||||
Query only the explicit PostgREST schema setting through its known configuration mechanism. Do not
|
||||
dump the whole environment. Confirm whether `thoth_sessions` already exists or is exposed.
|
||||
|
||||
**Step 3: Inspect schemas and migration state**
|
||||
|
||||
Through an approved administrative connection, run bounded catalog queries for existing schemas,
|
||||
owners, and any `thoth_sessions` tables/migration records. Do not change them.
|
||||
|
||||
**Step 4: Prove the intended DWH runtime identity is read-only**
|
||||
|
||||
Connect using the protected runtime credential mechanism and query `current_database()`,
|
||||
`current_user`, and grants for schema `datawarehouse`. Expected: USAGE/SELECT as required and no
|
||||
INSERT, UPDATE, DELETE, TRUNCATE, REFERENCES, TRIGGER, CREATE, or ownership privileges. Do not run a
|
||||
write probe against clinical tables.
|
||||
|
||||
**Step 5: Identify session-schema roles**
|
||||
|
||||
Record names or naming rules for a future migrator and runtime role. Do not create them. The final
|
||||
design uses the existing database plus schema `thoth_sessions`, never a new database.
|
||||
|
||||
### Task 8: Survey Git workspace and model boundaries
|
||||
|
||||
**Files:**
|
||||
- Create: `$PSD_SURVEY_ROOT/workspace-and-models.md`
|
||||
|
||||
**Step 1: Inspect the workspace remote read-only**
|
||||
|
||||
Record remote URL, branch, fetch credential path, current remote SHA, catalog entry, descriptor
|
||||
schema version, current `supported_transports`, Evidence tree, annotations blob, and deploy-key
|
||||
permissions. Do not push with the server's read-only deployment credential.
|
||||
|
||||
**Step 2: Inspect Pi/LLM policy**
|
||||
|
||||
Record selected provider/model/thinking level and credential references. Use bounded `tht pi status`
|
||||
and `pi doctor` where compatible. Do not output provider keys.
|
||||
|
||||
**Step 3: Check internal semantic capacity**
|
||||
|
||||
Record CPU/GPU availability, free storage, and whether Docker can run the pinned Qdrant and Ollama
|
||||
architectures. Do not pull images or models during the survey.
|
||||
|
||||
### Task 9: Produce the survey decision
|
||||
|
||||
**Files:**
|
||||
- Modify: `$PSD_SURVEY_ROOT/survey-report.md`
|
||||
|
||||
**Step 1: Complete the topology**
|
||||
|
||||
Include exact component owners and flows for user → load balancer → Nginx → Aritmolab/sidebar →
|
||||
ThothII, and core → Supabase DWH/auth session schema/Qdrant/Ollama/LLM/Authentik.
|
||||
|
||||
**Step 2: List exact intended change files**
|
||||
|
||||
Separate files owned by the new ThothII installation, workspace curator, Nginx, load balancer,
|
||||
Aritmolab, Authentik, and Supabase. Mark shared files as owner-gated.
|
||||
|
||||
**Step 3: State the scoped GO or NO-GO decisions**
|
||||
|
||||
State both `SURVEY_GO_PROJECT_A_PRIVATE` and `SURVEY_GO_PROJECT_B`. The private decision requires
|
||||
all paths, permissions, backup owners and rollback boundaries used by Project A; it may defer
|
||||
public-origin, load-balancer, Authentik and Mac REST closeout facts that Project A does not mutate.
|
||||
The Project B decision requires every shared/public fact plus the Mac acceptance, completed
|
||||
observation window and revoked legacy credential. Each NO-GO must name concrete missing facts and
|
||||
the person/system needed to resolve them.
|
||||
|
||||
**Step 4: Hash and retain the report**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
sha256sum "$PSD_SURVEY_ROOT/survey-report.md" > "$PSD_SURVEY_ROOT/survey-report.sha256"
|
||||
sha256sum --check "$PSD_SURVEY_ROOT/survey-report.sha256"
|
||||
```
|
||||
|
||||
Expected: checksum passes. Move the complete mode-0700 survey directory to the approved protected
|
||||
evidence root without changing its contents; record the final path and digest in the journal.
|
||||
@@ -1,909 +0,0 @@
|
||||
# Ristrutturazione delle Evidence — disegno approvato
|
||||
|
||||
**Stato:** approvato il 24 agosto 2026
|
||||
**Sostituisce:** il precedente disegno di Evidence canonica, disponibile nella storia Git
|
||||
**Ambito:** authoring, revisione, pubblicazione, indicizzazione e uso runtime delle Evidence
|
||||
|
||||
## 1. Obiettivo
|
||||
|
||||
Questo disegno introduce un processo semplice e verificabile per trasformare documenti
|
||||
di partenza non necessariamente ben organizzati in Evidence strutturate, revisionabili
|
||||
da una persona e ricercabili in modo efficace da ThothII.
|
||||
|
||||
La soluzione deve:
|
||||
|
||||
1. partire dai testi oggi presenti nel repository del workspace;
|
||||
2. riorganizzarli senza inventare informazioni;
|
||||
3. conservare sorgenti e risultato nello stesso repository Git;
|
||||
4. affidare a Git la revisione e l'approvazione umana;
|
||||
5. indicizzare soltanto le versioni approvate;
|
||||
6. sfruttare Qdrant senza moltiplicare collezioni e componenti;
|
||||
7. inserirsi nel workflow modulare attuale, nel quale Evidence è un modulo autonomo.
|
||||
|
||||
La fonte di verità rimane sempre il repository Git. Qdrant è un indice derivato che può
|
||||
essere ricostruito.
|
||||
|
||||
## 2. Principio guida
|
||||
|
||||
Il processo è diviso in due percorsi distinti.
|
||||
|
||||
- Il **percorso di authoring** prepara e revisiona le Evidence fuori dalle sessioni
|
||||
domanda→SQL.
|
||||
- Il **percorso runtime** è in sola lettura e consulta esclusivamente Evidence già
|
||||
pubblicate.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
S["Testi sorgente"] --> P["Pre-processing"]
|
||||
P --> C["Evidence curate"]
|
||||
C --> R["Revisione Git umana"]
|
||||
R --> M["Merge e attivazione revisione"]
|
||||
M --> I["Indicizzazione atomica"]
|
||||
I --> Q["Qdrant: indice attivo"]
|
||||
Q --> E["Evidence Module"]
|
||||
E --> W["Workflow F1-F8"]
|
||||
```
|
||||
|
||||
Una sessione può proporre una nuova formula o segnalare una lacuna, ma non modifica il
|
||||
repository e non pubblica autonomamente conoscenza.
|
||||
|
||||
## 3. Struttura nel repository del workspace
|
||||
|
||||
Ogni workspace adotta questa struttura sotto la propria directory `evidence/`:
|
||||
|
||||
```text
|
||||
evidence/
|
||||
├── README.md
|
||||
├── source/
|
||||
│ └── ... documenti originali ...
|
||||
├── curated/
|
||||
│ ├── glossary/
|
||||
│ ├── domain/
|
||||
│ ├── enum/
|
||||
│ ├── example/
|
||||
│ ├── mapping/
|
||||
│ ├── normalization/
|
||||
│ ├── formula/
|
||||
│ └── reference/
|
||||
├── manifest.yaml
|
||||
└── evaluation.yaml
|
||||
```
|
||||
|
||||
### 3.1 `source/`
|
||||
|
||||
Contiene i documenti originali. La prima versione accetta file Markdown, testo UTF-8 e
|
||||
file `.sql.md`. Un URL può essere descritto in un documento, ma non viene scaricato né
|
||||
interpretato automaticamente.
|
||||
|
||||
I sorgenti vengono preservati: il pre-processing non li riscrive.
|
||||
|
||||
### 3.2 `curated/`
|
||||
|
||||
Contiene una Evidence Unit per file. Le sottodirectory rendono immediatamente visibile
|
||||
il tipo anche a un lettore umano. Il campo `kind` nel documento resta comunque
|
||||
obbligatorio: la directory aiuta la navigazione, il campo è il contratto macchina.
|
||||
|
||||
### 3.3 `manifest.yaml`
|
||||
|
||||
È gestito dal comando di preparazione e registra:
|
||||
|
||||
- hash di ciascun sorgente;
|
||||
- Evidence Unit derivate da quel sorgente;
|
||||
- identificatori stabili;
|
||||
- versione del processo di preparazione;
|
||||
- unità orfane da controllare.
|
||||
|
||||
Il manifest permette di elaborare soltanto ciò che è cambiato. Non sostituisce Git e
|
||||
non contiene lo stato di approvazione.
|
||||
|
||||
### 3.4 `evaluation.yaml`
|
||||
|
||||
Contiene inizialmente circa venti domande rappresentative e gli identificatori delle
|
||||
Evidence che ci aspettiamo di recuperare. È il controllo minimo per evitare di
|
||||
considerare “migliore” una ricerca soltanto perché sembra sofisticata.
|
||||
|
||||
## 4. Una struttura comune, otto tipi distinti
|
||||
|
||||
La separazione tra tipi non viene eliminata. Ogni documento ha un involucro comune e
|
||||
una parte specializzata determinata da `kind`.
|
||||
|
||||
### 4.1 Campi comuni
|
||||
|
||||
```yaml
|
||||
schema_version: 1
|
||||
id: evidence:fascia-pediatrica
|
||||
title: Fascia pediatrica
|
||||
kind: formula
|
||||
purposes:
|
||||
- sql_generation
|
||||
- schema_linking
|
||||
applies_to:
|
||||
concepts:
|
||||
- fascia pediatrica
|
||||
tables:
|
||||
- clinical.patient
|
||||
columns:
|
||||
- clinical.patient.birth_date
|
||||
language: it
|
||||
provenance:
|
||||
source_file: source/10-domini-clinici/paziente.md
|
||||
source_sha256: sha256:0123456789abcdef...
|
||||
supporting_excerpts:
|
||||
- Per fascia pediatrica si intendono i pazienti con età inferiore a 18 anni.
|
||||
review_items: []
|
||||
```
|
||||
|
||||
I campi hanno ruoli diversi:
|
||||
|
||||
- `kind` dice **che cosa contiene** il documento;
|
||||
- `purposes` dice **in quali attività può essere utile**;
|
||||
- `applies_to` dice **a quali concetti o elementi del database si riferisce**;
|
||||
- `provenance` permette di risalire al testo di origine;
|
||||
- `review_items` rende visibili i dubbi ancora da risolvere.
|
||||
|
||||
Ogni unità contiene da uno a cinque `supporting_excerpts`, ciascuno lungo al massimo
|
||||
1.000 caratteri. Sono citazioni brevi che il validatore deve ritrovare nel sorgente dopo
|
||||
la stessa normalizzazione meccanica. Provano la tracciabilità, non la correttezza
|
||||
semantica: il revisore umano deve comunque verificare che sostengano davvero il
|
||||
contenuto ristrutturato.
|
||||
|
||||
Ogni `review_item` contiene soltanto:
|
||||
|
||||
```yaml
|
||||
code: ambiguous_source_statement
|
||||
message: Il sorgente non chiarisce se l'età sia calcolata alla data di ricovero.
|
||||
field: formula.sql # opzionale
|
||||
```
|
||||
|
||||
Non possiede stato, autore o timestamp. Tutti i review item bloccano la pubblicazione;
|
||||
il curatore corregge il documento e rimuove l'item, mentre Git conserva la storia.
|
||||
|
||||
### 4.2 Tipi iniziali
|
||||
|
||||
| `kind` | Contenuto | Esempio d'uso |
|
||||
| --- | --- | --- |
|
||||
| `glossary` | Definizione, sinonimi e varianti linguistiche | Capire che “ricovero” e “degenza” possono indicare lo stesso concetto |
|
||||
| `domain` | Regole e vincoli del dominio | Interpretare correttamente un episodio clinico |
|
||||
| `enum` | Valori ammessi e loro significato | Tradurre “dimesso” nel codice memorizzato nel DWH |
|
||||
| `example` | Domanda esemplificativa e interpretazione attesa | Riconoscere una formulazione già documentata |
|
||||
| `mapping` | Collegamento fra concetto e schema fisico | Individuare tabella e colonne pertinenti |
|
||||
| `normalization` | Regole di normalizzazione | Uniformare codici, date o varianti testuali |
|
||||
| `formula` | Espressione SQL riutilizzabile e relativi input | Calcolare la fascia pediatrica dalla data di nascita |
|
||||
| `reference` | Un riferimento esterno che è esso stesso contenuto recuperabile | Proporre all'utente il link a una specifica linea guida |
|
||||
|
||||
Un URL che documenta un'altra Evidence appartiene alla sua `provenance`. Un URL che
|
||||
deve essere recuperato come risposta autonoma è invece una Evidence `reference`.
|
||||
|
||||
Ogni Evidence Unit possiede un solo `kind`, scelto in base ai campi strutturati che ne
|
||||
definiscono il contenuto principale. `purposes` e `applies_to` possono invece avere più
|
||||
valori. Quando parti dello stesso sorgente hanno identità e regole di validazione
|
||||
indipendenti, vengono prodotte unità distinte; non si duplica un'unità soltanto perché è
|
||||
utile in più fasi del workflow.
|
||||
|
||||
La classificazione procede dai contenuti più strutturati a quelli più generali:
|
||||
`formula`, `enum`, `mapping`, `normalization`, `glossary`, `domain`, `example` e
|
||||
`reference`. `domain` è il tipo di ripiego per una regola del dominio che non soddisfa
|
||||
uno schema più specifico; `reference` si applica soltanto quando il collegamento deve
|
||||
essere restituito come contenuto autonomo.
|
||||
|
||||
L'identificatore non incorpora il `kind`: una riclassificazione conserva l'ID, mentre
|
||||
una vera divisione semantica assegna nuovi ID alle nuove unità. Un rinominamento
|
||||
univocamente riconoscibile del Source Evidence tramite hash aggiorna la provenienza e
|
||||
conserva gli ID esistenti.
|
||||
|
||||
Un nuovo identificatore usa la forma leggibile `evidence:<slug>`, viene assegnato una
|
||||
sola volta e non viene ricalcolato da titolo, percorso o hash. Le collisioni ricevono un
|
||||
suffisso deterministico. Dopo la prima pubblicazione cambiare ID equivale a ritirare
|
||||
l'unità esistente e crearne una nuova.
|
||||
|
||||
### 4.3 Dati specifici per tipo
|
||||
|
||||
La parte specializzata è una unione discriminata: ogni `kind` ammette e richiede campi
|
||||
diversi. Alcuni esempi:
|
||||
|
||||
```yaml
|
||||
# formula
|
||||
formula:
|
||||
concept: fascia pediatrica
|
||||
columns:
|
||||
- clinical.patient.birth_date
|
||||
sql: |
|
||||
CASE WHEN age < 18 THEN 'pediatrica' ELSE 'adulta' END
|
||||
```
|
||||
|
||||
```yaml
|
||||
# reference
|
||||
reference:
|
||||
url: https://example.org/linea-guida
|
||||
label: Linea guida clinica
|
||||
description: Criteri usati per classificare gli episodi.
|
||||
```
|
||||
|
||||
```yaml
|
||||
# enum
|
||||
enum:
|
||||
column: clinical.episode.discharge_status
|
||||
values:
|
||||
D: dimesso
|
||||
T: trasferito
|
||||
```
|
||||
|
||||
I tipi restano quindi sfruttabili sia in validazione sia in ricerca. Una formula non è
|
||||
un semplice testo etichettato: possiede obbligatoriamente un concetto, le colonne di
|
||||
input e una singola espressione PostgreSQL componibile. `SELECT`, `WITH`, DDL e DML come
|
||||
statement completi non sono Formula Evidence; una query completa documentata appartiene
|
||||
a `example`. Le formule legacy incompatibili diventano review item durante la migrazione.
|
||||
|
||||
## 5. Pre-processing dei testi sorgente
|
||||
|
||||
Il comando concettuale è:
|
||||
|
||||
```text
|
||||
tht evidence prepare <workspace-root>
|
||||
```
|
||||
|
||||
Per l'utente è una sola operazione. Internamente esegue quattro passaggi.
|
||||
|
||||
### 5.1 Estrazione deterministica
|
||||
|
||||
Il sistema:
|
||||
|
||||
- individua i file ammessi in `evidence/source/`;
|
||||
- verifica dimensione, codifica UTF-8 e percorso sicuro;
|
||||
- calcola l'hash del contenuto;
|
||||
- confronta il risultato con `manifest.yaml`;
|
||||
- carica, quando esiste, la precedente versione curata collegata al sorgente.
|
||||
|
||||
Un sorgente invariato non viene nuovamente elaborato.
|
||||
|
||||
Se la `pipeline_version` del manifest non è compatibile con quella installata, il
|
||||
normale `prepare` termina senza scrivere. Il curatore può scegliere esplicitamente
|
||||
`prepare --upgrade`, esclusivamente su un repository pulito, per rielaborare tutti i
|
||||
sorgenti e revisionare il diff completo.
|
||||
|
||||
### 5.2 Normalizzazione deterministica
|
||||
|
||||
Prima del modello vengono normalizzati soltanto aspetti meccanici:
|
||||
|
||||
- terminatori di riga e Unicode;
|
||||
- spaziatura e intestazioni palesemente riconoscibili;
|
||||
- elenchi, tabelle, blocchi SQL e URL;
|
||||
- metadati già esplicitamente presenti;
|
||||
- riferimenti a tabelle e colonne riconoscibili.
|
||||
|
||||
Questa fase non interpreta il significato e non inventa strutture semantiche.
|
||||
|
||||
### 5.3 Una sola ristrutturazione assistita dal modello
|
||||
|
||||
Per ogni sorgente cambiato il modello riceve:
|
||||
|
||||
- il testo normalizzato;
|
||||
- gli otto schemi ammessi;
|
||||
- le regole “non inventare” e “segnala il dubbio”;
|
||||
- le precedenti Evidence curate derivate da quel sorgente;
|
||||
- gli identificatori già assegnati.
|
||||
|
||||
Per un'unità già esistente il modello può restituire soltanto uno degli identificatori
|
||||
ricevuti. Per una nuova unità non propone l'ID: il preparatore assegna una sola volta
|
||||
`evidence:<slug>` e l'eventuale suffisso deterministico. Un identificatore sconosciuto
|
||||
prodotto dal modello rende la risposta non valida.
|
||||
|
||||
Può:
|
||||
|
||||
- assegnare titoli;
|
||||
- classificare il tipo;
|
||||
- separare un sorgente in più Evidence Unit;
|
||||
- riordinare e riscrivere per chiarezza;
|
||||
- compilare campi strutturati con fatti presenti nel sorgente.
|
||||
- citare da uno a cinque brevi estratti del sorgente che sostengono ciascuna unità.
|
||||
|
||||
Non può:
|
||||
|
||||
- fondere automaticamente sorgenti diversi;
|
||||
- aggiungere fatti non documentati;
|
||||
- risolvere silenziosamente un'ambiguità;
|
||||
- cancellare un'unità precedentemente revisionata.
|
||||
|
||||
Se il sorgente esiste ancora ma non sostiene più un'unità precedente, il modello la
|
||||
restituisce come retirement candidate con il `review_item`
|
||||
`source_no_longer_supports_unit`. Il curatore decide se eliminarla o riscriverla; fino a
|
||||
quel momento la pubblicazione resta bloccata.
|
||||
|
||||
Quando una precedente unità viene realmente divisa in più unità autonome, le nuove
|
||||
unità ricevono nuovi ID e la precedente rimane una retirement candidate finché il
|
||||
curatore non la ritira esplicitamente.
|
||||
|
||||
Pi viene usato in modalità non interattiva e senza strumenti di scrittura. È un
|
||||
dettaglio interno del comando, non una nuova tipologia di sessione ThothII.
|
||||
Timeout, uscita non valida o JSON malformato interrompono il comando con un errore
|
||||
attribuito al sorgente. La prima versione non esegue retry automatici.
|
||||
|
||||
### 5.4 Validazione deterministica
|
||||
|
||||
L'output del modello non viene scritto direttamente. Viene prima controllato:
|
||||
|
||||
- schema comune e schema specifico del `kind`;
|
||||
- unicità e stabilità degli identificatori;
|
||||
- appartenenza alle enumerazioni ammesse;
|
||||
- esistenza e hash del sorgente;
|
||||
- presenza nel sorgente normalizzato di ogni `supporting_excerpt`;
|
||||
- correttezza sintattica di URL, tabelle, colonne e SQL dove applicabile;
|
||||
- assenza di credenziali;
|
||||
- coerenza tra directory e `kind`;
|
||||
- assenza di collegamenti a sorgenti diversi nella stessa unità.
|
||||
|
||||
Esistono tre esiti.
|
||||
|
||||
| Esito | Comportamento |
|
||||
| --- | --- |
|
||||
| Valido | Il documento è pronto per la revisione Git |
|
||||
| Valido con dubbi | Il documento viene scritto con `review_items`; non è indicizzabile |
|
||||
| Non valido | Il documento non è pubblicabile e il rapporto spiega l'errore |
|
||||
|
||||
Un dubbio reale può essere mantenuto soltanto se il revisore lo trasforma in una
|
||||
limitazione esplicita del contenuto e svuota `review_items`.
|
||||
|
||||
### 5.5 Applicazione atomica
|
||||
|
||||
Tutti gli output dei sorgenti cambiati vengono costruiti e validati in un'area
|
||||
temporanea. Soltanto quando l'intero batch è valido, il comando sostituisce insieme i
|
||||
documenti interessati e `manifest.yaml`. Un singolo errore lascia il worktree invariato
|
||||
e il rapporto limitato elenca tutti i problemi rilevati. Non esiste successo parziale.
|
||||
|
||||
## 6. Aggiornamenti incrementali e revisione delle correzioni umane
|
||||
|
||||
La precedente versione curata è un input, non un file usa-e-getta. Il modello deve
|
||||
proporre una modifica minima senza ricominciare da zero, ma questa istruzione non viene
|
||||
presentata come una garanzia semantica. La garanzia è Git: la versione precedente resta
|
||||
recuperabile, ogni variazione è visibile nel diff e nessuna proposta diventa Published
|
||||
Evidence senza una nuova revisione umana.
|
||||
|
||||
Il comando:
|
||||
|
||||
- si rifiuta di operare se `evidence/curated/` o `evidence/manifest.yaml` contengono
|
||||
modifiche Git non salvate;
|
||||
- mantiene gli ID associati a contenuti che rappresentano ancora la stessa unità;
|
||||
- riconosce come rinominato un sorgente nuovo che corrisponde univocamente all'hash di
|
||||
un sorgente rimosso e ne aggiorna la provenienza senza cambiare gli ID;
|
||||
- mostra come diff le variazioni proposte;
|
||||
- non modifica i file derivati da sorgenti invariati;
|
||||
- segnala come orfana un'unità il cui sorgente è stato rimosso;
|
||||
- non elimina mai automaticamente un'unità orfana;
|
||||
- blocca la pubblicazione finché ogni unità orfana non viene eliminata, ricollegata o
|
||||
ricondotta a un sorgente ripristinato.
|
||||
|
||||
Git fornisce confronto, revisione, cronologia e recupero. Non viene introdotto un
|
||||
database di authoring parallelo.
|
||||
|
||||
Orfani e retirement candidate vengono risolti senza modificare manualmente il manifest:
|
||||
|
||||
```text
|
||||
tht evidence resolve <evidence-id> --retire
|
||||
tht evidence resolve <evidence-id> --source <source-path>
|
||||
```
|
||||
|
||||
Le due azioni sono mutuamente esclusive, richiedono un worktree pulito e aggiornano
|
||||
atomicamente file curato e manifest. `--retire` rimuove l'unità dal corpus di authoring;
|
||||
`--source` aggiorna provenienza e hash soltanto verso un sorgente esistente. Entrambe
|
||||
lasciano un diff Git recuperabile, senza commit o pubblicazione automatica. Se l'unità
|
||||
deve essere riscritta, il curatore modifica invece il documento e poi esegue
|
||||
`validate`.
|
||||
|
||||
## 7. Revisione e pubblicazione
|
||||
|
||||
Il flusso di pubblicazione è:
|
||||
|
||||
```text
|
||||
prepare → revisione Git → validate → merge → attivazione workspace
|
||||
→ preprocess evidence candidata → evaluate candidata
|
||||
→ pubblicazione atomica della generazione Qdrant
|
||||
```
|
||||
|
||||
### 7.1 Approvazione umana
|
||||
|
||||
L'approvazione coincide con il normale processo Git del repository del workspace:
|
||||
|
||||
1. il curatore esegue `prepare` in un clone di authoring;
|
||||
2. legge i documenti e il diff;
|
||||
3. corregge i contenuti;
|
||||
4. esegue `tht evidence validate`;
|
||||
5. apre o approva la pull request;
|
||||
6. esegue il merge.
|
||||
|
||||
La prima versione non crea automaticamente branch, commit o pull request.
|
||||
|
||||
### 7.2 Quando un documento diventa Published Evidence
|
||||
|
||||
Una Curated Evidence diventa Published Evidence soltanto quando:
|
||||
|
||||
- appartiene a una revisione Git pulita che ha superato il processo umano di revisione;
|
||||
- la revisione è stata attivata dal registry di ThothII;
|
||||
- non contiene `review_items` irrisolti;
|
||||
- il manifest non contiene unità orfane;
|
||||
- l'intero corpus supera la validazione;
|
||||
- la Candidate Evidence Generation supera il gate top-10;
|
||||
- la generazione Qdrant viene quindi pubblicata atomicamente.
|
||||
|
||||
Il runtime non tenta di ricostruire come sia avvenuta l'approvazione Git e il manifest
|
||||
non contiene un flag `approved`. Il confine verificabile di pubblicazione è la
|
||||
combinazione di revisione attiva, validazione superata e generazione Evidence attiva.
|
||||
|
||||
Il descriptor filesystem deve indicizzare solo `curated/**/*.md`. I sorgenti e i file
|
||||
di supporto restano materializzati per tracciabilità, ma non entrano nell'indice.
|
||||
|
||||
## 8. Indicizzazione e generazioni
|
||||
|
||||
L'indicizzazione continua a usare il meccanismo già implementato dal modulo Evidence:
|
||||
|
||||
1. legge la radice materializzata della revisione Git attiva;
|
||||
2. valida nuovamente tutte le Evidence;
|
||||
3. costruisce Evidence Fragment secondo sezioni semantiche;
|
||||
4. verifica il vettore dense predefinito già usato da Schema e Memory;
|
||||
5. aggiunge in modo non distruttivo il vettore sparse `bm25` se manca;
|
||||
6. genera le rappresentazioni dense degli Evidence Fragment;
|
||||
7. chiede a Qdrant di generare per gli stessi frammenti la rappresentazione lessicale
|
||||
BM25;
|
||||
8. carica i punti con la nuova `vector_generation`;
|
||||
9. verifica manifest, conteggi e leggibilità;
|
||||
10. rende attiva la nuova generazione;
|
||||
11. conserva le generazioni precedenti previste dalla policy.
|
||||
|
||||
Se uno dei passaggi fallisce, la generazione precedente rimane attiva. I punti caricati
|
||||
parzialmente vengono compensati secondo il meccanismo transazionale già esistente.
|
||||
L'eventuale configurazione `bm25` già aggiunta rimane: è compatibile con i punti dense
|
||||
esistenti e non richiede rollback. Se `bm25` esiste con una configurazione diversa da
|
||||
`modifier: idf`, la procedura fallisce senza modificarla.
|
||||
|
||||
L'upgrade non ricrea la collezione. I record `schema_table`, `schema_column`, `memory` e
|
||||
`solved_question` restano invariati e continuano a usare il vettore dense predefinito.
|
||||
Soltanto gli Evidence Fragment ricevono anche `bm25`. Durante la finestra fra aggiunta
|
||||
del vettore e pubblicazione della prima candidata ibrida, Evidence è `unavailable`, ma
|
||||
Schema e Memory continuano a funzionare.
|
||||
|
||||
## 9. Qdrant spiegato senza presupporre conoscenze vettoriali
|
||||
|
||||
### 9.1 L'analogia della biblioteca
|
||||
|
||||
Si può immaginare Qdrant come il catalogo di una biblioteca.
|
||||
|
||||
- Le **Evidence Unit** sono i documenti completi conservati negli scaffali Git.
|
||||
- Gli **Evidence Fragment** sono le schede del catalogo relative alle singole sezioni.
|
||||
- I **vettori** sono rappresentazioni numeriche usate per confrontare una domanda con
|
||||
quelle schede.
|
||||
- Il **payload** è l'insieme delle etichette leggibili: tipo, scopo, tabelle, colonne,
|
||||
revisione e documento di origine.
|
||||
|
||||
Qdrant non decide se una Evidence è vera e non sostituisce il documento. Aiuta soltanto
|
||||
a trovare rapidamente le schede più promettenti.
|
||||
|
||||
### 9.2 Ricerca per significato: vettore dense
|
||||
|
||||
La rappresentazione dense descrive il significato generale di una frase. Permette, per
|
||||
esempio, di avvicinare “pazienti minorenni” a “fascia pediatrica” anche quando le parole
|
||||
non coincidono.
|
||||
|
||||
È utile per il linguaggio naturale, ma può essere meno precisa con codici, acronimi,
|
||||
nomi di colonne e formule.
|
||||
|
||||
### 9.3 Ricerca per parole e identificatori: BM25 sparse
|
||||
|
||||
La rappresentazione sparse conserva il peso delle parole presenti. È adatta a termini
|
||||
come `ICD-10`, `discharge_status`, `ADT`, un valore enum o un nome esatto di colonna.
|
||||
|
||||
Qdrant 1.18.2 può generare questa rappresentazione direttamente sul server usando
|
||||
`qdrant/bm25`; per il corpus italiano si passa `language: italian` sia durante il
|
||||
caricamento sia durante la ricerca. Non serve aggiungere FastEmbed o un nuovo servizio.
|
||||
|
||||
Un test L0 avvia esattamente l'immagine Qdrant dichiarata da `compose.yaml`, crea una
|
||||
collezione temporanea, indicizza due testi italiani mediante `qdrant/bm25`, verifica una
|
||||
ricerca lessicale e infine elimina la collezione. Questo rende controllabile la capacità
|
||||
locale richiesta prima di qualunque migrazione reale. Se la prova fallisce non esiste
|
||||
un fallback silenzioso a un motore diverso.
|
||||
|
||||
Il nome “sparse” significa soltanto che, tra moltissime parole possibili, ogni testo ne
|
||||
usa poche. Qdrant mantiene anche l'IDF: una parola rara pesa più di una parola presente
|
||||
quasi ovunque.
|
||||
|
||||
### 9.4 Perché combinarle
|
||||
|
||||
Una domanda può richiedere contemporaneamente comprensione e precisione lessicale:
|
||||
|
||||
> “Qual è la formula per distinguere la fascia pediatrica usando
|
||||
> `patient.birth_date`?”
|
||||
|
||||
La ricerca dense riconosce il concetto; BM25 riconosce con forza “formula” e il nome
|
||||
della colonna. Qdrant esegue entrambe e produce due graduatorie.
|
||||
|
||||
### 9.5 Reciprocal Rank Fusion
|
||||
|
||||
Reciprocal Rank Fusion, o RRF, combina le due graduatorie usando la posizione dei
|
||||
risultati invece di confrontare direttamente punteggi di natura diversa.
|
||||
|
||||
In termini pratici:
|
||||
|
||||
- un documento alto in entrambe le liste sale;
|
||||
- un documento molto forte in una sola lista può comunque emergere;
|
||||
- non occorre inventare una conversione fragile fra “similarità semantica” e “punteggio
|
||||
delle parole”.
|
||||
|
||||
Si parte con i pesi predefiniti. Pesi diversi saranno introdotti soltanto se
|
||||
`evaluation.yaml` dimostrerà un miglioramento.
|
||||
|
||||
### 9.6 Il ruolo dei metadati
|
||||
|
||||
Ogni punto Qdrant conserva almeno:
|
||||
|
||||
```text
|
||||
workspace_id
|
||||
workspace_revision
|
||||
vector_generation
|
||||
record_kind
|
||||
evidence_id
|
||||
evidence_kind
|
||||
purposes
|
||||
concepts
|
||||
tables
|
||||
columns
|
||||
language
|
||||
source_file
|
||||
source_sha256
|
||||
fragment_ordinal
|
||||
```
|
||||
|
||||
I metadati hanno due usi:
|
||||
|
||||
- workspace, revisione, generazione e purpose sono filtri obbligatori;
|
||||
- tipo, concetti, tabelle e colonne diventano filtri soltanto quando il chiamante li
|
||||
dichiara vincolanti; altrimenti contribuiscono al testo della query e alla spiegazione
|
||||
del risultato.
|
||||
|
||||
La prima versione non aggiunge bonus automatici per `kind` o `applies_to`. Durante la
|
||||
generazione SQL una Formula Evidence viene imposta soltanto quando il workflow richiede
|
||||
esplicitamente `kind=formula`; negli altri casi dense, BM25 e RRF determinano l'ordine.
|
||||
|
||||
Quando concetti, tabelle o colonne non sono vincoli, il modulo costruisce un solo testo
|
||||
deterministico, identico per dense e BM25:
|
||||
|
||||
```text
|
||||
Domanda: <domanda originale>
|
||||
Concetti: <valori deduplicati e ordinati>
|
||||
Tabelle: <valori deduplicati e ordinati>
|
||||
Colonne: <valori deduplicati e ordinati>
|
||||
```
|
||||
|
||||
Le righe vuote sono omesse. La domanda conserva formulazione e ordine originali; il
|
||||
renderer applica soltanto Unicode NFC, converte CRLF e CR in `\n`, rimuove gli spazi
|
||||
esterni e rifiuta una domanda vuota. Non cambia maiuscole, punteggiatura o spazi interni.
|
||||
Ai valori contestuali applica NFC e `strip`, elimina stringhe vuote e duplicati esatti e
|
||||
li ordina per valore Unicode senza `lower()` o `casefold()`: gli identificatori
|
||||
PostgreSQL quotati possono essere sensibili alle maiuscole. In questo modo due richieste
|
||||
equivalenti non cambiano per effetto dell'ordine occasionale dei metadati.
|
||||
|
||||
### 9.7 Perché non creare una collezione per tipo
|
||||
|
||||
Una domanda spesso attraversa più tipi: una formula può dipendere da un mapping, da un
|
||||
enum e da una regola di dominio. Collezioni separate richiederebbero più interrogazioni,
|
||||
fusione applicativa e più operazioni di manutenzione.
|
||||
|
||||
La soluzione usa la collezione semantica già posseduta dal workspace, conserva il suo
|
||||
vettore dense predefinito e aggiunge soltanto il vettore sparse denominato `bm25`. I
|
||||
payload indicizzati distinguono i tipi. È più semplice, evita di ricostruire Schema e
|
||||
Memory e permette a Qdrant di eseguire ricerca ibrida e filtri nella stessa Query API.
|
||||
|
||||
### 9.8 Perché indicizzare frammenti ma restituire unità
|
||||
|
||||
Un documento lungo può contenere sezioni diverse. Un unico vettore ne diluirebbe il
|
||||
significato; frammenti arbitrari di lunghezza fissa spezzerebbero invece formule o
|
||||
regole.
|
||||
|
||||
La divisione segue intestazioni, campi tipizzati e confini di paragrafo. Non divide mai
|
||||
una formula, una coppia valore/significato, un mapping, una regola o un URL. Se uno di
|
||||
questi elementi atomici supera da solo `max_chunk_chars`, la preparazione aggiunge il
|
||||
Review item stabile `atomic_content_too_large` e blocca la pubblicazione: non usa un
|
||||
taglio a dimensione fissa che ne altererebbe il significato. Il limite è quello già
|
||||
presente nella configurazione degli embeddings, pari per default a 4.000 caratteri, e
|
||||
si applica all'intero testo reso che sarà inviato all'embedder, incluse etichette e
|
||||
metadati testuali. Non viene introdotta una seconda impostazione. Qdrant trova i
|
||||
frammenti, poi l'Evidence Module li raggruppa per `evidence_id` e restituisce un solo
|
||||
Evidence Result con i migliori estratti, provenienza, citazione e riferimento al
|
||||
documento completo. Il contenuto completo viene risolto soltanto quando il workflow ne
|
||||
ha bisogno.
|
||||
|
||||
### 9.9 Cosa non introduciamo nella prima versione
|
||||
|
||||
- una collezione per ogni tipo;
|
||||
- ColBERT o multivettori late-interaction;
|
||||
- un reranker basato su un altro modello;
|
||||
- pesi RRF regolati a mano senza misurazioni;
|
||||
- un servizio separato per BM25;
|
||||
- ricerca automatica sul web.
|
||||
|
||||
Queste possibilità rimangono future ottimizzazioni, non prerequisiti.
|
||||
|
||||
Riferimenti tecnici ufficiali:
|
||||
|
||||
- [Qdrant: Text Search](https://qdrant.tech/documentation/search/text-search/)
|
||||
- [Qdrant: server-side BM25](https://qdrant.tech/documentation/inference/inference-bm25/)
|
||||
- [Qdrant: Hybrid Queries e RRF](https://qdrant.tech/documentation/search/hybrid-queries/)
|
||||
- [Qdrant: aggiornamento dello schema dei vettori](https://qdrant.tech/documentation/manage-data/collections/#update-vector-schema)
|
||||
- [Qdrant: payload indexing](https://qdrant.tech/documentation/manage-data/indexing/)
|
||||
- [Qdrant: multitenancy](https://qdrant.tech/documentation/manage-data/multitenancy/)
|
||||
|
||||
## 10. Contratto di ricerca del modulo Evidence
|
||||
|
||||
Il workflow non costruisce query Qdrant. Usa una sola interfaccia concettuale:
|
||||
|
||||
```python
|
||||
search(
|
||||
query: str,
|
||||
purpose: EvidencePurpose,
|
||||
context: EvidenceSearchContext,
|
||||
) -> EvidenceSearchOutcome
|
||||
```
|
||||
|
||||
`EvidenceSearchContext` può specificare tabelle, colonne e concetti da aggiungere alla
|
||||
query, oltre a vincoli espliciti su tipo, tabelle, colonne o concetti.
|
||||
|
||||
Il modulo rende domanda e contesto una sola volta nel formato `Domanda`, `Concetti`,
|
||||
`Tabelle`, `Colonne` definito sopra e passa esattamente quel testo sia all'embedder dense
|
||||
sia a `qdrant/bm25`.
|
||||
|
||||
`EvidenceSearchOutcome` distingue due stati:
|
||||
|
||||
- `available`, con la generazione interrogata e zero o più `EvidenceResult`;
|
||||
- `unavailable`, senza risultati e con un codice di errore stabile e un messaggio
|
||||
limitato.
|
||||
|
||||
Una lista vuota nello stato `available` significa che la ricerca ha funzionato ma non
|
||||
ha trovato corrispondenze. Non equivale a un errore tecnico.
|
||||
|
||||
Il modulo Evidence possiede interamente:
|
||||
|
||||
- generazione della query dense;
|
||||
- query BM25 con lingua coerente;
|
||||
- filtri su workspace, revisione, generazione e purpose;
|
||||
- RRF;
|
||||
- vincoli espliciti su `kind` e `applies_to`;
|
||||
- raggruppamento dei frammenti;
|
||||
- risoluzione di provenienza e citazioni;
|
||||
- controllo della revisione attiva.
|
||||
|
||||
Il workflow riceve candidati spiegabili, mai verità automatiche.
|
||||
|
||||
## 11. Inserimento nel workflow modulare ThothII
|
||||
|
||||
Evidence rimane un modulo autonomo con due responsabilità pubbliche.
|
||||
|
||||
### 11.1 Authoring
|
||||
|
||||
```text
|
||||
prepare → validate → evaluate
|
||||
```
|
||||
|
||||
Questa superficie è usata dal curatore e non dalle sessioni.
|
||||
|
||||
### 11.2 Runtime
|
||||
|
||||
```text
|
||||
search → resolve citation → project into session
|
||||
```
|
||||
|
||||
Evidence è un contributore degli stage esistenti, non uno stage aggiuntivo. Non emette
|
||||
decisioni, non scrive gli artifact canonici e non modifica il ledger o lo stato del
|
||||
workflow. Lo stage chiamante decide come usare i candidati restituiti.
|
||||
|
||||
L'integrazione usa l'identità semantica dello stage, non il display code:
|
||||
|
||||
| Stage semantico | Display code attuale | Evidence purpose |
|
||||
|---|---:|---|
|
||||
| `clarification` | F1 | `disambiguation` |
|
||||
| `rewriting` | F3 | `rewriting` |
|
||||
| `schema_linking` | F4 | `schema_linking` |
|
||||
| `cte` | F6 | `sql_generation` |
|
||||
| `final_sql` | F7 | `sql_generation` |
|
||||
|
||||
Lo stage `memory` (F2) usa il Memory Module. Lo stage `synthesis` (F5) verifica e
|
||||
riassume lo schema linking già approvato e non avvia una nuova ricerca Evidence.
|
||||
|
||||
Ogni stage elencato esegue una ricerca indipendente con gli input disponibili in quel
|
||||
momento. In particolare `cte` usa domanda riscritta e schema approvato, mentre
|
||||
`final_sql` aggiunge il piano CTE approvato. La prima versione non introduce una cache
|
||||
condivisa fra stage.
|
||||
|
||||
Il chiamante conserva nella sessione una Evidence receipt con stage, purpose,
|
||||
generazione e ID restituiti. Il testo non viene copiato: rimane nel repository del
|
||||
workspace e viene risolto attraverso la provenienza della Published Evidence.
|
||||
|
||||
Gli stage passano `purpose` e contesto al modulo, ma non conoscono collezioni, nomi di
|
||||
vettori, generazioni o sintassi Qdrant.
|
||||
|
||||
Le istruzioni Pi relative alla consultazione delle Evidence vengono spostate in
|
||||
frammenti del modulo Evidence e poi proiettate nel `SKILL.md` generato, seguendo il
|
||||
meccanismo modulare già usato da Disambiguation e Memory.
|
||||
|
||||
## 12. Formule
|
||||
|
||||
Le formule approvate oggi presenti nello store `formulas/*.sql.md` vengono convertite in
|
||||
Evidence `kind: formula`. Dopo la migrazione non esistono due archivi runtime.
|
||||
|
||||
Una nuova formula scoperta in F4 segue invece questo percorso:
|
||||
|
||||
```text
|
||||
sessione → Formula proposal nell'artefatto di sessione
|
||||
→ importazione di manutenzione
|
||||
→ Curated Evidence formula
|
||||
→ revisione Git
|
||||
→ Published Evidence
|
||||
```
|
||||
|
||||
Le decisioni `concept_formula_approved` e `concept_formula_rejected` continuano a
|
||||
descrivere la scelta fatta nella singola sessione. Non equivalgono alla pubblicazione
|
||||
globale nel workspace.
|
||||
|
||||
## 13. Comportamento in caso di errore
|
||||
|
||||
### 13.1 Durante l'authoring
|
||||
|
||||
- un file non UTF-8, troppo grande o strutturalmente invalido produce un errore chiaro;
|
||||
- un dubbio semantico produce un `review_item`;
|
||||
- un albero Git sporco impedisce la scrittura di nuove proposte;
|
||||
- un sorgente rimosso produce un'unità orfana, non una cancellazione, e blocca la
|
||||
pubblicazione finché il curatore non la risolve.
|
||||
|
||||
### 13.2 Durante l'indicizzazione
|
||||
|
||||
- la nuova generazione viene preparata senza toccare quella attiva;
|
||||
- la generazione candidata viene interrogata esplicitamente per la valutazione senza
|
||||
renderla visibile alle sessioni;
|
||||
- una valutazione fallita lascia inattiva la candidata;
|
||||
- un caricamento o una verifica falliti non cambiano il puntatore attivo;
|
||||
- i dati parziali vengono rimossi quando possibile e comunque non sono leggibili dal
|
||||
runtime perché manca l'attivazione.
|
||||
|
||||
Poiché la revisione del workspace viene attivata prima di costruire la candidata, il
|
||||
runtime può attraversare una finestra di manutenzione in cui la vecchia generazione non
|
||||
corrisponde alla revisione. In questa finestra la ricerca Evidence è `unavailable` e
|
||||
blocca lo stage chiamante. La prima versione accetta questa degradazione fail-closed
|
||||
invece di introdurre una transazione distribuita fra Git, registry e Qdrant. Se il gate
|
||||
fallisce, l'operatore corregge il corpus oppure ripristina esplicitamente la revisione
|
||||
precedente.
|
||||
|
||||
### 13.3 Durante una sessione
|
||||
|
||||
Una ricerca `available` senza corrispondenze produce una Evidence receipt vuota, viene
|
||||
mostrata come tale e non impedisce allo stage di continuare.
|
||||
|
||||
Se Qdrant, il corpus attivo o la revisione attesa non sono disponibili:
|
||||
|
||||
- l'outcome è `unavailable`, non una lista vuota valida;
|
||||
- viene restituito un codice stabile con un messaggio limitato;
|
||||
- lo stage chiamante resta bloccato e può essere ritentato;
|
||||
- non vengono usate revisioni precedenti;
|
||||
- non viene ripetuta la ricerca con un purpose diverso.
|
||||
|
||||
Questo comportamento è fail-closed: un'assenza reale di corrispondenze non ferma il
|
||||
workflow, mentre un guasto non viene mascherato come assenza di conoscenza.
|
||||
|
||||
## 14. Valutazione minima
|
||||
|
||||
`tht evidence evaluate` esegue le domande in `evaluation.yaml` contro una generazione
|
||||
indicata esplicitamente oppure, per il monitoraggio ordinario, contro l'indice attivo.
|
||||
Riporta almeno:
|
||||
|
||||
- quante domande hanno trovato una Evidence attesa nei primi 5 e nei primi 10 risultati;
|
||||
- quali tipi attesi sono mancati;
|
||||
- quali query non hanno prodotto risultati;
|
||||
- per ogni Evidence attesa, la posizione nella graduatoria dense, BM25 e fused;
|
||||
- revisione Git, generazione e configurazione di ricerca usate.
|
||||
|
||||
Il file classifica ogni domanda come `lexical`, `semantic` o `mixed` e contiene almeno
|
||||
un caso per profilo. L'evaluator esegue i due rami anche separatamente per renderli
|
||||
diagnosticabili, oltre alla ricerca ibrida usata dal runtime. La prima baseline deve
|
||||
essere salvata prima di regolare pesi o introdurre altri modelli. Il comando non
|
||||
modifica l'indice. La valutazione supera il gate minimo soltanto quando ogni query trova
|
||||
almeno una delle Evidence attese nei primi dieci risultati fused. Le posizioni dei
|
||||
singoli rami e `hit@5` restano informative e non bloccano la pubblicazione.
|
||||
|
||||
## 15. Comandi e responsabilità
|
||||
|
||||
| Comando | Dove opera | Scrive |
|
||||
| --- | --- | --- |
|
||||
| `tht evidence prepare <workspace-root> [--upgrade]` | clone Git di authoring | `curated/`, `manifest.yaml` |
|
||||
| `tht evidence validate <workspace-root>` | clone Git o CI | nulla |
|
||||
| `tht evidence resolve <id> (--retire | --source <path>)` | clone Git di authoring | unità interessata, `manifest.yaml` |
|
||||
| `tht evidence evaluate ... [--generation <id>]` | generazione candidata o attiva | solo rapporto su stdout/JSON |
|
||||
| `tht ... workspace preprocess evidence` | installazione/runtime | generazione candidata, poi attiva soltanto dopo il gate |
|
||||
|
||||
`prepare` non crea commit. `preprocess evidence` non modifica il repository Git.
|
||||
|
||||
## 16. Migrazione iniziale del workspace PSD
|
||||
|
||||
Il corpus attuale comprende 36 file Markdown organizzati in glossario, domini clinici,
|
||||
enum, esempi NLQ, mapping e normalizzazione. La migrazione avviene così:
|
||||
|
||||
1. spostare gli originali sotto `evidence/source/`, conservandone la gerarchia;
|
||||
2. eseguire `prepare` e generare `curated/`;
|
||||
3. revisionare tutte le unità e risolvere i `review_items`;
|
||||
4. importare eventuali formule approvate come `kind: formula`;
|
||||
5. compilare circa venti query in `evaluation.yaml`;
|
||||
6. configurare il descriptor con `patterns: ["curated/**/*.md"]`;
|
||||
7. validare, fare merge e attivare la revisione in una finestra di manutenzione;
|
||||
8. registrare conteggi e ID campione di Schema e Memory;
|
||||
9. eseguire `preprocess evidence`, che aggiunge `bm25` senza ricreare la collezione e
|
||||
costruisce la generazione candidata;
|
||||
10. verificare che conteggi, ID campione e ricerche dense di Schema e Memory siano
|
||||
invariati;
|
||||
11. valutare la candidata e pubblicarla soltanto se supera il gate top-10;
|
||||
12. salvare la baseline e svolgere una verifica umana degli stage `clarification`,
|
||||
`rewriting`, `schema_linking`, `cte` e `final_sql`.
|
||||
|
||||
Non serve mantenere v1 e v2 attivi contemporaneamente nel runtime: Git conserva la
|
||||
vecchia revisione e il meccanismo delle generazioni conserva il rollback dell'indice.
|
||||
|
||||
## 17. Criteri di accettazione
|
||||
|
||||
La prima versione è completa quando:
|
||||
|
||||
1. un sorgente poco strutturato produce una o più unità tipizzate senza perdere la
|
||||
provenienza;
|
||||
2. sorgenti invariati sono un no-op;
|
||||
3. ogni modifica proposta a contenuti revisionati è recuperabile e visibile nel diff
|
||||
Git prima della pubblicazione;
|
||||
4. ogni unità cita brevi estratti verificabili del proprio sorgente;
|
||||
5. gli ID nuovi sono assegnati dal codice e il modello può soltanto riutilizzare ID
|
||||
precedenti esplicitamente forniti;
|
||||
6. il batch di preparazione è tutto-o-niente e non esegue retry automatici;
|
||||
7. un `review_item` impedisce l'indicizzazione;
|
||||
8. un'unità orfana blocca la pubblicazione finché non viene risolta;
|
||||
9. un'unità non più sostenuta dal proprio sorgente diventa una retirement candidate e
|
||||
blocca la pubblicazione;
|
||||
10. il comando `resolve` ritira o ricollega un'unità con un diff Git recuperabile;
|
||||
11. tutte le otto varianti hanno validazione specifica e un solo `kind` primario;
|
||||
12. una riclassificazione conserva l'ID e un rinominamento univoco del sorgente conserva
|
||||
gli ID delle unità collegate;
|
||||
13. ogni ID usa `evidence:<slug>`, non viene ricalcolato automaticamente e non contiene
|
||||
il kind;
|
||||
14. ogni Formula Evidence contiene una sola espressione PostgreSQL componibile;
|
||||
15. le formule approvate sono ricercate tramite lo stesso modulo delle altre Evidence;
|
||||
16. soltanto `curated/**/*.md` entra nel corpus runtime;
|
||||
17. la collezione Qdrant conserva il dense predefinito e aggiunge `bm25` con IDF senza
|
||||
ricostruzione distruttiva;
|
||||
18. la Query API esegue i due prefetch e la fusione RRF;
|
||||
19. risultati di frammenti della stessa unità diventano un solo Evidence Result con
|
||||
estratti e riferimento al documento completo;
|
||||
20. una ricerca disponibile può restituire zero Evidence, mentre una revisione o
|
||||
generazione non corrispondente produce `unavailable` e blocca lo stage;
|
||||
21. ogni ricerca applica il purpose come filtro obbligatorio e non usa bonus impliciti
|
||||
per kind o ambito;
|
||||
22. la generazione candidata diventa attiva soltanto quando ogni query recupera almeno
|
||||
una Evidence attesa nei primi dieci risultati; il rapporto include anche `hit@5`;
|
||||
23. i cinque stage mappati interrogano Evidence indipendentemente, mentre `memory` e
|
||||
`synthesis` non lo invocano;
|
||||
24. ogni ricerca disponibile conserva una ricevuta minima senza duplicare il testo;
|
||||
25. una sessione completa riprende dopo la finestra fail-closed mediante retry o
|
||||
rollback esplicito;
|
||||
26. conteggi, ID campione e ricerche dense dimostrano che Schema e Memory non cambiano
|
||||
durante l'upgrade;
|
||||
27. una prova L0 dimostra `qdrant/bm25` sull'immagine locale effettivamente dichiarata;
|
||||
28. dense e BM25 ricevono lo stesso testo di query deterministico;
|
||||
29. nessun elemento atomico viene spezzato per rispettare la dimensione dei frammenti;
|
||||
30. la valutazione copre casi lessicali, semantici e misti e mostra separatamente i tre
|
||||
ranking;
|
||||
31. il testo reso di ogni frammento rispetta il solo `max_chunk_chars` esistente;
|
||||
32. la query conserva maiuscole, punteggiatura e spazi interni e non altera gli
|
||||
identificatori PostgreSQL sensibili alle maiuscole;
|
||||
33. documentazione e comandi descrivono lo stesso contratto.
|
||||
|
||||
## 18. Decisioni rinviate
|
||||
|
||||
Saranno considerate soltanto dopo la baseline:
|
||||
|
||||
- pesi RRF diversi da quelli predefiniti;
|
||||
- reranking;
|
||||
- ColBERT o multivettori;
|
||||
- acquisizione automatica di PDF, Word, HTML o pagine web;
|
||||
- creazione automatica di branch e pull request;
|
||||
- fusione assistita di Evidence provenienti da sorgenti diversi.
|
||||
|
||||
Queste esclusioni mantengono la prima implementazione comprensibile, realizzabile,
|
||||
manutenibile e documentabile.
|
||||
Reference in New Issue
Block a user