diff --git a/docs/operations/psd-server-survey-remediation-checklist.md b/docs/operations/psd-server-survey-remediation-checklist.md index e2b447bb..7f5eb13c 100644 --- a/docs/operations/psd-server-survey-remediation-checklist.md +++ b/docs/operations/psd-server-survey-remediation-checklist.md @@ -29,7 +29,8 @@ Authentik, Aritmolab o repository esterni. - Current activity: `1` - Title: Rotate or revoke the exposed DWH credential safely -- Resume from: Activity 1, identify the accountable credential owner and the credential type without revealing its value +- Resume from: Activity 1, review the written per-installation authentication design, then prepare + its executable implementation and rotation plan - Discussion rule: una sola attività può essere `IN_DISCUSSION` - Allowed states: `PENDING`, `IN_DISCUSSION`, `BLOCKED`, `PASS` @@ -51,7 +52,7 @@ Authentik, Aritmolab o repository esterni. | ID | Attività | Stato | Responsabile | Prossimo gate | |---|---|---|---|---| -| 1 | Rotazione controllata della credenziale DWH esposta | `IN_DISCUSSION` | Unassigned | Identificare owner e tipo di credenziale | +| 1 | Rotazione controllata della credenziale DWH esposta | `IN_DISCUSSION` | Proprietario del progetto | Revisionare la specifica scritta e approvare il piano eseguibile | | 2 | Assegnazione dei responsabili dei componenti condivisi | `PENDING` | Unassigned | Elenco owner confermato | | 3 | Risoluzione del dominio pubblico `.it` oppure `.com` | `PENDING` | Unassigned | Origine autorevole documentata | | 4 | Topologia e responsabilità del load balancer | `PENDING` | Unassigned | Route, health, TLS, rollback e allowlist verificati | @@ -65,7 +66,7 @@ Authentik, Aritmolab o repository esterni. ## Activity 1: Rotate or revoke the exposed DWH credential safely - Status: `IN_DISCUSSION` -- Accountable owner: Unassigned +- Accountable owner: Proprietario del progetto (confermato dall'utente) - Objective: sostituire o revocare in modo controllato la credenziale DWH comparsa nell'output interno del survey, senza interrompere consumer legittimi e senza esporne nuovamente il valore. - Why this is required: la credenziale deve essere considerata compromessa; non può essere usata @@ -89,14 +90,82 @@ Authentik, Aritmolab o repository esterni. - timestamp e risultati dei test positivi e negativi; - conferma di revoca della credenziale precedente; - procedura di rollback e relativo esito. -- Discussion notes: il survey ha osservato la credenziale durante l'analisi della configurazione - Nginx relativa al DWH. Non è ancora provato se sia una chiave API, un header condiviso o un altro - tipo di secret; non va confusa automaticamente con una password PostgreSQL. -- Decision: No decision recorded -- Blockers: owner non identificato; tipo di credenziale non verificato; consumer e supporto alla - doppia credenziale non inventariati. -- Next step: il proprietario identifica il responsabile della credenziale della route DWH e indica - se appartiene al team Nginx, Supabase/PostgreSQL o a un altro team. +- Discussion notes: + - la credenziale osservata è stata identificata, senza rileggerne il valore, come chiave statica + `X-API-Key` applicata da Nginx alla route DWH REST `/dwh/`; è distinta dalle password PostgreSQL; + - `/home/chirone/chirone/etl` documenta PostgREST come integrazione HTTP esterna, mentre i processi + ETL e Superset usano PostgreSQL diretto; + - il vecchio ThothII e Chirone WP3 usano PostgreSQL diretto nei profili locali/server; Supabase + Studio è un pannello amministrativo e non appartiene al data-plane applicativo; + - il design approvato resta invariato: il Mac usa `rest_api`, il nuovo ThothII sul server PSD usa + `postgres_direct` con ruolo DWH dedicato e realmente read-only; + - la ricognizione dei repository ha rilevato materiale sensibile hardcoded in file tracciati, + senza riportarne i valori. La bonifica e la rotazione dei segreti coinvolti restano obbligatorie. + - il censimento statico non trova consumer `/dwh/` in ETL, Superset o nel Chirone WP3 attivo: + usano PostgreSQL diretto. Il vecchio container `thothii-core-1` è configurato con + `transport: direct`; i due ThothII restano comunque tecnicamente capaci di usare REST; + - i log Nginx redatti provano 18.523 richieste `/dwh/` dal 23 luglio al 19 agosto 2026: + 18.481 hanno user-agent classificato `python-requests` e gli endpoint RPC corrispondono + prevalentemente a introspezione e campionamento Thoth. La sorgente è una sola, privata e + compatibile con un proxy/load balancer; non prova che esista un solo client finale; + - l'ipotesi che il traffico sia generato dal job ETL delle 03:00 è smentita: nella finestra + 02:30–03:30 Europe/Rome non compare nessuna richiesta `/dwh/`. Il 99,37% del traffico è + concentrato il 13 agosto tra le 17:38 e le 20:03; + - la firma del 13 agosto corrisponde a undici preprocessing Thoth: undici `list_tables`, e + per ciascuna esecuzione 163 chiamate a ognuno dei tre RPC per-tabella più 1.180 `top_values`, + cioè 1.670 richieste per ciclo. `PROJECT_STATE.md` registra proprio il preprocessing PSD live + del 13 agosto su 163 tabelle, con più rerun e correzioni emerse durante l'esecuzione; + - il DAG ETL `nightly_etl_orchestrator` è schedulato con `0 3 * * *`, ma scrive il DWH tramite + PostgreSQL/`psycopg2` diretto. Nel codice tracciato non chiama `/dwh/`, gli RPC Thoth o + `tht workspace preprocess`, né emerge un trigger indiretto verso Thoth; + - la configurazione Nginx nominalmente attiva accetta una sola chiave tramite confronto letterale + in un endpoint `auth_request`. Non esiste una mappa a più chiavi: la doppia credenziale richiede + un refactor, backup, `nginx -t`, reload e rollback in una fase di mutazione autorizzata. + - il proprietario conferma che il ThothII sul Mac deve continuare a usare REST e che sono previste + molte altre installazioni remote, senza tunnel SSH verso Supabase. `/dwh/` è quindi + un'interfaccia remota stabile e multi-client, non una compatibilità temporanea. + - il proprietario decide di mantenere il certificato TLS corrente. L'endpoint esterno REST + presenta lo stesso certificato self-issued di Nginx, valido fino al 21 giugno 2027 e con SAN + per `supabase-aritmolab.policlinicosandonato.it`; non copre un eventuale dominio `.com`; + - il manuale di installazione deve trattare `TLS_CA_FILE` come necessario per ogni client che + non abbia già quel certificato nel proprio trust store, spiegando consegna affidabile, + verifica del fingerprint, rinnovo e aggiornamento coordinato delle installazioni; + - non esiste un ambiente di test. La rotazione dovrà quindi usare una verifica production-safe: + backup, finestra dual-key, RPC `ping` senza dati clinici, test positivo/negativo e rollback. + - il proprietario approva un componente `dwh-auth` riutilizzabile ma opzionale, incluso nel + repository senza modificare il protocollo dei client portabili o il CLI `tht`; + - su PSD `dwh-auth` avrà un lifecycle `systemd` indipendente dallo stack ThothII e comunicherà + con Nginx tramite socket Unix. Lo stop o la sostituzione di ThothII non dovrà interrompere i + client REST; + - il registro sarà composto da file protetti, versionati e aggiornati atomicamente, con un file + per generazione della chiave. Conterrà digest SHA-256 di segreti casuali da almeno 256 bit e + metadati non sensibili, senza SQLite o nuove dipendenze runtime; + - le chiavi saranno assegnate alle installazioni, non alle persone. Saranno prive di scadenza + predefinita, con scadenza opzionale e revoca manuale; + - creazione e import leggeranno o scriveranno soltanto file protetti. Nessun segreto sarà + accettato come argomento, stampato o inserito in log, JSON, documenti o repository; + - la gestione sarà fail-closed: credenziali non valide riceveranno `401`, mentre guasti del + servizio o del registro saranno mappati a `503` senza fallback permissivo; + - il proprietario conferma che le sessioni, gli indici e le cache del vecchio ThothII erano solo + test e non richiedono migrazione o attività di salvaguardia dedicate. Restano necessari il + rollback della route DWH condivisa e il rispetto del gate esplicito prima di fermare lo stack; + - il design approvato è scritto in + `docs/superpowers/specs/2026-08-20-dwh-rest-per-installation-auth-design.md` e richiede una + revisione del proprietario prima del piano di implementazione. +- Decision: il nuovo ThothII server userà PostgreSQL diretto; il Mac e le future installazioni + remote useranno REST. La chiave condivisa corrente non è un modello finale accettabile: la + rotazione esposta userà una transizione a doppia credenziale e l'architettura target assegnerà + un'identità revocabile distinta a ogni installazione. Supabase Studio non sarà usato come + trasporto. Il proprietario del progetto è accountable per coordinare la rotazione. Il meccanismo + target è il servizio server-only `dwh-auth`, indipendente dallo stack ThothII, con registro a file + e una chiave revocabile per installazione. +- Blockers: la specifica scritta attende la revisione del proprietario; mancano ancora il piano + eseguibile, la sorgente protetta per importare la chiave legacy, la procedura realizzata di + aggiornamento/verifica del Mac, la documentazione TLS completa, l'autorizzazione alla mutazione + e la revoca provata della vecchia chiave. +- Next step: dopo l'approvazione della specifica scritta, preparare il piano eseguibile di + implementazione e rotazione dual-key con test automatici, aggiornamento Mac, prova + positiva/negativa e rollback. Nessuna modifica Nginx avviene durante questa discussione. ## Activity 2: Identify accountable owners for shared components diff --git a/docs/superpowers/specs/2026-08-20-dwh-rest-per-installation-auth-design.md b/docs/superpowers/specs/2026-08-20-dwh-rest-per-installation-auth-design.md new file mode 100644 index 00000000..2017bb2f --- /dev/null +++ b/docs/superpowers/specs/2026-08-20-dwh-rest-per-installation-auth-design.md @@ -0,0 +1,383 @@ +# DWH REST Per-Installation Authentication — Design + +**Date:** 2026-08-20 + +**Status:** Approved in discussion; awaiting review of this written specification + +## Purpose + +Replace the single static `X-API-Key` currently protecting the PSD `/dwh/` route with a reusable, +server-side authentication component that assigns one independently revocable credential to each +ThothII installation. + +The design must preserve the existing portable connector contract. A ThothII installation on +macOS, Windows, or Linux continues to send `X-API-Key` over HTTPS when its selected transport is +`rest_api`. Installations using `postgres_direct` or `ssh_tunnel` are unaffected. + +The component belongs in the ThothII source repository so that it can be reused on another DWH +server, but it is optional server infrastructure. It is not part of the portable application +runtime and is not started or stopped by `tht`. + +## Confirmed context + +- The exposed credential must be considered compromised and must eventually be revoked. +- The current PSD route accepts one shared static `X-API-Key` through Nginx `auth_request`. +- The Mac installation must continue to access PSD through REST. +- Future remote ThothII installations will also use REST and cannot depend on an SSH tunnel. +- The new ThothII installation on the PSD server will use `postgres_direct` with a dedicated, + demonstrably read-only DWH role. +- The current self-issued TLS certificate remains in use. Clients that do not already trust its + issuer require a separately delivered `TLS_CA_FILE` and must verify the approved fingerprint. +- There is no separate test environment. This is accepted and is not itself a blocker. +- Existing ThothII sessions, derived indexes, and model caches are test data. They are not migrated + or protected by session-specific backup work. +- The legacy stack remains unchanged until the deployment program reaches its explicit stop gate. + This is an authorization boundary, not a session-preservation requirement. + +## Scope + +This design includes: + +- the reusable `dwh-auth` service and its administrative command; +- a protected file-based credential registry; +- Nginx `auth_request` integration; +- one credential identity per ThothII installation; +- creation, delivery, rotation, optional expiry, and revocation; +- a dual-key transition from the exposed shared credential; +- generic and PSD-specific operator documentation; +- automated tests and production-safe acceptance checks. + +This design does not: + +- change the ThothII REST header or connector protocol; +- add DWH credential administration to the portable `tht` CLI; +- place server credentials in a workspace repository; +- replace PostgreSQL grants, RLS, or the PostgREST read-only role boundary; +- preserve or migrate legacy ThothII sessions, Qdrant indexes, or Ollama caches; +- change the current certificate or resolve the separate `.it` versus `.com` public-origin issue; +- authorize any current server, Nginx, database, or service mutation. + +## Alternatives considered + +### Extend the portable `tht` CLI + +This would provide a single operator command surface, but it would distribute server-only DWH +administration code to every macOS and Windows installation. The DWH route also has a lifecycle +independent from the local ThothII application. This option is rejected. + +### Maintain a manual Nginx key map + +This is initially small, but it encourages plaintext credentials in Nginx configuration and makes +atomic rotation, redacted inventory, and reliable revocation harder. This option is rejected. + +### Use SQLite + +SQLite is not a declared project dependency and is unnecessary for the expected registry size and +write rate. Introducing a database would add packaging, migration, backup, and corruption-recovery +work without improving the required contract. This option is rejected. + +### Selected approach + +Add a standalone, standard-library Go component with a protected file registry. On PSD it runs as +an independent `systemd` service and communicates with host Nginx through a Unix socket. + +## Component and lifecycle boundary + +The repository will contain a separate component, expected under `tools/dwh-auth/`, plus generic +deployment templates and documentation. Its build and tests are isolated from the portable +application entrypoints. + +On PSD: + +- `dwh-auth` runs under a dedicated, unprivileged system account; +- `systemd` owns its lifecycle; +- the service is not included in the canonical ThothII Compose stack; +- the registry resides outside both the Git checkout and ThothII application data; +- the service receives read-only access to active credential records; +- Nginx is the only runtime caller and reaches it through a permission-restricted Unix socket; +- stopping, updating, or replacing ThothII does not interrupt `/dwh/` authentication. + +The same source may be deployed on another Linux DWH server. Platform-specific service packaging +beyond the approved PSD `systemd` deployment is not required by this implementation. + +## Credential model + +One credential identifies one ThothII installation, not one human user. An installation may have +more than one temporarily active generation during rotation. + +The external key has two components: + +```text +. +``` + +Requirements: + +- the public key ID is generated by the tool and contains only a strict safe alphabet; +- the secret is generated with the operating system cryptographic random source; +- the secret has at least 256 bits of entropy and uses an unambiguous transport-safe encoding; +- the complete header has a fixed maximum length; +- installation IDs are operator-provided, non-secret, unique labels and must not contain personal + or clinical information; +- descriptions are optional non-secret operator metadata. + +Because the secret is a uniformly random 256-bit value rather than a human password, the registry +stores a SHA-256 digest and verifies it with a constant-time comparison. No password hashing or +external cryptographic runtime dependency is needed. + +## Protected file registry + +The registry has versioned active and revoked records: + +```text +/ + active/.json + revoked/.json +``` + +An active record contains: + +- schema version; +- public key ID; +- installation ID and optional description; +- encoded SHA-256 digest; +- creation timestamp; +- optional expiry timestamp, absent by default. + +A revoked record additionally contains the revocation timestamp and a non-secret reason. The +original key never appears in either directory. + +Multiple key IDs may refer to the same installation ID during a rotation. Creation uses an +exclusive temporary file, file flush, directory-safe atomic rename, restrictive ownership and +mode checks, and refusal of symlinks or unsafe paths. Revocation atomically moves a record from +`active` to `revoked` on the same filesystem. A malformed or unsafe record never authenticates. + +The runtime service does not update registry files. Successful and failed use is recorded only in +the sanitized service journal, avoiding per-request writes and keeping the service's registry +access read-only. + +## Administrative interface + +The standalone binary provides a server-only administrative surface equivalent to: + +```text +dwh-auth key create --installation-id ID --description TEXT --output ABSOLUTE_FILE +dwh-auth key import --installation-id ID --from-file ABSOLUTE_FILE +dwh-auth key list [--json] +dwh-auth key revoke --key-id ID --reason TEXT +dwh-auth key status --key-id ID [--json] +dwh-auth check +``` + +The exact syntax will be frozen in the implementation plan and tests. The interface must obey +these invariants: + +- secrets are never accepted as command-line values; +- `create` writes the generated credential once to a new absolute file with restrictive access; +- `import` reads an existing protected file without displaying its content; +- normal stdout contains only non-secret identifiers, abbreviated fingerprints, status, and + paths; +- JSON output is pristine and contains no credential value or digest; +- an existing output file is never overwritten; +- listing and status commands never expose digests; +- errors and logs pass explicit secret-leak regression tests. + +## Provisioning and delivery + +The operator creates one key for each installation and transfers it through an approved protected +channel, such as an organizational password manager, a secret manager or MDM, or authenticated +file transfer when the recipient has suitable access. Email, ordinary chat, and ticket bodies are +not approved delivery channels. + +On the recipient installation: + +- the normal interactive path saves the value through authenticated Workspace management, where + ThothII keeps it in the installation-local encrypted workspace vault; +- a supported headless deployment may place it in a protected file referenced by `API_KEY_FILE`; +- the credential never enters the shared workspace repository or ordinary environment values; +- the private CA is delivered separately and configured with `TLS_CA_FILE` when the certificate + chain is not already trusted; +- the operator verifies the documented certificate fingerprint before trusting the CA file. + +After configuration, the recipient runs the existing workspace connection test against the +harmless `/rpc/ping` diagnostic. Only after the server and recipient both confirm the public key +ID may the one-time delivery copy be removed according to the organization's secret-handling +procedure. + +## Expiry and rotation + +Keys have no automatic expiry by default. This avoids unplanned outages for remote installations +that may not be continuously administered. An optional expiry timestamp is supported for sites +whose policy requires it; an expired key is rejected without fallback. Administrative status +output warns about approaching expiries without exposing secrets. + +Normal zero-downtime rotation is: + +1. create a second key generation for the installation; +2. deliver and configure it on the client; +3. verify `/rpc/ping` and the sanitized server key ID; +4. revoke the earlier key; +5. prove that the revoked key receives `401`; +6. remove temporary delivery material after recipient confirmation. + +## Request flow and Nginx contract + +```text +remote ThothII + -> HTTPS /dwh/ with X-API-Key + -> Nginx auth_request subrequest + -> dwh-auth over a Unix socket + -> valid: Nginx proxies the original request to internal PostgREST + -> invalid or revoked: Nginx returns 401 + -> authenticator fault: Nginx returns 503 +``` + +`dwh-auth` authenticates only. It does not proxy the original request, read a clinical response, +or connect to PostgreSQL. PostgREST remains inaccessible as a public direct backend. + +Nginx integration must: + +- forward only the bounded credential header to the internal verification endpoint; +- suppress the original request body in the auth subrequest; +- discard client-supplied identity or audit headers; +- accept only the authenticator's success result; +- return the same generic `401` for missing, malformed, unknown, expired, and revoked keys; +- map authenticator or registry faults to `503`, never to permissive access; +- apply bounded failure-rate protection; +- log only timestamp, result, request metadata already approved for the route, and the verified + public key ID; +- preserve the existing PostgREST path and response contract for successful requests. + +The authenticator returns a verified public key ID only after successful authentication. Nginx may +use that value in its sanitized access log but does not need to forward it to PostgREST. + +## Fail-closed behavior + +The following conditions deny access: + +- header absent, duplicated, oversized, malformed, or encoded incorrectly; +- unknown public key ID; +- digest mismatch; +- revoked or expired record; +- unsafe permissions, symlink, malformed JSON, schema mismatch, or inconsistent record identity; +- unreadable registry; +- service startup or runtime failure. + +Credential failures return a generic `401`. Infrastructure failures return `503` after the Nginx +mapping is applied. Neither response reveals whether an installation ID exists. Health reporting +is local-only and does not weaken the authentication decision. + +## PSD dual-key transition + +The exposed shared credential is represented temporarily as `legacy-shared`. It is imported only +from an approved protected source and is never placed in a command argument, terminal output, +document, or evidence file. + +The production route does not switch to the new authenticator until all of the following hold: + +1. protected backup and exact rollback instructions exist for the affected Nginx and service + configuration; +2. the new service passes local synthetic checks while disconnected from the public route; +3. `legacy-shared` passes `/rpc/ping` through the candidate authentication path; +4. a new per-installation Mac key passes the same diagnostic; +5. a random invalid key is rejected; +6. `nginx -t` passes and the authorized operator approves reload. + +After the Mac uses its new credential successfully for the agreed observation window, +`legacy-shared` is revoked. Acceptance requires a positive test with the new key and a negative +test proving `401` with the legacy key. The new secret must not appear in service, Nginx, shell, or +application logs. + +Rollback during the dual-key window restores only the reviewed authentication route and service +configuration needed to keep REST clients working. It does not preserve or restore legacy +ThothII sessions, indexes, or model caches. + +## PostgreSQL authorization boundary + +Successful API-key authentication is not proof of read-only database authorization. The existing +PostgREST role, exposed schemas, function privileges, default privileges, and direct network +reachability remain subject to the catalog-only survey activity. Project execution cannot treat +`dwh-auth` as a substitute for a demonstrably read-only `dwh_reader` boundary. + +## Verification strategy + +Automated tests use synthetic credentials and no clinical data. They cover: + +- key generation entropy and syntax; +- create, import, list, status, optional expiry, rotation, and revocation; +- constant-time digest verification; +- missing, duplicate, malformed, oversized, unknown, expired, and revoked credentials; +- corrupt records, unsafe modes, symlinks, path traversal, and partial files; +- atomic creation and revocation under concurrent reads; +- clean failure when the registry or socket is unavailable; +- absence of credential values and digests in human output, JSON, errors, and logs; +- Nginx `auth_request` integration against a synthetic upstream and `/rpc/ping` fixture; +- `401` for credential failures and `503` for infrastructure failures; +- unchanged successful request path and response body; +- independent service lifecycle from the ThothII stack. + +Portability regression checks prove that: + +- the existing REST connector still sends the same `X-API-Key` header; +- `postgres_direct` and `ssh_tunnel` configuration remain unchanged; +- canonical local and server ThothII Compose rendering does not acquire `dwh-auth`; +- macOS and Windows installation contracts do not require the new binary or files; +- existing `tht`, backend, frontend, and harness gates remain green in proportion to the touched + source boundaries. + +There is no separate PSD test environment. Local synthetic and integration tests precede a +bounded, reversible production rollout using only `/rpc/ping`. The absence of a test environment +does not trigger preservation work for legacy test sessions. + +## Documentation deliverables + +Implementation is incomplete until the following clear, secret-free documents exist and pass +their documentation checks: + +1. a generic server installation and administration guide for `dwh-auth`; +2. a generic enrollment guide for issuing a key to a macOS, Windows, Linux, or headless ThothII + installation; +3. exact GUI and `API_KEY_FILE` configuration paths; +4. a TLS guide covering `TLS_CA_FILE`, trusted delivery, fingerprint verification, renewal, and + coordinated client updates; +5. a PSD runbook for backup, installation, legacy import, dual-key activation, Mac update, + observation, revocation, negative testing, and rollback; +6. a troubleshooting table for `401`, `503`, TLS failures, revoked keys, and unreadable registry + records; +7. sanitized evidence templates that never request secret values or digests; +8. links from the existing local, server, workspace, and final unified user manuals. + +Examples use synthetic hostnames, IDs, fingerprints, and credentials. No real key, certificate +body, password, connection string, or clinical result is committed. + +## Acceptance criteria + +The design is implemented only when: + +- each REST installation can be identified and revoked independently; +- no plaintext server credential is stored in the registry; +- the portable ThothII connector contract is unchanged; +- the PSD ThothII server continues to select `postgres_direct`; +- stopping ThothII does not stop `dwh-auth` or `/dwh/` authentication; +- the authenticator fails closed and its logs are sanitized; +- the dual-key production procedure proves both positive and negative outcomes; +- the exposed legacy credential is demonstrably rejected; +- the current TLS limitations are documented precisely rather than silently bypassed; +- database read-only authorization is independently verified; +- all documentation deliverables are reviewed and reproducible by an operator who did not write + the implementation. + +## Authorization gates + +Approval of this design authorizes documentation and planning only. It does not authorize: + +- reading or importing the legacy credential; +- creating real keys; +- installing a binary or `systemd` unit; +- writing the registry directory; +- modifying or reloading Nginx; +- querying or changing PostgreSQL; +- changing, stopping, or removing the legacy ThothII stack. + +Those actions require the executable plan, the remaining survey gates, explicit mutation +authorization, and retained sanitized evidence.