docs: design per-installation DWH REST auth

This commit is contained in:
User
2026-08-20 22:11:23 +02:00
parent b0ead4be54
commit 8a1c23536f
2 changed files with 463 additions and 11 deletions
@@ -29,7 +29,8 @@ Authentik, Aritmolab o repository esterni.
- Current activity: `1`
- Title: Rotate or revoke the exposed DWH credential safely
- Resume from: Activity 1, identify the accountable credential owner and the credential type without revealing its value
- Resume from: Activity 1, review the written per-installation authentication design, then prepare
its executable implementation and rotation plan
- Discussion rule: una sola attività può essere `IN_DISCUSSION`
- Allowed states: `PENDING`, `IN_DISCUSSION`, `BLOCKED`, `PASS`
@@ -51,7 +52,7 @@ Authentik, Aritmolab o repository esterni.
| ID | Attività | Stato | Responsabile | Prossimo gate |
|---|---|---|---|---|
| 1 | Rotazione controllata della credenziale DWH esposta | `IN_DISCUSSION` | Unassigned | Identificare owner e tipo di credenziale |
| 1 | Rotazione controllata della credenziale DWH esposta | `IN_DISCUSSION` | Proprietario del progetto | Revisionare la specifica scritta e approvare il piano eseguibile |
| 2 | Assegnazione dei responsabili dei componenti condivisi | `PENDING` | Unassigned | Elenco owner confermato |
| 3 | Risoluzione del dominio pubblico `.it` oppure `.com` | `PENDING` | Unassigned | Origine autorevole documentata |
| 4 | Topologia e responsabilità del load balancer | `PENDING` | Unassigned | Route, health, TLS, rollback e allowlist verificati |
@@ -65,7 +66,7 @@ Authentik, Aritmolab o repository esterni.
## Activity 1: Rotate or revoke the exposed DWH credential safely
- Status: `IN_DISCUSSION`
- Accountable owner: Unassigned
- Accountable owner: Proprietario del progetto (confermato dall'utente)
- Objective: sostituire o revocare in modo controllato la credenziale DWH comparsa nell'output
interno del survey, senza interrompere consumer legittimi e senza esporne nuovamente il valore.
- Why this is required: la credenziale deve essere considerata compromessa; non può essere usata
@@ -89,14 +90,82 @@ Authentik, Aritmolab o repository esterni.
- timestamp e risultati dei test positivi e negativi;
- conferma di revoca della credenziale precedente;
- procedura di rollback e relativo esito.
- Discussion notes: il survey ha osservato la credenziale durante l'analisi della configurazione
Nginx relativa al DWH. Non è ancora provato se sia una chiave API, un header condiviso o un altro
tipo di secret; non va confusa automaticamente con una password PostgreSQL.
- Decision: No decision recorded
- Blockers: owner non identificato; tipo di credenziale non verificato; consumer e supporto alla
doppia credenziale non inventariati.
- Next step: il proprietario identifica il responsabile della credenziale della route DWH e indica
se appartiene al team Nginx, Supabase/PostgreSQL o a un altro team.
- Discussion notes:
- la credenziale osservata è stata identificata, senza rileggerne il valore, come chiave statica
`X-API-Key` applicata da Nginx alla route DWH REST `/dwh/`; è distinta dalle password PostgreSQL;
- `/home/chirone/chirone/etl` documenta PostgREST come integrazione HTTP esterna, mentre i processi
ETL e Superset usano PostgreSQL diretto;
- il vecchio ThothII e Chirone WP3 usano PostgreSQL diretto nei profili locali/server; Supabase
Studio è un pannello amministrativo e non appartiene al data-plane applicativo;
- il design approvato resta invariato: il Mac usa `rest_api`, il nuovo ThothII sul server PSD usa
`postgres_direct` con ruolo DWH dedicato e realmente read-only;
- la ricognizione dei repository ha rilevato materiale sensibile hardcoded in file tracciati,
senza riportarne i valori. La bonifica e la rotazione dei segreti coinvolti restano obbligatorie.
- il censimento statico non trova consumer `/dwh/` in ETL, Superset o nel Chirone WP3 attivo:
usano PostgreSQL diretto. Il vecchio container `thothii-core-1` è configurato con
`transport: direct`; i due ThothII restano comunque tecnicamente capaci di usare REST;
- i log Nginx redatti provano 18.523 richieste `/dwh/` dal 23 luglio al 19 agosto 2026:
18.481 hanno user-agent classificato `python-requests` e gli endpoint RPC corrispondono
prevalentemente a introspezione e campionamento Thoth. La sorgente è una sola, privata e
compatibile con un proxy/load balancer; non prova che esista un solo client finale;
- l'ipotesi che il traffico sia generato dal job ETL delle 03:00 è smentita: nella finestra
02:30–03:30 Europe/Rome non compare nessuna richiesta `/dwh/`. Il 99,37% del traffico è
concentrato il 13 agosto tra le 17:38 e le 20:03;
- la firma del 13 agosto corrisponde a undici preprocessing Thoth: undici `list_tables`, e
per ciascuna esecuzione 163 chiamate a ognuno dei tre RPC per-tabella più 1.180 `top_values`,
cioè 1.670 richieste per ciclo. `PROJECT_STATE.md` registra proprio il preprocessing PSD live
del 13 agosto su 163 tabelle, con più rerun e correzioni emerse durante l'esecuzione;
- il DAG ETL `nightly_etl_orchestrator` è schedulato con `0 3 * * *`, ma scrive il DWH tramite
PostgreSQL/`psycopg2` diretto. Nel codice tracciato non chiama `/dwh/`, gli RPC Thoth o
`tht workspace preprocess`, né emerge un trigger indiretto verso Thoth;
- la configurazione Nginx nominalmente attiva accetta una sola chiave tramite confronto letterale
in un endpoint `auth_request`. Non esiste una mappa a più chiavi: la doppia credenziale richiede
un refactor, backup, `nginx -t`, reload e rollback in una fase di mutazione autorizzata.
- il proprietario conferma che il ThothII sul Mac deve continuare a usare REST e che sono previste
molte altre installazioni remote, senza tunnel SSH verso Supabase. `/dwh/` è quindi
un'interfaccia remota stabile e multi-client, non una compatibilità temporanea.
- il proprietario decide di mantenere il certificato TLS corrente. L'endpoint esterno REST
presenta lo stesso certificato self-issued di Nginx, valido fino al 21 giugno 2027 e con SAN
per `supabase-aritmolab.policlinicosandonato.it`; non copre un eventuale dominio `.com`;
- il manuale di installazione deve trattare `TLS_CA_FILE` come necessario per ogni client che
non abbia già quel certificato nel proprio trust store, spiegando consegna affidabile,
verifica del fingerprint, rinnovo e aggiornamento coordinato delle installazioni;
- non esiste un ambiente di test. La rotazione dovrà quindi usare una verifica production-safe:
backup, finestra dual-key, RPC `ping` senza dati clinici, test positivo/negativo e rollback.
- il proprietario approva un componente `dwh-auth` riutilizzabile ma opzionale, incluso nel
repository senza modificare il protocollo dei client portabili o il CLI `tht`;
- su PSD `dwh-auth` avrà un lifecycle `systemd` indipendente dallo stack ThothII e comunicherà
con Nginx tramite socket Unix. Lo stop o la sostituzione di ThothII non dovrà interrompere i
client REST;
- il registro sarà composto da file protetti, versionati e aggiornati atomicamente, con un file
per generazione della chiave. Conterrà digest SHA-256 di segreti casuali da almeno 256 bit e
metadati non sensibili, senza SQLite o nuove dipendenze runtime;
- le chiavi saranno assegnate alle installazioni, non alle persone. Saranno prive di scadenza
predefinita, con scadenza opzionale e revoca manuale;
- creazione e import leggeranno o scriveranno soltanto file protetti. Nessun segreto sarà
accettato come argomento, stampato o inserito in log, JSON, documenti o repository;
- la gestione sarà fail-closed: credenziali non valide riceveranno `401`, mentre guasti del
servizio o del registro saranno mappati a `503` senza fallback permissivo;
- il proprietario conferma che le sessioni, gli indici e le cache del vecchio ThothII erano solo
test e non richiedono migrazione o attività di salvaguardia dedicate. Restano necessari il
rollback della route DWH condivisa e il rispetto del gate esplicito prima di fermare lo stack;
- il design approvato è scritto in
`docs/superpowers/specs/2026-08-20-dwh-rest-per-installation-auth-design.md` e richiede una
revisione del proprietario prima del piano di implementazione.
- Decision: il nuovo ThothII server userà PostgreSQL diretto; il Mac e le future installazioni
remote useranno REST. La chiave condivisa corrente non è un modello finale accettabile: la
rotazione esposta userà una transizione a doppia credenziale e l'architettura target assegnerà
un'identità revocabile distinta a ogni installazione. Supabase Studio non sarà usato come
trasporto. Il proprietario del progetto è accountable per coordinare la rotazione. Il meccanismo
target è il servizio server-only `dwh-auth`, indipendente dallo stack ThothII, con registro a file
e una chiave revocabile per installazione.
- Blockers: la specifica scritta attende la revisione del proprietario; mancano ancora il piano
eseguibile, la sorgente protetta per importare la chiave legacy, la procedura realizzata di
aggiornamento/verifica del Mac, la documentazione TLS completa, l'autorizzazione alla mutazione
e la revoca provata della vecchia chiave.
- Next step: dopo l'approvazione della specifica scritta, preparare il piano eseguibile di
implementazione e rotazione dual-key con test automatici, aggiornamento Mac, prova
positiva/negativa e rollback. Nessuna modifica Nginx avviene durante questa discussione.
## Activity 2: Identify accountable owners for shared components
@@ -0,0 +1,383 @@
# DWH REST Per-Installation Authentication — Design
**Date:** 2026-08-20
**Status:** Approved in discussion; awaiting review of this written specification
## Purpose
Replace the single static `X-API-Key` currently protecting the PSD `/dwh/` route with a reusable,
server-side authentication component that assigns one independently revocable credential to each
ThothII installation.
The design must preserve the existing portable connector contract. A ThothII installation on
macOS, Windows, or Linux continues to send `X-API-Key` over HTTPS when its selected transport is
`rest_api`. Installations using `postgres_direct` or `ssh_tunnel` are unaffected.
The component belongs in the ThothII source repository so that it can be reused on another DWH
server, but it is optional server infrastructure. It is not part of the portable application
runtime and is not started or stopped by `tht`.
## Confirmed context
- The exposed credential must be considered compromised and must eventually be revoked.
- The current PSD route accepts one shared static `X-API-Key` through Nginx `auth_request`.
- The Mac installation must continue to access PSD through REST.
- Future remote ThothII installations will also use REST and cannot depend on an SSH tunnel.
- The new ThothII installation on the PSD server will use `postgres_direct` with a dedicated,
demonstrably read-only DWH role.
- The current self-issued TLS certificate remains in use. Clients that do not already trust its
issuer require a separately delivered `TLS_CA_FILE` and must verify the approved fingerprint.
- There is no separate test environment. This is accepted and is not itself a blocker.
- Existing ThothII sessions, derived indexes, and model caches are test data. They are not migrated
or protected by session-specific backup work.
- The legacy stack remains unchanged until the deployment program reaches its explicit stop gate.
This is an authorization boundary, not a session-preservation requirement.
## Scope
This design includes:
- the reusable `dwh-auth` service and its administrative command;
- a protected file-based credential registry;
- Nginx `auth_request` integration;
- one credential identity per ThothII installation;
- creation, delivery, rotation, optional expiry, and revocation;
- a dual-key transition from the exposed shared credential;
- generic and PSD-specific operator documentation;
- automated tests and production-safe acceptance checks.
This design does not:
- change the ThothII REST header or connector protocol;
- add DWH credential administration to the portable `tht` CLI;
- place server credentials in a workspace repository;
- replace PostgreSQL grants, RLS, or the PostgREST read-only role boundary;
- preserve or migrate legacy ThothII sessions, Qdrant indexes, or Ollama caches;
- change the current certificate or resolve the separate `.it` versus `.com` public-origin issue;
- authorize any current server, Nginx, database, or service mutation.
## Alternatives considered
### Extend the portable `tht` CLI
This would provide a single operator command surface, but it would distribute server-only DWH
administration code to every macOS and Windows installation. The DWH route also has a lifecycle
independent from the local ThothII application. This option is rejected.
### Maintain a manual Nginx key map
This is initially small, but it encourages plaintext credentials in Nginx configuration and makes
atomic rotation, redacted inventory, and reliable revocation harder. This option is rejected.
### Use SQLite
SQLite is not a declared project dependency and is unnecessary for the expected registry size and
write rate. Introducing a database would add packaging, migration, backup, and corruption-recovery
work without improving the required contract. This option is rejected.
### Selected approach
Add a standalone, standard-library Go component with a protected file registry. On PSD it runs as
an independent `systemd` service and communicates with host Nginx through a Unix socket.
## Component and lifecycle boundary
The repository will contain a separate component, expected under `tools/dwh-auth/`, plus generic
deployment templates and documentation. Its build and tests are isolated from the portable
application entrypoints.
On PSD:
- `dwh-auth` runs under a dedicated, unprivileged system account;
- `systemd` owns its lifecycle;
- the service is not included in the canonical ThothII Compose stack;
- the registry resides outside both the Git checkout and ThothII application data;
- the service receives read-only access to active credential records;
- Nginx is the only runtime caller and reaches it through a permission-restricted Unix socket;
- stopping, updating, or replacing ThothII does not interrupt `/dwh/` authentication.
The same source may be deployed on another Linux DWH server. Platform-specific service packaging
beyond the approved PSD `systemd` deployment is not required by this implementation.
## Credential model
One credential identifies one ThothII installation, not one human user. An installation may have
more than one temporarily active generation during rotation.
The external key has two components:
```text
<public-key-id>.<random-secret>
```
Requirements:
- the public key ID is generated by the tool and contains only a strict safe alphabet;
- the secret is generated with the operating system cryptographic random source;
- the secret has at least 256 bits of entropy and uses an unambiguous transport-safe encoding;
- the complete header has a fixed maximum length;
- installation IDs are operator-provided, non-secret, unique labels and must not contain personal
or clinical information;
- descriptions are optional non-secret operator metadata.
Because the secret is a uniformly random 256-bit value rather than a human password, the registry
stores a SHA-256 digest and verifies it with a constant-time comparison. No password hashing or
external cryptographic runtime dependency is needed.
## Protected file registry
The registry has versioned active and revoked records:
```text
<registry-root>/
active/<public-key-id>.json
revoked/<public-key-id>.json
```
An active record contains:
- schema version;
- public key ID;
- installation ID and optional description;
- encoded SHA-256 digest;
- creation timestamp;
- optional expiry timestamp, absent by default.
A revoked record additionally contains the revocation timestamp and a non-secret reason. The
original key never appears in either directory.
Multiple key IDs may refer to the same installation ID during a rotation. Creation uses an
exclusive temporary file, file flush, directory-safe atomic rename, restrictive ownership and
mode checks, and refusal of symlinks or unsafe paths. Revocation atomically moves a record from
`active` to `revoked` on the same filesystem. A malformed or unsafe record never authenticates.
The runtime service does not update registry files. Successful and failed use is recorded only in
the sanitized service journal, avoiding per-request writes and keeping the service's registry
access read-only.
## Administrative interface
The standalone binary provides a server-only administrative surface equivalent to:
```text
dwh-auth key create --installation-id ID --description TEXT --output ABSOLUTE_FILE
dwh-auth key import --installation-id ID --from-file ABSOLUTE_FILE
dwh-auth key list [--json]
dwh-auth key revoke --key-id ID --reason TEXT
dwh-auth key status --key-id ID [--json]
dwh-auth check
```
The exact syntax will be frozen in the implementation plan and tests. The interface must obey
these invariants:
- secrets are never accepted as command-line values;
- `create` writes the generated credential once to a new absolute file with restrictive access;
- `import` reads an existing protected file without displaying its content;
- normal stdout contains only non-secret identifiers, abbreviated fingerprints, status, and
paths;
- JSON output is pristine and contains no credential value or digest;
- an existing output file is never overwritten;
- listing and status commands never expose digests;
- errors and logs pass explicit secret-leak regression tests.
## Provisioning and delivery
The operator creates one key for each installation and transfers it through an approved protected
channel, such as an organizational password manager, a secret manager or MDM, or authenticated
file transfer when the recipient has suitable access. Email, ordinary chat, and ticket bodies are
not approved delivery channels.
On the recipient installation:
- the normal interactive path saves the value through authenticated Workspace management, where
ThothII keeps it in the installation-local encrypted workspace vault;
- a supported headless deployment may place it in a protected file referenced by `API_KEY_FILE`;
- the credential never enters the shared workspace repository or ordinary environment values;
- the private CA is delivered separately and configured with `TLS_CA_FILE` when the certificate
chain is not already trusted;
- the operator verifies the documented certificate fingerprint before trusting the CA file.
After configuration, the recipient runs the existing workspace connection test against the
harmless `/rpc/ping` diagnostic. Only after the server and recipient both confirm the public key
ID may the one-time delivery copy be removed according to the organization's secret-handling
procedure.
## Expiry and rotation
Keys have no automatic expiry by default. This avoids unplanned outages for remote installations
that may not be continuously administered. An optional expiry timestamp is supported for sites
whose policy requires it; an expired key is rejected without fallback. Administrative status
output warns about approaching expiries without exposing secrets.
Normal zero-downtime rotation is:
1. create a second key generation for the installation;
2. deliver and configure it on the client;
3. verify `/rpc/ping` and the sanitized server key ID;
4. revoke the earlier key;
5. prove that the revoked key receives `401`;
6. remove temporary delivery material after recipient confirmation.
## Request flow and Nginx contract
```text
remote ThothII
-> HTTPS /dwh/ with X-API-Key
-> Nginx auth_request subrequest
-> dwh-auth over a Unix socket
-> valid: Nginx proxies the original request to internal PostgREST
-> invalid or revoked: Nginx returns 401
-> authenticator fault: Nginx returns 503
```
`dwh-auth` authenticates only. It does not proxy the original request, read a clinical response,
or connect to PostgreSQL. PostgREST remains inaccessible as a public direct backend.
Nginx integration must:
- forward only the bounded credential header to the internal verification endpoint;
- suppress the original request body in the auth subrequest;
- discard client-supplied identity or audit headers;
- accept only the authenticator's success result;
- return the same generic `401` for missing, malformed, unknown, expired, and revoked keys;
- map authenticator or registry faults to `503`, never to permissive access;
- apply bounded failure-rate protection;
- log only timestamp, result, request metadata already approved for the route, and the verified
public key ID;
- preserve the existing PostgREST path and response contract for successful requests.
The authenticator returns a verified public key ID only after successful authentication. Nginx may
use that value in its sanitized access log but does not need to forward it to PostgREST.
## Fail-closed behavior
The following conditions deny access:
- header absent, duplicated, oversized, malformed, or encoded incorrectly;
- unknown public key ID;
- digest mismatch;
- revoked or expired record;
- unsafe permissions, symlink, malformed JSON, schema mismatch, or inconsistent record identity;
- unreadable registry;
- service startup or runtime failure.
Credential failures return a generic `401`. Infrastructure failures return `503` after the Nginx
mapping is applied. Neither response reveals whether an installation ID exists. Health reporting
is local-only and does not weaken the authentication decision.
## PSD dual-key transition
The exposed shared credential is represented temporarily as `legacy-shared`. It is imported only
from an approved protected source and is never placed in a command argument, terminal output,
document, or evidence file.
The production route does not switch to the new authenticator until all of the following hold:
1. protected backup and exact rollback instructions exist for the affected Nginx and service
configuration;
2. the new service passes local synthetic checks while disconnected from the public route;
3. `legacy-shared` passes `/rpc/ping` through the candidate authentication path;
4. a new per-installation Mac key passes the same diagnostic;
5. a random invalid key is rejected;
6. `nginx -t` passes and the authorized operator approves reload.
After the Mac uses its new credential successfully for the agreed observation window,
`legacy-shared` is revoked. Acceptance requires a positive test with the new key and a negative
test proving `401` with the legacy key. The new secret must not appear in service, Nginx, shell, or
application logs.
Rollback during the dual-key window restores only the reviewed authentication route and service
configuration needed to keep REST clients working. It does not preserve or restore legacy
ThothII sessions, indexes, or model caches.
## PostgreSQL authorization boundary
Successful API-key authentication is not proof of read-only database authorization. The existing
PostgREST role, exposed schemas, function privileges, default privileges, and direct network
reachability remain subject to the catalog-only survey activity. Project execution cannot treat
`dwh-auth` as a substitute for a demonstrably read-only `dwh_reader` boundary.
## Verification strategy
Automated tests use synthetic credentials and no clinical data. They cover:
- key generation entropy and syntax;
- create, import, list, status, optional expiry, rotation, and revocation;
- constant-time digest verification;
- missing, duplicate, malformed, oversized, unknown, expired, and revoked credentials;
- corrupt records, unsafe modes, symlinks, path traversal, and partial files;
- atomic creation and revocation under concurrent reads;
- clean failure when the registry or socket is unavailable;
- absence of credential values and digests in human output, JSON, errors, and logs;
- Nginx `auth_request` integration against a synthetic upstream and `/rpc/ping` fixture;
- `401` for credential failures and `503` for infrastructure failures;
- unchanged successful request path and response body;
- independent service lifecycle from the ThothII stack.
Portability regression checks prove that:
- the existing REST connector still sends the same `X-API-Key` header;
- `postgres_direct` and `ssh_tunnel` configuration remain unchanged;
- canonical local and server ThothII Compose rendering does not acquire `dwh-auth`;
- macOS and Windows installation contracts do not require the new binary or files;
- existing `tht`, backend, frontend, and harness gates remain green in proportion to the touched
source boundaries.
There is no separate PSD test environment. Local synthetic and integration tests precede a
bounded, reversible production rollout using only `/rpc/ping`. The absence of a test environment
does not trigger preservation work for legacy test sessions.
## Documentation deliverables
Implementation is incomplete until the following clear, secret-free documents exist and pass
their documentation checks:
1. a generic server installation and administration guide for `dwh-auth`;
2. a generic enrollment guide for issuing a key to a macOS, Windows, Linux, or headless ThothII
installation;
3. exact GUI and `API_KEY_FILE` configuration paths;
4. a TLS guide covering `TLS_CA_FILE`, trusted delivery, fingerprint verification, renewal, and
coordinated client updates;
5. a PSD runbook for backup, installation, legacy import, dual-key activation, Mac update,
observation, revocation, negative testing, and rollback;
6. a troubleshooting table for `401`, `503`, TLS failures, revoked keys, and unreadable registry
records;
7. sanitized evidence templates that never request secret values or digests;
8. links from the existing local, server, workspace, and final unified user manuals.
Examples use synthetic hostnames, IDs, fingerprints, and credentials. No real key, certificate
body, password, connection string, or clinical result is committed.
## Acceptance criteria
The design is implemented only when:
- each REST installation can be identified and revoked independently;
- no plaintext server credential is stored in the registry;
- the portable ThothII connector contract is unchanged;
- the PSD ThothII server continues to select `postgres_direct`;
- stopping ThothII does not stop `dwh-auth` or `/dwh/` authentication;
- the authenticator fails closed and its logs are sanitized;
- the dual-key production procedure proves both positive and negative outcomes;
- the exposed legacy credential is demonstrably rejected;
- the current TLS limitations are documented precisely rather than silently bypassed;
- database read-only authorization is independently verified;
- all documentation deliverables are reviewed and reproducible by an operator who did not write
the implementation.
## Authorization gates
Approval of this design authorizes documentation and planning only. It does not authorize:
- reading or importing the legacy credential;
- creating real keys;
- installing a binary or `systemd` unit;
- writing the registry directory;
- modifying or reloading Nginx;
- querying or changing PostgreSQL;
- changing, stopping, or removing the legacy ThothII stack.
Those actions require the executable plan, the remaining survey gates, explicit mutation
authorization, and retained sanitized evidence.