ThothII/docs/superpowers/specs/2026-08-20-dwh-rest-per-installation-auth-design.md

DWH REST Per-Installation Authentication — Design

Date: 2026-08-20

Status: Approved by the owner

Purpose

Replace the single static X-API-Key currently protecting the PSD /dwh/ route with a reusable, server-side authentication component that assigns one independently revocable credential to each ThothII installation.

The design must preserve the existing portable connector contract. A ThothII installation on macOS, Windows, or Linux continues to send X-API-Key over HTTPS when its selected transport is rest_api. Installations using postgres_direct or ssh_tunnel are unaffected.

The component belongs in the ThothII source repository so that it can be reused on another DWH server, but it is optional server infrastructure. It is not part of the portable application runtime and is not started or stopped by tht.

Confirmed context

• The exposed credential must be considered compromised and must eventually be revoked.
• The current PSD route accepts one shared static X-API-Key through Nginx auth_request.
• The Mac installation must continue to access PSD through REST.
• Future remote ThothII installations will also use REST and cannot depend on an SSH tunnel.
• The new ThothII installation on the PSD server will use postgres_direct with a dedicated, demonstrably read-only DWH role.
• The current self-issued TLS certificate remains in use. Clients that do not already trust its issuer require a separately delivered TLS_CA_FILE and must verify the approved fingerprint.
• There is no separate test environment. This is accepted and is not itself a blocker.
• Existing ThothII sessions, derived indexes, and model caches are test data. They are not migrated or protected by session-specific backup work.
• The legacy stack remains unchanged until the deployment program reaches its explicit stop gate. This is an authorization boundary, not a session-preservation requirement.

Scope

This design includes:

• the reusable dwh-auth service and its administrative command;
• a protected file-based credential registry;
• Nginx auth_request integration;
• one credential identity per ThothII installation;
• creation, delivery, rotation, optional expiry, and revocation;
• a dual-key transition from the exposed shared credential;
• generic and PSD-specific operator documentation;
• automated tests and production-safe acceptance checks.

This design does not:

• change the ThothII REST header or connector protocol;
• add DWH credential administration to the portable tht CLI;
• place server credentials in a workspace repository;
• replace PostgreSQL grants, RLS, or the PostgREST read-only role boundary;
• preserve or migrate legacy ThothII sessions, Qdrant indexes, or Ollama caches;
• change the current certificate or resolve the separate .it versus .com public-origin issue;
• authorize any current server, Nginx, database, or service mutation.

Alternatives considered

Extend the portable tht CLI

This would provide a single operator command surface, but it would distribute server-only DWH administration code to every macOS and Windows installation. The DWH route also has a lifecycle independent from the local ThothII application. This option is rejected.

Maintain a manual Nginx key map

This is initially small, but it encourages plaintext credentials in Nginx configuration and makes atomic rotation, redacted inventory, and reliable revocation harder. This option is rejected.

Use SQLite

SQLite is not a declared project dependency and is unnecessary for the expected registry size and write rate. Introducing a database would add packaging, migration, backup, and corruption-recovery work without improving the required contract. This option is rejected.

Selected approach

Add a standalone, standard-library Go component with a protected file registry. On PSD it runs as an independent systemd service and communicates with host Nginx through a Unix socket.

Component and lifecycle boundary

The repository will contain a separate component, expected under tools/dwh-auth/, plus generic deployment templates and documentation. Its build and tests are isolated from the portable application entrypoints.

On PSD:

• dwh-auth runs under a dedicated, unprivileged system account;
• systemd owns its lifecycle;
• the service is not included in the canonical ThothII Compose stack;
• the registry resides outside both the Git checkout and ThothII application data;
• the service receives read-only access to active credential records;
• Nginx is the only runtime caller and reaches it through a permission-restricted Unix socket;
• stopping, updating, or replacing ThothII does not interrupt /dwh/ authentication.

The same source may be deployed on another Linux DWH server. Platform-specific service packaging beyond the approved PSD systemd deployment is not required by this implementation.

Credential model

One credential identifies one ThothII installation, not one human user. An installation may have more than one temporarily active generation during rotation.

The external key contains a version marker, a public key ID, and a random secret:

thtdwh_v1.<public-key-id>.<random-secret>

Requirements:

• the public key ID is generated by the tool and contains only a strict safe alphabet;
• the secret is generated with the operating system cryptographic random source;
• the secret has at least 256 bits of entropy and uses an unambiguous transport-safe encoding;
• the complete header has a fixed maximum length;
• installation IDs are operator-provided, non-secret, unique labels and must not contain personal or clinical information;
• descriptions are optional non-secret operator metadata.

Because the secret is a uniformly random 256-bit value rather than a human password, the registry stores a SHA-256 digest and verifies it with a constant-time comparison. No password hashing or external cryptographic runtime dependency is needed.

Protected file registry

The registry has versioned active and revoked records:

<registry-root>/
  active/<public-key-id>.json
  revoked/<public-key-id>.json

An active record contains:

• schema version;
• public key ID;
• installation ID and optional description;
• encoded SHA-256 digest;
• creation timestamp;
• optional expiry timestamp, absent by default.

A revoked record additionally contains the revocation timestamp and a non-secret reason. The original key never appears in either directory.

Multiple key IDs may refer to the same installation ID during a rotation. Creation uses an exclusive temporary file, file flush, directory-safe atomic rename, restrictive ownership and mode checks, and refusal of symlinks or unsafe paths. Revocation atomically moves a record from active to revoked on the same filesystem. A malformed or unsafe record never authenticates.

The runtime service does not update registry files. Successful and failed use is recorded only in the sanitized service journal, avoiding per-request writes and keeping the service's registry access read-only.

Administrative interface

The standalone binary provides a server-only administrative surface equivalent to:

dwh-auth key create --installation-id ID --description TEXT --output ABSOLUTE_FILE
dwh-auth key import --installation-id ID --from-file ABSOLUTE_FILE
dwh-auth key list [--json]
dwh-auth key revoke --key-id ID --reason TEXT
dwh-auth key status --key-id ID [--json]
dwh-auth check

The exact syntax will be frozen in the implementation plan and tests. The interface must obey these invariants:

• secrets are never accepted as command-line values;
• create writes the generated credential once to a new absolute file with restrictive access;
• import reads an existing protected file without displaying its content;
• normal stdout contains only non-secret identifiers, abbreviated fingerprints, status, and paths;
• JSON output is pristine and contains no credential value or digest;
• an existing output file is never overwritten;
• listing and status commands never expose digests;
• errors and logs pass explicit secret-leak regression tests.

Provisioning and delivery

The operator creates one key for each installation and transfers it through an approved protected channel, such as an organizational password manager, a secret manager or MDM, or authenticated file transfer when the recipient has suitable access. Email, ordinary chat, and ticket bodies are not approved delivery channels.

On the recipient installation:

• the normal interactive path saves the value through authenticated Workspace management, where ThothII keeps it in the installation-local encrypted workspace vault;
• a supported headless deployment may place it in a protected file referenced by API_KEY_FILE;
• the credential never enters the shared workspace repository or ordinary environment values;
• the private CA is delivered separately and configured with TLS_CA_FILE when the certificate chain is not already trusted;
• the operator verifies the documented certificate fingerprint before trusting the CA file.

After configuration, the recipient runs the existing workspace connection test against the harmless /rpc/ping diagnostic. Only after the server and recipient both confirm the public key ID may the one-time delivery copy be removed according to the organization's secret-handling procedure.

Expiry and rotation

Keys have no automatic expiry by default. This avoids unplanned outages for remote installations that may not be continuously administered. An optional expiry timestamp is supported for sites whose policy requires it; an expired key is rejected without fallback. Administrative status output warns about approaching expiries without exposing secrets.

Normal zero-downtime rotation is:

1. create a second key generation for the installation;
2. deliver and configure it on the client;
3. verify /rpc/ping and the sanitized server key ID;
4. revoke the earlier key;
5. prove that the revoked key receives 401;
6. remove temporary delivery material after recipient confirmation.

Request flow and Nginx contract

remote ThothII
  -> HTTPS /dwh/ with X-API-Key
  -> Nginx auth_request subrequest
  -> dwh-auth over a Unix socket
  -> valid: Nginx proxies the original request to internal PostgREST
  -> invalid or revoked: Nginx returns 401
  -> authenticator fault: Nginx returns 503

dwh-auth authenticates only. It does not proxy the original request, read a clinical response, or connect to PostgreSQL. PostgREST remains inaccessible as a public direct backend.

Nginx integration must:

• forward only the bounded credential header to the internal verification endpoint;
• suppress the original request body in the auth subrequest;
• discard client-supplied identity or audit headers;
• accept only the authenticator's success result;
• return the same generic 401 for missing, malformed, unknown, expired, and revoked keys;
• map authenticator or registry faults to 503, never to permissive access;
• apply bounded failure-rate protection;
• log only timestamp, result, request metadata already approved for the route, and the verified public key ID;
• preserve the existing PostgREST path and response contract for successful requests.

The authenticator returns a verified public key ID only after successful authentication. Nginx may use that value in its sanitized access log but does not need to forward it to PostgREST.

Fail-closed behavior

The following conditions deny access:

• header absent, duplicated, oversized, malformed, or encoded incorrectly;
• unknown public key ID;
• digest mismatch;
• revoked or expired record;
• unsafe permissions, symlink, malformed JSON, schema mismatch, or inconsistent record identity;
• unreadable registry;
• service startup or runtime failure.

Credential failures return a generic 401. Infrastructure failures return 503 after the Nginx mapping is applied. Neither response reveals whether an installation ID exists. Health reporting is local-only and does not weaken the authentication decision.

PSD dual-key transition

The exposed shared credential is represented temporarily as legacy-shared. It is imported only from an approved protected source and is never placed in a command argument, terminal output, document, or evidence file. Because the existing value does not use the new versioned key format, it is stored as the only permitted legacy_raw record. The service compares the digest of the complete opaque legacy header only for that reserved record. After legacy-shared is revoked, no unversioned credential is accepted.

The production route does not switch to the new authenticator until all of the following hold:

1. protected backup and exact rollback instructions exist for the affected Nginx and service configuration;
2. the new service passes local synthetic checks while disconnected from the public route;
3. legacy-shared passes /rpc/ping through the candidate authentication path;
4. a new per-installation Mac key passes the same diagnostic;
5. a random invalid key is rejected;
6. nginx -t passes and the authorized operator approves reload.

After the Mac uses its new credential successfully for the agreed observation window, legacy-shared is revoked. Acceptance requires a positive test with the new key and a negative test proving 401 with the legacy key. The new secret must not appear in service, Nginx, shell, or application logs.

Rollback during the dual-key window restores only the reviewed authentication route and service configuration needed to keep REST clients working. It does not preserve or restore legacy ThothII sessions, indexes, or model caches.

PostgreSQL authorization boundary

Successful API-key authentication is not proof of read-only database authorization. The existing PostgREST role, exposed schemas, function privileges, default privileges, and direct network reachability remain subject to the catalog-only survey activity. Project execution cannot treat dwh-auth as a substitute for a demonstrably read-only dwh_reader boundary.

Verification strategy

Automated tests use synthetic credentials and no clinical data. They cover:

• key generation entropy and syntax;
• create, import, list, status, optional expiry, rotation, and revocation;
• constant-time digest verification;
• missing, duplicate, malformed, oversized, unknown, expired, and revoked credentials;
• corrupt records, unsafe modes, symlinks, path traversal, and partial files;
• atomic creation and revocation under concurrent reads;
• clean failure when the registry or socket is unavailable;
• absence of credential values and digests in human output, JSON, errors, and logs;
• Nginx auth_request integration against a synthetic upstream and /rpc/ping fixture;
• 401 for credential failures and 503 for infrastructure failures;
• unchanged successful request path and response body;
• independent service lifecycle from the ThothII stack.

Portability regression checks prove that:

• the existing REST connector still sends the same X-API-Key header;
• postgres_direct and ssh_tunnel configuration remain unchanged;
• canonical local and server ThothII Compose rendering does not acquire dwh-auth;
• macOS and Windows installation contracts do not require the new binary or files;
• existing tht, backend, frontend, and harness gates remain green in proportion to the touched source boundaries.

There is no separate PSD test environment. Local synthetic and integration tests precede a bounded, reversible production rollout using only /rpc/ping. The absence of a test environment does not trigger preservation work for legacy test sessions.

Documentation deliverables

Implementation is incomplete until the following clear, secret-free documents exist and pass their documentation checks:

1. a generic server installation and administration guide for dwh-auth;
2. a generic enrollment guide for issuing a key to a macOS, Windows, Linux, or headless ThothII installation;
3. exact GUI and API_KEY_FILE configuration paths;
4. a TLS guide covering TLS_CA_FILE, trusted delivery, fingerprint verification, renewal, and coordinated client updates;
5. a PSD runbook for backup, installation, legacy import, dual-key activation, Mac update, observation, revocation, negative testing, and rollback;
6. a troubleshooting table for 401, 503, TLS failures, revoked keys, and unreadable registry records;
7. sanitized evidence templates that never request secret values or digests;
8. links from the existing local, server, workspace, and final unified user manuals.

Examples use synthetic hostnames, IDs, fingerprints, and credentials. No real key, certificate body, password, connection string, or clinical result is committed.

Acceptance criteria

The design is implemented only when:

• each REST installation can be identified and revoked independently;
• no plaintext server credential is stored in the registry;
• the portable ThothII connector contract is unchanged;
• the PSD ThothII server continues to select postgres_direct;
• stopping ThothII does not stop dwh-auth or /dwh/ authentication;
• the authenticator fails closed and its logs are sanitized;
• the dual-key production procedure proves both positive and negative outcomes;
• the exposed legacy credential is demonstrably rejected;
• the current TLS limitations are documented precisely rather than silently bypassed;
• database read-only authorization is independently verified;
• all documentation deliverables are reviewed and reproducible by an operator who did not write the implementation.

Authorization gates

Approval of this design authorizes documentation and planning only. It does not authorize:

• reading or importing the legacy credential;
• creating real keys;
• installing a binary or systemd unit;
• writing the registry directory;
• modifying or reloading Nginx;
• querying or changing PostgreSQL;
• changing, stopping, or removing the legacy ThothII stack.

Those actions require the executable plan, the remaining survey gates, explicit mutation authorization, and retained sanitized evidence.