ThothII/PROJECT_STATE.md

# ThothII — Project State

Last updated: 2026-08-27.

This file is the short operational snapshot. Stable commands and the architecture mental model
live in `AGENTS.md`; current design and runtime contracts live under `docs/architecture/`,
`docs/contracts/`, `docs/adr/`, and `docs/evidence.md`. Superseded plans and reports are
available from Git history rather than duplicated in the working tree.

## Current product shape

ThothII is a human-in-the-loop datamart builder with three independently built layers:

```text
frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness → DWH (read-only)
```

The harness owns the deterministic eight-phase NL→SQL workflow and all session persistence.
The backend remains a process/RPC/SSE bridge for sessions and now also owns an isolated PostgreSQL
metadata catalog for administrative database configuration. The frontend renders the review gates
and keeps the live transcript in memory. See
`docs/architecture/components.md` for the detailed component and data-flow map.

## Evidence restructuring — accepted

The evidence restructuring and PSD migration completed real acceptance on 2026-08-25.

- The curated PSD revision contains 35 approved Evidence units and 60 review items.
- The PSD authoring repository publishes all 35 units using Curated unit schema v3. Its table-free
  presentation uses hidden canonical metadata, wrapping Markdown scope lists, list-based enum
  values, and collapsed technical provenance. Long domain rules now have a deterministic
  human-readable presentation while retaining their exact canonical text for vector ingestion.
  `tht evidence migrate <workspace-root>` performs the deterministic v1/v2 upgrade and older-v3
  presentation rewrite without model calls. The structured-rule PSD rewrite is currently local and
  pending commit/publication.
- The accepted snapshot is
  `psd-clinical-675990d90eae51da6f2bd51b1ae2609f245772ef-snapshot`.
- The active generation is `gen:f968b3bd7a553dbfef3cf47093698f2bc7f95f11`.
- Retrieval acceptance reached 20/20 Hit@10.
- A real session, `20301df7-cad7-403d-a4c1-9f35c9d07b66`, completed F1–F8 with five
  receipts, three CTEs, and a final result of 78 patients.
- The durable acceptance record is
  `docs/testing/evidence/evidence-restructuring-psd-acceptance-2026-08-25.md`.

The canonical authoring, validation, publication, materialization, and preprocessing flow is
documented in `docs/evidence.md`. The governing contracts are
`docs/contracts/workspace-evidence-v3.md` and
`docs/contracts/workspace-preprocessing-cli.md`.

## Workspace preprocessing and configuration

The native host CLI `tht` is the operator surface. Workspace preprocessing runs through:

```sh
tht --installation /absolute/path/thothii-installation.yaml workspace preprocess evidence
tht --installation /absolute/path/thothii-installation.yaml workspace preprocess dwh
```

These commands use the profile-gated `workspace-maintenance` service. The former standalone
preprocessing Compose fixtures are retired.

Workspace descriptors use schema v3. For PSD, workspace content and runtime roots point to the
separate uncommitted repository `/Users/mp/projects/tht-workspace-psd`. Secrets remain outside
Git and are supplied only through installation-local protected files.

## Database management

The database, table, and authoritative physical-schema catalog slices are implemented. Database
management opens a responsive AG Grid master-detail surface, lists every YAML workspace, creates
at most one PostgreSQL database configuration per workspace, edits direct PostgreSQL, REST API, or
SSH-tunnel installation bindings, replaces write-only encrypted secrets, and tests supported
connector bindings.

Configured databases use pure hierarchical navigation through `Overview`, `Tables`, and
`Relationships`; a selected table has `Overview` and `Columns`. Physical membership, source
comments, column types/default/nullability/PK positions, and constraint-level ordered FK pairs are
projections of the external schema. They cannot be created, renamed, or structurally edited by
hand, but administrators can explicitly clear catalog tables, columns, or relationships without
touching the source database, binding, configuration, or secrets. Table deletion cascades through
columns and relationships; table-scoped relationship cleanup includes incoming and outgoing
relationships. Curated and generated descriptions are editable; generated descriptions start null
and Database Management can generate or consolidate them for selected tables, selected columns,
all targets, or only targets whose Generated Description is missing.

Schema refresh is one durable asynchronous engine with database-table, database-column,
selected-table-column, relationship, and full-database actions. Database-level menus expose the
table, all-column, relationship, and full scopes separately; selecting tables exposes column
synchronization plus manual column and relationship cleanup for that subset. Database selections
also expose manual table and relationship cleanup. Cleanup selections are atomic and share the
one-active-operation-per-database exclusion with synchronization. Runs have leases and
restart recovery, atomic apply, destructive-diff confirmation with re-scan, cancellation before
apply, retained history, and a live SSE log with polling fallback. Null metadata renders blank
rather than as a placeholder.

Direct PostgreSQL and strict known-host-verified OpenSSH use `pg_catalog`. REST bindings use the
typed full-snapshot `POST /rpc/schema_snapshot` contract when available. Servers such as the
current PSD endpoint that exposes only `POST /rpc/run_query` use one catalog-owned read-only query
to return the exact same strict v1 snapshot in a single round trip. Both paths remain fail-closed:
an absent capability, query error, partial result, or invalid snapshot applies no catalog changes.
SSH is not yet enabled for NL→SQL session runtime.

The catalog runs in the internal `catalog-db` PostgreSQL service. Kysely migrations are an explicit
one-shot `catalog-migrate` operation; `scripts/run-stack.sh` runs it before local startup. Runtime
sessions still consume the existing workspace configuration in this slice: database-management
records do not yet change the NL→SQL handoff. The accepted design is recorded in
`docs/plans/2026-08-26-metadata-catalog-from-thothai.md`, the snapshot contract under
`docs/contracts/`, and ADRs 0001–0011.

Semantic aliases, value descriptions, synonyms, concepts, and logical relationships remain
deferred to their dedicated slices.

AI Description Generation uses the catalog's human-owned Sensitive Data Flag. The flag defaults to
`false`, including for newly synchronized columns. An administrator may request an AI proposal based
only on structural metadata for one selected database, selected tables, or selected columns. The
backend divides large scopes into deterministic model requests of at most ten columns, also bounded
by helper message size, and combines their results, but the proposal remains an unsaved draft until
the human reviews and saves it.
For unprotected columns, up to five source rows and five representative non-null values may be sent
transiently to the configured model provider. Protected columns are omitted from source reads and
replaced in the prompt by deterministic plausible values derived only from their metadata. Existing
descriptions are not regenerated when a flag changes.

The accepted AI-description design is recorded in
`docs/plans/2026-08-28-ai-catalog-description-generation.md`, with the formal specification in the
adjacent `-spec.md` document and Gitea issue #4. Gitea issues #5–#11 deliver the implementation.
The runtime deliberately keeps ThothAI's simple operating model: one installation-wide sequential
run owned by the backend, one short-lived Python/LiteLLM completion helper per request, and
persistence limited to the run, its safe ordered text events, and each Generated Description as
soon as it succeeds. The helper performs at most one provider retry and never falls back to another
model. Stop terminates the current helper and retains prior results; three consecutive exhausted
technical batches fail the run. Startup marks stale queued/running work interrupted, and Unlock is
available only when no local start, worker, or helper is live. Runs remain inspectable through a
live SSE log with ordered polling fallback; there is no automatic resume or user-facing generation
CLI. ADRs 0009–0010 record the runtime and source-sampling decisions.

Metadata-generation setup accepts the protected `DEEPSEEK_API_KEY` and `ZAI_API_KEY` references.
It also accepts a model with no secret reference only when its OpenAI-compatible endpoint is
explicit; this covers the VPN-only AritmoLab Qwen 3.6 server without creating a fake operator
credential. The Python client supplies only its fixed non-secret compatibility placeholder.
The AritmoLab entry also sets `disableThinking: true`, mapped to the endpoint's chat-template flag,
because its default reasoning prose would violate the worker's exact JSON response contract.

Integration of the completed metadata catalog with core schema-linking is explicitly deferred
until the database, table, column, relationship, and synchronization slices are complete. At that
point the next required design gate is to compare the catalog snapshot with the current DWH
preprocessing/schema-linking contracts and plan the cutover; this follow-up must not be treated as
optional cleanup or silently omitted.

**Deferred follow-up — Sensitive Data Policy in schema-linking.** The policy is first delivered
and tested in catalog description generation. Its enforcement for core schema-linking remains
out of scope until the current tickets are closed and the owner has completed the acceptance test.
At that gate, resume the design: `tht` must receive a read-only projection of the current Sensitive
Data Flags and exclude values from columns marked sensitive from every LSH result before it is
given to Pi. Do not start this integration before the owner gives final approval after that test.

## Active deployment work and manual gates

### PSD server deployment program

The approved design and executable entry point are:

- `docs/plans/2026-08-20-psd-server-deployment-program-design.md`
- `docs/plans/2026-08-20-psd-server-deployment-program.md`
- `docs/plans/2026-08-20-psd-server-survey.md`
- `docs/plans/2026-08-20-psd-server-project-a-standalone.md`
- `docs/plans/2026-08-20-psd-server-project-b-authentik.md`

Last recorded state:

- survey: `SURVEY_NO_GO`;
- Project A: `BLOCKED_BY_SURVEY_AND_MUTATION_GATE`;
- Project B: `BLOCKED_BY_PROJECT_A_AND_PRE_B_GATE`.

The deployment is a clean replacement: legacy sessions, indexes, and application configuration
are not migration inputs. The existing stack remains intact until its documented mutation and
rollback gates are explicitly approved. Shared Omics/LocalLLM networks, ETL Evidence, DWH,
`dwh-auth`, Supabase, Authentik, Superset, and Aritmolab are outside cleanup scope.

Human acceptance guides and sanitized report templates live under `docs/testing/` and
`docs/testing/evidence/`. The remediation checklist is
`docs/operations/psd-server-survey-remediation-checklist.md`.

### Authentication

The local/OIDC authentication remediation passed its automated review on 2026-08-18. Release and
PSD mutation gates remain governed by:

- `docs/architecture/authentication.md`;
- `docs/plans/2026-08-18-thothii-authentication-acceptance-and-psd-deployment.md`;
- `docs/operations/psd-dwh-auth-rollout.md`;
- `docs/testing/authentication-manual-acceptance.md`.

Do not infer authorization for server, Nginx, Authentik, database, credential, or cutover changes
from an automated PASS.

## Verification status

- The P1.1 workspace-directory registry and P2–P6 preprocessing workstreams are implemented and
  have automated coverage.
- Evidence restructuring has a real PSD acceptance PASS as recorded above.
- AI Description Generation has automated coverage across installation setup, model selection,
  generation/consolidation scopes, bounded sampling, cancellation/recovery, history, SSE/polling,
  and the LiteLLM helper boundary.
- L2 tests requiring real providers or remote databases remain opt-in.
- Server deployment, release, and owner-operated acceptance steps remain pending wherever the
  referenced runbooks require explicit approval.

Run the layer-specific checks documented in `AGENTS.md`. For release-sensitive changes, also run
the repository contract scripts in `scripts/` and build the MkDocs site.

## Operational invariants

- `tht`'s `-c`/`--config` option follows the subcommand; it is not a global option.
- `--json` commands write pristine JSON to stdout.
- Persisted phase documents and the decision ledger are the source of session truth; chat is not.
- UI chrome is English; workspace document content retains the workspace language.
- The backend refuses resume for finalized or archived sessions.
- A resume must send `/riprendi-sessione <id>`; a new session must send `/nuova-domanda`.
- DWH access is read-only.