# ThothII — Project State Last updated: 2026-09-02. This file is the short operational snapshot. Stable commands and the architecture mental model live in `AGENTS.md`; current design and runtime contracts live under `docs/architecture/`, `docs/contracts/`, `docs/adr/`, and `docs/evidence.md`. Superseded plans and reports are available from Git history rather than duplicated in the working tree. ## Current product shape ThothII is a human-in-the-loop datamart builder with three independently built layers: ```text frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness → DWH (read-only) ``` The harness owns the deterministic eight-phase NL→SQL workflow and all session persistence. The backend remains a process/RPC/SSE bridge for sessions and now also owns an isolated PostgreSQL metadata catalog for administrative database configuration. The frontend renders the review gates and keeps the live transcript in memory. See `docs/architecture/components.md` for the detailed component and data-flow map. ## Evidence restructuring — accepted The evidence restructuring and PSD migration completed real acceptance on 2026-08-25. - The curated PSD revision contains 35 approved Evidence units and 60 review items. - The PSD authoring repository publishes all 35 units using Curated unit schema v3. Its table-free presentation uses hidden canonical metadata, wrapping Markdown scope lists, list-based enum values, and collapsed technical provenance. Long domain rules now have a deterministic human-readable presentation while retaining their exact canonical text for vector ingestion. `tht evidence migrate ` performs the deterministic v1/v2 upgrade and older-v3 presentation rewrite without model calls. The structured-rule PSD rewrite is currently local and pending commit/publication. - The accepted snapshot is `psd-clinical-675990d90eae51da6f2bd51b1ae2609f245772ef-snapshot`. - The active generation is `gen:f968b3bd7a553dbfef3cf47093698f2bc7f95f11`. - Retrieval acceptance reached 20/20 Hit@10. - A real session, `20301df7-cad7-403d-a4c1-9f35c9d07b66`, completed F1–F8 with five receipts, three CTEs, and a final result of 78 patients. - The durable acceptance record is `docs/testing/evidence/evidence-restructuring-psd-acceptance-2026-08-25.md`. The canonical authoring, validation, publication, materialization, and preprocessing flow is documented in `docs/evidence.md`. The governing contracts are `docs/contracts/workspace-evidence-v3.md` and `docs/contracts/workspace-preprocessing-cli.md`. ## Workspace preprocessing and configuration The native host CLI `tht` is the operator surface. Workspace preprocessing runs through: ```sh tht --installation /absolute/path/thothii-installation.yaml workspace preprocess evidence tht --installation /absolute/path/thothii-installation.yaml workspace preprocess dwh ``` These commands use the profile-gated `workspace-maintenance` service. The former standalone preprocessing Compose fixtures are retired. Workspace descriptors use schema v4 and contain only database, Evidence, diagnostics, and binding concerns; model, provider, embedding, and vector-store configuration is installation-owned. For PSD, workspace content and runtime roots point to the separate uncommitted repository `/Users/mp/projects/tht-workspace-psd`. Secrets remain outside Git and are supplied only through installation-local protected files. ## Installation Model Catalog `thothii-installation.yaml` schema version 2 is the only operator-authored source for session, metadata-generation, and embedding models. The host `tht` lifecycle validates `modelCatalog` and regenerates the backend catalog, Pi `models.json`/`settings.json`, and Compose override under the installation-local `generated/` directory. Those projections are replaceable runtime adapters: they are not edited, backed up, or treated as configuration. Session and metadata defaults use canonical `provider/model` IDs. Provider authentication declares one explicit mode (`secret_env`, `pi_auth`, or `none`); `secret_env` names a protected bundle key. The backend settings store now owns only the selected workspace and thinking level. Existing v1 installations use the explicit catalog migration command; schema-v3 workspace descriptors are converted deterministically in their curator-owned repository before commit. Strict runtime loading does not silently infer or merge legacy sources. ADR 0013 and `docs/plans/2026-09-02-installation-model-catalog.md` record the decision and implementation. ## Database management The database, table, and authoritative physical-schema catalog slices are implemented. Database Management now opens the Fleet Ledger presentation by default inside `AppShell`, lists every YAML workspace, creates at most one PostgreSQL database configuration per workspace, edits direct PostgreSQL, REST API, or SSH-tunnel installation bindings, replaces write-only encrypted secrets, and tests supported connector bindings. The surface keeps one responsive AG Grid visible at a time: databases lead to tables, tables lead to columns, and relationships are a sibling database view. Parent navigation remains explicit through the breadcrumb and emphasized back control. Selection-scoped operations use one action selector plus an explicit **Run** control; ineligible actions remain visible with their disabled reason, while row-scoped actions stay in the pinned final column. The KPI strip reads installation-wide or selected-database aggregates from `GET /catalog/metrics`. Database configuration, metadata editors, synchronization history, description history, and sensitive-field review/history use the production APIs in right-side drawers rather than prototype fixtures; closing a history drawer does not stop its background run. Sensitive-field review is now driven by the versioned local `sensitivity-v2` policy, not by a catalog model. The backend reads selected source tables through read-only, database-specific adapters and makes every `sensitive | non_sensitive` draft decision in the TypeScript `SensitivityClassifier`. A single validated match protects the column. Tables up to 1,000 rows are fully scanned; larger tables use breadth-first 300, 1,000, and text-only 3,000-value targets, with a five-second limit per source query and no global request deadline. Source failures fail the run instead of yielding `unknown`; coverage remains visible separately from the proposal. Draft assessments remain transient until an administrator explicitly saves them. Optional GLiNER2 evidence is CPU-only, offline, opt-in, and never replaces the deterministic decision point; see `docs/operations/sensitivity-analysis.md`. The earlier v1 PSD shadow comparison kept NER disabled by default; see `docs/reports/2026-09-02-psd-sensitivity-shadow.md`. The v2 comparison completed all 2,275 columns: CPU NER added 18 sensitive proposals and increased warm runtime from 50.1 to 61.3 seconds; see `docs/reports/2026-09-03-psd-progressive-sensitivity-shadow.md`. Physical membership, source comments, column types/default/nullability/PK positions, and constraint-level ordered FK pairs are projections of the external schema. They cannot be created, renamed, or structurally edited by hand, but administrators can explicitly clear catalog tables, columns, or relationships without touching the source database, binding, configuration, or secrets. Table deletion cascades through columns and relationships; table-scoped relationship cleanup includes incoming and outgoing relationships. Curated and generated descriptions are editable; generated descriptions start null and Database Management can generate or consolidate them for selected tables, selected columns, all targets, or only targets whose Generated Description is missing. Relationship Management is now reachable directly from each configured Fleet database. One Relationship Map shows read-only Physical Relationships together with Generated and Manual Logical Relationships, with Active, Excluded, and All filters. Administrators can add a single-column relationship, run deterministic name/PK/type inference, exclude or restore a logical relationship, or delete it permanently. Exclusion retains a tombstone that a rebuild cannot reactivate; permanent deletion allows a later rebuild to infer the same endpoints again. Inference uses no LLM, embedding, or source values. It supports normalized table-qualified names, unique non-generic PK names, composite-PK source columns, and the `*time_key -> dim_time.` warehouse convention while ignoring bare generic names. Explicit table/column metadata cleanup remains a destructive boundary: it removes the attached logical relationships and exclusions and requires a full schema synchronization before inference or runtime publication can continue. The previous Database Management renderer remains a temporary comparison fallback for development and staging only: `?db-ui=legacy` is honored in Vite development or when `VITE_DB_MANAGEMENT_LEGACY=true`; it is not a production presentation. The standalone Fleet Ledger prototype on port `5173` also remains temporary until owner acceptance of the integrated surface, after which both migration aids can be removed. Schema refresh is one durable asynchronous engine with database-table, selected-table-column, relationship, and full-database actions. Database-level menus expose only the table, relationship, and full scopes; selecting tables exposes column synchronization plus manual column and relationship cleanup for that subset. Database selections also expose manual table and relationship cleanup. Cleanup selections are atomic and share the one-active-operation-per-database exclusion with synchronization. Runs have leases and restart recovery, atomic apply, destructive-diff confirmation with re-scan, cancellation before apply, retained history, and a live SSE log with polling fallback. Null metadata renders blank rather than as a placeholder. Direct PostgreSQL and strict known-host-verified OpenSSH use `pg_catalog`. REST bindings use the typed full-snapshot `POST /rpc/schema_snapshot` contract when available. Servers such as the current PSD endpoint that exposes only `POST /rpc/run_query` use one catalog-owned read-only query to return the exact same strict v1 snapshot in a single round trip. Both paths remain fail-closed: an absent capability, query error, partial result, or invalid snapshot applies no catalog changes. SSH is not yet enabled for NL→SQL session runtime. The catalog runs in the internal `catalog-db` PostgreSQL service. Kysely migrations are an explicit one-shot `catalog-migrate` operation; `scripts/run-stack.sh` runs it before local startup. Runtime sessions now consume the Catalog's active effective relationship map through an immutable JSON snapshot tied to the runtime-config lease. The harness uses that snapshot as its exclusive relationship source while retaining Git-pinned annotations for descriptive metadata; legacy runtimes without a snapshot keep the previous merge behavior. The accepted design is recorded in `docs/plans/2026-08-26-metadata-catalog-from-thothai.md`, the snapshot contract under `docs/contracts/`, and ADRs 0001–0012. Semantic aliases, value descriptions, synonyms, and concepts remain deferred to their dedicated slices. AI Description Generation uses the catalog's human-owned Sensitive Data Flag. The flag defaults to `false`, including for newly synchronized columns. An administrator may request a local sensitivity analysis for one selected database, selected tables, or selected columns. One deterministic TypeScript classifier combines metadata, bounded source-content rules, and optional CPU-only NER; no generative model decides the result. Its `sensitive` or `non_sensitive` assessments remain an unsaved draft until the human reviews and saves any chosen flag changes, including a downgrade to non-sensitive. Coverage is reported separately; interrupted history may count unprocessed columns. Each started analysis records a separate Sensitivity Analysis Run with aggregate counters and safe ordered events. This operational history never stores per-column assessments, source values, matched spans, prompts, or free-form diagnostics; reloading still discards an unsaved review draft. For unprotected columns, up to five source rows and five representative non-null values may be sent transiently to the configured model provider. Protected columns are omitted from source reads and replaced in the prompt by deterministic plausible values derived only from their metadata. Existing descriptions are not regenerated when a flag changes. The accepted AI-description design is recorded in `docs/plans/2026-08-28-ai-catalog-description-generation.md`, with the formal specification in the adjacent `-spec.md` document and Gitea issue #4. Gitea issues #5–#11 deliver the implementation. The runtime deliberately keeps ThothAI's simple operating model: one installation-wide sequential run owned by the backend, one short-lived Python/LiteLLM completion helper per request, and persistence limited to the run, its safe ordered text events, and each Generated Description as soon as it succeeds. The helper performs at most one provider retry and never falls back to another model. Stop terminates the current helper and retains prior results; three consecutive exhausted technical batches fail the run. Startup marks stale queued/running work interrupted, and Unlock is available only when no local start, worker, or helper is live. Runs remain inspectable through a live SSE log with ordered polling fallback; there is no automatic resume or user-facing generation CLI. ADRs 0009–0010 record the runtime and source-sampling decisions. The Installation Model Catalog accepts the protected `DEEPSEEK_API_KEY` and `ZAI_API_KEY` references for metadata-generation providers. It also accepts a model with no secret reference only when its OpenAI-compatible endpoint is explicit; this covers the VPN-only AritmoLab Qwen 3.6 server without creating a fake operator credential. The Python client supplies only its fixed non-secret compatibility placeholder. The AritmoLab entry also sets `disableThinking: true`, mapped to the endpoint's chat-template flag, because its default reasoning prose would violate the worker's exact JSON response contract. Logical relationship integration with core schema-linking is complete: session creation and resume materialize the active physical/generated/manual map, retrieval-pack generation and Pi receive the same runtime config, and snapshot validation fails closed on a declared missing, invalid, or orphaned endpoint. Broader publication of other Catalog metadata to schema-linking remains a separate future slice. **Deferred follow-up — Sensitive Data Policy in schema-linking.** The policy is first delivered and tested in catalog description generation. Its enforcement for core schema-linking remains out of scope until the current tickets are closed and the owner has completed the acceptance test. At that gate, resume the design: `tht` must receive a read-only projection of the current Sensitive Data Flags and exclude values from columns marked sensitive from every LSH result before it is given to Pi. Do not start this integration before the owner gives final approval after that test. ## Active deployment work and manual gates ### PSD server deployment program The approved design and executable entry point are: - `docs/plans/2026-08-20-psd-server-deployment-program-design.md` - `docs/plans/2026-08-20-psd-server-deployment-program.md` - `docs/plans/2026-08-20-psd-server-survey.md` - `docs/plans/2026-08-20-psd-server-project-a-standalone.md` - `docs/plans/2026-08-20-psd-server-project-b-authentik.md` Last recorded state: - survey: `SURVEY_NO_GO`; - Project A: `BLOCKED_BY_SURVEY_AND_MUTATION_GATE`; - Project B: `BLOCKED_BY_PROJECT_A_AND_PRE_B_GATE`. The deployment is a clean replacement: legacy sessions, indexes, and application configuration are not migration inputs. The existing stack remains intact until its documented mutation and rollback gates are explicitly approved. Shared Omics/LocalLLM networks, ETL Evidence, DWH, `dwh-auth`, Supabase, Authentik, Superset, and Aritmolab are outside cleanup scope. Human acceptance guides and sanitized report templates live under `docs/testing/` and `docs/testing/evidence/`. The remediation checklist is `docs/operations/psd-server-survey-remediation-checklist.md`. ### Authentication The local/OIDC authentication remediation passed its automated review on 2026-08-18. Release and PSD mutation gates remain governed by: - `docs/architecture/authentication.md`; - `docs/plans/2026-08-18-thothii-authentication-acceptance-and-psd-deployment.md`; - `docs/operations/psd-dwh-auth-rollout.md`; - `docs/testing/authentication-manual-acceptance.md`. Do not infer authorization for server, Nginx, Authentik, database, credential, or cutover changes from an automated PASS. ## Verification status - The P1.1 workspace-directory registry and P2–P6 preprocessing workstreams are implemented and have automated coverage. - Evidence restructuring has a real PSD acceptance PASS as recorded above. - AI Description Generation has automated coverage across installation setup, model selection, generation/consolidation scopes, bounded sampling, cancellation/recovery, history, SSE/polling, and the LiteLLM helper boundary. - L2 tests requiring real providers or remote databases remain opt-in. - Server deployment, release, and owner-operated acceptance steps remain pending wherever the referenced runbooks require explicit approval. Run the layer-specific checks documented in `AGENTS.md`. For release-sensitive changes, also run the repository contract scripts in `scripts/` and build the MkDocs site. ## Operational invariants - `tht`'s `-c`/`--config` option follows the subcommand; it is not a global option. - `--json` commands write pristine JSON to stdout. - Persisted phase documents and the decision ledger are the source of session truth; chat is not. - UI chrome is English; workspace document content retains the workspace language. - The backend refuses resume for finalized or archived sessions. - A resume must send `/riprendi-sessione `; a new session must send `/nuova-domanda`. - DWH access is read-only.