ThothII/PROJECT_STATE.md

ThothII — Project State

Last updated: 2026-08-27.

This file is the short operational snapshot. Stable commands and the architecture mental model live in AGENTS.md; current design and runtime contracts live under docs/architecture/, docs/contracts/, docs/adr/, and docs/evidence.md. Superseded plans and reports are available from Git history rather than duplicated in the working tree.

Current product shape

ThothII is a human-in-the-loop datamart builder with three independently built layers:

frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness → DWH (read-only)

The harness owns the deterministic eight-phase NL→SQL workflow and all session persistence. The backend remains a process/RPC/SSE bridge for sessions and now also owns an isolated PostgreSQL metadata catalog for administrative database configuration. The frontend renders the review gates and keeps the live transcript in memory. See docs/architecture/components.md for the detailed component and data-flow map.

Evidence restructuring — accepted

The evidence restructuring and PSD migration completed real acceptance on 2026-08-25.

• The curated PSD revision contains 35 approved Evidence units and 60 review items.
• The PSD authoring repository publishes all 35 units using Curated unit schema v3. Its table-free presentation uses hidden canonical metadata, wrapping Markdown scope lists, list-based enum values, and collapsed technical provenance. Long domain rules now have a deterministic human-readable presentation while retaining their exact canonical text for vector ingestion. tht evidence migrate <workspace-root> performs the deterministic v1/v2 upgrade and older-v3 presentation rewrite without model calls. The structured-rule PSD rewrite is currently local and pending commit/publication.
• The accepted snapshot is psd-clinical-675990d90eae51da6f2bd51b1ae2609f245772ef-snapshot.
• The active generation is gen:f968b3bd7a553dbfef3cf47093698f2bc7f95f11.
• Retrieval acceptance reached 20/20 Hit@10.
• A real session, 20301df7-cad7-403d-a4c1-9f35c9d07b66, completed F1–F8 with five receipts, three CTEs, and a final result of 78 patients.
• The durable acceptance record is docs/testing/evidence/evidence-restructuring-psd-acceptance-2026-08-25.md.

The canonical authoring, validation, publication, materialization, and preprocessing flow is documented in docs/evidence.md. The governing contracts are docs/contracts/workspace-evidence-v3.md and docs/contracts/workspace-preprocessing-cli.md.

Workspace preprocessing and configuration

The native host CLI tht is the operator surface. Workspace preprocessing runs through:

tht --installation /absolute/path/thothii-installation.yaml workspace preprocess evidence
tht --installation /absolute/path/thothii-installation.yaml workspace preprocess dwh

These commands use the profile-gated workspace-maintenance service. The former standalone preprocessing Compose fixtures are retired.

Workspace descriptors use schema v3. For PSD, workspace content and runtime roots point to the separate uncommitted repository /Users/mp/projects/tht-workspace-psd. Secrets remain outside Git and are supplied only through installation-local protected files.

Database management

The database, table, and authoritative physical-schema catalog slices are implemented. Database management opens a responsive AG Grid master-detail surface, lists every YAML workspace, creates at most one PostgreSQL database configuration per workspace, edits direct PostgreSQL, REST API, or SSH-tunnel installation bindings, replaces write-only encrypted secrets, and tests supported connector bindings.

Configured databases use pure hierarchical navigation through Overview, Tables, and Relationships; a selected table has Overview and Columns. Physical membership, source comments, column types/default/nullability/PK positions, and constraint-level ordered FK pairs are projections of the external schema. They cannot be created, renamed, or structurally edited by hand, but administrators can explicitly clear catalog tables, columns, or relationships without touching the source database, binding, configuration, or secrets. Table deletion cascades through columns and relationships; table-scoped relationship cleanup includes incoming and outgoing relationships. Curated and generated descriptions are editable; generated descriptions start null and AI generation/consolidation is deferred.

Schema refresh is one durable asynchronous engine with database-table, database-column, selected-table-column, relationship, and full-database actions. Database-level menus expose the table, all-column, relationship, and full scopes separately; selecting tables exposes column synchronization plus manual column and relationship cleanup for that subset. Database selections also expose manual table and relationship cleanup. Cleanup selections are atomic and share the one-active-operation-per-database exclusion with synchronization. Runs have leases and restart recovery, atomic apply, destructive-diff confirmation with re-scan, cancellation before apply, retained history, and a live SSE log with polling fallback. Null metadata renders blank rather than as a placeholder.

Direct PostgreSQL and strict known-host-verified OpenSSH use pg_catalog. REST bindings use the typed full-snapshot POST /rpc/schema_snapshot contract when available. Servers such as the current PSD endpoint that exposes only POST /rpc/run_query use one catalog-owned read-only query to return the exact same strict v1 snapshot in a single round trip. Both paths remain fail-closed: an absent capability, query error, partial result, or invalid snapshot applies no catalog changes. SSH is not yet enabled for NL→SQL session runtime.

The catalog runs in the internal catalog-db PostgreSQL service. Kysely migrations are an explicit one-shot catalog-migrate operation; scripts/run-stack.sh runs it before local startup. Runtime sessions still consume the existing workspace configuration in this slice: database-management records do not yet change the NL→SQL handoff. The accepted design is recorded in docs/plans/2026-08-26-metadata-catalog-from-thothai.md, the snapshot contract under docs/contracts/, and ADRs 0001–0008.

Semantic aliases, value descriptions, synonyms, concepts, AI metadata generation/consolidation, and logical relationships remain deferred to their dedicated slices.

Integration of the completed metadata catalog with core schema-linking is explicitly deferred until the database, table, column, relationship, and synchronization slices are complete. At that point the next required design gate is to compare the catalog snapshot with the current DWH preprocessing/schema-linking contracts and plan the cutover; this follow-up must not be treated as optional cleanup or silently omitted.

Active deployment work and manual gates

PSD server deployment program

The approved design and executable entry point are:

• docs/plans/2026-08-20-psd-server-deployment-program-design.md
• docs/plans/2026-08-20-psd-server-deployment-program.md
• docs/plans/2026-08-20-psd-server-survey.md
• docs/plans/2026-08-20-psd-server-project-a-standalone.md
• docs/plans/2026-08-20-psd-server-project-b-authentik.md

Last recorded state:

• survey: SURVEY_NO_GO;
• Project A: BLOCKED_BY_SURVEY_AND_MUTATION_GATE;
• Project B: BLOCKED_BY_PROJECT_A_AND_PRE_B_GATE.

The deployment is a clean replacement: legacy sessions, indexes, and application configuration are not migration inputs. The existing stack remains intact until its documented mutation and rollback gates are explicitly approved. Shared Omics/LocalLLM networks, ETL Evidence, DWH, dwh-auth, Supabase, Authentik, Superset, and Aritmolab are outside cleanup scope.

Human acceptance guides and sanitized report templates live under docs/testing/ and docs/testing/evidence/. The remediation checklist is docs/operations/psd-server-survey-remediation-checklist.md.

Authentication

The local/OIDC authentication remediation passed its automated review on 2026-08-18. Release and PSD mutation gates remain governed by:

• docs/architecture/authentication.md;
• docs/plans/2026-08-18-thothii-authentication-acceptance-and-psd-deployment.md;
• docs/operations/psd-dwh-auth-rollout.md;
• docs/testing/authentication-manual-acceptance.md.

Do not infer authorization for server, Nginx, Authentik, database, credential, or cutover changes from an automated PASS.

Verification status

• The P1.1 workspace-directory registry and P2–P6 preprocessing workstreams are implemented and have automated coverage.
• Evidence restructuring has a real PSD acceptance PASS as recorded above.
• L2 tests requiring real providers or remote databases remain opt-in.
• Server deployment, release, and owner-operated acceptance steps remain pending wherever the referenced runbooks require explicit approval.

Run the layer-specific checks documented in AGENTS.md. For release-sensitive changes, also run the repository contract scripts in scripts/ and build the MkDocs site.

Operational invariants

• tht's -c/--config option follows the subcommand; it is not a global option.
• --json commands write pristine JSON to stdout.
• Persisted phase documents and the decision ledger are the source of session truth; chat is not.
• UI chrome is English; workspace document content retains the workspace language.
• The backend refuses resume for finalized or archived sessions.
• A resume must send /riprendi-sessione <id>; a new session must send /nuova-domanda.
• DWH access is read-only.