mptyl/ThothII
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
90514636545ada5b2da449e444816270a9f7150e
ThothII/PROJECT_STATE.md
T

45 KiB
Raw Blame History

ThothII — Project State

Last updated: 2026-09-13.

This file is the short operational snapshot. Stable commands and the architecture mental model live in AGENTS.md; current design and runtime contracts live under docs/architecture/, docs/contracts/, docs/adr/, and docs/evidence.md. Superseded plans and reports are available from Git history rather than duplicated in the working tree.

The guarded server migration from a legacy checkout to the schema-v2 installation, Gitea source, Authentik, internal catalog/embedding services, and the PSD workspace repository is documented in docs/operations/server-upgrade-gitea-workspace-v2.md. Treat its operator gates and rollback requirements as mandatory; do not replace the running server stack in place.

Full/embedded shell and bilingual interface

Full/embedded shell, EN/IT UI, immutable session interaction language, dark theme, fullscreen and full-mode logout are implemented. The local Mac descriptor explicitly sets shell.mode: full and shell.defaultLocale: en; the native tht and local core/frontend images were updated on 2026-09-13. Omics uses embedded with server-side identity verification and a replaceable presentation-only PortalAdapter; this does not mean this branch was verified on the production server. Its changes are in Omics commit 95154e1; production deployment remains pending. See docs/operations/shell-and-localization.md for integration and installation instructions and docs/reports/2026-09-13-full-shell-implementation.md for tests, independent reviews, local browser checks and rollback details.

Documentation handoff before branch closure — 2026-09-13

Current rendering architecture is in docs/architecture/application-shell.md; the exact portal identity/proxy contract is in docs/install/authentication-upstream.md. docs/operations/shell-and-localization.md is the configuration and coordinated Omics delivery/deploy runbook, including server-side fetch GitHub → explicit push Gitea PSD from /home/chirone/omics_portal. The acceptance matrix is docs/testing/authentication-manual-acceptance.md. README, documentation navigation, user/installation/authentication guides and descriptor examples point to these paths. Local examples explicitly use full/en; the projected server example is for standalone OIDC, not Omics upstream. No runtime configuration or deployment was changed by this documentation pass. Closing/merging the branch and actual PSD acceptance remain separate, unperformed steps.

Agreed Omics delivery route — 2026-09-13

The owner approved GitHub as an intermediate transport for Omics: publish only codex/thothii-embedded-shell from the Mac to https://github.com/Dallavilla-Tiziano/omics_portal.git. The operator then works in /home/chirone/omics_portal on the PSD server, fetches that branch without changing the shared checkout, verifies the delivered SHA, and pushes only that branch to ssh://git@localhost:2222/aritmolab/omics_portal.git using existing server credentials. Do not require a PSD Gitea token on the Mac to continue. Do not use the server's dual-push origin or merge into its current branch as part of transport. Production integration/rebuild is a separate operator gate. ThothII still uses its canonical TYL Gitea; this exception is for Omics only. Exact commands and the handoff are linked from docs/operations/shell-and-localization.md; the full server procedure lives in Omics docs/thothii-integration.md. Never report the PSD push or production deployment as complete until the operator supplies confirmation.

GitHub delivery verified on 2026-09-13 at fca10901a73666ca257d8f4cc4b77066295c400a (functional commit 95154e1 plus the server relay runbook). GitHub master remains aff7581. The operator's PSD Gitea push and production deployment are still pending.

Current product shape

Shared Memory/Evidence typography — 2026-09-13

Both detail readers now share Manrope and fixed reading roles: 24px card title, 20px section headings, 16px/1.65 narrative text, and 14px metadata/code. Authored Markdown subheadings remain subordinate to field headings; inline/fenced code no longer shrinks cumulatively. Full-width layout, paragraph separation and isolated FAKE examples are retained. All 762 frontend tests and 18 browser scenarios pass, including computed typography checks at 390/1280/2400px; typecheck, i18n and Docker build pass. Only Mac frontend was recreated: 87fca0dc5e19, image 19f1704b6bb2, healthy. Core and data services are unchanged. Rollback image: thothii-frontend:before-knowledge-typography-20260913. See docs/reports/2026-09-13-knowledge-typography.md. No PSD/Omics deployment.

Knowledge reading and isolated fake Memory examples — 2026-09-13

Memory/Evidence details now use all available width, with display-only paragraph splitting for long plain prose and preserved code/Markdown/source data. Evidence provenance renders Markdown; copy controls are accessible two-sheet icons. Memory's separate Formatting examples section contains four FAKE cards (one per family), automatically expanded for an empty archive. These client-side examples cannot be edited/saved/indexed and are never submitted to the model or Memory API. No real archive content was changed. 762 tests, 18 browser scenarios, typecheck, i18n and Docker build pass. Only the Mac frontend was recreated: fc4286b5406d, image cbe0fe0dce55, healthy; other services unchanged. Rollback image: thothii-frontend:before-knowledge-reading-20260913. Details in docs/reports/2026-09-13-knowledge-reading.md; no PSD/Omics deployment.

Unified Session entry and complete tab borders — 2026-09-13

The sidebar has one Session/Sessione button: return to the current unfinished session (including pending creation) without resetting/reconnecting it; otherwise prepare a new question with existing readiness and dirty-edit guards. Creation still requires submitting a question. A cold document panel is not a running session. Session tabs now have 11px horizontal/3px vertical padding, at least 38px height, and matching 1px borders on every side (gray inactive, red active), with no shared baseline or overlapping bottom border. This supersedes the earlier border removal. 757 frontend tests, 15 browser scenarios, typecheck, i18n and Docker build pass. Mac frontend 2844778d311c, image d58dccfee3f9, is healthy; only that service was recreated. Core/data services are unchanged, with no Omics or server deploy. Rollback image: thothii-frontend:before-session-navigation-20260913.

Login copy and language-selector focus — 2026-09-13

Removed the redundant login eyebrow/icon and installation-account explanation; the main sign-in heading remains. The full-header language select no longer shows an outer focus ring after pointer interaction; keyboard focus remains visible, including after returning with Tab. EN/IT login and both themes are covered by 36 targeted tests and 14 browser scenarios; typecheck/build pass. Only the Mac frontend was rebuilt/recreated: 204efd40d3eb, image 7dd0752ff823, healthy. Other containers are unchanged. Rollback image: thothii-frontend:before-login-focus-20260913. No production deployment.

Full-header and layout refinements — 2026-09-13

The owner's four visual adjustments are implemented: full header uses Omics #CB333B in both themes with a light complete wordmark/controls; Database status has 16px clearance below its top divider; workspace/model/Done share a compact desktop row; session-scope tabs have no bottom border. Embedded has no extra header. Verified with 71 targeted frontend tests, 11 browser scenarios, typecheck and Docker production build. The Mac frontend was recreated alone and is healthy (8a1608ad7fc6, image a2f489ebe9de); Core/data-service containers are unchanged. Full/en is retained. Rollback image: thothii-frontend:before-header-layout-20260913. See docs/reports/2026-09-13-header-layout-refinements.md for validation and rollback.

Visual review integrated with the full/embedded shell

At the owner's request, the seven commits through a59624a6 from codex/ui-visual-review are integrated with the shell/i18n work in this checkout, codex/prototype-administration-pages. Both branches started at 2d1b714e; the first full-shell build omitted that lateral branch and regressed the installed UI. The integrated source retains bundled Manrope, shared type roles, catalog/context alignment, wordmark sizes and composer autosizing alongside full/embedded and EN/IT.

Local Docker was updated at 12:54 UTC on 2026-09-13: frontend image 605a6e6bba68, healthy; Core and all data-service containers were unchanged. Mac remains full/en. The immediate pre-merge frontend is retained as thothii-frontend:before-visual-shell-merge-20260913.

Verification: 755 frontend tests, 11 Playwright visual/interaction scenarios at five widths, translation-catalog checks, typecheck and production build. See docs/reports/2026-09-13-visual-shell-integration.md for delivery, provenance and rollback details. The separate visual-review worktree and its rollback images are retained. No merge to main, push or production deployment is part of this delivery.

Gitea #28–#31 follow-up is implemented in this checkout: Workspace's four tabs, Database list-first entry without the preparation footer, full-height Pi instructions with installation-host OS selection, and short Admin navigation labels. Core and session behavior are retained. At the owner's request, these changes were rebuilt into local Docker on 2026-09-12 at 18:31 UTC. Core/frontend are healthy and the UI at http://127.0.0.1:8080 serves the updated bundle. The host projection now supplies THT_HOST_PLATFORM=darwin, so Pi selects macOS rather than the container's Linux OS. Persistent dependency containers and volumes were unchanged. Rollback images are tagged thothii-core:before-admin-28-31-20260912 and thothii-frontend:before-admin-28-31-20260912. See docs/reports/2026-09-12-admin-issues-28-31.md for verification and deployment details.

The latest context-shelf A and five Administration pages are implemented locally. At the owner's request, local Docker project thothii-18998cca7b0a was rebuilt and its core/frontend recreated on 2026-09-12. The real UI at http://127.0.0.1:8080 serves shelf A; both services and their existing dependencies are healthy. Runtime model projections were regenerated as schema v2 with the single zai/glm-5.3 interaction default. Persistent services/volumes were not recreated. The local launcher /private/tmp/thothii-memory-preview.sh now adds /private/tmp/thothii-context-a.compose.yaml last, building from this checkout instead of the earlier Memory worktree. Previous images are retained under thothii-core:before-context-a-20260912 and thothii-frontend:before-context-a-20260912. Remote server deployment remains pending. Core/session behavior is retained; one independently remembered workspace/model pair controls both Core and Admin. Unsaved Admin changes block navigation and context changes; operation activity locks the selectors. See docs/reports/2026-09-12-context-shelf-a-implementation.md for verification and the remaining server/Omics integration gate. Prototype alternatives remain untouched.

ThothII is a human-in-the-loop datamart builder with three independently built layers:

frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness → DWH (read-only)

The harness owns the deterministic eight-phase NL→SQL workflow and all session persistence. The backend remains a process/RPC/SSE bridge for sessions and now also owns an isolated PostgreSQL metadata catalog for administrative database configuration. The frontend renders the review gates and keeps the live transcript in memory. See docs/architecture/components.md for the detailed component and data-flow map.

Memory M1–M3, Evidence E1–E3 and joint workflow repair X1 implemented

Browser authorization now derives permissions from validated session roles using the current catalog. This fixes Memory/Evidence navigation remaining disabled for administrators whose remembered login predates those permissions; refreshing the page loads the updated permissions.

The owner requested two independent administration projects: Memory management and Evidence management. Their navigation entries must sit immediately after Database management, as peers; neither page belongs to Database management. Both require complete browsing, filtering, and CRUD without an active core session. Shared requirements and the two project briefs are linked from docs/plans/2026-09-08-memory-evidence-administration.md. Memory administration is implemented; Evidence administration is implemented through external file editing and explicit consolidation. Evidence editing requires an explicit evolution of the current authoring/publication contract. The M1 specification is at docs/plans/2026-09-08-memory-m1-spec.md and published as M1 — Archivio autorevole e amministrazione delle Memory Card with the ready-for-agent and enhancement labels. It covers the authoritative store, administrative CRUD, projection recovery, and current Memory/exemplar producer integration. Its accepted test boundaries are the public harness service, Fastify APIs, and the AppShell page, joined by a focused real-stack browser path. The owner confirmed those boundaries and authorized publication on 2026-09-08. M1 is implemented locally: PostgreSQL authority in thoth_memory, versioned installation migrations, admin CRUD for all four families, structured dependencies and links, explicit projection recovery, verified recall and current workflow producers. The page requires no active session or DWH binding. The existing catalog-migrate preparation service now runs Memory migrations as well. Existing installations need that preparation before using M1; the initial implementation did not deploy or migrate the owner's stacks. The integrated browser check passed with real authentication, Fastify, ThtRunner, harness, PostgreSQL and Qdrant; embeddings were deterministic and unrelated Pi/session activity used test fixtures. See docs/plans/2026-09-08-memory-m1-validation.md for results and commands. M2 is implemented locally: Memory dense/BM25 fusion, physical and business scope filters, bounded outgoing-link expansion and joint ranking, all resolved against current PostgreSQL authority. Migration 002_hybrid_projection.sql makes old dense projections pending until explicit retry/rebuild; Reference remains separate. The real retrieval check uses the configured qwen3-embedding:0.6b model, a separate Ollama process with a read-only model-volume mount, and isolated PostgreSQL/Qdrant resources. See docs/plans/2026-09-09-memory-m2-validation.md. M3 is implemented: editable F8 summary grounded in effective approved decisions, explicit updates with concurrent-edit protection, selected-card/link transactions and durable review receipts. Finalization no longer saves exemplars implicitly. SQL rules and explained errors are consulted in the existing F4/F6/F7 gates. Successful Catalog physical synchronization performs dependency cleanup, preserving the original removals for recovery across restarts. Migration 003_review_receipts.sql is required. Validation includes a real GLM 5.3 generation case against synthetic PostgreSQL data; see docs/plans/2026-09-09-memory-m3-validation.md. Evidence administration and the joint X1 conflict-repair increment are now implemented. The same local preview was subsequently rebuilt with M3 and migration 003 applied. Core and frontend now include the final Memory review and physical dependency cleanup. On 2026-09-09, at the owner's request, the local PSD Docker installation thothii-18998cca7b0a was updated from this worktree. Core/frontend images were rebuilt, Catalog and Memory migrations completed, and all five services became healthy. The UI is at http://127.0.0.1:8080, using the existing local authentication and persistent volumes. The psd-clinical authoritative Memory archive is initially empty (no legacy import). The existing installation configuration remains in /Users/mp/projects/ThothII/deploy/psd/; /private/tmp/thothii-memory-preview.sh invokes its Compose files with a final build-context and migration-command override from this worktree. A future build from the main checkout will use that checkout's code, so retain the worktree override until the changes are integrated. Memory revision history and compatibility with existing development sessions are not requirements. The agreed Memory scope includes reusable domain clarifications, SQL construction rules, solved questions, and explained, approved mistakes to avoid. This extends the current runtime's concept_clarified-only reusable Memory contract. The owner also approved an editable final summary for proposed additions/updates and Memory consumption in the relevant existing review gates. Further agreed behavior includes persistent corrections for Memory/Evidence conflicts through explicit choices, deletion of Memory with invalid dependencies after successful physical schema synchronization, and bounded functional/regression tests instead of a general quality benchmark. In-house Memory evolution is approved, including hybrid Qdrant retrieval and explicit card links traversed in core, without a dedicated graph database. Both capabilities belong to the current scope. PostgreSQL is the agreed authority for Memory Cards, links, and schema dependencies; Qdrant is a rebuildable index. Links are reviewed with cards and editable in Administration; deleting a card removes its incident links while preserving the other cards. The accepted, implemented architecture is recorded in docs/adr/0018-use-postgres-for-memory-and-qdrant-for-retrieval.md. For Evidence administration, Q12 rejects a separate editorial publication workflow. The final R0 direction uses external editors and a manual consolidation command that validates structure, reports required corrections, updates derived metadata, and activates the local Evidence index. The operator then checks the diff and runs Git commit/push manually. No watcher, automated Git, or web editor is required. The owner's later simplification instruction removes the requirement to support administrative edits while core work is in progress. Deliberate corrections made by the workflow itself remain supported. The context specialist writes Evidence drafts independently of the installation and without PostgreSQL access; the system refines them and stores them locally for use and maintenance. The revised Q11 recommendation separates external drafts from a durable local canonical file archive, removing automatic commit/push from CRUD. The owner has now accepted local files and requires discoverable, editable Markdown for nontechnical domain specialists, with no JSONL management surface. Editors on Mac/PC or vim/nano on the server are the chosen R0 editing surface. Evidence management keeps browsing/filtering/detail and clearly identifies the persistent working tree, each Markdown file's actual host path, and the manual commands. E1 implements editable Curated Evidence v4, deterministic legacy conversion, persistent local files and a consolidation API with immutable candidates, manual provenance, deletion records and recoverable activation. E2 connects its installed command, runtime source selection and administration page. The operator completes Git steps manually. All 35 PSD units were converted on an isolated copy with identical IDs and typed content. Tests exercise visible edits through normalization, indexing and recall with real Qdrant; see docs/plans/2026-09-09-evidence-e1-validation.md and docs/contracts/curated-evidence-v4.md. E2 is now running on the local Docker preview: all 35 PSD units were converted and indexed through the installed command. The editable host archive is /Users/mp/projects/ThothII/deploy/psd/evidence-registry/repo/psd-clinical/evidence. The original registry-volume checkout and original author repository were retained. The installation descriptor now includes workspace-bindings.yaml and evidence-host.yaml so native maintenance uses the same volumes and checkout as the preview. Descriptor backup: /private/tmp/thothii-installation-before-e2.yaml; previous native binary: /private/tmp/tht-before-e2. The launcher remains bash /private/tmp/thothii-memory-preview.sh; its worktree build override is still required until integration. Runtime lease filenames now include rendered bytes so upgrading the Evidence renderer does not collide with old immutable configs; Catalog input fingerprints and readiness are unchanged. See docs/plans/2026-09-09-evidence-e2-validation.md. E3 adds explicit source acquisition/refinement and durable comparisons in Evidence management. Local drafts go in evidence/incoming/; original local documents and configured HTTP/S3 sources are reacquired only by Import or refresh sources or installed tht workspace evidence refresh --workspace <id>. Keep/replace decisions, including the native workspace evidence decide command, activate through the existing archive/index path and have durable retry state. Acquired source versions and remote provenance stay local; old documentary lineage can coexist with current manual declarations. The installed preview's 35 PSD sources were verified unchanged with no new proposals. The previous native CLI is backed up at /private/tmp/tht-before-e3. See docs/plans/2026-09-09-evidence-e3-validation.md for tests and validation limits. X1 adds reviewer_archive_repair: closed alternatives with complete before/after content, explicit rejection/reformulation, admin-only application, durable session receipts and retry of saved but inactive corrections. The coordinator uses the canonical Memory and Evidence services; phase approval remains separate. Migration 004_archive_repairs.sql is required. See docs/contracts/archive-repair.md for authorization and recovery limits. X1 validation is recorded in docs/plans/2026-09-09-archive-repair-x1-validation.md: both archives were corrected and retrieved through the actual CLI with real PostgreSQL and Qdrant; desktop/mobile widget behavior and permission failures were verified separately. The local preview images include X1 and migration 004 is applied. Follow-up technical acceptance passed with configured GLM 5.3 generating closed conflict alternatives and the chosen Memory correction persisted and retrieved. An authenticated browser test also verified both administration pages, navigation order, Evidence filtering, workspace 404s, desktop/mobile layout, and Evidence survival after Memory deletion. All approved technical increments/checks are complete. A real PSD domain-conflict session remains the reviewer's semantic acceptance check; automated cases did not modify PSD knowledge. The joint browser inspection also fixed mobile archive navigation: below 768px, Memory and Evidence keep the full content width and open navigation in the shared accessible dialog. Desktop retains its sidebar; the local frontend image includes this fix. The owner accepted the remaining simplifications and requested explicit clarification of the core format change and the simple terminal-based Git check. E1 must adapt the Evidence parser, renderer, authoring, validation, and normalization; convert existing files and reindex; and verify that visible edits reach core consumption. Preserve the internal typed model where possible. This precedes the administrative page and is not merely a presentation change. Human inspection uses normal Git status and optional line-level diff commands; no custom diff viewer or mandatory double review is required. The suggestion of making PostgreSQL the Evidence authority was withdrawn after this clarification; it was never implemented or accepted as a replacement for Q11. Q13 keeps manual corrections active when updated sources contradict them, until an administrator resolves the comparison; deleted Evidence must not be regenerated automatically. Q14 allows direct manual creation and records a manual declaration as the current source, preserving any original document as distinct provenance. Q15 refreshes external sources only on explicit administrator request; normal saves and lookups do not reacquire them. These decisions, now implemented through E1–E3, are recorded in docs/adr/0019-author-evidence-in-app-with-automatic-activation.md. The decision-by-decision review is recorded in docs/plans/2026-09-08-memory-evidence-simplification-review.md; it distinguishes the new constraints from the revised technical recommendations. It also specifies a sequential save with minimal durable retry state and direct Memory cleanup after successful schema synchronization, without new queues or event infrastructure. The delivery order remains Memory, Evidence, and persistent conflict repair between both modules. Local curated Evidence must survive preprocessing Clear and be backed up as primary data; the Qdrant projection remains rebuildable. No runtime gate change or administrative page is implemented yet.

Evidence restructuring — accepted

The evidence restructuring and PSD migration completed real acceptance on 2026-08-25.

  • The curated PSD revision contains 35 approved Evidence units and 60 review items.
  • The PSD authoring repository publishes all 35 units using Curated unit schema v3. Its table-free presentation uses hidden canonical metadata, wrapping Markdown scope lists, list-based enum values, and collapsed technical provenance. Long domain rules now have a deterministic human-readable presentation while retaining their exact canonical text for vector ingestion. tht evidence migrate <workspace-root> performs the deterministic v1/v2 upgrade and older-v3 presentation rewrite without model calls. The structured-rule PSD rewrite is currently local and pending commit/publication.
  • The accepted snapshot is psd-clinical-675990d90eae51da6f2bd51b1ae2609f245772ef-snapshot.
  • The active generation is gen:f968b3bd7a553dbfef3cf47093698f2bc7f95f11.
  • Retrieval acceptance reached 20/20 Hit@10.
  • A real session, 20301df7-cad7-403d-a4c1-9f35c9d07b66, completed F1–F8 with five receipts, three CTEs, and a final result of 78 patients.
  • The durable acceptance record is docs/testing/evidence/evidence-restructuring-psd-acceptance-2026-08-25.md.

The canonical authoring, validation, publication, materialization, and preprocessing flow is documented in docs/evidence.md. The governing contracts are docs/contracts/workspace-evidence-v3.md and docs/contracts/workspace-preprocessing-cli.md. The incremental server procedure for the 260906-preprocessing-complete release is docs/operations/server-handoff-260906-preprocessing-complete.md.

Workspace preprocessing and configuration

The native host CLI tht is the operator surface. Workspace preprocessing runs through:

tht --installation /absolute/path/thothii-installation.yaml \
  workspace preprocess run --workspace <workspace-id>

This complete one-shot command uses the profile-gated workspace-maintenance service. Partial DWH, schema, Evidence, and vector mutation commands are retired.

The right Administration sidebar invokes that same operation for the selected workspace. It shows only current readiness or the latest bounded failure diagnostic; there is no preprocessing history. Known non-ready state disables Session when it would start a new question, while backend admission remains authoritative. The same control exposes an inline-confirmed Clear action to remove replaceable reference vectors, LSH, corpus, and checkpoints while preserving the separate Memory collection. The host CLI equivalent is workspace preprocess clear.

Each workspace now uses <workspace>-reference for Schema, relationships, and Evidence and <workspace>-memory for memory and solved_question. Clear and preprocessing own only the former. LSH ownership additionally binds the Catalog database ID and Metadata Content Revision, so derived values cannot be reused across database identities or Catalog revisions.

Workspace descriptors use schema v4 and contain only workspace identity and optional Evidence. PostgreSQL Metadata Catalog owns database identity, binding, schema, descriptions, sensitivity, and relationships; model, provider, embedding, and vector-store configuration is installation-owned. For PSD, workspace content and runtime roots point to the separate uncommitted repository /Users/mp/projects/tht-workspace-psd. Secrets remain outside Git and are supplied only through installation-local protected files.

Installation Model Catalog

thothii-installation.yaml schema version 2 is the only operator-authored source for session, metadata-generation, and embedding models. The host tht lifecycle validates modelCatalog and regenerates the backend catalog, Pi models.json/settings.json, and Compose override under the installation-local generated/ directory. Those projections are replaceable runtime adapters: they are not edited, backed up, or treated as configuration.

Core and Administration now share one canonical modelCatalog.defaults.interaction per installation, independent of workspace. Runtime catalog schema v2 contains only defaultInteraction; apply the host, backend, and regenerated projections together. Equal legacy defaults normalize on read; conflicting ones require an explicit operator choice. With Admin AI configured, selectable models are the intersection of the Pi and LiteLLM adapters; Core-only installations remain supported without Admin AI. The existing Core and Database controls share the operational selection. Explicit model choices are remembered per authenticated user/application mount in this browser, not in installation settings. Resume uses that selection/default while preserving the historical workspace/revision and manifest. Every model must be manually exercised in both Core and Admin as documented in docs/general/pi-configuration.md; validation is not a live model certification. These changes and shelf A are now deployed to local Docker; remote deployment remains pending.

PSD DeepSeek Pro/Flash now share canonical deepseek/... identities across native Pi and LiteLLM, using the existing DEEPSEEK_API_KEY bundle entry. Its value was confirmed identical to the working Pi key without exposing it; no secret files were changed. The duplicate deepseek-metadata descriptor is removed locally and the tracked example uses the shared provider. secret_env overrides legacy Pi auth only inside temporary runtime snapshots and fails closed if the bundle key is missing. The original auth store/history and zai/glm-5.3 default are preserved. Local Docker projections and core/frontend were updated together on 2026-09-12. Regenerate projections with the matching release for the separate server deployment.

Provider authentication declares one explicit mode (secret_env, pi_auth, or none); secret_env names a protected bundle key. The backend settings store now owns only the selected workspace and thinking level. Existing v1 installations use the explicit catalog migration command; schema-v3 workspace descriptors are converted deterministically in their curator-owned repository before commit. Strict runtime loading does not silently infer or merge legacy sources. ADR 0013 and docs/plans/2026-09-02-installation-model-catalog.md record the decision and implementation.

Database management

The database, table, and authoritative physical-schema catalog slices are implemented. Database Management now opens the Fleet Ledger presentation by default inside AppShell, lists every YAML workspace, creates at most one PostgreSQL database configuration per workspace, edits direct PostgreSQL, REST API, or SSH-tunnel installation bindings, replaces write-only encrypted secrets, and tests supported connector bindings. The surface keeps one responsive AG Grid visible at a time: databases lead to tables, tables lead to columns, and relationships are a sibling database view. Parent navigation remains explicit through the breadcrumb and emphasized back control.

Selection-scoped operations use one action selector plus an explicit Run control; ineligible actions remain visible with their disabled reason, while row-scoped actions stay in the pinned final column. The KPI strip reads installation-wide or selected-database aggregates from GET /catalog/metrics. Database configuration, metadata editors, synchronization history, description history, and sensitive-field review/history use the production APIs in right-side drawers rather than prototype fixtures; closing a history drawer does not stop its background run.

Sensitive-field review is now driven by the versioned local sensitivity-v4 policy, not by a catalog model. The backend reads selected source tables through read-only, database-specific adapters and makes every sensitive | non_sensitive draft decision in the TypeScript SensitivityClassifier. A single validated match protects the column. Tables up to 1,000 rows are fully scanned; larger tables use breadth-first 300, 1,000, and text-only 3,000-value targets, with a five-second limit per source query and no global request deadline. Source failures fail the run instead of yielding unknown; coverage remains visible separately from the proposal. Draft assessments remain transient until an administrator explicitly saves them. Optional GLiNER2 evidence is CPU-only, offline, opt-in, and never replaces the deterministic decision point; see docs/operations/sensitivity-analysis.md. The earlier v1 PSD shadow comparison kept NER disabled by default; see docs/reports/2026-09-02-psd-sensitivity-shadow.md. The v2 comparison completed all 2,275 columns: CPU NER added 18 sensitive proposals and increased warm runtime from 50.1 to 61.3 seconds; see docs/reports/2026-09-03-psd-progressive-sensitivity-shadow.md. Version 4 excludes declared bigint primary-key columns and conventionally named pk bigint columns before source inspection, reporting both as non-informative structural identifiers while distinguishing declared constraints from inferred roles.

Physical membership, source comments, column types/default/nullability/PK positions, and constraint-level ordered FK pairs are projections of the external schema. They cannot be created, renamed, or structurally edited by hand, but administrators can explicitly clear catalog tables, columns, or relationships without touching the source database, binding, configuration, or secrets. Table deletion cascades through columns and relationships; table-scoped relationship cleanup includes incoming and outgoing relationships. Curated and generated descriptions are editable; generated descriptions start null and Database Management can generate or consolidate them for selected tables, selected columns, all targets, or only targets whose Generated Description is missing.

Relationship Management is now reachable directly from each configured Fleet database. One Relationship Map shows read-only Physical Relationships together with Generated and Manual Logical Relationships, with Active, Excluded, and All filters. Administrators can add a single-column relationship, run deterministic name/PK/type inference, exclude or restore a logical relationship, or delete it permanently. Exclusion retains a tombstone that a rebuild cannot reactivate; permanent deletion allows a later rebuild to infer the same endpoints again. Inference uses no LLM, embedding, or source values. It supports normalized table-qualified names, unique non-generic PK names, composite-PK source columns, and the *time_key -> dim_time.<single PK> warehouse convention while ignoring bare generic names. Explicit table/column metadata cleanup remains a destructive boundary: it removes the attached logical relationships and exclusions and requires a full schema synchronization before inference or runtime publication can continue.

The previous Database Management renderer remains a temporary comparison fallback for development and staging only: ?db-ui=legacy is honored in Vite development or when VITE_DB_MANAGEMENT_LEGACY=true; it is not a production presentation. The standalone Fleet Ledger prototype on port 5173 also remains temporary until owner acceptance of the integrated surface, after which both migration aids can be removed.

Schema refresh is one durable asynchronous engine with database-table, selected-table-column, relationship, and full-database actions. Database-level menus expose only the table, relationship, and full scopes; selecting tables exposes column synchronization plus manual column and relationship cleanup for that subset. Database selections also expose manual table and relationship cleanup. Cleanup selections are atomic and share the one-active-operation-per-database exclusion with synchronization. Runs have leases and restart recovery, atomic apply, destructive-diff confirmation with re-scan, cancellation before apply, retained history, and a live SSE log with polling fallback. Null metadata renders blank rather than as a placeholder.

Direct PostgreSQL and strict known-host-verified OpenSSH use pg_catalog. REST bindings use the typed full-snapshot POST /rpc/schema_snapshot contract when available. Servers such as the current PSD endpoint that exposes only POST /rpc/run_query use one catalog-owned read-only query to return the exact same strict v1 snapshot in a single round trip. Both paths remain fail-closed: an absent capability, query error, partial result, or invalid snapshot applies no catalog changes. SSH is not yet enabled for NL→SQL session runtime.

The catalog runs in the internal catalog-db PostgreSQL service. Kysely migrations are an explicit one-shot catalog-migrate operation; scripts/run-stack.sh runs it before local startup. Runtime sessions consume an immutable Catalog JSON snapshot tied to the runtime-config lease. It contains the tables, columns, effective descriptions, sensitivity flags, and active relationships used by the harness; PostgreSQL is the exclusive runtime authority for database metadata. Authored workspace YAML remains limited to workspace identity and optional Evidence configuration. The accepted design is recorded in ADR 0016 and the contracts under docs/contracts/.

Semantic aliases, value descriptions, synonyms, and concepts remain deferred to their dedicated slices.

AI Description Generation uses the catalog's human-owned Sensitive Data Flag. The flag defaults to false, including for newly synchronized columns. An administrator may request a local sensitivity analysis for one selected database, selected tables, or selected columns. One deterministic TypeScript classifier combines metadata, bounded source-content rules, and optional CPU-only NER; no generative model decides the result. Its sensitive or non_sensitive assessments remain an unsaved draft until the human reviews and saves any chosen flag changes, including a downgrade to non-sensitive. Coverage is reported separately; interrupted history may count unprocessed columns. Each started analysis records a separate Sensitivity Analysis Run with aggregate counters and safe ordered events. The progress drawer opens before the synchronous request completes, polls the run, and displays sanitized source-scan and local-NER phase/batch activity while classification is in progress. This operational history never stores per-column assessments, source values, matched spans, prompts, or free-form diagnostics. Saving a sensitive decision persists a sanitized Sensitivity Reason as column Catalog Metadata alongside the human-owned flag; clearing the flag clears that reason. Reloading still discards an unsaved review draft. For unprotected columns, up to five source rows and five representative non-null values may be sent transiently to the configured model provider. Protected columns are omitted from source reads and replaced in the prompt by deterministic plausible values derived only from their metadata. Existing descriptions are not regenerated when a flag changes.

The accepted AI-description design is recorded in docs/plans/2026-08-28-ai-catalog-description-generation.md, with the formal specification in the adjacent -spec.md document and Gitea issue #4. Gitea issues #5–#11 deliver the implementation. The runtime deliberately keeps ThothAI's simple operating model: one installation-wide sequential run owned by the backend, one short-lived Python/LiteLLM completion helper per request, and persistence limited to the run, its safe ordered text events, and each Generated Description as soon as it succeeds. The helper performs at most one provider retry and never falls back to another model. Stop terminates the current helper and retains prior results; three consecutive exhausted technical batches fail the run. Startup marks stale queued/running work interrupted, and Unlock is available only when no local start, worker, or helper is live. Runs remain inspectable through a live SSE log with ordered polling fallback; there is no automatic resume or user-facing generation CLI. ADRs 0009–0010 record the runtime and source-sampling decisions.

The Installation Model Catalog accepts the protected DEEPSEEK_API_KEY and ZAI_API_KEY references for metadata-generation providers. It also accepts a model with no secret reference only when its OpenAI-compatible endpoint is explicit; this covers the VPN-only AritmoLab Qwen 3.6 server without creating a fake operator credential. The Python client supplies only its fixed non-secret compatibility placeholder. The AritmoLab entry also sets disableThinking: true, mapped to the endpoint's chat-template flag, because its default reasoning prose would violate the worker's exact JSON response contract.

Logical relationship integration with core schema-linking is complete: session creation and resume materialize the active physical/generated/manual map, retrieval-pack generation and Pi receive the same runtime config, and snapshot validation fails closed on a declared missing, invalid, or orphaned endpoint. Broader publication of other Catalog metadata to schema-linking remains a separate future slice.

Deferred follow-up — Sensitive Data Policy in schema-linking. The policy is first delivered and tested in catalog description generation. Its enforcement for core schema-linking remains out of scope until the current tickets are closed and the owner has completed the acceptance test. At that gate, resume the design: tht must receive a read-only projection of the current Sensitive Data Flags and exclude values from columns marked sensitive from every LSH result before it is given to Pi. Do not start this integration before the owner gives final approval after that test.

Active deployment work and manual gates

PSD server deployment program

The approved design and executable entry point are:

  • docs/plans/2026-08-20-psd-server-deployment-program-design.md
  • docs/plans/2026-08-20-psd-server-deployment-program.md
  • docs/plans/2026-08-20-psd-server-survey.md
  • docs/plans/2026-08-20-psd-server-project-a-standalone.md
  • docs/plans/2026-08-20-psd-server-project-b-authentik.md

Last recorded state:

  • survey: SURVEY_NO_GO;
  • Project A: BLOCKED_BY_SURVEY_AND_MUTATION_GATE;
  • Project B: BLOCKED_BY_PROJECT_A_AND_PRE_B_GATE.

The deployment is a clean replacement: legacy sessions, indexes, and application configuration are not migration inputs. The existing stack remains intact until its documented mutation and rollback gates are explicitly approved. Shared Omics/LocalLLM networks, ETL Evidence, DWH, dwh-auth, Supabase, Authentik, Superset, and Aritmolab are outside cleanup scope.

Human acceptance guides and sanitized report templates live under docs/testing/ and docs/testing/evidence/. The remediation checklist is docs/operations/psd-server-survey-remediation-checklist.md.

Authentication

The local/OIDC authentication remediation passed its automated review on 2026-08-18. Release and PSD mutation gates remain governed by:

  • docs/architecture/authentication.md;
  • docs/plans/2026-08-18-thothii-authentication-acceptance-and-psd-deployment.md;
  • docs/operations/psd-dwh-auth-rollout.md;
  • docs/testing/authentication-manual-acceptance.md.

Do not infer authorization for server, Nginx, Authentik, database, credential, or cutover changes from an automated PASS.

Verification status

  • The P1.1 workspace-directory registry and P2–P6 preprocessing workstreams are implemented and have automated coverage.
  • Evidence restructuring has a real PSD acceptance PASS as recorded above.
  • AI Description Generation has automated coverage across installation setup, model selection, generation/consolidation scopes, bounded sampling, cancellation/recovery, history, SSE/polling, and the LiteLLM helper boundary.
  • L2 tests requiring real providers or remote databases remain opt-in.
  • Server deployment, release, and owner-operated acceptance steps remain pending wherever the referenced runbooks require explicit approval.

Run the layer-specific checks documented in AGENTS.md. For release-sensitive changes, also run the repository contract scripts in scripts/ and build the MkDocs site.

Operational invariants

  • tht's -c/--config option follows the subcommand; it is not a global option.
  • --json commands write pristine JSON to stdout.
  • Persisted phase documents and the decision ledger are the source of session truth; chat is not.
  • UI chrome is English; workspace document content retains the workspace language.
  • The backend refuses resume for finalized or archived sessions.
  • A resume must send /riprendi-sessione <id>; a new session must send /nuova-domanda.
  • DWH access is read-only.
Reference in New Issue View Git Blame Copy Permalink