46 KiB
ThothII — Project State
Last updated: 2026-09-14.
This file is the short operational snapshot. Stable commands and the architecture mental model
live in AGENTS.md; current design and runtime contracts live under docs/architecture/,
docs/contracts/, docs/adr/, and docs/evidence.md. Superseded plans and reports are
available from Git history rather than duplicated in the working tree.
The guarded server migration from a legacy checkout to the schema-v2 installation, Gitea source,
Authentik, internal catalog/embedding services, and the PSD workspace repository is documented in
docs/operations/server-upgrade-gitea-workspace-v2.md. Treat its operator gates and rollback
requirements as mandatory; do not replace the running server stack in place.
Full/embedded shell and bilingual interface
Full/embedded shell, EN/IT UI, immutable session interaction language, dark theme,
fullscreen and full-mode logout are implemented. The local Mac descriptor explicitly
sets shell.mode: full and shell.defaultLocale: en; the native tht and local
core/frontend images were updated on 2026-09-13. Omics uses embedded with server-side
identity verification and a replaceable presentation-only PortalAdapter; this does
not mean this branch was verified on the production server. Its changes are
in Omics commit 95154e1; production deployment remains pending.
See docs/operations/shell-and-localization.md for integration and installation
instructions and docs/reports/2026-09-13-full-shell-implementation.md for tests,
independent reviews, local browser checks and rollback details.
Documentation handoff before branch closure — 2026-09-13
Current rendering architecture is in docs/architecture/application-shell.md;
the exact portal identity/proxy contract is in docs/install/authentication-upstream.md.
docs/operations/server-codex-handoff.md is the current server delivery/deploy
runbook, including Omics source integration from GitHub, configuration, tests and
rollback. It supersedes the earlier Omics repository-relay instructions. The acceptance matrix is
docs/testing/authentication-manual-acceptance.md. README, documentation navigation,
user/installation/authentication guides and descriptor examples point to these
paths. Local examples explicitly use full/en; the projected server example is
for standalone OIDC, not Omics upstream. No runtime configuration or deployment
was changed by that documentation pass. The 2026-09-14 delivery is prepared for
promotion to main; the actual merge is recorded in Git. PSD acceptance remains pending.
Navigation readiness and session accordions — 2026-09-13
The Workspace navigation button now carries an accessible green/red readiness
dot instead of a separate text row. Green requires a selected workspace and a
successful, current ready response; checking, unavailable and other states are
red, with the state exposed through the tooltip and accessible description.
The backend readiness gate is unchanged.
Active sessions (non-archived) and Archive both start collapsed. Their adjacent
headers are the only visible content below the scope tabs: no Sessions heading
or external selection toolbar. Each nonempty panel owns its Select all and
bulk-delete controls, scoped to that list and preserving the other's selection.
As of 2026-09-14, only one section can be open at a time, and either can be
collapsed. Empty lists show only "No sessions yet." The open section uses the
remaining sidebar height, with scrolling content capped at min(18rem, 35dvh). The mobile
navigation dialog also provides a bounded height. Keyboard controls and labels
are retained. See DESIGN.md and docs/guida-utente.md for the UI contract.
Verification: 768 frontend unit tests, 20 browser scenarios (including 80 mocked
sessions at 390/1280px), TypeScript and the production frontend build passed.
Only the Mac frontend was recreated; it is healthy at 127.0.0.1:8080.
Core, catalog, Qdrant and embedding containers were not changed. The prior
frontend image is retained as
thothii-frontend:before-single-session-accordion-20260914 for rollback.
Current Omics delivery and server handoff — 2026-09-14
The owner corrected the delivery requirement: Omics is obtained from GitHub, not relayed to another repository as part of this deployment. Any optional server-side repository copy is solely the owner's separate concern. This supersedes the 2026-09-13 relay agreement, including historical delivery notes in the Omics branch. Do not make another remote publication a prerequisite.
GitHub branch codex/thothii-embedded-shell at
https://github.com/Dallavilla-Tiziano/omics_portal.git was reverified at
fca10901a73666ca257d8f4cc4b77066295c400a (functional commit 95154e1).
The server operator integrates it with the current code in the confirmed Omics
checkout, normally /home/chirone/omics_portal, preserving later server changes.
docs/operations/server-codex-handoff.md is the authoritative ordered procedure:
inventory, source verification, native CLI and installation projections,
embedded/upstream auth, Omics templates/static assets/proxy, coordinated rollout,
acceptance and rollback. Server deployment and real IdP acceptance remain pending.
Current product shape
Shared Memory/Evidence typography — 2026-09-13
Both detail readers now share Manrope and fixed reading roles: 24px card title,
20px section headings, 16px/1.65 narrative text, and 14px metadata/code. Authored
Markdown subheadings remain subordinate to field headings; inline/fenced code no
longer shrinks cumulatively. Full-width layout, paragraph separation and isolated
FAKE examples are retained. All 762 frontend tests and 18 browser scenarios pass,
including computed typography checks at 390/1280/2400px; typecheck, i18n and Docker
build pass. Only Mac frontend was recreated: 87fca0dc5e19, image 19f1704b6bb2,
healthy. Core and data services are unchanged. Rollback image:
thothii-frontend:before-knowledge-typography-20260913. See
docs/reports/2026-09-13-knowledge-typography.md. No PSD/Omics deployment.
Knowledge reading and isolated fake Memory examples — 2026-09-13
Memory/Evidence details now use all available width, with display-only paragraph
splitting for long plain prose and preserved code/Markdown/source data. Evidence
provenance renders Markdown; copy controls are accessible two-sheet icons.
Memory's separate Formatting examples section contains four FAKE cards (one per
family), automatically expanded for an empty archive. These client-side examples
cannot be edited/saved/indexed and are never submitted to the model or Memory API.
No real archive content was changed. 762 tests, 18 browser scenarios, typecheck,
i18n and Docker build pass. Only the Mac frontend was recreated: fc4286b5406d,
image cbe0fe0dce55, healthy; other services unchanged. Rollback image:
thothii-frontend:before-knowledge-reading-20260913. Details in
docs/reports/2026-09-13-knowledge-reading.md; no PSD/Omics deployment.
Unified Session entry and complete tab borders — 2026-09-13
The sidebar has one Session/Sessione button: return to the current unfinished
session (including pending creation) without resetting/reconnecting it; otherwise
prepare a new question with existing readiness and dirty-edit guards. Creation
still requires submitting a question. A cold document panel is not a running session.
Session tabs now have 11px horizontal/3px vertical padding, at least 38px height,
and matching 1px borders on every side (gray inactive, red active), with no shared
baseline or overlapping bottom border. This supersedes the earlier border removal.
757 frontend tests, 15 browser scenarios, typecheck, i18n and Docker build pass.
Mac frontend 2844778d311c, image d58dccfee3f9, is healthy; only that service
was recreated. Core/data services are unchanged, with no Omics or server deploy.
Rollback image: thothii-frontend:before-session-navigation-20260913.
Login copy and language-selector focus — 2026-09-13
Removed the redundant login eyebrow/icon and installation-account explanation;
the main sign-in heading remains. The full-header language select no longer
shows an outer focus ring after pointer interaction; keyboard focus remains
visible, including after returning with Tab. EN/IT login and both themes are
covered by 36 targeted tests and 14 browser scenarios; typecheck/build pass.
Only the Mac frontend was rebuilt/recreated: 204efd40d3eb, image 7dd0752ff823,
healthy. Other containers are unchanged. Rollback image:
thothii-frontend:before-login-focus-20260913. No production deployment.
Full-header and layout refinements — 2026-09-13
The owner's four visual adjustments are implemented: full header uses Omics
#CB333B in both themes with a light complete wordmark/controls; Database status
has 16px clearance below its top divider; workspace/model/Done share a compact
desktop row; session-scope tabs have no bottom border. Embedded has no extra header.
Verified with 71 targeted frontend tests, 11 browser scenarios, typecheck and
Docker production build. The Mac frontend was recreated alone and is healthy
(8a1608ad7fc6, image a2f489ebe9de); Core/data-service containers are unchanged.
Full/en is retained. Rollback image: thothii-frontend:before-header-layout-20260913.
See docs/reports/2026-09-13-header-layout-refinements.md for validation and rollback.
Visual review integrated with the full/embedded shell
At the owner's request, the seven commits through a59624a6 from
codex/ui-visual-review are integrated with the shell/i18n work in this checkout,
codex/prototype-administration-pages. Both branches started at 2d1b714e; the
first full-shell build omitted that lateral branch and regressed the installed UI.
The integrated source retains bundled Manrope, shared type roles, catalog/context
alignment, wordmark sizes and composer autosizing alongside full/embedded and EN/IT.
Local Docker was updated at 12:54 UTC on 2026-09-13: frontend image 605a6e6bba68,
healthy; Core and all data-service containers were unchanged. Mac remains full/en.
The immediate pre-merge frontend is retained as
thothii-frontend:before-visual-shell-merge-20260913.
Verification: 755 frontend tests, 11 Playwright visual/interaction scenarios at five
widths, translation-catalog checks, typecheck and production build. See
docs/reports/2026-09-13-visual-shell-integration.md for delivery, provenance and
rollback details. The separate visual-review worktree and its rollback images are
retained. No merge to main, push or production deployment is part of this delivery.
Gitea #28–#31 follow-up is implemented in this checkout: Workspace's four tabs,
Database list-first entry without the preparation footer, full-height Pi instructions
with installation-host OS selection, and short Admin navigation labels. Core and
session behavior are retained. At the owner's request, these changes were rebuilt into
local Docker on 2026-09-12 at 18:31 UTC. Core/frontend are healthy and the UI at
http://127.0.0.1:8080 serves the updated bundle. The host projection now supplies
THT_HOST_PLATFORM=darwin, so Pi selects macOS rather than the container's Linux OS.
Persistent dependency containers and volumes were unchanged. Rollback images are tagged
thothii-core:before-admin-28-31-20260912 and
thothii-frontend:before-admin-28-31-20260912. See
docs/reports/2026-09-12-admin-issues-28-31.md for verification and deployment details.
The latest context-shelf A and five Administration pages are implemented locally.
At the owner's request, local Docker project thothii-18998cca7b0a was rebuilt and
its core/frontend recreated on 2026-09-12. The real UI at http://127.0.0.1:8080
serves shelf A; both services and their existing dependencies are healthy.
Runtime model projections were regenerated as schema v2 with the single
zai/glm-5.3 interaction default. Persistent services/volumes were not recreated.
The local launcher /private/tmp/thothii-memory-preview.sh now adds
/private/tmp/thothii-context-a.compose.yaml last, building from this checkout
instead of the earlier Memory worktree. Previous images are retained under
thothii-core:before-context-a-20260912 and
thothii-frontend:before-context-a-20260912. Remote server deployment remains pending.
Core/session behavior is retained; one independently remembered workspace/model
pair controls both Core and Admin. Unsaved Admin changes block navigation and
context changes; operation activity locks the selectors. See
docs/reports/2026-09-12-context-shelf-a-implementation.md for verification and the
remaining server/Omics integration gate. Prototype alternatives remain untouched.
ThothII is a human-in-the-loop datamart builder with three independently built layers:
frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness → DWH (read-only)
The harness owns the deterministic eight-phase NL→SQL workflow and all session persistence.
The backend remains a process/RPC/SSE bridge for sessions and now also owns an isolated PostgreSQL
metadata catalog for administrative database configuration. The frontend renders the review gates
and keeps the live transcript in memory. See
docs/architecture/components.md for the detailed component and data-flow map.
Memory M1–M3, Evidence E1–E3 and joint workflow repair X1 implemented
Browser authorization now derives permissions from validated session roles using the current catalog. This fixes Memory/Evidence navigation remaining disabled for administrators whose remembered login predates those permissions; refreshing the page loads the updated permissions.
The owner requested two independent administration projects: Memory management and Evidence
management. Their navigation entries must sit immediately after Database management, as peers;
neither page belongs to Database management. Both require complete browsing, filtering, and CRUD
without an active core session. Shared requirements and the two project briefs are linked from
docs/plans/2026-09-08-memory-evidence-administration.md. Memory administration is implemented;
Evidence administration is implemented through external file editing and explicit consolidation.
Evidence editing requires an explicit evolution of the current authoring/publication contract.
The M1 specification is at docs/plans/2026-09-08-memory-m1-spec.md and published as
M1 — Archivio autorevole e amministrazione delle Memory Card
with the ready-for-agent and enhancement labels.
It covers the authoritative store, administrative CRUD, projection recovery, and current
Memory/exemplar producer integration. Its accepted test boundaries are the public harness
service, Fastify APIs, and the AppShell page, joined by a focused real-stack browser path.
The owner confirmed those boundaries and authorized publication on 2026-09-08.
M1 is implemented locally: PostgreSQL authority in thoth_memory, versioned installation
migrations, admin CRUD for all four families, structured dependencies and links, explicit
projection recovery, verified recall and current workflow producers. The page requires no
active session or DWH binding. The existing catalog-migrate preparation service now runs
Memory migrations as well. Existing installations need that preparation before using M1;
the initial implementation did not deploy or migrate the owner's stacks.
The integrated browser check passed with real authentication, Fastify, ThtRunner, harness,
PostgreSQL and Qdrant; embeddings were deterministic and unrelated Pi/session activity used
test fixtures. See docs/plans/2026-09-08-memory-m1-validation.md for results and commands.
M2 is implemented locally: Memory dense/BM25 fusion, physical and business scope filters,
bounded outgoing-link expansion and joint ranking, all resolved against current PostgreSQL
authority. Migration 002_hybrid_projection.sql makes old dense projections pending until
explicit retry/rebuild; Reference remains separate. The real retrieval check uses the configured
qwen3-embedding:0.6b model, a separate Ollama process with a read-only model-volume mount,
and isolated PostgreSQL/Qdrant resources. See docs/plans/2026-09-09-memory-m2-validation.md.
M3 is implemented: editable F8 summary grounded in effective approved decisions, explicit
updates with concurrent-edit protection, selected-card/link transactions and durable review
receipts. Finalization no longer saves exemplars implicitly. SQL rules and explained errors
are consulted in the existing F4/F6/F7 gates. Successful Catalog physical synchronization
performs dependency cleanup, preserving the original removals for recovery across restarts.
Migration 003_review_receipts.sql is required. Validation includes a real GLM 5.3 generation
case against synthetic PostgreSQL data; see docs/plans/2026-09-09-memory-m3-validation.md.
Evidence administration and the joint X1 conflict-repair increment are now implemented.
The same local preview was subsequently rebuilt with M3 and migration 003 applied.
Core and frontend now include the final Memory review and physical dependency cleanup.
On 2026-09-09, at the owner's request, the local PSD Docker installation
thothii-18998cca7b0a was updated from this worktree. Core/frontend images were rebuilt,
Catalog and Memory migrations completed, and all five services became healthy. The UI is at
http://127.0.0.1:8080, using the existing local authentication and persistent volumes.
The psd-clinical authoritative Memory archive is initially empty (no legacy import).
The existing installation configuration remains in /Users/mp/projects/ThothII/deploy/psd/;
/private/tmp/thothii-memory-preview.sh invokes its Compose files with a final build-context
and migration-command override from this worktree. A future build from the main checkout
will use that checkout's code, so retain the worktree override until the changes are integrated.
Memory revision history and compatibility with existing development sessions are not requirements.
The agreed Memory scope includes reusable domain clarifications, SQL construction rules, solved
questions, and explained, approved mistakes to avoid. This extends the current runtime's
concept_clarified-only reusable Memory contract. The owner also approved an editable final summary
for proposed additions/updates and Memory consumption in the relevant existing review gates.
Further agreed behavior includes persistent corrections for Memory/Evidence conflicts through
explicit choices, deletion of Memory with invalid dependencies after successful physical schema
synchronization, and bounded functional/regression tests instead of a general quality benchmark.
In-house Memory evolution is approved, including hybrid Qdrant retrieval and explicit card links
traversed in core, without a dedicated graph database. Both capabilities belong to the current
scope. PostgreSQL is the agreed authority for Memory Cards, links, and schema dependencies;
Qdrant is a rebuildable index. Links are reviewed with cards and editable in Administration;
deleting a card removes its incident links while preserving the other cards. The accepted,
implemented architecture is recorded in
docs/adr/0018-use-postgres-for-memory-and-qdrant-for-retrieval.md.
For Evidence administration, Q12 rejects a separate editorial publication workflow.
The final R0 direction uses external editors and a manual consolidation command that
validates structure, reports required corrections, updates derived metadata, and
activates the local Evidence index. The operator then checks the diff and runs Git
commit/push manually. No watcher, automated Git, or web editor is required.
The owner's later simplification
instruction removes the requirement to support administrative edits while core work
is in progress. Deliberate corrections made by the workflow itself remain supported.
The context specialist writes Evidence drafts independently of the installation and
without PostgreSQL access; the system refines them and stores them locally for use
and maintenance. The revised Q11 recommendation separates external drafts from a
durable local canonical file archive, removing automatic commit/push from CRUD.
The owner has now accepted local files and requires discoverable, editable Markdown
for nontechnical domain specialists, with no JSONL management surface. Editors on
Mac/PC or vim/nano on the server are the chosen R0 editing surface. Evidence
management keeps browsing/filtering/detail and clearly identifies the persistent
working tree, each Markdown file's actual host path, and the manual commands.
E1 implements editable Curated Evidence v4, deterministic legacy conversion, persistent
local files and a consolidation API with immutable candidates, manual provenance,
deletion records and recoverable activation. E2 connects its installed command, runtime source
selection and administration page. The operator completes Git steps manually.
All 35 PSD units were converted on an isolated copy with identical IDs and typed content.
Tests exercise visible edits through normalization, indexing and recall with real Qdrant;
see docs/plans/2026-09-09-evidence-e1-validation.md and
docs/contracts/curated-evidence-v4.md. E2 is now running on the local Docker preview:
all 35 PSD units were converted and indexed through the installed command. The editable
host archive is /Users/mp/projects/ThothII/deploy/psd/evidence-registry/repo/psd-clinical/evidence.
The original registry-volume checkout and original author repository were retained.
The installation descriptor now includes workspace-bindings.yaml and evidence-host.yaml
so native maintenance uses the same volumes and checkout as the preview. Descriptor backup:
/private/tmp/thothii-installation-before-e2.yaml; previous native binary: /private/tmp/tht-before-e2.
The launcher remains bash /private/tmp/thothii-memory-preview.sh; its worktree build override
is still required until integration. Runtime lease filenames now include rendered bytes so
upgrading the Evidence renderer does not collide with old immutable configs; Catalog
input fingerprints and readiness are unchanged. See docs/plans/2026-09-09-evidence-e2-validation.md.
E3 adds explicit source acquisition/refinement and durable comparisons in Evidence management.
Local drafts go in evidence/incoming/; original local documents and configured HTTP/S3
sources are reacquired only by Import or refresh sources or installed
tht workspace evidence refresh --workspace <id>. Keep/replace decisions, including the
native workspace evidence decide command, activate through the existing archive/index
path and have durable retry state. Acquired source versions and remote provenance stay
local; old documentary lineage can coexist with current manual declarations.
The installed preview's 35 PSD sources were verified unchanged with no new proposals.
The previous native CLI is backed up at /private/tmp/tht-before-e3.
See docs/plans/2026-09-09-evidence-e3-validation.md for tests and validation limits.
X1 adds reviewer_archive_repair: closed alternatives with complete before/after content,
explicit rejection/reformulation, admin-only application, durable session receipts and
retry of saved but inactive corrections. The coordinator uses the canonical Memory and
Evidence services; phase approval remains separate. Migration 004_archive_repairs.sql
is required. See docs/contracts/archive-repair.md for authorization and recovery limits.
X1 validation is recorded in docs/plans/2026-09-09-archive-repair-x1-validation.md:
both archives were corrected and retrieved through the actual CLI with real PostgreSQL
and Qdrant; desktop/mobile widget behavior and permission failures were verified separately.
The local preview images include X1 and migration 004 is applied. Follow-up technical
acceptance passed with configured GLM 5.3 generating closed conflict alternatives and
the chosen Memory correction persisted and retrieved. An authenticated browser test
also verified both administration pages, navigation order, Evidence filtering, workspace
404s, desktop/mobile layout, and Evidence survival after Memory deletion. All approved
technical increments/checks are complete. A real PSD domain-conflict session remains
the reviewer's semantic acceptance check; automated cases did not modify PSD knowledge.
The joint browser inspection also fixed mobile archive navigation: below 768px,
Memory and Evidence keep the full content width and open navigation in the shared
accessible dialog. Desktop retains its sidebar; the local frontend image includes this fix.
The owner accepted the remaining simplifications and requested explicit clarification
of the core format change and the simple terminal-based Git check. E1 must adapt
the Evidence parser, renderer, authoring, validation, and normalization; convert
existing files and reindex; and verify that visible edits reach core consumption.
Preserve the internal typed model where possible. This precedes the administrative
page and is not merely a presentation change. Human inspection uses normal Git
status and optional line-level diff commands; no custom diff viewer or mandatory
double review is required.
The suggestion of making PostgreSQL the Evidence authority was withdrawn after this
clarification; it was never implemented or accepted as a replacement for Q11.
Q13 keeps manual corrections active when updated sources contradict them, until an
administrator resolves the comparison; deleted Evidence must not be regenerated
automatically. Q14 allows direct manual creation and records a manual declaration
as the current source, preserving any original document as distinct provenance.
Q15 refreshes external sources only on explicit administrator request; normal saves
and lookups do not reacquire them. These decisions, now implemented through E1–E3, are recorded in
docs/adr/0019-author-evidence-in-app-with-automatic-activation.md.
The decision-by-decision review is recorded in
docs/plans/2026-09-08-memory-evidence-simplification-review.md; it distinguishes the
new constraints from the revised technical recommendations. It also specifies a
sequential save with minimal durable retry state and direct Memory cleanup after
successful schema synchronization, without new queues or event infrastructure.
The delivery order remains Memory, Evidence, and persistent conflict repair between
both modules. Local curated Evidence must survive preprocessing Clear and be backed
up as primary data; the Qdrant projection remains rebuildable.
No runtime gate change or administrative page is implemented yet.
Evidence restructuring — accepted
The evidence restructuring and PSD migration completed real acceptance on 2026-08-25.
- The curated PSD revision contains 35 approved Evidence units and 60 review items.
- The PSD authoring repository publishes all 35 units using Curated unit schema v3. Its table-free
presentation uses hidden canonical metadata, wrapping Markdown scope lists, list-based enum
values, and collapsed technical provenance. Long domain rules now have a deterministic
human-readable presentation while retaining their exact canonical text for vector ingestion.
tht evidence migrate <workspace-root>performs the deterministic v1/v2 upgrade and older-v3 presentation rewrite without model calls. The structured-rule PSD rewrite is currently local and pending commit/publication. - The accepted snapshot is
psd-clinical-675990d90eae51da6f2bd51b1ae2609f245772ef-snapshot. - The active generation is
gen:f968b3bd7a553dbfef3cf47093698f2bc7f95f11. - Retrieval acceptance reached 20/20 Hit@10.
- A real session,
20301df7-cad7-403d-a4c1-9f35c9d07b66, completed F1–F8 with five receipts, three CTEs, and a final result of 78 patients. - The durable acceptance record is
docs/testing/evidence/evidence-restructuring-psd-acceptance-2026-08-25.md.
The canonical authoring, validation, publication, materialization, and preprocessing flow is
documented in docs/evidence.md. The governing contracts are
docs/contracts/workspace-evidence-v3.md and
docs/contracts/workspace-preprocessing-cli.md.
The incremental server procedure for the 260906-preprocessing-complete release is
docs/operations/server-handoff-260906-preprocessing-complete.md.
Workspace preprocessing and configuration
The native host CLI tht is the operator surface. Workspace preprocessing runs through:
tht --installation /absolute/path/thothii-installation.yaml \
workspace preprocess run --workspace <workspace-id>
This complete one-shot command uses the profile-gated workspace-maintenance service. Partial DWH,
schema, Evidence, and vector mutation commands are retired.
The right Administration sidebar invokes that same operation for the selected workspace. It shows
only current readiness or the latest bounded failure diagnostic; there is no preprocessing history.
Known non-ready state disables Session when it would start a new question,
while backend admission remains authoritative.
The same control exposes an inline-confirmed Clear action to remove replaceable reference
vectors, LSH, corpus, and checkpoints while preserving the separate Memory collection. The host CLI
equivalent is workspace preprocess clear.
Each workspace now uses <workspace>-reference for Schema, relationships, and Evidence and
<workspace>-memory for memory and solved_question. Clear and preprocessing own only the former.
LSH ownership additionally binds the Catalog database ID and Metadata Content Revision, so derived
values cannot be reused across database identities or Catalog revisions.
Workspace descriptors use schema v4 and contain only workspace identity and optional Evidence.
PostgreSQL Metadata Catalog owns database identity, binding, schema, descriptions, sensitivity, and
relationships; model, provider, embedding, and vector-store configuration is installation-owned. For
PSD, workspace content and runtime roots point to the
separate uncommitted repository /Users/mp/projects/tht-workspace-psd. Secrets remain outside
Git and are supplied only through installation-local protected files.
Installation Model Catalog
thothii-installation.yaml schema version 2 is the only operator-authored source for session,
metadata-generation, and embedding models. The host tht lifecycle validates modelCatalog and
regenerates the backend catalog, Pi models.json/settings.json, and Compose override under the
installation-local generated/ directory. Those projections are replaceable runtime adapters:
they are not edited, backed up, or treated as configuration.
Core and Administration now share one canonical modelCatalog.defaults.interaction per installation,
independent of workspace. Runtime catalog schema v2 contains only defaultInteraction; apply the host,
backend, and regenerated projections together. Equal legacy defaults normalize on read; conflicting
ones require an explicit operator choice. With Admin AI configured, selectable models are the
intersection of the Pi and LiteLLM adapters; Core-only installations remain supported without Admin AI.
The existing Core and Database controls share the operational selection. Explicit model choices are
remembered per authenticated user/application mount in this browser, not in installation settings.
Resume uses that selection/default while preserving the historical workspace/revision and manifest.
Every model must be manually exercised in both Core and Admin as documented in
docs/general/pi-configuration.md; validation is not a live model certification.
These changes and shelf A are now deployed to local Docker; remote deployment remains pending.
PSD DeepSeek Pro/Flash now share canonical deepseek/... identities across native Pi and LiteLLM,
using the existing DEEPSEEK_API_KEY bundle entry. Its value was confirmed identical to the working
Pi key without exposing it; no secret files were changed. The duplicate deepseek-metadata descriptor
is removed locally and the tracked example uses the shared provider. secret_env overrides legacy
Pi auth only inside temporary runtime snapshots and fails closed if the bundle key is missing.
The original auth store/history and zai/glm-5.3 default are preserved. Local Docker projections
and core/frontend were updated together on 2026-09-12. Regenerate projections with
the matching release for the separate server deployment.
Provider authentication declares
one explicit mode (secret_env, pi_auth, or none); secret_env names a protected bundle key.
The backend settings store now owns only the selected workspace and thinking level. Existing v1
installations use the explicit catalog migration command; schema-v3 workspace descriptors are
converted deterministically in their curator-owned repository before commit. Strict runtime loading
does not silently infer or merge legacy sources. ADR 0013 and
docs/plans/2026-09-02-installation-model-catalog.md record the decision and implementation.
Database management
The database, table, and authoritative physical-schema catalog slices are implemented. Database
Management now opens the Fleet Ledger presentation by default inside AppShell, lists every YAML
workspace, creates at most one PostgreSQL database configuration per workspace, edits direct
PostgreSQL, REST API, or SSH-tunnel installation bindings, replaces write-only encrypted secrets,
and tests supported connector bindings. The surface keeps one responsive AG Grid visible at a time:
databases lead to tables, tables lead to columns, and relationships are a sibling database view.
Parent navigation remains explicit through the breadcrumb and emphasized back control.
Selection-scoped operations use one action selector plus an explicit Run control; ineligible
actions remain visible with their disabled reason, while row-scoped actions stay in the pinned final
column. The KPI strip reads installation-wide or selected-database aggregates from
GET /catalog/metrics. Database configuration, metadata editors, synchronization history,
description history, and sensitive-field review/history use the production APIs in right-side
drawers rather than prototype fixtures; closing a history drawer does not stop its background run.
Sensitive-field review is now driven by the versioned local sensitivity-v4 policy, not by a
catalog model. The backend reads selected source tables through read-only, database-specific
adapters and makes every sensitive | non_sensitive draft decision in the TypeScript
SensitivityClassifier. A single validated match protects the column. Tables up to 1,000 rows are
fully scanned; larger tables use breadth-first 300, 1,000, and text-only 3,000-value targets, with a
five-second limit per source query and no global request deadline. Source failures fail the run
instead of yielding unknown; coverage remains visible separately from the proposal. Draft
assessments remain transient until an administrator explicitly saves them. Optional GLiNER2
evidence is CPU-only, offline, opt-in, and never replaces the deterministic decision point; see
docs/operations/sensitivity-analysis.md. The earlier v1 PSD shadow comparison kept NER disabled by
default; see docs/reports/2026-09-02-psd-sensitivity-shadow.md. The v2 comparison completed all
2,275 columns: CPU NER added 18 sensitive proposals and increased warm runtime from 50.1 to 61.3
seconds; see docs/reports/2026-09-03-psd-progressive-sensitivity-shadow.md.
Version 4 excludes declared bigint primary-key columns and conventionally named pk bigint
columns before source inspection, reporting both as non-informative structural identifiers while
distinguishing declared constraints from inferred roles.
Physical membership, source comments, column types/default/nullability/PK positions, and constraint-level ordered FK pairs are projections of the external schema. They cannot be created, renamed, or structurally edited by hand, but administrators can explicitly clear catalog tables, columns, or relationships without touching the source database, binding, configuration, or secrets. Table deletion cascades through columns and relationships; table-scoped relationship cleanup includes incoming and outgoing relationships. Curated and generated descriptions are editable; generated descriptions start null and Database Management can generate or consolidate them for selected tables, selected columns, all targets, or only targets whose Generated Description is missing.
Relationship Management is now reachable directly from each configured Fleet database. One
Relationship Map shows read-only Physical Relationships together with Generated and Manual Logical
Relationships, with Active, Excluded, and All filters. Administrators can add a single-column
relationship, run deterministic name/PK/type inference, exclude or restore a logical relationship,
or delete it permanently. Exclusion retains a tombstone that a rebuild cannot reactivate; permanent
deletion allows a later rebuild to infer the same endpoints again. Inference uses no LLM, embedding,
or source values. It supports normalized table-qualified names, unique non-generic PK names,
composite-PK source columns, and the *time_key -> dim_time.<single PK> warehouse convention while
ignoring bare generic names. Explicit table/column metadata cleanup remains a destructive boundary: it removes
the attached logical relationships and exclusions and requires a full schema synchronization before
inference or runtime publication can continue.
The previous Database Management renderer remains a temporary comparison fallback for development
and staging only: ?db-ui=legacy is honored in Vite development or when
VITE_DB_MANAGEMENT_LEGACY=true; it is not a production presentation. The standalone Fleet Ledger
prototype on port 5173 also remains temporary until owner acceptance of the integrated surface,
after which both migration aids can be removed.
Schema refresh is one durable asynchronous engine with database-table, selected-table-column, relationship, and full-database actions. Database-level menus expose only the table, relationship, and full scopes; selecting tables exposes column synchronization plus manual column and relationship cleanup for that subset. Database selections also expose manual table and relationship cleanup. Cleanup selections are atomic and share the one-active-operation-per-database exclusion with synchronization. Runs have leases and restart recovery, atomic apply, destructive-diff confirmation with re-scan, cancellation before apply, retained history, and a live SSE log with polling fallback. Null metadata renders blank rather than as a placeholder.
Direct PostgreSQL and strict known-host-verified OpenSSH use pg_catalog. REST bindings use the
typed full-snapshot POST /rpc/schema_snapshot contract when available. Servers such as the
current PSD endpoint that exposes only POST /rpc/run_query use one catalog-owned read-only query
to return the exact same strict v1 snapshot in a single round trip. Both paths remain fail-closed:
an absent capability, query error, partial result, or invalid snapshot applies no catalog changes.
SSH is not yet enabled for NL→SQL session runtime.
The catalog runs in the internal catalog-db PostgreSQL service. Kysely migrations are an explicit
one-shot catalog-migrate operation; scripts/run-stack.sh runs it before local startup. Runtime
sessions consume an immutable Catalog JSON snapshot tied to the runtime-config lease. It contains
the tables, columns, effective descriptions, sensitivity flags, and active relationships used by
the harness; PostgreSQL is the exclusive runtime authority for database metadata. Authored
workspace YAML remains limited to workspace identity and optional Evidence configuration. The
accepted design is recorded in ADR 0016 and the contracts under docs/contracts/.
Semantic aliases, value descriptions, synonyms, and concepts remain deferred to their dedicated slices.
AI Description Generation uses the catalog's human-owned Sensitive Data Flag. The flag defaults to
false, including for newly synchronized columns. An administrator may request a local sensitivity
analysis for one selected database, selected tables, or selected columns. One deterministic
TypeScript classifier combines metadata, bounded source-content rules, and optional CPU-only NER;
no generative model decides the result. Its sensitive or non_sensitive assessments remain an
unsaved draft until the human reviews and saves any chosen flag changes, including a downgrade to
non-sensitive. Coverage is reported separately; interrupted history may count unprocessed columns.
Each started analysis records a separate Sensitivity Analysis Run with aggregate counters and safe
ordered events. The progress drawer opens before the synchronous request completes, polls the run,
and displays sanitized source-scan and local-NER phase/batch activity while classification is in
progress. This operational history never stores per-column assessments, source values,
matched spans, prompts, or free-form diagnostics. Saving a sensitive decision persists a sanitized
Sensitivity Reason as column Catalog Metadata alongside the human-owned flag; clearing the flag
clears that reason. Reloading still discards an unsaved review draft.
For unprotected columns, up to five source rows and five representative non-null values may be sent
transiently to the configured model provider. Protected columns are omitted from source reads and
replaced in the prompt by deterministic plausible values derived only from their metadata. Existing
descriptions are not regenerated when a flag changes.
The accepted AI-description design is recorded in
docs/plans/2026-08-28-ai-catalog-description-generation.md, with the formal specification in the
adjacent -spec.md document and Gitea issue #4. Gitea issues #5–#11 deliver the implementation.
The runtime deliberately keeps ThothAI's simple operating model: one installation-wide sequential
run owned by the backend, one short-lived Python/LiteLLM completion helper per request, and
persistence limited to the run, its safe ordered text events, and each Generated Description as
soon as it succeeds. The helper performs at most one provider retry and never falls back to another
model. Stop terminates the current helper and retains prior results; three consecutive exhausted
technical batches fail the run. Startup marks stale queued/running work interrupted, and Unlock is
available only when no local start, worker, or helper is live. Runs remain inspectable through a
live SSE log with ordered polling fallback; there is no automatic resume or user-facing generation
CLI. ADRs 0009–0010 record the runtime and source-sampling decisions.
The Installation Model Catalog accepts the protected DEEPSEEK_API_KEY and ZAI_API_KEY
references for metadata-generation providers.
It also accepts a model with no secret reference only when its OpenAI-compatible endpoint is
explicit; this covers the VPN-only AritmoLab Qwen 3.6 server without creating a fake operator
credential. The Python client supplies only its fixed non-secret compatibility placeholder.
The AritmoLab entry also sets disableThinking: true, mapped to the endpoint's chat-template flag,
because its default reasoning prose would violate the worker's exact JSON response contract.
Logical relationship integration with core schema-linking is complete: session creation and resume materialize the active physical/generated/manual map, retrieval-pack generation and Pi receive the same runtime config, and snapshot validation fails closed on a declared missing, invalid, or orphaned endpoint. Broader publication of other Catalog metadata to schema-linking remains a separate future slice.
Deferred follow-up — Sensitive Data Policy in schema-linking. The policy is first delivered
and tested in catalog description generation. Its enforcement for core schema-linking remains
out of scope until the current tickets are closed and the owner has completed the acceptance test.
At that gate, resume the design: tht must receive a read-only projection of the current Sensitive
Data Flags and exclude values from columns marked sensitive from every LSH result before it is
given to Pi. Do not start this integration before the owner gives final approval after that test.
Active deployment work and manual gates
PSD server deployment program
The approved design and executable entry point are:
docs/plans/2026-08-20-psd-server-deployment-program-design.mddocs/plans/2026-08-20-psd-server-deployment-program.mddocs/plans/2026-08-20-psd-server-survey.mddocs/plans/2026-08-20-psd-server-project-a-standalone.mddocs/plans/2026-08-20-psd-server-project-b-authentik.md
Last recorded state:
- survey:
SURVEY_NO_GO; - Project A:
BLOCKED_BY_SURVEY_AND_MUTATION_GATE; - Project B:
BLOCKED_BY_PROJECT_A_AND_PRE_B_GATE.
The deployment is a clean replacement: legacy sessions, indexes, and application configuration
are not migration inputs. The existing stack remains intact until its documented mutation and
rollback gates are explicitly approved. Shared Omics/LocalLLM networks, ETL Evidence, DWH,
dwh-auth, Supabase, Authentik, Superset, and Aritmolab are outside cleanup scope.
Human acceptance guides and sanitized report templates live under docs/testing/ and
docs/testing/evidence/. The remediation checklist is
docs/operations/psd-server-survey-remediation-checklist.md.
Authentication
The local/OIDC authentication remediation passed its automated review on 2026-08-18. Release and PSD mutation gates remain governed by:
docs/architecture/authentication.md;docs/plans/2026-08-18-thothii-authentication-acceptance-and-psd-deployment.md;docs/operations/psd-dwh-auth-rollout.md;docs/testing/authentication-manual-acceptance.md.
Do not infer authorization for server, Nginx, Authentik, database, credential, or cutover changes from an automated PASS.
Verification status
- The P1.1 workspace-directory registry and P2–P6 preprocessing workstreams are implemented and have automated coverage.
- Evidence restructuring has a real PSD acceptance PASS as recorded above.
- AI Description Generation has automated coverage across installation setup, model selection, generation/consolidation scopes, bounded sampling, cancellation/recovery, history, SSE/polling, and the LiteLLM helper boundary.
- L2 tests requiring real providers or remote databases remain opt-in.
- Server deployment, release, and owner-operated acceptance steps remain pending wherever the referenced runbooks require explicit approval.
Run the layer-specific checks documented in AGENTS.md. For release-sensitive changes, also run
the repository contract scripts in scripts/ and build the MkDocs site.
Operational invariants
tht's-c/--configoption follows the subcommand; it is not a global option.--jsoncommands write pristine JSON to stdout.- Persisted phase documents and the decision ledger are the source of session truth; chat is not.
- UI chrome is English; workspace document content retains the workspace language.
- The backend refuses resume for finalized or archived sessions.
- A resume must send
/riprendi-sessione <id>; a new session must send/nuova-domanda. - DWH access is read-only.