mptyl/ThothII
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs: consolidate historical records and verify public manual publication
Publish documentation / publish (push) Successful in 29s
Details

Browse Source
This commit is contained in:
Codex
2026-09-15 14:37:29 +02:00
parent 5f3a7f5975
commit 4ff91e8d6e
63 changed files with 633 additions and 6598 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PROJECT_STATE.md
+95 -719
View File
@@ -1,719 +1,95 @@
# ThothII — Project State
Last updated: 2026-09-15.
This file is the short operational snapshot. Stable commands and the architecture mental model
live in `AGENTS.md`; current design and runtime contracts live under `docs/architecture/`,
`docs/contracts/`, `docs/adr/`, and `docs/evidence.md`. Superseded plans and reports are
available from Git history rather than duplicated in the working tree.
The guarded server migration from a legacy checkout to the schema-v2 installation, Gitea source,
Authentik, internal catalog/embedding services, and the PSD workspace repository is documented in
`docs/operations/server-upgrade-gitea-workspace-v2.md`. Treat its operator gates and rollback
requirements as mandatory; do not replace the running server stack in place.
## Public documentation boundary — 2026-09-15
MkDocs now publishes only its explicit public-manual allowlist: 20 product, installation,
usage and administration pages, plus approved assets. Internal developer/project documents
remain in Git but are excluded from generated pages and search. The build and publication
workflow enforce this boundary with `scripts/verify-public-docs.py` and negative fixtures.
The critical review and proposed (not executed) source-file retirements are in
`docs/maintenance/2026-09-15-documentation-cleanup.md`. It also records pre-existing stale
paths later in this snapshot and obsolete authentication/DWH documentation verifiers;
those require a separate reference/test cleanup, not restoration of superseded guides.
Commit `043ffdfa` passed Gitea publication run 122 and the generated `pages` search index
contains only the 20 public pages. The served `/thothii-docs/` URL is still a stale static
copy dated 2026-08-26 with internal pages: locating/updating its actual server publication
path remains blocked on verified administrative access. Do not claim the live site is clean.
## Session review layout deployed — 2026-09-14
Session-only dialogs now use the visible app bounds: artifact/column review grows
up to 80rem wide and the available height; short confirmations grow to 40rem.
Existing primary actions appear above and below session forms and review content,
with shared handlers, validation and pending state. Stop/delete focus Cancel;
rename focuses the name field. Administration surfaces are unchanged. The activity
shortcut is relabelled To Administration / Vai all’Amministrazione, retaining its
workspace-management destination. 778 frontend tests, five responsive browser
scenarios, typecheck, translations and the production build passed. The owner
authorized deployment and the server frontend was recreated at 18:09 CEST with tag `b1723c34-session-dialogs-20260914`. Frontend is healthy,
Omics serves the new assets, and doctor passes 13/13 checks. Core and Omics web
were not restarted. See DESIGN.md for the layout contract and
`docs/reports/2026-09-14-session-dialogs-release.md` for provenance and rollback.
## Session composer and empty Memory fix deployed — 2026-09-14
The embedded shell now caps its height at the portal mount height, keeping steering
and Stop & save visible. Empty authoritative Memory archives return zero results
without requiring embedding/BM25; SQL-rule embedding is lazy and shared. The live
empty PSD archive reproduces 503 with the current image and succeeds with the
candidate. 54 Memory tests, 90 frontend tests, five browser scenarios and both image
builds passed. The owner authorized the restart and core/frontend were recreated on
2026-09-14 at 17:18 CEST with tags `49333a2d-session-memory-fix`, from code committed
as `d6cdffea`. Both are healthy; production Memory search returns `[]`, Omics serves
the corrected CSS, native doctor passes 13/13 checks, and admissions are reopened.
Session inventory is preserved. Backups and rollback images are retained. Native
CLI diagnostics must run as installation owner UID 10001 with access to Compose;
see `docs/reports/2026-09-14-session-layout-memory-fix.md` for exact commands and
the remaining interactive browser acceptance.
## Full/embedded shell and bilingual interface
Full/embedded shell, EN/IT UI, immutable session interaction language, dark theme,
fullscreen and full-mode logout are implemented. The local Mac descriptor explicitly
sets `shell.mode: full` and `shell.defaultLocale: en`; the native `tht` and local
core/frontend images were updated on 2026-09-13. Omics uses embedded with server-side
identity verification and a replaceable presentation-only PortalAdapter; this does
not mean this branch was verified on the production server. Its changes are
in Omics commit `95154e1`; production deployment remains pending.
See `docs/operations/shell-and-localization.md` for integration and installation
instructions and `docs/reports/2026-09-13-full-shell-implementation.md` for tests,
independent reviews, local browser checks and rollback details.
### Documentation handoff before branch closure — 2026-09-13
Current rendering architecture is in `docs/architecture/application-shell.md`;
the exact portal identity/proxy contract is in `docs/install/authentication-upstream.md`.
`docs/operations/server-codex-handoff.md` is the current server delivery/deploy
runbook, including Omics source integration from GitHub, configuration, tests and
rollback. It supersedes the earlier Omics repository-relay instructions. The acceptance matrix is
`docs/testing/authentication-manual-acceptance.md`. README, documentation navigation,
user/installation/authentication guides and descriptor examples point to these
paths. Local examples explicitly use full/en; the projected server example is
for standalone OIDC, not Omics upstream. No runtime configuration or deployment
was changed by that documentation pass. The 2026-09-14 delivery is prepared for
promotion to main; the actual merge is recorded in Git. PSD acceptance remains pending.
### Navigation readiness and session accordions — 2026-09-13
The Workspace navigation button now carries an accessible green/red readiness
dot instead of a separate text row. Green requires a selected workspace and a
successful, current `ready` response; checking, unavailable and other states are
red, with the state exposed through the tooltip and accessible description.
The backend readiness gate is unchanged.
Active sessions (non-archived) and Archive both start collapsed. Their adjacent
headers are the only visible content below the scope tabs: no Sessions heading
or external selection toolbar. Each nonempty panel owns its Select all and
bulk-delete controls, scoped to that list and preserving the other's selection.
As of 2026-09-14, only one section can be open at a time, and either can be
collapsed. Empty lists show only "No sessions yet." The open section uses the
remaining sidebar height, with scrolling content capped at `min(18rem, 35dvh)`. The mobile
navigation dialog also provides a bounded height. Keyboard controls and labels
are retained. See `DESIGN.md` and `docs/guida-utente.md` for the UI contract.
Verification: 768 frontend unit tests, 20 browser scenarios (including 80 mocked
sessions at 390/1280px), TypeScript and the production frontend build passed.
Only the Mac frontend was recreated; it is healthy at `127.0.0.1:8080`.
Core, catalog, Qdrant and embedding containers were not changed. The prior
frontend image is retained as
`thothii-frontend:before-single-session-accordion-20260914` for rollback.
### Current Omics delivery and server handoff — 2026-09-14
The owner corrected the delivery requirement: Omics is obtained from GitHub,
not relayed to another repository as part of this deployment. Any optional
server-side repository copy is solely the owner's separate concern. This
supersedes the 2026-09-13 relay agreement, including historical delivery notes
in the Omics branch. Do not make another remote publication a prerequisite.
GitHub branch `codex/thothii-embedded-shell` at
`https://github.com/Dallavilla-Tiziano/omics_portal.git` was reverified at
`fca10901a73666ca257d8f4cc4b77066295c400a` (functional commit `95154e1`).
The server operator integrates it with the current code in the confirmed Omics
checkout, normally `/home/chirone/omics_portal`, preserving later server changes.
`docs/operations/server-codex-handoff.md` is the authoritative ordered procedure:
inventory, source verification, native CLI and installation projections,
embedded/upstream auth, Omics templates/static assets/proxy, coordinated rollout,
acceptance and rollback. Server deployment and real IdP acceptance remain pending.
## Current product shape
### Shared Memory/Evidence typography — 2026-09-13
Both detail readers now share Manrope and fixed reading roles: 24px card title,
20px section headings, 16px/1.65 narrative text, and 14px metadata/code. Authored
Markdown subheadings remain subordinate to field headings; inline/fenced code no
longer shrinks cumulatively. Full-width layout, paragraph separation and isolated
FAKE examples are retained. All 762 frontend tests and 18 browser scenarios pass,
including computed typography checks at 390/1280/2400px; typecheck, i18n and Docker
build pass. Only Mac frontend was recreated: `87fca0dc5e19`, image `19f1704b6bb2`,
healthy. Core and data services are unchanged. Rollback image:
`thothii-frontend:before-knowledge-typography-20260913`. See
`docs/reports/2026-09-13-knowledge-typography.md`. No PSD/Omics deployment.
### Knowledge reading and isolated fake Memory examples — 2026-09-13
Memory/Evidence details now use all available width, with display-only paragraph
splitting for long plain prose and preserved code/Markdown/source data. Evidence
provenance renders Markdown; copy controls are accessible two-sheet icons.
Memory's separate Formatting examples section contains four FAKE cards (one per
family), automatically expanded for an empty archive. These client-side examples
cannot be edited/saved/indexed and are never submitted to the model or Memory API.
No real archive content was changed. 762 tests, 18 browser scenarios, typecheck,
i18n and Docker build pass. Only the Mac frontend was recreated: `fc4286b5406d`,
image `cbe0fe0dce55`, healthy; other services unchanged. Rollback image:
`thothii-frontend:before-knowledge-reading-20260913`. Details in
`docs/reports/2026-09-13-knowledge-reading.md`; no PSD/Omics deployment.
### Unified Session entry and complete tab borders — 2026-09-13
The sidebar has one Session/Sessione button: return to the current unfinished
session (including pending creation) without resetting/reconnecting it; otherwise
prepare a new question with existing readiness and dirty-edit guards. Creation
still requires submitting a question. A cold document panel is not a running session.
Session tabs now have 11px horizontal/3px vertical padding, at least 38px height,
and matching 1px borders on every side (gray inactive, red active), with no shared
baseline or overlapping bottom border. This supersedes the earlier border removal.
757 frontend tests, 15 browser scenarios, typecheck, i18n and Docker build pass.
Mac frontend `2844778d311c`, image `d58dccfee3f9`, is healthy; only that service
was recreated. Core/data services are unchanged, with no Omics or server deploy.
Rollback image: `thothii-frontend:before-session-navigation-20260913`.
### Login copy and language-selector focus — 2026-09-13
Removed the redundant login eyebrow/icon and installation-account explanation;
the main sign-in heading remains. The full-header language select no longer
shows an outer focus ring after pointer interaction; keyboard focus remains
visible, including after returning with Tab. EN/IT login and both themes are
covered by 36 targeted tests and 14 browser scenarios; typecheck/build pass.
Only the Mac frontend was rebuilt/recreated: `204efd40d3eb`, image `7dd0752ff823`,
healthy. Other containers are unchanged. Rollback image:
`thothii-frontend:before-login-focus-20260913`. No production deployment.
### Full-header and layout refinements — 2026-09-13
The owner's four visual adjustments are implemented: full header uses Omics
`#CB333B` in both themes with a light complete wordmark/controls; Database status
has 16px clearance below its top divider; workspace/model/Done share a compact
desktop row; session-scope tabs have no bottom border. Embedded has no extra header.
Verified with 71 targeted frontend tests, 11 browser scenarios, typecheck and
Docker production build. The Mac frontend was recreated alone and is healthy
(`8a1608ad7fc6`, image `a2f489ebe9de`); Core/data-service containers are unchanged.
Full/en is retained. Rollback image: `thothii-frontend:before-header-layout-20260913`.
See `docs/reports/2026-09-13-header-layout-refinements.md` for validation and rollback.
### Visual review integrated with the full/embedded shell
At the owner's request, the seven commits through `a59624a6` from
`codex/ui-visual-review` are integrated with the shell/i18n work in this checkout,
`codex/prototype-administration-pages`. Both branches started at `2d1b714e`; the
first full-shell build omitted that lateral branch and regressed the installed UI.
The integrated source retains bundled Manrope, shared type roles, catalog/context
alignment, wordmark sizes and composer autosizing alongside full/embedded and EN/IT.
Local Docker was updated at 12:54 UTC on 2026-09-13: frontend image `605a6e6bba68`,
healthy; Core and all data-service containers were unchanged. Mac remains full/en.
The immediate pre-merge frontend is retained as
`thothii-frontend:before-visual-shell-merge-20260913`.
Verification: 755 frontend tests, 11 Playwright visual/interaction scenarios at five
widths, translation-catalog checks, typecheck and production build. See
`docs/reports/2026-09-13-visual-shell-integration.md` for delivery, provenance and
rollback details. The separate visual-review worktree and its rollback images are
retained. No merge to `main`, push or production deployment is part of this delivery.
Gitea #28–#31 follow-up is implemented in this checkout: Workspace's four tabs,
Database list-first entry without the preparation footer, full-height Pi instructions
with installation-host OS selection, and short Admin navigation labels. Core and
session behavior are retained. At the owner's request, these changes were rebuilt into
local Docker on 2026-09-12 at 18:31 UTC. Core/frontend are healthy and the UI at
`http://127.0.0.1:8080` serves the updated bundle. The host projection now supplies
`THT_HOST_PLATFORM=darwin`, so Pi selects macOS rather than the container's Linux OS.
Persistent dependency containers and volumes were unchanged. Rollback images are tagged
`thothii-core:before-admin-28-31-20260912` and
`thothii-frontend:before-admin-28-31-20260912`. See
`docs/reports/2026-09-12-admin-issues-28-31.md` for verification and deployment details.
The latest context-shelf A and five Administration pages are implemented locally.
At the owner's request, local Docker project `thothii-18998cca7b0a` was rebuilt and
its core/frontend recreated on 2026-09-12. The real UI at `http://127.0.0.1:8080`
serves shelf A; both services and their existing dependencies are healthy.
Runtime model projections were regenerated as schema v2 with the single
`zai/glm-5.3` interaction default. Persistent services/volumes were not recreated.
The local launcher `/private/tmp/thothii-memory-preview.sh` now adds
`/private/tmp/thothii-context-a.compose.yaml` last, building from this checkout
instead of the earlier Memory worktree. Previous images are retained under
`thothii-core:before-context-a-20260912` and
`thothii-frontend:before-context-a-20260912`. Remote server deployment remains pending.
Core/session behavior is retained; one independently remembered workspace/model
pair controls both Core and Admin. Unsaved Admin changes block navigation and
context changes; operation activity locks the selectors. See
`docs/reports/2026-09-12-context-shelf-a-implementation.md` for verification and the
remaining server/Omics integration gate. Prototype alternatives remain untouched.
ThothII is a human-in-the-loop datamart builder with three independently built layers:
```text
frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness → DWH (read-only)
```
The harness owns the deterministic eight-phase NL→SQL workflow and all session persistence.
The backend remains a process/RPC/SSE bridge for sessions and now also owns an isolated PostgreSQL
metadata catalog for administrative database configuration. The frontend renders the review gates
and keeps the live transcript in memory. See
`docs/architecture/components.md` for the detailed component and data-flow map.
## Memory M1–M3, Evidence E1–E3 and joint workflow repair X1 implemented
Browser authorization now derives permissions from validated session roles using the current
catalog. This fixes Memory/Evidence navigation remaining disabled for administrators whose
remembered login predates those permissions; refreshing the page loads the updated permissions.
The owner requested two independent administration projects: Memory management and Evidence
management. Their navigation entries must sit immediately after Database management, as peers;
neither page belongs to Database management. Both require complete browsing, filtering, and CRUD
without an active core session. Shared requirements and the two project briefs are linked from
`docs/plans/2026-09-08-memory-evidence-administration.md`. Memory administration is implemented;
Evidence administration is implemented through external file editing and explicit consolidation.
Evidence editing requires an explicit evolution of the current authoring/publication contract.
The M1 specification is at `docs/plans/2026-09-08-memory-m1-spec.md` and published as
[M1 — Archivio autorevole e amministrazione delle Memory Card](https://git.tylconsulting.it/mptyl/ThothII/issues/27)
with the `ready-for-agent` and `enhancement` labels.
It covers the authoritative store, administrative CRUD, projection recovery, and current
Memory/exemplar producer integration. Its accepted test boundaries are the public harness
service, Fastify APIs, and the AppShell page, joined by a focused real-stack browser path.
The owner confirmed those boundaries and authorized publication on 2026-09-08.
M1 is implemented locally: PostgreSQL authority in `thoth_memory`, versioned installation
migrations, admin CRUD for all four families, structured dependencies and links, explicit
projection recovery, verified recall and current workflow producers. The page requires no
active session or DWH binding. The existing `catalog-migrate` preparation service now runs
Memory migrations as well. Existing installations need that preparation before using M1;
the initial implementation did not deploy or migrate the owner's stacks.
The integrated browser check passed with real authentication, Fastify, ThtRunner, harness,
PostgreSQL and Qdrant; embeddings were deterministic and unrelated Pi/session activity used
test fixtures. See `docs/plans/2026-09-08-memory-m1-validation.md` for results and commands.
M2 is implemented locally: Memory dense/BM25 fusion, physical and business scope filters,
bounded outgoing-link expansion and joint ranking, all resolved against current PostgreSQL
authority. Migration `002_hybrid_projection.sql` makes old dense projections pending until
explicit retry/rebuild; Reference remains separate. The real retrieval check uses the configured
`qwen3-embedding:0.6b` model, a separate Ollama process with a read-only model-volume mount,
and isolated PostgreSQL/Qdrant resources. See `docs/plans/2026-09-09-memory-m2-validation.md`.
M3 is implemented: editable F8 summary grounded in effective approved decisions, explicit
updates with concurrent-edit protection, selected-card/link transactions and durable review
receipts. Finalization no longer saves exemplars implicitly. SQL rules and explained errors
are consulted in the existing F4/F6/F7 gates. Successful Catalog physical synchronization
performs dependency cleanup, preserving the original removals for recovery across restarts.
Migration `003_review_receipts.sql` is required. Validation includes a real GLM 5.3 generation
case against synthetic PostgreSQL data; see `docs/plans/2026-09-09-memory-m3-validation.md`.
Evidence administration and the joint X1 conflict-repair increment are now implemented.
The same local preview was subsequently rebuilt with M3 and migration 003 applied.
Core and frontend now include the final Memory review and physical dependency cleanup.
On 2026-09-09, at the owner's request, the local PSD Docker installation
`thothii-18998cca7b0a` was updated from this worktree. Core/frontend images were rebuilt,
Catalog and Memory migrations completed, and all five services became healthy. The UI is at
`http://127.0.0.1:8080`, using the existing local authentication and persistent volumes.
The `psd-clinical` authoritative Memory archive is initially empty (no legacy import).
The existing installation configuration remains in `/Users/mp/projects/ThothII/deploy/psd/`;
`/private/tmp/thothii-memory-preview.sh` invokes its Compose files with a final build-context
and migration-command override from this worktree. A future build from the main checkout
will use that checkout's code, so retain the worktree override until the changes are integrated.
Memory revision history and compatibility with existing development sessions are not requirements.
The agreed Memory scope includes reusable domain clarifications, SQL construction rules, solved
questions, and explained, approved mistakes to avoid. This extends the current runtime's
`concept_clarified`-only reusable Memory contract. The owner also approved an editable final summary
for proposed additions/updates and Memory consumption in the relevant existing review gates.
Further agreed behavior includes persistent corrections for Memory/Evidence conflicts through
explicit choices, deletion of Memory with invalid dependencies after successful physical schema
synchronization, and bounded functional/regression tests instead of a general quality benchmark.
In-house Memory evolution is approved, including hybrid Qdrant retrieval and explicit card links
traversed in core, without a dedicated graph database. Both capabilities belong to the current
scope. PostgreSQL is the agreed authority for Memory Cards, links, and schema dependencies;
Qdrant is a rebuildable index. Links are reviewed with cards and editable in Administration;
deleting a card removes its incident links while preserving the other cards. The accepted,
implemented architecture is recorded in
`docs/adr/0018-use-postgres-for-memory-and-qdrant-for-retrieval.md`.
For Evidence administration, Q12 rejects a separate editorial publication workflow.
The final R0 direction uses external editors and a manual consolidation command that
validates structure, reports required corrections, updates derived metadata, and
activates the local Evidence index. The operator then checks the diff and runs Git
commit/push manually. No watcher, automated Git, or web editor is required.
The owner's later simplification
instruction removes the requirement to support administrative edits while core work
is in progress. Deliberate corrections made by the workflow itself remain supported.
The context specialist writes Evidence drafts independently of the installation and
without PostgreSQL access; the system refines them and stores them locally for use
and maintenance. The revised Q11 recommendation separates external drafts from a
durable local canonical file archive, removing automatic commit/push from CRUD.
The owner has now accepted local files and requires discoverable, editable Markdown
for nontechnical domain specialists, with no JSONL management surface. Editors on
Mac/PC or vim/nano on the server are the chosen R0 editing surface. Evidence
management keeps browsing/filtering/detail and clearly identifies the persistent
working tree, each Markdown file's actual host path, and the manual commands.
E1 implements editable Curated Evidence v4, deterministic legacy conversion, persistent
local files and a consolidation API with immutable candidates, manual provenance,
deletion records and recoverable activation. E2 connects its installed command, runtime source
selection and administration page. The operator completes Git steps manually.
All 35 PSD units were converted on an isolated copy with identical IDs and typed content.
Tests exercise visible edits through normalization, indexing and recall with real Qdrant;
see `docs/plans/2026-09-09-evidence-e1-validation.md` and
`docs/contracts/curated-evidence-v4.md`. E2 is now running on the local Docker preview:
all 35 PSD units were converted and indexed through the installed command. The editable
host archive is `/Users/mp/projects/ThothII/deploy/psd/evidence-registry/repo/psd-clinical/evidence`.
The original registry-volume checkout and original author repository were retained.
The installation descriptor now includes `workspace-bindings.yaml` and `evidence-host.yaml`
so native maintenance uses the same volumes and checkout as the preview. Descriptor backup:
`/private/tmp/thothii-installation-before-e2.yaml`; previous native binary: `/private/tmp/tht-before-e2`.
The launcher remains `bash /private/tmp/thothii-memory-preview.sh`; its worktree build override
is still required until integration. Runtime lease filenames now include rendered bytes so
upgrading the Evidence renderer does not collide with old immutable configs; Catalog
input fingerprints and readiness are unchanged. See `docs/plans/2026-09-09-evidence-e2-validation.md`.
E3 adds explicit source acquisition/refinement and durable comparisons in Evidence management.
Local drafts go in `evidence/incoming/`; original local documents and configured HTTP/S3
sources are reacquired only by **Import or refresh sources** or installed
`tht workspace evidence refresh --workspace <id>`. Keep/replace decisions, including the
native `workspace evidence decide` command, activate through the existing archive/index
path and have durable retry state. Acquired source versions and remote provenance stay
local; old documentary lineage can coexist with current manual declarations.
The installed preview's 35 PSD sources were verified unchanged with no new proposals.
The previous native CLI is backed up at `/private/tmp/tht-before-e3`.
See `docs/plans/2026-09-09-evidence-e3-validation.md` for tests and validation limits.
X1 adds `reviewer_archive_repair`: closed alternatives with complete before/after content,
explicit rejection/reformulation, admin-only application, durable session receipts and
retry of saved but inactive corrections. The coordinator uses the canonical Memory and
Evidence services; phase approval remains separate. Migration `004_archive_repairs.sql`
is required. See `docs/contracts/archive-repair.md` for authorization and recovery limits.
X1 validation is recorded in `docs/plans/2026-09-09-archive-repair-x1-validation.md`:
both archives were corrected and retrieved through the actual CLI with real PostgreSQL
and Qdrant; desktop/mobile widget behavior and permission failures were verified separately.
The local preview images include X1 and migration 004 is applied. Follow-up technical
acceptance passed with configured GLM 5.3 generating closed conflict alternatives and
the chosen Memory correction persisted and retrieved. An authenticated browser test
also verified both administration pages, navigation order, Evidence filtering, workspace
404s, desktop/mobile layout, and Evidence survival after Memory deletion. All approved
technical increments/checks are complete. A real PSD domain-conflict session remains
the reviewer's semantic acceptance check; automated cases did not modify PSD knowledge.
The joint browser inspection also fixed mobile archive navigation: below 768px,
Memory and Evidence keep the full content width and open navigation in the shared
accessible dialog. Desktop retains its sidebar; the local frontend image includes this fix.
The owner accepted the remaining simplifications and requested explicit clarification
of the core format change and the simple terminal-based Git check. E1 must adapt
the Evidence parser, renderer, authoring, validation, and normalization; convert
existing files and reindex; and verify that visible edits reach core consumption.
Preserve the internal typed model where possible. This precedes the administrative
page and is not merely a presentation change. Human inspection uses normal Git
status and optional line-level diff commands; no custom diff viewer or mandatory
double review is required.
The suggestion of making PostgreSQL the Evidence authority was withdrawn after this
clarification; it was never implemented or accepted as a replacement for Q11.
Q13 keeps manual corrections active when updated sources contradict them, until an
administrator resolves the comparison; deleted Evidence must not be regenerated
automatically. Q14 allows direct manual creation and records a manual declaration
as the current source, preserving any original document as distinct provenance.
Q15 refreshes external sources only on explicit administrator request; normal saves
and lookups do not reacquire them. These decisions, now implemented through E1–E3, are recorded in
`docs/adr/0019-author-evidence-in-app-with-automatic-activation.md`.
The decision-by-decision review is recorded in
`docs/plans/2026-09-08-memory-evidence-simplification-review.md`; it distinguishes the
new constraints from the revised technical recommendations. It also specifies a
sequential save with minimal durable retry state and direct Memory cleanup after
successful schema synchronization, without new queues or event infrastructure.
The delivery order remains Memory, Evidence, and persistent conflict repair between
both modules. Local curated Evidence must survive preprocessing Clear and be backed
up as primary data; the Qdrant projection remains rebuildable.
No runtime gate change or administrative page is implemented yet.
## Evidence restructuring — accepted
The evidence restructuring and PSD migration completed real acceptance on 2026-08-25.
- The curated PSD revision contains 35 approved Evidence units and 60 review items.
- The PSD authoring repository publishes all 35 units using Curated unit schema v3. Its table-free
presentation uses hidden canonical metadata, wrapping Markdown scope lists, list-based enum
values, and collapsed technical provenance. Long domain rules now have a deterministic
human-readable presentation while retaining their exact canonical text for vector ingestion.
`tht evidence migrate <workspace-root>` performs the deterministic v1/v2 upgrade and older-v3
presentation rewrite without model calls. The structured-rule PSD rewrite is currently local and
pending commit/publication.
- The accepted snapshot is
`psd-clinical-675990d90eae51da6f2bd51b1ae2609f245772ef-snapshot`.
- The active generation is `gen:f968b3bd7a553dbfef3cf47093698f2bc7f95f11`.
- Retrieval acceptance reached 20/20 Hit@10.
- A real session, `20301df7-cad7-403d-a4c1-9f35c9d07b66`, completed F1–F8 with five
receipts, three CTEs, and a final result of 78 patients.
- The durable acceptance record is
`docs/testing/evidence/evidence-restructuring-psd-acceptance-2026-08-25.md`.
The canonical authoring, validation, publication, materialization, and preprocessing flow is
documented in `docs/evidence.md`. The governing contracts are
`docs/contracts/workspace-evidence-v3.md` and
`docs/contracts/workspace-preprocessing-cli.md`.
The incremental server procedure for the `260906-preprocessing-complete` release is
`docs/operations/server-handoff-260906-preprocessing-complete.md`.
## Workspace preprocessing and configuration
The native host CLI `tht` is the operator surface. Workspace preprocessing runs through:
```sh
tht --installation /absolute/path/thothii-installation.yaml \
workspace preprocess run --workspace <workspace-id>
```
This complete one-shot command uses the profile-gated `workspace-maintenance` service. Partial DWH,
schema, Evidence, and vector mutation commands are retired.
The right Administration sidebar invokes that same operation for the selected workspace. It shows
only current readiness or the latest bounded failure diagnostic; there is no preprocessing history.
Known non-ready state disables **Session** when it would start a new question,
while backend admission remains authoritative.
The same control exposes an inline-confirmed **Clear** action to remove replaceable reference
vectors, LSH, corpus, and checkpoints while preserving the separate Memory collection. The host CLI
equivalent is `workspace preprocess clear`.
Each workspace now uses `<workspace>-reference` for Schema, relationships, and Evidence and
`<workspace>-memory` for `memory` and `solved_question`. Clear and preprocessing own only the former.
LSH ownership additionally binds the Catalog database ID and Metadata Content Revision, so derived
values cannot be reused across database identities or Catalog revisions.
Workspace descriptors use schema v4 and contain only workspace identity and optional Evidence.
PostgreSQL Metadata Catalog owns database identity, binding, schema, descriptions, sensitivity, and
relationships; model, provider, embedding, and vector-store configuration is installation-owned. For
PSD, workspace content and runtime roots point to the
separate uncommitted repository `/Users/mp/projects/tht-workspace-psd`. Secrets remain outside
Git and are supplied only through installation-local protected files.
## Installation Model Catalog
`thothii-installation.yaml` schema version 2 is the only operator-authored source for session,
metadata-generation, and embedding models. The host `tht` lifecycle validates `modelCatalog` and
regenerates the backend catalog, Pi `models.json`/`settings.json`, and Compose override under the
installation-local `generated/` directory. Those projections are replaceable runtime adapters:
they are not edited, backed up, or treated as configuration.
Core and Administration now share one canonical `modelCatalog.defaults.interaction` per installation,
independent of workspace. Runtime catalog schema v2 contains only `defaultInteraction`; apply the host,
backend, and regenerated projections together. Equal legacy defaults normalize on read; conflicting
ones require an explicit operator choice. With Admin AI configured, selectable models are the
intersection of the Pi and LiteLLM adapters; Core-only installations remain supported without Admin AI.
The existing Core and Database controls share the operational selection. Explicit model choices are
remembered per authenticated user/application mount in this browser, not in installation settings.
Resume uses that selection/default while preserving the historical workspace/revision and manifest.
Every model must be manually exercised in both Core and Admin as documented in
`docs/general/pi-configuration.md`; validation is not a live model certification.
These changes and shelf A are now deployed to local Docker; remote deployment remains pending.
PSD DeepSeek Pro/Flash now share canonical `deepseek/...` identities across native Pi and LiteLLM,
using the existing `DEEPSEEK_API_KEY` bundle entry. Its value was confirmed identical to the working
Pi key without exposing it; no secret files were changed. The duplicate `deepseek-metadata` descriptor
is removed locally and the tracked example uses the shared provider. `secret_env` overrides legacy
Pi auth only inside temporary runtime snapshots and fails closed if the bundle key is missing.
The original auth store/history and `zai/glm-5.3` default are preserved. Local Docker projections
and core/frontend were updated together on 2026-09-12. Regenerate projections with
the matching release for the separate server deployment.
Provider authentication declares
one explicit mode (`secret_env`, `pi_auth`, or `none`); `secret_env` names a protected bundle key.
The backend settings store now owns only the selected workspace and thinking level. Existing v1
installations use the explicit catalog migration command; schema-v3 workspace descriptors are
converted deterministically in their curator-owned repository before commit. Strict runtime loading
does not silently infer or merge legacy sources. ADR 0013 and
`docs/plans/2026-09-02-installation-model-catalog.md` record the decision and implementation.
## Database management
The database, table, and authoritative physical-schema catalog slices are implemented. Database
Management now opens the Fleet Ledger presentation by default inside `AppShell`, lists every YAML
workspace, creates at most one PostgreSQL database configuration per workspace, edits direct
PostgreSQL, REST API, or SSH-tunnel installation bindings, replaces write-only encrypted secrets,
and tests supported connector bindings. The surface keeps one responsive AG Grid visible at a time:
databases lead to tables, tables lead to columns, and relationships are a sibling database view.
Parent navigation remains explicit through the breadcrumb and emphasized back control.
Selection-scoped operations use one action selector plus an explicit **Run** control; ineligible
actions remain visible with their disabled reason, while row-scoped actions stay in the pinned final
column. The KPI strip reads installation-wide or selected-database aggregates from
`GET /catalog/metrics`. Database configuration, metadata editors, synchronization history,
description history, and sensitive-field review/history use the production APIs in right-side
drawers rather than prototype fixtures; closing a history drawer does not stop its background run.
Sensitive-field review is now driven by the versioned local `sensitivity-v4` policy, not by a
catalog model. The backend reads selected source tables through read-only, database-specific
adapters and makes every `sensitive | non_sensitive` draft decision in the TypeScript
`SensitivityClassifier`. A single validated match protects the column. Tables up to 1,000 rows are
fully scanned; larger tables use breadth-first 300, 1,000, and text-only 3,000-value targets, with a
five-second limit per source query and no global request deadline. Source failures fail the run
instead of yielding `unknown`; coverage remains visible separately from the proposal. Draft
assessments remain transient until an administrator explicitly saves them. Optional GLiNER2
evidence is CPU-only, offline, opt-in, and never replaces the deterministic decision point; see
`docs/operations/sensitivity-analysis.md`. The earlier v1 PSD shadow comparison kept NER disabled by
default; see `docs/reports/2026-09-02-psd-sensitivity-shadow.md`. The v2 comparison completed all
2,275 columns: CPU NER added 18 sensitive proposals and increased warm runtime from 50.1 to 61.3
seconds; see `docs/reports/2026-09-03-psd-progressive-sensitivity-shadow.md`.
Version 4 excludes declared `bigint` primary-key columns and conventionally named `pk bigint`
columns before source inspection, reporting both as non-informative structural identifiers while
distinguishing declared constraints from inferred roles.
Physical membership, source
comments, column types/default/nullability/PK positions, and constraint-level ordered FK pairs are
projections of the external schema. They cannot be created, renamed, or structurally edited by
hand, but administrators can explicitly clear catalog tables, columns, or relationships without
touching the source database, binding, configuration, or secrets. Table deletion cascades through
columns and relationships; table-scoped relationship cleanup includes incoming and outgoing
relationships. Curated and generated descriptions are editable; generated descriptions start null
and Database Management can generate or consolidate them for selected tables, selected columns,
all targets, or only targets whose Generated Description is missing.
Relationship Management is now reachable directly from each configured Fleet database. One
Relationship Map shows read-only Physical Relationships together with Generated and Manual Logical
Relationships, with Active, Excluded, and All filters. Administrators can add a single-column
relationship, run deterministic name/PK/type inference, exclude or restore a logical relationship,
or delete it permanently. Exclusion retains a tombstone that a rebuild cannot reactivate; permanent
deletion allows a later rebuild to infer the same endpoints again. Inference uses no LLM, embedding,
or source values. It supports normalized table-qualified names, unique non-generic PK names,
composite-PK source columns, and the `*time_key -> dim_time.<single PK>` warehouse convention while
ignoring bare generic names. Explicit table/column metadata cleanup remains a destructive boundary: it removes
the attached logical relationships and exclusions and requires a full schema synchronization before
inference or runtime publication can continue.
The previous Database Management renderer remains a temporary comparison fallback for development
and staging only: `?db-ui=legacy` is honored in Vite development or when
`VITE_DB_MANAGEMENT_LEGACY=true`; it is not a production presentation. The standalone Fleet Ledger
prototype on port `5173` also remains temporary until owner acceptance of the integrated surface,
after which both migration aids can be removed.
Schema refresh is one durable asynchronous engine with database-table, selected-table-column,
relationship, and full-database actions. Database-level menus expose only the table, relationship,
and full scopes; selecting tables exposes column synchronization plus manual column and relationship cleanup for that subset. Database selections
also expose manual table and relationship cleanup. Cleanup selections are atomic and share the
one-active-operation-per-database exclusion with synchronization. Runs have leases and
restart recovery, atomic apply, destructive-diff confirmation with re-scan, cancellation before
apply, retained history, and a live SSE log with polling fallback. Null metadata renders blank
rather than as a placeholder.
Direct PostgreSQL and strict known-host-verified OpenSSH use `pg_catalog`. REST bindings use the
typed full-snapshot `POST /rpc/schema_snapshot` contract when available. Servers such as the
current PSD endpoint that exposes only `POST /rpc/run_query` use one catalog-owned read-only query
to return the exact same strict v1 snapshot in a single round trip. Both paths remain fail-closed:
an absent capability, query error, partial result, or invalid snapshot applies no catalog changes.
SSH is not yet enabled for NL→SQL session runtime.
The catalog runs in the internal `catalog-db` PostgreSQL service. Kysely migrations are an explicit
one-shot `catalog-migrate` operation; `scripts/run-stack.sh` runs it before local startup. Runtime
sessions consume an immutable Catalog JSON snapshot tied to the runtime-config lease. It contains
the tables, columns, effective descriptions, sensitivity flags, and active relationships used by
the harness; PostgreSQL is the exclusive runtime authority for database metadata. Authored
workspace YAML remains limited to workspace identity and optional Evidence configuration. The
accepted design is recorded in ADR 0016 and the contracts under `docs/contracts/`.
Semantic aliases, value descriptions, synonyms, and concepts remain deferred to their dedicated
slices.
AI Description Generation uses the catalog's human-owned Sensitive Data Flag. The flag defaults to
`false`, including for newly synchronized columns. An administrator may request a local sensitivity
analysis for one selected database, selected tables, or selected columns. One deterministic
TypeScript classifier combines metadata, bounded source-content rules, and optional CPU-only NER;
no generative model decides the result. Its `sensitive` or `non_sensitive` assessments remain an
unsaved draft until the human reviews and saves any chosen flag changes, including a downgrade to
non-sensitive. Coverage is reported separately; interrupted history may count unprocessed columns.
Each started analysis records a separate Sensitivity Analysis Run with aggregate counters and safe
ordered events. The progress drawer opens before the synchronous request completes, polls the run,
and displays sanitized source-scan and local-NER phase/batch activity while classification is in
progress. This operational history never stores per-column assessments, source values,
matched spans, prompts, or free-form diagnostics. Saving a sensitive decision persists a sanitized
Sensitivity Reason as column Catalog Metadata alongside the human-owned flag; clearing the flag
clears that reason. Reloading still discards an unsaved review draft.
For unprotected columns, up to five source rows and five representative non-null values may be sent
transiently to the configured model provider. Protected columns are omitted from source reads and
replaced in the prompt by deterministic plausible values derived only from their metadata. Existing
descriptions are not regenerated when a flag changes.
The accepted AI-description design is recorded in
`docs/plans/2026-08-28-ai-catalog-description-generation.md`, with the formal specification in the
adjacent `-spec.md` document and Gitea issue #4. Gitea issues #5–#11 deliver the implementation.
The runtime deliberately keeps ThothAI's simple operating model: one installation-wide sequential
run owned by the backend, one short-lived Python/LiteLLM completion helper per request, and
persistence limited to the run, its safe ordered text events, and each Generated Description as
soon as it succeeds. The helper performs at most one provider retry and never falls back to another
model. Stop terminates the current helper and retains prior results; three consecutive exhausted
technical batches fail the run. Startup marks stale queued/running work interrupted, and Unlock is
available only when no local start, worker, or helper is live. Runs remain inspectable through a
live SSE log with ordered polling fallback; there is no automatic resume or user-facing generation
CLI. ADRs 0009–0010 record the runtime and source-sampling decisions.
The Installation Model Catalog accepts the protected `DEEPSEEK_API_KEY` and `ZAI_API_KEY`
references for metadata-generation providers.
It also accepts a model with no secret reference only when its OpenAI-compatible endpoint is
explicit; this covers the VPN-only AritmoLab Qwen 3.6 server without creating a fake operator
credential. The Python client supplies only its fixed non-secret compatibility placeholder.
The AritmoLab entry also sets `disableThinking: true`, mapped to the endpoint's chat-template flag,
because its default reasoning prose would violate the worker's exact JSON response contract.
Logical relationship integration with core schema-linking is complete: session creation and resume
materialize the active physical/generated/manual map, retrieval-pack generation and Pi receive the
same runtime config, and snapshot validation fails closed on a declared missing, invalid, or orphaned
endpoint. Broader publication of other Catalog metadata to schema-linking remains a separate future
slice.
**Deferred follow-up — Sensitive Data Policy in schema-linking.** The policy is first delivered
and tested in catalog description generation. Its enforcement for core schema-linking remains
out of scope until the current tickets are closed and the owner has completed the acceptance test.
At that gate, resume the design: `tht` must receive a read-only projection of the current Sensitive
Data Flags and exclude values from columns marked sensitive from every LSH result before it is
given to Pi. Do not start this integration before the owner gives final approval after that test.
## Active deployment work and manual gates
### PSD server deployment program
The approved design and executable entry point are:
- `docs/plans/2026-08-20-psd-server-deployment-program-design.md`
- `docs/plans/2026-08-20-psd-server-deployment-program.md`
- `docs/plans/2026-08-20-psd-server-survey.md`
- `docs/plans/2026-08-20-psd-server-project-a-standalone.md`
- `docs/plans/2026-08-20-psd-server-project-b-authentik.md`
Last recorded state:
- survey: `SURVEY_NO_GO`;
- Project A: `BLOCKED_BY_SURVEY_AND_MUTATION_GATE`;
- Project B: `BLOCKED_BY_PROJECT_A_AND_PRE_B_GATE`.
The deployment is a clean replacement: legacy sessions, indexes, and application configuration
are not migration inputs. The existing stack remains intact until its documented mutation and
rollback gates are explicitly approved. Shared Omics/LocalLLM networks, ETL Evidence, DWH,
`dwh-auth`, Supabase, Authentik, Superset, and Aritmolab are outside cleanup scope.
Human acceptance guides and sanitized report templates live under `docs/testing/` and
`docs/testing/evidence/`. The remediation checklist is
`docs/operations/psd-server-survey-remediation-checklist.md`.
### Authentication
The local/OIDC authentication remediation passed its automated review on 2026-08-18. Release and
PSD mutation gates remain governed by:
- `docs/architecture/authentication.md`;
- `docs/plans/2026-08-18-thothii-authentication-acceptance-and-psd-deployment.md`;
- `docs/operations/psd-dwh-auth-rollout.md`;
- `docs/testing/authentication-manual-acceptance.md`.
Do not infer authorization for server, Nginx, Authentik, database, credential, or cutover changes
from an automated PASS.
## Verification status
- The P1.1 workspace-directory registry and P2–P6 preprocessing workstreams are implemented and
have automated coverage.
- Evidence restructuring has a real PSD acceptance PASS as recorded above.
- AI Description Generation has automated coverage across installation setup, model selection,
generation/consolidation scopes, bounded sampling, cancellation/recovery, history, SSE/polling,
and the LiteLLM helper boundary.
- L2 tests requiring real providers or remote databases remain opt-in.
- Server deployment, release, and owner-operated acceptance steps remain pending wherever the
referenced runbooks require explicit approval.
Run the layer-specific checks documented in `AGENTS.md`. For release-sensitive changes, also run
the repository contract scripts in `scripts/` and build the MkDocs site.
## Operational invariants
- `tht`'s `-c`/`--config` option follows the subcommand; it is not a global option.
- `--json` commands write pristine JSON to stdout.
- Persisted phase documents and the decision ledger are the source of session truth; chat is not.
- UI chrome is English; workspace document content retains the workspace language.
- The backend refuses resume for finalized or archived sessions.
- A resume must send `/riprendi-sessione <id>`; a new session must send `/nuova-domanda`.
- DWH access is read-only.
# Project state
Updated: 2026-09-15. This is a current snapshot, not a release diary. Stable commands
and invariants are in [AGENTS.md](AGENTS.md); prior snapshots remain in Git.
## Current contracts
- React supports full/embedded rendering independently of local/OIDC/upstream auth,
with EN/IT UI and immutable session interaction language. See
[application shell](docs/architecture/application-shell.md) and
[localization](docs/operations/shell-and-localization.md).
- PostgreSQL Metadata Catalog owns database identity, binding, schema, descriptions,
sensitivity and relationships for all core consumers. Workspace schema v4 contains
identity and optional Evidence only. Installation schema v2 is the authored model
catalog source. See [overview](docs/architecture/overview.md) and
[model configuration](docs/general/pi-configuration.md).
- The harness owns workflow persistence; chat is not the durable session record.
Memory uses PostgreSQL authority and derived Qdrant dense/BM25 search. Editable
Evidence has local archive authority and manual consolidation. File save, search
activation and Git publication have distinct outcomes. See
[Memory](docs/gestione-memory.md), [Evidence](docs/contracts/curated-evidence-v4.md)
and [consolidated release evidence](docs/reports/knowledge-archives-release.md).
- Reference preprocessing must not clear Memory. Use the installation-scoped
`tht --installation /absolute/path/thothii-installation.yaml workspace preprocess run`
and its [contract](docs/contracts/workspace-preprocessing-cli.md).
- DWH sessions are read-only. SSH tunnels support database-management diagnostics
and metadata synchronization, not NL→SQL session creation; use direct or REST
transport for sessions.
## Installation and workspace boundaries
Fresh standalone installations follow the manual terminal procedures in
[Italian](docs/install/standalone-manual-it.md) or
[English](docs/install/standalone-manual-en.md), without an installer or launcher.
The [Compose reference](docs/operations/compose-reference.md) is for maintainers,
not another quick start. Catalog and Memory migrations are explicit.
Descriptors, authentication, provider credentials, certificates and runtime bindings
stay in protected installation-local paths. Do not copy secrets into examples or
workspace Git. Legacy runtime snapshots may use absolute paths and protected
`harness/.env`; do not silently relocate them.
PSD authoring is separate at `/Users/mp/projects/tht-workspace-psd`. Its GitHub
repository was copied to private Gitea
[workspace_psd](https://git.tylconsulting.it/mptyl/workspace_psd), preserving both
branches. It is a copy, not automatic synchronization. Running installations were
not repointed to a different workspace remote.
## Recorded deployments and server authority
Use the ordered [server handoff](docs/operations/server-codex-handoff.md) for
coordinated ThothII/Omics upgrades. Omics integration uses GitHub
`Dallavilla-Tiziano/omics_portal`, with no Gitea relay prerequisite. Omics uses
embedded/upstream identity, not another ThothII OIDC login. Read
[upstream authentication](docs/install/authentication-upstream.md) before changes.
Last recorded application deliveries (not a fresh runtime attestation):
- [Session dialogs](docs/reports/2026-09-14-session-dialogs-release.md):
`b1723c34-session-dialogs-20260914`, frontend-only.
- [Session layout/Memory fix](docs/reports/2026-09-14-session-layout-memory-fix.md):
`49333a2d-session-memory-fix`, core/frontend.
Keep those reports and rollback instructions while operator gates remain open.
A later deployment does not prove every earlier acceptance item passed.
## Remaining acceptance and design gates
- Fresh-machine Mac, Windows/WSL2 and Linux installation acceptance, including real
DWH/model endpoints, remains a separate operator exercise.
- Real IdP/portal login, logout, embedded interaction and PSD semantic acceptance
follow the [manual matrix](docs/testing/authentication-manual-acceptance.md) and
delivery reports; synthetic tests do not close them.
- [Security hardening](docs/plans/2026-09-08-security-hardening-prd.md) is a draft.
Revalidate SEC01–12 and obtain design approval before implementation or real
server/IdP/DWH mutation.
- Optional NER remains opt-in; labeled Italian quality, benchmark and licensing
acceptance are not implied by document cleanup.
- Semantic aliases, value descriptions, synonyms/concepts, dialect and multi-schema
extensions remain explicit design work. Current sensitivity delivery follows the
Catalog contract; additional policies require their own acceptance.
- Legacy database UI fallback (`?db-ui=legacy`, dev/staging) and prototype removal
remain subject to owner acceptance.
## Documentation maintenance
MkDocs publishes only 20 product/operator pages and five approved assets.
Architecture, contracts, ADRs, plans, research, tests and release evidence are
excluded from HTML and search. The repository itself is public: editorial exclusion
is not confidentiality.
The [cleanup record](docs/maintenance/2026-09-15-documentation-cleanup.md) records
retired sources and retained gates. Main contains source; Actions generates the
`pages` branch. The live site requires the separate explicit deployment described
in [public manual publication](docs/operations/public-docs-publication.md).