Compare commits

...
Author SHA1 Message Date
Codex ec0421e9fd feat: publish verified installation images and native release bundles 2026-09-28 17:46:09 +02:00
Codex 55f3569e55 feat(cli): validate prerequisites and seal installation plans 2026-09-28 17:03:25 +02:00
Codex b9c3369e7b feat(cli): prepare and validate application documents offline 2026-09-28 16:33:07 +02:00
Codex 64e6b9664a feat(cli): prepare and validate workspace documents offline
Reuse the runtime catalog and workspace parsers in a standalone helper paired with tht. Add document templates, safe diagnostics, local Evidence checks, native bundle builds, shared CLI fixtures and IT/EN preparation guides. Record the approved document-first specification and ticket breakdown. Refs #43.
2026-09-28 15:35:25 +02:00
Codex 67ee52624c wip: guided standalone installation and workspace checks 2026-09-26 16:41:15 +02:00
Codex 0d2e573e0d fix(ui): reset session view on stop and exit 2026-09-26 16:40:37 +02:00
Codex bd416f7327 Fix new-question landing and question-language HITL
Publish documentation / publish (push) Successful in 34s
Reset the activity panel when starting a new question so the landing navigation is restored. Detect and persist the original question language, pass it through runtime and widget descriptors, and scope HITL controls to that language.

Validated with gate, session, backend and frontend tests, TypeScript checks, Ruff and strict docs build. Rebuilt and restarted local core/frontend; both healthy and serving HTTP successfully.
2026-09-21 19:47:22 +02:00
Codex 23e52c80de Record verified Qwen documentation publication
Publish documentation / publish (push) Successful in 23s
2026-09-21 16:27:10 +02:00
Codex 84084bba37 Fix Qwen session tool calls and expose thinking compatibility
Publish documentation / publish (push) Successful in 30s
2026-09-21 16:23:51 +02:00
pinoricci1956 efd7d788d9 correzione scroller verticale pagina di configurazione catalogo 2026-09-16 11:59:51 +02:00
Codex b1c510a097 fix(docs): preserve theme assets in deny-by-default publication
Publish documentation / publish (push) Successful in 36s
2026-09-16 09:36:30 +02:00
Codex 5f3680a0fb docs: record verified live manual publication
Publish documentation / publish (push) Successful in 28s
2026-09-15 14:39:46 +02:00
Codex 4ff91e8d6e docs: consolidate historical records and verify public manual publication
Publish documentation / publish (push) Successful in 29s
2026-09-15 14:37:29 +02:00
Codex 5f3a7f5975 docs: record stale public site publication blocker
Publish documentation / publish (push) Successful in 23s
2026-09-15 10:28:49 +02:00
Codex 043ffdfad6 docs: separate public manual from internal project documentation
Publish documentation / publish (push) Successful in 27s
2026-09-15 10:26:35 +02:00
Codex 6a4634dcf1 Merge manual standalone installation documentation
Publish documentation / publish (push) Successful in 35s
2026-09-15 10:06:33 +02:00
Codex 84804be9f8 docs: publish bilingual manual standalone installation guides 2026-09-15 10:06:28 +02:00
User c3caba94dd fix(ui): expand session dialogs and repeat confirmation actions
Publish documentation / publish (push) Successful in 24s
2026-09-14 18:13:53 +02:00
User b1723c34c4 docs: record server rollout of session and memory fixes
Publish documentation / publish (push) Successful in 32s
2026-09-14 17:25:23 +02:00
User d6cdffea62 fix: keep embedded session controls visible and handle empty memory
Publish documentation / publish (push) Successful in 34s
Cap the embedded shell at its portal container height so steering and stop controls remain accessible. Skip vector retrieval for an empty authoritative Memory archive and compute SQL-rule embeddings lazily.

Validated with 54 Memory tests, 90 frontend tests, five browser scenarios, frontend and Docker builds, and a read-only comparison against the real empty Memory archive.
2026-09-14 17:15:00 +02:00
Codex 49333a2d35 Merge full and embedded shell, administration UI and server handoff
Publish documentation / publish (push) Successful in 1m21s
2026-09-14 15:08:47 +02:00
Codex b006b94479 docs: prepare server Codex deployment handoff for ThothII and Omics 2026-09-14 15:08:46 +02:00
Codex bdcd8fcd28 fix(ui): open one session accordion panel at a time 2026-09-14 01:09:45 +02:00
Codex cf90c1bd51 fix(ui): collapse session lists and scope selection controls to panels 2026-09-13 18:12:27 +02:00
Codex 571a4bcaa2 fix(ui): show workspace readiness dot and bounded session accordions 2026-09-13 17:39:33 +02:00
Codex 9051463654 docs: document full and embedded rendering with server authentication 2026-09-13 17:28:10 +02:00
Codex 26c5605ff7 fix(ui): unify Memory and Evidence reading typography 2026-09-13 17:07:37 +02:00
Codex 3535fda958 fix(ui): improve knowledge reading and add isolated formatting examples 2026-09-13 16:52:53 +02:00
Codex 2953f6b608 fix(ui): unify Session navigation and restore uniform tab borders 2026-09-13 16:22:20 +02:00
Codex 45db3a239b fix(ui): simplify login and suppress pointer focus ring on locale select 2026-09-13 15:57:57 +02:00
Codex 7d826e46c0 fix(ui): match Omics header and compact workspace layout 2026-09-13 15:36:08 +02:00
Codex 648434a32e docs: record approved Omics GitHub to PSD relay handoff 2026-09-13 15:22:30 +02:00
Codex 023b822f83 Merge visual review into full shell and preserve bilingual layout 2026-09-13 14:55:58 +02:00
Codex d8a29bfbdd Add full shell, replaceable Omics adapter and bilingual interaction
Implement approved specification #32 and tickets #33-#37. Keep host authentication server-verified and pin session interaction language. Compile scoped base selectors for browser compatibility and retain full gutters during CSS pruning.
2026-09-13 14:26:39 +02:00
408 changed files with 21404 additions and 9119 deletions
+4
View File
@@ -30,3 +30,7 @@ coverage/
data/
sessions/
workspace-registry/
.tht/
deploy/local/
+20 -4
View File
@@ -7,6 +7,11 @@ on:
paths:
- "docs/**"
- "mkdocs.yml"
- "scripts/build-docs.sh"
- "scripts/verify-public-docs.py"
- "scripts/test-verify-public-docs.py"
- "scripts/verify-auth-docs.py"
- "scripts/test-verify-auth-docs.py"
- "docs/requirements.txt"
- ".gitea/workflows/publish-docs.yml"
workflow_dispatch:
@@ -34,14 +39,25 @@ jobs:
with:
python-version: "3.x"
cache: pip
cache-dependency-path: docs/requirements.txt
cache-dependency-path: docs/requirements.lock
- name: Install MkDocs dependencies
run: python -m pip install -r docs/requirements.txt
run: python -m pip install -r docs/requirements.lock
- name: Test public documentation boundary
run: python scripts/test-verify-public-docs.py
- name: Test current authentication documentation
run: |
python scripts/verify-auth-docs.py auth
python scripts/verify-auth-docs.py dwh
python scripts/test-verify-auth-docs.py auth
python scripts/test-verify-auth-docs.py dwh
- name: Build documentation
# Some documented source files intentionally live outside docs/.
run: mkdocs build
run: |
mkdocs build --strict
python scripts/verify-public-docs.py
- name: Publish generated site to the pages branch
working-directory: site
+1
View File
@@ -46,6 +46,7 @@ deploy/secrets/*
# Per-installation configuration generated by `tht setup` (examples stay tracked).
deploy/*/thothii-installation.yaml
deploy/*/operator.env
deploy/*/auth/
deploy/*/generated/
deploy/*/secrets/*
!deploy/*/secrets/.gitkeep
+12 -2
View File
@@ -98,8 +98,18 @@ frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness →
- **`tht`'s `-c`/`--config` is a PER-COMMAND option** — it must follow the subcommand, never
precede it (`ThtRunner.buildArgv` enforces this; prepending caused live 500s).
- **`--json` output must be pristine** (only valid JSON on stdout) — used as a machine contract.
- **UI strings are English; document *content* stays the workspace language** (Italian for
`psd`) because it's the real data. Only chrome/labels are English.
- **Localization:** deterministic UI uses the EN/IT catalogs with English fallback;
model interaction uses the session manifest's immutable `interaction_language`.
Workspace content, SQL, and identifiers remain unchanged. For shell modes, portal
integration, or translations, read `docs/operations/shell-and-localization.md`.
- **Server deployment:** for the coordinated ThothII/Omics upgrade, follow
`docs/operations/server-codex-handoff.md`; it supersedes earlier Omics delivery
instructions. Omics source integration uses GitHub with no repository relay prerequisite.
- **Server identity:** for portal login/logout, proxy headers, or Omics deploy, read
`docs/install/authentication-upstream.md` before changing authentication. Omics
uses embedded/upstream, not a second ThothII OIDC login. Full/embedded rendering
is documented in `docs/architecture/application-shell.md`; release acceptance
is in `docs/testing/authentication-manual-acceptance.md`.
- **Workspace schema v4** defines workspace identity and optional Evidence only. PostgreSQL Metadata
Catalog owns database identity, binding, schema, descriptions, sensitivity, and relationships;
embedding/model facts come from the installation catalog. The legacy `harness/workspaces/*.yaml` runtime snapshots still use
+54
View File
@@ -595,6 +595,60 @@ essere ripristinabile con refresh e cronologia e non contiene valori transitori
un portale host. Conserva la propria gerarchia funzionale, ma deve rispettare la geometria,
l'autenticazione e le regole responsive del portale host.
**Full Thoth Shell** — L'esperienza Thoth autonoma che possiede il proprio header e il proprio
layout di pagina. Non replica la navigazione amministrativa del portale host e non dipende dal suo
template visuale.
**Shell mode** — La scelta di installazione fra `embedded` e `full`. Determina chi possiede il
chrome globale, i comandi di identità e le integrazioni visuali, ma non cambia il workflow o la
persistenza delle sessioni.
**Fullscreen state** — Lo stato temporaneo in cui il documento applicativo occupa il fullscreen
del browser. È distinto da `Shell mode`: una Full Thoth Shell può essere aperta senza fullscreen;
il passaggio è attivato da un comando esplicito e può essere annullato con la stessa azione o con
il comando nativo del browser.
**Portal Shell Adapter** — Il confine sostituibile che traduce lo stato e i comandi del chrome di
un portale host nel modello semantico usato da Thoth. L'adapter non possiede autorizzazione,
sessioni di workflow o contenuti del modello.
**Host Shell State** — Il minimo stato visuale fornito dal portale host: locale UI, tema e stato
fullscreen. In una Embedded Thoth Shell è la fonte autorevole per queste preferenze;
non include identità, token o stato di autenticazione, che restano responsabilità dell'accesso.
**UI locale** — La lingua delle label, dei messaggi, dei tooltip, degli stati e delle istruzioni
non generate dal modello nell'interfaccia Thoth. È distinta dalla lingua dei contenuti di un
workspace.
**Interaction language** — La lingua in cui il modello presenta domande, spiegazioni e proposte
al revisore durante una sessione. Viene fissata alla creazione della sessione e rimane invariata
durante una ripresa, anche se la UI locale corrente cambia.
**Administrative Page Family** — L'insieme delle cinque Administration Page che condividono shell,
navigazione, tipografia e regole responsive, pur mantenendo contenuti e operazioni specifici:
Workspace, Evidence, Memory, Database e Pi.
## Installazione
**Manual standalone installation** — Una copia di ThothII predisposta per l'uso autonomo da una
persona che possiede il computer, con una Full Thoth Shell e servizi applicativi locali. La
procedura non implica che DWH o provider LLM siano locali o disponibili offline.
**Installation bootstrap** — L'insieme delle attività iniziali che rende disponibile una
installazione manuale: verifica dell'host, generazione della configurazione, predisposizione
delle credenziali protette e avvio dei servizi. Non è un installer dell'applicazione.
**Installation preparation** — La predisposizione dei documenti che descrivono i workspace
e i parametri dell'installazione, prima di applicarli. Permette all'operatore di raccogliere
e correggere le informazioni senza avviare l'applicazione.
**Installation validation** — La verifica ripetibile della completezza e coerenza dei
documenti e delle precondizioni di un'installazione. Distingue ciò che è stato verificato
da ciò che richiede un'applicazione già avviata.
**Installation execution** — L'applicazione dei documenti verificati per predisporre e
avviare ThothII. Non raccoglie nuovi parametri dall'operatore durante l'esecuzione.
**Platform acceptance** — La verifica che una Manual standalone installation possa essere
predisposta e avviata su una specifica combinazione di sistema operativo, architettura e runtime,
distinta dalla verifica funzionale del collegamento a DWH e provider LLM.
+84 -3
View File
@@ -160,6 +160,15 @@ users can scan structure without adding nested containers.
## Colors
The full-mode application header matches Omics Portal's `--gsd-red-primary`
(`#CB333B`) in both themes. Its complete wordmark, including `II`, and controls
use a near-white foreground. This header is absent in embedded mode. The sidebar
and welcome wordmarks retain their red suffix. Context editing places workspace,
model and Done in one desktop row, stacking on narrow containers. Session-scope
tabs retain their selected fill and accessible keyboard state with a uniform one-pixel
border on every side, gray when inactive and red when active. Their padding is 11px
horizontal and 3px vertical, with a 38px minimum height and wrapping labels.
The palette combines warm porcelain surfaces, warm graphite text, and an instrument red used only
for action, focus, and important state. OKLCH values in the frontmatter are normative because the
frontend uses OKLCH tokens directly.
@@ -299,6 +308,27 @@ default, hover, focus, active, disabled, loading, and error behavior where those
### Navigation
- **Workspace readiness:** the Workspace navigation button carries an 8px dot to
the right of its label. Green means a selected workspace with confirmed ready
preprocessing and no query error; all other states are red. The button's
tooltip and accessible description retain the translated exact state. Do not
add a separate readiness text row or change the backend readiness gate.
- **Session groups:** one accessible single-open accordion contains Active sessions
and Archive, both initially closed. Below the scope tabs, show only their
adjacent section headers, without a redundant Sessions heading. Selection and
bulk-delete controls belong inside each panel and only appear for nonempty
lists. Select all affects that list only, preserves the other list's selection,
and exposes a mixed state for partial selection. Preserve the existing archived
flag as the grouping rule, independent of whether a Pi process is running.
Opening a section closes the other; either can be collapsed, including both.
Empty lists show only the translated "No sessions yet." message.
The open section uses the rail's remaining height; its list scrolls internally
with a cap of `min(18rem, 35dvh)`, while its trigger remains outside that scroll
area. The mobile navigation dialog supplies a bounded viewport-height container.
Keyboard users can focus and scroll each labelled panel.
- **Session entry:** one Session button returns to the current unfinished session,
including provisional creation, without resetting or reconnecting it. Otherwise
it prepares a new question using the normal readiness and unsaved-work guards.
- **Style:** compact session rows use `8px` corners and restrained vertical padding.
- **Default / Hover / Active:** porcelain at rest, Sunken Surface on hover, and a muted Navigation
Active red with a defined border when current. Exactly one top-level navigation control is current.
@@ -311,13 +341,31 @@ default, hover, focus, active, disabled, loading, and error behavior where those
width; a Navigation button opens the shared accessible dialog. Selecting another archive
page or pressing Escape closes it. Desktop retains the right session sidebar and its My sessions /
All sessions tabs. Core retains question/answer, eight phases, reviewer gates and the left log.
The portal owns the red header and left sidebar; ThothII must not duplicate them. Size to the
In embedded mode the portal owns the red header and left sidebar; ThothII must not duplicate them. Size to the
actual application container. Narrow session document panels may use the available width.
### Session review and confirmations
Session dialogs use the visible application area, including the portal's header
and side rail. Artifact and schema-column review can grow to 80rem wide and the
available height; short confirmations use up to 40rem and at least 18rem when
space permits. Keep a 24px outer margin on desktop and 8px on small or short
screens. Long review content scrolls internally; on very short screens the
whole dialog can also scroll so every action remains reachable.
Session forms and review gates repeat their existing primary confirmation above
and below the content, sharing selection, validation, pending state and response
handlers. Alternate-response inputs follow the same rule. Reserved navigation
controls remain below the review. Stop/delete initially focus Cancel; rename
initially focuses the name field. Administration dialogs and forms retain their
existing layout and actions.
### Tabs
- **Shape:** compact label tabs sit on a shared baseline with rounded top corners and a two-pixel
lower edge. Inactive labels retain a complete Quiet Border and Porcelain Card surface, so every
lower edge, except session-scope tabs which use a uniform one-pixel border, rounded
corners and a 4px gap without a shared border or negative bottom margin.
Inactive labels retain a Quiet Border and Porcelain Card surface, so every
label reads as a tab before interaction; hover feedback reinforces clickability.
- **Current:** the selected tab uses the muted Navigation Active red for its fill, text, and defined border.
It must expose `aria-selected`, participate in a labelled `tablist`/`tabpanel`, and be the only
@@ -337,6 +385,38 @@ default, hover, focus, active, disabled, loading, and error behavior where those
### Curated Evidence Documents
Memory and Evidence share the `thot-knowledge-reader` reading contract. Use locally
bundled Manrope with normal tracking for prose and labels, and these fixed roles:
- Card title: 24px, weight 600, line-height 1.3 (`thot-knowledge-title`).
- Field/section heading, including Scope and Provenance: 20px, weight 600,
line-height 1.4, 8px clearance below (`thot-knowledge-heading`).
- All narrative text, including scope, lists and provenance: 16px, weight 400,
line-height 1.65. Do not apply compact UI text sizes to these fields.
- Authored Markdown subheadings inside a field: 16px, weight 600, line-height 1.5,
24px above/8px below. They remain subordinate to the enclosing field heading;
their semantic heading levels and original content are preserved.
- Technical metadata labels/values: 14px/1.5, with weight 600 for labels.
Only code, paths and machine identifiers use the technical monospace family at
14px/1.65, identical for inline and fenced code (never compound `em` shrinkage).
Separate reading sections by 24px; keep the first Markdown block flush with its
field heading's 8px bottom gap. The same typography applies in light/dark and at
all responsive widths. Controls and archive indexes retain their compact UI roles.
Memory and Evidence detail readers use the entire available content width, without
the ordinary 72–75ch prose cap. This is the owner's explicit reading-layout choice.
Long unstructured paragraphs are split for display at existing sentence/semicolon
boundaries outside inline code and links; authored Markdown structure and stored
content are unchanged. Paragraph spacing is 1.25em. Scope and provenance share the
available width; provenance excerpts render Markdown rather than literal markers.
Copy actions use the two-overlapping-sheets icon, an accessible name/tooltip and
live success/failure feedback instead of a visible Copy label.
Memory has four explicitly FAKE formatting examples, one per family, in a separate
expandable section. They reuse the real detail reader but never enter persistence,
indexing, link search or model recall, and expose no edit/delete/save actions.
Curated evidence follows a fixed reading order: title, compact type and purpose summary, scope,
typed content, supporting excerpts, review items, then technical provenance. Curated v4 files
use short, visible YAML frontmatter for identity and classification. The Markdown title and
@@ -362,7 +442,8 @@ paths with copy controls, never browser file links to container-only locations.
- **Do** preserve information density with headings, rhythm, and progressive disclosure.
- **Do** keep keyboard focus explicit and pair color with text, shape, icon, or position.
- **Do** respect `prefers-reduced-motion` while preserving immediate non-kinetic feedback.
- **Do** use English for interface chrome and the workspace language for persisted document content.
- **Do** use the selected interface language (English by default) for chrome and preserve the
workspace language for persisted domain content. Session interaction language remains pinned.
- **Do** render curated metadata and scope as Markdown prose or lists, never as a frontmatter table.
- **Do** break long curated rules into paragraphs, labelled subsections, and lists at existing
punctuation boundaries while preserving the exact canonical text for machines.
+109 -536
View File
@@ -1,536 +1,109 @@
# ThothII — Project State
Last updated: 2026-09-12.
This file is the short operational snapshot. Stable commands and the architecture mental model
live in `AGENTS.md`; current design and runtime contracts live under `docs/architecture/`,
`docs/contracts/`, `docs/adr/`, and `docs/evidence.md`. Superseded plans and reports are
available from Git history rather than duplicated in the working tree.
The guarded server migration from a legacy checkout to the schema-v2 installation, Gitea source,
Authentik, internal catalog/embedding services, and the PSD workspace repository is documented in
`docs/operations/server-upgrade-gitea-workspace-v2.md`. Treat its operator gates and rollback
requirements as mandatory; do not replace the running server stack in place.
## Current product shape
### Isolated visual review, awaiting owner acceptance
The UI revision is isolated in `/Users/mp/projects/ThothII-visual-review`, branch
`codex/ui-visual-review`, based on `2d1b714e`. The original checkout and its prototypes
remain intact. No merge to `main` has been performed.
Local Docker at `http://127.0.0.1:8080/` now runs the frontend image
`thothii-frontend:visual-review-20260912`. Only frontend was recreated; Core and all
data services/volumes/configurations are unchanged. The prior look is preserved as
`thothii-frontend:before-visual-review-20260912`. All five running services are healthy.
The revision uses bundled Manrope throughout, shared type roles, restrained semantic
colors, clearer Admin copy and a readable compact session-document panel. Core and
session behavior remain covered by the existing regression suite. Verification:
679 frontend tests, 7 Playwright visual/interaction scenarios and a production build.
See `docs/reports/2026-09-12-ui-visual-review-delivery.md` for scope, limits and the
exact local rollback command. Visual approval is required before adoption on `main`.
Gitea #28–#31 follow-up is implemented in this checkout: Workspace's four tabs,
Database list-first entry without the preparation footer, full-height Pi instructions
with installation-host OS selection, and short Admin navigation labels. Core and
session behavior are retained. At the owner's request, these changes were rebuilt into
local Docker on 2026-09-12 at 18:31 UTC. Core/frontend are healthy and the UI at
`http://127.0.0.1:8080` serves the updated bundle. The host projection now supplies
`THT_HOST_PLATFORM=darwin`, so Pi selects macOS rather than the container's Linux OS.
Persistent dependency containers and volumes were unchanged. Rollback images are tagged
`thothii-core:before-admin-28-31-20260912` and
`thothii-frontend:before-admin-28-31-20260912`. See
`docs/reports/2026-09-12-admin-issues-28-31.md` for verification and deployment details.
The latest context-shelf A and five Administration pages are implemented locally.
At the owner's request, local Docker project `thothii-18998cca7b0a` was rebuilt and
its core/frontend recreated on 2026-09-12. The real UI at `http://127.0.0.1:8080`
serves shelf A; both services and their existing dependencies are healthy.
Runtime model projections were regenerated as schema v2 with the single
`zai/glm-5.3` interaction default. Persistent services/volumes were not recreated.
The local launcher `/private/tmp/thothii-memory-preview.sh` now adds
`/private/tmp/thothii-context-a.compose.yaml` last, building from this checkout
instead of the earlier Memory worktree. Previous images are retained under
`thothii-core:before-context-a-20260912` and
`thothii-frontend:before-context-a-20260912`. Remote server deployment remains pending.
Core/session behavior is retained; one independently remembered workspace/model
pair controls both Core and Admin. Unsaved Admin changes block navigation and
context changes; operation activity locks the selectors. See
`docs/reports/2026-09-12-context-shelf-a-implementation.md` for verification and the
remaining server/Omics integration gate. Prototype alternatives remain untouched.
ThothII is a human-in-the-loop datamart builder with three independently built layers:
```text
frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness → DWH (read-only)
```
The harness owns the deterministic eight-phase NL→SQL workflow and all session persistence.
The backend remains a process/RPC/SSE bridge for sessions and now also owns an isolated PostgreSQL
metadata catalog for administrative database configuration. The frontend renders the review gates
and keeps the live transcript in memory. See
`docs/architecture/components.md` for the detailed component and data-flow map.
## Memory M1–M3, Evidence E1–E3 and joint workflow repair X1 implemented
Browser authorization now derives permissions from validated session roles using the current
catalog. This fixes Memory/Evidence navigation remaining disabled for administrators whose
remembered login predates those permissions; refreshing the page loads the updated permissions.
The owner requested two independent administration projects: Memory management and Evidence
management. Their navigation entries must sit immediately after Database management, as peers;
neither page belongs to Database management. Both require complete browsing, filtering, and CRUD
without an active core session. Shared requirements and the two project briefs are linked from
`docs/plans/2026-09-08-memory-evidence-administration.md`. Memory administration is implemented;
Evidence administration is implemented through external file editing and explicit consolidation.
Evidence editing requires an explicit evolution of the current authoring/publication contract.
The M1 specification is at `docs/plans/2026-09-08-memory-m1-spec.md` and published as
[M1 — Archivio autorevole e amministrazione delle Memory Card](https://git.tylconsulting.it/mptyl/ThothII/issues/27)
with the `ready-for-agent` and `enhancement` labels.
It covers the authoritative store, administrative CRUD, projection recovery, and current
Memory/exemplar producer integration. Its accepted test boundaries are the public harness
service, Fastify APIs, and the AppShell page, joined by a focused real-stack browser path.
The owner confirmed those boundaries and authorized publication on 2026-09-08.
M1 is implemented locally: PostgreSQL authority in `thoth_memory`, versioned installation
migrations, admin CRUD for all four families, structured dependencies and links, explicit
projection recovery, verified recall and current workflow producers. The page requires no
active session or DWH binding. The existing `catalog-migrate` preparation service now runs
Memory migrations as well. Existing installations need that preparation before using M1;
the initial implementation did not deploy or migrate the owner's stacks.
The integrated browser check passed with real authentication, Fastify, ThtRunner, harness,
PostgreSQL and Qdrant; embeddings were deterministic and unrelated Pi/session activity used
test fixtures. See `docs/plans/2026-09-08-memory-m1-validation.md` for results and commands.
M2 is implemented locally: Memory dense/BM25 fusion, physical and business scope filters,
bounded outgoing-link expansion and joint ranking, all resolved against current PostgreSQL
authority. Migration `002_hybrid_projection.sql` makes old dense projections pending until
explicit retry/rebuild; Reference remains separate. The real retrieval check uses the configured
`qwen3-embedding:0.6b` model, a separate Ollama process with a read-only model-volume mount,
and isolated PostgreSQL/Qdrant resources. See `docs/plans/2026-09-09-memory-m2-validation.md`.
M3 is implemented: editable F8 summary grounded in effective approved decisions, explicit
updates with concurrent-edit protection, selected-card/link transactions and durable review
receipts. Finalization no longer saves exemplars implicitly. SQL rules and explained errors
are consulted in the existing F4/F6/F7 gates. Successful Catalog physical synchronization
performs dependency cleanup, preserving the original removals for recovery across restarts.
Migration `003_review_receipts.sql` is required. Validation includes a real GLM 5.3 generation
case against synthetic PostgreSQL data; see `docs/plans/2026-09-09-memory-m3-validation.md`.
Evidence administration and the joint X1 conflict-repair increment are now implemented.
The same local preview was subsequently rebuilt with M3 and migration 003 applied.
Core and frontend now include the final Memory review and physical dependency cleanup.
On 2026-09-09, at the owner's request, the local PSD Docker installation
`thothii-18998cca7b0a` was updated from this worktree. Core/frontend images were rebuilt,
Catalog and Memory migrations completed, and all five services became healthy. The UI is at
`http://127.0.0.1:8080`, using the existing local authentication and persistent volumes.
The `psd-clinical` authoritative Memory archive is initially empty (no legacy import).
The existing installation configuration remains in `/Users/mp/projects/ThothII/deploy/psd/`;
`/private/tmp/thothii-memory-preview.sh` invokes its Compose files with a final build-context
and migration-command override from this worktree. A future build from the main checkout
will use that checkout's code, so retain the worktree override until the changes are integrated.
Memory revision history and compatibility with existing development sessions are not requirements.
The agreed Memory scope includes reusable domain clarifications, SQL construction rules, solved
questions, and explained, approved mistakes to avoid. This extends the current runtime's
`concept_clarified`-only reusable Memory contract. The owner also approved an editable final summary
for proposed additions/updates and Memory consumption in the relevant existing review gates.
Further agreed behavior includes persistent corrections for Memory/Evidence conflicts through
explicit choices, deletion of Memory with invalid dependencies after successful physical schema
synchronization, and bounded functional/regression tests instead of a general quality benchmark.
In-house Memory evolution is approved, including hybrid Qdrant retrieval and explicit card links
traversed in core, without a dedicated graph database. Both capabilities belong to the current
scope. PostgreSQL is the agreed authority for Memory Cards, links, and schema dependencies;
Qdrant is a rebuildable index. Links are reviewed with cards and editable in Administration;
deleting a card removes its incident links while preserving the other cards. The accepted,
implemented architecture is recorded in
`docs/adr/0018-use-postgres-for-memory-and-qdrant-for-retrieval.md`.
For Evidence administration, Q12 rejects a separate editorial publication workflow.
The final R0 direction uses external editors and a manual consolidation command that
validates structure, reports required corrections, updates derived metadata, and
activates the local Evidence index. The operator then checks the diff and runs Git
commit/push manually. No watcher, automated Git, or web editor is required.
The owner's later simplification
instruction removes the requirement to support administrative edits while core work
is in progress. Deliberate corrections made by the workflow itself remain supported.
The context specialist writes Evidence drafts independently of the installation and
without PostgreSQL access; the system refines them and stores them locally for use
and maintenance. The revised Q11 recommendation separates external drafts from a
durable local canonical file archive, removing automatic commit/push from CRUD.
The owner has now accepted local files and requires discoverable, editable Markdown
for nontechnical domain specialists, with no JSONL management surface. Editors on
Mac/PC or vim/nano on the server are the chosen R0 editing surface. Evidence
management keeps browsing/filtering/detail and clearly identifies the persistent
working tree, each Markdown file's actual host path, and the manual commands.
E1 implements editable Curated Evidence v4, deterministic legacy conversion, persistent
local files and a consolidation API with immutable candidates, manual provenance,
deletion records and recoverable activation. E2 connects its installed command, runtime source
selection and administration page. The operator completes Git steps manually.
All 35 PSD units were converted on an isolated copy with identical IDs and typed content.
Tests exercise visible edits through normalization, indexing and recall with real Qdrant;
see `docs/plans/2026-09-09-evidence-e1-validation.md` and
`docs/contracts/curated-evidence-v4.md`. E2 is now running on the local Docker preview:
all 35 PSD units were converted and indexed through the installed command. The editable
host archive is `/Users/mp/projects/ThothII/deploy/psd/evidence-registry/repo/psd-clinical/evidence`.
The original registry-volume checkout and original author repository were retained.
The installation descriptor now includes `workspace-bindings.yaml` and `evidence-host.yaml`
so native maintenance uses the same volumes and checkout as the preview. Descriptor backup:
`/private/tmp/thothii-installation-before-e2.yaml`; previous native binary: `/private/tmp/tht-before-e2`.
The launcher remains `bash /private/tmp/thothii-memory-preview.sh`; its worktree build override
is still required until integration. Runtime lease filenames now include rendered bytes so
upgrading the Evidence renderer does not collide with old immutable configs; Catalog
input fingerprints and readiness are unchanged. See `docs/plans/2026-09-09-evidence-e2-validation.md`.
E3 adds explicit source acquisition/refinement and durable comparisons in Evidence management.
Local drafts go in `evidence/incoming/`; original local documents and configured HTTP/S3
sources are reacquired only by **Import or refresh sources** or installed
`tht workspace evidence refresh --workspace <id>`. Keep/replace decisions, including the
native `workspace evidence decide` command, activate through the existing archive/index
path and have durable retry state. Acquired source versions and remote provenance stay
local; old documentary lineage can coexist with current manual declarations.
The installed preview's 35 PSD sources were verified unchanged with no new proposals.
The previous native CLI is backed up at `/private/tmp/tht-before-e3`.
See `docs/plans/2026-09-09-evidence-e3-validation.md` for tests and validation limits.
X1 adds `reviewer_archive_repair`: closed alternatives with complete before/after content,
explicit rejection/reformulation, admin-only application, durable session receipts and
retry of saved but inactive corrections. The coordinator uses the canonical Memory and
Evidence services; phase approval remains separate. Migration `004_archive_repairs.sql`
is required. See `docs/contracts/archive-repair.md` for authorization and recovery limits.
X1 validation is recorded in `docs/plans/2026-09-09-archive-repair-x1-validation.md`:
both archives were corrected and retrieved through the actual CLI with real PostgreSQL
and Qdrant; desktop/mobile widget behavior and permission failures were verified separately.
The local preview images include X1 and migration 004 is applied. Follow-up technical
acceptance passed with configured GLM 5.3 generating closed conflict alternatives and
the chosen Memory correction persisted and retrieved. An authenticated browser test
also verified both administration pages, navigation order, Evidence filtering, workspace
404s, desktop/mobile layout, and Evidence survival after Memory deletion. All approved
technical increments/checks are complete. A real PSD domain-conflict session remains
the reviewer's semantic acceptance check; automated cases did not modify PSD knowledge.
The joint browser inspection also fixed mobile archive navigation: below 768px,
Memory and Evidence keep the full content width and open navigation in the shared
accessible dialog. Desktop retains its sidebar; the local frontend image includes this fix.
The owner accepted the remaining simplifications and requested explicit clarification
of the core format change and the simple terminal-based Git check. E1 must adapt
the Evidence parser, renderer, authoring, validation, and normalization; convert
existing files and reindex; and verify that visible edits reach core consumption.
Preserve the internal typed model where possible. This precedes the administrative
page and is not merely a presentation change. Human inspection uses normal Git
status and optional line-level diff commands; no custom diff viewer or mandatory
double review is required.
The suggestion of making PostgreSQL the Evidence authority was withdrawn after this
clarification; it was never implemented or accepted as a replacement for Q11.
Q13 keeps manual corrections active when updated sources contradict them, until an
administrator resolves the comparison; deleted Evidence must not be regenerated
automatically. Q14 allows direct manual creation and records a manual declaration
as the current source, preserving any original document as distinct provenance.
Q15 refreshes external sources only on explicit administrator request; normal saves
and lookups do not reacquire them. These decisions, now implemented through E1–E3, are recorded in
`docs/adr/0019-author-evidence-in-app-with-automatic-activation.md`.
The decision-by-decision review is recorded in
`docs/plans/2026-09-08-memory-evidence-simplification-review.md`; it distinguishes the
new constraints from the revised technical recommendations. It also specifies a
sequential save with minimal durable retry state and direct Memory cleanup after
successful schema synchronization, without new queues or event infrastructure.
The delivery order remains Memory, Evidence, and persistent conflict repair between
both modules. Local curated Evidence must survive preprocessing Clear and be backed
up as primary data; the Qdrant projection remains rebuildable.
No runtime gate change or administrative page is implemented yet.
## Evidence restructuring — accepted
The evidence restructuring and PSD migration completed real acceptance on 2026-08-25.
- The curated PSD revision contains 35 approved Evidence units and 60 review items.
- The PSD authoring repository publishes all 35 units using Curated unit schema v3. Its table-free
presentation uses hidden canonical metadata, wrapping Markdown scope lists, list-based enum
values, and collapsed technical provenance. Long domain rules now have a deterministic
human-readable presentation while retaining their exact canonical text for vector ingestion.
`tht evidence migrate <workspace-root>` performs the deterministic v1/v2 upgrade and older-v3
presentation rewrite without model calls. The structured-rule PSD rewrite is currently local and
pending commit/publication.
- The accepted snapshot is
`psd-clinical-675990d90eae51da6f2bd51b1ae2609f245772ef-snapshot`.
- The active generation is `gen:f968b3bd7a553dbfef3cf47093698f2bc7f95f11`.
- Retrieval acceptance reached 20/20 Hit@10.
- A real session, `20301df7-cad7-403d-a4c1-9f35c9d07b66`, completed F1–F8 with five
receipts, three CTEs, and a final result of 78 patients.
- The durable acceptance record is
`docs/testing/evidence/evidence-restructuring-psd-acceptance-2026-08-25.md`.
The canonical authoring, validation, publication, materialization, and preprocessing flow is
documented in `docs/evidence.md`. The governing contracts are
`docs/contracts/workspace-evidence-v3.md` and
`docs/contracts/workspace-preprocessing-cli.md`.
The incremental server procedure for the `260906-preprocessing-complete` release is
`docs/operations/server-handoff-260906-preprocessing-complete.md`.
## Workspace preprocessing and configuration
The native host CLI `tht` is the operator surface. Workspace preprocessing runs through:
```sh
tht --installation /absolute/path/thothii-installation.yaml \
workspace preprocess run --workspace <workspace-id>
```
This complete one-shot command uses the profile-gated `workspace-maintenance` service. Partial DWH,
schema, Evidence, and vector mutation commands are retired.
The right Administration sidebar invokes that same operation for the selected workspace. It shows
only current readiness or the latest bounded failure diagnostic; there is no preprocessing history.
Known non-ready state disables **New session**, while backend admission remains authoritative.
The same control exposes an inline-confirmed **Clear** action to remove replaceable reference
vectors, LSH, corpus, and checkpoints while preserving the separate Memory collection. The host CLI
equivalent is `workspace preprocess clear`.
Each workspace now uses `<workspace>-reference` for Schema, relationships, and Evidence and
`<workspace>-memory` for `memory` and `solved_question`. Clear and preprocessing own only the former.
LSH ownership additionally binds the Catalog database ID and Metadata Content Revision, so derived
values cannot be reused across database identities or Catalog revisions.
Workspace descriptors use schema v4 and contain only workspace identity and optional Evidence.
PostgreSQL Metadata Catalog owns database identity, binding, schema, descriptions, sensitivity, and
relationships; model, provider, embedding, and vector-store configuration is installation-owned. For
PSD, workspace content and runtime roots point to the
separate uncommitted repository `/Users/mp/projects/tht-workspace-psd`. Secrets remain outside
Git and are supplied only through installation-local protected files.
## Installation Model Catalog
`thothii-installation.yaml` schema version 2 is the only operator-authored source for session,
metadata-generation, and embedding models. The host `tht` lifecycle validates `modelCatalog` and
regenerates the backend catalog, Pi `models.json`/`settings.json`, and Compose override under the
installation-local `generated/` directory. Those projections are replaceable runtime adapters:
they are not edited, backed up, or treated as configuration.
Core and Administration now share one canonical `modelCatalog.defaults.interaction` per installation,
independent of workspace. Runtime catalog schema v2 contains only `defaultInteraction`; apply the host,
backend, and regenerated projections together. Equal legacy defaults normalize on read; conflicting
ones require an explicit operator choice. With Admin AI configured, selectable models are the
intersection of the Pi and LiteLLM adapters; Core-only installations remain supported without Admin AI.
The existing Core and Database controls share the operational selection. Explicit model choices are
remembered per authenticated user/application mount in this browser, not in installation settings.
Resume uses that selection/default while preserving the historical workspace/revision and manifest.
Every model must be manually exercised in both Core and Admin as documented in
`docs/general/pi-configuration.md`; validation is not a live model certification.
These changes and shelf A are now deployed to local Docker; remote deployment remains pending.
PSD DeepSeek Pro/Flash now share canonical `deepseek/...` identities across native Pi and LiteLLM,
using the existing `DEEPSEEK_API_KEY` bundle entry. Its value was confirmed identical to the working
Pi key without exposing it; no secret files were changed. The duplicate `deepseek-metadata` descriptor
is removed locally and the tracked example uses the shared provider. `secret_env` overrides legacy
Pi auth only inside temporary runtime snapshots and fails closed if the bundle key is missing.
The original auth store/history and `zai/glm-5.3` default are preserved. Local Docker projections
and core/frontend were updated together on 2026-09-12. Regenerate projections with
the matching release for the separate server deployment.
Provider authentication declares
one explicit mode (`secret_env`, `pi_auth`, or `none`); `secret_env` names a protected bundle key.
The backend settings store now owns only the selected workspace and thinking level. Existing v1
installations use the explicit catalog migration command; schema-v3 workspace descriptors are
converted deterministically in their curator-owned repository before commit. Strict runtime loading
does not silently infer or merge legacy sources. ADR 0013 and
`docs/plans/2026-09-02-installation-model-catalog.md` record the decision and implementation.
## Database management
The database, table, and authoritative physical-schema catalog slices are implemented. Database
Management now opens the Fleet Ledger presentation by default inside `AppShell`, lists every YAML
workspace, creates at most one PostgreSQL database configuration per workspace, edits direct
PostgreSQL, REST API, or SSH-tunnel installation bindings, replaces write-only encrypted secrets,
and tests supported connector bindings. The surface keeps one responsive AG Grid visible at a time:
databases lead to tables, tables lead to columns, and relationships are a sibling database view.
Parent navigation remains explicit through the breadcrumb and emphasized back control.
Selection-scoped operations use one action selector plus an explicit **Run** control; ineligible
actions remain visible with their disabled reason, while row-scoped actions stay in the pinned final
column. The KPI strip reads installation-wide or selected-database aggregates from
`GET /catalog/metrics`. Database configuration, metadata editors, synchronization history,
description history, and sensitive-field review/history use the production APIs in right-side
drawers rather than prototype fixtures; closing a history drawer does not stop its background run.
Sensitive-field review is now driven by the versioned local `sensitivity-v4` policy, not by a
catalog model. The backend reads selected source tables through read-only, database-specific
adapters and makes every `sensitive | non_sensitive` draft decision in the TypeScript
`SensitivityClassifier`. A single validated match protects the column. Tables up to 1,000 rows are
fully scanned; larger tables use breadth-first 300, 1,000, and text-only 3,000-value targets, with a
five-second limit per source query and no global request deadline. Source failures fail the run
instead of yielding `unknown`; coverage remains visible separately from the proposal. Draft
assessments remain transient until an administrator explicitly saves them. Optional GLiNER2
evidence is CPU-only, offline, opt-in, and never replaces the deterministic decision point; see
`docs/operations/sensitivity-analysis.md`. The earlier v1 PSD shadow comparison kept NER disabled by
default; see `docs/reports/2026-09-02-psd-sensitivity-shadow.md`. The v2 comparison completed all
2,275 columns: CPU NER added 18 sensitive proposals and increased warm runtime from 50.1 to 61.3
seconds; see `docs/reports/2026-09-03-psd-progressive-sensitivity-shadow.md`.
Version 4 excludes declared `bigint` primary-key columns and conventionally named `pk bigint`
columns before source inspection, reporting both as non-informative structural identifiers while
distinguishing declared constraints from inferred roles.
Physical membership, source
comments, column types/default/nullability/PK positions, and constraint-level ordered FK pairs are
projections of the external schema. They cannot be created, renamed, or structurally edited by
hand, but administrators can explicitly clear catalog tables, columns, or relationships without
touching the source database, binding, configuration, or secrets. Table deletion cascades through
columns and relationships; table-scoped relationship cleanup includes incoming and outgoing
relationships. Curated and generated descriptions are editable; generated descriptions start null
and Database Management can generate or consolidate them for selected tables, selected columns,
all targets, or only targets whose Generated Description is missing.
Relationship Management is now reachable directly from each configured Fleet database. One
Relationship Map shows read-only Physical Relationships together with Generated and Manual Logical
Relationships, with Active, Excluded, and All filters. Administrators can add a single-column
relationship, run deterministic name/PK/type inference, exclude or restore a logical relationship,
or delete it permanently. Exclusion retains a tombstone that a rebuild cannot reactivate; permanent
deletion allows a later rebuild to infer the same endpoints again. Inference uses no LLM, embedding,
or source values. It supports normalized table-qualified names, unique non-generic PK names,
composite-PK source columns, and the `*time_key -> dim_time.<single PK>` warehouse convention while
ignoring bare generic names. Explicit table/column metadata cleanup remains a destructive boundary: it removes
the attached logical relationships and exclusions and requires a full schema synchronization before
inference or runtime publication can continue.
The previous Database Management renderer remains a temporary comparison fallback for development
and staging only: `?db-ui=legacy` is honored in Vite development or when
`VITE_DB_MANAGEMENT_LEGACY=true`; it is not a production presentation. The standalone Fleet Ledger
prototype on port `5173` also remains temporary until owner acceptance of the integrated surface,
after which both migration aids can be removed.
Schema refresh is one durable asynchronous engine with database-table, selected-table-column,
relationship, and full-database actions. Database-level menus expose only the table, relationship,
and full scopes; selecting tables exposes column synchronization plus manual column and relationship cleanup for that subset. Database selections
also expose manual table and relationship cleanup. Cleanup selections are atomic and share the
one-active-operation-per-database exclusion with synchronization. Runs have leases and
restart recovery, atomic apply, destructive-diff confirmation with re-scan, cancellation before
apply, retained history, and a live SSE log with polling fallback. Null metadata renders blank
rather than as a placeholder.
Direct PostgreSQL and strict known-host-verified OpenSSH use `pg_catalog`. REST bindings use the
typed full-snapshot `POST /rpc/schema_snapshot` contract when available. Servers such as the
current PSD endpoint that exposes only `POST /rpc/run_query` use one catalog-owned read-only query
to return the exact same strict v1 snapshot in a single round trip. Both paths remain fail-closed:
an absent capability, query error, partial result, or invalid snapshot applies no catalog changes.
SSH is not yet enabled for NL→SQL session runtime.
The catalog runs in the internal `catalog-db` PostgreSQL service. Kysely migrations are an explicit
one-shot `catalog-migrate` operation; `scripts/run-stack.sh` runs it before local startup. Runtime
sessions consume an immutable Catalog JSON snapshot tied to the runtime-config lease. It contains
the tables, columns, effective descriptions, sensitivity flags, and active relationships used by
the harness; PostgreSQL is the exclusive runtime authority for database metadata. Authored
workspace YAML remains limited to workspace identity and optional Evidence configuration. The
accepted design is recorded in ADR 0016 and the contracts under `docs/contracts/`.
Semantic aliases, value descriptions, synonyms, and concepts remain deferred to their dedicated
slices.
AI Description Generation uses the catalog's human-owned Sensitive Data Flag. The flag defaults to
`false`, including for newly synchronized columns. An administrator may request a local sensitivity
analysis for one selected database, selected tables, or selected columns. One deterministic
TypeScript classifier combines metadata, bounded source-content rules, and optional CPU-only NER;
no generative model decides the result. Its `sensitive` or `non_sensitive` assessments remain an
unsaved draft until the human reviews and saves any chosen flag changes, including a downgrade to
non-sensitive. Coverage is reported separately; interrupted history may count unprocessed columns.
Each started analysis records a separate Sensitivity Analysis Run with aggregate counters and safe
ordered events. The progress drawer opens before the synchronous request completes, polls the run,
and displays sanitized source-scan and local-NER phase/batch activity while classification is in
progress. This operational history never stores per-column assessments, source values,
matched spans, prompts, or free-form diagnostics. Saving a sensitive decision persists a sanitized
Sensitivity Reason as column Catalog Metadata alongside the human-owned flag; clearing the flag
clears that reason. Reloading still discards an unsaved review draft.
For unprotected columns, up to five source rows and five representative non-null values may be sent
transiently to the configured model provider. Protected columns are omitted from source reads and
replaced in the prompt by deterministic plausible values derived only from their metadata. Existing
descriptions are not regenerated when a flag changes.
The accepted AI-description design is recorded in
`docs/plans/2026-08-28-ai-catalog-description-generation.md`, with the formal specification in the
adjacent `-spec.md` document and Gitea issue #4. Gitea issues #5–#11 deliver the implementation.
The runtime deliberately keeps ThothAI's simple operating model: one installation-wide sequential
run owned by the backend, one short-lived Python/LiteLLM completion helper per request, and
persistence limited to the run, its safe ordered text events, and each Generated Description as
soon as it succeeds. The helper performs at most one provider retry and never falls back to another
model. Stop terminates the current helper and retains prior results; three consecutive exhausted
technical batches fail the run. Startup marks stale queued/running work interrupted, and Unlock is
available only when no local start, worker, or helper is live. Runs remain inspectable through a
live SSE log with ordered polling fallback; there is no automatic resume or user-facing generation
CLI. ADRs 0009–0010 record the runtime and source-sampling decisions.
The Installation Model Catalog accepts the protected `DEEPSEEK_API_KEY` and `ZAI_API_KEY`
references for metadata-generation providers.
It also accepts a model with no secret reference only when its OpenAI-compatible endpoint is
explicit; this covers the VPN-only AritmoLab Qwen 3.6 server without creating a fake operator
credential. The Python client supplies only its fixed non-secret compatibility placeholder.
The AritmoLab entry also sets `disableThinking: true`, mapped to the endpoint's chat-template flag,
because its default reasoning prose would violate the worker's exact JSON response contract.
Logical relationship integration with core schema-linking is complete: session creation and resume
materialize the active physical/generated/manual map, retrieval-pack generation and Pi receive the
same runtime config, and snapshot validation fails closed on a declared missing, invalid, or orphaned
endpoint. Broader publication of other Catalog metadata to schema-linking remains a separate future
slice.
**Deferred follow-up — Sensitive Data Policy in schema-linking.** The policy is first delivered
and tested in catalog description generation. Its enforcement for core schema-linking remains
out of scope until the current tickets are closed and the owner has completed the acceptance test.
At that gate, resume the design: `tht` must receive a read-only projection of the current Sensitive
Data Flags and exclude values from columns marked sensitive from every LSH result before it is
given to Pi. Do not start this integration before the owner gives final approval after that test.
## Active deployment work and manual gates
### PSD server deployment program
The approved design and executable entry point are:
- `docs/plans/2026-08-20-psd-server-deployment-program-design.md`
- `docs/plans/2026-08-20-psd-server-deployment-program.md`
- `docs/plans/2026-08-20-psd-server-survey.md`
- `docs/plans/2026-08-20-psd-server-project-a-standalone.md`
- `docs/plans/2026-08-20-psd-server-project-b-authentik.md`
Last recorded state:
- survey: `SURVEY_NO_GO`;
- Project A: `BLOCKED_BY_SURVEY_AND_MUTATION_GATE`;
- Project B: `BLOCKED_BY_PROJECT_A_AND_PRE_B_GATE`.
The deployment is a clean replacement: legacy sessions, indexes, and application configuration
are not migration inputs. The existing stack remains intact until its documented mutation and
rollback gates are explicitly approved. Shared Omics/LocalLLM networks, ETL Evidence, DWH,
`dwh-auth`, Supabase, Authentik, Superset, and Aritmolab are outside cleanup scope.
Human acceptance guides and sanitized report templates live under `docs/testing/` and
`docs/testing/evidence/`. The remediation checklist is
`docs/operations/psd-server-survey-remediation-checklist.md`.
### Authentication
The local/OIDC authentication remediation passed its automated review on 2026-08-18. Release and
PSD mutation gates remain governed by:
- `docs/architecture/authentication.md`;
- `docs/plans/2026-08-18-thothii-authentication-acceptance-and-psd-deployment.md`;
- `docs/operations/psd-dwh-auth-rollout.md`;
- `docs/testing/authentication-manual-acceptance.md`.
Do not infer authorization for server, Nginx, Authentik, database, credential, or cutover changes
from an automated PASS.
## Verification status
- The P1.1 workspace-directory registry and P2–P6 preprocessing workstreams are implemented and
have automated coverage.
- Evidence restructuring has a real PSD acceptance PASS as recorded above.
- AI Description Generation has automated coverage across installation setup, model selection,
generation/consolidation scopes, bounded sampling, cancellation/recovery, history, SSE/polling,
and the LiteLLM helper boundary.
- L2 tests requiring real providers or remote databases remain opt-in.
- Server deployment, release, and owner-operated acceptance steps remain pending wherever the
referenced runbooks require explicit approval.
Run the layer-specific checks documented in `AGENTS.md`. For release-sensitive changes, also run
the repository contract scripts in `scripts/` and build the MkDocs site.
## Operational invariants
- `tht`'s `-c`/`--config` option follows the subcommand; it is not a global option.
- `--json` commands write pristine JSON to stdout.
- Persisted phase documents and the decision ledger are the source of session truth; chat is not.
- UI chrome is English; workspace document content retains the workspace language.
- The backend refuses resume for finalized or archived sessions.
- A resume must send `/riprendi-sessione <id>`; a new session must send `/nuova-domanda`.
- DWH access is read-only.
# Project state
Updated: 2026-09-28. This is a current snapshot, not a release diary. Stable commands
and invariants are in [AGENTS.md](AGENTS.md); prior snapshots remain in Git.
## Current contracts
- Document-first installation ticket #43 provides offline `tht workspace prepare`
and `tht workspace validate` through a native two-executable bundle. Build and
test instructions are in [the host CLI guide](tools/tht/README.md). Local validation
reuses runtime workspace/catalog parsers and explicitly defers runtime Evidence,
database binding and readiness checks. Ticket #44 adds `tht installation prepare`,
explicit `installation credentials`, and `installation validate --workspaces PATH`
for protected application documents, canonical model settings and schema-v1
database bootstrap inputs. These commands do not start services or import Catalog
bindings. Ticket #45 adds host `installation preflight` and `installation plan`
with release/image checks, canonical external diagnostics and private input seals;
see [the preflight reference](docs/install/installation-preflight.md).
The remainder of the installation tickets,
Docker Hub publication and example databases remain pending.
- React supports full/embedded rendering independently of local/OIDC/upstream auth,
with EN/IT UI and immutable session interaction language. See
[application shell](docs/architecture/application-shell.md) and
[localization](docs/operations/shell-and-localization.md).
- PostgreSQL Metadata Catalog owns database identity, binding, schema, descriptions,
sensitivity and relationships for all core consumers. Workspace schema v4 contains
identity and optional Evidence only. Installation schema v2 is the authored model
catalog source. See [overview](docs/architecture/overview.md) and
[model configuration](docs/general/pi-configuration.md).
- The harness owns workflow persistence; chat is not the durable session record.
Memory uses PostgreSQL authority and derived Qdrant dense/BM25 search. Editable
Evidence has local archive authority and manual consolidation. File save, search
activation and Git publication have distinct outcomes. See
[Memory](docs/gestione-memory.md), [Evidence](docs/contracts/curated-evidence-v4.md)
and [consolidated release evidence](docs/reports/knowledge-archives-release.md).
- Reference preprocessing must not clear Memory. Use the installation-scoped
`tht --installation /absolute/path/thothii-installation.yaml workspace preprocess run`
and its [contract](docs/contracts/workspace-preprocessing-cli.md).
- DWH sessions are read-only. SSH tunnels support database-management diagnostics
and metadata synchronization, not NL→SQL session creation; use direct or REST
transport for sessions.
## Installation and workspace boundaries
Fresh standalone installations follow the manual terminal procedures in
[Italian](docs/install/standalone-manual-it.md) or
[English](docs/install/standalone-manual-en.md), without an installer or launcher.
The [Compose reference](docs/operations/compose-reference.md) is for maintainers,
not another quick start. Catalog and Memory migrations are explicit.
Descriptors, authentication, provider credentials, certificates and runtime bindings
stay in protected installation-local paths. Do not copy secrets into examples or
workspace Git. Legacy runtime snapshots may use absolute paths and protected
`harness/.env`; do not silently relocate them.
PSD authoring is separate at `/Users/mp/projects/tht-workspace-psd`. Its GitHub
repository was copied to private Gitea
[workspace_psd](https://git.tylconsulting.it/mptyl/workspace_psd), preserving both
branches. It is a copy, not automatic synchronization. Running installations were
not repointed to a different workspace remote.
## Recorded deployments and server authority
Use the ordered [server handoff](docs/operations/server-codex-handoff.md) for
coordinated ThothII/Omics upgrades. Omics integration uses GitHub
`Dallavilla-Tiziano/omics_portal`, with no Gitea relay prerequisite. Omics uses
embedded/upstream identity, not another ThothII OIDC login. Read
[upstream authentication](docs/install/authentication-upstream.md) before changes.
Last recorded application deliveries (not a fresh runtime attestation):
- [Session dialogs](docs/reports/2026-09-14-session-dialogs-release.md):
`b1723c34-session-dialogs-20260914`, frontend-only.
- [Session layout/Memory fix](docs/reports/2026-09-14-session-layout-memory-fix.md):
`49333a2d-session-memory-fix`, core/frontend.
Keep those reports and rollback instructions while operator gates remain open.
A later deployment does not prove every earlier acceptance item passed.
## Remaining acceptance and design gates
- Fresh-machine Mac, Windows/WSL2 and Linux installation acceptance, including real
DWH/model endpoints, remains a separate operator exercise.
- Real IdP/portal login, logout, embedded interaction and PSD semantic acceptance
follow the [manual matrix](docs/testing/authentication-manual-acceptance.md) and
delivery reports; synthetic tests do not close them.
- [Security hardening](docs/plans/2026-09-08-security-hardening-prd.md) is a draft.
Revalidate SEC01–12 and obtain design approval before implementation or real
server/IdP/DWH mutation.
- Optional NER remains opt-in; labeled Italian quality, benchmark and licensing
acceptance are not implied by document cleanup.
- Semantic aliases, value descriptions, synonyms/concepts, dialect and multi-schema
extensions remain explicit design work. Current sensitivity delivery follows the
Catalog contract; additional policies require their own acceptance.
- Legacy database UI fallback (`?db-ui=legacy`, dev/staging) and prototype removal
remain subject to owner acceptance.
## Documentation maintenance
MkDocs publishes only 20 product/operator pages and five approved assets.
Architecture, contracts, ADRs, plans, research, tests and release evidence are
excluded from HTML and search. The repository itself is public: editorial exclusion
is not confidentiality.
The [cleanup record](docs/maintenance/2026-09-15-documentation-cleanup.md) records
retired sources and retained gates. Main contains source; Actions generates the
`pages` branch. The live site requires the separate explicit deployment described
in [public manual publication](docs/operations/public-docs-publication.md).
+38 -58
View File
@@ -4,72 +4,52 @@ ThothII is a human-reviewed NL-to-SQL workflow with a React frontend and a Fasti
core. The portable deployment runs two application services plus the installation-local metadata
catalog; DWH and LLM services remain external. Semantic services are bundled in Compose.
Authentication is configured through the single host CLI tht: see the [local authentication guide](docs/install/authentication-local.md),
[generic OIDC guide](docs/install/authentication-oidc.md), and [manual acceptance matrix](docs/testing/authentication-manual-acceptance.md).
The same frontend supports **full** (its own header) and **embedded** (inside a
portal). This choice is independent of authentication: the Mac uses full/local,
Omics uses embedded/upstream with its existing login, and a standalone server
can use full/OIDC. See [rendering architecture](docs/architecture/application-shell.md)
and [configuration, Omics delivery and deploy](docs/operations/shell-and-localization.md).
## Docker Compose: local startup
For the current server upgrade with Omics Portal, follow the ordered
[Codex server handoff](docs/operations/server-codex-handoff.md), including source
integration, embedded/upstream configuration, coordinated rollout and rollback.
Requirements: Docker Engine with Compose v2. The mandatory stack is `frontend`, `core`,
`catalog-db`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`. DWH and LLM remain external,
configurable endpoints—even when they are co-located with ThothII.
Local/OIDC authentication is configured through the host CLI `tht`; portal
authentication is established by the trusted server proxy. See the
[local guide](docs/install/authentication-local.md),
[OIDC guide](docs/install/authentication-oidc.md),
[upstream integration](docs/install/authentication-upstream.md), and
[manual acceptance matrix](docs/testing/authentication-manual-acceptance.md).
From a fresh clone, run these commands from the repository root:
For the clone-based manual standalone installation test on macOS, Windows, and Linux, use the
[Italian procedure](docs/install/standalone-manual-it.md) or the
[English procedure](docs/install/standalone-manual-en.md).
```sh
cp deploy/env/local.env.example deploy/env/local.env
# Copy docs/install/examples/thothii-installation.local.yaml to a protected operator path,
# replace its placeholders, chmod it 600, and set that exact THT_INSTALLATION_CONFIG_SOURCE.
# Edit deploy/env/local.env, including PI_AUTH_FILE, THT_SECRETS_FILE, and external endpoints.
./scripts/run-stack.sh
```
The [public manual](https://git.tylconsulting.it/thothii-docs/) covers the product,
installation, use and administration. Developer architecture, contracts, ADRs, tests,
plans and release records remain in this repository but are excluded from MkDocs
pages and search. This is an editorial boundary, not an access restriction on the
public repository. See the [documentation cleanup review](docs/maintenance/2026-09-15-documentation-cleanup.md)
for the executed consolidation and the inventory of historical sources retained in Git.
The launcher builds the core, starts `catalog-db`, runs the explicit one-shot Kysely migrations,
then runs the base+local stack in the foreground. Migrations never run implicitly in backend
startup. The core image contains its Pi runtime; no host `pi` executable is used. For a server
installation, build the image, start the catalog, and run the same migration service before the
application rollout:
## Docker Compose and installation
```sh
cp deploy/env/server.env.example deploy/env/server.env
# Prepare a mode-600 thothii-installation.yaml from the server example and set its exact
# path as THT_INSTALLATION_CONFIG_SOURCE. Edit all remaining storage/secret/endpoint paths.
sudo scripts/prepare-server-pi-state.sh /srv/thothii/pi-state 10001 10001
docker compose --env-file deploy/env/server.env \
-f compose.yaml -f deploy/compose.server.yaml \
-f deploy/compose.session-server.yaml.example build core
docker compose --env-file deploy/env/server.env \
-f compose.yaml -f deploy/compose.server.yaml \
-f deploy/compose.session-server.yaml.example up -d catalog-db
docker compose --env-file deploy/env/server.env \
-f compose.yaml -f deploy/compose.server.yaml \
-f deploy/compose.session-server.yaml.example run --rm catalog-migrate
docker compose --env-file deploy/env/server.env \
-f compose.yaml -f deploy/compose.server.yaml \
-f deploy/compose.session-server.yaml.example up --build -d
```
For a fresh installation, follow the guided terminal procedure in
[Italian](docs/install/standalone-manual-it.md) or
[English](docs/install/standalone-manual-en.md). The single
`tht setup --complete` command validates protected files, builds the images, runs
Catalog migration, starts the stack and imports the configured workspace repository.
There is no graphical installer or native launcher.
The initializer is required for an empty or restored server Pi-state bind. It atomically creates
the three regular targets hidden below the writable parent bind; protected Pi auth and tracked
model/settings sources remain separate read-only mounts. See the server manual before substituting
a root other than `/srv/thothii/pi-state`.
The mandatory stack includes frontend, core, PostgreSQL catalog, Qdrant, Ollama
and the embedding initializer. DWH and LLM endpoints remain external dependencies.
Pi is included in the core image. Credentials and certificates belong in protected
installation-local files, never in the workspace repository.
Workspace descriptors come from the Git remote configured by `THT_WORKSPACE_GIT_REMOTE`; their
runtime endpoint and secret bindings remain installation-local. Open
<http://127.0.0.1:8080> (set `THOTH_HTTP_PORT` in `deploy/env/local.env` to choose another
loopback port).
Credentials and certificates are local protected files. Do not put them in environment examples,
workspace YAML, URLs, or Compose interpolation values.
Application state is split across the named `settings`, `pi-state`, `workspace-registry`,
`sessions`, `qdrant-data`, and `embedding-models` volumes. `docker compose down` keeps them.
`qdrant-data` is a derived but persistent index store; `embedding-models` is an Ollama model
cache for `qwen3-embedding:0.6b` with fixed `1024`-dimension embeddings. Only an explicit destructive command such as `docker compose
down --volumes` removes them.
The frontend depends on the core health check and proxies `/health` and `/api/*` to it. The
application health endpoint intentionally checks process readiness only; external dependency
diagnostics are exposed by `tht doctor` and do not prevent the UI from starting.
For developer topology, overlays and lifecycle details, see the internal
[Compose reference](docs/operations/compose-reference.md). Ordinary stop/down keeps
persistent data; removing volumes is destructive and is not an upgrade step.
Process health is distinct from external dependency checks performed by doctor.
## Git-backed workspace repository
+609
View File
@@ -6,11 +6,13 @@
"": {
"name": "thothii-backend",
"dependencies": {
"@aws-sdk/client-s3": "3.1141.0",
"@fastify/cookie": "11.1.2",
"@fastify/cors": "^11.2.0",
"@fastify/rate-limit": "11.2.0",
"@types/pg": "^8.20.3",
"fastify": "^5.0.0",
"ipaddr.js": "2.4.0",
"kysely": "^0.29.5",
"libphonenumber-js": "1.13.12",
"openid-client": "6.8.5",
@@ -23,11 +25,320 @@
"@testcontainers/postgresql": "^12.1.0",
"@types/node": "24.13.3",
"@types/validator": "13.15.10",
"bun": "1.4.2",
"tsx": "^4.19.0",
"typescript": "^5.6.0",
"vitest": "^2.1.0"
}
},
"node_modules/@aws-sdk/checksums": {
"version": "3.1001.1",
"resolved": "https://registry.npmjs.org/@aws-sdk/checksums/-/checksums-3.1001.1.tgz",
"integrity": "sha512-x12Q17KYlJAd3nKf8LV5LV0vt8sh8/6YfQLGPtrGnQf/tW4jqxPGq5GPpuVitpQYM3eUR4XB7CbxZf751NMbLw==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/client-s3": {
"version": "3.1141.0",
"resolved": "https://registry.npmjs.org/@aws-sdk/client-s3/-/client-s3-3.1141.0.tgz",
"integrity": "sha512-uOVH37xGLenAdJkCPCin/JJG2PgWrFcSsDnQ9+C9Zq8N9Oalo5ol4xmn5fG28iWAlA/b/9boQZgHbMh+UsIhcg==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/checksums": "^3.1001.1",
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/credential-provider-node": "^3.972.84",
"@aws-sdk/middleware-sdk-s3": "^3.972.77",
"@aws-sdk/signature-v4-multi-region": "^3.996.47",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/fetch-http-handler": "^5.8.0",
"@smithy/node-http-handler": "^4.12.1",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/core": {
"version": "3.978.1",
"resolved": "https://registry.npmjs.org/@aws-sdk/core/-/core-3.978.1.tgz",
"integrity": "sha512-LbY9aGsEiznDWmUc30Nwv3aIX/+dbwTx8KfS0yOC3NPYMO+O91e6jkT1azf34FwjOndq8/Q+RcVVZz5xnerwdg==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/types": "^3.974.6",
"@aws-sdk/xml-builder": "^3.972.41",
"@aws/lambda-invoke-store": "^0.3.0",
"@smithy/core": "^3.35.0",
"@smithy/signature-v4": "^5.7.3",
"@smithy/types": "^4.19.0",
"bowser": "^2.11.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/credential-provider-env": {
"version": "3.972.72",
"resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-env/-/credential-provider-env-3.972.72.tgz",
"integrity": "sha512-xTKO/FWJPozTIXbozVnVGoNBhaGba8TBcx+KyUjRVeOlXE+dUc7GTR1cLvu0uTdIdmemzaFbqqCshXeZA1fZew==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/credential-provider-http": {
"version": "3.972.74",
"resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-http/-/credential-provider-http-3.972.74.tgz",
"integrity": "sha512-u91E/hT8f4d1xy0Jl7VG4nVKJ3lxbrZkoBTeSVoJdWBiSEUMwMS/9+e0H/aJVQV//Lt5wuzP+E69v4aRSsNTmw==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/fetch-http-handler": "^5.8.0",
"@smithy/node-http-handler": "^4.12.1",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/credential-provider-ini": {
"version": "3.973.17",
"resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-ini/-/credential-provider-ini-3.973.17.tgz",
"integrity": "sha512-ged4KXdBkvIC81bLvNHHuQKdKak/VXhQTR1NWYTTqW0474nlmsxy9O/vlgTIohDDWH3xpBdtVMZRyjb+DnocDA==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/credential-provider-env": "^3.972.72",
"@aws-sdk/credential-provider-http": "^3.972.74",
"@aws-sdk/credential-provider-login": "^3.972.79",
"@aws-sdk/credential-provider-process": "^3.972.72",
"@aws-sdk/credential-provider-sso": "^3.973.16",
"@aws-sdk/credential-provider-web-identity": "^3.972.78",
"@aws-sdk/nested-clients": "^3.997.46",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/credential-provider-imds": "^4.5.2",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/credential-provider-login": {
"version": "3.972.79",
"resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-login/-/credential-provider-login-3.972.79.tgz",
"integrity": "sha512-L+Z85anONJd8MaiuraO4wRxATCdEejBZ3K3eymzWI5JPXa9sOS9CkIm72PBKqXKX+Z9p9NGMX5AIMXm0LEflgw==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/nested-clients": "^3.997.46",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/credential-provider-node": {
"version": "3.972.84",
"resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-node/-/credential-provider-node-3.972.84.tgz",
"integrity": "sha512-oHt854odINVwzwsh+c5x69j0ajm4DbqqqVJ+O1ECsCIZeMDAbzFpXItaqP7UZstJj/ATdTk/KFSH0LaNAgV+kA==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/credential-provider-env": "^3.972.72",
"@aws-sdk/credential-provider-http": "^3.972.74",
"@aws-sdk/credential-provider-ini": "^3.973.17",
"@aws-sdk/credential-provider-process": "^3.972.72",
"@aws-sdk/credential-provider-sso": "^3.973.16",
"@aws-sdk/credential-provider-web-identity": "^3.972.78",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/credential-provider-imds": "^4.5.2",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/credential-provider-process": {
"version": "3.972.72",
"resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-process/-/credential-provider-process-3.972.72.tgz",
"integrity": "sha512-rLIp2xbMjX/k9/od7APpqq1ZgXXnV0pOL1Th3ZsL8Wu0TRtBsDTVS8iPqcfRFcHakFxPvR04OSTv2ka2qOb/2A==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/credential-provider-sso": {
"version": "3.973.16",
"resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-sso/-/credential-provider-sso-3.973.16.tgz",
"integrity": "sha512-IGihaJfFZYacJJr/odqILCoK7W/mvrZ7cuK7ECn3sAu4vLC6u0V8bS7mCGbdugJ8Aum2tnvqmx0F2MRFp2rn9g==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/nested-clients": "^3.997.46",
"@aws-sdk/token-providers": "3.1138.0",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/credential-provider-web-identity": {
"version": "3.972.78",
"resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-web-identity/-/credential-provider-web-identity-3.972.78.tgz",
"integrity": "sha512-/y9WvNtlcPBGLR0qc1a+9J/xtYZfVczvLUOuXaVWylzttH7ewsxwHtjmiJSolNrVSDorIxHGHMU61CbonRkmwA==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/nested-clients": "^3.997.46",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/middleware-sdk-s3": {
"version": "3.972.77",
"resolved": "https://registry.npmjs.org/@aws-sdk/middleware-sdk-s3/-/middleware-sdk-s3-3.972.77.tgz",
"integrity": "sha512-E7W2UOeUoc+lg3uIfR/dM7ZwusHwhBQrKMnlkRv4EXRR+C0YtV1pg25xC7GdZIhXH+NAMgZPCbE7o5to2cjFiw==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/signature-v4-multi-region": "^3.996.47",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/nested-clients": {
"version": "3.997.46",
"resolved": "https://registry.npmjs.org/@aws-sdk/nested-clients/-/nested-clients-3.997.46.tgz",
"integrity": "sha512-oRxtBcka/JGHGs9l9p9IVajGoTP8vTPmoAzdHGy4Qcy9P5vPnDf6nhIeM/COQNY9k/OahImTRaLkHftoXvfcmQ==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/signature-v4-multi-region": "^3.996.47",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/fetch-http-handler": "^5.8.0",
"@smithy/node-http-handler": "^4.12.1",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/signature-v4-multi-region": {
"version": "3.996.47",
"resolved": "https://registry.npmjs.org/@aws-sdk/signature-v4-multi-region/-/signature-v4-multi-region-3.996.47.tgz",
"integrity": "sha512-Zk08macMvQTHzQJCLJVkOlviVoqwYMrpXv4lmLN7b7sAbiMoOK7Go0NYdR5UeF+MW8LIbRmwrNy9u/5VvX1U5g==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/types": "^3.974.6",
"@smithy/signature-v4": "^5.7.3",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/token-providers": {
"version": "3.1138.0",
"resolved": "https://registry.npmjs.org/@aws-sdk/token-providers/-/token-providers-3.1138.0.tgz",
"integrity": "sha512-GpyAr0DD63YOEmYFM6Df+gJuIgC92MMTiBK4FTKfxii5MJ9ge20epR7LyroulscYlG89J+ZB2ivFDPjvfQhzdw==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/nested-clients": "^3.997.46",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/types": {
"version": "3.974.6",
"resolved": "https://registry.npmjs.org/@aws-sdk/types/-/types-3.974.6.tgz",
"integrity": "sha512-v/clNZzZnDxGyvpHMOGpJKVXFAExJzUNAAjaWGdcx8QAcXLGwTaOkw33p5SHAi0YAioK32xB3hWwOekRVfmfKg==",
"license": "Apache-2.0",
"dependencies": {
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/xml-builder": {
"version": "3.972.41",
"resolved": "https://registry.npmjs.org/@aws-sdk/xml-builder/-/xml-builder-3.972.41.tgz",
"integrity": "sha512-ctjVSyCMegrWfXlx6VqzSBFI6UqmQ5ZlnfMhdLIiWmhoH8UAQxSCP5N3OpG7X3k4LnS7ou74C4mt20+bfTW2aQ==",
"license": "Apache-2.0",
"dependencies": {
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws/lambda-invoke-store": {
"version": "0.3.0",
"resolved": "https://registry.npmjs.org/@aws/lambda-invoke-store/-/lambda-invoke-store-0.3.0.tgz",
"integrity": "sha512-sl4Bm6yiMNYrZKkqqDFWN0UfnWhlS8ivKxrYl+6t0gCLrqr8y3B2IqZZbFRkfaVVp7C/baApyh71P+LeE1A2sQ==",
"license": "Apache-2.0",
"engines": {
"node": ">=18.0.0"
}
},
"node_modules/@balena/dockerignore": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/@balena/dockerignore/-/dockerignore-1.0.2.tgz",
@@ -802,6 +1113,174 @@
"node": ">=8"
}
},
"node_modules/@oven/bun-darwin-aarch64": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-darwin-aarch64/-/bun-darwin-aarch64-1.4.2.tgz",
"integrity": "sha512-MXdZkP1featqxZ+/VTXWG1BVjM4OGBehVY2Q88EeUj/7L0UMeCGItmyPYTN+wxvlGJ6F66JEtzsw+GvQWewnag==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"darwin"
]
},
"node_modules/@oven/bun-darwin-x64": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-darwin-x64/-/bun-darwin-x64-1.4.2.tgz",
"integrity": "sha512-gZTxZuLjkUhAWjTETu3tw0WhsEdNkJ64daj60ybhPf835a2yollV3yTkK9JozvzKPx4TRFzLSl8C+U525pxVbw==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"darwin"
]
},
"node_modules/@oven/bun-freebsd-aarch64": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-freebsd-aarch64/-/bun-freebsd-aarch64-1.4.2.tgz",
"integrity": "sha512-SMNItMw1Z8QeeQVKnw8jA7xQNkeXdP+OPgin4Wi/QTx/B8RHHLnuZfqmFy7NtVeT2NF0kKYppW4WWd2CCYZjhQ==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"freebsd"
]
},
"node_modules/@oven/bun-freebsd-x64": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-freebsd-x64/-/bun-freebsd-x64-1.4.2.tgz",
"integrity": "sha512-THbPKXhO54N0DpFRKZNDZpQ7dpbX0bWASuARckAUS9wRtFIHsiY+uULXJvxJGo2YD1YewvXQ4G8Fj7XT5oBCiw==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"freebsd"
]
},
"node_modules/@oven/bun-linux-aarch64": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-linux-aarch64/-/bun-linux-aarch64-1.4.2.tgz",
"integrity": "sha512-3BBP9ovJ2RGHFH6Ae1CAtxNtG1+YY6GD6rmYbsUosoAk9+OEl6zeDQ/k4fBkc6dYOJCtWnx8hUxzNzQATSmvYQ==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/@oven/bun-linux-aarch64-android": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-linux-aarch64-android/-/bun-linux-aarch64-android-1.4.2.tgz",
"integrity": "sha512-3mZKO2rhsNgbAUtAHC1UKUlF2zTxFraDZT/Elv8wzyH0fJL9h+Iv3TgB9lO63w89PRn3eFe+NRA1bhVgikKNPQ==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"android"
]
},
"node_modules/@oven/bun-linux-aarch64-musl": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-linux-aarch64-musl/-/bun-linux-aarch64-musl-1.4.2.tgz",
"integrity": "sha512-+Sm6y+lSiSFBOtXmnekp5Q6n1tUKlyv71FCPWBc61Cgb14T5eBs8SN/nh4MUCOKzONkI3O+as3MGUgikS4aCBQ==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/@oven/bun-linux-x64": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-linux-x64/-/bun-linux-x64-1.4.2.tgz",
"integrity": "sha512-9/E/UXOTpSo3YsV5g+FhtTd/qTpiWoKuxS12cqtuYA1ssu9fRAoPQnipFgGyck3tWO63iUdxBiygq+kELFawng==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/@oven/bun-linux-x64-android": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-linux-x64-android/-/bun-linux-x64-android-1.4.2.tgz",
"integrity": "sha512-6HC5tzcC79113n2IHCTJMWv+HsQImv4ZFEK2XpYLxY6HbT8tM4cUM2Zv1bHZBQsS3jv/zYBamDJ1UX7If0d5tw==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"android"
]
},
"node_modules/@oven/bun-linux-x64-musl": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-linux-x64-musl/-/bun-linux-x64-musl-1.4.2.tgz",
"integrity": "sha512-vVTKUg1bnPhRP/Hp73jIVoFh2vPFNYEqYX0ERKfZBOQEEHitNAeukZzzuUDZS0SoDCIpuWUGSpd/CDMbjdR+Uw==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/@oven/bun-windows-aarch64": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-windows-aarch64/-/bun-windows-aarch64-1.4.2.tgz",
"integrity": "sha512-8EJ1ST7339WJE3poPW5nBgVW/lWf9HBz4W27ZUNhburKmcBLOByPyE6DP9fHD8FQGm5c+ilUN2hX1mrW0jxq9Q==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"win32"
]
},
"node_modules/@oven/bun-windows-x64": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-windows-x64/-/bun-windows-x64-1.4.2.tgz",
"integrity": "sha512-+bN6OuVld/9diT/RLSXSW7JE6CvNE3gL9XsAEjULi1nUsXd6DNO6GuA9jNdNb3r8PdJFnYHr5aypNV1Oj3Rd9g==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"win32"
]
},
"node_modules/@pinojs/redact": {
"version": "0.4.0",
"resolved": "https://registry.npmjs.org/@pinojs/redact/-/redact-0.4.0.tgz",
@@ -1235,6 +1714,87 @@
"win32"
]
},
"node_modules/@smithy/core": {
"version": "3.35.0",
"resolved": "https://registry.npmjs.org/@smithy/core/-/core-3.35.0.tgz",
"integrity": "sha512-zRMhfkByhT2snNdr1si24vJitU6Cr9ix2MikUfWmkAgp4jrNP0GcKSP5YvwQ+TlI8AZXER5QOGJn3JsVtSD9/A==",
"license": "Apache-2.0",
"dependencies": {
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=18.0.0"
}
},
"node_modules/@smithy/credential-provider-imds": {
"version": "4.5.2",
"resolved": "https://registry.npmjs.org/@smithy/credential-provider-imds/-/credential-provider-imds-4.5.2.tgz",
"integrity": "sha512-A9uSdn72ozbRUSit0eib0TW7nXuNPlaeM0zcGkJ+nE6tFcSDbnmtwoxbTCFBukVQcszDAyvsd7+rTduPTXpygg==",
"license": "Apache-2.0",
"dependencies": {
"@smithy/core": "^3.33.2",
"@smithy/types": "^4.17.2",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=18.0.0"
}
},
"node_modules/@smithy/fetch-http-handler": {
"version": "5.8.0",
"resolved": "https://registry.npmjs.org/@smithy/fetch-http-handler/-/fetch-http-handler-5.8.0.tgz",
"integrity": "sha512-ycSJu3tFAQ4v04CBB0agqFMVsSQ1iG3yw+SpgxRqKfaURpQD4CZ8Wn0zPMmSnOuTpTh65Vz+EA0rMrw089wvkA==",
"license": "Apache-2.0",
"dependencies": {
"@smithy/core": "^3.33.3",
"@smithy/types": "^4.18.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=18.0.0"
}
},
"node_modules/@smithy/node-http-handler": {
"version": "4.12.1",
"resolved": "https://registry.npmjs.org/@smithy/node-http-handler/-/node-http-handler-4.12.1.tgz",
"integrity": "sha512-ThMkboGeONWXAelq9FvGsuJC4rOi+qyC4/zhUF58xYpxUg5sQKx2VXZYJmtNjr4dSuBJ1HeJXETQILCz3wOHvw==",
"license": "Apache-2.0",
"dependencies": {
"@smithy/core": "^3.33.3",
"@smithy/types": "^4.18.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=18.0.0"
}
},
"node_modules/@smithy/signature-v4": {
"version": "5.7.4",
"resolved": "https://registry.npmjs.org/@smithy/signature-v4/-/signature-v4-5.7.4.tgz",
"integrity": "sha512-tHy0K0VtqNd5Y7Y41h0a0Lhh0L1GzC08dTWg0F7vRJWFtTENg7IZikf3wQkanYIRdb7ngoIPMTmqgUi401fEeQ==",
"license": "Apache-2.0",
"dependencies": {
"@smithy/core": "^3.35.0",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=18.0.0"
}
},
"node_modules/@smithy/types": {
"version": "4.19.0",
"resolved": "https://registry.npmjs.org/@smithy/types/-/types-4.19.0.tgz",
"integrity": "sha512-r7jh49VJxGerfAcTQA6gXcKc+98zOp/tqRwzYjgOE+iSQsP6cEU1hq2QzbuipmP68QtYdY9wKEhiCQZIzHgZ4Q==",
"license": "Apache-2.0",
"dependencies": {
"tslib": "^2.6.2"
},
"engines": {
"node": ">=18.0.0"
}
},
"node_modules/@testcontainers/postgresql": {
"version": "12.1.0",
"resolved": "https://registry.npmjs.org/@testcontainers/postgresql/-/postgresql-12.1.0.tgz",
@@ -1821,6 +2381,12 @@
"node": ">= 6"
}
},
"node_modules/bowser": {
"version": "2.14.1",
"resolved": "https://registry.npmjs.org/bowser/-/bowser-2.14.1.tgz",
"integrity": "sha512-tzPjzCxygAKWFOJP011oxFHs57HzIhOEracIgAePE4pqB3LikALKnSzUyU4MGs9/iCEUuHlAJTjTc5M+u7YEGg==",
"license": "MIT"
},
"node_modules/brace-expansion": {
"version": "2.1.4",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.1.4.tgz",
@@ -1876,6 +2442,43 @@
"node": ">=10.0.0"
}
},
"node_modules/bun": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/bun/-/bun-1.4.2.tgz",
"integrity": "sha512-TrSXo6HJfIEaczpb3kjX82I2pL47vK1QUNmHRCUdz9IzaOwa9lzOXSWwu2l18YHE3sNfGRapVLd4nNm+22vVVA==",
"cpu": [
"arm64",
"x64"
],
"dev": true,
"hasInstallScript": true,
"license": "MIT",
"os": [
"darwin",
"linux",
"android",
"freebsd",
"win32"
],
"bin": {
"bun": "bin/bun.exe",
"bunx": "bin/bunx.exe"
},
"optionalDependencies": {
"@oven/bun-darwin-aarch64": "1.4.2",
"@oven/bun-darwin-x64": "1.4.2",
"@oven/bun-freebsd-aarch64": "1.4.2",
"@oven/bun-freebsd-x64": "1.4.2",
"@oven/bun-linux-aarch64": "1.4.2",
"@oven/bun-linux-aarch64-android": "1.4.2",
"@oven/bun-linux-aarch64-musl": "1.4.2",
"@oven/bun-linux-x64": "1.4.2",
"@oven/bun-linux-x64-android": "1.4.2",
"@oven/bun-linux-x64-musl": "1.4.2",
"@oven/bun-windows-aarch64": "1.4.2",
"@oven/bun-windows-x64": "1.4.2"
}
},
"node_modules/byline": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/byline/-/byline-5.0.0.tgz",
@@ -4114,6 +4717,12 @@
"node": ">=20"
}
},
"node_modules/tslib": {
"version": "2.8.1",
"resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz",
"integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==",
"license": "0BSD"
},
"node_modules/tsx": {
"version": "4.22.4",
"resolved": "https://registry.npmjs.org/tsx/-/tsx-4.22.4.tgz",
+4
View File
@@ -3,6 +3,7 @@
"private": true,
"type": "module",
"scripts": {
"build:workspace-tools": "node scripts/build-workspace-tools.mjs",
"dev": "tsx watch src/server.ts",
"prebuild": "node scripts/clean-dist.mjs",
"build": "tsc -p tsconfig.json",
@@ -14,11 +15,13 @@
"test:schema-v3-verifier": "npm run test:schema-v4-verifier"
},
"dependencies": {
"@aws-sdk/client-s3": "3.1141.0",
"@fastify/cookie": "11.1.2",
"@fastify/cors": "^11.2.0",
"@fastify/rate-limit": "11.2.0",
"@types/pg": "^8.20.3",
"fastify": "^5.0.0",
"ipaddr.js": "2.4.0",
"kysely": "^0.29.5",
"libphonenumber-js": "1.13.12",
"openid-client": "6.8.5",
@@ -31,6 +34,7 @@
"@testcontainers/postgresql": "^12.1.0",
"@types/node": "24.13.3",
"@types/validator": "13.15.10",
"bun": "1.4.2",
"tsx": "^4.19.0",
"typescript": "^5.6.0",
"vitest": "^2.1.0"
+49
View File
@@ -0,0 +1,49 @@
// Maintainer-only build. The resulting two-binary bundle needs no extra host runtime.
import { spawnSync } from "node:child_process";
import { createHash } from "node:crypto";
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { dirname, join, resolve } from "node:path";
import { fileURLToPath } from "node:url";
const repository = process.env.THT_BUILD_SOURCE_ROOT ? resolve(process.env.THT_BUILD_SOURCE_ROOT) : resolve(dirname(fileURLToPath(import.meta.url)), "../..");
const backend = join(repository, "backend");
const native = `${process.platform === "win32" ? "windows" : process.platform}-${process.arch === "x64" ? "amd64" : process.arch}`;
const targets = {
"windows-amd64": ["windows", "amd64", "bun-windows-x64"],
"darwin-amd64": ["darwin", "amd64", "bun-darwin-x64"],
"darwin-arm64": ["darwin", "arm64", "bun-darwin-arm64"],
"linux-amd64": ["linux", "amd64", "bun-linux-x64"],
"linux-arm64": ["linux", "arm64", "bun-linux-arm64"],
};
const requested = process.argv.slice(2);
const selected = requested.length === 1 && requested[0] === "--all" ? Object.keys(targets) : requested.length ? requested : [native];
if (selected.some((target) => !targets[target])) {
console.error(`Usage: npm run build:workspace-tools -- [${Object.keys(targets).join("|")}|--all]`);
process.exit(2);
}
function run(command, args, cwd = backend, env = process.env) {
const result = spawnSync(command, args, { cwd, env, stdio: "inherit" });
if (result.error || result.status !== 0) throw new Error(`Build failed: ${command}`);
}
const revision = spawnSync("git", ["rev-parse", "HEAD"], { cwd: repository, encoding: "utf8" });
if (revision.status !== 0) throw new Error("Cannot read build revision");
const commit = revision.stdout.trim();
const buildTime = process.env.THT_BUILD_TIME ?? new Date().toISOString();
const releaseVersion = process.env.THT_BUILD_VERSION ?? "0.0.0-dev";
if (!/^[0-9]+\.[0-9]+\.[0-9]+(?:-[A-Za-z0-9.-]+)?$/.test(releaseVersion) || !Number.isFinite(Date.parse(buildTime))) throw new Error("Invalid build identity");
const module = "github.com/aritmolab/thothii/tools/tht/internal/version";
const bunPackage = JSON.parse(readFileSync(join(backend, "node_modules", "bun", "package.json"), "utf8"));
const bun = join(backend, "node_modules", "bun", bunPackage.bin.bun);
for (const target of selected) {
const [os, arch, bunTarget] = targets[target];
const output = join(repository, "dist", "workspace-tools", target);
mkdirSync(output, { recursive: true });
const extension = os === "windows" ? ".exe" : "";
const names = [`tht${extension}`, `tht-workspace-documents${extension}`];
run(bun, ["build", "src/workspace-documents-cli.ts", "--compile", `--target=${bunTarget}`, "--outfile", join(output, names[1])]);
run("go", ["build", "-trimpath", "-ldflags", `-s -w -X ${module}.semanticVersion=${releaseVersion} -X ${module}.commit=${commit} -X ${module}.buildTime=${buildTime}`, "-o", join(output, names[0]), "./cmd/tht"], join(repository, "tools", "tht"), { ...process.env, CGO_ENABLED: "0", GOOS: os, GOARCH: arch });
const hashes = names.map((name) => `${createHash("sha256").update(readFileSync(join(output, name))).digest("hex")} ${name}\n`).join("");
writeFileSync(join(output, "SHA256SUMS"), hashes);
writeFileSync(join(output, "build.json"), JSON.stringify({ version: releaseVersion, commit, buildTime, target, bun: JSON.parse(readFileSync(join(backend, "package.json"), "utf8")).devDependencies.bun }, null, 2) + "\n");
console.log(output);
}
+195
View File
@@ -0,0 +1,195 @@
#!/usr/bin/env node
// Maintainer-only producer. Consumers download the resulting native bundle.
import { spawn } from "node:child_process";
import { mkdirSync, mkdtempSync, readFileSync, writeFileSync, existsSync, realpathSync, renameSync, rmSync, statSync, copyFileSync, chmodSync, openSync, closeSync, readdirSync } from "node:fs";
import { tmpdir } from "node:os";
import { basename, dirname, join, resolve } from "node:path";
import { pathToFileURL } from "node:url";
import { parse } from "yaml";
import { prepareBundle, sha256 } from "./release-bundle.mjs";
import { dockerCredentials, dockerHub } from "./release-registry.mjs";
import { giteaHosting } from "./release-hosting.mjs";
import { publishVerifiedRelease } from "./release-publication.mjs";
const repositoryRoot = resolve(import.meta.dirname, "../..");
export function optionsFromArgs(args) {
const values = {};
for (let i = 0; i < args.length; i += 2) {
if (!['--revision', '--version', '--namespace', '--platforms', '--output', '--repository'].includes(args[i]) || !args[i + 1] || values[args[i]]) throw new Error("Usage: --revision REF --version VERSION --namespace DOCKER_HUB_NAMESPACE --platforms linux/amd64 --output NEW_OR_MATCHING_DIRECTORY [--repository HTTPS_GITEA_REPO]");
values[args[i]] = args[i + 1];
}
if (!values['--revision'] || values['--revision'].startsWith('-') || !/^[0-9]+\.[0-9]+\.[0-9]+(?:-[A-Za-z0-9.-]+)?$/.test(values['--version'] ?? '') || !/^[a-z0-9][a-z0-9_-]{1,38}$/.test(values['--namespace'] ?? '') || !values['--output']) throw new Error("Supply an explicit source revision, semantic release version, Docker Hub namespace and output directory.");
const platforms = (values['--platforms'] ?? '').split(',');
if (!platforms.length || new Set(platforms).size !== platforms.length || platforms.some((p) => !['linux/amd64', 'linux/arm64'].includes(p))) throw new Error("Select explicit Linux image platforms; begin with linux/amd64 for Windows/WSL2 and Omarchy.");
const repository = values['--repository'] ?? 'https://git.tylconsulting.it/mptyl/ThothII';
const url = new URL(repository);
if (url.protocol !== 'https:' || url.username || url.password || url.search || url.hash || !/^\/[A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+$/.test(url.pathname)) throw new Error("Use a credential-free HTTPS Gitea owner/repository URL.");
return { revision: values['--revision'], version: values['--version'], namespace: values['--namespace'], platforms: platforms.sort(), output: resolve(values['--output']), repository };
}
export function commandRunner(logs) {
let sequence = 0;
return async (command, args, { cwd = repositoryRoot, input = '', env = process.env, quiet = false, timeout = 120_000 } = {}) => {
const log = join(logs, `${++sequence}-${basename(command)}.log`);
return new Promise((accept, reject) => {
if (process.platform === 'win32') { reject(new Error('Run the release producer on Linux, WSL2 or macOS.')); return; }
const child = spawn(command, args, { cwd, env, detached: true, stdio: ['pipe', 'pipe', 'pipe'] });
const stdout = [], stderr = []; let size = 0, overflow = false, settled = false, reapTimer;
const terminate = () => {
if (overflow) return;
overflow = true;
try { process.kill(-child.pid, 'SIGKILL'); } catch { child.kill('SIGKILL'); }
// An escaped descendant must never retain our pipes indefinitely.
reapTimer = setTimeout(() => { child.stdout.destroy(); child.stderr.destroy(); finish(-1); }, 500);
};
const timer = setTimeout(terminate, timeout);
const collect = (chunks) => (data) => { size += data.length; if (size > 64 * 2 ** 20) terminate(); else chunks.push(data); };
child.stdout.on('data', collect(stdout)); child.stderr.on('data', collect(stderr));
child.stdin.on('error', () => {}); child.stdin.end(input);
child.on('error', () => { settled = true; clearTimeout(timer); clearTimeout(reapTimer); reject(new Error(`Required maintainer command ${basename(command)} is unavailable.`)); });
function finish(code) {
if (settled) return;
settled = true;
clearTimeout(timer); clearTimeout(reapTimer);
if (!quiet) writeFileSync(log, Buffer.concat([...stdout, ...stderr]), { mode: 0o600 });
if (code !== 0 || overflow) reject(new Error(`${basename(command)} failed${quiet ? '.' : `; inspect private log ${log}`}`));
else accept(Buffer.concat(stdout).toString());
}
child.on('close', finish);
});
};
}
function acquireOutput(output) {
if (!existsSync(output)) mkdirSync(output, { mode: 0o700 });
if (realpathSync(output) !== output || !statSync(output).isDirectory() || (statSync(output).mode & 0o077)) throw new Error("Use a canonical owner-only output directory.");
if (!existsSync(join(output, 'publication-state.json')) && readdirSync(output).length) throw new Error("Choose an empty output directory or the matching previous publication directory.");
const lock = join(output, '.publisher.lock');
if (existsSync(lock)) {
const pid = Number(readFileSync(lock, 'utf8'));
if (!Number.isInteger(pid) || pid <= 0) throw new Error("Inspect the incomplete publisher lock before retrying.");
let alive = true;
try { process.kill(pid, 0); } catch (error) { if (error.code === 'ESRCH') alive = false; }
if (alive) throw new Error("Another publisher owns this output directory.");
rmSync(lock);
}
const fd = openSync(lock, 'wx', 0o600); writeFileSync(fd, String(process.pid)); closeSync(fd);
return () => rmSync(lock, { force: true });
}
export async function publishInstallation(options) {
const unlock = acquireOutput(options.output);
const logs = join(options.output, 'logs'); mkdirSync(logs, { recursive: true, mode: 0o700 });
const run = commandRunner(logs);
let worktree, temporary;
try {
const revision = (await run('git', ['rev-parse', '--verify', `${options.revision}^{commit}`], { quiet: true })).trim();
if (!/^[a-f0-9]{40}$/.test(revision)) throw new Error("Source revision is not a commit.");
const identity = { revision, version: options.version, namespace: options.namespace, platforms: options.platforms, repository: options.repository };
const statePath = join(options.output, 'publication-state.json');
let state = { identity };
if (existsSync(statePath)) {
state = JSON.parse(readFileSync(statePath, 'utf8'));
if (JSON.stringify(state.identity) !== JSON.stringify(identity)) throw new Error("Output directory belongs to a different release; choose a new directory.");
}
const save = () => { const temp = statePath + '.tmp'; writeFileSync(temp, JSON.stringify(state, null, 2) + '\n', { mode: 0o600 }); renameSync(temp, statePath); };
save();
console.log(`Release ${identity.version}: ${identity.namespace}, ${identity.platforms.join(',')}, source ${revision}`);
await run('docker', ['info', '--format', '{{.OSType}}']);
await run('docker', ['buildx', 'version']);
const registry = dockerHub(await dockerCredentials(run));
const hosting = await giteaHosting({ run, repository: options.repository, identity, body: `Installer prerelease for ${identity.platforms.join(', ')}.\n\nImages: docker.io/${identity.namespace}/thothii-core:${identity.version} and docker.io/${identity.namespace}/thothii-frontend:${identity.version}.\n\nDownload the native operator bundle and SHA256SUMS.txt below. This release supplies images, document validation and preflight; complete non-interactive setup and Windows/Omarchy/macOS acceptance are subsequent tickets. No example databases, credentials or user workspace data are included.` });
async function prepare() {
if (state.assets) {
const names = [...identity.platforms.map((platform) => `thothii-${identity.version}-${platform.replace('/', '-')}.tar.gz`), 'SHA256SUMS.txt'];
if (state.assets.length !== names.length) throw new Error("Cached artifact set is incomplete.");
for (const asset of state.assets) if (!names.includes(asset.name) || asset.path !== join(options.output, asset.name) || !existsSync(asset.path) || sha256(readFileSync(asset.path)) !== asset.sha256) throw new Error("Previously built release artifact changed; do not overwrite an immutable version.");
for (const [platform, images] of Object.entries(state.images)) for (const reference of Object.values(images)) {
const [repository, digest] = reference.replace(/^docker.io\//, '').split('@');
await registry.inspect(repository, digest, platform, { anonymous: true });
}
return state.assets;
}
temporary = realpathSync(mkdtempSync(join(tmpdir(), 'thothii-release-')));
worktree = join(temporary, 'source');
await run('git', ['worktree', 'add', '--detach', worktree, revision]);
const sourceCompose = parse(readFileSync(join(worktree, 'compose.yaml'), 'utf8'));
for (const role of ['core', 'frontend']) {
console.log(`Preparing public repository ${identity.namespace}/thothii-${role}`);
await registry.ensurePublic(identity.namespace, `thothii-${role}`);
let existing = null;
for (const platform of identity.platforms) {
const image = await registry.inspect(`${identity.namespace}/thothii-${role}`, identity.version, platform, { allowMissing: true });
if (image && (image.labels['org.opencontainers.image.revision'] !== revision || image.labels['org.opencontainers.image.version'] !== identity.version)) throw new Error("Image tag already belongs to another immutable build; select a new release version.");
existing = existing || image;
}
if (!existing) {
console.log(`Building and publishing ${role} (${identity.platforms.join(', ')})`);
await run('docker', ['buildx', 'build', '--platform', identity.platforms.join(','), '--file', `docker/${role}.Dockerfile`, '--tag', `docker.io/${identity.namespace}/thothii-${role}:${identity.version}`, '--build-arg', `IMAGE_VERSION=${identity.version}`, '--label', `org.opencontainers.image.revision=${revision}`, '--label', `org.opencontainers.image.source=${identity.repository}`, '--provenance=false', '--sbom=false', '--push', '.'], { cwd: worktree, timeout: 45 * 60_000 });
}
}
state.images = {};
for (const platform of identity.platforms) {
const images = {};
for (const role of ['core', 'frontend']) images[role] = (await registry.inspect(`${identity.namespace}/thothii-${role}`, identity.version, platform, { anonymous: true })).reference;
for (const [role, service] of [['catalog', 'catalog-db'], ['qdrant', 'qdrant'], ['embedding', 'embedding']]) {
const [named, digest] = sourceCompose.services[service].image.split('@');
let repository = named.replace(/:[^/:]+$/, '').replace(/^docker.io\//, '');
if (!repository.includes('/')) repository = 'library/' + repository;
images[role] = (await registry.inspect(repository, digest, platform, { anonymous: true })).reference;
}
state.images[platform] = images;
}
save();
console.log('Building native operator bundles from the selected source');
await run('npm', ['ci'], { cwd: join(worktree, 'backend'), timeout: 10 * 60_000 });
const sourceTime = (await run('git', ['show', '-s', '--format=%cI', revision], { quiet: true })).trim();
const targets = identity.platforms.map((platform) => platform.replace('/', '-'));
await run('node', [join(repositoryRoot, 'backend/scripts/build-workspace-tools.mjs'), ...targets], { cwd: join(worktree, 'backend'), env: { ...process.env, THT_BUILD_SOURCE_ROOT: worktree, THT_BUILD_VERSION: identity.version, THT_BUILD_TIME: sourceTime }, timeout: 10 * 60_000 });
const assets = [];
for (const platform of identity.platforms) {
const target = platform.replace('/', '-');
const name = `thothii-${identity.version}-${target}`;
const bundle = join(options.output, name);
if (existsSync(bundle)) rmSync(bundle, { recursive: true }); // owned staging, never an installed runtime
mkdirSync(bundle);
prepareBundle({ source: worktree, destination: bundle, platform, version: identity.version, revision, images: state.images[platform] });
mkdirSync(join(bundle, 'bin'));
for (const executable of ['tht', 'tht-workspace-documents']) {
copyFileSync(join(worktree, 'dist/workspace-tools', target, executable), join(bundle, 'bin', executable));
chmodSync(join(bundle, 'bin', executable), 0o755);
}
// Resolve source-independent resource/config shape without any operator credentials.
await run('docker', ['compose', '-f', join(bundle, 'compose.yaml'), '-f', join(bundle, 'deploy/compose.local.yaml'), 'config', '--no-interpolate', '--no-env-resolution', '--format', 'json']);
const archive = join(options.output, name + '.tar.gz');
await run('tar', ['-czf', archive, '-C', options.output, name]);
assets.push({ name: basename(archive), path: archive, bytes: statSync(archive).size, sha256: sha256(readFileSync(archive)) });
}
const sums = join(options.output, 'SHA256SUMS.txt');
writeFileSync(sums, assets.map((asset) => `${asset.sha256} ${asset.name}\n`).join(''));
assets.push({ name: 'SHA256SUMS.txt', path: sums, bytes: statSync(sums).size, sha256: sha256(readFileSync(sums)) });
// An empty Docker configuration proves the consumer can pull without publisher credentials.
const publicConfig = join(temporary, 'public-docker'); mkdirSync(publicConfig);
for (const [platform, images] of Object.entries(state.images)) {
for (const reference of Object.values(images)) {
console.log(`Verifying anonymous pull ${reference.split('@')[0]} (${platform})`);
await run('docker', ['--config', publicConfig, 'pull', '--platform', platform, reference], { timeout: 20 * 60_000 });
}
console.log(`Smoke checking published images (${platform})`);
await run('docker', ['run', '--rm', '--platform', platform, '--network', 'none', '--entrypoint', '/bin/sh', images.core, '-ec', 'test "$(pi --version)" = "$PI_VERSION"; tht --help >/dev/null; test -f /app/backend/dist/catalog/migrate.js; test -x /app/docker/workspace-maintenance-entrypoint.sh; test -f /app/docker/catalog-migrate.sh; test ! -e /run/secrets/thothii.secrets'], { timeout: 5 * 60_000 });
await run('docker', ['run', '--rm', '--platform', platform, '--network', 'none', '--entrypoint', '/usr/local/bin/frontend-config-smoke', images.frontend], { timeout: 60_000 });
}
state.assets = assets; save();
return assets;
}
const published = await publishVerifiedRelease({ hosting, prepare });
state.releaseURL = published.html_url; state.complete = true; save();
console.log(`Published and verified: ${published.html_url}`);
return published;
} finally {
if (worktree && existsSync(worktree)) await run('git', ['worktree', 'remove', '--force', worktree]).catch(() => {});
if (temporary && !existsSync(worktree ?? '')) rmSync(temporary, { recursive: true, force: true });
unlock();
}
}
if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1])).href) {
try { await publishInstallation(optionsFromArgs(process.argv.slice(2))); }
catch (error) { console.error(error instanceof SyntaxError ? 'Invalid release metadata; no secret values are printed.' : error.message); process.exitCode = 1; }
}
+48
View File
@@ -0,0 +1,48 @@
import { createHash } from "node:crypto";
import { mkdirSync, readFileSync, writeFileSync, lstatSync } from "node:fs";
import { dirname, join } from "node:path";
import { parse, stringify } from "yaml";
export const serviceRoles = Object.freeze({ core: "core", frontend: "frontend", "catalog-db": "catalog", "catalog-migrate": "core", "workspace-maintenance": "core", qdrant: "qdrant", embedding: "embedding", "embedding-model-init": "embedding" });
export const sha256 = (data) => createHash("sha256").update(data).digest("hex");
/** Only distribution assets enter the bundle: never a checkout, environment file or workspace. */
export function prepareBundle({ source, destination, platform, version, revision, images }) {
const compose = parse(readFileSync(join(source, "compose.yaml"), "utf8"));
if (Object.keys(compose.services).sort().join() !== Object.keys(serviceRoles).sort().join()) throw new Error("Release service contract changed; review packaging before publishing.");
for (const [name, role] of Object.entries(serviceRoles)) {
if (!/^docker\.io\/[a-z0-9][a-z0-9._/-]*@sha256:[a-f0-9]{64}$/.test(images[role] ?? "")) throw new Error("Release requires immutable Docker Hub image references.");
delete compose.services[name].build;
delete compose.services[name].pull_policy;
compose.services[name].image = images[role];
compose.services[name].platform = platform;
}
const files = {};
const write = (name, data) => {
mkdirSync(dirname(join(destination, name)), { recursive: true });
writeFileSync(join(destination, name), data);
files[name] = sha256(data);
};
write("compose.yaml", stringify(compose));
for (const name of ["deploy/compose.local.yaml", "deploy/compose.git-https.yaml", "deploy/compose.git-ssh.yaml", "docker/catalog-db-init.sql", "docker/embedding-model-init.sh"]) {
const path = join(source, name);
if (!lstatSync(path).isFile()) throw new Error("Release asset must be a regular tracked file.");
write(name, readFileSync(path));
}
// Any future source bind mount must be deliberately added to the asset allowlist.
for (const service of Object.values(compose.services)) {
for (const volume of service.volumes ?? []) {
const sourcePath = typeof volume === "string" ? volume.split(":")[0] : volume.type === "bind" ? volume.source : undefined;
if (sourcePath?.startsWith(".") && !files[sourcePath.replace(/^\.\//, "")]) throw new Error("Source bind mount is missing from the release bundle.");
}
}
const manifest = { schema_version: 1, version, revision, validator_protocol: 1,
requirements: { cpus: 2, memory_bytes: 4 * 2 ** 30, disk_bytes: 10 * 2 ** 30 },
components: ["pi", "catalog-migrations", "workspace-maintenance"],
images: Object.fromEntries(Object.entries(images).map(([role, reference]) => [role, { [platform]: reference }])),
files, compose: ["compose.yaml", "deploy/compose.local.yaml"] };
writeFileSync(join(destination, "release-manifest.json"), JSON.stringify(manifest, null, 2) + "\n");
writeFileSync(join(destination, "README.md"), `# ThothII ${version} — ${platform}\n\nSource / Sorgente: ${revision}\n\nThis prerelease provides images and document/preflight tools. Non-interactive execution and real-host acceptance are separate follow-up tickets; this is not a certified complete installation.\nQuesta prerelease fornisce immagini e strumenti di preparazione/preflight. Esecuzione non interattiva e collaudi reali sono incrementi successivi: non è ancora un'installazione completa certificata.\n\nUse bin/tht and its sibling bin/tht-workspace-documents together; no Node, Python, Bun or application checkout is required on the consumer host.\nWindows: use the Linux amd64 bundle inside Ubuntu WSL2, not a native Windows shell.\n\n1. bin/tht workspace prepare --directory NEW_WORKSPACE --id practice --name Practice\n2. bin/tht workspace validate --directory WORKSPACE\n3. bin/tht installation prepare --directory NEW_PRIVATE_INSTALLATION\n4. bin/tht installation preflight --directory INSTALLATION --release ABSOLUTE_RELEASE_DIR/release-manifest.json\n5. Complete the commented documents, generate technical credentials explicitly, then run installation validate and installation plan with --installation ABSOLUTE_INSTALLATION_FILE.\n\nImages are pinned by digest; runtime credentials and user workspaces are never bundled.\nConsult the accompanying IT/EN guides for prepared documents and mandatory runtime checks.\n`);
for (const [name, target] of [["standalone-manual-it.md", "GUIDE-IT.md"], ["standalone-manual-en.md", "GUIDE-EN.md"], ["installation-preflight.md", "PREFLIGHT.md"]]) writeFileSync(join(destination, target), readFileSync(join(source, "docs/install", name)));
return manifest;
}
+35
View File
@@ -0,0 +1,35 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, readFileSync, rmSync, existsSync } from "node:fs";
import { tmpdir } from "node:os";
import { resolve, join } from "node:path";
import { parse } from "yaml";
import { prepareBundle } from "./release-bundle.mjs";
test("consumer bundle pins every service and carries all source bind resources", () => {
const output = mkdtempSync(join(tmpdir(), "thoth-release-test-"));
const images = Object.fromEntries(["core", "frontend", "catalog", "qdrant", "embedding"].map((role) => [role, `docker.io/tylconsulting/${role}@sha256:${"a".repeat(64)}`]));
try {
prepareBundle({ source: resolve(import.meta.dirname, "../.."), destination: output, platform: "linux/amd64", version: "0.1.0-install-preview.1", revision: "b".repeat(40), images });
const manifest = JSON.parse(readFileSync(join(output, "release-manifest.json")));
const compose = parse(readFileSync(join(output, "compose.yaml"), "utf8"));
assert.equal(compose.services.core.image, images.core);
assert.equal(compose.services["catalog-migrate"].image, images.core);
assert.equal(compose.services["workspace-maintenance"].image, images.core);
assert.equal(compose.services["embedding-model-init"].image, images.embedding);
for (const service of Object.values(compose.services)) {
assert.equal(service.build, undefined);
assert.equal(service.pull_policy, undefined);
assert.equal(service.platform, "linux/amd64");
for (const volume of service.volumes ?? []) {
if (typeof volume === "string" && volume.startsWith("./")) assert.ok(existsSync(join(output, volume.split(":")[0])));
}
}
assert.equal(manifest.validator_protocol, 1);
assert.deepEqual(manifest.compose, ["compose.yaml", "deploy/compose.local.yaml"]);
assert.ok(manifest.files["docker/catalog-db-init.sql"]);
assert.ok(manifest.files["docker/embedding-model-init.sh"]);
assert.ok(!existsSync(join(output, "backend")));
assert.ok(!existsSync(join(output, "harness")));
} finally { rmSync(output, { recursive: true, force: true }); }
});
+69
View File
@@ -0,0 +1,69 @@
import { readFileSync } from "node:fs";
import { boundedFetch, readLimited } from "./release-registry.mjs";
import { sha256 } from "./release-bundle.mjs";
export async function giteaHosting({ run, repository, identity, body }) {
const remote = new URL(repository);
const credential = await run("git", ["credential", "fill"], { input: `protocol=https\nhost=${remote.host}\n\n`, quiet: true, env: { ...process.env, GIT_TERMINAL_PROMPT: "0" } });
const fields = Object.fromEntries(credential.trim().split("\n").map((line) => { const at = line.indexOf("="); return [line.slice(0, at), line.slice(at + 1)]; }));
if (!fields.username || !fields.password) throw new Error("Gitea publishing credentials are unavailable in the Git credential store.");
const authorization = `Basic ${Buffer.from(`${fields.username}:${fields.password}`).toString("base64")}`;
const api = `${remote.origin}/api/v1/repos${remote.pathname.replace(/\.git$/, "")}`;
const marker = `<!-- thothii-release:${JSON.stringify(identity)} -->`;
const tag = `installation-v${identity.version}`;
async function request(path, options = {}, allowMissing = false) {
const response = await boundedFetch(api + path, { ...options, headers: { Authorization: authorization, ...options.headers } }, 120_000);
if (allowMissing && response.status === 404) return null;
if (!response.ok) throw new Error(`Gitea release operation failed (HTTP ${response.status}).`);
const bytes = await readLimited(response, 4 * 2 ** 20);
try { return JSON.parse(bytes.toString()); } catch { throw new Error("Gitea returned invalid release metadata."); }
}
const json = (method, value) => ({ method, headers: { "Content-Type": "application/json" }, body: JSON.stringify(value) });
async function verifyTag(required = false) {
const existing = await request(`/tags/${encodeURIComponent(tag)}`, {}, true);
if ((!existing && required) || (existing && existing.commit?.sha !== identity.revision)) throw new Error("Release Git tag does not match the requested source revision.");
}
async function assets(release) { return request(`/releases/${release.id}/assets`); }
async function verifyAsset(asset, expected, publicRead) {
const url = new URL(asset.browser_download_url);
if (url.origin !== remote.origin) throw new Error("Unexpected release asset origin.");
const response = await boundedFetch(url, { headers: publicRead ? {} : { Authorization: authorization } }, 120_000);
if (!response.ok || sha256(await readLimited(response, expected.bytes + 1)) !== expected.sha256) throw new Error("Published release asset does not match its verified checksum.");
}
return {
async open() {
const info = await request("");
if (info.private || !info.permissions?.push) throw new Error("Release hosting must be a public Gitea repository with publication rights.");
await verifyTag();
const existing = await request(`/releases/tags/${encodeURIComponent(tag)}`, {}, true);
if (existing) {
if (!existing.body?.includes(marker)) throw new Error("Release version already belongs to another source or publication identity; it will not be overwritten.");
return existing;
}
return request("/releases", json("POST", { tag_name: tag, target_commitish: identity.revision, name: `ThothII ${identity.version}`, body: `${body}\n\n${marker}`, draft: true, prerelease: true }));
},
async upload(release, asset) {
const matching = (await assets(release)).filter((item) => item.name === asset.name);
if (matching.length > 1) throw new Error("Ambiguous release assets; no published files were replaced.");
if (matching.length === 1) { await verifyAsset(matching[0], asset, false); return; }
const data = readFileSync(asset.path);
if (sha256(data) !== asset.sha256) throw new Error("Local release asset changed before upload.");
const form = new FormData(); form.append("attachment", new Blob([data]), asset.name);
await request(`/releases/${release.id}/assets?name=${encodeURIComponent(asset.name)}`, { method: "POST", body: form });
},
async verify(release, expected, publicRead) {
await verifyTag(publicRead);
const uploaded = await assets(release);
if (uploaded.length !== expected.length) throw new Error("Release asset set is incomplete or contains unexpected files.");
for (const asset of expected) {
const matching = uploaded.filter((item) => item.name === asset.name);
if (matching.length !== 1) throw new Error("Release asset missing or ambiguous.");
await verifyAsset(matching[0], asset, publicRead);
}
},
async publish(release) {
await verifyTag();
return request(`/releases/${release.id}`, json("PATCH", { draft: false }));
},
};
}
+43
View File
@@ -0,0 +1,43 @@
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { giteaHosting } from './release-hosting.mjs';
test('release hosting refuses a pre-existing Git tag on another commit', async () => {
const fetch = globalThis.fetch;
const revision = 'a'.repeat(40);
let writes = 0;
globalThis.fetch = async (url, options) => {
if (options.method) { writes++; return Response.json({ id: 1, draft: true }); }
if (String(url).includes('/releases/tags/')) return new Response('', { status: 404 });
if (String(url).includes('/tags/installation-v')) return Response.json({ commit: { sha: 'b'.repeat(40) } });
return Response.json({ private: false, permissions: { push: true } });
};
try {
const hosting = await giteaHosting({ run: async () => 'username=test\npassword=test\n', repository: 'https://example.test/owner/repo', identity: { revision, version: '1.0.0' }, body: '' });
await assert.rejects(hosting.open(), /tag.*revision/i);
assert.equal(writes, 0);
} finally { globalThis.fetch = fetch; }
});
test('matching Git tag is accepted and verified again before publishing', async () => {
const fetch = globalThis.fetch;
const identity = { revision: 'a'.repeat(40), version: '1.0.0' };
let sha = identity.revision;
let writes = 0;
globalThis.fetch = async (url, options) => {
if (options.method) { writes++; return Response.json({ id: 1, draft: false }); }
if (String(url).includes('/releases/tags/')) return Response.json({ id: 1, draft: true, body: `<!-- thothii-release:${JSON.stringify(identity)} -->` });
if (String(url).includes('/tags/')) return Response.json({ commit: { sha } });
if (String(url).endsWith('/assets')) return Response.json([]);
return Response.json({ private: false, permissions: { push: true } });
};
try {
const hosting = await giteaHosting({ run: async () => 'username=test\npassword=test\n', repository: 'https://example.test/owner/repo', identity, body: '' });
const release = await hosting.open();
await hosting.verify(release, [], false);
sha = 'b'.repeat(40);
await assert.rejects(hosting.verify(release, [], false), /tag.*revision/i);
await assert.rejects(hosting.publish(release), /tag.*revision/i);
assert.equal(writes, 0);
} finally { globalThis.fetch = fetch; }
});
+18
View File
@@ -0,0 +1,18 @@
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { commandRunner } from './publish-installation.mjs';
test('publisher timeout terminates descendants that retain output pipes', async () => {
const logs = mkdtempSync(join(tmpdir(), 'release-process-test-'));
try {
const started = Date.now();
await assert.rejects(commandRunner(logs)(process.execPath, ['-e', `
require('child_process').spawn(process.execPath, ['-e', 'setTimeout(() => {}, 2500)'], {stdio: 'inherit'});
setTimeout(() => {}, 2500);
`], { timeout: 250, quiet: true }));
assert.ok(Date.now() - started < 1800, 'timeout must not wait for the descendant to exit naturally');
} finally { rmSync(logs, { recursive: true, force: true }); }
});
+14
View File
@@ -0,0 +1,14 @@
/** Drafts are the publication boundary. A partial build/upload is never a consumer release. */
export async function publishVerifiedRelease({ hosting, prepare }) {
const release = await hosting.open();
const assets = await prepare();
if (!release.draft) {
await hosting.verify(release, assets, true);
return release;
}
for (const asset of assets) await hosting.upload(release, asset);
await hosting.verify(release, assets, false);
const published = await hosting.publish(release);
await hosting.verify(published, assets, true);
return published;
}
@@ -0,0 +1,34 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { publishVerifiedRelease } from "./release-publication.mjs";
test("an interrupted preparation stays draft and retry publishes only after every artifact verifies", async () => {
const events = [];
let failing = true;
const hosting = {
open: async () => ({ id: 7, draft: true }),
upload: async (_draft, asset) => events.push(`upload:${asset.name}`),
verify: async (_draft, assets, publicRead) => events.push(`verify:${publicRead}:${assets.length}`),
publish: async () => { events.push("publish"); return { html_url: "https://example.test/release" }; },
};
const prepare = async () => {
if (failing) throw new Error("frontend build unavailable");
return [{ name: "bundle.tar.gz" }, { name: "SHA256SUMS" }];
};
await assert.rejects(publishVerifiedRelease({ hosting, prepare }), /frontend/);
assert.deepEqual(events, []);
failing = false;
await publishVerifiedRelease({ hosting, prepare });
assert.deepEqual(events, ["upload:bundle.tar.gz", "upload:SHA256SUMS", "verify:false:2", "publish", "verify:true:2"]);
});
test("a published version is verified without replacing any asset", async () => {
const events = [];
await publishVerifiedRelease({ prepare: async () => [{ name: "bundle.tar.gz" }], hosting: {
open: async () => ({ id: 7, draft: false }),
upload: async () => { throw new Error("overwrote a published version"); },
publish: async () => { throw new Error("republished a version"); },
verify: async (_release, _assets, publicRead) => events.push(publicRead),
} });
assert.deepEqual(events, [true]);
});
+86
View File
@@ -0,0 +1,86 @@
import { readFileSync } from "node:fs";
import { homedir } from "node:os";
import { join } from "node:path";
import { sha256 } from "./release-bundle.mjs";
const accept = "application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json";
export async function boundedFetch(url, options = {}, timeout = 30_000) {
try { return await fetch(url, { ...options, redirect: options.redirect ?? "error", signal: AbortSignal.timeout(timeout) }); }
catch { throw new Error("Release network request failed; retry with the same output directory."); }
}
export async function readLimited(response, maximum) {
const chunks = []; let size = 0;
for await (const chunk of response.body) { size += chunk.length; if (size > maximum) throw new Error("Release response exceeded its bound."); chunks.push(chunk); }
return Buffer.concat(chunks);
}
async function jsonResponse(response, description) {
if (!response.ok) throw new Error(`${description} refused (HTTP ${response.status}).`);
const bytes = await readLimited(response, 4 * 2 ** 20);
try { return { bytes, value: JSON.parse(bytes.toString()) }; }
catch { throw new Error("Release service returned invalid metadata."); }
}
export async function dockerCredentials(run) {
const config = JSON.parse(readFileSync(join(process.env.DOCKER_CONFIG || join(homedir(), ".docker"), "config.json"), "utf8"));
const server = "https://index.docker.io/v1/";
const helper = config.credHelpers?.[server] || config.credsStore;
if (!helper || !/^[A-Za-z0-9._-]+$/.test(helper)) throw new Error("Use docker login with an OS credential store before publishing.");
const auth = JSON.parse(await run(`docker-credential-${helper}`, ["get"], { input: server + "\n", quiet: true }));
if (!auth.Username || !auth.Secret) throw new Error("Docker Hub login is unavailable.");
return auth;
}
export function dockerHub(auth) {
async function token(repository, anonymous) {
const url = new URL("https://auth.docker.io/token");
url.searchParams.set("service", "registry.docker.io");
url.searchParams.set("scope", `repository:${repository}:pull`);
const response = await boundedFetch(url, { headers: anonymous ? {} : { Authorization: `Basic ${Buffer.from(`${auth.Username}:${auth.Secret}`).toString("base64")}` } });
return (await jsonResponse(response, "Registry authentication")).value.token;
}
async function registryJSON(repository, route, anonymous, allowMissing = false) {
const bearer = await token(repository, anonymous);
let response = await boundedFetch(`https://registry-1.docker.io/v2/${repository}/${route}`, { redirect: "manual", headers: { Accept: accept, Authorization: `Bearer ${bearer}` } });
if ([302, 307].includes(response.status) && route.startsWith("blobs/")) {
const target = new URL(response.headers.get("location"));
if (target.protocol !== "https:" || target.username || target.password) throw new Error("Invalid registry blob redirect.");
// Signed blob URLs are fetched without forwarding registry credentials.
response = await boundedFetch(target);
}
if (allowMissing && response.status === 404) return null;
const { bytes, value } = await jsonResponse(response, "Registry read");
const digest = `sha256:${sha256(bytes)}`;
const advertised = response.headers.get("docker-content-digest");
if (advertised && advertised !== digest) throw new Error("Registry content digest mismatch.");
return { value, digest };
}
return {
async ensurePublic(namespace, name) {
const response = await boundedFetch("https://hub.docker.com/v2/auth/token", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ identifier: auth.Username, secret: auth.Secret }) });
const bearer = (await jsonResponse(response, "Docker Hub authentication")).value.access_token;
const headers = { Authorization: `Bearer ${bearer}`, "Content-Type": "application/json" };
const path = `https://hub.docker.com/v2/namespaces/${namespace}/repositories`;
let existing = await boundedFetch(`${path}/${name}`, { headers });
if (existing.status === 404) {
existing = await boundedFetch(path, { method: "POST", headers, body: JSON.stringify({ namespace, name, registry: "docker.io", is_private: false, description: `ThothII ${name.endsWith("core") ? "core with embedded Pi" : "standalone frontend"}` }) });
}
const data = (await jsonResponse(existing, "Public repository preparation")).value;
if (data.is_private !== false) throw new Error("Selected Docker Hub repository is private; make this release repository public before retrying.");
},
async inspect(repository, reference, platform, { anonymous = false, allowMissing = false } = {}) {
let image = await registryJSON(repository, `manifests/${reference}`, anonymous, allowMissing);
if (!image) return null;
if (reference.startsWith("sha256:") && image.digest !== reference) throw new Error("Requested image digest does not match registry content.");
if (image.value.manifests) {
const match = image.value.manifests.find((entry) => `${entry.platform?.os}/${entry.platform?.architecture}` === platform);
if (!match || !/^sha256:[a-f0-9]{64}$/.test(match.digest)) throw new Error("Image does not contain the requested platform.");
image = await registryJSON(repository, `manifests/${match.digest}`, anonymous);
if (image.digest !== match.digest) throw new Error("Image index digest mismatch.");
}
if (reference.startsWith("sha256:") && !image.value.config) throw new Error("Image metadata is incomplete.");
const configDigest = image.value.config?.digest;
if (!/^sha256:[a-f0-9]{64}$/.test(configDigest ?? "")) throw new Error("Image configuration digest is invalid.");
const config = await registryJSON(repository, `blobs/${configDigest}`, anonymous);
if (config.digest !== configDigest || `${config.value.os}/${config.value.architecture}` !== platform) throw new Error("Image configuration or platform mismatch.");
return { reference: `docker.io/${repository}@${image.digest}`, labels: config.value.config?.Labels ?? {} };
},
};
}
+28
View File
@@ -0,0 +1,28 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { dockerHub } from "./release-registry.mjs";
import { sha256 } from "./release-bundle.mjs";
test("registry verifies pinned config/platform and does not forward auth to blob storage", async () => {
const original = globalThis.fetch;
const config = JSON.stringify({ os: "linux", architecture: "amd64", config: { Labels: { "org.opencontainers.image.revision": "b".repeat(40) } } });
const configDigest = "sha256:" + sha256(config);
const manifest = JSON.stringify({ schemaVersion: 2, config: { digest: configDigest } });
const imageDigest = "sha256:" + sha256(manifest);
const auth = [];
globalThis.fetch = async (target, options) => {
const url = new URL(target);
if (url.hostname === "auth.docker.io") return new Response(JSON.stringify({ token: "registry-token" }));
if (url.hostname === "blob.example.test") { auth.push(options.headers?.Authorization); return new Response(config); }
if (url.pathname.includes("/blobs/")) return new Response(null, { status: 307, headers: { location: "https://blob.example.test/config" } });
return new Response(manifest, { headers: { "docker-content-digest": imageDigest } });
};
try {
const registry = dockerHub({ Username: "publisher", Secret: "PRIVATE_TOKEN" });
const image = await registry.inspect("example/core", imageDigest, "linux/amd64", { anonymous: true });
assert.equal(image.reference, `docker.io/example/core@${imageDigest}`);
assert.deepEqual(auth, [undefined]);
await assert.rejects(registry.inspect("example/core", imageDigest, "linux/arm64"), /platform mismatch/);
await assert.rejects(registry.inspect("example/core", "sha256:" + "0".repeat(64), "linux/amd64"), /Requested image digest/);
} finally { globalThis.fetch = original; }
});
+21
View File
@@ -0,0 +1,21 @@
import { readFileSync, lstatSync } from "node:fs";
import { parseAllDocuments } from "yaml";
import { decode, DocumentError, runWorkspaceDocuments } from "../workspaces/documents.js";
import { validateDatabaseBootstrap } from "./bootstrap-documents.js";
/** Internal sibling protocol: only references cross back to Go, never secret contents. */
export function runBootstrapValidation(args: string[]): { status: number; output: string } {
try {
if (args.length !== 5 || args[0] !== "--directory" || args[2] !== "--bootstrap" || args[4] !== "--json") throw new Error("usage");
const checked = runWorkspaceDocuments(["validate", "--directory", args[1], "--json"]);
if (checked.status !== 0) return checked;
const info = lstatSync(args[3]);
if (!info.isFile() || info.isSymbolicLink() || info.size > 1024 * 1024) throw new Error("file");
const validated = decode(readFileSync(args[3], "utf8"), "database-bootstrap.yaml", (source) =>
validateDatabaseBootstrap(parseAllDocuments(source)[0].toJSON(), args[1]), "database bootstrap schema v1 and the Catalog binding contract");
return { status: 0, output: JSON.stringify({ schema_version: 1, ok: true, secret_files: validated.secretFiles, warnings: validated.warnings, issues: [] }) };
} catch (error) {
const issue = error instanceof DocumentError ? error.issue : { document: "database-bootstrap.yaml", field: "$", code: "bootstrap_invalid", correction: "Supply one complete Catalog database configuration per workspace in a readable local bootstrap document." };
return { status: 1, output: JSON.stringify({ schema_version: 1, ok: false, issues: [issue] }) };
}
}
@@ -0,0 +1,49 @@
import { readFileSync } from "node:fs";
import { join } from "node:path";
import { z } from "zod";
import { databaseConfigurationSchema } from "./configuration-schema.js";
import { parseWorkspaceCatalogYaml } from "../workspaces/catalog.js";
import { parseWorkspaceYaml } from "../workspaces/schema.js";
import { discoverWorkspaceSecretRequirements } from "../workspaces/secret-requirements.js";
const reference = z.string().min(1).max(4096);
const database = databaseConfigurationSchema.extend({
secretFiles: z.object({ password: reference.optional(), apiKey: reference.optional(), sshPrivateKey: reference.optional(), sshPrivateKeyPassphrase: reference.optional(), sshKnownHosts: reference.optional(), tlsCa: reference.optional() }).strict(),
evidenceSecretFiles: z.object({ "evidence.signed_urls": reference.optional(), "evidence.access_key": reference.optional(), "evidence.secret_key": reference.optional(), "evidence.session_token": reference.optional() }).strict().optional(),
});
const bootstrap = z.object({ schemaVersion: z.literal(1), databases: z.array(database).min(1).max(1000) }).strict();
export const parseDatabaseBootstrap = (value: unknown) => bootstrap.parse(value);
export interface BootstrapReference { field: string; path: string }
/** Offline bootstrap boundary: runtime Catalog owns the resulting bindings after import. */
export function validateDatabaseBootstrap(value: unknown, workspaceRoot: string): { secretFiles: BootstrapReference[]; warnings: string[] } {
const document = bootstrap.parse(value);
const catalog = parseWorkspaceCatalogYaml(readFileSync(join(workspaceRoot, "thoth-workspaces.yaml"), "utf8"));
const expected = new Set(catalog.workspaces.map((entry) => entry.id));
const seen = new Set<string>();
const secretFiles: BootstrapReference[] = [];
const warnings: string[] = [];
const issue = (path: (string | number)[], message: string): never => { throw new z.ZodError([{ code: "custom", path, message }]); };
document.databases.forEach((entry, index) => {
if (!expected.has(entry.workspaceId) || seen.has(entry.workspaceId)) issue(["databases", index, "workspaceId"], "Declare each catalog workspace exactly once.");
seen.add(entry.workspaceId);
const required = entry.binding.transport === "rest_api"
? entry.binding.restAuth === "none" ? [] : ["apiKey"] as const
: entry.binding.transport === "ssh_tunnel" ? ["password", "sshPrivateKey", "sshKnownHosts"] as const : ["password"] as const;
for (const name of required) {
if (!entry.secretFiles[name]) issue(["databases", index, "secretFiles", name], "Supply a protected file reference for this transport.");
}
if (entry.binding.transport === "ssh_tunnel") warnings.push(`databases.${index}:ssh_tunnel supports Catalog diagnostics, not NL-to-SQL sessions; choose direct or REST for practice.`);
for (const [name, path] of Object.entries(entry.secretFiles)) secretFiles.push({ field: `databases.${index}.secretFiles.${name}`, path });
const workspace = parseWorkspaceYaml(readFileSync(join(workspaceRoot, entry.workspaceId, "workspace.yaml"), "utf8"));
const requirements = discoverWorkspaceSecretRequirements(workspace, {});
for (const requirement of requirements.filter((item) => item.connector === "evidence" && item.required)) {
if (!(entry.evidenceSecretFiles as Record<string, string> | undefined)?.[requirement.id]) issue(["databases", index, "evidenceSecretFiles"], "Supply the configured Evidence authentication file references.");
}
for (const [name, path] of Object.entries(entry.evidenceSecretFiles ?? {})) secretFiles.push({ field: `databases.${index}.evidenceSecretFiles.${name}`, path });
});
if (seen.size !== expected.size) issue(["databases"], "Add a database binding for every catalog workspace.");
return { secretFiles, warnings };
}
@@ -0,0 +1,47 @@
import { readFileSync } from "node:fs";
import { join } from "node:path";
import { S3Client, ListObjectsV2Command } from "@aws-sdk/client-s3";
import { parseWorkspaceYaml } from "../workspaces/schema.js";
import { evidencePolicy } from "../workspaces/evidence/preprocessing.js";
import type { parseDatabaseBootstrap } from "./bootstrap-documents.js";
import { probePublicEvidenceUrl, publicEvidenceAgent } from "./evidence-probe-http.js";
type Entry = ReturnType<typeof parseDatabaseBootstrap>["databases"][number];
/** Read-only availability probes; domain correctness and materialization remain runtime gates. */
export async function probeEvidence(entry: Entry, root: string): Promise<void> {
const evidence = parseWorkspaceYaml(readFileSync(join(root, entry.workspaceId, "workspace.yaml"), "utf8")).evidence;
if (!evidence || evidence.source.type === "filesystem") return;
if (evidencePolicy(evidence)) throw new Error("Evidence egress policy refused");
const secret = (name: keyof NonNullable<Entry["evidenceSecretFiles"]>) => {
const path = entry.evidenceSecretFiles?.[name];
if (!path) throw new Error("Evidence credential missing");
return readFileSync(path, "utf8").trim();
};
const source = evidence.source;
if (source.type === "http") {
const urls: unknown = source.authentication === "signed_urls_file"
? JSON.parse(secret("evidence.signed_urls")) : source.uris;
if (!Array.isArray(urls) || urls.length !== source.uris.length || urls.length > 1000) throw new Error("Invalid signed URLs");
for (const [index, value] of urls.entries()) {
if (typeof value !== "string") throw new Error("Invalid signed URL");
const url = new URL(value);
const provenance = new URL(source.uris[index]);
// Signed queries may authorize the same identity, never a different host/path.
if (url.origin !== provenance.origin || url.pathname !== provenance.pathname || url.username || url.password || url.hash) throw new Error("Invalid signed URL identity");
await probePublicEvidenceUrl(url);
}
return;
}
// The shared runtime policy currently permits trusted AWS endpoints with explicit file credentials.
const location = new URL(source.uri);
const client = new S3Client({
region: source.region ?? "us-east-1", maxAttempts: 1,
requestHandler: { httpsAgent: publicEvidenceAgent(), connectionTimeout: 5_000, requestTimeout: 5_000 },
credentials: { accessKeyId: secret("evidence.access_key"), secretAccessKey: secret("evidence.secret_key"),
...(entry.evidenceSecretFiles?.["evidence.session_token"] ? { sessionToken: secret("evidence.session_token") } : {}) },
});
try {
await client.send(new ListObjectsV2Command({ Bucket: location.hostname, Prefix: decodeURIComponent(location.pathname.slice(1)), MaxKeys: 1 }), { abortSignal: AbortSignal.timeout(5_000) });
} finally { client.destroy(); }
}
+53
View File
@@ -0,0 +1,53 @@
import { readFileSync } from "node:fs";
import { parse } from "yaml";
import { parseDatabaseBootstrap } from "./bootstrap-documents.js";
import { runBootstrapValidation } from "./bootstrap-cli.js";
import { createConcreteDiagnosticAdapters, type DiagnosticAdapters } from "../workspaces/diagnostics.js";
import { probeEvidence } from "./bootstrap-evidence-probes.js";
interface ProbeCheck { id: string; outcome: "passed" | "error"; field: string; action: string }
/** Uses the same read-only, authenticated connector diagnostics as the Catalog. */
export async function probeBootstrapDependencies(value: unknown, adapters: DiagnosticAdapters = createConcreteDiagnosticAdapters(), workspaceRoot?: string) {
const document = parseDatabaseBootstrap(value);
const checks: ProbeCheck[] = [];
for (const [index, entry] of document.databases.entries()) {
let outcome: ProbeCheck["outcome"] = "passed";
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), 5_000);
try {
if (entry.binding.transport === "ssh_tunnel") throw new Error("session transport unavailable");
await adapters.probeConnector({
role: "dwh", transport: entry.binding.transport,
host: entry.binding.host, port: entry.binding.port, user: entry.binding.username,
baseUrl: entry.binding.baseUrl,
credentialFile: entry.binding.transport === "rest_api" ? entry.secretFiles.apiKey : entry.secretFiles.password,
tlsCaFile: entry.secretFiles.tlsCa, tlsServername: entry.binding.tlsServername,
resource: { database: entry.databaseName, schema: entry.schema },
timeoutMs: 5_000, signal: controller.signal,
diagnostic: { method: "GET", path: entry.binding.restPath ?? "/health", auth: entry.binding.restAuth ?? "bearer" },
});
} catch { outcome = "error"; } finally { clearTimeout(timer); }
checks.push({ id: `database-${index}`, outcome, field: `database-bootstrap.databases.${index}`,
action: entry.binding.transport === "ssh_tunnel"
? "Choose postgres_direct or rest_api for NL-to-SQL practice; SSH diagnostics alone cannot establish session readiness."
: "Require an authenticated read-only connection and access to the configured database/schema; correct endpoint, permissions or protected credentials." });
if (workspaceRoot) {
let evidenceOutcome: ProbeCheck["outcome"] = "passed";
try { await probeEvidence(entry, workspaceRoot); } catch { evidenceOutcome = "error"; }
checks.push({ id: `evidence-${index}`, outcome: evidenceOutcome, field: `workspaces.${index}.evidence`, action: "Require readable local Evidence or authenticated bounded HTTP/S3 access under the canonical egress policy; domain meaning is verified during practice." });
}
}
return { schema_version: 1, ok: checks.every((check) => check.outcome === "passed"), checks };
}
export async function runBootstrapProbes(args: string[]) {
const validation = runBootstrapValidation(args);
if (validation.status !== 0) return validation;
try {
const report = await probeBootstrapDependencies(parse(readFileSync(args[3], "utf8")), undefined, args[1]);
return { status: report.ok ? 0 : 1, output: JSON.stringify(report) };
} catch {
return { status: 1, output: JSON.stringify({ schema_version: 1, ok: false, checks: [{ id: "database-probes", outcome: "error", field: "database-bootstrap", action: "Revalidate prepared documents and protected credential references." }] }) };
}
}
@@ -0,0 +1,44 @@
import { z } from "zod";
import { parseCredentialFreeHttpUrl } from "../auth/url-policy.js";
import { DATABASE_TRANSPORTS } from "./types.js";
const workspaceIdSchema = z.string().regex(/^[a-z][a-z0-9-]{2,62}$/);
const identifier = z.string().trim().min(1).max(128).regex(/^[A-Za-z_][A-Za-z0-9_$-]*$/);
const nonEmpty = z.string().trim().min(1).max(512);
const port = z.number().int().min(1).max(65_535);
const optionalText = nonEmpty.optional();
const sshHost = z.string().trim().min(1).max(255).regex(/^[A-Za-z0-9_.:\[\]-]+$/).optional();
const sshUsername = z.string().trim().min(1).max(128).regex(/^[A-Za-z0-9._-]+$/).optional();
const bindingSchema = z.object({
transport: z.enum(DATABASE_TRANSPORTS),
host: optionalText,
port: port.optional(),
username: optionalText,
baseUrl: z.string().max(2048)
.refine((value) => parseCredentialFreeHttpUrl(value) !== undefined)
.optional(),
restPath: z.string().regex(/^\/(?!\/)[^?#\\\u0000-\u001f]*$/).max(512).optional(),
restAuth: z.enum(["none", "bearer", "x-api-key"]).optional(),
tlsServername: optionalText,
sshHost,
sshPort: port.optional(),
sshUsername,
sshTargetHost: sshHost,
sshTargetPort: port.optional(),
}).strict().superRefine((binding, context) => {
const required = binding.transport === "postgres_direct"
? ["host", "port", "username"] as const
: binding.transport === "rest_api"
? ["baseUrl", "restPath", "restAuth"] as const
: ["username", "sshHost", "sshPort", "sshUsername", "sshTargetHost", "sshTargetPort"] as const;
for (const field of required) {
if (binding[field] === undefined) context.addIssue({ code: "custom", path: [field], message: "Required" });
}
});
export const databaseConfigurationSchema = z.object({
workspaceId: workspaceIdSchema,
engine: z.literal("postgres"),
databaseName: identifier,
schema: identifier,
binding: bindingSchema,
}).strict();
@@ -0,0 +1,57 @@
import { lookup } from "node:dns/promises";
import { request as httpRequest } from "node:http";
import { request as httpsRequest, Agent } from "node:https";
import type { LookupFunction } from "node:net";
import ipaddr from "ipaddr.js";
const refused = () => new Error("Evidence network policy refused");
function normalizedPublicAddress(value: string): string {
const address = ipaddr.process(value);
if (address.range() !== "unicast") throw refused();
return address.toString();
}
/** Reject the entire DNS answer set, then pin the connection to that verified set. */
export async function resolvePublicEvidenceHost(hostname: string) {
const values = await lookup(hostname.replace(/^\[|\]$/g, ""), { all: true });
if (!values.length) throw refused();
values.forEach((value) => normalizedPublicAddress(value.address));
return values;
}
const publicLookup: LookupFunction = (hostname, options, callback) => {
void resolvePublicEvidenceHost(hostname).then((values) => {
if (options.all) callback(null, values);
else callback(null, values[0].address, values[0].family);
}, () => callback(refused(), "", 0));
};
// Node's direct agent does not inherit HTTP proxy environment or ambient credentials.
export const publicEvidenceAgent = () => new Agent({ lookup: publicLookup });
export async function probePublicEvidenceUrl(url: URL): Promise<void> {
if (!['http:', 'https:'].includes(url.protocol)) throw refused();
const signal = AbortSignal.timeout(5_000);
const values = await Promise.race([
resolvePublicEvidenceHost(url.hostname),
new Promise<never>((_, reject) => signal.addEventListener("abort", () => reject(refused()), { once: true })),
]);
signal.throwIfAborted();
const allowed = new Set(values.map((value) => normalizedPublicAddress(value.address)));
const pinned: LookupFunction = (_hostname, options, callback) => {
if (options.all) callback(null, values);
else callback(null, values[0].address, values[0].family);
};
await new Promise<void>((resolve, reject) => {
const request = (url.protocol === "https:" ? httpsRequest : httpRequest)(url, {
method: "GET", lookup: pinned, signal, agent: false,
}, (response) => {
try {
const peer = response.socket.remoteAddress;
if (!peer || !allowed.has(normalizedPublicAddress(peer)) || !response.statusCode || response.statusCode < 200 || response.statusCode >= 300) throw refused();
resolve();
} catch { reject(refused()); } finally { response.destroy(); }
});
request.on("error", () => reject(refused()));
request.end();
});
}
+4 -1
View File
@@ -41,8 +41,11 @@ const runtimeModelSchema = z.object({
supportsReasoningEffort: z.boolean(),
supportsStore: z.boolean(),
maxTokensField: z.string().optional(),
thinkingFormat: z.enum(["qwen", "qwen-chat-template"]).optional(),
}).strict().optional(),
}).strict().optional(),
}).strict().refine((session) => !session.compatibility?.thinkingFormat || session.reasoning, {
message: "thinkingFormat requires reasoning: true",
}).optional(),
metadataGeneration: z.object({ disableThinking: z.boolean() }).strict().optional(),
}).strict();
+121 -1
View File
@@ -16,12 +16,16 @@ import { loadSettings } from "./settings/settings-store.js";
import { ThtRunner, type SessionRow } from "./tht/tht-runner.js";
import { WorkspaceRegistry } from "./workspaces/registry.js";
import { WorkspaceSecretStore } from "./workspaces/secret-store.js";
import { createProductionWorkspaceDiagnoser } from "./workspaces/diagnostics.js";
import { resolveCatalogRuntimeBinding } from "./catalog/runtime-binding.js";
import { CatalogService } from "./catalog/service.js";
import { validateOperationalWorkspace } from "./workspaces/schema.js";
import { createCatalogRepository } from "./catalog/repository.js";
import type { CatalogRepository } from "./catalog/types.js";
type OperatorAction = "maintenance-activate" | "maintenance-deactivate" | "maintenance-status"
| "session-inventory" | "workflow-doctor" | "workspace-integrity"
| "pi-test" | "effective-settings";
| "workspace-pull" | "workspace-test" | "pi-test" | "effective-settings";
const lifecyclePrincipal: PrincipalContext = {
issuer: "tht-operator-command",
@@ -119,6 +123,119 @@ async function workspaceIntegrity(config: AppConfig): Promise<{
return { ready: true, ...integrity };
}
async function workspacePull(config: AppConfig): Promise<{
ready: boolean;
status: "succeeded" | "degraded";
branch: string;
head?: string;
degraded: boolean;
}> {
const status = await new WorkspaceRegistry(config.workspaceRegistry).pull();
return {
ready: !status.degraded,
status: status.degraded ? "degraded" : "succeeded",
branch: status.branch,
...(status.head ? { head: status.head } : {}),
degraded: status.degraded,
};
}
interface WorkspaceTestReport {
id: string;
status: "ready" | "failed";
database: "reachable" | "not_configured" | "failed";
diagnostics: string[];
}
async function workspaceTest(config: AppConfig): Promise<{
ready: boolean;
workspaces: WorkspaceTestReport[];
}> {
if (!config.catalogDatabase) throw new Error("Catalog database is not configured");
const registry = new WorkspaceRegistry(config.workspaceRegistry);
const revisions = await registry.list();
const repository = createCatalogRepository(config.catalogDatabase);
try {
const secretStore = new WorkspaceSecretStore({
root: config.workspaceSecretStoreRoot,
runtimeRoot: config.workspaceSecretRuntimeRoot,
installationId: config.workspaceRegistry.installationId,
});
const catalogService = new CatalogService(
repository,
registry,
secretStore,
config.workspaceRegistry.secretRoots,
config.workspaceDiagnosticTimeoutMs,
);
const diagnose = createProductionWorkspaceDiagnoser(config.workspaceDiagnosticTimeoutMs, undefined, {
internalQdrantUrl: config.internalQdrantUrl,
internalEmbeddingUrl: config.internalEmbeddingUrl,
internalEmbeddingId: config.internalEmbeddingId,
internalEmbeddingModel: config.internalEmbeddingModel,
internalEmbeddingDimensions: config.internalEmbeddingDimensions,
});
const databases = await repository.list();
const reports: WorkspaceTestReport[] = [];
for (const revision of revisions) {
const diagnostics: string[] = [];
let workspace: ReturnType<typeof validateOperationalWorkspace>;
try {
workspace = validateOperationalWorkspace((await registry.read(revision.id)).workspace);
} catch {
reports.push({ id: revision.id, status: "failed", database: "not_configured", diagnostics: ["workspace_invalid"] });
continue;
}
const database = databases.find((candidate) => candidate.workspaceId === revision.id);
if (!database) {
reports.push({ id: revision.id, status: "failed", database: "not_configured", diagnostics: ["database_binding_missing"] });
continue;
}
let tested;
try {
tested = await catalogService.test(database);
} catch {
reports.push({ id: revision.id, status: "failed", database: "failed", diagnostics: ["database_unreachable"] });
continue;
}
if (!tested || tested.connectionStatus !== "reachable") {
reports.push({ id: revision.id, status: "failed", database: "failed", diagnostics: ["database_unreachable"] });
continue;
}
let lease: ReturnType<typeof resolveCatalogRuntimeBinding>;
try {
lease = resolveCatalogRuntimeBinding({
workspace,
database: tested,
environment: process.env,
secretRoots: config.workspaceRegistry.secretRoots,
secretStore,
});
} catch {
reports.push({ id: revision.id, status: "failed", database: "reachable", diagnostics: ["binding_missing"] });
continue;
}
try {
// CatalogService.test() above is the authoritative database probe and records its
// outcome. The remaining diagnoser pass checks Evidence and internal semantic services;
// skipping its legacy DWH probe avoids requiring a second response-shape contract for a
// REST health endpoint.
const result = await diagnose(lease.workspace, lease.bindings, { writeProbe: false, skipDwh: true });
diagnostics.push(...result.diagnostics.map((diagnostic) => diagnostic.code));
const ready = result.activatable;
reports.push({ id: revision.id, status: ready ? "ready" : "failed", database: "reachable", diagnostics });
} catch {
reports.push({ id: revision.id, status: "failed", database: "reachable", diagnostics: ["connector_unavailable"] });
} finally {
lease.release();
}
}
return { ready: reports.length > 0 && reports.every((report) => report.status === "ready"), workspaces: reports };
} finally {
await repository.close?.();
}
}
export async function runOperatorAction(
action: OperatorAction,
config: AppConfig,
@@ -132,6 +249,8 @@ export async function runOperatorAction(
if (action === "session-inventory") return await sessionInventory(config);
if (action === "workflow-doctor") return await workflowDiagnostics(config);
if (action === "workspace-integrity") return await workspaceIntegrity(config);
if (action === "workspace-pull") return await workspacePull(config);
if (action === "workspace-test") return await workspaceTest(config);
const modelCatalog = loadRuntimeModelCatalog(config.modelCatalogFile);
if (action === "effective-settings") {
return effectiveSettings(config, loadSettings(config), modelCatalog);
@@ -146,6 +265,7 @@ async function main(): Promise<void> {
if (!action || ![
"maintenance-activate", "maintenance-deactivate", "maintenance-status", "session-inventory",
"workflow-doctor", "workspace-integrity", "pi-test", "effective-settings",
"workspace-pull", "workspace-test",
].includes(action)) throw new Error("invalid operator action");
const result = await runOperatorAction(action, loadConfig(process.env));
process.stdout.write(`${JSON.stringify(result)}\n`);
+19 -6
View File
@@ -25,6 +25,7 @@ export interface SessionRuntime {
}
export interface RuntimeOptions {
interactionLanguage?: string | null;
provider?: string;
model?: string;
thinking?: string;
@@ -48,6 +49,7 @@ export class PiProcessManager {
private spawnFn: (
sessionId: string, author: string, provider: string | undefined, model: string | undefined,
principal?: PrincipalContext, runtimeConfigPath?: string,
interactionLanguage?: string | null,
) => ChildProcessWithoutNullStreams;
private loadAuthProviders: (agentDir: string) => ReadonlySet<string>;
private modelCatalog: RuntimeModelCatalog;
@@ -67,11 +69,11 @@ export class PiProcessManager {
this.loadAuthProviders = opts?.authProviders
?? ((agentDir) => loadPiAuthProviders({ agentDir }));
if (opts?.spawnFn) {
this.spawnFn = (sessionId, author, provider, model, principal, runtimeConfigPath) =>
this.spawnPi(opts.spawnFn!, sessionId, author, provider, model, principal, runtimeConfigPath);
this.spawnFn = (sessionId, author, provider, model, principal, runtimeConfigPath, language) =>
this.spawnPi(opts.spawnFn!, sessionId, author, provider, model, principal, runtimeConfigPath, language);
} else {
this.spawnFn = (sessionId, author, provider, model, principal, runtimeConfigPath) =>
this.spawnPi(nodeSpawn, sessionId, author, provider, model, principal, runtimeConfigPath);
this.spawnFn = (sessionId, author, provider, model, principal, runtimeConfigPath, language) =>
this.spawnPi(nodeSpawn, sessionId, author, provider, model, principal, runtimeConfigPath, language);
}
}
@@ -85,6 +87,7 @@ export class PiProcessManager {
private spawnPi(
spawnFn: SpawnFn, sessionId: string, author: string, provider: string | undefined,
model: string | undefined, principal?: PrincipalContext, runtimeConfigPath?: string,
interactionLanguage?: string | null,
): ChildProcessWithoutNullStreams {
// This is the final shared boundary for createFor(), spawnFor(), and resume(). Validate
// before auth-provider inspection, then make Pi consume the exact copied bytes rather than
@@ -110,6 +113,9 @@ export class PiProcessManager {
additions: { THT_SESSION: sessionId, THT_AUTHOR: author },
});
env.PI_CODING_AGENT_DIR = agent.agentDir;
// A launch hint only: the gate reads the authoritative manifest before each turn.
delete env.THT_INTERACTION_LANGUAGE;
if (interactionLanguage) env.THT_INTERACTION_LANGUAGE = interactionLanguage;
env.PI_CODING_AGENT_SESSION_DIR = agent.sessionDir;
clearPrincipalEnvironment(env);
if (principal) Object.assign(env, principalEnvironment(principal));
@@ -187,7 +193,9 @@ export class PiProcessManager {
const model = o.model ?? this.cfg.defaults.model;
let child: ChildProcessWithoutNullStreams;
try {
child = this.spawnFn(sessionId, author, provider, model, o.principal, o.runtimeConfig?.path);
child = this.spawnFn(
sessionId, author, provider, model, o.principal, o.runtimeConfig?.path, o.interactionLanguage,
);
} catch (error) {
o.runtimeConfig?.release();
throw error;
@@ -296,8 +304,13 @@ export class PiProcessManager {
}
async resume(sessionId: string, tht: ThtRunner): Promise<SessionRuntime> {
const manifest = await tht.sessionShow(sessionId) as { provider?: string; model?: string; thinking?: string } | null;
const manifest = await tht.sessionShow(sessionId) as {
provider?: string; model?: string; thinking?: string; interaction_language?: string | null;
} | null;
const language = manifest?.interaction_language
?? (await tht.ensureInteractionLanguage(sessionId)).interaction_language;
return this.spawnFor(sessionId, {
interactionLanguage: language,
provider: manifest?.provider,
model: manifest?.model,
thinking: manifest?.thinking,
+1 -42
View File
@@ -1,60 +1,19 @@
import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
import { z } from "zod";
import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
import { parseCredentialFreeHttpUrl } from "../auth/url-policy.js";
import { databaseConfigurationSchema as configSchema } from "../catalog/configuration-schema.js";
import { CatalogService, type CatalogSecretName } from "../catalog/service.js";
import { WorkspaceRegistryError } from "../workspaces/git-repository.js";
import {
CatalogConflictError,
CatalogOperationInProgressError,
CatalogUnavailableError,
DATABASE_TRANSPORTS,
type CatalogRepository,
type DatabaseConfigurationInput,
} from "../catalog/types.js";
import type { CatalogOperationCoordinator } from "../catalog/operation-coordinator.js";
const idSchema = z.uuid();
const workspaceIdSchema = z.string().regex(/^[a-z][a-z0-9-]{2,62}$/);
const identifier = z.string().trim().min(1).max(128).regex(/^[A-Za-z_][A-Za-z0-9_$-]*$/);
const nonEmpty = z.string().trim().min(1).max(512);
const port = z.number().int().min(1).max(65_535);
const optionalText = nonEmpty.optional();
const sshHost = z.string().trim().min(1).max(255).regex(/^[A-Za-z0-9_.:\[\]-]+$/).optional();
const sshUsername = z.string().trim().min(1).max(128).regex(/^[A-Za-z0-9._-]+$/).optional();
const bindingSchema = z.object({
transport: z.enum(DATABASE_TRANSPORTS),
host: optionalText,
port: port.optional(),
username: optionalText,
baseUrl: z.string().max(2048)
.refine((value) => parseCredentialFreeHttpUrl(value) !== undefined)
.optional(),
restPath: z.string().regex(/^\/(?!\/)[^?#\\\u0000-\u001f]*$/).max(512).optional(),
restAuth: z.enum(["none", "bearer", "x-api-key"]).optional(),
tlsServername: optionalText,
sshHost,
sshPort: port.optional(),
sshUsername,
sshTargetHost: sshHost,
sshTargetPort: port.optional(),
}).strict().superRefine((binding, context) => {
const required = binding.transport === "postgres_direct"
? ["host", "port", "username"] as const
: binding.transport === "rest_api"
? ["baseUrl", "restPath", "restAuth"] as const
: ["username", "sshHost", "sshPort", "sshUsername", "sshTargetHost", "sshTargetPort"] as const;
for (const field of required) {
if (binding[field] === undefined) context.addIssue({ code: "custom", path: [field], message: "Required" });
}
});
const configSchema = z.object({
workspaceId: workspaceIdSchema,
engine: z.literal("postgres"),
databaseName: identifier,
schema: identifier,
binding: bindingSchema,
}).strict();
const updateSchema = configSchema.extend({ version: z.number().int().positive() });
const secretNames = [
"password",
+29 -3
View File
@@ -13,6 +13,7 @@ import type { MaintenanceBarrier } from "../runtime/maintenance-gate.js";
import { hasPermission, isPrincipalContext, requirePermission } from "../auth/authorization.js";
import { splitCanonicalModelId, type RuntimeModelCatalog } from "../models/runtime-model-catalog.js";
import type { CatalogRepository } from "../catalog/types.js";
import { interactionLanguage } from "../tht/interaction-language.js";
const BOOTSTRAP_FAILURE_MESSAGE =
"Session startup failed. Check configuration and connectivity, then Resume the session.";
@@ -381,8 +382,13 @@ export function sessionRoutes(
app.post("/sessions", async (req, reply) => {
const b = req.body as {
question: string; name?: string; workspace?: string; workspaceId?: string;
provider?: string; model?: string; thinking?: string;
provider?: string; model?: string; thinking?: string; interactionLanguage?: unknown;
};
const language = interactionLanguage(b?.interactionLanguage);
if (!language) return reply.code(400).send({
code: "invalid_interaction_language",
error: "interactionLanguage must be a well-formed BCP-47 language tag",
});
if ((b.provider === undefined) !== (b.model === undefined)
|| (b.provider !== undefined && (typeof b.provider !== "string" || typeof b.model !== "string" || !b.provider || !b.model))) {
return reply.code(400).send({ error: "provider and model must be supplied together" });
@@ -503,11 +509,15 @@ export function sessionRoutes(
// registry snapshot. The legacy fallback stays available for sessions created before
// the browser-local preference migration.
let id: string;
let sessionLanguage = language;
try {
({ id } = await runner.sessionNew({
const created = await runner.sessionNew({
question: b.question, name: b.name, workspaceConfigPath,
workspaceId, workspaceRevision, provider, model, thinking,
}));
interactionLanguage: language,
});
id = created.id;
sessionLanguage = created.interaction_language ?? language;
manifestPersisted = true;
if (revisionLease) {
await revisionLease.markPersisted().catch((error: unknown) => {
@@ -520,6 +530,7 @@ export function sessionRoutes(
} catch { return storageFailure(reply); }
const options = {
provider, model, thinking,
interactionLanguage: sessionLanguage,
author: principal.displayName ?? principal.subject,
principal,
question: b.question,
@@ -664,6 +675,13 @@ export function sessionRoutes(
app.post("/sessions/:id/resume", async (req, reply) => {
const id = (req.params as any).id;
const principal = getPrincipal(req);
if (Object.hasOwn(req.body ?? {}, "interactionLanguage")
|| Object.hasOwn(req.body ?? {}, "interaction_language")) {
return reply.code(400).send({
code: "interaction_language_pinned",
error: "Resume uses the session's persisted interaction language; overrides are not accepted",
});
}
return withSessionLifecycle(id, async () => {
let settings: Settings;
let located: LocatedSession | undefined;
@@ -681,6 +699,7 @@ export function sessionRoutes(
const saved = manifest as {
provider?: string; model?: string; thinking?: string;
workspace_id?: string; workspace_revision?: string;
interaction_language?: string | null;
};
const requested = (req.body ?? {}) as { provider?: string; model?: string; thinking?: string };
if ((requested.provider === undefined) !== (requested.model === undefined)
@@ -719,6 +738,12 @@ export function sessionRoutes(
});
}
try { settings = await d.getSettings(principal); } catch { return storageFailure(reply); }
let language = saved.interaction_language;
if (language == null) {
try {
language = (await runner.ensureInteractionLanguage(id, workspaceConfigPath)).interaction_language;
} catch { return reply.code(503).send({ error: RESUME_FAILURE_MESSAGE }); }
}
// This check belongs inside the per-session lock: a preceding cold Resume may have
// installed a running runtime while this request was waiting.
const existing = d.mgr.get(id);
@@ -737,6 +762,7 @@ export function sessionRoutes(
});
const options = {
provider: selected.provider,
interactionLanguage: language,
model: selected.model,
thinking: saved?.thinking ?? settings.thinking,
author: principal.displayName ?? principal.subject,
+10
View File
@@ -0,0 +1,10 @@
/** Validate/canonicalize the tag; available translation catalogs belong to the UI. */
export function interactionLanguage(value: unknown): string | undefined {
if (typeof value !== "string") return undefined;
try {
const [canonical] = Intl.getCanonicalLocales(value);
return canonical;
} catch {
return undefined;
}
}
+10 -1
View File
@@ -59,6 +59,7 @@ export interface SessionRow {
author: string | null;
workspace_id?: string | null;
workspace_revision?: string | null;
interaction_language?: string | null;
archived?: boolean;
}
@@ -482,6 +483,7 @@ export class ThtRunner {
async sessionNew(o: {
question: string;
interactionLanguage?: string;
provider?: string;
model?: string;
thinking?: string;
@@ -497,6 +499,7 @@ export class ThtRunner {
["--provider", o.provider],
["--model", o.model],
["--thinking", o.thinking],
["--interaction-language", o.interactionLanguage],
["--name", o.name],
["--workspace-id", o.workspaceId],
["--workspace-revision", o.workspaceRevision],
@@ -504,7 +507,7 @@ export class ThtRunner {
if (v) a.push(f, v);
}
a.push("--json");
return this.json<{ id: string }>(a, o.workspaceConfigPath ?? o.workspace);
return this.json<{ id: string; interaction_language?: string }>(a, o.workspaceConfigPath ?? o.workspace);
}
/** Build and persist the deterministic F1 retrieval pack for a new session. */
@@ -533,6 +536,12 @@ export class ThtRunner {
return this.json<unknown>(["session", "show", id, "--json"], workspace);
}
ensureInteractionLanguage(id: string, workspace?: string) {
return this.json<{ interaction_language: string }>(
["session", "ensure-interaction-language", id, "--json"], workspace,
);
}
sqlPreview(id: string, p: { limit?: number; offset?: number }, workspace?: string) {
// No positional FILE: the harness resolves sql_final.sql from the session
// via _session_sql_file(cfg, session_id), which respects the workspace path.
+10
View File
@@ -0,0 +1,10 @@
/** Compiled with its runtime for the host CLI: no installation, Docker or host Node required. */
import { runWorkspaceDocuments } from "./workspaces/documents.js";
import { runBootstrapValidation } from "./catalog/bootstrap-cli.js";
import { runBootstrapProbes } from "./catalog/bootstrap-probes.js";
const args = process.argv.slice(2);
const result = args[0] === "probe" ? await runBootstrapProbes(args.slice(1))
: args[0] === "bootstrap" ? runBootstrapValidation(args.slice(1)) : runWorkspaceDocuments(args);
console.log(result.output);
process.exitCode = result.status;
+3 -3
View File
@@ -44,8 +44,8 @@ const catalogSchema = z.object({
});
});
function safeCatalogError(): Error {
return new Error("Workspace catalog is invalid");
function safeCatalogError(cause?: unknown): Error {
return new Error("Workspace catalog is invalid", { cause });
}
export function parseWorkspaceCatalogYaml(source: string): WorkspaceCatalog {
@@ -57,7 +57,7 @@ export function parseWorkspaceCatalogYaml(source: string): WorkspaceCatalog {
return catalogSchema.parse(document.toJSON()) as WorkspaceCatalog;
} catch (error) {
if (error instanceof Error && error.message === "Workspace catalog is invalid") throw error;
throw safeCatalogError();
throw safeCatalogError(error);
}
}
+222
View File
@@ -0,0 +1,222 @@
import { closeSync, lstatSync, mkdirSync, openSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { join, resolve } from "node:path";
import { parseAllDocuments, stringify } from "yaml";
import { ZodError } from "zod";
import { CATALOG_PATH, parseWorkspaceCatalogYaml, assertCatalogMatchesDescriptor } from "./catalog.js";
import { parseWorkspaceYaml, type WorkspaceDescriptor } from "./schema.js";
interface Issue { document: string; field: string; code: string; correction: string; line?: number }
interface Report {
schema_version: 1;
scope: "local-documents";
ok: boolean;
workspaces: { id: string; evidence: "absent" | "local-files" | "remote-deferred" }[];
issues: Issue[];
deferred_checks: string[];
}
const MAX_DOCUMENT_BYTES = 1024 * 1024;
const usage = "tht workspace prepare --directory NEW_PATH --id ID --name NAME [--language en|it] [--json]\ntht workspace validate --directory PATH [--json]";
export class DocumentError extends Error {
constructor(readonly issue: Issue) { super(issue.correction); }
}
function fail(document: string, field: string, code: string, correction: string): never {
throw new DocumentError({ document, field, code, correction });
}
/** Never include parser messages or submitted values: YAML and Zod errors can contain secrets. */
export function decode<T>(source: string, document: string, parser: (text: string) => T, contract = "workspace schema v4 or catalog schema v1"): T {
try {
const documents = parseAllDocuments(source, { uniqueKeys: true });
if (documents.length !== 1) fail(document, "$", "yaml_documents", "Keep exactly one YAML document in this file.");
const problem = [...documents[0].errors, ...documents[0].warnings][0];
if (problem) {
throw new DocumentError({ document, field: "$", code: "yaml_syntax", line: problem.linePos?.[0].line,
correction: "Correct YAML syntax, remove duplicate keys and unsupported tags at the indicated line." });
}
return parser(source);
} catch (error) {
if (error instanceof DocumentError) throw error;
const cause = error instanceof Error && error.cause instanceof ZodError ? error.cause : error;
if (cause instanceof ZodError) {
const issue = cause.issues[0];
// Strict schemas produce paths containing schema-defined keys and array indices only.
fail(document, issue.path.join(".") || "$", "schema_invalid",
issue.code === "unrecognized_keys" ? "Remove fields not defined by the current workspace/catalog contract."
: `Correct this field using ${contract}; check type, required value, uniqueness and allowed values.`);
}
fail(document, "$", "schema_invalid", "Use an authored workspace v4 descriptor; remove database configuration and keep it in the PostgreSQL Metadata Catalog.");
}
}
function stat(root: string, document: string, directory: boolean) {
let info;
try { info = lstatSync(join(root, document)); }
catch { fail(document, "$", "missing_reference", "Create the referenced local file or directory and grant read access."); }
if (info.isSymbolicLink() || (directory ? !info.isDirectory() : !info.isFile())) {
fail(document, "$", "unsafe_reference", "Use a regular local file or directory, without symbolic links or special files.");
}
return info;
}
function readDocument(root: string, document: string): string {
if (stat(root, document, false).size > MAX_DOCUMENT_BYTES) {
fail(document, "$", "document_too_large", "Keep YAML documents below the 1 MiB local validation limit.");
}
try { return readFileSync(join(root, document), "utf8"); }
catch { fail(document, "$", "unreadable", "Grant read access to this document and retry validation."); }
}
function directories(root: string, document: string) {
stat(root, document, true);
try { return readdirSync(join(root, document), { withFileTypes: true }).sort((a, b) => a.name.localeCompare(b.name)); }
catch { fail(document, "$", "unreadable", "Grant read and traversal access to this directory and retry validation."); }
}
function inspectEvidence(root: string, descriptor: WorkspaceDescriptor): "absent" | "local-files" | "remote-deferred" {
const evidence = descriptor.evidence;
if (!evidence) return "absent";
if (evidence.source.type !== "filesystem") return "remote-deferred";
const base = evidence.source.uri;
stat(root, base, true);
if (evidence.schema_version === 2) stat(root, `${base}/curated`, true);
const patterns = evidence.source.patterns ?? ["**/*.md"];
const literals = patterns.filter((pattern) => !/[?*\[]/.test(pattern));
for (const pattern of literals) {
const parts = pattern.split("/");
for (let index = 1; index < parts.length; index++) stat(root, `${base}/${parts.slice(0, index).join("/")}`, true);
stat(root, `${base}/${pattern}`, false);
}
// Inventory without following links; curated-unit interpretation remains a runtime check.
const pending = [base];
let count = 0;
while (pending.length) {
const current = pending.pop()!;
for (const entry of directories(root, current)) {
if (++count > 100_000) fail(base, "evidence.source", "inventory_limit", "Reduce the Evidence tree below 100,000 entries before local validation.");
const path = `${current}/${entry.name}`;
if (entry.isDirectory()) pending.push(path);
else {
const info = stat(root, path, false);
const relative = path.slice(base.length + 1);
// Only apply content checks to selections whose meaning is unambiguous locally.
// Arbitrary globs are expanded by the canonical Python adapter after startup.
const curated = evidence.schema_version === 2 && relative.startsWith("curated/") && relative.endsWith(".md");
const selected = curated || literals.includes(relative) || (patterns.includes("**/*.md") && relative.endsWith(".md"));
if (selected && info.size > evidence.source.max_bytes) fail(path, "evidence.source.max_bytes", "evidence_too_large", "Reduce the source file or increase the declared max_bytes limit deliberately.");
try { closeSync(openSync(join(root, path), "r")); }
catch { fail(path, "$", "unreadable", "Grant read access to this Evidence file and retry validation."); }
if (curated) {
const text = readDocument(root, path);
const frontmatter = text.match(/^---\r?\n([\s\S]*?)\r?\n---(?:\r?\n|$)/);
if (!frontmatter) fail(path, "$", "evidence_frontmatter", "Add a YAML frontmatter block delimited by --- to the curated Markdown unit; follow the Curated Evidence contract.");
decode(frontmatter[1], path, (source) => parseAllDocuments(source)[0].toJSON());
}
}
}
}
return "local-files";
}
function validate(root: string, report: Report): void {
directories(root, ".");
const catalog = decode(readDocument(root, CATALOG_PATH), CATALOG_PATH, parseWorkspaceCatalogYaml);
const ids = new Set(catalog.workspaces.map((entry) => entry.id));
for (const entry of directories(root, ".")) {
if (entry.name === ".git") continue;
if (entry.isSymbolicLink()) fail(".", "$", "unsafe_reference", "Replace repository-root symbolic links with regular files or directories.");
if (entry.isDirectory() && entry.name !== "workspace-docs" && !ids.has(entry.name)) {
fail(".", "workspaces", "unlisted_directory", "Every root directory except workspace-docs must match a catalog workspace id; remove or register the extra directory.");
}
if (entry.name === "workspace-docs") {
for (const docs of directories(root, "workspace-docs")) {
if (!ids.has(docs.name)) fail("workspace-docs", "$", "unlisted_documentation", "Keep documentation only for workspace ids listed in the catalog.");
for (const file of directories(root, `workspace-docs/${docs.name}`)) {
const path = `workspace-docs/${docs.name}/${file.name}`;
if (!["README.md", "contract.env.example"].includes(file.name)) fail(`workspace-docs/${docs.name}`, "$", "unsupported_documentation", "Keep only README.md and contract.env.example in workspace-docs/<id>. Place Evidence inside the workspace directory.");
stat(root, path, false);
}
}
}
}
for (const entry of catalog.workspaces) {
const document = `${entry.id}/workspace.yaml`;
try {
stat(root, entry.id, true);
const descriptor = decode(readDocument(root, document), document, parseWorkspaceYaml);
try { assertCatalogMatchesDescriptor(entry, descriptor); }
catch { fail(document, "workspace", "catalog_mismatch", "Make id, name and description identical in the catalog, descriptor and workspace directory name."); }
const evidence = inspectEvidence(root, descriptor);
report.workspaces.push({ id: entry.id, evidence });
if (evidence !== "absent") report.deferred_checks.push(`${entry.id}:evidence-source-selection`, `${entry.id}:evidence-content-provenance-and-indexing`);
if (evidence === "remote-deferred") report.deferred_checks.push(`${entry.id}:remote-evidence-access`);
} catch (error) {
if (error instanceof DocumentError) report.issues.push(error.issue);
else throw error;
}
}
}
function prepare(root: string, options: Map<string, string>): void {
const id = options.get("--id");
const name = options.get("--name");
const language = options.get("--language") ?? "en";
const catalog = stringify({ schema_version: 1, workspaces: [{ id, name }] });
const descriptor = stringify({ workspace: { schema_version: 4, id, name, language } });
decode(catalog, CATALOG_PATH, parseWorkspaceCatalogYaml);
decode(descriptor, "workspace.yaml", parseWorkspaceYaml);
try { mkdirSync(root); }
catch (error) {
if ((error as NodeJS.ErrnoException).code === "EEXIST") fail(".", "--directory", "destination_exists", "Choose a new directory; preparation never overwrites an existing directory or its documents.");
fail(".", "--directory", "destination_unavailable", "Create the parent directory and grant write access, then choose a new destination.");
}
try {
mkdirSync(join(root, id!));
mkdirSync(join(root, "workspace-docs", id!), { recursive: true });
writeFileSync(join(root, CATALOG_PATH), "# Index of workspace identities. Keep metadata identical to each descriptor.\n" + catalog, { flag: "wx" });
writeFileSync(join(root, id!, "workspace.yaml"), "# Authored workspace v4: optional Evidence; database binding belongs to the Metadata Catalog.\n" + descriptor, { flag: "wx" });
writeFileSync(join(root, "workspace-docs", id!, "README.md"),
"# Workspace documents / Documenti workspace\n\n" +
"EN: Edit thoth-workspaces.yaml and <id>/workspace.yaml together. Evidence is optional and absent by default. Add reviewed source material under <id>/evidence only when configured. Database connections and schema belong to the installation Metadata Catalog. No example databases are downloaded.\n\n" +
"IT: Modificare insieme thoth-workspaces.yaml e <id>/workspace.yaml. Le Evidence sono facoltative e inizialmente assenti. Inserire materiale verificato in <id>/evidence solo quando configurato. Connessioni e schema dei database appartengono al Metadata Catalog dell'installazione. Nessun database di esempio viene scaricato.\n\n" +
"Repeat / Ripetere: `tht workspace validate --directory <repository>`. Local success does not establish runtime readiness or semantic truth / Il successo locale non certifica readiness o verità semantica.\n", { flag: "wx" });
} catch {
rmSync(root, { recursive: true, force: true });
fail(".", "$", "prepare_failed", "Preparation could not write the documents; check disk space and permissions, then retry with a new directory.");
}
}
export function runWorkspaceDocuments(args: string[]): { status: number; output: string } {
const report: Report = { schema_version: 1, scope: "local-documents", ok: false, workspaces: [], issues: [], deferred_checks: ["catalog-database-binding", "database-connectivity", "runtime-preprocessing"] };
const json = args.includes("--json");
let status = 1;
try {
const [command, ...rest] = args;
if ((command === "prepare" || command === "validate") && rest.length === 1 && rest[0] === "--help") return { status: 0, output: usage };
const allowed = command === "prepare" ? ["--directory", "--id", "--name", "--language"] : command === "validate" ? ["--directory"] : [];
const options = new Map<string, string>();
const seen = new Set<string>();
for (let index = 0; index < rest.length; index++) {
const key = rest[index];
if (seen.has(key)) fail("CLI", "$", "usage", usage);
seen.add(key);
if (key === "--json") continue;
if (!allowed.includes(key) || !rest[index + 1] || rest[index + 1].startsWith("--")) fail("CLI", "$", "usage", usage);
options.set(key, rest[++index]);
}
if (!allowed.length || !options.has("--directory") || (command === "prepare" && (!options.has("--id") || !options.has("--name")))) fail("CLI", "$", "usage", usage);
const root = resolve(options.get("--directory")!);
if (command === "prepare") prepare(root, options);
validate(root, report);
report.ok = report.issues.length === 0;
status = report.ok ? 0 : 1;
} catch (error) {
report.issues.push(error instanceof DocumentError ? error.issue : { document: ".", field: "$", code: "io_error", correction: "Check local permissions and regular files, then retry; no services were started." });
if (report.issues[0].code === "usage") status = 2;
}
const output = json ? JSON.stringify(report) : [
report.ok ? "Workspace documents pass local validation." : "Workspace documents require corrections.",
...report.issues.map((issue) => `${issue.document}${issue.line ? `:${issue.line}` : ""} [${issue.field}] ${issue.code}: ${issue.correction}`),
...report.workspaces.map((entry) => `${entry.id}: Evidence ${entry.evidence}`),
`Deferred until runtime: ${report.deferred_checks.join(", ")}.`,
].join("\n");
return { status, output };
}
@@ -0,0 +1,49 @@
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { afterEach, expect, test } from "vitest";
import { validateDatabaseBootstrap } from "../src/catalog/bootstrap-documents.js";
import { runBootstrapValidation } from "../src/catalog/bootstrap-cli.js";
const roots: string[] = [];
afterEach(() => roots.splice(0).forEach((root) => rmSync(root, { recursive: true, force: true })));
function fixture() {
const root = mkdtempSync(join(tmpdir(), "bootstrap-documents-")); roots.push(root);
mkdirSync(join(root, "practice"));
writeFileSync(join(root, "thoth-workspaces.yaml"), "schema_version: 1\nworkspaces: [{id: practice, name: Practice}]\n");
writeFileSync(join(root, "practice/workspace.yaml"), "workspace: {schema_version: 4, id: practice, name: Practice, language: en}\n");
return root;
}
const entry = { workspaceId: "practice", engine: "postgres", databaseName: "practice", schema: "public", binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" }, secretFiles: { password: "/private/operator/db-password" } };
test("bootstrap reuses Catalog configuration and requires one complete binding per workspace", () => {
const root = fixture();
const valid = validateDatabaseBootstrap({ schemaVersion: 1, databases: [entry] }, root);
expect(valid.secretFiles).toContainEqual({ field: "databases.0.secretFiles.password", path: "/private/operator/db-password" });
for (const databases of [[], [entry, entry], [{ ...entry, workspaceId: "unknown" }], [{ ...entry, secretFiles: {} }], [{ ...entry, engine: "mysql" }], [{ ...entry, binding: { ...entry.binding, port: 70000 } }]]) {
expect(() => validateDatabaseBootstrap({ schemaVersion: 1, databases }, root)).toThrow();
}
});
test("REST authentication, SSH credentials and optional Evidence use their declared contracts", () => {
const root = fixture();
const rest = { ...entry, binding: { transport: "rest_api", baseUrl: "https://data.internal", restPath: "/query", restAuth: "none" }, secretFiles: {} };
expect(validateDatabaseBootstrap({ schemaVersion: 1, databases: [rest] }, root).secretFiles).toEqual([]);
expect(() => validateDatabaseBootstrap({ schemaVersion: 1, databases: [{ ...rest, binding: { ...rest.binding, restAuth: "bearer" } }] }, root)).toThrow();
const ssh = { ...entry, binding: { transport: "ssh_tunnel", username: "reader", sshHost: "bastion", sshPort: 22, sshUsername: "tunnel", sshTargetHost: "database", sshTargetPort: 5432 }, secretFiles: { password: "/password", sshPrivateKey: "/key", sshKnownHosts: "/hosts" } };
expect(validateDatabaseBootstrap({ schemaVersion: 1, databases: [ssh] }, root).warnings[0]).toContain("not NL-to-SQL");
writeFileSync(join(root, "practice/workspace.yaml"), "workspace: {schema_version: 4, id: practice, name: Practice, language: en}\nevidence:\n source:\n type: http\n uris: [https://docs.internal/manual.md]\n authentication: signed_urls_file\n");
expect(() => validateDatabaseBootstrap({ schemaVersion: 1, databases: [entry] }, root)).toThrow();
expect(validateDatabaseBootstrap({ schemaVersion: 1, databases: [{ ...entry, evidenceSecretFiles: { "evidence.signed_urls": "/urls.json" } }] }, root).secretFiles).toContainEqual({ field: "databases.0.evidenceSecretFiles.evidence.signed_urls", path: "/urls.json" });
});
test("bootstrap CLI never echoes arbitrary keys from submitted secret references", () => {
const root = fixture();
const path = join(root, "bootstrap.yaml");
for (const field of ["secretFiles", "evidenceSecretFiles"]) {
writeFileSync(path, JSON.stringify({ schemaVersion: 1, databases: [{ ...entry, [field]: { PRIVATE_CREDENTIAL_SENTINEL: "/path" } }] }));
const result = runBootstrapValidation(["--directory", root, "--bootstrap", path, "--json"]);
expect(result.status).toBe(1);
expect(result.output).not.toContain("PRIVATE_CREDENTIAL_SENTINEL");
}
});
+2 -2
View File
@@ -36,7 +36,7 @@ test("loop F1: crea sessione → SSE riceve il widget → risponde → il modell
ollamaEnsure: async () => ({ ok: true }),
searchPack: async () => {},
sessionNew: async () => ({ id: "s1" }),
sessionShow: async (_id: string) => ({ id: "s1", provider: undefined, model: undefined, thinking: undefined }),
sessionShow: async (_id: string) => ({ id: "s1", interaction_language: "en", provider: undefined, model: undefined, thinking: undefined }),
sessionList: async () => [],
} as any,
spawnFn: () => spawn("node", [FAKE, SCRIPT]) as any,
@@ -48,7 +48,7 @@ test("loop F1: crea sessione → SSE riceve il widget → risponde → il modell
const created = await fetch(`${base}/sessions`, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ workspace: "w", question: "q" }),
body: JSON.stringify({ workspace: "w", question: "q", interactionLanguage: "en" }),
});
expect(created.status).toBe(200);
@@ -0,0 +1,67 @@
import { spawnSync } from "node:child_process";
import { chmodSync, mkdtempSync, readFileSync, realpathSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { rootCertificates } from "node:tls";
import { afterEach, expect, test } from "vitest";
const binary = process.env.THT_INSTALLATION_TEST_CLI;
const roots: string[] = [];
afterEach(() => roots.splice(0).forEach((path) => rmSync(path, { recursive: true, force: true })));
function run(...args: string[]) {
return spawnSync(binary!, [...args, "--json"], { encoding: "utf8", env: { ...process.env, PATH: "" }, input: "" });
}
test.skipIf(!binary)("operator prepares, completes and repeatedly validates before any stack exists", () => {
const root = realpathSync(mkdtempSync(join(tmpdir(), "application-documents-"))); roots.push(root);
const workspace = join(root, "workspaces"), directory = join(root, "installation");
expect(run("workspace", "prepare", "--directory", workspace, "--id", "practice", "--name", "Practice").status).toBe(0);
expect(run("installation", "prepare", "--directory", directory).status).toBe(0);
const installation = join(directory, "thothii-installation.yaml");
const validate = () => run("--installation", installation, "installation", "validate", "--workspaces", workspace);
expect(validate().status).toBe(1); // Visible placeholders cannot be approved.
expect(run("installation", "credentials", "--directory", directory).status).toBe(0);
for (const name of ["thothii-installation.yaml", "operator.env", "database-bootstrap.yaml"]) {
const path = join(directory, name);
writeFileSync(path, readFileSync(path, "utf8").replaceAll("CHANGE_ME", "practice"));
}
for (const [name, contents] of Object.entries({ "secrets.env": "OPENAI_API_KEY=PRIVATE_PROVIDER_VALUE\n", "database-password": "PRIVATE_DATABASE_VALUE", "git-credentials": "", "git-ca.pem": rootCertificates[0] })) {
writeFileSync(join(directory, "secrets", name), contents, { mode: 0o600 });
}
const before = readFileSync(installation, "utf8");
for (let index = 0; index < 2; index++) {
const checked = validate();
expect(checked.stderr).toBe("");
expect(checked.stdout).not.toContain("PRIVATE_");
expect(checked.status, checked.stdout).toBe(0);
expect(JSON.parse(checked.stdout).deferred_checks).toContain("release-assets");
}
expect(readFileSync(installation, "utf8")).toBe(before);
writeFileSync(installation, before.replace("interaction: openai/gpt-4.1-mini", "interaction: openai/nonexistent"));
expect(validate().status).toBe(1);
writeFileSync(installation, before);
const authFile = join(directory, "secrets/pi-auth.json");
writeFileSync(authFile, "not-json");
expect(validate().status).toBe(1);
writeFileSync(installation, before.replace("{mode: secret_env, apiKeyEnv: OPENAI_API_KEY}", "{mode: pi_auth}"));
writeFileSync(authFile, JSON.stringify({ openai: { unrelated: true } }));
expect(validate().status).toBe(1);
writeFileSync(authFile, JSON.stringify({ " OpenAI ": { type: "api_key", key: "PRIVATE_PI_KEY" } }));
expect(validate().status).toBe(1);
writeFileSync(authFile, JSON.stringify({ openai: { type: "api_key", key: "PRIVATE_PI_KEY" } }));
expect(validate().status).toBe(0);
writeFileSync(installation, before);
writeFileSync(authFile, "{}");
const bootstrap = join(directory, "database-bootstrap.yaml");
const originalBootstrap = readFileSync(bootstrap, "utf8");
writeFileSync(join(workspace, "practice", "private-password"), "PRIVATE_DATABASE_VALUE", { mode: 0o600 });
writeFileSync(bootstrap, originalBootstrap.replace(join(directory, "secrets/database-password"), join(workspace, "practice/private-password")));
expect(validate().status).toBe(1);
writeFileSync(bootstrap, originalBootstrap);
chmodSync(join(directory, "secrets/database-password"), 0o644);
expect(validate().status).toBe(1);
chmodSync(join(directory, "secrets/database-password"), 0o600);
const env = join(directory, "operator.env");
writeFileSync(env, readFileSync(env, "utf8") + "THOTH_HTTP_PORT=8081\n");
expect(validate().status).toBe(1);
}, 15_000);
@@ -0,0 +1,43 @@
import { createServer } from "node:http";
import { mkdtempSync, writeFileSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { expect, it } from "vitest";
import { probeBootstrapDependencies } from "../src/catalog/bootstrap-probes.js";
import { probePublicEvidenceUrl } from "../src/catalog/evidence-probe-http.js";
it("refuses loopback literals and DNS answers before sending an Evidence GET", async () => {
for (const hostname of ["[::1]", "127.0.0.1", "localhost", "[::ffff:127.0.0.1]"]) {
await expect(probePublicEvidenceUrl(new URL(`http://${hostname}/private`))).rejects.toThrow("Evidence network policy refused");
}
});
it("authenticates a bounded read-only REST probe and blocks unavailable credentials/services", async () => {
const directory = mkdtempSync(join(tmpdir(), "tht-probe-"));
const secret = join(directory, "key");
writeFileSync(secret, "PRIVATE_SENTINEL", { mode: 0o600 });
let status = 200;
const requests: string[] = [];
const server = createServer((req, res) => {
requests.push(`${req.method} ${req.url}`);
res.writeHead(req.headers.authorization === "Bearer PRIVATE_SENTINEL" ? status : 401);
res.end("PRIVATE_SERVER_RESPONSE");
});
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
const address = server.address() as { port: number };
const document = { schemaVersion: 1, databases: [{ workspaceId: "demo", engine: "postgres", databaseName: "demo", schema: "public", binding: { transport: "rest_api", baseUrl: `http://127.0.0.1:${address.port}`, restPath: "/health", restAuth: "bearer" }, secretFiles: { apiKey: secret } }] };
try {
expect((await probeBootstrapDependencies(document)).ok).toBe(true);
status = 503;
const failed = await probeBootstrapDependencies(document);
expect(failed.ok).toBe(false);
expect(JSON.stringify(failed)).not.toContain("PRIVATE");
status = 200;
writeFileSync(secret, "rotated-but-invalid");
expect((await probeBootstrapDependencies(document)).ok).toBe(false);
expect(requests).toEqual(["GET /health", "GET /health", "GET /health"]);
} finally {
await new Promise<void>((resolve) => server.close(() => resolve()));
rmSync(directory, { recursive: true });
}
});
@@ -95,6 +95,35 @@ test("returns empty catalogs when no runtime projection is configured", () => {
expect(loadMetadataGenerationModels({}).catalog()).toEqual({ models: [], default: null });
});
test.each([
["qwen-chat-template", true, true],
["qwen", true, true],
["qwen-typo", true, false],
["qwen-chat-template", false, false],
])("validates Qwen thinking format %s with reasoning=%s", (thinkingFormat, reasoning, valid) => {
const { catalogFile } = runtimeCatalog({
defaultInteraction: "local/qwen",
models: [{
id: "local/qwen", provider: "local", model: "qwen", label: "Qwen",
upstreamModel: "qwen", endpoint: { baseUrl: "http://localhost:8000/v1" },
authentication: { mode: "none" }, sessionAdapter: { mode: "openai_compatible" },
session: {
reasoning, contextWindow: 32768, maxTokens: 8192,
compatibility: {
supportsDeveloperRole: false, supportsReasoningEffort: false,
supportsStore: false, maxTokensField: "max_tokens", thinkingFormat,
},
},
}],
});
if (valid) {
expect(loadRuntimeModelCatalog(catalogFile).sessionModels()[0].session?.compatibility)
.toMatchObject({ thinkingFormat });
} else {
expect(() => loadRuntimeModelCatalog(catalogFile)).toThrow("runtime model catalog is invalid");
}
});
test("rejects a drifted default and an unprotected projection", () => {
const drifted = runtimeCatalog({ defaultInteraction: "zai/missing" });
expect(() => loadRuntimeModelCatalog(drifted.catalogFile)).toThrow("interaction default is invalid");
+14
View File
@@ -36,6 +36,10 @@ vi.mock("../src/tht/tht-runner.js", () => ({
vi.mock("../src/workspaces/registry.js", () => ({
WorkspaceRegistry: class {
async pull() {
return { branch: "main", head: "a".repeat(40), ahead: 0, behind: 0, degraded: false };
}
async listRetainedSnapshots() {
return [{ snapshotPath: "/data/workspace-registry/snapshots/revision/workspace.yaml" }];
}
@@ -98,3 +102,13 @@ test("workflow doctor gives schema-v4 runtime rendering a live Catalog repositor
expect(fakes.releaseWorkspaceRuntime).toHaveBeenCalledOnce();
expect(fakes.catalogRepository.close).toHaveBeenCalledOnce();
});
test("workspace pull exposes only safe Git status", async () => {
await expect(runOperatorAction("workspace-pull", config)).resolves.toEqual({
ready: true,
status: "succeeded",
branch: "main",
head: "a".repeat(40),
degraded: false,
});
});
+23
View File
@@ -190,6 +190,29 @@ test("Pi receives the leased workspace runtime config and releases it on direct
expect(release).toHaveBeenCalledOnce();
});
test("Pi language launch hint comes from the session options and never ambient environment", () => {
const previous = process.env.THT_INTERACTION_LANGUAGE;
process.env.THT_INTERACTION_LANGUAGE = "it";
const environments: NodeJS.ProcessEnv[] = [];
const mgr = new PiProcessManager(loadConfig({}), {
spawnFn: (_command, _args, options) => {
environments.push(options.env);
return recordingChild() as any;
},
});
try {
mgr.createFor("explicit-language", { interactionLanguage: "en" });
mgr.teardown("explicit-language");
mgr.createFor("manifest-resolved-in-gate");
mgr.teardown("manifest-resolved-in-gate");
expect(environments[0].THT_INTERACTION_LANGUAGE).toBe("en");
expect(environments[1]).not.toHaveProperty("THT_INTERACTION_LANGUAGE");
} finally {
if (previous === undefined) delete process.env.THT_INTERACTION_LANGUAGE;
else process.env.THT_INTERACTION_LANGUAGE = previous;
}
});
test("a close-only child event releases its temporary Pi agent snapshot", () => {
const child = recordingChild();
let snapshotDir: string | undefined;
+196 -42
View File
@@ -53,7 +53,11 @@ const defaultWorkspaceRegistry = {
function buildApp(config: Parameters<typeof buildRealApp>[0], deps: Record<string, unknown> = {}) {
const thtRunner = deps.thtRunner
? { qdrantEnsure: async () => ({ ok: true }), ...(deps.thtRunner as object) }
? {
qdrantEnsure: async () => ({ ok: true }),
ensureInteractionLanguage: async () => ({ interaction_language: "en" }),
...(deps.thtRunner as object),
}
: undefined;
return buildRealApp(config, {
workspaceRuntimeSupport: () => true,
@@ -88,6 +92,156 @@ const aliceHeaders = {
"x-thoth-is-admin": "0",
};
test.each([undefined, null, "", "en--US", "en_US", "en\nIGNORE", "en<script>", 42])(
"new sessions reject invalid interaction language %s before persistence", async (interactionLanguage) => {
const app = mutApp({ sessionNew: async () => { throw new Error("must not create"); } });
try {
const response = await app.inject({
method: "POST", url: "/sessions", payload: { question: "Pazienti", interactionLanguage },
});
expect(response.statusCode).toBe(400);
expect(response.json()).toMatchObject({ code: "invalid_interaction_language" });
} finally { await app.close(); }
},
);
test.each([["en", "en"], ["fr-FR", "fr-FR"], ["FR-fr", "fr-FR"]])(
"new session forwards canonical interaction language %s without changing the question", async (language, canonical) => {
let created: any;
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
thtRunner: {
sessionNew: async (options: any) => { created = options; return { id: "language" }; },
searchPack: async () => {},
},
readiness: { ensure: async () => ({ ok: true }) },
mgr: {
get: () => undefined, createFor: () => ({ bridge: { onClientEvent: () => {} } }),
configure: async () => {}, start: () => {},
},
getSettings: () => ({ workspace: "default" }),
});
try {
const response = await app.inject({
method: "POST", url: "/sessions", payload: { question: "Pazienti", interactionLanguage: language },
});
expect(response.statusCode).toBe(200);
expect(created).toMatchObject({ question: "Pazienti", interactionLanguage: canonical });
} finally { await app.close(); }
});
test("resume rejects browser interaction language overrides", async () => {
const app = mutApp({ sessionShow: async () => ({ status: "open", interaction_language: "it" }) });
try {
const response = await app.inject({
method: "POST", url: "/sessions/s1/resume", payload: { interactionLanguage: "en" },
});
expect(response.statusCode).toBe(400);
expect(response.json()).toMatchObject({ code: "interaction_language_pinned" });
} finally { await app.close(); }
});
test("new session starts Pi with the question language persisted by the harness", async () => {
let runtime: any;
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
thtRunner: {
sessionNew: async () => ({ id: "english-question", interaction_language: "en" }),
searchPack: async () => {},
},
readiness: { ensure: async () => ({ ok: true }) },
mgr: {
get: () => undefined,
createFor: (_id: string, options: any) => {
runtime = options;
return { bridge: { onClientEvent: () => {} } };
},
configure: async () => {}, start: () => {},
},
getSettings: () => ({ workspace: "default" }),
});
try {
const response = await app.inject({ method: "POST", url: "/sessions", payload: {
question: "How many patients were admitted last year?", interactionLanguage: "it",
} });
expect(response.statusCode).toBe(200);
expect(runtime.interactionLanguage).toBe("en");
} finally { await app.close(); }
});
test("resume pins legacy interaction language using the resolved harness workspace", async () => {
let pinnedWorkspace: string | undefined;
let runtime: any;
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
thtRunner: {
sessionShow: async () => ({ id: "legacy", status: "closed" }),
ensureInteractionLanguage: async (_id: string, workspace: string) => {
pinnedWorkspace = workspace;
return { interaction_language: "it" };
},
reopenSession: async () => {},
},
readiness: { ensure: async () => ({ ok: true }) },
mgr: {
get: () => undefined,
createFor: (_id: string, options: any) => {
runtime = options;
return { bridge: { onClientEvent: () => {} } };
},
configure: async () => {}, start: () => {},
},
getSettings: () => ({ workspace: "other-browser-workspace" }),
});
try {
const response = await app.inject({ method: "POST", url: "/sessions/legacy/resume" });
expect(response.statusCode).toBe(200);
expect(pinnedWorkspace).toContain("/default.yaml");
expect(runtime).toMatchObject({ interactionLanguage: "it", mode: "resume" });
} finally { await app.close(); }
});
test("resume retains persisted interaction language despite different browser settings", async () => {
let options: any;
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
thtRunner: {
sessionShow: async () => ({ id: "saved", status: "closed", interaction_language: "it" }),
ensureInteractionLanguage: async () => { throw new Error("language is already pinned"); },
reopenSession: async () => {},
},
readiness: { ensure: async () => ({ ok: true }) },
mgr: {
get: () => undefined,
createFor: (_id: string, value: any) => {
options = value;
return { bridge: { onClientEvent: () => {} } };
},
configure: async () => {}, start: () => {},
},
getSettings: () => ({ workspace: "english-workspace", locale: "en" }),
});
try {
const response = await app.inject({ method: "POST", url: "/sessions/saved/resume" });
expect(response.statusCode).toBe(200);
expect(options.interactionLanguage).toBe("it");
} finally { await app.close(); }
});
test("resume cannot start Pi if legacy interaction language cannot be persisted", async () => {
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
thtRunner: {
sessionShow: async () => ({ id: "legacy", status: "closed" }),
ensureInteractionLanguage: async () => { throw new Error("private storage details"); },
reopenSession: async () => { throw new Error("must not reopen"); },
},
readiness: { ensure: async () => ({ ok: true }) },
mgr: { createFor: () => { throw new Error("must not spawn"); } },
getSettings: () => ({ workspace: "default" }),
});
try {
const response = await app.inject({ method: "POST", url: "/sessions/legacy/resume" });
expect(response.statusCode).toBe(503);
expect(response.body).not.toContain("private storage details");
} finally { await app.close(); }
});
test("upstream requests without a principal fail before a Pi runtime can be created", async () => {
let created = false;
const app = buildApp(loadConfig({ AUTH_MODE: "upstream", THT_HARNESS_DIR: "../harness" }), {
@@ -95,7 +249,7 @@ test("upstream requests without a principal fail before a Pi runtime can be crea
thtRunner: {} as any,
});
const response = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const response = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(response.statusCode).toBe(401);
expect(created).toBe(false);
@@ -133,7 +287,7 @@ test("maintenance rejects new and resumed session admission without interrupting
});
const create = await app.inject({
method: "POST", url: "/sessions", headers: aliceHeaders, payload: { question: "q" },
method: "POST", url: "/sessions", headers: aliceHeaders, payload: { interactionLanguage: "en", question: "q" },
});
const resume = await app.inject({ method: "POST", url: "/sessions/open/resume", headers: aliceHeaders });
@@ -154,7 +308,7 @@ test("a durable maintenance marker initializes admission closed after backend re
AUTH_MODE: "upstream", THT_HARNESS_DIR: "../harness", THT_MAINTENANCE_FILE: marker,
}), { thtRunner: {} as any });
const response = await app.inject({
method: "POST", url: "/sessions", headers: aliceHeaders, payload: { question: "q" },
method: "POST", url: "/sessions", headers: aliceHeaders, payload: { interactionLanguage: "en", question: "q" },
});
expect(response.statusCode).toBe(503);
expect(response.json()).toMatchObject({ code: "maintenance" });
@@ -492,7 +646,7 @@ test("new sessions are created through the authenticated principal, not a client
const response = await app.inject({
method: "POST", url: "/sessions", headers: aliceHeaders,
payload: { question: "q", owner: "mallory" },
payload: { interactionLanguage: "en", question: "q", owner: "mallory" },
});
expect(response.statusCode).toBe(200);
@@ -509,7 +663,7 @@ test("new sessions reject the client legacy workspace field unless local legacy
});
const response = await app.inject({
method: "POST", url: "/sessions", payload: { question: "q", workspace: "legacy" },
method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q", workspace: "legacy" },
});
expect(response.statusCode).toBe(409);
@@ -529,7 +683,7 @@ test("explicit local legacy mode permits the unpinned client workspace request",
});
const response = await app.inject({
method: "POST", url: "/sessions", payload: { question: "q", workspace: "legacy" },
method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q", workspace: "legacy" },
});
expect(response.statusCode).toBe(200);
@@ -567,7 +721,7 @@ test("creates a session from the active immutable workspace revision", async ()
await app.inject({
method: "POST", url: "/sessions",
payload: { question: "q", workspaceId: "psd-clinical", provider: "zai", model: "glm-5.2", thinking: "low" },
payload: { interactionLanguage: "en", question: "q", workspaceId: "psd-clinical", provider: "zai", model: "glm-5.2", thinking: "low" },
});
expect(sessionNew).toHaveBeenCalledWith(expect.objectContaining({
@@ -611,7 +765,7 @@ test("hands one Catalog-backed runtime to both retrieval and Pi", async () => {
const response = await app.inject({
method: "POST",
url: "/sessions",
payload: { question: "Which users placed orders?", workspaceId: "default" },
payload: { interactionLanguage: "en", question: "Which users placed orders?", workspaceId: "default" },
});
expect(response.statusCode).toBe(200);
@@ -687,7 +841,7 @@ test("refuses core admission when the workspace preprocessing fingerprint is sta
const response = await app.inject({
method: "POST",
url: "/sessions",
payload: { question: "q", workspaceId: "default" },
payload: { interactionLanguage: "en", question: "q", workspaceId: "default" },
});
expect(response.statusCode).toBe(409);
@@ -739,7 +893,7 @@ test("rejects an SSH-only Catalog binding before persisting or starting a sessio
} as any,
});
const response = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const response = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(response.statusCode).toBe(409);
expect(response.json()).toMatchObject({ code: "workspace_not_activatable" });
@@ -779,7 +933,7 @@ test("hands a revision lease to retention only after the session manifest is dur
workspaceRegistry: { acquireSessionRevision } as any,
});
const request = app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const request = app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
await new Promise((resolve) => setImmediate(resolve));
expect(markPersisted).not.toHaveBeenCalled();
expect(abort).not.toHaveBeenCalled();
@@ -810,7 +964,7 @@ test("creates a session from the configured default workspace revision when work
workspaceRegistry: registry as any,
});
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(registry.read).toHaveBeenCalledWith("psd-clinical");
expect(sessionNew).toHaveBeenCalledWith(expect.objectContaining({
@@ -896,7 +1050,7 @@ test("session lifecycle locates a B session when installation default is A", asy
} as any,
});
expect((await app.inject({ method: "POST", url: "/sessions", payload: {
expect((await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en",
question: "B question", workspaceId: "b-workspace", provider: "zai", model: "glm-5.2", thinking: "low",
} })).statusCode).toBe(200);
expect((await app.inject({ method: "GET", url: "/sessions" })).json()).toEqual([
@@ -939,7 +1093,7 @@ test("POST /sessions uses the catalog default with workspace/thinking settings a
],
spawnFn: () => nodeSpawn("node", [FAKE, SCRIPT]) as any,
});
const created = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const created = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(created.json()).toEqual({ id: "s1" });
expect(sessionNewArg.workspaceConfigPath).toContain(`/snapshots/${"e".repeat(40)}/w.yaml`);
expect(sessionNewArg.provider).toBe("zai");
@@ -1000,11 +1154,11 @@ test("POST /sessions stops the user's previous open Pi runtime before creating a
getSettings: () => ({ workspace: "local" }) as any,
});
expect((await app.inject({ method: "POST", url: "/sessions", payload: { question: "one" } })).statusCode)
expect((await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "one" } })).statusCode)
.toBe(200);
order.length = 0;
const second = await app.inject({ method: "POST", url: "/sessions", payload: { question: "two" } });
const second = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "two" } });
expect(second.statusCode).toBe(200);
expect(second.json()).toEqual({ id: "s2" });
@@ -1025,7 +1179,7 @@ test("POST /sessions refuses to create a session when the local DWH precheck fai
readiness: { ensure: async () => ({ ok: true }) } as any,
getSettings: () => ({ workspace: "w" }) as any,
});
const res = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const res = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(res.statusCode).toBe(503);
expect(res.json()).toMatchObject({ code: "dwh_unreachable" });
expect(pinged).toBe(1);
@@ -1046,7 +1200,7 @@ test("POST /sessions proceeds past a passing DWH precheck", async () => {
getSettings: () => ({ workspace: "w" }) as any,
spawnFn: () => nodeSpawn("node", [FAKE, SCRIPT]) as any,
});
const res = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const res = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(res.statusCode).toBe(200);
expect(pinged).toBe(1);
expect(created).toBe(1);
@@ -1065,7 +1219,7 @@ test("POST /sessions skips the DWH precheck when the flag is off (default)", asy
getSettings: () => ({ workspace: "w" }) as any,
spawnFn: () => nodeSpawn("node", [FAKE, SCRIPT]) as any,
});
const res = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const res = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(res.statusCode).toBe(200);
expect(pinged).toBe(0); // probe never runs without the flag
});
@@ -1099,7 +1253,7 @@ test("POST /sessions configura Pi con il thinking globale selezionato", async ()
],
});
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
await new Promise((resolve) => setImmediate(resolve));
expect(configured.thinking).toBe("high");
@@ -1427,7 +1581,7 @@ test("resuming a different session stops the user's previous Pi runtime", async
readiness: { ensure: async () => ({ ok: true }) } as any,
getSettings: () => ({ workspace: "local" }) as any,
});
expect((await app.inject({ method: "POST", url: "/sessions", payload: { question: "one" } })).statusCode)
expect((await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "one" } })).statusCode)
.toBe(200);
const resumed = await app.inject({ method: "POST", url: "/sessions/s2/resume" });
@@ -1815,13 +1969,13 @@ test.each([
getSettings: () => ({ workspace: "local" }) as any,
});
await app.inject({ method: "POST", url: "/sessions", payload: { question: "old" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "old" } });
if (lifecycle === "close") {
await app.inject({ method: "POST", url: "/sessions/s1/close" });
await app.inject({ method: "POST", url: "/sessions/s1/resume" });
} else {
await app.inject({ method: "DELETE", url: "/sessions/s1" });
await app.inject({ method: "POST", url: "/sessions", payload: { question: "replacement" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "replacement" } });
}
await new Promise((resolve) => setImmediate(resolve));
expect(current).toBe(replacement);
@@ -1885,7 +2039,7 @@ test.each(["resolve", "reject"] as const)(
getSettings: () => ({ workspace: "local" }) as any,
});
await app.inject({ method: "POST", url: "/sessions", payload: { question: "old" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "old" } });
await app.inject({ method: "DELETE", url: "/sessions/s1" });
published.length = 0;
@@ -2077,7 +2231,7 @@ test("Close suppresses a bootstrap that settles while close persistence is pendi
getSettings: () => ({ workspace: "local" }) as any,
});
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
const closeResponse = app.inject({ method: "POST", url: "/sessions/s1/close" });
await closeStarted;
published.length = 0;
@@ -2151,7 +2305,7 @@ test("bootstrap failure persists once and keeps Resume serialized behind that pe
getSettings: () => ({ workspace: "local" }) as any,
});
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
oldConfigure.reject(new Error("configure failed"));
await failureStarted;
const resumeResponse = app.inject({ method: "POST", url: "/sessions/s1/resume" })
@@ -2274,7 +2428,7 @@ test("a replaced runtime cannot publish or fail the newly resumed session", asyn
readiness: { ensure: async () => ({ ok: true }) } as any,
getSettings: () => ({ workspace: "local" }) as any,
});
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
await new Promise((resolve) => setImmediate(resolve));
oldBridge.setState("idle");
@@ -2334,7 +2488,7 @@ test("a deleted runtime cannot repopulate or fail the forgotten session", async
readiness: { ensure: async () => ({ ok: true }) } as any,
getSettings: () => ({ workspace: "local" }) as any,
});
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
await new Promise((resolve) => setImmediate(resolve));
const response = await app.inject({ method: "DELETE", url: "/sessions/s1" });
@@ -2379,7 +2533,7 @@ test("an unexpectedly exited runtime publishes its terminal sequence then releas
readiness: { ensure: async () => ({ ok: true }) } as any,
getSettings: () => ({ workspace: "local" }) as any,
});
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
await new Promise((resolve) => setImmediate(resolve));
published.length = 0;
@@ -2430,7 +2584,7 @@ test("agent_end releases the Pi runtime after the session was finalized", async
readiness: { ensure: async () => ({ ok: true }) } as any,
getSettings: () => ({ workspace: "local" }) as any,
});
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
await new Promise((resolve) => setImmediate(resolve));
bridge.emitClientEvent({ type: "system_event", event: "agent_end" });
@@ -2502,7 +2656,7 @@ test("POST /sessions/:id/response senza gate pendente risponde 409 (risposta sta
getSettings: () => ({ workspace: "w" }),
spawnFn: () => nodeSpawn("node", [FAKE, SCRIPT]) as any,
});
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
// The fake Pi never emitted a ui_request: the bridge has no pending descriptor, so a
// response (stale UI, double submit) must be rejected instead of forwarded to Pi.
const res = await app.inject({ method: "POST", url: "/sessions/s1/response",
@@ -2643,7 +2797,7 @@ test("POST /sessions readiness failure returns one fixed public message without
getSettings: () => ({ workspace: "psd" }) as any,
spawnFn: () => nodeSpawn("node", [FAKE, SCRIPT]) as any,
});
const res = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const res = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(res.statusCode).toBe(503);
expect(res.json()).toEqual({
error: "Session services are not ready. Check configuration and connectivity, then try again.",
@@ -2664,7 +2818,7 @@ test.each(["semantic_index_incompatible", "workspace_not_activatable"] as const)
});
const response = await app.inject({
method: "POST", url: "/sessions", payload: { question: "q" },
method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" },
});
expect(response.statusCode).toBe(503);
@@ -2687,7 +2841,7 @@ test("POST /sessions returns storage 503 before creating a Pi runtime when sessi
mgr: { createFor: () => { piCreated = true; throw new Error("must not spawn"); } } as any,
});
const response = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const response = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(response.statusCode).toBe(503);
expect(response.json()).toEqual({ error: "session storage is unavailable" });
@@ -2707,7 +2861,7 @@ test("POST /sessions proceeds when ollamaEnsure succeeds", async () => {
getSettings: () => ({ workspace: "psd" }) as any,
spawnFn: () => nodeSpawn("node", [FAKE, SCRIPT]) as any,
});
const res = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const res = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(res.json()).toEqual({ id: "s1" });
expect(qdrantEnsure).toHaveBeenCalledWith(operationalWorkspace("psd"), 60, "self_heal");
expect(ensureWs).toContain(`/snapshots/${"e".repeat(40)}/psd.yaml`);
@@ -2740,7 +2894,7 @@ test("POST /sessions rejects a stale requested model without silently using the
const res = await app.inject({
method: "POST",
url: "/sessions",
payload: { question: "q", provider: "deepseek", model: "deepseek-v4-pro" },
payload: { interactionLanguage: "en", question: "q", provider: "deepseek", model: "deepseek-v4-pro" },
});
expect(res.statusCode).toBe(503);
@@ -2775,7 +2929,7 @@ test("POST /sessions marks a persisted session failed when runtime construction
],
});
const res = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const res = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(res.statusCode).toBe(503);
expect(res.json()).toEqual({
@@ -2868,7 +3022,7 @@ test.each([
try {
const response = flow === "new"
? await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } })
? await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } })
: await app.inject({ method: "POST", url: `/sessions/${sessionId}/resume` });
const logs = consoleError.mock.calls.flat().map(String).join(" ");
@@ -2974,7 +3128,7 @@ test("POST /sessions returns after bridge attachment but starts only after retri
getSettings: () => ({ workspace: "psd" }) as any,
});
const res = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const res = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(res.json()).toEqual({ id: "s-early" });
expect(bridgeAttached).toBe(true);
expect(started).toBe(false);
@@ -3021,7 +3175,7 @@ test("POST /sessions bootstrap failure emits only a fixed recovery message", asy
const response = await app.inject({
method: "POST",
url: "/sessions",
payload: { question: "q" },
payload: { interactionLanguage: "en", question: "q" },
});
await new Promise((resolve) => setImmediate(resolve));
@@ -3117,7 +3271,7 @@ test.each([
})),
} as any,
});
const response = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const response = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(response.statusCode).toBe(expectedStatus);
expect(ensure).toHaveBeenCalledTimes(reachesReadiness ? 1 : 0);
+13
View File
@@ -34,6 +34,19 @@ test("sessionNew parses id from JSON", async () => {
expect(await r.sessionNew({ question: "q" })).toEqual({ id: "2026-06-27-100000-x" });
});
test("session language uses public per-command CLI flags and the selected config", async () => {
const runner = new ThtRunner({ thtBin: "tht", harnessDir: "/h", configPath: "config/tht.yaml" });
(spawn as any).mockClear();
await runner.sessionNew({ question: "Pazienti", interactionLanguage: "en" });
expect((spawn as any).mock.calls[0][1]).toEqual([
"session", "new", "Pazienti", "--interaction-language", "en", "--json", "-c", "config/tht.yaml",
]);
await runner.ensureInteractionLanguage("s1");
expect((spawn as any).mock.calls[1][1]).toEqual([
"session", "ensure-interaction-language", "s1", "--json", "-c", "config/tht.yaml",
]);
});
test("searchPack persists retrieval context with session and workspace", async () => {
const calls: any[] = [];
const r = new ThtRunner({ thtBin: "tht", harnessDir: "/h", configPath: "config/tht.yaml" });
@@ -0,0 +1,119 @@
import { spawnSync } from "node:child_process";
import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync, symlinkSync } from "node:fs";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { afterEach, expect, test } from "vitest";
import { parseWorkspaceCatalogYaml } from "../src/workspaces/catalog.js";
import { parseWorkspaceYaml } from "../src/workspaces/schema.js";
const roots: string[] = [];
function directory() {
const root = mkdtempSync(join(tmpdir(), "tht-documents-"));
roots.push(root);
return join(root, "workspaces");
}
afterEach(() => roots.splice(0).forEach((root) => rmSync(root, { recursive: true, force: true })));
function cli(...args: string[]) {
const packaged = process.env.THT_WORKSPACE_TEST_CLI;
const result = spawnSync(packaged ?? process.execPath, [...(packaged ? ["workspace"] : ["--import", "tsx", resolve("src/workspace-documents-cli.ts")]), ...args, "--json"], {
encoding: "utf8", env: { ...process.env, PATH: "" },
});
return { ...result, report: result.stdout.trim() ? JSON.parse(result.stdout) : null };
}
const descriptor = "workspace:\n schema_version: 4\n id: practice\n name: Practice\n language: en\n";
const catalog = "schema_version: 1\nworkspaces:\n - id: practice\n name: Practice\n";
const corpus = [
{ name: "minimal", descriptor, catalog, valid: true },
{ name: "optional Evidence", descriptor: descriptor + "evidence:\n source:\n type: http\n uris: [https://example.com/manual.md]\n", catalog, valid: true },
{ name: "duplicate YAML key", descriptor: descriptor + " id: practice\n", catalog, valid: false },
{ name: "ambiguous document", descriptor: descriptor + "---\n" + descriptor, catalog, valid: false },
{ name: "unknown workspace field", descriptor: descriptor + " secret: VERY_SECRET_VALUE\n", catalog, valid: false },
{ name: "database binding in authored descriptor", descriptor: descriptor + "dwh: {password: VERY_SECRET_VALUE}\n", catalog, valid: false },
{ name: "duplicate catalog id", descriptor, catalog: catalog + " - id: practice\n name: Practice\n", valid: false },
{ name: "unknown catalog field", descriptor, catalog: catalog + "secret: VERY_SECRET_VALUE\n", valid: false },
{ name: "invalid catalog YAML", descriptor, catalog: catalog + "schema_version: 1\n", valid: false },
{ name: "unsafe Evidence URI", descriptor: descriptor + "evidence:\n source:\n type: http\n uris: [https://user:VERY_SECRET_VALUE@example.com/file]\n", catalog, valid: false },
];
test.each(corpus)("CLI and runtime agree: $name", (fixture) => {
const root = directory();
mkdirSync(join(root, "practice"), { recursive: true });
writeFileSync(join(root, "thoth-workspaces.yaml"), fixture.catalog);
writeFileSync(join(root, "practice/workspace.yaml"), fixture.descriptor);
let accepted = true;
try { parseWorkspaceCatalogYaml(fixture.catalog); parseWorkspaceYaml(fixture.descriptor); }
catch { accepted = false; }
expect(accepted).toBe(fixture.valid);
const checked = cli("validate", "--directory", root);
expect(checked.status).toBe(fixture.valid ? 0 : 1);
expect(checked.report.ok).toBe(accepted);
expect(checked.stdout + checked.stderr).not.toContain("VERY_SECRET_VALUE");
if (!fixture.valid) {
expect(checked.report.issues[0].document).not.toBe(".");
expect(checked.report.issues[0].correction.length).toBeGreaterThan(10);
}
});
test("prepare refuses existing directories and invalid options without changing documents", () => {
const root = directory();
expect(cli("prepare", "--directory", root, "--id", "practice", "--name", "Practice").status).toBe(0);
const before = readFileSync(join(root, "thoth-workspaces.yaml"), "utf8");
expect(cli("prepare", "--directory", root, "--id", "other", "--name", "Other").report.issues[0].code).toBe("destination_exists");
expect(readFileSync(join(root, "thoth-workspaces.yaml"), "utf8")).toBe(before);
expect(cli("validate", "--directory", root, "--typo", "VERY_SECRET_VALUE").status).toBe(2);
});
test("validation rejects directory mismatches, missing references and symlinks without following them", () => {
const root = directory();
cli("prepare", "--directory", root, "--id", "practice", "--name", "Practice");
mkdirSync(join(root, "unlisted"));
expect(cli("validate", "--directory", root).report.issues.some((i: {code: string}) => i.code === "unlisted_directory")).toBe(true);
rmSync(join(root, "unlisted"), { recursive: true });
writeFileSync(join(root, "practice/workspace.yaml"), descriptor.replace("name: Practice", "name: Different"));
expect(cli("validate", "--directory", root).report.issues[0].code).toBe("catalog_mismatch");
writeFileSync(join(root, "practice/workspace.yaml"), descriptor + "evidence:\n source:\n type: filesystem\n uri: practice/evidence\n");
expect(cli("validate", "--directory", root).report.issues[0].code).toBe("missing_reference");
symlinkSync(roots[roots.length - 1], join(root, "practice/evidence"), "dir");
expect(cli("validate", "--directory", root).report.issues[0].code).toBe("unsafe_reference");
});
test("local Evidence checks references and YAML frontmatter, and explicitly defers canonical content validation", () => {
const root = directory();
cli("prepare", "--directory", root, "--id", "practice", "--name", "Practice");
writeFileSync(join(root, "practice/workspace.yaml"), descriptor + "evidence:\n schema_version: 2\n source:\n type: filesystem\n uri: practice/evidence\n patterns: ['curated/**/*.md']\n");
mkdirSync(join(root, "practice/evidence/curated/domain"), { recursive: true });
const evidence = join(root, "practice/evidence/curated/domain/rule.md");
writeFileSync(evidence, "---\nschema_version: 4\nid: evidence:rule\nkind: domain\nlanguage: en\npurposes: [sql_generation]\n---\n# Rule\n\n## Rule\nUse the order number.\n");
const checked = cli("validate", "--directory", root);
expect(checked.status).toBe(0);
expect(checked.report.workspaces).toEqual([{ id: "practice", evidence: "local-files" }]);
expect(checked.report.deferred_checks).toContain("practice:evidence-content-provenance-and-indexing");
writeFileSync(evidence, "---\nid: first\nid: VERY_SECRET_VALUE\n---\n# Rule\n");
const invalid = cli("validate", "--directory", root);
expect(invalid.status).toBe(1);
expect(invalid.report.issues[0].document).toBe("practice/evidence/curated/domain/rule.md");
expect(invalid.stdout).not.toContain("VERY_SECRET_VALUE");
writeFileSync(join(root, "practice/workspace.yaml"), descriptor + "evidence:\n source:\n type: filesystem\n uri: practice/evidence\n patterns: [missing.md]\n");
expect(cli("validate", "--directory", root).report.issues[0].code).toBe("missing_reference");
});
test("prepare creates documents accepted by the runtime and validate is repeatable without installation or services", () => {
const root = directory();
const prepared = cli("prepare", "--directory", root, "--id", "practice", "--name", "Practice");
expect(prepared.stderr).toBe("");
expect(prepared.status).toBe(0);
expect(prepared.report.ok).toBe(true);
const catalog = readFileSync(join(root, "thoth-workspaces.yaml"), "utf8");
const descriptor = readFileSync(join(root, "practice/workspace.yaml"), "utf8");
expect(parseWorkspaceCatalogYaml(catalog).workspaces[0].id).toBe("practice");
expect(parseWorkspaceYaml(descriptor).evidence).toBeUndefined();
for (let index = 0; index < 2; index++) {
const checked = cli("validate", "--directory", root);
expect(checked.status).toBe(0);
expect(checked.report.workspaces).toEqual([{ id: "practice", evidence: "absent" }]);
}
expect(readFileSync(join(root, "thoth-workspaces.yaml"), "utf8")).toBe(catalog);
expect(readFileSync(join(root, "practice/workspace.yaml"), "utf8")).toBe(descriptor);
});
@@ -0,0 +1,5 @@
# Local-only rollback to the full-shell frontend before visual integration.
# Apply after local installation profiles, with --no-build and --no-deps.
services:
frontend:
image: thothii-frontend:before-visual-shell-merge-20260913
@@ -2,6 +2,10 @@
# Replace every absolute path before using this as an advanced reference.
schemaVersion: 2
profile: local
# Mac standalone example; the Omics server requires embedded/upstream separately.
shell:
mode: full
defaultLocale: en
projectDirectory: "<abs>/projects/ThothII"
envFile: "<abs>/projects/ThothII/deploy/psd/operator.env"
workspaceRepository:
+10
View File
@@ -8,6 +8,11 @@ cp deploy/secrets/thothii.secrets.example deploy/secrets/thothii.secrets
chmod 600 deploy/secrets/thothii.secrets
```
For a normal local installation, `tht setup --complete` creates the active bundle at
`deploy/local/secrets/thothii.secrets` and creates the two Catalog password files beside it. The
generated `deploy/local/operator.env` contains only absolute paths to those files; never copy
secret values into `operator.env`.
The file uses strict `KEY=VALUE` lines (comments and blank lines are allowed). The supported
installation keys are `THT_MODEL_API_KEY`, `THT_DWH_API_KEY`, `THT_CA`, `THT_SSL_CA`,
`THT_OIDC_CLIENT_SECRET`, and `THT_AUTHENTIK_API_TOKEN`. Installation Model Catalog providers may
@@ -29,6 +34,11 @@ Session and metadata-generation runtimes read only the provider key named by
credential reference, not their execution lifecycle. Pi-owned authentication remains available only
to session-only built-in providers through `authentication.mode: pi_auth`.
Workspace database credentials are intentionally not part of this global bundle. Configure each
workspace's database binding, password/token, tunnel key and CA in Database Management; the
installation stores those values in its encrypted workspace secret store. The workspace Git
repository may declare database identity and Evidence, but must never contain these credentials.
Do not add vector or embedding endpoint credentials to the bundle. Active operator manuals use
internal Qdrant and Ollama services, so vector/embedding runtime endpoint secrets are not part of
the supported installation contract.
+21 -1
View File
@@ -1,6 +1,26 @@
#!/bin/sh
set -eu
test "$(cat /usr/share/nginx/html/config.js)" = 'window.__THOTHII_CONFIG__ = {};'
# The generic image contains an empty fallback; installed containers mount the generated
# public projection over it. An optional path lets host projection tests use this same check.
config_file=${1:-/usr/share/nginx/html/config.js}
test -f "$config_file" && test -r "$config_file"
config=$(tr -d '[:space:]' < "$config_file")
case "$config" in
'window.__THOTHII_CONFIG__={};')
test -z "${THT_FRONTEND_CONFIG_REVISION:-}"
;;
*)
# Installation Load validates BCP47 syntax. Here check only the public payload shape;
# catalog availability and fallback belong to the frontend, not the generic image.
locale='[A-Za-z][A-Za-z0-9]*(-[A-Za-z0-9]+)*'
full="\"mode\":\"full\",\"defaultLocale\":\"$locale\""
embedded="\"mode\":\"embedded\",\"defaultLocale\":\"$locale\",\"adapter\":\"omics-portal\""
if ! printf '%s\n' "$config" | LC_ALL=C grep -Eq "^window\.__THOTHII_CONFIG__=\{\"backendBaseUrl\":\"/api\",\"shell\":\{($full|$embedded)\}\};$"; then
echo "Invalid frontend public runtime configuration" >&2
exit 1
fi
;;
esac
printf '%s\n' "frontend runtime config smoke: ok"
@@ -0,0 +1,101 @@
# ADR 0021 — Shell separati e adapter sostituibile per il portale
- Stato: accettato
- Data: 2026-09-13
## Decisione
ThothII espone due modalità di installazione, selezionate da `shell.mode`:
- `embedded` (default): ThothII è ospitato da Omics Portal. Non renderizza alcun header e
riceve dal portale lingua, tema e fullscreen. L'accesso resta verificato dal server.
- `full`: ThothII è autonomo. Renderizza il proprio header, con selettore lingua, tema,
fullscreen e nome utente. Il click sul nome apre il logout. Non mostra mai la rotellina o
altri comandi amministrativi del portale. Mantiene un rail vuoto a sinistra di almeno 20 px.
Il fatto che la shell sia `full` è distinto dallo stato `fullscreen`: la prima decide quale
contenitore viene renderizzato, il secondo indica se è attiva la Fullscreen API del browser.
L'icona passa da “entra in fullscreen” a “torna alla modalità normale”; anche `Esc` aggiorna lo
stato visualizzato.
La configurazione installata resta semplice e retrocompatibile. Sul Mac di sviluppo il profilo
locale userà:
```yaml
shell:
mode: full
defaultLocale: en
```
Il deploy sul server userà invece:
```yaml
shell:
mode: embedded
adapter: omics-portal
```
`defaultLocale` indica la lingua iniziale della shell full; in embedded la fonte autorevole resta
il portale.
L'adapter è l'unico confine tra ThothII e il portale. La sua interfaccia pubblica è volutamente
profonda e minima: consegna solo snapshot dello stato, senza esporre comandi, token, identità o
dettagli di trasporto.
```ts
export type HostShellState = {
locale: string; // BCP-47, inizialmente it/en
theme: "light" | "dark";
fullscreen: boolean;
};
export interface PortalAdapter {
subscribe(
onState: (state: HostShellState) => void,
onError: (error: Error) => void,
): () => void;
}
```
`OmicsPortalAdapter` è l'implementazione corrente. Un adapter per un altro portale potrà
sostituirlo senza modificare shell, i18n o workflow. In `full` l'adapter non viene istanziato:
lo stato è gestito internamente dalla shell.
Per l'integrazione oggi operativa, che monta la SPA direttamente nel DOM di Omics Portal,
l'adapter legge la lingua effettiva dal selettore Omics, osserva l'attributo del tema e ascolta
il fullscreen del documento. Selettori e osservatori restano privati dell'implementazione Omics.
La lingua segue il normale ricaricamento Django; tema e fullscreen cambiano nella pagina aperta.
La revisione approvata del 2026-09-13 elimina il precedente handshake a eventi: non servono
messaggi personalizzati, versioni di trasporto o timeout di avvio. Un futuro adapter potrà usare
un diverso trasporto senza modificare l'interfaccia applicativa.
Se `shell` o l'adapter embedded sono omessi, si usa `embedded` con `omics-portal`. Questo default
supporta il documento Omics esistente; nomi adapter sconosciuti o dati host mancanti producono
un errore esplicito, senza attivare la shell full.
## Confini che restano invariati
L'identità e l'autorizzazione del backend non vengono ricostruite nel browser. In embedded,
Omics Portal continua a gestire login e logout e la catena server-side `auth_request` continua a
fornire i principal header già previsti. Lo stato UI non dichiara l'utente autenticato: il modulo
di accesso usa la verifica backend esistente anche alla riconnessione e al ritorno alla pagina.
Un rifiuto su una singola operazione non equivale automaticamente alla perdita dell'accesso.
La lingua UI e la lingua di interazione con il modello restano separate dalla lingua del
workspace; il relativo contratto è in [ADR 0022](0022-separate-ui-locale-from-session-interaction-language.md).
## Alternative scartate
- Duplicare l'header di Omics in embedded: crea due fonti di stato e incompatibilità visive.
- Spargere controlli `if embedded/full` nei componenti: lega ogni pagina al portale.
- Trasmettere utente o token nel bridge: aumenta superficie e accoppia UI e autenticazione.
- Introdurre un protocollo completo request/response: non aggiunge funzionalità richiesta.
- Usare `profile` per distinguere le shell: `profile` descrive la topologia dell'installazione,
non la sua presentazione.
## Conseguenze
La soluzione richiede un adapter nel frontend che osserva il documento condiviso e mantiene la
logica di shell locale a ThothII. Il backend non necessita di un nuovo protocollo di autenticazione
o di una nuova sessione browser. Un nuovo portale deve soddisfare anche il contratto server di
identità fidata: la sola sostituzione della classe UI non sostituisce quel contratto.
@@ -0,0 +1,51 @@
# Separate UI locale from session interaction language
status: accepted
ThothII distinguishes three language concepts:
- `workspace.language` remains the language of workspace-owned documents, descriptions and Evidence;
- `ui_locale` controls deterministic ThothII chrome such as labels, form help, placeholders, errors,
accessibility text and review-widget chrome;
- `interaction_language` is persisted in a session and controls model-generated questions,
explanations and reviewer proposals.
The initial locale catalog supports Italian and English and uses extensible BCP-47 language tags.
Missing deterministic translations fall back to English. The selected UI locale supplies the default
interaction language when a new session is created. A resumed session always uses its persisted
interaction language; changing the host or full-shell UI locale must not silently rewrite an existing
session or make its model output switch language mid-workflow.
The distinction is required because the current workspace contract already uses `language` for
content and the PSD workspace is Italian. Reusing that field for a browser preference would make a
visual choice mutate domain content semantics. The model receives the session interaction language
through the session/Pi workflow context. SQL, identifiers, database values and other technical
artifacts remain governed by their existing contracts and are not translated as UI strings.
In `full`, the local shell owns `ui_locale` and supplies it when starting a new session. In
`embedded`, the host adapter is authoritative for `ui_locale`; ThothII applies host changes to
deterministic UI immediately while preserving the interaction language of any active session.
We considered using only `workspace.language`, using only a global browser locale, and translating
the model output after generation. The first conflates domain content with UI preference; the second
cannot preserve a session's language or follow the host portal; and the third would be unsafe for
structured reviewer decisions and would not control the model's reasoning or proposal language.
## Considered Options
- One mutable `language` field for workspace, UI and session was rejected because the fields have
different owners and lifecycles.
- Client-only translation of reviewer choices was rejected because choices can be generated by the
model and must be requested in the intended language.
- An English-only deterministic chrome was rejected because embedded and full installations must
follow the selected host/user language.
## Consequences
- Session creation and the persisted manifest gain an explicit interaction-language value.
- Legacy manifests without that value use the workspace language, pinned idempotently on first
resume; the browser locale must not determine this compatibility value.
- Resume must read that value from the manifest and must not accept a new locale as an override.
- The workflow prompt contract and deterministic reviewer-widget builders need a locale-aware input.
- Frontend strings need a catalog and stable keys; backend events should expose stable codes where
the frontend is responsible for localization.
+135
View File
@@ -0,0 +1,135 @@
# Rendering full ed embedded
ThothII ha una sola applicazione React, una sola build Vite e gli stessi servizi
backend. «Doppio rendering» significa due modi di ospitare quella applicazione,
non due versioni delle pagine e non rendering React sul server. Django renderizza
il contenitore Omics; React renderizza ThothII nel browser, dentro `#root`.
## Tre decisioni indipendenti
| Decisione | Configurazione | Effetto |
| --- | --- | --- |
| Distribuzione | `profile: local` oppure `server` | Compose, percorsi e vincoli operativi |
| Presentazione | `shell.mode: full` oppure `embedded` | Proprietario di header e preferenze |
| Autenticazione | `auth.yaml` local/OIDC oppure `AUTH_MODE=upstream` | Chi verifica l'identità, come arriva al backend |
Il Mac usa **full + local**, con lingua iniziale inglese. L'integrazione Omics
usa **embedded + upstream**, con accesso già verificato dal portale. Un server
autonomo può usare **full + oidc**. Cambiare `shell.mode` non abilita un metodo
di autenticazione e non modifica permessi o proprietari delle sessioni.
Full con upstream può visualizzare un'identità già verificata dal proxy, ma non
ha un logout ThothII disponibile: non è il profilo autonomo con login/logout.
Embedded non avvia login locale o OIDC anche se il backend è configurato così;
questa combinazione non realizza il login unico Omics e non va usata come fallback.
## Composizione comune
```mermaid
flowchart TD
CONFIG["config.js pubblico"] --> SHELL["ShellProvider"]
FULL["Preferenze full nel browser"] --> SHELL
HOST["Documento Omics"] --> ADAPTER["OmicsPortalAdapter: solo presentazione"]
ADAPTER --> SHELL
SHELL --> GATE["AuthGate: verifica GET /me"]
GATE --> APP["AppShell: stesse pagine, sessioni e amministrazione"]
AUTH["Backend: cookie locale/OIDC o identità upstream"] --> GATE
```
`ShellProvider` risolve la configurazione, applica lingua/tema e monta i contenuti
solo dopo uno snapshot host valido in embedded. `AuthGate` verifica l'accesso;
`AppShell` e le pagine non devono leggere selettori o eventi specifici di Omics.
Il cambio utente smonta lo stato applicativo della precedente identità.
## Full
- Header ThothII rosso Omics `#CB333B` in entrambi i temi; logo interamente chiaro.
- Selettore EN/IT, tema light/dark, fullscreen e nome verificato dell'utente.
- Menu del nome con logout soltanto per local/OIDC; nessuna rotellina admin.
L'amministrazione resta nella navigazione applicativa, secondo i permessi.
- Margine sinistro vuoto e simmetrico al destro: `max(20px, 1.5rem)`, normalmente
24px con radice a 16px. Non è una seconda sidebar di navigazione.
- Lingua e tema ricordati sullo stesso origin in `localStorage`, nelle chiavi
`thothii:shell:locale` e `thothii:shell:theme`. Non sono preferenze server per
utente. In assenza di preferenze: `defaultLocale` e tema light.
- Fullscreen usa `document.documentElement.requestFullscreen()` e
`document.exitFullscreen()`: nasconde il contorno del browser dove supportato.
L'icona cambia sullo stato reale, anche dopo Esc; un rifiuto mostra un errore.
Non è un semplice ingrandimento CSS e non scatta automaticamente all'accesso.
## Embedded
- Nessun header ThothII, selettore lingua, toggle tema, login o logout autonomo.
I controlli rimangono nell'header generale Omics.
- React è nello stesso documento della pagina `/kokoro/datamart-builder/`, non
in un iframe. Non serve `postMessage` né un secondo protocollo di sessione.
- L'adapter legge la lingua Django già confermata, osserva il tema del documento
e ascolta il fullscreen reale. Le azioni rimangono di proprietà del portale.
- Un contesto Omics mancante o invalido mostra un errore d'integrazione; non
passa silenziosamente a full e non offre un secondo login.
- Il portale assegna l'altezza disponibile sotto il proprio header: catena flex
con `min-height: 0`, root contenuto e altezza applicativa vincolata al contenitore.
Il contratto ThothII espone `--thoth-app-height` (fallback `100dvh`); verificare
il contenitore reale, non presumere che l'intera viewport appartenga a React.
Il template Omics mantiene inoltre i suoi override di compatibilità.
Il reset CSS è limitato al mount React e ai popup dell'applicazione, senza
richiedere CSS `@scope`. I token e i popup seguono il tema applicativo. Questo
non rende indipendenti fogli di stile arbitrari caricati dal portale: la verifica
del documento condiviso rimane necessaria a ogni integrazione.
## Caricamento e configurazione pubblica
Il descrittore installato è la sorgente di verità. Il CLI genera
`generated/frontend/config.js` e il suo mount di sola lettura nella proiezione
`generated/compose.models.yaml`. Il file pubblico contiene solo `backendBaseUrl`
e `shell`, mai identità, token, password o percorsi host. Va caricato **prima** del
modulo React e servito senza cache. Nessuna build separata è richiesta per
cambiare modalità; occorre rigenerare e applicare i mount tramite il lifecycle.
L'ordine Omics è: config pubblico → override del solo prefisso API → asset dal
manifest Vite. L'override deve conservare `shell`; l'adapter non configura il proxy.
Il default completo di shell omessa è embedded/en/omics-portal. Il CLI normalizza
anche singoli campi omessi; un oggetto `shell` scritto manualmente nel browser
deve invece contenere `mode` e `defaultLocale`, altrimenti viene rifiutato.
## Lingua, continuità e dati
La lingua UI traduce il testo dell'applicazione, non i contenuti di dominio.
Alla creazione, la lingua UI viene acquisita come `interactionLanguage` di ripiego.
Il CLI riconosce la lingua della domanda originale e salva `interaction_language`
nel manifest; usa il ripiego solo per input troppo brevi, ambigui o composti da codice.
Questa lingua governa domande, spiegazioni, scelte e controlli HITL. Il gate la
include nei descrittori e il frontend la applica al sottoalbero dei widget,
senza cambiare la lingua della navigazione.
Alla ripresa vale la lingua salvata, non l'ultima scelta dell'header. Per i manifest
precedenti senza campo viene riconosciuta e fissata la lingua della domanda,
con la lingua workspace disponibile alla prima ripresa come ripiego.
Il cambio lingua Omics invia il form Django e ricarica la pagina. ThothII conserva
solo l'ID della selezione in `sessionStorage`, separato per pathname, issuer e
subject. Riapre i documenti, non avvia una generazione. Bozze non inviate e modifiche
non salvate richiedono conferma prima della navigazione; non sono una trascrizione
salvata. La ripresa operativa resta esplicita.
## Punti di implementazione e manutenzione
| Sorgente | Responsabilità |
| --- | --- |
| `tools/tht/internal/config/shell.go` | Normalizzazione e validazione del descrittore |
| `tools/tht/internal/modelprojection/projection.go` | Config pubblico e mount generati |
| `frontend/src/api/runtime-config.ts` | Validazione browser e prefisso API same-origin |
| `frontend/src/shell/host/ShellProvider.tsx` | Composizione, preferenze e tema |
| `frontend/src/shell/host/FullHeader.tsx` | Controlli solo full |
| `frontend/src/shell/host/OmicsPortalAdapter.ts` | Conoscenza del documento Omics |
| `frontend/src/auth/AuthGate.tsx` | Accesso e ricontrolli al ritorno alla pagina |
| `backend/src/auth/auth.ts` e `principal.ts` | Verifica server dell'identità |
Per un altro portale servono un'implementazione del
[PortalAdapter](../contracts/portal-shell-adapter-v1.md), la sua registrazione nei
validatori CLI/browser e nel punto di composizione, oltre al
[contratto di autenticazione server](../install/authentication-upstream.md).
Il nome di una classe non è un plugin caricabile dinamicamente da YAML.
Procedure: [configurazione e deploy](../operations/shell-and-localization.md),
[autenticazione](authentication.md), [accettazione](../testing/authentication-manual-acceptance.md).
+44 -5
View File
@@ -1,18 +1,27 @@
# Authentication architecture
ThothII has two production authentication modes: `local` and generic `oidc`. The host operator
surface is one CLI, `tht`; there is no separate authentication executable. The backend owns
opaque browser sessions and authorization, while `tht` owns protected configuration and local-user
files.
ThothII supports `local` and generic `oidc` through protected `auth.yaml`, plus
the trusted-proxy `upstream` path used by Omics. The host operator surface is
one CLI, `tht`; there is no separate authentication executable. The backend owns
authorization in all paths and opaque browser sessions only in local/OIDC.
`tht` owns protected local/OIDC configuration and local-user files; upstream
identity is supplied per request by the authenticated server proxy.
Presentation is separate: [full/embedded rendering](application-shell.md) does
not select authentication. The Mac uses full/local; Omics uses embedded/upstream;
a standalone server can use full/OIDC. Do not configure a second ThothII OIDC
login simply because Omics itself authenticates users through Authentik.
```mermaid
flowchart TB
BROWSER["Browser"] --> BOUNDARY["Authentication boundary"]
BOUNDARY --> LOCAL["Local users\nArgon2id hashes"]
BOUNDARY --> OIDC["OIDC provider\nAuthorization Code PKCE"]
BOUNDARY --> UPSTREAM["Trusted proxy\nVerified portal session"]
OIDC --> GROUPS["Groups claim\nexact mapping"]
LOCAL --> PRINCIPAL["Thoth principal"]
GROUPS --> PRINCIPAL
UPSTREAM --> PRINCIPAL
PRINCIPAL --> ROLES["Roles"]
ROLES --> PERMISSIONS["Permissions"]
PERMISSIONS --> ROUTES["Protected routes"]
@@ -21,6 +30,13 @@ flowchart TB
## Configuration and trust boundaries
The following protected-file configuration applies to local/OIDC. Upstream uses
`AUTH_MODE=upstream` without a mounted `auth.yaml` or authentication runtime
projection. The backend refuses both authorities together. `AUTH_MODE=none`
and `mock` are development/test modes, not production fallbacks. The exact
upstream setup, header contract, proxy hops and origin checks are in the
[server integration guide](../install/authentication-upstream.md).
The installation descriptor points to an operator-controlled authentication directory. It contains
non-secret `auth.yaml` and, for local mode, `users.yaml`. POSIX installations use a private
directory and owner-only regular files; Windows uses equivalent owner-only ACLs. Secret values are
@@ -49,6 +65,10 @@ Authentik is the first certified group-catalog adapter, not a special browser lo
## Group authorization
This section describes **ThothII's direct OIDC login**, not the embedded Omics
path. Omics checks its own capability and administrator status and supplies
normalized identity headers; ThothII does not repeat the OIDC groups exchange.
OIDC must return a direct, non-empty `groups` claim whose value is a JSON array of strings.
Missing, malformed, indirect, or overage-style claims fail closed. The browser callback returns
HTTP 401 with the generic code `oidc_callback_failed`; it does not expose the internal reason.
@@ -100,6 +120,10 @@ prerequisite fails.
## Browser sessions
This section applies only to **local and direct OIDC**. Upstream reuses the
portal's authenticated session at the proxy boundary, not a ThothII cookie;
its `/me` response has `session: null` and `csrfToken: null`.
The browser receives only an opaque `HttpOnly`, `SameSite=Lax` cookie named `thothii_session`.
State-changing cookie requests require the in-memory CSRF token, same-origin `Origin`, and Fetch
Metadata checks when present. The frontend never stores bearer tokens or session secrets in Web
@@ -116,5 +140,20 @@ affected sessions. Authentication configuration revision changes invalidate all
reload. Logout deletes the server record. Backup restore excludes active sessions and OIDC state,
recreates empty private auth-state directories, and therefore forces reauthentication.
Full local/OIDC logout calls `POST /auth/logout`, revokes the server session and
clears its cookie. It does not call the identity provider's global logout. If the
provider still has an SSO session, the next OIDC login can complete without
another password prompt. Embedded has no ThothII logout control: use the portal.
## Access revalidation
The frontend treats `/me` as the access authority. In embedded it does not fetch
`/auth/config` or offer local/OIDC login. On focus, pageshow, visibility return
and event-stream reconnection it rechecks access. A 401/403 from this probe clears
protected state; a 403 on one operation is not automatically an app-wide logout.
Neither the DOM adapter nor the proxy's initial SSE check guarantees instantaneous
revocation of streams already open in other tabs.
See the [local guide](../install/authentication-local.md), [generic OIDC guide](../install/authentication-oidc.md),
and [Authentik guide](../install/authentik.md) for operator procedures.
[Authentik guide](../install/authentik.md), [upstream integration](../install/authentication-upstream.md),
and [manual acceptance matrix](../testing/authentication-manual-acceptance.md).
+8
View File
@@ -44,6 +44,14 @@ Dipendenze principali:
## Session sequence
The shared frontend is wrapped by `ShellProvider` (full preferences or a
replaceable portal presentation adapter), then `AuthGate` (backend identity),
then `AppShell`. Omics-specific DOM details belong only to `OmicsPortalAdapter`;
credentials and principal validation belong to the server, never that adapter.
Full/embedded do not duplicate the session workflow below. See
[rendering architecture](application-shell.md) and
[upstream identity](../install/authentication-upstream.md) for both boundaries.
The main path starts with a user question and ends with an SSE event. Reviewer decisions use the same channel and are persisted by the harness.
```mermaid
+27 -15
View File
@@ -4,8 +4,11 @@
ThothII is a **human-in-the-loop datamart builder**. It turns a natural-language question into validated SQL, and optionally a dbt datamart, through a **deterministic eight-phase NL-to-SQL workflow** in which the model *proposes* and a human reviewer *decides* at gates.
Production authentication uses local authentication or generic OIDC. `tht` is the only operator CLI.
For sessions, roles, groups, diagnostics, and recovery, see the [authentication documentation](authentication.md).
Production authentication uses local authentication, generic OIDC, or the
trusted-proxy upstream path used by Omics. `tht` is the operator CLI for the
installation and local/OIDC configuration. For roles and recovery, see
[authentication](authentication.md). One React build supports full and embedded;
[rendering architecture](application-shell.md) separates presentation from identity.
```mermaid
flowchart LR
@@ -78,20 +81,20 @@ The model proposes; a human reviewer decides at gates through widgets:
The frontend renders these widget descriptors (registry in `src/widgets/`). It rebuilds the live transcript in memory from the SSE stream (`src/store/sessionStore.ts`); it is **not persisted**.
## Curated and immutable Evidence
## Evidence sources, local authority and search projections
The workspace repository is the publication boundary. The curator prepares `evidence/source/`,
reviews units in `evidence/curated/`, validates them, and merges them. With `evidence.schema_version: 2`,
the runtime materializes the full `evidence/` tree from the exact Git commit, but the renderer passes
only `curated/**/*.md` from the immutable revision root to preprocessing. Sources, manifests, and
evaluation data remain available for traceability. The runtime never modifies, stages, commits, or
publishes the authoring repository.
The workspace repository publishes revision-pinned workspace identities and Evidence sources.
An initialized editable Evidence archive is maintained locally through external editors and
explicit consolidation; normal preprocessing or source refresh must not overwrite its manual
corrections. File save, active search generation and a later human Git commit/push are separate
outcomes. See [the current Evidence contract](../contracts/curated-evidence-v4.md) and
[source/publication boundaries](../contracts/workspace-evidence-v3.md).
Before indexing, the curated corpus from the pinned revision is validated. Each workspace has two
physical Qdrant collections with different lifecycles. `reference` contains Schema, relationships,
and Evidence and may be replaced or cleared by preprocessing; `memory` contains `memory` and
`solved_question` records and is never preprocessing output. Only the `reference` collection has the
sparse `bm25` vector with `idf`, and only the Evidence stage writes sparse values.
Each workspace has separate Qdrant collections. `reference` contains Schema, relationships and
Evidence and may be replaced or cleared by preprocessing. Memory uses its own dense/BM25
projection, rebuilt from authoritative PostgreSQL cards rather than old vector payloads or
session artifacts. Both collections can contain sparse vectors; their ownership and cleanup
lifecycles remain separate. See [Memory](../gestione-memory.md).
The Administration control can clear the replaceable reference collection, LSH, corpus, and derived
checkpoints. The operation preserves the memory collection and makes preprocessing required before
@@ -101,7 +104,9 @@ the core can admit a new session.
- `tht -c`/`--config` is a **per-command** option. It must follow the subcommand, never precede it (`ThtRunner.buildArgv` enforces this).
- `--json` output must be plain JSON on stdout. It is a machine-readable contract.
- UI strings are in English. Document *content* stays in the workspace language because it is the actual data; only chrome and labels are in English.
- UI strings support English and Italian, with English fallback. Session interaction language
is pinned at creation; document content remains in the workspace language. See
[shell and localization](../operations/shell-and-localization.md).
- Each workspace defines identity and optional Evidence only. The PostgreSQL Metadata Catalog defines
its DWH target and binding; secrets remain in the protected workspace secret store.
- Settings are global (`backend/data/settings.json`: workspace/thinking); provider/model choices are
@@ -110,6 +115,13 @@ the core can admit a new session.
## Runtime composition
PostgreSQL is the current database-metadata authority for all core consumers, not a deferred
catalog-to-core integration. The historical research on a separate catalog service and
`annotations.yaml` publication is superseded by ADRs 0004 and 0016. Preserve read-only DWH access,
installation-local bindings/secrets, revision-pinned workspace identity, and fail-closed readiness
when changing those boundaries. The exact snapshot interface is the
[Catalog Schema Snapshot contract](../contracts/catalog-schema-snapshot.md).
The local stack is started by `./scripts/run-stack.sh` after the installation descriptor and
`deploy/env/local.env` exist. `core` includes Pi. `catalog-db`, Qdrant, and the Ollama embedding
service are internal Compose services; only the DWH and model-provider endpoint remain external.
Binary file not shown.

Before

Width:  |  Height:  |  Size: 132 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 131 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 189 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 186 KiB

@@ -1,32 +0,0 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>Archify automated browser evidence · thothii-core-sequence.html</title>
<style>
*{box-sizing:border-box}body{margin:0;padding:24px;background:#e9eef5;color:#172033;font:14px/1.5 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}header{max-width:1500px;margin:0 auto 18px}h1{margin:0 0 6px;font-size:20px}p{margin:0;color:#526176}.grid{max-width:1500px;margin:auto;display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:18px}figure{margin:0;padding:10px;background:white;border:1px solid #c9d4e3;border-radius:12px;box-shadow:0 10px 30px rgba(15,23,42,.08)}img{display:block;width:100%;height:auto;border:1px solid #e2e8f0}figcaption{padding:9px 4px 2px;color:#526176}@media(max-width:900px){.grid{grid-template-columns:1fr}}
</style>
</head>
<body>
<header><h1>Automated browser evidence</h1><p>thothii-core-sequence.html · visual-check containment pass · perceptual visual review pending</p></header>
<main class="grid">
<figure>
<img src="thothii-core-sequence.visual-check.1440x900.light.png" alt="light 1440 by 900">
<figcaption><strong>LIGHT</strong> · 1440×900 · containment pass</figcaption>
</figure>
<figure>
<img src="thothii-core-sequence.visual-check.1440x900.dark.png" alt="dark 1440 by 900">
<figcaption><strong>DARK</strong> · 1440×900 · containment pass</figcaption>
</figure>
<figure>
<img src="thothii-core-sequence.visual-check.2048x1320.light.png" alt="light 2048 by 1320">
<figcaption><strong>LIGHT</strong> · 2048×1320 · containment pass</figcaption>
</figure>
<figure>
<img src="thothii-core-sequence.visual-check.2048x1320.dark.png" alt="dark 2048 by 1320">
<figcaption><strong>DARK</strong> · 2048×1320 · containment pass</figcaption>
</figure>
</main>
</body>
</html>
@@ -1,548 +0,0 @@
{
"schemaVersion": 1,
"ok": true,
"command": "visual-check",
"evidenceKind": "automated-browser",
"status": "pass",
"visualReview": "pending",
"artifact": {
"path": "/Users/mp/projects/ThothII/docs/architecture/thothii-core-sequence.html",
"sha256": "b521eb942f7889cfc3a5e29010546ba59eeab1cf485c22c533d271ccef9a28d5",
"bytes": 716881
},
"state": {
"detail": "read",
"motion": "still"
},
"chrome": {
"status": "available",
"executable": "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"
},
"diagnostics": [],
"containment": {
"status": "pass",
"viewports": [
{
"width": 1440,
"height": 900,
"theme": "light",
"innerWidth": 1440,
"innerHeight": 900,
"scrollWidth": 1440,
"scrollHeight": 900,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1194,
"diagramWidth": 1164,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1600,
"height": 1000,
"theme": "light",
"innerWidth": 1600,
"innerHeight": 1000,
"scrollWidth": 1600,
"scrollHeight": 1000,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1242,
"diagramWidth": 1212,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1920,
"height": 1080,
"theme": "light",
"innerWidth": 1920,
"innerHeight": 1080,
"scrollWidth": 1920,
"scrollHeight": 1080,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1402,
"diagramWidth": 1372,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 2048,
"height": 1320,
"theme": "light",
"innerWidth": 2048,
"innerHeight": 1320,
"scrollWidth": 2048,
"scrollHeight": 1320,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1818,
"diagramWidth": 1768,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 41,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
}
]
},
"readability": {
"status": "pass",
"minimumProjectedNodeTextPx": 6,
"viewports": [
{
"width": 1440,
"height": 900,
"theme": "light",
"innerWidth": 1440,
"innerHeight": 900,
"scrollWidth": 1440,
"scrollHeight": 900,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1194,
"diagramWidth": 1164,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1600,
"height": 1000,
"theme": "light",
"innerWidth": 1600,
"innerHeight": 1000,
"scrollWidth": 1600,
"scrollHeight": 1000,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1242,
"diagramWidth": 1212,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1920,
"height": 1080,
"theme": "light",
"innerWidth": 1920,
"innerHeight": 1080,
"scrollWidth": 1920,
"scrollHeight": 1080,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1402,
"diagramWidth": 1372,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 2048,
"height": 1320,
"theme": "light",
"innerWidth": 2048,
"innerHeight": 1320,
"scrollWidth": 2048,
"scrollHeight": 1320,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1818,
"diagramWidth": 1768,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 41,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
}
]
},
"viewerChrome": {
"status": "pass",
"viewports": [
{
"width": 1440,
"height": 900,
"theme": "light",
"innerWidth": 1440,
"innerHeight": 900,
"scrollWidth": 1440,
"scrollHeight": 900,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1194,
"diagramWidth": 1164,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1600,
"height": 1000,
"theme": "light",
"innerWidth": 1600,
"innerHeight": 1000,
"scrollWidth": 1600,
"scrollHeight": 1000,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1242,
"diagramWidth": 1212,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1920,
"height": 1080,
"theme": "light",
"innerWidth": 1920,
"innerHeight": 1080,
"scrollWidth": 1920,
"scrollHeight": 1080,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1402,
"diagramWidth": 1372,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 2048,
"height": 1320,
"theme": "light",
"innerWidth": 2048,
"innerHeight": 1320,
"scrollWidth": 2048,
"scrollHeight": 1320,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1818,
"diagramWidth": 1768,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 41,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
}
]
},
"captures": {
"status": "pass",
"screenshots": [
{
"width": 1440,
"height": 900,
"theme": "light",
"innerWidth": 1440,
"innerHeight": 900,
"scrollWidth": 1440,
"scrollHeight": 900,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1194,
"diagramWidth": 1164,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light",
"file": "thothii-core-sequence.visual-check.1440x900.light.png"
},
{
"width": 1440,
"height": 900,
"theme": "dark",
"innerWidth": 1440,
"innerHeight": 900,
"scrollWidth": 1440,
"scrollHeight": 900,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1194,
"diagramWidth": 1164,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "dark",
"file": "thothii-core-sequence.visual-check.1440x900.dark.png"
},
{
"width": 2048,
"height": 1320,
"theme": "light",
"innerWidth": 2048,
"innerHeight": 1320,
"scrollWidth": 2048,
"scrollHeight": 1320,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1818,
"diagramWidth": 1768,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 41,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light",
"file": "thothii-core-sequence.visual-check.2048x1320.light.png"
},
{
"width": 2048,
"height": 1320,
"theme": "dark",
"innerWidth": 2048,
"innerHeight": 1320,
"scrollWidth": 2048,
"scrollHeight": 1320,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1818,
"diagramWidth": 1768,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 41,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "dark",
"file": "thothii-core-sequence.visual-check.2048x1320.dark.png"
}
],
"contactSheet": "thothii-core-sequence.visual-check.html"
},
"sidecars": {
"receipt": "thothii-core-sequence.visual-check.json",
"contactSheet": "thothii-core-sequence.visual-check.html"
}
}
Binary file not shown.

Before

Width:  |  Height:  |  Size: 137 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 136 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 183 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 182 KiB

@@ -1,32 +0,0 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>Archify automated browser evidence · thothii-runtime.html</title>
<style>
*{box-sizing:border-box}body{margin:0;padding:24px;background:#e9eef5;color:#172033;font:14px/1.5 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}header{max-width:1500px;margin:0 auto 18px}h1{margin:0 0 6px;font-size:20px}p{margin:0;color:#526176}.grid{max-width:1500px;margin:auto;display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:18px}figure{margin:0;padding:10px;background:white;border:1px solid #c9d4e3;border-radius:12px;box-shadow:0 10px 30px rgba(15,23,42,.08)}img{display:block;width:100%;height:auto;border:1px solid #e2e8f0}figcaption{padding:9px 4px 2px;color:#526176}@media(max-width:900px){.grid{grid-template-columns:1fr}}
</style>
</head>
<body>
<header><h1>Automated browser evidence</h1><p>thothii-runtime.html · visual-check containment pass · perceptual visual review pending</p></header>
<main class="grid">
<figure>
<img src="thothii-runtime.visual-check.1440x900.light.png" alt="light 1440 by 900">
<figcaption><strong>LIGHT</strong> · 1440×900 · containment pass</figcaption>
</figure>
<figure>
<img src="thothii-runtime.visual-check.1440x900.dark.png" alt="dark 1440 by 900">
<figcaption><strong>DARK</strong> · 1440×900 · containment pass</figcaption>
</figure>
<figure>
<img src="thothii-runtime.visual-check.2048x1320.light.png" alt="light 2048 by 1320">
<figcaption><strong>LIGHT</strong> · 2048×1320 · containment pass</figcaption>
</figure>
<figure>
<img src="thothii-runtime.visual-check.2048x1320.dark.png" alt="dark 2048 by 1320">
<figcaption><strong>DARK</strong> · 2048×1320 · containment pass</figcaption>
</figure>
</main>
</body>
</html>
@@ -1,548 +0,0 @@
{
"schemaVersion": 1,
"ok": true,
"command": "visual-check",
"evidenceKind": "automated-browser",
"status": "pass",
"visualReview": "pending",
"artifact": {
"path": "/Users/mp/projects/ThothII/docs/architecture/thothii-runtime.html",
"sha256": "ad19195852c7c643228663e5bf7f3cb273afb0456fedbe295d4df24bc345b6c7",
"bytes": 724277
},
"state": {
"detail": "read",
"motion": "still"
},
"chrome": {
"status": "available",
"executable": "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"
},
"diagnostics": [],
"containment": {
"status": "pass",
"viewports": [
{
"width": 1440,
"height": 900,
"theme": "light",
"innerWidth": 1440,
"innerHeight": 900,
"scrollWidth": 1440,
"scrollHeight": 900,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 968,
"diagramWidth": 938,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 6.20735294117647,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1600,
"height": 1000,
"theme": "light",
"innerWidth": 1600,
"innerHeight": 1000,
"scrollWidth": 1600,
"scrollHeight": 1000,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1009,
"diagramWidth": 979,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 6.478676470588235,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1920,
"height": 1080,
"theme": "light",
"innerWidth": 1920,
"innerHeight": 1080,
"scrollWidth": 1920,
"scrollHeight": 1080,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1172,
"diagramWidth": 1142,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 7.557352941176471,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 2048,
"height": 1320,
"theme": "light",
"innerWidth": 2048,
"innerHeight": 1320,
"scrollWidth": 2048,
"scrollHeight": 1320,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1583,
"diagramWidth": 1533,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 9,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 41,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
}
]
},
"readability": {
"status": "pass",
"minimumProjectedNodeTextPx": 6,
"viewports": [
{
"width": 1440,
"height": 900,
"theme": "light",
"innerWidth": 1440,
"innerHeight": 900,
"scrollWidth": 1440,
"scrollHeight": 900,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 968,
"diagramWidth": 938,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 6.20735294117647,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1600,
"height": 1000,
"theme": "light",
"innerWidth": 1600,
"innerHeight": 1000,
"scrollWidth": 1600,
"scrollHeight": 1000,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1009,
"diagramWidth": 979,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 6.478676470588235,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1920,
"height": 1080,
"theme": "light",
"innerWidth": 1920,
"innerHeight": 1080,
"scrollWidth": 1920,
"scrollHeight": 1080,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1172,
"diagramWidth": 1142,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 7.557352941176471,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 2048,
"height": 1320,
"theme": "light",
"innerWidth": 2048,
"innerHeight": 1320,
"scrollWidth": 2048,
"scrollHeight": 1320,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1583,
"diagramWidth": 1533,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 9,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 41,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
}
]
},
"viewerChrome": {
"status": "pass",
"viewports": [
{
"width": 1440,
"height": 900,
"theme": "light",
"innerWidth": 1440,
"innerHeight": 900,
"scrollWidth": 1440,
"scrollHeight": 900,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 968,
"diagramWidth": 938,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 6.20735294117647,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1600,
"height": 1000,
"theme": "light",
"innerWidth": 1600,
"innerHeight": 1000,
"scrollWidth": 1600,
"scrollHeight": 1000,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1009,
"diagramWidth": 979,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 6.478676470588235,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1920,
"height": 1080,
"theme": "light",
"innerWidth": 1920,
"innerHeight": 1080,
"scrollWidth": 1920,
"scrollHeight": 1080,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1172,
"diagramWidth": 1142,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 7.557352941176471,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 2048,
"height": 1320,
"theme": "light",
"innerWidth": 2048,
"innerHeight": 1320,
"scrollWidth": 2048,
"scrollHeight": 1320,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1583,
"diagramWidth": 1533,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 9,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 41,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
}
]
},
"captures": {
"status": "pass",
"screenshots": [
{
"width": 1440,
"height": 900,
"theme": "light",
"innerWidth": 1440,
"innerHeight": 900,
"scrollWidth": 1440,
"scrollHeight": 900,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 968,
"diagramWidth": 938,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 6.20735294117647,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light",
"file": "thothii-runtime.visual-check.1440x900.light.png"
},
{
"width": 1440,
"height": 900,
"theme": "dark",
"innerWidth": 1440,
"innerHeight": 900,
"scrollWidth": 1440,
"scrollHeight": 900,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 968,
"diagramWidth": 938,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 6.20735294117647,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "dark",
"file": "thothii-runtime.visual-check.1440x900.dark.png"
},
{
"width": 2048,
"height": 1320,
"theme": "light",
"innerWidth": 2048,
"innerHeight": 1320,
"scrollWidth": 2048,
"scrollHeight": 1320,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1583,
"diagramWidth": 1533,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 9,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 41,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light",
"file": "thothii-runtime.visual-check.2048x1320.light.png"
},
{
"width": 2048,
"height": 1320,
"theme": "dark",
"innerWidth": 2048,
"innerHeight": 1320,
"scrollWidth": 2048,
"scrollHeight": 1320,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1583,
"diagramWidth": 1533,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 9,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 41,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "dark",
"file": "thothii-runtime.visual-check.2048x1320.dark.png"
}
],
"contactSheet": "thothii-runtime.visual-check.html"
},
"sidecars": {
"receipt": "thothii-runtime.visual-check.json",
"contactSheet": "thothii-runtime.visual-check.html"
}
}
+2 -2
View File
@@ -244,5 +244,5 @@ The first installed consolidation performs this conversion automatically when th
legacy manifest is present, then validates and activates the result. Preserve the
existing checkout in backups before upgrading. The E1 validation used an isolated
copy; E2 also converted and indexed all 35 units on the running local preview.
See the [E1 validation report](../plans/2026-09-09-evidence-e1-validation.md) and
[E2 validation report](../plans/2026-09-09-evidence-e2-validation.md).
See the [E1 validation report](../reports/knowledge-archives-release.md) and
[E2 validation report](../reports/knowledge-archives-release.md).
+135
View File
@@ -0,0 +1,135 @@
# Portal Shell Adapter v1
Contratto minimo della presentazione embedded. La revisione approvata il 2026-09-13
sostituisce il precedente trasporto a eventi personalizzati con l'osservazione del
documento condiviso. Non trasferisce identità, token o stato di autenticazione.
## Configurazione
Sul Mac:
```yaml
shell:
mode: full
defaultLocale: en
```
Sul server Omics:
```yaml
shell:
mode: embedded
adapter: omics-portal
```
Se `shell` o `mode` sono omessi, la modalità è embedded. L'adapter embedded
predefinito è `omics-portal`; un nome sconosciuto è un errore di configurazione.
Questi default sono normalizzati dal CLI prima della proiezione. Nel browser,
shell interamente omessa ha gli stessi default, ma un oggetto `shell` parziale
senza `mode` o `defaultLocale` viene rifiutato: non scrivere proiezioni a mano.
Full non istanzia adapter. `defaultLocale` inizializza full; in embedded il locale
proviene dal portale. L'autenticazione si configura separatamente dalla shell.
## API applicativa
```ts
export type PortalSnapshot = {
locale: string;
theme: "light" | "dark";
fullscreen: boolean;
};
export interface PortalAdapter {
subscribe(
onState: (state: PortalSnapshot) => void,
onError: (error: Error) => void,
): () => void;
}
```
Una sottoscrizione installa gli osservatori e consegna lo snapshot iniziale senza
richiedere messaggi all'altro applicativo. Gli aggiornamenti contengono snapshot
completi e validati. La disiscrizione elimina tutti i listener e osservatori.
La lingua viene risolta tramite i cataloghi UI, con fallback inglese.
## Implementazione Omics
L'integrazione monta React nel documento Django, non in un iframe.
| Dato | Fonte privata dell'adapter | Aggiornamento |
| --- | --- | --- |
| Locale | `data-lang` del selettore `.omics-language-select` | nuova pagina Django dopo `set_language` |
| Tema | `data-bs-theme` su `html` | osservazione limitata a quell'attributo |
| Fullscreen | stato effettivo del documento | evento del browser, inclusa uscita con Esc |
Il template Omics aggiornato allinea anche `html lang` alla lingua Django, ma
la fonte dell'adapter rimane `select.omics-language-select[data-lang]`. Leggere
il valore renderizzato evita di anticipare un cambio lingua prima che il form
abbia successo. Cambiare soltanto `select.value` o `data-lang` senza il normale
reload non è un trasporto runtime implementato per la lingua.
L'assenza del contesto host atteso produce un errore di integrazione; non abilita
controlli locali. Non si introducono eventi `ready/state`, handshake, timeout,
versioni dei messaggi o comandi duplicati. Selettori e dettagli Omics non devono
essere letti dai componenti applicativi.
## Proprietà per modalità
| Funzione | Full | Embedded |
| --- | --- | --- |
| Header | ThothII | solo Omics |
| Lingua | selettore locale | selettore Omics, normale reload Django |
| Tema | toggle locale light/dark | stato Omics |
| Fullscreen | controllo locale, stato reale | controllo Omics, stato reale |
| Login/logout | ThothII local/OIDC; upstream non offre logout locale | autenticazione Omics esistente |
| Nome utente | header ThothII | header Omics |
| Rotellina amministrativa | mai | eventuale comando del portale |
## Accesso e continuità
Il server Omics verifica l'accesso a Datamart Builder e il proxy trasmette i
principal header normalizzati al backend ThothII. La UI usa `/me`; non effettua
un secondo login. Un altro portale deve soddisfare anche questo contratto server,
oltre a fornire una nuova implementazione dell'adapter UI.
Il [contratto upstream](../install/authentication-upstream.md) specifica header,
origine, rete e configurazioni incompatibili. Lo snapshot non può contenere
`authenticated`, utente, ruoli, cookie o token; un evento browser non autorizza
una richiesta API. Il prefisso API viene configurato separatamente prima del
caricamento React, non viene dedotto dall'adapter.
Il logout del portale segue la sua navigazione. Una perdita di accesso rilevata
dal server chiude lo stato protetto; un 403 di una singola operazione non equivale
automaticamente a logout. La riconnessione degli eventi e il ritorno alla pagina
ricontrollano l'accesso. Non si garantisce revoca istantanea di una connessione
aperta in un'altra scheda attraverso il solo controllo iniziale del proxy.
Il cambio lingua può ricaricare la pagina: conservare la selezione della sessione,
proteggere le modifiche non salvate e non avviare una nuova generazione al reload.
Non si conserva una trascrizione integrale nel browser. La lingua della sessione
rimane quella registrata nel manifest, secondo ADR 0022.
## Sostituzione e verifiche
Un nuovo adapter può usare un diverso documento o trasporto, ma deve rispettare
la stessa sottoscrizione e mantenere la conoscenza del portale nella propria
implementazione. Oggi `ShellProvider` istanzia direttamente `OmicsPortalAdapter`:
per sostituirlo aggiornare quel punto e i nomi accettati in
`tools/tht/internal/config/shell.go` e `frontend/src/api/runtime-config.ts`.
Non è disponibile il caricamento dinamico di classi da una stringa YAML.
Non occorre implementare ora iframe o un secondo portale.
La nuova implementazione deve pubblicare uno snapshot iniziale completo, poi gli
aggiornamenti; segnalare contesto invalido; liberare tutti i listener alla
disiscrizione. Locale ben formato ma non tradotto significa fallback inglese;
locale assente/malformato e tema diverso da light/dark sono errori di integrazione.
Non cambiare componenti applicativi o workflow per aggiungere selettori specifici
del nuovo portale.
Verificare snapshot prima/dopo il montaggio, tema, fullscreen con Esc, cleanup,
contesto host mancante, assenza di header ThothII embedded, accesso singolo,
locale dopo reload e compatibilità del prefisso API. Full deve funzionare senza
alcun elemento Omics presente.
Vedere anche [architettura del rendering](../architecture/application-shell.md)
e [matrice di accettazione](../testing/authentication-manual-acceptance.md).
+6 -6
View File
@@ -4,7 +4,7 @@ This page describes the complete workspace Evidence lifecycle: where original ma
## Editable local Evidence
E1 adds [Curated Evidence v4 and a persistent local archive](contracts/curated-evidence-v4.md).
E1 adds [Curated Evidence v4 and a persistent local archive](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/curated-evidence-v4.md).
The visible title and payload fields are authoritative Markdown. New manual units need no
external source; consolidation records their curator and distinguishes later corrections
from original documentary provenance. The core archive API creates immutable candidates
@@ -14,12 +14,12 @@ E2 adds **Administration → Evidence management**, actual host file paths, comp
browsing and filtering, and the installed `tht workspace evidence consolidate
--workspace <id>` command. Edit files externally, consolidate to activate them, then
review and run Git manually. Runtime consumes only the active local snapshot.
The [v4 contract](contracts/curated-evidence-v4.md#administration-and-installed-command)
The [v4 contract](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/curated-evidence-v4.md#administration-and-installed-command)
describes host mounting, first conversion, failure recovery and Clear behavior.
The local PSD preview has 35 converted and indexed units. E3 adds explicit import/refresh
from local drafts and configured HTTP/S3 sources, durable current/proposed comparisons,
and keep/replace decisions with activation and retry. See
[Import drafts and refresh sources](contracts/curated-evidence-v4.md#import-drafts-and-refresh-sources).
[Import drafts and refresh sources](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/curated-evidence-v4.md#import-drafts-and-refresh-sources).
Workflow gate corrections remain the subsequent shared increment, X1.
## Existing repository publication path
@@ -92,7 +92,7 @@ Join orders using the order number, financial year and company.
Use `tht evidence migrate <workspace-root>` for deterministic legacy conversion. The
conversion preserves typed content and initializes an archive baseline; it does not
activate the local corpus. See the [v4 contract](contracts/curated-evidence-v4.md) for
activate the local corpus. See the [v4 contract](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/curated-evidence-v4.md) for
all eight kinds, provenance, file layout and the E1/E2 boundary.
## Legacy v3 representation
@@ -262,5 +262,5 @@ Formulas use a format distinct from document Evidence. A formula proposed during
## Contract references
- [Workspace Evidence v3 contract](contracts/workspace-evidence-v3.md)
- [Preprocessing CLI contract](contracts/workspace-preprocessing-cli.md)
- [Workspace Evidence v3 contract](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/workspace-evidence-v3.md)
- [Preprocessing CLI contract](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/workspace-preprocessing-cli.md)
+45
View File
@@ -128,6 +128,51 @@ endpoint expects a model name different from the catalog key.
Provider integrations remain declarative. Do not register providers from
`harness/.pi/extensions/`; those extensions implement the workflow and human gates only.
### Qwen 3.6 sessions and thinking controls
For Qwen served through a vLLM-compatible chat template, declare the following inside the
model's `session` block, alongside its context and output limits:
```yaml
reasoning: true
compatibility:
supportsDeveloperRole: false
supportsReasoningEffort: false
supportsStore: false
maxTokensField: max_tokens
thinkingFormat: qwen-chat-template
```
This makes Pi send `chat_template_kwargs.enable_thinking` from the selected thinking level,
with `preserve_thinking: true`. Choose **off** to explicitly disable thinking. The alternative
`thinkingFormat: qwen` is for endpoints expecting top-level `enable_thinking`. Both formats
require `reasoning: true`; declaring `reasoning: false` does not tell the server to disable
thinking. Omit `thinkingFormat` to preserve Pi's default behavior for other providers.
Regenerate projections with the updated host CLI and recreate the local core container after
rebuilding it. Do not add these fields directly to generated Pi files. These controls do not
force tool calls or certify the workflow; perform the operator verification above.
```sh
tht --installation /absolute/path/thothii-installation.yaml installation generate
```
Use the model identifier exposed by your endpoint, such as `qwen3.6-35b-a3b`, and limits
supported by that deployment. The thinking format configures the Pi session adapter;
metadata generation continues to use its separate LiteLLM settings.
If a session displays text such as `{"type":"bash","command":"tht session show … --json"}`
and never opens a review widget, that text is not an executed tool call. A verified cause
was the Evidence JSON extension being loaded into interactive sessions and forcing
`response_format: {type: "json_object"}`. Upgrade to the core image containing the fix:
the extension belongs in `.pi/evidence-extensions/` and is loaded explicitly only by
Evidence authoring. It must not also remain in the automatically loaded `.pi/extensions/`
directory. Regenerating model configuration alone does not remove an extension from an old image.
After upgrading, reload the browser and resume the session. Verify that Pi executes
`tht session show` and opens a review widget. This fix does not require changing the Qwen
server, forcing every turn to call a tool, or teaching the model to print tool-call JSON.
## Authentication
Every provider chooses one explicit mode:
+1 -1
View File
@@ -226,5 +226,5 @@ error. Preparation is repeatable and checks migration checksums.
See [the M1 specification](plans/2026-09-08-memory-m1-spec.md) and
[ADR 0018](adr/0018-use-postgres-for-memory-and-qdrant-for-retrieval.md) for the
approved scope and acceptance boundaries.
See [M2 implementation and validation](plans/2026-09-09-memory-m2-validation.md)
See [M2 implementation and validation](reports/knowledge-archives-release.md)
for the retrieval checks and real embedding test command.
+39 -3
View File
@@ -4,10 +4,30 @@ This guide is for a reviewer using a configured ThothII installation. Installati
publication, preprocessing, and database administration are separate paths; links to them are at
the end of this page.
## Standalone or inside Omics
In **full** mode, ThothII has its own red header. Sign in using the installation's
local account or the configured identity provider. The header lets you select
English/Italian, light/dark, and fullscreen; Esc exits fullscreen. Open the user
name menu to log out of ThothII. OIDC logout does not necessarily log out other
applications using the same provider.
In **embedded** mode, first sign in to Omics and choose **Datamart Builder** in
its left menu. ThothII opens with that authenticated identity: there is no second
login or duplicate header. Use Omics's language, theme, fullscreen and logout
controls. If portal access expires, return to Omics, sign in and reopen the page.
The Mac starts in English unless the browser remembers another choice. Changing
the UI language affects labels, not saved domain content. A new session takes
the selected language for the model's questions and reviewer choices; an existing
session retains its saved language when resumed. Omics's language change reloads
the page: confirm or cancel any unsaved-work warning. Reopening the saved session
selection shows documents; it does not automatically restart generation.
## Before creating a session
An administrator must have selected a workspace and configured the installation-wide provider,
model, and thinking settings. The New session form deliberately asks only for the question.
model, and thinking settings. The new-question form deliberately asks only for the question.
The workspace is a pinned Git revision. A subsequent workspace update cannot alter a session
already created from an earlier revision. If a workspace cannot reach its configured runtime DWH,
@@ -15,7 +35,9 @@ new sessions are refused before any session state is written.
## Create and review a session
1. Sign in and select **New session**.
1. Sign in and select **Session** (**Sessione** in Italian). If a session is already
open and unfinished, this returns to it without restarting it. Otherwise it
opens a new question; no session is created until you submit that question.
2. Enter a precise business question, including the relevant time period and desired output. For
example: “List patients discharged in the last 30 days, with ward and discharge date.”
3. Review each gate and make the decision requested by the widget. A choice with a decision payload
@@ -37,6 +59,19 @@ The workflow phases are fixed:
## Resume, archive, and the meaning of saved state
In Administration, the dot beside **Workspace** is green when readiness is
confirmed and red otherwise. Hover the button for the exact state; assistive
technology receives the same description. Select Workspace to inspect preparation.
The session sidebar has two accordion sections: **Active sessions** and
**Archive**, both initially closed. Only their headers appear below the scope tabs.
Inside each nonempty list, **Select all** selects only that list; its delete action
also applies only to the selected sessions in that list. The other list's selection
is preserved. Opening a section closes the other; clicking the open section closes
it too. Empty lists show only "No sessions yet." Long lists scroll inside
their own panels. Here active means not archived, not necessarily a running model
process. Existing groups and session actions remain inside those sections.
The sidebar lists sessions and their current lifecycle. Resuming returns to the last incomplete
phase. A finalized or archived session cannot be resumed.
@@ -54,4 +89,5 @@ happen from the workflow’s point of view.
- To author material the workflow can retrieve, use [Evidence](evidence.md). A proposal from a
session does not become Evidence automatically: a curator must review and publish it in Git.
- For login and access recovery, use [local authentication](install/authentication-local.md) or
[OIDC authentication](install/authentication-oidc.md).
[OIDC authentication](install/authentication-oidc.md) for full, or contact the
portal administrator for [embedded/upstream access](install/shell-and-language.md).
+24 -20
View File
@@ -1,29 +1,33 @@
# ThothII documentation
ThothII is a human-reviewed datamart builder. It turns a question into validated SQL through an
eight-phase workflow: the model proposes; a reviewer makes the decisions that are persisted.
ThothII turns a natural-language question into reviewed SQL. The model proposes;
a human reviewer decides which interpretations, sources and results to accept.
Start with the path that matches the work you need to do:
This is the public product manual. Start with the task you need to perform:
| I need to… | Start here |
| --- | --- |
| Install or operate one instance | [Install and first start](install/first-start.md) |
| Add, update, or prepare a workspace | [Workspace operations](operations/workspaces.md) |
| Ask a question and review the SQL workflow | [User guide](guida-utente.md) |
| Configure and refresh an authoritative database catalog | [Database management](operations/database-management.md) |
| Author and publish domain Evidence | [Evidence](evidence.md) |
| Understand boundaries and persistence | [Architecture overview](architecture/overview.md) |
| Understand the product and its boundaries | [What ThothII does](product-overview.md) |
| Install on Mac, Windows through WSL2, or Linux | [Italian procedure](install/standalone-manual-it.md) · [English procedure](install/standalone-manual-en.md) |
| Choose display mode and language | [Display mode and language](install/shell-and-language.md) |
| Configure login | [Local accounts](install/authentication-local.md) · [OIDC](install/authentication-oidc.md) |
| Configure model providers | [Model configuration](general/pi-configuration.md) |
| Prepare a domain workspace | [Workspaces](operations/workspaces.md) |
| Configure databases and reviewed descriptions | [Database management](operations/database-management.md) |
| Ask a question and review SQL | [User guide](guida-utente.md) · [Workflow](skills.md) |
| Maintain domain knowledge | [Evidence](evidence.md) · [Memory](usage/memory.md) |
## How the documentation is organised
## Scope of this manual
- **Install and operate** documents host-side setup, authentication, lifecycle, workspaces, and
Pi administration.
- **Use ThothII** documents the two application paths: reviewed NL→SQL sessions and administrative
database management.
- **Architecture and contracts** explain why the system behaves as it does and define the
machine-facing boundaries. Consult them when integrating or changing an implementation; they
are not a substitute for an operator runbook.
The manual covers the product, installation, use and administration. Architecture,
code contracts, design decisions, implementation plans, test reports and site-specific
deployment handoffs are developer/project material maintained in the repository;
they are not pages of this site and are not included in its search index.
The host-side `tht` command is the operator interface. The Python `tht` inside `harness/` is a
different, internal workflow CLI invoked by the core. In examples, use an absolute installation
descriptor path whenever discovery is not unambiguous.
The installation procedures distinguish checked documentation from platform and
functional tests that still require execution. A clone does not transfer another
installation's credentials, data or network access.
The host-side `tht` command operates an installation. The Python workflow CLI inside
the runtime is a separate internal interface; do not substitute its commands for
the host installation procedure.
+11 -1
View File
@@ -1,6 +1,11 @@
# Local authentication
Use local mode for a standalone PC or Mac. Configure it through `tht`; passwords are entered at an
Use local mode for a standalone PC or Mac, with `shell.mode: full` and
`shell.defaultLocale: en` in the installation descriptor. Presentation and
authentication are independent: selecting full does not create accounts. Omics
embedded instead uses the [upstream guide](shell-and-language.md), not local users.
Configure local authentication through `tht`; passwords are entered at an
echo-free prompt or read from a protected `--password-file`, never from a command argument.
## Bootstrap
@@ -56,6 +61,11 @@ machine use; JSON output is pristine on stdout.
## Session behavior and recovery
Full shows its own login form and, after login, the verified display name in its
header. The name menu contains Log out. This sends a CSRF-protected request to
`/api/auth/logout`, revokes the session and returns to login. Language/theme
preferences may remain in the browser; they are not credentials.
An ordinary login expires after 2 hours idle or 12 hours absolute. Selecting **Remember me** makes
the cookie persistent and changes the limits to 7 days idle or 30 days absolute. Remembered
sessions survive a browser and backend restart, but not a user revision change, configuration
+24 -1
View File
@@ -1,10 +1,20 @@
# Generic OIDC authentication
Use this guide for **ThothII's own login**, normally `shell.mode: full` on an
autonomous server. It is not the integration procedure for an already logged-in
Omics user. That deployment uses [embedded/upstream](shell-and-language.md),
even when Omics's identity provider is Authentik.
OIDC mode supports a standards-based provider. The browser flow is generic: Authorization Code,
PKCE S256, state, nonce, issuer/signature/audience/expiry validation, and the fixed callback
`<publicUrl>/api/auth/oidc/callback`. The browser and API must use the same origin; configure the
reverse proxy to preserve that public origin and callback path.
`publicUrl` is the public origin, without an application subpath. The current
full OIDC browser entry and callback use `/api/auth/oidc/login` and
`/api/auth/oidc/callback`; arbitrary prefixed OIDC hosting is not implemented by
selecting a different `backendBaseUrl`.
Configure the installation with `tht`:
```sh
@@ -18,6 +28,9 @@ The OIDC client secret is supplied through the protected secret bundle under the
`THT_OIDC_CLIENT_SECRET`; it is never written into `auth.yaml`. The default scopes are exactly
`openid`, `profile`, and `email`.
Keep `AUTH_MODE` unset when using this file. A simultaneously mounted local/OIDC
configuration and `AUTH_MODE=upstream` is an error, not a fallback chain.
The non-secret OIDC configuration has this exact shape (replace angle-bracket placeholders with
operator values):
@@ -91,4 +104,14 @@ Any authentication failure prevents activation according to the static or live s
relevant diagnostic surface.
The complete closed diagnostic-code union and exact role-to-permission expansion are in the
[authentication architecture](../architecture/authentication.md).
[authentication architecture](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/architecture/authentication.md).
## Browser login and logout
ThothII redirects the browser to the provider and creates its own opaque session
after validating the callback. An existing provider SSO session may avoid another
password prompt, but this remains a distinct ThothII login/session, unlike Omics
upstream. Full's name menu logs out of ThothII only. It does not revoke the
provider session or log out other applications, so a subsequent login can return
immediately through SSO. No provider token is placed in the UI adapter or browser
storage. See the [manual acceptance matrix](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/testing/authentication-manual-acceptance.md).
+186
View File
@@ -0,0 +1,186 @@
# Autenticazione tramite portale e proxy fidato
Questa è la modalità **upstream** usata dall'integrazione Omics. Non è il login
OIDC diretto di ThothII: l'utente accede a Omics come già fa, poi sceglie
Datamart Builder e trova ThothII già autenticato. Non deve essere creato un utente
locale ThothII né effettuato un secondo scambio OIDC dall'applicazione embedded.
## Il confine di fiducia
```mermaid
sequenceDiagram
actor U as Utente già autenticato
participant N as Nginx Omics
participant D as Django Omics
participant T as Core ThothII upstream
U->>D: Apri Datamart Builder
D-->>U: Pagina autorizzata con mount React
U->>N: GET /datamart-builder/api/me (cookie Omics)
N->>D: Subrequest interna /datamart-builder/api-auth
D-->>N: 200 + identità verificata, oppure 403
N->>T: GET /me + intestazioni normalizzate (solo se autorizzato)
T-->>U: Identità e permessi applicativi, oppure rifiuto
```
L'header e l'adapter JavaScript non autenticano nessuno. Il backend accetta una
richiesta upstream solo con un'identità valida ricevuta da un percorso di rete
fidato. Gli header non sono firmati da ThothII: la protezione è il proxy che
verifica la sessione e sovrascrive l'identità, insieme all'isolamento del core.
Un core upstream direttamente raggiungibile da client non fidati è una falla,
non una modalità alternativa di accesso.
## Configurazione del core
Per Omics il descrittore pubblico deve contenere:
```yaml
shell:
mode: embedded
defaultLocale: en
adapter: omics-portal
```
Separatamente, il **processo core** deve ricevere `AUTH_MODE=upstream`. Definirlo
nell'override Compose approvato e incluso nell'installazione; una variabile nel
file di interpolazione `.env` non viene passata automaticamente al container:
```yaml
services:
core:
environment:
AUTH_MODE: upstream
```
È solo il frammento di selezione auth, non un file Compose completo né una
configurazione di rete sufficiente. Non aggiunge porte pubbliche.
Condizioni obbligatorie:
1. Nessun `auth.yaml` local/OIDC deve essere effettivamente montato al percorso
letto dal core (default `/run/thothii-auth/auth.yaml`). Se è presente insieme
ad `AUTH_MODE`, l'avvio fallisce. Non impostare `AUTH_MODE=local` o `oidc`:
questi due modi si selezionano dal file, non da quella variabile.
2. Non configurare `authentication.runtimeProjection` per questo percorso: è la
proiezione delle configurazioni cookie local/OIDC, non l'identità Omics.
Nemmeno `THT_AUTH_RUNTIME_PROJECTION_ROOT` deve attivarla nel core.
3. Il descrittore e Compose base continuano a richiedere `authentication.configDirectory`
e `THT_AUTH_CONFIG_ROOT` coerenti. Per una nuova installazione upstream usare
una directory dedicata senza `auth.yaml`, non cancellare la configurazione di
un'installazione esistente. I cambi di modalità richiedono un piano separato.
4. Non esiste `tht auth configure --mode upstream`: il CLI configura gli utenti
locali o l'OIDC diretto. Conservare il percorso proxy già operativo per Omics.
5. `profile: server`, storage delle sessioni e `THOTH_PUBLIC_EXPOSURE` hanno propri
vincoli, che rimangono attivi. La shell embedded non li soddisfa automaticamente.
Il sorgente considera upstream un percorso di compatibilità con il proxy; è
quello usato dall'integrazione Omics corrente. `none` e `mock` sono per sviluppo/test,
non soluzioni a errori di configurazione in produzione.
## Intestazioni richieste all'ingresso del core
| Header | Regola ThothII | Valore Omics |
| --- | --- | --- |
| `X-Thoth-Principal-Issuer` | Stringa stabile, obbligatoria | `portal` |
| `X-Thoth-Principal-Subject` | ID stabile dell'utente, obbligatorio | `str(user.pk)` Django |
| `X-Thoth-Principal-Display-Name` | Facoltativo, se presente non vuoto | Nome completo o username |
| `X-Thoth-Is-Admin` | Obbligatorio: `0`, `1`, `false` o `true` | Risultato di `is_authentik_admin(user)` |
Le stringhe sono ripulite degli spazi esterni, devono avere al massimo 512
caratteri e non contenere caratteri di controllo. Header mancanti o invalidi
producono 401. `false`/`0` assegna il ruolo `user`; `true`/`1` assegna `user` e
`admin`. Il core espande i permessi dal proprio catalogo, non da un array inviato
dal browser. `/me` richiede `session.use`.
La coppia `(issuer, subject)` identifica il proprietario delle sessioni.
Non sostituire il subject con un nome visualizzato o un'email modificabile; non
cambiare issuer/subject di utenti esistenti per correggere un problema grafico.
Passare da identità `portal` a identità OIDC diretta non migra la proprietà dei dati.
## Omics: percorsi e componenti esatti
| Percorso | Destinazione e funzione |
| --- | --- |
| `/kokoro/datamart-builder/` | Pagina Django con `datamart_builder.access` |
| `/datamart-builder/config.js` | Config pubblico del frontend, senza cache |
| `/datamart-builder/assets/…` | Asset frontend risolti dal manifest Vite |
| `/datamart-builder/api/…` | Nginx con `auth_request`, poi core senza il prefisso |
| `/_thothii_auth` | Location Nginx interna, non un login pubblico |
| `/datamart-builder/api-auth` | Django verifica sessione Omics e capability |
Nel repository Omics:
- `kokoro/datamart_catalog_views.py`: `DatamartBuilderView` e
`datamart_builder_api_auth`; la verifica API risponde 200 o 403, anche 403
quando la sessione è assente/scaduta. Non trasforma l'API in una pagina di login.
- `nginx/nginx.conf`: API direttamente a `thothii-core:8787`, config e asset a
`thothii-frontend:8080`; verificare alias e reti Docker effettivi sul server.
- `templates/kokoro/datamart_builder.html`: mount, config e override del prefisso.
- `kokoro/templatetags/vite.py`: manifest da
`http://thothii-frontend:8080/.vite/manifest.json`, cache Django di 30 secondi.
Nginx usa il cookie Omics nella subrequest a Django. Sulle richieste al core
sovrascrive i quattro header con i risultati della verifica e rimuove
`Cookie`, `Authorization` e `X-Authenticated-User`. Nessuna password o token del
portale deve essere copiato nel config pubblico, nello snapshot adapter o in Web Storage.
`GET /datamart-builder/api/me` restituisce l'identità e i permessi; in upstream
`session` e `csrfToken` sono `null`: non viene creata una sessione-cookie ThothII.
## Non confondere i due percorsi proxy
L'esempio generico `deploy/nginx-authenticated-proxy.conf.example` usa **due hop**:
proxy host → frontend Nginx ThothII → core. Sul tratto privato verso il frontend
trasporta `X-Thoth-Trusted-Principal-*` e `X-Thoth-Trusted-Is-Admin`; il frontend
li converte nei quattro header del core e li elimina prima dell'inoltro.
Omics usa invece **Nginx Omics → core direttamente** per le API e invia gli header
normalizzati senza `Trusted`. Non incollare l'esempio a due hop in questa location:
la famiglia di header sbagliata produce 401. In entrambi i casi i valori devono
venire dalla verifica server, mai dagli header del client. Il tratto privato del
percorso generico deve essere inaccessibile ai client non fidati.
## Origine delle richieste e stream
Browser e API devono restare sullo stesso origin. Il frontend accetta `/api` o un
prefisso same-origin come `/datamart-builder/api`, non un URL `http://core:8787`.
In upstream le scritture con `Origin` sono confrontate con protocollo e Host
percepiti dal core; non usano il token CSRF della sessione ThothII local/OIDC.
Le richieste senza Origin hanno il trattamento non-browser: l'autenticazione del
proxy rimane indispensabile anche per esse.
Nel Nginx Omics esaminato il TLS termina a monte e una mappa **esatta** converte
`https://aritmolab.policlinicosandonato.it` in
`http://aritmolab.policlinicosandonato.it` per il confronto interno. Le altre origini
rimangono invariate e devono essere negate quando non coincidono. È una scelta
specifica della topologia corrente, non un modello da estendere con wildcard,
cancellazione di Origin o riscrittura incondizionata. Verificare Host/protocollo
al core e i dinieghi cross-origin nella topologia realmente rilasciata.
La location API disabilita buffering/cache per SSE e mantiene timeout lunghi.
`auth_request` verifica ogni nuova richiesta, ma non interrompe istantaneamente
uno stream già aperto quando il portale revoca l'utente. ThothII ricontrolla `/me`
al ritorno alla pagina e alla riconnessione degli eventi; non promettere revoca
istantanea fra tutte le schede.
## Logout, rientro e diagnosi
In embedded logout e successivo login sono di Omics. ThothII non chiama
`/auth/logout`, non cancella il cookie Django e non apre un suo login.
Il rifiuto 401/403 di `/me` rimuove lo stato protetto e richiede il rientro dal
portale. Un 403 su una singola operazione non equivale al logout dell'applicazione.
| Sintomo | Controllo mirato |
| --- | --- |
| Secondo header | Config servito: deve essere embedded, non full |
| Nessuna UI e errore preferenze | Selettore Omics `data-lang` e `html data-bs-theme` |
| `/me` 401 dal core | Header obbligatori, famiglia Trusted/normalizzata, percorso proxy |
| `/me` 403 dal proxy | Sessione Omics e capability `datamart_builder.access` |
| `/me` funziona ma POST 403 | Distinguere permesso operativo da mismatch Origin/Host/protocollo |
| 502 o asset assenti | Alias/rete Docker e manifest Vite; attesa cache manifest 30 s |
| Avvio core rifiutato | Coesistenza di `auth.yaml` o runtime projection con `AUTH_MODE` |
| Logout full seguito da rientro IdP immediato | Il logout ThothII non è logout globale OIDC |
Non raccogliere cookie, token, segreti o dump completi delle configurazioni nei
report. Registrare codici HTTP, nomi dei percorsi, revisioni e risultati dei test.
Consegna e rilascio: [procedura Omics](../operations/shell-and-localization.md#verifica-prima-del-deploy-server).
Collaudo obbligatorio: [matrice di accettazione](../testing/authentication-manual-acceptance.md).
+9 -2
View File
@@ -1,7 +1,14 @@
# Authentik provider configuration
ThothII uses generic OIDC in the browser. Authentik provides the identity provider and group
catalog without adding a proprietary login flow.
For **full with direct OIDC**, ThothII uses generic OIDC in the browser. Authentik
provides the identity provider and group catalog without adding a proprietary flow.
The provider/client/group setup below applies to that case only.
For **embedded in Omics**, retain Omics's existing Authentik authentication and
configure ThothII as upstream. Omics verifies `datamart_builder.access` and
administrator status and the proxy supplies the identity; no additional ThothII
OIDC client, login or local user is required for that path. Follow the
[portal integration guide](shell-and-language.md).
```mermaid
sequenceDiagram
@@ -3,6 +3,9 @@
# Replace every absolute placeholder. Select exactly one Git transport override.
schemaVersion: 2
profile: local
shell:
mode: full
defaultLocale: en
projectDirectory: "/absolute/path/to/ThothII"
envFile: "/absolute/path/to/ThothII/deploy/env/local.env"
workspaceRepository:
@@ -3,6 +3,11 @@
# Replace every absolute placeholder. Select exactly one Git transport override.
schemaVersion: 2
profile: server
# Standalone server with protected direct OIDC auth, not the Omics upstream path.
# For Omics use authentication-upstream.md: embedded, no auth runtime projection.
shell:
mode: full
defaultLocale: en
projectDirectory: "/absolute/path/to/ThothII"
envFile: "/absolute/path/to/thothii-server-operator/server.env"
workspaceRepository:
+45 -64
View File
@@ -1,82 +1,63 @@
# Install and first start
This is the supported local installation path. It creates an installation-local configuration and
starts the Compose stack; it does not create a workspace repository or a database catalog entry.
Use the guided procedure for a fresh installation:
## Prerequisites and boundaries
- [Italian guided installation](standalone-manual-it.md)
- [English guided installation](standalone-manual-en.md)
Install Docker Engine with Compose v2, plus the host `tht` command. On macOS or Linux, install the
host command from the repository with `./scripts/install-tht.sh`; Windows uses
`./scripts/install-tht.ps1`. The installer builds or verifies the native command and checks that
`tht` is resolvable on `PATH`.
The procedure covers Windows through Ubuntu WSL2, macOS, and Linux including an Omarchy/Arch-like
host. It uses the application clone, one installation secret bundle, protected repository
credentials when needed, and a terminal command. No host Node.js, Python or Pi installation is
required.
The stack contains `frontend`, `core`, `catalog-db`, `qdrant`, `embedding`, and the one-shot
`embedding-model-init` and `catalog-migrate` services. DWH and model-provider endpoints are
external installation settings. Pi runs inside `core`; do not install a host Pi executable for
the application runtime.
## What must be ready
Secrets, certificates, Pi authentication, and workspace endpoint bindings are protected
installation-local files. Never put them in a workspace descriptor, an env file intended for
version control, a URL, or a command line.
You need Docker with Compose v2, Git, Bash, curl, OpenSSL and shasum. You also need access to the
workspace repository and the values supplied by its owner: repository URL/branch, DWH endpoint,
database/schema, transport, credentials or certificates, Evidence credentials when applicable,
and LLM provider/API-key information.
## Create the local installation
The workspace repository and the THothII application repository are different. A workspace
descriptor may declare Evidence, but database passwords and installation bindings are stored in
the installation Catalog, not in Git.
From the repository root, start the interactive setup and select the local profile:
## One guided command
```sh
tht setup --profile local
```
After cloning THothII, checking prerequisites and installing tht, run:
It writes the selected non-secret descriptor below `deploy/<installation-id>/`, the associated
operator env file, and can create protected secret templates. Keep the descriptor path: pass it
to commands as `--installation /absolute/path/thothii-installation.yaml` when more than one
installation can be discovered.
~~~
tht setup --complete --profile local --shell-mode full --shell-default-locale en
~~~
If the descriptor is prepared manually instead, begin with
[`thothii-installation.local.yaml`](examples/thothii-installation.local.yaml), set mode `0600` or
`0400`, and ensure `THT_INSTALLATION_CONFIG_SOURCE` in the selected env file points to that exact
file. Create the secret bundle from `deploy/secrets/thothii.secrets.example`, protect it, and set
the file locations and external endpoints in the env file. The required settings include:
The first run creates protected placeholders under deploy/local/secrets/. Fill the required
credential files and rerun the same command. The command validates the local files and paths,
renders Compose, builds the images, starts catalog-db, runs catalog-migrate, starts the full
stack, and pulls/activates the workspace repository. Evidence source files declared by the
workspace are imported during activation.
- `PI_AUTH_FILE`, `THT_SECRETS_FILE`, and the catalog password source files;
- `THT_INSTALLATION_CONFIG_SOURCE` and the workspace Git remote/branch;
- the DWH and model-provider endpoints; and
- `THT_AUTH_CONFIG_ROOT` for local authentication or the OIDC configuration selected during setup.
For an already configured installation:
For the supported secret names and the metadata-generation model credential boundary, see the
`deploy/secrets/README.md` file in the installation checkout. It is intentionally not published
as a documentation page because it describes a protected local-file contract.
## Start and verify
For the normal local path, use the launcher:
```sh
./scripts/run-stack.sh
```
It builds `core`, starts `catalog-db`, runs `catalog-migrate`, then keeps the base plus local
Compose profile in the foreground. Database migrations are deliberately not a hidden backend
startup action. Open `http://127.0.0.1:8080` unless `THOTH_HTTP_PORT` was changed.
In another terminal, verify the installation without changing it:
```sh
tht --installation /absolute/path/thothii-installation.yaml doctor --json
~~~
tht --installation /absolute/path/thothii-installation.yaml status
```
tht --installation /absolute/path/thothii-installation.yaml doctor --json
tht --installation /absolute/path/thothii-installation.yaml workspace test --json
~~~
`/health` verifies application-process readiness; `tht doctor` is the diagnostic surface for
Compose, configuration, workspace, workflow, and Pi prerequisites.
doctor --json is the non-destructive general core test. workspace test also probes the configured
database, Evidence, Qdrant and embedding service for every active workspace. It requires the
workspace database to have been configured in Database Management first.
## Routine lifecycle and next steps
## Installer-only completion
Use `tht start [--build]`, `tht stop`, `tht logs`, and `tht doctor` rather than composing ad-hoc
container commands. Named volumes retain settings, Pi state, workspace registry, sessions,
Qdrant data, and embedding models across `docker compose down`; removing them requires the
explicit destructive `--volumes` form.
The installer must still decide which LLMs and API keys are approved, configure and test each
workspace database, synchronize its schema, generate and consolidate descriptions, create Qdrant
entries, review naming-based FK suggestions alongside schema FKs, and load the approved
relationships. The final declaration of completeness requires green doctor and workspace test
results plus one real natural-language question completed through final SQL.
After the stack is healthy, configure authentication if setup did not do so, then continue with
[Workspace operations](../operations/workspaces.md). For server profile, reverse proxy, backups,
and recovery, use the deployment program and its manual gates; the server profile is not a
drop-in replacement for the local command above.
Migrations are part of setup --complete. Do not mix this installation’s descriptor or volumes with
a different Compose environment. Preserve the descriptor, credentials, Catalog and persistent
volumes; do not use docker compose down --volumes as a routine stop.
See the guides for the Windows/macOS/Linux prerequisite matrix, workspace repository explanation,
secret layout and Gate A/Gate B acceptance checks.
+91
View File
@@ -0,0 +1,91 @@
# Installation preflight and release manifest
The native operator protocol is version 1. `installation preflight` is the independent
step-3 host check. `installation plan` repeats document validation, performs step-5
live checks and writes an owner-only JSON plan plus a separate owner-only `.key` file.
Neither command creates containers, imports bindings, modifies a database or invokes
model generation. Exit codes are 0 (checks passed), 1 (blocking error), and 2 (usage).
JSON stdout contains only the report. Checks carry stable `id`, `outcome`, `field`
and `action`; outcomes are `passed`, `error`, `warning`, `deferred-to-runtime`.
## Release manifest schema 1
The manifest is a local JSON file shipped with the verified operator/release bundle.
Required keys:
| Key | Contract |
| --- | --- |
| `schema_version` | `1` |
| `version` | Semantic release version, optionally prerelease |
| `revision` | 40 lowercase hexadecimal Git commit characters |
| `validator_protocol` | `1`; incompatible consumers refuse the manifest |
| `requirements` | `cpus`, `memory_bytes`, `disk_bytes`; at least 2 CPUs, 4 GiB Docker memory and 10 GiB installation filesystem space |
| `components` | Includes `pi`, `catalog-migrations`, `workspace-maintenance` |
| `images` | Exactly `core`, `frontend`, `catalog`, `qdrant`, `embedding`; each maps `linux/amd64` and/or `linux/arm64` to a `docker.io/...@sha256:...` **single-platform image digest** |
| `files` | Relative packaged resource paths to SHA-256; no traversal, links or absolute paths; maximum 256 files, 32 MiB per resource |
| `compose` | Ordered relative Compose file paths present in `files` for this release configuration |
Include the selected `deploy/compose.git-https.yaml` or `deploy/compose.git-ssh.yaml`
transport overlay in `files`. Standard transport overlays are resolved from the
release while absent in the installation directory; existing authored overrides
remain input files. Compose must resolve all eight services: core, frontend,
catalog-db, catalog-migrate, workspace-maintenance, qdrant, embedding and
embedding-model-init. The two maintenance services share the core digest; embedding
initialization shares the embedding digest. Source builds and undeclared services
are rejected in this prebuilt path. The explicit source path is a separate ticket.
`docker manifest inspect --verbose` checks each selected immutable image and its
platform without pulling layers. `docker compose config --format json` checks the
effective service configuration. Compose receives only Docker connection/trust,
proxy and executable-discovery host variables; application parameters come from
the prepared environment file. Raw Docker output is never copied into reports.
Each Docker command has a 15-second bound. No release is currently certified merely
because controlled manifest tests pass: publication and real pull acceptance belong
to the publication/execution tickets.
## External checks and bounds
- Git HTTPS: authenticated `GET /info/refs?service=git-upload-pack`, configured CA,
no redirects, selected branch advertised, 1 MiB response and 5-second bound.
Git SSH uses its prepared key/known-hosts and `git-upload-pack --advertise-refs`
with the same response/time bounds; no checkout or push occurs.
- PostgreSQL: the existing Catalog diagnostic adapter authenticates and reads
`current_database()` plus schema `USAGE`; no user tables are modified.
- REST database transport: existing Catalog diagnostic `GET` with the configured
bearer/API-key header and status validation. SSH database bindings cannot pass
this NL-to-SQL installation plan because runtime sessions do not support them.
- Evidence: local paths were already validated. HTTP performs bounded GET requests
and cancels response bodies; signed URL identities must match authored provenance.
S3 performs one `ListObjectsV2` request with `MaxKeys: 1`, no retries, explicit
file credentials and the canonical Evidence egress policy. These metadata requests
can incur normal remote-service request charges; they do not run LLM generation.
Each external request has a 5-second bound; the native database/Evidence helper
has a 60-second aggregate bound. Correct unavailable services before repeating.
- Explicit model endpoints: DNS/TCP/TLS origin reachability, without generating
tokens. Built-in endpoint resolution, provider authentication and model smoke
operations use the bundled Pi SDK at runtime; the report never claims those
operations have already passed.
No unreachable configured external dependency is converted to a deferred success.
Runtime obligations have explicit identities: `container-network`,
`catalog-initialization`, `pi-operation`, `local-embedding`,
`workspace-preprocessing`, `workspace-readiness`. These must be discharged by the
execution/readiness tickets before final success. Host disk inspection cannot prove
Docker Desktop VM free storage; its separate storage warning remains explicit.
## Input identity and freshness
The plan records normalized installation configuration, release digests, validator
build identity, verified input paths, local workspace content and its Git HEAD when
available (otherwise a content snapshot). Files are bounded to 32 MiB each and
256 MiB total; workspace/auth trees to 10,000 entries, without links or special files.
Credential contents are never serialized. An HMAC covers the private inputs and
plan using a separate random 32-byte owner-only key; no public unkeyed secret hash
is generated. Credential rotation invalidates the plan. Added/deleted/changed
workspace files invalidate it as well. Changes during the checks abort publication.
Plan output never overwrites an existing plan or key.
Execution and resumption must call `VerifyPlanInputs` **and repeat live checks and
credential reads before mutations**. A valid seal alone does not certify current
network availability, Docker state or runtime readiness. Keep both plan files
private and outside the workspace repository; they are installation-local artifacts.
+94
View File
@@ -0,0 +1,94 @@
# Pubblicare immagini e pacchetto operatore / Publish images and operator bundle
## Italiano
Questo comando è riservato al manutentore. L'utente finale scarica il pacchetto e
le immagini già compilati. La prima prerelease riguarda **Linux amd64**, utilizzato
da Ubuntu WSL2 su Windows e successivamente da Omarchy; non certifica ancora il
percorso completo di installazione o i collaudi manuali.
Prerequisiti del manutentore: Git, Node 24/npm, Go indicato in `tools/tht/go.mod`,
Docker con Buildx e capacità di eseguire Linux amd64, `tar`. Bun viene installato
dal lock npm e serve soltanto per compilare il pacchetto. Il commit da pubblicare
deve essere già disponibile sul repository Gitea pubblico.
1. Eseguire `docker login` sul computer di pubblicazione con un account autorizzato
a creare repository pubblici e pubblicare immagini nel namespace scelto.
Usare un credential store Docker: il token non va passato sulla riga di comando,
scritto nel repository o copiato nel pacchetto.
2. Predisporre nel credential helper Git l'accesso al repository Gitea con diritto
di creare rilasci e allegati. Il comando riusa quelle credenziali senza stamparle.
3. Installare le dipendenze del produttore con `cd backend && npm ci`, poi eseguire:
```bash
node scripts/publish-installation.mjs \
--revision COMMIT_GIA_PUBBLICATO \
--version 0.1.0-install-preview.1 \
--namespace tylconsulting \
--platforms linux/amd64 \
--output /percorso/privato/rilascio-0.1.0-install-preview.1
```
La directory di output deve essere nuova o vuota e avere un genitore esistente.
Diventa privata e contiene stato di ripresa, log del produttore, archivi e checksum.
Non è una directory di installazione. Il produttore usa un worktree temporaneo al
commit richiesto, così modifiche locali, segreti e workspace non entrano nelle build.
Il comando prepara `tylconsulting/thothii-core` e `tylconsulting/thothii-frontend`
come repository pubblici, costruisce e pubblica le immagini versionate, risolve i
digest di tutte le immagini e crea gli archivi del comando nativo con Compose e
risorse di inizializzazione. Catalog migration e workspace maintenance usano lo
stesso digest core. PostgreSQL, Qdrant e Ollama restano immagini upstream.
Prima di rendere pubblico il rilascio Gitea, vengono verificati pull anonimi delle
immagini, smoke test senza rete di core/frontend, checksum e download degli allegati.
Una build o un upload incompleto lascia il rilascio **in bozza**. Per riprovare,
rieseguire lo stesso comando con gli stessi parametri e la stessa directory.
Immagini già presenti devono appartenere allo stesso commit/versione; allegati
esistenti devono avere lo stesso checksum. Il produttore non sostituisce versioni
pubblicate con contenuti diversi. Conservare la directory fino al completamento.
Il risultato pubblico comprende `thothii-VERSION-linux-amd64.tar.gz` e
`SHA256SUMS.txt`. L'archivio contiene `bin/tht`, il validatore affiancato,
`release-manifest.json`, Compose, SQL/script di inizializzazione e guide. Non
contiene credenziali, dati dei workspace o database di esempio. La verifica su
questa macchina di pubblicazione non sostituisce il successivo collaudo Windows.
## English
This is a maintainer command. Consumers download precompiled images and the native
operator bundle. The first prerelease targets **Linux amd64** for Ubuntu WSL2 and
later Omarchy; full installation and real-host acceptance remain separate work.
The maintainer needs Git, Node 24/npm, the Go toolchain from `tools/tht/go.mod`,
Docker Buildx with Linux amd64 execution support, and `tar`. The npm lock supplies
Bun for producer builds only. Push the selected source commit to the public Gitea
repository before publication. Use Docker's credential store for `docker login`
and Git's credential helper for Gitea release/attachment permissions. Never pass
tokens as command arguments or include them in a checkout or archive.
From `backend`, run `npm ci`, then the command above with an explicit revision,
version, namespace, platforms and a new private output directory. A temporary Git
worktree isolates the selected commit. The producer publishes core/frontend to
Docker Hub and retains PostgreSQL, Qdrant and Ollama upstream. All runtime and
maintenance services use resolved immutable platform digests.
The Gitea release stays a draft until images, anonymous pulls, network-isolated
smoke checks and uploaded bundle checksums pass. An interrupted run can be retried
with the same arguments and output directory. Existing images must match the
source/version, and existing attachments must match their checksum; published
versions are not overwritten. Producer logs and retry state stay local.
Download the matching `.tar.gz` and `SHA256SUMS.txt` from the public Gitea prerelease,
verify the archive checksum, then extract it. Keep the two executables together.
The bundle needs no application checkout, Node, Bun, Python or compiler on the
consumer host. It carries the manifest, Compose and initialization assets, but no
installation credentials, workspace data or example databases. Linux arm64 can be
selected explicitly for later release work; it does not imply macOS acceptance.
Developer regression checks:
```bash
cd backend
node --test scripts/release-*.test.mjs
```
+70
View File
@@ -0,0 +1,70 @@
# Display mode and language
ThothII can run with its own application header (**full**) or inside an integrated
portal (**embedded**). Display mode and authentication are separate choices.
| Installation | Display | Authentication |
| --- | --- | --- |
| Standalone local instance | `full` | Local ThothII account |
| Standalone server | `full` | Local accounts or configured OIDC provider |
| Integrated portal | `embedded` | Identity verified by the portal's trusted server proxy |
## Standalone setup
Follow the complete [Italian](standalone-manual-it.md) or
[English](standalone-manual-en.md) installation procedure. It explicitly selects
`--shell-mode full --shell-default-locale en` and separates configuration, credentials,
initial migrations and startup. Do not skip those steps by running setup alone.
The authored installation descriptor contains:
```yaml
shell:
mode: full
defaultLocale: en
```
Use `it` for an Italian initial interface. Existing browser language preferences can
override that initial value. Full mode remembers language and theme in the browser.
For an existing installation, preserve the current descriptor and edit only the intended
settings; do not rerun setup to overwrite it. With a current host `tht` binary:
```sh
tht --installation /absolute/path/thothii-installation.yaml installation generate
tht --installation /absolute/path/thothii-installation.yaml start
tht --installation /absolute/path/thothii-installation.yaml status
tht --installation /absolute/path/thothii-installation.yaml doctor --json
```
Generation updates derived configuration; it does not start services. `start` applies
the installation's normal lifecycle and is not guaranteed to touch only the frontend.
Follow the deployment's maintenance procedure and retain its network, authentication and
model settings. Do not edit generated files or remove persistent volumes.
## Authentication and embedded deployments
Full mode does not configure login by itself. See [local authentication](authentication-local.md)
or [OIDC](authentication-oidc.md), with [Authentik](authentik.md) as a provider option.
Embedded mode requires a compatible portal integration, not just a descriptor toggle.
The portal owns login/logout and supplies a server-verified identity. A presentation
adapter does not authenticate users. The core must not be reachable by a route that
bypasses the trusted proxy. Do not add a second OIDC login to an already authenticated
upstream deployment.
Portal implementation details belong to the
[developer integration reference in the repository](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/install/authentication-upstream.md),
not to the standalone installation procedure.
## Three different languages
- **Interface language** controls labels, forms and application messages.
- **Session interaction language** is captured when a session is created. Resuming it
retains that language even if the interface language changes later.
- **Workspace language** concerns domain content and retrieval; switching the interface
does not translate Evidence, SQL, identifiers or database values.
In embedded mode the interface follows the portal's language and theme. A portal language
change may reload the page. Saved session artifacts remain available, but resuming work
is explicit; a reload does not by itself request a new model generation.
+441
View File
@@ -0,0 +1,441 @@
# Guided standalone installation
[Versione italiana](standalone-manual-it.md)
This is the fresh-machine installation procedure for THothII. THothII receives a natural-language
question, queries an enterprise database read-only, and guides the user through SQL review. The
core, PostgreSQL catalog, Qdrant, embedding service, and Pi run in Docker; Node.js, Python, and Pi
are not required on the host.
## Prepare and validate workspace documents before starting the stack
The first two steps of the new flow work without Docker, Node, Python, Pi or an
installation descriptor. Use the platform bundle with **both** `tht` and
`tht-workspace-documents` in the same directory (`.exe` on Windows). Put that directory
on `PATH`, or invoke the absolute executable path. Older packages containing only
`tht` do not provide this capability. Maintainers can currently build the bundle;
publishing assets and Docker Hub images belongs to a later delivery step. The rest
of this guide still describes the existing installation path.
1. Choose a new directory outside the application checkout, with an existing parent:
```sh
tht workspace prepare --directory ./my-workspaces --id practice --name "Practice" --language en
```
This creates `thoth-workspaces.yaml`, `practice/workspace.yaml` and
`workspace-docs/practice/README.md`. Existing destinations, even empty ones, are
refused. No services start, Git is not initialized and no remote is contacted.
Example databases remain a deferred subproject. For a curator-supplied repository,
use a separate local copy and go straight to step 3; read access to the origin is
sufficient.
2. Edit the catalog (schema v1) and workspace descriptor (schema v4) at your own pace.
Keep `id`, `name` and optional `description` identical in both; the id must match
the directory name. Root directories must match catalog entries, except
`workspace-docs` and the local `.git` directory. Database connections, schema and
credentials belong to the installation Metadata Catalog. Evidence is optional
and initially absent.
3. Validate, correct the reported document/field, and repeat:
```sh
tht workspace validate --directory ./my-workspaces
tht workspace validate --directory ./my-workspaces --json
```
PowerShell uses the same arguments, for example
`C:\ThothII\bin\tht.exe workspace validate --directory C:\ThothII\my-workspaces`.
Validation changes no files. It rejects multiple/malformed YAML documents,
duplicate keys/ids, unknown fields, catalog/directory/descriptor mismatches,
missing local references and symbolic links. Fix the first error in each document
and repeat to reveal any subsequent errors.
Evidence `absent` is valid. Filesystem checks cover directories, accessibility and
literal references; standard Markdown selections also check declared size limits.
Evidence v2 requires `curated/` and syntactically valid YAML frontmatter. Local limits
are 1 MiB per document read and 100,000 entries per Evidence tree. Arbitrary patterns,
the complete curated-unit contract, provenance, HTTP/S3 access and indexing remain
explicit runtime checks. See the [Evidence guide](../evidence.md).
JSON includes `schema_version`, `scope: local-documents`, `ok`, `workspaces`, `issues`
and `deferred_checks`. Issues identify document, field, code, correction and YAML
line where available, without printing document values. Exit statuses: `0` local
success, `1` documents/access/bundle need correction, `2` invalid arguments. Local
success does not certify semantic truth, connectivity or readiness. The Git revision
activated later must contain the checked documents; this command does not publish
uncommitted files or empty directories.
## Prepare and validate application documents
After validating workspaces, create a local directory **outside their repository**:
```sh
tht installation prepare --directory ./my-installation
```
This creates private, commented `thothii-installation.yaml`, `operator.env`,
`database-bootstrap.yaml` and `README.md`. The destination must be new and its parent
must exist. It starts no services and does not implicitly generate passwords.
1. Choose models and providers in the descriptor. The template proposes
`openai/gpt-4.1-mini` for interaction and `ollama/qwen3-embedding:0.6b` with 1024
dimensions for embedding. Edit these before setup. `modelCatalog.defaults.interaction`
must support sessions and metadata generation when the latter is configured.
The template omits optional metadata generation. See
[model configuration](../general/pi-configuration.md) for custom providers.
2. Replace the Git remote in both the descriptor and `operator.env`; keep branch and
transport consistent. Paths are absolute and machine-local. `operator.env` accepts
one literal `KEY=value` assignment per line, without duplicate keys or shell
interpolation. Credentials belong in referenced protected files.
3. Complete `database-bootstrap.yaml` with exactly one entry per workspace. A complete
direct-connection example is:
```yaml
schemaVersion: 1
databases:
- workspaceId: practice
engine: postgres
databaseName: sales
schema: public
binding:
transport: postgres_direct
host: db.intranet
port: 5432
username: thoth_reader
secretFiles:
password: /private/path/my-installation/secrets/database-password
```
Use a read-only DWH account. `rest_api` requires `baseUrl`, `restPath`, `restAuth`
(`none`, `bearer`, `x-api-key`) and `secretFiles.apiKey` when authenticated.
`ssh_tunnel` requires `username`, `sshHost`, `sshPort`, `sshUsername`,
`sshTargetHost`, `sshTargetPort` and files `password`, `sshPrivateKey`,
`sshKnownHosts`; it supports Catalog diagnostics, not NL-to-SQL sessions.
`tlsCa` and `sshPrivateKeyPassphrase` are optional. Signed HTTP Evidence needs
`evidenceSecretFiles` with key `evidence.signed_urls`; static S3 credentials need
`evidence.access_key`, `evidence.secret_key` and optional `evidence.session_token`.
All values are private file paths. Workspace descriptors remain schema v4;
this bootstrap input is not a second runtime Catalog.
4. Explicitly generate technical credentials in the standard layout:
```sh
tht installation credentials --directory ./my-installation
```
This creates separate random Catalog runtime/migrator and administrator passwords,
`auth/auth.yaml`, `auth/users.yaml`, a `secrets/secrets.env` template and
`secrets/pi-auth.json`. Existing files are retained; invalid ones stop the command.
The initial administrator is `admin`; its password stays in private
`secrets/admin-password` and is never printed. The default is local authentication
at `http://localhost:8080`: review and edit `auth/auth.yaml` before validation.
This increment does not validate offline OIDC bootstrap for the existing path.
5. Fill the provider key in `secrets/secrets.env` and create the DWH password file.
For Git HTTPS supply the referenced credentials and CA files; empty credentials
are allowed for a public remote, and the CA file must be available. For SSH supply
a key and known_hosts and select the matching descriptor override. `pi_auth`
providers require prepared Pi credentials. Keep every secret outside workspace
Git with installer-only access (0600 on Unix, equivalent Windows ACLs).
6. Validate and repeat after each correction:
```sh
tht --installation /absolute/path/my-installation/thothii-installation.yaml installation validate --workspaces /absolute/path/my-workspaces --json
```
The default bootstrap is beside the descriptor; `--bootstrap PATH` selects another.
Validation changes no documents, generates no projections, uses no network and
writes no database. It rejects placeholders, inconsistencies, missing/non-private
files and secrets inside workspace Git. Reports identify document, field and
correction without secret values. Exit statuses: 0 local success, 1 corrections
needed, 2 invalid arguments.
Standard release Compose assets may still be absent at this stage; custom overrides
must already exist. Release assets, external connectivity, Catalog import and runtime
readiness remain explicit deferred checks. Success prepares the next preflight;
it neither skips those checks nor establishes a completed installation.
## Check prerequisites and produce the plan
At step 3, before completing all application parameters, check the machine and
the private installation directory already prepared:
```bash
tht installation preflight --directory /path/installation --json
```
This requires a reachable Linux Docker daemon, Compose 2.24 or newer, at least
2 CPUs, 4 GiB allocated to Docker and 10 GiB free on the installation filesystem.
A release may require more resources. On Windows run the Linux executable in
Ubuntu WSL2 with Docker Desktop integration; Pi is bundled in the core image.
At step 5, after `installation validate`, select the published release manifest
with its downloaded bundle resources and produce a new plan:
```bash
tht --installation /path/installation/thothii-installation.yaml installation plan \
--workspaces /path/workspaces \
--release /path/release/release-manifest.json \
--output /path/installation/installation-plan.json --json
```
This repeats document checks, verifies image digests and Compose, Git, available
external databases and Evidence, then saves the plan and its separate private
`.key` file. It does not execute setup. Missing images and unavailable existing
dependencies block the plan. Actual Docker Hub publication remains the next ticket;
an invented manifest cannot bypass publication.
Correct `error` outcomes and read `warning` outcomes. `deferred-to-runtime` entries
are mandatory checks after startup, not readiness already achieved. After changing
documents or rotating credentials, produce a new plan; existing files are never
overwritten. Keep both plan files outside workspace Git. External probes perform
bounded database authentication/schema reads, Git/HTTP/S3 reads and explicit model
endpoint reachability checks. They invoke no LLM generation; HTTP/S3 requests may
incur ordinary service request charges. See the
[preflight reference](installation-preflight.md) for limits, the manifest
format and runtime obligations.
## Before you start: the two repositories
There are two separate repositories:
1. the application repository cloned by the user:
https://git.tylconsulting.it/mptyl/ThothII.git;
2. the workspace repository supplied by the curator/installer. It is not the THothII repository
and must not be cloned inside the application directory.
The workspace repository normally contains:
~~~
thoth-workspaces.yaml
<workspace-id>/workspace.yaml
<workspace-id>/evidence/** # when Evidence is declared
~~~
workspace.yaml contains workspace identity, language and optional Evidence source. By design it does
not contain database passwords. Database identity, transport (PostgreSQL, REST, or tunnel), user,
password, token and certificates are installation-local settings stored encrypted by the Catalog.
This prevents credentials from being committed to the workspace repository.
## 0. Machine prerequisites
### Windows
- Windows 10/11 with Docker Desktop running and the WSL2 backend enabled.
- Ubuntu in WSL2, with Docker Desktop integration enabled for that distribution.
- Git, Bash, curl, OpenSSL and shasum inside WSL2.
- Do not install Node.js, Python or Pi on the host for this procedure.
If WSL2 is not installed, use the company procedure or, in PowerShell:
~~~
wsl --install -d Ubuntu
~~~
Run all commands inside Ubuntu WSL2, in a Linux directory such as $HOME/src, not under /mnt/c.
scripts/install-tht.ps1 exists for advanced native PowerShell scenarios; use WSL2 for the
reproducible test.
### macOS
- Docker Desktop installed and running, with several GB free for images and the embedding model.
- Git, Bash, curl, OpenSSL and shasum.
- Intel and Apple Silicon Macs are supported when Docker Desktop supports the architecture
reported by the Docker server.
- Do not install Node.js, Python or Pi on the host for this procedure.
### Linux, including Omarchy
- Git, Bash, curl, OpenSSL and shasum.
- Docker Engine and the Docker Compose v2 plugin. On Omarchy, check first:
~~~
command -v docker
docker compose version
docker info
~~~
If Docker is missing, install Docker and Compose using the distribution-approved package procedure,
then start the service. On an Arch-like distribution the typical route is:
~~~
sudo pacman -S docker docker-compose
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"
~~~
After adding the group, open a new session and repeat docker info. Node.js, Python and Pi are not
needed on the host: they are in the Docker images.
On every system run:
~~~
bash scripts/check-standalone-prerequisites.sh
docker version --format '{{.Server.Arch}}'
~~~
The architecture must be amd64, x86_64, arm64, or aarch64. You also need access to the
application Gitea repository, the workspace repository URL/branch and credentials, container
reachability to DWH/LLM endpoints, and the credentials, tokens or certificates associated with
the databases.
## 1. What to clone
Clone only the application:
~~~
mkdir -p "$HOME/src"
cd "$HOME/src"
git clone https://git.tylconsulting.it/mptyl/ThothII.git
cd ThothII
git rev-parse --short HEAD
~~~
Record the revision. tht setup --complete downloads the workspace repository into a persistent
Docker volume using the URL, branch and transport supplied during setup.
## 2. Install the terminal command
From the clone root:
~~~
bash scripts/check-standalone-prerequisites.sh
mkdir -p "$HOME/.local/bin"
THT_INSTALL_DIRECTORY="$HOME/.local/bin" bash scripts/install-tht.sh
export PATH="$HOME/.local/bin:$PATH"
tht version
~~~
tht is the only native component to install. It builds the binary with Docker and orchestrates
Compose; it is not a second application runtime.
## 3. Prepare a few secrets and run complete setup
The first execution creates protected placeholders under deploy/local/secrets/ and stops if a
required credential is missing. Fill in the requested files and rerun the same command; compatible
configuration files are reused.
~~~
tht setup --complete --profile local --shell-mode full --shell-default-locale en
~~~
The setup asks only for information the computer cannot know:
| Request | What to provide |
| --- | --- |
| Workspace repository | Data/configuration repository URL, not ThothII.git |
| Branch | normally main |
| Access | ssh with key and known_hosts, or https with credential file and CA |
| DWH/LLM URL | endpoint without a token in the URL |
| Local login | initial user and password requested by the prompt |
The setup generates random Catalog passwords and writes their paths, never their values, to
operator.env. It runs docker compose config, builds images, starts the Catalog, runs
catalog-migrate, starts the stack, and pulls the workspace repository. The pull also activates
declared Evidence; at minimum source files present in the workspace are materialized locally.
### The file the user fills in
The main file is:
~~~
deploy/local/secrets/thothii.secrets
~~~
Add only NAME=VALUE lines needed by modelCatalog and installation adapters, such as an LLM API key
(DEEPSEEK_API_KEY, OPENAI_API_KEY, or the key declared by the catalog) and, when applicable,
THT_DWH_API_KEY. Allowed names are documented in deploy/secrets/README.md. Never put tokens in
URLs, the repository, or copied shell commands.
Two distinctions prevent common errors:
- when the catalog uses pi_auth, the LLM token belongs in the Pi pi-auth.json file created by
setup; {} is only a placeholder and does not enable a model;
- workspace database credentials (PostgreSQL password, REST API token, tunnel SSH key,
known_hosts, CA) do not belong in the workspace repository. Enter them per workspace in
Database Management, which stores them encrypted in the Catalog. The workspace declares
database/schema and transport; the installer must obtain the actual values from the database owner.
A private workspace repository also needs the Git files required by its transport: an SSH key and
known_hosts, or an HTTPS credential file and CA. These are transport files, not a second bundle to
commit. To minimize manual files, use SSH with an already-authorized deploy key.
## 4. Automatic checks and terminal tests
Setup verifies files, permissions, descriptor, Compose, Docker, authentication, services, Pi and
the workspace. After startup, run these commands at any time:
~~~
INSTALLATION="$PWD/deploy/local/thothii-installation.yaml"
tht --installation "$INSTALLATION" doctor --json
tht --installation "$INSTALLATION" workspace pull --json
tht --installation "$INSTALLATION" workspace test --json
~~~
workspace test checks, for every active workspace, database binding and credentials, Evidence,
Qdrant, and the embedding service. It exits non-zero when the database binding is missing or a
connection is unusable. Before running it, the installer must configure the database in Database
Management: the workspace repository cannot contain the password by itself.
doctor --json is the repeatable, non-destructive core verification. The final functional test must
also open http://127.0.0.1:8080, sign in, and complete a real question through final SQL.
## Activities only the installer can complete
The procedure automates bootstrap, but it cannot invent enterprise decisions or authorizations.
The installer must complete and record:
1. usable LLMs, the modelCatalog, and linked API keys; then run tht pi test and tht doctor;
2. database configuration, connection test, schema synchronization, and description generation;
3. human consolidation of generated descriptions;
4. Qdrant semantic entries through workspace preprocess run;
5. naming-based FK suggestions as a complement to schema FKs, human review, and loading approved
relationships into Qdrant;
6. recurring tht doctor --json and tht workspace test --json checks;
7. one real question completed successfully without connection or model errors.
Configuration is complete only when all applicable activities are done, decisions are recorded, and
the two terminal tests are green. The core is usable only after the real question, not merely
because the frontend answers /health.
## Gate A and Gate B
### Gate A — platform
~~~
uname -a
docker version --format '{{.Server.Version}} {{.Server.Arch}}'
tht version
bash scripts/check-standalone-prerequisites.sh
bash scripts/verify-standalone-install.sh "$INSTALLATION"
~~~
### Gate B — usability
~~~
tht --installation "$INSTALLATION" doctor --json
tht --installation "$INSTALLATION" workspace test --json
curl --fail --silent --show-error http://127.0.0.1:8080/health
~~~
Then run a real question and stop/restart with tht stop and tht start. Do not use docker compose
down --volumes: it deletes the Catalog, sessions, Qdrant data and the embedding model.
## Quick diagnosis
| Symptom | Action |
| --- | --- |
| Docker Engine is not reachable | start Docker Desktop or systemctl and repeat docker info |
| Omarchy cannot find docker | install Docker/Compose, enable the service and open a new session |
| Windows sees Docker but Bash fails | use Ubuntu WSL2 and enable its Docker Desktop integration |
| workspace pull fails | check URL, branch, key/credential file and known_hosts from the container |
| workspace test reports a missing binding | configure database, token/password and CA in Database Management |
| Pi is not ready | fill pi-auth.json or the key declared by modelCatalog, then run tht pi test |
## Related documents
- [Install and first start](first-start.md)
- [Workspace operations](../operations/workspaces.md)
- [Database Management](../operations/database-management.md)
- [Model configuration](../general/pi-configuration.md)
- deploy/secrets/README.md
Publishing images on Docker Hub and native DMG/MSI/AppImage installers remain later work: this
procedure starts from the Gitea clone and does not require pre-published Docker Hub images.
+454
View File
@@ -0,0 +1,454 @@
# Installazione standalone guidata
[English version](standalone-manual-en.md)
Questa è la procedura per installare THothII da zero. THothII riceve una domanda in linguaggio
naturale, interroga in sola lettura un database aziendale e accompagna l’utente nella revisione
della SQL risultante. Il core, il catalogo PostgreSQL, Qdrant, il servizio di embedding e Pi vengono
eseguiti in Docker; sul computer non servono Node.js, Python o Pi.
## Preparazione e verifica dei workspace prima dello stack
I primi due passi del nuovo percorso funzionano senza Docker, Node, Python, Pi o
un file di installazione. Usare il bundle della propria piattaforma con **entrambi**
gli eseguibili `tht` e `tht-workspace-documents` nella stessa cartella (`.exe` su
Windows). Aggiungere la cartella al `PATH`, oppure usare il percorso completo.
Il vecchio pacchetto con il solo `tht` non contiene questa capacità. Il bundle è
attualmente producibile dal manutentore; pubblicazione degli asset e immagini
Docker Hub appartengono a una fase successiva. Il resto della guida descrive ancora
il percorso di installazione esistente.
1. Scegliere una cartella nuova, esterna all'applicazione, con padre già esistente:
```sh
tht workspace prepare --directory ./miei-workspace --id pratica --name "Pratica" --language it
```
Sono creati `thoth-workspaces.yaml`, `pratica/workspace.yaml` e
`workspace-docs/pratica/README.md`. Una destinazione esistente, anche vuota, viene
rifiutata. Nessun servizio viene avviato; Git non viene inizializzato, nessun remoto
viene contattato. I database di esempio restano un sottoprogetto differito. Per
un repository fornito dal curatore, usare una copia locale separata e passare al
punto 3: è sufficiente accesso in lettura all'origine.
2. Modificare con calma catalogo (schema v1) e descrittore workspace (schema v4).
Mantenere uguali `id`, `name` e l'eventuale `description` nei due file; l'id deve
coincidere con la cartella. Ogni cartella alla radice deve corrispondere a un
workspace elencato, salvo `workspace-docs` e la directory locale `.git`.
Connessioni, schema e credenziali dei database appartengono al Metadata Catalog
dell'installazione. Le Evidence sono facoltative e inizialmente assenti.
3. Verificare, correggere il documento/campo indicato e ripetere:
```sh
tht workspace validate --directory ./miei-workspace
tht workspace validate --directory ./miei-workspace --json
```
Su PowerShell usare gli stessi argomenti, ad esempio
`C:\ThothII\bin\tht.exe workspace validate --directory C:\ThothII\miei-workspace`.
Il controllo non modifica file. Rifiuta YAML multipli o malformati, chiavi/id
duplicati, campi sconosciuti, incoerenze tra catalogo, cartelle e descrittori,
riferimenti locali mancanti e link simbolici. Correggere il primo errore del
documento e ripetere per vedere eventuali errori successivi.
Evidence `absent` è valido. Per filesystem si verificano directory, accessibilità e
riferimenti letterali; per le selezioni Markdown standard anche i limiti dichiarati.
Evidence v2 richiede `curated/` e frontmatter con sintassi YAML valida. I limiti locali
sono 1 MiB per documento letto e 100.000 elementi per albero Evidence. Pattern
arbitrari, contratto completo delle unità curate, provenienza, accesso HTTP/S3 e
indicizzazione restano controlli runtime espliciti. Vedere la [guida Evidence](../evidence.md).
Il JSON espone `schema_version`, `scope: local-documents`, `ok`, `workspaces`, `issues`
e `deferred_checks`. I problemi riportano documento, campo, codice, correzione e,
quando disponibile, riga YAML, senza stampare i valori del documento. Exit status:
`0` successo locale, `1` documenti/accesso/bundle da correggere, `2` argomenti errati.
Il successo locale non certifica verità semantica, connettività o readiness. La
revisione Git attivata in seguito deve contenere i documenti verificati; il comando
non pubblica file non committati o directory vuote.
## Predisporre e verificare i documenti applicativi
Dopo la verifica dei workspace, creare una cartella locale **esterna al loro repository**:
```sh
tht installation prepare --directory ./mia-installazione
```
Il comando crea file privati commentati: `thothii-installation.yaml`, `operator.env`,
`database-bootstrap.yaml` e `README.md`. La cartella deve essere nuova e il padre
deve esistere. Non avvia servizi e non genera implicitamente password.
1. Nel descrittore, scegliere modelli e provider. Il template propone
`openai/gpt-4.1-mini` per l'interazione e `ollama/qwen3-embedding:0.6b` con 1024
dimensioni per l'embedding. Sono valori modificabili, non una selezione richiesta
durante il setup. `modelCatalog.defaults.interaction` deve essere utilizzabile
nelle sessioni e anche nella generazione metadati, se quest'ultima è configurata.
Il template omette la generazione metadati, che è facoltativa. Consultare la
[configurazione dei modelli](../general/pi-configuration.md) per provider personalizzati.
2. Sostituire il remoto Git sia nel descrittore sia in `operator.env`; mantenere
coerenti branch e trasporto. I percorsi sono assoluti e riferiti a questa macchina.
`operator.env` accetta una sola assegnazione letterale `KEY=value` per riga, senza
duplicati o interpolazioni shell. Le credenziali restano nei file referenziati.
3. Compilare `database-bootstrap.yaml`: una voce per ciascun workspace, senza
duplicati. Questo esempio mostra il contratto completo di un collegamento diretto:
```yaml
schemaVersion: 1
databases:
- workspaceId: pratica
engine: postgres
databaseName: vendite
schema: public
binding:
transport: postgres_direct
host: db.intranet
port: 5432
username: thoth_reader
secretFiles:
password: /percorso/privato/mia-installazione/secrets/database-password
```
Usare le credenziali di un utente DWH in sola lettura. Per `rest_api`, il binding
richiede `baseUrl`, `restPath` e `restAuth` (`none`, `bearer`, `x-api-key`); quando
serve autenticazione, aggiungere `secretFiles.apiKey`. `ssh_tunnel` richiede
`username`, `sshHost`, `sshPort`, `sshUsername`, `sshTargetHost`, `sshTargetPort` e
i file `password`, `sshPrivateKey`, `sshKnownHosts`; abilita diagnostica Catalog,
non sessioni NL→SQL. Sono facoltativi `tlsCa` e `sshPrivateKeyPassphrase`.
Le Evidence HTTP firmate richiedono `evidenceSecretFiles` con chiave
`evidence.signed_urls`; S3 con credenziali statiche richiede `evidence.access_key`
e `evidence.secret_key`, con `evidence.session_token` facoltativo. Tutti i valori
sono percorsi di file privati. I workspace rimangono nello schema v4: il bootstrap
è un input iniziale, non un secondo Catalog runtime.
4. Generare esplicitamente le credenziali tecniche nel layout standard:
```sh
tht installation credentials --directory ./mia-installazione
```
Sono creati password casuali separate per Catalog runtime/migrator e amministratore,
il relativo `auth/auth.yaml` con `auth/users.yaml`, un template `secrets/secrets.env`
e `secrets/pi-auth.json`. I file esistenti vengono conservati; se non validi, il
comando si ferma. L'amministratore iniziale è `admin`, la password è nel file
privato `secrets/admin-password` e non viene stampata. Il default è autenticazione
locale con URL pubblico `http://localhost:8080`: verificare e, se necessario,
modificare `auth/auth.yaml` prima del controllo. Questo incremento non valida il
bootstrap OIDC del percorso esistente.
5. Inserire la chiave del provider in `secrets/secrets.env` e creare il file della
password DWH. Per HTTPS Git fornire i file referenziati per credenziali e CA:
credenziali vuote sono ammesse per un remoto pubblico, la CA deve essere disponibile.
Per SSH fornire chiave e known_hosts, scegliendo il relativo override nel descrittore.
I provider `pi_auth` richiedono credenziali Pi già preparate. Conservare tutti
questi file fuori dal repository workspace e proteggere l'accesso al solo utente
installatore (0600 su Unix, ACL equivalenti su Windows). Non committarli.
6. Verificare e ripetere dopo ogni correzione:
```sh
tht --installation /percorso/assoluto/mia-installazione/thothii-installation.yaml installation validate --workspaces /percorso/assoluto/miei-workspace --json
```
Il bootstrap viene cercato accanto al descrittore; `--bootstrap PERCORSO` ne
seleziona uno diverso. Il controllo non cambia documenti, non genera proiezioni,
non usa la rete e non scrive database. Rifiuta placeholder, incoerenze, file
mancanti/non privati e segreti situati nel repository workspace. Il report indica
documento, campo e correzione senza valori riservati. Exit status: 0 successo
locale, 1 correzioni necessarie, 2 argomenti errati.
Gli asset Compose standard del rilascio possono ancora mancare in questa fase;
gli override personalizzati devono già esistere. Il report distingue i controlli
differiti: asset del rilascio, connettività esterna, import Catalog e readiness.
Un esito positivo prepara il successivo preflight: non autorizza a saltare tali
controlli e non equivale a un'installazione completata.
## Verificare le precondizioni e produrre il piano
Al passo 3, prima di completare tutti i parametri applicativi, controllare la
macchina e la directory privata già preparata:
```bash
tht installation preflight --directory /percorso/installazione --json
```
Servono Docker Linux raggiungibile, Compose 2.24 o successivo, almeno 2 CPU,
4 GiB assegnati a Docker e 10 GiB liberi nella directory di installazione. Il
rilascio può richiedere risorse maggiori. Su Windows eseguire il binario Linux
in Ubuntu WSL2 con integrazione Docker Desktop; Pi è incluso nell'immagine core.
Al passo 5, dopo `installation validate`, selezionare il manifest del rilascio
pubblicato, con le risorse del bundle già scaricate, e produrre un piano nuovo:
```bash
tht --installation /percorso/installazione/thothii-installation.yaml installation plan \
--workspaces /percorso/workspaces \
--release /percorso/rilascio/release-manifest.json \
--output /percorso/installazione/installation-plan.json --json
```
Il comando ripete i controlli dei documenti, verifica immagini/digest e Compose,
Git, database ed Evidence esterne disponibili, poi salva il piano e il suo file
privato `.key`. Non esegue il setup. Le immagini assenti e le dipendenze esterne
irraggiungibili bloccano il piano. Al momento la pubblicazione reale Docker Hub
è ancora il ticket successivo: un manifest inventato non permette di aggirarla.
Correggere gli esiti `error`; leggere gli `warning`. Gli esiti
`deferred-to-runtime` identificano controlli obbligatori dopo l'avvio, non una
readiness già ottenuta. Un piano valido non sostituisce questi gate. Dopo una
correzione o rotazione di credenziali produrre un nuovo piano; i file esistenti
non vengono sovrascritti. Conservare piano e chiave fuori dal Git dei workspace.
Le prove esterne sono letture limitate: autenticazione/schema del database,
letture Git e HTTP/S3, raggiungibilità degli endpoint modello espliciti. Nessuna
generazione LLM viene invocata; le richieste HTTP/S3 possono avere i normali costi
del servizio. Limiti, manifest e obblighi sono nel
[riferimento di preflight](installation-preflight.md).
## Prima di iniziare: i due repository
Servono due repository distinti:
1. il repository dell’applicazione, che l’utente clona:
https://git.tylconsulting.it/mptyl/ThothII.git;
2. il repository dei workspace, indicato dal curatore/installatore. Non è il repository di
THothII e non va clonato manualmente nella directory dell’applicazione.
Il repository workspace contiene il catalogo e una directory per ogni workspace, normalmente:
~~~
thoth-workspaces.yaml
<workspace-id>/workspace.yaml
<workspace-id>/evidence/** # se il workspace dichiara Evidence
~~~
Il file workspace.yaml descrive identità, lingua e, opzionalmente, la sorgente Evidence. Per scelta
architetturale non contiene password del database. L’identità del database, il trasporto
(PostgreSQL, REST o tunnel), username, password, token e certificati sono configurazione locale
dell’installazione, conservata cifrata dal Catalog. Questo evita di committare credenziali nel
repository workspace.
## 0. Prerequisiti della macchina
### Windows
- Windows 10/11 con Docker Desktop avviato e backend WSL2 abilitato.
- Ubuntu in WSL2, con integrazione Docker Desktop abilitata per quella distribuzione.
- Git, Bash, curl, OpenSSL e shasum nella distribuzione WSL2.
- Non installare Node.js, Python o Pi sull’host per questa procedura.
In PowerShell, se WSL2 non esiste ancora, usare la procedura aziendale oppure:
~~~
wsl --install -d Ubuntu
~~~
Eseguire poi tutti i comandi dentro Ubuntu WSL2, in una directory Linux come $HOME/src, non sotto
/mnt/c. Il percorso nativo scripts/install-tht.ps1 esiste per scenari PowerShell avanzati; per la
prova riproducibile usare WSL2.
### macOS
- Docker Desktop installato, avviato e con alcuni GB liberi per immagini e modello di embedding.
- Git, Bash, curl, OpenSSL e shasum.
- Sono supportati Mac Intel e Apple Silicon se Docker Desktop supporta l’architettura restituita
dal Docker server.
- Non installare Node.js, Python o Pi sull’host per questa procedura.
### Linux, incluso Omarchy
- Git, Bash, curl, OpenSSL e shasum.
- Docker Engine e il plugin Docker Compose v2. Su Omarchy verificare prima:
~~~
command -v docker
docker compose version
docker info
~~~
Se Docker manca, installare Docker e Compose con il gestore pacchetti/procedura approvata dalla
distribuzione, poi avviare il servizio. Su una distribuzione Arch-like il percorso tipico è:
~~~
sudo pacman -S docker docker-compose
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"
~~~
Dopo l’aggiunta al gruppo aprire una nuova sessione e ripetere docker info. Non installare Node.js,
Python o Pi sull’host: sono dentro le immagini Docker.
Su tutti i sistemi il controllo finale è:
~~~
bash scripts/check-standalone-prerequisites.sh
docker version --format '{{.Server.Arch}}'
~~~
L’architettura deve essere amd64, x86_64, arm64 o aarch64. Servono inoltre accesso al repository
Gitea dell’applicazione, URL/branch e credenziali del repository workspace, raggiungibilità dal
container degli endpoint DWH/LLM e le credenziali, token o certificati associati ai database.
## 1. Cosa clonare
Clonare solo l’applicazione:
~~~
mkdir -p "$HOME/src"
cd "$HOME/src"
git clone https://git.tylconsulting.it/mptyl/ThothII.git
cd ThothII
git rev-parse --short HEAD
~~~
Annotare la revisione. Il repository workspace verrà scaricato da tht setup --complete dentro un
volume Docker persistente, usando URL, branch e trasporto indicati durante il setup.
## 2. Installare il comando terminale
Dal root del clone:
~~~
bash scripts/check-standalone-prerequisites.sh
mkdir -p "$HOME/.local/bin"
THT_INSTALL_DIRECTORY="$HOME/.local/bin" bash scripts/install-tht.sh
export PATH="$HOME/.local/bin:$PATH"
tht version
~~~
Il comando tht è l’unico componente nativo da installare. Costruisce il binario con Docker e
orchestra Compose; non è un secondo runtime dell’applicazione.
## 3. Preparare pochi segreti e avviare il setup completo
La prima esecuzione crea i placeholder protetti sotto deploy/local/secrets/ e si ferma se manca
una credenziale necessaria. Compilare i file indicati e rilanciare lo stesso comando: i file di
configurazione già compatibili vengono riutilizzati.
~~~
tht setup --complete --profile local --shell-mode full --shell-default-locale en
~~~
Durante il setup servono solo le informazioni operative che il computer non può conoscere:
| Richiesta | Cosa inserire |
| --- | --- |
| Repository workspace | URL del repository dati/configurazione, non ThothII.git |
| Branch | normalmente main |
| Accesso | ssh con chiave e known_hosts, oppure https con credential file e CA |
| DWH/LLM URL | endpoint senza token nella URL |
| Login locale | utente e password iniziale richiesti dal prompt |
Il setup crea automaticamente le password casuali del Catalog e le scrive in operator.env come
percorsi, non come valori. Esegue docker compose config, costruisce le immagini, avvia il Catalog,
esegue catalog-migrate, avvia lo stack e importa il repository workspace. L’import attiva anche
l’Evidence dichiarata: almeno i file source presenti nel workspace vengono materializzati nel
registro locale.
### Il file da compilare
Il file principale è:
~~~
deploy/local/secrets/thothii.secrets
~~~
Inserire solo righe NOME=VALORE necessarie al modelCatalog e agli adapter, per esempio una API key
LLM (DEEPSEEK_API_KEY, OPENAI_API_KEY o quella dichiarata dal catalogo) ed eventualmente
THT_DWH_API_KEY. I nomi ammessi sono documentati in deploy/secrets/README.md. Non mettere token
nelle URL, nel repository o nei comandi.
Due precisazioni evitano gli errori più comuni:
- se il catalogo usa pi_auth, il token LLM va nel file Pi pi-auth.json creato dal setup; {} è solo
un placeholder e non abilita alcun modello;
- le credenziali specifiche di un database workspace (password PostgreSQL, API token REST, chiave
SSH del tunnel, known_hosts, CA) non vanno nel repository workspace: si inseriscono per workspace
in Database Management, che le conserva nel Catalog cifrato. Il workspace indica database/schema
e trasporto; l’installatore deve ottenere dal proprietario il valore corretto.
Per un repository workspace privato sono inoltre indispensabili i file Git richiesti dal trasporto:
una chiave SSH e known_hosts, oppure credential file HTTPS e CA. Sono file di trasporto, non un
secondo bundle da committare. Per ridurre i file da compilare, usare SSH con una deploy key già
autorizzata.
## 4. Controlli automatici e test da terminale
Il setup verifica file, permessi, descriptor, Compose, Docker, autenticazione, servizi, Pi e
workspace. Dopo l’avvio usare questi comandi in qualunque momento:
~~~
INSTALLATION="$PWD/deploy/local/thothii-installation.yaml"
tht --installation "$INSTALLATION" doctor --json
tht --installation "$INSTALLATION" workspace pull --json
tht --installation "$INSTALLATION" workspace test --json
~~~
workspace test prova, per ogni workspace attivo, il binding del database e le credenziali, Evidence,
Qdrant e il servizio embedding. Restituisce exit code diverso da zero se manca il binding del
database o una connessione non è utilizzabile. Prima di eseguirlo l’installatore deve aver
configurato il database in Database Management: il workspace repository da solo non può contenere
la password.
Per una verifica generale del core, doctor --json è il test ripetibile e non distruttivo. Il test
funzionale finale deve inoltre aprire http://127.0.0.1:8080, autenticarsi e completare una domanda
reale fino alla SQL finale.
## Attività che può svolgere solo l’installatore
La procedura automatizza il bootstrap, non può inventare decisioni o autorizzazioni aziendali.
L’installatore deve completare e registrare:
1. quali LLM sono utilizzabili, il relativo modelCatalog e le API key collegate; poi eseguire
tht pi test e tht doctor;
2. per ogni database: configurazione, test connessione, sincronizzazione dello schema e
generazione delle descrizioni;
3. consolidamento umano delle descrizioni generate;
4. generazione delle entry semantiche in Qdrant tramite workspace preprocess run;
5. generazione delle FK suggerite dal naming, come complemento alle FK lette dallo schema, revisione
umana delle proposte e caricamento delle relazioni approvate in Qdrant;
6. verifica periodica con tht doctor --json e tht workspace test --json;
7. una domanda reale completata con successo, senza errori di connessione o modello.
La configurazione è dichiarata completa solo quando tutti i punti applicabili sono stati eseguiti,
le decisioni sono state registrate e i due test terminali sono verdi. Il core è dichiarato usabile
solo dopo la domanda reale, non perché il frontend risponde a /health.
## Gate A e Gate B
### Gate A — piattaforma
~~~
uname -a
docker version --format '{{.Server.Version}} {{.Server.Arch}}'
tht version
bash scripts/check-standalone-prerequisites.sh
bash scripts/verify-standalone-install.sh "$INSTALLATION"
~~~
### Gate B — usabilità
~~~
tht --installation "$INSTALLATION" doctor --json
tht --installation "$INSTALLATION" workspace test --json
curl --fail --silent --show-error http://127.0.0.1:8080/health
~~~
Poi eseguire una domanda reale e fermare/riavviare con tht stop e tht start. Non usare docker
compose down --volumes: cancella Catalog, sessioni, Qdrant e il modello embedding.
## Diagnosi rapida
| Sintomo | Azione |
| --- | --- |
| Docker Engine is not reachable | avviare Docker Desktop o systemctl e ripetere docker info |
| Omarchy non trova docker | installare Docker/Compose, abilitare il servizio e riaprire la sessione |
| Windows vede Docker ma Bash fallisce | usare Ubuntu WSL2 e abilitarne l’integrazione in Docker Desktop |
| pull workspace fallisce | controllare URL, branch, chiave/credential file e known_hosts dal container |
| workspace test segnala binding mancante | configurare database, token/password e CA in Database Management |
| Pi non è pronto | compilare pi-auth.json o la chiave dichiarata dal modelCatalog, poi eseguire tht pi test |
## Documenti collegati
- [Installazione e primo avvio](first-start.md)
- [Operazioni sui workspace](../operations/workspaces.md)
- [Database Management](../operations/database-management.md)
- [Configurazione dei modelli](../general/pi-configuration.md)
- deploy/secrets/README.md
La pubblicazione di immagini su Docker Hub e gli installer nativi DMG/MSI/AppImage restano attività
successive: questa procedura parte dal clone Gitea e non richiede immagini Docker Hub pre-pubblicate.
-129
View File
@@ -1,129 +0,0 @@
# Docker installation in the current operating contexts
ThothII uses one Compose topology:
- `frontend`
- `core`
- `catalog-db`
- `qdrant`
- `embedding`
- `embedding-model-init`
- `catalog-migrate` (one-shot)
Qdrant and Ollama embedding are required internal Compose services. Only DWH and the LLM remain
external. The fixed model is `qwen3-embedding:0.6b` with 1024 dimensions and cosine distance;
`embedding-model-init` prepares it before `core` starts.
```mermaid
flowchart TB
INSTALL["Installation descriptor"] --> CONTEXT{Context}
CONTEXT --> LOCAL["Local\nCompose local"]
CONTEXT --> SERVER["Server\nCompose server"]
CONTEXT --> SESSION["Session server\noperator services"]
CONTEXT --> AUTH["Auth runtime\nprojection services"]
LOCAL --> BUNDLE["Common secret bundle"]
SERVER --> BUNDLE
SESSION --> BUNDLE
AUTH --> BUNDLE
BUNDLE --> SERVICES["Frontend, core, catalog DB, vector, embedding"]
```
## Short ownership contract
| Componente | Ownership | Contratto operativo |
| --- | --- | --- |
| DWH | External | External endpoint configured by the installation. |
| LLM | External | Endpoint or policy outside the internal semantic infrastructure. |
| Qdrant | Internal | Required internal Compose service with persistent `qdrant-data` volume. |
| Ollama embedding | Internal | Required internal Compose service for `qwen3-embedding:0.6b`. |
## Local path: setup, migration, start
The normal local path is [Install and first start](install/first-start.md). It uses `tht setup`
to create the selected installation-local descriptor and runs the catalog migration explicitly
before application startup.
If an operator intentionally prepares the descriptor and protected files by hand, the equivalent
foreground launch is:
```sh
cp deploy/env/local.env.example deploy/env/local.env
cp deploy/secrets/thothii.secrets.example deploy/secrets/thothii.secrets
chmod 600 deploy/secrets/thothii.secrets
# Copy docs/install/examples/thothii-installation.local.yaml to a protected operator path,
# replace every placeholder, chmod it 600, then set that exact path as
# THT_INSTALLATION_CONFIG_SOURCE in deploy/env/local.env.
./scripts/run-stack.sh
```
Set these values in `deploy/env/local.env`:
- `PI_AUTH_FILE`
- `THT_SECRETS_FILE`
- `THT_INSTALLATION_CONFIG_SOURCE` (the exact protected host `thothii-installation.yaml`)
- `THT_WORKSPACE_GIT_REMOTE`
- DWH endpoint
- LLM endpoint
Do not put secrets in `.env`. Runtime secrets belong in the
`deploy/secrets/thothii.secrets` bundle.
## Secret bundle
The documented and supported bundle keys are:
```dotenv
THT_MODEL_API_KEY=...
THT_DWH_API_KEY=...
OPENAI_API_KEY=...
```
`THT_MODEL_API_KEY` is available only as an explicitly declared catalog bundle key. A
metadata-generation provider references one audited bundle name from `deploy/secrets/README.md`
through `modelCatalog.providers.<provider>.authentication.apiKeyEnv`.
`apiKeyEnv` may be omitted only for an explicit endpoint that accepts unauthenticated requests;
hosted/default endpoints remain keyed.
Provider/model/endpoint settings stay in the protected installation descriptor; raw keys do not.
Compose mounts exactly `THT_INSTALLATION_CONFIG_SOURCE` into `core` as a read-only config and sets
the backend-only runtime path `THT_INSTALLATION_CONFIG_FILE` to
`/run/thothii-installation/thothii-installation.yaml`. Do not set the runtime path in the host env.
Descriptor and bundle changes are loaded only after application restart.
A private PEM CA remains outside the bundle and must be mounted through a reviewed Compose override.
## Preprocessing
Preprocessing runs through the native host CLI and the installation descriptor:
```sh
tht --installation /percorso/assoluto/thothii-installation.yaml \
workspace preprocess run --workspace <workspace-id>
```
Per rimuovere soltanto gli indici e gli artifact ricostruibili, preservando Memory e domande risolte:
```sh
tht --installation /percorso/assoluto/thothii-installation.yaml \
workspace preprocess clear --workspace <workspace-id>
```
The CLI runs the profile-gated `workspace-maintenance` service. See the
[preprocessing contract](contracts/workspace-preprocessing-cli.md) and the
[Evidence guide](evidence.md) for details.
## Server
For server installations, use the server profile with the session overlay:
```sh
docker compose --env-file deploy/env/server.env \
-f compose.yaml -f deploy/compose.server.yaml \
-f deploy/compose.session-server.yaml.example up --build -d
```
Also see [Workspace operations](operations/workspaces.md). Server deployment, reverse-proxy,
backup, and recovery remain manual-gated operations; do not treat the local profile as a server
replacement.
@@ -0,0 +1,435 @@
# Revisione e bonifica della documentazione
Data: 15 settembre 2026. Baseline: `6a4634dcf1b1df4ad29510a4da371245abd5c666`.
Ambito: 121 Markdown e 23 altri file già presenti in `docs/`, navigazione, collegamenti
locali, build e verificatori documentali. Il documento conserva la proposta iniziale
e registra sotto l'esecuzione successivamente autorizzata dall'utente. È interno,
non una pagina del manuale pubblico.
## Bonifica eseguita dopo approvazione
Ritirati 35 file dal working tree: 23 Markdown e 12 artefatti visuali generati
(otto PNG, due HTML e due JSON). Recuperabili integralmente dalla revisione
`5f3a7f5975b96fae1e1cdd0da08b4d60d41064cc`, precedente a questo intervento,
mediante `git show REVISIONE:percorso`. Non eliminata né riscritta la storia Git.
- Dettagli Compose utili consolidati in [riferimento developer](../operations/compose-reference.md);
README rinvia alle sole due procedure manuali IT/EN per installazioni nuove.
- M1–M3, E1–E3, X1 e i piani Catalog/model già implementati consolidati nel
[record di rilascio](../reports/knowledge-archives-release.md). Test storici non
presentati come collaudi attuali; estensioni e accettazioni aperte conservate.
- Prompt security assorbito nel PRD, senza approvare il progetto di hardening.
- Snapshot di progetto riscritto; overview corretta su Catalog, Memory ed Evidence.
- Verificatori auth/DWH e fixture riallineati ai documenti attuali: controlli su
permessi, diagnostica, segreti, TLS, separazione dei servizi e link. Le simulazioni
del vecchio journal scanner, non più presente nei documenti, non sono conservate
come finto collaudo del runtime. Le suite applicative non vengono modificate.
- Rimandi Markdown verificati; contratti, ADR (anche superati), progetti con
decisioni ancora aperte e runbook/report con rollback o accettazione pendente
restano nel repository, esclusi dal sito. Non spostati gate in nuove issue né
dichiarati chiusi: lo snapshot corrente li rende espliciti. ADR 0019 conservato
senza dedurre dal solo nome che tutte le sue estensioni siano implementate.
- Pubblicazione effettiva distinta dal branch `pages`; procedura ripetibile nel
[runbook del manuale](../operations/public-docs-publication.md).
File ritirati (percorsi dalla radice):
- `docs/architecture/thothii-core-sequence.visual-check.1440x900.dark.png`
- `docs/architecture/thothii-core-sequence.visual-check.1440x900.light.png`
- `docs/architecture/thothii-core-sequence.visual-check.2048x1320.dark.png`
- `docs/architecture/thothii-core-sequence.visual-check.2048x1320.light.png`
- `docs/architecture/thothii-core-sequence.visual-check.html`
- `docs/architecture/thothii-core-sequence.visual-check.json`
- `docs/architecture/thothii-runtime.visual-check.1440x900.dark.png`
- `docs/architecture/thothii-runtime.visual-check.1440x900.light.png`
- `docs/architecture/thothii-runtime.visual-check.2048x1320.dark.png`
- `docs/architecture/thothii-runtime.visual-check.2048x1320.light.png`
- `docs/architecture/thothii-runtime.visual-check.html`
- `docs/architecture/thothii-runtime.visual-check.json`
- `docs/installazione-docker-4-contesti.md`
- `docs/plans/2026-08-26-metadata-catalog-from-thothai.md`
- `docs/plans/2026-08-28-ai-catalog-description-generation-spec.md`
- `docs/plans/2026-08-28-ai-catalog-description-generation.md`
- `docs/plans/2026-09-02-installation-model-catalog.md`
- `docs/plans/2026-09-08-memory-m1-validation.md`
- `docs/plans/2026-09-08-security-hardening-resume-prompt.md`
- `docs/plans/2026-09-09-archive-repair-x1-validation.md`
- `docs/plans/2026-09-09-evidence-e1-validation.md`
- `docs/plans/2026-09-09-evidence-e2-validation.md`
- `docs/plans/2026-09-09-evidence-e3-validation.md`
- `docs/plans/2026-09-09-memory-m2-validation.md`
- `docs/plans/2026-09-09-memory-m3-validation.md`
- `docs/research/2026-08-23-legacy-thothai-metadata-capabilities.md`
- `docs/research/2026-08-23-postgresql-catalog-deployment-constraints.md`
- `docs/research/2026-08-23-thothii-metadata-publication-qdrant-seams.md`
- `docs/research/2026-09-07-antigravity-gemini-flash-value.md`
- `docs/research/2026-09-07-deepseek-harness-terminal.md`
- `docs/research/2026-09-07-omp-codex-zcode.md`
- `docs/research/2026-09-07-pi-config-vs-oh-my-pi.md`
- `docs/research/2026-09-07-zai-coding-plan-cli-quota.md`
- `docs/research/2026-09-07-zed-acp-omp-pi.md`
- `docs/research/text-to-sql-products.md`
Le sezioni successive e l'inventario fotografano la revisione iniziale: la presente
sezione prevale per le azioni eseguite. Gli altri ritiri sono rinviati finché non
si chiudono i relativi gate; non si cancellano documenti solo perché datati.
## Esito e modifica applicata
Il sito confondeva quattro funzioni: manuale del prodotto, riferimento del codice,
progettazione e diario delle consegne. Il problema non era soltanto la lunghezza della nav:
le pagine non elencate potevano comunque essere generate e indicizzate.
La configurazione ora pubblica soltanto 20 pagine e cinque asset esplicitamente ammessi.
Il resto è escluso da HTML, file copiati e indice di ricerca. Le nuove pagine pubbliche
sono `product-overview.md`, `install/shell-and-language.md` e `usage/memory.md`;
i documenti tecnici da cui sono state ricavate restano intatti nel repository.
Home e `install/first-start.md` sono state riscritte per dare un ingresso univoco.
Non è stato cancellato alcun documento della baseline.
`exclude_docs` nega per default la pubblicazione: aggiungere un file a `docs/` non basta
più a metterlo online. La build e la pipeline verificano corrispondenza tra nav ed
eccezioni, pagine generate, asset e ricerca. I riferimenti tecnici ancora necessari
al lettore rinviano esplicitamente ai sorgenti su Gitea, non a pagine interne del sito.
**Fuori dal manuale non significa privato.** Il repository ThothII è pubblico: questi
file e la cronologia restano leggibili su Gitea. Per riservatezza reale occorrerebbe una
decisione separata su repository/accessi e sulla storia già pubblicata. Questa bonifica
non modifica autorizzazioni, altri repository o l'installazione applicativa.
## Tre destinazioni, con regole diverse
| Destinazione | Contenuto | Regola |
| --- | --- | --- |
| Manuale pubblico | Prodotto, uso, installazione, configurazione, amministrazione | Descrivere ciò che il lettore può fare oggi; distinguere prerequisiti e limiti verificati. |
| Riferimento developer | Architettura, contratti, ADR, test ripetibili, integrazione | Conservare nel repository, fuori da MkDocs; un'autorità per ciascun contratto. |
| Lavoro temporaneo o storico | Piani attuati, prompt di ripresa, survey, report di singola consegna | Estrarre decisioni, difetti aperti e rollback ancora necessari; poi ritirare dal working tree con Git come archivio. |
Un file datato non è automaticamente obsoleto. Un contratto con `v3` nel nome non è
automaticamente sostituito da un descriptor v4: sono versioni di oggetti diversi.
Un ADR superato conserva il motivo della decisione e il collegamento al successore.
## Rilievi prioritari, con motivazione
### 1. Percorsi di installazione concorrenti — corretto per il pubblico
`install/first-start.md` proponeva ancora setup senza `--configure-only` e avvio con
`run-stack.sh`, mentre le guide manuali separano segreti, migrazione e avvio per lo
stesso descriptor/progetto. Ora è un punto d'ingresso alle due procedure complete,
non una terza ricetta. `installazione-docker-4-contesti.md` rimane interno: consolidarne
i dettagli ancora esclusivi in una procedura developer di distribuzione, poi ritirarlo.
Anche il percorso rapido nel README va riallineato prima di eliminare quei dettagli.
### 2. Ricerca architetturale presentata come stato corrente — ritirare dopo estrazione
`research/2026-08-23-thothii-metadata-publication-qdrant-seams.md` dichiara ancora il
passaggio catalog-to-core «deferred» e descrive `annotations.yaml` come input corrente.
ADR 0016 e gli attuali contratti registrano invece PostgreSQL come autorità per i
consumatori del core. Non usarlo per implementare nuovi comportamenti.
`research/2026-08-23-postgresql-catalog-deployment-constraints.md` dichiara esplicitamente
di essere parzialmente superato da ADR 0004. L'inventario legacy ThothAI del 23 agosto
è esplicitamente storico. Estrarre soltanto vincoli non già presenti in ADR/contratti;
poi eliminare i tre file dal working tree, conservandoli nella storia Git.
### 3. Decisioni correnti distribuite tra troppi piani — consolidare prima di eliminare
Per Memory/Evidence convivono piani di famiglia, M1, amministrazione comune, revisione
di semplificazione e sette resoconti M1–M3/E1–E3/X1. Conservare gli invarianti nei
contratti e gli scenari ripetibili in `testing/`; trasferire i gate non chiusi in issue.
Solo dopo si possono ritirare piani e validazioni di incremento.
`adr/0019-author-evidence-in-app-with-automatic-activation.md` ha un nome che suggerisce
editor in-app e attivazione automatica, ma il titolo descrive editor esterni e
consolidamento manuale. Contiene anche «Implementation is pending»: non è un affidabile
indicatore dello stato di tutte le parti della release. Conservare numero/decisione,
correggere metadati e rimandi e distinguere eventuali estensioni ancora pendenti.
### 4. Report operativi e piani con stato non aggiornato — non cancellare in blocco
Il report `2026-09-12-ui-visual-review-delivery.md` apre con «non integrata in main»;
le integrazioni successive sono documentate altrove. I piani full-shell dicono ancora
«deploy server non eseguito», mentre esistono report di rilascio server del 14 settembre.
Ciò prova che lo stato è distribuito, non che tutti i collaudi siano conclusi.
Consolidare report UI del 12–13 settembre in un record di rilascio per versione.
Non ritirare i report del 14 settembre né i runbook di upgrade finché rollback,
accettazione interattiva e finestra di osservazione non risultano chiusi dall'operatore.
`server-handoff-260906-preprocessing-complete.md` va confrontato con il runbook corrente
`server-codex-handoff.md`: estrarre eventuali passaggi di preprocessing esclusivi prima
di ritirare la consegna datata.
### 5. Ricerca su strumenti personali estranea al manuale — candidata all'eliminazione
I sei confronti CLI/editor/provider del 7 settembre (Antigravity, DeepSeek/CyberArk,
OMP/Codex/ZCode, Pi/Oh My Pi, quota Z.ai e Zed/ACP) non spiegano un contratto di ThothII.
Prezzi, quote e confronti non sono stati riverificati in questa revisione. Proposta:
toglierli dal repository del prodotto, mantenendoli in Git o trasferendoli, su scelta
del proprietario, a una raccolta personale. Stessa destinazione proposta per
`research/text-to-sql-products.md`, che è una ricognizione di mercato.
Non applicare automaticamente questa regola alla ricerca ERD, relationship e
classificatore sensibile: prima verificare se documenta decisioni ancora aperte.
### 6. Controlli e snapshot già disallineati — debito da correggere
Due verificatori falliscono su file già assenti nella baseline, non per l'esclusione
da MkDocs introdotta qui:
- `scripts/auth-docs-smoke.sh`: manca `docs/install/local.md`.
- `scripts/verify-dwh-auth-docs.sh`: manca `docs/operations/psd-dwh-auth-rollout.md`.
Le relative suite di fixture conservano la vecchia struttura. Non ricreare documenti
obsoleti per far passare i test: aggiornare i controlli ai contratti attuali, mantenendo
le verifiche su segreti, modalità di autenticazione e TLS. Questo riallineamento è
proposto, non eseguito in questa bonifica editoriale.
`PROJECT_STATE.md` cita nove percorsi `docs/*.md` non più esistenti, fra cui il programma
server PSD del 20 agosto, l'accettazione Evidence del 25 agosto e il rollout dwh-auth.
Inoltre accumula consegne anziché restare uno snapshot breve. Riscriverlo come stato
attuale, gate aperti e rimandi esistenti. I normali link Markdown relativi dei documenti
esistenti non presentano target mancanti nella verifica eseguita: i riferimenti rotti
qui citati sono anche percorsi in backtick o imposti dagli script.
## Proposta operativa in ordine
0. **Collegare il sito servito al risultato della pipeline:** il controllo remoto descritto
sotto ha rilevato una copia statica vecchia. Prima di dichiarare la bonifica online,
aggiornare il percorso di pubblicazione sul server con accesso amministrativo verificato.
1. **Separazione pubblica:** applicata; niente cancellazioni di sorgenti.
2. **Riallineare le autorità:** snapshot, verificatori, ADR 0019, README e stato delle
consegne; spostare gate aperti in issue senza marcarli completati.
3. **Ritiro a basso rischio, previa approvazione:** confronti personali, ricerca già
esplicitamente superata, prompt di ripresa dopo assorbimento nel PRD, output visuali
rigenerabili. Controllare prima i riferimenti in tutto il repository.
4. **Consolidamento:** piani completati e report incrementali; non perdere criteri di
accettazione, problemi aperti, provenienza delle immagini e rollback ancora validi.
5. **Riorganizzazione eventuale:** `docs/` per il pubblico, `developer-docs/` per
architettura/contratti/ADR/testing, `project-notes/` per lavoro in corso. È un secondo
intervento: aggiornare insieme tutti i link in README, AGENTS, CONTEXT, script e codice.
Per ogni ritiro usare un commit dedicato con motivazione e documento sostitutivo.
Non creare una cartella `archive/` piena di copie obsolete: la cronologia Git è già
l'archivio, salvo una necessità operativa o di conservazione esplicita.
## Inventario e destinazione proposta
L'inventario seguente distingue l'azione proposta dalla sola esclusione già applicata
al sito. «Consolidare» e «ritirare» non indicano cancellazioni eseguite.
<!-- inventory:start -->
| File (relativo a `docs/`) | Destinazione | Azione proposta |
| --- | --- | --- |
| `adr/0001-postgres-metadata-catalog.md` | Developer | Conservare decisione architetturale. |
| `adr/0002-workspace-database-secret-references.md` | Developer | Conservare decisione architetturale. |
| `adr/0003-installation-local-database-bindings.md` | Developer | Conservare decisione architetturale. |
| `adr/0004-fastify-kysely-metadata-catalog.md` | Developer | Conservare decisione architetturale. |
| `adr/0005-hard-delete-catalog-tables-during-synchronization.md` | Developer | Conservare ADR superato e catena dei successori; non eliminarlo. |
| `adr/0006-separate-physical-and-logical-relationships.md` | Developer | Conservare decisione architetturale. |
| `adr/0007-durable-authoritative-schema-synchronization.md` | Developer | Conservare decisione architetturale. |
| `adr/0008-allow-manual-catalog-metadata-cleanup.md` | Developer | Conservare decisione architetturale. |
| `adr/0009-use-one-sequential-description-generation-run.md` | Developer | Conservare decisione architetturale. |
| `adr/0010-allow-bounded-real-source-samples-for-description-generation.md` | Developer | Conservare decisione architetturale. |
| `adr/0011-gate-source-samples-with-a-sensitive-data-flag.md` | Developer | Conservare decisione architetturale. |
| `adr/0012-use-the-catalog-as-the-logical-relationship-authority.md` | Developer | Conservare decisione architetturale. |
| `adr/0013-use-one-installation-model-catalog-with-runtime-projections.md` | Developer | Conservare decisione architetturale. |
| `adr/0014-assess-sensitive-columns-locally-from-source-content.md` | Developer | Conservare ADR superato e catena dei successori; non eliminarlo. |
| `adr/0015-use-progressive-sampling-for-sensitive-columns.md` | Developer | Conservare decisione architetturale. |
| `adr/0016-use-postgres-metadata-for-all-core-consumers.md` | Developer | Conservare decisione architetturale. |
| `adr/0017-separate-reference-vectors-from-runtime-memory.md` | Developer | Conservare decisione architetturale. |
| `adr/0018-use-postgres-for-memory-and-qdrant-for-retrieval.md` | Developer | Conservare decisione architetturale. |
| `adr/0019-author-evidence-in-app-with-automatic-activation.md` | Developer | Conservare; allineare nome/stato alla decisione manuale effettiva. |
| `adr/0020-unify-administration-pages-and-use-namespaced-routes.md` | Developer | Conservare decisione architetturale. |
| `adr/0021-separate-shell-modes-and-replaceable-portal-adapter.md` | Developer | Conservare decisione architetturale. |
| `adr/0022-separate-ui-locale-from-session-interaction-language.md` | Developer | Conservare decisione architetturale. |
| `agents/domain.md` | Developer | Conservare regole di contribuzione/agenti, fuori dal sito. |
| `agents/issue-tracker.md` | Developer | Conservare regole di contribuzione/agenti, fuori dal sito. |
| `agents/triage-labels.md` | Developer | Conservare regole di contribuzione/agenti, fuori dal sito. |
| `architecture/application-shell.md` | Developer | Conservare riferimento corrente; distinguere il prodotto dalla struttura del codice. |
| `architecture/authentication.md` | Developer | Conservare riferimento corrente; distinguere il prodotto dalla struttura del codice. |
| `architecture/components.md` | Developer | Conservare riferimento corrente; distinguere il prodotto dalla struttura del codice. |
| `architecture/overview.md` | Developer | Conservare riferimento corrente; distinguere il prodotto dalla struttura del codice. |
| `architecture/thothii-core-sequence.html` | Developer/build | Conservare sorgente/diagramma tecnico; non pubblicare nel manuale. |
| `architecture/thothii-core-sequence.visual-check.1440x900.dark.png` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `architecture/thothii-core-sequence.visual-check.1440x900.light.png` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `architecture/thothii-core-sequence.visual-check.2048x1320.dark.png` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `architecture/thothii-core-sequence.visual-check.2048x1320.light.png` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `architecture/thothii-core-sequence.visual-check.html` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `architecture/thothii-core-sequence.visual-check.json` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `architecture/thothii-core.sequence.json` | Developer/build | Conservare sorgente/diagramma tecnico; non pubblicare nel manuale. |
| `architecture/thothii-runtime.architecture.json` | Developer/build | Conservare sorgente/diagramma tecnico; non pubblicare nel manuale. |
| `architecture/thothii-runtime.html` | Developer/build | Conservare sorgente/diagramma tecnico; non pubblicare nel manuale. |
| `architecture/thothii-runtime.visual-check.1440x900.dark.png` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `architecture/thothii-runtime.visual-check.1440x900.light.png` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `architecture/thothii-runtime.visual-check.2048x1320.dark.png` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `architecture/thothii-runtime.visual-check.2048x1320.light.png` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `architecture/thothii-runtime.visual-check.html` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `architecture/thothii-runtime.visual-check.json` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `contracts/archive-repair.md` | Developer | Conservare contratto e versioni semantiche; aggiornare insieme a codice/test. |
| `contracts/catalog-schema-snapshot.md` | Developer | Conservare contratto e versioni semantiche; aggiornare insieme a codice/test. |
| `contracts/curated-evidence-v4.md` | Developer | Conservare contratto e versioni semantiche; aggiornare insieme a codice/test. |
| `contracts/portal-shell-adapter-v1.md` | Developer | Conservare contratto e versioni semantiche; aggiornare insieme a codice/test. |
| `contracts/tht-dwh.md` | Developer | Conservare contratto e versioni semantiche; aggiornare insieme a codice/test. |
| `contracts/workspace-evidence-v3.md` | Developer | Conservare contratto e versioni semantiche; aggiornare insieme a codice/test. |
| `contracts/workspace-preprocessing-cli.md` | Developer | Conservare contratto e versioni semantiche; aggiornare insieme a codice/test. |
| `disambiguazione-iniziale.md` | Developer | Conservare invarianti di gate/ledger; il percorso utente è in skills.md. |
| `evidence.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `general/pi-configuration.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `gestione-memory.md` | Developer | Conservare contratto di Memory; istruzioni pubbliche in usage/memory.md. |
| `guida-utente.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `index.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `install/authentication-local.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `install/authentication-oidc.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `install/authentication-upstream.md` | Developer/integratore | Conservare integrazione e confine di fiducia; guida pubblica separata. |
| `install/authentik.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `install/dwh-auth-client-enrollment.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `install/dwh-auth-server.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `install/dwh-auth-tls.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `install/examples/thothii-installation.local.yaml` | Asset pubblico | Conservare asset o esempio operativo. |
| `install/examples/thothii-installation.server.yaml` | Asset pubblico | Conservare asset o esempio operativo. |
| `install/examples/workspace-bindings.env.example` | Asset pubblico | Conservare asset o esempio operativo. |
| `install/first-start.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `install/standalone-manual-en.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `install/standalone-manual-it.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `installazione-docker-4-contesti.md` | Misto/transitorio | Estrarre dettagli server unici, poi ritirare la ricetta duplicata. |
| `javascripts/layout-init.js` | Asset pubblico | Conservare asset o esempio operativo. |
| `operations/database-management.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `operations/docker-refresh.md` | Operazioni interne | Conservare finché descrive il contesto locale attivo; poi consolidare il lifecycle. |
| `operations/sensitivity-analysis.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `operations/server-codex-handoff.md` | Operazioni interne | Conservare runbook finché migrazione e rollback sono operativamente necessari. |
| `operations/server-handoff-260906-preprocessing-complete.md` | Operazioni/transitorio | Confrontare con server-codex-handoff, assorbire passaggi unici, poi ritirare. |
| `operations/server-upgrade-gitea-workspace-v2.md` | Operazioni interne | Conservare runbook finché migrazione e rollback sono operativamente necessari. |
| `operations/shell-and-localization.md` | Developer/integratore | Conservare integrazione e confine di fiducia; guida pubblica separata. |
| `operations/workspaces.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `plans/2026-08-26-metadata-catalog-from-thothai.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-08-28-ai-catalog-description-generation-spec.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-08-28-ai-catalog-description-generation.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-02-installation-model-catalog.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-08-evidence-management.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-08-memory-evidence-administration.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-08-memory-evidence-simplification-review.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-08-memory-m1-spec.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-08-memory-m1-validation.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-08-memory-management.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-08-security-hardening-prd.md` | Progetto attivo | Conservare PRD aperto; risolvere decisioni/issue prima del ritiro. |
| `plans/2026-09-08-security-hardening-resume-prompt.md` | Transitorio | Ritirare dopo trasferimento di istruzioni e questioni aperte nel PRD/issue. |
| `plans/2026-09-09-archive-repair-x1-validation.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-09-evidence-e1-validation.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-09-evidence-e2-validation.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-09-evidence-e3-validation.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-09-memory-m2-validation.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-09-memory-m3-validation.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-10-administration-pages-spec.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-13-full-shell-and-portal-integration.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-13-full-shell-spec.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-14-manual-standalone-installation.md` | Progetto attivo | Conservare fino alla verifica su tre piattaforme; poi assorbire gate e ritirare. |
| `plans/administration-pages/01-navigation.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/administration-pages/02-workspace-readiness.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/administration-pages/03-workbench-family.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/administration-pages/04-embedded-acceptance.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `reports/2026-09-02-psd-sensitivity-shadow.md` | Developer/benchmark | Conservare risultati aggregati come evidenza versionata, non garanzia corrente. |
| `reports/2026-09-02-sensitivity-ner-license-inventory.md` | Developer/release | Conservare provenienza licenze della release; rigenerare quando cambiano le dipendenze. |
| `reports/2026-09-03-psd-progressive-sensitivity-shadow.md` | Developer/benchmark | Conservare risultati aggregati come evidenza versionata, non garanzia corrente. |
| `reports/2026-09-12-admin-issues-28-31.md` | Progetto/transitorio | Consolidare in record di release; ritirare solo dopo trasferimento gate/rollback utili. |
| `reports/2026-09-12-context-shelf-a-implementation.md` | Progetto/transitorio | Consolidare in record di release; ritirare solo dopo trasferimento gate/rollback utili. |
| `reports/2026-09-12-ui-survey-and-graphic-revision-plan.md` | Progetto/transitorio | Consolidare in record di release; ritirare solo dopo trasferimento gate/rollback utili. |
| `reports/2026-09-12-ui-visual-review-delivery.md` | Progetto/transitorio | Consolidare in record di release; ritirare solo dopo trasferimento gate/rollback utili. |
| `reports/2026-09-12-unified-interaction-model.md` | Progetto/transitorio | Consolidare in record di release; ritirare solo dopo trasferimento gate/rollback utili. |
| `reports/2026-09-13-full-shell-implementation.md` | Progetto/transitorio | Consolidare in record di release; ritirare solo dopo trasferimento gate/rollback utili. |
| `reports/2026-09-13-header-layout-refinements.md` | Progetto/transitorio | Consolidare in record di release; ritirare solo dopo trasferimento gate/rollback utili. |
| `reports/2026-09-13-knowledge-reading.md` | Progetto/transitorio | Consolidare in record di release; ritirare solo dopo trasferimento gate/rollback utili. |
| `reports/2026-09-13-knowledge-typography.md` | Progetto/transitorio | Consolidare in record di release; ritirare solo dopo trasferimento gate/rollback utili. |
| `reports/2026-09-13-shell-simplification-review.md` | Progetto/transitorio | Consolidare in record di release; ritirare solo dopo trasferimento gate/rollback utili. |
| `reports/2026-09-13-visual-shell-integration.md` | Progetto/transitorio | Consolidare in record di release; ritirare solo dopo trasferimento gate/rollback utili. |
| `reports/2026-09-14-session-dialogs-release.md` | Operazioni/release | Conservare finché accettazione, osservazione e rollback non sono chiusi. |
| `reports/2026-09-14-session-layout-memory-fix.md` | Operazioni/release | Conservare finché accettazione, osservazione e rollback non sono chiusi. |
| `requirements.lock` | Developer/build | Conservare dipendenze e lock; esclusi dal sito. |
| `requirements.txt` | Developer/build | Conservare dipendenze e lock; esclusi dal sito. |
| `research/2026-08-23-legacy-thothai-metadata-capabilities.md` | Storico/superato | Estrarre vincoli non assorbiti da ADR/contratti, poi ritirare. |
| `research/2026-08-23-postgresql-catalog-deployment-constraints.md` | Storico/superato | Estrarre vincoli non assorbiti da ADR/contratti, poi ritirare. |
| `research/2026-08-23-thothii-metadata-publication-qdrant-seams.md` | Storico/superato | Estrarre vincoli non assorbiti da ADR/contratti, poi ritirare. |
| `research/2026-08-31-browser-erd-library-options.md` | Developer/ricerca | Verificare decisioni ancora aperte; assorbire conclusioni in issue/ADR, poi ritirare. |
| `research/2026-08-31-relationship-management-thothai-to-thothii.md` | Developer/ricerca | Verificare decisioni ancora aperte; assorbire conclusioni in issue/ADR, poi ritirare. |
| `research/2026-09-02-local-sensitive-column-classifier-libraries.md` | Developer/ricerca | Verificare decisioni ancora aperte; assorbire conclusioni in issue/ADR, poi ritirare. |
| `research/2026-09-07-antigravity-gemini-flash-value.md` | Fuori prodotto | Proposta di ritiro dal working tree; Git o raccolta personale su scelta del proprietario. |
| `research/2026-09-07-deepseek-harness-terminal.md` | Fuori prodotto | Proposta di ritiro dal working tree; Git o raccolta personale su scelta del proprietario. |
| `research/2026-09-07-omp-codex-zcode.md` | Fuori prodotto | Proposta di ritiro dal working tree; Git o raccolta personale su scelta del proprietario. |
| `research/2026-09-07-pi-config-vs-oh-my-pi.md` | Fuori prodotto | Proposta di ritiro dal working tree; Git o raccolta personale su scelta del proprietario. |
| `research/2026-09-07-zai-coding-plan-cli-quota.md` | Fuori prodotto | Proposta di ritiro dal working tree; Git o raccolta personale su scelta del proprietario. |
| `research/2026-09-07-zed-acp-omp-pi.md` | Fuori prodotto | Proposta di ritiro dal working tree; Git o raccolta personale su scelta del proprietario. |
| `research/text-to-sql-products.md` | Fuori prodotto | Proposta di ritiro dal working tree; Git o raccolta personale su scelta del proprietario. |
| `skills.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `stylesheets/extra.css` | Asset pubblico | Conservare asset o esempio operativo. |
| `testing/2026-08-29-ai-catalog-description-generation-acceptance.md` | Developer/QA | Conservare criteri ripetibili e gate; separare risultati di singola release. |
| `testing/2026-08-31-metadata-privacy-description-test-plan.md` | Developer/QA | Conservare scenari; ridurre duplicazioni e distinguere test proposti da eseguiti. |
| `testing/authentication-manual-acceptance.md` | Developer/QA | Conservare criteri ripetibili e gate; separare risultati di singola release. |
| `testing/evidence-lifecycle-test-plan.md` | Developer/QA | Conservare criteri ripetibili e gate; separare risultati di singola release. |
<!-- inventory:end -->
## Verifica iniziale, prima dell'esecuzione dei ritiri
- Build MkDocs con `--strict` e dipendenze bloccate: superata.
- Verifica confine pubblico: 20 pagine, nessun file interno generato o indicizzato.
- Otto fixture positive/negative del confine pubblico: superate.
- Suite corrente installazione/workspace: superata.
- Due verificatori legacy: fallimenti preesistenti descritti sopra; non dichiarati verdi.
- Nessuna certificazione di nuova installazione o nuovo collaudo applicativo.
La pubblicazione remota va verificata separatamente dalla build locale: il ramo sorgente
`main`, il ramo generato `pages` e il sito servito non sono la stessa prova.
### Primo esito remoto del 15 settembre 2026 — diagnosi storica
Il commit `043ffdfa` è stato pubblicato su `main`. La
[pipeline 122](https://git.tylconsulting.it/mptyl/ThothII/actions/runs/122) è terminata
con successo e il ramo `pages` contiene 20 pagine pubbliche, senza pagine interne
nell'indice di ricerca verificato anonimamente.
L'URL `https://git.tylconsulting.it/thothii-docs/` serve però ancora la home precedente,
con `Last-Modified: Wed, 26 Aug 2026 09:00:07 GMT`; il suo indice di ricerca include
ancora architettura e contratti. La richiesta senza cache restituisce lo stesso risultato.
Nel repository la pipeline aggiorna soltanto il ramo `pages`: non è documentato il
collegamento alla directory effettivamente servita dal web server.
Il tentativo SSH in sola lettura con la configurazione locale non è proseguito:
manca una host key ED25519 conosciuta per `git.tylconsulting.it`. Non è stata disabilitata
la verifica dell'identità del server. Servono host/alias amministrativo verificato e
percorso o meccanismo di pubblicazione prima di intervenire sul sito effettivo.
Al momento di quella verifica la copia servita restava da aggiornare. L'accesso è
stato poi risolto usando l'alias SSH già configurato e fidato `contabo`; nessuna
host key è stata aggirata.
## Verifica della bonifica eseguita
- Build MkDocs strict e confine pubblico: superati, 20 pagine.
- Otto fixture del confine pubblico: superate.
- Contratti auth/DWH correnti e relative fixture di mutazione: superati.
- Suite installazione/workspace, Compose canonico e contratto comandi deployment: superate.
- Link Markdown locali di README, PROJECT_STATE e tutti i documenti: nessun target mancante.
- `git diff --check`: superato.
- `test-no-deployment-coupling.sh`: resta rosso su rilievi preesistenti relativi
a contenuti locali PSD e route Omics. Eseguita anche la versione dello script
precedente alla bonifica: stesso esito e identico insieme di rilievi, nessuno nuovo.
Non è un collaudo del sito e non è stato indebolito per nascondere i risultati.
- Nessuna nuova certificazione applicativa, modifica del runtime o chiusura di
collaudi reali. I gate elencati nello snapshot restano espliciti.
## Pubblicazione effettiva completata — 15 settembre 2026
- Sorgente: `4ff91e8d6e771545bc3d8e7a0ac0b027ef66ba14`, pubblicata su `main`.
- [Pipeline 124](https://git.tylconsulting.it/mptyl/ThothII/actions/runs/124):
completata con successo, inclusi i nuovi controlli auth/DWH e pubblicazione `pages`.
- Release servita: `/srv/thothii-docs/releases/20260915T123738Z-4ff91e8d`.
Il symlink `current` punta a questa release; ricreato soltanto `thothii-docs`.
- Configurazione nginx copiata identica; container healthy. Confronto degli ID
dei container prima/dopo: nessun altro container ricreato o fermato.
- Verifica anonima HTTP: home, ricerca e guide manuali IT/EN rispondono 200.
SHA-256 di home e indice remoto identici alla build locale; ricerca: 20 pagine,
nessuna interna.
- Campioni interni `architecture/overview/`, `plans/2026-09-08-memory-management/`
e `operations/compose-reference/`: HTTP 404.
- Release precedente `releases/20260826T090022Z-e910c7d` conservata per rollback;
nessuna vecchia release cancellata. Pubblicazioni future richiedono il passaggio
esplicito del runbook, non il solo push di `main`.
Il precedente blocco sulla copia statica obsoleta è quindi risolto.
+48
View File
@@ -0,0 +1,48 @@
# Compose reference for maintainers
This is an internal topology reference, not a second fresh-installation recipe. Operators
start with the [Italian](../install/standalone-manual-it.md) or
[English](../install/standalone-manual-en.md) manual. It replaces the duplicated four-context
Docker guide without changing the runtime.
The base topology contains frontend, core, catalog-db, qdrant, embedding, embedding-model-init,
catalog-migrate and profile-gated workspace-maintenance. Pi runs in core; DWH and generative
model endpoints remain installation settings. Reference preprocessing and Memory have distinct
lifecycles and collections. See [preprocessing](../contracts/workspace-preprocessing-cli.md).
## Configuration and migration boundaries
- Use one physical absolute installation descriptor path, its generated operator environment,
Compose project name, base/profile overlays, transport overlays and generated model overlay.
Mixing a generic `deploy/env/local.env` invocation with a native `tht` installation creates
a different stack; it is not an equivalent lifecycle command.
- `THT_INSTALLATION_CONFIG_SOURCE` identifies the protected host descriptor. The read-only
backend runtime mount is `/run/thothii-installation/thothii-installation.yaml`, exposed through
`THT_INSTALLATION_CONFIG_FILE`; the runtime path is not a host source path.
- Catalog runtime/migrator passwords and provider credentials are protected files. A provider's
`authentication.apiKeyEnv` names an allowed bundle entry; it is not a raw key. Private CAs
are separately mounted PEM files, not bundle values. See the repository's
`deploy/secrets/README.md` and the [model catalog guide](../general/pi-configuration.md).
- Generate projections after descriptor edits. Do not edit generated model/auth/frontend files.
Apply the installation's normal restart process when authored configuration changes.
- Explicitly start catalog-db and run catalog-migrate before application rollout on a fresh
database or after an approved schema update. That service runs Catalog and Memory migrations;
neither normal backend startup nor `tht start` implicitly performs them.
- Server deployments retain their reviewed session/auth/network/storage overlays. A writable
server Pi-state parent must be initialized with the regular targets expected by the read-only
nested mounts; use `scripts/prepare-server-pi-state.sh` with the installation's verified UID/GID.
## Deployment-specific authority
For the prepared generic server environment, the base/profile/session override
combination is `-f compose.yaml -f deploy/compose.server.yaml
-f deploy/compose.session-server.yaml.example`, with `--env-file` supplied before
the overrides. Add the reviewed transport/generated overlays for that installation;
this fragment alone is not a complete startup command.
The current [server handoff](server-codex-handoff.md) and
[legacy upgrade runbook](server-upgrade-gitea-workspace-v2.md) retain maintenance, backup and
rollback gates. Embedded/upstream identity is not standalone OIDC. Local/standalone setup
does not authorize replacing a running server stack, resetting volumes, or copying another
machine's descriptor. `scripts/run-stack.sh` remains a low-level path for an explicitly
prepared generic environment, not the public manual's default startup command.
+3 -3
View File
@@ -67,7 +67,7 @@ SSH uses a private key, optional key passphrase, mandatory `known_hosts`, and op
TLS CA/server name. REST prefers `POST /rpc/schema_snapshot`; when it is absent, the catalog may
use the same strict v1 snapshot through one read-only `POST /rpc/run_query`. An unavailable
capability, malformed snapshot, or connector error applies no catalog changes. See the
[schema snapshot contract](../contracts/catalog-schema-snapshot.md).
[schema snapshot contract](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/catalog-schema-snapshot.md).
## Synchronize authoritative schema metadata
@@ -176,6 +176,6 @@ atomic operation. It skips empty generated descriptions, reports aggregate copie
counts, and retains the generated text. Because this can replace reviewed descriptions, the
interface requires explicit confirmation before applying it.
The decisions behind this surface are [ADRs 0001–0011](../adr/0001-postgres-metadata-catalog.md)
The decisions behind this surface are [ADRs 0001–0011](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/adr/0001-postgres-metadata-catalog.md)
and the detailed acceptance record is
[AI catalog description generation acceptance](../testing/2026-08-29-ai-catalog-description-generation-acceptance.md).
[AI catalog description generation acceptance](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/testing/2026-08-29-ai-catalog-description-generation-acceptance.md).
@@ -0,0 +1,98 @@
# Publishing the public manual
This is an internal operator runbook. Publishing documentation must not recreate
the ThothII application, Gitea, proxy, DWH or other documentation containers.
## Three separate states
1. `main` contains reviewed source and the public allowlist in `mkdocs.yml`.
2. Gitea Actions builds and validates the generated `pages` branch. A successful
Actions run alone does **not** update the live site.
3. The live [manual](https://git.tylconsulting.it/thothii-docs/) is served by the
dedicated `thothii-docs` nginx container on the host reached through the existing
trusted SSH alias `contabo`. Its Compose project/service are `thothii-docs`/`docs`.
This procedure is manual. No timer, webhook, credential or automatic deployment
has been added. Use the operator's existing SSH authorization and known host key;
never disable host verification to make a publication work.
## Build and stage
From a clean ThothII `main` checkout already pushed to Gitea:
```sh
git status --short
git rev-parse HEAD
./scripts/build-docs.sh
bash scripts/auth-docs-smoke.sh
bash scripts/verify-dwh-auth-docs.sh
bash scripts/test-auth-docs-smoke.sh
bash scripts/test-verify-dwh-auth-docs.sh
```
The strict build also verifies the public boundary: exactly 20 navigation pages,
five approved source assets, no internal pages/assets, and matching search entries.
The locked Windmill theme requires a separate exact allowlist of 25 static assets;
MkDocs applies `exclude_docs` to those files too. Never replace the list with broad
directory exceptions. Sources under `docs/` may not shadow theme asset paths.
The verifier follows stylesheet/script/image references and CSS font references
from generated pages and fails when a local dependency is absent.
Choose a unique release identifier consisting of UTC timestamp and source SHA.
Record the current symlink and container identity before proceeding:
```sh
ssh -oBatchMode=yes -oStrictHostKeyChecking=yes contabo \
'readlink /srv/thothii-docs/current; docker inspect --format "{{.Id}} {{.State.Health.Status}}" thothii-docs'
```
On the server, create `/srv/thothii-docs/releases/RELEASE/site` only if RELEASE
does not exist. Copy the current `nginx.conf` unchanged into the new release.
Transfer the local built `site/` into that new empty directory with rsync over
the same verified SSH connection. Do not use `--delete` against `current`, and do
not edit routing, TLS, network or authentication configuration.
## Activate only this site
Replace RELEASE below with the validated identifier, not a user-supplied path.
Keep the previously recorded release for rollback.
```sh
cd /srv/thothii-docs
test -f releases/RELEASE/site/index.html
test -f releases/RELEASE/nginx.conf
docker exec thothii-docs nginx -t
ln -s releases/RELEASE current.next
mv -Tf current.next current
docker compose --project-name thothii-docs -f docker-compose.yml \
up -d --no-deps --force-recreate docs
```
Check that `current.next` does not already exist before creating it; stop if it
does, because another publication or recovery may be in progress. Changing the
symlink alone is insufficient: an existing Docker bind mount still resolves to
the old release. The service recreation above remounts the new directory. It may
briefly interrupt this manual only.
## Verify, record, or roll back
- Wait for `docker inspect` to report this container healthy; verify mounted
paths and nginx configuration. Stop after a bounded timeout (for example 60 s).
- Without login/cookies, require HTTP 200 for home, search and both
`install/standalone-manual-it/` and `install/standalone-manual-en/`.
- Compare live home and `search/search_index.json` checksums to the build. Search
must contain only the 20 approved pages, never plans, reports or architecture.
- Check the actual stylesheet and JavaScript URLs in both installation pages:
HTTP 200, CSS served as `text/css`, JavaScript with a valid script content type,
and fonts available. In a browser confirm the stylesheets load and the layout is
styled. HTML 200 alone is not enough to accept a publication.
- Require HTTP 404 for retired/internal paths, including `architecture/overview/`,
`plans/2026-09-08-memory-management/`, and `operations/compose-reference/`.
- Record source SHA, release ID, previous release and checks in the cleanup/release
record. Do not call the publication complete solely because `pages` was pushed.
If health or public verification fails, point `current` back to the recorded
previous release using a fresh temporary symlink and the same atomic rename, then
recreate **only** service `docs` with the same Compose command. Verify health and
old site availability. Do not delete either release during recovery. Historical
releases are retained; pruning them requires a separate retention decision.
+6 -11
View File
@@ -118,12 +118,12 @@ other CPU workloads.
Run the first evaluation in shadow mode: read the source with its existing read-only role, do not
save proposed flags, and report only aggregate counts, rule IDs, coverage, and timings. Never copy
matched values into test output. Use a separately approved, labeled Italian corpus to calculate
precision and recall; raw PSD values must remain inside the authorized environment.
precision and recall; source values must remain inside the authorized environment.
Inside the configured core runtime, the non-mutating command is:
```bash
npm run sensitivity:shadow -- psd-clinical
npm run sensitivity:shadow -- <workspace-id>
```
It reads catalog metadata and source values but emits one aggregate JSON object with no database,
@@ -139,12 +139,7 @@ Enabling NER by default requires all of these gates:
If a gate fails, leave NER disabled. The deterministic policy remains available and produces the
binary draft from its scan coverage; no content is sent to an internal or external LLM.
The first aggregate PSD shadow comparison is recorded in
[`2026-09-02-psd-sensitivity-shadow.md`](../reports/2026-09-02-psd-sensitivity-shadow.md). On the
local CPU runner, NER found additional entities but reduced total coverage under the superseded
global deadline. The v2 benchmark removed that confounder: CPU NER added 18 sensitive proposals and
increased the warm analysis time from 50.1 to 61.3 seconds. It remains opt-in until a labeled Italian
evaluation establishes that the additional findings justify their false-positive rate and cost.
The deterministic progressive PSD run is recorded in
[`2026-09-03-psd-progressive-sensitivity-shadow.md`](../reports/2026-09-03-psd-progressive-sensitivity-shadow.md):
both deterministic and CPU-NER profiles assessed all 2,275 columns with zero `unknown` decisions.
Benchmarks from a particular installation are not a guarantee for another database or machine.
NER remains opt-in until an approved evaluation establishes that additional findings justify
their false-positive rate and operational cost. Keep benchmark and release records with the
installation's technical evidence, separate from this operator procedure.
+378
View File
@@ -0,0 +1,378 @@
# Consegna a Codex sul server: ThothII e Omics Portal
Revisione: **14 settembre 2026**. Destinazione: Datamart Builder nel portale
Omics esistente, non un nuovo sito standalone. Questo documento è la procedura
di riferimento per questa consegna e sostituisce le precedenti istruzioni di
trasporto/pubblicazione del codice Omics. La distribuzione parte dai sorgenti
ThothII aggiornati su `main` e dal branch Omics disponibile su GitHub; nessuna
replica del repository Omics ad altri servizi fa parte dell'intervento.
## Risultato da ottenere e limiti
- L'utente entra in Omics come oggi, sceglie **Datamart Builder** e trova ThothII
già autenticato, senza un secondo login.
- Omics mantiene header, navigazione sinistra, lingua, tema, fullscreen, nome
utente e logout. ThothII occupa soltanto la zona centrale: **embedded/upstream**.
- I dati e le identità esistenti, i workspace, i modelli approvati e le credenziali
del server restano quelli del server. Il Mac rimane **full/local**, default EN.
- L'intervento comprende il codice e la configurazione di entrambi gli
applicativi, la rigenerazione delle proiezioni, le immagini e il collaudo.
Un pull da solo non conclude l'installazione.
Codex può fare l'inventario, preparare modifiche e test isolati. Prima del fermo,
delle migrazioni, della modifica del proxy o della ricreazione di servizi
operativi, presenta i comandi risolti, backup e rollback e ottieni conferma della
finestra di rilascio. Ferma il passaggio interessato se manca una credenziale,
una decisione sulla migrazione o un prerequisito; non aggirare i controlli.
Non modificare Authentik o il DWH per correggere la UI. Il DWH resta read-only.
Non cancellare volumi, dati o modifiche locali e non stampare segreti nei report.
## 1. Identificare l'installazione realmente in uso
Leggi `AGENTS.md` e `PROJECT_STATE.md` nel checkout ThothII aggiornato. Individua
il checkout operativo Omics, normalmente `/home/chirone/omics_portal`; conferma
il percorso prima di usarlo. Registra per **entrambi** i progetti:
1. Percorso, branch, SHA, stato della working tree e revisioni delle immagini
effettivamente in esecuzione. Il checkout appena aggiornato può non coincidere
con quello da cui sono stati creati i container.
2. Nomi progetto Compose, file Compose/override ordinati, env file, servizi,
mount, porte e reti. Leggi le label Compose dei container per ricostruire
l'avvio; filtra gli inspect, evitando dump di variabili segrete.
3. Percorso assoluto del `thothii-installation.yaml`, suo schema/profile,
`projectDirectory`, `envFile`, `overrides`, `authentication`, `shell` e modello
dei dati persistenti. Usa solo percorsi Linux reali e file che esistono.
4. Configurazione effettiva del core: modalità auth, `THOTH_PUBLIC_EXPOSURE`,
`THT_SESSION_STORAGE`, binding workspace/database, percorsi degli archivi
Evidence e Memory e delle credenziali Pi/provider.
5. Origine HTTPS pubblica del portale, punto di terminazione TLS, percorso
autenticato delle API, alias di rete e possibilità di accesso diretto al core.
**Completato quando:** esiste un inventario senza segreti e ogni comando di
avvio è ricostruibile con percorsi/progetti effettivi. Non usare gli script
temporanei `/private/tmp/…` o i percorsi `/Users/mp/…` del Mac. Gli override vanno
conservati in un percorso operativo stabile sul server.
## 2. Verificare le revisioni dei due applicativi
### ThothII
Il checkout aggiornato deve essere su `main` e includere almeno
`bdcd8fcd28f3011471d77224db9c3f5baf227995` e questo documento. Registra anche lo
SHA effettivo di `main`, che include il commit di consegna e il merge successivi:
```bash
git status --short --branch
git rev-parse HEAD
git merge-base --is-ancestor bdcd8fcd28f3011471d77224db9c3f5baf227995 HEAD
```
Se il controllo fallisce, completa l'acquisizione della revisione approvata
prima di toccare l'installazione. Non ricostruire a mano le singole modifiche UI:
questa revisione contiene shell, autenticazione, i18n, workflow bilingue,
amministrazione, typography e navigazione aggiornate.
### Omics Portal
Il pull di ThothII **non aggiorna Omics**. Consegna Omics verificata su GitHub:
- Repository: `https://github.com/Dallavilla-Tiziano/omics_portal.git`.
- Branch: `codex/thothii-embedded-shell`.
- SHA della consegna: `fca10901a73666ca257d8f4cc4b77066295c400a`.
- Commit funzionale della shell: `95154e179144e2453b37ef2a63a65d6f377e4cf8`.
Nel checkout Omics confermato, acquisisci senza fare un pull/merge implicito:
```bash
cd /home/chirone/omics_portal
git status --short --branch
git rev-parse HEAD
git fetch --no-tags https://github.com/Dallavilla-Tiziano/omics_portal.git \
refs/heads/codex/thothii-embedded-shell:refs/remotes/thothii-delivery/omics-shell
git rev-parse refs/remotes/thothii-delivery/omics-shell
git merge-base --is-ancestor 95154e179144e2453b37ef2a63a65d6f377e4cf8 \
refs/remotes/thothii-delivery/omics-shell
git diff --stat HEAD...refs/remotes/thothii-delivery/omics-shell
```
Confronta lo SHA acquisito con quello sopra. In caso di consegna diversa chiedi
quale revisione usare. Confronta inoltre le modifiche con i progressi del server:
non sostituire l'intero portale con un checkout più vecchio. Se la consegna è già
integrata verifica i file, senza ripetere il merge; altrimenti prepara la sua
integrazione in un branch/worktree di revisione dal codice operativo. Risolvi
eventuali conflitti preservando i cambiamenti del server, testa, quindi applica
la revisione concordata nel rilascio. Non fare reset o force push.
I dettagli tecnici locali in `docs/thothii-integration.md` di Omics sono utili,
ma il percorso operativo di questa consegna è quello di **questo documento**.
La pubblicazione del codice Omics su altri remote non è un prerequisito.
**Completato quando:** una revisione integrata Omics conserva le funzionalità del
server e soddisfa tutti i controlli dei file nella sezione 4.
## 3. Adeguare ThothII senza importare la configurazione del Mac
### CLI, descrittore e proiezioni
Aggiorna il **CLI nativo host** dal checkout ThothII approvato, conservando il
vecchio binario per rollback. Non confonderlo con il CLI Python interno al core:
```bash
./scripts/install-tht.sh
command -v tht
tht --help
```
Il comando installa normalmente in `/usr/local/bin` e verifica la risoluzione
su PATH. Se il server usa un'altra directory, mantieni quella usando
`THT_INSTALL_DIRECTORY` con un percorso assoluto. Conserva proprietario e
permessi protetti del descrittore (`0600` o `0400`). Nel descrittore esistente
schema v2 modifica la sezione seguente, preservando gli altri valori:
```yaml
shell:
mode: embedded
defaultLocale: en
adapter: omics-portal
```
`en` è il fallback UI, non forza l'inglese sul portale: in embedded prevale la
lingua renderizzata da Django. Mantieni il `profile` server approvato e i percorsi,
la project/installation identity, lo storage e gli override del server. Se il
descrittore manca o è legacy, prepara una migrazione separata dopo l'inventario;
non rilanciare il setup locale e non copiare il descrittore Mac.
Usa una variabile di lavoro dedicata, valorizzata con il percorso **confermato**:
```bash
THTII_INSTALLATION=/percorso/reale/thothii-installation.yaml
tht --installation "$THTII_INSTALLATION" installation generate
```
La generazione non avvia i servizi. Controlla `generated/frontend/config.js` e
il suo mount read-only nel Compose generato; prima dell'override Omics deve
contenere `backendBaseUrl: "/api"` e la shell embedded completa. Conserva anche
le proiezioni generate di modelli/settings/Pi. Non mantenere copie manuali dei
file generati: `thothii-installation.yaml` resta la sorgente authored.
### Autenticazione già fornita dal portale
Nell'override Compose persistente dell'installazione deve esserci:
```yaml
services:
core:
environment:
AUTH_MODE: upstream
```
Aggiungi il percorso dell'override all'elenco `overrides` del descrittore se non
è già caricato. Controlla il Compose risolto: scrivere `AUTH_MODE` nel solo file
env non garantisce che la variabile arrivi al container.
- `authentication.configDirectory` e `THT_AUTH_CONFIG_ROOT` devono indicare la
directory protetta prevista, **senza un `auth.yaml` local/OIDC letto dal core**.
Non cancellare un file esistente: se trovato, fermati e prepara il cambio auth
con backup e una directory dedicata. `AUTH_MODE` insieme al file è rifiutato.
- Per Omics non usare `authentication.runtimeProjection` né
`THT_AUTH_RUNTIME_PROJECTION_ROOT`: appartengono all'accesso local/OIDC diretto.
- Non creare utenti/password ThothII, nuovi client OIDC o callback per questo
embedding. Non esiste `tht auth configure --mode upstream`.
- Preserva la coppia stabile `(issuer, subject)` degli utenti (`portal`, ID
Django); cambiarla può rendere invisibili le sessioni dei proprietari esistenti.
### Dati, storage e prerequisiti non grafici
La release corrente usa catalogo metadati interno PostgreSQL, workspace schema
v4, Installation Model Catalog v2, Qdrant e Ollama per embedding; Pi gira nel
core. Mantieni i provider e i binding reali del server, incluse credenziali e CA.
I default del Mac non sono una richiesta di cambiare modello o database.
Il catalogo metadati e l'eventuale database delle sessioni sono **due funzioni
distinte**. Con `THOTH_PUBLIC_EXPOSURE=true`, il core rifiuta
`THT_SESSION_STORAGE=local`: verifica che il percorso PostgreSQL delle sessioni
sia già configurato e validato. Se manca, presenta il piano di provisioning e
migrazione; non disabilitare il controllo public-exposure per ottenere l'avvio.
L'overlay session-server è opt-in e non migra automaticamente i vecchi archivi.
Se la versione operativa precede questi contratti, risolvi prima la migrazione
dei dati con backup verificati. Le migrazioni del catalogo si eseguono tramite
il job esplicito `catalog-migrate`; quelle delle sessioni, quando necessarie e
approvate, tramite `session-migrate`. Nessuna riguarda il DWH o è sostituita da
una sincronizzazione di schema dall'interfaccia.
**Completato quando:** il descrittore genera correttamente; la configurazione
risolta contiene shell embedded, upstream senza doppia auth, storage compatibile,
mount e reti corretti; ogni differenza infrastrutturale ha un piano approvato.
## 4. Verificare e integrare i file Omics
| File nel repository Omics | Risultato obbligatorio |
| --- | --- |
| `templates/kokoro/datamart_builder.html` | Mount `#root` nello stesso documento Django, senza iframe; altezza contenuta sotto la topbar e layout centrale responsive. Carica config, override limitato e asset in quest'ordine. |
| `templates/base.html` | `{% get_current_language as CURRENT_LANGUAGE %}` e `<html lang="{{ CURRENT_LANGUAGE }}">`, preservando i block del template. |
| `templates/partials/topbar.html` | `select.omics-language-select[data-lang]` con lingua Django, form `set_language` POST/CSRF/next; pulsante fullscreen con label ingresso/uscita e stato accessibile. Mantieni nome/logout Omics. |
| `static/js/app.js` | Fullscreen reale del documento con `requestFullscreen`/`exitFullscreen`; ascolta gli eventi del browser, inclusa uscita con Esc, aggiorna icona/stato/label e gestisce rifiuti senza simulare successo. |
| `locale/it/LC_MESSAGES/django.po` | Traduzioni dei nuovi controlli fullscreen; compilazione del catalogo distribuito. |
| `kokoro/datamart_catalog_views.py` e routing | La pagina richiede `datamart_builder.access`; l'endpoint `/datamart-builder/api-auth` verifica la sessione Django e restituisce identità verificata o 403. Conserva questa parte già esistente. |
| `nginx/nginx.conf` | API protette verso il core, asset/config verso il frontend, origine e SSE coerenti. Conserva anche le altre route del portale. |
| `kokoro/templatetags/vite.py` | Manifest da `http://thothii-frontend:8080/.vite/manifest.json`, asset dal manifest, cache di 30 secondi. |
| `kokoro/test_thothii_shell.py`, `test_support/thothii/` | Test isolati della pagina e dei controlli, da eseguire prima del rilascio. |
Il template deve fare questo **prima** di `{% vite_assets %}`:
```html
<script src="/datamart-builder/config.js"></script>
<script>
window.__THOTHII_CONFIG__ = Object.assign({}, window.__THOTHII_CONFIG__ || {}, {
backendBaseUrl: '/datamart-builder/api'
});
</script>
```
L'override non deve sostituire l'intero oggetto perdendo `shell`. Nel browser il
risultato deve avere `/datamart-builder/api` e `shell.mode === "embedded"`.
`config.js` deve avere `Cache-Control: no-store`; gli asset con hash possono
avere cache lunga. Non codificare a mano i nomi dei bundle Vite.
Il tema deve essere espresso come `data-bs-theme="light"` o `"dark"` su `html`.
`OmicsPortalAdapter` in ThothII osserva quel dato, legge il `data-lang` renderizzato
dal selettore e lo stato fullscreen. Non richiede nuovi eventi, handshake o
token JavaScript. Se il contesto manca va corretto il template, non aggirato
l'errore con un header full. I dettagli del portale restano nel solo adapter.
### Proxy e identità: controllo obbligatorio
Il flusso è browser → Nginx Omics → controllo Django → core ThothII:
1. `/datamart-builder/api/…` usa `auth_request /_thothii_auth`.
2. La location interna interroga `/datamart-builder/api-auth` usando il cookie
Omics. Django risponde 200 se autorizzato, 403 senza sessione/capability.
3. Nginx usa solo gli header **della risposta Django** e sovrascrive gli eventuali
valori client: `X-Thoth-Principal-Issuer: portal`,
`X-Thoth-Principal-Subject: <user.pk>`, `X-Thoth-Principal-Display-Name` e
`X-Thoth-Is-Admin: true|false` secondo `is_authentik_admin(user)`.
4. Il proxy rimuove `Cookie`, `Authorization` e `X-Authenticated-User` prima del
core, toglie il prefisso API e inoltra a `thothii-core:8787`. Non usare qui
gli header `X-Thoth-Trusted-*` dell'esempio generico a due hop.
5. Config/asset e manifest arrivano da `thothii-frontend:8080`. Omics web deve
raggiungere il manifest; Nginx deve raggiungere entrambi gli alias privati.
Integra i servizi nella rete effettiva del portale, con alias non ambigui;
non collegare due core candidati con lo stesso alias. Il core upstream deve
essere irraggiungibile direttamente da browser/client non fidati, inclusi
percorsi alternativi attraverso un frontend o proxy non protetto.
Conserva buffering/cache disattivati e timeout lunghi per SSE anche nel proxy
a monte. Verifica l'Origin delle scritture: il codice Omics contiene la mappa
esatta da `https://aritmolab.policlinicosandonato.it` a
`http://aritmolab.policlinicosandonato.it` per la terminazione TLS esterna.
Conferma che la topologia sia ancora quella. Se differisce, correggi Host,
protocollo e mappa esatta con un test di rifiuto cross-origin; non cancellare
Origin, non usare wildcard né rendere fidati gli header forniti dal browser.
Riferimento per errori 401/403 e contratto completo:
[autenticazione upstream](../install/authentication-upstream.md).
## 5. Test, backup e rilascio coordinato
Dal checkout Omics integrato, senza database operativo o volumi collegati:
```bash
docker build -f test_support/thothii/Dockerfile -t omics-portal:thothii-shell-tests .
docker run --rm --network none omics-portal:thothii-shell-tests
```
In ThothII verifica la build di frontend/core, i test auth e shell e la
generazione del descrittore con il CLI aggiornato. I test locali alla consegna
includono 768 test frontend, 20 scenari browser e build documentale strict;
non certificano il portale/IdP né i dati del server.
Prima del rilascio prepara un piano con i **comandi esatti risolti**. Per
installazioni già governate dal CLI usa `tht --installation …`; per launcher
server personalizzati conserva progetto, ordine di tutti gli override e bind.
Non alternare i due lifecycle se cambiano la project identity o i volumi.
Ordine da applicare nella finestra confermata:
1. Metti al sicuro revisioni, binario host, descrittore/env/override, immagini e
backup consistenti di catalogo, sessioni, registry/Evidence/Memory, settings,
Pi e indici. Proteggi i backup che contengono segreti. Verifica il ripristino
prima di una migrazione non reversibile e gestisci le sessioni in corso.
2. Genera le proiezioni dell'installazione; costruisci **core e frontend** dai
sorgenti approvati. Avvia i servizi di supporto necessari e, se richiesto dal
salto di versione, esegui le migrazioni esplicite con exit 0 prima del core.
Il normale `tht start --build` coordina il lifecycle, non è frontend-only.
3. Ricrea i servizi applicativi ThothII con configurazione embedded/upstream e
conserva il progetto/dati approvati. Controlla health e diagnostica:
```bash
tht --installation "$THTII_INSTALLATION" status
tht --installation "$THTII_INSTALLATION" doctor --json
```
4. Distribuisci la revisione Omics integrata tramite il suo normale rilascio,
includendo `web`, statici/cataloghi e configurazione `nginx`. Il suo entrypoint
esegue `migrate`, `compilemessages` e `collectstatic`: le modifiche della shell
non aggiungono migrazioni Django, ma controlla quelle pendenti del server prima
del riavvio. Verifica anche il catalogo Superset richiesto dalla build Omics.
5. Esegui `nginx -t` nel servizio candidato e applica il reload/riavvio secondo
la topologia registrata. Dopo la ricreazione dei container verifica che il
proxy risolva gli alias ai nuovi indirizzi, non a IP Docker precedenti.
6. Attendi almeno 30 secondi per la cache manifest Django o invalidala con il
meccanismo del portale. Verifica asset/config e svolgi il collaudo seguente.
Se uno step fallisce non marcare l'installazione conclusa. Un core healthy non
prova che auth, UI embedded o scritture attraverso il proxy funzionino.
## 6. Accettazione prima di dichiarare completato
Usa account di prova autorizzati e dati non operativi per i test che scrivono.
Le verifiche che richiedono login interattivo possono essere svolte dall'operatore:
riporta esplicitamente quelle ancora da fare, senza spuntarle per deduzione.
- **Accesso:** login Omics, apertura da menu, nessun login/header ThothII. `/me`
su `/datamart-builder/api/me` restituisce identità e permessi corretti;
`session`/`csrfToken` sono null in upstream.
- **Dinieghi:** senza sessione o capability il proxy nega; header principal
falsificati non danno accesso. Utente normale senza controlli admin; admin
autorizzato con controlli coerenti. Nessuna route diretta aggira il proxy.
- **Lingua e continuità:** IT/EN prima e dopo l'apertura, cambio attraverso Omics,
ripristino della selezione dopo reload senza generazione automatica. Nuove
sessioni ricevono la lingua UI; sessioni riprese mantengono
`interaction_language`. Per quelle legacy la prima ripresa fissa la lingua
del workspace in modo idempotente. SQL e contenuti authored non sono tradotti.
- **Tema/fullscreen:** light/dark cambia anche ThothII, incluse finestre e menu;
fullscreen nasconde il bordo browser, sostituisce l'icona e torna normale con
Esc. Header/sidebar Omics mantengono il proprio aspetto.
- **Logout:** il logout è soltanto quello Omics. Ritorno alla pagina e
riconnessione ricontrollano l'accesso; non promettere revoca istantanea di uno
stream già aperto in un'altra scheda.
- **Workflow:** una sessione di prova autorizzata può essere creata, ricevere
eventi SSE e domande/scelte nella lingua corretta, salvare e riprendere senza
perdere proprietà. Le scritture same-origin funzionano, quelle cross-origin
non autorizzate vengono negate.
- **Amministrazione/UI:** Database, Memory ed Evidence leggibili, font/layout
aggiornati e nessuna propagazione del reset CSS alla topbar Omics. Le memory
FAKE sono solo esempi UI isolati, non da importare nel catalogo o nel recall.
Puntino readiness Workspace, unico bottone Sessione, tab con bordi uniformi;
accordion inizialmente chiuso, un solo pannello aperto, selezione per lista,
scroll interno e nessuna frase “Inizia con Sessione”.
- **Operatività:** nessun errore di config/auth nei log, mount e permessi corretti,
indici/cataloghi e dati precedenti disponibili, nessuna modifica al DWH/IdP.
Compila il report con SHA ThothII/Omics, immagini, percorsi configurazione,
comandi eseguiti, risultati e prove manuali pendenti, senza cookie o token.
La [matrice auth completa](../testing/authentication-manual-acceptance.md)
approfondisce i casi di sicurezza.
## 7. Rollback
Ripristina la coppia compatibile di codice/immagini **Omics e ThothII**, il CLI,
descrittore e proiezioni registrati, seguendo il lifecycle approvato. Riavvia il
proxy se necessario per DNS/config e ricontrolla manifest, accesso e SSE.
I dati restano preservati: nessun `down --volumes`, cancellazione di archivi o
reset distruttivo. Se il rilascio ha migrato uno schema o scritto dati non
compatibili con la versione precedente, usa il piano di ripristino dati approvato,
non un semplice downgrade d'immagine. Il rollback termina solo dopo il collaudo
della versione ripristinata.
@@ -4,6 +4,22 @@ Questo runbook è il passaggio di consegne per il Codex che opererà sul server
installazione precedente alla configurazione corrente di ThothII senza modificare Authentik o il
DWH esterno e senza cancellare lo stack precedente durante il primo cutover.
## Scelta preliminare: server autonomo oppure Omics
I passaggi di questo runbook che configurano OIDC diretto, gruppi e
`authentication.runtimeProjection` riguardano **ThothII autonomo**, da rendere
con `shell.mode: full`. Non applicarli all'integrazione Datamart Builder: Omics
usa **embedded/upstream** e mantiene il proprio accesso Authentik. Il core riceve
l'identità verificata dal proxy senza un secondo login né un secondo auth.yaml.
Prima dell'inventario identificare quale percorso è approvato. Per Omics seguire
[autenticazione upstream](../install/authentication-upstream.md) e
[rilascio coordinato dei due repository](shell-and-localization.md#preparare-il-rilascio-coordinato);
i gate di backup, isolamento, catalogo, storage e rollback di questo runbook
rimangono validi, ma non copiare i passi auth del percorso autonomo. Il passaggio
da issuer `portal` a un issuer OIDC differente non trasferisce automaticamente
la proprietà delle sessioni.
La procedura si applica a `main` quando contiene almeno il commit
`eba6148511675fc6a187aabb69a975adc5e3c542`. Deve essere presente anche questo file. Il commit
minimo è un controllo di sicurezza, non un invito a fermarsi a quella revisione: installare sempre
@@ -11,6 +27,9 @@ la `origin/main` approvata dall'operatore.
## Regole non negoziabili
- Per il rilascio Omics seguire la [consegna corrente a Codex sul server](server-codex-handoff.md).
Acquisire il branch dedicato da GitHub, verificare SHA e integrarlo con i
progressi del server; nessuna replica del repository è richiesta.
- Eseguire prima l'intero inventario in sola lettura e consegnarlo all'operatore.
- Non stampare mai password, token, chiavi private, cookie, file `.env` o contenuti dei Docker
secret. Nei report sono ammessi solo percorsi, nomi delle variabili e valori non segreti.
@@ -51,7 +70,9 @@ La topologia base attesa è:
| `embedding-model-init` | scarica/verifica il modello | job one-shot, deve terminare con exit 0 |
`catalog-db` non è il DWH. Non pubblica porte sull'host e usa credenziali runtime e migrator
separate. Ollama non controlla le password utente: l'autenticazione resta OIDC tramite Authentik.
separate. Ollama non controlla le password utente: nel percorso autonomo vale
OIDC tramite Authentik; nel percorso embedded Omics verifica l'accesso e il core
usa upstream.
Il PostgreSQL per le **sessioni** è un'altra funzione ancora. L'overlay
`deploy/compose.session-server.yaml.example` è esplicitamente opt-in: non abilitarlo durante questo
@@ -701,6 +722,6 @@ la nuova installazione è ferma ma ispezionabile.
- [OIDC generico](../install/authentication-oidc.md)
- [Operazioni workspace](workspaces.md)
- [Configurazione dei modelli Pi](../general/pi-configuration.md)
- [Contesti Docker](../installazione-docker-4-contesti.md)
- [Contesti Docker](compose-reference.md)
- [Database Management](database-management.md)
- [Analisi locale della sensibilità](sensitivity-analysis.md)
+322
View File
@@ -0,0 +1,322 @@
# Shell, autenticazione e lingue
Questa è la procedura operativa del rendering corrente. Leggerla insieme a
[architettura full/embedded](../architecture/application-shell.md),
[autenticazione upstream](../install/authentication-upstream.md) e
[contratto PortalAdapter](../contracts/portal-shell-adapter-v1.md).
La [specifica approvata](../plans/2026-09-13-full-shell-spec.md) documenta la
progettazione, non sostituisce i vincoli verificati nel codice e riportati qui.
## Scegliere il contenitore
La modalità della shell è indipendente dal profilo di distribuzione e dal metodo
di autenticazione. Un server può ospitare full; un ambiente locale può ospitare
embedded per provare un'integrazione.
| Destinazione | Shell | Autorità di accesso | Login/logout visibile |
| --- | --- | --- | --- |
| Mac attuale | full, default en | `auth.yaml` local | ThothII |
| Server autonomo | full | `auth.yaml` OIDC | ThothII, con redirect al provider |
| Datamart Builder in Omics | embedded, adapter Omics | core `AUTH_MODE=upstream`, sessione Omics al proxy | Solo Omics |
Non confondere la lingua inglese iniziale del Mac con quella del workspace o
delle sessioni già create. Non copiare sul server l'intero descrittore del Mac:
contiene percorsi e scelte locali, oltre a full.
Per questo Mac, nel descrittore installato:
```yaml
shell:
mode: full
defaultLocale: en
```
Per il server Omics:
```yaml
shell:
mode: embedded
defaultLocale: en
adapter: omics-portal
```
L'assenza di `shell` conserva embedded con adapter Omics. Un nome adapter
sconosciuto è un errore, non una richiesta di fallback a full. Full ignora
l'adapter Omics riconosciuto e non lo istanzia. Un locale ben formato per cui non
esiste ancora un catalogo usa l'inglese nell'interfaccia.
`defaultLocale` è il valore iniziale, non un vincolo che annulla ogni scelta
dell'utente. Full ricorda lingua e tema nel browser; embedded segue soltanto
Omics. Le preferenze non modificano il descrittore installato.
## Applicare una modifica all'installazione
Per una nuova installazione autonoma, selezionare esplicitamente full:
```bash
tht setup --complete --profile local --shell-mode full --shell-default-locale en
```
Il setup senza opzioni shell conserva per compatibilità il default embedded.
Per un'installazione esistente non rilanciare setup per sovrascrivere il
descrittore: registrare la configurazione attuale, modificarne la sezione shell
e usare la generazione seguente. Le credenziali rimangono nei file protetti.
Aggiornare prima il binario nativo `tht`: le versioni precedenti rifiutano la
sezione `shell`. Modificare poi il descrittore e generare le proiezioni:
```bash
tht --installation /percorso/assoluto/thothii-installation.yaml installation generate
```
Il comando non avvia né arresta servizi. Genera anche
`generated/frontend/config.js`, che contiene configurazione pubblica, e il
relativo mount Compose. Non modificare a mano i file generati. `tht start`
rigenera le proiezioni nel normale percorso di avvio.
Applicare il normale processo di aggiornamento dei container dell'installazione.
Se si usa un launcher Compose personalizzato, deve includere la proiezione
Compose generata e ricreare il frontend quando cambia la configurazione. Il
fingerprint della configurazione pubblica permette a Compose di rilevare il cambio.
La stessa immagine frontend supporta entrambe le modalità.
Nel percorso standard del CLI, dopo avere approvato l'aggiornamento:
```bash
tht --installation /percorso/assoluto/thothii-installation.yaml start
tht --installation /percorso/assoluto/thothii-installation.yaml status
tht --installation /percorso/assoluto/thothii-installation.yaml doctor --json
```
Usare `start --build` per una revisione di codice che richiede nuove immagini,
non per la sola modifica della shell. Questo è un lifecycle dell'installazione,
non un comando garantito frontend-only. Un launcher personalizzato deve conservare
tutti gli override di rete, autenticazione, workspace e modelli già approvati.
Non usare `down --volumes`. Registrare gli identificatori delle immagini prima
dell'aggiornamento e conservare il descrittore precedente per il rollback.
Controllare il `config.js` effettivamente servito, che non deve essere memorizzato
in cache. In full la route API ordinaria è `/api`; Omics imposta nel template il
prefisso same-origin `/datamart-builder/api`, mantenendo le altre impostazioni.
Il file pubblico standalone deve essere equivalente a:
```javascript
window.__THOTHII_CONFIG__ = {
backendBaseUrl: "/api",
shell: { mode: "full", defaultLocale: "en" }
};
```
È un risultato da controllare, non un file da mantenere a mano. In Omics il
template carica `/datamart-builder/config.js`, conserva l'oggetto con
`Object.assign` cambiando solo `backendBaseUrl` in `/datamart-builder/api`, poi
carica gli asset dal manifest. Il config senza cache deve precedere ogni modulo
React; verificare nella rete del browser l'URL finale `/datamart-builder/api/me`.
## Accesso in parole semplici
In Omics l'utente effettua l'accesso al portale come oggi. Quando sceglie
Datamart Builder, il server controlla che possa usarlo e comunica a ThothII chi è.
ThothII apre l'applicazione per quella persona: non presenta un altro login e non
crea una seconda sessione browser. Le password non vengono trasmesse a ThothII.
Il nome e il comando Esci rimangono nell'header del portale.
Sul Mac full, ThothII presenta il proprio login locale. Dopo l'accesso mostra il
nome nell'header; il menu del nome contiene il logout. Riutilizza gli utenti e
la configurazione di accesso dell'installazione. Full supporta anche un'eventuale
autenticazione OIDC configurata; il suo logout termina la sessione ThothII, non
promette di disconnettere l'utente da tutti gli altri servizi OIDC.
Full/upstream non può terminare una sessione posseduta dal proxy e non mostra
quel comando logout. Embedded non presenta mai il login ThothII, neppure per
recuperare un errore di configurazione. Per un server autonomo con login/logout
ThothII usare full/OIDC, non full/upstream.
I controlli server restano autorevoli. Un errore su una singola operazione non
deve cancellare automaticamente l'accesso all'intera applicazione. Un rifiuto
della verifica dell'utente chiude invece lo stato protetto. L'accesso viene
ricontrollato anche al ritorno alla pagina e quando il collegamento eventi deve
riconnettersi. Questo non equivale a una revoca istantanea di ogni connessione
già aperta in altre schede.
## Contratto server Omics
Il percorso corrente usa il controllo Django della capability
`datamart_builder.access`, la subrequest nginx `auth_request` e gli header
normalizzati `X-Thoth-Principal-Issuer`, `X-Thoth-Principal-Subject`,
`X-Thoth-Principal-Display-Name` e `X-Thoth-Is-Admin`. Il browser non può
scegliere queste identità: il proxy ricava gli header dal controllo server e
sostituisce quelli eventualmente forniti dal client.
Mantenere il backend configurato per l'autenticazione upstream e i suoi controlli
di autorizzazione. Non esporre un percorso alternativo che permetta al browser
di raggiungerlo aggirando quel controllo. Non introdurre token nel documento,
negli eventi UI o nella configurazione pubblica.
La [guida upstream](../install/authentication-upstream.md) riporta i vincoli
esatti: `AUTH_MODE=upstream` nel core, nessun `auth.yaml` o runtime projection
contemporaneo, capability Django, quattro header obbligatori/facoltativi, percorso
diretto Omics distinto dal proxy generico a due hop, origine e SSE. Non usare
`tht auth configure --mode oidc` per «completare» l'accesso Omics già funzionante.
## Come funziona l'adapter
`OmicsPortalAdapter` è il solo modulo frontend che conosce il documento Omics.
Legge il `data-lang` del selettore lingua, osserva `data-bs-theme` e ascolta lo
stato fullscreen del documento. Fornisce snapshot `{ locale, theme, fullscreen }`
al controller di shell. Non invia comandi al portale e non usa un handshake.
La pagina Omics deve continuare a esporre il selettore
`select.omics-language-select` con la lingua renderizzata nel suo `data-lang`, e
il tema light/dark nell'attributo di `html`. Il selettore lingua usa il normale
form Django; non occorre convertirlo in una richiesta asincrona.
Per un altro portale, implementare la stessa sottoscrizione e selezionare il
nuovo adapter nel punto di composizione. Le pagine, l'i18n e il workflow non
devono acquisire riferimenti al nuovo portale. Sul lato server, il nuovo
contenitore deve anche fornire un'identità verificata conforme al contratto
upstream. Cambiare una classe JavaScript non sostituisce quel requisito.
## Lingue e sessioni
Ci sono tre scelte distinte:
| Scelta | Dove viene conservata | Cosa influenza |
| --- | --- | --- |
| Lingua UI | preferenza full o stato Omics | navigazione, amministrazione e messaggi generali |
| Lingua di interazione | `interaction_language` nel manifest | domande, spiegazioni, scelte e controlli HITL |
| Lingua workspace | configurazione del workspace | documenti, descrizioni e contenuti di dominio |
La creazione web acquisisce la lingua UI prima delle operazioni asincrone e la
invia come `interactionLanguage` di ripiego. Il CLI riconosce localmente la lingua
della domanda originale e la fissa nel manifest; usa il ripiego per testo troppo
breve, ambiguo o composto soltanto da codice. Un cambio successivo non modifica
quella richiesta. Il gate trasmette la lingua nei widget: anche i controlli HITL
seguono la sessione, mentre la navigazione conserva la lingua UI.
La ripresa legge il manifest e non usa il locale del browser come
override. SQL, identificatori, valori e citazioni dei contenuti rimangono invariati.
Per sessioni precedenti senza `interaction_language`, la prima ripresa fissa la
lingua della domanda in modo idempotente, usando la lingua del workspace come
ripiego. Le sessioni che hanno già una lingua fissata la conservano.
Il cambio lingua di Omics ricarica la pagina. ThothII ricorda soltanto l'identificatore
della sessione per utente e pagina, senza salvare una trascrizione nel browser.
Il recupero riapre il pannello dei documenti; la ripresa operativa è esplicita e
non avvia una generazione soltanto perché la pagina è stata ricaricata. Le bozze
non inviate e le modifiche amministrative richiedono protezione dalla navigazione.
## Aggiungere e verificare traduzioni
I messaggi inglesi fungono da identificatori gettext-style e fallback. I
cataloghi italiani sono divisi per area per agevolarne la manutenzione. Usare
`useI18n()` nei componenti e interpolazioni nominate, evitando concatenazioni
che rendano impossibile cambiare l'ordine delle parole. Non chiamare il traduttore
su SQL, testi del modello o descrizioni del workspace.
Per una nuova lingua aggiungere il catalogo, registrarlo nel risolutore e renderlo
disponibile nel selettore full. Il portale deve fornire il relativo locale. La
lingua delle sessioni è già esplicita e non richiede una nuova struttura del manifest.
Le traduzioni dei controlli della griglia provengono dal catalogo ufficiale della
stessa versione di AG Grid.
```bash
cd frontend
npm run check:i18n
npx tsc -b
npx vitest run
```
Il controllo dei cataloghi segnala messaggi statici mancanti, interpolazioni
incompatibili e traduzioni discordanti. Non può provare da solo la copertura di
tutti i messaggi dinamici: completarlo con i test delle pagine e la verifica visiva.
La build verifica anche il CSS effettivamente generato: il reset Tailwind viene
limitato al mount React e ai suoi popup con selettori ordinari, senza richiedere
supporto browser a `@scope`. Mantenere letterali le classi dei due modi della shell
per conservarle durante la rimozione del CSS inutilizzato. Le griglie usano il tema
CSS esistente con i token light/dark; non mescolarlo con la nuova Theming API di AG Grid.
## Verifica prima del deploy server
### Acquisire prima le modifiche al repository Omics
Procedura aggiornata il 14 settembre 2026: acquisire il codice Omics da GitHub
e integrarlo con il codice operativo del server. Non è richiesta alcuna replica
del repository su altri servizi; l'eventuale copia è un'attività distinta del
proprietario. Questa indicazione sostituisce le precedenti note di trasporto,
anche se ancora presenti nei documenti storici del branch Omics.
La [consegna corrente a Codex sul server](server-codex-handoff.md) contiene i
comandi esatti di acquisizione, gli SHA, i file da adeguare, la configurazione
embedded/upstream, i test, i gate di rilascio e il rollback. Usarla come procedura
ordinata per l'aggiornamento; le sezioni qui sotto restano il riepilogo tecnico.
Consegna GitHub riverificata: branch `codex/thothii-embedded-shell` di
`https://github.com/Dallavilla-Tiziano/omics_portal.git`, SHA
`fca10901a73666ca257d8f4cc4b77066295c400a`, incluso il commit funzionale
`95154e179144e2453b37ef2a63a65d6f377e4cf8`. Il pull di ThothII non aggiorna
Omics: il checkout del portale, normalmente `/home/chirone/omics_portal`, va
verificato e integrato separatamente preservando le modifiche successive del server.
### Preparare il rilascio coordinato
1. Registrare SHA approvati di **entrambi** i repository, immagini precedenti,
descriptor ThothII, file Compose/override e progetto realmente in uso. La testa
del branch di lavoro non è automaticamente una revisione approvata di produzione.
2. In un checkout di revisione separato, integrare Omics con il branch di rilascio
concordato. Non fare merge nel checkout operativo con modifiche altrui.
I file Omics da includere sono template Datamart Builder/topbar/base, asset
fullscreen e cataloghi Django del branch; conservare la verifica server
esistente in `kokoro/datamart_catalog_views.py` e le location Nginx protette.
3. Eseguire dal checkout Omics i test isolati, non i test contro il database operativo:
```bash
docker build -f test_support/thothii/Dockerfile -t omics-portal:thothii-shell-tests .
docker run --rm --network none omics-portal:thothii-shell-tests
```
4. Predisporre il descrittore ThothII embedded e il core upstream secondo la guida.
Verificare che Nginx Omics possa raggiungere gli alias privati `thothii-core:8787`
e `thothii-frontend:8080` e che non esista un ingresso non protetto al core.
Non sovrascrivere rete, mount o autenticazione usando il Compose locale del Mac.
5. Solo dopo il gate operatore, applicare le revisioni approvate seguendo il
lifecycle dei due progetti. In Omics i servizi sono `web` e `nginx`: includere
nel rebuild template, statici e cataloghi, mantenendo tutti gli override del
server. Verificare la configurazione Nginx con `nginx -t` nel servizio e lo
stato di entrambi. Non inventare opzioni Compose/progetto: usare quelle
registrate al punto 1. Gli entrypoint del portale possono avere altri effetti
operativi: questa modifica non richiede nuove migrazioni DB, ma non autorizza
a bypassare i controlli del suo rilascio.
6. Dopo l'aggiornamento del frontend, attendere la cache manifest Django (30 s)
oppure usare l'invalidazione prevista dal portale; ricaricare e controllare
config/asset/prefisso API prima di giudicare il risultato.
7. Compilare la matrice seguente. In caso di errore ripristinare revisioni,
immagini e configurazioni registrate, senza cancellare volumi. Il rollback
deve conservare una coppia compatibile di template Omics e frontend ThothII.
La consegna GitHub è verificata; il deploy della revisione integrata
Omics rimane da confermare dall'operatore. I test locali non attestano lo stato
attuale del server remoto.
### Accettazione dell'integrazione
Usare la [matrice completa full/embedded e autenticazione](../testing/authentication-manual-acceptance.md),
registrando per ogni prova revisione, ambiente e risultato. Non spuntare i casi
IdP/Omics reali soltanto perché passano i test con risposte simulate.
Provare l'apertura dal menu Omics con un utente autorizzato e uno senza accesso;
verificare assenza di un secondo login e di header ThothII, italiano/inglese già
selezionati prima dell'apertura, tema, fullscreen e uscita con Esc. Ripetere con
un menu o un form aperto, una bozza non inviata e una sessione esistente.
Verificare perdita dell'accesso, ritorno alla scheda e riconnessione degli eventi;
distinguere questi casi dal rifiuto di una sola operazione. Controllare che una
ripresa conservi la lingua salvata e che il reload non avvii una nuova generazione.
Eseguire i test Omics nel suo ambiente Docker e includere gli aggiornamenti
dei template, degli asset e dei cataloghi Django nel suo normale rebuild.
La verifica del codice e i test locali non costituiscono un deploy sul server
di produzione. Usare il normale processo di rilascio per applicare entrambe le
revisioni e annotare immagini, descrittore e revisioni realmente installate.
+2 -2
View File
@@ -73,9 +73,9 @@ only that run's safe stage, error code, and finish time; use `docker compose log
corresponding service log.
The contract gives exact validation, exit code, and JSON rules in
[Workspace preprocessing CLI](../contracts/workspace-preprocessing-cli.md). For Evidence source
[Workspace preprocessing CLI](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/workspace-preprocessing-cli.md). For Evidence source
forms and the schema-v4 descriptor contract, see
[Workspace Evidence v3](../contracts/workspace-evidence-v3.md).
[Workspace Evidence v3](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/workspace-evidence-v3.md).
## Transport and revision rules
@@ -1,695 +0,0 @@
# Metadata Catalog di ThothII: ricognizione ThothAI e percorso incrementale
Data: 2026-08-26; aggiornato 2026-08-27
Stato: ricognizione e progettazione completate; navigazione, CRUD Workspace Database, Catalog
Table, Catalog Column, Catalog Relationship e sincronizzazione durevole dello schema implementati
il 2026-08-27. Generazione AI e integrazione con il workflow core restano negli step successivi.
## Obiettivo
ThothII deve introdurre un contesto amministrativo separato, il **Metadata Catalog**, per gestire
il database associato a ciascun workspace, la sua struttura fisica introspezionata e i metadati
semantici oggi rappresentati da `schema/annotations.yaml`.
Il programma procede per step indipendenti. Il primo step ha aggiunto l'accesso dalla sidebar; il
secondo ha sostituito la superficie vuota con il CRUD di configurazione, il PostgreSQL interno e i
test di connessione; gli step successivi hanno aggiunto navigazione gerarchica, colonne, relazioni
fisiche e sincronizzazione durevole dell'intero schema. Non introduce ancora generazione AI o
integrazione con il workflow core.
Questa analisi usa come riferimento il working tree legacy osservato in
`Thoth/ThothAI`. Non è stato verificato che quel contenuto corrisponda a una release o a un tag
canonico; i percorsi e i comportamenti descrivono il sorgente disponibile il 2026-08-26.
## Decisioni già confermate
1. Ogni Workspace Database appartiene a un solo workspace tramite un `workspace_id` obbligatorio e
univoco; un workspace può avere al massimo un Workspace Database. Poiché i workspace non sono
righe del catalogo PostgreSQL, l'associazione è un riferimento logico validato contro
`thoth-workspaces.yaml`, non una foreign key SQL.
2. Il CRUD non crea né rinomina workspace. Identità e lista ordinata dei workspace restano
autorevoli in `thoth-workspaces.yaml`; il catalogo conserva il loro identificatore stabile.
3. La struttura fisica viene acquisita interrogando il database esterno tramite i dati di
connessione registrati per il Workspace Database.
4. I contenuti semantici equivalenti a `annotations.yaml` vengono generati con l'AI e conservati nel
PostgreSQL interno.
5. Per PSD è prevista l'importazione delle annotations esistenti. Gli altri database partiranno
dalla struttura introspezionata e genereranno i metadati semantici da zero.
6. `annotations.yaml` sarà sostituito anche come input del core in uno step futuro. Il repository è
in fase di test e non è richiesta la conservazione delle sessioni esistenti durante il cutover.
7. La gestione catalogo resta una superficie separata dal processo NL→SQL. La futura integrazione
deve essere esplicita e non deve modificare fasi, gate o semantica del workflow.
8. Il link iniziale è visibile agli utenti con `workspace.manage`, usa stato React locale e non
introduce un router.
9. La pagina iniziale è vuota, segue il tema, nasconde l'intera colonna core e non interrompe una
sessione live. Le azioni di apertura, resume o creazione sessione riportano al core.
10. La compatibilità con il modello ThothAI è semantica, non una copia letterale: configurazione e
contenuti semantici sono campi relazionali mutabili, mentre identità e appartenenza della
struttura fisica derivano dall'introspezione; i segreti restano nel secret store e lo stato dei
job non viene mescolato ai dati amministrativi.
11. Il CRUD amministra il Metadata Catalog e non esegue DDL sul database esterno, che resta
read-only.
12. La prima versione supporta PostgreSQL; il confine di introspezione dovrà permettere di
aggiungere altri dialetti senza cambiare il modello del catalogo.
13. I segreti dei Workspace Database riusano il secret store cifrato di ThothII. Il catalogo
conserva riferimenti ai segreti e nessuna API, esportazione o log ne restituisce i valori.
14. La UI usa AG Grid Community per la lista master e un pannello React separato per il dettaglio;
non dipende dalle funzionalità master-detail di AG Grid Enterprise.
15. Un Workspace Database il cui `workspace_id` scompare dal catalogo YAML non viene cancellato
automaticamente: diventa orphaned e può soltanto essere recuperato, riassegnato o eliminato
esplicitamente da un amministratore.
16. La prima vertical slice gestisce configurazione del Workspace Database, riferimenti ai segreti,
test di connessione e stato. La seconda gestisce le Catalog Table: la collezione e i nomi sono
controllati dall'introspezione, mentre la descrizione curata è modificabile. Le slice successive
hanno aggiunto Catalog Column, Catalog Relationship e sincronizzazione durevole dello schema.
17. Il modello non conserva il `name` libero di ThothAI: nome e ID visualizzati appartengono al
workspace YAML, mentre `database_name` identifica il database PostgreSQL esterno.
18. Database management supporta i tre trasporti già riconosciuti da ThothII: `postgres_direct`,
`rest_api` e `ssh_tunnel`. PSD rimane un solo Workspace Database: usa la connessione diretta sul
server e l'endpoint REST in locale tramite una Database Binding specifica dell'installazione.
Questo supporto non abilita automaticamente `ssh_tunnel` nel runtime NL→SQL.
19. Una configurazione può essere salvata prima di una connessione riuscita. Il test separato
produce uno stato `untested`, `reachable` o `failed`; attivazione e introspezione richiedono uno
stato raggiungibile.
20. Il CRUD e il test di connessione richiedono `database.manage`; inserimento e sostituzione dei
segreti continuano a richiedere `workspace.secrets.manage`.
21. Il Workspace Database e il modo di raggiungerlo sono entità distinte. Ogni catalogo di
installazione conserva una sola Database Binding attiva per workspace: PSD usa `rest_api` in
locale e `postgres_direct` sul server senza duplicare il Workspace Database.
22. Nel modello finale il Metadata Catalog è autorevole per engine, `database_name`, schema,
capacità e binding. Lo YAML resta autorevole per identità e contenuti del workspace; i campi
DWH correnti saranno importati, confrontati e rimossi soltanto durante un cutover esplicito.
23. La lista master è l'unione fra workspace YAML e record del catalogo: mostra workspace
`unconfigured`, database configurati e record `orphaned`.
24. Ogni introspezione registra le capability disponibili. Una capability `unavailable` non viene
rappresentata come una collezione osservata ma vuota; REST può completare con successo anche
quando indici o enum non sono supportati.
25. Il Metadata Catalog non introduce snapshot, draft o pubblicazioni. Configurazione e contenuti
semantici, inclusi quelli futuri generati dall'AI, sono normali campi modificabili; la struttura
osservata cambia soltanto con una sincronizzazione esplicita.
26. Il normale Delete elimina realmente il Workspace Database, la Database Binding e i relativi
record catalogo e segreti. Non modifica il DWH esterno né il repository YAML; il workspace torna
visibile nella lista master come `unconfigured`.
27. La prima versione gestisce un solo schema obbligatorio per Workspace Database, identificato
dalla coppia `database_name + schema`; per PSD la coppia è `postgres + datawarehouse`.
28. I record mantengono soltanto `created_at`, `updated_at` e un contatore `version` per optimistic
concurrency. Non esistono storico delle revisioni, rollback o audit applicativo delle modifiche.
29. `workspace_databases` conserva soltanto UUID, `workspace_id` unique, engine, `database_name`,
schema, timestamp e version. Il nome visualizzato appartiene al workspace YAML.
30. Ogni Workspace Database ha al massimo una riga `database_bindings`. Una singola tabella usa
check constraint dipendenti da `transport` per i campi direct, REST e SSH; non esiste un flag
`active`, perché ciascuna installazione conserva una sola binding.
31. `rest_api` configura il Thoth REST Connector tipizzato: base URL, autenticazione e TLS sono dati
della binding, mentre path RPC e shape delle risposte appartengono al contratto applicativo e non
sono liberamente configurabili.
32. Il test connessione usa soltanto una configurazione già salvata ed è associato alla sua
`version`. Ogni modifica della binding o dei segreti invalida il risultato precedente e riporta
lo stato a `untested`.
33. Password, API key e chiavi sono write-only: l'API espone soltanto `configured`, un campo vuoto
conserva il valore esistente e la sostituzione è un'azione esplicita. Delete rimuove anche i
segreti associati.
34. La pagina usa AG Grid come master e un form React come detail, con sezioni Database, Connection
e TLS/SSH condizionali. Non esiste un'azione globale `Add database`: ogni riga `unconfigured`
offre `Configure catalog`, apre il form già vincolato a quello specifico workspace YAML e crea il
record soltanto al Save; `workspace_id` non è selezionabile né modificabile.
35. La grid mostra separatamente revisione/Evidence del workspace, binding runtime NL→SQL e
configurazione del Metadata Catalog, oltre a database, schema, endpoint e ultimo aggiornamento.
Su schermi piccoli il dettaglio occupa il pannello completo. Il cambio riga con modifiche non
salvate e Delete richiedono conferma, senza conferma testuale tipizzata.
36. La Database Binding conserva `connection_status`, `tested_version`, `last_tested_at`, un codice
errore e un messaggio breve sanificato. Non conserva stack trace, DSN, credenziali o output grezzo
del driver.
37. Le API vivono sotto `/api/catalog`: list/create di `/databases`, get/patch/delete di
`/databases/:id`, sostituzione dei segreti sotto `/databases/:id/secrets`, test connessione sotto
`/databases/:id/test` e list/patch/sync delle tabelle sotto `/databases/:id/tables`.
38. `GET /api/catalog/databases` restituisce l'intera master list unificata; AG Grid Community applica
client-side ricerca, filtri e ordinamento. La prima versione non introduce paginazione server o
funzionalità AG Grid Enterprise.
39. Il Metadata Catalog vive nello stesso processo Fastify come modulo isolato con repository,
service, route, diagnostica e readiness proprie. L'indisponibilità del catalogo non modifica
sessioni, SSE o health del core e non giustifica ancora un microservizio separato.
40. Il backend mantiene `pg@8.22.0` e aggiunge `kysely@0.29.5` per query e transazioni tipizzate. Le
migrazioni Kysely sono timestampate, compilate con il backend ed eseguite da un comando
`catalog:migrate` separato; l'applicazione non migra automaticamente il database all'avvio.
41. Lo stack aggiunge un servizio interno `catalog-db` con volume persistente, ruolo runtime DML,
ruolo migrator DDL e job one-shot `catalog-migrate`. Un catalogo indisponibile produce 503 sulle
sole route catalogo.
42. La prima vertical slice è amministrativa: scrive il catalogo ma non cambia ancora il runtime di
sessioni e workflow, che continua a usare YAML e binding correnti fino al cutover esplicito.
43. `Configure` precompila senza salvare engine, database e schema dal descriptor e i dati non
sensibili dalla binding effettiva. L'amministratore verifica, inserisce i segreti e salva; non
esiste importazione silenziosa.
44. Unit e route test usano un repository fake; una suite PostgreSQL Testcontainers separata verifica
migrazioni, constraint, transazioni, optimistic concurrency e cascade. SQLite ed emulatori non
sono sostituti ammessi per questi test.
45. La navigazione delle entità catalogo è gerarchica e senza scorciatoie globali: `Databases →
Database → Overview | Tables → Table`. Non esistono una voce globale Tables, un filtro globale
Database o una preselezione implicita; Columns continuerà sotto Table e Relationships sotto
Database.
46. Una Catalog Table conserva nome fisico, `source_comment`, descrizione curata nullable,
`generated_description` nullable per lo step AI futuro, version e timestamp. La UI mostra come
tre campi indipendenti senza fallback visivo: source comment read-only, generated description
modificabile e description modificabile. I valori null restano celle e controlli vuoti.
47. Le Catalog Table non possono essere aggiunte o rinominate manualmente. Un amministratore può
però ripulire esplicitamente le proiezioni nel Metadata Catalog senza modificare il database
esterno; `Sync tables` legge le tabelle PostgreSQL ordinarie e partizionate dello schema scelto,
mentre viste e materialized view sono escluse.
48. La sincronizzazione è esplicita. La scansione avviene fuori dalla transazione del catalogo; il
diff viene applicato atomicamente soltanto se la version del Workspace Database è ancora quella
sottoposta a scansione. Una scansione fallita non modifica il catalogo.
49. Tabelle nuove vengono create, i commenti sorgente vengono aggiornati e quelle non più osservate
vengono eliminate definitivamente. La rimozione di tabelle, colonne o relazioni richiede la
conferma dell'esatto piano distruttivo; se il secondo scan produce una fotografia differente,
l'applicazione richiede una nuova conferma.
50. Un rename fisico è intenzionalmente delete più create e perde i metadati curati. Le colonne e
relazioni dipendenti vengono eliminate in cascade insieme alla Catalog Table.
51. L'introspezione vive nel modulo catalogo Fastify dietro un adapter. PostgreSQL diretto e tunnel
SSH usano il catalogo `pg_catalog`; REST preferisce il contratto tipizzato
`POST /rpc/schema_snapshot` e, quando quell'RPC non è esposto, usa come fallback compatibile una
singola query read-only tramite `POST /rpc/run_query`. Entrambi i percorsi devono produrre la
stessa fotografia v1 stretta descritta in `docs/contracts/catalog-schema-snapshot.md`.
52. Test connessione e sincronizzazione sono serializzati per Workspace Database, hanno timeout e
richiedono che la binding nella version corrente abbia un test `reachable` prima di qualsiasi
Catalog Sync Run. La scansione asincrona ha un timeout separato, di default dieci minuti.
53. Il tunnel SSH usa OpenSSH in modalità stdio `-W`, chiave privata e passphrase opzionale dal
secret store, `known_hosts` obbligatorio, `StrictHostKeyChecking=yes`, agent e configurazione
globale disabilitati. Non è ammesso TOFU. TLS PostgreSQL con CA e server name resta verificato
anche attraverso il tunnel.
54. In questo slice `ssh_tunnel` è una binding supportata da Database management per Test connection
e Schema Sync. Il renderer e il runtime delle sessioni NL→SQL restano fuori scope e continuano a
rifiutarla finché non verrà deciso il relativo cutover.
55. I menu di azione a livello Workspace Database espongono separatamente `Synchronize tables`,
`Synchronize relationships` e `Synchronize all`. Su una selezione di
database lo scope scelto viene avviato per ogni database idoneo; non viene sostituito
implicitamente con una sincronizzazione completa.
56. Lo scope Columns è disponibile dalla grid Tables e limita la riconciliazione alle tabelle
selezionate; la pagina Columns non espone azioni di sincronizzazione. La grid Tables espone
`Synchronize columns` sulle tabelle selezionate.
## Correzione del modello mentale corrente
`schema/annotations.yaml` non contiene l'intero schema del database.
- `physical.yaml` è un artefatto derivato dall'introspezione. Contiene database, schema, timestamp,
tabelle, colonne, tipi, nullability, default, primary key, commenti sorgente, esempi, foreign key
fisiche e indici.
- `annotations.yaml` contiene metadati curati: descrizioni e concetti delle tabelle; descrizioni,
sinonimi, concetti, evidence, note e override `eligible` delle colonne; foreign key logiche.
- Il rendering M-Schema fonde questi due input. Le annotations prevalgono sui commenti sorgente e
le relazioni logiche vengono unite alle foreign key fisiche.
La sostituzione del solo file annotations non elimina automaticamente l'introspezione fisica. Il
nuovo catalogo dovrà conservare una distinzione esplicita fra fatti osservati nel database e
contenuto semantico modificabile.
## Architettura ThothII rilevante
### Autorità e revisionamento attuali
Il repository dei workspace contiene:
```text
thoth-workspaces.yaml
<workspace-id>/workspace.yaml
<workspace-id>/schema/annotations.yaml
<workspace-id>/evidence/**
```
Il backend legge descriptor e annotations allo stesso commit Git. Durante l'attivazione valida il
blob, lo copia atomicamente nello snapshot immutabile della revisione e registra commit, blob ID e
digest. Le nuove sessioni vengono legate a quella revisione; resume e SQL salvato riaprono lo stesso
snapshot.
Punti principali:
- `backend/src/workspaces/schema.ts`: descriptor v3 e singolo `dwh.database`/`dwh.schema`;
- `backend/src/workspaces/git-repository.ts`: lettura sicura del blob annotations al commit;
- `backend/src/workspaces/registry.ts`: validazione e attivazione atomica;
- `backend/src/workspaces/annotations-sync.ts`: materializzazione revision-qualified;
- `backend/src/workspaces/runtime-config-lease.ts`: binding dello snapshot al runtime;
- `harness/tht/mschema/models.py`: contratti `PhysicalSchema` e `Annotations`;
- `harness/tht/mschema/render.py`: fusione fisico/semantico;
- `harness/tht/cli/vector_cmd.py`: indicizzazione schema in Qdrant.
### Consumatori da preservare al cutover futuro
Le annotations incidono oggi su:
- override `eligible` prima del campionamento LSH;
- suggerimento, controllo e accettazione delle foreign key logiche;
- descrizioni, concetti e sinonimi dei record schema in Qdrant;
- retrieval delle tabelle e colonne candidate;
- rendering M-Schema usato dal gate F4 e dalla generazione SQL;
- digest della revisione accettata durante il preprocessing.
Il futuro cutover non potrà limitarsi a rimuovere il file: dovrà fornire al core lo stesso contenuto
effettivo, con un'identità coerente e test di equivalenza. Poiché non occorre preservare le sessioni
di test esistenti, non serve progettare compatibilità con i vecchi manifest, ma resta necessario
evitare letture parziali o semanticamente incoerenti.
## Inventario ThothAI
### Modelli legacy
I modelli sono definiti in `Thoth/ThothAI/backend/thoth_core/models.py`.
#### `SqlDb`
Campi di connessione osservati:
- `name`;
- `db_host`, `db_port`;
- `db_type`;
- `db_name`, `schema`;
- `user_name`, `password`;
- `db_mode`;
- configurazione SSH e Informix opzionale.
Il modello contiene anche scope, JSON dello scope, ERD, direttive, campi GDPR, collegamento a
`VectorDb` e numerosi campi di stato/task/log per lavori AI asincroni.
I tipi legacy dichiarati sono Informix, MariaDB, MySQL, Oracle, PostgreSQL, SQL Server e SQLite.
Questo elenco non costituisce automaticamente un requisito per ThothII: il core corrente supporta
PostgreSQL e l'estensione ad altri dialetti dovrà essere decisa separatamente.
#### `SqlTable`
- `name`;
- `description`;
- `generated_comment`;
- foreign key obbligatoria a `SqlDb`, con cancellazione cascade.
#### `SqlColumn`
- `original_column_name` e alias `column_name`;
- `data_format` normalizzato;
- `column_description`;
- `generated_comment`;
- `value_description`;
- stringhe denormalizzate `pk_field` e `fk_field`;
- foreign key obbligatoria a `SqlTable`, con cancellazione cascade.
#### `Relationship`
Contiene quattro foreign key obbligatorie:
- `source_table` e `source_column`;
- `target_table` e `target_column`.
Il form admin verifica che le tabelle appartengano allo stesso database e che ogni colonna
appartenga alla tabella selezionata. Il database non impone però gli stessi check.
#### `Workspace`
ThothAI usa `Workspace.sql_db` come foreign key nullable verso `SqlDb`: un workspace seleziona un
solo DB, mentre lo stesso DB può essere riusato da più workspace. ThothII adotterà invece una
relazione uno-a-uno: `workspace_id` deve essere unico nel catalogo.
### Lacune dei constraint legacy
Non risultano constraint database-level per:
- unicità del nome database nel workspace;
- unicità `(database, table name)`;
- unicità `(table, column name)`;
- unicità degli estremi di una relationship;
- appartenenza degli estremi della relationship allo stesso database;
- corrispondenza fra colonna e tabella dichiarata.
ThothII deve applicare queste invarianti sia nel database interno sia nel servizio applicativo. La
sola validazione del form non è sufficiente perché API, import e job la possono aggirare.
### Django Admin e UX da replicare concettualmente
ThothAI espone il CRUD tramite il Django Admin standard, registrato da
`backend/thoth_core/admin.py` e pubblicato su `/admin/`.
Capacità utili:
- lista database con ricerca per nome, host, tipo, database e schema;
- fieldset separati per identità, connessione, autenticazione, SSH e stato;
- lista tabelle filtrabile per database;
- lista colonne filtrabile in cascata per database e tabella;
- lista relazioni con estremi leggibili e filtri per database e tabelle;
- form relazione con dropdown dipendenti database → tabella → colonna;
- validazione degli estremi prima del salvataggio;
- azioni separate per test connessione, introspezione, import/export e generazione AI;
- azioni bulk sulle righe selezionate.
ThothII deve replicare i contratti di interazione e validazione, non il rendering server-side o i
template Django.
### Introspezione legacy
`Thoth/ThothAI/backend/thoth_core/dbmanagement.py` usa `thoth-dbmanager` per:
1. costruire l'adapter del dialetto;
2. acquisire tabelle;
3. acquisire e normalizzare colonne e tipi;
4. acquisire relazioni;
5. creare le eventuali colonne mancanti necessarie alle relazioni;
6. aggiornare i campi PK/FK denormalizzati.
Il comportamento è principalmente additivo: usa `get_or_create` o controlli `exists`, aggiorna
alcuni commenti, ma non riconcilia in modo completo rename, rimozioni o drift. Non va copiato così
com'è. Il processo ThothII implementato distingue scansione, differenze osservate e applicazione
della nuova snapshot.
### Generazione AI legacy
ThothAI dispone di azioni e workflow per:
- commenti delle tabelle;
- commenti delle colonne;
- scope del database;
- ERD Mermaid;
- documentazione del database;
- analisi GDPR.
Per il requisito attuale sono direttamente rilevanti descrizioni di tabelle e colonne, scope e
metadati semantici. ERD, documentazione aggregata e GDPR sono estensioni future, non prerequisiti
del CRUD iniziale.
La separazione `description`/`generated_comment` del legacy non offre versioning o approvazione
robusti. Nei passi successivi andrà deciso se l'output AI è una proposta revisionabile o diventa
immediatamente il valore editabile corrente.
### Import ed export legacy
ThothAI offre:
- CSV di database, tabelle, colonne e relazioni;
- export di struttura per workspace;
- import mediante `import_db_structure`;
- script SQL dei commenti per più dialetti;
- aggiornamento delle descrizioni colonna da CSV.
Il futuro import PSD dovrà leggere il contratto YAML corrente e convertirlo su chiavi naturali,
non riutilizzare gli ID numerici Django. Deve essere idempotente e produrre un report di elementi
creati, aggiornati, ignorati o non risolti.
## Comandi osservati in ThothAI
### Backend locale
Eseguiti da `Thoth/ThothAI/backend`:
```sh
uv sync
uv run python manage.py migrate
uv run python manage.py createsuperuser
uv run python manage.py runserver 8200
uv run pytest
```
Import catalogo legacy:
```sh
uv run python manage.py import_db_structure --source local
uv run python manage.py load_defaults --only-level 4 --source local
```
Test mirati rilevanti:
```sh
uv run pytest tests/test_relational_database_operations.py -v
uv run pytest tests/test_ssh_tunnel_configuration.py -v
```
### Stack Docker legacy
ThothAI dichiara `postgres:16-alpine` nel profilo `internal-db`, con volume persistente e
healthcheck `pg_isready`.
```sh
docker compose --profile internal-db up --build
```
Il wrapper legacy abilita lo stesso profilo quando `POSTGRES_INTERNAL=true`:
```sh
POSTGRES_INTERNAL=true ./docker-up.sh
```
Questi comandi documentano il riferimento osservato; non sono comandi di installazione per
ThothII.
## Cosa copiare in ThothII
### Parità necessaria
- gerarchia Workspace Database → Table → Column;
- relazione strutturale fra colonne sorgente e destinazione;
- navigazione e filtri dipendenti workspace/database/tabella;
- test di connessione separato dal salvataggio;
- introspezione esplicita e ripetibile;
- descrizioni generate dall'AI ma modificabili dall'utente;
- validazione cross-entity delle relazioni;
- azioni di import/export senza segreti;
- stato leggibile dei job lunghi;
- PostgreSQL interno persistente con migrazioni esplicite;
- test di CRUD, cardinalità, cascade/restrict, isolamento per workspace e idempotenza.
### Parità semantica con `annotations.yaml`
Il modello futuro deve poter rappresentare almeno:
- descrizione, concetti e note per tabella;
- descrizione, sinonimi, concetti, evidence, note ed `eligible` per colonna;
- foreign key logiche;
- distinzione fra commento fisico osservato e descrizione curata;
- provenienza del contenuto importato o generato.
L'eventuale esclusione di uno di questi campi deve essere una decisione esplicita perché cambia
rendering, retrieval, LSH o SQL generation.
### Vincoli minimi da progettare
- `workspace_id` obbligatorio e unico sul Workspace Database, con esistenza validata contro il
catalogo YAML dal servizio applicativo;
- nome tabella unico nel database e schema appropriato;
- nome colonna unico nella tabella;
- relationship unica secondo il modello, anche per chiavi composite;
- estremi della relationship nello stesso Workspace Database;
- appartenenza certa della colonna alla tabella;
- mutazioni aggregate transazionali;
- gestione esplicita di concorrenza fra CRUD e introspezione.
## Cosa non copiare
- Django, Django Admin, Django ORM, DRF, template admin e frontend Next;
- modello Workspace legacy e condivisione dello stesso DB fra più workspace;
- password o passphrase come normali campi testuali;
- password incluse in CSV o export completi;
- token SSO inseriti nella query string;
- migrazioni generate automaticamente all'avvio;
- validazioni presenti soltanto nel form;
- `pk_field` e `fk_field` testuali come fonte di verità;
- duplicazione di tabella e colonna negli estremi senza constraint coerenti;
- introspezione additiva che non segnala rename, delete o drift;
- azioni admin che possono mostrare successo dopo output AI non valido;
- dipendenza del workflow core dalla disponibilità della UI o del PostgreSQL amministrativo.
## Aspetti di sicurezza da non ereditare
L'export legacy della struttura include username e password in chiaro. Il modello conserva inoltre
password, passphrase SSH e altri segreti in `CharField`; non è stata trovata cifratura applicativa,
nonostante un testo admin affermi il contrario.
ThothII distingue i metadati di connessione dai riferimenti al secret store cifrato. In ogni caso:
- nessun endpoint o export deve restituire segreti;
- log ed errori devono sanificare DSN e credenziali;
- le credenziali di migrazione non devono essere disponibili al runtime CRUD;
- il catalogo non deve riusare credenziali del DWH, delle sessioni o di Qdrant;
- test connessione e introspezione devono usare timeout e privilegi read-only.
La binding REST corrente richiede una verifica prima del cutover: il renderer emette
`ssl_ca_file`, mentre il modello Python espone `ssl_ca`; il percorso della CA privata potrebbe quindi
non essere consumato. PSD richiede TLS con CA privata in locale, perciò questo disallineamento deve
essere corretto e coperto da un test end-to-end prima di affidare il profilo REST al catalogo.
## Percorso incrementale
### Step 1: accesso alla superficie vuota
Implementato in questo worktree:
- pulsante `Database management` nella sidebar destra;
- visibilità legata a `workspace.manage`;
- superficie centrale React separata e vuota;
- nessun router, endpoint, fetch o stato catalogo;
- sessione e SSE conservati in background;
- ritorno al core tramite creazione, apertura o resume di una sessione;
- test frontend dedicati.
Comandi di verifica:
```sh
cd frontend
npx vitest run src/shell/AppShell.database-management.test.tsx
npx vitest run src/shell/AppShell.new-session.test.tsx \
src/shell/AppShell.session-target.test.tsx \
src/shell/AppShell.session-mgmt.test.tsx
npx tsc -b
```
### Step 2: contratto di dominio e schema relazionale
Progettazione della vertical slice completata: Workspace Database, Database Binding, singolo schema,
riferimenti al secret store, optimistic concurrency e capability per trasporto hanno contratti
espliciti. Configurazione e contenuti semantici restano mutabili; la struttura fisica osservata è
sincronizzata e non modificabile manualmente.
### Step 3: PostgreSQL interno e migrazioni
PostgreSQL interno con volume e ruoli runtime/migrator separati. Il modulo catalogo usa Kysely sopra
il driver `pg`; le migrazioni compilate vengono applicate soltanto dal comando `catalog:migrate` e
mai allo startup Fastify. Health, readiness e diagnostica restano dedicate; l'indisponibilità del
catalogo non cambia `core /health` e non interrompe una sessione.
### Step 4: API CRUD
Contratti HTTP, autorizzazione, paginazione, filtri, errori, optimistic concurrency e transazioni.
Gli endpoint dovranno vivere sotto un namespace catalogo e non riutilizzare le route sessione.
### Step 5: UI CRUD
Workspace Database, Catalog Table, Catalog Column e Catalog Relationship sono implementati con
React/Vite e il design system ThothII.
La navigazione è gerarchica e locale al database (`Overview | Tables`), senza menu o filtri globali
per tipo di entità. La grid delle tabelle non offre Add o cancellazione della singola configurazione;
le selezioni espongono invece la pulizia esplicita dei metadati. Il dettaglio full-width mantiene
immutabili i fatti fisici e consente di modificare separatamente Description e Generated
Description. Colonne e relazioni seguono la stessa gerarchia: Columns appartiene al dettaglio
della tabella, Relationships al database. I valori descrittivi null sono mostrati come celle e
campi vuoti, senza fallback visivi o placeholder `Not set` che nascondano quale sorgente è
effettivamente valorizzata.
Le griglie che dispongono di azioni massive usano checkbox e una toolbar contestuale con conteggio,
menu `Actions` e cancellazione della selezione. La selezione identifica ID espliciti, può essere
accumulata attraverso i filtri e viene azzerata dopo successo, nuova sincronizzazione o uscita
dalla pagina; un'azione è all-or-nothing se un elemento non è idoneo. I menu a livello database
espongono gli scope fisici come azioni distinte: `Synchronize tables`, `Synchronize relationships`
e `Synchronize all`. La grid Tables espone invece `Synchronize columns` per le tabelle selezionate;
la pagina Columns non espone sincronizzazione. Le selezioni database aggiungono `Delete all tables` e
`Delete all relationships`; le selezioni tabelle aggiungono `Delete all columns` e `Delete all
relationships`. Queste operazioni sono atomiche, richiedono conferma e non modificano database
esterno, binding, configurazione o segreti. Test connection resta un'azione distinta; griglie senza
azioni non mostrano controlli di selezione inerti.
### Step 6: introspezione
Catalog Table, Catalog Column e Catalog Relationship sono implementate per PostgreSQL diretto,
Thoth REST Connector e tunnel SSH. La scansione read-only è separata dalla transazione; una
riconciliazione atomica crea, aggiorna i commenti sorgente ed elimina, dopo conferma, i fatti fisici
assenti senza rendere modificabile manualmente la struttura osservata. Gli scope autorevoli sono
Tables per database e Physical Relationships per database. Per Columns, `tableIds` vuoto include
tutte le Catalog Table correnti, mentre una lista di ID limita lo scope al sottoinsieme esplicito;
`Synchronize all` osserva tutti e tre gli scope in un unico snapshot e li riconcilia insieme. Tutti
gli scope sono eseguiti come Catalog Sync Run durevoli in background, non attraverso implementazioni
sincrone e asincrone separate. Un run che prevede cancellazioni conserva il diff, attende una
conferma esplicita e verifica nuovamente lo snapshot prima dell'applicazione; se la sorgente è
cambiata, invalida la conferma. Ogni applicazione è atomica e fail-closed: errori, timeout o
capability non disponibili non producono aggiornamenti parziali.
PK e FK devono essere visibili sulle Catalog Column senza duplicare le stringhe denormalizzate di
ThothAI. La posizione nella primary key è un fatto osservato della colonna; membership e conteggio
FK sono proiezioni derivate dalle Catalog Relationship e dalle loro coppie ordinate, aggiornate
nella stessa transazione di riconciliazione.
Ogni scope registra la versione della Database Binding osservata e l'istante dell'ultima
sincronizzazione. Una modifica della binding conserva il catalogo precedente ma lo marca stale;
solo un `Synchronize all` riuscito rende nuovamente corrente l'intero schema.
### Step 7: generazione AI dei metadati
Generated Description è una proposta distinta e modificabile: un revisore può correggerla prima
di consolidarla esplicitamente come Description. Lo slice AI dovrà decidere e implementare anche
alias semantici, descrizioni dei valori, sinonimi e concetti per tabelle e colonne, oltre alla
gestione esplicita di errori e output non validi. La generazione AI e l'azione di consolidamento non
appartengono allo slice di introspezione dello schema.
### Step 8: migrazione PSD
Import idempotente delle annotations PSD, riconciliazione contro la struttura introspezionata,
report degli orfani e confronto semantico con il rendering corrente. Gli altri workspace non
ricevono import legacy.
### Step 9: sostituzione dell'input core
Rimuovere la dipendenza da `annotations.yaml` soltanto dopo avere un contratto equivalente,
test di rendering/search/Qdrant e una policy di disponibilità. Le sessioni di test esistenti
possono essere eliminate, ma le nuove sessioni non devono osservare aggiornamenti parziali.
Questo cutover è esplicitamente rinviato fino al completamento del database dei metadati. Il primo
gate successivo obbligatorio sarà valutare l'integrazione del Catalog Schema Snapshot con il
workflow core e lo schema-linking corrente; il rinvio non autorizza a dimenticare o assorbire
implicitamente il lavoro in altri slice.
### Step 10: operazioni e accettazione
Backup/restore reale, diagnostica, metriche, permessi definitivi, hardening degli export e
test di failure isolation fra catalogo e workflow. I Catalog Sync Run hanno un solo job attivo per
Workspace Database, sono concorrenti fra database diversi e usano un lock persistente. Un pannello
operativo non modale rimane visibile durante la navigazione del database, mostra fasi, contatori,
tempo trascorso e log sanitizzato via SSE con polling di fallback, e offre Confirm, Cancel e Retry
quando consentiti. Un restart marca `interrupted` i run rimasti attivi; il retry crea un nuovo run.
Le modifiche ai metadati restano consentite durante la scansione e sono preservate dall'applicazione.
Il worker gira inizialmente nello stesso servizio Fastify ma dietro un'interfaccia estraibile, con
coda, lease e heartbeat persistiti nel catalog-db. I riepiloghi dei run non scadono; gli eventi
dettagliati sono conservati per 30 giorni, mentre snapshot e diff completi vengono eliminati dopo
la conclusione lasciando conteggi, decisioni e una sintesi sanitizzata dell'esito.
## Verifiche del core da conservare per il cutover
Comandi attuali rilevanti:
```sh
tht --installation <absolute>/thothii-installation.yaml workspace preprocess dwh \
--workspace <id> --json
tht --installation <absolute>/thothii-installation.yaml workspace schema suggest-fks \
--workspace <id> --json
tht --installation <absolute>/thothii-installation.yaml workspace schema check \
--workspace <id> --json
tht --installation <absolute>/thothii-installation.yaml workspace schema accept \
--workspace <id> --run <run-id> --yes --json
tht --installation <absolute>/thothii-installation.yaml workspace index-schema \
--workspace <id> --json
```
Suite che documentano il comportamento da preservare:
```sh
cd backend
npx vitest run test/workspaces-git-annotations.test.ts \
test/registry-annotations.test.ts \
test/annotations-sync.test.ts \
test/workspace-runtime-config-lease.test.ts \
test/workspace-preprocessing-service.test.ts
npx tsc --noEmit -p .
cd ../harness
.venv/bin/pytest -q \
tests/test_annotations_root.py \
tests/test_schema_fk_annotations.py \
tests/test_mschema_render.py \
tests/test_qdrant_cli_commands.py
```
Questi test non implicano che la futura implementazione debba continuare a usare file YAML.
Definiscono gli effetti semantici e le guardie da mantenere o sostituire consapevolmente.
## Decisioni rinviate
Le seguenti scelte non appartengono allo step 1:
- lifecycle dei riferimenti ai segreti durante sostituzione e cancellazione;
- criteri per aggiungere dialetti successivi a PostgreSQL;
- criteri per un'eventuale estensione futura a più schemi per database;
- lifecycle e gestione amministrativa delle future Logical Relationship;
- alias semantici, descrizioni dei valori, sinonimi e concetti prodotti o assistiti dall'AI;
- formato e momento del cutover dal file al database interno;
- permission definitiva separata da `workspace.manage`.
Ognuna sarà affrontata nel relativo step, senza anticipare scelte tecnologiche nel presente
documento.
@@ -1,251 +0,0 @@
# AI-generated descriptions for Catalog Tables and Catalog Columns
## Problem Statement
ThothII already stores a Generated Description separately from the curated Description for Catalog
Tables and Catalog Columns, but administrators cannot populate it with AI. ThothAI provides the
useful core workflow—generate table and column comments from schema context and small real-data
samples—but its execution, configuration, and interaction model cannot be copied directly into
ThothII.
Administrators need an asynchronous workflow integrated into Database Management. They must be
able to choose an installation-approved model, generate descriptions for selected or missing
targets, observe understandable progress, stop or recover a stuck operation, review generated
text, and explicitly consolidate it. The solution must retain ThothAI's practical simplicity and
must not introduce a general job platform, model gateway, distributed scheduler, or competing
user-facing CLI.
## Solution
Add Description Generation to Database Management as one installation-wide, sequential background
run owned by the Fastify backend. The browser starts a run and remains responsive while the backend
processes bounded requests one at a time. Each completion is delegated to a short-lived internal
Python helper using LiteLLM. Models, their default, and any API-key secret references are declared in
application setup YAML and are independent of both workspaces and Pi configuration.
Each valid result is written immediately to the target's Generated Description. A minimal run row
and ordered text events provide status, counters, history, and a live log. A stopped or crashed run
is not resumed automatically; completed results remain in place and Generate Missing supplies the
simple recovery path. An Unlock action marks a stale recorded run interrupted only when no helper
or backend generation loop is alive.
Prompts use catalog context and, when available, no more than five real rows and five representative
non-null examples. Samples are transient and never logged or persisted. A valid inability to infer
a description produces a standard application-localized value such as `Non generabile`; provider,
timeout, and response-validation failures remain technical errors.
Generated text remains separate from Description until an administrator uses the existing
checkbox selection and Actions control to consolidate it. Consolidation retains Generated
Description and never writes comments to the external Workspace Database.
## User Stories
1. As an installation operator, I want to declare the models allowed for metadata generation in setup YAML, so that model availability is controlled centrally.
2. As an installation operator, I want to declare one default metadata-generation model, so that administrators begin with a safe operational choice.
3. As an installation operator, I want each model to reference its own API-key secret, so that credentials are not stored in workspaces or browser-visible settings.
4. As an installation operator, I want metadata-generation models to remain independent of Pi models, so that changing this workflow cannot disrupt the core NL-to-SQL experience.
5. As an installation operator, I want invalid model setup to fail validation clearly, so that the application does not start with ambiguous provider behavior.
6. As a Catalog Administrator, I want generation controls to explain when no model is configured, so that I know why the action is unavailable.
7. As a Catalog Administrator, I want to select an approved model from a selector initialized to the setup default, so that I control which model performs the work.
8. As a Catalog Administrator, I want to generate descriptions for selected Catalog Tables, so that I can work on a focused part of the catalog.
9. As a Catalog Administrator, I want to generate descriptions for selected Catalog Columns, so that I can work on individual fields without regenerating a whole table.
10. As a Catalog Administrator, I want to generate all eligible descriptions for a Workspace Database, so that I can initialize a catalog in one operation.
11. As a Catalog Administrator, I want to generate only missing descriptions, so that I can continue interrupted work without replacing completed proposals.
12. As a Catalog Administrator, I want a full run to process Catalog Columns before their Catalog Tables, so that table descriptions can benefit from column descriptions.
13. As a Catalog Administrator, I want generation to run asynchronously after I start it, so that the browser remains usable and progress is not tied to one HTTP request.
14. As a Catalog Administrator, I want only one Description Generation Run active in the installation, so that provider traffic and operational behavior remain predictable.
15. As a Catalog Administrator, I want a second start attempt to return a clear conflict, so that I cannot accidentally overlap generation runs.
16. As a Catalog Administrator, I want synchronization, cleanup, consolidation, and edits for the target Workspace Database blocked during generation, so that the simple sequential run sees stable catalog state.
17. As a Catalog Administrator, I want to see the run's model, scope, status, counters, and timestamps, so that I understand what is happening.
18. As a Catalog Administrator, I want a chronological text log, so that I can follow completed targets and diagnose errors.
19. As a Catalog Administrator, I want live log updates with a polling fallback, so that temporary SSE problems do not hide run progress.
20. As a Catalog Administrator, I want completed and interrupted runs to remain inspectable, so that I can understand prior activity.
21. As a Catalog Administrator, I want to stop an active run, so that I can halt an incorrect or unexpectedly costly operation.
22. As a Catalog Administrator, I want stopping a run to terminate its current model helper and prevent later targets from starting, so that stop has prompt operational effect.
23. As a Catalog Administrator, I want valid results completed before a stop or failure to remain saved, so that useful work is not discarded.
24. As a Catalog Administrator, I want a run left active by a backend restart to become interrupted, so that the UI does not claim nonexistent work is still running.
25. As a Catalog Administrator, I want to unlock a stale active run when no generation process is alive, so that an erroneous recorded lock cannot block future work.
26. As a Catalog Administrator, I want Unlock rejected while a live generation process exists, so that recovery cannot create an overlapping run.
27. As a Catalog Administrator, I want Generate Missing to continue after interruption, so that recovery does not require a special resume mechanism.
28. As a Catalog Administrator, I want one retry for a transient model failure, so that a brief provider fault does not immediately lose a batch.
29. As a Catalog Administrator, I want the run to fail after three consecutive technical failures, so that a broken provider does not generate an unbounded stream of attempts.
30. As a Catalog Administrator, I want a successful request to reset the consecutive-failure count, so that isolated errors do not prematurely stop a useful run.
31. As a Catalog Administrator, I want a completed-with-errors result when isolated batches fail but the run reaches its end, so that partial problems remain visible.
32. As a Catalog Administrator, I want no automatic fallback to a different model, so that the selected model remains truthful and predictable.
33. As a Catalog Administrator, I want malformed or ambiguous model output rejected without writing it, so that descriptions cannot be assigned to the wrong target.
34. As a Catalog Administrator, I want an inability to infer a description represented by standard localized text, so that every valid outcome is understandable in the workspace language.
35. As a Catalog Administrator, I want technical failures kept distinct from non-generatable outcomes, so that provider problems are not mistaken for catalog knowledge.
36. As a Catalog Administrator, I want generated prose written in the workspace language, so that it matches the catalog's intended audience.
37. As a Catalog Administrator, I want generated text stored separately from curated Description, so that AI output remains a reviewable proposal.
38. As a Catalog Administrator, I want to edit a Generated Description manually, so that I can improve a proposal before consolidation.
39. As a Catalog Administrator, I want to select one or more tables or columns and run “Move generated description to Description” from the existing Actions control, so that review remains integrated into the current grids.
40. As a Catalog Administrator, I want consolidation to retain the Generated Description, so that I can still see the proposal from which the curated text was copied.
41. As a Catalog Administrator, I want selected records without a Generated Description skipped and reported, so that the bulk action does not erase curated text.
42. As a Catalog Administrator, I want consolidation and generation to modify only the Metadata Catalog, so that no external database comment is changed.
43. As a Catalog Administrator, I want prompts to use schema facts and existing catalog text, so that generated descriptions are grounded in available metadata.
44. As a Catalog Administrator, I want prompts to use at most five real source rows and five representative values when available, so that the model has useful examples without unbounded disclosure.
45. As a Catalog Administrator, I want to be warned that real source samples are sent to the selected provider, so that I can make an informed disclosure decision.
46. As a Catalog Administrator, I want sampled rows and values excluded from persistence and logs, so that operational history does not become a secondary data store.
47. As a security operator, I want API keys, prompts, samples, and complete provider payloads redacted from logs, so that diagnostics do not leak secrets or source data.
48. As a support operator, I want concise per-target and per-batch event messages, so that failures can be diagnosed without provider-specific internals.
49. As an authorized administrator, I want all generation, cancellation, unlock, and consolidation actions protected by database-management permission, so that ordinary users cannot mutate catalog metadata.
50. As an unauthorized user, I want generation controls hidden or disabled and API calls rejected, so that frontend visibility is not treated as authorization.
51. As an operator, I want setup changes to take effect after an application restart, so that configuration lifecycle remains simple and explicit.
52. As a product owner, I want the first release to avoid queues, parallel calls, distributed locks, and automatic resume, so that effort remains focused on generating and reviewing useful descriptions.
## Implementation Decisions
- The Fastify backend owns one installation-wide Description Generation Run and its sequential
processing loop. It does not delegate lifecycle ownership to Pi or Python.
- A Description Generation Run has one of `queued`, `running`, `completed`,
`completed_with_errors`, `cancelled`, `failed`, or `interrupted`. It stores the Workspace
Database, requested scope, selected model identifier, workspace language, progress counters,
timestamps, and an optional final error summary.
- Ordered Description Generation Events store timestamp, severity, and safe human-readable text.
When an event identifies a target, it uses the object type and qualified physical name, such as
`Column "patients.birth_date"` or `Table "patients"`; catalog UUIDs remain internal identifiers.
No durable per-target jobs, model invocation rows, prompt snapshots, sample snapshots, leases,
heartbeats, registry revisions, or provenance chains are introduced.
- Starting a run schedules an in-process background loop and returns the run immediately. The API
exposes start, run/history lookup, event listing and streaming, cancellation, stale-run unlock,
and the safe list of configured model choices. There are no retry-item or resume endpoints.
- One in-memory generation manager enforces the installation-wide active-run rule. The existing
Catalog Operation Coordinator reserves the target Workspace Database for the duration of the
run, without being generalized into a new operation framework.
- On backend startup, persisted `queued` or `running` Description Generation Runs become
`interrupted`. The application performs no automatic replay or resume.
- Unlock succeeds only when no live generation loop or helper child exists. It marks the stale run
interrupted and releases the local reservation; it is not a distributed lock recovery protocol.
- Each model completion uses a short-lived Python helper backed by LiteLLM. Structured input is
supplied over stdin, structured output alone is emitted on stdout, diagnostics use stderr, and
the helper can be terminated by cancellation.
- A completion request contains no more than ten targets. Requests run one at a time. The helper
performs at most one retry for a transient technical failure.
- Three consecutive model-request failures fail the run. A successful request resets that count.
Isolated exhausted failures may be logged and skipped, producing `completed_with_errors` if the
run later reaches its end.
- Every valid generated or non-generatable result is applied immediately to Generated Description.
Earlier writes are retained after cancellation, interruption, or later failure.
- A response must identify requested targets unambiguously and classify each returned result as
generated or non-generatable. Duplicate, unknown, missing, or malformed mappings cause a
technical request failure and no result from that ambiguous response is applied.
- The parser also tolerates one JSON object enclosed by one complete `json` code fence, because
some supported models add that formatting despite the prompt. Any prose outside the fence,
multiple payloads, or malformed/ambiguous mappings remain invalid.
- The application supplies localized standard non-generatable text. Provider wording is not used
as the standard value, and technical errors never write that value.
- A full-database run generates eligible Catalog Columns before Catalog Tables. Generate Missing
excludes targets whose Generated Description is already non-empty; all-generation may replace
existing generated proposals only after the initiating action makes that scope explicit.
- Model choices are declared under a metadata-generation section in installation setup YAML. Each
choice has a stable identifier, display label, LiteLLM provider/model settings, optional endpoint
settings, and an optional environment-secret reference for its API key. The reference may be
omitted only when an explicit endpoint is configured for unauthenticated access. One identifier
is the default.
- An explicit endpoint may set `disableThinking: true`; the helper translates it only to the
Qwen-compatible chat-template switch needed to keep the response within the strict JSON contract.
- Metadata-generation setup is separate from application settings for Pi and from workspace
`llm_policy`. Raw keys never enter setup YAML, the catalog database, API responses, process
arguments, or event text. Configuration reload is restart-only.
- If setup defines no usable model, the safe model-list response is empty and the UI disables
generation with an explanation. The backend still rejects direct generation attempts.
- Prompt construction treats schema names, comments, descriptions, and values as untrusted data.
It requests output in the workspace language and separates instructions from catalog content.
- A request may contain up to five real source rows and up to five representative distinct,
non-null values for relevant columns. Inputs are bounded before prompt construction and are not
persisted or logged.
- The UI discloses that real data can be sent to the selected provider. A future Sensitive Data
Policy will classify values and exclude or anonymize protected data; that policy is not silently
approximated in this slice.
- The generation UI reuses Database Management's table and column selections, model selector,
Actions control, run drawer conventions, SSE delivery, and polling fallback where practical.
Visual parity with Catalog Sync Run logs is not required.
- The consolidation action copies each selected, non-empty Generated Description into Description
in a catalog transaction, retains Generated Description, skips empty proposals, and reports
copied and skipped counts. It never writes to the external Workspace Database.
- Generation, cancellation, unlock, and consolidation require the existing database-management
permission and are validated by the backend independently of UI state.
- No user-facing generation CLI is added. The Python process is an internal completion adapter,
not an operator surface or a long-lived service.
## Testing Decisions
- Tests assert externally observable behavior rather than private loop structure, process timing,
or LiteLLM implementation details.
- The primary and highest test seam is the Fastify catalog API with a test PostgreSQL catalog and
an injected fake Model Completer. It verifies complete paths through authorization, run
persistence, sequential processing, event delivery, Generated Description updates, and final
status without contacting a real provider.
- API tests cover each generation scope, column-before-table order, the ten-target request bound,
model validation, one-active-run conflict, target-database exclusion, cancellation, startup
interruption, Unlock safeguards, Generate Missing, partial success, consecutive failure
handling, non-generatable localization, malformed responses, redacted events, and permissions.
- Catalog repository integration tests verify the migration, run and event ordering, active-run
constraint, immediate description writes, history queries, startup interruption, and bulk
consolidation behavior against PostgreSQL.
- The Python helper has a small black-box contract suite using a simulated LiteLLM adapter. It
verifies stdin/stdout framing, pristine stdout, stderr diagnostics, normalized success and
failure output, one transient retry, secret redaction, and termination behavior.
- Setup-validation tests cover duplicate model identifiers, missing or unknown defaults, malformed
provider settings, missing secret references, safe public model projection, and strict separation
from Pi and workspace model settings.
- Database Management tests use the existing browser-level component seam with MSW. They verify
model selection and default, selected/all/missing actions, disabled state without models, running
progress and logs, polling recovery, cancellation, Unlock visibility, terminal summaries,
generated-text refresh, and selected consolidation with copied/skipped counts.
- Existing Catalog Sync Run route, repository, SSE, and drawer tests are prior art for asynchronous
status and event behavior. Existing catalog table/column editing and Database Management tests
are prior art for optimistic catalog updates, permissions, selection, and action controls.
- One required manual acceptance gate, outside deterministic CI, uses the installation's configured
default model and a disposable PostgreSQL database containing only invented data. Its application
credentials are read-only. It generates Italian text for one Catalog Column and one Catalog
Table, verifies their Generated Description, inspects the safe activity log, confirms that no key
or sample value is exposed, and consolidates one selected result. If the configured secret is not
available, acceptance stops without exposing or requesting the key in conversation.
- Delivery includes a strict MkDocs build executed through repository-managed, reproducible
documentation dependencies rather than globally installed Python packages. A readable direct
dependency file is retained, a complete transitive lock is generated with `uv`, and one canonical
repository command performs the strict build from that lock.
- Successful real-provider acceptance is recorded in a short sanitized report under
`docs/testing/`. It identifies the model and checks performed but contains no credentials,
prompts, source samples, complete provider payloads, or generated database values.
- No tests are added for worker queues, parallel generation, distributed locking, multi-replica
recovery, automatic resume, cost accounting, or model fallback because those behaviors are out
of scope.
## Out of Scope
- Reusing Pi to execute Description Generation or changing Pi's model configuration.
- A shared Installation Model Registry, model gateway, long-lived Python sidecar, or provider
management platform.
- A user-facing generation CLI.
- Parallel model calls, worker queues, adaptive rate limiting, distributed locks, leases,
heartbeats, automatic resume, or multi-replica execution.
- Durable target jobs, invocation history, prompts, samples, token usage, cost accounting,
provenance chains, target snapshots, or advanced retention controls.
- Automatic retry or resume of individual targets beyond one technical helper retry and a new
Generate Missing run.
- Automatic fallback to a different model.
- Writing generated text into comments of the external Workspace Database.
- Generating logical relationships or other catalog metadata beyond Catalog Table and Catalog
Column descriptions.
- Implementing the Sensitive Data Policy. Its definition and exclusion/anonymization behavior are
a required follow-up improvement.
- Generalizing the log viewer across unrelated metadata operations. That broader concern remains
related to Gitea issue #2.
## Further Notes
- The design deliberately follows ThothAI's proven simple workflow while adapting it to ThothII's
asynchronous browser interaction, setup ownership, and existing Generated Description model.
- The source-sampling disclosure is a release requirement, not merely documentation for operators.
- `completed_with_errors` is reserved for a run that reaches the end after isolated technical
failures. Three consecutive failures end the run as `failed`.
- Successful values are their own recovery record: after interruption, Generate Missing naturally
skips them without needing replay state.
- The implementation is available without a feature flag once the catalog migration and valid
setup are present. With no configured model, the feature remains visibly unavailable rather than
partially initialized.
- The final manual gate is intentionally narrow: one real-provider run covers one Catalog Column
and one Catalog Table, generated Italian text, safe events, and one consolidation. Automated
tests remain the evidence for All, Missing, Stop, restart interruption, Unlock, and failure paths.

Some files were not shown because too many files have changed in this diff Show More