Compare commits

...
Author SHA1 Message Date
Codex 6a4634dcf1 Merge manual standalone installation documentation
Publish documentation / publish (push) Successful in 35s
2026-09-15 10:06:33 +02:00
Codex 84804be9f8 docs: publish bilingual manual standalone installation guides 2026-09-15 10:06:28 +02:00
User c3caba94dd fix(ui): expand session dialogs and repeat confirmation actions
Publish documentation / publish (push) Successful in 24s
2026-09-14 18:13:53 +02:00
User b1723c34c4 docs: record server rollout of session and memory fixes
Publish documentation / publish (push) Successful in 32s
2026-09-14 17:25:23 +02:00
User d6cdffea62 fix: keep embedded session controls visible and handle empty memory
Publish documentation / publish (push) Successful in 34s
Cap the embedded shell at its portal container height so steering and stop controls remain accessible. Skip vector retrieval for an empty authoritative Memory archive and compute SQL-rule embeddings lazily.

Validated with 54 Memory tests, 90 frontend tests, five browser scenarios, frontend and Docker builds, and a read-only comparison against the real empty Memory archive.
2026-09-14 17:15:00 +02:00
Codex 49333a2d35 Merge full and embedded shell, administration UI and server handoff
Publish documentation / publish (push) Successful in 1m21s
2026-09-14 15:08:47 +02:00
Codex b006b94479 docs: prepare server Codex deployment handoff for ThothII and Omics 2026-09-14 15:08:46 +02:00
Codex bdcd8fcd28 fix(ui): open one session accordion panel at a time 2026-09-14 01:09:45 +02:00
Codex cf90c1bd51 fix(ui): collapse session lists and scope selection controls to panels 2026-09-13 18:12:27 +02:00
Codex 571a4bcaa2 fix(ui): show workspace readiness dot and bounded session accordions 2026-09-13 17:39:33 +02:00
Codex 9051463654 docs: document full and embedded rendering with server authentication 2026-09-13 17:28:10 +02:00
Codex 26c5605ff7 fix(ui): unify Memory and Evidence reading typography 2026-09-13 17:07:37 +02:00
Codex 3535fda958 fix(ui): improve knowledge reading and add isolated formatting examples 2026-09-13 16:52:53 +02:00
Codex 2953f6b608 fix(ui): unify Session navigation and restore uniform tab borders 2026-09-13 16:22:20 +02:00
Codex 45db3a239b fix(ui): simplify login and suppress pointer focus ring on locale select 2026-09-13 15:57:57 +02:00
Codex 7d826e46c0 fix(ui): match Omics header and compact workspace layout 2026-09-13 15:36:08 +02:00
Codex 648434a32e docs: record approved Omics GitHub to PSD relay handoff 2026-09-13 15:22:30 +02:00
Codex 023b822f83 Merge visual review into full shell and preserve bilingual layout 2026-09-13 14:55:58 +02:00
Codex d8a29bfbdd Add full shell, replaceable Omics adapter and bilingual interaction
Implement approved specification #32 and tickets #33-#37. Keep host authentication server-verified and pin session interaction language. Compile scoped base selectors for browser compatibility and retain full gutters during CSS pruning.
2026-09-13 14:26:39 +02:00
Codex a59624a68f style: align global context panel and simplify selection copy 2026-09-13 10:43:42 +02:00
Codex 5af4408194 style: unify database block gutters and content alignment 2026-09-13 10:32:55 +02:00
Codex e088abd60a style: align catalog status with summary grid 2026-09-13 01:44:48 +02:00
Codex 803e9e9201 fix: remeasure composer after hidden Core becomes visible 2026-09-13 01:39:19 +02:00
Codex 8c81996896 style: place catalog status indicators after their labels 2026-09-13 01:23:46 +02:00
Codex eed398e569 style: restore prominent ThothII application wordmarks 2026-09-13 01:18:06 +02:00
Codex c8d276ddc6 style: unify workbench typography and prepare isolated visual review 2026-09-12 22:51:58 +02:00
Codex 2d1b714ebe fix: resolve admin issue review findings and record verification 2026-09-12 18:24:23 +02:00
Codex c7e5f295e6 fix: address administration layout and navigation issues #28 #29 #30 #31 2026-09-12 18:16:52 +02:00
Codex f52bf22e05 feat: establish unified administration and model context baseline 2026-09-12 18:03:15 +02:00
Codex 840344706f prototype: restore original Core within context shelf alternatives 2026-09-12 14:00:52 +02:00
Codex 41b9fed4d5 prototype: refine context shelf with remembered defaults and session tabs 2026-09-12 12:29:02 +02:00
Codex 36bf659ea9 prototype: explore global context with one operation at a time 2026-09-12 11:53:52 +02:00
Codex 4a67d60233 prototype: revisit five administration workflows after design review 2026-09-10 19:40:21 +02:00
Codex debb63d87b prototype unified administration page layouts 2026-09-10 16:48:25 +02:00
Codex 3943022a97 Merge remote-tracking branch 'origin/main'
# Conflicts:
#	mkdocs.yml
2026-09-10 12:57:43 +02:00
Codex f5ec2d9313 docs: track security evidence and research notes 2026-09-10 12:53:13 +02:00
Codex 82e2c91f42 feat: implement memory and evidence administration with guided repairs
Publish documentation / publish (push) Successful in 1m27s
Add PostgreSQL-backed memory, editable evidence with source review and activation, and human-approved archive repairs across the harness, API, and UI. Include migrations, deployment support, regression coverage, and validation documentation.

Refresh permissions from validated session roles so existing administrator logins can access newly deployed archive management features.
2026-09-10 10:31:34 +02:00
User 8fe526dd6e fix(frontend): show session catalog in pi management 2026-09-08 14:58:18 +02:00
User e68e80a33d fix(frontend): prevent catalog header overlap 2026-09-08 14:35:36 +02:00
Codex 818563c408 fix(core): keep workspace runtime available 2026-09-08 13:37:10 +02:00
Codex 50c546e42d fix(frontend): accept catalog-owned workspace descriptors 2026-09-07 15:01:31 +02:00
Codex 651a5c7902 fix(frontend): isolate administration rail from portal CSS 2026-09-07 11:05:38 +02:00
493 changed files with 32289 additions and 4685 deletions
+1
View File
@@ -17,6 +17,7 @@ frontend/vite.database-management-prototype.config.ts
!deploy/env/*.env.example
deploy/thothii.env
deploy/secrets/
deploy/psd/
harness/workspaces/*.yaml
!harness/workspaces/local.yaml
!harness/workspaces/tht.example.yaml
+1
View File
@@ -46,6 +46,7 @@ deploy/secrets/*
# Per-installation configuration generated by `tht setup` (examples stay tracked).
deploy/*/thothii-installation.yaml
deploy/*/operator.env
deploy/*/auth/
deploy/*/generated/
deploy/*/secrets/*
!deploy/*/secrets/.gitkeep
+12 -2
View File
@@ -98,8 +98,18 @@ frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness →
- **`tht`'s `-c`/`--config` is a PER-COMMAND option** — it must follow the subcommand, never
precede it (`ThtRunner.buildArgv` enforces this; prepending caused live 500s).
- **`--json` output must be pristine** (only valid JSON on stdout) — used as a machine contract.
- **UI strings are English; document *content* stays the workspace language** (Italian for
`psd`) because it's the real data. Only chrome/labels are English.
- **Localization:** deterministic UI uses the EN/IT catalogs with English fallback;
model interaction uses the session manifest's immutable `interaction_language`.
Workspace content, SQL, and identifiers remain unchanged. For shell modes, portal
integration, or translations, read `docs/operations/shell-and-localization.md`.
- **Server deployment:** for the coordinated ThothII/Omics upgrade, follow
`docs/operations/server-codex-handoff.md`; it supersedes earlier Omics delivery
instructions. Omics source integration uses GitHub with no repository relay prerequisite.
- **Server identity:** for portal login/logout, proxy headers, or Omics deploy, read
`docs/install/authentication-upstream.md` before changing authentication. Omics
uses embedded/upstream, not a second ThothII OIDC login. Full/embedded rendering
is documented in `docs/architecture/application-shell.md`; release acceptance
is in `docs/testing/authentication-manual-acceptance.md`.
- **Workspace schema v4** defines workspace identity and optional Evidence only. PostgreSQL Metadata
Catalog owns database identity, binding, schema, descriptions, sensitivity, and relationships;
embedding/model facts come from the installation catalog. The legacy `harness/workspaces/*.yaml` runtime snapshots still use
+147 -12
View File
@@ -91,21 +91,79 @@ correzione successiva crea una nuova sessione derivata, collegata a quella prece
dopo la finalizzazione. Non può modificare il ledger, gli artifact canonici o lo stato
terminale della sessione.
## Memory
**Memory Module** — Il modulo che possiede le conoscenze ed esperienze curate per
migliorare schema linking e generazione SQL di domande future. Le Memory appartengono
a un workspace e rimangono distinte dalle Evidence.
**Memory Card** — L'unità di contenuto gestibile del Memory Module, con identità,
ambito di applicazione e provenienza. Il formato è allineato per analogia alle
Evidence, senza implicare la stessa origine o lo stesso percorso di pubblicazione.
**Reusable Memory** — Una Memory Card che esprime un chiarimento di dominio, una
regola di costruzione SQL o un errore da evitare con motivo compreso e approvato.
La sua validità è circoscritta a un ambito esplicito e non deriva dalla sola
approvazione di una scelta occasionale in una domanda.
**Solved Question** — Una Memory Card che conserva una domanda risolta con la
relativa soluzione SQL e il contesto necessario a interpretarla. È un exemplar
consultativo: i parametri e le scelte del caso non diventano regole generali.
**Memory Graph** — L'insieme dei collegamenti espliciti fra card che contribuisce
al recupero di conoscenze pertinenti oltre alla somiglianza del contenuto. Il
ritrovamento di una card tramite un collegamento non ne implica l'approvazione.
**Memory Link** — Un collegamento curato fra card, con destinazione e significato
espliciti, che contribuisce alla consultazione di contenuti pertinenti. La sua
rimozione non comporta la cancellazione delle card collegate.
## Evidence
**Context specialist** — La persona competente sul dominio che redige e cura il
contenuto delle Evidence. Può essere distinta da chi amministra l'installazione;
il suo lavoro di redazione non richiede accesso al database applicativo.
**Evidence draft** — Il documento iniziale scritto dallo specialista di contesto,
che il sistema acquisisce e raffina in Evidence Unit. Può essere redatto e
consegnato indipendentemente dall'installazione che userà le Evidence risultanti.
**Evidence Module** — Il modulo autonomo che possiede la preparazione delle Evidence e
la loro consultazione durante il workflow. La preparazione avviene fuori dalle singole
sessioni; il workflow usa soltanto contenuti già pubblicati. A runtime contribuisce agli
stage semantici esistenti, senza diventare uno stage visibile e senza modificare ledger,
artifact o stato del workflow.
**Source Evidence** — Un documento originale del workspace, conservato senza modifiche
come riferimento umano e origine della successiva ristrutturazione.
**Source Evidence** — Il documento o la dichiarazione che sostiene il contenuto
corrente di una Evidence Unit. Un documento acquisito viene conservato come
riferimento umano; una dichiarazione manuale attribuisce il contenuto alla persona
che lo ha scritto e approvato.
**Manual Evidence declaration** — Una dichiarazione esplicita dell'amministratore
che sostiene una Evidence creata direttamente o una correzione del suo significato.
Non implica una verifica indipendente da parte di una fonte documentale esterna.
**Evidence origin** — Il documento da cui una Evidence Unit è stata inizialmente
derivata. Può restare collegato per provenienza e confronto con gli aggiornamenti
anche quando una dichiarazione manuale sostiene il testo corrente. La sola origine
non dimostra il supporto semantico di una successiva correzione.
**Local Evidence archive** — L'insieme delle Evidence curate custodite
dall'installazione, distinto dalle draft originali e dai contenuti derivati per
la ricerca. Comprende le correzioni manuali e i ritiri deliberati.
**Consolidated Evidence** — Una versione delle Evidence locali controllata come
insieme coerente e pronta per l'attivazione. I file ancora in modifica non ne
cambiano il contenuto.
**Active Evidence** — La versione consolidata disponibile alla consultazione del
core. Un tentativo di aggiornamento fallito conserva la versione attiva precedente.
**Evidence Unit** — La più piccola unità semantica coerente, revisionabile e ricercabile
derivata da una sola Source Evidence. Possiede un identificatore stabile indipendente
dal kind, assegnato una volta nella forma `evidence:<slug>`; fonti diverse non vengono
fuse automaticamente.
fondata su una Source Evidence corrente, anche manuale, e con eventuale origine
documentale distinta. Possiede un identificatore stabile indipendente dal kind,
assegnato una volta nella forma `evidence:<slug>`; fonti diverse non vengono fuse
automaticamente.
**Evidence kind** — La categoria semantica di una Evidence Unit, che ne determina i
campi specifici e ne orienta l'uso. Ogni unità ha un solo kind primario; i tipi iniziali
@@ -151,10 +209,9 @@ avanzare fino a un retry riuscito.
nella sessione: stage semantico, purpose, generazione interrogata e identificatori delle
Evidence restituite. Non duplica il contenuto delle Evidence.
**Curated Evidence** — Una o più Evidence Unit ristrutturate a partire da una Source
Evidence e conservate nel repository del workspace come proposte per la revisione
umana. Git conserva la versione precedente e rende visibile ogni modifica; una Curated
Evidence non è ancora contenuto autorevole del runtime.
**Curated Evidence** — Una o più Evidence Unit preparate da documenti o curate
manualmente. La presenza nell'archivio curato non implica da sola che il contenuto
sia già attivo per il workflow.
**Published Evidence** — Le Curated Evidence valide appartenenti alla revisione attiva
del workspace e alla generazione Evidence pubblicata. L'approvazione umana precede
@@ -176,9 +233,17 @@ una Evidence Unit. Il sistema ne verifica deterministicamente la presenza dopo l
normalizzazione meccanica; il curatore resta responsabile di verificarne la sufficienza
semantica.
**Evidence resolution** — L'operazione esplicita con cui un curatore ritira una
Evidence Unit oppure la ricollega a un Source Evidence esistente. Aggiorna documento e
manifest insieme, lascia un diff Git revisionabile e non pubblica né crea commit.
**Evidence resolution** — La decisione esplicita con cui un curatore risolve un
problema di una Evidence Unit, correggendola, ritirandola oppure ricollegandola a
una fonte adeguata.
**Source update conflict** — Un contrasto fra una fonte aggiornata e una correzione
manuale già approvata. La correzione resta in uso fino alla risoluzione esplicita
del confronto da parte dell'amministratore.
**Evidence source refresh** — La riacquisizione delle fonti esterne richiesta
dall'amministratore per rilevarne le modifiche. Fra due aggiornamenti il contenuto
già acquisito resta il riferimento per preparazione e consultazione.
**Review item** — Un blocco di revisione descritto da codice stabile, messaggio umano e
campo opzionale. Finché viene mantenuto nell'Evidence Unit, ne impedisce la
@@ -506,3 +571,73 @@ _Avoid_: Sensitive Data Suggestion Event
**Introspection Capability** — Una categoria di struttura fisica che una Database Binding
può osservare, come tabelle, colonne, relazioni, indici o enum. Una capability non disponibile
è distinta da una capability osservata che non ha restituito elementi.
## Amministrazione e integrazione
**Workspace Readiness** — La preparazione di uno specifico Workspace per l'uso nel
workflow, comprensiva della disponibilità degli artefatti derivati dai suoi metadati
Database e dalle sue Evidence. Il preprocessing appartiene a questa preparazione;
la configurazione e la sincronizzazione del catalogo restano responsabilità Database.
**Administration Surface** — Una superficie amministrativa autonoma per configurare o curare una
parte dell'installazione. Workspace, Evidence, Memory, Database e Pi sono superfici peer e non
dipendono dall'esistenza di una sessione attiva.
**Administration Page** — La rappresentazione a pagina intera di una Administration Surface, con
una gerarchia condivisa per identità, stato, azioni e contenuto. Un form amministrativo appartiene
alla pagina e non a una popup come contenitore principale.
_Avoid_: management popup, settings modal
**Administration Route** — L'identità navigabile di una Administration Surface nel browser. Deve
essere ripristinabile con refresh e cronologia e non contiene valori transitori o segreti dei form.
**Embedded Thoth Shell** — L'esperienza Thoth ospitata dentro il documento e il contesto visuale di
un portale host. Conserva la propria gerarchia funzionale, ma deve rispettare la geometria,
l'autenticazione e le regole responsive del portale host.
**Full Thoth Shell** — L'esperienza Thoth autonoma che possiede il proprio header e il proprio
layout di pagina. Non replica la navigazione amministrativa del portale host e non dipende dal suo
template visuale.
**Shell mode** — La scelta di installazione fra `embedded` e `full`. Determina chi possiede il
chrome globale, i comandi di identità e le integrazioni visuali, ma non cambia il workflow o la
persistenza delle sessioni.
**Fullscreen state** — Lo stato temporaneo in cui il documento applicativo occupa il fullscreen
del browser. È distinto da `Shell mode`: una Full Thoth Shell può essere aperta senza fullscreen;
il passaggio è attivato da un comando esplicito e può essere annullato con la stessa azione o con
il comando nativo del browser.
**Portal Shell Adapter** — Il confine sostituibile che traduce lo stato e i comandi del chrome di
un portale host nel modello semantico usato da Thoth. L'adapter non possiede autorizzazione,
sessioni di workflow o contenuti del modello.
**Host Shell State** — Il minimo stato visuale fornito dal portale host: locale UI, tema e stato
fullscreen. In una Embedded Thoth Shell è la fonte autorevole per queste preferenze;
non include identità, token o stato di autenticazione, che restano responsabilità dell'accesso.
**UI locale** — La lingua delle label, dei messaggi, dei tooltip, degli stati e delle istruzioni
non generate dal modello nell'interfaccia Thoth. È distinta dalla lingua dei contenuti di un
workspace.
**Interaction language** — La lingua in cui il modello presenta domande, spiegazioni e proposte
al revisore durante una sessione. Viene fissata alla creazione della sessione e rimane invariata
durante una ripresa, anche se la UI locale corrente cambia.
**Administrative Page Family** — L'insieme delle cinque Administration Page che condividono shell,
navigazione, tipografia e regole responsive, pur mantenendo contenuti e operazioni specifici:
Workspace, Evidence, Memory, Database e Pi.
## Installazione
**Manual standalone installation** — Una copia di ThothII predisposta per l'uso autonomo da una
persona che possiede il computer, con una Full Thoth Shell e servizi applicativi locali. La
procedura non implica che DWH o provider LLM siano locali o disponibili offline.
**Installation bootstrap** — L'insieme delle attività iniziali che rende disponibile una
installazione manuale: verifica dell'host, generazione della configurazione, predisposizione
delle credenziali protette e avvio dei servizi. Non è un installer dell'applicazione.
**Platform acceptance** — La verifica che una Manual standalone installation possa essere
predisposta e avviata su una specifica combinazione di sistema operativo, architettura e runtime,
distinta dalla verifica funzionale del collegamento a DWH e provider LLM.
+146 -44
View File
@@ -11,50 +11,50 @@ colors:
warm-graphite: "oklch(26.78% 0.0097 355.6)"
muted-graphite: "oklch(51.33% 0.0088 345.6)"
quiet-border: "oklch(90.93% 0.0035 354.7)"
success-mint: "oklch(75.77% 0.1581 165)"
success-mint: "oklch(46% 0.095 160)"
navigation-active: "oklch(92.5% 0.052 23.2)"
navigation-active-hover: "oklch(89.5% 0.071 23.2)"
navigation-active-foreground: "oklch(36.5% 0.11 23.2)"
navigation-active-border: "oklch(60% 0.135 23.2)"
warning-amber: "oklch(85.23% 0.1386 78.9)"
information-blue: "oklch(70.35% 0.1128 221.3)"
warning-amber: "oklch(48% 0.09 70)"
information-neutral: "oklch(51.33% 0.0088 345.6)"
typography:
display:
fontFamily: "Fraunces, Source Serif Pro, Georgia, Times New Roman, serif"
fontSize: "3rem"
fontFamily: "Manrope Variable, Manrope, system-ui, sans-serif"
fontSize: "1.5rem"
fontWeight: 600
lineHeight: 1.03
letterSpacing: "-0.025em"
headline:
fontFamily: "Fraunces, Source Serif Pro, Georgia, Times New Roman, serif"
fontSize: "1.875rem"
fontFamily: "Manrope Variable, Manrope, system-ui, sans-serif"
fontSize: "1.5rem"
fontWeight: 600
lineHeight: 1.15
letterSpacing: "-0.015em"
title:
fontFamily: "Fraunces, Source Serif Pro, Georgia, Times New Roman, serif"
fontSize: "1.2rem"
fontFamily: "Manrope Variable, Manrope, system-ui, sans-serif"
fontSize: "1.25rem"
fontWeight: 600
lineHeight: 1.25
letterSpacing: "-0.01em"
body:
fontFamily: "Manrope, -apple-system, BlinkMacSystemFont, Segoe UI, system-ui, Arial, sans-serif"
fontSize: "0.9375rem"
fontFamily: "Manrope Variable, Manrope, -apple-system, BlinkMacSystemFont, Segoe UI, system-ui, Arial, sans-serif"
fontSize: "1rem"
fontWeight: 400
lineHeight: 1.65
letterSpacing: "normal"
control:
fontFamily: "Manrope, -apple-system, BlinkMacSystemFont, Segoe UI, system-ui, Arial, sans-serif"
fontFamily: "Manrope Variable, Manrope, -apple-system, BlinkMacSystemFont, Segoe UI, system-ui, Arial, sans-serif"
fontSize: "0.875rem"
fontWeight: 600
lineHeight: 1.25
letterSpacing: "0.005em"
label:
fontFamily: "ui-monospace, SF Mono, Cascadia Code, Menlo, Consolas, monospace"
fontSize: "0.6875rem"
fontFamily: "Manrope Variable, Manrope, system-ui, sans-serif"
fontSize: "0.75rem"
fontWeight: 600
lineHeight: 1.25
letterSpacing: "0.06em"
letterSpacing: "normal"
rounded:
xs: "4px"
sm: "6px"
@@ -113,6 +113,16 @@ components:
# Design System: ThothII
## Visual review branch, September 2026
The revision on `codex/ui-visual-review` is approved for implementation and Docker visual review,
not yet for adoption on `main`. The previous look remains recoverable from the base commit and
the preserved Docker image. Historical prototypes must remain untouched.
This revision follows Impeccable's product register: one locally bundled Manrope family for the
whole UI, five fixed size roles, red as the sole brand accent and additional color only for meaningful
state. The primary scene remains an analyst reading data and SQL in a well-lit office.
## Overview
**Creative North Star: "The Clinical Workbench"**
@@ -134,7 +144,7 @@ disciplined and tactile, never playful, sluggish, or visually unstable.
**Key Characteristics:**
- Warm, restrained surfaces with one scarce red accent.
- Editorial headings paired with highly legible operational body text.
- One sans-serif family, with hierarchy expressed through size, weight and spacing.
- Dense information organized through hierarchy, rhythm, and progressive disclosure.
- Persisted artifacts and reviewer decisions presented as the visual source of truth.
- Fast state feedback with reduced-motion parity.
@@ -150,6 +160,15 @@ users can scan structure without adding nested containers.
## Colors
The full-mode application header matches Omics Portal's `--gsd-red-primary`
(`#CB333B`) in both themes. Its complete wordmark, including `II`, and controls
use a near-white foreground. This header is absent in embedded mode. The sidebar
and welcome wordmarks retain their red suffix. Context editing places workspace,
model and Done in one desktop row, stacking on narrow containers. Session-scope
tabs retain their selected fill and accessible keyboard state with a uniform one-pixel
border on every side, gray when inactive and red when active. Their padding is 11px
horizontal and 3px vertical, with a 38px minimum height and wrapping labels.
The palette combines warm porcelain surfaces, warm graphite text, and an instrument red used only
for action, focus, and important state. OKLCH values in the frontmatter are normative because the
frontend uses OKLCH tokens directly.
@@ -178,7 +197,8 @@ frontend uses OKLCH tokens directly.
foreground. It shares Instrument Red's hue but uses a lighter, lower-chroma fill, so location is
visible without carrying the full weight of a primary action.
- **Warning Amber** (`warning-amber`): waiting, attention, and in-progress states.
- **Information Blue** (`information-blue`): informational state when red would imply action.
- **Information**: neutral text and indicators for dates, protocols and ordinary status. The legacy
`--info` token resolves to muted foreground, not an additional blue accent.
The dark theme keeps the same semantic mapping with neutral near-black surfaces and a slightly
lighter red accent. Do not introduce a second visual identity for dark mode.
@@ -191,30 +211,33 @@ for their named states. Color is never the only state indicator.
## Typography
**Display Font:** Fraunces, with Source Serif Pro, Georgia, and Times New Roman fallbacks
**Body Font:** Manrope, with native system sans-serif fallbacks
**Label/Mono Font:** SF Mono or Cascadia Code, with Menlo and Consolas fallbacks
**UI Font:** locally bundled Manrope Variable, with Manrope and native sans-serif fallbacks.
**Technical Font:** SF Mono or Cascadia Code, with Menlo and Consolas fallbacks.
**Character:** Fraunces gives persisted artifacts and key headings editorial authority. Manrope
keeps dense controls and prose calm and readable. The mono register separates machine identity,
metadata, SQL, identifiers, and micro-labels from natural-language content.
Manrope covers headings, labels, controls, navigation and document reading. Monospace is reserved
for SQL, code, paths and machine identifiers, never for ordinary UI labels or status headings.
### Hierarchy
- **Display** (600, `3rem`, `1.03`): authentication and exceptional page-level statements only.
- **Headline** (600, `1.875rem`, `1.15`): major page or artifact titles.
- **Title** (600, `1.2rem`, `1.25`): panel and document section hierarchy.
- **Body** (400, `0.9375rem`, `1.65`): operational prose, with a target line length of 65 to 75
- **Headline** (600, `1.5rem`, `1.3`): page or artifact titles, `--text-page`.
- **Title** (600, `1.25rem`, `1.4`): section hierarchy, `--text-section`.
- **Body** (400, `1rem`, `1.6`): operational prose, `--text-body`, with a target line length of 65 to 75
characters where the surface controls width.
- **Control** (600, `0.875rem`, `1.25`): buttons, inputs, tabs, and compact actions.
- **Label** (600, `0.6875rem`, `0.06em` tracking): uppercase micro-labels, state metadata, and panel
headers. Labels use the mono family.
- **Control** (400–600, `0.875rem`, `1.5`): buttons, inputs, tables, tabs and compact subheadings,
`--text-control`.
- **Metadata** (400–600, `0.75rem`, `1.5`): secondary status, counts and timestamps, `--text-meta`.
Labels use sentence case and normal tracking. Ordinary operational text never falls below 12px.
Typography uses fixed sizes. Responsive changes happen at structural breakpoints, not through fluid
type scaling. Numeric data and identifiers use tabular numerals where comparison matters.
**The Three Registers Rule.** Serif means authority, sans means interaction and reading, mono means
machine identity. Do not exchange these roles for novelty.
**Application wordmark:** ThothII is a brand mark, not a page title: use Manrope semibold at
48px (`3rem`) in the Core welcome area and 32px (`2rem`) in the session sidebar, with the
`II` suffix in brand red. Preserve these sizes across responsive layouts.
**The One Family Rule.** The UI and document readers use sans-serif throughout. The legacy
`--font-heading` alias resolves to `--font-sans`. Preserve technical monospace without turning it
into a second decorative hierarchy. Do not shrink text to solve layout constraints.
**The Read Once Rule.** A heading, label, and body must be distinguishable on first glance through
size and weight. Do not repeat headings in explanatory copy.
@@ -279,27 +302,70 @@ default, hover, focus, active, disabled, loading, and error behavior where those
- **Focus:** three-pixel Instrument Red ring with a clear border shift.
- **Error / Disabled:** errors combine destructive color with explanatory text; disabled controls
retain readable contrast and use 50 percent opacity.
- **Metadata catalog model:** Database Management keeps one compact, installation-level
metadata-generation LLM selector in the application header. The selection persists across
database, table, column, and relationship views; when no usable profile is configured, the
disabled control explains: “No metadata-generation LLM model is configured for this installation.”
- **Global context:** the collapsible top shelf is the sole workspace/model selector for Core and
Admin. Preserve independent remembered choices, installation defaults, operation locks and unsaved
edit guards. Never introduce a separate metadata-generation default or selector.
### Navigation
- **Workspace readiness:** the Workspace navigation button carries an 8px dot to
the right of its label. Green means a selected workspace with confirmed ready
preprocessing and no query error; all other states are red. The button's
tooltip and accessible description retain the translated exact state. Do not
add a separate readiness text row or change the backend readiness gate.
- **Session groups:** one accessible single-open accordion contains Active sessions
and Archive, both initially closed. Below the scope tabs, show only their
adjacent section headers, without a redundant Sessions heading. Selection and
bulk-delete controls belong inside each panel and only appear for nonempty
lists. Select all affects that list only, preserves the other list's selection,
and exposes a mixed state for partial selection. Preserve the existing archived
flag as the grouping rule, independent of whether a Pi process is running.
Opening a section closes the other; either can be collapsed, including both.
Empty lists show only the translated "No sessions yet." message.
The open section uses the rail's remaining height; its list scrolls internally
with a cap of `min(18rem, 35dvh)`, while its trigger remains outside that scroll
area. The mobile navigation dialog supplies a bounded viewport-height container.
Keyboard users can focus and scroll each labelled panel.
- **Session entry:** one Session button returns to the current unfinished session,
including provisional creation, without resetting or reconnecting it. Otherwise
it prepares a new question using the normal readiness and unsaved-work guards.
- **Style:** compact session rows use `8px` corners and restrained vertical padding.
- **Default / Hover / Active:** porcelain at rest, Sunken Surface on hover, and a muted Navigation
Active red with a defined border when current. Exactly one top-level navigation control is current.
- **Administrative controls:** the admin-only Administration accordion groups Database management,
a structural divider, Workspace management, and Pi management in that order. Its trigger exposes
- **Administrative controls:** the admin-only Administration accordion groups Database,
Memory, Evidence, a structural divider, Workspace, and Pi configuration in that order. Its trigger exposes
expanded state and starts collapsed by default, while non-admin users do not receive the accordion
or its navigation actions.
- **Responsive:** collapse navigation structurally at the application breakpoint. Do not shrink
labels into illegibility.
labels into illegibility. Below 768px, Memory and Evidence management use the full content
width; a Navigation button opens the shared accessible dialog. Selecting another archive
page or pressing Escape closes it. Desktop retains the right session sidebar and its My sessions /
All sessions tabs. Core retains question/answer, eight phases, reviewer gates and the left log.
In embedded mode the portal owns the red header and left sidebar; ThothII must not duplicate them. Size to the
actual application container. Narrow session document panels may use the available width.
### Session review and confirmations
Session dialogs use the visible application area, including the portal's header
and side rail. Artifact and schema-column review can grow to 80rem wide and the
available height; short confirmations use up to 40rem and at least 18rem when
space permits. Keep a 24px outer margin on desktop and 8px on small or short
screens. Long review content scrolls internally; on very short screens the
whole dialog can also scroll so every action remains reachable.
Session forms and review gates repeat their existing primary confirmation above
and below the content, sharing selection, validation, pending state and response
handlers. Alternate-response inputs follow the same rule. Reserved navigation
controls remain below the review. Stop/delete initially focus Cancel; rename
initially focuses the name field. Administration dialogs and forms retain their
existing layout and actions.
### Tabs
- **Shape:** compact label tabs sit on a shared baseline with rounded top corners and a two-pixel
lower edge. Inactive labels retain a complete Quiet Border and Porcelain Card surface, so every
lower edge, except session-scope tabs which use a uniform one-pixel border, rounded
corners and a 4px gap without a shared border or negative bottom margin.
Inactive labels retain a Quiet Border and Porcelain Card surface, so every
label reads as a tab before interaction; hover feedback reinforces clickability.
- **Current:** the selected tab uses the muted Navigation Active red for its fill, text, and defined border.
It must expose `aria-selected`, participate in a labelled `tablist`/`tabpanel`, and be the only
@@ -319,9 +385,42 @@ default, hover, focus, active, disabled, loading, and error behavior where those
### Curated Evidence Documents
Memory and Evidence share the `thot-knowledge-reader` reading contract. Use locally
bundled Manrope with normal tracking for prose and labels, and these fixed roles:
- Card title: 24px, weight 600, line-height 1.3 (`thot-knowledge-title`).
- Field/section heading, including Scope and Provenance: 20px, weight 600,
line-height 1.4, 8px clearance below (`thot-knowledge-heading`).
- All narrative text, including scope, lists and provenance: 16px, weight 400,
line-height 1.65. Do not apply compact UI text sizes to these fields.
- Authored Markdown subheadings inside a field: 16px, weight 600, line-height 1.5,
24px above/8px below. They remain subordinate to the enclosing field heading;
their semantic heading levels and original content are preserved.
- Technical metadata labels/values: 14px/1.5, with weight 600 for labels.
Only code, paths and machine identifiers use the technical monospace family at
14px/1.65, identical for inline and fenced code (never compound `em` shrinkage).
Separate reading sections by 24px; keep the first Markdown block flush with its
field heading's 8px bottom gap. The same typography applies in light/dark and at
all responsive widths. Controls and archive indexes retain their compact UI roles.
Memory and Evidence detail readers use the entire available content width, without
the ordinary 72–75ch prose cap. This is the owner's explicit reading-layout choice.
Long unstructured paragraphs are split for display at existing sentence/semicolon
boundaries outside inline code and links; authored Markdown structure and stored
content are unchanged. Paragraph spacing is 1.25em. Scope and provenance share the
available width; provenance excerpts render Markdown rather than literal markers.
Copy actions use the two-overlapping-sheets icon, an accessible name/tooltip and
live success/failure feedback instead of a visible Copy label.
Memory has four explicitly FAKE formatting examples, one per family, in a separate
expandable section. They reuse the real detail reader but never enter persistence,
indexing, link search or model recall, and expose no edit/delete/save actions.
Curated evidence follows a fixed reading order: title, compact type and purpose summary, scope,
typed content, supporting excerpts, review items, then collapsed technical provenance. Machine
metadata stays in invisible comments so GitHub Preview shows only the reviewable document.
typed content, supporting excerpts, review items, then technical provenance. Curated v4 files
use short, visible YAML frontmatter for identity and classification. The Markdown title and
body are authoritative; hidden payload comments are a legacy format converted on consolidation.
`applies_to` is rendered as “Ambito di applicazione” with separate bullet lists for concepts,
tables, and columns. Enum values also use lists. Tables are forbidden for metadata, scope, or any
@@ -329,7 +428,9 @@ one-dimensional collection; reserve tables for genuinely two-dimensional dataset
identifiers use inline code. SQL uses fenced code. Supporting excerpts use blockquotes.
**The Review Surface Rule.** The visible Markdown must be readable without understanding the
machine contract. Technical metadata belongs in progressive disclosure, not above the title.
machine contract. In Administration, explain current and original provenance separately and
keep file-editing templates and Git instructions in progressive disclosure. Show actual host
paths with copy controls, never browser file links to container-only locations.
## Do's and Don'ts
@@ -341,7 +442,8 @@ machine contract. Technical metadata belongs in progressive disclosure, not abov
- **Do** preserve information density with headings, rhythm, and progressive disclosure.
- **Do** keep keyboard focus explicit and pair color with text, shape, icon, or position.
- **Do** respect `prefers-reduced-motion` while preserving immediate non-kinetic feedback.
- **Do** use English for interface chrome and the workspace language for persisted document content.
- **Do** use the selected interface language (English by default) for chrome and preserve the
workspace language for persisted domain content. Session interaction language remains pinned.
- **Do** render curated metadata and scope as Markdown prose or lists, never as a frontmatter table.
- **Do** break long curated rules into paragraphs, labelled subsections, and lists at existing
punctuation boundaries while preserving the exact canonical text for machines.
+398 -3
View File
@@ -1,6 +1,6 @@
# ThothII — Project State
Last updated: 2026-09-06.
Last updated: 2026-09-14.
This file is the short operational snapshot. Stable commands and the architecture mental model
live in `AGENTS.md`; current design and runtime contracts live under `docs/architecture/`,
@@ -12,8 +12,222 @@ Authentik, internal catalog/embedding services, and the PSD workspace repository
`docs/operations/server-upgrade-gitea-workspace-v2.md`. Treat its operator gates and rollback
requirements as mandatory; do not replace the running server stack in place.
## Session review layout deployed — 2026-09-14
Session-only dialogs now use the visible app bounds: artifact/column review grows
up to 80rem wide and the available height; short confirmations grow to 40rem.
Existing primary actions appear above and below session forms and review content,
with shared handlers, validation and pending state. Stop/delete focus Cancel;
rename focuses the name field. Administration surfaces are unchanged. The activity
shortcut is relabelled To Administration / Vai all’Amministrazione, retaining its
workspace-management destination. 778 frontend tests, five responsive browser
scenarios, typecheck, translations and the production build passed. The owner
authorized deployment and the server frontend was recreated at 18:09 CEST with tag `b1723c34-session-dialogs-20260914`. Frontend is healthy,
Omics serves the new assets, and doctor passes 13/13 checks. Core and Omics web
were not restarted. See DESIGN.md for the layout contract and
`docs/reports/2026-09-14-session-dialogs-release.md` for provenance and rollback.
## Session composer and empty Memory fix deployed — 2026-09-14
The embedded shell now caps its height at the portal mount height, keeping steering
and Stop & save visible. Empty authoritative Memory archives return zero results
without requiring embedding/BM25; SQL-rule embedding is lazy and shared. The live
empty PSD archive reproduces 503 with the current image and succeeds with the
candidate. 54 Memory tests, 90 frontend tests, five browser scenarios and both image
builds passed. The owner authorized the restart and core/frontend were recreated on
2026-09-14 at 17:18 CEST with tags `49333a2d-session-memory-fix`, from code committed
as `d6cdffea`. Both are healthy; production Memory search returns `[]`, Omics serves
the corrected CSS, native doctor passes 13/13 checks, and admissions are reopened.
Session inventory is preserved. Backups and rollback images are retained. Native
CLI diagnostics must run as installation owner UID 10001 with access to Compose;
see `docs/reports/2026-09-14-session-layout-memory-fix.md` for exact commands and
the remaining interactive browser acceptance.
## Full/embedded shell and bilingual interface
Full/embedded shell, EN/IT UI, immutable session interaction language, dark theme,
fullscreen and full-mode logout are implemented. The local Mac descriptor explicitly
sets `shell.mode: full` and `shell.defaultLocale: en`; the native `tht` and local
core/frontend images were updated on 2026-09-13. Omics uses embedded with server-side
identity verification and a replaceable presentation-only PortalAdapter; this does
not mean this branch was verified on the production server. Its changes are
in Omics commit `95154e1`; production deployment remains pending.
See `docs/operations/shell-and-localization.md` for integration and installation
instructions and `docs/reports/2026-09-13-full-shell-implementation.md` for tests,
independent reviews, local browser checks and rollback details.
### Documentation handoff before branch closure — 2026-09-13
Current rendering architecture is in `docs/architecture/application-shell.md`;
the exact portal identity/proxy contract is in `docs/install/authentication-upstream.md`.
`docs/operations/server-codex-handoff.md` is the current server delivery/deploy
runbook, including Omics source integration from GitHub, configuration, tests and
rollback. It supersedes the earlier Omics repository-relay instructions. The acceptance matrix is
`docs/testing/authentication-manual-acceptance.md`. README, documentation navigation,
user/installation/authentication guides and descriptor examples point to these
paths. Local examples explicitly use full/en; the projected server example is
for standalone OIDC, not Omics upstream. No runtime configuration or deployment
was changed by that documentation pass. The 2026-09-14 delivery is prepared for
promotion to main; the actual merge is recorded in Git. PSD acceptance remains pending.
### Navigation readiness and session accordions — 2026-09-13
The Workspace navigation button now carries an accessible green/red readiness
dot instead of a separate text row. Green requires a selected workspace and a
successful, current `ready` response; checking, unavailable and other states are
red, with the state exposed through the tooltip and accessible description.
The backend readiness gate is unchanged.
Active sessions (non-archived) and Archive both start collapsed. Their adjacent
headers are the only visible content below the scope tabs: no Sessions heading
or external selection toolbar. Each nonempty panel owns its Select all and
bulk-delete controls, scoped to that list and preserving the other's selection.
As of 2026-09-14, only one section can be open at a time, and either can be
collapsed. Empty lists show only "No sessions yet." The open section uses the
remaining sidebar height, with scrolling content capped at `min(18rem, 35dvh)`. The mobile
navigation dialog also provides a bounded height. Keyboard controls and labels
are retained. See `DESIGN.md` and `docs/guida-utente.md` for the UI contract.
Verification: 768 frontend unit tests, 20 browser scenarios (including 80 mocked
sessions at 390/1280px), TypeScript and the production frontend build passed.
Only the Mac frontend was recreated; it is healthy at `127.0.0.1:8080`.
Core, catalog, Qdrant and embedding containers were not changed. The prior
frontend image is retained as
`thothii-frontend:before-single-session-accordion-20260914` for rollback.
### Current Omics delivery and server handoff — 2026-09-14
The owner corrected the delivery requirement: Omics is obtained from GitHub,
not relayed to another repository as part of this deployment. Any optional
server-side repository copy is solely the owner's separate concern. This
supersedes the 2026-09-13 relay agreement, including historical delivery notes
in the Omics branch. Do not make another remote publication a prerequisite.
GitHub branch `codex/thothii-embedded-shell` at
`https://github.com/Dallavilla-Tiziano/omics_portal.git` was reverified at
`fca10901a73666ca257d8f4cc4b77066295c400a` (functional commit `95154e1`).
The server operator integrates it with the current code in the confirmed Omics
checkout, normally `/home/chirone/omics_portal`, preserving later server changes.
`docs/operations/server-codex-handoff.md` is the authoritative ordered procedure:
inventory, source verification, native CLI and installation projections,
embedded/upstream auth, Omics templates/static assets/proxy, coordinated rollout,
acceptance and rollback. Server deployment and real IdP acceptance remain pending.
## Current product shape
### Shared Memory/Evidence typography — 2026-09-13
Both detail readers now share Manrope and fixed reading roles: 24px card title,
20px section headings, 16px/1.65 narrative text, and 14px metadata/code. Authored
Markdown subheadings remain subordinate to field headings; inline/fenced code no
longer shrinks cumulatively. Full-width layout, paragraph separation and isolated
FAKE examples are retained. All 762 frontend tests and 18 browser scenarios pass,
including computed typography checks at 390/1280/2400px; typecheck, i18n and Docker
build pass. Only Mac frontend was recreated: `87fca0dc5e19`, image `19f1704b6bb2`,
healthy. Core and data services are unchanged. Rollback image:
`thothii-frontend:before-knowledge-typography-20260913`. See
`docs/reports/2026-09-13-knowledge-typography.md`. No PSD/Omics deployment.
### Knowledge reading and isolated fake Memory examples — 2026-09-13
Memory/Evidence details now use all available width, with display-only paragraph
splitting for long plain prose and preserved code/Markdown/source data. Evidence
provenance renders Markdown; copy controls are accessible two-sheet icons.
Memory's separate Formatting examples section contains four FAKE cards (one per
family), automatically expanded for an empty archive. These client-side examples
cannot be edited/saved/indexed and are never submitted to the model or Memory API.
No real archive content was changed. 762 tests, 18 browser scenarios, typecheck,
i18n and Docker build pass. Only the Mac frontend was recreated: `fc4286b5406d`,
image `cbe0fe0dce55`, healthy; other services unchanged. Rollback image:
`thothii-frontend:before-knowledge-reading-20260913`. Details in
`docs/reports/2026-09-13-knowledge-reading.md`; no PSD/Omics deployment.
### Unified Session entry and complete tab borders — 2026-09-13
The sidebar has one Session/Sessione button: return to the current unfinished
session (including pending creation) without resetting/reconnecting it; otherwise
prepare a new question with existing readiness and dirty-edit guards. Creation
still requires submitting a question. A cold document panel is not a running session.
Session tabs now have 11px horizontal/3px vertical padding, at least 38px height,
and matching 1px borders on every side (gray inactive, red active), with no shared
baseline or overlapping bottom border. This supersedes the earlier border removal.
757 frontend tests, 15 browser scenarios, typecheck, i18n and Docker build pass.
Mac frontend `2844778d311c`, image `d58dccfee3f9`, is healthy; only that service
was recreated. Core/data services are unchanged, with no Omics or server deploy.
Rollback image: `thothii-frontend:before-session-navigation-20260913`.
### Login copy and language-selector focus — 2026-09-13
Removed the redundant login eyebrow/icon and installation-account explanation;
the main sign-in heading remains. The full-header language select no longer
shows an outer focus ring after pointer interaction; keyboard focus remains
visible, including after returning with Tab. EN/IT login and both themes are
covered by 36 targeted tests and 14 browser scenarios; typecheck/build pass.
Only the Mac frontend was rebuilt/recreated: `204efd40d3eb`, image `7dd0752ff823`,
healthy. Other containers are unchanged. Rollback image:
`thothii-frontend:before-login-focus-20260913`. No production deployment.
### Full-header and layout refinements — 2026-09-13
The owner's four visual adjustments are implemented: full header uses Omics
`#CB333B` in both themes with a light complete wordmark/controls; Database status
has 16px clearance below its top divider; workspace/model/Done share a compact
desktop row; session-scope tabs have no bottom border. Embedded has no extra header.
Verified with 71 targeted frontend tests, 11 browser scenarios, typecheck and
Docker production build. The Mac frontend was recreated alone and is healthy
(`8a1608ad7fc6`, image `a2f489ebe9de`); Core/data-service containers are unchanged.
Full/en is retained. Rollback image: `thothii-frontend:before-header-layout-20260913`.
See `docs/reports/2026-09-13-header-layout-refinements.md` for validation and rollback.
### Visual review integrated with the full/embedded shell
At the owner's request, the seven commits through `a59624a6` from
`codex/ui-visual-review` are integrated with the shell/i18n work in this checkout,
`codex/prototype-administration-pages`. Both branches started at `2d1b714e`; the
first full-shell build omitted that lateral branch and regressed the installed UI.
The integrated source retains bundled Manrope, shared type roles, catalog/context
alignment, wordmark sizes and composer autosizing alongside full/embedded and EN/IT.
Local Docker was updated at 12:54 UTC on 2026-09-13: frontend image `605a6e6bba68`,
healthy; Core and all data-service containers were unchanged. Mac remains full/en.
The immediate pre-merge frontend is retained as
`thothii-frontend:before-visual-shell-merge-20260913`.
Verification: 755 frontend tests, 11 Playwright visual/interaction scenarios at five
widths, translation-catalog checks, typecheck and production build. See
`docs/reports/2026-09-13-visual-shell-integration.md` for delivery, provenance and
rollback details. The separate visual-review worktree and its rollback images are
retained. No merge to `main`, push or production deployment is part of this delivery.
Gitea #28–#31 follow-up is implemented in this checkout: Workspace's four tabs,
Database list-first entry without the preparation footer, full-height Pi instructions
with installation-host OS selection, and short Admin navigation labels. Core and
session behavior are retained. At the owner's request, these changes were rebuilt into
local Docker on 2026-09-12 at 18:31 UTC. Core/frontend are healthy and the UI at
`http://127.0.0.1:8080` serves the updated bundle. The host projection now supplies
`THT_HOST_PLATFORM=darwin`, so Pi selects macOS rather than the container's Linux OS.
Persistent dependency containers and volumes were unchanged. Rollback images are tagged
`thothii-core:before-admin-28-31-20260912` and
`thothii-frontend:before-admin-28-31-20260912`. See
`docs/reports/2026-09-12-admin-issues-28-31.md` for verification and deployment details.
The latest context-shelf A and five Administration pages are implemented locally.
At the owner's request, local Docker project `thothii-18998cca7b0a` was rebuilt and
its core/frontend recreated on 2026-09-12. The real UI at `http://127.0.0.1:8080`
serves shelf A; both services and their existing dependencies are healthy.
Runtime model projections were regenerated as schema v2 with the single
`zai/glm-5.3` interaction default. Persistent services/volumes were not recreated.
The local launcher `/private/tmp/thothii-memory-preview.sh` now adds
`/private/tmp/thothii-context-a.compose.yaml` last, building from this checkout
instead of the earlier Memory worktree. Previous images are retained under
`thothii-core:before-context-a-20260912` and
`thothii-frontend:before-context-a-20260912`. Remote server deployment remains pending.
Core/session behavior is retained; one independently remembered workspace/model
pair controls both Core and Admin. Unsaved Admin changes block navigation and
context changes; operation activity locks the selectors. See
`docs/reports/2026-09-12-context-shelf-a-implementation.md` for verification and the
remaining server/Omics integration gate. Prototype alternatives remain untouched.
ThothII is a human-in-the-loop datamart builder with three independently built layers:
```text
@@ -26,6 +240,165 @@ metadata catalog for administrative database configuration. The frontend renders
and keeps the live transcript in memory. See
`docs/architecture/components.md` for the detailed component and data-flow map.
## Memory M1–M3, Evidence E1–E3 and joint workflow repair X1 implemented
Browser authorization now derives permissions from validated session roles using the current
catalog. This fixes Memory/Evidence navigation remaining disabled for administrators whose
remembered login predates those permissions; refreshing the page loads the updated permissions.
The owner requested two independent administration projects: Memory management and Evidence
management. Their navigation entries must sit immediately after Database management, as peers;
neither page belongs to Database management. Both require complete browsing, filtering, and CRUD
without an active core session. Shared requirements and the two project briefs are linked from
`docs/plans/2026-09-08-memory-evidence-administration.md`. Memory administration is implemented;
Evidence administration is implemented through external file editing and explicit consolidation.
Evidence editing requires an explicit evolution of the current authoring/publication contract.
The M1 specification is at `docs/plans/2026-09-08-memory-m1-spec.md` and published as
[M1 — Archivio autorevole e amministrazione delle Memory Card](https://git.tylconsulting.it/mptyl/ThothII/issues/27)
with the `ready-for-agent` and `enhancement` labels.
It covers the authoritative store, administrative CRUD, projection recovery, and current
Memory/exemplar producer integration. Its accepted test boundaries are the public harness
service, Fastify APIs, and the AppShell page, joined by a focused real-stack browser path.
The owner confirmed those boundaries and authorized publication on 2026-09-08.
M1 is implemented locally: PostgreSQL authority in `thoth_memory`, versioned installation
migrations, admin CRUD for all four families, structured dependencies and links, explicit
projection recovery, verified recall and current workflow producers. The page requires no
active session or DWH binding. The existing `catalog-migrate` preparation service now runs
Memory migrations as well. Existing installations need that preparation before using M1;
the initial implementation did not deploy or migrate the owner's stacks.
The integrated browser check passed with real authentication, Fastify, ThtRunner, harness,
PostgreSQL and Qdrant; embeddings were deterministic and unrelated Pi/session activity used
test fixtures. See `docs/plans/2026-09-08-memory-m1-validation.md` for results and commands.
M2 is implemented locally: Memory dense/BM25 fusion, physical and business scope filters,
bounded outgoing-link expansion and joint ranking, all resolved against current PostgreSQL
authority. Migration `002_hybrid_projection.sql` makes old dense projections pending until
explicit retry/rebuild; Reference remains separate. The real retrieval check uses the configured
`qwen3-embedding:0.6b` model, a separate Ollama process with a read-only model-volume mount,
and isolated PostgreSQL/Qdrant resources. See `docs/plans/2026-09-09-memory-m2-validation.md`.
M3 is implemented: editable F8 summary grounded in effective approved decisions, explicit
updates with concurrent-edit protection, selected-card/link transactions and durable review
receipts. Finalization no longer saves exemplars implicitly. SQL rules and explained errors
are consulted in the existing F4/F6/F7 gates. Successful Catalog physical synchronization
performs dependency cleanup, preserving the original removals for recovery across restarts.
Migration `003_review_receipts.sql` is required. Validation includes a real GLM 5.3 generation
case against synthetic PostgreSQL data; see `docs/plans/2026-09-09-memory-m3-validation.md`.
Evidence administration and the joint X1 conflict-repair increment are now implemented.
The same local preview was subsequently rebuilt with M3 and migration 003 applied.
Core and frontend now include the final Memory review and physical dependency cleanup.
On 2026-09-09, at the owner's request, the local PSD Docker installation
`thothii-18998cca7b0a` was updated from this worktree. Core/frontend images were rebuilt,
Catalog and Memory migrations completed, and all five services became healthy. The UI is at
`http://127.0.0.1:8080`, using the existing local authentication and persistent volumes.
The `psd-clinical` authoritative Memory archive is initially empty (no legacy import).
The existing installation configuration remains in `/Users/mp/projects/ThothII/deploy/psd/`;
`/private/tmp/thothii-memory-preview.sh` invokes its Compose files with a final build-context
and migration-command override from this worktree. A future build from the main checkout
will use that checkout's code, so retain the worktree override until the changes are integrated.
Memory revision history and compatibility with existing development sessions are not requirements.
The agreed Memory scope includes reusable domain clarifications, SQL construction rules, solved
questions, and explained, approved mistakes to avoid. This extends the current runtime's
`concept_clarified`-only reusable Memory contract. The owner also approved an editable final summary
for proposed additions/updates and Memory consumption in the relevant existing review gates.
Further agreed behavior includes persistent corrections for Memory/Evidence conflicts through
explicit choices, deletion of Memory with invalid dependencies after successful physical schema
synchronization, and bounded functional/regression tests instead of a general quality benchmark.
In-house Memory evolution is approved, including hybrid Qdrant retrieval and explicit card links
traversed in core, without a dedicated graph database. Both capabilities belong to the current
scope. PostgreSQL is the agreed authority for Memory Cards, links, and schema dependencies;
Qdrant is a rebuildable index. Links are reviewed with cards and editable in Administration;
deleting a card removes its incident links while preserving the other cards. The accepted,
implemented architecture is recorded in
`docs/adr/0018-use-postgres-for-memory-and-qdrant-for-retrieval.md`.
For Evidence administration, Q12 rejects a separate editorial publication workflow.
The final R0 direction uses external editors and a manual consolidation command that
validates structure, reports required corrections, updates derived metadata, and
activates the local Evidence index. The operator then checks the diff and runs Git
commit/push manually. No watcher, automated Git, or web editor is required.
The owner's later simplification
instruction removes the requirement to support administrative edits while core work
is in progress. Deliberate corrections made by the workflow itself remain supported.
The context specialist writes Evidence drafts independently of the installation and
without PostgreSQL access; the system refines them and stores them locally for use
and maintenance. The revised Q11 recommendation separates external drafts from a
durable local canonical file archive, removing automatic commit/push from CRUD.
The owner has now accepted local files and requires discoverable, editable Markdown
for nontechnical domain specialists, with no JSONL management surface. Editors on
Mac/PC or vim/nano on the server are the chosen R0 editing surface. Evidence
management keeps browsing/filtering/detail and clearly identifies the persistent
working tree, each Markdown file's actual host path, and the manual commands.
E1 implements editable Curated Evidence v4, deterministic legacy conversion, persistent
local files and a consolidation API with immutable candidates, manual provenance,
deletion records and recoverable activation. E2 connects its installed command, runtime source
selection and administration page. The operator completes Git steps manually.
All 35 PSD units were converted on an isolated copy with identical IDs and typed content.
Tests exercise visible edits through normalization, indexing and recall with real Qdrant;
see `docs/plans/2026-09-09-evidence-e1-validation.md` and
`docs/contracts/curated-evidence-v4.md`. E2 is now running on the local Docker preview:
all 35 PSD units were converted and indexed through the installed command. The editable
host archive is `/Users/mp/projects/ThothII/deploy/psd/evidence-registry/repo/psd-clinical/evidence`.
The original registry-volume checkout and original author repository were retained.
The installation descriptor now includes `workspace-bindings.yaml` and `evidence-host.yaml`
so native maintenance uses the same volumes and checkout as the preview. Descriptor backup:
`/private/tmp/thothii-installation-before-e2.yaml`; previous native binary: `/private/tmp/tht-before-e2`.
The launcher remains `bash /private/tmp/thothii-memory-preview.sh`; its worktree build override
is still required until integration. Runtime lease filenames now include rendered bytes so
upgrading the Evidence renderer does not collide with old immutable configs; Catalog
input fingerprints and readiness are unchanged. See `docs/plans/2026-09-09-evidence-e2-validation.md`.
E3 adds explicit source acquisition/refinement and durable comparisons in Evidence management.
Local drafts go in `evidence/incoming/`; original local documents and configured HTTP/S3
sources are reacquired only by **Import or refresh sources** or installed
`tht workspace evidence refresh --workspace <id>`. Keep/replace decisions, including the
native `workspace evidence decide` command, activate through the existing archive/index
path and have durable retry state. Acquired source versions and remote provenance stay
local; old documentary lineage can coexist with current manual declarations.
The installed preview's 35 PSD sources were verified unchanged with no new proposals.
The previous native CLI is backed up at `/private/tmp/tht-before-e3`.
See `docs/plans/2026-09-09-evidence-e3-validation.md` for tests and validation limits.
X1 adds `reviewer_archive_repair`: closed alternatives with complete before/after content,
explicit rejection/reformulation, admin-only application, durable session receipts and
retry of saved but inactive corrections. The coordinator uses the canonical Memory and
Evidence services; phase approval remains separate. Migration `004_archive_repairs.sql`
is required. See `docs/contracts/archive-repair.md` for authorization and recovery limits.
X1 validation is recorded in `docs/plans/2026-09-09-archive-repair-x1-validation.md`:
both archives were corrected and retrieved through the actual CLI with real PostgreSQL
and Qdrant; desktop/mobile widget behavior and permission failures were verified separately.
The local preview images include X1 and migration 004 is applied. Follow-up technical
acceptance passed with configured GLM 5.3 generating closed conflict alternatives and
the chosen Memory correction persisted and retrieved. An authenticated browser test
also verified both administration pages, navigation order, Evidence filtering, workspace
404s, desktop/mobile layout, and Evidence survival after Memory deletion. All approved
technical increments/checks are complete. A real PSD domain-conflict session remains
the reviewer's semantic acceptance check; automated cases did not modify PSD knowledge.
The joint browser inspection also fixed mobile archive navigation: below 768px,
Memory and Evidence keep the full content width and open navigation in the shared
accessible dialog. Desktop retains its sidebar; the local frontend image includes this fix.
The owner accepted the remaining simplifications and requested explicit clarification
of the core format change and the simple terminal-based Git check. E1 must adapt
the Evidence parser, renderer, authoring, validation, and normalization; convert
existing files and reindex; and verify that visible edits reach core consumption.
Preserve the internal typed model where possible. This precedes the administrative
page and is not merely a presentation change. Human inspection uses normal Git
status and optional line-level diff commands; no custom diff viewer or mandatory
double review is required.
The suggestion of making PostgreSQL the Evidence authority was withdrawn after this
clarification; it was never implemented or accepted as a replacement for Q11.
Q13 keeps manual corrections active when updated sources contradict them, until an
administrator resolves the comparison; deleted Evidence must not be regenerated
automatically. Q14 allows direct manual creation and records a manual declaration
as the current source, preserving any original document as distinct provenance.
Q15 refreshes external sources only on explicit administrator request; normal saves
and lookups do not reacquire them. These decisions, now implemented through E1–E3, are recorded in
`docs/adr/0019-author-evidence-in-app-with-automatic-activation.md`.
The decision-by-decision review is recorded in
`docs/plans/2026-09-08-memory-evidence-simplification-review.md`; it distinguishes the
new constraints from the revised technical recommendations. It also specifies a
sequential save with minimal durable retry state and direct Memory cleanup after
successful schema synchronization, without new queues or event infrastructure.
The delivery order remains Memory, Evidence, and persistent conflict repair between
both modules. Local curated Evidence must survive preprocessing Clear and be backed
up as primary data; the Qdrant projection remains rebuildable.
No runtime gate change or administrative page is implemented yet.
## Evidence restructuring — accepted
The evidence restructuring and PSD migration completed real acceptance on 2026-08-25.
@@ -68,7 +441,8 @@ schema, Evidence, and vector mutation commands are retired.
The right Administration sidebar invokes that same operation for the selected workspace. It shows
only current readiness or the latest bounded failure diagnostic; there is no preprocessing history.
Known non-ready state disables **New session**, while backend admission remains authoritative.
Known non-ready state disables **Session** when it would start a new question,
while backend admission remains authoritative.
The same control exposes an inline-confirmed **Clear** action to remove replaceable reference
vectors, LSH, corpus, and checkpoints while preserving the separate Memory collection. The host CLI
equivalent is `workspace preprocess clear`.
@@ -93,7 +467,28 @@ regenerates the backend catalog, Pi `models.json`/`settings.json`, and Compose o
installation-local `generated/` directory. Those projections are replaceable runtime adapters:
they are not edited, backed up, or treated as configuration.
Session and metadata defaults use canonical `provider/model` IDs. Provider authentication declares
Core and Administration now share one canonical `modelCatalog.defaults.interaction` per installation,
independent of workspace. Runtime catalog schema v2 contains only `defaultInteraction`; apply the host,
backend, and regenerated projections together. Equal legacy defaults normalize on read; conflicting
ones require an explicit operator choice. With Admin AI configured, selectable models are the
intersection of the Pi and LiteLLM adapters; Core-only installations remain supported without Admin AI.
The existing Core and Database controls share the operational selection. Explicit model choices are
remembered per authenticated user/application mount in this browser, not in installation settings.
Resume uses that selection/default while preserving the historical workspace/revision and manifest.
Every model must be manually exercised in both Core and Admin as documented in
`docs/general/pi-configuration.md`; validation is not a live model certification.
These changes and shelf A are now deployed to local Docker; remote deployment remains pending.
PSD DeepSeek Pro/Flash now share canonical `deepseek/...` identities across native Pi and LiteLLM,
using the existing `DEEPSEEK_API_KEY` bundle entry. Its value was confirmed identical to the working
Pi key without exposing it; no secret files were changed. The duplicate `deepseek-metadata` descriptor
is removed locally and the tracked example uses the shared provider. `secret_env` overrides legacy
Pi auth only inside temporary runtime snapshots and fails closed if the bundle key is missing.
The original auth store/history and `zai/glm-5.3` default are preserved. Local Docker projections
and core/frontend were updated together on 2026-09-12. Regenerate projections with
the matching release for the separate server deployment.
Provider authentication declares
one explicit mode (`secret_env`, `pi_auth`, or `none`); `secret_env` names a protected bundle key.
The backend settings store now owns only the selected workspace and thinking level. Existing v1
installations use the explicit catalog migration command; schema-v3 workspace descriptors are
+21 -3
View File
@@ -4,8 +4,26 @@ ThothII is a human-reviewed NL-to-SQL workflow with a React frontend and a Fasti
core. The portable deployment runs two application services plus the installation-local metadata
catalog; DWH and LLM services remain external. Semantic services are bundled in Compose.
Authentication is configured through the single host CLI tht: see the [local authentication guide](docs/install/authentication-local.md),
[generic OIDC guide](docs/install/authentication-oidc.md), and [manual acceptance matrix](docs/testing/authentication-manual-acceptance.md).
The same frontend supports **full** (its own header) and **embedded** (inside a
portal). This choice is independent of authentication: the Mac uses full/local,
Omics uses embedded/upstream with its existing login, and a standalone server
can use full/OIDC. See [rendering architecture](docs/architecture/application-shell.md)
and [configuration, Omics delivery and deploy](docs/operations/shell-and-localization.md).
For the current server upgrade with Omics Portal, follow the ordered
[Codex server handoff](docs/operations/server-codex-handoff.md), including source
integration, embedded/upstream configuration, coordinated rollout and rollback.
Local/OIDC authentication is configured through the host CLI `tht`; portal
authentication is established by the trusted server proxy. See the
[local guide](docs/install/authentication-local.md),
[OIDC guide](docs/install/authentication-oidc.md),
[upstream integration](docs/install/authentication-upstream.md), and
[manual acceptance matrix](docs/testing/authentication-manual-acceptance.md).
For the clone-based manual standalone installation test on macOS, Windows, and Linux, use the
[Italian procedure](docs/install/standalone-manual-it.md) or the
[English procedure](docs/install/standalone-manual-en.md).
## Docker Compose: local startup
@@ -23,7 +41,7 @@ cp deploy/env/local.env.example deploy/env/local.env
./scripts/run-stack.sh
```
The launcher builds the core, starts `catalog-db`, runs the explicit one-shot Kysely migrations,
The low-level stack script builds the core, starts `catalog-db`, runs the explicit one-shot Kysely migrations,
then runs the base+local stack in the foreground. Migrations never run implicitly in backend
startup. The core image contains its Pi runtime; no host `pi` executable is used. For a server
installation, build the image, start the catalog, and run the same migration service before the
+18 -1
View File
@@ -7,6 +7,7 @@ import { fileURLToPath } from "node:url";
import { tmpdir } from "node:os";
import type { AppConfig } from "./config.js";
import { ThtRunner } from "./tht/tht-runner.js";
import { createMemoryCleanup } from "./catalog/memory-cleanup.js";
import { PiProcessManager } from "./pi/pi-process-manager.js";
import { SseHub } from "./sse/sse-hub.js";
import { authenticateSession, captureAuthConfigSnapshot, configuredOrigin } from "./auth/auth.js";
@@ -80,6 +81,8 @@ import { createProductionWorkspacePreprocessingService } from "./workspace-maint
import type { WorkspacePreprocessingService } from "./workspaces/preprocessing-service.js";
import { PreprocessingStateStore } from "./workspaces/preprocessing-state.js";
import { workspacePreprocessingRoutes } from "./routes/workspace-preprocessing.js";
import { memoryRoutes } from "./routes/memory.js";
import { evidenceRoutes } from "./routes/evidence.js";
export interface BuildAppDeps {
thtRunner?: ThtRunner;
@@ -93,7 +96,7 @@ export interface BuildAppDeps {
workspaceDiagnoser?: WorkspaceDiagnoser;
workspaceDatabaseTester?: WorkspaceDatabaseTester;
workspaceSecretStore?: WorkspaceSecretStore;
workspacePreprocessingService?: Pick<WorkspacePreprocessingService, "run" | "clear">;
workspacePreprocessingService?: Pick<WorkspacePreprocessingService, "run" | "clear"> & Partial<Pick<WorkspacePreprocessingService, "consolidateEvidence" | "evidenceSources">>;
catalogRepository?: CatalogRepository;
catalogService?: CatalogService;
catalogPostgresAccess?: CatalogPostgresAccess;
@@ -265,6 +268,11 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
catalogSchemaIntrospector,
catalogOperationCoordinator,
config.catalogSyncTimeoutMs,
createMemoryCleanup(tht as ThtRunner, {
internalQdrantUrl: config.internalQdrantUrl, internalEmbeddingUrl: config.internalEmbeddingUrl,
internalEmbeddingId: config.internalEmbeddingId, internalEmbeddingModel: config.internalEmbeddingModel,
internalEmbeddingDimensions: config.internalEmbeddingDimensions,
}),
);
app.addHook("onReady", async () => { await catalogSyncWorker.initialize(); });
app.addHook("onReady", async () => { await descriptionGenerationWorker.initialize(); });
@@ -493,7 +501,16 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
return maintenanceBarrier.status();
});
sqlRoutes(app, { tht: tht as ThtRunner, getSettings, workspaceRegistry });
memoryRoutes(app, { runner: tht as ThtRunner, registry: workspaceRegistry, runtime: {
internalQdrantUrl: config.internalQdrantUrl,
internalEmbeddingUrl: config.internalEmbeddingUrl,
internalEmbeddingModel: config.internalEmbeddingModel,
internalEmbeddingDimensions: config.internalEmbeddingDimensions,
} });
metaRoutes(app, { harnessDir: config.harnessDir, modelCatalog: runtimeModelCatalog });
evidenceRoutes(app, { runner: tht as ThtRunner, registry: workspaceRegistry,
registryRoot: config.workspaceRegistry.root, hostRegistryRoot: config.evidenceHostRegistryRoot,
service: workspacePreprocessingService });
workspaceRoutes(app, {
registry: workspaceRegistry,
config: config.workspaceRegistry,
+3 -1
View File
@@ -119,7 +119,9 @@ export function authenticateSession(deps: AuthDependencies): preHandlerHookHandl
subject: session.subject,
...(session.displayName === undefined ? {} : { displayName: session.displayName }),
roles: session.roles,
permissions: session.permissions,
// Sessions can outlive a deployment that changes the role permission catalog.
// resolve() has already checked validity, including current local user roles.
permissions: rolesToPermissions(session.roles),
isAdmin: session.roles.includes("admin"),
};
if (STATE_CHANGING_METHODS.has(request.method)) {
+1 -1
View File
@@ -38,7 +38,7 @@ const MAX_MAPPED_GROUPS = 128;
const ROLES = ["user", "admin"] as const;
export const PERMISSION_CATALOG: readonly Permission[] = [
"session.use", "session.read_all", "session.manage_all", "settings.manage",
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
"workspace.manage", "workspace.secrets.manage", "database.manage", "memory.manage", "evidence.manage", "pi.manage", "auth.diagnostics.read",
];
const invalid = (): Error => new Error("authentication configuration is invalid");
+1 -1
View File
@@ -33,7 +33,7 @@ const EMPTY_HKDF_SALT = Buffer.alloc(0);
const ROLES = ["user", "admin"] as const;
const PERMISSIONS = [
"session.use", "session.read_all", "session.manage_all", "settings.manage",
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
"workspace.manage", "workspace.secrets.manage", "database.manage", "memory.manage", "evidence.manage", "pi.manage", "auth.diagnostics.read",
] as const satisfies readonly Permission[];
const invalid = (): Error => new Error("auth_session_store_invalid");
+1 -1
View File
@@ -5,7 +5,7 @@ export type Role = "user" | "admin";
export type Permission =
| "session.use" | "session.read_all" | "session.manage_all"
| "settings.manage" | "workspace.manage" | "workspace.secrets.manage"
| "database.manage" | "pi.manage" | "auth.diagnostics.read";
| "database.manage" | "memory.manage" | "evidence.manage" | "pi.manage" | "auth.diagnostics.read";
export interface AuthenticationSessionConfig {
regularTtlSeconds: number;
+23
View File
@@ -0,0 +1,23 @@
import type { ThtRunner } from "../tht/tht-runner.js";
import type { SemanticRuntimeConfig } from "../workspaces/runtime-renderer.js";
import type { CatalogSyncRun, WorkspaceDatabase } from "./types.js";
/** Internal continuation of an applied physical sync, using the harness Memory boundary. */
export function createMemoryCleanup(runner: Pick<ThtRunner, "withPrincipal">, runtime: SemanticRuntimeConfig) {
return async (database: WorkspaceDatabase, run: CatalogSyncRun): Promise<number> => {
if (run.phase !== "memory_cleanup" || !run.plannedDiff) throw new Error("Physical cleanup is not committed");
const result = await runner.withPrincipal({ issuer: "installation", subject: "catalog-sync",
roles: ["admin"], permissions: ["memory.manage"], isAdmin: true,
}).runWithRuntimeSnapshot(["memory", "admin", "--workspace", database.workspaceId], JSON.stringify({
action: "cleanup", runtime, request: { sync_id: run.id, database: database.databaseName,
schema_name: database.schema, removed_tables: run.plannedDiff.deletedTables,
removed_columns: run.plannedDiff.deletedColumns.map(column => ({ table: column.tableName, column: column.columnName })),
},
}));
const payload = JSON.parse(result.stdout);
if (result.code !== 0 || payload.indexed !== true || !Number.isInteger(payload.deleted) || payload.deleted < 0) {
throw new Error("Memory cleanup is incomplete");
}
return payload.deleted;
};
}
+3 -1
View File
@@ -926,6 +926,7 @@ export class MemoryCatalogRepository implements CatalogRepository {
scope: CatalogSyncScope,
tableIds: readonly string[],
snapshot: ObservedSchemaSnapshot,
syncRunId?: string,
): Promise<CatalogSyncCounts | undefined> {
const database = this.records.get(databaseId);
if (!database || database.version !== expectedDatabaseVersion) return undefined;
@@ -1141,6 +1142,7 @@ export class MemoryCatalogRepository implements CatalogRepository {
if (scope === "all") {
this.records.set(databaseId, { ...database, schemaSyncedVersion: expectedDatabaseVersion, schemaSyncedAt: now });
}
if (syncRunId) await this.updateSyncRun(syncRunId, { phase: "memory_cleanup" });
return {
tables: (await this.listTables(databaseId)).length,
columns: [...this.columns.values()].filter((column) => this.tables.get(column.tableId)?.databaseId === databaseId).length,
@@ -1254,7 +1256,7 @@ export class MemoryCatalogRepository implements CatalogRepository {
for (const run of this.syncRuns.values()) {
if (["queued", "running", "awaiting_confirmation", "applying"].includes(run.state)) {
await this.updateSyncRun(run.id, {
state: "interrupted", phase: "completed", finishedAt: new Date().toISOString(),
state: "interrupted", phase: run.phase === "memory_cleanup" ? "memory_cleanup" : "completed", finishedAt: new Date().toISOString(),
errorCode: "SYNC_INTERRUPTED", errorMessage: "Synchronization was interrupted by a service restart",
});
}
@@ -102,7 +102,7 @@ export function loadMetadataGenerationModels(options: {
...(apiKeyEnv ? { apiKeyEnv, apiKey } : {}),
}));
}
const result = new RestartLoadedMetadataGenerationModels(models, catalog.defaultMetadataGeneration);
const result = new RestartLoadedMetadataGenerationModels(models, models.size ? catalog.defaultInteraction : null);
const safe = result.catalog();
return {
catalog: () => ({
+6 -1
View File
@@ -1604,6 +1604,7 @@ export class KyselyCatalogRepository implements CatalogRepository {
scope: CatalogSyncScope,
tableIds: readonly string[],
snapshot: ObservedSchemaSnapshot,
syncRunId?: string,
): Promise<CatalogSyncCounts | undefined> {
return await this.db.transaction().execute(async (trx) => {
const database = await trx.selectFrom("workspaceDatabases").select("version")
@@ -1779,6 +1780,10 @@ export class KyselyCatalogRepository implements CatalogRepository {
schemaSyncedAt: now,
}).where("id", "=", databaseId).execute();
}
if (syncRunId) {
await trx.updateTable("catalogSyncRuns").set({ phase: "memory_cleanup" })
.where("id", "=", syncRunId).where("databaseId", "=", databaseId).execute();
}
return {
tables: snapshot.tables.length,
columns: scope === "tables" ? undefined : snapshot.columns.length,
@@ -1882,7 +1887,7 @@ export class KyselyCatalogRepository implements CatalogRepository {
async interruptActiveSyncRuns(): Promise<void> {
await this.db.updateTable("catalogSyncRuns").set({
state: "interrupted", phase: "completed", errorCode: "worker_restarted",
state: "interrupted", phase: sql`case when phase='memory_cleanup' then phase else 'completed' end`, errorCode: "worker_restarted",
errorMessage: "Synchronization was interrupted by a backend restart.",
finishedAt: sql`now()`, updatedAt: sql`now()`,
leaseOwner: null, leaseExpiresAt: null,
+46 -19
View File
@@ -58,6 +58,8 @@ export class CatalogSyncWorker {
private readonly introspector: CatalogSchemaIntrospector,
private readonly operations: CatalogOperationCoordinator,
private readonly timeoutMs: number,
private readonly cleanupMemory: (database: WorkspaceDatabase, run: CatalogSyncRun) => Promise<number>
= async () => 0,
) {}
async initialize(): Promise<void> {
@@ -67,6 +69,9 @@ export class CatalogSyncWorker {
}
async start(database: WorkspaceDatabase, scope: CatalogSyncScope, tableIds: readonly string[]): Promise<CatalogSyncRun> {
if ((await this.repository.listSyncRuns(database.id, 100)).some(run => run.phase === "memory_cleanup")) {
throw new CatalogConflictError("Retry the pending Memory cleanup before starting another synchronization");
}
const uniqueTableIds = [...new Set(tableIds)];
if (scope === "columns") {
const tables = await Promise.all(uniqueTableIds.map((tableId) => this.repository.getTable(database.id, tableId)));
@@ -109,7 +114,7 @@ export class CatalogSyncWorker {
async cancel(runId: string): Promise<CatalogSyncRun | undefined> {
const run = await this.repository.getSyncRun(runId);
if (!run) return undefined;
if (run.state === "applying" || TERMINAL_STATES.has(run.state)) return run;
if (run.phase === "memory_cleanup" || run.state === "applying" || TERMINAL_STATES.has(run.state)) return run;
await this.repository.requestSyncRunCancellation(runId);
this.controllers.get(runId)?.abort();
if (run.state === "queued" || run.state === "awaiting_confirmation") {
@@ -138,6 +143,16 @@ export class CatalogSyncWorker {
}
const database = await this.repository.get(previous.databaseId);
if (!database) return undefined;
if (previous.phase === "memory_cleanup") {
const release = this.operations.reserve(database.id);
try {
const queued = await this.repository.updateSyncRun(runId, { state: "queued", cancelRequested: false,
errorCode: null, errorMessage: null, finishedAt: null, leaseOwner: null, leaseExpiresAt: null });
this.reservations.set(runId, release);
this.launch(runId);
return queued;
} catch (error) { release(); throw error; }
}
return await this.start(database, previous.scope, previous.tableIds);
}
@@ -179,6 +194,10 @@ export class CatalogSyncWorker {
if (!database || database.version !== claimed.requestedDatabaseVersion) {
throw new CatalogConflictError("Database binding changed before synchronization started");
}
if (claimed.phase === "memory_cleanup") {
await this.finishMemoryCleanup(database, claimed);
return;
}
const progress: CatalogSchemaScanProgress = async (phase, counts) => {
await this.checkCancelled(runId);
await this.repository.updateSyncRun(runId, {
@@ -230,36 +249,30 @@ export class CatalogSyncWorker {
claimed.scope,
claimed.tableIds,
snapshot,
runId,
);
if (!applied) throw new CatalogConflictError("Database binding changed before schema changes were applied");
await this.repository.updateSyncRun(runId, {
state: "succeeded",
phase: "completed",
counts: applied,
finishedAt: new Date().toISOString(),
observedSnapshot: null,
plannedDiff: null,
confirmationToken: null,
heartbeatAt: new Date().toISOString(),
leaseOwner: null,
leaseExpiresAt: null,
});
await this.repository.appendSyncEvent(runId, "info", "succeeded", "Synchronization completed.", { ...applied });
this.release(runId);
const cleanupRun = await this.repository.updateSyncRun(runId, { counts: applied });
if (!cleanupRun) throw new Error("Synchronization run disappeared");
await this.finishMemoryCleanup(database, cleanupRun);
/* Completion is recorded only after the durable Memory cleanup succeeds. */
return;
} catch (error) {
const current = await this.repository.getSyncRun(runId);
const cancelled = !timedOut && (error instanceof SyncCancelledError || controller.signal.aborted || current?.cancelRequested);
const failure = timedOut
const failure = current?.phase === "memory_cleanup"
? { code: "memory_cleanup_pending", message: "Catalog synchronized. Memory cleanup is pending; retry this synchronization to complete it." }
: timedOut
? { code: "schema_sync_timed_out", message: "Schema synchronization timed out." }
: safeFailure(error);
await this.repository.updateSyncRun(runId, {
state: cancelled ? "cancelled" : "failed",
phase: "completed",
phase: current?.phase === "memory_cleanup" ? "memory_cleanup" : "completed",
errorCode: cancelled ? null : failure.code,
errorMessage: cancelled ? null : failure.message,
finishedAt: new Date().toISOString(),
observedSnapshot: null,
plannedDiff: null,
observedSnapshot: current?.phase === "memory_cleanup" ? current.observedSnapshot : null,
plannedDiff: current?.phase === "memory_cleanup" ? current.plannedDiff : null,
confirmationToken: null,
leaseOwner: null,
leaseExpiresAt: null,
@@ -278,6 +291,20 @@ export class CatalogSyncWorker {
}
}
private async finishMemoryCleanup(database: WorkspaceDatabase, run: CatalogSyncRun): Promise<void> {
if (!run.plannedDiff || run.phase !== "memory_cleanup") throw new Error("Missing committed cleanup context");
await this.repository.appendSyncEvent(run.id, "info", "memory_cleanup", "Removing Memory cards with deleted physical dependencies.");
const memoryDeleted = await this.cleanupMemory(database, run);
const counts = { ...run.counts, memoryDeleted };
await this.repository.updateSyncRun(run.id, {
state: "succeeded", phase: "completed", counts, finishedAt: new Date().toISOString(),
observedSnapshot: null, plannedDiff: null, confirmationToken: null,
heartbeatAt: new Date().toISOString(), leaseOwner: null, leaseExpiresAt: null,
});
await this.repository.appendSyncEvent(run.id, "info", "succeeded", "Synchronization completed.", counts);
this.release(run.id);
}
private assertCapability(scope: CatalogSyncScope, snapshot: ObservedSchemaSnapshot): void {
const required = scope === "all" ? ["tables", "columns", "relationships"] as const : [scope] as const;
for (const name of required) {
+3 -1
View File
@@ -366,7 +366,7 @@ export type CatalogSyncState =
export type CatalogSyncPhase =
| "queued" | "connecting" | "scanning_tables" | "scanning_columns"
| "scanning_relationships" | "planning" | "awaiting_confirmation"
| "applying" | "completed";
| "applying" | "memory_cleanup" | "completed";
export interface CatalogSchemaDiff {
deletedTables: string[];
@@ -375,6 +375,7 @@ export interface CatalogSchemaDiff {
}
export interface CatalogSyncCounts {
memoryDeleted?: number;
tables?: number;
columns?: number;
relationships?: number;
@@ -587,6 +588,7 @@ export interface CatalogRepository {
scope: CatalogSyncScope,
tableIds: readonly string[],
snapshot: ObservedSchemaSnapshot,
syncRunId?: string,
): Promise<CatalogSyncCounts | undefined>;
createSyncRun(
databaseId: string,
+5
View File
@@ -30,8 +30,11 @@ export interface AppConfig {
dataRoot?: string;
ollamaEnsureTimeoutMs: number;
piManagementTimeoutMs: number;
/** Host CLI platform projected into Docker; not the browser or container OS. */
hostPlatform?: string;
secretsFile?: string;
installationConfigFile?: string;
evidenceHostRegistryRoot?: string;
modelCatalogFile?: string;
sensitivityNer?: {
pythonExecutable: string;
@@ -474,8 +477,10 @@ export function loadConfig(
dataRoot: env.THT_DATA_ROOT,
ollamaEnsureTimeoutMs: Number(env.OLLAMA_ENSURE_TIMEOUT_MS ?? 60000),
piManagementTimeoutMs: piManagementTimeout(env.PI_MANAGEMENT_TIMEOUT_MS),
hostPlatform: env.THT_HOST_PLATFORM,
secretsFile,
installationConfigFile,
evidenceHostRegistryRoot: env.THT_EVIDENCE_HOST_REGISTRY_ROOT || undefined,
modelCatalogFile,
sensitivityNer,
piAuthFile,
+14 -16
View File
@@ -47,9 +47,8 @@ const runtimeModelSchema = z.object({
}).strict();
const catalogSchema = z.object({
schemaVersion: z.literal(1),
defaultSession: canonicalId,
defaultMetadataGeneration: canonicalId.optional(),
schemaVersion: z.literal(2),
defaultInteraction: canonicalId,
embedding: z.object({ id: canonicalId, dimensions: z.number().int().positive() }).strict(),
models: z.array(runtimeModelSchema).max(64),
}).strict();
@@ -57,8 +56,7 @@ const catalogSchema = z.object({
export type RuntimeModel = z.infer<typeof runtimeModelSchema>;
export interface RuntimeModelCatalog {
readonly defaultSession: string | null;
readonly defaultMetadataGeneration: string | null;
readonly defaultInteraction: string | null;
readonly embedding: Readonly<{ id: string; dimensions: number }> | null;
sessionModels(): readonly RuntimeModel[];
metadataModels(): readonly RuntimeModel[];
@@ -66,19 +64,21 @@ export interface RuntimeModelCatalog {
}
class RestartLoadedRuntimeModelCatalog implements RuntimeModelCatalog {
readonly defaultSession: string | null;
readonly defaultMetadataGeneration: string | null;
readonly defaultInteraction: string | null;
readonly embedding: Readonly<{ id: string; dimensions: number }> | null;
readonly #sessions: readonly RuntimeModel[];
readonly #metadata: readonly RuntimeModel[];
readonly #sessionIds: ReadonlySet<string>;
constructor(catalog?: z.infer<typeof catalogSchema>) {
this.defaultSession = catalog?.defaultSession ?? null;
this.defaultMetadataGeneration = catalog?.defaultMetadataGeneration ?? null;
this.defaultInteraction = catalog?.defaultInteraction ?? null;
this.embedding = catalog ? Object.freeze({ ...catalog.embedding }) : null;
this.#sessions = Object.freeze((catalog?.models ?? []).filter((model) => model.session !== undefined));
this.#metadata = Object.freeze((catalog?.models ?? []).filter((model) => model.metadataGeneration !== undefined));
// One operational list. Session-only installations can still run Core, but once Admin
// LLM models are configured every selectable model must support both adapters.
const hasMetadata = catalog?.models.some((model) => model.metadataGeneration !== undefined);
this.#sessions = Object.freeze((catalog?.models ?? []).filter((model) =>
model.session !== undefined && (!hasMetadata || model.metadataGeneration !== undefined)));
this.#metadata = Object.freeze(this.#sessions.filter((model) => model.metadataGeneration !== undefined));
this.#sessionIds = new Set(this.#sessions.map((model) => model.id));
}
@@ -129,11 +129,9 @@ export function loadRuntimeModelCatalog(file?: string): RuntimeModelCatalog {
if (ids.size !== parsed.data.models.length) throw new Error("runtime model catalog contains duplicate models");
const sessions = parsed.data.models.filter((model) => model.session !== undefined).map((model) => model.id);
const metadata = parsed.data.models.filter((model) => model.metadataGeneration !== undefined).map((model) => model.id);
if (!sessions.includes(parsed.data.defaultSession)) throw new Error("runtime model catalog session default is invalid");
if ((metadata.length > 0) !== (parsed.data.defaultMetadataGeneration !== undefined)
|| (parsed.data.defaultMetadataGeneration !== undefined
&& !metadata.includes(parsed.data.defaultMetadataGeneration))) {
throw new Error("runtime model catalog metadata default is invalid");
if (!sessions.includes(parsed.data.defaultInteraction)
|| (metadata.length > 0 && !metadata.includes(parsed.data.defaultInteraction))) {
throw new Error("runtime model catalog interaction default is invalid");
}
return new RestartLoadedRuntimeModelCatalog(parsed.data);
}
+14 -9
View File
@@ -88,17 +88,22 @@ async function workflowDiagnostics(config: AppConfig): Promise<{ ready: true; wo
if (revisions.length === 0) throw new Error("workflow diagnostics unavailable");
return await withOperatorRunner(config, async (runner) => {
for (const revision of revisions) {
const result = await runner.run(["doctor", "--json"], revision.snapshotPath);
let payload: unknown;
const runtime = await runner.acquireWorkspaceRuntime(revision.snapshotPath);
try {
payload = JSON.parse(result.stdout);
} catch {
throw new Error("workflow diagnostics failed");
const result = await runner.run(["doctor", "--json"], runtime.path);
let payload: unknown;
try {
payload = JSON.parse(result.stdout);
} catch {
throw new Error("workflow diagnostics failed");
}
if (
result.code !== 0 || !payload || typeof payload !== "object"
|| (payload as { ok?: unknown }).ok !== true
) throw new Error("workflow diagnostics failed");
} finally {
runtime.release();
}
if (
result.code !== 0 || !payload || typeof payload !== "object"
|| (payload as { ok?: unknown }).ok !== true
) throw new Error("workflow diagnostics failed");
}
return { ready: true, workspaces: revisions.length };
});
+9
View File
@@ -11,6 +11,7 @@ import {
validateDeclarativePiConfig,
} from "./managed-config.js";
import type { RuntimeModelCatalog } from "../models/runtime-model-catalog.js";
import { secretValue } from "../config/secret-bundle.js";
export interface PiModel {
provider: string;
@@ -64,6 +65,14 @@ export function createPiModelLister(cfg: AppConfig, opts: Opts = {}): ListModels
}
const env = buildPiChildEnv({});
// Pi's availability enumeration also needs the catalog-owned credentials for built-in
// providers. It must keep working after their obsolete Pi auth entries are removed.
for (const model of opts.modelCatalog?.sessionModels() ?? []) {
const name = model.authentication.mode === "secret_env" ? model.authentication.apiKeyEnv : undefined;
if (!name) continue;
const value = secretValue(cfg, name);
if (value) env[name] = value;
}
delete env.THT_DATA_ROOT;
if (cfg.dataRoot !== undefined) env.THT_DATA_ROOT = cfg.dataRoot;
const child = spawnFn(cfg.piBin, ["--mode", "rpc"], { cwd: cfg.harnessDir, env });
+13 -2
View File
@@ -138,7 +138,7 @@ export interface PiRuntimeAgentSnapshot {
* Bind a session Pi process to the exact managed auth/model bytes validated at spawn time.
* Other agent resources remain live through symlinks, while session storage stays persistent.
*/
export function createPiRuntimeAgentSnapshot(): PiRuntimeAgentSnapshot {
export function createPiRuntimeAgentSnapshot(options: { excludeAuthProvider?: string } = {}): PiRuntimeAgentSnapshot {
const sourceAgentDir = configuredPiAgentDir();
const auth = readPiAgentFile(sourceAgentDir, "auth.json", true);
const models = readPiAgentFile(sourceAgentDir, "models.json", true);
@@ -165,7 +165,18 @@ export function createPiRuntimeAgentSnapshot(): PiRuntimeAgentSnapshot {
);
}
if (auth !== undefined) {
writeFileSync(join(snapshotDir, "auth.json"), auth, { flag: "wx", mode: 0o600 });
let effectiveAuth = auth;
if (options.excludeAuthProvider) {
const parsed = parsePiConfigJson(auth);
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) throw new PiManagedConfigError();
const provider = options.excludeAuthProvider.trim().toLowerCase();
effectiveAuth = JSON.stringify(Object.fromEntries(
Object.entries(parsed).filter(([key]) => key.trim().toLowerCase() !== provider),
));
}
// secret_env is authoritative for this provider. Keep the operator's auth file intact,
// but do not let an old Pi credential override the shared bundle inside this child.
writeFileSync(join(snapshotDir, "auth.json"), effectiveAuth, { flag: "wx", mode: 0o600 });
}
if (models !== undefined) {
writeFileSync(join(snapshotDir, "models.json"), models, { flag: "wx", mode: 0o600 });
+14 -10
View File
@@ -36,6 +36,7 @@ export interface PiInstallationConfig {
}
export interface PiStatus {
hostPlatform: "linux" | "macos" | "windows";
version?: string;
ready: boolean;
credentials: PiCredentialStatus;
@@ -92,6 +93,9 @@ interface PiManagementDeps {
}
export function createPiManagement(config: AppConfig, deps: PiManagementDeps): PiManagementService {
const platform = config.hostPlatform ?? process.platform;
const hostPlatform = platform === "darwin" ? "macos"
: platform === "windows" || platform === "win32" ? "windows" : "linux";
const now = deps.now ?? (() => new Date());
const diagnostics: string[] = [];
const addDiagnostic = (message: string): void => {
@@ -106,23 +110,23 @@ export function createPiManagement(config: AppConfig, deps: PiManagementDeps): P
});
const credentialStatus = deps.credentialStatus ?? ((provider: string | undefined) => {
try {
const model = deps.modelCatalog.defaultSession
? deps.modelCatalog.sessionModels().find((entry) => entry.id === deps.modelCatalog.defaultSession)
const model = deps.modelCatalog.defaultInteraction
? deps.modelCatalog.sessionModels().find((entry) => entry.id === deps.modelCatalog.defaultInteraction)
: undefined;
const credentialName = model?.authentication.mode === "secret_env"
? model.authentication.apiKeyEnv
: undefined;
const configuredApiKey = configuredPiProviderApiKey(
const configuredApiKey = credentialName ? `$${credentialName}` : configuredPiProviderApiKey(
readConfiguredPiAgentFile("models.json", true),
provider,
) ?? (credentialName ? `$${credentialName}` : undefined);
);
return piProviderCredentialStatus({
provider,
authProviders: loadPiAuthProviders(),
authProviders: credentialName ? new Set() : loadPiAuthProviders(),
resolveCredentialValue: () => credentialName
? secretValue(config, credentialName)
: config.modelCatalogFile ? undefined : secretValue(config, "THT_MODEL_API_KEY"),
credentialFile: config.modelApiKeyFile,
credentialFile: credentialName ? undefined : config.modelApiKeyFile,
configuredApiKey,
});
} catch {
@@ -152,8 +156,8 @@ export function createPiManagement(config: AppConfig, deps: PiManagementDeps): P
const installationConfig = (): PiInstallationConfig => {
const settings = readSettings();
const reasoning = config.defaults.thinking ?? settings.thinking;
const selected = deps.modelCatalog.defaultSession
? splitCanonicalModelId(deps.modelCatalog.defaultSession)
const selected = deps.modelCatalog.defaultInteraction
? splitCanonicalModelId(deps.modelCatalog.defaultInteraction)
: undefined;
return {
...(selected ? selected : {}),
@@ -169,11 +173,11 @@ export function createPiManagement(config: AppConfig, deps: PiManagementDeps): P
try {
const currentVersion = await version();
addDiagnostic("Pi version probe succeeded");
return { version: currentVersion, ready: true, credentials, config: current, checkedAt };
return { hostPlatform, version: currentVersion, ready: true, credentials, config: current, checkedAt };
} catch (error) {
const message = stableMessage(error, "Pi runtime is unavailable");
addDiagnostic(message);
return { ready: false, credentials, config: current, checkedAt, message };
return { hostPlatform, ready: false, credentials, config: current, checkedAt, message };
}
},
+30 -19
View File
@@ -25,6 +25,7 @@ export interface SessionRuntime {
}
export interface RuntimeOptions {
interactionLanguage?: string | null;
provider?: string;
model?: string;
thinking?: string;
@@ -48,6 +49,7 @@ export class PiProcessManager {
private spawnFn: (
sessionId: string, author: string, provider: string | undefined, model: string | undefined,
principal?: PrincipalContext, runtimeConfigPath?: string,
interactionLanguage?: string | null,
) => ChildProcessWithoutNullStreams;
private loadAuthProviders: (agentDir: string) => ReadonlySet<string>;
private modelCatalog: RuntimeModelCatalog;
@@ -63,15 +65,15 @@ export class PiProcessManager {
) {
this.modelCatalog = opts?.modelCatalog ?? loadRuntimeModelCatalog(cfg.modelCatalogFile);
this.modelCatalogConfigured = cfg.modelCatalogFile !== undefined
|| this.modelCatalog.defaultSession !== null;
|| this.modelCatalog.defaultInteraction !== null;
this.loadAuthProviders = opts?.authProviders
?? ((agentDir) => loadPiAuthProviders({ agentDir }));
if (opts?.spawnFn) {
this.spawnFn = (sessionId, author, provider, model, principal, runtimeConfigPath) =>
this.spawnPi(opts.spawnFn!, sessionId, author, provider, model, principal, runtimeConfigPath);
this.spawnFn = (sessionId, author, provider, model, principal, runtimeConfigPath, language) =>
this.spawnPi(opts.spawnFn!, sessionId, author, provider, model, principal, runtimeConfigPath, language);
} else {
this.spawnFn = (sessionId, author, provider, model, principal, runtimeConfigPath) =>
this.spawnPi(nodeSpawn, sessionId, author, provider, model, principal, runtimeConfigPath);
this.spawnFn = (sessionId, author, provider, model, principal, runtimeConfigPath, language) =>
this.spawnPi(nodeSpawn, sessionId, author, provider, model, principal, runtimeConfigPath, language);
}
}
@@ -85,33 +87,35 @@ export class PiProcessManager {
private spawnPi(
spawnFn: SpawnFn, sessionId: string, author: string, provider: string | undefined,
model: string | undefined, principal?: PrincipalContext, runtimeConfigPath?: string,
interactionLanguage?: string | null,
): ChildProcessWithoutNullStreams {
// This is the final shared boundary for createFor(), spawnFor(), and resume(). Validate
// before auth-provider inspection, then make Pi consume the exact copied bytes rather than
// reopening mutable mounted auth/models files after this check.
const agent = createPiRuntimeAgentSnapshot();
const catalogModel = provider && model
? this.modelCatalog.sessionModels().find((entry) => entry.provider === provider && entry.model === model)
: undefined;
const credentialName = catalogModel?.authentication.mode === "secret_env"
? catalogModel.authentication.apiKeyEnv : undefined;
const agent = createPiRuntimeAgentSnapshot({ excludeAuthProvider: credentialName ? provider : undefined });
let child: ChildProcessWithoutNullStreams | undefined;
try {
const catalogModel = provider && model
? this.modelCatalog.sessionModels()
.find((entry) => entry.provider === provider && entry.model === model)
: undefined;
const credentialName = catalogModel?.authentication.mode === "secret_env"
? catalogModel.authentication.apiKeyEnv
: undefined;
const projectedApiKey = configuredPiProviderApiKey(agent.models, provider)
?? (credentialName ? `$${credentialName}` : undefined);
const projectedApiKey = credentialName ? `$${credentialName}`
: configuredPiProviderApiKey(agent.models, provider);
const env = buildPiChildEnv({
provider,
authProviders: this.loadAuthProviders(agent.agentDir),
authProviders: credentialName ? new Set() : this.loadAuthProviders(agent.agentDir),
credentialValue: credentialName
? secretValue(this.cfg, credentialName)
: this.modelCatalogConfigured ? undefined : secretValue(this.cfg, "THT_MODEL_API_KEY"),
credentialFile: this.cfg.modelApiKeyFile,
credentialFile: credentialName ? undefined : this.cfg.modelApiKeyFile,
configuredApiKey: projectedApiKey,
additions: { THT_SESSION: sessionId, THT_AUTHOR: author },
});
env.PI_CODING_AGENT_DIR = agent.agentDir;
// A launch hint only: the gate reads the authoritative manifest before each turn.
delete env.THT_INTERACTION_LANGUAGE;
if (interactionLanguage) env.THT_INTERACTION_LANGUAGE = interactionLanguage;
env.PI_CODING_AGENT_SESSION_DIR = agent.sessionDir;
clearPrincipalEnvironment(env);
if (principal) Object.assign(env, principalEnvironment(principal));
@@ -189,7 +193,9 @@ export class PiProcessManager {
const model = o.model ?? this.cfg.defaults.model;
let child: ChildProcessWithoutNullStreams;
try {
child = this.spawnFn(sessionId, author, provider, model, o.principal, o.runtimeConfig?.path);
child = this.spawnFn(
sessionId, author, provider, model, o.principal, o.runtimeConfig?.path, o.interactionLanguage,
);
} catch (error) {
o.runtimeConfig?.release();
throw error;
@@ -298,8 +304,13 @@ export class PiProcessManager {
}
async resume(sessionId: string, tht: ThtRunner): Promise<SessionRuntime> {
const manifest = await tht.sessionShow(sessionId) as { provider?: string; model?: string; thinking?: string } | null;
const manifest = await tht.sessionShow(sessionId) as {
provider?: string; model?: string; thinking?: string; interaction_language?: string | null;
} | null;
const language = manifest?.interaction_language
?? (await tht.ensureInteractionLanguage(sessionId)).interaction_language;
return this.spawnFor(sessionId, {
interactionLanguage: language,
provider: manifest?.provider,
model: manifest?.model,
thinking: manifest?.thinking,
+6 -5
View File
@@ -70,28 +70,29 @@ export function createPiProviderSmoke(
try {
const canonicalProvider = canonicalPiProvider(provider);
if (!canonicalProvider || timeoutMs <= 0) throw providerFailure();
const configuredAuthProviders = authProviders();
const configuredAuthProviders = new Set(authProviders());
const configuredModels = options.readModelsStore
? options.readModelsStore()
: readConfiguredPiAgentFile("models.json", true);
const catalog = options.modelCatalog ?? loadRuntimeModelCatalog(config.modelCatalogFile);
const catalogConfigured = config.modelCatalogFile !== undefined
|| catalog.defaultSession !== null;
|| catalog.defaultInteraction !== null;
const catalogModel = catalog.sessionModels()
.find((entry) => entry.provider === canonicalProvider && entry.model === model);
const upstreamModel = catalogModel?.upstreamModel ?? model;
const credentialName = catalogModel?.authentication.mode === "secret_env"
? catalogModel.authentication.apiKeyEnv
: undefined;
const projectedApiKey = configuredPiProviderApiKey(configuredModels, canonicalProvider)
?? (credentialName ? `$${credentialName}` : undefined);
if (credentialName) configuredAuthProviders.delete(canonicalProvider);
const projectedApiKey = credentialName ? `$${credentialName}`
: configuredPiProviderApiKey(configuredModels, canonicalProvider);
const env = buildPiChildEnv({
provider: canonicalProvider,
authProviders: configuredAuthProviders,
credentialValue: credentialName
? secretValue(config, credentialName)
: catalogConfigured ? undefined : secretValue(config, "THT_MODEL_API_KEY"),
credentialFile: config.modelApiKeyFile,
credentialFile: credentialName ? undefined : config.modelApiKeyFile,
configuredApiKey: projectedApiKey,
});
clearPrincipalEnvironment(env);
+90
View File
@@ -0,0 +1,90 @@
import path from "node:path";
import type { FastifyInstance } from "fastify";
import { z } from "zod";
import { requirePermission, isPrincipalContext } from "../auth/authorization.js";
import type { ThtRunner } from "../tht/tht-runner.js";
import type { WorkspaceRegistry } from "../workspaces/registry.js";
import type { WorkspacePreprocessingService } from "../workspaces/preprocessing-service.js";
const params = z.object({ workspaceId: z.string().regex(/^[a-z][a-z0-9-]{2,62}$/),
evidenceId: z.string().regex(/^evidence:[a-z0-9]+(?:-[a-z0-9]+)*$/).optional() });
const query = z.object({
q: z.string().max(1000).optional(), kind: z.enum(["domain", "glossary", "enum", "example", "mapping", "normalization", "formula", "reference"]).optional(),
purpose: z.enum(["disambiguation", "rewriting", "schema_linking", "sql_generation"]).optional(),
status: z.enum(["new", "modified", "active", "removed", "review_required", "legacy", "invalid"]).optional(),
concept: z.string().max(300).optional(), table: z.string().max(300).optional(), column: z.string().max(300).optional(),
source: z.string().max(300).optional(), language: z.string().max(30).optional(),
sort: z.enum(["title", "id", "kind", "status"]).optional(), direction: z.enum(["asc", "desc"]).optional(),
page: z.coerce.number().int().positive().optional(), page_size: z.coerce.number().int().min(1).max(100).optional(),
}).strict();
const quote = (value: string) => `'${value.replaceAll("'", "'\\''")}'`;
export function evidenceRoutes(app: FastifyInstance, deps: {
runner: Pick<ThtRunner, "withPrincipal">; registry: Pick<WorkspaceRegistry, "list">;
registryRoot: string; hostRegistryRoot?: string;
service: Partial<Pick<WorkspacePreprocessingService, "consolidateEvidence" | "evidenceSources">>;
}) {
app.post("/workspaces/:workspaceId/evidence/sources", async (request, reply) => {
const principal = requirePermission(request, reply, "evidence.manage");
if (!isPrincipalContext(principal)) return reply;
try {
const { workspaceId } = params.parse(request.params);
const body = z.discriminatedUnion("action", [
z.object({ action: z.literal("refresh") }).strict(),
z.object({ action: z.literal("decide"), sourceId: z.string().regex(/^[a-f0-9]{64}$/),
revision: z.string().regex(/^[a-f0-9]{64}$/), decision: z.enum(["keep", "replace"]) }).strict(),
]).parse(request.body);
if (!(await deps.registry.list()).some(w => w.id === workspaceId)) return reply.code(404).send({ code: "workspace_invalid" });
if (!deps.service.evidenceSources) throw new Error("Evidence service unavailable");
return await deps.service.evidenceSources({ workspaceId, ...body, actor: principal.subject });
} catch (error) {
return reply.code(error instanceof z.ZodError ? 400 : 503).send({ code: "evidence_unavailable",
message: error instanceof z.ZodError ? "Invalid source request." : "Evidence source service is unavailable." });
}
});
app.get<{ Params: { workspaceId: string; evidenceId?: string } }>("/workspaces/:workspaceId/evidence", read);
app.get<{ Params: { workspaceId: string; evidenceId?: string } }>("/workspaces/:workspaceId/evidence/:evidenceId", read);
async function read(request: import("fastify").FastifyRequest, reply: import("fastify").FastifyReply) {
const principal = requirePermission(request, reply, "evidence.manage");
if (!isPrincipalContext(principal)) return reply;
try {
const { workspaceId, evidenceId } = params.parse(request.params);
const filters = query.parse(request.query);
if (!(await deps.registry.list()).some(w => w.id === workspaceId)) return reply.code(404).send({ code: "workspace_invalid" });
const root = path.join(deps.registryRoot, "repo", workspaceId);
const result = await deps.runner.withPrincipal(principal).runWithRuntimeSnapshot(
["evidence", "admin", "--workspace", workspaceId],
JSON.stringify({ root, query: { ...filters, ...(evidenceId ? { id: evidenceId } : {}) } }),
);
const payload = JSON.parse(result.stdout);
if (result.code !== 0) return reply.code(503).send(payload);
if (evidenceId && !payload.item) return reply.code(404).send({ code: "evidence_not_found", message: "Evidence was not found." });
const host = deps.hostRegistryRoot;
const hostPath = host && (path.isAbsolute(host) || path.win32.isAbsolute(host))
? (path.win32.isAbsolute(host) && !path.isAbsolute(host) ? path.win32 : path).join(host, "repo") : null;
const hostJoin = hostPath && path.win32.isAbsolute(hostPath) && !path.isAbsolute(hostPath) ? path.win32.join : path.join;
return { ...payload, location: { repository: hostPath, workspace: hostPath ? hostJoin(hostPath, workspaceId) : null,
runtime_workspace: root, host: "Installation host", command: `tht workspace evidence consolidate --workspace ${workspaceId}`,
git_commands: hostPath ? [
`git -C ${quote(hostPath)} status --short -- ${quote(workspaceId + "/evidence")}`,
`git -C ${quote(hostPath)} diff -- ${quote(workspaceId + "/evidence")}`,
`git -C ${quote(hostPath)} add -A -- ${quote(workspaceId + "/evidence")}`,
`git -C ${quote(hostPath)} commit --only -m 'Curate Evidence' -- ${quote(workspaceId + "/evidence")}`,
`git -C ${quote(hostPath)} push`,
] : [] } };
} catch (error) {
return reply.code(error instanceof z.ZodError ? 400 : 503).send({ code: "evidence_unavailable",
message: error instanceof z.ZodError ? "Evidence filters are invalid." : "Evidence archive is unavailable." });
}
}
app.post("/workspaces/:workspaceId/evidence/consolidate", async (request, reply) => {
const principal = requirePermission(request, reply, "evidence.manage");
if (!isPrincipalContext(principal)) return reply;
try {
const { workspaceId } = params.parse(request.params);
if (!(await deps.registry.list()).some(w => w.id === workspaceId)) return reply.code(404).send({ code: "workspace_invalid" });
if (!deps.service.consolidateEvidence) throw new Error("Evidence service unavailable");
return await deps.service.consolidateEvidence({ workspaceId });
} catch { return reply.code(503).send({ code: "evidence_unavailable", message: "Evidence consolidation is unavailable." }); }
});
}
+94
View File
@@ -0,0 +1,94 @@
import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
import { z } from "zod";
import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
import type { ThtRunner } from "../tht/tht-runner.js";
import type { WorkspaceRegistry } from "../workspaces/registry.js";
import type { SemanticRuntimeConfig } from "../workspaces/runtime-renderer.js";
const paramsSchema = z.object({
workspaceId: z.string().regex(/^[a-z][a-z0-9_-]{0,63}$/),
cardId: z.string().regex(/^mem-[0-9a-f-]{36}$/).optional(),
});
const family = z.enum(["domain_clarification", "sql_rule", "solved_question", "explained_error"]);
const cardSchema = z.object({
family, subject: z.string().trim().min(1).max(1000),
detail: z.string().max(50000).default(""), scope: z.string().trim().min(1).max(10000),
rationale: z.string().max(10000).default(""), question: z.string().max(10000).default(""),
sql: z.string().max(100000).default(""),
concepts: z.array(z.string().trim().min(1).max(200)).max(100).default([]),
dependencies: z.array(z.object({
database: z.string().trim().min(1).max(200), schema_name: z.string().max(200).default(""),
table: z.string().max(200).default(""), column: z.string().max(200).default(""),
}).strict()).max(200).default([]),
links: z.array(z.object({
target_id: z.string().min(1).max(100), meaning: z.string().trim().min(1).max(1000),
}).strict()).max(200).default([]),
}).strict();
const querySchema = z.object({
q: z.string().max(1000).optional(), family: family.optional(),
concept: z.string().max(200).optional(), database: z.string().max(200).optional(),
table: z.string().max(200).optional(), column: z.string().max(200).optional(),
origin: z.enum(["manual", "workflow"]).optional(),
updated_after: z.iso.datetime({ offset: true }).optional(),
updated_before: z.iso.datetime({ offset: true }).optional(),
page: z.coerce.number().int().positive().optional(),
page_size: z.coerce.number().int().min(1).max(100).optional(),
sort: z.enum(["updated_at", "created_at", "subject", "family"]).optional(),
direction: z.enum(["asc", "desc"]).optional(),
}).strict();
export function memoryRoutes(app: FastifyInstance, deps: {
runner: Pick<ThtRunner, "withPrincipal">;
registry: Pick<WorkspaceRegistry, "list" | "read">;
runtime: SemanticRuntimeConfig;
}) {
const invoke = (action: string) => async (request: FastifyRequest, reply: FastifyReply) => {
const principal = requirePermission(request, reply, "memory.manage");
if (!isPrincipalContext(principal)) return reply;
try {
const { workspaceId, cardId } = paramsSchema.parse(request.params);
const input = action === "list" ? querySchema.parse(request.query)
: action === "create" || action === "update"
? { card: cardSchema.parse(request.body), ...(cardId ? { id: cardId } : {}) }
: cardId ? { id: cardId } : {};
// Registry identity is enough: administrative browsing must not require a DWH binding.
const workspaces = await deps.registry.list();
if (!workspaces.some(workspace => workspace.id === workspaceId)) {
return reply.code(404).send({ code: "workspace_invalid", message: "Workspace was not found." });
}
const { workspace } = await deps.registry.read(workspaceId);
const result = await deps.runner.withPrincipal(principal).runWithRuntimeSnapshot(
["memory", "admin", "--workspace", workspaceId],
JSON.stringify({ action, request: input, runtime: {
...deps.runtime, memoryLanguage: workspace.workspace.language,
} }),
);
let payload;
try { payload = JSON.parse(result.stdout); } catch {
return reply.code(503).send({ code: "memory_unavailable", message: "Memory archive is unavailable." });
}
if (result.code !== 0) {
const codes: Record<string, number> = {
memory_invalid: 400, memory_forbidden: 403, memory_not_found: 404,
memory_conflict: 409, memory_unavailable: 503,
};
const code = typeof payload?.code === "string" && payload.code in codes
? payload.code : "memory_unavailable";
return reply.code(codes[code]).send({ code, message: "Memory operation could not be completed." });
}
return reply.code(action === "create" ? 201 : 200).send(payload);
} catch (error) {
if (error instanceof z.ZodError) {
return reply.code(400).send({ code: "memory_invalid", message: "Memory request is invalid." });
}
return reply.code(503).send({ code: "memory_unavailable", message: "Memory archive is unavailable." });
}
};
app.get("/workspaces/:workspaceId/memory", invoke("list"));
app.post("/workspaces/:workspaceId/memory", invoke("create"));
app.get("/workspaces/:workspaceId/memory/pending", invoke("pending"));
app.get("/workspaces/:workspaceId/memory/:cardId", invoke("show"));
app.put("/workspaces/:workspaceId/memory/:cardId", invoke("update"));
app.delete("/workspaces/:workspaceId/memory/:cardId", invoke("delete"));
app.post("/workspaces/:workspaceId/memory/:cardId/retry", invoke("retry"));
}
+92 -21
View File
@@ -13,6 +13,7 @@ import type { MaintenanceBarrier } from "../runtime/maintenance-gate.js";
import { hasPermission, isPrincipalContext, requirePermission } from "../auth/authorization.js";
import { splitCanonicalModelId, type RuntimeModelCatalog } from "../models/runtime-model-catalog.js";
import type { CatalogRepository } from "../catalog/types.js";
import { interactionLanguage } from "../tht/interaction-language.js";
const BOOTSTRAP_FAILURE_MESSAGE =
"Session startup failed. Check configuration and connectivity, then Resume the session.";
@@ -150,13 +151,34 @@ export function sessionRoutes(
});
app.addHook("onResponse", async (req) => { admissionLeases.get(req)?.(); });
type SessionRevisionScan = {
revisions: Awaited<ReturnType<typeof d.workspaceRegistry.list>>;
retainedComplete: boolean;
};
let retainedSnapshotWarning: string | null = null;
/** Include retained historical descriptors so removed workspaces remain resumable. */
const sessionRevisions = async () => {
const sessionRevisions = async (): Promise<SessionRevisionScan> => {
const registry = d.workspaceRegistry as Partial<WorkspaceRegistry>;
if (typeof registry.listRetainedSnapshots === "function") {
return await registry.listRetainedSnapshots();
try {
const revisions = await registry.listRetainedSnapshots();
retainedSnapshotWarning = null;
return { revisions, retainedComplete: true };
} catch (error) {
// A legacy/corrupt historical snapshot must not make active sessions (and their SSE
// reviewer gates) unreachable. The registry still rejects that snapshot; this fallback
// exposes only descriptors from the verified active state and deliberately disables
// retention reconciliation because the resulting session view is incomplete.
const detail = error instanceof Error ? error.message : "unknown error";
if (retainedSnapshotWarning !== detail) {
console.warn("[sessions] retained snapshot discovery failed; using active snapshots:", detail);
retainedSnapshotWarning = detail;
}
return { revisions: await d.workspaceRegistry.list(), retainedComplete: false };
}
}
return await d.workspaceRegistry.list();
return { revisions: await d.workspaceRegistry.list(), retainedComplete: true };
};
const isNotFound = (error: unknown) =>
@@ -200,7 +222,7 @@ export function sessionRoutes(
};
let revisions: Awaited<ReturnType<typeof d.workspaceRegistry.list>>;
try {
revisions = await sessionRevisions();
({ revisions } = await sessionRevisions());
} catch (registryError) {
// Sessions created before revision pinning still live under the installation's legacy
// default config. Keep that compatibility path available when a fresh installation has
@@ -360,8 +382,17 @@ export function sessionRoutes(
app.post("/sessions", async (req, reply) => {
const b = req.body as {
question: string; name?: string; workspace?: string; workspaceId?: string;
provider?: string; model?: string; thinking?: string;
provider?: string; model?: string; thinking?: string; interactionLanguage?: unknown;
};
const language = interactionLanguage(b?.interactionLanguage);
if (!language) return reply.code(400).send({
code: "invalid_interaction_language",
error: "interactionLanguage must be a well-formed BCP-47 language tag",
});
if ((b.provider === undefined) !== (b.model === undefined)
|| (b.provider !== undefined && (typeof b.provider !== "string" || typeof b.model !== "string" || !b.provider || !b.model))) {
return reply.code(400).send({ error: "provider and model must be supplied together" });
}
const principal = getPrincipal(req);
let s: Settings;
try { s = await d.getSettings(principal); } catch { return storageFailure(reply); }
@@ -425,11 +456,9 @@ export function sessionRoutes(
}
}
const requestedCanonical = b.provider && b.model ? `${b.provider}/${b.model}` : undefined;
let selectedCanonical = requestedCanonical ?? d.modelCatalog.defaultSession;
let modelWarning: string | undefined;
if (selectedCanonical && d.modelCatalog.defaultSession && !d.modelCatalog.hasSession(selectedCanonical)) {
selectedCanonical = d.modelCatalog.defaultSession;
modelWarning = `Configured model ${requestedCanonical ?? "selection"} is unavailable; using ${selectedCanonical}.`;
const selectedCanonical = requestedCanonical ?? d.modelCatalog.defaultInteraction;
if (selectedCanonical && d.modelCatalog.defaultInteraction && !d.modelCatalog.hasSession(selectedCanonical)) {
return reply.code(503).send({ error: MODEL_UNAVAILABLE_MESSAGE, code: "model_unavailable" });
}
const selected = selectedCanonical ? splitCanonicalModelId(selectedCanonical) : undefined;
const provider = selected?.provider ?? b.provider;
@@ -484,6 +513,7 @@ export function sessionRoutes(
({ id } = await runner.sessionNew({
question: b.question, name: b.name, workspaceConfigPath,
workspaceId, workspaceRevision, provider, model, thinking,
interactionLanguage: language,
}));
manifestPersisted = true;
if (revisionLease) {
@@ -497,6 +527,7 @@ export function sessionRoutes(
} catch { return storageFailure(reply); }
const options = {
provider, model, thinking,
interactionLanguage: language,
author: principal.displayName ?? principal.subject,
principal,
question: b.question,
@@ -533,7 +564,7 @@ export function sessionRoutes(
),
() => d.mgr.start(id, rt, runtimeOptions),
);
return { id, ...(modelWarning ? { warning: modelWarning } : {}) };
return { id };
} finally {
if (revisionLease && !manifestPersisted) {
await revisionLease.abort().catch((error: unknown) => {
@@ -559,8 +590,8 @@ export function sessionRoutes(
? { ...principal, isAdmin: false }
: ownershipPrincipal(principal, "session.read_all");
const runner = runnerFor(scopedPrincipal);
const revisions = await sessionRevisions();
const lists = await Promise.all(revisions
const revisionScan = await sessionRevisions();
const lists = await Promise.all(revisionScan.revisions
.map((revision) => runner.sessionList(revision.snapshotPath) as Promise<SessionRow[]>));
const sessions = new Map<string, SessionRow>();
for (const row of lists.flat()) {
@@ -570,7 +601,8 @@ export function sessionRoutes(
// Only an administrator-visible complete list (or the single local principal) is safe
// input for retention. A remote per-user view can never discard another principal's pin.
const reconcileSnapshotRetention = (d.workspaceRegistry as Partial<WorkspaceRegistry>).reconcileSnapshotRetention;
const hasCompleteRetentionView = (scope === "all" || principal.issuer === "local")
const hasCompleteRetentionView = revisionScan.retainedComplete
&& (scope === "all" || principal.issuer === "local")
&& hasPermission(principal, "session.read_all");
if (hasCompleteRetentionView && typeof reconcileSnapshotRetention === "function") {
const retained = [...new Set(list
@@ -604,6 +636,23 @@ export function sessionRoutes(
} catch (error) { return lifecycleFailure(reply, error); }
const rt = d.mgr.get(id);
if (!rt) return reply.code(404).send({ error: "sessione non attiva" });
const pending = rt.bridge.pendingWidget?.() as any;
const response = (req.body as any)?.ui_response;
if (pending?.widget === "archive-repair" && response?.id === pending.id && !response.control) {
const choices = response.choices;
if (!Array.isArray(choices) || choices.length !== 1)
return reply.code(400).send({ error: "Select one archive repair choice" });
if (choices[0] !== "continue" && choices[0] !== "reject") {
const option = pending.repair?.options?.find((item: any) => item.id === choices[0]);
if (!option || !["memory", "evidence"].includes(option.archive))
return reply.code(400).send({ error: "Unknown archive repair choice" });
const permission = option.archive === "memory" ? "memory.manage" : "evidence.manage";
if (!principal.isAdmin || !hasPermission(principal, permission))
return reply.code(403).send({ error: "Archive corrections require an administrator" });
if (rt.ownerKey !== `${principal.issuer}\0${principal.subject}`)
return reply.code(409).send({ error: "Resume the session with your account before correcting an archive" });
}
}
if (!rt.bridge.respond((req.body as any).ui_response)) {
return reply.code(409).send({ error: "risposta non corrispondente al gate in attesa" });
}
@@ -623,6 +672,13 @@ export function sessionRoutes(
app.post("/sessions/:id/resume", async (req, reply) => {
const id = (req.params as any).id;
const principal = getPrincipal(req);
if (Object.hasOwn(req.body ?? {}, "interactionLanguage")
|| Object.hasOwn(req.body ?? {}, "interaction_language")) {
return reply.code(400).send({
code: "interaction_language_pinned",
error: "Resume uses the session's persisted interaction language; overrides are not accepted",
});
}
return withSessionLifecycle(id, async () => {
let settings: Settings;
let located: LocatedSession | undefined;
@@ -640,11 +696,19 @@ export function sessionRoutes(
const saved = manifest as {
provider?: string; model?: string; thinking?: string;
workspace_id?: string; workspace_revision?: string;
interaction_language?: string | null;
};
const savedCanonical = saved.provider && saved.model ? `${saved.provider}/${saved.model}` : "";
if (d.modelCatalog.defaultSession && (!savedCanonical || !d.modelCatalog.hasSession(savedCanonical))) {
const requested = (req.body ?? {}) as { provider?: string; model?: string; thinking?: string };
if ((requested.provider === undefined) !== (requested.model === undefined)
|| (requested.provider !== undefined && (typeof requested.provider !== "string" || typeof requested.model !== "string" || !requested.provider || !requested.model))) {
return reply.code(400).send({ error: "provider and model must be supplied together" });
}
const selectedCanonical = requested.provider && requested.model
? `${requested.provider}/${requested.model}` : d.modelCatalog.defaultInteraction;
if (d.modelCatalog.defaultInteraction && (!selectedCanonical || !d.modelCatalog.hasSession(selectedCanonical))) {
return reply.code(503).send({ error: MODEL_UNAVAILABLE_MESSAGE, code: "model_unavailable" });
}
const selected = selectedCanonical ? splitCanonicalModelId(selectedCanonical) : saved;
let workspaceConfigPath: string;
let workspaceDescriptor: WorkspaceDescriptor | undefined;
try {
@@ -671,13 +735,19 @@ export function sessionRoutes(
});
}
try { settings = await d.getSettings(principal); } catch { return storageFailure(reply); }
let language = saved.interaction_language;
if (language == null) {
try {
language = (await runner.ensureInteractionLanguage(id, workspaceConfigPath)).interaction_language;
} catch { return reply.code(503).send({ error: RESUME_FAILURE_MESSAGE }); }
}
// This check belongs inside the per-session lock: a preceding cold Resume may have
// installed a running runtime while this request was waiting.
const existing = d.mgr.get(id);
if (existing) {
const state = existing.bridge.turnState();
if (state === "running" || state === "waiting") {
return reply.code(200).send({ id, alreadyActive: true });
return reply.code(200).send({ id, alreadyActive: true, workspaceId: saved.workspace_id });
}
}
const ensure = await d.readiness.ensure(
@@ -688,8 +758,9 @@ export function sessionRoutes(
...(ensure.code ? { code: ensure.code } : {}),
});
const options = {
provider: saved?.provider,
model: saved?.model,
provider: selected.provider,
interactionLanguage: language,
model: selected.model,
thinking: saved?.thinking ?? settings.thinking,
author: principal.displayName ?? principal.subject,
principal,
@@ -711,7 +782,7 @@ export function sessionRoutes(
if (current) {
const state = current.bridge.turnState();
if (state === "running" || state === "waiting") {
return reply.code(200).send({ id, alreadyActive: true });
return reply.code(200).send({ id, alreadyActive: true, workspaceId: saved.workspace_id });
}
}
@@ -752,7 +823,7 @@ export function sessionRoutes(
d.mgr.configure(rt, runtimeOptions), null,
() => d.mgr.start(id, rt, runtimeOptions),
);
return reply.code(200).send({ id, alreadyActive: false });
return reply.code(200).send({ id, alreadyActive: false, workspaceId: saved.workspace_id });
});
});
app.post("/sessions/:id/close", async (req, reply) => {
+2 -2
View File
@@ -16,8 +16,8 @@ export function effectiveSettings(
modelCatalog?: RuntimeModelCatalog,
): Settings {
const workspaces = listWorkspaces(cfg.harnessDir);
const selected = modelCatalog?.defaultSession
? splitCanonicalModelId(modelCatalog.defaultSession)
const selected = modelCatalog?.defaultInteraction
? splitCanonicalModelId(modelCatalog.defaultInteraction)
: undefined;
return {
workspace: stored.workspace ?? workspaces[0]?.name,
+10
View File
@@ -0,0 +1,10 @@
/** Validate/canonicalize the tag; available translation catalogs belong to the UI. */
export function interactionLanguage(value: unknown): string | undefined {
if (typeof value !== "string") return undefined;
try {
const [canonical] = Intl.getCanonicalLocales(value);
return canonical;
} catch {
return undefined;
}
}
+9
View File
@@ -59,6 +59,7 @@ export interface SessionRow {
author: string | null;
workspace_id?: string | null;
workspace_revision?: string | null;
interaction_language?: string | null;
archived?: boolean;
}
@@ -482,6 +483,7 @@ export class ThtRunner {
async sessionNew(o: {
question: string;
interactionLanguage?: string;
provider?: string;
model?: string;
thinking?: string;
@@ -497,6 +499,7 @@ export class ThtRunner {
["--provider", o.provider],
["--model", o.model],
["--thinking", o.thinking],
["--interaction-language", o.interactionLanguage],
["--name", o.name],
["--workspace-id", o.workspaceId],
["--workspace-revision", o.workspaceRevision],
@@ -533,6 +536,12 @@ export class ThtRunner {
return this.json<unknown>(["session", "show", id, "--json"], workspace);
}
ensureInteractionLanguage(id: string, workspace?: string) {
return this.json<{ interaction_language: string }>(
["session", "ensure-interaction-language", id, "--json"], workspace,
);
}
sqlPreview(id: string, p: { limit?: number; offset?: number }, workspace?: string) {
// No positional FILE: the harness resolves sql_final.sql from the session
// via _session_sql_file(cfg, session_id), which respects the workspace path.
+16 -2
View File
@@ -18,7 +18,7 @@ export interface WorkspaceMaintenanceIo {
writeStderr(value: string): void;
}
type Command = "inspect" | "preprocess-run" | "preprocess-clear";
type Command = "inspect" | "preprocess-run" | "preprocess-clear" | "evidence-consolidate" | "evidence-refresh" | "evidence-decide";
function failureResult(
operation: string,
@@ -65,6 +65,9 @@ function parseRequest(command: string, stdin: string): Record<string, unknown> {
inspect: ["schemaVersion", "workspaceId"],
"preprocess-run": ["schemaVersion", "workspaceId"],
"preprocess-clear": ["schemaVersion", "workspaceId"],
"evidence-consolidate": ["schemaVersion", "workspaceId"],
"evidence-refresh": ["schemaVersion", "workspaceId"],
"evidence-decide": ["schemaVersion", "workspaceId", "sourceId", "revision", "decision"],
};
const allowed = allowedByCommand[command];
if (!allowed) throw new Error("unknown command");
@@ -81,6 +84,16 @@ function exitCodeFor(result: WorkspaceOperationResult): number {
async function dispatch(command: Command, service: WorkspacePreprocessingService, request: Record<string, unknown>): Promise<WorkspaceOperationResult> {
switch (command) {
case "evidence-refresh":
return await service.evidenceSources({ workspaceId: request.workspaceId as string, action: "refresh" });
case "evidence-decide":
if (typeof request.sourceId !== "string" || !/^[a-f0-9]{64}$/.test(request.sourceId)
|| typeof request.revision !== "string" || !/^[a-f0-9]{64}$/.test(request.revision)
|| (request.decision !== "keep" && request.decision !== "replace")) throw new Error("invalid request");
return await service.evidenceSources({ workspaceId: request.workspaceId as string, action: "decide",
sourceId: request.sourceId, revision: request.revision, decision: request.decision });
case "evidence-consolidate":
return await service.consolidateEvidence({ workspaceId: request.workspaceId as string });
case "inspect":
return await service.inspect({ workspaceId: request.workspaceId as string });
case "preprocess-run":
@@ -127,7 +140,8 @@ export async function runWorkspaceMaintenanceCli(
|| message === "unexpected request field"
|| message === "invalid workspace id";
return command in {
inspect: true, "preprocess-run": true, "preprocess-clear": true,
inspect: true, "preprocess-run": true, "preprocess-clear": true, "evidence-consolidate": true,
"evidence-refresh": true, "evidence-decide": true,
} ? (requestError ? 2 : 1) : 2;
}
}
@@ -22,6 +22,7 @@ export interface EvidencePreprocessingRequest {
evidence: EvidenceConfig;
job: EvidenceJobState;
dryRun?: boolean;
consolidate?: boolean;
httpPrivateHostAllowlist?: readonly string[];
}
@@ -56,7 +57,7 @@ function isPrivateHost(hostname: string): boolean {
return hostname.endsWith(".internal");
}
function evidencePolicy(
export function evidencePolicy(
evidence: EvidenceConfig,
httpPrivateHostAllowlist?: readonly string[],
): EvidencePreprocessingOutcome | undefined {
@@ -95,13 +96,14 @@ function jobResult(job: EvidenceJobState): Pick<
};
}
async function runEvidenceStage(
export async function runEvidenceStage(
request: EvidencePreprocessingRequest,
deps: EvidencePreprocessingDependencies,
): Promise<EvidencePreprocessingOutcome> {
const payload = await deps.runStage([
"preprocess",
"evidence",
...(request.consolidate ? ["--consolidate"] : []),
...(request.dryRun ? ["--dry-run"] : []),
...(request.job.childRuns.evidence
? ["--resume", request.job.childRuns.evidence]
@@ -3,6 +3,8 @@ import { renameSync, rmSync, writeFileSync, mkdirSync } from "node:fs";
import { join } from "node:path";
import {
continueEvidencePreprocessing,
evidencePolicy,
runEvidenceStage,
type EvidencePreprocessingDependencies,
type EvidencePreprocessingOutcome,
} from "./evidence/preprocessing.js";
@@ -108,6 +110,72 @@ function baseResult(
export class WorkspacePreprocessingService {
constructor(private readonly deps: WorkspacePreprocessingServiceDeps) {}
async evidenceSources(options: { workspaceId: string; action: "refresh" | "decide";
sourceId?: string; revision?: string; decision?: "keep" | "replace"; actor?: string }): Promise<WorkspaceOperationResult> {
const runtime = await this.deps.acquireActiveRuntime(options.workspaceId);
try {
if (options.action === "refresh") {
const refused = evidencePolicy(runtime.workspace.evidence, this.deps.httpPrivateHostAllowlist);
if (refused) return baseResult(runtime, "evidence refresh", "failed", refused.code);
}
const argv = ["evidence", "sources", options.action, "--json", "-c", "/dev/fd/3"];
if (options.action === "decide") {
if (!/^[a-f0-9]{64}$/.test(options.sourceId ?? "") || !/^[a-f0-9]{64}$/.test(options.revision ?? "")
|| !["keep", "replace"].includes(options.decision ?? "")) throw new Error("Invalid source decision");
const preflight = await this.deps.evidencePreflight(runtime.workspace);
if (!preflight.ok) return baseResult(runtime, "evidence sources", "failed", preflight.code);
argv.push("--source-id", options.sourceId!, "--revision", options.revision!, "--decision", options.decision!);
}
argv.push("--actor", options.actor ?? "installation operator");
const result = await this.deps.runChild({ argv, configPath: runtime.configLease.path });
const payload = JSON.parse(result.stdout);
if (result.exitCode !== 0 || payload.status !== "succeeded") throw new Error(
typeof payload.error === "string" ? payload.error.slice(0, 1500) : "Evidence source operation failed");
return baseResult(runtime, `evidence ${options.action}`, "succeeded", "ok", {
counts: this.numberRecord(payload.counts),
warnings: [options.action === "refresh" ? "Source comparisons are ready in Evidence management. Active content is unchanged."
: "Source decision activated locally. Commit and push the Evidence tree manually."],
});
} catch (error) {
return baseResult(runtime, `evidence ${options.action}`, "failed", "evidence_materialization_required", {
warnings: [error instanceof Error ? error.message : "Evidence source operation failed"],
});
}
}
async consolidateEvidence(options: { workspaceId: string }): Promise<WorkspaceOperationResult> {
const runtime = await this.deps.acquireActiveRuntime(options.workspaceId);
const preflight = await this.deps.evidencePreflight(runtime.workspace);
if (!preflight.ok) return baseResult(runtime, "evidence consolidate", "failed", preflight.code);
// Evidence has its own durable archive/corpus jobs. Do not alter Catalog readiness
// or the full preprocessing job when publishing this one component.
try {
const outcome = await runEvidenceStage({ evidence: runtime.workspace.evidence,
consolidate: true, job: { runId: randomBytes(16).toString("hex"), completedStages: [], childRuns: {} },
}, {
runStage: async argv => {
const result = await this.deps.runChild({ argv, configPath: runtime.configLease.path });
const payload = JSON.parse(result.stdout);
if (result.exitCode !== 0 || payload.status !== "succeeded") {
return Promise.reject(new Error(typeof payload.error === "string" ? payload.error.slice(0, 1500) : "Evidence consolidation failed. Retry the command."));
}
return payload;
},
persistJob: () => undefined,
evidencePreflight: async () => preflight,
requireRunId: value => this.requireRunId(value),
numberRecord: value => this.numberRecord(value),
});
return baseResult(runtime, "evidence consolidate", "succeeded", "ok", {
...outcome, warnings: ["Evidence is active locally. Catalog and Schema readiness are unchanged. Commit and push the Evidence files manually."],
});
} catch (error) {
return baseResult(runtime, "evidence consolidate", "failed", "evidence_materialization_required", {
warnings: [error instanceof Error ? error.message : "Evidence consolidation failed. Retry the command."],
});
}
}
async inspect(options: { workspaceId: string }): Promise<WorkspaceOperationResult> {
try {
const runtime = await this.deps.acquireActiveRuntime(options.workspaceId);
@@ -497,7 +497,10 @@ export async function publishDeterministicRuntimeConfigLease(options: {
rendered.workspaceRevision,
renderedConfigObject,
);
const identitySuffix = inputFingerprintValue.slice(7, 23);
// The Catalog input identity intentionally excludes representation-only changes.
// A lease also identifies its bytes, so a new renderer never collides with an
// immutable config produced by an earlier release for the same Catalog inputs.
const identitySuffix = sha256(inputFingerprintValue + "\n" + publishedConfig).slice(7, 23);
const preprocessingRoot = ensureTrustedDirectory(join(
options.dataRoot,
+2 -1
View File
@@ -1,4 +1,4 @@
import { basename, join } from "node:path";
import { basename, dirname, join } from "node:path";
import { stringify } from "yaml";
import { buildInstallationContract } from "./contracts.js";
import { validateWorkspaceDescriptor, type WorkspaceDescriptor } from "./schema.js";
@@ -179,6 +179,7 @@ function renderEvidence(
return {
evidence: {
...(workspace.evidence.schema_version === 2 ? { schema_version: 2 } : {}),
local_archive_root: join(dirname(dirname(context.revisionContentRoot)), "repo", context.workspaceId),
sources: [renderedSource],
},
vector: {
@@ -0,0 +1,48 @@
import Fastify from "fastify";
import { expect, test, vi } from "vitest";
import { sessionRoutes } from "../src/routes/sessions.js";
import type { PrincipalContext } from "../src/auth/principal.js";
test.each([
[false, "m", 403], [false, "e", 403], [false, "reject", 204],
[true, "m", 204], [true, "e", 204], [true, "forged", 400],
] as const)("archive repair response enforces the responding principal (%s, %s)", async (admin, choice, status) => {
const app = Fastify();
const principal: PrincipalContext = { issuer: "test", subject: "reviewer", isAdmin: admin,
roles: admin ? ["admin"] : ["user"],
permissions: admin ? ["session.use", "memory.manage", "evidence.manage"] : ["session.use"] };
app.addHook("preHandler", async request => { request.principal = principal; });
const respond = vi.fn(() => true);
sessionRoutes(app, {
tht: {}, mgr: { get: () => ({ ownerKey: "test\0reviewer", bridge: {
respond, pendingWidget: () => ({ id: "gate", widget: "archive-repair", repair: { options: [
{ id: "m", archive: "memory" }, { id: "e", archive: "evidence" },
] } }),
} }) },
} as unknown as Parameters<typeof sessionRoutes>[1]);
try {
const result = await app.inject({ method: "POST", url: "/sessions/s/response",
payload: { ui_response: { id: "gate", choices: [choice] } } });
expect(result.statusCode).toBe(status);
expect(respond).toHaveBeenCalledTimes(status === 204 ? 1 : 0);
} finally { await app.close(); }
});
test("an administrator cannot attribute a repair to another runtime's principal", async () => {
const app = Fastify();
app.addHook("preHandler", async request => { request.principal = {
issuer: "test", subject: "second-admin", isAdmin: true, roles: ["admin"],
permissions: ["session.use", "memory.manage"],
}; });
const respond = vi.fn();
sessionRoutes(app, { tht: {}, mgr: { get: () => ({ ownerKey: "test\0first-admin", bridge: {
respond, pendingWidget: () => ({ id: "gate", widget: "archive-repair",
repair: { options: [{ id: "m", archive: "memory" }] } }),
} }) } } as unknown as Parameters<typeof sessionRoutes>[1]);
try {
const result = await app.inject({ method: "POST", url: "/sessions/s/response",
payload: { ui_response: { id: "gate", choices: ["m"] } } });
expect(result.statusCode).toBe(409);
expect(respond).not.toHaveBeenCalled();
} finally { await app.close(); }
});
+2
View File
@@ -188,6 +188,8 @@ test("roles collapse duplicates and admin contains all administrative permission
"workspace.manage",
"workspace.secrets.manage",
"database.manage",
"memory.manage",
"evidence.manage",
"pi.manage",
"auth.diagnostics.read",
]);
+1 -1
View File
@@ -130,7 +130,7 @@ test("local login sets a non-persistent opaque session cookie and exposes only a
roles: ["admin"],
permissions: [
"session.use", "session.read_all", "session.manage_all", "settings.manage",
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
"workspace.manage", "workspace.secrets.manage", "database.manage", "memory.manage", "evidence.manage", "pi.manage", "auth.diagnostics.read",
],
isAdmin: true,
csrfToken: expect.stringMatching(/^[A-Za-z0-9_-]{43}$/),
+32 -6
View File
@@ -7,6 +7,8 @@ import { join } from "node:path";
import { expandLocalHome, localPrincipal, upstreamPrincipal } from "../src/auth/principal.js";
import { buildApp } from "../src/app.js";
import { loadConfig } from "../src/config.js";
import { rolesToPermissions } from "../src/auth/config.js";
import type { Permission, Role } from "../src/auth/types.js";
test("server smoke rejects retired trusted claims under OIDC authentication", () => {
const smoke = readFileSync("../scripts/unified-deployment-smoke.sh", "utf8");
@@ -32,7 +34,7 @@ test("local mode resolves a stable local principal", async () => {
roles: ["admin"],
permissions: [
"session.use", "session.read_all", "session.manage_all", "settings.manage",
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
"workspace.manage", "workspace.secrets.manage", "database.manage", "memory.manage", "evidence.manage", "pi.manage", "auth.diagnostics.read",
],
isAdmin: true,
});
@@ -74,7 +76,7 @@ test("upstream mode accepts only normalized proxy principal headers", async () =
issuer: "portal", subject: "42", displayName: "Alice", roles: ["user", "admin"],
permissions: [
"session.use", "session.read_all", "session.manage_all", "settings.manage",
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
"workspace.manage", "workspace.secrets.manage", "database.manage", "memory.manage", "evidence.manage", "pi.manage", "auth.diagnostics.read",
],
isAdmin: true,
});
@@ -206,15 +208,33 @@ test("the session boundary rejects tht maintenance headers outside exact loopbac
})).statusCode).toBe(503);
});
test("the session boundary touches a valid cookie session through the bounded Task 7 store operation", async () => {
test.each<{
name: string;
roles: Role[];
storedPermissions: Permission[];
}>([
{ name: "current user", roles: ["user"], storedPermissions: ["session.use"] },
{
name: "administrator signed in before archive permissions existed",
roles: ["admin"],
storedPermissions: rolesToPermissions(["admin"]).filter(
(permission) => permission !== "memory.manage" && permission !== "evidence.manage",
),
},
{
name: "user with obsolete administrative permissions",
roles: ["user"],
storedPermissions: ["session.use", "memory.manage", "evidence.manage"],
},
])("the session boundary derives current permissions and touches a valid cookie: $name", async ({ roles, storedPermissions }) => {
const sessions = {
resolve: vi.fn(async () => ({
version: 1,
issuer: "local",
subject: "user-1",
method: "local",
roles: ["user"],
permissions: ["session.use"],
roles,
permissions: storedPermissions,
userAuthRevision: 1,
authConfigRevision: "b".repeat(64),
remembered: false,
@@ -249,7 +269,13 @@ test("the session boundary touches a valid cookie session through the bounded Ta
app.get("/private", async (request) => getPrincipal(request));
const token = "z".repeat(43);
expect((await app.inject({ method: "GET", url: "/private", headers: { cookie: `thothii_session=${token}` } })).statusCode).toBe(200);
const response = await app.inject({ method: "GET", url: "/private", headers: { cookie: `thothii_session=${token}` } });
expect(response.statusCode).toBe(200);
expect(response.json()).toMatchObject({
roles,
permissions: rolesToPermissions(roles),
isAdmin: roles.includes("admin"),
});
expect(sessions.touch).toHaveBeenCalledWith(token);
});
@@ -124,8 +124,17 @@ test.skipIf(!dockerAvailable)("PostgreSQL migration enforces one database per wo
],
relationships: [],
};
expect(await repository.applySchemaSync(created.id, 1, "columns", [], fullColumnsSnapshot))
const memoryCleanupRun = await repository.createSyncRun(created.id, "columns", [], 1);
await repository.updateSyncRun(memoryCleanupRun.id, { state: "applying", phase: "applying" });
await expect(repository.applySchemaSync(created.id, 999, "columns", [], fullColumnsSnapshot, memoryCleanupRun.id))
.resolves.toBeUndefined();
expect((await repository.getSyncRun(memoryCleanupRun.id))?.phase).toBe("applying");
expect(await repository.applySchemaSync(created.id, 1, "columns", [], fullColumnsSnapshot, memoryCleanupRun.id))
.toMatchObject({ created: 4, deleted: 0 });
expect((await repository.getSyncRun(memoryCleanupRun.id))?.phase).toBe("memory_cleanup");
await repository.interruptActiveSyncRuns();
expect(await repository.getSyncRun(memoryCleanupRun.id))
.toMatchObject({ state: "interrupted", phase: "memory_cleanup" });
expect((await repository.listColumns(created.id, patients.id)).map((column) => ({
name: column.name,
sensitive: column.sensitive,
@@ -233,6 +242,7 @@ test.skipIf(!dockerAvailable)("PostgreSQL migration enforces one database per wo
});
expect(await repository.listSyncRuns(created.id)).toEqual([
expect.objectContaining({ id: syncRun.id, tableIds: [patients.id] }),
expect.objectContaining({ id: memoryCleanupRun.id, phase: "memory_cleanup" }),
]);
const lockedDatabase = await repository.create({
+42 -2
View File
@@ -117,6 +117,7 @@ async function setup(env: Record<string, string> = {}) {
});
const introspector: CatalogSchemaIntrospector = { scan };
const operations = new CatalogOperationCoordinator();
const cleanup = vi.fn(async () => ({ code: 0, stdout: JSON.stringify({ indexed: true, deleted: 0 }), stderr: "" }));
const registry = {
list: vi.fn(async () => [revision]),
listCatalog: vi.fn(async () => [{ id: "psd-clinical", name: "Policlinico San Donato", configurationState: "ready", revision }]),
@@ -124,7 +125,7 @@ async function setup(env: Record<string, string> = {}) {
readPinned: vi.fn(async () => ({ workspace, workspaceConfigPath: revision.snapshotPath })),
} as unknown as WorkspaceRegistry;
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "/missing", NODE_ENV: "test", ...env }), {
thtRunner: {} as never,
thtRunner: { withPrincipal: () => ({ runWithRuntimeSnapshot: cleanup }) } as never,
workspaceRegistry: registry,
workspaceSecretStore: new WorkspaceSecretStore({ root: secretRoot, runtimeRoot, installationId: "test" }),
catalogRepository: repository,
@@ -133,7 +134,7 @@ async function setup(env: Record<string, string> = {}) {
workspaceDiagnoser: vi.fn(),
});
return {
app, repository, database: (await repository.get(created.id))!, scan, operations,
app, repository, database: (await repository.get(created.id))!, scan, operations, cleanup,
setObserved(next: ObservedSchemaSnapshot) { observed = next; },
};
}
@@ -614,6 +615,45 @@ test("waits for confirmation and rescans before applying destructive changes", a
expect(scan).toHaveBeenCalledTimes(3);
});
test("retries committed Memory cleanup with the original removals without rescanning", async () => {
const { app, repository, database, scan, setObserved, cleanup } = await setup();
await seedCatalog(repository, database);
const observed = snapshot();
observed.tables = observed.tables.filter(t => t.name !== "visits");
observed.columns = observed.columns.filter(c => c.tableName !== "visits");
observed.relationships = [];
setObserved(observed);
cleanup.mockResolvedValueOnce({ code: 0, stdout: JSON.stringify({ indexed: false, deleted: 2 }), stderr: "" });
const started = await app.inject({ method: "POST", url: `/catalog/databases/${database.id}/sync-runs`,
payload: { version: database.version, scope: "all", tableIds: [] } });
const waiting = await waitFor(repository, started.json().id, "awaiting_confirmation");
expect(cleanup).not.toHaveBeenCalled();
await app.inject({ method: "POST", url: `/catalog/sync-runs/${waiting.id}/confirm`,
payload: { confirmationToken: waiting.confirmationToken } });
const failed = await waitFor(repository, waiting.id, "failed");
expect(failed).toMatchObject({ phase: "memory_cleanup", errorCode: "memory_cleanup_pending",
plannedDiff: { deletedTables: ["visits"] } });
expect((await repository.listTables(database.id)).map(t => t.name)).toEqual(["patients"]);
const callsBeforeRetry = scan.mock.calls.length;
const blocked = await app.inject({ method: "POST", url: `/catalog/databases/${database.id}/sync-runs`,
payload: { version: database.version, scope: "all", tableIds: [] } });
expect(blocked.statusCode).toBe(409);
// Simulate startup recovery and an unreachable DWH after schema application.
await repository.interruptActiveSyncRuns();
scan.mockRejectedValue(new Error("DWH unavailable"));
cleanup.mockResolvedValue({ code: 0, stdout: JSON.stringify({ indexed: true, deleted: 2 }), stderr: "" });
const retried = await app.inject({ method: "POST", url: `/catalog/sync-runs/${waiting.id}/retry` });
expect(retried.statusCode).toBe(202);
const completed = await waitFor(repository, waiting.id, "succeeded");
expect(completed.counts).toMatchObject({ memoryDeleted: 2 });
expect(scan.mock.calls.length).toBe(callsBeforeRetry);
expect(cleanup).toHaveBeenCalledTimes(2);
expect(cleanup.mock.calls[1]).toEqual(cleanup.mock.calls[0]);
const request = JSON.parse((cleanup.mock.calls[0] as unknown as string[])[1]!).request;
expect(request).toMatchObject({ sync_id: waiting.id, database: "warehouse",
schema_name: "datawarehouse", removed_tables: ["visits"] });
});
test("deletes every catalog table for multiple selected databases and cascades dependent metadata", async () => {
const { app, repository, database } = await setup();
const second = await repository.create({
+2 -2
View File
@@ -36,7 +36,7 @@ test("loop F1: crea sessione → SSE riceve il widget → risponde → il modell
ollamaEnsure: async () => ({ ok: true }),
searchPack: async () => {},
sessionNew: async () => ({ id: "s1" }),
sessionShow: async (_id: string) => ({ id: "s1", provider: undefined, model: undefined, thinking: undefined }),
sessionShow: async (_id: string) => ({ id: "s1", interaction_language: "en", provider: undefined, model: undefined, thinking: undefined }),
sessionList: async () => [],
} as any,
spawnFn: () => spawn("node", [FAKE, SCRIPT]) as any,
@@ -48,7 +48,7 @@ test("loop F1: crea sessione → SSE riceve il widget → risponde → il modell
const created = await fetch(`${base}/sessions`, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ workspace: "w", question: "q" }),
body: JSON.stringify({ workspace: "w", question: "q", interactionLanguage: "en" }),
});
expect(created.status).toBe(200);
+69
View File
@@ -0,0 +1,69 @@
import Fastify from "fastify";
import { afterEach, expect, test, vi } from "vitest";
import { evidenceRoutes } from "../src/routes/evidence.js";
import type { ThtRunner } from "../src/tht/tht-runner.js";
import type { PrincipalContext } from "../src/auth/principal.js";
const apps: ReturnType<typeof Fastify>[] = [];
afterEach(async () => { await Promise.all(apps.splice(0).map(app => app.close())); });
function setup(admin = true) {
const app = Fastify(); apps.push(app);
const principal: PrincipalContext = { issuer: "test", subject: "curator", roles: [admin ? "admin" : "user"],
permissions: admin ? ["evidence.manage"] : ["session.use"], isAdmin: admin };
app.addHook("preHandler", async request => { request.principal = principal; });
const run = vi.fn(async () => ({ code: 0, stdout: JSON.stringify({ items: [], total: 0 }), stderr: "" }));
const withPrincipal = vi.fn(() => ({ runWithRuntimeSnapshot: run }) as unknown as ThtRunner);
const consolidateEvidence = vi.fn();
const evidenceSources = vi.fn().mockResolvedValue({status: "succeeded"});
evidenceRoutes(app, { runner: { withPrincipal }, registryRoot: "/data/registry", hostRegistryRoot: "/srv/Thoth workspaces",
registry: { list: async () => [{ id: "sales", commit: "a".repeat(40), blob: "b".repeat(40), snapshotPath: "/missing" }] },
service: { consolidateEvidence, evidenceSources } });
return { app, run, withPrincipal, principal, consolidateEvidence, evidenceSources };
}
test("browses complete local files with host paths, without a database runtime", async () => {
const { app, run, withPrincipal, principal } = setup();
const response = await app.inject("/workspaces/sales/evidence?kind=domain&concept=orders&page=2");
expect(response.statusCode).toBe(200);
expect(withPrincipal).toHaveBeenCalledWith(principal);
const [argv, snapshot] = run.mock.calls[0] as unknown as [string[], string];
expect(argv).toEqual(["evidence", "admin", "--workspace", "sales"]);
expect(JSON.parse(snapshot)).toEqual({ root: "/data/registry/repo/sales", query: { kind: "domain", concept: "orders", page: 2 } });
expect(response.json().location.workspace).toBe("/srv/Thoth workspaces/repo/sales");
expect(response.json().location.git_commands[0]).toContain("git -C '/srv/Thoth workspaces/repo'");
});
test("source actions require admin, bind the actor, and reject cross-workspace or arbitrary inputs", async () => {
const denied = setup(false);
expect((await denied.app.inject({method: "POST", url: "/workspaces/sales/evidence/sources", payload: {action: "refresh"}})).statusCode).toBe(403);
expect(denied.evidenceSources).not.toHaveBeenCalled();
const allowed = setup();
expect((await allowed.app.inject({method: "POST", url: "/workspaces/sales/evidence/sources", payload: {action: "refresh"}})).statusCode).toBe(200);
expect(allowed.evidenceSources).toHaveBeenCalledWith({workspaceId: "sales", action: "refresh", actor: "curator"});
const choice = {action: "decide", sourceId: "a".repeat(64), revision: "b".repeat(64), decision: "keep"};
expect((await allowed.app.inject({method: "POST", url: "/workspaces/sales/evidence/sources", payload: choice})).statusCode).toBe(200);
expect(allowed.evidenceSources).toHaveBeenLastCalledWith({...choice, workspaceId: "sales", actor: "curator"});
for (const payload of [{action: "refresh", url: "http://localhost"}, {...choice, actor: "forged"}, {...choice, revision: "../other"}]) {
expect((await allowed.app.inject({method: "POST", url: "/workspaces/sales/evidence/sources", payload})).statusCode).toBe(400);
}
expect((await allowed.app.inject({method: "POST", url: "/workspaces/other/evidence/sources", payload: choice})).statusCode).toBe(404);
expect(allowed.evidenceSources).toHaveBeenCalledTimes(2);
});
test.each(["/", "/evidence:rule", "/consolidate"])("refuses non-admin access %s", async suffix => {
const { app, run, consolidateEvidence } = setup(false);
const response = await app.inject({ method: suffix === "/consolidate" ? "POST" : "GET", url: `/workspaces/sales/evidence${suffix === "/" ? "" : suffix}` });
expect(response.statusCode).toBe(403); expect(run).not.toHaveBeenCalled(); expect(consolidateEvidence).not.toHaveBeenCalled();
});
test("rejects workspace escape, unknown workspace and unsupported filters", async () => {
const { app, run } = setup();
expect((await app.inject("/workspaces/foreign/evidence")).statusCode).toBe(404);
expect((await app.inject("/workspaces/sales/evidence?root=/private")).statusCode).toBe(400);
expect((await app.inject("/workspaces/sales/evidence?page_size=101")).statusCode).toBe(400);
expect(run).not.toHaveBeenCalled();
});
test("a missing unit is 404 and consolidation preserves the partial outcome", async () => {
const { app, consolidateEvidence } = setup();
expect((await app.inject("/workspaces/sales/evidence/evidence:missing")).statusCode).toBe(404);
consolidateEvidence.mockResolvedValue({ status: "failed", warnings: ["Saved; retry indexing."] });
const response = await app.inject({ method: "POST", url: "/workspaces/sales/evidence/consolidate" });
expect(response.json()).toEqual({ status: "failed", warnings: ["Saved; retry indexing."] });
});
+33 -1
View File
@@ -56,7 +56,7 @@ test("catalog listing translates upstream Pi IDs back to canonical model keys",
session: { reasoning: true, contextWindow: 32768, maxTokens: 8192 },
};
const modelCatalog: RuntimeModelCatalog = {
defaultSession: model.id, defaultMetadataGeneration: null, embedding: null,
defaultInteraction: model.id, embedding: null,
sessionModels: () => [model], metadataModels: () => [], hasSession: (id) => id === model.id,
};
try {
@@ -75,6 +75,38 @@ test("catalog listing translates upstream Pi IDs back to canonical model keys",
}
});
test("catalog listing supplies the shared DeepSeek key without requiring Pi auth", async () => {
const script = scriptWith([{ provider: "deepseek", id: "deepseek-v4-pro", name: "DeepSeek V4 Pro" }]);
const secret = join(path.dirname(script), "thothii.secrets");
writeFileSync(secret, "DEEPSEEK_API_KEY=shared-key\nOPENAI_API_KEY=unrelated-key\n", { mode: 0o600 });
const model: RuntimeModel = {
id: "deepseek/deepseek-v4-pro", provider: "deepseek", model: "deepseek-v4-pro",
label: "DeepSeek V4 Pro", upstreamModel: "deepseek-v4-pro",
authentication: { mode: "secret_env", apiKeyEnv: "DEEPSEEK_API_KEY" },
sessionAdapter: { mode: "pi_builtin" }, session: { reasoning: false },
};
const modelCatalog: RuntimeModelCatalog = {
defaultInteraction: model.id, embedding: null,
sessionModels: () => [model], metadataModels: () => [], hasSession: (id) => id === model.id,
};
try {
const lister = createPiModelLister(loadConfig({ THT_SECRETS_FILE: secret }), {
...noManagedModels, modelCatalog, loadEnabledModels: enabled(model.id),
spawnFn: (_command, _args, options) => {
expect(options.env.DEEPSEEK_API_KEY).toBe("shared-key");
expect(options.env).not.toHaveProperty("OPENAI_API_KEY");
expect(options.env).not.toHaveProperty("THT_SECRETS_FILE");
return spawn("node", [FAKE, script], { env: options.env }) as any;
},
});
await expect(lister()).resolves.toEqual([{
provider: model.provider, id: model.model, name: model.label, reasoning: false,
}]);
} finally {
rmSync(path.dirname(script), { recursive: true, force: true });
}
});
test("createPiModelLister caches within ttl (spawns once for two calls)", async () => {
const script = scriptWith([{ provider: "zai", id: "glm-5.2", name: "GLM 5.2", reasoning: true }]);
try {
+81
View File
@@ -0,0 +1,81 @@
import Fastify from "fastify";
import { afterEach, expect, test, vi } from "vitest";
import { memoryRoutes } from "../src/routes/memory.js";
import { DEFAULT_SEMANTIC_RUNTIME } from "../src/workspaces/runtime-renderer.js";
import type { ThtRunner } from "../src/tht/tht-runner.js";
import type { PrincipalContext } from "../src/auth/principal.js";
const apps: ReturnType<typeof Fastify>[] = [];
afterEach(async () => { await Promise.all(apps.splice(0).map(app => app.close())); });
const id = "mem-11111111-1111-4111-8111-111111111111";
const input = { family: "domain_clarification", subject: "Order", scope: "Sales" };
function setup(admin = true) {
const app = Fastify(); apps.push(app);
const principal: PrincipalContext = { issuer: "test", subject: "operator", roles: [admin ? "admin" : "user"],
permissions: admin ? ["memory.manage"] : ["session.use"], isAdmin: admin };
app.addHook("preHandler", async request => { request.principal = principal; });
const run = vi.fn(async () => ({ code: 0, stdout: JSON.stringify({ items: [], total: 0 }), stderr: "" }));
const bound = { runWithRuntimeSnapshot: run } as unknown as ThtRunner;
const withPrincipal = vi.fn(() => bound);
memoryRoutes(app, { runner: { withPrincipal }, runtime: DEFAULT_SEMANTIC_RUNTIME,
registry: {
list: async () => [{ id: "sales", commit: "a".repeat(40), blob: "b".repeat(40), snapshotPath: "/missing" }],
read: vi.fn().mockResolvedValue({ workspace: { workspace: { language: "it" } } }),
} });
return { app, run, withPrincipal, principal };
}
test("list binds principal and workspace without requiring a DWH runtime", async () => {
const { app, run, withPrincipal, principal } = setup();
const response = await app.inject("/workspaces/sales/memory?q=Order&page=2&column=id");
expect(response.statusCode).toBe(200);
expect(withPrincipal).toHaveBeenCalledWith(principal);
const [args, snapshot] = run.mock.calls[0] as unknown as [string[], string];
expect(args).toEqual(["memory", "admin", "--workspace", "sales"]);
expect(JSON.parse(snapshot)).toMatchObject({ action: "list", request: { q: "Order", page: 2, column: "id" } });
expect(JSON.parse(snapshot).runtime.memoryLanguage).toBe("it");
});
test.each([
["GET", ""], ["POST", ""], ["GET", "/pending"], ["GET", `/${id}`],
["PUT", `/${id}`], ["DELETE", `/${id}`], ["POST", `/${id}/retry`],
] as const)("non-admin cannot %s %s", async (method, suffix) => {
const { app, run } = setup(false);
const response = await app.inject({ method, url: `/workspaces/sales/memory${suffix}`,
...(method === "PUT" || method === "POST" && suffix === "" ? { payload: input } : {}) });
expect(response.statusCode).toBe(403); expect(run).not.toHaveBeenCalled();
});
test("unknown workspace and invalid input do not invoke the harness", async () => {
const { app, run } = setup();
expect((await app.inject("/workspaces/missing/memory")).statusCode).toBe(404);
expect((await app.inject("/workspaces/sales/memory?page_size=101")).statusCode).toBe(400);
expect((await app.inject({ method: "POST", url: "/workspaces/sales/memory",
payload: { ...input, workspace_id: "foreign" } })).statusCode).toBe(400);
expect(run).not.toHaveBeenCalled();
});
test("create preserves saved-but-unindexed outcome, update carries complete related changes", async () => {
const { app, run } = setup();
run.mockResolvedValue({ code: 0, stdout: JSON.stringify({ id, saved: true, indexed: false }), stderr: "" });
const result = await app.inject({ method: "POST", url: "/workspaces/sales/memory", payload: input });
expect(result.statusCode).toBe(201); expect(result.json()).toEqual({ id, saved: true, indexed: false });
await app.inject({ method: "PUT", url: `/workspaces/sales/memory/${id}`, payload: {
...input, links: [{ target_id: id, meaning: "Related" }], dependencies: [{ database: "dwh" }],
} });
const [, snapshot] = run.mock.calls[1] as unknown as [string[], string];
expect(JSON.parse(snapshot)).toMatchObject({ action: "update", request: { id, card: {
links: [{ target_id: id, meaning: "Related" }], dependencies: [{ database: "dwh" }],
} } });
});
test.each([["memory_not_found", 404], ["memory_conflict", 409], ["memory_unavailable", 503]])(
"maps %s without leaking stderr", async (code, status) => {
const { app, run } = setup();
run.mockResolvedValue({ code: 1, stdout: JSON.stringify({ code, message: "sensitive detail" }), stderr: "secret" });
const result = await app.inject("/workspaces/sales/memory");
expect(result.statusCode).toBe(status); expect(result.json().code).toBe(code);
expect(result.body).not.toMatch(/secret|sensitive/);
},
);
@@ -20,9 +20,8 @@ function runtimeCatalog(overrides: Record<string, unknown> = {}, secrets = "OPEN
const catalogFile = join(root, "catalog.json");
const secretsFile = join(root, "thothii.secrets");
const catalog = {
schemaVersion: 1,
defaultSession: "zai/glm-5.3",
defaultMetadataGeneration: "zai/glm-5.3",
schemaVersion: 2,
defaultInteraction: "zai/glm-5.3",
embedding: { id: "ollama/qwen3-embedding:0.6b", dimensions: 1024 },
models: [
{
@@ -55,8 +54,8 @@ test("loads session default and safe metadata choices from the normalized runtim
const runtime = loadRuntimeModelCatalog(catalogFile);
const metadata = loadMetadataGenerationModels({ catalogFile, secretsFile });
expect(runtime.defaultSession).toBe("zai/glm-5.3");
expect(runtime.hasSession("deepseek/deepseek-v4-pro")).toBe(true);
expect(runtime.defaultInteraction).toBe("zai/glm-5.3");
expect(runtime.hasSession("deepseek/deepseek-v4-pro")).toBe(false);
expect(metadata.catalog()).toEqual({
models: [{ id: "zai/glm-5.3", label: "GLM 5.3" }],
default: "zai/glm-5.3",
@@ -69,14 +68,36 @@ test("loads session default and safe metadata choices from the normalized runtim
expect(() => metadata.resolve("zai/missing")).toThrow(MetadataGenerationModelUnavailableError);
});
test("DeepSeek uses one identity and bundle credential for native Pi and LiteLLM", () => {
const models = ["deepseek-v4-pro", "deepseek-v4-flash"].map((model) => ({
id: `deepseek/${model}`, provider: "deepseek", model, label: model, upstreamModel: model,
authentication: { mode: "secret_env", apiKeyEnv: "DEEPSEEK_API_KEY" },
sessionAdapter: { mode: "pi_builtin" }, metadataAdapter: { litellmProvider: "deepseek" },
session: { reasoning: false }, metadataGeneration: { disableThinking: false },
}));
const files = runtimeCatalog({ defaultInteraction: models[0].id, models }, "DEEPSEEK_API_KEY=shared-key\n");
const runtime = loadRuntimeModelCatalog(files.catalogFile);
const metadata = loadMetadataGenerationModels(files);
expect(runtime.sessionModels().map((model) => model.id)).toEqual(metadata.catalog().models.map((model) => model.id));
expect(metadata.catalog().default).toBe(runtime.defaultInteraction);
for (const model of models) {
expect(metadata.resolve(model.id)).toMatchObject({
id: model.id, provider: "deepseek", model: model.model,
apiKeyEnv: "DEEPSEEK_API_KEY", apiKey: "shared-key",
});
}
expect(() => metadata.resolve("deepseek-metadata/deepseek-v4-pro"))
.toThrow(MetadataGenerationModelUnavailableError);
});
test("returns empty catalogs when no runtime projection is configured", () => {
expect(loadRuntimeModelCatalog().defaultSession).toBeNull();
expect(loadRuntimeModelCatalog().defaultInteraction).toBeNull();
expect(loadMetadataGenerationModels({}).catalog()).toEqual({ models: [], default: null });
});
test("rejects a drifted default and an unprotected projection", () => {
const drifted = runtimeCatalog({ defaultSession: "zai/missing" });
expect(() => loadRuntimeModelCatalog(drifted.catalogFile)).toThrow("session default is invalid");
const drifted = runtimeCatalog({ defaultInteraction: "zai/missing" });
expect(() => loadRuntimeModelCatalog(drifted.catalogFile)).toThrow("interaction default is invalid");
const unprotected = runtimeCatalog();
chmodSync(unprotected.catalogFile, 0o666);
@@ -85,7 +106,7 @@ test("rejects a drifted default and an unprotected projection", () => {
test("rejects authentication semantics that cannot come from the installation catalog", () => {
const invalid = runtimeCatalog({
defaultMetadataGeneration: undefined,
defaultInteraction: undefined,
models: [{
id: "zai/glm-5.3",
provider: "zai",
@@ -112,3 +133,13 @@ test("splits canonical session identities without provider aliases", () => {
expect(splitCanonicalModelId("zai/glm-5.3")).toEqual({ provider: "zai", model: "glm-5.3" });
expect(() => splitCanonicalModelId("glm-5.3")).toThrow("model identity is invalid");
});
test("rejects a default supported by only one configured use", () => {
const { catalogFile } = runtimeCatalog({ defaultInteraction: "deepseek/deepseek-v4-pro" });
expect(() => loadRuntimeModelCatalog(catalogFile)).toThrow("interaction default is invalid");
});
test("requires regenerated runtime schema v2 rather than interpreting two legacy defaults", () => {
const { catalogFile } = runtimeCatalog({ schemaVersion: 1, defaultSession: "zai/glm-5.3" });
expect(() => loadRuntimeModelCatalog(catalogFile)).toThrow("runtime model catalog is invalid");
});
+18
View File
@@ -5,6 +5,8 @@ const fakes = vi.hoisted(() => ({
catalogRepository: { close: vi.fn(async () => {}) },
createCatalogRepository: vi.fn(),
runnerConfig: undefined as Record<string, unknown> | undefined,
acquireWorkspaceRuntime: vi.fn(),
releaseWorkspaceRuntime: vi.fn(),
run: vi.fn(async () => ({
code: 0,
stdout: JSON.stringify({ ok: true }),
@@ -24,6 +26,8 @@ vi.mock("../src/tht/tht-runner.js", () => ({
run = fakes.run;
acquireWorkspaceRuntime = fakes.acquireWorkspaceRuntime;
withPrincipal() {
return this;
}
@@ -67,6 +71,12 @@ beforeEach(() => {
fakes.createCatalogRepository.mockReset();
fakes.createCatalogRepository.mockReturnValue(fakes.catalogRepository);
fakes.run.mockClear();
fakes.acquireWorkspaceRuntime.mockReset();
fakes.acquireWorkspaceRuntime.mockResolvedValue({
path: "/data/workspace-registry/snapshots/runtime/workspace.yaml",
release: fakes.releaseWorkspaceRuntime,
});
fakes.releaseWorkspaceRuntime.mockClear();
fakes.runnerConfig = undefined;
});
@@ -78,5 +88,13 @@ test("workflow doctor gives schema-v4 runtime rendering a live Catalog repositor
expect(fakes.createCatalogRepository).toHaveBeenCalledWith(config.catalogDatabase);
expect(fakes.runnerConfig?.catalogRepository).toBe(fakes.catalogRepository);
expect(fakes.acquireWorkspaceRuntime).toHaveBeenCalledWith(
"/data/workspace-registry/snapshots/revision/workspace.yaml",
);
expect(fakes.run).toHaveBeenCalledWith(
["doctor", "--json"],
"/data/workspace-registry/snapshots/runtime/workspace.yaml",
);
expect(fakes.releaseWorkspaceRuntime).toHaveBeenCalledOnce();
expect(fakes.catalogRepository.close).toHaveBeenCalledOnce();
});
+43 -4
View File
@@ -1,4 +1,4 @@
import { mkdtempSync } from "node:fs";
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { expect, test, vi } from "vitest";
@@ -7,7 +7,34 @@ import {
createPiManagement,
type PiExecFile,
} from "../src/pi/management.js";
import type { RuntimeModelCatalog } from "../src/models/runtime-model-catalog.js";
import type { RuntimeModel, RuntimeModelCatalog } from "../src/models/runtime-model-catalog.js";
test.each([false, true])("catalog credential status ignores legacy Pi auth, missing bundle key: %s", async (missingKey) => {
const root = mkdtempSync(join(tmpdir(), "tht-catalog-status-"));
writeFileSync(join(root, "auth.json"), JSON.stringify({ deepseek: { type: "api_key", key: "stale-key" } }), { mode: 0o600 });
const secret = join(root, "thothii.secrets");
writeFileSync(secret, missingKey ? "THT_MODEL_API_KEY=legacy-key\n" : "DEEPSEEK_API_KEY=shared-key\n", { mode: 0o600 });
vi.stubEnv("PI_CODING_AGENT_DIR", root);
const model: RuntimeModel = {
id: "deepseek/deepseek-v4-pro", provider: "deepseek", model: "deepseek-v4-pro",
label: "DeepSeek", upstreamModel: "deepseek-v4-pro",
authentication: { mode: "secret_env", apiKeyEnv: "DEEPSEEK_API_KEY" },
sessionAdapter: { mode: "pi_builtin" }, session: { reasoning: false },
};
try {
const service = createPiManagement(loadConfig({ THT_SECRETS_FILE: secret }), {
modelCatalog: {
defaultInteraction: model.id, embedding: null,
sessionModels: () => [model], metadataModels: () => [], hasSession: (id) => id === model.id,
},
execute: successfulExec([]), readSettings: () => ({ thinking: "medium" }),
});
expect((await service.status()).credentials).toBe(missingKey ? "missing" : "present");
} finally {
vi.unstubAllEnvs();
rmSync(root, { recursive: true, force: true });
}
});
function configFor(settingsFile = join(mkdtempSync(join(tmpdir(), "tht-pi-management-")), "settings.json")) {
return loadConfig({
@@ -15,12 +42,12 @@ function configFor(settingsFile = join(mkdtempSync(join(tmpdir(), "tht-pi-manage
SETTINGS_FILE: settingsFile,
PI_BIN: "/usr/local/bin/pi",
PI_MANAGEMENT_TIMEOUT_MS: "750",
THT_HOST_PLATFORM: "linux",
});
}
const modelCatalog: RuntimeModelCatalog = {
defaultSession: "zai/glm-5.2",
defaultMetadataGeneration: null,
defaultInteraction: "zai/glm-5.2",
embedding: { id: "ollama/qwen3-embedding:0.6b", dimensions: 1024 },
sessionModels: () => [],
metadataModels: () => [],
@@ -34,6 +61,16 @@ function successfulExec(calls: Array<{ command: string; args: string[]; timeout:
};
}
test.each([
["linux", "linux"], ["darwin", "macos"], ["windows", "windows"],
])("reports installation host %s independently of the backend container OS", async (host, expected) => {
const service = createPiManagement(loadConfig({ THT_HOST_PLATFORM: host }), {
modelCatalog, execute: successfulExec([]), readSettings: () => ({ thinking: "medium" }),
credentialStatus: () => "missing",
});
expect((await service.status()).hostPlatform).toBe(expected);
});
// Catches a Pi executable that emits unexpected text or is invoked through a shell, which could
// turn a version display into a command-injection or information-disclosure surface.
test("status parses only a Pi version from a fixed execFile argument array", async () => {
@@ -47,6 +84,7 @@ test("status parses only a Pi version from a fixed execFile argument array", asy
});
await expect(service.status()).resolves.toEqual({
hostPlatform: "linux",
version: "0.80.3",
ready: true,
credentials: "missing",
@@ -78,6 +116,7 @@ test.each(["present", "missing"] as const)(
const status = await service.status();
expect(status).toEqual({
hostPlatform: "linux",
version: "0.80.3",
ready: true,
credentials,
+54 -16
View File
@@ -190,6 +190,29 @@ test("Pi receives the leased workspace runtime config and releases it on direct
expect(release).toHaveBeenCalledOnce();
});
test("Pi language launch hint comes from the session options and never ambient environment", () => {
const previous = process.env.THT_INTERACTION_LANGUAGE;
process.env.THT_INTERACTION_LANGUAGE = "it";
const environments: NodeJS.ProcessEnv[] = [];
const mgr = new PiProcessManager(loadConfig({}), {
spawnFn: (_command, _args, options) => {
environments.push(options.env);
return recordingChild() as any;
},
});
try {
mgr.createFor("explicit-language", { interactionLanguage: "en" });
mgr.teardown("explicit-language");
mgr.createFor("manifest-resolved-in-gate");
mgr.teardown("manifest-resolved-in-gate");
expect(environments[0].THT_INTERACTION_LANGUAGE).toBe("en");
expect(environments[1]).not.toHaveProperty("THT_INTERACTION_LANGUAGE");
} finally {
if (previous === undefined) delete process.env.THT_INTERACTION_LANGUAGE;
else process.env.THT_INTERACTION_LANGUAGE = previous;
}
});
test("a close-only child event releases its temporary Pi agent snapshot", () => {
const child = recordingChild();
let snapshotDir: string | undefined;
@@ -642,27 +665,33 @@ test("session Pi spawn reads the single secret bundle and scrubs its path", asyn
}
});
test("session Pi spawn resolves the selected catalog credential from the secret bundle", () => {
test.each([false, true])("session Pi uses the catalog bundle despite stale Pi auth: %s", (missingKey) => {
const root = mkdtempSync(path.join(tmpdir(), "thothii-catalog-credential-"));
const agentDir = path.join(root, "agent");
mkdirSync(agentDir, { mode: 0o700 });
writeFileSync(path.join(agentDir, "auth.json"), "{}\n", { mode: 0o600 });
const originalAuth = JSON.stringify({
deepseek: { type: "api_key", key: "stale-key" },
anthropic: { type: "api_key", key: "unrelated-key" },
});
writeFileSync(path.join(agentDir, "auth.json"), originalAuth, { mode: 0o600 });
writeFileSync(path.join(agentDir, "models.json"), '{"providers":{}}\n', { mode: 0o600 });
const secret = path.join(root, "thothii.secrets");
writeFileSync(secret, "ZAI_API_KEY=catalog-secret\nTHT_MODEL_API_KEY=legacy-secret\n", { mode: 0o600 });
writeFileSync(secret, missingKey ? "THT_MODEL_API_KEY=legacy-secret\n"
: "DEEPSEEK_API_KEY=catalog-secret\nTHT_MODEL_API_KEY=legacy-secret\n", { mode: 0o600 });
const legacyKey = path.join(root, "legacy-key");
writeFileSync(legacyKey, "legacy-file-key", { mode: 0o600 });
const model: RuntimeModel = {
id: "openai/test-model",
provider: "openai",
model: "test-model",
label: "Test model",
upstreamModel: "test-model",
authentication: { mode: "secret_env", apiKeyEnv: "ZAI_API_KEY" },
id: "deepseek/deepseek-v4-pro",
provider: "deepseek",
model: "deepseek-v4-pro",
label: "DeepSeek V4 Pro",
upstreamModel: "deepseek-v4-pro",
authentication: { mode: "secret_env", apiKeyEnv: "DEEPSEEK_API_KEY" },
sessionAdapter: { mode: "pi_builtin" },
session: { reasoning: false },
};
const modelCatalog: RuntimeModelCatalog = {
defaultSession: model.id,
defaultMetadataGeneration: null,
defaultInteraction: model.id,
embedding: null,
sessionModels: () => [model],
metadataModels: () => [],
@@ -672,17 +701,26 @@ test("session Pi spawn resolves the selected catalog credential from the secret
const child = recordingChild();
child.stderr.resume = () => {};
vi.stubEnv("PI_CODING_AGENT_DIR", agentDir);
const mgr = new PiProcessManager(loadConfig({ THT_SECRETS_FILE: secret }), {
const mgr = new PiProcessManager(loadConfig({ THT_SECRETS_FILE: secret, THT_MODEL_API_KEY_FILE: legacyKey }), {
modelCatalog,
authProviders: () => new Set(),
authProviders: () => new Set(["deepseek"]),
spawnFn: (...args: any[]) => { calls.push(args); return child as any; },
});
try {
mgr.createFor("catalog-credential", { provider: "openai", model: "test-model" });
expect(calls[0][2].env.ZAI_API_KEY).toBe("catalog-secret");
const create = () => mgr.createFor("catalog-credential", { provider: model.provider, model: model.model });
if (missingKey) {
expect(create).toThrow("model provider credential is unavailable");
expect(calls).toHaveLength(0);
return;
}
create();
expect(calls[0][2].env.DEEPSEEK_API_KEY).toBe("catalog-secret");
expect(calls[0][2].env).not.toHaveProperty("OPENAI_API_KEY");
expect(calls[0][2].env).not.toHaveProperty("THT_MODEL_API_KEY");
expect(JSON.parse(readFileSync(path.join(calls[0][2].env.PI_CODING_AGENT_DIR, "auth.json"), "utf8")))
.toEqual({ anthropic: { type: "api_key", key: "unrelated-key" } });
} finally {
expect(readFileSync(path.join(agentDir, "auth.json"), "utf8")).toBe(originalAuth);
mgr.teardown("catalog-credential");
vi.unstubAllEnvs();
rmSync(root, { recursive: true, force: true });
@@ -750,7 +788,7 @@ test("set_model translates a canonical catalog key to its upstream Pi model ID",
session: { reasoning: false, contextWindow: 32768, maxTokens: 8192 },
};
const modelCatalog: RuntimeModelCatalog = {
defaultSession: model.id, defaultMetadataGeneration: null, embedding: null,
defaultInteraction: model.id, embedding: null,
sessionModels: () => [model], metadataModels: () => [], hasSession: (id) => id === model.id,
};
const mgr = new PiProcessManager(loadConfig({ PI_BIN: "/usr/local/bin/pi" }), {
+7 -4
View File
@@ -61,20 +61,22 @@ test("provider smoke resolves the selected catalog credential from the secret bu
session: { reasoning: false },
};
const modelCatalog: RuntimeModelCatalog = {
defaultSession: model.id,
defaultMetadataGeneration: null,
defaultInteraction: model.id,
embedding: null,
sessionModels: () => [model],
metadataModels: () => [],
hasSession: (id) => id === model.id,
};
let spawnEnv: NodeJS.ProcessEnv | undefined;
const readAuthStore = vi.fn(() => JSON.stringify({ openai: { type: "api_key", key: "stale-key" } }));
const smoke = createPiProviderSmoke(loadConfig({ THT_SECRETS_FILE: secret }), {
modelCatalog,
authProviders: () => new Set(),
authProviders: () => new Set(["openai"]),
readAuthStore,
readModelsStore: () => undefined,
spawnFn: (_command, _args, options) => {
spawnEnv = options.env;
expect(existsSync(join(options.env.PI_CODING_AGENT_DIR!, "auth.json"))).toBe(false);
return successfulProviderChild();
},
});
@@ -85,6 +87,7 @@ test("provider smoke resolves the selected catalog credential from the secret bu
expect(spawnEnv?.ZAI_API_KEY).toBe("catalog-secret");
expect(spawnEnv).not.toHaveProperty("OPENAI_API_KEY");
expect(spawnEnv).not.toHaveProperty("THT_MODEL_API_KEY");
expect(readAuthStore).not.toHaveBeenCalled();
} finally {
rmSync(root, { recursive: true, force: true });
}
@@ -306,7 +309,7 @@ test("provider smoke makes one configured request from an isolated no-capability
sessionAdapter: { mode: "pi_builtin" }, session: { reasoning: true },
};
const smokeCatalog: RuntimeModelCatalog = {
defaultSession: smokeModel.id, defaultMetadataGeneration: null, embedding: null,
defaultInteraction: smokeModel.id, embedding: null,
sessionModels: () => [smokeModel], metadataModels: () => [],
hasSession: (id) => id === smokeModel.id,
};
+259 -54
View File
@@ -27,10 +27,9 @@ function operationalWorkspace(id = "default") {
} as const;
}
function sessionCatalog(defaultSession = "zai/glm-5.2", available = [defaultSession]) {
function sessionCatalog(defaultInteraction = "zai/glm-5.2", available = [defaultInteraction]) {
return {
defaultSession,
defaultMetadataGeneration: null,
defaultInteraction,
embedding: { id: "ollama/qwen3-embedding:0.6b", dimensions: 1024 },
sessionModels: () => [],
metadataModels: () => [],
@@ -54,7 +53,11 @@ const defaultWorkspaceRegistry = {
function buildApp(config: Parameters<typeof buildRealApp>[0], deps: Record<string, unknown> = {}) {
const thtRunner = deps.thtRunner
? { qdrantEnsure: async () => ({ ok: true }), ...(deps.thtRunner as object) }
? {
qdrantEnsure: async () => ({ ok: true }),
ensureInteractionLanguage: async () => ({ interaction_language: "en" }),
...(deps.thtRunner as object),
}
: undefined;
return buildRealApp(config, {
workspaceRuntimeSupport: () => true,
@@ -89,6 +92,129 @@ const aliceHeaders = {
"x-thoth-is-admin": "0",
};
test.each([undefined, null, "", "en--US", "en_US", "en\nIGNORE", "en<script>", 42])(
"new sessions reject invalid interaction language %s before persistence", async (interactionLanguage) => {
const app = mutApp({ sessionNew: async () => { throw new Error("must not create"); } });
try {
const response = await app.inject({
method: "POST", url: "/sessions", payload: { question: "Pazienti", interactionLanguage },
});
expect(response.statusCode).toBe(400);
expect(response.json()).toMatchObject({ code: "invalid_interaction_language" });
} finally { await app.close(); }
},
);
test.each([["en", "en"], ["fr-FR", "fr-FR"], ["FR-fr", "fr-FR"]])(
"new session forwards canonical interaction language %s without changing the question", async (language, canonical) => {
let created: any;
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
thtRunner: {
sessionNew: async (options: any) => { created = options; return { id: "language" }; },
searchPack: async () => {},
},
readiness: { ensure: async () => ({ ok: true }) },
mgr: {
get: () => undefined, createFor: () => ({ bridge: { onClientEvent: () => {} } }),
configure: async () => {}, start: () => {},
},
getSettings: () => ({ workspace: "default" }),
});
try {
const response = await app.inject({
method: "POST", url: "/sessions", payload: { question: "Pazienti", interactionLanguage: language },
});
expect(response.statusCode).toBe(200);
expect(created).toMatchObject({ question: "Pazienti", interactionLanguage: canonical });
} finally { await app.close(); }
});
test("resume rejects browser interaction language overrides", async () => {
const app = mutApp({ sessionShow: async () => ({ status: "open", interaction_language: "it" }) });
try {
const response = await app.inject({
method: "POST", url: "/sessions/s1/resume", payload: { interactionLanguage: "en" },
});
expect(response.statusCode).toBe(400);
expect(response.json()).toMatchObject({ code: "interaction_language_pinned" });
} finally { await app.close(); }
});
test("resume pins legacy interaction language using the resolved harness workspace", async () => {
let pinnedWorkspace: string | undefined;
let runtime: any;
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
thtRunner: {
sessionShow: async () => ({ id: "legacy", status: "closed" }),
ensureInteractionLanguage: async (_id: string, workspace: string) => {
pinnedWorkspace = workspace;
return { interaction_language: "it" };
},
reopenSession: async () => {},
},
readiness: { ensure: async () => ({ ok: true }) },
mgr: {
get: () => undefined,
createFor: (_id: string, options: any) => {
runtime = options;
return { bridge: { onClientEvent: () => {} } };
},
configure: async () => {}, start: () => {},
},
getSettings: () => ({ workspace: "other-browser-workspace" }),
});
try {
const response = await app.inject({ method: "POST", url: "/sessions/legacy/resume" });
expect(response.statusCode).toBe(200);
expect(pinnedWorkspace).toContain("/default.yaml");
expect(runtime).toMatchObject({ interactionLanguage: "it", mode: "resume" });
} finally { await app.close(); }
});
test("resume retains persisted interaction language despite different browser settings", async () => {
let options: any;
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
thtRunner: {
sessionShow: async () => ({ id: "saved", status: "closed", interaction_language: "it" }),
ensureInteractionLanguage: async () => { throw new Error("language is already pinned"); },
reopenSession: async () => {},
},
readiness: { ensure: async () => ({ ok: true }) },
mgr: {
get: () => undefined,
createFor: (_id: string, value: any) => {
options = value;
return { bridge: { onClientEvent: () => {} } };
},
configure: async () => {}, start: () => {},
},
getSettings: () => ({ workspace: "english-workspace", locale: "en" }),
});
try {
const response = await app.inject({ method: "POST", url: "/sessions/saved/resume" });
expect(response.statusCode).toBe(200);
expect(options.interactionLanguage).toBe("it");
} finally { await app.close(); }
});
test("resume cannot start Pi if legacy interaction language cannot be persisted", async () => {
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
thtRunner: {
sessionShow: async () => ({ id: "legacy", status: "closed" }),
ensureInteractionLanguage: async () => { throw new Error("private storage details"); },
reopenSession: async () => { throw new Error("must not reopen"); },
},
readiness: { ensure: async () => ({ ok: true }) },
mgr: { createFor: () => { throw new Error("must not spawn"); } },
getSettings: () => ({ workspace: "default" }),
});
try {
const response = await app.inject({ method: "POST", url: "/sessions/legacy/resume" });
expect(response.statusCode).toBe(503);
expect(response.body).not.toContain("private storage details");
} finally { await app.close(); }
});
test("upstream requests without a principal fail before a Pi runtime can be created", async () => {
let created = false;
const app = buildApp(loadConfig({ AUTH_MODE: "upstream", THT_HARNESS_DIR: "../harness" }), {
@@ -96,7 +222,7 @@ test("upstream requests without a principal fail before a Pi runtime can be crea
thtRunner: {} as any,
});
const response = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const response = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(response.statusCode).toBe(401);
expect(created).toBe(false);
@@ -134,7 +260,7 @@ test("maintenance rejects new and resumed session admission without interrupting
});
const create = await app.inject({
method: "POST", url: "/sessions", headers: aliceHeaders, payload: { question: "q" },
method: "POST", url: "/sessions", headers: aliceHeaders, payload: { interactionLanguage: "en", question: "q" },
});
const resume = await app.inject({ method: "POST", url: "/sessions/open/resume", headers: aliceHeaders });
@@ -155,7 +281,7 @@ test("a durable maintenance marker initializes admission closed after backend re
AUTH_MODE: "upstream", THT_HARNESS_DIR: "../harness", THT_MAINTENANCE_FILE: marker,
}), { thtRunner: {} as any });
const response = await app.inject({
method: "POST", url: "/sessions", headers: aliceHeaders, payload: { question: "q" },
method: "POST", url: "/sessions", headers: aliceHeaders, payload: { interactionLanguage: "en", question: "q" },
});
expect(response.statusCode).toBe(503);
expect(response.json()).toMatchObject({ code: "maintenance" });
@@ -364,6 +490,65 @@ test("retention scans a removed workspace's retained snapshot", async () => {
expect(response.json()).toEqual([expect.objectContaining({ id: "resumable" })]);
});
test("active sessions remain available when retained snapshot discovery is unreadable", async () => {
const activeSnapshot = "/registry/snapshots/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/active.yaml";
const retainedFailure = "invalid retained snapshot /registry/snapshots/secret/legacy.yaml";
const reconcileSnapshotRetention = vi.fn(async () => {});
const subscribed = vi.fn();
const warning = vi.spyOn(console, "warn").mockImplementation(() => {});
const manifest = {
id: "live-session", status: "open", archived: false,
workspace_id: "active", workspace_revision: "a".repeat(40),
};
const app = buildApp(loadConfig({ AUTH_MODE: "upstream", THT_HARNESS_DIR: "../harness" }), {
thtRunner: {
withPrincipal: () => ({
sessionList: async (snapshotPath: string) => {
expect(snapshotPath).toBe(activeSnapshot);
return [manifest];
},
sessionShow: async (id: string, snapshotPath?: string) => {
if (id === manifest.id && snapshotPath === activeSnapshot) return manifest;
throw new Error("session not found");
},
}),
} as any,
mgr: { get: () => undefined } as any,
hub: {
subscribe: (_id: string, _send: unknown, options: { close?: () => void }) => {
subscribed();
options.close?.();
return () => {};
},
} as any,
workspaceRegistry: {
listRetainedSnapshots: async () => { throw new Error(retainedFailure); },
list: async () => [{
id: "active", commit: "a".repeat(40), blob: "b".repeat(40), snapshotPath: activeSnapshot,
}],
reconcileSnapshotRetention,
} as any,
});
try {
const list = await app.inject({ method: "GET", url: "/sessions", headers: aliceHeaders });
const detail = await app.inject({ method: "GET", url: `/sessions/${manifest.id}`, headers: aliceHeaders });
const events = await app.inject({ method: "GET", url: `/sessions/${manifest.id}/events`, headers: aliceHeaders });
expect(list.statusCode).toBe(200);
expect(list.json()).toEqual([expect.objectContaining({ id: manifest.id, active: false })]);
expect(detail.statusCode).toBe(200);
expect(detail.json()).toMatchObject(manifest);
expect(events.statusCode).toBe(200);
expect(subscribed).toHaveBeenCalledOnce();
expect(reconcileSnapshotRetention).not.toHaveBeenCalled();
expect(list.body + detail.body + events.body).not.toContain(retainedFailure);
} finally {
warning.mockRestore();
await app.close();
}
});
test("the single local installation listing reconciles its resumable workspace pins", async () => {
const retained = vi.fn(async () => {});
const retainedRevision = "d".repeat(40);
@@ -434,7 +619,7 @@ test("new sessions are created through the authenticated principal, not a client
const response = await app.inject({
method: "POST", url: "/sessions", headers: aliceHeaders,
payload: { question: "q", owner: "mallory" },
payload: { interactionLanguage: "en", question: "q", owner: "mallory" },
});
expect(response.statusCode).toBe(200);
@@ -451,7 +636,7 @@ test("new sessions reject the client legacy workspace field unless local legacy
});
const response = await app.inject({
method: "POST", url: "/sessions", payload: { question: "q", workspace: "legacy" },
method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q", workspace: "legacy" },
});
expect(response.statusCode).toBe(409);
@@ -471,7 +656,7 @@ test("explicit local legacy mode permits the unpinned client workspace request",
});
const response = await app.inject({
method: "POST", url: "/sessions", payload: { question: "q", workspace: "legacy" },
method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q", workspace: "legacy" },
});
expect(response.statusCode).toBe(200);
@@ -509,7 +694,7 @@ test("creates a session from the active immutable workspace revision", async ()
await app.inject({
method: "POST", url: "/sessions",
payload: { question: "q", workspaceId: "psd-clinical", provider: "zai", model: "glm-5.2", thinking: "low" },
payload: { interactionLanguage: "en", question: "q", workspaceId: "psd-clinical", provider: "zai", model: "glm-5.2", thinking: "low" },
});
expect(sessionNew).toHaveBeenCalledWith(expect.objectContaining({
@@ -553,7 +738,7 @@ test("hands one Catalog-backed runtime to both retrieval and Pi", async () => {
const response = await app.inject({
method: "POST",
url: "/sessions",
payload: { question: "Which users placed orders?", workspaceId: "default" },
payload: { interactionLanguage: "en", question: "Which users placed orders?", workspaceId: "default" },
});
expect(response.statusCode).toBe(200);
@@ -629,7 +814,7 @@ test("refuses core admission when the workspace preprocessing fingerprint is sta
const response = await app.inject({
method: "POST",
url: "/sessions",
payload: { question: "q", workspaceId: "default" },
payload: { interactionLanguage: "en", question: "q", workspaceId: "default" },
});
expect(response.statusCode).toBe(409);
@@ -681,7 +866,7 @@ test("rejects an SSH-only Catalog binding before persisting or starting a sessio
} as any,
});
const response = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const response = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(response.statusCode).toBe(409);
expect(response.json()).toMatchObject({ code: "workspace_not_activatable" });
@@ -721,7 +906,7 @@ test("hands a revision lease to retention only after the session manifest is dur
workspaceRegistry: { acquireSessionRevision } as any,
});
const request = app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const request = app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
await new Promise((resolve) => setImmediate(resolve));
expect(markPersisted).not.toHaveBeenCalled();
expect(abort).not.toHaveBeenCalled();
@@ -752,7 +937,7 @@ test("creates a session from the configured default workspace revision when work
workspaceRegistry: registry as any,
});
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(registry.read).toHaveBeenCalledWith("psd-clinical");
expect(sessionNew).toHaveBeenCalledWith(expect.objectContaining({
@@ -838,7 +1023,7 @@ test("session lifecycle locates a B session when installation default is A", asy
} as any,
});
expect((await app.inject({ method: "POST", url: "/sessions", payload: {
expect((await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en",
question: "B question", workspaceId: "b-workspace", provider: "zai", model: "glm-5.2", thinking: "low",
} })).statusCode).toBe(200);
expect((await app.inject({ method: "GET", url: "/sessions" })).json()).toEqual([
@@ -850,7 +1035,7 @@ test("session lifecycle locates a B session when installation default is A", asy
active = undefined;
expect((await app.inject({ method: "POST", url: "/sessions/session-b/resume" })).json())
.toEqual({ id: "session-b", alreadyActive: false });
.toEqual({ id: "session-b", alreadyActive: false, workspaceId: "b-workspace" });
expect(calls).toContain(`new:${bPath}`);
expect(calls).toContain(`list:${bPath}`);
expect(calls).toContain(`show:${bPath}`);
@@ -881,7 +1066,7 @@ test("POST /sessions uses the catalog default with workspace/thinking settings a
],
spawnFn: () => nodeSpawn("node", [FAKE, SCRIPT]) as any,
});
const created = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const created = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(created.json()).toEqual({ id: "s1" });
expect(sessionNewArg.workspaceConfigPath).toContain(`/snapshots/${"e".repeat(40)}/w.yaml`);
expect(sessionNewArg.provider).toBe("zai");
@@ -942,11 +1127,11 @@ test("POST /sessions stops the user's previous open Pi runtime before creating a
getSettings: () => ({ workspace: "local" }) as any,
});
expect((await app.inject({ method: "POST", url: "/sessions", payload: { question: "one" } })).statusCode)
expect((await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "one" } })).statusCode)
.toBe(200);
order.length = 0;
const second = await app.inject({ method: "POST", url: "/sessions", payload: { question: "two" } });
const second = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "two" } });
expect(second.statusCode).toBe(200);
expect(second.json()).toEqual({ id: "s2" });
@@ -967,7 +1152,7 @@ test("POST /sessions refuses to create a session when the local DWH precheck fai
readiness: { ensure: async () => ({ ok: true }) } as any,
getSettings: () => ({ workspace: "w" }) as any,
});
const res = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const res = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(res.statusCode).toBe(503);
expect(res.json()).toMatchObject({ code: "dwh_unreachable" });
expect(pinged).toBe(1);
@@ -988,7 +1173,7 @@ test("POST /sessions proceeds past a passing DWH precheck", async () => {
getSettings: () => ({ workspace: "w" }) as any,
spawnFn: () => nodeSpawn("node", [FAKE, SCRIPT]) as any,
});
const res = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const res = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(res.statusCode).toBe(200);
expect(pinged).toBe(1);
expect(created).toBe(1);
@@ -1007,7 +1192,7 @@ test("POST /sessions skips the DWH precheck when the flag is off (default)", asy
getSettings: () => ({ workspace: "w" }) as any,
spawnFn: () => nodeSpawn("node", [FAKE, SCRIPT]) as any,
});
const res = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const res = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(res.statusCode).toBe(200);
expect(pinged).toBe(0); // probe never runs without the flag
});
@@ -1041,7 +1226,7 @@ test("POST /sessions configura Pi con il thinking globale selezionato", async ()
],
});
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
await new Promise((resolve) => setImmediate(resolve));
expect(configured.thinking).toBe("high");
@@ -1120,6 +1305,29 @@ test("POST /sessions/:id/resume uses the manifest's retained workspace revision"
);
});
test.each([undefined, { provider: "local", model: "qwen" }])("resume uses the global model/default, not the historical manifest (%j)", async (payload) => {
const configure = vi.fn(async () => {});
const manifest = { status: "open", archived: false, provider: "retired", model: "old-model" };
const runtime = { bridge: { onClientEvent: () => {}, emitClientEvent: () => {} } };
let current: any;
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
mgr: {
get: () => current,
createFor: () => { current = runtime; return runtime; },
configure, start: () => {},
} as any,
thtRunner: { sessionShow: async () => manifest, reopenSession: async () => {} } as any,
readiness: { ensure: async () => ({ ok: true }) } as any,
runtimeModelCatalog: sessionCatalog("zai/glm-5.2", ["zai/glm-5.2", "local/qwen"]),
getSettings: () => ({ workspace: "psd", thinking: "medium" }) as any,
});
const response = await app.inject({ method: "POST", url: "/sessions/model-resume/resume", ...(payload ? { payload } : {}) });
expect(response.statusCode).toBe(200);
await new Promise((resolve) => setImmediate(resolve));
expect(configure).toHaveBeenCalledWith(expect.anything(), expect.objectContaining(payload ?? { provider: "zai", model: "glm-5.2" }));
expect(manifest).toMatchObject({ provider: "retired", model: "old-model" });
});
test("POST /sessions/:id/resume returns a sanitized error when its retained revision is unavailable", async () => {
const rawFailure = "cannot read /data/workspace-registry/snapshots/secret-revision";
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
@@ -1346,7 +1554,7 @@ test("resuming a different session stops the user's previous Pi runtime", async
readiness: { ensure: async () => ({ ok: true }) } as any,
getSettings: () => ({ workspace: "local" }) as any,
});
expect((await app.inject({ method: "POST", url: "/sessions", payload: { question: "one" } })).statusCode)
expect((await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "one" } })).statusCode)
.toBe(200);
const resumed = await app.inject({ method: "POST", url: "/sessions/s2/resume" });
@@ -1734,13 +1942,13 @@ test.each([
getSettings: () => ({ workspace: "local" }) as any,
});
await app.inject({ method: "POST", url: "/sessions", payload: { question: "old" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "old" } });
if (lifecycle === "close") {
await app.inject({ method: "POST", url: "/sessions/s1/close" });
await app.inject({ method: "POST", url: "/sessions/s1/resume" });
} else {
await app.inject({ method: "DELETE", url: "/sessions/s1" });
await app.inject({ method: "POST", url: "/sessions", payload: { question: "replacement" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "replacement" } });
}
await new Promise((resolve) => setImmediate(resolve));
expect(current).toBe(replacement);
@@ -1804,7 +2012,7 @@ test.each(["resolve", "reject"] as const)(
getSettings: () => ({ workspace: "local" }) as any,
});
await app.inject({ method: "POST", url: "/sessions", payload: { question: "old" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "old" } });
await app.inject({ method: "DELETE", url: "/sessions/s1" });
published.length = 0;
@@ -1996,7 +2204,7 @@ test("Close suppresses a bootstrap that settles while close persistence is pendi
getSettings: () => ({ workspace: "local" }) as any,
});
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
const closeResponse = app.inject({ method: "POST", url: "/sessions/s1/close" });
await closeStarted;
published.length = 0;
@@ -2070,7 +2278,7 @@ test("bootstrap failure persists once and keeps Resume serialized behind that pe
getSettings: () => ({ workspace: "local" }) as any,
});
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
oldConfigure.reject(new Error("configure failed"));
await failureStarted;
const resumeResponse = app.inject({ method: "POST", url: "/sessions/s1/resume" })
@@ -2193,7 +2401,7 @@ test("a replaced runtime cannot publish or fail the newly resumed session", asyn
readiness: { ensure: async () => ({ ok: true }) } as any,
getSettings: () => ({ workspace: "local" }) as any,
});
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
await new Promise((resolve) => setImmediate(resolve));
oldBridge.setState("idle");
@@ -2253,7 +2461,7 @@ test("a deleted runtime cannot repopulate or fail the forgotten session", async
readiness: { ensure: async () => ({ ok: true }) } as any,
getSettings: () => ({ workspace: "local" }) as any,
});
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
await new Promise((resolve) => setImmediate(resolve));
const response = await app.inject({ method: "DELETE", url: "/sessions/s1" });
@@ -2298,7 +2506,7 @@ test("an unexpectedly exited runtime publishes its terminal sequence then releas
readiness: { ensure: async () => ({ ok: true }) } as any,
getSettings: () => ({ workspace: "local" }) as any,
});
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
await new Promise((resolve) => setImmediate(resolve));
published.length = 0;
@@ -2349,7 +2557,7 @@ test("agent_end releases the Pi runtime after the session was finalized", async
readiness: { ensure: async () => ({ ok: true }) } as any,
getSettings: () => ({ workspace: "local" }) as any,
});
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
await new Promise((resolve) => setImmediate(resolve));
bridge.emitClientEvent({ type: "system_event", event: "agent_end" });
@@ -2421,7 +2629,7 @@ test("POST /sessions/:id/response senza gate pendente risponde 409 (risposta sta
getSettings: () => ({ workspace: "w" }),
spawnFn: () => nodeSpawn("node", [FAKE, SCRIPT]) as any,
});
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
// The fake Pi never emitted a ui_request: the bridge has no pending descriptor, so a
// response (stale UI, double submit) must be rejected instead of forwarded to Pi.
const res = await app.inject({ method: "POST", url: "/sessions/s1/response",
@@ -2562,7 +2770,7 @@ test("POST /sessions readiness failure returns one fixed public message without
getSettings: () => ({ workspace: "psd" }) as any,
spawnFn: () => nodeSpawn("node", [FAKE, SCRIPT]) as any,
});
const res = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const res = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(res.statusCode).toBe(503);
expect(res.json()).toEqual({
error: "Session services are not ready. Check configuration and connectivity, then try again.",
@@ -2583,7 +2791,7 @@ test.each(["semantic_index_incompatible", "workspace_not_activatable"] as const)
});
const response = await app.inject({
method: "POST", url: "/sessions", payload: { question: "q" },
method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" },
});
expect(response.statusCode).toBe(503);
@@ -2606,7 +2814,7 @@ test("POST /sessions returns storage 503 before creating a Pi runtime when sessi
mgr: { createFor: () => { piCreated = true; throw new Error("must not spawn"); } } as any,
});
const response = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const response = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(response.statusCode).toBe(503);
expect(response.json()).toEqual({ error: "session storage is unavailable" });
@@ -2626,13 +2834,13 @@ test("POST /sessions proceeds when ollamaEnsure succeeds", async () => {
getSettings: () => ({ workspace: "psd" }) as any,
spawnFn: () => nodeSpawn("node", [FAKE, SCRIPT]) as any,
});
const res = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const res = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(res.json()).toEqual({ id: "s1" });
expect(qdrantEnsure).toHaveBeenCalledWith(operationalWorkspace("psd"), 60, "self_heal");
expect(ensureWs).toContain(`/snapshots/${"e".repeat(40)}/psd.yaml`);
});
test("POST /sessions falls back from a stale requested model to the catalog default", async () => {
test("POST /sessions rejects a stale requested model without silently using the default", async () => {
let created = 0;
let persisted: any;
const runtime = { bridge: { onClientEvent: () => {} } };
@@ -2659,16 +2867,13 @@ test("POST /sessions falls back from a stale requested model to the catalog defa
const res = await app.inject({
method: "POST",
url: "/sessions",
payload: { question: "q", provider: "deepseek", model: "deepseek-v4-pro" },
payload: { interactionLanguage: "en", question: "q", provider: "deepseek", model: "deepseek-v4-pro" },
});
expect(res.statusCode).toBe(200);
expect(res.json()).toEqual({
id: "fallback",
warning: "Configured model deepseek/deepseek-v4-pro is unavailable; using zai/glm-5.2.",
});
expect(persisted).toMatchObject({ provider: "zai", model: "glm-5.2" });
expect(created).toBe(1);
expect(res.statusCode).toBe(503);
expect(res.json()).toMatchObject({ code: "model_unavailable" });
expect(persisted).toBeUndefined();
expect(created).toBe(0);
});
test("POST /sessions marks a persisted session failed when runtime construction throws", async () => {
@@ -2697,7 +2902,7 @@ test("POST /sessions marks a persisted session failed when runtime construction
],
});
const res = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const res = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(res.statusCode).toBe(503);
expect(res.json()).toEqual({
@@ -2790,7 +2995,7 @@ test.each([
try {
const response = flow === "new"
? await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } })
? await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } })
: await app.inject({ method: "POST", url: `/sessions/${sessionId}/resume` });
const logs = consoleError.mock.calls.flat().map(String).join(" ");
@@ -2896,7 +3101,7 @@ test("POST /sessions returns after bridge attachment but starts only after retri
getSettings: () => ({ workspace: "psd" }) as any,
});
const res = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const res = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(res.json()).toEqual({ id: "s-early" });
expect(bridgeAttached).toBe(true);
expect(started).toBe(false);
@@ -2943,7 +3148,7 @@ test("POST /sessions bootstrap failure emits only a fixed recovery message", asy
const response = await app.inject({
method: "POST",
url: "/sessions",
payload: { question: "q" },
payload: { interactionLanguage: "en", question: "q" },
});
await new Promise((resolve) => setImmediate(resolve));
@@ -3039,7 +3244,7 @@ test.each([
})),
} as any,
});
const response = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const response = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
expect(response.statusCode).toBe(expectedStatus);
expect(ensure).toHaveBeenCalledTimes(reachesReadiness ? 1 : 0);
+2 -4
View File
@@ -160,8 +160,7 @@ test("GET /models returns session choices from the installation model catalog",
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
thtRunner: {} as any,
runtimeModelCatalog: {
defaultSession: "zai/glm-5.2",
defaultMetadataGeneration: null,
defaultInteraction: "zai/glm-5.2",
embedding: { id: "ollama/qwen3-embedding:0.6b", dimensions: 1024 },
sessionModels: () => [{
id: "zai/glm-5.2", provider: "zai", model: "glm-5.2", label: "GLM 5.2",
@@ -187,8 +186,7 @@ test("GET /models returns an empty list when the catalog has no session models",
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
thtRunner: {} as any,
runtimeModelCatalog: {
defaultSession: null,
defaultMetadataGeneration: null,
defaultInteraction: null,
embedding: null,
sessionModels: () => [],
metadataModels: () => [],
+13
View File
@@ -34,6 +34,19 @@ test("sessionNew parses id from JSON", async () => {
expect(await r.sessionNew({ question: "q" })).toEqual({ id: "2026-06-27-100000-x" });
});
test("session language uses public per-command CLI flags and the selected config", async () => {
const runner = new ThtRunner({ thtBin: "tht", harnessDir: "/h", configPath: "config/tht.yaml" });
(spawn as any).mockClear();
await runner.sessionNew({ question: "Pazienti", interactionLanguage: "en" });
expect((spawn as any).mock.calls[0][1]).toEqual([
"session", "new", "Pazienti", "--interaction-language", "en", "--json", "-c", "config/tht.yaml",
]);
await runner.ensureInteractionLanguage("s1");
expect((spawn as any).mock.calls[1][1]).toEqual([
"session", "ensure-interaction-language", "s1", "--json", "-c", "config/tht.yaml",
]);
});
test("searchPack persists retrieval context with session and workspace", async () => {
const calls: any[] = [];
const r = new ThtRunner({ thtBin: "tht", harnessDir: "/h", configPath: "config/tht.yaml" });
@@ -10,6 +10,28 @@ import {
const roots: string[] = [];
test("Evidence consolidation runs only its stage and preserves blocked Catalog readiness", async () => {
const catalog = repository();
const runChild = vi.fn(async () => ({ exitCode: 0, stdout: JSON.stringify({ status: "succeeded", counts: { documents: 2 } }), stderr: "" }));
const result = await service({ repository: catalog, runChild }).consolidateEvidence({ workspaceId: "catalog-workspace" });
expect(result.status).toBe("succeeded");
expect(runChild).toHaveBeenCalledOnce();
expect(runChild.mock.calls[0]?.[0]).toMatchObject({ argv: ["preprocess", "evidence", "--consolidate", "--json", "-c", "/dev/fd/3"] });
expect(catalog.beginPreprocessing).not.toHaveBeenCalled();
expect(catalog.finishPreprocessing).not.toHaveBeenCalled();
expect(catalog.clearPreprocessing).not.toHaveBeenCalled();
});
test("Evidence index failure reports retry without changing Catalog state", async () => {
const catalog = repository();
const runChild = vi.fn(async () => ({ exitCode: 1, stdout: JSON.stringify({ status: "failed", saved: true, error: "Saved; retry indexing." }), stderr: "private provider endpoint" }));
const result = await service({ repository: catalog, runChild }).consolidateEvidence({ workspaceId: "catalog-workspace" });
expect(result.status).toBe("failed");
expect(result.warnings).toEqual(["Saved; retry indexing."]);
expect(catalog.finishPreprocessing).not.toHaveBeenCalled();
expect(JSON.stringify(result)).not.toContain("private provider");
});
afterEach(() => {
roots.splice(0).forEach((root) => rmSync(root, { recursive: true, force: true }));
});
@@ -254,6 +276,24 @@ test("semantic preflight failure is persisted before returning", async () => {
);
});
test.each(["refresh", "decide"] as const)("Evidence %s calls only the source worker and preserves Catalog readiness", async action => {
const catalog = repository();
const runChild = vi.fn(async () => ({exitCode: 0, stdout: JSON.stringify({status: "succeeded", counts: {changed: 1}}), stderr: ""}));
const result = await service({repository: catalog, runChild}).evidenceSources({workspaceId: "catalog-workspace", action,
...(action === "decide" ? {sourceId: "a".repeat(64), revision: "b".repeat(64), decision: "replace" as const} : {}), actor: "Curator"});
expect(result.status).toBe("succeeded");
expect(runChild).toHaveBeenCalledWith({configPath: expect.any(String), argv: expect.arrayContaining(["evidence", "sources", action, "-c", "/dev/fd/3", "--actor", "Curator"])});
expect(catalog.beginPreprocessing).not.toHaveBeenCalled();
expect(catalog.finishPreprocessing).not.toHaveBeenCalled();
expect(catalog.clearPreprocessing).not.toHaveBeenCalled();
});
test("failed source activation reports the saved decision for retry", async () => {
const result = await service({runChild: async () => ({exitCode: 1, stdout: JSON.stringify({status: "failed", saved: true, error: "Decision saved; retry indexing"}), stderr: "secret"})})
.evidenceSources({workspaceId: "catalog-workspace", action: "decide", sourceId: "a".repeat(64), revision: "b".repeat(64), decision: "keep"});
expect(result).toMatchObject({status: "failed", warnings: ["Decision saved; retry indexing"]});
});
test("clear invalidates Catalog readiness before clearing only derived worker data", async () => {
const catalog = repository();
const runChild = vi.fn(async () => ({
@@ -1,4 +1,5 @@
import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
import {
existsSync,
mkdtempSync,
@@ -202,7 +203,7 @@ test("deterministic operator leases are keyed by logical identity and stable acr
workspaceSecretStore: f.workspaceSecretStore,
});
const suffix = first.inputFingerprint.slice(7, 23);
const suffix = createHash("sha256").update(first.inputFingerprint + "\n" + readFileSync(first.path, "utf8")).digest("hex").slice(0, 16);
expect(second.path).toBe(first.path);
expect(first.path).toBe(join(
f.dataRoot,
@@ -242,6 +242,7 @@ test("separate runtime leases hand off byte-identical revision Evidence configs
source_identity: "workspace://psd-clinical",
});
expect(parse(firstYaml).evidence).toEqual({
local_archive_root: join(f.registryConfig.root, "repo", "psd-clinical"),
sources: [{
type: "filesystem",
root: expectedRoot,
@@ -192,6 +192,7 @@ test("renders filesystem Evidence below the immutable revision content root with
expect(rendered.runtime_identity.workspace_revision).toBe(evidenceRevision);
expect(rendered.evidence).toEqual({
schema_version: 2,
local_archive_root: "/srv/registry/repo/psd-clinical",
sources: [{
type: "filesystem",
root: `/srv/registry/snapshots/${evidenceRevision}/psd-clinical/evidence`,
@@ -203,7 +204,7 @@ test("renders filesystem Evidence below the immutable revision content root with
max_chunk_chars: 4_000,
retain_published_generations: 3,
});
expect(yaml).not.toContain("/srv/registry/repo");
expect(rendered.evidence.sources[0].root).not.toContain("/srv/registry/repo");
});
test("renders public HTTP Evidence with exact fractional-second timeouts and every policy limit", () => {
@@ -223,6 +224,7 @@ test("renders public HTTP Evidence with exact fractional-second timeouts and eve
}));
expect(rendered.evidence).toEqual({
local_archive_root: "/srv/registry/repo/psd-clinical",
sources: [{
type: "http",
urls: ["https://evidence.example.test/guide.md"],
+3 -2
View File
@@ -13,6 +13,7 @@ services:
THT_BIN: /opt/venv/bin/tht
THT_DATA_ROOT: /data
SETTINGS_FILE: /data/settings/settings.json
THT_EVIDENCE_HOST_REGISTRY_ROOT: ${THT_WORKSPACE_REGISTRY_ROOT:-}
THT_MAINTENANCE_FILE: /data/settings/maintenance.json
THT_WORKSPACE_REGISTRY_ROOT: /data/workspace-registry
THT_WORKSPACE_GIT_REMOTE: ${THT_WORKSPACE_GIT_REMOTE:?set THT_WORKSPACE_GIT_REMOTE}
@@ -99,7 +100,7 @@ services:
image: thothii-core:local
profiles: [catalog-maintenance]
pull_policy: never
command: ["node", "/app/backend/dist/catalog/migrate.js"]
command: ["bash", "/app/docker/catalog-migrate.sh"]
environment:
THT_CATALOG_DB_HOST: catalog-db
THT_CATALOG_DB_PORT: "5432"
@@ -153,7 +154,7 @@ services:
- type: volume
source: workspace-registry
target: /data/workspace-registry
read_only: true
read_only: false
- type: volume
source: sessions
target: /data/sessions
+20
View File
@@ -0,0 +1,20 @@
# Optional local profile: expose the existing curator checkout on the installation host.
# Copy the existing registry repo to this path before enabling the override. Snapshot/state
# volumes remain unchanged. THT_EVIDENCE_HOST_REGISTRY_ROOT must be an absolute host path.
services:
core:
environment:
THT_EVIDENCE_HOST_REGISTRY_ROOT: ${THT_EVIDENCE_HOST_REGISTRY_ROOT:?set an absolute host registry path}
volumes:
- type: bind
source: ${THT_EVIDENCE_HOST_REGISTRY_ROOT:?set an absolute host registry path}/repo
target: /data/workspace-registry/repo
bind:
create_host_path: false
workspace-maintenance:
volumes:
- type: bind
source: ${THT_EVIDENCE_HOST_REGISTRY_ROOT:?set an absolute host registry path}/repo
target: /data/workspace-registry/repo
bind:
create_host_path: false
+1 -1
View File
@@ -45,7 +45,7 @@ services:
- type: bind
source: ${THT_WORKSPACE_REGISTRY_ROOT:?set THT_WORKSPACE_REGISTRY_ROOT}
target: /data/workspace-registry
read_only: true
read_only: false
- type: bind
source: ${THT_DATA_ROOT:?set THT_DATA_ROOT}/workspace-secrets
target: /data/workspace-secrets
@@ -0,0 +1,4 @@
# The image is tagged from the running frontend before the visual review is published.
services:
frontend:
image: thothii-frontend:before-visual-review-20260912
+8
View File
@@ -0,0 +1,8 @@
# Local-only review override. Apply after the existing local installation profiles.
# Changes only the frontend image; Core, configuration and volumes remain unchanged.
services:
frontend:
image: thothii-frontend:visual-review-20260912
build:
context: /Users/mp/projects/ThothII-visual-review
dockerfile: docker/frontend.Dockerfile
@@ -0,0 +1,5 @@
# Local-only rollback to the full-shell frontend before visual integration.
# Apply after local installation profiles, with --no-build and --no-deps.
services:
frontend:
image: thothii-frontend:before-visual-shell-merge-20260913
+17
View File
@@ -0,0 +1,17 @@
services:
core:
environment:
THT_EVIDENCE_HOST_REGISTRY_ROOT: /Users/mp/projects/ThothII/deploy/psd/evidence-registry
volumes:
- type: bind
source: /Users/mp/projects/ThothII/deploy/psd/evidence-registry/repo
target: /data/workspace-registry/repo
bind:
create_host_path: false
workspace-maintenance:
volumes:
- type: bind
source: /Users/mp/projects/ThothII/deploy/psd/evidence-registry/repo
target: /data/workspace-registry/repo
bind:
create_host_path: false
Submodule deploy/psd/evidence-registry/repo added at 24fb53236b
+13 -3
View File
@@ -2,6 +2,10 @@
# Replace every absolute path before using this as an advanced reference.
schemaVersion: 2
profile: local
# Mac standalone example; the Omics server requires embedded/upstream separately.
shell:
mode: full
defaultLocale: en
projectDirectory: "<abs>/projects/ThothII"
envFile: "<abs>/projects/ThothII/deploy/psd/operator.env"
workspaceRepository:
@@ -10,22 +14,28 @@ workspaceRepository:
access: ssh
modelCatalog:
defaults:
session: zai/glm-5.3
metadataGeneration: zai/glm-5.3
interaction: zai/glm-5.3
embedding:
id: ollama/qwen3-embedding:0.6b
dimensions: 1024
providers:
deepseek:
authentication:
mode: pi_auth
mode: secret_env
apiKeyEnv: DEEPSEEK_API_KEY
session:
mode: pi_builtin
metadataGeneration:
litellmProvider: deepseek
models:
deepseek-v4-pro:
label: DeepSeek V4 Pro
session: {}
metadataGeneration: {}
deepseek-v4-flash:
label: DeepSeek V4 Flash
session: {}
metadataGeneration: {}
zai:
endpoint:
baseUrl: https://api.z.ai/api/coding/paas/v4
+4
View File
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
set -euo pipefail
node /app/backend/dist/catalog/migrate.js
/opt/venv/bin/python -m tht.memory.migrate
+1 -1
View File
@@ -131,7 +131,7 @@ ENV PATH="/opt/venv/bin:/usr/local/bin:$PATH" \
HOME=/home/thoth
COPY scripts/verify-line-endings.sh /usr/local/bin/verify-line-endings
COPY docker/core-entrypoint.sh docker/workspace-maintenance-entrypoint.sh docker/session-migrate.sh docker/ensure-pi-trust.mjs docker/embedding-model-init.sh /app/docker/
COPY docker/core-entrypoint.sh docker/workspace-maintenance-entrypoint.sh docker/session-migrate.sh docker/catalog-migrate.sh docker/ensure-pi-trust.mjs docker/embedding-model-init.sh /app/docker/
COPY docker/smoke/core-smoke.sh /app/docker/smoke/core-smoke.sh
RUN /usr/local/bin/verify-line-endings /app/docker \
&& chmod +x /app/docker/core-entrypoint.sh /app/docker/workspace-maintenance-entrypoint.sh /app/docker/session-migrate.sh /app/docker/embedding-model-init.sh /app/docker/smoke/core-smoke.sh
+21 -1
View File
@@ -1,6 +1,26 @@
#!/bin/sh
set -eu
test "$(cat /usr/share/nginx/html/config.js)" = 'window.__THOTHII_CONFIG__ = {};'
# The generic image contains an empty fallback; installed containers mount the generated
# public projection over it. An optional path lets host projection tests use this same check.
config_file=${1:-/usr/share/nginx/html/config.js}
test -f "$config_file" && test -r "$config_file"
config=$(tr -d '[:space:]' < "$config_file")
case "$config" in
'window.__THOTHII_CONFIG__={};')
test -z "${THT_FRONTEND_CONFIG_REVISION:-}"
;;
*)
# Installation Load validates BCP47 syntax. Here check only the public payload shape;
# catalog availability and fallback belong to the frontend, not the generic image.
locale='[A-Za-z][A-Za-z0-9]*(-[A-Za-z0-9]+)*'
full="\"mode\":\"full\",\"defaultLocale\":\"$locale\""
embedded="\"mode\":\"embedded\",\"defaultLocale\":\"$locale\",\"adapter\":\"omics-portal\""
if ! printf '%s\n' "$config" | LC_ALL=C grep -Eq "^window\.__THOTHII_CONFIG__=\{\"backendBaseUrl\":\"/api\",\"shell\":\{($full|$embedded)\}\};$"; then
echo "Invalid frontend public runtime configuration" >&2
exit 1
fi
;;
esac
printf '%s\n' "frontend runtime config smoke: ok"
@@ -0,0 +1,59 @@
---
status: accepted
date: 2026-09-08
---
# Use PostgreSQL for Memory and Qdrant for retrieval
The new Memory module uses the installation's existing PostgreSQL service as the
authority for Memory Cards, their links, and structured schema dependencies.
Qdrant holds a rebuildable search projection. This replaces the JSONL registry and
allows related administrative changes to be coordinated in one database transaction.
The owner accepted this direction in Q9 of the Memory design interview; implementation
is pending.
Memory uses its own tables and remains a separate module from the Metadata Catalog.
Sharing the PostgreSQL service does not transfer Memory ownership to Database
management or make Evidence and Memory one canonical domain.
## Retrieval and graph
The owner also accepted hybrid semantic/lexical Qdrant retrieval with scope filters
and explicit card links traversed in core. Both belong to the planned first version;
there is no dedicated graph database or external Memory framework.
This extends [ADR 0017](0017-separate-reference-vectors-from-runtime-memory.md):
the separate reference and memory collections remain, while Memory gains sparse
lexical indexing in addition to dense vectors. Memory projections become rebuildable
from the module's PostgreSQL authority. Preprocessing Clear still preserves Memory;
this decision does not add it to the preprocessing cleanup scope.
Links support discovery. Finding a card through a link does not approve its use.
The core proposes links with the cards for the same final review; Administration
provides manual creation, editing, and deletion. Deleting a card removes its incident
links without deleting the other linked cards.
## Considered options
- Retaining JSONL preserves the current storage format, but leaves coordinated
card/link/dependency mutations and concurrent administrative writes to application code.
- Using Qdrant as the sole authority is a viable alternative for record storage and
retrieval. PostgreSQL is preferred for the coordinated mutations of the new module,
with Qdrant reserved for its search projection.
- Adding a dedicated graph database would introduce another service; the selected
bounded traversal can be implemented in core over persisted links.
## Consequences
The module needs a persistence contract, PostgreSQL schema, and explicit propagation
of additions, updates, and deletions to Qdrant. Choosing PostgreSQL does not make this
propagation atomic across both systems: failures, retries, and invalidation of stale
search content must be handled and tested. An index rebuild uses the original card
content and cannot resurrect deleted cards.
The decision does not introduce Memory revision history or require compatibility with
existing development sessions. Evidence authoring and publication remain governed by
their own contract until the Evidence management project defines its evolution.
The [Memory management project](../plans/2026-09-08-memory-management.md) records the
approved behavior, scope, and integration work.
@@ -0,0 +1,152 @@
---
status: accepted
date: 2026-09-08
---
# Maintain local Evidence with external editors and manual consolidation
Evidence management must provide complete access to registered Evidence and a
maintenance path, as accepted in Q11–Q15 and revised during simplification. The owner subsequently
clarified that a context specialist writes drafts independently of the installation
and without PostgreSQL access; the system refines and stores them locally for use
and maintenance. Implementation is pending.
The storage mechanics of Q11 were revisited in the
[simplification review](../plans/2026-09-08-memory-evidence-simplification-review.md).
The original choice combined the workspace repository as Evidence authority with
application-managed Git writes. The owner accepted the separation of external draft
files/repositories from a durable local canonical file archive and removes
commit/push from normal CRUD. The management surface must expose discoverable,
editable Markdown for domain specialists, without JSONL editing. The owner chose
external editors for release 0: their preferred Mac/PC editor or vim/nano on the
server. No web editor or content form is required. The page keeps browsing,
filtering, and detail, with actual host file paths and maintenance instructions.
The owner also requested manual consolidation followed by manual Git diff, commit,
and push, relying on operator discipline rather than automation. The owner accepted
the remaining simplifications and requested explicit clarification of the core
format change and the simple terminal-based Git check. The original application Git writer
is not the final implementation prescription. A suggestion
to move Evidence authority into PostgreSQL was withdrawn after the clarification.
Memory remains a distinct module with its own PostgreSQL authority under
[ADR 0018](0018-use-postgres-for-memory-and-qdrant-for-retrieval.md).
## Manual consolidation and Git follow-up
The external-editor decision replaces the form's single Save with saving files and
running one consolidation command. The command checks expected structure, required
fields, types, identifiers, references, and provenance. It reports the affected file
and the necessary correction. Invalid input stops activation; valid input updates
derived metadata, the corpus, and the Evidence index. New and deleted files use the
same path. An inaccessible archive must not be interpreted as deleted Evidence.
Structural validation does not prove semantic correctness or rerun model refinement
to rewrite manually edited rules.
The core reads only successfully consolidated active content, never the editable
working files directly. Candidate validation and activation must preserve the last
valid corpus on failure, or report unavailability if its integrity cannot be
guaranteed. Unconsolidated edits must not mix new text with old retrieval data.
Consolidation reports full local success only once the updated content is available
to the core. If persistence succeeds but activation fails, the command reports the
partial outcome and can be rerun without duplicating units or losing edits.
The operation does not imply atomicity across
authoritative storage and Qdrant; their failure and recovery behavior requires
explicit implementation. An interrupted operation must not leave a partial corpus
advertised as ready for subsequent use.
The maintained archive is a persistent Git working tree containing the local
Evidence; it may reuse the workspace repository. Draft sources and curated units
remain distinct even when in the same repository. Setup and the page identify the
repository, working tree, actual host paths, branch, and configured remote. A
rebuildable runtime snapshot is not the editing location.
Before consolidation, the operator checks Git status, including inspection of new
untracked files. Normal terminal Git diff commands are available for line-level
inspection when needed; no custom viewer or mandatory double review is required.
After consolidation, the operator stages the Evidence and required metadata changes,
commits, and pushes using normal Git commands. No watcher, automatic Git writes, retrying push, pull, merge,
or dedicated web execution control is required. Missing credentials, unconfigured
upstreams, and Git conflicts are handled by the operator.
Consolidation activates local changes before commit/push. If the operator omits the
Git steps or push fails, the local update remains effective while transfer to the
remote is incomplete. Pushing does not automatically update other installations.
The system relies on operator discipline to complete the sequence; it does not add
a separate editorial publication state machine or require a second reviewer.
Approved conflict repairs from the core still call the same persistence and
activation service directly. They do not require an external editor or manual
consolidation before the session can use its own approved correction. Their local
file changes are included in the operator's subsequent Git maintenance.
## Administration without concurrent core work
The owner's simplification instruction replaces the earlier Q12 requirement to
refresh open sessions after administrative edits. Core activity can be assumed
absent during administration, or its overlap can be ignored. No dedicated live
update, session notification, restart, maintenance mode, or reader coordination
is required. Completed administrative changes apply to subsequent work.
Deliberate writes from the workflow itself still exist: approved conflict repairs
and the final Memory summary use the same persistence services and handle their
outcome before proceeding. In particular, a session must be able to use its own
approved Evidence correction. This does not require updating all other sessions
or rewriting previously approved decisions, artifacts, or SQL.
## Manual corrections survive source updates
When an updated source contradicts an administrator's correction, the saved manual
Evidence remains active. Evidence management shows the conflicting content for an
explicit decision and subsequent save. The source's newer content does not
automatically override the correction. The owner accepts that the manual rule can
remain in use until that comparison is resolved.
Deleted Evidence must not silently reappear after preparation or reindexing. The
authoring implementation must preserve both manual corrections and suppression of
deleted units through source refreshes. The current protection for uncommitted Git
changes is insufficient: it does not preserve already committed manual corrections
against regeneration, and retirement currently removes the unit without recording
suppression for later preparation.
## Manual creation and explicit source refresh
Administrators can create Evidence without providing an external document. In R0,
they add Markdown using the documented example and consolidate it. The application
records a manual declaration as the managed source of the current statement.
Explicit consolidation approves that declaration; it does not
claim independent documentary verification.
A correction that changes a unit's meaning uses the same explicit manual origin.
The original document remains linked for provenance and source-change detection,
but its excerpt is not presented as support for a rule it does not contain. The
canonical contract must distinguish the current supporting source from the original
document rather than silently retaining outdated support metadata.
External sources are reacquired only when an administrator requests a source
refresh. Ordinary saves and session lookups use the acquired local content; they
do not poll sources or fetch their current versions. A remote change is therefore
detected at the next requested refresh, when the manual-precedence rule applies.
The revised storage recommendation reads the specialist's repository as an input;
ordinary local edits do not write back to it or refresh its content automatically.
## Implementation consequences
The existing [Evidence lifecycle](../evidence.md) and
[Workspace Evidence v3 contract](../contracts/workspace-evidence-v3.md) require
explicit evolution for editable local Markdown, manual consolidation, and manual
precedence. Source provenance and coherent canonical metadata remain requirements;
manual changes must not be presented as statements supported by an unrelated source
excerpt. Under the accepted local-file direction, curated files are primary
installation data preserved by Clear and backed up separately from rebuildable
indexes. Visible Markdown content must become authoritative for editing; operators
must not maintain hidden duplicate text, hashes, or manifest entries themselves.
This is a core Evidence contract change: parser, renderer, authoring, validation,
normalization, and their integration with indexing and recall must be adapted
together. E1 includes a new Curated unit format version, conversion of existing
Evidence, reindexing, and end-to-end verification that visible edits reach core
consumption. Preserve the internal typed model where possible; the change does
not require redesigning the NL-to-SQL workflow phases.
Local edits do not automatically change external drafts or another installation;
Git versioning and transfer occur through the explicit manual follow-up.
The [Evidence management project](../plans/2026-09-08-evidence-management.md)
records the implementation sequence and acceptance checks.
@@ -0,0 +1,27 @@
# Unify administration pages and use namespaced routes
status: accepted
Workspace and Pi management will become full Administration Pages alongside Evidence, Memory and
Database, using one shared Administrative Page Family. The Embedded Thoth Shell will identify the
active Administration Surface through a namespaced browser route, `thoth_route=administration/<surface>`,
so links remain reloadable and support Back/Forward without requiring new host-portal server routes.
Transient form values and secrets stay outside the route; destructive confirmations may remain
short-lived dialogs because they are actions, not page containers.
We considered React-only state, path-based routes and hash routes. React-only state loses refresh and
deep-link behavior, path routes require a host catch-all route that does not yet exist in Omics Portal,
and hash routes can conflict with host-page fragments. A namespaced query route preserves the current
same-document integration boundary while leaving room for a future path adapter.
The accepted visual direction is A / Workbench for all five surfaces; B and C remain recoverable
prototypes. Workspace owns readiness and preprocessing, consuming the Database catalog as a
prerequisite. Database owns connection/binding, schema synchronization, descriptions and sensitivity;
the preprocessing action remains exclusively in Workspace preparation.
On 2026-09-12, the owner refined this direction in
[Gitea #29](https://git.tylconsulting.it/mptyl/ThothII/issues/29): Database opens at its
catalog list, with its original table space and margins, and no Workspace preparation footer.
This supersedes the earlier requirement for a preparation link in that page. Workspace
remains reachable through the shared Administration navigation; its Preparation tab still
links to Database configuration and schema when catalog work is needed.
@@ -0,0 +1,101 @@
# ADR 0021 — Shell separati e adapter sostituibile per il portale
- Stato: accettato
- Data: 2026-09-13
## Decisione
ThothII espone due modalità di installazione, selezionate da `shell.mode`:
- `embedded` (default): ThothII è ospitato da Omics Portal. Non renderizza alcun header e
riceve dal portale lingua, tema e fullscreen. L'accesso resta verificato dal server.
- `full`: ThothII è autonomo. Renderizza il proprio header, con selettore lingua, tema,
fullscreen e nome utente. Il click sul nome apre il logout. Non mostra mai la rotellina o
altri comandi amministrativi del portale. Mantiene un rail vuoto a sinistra di almeno 20 px.
Il fatto che la shell sia `full` è distinto dallo stato `fullscreen`: la prima decide quale
contenitore viene renderizzato, il secondo indica se è attiva la Fullscreen API del browser.
L'icona passa da “entra in fullscreen” a “torna alla modalità normale”; anche `Esc` aggiorna lo
stato visualizzato.
La configurazione installata resta semplice e retrocompatibile. Sul Mac di sviluppo il profilo
locale userà:
```yaml
shell:
mode: full
defaultLocale: en
```
Il deploy sul server userà invece:
```yaml
shell:
mode: embedded
adapter: omics-portal
```
`defaultLocale` indica la lingua iniziale della shell full; in embedded la fonte autorevole resta
il portale.
L'adapter è l'unico confine tra ThothII e il portale. La sua interfaccia pubblica è volutamente
profonda e minima: consegna solo snapshot dello stato, senza esporre comandi, token, identità o
dettagli di trasporto.
```ts
export type HostShellState = {
locale: string; // BCP-47, inizialmente it/en
theme: "light" | "dark";
fullscreen: boolean;
};
export interface PortalAdapter {
subscribe(
onState: (state: HostShellState) => void,
onError: (error: Error) => void,
): () => void;
}
```
`OmicsPortalAdapter` è l'implementazione corrente. Un adapter per un altro portale potrà
sostituirlo senza modificare shell, i18n o workflow. In `full` l'adapter non viene istanziato:
lo stato è gestito internamente dalla shell.
Per l'integrazione oggi operativa, che monta la SPA direttamente nel DOM di Omics Portal,
l'adapter legge la lingua effettiva dal selettore Omics, osserva l'attributo del tema e ascolta
il fullscreen del documento. Selettori e osservatori restano privati dell'implementazione Omics.
La lingua segue il normale ricaricamento Django; tema e fullscreen cambiano nella pagina aperta.
La revisione approvata del 2026-09-13 elimina il precedente handshake a eventi: non servono
messaggi personalizzati, versioni di trasporto o timeout di avvio. Un futuro adapter potrà usare
un diverso trasporto senza modificare l'interfaccia applicativa.
Se `shell` o l'adapter embedded sono omessi, si usa `embedded` con `omics-portal`. Questo default
supporta il documento Omics esistente; nomi adapter sconosciuti o dati host mancanti producono
un errore esplicito, senza attivare la shell full.
## Confini che restano invariati
L'identità e l'autorizzazione del backend non vengono ricostruite nel browser. In embedded,
Omics Portal continua a gestire login e logout e la catena server-side `auth_request` continua a
fornire i principal header già previsti. Lo stato UI non dichiara l'utente autenticato: il modulo
di accesso usa la verifica backend esistente anche alla riconnessione e al ritorno alla pagina.
Un rifiuto su una singola operazione non equivale automaticamente alla perdita dell'accesso.
La lingua UI e la lingua di interazione con il modello restano separate dalla lingua del
workspace; il relativo contratto è in [ADR 0022](0022-separate-ui-locale-from-session-interaction-language.md).
## Alternative scartate
- Duplicare l'header di Omics in embedded: crea due fonti di stato e incompatibilità visive.
- Spargere controlli `if embedded/full` nei componenti: lega ogni pagina al portale.
- Trasmettere utente o token nel bridge: aumenta superficie e accoppia UI e autenticazione.
- Introdurre un protocollo completo request/response: non aggiunge funzionalità richiesta.
- Usare `profile` per distinguere le shell: `profile` descrive la topologia dell'installazione,
non la sua presentazione.
## Conseguenze
La soluzione richiede un adapter nel frontend che osserva il documento condiviso e mantiene la
logica di shell locale a ThothII. Il backend non necessita di un nuovo protocollo di autenticazione
o di una nuova sessione browser. Un nuovo portale deve soddisfare anche il contratto server di
identità fidata: la sola sostituzione della classe UI non sostituisce quel contratto.
@@ -0,0 +1,51 @@
# Separate UI locale from session interaction language
status: accepted
ThothII distinguishes three language concepts:
- `workspace.language` remains the language of workspace-owned documents, descriptions and Evidence;
- `ui_locale` controls deterministic ThothII chrome such as labels, form help, placeholders, errors,
accessibility text and review-widget chrome;
- `interaction_language` is persisted in a session and controls model-generated questions,
explanations and reviewer proposals.
The initial locale catalog supports Italian and English and uses extensible BCP-47 language tags.
Missing deterministic translations fall back to English. The selected UI locale supplies the default
interaction language when a new session is created. A resumed session always uses its persisted
interaction language; changing the host or full-shell UI locale must not silently rewrite an existing
session or make its model output switch language mid-workflow.
The distinction is required because the current workspace contract already uses `language` for
content and the PSD workspace is Italian. Reusing that field for a browser preference would make a
visual choice mutate domain content semantics. The model receives the session interaction language
through the session/Pi workflow context. SQL, identifiers, database values and other technical
artifacts remain governed by their existing contracts and are not translated as UI strings.
In `full`, the local shell owns `ui_locale` and supplies it when starting a new session. In
`embedded`, the host adapter is authoritative for `ui_locale`; ThothII applies host changes to
deterministic UI immediately while preserving the interaction language of any active session.
We considered using only `workspace.language`, using only a global browser locale, and translating
the model output after generation. The first conflates domain content with UI preference; the second
cannot preserve a session's language or follow the host portal; and the third would be unsafe for
structured reviewer decisions and would not control the model's reasoning or proposal language.
## Considered Options
- One mutable `language` field for workspace, UI and session was rejected because the fields have
different owners and lifecycles.
- Client-only translation of reviewer choices was rejected because choices can be generated by the
model and must be requested in the intended language.
- An English-only deterministic chrome was rejected because embedded and full installations must
follow the selected host/user language.
## Consequences
- Session creation and the persisted manifest gain an explicit interaction-language value.
- Legacy manifests without that value use the workspace language, pinned idempotently on first
resume; the browser locale must not determine this compatibility value.
- Resume must read that value from the manifest and must not accept a new locale as an override.
- The workflow prompt contract and deterministic reviewer-widget builders need a locale-aware input.
- Frontend strings need a catalog and stable keys; backend events should expose stable codes where
the frontend is responsible for localization.
+130
View File
@@ -0,0 +1,130 @@
# Rendering full ed embedded
ThothII ha una sola applicazione React, una sola build Vite e gli stessi servizi
backend. «Doppio rendering» significa due modi di ospitare quella applicazione,
non due versioni delle pagine e non rendering React sul server. Django renderizza
il contenitore Omics; React renderizza ThothII nel browser, dentro `#root`.
## Tre decisioni indipendenti
| Decisione | Configurazione | Effetto |
| --- | --- | --- |
| Distribuzione | `profile: local` oppure `server` | Compose, percorsi e vincoli operativi |
| Presentazione | `shell.mode: full` oppure `embedded` | Proprietario di header e preferenze |
| Autenticazione | `auth.yaml` local/OIDC oppure `AUTH_MODE=upstream` | Chi verifica l'identità, come arriva al backend |
Il Mac usa **full + local**, con lingua iniziale inglese. L'integrazione Omics
usa **embedded + upstream**, con accesso già verificato dal portale. Un server
autonomo può usare **full + oidc**. Cambiare `shell.mode` non abilita un metodo
di autenticazione e non modifica permessi o proprietari delle sessioni.
Full con upstream può visualizzare un'identità già verificata dal proxy, ma non
ha un logout ThothII disponibile: non è il profilo autonomo con login/logout.
Embedded non avvia login locale o OIDC anche se il backend è configurato così;
questa combinazione non realizza il login unico Omics e non va usata come fallback.
## Composizione comune
```mermaid
flowchart TD
CONFIG["config.js pubblico"] --> SHELL["ShellProvider"]
FULL["Preferenze full nel browser"] --> SHELL
HOST["Documento Omics"] --> ADAPTER["OmicsPortalAdapter: solo presentazione"]
ADAPTER --> SHELL
SHELL --> GATE["AuthGate: verifica GET /me"]
GATE --> APP["AppShell: stesse pagine, sessioni e amministrazione"]
AUTH["Backend: cookie locale/OIDC o identità upstream"] --> GATE
```
`ShellProvider` risolve la configurazione, applica lingua/tema e monta i contenuti
solo dopo uno snapshot host valido in embedded. `AuthGate` verifica l'accesso;
`AppShell` e le pagine non devono leggere selettori o eventi specifici di Omics.
Il cambio utente smonta lo stato applicativo della precedente identità.
## Full
- Header ThothII rosso Omics `#CB333B` in entrambi i temi; logo interamente chiaro.
- Selettore EN/IT, tema light/dark, fullscreen e nome verificato dell'utente.
- Menu del nome con logout soltanto per local/OIDC; nessuna rotellina admin.
L'amministrazione resta nella navigazione applicativa, secondo i permessi.
- Margine sinistro vuoto e simmetrico al destro: `max(20px, 1.5rem)`, normalmente
24px con radice a 16px. Non è una seconda sidebar di navigazione.
- Lingua e tema ricordati sullo stesso origin in `localStorage`, nelle chiavi
`thothii:shell:locale` e `thothii:shell:theme`. Non sono preferenze server per
utente. In assenza di preferenze: `defaultLocale` e tema light.
- Fullscreen usa `document.documentElement.requestFullscreen()` e
`document.exitFullscreen()`: nasconde il contorno del browser dove supportato.
L'icona cambia sullo stato reale, anche dopo Esc; un rifiuto mostra un errore.
Non è un semplice ingrandimento CSS e non scatta automaticamente all'accesso.
## Embedded
- Nessun header ThothII, selettore lingua, toggle tema, login o logout autonomo.
I controlli rimangono nell'header generale Omics.
- React è nello stesso documento della pagina `/kokoro/datamart-builder/`, non
in un iframe. Non serve `postMessage` né un secondo protocollo di sessione.
- L'adapter legge la lingua Django già confermata, osserva il tema del documento
e ascolta il fullscreen reale. Le azioni rimangono di proprietà del portale.
- Un contesto Omics mancante o invalido mostra un errore d'integrazione; non
passa silenziosamente a full e non offre un secondo login.
- Il portale assegna l'altezza disponibile sotto il proprio header: catena flex
con `min-height: 0`, root contenuto e altezza applicativa vincolata al contenitore.
Il contratto ThothII espone `--thoth-app-height` (fallback `100dvh`); verificare
il contenitore reale, non presumere che l'intera viewport appartenga a React.
Il template Omics mantiene inoltre i suoi override di compatibilità.
Il reset CSS è limitato al mount React e ai popup dell'applicazione, senza
richiedere CSS `@scope`. I token e i popup seguono il tema applicativo. Questo
non rende indipendenti fogli di stile arbitrari caricati dal portale: la verifica
del documento condiviso rimane necessaria a ogni integrazione.
## Caricamento e configurazione pubblica
Il descrittore installato è la sorgente di verità. Il CLI genera
`generated/frontend/config.js` e il suo mount di sola lettura nella proiezione
`generated/compose.models.yaml`. Il file pubblico contiene solo `backendBaseUrl`
e `shell`, mai identità, token, password o percorsi host. Va caricato **prima** del
modulo React e servito senza cache. Nessuna build separata è richiesta per
cambiare modalità; occorre rigenerare e applicare i mount tramite il lifecycle.
L'ordine Omics è: config pubblico → override del solo prefisso API → asset dal
manifest Vite. L'override deve conservare `shell`; l'adapter non configura il proxy.
Il default completo di shell omessa è embedded/en/omics-portal. Il CLI normalizza
anche singoli campi omessi; un oggetto `shell` scritto manualmente nel browser
deve invece contenere `mode` e `defaultLocale`, altrimenti viene rifiutato.
## Lingua, continuità e dati
La lingua UI traduce il testo dell'applicazione, non i contenuti di dominio.
Alla creazione, la lingua UI viene acquisita come `interactionLanguage`; il
manifest salva `interaction_language`, che governa domande e scelte del modello.
Alla ripresa vale la lingua salvata, non l'ultima scelta dell'header. Per i manifest
precedenti senza campo viene fissata la lingua workspace disponibile alla prima ripresa.
Il cambio lingua Omics invia il form Django e ricarica la pagina. ThothII conserva
solo l'ID della selezione in `sessionStorage`, separato per pathname, issuer e
subject. Riapre i documenti, non avvia una generazione. Bozze non inviate e modifiche
non salvate richiedono conferma prima della navigazione; non sono una trascrizione
salvata. La ripresa operativa resta esplicita.
## Punti di implementazione e manutenzione
| Sorgente | Responsabilità |
| --- | --- |
| `tools/tht/internal/config/shell.go` | Normalizzazione e validazione del descrittore |
| `tools/tht/internal/modelprojection/projection.go` | Config pubblico e mount generati |
| `frontend/src/api/runtime-config.ts` | Validazione browser e prefisso API same-origin |
| `frontend/src/shell/host/ShellProvider.tsx` | Composizione, preferenze e tema |
| `frontend/src/shell/host/FullHeader.tsx` | Controlli solo full |
| `frontend/src/shell/host/OmicsPortalAdapter.ts` | Conoscenza del documento Omics |
| `frontend/src/auth/AuthGate.tsx` | Accesso e ricontrolli al ritorno alla pagina |
| `backend/src/auth/auth.ts` e `principal.ts` | Verifica server dell'identità |
Per un altro portale servono un'implementazione del
[PortalAdapter](../contracts/portal-shell-adapter-v1.md), la sua registrazione nei
validatori CLI/browser e nel punto di composizione, oltre al
[contratto di autenticazione server](../install/authentication-upstream.md).
Il nome di una classe non è un plugin caricabile dinamicamente da YAML.
Procedure: [configurazione e deploy](../operations/shell-and-localization.md),
[autenticazione](authentication.md), [accettazione](../testing/authentication-manual-acceptance.md).
+50 -6
View File
@@ -1,18 +1,27 @@
# Authentication architecture
ThothII has two production authentication modes: `local` and generic `oidc`. The host operator
surface is one CLI, `tht`; there is no separate authentication executable. The backend owns
opaque browser sessions and authorization, while `tht` owns protected configuration and local-user
files.
ThothII supports `local` and generic `oidc` through protected `auth.yaml`, plus
the trusted-proxy `upstream` path used by Omics. The host operator surface is
one CLI, `tht`; there is no separate authentication executable. The backend owns
authorization in all paths and opaque browser sessions only in local/OIDC.
`tht` owns protected local/OIDC configuration and local-user files; upstream
identity is supplied per request by the authenticated server proxy.
Presentation is separate: [full/embedded rendering](application-shell.md) does
not select authentication. The Mac uses full/local; Omics uses embedded/upstream;
a standalone server can use full/OIDC. Do not configure a second ThothII OIDC
login simply because Omics itself authenticates users through Authentik.
```mermaid
flowchart TB
BROWSER["Browser"] --> BOUNDARY["Authentication boundary"]
BOUNDARY --> LOCAL["Local users\nArgon2id hashes"]
BOUNDARY --> OIDC["OIDC provider\nAuthorization Code PKCE"]
BOUNDARY --> UPSTREAM["Trusted proxy\nVerified portal session"]
OIDC --> GROUPS["Groups claim\nexact mapping"]
LOCAL --> PRINCIPAL["Thoth principal"]
GROUPS --> PRINCIPAL
UPSTREAM --> PRINCIPAL
PRINCIPAL --> ROLES["Roles"]
ROLES --> PERMISSIONS["Permissions"]
PERMISSIONS --> ROUTES["Protected routes"]
@@ -21,6 +30,13 @@ flowchart TB
## Configuration and trust boundaries
The following protected-file configuration applies to local/OIDC. Upstream uses
`AUTH_MODE=upstream` without a mounted `auth.yaml` or authentication runtime
projection. The backend refuses both authorities together. `AUTH_MODE=none`
and `mock` are development/test modes, not production fallbacks. The exact
upstream setup, header contract, proxy hops and origin checks are in the
[server integration guide](../install/authentication-upstream.md).
The installation descriptor points to an operator-controlled authentication directory. It contains
non-secret `auth.yaml` and, for local mode, `users.yaml`. POSIX installations use a private
directory and owner-only regular files; Windows uses equivalent owner-only ACLs. Secret values are
@@ -33,17 +49,26 @@ The production role expansion from `backend/src/auth/config.ts` is exact:
| Role | Permissions |
|---|---|
| `user` | `session.use` |
| `admin` | `session.use`, `session.read_all`, `session.manage_all`, `settings.manage`, `workspace.manage`, `workspace.secrets.manage`, `database.manage`, `pi.manage`, `auth.diagnostics.read` |
| `admin` | `session.use`, `session.read_all`, `session.manage_all`, `settings.manage`, `workspace.manage`, `workspace.secrets.manage`, `database.manage`, `memory.manage`, `evidence.manage`, `pi.manage`, `auth.diagnostics.read` |
`admin` therefore includes the ordinary `session.use` permission. No other role or permission
label is part of the production catalog.
After validating a browser session, the backend expands its roles through the current permission
catalog on every request. The permissions saved at login are a historical snapshot, so existing
administrator sessions can use newly deployed administration features without signing in again.
Session expiry, revocation and local-user role validation still apply before role expansion.
OIDC is provider-neutral at the browser protocol boundary. Authorization Code + PKCE, issuer,
signature, audience, expiry, state, and nonce are validated before a principal is created.
Authentik is the first certified group-catalog adapter, not a special browser login mode.
## Group authorization
This section describes **ThothII's direct OIDC login**, not the embedded Omics
path. Omics checks its own capability and administrator status and supplies
normalized identity headers; ThothII does not repeat the OIDC groups exchange.
OIDC must return a direct, non-empty `groups` claim whose value is a JSON array of strings.
Missing, malformed, indirect, or overage-style claims fail closed. The browser callback returns
HTTP 401 with the generic code `oidc_callback_failed`; it does not expose the internal reason.
@@ -95,6 +120,10 @@ prerequisite fails.
## Browser sessions
This section applies only to **local and direct OIDC**. Upstream reuses the
portal's authenticated session at the proxy boundary, not a ThothII cookie;
its `/me` response has `session: null` and `csrfToken: null`.
The browser receives only an opaque `HttpOnly`, `SameSite=Lax` cookie named `thothii_session`.
State-changing cookie requests require the in-memory CSRF token, same-origin `Origin`, and Fetch
Metadata checks when present. The frontend never stores bearer tokens or session secrets in Web
@@ -111,5 +140,20 @@ affected sessions. Authentication configuration revision changes invalidate all
reload. Logout deletes the server record. Backup restore excludes active sessions and OIDC state,
recreates empty private auth-state directories, and therefore forces reauthentication.
Full local/OIDC logout calls `POST /auth/logout`, revokes the server session and
clears its cookie. It does not call the identity provider's global logout. If the
provider still has an SSO session, the next OIDC login can complete without
another password prompt. Embedded has no ThothII logout control: use the portal.
## Access revalidation
The frontend treats `/me` as the access authority. In embedded it does not fetch
`/auth/config` or offer local/OIDC login. On focus, pageshow, visibility return
and event-stream reconnection it rechecks access. A 401/403 from this probe clears
protected state; a 403 on one operation is not automatically an app-wide logout.
Neither the DOM adapter nor the proxy's initial SSE check guarantees instantaneous
revocation of streams already open in other tabs.
See the [local guide](../install/authentication-local.md), [generic OIDC guide](../install/authentication-oidc.md),
and [Authentik guide](../install/authentik.md) for operator procedures.
[Authentik guide](../install/authentik.md), [upstream integration](../install/authentication-upstream.md),
and [manual acceptance matrix](../testing/authentication-manual-acceptance.md).
+12 -1
View File
@@ -7,7 +7,9 @@ This page complements the [architecture overview](overview.md) with the module s
The frontend communicates with the backend through REST and SSE. The backend does not own session
persistence: it starts Pi, invokes the `tht` CLI, and forwards events. It does own the separate
installation-local database catalog. The harness contains the workflow, the Python CLI, and
adapters for the DWH and vector store.
adapters for the DWH and vector store. Its Memory module also owns the authoritative
PostgreSQL archive of cards, links, dependencies and pending Qdrant projections.
Administrative API calls use the same harness service as workflow producers and recall.
```mermaid
flowchart LR
@@ -20,6 +22,7 @@ flowchart LR
THT --> FS["Sessions and artifacts\nworkspace repository"]
THT --> DWH["DWH\nread-only"]
THT --> VDB["Qdrant / vector store"]
THT --> MEM["thoth_memory\nPostgreSQL Memory archive"]
BE --> CFG["settings.json\nworkspace + thinking"]
BE --> MODELS["generated runtime catalog\nfrom installation YAML"]
BE --> CAT["catalog-db\nPostgreSQL + Kysely"]
@@ -41,6 +44,14 @@ Dipendenze principali:
## Session sequence
The shared frontend is wrapped by `ShellProvider` (full preferences or a
replaceable portal presentation adapter), then `AuthGate` (backend identity),
then `AppShell`. Omics-specific DOM details belong only to `OmicsPortalAdapter`;
credentials and principal validation belong to the server, never that adapter.
Full/embedded do not duplicate the session workflow below. See
[rendering architecture](application-shell.md) and
[upstream identity](../install/authentication-upstream.md) for both boundaries.
The main path starts with a user question and ends with an SSE event. Reviewer decisions use the same channel and are persisted by the harness.
```mermaid
+8 -3
View File
@@ -4,8 +4,11 @@
ThothII is a **human-in-the-loop datamart builder**. It turns a natural-language question into validated SQL, and optionally a dbt datamart, through a **deterministic eight-phase NL-to-SQL workflow** in which the model *proposes* and a human reviewer *decides* at gates.
Production authentication uses local authentication or generic OIDC. `tht` is the only operator CLI.
For sessions, roles, groups, diagnostics, and recovery, see the [authentication documentation](authentication.md).
Production authentication uses local authentication, generic OIDC, or the
trusted-proxy upstream path used by Omics. `tht` is the operator CLI for the
installation and local/OIDC configuration. For roles and recovery, see
[authentication](authentication.md). One React build supports full and embedded;
[rendering architecture](application-shell.md) separates presentation from identity.
```mermaid
flowchart LR
@@ -101,7 +104,9 @@ the core can admit a new session.
- `tht -c`/`--config` is a **per-command** option. It must follow the subcommand, never precede it (`ThtRunner.buildArgv` enforces this).
- `--json` output must be plain JSON on stdout. It is a machine-readable contract.
- UI strings are in English. Document *content* stays in the workspace language because it is the actual data; only chrome and labels are in English.
- UI strings support English and Italian, with English fallback. Session interaction language
is pinned at creation; document content remains in the workspace language. See
[shell and localization](../operations/shell-and-localization.md).
- Each workspace defines identity and optional Evidence only. The PostgreSQL Metadata Catalog defines
its DWH target and binding; secrets remain in the protected workspace secret store.
- Settings are global (`backend/data/settings.json`: workspace/thinking); provider/model choices are
+65
View File
@@ -0,0 +1,65 @@
# Session corrections to Memory and Evidence
`reviewer_archive_repair` presents one to five closed alternatives for a conflict.
Each alternative updates one existing Memory Card or one existing local Evidence unit,
showing complete current and resulting content. The reviewer chooses one alternative,
rejects all as inadequate, or asks for reformulation. A correction does not approve or
advance the session phase; the ordinary gate still reviews its use in the current question.
The harness coordinator `tht/archive_repair.py` joins two independent domains. It uses
Memory's PostgreSQL repository, workspace mutation lock and projection service, and the
same local Evidence save/consolidation boundary as administration. Evidence activation
runs the existing corpus pipeline without reacquiring configured external sources.
No database binding, schema configuration, other archive, or Git repository is changed.
## Authority and recovery
Migration `004_archive_repairs.sql` stores session-bound proposals and receipts in
`thoth_memory.archive_repairs`, isolated by workspace RLS. Each receipt captures the
session decision context, current target revision, complete resulting content, selected
choice, acting principal and publication outcome. The preparation command changes no
archive content. Application accepts only an option ID from the persisted proposal.
Memory writes its new revision and receipt in one SQL transaction. Projection failures
leave a durable pending operation; retry propagates the saved revision. A later card
edit invalidates replay of the correction.
Evidence records the exact choice before writing its canonical file. Recovery accepts
either the reviewed original revision or the already-written approved result; it never
overwrites a different intervening correction. Before a first proposal, the editable
checkout must match its active snapshot. Other curated files are fingerprinted and
checked again before application/retry so a session decision cannot publish unrelated
external edits. A crash after replacement is recoverable by the same receipt. Original
document provenance is retained as the lineage of the manual correction.
The gate displays these outcomes:
| Status | Meaning |
| --- | --- |
| `proposed` | Waiting for a human choice; no content saved |
| `rejected` | All alternatives declined; reformulation required |
| `applying` | Choice recorded; file write or candidate recovery still required |
| `pending_activation` | Content saved; index activation incomplete |
| `active` | Saved correction matches the currently active revision |
| `superseded` | The target was removed or changed after the saved correction |
The reviewer may retry a pending correction or continue the current question while
leaving activation explicitly pending. The latter is not persistent-resolution success.
`repair-show` refreshes target status; `repairs` lists the historical receipt status.
The final Memory summary must not create a duplicate of a correction already saved here.
## Authorization and commands
Inspection/preparation requires an accessible open session in the same workspace.
Applying either archive correction requires an administrator. Browser responses are
also checked against the responding principal's `memory.manage` or `evidence.manage`
permission and the Pi runtime owner. An administrator using another principal's runtime
must resume it under their own account first, preserving truthful receipt attribution.
Non-administrators may reject proposals or continue question review without changing
the archives. The Pi shell guard blocks `repair-apply`; only the human gate invokes it.
The Python workflow CLI provides `memory repair-target`, `repair-prepare`, `repair-show`,
`repair-apply`, and `repairs`. These are workflow integration commands, not new native
installation commands. All accept `--session` and command-local `-c`; JSON output remains
machine-readable. Proposal bodies are bounded to 1 MB. Durable receipts support both
filesystem and server session storage without a persisted chat transcript.
+248
View File
@@ -0,0 +1,248 @@
# Editable Curated Evidence v4
Curated unit v4 makes the visible Markdown body authoritative. It uses the existing
typed Evidence payloads and stable identifiers. Workspace descriptor v4 and Evidence
descriptor v1/v2 are separate version numbers.
E1 implements the format, explicit conversion, local archive and consolidation API.
E2 connects that API to the installed consolidation command, Evidence management
page and runtime source selection. The local PSD preview now uses 35 converted units.
## Write or edit a file
Place units in `<workspace-root>/evidence/curated/<kind>/<name>.md`. Keep the existing
`id` when editing. The directory must match `kind`. A new manual unit needs no external
document, hash or encoded metadata:
```markdown
---
schema_version: 4
id: evidence:order-key
kind: domain
language: en
purposes: [sql_generation]
applies_to:
tables: [sales.orders]
---
# Order key
## Rule
Join orders using the order number, financial year and company.
```
Required metadata is `schema_version`, `id`, `kind`, `language`, and a nonempty
`purposes` list. Purposes are `disambiguation`, `rewriting`, `schema_linking`, and
`sql_generation`. Optional `applies_to` contains `concepts`, `tables`, and `columns`.
Tables use `schema.table`; columns use `schema.table.column`.
The first H1 is the title. H2 headings identify the payload fields below. Heading
spelling follows `language`: Italian for `it` and its regional variants, English
otherwise. Keep structural headings when editing the text below them.
| Kind | English field headings | Italian field headings |
| --- | --- | --- |
| `domain` | Rule | Regola |
| `glossary` | Definition, Synonyms, Variants | Definizione, Sinonimi, Varianti |
| `enum` | Column, Values | Colonna, Valori |
| `example` | Question, Interpretation | Domanda, Interpretazione |
| `mapping` | Concept, Tables, Columns | Concetto, Tabelle, Colonne |
| `normalization` | Input, Output, Rule | Input, Output, Regola |
| `formula` | Concept, Columns, SQL | Concetto, Colonne, SQL |
| `reference` | URL, Label, Description | URL, Etichetta, Descrizione |
List fields use one `- value` per line. Empty optional lists may be omitted. Quoted
JSON strings within bullets preserve unusual or multiline values during conversion.
Enum values use `### "stored value"`, followed by their meaning; `### ""` represents
an empty stored value. Formula SQL uses a fenced `sql` block containing one PostgreSQL
expression. Whole queries and mutation statements remain invalid.
Nested prose headings and fenced examples are supported inside text fields. An H2
matching a field heading is structural outside a code fence. Duplicate fields, missing
required fields, duplicate metadata keys and malformed payloads are rejected. There
is no second title or payload in frontmatter and no hidden authoritative rule text.
Conversion fails explicitly if a legacy payload cannot be represented losslessly.
## Current provenance and original source
The host supplies document provenance when refining source material: relative source
path, normalized source hash and exact supporting excerpts. A manual unit can omit
`provenance`. Consolidation records `kind: manual` and the supplied curator identity.
A visible change to a previously recorded unit becomes a manual declaration. If that
unit originated from a document, its former document provenance is retained under
`original`. The original excerpts establish lineage; they do not assert that the
source contains the new wording. Retrieval carries this distinction through typed
metadata. Curators edit content; the service updates managed provenance.
Unchanged document declarations still require matching source bytes and excerpts.
Replacing a source requires an explicit refresh through the E3 source-review path. Ordinary preparation
is blocked on an initialized local archive, preventing regenerated source material
from overwriting corrections or restoring deletions. Legacy `resolve` is likewise
blocked there; local file corrections and the archive API own those changes.
## Persistent archive and activation
```text
<workspace-root>/
.evidence-archive.lock
evidence/
source/ # acquired original documents
curated/<kind>/*.md # editable primary content
local-manifest.yaml # derived declarations and deletion records
.local/
state.yaml # baseline, pending and active revisions
snapshots/<revision>/ # immutable units, source bytes and manifest
```
Keep the complete Evidence tree and its managed metadata in backups and the operator's
Git review. Historical source bytes and deletion records are needed to preserve manual
care across subsequent imports. The separate corpus cache and Qdrant index are derived.
The lock file only coordinates local service operations.
`LocalEvidenceArchive.initialize()` records the pre-edit baseline without activation.
`consolidate(actor=..., activate=...)` validates files, derives provenance, records
deletions and source suppression, and creates an immutable candidate. The activation
callback receives that snapshot and must raise if indexing is blocked or fails. Only
successful activation advances `active_snapshot()`. Without a callback the result is
explicitly `pending_activation`; saving a file alone never changes this pointer.
The same candidate can be retried after an index failure. Interrupted managed writes
are replayed only when the operator's file bytes have not changed. A missing curated
directory is an availability error, not proof that all units were deleted. Removing
unit files from an accessible directory records deletion without deleting their sources.
The API also provides `get`, revision-checked `save`, and revision-checked `remove` for
future deliberate workflow corrections. Concurrent external edits produce conflicts.
These boundaries are exercised with the existing corpus pipeline and real Qdrant.
An initialized installation reads only its active local snapshot, including during
ordinary preprocessing. Unconsolidated edits are visible in Administration but do
not enter retrieval. Before initialization, the pinned repository source still works.
## Administration and installed command
Open **Administration → Evidence management**. This independent page requires the
`evidence.manage` permission and no active session. It shows complete units, source
lineage, review items, file errors, filters, and changes relative to the active snapshot.
Edit the displayed Markdown path using an external editor. Refresh files to inspect
the result, then run the command shown by the page:
```sh
tht --installation /absolute/path/thothii-installation.yaml \
workspace evidence consolidate --workspace psd-clinical
```
The installed command accepts `--json`. It validates and activates Evidence using
the existing corpus pipeline and embedding service. It does not scan the DWH, run
full workspace preprocessing, change Catalog/Schema readiness, or execute Git.
If indexing fails after saving, the archive retains the candidate for retry and the
previous active revision remains selected. Validation errors identify corrections
to make in the files. Structural and review checks still apply; initialized archives
do not require the legacy repository's fixed retrieval-evaluation fixture, whose
expected IDs would otherwise prevent deliberate local deletions.
The canonical workspace root is `<workspace-registry-root>/repo/<workspace-id>`.
Snapshots and the derived corpus are separate. For a registry in a Docker volume,
copy its existing `repo` to a persistent host directory before enabling
`deploy/compose.evidence-host.yaml`; set `THT_EVIDENCE_HOST_REGISTRY_ROOT` to that
directory and include the override in the installation descriptor. Core and
workspace-maintenance must mount the same checkout. Keep registry state/snapshots
on their existing volume. The page only presents a host path when configured; it
does not label an internal container path as a usable editor path.
After successful consolidation, inspect and commit the complete workspace Evidence
tree, including managed manifests, snapshots, and deletion records, then push manually.
The page provides quoted POSIX-shell examples for status, diff, add, commit, and push.
Do not commit only the edited Markdown. Ordinary Git operations remain the operator's
responsibility. E3 source decisions use this same activation boundary.
**Clear** removes derived Reference/corpus data while retaining editable files,
snapshots and Memory. It still requires full workspace preprocessing to recreate
Reference/Schema readiness; Evidence consolidation does not satisfy that gate.
## Import drafts and refresh sources
Place externally authored Markdown drafts in `<workspace-root>/evidence/incoming/`.
The specialist needs no installation account or database access to write a draft.
Copying the draft to the installation and choosing **Import or refresh sources** in
Evidence management explicitly starts acquisition and refinement. The equivalent
installed command is:
```sh
tht --installation /absolute/path/thothii-installation.yaml \
workspace evidence refresh --workspace psd-clinical
```
This reads local `incoming/**/*.md` and original `source/**/*.md` files, excluding
managed `source/acquired/` versions. It also reads configured HTTP and S3 sources
through the existing read-only adapters and network policies. After local archive
initialization, filesystem descriptors use this local authoring tree; ordinary
runtime/preprocessing never fetches remote source changes. No source-server write
credential is needed. The existing Pi authoring refiner runs once for each changed
document, without session state or tools. Unchanged source hashes skip refinement.
All acquisitions and proposals must succeed before the new comparison set is
recorded. An access or refinement failure preserves previous comparisons and active
Evidence. A source absent from a successful discovery is marked missing and never
treated as permission to delete units. Restore an accidentally missing local original
file before consolidating its document-derived units, or explicitly retire those units.
Each changed source has a durable comparison showing current local units, complete
proposed content, supporting excerpts, and IDs that replacement would retire:
- **Keep local Evidence** retains the current wording as a manual declaration, with
its original documentary lineage preserved. The changed source is acknowledged;
the next unchanged refresh does not reopen that decision.
- **Use proposed Evidence** adopts the displayed proposal and explicitly retires the
displayed omitted IDs. The source version and provenance change together.
Both choices save and activate through the same local consolidation/index pipeline.
Review items block adoption; correct the original draft and refresh, or keep local
content. There is no automatic merge based on a model's semantic conflict assessment.
Any change to an affected curated file invalidates the comparison and requires a new
refresh. If indexing fails after the decision is saved, use **Retry saved decision**;
this reuses acquired content without fetching sources again. An intervening external
edit is never silently overwritten by recovery.
Headless operators can make the same decision using the source ID and comparison
revision from the local source registry or administration response:
```sh
tht --installation /absolute/path/thothii-installation.yaml \
workspace evidence decide --workspace psd-clinical \
--source-id <64-hex-source-id> --revision <64-hex-comparison-revision> \
--decision keep
```
Use `--decision replace` to adopt the proposal. All installed commands accept `--json`.
The Python `evidence sources` worker is internal to this installed command/API surface.
`evidence/.local/sources.json` stores source identities, comparisons and retry journals.
`evidence/.local/acquisitions/` preserves original acquired bytes and credential-free
remote provenance. Adopted normalized documents live under
`evidence/source/acquired/<source-id>/<content-hash>.md`. Versioned paths let a new
document and an older manual declaration's original source coexist. Include all of
these files in the existing manual Git/backup sequence. Do not edit managed acquired
versions: edit the original local draft or refresh its remote origin.
Deleted IDs remain reserved. Once a source has had a curated deletion, fresh model
IDs from that source are conservatively suppressed as well: changing an ID must not
restore retired knowledge. Existing surviving IDs can still receive reviewed updates;
deliberate new knowledge can be written as a manual Evidence file. Refresh is bounded
to 200 documents and 100 MiB per operation, in addition to each adapter's limits.
## Convert an existing workspace
The existing workflow CLI command `tht evidence migrate <workspace-root>` converts
unit versions 1–3 to 4 deterministically and initializes the archive baseline. It
preserves IDs, typed content, provenance and review items, with no model call or Git
commit. It does not activate a local index. Review items still block consolidation.
The command's existing Git-worktree path check remains in effect.
The first installed consolidation performs this conversion automatically when the
legacy manifest is present, then validates and activates the result. Preserve the
existing checkout in backups before upgrading. The E1 validation used an isolated
copy; E2 also converted and indexed all 35 units on the running local preview.
See the [E1 validation report](../plans/2026-09-09-evidence-e1-validation.md) and
[E2 validation report](../plans/2026-09-09-evidence-e2-validation.md).
+135
View File
@@ -0,0 +1,135 @@
# Portal Shell Adapter v1
Contratto minimo della presentazione embedded. La revisione approvata il 2026-09-13
sostituisce il precedente trasporto a eventi personalizzati con l'osservazione del
documento condiviso. Non trasferisce identità, token o stato di autenticazione.
## Configurazione
Sul Mac:
```yaml
shell:
mode: full
defaultLocale: en
```
Sul server Omics:
```yaml
shell:
mode: embedded
adapter: omics-portal
```
Se `shell` o `mode` sono omessi, la modalità è embedded. L'adapter embedded
predefinito è `omics-portal`; un nome sconosciuto è un errore di configurazione.
Questi default sono normalizzati dal CLI prima della proiezione. Nel browser,
shell interamente omessa ha gli stessi default, ma un oggetto `shell` parziale
senza `mode` o `defaultLocale` viene rifiutato: non scrivere proiezioni a mano.
Full non istanzia adapter. `defaultLocale` inizializza full; in embedded il locale
proviene dal portale. L'autenticazione si configura separatamente dalla shell.
## API applicativa
```ts
export type PortalSnapshot = {
locale: string;
theme: "light" | "dark";
fullscreen: boolean;
};
export interface PortalAdapter {
subscribe(
onState: (state: PortalSnapshot) => void,
onError: (error: Error) => void,
): () => void;
}
```
Una sottoscrizione installa gli osservatori e consegna lo snapshot iniziale senza
richiedere messaggi all'altro applicativo. Gli aggiornamenti contengono snapshot
completi e validati. La disiscrizione elimina tutti i listener e osservatori.
La lingua viene risolta tramite i cataloghi UI, con fallback inglese.
## Implementazione Omics
L'integrazione monta React nel documento Django, non in un iframe.
| Dato | Fonte privata dell'adapter | Aggiornamento |
| --- | --- | --- |
| Locale | `data-lang` del selettore `.omics-language-select` | nuova pagina Django dopo `set_language` |
| Tema | `data-bs-theme` su `html` | osservazione limitata a quell'attributo |
| Fullscreen | stato effettivo del documento | evento del browser, inclusa uscita con Esc |
Il template Omics aggiornato allinea anche `html lang` alla lingua Django, ma
la fonte dell'adapter rimane `select.omics-language-select[data-lang]`. Leggere
il valore renderizzato evita di anticipare un cambio lingua prima che il form
abbia successo. Cambiare soltanto `select.value` o `data-lang` senza il normale
reload non è un trasporto runtime implementato per la lingua.
L'assenza del contesto host atteso produce un errore di integrazione; non abilita
controlli locali. Non si introducono eventi `ready/state`, handshake, timeout,
versioni dei messaggi o comandi duplicati. Selettori e dettagli Omics non devono
essere letti dai componenti applicativi.
## Proprietà per modalità
| Funzione | Full | Embedded |
| --- | --- | --- |
| Header | ThothII | solo Omics |
| Lingua | selettore locale | selettore Omics, normale reload Django |
| Tema | toggle locale light/dark | stato Omics |
| Fullscreen | controllo locale, stato reale | controllo Omics, stato reale |
| Login/logout | ThothII local/OIDC; upstream non offre logout locale | autenticazione Omics esistente |
| Nome utente | header ThothII | header Omics |
| Rotellina amministrativa | mai | eventuale comando del portale |
## Accesso e continuità
Il server Omics verifica l'accesso a Datamart Builder e il proxy trasmette i
principal header normalizzati al backend ThothII. La UI usa `/me`; non effettua
un secondo login. Un altro portale deve soddisfare anche questo contratto server,
oltre a fornire una nuova implementazione dell'adapter UI.
Il [contratto upstream](../install/authentication-upstream.md) specifica header,
origine, rete e configurazioni incompatibili. Lo snapshot non può contenere
`authenticated`, utente, ruoli, cookie o token; un evento browser non autorizza
una richiesta API. Il prefisso API viene configurato separatamente prima del
caricamento React, non viene dedotto dall'adapter.
Il logout del portale segue la sua navigazione. Una perdita di accesso rilevata
dal server chiude lo stato protetto; un 403 di una singola operazione non equivale
automaticamente a logout. La riconnessione degli eventi e il ritorno alla pagina
ricontrollano l'accesso. Non si garantisce revoca istantanea di una connessione
aperta in un'altra scheda attraverso il solo controllo iniziale del proxy.
Il cambio lingua può ricaricare la pagina: conservare la selezione della sessione,
proteggere le modifiche non salvate e non avviare una nuova generazione al reload.
Non si conserva una trascrizione integrale nel browser. La lingua della sessione
rimane quella registrata nel manifest, secondo ADR 0022.
## Sostituzione e verifiche
Un nuovo adapter può usare un diverso documento o trasporto, ma deve rispettare
la stessa sottoscrizione e mantenere la conoscenza del portale nella propria
implementazione. Oggi `ShellProvider` istanzia direttamente `OmicsPortalAdapter`:
per sostituirlo aggiornare quel punto e i nomi accettati in
`tools/tht/internal/config/shell.go` e `frontend/src/api/runtime-config.ts`.
Non è disponibile il caricamento dinamico di classi da una stringa YAML.
Non occorre implementare ora iframe o un secondo portale.
La nuova implementazione deve pubblicare uno snapshot iniziale completo, poi gli
aggiornamenti; segnalare contesto invalido; liberare tutti i listener alla
disiscrizione. Locale ben formato ma non tradotto significa fallback inglese;
locale assente/malformato e tema diverso da light/dark sono errori di integrazione.
Non cambiare componenti applicativi o workflow per aggiungere selettori specifici
del nuovo portale.
Verificare snapshot prima/dopo il montaggio, tema, fullscreen con Esc, cleanup,
contesto host mancante, assenza di header ThothII embedded, accesso singolo,
locale dopo reload e compatibilità del prefisso API. Full deve funzionare senza
alcun elemento Omics presente.
Vedere anche [architettura del rendering](../architecture/application-shell.md)
e [matrice di accettazione](../testing/authentication-manual-acceptance.md).
+8 -5
View File
@@ -6,8 +6,9 @@ When present, `evidence` is strict: it contains `source` and a defaulted strict
source variant and the policy reject unknown keys.
The version numbers are intentionally separate: the Evidence descriptor supports v1/v2, while the
latest Curated Evidence Unit format is v3. There is no Evidence descriptor v3/v4 and no Curated
Evidence Unit v4.
latest Curated Evidence Unit format is [v4](curated-evidence-v4.md). There is no Evidence
descriptor v3/v4. The editable local archive is implemented in E1; installation integration
is the next increment, E2.
```mermaid
flowchart LR
@@ -67,9 +68,11 @@ ignore it. Domain rules also retain their exact canonical text in an invisible `
comment while presenting long prose as paragraphs, labelled subsections, and semicolon-derived
lists. Runtime chunking reads the parsed canonical rule, not this review-only presentation.
Newly prepared units use v3. `tht evidence migrate <workspace-root>` upgrades v1 and v2 units and
canonicalizes an older v3 presentation locally without a model call, commit, publication, or
semantic change.
The representations above are legacy conversion inputs. Newly prepared units use editable v4:
short YAML metadata, a visible H1 title and typed H2 payload fields, with no hidden content copy.
`tht evidence migrate <workspace-root>` converts v1–v3 to v4 and initializes a local archive
baseline without a model call, commit, activation, or semantic change. See the
[v4 editing and consolidation contract](curated-evidence-v4.md).
### Example: filesystem
+26 -7
View File
@@ -14,12 +14,30 @@ tht --installation <absolute>/thothii-installation.yaml workspace preprocess run
tht --installation <absolute>/thothii-installation.yaml workspace preprocess clear
--workspace <id> [--json]
tht --installation <absolute>/thothii-installation.yaml workspace evidence consolidate
--workspace <id> [--json]
tht --installation <absolute>/thothii-installation.yaml workspace evidence refresh
--workspace <id> [--json]
tht --installation <absolute>/thothii-installation.yaml workspace evidence decide
--workspace <id> --source-id <64-hex> --revision <64-hex>
--decision keep|replace [--json]
```
There are no public partial commands for DWH introspection, LSH, FK suggestions, schema indexing,
Evidence indexing, or Qdrant rebuild. `preprocess run` does not accept `--resume`, `--dry-run`, a
or Qdrant rebuild. The separate Evidence curation command validates editable local files and
activates only their index; it does not satisfy workspace preprocessing readiness or execute Git.
See [Curated Evidence v4](curated-evidence-v4.md#administration-and-installed-command).
Source refresh acquires and refines only on explicit request, saving comparisons without
changing active Evidence. Source decisions activate through the same Evidence-only
pipeline and preserve Catalog readiness. Their envelopes reject arbitrary URLs, paths
and extra flags; source locations and credentials come from installation configuration.
`preprocess run` does not accept `--resume`, `--dry-run`, a
generation identifier, or a rollback option. Re-running it replaces the preceding derived output.
`preprocess clear` removes all replaceable preprocessing output and preserves runtime memory.
`preprocess clear` removes all replaceable preprocessing output and preserves runtime memory
and the canonical local Evidence archive. Full preprocessing remains required afterward.
## Sources of truth
@@ -29,8 +47,10 @@ generation identifier, or a rollback option. Re-running it replaces the precedin
- PostgreSQL Metadata Catalog is the sole database authority. It owns the workspace/database
association, installation-local binding, tables, columns, descriptions, sensitivity flags,
physical foreign keys, and active logical relationships.
- The workspace Git revision remains authoritative for Evidence. Evidence Descriptor v1/v2 and
Curated Evidence Unit v3 are unchanged; there is no Evidence v4.
- Evidence Descriptor v1/v2 still configures the initial source. Initialized archives use
editable Curated Evidence Unit v4 and the last successfully activated local snapshot.
Ordinary preprocessing never imports unconsolidated working-tree edits. Before initialization,
the pinned workspace Git source remains supported.
No metadata is imported from legacy workspace YAML or `physical.yaml`/`annotations.yaml`.
@@ -98,9 +118,8 @@ generation produced for another database or revision.
`workspace preprocess clear` is intentionally narrower than deleting all semantic data. It:
1. copies any `memory` and `solved_question` records still present in the pre-split workspace
collection to `<workspace>-memory`, then retires that legacy collection (a normal preprocessing
write performs the same one-time cutover if clear was not invoked first);
1. leaves `<workspace>-memory` unchanged; Memory projections are reconstructed only from
the authoritative PostgreSQL archive, never imported from legacy vector payloads;
2. deletes `<workspace>-reference`;
3. removes the active LSH generation, Evidence corpus, private Catalog snapshot, and derived job
checkpoints;
+52 -3
View File
@@ -2,7 +2,30 @@
This page describes the complete workspace Evidence lifecycle: where original material lives, how curated units are produced, when they become available at runtime, and what the author and reviewer are responsible for.
## Publication rule
## Editable local Evidence
E1 adds [Curated Evidence v4 and a persistent local archive](contracts/curated-evidence-v4.md).
The visible title and payload fields are authoritative Markdown. New manual units need no
external source; consolidation records their curator and distinguishes later corrections
from original documentary provenance. The core archive API creates immutable candidates
and advances its active pointer only after successful indexing.
E2 adds **Administration → Evidence management**, actual host file paths, complete
browsing and filtering, and the installed `tht workspace evidence consolidate
--workspace <id>` command. Edit files externally, consolidate to activate them, then
review and run Git manually. Runtime consumes only the active local snapshot.
The [v4 contract](contracts/curated-evidence-v4.md#administration-and-installed-command)
describes host mounting, first conversion, failure recovery and Clear behavior.
The local PSD preview has 35 converted and indexed units. E3 adds explicit import/refresh
from local drafts and configured HTTP/S3 sources, durable current/proposed comparisons,
and keep/replace decisions with activation and retry. See
[Import drafts and refresh sources](contracts/curated-evidence-v4.md#import-drafts-and-refresh-sources).
Workflow gate corrections remain the subsequent shared increment, X1.
## Existing repository publication path
The remainder describes the legacy, uninitialized repository source path. Initialized
v4 local archives use the lifecycle above; manual declarations need no source document.
The workspace repository is the versioned source. ThothII reads it, validates it, and publishes an atomic generation. It does not modify, commit, or push the author's repository.
@@ -46,7 +69,33 @@ The legacy configuration may expose `source_root`, such as `${THT_DOCS_ROOT}` or
HTTP and S3 are separate adapters. They do not use the filesystem structure `source/` and `curated/`, but they must still provide stable provenance, without credentials in URIs, under the adapter-specific contract.
## What a curated unit must contain
## What an editable curated unit contains
New preparation produces unit schema v4. The first H1 contains its title; documented H2
sections contain the typed payload. A minimal manual unit is:
```markdown
---
schema_version: 4
id: evidence:order-key
kind: domain
language: en
purposes: [sql_generation]
---
# Order key
## Rule
Join orders using the order number, financial year and company.
```
Use `tht evidence migrate <workspace-root>` for deterministic legacy conversion. The
conversion preserves typed content and initializes an archive baseline; it does not
activate the local corpus. See the [v4 contract](contracts/curated-evidence-v4.md) for
all eight kinds, provenance, file layout and the E1/E2 boundary.
## Legacy v3 representation
Canonical Curated Evidence v3 hides canonical machine metadata in an HTML comment and renders the
whole review surface as real Markdown. GitHub therefore shows no frontmatter table. The body layout
@@ -94,7 +143,7 @@ La fascia pediatrica comprende i pazienti con età inferiore a 18 anni.
The actual files contain invisible `tht:` comments for canonical metadata and typed-field
boundaries. Removing, duplicating, or desynchronizing them makes validation fail closed instead of
silently ignoring content. Unit schemas v1 and v2 remain readable for compatibility, but newly
prepared units use v3.
prepared units use v4. Unit v3 remains readable as a conversion input.
Curated units must be atomic, readable by a second reviewer, and supported by the source.
Provenance references must lead back to the original file and the passage that supports the claim.
+85 -9
View File
@@ -8,14 +8,26 @@ declare providers, model allowlists, defaults, embeddings, dimensions, or vector
Do not edit `deploy/pi/models.json`, `deploy/pi/settings.json`, files under `generated/`, or
provider/model environment defaults. Those former sources are retired.
## Host instructions in Administration
The **Pi configuration** navigation button opens **Pi management**. Its Host maintenance
section selects the installation host's Linux, macOS, or Windows tab automatically; the
browser's operating system does not affect it. Selecting another tab is still possible.
The host CLI includes `THT_HOST_PLATFORM` in the generated Compose projection using the OS
on which it runs. Generate projections on the destination host, not on another computer,
and do not edit generated files. Without this projection (older deployments or native
development), the backend reports its own OS; a Linux Docker container cannot discover
whether the physical host is macOS or Windows. Run the updated host CLI's normal
configuration reload on the destination installation to regenerate this information.
## Minimal catalog
```yaml
schemaVersion: 2
modelCatalog:
defaults:
session: zai/glm-5.3
metadataGeneration: zai/glm-5.3
interaction: zai/glm-5.3
embedding:
id: ollama/qwen3-embedding:0.6b
@@ -50,23 +62,63 @@ it contains that use block:
- `metadataGeneration` makes it selectable for description generation;
- `embedding` is a single installation-level model rather than a selectable list.
`defaults.session` is required. `defaults.metadataGeneration` is required exactly when at least
one metadata-generation model exists. A session manifest pins its canonical identity, so removing a
model never silently changes an existing session: resume fails with `model_unavailable`.
`defaults.interaction` is the only LLM default, required once per installation, never per workspace.
Core and Administration share the user's operational model choice. An explicit choice takes priority
over the default and is remembered in this browser for the authenticated user and application mount.
Switching workspace does not change the model. Browser-storage restrictions may limit remembering
to the current visit; preferences do not synchronize across devices or change installation YAML.
An unavailable remembered model is not silently replaced: select another configured model.
When any metadata-generation models are configured, the default and the operational list must
support both `session` and `metadataGeneration`. Entries for just one adapter may remain in the
installation inventory, but are not selectable for global interaction. Core-only installations remain
supported when no metadata-generation model is configured; Admin AI is then unavailable.
The embedding model remains separate and is unaffected by the interaction selector.
New sessions record the selected model in their manifest. Resume retains the session's workspace and
revision, but uses the current global model (the installation default for clients that omit a model).
Historical manifest model fields are not rewritten by resume. Archived/finalized sessions remain read-only.
### Required operator verification for every model
Catalog validation checks configuration, not model behavior. Before offering a model to users, and
after changing its endpoint, adapters, or the Pi/LiteLLM versions, the operator must verify **both**:
1. **Core / Pi:** select the model, start a test session in a prepared test workspace, exercise an
actual tool call and its returned result, a human review gate, and stop/resume. Check streaming,
tool arguments, authentication, and reasoning/token-limit compatibility. A plain chat reply or
`tht pi test` alone is not sufficient.
2. **Administration / LiteLLM:** select the same model and generate descriptions for a small,
non-sensitive test table. Check the structured result is accepted and the generation completes.
Review the output quality before using it on real metadata. This action writes test metadata
and may incur provider charges: use an authorized test database and approved data.
There is no automatic certification flag or startup model probe. The operator owns this verification;
do not infer compatibility from the model label or from success in just one path. Both adapters point
to one catalog identity; Pi does not need to route through a new LiteLLM proxy. Models using only
`pi_auth` cannot serve the current LiteLLM path and are excluded from shared selection.
## Session adapters
Use `pi_builtin` for a model whose technical definition ships with Pi:
Use `pi_builtin` for a model whose technical definition ships with Pi. This does not require
`pi_auth`: a shared bundle credential lets native Pi and LiteLLM use the same provider identity:
```yaml
deepseek:
authentication:
mode: pi_auth
mode: secret_env
apiKeyEnv: DEEPSEEK_API_KEY
session:
mode: pi_builtin
metadataGeneration:
litellmProvider: deepseek
models:
deepseek-v4-pro:
session: {}
metadataGeneration: {}
deepseek-v4-flash:
session: {}
metadataGeneration: {}
```
Use `openai_compatible` for an explicit compatible endpoint. Each eligible session model must then
@@ -89,6 +141,18 @@ Secret values never belong in installation YAML, generated files, logs, CLI argu
requests. The YAML contains only an environment-variable name or an authentication mode. Pi's
protected credential file remains selected by the installation authentication configuration.
For catalog providers using `secret_env`, the bundle is authoritative in both Core and Admin.
ThothII removes only the selected provider's old auth entry from the temporary Pi session snapshot;
the operator's original Pi auth store and other providers are unchanged. Provider smoke checks use
the same precedence, and model enumeration receives the catalog-declared bundle keys. A missing
declared key is an error, not permission to fall back to Pi auth or the legacy generic key file.
After rotating a bundle key, apply the normal installation lifecycle so processes reload it.
The PSD descriptor now declares only `deepseek/deepseek-v4-pro` and `deepseek/deepseek-v4-flash`
for both uses; it no longer duplicates them under `deepseek-metadata`. Historical records are not
rewritten. A saved obsolete identity must be explicitly reselected from the current catalog;
it is not silently remapped to another model or account.
## Generated runtime projections
Before Compose starts, `tht` validates the installation and atomically writes deterministic files
@@ -133,6 +197,16 @@ configuration command. There is no `tht pi configure` and no separate apply comm
## Migrating a legacy installation
For schema-v2 descriptors with the former `defaults.session` and `defaults.metadataGeneration`,
replace both with `defaults.interaction`. Equal legacy values are accepted and normalized in memory;
the loader never rewrites the descriptor. Different values fail with `migration_required`: explicitly
choose a model supporting both uses, remove both old fields, and set the single new field. Do not mix
new and legacy fields. A Core-only legacy session default can be normalized when Admin AI is absent.
The generated runtime catalog now uses schema version **2** and only `defaultInteraction`. Regenerate
and apply all runtime projections with the matching host/backend release using the normal installation
lifecycle; do not deploy only the backend against an old generated catalog or hand-edit generated JSON.
The migrator reads the former installation `metadataGeneration` block and the two former Pi JSON
files, but never modifies them. Supply the facts that cannot be inferred safely and write a separate
candidate:
@@ -148,13 +222,15 @@ tht --installation /absolute/path/legacy/thothii-installation.yaml installation
Review the candidate, move the legacy source files out of the installation only after approval,
then select the v2 descriptor. Ambiguous aliases, endpoint conflicts, or missing authentication
facts produce field-level errors; the migrator does not guess.
The legacy CLI flag `--session-default` now supplies the unified interaction default in the candidate;
if it conflicts with the legacy metadata default, align that choice explicitly before retrying.
## Troubleshooting
| Symptom | Meaning | Action |
| --- | --- | --- |
| `migration_required` | A retired model source or installation schema is still present | Run the installation migrator and review its candidate |
| Unknown session or metadata default | The canonical ID is missing the corresponding use block | Correct the provider/model key or add the intended use block |
| Invalid interaction default | The canonical ID does not support all configured uses | Correct `defaults.interaction` or the intended adapter blocks |
| Generated projection drift | Runtime files differ from the descriptor-derived bytes | Run `tht start` or `tht pi restart --yes --drain` |
| `model_unavailable` on resume | The session's pinned model is no longer session-eligible | Restore that catalog entry or keep the session unavailable; do not remap it |
| `model_unavailable` on create/resume | The selected global model is no longer eligible | Explicitly choose an eligible model; no fallback is applied |
| Provider smoke failure | Credentials, endpoint, or provider availability is invalid | Correct the protected credential or catalog endpoint, restart, then run `tht pi test` |
+208 -222
View File
@@ -1,244 +1,230 @@
# Memory management
This document describes how Memory currently works in ThothII: its conceptual model, lifecycle, persistence, semantic search, review gates, session display, and main technical limits.
Memory holds reusable knowledge for a workspace. PostgreSQL is the authoritative
archive; Qdrant contains a rebuildable search projection. The harness owns both
the administrative operations and the verification of retrieved results.
## Architectural summary
## Administration
A Memory item is domain knowledge that can be reused across questions. It is not a copy of one question's schema linking.
Open **Administration → Memory management**, immediately after Database management.
An administrator selects the workspace explicitly. No active session, DWH binding,
embedding service or Qdrant connection is required to browse and edit the archive.
PostgreSQL must be available.
The page provides a paginated list, text/ID search, stable sorting, and combined
filters for family, concepts, database, table, column, origin and update date.
Filtering happens across the entire archive before pagination. Each card has a
complete detail view, an editable form, cancellation of unsaved changes, and an
explicit deletion confirmation.
| Family | Content |
| --- | --- |
| Domain clarification | A reusable definition or interpretation, with scope and context. |
| SQL rule | Guidance for constructing SQL, with scope and rationale. |
| Solved question | A question, its approved SQL and context; used as a consultative exemplar. |
| Explained error | A correction and its rationale, to avoid repeating a known error. |
All cards have a stable `mem-<UUID>` identity, title, scope, origin and timestamps.
Manual cards have no invented source session or decision. Workflow cards retain
their source references when an administrator edits them. There is no editorial
revision history.
The form also manages concepts, structured database/schema/table/column
dependencies, and links to other cards with an explicit meaning. Links can only
connect cards in the same workspace. Card, dependency and outgoing-link changes
are committed together. Deleting a card removes its incident links and dependencies,
while keeping the other cards, Evidence and Catalog metadata.
## Save, failure and recovery
```mermaid
flowchart TB
CLARIFY["F1 concept clarified"] --> REVIEW["F8 reviewer review"]
REVIEW -->|"accepted"| REGISTRY["registry.jsonl"]
REVIEW -->|"declined"| LOCAL["Session decision only"]
REGISTRY --> VECTOR["Qdrant semantic index"]
VECTOR --> FUTURE["Future F2 retrieval"]
FUTURE --> PROPOSAL["Reviewer proposal"]
flowchart LR
EDIT["Admin or workflow"] --> SQL["PostgreSQL transaction"]
SQL --> CARD["Current card and links"]
SQL --> WORK["Pending projection"]
WORK --> Q["Qdrant"]
Q --> CHECK["Verify current card and projection"]
CARD --> CHECK
CHECK --> REVIEW["Recall for human review"]
```
```text
F1: clarify a concept
│
▼
concept_clarified decision in the session ledger
│
▼
F8: reviewer decides whether to promote it
│
├── global registry registry.jsonl
└── Qdrant semantic index
│
▼
F2 in a future session
search and proposal to the reviewer
```
The service serializes each workspace's mutations. It first commits content and
the projection operation in PostgreSQL, then propagates to Qdrant. A failed save
is different from **saved, index update incomplete**. In the latter case the
current content is already available in administration, while its previous vector
result is excluded from recall.
The main invariant is `REUSABLE_TYPES = {"concept_clarified"}`: only clarified concepts can be generated, saved, searched, or proposed as Memory. Decisions such as `table_promoted`, `table_excluded`, and `column_promoted` remain local to the question.
The **Pending index updates** section offers an explicit retry, including cleanup
for deleted cards. These operations survive a restart. Retry reads the current
archive state and cannot restore an earlier edit or a deleted card. Technical
revisions are internal consistency markers, not user-managed card statuses.
## The three management levels
Every recall hit must resolve to a current card in the requested workspace, with
a matching valid projection. The response is reconstructed from PostgreSQL, never
from an unverified Qdrant payload. Deleted, orphaned and stale points are excluded
for both domain Memory and solved-question exemplars. An unavailable archive is
an operational error, not a successful empty archive.
| Level | Content | Function |
Rebuilding Memory uses only authoritative cards. It does not import JSONL files,
historical session artifacts or old vector payloads. Reference preprocessing keeps
the Memory collection separate and does not migrate legacy Memory payloads.
## Hybrid recall and links
Memory uses dense embeddings and Qdrant BM25, fused with reciprocal rank fusion.
Both branches receive the same workspace, family and scope filters before candidate
selection. Indexed text includes content, scope, rationale, question, exemplar SQL,
concepts and qualified physical dependencies. Both indexing and querying use the
workspace language (`en` or `it`), including manual administration without a DWH binding.
The workflow CLI binds recall to its configured database and schema. Optional
`--filters` JSON can narrow the business `scope`, `concepts`, `table` and `column`.
Business scope is an exact string; every requested concept must be present. Physical
context matches a single structured dependency: database, schema, table and column
cannot be satisfied by unrelated entries. A card without dependencies is workspace-wide;
a database-only or table-only dependency also applies to descendants. Without a table
filter, table-specific knowledge in the selected database/schema remains discoverable.
Filters cannot override the configured database/schema. Free-text business scope is
not automatically interpreted or inferred from the question.
The core expands outgoing links from current, eligible search candidates. It applies
the same filters and workflow-family restrictions to destinations and intermediate
cards. Removed, pending, previously decided and out-of-scope cards cannot act as bridges.
Traversal allows two hops, at most 20 outgoing links per card (stable target-ID order),
200 distinct card lookups and 400 inspected links in total. Search requests
`min(100, max(20, 3 × top))` seeds; the result limit remains between 1 and 100.
All candidates are ranked together using `1 / (60 + seed rank)` for direct hits,
plus the strongest linked contribution, decayed by `0.5` per hop. Repeated paths
do not accumulate votes. Ties use card ID. The returned `score` is a ranking score,
not cosine similarity or a confidence estimate. `retrieval.path` shows the strongest
link path, or just the card ID for an exclusively direct result. PostgreSQL content,
links and eligibility are resolved under the workspace operation lock after search.
Migration `002_hybrid_projection.sql` marks the format of old dense projections as
incompatible without changing their authoritative cards. They appear in pending
updates and are excluded from recall until explicit retry or `memory index` succeeds.
Memory writes can add a missing BM25 sparse vector to their collection; an incompatible
existing vector configuration fails visibly and leaves recovery pending. Reading never
silently falls back to dense retrieval. Reference remains independently managed.
An explicit `memory index` can also recreate a missing Memory collection before
rebuilding its projections; it never imports records from another source.
## Workflow integration
During the workflow, Pi prepares reusable proposals in the session artifact
`memory_proposals.json`. Each proposal names effective approved source decisions,
the content and scope, its rationale, and any physical dependencies or links.
Unexplained failures, rejected options and simple table selections do not create
reusable knowledge. Exact existing cards are reused; semantic similarity alone
never authorizes replacement. Updates name the existing card and its current revision.
At F8, `reviewer_memory_promote` presents one editable Memory summary, including
the approved solved question. The reviewer chooses additions and updates, edits
their content, scope, dependencies and links, or declines everything. Approved SQL
is read only here: changing the solution requires returning to SQL review.
Only selected cards and their links are committed. Invalid links or stale updates
roll back the entire selection. Links between selected new cards are resolved
inside the same transaction. An explicit update preserves the existing identity
and origin, including manually authored content.
A durable review receipt makes repeated delivery idempotent and recovers the
gap between saving Memory and recording `memory_summary_reviewed` in the session
ledger. Retry never recreates a deleted card. The gate then closes F8 and finalizes;
finalization itself performs no automatic Memory writes. Pending indexing remains
visible and recoverable in administration.
F2 consumes domain clarifications and excludes already decided Memory. In F4,
F6 and F7, `memory rules` retrieves applicable SQL rules and explained errors.
Pi presents their use in the existing schema, CTE or SQL approval gate.
Exemplar search remains consultative. Retrieval never constitutes approval.
Persistent Memory/Evidence conflict repair remains part of the joint X1 increment.
## Physical schema changes
After a successful physical Catalog synchronization, the backend passes the exact
removed tables and columns, database, schema and run identity to Memory. Only cards
with matching structured dependencies are deleted, together with their incident
links and searchable projections. Global cards and objects outside the synchronized
scope survive. Manual Catalog cleanup and failed DWH scans never trigger this deletion.
The Catalog transaction records a pending `memory_cleanup` phase before committing
the physical change. If cleanup or indexing fails, the run retains the original
removals. Retry completes that same operation without rescanning the DWH or relying
on a new diff. A new synchronization is blocked until this cleanup is completed.
The Memory deletion receipt and vector tombstones make the operation repeatable
across restarts. The synchronization drawer reports deleted Memory cards and errors.
## API and commands
The administrative HTTP surface requires `memory.manage`, included in the existing
admin role. The backend checks workspace identity and passes its trusted principal
and a protected request snapshot to the harness. The harness independently checks
the principal; ordinary users cannot bypass administration through the CLI.
Production authentication and CSRF protections apply to the new routes.
| Method | Path below `/api/workspaces/:workspaceId/memory` | Operation |
| --- | --- | --- |
| Session ledger | `concept_clarified`, `memory_promoted`, `memory_promotion_declined` | Audit and state for one session |
| Global registry | `mem-XXXX` records in `registry.jsonl` | Current canonical Memory archive |
| Qdrant index | Embeddings and metadata derived from the registry | Semantic search |
| GET | root | Search, filter and paginate the archive |
| POST | root | Create a card |
| GET | `/:cardId` | Read a complete card |
| PUT | `/:cardId` | Save content, links and dependencies together |
| DELETE | `/:cardId` | Delete a card and incident links |
| GET | `/pending` | List incomplete projection operations |
| POST | `/:cardId/retry` | Retry current projection work, including deletion |
The ledger contains provenance and human decisions. The global record contains reusable text. The vector index is a search projection, not the place where the workflow records decisions directly.
## What can become Memory
During F1, the workflow records clarifications as `concept_clarified` decisions. A clarification can express:
- definitions of production or organizational concepts;
- inclusion and exclusion criteria for a product line;
- formulas and calculation methods;
- interpretations of time periods;
- mappings to specific tables and columns;
- the meaning of flags, codes, or indicators.
The `MemoryRecord` model contains:
- `id`, such as `mem-0001`;
- the timestamp, session, and sequence of the original decision;
- `type`;
- `subject`;
- `detail`;
- `rationale`;
- `question_context`;
- `tables` e `concepts`.
For new Memory items, the type is always `concept_clarified` and `tables` starts empty. A table or column may appear in the explanation as a technical mapping, but it cannot be the Memory item's standalone concept.
Valid example:
> Ablation means a procedure with `ablazione_transcatetere = TRUE`, counted with `COUNT(DISTINCT cod_paz)` by year.
Invalid examples:
- `fact_cardioversione` as approved Memory;
- `dim_time` as rejected Memory;
- an "include this table" decision saved for future questions.
The workflow skill also documents this rule in [harness/.pi/skills/tht-sessione/SKILL.md](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/harness/.pi/skills/tht-sessione/SKILL.md#L217).
## Promotion at the end of the session: F8
At the end of the workflow, the `reviewer_memory_promote` gate runs a deterministic preview:
CLI configuration remains a per-command option. Commands emit pure JSON:
```text
tht memory promote --session <id> --preview --json
tht memory list --filters '{"family":"sql_rule","page":1}' -c <runtime.yaml>
tht memory show <card-id> -c <runtime.yaml>
tht memory create --data <card.json> -c <runtime.yaml>
tht memory update <card-id> --data <card.json> -c <runtime.yaml>
tht memory delete <card-id> --yes -c <runtime.yaml>
tht memory pending -c <runtime.yaml>
tht memory retry <card-id> -c <runtime.yaml>
tht memory index -c <runtime.yaml>
tht memory search "<question>" --session <id> --json -c <runtime.yaml>
tht memory search "<question>" --filters '{"table":"orders","column":"id","scope":"Sales"}' --json -c <runtime.yaml>
tht memory solved-search "<question>" --json -c <runtime.yaml>
tht memory rules "<question>" --session <id> --json -c <runtime.yaml>
tht memory propose --session <id> --data <proposals.json> -c <runtime.yaml>
tht memory summary --session <id> --json -c <runtime.yaml>
tht memory solved-index <session-id> --json -c <runtime.yaml>
```
The preview:
`memory index` rebuilds both domain and solved-question projections.
`solved-index` only retries an existing authoritative source receipt.
The reviewer gate owns `memory review-apply`; Pi must not call it directly.
Migration `003_review_receipts.sql` adds durable review and physical-cleanup receipts.
The backend uses `memory admin --workspace <id> -c <protected-request.json>` so
administration does not materialize session or DWH configuration.
1. reads the session's effective decisions;
2. considers only `concept_clarified`;
3. discards decisions already promoted;
4. discards sequences already declined in F8;
5. deduplicates equivalent content;
6. proposes at most five candidates.
## Installation and storage
The code applies filtering and deduplication in
[harness/tht/memory/core.py](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/harness/tht/memory/core.py); the gate applies an additional defensive filter in
[harness/.pi/extensions/gate/memory/index.js](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/harness/.pi/extensions/gate/memory/index.js).
Memory shares the installation's existing PostgreSQL service, using its own
`thoth_memory` schema and versioned harness migration pack. The existing
`catalog-migrate` preparation service runs Catalog migrations followed by
`python -m tht.memory.migrate`. The core image includes both migration runners.
Ordinary API requests never migrate the schema.
The reviewer sees one preselected checklist. For each candidate:
Connection credentials come from the existing generated `THT_CATALOG_DB_HOST`,
`THT_CATALOG_DB_PORT`, `THT_CATALOG_DB_NAME`, `THT_CATALOG_RUNTIME_USER` and
`THT_CATALOG_RUNTIME_PASSWORD_FILE`. Migration uses the corresponding migrator
user/password file. Direct URL environments can use `THT_CATALOG_DATABASE_URL`
(runtime) and `THT_CATALOG_MIGRATOR_DATABASE_URL`; the harness also accepts
`THT_CATALOG_RUNTIME_DATABASE_URL`. Credentials are not authored in workspace YAML.
- selected: `tht memory save-one` runs, followed by a `memory_promoted` record;
- deselected: records `memory_promotion_declined`;
- no candidates: F8 closes automatically.
The migration grants the installation login membership in the restricted
`thoth_memory_runtime` role. Every repository transaction sets that role and a
workspace context. Forced row-level policies isolate cards, links, dependencies
and projection operations. This role has Memory DML and migration-status read
access, with no runtime DDL privilege. There are no cascading foreign keys to
Catalog or Evidence. A missing or incompatible schema returns a clear operational
error. Preparation is repeatable and checks migration checksums.
The `memory_promoted` marker uses `detail: seq:N`, a reference to the original `concept_clarified` decision. The flow is in [harness/.pi/extensions/tht-gate.js](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/harness/.pi/extensions/tht-gate.js#L1691).
Promotion does not run in F2, and the model cannot invent F8 candidates. Direct promotion commands are also protected by the anti-bypass gate.
## Global persistence
### JSONL registry
The current registry is:
```text
<artifacts>/memory/registry.jsonl
```
The registry is written through a temporary file and `os.replace`, so replacement is atomic. Promotion is idempotent on the `session_id + decision_seq` pair: the same decision from the same session cannot create two global records.
### Qdrant
After promotion, `save-one` builds one `VectorRecord` and sends it to the Qdrant index. The indexed text includes:
- type and subject;
- detail;
- rationale;
- question context;
- any concepts and mappings.
The vector record uses the ID `memory:mem-XXXX`. Its metadata stores `subject`, `detail`, `rationale`, `tables`, `concepts`, and the `kind` discriminator. The content's SHA-256 hash prevents embedding and upsert work when the text has not changed.
This behavior is implemented in
[harness/tht/memory/core.py](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/harness/tht/memory/core.py).
### Current canonical source
The JSONL registry remains the application's canonical source, while Qdrant is a derived but persistent index. The workflow does not record decisions directly in the vector database. It uses Qdrant as a searchable projection of the registry and effective ledger.
## Reuse in F2
In a future session, F2 runs:
```text
tht memory search "<question>" --session <id> --json
```
The command:
1. creates an embedding for the question;
2. searches the vector store for records with `kind=memory` only;
3. resolves each hit in the JSONL registry through its `ref`;
4. discards records missing from the registry;
5. discards every type other than `concept_clarified`;
6. excludes Memory already decided in the current session;
7. returns results ordered by similarity.
Memory is never applied automatically. The model must present it in one `reviewer_decide` choice:
- a selected Memory item is recorded as a new `concept_clarified` in the current session;
- the rationale must cite the original `mem-XXXX` ID;
- a deselected Memory item means "do not apply it now", not "delete it globally".
If F2 is reopened, a deselected Memory item can be proposed again. `memory_rejected` remains supported for legacy decisions and sessions, but it is not the normal behavior for current F2 deselection.
## Effective ledger, rollback, and reopening
The ledger is append-only. Reopening and withdrawing decisions do not delete earlier rows, but they change which decisions are effective.
Memory helpers use `effective_decisions()` to:
- exclude withdrawn decisions;
- ignore decisions from phases that became stale after a rollback;
- prevent promotion of clarifications that are no longer valid.
The effective view is defined in [harness/tht/phase.py](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/harness/tht/phase.py#L82).
## Display in the session summary
The "Memories" section of the summary is projected at runtime from the session ledger; it is not a direct copy of the global registry.
The projection:
- shows approved Memory first;
- then shows declined Memory;
- resolves `seq:N` to the original `concept_clarified`;
- hides markers whose original record is not `concept_clarified`;
- hides subjects that match schema-linking tables;
- hides standalone subjects shaped like `fact_*` or `dim_*`;
- keeps table and field references when they are part of the conceptual explanation.
The logic is in [harness/tht/session/store.py](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/harness/tht/session/store.py#L237). The frontend renders a structured list and treats `subject`, `detail`, and `rationale` as Markdown instead of showing raw Markdown.
The transformation happens on read. Historical sessions use the current layout and filters without rewriting their original artifacts.
## Enforced invariants
The protections are distributed across several boundaries:
1. `REUSABLE_TYPES` in the Python core;
2. the `memory search` command filter;
3. the F8 preview filter;
4. filtering and deduplication in the Pi gate;
5. exclusion of table Memory from the UI projection.
This prevents one prompt or component change from reintroducing tables as Memory.
## Remaining limits and risks
### The registry and index are not one transaction
Saving broadly follows this sequence:
```text
JSONL registry → Qdrant → memory_promoted marker in the ledger
```
If Qdrant is unavailable, the registry can contain Memory that is not yet searchable. The command reports that reindexing is required.
If the ledger marker fails after the vector database save, the Memory item can exist globally without a complete session audit. The gate returns a manual recovery command.
### Five-candidate limit
F8 proposes at most five Memory items. If a session produces more than five valid concepts, the extra items are not shown and the session can be finalized without promoting them.
### Deduplication is not global
Deduplication prevents duplicates within one proposal, and idempotency prevents the same decision from being promoted twice. There is no global merge of semantically similar Memory items from different sessions.
### Old physical records
Old `table_promoted` or `table_excluded` records may still exist in historical artifacts or indexes. The current code makes them unusable by filtering by type and does not show them in session projections. Physically removing them from the vector database remains a separate cleanup task.
## Final assessment
The current implementation matches the functional requirement: Memory is reusable conceptual knowledge, not a schema-linking choice.
The strongest part is the multilayer protection of the `concept_clarified` type. The main technical debt is the coexistence of the JSONL registry and Qdrant, with no single transaction spanning the global archive, semantic index, and session ledger.
See [the M1 specification](plans/2026-09-08-memory-m1-spec.md) and
[ADR 0018](adr/0018-use-postgres-for-memory-and-qdrant-for-retrieval.md) for the
approved scope and acceptance boundaries.
See [M2 implementation and validation](plans/2026-09-09-memory-m2-validation.md)
for the retrieval checks and real embedding test command.
+39 -3
View File
@@ -4,10 +4,30 @@ This guide is for a reviewer using a configured ThothII installation. Installati
publication, preprocessing, and database administration are separate paths; links to them are at
the end of this page.
## Standalone or inside Omics
In **full** mode, ThothII has its own red header. Sign in using the installation's
local account or the configured identity provider. The header lets you select
English/Italian, light/dark, and fullscreen; Esc exits fullscreen. Open the user
name menu to log out of ThothII. OIDC logout does not necessarily log out other
applications using the same provider.
In **embedded** mode, first sign in to Omics and choose **Datamart Builder** in
its left menu. ThothII opens with that authenticated identity: there is no second
login or duplicate header. Use Omics's language, theme, fullscreen and logout
controls. If portal access expires, return to Omics, sign in and reopen the page.
The Mac starts in English unless the browser remembers another choice. Changing
the UI language affects labels, not saved domain content. A new session takes
the selected language for the model's questions and reviewer choices; an existing
session retains its saved language when resumed. Omics's language change reloads
the page: confirm or cancel any unsaved-work warning. Reopening the saved session
selection shows documents; it does not automatically restart generation.
## Before creating a session
An administrator must have selected a workspace and configured the installation-wide provider,
model, and thinking settings. The New session form deliberately asks only for the question.
model, and thinking settings. The new-question form deliberately asks only for the question.
The workspace is a pinned Git revision. A subsequent workspace update cannot alter a session
already created from an earlier revision. If a workspace cannot reach its configured runtime DWH,
@@ -15,7 +35,9 @@ new sessions are refused before any session state is written.
## Create and review a session
1. Sign in and select **New session**.
1. Sign in and select **Session** (**Sessione** in Italian). If a session is already
open and unfinished, this returns to it without restarting it. Otherwise it
opens a new question; no session is created until you submit that question.
2. Enter a precise business question, including the relevant time period and desired output. For
example: “List patients discharged in the last 30 days, with ward and discharge date.”
3. Review each gate and make the decision requested by the widget. A choice with a decision payload
@@ -37,6 +59,19 @@ The workflow phases are fixed:
## Resume, archive, and the meaning of saved state
In Administration, the dot beside **Workspace** is green when readiness is
confirmed and red otherwise. Hover the button for the exact state; assistive
technology receives the same description. Select Workspace to inspect preparation.
The session sidebar has two accordion sections: **Active sessions** and
**Archive**, both initially closed. Only their headers appear below the scope tabs.
Inside each nonempty list, **Select all** selects only that list; its delete action
also applies only to the selected sessions in that list. The other list's selection
is preserved. Opening a section closes the other; clicking the open section closes
it too. Empty lists show only "No sessions yet." Long lists scroll inside
their own panels. Here active means not archived, not necessarily a running model
process. Existing groups and session actions remain inside those sections.
The sidebar lists sessions and their current lifecycle. Resuming returns to the last incomplete
phase. A finalized or archived session cannot be resumed.
@@ -54,4 +89,5 @@ happen from the workflow’s point of view.
- To author material the workflow can retrieve, use [Evidence](evidence.md). A proposal from a
session does not become Evidence automatically: a curator must review and publish it in Git.
- For login and access recovery, use [local authentication](install/authentication-local.md) or
[OIDC authentication](install/authentication-oidc.md).
[OIDC authentication](install/authentication-oidc.md) for full, or contact the
portal administrator for [embedded/upstream access](install/authentication-upstream.md).
+6
View File
@@ -8,6 +8,12 @@ Start with the path that matches the work you need to do:
| I need to… | Start here |
| --- | --- |
| Install or operate one instance | [Install and first start](install/first-start.md) |
| Clone and manually install on macOS, Windows, or Linux | [Italian procedure](install/standalone-manual-it.md) · [English procedure](install/standalone-manual-en.md) |
| Upgrade the server and integrate Omics Portal | [Codex server handoff](operations/server-codex-handoff.md) |
| Choose full/embedded and configure the shell | [Shell and localization](operations/shell-and-localization.md) |
| Reuse an authenticated server portal without a second login | [Upstream authentication](install/authentication-upstream.md) |
| Understand how the two renderings share the same application | [Rendering architecture](architecture/application-shell.md) |
| Validate login, logout and portal integration before release | [Acceptance matrix](testing/authentication-manual-acceptance.md) |
| Add, update, or prepare a workspace | [Workspace operations](operations/workspaces.md) |
| Ask a question and review the SQL workflow | [User guide](guida-utente.md) |
| Configure and refresh an authoritative database catalog | [Database management](operations/database-management.md) |
+11 -1
View File
@@ -1,6 +1,11 @@
# Local authentication
Use local mode for a standalone PC or Mac. Configure it through `tht`; passwords are entered at an
Use local mode for a standalone PC or Mac, with `shell.mode: full` and
`shell.defaultLocale: en` in the installation descriptor. Presentation and
authentication are independent: selecting full does not create accounts. Omics
embedded instead uses the [upstream guide](authentication-upstream.md), not local users.
Configure local authentication through `tht`; passwords are entered at an
echo-free prompt or read from a protected `--password-file`, never from a command argument.
## Bootstrap
@@ -56,6 +61,11 @@ machine use; JSON output is pristine on stdout.
## Session behavior and recovery
Full shows its own login form and, after login, the verified display name in its
header. The name menu contains Log out. This sends a CSRF-protected request to
`/api/auth/logout`, revokes the session and returns to login. Language/theme
preferences may remain in the browser; they are not credentials.
An ordinary login expires after 2 hours idle or 12 hours absolute. Selecting **Remember me** makes
the cookie persistent and changes the limits to 7 days idle or 30 days absolute. Remembered
sessions survive a browser and backend restart, but not a user revision change, configuration
+23
View File
@@ -1,10 +1,20 @@
# Generic OIDC authentication
Use this guide for **ThothII's own login**, normally `shell.mode: full` on an
autonomous server. It is not the integration procedure for an already logged-in
Omics user. That deployment uses [embedded/upstream](authentication-upstream.md),
even when Omics's identity provider is Authentik.
OIDC mode supports a standards-based provider. The browser flow is generic: Authorization Code,
PKCE S256, state, nonce, issuer/signature/audience/expiry validation, and the fixed callback
`<publicUrl>/api/auth/oidc/callback`. The browser and API must use the same origin; configure the
reverse proxy to preserve that public origin and callback path.
`publicUrl` is the public origin, without an application subpath. The current
full OIDC browser entry and callback use `/api/auth/oidc/login` and
`/api/auth/oidc/callback`; arbitrary prefixed OIDC hosting is not implemented by
selecting a different `backendBaseUrl`.
Configure the installation with `tht`:
```sh
@@ -18,6 +28,9 @@ The OIDC client secret is supplied through the protected secret bundle under the
`THT_OIDC_CLIENT_SECRET`; it is never written into `auth.yaml`. The default scopes are exactly
`openid`, `profile`, and `email`.
Keep `AUTH_MODE` unset when using this file. A simultaneously mounted local/OIDC
configuration and `AUTH_MODE=upstream` is an error, not a fallback chain.
The non-secret OIDC configuration has this exact shape (replace angle-bracket placeholders with
operator values):
@@ -92,3 +105,13 @@ relevant diagnostic surface.
The complete closed diagnostic-code union and exact role-to-permission expansion are in the
[authentication architecture](../architecture/authentication.md).
## Browser login and logout
ThothII redirects the browser to the provider and creates its own opaque session
after validating the callback. An existing provider SSO session may avoid another
password prompt, but this remains a distinct ThothII login/session, unlike Omics
upstream. Full's name menu logs out of ThothII only. It does not revoke the
provider session or log out other applications, so a subsequent login can return
immediately through SSO. No provider token is placed in the UI adapter or browser
storage. See the [manual acceptance matrix](../testing/authentication-manual-acceptance.md).
+186
View File
@@ -0,0 +1,186 @@
# Autenticazione tramite portale e proxy fidato
Questa è la modalità **upstream** usata dall'integrazione Omics. Non è il login
OIDC diretto di ThothII: l'utente accede a Omics come già fa, poi sceglie
Datamart Builder e trova ThothII già autenticato. Non deve essere creato un utente
locale ThothII né effettuato un secondo scambio OIDC dall'applicazione embedded.
## Il confine di fiducia
```mermaid
sequenceDiagram
actor U as Utente già autenticato
participant N as Nginx Omics
participant D as Django Omics
participant T as Core ThothII upstream
U->>D: Apri Datamart Builder
D-->>U: Pagina autorizzata con mount React
U->>N: GET /datamart-builder/api/me (cookie Omics)
N->>D: Subrequest interna /datamart-builder/api-auth
D-->>N: 200 + identità verificata, oppure 403
N->>T: GET /me + intestazioni normalizzate (solo se autorizzato)
T-->>U: Identità e permessi applicativi, oppure rifiuto
```
L'header e l'adapter JavaScript non autenticano nessuno. Il backend accetta una
richiesta upstream solo con un'identità valida ricevuta da un percorso di rete
fidato. Gli header non sono firmati da ThothII: la protezione è il proxy che
verifica la sessione e sovrascrive l'identità, insieme all'isolamento del core.
Un core upstream direttamente raggiungibile da client non fidati è una falla,
non una modalità alternativa di accesso.
## Configurazione del core
Per Omics il descrittore pubblico deve contenere:
```yaml
shell:
mode: embedded
defaultLocale: en
adapter: omics-portal
```
Separatamente, il **processo core** deve ricevere `AUTH_MODE=upstream`. Definirlo
nell'override Compose approvato e incluso nell'installazione; una variabile nel
file di interpolazione `.env` non viene passata automaticamente al container:
```yaml
services:
core:
environment:
AUTH_MODE: upstream
```
È solo il frammento di selezione auth, non un file Compose completo né una
configurazione di rete sufficiente. Non aggiunge porte pubbliche.
Condizioni obbligatorie:
1. Nessun `auth.yaml` local/OIDC deve essere effettivamente montato al percorso
letto dal core (default `/run/thothii-auth/auth.yaml`). Se è presente insieme
ad `AUTH_MODE`, l'avvio fallisce. Non impostare `AUTH_MODE=local` o `oidc`:
questi due modi si selezionano dal file, non da quella variabile.
2. Non configurare `authentication.runtimeProjection` per questo percorso: è la
proiezione delle configurazioni cookie local/OIDC, non l'identità Omics.
Nemmeno `THT_AUTH_RUNTIME_PROJECTION_ROOT` deve attivarla nel core.
3. Il descrittore e Compose base continuano a richiedere `authentication.configDirectory`
e `THT_AUTH_CONFIG_ROOT` coerenti. Per una nuova installazione upstream usare
una directory dedicata senza `auth.yaml`, non cancellare la configurazione di
un'installazione esistente. I cambi di modalità richiedono un piano separato.
4. Non esiste `tht auth configure --mode upstream`: il CLI configura gli utenti
locali o l'OIDC diretto. Conservare il percorso proxy già operativo per Omics.
5. `profile: server`, storage delle sessioni e `THOTH_PUBLIC_EXPOSURE` hanno propri
vincoli, che rimangono attivi. La shell embedded non li soddisfa automaticamente.
Il sorgente considera upstream un percorso di compatibilità con il proxy; è
quello usato dall'integrazione Omics corrente. `none` e `mock` sono per sviluppo/test,
non soluzioni a errori di configurazione in produzione.
## Intestazioni richieste all'ingresso del core
| Header | Regola ThothII | Valore Omics |
| --- | --- | --- |
| `X-Thoth-Principal-Issuer` | Stringa stabile, obbligatoria | `portal` |
| `X-Thoth-Principal-Subject` | ID stabile dell'utente, obbligatorio | `str(user.pk)` Django |
| `X-Thoth-Principal-Display-Name` | Facoltativo, se presente non vuoto | Nome completo o username |
| `X-Thoth-Is-Admin` | Obbligatorio: `0`, `1`, `false` o `true` | Risultato di `is_authentik_admin(user)` |
Le stringhe sono ripulite degli spazi esterni, devono avere al massimo 512
caratteri e non contenere caratteri di controllo. Header mancanti o invalidi
producono 401. `false`/`0` assegna il ruolo `user`; `true`/`1` assegna `user` e
`admin`. Il core espande i permessi dal proprio catalogo, non da un array inviato
dal browser. `/me` richiede `session.use`.
La coppia `(issuer, subject)` identifica il proprietario delle sessioni.
Non sostituire il subject con un nome visualizzato o un'email modificabile; non
cambiare issuer/subject di utenti esistenti per correggere un problema grafico.
Passare da identità `portal` a identità OIDC diretta non migra la proprietà dei dati.
## Omics: percorsi e componenti esatti
| Percorso | Destinazione e funzione |
| --- | --- |
| `/kokoro/datamart-builder/` | Pagina Django con `datamart_builder.access` |
| `/datamart-builder/config.js` | Config pubblico del frontend, senza cache |
| `/datamart-builder/assets/…` | Asset frontend risolti dal manifest Vite |
| `/datamart-builder/api/…` | Nginx con `auth_request`, poi core senza il prefisso |
| `/_thothii_auth` | Location Nginx interna, non un login pubblico |
| `/datamart-builder/api-auth` | Django verifica sessione Omics e capability |
Nel repository Omics:
- `kokoro/datamart_catalog_views.py`: `DatamartBuilderView` e
`datamart_builder_api_auth`; la verifica API risponde 200 o 403, anche 403
quando la sessione è assente/scaduta. Non trasforma l'API in una pagina di login.
- `nginx/nginx.conf`: API direttamente a `thothii-core:8787`, config e asset a
`thothii-frontend:8080`; verificare alias e reti Docker effettivi sul server.
- `templates/kokoro/datamart_builder.html`: mount, config e override del prefisso.
- `kokoro/templatetags/vite.py`: manifest da
`http://thothii-frontend:8080/.vite/manifest.json`, cache Django di 30 secondi.
Nginx usa il cookie Omics nella subrequest a Django. Sulle richieste al core
sovrascrive i quattro header con i risultati della verifica e rimuove
`Cookie`, `Authorization` e `X-Authenticated-User`. Nessuna password o token del
portale deve essere copiato nel config pubblico, nello snapshot adapter o in Web Storage.
`GET /datamart-builder/api/me` restituisce l'identità e i permessi; in upstream
`session` e `csrfToken` sono `null`: non viene creata una sessione-cookie ThothII.
## Non confondere i due percorsi proxy
L'esempio generico `deploy/nginx-authenticated-proxy.conf.example` usa **due hop**:
proxy host → frontend Nginx ThothII → core. Sul tratto privato verso il frontend
trasporta `X-Thoth-Trusted-Principal-*` e `X-Thoth-Trusted-Is-Admin`; il frontend
li converte nei quattro header del core e li elimina prima dell'inoltro.
Omics usa invece **Nginx Omics → core direttamente** per le API e invia gli header
normalizzati senza `Trusted`. Non incollare l'esempio a due hop in questa location:
la famiglia di header sbagliata produce 401. In entrambi i casi i valori devono
venire dalla verifica server, mai dagli header del client. Il tratto privato del
percorso generico deve essere inaccessibile ai client non fidati.
## Origine delle richieste e stream
Browser e API devono restare sullo stesso origin. Il frontend accetta `/api` o un
prefisso same-origin come `/datamart-builder/api`, non un URL `http://core:8787`.
In upstream le scritture con `Origin` sono confrontate con protocollo e Host
percepiti dal core; non usano il token CSRF della sessione ThothII local/OIDC.
Le richieste senza Origin hanno il trattamento non-browser: l'autenticazione del
proxy rimane indispensabile anche per esse.
Nel Nginx Omics esaminato il TLS termina a monte e una mappa **esatta** converte
`https://aritmolab.policlinicosandonato.it` in
`http://aritmolab.policlinicosandonato.it` per il confronto interno. Le altre origini
rimangono invariate e devono essere negate quando non coincidono. È una scelta
specifica della topologia corrente, non un modello da estendere con wildcard,
cancellazione di Origin o riscrittura incondizionata. Verificare Host/protocollo
al core e i dinieghi cross-origin nella topologia realmente rilasciata.
La location API disabilita buffering/cache per SSE e mantiene timeout lunghi.
`auth_request` verifica ogni nuova richiesta, ma non interrompe istantaneamente
uno stream già aperto quando il portale revoca l'utente. ThothII ricontrolla `/me`
al ritorno alla pagina e alla riconnessione degli eventi; non promettere revoca
istantanea fra tutte le schede.
## Logout, rientro e diagnosi
In embedded logout e successivo login sono di Omics. ThothII non chiama
`/auth/logout`, non cancella il cookie Django e non apre un suo login.
Il rifiuto 401/403 di `/me` rimuove lo stato protetto e richiede il rientro dal
portale. Un 403 su una singola operazione non equivale al logout dell'applicazione.
| Sintomo | Controllo mirato |
| --- | --- |
| Secondo header | Config servito: deve essere embedded, non full |
| Nessuna UI e errore preferenze | Selettore Omics `data-lang` e `html data-bs-theme` |
| `/me` 401 dal core | Header obbligatori, famiglia Trusted/normalizzata, percorso proxy |
| `/me` 403 dal proxy | Sessione Omics e capability `datamart_builder.access` |
| `/me` funziona ma POST 403 | Distinguere permesso operativo da mismatch Origin/Host/protocollo |
| 502 o asset assenti | Alias/rete Docker e manifest Vite; attesa cache manifest 30 s |
| Avvio core rifiutato | Coesistenza di `auth.yaml` o runtime projection con `AUTH_MODE` |
| Logout full seguito da rientro IdP immediato | Il logout ThothII non è logout globale OIDC |
Non raccogliere cookie, token, segreti o dump completi delle configurazioni nei
report. Registrare codici HTTP, nomi dei percorsi, revisioni e risultati dei test.
Consegna e rilascio: [procedura Omics](../operations/shell-and-localization.md#verifica-prima-del-deploy-server).
Collaudo obbligatorio: [matrice di accettazione](../testing/authentication-manual-acceptance.md).
+9 -2
View File
@@ -1,7 +1,14 @@
# Authentik provider configuration
ThothII uses generic OIDC in the browser. Authentik provides the identity provider and group
catalog without adding a proprietary login flow.
For **full with direct OIDC**, ThothII uses generic OIDC in the browser. Authentik
provides the identity provider and group catalog without adding a proprietary flow.
The provider/client/group setup below applies to that case only.
For **embedded in Omics**, retain Omics's existing Authentik authentication and
configure ThothII as upstream. Omics verifies `datamart_builder.access` and
administrator status and the proxy supplies the identity; no additional ThothII
OIDC client, login or local user is required for that path. Follow the
[portal integration guide](authentication-upstream.md).
```mermaid
sequenceDiagram
@@ -3,6 +3,9 @@
# Replace every absolute placeholder. Select exactly one Git transport override.
schemaVersion: 2
profile: local
shell:
mode: full
defaultLocale: en
projectDirectory: "/absolute/path/to/ThothII"
envFile: "/absolute/path/to/ThothII/deploy/env/local.env"
workspaceRepository:
@@ -3,6 +3,11 @@
# Replace every absolute placeholder. Select exactly one Git transport override.
schemaVersion: 2
profile: server
# Standalone server with protected direct OIDC auth, not the Omics upstream path.
# For Omics use authentication-upstream.md: embedded, no auth runtime projection.
shell:
mode: full
defaultLocale: en
projectDirectory: "/absolute/path/to/ThothII"
envFile: "/absolute/path/to/thothii-server-operator/server.env"
workspaceRepository:
+17 -1
View File
@@ -1,5 +1,9 @@
# Install and first start
For the ordered clone-to-start procedure on Mac, Windows WSL2 and Linux, use the
[Italian manual guide](standalone-manual-it.md) or [English manual guide](standalone-manual-en.md),
including protected credentials and the explicit initial catalog migration.
This is the supported local installation path. It creates an installation-local configuration and
starts the Compose stack; it does not create a workspace repository or a database catalog entry.
@@ -24,7 +28,7 @@ version control, a URL, or a command line.
From the repository root, start the interactive setup and select the local profile:
```sh
tht setup --profile local
tht setup --profile local --shell-mode full --shell-default-locale en
```
It writes the selected non-secret descriptor below `deploy/<installation-id>/`, the associated
@@ -32,6 +36,18 @@ operator env file, and can create protected secret templates. Keep the descripto
to commands as `--installation /absolute/path/thothii-installation.yaml` when more than one
installation can be discovered.
The explicit shell options are important: compatibility defaults without them
are embedded/en/omics-portal, which expects an Omics document. The Mac's standalone
installation must use full, with English as its initial locale. Existing browser
language preferences take precedence over that initial value. Full does not
configure authentication; setup separately bootstraps local login.
For a server portal, use the [embedded/upstream procedure](authentication-upstream.md)
instead of creating a second ThothII login. For a standalone server, use full
with [direct OIDC](authentication-oidc.md). Shell mode does not follow `profile`
automatically. See [configuration and regeneration](../operations/shell-and-localization.md)
before modifying an existing installation.
If the descriptor is prepared manually instead, begin with
[`thothii-installation.local.yaml`](examples/thothii-installation.local.yaml), set mode `0600` or
`0400`, and ensure `THT_INSTALLATION_CONFIG_SOURCE` in the selected env file points to that exact
+324
View File
@@ -0,0 +1,324 @@
# Manual standalone installation
[Versione italiana](standalone-manual-it.md)
This is the verification procedure for preparing THothII as a standalone application in `full`
mode on macOS, Windows, and Linux.
In this document, “standalone” means that the user does not need to install Node.js, Python or Pi
on the host: the application services and local semantic
services run through Docker. DWH and LLM providers remain external endpoints configured by the
installation; this is not an offline package.
This path uses no graphical installer, native launcher, or Docker Hub image. It starts from a Gitea
clone and uses explicit terminal commands. Publishing pre-built images is a later step.
## Verification matrix
| System | Recommended terminal | Runtime | Test architecture |
| --- | --- | --- | --- |
| macOS supported by the installed Docker Desktop version | Bash in Terminal | Docker Desktop | Apple Silicon (`arm64`) |
| Windows 11 | Ubuntu inside WSL2 | Docker Desktop with WSL2 integration | x64 (`amd64`) |
| Ubuntu Linux 22.04 or 24.04 | Bash | Docker Engine + Compose v2 | x64 (`amd64`) |
Intel macOS is excluded from the first verification campaign. ARM Linux may be tested when the
machine’s Docker runtime reports `arm64`, but it is not part of the minimum matrix.
Documentation and Mac prerequisite checks have passed. Complete fresh installations on all three
systems remain pending; this matrix describes the tests to perform, not completed certification.
## Before you start
You need:
- access to the THothII Gitea repository and the workspace Git repository;
- Git;
- Docker Desktop on macOS and Windows, or Docker Engine with the Compose v2 plugin on Linux;
- Bash, `curl`, OpenSSL and `shasum` (Ubuntu package: `libdigest-sha-perl`);
- enough disk space to build the images and download the embedding model;
- the DWH and LLM endpoints, plus the credentials required by the installation.
On Linux, the current user must be able to run Docker. If the system requires `sudo`, add the user
to the Docker group according to local policy and open a new session before continuing.
On Windows, run every command in this guide from Ubuntu under WSL2. In Docker Desktop, enable WSL2
integration for that distribution. Clone the project inside the WSL2 Linux filesystem, for example
under `~/src`, rather than under `/mnt/c`: this avoids slow builds and path/line-ending issues. Pi
does not need to be installed on the host.
Check the runtime before or immediately after cloning:
```sh
docker version
docker compose version
docker version --format '{{.Server.Arch}}'
```
The last command must return `amd64`, `x86_64`, `arm64`, or `aarch64`.
## 1. Clone a project revision
Use the project repository on Gitea:
```sh
mkdir -p "$HOME/src"
cd "$HOME/src"
git clone https://git.tylconsulting.it/mptyl/ThothII.git
cd ThothII
git rev-parse --short HEAD
```
For an SSH clone, when the key is already authorized on Gitea:
```sh
git clone git@git.tylconsulting.it:mptyl/ThothII.git
```
Record the hash printed by `git rev-parse` for a repeatable test. In a later campaign, use the
maintainer-approved revision/tag rather than implicitly following a mutable `main` branch.
## 2. Check prerequisites and install the operator command
From the clone root:
```sh
bash scripts/check-standalone-prerequisites.sh
export PATH="$HOME/.local/bin:$PATH"
THT_INSTALL_DIRECTORY="$HOME/.local/bin" bash scripts/install-tht.sh
tht version
```
`install-tht.sh` bootstraps only the native `tht` operator command; it does not install a desktop
version of THothII. It uses the repository’s Docker builder, installs the binary for the current
terminal environment, and installs it in the user directory. Persist `$HOME/.local/bin` in your
shell PATH for new terminals too. An existing `tht` in this directory will be updated.
On Windows, run these commands inside WSL2. The installed `tht` binary is the Linux binary inside
WSL2; the application runtime remains Docker Desktop. Do not use `scripts/install-tht.ps1` as the
primary path for this test.
## 3. Configure and start the local installation
Run the remaining blocks in one Bash session from the physical clone root (`pwd -P`).
First create two distinct catalog passwords, preserving any existing files:
```bash
umask 077
mkdir -p deploy/local/secrets
for name in catalog-runtime-password catalog-migrator-password; do
target="deploy/local/secrets/$name"
if [ ! -e "$target" ]; then
(set -C; openssl rand -hex 32 > "$target") || exit 1
fi
done
export THT_CATALOG_RUNTIME_PASSWORD_SOURCE="$(pwd -P)/deploy/local/secrets/catalog-runtime-password"
export THT_CATALOG_MIGRATOR_PASSWORD_SOURCE="$(pwd -P)/deploy/local/secrets/catalog-migrator-password"
```
Do not regenerate passwords for an initialized catalog. Configure without starting services:
```sh
tht setup --profile local --shell-mode full --shell-default-locale en --configure-only
```
Answer the prompts as follows:
| Prompt | Value or rule |
| --- | --- |
| Installation ID | `local`, unless one clone hosts multiple installations |
| Deployment profile | `local` |
| DWH API endpoint | An `http(s)` URL without user, password, query, or fragment; may be empty for a smoke-only test |
| LLM API endpoint | An `http(s)` URL without credentials; may be empty for a smoke-only test |
| Workspace repository URL | The workspace repository URL, not the THothII source clone |
| Workspace branch | Normally `main` |
| Workspace access | `ssh` with a deploy key, or `https` with a protected credential file |
| File paths | Accept the default paths under `deploy/local/secrets/` for the first test |
| Secret templates | Answer `yes` when protected files do not exist yet |
| Authentication | Configure the local login required by the installation; never put passwords on a command line |
The generated configuration is local and ignored by Git:
```text
deploy/local/thothii-installation.yaml
deploy/local/operator.env
deploy/local/auth/
deploy/local/secrets/
```
Edit secrets only in protected local files; never commit them. `deploy/env/local.env.example` is a tracked reference; the
generated path `deploy/local/operator.env` is the active path for this installation.
### Complete protected files
If setup created blank templates, enter the values with a local editor:
```sh
chmod 600 deploy/local/secrets/*
"${EDITOR:-vi}" deploy/local/secrets/thothii.secrets
```
The bundle must contain only `KEY=VALUE` lines for credentials actually used by `modelCatalog`. The
allowed names and credential boundary are documented in the local file
`deploy/secrets/README.md`. Do not put tokens in URLs, the YAML
descriptor, the Git repository, or commands copied into the shell.
For SSH workspace access, also provide the private key and `known_hosts` file requested by setup.
For HTTPS access, provide the Git credential file and any required CA. Both must remain protected
and outside version control.
Before starting, complete these additional configuration steps:
1. Add `THT_CATALOG_RUNTIME_PASSWORD_SOURCE` and `THT_CATALOG_MIGRATOR_PASSWORD_SOURCE` to
`deploy/local/operator.env`, with the same absolute paths exported above. Setup does not persist
these two variables. Store paths, not passwords.
2. Replace the descriptor's generic `modelCatalog` with the approved provider/model configuration.
The generated defaults do not replicate the existing Mac. See [Pi/model configuration](../general/pi-configuration.md)
and the local example `deploy/psd/thothii-installation.yaml.example`.
3. Populate the keys referenced by `authentication.apiKeyEnv` in `thothii.secrets`. Providers using
`pi_auth` need valid credentials at `PI_AUTH_FILE`; the `{}` template is not authentication.
4. Complete the workspace Git files. SSH requires an authorized deploy key and verified known-hosts;
HTTPS requires the credential file and CA bundle expected by the overlay. Blank templates cannot
provide repository access.
After editing generated configuration, do not rerun setup: it rejects different existing content.
Generate the projections and run the explicit migration below. Use `THT_GIT_ACCESS=https` if that
was selected during setup. This block targets the fresh `local` descriptor with only the Git overlay;
custom installations must include their extra descriptor overlays in the same order.
```bash
INSTALLATION="$(pwd -P)/deploy/local/thothii-installation.yaml"
tht --installation "$INSTALLATION" installation generate
THT_PROJECT="thothii-$(printf '%s' "$INSTALLATION" | shasum -a 256 | cut -c 1-12)"
THT_GIT_ACCESS=ssh
compose=(
docker compose --project-name "$THT_PROJECT" --project-directory "$(pwd -P)"
--env-file "$(pwd -P)/deploy/local/operator.env"
-f compose.yaml -f deploy/compose.local.yaml
-f "deploy/compose.git-$THT_GIT_ACCESS.yaml"
-f deploy/local/generated/compose.models.yaml
)
"${compose[@]}" config --quiet
"${compose[@]}" build core frontend
"${compose[@]}" up -d catalog-db
"${compose[@]}" run --rm catalog-migrate
tht --installation "$INSTALLATION" start
```
Stop if a command fails. The project name matches the hash used by `tht`, preserving volume
identity. `catalog-migrate` applies Catalog and Memory migrations; `tht start` does not run it
automatically. Initial embedding-model download may take time. Use this installation-specific
sequence, not `run-stack.sh` with a different environment/project name.
## 4. Verify the installation
The descriptor generated for the default ID is:
```sh
INSTALLATION="$(pwd -P)/deploy/local/thothii-installation.yaml"
test -f "$INSTALLATION"
bash scripts/verify-standalone-install.sh "$INSTALLATION"
```
The verifier is read-only: it runs `tht doctor --json` and `tht status` without restarting the
stack, regenerating configuration, or printing secret contents.
### Gate A — platform smoke test on all three computers
Record the following for each machine:
```sh
uname -a
docker version --format '{{.Server.Version}} {{.Server.Arch}}'
tht version
bash scripts/check-standalone-prerequisites.sh
bash scripts/verify-standalone-install.sh "$INSTALLATION"
```
The gate passes when the clone is intact, Docker and Compose are reachable, `tht doctor` is OK, the
stack is running, and the frontend responds at the default local URL `http://127.0.0.1:8080`.
Doctor also checks workspace and Pi: record their failures separately rather than labeling every
failure as a platform problem. Check HTTP readiness with:
```sh
curl --fail --silent --show-error http://127.0.0.1:8080/health
```
### Gate B — functional verification
Run this on at least one machine with available endpoints and credentials:
First follow [Workspace operations](../operations/workspaces.md) to import/prepare the workspace
and configure the Database and local binding. The source clone does not transfer catalog data,
secrets or sessions from the Mac. Connect any VPN required by DWH/model endpoints and verify
that their names are reachable from containers too.
1. open `http://127.0.0.1:8080`;
2. sign in with the configured local account;
3. verify that the configured workspace is readable;
4. start a real question and complete the review gates through final SQL;
5. stop and restart the installation, then run `verify-standalone-install.sh` again.
A Gate B failure involving DWH, the LLM provider, workspace Git, or credentials does not by itself
prove a Docker portability problem: record the failed endpoint or component separately.
## Daily lifecycle
Use the explicit descriptor when more than one installation may be discoverable:
```sh
INSTALLATION="$PWD/deploy/local/thothii-installation.yaml"
tht --installation "$INSTALLATION" status
tht --installation "$INSTALLATION" start
tht --installation "$INSTALLATION" start --build
tht --installation "$INSTALLATION" logs
tht --installation "$INSTALLATION" doctor --json
tht --installation "$INSTALLATION" stop
```
Use `start --build` after source changes or to rebuild images from the current checkout. `stop`
preserves volumes, sessions, the catalog, Pi state, Qdrant data, and the embedding model. Do not
use `docker compose down --volumes` during a normal test: it is destructive and removes local data.
For upgrades requiring migrations, follow the release runbook before starting the new application.
## Quick diagnosis
| Symptom | Check |
| --- | --- |
| `Docker Engine is not reachable` | start Docker Desktop or the Docker service and rerun `docker info` |
| Windows sees Docker but Bash fails | run the guide inside Ubuntu WSL2 and enable that distribution in Docker Desktop |
| `tht: command not found` | open a new shell and check `command -v tht`; rerun the bootstrap if needed |
| line-ending or executable-script errors | use a clone in the WSL2/Linux filesystem and rerun `bash scripts/...` |
| unsupported architecture | check `docker version --format '{{.Server.Arch}}'`; the test requires `amd64` or `arm64` |
| missing descriptor or env file | use `deploy/local/...` generated by `tht setup`, not an arbitrary copied file |
| healthy stack but workflow failure | check external URLs, the credential bundle, workspace Git, and authentication separately |
| data appears missing | check that `down --volumes` was not used; `stop` does not remove volumes |
## Acceptance checklist
- [ ] The clone comes from the expected Gitea repository and the revision is recorded.
- [ ] Docker Desktop/Engine and Compose v2 are available.
- [ ] The runtime reports an allowed architecture.
- [ ] `tht` was built from the repository and responds to `tht version`.
- [ ] Setup uses `profile: local`, `shell.mode: full`, and `shell.defaultLocale: en`.
- [ ] The descriptor, `operator.env`, authentication, and secrets exist only under `deploy/local/`.
- [ ] No secret appears in Git, URLs, public YAML, or recorded commands.
- [ ] Gate A passes on Apple Silicon macOS, x64 Windows WSL2, and x64 Linux.
- [ ] Gate B runs on at least one machine with DWH and LLM available.
- [ ] Stop/start and final verification complete without deleting volumes.
## Out of scope for this release
The following remain future work:
- publishing pre-built images on Docker Hub;
- reducing prompts through a dedicated non-interactive configuration;
- creating DMG, MSI/EXE, AppImage, or other native installers;
- providing an offline runtime or bundling a local DWH/LLM into the application.
## Related documents
- [Install and first start](first-start.md)
- [Shell and localization](../operations/shell-and-localization.md)
- [Workspace operations](../operations/workspaces.md)
- `deploy/secrets/README.md` (runtime secrets)
+329
View File
@@ -0,0 +1,329 @@
# Installazione manuale standalone
[English version](standalone-manual-en.md)
Questa è la procedura di prova per predisporre THothII come applicazione autonoma in modalità
`full` su macOS, Windows e Linux.
In questo documento “autonoma” significa che l’utente non deve installare Node.js, Python o Pi
sull'host: i servizi applicativi e i servizi semantici
locali vengono eseguiti con Docker. DWH e provider LLM restano endpoint esterni configurati
dall’installazione; questa procedura non è un pacchetto offline.
Il percorso non usa installer grafici, launcher nativi o immagini Docker Hub. Si parte da un clone
Gitea e si usano comandi espliciti da terminale. La pubblicazione di immagini pre-costruite è una
fase successiva.
## Matrice di verifica
| Sistema | Terminale raccomandato | Runtime | Architettura della prova |
| --- | --- | --- | --- |
| macOS supportato dalla versione Docker Desktop installata | Bash nel Terminale | Docker Desktop | Apple Silicon (`arm64`) |
| Windows 11 | Ubuntu dentro WSL2 | Docker Desktop con integrazione WSL2 | x64 (`amd64`) |
| Linux Ubuntu 22.04 o 24.04 | Bash | Docker Engine + Compose v2 | x64 (`amd64`) |
Intel macOS non fa parte della prima campagna di verifica. ARM Linux può essere provato quando il
runtime Docker della macchina restituisce `arm64`, ma non è un requisito della matrice minima.
Le verifiche documentali e dei prerequisiti Mac sono passate. Le installazioni complete da zero
sui tre sistemi restano da eseguire: la matrice indica le prove previste, non una certificazione.
## Cosa serve prima di iniziare
Servono:
- accesso al repository Gitea di THothII e al repository Git dei workspace;
- Git;
- Docker Desktop su macOS e Windows, oppure Docker Engine con il plugin Compose v2 su Linux;
- Bash, `curl`, OpenSSL e `shasum` (su Ubuntu, pacchetto `libdigest-sha-perl`);
- spazio disco sufficiente per compilare le immagini e scaricare il modello di embedding;
- gli endpoint DWH e LLM, più le credenziali che l’installazione deve usare.
Su Linux l’utente corrente deve poter eseguire Docker. Se il sistema richiede `sudo`, aggiungere
l’utente al gruppo Docker secondo la policy locale e aprire una nuova sessione prima di continuare.
Su Windows usare Ubuntu in WSL2 per tutti i comandi di questa guida. In Docker Desktop attivare
l’integrazione WSL2 per quella distribuzione. Clonare il progetto nel filesystem Linux di WSL2,
per esempio sotto `~/src`, e non sotto `/mnt/c`: si evitano rallentamenti e problemi di permessi o
line ending. Non è necessario installare Pi sull’host.
Verificare il runtime prima del clone o subito dopo:
```sh
docker version
docker compose version
docker version --format '{{.Server.Arch}}'
```
L’ultima istruzione deve restituire `amd64`, `x86_64`, `arm64` o `aarch64`.
## 1. Clonare una revisione del progetto
Usare il repository di progetto su Gitea:
```sh
mkdir -p "$HOME/src"
cd "$HOME/src"
git clone https://git.tylconsulting.it/mptyl/ThothII.git
cd ThothII
git rev-parse --short HEAD
```
Per un clone SSH usare, se la chiave è già autorizzata su Gitea:
```sh
git clone git@git.tylconsulting.it:mptyl/ThothII.git
```
Per una prova ripetibile annotare l’hash stampato da `git rev-parse`. In una campagna successiva
usare la revisione/tag approvata dal maintainer invece di seguire implicitamente una `main` che
può cambiare.
## 2. Verificare i prerequisiti e installare il comando operatore
Dal root del clone:
```sh
bash scripts/check-standalone-prerequisites.sh
export PATH="$HOME/.local/bin:$PATH"
THT_INSTALL_DIRECTORY="$HOME/.local/bin" bash scripts/install-tht.sh
tht version
```
`install-tht.sh` è un bootstrap del solo comando operatore nativo `tht`; non installa una versione
desktop di THothII. Usa il builder Docker del repository, installa il binario adatto all’ambiente
del terminale e lo installa nella directory utente. Aggiungere `$HOME/.local/bin` al PATH della
shell anche per i terminali successivi. Un `tht` già presente in quella directory viene aggiornato.
Su Windows, eseguire questi comandi dentro WSL2. Il binario `tht` installato è quello Linux di WSL2;
il runtime dell’applicazione rimane Docker Desktop. Non usare `scripts/install-tht.ps1` come percorso
principale di questa prova.
## 3. Configurare e avviare l’installazione locale
Eseguire i blocchi successivi nella stessa sessione Bash dal root fisico del clone (`pwd -P`).
Creare prima le due password distinte del catalogo, conservando eventuali file già esistenti:
```bash
umask 077
mkdir -p deploy/local/secrets
for name in catalog-runtime-password catalog-migrator-password; do
target="deploy/local/secrets/$name"
if [ ! -e "$target" ]; then
(set -C; openssl rand -hex 32 > "$target") || exit 1
fi
done
export THT_CATALOG_RUNTIME_PASSWORD_SOURCE="$(pwd -P)/deploy/local/secrets/catalog-runtime-password"
export THT_CATALOG_MIGRATOR_PASSWORD_SOURCE="$(pwd -P)/deploy/local/secrets/catalog-migrator-password"
```
Non rigenerare le password di un catalogo già inizializzato. Configurare senza avviare i servizi:
```sh
tht setup --profile local --shell-mode full --shell-default-locale en --configure-only
```
Rispondere ai prompt nel seguente modo:
| Prompt | Valore o regola |
| --- | --- |
| Installation ID | `local`, salvo necessità di più installazioni nello stesso clone |
| Deployment profile | `local` |
| DWH API endpoint | URL `http(s)` senza user, password, query o fragment; può restare vuoto per il solo smoke test |
| LLM API endpoint | URL `http(s)` senza credenziali; può restare vuoto per il solo smoke test |
| Workspace repository URL | URL del repository dei workspace, non il clone sorgente di THothII |
| Workspace branch | normalmente `main` |
| Workspace access | `ssh` se si usa una chiave deploy; altrimenti `https` con credential file protetto |
| Percorsi dei file | accettare i percorsi predefiniti sotto `deploy/local/secrets/` nella prima prova |
| Secret templates | rispondere `yes` quando i file protetti non esistono ancora |
| Autenticazione | configurare il login locale richiesto dall’installazione; non inserire password in una riga di comando |
La configurazione generata è locale e ignorata da Git:
```text
deploy/local/thothii-installation.yaml
deploy/local/operator.env
deploy/local/auth/
deploy/local/secrets/
```
Modificare i segreti solo nei file locali protetti; non committarli. Il file `deploy/env/local.env.example` è un riferimento
tracciato; il percorso generato da `tht setup`, `deploy/local/operator.env`, è quello da usare per
questa installazione.
### Completare i file protetti
Se il setup ha creato template vuoti, inserire i valori con un editor locale:
```sh
chmod 600 deploy/local/secrets/*
"${EDITOR:-vi}" deploy/local/secrets/thothii.secrets
```
Il bundle deve contenere solo righe `KEY=VALUE` per le credenziali effettivamente usate dal
`modelCatalog`. I nomi ammessi e il confine delle credenziali sono descritti nel file locale
`deploy/secrets/README.md`. Non mettere token nelle URL, nel
descriptor YAML, nel repository Git o nei comandi copiati nella shell.
Per accesso workspace SSH, predisporre anche la chiave privata e il file `known_hosts` indicati dal
setup. Per accesso HTTPS, predisporre il credential file Git e l’eventuale CA. Entrambi devono
restare protetti e fuori dal controllo versione.
Prima dell'avvio completare anche questi passaggi:
1. Aggiungere `THT_CATALOG_RUNTIME_PASSWORD_SOURCE` e `THT_CATALOG_MIGRATOR_PASSWORD_SOURCE` a
`deploy/local/operator.env`, con gli stessi percorsi assoluti esportati sopra. Il setup non salva
queste due variabili. Inserire i percorsi, non le password.
2. Sostituire il `modelCatalog` generico nel descriptor con la configurazione provider/modelli
approvata. I default generati non replicano il Mac esistente. Vedere [configurazione Pi/modelli](../general/pi-configuration.md)
e l'esempio locale `deploy/psd/thothii-installation.yaml.example`.
3. Inserire in `thothii.secrets` le chiavi referenziate da `authentication.apiKeyEnv`. I provider
`pi_auth` richiedono credenziali valide nel file `PI_AUTH_FILE`; il template `{}` non autentica.
4. Completare i file Git del workspace. SSH richiede una chiave deploy autorizzata e known-hosts
verificato; HTTPS richiede il credential file e il bundle CA previsti dall'overlay. I template
vuoti non consentono l'accesso al repository.
Dopo aver modificato la configurazione generata, non rilanciare setup: rifiuta file esistenti con
contenuti diversi. Generare le proiezioni ed eseguire la migrazione esplicita indicata sotto.
Impostare `THT_GIT_ACCESS=https` se scelto nel setup. Il blocco usa il nuovo descriptor `local`
con il solo overlay Git; per installazioni personalizzate includere gli overlay aggiuntivi
nello stesso ordine del descriptor.
```bash
INSTALLATION="$(pwd -P)/deploy/local/thothii-installation.yaml"
tht --installation "$INSTALLATION" installation generate
THT_PROJECT="thothii-$(printf '%s' "$INSTALLATION" | shasum -a 256 | cut -c 1-12)"
THT_GIT_ACCESS=ssh
compose=(
docker compose --project-name "$THT_PROJECT" --project-directory "$(pwd -P)"
--env-file "$(pwd -P)/deploy/local/operator.env"
-f compose.yaml -f deploy/compose.local.yaml
-f "deploy/compose.git-$THT_GIT_ACCESS.yaml"
-f deploy/local/generated/compose.models.yaml
)
"${compose[@]}" config --quiet
"${compose[@]}" build core frontend
"${compose[@]}" up -d catalog-db
"${compose[@]}" run --rm catalog-migrate
tht --installation "$INSTALLATION" start
```
Fermarsi se un comando fallisce. Il nome progetto coincide con l'hash usato da `tht`, preservando
l'identità dei volumi. `catalog-migrate` applica le migrazioni Catalog e Memory; `tht start` non
lo esegue automaticamente. Il primo download del modello embedding può richiedere tempo.
Usare questa sequenza legata all'installazione, non `run-stack.sh` con env/progetto diversi.
## 4. Verificare l’installazione
Il descriptor generato per l’ID predefinito è:
```sh
INSTALLATION="$(pwd -P)/deploy/local/thothii-installation.yaml"
test -f "$INSTALLATION"
bash scripts/verify-standalone-install.sh "$INSTALLATION"
```
Il verificatore è read-only: esegue `tht doctor --json` e `tht status`, senza ristartare lo stack,
rigenerare la configurazione o stampare il contenuto dei segreti.
### Gate A — smoke di piattaforma, su tutti e tre i computer
Registrare per ogni macchina:
```sh
uname -a
docker version --format '{{.Server.Version}} {{.Server.Arch}}'
tht version
bash scripts/check-standalone-prerequisites.sh
bash scripts/verify-standalone-install.sh "$INSTALLATION"
```
Il gate passa quando il clone è integro, Docker e Compose sono raggiungibili, `tht doctor` è OK,
lo stack è avviato e il frontend risponde sulla porta locale predefinita `http://127.0.0.1:8080`.
Il doctor controlla anche workspace e Pi: registrare separatamente i loro errori senza attribuire
ogni fallimento alla piattaforma. Verificare la disponibilità HTTP con:
```sh
curl --fail --silent --show-error http://127.0.0.1:8080/health
```
### Gate B — verifica funzionale
Eseguire almeno su una macchina con endpoint e credenziali disponibili:
Seguire prima [Workspace operations](../operations/workspaces.md) per importare/preparare il
workspace e configurare Database e binding locale. Il clone sorgente non trasferisce catalogo,
segreti o sessioni del Mac. Connettere l'eventuale VPN richiesta da DWH/modelli e verificare
che i relativi nomi siano raggiungibili anche dai container.
1. aprire `http://127.0.0.1:8080`;
2. autenticarsi con l’account locale configurato;
3. verificare che il workspace configurato sia leggibile;
4. avviare una domanda reale e completare i gate di revisione fino alla SQL finale;
5. fermare e riavviare l’installazione, poi ripetere `verify-standalone-install.sh`.
Un fallimento del Gate B per DWH, provider LLM, Git workspace o credenziali non dimostra da solo un
problema di portabilità Docker: registrare separatamente l’endpoint o il componente fallito.
## Ciclo di vita quotidiano
Usare il descriptor esplicito quando più installazioni possono essere scoperte:
```sh
INSTALLATION="$PWD/deploy/local/thothii-installation.yaml"
tht --installation "$INSTALLATION" status
tht --installation "$INSTALLATION" start
tht --installation "$INSTALLATION" start --build
tht --installation "$INSTALLATION" logs
tht --installation "$INSTALLATION" doctor --json
tht --installation "$INSTALLATION" stop
```
`start --build` è necessario dopo una modifica al codice o per ricostruire le immagini dal clone
corrente. `stop` conserva volumi, sessioni, catalogo, stato Pi, dati Qdrant e modello embedding.
Non usare `docker compose down --volumes` durante una prova normale: è un’operazione distruttiva
che cancella i dati locali.
Per aggiornamenti che richiedono migrazioni, seguire il runbook della release prima dell'avvio.
## Diagnosi rapida
| Sintomo | Controllo |
| --- | --- |
| `Docker Engine is not reachable` | avviare Docker Desktop oppure il servizio Docker e ripetere `docker info` |
| Windows vede Docker ma Bash fallisce | eseguire la guida dentro Ubuntu WSL2 e abilitare l’integrazione della distribuzione in Docker Desktop |
| `tht: command not found` | aprire una nuova shell e verificare `command -v tht`; se necessario ripetere il bootstrap |
| line ending o script non eseguibile | usare un clone nel filesystem WSL2/Linux e rieseguire `bash scripts/...` |
| architettura non supportata | verificare `docker version --format '{{.Server.Arch}}'`; la prova richiede `amd64` o `arm64` |
| descriptor o env file mancanti | usare il percorso `deploy/local/...` generato da `tht setup`, non un file copiato casualmente |
| stack sano ma workflow fallisce | controllare separatamente URL, credential bundle, workspace Git e autenticazione |
| dati apparentemente persi | verificare che non sia stato usato `down --volumes`; `stop` non rimuove i volumi |
## Checklist di accettazione
- [ ] Il clone proviene dal repository Gitea atteso e la revisione è stata annotata.
- [ ] Docker Desktop/Engine e Compose v2 sono disponibili.
- [ ] Il runtime restituisce un’architettura ammessa.
- [ ] `tht` è stato costruito dal repository e risponde a `tht version`.
- [ ] Il setup usa `profile: local`, `shell.mode: full` e `shell.defaultLocale: en`.
- [ ] Descriptor, `operator.env`, autenticazione e segreti sono presenti solo in `deploy/local/`.
- [ ] Nessun segreto compare in Git, URL, YAML pubblico o comandi registrati.
- [ ] Gate A superato su macOS Apple Silicon, Windows WSL2/x64 e Linux x64.
- [ ] Gate B eseguito almeno su una macchina con DWH e LLM disponibili.
- [ ] Stop/start e verifica finale completati senza cancellare i volumi.
## Fuori perimetro di questa release
Restano attività successive:
- pubblicare immagini pre-costruite su Docker Hub;
- ridurre ulteriormente i prompt tramite una configurazione non interattiva dedicata;
- creare pacchetti DMG, MSI/EXE, AppImage o altri installer nativi;
- fornire un runtime offline o un DWH/LLM locale incluso nell’applicazione.
## Documenti collegati
- [Install and first start](first-start.md)
- [Shell and localization](../operations/shell-and-localization.md)
- [Workspace operations](../operations/workspaces.md)
- `deploy/secrets/README.md` (runtime secrets)
+8
View File
@@ -1,5 +1,13 @@
# Docker installation in the current operating contexts
Rendering and authentication are independent of these Docker contexts. Explicitly
select full/en for the Mac, full/OIDC for an autonomous server, or
embedded/upstream for Omics. The same frontend image supports both renderings;
generated `config.js` and the host page decide the container, while the backend
and trusted proxy decide identity. See [shell configuration and deploy](operations/shell-and-localization.md)
and [server portal authentication](install/authentication-upstream.md). Do not
apply the standalone server authentication projection to the Omics upstream path.
ThothII uses one Compose topology:
- `frontend`

Some files were not shown because too many files have changed in this diff Show More