Compare commits
42
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6a4634dcf1 | ||
|
|
84804be9f8 | ||
|
|
c3caba94dd | ||
|
|
b1723c34c4 | ||
|
|
d6cdffea62 | ||
|
|
49333a2d35 | ||
|
|
b006b94479 | ||
|
|
bdcd8fcd28 | ||
|
|
cf90c1bd51 | ||
|
|
571a4bcaa2 | ||
|
|
9051463654 | ||
|
|
26c5605ff7 | ||
|
|
3535fda958 | ||
|
|
2953f6b608 | ||
|
|
45db3a239b | ||
|
|
7d826e46c0 | ||
|
|
648434a32e | ||
|
|
023b822f83 | ||
|
|
d8a29bfbdd | ||
|
|
a59624a68f | ||
|
|
5af4408194 | ||
|
|
e088abd60a | ||
|
|
803e9e9201 | ||
|
|
8c81996896 | ||
|
|
eed398e569 | ||
|
|
c8d276ddc6 | ||
|
|
2d1b714ebe | ||
|
|
c7e5f295e6 | ||
|
|
f52bf22e05 | ||
|
|
840344706f | ||
|
|
41b9fed4d5 | ||
|
|
36bf659ea9 | ||
|
|
4a67d60233 | ||
|
|
debb63d87b | ||
|
|
3943022a97 | ||
|
|
f5ec2d9313 | ||
|
|
82e2c91f42 | ||
|
|
8fe526dd6e | ||
|
|
e68e80a33d | ||
|
|
818563c408 | ||
|
|
50c546e42d | ||
|
|
651a5c7902 |
@@ -17,6 +17,7 @@ frontend/vite.database-management-prototype.config.ts
|
||||
!deploy/env/*.env.example
|
||||
deploy/thothii.env
|
||||
deploy/secrets/
|
||||
deploy/psd/
|
||||
harness/workspaces/*.yaml
|
||||
!harness/workspaces/local.yaml
|
||||
!harness/workspaces/tht.example.yaml
|
||||
|
||||
@@ -46,6 +46,7 @@ deploy/secrets/*
|
||||
# Per-installation configuration generated by `tht setup` (examples stay tracked).
|
||||
deploy/*/thothii-installation.yaml
|
||||
deploy/*/operator.env
|
||||
deploy/*/auth/
|
||||
deploy/*/generated/
|
||||
deploy/*/secrets/*
|
||||
!deploy/*/secrets/.gitkeep
|
||||
|
||||
@@ -98,8 +98,18 @@ frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness →
|
||||
- **`tht`'s `-c`/`--config` is a PER-COMMAND option** — it must follow the subcommand, never
|
||||
precede it (`ThtRunner.buildArgv` enforces this; prepending caused live 500s).
|
||||
- **`--json` output must be pristine** (only valid JSON on stdout) — used as a machine contract.
|
||||
- **UI strings are English; document *content* stays the workspace language** (Italian for
|
||||
`psd`) because it's the real data. Only chrome/labels are English.
|
||||
- **Localization:** deterministic UI uses the EN/IT catalogs with English fallback;
|
||||
model interaction uses the session manifest's immutable `interaction_language`.
|
||||
Workspace content, SQL, and identifiers remain unchanged. For shell modes, portal
|
||||
integration, or translations, read `docs/operations/shell-and-localization.md`.
|
||||
- **Server deployment:** for the coordinated ThothII/Omics upgrade, follow
|
||||
`docs/operations/server-codex-handoff.md`; it supersedes earlier Omics delivery
|
||||
instructions. Omics source integration uses GitHub with no repository relay prerequisite.
|
||||
- **Server identity:** for portal login/logout, proxy headers, or Omics deploy, read
|
||||
`docs/install/authentication-upstream.md` before changing authentication. Omics
|
||||
uses embedded/upstream, not a second ThothII OIDC login. Full/embedded rendering
|
||||
is documented in `docs/architecture/application-shell.md`; release acceptance
|
||||
is in `docs/testing/authentication-manual-acceptance.md`.
|
||||
- **Workspace schema v4** defines workspace identity and optional Evidence only. PostgreSQL Metadata
|
||||
Catalog owns database identity, binding, schema, descriptions, sensitivity, and relationships;
|
||||
embedding/model facts come from the installation catalog. The legacy `harness/workspaces/*.yaml` runtime snapshots still use
|
||||
|
||||
+147
-12
@@ -91,21 +91,79 @@ correzione successiva crea una nuova sessione derivata, collegata a quella prece
|
||||
dopo la finalizzazione. Non può modificare il ledger, gli artifact canonici o lo stato
|
||||
terminale della sessione.
|
||||
|
||||
## Memory
|
||||
|
||||
**Memory Module** — Il modulo che possiede le conoscenze ed esperienze curate per
|
||||
migliorare schema linking e generazione SQL di domande future. Le Memory appartengono
|
||||
a un workspace e rimangono distinte dalle Evidence.
|
||||
|
||||
**Memory Card** — L'unità di contenuto gestibile del Memory Module, con identità,
|
||||
ambito di applicazione e provenienza. Il formato è allineato per analogia alle
|
||||
Evidence, senza implicare la stessa origine o lo stesso percorso di pubblicazione.
|
||||
|
||||
**Reusable Memory** — Una Memory Card che esprime un chiarimento di dominio, una
|
||||
regola di costruzione SQL o un errore da evitare con motivo compreso e approvato.
|
||||
La sua validità è circoscritta a un ambito esplicito e non deriva dalla sola
|
||||
approvazione di una scelta occasionale in una domanda.
|
||||
|
||||
**Solved Question** — Una Memory Card che conserva una domanda risolta con la
|
||||
relativa soluzione SQL e il contesto necessario a interpretarla. È un exemplar
|
||||
consultativo: i parametri e le scelte del caso non diventano regole generali.
|
||||
|
||||
**Memory Graph** — L'insieme dei collegamenti espliciti fra card che contribuisce
|
||||
al recupero di conoscenze pertinenti oltre alla somiglianza del contenuto. Il
|
||||
ritrovamento di una card tramite un collegamento non ne implica l'approvazione.
|
||||
|
||||
**Memory Link** — Un collegamento curato fra card, con destinazione e significato
|
||||
espliciti, che contribuisce alla consultazione di contenuti pertinenti. La sua
|
||||
rimozione non comporta la cancellazione delle card collegate.
|
||||
|
||||
## Evidence
|
||||
|
||||
**Context specialist** — La persona competente sul dominio che redige e cura il
|
||||
contenuto delle Evidence. Può essere distinta da chi amministra l'installazione;
|
||||
il suo lavoro di redazione non richiede accesso al database applicativo.
|
||||
|
||||
**Evidence draft** — Il documento iniziale scritto dallo specialista di contesto,
|
||||
che il sistema acquisisce e raffina in Evidence Unit. Può essere redatto e
|
||||
consegnato indipendentemente dall'installazione che userà le Evidence risultanti.
|
||||
|
||||
**Evidence Module** — Il modulo autonomo che possiede la preparazione delle Evidence e
|
||||
la loro consultazione durante il workflow. La preparazione avviene fuori dalle singole
|
||||
sessioni; il workflow usa soltanto contenuti già pubblicati. A runtime contribuisce agli
|
||||
stage semantici esistenti, senza diventare uno stage visibile e senza modificare ledger,
|
||||
artifact o stato del workflow.
|
||||
|
||||
**Source Evidence** — Un documento originale del workspace, conservato senza modifiche
|
||||
come riferimento umano e origine della successiva ristrutturazione.
|
||||
**Source Evidence** — Il documento o la dichiarazione che sostiene il contenuto
|
||||
corrente di una Evidence Unit. Un documento acquisito viene conservato come
|
||||
riferimento umano; una dichiarazione manuale attribuisce il contenuto alla persona
|
||||
che lo ha scritto e approvato.
|
||||
|
||||
**Manual Evidence declaration** — Una dichiarazione esplicita dell'amministratore
|
||||
che sostiene una Evidence creata direttamente o una correzione del suo significato.
|
||||
Non implica una verifica indipendente da parte di una fonte documentale esterna.
|
||||
|
||||
**Evidence origin** — Il documento da cui una Evidence Unit è stata inizialmente
|
||||
derivata. Può restare collegato per provenienza e confronto con gli aggiornamenti
|
||||
anche quando una dichiarazione manuale sostiene il testo corrente. La sola origine
|
||||
non dimostra il supporto semantico di una successiva correzione.
|
||||
|
||||
**Local Evidence archive** — L'insieme delle Evidence curate custodite
|
||||
dall'installazione, distinto dalle draft originali e dai contenuti derivati per
|
||||
la ricerca. Comprende le correzioni manuali e i ritiri deliberati.
|
||||
|
||||
**Consolidated Evidence** — Una versione delle Evidence locali controllata come
|
||||
insieme coerente e pronta per l'attivazione. I file ancora in modifica non ne
|
||||
cambiano il contenuto.
|
||||
|
||||
**Active Evidence** — La versione consolidata disponibile alla consultazione del
|
||||
core. Un tentativo di aggiornamento fallito conserva la versione attiva precedente.
|
||||
|
||||
**Evidence Unit** — La più piccola unità semantica coerente, revisionabile e ricercabile
|
||||
derivata da una sola Source Evidence. Possiede un identificatore stabile indipendente
|
||||
dal kind, assegnato una volta nella forma `evidence:<slug>`; fonti diverse non vengono
|
||||
fuse automaticamente.
|
||||
fondata su una Source Evidence corrente, anche manuale, e con eventuale origine
|
||||
documentale distinta. Possiede un identificatore stabile indipendente dal kind,
|
||||
assegnato una volta nella forma `evidence:<slug>`; fonti diverse non vengono fuse
|
||||
automaticamente.
|
||||
|
||||
**Evidence kind** — La categoria semantica di una Evidence Unit, che ne determina i
|
||||
campi specifici e ne orienta l'uso. Ogni unità ha un solo kind primario; i tipi iniziali
|
||||
@@ -151,10 +209,9 @@ avanzare fino a un retry riuscito.
|
||||
nella sessione: stage semantico, purpose, generazione interrogata e identificatori delle
|
||||
Evidence restituite. Non duplica il contenuto delle Evidence.
|
||||
|
||||
**Curated Evidence** — Una o più Evidence Unit ristrutturate a partire da una Source
|
||||
Evidence e conservate nel repository del workspace come proposte per la revisione
|
||||
umana. Git conserva la versione precedente e rende visibile ogni modifica; una Curated
|
||||
Evidence non è ancora contenuto autorevole del runtime.
|
||||
**Curated Evidence** — Una o più Evidence Unit preparate da documenti o curate
|
||||
manualmente. La presenza nell'archivio curato non implica da sola che il contenuto
|
||||
sia già attivo per il workflow.
|
||||
|
||||
**Published Evidence** — Le Curated Evidence valide appartenenti alla revisione attiva
|
||||
del workspace e alla generazione Evidence pubblicata. L'approvazione umana precede
|
||||
@@ -176,9 +233,17 @@ una Evidence Unit. Il sistema ne verifica deterministicamente la presenza dopo l
|
||||
normalizzazione meccanica; il curatore resta responsabile di verificarne la sufficienza
|
||||
semantica.
|
||||
|
||||
**Evidence resolution** — L'operazione esplicita con cui un curatore ritira una
|
||||
Evidence Unit oppure la ricollega a un Source Evidence esistente. Aggiorna documento e
|
||||
manifest insieme, lascia un diff Git revisionabile e non pubblica né crea commit.
|
||||
**Evidence resolution** — La decisione esplicita con cui un curatore risolve un
|
||||
problema di una Evidence Unit, correggendola, ritirandola oppure ricollegandola a
|
||||
una fonte adeguata.
|
||||
|
||||
**Source update conflict** — Un contrasto fra una fonte aggiornata e una correzione
|
||||
manuale già approvata. La correzione resta in uso fino alla risoluzione esplicita
|
||||
del confronto da parte dell'amministratore.
|
||||
|
||||
**Evidence source refresh** — La riacquisizione delle fonti esterne richiesta
|
||||
dall'amministratore per rilevarne le modifiche. Fra due aggiornamenti il contenuto
|
||||
già acquisito resta il riferimento per preparazione e consultazione.
|
||||
|
||||
**Review item** — Un blocco di revisione descritto da codice stabile, messaggio umano e
|
||||
campo opzionale. Finché viene mantenuto nell'Evidence Unit, ne impedisce la
|
||||
@@ -506,3 +571,73 @@ _Avoid_: Sensitive Data Suggestion Event
|
||||
**Introspection Capability** — Una categoria di struttura fisica che una Database Binding
|
||||
può osservare, come tabelle, colonne, relazioni, indici o enum. Una capability non disponibile
|
||||
è distinta da una capability osservata che non ha restituito elementi.
|
||||
|
||||
## Amministrazione e integrazione
|
||||
|
||||
**Workspace Readiness** — La preparazione di uno specifico Workspace per l'uso nel
|
||||
workflow, comprensiva della disponibilità degli artefatti derivati dai suoi metadati
|
||||
Database e dalle sue Evidence. Il preprocessing appartiene a questa preparazione;
|
||||
la configurazione e la sincronizzazione del catalogo restano responsabilità Database.
|
||||
|
||||
**Administration Surface** — Una superficie amministrativa autonoma per configurare o curare una
|
||||
parte dell'installazione. Workspace, Evidence, Memory, Database e Pi sono superfici peer e non
|
||||
dipendono dall'esistenza di una sessione attiva.
|
||||
|
||||
**Administration Page** — La rappresentazione a pagina intera di una Administration Surface, con
|
||||
una gerarchia condivisa per identità, stato, azioni e contenuto. Un form amministrativo appartiene
|
||||
alla pagina e non a una popup come contenitore principale.
|
||||
_Avoid_: management popup, settings modal
|
||||
|
||||
**Administration Route** — L'identità navigabile di una Administration Surface nel browser. Deve
|
||||
essere ripristinabile con refresh e cronologia e non contiene valori transitori o segreti dei form.
|
||||
|
||||
**Embedded Thoth Shell** — L'esperienza Thoth ospitata dentro il documento e il contesto visuale di
|
||||
un portale host. Conserva la propria gerarchia funzionale, ma deve rispettare la geometria,
|
||||
l'autenticazione e le regole responsive del portale host.
|
||||
|
||||
**Full Thoth Shell** — L'esperienza Thoth autonoma che possiede il proprio header e il proprio
|
||||
layout di pagina. Non replica la navigazione amministrativa del portale host e non dipende dal suo
|
||||
template visuale.
|
||||
|
||||
**Shell mode** — La scelta di installazione fra `embedded` e `full`. Determina chi possiede il
|
||||
chrome globale, i comandi di identità e le integrazioni visuali, ma non cambia il workflow o la
|
||||
persistenza delle sessioni.
|
||||
|
||||
**Fullscreen state** — Lo stato temporaneo in cui il documento applicativo occupa il fullscreen
|
||||
del browser. È distinto da `Shell mode`: una Full Thoth Shell può essere aperta senza fullscreen;
|
||||
il passaggio è attivato da un comando esplicito e può essere annullato con la stessa azione o con
|
||||
il comando nativo del browser.
|
||||
|
||||
**Portal Shell Adapter** — Il confine sostituibile che traduce lo stato e i comandi del chrome di
|
||||
un portale host nel modello semantico usato da Thoth. L'adapter non possiede autorizzazione,
|
||||
sessioni di workflow o contenuti del modello.
|
||||
|
||||
**Host Shell State** — Il minimo stato visuale fornito dal portale host: locale UI, tema e stato
|
||||
fullscreen. In una Embedded Thoth Shell è la fonte autorevole per queste preferenze;
|
||||
non include identità, token o stato di autenticazione, che restano responsabilità dell'accesso.
|
||||
|
||||
**UI locale** — La lingua delle label, dei messaggi, dei tooltip, degli stati e delle istruzioni
|
||||
non generate dal modello nell'interfaccia Thoth. È distinta dalla lingua dei contenuti di un
|
||||
workspace.
|
||||
|
||||
**Interaction language** — La lingua in cui il modello presenta domande, spiegazioni e proposte
|
||||
al revisore durante una sessione. Viene fissata alla creazione della sessione e rimane invariata
|
||||
durante una ripresa, anche se la UI locale corrente cambia.
|
||||
|
||||
**Administrative Page Family** — L'insieme delle cinque Administration Page che condividono shell,
|
||||
navigazione, tipografia e regole responsive, pur mantenendo contenuti e operazioni specifici:
|
||||
Workspace, Evidence, Memory, Database e Pi.
|
||||
|
||||
## Installazione
|
||||
|
||||
**Manual standalone installation** — Una copia di ThothII predisposta per l'uso autonomo da una
|
||||
persona che possiede il computer, con una Full Thoth Shell e servizi applicativi locali. La
|
||||
procedura non implica che DWH o provider LLM siano locali o disponibili offline.
|
||||
|
||||
**Installation bootstrap** — L'insieme delle attività iniziali che rende disponibile una
|
||||
installazione manuale: verifica dell'host, generazione della configurazione, predisposizione
|
||||
delle credenziali protette e avvio dei servizi. Non è un installer dell'applicazione.
|
||||
|
||||
**Platform acceptance** — La verifica che una Manual standalone installation possa essere
|
||||
predisposta e avviata su una specifica combinazione di sistema operativo, architettura e runtime,
|
||||
distinta dalla verifica funzionale del collegamento a DWH e provider LLM.
|
||||
|
||||
@@ -11,50 +11,50 @@ colors:
|
||||
warm-graphite: "oklch(26.78% 0.0097 355.6)"
|
||||
muted-graphite: "oklch(51.33% 0.0088 345.6)"
|
||||
quiet-border: "oklch(90.93% 0.0035 354.7)"
|
||||
success-mint: "oklch(75.77% 0.1581 165)"
|
||||
success-mint: "oklch(46% 0.095 160)"
|
||||
navigation-active: "oklch(92.5% 0.052 23.2)"
|
||||
navigation-active-hover: "oklch(89.5% 0.071 23.2)"
|
||||
navigation-active-foreground: "oklch(36.5% 0.11 23.2)"
|
||||
navigation-active-border: "oklch(60% 0.135 23.2)"
|
||||
warning-amber: "oklch(85.23% 0.1386 78.9)"
|
||||
information-blue: "oklch(70.35% 0.1128 221.3)"
|
||||
warning-amber: "oklch(48% 0.09 70)"
|
||||
information-neutral: "oklch(51.33% 0.0088 345.6)"
|
||||
typography:
|
||||
display:
|
||||
fontFamily: "Fraunces, Source Serif Pro, Georgia, Times New Roman, serif"
|
||||
fontSize: "3rem"
|
||||
fontFamily: "Manrope Variable, Manrope, system-ui, sans-serif"
|
||||
fontSize: "1.5rem"
|
||||
fontWeight: 600
|
||||
lineHeight: 1.03
|
||||
letterSpacing: "-0.025em"
|
||||
headline:
|
||||
fontFamily: "Fraunces, Source Serif Pro, Georgia, Times New Roman, serif"
|
||||
fontSize: "1.875rem"
|
||||
fontFamily: "Manrope Variable, Manrope, system-ui, sans-serif"
|
||||
fontSize: "1.5rem"
|
||||
fontWeight: 600
|
||||
lineHeight: 1.15
|
||||
letterSpacing: "-0.015em"
|
||||
title:
|
||||
fontFamily: "Fraunces, Source Serif Pro, Georgia, Times New Roman, serif"
|
||||
fontSize: "1.2rem"
|
||||
fontFamily: "Manrope Variable, Manrope, system-ui, sans-serif"
|
||||
fontSize: "1.25rem"
|
||||
fontWeight: 600
|
||||
lineHeight: 1.25
|
||||
letterSpacing: "-0.01em"
|
||||
body:
|
||||
fontFamily: "Manrope, -apple-system, BlinkMacSystemFont, Segoe UI, system-ui, Arial, sans-serif"
|
||||
fontSize: "0.9375rem"
|
||||
fontFamily: "Manrope Variable, Manrope, -apple-system, BlinkMacSystemFont, Segoe UI, system-ui, Arial, sans-serif"
|
||||
fontSize: "1rem"
|
||||
fontWeight: 400
|
||||
lineHeight: 1.65
|
||||
letterSpacing: "normal"
|
||||
control:
|
||||
fontFamily: "Manrope, -apple-system, BlinkMacSystemFont, Segoe UI, system-ui, Arial, sans-serif"
|
||||
fontFamily: "Manrope Variable, Manrope, -apple-system, BlinkMacSystemFont, Segoe UI, system-ui, Arial, sans-serif"
|
||||
fontSize: "0.875rem"
|
||||
fontWeight: 600
|
||||
lineHeight: 1.25
|
||||
letterSpacing: "0.005em"
|
||||
label:
|
||||
fontFamily: "ui-monospace, SF Mono, Cascadia Code, Menlo, Consolas, monospace"
|
||||
fontSize: "0.6875rem"
|
||||
fontFamily: "Manrope Variable, Manrope, system-ui, sans-serif"
|
||||
fontSize: "0.75rem"
|
||||
fontWeight: 600
|
||||
lineHeight: 1.25
|
||||
letterSpacing: "0.06em"
|
||||
letterSpacing: "normal"
|
||||
rounded:
|
||||
xs: "4px"
|
||||
sm: "6px"
|
||||
@@ -113,6 +113,16 @@ components:
|
||||
|
||||
# Design System: ThothII
|
||||
|
||||
## Visual review branch, September 2026
|
||||
|
||||
The revision on `codex/ui-visual-review` is approved for implementation and Docker visual review,
|
||||
not yet for adoption on `main`. The previous look remains recoverable from the base commit and
|
||||
the preserved Docker image. Historical prototypes must remain untouched.
|
||||
|
||||
This revision follows Impeccable's product register: one locally bundled Manrope family for the
|
||||
whole UI, five fixed size roles, red as the sole brand accent and additional color only for meaningful
|
||||
state. The primary scene remains an analyst reading data and SQL in a well-lit office.
|
||||
|
||||
## Overview
|
||||
|
||||
**Creative North Star: "The Clinical Workbench"**
|
||||
@@ -134,7 +144,7 @@ disciplined and tactile, never playful, sluggish, or visually unstable.
|
||||
**Key Characteristics:**
|
||||
|
||||
- Warm, restrained surfaces with one scarce red accent.
|
||||
- Editorial headings paired with highly legible operational body text.
|
||||
- One sans-serif family, with hierarchy expressed through size, weight and spacing.
|
||||
- Dense information organized through hierarchy, rhythm, and progressive disclosure.
|
||||
- Persisted artifacts and reviewer decisions presented as the visual source of truth.
|
||||
- Fast state feedback with reduced-motion parity.
|
||||
@@ -150,6 +160,15 @@ users can scan structure without adding nested containers.
|
||||
|
||||
## Colors
|
||||
|
||||
The full-mode application header matches Omics Portal's `--gsd-red-primary`
|
||||
(`#CB333B`) in both themes. Its complete wordmark, including `II`, and controls
|
||||
use a near-white foreground. This header is absent in embedded mode. The sidebar
|
||||
and welcome wordmarks retain their red suffix. Context editing places workspace,
|
||||
model and Done in one desktop row, stacking on narrow containers. Session-scope
|
||||
tabs retain their selected fill and accessible keyboard state with a uniform one-pixel
|
||||
border on every side, gray when inactive and red when active. Their padding is 11px
|
||||
horizontal and 3px vertical, with a 38px minimum height and wrapping labels.
|
||||
|
||||
The palette combines warm porcelain surfaces, warm graphite text, and an instrument red used only
|
||||
for action, focus, and important state. OKLCH values in the frontmatter are normative because the
|
||||
frontend uses OKLCH tokens directly.
|
||||
@@ -178,7 +197,8 @@ frontend uses OKLCH tokens directly.
|
||||
foreground. It shares Instrument Red's hue but uses a lighter, lower-chroma fill, so location is
|
||||
visible without carrying the full weight of a primary action.
|
||||
- **Warning Amber** (`warning-amber`): waiting, attention, and in-progress states.
|
||||
- **Information Blue** (`information-blue`): informational state when red would imply action.
|
||||
- **Information**: neutral text and indicators for dates, protocols and ordinary status. The legacy
|
||||
`--info` token resolves to muted foreground, not an additional blue accent.
|
||||
|
||||
The dark theme keeps the same semantic mapping with neutral near-black surfaces and a slightly
|
||||
lighter red accent. Do not introduce a second visual identity for dark mode.
|
||||
@@ -191,30 +211,33 @@ for their named states. Color is never the only state indicator.
|
||||
|
||||
## Typography
|
||||
|
||||
**Display Font:** Fraunces, with Source Serif Pro, Georgia, and Times New Roman fallbacks
|
||||
**Body Font:** Manrope, with native system sans-serif fallbacks
|
||||
**Label/Mono Font:** SF Mono or Cascadia Code, with Menlo and Consolas fallbacks
|
||||
**UI Font:** locally bundled Manrope Variable, with Manrope and native sans-serif fallbacks.
|
||||
**Technical Font:** SF Mono or Cascadia Code, with Menlo and Consolas fallbacks.
|
||||
|
||||
**Character:** Fraunces gives persisted artifacts and key headings editorial authority. Manrope
|
||||
keeps dense controls and prose calm and readable. The mono register separates machine identity,
|
||||
metadata, SQL, identifiers, and micro-labels from natural-language content.
|
||||
Manrope covers headings, labels, controls, navigation and document reading. Monospace is reserved
|
||||
for SQL, code, paths and machine identifiers, never for ordinary UI labels or status headings.
|
||||
|
||||
### Hierarchy
|
||||
|
||||
- **Display** (600, `3rem`, `1.03`): authentication and exceptional page-level statements only.
|
||||
- **Headline** (600, `1.875rem`, `1.15`): major page or artifact titles.
|
||||
- **Title** (600, `1.2rem`, `1.25`): panel and document section hierarchy.
|
||||
- **Body** (400, `0.9375rem`, `1.65`): operational prose, with a target line length of 65 to 75
|
||||
- **Headline** (600, `1.5rem`, `1.3`): page or artifact titles, `--text-page`.
|
||||
- **Title** (600, `1.25rem`, `1.4`): section hierarchy, `--text-section`.
|
||||
- **Body** (400, `1rem`, `1.6`): operational prose, `--text-body`, with a target line length of 65 to 75
|
||||
characters where the surface controls width.
|
||||
- **Control** (600, `0.875rem`, `1.25`): buttons, inputs, tabs, and compact actions.
|
||||
- **Label** (600, `0.6875rem`, `0.06em` tracking): uppercase micro-labels, state metadata, and panel
|
||||
headers. Labels use the mono family.
|
||||
- **Control** (400–600, `0.875rem`, `1.5`): buttons, inputs, tables, tabs and compact subheadings,
|
||||
`--text-control`.
|
||||
- **Metadata** (400–600, `0.75rem`, `1.5`): secondary status, counts and timestamps, `--text-meta`.
|
||||
Labels use sentence case and normal tracking. Ordinary operational text never falls below 12px.
|
||||
|
||||
Typography uses fixed sizes. Responsive changes happen at structural breakpoints, not through fluid
|
||||
type scaling. Numeric data and identifiers use tabular numerals where comparison matters.
|
||||
|
||||
**The Three Registers Rule.** Serif means authority, sans means interaction and reading, mono means
|
||||
machine identity. Do not exchange these roles for novelty.
|
||||
**Application wordmark:** ThothII is a brand mark, not a page title: use Manrope semibold at
|
||||
48px (`3rem`) in the Core welcome area and 32px (`2rem`) in the session sidebar, with the
|
||||
`II` suffix in brand red. Preserve these sizes across responsive layouts.
|
||||
|
||||
**The One Family Rule.** The UI and document readers use sans-serif throughout. The legacy
|
||||
`--font-heading` alias resolves to `--font-sans`. Preserve technical monospace without turning it
|
||||
into a second decorative hierarchy. Do not shrink text to solve layout constraints.
|
||||
|
||||
**The Read Once Rule.** A heading, label, and body must be distinguishable on first glance through
|
||||
size and weight. Do not repeat headings in explanatory copy.
|
||||
@@ -279,27 +302,70 @@ default, hover, focus, active, disabled, loading, and error behavior where those
|
||||
- **Focus:** three-pixel Instrument Red ring with a clear border shift.
|
||||
- **Error / Disabled:** errors combine destructive color with explanatory text; disabled controls
|
||||
retain readable contrast and use 50 percent opacity.
|
||||
- **Metadata catalog model:** Database Management keeps one compact, installation-level
|
||||
metadata-generation LLM selector in the application header. The selection persists across
|
||||
database, table, column, and relationship views; when no usable profile is configured, the
|
||||
disabled control explains: “No metadata-generation LLM model is configured for this installation.”
|
||||
- **Global context:** the collapsible top shelf is the sole workspace/model selector for Core and
|
||||
Admin. Preserve independent remembered choices, installation defaults, operation locks and unsaved
|
||||
edit guards. Never introduce a separate metadata-generation default or selector.
|
||||
|
||||
### Navigation
|
||||
|
||||
- **Workspace readiness:** the Workspace navigation button carries an 8px dot to
|
||||
the right of its label. Green means a selected workspace with confirmed ready
|
||||
preprocessing and no query error; all other states are red. The button's
|
||||
tooltip and accessible description retain the translated exact state. Do not
|
||||
add a separate readiness text row or change the backend readiness gate.
|
||||
- **Session groups:** one accessible single-open accordion contains Active sessions
|
||||
and Archive, both initially closed. Below the scope tabs, show only their
|
||||
adjacent section headers, without a redundant Sessions heading. Selection and
|
||||
bulk-delete controls belong inside each panel and only appear for nonempty
|
||||
lists. Select all affects that list only, preserves the other list's selection,
|
||||
and exposes a mixed state for partial selection. Preserve the existing archived
|
||||
flag as the grouping rule, independent of whether a Pi process is running.
|
||||
Opening a section closes the other; either can be collapsed, including both.
|
||||
Empty lists show only the translated "No sessions yet." message.
|
||||
The open section uses the rail's remaining height; its list scrolls internally
|
||||
with a cap of `min(18rem, 35dvh)`, while its trigger remains outside that scroll
|
||||
area. The mobile navigation dialog supplies a bounded viewport-height container.
|
||||
Keyboard users can focus and scroll each labelled panel.
|
||||
- **Session entry:** one Session button returns to the current unfinished session,
|
||||
including provisional creation, without resetting or reconnecting it. Otherwise
|
||||
it prepares a new question using the normal readiness and unsaved-work guards.
|
||||
- **Style:** compact session rows use `8px` corners and restrained vertical padding.
|
||||
- **Default / Hover / Active:** porcelain at rest, Sunken Surface on hover, and a muted Navigation
|
||||
Active red with a defined border when current. Exactly one top-level navigation control is current.
|
||||
- **Administrative controls:** the admin-only Administration accordion groups Database management,
|
||||
a structural divider, Workspace management, and Pi management in that order. Its trigger exposes
|
||||
- **Administrative controls:** the admin-only Administration accordion groups Database,
|
||||
Memory, Evidence, a structural divider, Workspace, and Pi configuration in that order. Its trigger exposes
|
||||
expanded state and starts collapsed by default, while non-admin users do not receive the accordion
|
||||
or its navigation actions.
|
||||
- **Responsive:** collapse navigation structurally at the application breakpoint. Do not shrink
|
||||
labels into illegibility.
|
||||
labels into illegibility. Below 768px, Memory and Evidence management use the full content
|
||||
width; a Navigation button opens the shared accessible dialog. Selecting another archive
|
||||
page or pressing Escape closes it. Desktop retains the right session sidebar and its My sessions /
|
||||
All sessions tabs. Core retains question/answer, eight phases, reviewer gates and the left log.
|
||||
In embedded mode the portal owns the red header and left sidebar; ThothII must not duplicate them. Size to the
|
||||
actual application container. Narrow session document panels may use the available width.
|
||||
|
||||
### Session review and confirmations
|
||||
|
||||
Session dialogs use the visible application area, including the portal's header
|
||||
and side rail. Artifact and schema-column review can grow to 80rem wide and the
|
||||
available height; short confirmations use up to 40rem and at least 18rem when
|
||||
space permits. Keep a 24px outer margin on desktop and 8px on small or short
|
||||
screens. Long review content scrolls internally; on very short screens the
|
||||
whole dialog can also scroll so every action remains reachable.
|
||||
|
||||
Session forms and review gates repeat their existing primary confirmation above
|
||||
and below the content, sharing selection, validation, pending state and response
|
||||
handlers. Alternate-response inputs follow the same rule. Reserved navigation
|
||||
controls remain below the review. Stop/delete initially focus Cancel; rename
|
||||
initially focuses the name field. Administration dialogs and forms retain their
|
||||
existing layout and actions.
|
||||
|
||||
### Tabs
|
||||
|
||||
- **Shape:** compact label tabs sit on a shared baseline with rounded top corners and a two-pixel
|
||||
lower edge. Inactive labels retain a complete Quiet Border and Porcelain Card surface, so every
|
||||
lower edge, except session-scope tabs which use a uniform one-pixel border, rounded
|
||||
corners and a 4px gap without a shared border or negative bottom margin.
|
||||
Inactive labels retain a Quiet Border and Porcelain Card surface, so every
|
||||
label reads as a tab before interaction; hover feedback reinforces clickability.
|
||||
- **Current:** the selected tab uses the muted Navigation Active red for its fill, text, and defined border.
|
||||
It must expose `aria-selected`, participate in a labelled `tablist`/`tabpanel`, and be the only
|
||||
@@ -319,9 +385,42 @@ default, hover, focus, active, disabled, loading, and error behavior where those
|
||||
|
||||
### Curated Evidence Documents
|
||||
|
||||
Memory and Evidence share the `thot-knowledge-reader` reading contract. Use locally
|
||||
bundled Manrope with normal tracking for prose and labels, and these fixed roles:
|
||||
|
||||
- Card title: 24px, weight 600, line-height 1.3 (`thot-knowledge-title`).
|
||||
- Field/section heading, including Scope and Provenance: 20px, weight 600,
|
||||
line-height 1.4, 8px clearance below (`thot-knowledge-heading`).
|
||||
- All narrative text, including scope, lists and provenance: 16px, weight 400,
|
||||
line-height 1.65. Do not apply compact UI text sizes to these fields.
|
||||
- Authored Markdown subheadings inside a field: 16px, weight 600, line-height 1.5,
|
||||
24px above/8px below. They remain subordinate to the enclosing field heading;
|
||||
their semantic heading levels and original content are preserved.
|
||||
- Technical metadata labels/values: 14px/1.5, with weight 600 for labels.
|
||||
Only code, paths and machine identifiers use the technical monospace family at
|
||||
14px/1.65, identical for inline and fenced code (never compound `em` shrinkage).
|
||||
|
||||
Separate reading sections by 24px; keep the first Markdown block flush with its
|
||||
field heading's 8px bottom gap. The same typography applies in light/dark and at
|
||||
all responsive widths. Controls and archive indexes retain their compact UI roles.
|
||||
|
||||
Memory and Evidence detail readers use the entire available content width, without
|
||||
the ordinary 72–75ch prose cap. This is the owner's explicit reading-layout choice.
|
||||
Long unstructured paragraphs are split for display at existing sentence/semicolon
|
||||
boundaries outside inline code and links; authored Markdown structure and stored
|
||||
content are unchanged. Paragraph spacing is 1.25em. Scope and provenance share the
|
||||
available width; provenance excerpts render Markdown rather than literal markers.
|
||||
Copy actions use the two-overlapping-sheets icon, an accessible name/tooltip and
|
||||
live success/failure feedback instead of a visible Copy label.
|
||||
|
||||
Memory has four explicitly FAKE formatting examples, one per family, in a separate
|
||||
expandable section. They reuse the real detail reader but never enter persistence,
|
||||
indexing, link search or model recall, and expose no edit/delete/save actions.
|
||||
|
||||
Curated evidence follows a fixed reading order: title, compact type and purpose summary, scope,
|
||||
typed content, supporting excerpts, review items, then collapsed technical provenance. Machine
|
||||
metadata stays in invisible comments so GitHub Preview shows only the reviewable document.
|
||||
typed content, supporting excerpts, review items, then technical provenance. Curated v4 files
|
||||
use short, visible YAML frontmatter for identity and classification. The Markdown title and
|
||||
body are authoritative; hidden payload comments are a legacy format converted on consolidation.
|
||||
|
||||
`applies_to` is rendered as “Ambito di applicazione” with separate bullet lists for concepts,
|
||||
tables, and columns. Enum values also use lists. Tables are forbidden for metadata, scope, or any
|
||||
@@ -329,7 +428,9 @@ one-dimensional collection; reserve tables for genuinely two-dimensional dataset
|
||||
identifiers use inline code. SQL uses fenced code. Supporting excerpts use blockquotes.
|
||||
|
||||
**The Review Surface Rule.** The visible Markdown must be readable without understanding the
|
||||
machine contract. Technical metadata belongs in progressive disclosure, not above the title.
|
||||
machine contract. In Administration, explain current and original provenance separately and
|
||||
keep file-editing templates and Git instructions in progressive disclosure. Show actual host
|
||||
paths with copy controls, never browser file links to container-only locations.
|
||||
|
||||
## Do's and Don'ts
|
||||
|
||||
@@ -341,7 +442,8 @@ machine contract. Technical metadata belongs in progressive disclosure, not abov
|
||||
- **Do** preserve information density with headings, rhythm, and progressive disclosure.
|
||||
- **Do** keep keyboard focus explicit and pair color with text, shape, icon, or position.
|
||||
- **Do** respect `prefers-reduced-motion` while preserving immediate non-kinetic feedback.
|
||||
- **Do** use English for interface chrome and the workspace language for persisted document content.
|
||||
- **Do** use the selected interface language (English by default) for chrome and preserve the
|
||||
workspace language for persisted domain content. Session interaction language remains pinned.
|
||||
- **Do** render curated metadata and scope as Markdown prose or lists, never as a frontmatter table.
|
||||
- **Do** break long curated rules into paragraphs, labelled subsections, and lists at existing
|
||||
punctuation boundaries while preserving the exact canonical text for machines.
|
||||
|
||||
+398
-3
@@ -1,6 +1,6 @@
|
||||
# ThothII — Project State
|
||||
|
||||
Last updated: 2026-09-06.
|
||||
Last updated: 2026-09-14.
|
||||
|
||||
This file is the short operational snapshot. Stable commands and the architecture mental model
|
||||
live in `AGENTS.md`; current design and runtime contracts live under `docs/architecture/`,
|
||||
@@ -12,8 +12,222 @@ Authentik, internal catalog/embedding services, and the PSD workspace repository
|
||||
`docs/operations/server-upgrade-gitea-workspace-v2.md`. Treat its operator gates and rollback
|
||||
requirements as mandatory; do not replace the running server stack in place.
|
||||
|
||||
## Session review layout deployed — 2026-09-14
|
||||
|
||||
Session-only dialogs now use the visible app bounds: artifact/column review grows
|
||||
up to 80rem wide and the available height; short confirmations grow to 40rem.
|
||||
Existing primary actions appear above and below session forms and review content,
|
||||
with shared handlers, validation and pending state. Stop/delete focus Cancel;
|
||||
rename focuses the name field. Administration surfaces are unchanged. The activity
|
||||
shortcut is relabelled To Administration / Vai all’Amministrazione, retaining its
|
||||
workspace-management destination. 778 frontend tests, five responsive browser
|
||||
scenarios, typecheck, translations and the production build passed. The owner
|
||||
authorized deployment and the server frontend was recreated at 18:09 CEST with tag `b1723c34-session-dialogs-20260914`. Frontend is healthy,
|
||||
Omics serves the new assets, and doctor passes 13/13 checks. Core and Omics web
|
||||
were not restarted. See DESIGN.md for the layout contract and
|
||||
`docs/reports/2026-09-14-session-dialogs-release.md` for provenance and rollback.
|
||||
|
||||
## Session composer and empty Memory fix deployed — 2026-09-14
|
||||
|
||||
The embedded shell now caps its height at the portal mount height, keeping steering
|
||||
and Stop & save visible. Empty authoritative Memory archives return zero results
|
||||
without requiring embedding/BM25; SQL-rule embedding is lazy and shared. The live
|
||||
empty PSD archive reproduces 503 with the current image and succeeds with the
|
||||
candidate. 54 Memory tests, 90 frontend tests, five browser scenarios and both image
|
||||
builds passed. The owner authorized the restart and core/frontend were recreated on
|
||||
2026-09-14 at 17:18 CEST with tags `49333a2d-session-memory-fix`, from code committed
|
||||
as `d6cdffea`. Both are healthy; production Memory search returns `[]`, Omics serves
|
||||
the corrected CSS, native doctor passes 13/13 checks, and admissions are reopened.
|
||||
Session inventory is preserved. Backups and rollback images are retained. Native
|
||||
CLI diagnostics must run as installation owner UID 10001 with access to Compose;
|
||||
see `docs/reports/2026-09-14-session-layout-memory-fix.md` for exact commands and
|
||||
the remaining interactive browser acceptance.
|
||||
|
||||
## Full/embedded shell and bilingual interface
|
||||
|
||||
Full/embedded shell, EN/IT UI, immutable session interaction language, dark theme,
|
||||
fullscreen and full-mode logout are implemented. The local Mac descriptor explicitly
|
||||
sets `shell.mode: full` and `shell.defaultLocale: en`; the native `tht` and local
|
||||
core/frontend images were updated on 2026-09-13. Omics uses embedded with server-side
|
||||
identity verification and a replaceable presentation-only PortalAdapter; this does
|
||||
not mean this branch was verified on the production server. Its changes are
|
||||
in Omics commit `95154e1`; production deployment remains pending.
|
||||
See `docs/operations/shell-and-localization.md` for integration and installation
|
||||
instructions and `docs/reports/2026-09-13-full-shell-implementation.md` for tests,
|
||||
independent reviews, local browser checks and rollback details.
|
||||
|
||||
### Documentation handoff before branch closure — 2026-09-13
|
||||
|
||||
Current rendering architecture is in `docs/architecture/application-shell.md`;
|
||||
the exact portal identity/proxy contract is in `docs/install/authentication-upstream.md`.
|
||||
`docs/operations/server-codex-handoff.md` is the current server delivery/deploy
|
||||
runbook, including Omics source integration from GitHub, configuration, tests and
|
||||
rollback. It supersedes the earlier Omics repository-relay instructions. The acceptance matrix is
|
||||
`docs/testing/authentication-manual-acceptance.md`. README, documentation navigation,
|
||||
user/installation/authentication guides and descriptor examples point to these
|
||||
paths. Local examples explicitly use full/en; the projected server example is
|
||||
for standalone OIDC, not Omics upstream. No runtime configuration or deployment
|
||||
was changed by that documentation pass. The 2026-09-14 delivery is prepared for
|
||||
promotion to main; the actual merge is recorded in Git. PSD acceptance remains pending.
|
||||
|
||||
### Navigation readiness and session accordions — 2026-09-13
|
||||
|
||||
The Workspace navigation button now carries an accessible green/red readiness
|
||||
dot instead of a separate text row. Green requires a selected workspace and a
|
||||
successful, current `ready` response; checking, unavailable and other states are
|
||||
red, with the state exposed through the tooltip and accessible description.
|
||||
The backend readiness gate is unchanged.
|
||||
|
||||
Active sessions (non-archived) and Archive both start collapsed. Their adjacent
|
||||
headers are the only visible content below the scope tabs: no Sessions heading
|
||||
or external selection toolbar. Each nonempty panel owns its Select all and
|
||||
bulk-delete controls, scoped to that list and preserving the other's selection.
|
||||
As of 2026-09-14, only one section can be open at a time, and either can be
|
||||
collapsed. Empty lists show only "No sessions yet." The open section uses the
|
||||
remaining sidebar height, with scrolling content capped at `min(18rem, 35dvh)`. The mobile
|
||||
navigation dialog also provides a bounded height. Keyboard controls and labels
|
||||
are retained. See `DESIGN.md` and `docs/guida-utente.md` for the UI contract.
|
||||
|
||||
Verification: 768 frontend unit tests, 20 browser scenarios (including 80 mocked
|
||||
sessions at 390/1280px), TypeScript and the production frontend build passed.
|
||||
Only the Mac frontend was recreated; it is healthy at `127.0.0.1:8080`.
|
||||
Core, catalog, Qdrant and embedding containers were not changed. The prior
|
||||
frontend image is retained as
|
||||
`thothii-frontend:before-single-session-accordion-20260914` for rollback.
|
||||
|
||||
### Current Omics delivery and server handoff — 2026-09-14
|
||||
|
||||
The owner corrected the delivery requirement: Omics is obtained from GitHub,
|
||||
not relayed to another repository as part of this deployment. Any optional
|
||||
server-side repository copy is solely the owner's separate concern. This
|
||||
supersedes the 2026-09-13 relay agreement, including historical delivery notes
|
||||
in the Omics branch. Do not make another remote publication a prerequisite.
|
||||
|
||||
GitHub branch `codex/thothii-embedded-shell` at
|
||||
`https://github.com/Dallavilla-Tiziano/omics_portal.git` was reverified at
|
||||
`fca10901a73666ca257d8f4cc4b77066295c400a` (functional commit `95154e1`).
|
||||
The server operator integrates it with the current code in the confirmed Omics
|
||||
checkout, normally `/home/chirone/omics_portal`, preserving later server changes.
|
||||
`docs/operations/server-codex-handoff.md` is the authoritative ordered procedure:
|
||||
inventory, source verification, native CLI and installation projections,
|
||||
embedded/upstream auth, Omics templates/static assets/proxy, coordinated rollout,
|
||||
acceptance and rollback. Server deployment and real IdP acceptance remain pending.
|
||||
|
||||
## Current product shape
|
||||
|
||||
### Shared Memory/Evidence typography — 2026-09-13
|
||||
|
||||
Both detail readers now share Manrope and fixed reading roles: 24px card title,
|
||||
20px section headings, 16px/1.65 narrative text, and 14px metadata/code. Authored
|
||||
Markdown subheadings remain subordinate to field headings; inline/fenced code no
|
||||
longer shrinks cumulatively. Full-width layout, paragraph separation and isolated
|
||||
FAKE examples are retained. All 762 frontend tests and 18 browser scenarios pass,
|
||||
including computed typography checks at 390/1280/2400px; typecheck, i18n and Docker
|
||||
build pass. Only Mac frontend was recreated: `87fca0dc5e19`, image `19f1704b6bb2`,
|
||||
healthy. Core and data services are unchanged. Rollback image:
|
||||
`thothii-frontend:before-knowledge-typography-20260913`. See
|
||||
`docs/reports/2026-09-13-knowledge-typography.md`. No PSD/Omics deployment.
|
||||
|
||||
### Knowledge reading and isolated fake Memory examples — 2026-09-13
|
||||
|
||||
Memory/Evidence details now use all available width, with display-only paragraph
|
||||
splitting for long plain prose and preserved code/Markdown/source data. Evidence
|
||||
provenance renders Markdown; copy controls are accessible two-sheet icons.
|
||||
Memory's separate Formatting examples section contains four FAKE cards (one per
|
||||
family), automatically expanded for an empty archive. These client-side examples
|
||||
cannot be edited/saved/indexed and are never submitted to the model or Memory API.
|
||||
No real archive content was changed. 762 tests, 18 browser scenarios, typecheck,
|
||||
i18n and Docker build pass. Only the Mac frontend was recreated: `fc4286b5406d`,
|
||||
image `cbe0fe0dce55`, healthy; other services unchanged. Rollback image:
|
||||
`thothii-frontend:before-knowledge-reading-20260913`. Details in
|
||||
`docs/reports/2026-09-13-knowledge-reading.md`; no PSD/Omics deployment.
|
||||
|
||||
### Unified Session entry and complete tab borders — 2026-09-13
|
||||
|
||||
The sidebar has one Session/Sessione button: return to the current unfinished
|
||||
session (including pending creation) without resetting/reconnecting it; otherwise
|
||||
prepare a new question with existing readiness and dirty-edit guards. Creation
|
||||
still requires submitting a question. A cold document panel is not a running session.
|
||||
Session tabs now have 11px horizontal/3px vertical padding, at least 38px height,
|
||||
and matching 1px borders on every side (gray inactive, red active), with no shared
|
||||
baseline or overlapping bottom border. This supersedes the earlier border removal.
|
||||
757 frontend tests, 15 browser scenarios, typecheck, i18n and Docker build pass.
|
||||
Mac frontend `2844778d311c`, image `d58dccfee3f9`, is healthy; only that service
|
||||
was recreated. Core/data services are unchanged, with no Omics or server deploy.
|
||||
Rollback image: `thothii-frontend:before-session-navigation-20260913`.
|
||||
|
||||
### Login copy and language-selector focus — 2026-09-13
|
||||
|
||||
Removed the redundant login eyebrow/icon and installation-account explanation;
|
||||
the main sign-in heading remains. The full-header language select no longer
|
||||
shows an outer focus ring after pointer interaction; keyboard focus remains
|
||||
visible, including after returning with Tab. EN/IT login and both themes are
|
||||
covered by 36 targeted tests and 14 browser scenarios; typecheck/build pass.
|
||||
Only the Mac frontend was rebuilt/recreated: `204efd40d3eb`, image `7dd0752ff823`,
|
||||
healthy. Other containers are unchanged. Rollback image:
|
||||
`thothii-frontend:before-login-focus-20260913`. No production deployment.
|
||||
|
||||
### Full-header and layout refinements — 2026-09-13
|
||||
|
||||
The owner's four visual adjustments are implemented: full header uses Omics
|
||||
`#CB333B` in both themes with a light complete wordmark/controls; Database status
|
||||
has 16px clearance below its top divider; workspace/model/Done share a compact
|
||||
desktop row; session-scope tabs have no bottom border. Embedded has no extra header.
|
||||
Verified with 71 targeted frontend tests, 11 browser scenarios, typecheck and
|
||||
Docker production build. The Mac frontend was recreated alone and is healthy
|
||||
(`8a1608ad7fc6`, image `a2f489ebe9de`); Core/data-service containers are unchanged.
|
||||
Full/en is retained. Rollback image: `thothii-frontend:before-header-layout-20260913`.
|
||||
See `docs/reports/2026-09-13-header-layout-refinements.md` for validation and rollback.
|
||||
|
||||
### Visual review integrated with the full/embedded shell
|
||||
|
||||
At the owner's request, the seven commits through `a59624a6` from
|
||||
`codex/ui-visual-review` are integrated with the shell/i18n work in this checkout,
|
||||
`codex/prototype-administration-pages`. Both branches started at `2d1b714e`; the
|
||||
first full-shell build omitted that lateral branch and regressed the installed UI.
|
||||
The integrated source retains bundled Manrope, shared type roles, catalog/context
|
||||
alignment, wordmark sizes and composer autosizing alongside full/embedded and EN/IT.
|
||||
|
||||
Local Docker was updated at 12:54 UTC on 2026-09-13: frontend image `605a6e6bba68`,
|
||||
healthy; Core and all data-service containers were unchanged. Mac remains full/en.
|
||||
The immediate pre-merge frontend is retained as
|
||||
`thothii-frontend:before-visual-shell-merge-20260913`.
|
||||
|
||||
Verification: 755 frontend tests, 11 Playwright visual/interaction scenarios at five
|
||||
widths, translation-catalog checks, typecheck and production build. See
|
||||
`docs/reports/2026-09-13-visual-shell-integration.md` for delivery, provenance and
|
||||
rollback details. The separate visual-review worktree and its rollback images are
|
||||
retained. No merge to `main`, push or production deployment is part of this delivery.
|
||||
|
||||
Gitea #28–#31 follow-up is implemented in this checkout: Workspace's four tabs,
|
||||
Database list-first entry without the preparation footer, full-height Pi instructions
|
||||
with installation-host OS selection, and short Admin navigation labels. Core and
|
||||
session behavior are retained. At the owner's request, these changes were rebuilt into
|
||||
local Docker on 2026-09-12 at 18:31 UTC. Core/frontend are healthy and the UI at
|
||||
`http://127.0.0.1:8080` serves the updated bundle. The host projection now supplies
|
||||
`THT_HOST_PLATFORM=darwin`, so Pi selects macOS rather than the container's Linux OS.
|
||||
Persistent dependency containers and volumes were unchanged. Rollback images are tagged
|
||||
`thothii-core:before-admin-28-31-20260912` and
|
||||
`thothii-frontend:before-admin-28-31-20260912`. See
|
||||
`docs/reports/2026-09-12-admin-issues-28-31.md` for verification and deployment details.
|
||||
|
||||
The latest context-shelf A and five Administration pages are implemented locally.
|
||||
At the owner's request, local Docker project `thothii-18998cca7b0a` was rebuilt and
|
||||
its core/frontend recreated on 2026-09-12. The real UI at `http://127.0.0.1:8080`
|
||||
serves shelf A; both services and their existing dependencies are healthy.
|
||||
Runtime model projections were regenerated as schema v2 with the single
|
||||
`zai/glm-5.3` interaction default. Persistent services/volumes were not recreated.
|
||||
The local launcher `/private/tmp/thothii-memory-preview.sh` now adds
|
||||
`/private/tmp/thothii-context-a.compose.yaml` last, building from this checkout
|
||||
instead of the earlier Memory worktree. Previous images are retained under
|
||||
`thothii-core:before-context-a-20260912` and
|
||||
`thothii-frontend:before-context-a-20260912`. Remote server deployment remains pending.
|
||||
Core/session behavior is retained; one independently remembered workspace/model
|
||||
pair controls both Core and Admin. Unsaved Admin changes block navigation and
|
||||
context changes; operation activity locks the selectors. See
|
||||
`docs/reports/2026-09-12-context-shelf-a-implementation.md` for verification and the
|
||||
remaining server/Omics integration gate. Prototype alternatives remain untouched.
|
||||
|
||||
ThothII is a human-in-the-loop datamart builder with three independently built layers:
|
||||
|
||||
```text
|
||||
@@ -26,6 +240,165 @@ metadata catalog for administrative database configuration. The frontend renders
|
||||
and keeps the live transcript in memory. See
|
||||
`docs/architecture/components.md` for the detailed component and data-flow map.
|
||||
|
||||
## Memory M1–M3, Evidence E1–E3 and joint workflow repair X1 implemented
|
||||
|
||||
Browser authorization now derives permissions from validated session roles using the current
|
||||
catalog. This fixes Memory/Evidence navigation remaining disabled for administrators whose
|
||||
remembered login predates those permissions; refreshing the page loads the updated permissions.
|
||||
|
||||
The owner requested two independent administration projects: Memory management and Evidence
|
||||
management. Their navigation entries must sit immediately after Database management, as peers;
|
||||
neither page belongs to Database management. Both require complete browsing, filtering, and CRUD
|
||||
without an active core session. Shared requirements and the two project briefs are linked from
|
||||
`docs/plans/2026-09-08-memory-evidence-administration.md`. Memory administration is implemented;
|
||||
Evidence administration is implemented through external file editing and explicit consolidation.
|
||||
Evidence editing requires an explicit evolution of the current authoring/publication contract.
|
||||
The M1 specification is at `docs/plans/2026-09-08-memory-m1-spec.md` and published as
|
||||
[M1 — Archivio autorevole e amministrazione delle Memory Card](https://git.tylconsulting.it/mptyl/ThothII/issues/27)
|
||||
with the `ready-for-agent` and `enhancement` labels.
|
||||
It covers the authoritative store, administrative CRUD, projection recovery, and current
|
||||
Memory/exemplar producer integration. Its accepted test boundaries are the public harness
|
||||
service, Fastify APIs, and the AppShell page, joined by a focused real-stack browser path.
|
||||
The owner confirmed those boundaries and authorized publication on 2026-09-08.
|
||||
M1 is implemented locally: PostgreSQL authority in `thoth_memory`, versioned installation
|
||||
migrations, admin CRUD for all four families, structured dependencies and links, explicit
|
||||
projection recovery, verified recall and current workflow producers. The page requires no
|
||||
active session or DWH binding. The existing `catalog-migrate` preparation service now runs
|
||||
Memory migrations as well. Existing installations need that preparation before using M1;
|
||||
the initial implementation did not deploy or migrate the owner's stacks.
|
||||
The integrated browser check passed with real authentication, Fastify, ThtRunner, harness,
|
||||
PostgreSQL and Qdrant; embeddings were deterministic and unrelated Pi/session activity used
|
||||
test fixtures. See `docs/plans/2026-09-08-memory-m1-validation.md` for results and commands.
|
||||
M2 is implemented locally: Memory dense/BM25 fusion, physical and business scope filters,
|
||||
bounded outgoing-link expansion and joint ranking, all resolved against current PostgreSQL
|
||||
authority. Migration `002_hybrid_projection.sql` makes old dense projections pending until
|
||||
explicit retry/rebuild; Reference remains separate. The real retrieval check uses the configured
|
||||
`qwen3-embedding:0.6b` model, a separate Ollama process with a read-only model-volume mount,
|
||||
and isolated PostgreSQL/Qdrant resources. See `docs/plans/2026-09-09-memory-m2-validation.md`.
|
||||
M3 is implemented: editable F8 summary grounded in effective approved decisions, explicit
|
||||
updates with concurrent-edit protection, selected-card/link transactions and durable review
|
||||
receipts. Finalization no longer saves exemplars implicitly. SQL rules and explained errors
|
||||
are consulted in the existing F4/F6/F7 gates. Successful Catalog physical synchronization
|
||||
performs dependency cleanup, preserving the original removals for recovery across restarts.
|
||||
Migration `003_review_receipts.sql` is required. Validation includes a real GLM 5.3 generation
|
||||
case against synthetic PostgreSQL data; see `docs/plans/2026-09-09-memory-m3-validation.md`.
|
||||
Evidence administration and the joint X1 conflict-repair increment are now implemented.
|
||||
The same local preview was subsequently rebuilt with M3 and migration 003 applied.
|
||||
Core and frontend now include the final Memory review and physical dependency cleanup.
|
||||
On 2026-09-09, at the owner's request, the local PSD Docker installation
|
||||
`thothii-18998cca7b0a` was updated from this worktree. Core/frontend images were rebuilt,
|
||||
Catalog and Memory migrations completed, and all five services became healthy. The UI is at
|
||||
`http://127.0.0.1:8080`, using the existing local authentication and persistent volumes.
|
||||
The `psd-clinical` authoritative Memory archive is initially empty (no legacy import).
|
||||
The existing installation configuration remains in `/Users/mp/projects/ThothII/deploy/psd/`;
|
||||
`/private/tmp/thothii-memory-preview.sh` invokes its Compose files with a final build-context
|
||||
and migration-command override from this worktree. A future build from the main checkout
|
||||
will use that checkout's code, so retain the worktree override until the changes are integrated.
|
||||
Memory revision history and compatibility with existing development sessions are not requirements.
|
||||
The agreed Memory scope includes reusable domain clarifications, SQL construction rules, solved
|
||||
questions, and explained, approved mistakes to avoid. This extends the current runtime's
|
||||
`concept_clarified`-only reusable Memory contract. The owner also approved an editable final summary
|
||||
for proposed additions/updates and Memory consumption in the relevant existing review gates.
|
||||
Further agreed behavior includes persistent corrections for Memory/Evidence conflicts through
|
||||
explicit choices, deletion of Memory with invalid dependencies after successful physical schema
|
||||
synchronization, and bounded functional/regression tests instead of a general quality benchmark.
|
||||
In-house Memory evolution is approved, including hybrid Qdrant retrieval and explicit card links
|
||||
traversed in core, without a dedicated graph database. Both capabilities belong to the current
|
||||
scope. PostgreSQL is the agreed authority for Memory Cards, links, and schema dependencies;
|
||||
Qdrant is a rebuildable index. Links are reviewed with cards and editable in Administration;
|
||||
deleting a card removes its incident links while preserving the other cards. The accepted,
|
||||
implemented architecture is recorded in
|
||||
`docs/adr/0018-use-postgres-for-memory-and-qdrant-for-retrieval.md`.
|
||||
For Evidence administration, Q12 rejects a separate editorial publication workflow.
|
||||
The final R0 direction uses external editors and a manual consolidation command that
|
||||
validates structure, reports required corrections, updates derived metadata, and
|
||||
activates the local Evidence index. The operator then checks the diff and runs Git
|
||||
commit/push manually. No watcher, automated Git, or web editor is required.
|
||||
The owner's later simplification
|
||||
instruction removes the requirement to support administrative edits while core work
|
||||
is in progress. Deliberate corrections made by the workflow itself remain supported.
|
||||
The context specialist writes Evidence drafts independently of the installation and
|
||||
without PostgreSQL access; the system refines them and stores them locally for use
|
||||
and maintenance. The revised Q11 recommendation separates external drafts from a
|
||||
durable local canonical file archive, removing automatic commit/push from CRUD.
|
||||
The owner has now accepted local files and requires discoverable, editable Markdown
|
||||
for nontechnical domain specialists, with no JSONL management surface. Editors on
|
||||
Mac/PC or vim/nano on the server are the chosen R0 editing surface. Evidence
|
||||
management keeps browsing/filtering/detail and clearly identifies the persistent
|
||||
working tree, each Markdown file's actual host path, and the manual commands.
|
||||
E1 implements editable Curated Evidence v4, deterministic legacy conversion, persistent
|
||||
local files and a consolidation API with immutable candidates, manual provenance,
|
||||
deletion records and recoverable activation. E2 connects its installed command, runtime source
|
||||
selection and administration page. The operator completes Git steps manually.
|
||||
All 35 PSD units were converted on an isolated copy with identical IDs and typed content.
|
||||
Tests exercise visible edits through normalization, indexing and recall with real Qdrant;
|
||||
see `docs/plans/2026-09-09-evidence-e1-validation.md` and
|
||||
`docs/contracts/curated-evidence-v4.md`. E2 is now running on the local Docker preview:
|
||||
all 35 PSD units were converted and indexed through the installed command. The editable
|
||||
host archive is `/Users/mp/projects/ThothII/deploy/psd/evidence-registry/repo/psd-clinical/evidence`.
|
||||
The original registry-volume checkout and original author repository were retained.
|
||||
The installation descriptor now includes `workspace-bindings.yaml` and `evidence-host.yaml`
|
||||
so native maintenance uses the same volumes and checkout as the preview. Descriptor backup:
|
||||
`/private/tmp/thothii-installation-before-e2.yaml`; previous native binary: `/private/tmp/tht-before-e2`.
|
||||
The launcher remains `bash /private/tmp/thothii-memory-preview.sh`; its worktree build override
|
||||
is still required until integration. Runtime lease filenames now include rendered bytes so
|
||||
upgrading the Evidence renderer does not collide with old immutable configs; Catalog
|
||||
input fingerprints and readiness are unchanged. See `docs/plans/2026-09-09-evidence-e2-validation.md`.
|
||||
E3 adds explicit source acquisition/refinement and durable comparisons in Evidence management.
|
||||
Local drafts go in `evidence/incoming/`; original local documents and configured HTTP/S3
|
||||
sources are reacquired only by **Import or refresh sources** or installed
|
||||
`tht workspace evidence refresh --workspace <id>`. Keep/replace decisions, including the
|
||||
native `workspace evidence decide` command, activate through the existing archive/index
|
||||
path and have durable retry state. Acquired source versions and remote provenance stay
|
||||
local; old documentary lineage can coexist with current manual declarations.
|
||||
The installed preview's 35 PSD sources were verified unchanged with no new proposals.
|
||||
The previous native CLI is backed up at `/private/tmp/tht-before-e3`.
|
||||
See `docs/plans/2026-09-09-evidence-e3-validation.md` for tests and validation limits.
|
||||
X1 adds `reviewer_archive_repair`: closed alternatives with complete before/after content,
|
||||
explicit rejection/reformulation, admin-only application, durable session receipts and
|
||||
retry of saved but inactive corrections. The coordinator uses the canonical Memory and
|
||||
Evidence services; phase approval remains separate. Migration `004_archive_repairs.sql`
|
||||
is required. See `docs/contracts/archive-repair.md` for authorization and recovery limits.
|
||||
X1 validation is recorded in `docs/plans/2026-09-09-archive-repair-x1-validation.md`:
|
||||
both archives were corrected and retrieved through the actual CLI with real PostgreSQL
|
||||
and Qdrant; desktop/mobile widget behavior and permission failures were verified separately.
|
||||
The local preview images include X1 and migration 004 is applied. Follow-up technical
|
||||
acceptance passed with configured GLM 5.3 generating closed conflict alternatives and
|
||||
the chosen Memory correction persisted and retrieved. An authenticated browser test
|
||||
also verified both administration pages, navigation order, Evidence filtering, workspace
|
||||
404s, desktop/mobile layout, and Evidence survival after Memory deletion. All approved
|
||||
technical increments/checks are complete. A real PSD domain-conflict session remains
|
||||
the reviewer's semantic acceptance check; automated cases did not modify PSD knowledge.
|
||||
The joint browser inspection also fixed mobile archive navigation: below 768px,
|
||||
Memory and Evidence keep the full content width and open navigation in the shared
|
||||
accessible dialog. Desktop retains its sidebar; the local frontend image includes this fix.
|
||||
The owner accepted the remaining simplifications and requested explicit clarification
|
||||
of the core format change and the simple terminal-based Git check. E1 must adapt
|
||||
the Evidence parser, renderer, authoring, validation, and normalization; convert
|
||||
existing files and reindex; and verify that visible edits reach core consumption.
|
||||
Preserve the internal typed model where possible. This precedes the administrative
|
||||
page and is not merely a presentation change. Human inspection uses normal Git
|
||||
status and optional line-level diff commands; no custom diff viewer or mandatory
|
||||
double review is required.
|
||||
The suggestion of making PostgreSQL the Evidence authority was withdrawn after this
|
||||
clarification; it was never implemented or accepted as a replacement for Q11.
|
||||
Q13 keeps manual corrections active when updated sources contradict them, until an
|
||||
administrator resolves the comparison; deleted Evidence must not be regenerated
|
||||
automatically. Q14 allows direct manual creation and records a manual declaration
|
||||
as the current source, preserving any original document as distinct provenance.
|
||||
Q15 refreshes external sources only on explicit administrator request; normal saves
|
||||
and lookups do not reacquire them. These decisions, now implemented through E1–E3, are recorded in
|
||||
`docs/adr/0019-author-evidence-in-app-with-automatic-activation.md`.
|
||||
The decision-by-decision review is recorded in
|
||||
`docs/plans/2026-09-08-memory-evidence-simplification-review.md`; it distinguishes the
|
||||
new constraints from the revised technical recommendations. It also specifies a
|
||||
sequential save with minimal durable retry state and direct Memory cleanup after
|
||||
successful schema synchronization, without new queues or event infrastructure.
|
||||
The delivery order remains Memory, Evidence, and persistent conflict repair between
|
||||
both modules. Local curated Evidence must survive preprocessing Clear and be backed
|
||||
up as primary data; the Qdrant projection remains rebuildable.
|
||||
No runtime gate change or administrative page is implemented yet.
|
||||
|
||||
## Evidence restructuring — accepted
|
||||
|
||||
The evidence restructuring and PSD migration completed real acceptance on 2026-08-25.
|
||||
@@ -68,7 +441,8 @@ schema, Evidence, and vector mutation commands are retired.
|
||||
|
||||
The right Administration sidebar invokes that same operation for the selected workspace. It shows
|
||||
only current readiness or the latest bounded failure diagnostic; there is no preprocessing history.
|
||||
Known non-ready state disables **New session**, while backend admission remains authoritative.
|
||||
Known non-ready state disables **Session** when it would start a new question,
|
||||
while backend admission remains authoritative.
|
||||
The same control exposes an inline-confirmed **Clear** action to remove replaceable reference
|
||||
vectors, LSH, corpus, and checkpoints while preserving the separate Memory collection. The host CLI
|
||||
equivalent is `workspace preprocess clear`.
|
||||
@@ -93,7 +467,28 @@ regenerates the backend catalog, Pi `models.json`/`settings.json`, and Compose o
|
||||
installation-local `generated/` directory. Those projections are replaceable runtime adapters:
|
||||
they are not edited, backed up, or treated as configuration.
|
||||
|
||||
Session and metadata defaults use canonical `provider/model` IDs. Provider authentication declares
|
||||
Core and Administration now share one canonical `modelCatalog.defaults.interaction` per installation,
|
||||
independent of workspace. Runtime catalog schema v2 contains only `defaultInteraction`; apply the host,
|
||||
backend, and regenerated projections together. Equal legacy defaults normalize on read; conflicting
|
||||
ones require an explicit operator choice. With Admin AI configured, selectable models are the
|
||||
intersection of the Pi and LiteLLM adapters; Core-only installations remain supported without Admin AI.
|
||||
The existing Core and Database controls share the operational selection. Explicit model choices are
|
||||
remembered per authenticated user/application mount in this browser, not in installation settings.
|
||||
Resume uses that selection/default while preserving the historical workspace/revision and manifest.
|
||||
Every model must be manually exercised in both Core and Admin as documented in
|
||||
`docs/general/pi-configuration.md`; validation is not a live model certification.
|
||||
These changes and shelf A are now deployed to local Docker; remote deployment remains pending.
|
||||
|
||||
PSD DeepSeek Pro/Flash now share canonical `deepseek/...` identities across native Pi and LiteLLM,
|
||||
using the existing `DEEPSEEK_API_KEY` bundle entry. Its value was confirmed identical to the working
|
||||
Pi key without exposing it; no secret files were changed. The duplicate `deepseek-metadata` descriptor
|
||||
is removed locally and the tracked example uses the shared provider. `secret_env` overrides legacy
|
||||
Pi auth only inside temporary runtime snapshots and fails closed if the bundle key is missing.
|
||||
The original auth store/history and `zai/glm-5.3` default are preserved. Local Docker projections
|
||||
and core/frontend were updated together on 2026-09-12. Regenerate projections with
|
||||
the matching release for the separate server deployment.
|
||||
|
||||
Provider authentication declares
|
||||
one explicit mode (`secret_env`, `pi_auth`, or `none`); `secret_env` names a protected bundle key.
|
||||
The backend settings store now owns only the selected workspace and thinking level. Existing v1
|
||||
installations use the explicit catalog migration command; schema-v3 workspace descriptors are
|
||||
|
||||
@@ -4,8 +4,26 @@ ThothII is a human-reviewed NL-to-SQL workflow with a React frontend and a Fasti
|
||||
core. The portable deployment runs two application services plus the installation-local metadata
|
||||
catalog; DWH and LLM services remain external. Semantic services are bundled in Compose.
|
||||
|
||||
Authentication is configured through the single host CLI tht: see the [local authentication guide](docs/install/authentication-local.md),
|
||||
[generic OIDC guide](docs/install/authentication-oidc.md), and [manual acceptance matrix](docs/testing/authentication-manual-acceptance.md).
|
||||
The same frontend supports **full** (its own header) and **embedded** (inside a
|
||||
portal). This choice is independent of authentication: the Mac uses full/local,
|
||||
Omics uses embedded/upstream with its existing login, and a standalone server
|
||||
can use full/OIDC. See [rendering architecture](docs/architecture/application-shell.md)
|
||||
and [configuration, Omics delivery and deploy](docs/operations/shell-and-localization.md).
|
||||
|
||||
For the current server upgrade with Omics Portal, follow the ordered
|
||||
[Codex server handoff](docs/operations/server-codex-handoff.md), including source
|
||||
integration, embedded/upstream configuration, coordinated rollout and rollback.
|
||||
|
||||
Local/OIDC authentication is configured through the host CLI `tht`; portal
|
||||
authentication is established by the trusted server proxy. See the
|
||||
[local guide](docs/install/authentication-local.md),
|
||||
[OIDC guide](docs/install/authentication-oidc.md),
|
||||
[upstream integration](docs/install/authentication-upstream.md), and
|
||||
[manual acceptance matrix](docs/testing/authentication-manual-acceptance.md).
|
||||
|
||||
For the clone-based manual standalone installation test on macOS, Windows, and Linux, use the
|
||||
[Italian procedure](docs/install/standalone-manual-it.md) or the
|
||||
[English procedure](docs/install/standalone-manual-en.md).
|
||||
|
||||
## Docker Compose: local startup
|
||||
|
||||
@@ -23,7 +41,7 @@ cp deploy/env/local.env.example deploy/env/local.env
|
||||
./scripts/run-stack.sh
|
||||
```
|
||||
|
||||
The launcher builds the core, starts `catalog-db`, runs the explicit one-shot Kysely migrations,
|
||||
The low-level stack script builds the core, starts `catalog-db`, runs the explicit one-shot Kysely migrations,
|
||||
then runs the base+local stack in the foreground. Migrations never run implicitly in backend
|
||||
startup. The core image contains its Pi runtime; no host `pi` executable is used. For a server
|
||||
installation, build the image, start the catalog, and run the same migration service before the
|
||||
|
||||
+18
-1
@@ -7,6 +7,7 @@ import { fileURLToPath } from "node:url";
|
||||
import { tmpdir } from "node:os";
|
||||
import type { AppConfig } from "./config.js";
|
||||
import { ThtRunner } from "./tht/tht-runner.js";
|
||||
import { createMemoryCleanup } from "./catalog/memory-cleanup.js";
|
||||
import { PiProcessManager } from "./pi/pi-process-manager.js";
|
||||
import { SseHub } from "./sse/sse-hub.js";
|
||||
import { authenticateSession, captureAuthConfigSnapshot, configuredOrigin } from "./auth/auth.js";
|
||||
@@ -80,6 +81,8 @@ import { createProductionWorkspacePreprocessingService } from "./workspace-maint
|
||||
import type { WorkspacePreprocessingService } from "./workspaces/preprocessing-service.js";
|
||||
import { PreprocessingStateStore } from "./workspaces/preprocessing-state.js";
|
||||
import { workspacePreprocessingRoutes } from "./routes/workspace-preprocessing.js";
|
||||
import { memoryRoutes } from "./routes/memory.js";
|
||||
import { evidenceRoutes } from "./routes/evidence.js";
|
||||
|
||||
export interface BuildAppDeps {
|
||||
thtRunner?: ThtRunner;
|
||||
@@ -93,7 +96,7 @@ export interface BuildAppDeps {
|
||||
workspaceDiagnoser?: WorkspaceDiagnoser;
|
||||
workspaceDatabaseTester?: WorkspaceDatabaseTester;
|
||||
workspaceSecretStore?: WorkspaceSecretStore;
|
||||
workspacePreprocessingService?: Pick<WorkspacePreprocessingService, "run" | "clear">;
|
||||
workspacePreprocessingService?: Pick<WorkspacePreprocessingService, "run" | "clear"> & Partial<Pick<WorkspacePreprocessingService, "consolidateEvidence" | "evidenceSources">>;
|
||||
catalogRepository?: CatalogRepository;
|
||||
catalogService?: CatalogService;
|
||||
catalogPostgresAccess?: CatalogPostgresAccess;
|
||||
@@ -265,6 +268,11 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
|
||||
catalogSchemaIntrospector,
|
||||
catalogOperationCoordinator,
|
||||
config.catalogSyncTimeoutMs,
|
||||
createMemoryCleanup(tht as ThtRunner, {
|
||||
internalQdrantUrl: config.internalQdrantUrl, internalEmbeddingUrl: config.internalEmbeddingUrl,
|
||||
internalEmbeddingId: config.internalEmbeddingId, internalEmbeddingModel: config.internalEmbeddingModel,
|
||||
internalEmbeddingDimensions: config.internalEmbeddingDimensions,
|
||||
}),
|
||||
);
|
||||
app.addHook("onReady", async () => { await catalogSyncWorker.initialize(); });
|
||||
app.addHook("onReady", async () => { await descriptionGenerationWorker.initialize(); });
|
||||
@@ -493,7 +501,16 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
|
||||
return maintenanceBarrier.status();
|
||||
});
|
||||
sqlRoutes(app, { tht: tht as ThtRunner, getSettings, workspaceRegistry });
|
||||
memoryRoutes(app, { runner: tht as ThtRunner, registry: workspaceRegistry, runtime: {
|
||||
internalQdrantUrl: config.internalQdrantUrl,
|
||||
internalEmbeddingUrl: config.internalEmbeddingUrl,
|
||||
internalEmbeddingModel: config.internalEmbeddingModel,
|
||||
internalEmbeddingDimensions: config.internalEmbeddingDimensions,
|
||||
} });
|
||||
metaRoutes(app, { harnessDir: config.harnessDir, modelCatalog: runtimeModelCatalog });
|
||||
evidenceRoutes(app, { runner: tht as ThtRunner, registry: workspaceRegistry,
|
||||
registryRoot: config.workspaceRegistry.root, hostRegistryRoot: config.evidenceHostRegistryRoot,
|
||||
service: workspacePreprocessingService });
|
||||
workspaceRoutes(app, {
|
||||
registry: workspaceRegistry,
|
||||
config: config.workspaceRegistry,
|
||||
|
||||
@@ -119,7 +119,9 @@ export function authenticateSession(deps: AuthDependencies): preHandlerHookHandl
|
||||
subject: session.subject,
|
||||
...(session.displayName === undefined ? {} : { displayName: session.displayName }),
|
||||
roles: session.roles,
|
||||
permissions: session.permissions,
|
||||
// Sessions can outlive a deployment that changes the role permission catalog.
|
||||
// resolve() has already checked validity, including current local user roles.
|
||||
permissions: rolesToPermissions(session.roles),
|
||||
isAdmin: session.roles.includes("admin"),
|
||||
};
|
||||
if (STATE_CHANGING_METHODS.has(request.method)) {
|
||||
|
||||
@@ -38,7 +38,7 @@ const MAX_MAPPED_GROUPS = 128;
|
||||
const ROLES = ["user", "admin"] as const;
|
||||
export const PERMISSION_CATALOG: readonly Permission[] = [
|
||||
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
||||
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
|
||||
"workspace.manage", "workspace.secrets.manage", "database.manage", "memory.manage", "evidence.manage", "pi.manage", "auth.diagnostics.read",
|
||||
];
|
||||
|
||||
const invalid = (): Error => new Error("authentication configuration is invalid");
|
||||
|
||||
@@ -33,7 +33,7 @@ const EMPTY_HKDF_SALT = Buffer.alloc(0);
|
||||
const ROLES = ["user", "admin"] as const;
|
||||
const PERMISSIONS = [
|
||||
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
||||
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
|
||||
"workspace.manage", "workspace.secrets.manage", "database.manage", "memory.manage", "evidence.manage", "pi.manage", "auth.diagnostics.read",
|
||||
] as const satisfies readonly Permission[];
|
||||
|
||||
const invalid = (): Error => new Error("auth_session_store_invalid");
|
||||
|
||||
@@ -5,7 +5,7 @@ export type Role = "user" | "admin";
|
||||
export type Permission =
|
||||
| "session.use" | "session.read_all" | "session.manage_all"
|
||||
| "settings.manage" | "workspace.manage" | "workspace.secrets.manage"
|
||||
| "database.manage" | "pi.manage" | "auth.diagnostics.read";
|
||||
| "database.manage" | "memory.manage" | "evidence.manage" | "pi.manage" | "auth.diagnostics.read";
|
||||
|
||||
export interface AuthenticationSessionConfig {
|
||||
regularTtlSeconds: number;
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
import type { ThtRunner } from "../tht/tht-runner.js";
|
||||
import type { SemanticRuntimeConfig } from "../workspaces/runtime-renderer.js";
|
||||
import type { CatalogSyncRun, WorkspaceDatabase } from "./types.js";
|
||||
|
||||
/** Internal continuation of an applied physical sync, using the harness Memory boundary. */
|
||||
export function createMemoryCleanup(runner: Pick<ThtRunner, "withPrincipal">, runtime: SemanticRuntimeConfig) {
|
||||
return async (database: WorkspaceDatabase, run: CatalogSyncRun): Promise<number> => {
|
||||
if (run.phase !== "memory_cleanup" || !run.plannedDiff) throw new Error("Physical cleanup is not committed");
|
||||
const result = await runner.withPrincipal({ issuer: "installation", subject: "catalog-sync",
|
||||
roles: ["admin"], permissions: ["memory.manage"], isAdmin: true,
|
||||
}).runWithRuntimeSnapshot(["memory", "admin", "--workspace", database.workspaceId], JSON.stringify({
|
||||
action: "cleanup", runtime, request: { sync_id: run.id, database: database.databaseName,
|
||||
schema_name: database.schema, removed_tables: run.plannedDiff.deletedTables,
|
||||
removed_columns: run.plannedDiff.deletedColumns.map(column => ({ table: column.tableName, column: column.columnName })),
|
||||
},
|
||||
}));
|
||||
const payload = JSON.parse(result.stdout);
|
||||
if (result.code !== 0 || payload.indexed !== true || !Number.isInteger(payload.deleted) || payload.deleted < 0) {
|
||||
throw new Error("Memory cleanup is incomplete");
|
||||
}
|
||||
return payload.deleted;
|
||||
};
|
||||
}
|
||||
@@ -926,6 +926,7 @@ export class MemoryCatalogRepository implements CatalogRepository {
|
||||
scope: CatalogSyncScope,
|
||||
tableIds: readonly string[],
|
||||
snapshot: ObservedSchemaSnapshot,
|
||||
syncRunId?: string,
|
||||
): Promise<CatalogSyncCounts | undefined> {
|
||||
const database = this.records.get(databaseId);
|
||||
if (!database || database.version !== expectedDatabaseVersion) return undefined;
|
||||
@@ -1141,6 +1142,7 @@ export class MemoryCatalogRepository implements CatalogRepository {
|
||||
if (scope === "all") {
|
||||
this.records.set(databaseId, { ...database, schemaSyncedVersion: expectedDatabaseVersion, schemaSyncedAt: now });
|
||||
}
|
||||
if (syncRunId) await this.updateSyncRun(syncRunId, { phase: "memory_cleanup" });
|
||||
return {
|
||||
tables: (await this.listTables(databaseId)).length,
|
||||
columns: [...this.columns.values()].filter((column) => this.tables.get(column.tableId)?.databaseId === databaseId).length,
|
||||
@@ -1254,7 +1256,7 @@ export class MemoryCatalogRepository implements CatalogRepository {
|
||||
for (const run of this.syncRuns.values()) {
|
||||
if (["queued", "running", "awaiting_confirmation", "applying"].includes(run.state)) {
|
||||
await this.updateSyncRun(run.id, {
|
||||
state: "interrupted", phase: "completed", finishedAt: new Date().toISOString(),
|
||||
state: "interrupted", phase: run.phase === "memory_cleanup" ? "memory_cleanup" : "completed", finishedAt: new Date().toISOString(),
|
||||
errorCode: "SYNC_INTERRUPTED", errorMessage: "Synchronization was interrupted by a service restart",
|
||||
});
|
||||
}
|
||||
|
||||
@@ -102,7 +102,7 @@ export function loadMetadataGenerationModels(options: {
|
||||
...(apiKeyEnv ? { apiKeyEnv, apiKey } : {}),
|
||||
}));
|
||||
}
|
||||
const result = new RestartLoadedMetadataGenerationModels(models, catalog.defaultMetadataGeneration);
|
||||
const result = new RestartLoadedMetadataGenerationModels(models, models.size ? catalog.defaultInteraction : null);
|
||||
const safe = result.catalog();
|
||||
return {
|
||||
catalog: () => ({
|
||||
|
||||
@@ -1604,6 +1604,7 @@ export class KyselyCatalogRepository implements CatalogRepository {
|
||||
scope: CatalogSyncScope,
|
||||
tableIds: readonly string[],
|
||||
snapshot: ObservedSchemaSnapshot,
|
||||
syncRunId?: string,
|
||||
): Promise<CatalogSyncCounts | undefined> {
|
||||
return await this.db.transaction().execute(async (trx) => {
|
||||
const database = await trx.selectFrom("workspaceDatabases").select("version")
|
||||
@@ -1779,6 +1780,10 @@ export class KyselyCatalogRepository implements CatalogRepository {
|
||||
schemaSyncedAt: now,
|
||||
}).where("id", "=", databaseId).execute();
|
||||
}
|
||||
if (syncRunId) {
|
||||
await trx.updateTable("catalogSyncRuns").set({ phase: "memory_cleanup" })
|
||||
.where("id", "=", syncRunId).where("databaseId", "=", databaseId).execute();
|
||||
}
|
||||
return {
|
||||
tables: snapshot.tables.length,
|
||||
columns: scope === "tables" ? undefined : snapshot.columns.length,
|
||||
@@ -1882,7 +1887,7 @@ export class KyselyCatalogRepository implements CatalogRepository {
|
||||
|
||||
async interruptActiveSyncRuns(): Promise<void> {
|
||||
await this.db.updateTable("catalogSyncRuns").set({
|
||||
state: "interrupted", phase: "completed", errorCode: "worker_restarted",
|
||||
state: "interrupted", phase: sql`case when phase='memory_cleanup' then phase else 'completed' end`, errorCode: "worker_restarted",
|
||||
errorMessage: "Synchronization was interrupted by a backend restart.",
|
||||
finishedAt: sql`now()`, updatedAt: sql`now()`,
|
||||
leaseOwner: null, leaseExpiresAt: null,
|
||||
|
||||
@@ -58,6 +58,8 @@ export class CatalogSyncWorker {
|
||||
private readonly introspector: CatalogSchemaIntrospector,
|
||||
private readonly operations: CatalogOperationCoordinator,
|
||||
private readonly timeoutMs: number,
|
||||
private readonly cleanupMemory: (database: WorkspaceDatabase, run: CatalogSyncRun) => Promise<number>
|
||||
= async () => 0,
|
||||
) {}
|
||||
|
||||
async initialize(): Promise<void> {
|
||||
@@ -67,6 +69,9 @@ export class CatalogSyncWorker {
|
||||
}
|
||||
|
||||
async start(database: WorkspaceDatabase, scope: CatalogSyncScope, tableIds: readonly string[]): Promise<CatalogSyncRun> {
|
||||
if ((await this.repository.listSyncRuns(database.id, 100)).some(run => run.phase === "memory_cleanup")) {
|
||||
throw new CatalogConflictError("Retry the pending Memory cleanup before starting another synchronization");
|
||||
}
|
||||
const uniqueTableIds = [...new Set(tableIds)];
|
||||
if (scope === "columns") {
|
||||
const tables = await Promise.all(uniqueTableIds.map((tableId) => this.repository.getTable(database.id, tableId)));
|
||||
@@ -109,7 +114,7 @@ export class CatalogSyncWorker {
|
||||
async cancel(runId: string): Promise<CatalogSyncRun | undefined> {
|
||||
const run = await this.repository.getSyncRun(runId);
|
||||
if (!run) return undefined;
|
||||
if (run.state === "applying" || TERMINAL_STATES.has(run.state)) return run;
|
||||
if (run.phase === "memory_cleanup" || run.state === "applying" || TERMINAL_STATES.has(run.state)) return run;
|
||||
await this.repository.requestSyncRunCancellation(runId);
|
||||
this.controllers.get(runId)?.abort();
|
||||
if (run.state === "queued" || run.state === "awaiting_confirmation") {
|
||||
@@ -138,6 +143,16 @@ export class CatalogSyncWorker {
|
||||
}
|
||||
const database = await this.repository.get(previous.databaseId);
|
||||
if (!database) return undefined;
|
||||
if (previous.phase === "memory_cleanup") {
|
||||
const release = this.operations.reserve(database.id);
|
||||
try {
|
||||
const queued = await this.repository.updateSyncRun(runId, { state: "queued", cancelRequested: false,
|
||||
errorCode: null, errorMessage: null, finishedAt: null, leaseOwner: null, leaseExpiresAt: null });
|
||||
this.reservations.set(runId, release);
|
||||
this.launch(runId);
|
||||
return queued;
|
||||
} catch (error) { release(); throw error; }
|
||||
}
|
||||
return await this.start(database, previous.scope, previous.tableIds);
|
||||
}
|
||||
|
||||
@@ -179,6 +194,10 @@ export class CatalogSyncWorker {
|
||||
if (!database || database.version !== claimed.requestedDatabaseVersion) {
|
||||
throw new CatalogConflictError("Database binding changed before synchronization started");
|
||||
}
|
||||
if (claimed.phase === "memory_cleanup") {
|
||||
await this.finishMemoryCleanup(database, claimed);
|
||||
return;
|
||||
}
|
||||
const progress: CatalogSchemaScanProgress = async (phase, counts) => {
|
||||
await this.checkCancelled(runId);
|
||||
await this.repository.updateSyncRun(runId, {
|
||||
@@ -230,36 +249,30 @@ export class CatalogSyncWorker {
|
||||
claimed.scope,
|
||||
claimed.tableIds,
|
||||
snapshot,
|
||||
runId,
|
||||
);
|
||||
if (!applied) throw new CatalogConflictError("Database binding changed before schema changes were applied");
|
||||
await this.repository.updateSyncRun(runId, {
|
||||
state: "succeeded",
|
||||
phase: "completed",
|
||||
counts: applied,
|
||||
finishedAt: new Date().toISOString(),
|
||||
observedSnapshot: null,
|
||||
plannedDiff: null,
|
||||
confirmationToken: null,
|
||||
heartbeatAt: new Date().toISOString(),
|
||||
leaseOwner: null,
|
||||
leaseExpiresAt: null,
|
||||
});
|
||||
await this.repository.appendSyncEvent(runId, "info", "succeeded", "Synchronization completed.", { ...applied });
|
||||
this.release(runId);
|
||||
const cleanupRun = await this.repository.updateSyncRun(runId, { counts: applied });
|
||||
if (!cleanupRun) throw new Error("Synchronization run disappeared");
|
||||
await this.finishMemoryCleanup(database, cleanupRun);
|
||||
/* Completion is recorded only after the durable Memory cleanup succeeds. */
|
||||
return;
|
||||
} catch (error) {
|
||||
const current = await this.repository.getSyncRun(runId);
|
||||
const cancelled = !timedOut && (error instanceof SyncCancelledError || controller.signal.aborted || current?.cancelRequested);
|
||||
const failure = timedOut
|
||||
const failure = current?.phase === "memory_cleanup"
|
||||
? { code: "memory_cleanup_pending", message: "Catalog synchronized. Memory cleanup is pending; retry this synchronization to complete it." }
|
||||
: timedOut
|
||||
? { code: "schema_sync_timed_out", message: "Schema synchronization timed out." }
|
||||
: safeFailure(error);
|
||||
await this.repository.updateSyncRun(runId, {
|
||||
state: cancelled ? "cancelled" : "failed",
|
||||
phase: "completed",
|
||||
phase: current?.phase === "memory_cleanup" ? "memory_cleanup" : "completed",
|
||||
errorCode: cancelled ? null : failure.code,
|
||||
errorMessage: cancelled ? null : failure.message,
|
||||
finishedAt: new Date().toISOString(),
|
||||
observedSnapshot: null,
|
||||
plannedDiff: null,
|
||||
observedSnapshot: current?.phase === "memory_cleanup" ? current.observedSnapshot : null,
|
||||
plannedDiff: current?.phase === "memory_cleanup" ? current.plannedDiff : null,
|
||||
confirmationToken: null,
|
||||
leaseOwner: null,
|
||||
leaseExpiresAt: null,
|
||||
@@ -278,6 +291,20 @@ export class CatalogSyncWorker {
|
||||
}
|
||||
}
|
||||
|
||||
private async finishMemoryCleanup(database: WorkspaceDatabase, run: CatalogSyncRun): Promise<void> {
|
||||
if (!run.plannedDiff || run.phase !== "memory_cleanup") throw new Error("Missing committed cleanup context");
|
||||
await this.repository.appendSyncEvent(run.id, "info", "memory_cleanup", "Removing Memory cards with deleted physical dependencies.");
|
||||
const memoryDeleted = await this.cleanupMemory(database, run);
|
||||
const counts = { ...run.counts, memoryDeleted };
|
||||
await this.repository.updateSyncRun(run.id, {
|
||||
state: "succeeded", phase: "completed", counts, finishedAt: new Date().toISOString(),
|
||||
observedSnapshot: null, plannedDiff: null, confirmationToken: null,
|
||||
heartbeatAt: new Date().toISOString(), leaseOwner: null, leaseExpiresAt: null,
|
||||
});
|
||||
await this.repository.appendSyncEvent(run.id, "info", "succeeded", "Synchronization completed.", counts);
|
||||
this.release(run.id);
|
||||
}
|
||||
|
||||
private assertCapability(scope: CatalogSyncScope, snapshot: ObservedSchemaSnapshot): void {
|
||||
const required = scope === "all" ? ["tables", "columns", "relationships"] as const : [scope] as const;
|
||||
for (const name of required) {
|
||||
|
||||
@@ -366,7 +366,7 @@ export type CatalogSyncState =
|
||||
export type CatalogSyncPhase =
|
||||
| "queued" | "connecting" | "scanning_tables" | "scanning_columns"
|
||||
| "scanning_relationships" | "planning" | "awaiting_confirmation"
|
||||
| "applying" | "completed";
|
||||
| "applying" | "memory_cleanup" | "completed";
|
||||
|
||||
export interface CatalogSchemaDiff {
|
||||
deletedTables: string[];
|
||||
@@ -375,6 +375,7 @@ export interface CatalogSchemaDiff {
|
||||
}
|
||||
|
||||
export interface CatalogSyncCounts {
|
||||
memoryDeleted?: number;
|
||||
tables?: number;
|
||||
columns?: number;
|
||||
relationships?: number;
|
||||
@@ -587,6 +588,7 @@ export interface CatalogRepository {
|
||||
scope: CatalogSyncScope,
|
||||
tableIds: readonly string[],
|
||||
snapshot: ObservedSchemaSnapshot,
|
||||
syncRunId?: string,
|
||||
): Promise<CatalogSyncCounts | undefined>;
|
||||
createSyncRun(
|
||||
databaseId: string,
|
||||
|
||||
@@ -30,8 +30,11 @@ export interface AppConfig {
|
||||
dataRoot?: string;
|
||||
ollamaEnsureTimeoutMs: number;
|
||||
piManagementTimeoutMs: number;
|
||||
/** Host CLI platform projected into Docker; not the browser or container OS. */
|
||||
hostPlatform?: string;
|
||||
secretsFile?: string;
|
||||
installationConfigFile?: string;
|
||||
evidenceHostRegistryRoot?: string;
|
||||
modelCatalogFile?: string;
|
||||
sensitivityNer?: {
|
||||
pythonExecutable: string;
|
||||
@@ -474,8 +477,10 @@ export function loadConfig(
|
||||
dataRoot: env.THT_DATA_ROOT,
|
||||
ollamaEnsureTimeoutMs: Number(env.OLLAMA_ENSURE_TIMEOUT_MS ?? 60000),
|
||||
piManagementTimeoutMs: piManagementTimeout(env.PI_MANAGEMENT_TIMEOUT_MS),
|
||||
hostPlatform: env.THT_HOST_PLATFORM,
|
||||
secretsFile,
|
||||
installationConfigFile,
|
||||
evidenceHostRegistryRoot: env.THT_EVIDENCE_HOST_REGISTRY_ROOT || undefined,
|
||||
modelCatalogFile,
|
||||
sensitivityNer,
|
||||
piAuthFile,
|
||||
|
||||
@@ -47,9 +47,8 @@ const runtimeModelSchema = z.object({
|
||||
}).strict();
|
||||
|
||||
const catalogSchema = z.object({
|
||||
schemaVersion: z.literal(1),
|
||||
defaultSession: canonicalId,
|
||||
defaultMetadataGeneration: canonicalId.optional(),
|
||||
schemaVersion: z.literal(2),
|
||||
defaultInteraction: canonicalId,
|
||||
embedding: z.object({ id: canonicalId, dimensions: z.number().int().positive() }).strict(),
|
||||
models: z.array(runtimeModelSchema).max(64),
|
||||
}).strict();
|
||||
@@ -57,8 +56,7 @@ const catalogSchema = z.object({
|
||||
export type RuntimeModel = z.infer<typeof runtimeModelSchema>;
|
||||
|
||||
export interface RuntimeModelCatalog {
|
||||
readonly defaultSession: string | null;
|
||||
readonly defaultMetadataGeneration: string | null;
|
||||
readonly defaultInteraction: string | null;
|
||||
readonly embedding: Readonly<{ id: string; dimensions: number }> | null;
|
||||
sessionModels(): readonly RuntimeModel[];
|
||||
metadataModels(): readonly RuntimeModel[];
|
||||
@@ -66,19 +64,21 @@ export interface RuntimeModelCatalog {
|
||||
}
|
||||
|
||||
class RestartLoadedRuntimeModelCatalog implements RuntimeModelCatalog {
|
||||
readonly defaultSession: string | null;
|
||||
readonly defaultMetadataGeneration: string | null;
|
||||
readonly defaultInteraction: string | null;
|
||||
readonly embedding: Readonly<{ id: string; dimensions: number }> | null;
|
||||
readonly #sessions: readonly RuntimeModel[];
|
||||
readonly #metadata: readonly RuntimeModel[];
|
||||
readonly #sessionIds: ReadonlySet<string>;
|
||||
|
||||
constructor(catalog?: z.infer<typeof catalogSchema>) {
|
||||
this.defaultSession = catalog?.defaultSession ?? null;
|
||||
this.defaultMetadataGeneration = catalog?.defaultMetadataGeneration ?? null;
|
||||
this.defaultInteraction = catalog?.defaultInteraction ?? null;
|
||||
this.embedding = catalog ? Object.freeze({ ...catalog.embedding }) : null;
|
||||
this.#sessions = Object.freeze((catalog?.models ?? []).filter((model) => model.session !== undefined));
|
||||
this.#metadata = Object.freeze((catalog?.models ?? []).filter((model) => model.metadataGeneration !== undefined));
|
||||
// One operational list. Session-only installations can still run Core, but once Admin
|
||||
// LLM models are configured every selectable model must support both adapters.
|
||||
const hasMetadata = catalog?.models.some((model) => model.metadataGeneration !== undefined);
|
||||
this.#sessions = Object.freeze((catalog?.models ?? []).filter((model) =>
|
||||
model.session !== undefined && (!hasMetadata || model.metadataGeneration !== undefined)));
|
||||
this.#metadata = Object.freeze(this.#sessions.filter((model) => model.metadataGeneration !== undefined));
|
||||
this.#sessionIds = new Set(this.#sessions.map((model) => model.id));
|
||||
}
|
||||
|
||||
@@ -129,11 +129,9 @@ export function loadRuntimeModelCatalog(file?: string): RuntimeModelCatalog {
|
||||
if (ids.size !== parsed.data.models.length) throw new Error("runtime model catalog contains duplicate models");
|
||||
const sessions = parsed.data.models.filter((model) => model.session !== undefined).map((model) => model.id);
|
||||
const metadata = parsed.data.models.filter((model) => model.metadataGeneration !== undefined).map((model) => model.id);
|
||||
if (!sessions.includes(parsed.data.defaultSession)) throw new Error("runtime model catalog session default is invalid");
|
||||
if ((metadata.length > 0) !== (parsed.data.defaultMetadataGeneration !== undefined)
|
||||
|| (parsed.data.defaultMetadataGeneration !== undefined
|
||||
&& !metadata.includes(parsed.data.defaultMetadataGeneration))) {
|
||||
throw new Error("runtime model catalog metadata default is invalid");
|
||||
if (!sessions.includes(parsed.data.defaultInteraction)
|
||||
|| (metadata.length > 0 && !metadata.includes(parsed.data.defaultInteraction))) {
|
||||
throw new Error("runtime model catalog interaction default is invalid");
|
||||
}
|
||||
return new RestartLoadedRuntimeModelCatalog(parsed.data);
|
||||
}
|
||||
|
||||
@@ -88,17 +88,22 @@ async function workflowDiagnostics(config: AppConfig): Promise<{ ready: true; wo
|
||||
if (revisions.length === 0) throw new Error("workflow diagnostics unavailable");
|
||||
return await withOperatorRunner(config, async (runner) => {
|
||||
for (const revision of revisions) {
|
||||
const result = await runner.run(["doctor", "--json"], revision.snapshotPath);
|
||||
let payload: unknown;
|
||||
const runtime = await runner.acquireWorkspaceRuntime(revision.snapshotPath);
|
||||
try {
|
||||
payload = JSON.parse(result.stdout);
|
||||
} catch {
|
||||
throw new Error("workflow diagnostics failed");
|
||||
const result = await runner.run(["doctor", "--json"], runtime.path);
|
||||
let payload: unknown;
|
||||
try {
|
||||
payload = JSON.parse(result.stdout);
|
||||
} catch {
|
||||
throw new Error("workflow diagnostics failed");
|
||||
}
|
||||
if (
|
||||
result.code !== 0 || !payload || typeof payload !== "object"
|
||||
|| (payload as { ok?: unknown }).ok !== true
|
||||
) throw new Error("workflow diagnostics failed");
|
||||
} finally {
|
||||
runtime.release();
|
||||
}
|
||||
if (
|
||||
result.code !== 0 || !payload || typeof payload !== "object"
|
||||
|| (payload as { ok?: unknown }).ok !== true
|
||||
) throw new Error("workflow diagnostics failed");
|
||||
}
|
||||
return { ready: true, workspaces: revisions.length };
|
||||
});
|
||||
|
||||
@@ -11,6 +11,7 @@ import {
|
||||
validateDeclarativePiConfig,
|
||||
} from "./managed-config.js";
|
||||
import type { RuntimeModelCatalog } from "../models/runtime-model-catalog.js";
|
||||
import { secretValue } from "../config/secret-bundle.js";
|
||||
|
||||
export interface PiModel {
|
||||
provider: string;
|
||||
@@ -64,6 +65,14 @@ export function createPiModelLister(cfg: AppConfig, opts: Opts = {}): ListModels
|
||||
}
|
||||
|
||||
const env = buildPiChildEnv({});
|
||||
// Pi's availability enumeration also needs the catalog-owned credentials for built-in
|
||||
// providers. It must keep working after their obsolete Pi auth entries are removed.
|
||||
for (const model of opts.modelCatalog?.sessionModels() ?? []) {
|
||||
const name = model.authentication.mode === "secret_env" ? model.authentication.apiKeyEnv : undefined;
|
||||
if (!name) continue;
|
||||
const value = secretValue(cfg, name);
|
||||
if (value) env[name] = value;
|
||||
}
|
||||
delete env.THT_DATA_ROOT;
|
||||
if (cfg.dataRoot !== undefined) env.THT_DATA_ROOT = cfg.dataRoot;
|
||||
const child = spawnFn(cfg.piBin, ["--mode", "rpc"], { cwd: cfg.harnessDir, env });
|
||||
|
||||
@@ -138,7 +138,7 @@ export interface PiRuntimeAgentSnapshot {
|
||||
* Bind a session Pi process to the exact managed auth/model bytes validated at spawn time.
|
||||
* Other agent resources remain live through symlinks, while session storage stays persistent.
|
||||
*/
|
||||
export function createPiRuntimeAgentSnapshot(): PiRuntimeAgentSnapshot {
|
||||
export function createPiRuntimeAgentSnapshot(options: { excludeAuthProvider?: string } = {}): PiRuntimeAgentSnapshot {
|
||||
const sourceAgentDir = configuredPiAgentDir();
|
||||
const auth = readPiAgentFile(sourceAgentDir, "auth.json", true);
|
||||
const models = readPiAgentFile(sourceAgentDir, "models.json", true);
|
||||
@@ -165,7 +165,18 @@ export function createPiRuntimeAgentSnapshot(): PiRuntimeAgentSnapshot {
|
||||
);
|
||||
}
|
||||
if (auth !== undefined) {
|
||||
writeFileSync(join(snapshotDir, "auth.json"), auth, { flag: "wx", mode: 0o600 });
|
||||
let effectiveAuth = auth;
|
||||
if (options.excludeAuthProvider) {
|
||||
const parsed = parsePiConfigJson(auth);
|
||||
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) throw new PiManagedConfigError();
|
||||
const provider = options.excludeAuthProvider.trim().toLowerCase();
|
||||
effectiveAuth = JSON.stringify(Object.fromEntries(
|
||||
Object.entries(parsed).filter(([key]) => key.trim().toLowerCase() !== provider),
|
||||
));
|
||||
}
|
||||
// secret_env is authoritative for this provider. Keep the operator's auth file intact,
|
||||
// but do not let an old Pi credential override the shared bundle inside this child.
|
||||
writeFileSync(join(snapshotDir, "auth.json"), effectiveAuth, { flag: "wx", mode: 0o600 });
|
||||
}
|
||||
if (models !== undefined) {
|
||||
writeFileSync(join(snapshotDir, "models.json"), models, { flag: "wx", mode: 0o600 });
|
||||
|
||||
@@ -36,6 +36,7 @@ export interface PiInstallationConfig {
|
||||
}
|
||||
|
||||
export interface PiStatus {
|
||||
hostPlatform: "linux" | "macos" | "windows";
|
||||
version?: string;
|
||||
ready: boolean;
|
||||
credentials: PiCredentialStatus;
|
||||
@@ -92,6 +93,9 @@ interface PiManagementDeps {
|
||||
}
|
||||
|
||||
export function createPiManagement(config: AppConfig, deps: PiManagementDeps): PiManagementService {
|
||||
const platform = config.hostPlatform ?? process.platform;
|
||||
const hostPlatform = platform === "darwin" ? "macos"
|
||||
: platform === "windows" || platform === "win32" ? "windows" : "linux";
|
||||
const now = deps.now ?? (() => new Date());
|
||||
const diagnostics: string[] = [];
|
||||
const addDiagnostic = (message: string): void => {
|
||||
@@ -106,23 +110,23 @@ export function createPiManagement(config: AppConfig, deps: PiManagementDeps): P
|
||||
});
|
||||
const credentialStatus = deps.credentialStatus ?? ((provider: string | undefined) => {
|
||||
try {
|
||||
const model = deps.modelCatalog.defaultSession
|
||||
? deps.modelCatalog.sessionModels().find((entry) => entry.id === deps.modelCatalog.defaultSession)
|
||||
const model = deps.modelCatalog.defaultInteraction
|
||||
? deps.modelCatalog.sessionModels().find((entry) => entry.id === deps.modelCatalog.defaultInteraction)
|
||||
: undefined;
|
||||
const credentialName = model?.authentication.mode === "secret_env"
|
||||
? model.authentication.apiKeyEnv
|
||||
: undefined;
|
||||
const configuredApiKey = configuredPiProviderApiKey(
|
||||
const configuredApiKey = credentialName ? `$${credentialName}` : configuredPiProviderApiKey(
|
||||
readConfiguredPiAgentFile("models.json", true),
|
||||
provider,
|
||||
) ?? (credentialName ? `$${credentialName}` : undefined);
|
||||
);
|
||||
return piProviderCredentialStatus({
|
||||
provider,
|
||||
authProviders: loadPiAuthProviders(),
|
||||
authProviders: credentialName ? new Set() : loadPiAuthProviders(),
|
||||
resolveCredentialValue: () => credentialName
|
||||
? secretValue(config, credentialName)
|
||||
: config.modelCatalogFile ? undefined : secretValue(config, "THT_MODEL_API_KEY"),
|
||||
credentialFile: config.modelApiKeyFile,
|
||||
credentialFile: credentialName ? undefined : config.modelApiKeyFile,
|
||||
configuredApiKey,
|
||||
});
|
||||
} catch {
|
||||
@@ -152,8 +156,8 @@ export function createPiManagement(config: AppConfig, deps: PiManagementDeps): P
|
||||
const installationConfig = (): PiInstallationConfig => {
|
||||
const settings = readSettings();
|
||||
const reasoning = config.defaults.thinking ?? settings.thinking;
|
||||
const selected = deps.modelCatalog.defaultSession
|
||||
? splitCanonicalModelId(deps.modelCatalog.defaultSession)
|
||||
const selected = deps.modelCatalog.defaultInteraction
|
||||
? splitCanonicalModelId(deps.modelCatalog.defaultInteraction)
|
||||
: undefined;
|
||||
return {
|
||||
...(selected ? selected : {}),
|
||||
@@ -169,11 +173,11 @@ export function createPiManagement(config: AppConfig, deps: PiManagementDeps): P
|
||||
try {
|
||||
const currentVersion = await version();
|
||||
addDiagnostic("Pi version probe succeeded");
|
||||
return { version: currentVersion, ready: true, credentials, config: current, checkedAt };
|
||||
return { hostPlatform, version: currentVersion, ready: true, credentials, config: current, checkedAt };
|
||||
} catch (error) {
|
||||
const message = stableMessage(error, "Pi runtime is unavailable");
|
||||
addDiagnostic(message);
|
||||
return { ready: false, credentials, config: current, checkedAt, message };
|
||||
return { hostPlatform, ready: false, credentials, config: current, checkedAt, message };
|
||||
}
|
||||
},
|
||||
|
||||
|
||||
@@ -25,6 +25,7 @@ export interface SessionRuntime {
|
||||
}
|
||||
|
||||
export interface RuntimeOptions {
|
||||
interactionLanguage?: string | null;
|
||||
provider?: string;
|
||||
model?: string;
|
||||
thinking?: string;
|
||||
@@ -48,6 +49,7 @@ export class PiProcessManager {
|
||||
private spawnFn: (
|
||||
sessionId: string, author: string, provider: string | undefined, model: string | undefined,
|
||||
principal?: PrincipalContext, runtimeConfigPath?: string,
|
||||
interactionLanguage?: string | null,
|
||||
) => ChildProcessWithoutNullStreams;
|
||||
private loadAuthProviders: (agentDir: string) => ReadonlySet<string>;
|
||||
private modelCatalog: RuntimeModelCatalog;
|
||||
@@ -63,15 +65,15 @@ export class PiProcessManager {
|
||||
) {
|
||||
this.modelCatalog = opts?.modelCatalog ?? loadRuntimeModelCatalog(cfg.modelCatalogFile);
|
||||
this.modelCatalogConfigured = cfg.modelCatalogFile !== undefined
|
||||
|| this.modelCatalog.defaultSession !== null;
|
||||
|| this.modelCatalog.defaultInteraction !== null;
|
||||
this.loadAuthProviders = opts?.authProviders
|
||||
?? ((agentDir) => loadPiAuthProviders({ agentDir }));
|
||||
if (opts?.spawnFn) {
|
||||
this.spawnFn = (sessionId, author, provider, model, principal, runtimeConfigPath) =>
|
||||
this.spawnPi(opts.spawnFn!, sessionId, author, provider, model, principal, runtimeConfigPath);
|
||||
this.spawnFn = (sessionId, author, provider, model, principal, runtimeConfigPath, language) =>
|
||||
this.spawnPi(opts.spawnFn!, sessionId, author, provider, model, principal, runtimeConfigPath, language);
|
||||
} else {
|
||||
this.spawnFn = (sessionId, author, provider, model, principal, runtimeConfigPath) =>
|
||||
this.spawnPi(nodeSpawn, sessionId, author, provider, model, principal, runtimeConfigPath);
|
||||
this.spawnFn = (sessionId, author, provider, model, principal, runtimeConfigPath, language) =>
|
||||
this.spawnPi(nodeSpawn, sessionId, author, provider, model, principal, runtimeConfigPath, language);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -85,33 +87,35 @@ export class PiProcessManager {
|
||||
private spawnPi(
|
||||
spawnFn: SpawnFn, sessionId: string, author: string, provider: string | undefined,
|
||||
model: string | undefined, principal?: PrincipalContext, runtimeConfigPath?: string,
|
||||
interactionLanguage?: string | null,
|
||||
): ChildProcessWithoutNullStreams {
|
||||
// This is the final shared boundary for createFor(), spawnFor(), and resume(). Validate
|
||||
// before auth-provider inspection, then make Pi consume the exact copied bytes rather than
|
||||
// reopening mutable mounted auth/models files after this check.
|
||||
const agent = createPiRuntimeAgentSnapshot();
|
||||
const catalogModel = provider && model
|
||||
? this.modelCatalog.sessionModels().find((entry) => entry.provider === provider && entry.model === model)
|
||||
: undefined;
|
||||
const credentialName = catalogModel?.authentication.mode === "secret_env"
|
||||
? catalogModel.authentication.apiKeyEnv : undefined;
|
||||
const agent = createPiRuntimeAgentSnapshot({ excludeAuthProvider: credentialName ? provider : undefined });
|
||||
let child: ChildProcessWithoutNullStreams | undefined;
|
||||
try {
|
||||
const catalogModel = provider && model
|
||||
? this.modelCatalog.sessionModels()
|
||||
.find((entry) => entry.provider === provider && entry.model === model)
|
||||
: undefined;
|
||||
const credentialName = catalogModel?.authentication.mode === "secret_env"
|
||||
? catalogModel.authentication.apiKeyEnv
|
||||
: undefined;
|
||||
const projectedApiKey = configuredPiProviderApiKey(agent.models, provider)
|
||||
?? (credentialName ? `$${credentialName}` : undefined);
|
||||
const projectedApiKey = credentialName ? `$${credentialName}`
|
||||
: configuredPiProviderApiKey(agent.models, provider);
|
||||
const env = buildPiChildEnv({
|
||||
provider,
|
||||
authProviders: this.loadAuthProviders(agent.agentDir),
|
||||
authProviders: credentialName ? new Set() : this.loadAuthProviders(agent.agentDir),
|
||||
credentialValue: credentialName
|
||||
? secretValue(this.cfg, credentialName)
|
||||
: this.modelCatalogConfigured ? undefined : secretValue(this.cfg, "THT_MODEL_API_KEY"),
|
||||
credentialFile: this.cfg.modelApiKeyFile,
|
||||
credentialFile: credentialName ? undefined : this.cfg.modelApiKeyFile,
|
||||
configuredApiKey: projectedApiKey,
|
||||
additions: { THT_SESSION: sessionId, THT_AUTHOR: author },
|
||||
});
|
||||
env.PI_CODING_AGENT_DIR = agent.agentDir;
|
||||
// A launch hint only: the gate reads the authoritative manifest before each turn.
|
||||
delete env.THT_INTERACTION_LANGUAGE;
|
||||
if (interactionLanguage) env.THT_INTERACTION_LANGUAGE = interactionLanguage;
|
||||
env.PI_CODING_AGENT_SESSION_DIR = agent.sessionDir;
|
||||
clearPrincipalEnvironment(env);
|
||||
if (principal) Object.assign(env, principalEnvironment(principal));
|
||||
@@ -189,7 +193,9 @@ export class PiProcessManager {
|
||||
const model = o.model ?? this.cfg.defaults.model;
|
||||
let child: ChildProcessWithoutNullStreams;
|
||||
try {
|
||||
child = this.spawnFn(sessionId, author, provider, model, o.principal, o.runtimeConfig?.path);
|
||||
child = this.spawnFn(
|
||||
sessionId, author, provider, model, o.principal, o.runtimeConfig?.path, o.interactionLanguage,
|
||||
);
|
||||
} catch (error) {
|
||||
o.runtimeConfig?.release();
|
||||
throw error;
|
||||
@@ -298,8 +304,13 @@ export class PiProcessManager {
|
||||
}
|
||||
|
||||
async resume(sessionId: string, tht: ThtRunner): Promise<SessionRuntime> {
|
||||
const manifest = await tht.sessionShow(sessionId) as { provider?: string; model?: string; thinking?: string } | null;
|
||||
const manifest = await tht.sessionShow(sessionId) as {
|
||||
provider?: string; model?: string; thinking?: string; interaction_language?: string | null;
|
||||
} | null;
|
||||
const language = manifest?.interaction_language
|
||||
?? (await tht.ensureInteractionLanguage(sessionId)).interaction_language;
|
||||
return this.spawnFor(sessionId, {
|
||||
interactionLanguage: language,
|
||||
provider: manifest?.provider,
|
||||
model: manifest?.model,
|
||||
thinking: manifest?.thinking,
|
||||
|
||||
@@ -70,28 +70,29 @@ export function createPiProviderSmoke(
|
||||
try {
|
||||
const canonicalProvider = canonicalPiProvider(provider);
|
||||
if (!canonicalProvider || timeoutMs <= 0) throw providerFailure();
|
||||
const configuredAuthProviders = authProviders();
|
||||
const configuredAuthProviders = new Set(authProviders());
|
||||
const configuredModels = options.readModelsStore
|
||||
? options.readModelsStore()
|
||||
: readConfiguredPiAgentFile("models.json", true);
|
||||
const catalog = options.modelCatalog ?? loadRuntimeModelCatalog(config.modelCatalogFile);
|
||||
const catalogConfigured = config.modelCatalogFile !== undefined
|
||||
|| catalog.defaultSession !== null;
|
||||
|| catalog.defaultInteraction !== null;
|
||||
const catalogModel = catalog.sessionModels()
|
||||
.find((entry) => entry.provider === canonicalProvider && entry.model === model);
|
||||
const upstreamModel = catalogModel?.upstreamModel ?? model;
|
||||
const credentialName = catalogModel?.authentication.mode === "secret_env"
|
||||
? catalogModel.authentication.apiKeyEnv
|
||||
: undefined;
|
||||
const projectedApiKey = configuredPiProviderApiKey(configuredModels, canonicalProvider)
|
||||
?? (credentialName ? `$${credentialName}` : undefined);
|
||||
if (credentialName) configuredAuthProviders.delete(canonicalProvider);
|
||||
const projectedApiKey = credentialName ? `$${credentialName}`
|
||||
: configuredPiProviderApiKey(configuredModels, canonicalProvider);
|
||||
const env = buildPiChildEnv({
|
||||
provider: canonicalProvider,
|
||||
authProviders: configuredAuthProviders,
|
||||
credentialValue: credentialName
|
||||
? secretValue(config, credentialName)
|
||||
: catalogConfigured ? undefined : secretValue(config, "THT_MODEL_API_KEY"),
|
||||
credentialFile: config.modelApiKeyFile,
|
||||
credentialFile: credentialName ? undefined : config.modelApiKeyFile,
|
||||
configuredApiKey: projectedApiKey,
|
||||
});
|
||||
clearPrincipalEnvironment(env);
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
import path from "node:path";
|
||||
import type { FastifyInstance } from "fastify";
|
||||
import { z } from "zod";
|
||||
import { requirePermission, isPrincipalContext } from "../auth/authorization.js";
|
||||
import type { ThtRunner } from "../tht/tht-runner.js";
|
||||
import type { WorkspaceRegistry } from "../workspaces/registry.js";
|
||||
import type { WorkspacePreprocessingService } from "../workspaces/preprocessing-service.js";
|
||||
|
||||
const params = z.object({ workspaceId: z.string().regex(/^[a-z][a-z0-9-]{2,62}$/),
|
||||
evidenceId: z.string().regex(/^evidence:[a-z0-9]+(?:-[a-z0-9]+)*$/).optional() });
|
||||
const query = z.object({
|
||||
q: z.string().max(1000).optional(), kind: z.enum(["domain", "glossary", "enum", "example", "mapping", "normalization", "formula", "reference"]).optional(),
|
||||
purpose: z.enum(["disambiguation", "rewriting", "schema_linking", "sql_generation"]).optional(),
|
||||
status: z.enum(["new", "modified", "active", "removed", "review_required", "legacy", "invalid"]).optional(),
|
||||
concept: z.string().max(300).optional(), table: z.string().max(300).optional(), column: z.string().max(300).optional(),
|
||||
source: z.string().max(300).optional(), language: z.string().max(30).optional(),
|
||||
sort: z.enum(["title", "id", "kind", "status"]).optional(), direction: z.enum(["asc", "desc"]).optional(),
|
||||
page: z.coerce.number().int().positive().optional(), page_size: z.coerce.number().int().min(1).max(100).optional(),
|
||||
}).strict();
|
||||
const quote = (value: string) => `'${value.replaceAll("'", "'\\''")}'`;
|
||||
|
||||
export function evidenceRoutes(app: FastifyInstance, deps: {
|
||||
runner: Pick<ThtRunner, "withPrincipal">; registry: Pick<WorkspaceRegistry, "list">;
|
||||
registryRoot: string; hostRegistryRoot?: string;
|
||||
service: Partial<Pick<WorkspacePreprocessingService, "consolidateEvidence" | "evidenceSources">>;
|
||||
}) {
|
||||
app.post("/workspaces/:workspaceId/evidence/sources", async (request, reply) => {
|
||||
const principal = requirePermission(request, reply, "evidence.manage");
|
||||
if (!isPrincipalContext(principal)) return reply;
|
||||
try {
|
||||
const { workspaceId } = params.parse(request.params);
|
||||
const body = z.discriminatedUnion("action", [
|
||||
z.object({ action: z.literal("refresh") }).strict(),
|
||||
z.object({ action: z.literal("decide"), sourceId: z.string().regex(/^[a-f0-9]{64}$/),
|
||||
revision: z.string().regex(/^[a-f0-9]{64}$/), decision: z.enum(["keep", "replace"]) }).strict(),
|
||||
]).parse(request.body);
|
||||
if (!(await deps.registry.list()).some(w => w.id === workspaceId)) return reply.code(404).send({ code: "workspace_invalid" });
|
||||
if (!deps.service.evidenceSources) throw new Error("Evidence service unavailable");
|
||||
return await deps.service.evidenceSources({ workspaceId, ...body, actor: principal.subject });
|
||||
} catch (error) {
|
||||
return reply.code(error instanceof z.ZodError ? 400 : 503).send({ code: "evidence_unavailable",
|
||||
message: error instanceof z.ZodError ? "Invalid source request." : "Evidence source service is unavailable." });
|
||||
}
|
||||
});
|
||||
app.get<{ Params: { workspaceId: string; evidenceId?: string } }>("/workspaces/:workspaceId/evidence", read);
|
||||
app.get<{ Params: { workspaceId: string; evidenceId?: string } }>("/workspaces/:workspaceId/evidence/:evidenceId", read);
|
||||
async function read(request: import("fastify").FastifyRequest, reply: import("fastify").FastifyReply) {
|
||||
const principal = requirePermission(request, reply, "evidence.manage");
|
||||
if (!isPrincipalContext(principal)) return reply;
|
||||
try {
|
||||
const { workspaceId, evidenceId } = params.parse(request.params);
|
||||
const filters = query.parse(request.query);
|
||||
if (!(await deps.registry.list()).some(w => w.id === workspaceId)) return reply.code(404).send({ code: "workspace_invalid" });
|
||||
const root = path.join(deps.registryRoot, "repo", workspaceId);
|
||||
const result = await deps.runner.withPrincipal(principal).runWithRuntimeSnapshot(
|
||||
["evidence", "admin", "--workspace", workspaceId],
|
||||
JSON.stringify({ root, query: { ...filters, ...(evidenceId ? { id: evidenceId } : {}) } }),
|
||||
);
|
||||
const payload = JSON.parse(result.stdout);
|
||||
if (result.code !== 0) return reply.code(503).send(payload);
|
||||
if (evidenceId && !payload.item) return reply.code(404).send({ code: "evidence_not_found", message: "Evidence was not found." });
|
||||
const host = deps.hostRegistryRoot;
|
||||
const hostPath = host && (path.isAbsolute(host) || path.win32.isAbsolute(host))
|
||||
? (path.win32.isAbsolute(host) && !path.isAbsolute(host) ? path.win32 : path).join(host, "repo") : null;
|
||||
const hostJoin = hostPath && path.win32.isAbsolute(hostPath) && !path.isAbsolute(hostPath) ? path.win32.join : path.join;
|
||||
return { ...payload, location: { repository: hostPath, workspace: hostPath ? hostJoin(hostPath, workspaceId) : null,
|
||||
runtime_workspace: root, host: "Installation host", command: `tht workspace evidence consolidate --workspace ${workspaceId}`,
|
||||
git_commands: hostPath ? [
|
||||
`git -C ${quote(hostPath)} status --short -- ${quote(workspaceId + "/evidence")}`,
|
||||
`git -C ${quote(hostPath)} diff -- ${quote(workspaceId + "/evidence")}`,
|
||||
`git -C ${quote(hostPath)} add -A -- ${quote(workspaceId + "/evidence")}`,
|
||||
`git -C ${quote(hostPath)} commit --only -m 'Curate Evidence' -- ${quote(workspaceId + "/evidence")}`,
|
||||
`git -C ${quote(hostPath)} push`,
|
||||
] : [] } };
|
||||
} catch (error) {
|
||||
return reply.code(error instanceof z.ZodError ? 400 : 503).send({ code: "evidence_unavailable",
|
||||
message: error instanceof z.ZodError ? "Evidence filters are invalid." : "Evidence archive is unavailable." });
|
||||
}
|
||||
}
|
||||
app.post("/workspaces/:workspaceId/evidence/consolidate", async (request, reply) => {
|
||||
const principal = requirePermission(request, reply, "evidence.manage");
|
||||
if (!isPrincipalContext(principal)) return reply;
|
||||
try {
|
||||
const { workspaceId } = params.parse(request.params);
|
||||
if (!(await deps.registry.list()).some(w => w.id === workspaceId)) return reply.code(404).send({ code: "workspace_invalid" });
|
||||
if (!deps.service.consolidateEvidence) throw new Error("Evidence service unavailable");
|
||||
return await deps.service.consolidateEvidence({ workspaceId });
|
||||
} catch { return reply.code(503).send({ code: "evidence_unavailable", message: "Evidence consolidation is unavailable." }); }
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
|
||||
import { z } from "zod";
|
||||
import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
|
||||
import type { ThtRunner } from "../tht/tht-runner.js";
|
||||
import type { WorkspaceRegistry } from "../workspaces/registry.js";
|
||||
import type { SemanticRuntimeConfig } from "../workspaces/runtime-renderer.js";
|
||||
|
||||
const paramsSchema = z.object({
|
||||
workspaceId: z.string().regex(/^[a-z][a-z0-9_-]{0,63}$/),
|
||||
cardId: z.string().regex(/^mem-[0-9a-f-]{36}$/).optional(),
|
||||
});
|
||||
const family = z.enum(["domain_clarification", "sql_rule", "solved_question", "explained_error"]);
|
||||
const cardSchema = z.object({
|
||||
family, subject: z.string().trim().min(1).max(1000),
|
||||
detail: z.string().max(50000).default(""), scope: z.string().trim().min(1).max(10000),
|
||||
rationale: z.string().max(10000).default(""), question: z.string().max(10000).default(""),
|
||||
sql: z.string().max(100000).default(""),
|
||||
concepts: z.array(z.string().trim().min(1).max(200)).max(100).default([]),
|
||||
dependencies: z.array(z.object({
|
||||
database: z.string().trim().min(1).max(200), schema_name: z.string().max(200).default(""),
|
||||
table: z.string().max(200).default(""), column: z.string().max(200).default(""),
|
||||
}).strict()).max(200).default([]),
|
||||
links: z.array(z.object({
|
||||
target_id: z.string().min(1).max(100), meaning: z.string().trim().min(1).max(1000),
|
||||
}).strict()).max(200).default([]),
|
||||
}).strict();
|
||||
const querySchema = z.object({
|
||||
q: z.string().max(1000).optional(), family: family.optional(),
|
||||
concept: z.string().max(200).optional(), database: z.string().max(200).optional(),
|
||||
table: z.string().max(200).optional(), column: z.string().max(200).optional(),
|
||||
origin: z.enum(["manual", "workflow"]).optional(),
|
||||
updated_after: z.iso.datetime({ offset: true }).optional(),
|
||||
updated_before: z.iso.datetime({ offset: true }).optional(),
|
||||
page: z.coerce.number().int().positive().optional(),
|
||||
page_size: z.coerce.number().int().min(1).max(100).optional(),
|
||||
sort: z.enum(["updated_at", "created_at", "subject", "family"]).optional(),
|
||||
direction: z.enum(["asc", "desc"]).optional(),
|
||||
}).strict();
|
||||
|
||||
export function memoryRoutes(app: FastifyInstance, deps: {
|
||||
runner: Pick<ThtRunner, "withPrincipal">;
|
||||
registry: Pick<WorkspaceRegistry, "list" | "read">;
|
||||
runtime: SemanticRuntimeConfig;
|
||||
}) {
|
||||
const invoke = (action: string) => async (request: FastifyRequest, reply: FastifyReply) => {
|
||||
const principal = requirePermission(request, reply, "memory.manage");
|
||||
if (!isPrincipalContext(principal)) return reply;
|
||||
try {
|
||||
const { workspaceId, cardId } = paramsSchema.parse(request.params);
|
||||
const input = action === "list" ? querySchema.parse(request.query)
|
||||
: action === "create" || action === "update"
|
||||
? { card: cardSchema.parse(request.body), ...(cardId ? { id: cardId } : {}) }
|
||||
: cardId ? { id: cardId } : {};
|
||||
// Registry identity is enough: administrative browsing must not require a DWH binding.
|
||||
const workspaces = await deps.registry.list();
|
||||
if (!workspaces.some(workspace => workspace.id === workspaceId)) {
|
||||
return reply.code(404).send({ code: "workspace_invalid", message: "Workspace was not found." });
|
||||
}
|
||||
const { workspace } = await deps.registry.read(workspaceId);
|
||||
const result = await deps.runner.withPrincipal(principal).runWithRuntimeSnapshot(
|
||||
["memory", "admin", "--workspace", workspaceId],
|
||||
JSON.stringify({ action, request: input, runtime: {
|
||||
...deps.runtime, memoryLanguage: workspace.workspace.language,
|
||||
} }),
|
||||
);
|
||||
let payload;
|
||||
try { payload = JSON.parse(result.stdout); } catch {
|
||||
return reply.code(503).send({ code: "memory_unavailable", message: "Memory archive is unavailable." });
|
||||
}
|
||||
if (result.code !== 0) {
|
||||
const codes: Record<string, number> = {
|
||||
memory_invalid: 400, memory_forbidden: 403, memory_not_found: 404,
|
||||
memory_conflict: 409, memory_unavailable: 503,
|
||||
};
|
||||
const code = typeof payload?.code === "string" && payload.code in codes
|
||||
? payload.code : "memory_unavailable";
|
||||
return reply.code(codes[code]).send({ code, message: "Memory operation could not be completed." });
|
||||
}
|
||||
return reply.code(action === "create" ? 201 : 200).send(payload);
|
||||
} catch (error) {
|
||||
if (error instanceof z.ZodError) {
|
||||
return reply.code(400).send({ code: "memory_invalid", message: "Memory request is invalid." });
|
||||
}
|
||||
return reply.code(503).send({ code: "memory_unavailable", message: "Memory archive is unavailable." });
|
||||
}
|
||||
};
|
||||
app.get("/workspaces/:workspaceId/memory", invoke("list"));
|
||||
app.post("/workspaces/:workspaceId/memory", invoke("create"));
|
||||
app.get("/workspaces/:workspaceId/memory/pending", invoke("pending"));
|
||||
app.get("/workspaces/:workspaceId/memory/:cardId", invoke("show"));
|
||||
app.put("/workspaces/:workspaceId/memory/:cardId", invoke("update"));
|
||||
app.delete("/workspaces/:workspaceId/memory/:cardId", invoke("delete"));
|
||||
app.post("/workspaces/:workspaceId/memory/:cardId/retry", invoke("retry"));
|
||||
}
|
||||
@@ -13,6 +13,7 @@ import type { MaintenanceBarrier } from "../runtime/maintenance-gate.js";
|
||||
import { hasPermission, isPrincipalContext, requirePermission } from "../auth/authorization.js";
|
||||
import { splitCanonicalModelId, type RuntimeModelCatalog } from "../models/runtime-model-catalog.js";
|
||||
import type { CatalogRepository } from "../catalog/types.js";
|
||||
import { interactionLanguage } from "../tht/interaction-language.js";
|
||||
|
||||
const BOOTSTRAP_FAILURE_MESSAGE =
|
||||
"Session startup failed. Check configuration and connectivity, then Resume the session.";
|
||||
@@ -150,13 +151,34 @@ export function sessionRoutes(
|
||||
});
|
||||
app.addHook("onResponse", async (req) => { admissionLeases.get(req)?.(); });
|
||||
|
||||
type SessionRevisionScan = {
|
||||
revisions: Awaited<ReturnType<typeof d.workspaceRegistry.list>>;
|
||||
retainedComplete: boolean;
|
||||
};
|
||||
let retainedSnapshotWarning: string | null = null;
|
||||
|
||||
/** Include retained historical descriptors so removed workspaces remain resumable. */
|
||||
const sessionRevisions = async () => {
|
||||
const sessionRevisions = async (): Promise<SessionRevisionScan> => {
|
||||
const registry = d.workspaceRegistry as Partial<WorkspaceRegistry>;
|
||||
if (typeof registry.listRetainedSnapshots === "function") {
|
||||
return await registry.listRetainedSnapshots();
|
||||
try {
|
||||
const revisions = await registry.listRetainedSnapshots();
|
||||
retainedSnapshotWarning = null;
|
||||
return { revisions, retainedComplete: true };
|
||||
} catch (error) {
|
||||
// A legacy/corrupt historical snapshot must not make active sessions (and their SSE
|
||||
// reviewer gates) unreachable. The registry still rejects that snapshot; this fallback
|
||||
// exposes only descriptors from the verified active state and deliberately disables
|
||||
// retention reconciliation because the resulting session view is incomplete.
|
||||
const detail = error instanceof Error ? error.message : "unknown error";
|
||||
if (retainedSnapshotWarning !== detail) {
|
||||
console.warn("[sessions] retained snapshot discovery failed; using active snapshots:", detail);
|
||||
retainedSnapshotWarning = detail;
|
||||
}
|
||||
return { revisions: await d.workspaceRegistry.list(), retainedComplete: false };
|
||||
}
|
||||
}
|
||||
return await d.workspaceRegistry.list();
|
||||
return { revisions: await d.workspaceRegistry.list(), retainedComplete: true };
|
||||
};
|
||||
|
||||
const isNotFound = (error: unknown) =>
|
||||
@@ -200,7 +222,7 @@ export function sessionRoutes(
|
||||
};
|
||||
let revisions: Awaited<ReturnType<typeof d.workspaceRegistry.list>>;
|
||||
try {
|
||||
revisions = await sessionRevisions();
|
||||
({ revisions } = await sessionRevisions());
|
||||
} catch (registryError) {
|
||||
// Sessions created before revision pinning still live under the installation's legacy
|
||||
// default config. Keep that compatibility path available when a fresh installation has
|
||||
@@ -360,8 +382,17 @@ export function sessionRoutes(
|
||||
app.post("/sessions", async (req, reply) => {
|
||||
const b = req.body as {
|
||||
question: string; name?: string; workspace?: string; workspaceId?: string;
|
||||
provider?: string; model?: string; thinking?: string;
|
||||
provider?: string; model?: string; thinking?: string; interactionLanguage?: unknown;
|
||||
};
|
||||
const language = interactionLanguage(b?.interactionLanguage);
|
||||
if (!language) return reply.code(400).send({
|
||||
code: "invalid_interaction_language",
|
||||
error: "interactionLanguage must be a well-formed BCP-47 language tag",
|
||||
});
|
||||
if ((b.provider === undefined) !== (b.model === undefined)
|
||||
|| (b.provider !== undefined && (typeof b.provider !== "string" || typeof b.model !== "string" || !b.provider || !b.model))) {
|
||||
return reply.code(400).send({ error: "provider and model must be supplied together" });
|
||||
}
|
||||
const principal = getPrincipal(req);
|
||||
let s: Settings;
|
||||
try { s = await d.getSettings(principal); } catch { return storageFailure(reply); }
|
||||
@@ -425,11 +456,9 @@ export function sessionRoutes(
|
||||
}
|
||||
}
|
||||
const requestedCanonical = b.provider && b.model ? `${b.provider}/${b.model}` : undefined;
|
||||
let selectedCanonical = requestedCanonical ?? d.modelCatalog.defaultSession;
|
||||
let modelWarning: string | undefined;
|
||||
if (selectedCanonical && d.modelCatalog.defaultSession && !d.modelCatalog.hasSession(selectedCanonical)) {
|
||||
selectedCanonical = d.modelCatalog.defaultSession;
|
||||
modelWarning = `Configured model ${requestedCanonical ?? "selection"} is unavailable; using ${selectedCanonical}.`;
|
||||
const selectedCanonical = requestedCanonical ?? d.modelCatalog.defaultInteraction;
|
||||
if (selectedCanonical && d.modelCatalog.defaultInteraction && !d.modelCatalog.hasSession(selectedCanonical)) {
|
||||
return reply.code(503).send({ error: MODEL_UNAVAILABLE_MESSAGE, code: "model_unavailable" });
|
||||
}
|
||||
const selected = selectedCanonical ? splitCanonicalModelId(selectedCanonical) : undefined;
|
||||
const provider = selected?.provider ?? b.provider;
|
||||
@@ -484,6 +513,7 @@ export function sessionRoutes(
|
||||
({ id } = await runner.sessionNew({
|
||||
question: b.question, name: b.name, workspaceConfigPath,
|
||||
workspaceId, workspaceRevision, provider, model, thinking,
|
||||
interactionLanguage: language,
|
||||
}));
|
||||
manifestPersisted = true;
|
||||
if (revisionLease) {
|
||||
@@ -497,6 +527,7 @@ export function sessionRoutes(
|
||||
} catch { return storageFailure(reply); }
|
||||
const options = {
|
||||
provider, model, thinking,
|
||||
interactionLanguage: language,
|
||||
author: principal.displayName ?? principal.subject,
|
||||
principal,
|
||||
question: b.question,
|
||||
@@ -533,7 +564,7 @@ export function sessionRoutes(
|
||||
),
|
||||
() => d.mgr.start(id, rt, runtimeOptions),
|
||||
);
|
||||
return { id, ...(modelWarning ? { warning: modelWarning } : {}) };
|
||||
return { id };
|
||||
} finally {
|
||||
if (revisionLease && !manifestPersisted) {
|
||||
await revisionLease.abort().catch((error: unknown) => {
|
||||
@@ -559,8 +590,8 @@ export function sessionRoutes(
|
||||
? { ...principal, isAdmin: false }
|
||||
: ownershipPrincipal(principal, "session.read_all");
|
||||
const runner = runnerFor(scopedPrincipal);
|
||||
const revisions = await sessionRevisions();
|
||||
const lists = await Promise.all(revisions
|
||||
const revisionScan = await sessionRevisions();
|
||||
const lists = await Promise.all(revisionScan.revisions
|
||||
.map((revision) => runner.sessionList(revision.snapshotPath) as Promise<SessionRow[]>));
|
||||
const sessions = new Map<string, SessionRow>();
|
||||
for (const row of lists.flat()) {
|
||||
@@ -570,7 +601,8 @@ export function sessionRoutes(
|
||||
// Only an administrator-visible complete list (or the single local principal) is safe
|
||||
// input for retention. A remote per-user view can never discard another principal's pin.
|
||||
const reconcileSnapshotRetention = (d.workspaceRegistry as Partial<WorkspaceRegistry>).reconcileSnapshotRetention;
|
||||
const hasCompleteRetentionView = (scope === "all" || principal.issuer === "local")
|
||||
const hasCompleteRetentionView = revisionScan.retainedComplete
|
||||
&& (scope === "all" || principal.issuer === "local")
|
||||
&& hasPermission(principal, "session.read_all");
|
||||
if (hasCompleteRetentionView && typeof reconcileSnapshotRetention === "function") {
|
||||
const retained = [...new Set(list
|
||||
@@ -604,6 +636,23 @@ export function sessionRoutes(
|
||||
} catch (error) { return lifecycleFailure(reply, error); }
|
||||
const rt = d.mgr.get(id);
|
||||
if (!rt) return reply.code(404).send({ error: "sessione non attiva" });
|
||||
const pending = rt.bridge.pendingWidget?.() as any;
|
||||
const response = (req.body as any)?.ui_response;
|
||||
if (pending?.widget === "archive-repair" && response?.id === pending.id && !response.control) {
|
||||
const choices = response.choices;
|
||||
if (!Array.isArray(choices) || choices.length !== 1)
|
||||
return reply.code(400).send({ error: "Select one archive repair choice" });
|
||||
if (choices[0] !== "continue" && choices[0] !== "reject") {
|
||||
const option = pending.repair?.options?.find((item: any) => item.id === choices[0]);
|
||||
if (!option || !["memory", "evidence"].includes(option.archive))
|
||||
return reply.code(400).send({ error: "Unknown archive repair choice" });
|
||||
const permission = option.archive === "memory" ? "memory.manage" : "evidence.manage";
|
||||
if (!principal.isAdmin || !hasPermission(principal, permission))
|
||||
return reply.code(403).send({ error: "Archive corrections require an administrator" });
|
||||
if (rt.ownerKey !== `${principal.issuer}\0${principal.subject}`)
|
||||
return reply.code(409).send({ error: "Resume the session with your account before correcting an archive" });
|
||||
}
|
||||
}
|
||||
if (!rt.bridge.respond((req.body as any).ui_response)) {
|
||||
return reply.code(409).send({ error: "risposta non corrispondente al gate in attesa" });
|
||||
}
|
||||
@@ -623,6 +672,13 @@ export function sessionRoutes(
|
||||
app.post("/sessions/:id/resume", async (req, reply) => {
|
||||
const id = (req.params as any).id;
|
||||
const principal = getPrincipal(req);
|
||||
if (Object.hasOwn(req.body ?? {}, "interactionLanguage")
|
||||
|| Object.hasOwn(req.body ?? {}, "interaction_language")) {
|
||||
return reply.code(400).send({
|
||||
code: "interaction_language_pinned",
|
||||
error: "Resume uses the session's persisted interaction language; overrides are not accepted",
|
||||
});
|
||||
}
|
||||
return withSessionLifecycle(id, async () => {
|
||||
let settings: Settings;
|
||||
let located: LocatedSession | undefined;
|
||||
@@ -640,11 +696,19 @@ export function sessionRoutes(
|
||||
const saved = manifest as {
|
||||
provider?: string; model?: string; thinking?: string;
|
||||
workspace_id?: string; workspace_revision?: string;
|
||||
interaction_language?: string | null;
|
||||
};
|
||||
const savedCanonical = saved.provider && saved.model ? `${saved.provider}/${saved.model}` : "";
|
||||
if (d.modelCatalog.defaultSession && (!savedCanonical || !d.modelCatalog.hasSession(savedCanonical))) {
|
||||
const requested = (req.body ?? {}) as { provider?: string; model?: string; thinking?: string };
|
||||
if ((requested.provider === undefined) !== (requested.model === undefined)
|
||||
|| (requested.provider !== undefined && (typeof requested.provider !== "string" || typeof requested.model !== "string" || !requested.provider || !requested.model))) {
|
||||
return reply.code(400).send({ error: "provider and model must be supplied together" });
|
||||
}
|
||||
const selectedCanonical = requested.provider && requested.model
|
||||
? `${requested.provider}/${requested.model}` : d.modelCatalog.defaultInteraction;
|
||||
if (d.modelCatalog.defaultInteraction && (!selectedCanonical || !d.modelCatalog.hasSession(selectedCanonical))) {
|
||||
return reply.code(503).send({ error: MODEL_UNAVAILABLE_MESSAGE, code: "model_unavailable" });
|
||||
}
|
||||
const selected = selectedCanonical ? splitCanonicalModelId(selectedCanonical) : saved;
|
||||
let workspaceConfigPath: string;
|
||||
let workspaceDescriptor: WorkspaceDescriptor | undefined;
|
||||
try {
|
||||
@@ -671,13 +735,19 @@ export function sessionRoutes(
|
||||
});
|
||||
}
|
||||
try { settings = await d.getSettings(principal); } catch { return storageFailure(reply); }
|
||||
let language = saved.interaction_language;
|
||||
if (language == null) {
|
||||
try {
|
||||
language = (await runner.ensureInteractionLanguage(id, workspaceConfigPath)).interaction_language;
|
||||
} catch { return reply.code(503).send({ error: RESUME_FAILURE_MESSAGE }); }
|
||||
}
|
||||
// This check belongs inside the per-session lock: a preceding cold Resume may have
|
||||
// installed a running runtime while this request was waiting.
|
||||
const existing = d.mgr.get(id);
|
||||
if (existing) {
|
||||
const state = existing.bridge.turnState();
|
||||
if (state === "running" || state === "waiting") {
|
||||
return reply.code(200).send({ id, alreadyActive: true });
|
||||
return reply.code(200).send({ id, alreadyActive: true, workspaceId: saved.workspace_id });
|
||||
}
|
||||
}
|
||||
const ensure = await d.readiness.ensure(
|
||||
@@ -688,8 +758,9 @@ export function sessionRoutes(
|
||||
...(ensure.code ? { code: ensure.code } : {}),
|
||||
});
|
||||
const options = {
|
||||
provider: saved?.provider,
|
||||
model: saved?.model,
|
||||
provider: selected.provider,
|
||||
interactionLanguage: language,
|
||||
model: selected.model,
|
||||
thinking: saved?.thinking ?? settings.thinking,
|
||||
author: principal.displayName ?? principal.subject,
|
||||
principal,
|
||||
@@ -711,7 +782,7 @@ export function sessionRoutes(
|
||||
if (current) {
|
||||
const state = current.bridge.turnState();
|
||||
if (state === "running" || state === "waiting") {
|
||||
return reply.code(200).send({ id, alreadyActive: true });
|
||||
return reply.code(200).send({ id, alreadyActive: true, workspaceId: saved.workspace_id });
|
||||
}
|
||||
}
|
||||
|
||||
@@ -752,7 +823,7 @@ export function sessionRoutes(
|
||||
d.mgr.configure(rt, runtimeOptions), null,
|
||||
() => d.mgr.start(id, rt, runtimeOptions),
|
||||
);
|
||||
return reply.code(200).send({ id, alreadyActive: false });
|
||||
return reply.code(200).send({ id, alreadyActive: false, workspaceId: saved.workspace_id });
|
||||
});
|
||||
});
|
||||
app.post("/sessions/:id/close", async (req, reply) => {
|
||||
|
||||
@@ -16,8 +16,8 @@ export function effectiveSettings(
|
||||
modelCatalog?: RuntimeModelCatalog,
|
||||
): Settings {
|
||||
const workspaces = listWorkspaces(cfg.harnessDir);
|
||||
const selected = modelCatalog?.defaultSession
|
||||
? splitCanonicalModelId(modelCatalog.defaultSession)
|
||||
const selected = modelCatalog?.defaultInteraction
|
||||
? splitCanonicalModelId(modelCatalog.defaultInteraction)
|
||||
: undefined;
|
||||
return {
|
||||
workspace: stored.workspace ?? workspaces[0]?.name,
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
/** Validate/canonicalize the tag; available translation catalogs belong to the UI. */
|
||||
export function interactionLanguage(value: unknown): string | undefined {
|
||||
if (typeof value !== "string") return undefined;
|
||||
try {
|
||||
const [canonical] = Intl.getCanonicalLocales(value);
|
||||
return canonical;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
@@ -59,6 +59,7 @@ export interface SessionRow {
|
||||
author: string | null;
|
||||
workspace_id?: string | null;
|
||||
workspace_revision?: string | null;
|
||||
interaction_language?: string | null;
|
||||
archived?: boolean;
|
||||
}
|
||||
|
||||
@@ -482,6 +483,7 @@ export class ThtRunner {
|
||||
|
||||
async sessionNew(o: {
|
||||
question: string;
|
||||
interactionLanguage?: string;
|
||||
provider?: string;
|
||||
model?: string;
|
||||
thinking?: string;
|
||||
@@ -497,6 +499,7 @@ export class ThtRunner {
|
||||
["--provider", o.provider],
|
||||
["--model", o.model],
|
||||
["--thinking", o.thinking],
|
||||
["--interaction-language", o.interactionLanguage],
|
||||
["--name", o.name],
|
||||
["--workspace-id", o.workspaceId],
|
||||
["--workspace-revision", o.workspaceRevision],
|
||||
@@ -533,6 +536,12 @@ export class ThtRunner {
|
||||
return this.json<unknown>(["session", "show", id, "--json"], workspace);
|
||||
}
|
||||
|
||||
ensureInteractionLanguage(id: string, workspace?: string) {
|
||||
return this.json<{ interaction_language: string }>(
|
||||
["session", "ensure-interaction-language", id, "--json"], workspace,
|
||||
);
|
||||
}
|
||||
|
||||
sqlPreview(id: string, p: { limit?: number; offset?: number }, workspace?: string) {
|
||||
// No positional FILE: the harness resolves sql_final.sql from the session
|
||||
// via _session_sql_file(cfg, session_id), which respects the workspace path.
|
||||
|
||||
@@ -18,7 +18,7 @@ export interface WorkspaceMaintenanceIo {
|
||||
writeStderr(value: string): void;
|
||||
}
|
||||
|
||||
type Command = "inspect" | "preprocess-run" | "preprocess-clear";
|
||||
type Command = "inspect" | "preprocess-run" | "preprocess-clear" | "evidence-consolidate" | "evidence-refresh" | "evidence-decide";
|
||||
|
||||
function failureResult(
|
||||
operation: string,
|
||||
@@ -65,6 +65,9 @@ function parseRequest(command: string, stdin: string): Record<string, unknown> {
|
||||
inspect: ["schemaVersion", "workspaceId"],
|
||||
"preprocess-run": ["schemaVersion", "workspaceId"],
|
||||
"preprocess-clear": ["schemaVersion", "workspaceId"],
|
||||
"evidence-consolidate": ["schemaVersion", "workspaceId"],
|
||||
"evidence-refresh": ["schemaVersion", "workspaceId"],
|
||||
"evidence-decide": ["schemaVersion", "workspaceId", "sourceId", "revision", "decision"],
|
||||
};
|
||||
const allowed = allowedByCommand[command];
|
||||
if (!allowed) throw new Error("unknown command");
|
||||
@@ -81,6 +84,16 @@ function exitCodeFor(result: WorkspaceOperationResult): number {
|
||||
|
||||
async function dispatch(command: Command, service: WorkspacePreprocessingService, request: Record<string, unknown>): Promise<WorkspaceOperationResult> {
|
||||
switch (command) {
|
||||
case "evidence-refresh":
|
||||
return await service.evidenceSources({ workspaceId: request.workspaceId as string, action: "refresh" });
|
||||
case "evidence-decide":
|
||||
if (typeof request.sourceId !== "string" || !/^[a-f0-9]{64}$/.test(request.sourceId)
|
||||
|| typeof request.revision !== "string" || !/^[a-f0-9]{64}$/.test(request.revision)
|
||||
|| (request.decision !== "keep" && request.decision !== "replace")) throw new Error("invalid request");
|
||||
return await service.evidenceSources({ workspaceId: request.workspaceId as string, action: "decide",
|
||||
sourceId: request.sourceId, revision: request.revision, decision: request.decision });
|
||||
case "evidence-consolidate":
|
||||
return await service.consolidateEvidence({ workspaceId: request.workspaceId as string });
|
||||
case "inspect":
|
||||
return await service.inspect({ workspaceId: request.workspaceId as string });
|
||||
case "preprocess-run":
|
||||
@@ -127,7 +140,8 @@ export async function runWorkspaceMaintenanceCli(
|
||||
|| message === "unexpected request field"
|
||||
|| message === "invalid workspace id";
|
||||
return command in {
|
||||
inspect: true, "preprocess-run": true, "preprocess-clear": true,
|
||||
inspect: true, "preprocess-run": true, "preprocess-clear": true, "evidence-consolidate": true,
|
||||
"evidence-refresh": true, "evidence-decide": true,
|
||||
} ? (requestError ? 2 : 1) : 2;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -22,6 +22,7 @@ export interface EvidencePreprocessingRequest {
|
||||
evidence: EvidenceConfig;
|
||||
job: EvidenceJobState;
|
||||
dryRun?: boolean;
|
||||
consolidate?: boolean;
|
||||
httpPrivateHostAllowlist?: readonly string[];
|
||||
}
|
||||
|
||||
@@ -56,7 +57,7 @@ function isPrivateHost(hostname: string): boolean {
|
||||
return hostname.endsWith(".internal");
|
||||
}
|
||||
|
||||
function evidencePolicy(
|
||||
export function evidencePolicy(
|
||||
evidence: EvidenceConfig,
|
||||
httpPrivateHostAllowlist?: readonly string[],
|
||||
): EvidencePreprocessingOutcome | undefined {
|
||||
@@ -95,13 +96,14 @@ function jobResult(job: EvidenceJobState): Pick<
|
||||
};
|
||||
}
|
||||
|
||||
async function runEvidenceStage(
|
||||
export async function runEvidenceStage(
|
||||
request: EvidencePreprocessingRequest,
|
||||
deps: EvidencePreprocessingDependencies,
|
||||
): Promise<EvidencePreprocessingOutcome> {
|
||||
const payload = await deps.runStage([
|
||||
"preprocess",
|
||||
"evidence",
|
||||
...(request.consolidate ? ["--consolidate"] : []),
|
||||
...(request.dryRun ? ["--dry-run"] : []),
|
||||
...(request.job.childRuns.evidence
|
||||
? ["--resume", request.job.childRuns.evidence]
|
||||
|
||||
@@ -3,6 +3,8 @@ import { renameSync, rmSync, writeFileSync, mkdirSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import {
|
||||
continueEvidencePreprocessing,
|
||||
evidencePolicy,
|
||||
runEvidenceStage,
|
||||
type EvidencePreprocessingDependencies,
|
||||
type EvidencePreprocessingOutcome,
|
||||
} from "./evidence/preprocessing.js";
|
||||
@@ -108,6 +110,72 @@ function baseResult(
|
||||
export class WorkspacePreprocessingService {
|
||||
constructor(private readonly deps: WorkspacePreprocessingServiceDeps) {}
|
||||
|
||||
async evidenceSources(options: { workspaceId: string; action: "refresh" | "decide";
|
||||
sourceId?: string; revision?: string; decision?: "keep" | "replace"; actor?: string }): Promise<WorkspaceOperationResult> {
|
||||
const runtime = await this.deps.acquireActiveRuntime(options.workspaceId);
|
||||
try {
|
||||
if (options.action === "refresh") {
|
||||
const refused = evidencePolicy(runtime.workspace.evidence, this.deps.httpPrivateHostAllowlist);
|
||||
if (refused) return baseResult(runtime, "evidence refresh", "failed", refused.code);
|
||||
}
|
||||
const argv = ["evidence", "sources", options.action, "--json", "-c", "/dev/fd/3"];
|
||||
if (options.action === "decide") {
|
||||
if (!/^[a-f0-9]{64}$/.test(options.sourceId ?? "") || !/^[a-f0-9]{64}$/.test(options.revision ?? "")
|
||||
|| !["keep", "replace"].includes(options.decision ?? "")) throw new Error("Invalid source decision");
|
||||
const preflight = await this.deps.evidencePreflight(runtime.workspace);
|
||||
if (!preflight.ok) return baseResult(runtime, "evidence sources", "failed", preflight.code);
|
||||
argv.push("--source-id", options.sourceId!, "--revision", options.revision!, "--decision", options.decision!);
|
||||
}
|
||||
argv.push("--actor", options.actor ?? "installation operator");
|
||||
const result = await this.deps.runChild({ argv, configPath: runtime.configLease.path });
|
||||
const payload = JSON.parse(result.stdout);
|
||||
if (result.exitCode !== 0 || payload.status !== "succeeded") throw new Error(
|
||||
typeof payload.error === "string" ? payload.error.slice(0, 1500) : "Evidence source operation failed");
|
||||
return baseResult(runtime, `evidence ${options.action}`, "succeeded", "ok", {
|
||||
counts: this.numberRecord(payload.counts),
|
||||
warnings: [options.action === "refresh" ? "Source comparisons are ready in Evidence management. Active content is unchanged."
|
||||
: "Source decision activated locally. Commit and push the Evidence tree manually."],
|
||||
});
|
||||
} catch (error) {
|
||||
return baseResult(runtime, `evidence ${options.action}`, "failed", "evidence_materialization_required", {
|
||||
warnings: [error instanceof Error ? error.message : "Evidence source operation failed"],
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
async consolidateEvidence(options: { workspaceId: string }): Promise<WorkspaceOperationResult> {
|
||||
const runtime = await this.deps.acquireActiveRuntime(options.workspaceId);
|
||||
const preflight = await this.deps.evidencePreflight(runtime.workspace);
|
||||
if (!preflight.ok) return baseResult(runtime, "evidence consolidate", "failed", preflight.code);
|
||||
// Evidence has its own durable archive/corpus jobs. Do not alter Catalog readiness
|
||||
// or the full preprocessing job when publishing this one component.
|
||||
try {
|
||||
const outcome = await runEvidenceStage({ evidence: runtime.workspace.evidence,
|
||||
consolidate: true, job: { runId: randomBytes(16).toString("hex"), completedStages: [], childRuns: {} },
|
||||
}, {
|
||||
runStage: async argv => {
|
||||
const result = await this.deps.runChild({ argv, configPath: runtime.configLease.path });
|
||||
const payload = JSON.parse(result.stdout);
|
||||
if (result.exitCode !== 0 || payload.status !== "succeeded") {
|
||||
return Promise.reject(new Error(typeof payload.error === "string" ? payload.error.slice(0, 1500) : "Evidence consolidation failed. Retry the command."));
|
||||
}
|
||||
return payload;
|
||||
},
|
||||
persistJob: () => undefined,
|
||||
evidencePreflight: async () => preflight,
|
||||
requireRunId: value => this.requireRunId(value),
|
||||
numberRecord: value => this.numberRecord(value),
|
||||
});
|
||||
return baseResult(runtime, "evidence consolidate", "succeeded", "ok", {
|
||||
...outcome, warnings: ["Evidence is active locally. Catalog and Schema readiness are unchanged. Commit and push the Evidence files manually."],
|
||||
});
|
||||
} catch (error) {
|
||||
return baseResult(runtime, "evidence consolidate", "failed", "evidence_materialization_required", {
|
||||
warnings: [error instanceof Error ? error.message : "Evidence consolidation failed. Retry the command."],
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
async inspect(options: { workspaceId: string }): Promise<WorkspaceOperationResult> {
|
||||
try {
|
||||
const runtime = await this.deps.acquireActiveRuntime(options.workspaceId);
|
||||
|
||||
@@ -497,7 +497,10 @@ export async function publishDeterministicRuntimeConfigLease(options: {
|
||||
rendered.workspaceRevision,
|
||||
renderedConfigObject,
|
||||
);
|
||||
const identitySuffix = inputFingerprintValue.slice(7, 23);
|
||||
// The Catalog input identity intentionally excludes representation-only changes.
|
||||
// A lease also identifies its bytes, so a new renderer never collides with an
|
||||
// immutable config produced by an earlier release for the same Catalog inputs.
|
||||
const identitySuffix = sha256(inputFingerprintValue + "\n" + publishedConfig).slice(7, 23);
|
||||
|
||||
const preprocessingRoot = ensureTrustedDirectory(join(
|
||||
options.dataRoot,
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { basename, join } from "node:path";
|
||||
import { basename, dirname, join } from "node:path";
|
||||
import { stringify } from "yaml";
|
||||
import { buildInstallationContract } from "./contracts.js";
|
||||
import { validateWorkspaceDescriptor, type WorkspaceDescriptor } from "./schema.js";
|
||||
@@ -179,6 +179,7 @@ function renderEvidence(
|
||||
return {
|
||||
evidence: {
|
||||
...(workspace.evidence.schema_version === 2 ? { schema_version: 2 } : {}),
|
||||
local_archive_root: join(dirname(dirname(context.revisionContentRoot)), "repo", context.workspaceId),
|
||||
sources: [renderedSource],
|
||||
},
|
||||
vector: {
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
import Fastify from "fastify";
|
||||
import { expect, test, vi } from "vitest";
|
||||
import { sessionRoutes } from "../src/routes/sessions.js";
|
||||
import type { PrincipalContext } from "../src/auth/principal.js";
|
||||
|
||||
test.each([
|
||||
[false, "m", 403], [false, "e", 403], [false, "reject", 204],
|
||||
[true, "m", 204], [true, "e", 204], [true, "forged", 400],
|
||||
] as const)("archive repair response enforces the responding principal (%s, %s)", async (admin, choice, status) => {
|
||||
const app = Fastify();
|
||||
const principal: PrincipalContext = { issuer: "test", subject: "reviewer", isAdmin: admin,
|
||||
roles: admin ? ["admin"] : ["user"],
|
||||
permissions: admin ? ["session.use", "memory.manage", "evidence.manage"] : ["session.use"] };
|
||||
app.addHook("preHandler", async request => { request.principal = principal; });
|
||||
const respond = vi.fn(() => true);
|
||||
sessionRoutes(app, {
|
||||
tht: {}, mgr: { get: () => ({ ownerKey: "test\0reviewer", bridge: {
|
||||
respond, pendingWidget: () => ({ id: "gate", widget: "archive-repair", repair: { options: [
|
||||
{ id: "m", archive: "memory" }, { id: "e", archive: "evidence" },
|
||||
] } }),
|
||||
} }) },
|
||||
} as unknown as Parameters<typeof sessionRoutes>[1]);
|
||||
try {
|
||||
const result = await app.inject({ method: "POST", url: "/sessions/s/response",
|
||||
payload: { ui_response: { id: "gate", choices: [choice] } } });
|
||||
expect(result.statusCode).toBe(status);
|
||||
expect(respond).toHaveBeenCalledTimes(status === 204 ? 1 : 0);
|
||||
} finally { await app.close(); }
|
||||
});
|
||||
|
||||
test("an administrator cannot attribute a repair to another runtime's principal", async () => {
|
||||
const app = Fastify();
|
||||
app.addHook("preHandler", async request => { request.principal = {
|
||||
issuer: "test", subject: "second-admin", isAdmin: true, roles: ["admin"],
|
||||
permissions: ["session.use", "memory.manage"],
|
||||
}; });
|
||||
const respond = vi.fn();
|
||||
sessionRoutes(app, { tht: {}, mgr: { get: () => ({ ownerKey: "test\0first-admin", bridge: {
|
||||
respond, pendingWidget: () => ({ id: "gate", widget: "archive-repair",
|
||||
repair: { options: [{ id: "m", archive: "memory" }] } }),
|
||||
} }) } } as unknown as Parameters<typeof sessionRoutes>[1]);
|
||||
try {
|
||||
const result = await app.inject({ method: "POST", url: "/sessions/s/response",
|
||||
payload: { ui_response: { id: "gate", choices: ["m"] } } });
|
||||
expect(result.statusCode).toBe(409);
|
||||
expect(respond).not.toHaveBeenCalled();
|
||||
} finally { await app.close(); }
|
||||
});
|
||||
@@ -188,6 +188,8 @@ test("roles collapse duplicates and admin contains all administrative permission
|
||||
"workspace.manage",
|
||||
"workspace.secrets.manage",
|
||||
"database.manage",
|
||||
"memory.manage",
|
||||
"evidence.manage",
|
||||
"pi.manage",
|
||||
"auth.diagnostics.read",
|
||||
]);
|
||||
|
||||
@@ -130,7 +130,7 @@ test("local login sets a non-persistent opaque session cookie and exposes only a
|
||||
roles: ["admin"],
|
||||
permissions: [
|
||||
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
||||
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
|
||||
"workspace.manage", "workspace.secrets.manage", "database.manage", "memory.manage", "evidence.manage", "pi.manage", "auth.diagnostics.read",
|
||||
],
|
||||
isAdmin: true,
|
||||
csrfToken: expect.stringMatching(/^[A-Za-z0-9_-]{43}$/),
|
||||
|
||||
@@ -7,6 +7,8 @@ import { join } from "node:path";
|
||||
import { expandLocalHome, localPrincipal, upstreamPrincipal } from "../src/auth/principal.js";
|
||||
import { buildApp } from "../src/app.js";
|
||||
import { loadConfig } from "../src/config.js";
|
||||
import { rolesToPermissions } from "../src/auth/config.js";
|
||||
import type { Permission, Role } from "../src/auth/types.js";
|
||||
|
||||
test("server smoke rejects retired trusted claims under OIDC authentication", () => {
|
||||
const smoke = readFileSync("../scripts/unified-deployment-smoke.sh", "utf8");
|
||||
@@ -32,7 +34,7 @@ test("local mode resolves a stable local principal", async () => {
|
||||
roles: ["admin"],
|
||||
permissions: [
|
||||
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
||||
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
|
||||
"workspace.manage", "workspace.secrets.manage", "database.manage", "memory.manage", "evidence.manage", "pi.manage", "auth.diagnostics.read",
|
||||
],
|
||||
isAdmin: true,
|
||||
});
|
||||
@@ -74,7 +76,7 @@ test("upstream mode accepts only normalized proxy principal headers", async () =
|
||||
issuer: "portal", subject: "42", displayName: "Alice", roles: ["user", "admin"],
|
||||
permissions: [
|
||||
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
||||
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
|
||||
"workspace.manage", "workspace.secrets.manage", "database.manage", "memory.manage", "evidence.manage", "pi.manage", "auth.diagnostics.read",
|
||||
],
|
||||
isAdmin: true,
|
||||
});
|
||||
@@ -206,15 +208,33 @@ test("the session boundary rejects tht maintenance headers outside exact loopbac
|
||||
})).statusCode).toBe(503);
|
||||
});
|
||||
|
||||
test("the session boundary touches a valid cookie session through the bounded Task 7 store operation", async () => {
|
||||
test.each<{
|
||||
name: string;
|
||||
roles: Role[];
|
||||
storedPermissions: Permission[];
|
||||
}>([
|
||||
{ name: "current user", roles: ["user"], storedPermissions: ["session.use"] },
|
||||
{
|
||||
name: "administrator signed in before archive permissions existed",
|
||||
roles: ["admin"],
|
||||
storedPermissions: rolesToPermissions(["admin"]).filter(
|
||||
(permission) => permission !== "memory.manage" && permission !== "evidence.manage",
|
||||
),
|
||||
},
|
||||
{
|
||||
name: "user with obsolete administrative permissions",
|
||||
roles: ["user"],
|
||||
storedPermissions: ["session.use", "memory.manage", "evidence.manage"],
|
||||
},
|
||||
])("the session boundary derives current permissions and touches a valid cookie: $name", async ({ roles, storedPermissions }) => {
|
||||
const sessions = {
|
||||
resolve: vi.fn(async () => ({
|
||||
version: 1,
|
||||
issuer: "local",
|
||||
subject: "user-1",
|
||||
method: "local",
|
||||
roles: ["user"],
|
||||
permissions: ["session.use"],
|
||||
roles,
|
||||
permissions: storedPermissions,
|
||||
userAuthRevision: 1,
|
||||
authConfigRevision: "b".repeat(64),
|
||||
remembered: false,
|
||||
@@ -249,7 +269,13 @@ test("the session boundary touches a valid cookie session through the bounded Ta
|
||||
app.get("/private", async (request) => getPrincipal(request));
|
||||
|
||||
const token = "z".repeat(43);
|
||||
expect((await app.inject({ method: "GET", url: "/private", headers: { cookie: `thothii_session=${token}` } })).statusCode).toBe(200);
|
||||
const response = await app.inject({ method: "GET", url: "/private", headers: { cookie: `thothii_session=${token}` } });
|
||||
expect(response.statusCode).toBe(200);
|
||||
expect(response.json()).toMatchObject({
|
||||
roles,
|
||||
permissions: rolesToPermissions(roles),
|
||||
isAdmin: roles.includes("admin"),
|
||||
});
|
||||
expect(sessions.touch).toHaveBeenCalledWith(token);
|
||||
});
|
||||
|
||||
|
||||
@@ -124,8 +124,17 @@ test.skipIf(!dockerAvailable)("PostgreSQL migration enforces one database per wo
|
||||
],
|
||||
relationships: [],
|
||||
};
|
||||
expect(await repository.applySchemaSync(created.id, 1, "columns", [], fullColumnsSnapshot))
|
||||
const memoryCleanupRun = await repository.createSyncRun(created.id, "columns", [], 1);
|
||||
await repository.updateSyncRun(memoryCleanupRun.id, { state: "applying", phase: "applying" });
|
||||
await expect(repository.applySchemaSync(created.id, 999, "columns", [], fullColumnsSnapshot, memoryCleanupRun.id))
|
||||
.resolves.toBeUndefined();
|
||||
expect((await repository.getSyncRun(memoryCleanupRun.id))?.phase).toBe("applying");
|
||||
expect(await repository.applySchemaSync(created.id, 1, "columns", [], fullColumnsSnapshot, memoryCleanupRun.id))
|
||||
.toMatchObject({ created: 4, deleted: 0 });
|
||||
expect((await repository.getSyncRun(memoryCleanupRun.id))?.phase).toBe("memory_cleanup");
|
||||
await repository.interruptActiveSyncRuns();
|
||||
expect(await repository.getSyncRun(memoryCleanupRun.id))
|
||||
.toMatchObject({ state: "interrupted", phase: "memory_cleanup" });
|
||||
expect((await repository.listColumns(created.id, patients.id)).map((column) => ({
|
||||
name: column.name,
|
||||
sensitive: column.sensitive,
|
||||
@@ -233,6 +242,7 @@ test.skipIf(!dockerAvailable)("PostgreSQL migration enforces one database per wo
|
||||
});
|
||||
expect(await repository.listSyncRuns(created.id)).toEqual([
|
||||
expect.objectContaining({ id: syncRun.id, tableIds: [patients.id] }),
|
||||
expect.objectContaining({ id: memoryCleanupRun.id, phase: "memory_cleanup" }),
|
||||
]);
|
||||
|
||||
const lockedDatabase = await repository.create({
|
||||
|
||||
@@ -117,6 +117,7 @@ async function setup(env: Record<string, string> = {}) {
|
||||
});
|
||||
const introspector: CatalogSchemaIntrospector = { scan };
|
||||
const operations = new CatalogOperationCoordinator();
|
||||
const cleanup = vi.fn(async () => ({ code: 0, stdout: JSON.stringify({ indexed: true, deleted: 0 }), stderr: "" }));
|
||||
const registry = {
|
||||
list: vi.fn(async () => [revision]),
|
||||
listCatalog: vi.fn(async () => [{ id: "psd-clinical", name: "Policlinico San Donato", configurationState: "ready", revision }]),
|
||||
@@ -124,7 +125,7 @@ async function setup(env: Record<string, string> = {}) {
|
||||
readPinned: vi.fn(async () => ({ workspace, workspaceConfigPath: revision.snapshotPath })),
|
||||
} as unknown as WorkspaceRegistry;
|
||||
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "/missing", NODE_ENV: "test", ...env }), {
|
||||
thtRunner: {} as never,
|
||||
thtRunner: { withPrincipal: () => ({ runWithRuntimeSnapshot: cleanup }) } as never,
|
||||
workspaceRegistry: registry,
|
||||
workspaceSecretStore: new WorkspaceSecretStore({ root: secretRoot, runtimeRoot, installationId: "test" }),
|
||||
catalogRepository: repository,
|
||||
@@ -133,7 +134,7 @@ async function setup(env: Record<string, string> = {}) {
|
||||
workspaceDiagnoser: vi.fn(),
|
||||
});
|
||||
return {
|
||||
app, repository, database: (await repository.get(created.id))!, scan, operations,
|
||||
app, repository, database: (await repository.get(created.id))!, scan, operations, cleanup,
|
||||
setObserved(next: ObservedSchemaSnapshot) { observed = next; },
|
||||
};
|
||||
}
|
||||
@@ -614,6 +615,45 @@ test("waits for confirmation and rescans before applying destructive changes", a
|
||||
expect(scan).toHaveBeenCalledTimes(3);
|
||||
});
|
||||
|
||||
test("retries committed Memory cleanup with the original removals without rescanning", async () => {
|
||||
const { app, repository, database, scan, setObserved, cleanup } = await setup();
|
||||
await seedCatalog(repository, database);
|
||||
const observed = snapshot();
|
||||
observed.tables = observed.tables.filter(t => t.name !== "visits");
|
||||
observed.columns = observed.columns.filter(c => c.tableName !== "visits");
|
||||
observed.relationships = [];
|
||||
setObserved(observed);
|
||||
cleanup.mockResolvedValueOnce({ code: 0, stdout: JSON.stringify({ indexed: false, deleted: 2 }), stderr: "" });
|
||||
const started = await app.inject({ method: "POST", url: `/catalog/databases/${database.id}/sync-runs`,
|
||||
payload: { version: database.version, scope: "all", tableIds: [] } });
|
||||
const waiting = await waitFor(repository, started.json().id, "awaiting_confirmation");
|
||||
expect(cleanup).not.toHaveBeenCalled();
|
||||
await app.inject({ method: "POST", url: `/catalog/sync-runs/${waiting.id}/confirm`,
|
||||
payload: { confirmationToken: waiting.confirmationToken } });
|
||||
const failed = await waitFor(repository, waiting.id, "failed");
|
||||
expect(failed).toMatchObject({ phase: "memory_cleanup", errorCode: "memory_cleanup_pending",
|
||||
plannedDiff: { deletedTables: ["visits"] } });
|
||||
expect((await repository.listTables(database.id)).map(t => t.name)).toEqual(["patients"]);
|
||||
const callsBeforeRetry = scan.mock.calls.length;
|
||||
const blocked = await app.inject({ method: "POST", url: `/catalog/databases/${database.id}/sync-runs`,
|
||||
payload: { version: database.version, scope: "all", tableIds: [] } });
|
||||
expect(blocked.statusCode).toBe(409);
|
||||
// Simulate startup recovery and an unreachable DWH after schema application.
|
||||
await repository.interruptActiveSyncRuns();
|
||||
scan.mockRejectedValue(new Error("DWH unavailable"));
|
||||
cleanup.mockResolvedValue({ code: 0, stdout: JSON.stringify({ indexed: true, deleted: 2 }), stderr: "" });
|
||||
const retried = await app.inject({ method: "POST", url: `/catalog/sync-runs/${waiting.id}/retry` });
|
||||
expect(retried.statusCode).toBe(202);
|
||||
const completed = await waitFor(repository, waiting.id, "succeeded");
|
||||
expect(completed.counts).toMatchObject({ memoryDeleted: 2 });
|
||||
expect(scan.mock.calls.length).toBe(callsBeforeRetry);
|
||||
expect(cleanup).toHaveBeenCalledTimes(2);
|
||||
expect(cleanup.mock.calls[1]).toEqual(cleanup.mock.calls[0]);
|
||||
const request = JSON.parse((cleanup.mock.calls[0] as unknown as string[])[1]!).request;
|
||||
expect(request).toMatchObject({ sync_id: waiting.id, database: "warehouse",
|
||||
schema_name: "datawarehouse", removed_tables: ["visits"] });
|
||||
});
|
||||
|
||||
test("deletes every catalog table for multiple selected databases and cascades dependent metadata", async () => {
|
||||
const { app, repository, database } = await setup();
|
||||
const second = await repository.create({
|
||||
|
||||
@@ -36,7 +36,7 @@ test("loop F1: crea sessione → SSE riceve il widget → risponde → il modell
|
||||
ollamaEnsure: async () => ({ ok: true }),
|
||||
searchPack: async () => {},
|
||||
sessionNew: async () => ({ id: "s1" }),
|
||||
sessionShow: async (_id: string) => ({ id: "s1", provider: undefined, model: undefined, thinking: undefined }),
|
||||
sessionShow: async (_id: string) => ({ id: "s1", interaction_language: "en", provider: undefined, model: undefined, thinking: undefined }),
|
||||
sessionList: async () => [],
|
||||
} as any,
|
||||
spawnFn: () => spawn("node", [FAKE, SCRIPT]) as any,
|
||||
@@ -48,7 +48,7 @@ test("loop F1: crea sessione → SSE riceve il widget → risponde → il modell
|
||||
const created = await fetch(`${base}/sessions`, {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify({ workspace: "w", question: "q" }),
|
||||
body: JSON.stringify({ workspace: "w", question: "q", interactionLanguage: "en" }),
|
||||
});
|
||||
expect(created.status).toBe(200);
|
||||
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
import Fastify from "fastify";
|
||||
import { afterEach, expect, test, vi } from "vitest";
|
||||
import { evidenceRoutes } from "../src/routes/evidence.js";
|
||||
import type { ThtRunner } from "../src/tht/tht-runner.js";
|
||||
import type { PrincipalContext } from "../src/auth/principal.js";
|
||||
|
||||
const apps: ReturnType<typeof Fastify>[] = [];
|
||||
afterEach(async () => { await Promise.all(apps.splice(0).map(app => app.close())); });
|
||||
function setup(admin = true) {
|
||||
const app = Fastify(); apps.push(app);
|
||||
const principal: PrincipalContext = { issuer: "test", subject: "curator", roles: [admin ? "admin" : "user"],
|
||||
permissions: admin ? ["evidence.manage"] : ["session.use"], isAdmin: admin };
|
||||
app.addHook("preHandler", async request => { request.principal = principal; });
|
||||
const run = vi.fn(async () => ({ code: 0, stdout: JSON.stringify({ items: [], total: 0 }), stderr: "" }));
|
||||
const withPrincipal = vi.fn(() => ({ runWithRuntimeSnapshot: run }) as unknown as ThtRunner);
|
||||
const consolidateEvidence = vi.fn();
|
||||
const evidenceSources = vi.fn().mockResolvedValue({status: "succeeded"});
|
||||
evidenceRoutes(app, { runner: { withPrincipal }, registryRoot: "/data/registry", hostRegistryRoot: "/srv/Thoth workspaces",
|
||||
registry: { list: async () => [{ id: "sales", commit: "a".repeat(40), blob: "b".repeat(40), snapshotPath: "/missing" }] },
|
||||
service: { consolidateEvidence, evidenceSources } });
|
||||
return { app, run, withPrincipal, principal, consolidateEvidence, evidenceSources };
|
||||
}
|
||||
test("browses complete local files with host paths, without a database runtime", async () => {
|
||||
const { app, run, withPrincipal, principal } = setup();
|
||||
const response = await app.inject("/workspaces/sales/evidence?kind=domain&concept=orders&page=2");
|
||||
expect(response.statusCode).toBe(200);
|
||||
expect(withPrincipal).toHaveBeenCalledWith(principal);
|
||||
const [argv, snapshot] = run.mock.calls[0] as unknown as [string[], string];
|
||||
expect(argv).toEqual(["evidence", "admin", "--workspace", "sales"]);
|
||||
expect(JSON.parse(snapshot)).toEqual({ root: "/data/registry/repo/sales", query: { kind: "domain", concept: "orders", page: 2 } });
|
||||
expect(response.json().location.workspace).toBe("/srv/Thoth workspaces/repo/sales");
|
||||
expect(response.json().location.git_commands[0]).toContain("git -C '/srv/Thoth workspaces/repo'");
|
||||
});
|
||||
|
||||
test("source actions require admin, bind the actor, and reject cross-workspace or arbitrary inputs", async () => {
|
||||
const denied = setup(false);
|
||||
expect((await denied.app.inject({method: "POST", url: "/workspaces/sales/evidence/sources", payload: {action: "refresh"}})).statusCode).toBe(403);
|
||||
expect(denied.evidenceSources).not.toHaveBeenCalled();
|
||||
const allowed = setup();
|
||||
expect((await allowed.app.inject({method: "POST", url: "/workspaces/sales/evidence/sources", payload: {action: "refresh"}})).statusCode).toBe(200);
|
||||
expect(allowed.evidenceSources).toHaveBeenCalledWith({workspaceId: "sales", action: "refresh", actor: "curator"});
|
||||
const choice = {action: "decide", sourceId: "a".repeat(64), revision: "b".repeat(64), decision: "keep"};
|
||||
expect((await allowed.app.inject({method: "POST", url: "/workspaces/sales/evidence/sources", payload: choice})).statusCode).toBe(200);
|
||||
expect(allowed.evidenceSources).toHaveBeenLastCalledWith({...choice, workspaceId: "sales", actor: "curator"});
|
||||
for (const payload of [{action: "refresh", url: "http://localhost"}, {...choice, actor: "forged"}, {...choice, revision: "../other"}]) {
|
||||
expect((await allowed.app.inject({method: "POST", url: "/workspaces/sales/evidence/sources", payload})).statusCode).toBe(400);
|
||||
}
|
||||
expect((await allowed.app.inject({method: "POST", url: "/workspaces/other/evidence/sources", payload: choice})).statusCode).toBe(404);
|
||||
expect(allowed.evidenceSources).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
test.each(["/", "/evidence:rule", "/consolidate"])("refuses non-admin access %s", async suffix => {
|
||||
const { app, run, consolidateEvidence } = setup(false);
|
||||
const response = await app.inject({ method: suffix === "/consolidate" ? "POST" : "GET", url: `/workspaces/sales/evidence${suffix === "/" ? "" : suffix}` });
|
||||
expect(response.statusCode).toBe(403); expect(run).not.toHaveBeenCalled(); expect(consolidateEvidence).not.toHaveBeenCalled();
|
||||
});
|
||||
test("rejects workspace escape, unknown workspace and unsupported filters", async () => {
|
||||
const { app, run } = setup();
|
||||
expect((await app.inject("/workspaces/foreign/evidence")).statusCode).toBe(404);
|
||||
expect((await app.inject("/workspaces/sales/evidence?root=/private")).statusCode).toBe(400);
|
||||
expect((await app.inject("/workspaces/sales/evidence?page_size=101")).statusCode).toBe(400);
|
||||
expect(run).not.toHaveBeenCalled();
|
||||
});
|
||||
test("a missing unit is 404 and consolidation preserves the partial outcome", async () => {
|
||||
const { app, consolidateEvidence } = setup();
|
||||
expect((await app.inject("/workspaces/sales/evidence/evidence:missing")).statusCode).toBe(404);
|
||||
consolidateEvidence.mockResolvedValue({ status: "failed", warnings: ["Saved; retry indexing."] });
|
||||
const response = await app.inject({ method: "POST", url: "/workspaces/sales/evidence/consolidate" });
|
||||
expect(response.json()).toEqual({ status: "failed", warnings: ["Saved; retry indexing."] });
|
||||
});
|
||||
@@ -56,7 +56,7 @@ test("catalog listing translates upstream Pi IDs back to canonical model keys",
|
||||
session: { reasoning: true, contextWindow: 32768, maxTokens: 8192 },
|
||||
};
|
||||
const modelCatalog: RuntimeModelCatalog = {
|
||||
defaultSession: model.id, defaultMetadataGeneration: null, embedding: null,
|
||||
defaultInteraction: model.id, embedding: null,
|
||||
sessionModels: () => [model], metadataModels: () => [], hasSession: (id) => id === model.id,
|
||||
};
|
||||
try {
|
||||
@@ -75,6 +75,38 @@ test("catalog listing translates upstream Pi IDs back to canonical model keys",
|
||||
}
|
||||
});
|
||||
|
||||
test("catalog listing supplies the shared DeepSeek key without requiring Pi auth", async () => {
|
||||
const script = scriptWith([{ provider: "deepseek", id: "deepseek-v4-pro", name: "DeepSeek V4 Pro" }]);
|
||||
const secret = join(path.dirname(script), "thothii.secrets");
|
||||
writeFileSync(secret, "DEEPSEEK_API_KEY=shared-key\nOPENAI_API_KEY=unrelated-key\n", { mode: 0o600 });
|
||||
const model: RuntimeModel = {
|
||||
id: "deepseek/deepseek-v4-pro", provider: "deepseek", model: "deepseek-v4-pro",
|
||||
label: "DeepSeek V4 Pro", upstreamModel: "deepseek-v4-pro",
|
||||
authentication: { mode: "secret_env", apiKeyEnv: "DEEPSEEK_API_KEY" },
|
||||
sessionAdapter: { mode: "pi_builtin" }, session: { reasoning: false },
|
||||
};
|
||||
const modelCatalog: RuntimeModelCatalog = {
|
||||
defaultInteraction: model.id, embedding: null,
|
||||
sessionModels: () => [model], metadataModels: () => [], hasSession: (id) => id === model.id,
|
||||
};
|
||||
try {
|
||||
const lister = createPiModelLister(loadConfig({ THT_SECRETS_FILE: secret }), {
|
||||
...noManagedModels, modelCatalog, loadEnabledModels: enabled(model.id),
|
||||
spawnFn: (_command, _args, options) => {
|
||||
expect(options.env.DEEPSEEK_API_KEY).toBe("shared-key");
|
||||
expect(options.env).not.toHaveProperty("OPENAI_API_KEY");
|
||||
expect(options.env).not.toHaveProperty("THT_SECRETS_FILE");
|
||||
return spawn("node", [FAKE, script], { env: options.env }) as any;
|
||||
},
|
||||
});
|
||||
await expect(lister()).resolves.toEqual([{
|
||||
provider: model.provider, id: model.model, name: model.label, reasoning: false,
|
||||
}]);
|
||||
} finally {
|
||||
rmSync(path.dirname(script), { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test("createPiModelLister caches within ttl (spawns once for two calls)", async () => {
|
||||
const script = scriptWith([{ provider: "zai", id: "glm-5.2", name: "GLM 5.2", reasoning: true }]);
|
||||
try {
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
import Fastify from "fastify";
|
||||
import { afterEach, expect, test, vi } from "vitest";
|
||||
import { memoryRoutes } from "../src/routes/memory.js";
|
||||
import { DEFAULT_SEMANTIC_RUNTIME } from "../src/workspaces/runtime-renderer.js";
|
||||
import type { ThtRunner } from "../src/tht/tht-runner.js";
|
||||
import type { PrincipalContext } from "../src/auth/principal.js";
|
||||
|
||||
const apps: ReturnType<typeof Fastify>[] = [];
|
||||
afterEach(async () => { await Promise.all(apps.splice(0).map(app => app.close())); });
|
||||
const id = "mem-11111111-1111-4111-8111-111111111111";
|
||||
const input = { family: "domain_clarification", subject: "Order", scope: "Sales" };
|
||||
|
||||
function setup(admin = true) {
|
||||
const app = Fastify(); apps.push(app);
|
||||
const principal: PrincipalContext = { issuer: "test", subject: "operator", roles: [admin ? "admin" : "user"],
|
||||
permissions: admin ? ["memory.manage"] : ["session.use"], isAdmin: admin };
|
||||
app.addHook("preHandler", async request => { request.principal = principal; });
|
||||
const run = vi.fn(async () => ({ code: 0, stdout: JSON.stringify({ items: [], total: 0 }), stderr: "" }));
|
||||
const bound = { runWithRuntimeSnapshot: run } as unknown as ThtRunner;
|
||||
const withPrincipal = vi.fn(() => bound);
|
||||
memoryRoutes(app, { runner: { withPrincipal }, runtime: DEFAULT_SEMANTIC_RUNTIME,
|
||||
registry: {
|
||||
list: async () => [{ id: "sales", commit: "a".repeat(40), blob: "b".repeat(40), snapshotPath: "/missing" }],
|
||||
read: vi.fn().mockResolvedValue({ workspace: { workspace: { language: "it" } } }),
|
||||
} });
|
||||
return { app, run, withPrincipal, principal };
|
||||
}
|
||||
|
||||
test("list binds principal and workspace without requiring a DWH runtime", async () => {
|
||||
const { app, run, withPrincipal, principal } = setup();
|
||||
const response = await app.inject("/workspaces/sales/memory?q=Order&page=2&column=id");
|
||||
expect(response.statusCode).toBe(200);
|
||||
expect(withPrincipal).toHaveBeenCalledWith(principal);
|
||||
const [args, snapshot] = run.mock.calls[0] as unknown as [string[], string];
|
||||
expect(args).toEqual(["memory", "admin", "--workspace", "sales"]);
|
||||
expect(JSON.parse(snapshot)).toMatchObject({ action: "list", request: { q: "Order", page: 2, column: "id" } });
|
||||
expect(JSON.parse(snapshot).runtime.memoryLanguage).toBe("it");
|
||||
});
|
||||
|
||||
test.each([
|
||||
["GET", ""], ["POST", ""], ["GET", "/pending"], ["GET", `/${id}`],
|
||||
["PUT", `/${id}`], ["DELETE", `/${id}`], ["POST", `/${id}/retry`],
|
||||
] as const)("non-admin cannot %s %s", async (method, suffix) => {
|
||||
const { app, run } = setup(false);
|
||||
const response = await app.inject({ method, url: `/workspaces/sales/memory${suffix}`,
|
||||
...(method === "PUT" || method === "POST" && suffix === "" ? { payload: input } : {}) });
|
||||
expect(response.statusCode).toBe(403); expect(run).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test("unknown workspace and invalid input do not invoke the harness", async () => {
|
||||
const { app, run } = setup();
|
||||
expect((await app.inject("/workspaces/missing/memory")).statusCode).toBe(404);
|
||||
expect((await app.inject("/workspaces/sales/memory?page_size=101")).statusCode).toBe(400);
|
||||
expect((await app.inject({ method: "POST", url: "/workspaces/sales/memory",
|
||||
payload: { ...input, workspace_id: "foreign" } })).statusCode).toBe(400);
|
||||
expect(run).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test("create preserves saved-but-unindexed outcome, update carries complete related changes", async () => {
|
||||
const { app, run } = setup();
|
||||
run.mockResolvedValue({ code: 0, stdout: JSON.stringify({ id, saved: true, indexed: false }), stderr: "" });
|
||||
const result = await app.inject({ method: "POST", url: "/workspaces/sales/memory", payload: input });
|
||||
expect(result.statusCode).toBe(201); expect(result.json()).toEqual({ id, saved: true, indexed: false });
|
||||
await app.inject({ method: "PUT", url: `/workspaces/sales/memory/${id}`, payload: {
|
||||
...input, links: [{ target_id: id, meaning: "Related" }], dependencies: [{ database: "dwh" }],
|
||||
} });
|
||||
const [, snapshot] = run.mock.calls[1] as unknown as [string[], string];
|
||||
expect(JSON.parse(snapshot)).toMatchObject({ action: "update", request: { id, card: {
|
||||
links: [{ target_id: id, meaning: "Related" }], dependencies: [{ database: "dwh" }],
|
||||
} } });
|
||||
});
|
||||
|
||||
test.each([["memory_not_found", 404], ["memory_conflict", 409], ["memory_unavailable", 503]])(
|
||||
"maps %s without leaking stderr", async (code, status) => {
|
||||
const { app, run } = setup();
|
||||
run.mockResolvedValue({ code: 1, stdout: JSON.stringify({ code, message: "sensitive detail" }), stderr: "secret" });
|
||||
const result = await app.inject("/workspaces/sales/memory");
|
||||
expect(result.statusCode).toBe(status); expect(result.json().code).toBe(code);
|
||||
expect(result.body).not.toMatch(/secret|sensitive/);
|
||||
},
|
||||
);
|
||||
@@ -20,9 +20,8 @@ function runtimeCatalog(overrides: Record<string, unknown> = {}, secrets = "OPEN
|
||||
const catalogFile = join(root, "catalog.json");
|
||||
const secretsFile = join(root, "thothii.secrets");
|
||||
const catalog = {
|
||||
schemaVersion: 1,
|
||||
defaultSession: "zai/glm-5.3",
|
||||
defaultMetadataGeneration: "zai/glm-5.3",
|
||||
schemaVersion: 2,
|
||||
defaultInteraction: "zai/glm-5.3",
|
||||
embedding: { id: "ollama/qwen3-embedding:0.6b", dimensions: 1024 },
|
||||
models: [
|
||||
{
|
||||
@@ -55,8 +54,8 @@ test("loads session default and safe metadata choices from the normalized runtim
|
||||
const runtime = loadRuntimeModelCatalog(catalogFile);
|
||||
const metadata = loadMetadataGenerationModels({ catalogFile, secretsFile });
|
||||
|
||||
expect(runtime.defaultSession).toBe("zai/glm-5.3");
|
||||
expect(runtime.hasSession("deepseek/deepseek-v4-pro")).toBe(true);
|
||||
expect(runtime.defaultInteraction).toBe("zai/glm-5.3");
|
||||
expect(runtime.hasSession("deepseek/deepseek-v4-pro")).toBe(false);
|
||||
expect(metadata.catalog()).toEqual({
|
||||
models: [{ id: "zai/glm-5.3", label: "GLM 5.3" }],
|
||||
default: "zai/glm-5.3",
|
||||
@@ -69,14 +68,36 @@ test("loads session default and safe metadata choices from the normalized runtim
|
||||
expect(() => metadata.resolve("zai/missing")).toThrow(MetadataGenerationModelUnavailableError);
|
||||
});
|
||||
|
||||
test("DeepSeek uses one identity and bundle credential for native Pi and LiteLLM", () => {
|
||||
const models = ["deepseek-v4-pro", "deepseek-v4-flash"].map((model) => ({
|
||||
id: `deepseek/${model}`, provider: "deepseek", model, label: model, upstreamModel: model,
|
||||
authentication: { mode: "secret_env", apiKeyEnv: "DEEPSEEK_API_KEY" },
|
||||
sessionAdapter: { mode: "pi_builtin" }, metadataAdapter: { litellmProvider: "deepseek" },
|
||||
session: { reasoning: false }, metadataGeneration: { disableThinking: false },
|
||||
}));
|
||||
const files = runtimeCatalog({ defaultInteraction: models[0].id, models }, "DEEPSEEK_API_KEY=shared-key\n");
|
||||
const runtime = loadRuntimeModelCatalog(files.catalogFile);
|
||||
const metadata = loadMetadataGenerationModels(files);
|
||||
expect(runtime.sessionModels().map((model) => model.id)).toEqual(metadata.catalog().models.map((model) => model.id));
|
||||
expect(metadata.catalog().default).toBe(runtime.defaultInteraction);
|
||||
for (const model of models) {
|
||||
expect(metadata.resolve(model.id)).toMatchObject({
|
||||
id: model.id, provider: "deepseek", model: model.model,
|
||||
apiKeyEnv: "DEEPSEEK_API_KEY", apiKey: "shared-key",
|
||||
});
|
||||
}
|
||||
expect(() => metadata.resolve("deepseek-metadata/deepseek-v4-pro"))
|
||||
.toThrow(MetadataGenerationModelUnavailableError);
|
||||
});
|
||||
|
||||
test("returns empty catalogs when no runtime projection is configured", () => {
|
||||
expect(loadRuntimeModelCatalog().defaultSession).toBeNull();
|
||||
expect(loadRuntimeModelCatalog().defaultInteraction).toBeNull();
|
||||
expect(loadMetadataGenerationModels({}).catalog()).toEqual({ models: [], default: null });
|
||||
});
|
||||
|
||||
test("rejects a drifted default and an unprotected projection", () => {
|
||||
const drifted = runtimeCatalog({ defaultSession: "zai/missing" });
|
||||
expect(() => loadRuntimeModelCatalog(drifted.catalogFile)).toThrow("session default is invalid");
|
||||
const drifted = runtimeCatalog({ defaultInteraction: "zai/missing" });
|
||||
expect(() => loadRuntimeModelCatalog(drifted.catalogFile)).toThrow("interaction default is invalid");
|
||||
|
||||
const unprotected = runtimeCatalog();
|
||||
chmodSync(unprotected.catalogFile, 0o666);
|
||||
@@ -85,7 +106,7 @@ test("rejects a drifted default and an unprotected projection", () => {
|
||||
|
||||
test("rejects authentication semantics that cannot come from the installation catalog", () => {
|
||||
const invalid = runtimeCatalog({
|
||||
defaultMetadataGeneration: undefined,
|
||||
defaultInteraction: undefined,
|
||||
models: [{
|
||||
id: "zai/glm-5.3",
|
||||
provider: "zai",
|
||||
@@ -112,3 +133,13 @@ test("splits canonical session identities without provider aliases", () => {
|
||||
expect(splitCanonicalModelId("zai/glm-5.3")).toEqual({ provider: "zai", model: "glm-5.3" });
|
||||
expect(() => splitCanonicalModelId("glm-5.3")).toThrow("model identity is invalid");
|
||||
});
|
||||
|
||||
test("rejects a default supported by only one configured use", () => {
|
||||
const { catalogFile } = runtimeCatalog({ defaultInteraction: "deepseek/deepseek-v4-pro" });
|
||||
expect(() => loadRuntimeModelCatalog(catalogFile)).toThrow("interaction default is invalid");
|
||||
});
|
||||
|
||||
test("requires regenerated runtime schema v2 rather than interpreting two legacy defaults", () => {
|
||||
const { catalogFile } = runtimeCatalog({ schemaVersion: 1, defaultSession: "zai/glm-5.3" });
|
||||
expect(() => loadRuntimeModelCatalog(catalogFile)).toThrow("runtime model catalog is invalid");
|
||||
});
|
||||
|
||||
@@ -5,6 +5,8 @@ const fakes = vi.hoisted(() => ({
|
||||
catalogRepository: { close: vi.fn(async () => {}) },
|
||||
createCatalogRepository: vi.fn(),
|
||||
runnerConfig: undefined as Record<string, unknown> | undefined,
|
||||
acquireWorkspaceRuntime: vi.fn(),
|
||||
releaseWorkspaceRuntime: vi.fn(),
|
||||
run: vi.fn(async () => ({
|
||||
code: 0,
|
||||
stdout: JSON.stringify({ ok: true }),
|
||||
@@ -24,6 +26,8 @@ vi.mock("../src/tht/tht-runner.js", () => ({
|
||||
|
||||
run = fakes.run;
|
||||
|
||||
acquireWorkspaceRuntime = fakes.acquireWorkspaceRuntime;
|
||||
|
||||
withPrincipal() {
|
||||
return this;
|
||||
}
|
||||
@@ -67,6 +71,12 @@ beforeEach(() => {
|
||||
fakes.createCatalogRepository.mockReset();
|
||||
fakes.createCatalogRepository.mockReturnValue(fakes.catalogRepository);
|
||||
fakes.run.mockClear();
|
||||
fakes.acquireWorkspaceRuntime.mockReset();
|
||||
fakes.acquireWorkspaceRuntime.mockResolvedValue({
|
||||
path: "/data/workspace-registry/snapshots/runtime/workspace.yaml",
|
||||
release: fakes.releaseWorkspaceRuntime,
|
||||
});
|
||||
fakes.releaseWorkspaceRuntime.mockClear();
|
||||
fakes.runnerConfig = undefined;
|
||||
});
|
||||
|
||||
@@ -78,5 +88,13 @@ test("workflow doctor gives schema-v4 runtime rendering a live Catalog repositor
|
||||
|
||||
expect(fakes.createCatalogRepository).toHaveBeenCalledWith(config.catalogDatabase);
|
||||
expect(fakes.runnerConfig?.catalogRepository).toBe(fakes.catalogRepository);
|
||||
expect(fakes.acquireWorkspaceRuntime).toHaveBeenCalledWith(
|
||||
"/data/workspace-registry/snapshots/revision/workspace.yaml",
|
||||
);
|
||||
expect(fakes.run).toHaveBeenCalledWith(
|
||||
["doctor", "--json"],
|
||||
"/data/workspace-registry/snapshots/runtime/workspace.yaml",
|
||||
);
|
||||
expect(fakes.releaseWorkspaceRuntime).toHaveBeenCalledOnce();
|
||||
expect(fakes.catalogRepository.close).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { mkdtempSync } from "node:fs";
|
||||
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { expect, test, vi } from "vitest";
|
||||
@@ -7,7 +7,34 @@ import {
|
||||
createPiManagement,
|
||||
type PiExecFile,
|
||||
} from "../src/pi/management.js";
|
||||
import type { RuntimeModelCatalog } from "../src/models/runtime-model-catalog.js";
|
||||
import type { RuntimeModel, RuntimeModelCatalog } from "../src/models/runtime-model-catalog.js";
|
||||
|
||||
test.each([false, true])("catalog credential status ignores legacy Pi auth, missing bundle key: %s", async (missingKey) => {
|
||||
const root = mkdtempSync(join(tmpdir(), "tht-catalog-status-"));
|
||||
writeFileSync(join(root, "auth.json"), JSON.stringify({ deepseek: { type: "api_key", key: "stale-key" } }), { mode: 0o600 });
|
||||
const secret = join(root, "thothii.secrets");
|
||||
writeFileSync(secret, missingKey ? "THT_MODEL_API_KEY=legacy-key\n" : "DEEPSEEK_API_KEY=shared-key\n", { mode: 0o600 });
|
||||
vi.stubEnv("PI_CODING_AGENT_DIR", root);
|
||||
const model: RuntimeModel = {
|
||||
id: "deepseek/deepseek-v4-pro", provider: "deepseek", model: "deepseek-v4-pro",
|
||||
label: "DeepSeek", upstreamModel: "deepseek-v4-pro",
|
||||
authentication: { mode: "secret_env", apiKeyEnv: "DEEPSEEK_API_KEY" },
|
||||
sessionAdapter: { mode: "pi_builtin" }, session: { reasoning: false },
|
||||
};
|
||||
try {
|
||||
const service = createPiManagement(loadConfig({ THT_SECRETS_FILE: secret }), {
|
||||
modelCatalog: {
|
||||
defaultInteraction: model.id, embedding: null,
|
||||
sessionModels: () => [model], metadataModels: () => [], hasSession: (id) => id === model.id,
|
||||
},
|
||||
execute: successfulExec([]), readSettings: () => ({ thinking: "medium" }),
|
||||
});
|
||||
expect((await service.status()).credentials).toBe(missingKey ? "missing" : "present");
|
||||
} finally {
|
||||
vi.unstubAllEnvs();
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
function configFor(settingsFile = join(mkdtempSync(join(tmpdir(), "tht-pi-management-")), "settings.json")) {
|
||||
return loadConfig({
|
||||
@@ -15,12 +42,12 @@ function configFor(settingsFile = join(mkdtempSync(join(tmpdir(), "tht-pi-manage
|
||||
SETTINGS_FILE: settingsFile,
|
||||
PI_BIN: "/usr/local/bin/pi",
|
||||
PI_MANAGEMENT_TIMEOUT_MS: "750",
|
||||
THT_HOST_PLATFORM: "linux",
|
||||
});
|
||||
}
|
||||
|
||||
const modelCatalog: RuntimeModelCatalog = {
|
||||
defaultSession: "zai/glm-5.2",
|
||||
defaultMetadataGeneration: null,
|
||||
defaultInteraction: "zai/glm-5.2",
|
||||
embedding: { id: "ollama/qwen3-embedding:0.6b", dimensions: 1024 },
|
||||
sessionModels: () => [],
|
||||
metadataModels: () => [],
|
||||
@@ -34,6 +61,16 @@ function successfulExec(calls: Array<{ command: string; args: string[]; timeout:
|
||||
};
|
||||
}
|
||||
|
||||
test.each([
|
||||
["linux", "linux"], ["darwin", "macos"], ["windows", "windows"],
|
||||
])("reports installation host %s independently of the backend container OS", async (host, expected) => {
|
||||
const service = createPiManagement(loadConfig({ THT_HOST_PLATFORM: host }), {
|
||||
modelCatalog, execute: successfulExec([]), readSettings: () => ({ thinking: "medium" }),
|
||||
credentialStatus: () => "missing",
|
||||
});
|
||||
expect((await service.status()).hostPlatform).toBe(expected);
|
||||
});
|
||||
|
||||
// Catches a Pi executable that emits unexpected text or is invoked through a shell, which could
|
||||
// turn a version display into a command-injection or information-disclosure surface.
|
||||
test("status parses only a Pi version from a fixed execFile argument array", async () => {
|
||||
@@ -47,6 +84,7 @@ test("status parses only a Pi version from a fixed execFile argument array", asy
|
||||
});
|
||||
|
||||
await expect(service.status()).resolves.toEqual({
|
||||
hostPlatform: "linux",
|
||||
version: "0.80.3",
|
||||
ready: true,
|
||||
credentials: "missing",
|
||||
@@ -78,6 +116,7 @@ test.each(["present", "missing"] as const)(
|
||||
|
||||
const status = await service.status();
|
||||
expect(status).toEqual({
|
||||
hostPlatform: "linux",
|
||||
version: "0.80.3",
|
||||
ready: true,
|
||||
credentials,
|
||||
|
||||
@@ -190,6 +190,29 @@ test("Pi receives the leased workspace runtime config and releases it on direct
|
||||
expect(release).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
test("Pi language launch hint comes from the session options and never ambient environment", () => {
|
||||
const previous = process.env.THT_INTERACTION_LANGUAGE;
|
||||
process.env.THT_INTERACTION_LANGUAGE = "it";
|
||||
const environments: NodeJS.ProcessEnv[] = [];
|
||||
const mgr = new PiProcessManager(loadConfig({}), {
|
||||
spawnFn: (_command, _args, options) => {
|
||||
environments.push(options.env);
|
||||
return recordingChild() as any;
|
||||
},
|
||||
});
|
||||
try {
|
||||
mgr.createFor("explicit-language", { interactionLanguage: "en" });
|
||||
mgr.teardown("explicit-language");
|
||||
mgr.createFor("manifest-resolved-in-gate");
|
||||
mgr.teardown("manifest-resolved-in-gate");
|
||||
expect(environments[0].THT_INTERACTION_LANGUAGE).toBe("en");
|
||||
expect(environments[1]).not.toHaveProperty("THT_INTERACTION_LANGUAGE");
|
||||
} finally {
|
||||
if (previous === undefined) delete process.env.THT_INTERACTION_LANGUAGE;
|
||||
else process.env.THT_INTERACTION_LANGUAGE = previous;
|
||||
}
|
||||
});
|
||||
|
||||
test("a close-only child event releases its temporary Pi agent snapshot", () => {
|
||||
const child = recordingChild();
|
||||
let snapshotDir: string | undefined;
|
||||
@@ -642,27 +665,33 @@ test("session Pi spawn reads the single secret bundle and scrubs its path", asyn
|
||||
}
|
||||
});
|
||||
|
||||
test("session Pi spawn resolves the selected catalog credential from the secret bundle", () => {
|
||||
test.each([false, true])("session Pi uses the catalog bundle despite stale Pi auth: %s", (missingKey) => {
|
||||
const root = mkdtempSync(path.join(tmpdir(), "thothii-catalog-credential-"));
|
||||
const agentDir = path.join(root, "agent");
|
||||
mkdirSync(agentDir, { mode: 0o700 });
|
||||
writeFileSync(path.join(agentDir, "auth.json"), "{}\n", { mode: 0o600 });
|
||||
const originalAuth = JSON.stringify({
|
||||
deepseek: { type: "api_key", key: "stale-key" },
|
||||
anthropic: { type: "api_key", key: "unrelated-key" },
|
||||
});
|
||||
writeFileSync(path.join(agentDir, "auth.json"), originalAuth, { mode: 0o600 });
|
||||
writeFileSync(path.join(agentDir, "models.json"), '{"providers":{}}\n', { mode: 0o600 });
|
||||
const secret = path.join(root, "thothii.secrets");
|
||||
writeFileSync(secret, "ZAI_API_KEY=catalog-secret\nTHT_MODEL_API_KEY=legacy-secret\n", { mode: 0o600 });
|
||||
writeFileSync(secret, missingKey ? "THT_MODEL_API_KEY=legacy-secret\n"
|
||||
: "DEEPSEEK_API_KEY=catalog-secret\nTHT_MODEL_API_KEY=legacy-secret\n", { mode: 0o600 });
|
||||
const legacyKey = path.join(root, "legacy-key");
|
||||
writeFileSync(legacyKey, "legacy-file-key", { mode: 0o600 });
|
||||
const model: RuntimeModel = {
|
||||
id: "openai/test-model",
|
||||
provider: "openai",
|
||||
model: "test-model",
|
||||
label: "Test model",
|
||||
upstreamModel: "test-model",
|
||||
authentication: { mode: "secret_env", apiKeyEnv: "ZAI_API_KEY" },
|
||||
id: "deepseek/deepseek-v4-pro",
|
||||
provider: "deepseek",
|
||||
model: "deepseek-v4-pro",
|
||||
label: "DeepSeek V4 Pro",
|
||||
upstreamModel: "deepseek-v4-pro",
|
||||
authentication: { mode: "secret_env", apiKeyEnv: "DEEPSEEK_API_KEY" },
|
||||
sessionAdapter: { mode: "pi_builtin" },
|
||||
session: { reasoning: false },
|
||||
};
|
||||
const modelCatalog: RuntimeModelCatalog = {
|
||||
defaultSession: model.id,
|
||||
defaultMetadataGeneration: null,
|
||||
defaultInteraction: model.id,
|
||||
embedding: null,
|
||||
sessionModels: () => [model],
|
||||
metadataModels: () => [],
|
||||
@@ -672,17 +701,26 @@ test("session Pi spawn resolves the selected catalog credential from the secret
|
||||
const child = recordingChild();
|
||||
child.stderr.resume = () => {};
|
||||
vi.stubEnv("PI_CODING_AGENT_DIR", agentDir);
|
||||
const mgr = new PiProcessManager(loadConfig({ THT_SECRETS_FILE: secret }), {
|
||||
const mgr = new PiProcessManager(loadConfig({ THT_SECRETS_FILE: secret, THT_MODEL_API_KEY_FILE: legacyKey }), {
|
||||
modelCatalog,
|
||||
authProviders: () => new Set(),
|
||||
authProviders: () => new Set(["deepseek"]),
|
||||
spawnFn: (...args: any[]) => { calls.push(args); return child as any; },
|
||||
});
|
||||
try {
|
||||
mgr.createFor("catalog-credential", { provider: "openai", model: "test-model" });
|
||||
expect(calls[0][2].env.ZAI_API_KEY).toBe("catalog-secret");
|
||||
const create = () => mgr.createFor("catalog-credential", { provider: model.provider, model: model.model });
|
||||
if (missingKey) {
|
||||
expect(create).toThrow("model provider credential is unavailable");
|
||||
expect(calls).toHaveLength(0);
|
||||
return;
|
||||
}
|
||||
create();
|
||||
expect(calls[0][2].env.DEEPSEEK_API_KEY).toBe("catalog-secret");
|
||||
expect(calls[0][2].env).not.toHaveProperty("OPENAI_API_KEY");
|
||||
expect(calls[0][2].env).not.toHaveProperty("THT_MODEL_API_KEY");
|
||||
expect(JSON.parse(readFileSync(path.join(calls[0][2].env.PI_CODING_AGENT_DIR, "auth.json"), "utf8")))
|
||||
.toEqual({ anthropic: { type: "api_key", key: "unrelated-key" } });
|
||||
} finally {
|
||||
expect(readFileSync(path.join(agentDir, "auth.json"), "utf8")).toBe(originalAuth);
|
||||
mgr.teardown("catalog-credential");
|
||||
vi.unstubAllEnvs();
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
@@ -750,7 +788,7 @@ test("set_model translates a canonical catalog key to its upstream Pi model ID",
|
||||
session: { reasoning: false, contextWindow: 32768, maxTokens: 8192 },
|
||||
};
|
||||
const modelCatalog: RuntimeModelCatalog = {
|
||||
defaultSession: model.id, defaultMetadataGeneration: null, embedding: null,
|
||||
defaultInteraction: model.id, embedding: null,
|
||||
sessionModels: () => [model], metadataModels: () => [], hasSession: (id) => id === model.id,
|
||||
};
|
||||
const mgr = new PiProcessManager(loadConfig({ PI_BIN: "/usr/local/bin/pi" }), {
|
||||
|
||||
@@ -61,20 +61,22 @@ test("provider smoke resolves the selected catalog credential from the secret bu
|
||||
session: { reasoning: false },
|
||||
};
|
||||
const modelCatalog: RuntimeModelCatalog = {
|
||||
defaultSession: model.id,
|
||||
defaultMetadataGeneration: null,
|
||||
defaultInteraction: model.id,
|
||||
embedding: null,
|
||||
sessionModels: () => [model],
|
||||
metadataModels: () => [],
|
||||
hasSession: (id) => id === model.id,
|
||||
};
|
||||
let spawnEnv: NodeJS.ProcessEnv | undefined;
|
||||
const readAuthStore = vi.fn(() => JSON.stringify({ openai: { type: "api_key", key: "stale-key" } }));
|
||||
const smoke = createPiProviderSmoke(loadConfig({ THT_SECRETS_FILE: secret }), {
|
||||
modelCatalog,
|
||||
authProviders: () => new Set(),
|
||||
authProviders: () => new Set(["openai"]),
|
||||
readAuthStore,
|
||||
readModelsStore: () => undefined,
|
||||
spawnFn: (_command, _args, options) => {
|
||||
spawnEnv = options.env;
|
||||
expect(existsSync(join(options.env.PI_CODING_AGENT_DIR!, "auth.json"))).toBe(false);
|
||||
return successfulProviderChild();
|
||||
},
|
||||
});
|
||||
@@ -85,6 +87,7 @@ test("provider smoke resolves the selected catalog credential from the secret bu
|
||||
expect(spawnEnv?.ZAI_API_KEY).toBe("catalog-secret");
|
||||
expect(spawnEnv).not.toHaveProperty("OPENAI_API_KEY");
|
||||
expect(spawnEnv).not.toHaveProperty("THT_MODEL_API_KEY");
|
||||
expect(readAuthStore).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
@@ -306,7 +309,7 @@ test("provider smoke makes one configured request from an isolated no-capability
|
||||
sessionAdapter: { mode: "pi_builtin" }, session: { reasoning: true },
|
||||
};
|
||||
const smokeCatalog: RuntimeModelCatalog = {
|
||||
defaultSession: smokeModel.id, defaultMetadataGeneration: null, embedding: null,
|
||||
defaultInteraction: smokeModel.id, embedding: null,
|
||||
sessionModels: () => [smokeModel], metadataModels: () => [],
|
||||
hasSession: (id) => id === smokeModel.id,
|
||||
};
|
||||
|
||||
@@ -27,10 +27,9 @@ function operationalWorkspace(id = "default") {
|
||||
} as const;
|
||||
}
|
||||
|
||||
function sessionCatalog(defaultSession = "zai/glm-5.2", available = [defaultSession]) {
|
||||
function sessionCatalog(defaultInteraction = "zai/glm-5.2", available = [defaultInteraction]) {
|
||||
return {
|
||||
defaultSession,
|
||||
defaultMetadataGeneration: null,
|
||||
defaultInteraction,
|
||||
embedding: { id: "ollama/qwen3-embedding:0.6b", dimensions: 1024 },
|
||||
sessionModels: () => [],
|
||||
metadataModels: () => [],
|
||||
@@ -54,7 +53,11 @@ const defaultWorkspaceRegistry = {
|
||||
|
||||
function buildApp(config: Parameters<typeof buildRealApp>[0], deps: Record<string, unknown> = {}) {
|
||||
const thtRunner = deps.thtRunner
|
||||
? { qdrantEnsure: async () => ({ ok: true }), ...(deps.thtRunner as object) }
|
||||
? {
|
||||
qdrantEnsure: async () => ({ ok: true }),
|
||||
ensureInteractionLanguage: async () => ({ interaction_language: "en" }),
|
||||
...(deps.thtRunner as object),
|
||||
}
|
||||
: undefined;
|
||||
return buildRealApp(config, {
|
||||
workspaceRuntimeSupport: () => true,
|
||||
@@ -89,6 +92,129 @@ const aliceHeaders = {
|
||||
"x-thoth-is-admin": "0",
|
||||
};
|
||||
|
||||
test.each([undefined, null, "", "en--US", "en_US", "en\nIGNORE", "en<script>", 42])(
|
||||
"new sessions reject invalid interaction language %s before persistence", async (interactionLanguage) => {
|
||||
const app = mutApp({ sessionNew: async () => { throw new Error("must not create"); } });
|
||||
try {
|
||||
const response = await app.inject({
|
||||
method: "POST", url: "/sessions", payload: { question: "Pazienti", interactionLanguage },
|
||||
});
|
||||
expect(response.statusCode).toBe(400);
|
||||
expect(response.json()).toMatchObject({ code: "invalid_interaction_language" });
|
||||
} finally { await app.close(); }
|
||||
},
|
||||
);
|
||||
|
||||
test.each([["en", "en"], ["fr-FR", "fr-FR"], ["FR-fr", "fr-FR"]])(
|
||||
"new session forwards canonical interaction language %s without changing the question", async (language, canonical) => {
|
||||
let created: any;
|
||||
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
|
||||
thtRunner: {
|
||||
sessionNew: async (options: any) => { created = options; return { id: "language" }; },
|
||||
searchPack: async () => {},
|
||||
},
|
||||
readiness: { ensure: async () => ({ ok: true }) },
|
||||
mgr: {
|
||||
get: () => undefined, createFor: () => ({ bridge: { onClientEvent: () => {} } }),
|
||||
configure: async () => {}, start: () => {},
|
||||
},
|
||||
getSettings: () => ({ workspace: "default" }),
|
||||
});
|
||||
try {
|
||||
const response = await app.inject({
|
||||
method: "POST", url: "/sessions", payload: { question: "Pazienti", interactionLanguage: language },
|
||||
});
|
||||
expect(response.statusCode).toBe(200);
|
||||
expect(created).toMatchObject({ question: "Pazienti", interactionLanguage: canonical });
|
||||
} finally { await app.close(); }
|
||||
});
|
||||
|
||||
test("resume rejects browser interaction language overrides", async () => {
|
||||
const app = mutApp({ sessionShow: async () => ({ status: "open", interaction_language: "it" }) });
|
||||
try {
|
||||
const response = await app.inject({
|
||||
method: "POST", url: "/sessions/s1/resume", payload: { interactionLanguage: "en" },
|
||||
});
|
||||
expect(response.statusCode).toBe(400);
|
||||
expect(response.json()).toMatchObject({ code: "interaction_language_pinned" });
|
||||
} finally { await app.close(); }
|
||||
});
|
||||
|
||||
test("resume pins legacy interaction language using the resolved harness workspace", async () => {
|
||||
let pinnedWorkspace: string | undefined;
|
||||
let runtime: any;
|
||||
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
|
||||
thtRunner: {
|
||||
sessionShow: async () => ({ id: "legacy", status: "closed" }),
|
||||
ensureInteractionLanguage: async (_id: string, workspace: string) => {
|
||||
pinnedWorkspace = workspace;
|
||||
return { interaction_language: "it" };
|
||||
},
|
||||
reopenSession: async () => {},
|
||||
},
|
||||
readiness: { ensure: async () => ({ ok: true }) },
|
||||
mgr: {
|
||||
get: () => undefined,
|
||||
createFor: (_id: string, options: any) => {
|
||||
runtime = options;
|
||||
return { bridge: { onClientEvent: () => {} } };
|
||||
},
|
||||
configure: async () => {}, start: () => {},
|
||||
},
|
||||
getSettings: () => ({ workspace: "other-browser-workspace" }),
|
||||
});
|
||||
try {
|
||||
const response = await app.inject({ method: "POST", url: "/sessions/legacy/resume" });
|
||||
expect(response.statusCode).toBe(200);
|
||||
expect(pinnedWorkspace).toContain("/default.yaml");
|
||||
expect(runtime).toMatchObject({ interactionLanguage: "it", mode: "resume" });
|
||||
} finally { await app.close(); }
|
||||
});
|
||||
|
||||
test("resume retains persisted interaction language despite different browser settings", async () => {
|
||||
let options: any;
|
||||
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
|
||||
thtRunner: {
|
||||
sessionShow: async () => ({ id: "saved", status: "closed", interaction_language: "it" }),
|
||||
ensureInteractionLanguage: async () => { throw new Error("language is already pinned"); },
|
||||
reopenSession: async () => {},
|
||||
},
|
||||
readiness: { ensure: async () => ({ ok: true }) },
|
||||
mgr: {
|
||||
get: () => undefined,
|
||||
createFor: (_id: string, value: any) => {
|
||||
options = value;
|
||||
return { bridge: { onClientEvent: () => {} } };
|
||||
},
|
||||
configure: async () => {}, start: () => {},
|
||||
},
|
||||
getSettings: () => ({ workspace: "english-workspace", locale: "en" }),
|
||||
});
|
||||
try {
|
||||
const response = await app.inject({ method: "POST", url: "/sessions/saved/resume" });
|
||||
expect(response.statusCode).toBe(200);
|
||||
expect(options.interactionLanguage).toBe("it");
|
||||
} finally { await app.close(); }
|
||||
});
|
||||
|
||||
test("resume cannot start Pi if legacy interaction language cannot be persisted", async () => {
|
||||
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
|
||||
thtRunner: {
|
||||
sessionShow: async () => ({ id: "legacy", status: "closed" }),
|
||||
ensureInteractionLanguage: async () => { throw new Error("private storage details"); },
|
||||
reopenSession: async () => { throw new Error("must not reopen"); },
|
||||
},
|
||||
readiness: { ensure: async () => ({ ok: true }) },
|
||||
mgr: { createFor: () => { throw new Error("must not spawn"); } },
|
||||
getSettings: () => ({ workspace: "default" }),
|
||||
});
|
||||
try {
|
||||
const response = await app.inject({ method: "POST", url: "/sessions/legacy/resume" });
|
||||
expect(response.statusCode).toBe(503);
|
||||
expect(response.body).not.toContain("private storage details");
|
||||
} finally { await app.close(); }
|
||||
});
|
||||
|
||||
test("upstream requests without a principal fail before a Pi runtime can be created", async () => {
|
||||
let created = false;
|
||||
const app = buildApp(loadConfig({ AUTH_MODE: "upstream", THT_HARNESS_DIR: "../harness" }), {
|
||||
@@ -96,7 +222,7 @@ test("upstream requests without a principal fail before a Pi runtime can be crea
|
||||
thtRunner: {} as any,
|
||||
});
|
||||
|
||||
const response = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
|
||||
const response = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
|
||||
|
||||
expect(response.statusCode).toBe(401);
|
||||
expect(created).toBe(false);
|
||||
@@ -134,7 +260,7 @@ test("maintenance rejects new and resumed session admission without interrupting
|
||||
});
|
||||
|
||||
const create = await app.inject({
|
||||
method: "POST", url: "/sessions", headers: aliceHeaders, payload: { question: "q" },
|
||||
method: "POST", url: "/sessions", headers: aliceHeaders, payload: { interactionLanguage: "en", question: "q" },
|
||||
});
|
||||
const resume = await app.inject({ method: "POST", url: "/sessions/open/resume", headers: aliceHeaders });
|
||||
|
||||
@@ -155,7 +281,7 @@ test("a durable maintenance marker initializes admission closed after backend re
|
||||
AUTH_MODE: "upstream", THT_HARNESS_DIR: "../harness", THT_MAINTENANCE_FILE: marker,
|
||||
}), { thtRunner: {} as any });
|
||||
const response = await app.inject({
|
||||
method: "POST", url: "/sessions", headers: aliceHeaders, payload: { question: "q" },
|
||||
method: "POST", url: "/sessions", headers: aliceHeaders, payload: { interactionLanguage: "en", question: "q" },
|
||||
});
|
||||
expect(response.statusCode).toBe(503);
|
||||
expect(response.json()).toMatchObject({ code: "maintenance" });
|
||||
@@ -364,6 +490,65 @@ test("retention scans a removed workspace's retained snapshot", async () => {
|
||||
expect(response.json()).toEqual([expect.objectContaining({ id: "resumable" })]);
|
||||
});
|
||||
|
||||
test("active sessions remain available when retained snapshot discovery is unreadable", async () => {
|
||||
const activeSnapshot = "/registry/snapshots/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/active.yaml";
|
||||
const retainedFailure = "invalid retained snapshot /registry/snapshots/secret/legacy.yaml";
|
||||
const reconcileSnapshotRetention = vi.fn(async () => {});
|
||||
const subscribed = vi.fn();
|
||||
const warning = vi.spyOn(console, "warn").mockImplementation(() => {});
|
||||
const manifest = {
|
||||
id: "live-session", status: "open", archived: false,
|
||||
workspace_id: "active", workspace_revision: "a".repeat(40),
|
||||
};
|
||||
const app = buildApp(loadConfig({ AUTH_MODE: "upstream", THT_HARNESS_DIR: "../harness" }), {
|
||||
thtRunner: {
|
||||
withPrincipal: () => ({
|
||||
sessionList: async (snapshotPath: string) => {
|
||||
expect(snapshotPath).toBe(activeSnapshot);
|
||||
return [manifest];
|
||||
},
|
||||
sessionShow: async (id: string, snapshotPath?: string) => {
|
||||
if (id === manifest.id && snapshotPath === activeSnapshot) return manifest;
|
||||
throw new Error("session not found");
|
||||
},
|
||||
}),
|
||||
} as any,
|
||||
mgr: { get: () => undefined } as any,
|
||||
hub: {
|
||||
subscribe: (_id: string, _send: unknown, options: { close?: () => void }) => {
|
||||
subscribed();
|
||||
options.close?.();
|
||||
return () => {};
|
||||
},
|
||||
} as any,
|
||||
workspaceRegistry: {
|
||||
listRetainedSnapshots: async () => { throw new Error(retainedFailure); },
|
||||
list: async () => [{
|
||||
id: "active", commit: "a".repeat(40), blob: "b".repeat(40), snapshotPath: activeSnapshot,
|
||||
}],
|
||||
reconcileSnapshotRetention,
|
||||
} as any,
|
||||
});
|
||||
|
||||
try {
|
||||
const list = await app.inject({ method: "GET", url: "/sessions", headers: aliceHeaders });
|
||||
const detail = await app.inject({ method: "GET", url: `/sessions/${manifest.id}`, headers: aliceHeaders });
|
||||
const events = await app.inject({ method: "GET", url: `/sessions/${manifest.id}/events`, headers: aliceHeaders });
|
||||
|
||||
expect(list.statusCode).toBe(200);
|
||||
expect(list.json()).toEqual([expect.objectContaining({ id: manifest.id, active: false })]);
|
||||
expect(detail.statusCode).toBe(200);
|
||||
expect(detail.json()).toMatchObject(manifest);
|
||||
expect(events.statusCode).toBe(200);
|
||||
expect(subscribed).toHaveBeenCalledOnce();
|
||||
expect(reconcileSnapshotRetention).not.toHaveBeenCalled();
|
||||
expect(list.body + detail.body + events.body).not.toContain(retainedFailure);
|
||||
} finally {
|
||||
warning.mockRestore();
|
||||
await app.close();
|
||||
}
|
||||
});
|
||||
|
||||
test("the single local installation listing reconciles its resumable workspace pins", async () => {
|
||||
const retained = vi.fn(async () => {});
|
||||
const retainedRevision = "d".repeat(40);
|
||||
@@ -434,7 +619,7 @@ test("new sessions are created through the authenticated principal, not a client
|
||||
|
||||
const response = await app.inject({
|
||||
method: "POST", url: "/sessions", headers: aliceHeaders,
|
||||
payload: { question: "q", owner: "mallory" },
|
||||
payload: { interactionLanguage: "en", question: "q", owner: "mallory" },
|
||||
});
|
||||
|
||||
expect(response.statusCode).toBe(200);
|
||||
@@ -451,7 +636,7 @@ test("new sessions reject the client legacy workspace field unless local legacy
|
||||
});
|
||||
|
||||
const response = await app.inject({
|
||||
method: "POST", url: "/sessions", payload: { question: "q", workspace: "legacy" },
|
||||
method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q", workspace: "legacy" },
|
||||
});
|
||||
|
||||
expect(response.statusCode).toBe(409);
|
||||
@@ -471,7 +656,7 @@ test("explicit local legacy mode permits the unpinned client workspace request",
|
||||
});
|
||||
|
||||
const response = await app.inject({
|
||||
method: "POST", url: "/sessions", payload: { question: "q", workspace: "legacy" },
|
||||
method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q", workspace: "legacy" },
|
||||
});
|
||||
|
||||
expect(response.statusCode).toBe(200);
|
||||
@@ -509,7 +694,7 @@ test("creates a session from the active immutable workspace revision", async ()
|
||||
|
||||
await app.inject({
|
||||
method: "POST", url: "/sessions",
|
||||
payload: { question: "q", workspaceId: "psd-clinical", provider: "zai", model: "glm-5.2", thinking: "low" },
|
||||
payload: { interactionLanguage: "en", question: "q", workspaceId: "psd-clinical", provider: "zai", model: "glm-5.2", thinking: "low" },
|
||||
});
|
||||
|
||||
expect(sessionNew).toHaveBeenCalledWith(expect.objectContaining({
|
||||
@@ -553,7 +738,7 @@ test("hands one Catalog-backed runtime to both retrieval and Pi", async () => {
|
||||
const response = await app.inject({
|
||||
method: "POST",
|
||||
url: "/sessions",
|
||||
payload: { question: "Which users placed orders?", workspaceId: "default" },
|
||||
payload: { interactionLanguage: "en", question: "Which users placed orders?", workspaceId: "default" },
|
||||
});
|
||||
|
||||
expect(response.statusCode).toBe(200);
|
||||
@@ -629,7 +814,7 @@ test("refuses core admission when the workspace preprocessing fingerprint is sta
|
||||
const response = await app.inject({
|
||||
method: "POST",
|
||||
url: "/sessions",
|
||||
payload: { question: "q", workspaceId: "default" },
|
||||
payload: { interactionLanguage: "en", question: "q", workspaceId: "default" },
|
||||
});
|
||||
|
||||
expect(response.statusCode).toBe(409);
|
||||
@@ -681,7 +866,7 @@ test("rejects an SSH-only Catalog binding before persisting or starting a sessio
|
||||
} as any,
|
||||
});
|
||||
|
||||
const response = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
|
||||
const response = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
|
||||
|
||||
expect(response.statusCode).toBe(409);
|
||||
expect(response.json()).toMatchObject({ code: "workspace_not_activatable" });
|
||||
@@ -721,7 +906,7 @@ test("hands a revision lease to retention only after the session manifest is dur
|
||||
workspaceRegistry: { acquireSessionRevision } as any,
|
||||
});
|
||||
|
||||
const request = app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
|
||||
const request = app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
expect(markPersisted).not.toHaveBeenCalled();
|
||||
expect(abort).not.toHaveBeenCalled();
|
||||
@@ -752,7 +937,7 @@ test("creates a session from the configured default workspace revision when work
|
||||
workspaceRegistry: registry as any,
|
||||
});
|
||||
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
|
||||
|
||||
expect(registry.read).toHaveBeenCalledWith("psd-clinical");
|
||||
expect(sessionNew).toHaveBeenCalledWith(expect.objectContaining({
|
||||
@@ -838,7 +1023,7 @@ test("session lifecycle locates a B session when installation default is A", asy
|
||||
} as any,
|
||||
});
|
||||
|
||||
expect((await app.inject({ method: "POST", url: "/sessions", payload: {
|
||||
expect((await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en",
|
||||
question: "B question", workspaceId: "b-workspace", provider: "zai", model: "glm-5.2", thinking: "low",
|
||||
} })).statusCode).toBe(200);
|
||||
expect((await app.inject({ method: "GET", url: "/sessions" })).json()).toEqual([
|
||||
@@ -850,7 +1035,7 @@ test("session lifecycle locates a B session when installation default is A", asy
|
||||
|
||||
active = undefined;
|
||||
expect((await app.inject({ method: "POST", url: "/sessions/session-b/resume" })).json())
|
||||
.toEqual({ id: "session-b", alreadyActive: false });
|
||||
.toEqual({ id: "session-b", alreadyActive: false, workspaceId: "b-workspace" });
|
||||
expect(calls).toContain(`new:${bPath}`);
|
||||
expect(calls).toContain(`list:${bPath}`);
|
||||
expect(calls).toContain(`show:${bPath}`);
|
||||
@@ -881,7 +1066,7 @@ test("POST /sessions uses the catalog default with workspace/thinking settings a
|
||||
],
|
||||
spawnFn: () => nodeSpawn("node", [FAKE, SCRIPT]) as any,
|
||||
});
|
||||
const created = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
|
||||
const created = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
|
||||
expect(created.json()).toEqual({ id: "s1" });
|
||||
expect(sessionNewArg.workspaceConfigPath).toContain(`/snapshots/${"e".repeat(40)}/w.yaml`);
|
||||
expect(sessionNewArg.provider).toBe("zai");
|
||||
@@ -942,11 +1127,11 @@ test("POST /sessions stops the user's previous open Pi runtime before creating a
|
||||
getSettings: () => ({ workspace: "local" }) as any,
|
||||
});
|
||||
|
||||
expect((await app.inject({ method: "POST", url: "/sessions", payload: { question: "one" } })).statusCode)
|
||||
expect((await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "one" } })).statusCode)
|
||||
.toBe(200);
|
||||
order.length = 0;
|
||||
|
||||
const second = await app.inject({ method: "POST", url: "/sessions", payload: { question: "two" } });
|
||||
const second = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "two" } });
|
||||
|
||||
expect(second.statusCode).toBe(200);
|
||||
expect(second.json()).toEqual({ id: "s2" });
|
||||
@@ -967,7 +1152,7 @@ test("POST /sessions refuses to create a session when the local DWH precheck fai
|
||||
readiness: { ensure: async () => ({ ok: true }) } as any,
|
||||
getSettings: () => ({ workspace: "w" }) as any,
|
||||
});
|
||||
const res = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
|
||||
const res = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
|
||||
expect(res.statusCode).toBe(503);
|
||||
expect(res.json()).toMatchObject({ code: "dwh_unreachable" });
|
||||
expect(pinged).toBe(1);
|
||||
@@ -988,7 +1173,7 @@ test("POST /sessions proceeds past a passing DWH precheck", async () => {
|
||||
getSettings: () => ({ workspace: "w" }) as any,
|
||||
spawnFn: () => nodeSpawn("node", [FAKE, SCRIPT]) as any,
|
||||
});
|
||||
const res = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
|
||||
const res = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
|
||||
expect(res.statusCode).toBe(200);
|
||||
expect(pinged).toBe(1);
|
||||
expect(created).toBe(1);
|
||||
@@ -1007,7 +1192,7 @@ test("POST /sessions skips the DWH precheck when the flag is off (default)", asy
|
||||
getSettings: () => ({ workspace: "w" }) as any,
|
||||
spawnFn: () => nodeSpawn("node", [FAKE, SCRIPT]) as any,
|
||||
});
|
||||
const res = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
|
||||
const res = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
|
||||
expect(res.statusCode).toBe(200);
|
||||
expect(pinged).toBe(0); // probe never runs without the flag
|
||||
});
|
||||
@@ -1041,7 +1226,7 @@ test("POST /sessions configura Pi con il thinking globale selezionato", async ()
|
||||
],
|
||||
});
|
||||
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
|
||||
expect(configured.thinking).toBe("high");
|
||||
@@ -1120,6 +1305,29 @@ test("POST /sessions/:id/resume uses the manifest's retained workspace revision"
|
||||
);
|
||||
});
|
||||
|
||||
test.each([undefined, { provider: "local", model: "qwen" }])("resume uses the global model/default, not the historical manifest (%j)", async (payload) => {
|
||||
const configure = vi.fn(async () => {});
|
||||
const manifest = { status: "open", archived: false, provider: "retired", model: "old-model" };
|
||||
const runtime = { bridge: { onClientEvent: () => {}, emitClientEvent: () => {} } };
|
||||
let current: any;
|
||||
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
|
||||
mgr: {
|
||||
get: () => current,
|
||||
createFor: () => { current = runtime; return runtime; },
|
||||
configure, start: () => {},
|
||||
} as any,
|
||||
thtRunner: { sessionShow: async () => manifest, reopenSession: async () => {} } as any,
|
||||
readiness: { ensure: async () => ({ ok: true }) } as any,
|
||||
runtimeModelCatalog: sessionCatalog("zai/glm-5.2", ["zai/glm-5.2", "local/qwen"]),
|
||||
getSettings: () => ({ workspace: "psd", thinking: "medium" }) as any,
|
||||
});
|
||||
const response = await app.inject({ method: "POST", url: "/sessions/model-resume/resume", ...(payload ? { payload } : {}) });
|
||||
expect(response.statusCode).toBe(200);
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
expect(configure).toHaveBeenCalledWith(expect.anything(), expect.objectContaining(payload ?? { provider: "zai", model: "glm-5.2" }));
|
||||
expect(manifest).toMatchObject({ provider: "retired", model: "old-model" });
|
||||
});
|
||||
|
||||
test("POST /sessions/:id/resume returns a sanitized error when its retained revision is unavailable", async () => {
|
||||
const rawFailure = "cannot read /data/workspace-registry/snapshots/secret-revision";
|
||||
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
|
||||
@@ -1346,7 +1554,7 @@ test("resuming a different session stops the user's previous Pi runtime", async
|
||||
readiness: { ensure: async () => ({ ok: true }) } as any,
|
||||
getSettings: () => ({ workspace: "local" }) as any,
|
||||
});
|
||||
expect((await app.inject({ method: "POST", url: "/sessions", payload: { question: "one" } })).statusCode)
|
||||
expect((await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "one" } })).statusCode)
|
||||
.toBe(200);
|
||||
|
||||
const resumed = await app.inject({ method: "POST", url: "/sessions/s2/resume" });
|
||||
@@ -1734,13 +1942,13 @@ test.each([
|
||||
getSettings: () => ({ workspace: "local" }) as any,
|
||||
});
|
||||
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { question: "old" } });
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "old" } });
|
||||
if (lifecycle === "close") {
|
||||
await app.inject({ method: "POST", url: "/sessions/s1/close" });
|
||||
await app.inject({ method: "POST", url: "/sessions/s1/resume" });
|
||||
} else {
|
||||
await app.inject({ method: "DELETE", url: "/sessions/s1" });
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { question: "replacement" } });
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "replacement" } });
|
||||
}
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
expect(current).toBe(replacement);
|
||||
@@ -1804,7 +2012,7 @@ test.each(["resolve", "reject"] as const)(
|
||||
getSettings: () => ({ workspace: "local" }) as any,
|
||||
});
|
||||
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { question: "old" } });
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "old" } });
|
||||
await app.inject({ method: "DELETE", url: "/sessions/s1" });
|
||||
published.length = 0;
|
||||
|
||||
@@ -1996,7 +2204,7 @@ test("Close suppresses a bootstrap that settles while close persistence is pendi
|
||||
getSettings: () => ({ workspace: "local" }) as any,
|
||||
});
|
||||
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
|
||||
const closeResponse = app.inject({ method: "POST", url: "/sessions/s1/close" });
|
||||
await closeStarted;
|
||||
published.length = 0;
|
||||
@@ -2070,7 +2278,7 @@ test("bootstrap failure persists once and keeps Resume serialized behind that pe
|
||||
getSettings: () => ({ workspace: "local" }) as any,
|
||||
});
|
||||
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
|
||||
oldConfigure.reject(new Error("configure failed"));
|
||||
await failureStarted;
|
||||
const resumeResponse = app.inject({ method: "POST", url: "/sessions/s1/resume" })
|
||||
@@ -2193,7 +2401,7 @@ test("a replaced runtime cannot publish or fail the newly resumed session", asyn
|
||||
readiness: { ensure: async () => ({ ok: true }) } as any,
|
||||
getSettings: () => ({ workspace: "local" }) as any,
|
||||
});
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
oldBridge.setState("idle");
|
||||
|
||||
@@ -2253,7 +2461,7 @@ test("a deleted runtime cannot repopulate or fail the forgotten session", async
|
||||
readiness: { ensure: async () => ({ ok: true }) } as any,
|
||||
getSettings: () => ({ workspace: "local" }) as any,
|
||||
});
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
|
||||
const response = await app.inject({ method: "DELETE", url: "/sessions/s1" });
|
||||
@@ -2298,7 +2506,7 @@ test("an unexpectedly exited runtime publishes its terminal sequence then releas
|
||||
readiness: { ensure: async () => ({ ok: true }) } as any,
|
||||
getSettings: () => ({ workspace: "local" }) as any,
|
||||
});
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
published.length = 0;
|
||||
|
||||
@@ -2349,7 +2557,7 @@ test("agent_end releases the Pi runtime after the session was finalized", async
|
||||
readiness: { ensure: async () => ({ ok: true }) } as any,
|
||||
getSettings: () => ({ workspace: "local" }) as any,
|
||||
});
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
|
||||
bridge.emitClientEvent({ type: "system_event", event: "agent_end" });
|
||||
@@ -2421,7 +2629,7 @@ test("POST /sessions/:id/response senza gate pendente risponde 409 (risposta sta
|
||||
getSettings: () => ({ workspace: "w" }),
|
||||
spawnFn: () => nodeSpawn("node", [FAKE, SCRIPT]) as any,
|
||||
});
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
|
||||
await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
|
||||
// The fake Pi never emitted a ui_request: the bridge has no pending descriptor, so a
|
||||
// response (stale UI, double submit) must be rejected instead of forwarded to Pi.
|
||||
const res = await app.inject({ method: "POST", url: "/sessions/s1/response",
|
||||
@@ -2562,7 +2770,7 @@ test("POST /sessions readiness failure returns one fixed public message without
|
||||
getSettings: () => ({ workspace: "psd" }) as any,
|
||||
spawnFn: () => nodeSpawn("node", [FAKE, SCRIPT]) as any,
|
||||
});
|
||||
const res = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
|
||||
const res = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
|
||||
expect(res.statusCode).toBe(503);
|
||||
expect(res.json()).toEqual({
|
||||
error: "Session services are not ready. Check configuration and connectivity, then try again.",
|
||||
@@ -2583,7 +2791,7 @@ test.each(["semantic_index_incompatible", "workspace_not_activatable"] as const)
|
||||
});
|
||||
|
||||
const response = await app.inject({
|
||||
method: "POST", url: "/sessions", payload: { question: "q" },
|
||||
method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" },
|
||||
});
|
||||
|
||||
expect(response.statusCode).toBe(503);
|
||||
@@ -2606,7 +2814,7 @@ test("POST /sessions returns storage 503 before creating a Pi runtime when sessi
|
||||
mgr: { createFor: () => { piCreated = true; throw new Error("must not spawn"); } } as any,
|
||||
});
|
||||
|
||||
const response = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
|
||||
const response = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
|
||||
|
||||
expect(response.statusCode).toBe(503);
|
||||
expect(response.json()).toEqual({ error: "session storage is unavailable" });
|
||||
@@ -2626,13 +2834,13 @@ test("POST /sessions proceeds when ollamaEnsure succeeds", async () => {
|
||||
getSettings: () => ({ workspace: "psd" }) as any,
|
||||
spawnFn: () => nodeSpawn("node", [FAKE, SCRIPT]) as any,
|
||||
});
|
||||
const res = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
|
||||
const res = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
|
||||
expect(res.json()).toEqual({ id: "s1" });
|
||||
expect(qdrantEnsure).toHaveBeenCalledWith(operationalWorkspace("psd"), 60, "self_heal");
|
||||
expect(ensureWs).toContain(`/snapshots/${"e".repeat(40)}/psd.yaml`);
|
||||
});
|
||||
|
||||
test("POST /sessions falls back from a stale requested model to the catalog default", async () => {
|
||||
test("POST /sessions rejects a stale requested model without silently using the default", async () => {
|
||||
let created = 0;
|
||||
let persisted: any;
|
||||
const runtime = { bridge: { onClientEvent: () => {} } };
|
||||
@@ -2659,16 +2867,13 @@ test("POST /sessions falls back from a stale requested model to the catalog defa
|
||||
const res = await app.inject({
|
||||
method: "POST",
|
||||
url: "/sessions",
|
||||
payload: { question: "q", provider: "deepseek", model: "deepseek-v4-pro" },
|
||||
payload: { interactionLanguage: "en", question: "q", provider: "deepseek", model: "deepseek-v4-pro" },
|
||||
});
|
||||
|
||||
expect(res.statusCode).toBe(200);
|
||||
expect(res.json()).toEqual({
|
||||
id: "fallback",
|
||||
warning: "Configured model deepseek/deepseek-v4-pro is unavailable; using zai/glm-5.2.",
|
||||
});
|
||||
expect(persisted).toMatchObject({ provider: "zai", model: "glm-5.2" });
|
||||
expect(created).toBe(1);
|
||||
expect(res.statusCode).toBe(503);
|
||||
expect(res.json()).toMatchObject({ code: "model_unavailable" });
|
||||
expect(persisted).toBeUndefined();
|
||||
expect(created).toBe(0);
|
||||
});
|
||||
|
||||
test("POST /sessions marks a persisted session failed when runtime construction throws", async () => {
|
||||
@@ -2697,7 +2902,7 @@ test("POST /sessions marks a persisted session failed when runtime construction
|
||||
],
|
||||
});
|
||||
|
||||
const res = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
|
||||
const res = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
|
||||
|
||||
expect(res.statusCode).toBe(503);
|
||||
expect(res.json()).toEqual({
|
||||
@@ -2790,7 +2995,7 @@ test.each([
|
||||
|
||||
try {
|
||||
const response = flow === "new"
|
||||
? await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } })
|
||||
? await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } })
|
||||
: await app.inject({ method: "POST", url: `/sessions/${sessionId}/resume` });
|
||||
const logs = consoleError.mock.calls.flat().map(String).join(" ");
|
||||
|
||||
@@ -2896,7 +3101,7 @@ test("POST /sessions returns after bridge attachment but starts only after retri
|
||||
getSettings: () => ({ workspace: "psd" }) as any,
|
||||
});
|
||||
|
||||
const res = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
|
||||
const res = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
|
||||
expect(res.json()).toEqual({ id: "s-early" });
|
||||
expect(bridgeAttached).toBe(true);
|
||||
expect(started).toBe(false);
|
||||
@@ -2943,7 +3148,7 @@ test("POST /sessions bootstrap failure emits only a fixed recovery message", asy
|
||||
const response = await app.inject({
|
||||
method: "POST",
|
||||
url: "/sessions",
|
||||
payload: { question: "q" },
|
||||
payload: { interactionLanguage: "en", question: "q" },
|
||||
});
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
|
||||
@@ -3039,7 +3244,7 @@ test.each([
|
||||
})),
|
||||
} as any,
|
||||
});
|
||||
const response = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
|
||||
const response = await app.inject({ method: "POST", url: "/sessions", payload: { interactionLanguage: "en", question: "q" } });
|
||||
|
||||
expect(response.statusCode).toBe(expectedStatus);
|
||||
expect(ensure).toHaveBeenCalledTimes(reachesReadiness ? 1 : 0);
|
||||
|
||||
@@ -160,8 +160,7 @@ test("GET /models returns session choices from the installation model catalog",
|
||||
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
|
||||
thtRunner: {} as any,
|
||||
runtimeModelCatalog: {
|
||||
defaultSession: "zai/glm-5.2",
|
||||
defaultMetadataGeneration: null,
|
||||
defaultInteraction: "zai/glm-5.2",
|
||||
embedding: { id: "ollama/qwen3-embedding:0.6b", dimensions: 1024 },
|
||||
sessionModels: () => [{
|
||||
id: "zai/glm-5.2", provider: "zai", model: "glm-5.2", label: "GLM 5.2",
|
||||
@@ -187,8 +186,7 @@ test("GET /models returns an empty list when the catalog has no session models",
|
||||
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
|
||||
thtRunner: {} as any,
|
||||
runtimeModelCatalog: {
|
||||
defaultSession: null,
|
||||
defaultMetadataGeneration: null,
|
||||
defaultInteraction: null,
|
||||
embedding: null,
|
||||
sessionModels: () => [],
|
||||
metadataModels: () => [],
|
||||
|
||||
@@ -34,6 +34,19 @@ test("sessionNew parses id from JSON", async () => {
|
||||
expect(await r.sessionNew({ question: "q" })).toEqual({ id: "2026-06-27-100000-x" });
|
||||
});
|
||||
|
||||
test("session language uses public per-command CLI flags and the selected config", async () => {
|
||||
const runner = new ThtRunner({ thtBin: "tht", harnessDir: "/h", configPath: "config/tht.yaml" });
|
||||
(spawn as any).mockClear();
|
||||
await runner.sessionNew({ question: "Pazienti", interactionLanguage: "en" });
|
||||
expect((spawn as any).mock.calls[0][1]).toEqual([
|
||||
"session", "new", "Pazienti", "--interaction-language", "en", "--json", "-c", "config/tht.yaml",
|
||||
]);
|
||||
await runner.ensureInteractionLanguage("s1");
|
||||
expect((spawn as any).mock.calls[1][1]).toEqual([
|
||||
"session", "ensure-interaction-language", "s1", "--json", "-c", "config/tht.yaml",
|
||||
]);
|
||||
});
|
||||
|
||||
test("searchPack persists retrieval context with session and workspace", async () => {
|
||||
const calls: any[] = [];
|
||||
const r = new ThtRunner({ thtBin: "tht", harnessDir: "/h", configPath: "config/tht.yaml" });
|
||||
|
||||
@@ -10,6 +10,28 @@ import {
|
||||
|
||||
const roots: string[] = [];
|
||||
|
||||
test("Evidence consolidation runs only its stage and preserves blocked Catalog readiness", async () => {
|
||||
const catalog = repository();
|
||||
const runChild = vi.fn(async () => ({ exitCode: 0, stdout: JSON.stringify({ status: "succeeded", counts: { documents: 2 } }), stderr: "" }));
|
||||
const result = await service({ repository: catalog, runChild }).consolidateEvidence({ workspaceId: "catalog-workspace" });
|
||||
expect(result.status).toBe("succeeded");
|
||||
expect(runChild).toHaveBeenCalledOnce();
|
||||
expect(runChild.mock.calls[0]?.[0]).toMatchObject({ argv: ["preprocess", "evidence", "--consolidate", "--json", "-c", "/dev/fd/3"] });
|
||||
expect(catalog.beginPreprocessing).not.toHaveBeenCalled();
|
||||
expect(catalog.finishPreprocessing).not.toHaveBeenCalled();
|
||||
expect(catalog.clearPreprocessing).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test("Evidence index failure reports retry without changing Catalog state", async () => {
|
||||
const catalog = repository();
|
||||
const runChild = vi.fn(async () => ({ exitCode: 1, stdout: JSON.stringify({ status: "failed", saved: true, error: "Saved; retry indexing." }), stderr: "private provider endpoint" }));
|
||||
const result = await service({ repository: catalog, runChild }).consolidateEvidence({ workspaceId: "catalog-workspace" });
|
||||
expect(result.status).toBe("failed");
|
||||
expect(result.warnings).toEqual(["Saved; retry indexing."]);
|
||||
expect(catalog.finishPreprocessing).not.toHaveBeenCalled();
|
||||
expect(JSON.stringify(result)).not.toContain("private provider");
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
roots.splice(0).forEach((root) => rmSync(root, { recursive: true, force: true }));
|
||||
});
|
||||
@@ -254,6 +276,24 @@ test("semantic preflight failure is persisted before returning", async () => {
|
||||
);
|
||||
});
|
||||
|
||||
test.each(["refresh", "decide"] as const)("Evidence %s calls only the source worker and preserves Catalog readiness", async action => {
|
||||
const catalog = repository();
|
||||
const runChild = vi.fn(async () => ({exitCode: 0, stdout: JSON.stringify({status: "succeeded", counts: {changed: 1}}), stderr: ""}));
|
||||
const result = await service({repository: catalog, runChild}).evidenceSources({workspaceId: "catalog-workspace", action,
|
||||
...(action === "decide" ? {sourceId: "a".repeat(64), revision: "b".repeat(64), decision: "replace" as const} : {}), actor: "Curator"});
|
||||
expect(result.status).toBe("succeeded");
|
||||
expect(runChild).toHaveBeenCalledWith({configPath: expect.any(String), argv: expect.arrayContaining(["evidence", "sources", action, "-c", "/dev/fd/3", "--actor", "Curator"])});
|
||||
expect(catalog.beginPreprocessing).not.toHaveBeenCalled();
|
||||
expect(catalog.finishPreprocessing).not.toHaveBeenCalled();
|
||||
expect(catalog.clearPreprocessing).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test("failed source activation reports the saved decision for retry", async () => {
|
||||
const result = await service({runChild: async () => ({exitCode: 1, stdout: JSON.stringify({status: "failed", saved: true, error: "Decision saved; retry indexing"}), stderr: "secret"})})
|
||||
.evidenceSources({workspaceId: "catalog-workspace", action: "decide", sourceId: "a".repeat(64), revision: "b".repeat(64), decision: "keep"});
|
||||
expect(result).toMatchObject({status: "failed", warnings: ["Decision saved; retry indexing"]});
|
||||
});
|
||||
|
||||
test("clear invalidates Catalog readiness before clearing only derived worker data", async () => {
|
||||
const catalog = repository();
|
||||
const runChild = vi.fn(async () => ({
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { execFile } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import {
|
||||
existsSync,
|
||||
mkdtempSync,
|
||||
@@ -202,7 +203,7 @@ test("deterministic operator leases are keyed by logical identity and stable acr
|
||||
workspaceSecretStore: f.workspaceSecretStore,
|
||||
});
|
||||
|
||||
const suffix = first.inputFingerprint.slice(7, 23);
|
||||
const suffix = createHash("sha256").update(first.inputFingerprint + "\n" + readFileSync(first.path, "utf8")).digest("hex").slice(0, 16);
|
||||
expect(second.path).toBe(first.path);
|
||||
expect(first.path).toBe(join(
|
||||
f.dataRoot,
|
||||
|
||||
@@ -242,6 +242,7 @@ test("separate runtime leases hand off byte-identical revision Evidence configs
|
||||
source_identity: "workspace://psd-clinical",
|
||||
});
|
||||
expect(parse(firstYaml).evidence).toEqual({
|
||||
local_archive_root: join(f.registryConfig.root, "repo", "psd-clinical"),
|
||||
sources: [{
|
||||
type: "filesystem",
|
||||
root: expectedRoot,
|
||||
|
||||
@@ -192,6 +192,7 @@ test("renders filesystem Evidence below the immutable revision content root with
|
||||
expect(rendered.runtime_identity.workspace_revision).toBe(evidenceRevision);
|
||||
expect(rendered.evidence).toEqual({
|
||||
schema_version: 2,
|
||||
local_archive_root: "/srv/registry/repo/psd-clinical",
|
||||
sources: [{
|
||||
type: "filesystem",
|
||||
root: `/srv/registry/snapshots/${evidenceRevision}/psd-clinical/evidence`,
|
||||
@@ -203,7 +204,7 @@ test("renders filesystem Evidence below the immutable revision content root with
|
||||
max_chunk_chars: 4_000,
|
||||
retain_published_generations: 3,
|
||||
});
|
||||
expect(yaml).not.toContain("/srv/registry/repo");
|
||||
expect(rendered.evidence.sources[0].root).not.toContain("/srv/registry/repo");
|
||||
});
|
||||
|
||||
test("renders public HTTP Evidence with exact fractional-second timeouts and every policy limit", () => {
|
||||
@@ -223,6 +224,7 @@ test("renders public HTTP Evidence with exact fractional-second timeouts and eve
|
||||
}));
|
||||
|
||||
expect(rendered.evidence).toEqual({
|
||||
local_archive_root: "/srv/registry/repo/psd-clinical",
|
||||
sources: [{
|
||||
type: "http",
|
||||
urls: ["https://evidence.example.test/guide.md"],
|
||||
|
||||
+3
-2
@@ -13,6 +13,7 @@ services:
|
||||
THT_BIN: /opt/venv/bin/tht
|
||||
THT_DATA_ROOT: /data
|
||||
SETTINGS_FILE: /data/settings/settings.json
|
||||
THT_EVIDENCE_HOST_REGISTRY_ROOT: ${THT_WORKSPACE_REGISTRY_ROOT:-}
|
||||
THT_MAINTENANCE_FILE: /data/settings/maintenance.json
|
||||
THT_WORKSPACE_REGISTRY_ROOT: /data/workspace-registry
|
||||
THT_WORKSPACE_GIT_REMOTE: ${THT_WORKSPACE_GIT_REMOTE:?set THT_WORKSPACE_GIT_REMOTE}
|
||||
@@ -99,7 +100,7 @@ services:
|
||||
image: thothii-core:local
|
||||
profiles: [catalog-maintenance]
|
||||
pull_policy: never
|
||||
command: ["node", "/app/backend/dist/catalog/migrate.js"]
|
||||
command: ["bash", "/app/docker/catalog-migrate.sh"]
|
||||
environment:
|
||||
THT_CATALOG_DB_HOST: catalog-db
|
||||
THT_CATALOG_DB_PORT: "5432"
|
||||
@@ -153,7 +154,7 @@ services:
|
||||
- type: volume
|
||||
source: workspace-registry
|
||||
target: /data/workspace-registry
|
||||
read_only: true
|
||||
read_only: false
|
||||
- type: volume
|
||||
source: sessions
|
||||
target: /data/sessions
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
# Optional local profile: expose the existing curator checkout on the installation host.
|
||||
# Copy the existing registry repo to this path before enabling the override. Snapshot/state
|
||||
# volumes remain unchanged. THT_EVIDENCE_HOST_REGISTRY_ROOT must be an absolute host path.
|
||||
services:
|
||||
core:
|
||||
environment:
|
||||
THT_EVIDENCE_HOST_REGISTRY_ROOT: ${THT_EVIDENCE_HOST_REGISTRY_ROOT:?set an absolute host registry path}
|
||||
volumes:
|
||||
- type: bind
|
||||
source: ${THT_EVIDENCE_HOST_REGISTRY_ROOT:?set an absolute host registry path}/repo
|
||||
target: /data/workspace-registry/repo
|
||||
bind:
|
||||
create_host_path: false
|
||||
workspace-maintenance:
|
||||
volumes:
|
||||
- type: bind
|
||||
source: ${THT_EVIDENCE_HOST_REGISTRY_ROOT:?set an absolute host registry path}/repo
|
||||
target: /data/workspace-registry/repo
|
||||
bind:
|
||||
create_host_path: false
|
||||
@@ -45,7 +45,7 @@ services:
|
||||
- type: bind
|
||||
source: ${THT_WORKSPACE_REGISTRY_ROOT:?set THT_WORKSPACE_REGISTRY_ROOT}
|
||||
target: /data/workspace-registry
|
||||
read_only: true
|
||||
read_only: false
|
||||
- type: bind
|
||||
source: ${THT_DATA_ROOT:?set THT_DATA_ROOT}/workspace-secrets
|
||||
target: /data/workspace-secrets
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
# The image is tagged from the running frontend before the visual review is published.
|
||||
services:
|
||||
frontend:
|
||||
image: thothii-frontend:before-visual-review-20260912
|
||||
@@ -0,0 +1,8 @@
|
||||
# Local-only review override. Apply after the existing local installation profiles.
|
||||
# Changes only the frontend image; Core, configuration and volumes remain unchanged.
|
||||
services:
|
||||
frontend:
|
||||
image: thothii-frontend:visual-review-20260912
|
||||
build:
|
||||
context: /Users/mp/projects/ThothII-visual-review
|
||||
dockerfile: docker/frontend.Dockerfile
|
||||
@@ -0,0 +1,5 @@
|
||||
# Local-only rollback to the full-shell frontend before visual integration.
|
||||
# Apply after local installation profiles, with --no-build and --no-deps.
|
||||
services:
|
||||
frontend:
|
||||
image: thothii-frontend:before-visual-shell-merge-20260913
|
||||
@@ -0,0 +1,17 @@
|
||||
services:
|
||||
core:
|
||||
environment:
|
||||
THT_EVIDENCE_HOST_REGISTRY_ROOT: /Users/mp/projects/ThothII/deploy/psd/evidence-registry
|
||||
volumes:
|
||||
- type: bind
|
||||
source: /Users/mp/projects/ThothII/deploy/psd/evidence-registry/repo
|
||||
target: /data/workspace-registry/repo
|
||||
bind:
|
||||
create_host_path: false
|
||||
workspace-maintenance:
|
||||
volumes:
|
||||
- type: bind
|
||||
source: /Users/mp/projects/ThothII/deploy/psd/evidence-registry/repo
|
||||
target: /data/workspace-registry/repo
|
||||
bind:
|
||||
create_host_path: false
|
||||
Submodule
+1
Submodule deploy/psd/evidence-registry/repo added at 24fb53236b
@@ -2,6 +2,10 @@
|
||||
# Replace every absolute path before using this as an advanced reference.
|
||||
schemaVersion: 2
|
||||
profile: local
|
||||
# Mac standalone example; the Omics server requires embedded/upstream separately.
|
||||
shell:
|
||||
mode: full
|
||||
defaultLocale: en
|
||||
projectDirectory: "<abs>/projects/ThothII"
|
||||
envFile: "<abs>/projects/ThothII/deploy/psd/operator.env"
|
||||
workspaceRepository:
|
||||
@@ -10,22 +14,28 @@ workspaceRepository:
|
||||
access: ssh
|
||||
modelCatalog:
|
||||
defaults:
|
||||
session: zai/glm-5.3
|
||||
metadataGeneration: zai/glm-5.3
|
||||
interaction: zai/glm-5.3
|
||||
embedding:
|
||||
id: ollama/qwen3-embedding:0.6b
|
||||
dimensions: 1024
|
||||
providers:
|
||||
deepseek:
|
||||
authentication:
|
||||
mode: pi_auth
|
||||
mode: secret_env
|
||||
apiKeyEnv: DEEPSEEK_API_KEY
|
||||
session:
|
||||
mode: pi_builtin
|
||||
metadataGeneration:
|
||||
litellmProvider: deepseek
|
||||
models:
|
||||
deepseek-v4-pro:
|
||||
label: DeepSeek V4 Pro
|
||||
session: {}
|
||||
metadataGeneration: {}
|
||||
deepseek-v4-flash:
|
||||
label: DeepSeek V4 Flash
|
||||
session: {}
|
||||
metadataGeneration: {}
|
||||
zai:
|
||||
endpoint:
|
||||
baseUrl: https://api.z.ai/api/coding/paas/v4
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
node /app/backend/dist/catalog/migrate.js
|
||||
/opt/venv/bin/python -m tht.memory.migrate
|
||||
@@ -131,7 +131,7 @@ ENV PATH="/opt/venv/bin:/usr/local/bin:$PATH" \
|
||||
HOME=/home/thoth
|
||||
|
||||
COPY scripts/verify-line-endings.sh /usr/local/bin/verify-line-endings
|
||||
COPY docker/core-entrypoint.sh docker/workspace-maintenance-entrypoint.sh docker/session-migrate.sh docker/ensure-pi-trust.mjs docker/embedding-model-init.sh /app/docker/
|
||||
COPY docker/core-entrypoint.sh docker/workspace-maintenance-entrypoint.sh docker/session-migrate.sh docker/catalog-migrate.sh docker/ensure-pi-trust.mjs docker/embedding-model-init.sh /app/docker/
|
||||
COPY docker/smoke/core-smoke.sh /app/docker/smoke/core-smoke.sh
|
||||
RUN /usr/local/bin/verify-line-endings /app/docker \
|
||||
&& chmod +x /app/docker/core-entrypoint.sh /app/docker/workspace-maintenance-entrypoint.sh /app/docker/session-migrate.sh /app/docker/embedding-model-init.sh /app/docker/smoke/core-smoke.sh
|
||||
|
||||
@@ -1,6 +1,26 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
|
||||
test "$(cat /usr/share/nginx/html/config.js)" = 'window.__THOTHII_CONFIG__ = {};'
|
||||
# The generic image contains an empty fallback; installed containers mount the generated
|
||||
# public projection over it. An optional path lets host projection tests use this same check.
|
||||
config_file=${1:-/usr/share/nginx/html/config.js}
|
||||
test -f "$config_file" && test -r "$config_file"
|
||||
config=$(tr -d '[:space:]' < "$config_file")
|
||||
case "$config" in
|
||||
'window.__THOTHII_CONFIG__={};')
|
||||
test -z "${THT_FRONTEND_CONFIG_REVISION:-}"
|
||||
;;
|
||||
*)
|
||||
# Installation Load validates BCP47 syntax. Here check only the public payload shape;
|
||||
# catalog availability and fallback belong to the frontend, not the generic image.
|
||||
locale='[A-Za-z][A-Za-z0-9]*(-[A-Za-z0-9]+)*'
|
||||
full="\"mode\":\"full\",\"defaultLocale\":\"$locale\""
|
||||
embedded="\"mode\":\"embedded\",\"defaultLocale\":\"$locale\",\"adapter\":\"omics-portal\""
|
||||
if ! printf '%s\n' "$config" | LC_ALL=C grep -Eq "^window\.__THOTHII_CONFIG__=\{\"backendBaseUrl\":\"/api\",\"shell\":\{($full|$embedded)\}\};$"; then
|
||||
echo "Invalid frontend public runtime configuration" >&2
|
||||
exit 1
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
|
||||
printf '%s\n' "frontend runtime config smoke: ok"
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
---
|
||||
status: accepted
|
||||
date: 2026-09-08
|
||||
---
|
||||
|
||||
# Use PostgreSQL for Memory and Qdrant for retrieval
|
||||
|
||||
The new Memory module uses the installation's existing PostgreSQL service as the
|
||||
authority for Memory Cards, their links, and structured schema dependencies.
|
||||
Qdrant holds a rebuildable search projection. This replaces the JSONL registry and
|
||||
allows related administrative changes to be coordinated in one database transaction.
|
||||
The owner accepted this direction in Q9 of the Memory design interview; implementation
|
||||
is pending.
|
||||
|
||||
Memory uses its own tables and remains a separate module from the Metadata Catalog.
|
||||
Sharing the PostgreSQL service does not transfer Memory ownership to Database
|
||||
management or make Evidence and Memory one canonical domain.
|
||||
|
||||
## Retrieval and graph
|
||||
|
||||
The owner also accepted hybrid semantic/lexical Qdrant retrieval with scope filters
|
||||
and explicit card links traversed in core. Both belong to the planned first version;
|
||||
there is no dedicated graph database or external Memory framework.
|
||||
|
||||
This extends [ADR 0017](0017-separate-reference-vectors-from-runtime-memory.md):
|
||||
the separate reference and memory collections remain, while Memory gains sparse
|
||||
lexical indexing in addition to dense vectors. Memory projections become rebuildable
|
||||
from the module's PostgreSQL authority. Preprocessing Clear still preserves Memory;
|
||||
this decision does not add it to the preprocessing cleanup scope.
|
||||
|
||||
Links support discovery. Finding a card through a link does not approve its use.
|
||||
The core proposes links with the cards for the same final review; Administration
|
||||
provides manual creation, editing, and deletion. Deleting a card removes its incident
|
||||
links without deleting the other linked cards.
|
||||
|
||||
## Considered options
|
||||
|
||||
- Retaining JSONL preserves the current storage format, but leaves coordinated
|
||||
card/link/dependency mutations and concurrent administrative writes to application code.
|
||||
- Using Qdrant as the sole authority is a viable alternative for record storage and
|
||||
retrieval. PostgreSQL is preferred for the coordinated mutations of the new module,
|
||||
with Qdrant reserved for its search projection.
|
||||
- Adding a dedicated graph database would introduce another service; the selected
|
||||
bounded traversal can be implemented in core over persisted links.
|
||||
|
||||
## Consequences
|
||||
|
||||
The module needs a persistence contract, PostgreSQL schema, and explicit propagation
|
||||
of additions, updates, and deletions to Qdrant. Choosing PostgreSQL does not make this
|
||||
propagation atomic across both systems: failures, retries, and invalidation of stale
|
||||
search content must be handled and tested. An index rebuild uses the original card
|
||||
content and cannot resurrect deleted cards.
|
||||
|
||||
The decision does not introduce Memory revision history or require compatibility with
|
||||
existing development sessions. Evidence authoring and publication remain governed by
|
||||
their own contract until the Evidence management project defines its evolution.
|
||||
|
||||
The [Memory management project](../plans/2026-09-08-memory-management.md) records the
|
||||
approved behavior, scope, and integration work.
|
||||
@@ -0,0 +1,152 @@
|
||||
---
|
||||
status: accepted
|
||||
date: 2026-09-08
|
||||
---
|
||||
|
||||
# Maintain local Evidence with external editors and manual consolidation
|
||||
|
||||
Evidence management must provide complete access to registered Evidence and a
|
||||
maintenance path, as accepted in Q11–Q15 and revised during simplification. The owner subsequently
|
||||
clarified that a context specialist writes drafts independently of the installation
|
||||
and without PostgreSQL access; the system refines and stores them locally for use
|
||||
and maintenance. Implementation is pending.
|
||||
|
||||
The storage mechanics of Q11 were revisited in the
|
||||
[simplification review](../plans/2026-09-08-memory-evidence-simplification-review.md).
|
||||
The original choice combined the workspace repository as Evidence authority with
|
||||
application-managed Git writes. The owner accepted the separation of external draft
|
||||
files/repositories from a durable local canonical file archive and removes
|
||||
commit/push from normal CRUD. The management surface must expose discoverable,
|
||||
editable Markdown for domain specialists, without JSONL editing. The owner chose
|
||||
external editors for release 0: their preferred Mac/PC editor or vim/nano on the
|
||||
server. No web editor or content form is required. The page keeps browsing,
|
||||
filtering, and detail, with actual host file paths and maintenance instructions.
|
||||
The owner also requested manual consolidation followed by manual Git diff, commit,
|
||||
and push, relying on operator discipline rather than automation. The owner accepted
|
||||
the remaining simplifications and requested explicit clarification of the core
|
||||
format change and the simple terminal-based Git check. The original application Git writer
|
||||
is not the final implementation prescription. A suggestion
|
||||
to move Evidence authority into PostgreSQL was withdrawn after the clarification.
|
||||
Memory remains a distinct module with its own PostgreSQL authority under
|
||||
[ADR 0018](0018-use-postgres-for-memory-and-qdrant-for-retrieval.md).
|
||||
|
||||
## Manual consolidation and Git follow-up
|
||||
|
||||
The external-editor decision replaces the form's single Save with saving files and
|
||||
running one consolidation command. The command checks expected structure, required
|
||||
fields, types, identifiers, references, and provenance. It reports the affected file
|
||||
and the necessary correction. Invalid input stops activation; valid input updates
|
||||
derived metadata, the corpus, and the Evidence index. New and deleted files use the
|
||||
same path. An inaccessible archive must not be interpreted as deleted Evidence.
|
||||
Structural validation does not prove semantic correctness or rerun model refinement
|
||||
to rewrite manually edited rules.
|
||||
|
||||
The core reads only successfully consolidated active content, never the editable
|
||||
working files directly. Candidate validation and activation must preserve the last
|
||||
valid corpus on failure, or report unavailability if its integrity cannot be
|
||||
guaranteed. Unconsolidated edits must not mix new text with old retrieval data.
|
||||
|
||||
Consolidation reports full local success only once the updated content is available
|
||||
to the core. If persistence succeeds but activation fails, the command reports the
|
||||
partial outcome and can be rerun without duplicating units or losing edits.
|
||||
The operation does not imply atomicity across
|
||||
authoritative storage and Qdrant; their failure and recovery behavior requires
|
||||
explicit implementation. An interrupted operation must not leave a partial corpus
|
||||
advertised as ready for subsequent use.
|
||||
|
||||
The maintained archive is a persistent Git working tree containing the local
|
||||
Evidence; it may reuse the workspace repository. Draft sources and curated units
|
||||
remain distinct even when in the same repository. Setup and the page identify the
|
||||
repository, working tree, actual host paths, branch, and configured remote. A
|
||||
rebuildable runtime snapshot is not the editing location.
|
||||
|
||||
Before consolidation, the operator checks Git status, including inspection of new
|
||||
untracked files. Normal terminal Git diff commands are available for line-level
|
||||
inspection when needed; no custom viewer or mandatory double review is required.
|
||||
After consolidation, the operator stages the Evidence and required metadata changes,
|
||||
commits, and pushes using normal Git commands. No watcher, automatic Git writes, retrying push, pull, merge,
|
||||
or dedicated web execution control is required. Missing credentials, unconfigured
|
||||
upstreams, and Git conflicts are handled by the operator.
|
||||
|
||||
Consolidation activates local changes before commit/push. If the operator omits the
|
||||
Git steps or push fails, the local update remains effective while transfer to the
|
||||
remote is incomplete. Pushing does not automatically update other installations.
|
||||
The system relies on operator discipline to complete the sequence; it does not add
|
||||
a separate editorial publication state machine or require a second reviewer.
|
||||
|
||||
Approved conflict repairs from the core still call the same persistence and
|
||||
activation service directly. They do not require an external editor or manual
|
||||
consolidation before the session can use its own approved correction. Their local
|
||||
file changes are included in the operator's subsequent Git maintenance.
|
||||
|
||||
## Administration without concurrent core work
|
||||
|
||||
The owner's simplification instruction replaces the earlier Q12 requirement to
|
||||
refresh open sessions after administrative edits. Core activity can be assumed
|
||||
absent during administration, or its overlap can be ignored. No dedicated live
|
||||
update, session notification, restart, maintenance mode, or reader coordination
|
||||
is required. Completed administrative changes apply to subsequent work.
|
||||
|
||||
Deliberate writes from the workflow itself still exist: approved conflict repairs
|
||||
and the final Memory summary use the same persistence services and handle their
|
||||
outcome before proceeding. In particular, a session must be able to use its own
|
||||
approved Evidence correction. This does not require updating all other sessions
|
||||
or rewriting previously approved decisions, artifacts, or SQL.
|
||||
|
||||
## Manual corrections survive source updates
|
||||
|
||||
When an updated source contradicts an administrator's correction, the saved manual
|
||||
Evidence remains active. Evidence management shows the conflicting content for an
|
||||
explicit decision and subsequent save. The source's newer content does not
|
||||
automatically override the correction. The owner accepts that the manual rule can
|
||||
remain in use until that comparison is resolved.
|
||||
|
||||
Deleted Evidence must not silently reappear after preparation or reindexing. The
|
||||
authoring implementation must preserve both manual corrections and suppression of
|
||||
deleted units through source refreshes. The current protection for uncommitted Git
|
||||
changes is insufficient: it does not preserve already committed manual corrections
|
||||
against regeneration, and retirement currently removes the unit without recording
|
||||
suppression for later preparation.
|
||||
|
||||
## Manual creation and explicit source refresh
|
||||
|
||||
Administrators can create Evidence without providing an external document. In R0,
|
||||
they add Markdown using the documented example and consolidate it. The application
|
||||
records a manual declaration as the managed source of the current statement.
|
||||
Explicit consolidation approves that declaration; it does not
|
||||
claim independent documentary verification.
|
||||
|
||||
A correction that changes a unit's meaning uses the same explicit manual origin.
|
||||
The original document remains linked for provenance and source-change detection,
|
||||
but its excerpt is not presented as support for a rule it does not contain. The
|
||||
canonical contract must distinguish the current supporting source from the original
|
||||
document rather than silently retaining outdated support metadata.
|
||||
|
||||
External sources are reacquired only when an administrator requests a source
|
||||
refresh. Ordinary saves and session lookups use the acquired local content; they
|
||||
do not poll sources or fetch their current versions. A remote change is therefore
|
||||
detected at the next requested refresh, when the manual-precedence rule applies.
|
||||
The revised storage recommendation reads the specialist's repository as an input;
|
||||
ordinary local edits do not write back to it or refresh its content automatically.
|
||||
|
||||
## Implementation consequences
|
||||
|
||||
The existing [Evidence lifecycle](../evidence.md) and
|
||||
[Workspace Evidence v3 contract](../contracts/workspace-evidence-v3.md) require
|
||||
explicit evolution for editable local Markdown, manual consolidation, and manual
|
||||
precedence. Source provenance and coherent canonical metadata remain requirements;
|
||||
manual changes must not be presented as statements supported by an unrelated source
|
||||
excerpt. Under the accepted local-file direction, curated files are primary
|
||||
installation data preserved by Clear and backed up separately from rebuildable
|
||||
indexes. Visible Markdown content must become authoritative for editing; operators
|
||||
must not maintain hidden duplicate text, hashes, or manifest entries themselves.
|
||||
This is a core Evidence contract change: parser, renderer, authoring, validation,
|
||||
normalization, and their integration with indexing and recall must be adapted
|
||||
together. E1 includes a new Curated unit format version, conversion of existing
|
||||
Evidence, reindexing, and end-to-end verification that visible edits reach core
|
||||
consumption. Preserve the internal typed model where possible; the change does
|
||||
not require redesigning the NL-to-SQL workflow phases.
|
||||
Local edits do not automatically change external drafts or another installation;
|
||||
Git versioning and transfer occur through the explicit manual follow-up.
|
||||
The [Evidence management project](../plans/2026-09-08-evidence-management.md)
|
||||
records the implementation sequence and acceptance checks.
|
||||
@@ -0,0 +1,27 @@
|
||||
# Unify administration pages and use namespaced routes
|
||||
|
||||
status: accepted
|
||||
|
||||
Workspace and Pi management will become full Administration Pages alongside Evidence, Memory and
|
||||
Database, using one shared Administrative Page Family. The Embedded Thoth Shell will identify the
|
||||
active Administration Surface through a namespaced browser route, `thoth_route=administration/<surface>`,
|
||||
so links remain reloadable and support Back/Forward without requiring new host-portal server routes.
|
||||
Transient form values and secrets stay outside the route; destructive confirmations may remain
|
||||
short-lived dialogs because they are actions, not page containers.
|
||||
|
||||
We considered React-only state, path-based routes and hash routes. React-only state loses refresh and
|
||||
deep-link behavior, path routes require a host catch-all route that does not yet exist in Omics Portal,
|
||||
and hash routes can conflict with host-page fragments. A namespaced query route preserves the current
|
||||
same-document integration boundary while leaving room for a future path adapter.
|
||||
|
||||
The accepted visual direction is A / Workbench for all five surfaces; B and C remain recoverable
|
||||
prototypes. Workspace owns readiness and preprocessing, consuming the Database catalog as a
|
||||
prerequisite. Database owns connection/binding, schema synchronization, descriptions and sensitivity;
|
||||
the preprocessing action remains exclusively in Workspace preparation.
|
||||
|
||||
On 2026-09-12, the owner refined this direction in
|
||||
[Gitea #29](https://git.tylconsulting.it/mptyl/ThothII/issues/29): Database opens at its
|
||||
catalog list, with its original table space and margins, and no Workspace preparation footer.
|
||||
This supersedes the earlier requirement for a preparation link in that page. Workspace
|
||||
remains reachable through the shared Administration navigation; its Preparation tab still
|
||||
links to Database configuration and schema when catalog work is needed.
|
||||
@@ -0,0 +1,101 @@
|
||||
# ADR 0021 — Shell separati e adapter sostituibile per il portale
|
||||
|
||||
- Stato: accettato
|
||||
- Data: 2026-09-13
|
||||
|
||||
## Decisione
|
||||
|
||||
ThothII espone due modalità di installazione, selezionate da `shell.mode`:
|
||||
|
||||
- `embedded` (default): ThothII è ospitato da Omics Portal. Non renderizza alcun header e
|
||||
riceve dal portale lingua, tema e fullscreen. L'accesso resta verificato dal server.
|
||||
- `full`: ThothII è autonomo. Renderizza il proprio header, con selettore lingua, tema,
|
||||
fullscreen e nome utente. Il click sul nome apre il logout. Non mostra mai la rotellina o
|
||||
altri comandi amministrativi del portale. Mantiene un rail vuoto a sinistra di almeno 20 px.
|
||||
|
||||
Il fatto che la shell sia `full` è distinto dallo stato `fullscreen`: la prima decide quale
|
||||
contenitore viene renderizzato, il secondo indica se è attiva la Fullscreen API del browser.
|
||||
L'icona passa da “entra in fullscreen” a “torna alla modalità normale”; anche `Esc` aggiorna lo
|
||||
stato visualizzato.
|
||||
|
||||
La configurazione installata resta semplice e retrocompatibile. Sul Mac di sviluppo il profilo
|
||||
locale userà:
|
||||
|
||||
```yaml
|
||||
shell:
|
||||
mode: full
|
||||
defaultLocale: en
|
||||
```
|
||||
|
||||
Il deploy sul server userà invece:
|
||||
|
||||
```yaml
|
||||
shell:
|
||||
mode: embedded
|
||||
adapter: omics-portal
|
||||
```
|
||||
|
||||
`defaultLocale` indica la lingua iniziale della shell full; in embedded la fonte autorevole resta
|
||||
il portale.
|
||||
|
||||
L'adapter è l'unico confine tra ThothII e il portale. La sua interfaccia pubblica è volutamente
|
||||
profonda e minima: consegna solo snapshot dello stato, senza esporre comandi, token, identità o
|
||||
dettagli di trasporto.
|
||||
|
||||
```ts
|
||||
export type HostShellState = {
|
||||
locale: string; // BCP-47, inizialmente it/en
|
||||
theme: "light" | "dark";
|
||||
fullscreen: boolean;
|
||||
};
|
||||
|
||||
export interface PortalAdapter {
|
||||
subscribe(
|
||||
onState: (state: HostShellState) => void,
|
||||
onError: (error: Error) => void,
|
||||
): () => void;
|
||||
}
|
||||
```
|
||||
|
||||
`OmicsPortalAdapter` è l'implementazione corrente. Un adapter per un altro portale potrà
|
||||
sostituirlo senza modificare shell, i18n o workflow. In `full` l'adapter non viene istanziato:
|
||||
lo stato è gestito internamente dalla shell.
|
||||
|
||||
Per l'integrazione oggi operativa, che monta la SPA direttamente nel DOM di Omics Portal,
|
||||
l'adapter legge la lingua effettiva dal selettore Omics, osserva l'attributo del tema e ascolta
|
||||
il fullscreen del documento. Selettori e osservatori restano privati dell'implementazione Omics.
|
||||
La lingua segue il normale ricaricamento Django; tema e fullscreen cambiano nella pagina aperta.
|
||||
La revisione approvata del 2026-09-13 elimina il precedente handshake a eventi: non servono
|
||||
messaggi personalizzati, versioni di trasporto o timeout di avvio. Un futuro adapter potrà usare
|
||||
un diverso trasporto senza modificare l'interfaccia applicativa.
|
||||
|
||||
Se `shell` o l'adapter embedded sono omessi, si usa `embedded` con `omics-portal`. Questo default
|
||||
supporta il documento Omics esistente; nomi adapter sconosciuti o dati host mancanti producono
|
||||
un errore esplicito, senza attivare la shell full.
|
||||
|
||||
## Confini che restano invariati
|
||||
|
||||
L'identità e l'autorizzazione del backend non vengono ricostruite nel browser. In embedded,
|
||||
Omics Portal continua a gestire login e logout e la catena server-side `auth_request` continua a
|
||||
fornire i principal header già previsti. Lo stato UI non dichiara l'utente autenticato: il modulo
|
||||
di accesso usa la verifica backend esistente anche alla riconnessione e al ritorno alla pagina.
|
||||
Un rifiuto su una singola operazione non equivale automaticamente alla perdita dell'accesso.
|
||||
|
||||
La lingua UI e la lingua di interazione con il modello restano separate dalla lingua del
|
||||
workspace; il relativo contratto è in [ADR 0022](0022-separate-ui-locale-from-session-interaction-language.md).
|
||||
|
||||
## Alternative scartate
|
||||
|
||||
- Duplicare l'header di Omics in embedded: crea due fonti di stato e incompatibilità visive.
|
||||
- Spargere controlli `if embedded/full` nei componenti: lega ogni pagina al portale.
|
||||
- Trasmettere utente o token nel bridge: aumenta superficie e accoppia UI e autenticazione.
|
||||
- Introdurre un protocollo completo request/response: non aggiunge funzionalità richiesta.
|
||||
- Usare `profile` per distinguere le shell: `profile` descrive la topologia dell'installazione,
|
||||
non la sua presentazione.
|
||||
|
||||
## Conseguenze
|
||||
|
||||
La soluzione richiede un adapter nel frontend che osserva il documento condiviso e mantiene la
|
||||
logica di shell locale a ThothII. Il backend non necessita di un nuovo protocollo di autenticazione
|
||||
o di una nuova sessione browser. Un nuovo portale deve soddisfare anche il contratto server di
|
||||
identità fidata: la sola sostituzione della classe UI non sostituisce quel contratto.
|
||||
@@ -0,0 +1,51 @@
|
||||
# Separate UI locale from session interaction language
|
||||
|
||||
status: accepted
|
||||
|
||||
ThothII distinguishes three language concepts:
|
||||
|
||||
- `workspace.language` remains the language of workspace-owned documents, descriptions and Evidence;
|
||||
- `ui_locale` controls deterministic ThothII chrome such as labels, form help, placeholders, errors,
|
||||
accessibility text and review-widget chrome;
|
||||
- `interaction_language` is persisted in a session and controls model-generated questions,
|
||||
explanations and reviewer proposals.
|
||||
|
||||
The initial locale catalog supports Italian and English and uses extensible BCP-47 language tags.
|
||||
Missing deterministic translations fall back to English. The selected UI locale supplies the default
|
||||
interaction language when a new session is created. A resumed session always uses its persisted
|
||||
interaction language; changing the host or full-shell UI locale must not silently rewrite an existing
|
||||
session or make its model output switch language mid-workflow.
|
||||
|
||||
The distinction is required because the current workspace contract already uses `language` for
|
||||
content and the PSD workspace is Italian. Reusing that field for a browser preference would make a
|
||||
visual choice mutate domain content semantics. The model receives the session interaction language
|
||||
through the session/Pi workflow context. SQL, identifiers, database values and other technical
|
||||
artifacts remain governed by their existing contracts and are not translated as UI strings.
|
||||
|
||||
In `full`, the local shell owns `ui_locale` and supplies it when starting a new session. In
|
||||
`embedded`, the host adapter is authoritative for `ui_locale`; ThothII applies host changes to
|
||||
deterministic UI immediately while preserving the interaction language of any active session.
|
||||
|
||||
We considered using only `workspace.language`, using only a global browser locale, and translating
|
||||
the model output after generation. The first conflates domain content with UI preference; the second
|
||||
cannot preserve a session's language or follow the host portal; and the third would be unsafe for
|
||||
structured reviewer decisions and would not control the model's reasoning or proposal language.
|
||||
|
||||
## Considered Options
|
||||
|
||||
- One mutable `language` field for workspace, UI and session was rejected because the fields have
|
||||
different owners and lifecycles.
|
||||
- Client-only translation of reviewer choices was rejected because choices can be generated by the
|
||||
model and must be requested in the intended language.
|
||||
- An English-only deterministic chrome was rejected because embedded and full installations must
|
||||
follow the selected host/user language.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Session creation and the persisted manifest gain an explicit interaction-language value.
|
||||
- Legacy manifests without that value use the workspace language, pinned idempotently on first
|
||||
resume; the browser locale must not determine this compatibility value.
|
||||
- Resume must read that value from the manifest and must not accept a new locale as an override.
|
||||
- The workflow prompt contract and deterministic reviewer-widget builders need a locale-aware input.
|
||||
- Frontend strings need a catalog and stable keys; backend events should expose stable codes where
|
||||
the frontend is responsible for localization.
|
||||
@@ -0,0 +1,130 @@
|
||||
# Rendering full ed embedded
|
||||
|
||||
ThothII ha una sola applicazione React, una sola build Vite e gli stessi servizi
|
||||
backend. «Doppio rendering» significa due modi di ospitare quella applicazione,
|
||||
non due versioni delle pagine e non rendering React sul server. Django renderizza
|
||||
il contenitore Omics; React renderizza ThothII nel browser, dentro `#root`.
|
||||
|
||||
## Tre decisioni indipendenti
|
||||
|
||||
| Decisione | Configurazione | Effetto |
|
||||
| --- | --- | --- |
|
||||
| Distribuzione | `profile: local` oppure `server` | Compose, percorsi e vincoli operativi |
|
||||
| Presentazione | `shell.mode: full` oppure `embedded` | Proprietario di header e preferenze |
|
||||
| Autenticazione | `auth.yaml` local/OIDC oppure `AUTH_MODE=upstream` | Chi verifica l'identità, come arriva al backend |
|
||||
|
||||
Il Mac usa **full + local**, con lingua iniziale inglese. L'integrazione Omics
|
||||
usa **embedded + upstream**, con accesso già verificato dal portale. Un server
|
||||
autonomo può usare **full + oidc**. Cambiare `shell.mode` non abilita un metodo
|
||||
di autenticazione e non modifica permessi o proprietari delle sessioni.
|
||||
|
||||
Full con upstream può visualizzare un'identità già verificata dal proxy, ma non
|
||||
ha un logout ThothII disponibile: non è il profilo autonomo con login/logout.
|
||||
Embedded non avvia login locale o OIDC anche se il backend è configurato così;
|
||||
questa combinazione non realizza il login unico Omics e non va usata come fallback.
|
||||
|
||||
## Composizione comune
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
CONFIG["config.js pubblico"] --> SHELL["ShellProvider"]
|
||||
FULL["Preferenze full nel browser"] --> SHELL
|
||||
HOST["Documento Omics"] --> ADAPTER["OmicsPortalAdapter: solo presentazione"]
|
||||
ADAPTER --> SHELL
|
||||
SHELL --> GATE["AuthGate: verifica GET /me"]
|
||||
GATE --> APP["AppShell: stesse pagine, sessioni e amministrazione"]
|
||||
AUTH["Backend: cookie locale/OIDC o identità upstream"] --> GATE
|
||||
```
|
||||
|
||||
`ShellProvider` risolve la configurazione, applica lingua/tema e monta i contenuti
|
||||
solo dopo uno snapshot host valido in embedded. `AuthGate` verifica l'accesso;
|
||||
`AppShell` e le pagine non devono leggere selettori o eventi specifici di Omics.
|
||||
Il cambio utente smonta lo stato applicativo della precedente identità.
|
||||
|
||||
## Full
|
||||
|
||||
- Header ThothII rosso Omics `#CB333B` in entrambi i temi; logo interamente chiaro.
|
||||
- Selettore EN/IT, tema light/dark, fullscreen e nome verificato dell'utente.
|
||||
- Menu del nome con logout soltanto per local/OIDC; nessuna rotellina admin.
|
||||
L'amministrazione resta nella navigazione applicativa, secondo i permessi.
|
||||
- Margine sinistro vuoto e simmetrico al destro: `max(20px, 1.5rem)`, normalmente
|
||||
24px con radice a 16px. Non è una seconda sidebar di navigazione.
|
||||
- Lingua e tema ricordati sullo stesso origin in `localStorage`, nelle chiavi
|
||||
`thothii:shell:locale` e `thothii:shell:theme`. Non sono preferenze server per
|
||||
utente. In assenza di preferenze: `defaultLocale` e tema light.
|
||||
- Fullscreen usa `document.documentElement.requestFullscreen()` e
|
||||
`document.exitFullscreen()`: nasconde il contorno del browser dove supportato.
|
||||
L'icona cambia sullo stato reale, anche dopo Esc; un rifiuto mostra un errore.
|
||||
Non è un semplice ingrandimento CSS e non scatta automaticamente all'accesso.
|
||||
|
||||
## Embedded
|
||||
|
||||
- Nessun header ThothII, selettore lingua, toggle tema, login o logout autonomo.
|
||||
I controlli rimangono nell'header generale Omics.
|
||||
- React è nello stesso documento della pagina `/kokoro/datamart-builder/`, non
|
||||
in un iframe. Non serve `postMessage` né un secondo protocollo di sessione.
|
||||
- L'adapter legge la lingua Django già confermata, osserva il tema del documento
|
||||
e ascolta il fullscreen reale. Le azioni rimangono di proprietà del portale.
|
||||
- Un contesto Omics mancante o invalido mostra un errore d'integrazione; non
|
||||
passa silenziosamente a full e non offre un secondo login.
|
||||
- Il portale assegna l'altezza disponibile sotto il proprio header: catena flex
|
||||
con `min-height: 0`, root contenuto e altezza applicativa vincolata al contenitore.
|
||||
Il contratto ThothII espone `--thoth-app-height` (fallback `100dvh`); verificare
|
||||
il contenitore reale, non presumere che l'intera viewport appartenga a React.
|
||||
Il template Omics mantiene inoltre i suoi override di compatibilità.
|
||||
|
||||
Il reset CSS è limitato al mount React e ai popup dell'applicazione, senza
|
||||
richiedere CSS `@scope`. I token e i popup seguono il tema applicativo. Questo
|
||||
non rende indipendenti fogli di stile arbitrari caricati dal portale: la verifica
|
||||
del documento condiviso rimane necessaria a ogni integrazione.
|
||||
|
||||
## Caricamento e configurazione pubblica
|
||||
|
||||
Il descrittore installato è la sorgente di verità. Il CLI genera
|
||||
`generated/frontend/config.js` e il suo mount di sola lettura nella proiezione
|
||||
`generated/compose.models.yaml`. Il file pubblico contiene solo `backendBaseUrl`
|
||||
e `shell`, mai identità, token, password o percorsi host. Va caricato **prima** del
|
||||
modulo React e servito senza cache. Nessuna build separata è richiesta per
|
||||
cambiare modalità; occorre rigenerare e applicare i mount tramite il lifecycle.
|
||||
|
||||
L'ordine Omics è: config pubblico → override del solo prefisso API → asset dal
|
||||
manifest Vite. L'override deve conservare `shell`; l'adapter non configura il proxy.
|
||||
Il default completo di shell omessa è embedded/en/omics-portal. Il CLI normalizza
|
||||
anche singoli campi omessi; un oggetto `shell` scritto manualmente nel browser
|
||||
deve invece contenere `mode` e `defaultLocale`, altrimenti viene rifiutato.
|
||||
|
||||
## Lingua, continuità e dati
|
||||
|
||||
La lingua UI traduce il testo dell'applicazione, non i contenuti di dominio.
|
||||
Alla creazione, la lingua UI viene acquisita come `interactionLanguage`; il
|
||||
manifest salva `interaction_language`, che governa domande e scelte del modello.
|
||||
Alla ripresa vale la lingua salvata, non l'ultima scelta dell'header. Per i manifest
|
||||
precedenti senza campo viene fissata la lingua workspace disponibile alla prima ripresa.
|
||||
|
||||
Il cambio lingua Omics invia il form Django e ricarica la pagina. ThothII conserva
|
||||
solo l'ID della selezione in `sessionStorage`, separato per pathname, issuer e
|
||||
subject. Riapre i documenti, non avvia una generazione. Bozze non inviate e modifiche
|
||||
non salvate richiedono conferma prima della navigazione; non sono una trascrizione
|
||||
salvata. La ripresa operativa resta esplicita.
|
||||
|
||||
## Punti di implementazione e manutenzione
|
||||
|
||||
| Sorgente | Responsabilità |
|
||||
| --- | --- |
|
||||
| `tools/tht/internal/config/shell.go` | Normalizzazione e validazione del descrittore |
|
||||
| `tools/tht/internal/modelprojection/projection.go` | Config pubblico e mount generati |
|
||||
| `frontend/src/api/runtime-config.ts` | Validazione browser e prefisso API same-origin |
|
||||
| `frontend/src/shell/host/ShellProvider.tsx` | Composizione, preferenze e tema |
|
||||
| `frontend/src/shell/host/FullHeader.tsx` | Controlli solo full |
|
||||
| `frontend/src/shell/host/OmicsPortalAdapter.ts` | Conoscenza del documento Omics |
|
||||
| `frontend/src/auth/AuthGate.tsx` | Accesso e ricontrolli al ritorno alla pagina |
|
||||
| `backend/src/auth/auth.ts` e `principal.ts` | Verifica server dell'identità |
|
||||
|
||||
Per un altro portale servono un'implementazione del
|
||||
[PortalAdapter](../contracts/portal-shell-adapter-v1.md), la sua registrazione nei
|
||||
validatori CLI/browser e nel punto di composizione, oltre al
|
||||
[contratto di autenticazione server](../install/authentication-upstream.md).
|
||||
Il nome di una classe non è un plugin caricabile dinamicamente da YAML.
|
||||
|
||||
Procedure: [configurazione e deploy](../operations/shell-and-localization.md),
|
||||
[autenticazione](authentication.md), [accettazione](../testing/authentication-manual-acceptance.md).
|
||||
@@ -1,18 +1,27 @@
|
||||
# Authentication architecture
|
||||
|
||||
ThothII has two production authentication modes: `local` and generic `oidc`. The host operator
|
||||
surface is one CLI, `tht`; there is no separate authentication executable. The backend owns
|
||||
opaque browser sessions and authorization, while `tht` owns protected configuration and local-user
|
||||
files.
|
||||
ThothII supports `local` and generic `oidc` through protected `auth.yaml`, plus
|
||||
the trusted-proxy `upstream` path used by Omics. The host operator surface is
|
||||
one CLI, `tht`; there is no separate authentication executable. The backend owns
|
||||
authorization in all paths and opaque browser sessions only in local/OIDC.
|
||||
`tht` owns protected local/OIDC configuration and local-user files; upstream
|
||||
identity is supplied per request by the authenticated server proxy.
|
||||
|
||||
Presentation is separate: [full/embedded rendering](application-shell.md) does
|
||||
not select authentication. The Mac uses full/local; Omics uses embedded/upstream;
|
||||
a standalone server can use full/OIDC. Do not configure a second ThothII OIDC
|
||||
login simply because Omics itself authenticates users through Authentik.
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
BROWSER["Browser"] --> BOUNDARY["Authentication boundary"]
|
||||
BOUNDARY --> LOCAL["Local users\nArgon2id hashes"]
|
||||
BOUNDARY --> OIDC["OIDC provider\nAuthorization Code PKCE"]
|
||||
BOUNDARY --> UPSTREAM["Trusted proxy\nVerified portal session"]
|
||||
OIDC --> GROUPS["Groups claim\nexact mapping"]
|
||||
LOCAL --> PRINCIPAL["Thoth principal"]
|
||||
GROUPS --> PRINCIPAL
|
||||
UPSTREAM --> PRINCIPAL
|
||||
PRINCIPAL --> ROLES["Roles"]
|
||||
ROLES --> PERMISSIONS["Permissions"]
|
||||
PERMISSIONS --> ROUTES["Protected routes"]
|
||||
@@ -21,6 +30,13 @@ flowchart TB
|
||||
|
||||
## Configuration and trust boundaries
|
||||
|
||||
The following protected-file configuration applies to local/OIDC. Upstream uses
|
||||
`AUTH_MODE=upstream` without a mounted `auth.yaml` or authentication runtime
|
||||
projection. The backend refuses both authorities together. `AUTH_MODE=none`
|
||||
and `mock` are development/test modes, not production fallbacks. The exact
|
||||
upstream setup, header contract, proxy hops and origin checks are in the
|
||||
[server integration guide](../install/authentication-upstream.md).
|
||||
|
||||
The installation descriptor points to an operator-controlled authentication directory. It contains
|
||||
non-secret `auth.yaml` and, for local mode, `users.yaml`. POSIX installations use a private
|
||||
directory and owner-only regular files; Windows uses equivalent owner-only ACLs. Secret values are
|
||||
@@ -33,17 +49,26 @@ The production role expansion from `backend/src/auth/config.ts` is exact:
|
||||
| Role | Permissions |
|
||||
|---|---|
|
||||
| `user` | `session.use` |
|
||||
| `admin` | `session.use`, `session.read_all`, `session.manage_all`, `settings.manage`, `workspace.manage`, `workspace.secrets.manage`, `database.manage`, `pi.manage`, `auth.diagnostics.read` |
|
||||
| `admin` | `session.use`, `session.read_all`, `session.manage_all`, `settings.manage`, `workspace.manage`, `workspace.secrets.manage`, `database.manage`, `memory.manage`, `evidence.manage`, `pi.manage`, `auth.diagnostics.read` |
|
||||
|
||||
`admin` therefore includes the ordinary `session.use` permission. No other role or permission
|
||||
label is part of the production catalog.
|
||||
|
||||
After validating a browser session, the backend expands its roles through the current permission
|
||||
catalog on every request. The permissions saved at login are a historical snapshot, so existing
|
||||
administrator sessions can use newly deployed administration features without signing in again.
|
||||
Session expiry, revocation and local-user role validation still apply before role expansion.
|
||||
|
||||
OIDC is provider-neutral at the browser protocol boundary. Authorization Code + PKCE, issuer,
|
||||
signature, audience, expiry, state, and nonce are validated before a principal is created.
|
||||
Authentik is the first certified group-catalog adapter, not a special browser login mode.
|
||||
|
||||
## Group authorization
|
||||
|
||||
This section describes **ThothII's direct OIDC login**, not the embedded Omics
|
||||
path. Omics checks its own capability and administrator status and supplies
|
||||
normalized identity headers; ThothII does not repeat the OIDC groups exchange.
|
||||
|
||||
OIDC must return a direct, non-empty `groups` claim whose value is a JSON array of strings.
|
||||
Missing, malformed, indirect, or overage-style claims fail closed. The browser callback returns
|
||||
HTTP 401 with the generic code `oidc_callback_failed`; it does not expose the internal reason.
|
||||
@@ -95,6 +120,10 @@ prerequisite fails.
|
||||
|
||||
## Browser sessions
|
||||
|
||||
This section applies only to **local and direct OIDC**. Upstream reuses the
|
||||
portal's authenticated session at the proxy boundary, not a ThothII cookie;
|
||||
its `/me` response has `session: null` and `csrfToken: null`.
|
||||
|
||||
The browser receives only an opaque `HttpOnly`, `SameSite=Lax` cookie named `thothii_session`.
|
||||
State-changing cookie requests require the in-memory CSRF token, same-origin `Origin`, and Fetch
|
||||
Metadata checks when present. The frontend never stores bearer tokens or session secrets in Web
|
||||
@@ -111,5 +140,20 @@ affected sessions. Authentication configuration revision changes invalidate all
|
||||
reload. Logout deletes the server record. Backup restore excludes active sessions and OIDC state,
|
||||
recreates empty private auth-state directories, and therefore forces reauthentication.
|
||||
|
||||
Full local/OIDC logout calls `POST /auth/logout`, revokes the server session and
|
||||
clears its cookie. It does not call the identity provider's global logout. If the
|
||||
provider still has an SSO session, the next OIDC login can complete without
|
||||
another password prompt. Embedded has no ThothII logout control: use the portal.
|
||||
|
||||
## Access revalidation
|
||||
|
||||
The frontend treats `/me` as the access authority. In embedded it does not fetch
|
||||
`/auth/config` or offer local/OIDC login. On focus, pageshow, visibility return
|
||||
and event-stream reconnection it rechecks access. A 401/403 from this probe clears
|
||||
protected state; a 403 on one operation is not automatically an app-wide logout.
|
||||
Neither the DOM adapter nor the proxy's initial SSE check guarantees instantaneous
|
||||
revocation of streams already open in other tabs.
|
||||
|
||||
See the [local guide](../install/authentication-local.md), [generic OIDC guide](../install/authentication-oidc.md),
|
||||
and [Authentik guide](../install/authentik.md) for operator procedures.
|
||||
[Authentik guide](../install/authentik.md), [upstream integration](../install/authentication-upstream.md),
|
||||
and [manual acceptance matrix](../testing/authentication-manual-acceptance.md).
|
||||
|
||||
@@ -7,7 +7,9 @@ This page complements the [architecture overview](overview.md) with the module s
|
||||
The frontend communicates with the backend through REST and SSE. The backend does not own session
|
||||
persistence: it starts Pi, invokes the `tht` CLI, and forwards events. It does own the separate
|
||||
installation-local database catalog. The harness contains the workflow, the Python CLI, and
|
||||
adapters for the DWH and vector store.
|
||||
adapters for the DWH and vector store. Its Memory module also owns the authoritative
|
||||
PostgreSQL archive of cards, links, dependencies and pending Qdrant projections.
|
||||
Administrative API calls use the same harness service as workflow producers and recall.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
@@ -20,6 +22,7 @@ flowchart LR
|
||||
THT --> FS["Sessions and artifacts\nworkspace repository"]
|
||||
THT --> DWH["DWH\nread-only"]
|
||||
THT --> VDB["Qdrant / vector store"]
|
||||
THT --> MEM["thoth_memory\nPostgreSQL Memory archive"]
|
||||
BE --> CFG["settings.json\nworkspace + thinking"]
|
||||
BE --> MODELS["generated runtime catalog\nfrom installation YAML"]
|
||||
BE --> CAT["catalog-db\nPostgreSQL + Kysely"]
|
||||
@@ -41,6 +44,14 @@ Dipendenze principali:
|
||||
|
||||
## Session sequence
|
||||
|
||||
The shared frontend is wrapped by `ShellProvider` (full preferences or a
|
||||
replaceable portal presentation adapter), then `AuthGate` (backend identity),
|
||||
then `AppShell`. Omics-specific DOM details belong only to `OmicsPortalAdapter`;
|
||||
credentials and principal validation belong to the server, never that adapter.
|
||||
Full/embedded do not duplicate the session workflow below. See
|
||||
[rendering architecture](application-shell.md) and
|
||||
[upstream identity](../install/authentication-upstream.md) for both boundaries.
|
||||
|
||||
The main path starts with a user question and ends with an SSE event. Reviewer decisions use the same channel and are persisted by the harness.
|
||||
|
||||
```mermaid
|
||||
|
||||
@@ -4,8 +4,11 @@
|
||||
|
||||
ThothII is a **human-in-the-loop datamart builder**. It turns a natural-language question into validated SQL, and optionally a dbt datamart, through a **deterministic eight-phase NL-to-SQL workflow** in which the model *proposes* and a human reviewer *decides* at gates.
|
||||
|
||||
Production authentication uses local authentication or generic OIDC. `tht` is the only operator CLI.
|
||||
For sessions, roles, groups, diagnostics, and recovery, see the [authentication documentation](authentication.md).
|
||||
Production authentication uses local authentication, generic OIDC, or the
|
||||
trusted-proxy upstream path used by Omics. `tht` is the operator CLI for the
|
||||
installation and local/OIDC configuration. For roles and recovery, see
|
||||
[authentication](authentication.md). One React build supports full and embedded;
|
||||
[rendering architecture](application-shell.md) separates presentation from identity.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
@@ -101,7 +104,9 @@ the core can admit a new session.
|
||||
|
||||
- `tht -c`/`--config` is a **per-command** option. It must follow the subcommand, never precede it (`ThtRunner.buildArgv` enforces this).
|
||||
- `--json` output must be plain JSON on stdout. It is a machine-readable contract.
|
||||
- UI strings are in English. Document *content* stays in the workspace language because it is the actual data; only chrome and labels are in English.
|
||||
- UI strings support English and Italian, with English fallback. Session interaction language
|
||||
is pinned at creation; document content remains in the workspace language. See
|
||||
[shell and localization](../operations/shell-and-localization.md).
|
||||
- Each workspace defines identity and optional Evidence only. The PostgreSQL Metadata Catalog defines
|
||||
its DWH target and binding; secrets remain in the protected workspace secret store.
|
||||
- Settings are global (`backend/data/settings.json`: workspace/thinking); provider/model choices are
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
# Session corrections to Memory and Evidence
|
||||
|
||||
`reviewer_archive_repair` presents one to five closed alternatives for a conflict.
|
||||
Each alternative updates one existing Memory Card or one existing local Evidence unit,
|
||||
showing complete current and resulting content. The reviewer chooses one alternative,
|
||||
rejects all as inadequate, or asks for reformulation. A correction does not approve or
|
||||
advance the session phase; the ordinary gate still reviews its use in the current question.
|
||||
|
||||
The harness coordinator `tht/archive_repair.py` joins two independent domains. It uses
|
||||
Memory's PostgreSQL repository, workspace mutation lock and projection service, and the
|
||||
same local Evidence save/consolidation boundary as administration. Evidence activation
|
||||
runs the existing corpus pipeline without reacquiring configured external sources.
|
||||
No database binding, schema configuration, other archive, or Git repository is changed.
|
||||
|
||||
## Authority and recovery
|
||||
|
||||
Migration `004_archive_repairs.sql` stores session-bound proposals and receipts in
|
||||
`thoth_memory.archive_repairs`, isolated by workspace RLS. Each receipt captures the
|
||||
session decision context, current target revision, complete resulting content, selected
|
||||
choice, acting principal and publication outcome. The preparation command changes no
|
||||
archive content. Application accepts only an option ID from the persisted proposal.
|
||||
|
||||
Memory writes its new revision and receipt in one SQL transaction. Projection failures
|
||||
leave a durable pending operation; retry propagates the saved revision. A later card
|
||||
edit invalidates replay of the correction.
|
||||
|
||||
Evidence records the exact choice before writing its canonical file. Recovery accepts
|
||||
either the reviewed original revision or the already-written approved result; it never
|
||||
overwrites a different intervening correction. Before a first proposal, the editable
|
||||
checkout must match its active snapshot. Other curated files are fingerprinted and
|
||||
checked again before application/retry so a session decision cannot publish unrelated
|
||||
external edits. A crash after replacement is recoverable by the same receipt. Original
|
||||
document provenance is retained as the lineage of the manual correction.
|
||||
|
||||
The gate displays these outcomes:
|
||||
|
||||
| Status | Meaning |
|
||||
| --- | --- |
|
||||
| `proposed` | Waiting for a human choice; no content saved |
|
||||
| `rejected` | All alternatives declined; reformulation required |
|
||||
| `applying` | Choice recorded; file write or candidate recovery still required |
|
||||
| `pending_activation` | Content saved; index activation incomplete |
|
||||
| `active` | Saved correction matches the currently active revision |
|
||||
| `superseded` | The target was removed or changed after the saved correction |
|
||||
|
||||
The reviewer may retry a pending correction or continue the current question while
|
||||
leaving activation explicitly pending. The latter is not persistent-resolution success.
|
||||
`repair-show` refreshes target status; `repairs` lists the historical receipt status.
|
||||
The final Memory summary must not create a duplicate of a correction already saved here.
|
||||
|
||||
## Authorization and commands
|
||||
|
||||
Inspection/preparation requires an accessible open session in the same workspace.
|
||||
Applying either archive correction requires an administrator. Browser responses are
|
||||
also checked against the responding principal's `memory.manage` or `evidence.manage`
|
||||
permission and the Pi runtime owner. An administrator using another principal's runtime
|
||||
must resume it under their own account first, preserving truthful receipt attribution.
|
||||
Non-administrators may reject proposals or continue question review without changing
|
||||
the archives. The Pi shell guard blocks `repair-apply`; only the human gate invokes it.
|
||||
|
||||
The Python workflow CLI provides `memory repair-target`, `repair-prepare`, `repair-show`,
|
||||
`repair-apply`, and `repairs`. These are workflow integration commands, not new native
|
||||
installation commands. All accept `--session` and command-local `-c`; JSON output remains
|
||||
machine-readable. Proposal bodies are bounded to 1 MB. Durable receipts support both
|
||||
filesystem and server session storage without a persisted chat transcript.
|
||||
@@ -0,0 +1,248 @@
|
||||
# Editable Curated Evidence v4
|
||||
|
||||
Curated unit v4 makes the visible Markdown body authoritative. It uses the existing
|
||||
typed Evidence payloads and stable identifiers. Workspace descriptor v4 and Evidence
|
||||
descriptor v1/v2 are separate version numbers.
|
||||
|
||||
E1 implements the format, explicit conversion, local archive and consolidation API.
|
||||
E2 connects that API to the installed consolidation command, Evidence management
|
||||
page and runtime source selection. The local PSD preview now uses 35 converted units.
|
||||
|
||||
## Write or edit a file
|
||||
|
||||
Place units in `<workspace-root>/evidence/curated/<kind>/<name>.md`. Keep the existing
|
||||
`id` when editing. The directory must match `kind`. A new manual unit needs no external
|
||||
document, hash or encoded metadata:
|
||||
|
||||
```markdown
|
||||
---
|
||||
schema_version: 4
|
||||
id: evidence:order-key
|
||||
kind: domain
|
||||
language: en
|
||||
purposes: [sql_generation]
|
||||
applies_to:
|
||||
tables: [sales.orders]
|
||||
---
|
||||
|
||||
# Order key
|
||||
|
||||
## Rule
|
||||
|
||||
Join orders using the order number, financial year and company.
|
||||
```
|
||||
|
||||
Required metadata is `schema_version`, `id`, `kind`, `language`, and a nonempty
|
||||
`purposes` list. Purposes are `disambiguation`, `rewriting`, `schema_linking`, and
|
||||
`sql_generation`. Optional `applies_to` contains `concepts`, `tables`, and `columns`.
|
||||
Tables use `schema.table`; columns use `schema.table.column`.
|
||||
|
||||
The first H1 is the title. H2 headings identify the payload fields below. Heading
|
||||
spelling follows `language`: Italian for `it` and its regional variants, English
|
||||
otherwise. Keep structural headings when editing the text below them.
|
||||
|
||||
| Kind | English field headings | Italian field headings |
|
||||
| --- | --- | --- |
|
||||
| `domain` | Rule | Regola |
|
||||
| `glossary` | Definition, Synonyms, Variants | Definizione, Sinonimi, Varianti |
|
||||
| `enum` | Column, Values | Colonna, Valori |
|
||||
| `example` | Question, Interpretation | Domanda, Interpretazione |
|
||||
| `mapping` | Concept, Tables, Columns | Concetto, Tabelle, Colonne |
|
||||
| `normalization` | Input, Output, Rule | Input, Output, Regola |
|
||||
| `formula` | Concept, Columns, SQL | Concetto, Colonne, SQL |
|
||||
| `reference` | URL, Label, Description | URL, Etichetta, Descrizione |
|
||||
|
||||
List fields use one `- value` per line. Empty optional lists may be omitted. Quoted
|
||||
JSON strings within bullets preserve unusual or multiline values during conversion.
|
||||
Enum values use `### "stored value"`, followed by their meaning; `### ""` represents
|
||||
an empty stored value. Formula SQL uses a fenced `sql` block containing one PostgreSQL
|
||||
expression. Whole queries and mutation statements remain invalid.
|
||||
|
||||
Nested prose headings and fenced examples are supported inside text fields. An H2
|
||||
matching a field heading is structural outside a code fence. Duplicate fields, missing
|
||||
required fields, duplicate metadata keys and malformed payloads are rejected. There
|
||||
is no second title or payload in frontmatter and no hidden authoritative rule text.
|
||||
Conversion fails explicitly if a legacy payload cannot be represented losslessly.
|
||||
|
||||
## Current provenance and original source
|
||||
|
||||
The host supplies document provenance when refining source material: relative source
|
||||
path, normalized source hash and exact supporting excerpts. A manual unit can omit
|
||||
`provenance`. Consolidation records `kind: manual` and the supplied curator identity.
|
||||
|
||||
A visible change to a previously recorded unit becomes a manual declaration. If that
|
||||
unit originated from a document, its former document provenance is retained under
|
||||
`original`. The original excerpts establish lineage; they do not assert that the
|
||||
source contains the new wording. Retrieval carries this distinction through typed
|
||||
metadata. Curators edit content; the service updates managed provenance.
|
||||
|
||||
Unchanged document declarations still require matching source bytes and excerpts.
|
||||
Replacing a source requires an explicit refresh through the E3 source-review path. Ordinary preparation
|
||||
is blocked on an initialized local archive, preventing regenerated source material
|
||||
from overwriting corrections or restoring deletions. Legacy `resolve` is likewise
|
||||
blocked there; local file corrections and the archive API own those changes.
|
||||
|
||||
## Persistent archive and activation
|
||||
|
||||
```text
|
||||
<workspace-root>/
|
||||
.evidence-archive.lock
|
||||
evidence/
|
||||
source/ # acquired original documents
|
||||
curated/<kind>/*.md # editable primary content
|
||||
local-manifest.yaml # derived declarations and deletion records
|
||||
.local/
|
||||
state.yaml # baseline, pending and active revisions
|
||||
snapshots/<revision>/ # immutable units, source bytes and manifest
|
||||
```
|
||||
|
||||
Keep the complete Evidence tree and its managed metadata in backups and the operator's
|
||||
Git review. Historical source bytes and deletion records are needed to preserve manual
|
||||
care across subsequent imports. The separate corpus cache and Qdrant index are derived.
|
||||
The lock file only coordinates local service operations.
|
||||
|
||||
`LocalEvidenceArchive.initialize()` records the pre-edit baseline without activation.
|
||||
`consolidate(actor=..., activate=...)` validates files, derives provenance, records
|
||||
deletions and source suppression, and creates an immutable candidate. The activation
|
||||
callback receives that snapshot and must raise if indexing is blocked or fails. Only
|
||||
successful activation advances `active_snapshot()`. Without a callback the result is
|
||||
explicitly `pending_activation`; saving a file alone never changes this pointer.
|
||||
|
||||
The same candidate can be retried after an index failure. Interrupted managed writes
|
||||
are replayed only when the operator's file bytes have not changed. A missing curated
|
||||
directory is an availability error, not proof that all units were deleted. Removing
|
||||
unit files from an accessible directory records deletion without deleting their sources.
|
||||
The API also provides `get`, revision-checked `save`, and revision-checked `remove` for
|
||||
future deliberate workflow corrections. Concurrent external edits produce conflicts.
|
||||
|
||||
These boundaries are exercised with the existing corpus pipeline and real Qdrant.
|
||||
An initialized installation reads only its active local snapshot, including during
|
||||
ordinary preprocessing. Unconsolidated edits are visible in Administration but do
|
||||
not enter retrieval. Before initialization, the pinned repository source still works.
|
||||
|
||||
## Administration and installed command
|
||||
|
||||
Open **Administration → Evidence management**. This independent page requires the
|
||||
`evidence.manage` permission and no active session. It shows complete units, source
|
||||
lineage, review items, file errors, filters, and changes relative to the active snapshot.
|
||||
Edit the displayed Markdown path using an external editor. Refresh files to inspect
|
||||
the result, then run the command shown by the page:
|
||||
|
||||
```sh
|
||||
tht --installation /absolute/path/thothii-installation.yaml \
|
||||
workspace evidence consolidate --workspace psd-clinical
|
||||
```
|
||||
|
||||
The installed command accepts `--json`. It validates and activates Evidence using
|
||||
the existing corpus pipeline and embedding service. It does not scan the DWH, run
|
||||
full workspace preprocessing, change Catalog/Schema readiness, or execute Git.
|
||||
If indexing fails after saving, the archive retains the candidate for retry and the
|
||||
previous active revision remains selected. Validation errors identify corrections
|
||||
to make in the files. Structural and review checks still apply; initialized archives
|
||||
do not require the legacy repository's fixed retrieval-evaluation fixture, whose
|
||||
expected IDs would otherwise prevent deliberate local deletions.
|
||||
|
||||
The canonical workspace root is `<workspace-registry-root>/repo/<workspace-id>`.
|
||||
Snapshots and the derived corpus are separate. For a registry in a Docker volume,
|
||||
copy its existing `repo` to a persistent host directory before enabling
|
||||
`deploy/compose.evidence-host.yaml`; set `THT_EVIDENCE_HOST_REGISTRY_ROOT` to that
|
||||
directory and include the override in the installation descriptor. Core and
|
||||
workspace-maintenance must mount the same checkout. Keep registry state/snapshots
|
||||
on their existing volume. The page only presents a host path when configured; it
|
||||
does not label an internal container path as a usable editor path.
|
||||
|
||||
After successful consolidation, inspect and commit the complete workspace Evidence
|
||||
tree, including managed manifests, snapshots, and deletion records, then push manually.
|
||||
The page provides quoted POSIX-shell examples for status, diff, add, commit, and push.
|
||||
Do not commit only the edited Markdown. Ordinary Git operations remain the operator's
|
||||
responsibility. E3 source decisions use this same activation boundary.
|
||||
|
||||
**Clear** removes derived Reference/corpus data while retaining editable files,
|
||||
snapshots and Memory. It still requires full workspace preprocessing to recreate
|
||||
Reference/Schema readiness; Evidence consolidation does not satisfy that gate.
|
||||
|
||||
## Import drafts and refresh sources
|
||||
|
||||
Place externally authored Markdown drafts in `<workspace-root>/evidence/incoming/`.
|
||||
The specialist needs no installation account or database access to write a draft.
|
||||
Copying the draft to the installation and choosing **Import or refresh sources** in
|
||||
Evidence management explicitly starts acquisition and refinement. The equivalent
|
||||
installed command is:
|
||||
|
||||
```sh
|
||||
tht --installation /absolute/path/thothii-installation.yaml \
|
||||
workspace evidence refresh --workspace psd-clinical
|
||||
```
|
||||
|
||||
This reads local `incoming/**/*.md` and original `source/**/*.md` files, excluding
|
||||
managed `source/acquired/` versions. It also reads configured HTTP and S3 sources
|
||||
through the existing read-only adapters and network policies. After local archive
|
||||
initialization, filesystem descriptors use this local authoring tree; ordinary
|
||||
runtime/preprocessing never fetches remote source changes. No source-server write
|
||||
credential is needed. The existing Pi authoring refiner runs once for each changed
|
||||
document, without session state or tools. Unchanged source hashes skip refinement.
|
||||
|
||||
All acquisitions and proposals must succeed before the new comparison set is
|
||||
recorded. An access or refinement failure preserves previous comparisons and active
|
||||
Evidence. A source absent from a successful discovery is marked missing and never
|
||||
treated as permission to delete units. Restore an accidentally missing local original
|
||||
file before consolidating its document-derived units, or explicitly retire those units.
|
||||
|
||||
Each changed source has a durable comparison showing current local units, complete
|
||||
proposed content, supporting excerpts, and IDs that replacement would retire:
|
||||
|
||||
- **Keep local Evidence** retains the current wording as a manual declaration, with
|
||||
its original documentary lineage preserved. The changed source is acknowledged;
|
||||
the next unchanged refresh does not reopen that decision.
|
||||
- **Use proposed Evidence** adopts the displayed proposal and explicitly retires the
|
||||
displayed omitted IDs. The source version and provenance change together.
|
||||
|
||||
Both choices save and activate through the same local consolidation/index pipeline.
|
||||
Review items block adoption; correct the original draft and refresh, or keep local
|
||||
content. There is no automatic merge based on a model's semantic conflict assessment.
|
||||
Any change to an affected curated file invalidates the comparison and requires a new
|
||||
refresh. If indexing fails after the decision is saved, use **Retry saved decision**;
|
||||
this reuses acquired content without fetching sources again. An intervening external
|
||||
edit is never silently overwritten by recovery.
|
||||
|
||||
Headless operators can make the same decision using the source ID and comparison
|
||||
revision from the local source registry or administration response:
|
||||
|
||||
```sh
|
||||
tht --installation /absolute/path/thothii-installation.yaml \
|
||||
workspace evidence decide --workspace psd-clinical \
|
||||
--source-id <64-hex-source-id> --revision <64-hex-comparison-revision> \
|
||||
--decision keep
|
||||
```
|
||||
|
||||
Use `--decision replace` to adopt the proposal. All installed commands accept `--json`.
|
||||
The Python `evidence sources` worker is internal to this installed command/API surface.
|
||||
|
||||
`evidence/.local/sources.json` stores source identities, comparisons and retry journals.
|
||||
`evidence/.local/acquisitions/` preserves original acquired bytes and credential-free
|
||||
remote provenance. Adopted normalized documents live under
|
||||
`evidence/source/acquired/<source-id>/<content-hash>.md`. Versioned paths let a new
|
||||
document and an older manual declaration's original source coexist. Include all of
|
||||
these files in the existing manual Git/backup sequence. Do not edit managed acquired
|
||||
versions: edit the original local draft or refresh its remote origin.
|
||||
|
||||
Deleted IDs remain reserved. Once a source has had a curated deletion, fresh model
|
||||
IDs from that source are conservatively suppressed as well: changing an ID must not
|
||||
restore retired knowledge. Existing surviving IDs can still receive reviewed updates;
|
||||
deliberate new knowledge can be written as a manual Evidence file. Refresh is bounded
|
||||
to 200 documents and 100 MiB per operation, in addition to each adapter's limits.
|
||||
|
||||
## Convert an existing workspace
|
||||
|
||||
The existing workflow CLI command `tht evidence migrate <workspace-root>` converts
|
||||
unit versions 1–3 to 4 deterministically and initializes the archive baseline. It
|
||||
preserves IDs, typed content, provenance and review items, with no model call or Git
|
||||
commit. It does not activate a local index. Review items still block consolidation.
|
||||
The command's existing Git-worktree path check remains in effect.
|
||||
|
||||
The first installed consolidation performs this conversion automatically when the
|
||||
legacy manifest is present, then validates and activates the result. Preserve the
|
||||
existing checkout in backups before upgrading. The E1 validation used an isolated
|
||||
copy; E2 also converted and indexed all 35 units on the running local preview.
|
||||
See the [E1 validation report](../plans/2026-09-09-evidence-e1-validation.md) and
|
||||
[E2 validation report](../plans/2026-09-09-evidence-e2-validation.md).
|
||||
@@ -0,0 +1,135 @@
|
||||
# Portal Shell Adapter v1
|
||||
|
||||
Contratto minimo della presentazione embedded. La revisione approvata il 2026-09-13
|
||||
sostituisce il precedente trasporto a eventi personalizzati con l'osservazione del
|
||||
documento condiviso. Non trasferisce identità, token o stato di autenticazione.
|
||||
|
||||
## Configurazione
|
||||
|
||||
Sul Mac:
|
||||
|
||||
```yaml
|
||||
shell:
|
||||
mode: full
|
||||
defaultLocale: en
|
||||
```
|
||||
|
||||
Sul server Omics:
|
||||
|
||||
```yaml
|
||||
shell:
|
||||
mode: embedded
|
||||
adapter: omics-portal
|
||||
```
|
||||
|
||||
Se `shell` o `mode` sono omessi, la modalità è embedded. L'adapter embedded
|
||||
predefinito è `omics-portal`; un nome sconosciuto è un errore di configurazione.
|
||||
Questi default sono normalizzati dal CLI prima della proiezione. Nel browser,
|
||||
shell interamente omessa ha gli stessi default, ma un oggetto `shell` parziale
|
||||
senza `mode` o `defaultLocale` viene rifiutato: non scrivere proiezioni a mano.
|
||||
Full non istanzia adapter. `defaultLocale` inizializza full; in embedded il locale
|
||||
proviene dal portale. L'autenticazione si configura separatamente dalla shell.
|
||||
|
||||
## API applicativa
|
||||
|
||||
```ts
|
||||
export type PortalSnapshot = {
|
||||
locale: string;
|
||||
theme: "light" | "dark";
|
||||
fullscreen: boolean;
|
||||
};
|
||||
|
||||
export interface PortalAdapter {
|
||||
subscribe(
|
||||
onState: (state: PortalSnapshot) => void,
|
||||
onError: (error: Error) => void,
|
||||
): () => void;
|
||||
}
|
||||
```
|
||||
|
||||
Una sottoscrizione installa gli osservatori e consegna lo snapshot iniziale senza
|
||||
richiedere messaggi all'altro applicativo. Gli aggiornamenti contengono snapshot
|
||||
completi e validati. La disiscrizione elimina tutti i listener e osservatori.
|
||||
La lingua viene risolta tramite i cataloghi UI, con fallback inglese.
|
||||
|
||||
## Implementazione Omics
|
||||
|
||||
L'integrazione monta React nel documento Django, non in un iframe.
|
||||
|
||||
| Dato | Fonte privata dell'adapter | Aggiornamento |
|
||||
| --- | --- | --- |
|
||||
| Locale | `data-lang` del selettore `.omics-language-select` | nuova pagina Django dopo `set_language` |
|
||||
| Tema | `data-bs-theme` su `html` | osservazione limitata a quell'attributo |
|
||||
| Fullscreen | stato effettivo del documento | evento del browser, inclusa uscita con Esc |
|
||||
|
||||
Il template Omics aggiornato allinea anche `html lang` alla lingua Django, ma
|
||||
la fonte dell'adapter rimane `select.omics-language-select[data-lang]`. Leggere
|
||||
il valore renderizzato evita di anticipare un cambio lingua prima che il form
|
||||
abbia successo. Cambiare soltanto `select.value` o `data-lang` senza il normale
|
||||
reload non è un trasporto runtime implementato per la lingua.
|
||||
|
||||
L'assenza del contesto host atteso produce un errore di integrazione; non abilita
|
||||
controlli locali. Non si introducono eventi `ready/state`, handshake, timeout,
|
||||
versioni dei messaggi o comandi duplicati. Selettori e dettagli Omics non devono
|
||||
essere letti dai componenti applicativi.
|
||||
|
||||
## Proprietà per modalità
|
||||
|
||||
| Funzione | Full | Embedded |
|
||||
| --- | --- | --- |
|
||||
| Header | ThothII | solo Omics |
|
||||
| Lingua | selettore locale | selettore Omics, normale reload Django |
|
||||
| Tema | toggle locale light/dark | stato Omics |
|
||||
| Fullscreen | controllo locale, stato reale | controllo Omics, stato reale |
|
||||
| Login/logout | ThothII local/OIDC; upstream non offre logout locale | autenticazione Omics esistente |
|
||||
| Nome utente | header ThothII | header Omics |
|
||||
| Rotellina amministrativa | mai | eventuale comando del portale |
|
||||
|
||||
## Accesso e continuità
|
||||
|
||||
Il server Omics verifica l'accesso a Datamart Builder e il proxy trasmette i
|
||||
principal header normalizzati al backend ThothII. La UI usa `/me`; non effettua
|
||||
un secondo login. Un altro portale deve soddisfare anche questo contratto server,
|
||||
oltre a fornire una nuova implementazione dell'adapter UI.
|
||||
|
||||
Il [contratto upstream](../install/authentication-upstream.md) specifica header,
|
||||
origine, rete e configurazioni incompatibili. Lo snapshot non può contenere
|
||||
`authenticated`, utente, ruoli, cookie o token; un evento browser non autorizza
|
||||
una richiesta API. Il prefisso API viene configurato separatamente prima del
|
||||
caricamento React, non viene dedotto dall'adapter.
|
||||
|
||||
Il logout del portale segue la sua navigazione. Una perdita di accesso rilevata
|
||||
dal server chiude lo stato protetto; un 403 di una singola operazione non equivale
|
||||
automaticamente a logout. La riconnessione degli eventi e il ritorno alla pagina
|
||||
ricontrollano l'accesso. Non si garantisce revoca istantanea di una connessione
|
||||
aperta in un'altra scheda attraverso il solo controllo iniziale del proxy.
|
||||
|
||||
Il cambio lingua può ricaricare la pagina: conservare la selezione della sessione,
|
||||
proteggere le modifiche non salvate e non avviare una nuova generazione al reload.
|
||||
Non si conserva una trascrizione integrale nel browser. La lingua della sessione
|
||||
rimane quella registrata nel manifest, secondo ADR 0022.
|
||||
|
||||
## Sostituzione e verifiche
|
||||
|
||||
Un nuovo adapter può usare un diverso documento o trasporto, ma deve rispettare
|
||||
la stessa sottoscrizione e mantenere la conoscenza del portale nella propria
|
||||
implementazione. Oggi `ShellProvider` istanzia direttamente `OmicsPortalAdapter`:
|
||||
per sostituirlo aggiornare quel punto e i nomi accettati in
|
||||
`tools/tht/internal/config/shell.go` e `frontend/src/api/runtime-config.ts`.
|
||||
Non è disponibile il caricamento dinamico di classi da una stringa YAML.
|
||||
Non occorre implementare ora iframe o un secondo portale.
|
||||
|
||||
La nuova implementazione deve pubblicare uno snapshot iniziale completo, poi gli
|
||||
aggiornamenti; segnalare contesto invalido; liberare tutti i listener alla
|
||||
disiscrizione. Locale ben formato ma non tradotto significa fallback inglese;
|
||||
locale assente/malformato e tema diverso da light/dark sono errori di integrazione.
|
||||
Non cambiare componenti applicativi o workflow per aggiungere selettori specifici
|
||||
del nuovo portale.
|
||||
|
||||
Verificare snapshot prima/dopo il montaggio, tema, fullscreen con Esc, cleanup,
|
||||
contesto host mancante, assenza di header ThothII embedded, accesso singolo,
|
||||
locale dopo reload e compatibilità del prefisso API. Full deve funzionare senza
|
||||
alcun elemento Omics presente.
|
||||
|
||||
Vedere anche [architettura del rendering](../architecture/application-shell.md)
|
||||
e [matrice di accettazione](../testing/authentication-manual-acceptance.md).
|
||||
@@ -6,8 +6,9 @@ When present, `evidence` is strict: it contains `source` and a defaulted strict
|
||||
source variant and the policy reject unknown keys.
|
||||
|
||||
The version numbers are intentionally separate: the Evidence descriptor supports v1/v2, while the
|
||||
latest Curated Evidence Unit format is v3. There is no Evidence descriptor v3/v4 and no Curated
|
||||
Evidence Unit v4.
|
||||
latest Curated Evidence Unit format is [v4](curated-evidence-v4.md). There is no Evidence
|
||||
descriptor v3/v4. The editable local archive is implemented in E1; installation integration
|
||||
is the next increment, E2.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
@@ -67,9 +68,11 @@ ignore it. Domain rules also retain their exact canonical text in an invisible `
|
||||
comment while presenting long prose as paragraphs, labelled subsections, and semicolon-derived
|
||||
lists. Runtime chunking reads the parsed canonical rule, not this review-only presentation.
|
||||
|
||||
Newly prepared units use v3. `tht evidence migrate <workspace-root>` upgrades v1 and v2 units and
|
||||
canonicalizes an older v3 presentation locally without a model call, commit, publication, or
|
||||
semantic change.
|
||||
The representations above are legacy conversion inputs. Newly prepared units use editable v4:
|
||||
short YAML metadata, a visible H1 title and typed H2 payload fields, with no hidden content copy.
|
||||
`tht evidence migrate <workspace-root>` converts v1–v3 to v4 and initializes a local archive
|
||||
baseline without a model call, commit, activation, or semantic change. See the
|
||||
[v4 editing and consolidation contract](curated-evidence-v4.md).
|
||||
|
||||
### Example: filesystem
|
||||
|
||||
|
||||
@@ -14,12 +14,30 @@ tht --installation <absolute>/thothii-installation.yaml workspace preprocess run
|
||||
|
||||
tht --installation <absolute>/thothii-installation.yaml workspace preprocess clear
|
||||
--workspace <id> [--json]
|
||||
|
||||
tht --installation <absolute>/thothii-installation.yaml workspace evidence consolidate
|
||||
--workspace <id> [--json]
|
||||
|
||||
tht --installation <absolute>/thothii-installation.yaml workspace evidence refresh
|
||||
--workspace <id> [--json]
|
||||
|
||||
tht --installation <absolute>/thothii-installation.yaml workspace evidence decide
|
||||
--workspace <id> --source-id <64-hex> --revision <64-hex>
|
||||
--decision keep|replace [--json]
|
||||
```
|
||||
|
||||
There are no public partial commands for DWH introspection, LSH, FK suggestions, schema indexing,
|
||||
Evidence indexing, or Qdrant rebuild. `preprocess run` does not accept `--resume`, `--dry-run`, a
|
||||
or Qdrant rebuild. The separate Evidence curation command validates editable local files and
|
||||
activates only their index; it does not satisfy workspace preprocessing readiness or execute Git.
|
||||
See [Curated Evidence v4](curated-evidence-v4.md#administration-and-installed-command).
|
||||
Source refresh acquires and refines only on explicit request, saving comparisons without
|
||||
changing active Evidence. Source decisions activate through the same Evidence-only
|
||||
pipeline and preserve Catalog readiness. Their envelopes reject arbitrary URLs, paths
|
||||
and extra flags; source locations and credentials come from installation configuration.
|
||||
`preprocess run` does not accept `--resume`, `--dry-run`, a
|
||||
generation identifier, or a rollback option. Re-running it replaces the preceding derived output.
|
||||
`preprocess clear` removes all replaceable preprocessing output and preserves runtime memory.
|
||||
`preprocess clear` removes all replaceable preprocessing output and preserves runtime memory
|
||||
and the canonical local Evidence archive. Full preprocessing remains required afterward.
|
||||
|
||||
## Sources of truth
|
||||
|
||||
@@ -29,8 +47,10 @@ generation identifier, or a rollback option. Re-running it replaces the precedin
|
||||
- PostgreSQL Metadata Catalog is the sole database authority. It owns the workspace/database
|
||||
association, installation-local binding, tables, columns, descriptions, sensitivity flags,
|
||||
physical foreign keys, and active logical relationships.
|
||||
- The workspace Git revision remains authoritative for Evidence. Evidence Descriptor v1/v2 and
|
||||
Curated Evidence Unit v3 are unchanged; there is no Evidence v4.
|
||||
- Evidence Descriptor v1/v2 still configures the initial source. Initialized archives use
|
||||
editable Curated Evidence Unit v4 and the last successfully activated local snapshot.
|
||||
Ordinary preprocessing never imports unconsolidated working-tree edits. Before initialization,
|
||||
the pinned workspace Git source remains supported.
|
||||
|
||||
No metadata is imported from legacy workspace YAML or `physical.yaml`/`annotations.yaml`.
|
||||
|
||||
@@ -98,9 +118,8 @@ generation produced for another database or revision.
|
||||
|
||||
`workspace preprocess clear` is intentionally narrower than deleting all semantic data. It:
|
||||
|
||||
1. copies any `memory` and `solved_question` records still present in the pre-split workspace
|
||||
collection to `<workspace>-memory`, then retires that legacy collection (a normal preprocessing
|
||||
write performs the same one-time cutover if clear was not invoked first);
|
||||
1. leaves `<workspace>-memory` unchanged; Memory projections are reconstructed only from
|
||||
the authoritative PostgreSQL archive, never imported from legacy vector payloads;
|
||||
2. deletes `<workspace>-reference`;
|
||||
3. removes the active LSH generation, Evidence corpus, private Catalog snapshot, and derived job
|
||||
checkpoints;
|
||||
|
||||
+52
-3
@@ -2,7 +2,30 @@
|
||||
|
||||
This page describes the complete workspace Evidence lifecycle: where original material lives, how curated units are produced, when they become available at runtime, and what the author and reviewer are responsible for.
|
||||
|
||||
## Publication rule
|
||||
## Editable local Evidence
|
||||
|
||||
E1 adds [Curated Evidence v4 and a persistent local archive](contracts/curated-evidence-v4.md).
|
||||
The visible title and payload fields are authoritative Markdown. New manual units need no
|
||||
external source; consolidation records their curator and distinguishes later corrections
|
||||
from original documentary provenance. The core archive API creates immutable candidates
|
||||
and advances its active pointer only after successful indexing.
|
||||
|
||||
E2 adds **Administration → Evidence management**, actual host file paths, complete
|
||||
browsing and filtering, and the installed `tht workspace evidence consolidate
|
||||
--workspace <id>` command. Edit files externally, consolidate to activate them, then
|
||||
review and run Git manually. Runtime consumes only the active local snapshot.
|
||||
The [v4 contract](contracts/curated-evidence-v4.md#administration-and-installed-command)
|
||||
describes host mounting, first conversion, failure recovery and Clear behavior.
|
||||
The local PSD preview has 35 converted and indexed units. E3 adds explicit import/refresh
|
||||
from local drafts and configured HTTP/S3 sources, durable current/proposed comparisons,
|
||||
and keep/replace decisions with activation and retry. See
|
||||
[Import drafts and refresh sources](contracts/curated-evidence-v4.md#import-drafts-and-refresh-sources).
|
||||
Workflow gate corrections remain the subsequent shared increment, X1.
|
||||
|
||||
## Existing repository publication path
|
||||
|
||||
The remainder describes the legacy, uninitialized repository source path. Initialized
|
||||
v4 local archives use the lifecycle above; manual declarations need no source document.
|
||||
|
||||
The workspace repository is the versioned source. ThothII reads it, validates it, and publishes an atomic generation. It does not modify, commit, or push the author's repository.
|
||||
|
||||
@@ -46,7 +69,33 @@ The legacy configuration may expose `source_root`, such as `${THT_DOCS_ROOT}` or
|
||||
|
||||
HTTP and S3 are separate adapters. They do not use the filesystem structure `source/` and `curated/`, but they must still provide stable provenance, without credentials in URIs, under the adapter-specific contract.
|
||||
|
||||
## What a curated unit must contain
|
||||
## What an editable curated unit contains
|
||||
|
||||
New preparation produces unit schema v4. The first H1 contains its title; documented H2
|
||||
sections contain the typed payload. A minimal manual unit is:
|
||||
|
||||
```markdown
|
||||
---
|
||||
schema_version: 4
|
||||
id: evidence:order-key
|
||||
kind: domain
|
||||
language: en
|
||||
purposes: [sql_generation]
|
||||
---
|
||||
|
||||
# Order key
|
||||
|
||||
## Rule
|
||||
|
||||
Join orders using the order number, financial year and company.
|
||||
```
|
||||
|
||||
Use `tht evidence migrate <workspace-root>` for deterministic legacy conversion. The
|
||||
conversion preserves typed content and initializes an archive baseline; it does not
|
||||
activate the local corpus. See the [v4 contract](contracts/curated-evidence-v4.md) for
|
||||
all eight kinds, provenance, file layout and the E1/E2 boundary.
|
||||
|
||||
## Legacy v3 representation
|
||||
|
||||
Canonical Curated Evidence v3 hides canonical machine metadata in an HTML comment and renders the
|
||||
whole review surface as real Markdown. GitHub therefore shows no frontmatter table. The body layout
|
||||
@@ -94,7 +143,7 @@ La fascia pediatrica comprende i pazienti con età inferiore a 18 anni.
|
||||
The actual files contain invisible `tht:` comments for canonical metadata and typed-field
|
||||
boundaries. Removing, duplicating, or desynchronizing them makes validation fail closed instead of
|
||||
silently ignoring content. Unit schemas v1 and v2 remain readable for compatibility, but newly
|
||||
prepared units use v3.
|
||||
prepared units use v4. Unit v3 remains readable as a conversion input.
|
||||
|
||||
Curated units must be atomic, readable by a second reviewer, and supported by the source.
|
||||
Provenance references must lead back to the original file and the passage that supports the claim.
|
||||
|
||||
@@ -8,14 +8,26 @@ declare providers, model allowlists, defaults, embeddings, dimensions, or vector
|
||||
Do not edit `deploy/pi/models.json`, `deploy/pi/settings.json`, files under `generated/`, or
|
||||
provider/model environment defaults. Those former sources are retired.
|
||||
|
||||
## Host instructions in Administration
|
||||
|
||||
The **Pi configuration** navigation button opens **Pi management**. Its Host maintenance
|
||||
section selects the installation host's Linux, macOS, or Windows tab automatically; the
|
||||
browser's operating system does not affect it. Selecting another tab is still possible.
|
||||
|
||||
The host CLI includes `THT_HOST_PLATFORM` in the generated Compose projection using the OS
|
||||
on which it runs. Generate projections on the destination host, not on another computer,
|
||||
and do not edit generated files. Without this projection (older deployments or native
|
||||
development), the backend reports its own OS; a Linux Docker container cannot discover
|
||||
whether the physical host is macOS or Windows. Run the updated host CLI's normal
|
||||
configuration reload on the destination installation to regenerate this information.
|
||||
|
||||
## Minimal catalog
|
||||
|
||||
```yaml
|
||||
schemaVersion: 2
|
||||
modelCatalog:
|
||||
defaults:
|
||||
session: zai/glm-5.3
|
||||
metadataGeneration: zai/glm-5.3
|
||||
interaction: zai/glm-5.3
|
||||
|
||||
embedding:
|
||||
id: ollama/qwen3-embedding:0.6b
|
||||
@@ -50,23 +62,63 @@ it contains that use block:
|
||||
- `metadataGeneration` makes it selectable for description generation;
|
||||
- `embedding` is a single installation-level model rather than a selectable list.
|
||||
|
||||
`defaults.session` is required. `defaults.metadataGeneration` is required exactly when at least
|
||||
one metadata-generation model exists. A session manifest pins its canonical identity, so removing a
|
||||
model never silently changes an existing session: resume fails with `model_unavailable`.
|
||||
`defaults.interaction` is the only LLM default, required once per installation, never per workspace.
|
||||
Core and Administration share the user's operational model choice. An explicit choice takes priority
|
||||
over the default and is remembered in this browser for the authenticated user and application mount.
|
||||
Switching workspace does not change the model. Browser-storage restrictions may limit remembering
|
||||
to the current visit; preferences do not synchronize across devices or change installation YAML.
|
||||
An unavailable remembered model is not silently replaced: select another configured model.
|
||||
|
||||
When any metadata-generation models are configured, the default and the operational list must
|
||||
support both `session` and `metadataGeneration`. Entries for just one adapter may remain in the
|
||||
installation inventory, but are not selectable for global interaction. Core-only installations remain
|
||||
supported when no metadata-generation model is configured; Admin AI is then unavailable.
|
||||
The embedding model remains separate and is unaffected by the interaction selector.
|
||||
|
||||
New sessions record the selected model in their manifest. Resume retains the session's workspace and
|
||||
revision, but uses the current global model (the installation default for clients that omit a model).
|
||||
Historical manifest model fields are not rewritten by resume. Archived/finalized sessions remain read-only.
|
||||
|
||||
### Required operator verification for every model
|
||||
|
||||
Catalog validation checks configuration, not model behavior. Before offering a model to users, and
|
||||
after changing its endpoint, adapters, or the Pi/LiteLLM versions, the operator must verify **both**:
|
||||
|
||||
1. **Core / Pi:** select the model, start a test session in a prepared test workspace, exercise an
|
||||
actual tool call and its returned result, a human review gate, and stop/resume. Check streaming,
|
||||
tool arguments, authentication, and reasoning/token-limit compatibility. A plain chat reply or
|
||||
`tht pi test` alone is not sufficient.
|
||||
2. **Administration / LiteLLM:** select the same model and generate descriptions for a small,
|
||||
non-sensitive test table. Check the structured result is accepted and the generation completes.
|
||||
Review the output quality before using it on real metadata. This action writes test metadata
|
||||
and may incur provider charges: use an authorized test database and approved data.
|
||||
|
||||
There is no automatic certification flag or startup model probe. The operator owns this verification;
|
||||
do not infer compatibility from the model label or from success in just one path. Both adapters point
|
||||
to one catalog identity; Pi does not need to route through a new LiteLLM proxy. Models using only
|
||||
`pi_auth` cannot serve the current LiteLLM path and are excluded from shared selection.
|
||||
|
||||
## Session adapters
|
||||
|
||||
Use `pi_builtin` for a model whose technical definition ships with Pi:
|
||||
Use `pi_builtin` for a model whose technical definition ships with Pi. This does not require
|
||||
`pi_auth`: a shared bundle credential lets native Pi and LiteLLM use the same provider identity:
|
||||
|
||||
```yaml
|
||||
deepseek:
|
||||
authentication:
|
||||
mode: pi_auth
|
||||
mode: secret_env
|
||||
apiKeyEnv: DEEPSEEK_API_KEY
|
||||
session:
|
||||
mode: pi_builtin
|
||||
metadataGeneration:
|
||||
litellmProvider: deepseek
|
||||
models:
|
||||
deepseek-v4-pro:
|
||||
session: {}
|
||||
metadataGeneration: {}
|
||||
deepseek-v4-flash:
|
||||
session: {}
|
||||
metadataGeneration: {}
|
||||
```
|
||||
|
||||
Use `openai_compatible` for an explicit compatible endpoint. Each eligible session model must then
|
||||
@@ -89,6 +141,18 @@ Secret values never belong in installation YAML, generated files, logs, CLI argu
|
||||
requests. The YAML contains only an environment-variable name or an authentication mode. Pi's
|
||||
protected credential file remains selected by the installation authentication configuration.
|
||||
|
||||
For catalog providers using `secret_env`, the bundle is authoritative in both Core and Admin.
|
||||
ThothII removes only the selected provider's old auth entry from the temporary Pi session snapshot;
|
||||
the operator's original Pi auth store and other providers are unchanged. Provider smoke checks use
|
||||
the same precedence, and model enumeration receives the catalog-declared bundle keys. A missing
|
||||
declared key is an error, not permission to fall back to Pi auth or the legacy generic key file.
|
||||
After rotating a bundle key, apply the normal installation lifecycle so processes reload it.
|
||||
|
||||
The PSD descriptor now declares only `deepseek/deepseek-v4-pro` and `deepseek/deepseek-v4-flash`
|
||||
for both uses; it no longer duplicates them under `deepseek-metadata`. Historical records are not
|
||||
rewritten. A saved obsolete identity must be explicitly reselected from the current catalog;
|
||||
it is not silently remapped to another model or account.
|
||||
|
||||
## Generated runtime projections
|
||||
|
||||
Before Compose starts, `tht` validates the installation and atomically writes deterministic files
|
||||
@@ -133,6 +197,16 @@ configuration command. There is no `tht pi configure` and no separate apply comm
|
||||
|
||||
## Migrating a legacy installation
|
||||
|
||||
For schema-v2 descriptors with the former `defaults.session` and `defaults.metadataGeneration`,
|
||||
replace both with `defaults.interaction`. Equal legacy values are accepted and normalized in memory;
|
||||
the loader never rewrites the descriptor. Different values fail with `migration_required`: explicitly
|
||||
choose a model supporting both uses, remove both old fields, and set the single new field. Do not mix
|
||||
new and legacy fields. A Core-only legacy session default can be normalized when Admin AI is absent.
|
||||
|
||||
The generated runtime catalog now uses schema version **2** and only `defaultInteraction`. Regenerate
|
||||
and apply all runtime projections with the matching host/backend release using the normal installation
|
||||
lifecycle; do not deploy only the backend against an old generated catalog or hand-edit generated JSON.
|
||||
|
||||
The migrator reads the former installation `metadataGeneration` block and the two former Pi JSON
|
||||
files, but never modifies them. Supply the facts that cannot be inferred safely and write a separate
|
||||
candidate:
|
||||
@@ -148,13 +222,15 @@ tht --installation /absolute/path/legacy/thothii-installation.yaml installation
|
||||
Review the candidate, move the legacy source files out of the installation only after approval,
|
||||
then select the v2 descriptor. Ambiguous aliases, endpoint conflicts, or missing authentication
|
||||
facts produce field-level errors; the migrator does not guess.
|
||||
The legacy CLI flag `--session-default` now supplies the unified interaction default in the candidate;
|
||||
if it conflicts with the legacy metadata default, align that choice explicitly before retrying.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Meaning | Action |
|
||||
| --- | --- | --- |
|
||||
| `migration_required` | A retired model source or installation schema is still present | Run the installation migrator and review its candidate |
|
||||
| Unknown session or metadata default | The canonical ID is missing the corresponding use block | Correct the provider/model key or add the intended use block |
|
||||
| Invalid interaction default | The canonical ID does not support all configured uses | Correct `defaults.interaction` or the intended adapter blocks |
|
||||
| Generated projection drift | Runtime files differ from the descriptor-derived bytes | Run `tht start` or `tht pi restart --yes --drain` |
|
||||
| `model_unavailable` on resume | The session's pinned model is no longer session-eligible | Restore that catalog entry or keep the session unavailable; do not remap it |
|
||||
| `model_unavailable` on create/resume | The selected global model is no longer eligible | Explicitly choose an eligible model; no fallback is applied |
|
||||
| Provider smoke failure | Credentials, endpoint, or provider availability is invalid | Correct the protected credential or catalog endpoint, restart, then run `tht pi test` |
|
||||
|
||||
+208
-222
@@ -1,244 +1,230 @@
|
||||
# Memory management
|
||||
|
||||
This document describes how Memory currently works in ThothII: its conceptual model, lifecycle, persistence, semantic search, review gates, session display, and main technical limits.
|
||||
Memory holds reusable knowledge for a workspace. PostgreSQL is the authoritative
|
||||
archive; Qdrant contains a rebuildable search projection. The harness owns both
|
||||
the administrative operations and the verification of retrieved results.
|
||||
|
||||
## Architectural summary
|
||||
## Administration
|
||||
|
||||
A Memory item is domain knowledge that can be reused across questions. It is not a copy of one question's schema linking.
|
||||
Open **Administration → Memory management**, immediately after Database management.
|
||||
An administrator selects the workspace explicitly. No active session, DWH binding,
|
||||
embedding service or Qdrant connection is required to browse and edit the archive.
|
||||
PostgreSQL must be available.
|
||||
|
||||
The page provides a paginated list, text/ID search, stable sorting, and combined
|
||||
filters for family, concepts, database, table, column, origin and update date.
|
||||
Filtering happens across the entire archive before pagination. Each card has a
|
||||
complete detail view, an editable form, cancellation of unsaved changes, and an
|
||||
explicit deletion confirmation.
|
||||
|
||||
| Family | Content |
|
||||
| --- | --- |
|
||||
| Domain clarification | A reusable definition or interpretation, with scope and context. |
|
||||
| SQL rule | Guidance for constructing SQL, with scope and rationale. |
|
||||
| Solved question | A question, its approved SQL and context; used as a consultative exemplar. |
|
||||
| Explained error | A correction and its rationale, to avoid repeating a known error. |
|
||||
|
||||
All cards have a stable `mem-<UUID>` identity, title, scope, origin and timestamps.
|
||||
Manual cards have no invented source session or decision. Workflow cards retain
|
||||
their source references when an administrator edits them. There is no editorial
|
||||
revision history.
|
||||
|
||||
The form also manages concepts, structured database/schema/table/column
|
||||
dependencies, and links to other cards with an explicit meaning. Links can only
|
||||
connect cards in the same workspace. Card, dependency and outgoing-link changes
|
||||
are committed together. Deleting a card removes its incident links and dependencies,
|
||||
while keeping the other cards, Evidence and Catalog metadata.
|
||||
|
||||
## Save, failure and recovery
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
CLARIFY["F1 concept clarified"] --> REVIEW["F8 reviewer review"]
|
||||
REVIEW -->|"accepted"| REGISTRY["registry.jsonl"]
|
||||
REVIEW -->|"declined"| LOCAL["Session decision only"]
|
||||
REGISTRY --> VECTOR["Qdrant semantic index"]
|
||||
VECTOR --> FUTURE["Future F2 retrieval"]
|
||||
FUTURE --> PROPOSAL["Reviewer proposal"]
|
||||
flowchart LR
|
||||
EDIT["Admin or workflow"] --> SQL["PostgreSQL transaction"]
|
||||
SQL --> CARD["Current card and links"]
|
||||
SQL --> WORK["Pending projection"]
|
||||
WORK --> Q["Qdrant"]
|
||||
Q --> CHECK["Verify current card and projection"]
|
||||
CARD --> CHECK
|
||||
CHECK --> REVIEW["Recall for human review"]
|
||||
```
|
||||
|
||||
```text
|
||||
F1: clarify a concept
|
||||
│
|
||||
▼
|
||||
concept_clarified decision in the session ledger
|
||||
│
|
||||
▼
|
||||
F8: reviewer decides whether to promote it
|
||||
│
|
||||
├── global registry registry.jsonl
|
||||
└── Qdrant semantic index
|
||||
│
|
||||
▼
|
||||
F2 in a future session
|
||||
search and proposal to the reviewer
|
||||
```
|
||||
The service serializes each workspace's mutations. It first commits content and
|
||||
the projection operation in PostgreSQL, then propagates to Qdrant. A failed save
|
||||
is different from **saved, index update incomplete**. In the latter case the
|
||||
current content is already available in administration, while its previous vector
|
||||
result is excluded from recall.
|
||||
|
||||
The main invariant is `REUSABLE_TYPES = {"concept_clarified"}`: only clarified concepts can be generated, saved, searched, or proposed as Memory. Decisions such as `table_promoted`, `table_excluded`, and `column_promoted` remain local to the question.
|
||||
The **Pending index updates** section offers an explicit retry, including cleanup
|
||||
for deleted cards. These operations survive a restart. Retry reads the current
|
||||
archive state and cannot restore an earlier edit or a deleted card. Technical
|
||||
revisions are internal consistency markers, not user-managed card statuses.
|
||||
|
||||
## The three management levels
|
||||
Every recall hit must resolve to a current card in the requested workspace, with
|
||||
a matching valid projection. The response is reconstructed from PostgreSQL, never
|
||||
from an unverified Qdrant payload. Deleted, orphaned and stale points are excluded
|
||||
for both domain Memory and solved-question exemplars. An unavailable archive is
|
||||
an operational error, not a successful empty archive.
|
||||
|
||||
| Level | Content | Function |
|
||||
Rebuilding Memory uses only authoritative cards. It does not import JSONL files,
|
||||
historical session artifacts or old vector payloads. Reference preprocessing keeps
|
||||
the Memory collection separate and does not migrate legacy Memory payloads.
|
||||
|
||||
## Hybrid recall and links
|
||||
|
||||
Memory uses dense embeddings and Qdrant BM25, fused with reciprocal rank fusion.
|
||||
Both branches receive the same workspace, family and scope filters before candidate
|
||||
selection. Indexed text includes content, scope, rationale, question, exemplar SQL,
|
||||
concepts and qualified physical dependencies. Both indexing and querying use the
|
||||
workspace language (`en` or `it`), including manual administration without a DWH binding.
|
||||
|
||||
The workflow CLI binds recall to its configured database and schema. Optional
|
||||
`--filters` JSON can narrow the business `scope`, `concepts`, `table` and `column`.
|
||||
Business scope is an exact string; every requested concept must be present. Physical
|
||||
context matches a single structured dependency: database, schema, table and column
|
||||
cannot be satisfied by unrelated entries. A card without dependencies is workspace-wide;
|
||||
a database-only or table-only dependency also applies to descendants. Without a table
|
||||
filter, table-specific knowledge in the selected database/schema remains discoverable.
|
||||
Filters cannot override the configured database/schema. Free-text business scope is
|
||||
not automatically interpreted or inferred from the question.
|
||||
|
||||
The core expands outgoing links from current, eligible search candidates. It applies
|
||||
the same filters and workflow-family restrictions to destinations and intermediate
|
||||
cards. Removed, pending, previously decided and out-of-scope cards cannot act as bridges.
|
||||
Traversal allows two hops, at most 20 outgoing links per card (stable target-ID order),
|
||||
200 distinct card lookups and 400 inspected links in total. Search requests
|
||||
`min(100, max(20, 3 × top))` seeds; the result limit remains between 1 and 100.
|
||||
|
||||
All candidates are ranked together using `1 / (60 + seed rank)` for direct hits,
|
||||
plus the strongest linked contribution, decayed by `0.5` per hop. Repeated paths
|
||||
do not accumulate votes. Ties use card ID. The returned `score` is a ranking score,
|
||||
not cosine similarity or a confidence estimate. `retrieval.path` shows the strongest
|
||||
link path, or just the card ID for an exclusively direct result. PostgreSQL content,
|
||||
links and eligibility are resolved under the workspace operation lock after search.
|
||||
|
||||
Migration `002_hybrid_projection.sql` marks the format of old dense projections as
|
||||
incompatible without changing their authoritative cards. They appear in pending
|
||||
updates and are excluded from recall until explicit retry or `memory index` succeeds.
|
||||
Memory writes can add a missing BM25 sparse vector to their collection; an incompatible
|
||||
existing vector configuration fails visibly and leaves recovery pending. Reading never
|
||||
silently falls back to dense retrieval. Reference remains independently managed.
|
||||
An explicit `memory index` can also recreate a missing Memory collection before
|
||||
rebuilding its projections; it never imports records from another source.
|
||||
|
||||
## Workflow integration
|
||||
|
||||
During the workflow, Pi prepares reusable proposals in the session artifact
|
||||
`memory_proposals.json`. Each proposal names effective approved source decisions,
|
||||
the content and scope, its rationale, and any physical dependencies or links.
|
||||
Unexplained failures, rejected options and simple table selections do not create
|
||||
reusable knowledge. Exact existing cards are reused; semantic similarity alone
|
||||
never authorizes replacement. Updates name the existing card and its current revision.
|
||||
|
||||
At F8, `reviewer_memory_promote` presents one editable Memory summary, including
|
||||
the approved solved question. The reviewer chooses additions and updates, edits
|
||||
their content, scope, dependencies and links, or declines everything. Approved SQL
|
||||
is read only here: changing the solution requires returning to SQL review.
|
||||
Only selected cards and their links are committed. Invalid links or stale updates
|
||||
roll back the entire selection. Links between selected new cards are resolved
|
||||
inside the same transaction. An explicit update preserves the existing identity
|
||||
and origin, including manually authored content.
|
||||
|
||||
A durable review receipt makes repeated delivery idempotent and recovers the
|
||||
gap between saving Memory and recording `memory_summary_reviewed` in the session
|
||||
ledger. Retry never recreates a deleted card. The gate then closes F8 and finalizes;
|
||||
finalization itself performs no automatic Memory writes. Pending indexing remains
|
||||
visible and recoverable in administration.
|
||||
|
||||
F2 consumes domain clarifications and excludes already decided Memory. In F4,
|
||||
F6 and F7, `memory rules` retrieves applicable SQL rules and explained errors.
|
||||
Pi presents their use in the existing schema, CTE or SQL approval gate.
|
||||
Exemplar search remains consultative. Retrieval never constitutes approval.
|
||||
Persistent Memory/Evidence conflict repair remains part of the joint X1 increment.
|
||||
|
||||
## Physical schema changes
|
||||
|
||||
After a successful physical Catalog synchronization, the backend passes the exact
|
||||
removed tables and columns, database, schema and run identity to Memory. Only cards
|
||||
with matching structured dependencies are deleted, together with their incident
|
||||
links and searchable projections. Global cards and objects outside the synchronized
|
||||
scope survive. Manual Catalog cleanup and failed DWH scans never trigger this deletion.
|
||||
|
||||
The Catalog transaction records a pending `memory_cleanup` phase before committing
|
||||
the physical change. If cleanup or indexing fails, the run retains the original
|
||||
removals. Retry completes that same operation without rescanning the DWH or relying
|
||||
on a new diff. A new synchronization is blocked until this cleanup is completed.
|
||||
The Memory deletion receipt and vector tombstones make the operation repeatable
|
||||
across restarts. The synchronization drawer reports deleted Memory cards and errors.
|
||||
|
||||
## API and commands
|
||||
|
||||
The administrative HTTP surface requires `memory.manage`, included in the existing
|
||||
admin role. The backend checks workspace identity and passes its trusted principal
|
||||
and a protected request snapshot to the harness. The harness independently checks
|
||||
the principal; ordinary users cannot bypass administration through the CLI.
|
||||
Production authentication and CSRF protections apply to the new routes.
|
||||
|
||||
| Method | Path below `/api/workspaces/:workspaceId/memory` | Operation |
|
||||
| --- | --- | --- |
|
||||
| Session ledger | `concept_clarified`, `memory_promoted`, `memory_promotion_declined` | Audit and state for one session |
|
||||
| Global registry | `mem-XXXX` records in `registry.jsonl` | Current canonical Memory archive |
|
||||
| Qdrant index | Embeddings and metadata derived from the registry | Semantic search |
|
||||
| GET | root | Search, filter and paginate the archive |
|
||||
| POST | root | Create a card |
|
||||
| GET | `/:cardId` | Read a complete card |
|
||||
| PUT | `/:cardId` | Save content, links and dependencies together |
|
||||
| DELETE | `/:cardId` | Delete a card and incident links |
|
||||
| GET | `/pending` | List incomplete projection operations |
|
||||
| POST | `/:cardId/retry` | Retry current projection work, including deletion |
|
||||
|
||||
The ledger contains provenance and human decisions. The global record contains reusable text. The vector index is a search projection, not the place where the workflow records decisions directly.
|
||||
|
||||
## What can become Memory
|
||||
|
||||
During F1, the workflow records clarifications as `concept_clarified` decisions. A clarification can express:
|
||||
|
||||
- definitions of production or organizational concepts;
|
||||
- inclusion and exclusion criteria for a product line;
|
||||
- formulas and calculation methods;
|
||||
- interpretations of time periods;
|
||||
- mappings to specific tables and columns;
|
||||
- the meaning of flags, codes, or indicators.
|
||||
|
||||
The `MemoryRecord` model contains:
|
||||
|
||||
- `id`, such as `mem-0001`;
|
||||
- the timestamp, session, and sequence of the original decision;
|
||||
- `type`;
|
||||
- `subject`;
|
||||
- `detail`;
|
||||
- `rationale`;
|
||||
- `question_context`;
|
||||
- `tables` e `concepts`.
|
||||
|
||||
For new Memory items, the type is always `concept_clarified` and `tables` starts empty. A table or column may appear in the explanation as a technical mapping, but it cannot be the Memory item's standalone concept.
|
||||
|
||||
Valid example:
|
||||
|
||||
> Ablation means a procedure with `ablazione_transcatetere = TRUE`, counted with `COUNT(DISTINCT cod_paz)` by year.
|
||||
|
||||
Invalid examples:
|
||||
|
||||
- `fact_cardioversione` as approved Memory;
|
||||
- `dim_time` as rejected Memory;
|
||||
- an "include this table" decision saved for future questions.
|
||||
|
||||
The workflow skill also documents this rule in [harness/.pi/skills/tht-sessione/SKILL.md](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/harness/.pi/skills/tht-sessione/SKILL.md#L217).
|
||||
|
||||
## Promotion at the end of the session: F8
|
||||
|
||||
At the end of the workflow, the `reviewer_memory_promote` gate runs a deterministic preview:
|
||||
CLI configuration remains a per-command option. Commands emit pure JSON:
|
||||
|
||||
```text
|
||||
tht memory promote --session <id> --preview --json
|
||||
tht memory list --filters '{"family":"sql_rule","page":1}' -c <runtime.yaml>
|
||||
tht memory show <card-id> -c <runtime.yaml>
|
||||
tht memory create --data <card.json> -c <runtime.yaml>
|
||||
tht memory update <card-id> --data <card.json> -c <runtime.yaml>
|
||||
tht memory delete <card-id> --yes -c <runtime.yaml>
|
||||
tht memory pending -c <runtime.yaml>
|
||||
tht memory retry <card-id> -c <runtime.yaml>
|
||||
tht memory index -c <runtime.yaml>
|
||||
tht memory search "<question>" --session <id> --json -c <runtime.yaml>
|
||||
tht memory search "<question>" --filters '{"table":"orders","column":"id","scope":"Sales"}' --json -c <runtime.yaml>
|
||||
tht memory solved-search "<question>" --json -c <runtime.yaml>
|
||||
tht memory rules "<question>" --session <id> --json -c <runtime.yaml>
|
||||
tht memory propose --session <id> --data <proposals.json> -c <runtime.yaml>
|
||||
tht memory summary --session <id> --json -c <runtime.yaml>
|
||||
tht memory solved-index <session-id> --json -c <runtime.yaml>
|
||||
```
|
||||
|
||||
The preview:
|
||||
`memory index` rebuilds both domain and solved-question projections.
|
||||
`solved-index` only retries an existing authoritative source receipt.
|
||||
The reviewer gate owns `memory review-apply`; Pi must not call it directly.
|
||||
Migration `003_review_receipts.sql` adds durable review and physical-cleanup receipts.
|
||||
The backend uses `memory admin --workspace <id> -c <protected-request.json>` so
|
||||
administration does not materialize session or DWH configuration.
|
||||
|
||||
1. reads the session's effective decisions;
|
||||
2. considers only `concept_clarified`;
|
||||
3. discards decisions already promoted;
|
||||
4. discards sequences already declined in F8;
|
||||
5. deduplicates equivalent content;
|
||||
6. proposes at most five candidates.
|
||||
## Installation and storage
|
||||
|
||||
The code applies filtering and deduplication in
|
||||
[harness/tht/memory/core.py](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/harness/tht/memory/core.py); the gate applies an additional defensive filter in
|
||||
[harness/.pi/extensions/gate/memory/index.js](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/harness/.pi/extensions/gate/memory/index.js).
|
||||
Memory shares the installation's existing PostgreSQL service, using its own
|
||||
`thoth_memory` schema and versioned harness migration pack. The existing
|
||||
`catalog-migrate` preparation service runs Catalog migrations followed by
|
||||
`python -m tht.memory.migrate`. The core image includes both migration runners.
|
||||
Ordinary API requests never migrate the schema.
|
||||
|
||||
The reviewer sees one preselected checklist. For each candidate:
|
||||
Connection credentials come from the existing generated `THT_CATALOG_DB_HOST`,
|
||||
`THT_CATALOG_DB_PORT`, `THT_CATALOG_DB_NAME`, `THT_CATALOG_RUNTIME_USER` and
|
||||
`THT_CATALOG_RUNTIME_PASSWORD_FILE`. Migration uses the corresponding migrator
|
||||
user/password file. Direct URL environments can use `THT_CATALOG_DATABASE_URL`
|
||||
(runtime) and `THT_CATALOG_MIGRATOR_DATABASE_URL`; the harness also accepts
|
||||
`THT_CATALOG_RUNTIME_DATABASE_URL`. Credentials are not authored in workspace YAML.
|
||||
|
||||
- selected: `tht memory save-one` runs, followed by a `memory_promoted` record;
|
||||
- deselected: records `memory_promotion_declined`;
|
||||
- no candidates: F8 closes automatically.
|
||||
The migration grants the installation login membership in the restricted
|
||||
`thoth_memory_runtime` role. Every repository transaction sets that role and a
|
||||
workspace context. Forced row-level policies isolate cards, links, dependencies
|
||||
and projection operations. This role has Memory DML and migration-status read
|
||||
access, with no runtime DDL privilege. There are no cascading foreign keys to
|
||||
Catalog or Evidence. A missing or incompatible schema returns a clear operational
|
||||
error. Preparation is repeatable and checks migration checksums.
|
||||
|
||||
The `memory_promoted` marker uses `detail: seq:N`, a reference to the original `concept_clarified` decision. The flow is in [harness/.pi/extensions/tht-gate.js](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/harness/.pi/extensions/tht-gate.js#L1691).
|
||||
|
||||
Promotion does not run in F2, and the model cannot invent F8 candidates. Direct promotion commands are also protected by the anti-bypass gate.
|
||||
|
||||
## Global persistence
|
||||
|
||||
### JSONL registry
|
||||
|
||||
The current registry is:
|
||||
|
||||
```text
|
||||
<artifacts>/memory/registry.jsonl
|
||||
```
|
||||
|
||||
The registry is written through a temporary file and `os.replace`, so replacement is atomic. Promotion is idempotent on the `session_id + decision_seq` pair: the same decision from the same session cannot create two global records.
|
||||
|
||||
### Qdrant
|
||||
|
||||
After promotion, `save-one` builds one `VectorRecord` and sends it to the Qdrant index. The indexed text includes:
|
||||
|
||||
- type and subject;
|
||||
- detail;
|
||||
- rationale;
|
||||
- question context;
|
||||
- any concepts and mappings.
|
||||
|
||||
The vector record uses the ID `memory:mem-XXXX`. Its metadata stores `subject`, `detail`, `rationale`, `tables`, `concepts`, and the `kind` discriminator. The content's SHA-256 hash prevents embedding and upsert work when the text has not changed.
|
||||
|
||||
This behavior is implemented in
|
||||
[harness/tht/memory/core.py](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/harness/tht/memory/core.py).
|
||||
|
||||
### Current canonical source
|
||||
|
||||
The JSONL registry remains the application's canonical source, while Qdrant is a derived but persistent index. The workflow does not record decisions directly in the vector database. It uses Qdrant as a searchable projection of the registry and effective ledger.
|
||||
|
||||
## Reuse in F2
|
||||
|
||||
In a future session, F2 runs:
|
||||
|
||||
```text
|
||||
tht memory search "<question>" --session <id> --json
|
||||
```
|
||||
|
||||
The command:
|
||||
|
||||
1. creates an embedding for the question;
|
||||
2. searches the vector store for records with `kind=memory` only;
|
||||
3. resolves each hit in the JSONL registry through its `ref`;
|
||||
4. discards records missing from the registry;
|
||||
5. discards every type other than `concept_clarified`;
|
||||
6. excludes Memory already decided in the current session;
|
||||
7. returns results ordered by similarity.
|
||||
|
||||
Memory is never applied automatically. The model must present it in one `reviewer_decide` choice:
|
||||
|
||||
- a selected Memory item is recorded as a new `concept_clarified` in the current session;
|
||||
- the rationale must cite the original `mem-XXXX` ID;
|
||||
- a deselected Memory item means "do not apply it now", not "delete it globally".
|
||||
|
||||
If F2 is reopened, a deselected Memory item can be proposed again. `memory_rejected` remains supported for legacy decisions and sessions, but it is not the normal behavior for current F2 deselection.
|
||||
|
||||
## Effective ledger, rollback, and reopening
|
||||
|
||||
The ledger is append-only. Reopening and withdrawing decisions do not delete earlier rows, but they change which decisions are effective.
|
||||
|
||||
Memory helpers use `effective_decisions()` to:
|
||||
|
||||
- exclude withdrawn decisions;
|
||||
- ignore decisions from phases that became stale after a rollback;
|
||||
- prevent promotion of clarifications that are no longer valid.
|
||||
|
||||
The effective view is defined in [harness/tht/phase.py](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/harness/tht/phase.py#L82).
|
||||
|
||||
## Display in the session summary
|
||||
|
||||
The "Memories" section of the summary is projected at runtime from the session ledger; it is not a direct copy of the global registry.
|
||||
|
||||
The projection:
|
||||
|
||||
- shows approved Memory first;
|
||||
- then shows declined Memory;
|
||||
- resolves `seq:N` to the original `concept_clarified`;
|
||||
- hides markers whose original record is not `concept_clarified`;
|
||||
- hides subjects that match schema-linking tables;
|
||||
- hides standalone subjects shaped like `fact_*` or `dim_*`;
|
||||
- keeps table and field references when they are part of the conceptual explanation.
|
||||
|
||||
The logic is in [harness/tht/session/store.py](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/harness/tht/session/store.py#L237). The frontend renders a structured list and treats `subject`, `detail`, and `rationale` as Markdown instead of showing raw Markdown.
|
||||
|
||||
The transformation happens on read. Historical sessions use the current layout and filters without rewriting their original artifacts.
|
||||
|
||||
## Enforced invariants
|
||||
|
||||
The protections are distributed across several boundaries:
|
||||
|
||||
1. `REUSABLE_TYPES` in the Python core;
|
||||
2. the `memory search` command filter;
|
||||
3. the F8 preview filter;
|
||||
4. filtering and deduplication in the Pi gate;
|
||||
5. exclusion of table Memory from the UI projection.
|
||||
|
||||
This prevents one prompt or component change from reintroducing tables as Memory.
|
||||
|
||||
## Remaining limits and risks
|
||||
|
||||
### The registry and index are not one transaction
|
||||
|
||||
Saving broadly follows this sequence:
|
||||
|
||||
```text
|
||||
JSONL registry → Qdrant → memory_promoted marker in the ledger
|
||||
```
|
||||
|
||||
If Qdrant is unavailable, the registry can contain Memory that is not yet searchable. The command reports that reindexing is required.
|
||||
|
||||
If the ledger marker fails after the vector database save, the Memory item can exist globally without a complete session audit. The gate returns a manual recovery command.
|
||||
|
||||
### Five-candidate limit
|
||||
|
||||
F8 proposes at most five Memory items. If a session produces more than five valid concepts, the extra items are not shown and the session can be finalized without promoting them.
|
||||
|
||||
### Deduplication is not global
|
||||
|
||||
Deduplication prevents duplicates within one proposal, and idempotency prevents the same decision from being promoted twice. There is no global merge of semantically similar Memory items from different sessions.
|
||||
|
||||
### Old physical records
|
||||
|
||||
Old `table_promoted` or `table_excluded` records may still exist in historical artifacts or indexes. The current code makes them unusable by filtering by type and does not show them in session projections. Physically removing them from the vector database remains a separate cleanup task.
|
||||
|
||||
## Final assessment
|
||||
|
||||
The current implementation matches the functional requirement: Memory is reusable conceptual knowledge, not a schema-linking choice.
|
||||
|
||||
The strongest part is the multilayer protection of the `concept_clarified` type. The main technical debt is the coexistence of the JSONL registry and Qdrant, with no single transaction spanning the global archive, semantic index, and session ledger.
|
||||
See [the M1 specification](plans/2026-09-08-memory-m1-spec.md) and
|
||||
[ADR 0018](adr/0018-use-postgres-for-memory-and-qdrant-for-retrieval.md) for the
|
||||
approved scope and acceptance boundaries.
|
||||
See [M2 implementation and validation](plans/2026-09-09-memory-m2-validation.md)
|
||||
for the retrieval checks and real embedding test command.
|
||||
|
||||
+39
-3
@@ -4,10 +4,30 @@ This guide is for a reviewer using a configured ThothII installation. Installati
|
||||
publication, preprocessing, and database administration are separate paths; links to them are at
|
||||
the end of this page.
|
||||
|
||||
## Standalone or inside Omics
|
||||
|
||||
In **full** mode, ThothII has its own red header. Sign in using the installation's
|
||||
local account or the configured identity provider. The header lets you select
|
||||
English/Italian, light/dark, and fullscreen; Esc exits fullscreen. Open the user
|
||||
name menu to log out of ThothII. OIDC logout does not necessarily log out other
|
||||
applications using the same provider.
|
||||
|
||||
In **embedded** mode, first sign in to Omics and choose **Datamart Builder** in
|
||||
its left menu. ThothII opens with that authenticated identity: there is no second
|
||||
login or duplicate header. Use Omics's language, theme, fullscreen and logout
|
||||
controls. If portal access expires, return to Omics, sign in and reopen the page.
|
||||
|
||||
The Mac starts in English unless the browser remembers another choice. Changing
|
||||
the UI language affects labels, not saved domain content. A new session takes
|
||||
the selected language for the model's questions and reviewer choices; an existing
|
||||
session retains its saved language when resumed. Omics's language change reloads
|
||||
the page: confirm or cancel any unsaved-work warning. Reopening the saved session
|
||||
selection shows documents; it does not automatically restart generation.
|
||||
|
||||
## Before creating a session
|
||||
|
||||
An administrator must have selected a workspace and configured the installation-wide provider,
|
||||
model, and thinking settings. The New session form deliberately asks only for the question.
|
||||
model, and thinking settings. The new-question form deliberately asks only for the question.
|
||||
|
||||
The workspace is a pinned Git revision. A subsequent workspace update cannot alter a session
|
||||
already created from an earlier revision. If a workspace cannot reach its configured runtime DWH,
|
||||
@@ -15,7 +35,9 @@ new sessions are refused before any session state is written.
|
||||
|
||||
## Create and review a session
|
||||
|
||||
1. Sign in and select **New session**.
|
||||
1. Sign in and select **Session** (**Sessione** in Italian). If a session is already
|
||||
open and unfinished, this returns to it without restarting it. Otherwise it
|
||||
opens a new question; no session is created until you submit that question.
|
||||
2. Enter a precise business question, including the relevant time period and desired output. For
|
||||
example: “List patients discharged in the last 30 days, with ward and discharge date.”
|
||||
3. Review each gate and make the decision requested by the widget. A choice with a decision payload
|
||||
@@ -37,6 +59,19 @@ The workflow phases are fixed:
|
||||
|
||||
## Resume, archive, and the meaning of saved state
|
||||
|
||||
In Administration, the dot beside **Workspace** is green when readiness is
|
||||
confirmed and red otherwise. Hover the button for the exact state; assistive
|
||||
technology receives the same description. Select Workspace to inspect preparation.
|
||||
|
||||
The session sidebar has two accordion sections: **Active sessions** and
|
||||
**Archive**, both initially closed. Only their headers appear below the scope tabs.
|
||||
Inside each nonempty list, **Select all** selects only that list; its delete action
|
||||
also applies only to the selected sessions in that list. The other list's selection
|
||||
is preserved. Opening a section closes the other; clicking the open section closes
|
||||
it too. Empty lists show only "No sessions yet." Long lists scroll inside
|
||||
their own panels. Here active means not archived, not necessarily a running model
|
||||
process. Existing groups and session actions remain inside those sections.
|
||||
|
||||
The sidebar lists sessions and their current lifecycle. Resuming returns to the last incomplete
|
||||
phase. A finalized or archived session cannot be resumed.
|
||||
|
||||
@@ -54,4 +89,5 @@ happen from the workflow’s point of view.
|
||||
- To author material the workflow can retrieve, use [Evidence](evidence.md). A proposal from a
|
||||
session does not become Evidence automatically: a curator must review and publish it in Git.
|
||||
- For login and access recovery, use [local authentication](install/authentication-local.md) or
|
||||
[OIDC authentication](install/authentication-oidc.md).
|
||||
[OIDC authentication](install/authentication-oidc.md) for full, or contact the
|
||||
portal administrator for [embedded/upstream access](install/authentication-upstream.md).
|
||||
|
||||
@@ -8,6 +8,12 @@ Start with the path that matches the work you need to do:
|
||||
| I need to… | Start here |
|
||||
| --- | --- |
|
||||
| Install or operate one instance | [Install and first start](install/first-start.md) |
|
||||
| Clone and manually install on macOS, Windows, or Linux | [Italian procedure](install/standalone-manual-it.md) · [English procedure](install/standalone-manual-en.md) |
|
||||
| Upgrade the server and integrate Omics Portal | [Codex server handoff](operations/server-codex-handoff.md) |
|
||||
| Choose full/embedded and configure the shell | [Shell and localization](operations/shell-and-localization.md) |
|
||||
| Reuse an authenticated server portal without a second login | [Upstream authentication](install/authentication-upstream.md) |
|
||||
| Understand how the two renderings share the same application | [Rendering architecture](architecture/application-shell.md) |
|
||||
| Validate login, logout and portal integration before release | [Acceptance matrix](testing/authentication-manual-acceptance.md) |
|
||||
| Add, update, or prepare a workspace | [Workspace operations](operations/workspaces.md) |
|
||||
| Ask a question and review the SQL workflow | [User guide](guida-utente.md) |
|
||||
| Configure and refresh an authoritative database catalog | [Database management](operations/database-management.md) |
|
||||
|
||||
@@ -1,6 +1,11 @@
|
||||
# Local authentication
|
||||
|
||||
Use local mode for a standalone PC or Mac. Configure it through `tht`; passwords are entered at an
|
||||
Use local mode for a standalone PC or Mac, with `shell.mode: full` and
|
||||
`shell.defaultLocale: en` in the installation descriptor. Presentation and
|
||||
authentication are independent: selecting full does not create accounts. Omics
|
||||
embedded instead uses the [upstream guide](authentication-upstream.md), not local users.
|
||||
|
||||
Configure local authentication through `tht`; passwords are entered at an
|
||||
echo-free prompt or read from a protected `--password-file`, never from a command argument.
|
||||
|
||||
## Bootstrap
|
||||
@@ -56,6 +61,11 @@ machine use; JSON output is pristine on stdout.
|
||||
|
||||
## Session behavior and recovery
|
||||
|
||||
Full shows its own login form and, after login, the verified display name in its
|
||||
header. The name menu contains Log out. This sends a CSRF-protected request to
|
||||
`/api/auth/logout`, revokes the session and returns to login. Language/theme
|
||||
preferences may remain in the browser; they are not credentials.
|
||||
|
||||
An ordinary login expires after 2 hours idle or 12 hours absolute. Selecting **Remember me** makes
|
||||
the cookie persistent and changes the limits to 7 days idle or 30 days absolute. Remembered
|
||||
sessions survive a browser and backend restart, but not a user revision change, configuration
|
||||
|
||||
@@ -1,10 +1,20 @@
|
||||
# Generic OIDC authentication
|
||||
|
||||
Use this guide for **ThothII's own login**, normally `shell.mode: full` on an
|
||||
autonomous server. It is not the integration procedure for an already logged-in
|
||||
Omics user. That deployment uses [embedded/upstream](authentication-upstream.md),
|
||||
even when Omics's identity provider is Authentik.
|
||||
|
||||
OIDC mode supports a standards-based provider. The browser flow is generic: Authorization Code,
|
||||
PKCE S256, state, nonce, issuer/signature/audience/expiry validation, and the fixed callback
|
||||
`<publicUrl>/api/auth/oidc/callback`. The browser and API must use the same origin; configure the
|
||||
reverse proxy to preserve that public origin and callback path.
|
||||
|
||||
`publicUrl` is the public origin, without an application subpath. The current
|
||||
full OIDC browser entry and callback use `/api/auth/oidc/login` and
|
||||
`/api/auth/oidc/callback`; arbitrary prefixed OIDC hosting is not implemented by
|
||||
selecting a different `backendBaseUrl`.
|
||||
|
||||
Configure the installation with `tht`:
|
||||
|
||||
```sh
|
||||
@@ -18,6 +28,9 @@ The OIDC client secret is supplied through the protected secret bundle under the
|
||||
`THT_OIDC_CLIENT_SECRET`; it is never written into `auth.yaml`. The default scopes are exactly
|
||||
`openid`, `profile`, and `email`.
|
||||
|
||||
Keep `AUTH_MODE` unset when using this file. A simultaneously mounted local/OIDC
|
||||
configuration and `AUTH_MODE=upstream` is an error, not a fallback chain.
|
||||
|
||||
The non-secret OIDC configuration has this exact shape (replace angle-bracket placeholders with
|
||||
operator values):
|
||||
|
||||
@@ -92,3 +105,13 @@ relevant diagnostic surface.
|
||||
|
||||
The complete closed diagnostic-code union and exact role-to-permission expansion are in the
|
||||
[authentication architecture](../architecture/authentication.md).
|
||||
|
||||
## Browser login and logout
|
||||
|
||||
ThothII redirects the browser to the provider and creates its own opaque session
|
||||
after validating the callback. An existing provider SSO session may avoid another
|
||||
password prompt, but this remains a distinct ThothII login/session, unlike Omics
|
||||
upstream. Full's name menu logs out of ThothII only. It does not revoke the
|
||||
provider session or log out other applications, so a subsequent login can return
|
||||
immediately through SSO. No provider token is placed in the UI adapter or browser
|
||||
storage. See the [manual acceptance matrix](../testing/authentication-manual-acceptance.md).
|
||||
|
||||
@@ -0,0 +1,186 @@
|
||||
# Autenticazione tramite portale e proxy fidato
|
||||
|
||||
Questa è la modalità **upstream** usata dall'integrazione Omics. Non è il login
|
||||
OIDC diretto di ThothII: l'utente accede a Omics come già fa, poi sceglie
|
||||
Datamart Builder e trova ThothII già autenticato. Non deve essere creato un utente
|
||||
locale ThothII né effettuato un secondo scambio OIDC dall'applicazione embedded.
|
||||
|
||||
## Il confine di fiducia
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
actor U as Utente già autenticato
|
||||
participant N as Nginx Omics
|
||||
participant D as Django Omics
|
||||
participant T as Core ThothII upstream
|
||||
U->>D: Apri Datamart Builder
|
||||
D-->>U: Pagina autorizzata con mount React
|
||||
U->>N: GET /datamart-builder/api/me (cookie Omics)
|
||||
N->>D: Subrequest interna /datamart-builder/api-auth
|
||||
D-->>N: 200 + identità verificata, oppure 403
|
||||
N->>T: GET /me + intestazioni normalizzate (solo se autorizzato)
|
||||
T-->>U: Identità e permessi applicativi, oppure rifiuto
|
||||
```
|
||||
|
||||
L'header e l'adapter JavaScript non autenticano nessuno. Il backend accetta una
|
||||
richiesta upstream solo con un'identità valida ricevuta da un percorso di rete
|
||||
fidato. Gli header non sono firmati da ThothII: la protezione è il proxy che
|
||||
verifica la sessione e sovrascrive l'identità, insieme all'isolamento del core.
|
||||
Un core upstream direttamente raggiungibile da client non fidati è una falla,
|
||||
non una modalità alternativa di accesso.
|
||||
|
||||
## Configurazione del core
|
||||
|
||||
Per Omics il descrittore pubblico deve contenere:
|
||||
|
||||
```yaml
|
||||
shell:
|
||||
mode: embedded
|
||||
defaultLocale: en
|
||||
adapter: omics-portal
|
||||
```
|
||||
|
||||
Separatamente, il **processo core** deve ricevere `AUTH_MODE=upstream`. Definirlo
|
||||
nell'override Compose approvato e incluso nell'installazione; una variabile nel
|
||||
file di interpolazione `.env` non viene passata automaticamente al container:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
core:
|
||||
environment:
|
||||
AUTH_MODE: upstream
|
||||
```
|
||||
|
||||
È solo il frammento di selezione auth, non un file Compose completo né una
|
||||
configurazione di rete sufficiente. Non aggiunge porte pubbliche.
|
||||
|
||||
Condizioni obbligatorie:
|
||||
|
||||
1. Nessun `auth.yaml` local/OIDC deve essere effettivamente montato al percorso
|
||||
letto dal core (default `/run/thothii-auth/auth.yaml`). Se è presente insieme
|
||||
ad `AUTH_MODE`, l'avvio fallisce. Non impostare `AUTH_MODE=local` o `oidc`:
|
||||
questi due modi si selezionano dal file, non da quella variabile.
|
||||
2. Non configurare `authentication.runtimeProjection` per questo percorso: è la
|
||||
proiezione delle configurazioni cookie local/OIDC, non l'identità Omics.
|
||||
Nemmeno `THT_AUTH_RUNTIME_PROJECTION_ROOT` deve attivarla nel core.
|
||||
3. Il descrittore e Compose base continuano a richiedere `authentication.configDirectory`
|
||||
e `THT_AUTH_CONFIG_ROOT` coerenti. Per una nuova installazione upstream usare
|
||||
una directory dedicata senza `auth.yaml`, non cancellare la configurazione di
|
||||
un'installazione esistente. I cambi di modalità richiedono un piano separato.
|
||||
4. Non esiste `tht auth configure --mode upstream`: il CLI configura gli utenti
|
||||
locali o l'OIDC diretto. Conservare il percorso proxy già operativo per Omics.
|
||||
5. `profile: server`, storage delle sessioni e `THOTH_PUBLIC_EXPOSURE` hanno propri
|
||||
vincoli, che rimangono attivi. La shell embedded non li soddisfa automaticamente.
|
||||
|
||||
Il sorgente considera upstream un percorso di compatibilità con il proxy; è
|
||||
quello usato dall'integrazione Omics corrente. `none` e `mock` sono per sviluppo/test,
|
||||
non soluzioni a errori di configurazione in produzione.
|
||||
|
||||
## Intestazioni richieste all'ingresso del core
|
||||
|
||||
| Header | Regola ThothII | Valore Omics |
|
||||
| --- | --- | --- |
|
||||
| `X-Thoth-Principal-Issuer` | Stringa stabile, obbligatoria | `portal` |
|
||||
| `X-Thoth-Principal-Subject` | ID stabile dell'utente, obbligatorio | `str(user.pk)` Django |
|
||||
| `X-Thoth-Principal-Display-Name` | Facoltativo, se presente non vuoto | Nome completo o username |
|
||||
| `X-Thoth-Is-Admin` | Obbligatorio: `0`, `1`, `false` o `true` | Risultato di `is_authentik_admin(user)` |
|
||||
|
||||
Le stringhe sono ripulite degli spazi esterni, devono avere al massimo 512
|
||||
caratteri e non contenere caratteri di controllo. Header mancanti o invalidi
|
||||
producono 401. `false`/`0` assegna il ruolo `user`; `true`/`1` assegna `user` e
|
||||
`admin`. Il core espande i permessi dal proprio catalogo, non da un array inviato
|
||||
dal browser. `/me` richiede `session.use`.
|
||||
|
||||
La coppia `(issuer, subject)` identifica il proprietario delle sessioni.
|
||||
Non sostituire il subject con un nome visualizzato o un'email modificabile; non
|
||||
cambiare issuer/subject di utenti esistenti per correggere un problema grafico.
|
||||
Passare da identità `portal` a identità OIDC diretta non migra la proprietà dei dati.
|
||||
|
||||
## Omics: percorsi e componenti esatti
|
||||
|
||||
| Percorso | Destinazione e funzione |
|
||||
| --- | --- |
|
||||
| `/kokoro/datamart-builder/` | Pagina Django con `datamart_builder.access` |
|
||||
| `/datamart-builder/config.js` | Config pubblico del frontend, senza cache |
|
||||
| `/datamart-builder/assets/…` | Asset frontend risolti dal manifest Vite |
|
||||
| `/datamart-builder/api/…` | Nginx con `auth_request`, poi core senza il prefisso |
|
||||
| `/_thothii_auth` | Location Nginx interna, non un login pubblico |
|
||||
| `/datamart-builder/api-auth` | Django verifica sessione Omics e capability |
|
||||
|
||||
Nel repository Omics:
|
||||
|
||||
- `kokoro/datamart_catalog_views.py`: `DatamartBuilderView` e
|
||||
`datamart_builder_api_auth`; la verifica API risponde 200 o 403, anche 403
|
||||
quando la sessione è assente/scaduta. Non trasforma l'API in una pagina di login.
|
||||
- `nginx/nginx.conf`: API direttamente a `thothii-core:8787`, config e asset a
|
||||
`thothii-frontend:8080`; verificare alias e reti Docker effettivi sul server.
|
||||
- `templates/kokoro/datamart_builder.html`: mount, config e override del prefisso.
|
||||
- `kokoro/templatetags/vite.py`: manifest da
|
||||
`http://thothii-frontend:8080/.vite/manifest.json`, cache Django di 30 secondi.
|
||||
|
||||
Nginx usa il cookie Omics nella subrequest a Django. Sulle richieste al core
|
||||
sovrascrive i quattro header con i risultati della verifica e rimuove
|
||||
`Cookie`, `Authorization` e `X-Authenticated-User`. Nessuna password o token del
|
||||
portale deve essere copiato nel config pubblico, nello snapshot adapter o in Web Storage.
|
||||
`GET /datamart-builder/api/me` restituisce l'identità e i permessi; in upstream
|
||||
`session` e `csrfToken` sono `null`: non viene creata una sessione-cookie ThothII.
|
||||
|
||||
## Non confondere i due percorsi proxy
|
||||
|
||||
L'esempio generico `deploy/nginx-authenticated-proxy.conf.example` usa **due hop**:
|
||||
proxy host → frontend Nginx ThothII → core. Sul tratto privato verso il frontend
|
||||
trasporta `X-Thoth-Trusted-Principal-*` e `X-Thoth-Trusted-Is-Admin`; il frontend
|
||||
li converte nei quattro header del core e li elimina prima dell'inoltro.
|
||||
|
||||
Omics usa invece **Nginx Omics → core direttamente** per le API e invia gli header
|
||||
normalizzati senza `Trusted`. Non incollare l'esempio a due hop in questa location:
|
||||
la famiglia di header sbagliata produce 401. In entrambi i casi i valori devono
|
||||
venire dalla verifica server, mai dagli header del client. Il tratto privato del
|
||||
percorso generico deve essere inaccessibile ai client non fidati.
|
||||
|
||||
## Origine delle richieste e stream
|
||||
|
||||
Browser e API devono restare sullo stesso origin. Il frontend accetta `/api` o un
|
||||
prefisso same-origin come `/datamart-builder/api`, non un URL `http://core:8787`.
|
||||
In upstream le scritture con `Origin` sono confrontate con protocollo e Host
|
||||
percepiti dal core; non usano il token CSRF della sessione ThothII local/OIDC.
|
||||
Le richieste senza Origin hanno il trattamento non-browser: l'autenticazione del
|
||||
proxy rimane indispensabile anche per esse.
|
||||
|
||||
Nel Nginx Omics esaminato il TLS termina a monte e una mappa **esatta** converte
|
||||
`https://aritmolab.policlinicosandonato.it` in
|
||||
`http://aritmolab.policlinicosandonato.it` per il confronto interno. Le altre origini
|
||||
rimangono invariate e devono essere negate quando non coincidono. È una scelta
|
||||
specifica della topologia corrente, non un modello da estendere con wildcard,
|
||||
cancellazione di Origin o riscrittura incondizionata. Verificare Host/protocollo
|
||||
al core e i dinieghi cross-origin nella topologia realmente rilasciata.
|
||||
|
||||
La location API disabilita buffering/cache per SSE e mantiene timeout lunghi.
|
||||
`auth_request` verifica ogni nuova richiesta, ma non interrompe istantaneamente
|
||||
uno stream già aperto quando il portale revoca l'utente. ThothII ricontrolla `/me`
|
||||
al ritorno alla pagina e alla riconnessione degli eventi; non promettere revoca
|
||||
istantanea fra tutte le schede.
|
||||
|
||||
## Logout, rientro e diagnosi
|
||||
|
||||
In embedded logout e successivo login sono di Omics. ThothII non chiama
|
||||
`/auth/logout`, non cancella il cookie Django e non apre un suo login.
|
||||
Il rifiuto 401/403 di `/me` rimuove lo stato protetto e richiede il rientro dal
|
||||
portale. Un 403 su una singola operazione non equivale al logout dell'applicazione.
|
||||
|
||||
| Sintomo | Controllo mirato |
|
||||
| --- | --- |
|
||||
| Secondo header | Config servito: deve essere embedded, non full |
|
||||
| Nessuna UI e errore preferenze | Selettore Omics `data-lang` e `html data-bs-theme` |
|
||||
| `/me` 401 dal core | Header obbligatori, famiglia Trusted/normalizzata, percorso proxy |
|
||||
| `/me` 403 dal proxy | Sessione Omics e capability `datamart_builder.access` |
|
||||
| `/me` funziona ma POST 403 | Distinguere permesso operativo da mismatch Origin/Host/protocollo |
|
||||
| 502 o asset assenti | Alias/rete Docker e manifest Vite; attesa cache manifest 30 s |
|
||||
| Avvio core rifiutato | Coesistenza di `auth.yaml` o runtime projection con `AUTH_MODE` |
|
||||
| Logout full seguito da rientro IdP immediato | Il logout ThothII non è logout globale OIDC |
|
||||
|
||||
Non raccogliere cookie, token, segreti o dump completi delle configurazioni nei
|
||||
report. Registrare codici HTTP, nomi dei percorsi, revisioni e risultati dei test.
|
||||
|
||||
Consegna e rilascio: [procedura Omics](../operations/shell-and-localization.md#verifica-prima-del-deploy-server).
|
||||
Collaudo obbligatorio: [matrice di accettazione](../testing/authentication-manual-acceptance.md).
|
||||
@@ -1,7 +1,14 @@
|
||||
# Authentik provider configuration
|
||||
|
||||
ThothII uses generic OIDC in the browser. Authentik provides the identity provider and group
|
||||
catalog without adding a proprietary login flow.
|
||||
For **full with direct OIDC**, ThothII uses generic OIDC in the browser. Authentik
|
||||
provides the identity provider and group catalog without adding a proprietary flow.
|
||||
The provider/client/group setup below applies to that case only.
|
||||
|
||||
For **embedded in Omics**, retain Omics's existing Authentik authentication and
|
||||
configure ThothII as upstream. Omics verifies `datamart_builder.access` and
|
||||
administrator status and the proxy supplies the identity; no additional ThothII
|
||||
OIDC client, login or local user is required for that path. Follow the
|
||||
[portal integration guide](authentication-upstream.md).
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
|
||||
@@ -3,6 +3,9 @@
|
||||
# Replace every absolute placeholder. Select exactly one Git transport override.
|
||||
schemaVersion: 2
|
||||
profile: local
|
||||
shell:
|
||||
mode: full
|
||||
defaultLocale: en
|
||||
projectDirectory: "/absolute/path/to/ThothII"
|
||||
envFile: "/absolute/path/to/ThothII/deploy/env/local.env"
|
||||
workspaceRepository:
|
||||
|
||||
@@ -3,6 +3,11 @@
|
||||
# Replace every absolute placeholder. Select exactly one Git transport override.
|
||||
schemaVersion: 2
|
||||
profile: server
|
||||
# Standalone server with protected direct OIDC auth, not the Omics upstream path.
|
||||
# For Omics use authentication-upstream.md: embedded, no auth runtime projection.
|
||||
shell:
|
||||
mode: full
|
||||
defaultLocale: en
|
||||
projectDirectory: "/absolute/path/to/ThothII"
|
||||
envFile: "/absolute/path/to/thothii-server-operator/server.env"
|
||||
workspaceRepository:
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
# Install and first start
|
||||
|
||||
For the ordered clone-to-start procedure on Mac, Windows WSL2 and Linux, use the
|
||||
[Italian manual guide](standalone-manual-it.md) or [English manual guide](standalone-manual-en.md),
|
||||
including protected credentials and the explicit initial catalog migration.
|
||||
|
||||
This is the supported local installation path. It creates an installation-local configuration and
|
||||
starts the Compose stack; it does not create a workspace repository or a database catalog entry.
|
||||
|
||||
@@ -24,7 +28,7 @@ version control, a URL, or a command line.
|
||||
From the repository root, start the interactive setup and select the local profile:
|
||||
|
||||
```sh
|
||||
tht setup --profile local
|
||||
tht setup --profile local --shell-mode full --shell-default-locale en
|
||||
```
|
||||
|
||||
It writes the selected non-secret descriptor below `deploy/<installation-id>/`, the associated
|
||||
@@ -32,6 +36,18 @@ operator env file, and can create protected secret templates. Keep the descripto
|
||||
to commands as `--installation /absolute/path/thothii-installation.yaml` when more than one
|
||||
installation can be discovered.
|
||||
|
||||
The explicit shell options are important: compatibility defaults without them
|
||||
are embedded/en/omics-portal, which expects an Omics document. The Mac's standalone
|
||||
installation must use full, with English as its initial locale. Existing browser
|
||||
language preferences take precedence over that initial value. Full does not
|
||||
configure authentication; setup separately bootstraps local login.
|
||||
|
||||
For a server portal, use the [embedded/upstream procedure](authentication-upstream.md)
|
||||
instead of creating a second ThothII login. For a standalone server, use full
|
||||
with [direct OIDC](authentication-oidc.md). Shell mode does not follow `profile`
|
||||
automatically. See [configuration and regeneration](../operations/shell-and-localization.md)
|
||||
before modifying an existing installation.
|
||||
|
||||
If the descriptor is prepared manually instead, begin with
|
||||
[`thothii-installation.local.yaml`](examples/thothii-installation.local.yaml), set mode `0600` or
|
||||
`0400`, and ensure `THT_INSTALLATION_CONFIG_SOURCE` in the selected env file points to that exact
|
||||
|
||||
@@ -0,0 +1,324 @@
|
||||
# Manual standalone installation
|
||||
|
||||
[Versione italiana](standalone-manual-it.md)
|
||||
|
||||
This is the verification procedure for preparing THothII as a standalone application in `full`
|
||||
mode on macOS, Windows, and Linux.
|
||||
|
||||
In this document, “standalone” means that the user does not need to install Node.js, Python or Pi
|
||||
on the host: the application services and local semantic
|
||||
services run through Docker. DWH and LLM providers remain external endpoints configured by the
|
||||
installation; this is not an offline package.
|
||||
|
||||
This path uses no graphical installer, native launcher, or Docker Hub image. It starts from a Gitea
|
||||
clone and uses explicit terminal commands. Publishing pre-built images is a later step.
|
||||
|
||||
## Verification matrix
|
||||
|
||||
| System | Recommended terminal | Runtime | Test architecture |
|
||||
| --- | --- | --- | --- |
|
||||
| macOS supported by the installed Docker Desktop version | Bash in Terminal | Docker Desktop | Apple Silicon (`arm64`) |
|
||||
| Windows 11 | Ubuntu inside WSL2 | Docker Desktop with WSL2 integration | x64 (`amd64`) |
|
||||
| Ubuntu Linux 22.04 or 24.04 | Bash | Docker Engine + Compose v2 | x64 (`amd64`) |
|
||||
|
||||
Intel macOS is excluded from the first verification campaign. ARM Linux may be tested when the
|
||||
machine’s Docker runtime reports `arm64`, but it is not part of the minimum matrix.
|
||||
|
||||
Documentation and Mac prerequisite checks have passed. Complete fresh installations on all three
|
||||
systems remain pending; this matrix describes the tests to perform, not completed certification.
|
||||
|
||||
## Before you start
|
||||
|
||||
You need:
|
||||
|
||||
- access to the THothII Gitea repository and the workspace Git repository;
|
||||
- Git;
|
||||
- Docker Desktop on macOS and Windows, or Docker Engine with the Compose v2 plugin on Linux;
|
||||
- Bash, `curl`, OpenSSL and `shasum` (Ubuntu package: `libdigest-sha-perl`);
|
||||
- enough disk space to build the images and download the embedding model;
|
||||
- the DWH and LLM endpoints, plus the credentials required by the installation.
|
||||
|
||||
On Linux, the current user must be able to run Docker. If the system requires `sudo`, add the user
|
||||
to the Docker group according to local policy and open a new session before continuing.
|
||||
|
||||
On Windows, run every command in this guide from Ubuntu under WSL2. In Docker Desktop, enable WSL2
|
||||
integration for that distribution. Clone the project inside the WSL2 Linux filesystem, for example
|
||||
under `~/src`, rather than under `/mnt/c`: this avoids slow builds and path/line-ending issues. Pi
|
||||
does not need to be installed on the host.
|
||||
|
||||
Check the runtime before or immediately after cloning:
|
||||
|
||||
```sh
|
||||
docker version
|
||||
docker compose version
|
||||
docker version --format '{{.Server.Arch}}'
|
||||
```
|
||||
|
||||
The last command must return `amd64`, `x86_64`, `arm64`, or `aarch64`.
|
||||
|
||||
## 1. Clone a project revision
|
||||
|
||||
Use the project repository on Gitea:
|
||||
|
||||
```sh
|
||||
mkdir -p "$HOME/src"
|
||||
cd "$HOME/src"
|
||||
git clone https://git.tylconsulting.it/mptyl/ThothII.git
|
||||
cd ThothII
|
||||
git rev-parse --short HEAD
|
||||
```
|
||||
|
||||
For an SSH clone, when the key is already authorized on Gitea:
|
||||
|
||||
```sh
|
||||
git clone git@git.tylconsulting.it:mptyl/ThothII.git
|
||||
```
|
||||
|
||||
Record the hash printed by `git rev-parse` for a repeatable test. In a later campaign, use the
|
||||
maintainer-approved revision/tag rather than implicitly following a mutable `main` branch.
|
||||
|
||||
## 2. Check prerequisites and install the operator command
|
||||
|
||||
From the clone root:
|
||||
|
||||
```sh
|
||||
bash scripts/check-standalone-prerequisites.sh
|
||||
export PATH="$HOME/.local/bin:$PATH"
|
||||
THT_INSTALL_DIRECTORY="$HOME/.local/bin" bash scripts/install-tht.sh
|
||||
tht version
|
||||
```
|
||||
|
||||
`install-tht.sh` bootstraps only the native `tht` operator command; it does not install a desktop
|
||||
version of THothII. It uses the repository’s Docker builder, installs the binary for the current
|
||||
terminal environment, and installs it in the user directory. Persist `$HOME/.local/bin` in your
|
||||
shell PATH for new terminals too. An existing `tht` in this directory will be updated.
|
||||
|
||||
On Windows, run these commands inside WSL2. The installed `tht` binary is the Linux binary inside
|
||||
WSL2; the application runtime remains Docker Desktop. Do not use `scripts/install-tht.ps1` as the
|
||||
primary path for this test.
|
||||
|
||||
## 3. Configure and start the local installation
|
||||
|
||||
Run the remaining blocks in one Bash session from the physical clone root (`pwd -P`).
|
||||
First create two distinct catalog passwords, preserving any existing files:
|
||||
|
||||
```bash
|
||||
umask 077
|
||||
mkdir -p deploy/local/secrets
|
||||
for name in catalog-runtime-password catalog-migrator-password; do
|
||||
target="deploy/local/secrets/$name"
|
||||
if [ ! -e "$target" ]; then
|
||||
(set -C; openssl rand -hex 32 > "$target") || exit 1
|
||||
fi
|
||||
done
|
||||
export THT_CATALOG_RUNTIME_PASSWORD_SOURCE="$(pwd -P)/deploy/local/secrets/catalog-runtime-password"
|
||||
export THT_CATALOG_MIGRATOR_PASSWORD_SOURCE="$(pwd -P)/deploy/local/secrets/catalog-migrator-password"
|
||||
```
|
||||
|
||||
Do not regenerate passwords for an initialized catalog. Configure without starting services:
|
||||
|
||||
```sh
|
||||
tht setup --profile local --shell-mode full --shell-default-locale en --configure-only
|
||||
```
|
||||
|
||||
Answer the prompts as follows:
|
||||
|
||||
| Prompt | Value or rule |
|
||||
| --- | --- |
|
||||
| Installation ID | `local`, unless one clone hosts multiple installations |
|
||||
| Deployment profile | `local` |
|
||||
| DWH API endpoint | An `http(s)` URL without user, password, query, or fragment; may be empty for a smoke-only test |
|
||||
| LLM API endpoint | An `http(s)` URL without credentials; may be empty for a smoke-only test |
|
||||
| Workspace repository URL | The workspace repository URL, not the THothII source clone |
|
||||
| Workspace branch | Normally `main` |
|
||||
| Workspace access | `ssh` with a deploy key, or `https` with a protected credential file |
|
||||
| File paths | Accept the default paths under `deploy/local/secrets/` for the first test |
|
||||
| Secret templates | Answer `yes` when protected files do not exist yet |
|
||||
| Authentication | Configure the local login required by the installation; never put passwords on a command line |
|
||||
|
||||
The generated configuration is local and ignored by Git:
|
||||
|
||||
```text
|
||||
deploy/local/thothii-installation.yaml
|
||||
deploy/local/operator.env
|
||||
deploy/local/auth/
|
||||
deploy/local/secrets/
|
||||
```
|
||||
|
||||
Edit secrets only in protected local files; never commit them. `deploy/env/local.env.example` is a tracked reference; the
|
||||
generated path `deploy/local/operator.env` is the active path for this installation.
|
||||
|
||||
### Complete protected files
|
||||
|
||||
If setup created blank templates, enter the values with a local editor:
|
||||
|
||||
```sh
|
||||
chmod 600 deploy/local/secrets/*
|
||||
"${EDITOR:-vi}" deploy/local/secrets/thothii.secrets
|
||||
```
|
||||
|
||||
The bundle must contain only `KEY=VALUE` lines for credentials actually used by `modelCatalog`. The
|
||||
allowed names and credential boundary are documented in the local file
|
||||
`deploy/secrets/README.md`. Do not put tokens in URLs, the YAML
|
||||
descriptor, the Git repository, or commands copied into the shell.
|
||||
|
||||
For SSH workspace access, also provide the private key and `known_hosts` file requested by setup.
|
||||
For HTTPS access, provide the Git credential file and any required CA. Both must remain protected
|
||||
and outside version control.
|
||||
|
||||
Before starting, complete these additional configuration steps:
|
||||
|
||||
1. Add `THT_CATALOG_RUNTIME_PASSWORD_SOURCE` and `THT_CATALOG_MIGRATOR_PASSWORD_SOURCE` to
|
||||
`deploy/local/operator.env`, with the same absolute paths exported above. Setup does not persist
|
||||
these two variables. Store paths, not passwords.
|
||||
2. Replace the descriptor's generic `modelCatalog` with the approved provider/model configuration.
|
||||
The generated defaults do not replicate the existing Mac. See [Pi/model configuration](../general/pi-configuration.md)
|
||||
and the local example `deploy/psd/thothii-installation.yaml.example`.
|
||||
3. Populate the keys referenced by `authentication.apiKeyEnv` in `thothii.secrets`. Providers using
|
||||
`pi_auth` need valid credentials at `PI_AUTH_FILE`; the `{}` template is not authentication.
|
||||
4. Complete the workspace Git files. SSH requires an authorized deploy key and verified known-hosts;
|
||||
HTTPS requires the credential file and CA bundle expected by the overlay. Blank templates cannot
|
||||
provide repository access.
|
||||
|
||||
After editing generated configuration, do not rerun setup: it rejects different existing content.
|
||||
Generate the projections and run the explicit migration below. Use `THT_GIT_ACCESS=https` if that
|
||||
was selected during setup. This block targets the fresh `local` descriptor with only the Git overlay;
|
||||
custom installations must include their extra descriptor overlays in the same order.
|
||||
|
||||
```bash
|
||||
INSTALLATION="$(pwd -P)/deploy/local/thothii-installation.yaml"
|
||||
tht --installation "$INSTALLATION" installation generate
|
||||
THT_PROJECT="thothii-$(printf '%s' "$INSTALLATION" | shasum -a 256 | cut -c 1-12)"
|
||||
THT_GIT_ACCESS=ssh
|
||||
compose=(
|
||||
docker compose --project-name "$THT_PROJECT" --project-directory "$(pwd -P)"
|
||||
--env-file "$(pwd -P)/deploy/local/operator.env"
|
||||
-f compose.yaml -f deploy/compose.local.yaml
|
||||
-f "deploy/compose.git-$THT_GIT_ACCESS.yaml"
|
||||
-f deploy/local/generated/compose.models.yaml
|
||||
)
|
||||
"${compose[@]}" config --quiet
|
||||
"${compose[@]}" build core frontend
|
||||
"${compose[@]}" up -d catalog-db
|
||||
"${compose[@]}" run --rm catalog-migrate
|
||||
tht --installation "$INSTALLATION" start
|
||||
```
|
||||
|
||||
Stop if a command fails. The project name matches the hash used by `tht`, preserving volume
|
||||
identity. `catalog-migrate` applies Catalog and Memory migrations; `tht start` does not run it
|
||||
automatically. Initial embedding-model download may take time. Use this installation-specific
|
||||
sequence, not `run-stack.sh` with a different environment/project name.
|
||||
|
||||
## 4. Verify the installation
|
||||
|
||||
The descriptor generated for the default ID is:
|
||||
|
||||
```sh
|
||||
INSTALLATION="$(pwd -P)/deploy/local/thothii-installation.yaml"
|
||||
test -f "$INSTALLATION"
|
||||
bash scripts/verify-standalone-install.sh "$INSTALLATION"
|
||||
```
|
||||
|
||||
The verifier is read-only: it runs `tht doctor --json` and `tht status` without restarting the
|
||||
stack, regenerating configuration, or printing secret contents.
|
||||
|
||||
### Gate A — platform smoke test on all three computers
|
||||
|
||||
Record the following for each machine:
|
||||
|
||||
```sh
|
||||
uname -a
|
||||
docker version --format '{{.Server.Version}} {{.Server.Arch}}'
|
||||
tht version
|
||||
bash scripts/check-standalone-prerequisites.sh
|
||||
bash scripts/verify-standalone-install.sh "$INSTALLATION"
|
||||
```
|
||||
|
||||
The gate passes when the clone is intact, Docker and Compose are reachable, `tht doctor` is OK, the
|
||||
stack is running, and the frontend responds at the default local URL `http://127.0.0.1:8080`.
|
||||
Doctor also checks workspace and Pi: record their failures separately rather than labeling every
|
||||
failure as a platform problem. Check HTTP readiness with:
|
||||
|
||||
```sh
|
||||
curl --fail --silent --show-error http://127.0.0.1:8080/health
|
||||
```
|
||||
|
||||
### Gate B — functional verification
|
||||
|
||||
Run this on at least one machine with available endpoints and credentials:
|
||||
|
||||
First follow [Workspace operations](../operations/workspaces.md) to import/prepare the workspace
|
||||
and configure the Database and local binding. The source clone does not transfer catalog data,
|
||||
secrets or sessions from the Mac. Connect any VPN required by DWH/model endpoints and verify
|
||||
that their names are reachable from containers too.
|
||||
|
||||
1. open `http://127.0.0.1:8080`;
|
||||
2. sign in with the configured local account;
|
||||
3. verify that the configured workspace is readable;
|
||||
4. start a real question and complete the review gates through final SQL;
|
||||
5. stop and restart the installation, then run `verify-standalone-install.sh` again.
|
||||
|
||||
A Gate B failure involving DWH, the LLM provider, workspace Git, or credentials does not by itself
|
||||
prove a Docker portability problem: record the failed endpoint or component separately.
|
||||
|
||||
## Daily lifecycle
|
||||
|
||||
Use the explicit descriptor when more than one installation may be discoverable:
|
||||
|
||||
```sh
|
||||
INSTALLATION="$PWD/deploy/local/thothii-installation.yaml"
|
||||
|
||||
tht --installation "$INSTALLATION" status
|
||||
tht --installation "$INSTALLATION" start
|
||||
tht --installation "$INSTALLATION" start --build
|
||||
tht --installation "$INSTALLATION" logs
|
||||
tht --installation "$INSTALLATION" doctor --json
|
||||
tht --installation "$INSTALLATION" stop
|
||||
```
|
||||
|
||||
Use `start --build` after source changes or to rebuild images from the current checkout. `stop`
|
||||
preserves volumes, sessions, the catalog, Pi state, Qdrant data, and the embedding model. Do not
|
||||
use `docker compose down --volumes` during a normal test: it is destructive and removes local data.
|
||||
For upgrades requiring migrations, follow the release runbook before starting the new application.
|
||||
|
||||
## Quick diagnosis
|
||||
|
||||
| Symptom | Check |
|
||||
| --- | --- |
|
||||
| `Docker Engine is not reachable` | start Docker Desktop or the Docker service and rerun `docker info` |
|
||||
| Windows sees Docker but Bash fails | run the guide inside Ubuntu WSL2 and enable that distribution in Docker Desktop |
|
||||
| `tht: command not found` | open a new shell and check `command -v tht`; rerun the bootstrap if needed |
|
||||
| line-ending or executable-script errors | use a clone in the WSL2/Linux filesystem and rerun `bash scripts/...` |
|
||||
| unsupported architecture | check `docker version --format '{{.Server.Arch}}'`; the test requires `amd64` or `arm64` |
|
||||
| missing descriptor or env file | use `deploy/local/...` generated by `tht setup`, not an arbitrary copied file |
|
||||
| healthy stack but workflow failure | check external URLs, the credential bundle, workspace Git, and authentication separately |
|
||||
| data appears missing | check that `down --volumes` was not used; `stop` does not remove volumes |
|
||||
|
||||
## Acceptance checklist
|
||||
|
||||
- [ ] The clone comes from the expected Gitea repository and the revision is recorded.
|
||||
- [ ] Docker Desktop/Engine and Compose v2 are available.
|
||||
- [ ] The runtime reports an allowed architecture.
|
||||
- [ ] `tht` was built from the repository and responds to `tht version`.
|
||||
- [ ] Setup uses `profile: local`, `shell.mode: full`, and `shell.defaultLocale: en`.
|
||||
- [ ] The descriptor, `operator.env`, authentication, and secrets exist only under `deploy/local/`.
|
||||
- [ ] No secret appears in Git, URLs, public YAML, or recorded commands.
|
||||
- [ ] Gate A passes on Apple Silicon macOS, x64 Windows WSL2, and x64 Linux.
|
||||
- [ ] Gate B runs on at least one machine with DWH and LLM available.
|
||||
- [ ] Stop/start and final verification complete without deleting volumes.
|
||||
|
||||
## Out of scope for this release
|
||||
|
||||
The following remain future work:
|
||||
|
||||
- publishing pre-built images on Docker Hub;
|
||||
- reducing prompts through a dedicated non-interactive configuration;
|
||||
- creating DMG, MSI/EXE, AppImage, or other native installers;
|
||||
- providing an offline runtime or bundling a local DWH/LLM into the application.
|
||||
|
||||
## Related documents
|
||||
|
||||
- [Install and first start](first-start.md)
|
||||
- [Shell and localization](../operations/shell-and-localization.md)
|
||||
- [Workspace operations](../operations/workspaces.md)
|
||||
- `deploy/secrets/README.md` (runtime secrets)
|
||||
@@ -0,0 +1,329 @@
|
||||
# Installazione manuale standalone
|
||||
|
||||
[English version](standalone-manual-en.md)
|
||||
|
||||
Questa è la procedura di prova per predisporre THothII come applicazione autonoma in modalità
|
||||
`full` su macOS, Windows e Linux.
|
||||
|
||||
In questo documento “autonoma” significa che l’utente non deve installare Node.js, Python o Pi
|
||||
sull'host: i servizi applicativi e i servizi semantici
|
||||
locali vengono eseguiti con Docker. DWH e provider LLM restano endpoint esterni configurati
|
||||
dall’installazione; questa procedura non è un pacchetto offline.
|
||||
|
||||
Il percorso non usa installer grafici, launcher nativi o immagini Docker Hub. Si parte da un clone
|
||||
Gitea e si usano comandi espliciti da terminale. La pubblicazione di immagini pre-costruite è una
|
||||
fase successiva.
|
||||
|
||||
## Matrice di verifica
|
||||
|
||||
| Sistema | Terminale raccomandato | Runtime | Architettura della prova |
|
||||
| --- | --- | --- | --- |
|
||||
| macOS supportato dalla versione Docker Desktop installata | Bash nel Terminale | Docker Desktop | Apple Silicon (`arm64`) |
|
||||
| Windows 11 | Ubuntu dentro WSL2 | Docker Desktop con integrazione WSL2 | x64 (`amd64`) |
|
||||
| Linux Ubuntu 22.04 o 24.04 | Bash | Docker Engine + Compose v2 | x64 (`amd64`) |
|
||||
|
||||
Intel macOS non fa parte della prima campagna di verifica. ARM Linux può essere provato quando il
|
||||
runtime Docker della macchina restituisce `arm64`, ma non è un requisito della matrice minima.
|
||||
|
||||
Le verifiche documentali e dei prerequisiti Mac sono passate. Le installazioni complete da zero
|
||||
sui tre sistemi restano da eseguire: la matrice indica le prove previste, non una certificazione.
|
||||
|
||||
## Cosa serve prima di iniziare
|
||||
|
||||
Servono:
|
||||
|
||||
- accesso al repository Gitea di THothII e al repository Git dei workspace;
|
||||
- Git;
|
||||
- Docker Desktop su macOS e Windows, oppure Docker Engine con il plugin Compose v2 su Linux;
|
||||
- Bash, `curl`, OpenSSL e `shasum` (su Ubuntu, pacchetto `libdigest-sha-perl`);
|
||||
- spazio disco sufficiente per compilare le immagini e scaricare il modello di embedding;
|
||||
- gli endpoint DWH e LLM, più le credenziali che l’installazione deve usare.
|
||||
|
||||
Su Linux l’utente corrente deve poter eseguire Docker. Se il sistema richiede `sudo`, aggiungere
|
||||
l’utente al gruppo Docker secondo la policy locale e aprire una nuova sessione prima di continuare.
|
||||
|
||||
Su Windows usare Ubuntu in WSL2 per tutti i comandi di questa guida. In Docker Desktop attivare
|
||||
l’integrazione WSL2 per quella distribuzione. Clonare il progetto nel filesystem Linux di WSL2,
|
||||
per esempio sotto `~/src`, e non sotto `/mnt/c`: si evitano rallentamenti e problemi di permessi o
|
||||
line ending. Non è necessario installare Pi sull’host.
|
||||
|
||||
Verificare il runtime prima del clone o subito dopo:
|
||||
|
||||
```sh
|
||||
docker version
|
||||
docker compose version
|
||||
docker version --format '{{.Server.Arch}}'
|
||||
```
|
||||
|
||||
L’ultima istruzione deve restituire `amd64`, `x86_64`, `arm64` o `aarch64`.
|
||||
|
||||
## 1. Clonare una revisione del progetto
|
||||
|
||||
Usare il repository di progetto su Gitea:
|
||||
|
||||
```sh
|
||||
mkdir -p "$HOME/src"
|
||||
cd "$HOME/src"
|
||||
git clone https://git.tylconsulting.it/mptyl/ThothII.git
|
||||
cd ThothII
|
||||
git rev-parse --short HEAD
|
||||
```
|
||||
|
||||
Per un clone SSH usare, se la chiave è già autorizzata su Gitea:
|
||||
|
||||
```sh
|
||||
git clone git@git.tylconsulting.it:mptyl/ThothII.git
|
||||
```
|
||||
|
||||
Per una prova ripetibile annotare l’hash stampato da `git rev-parse`. In una campagna successiva
|
||||
usare la revisione/tag approvata dal maintainer invece di seguire implicitamente una `main` che
|
||||
può cambiare.
|
||||
|
||||
## 2. Verificare i prerequisiti e installare il comando operatore
|
||||
|
||||
Dal root del clone:
|
||||
|
||||
```sh
|
||||
bash scripts/check-standalone-prerequisites.sh
|
||||
export PATH="$HOME/.local/bin:$PATH"
|
||||
THT_INSTALL_DIRECTORY="$HOME/.local/bin" bash scripts/install-tht.sh
|
||||
tht version
|
||||
```
|
||||
|
||||
`install-tht.sh` è un bootstrap del solo comando operatore nativo `tht`; non installa una versione
|
||||
desktop di THothII. Usa il builder Docker del repository, installa il binario adatto all’ambiente
|
||||
del terminale e lo installa nella directory utente. Aggiungere `$HOME/.local/bin` al PATH della
|
||||
shell anche per i terminali successivi. Un `tht` già presente in quella directory viene aggiornato.
|
||||
|
||||
Su Windows, eseguire questi comandi dentro WSL2. Il binario `tht` installato è quello Linux di WSL2;
|
||||
il runtime dell’applicazione rimane Docker Desktop. Non usare `scripts/install-tht.ps1` come percorso
|
||||
principale di questa prova.
|
||||
|
||||
## 3. Configurare e avviare l’installazione locale
|
||||
|
||||
Eseguire i blocchi successivi nella stessa sessione Bash dal root fisico del clone (`pwd -P`).
|
||||
Creare prima le due password distinte del catalogo, conservando eventuali file già esistenti:
|
||||
|
||||
```bash
|
||||
umask 077
|
||||
mkdir -p deploy/local/secrets
|
||||
for name in catalog-runtime-password catalog-migrator-password; do
|
||||
target="deploy/local/secrets/$name"
|
||||
if [ ! -e "$target" ]; then
|
||||
(set -C; openssl rand -hex 32 > "$target") || exit 1
|
||||
fi
|
||||
done
|
||||
export THT_CATALOG_RUNTIME_PASSWORD_SOURCE="$(pwd -P)/deploy/local/secrets/catalog-runtime-password"
|
||||
export THT_CATALOG_MIGRATOR_PASSWORD_SOURCE="$(pwd -P)/deploy/local/secrets/catalog-migrator-password"
|
||||
```
|
||||
|
||||
Non rigenerare le password di un catalogo già inizializzato. Configurare senza avviare i servizi:
|
||||
|
||||
```sh
|
||||
tht setup --profile local --shell-mode full --shell-default-locale en --configure-only
|
||||
```
|
||||
|
||||
Rispondere ai prompt nel seguente modo:
|
||||
|
||||
| Prompt | Valore o regola |
|
||||
| --- | --- |
|
||||
| Installation ID | `local`, salvo necessità di più installazioni nello stesso clone |
|
||||
| Deployment profile | `local` |
|
||||
| DWH API endpoint | URL `http(s)` senza user, password, query o fragment; può restare vuoto per il solo smoke test |
|
||||
| LLM API endpoint | URL `http(s)` senza credenziali; può restare vuoto per il solo smoke test |
|
||||
| Workspace repository URL | URL del repository dei workspace, non il clone sorgente di THothII |
|
||||
| Workspace branch | normalmente `main` |
|
||||
| Workspace access | `ssh` se si usa una chiave deploy; altrimenti `https` con credential file protetto |
|
||||
| Percorsi dei file | accettare i percorsi predefiniti sotto `deploy/local/secrets/` nella prima prova |
|
||||
| Secret templates | rispondere `yes` quando i file protetti non esistono ancora |
|
||||
| Autenticazione | configurare il login locale richiesto dall’installazione; non inserire password in una riga di comando |
|
||||
|
||||
La configurazione generata è locale e ignorata da Git:
|
||||
|
||||
```text
|
||||
deploy/local/thothii-installation.yaml
|
||||
deploy/local/operator.env
|
||||
deploy/local/auth/
|
||||
deploy/local/secrets/
|
||||
```
|
||||
|
||||
Modificare i segreti solo nei file locali protetti; non committarli. Il file `deploy/env/local.env.example` è un riferimento
|
||||
tracciato; il percorso generato da `tht setup`, `deploy/local/operator.env`, è quello da usare per
|
||||
questa installazione.
|
||||
|
||||
### Completare i file protetti
|
||||
|
||||
Se il setup ha creato template vuoti, inserire i valori con un editor locale:
|
||||
|
||||
```sh
|
||||
chmod 600 deploy/local/secrets/*
|
||||
"${EDITOR:-vi}" deploy/local/secrets/thothii.secrets
|
||||
```
|
||||
|
||||
Il bundle deve contenere solo righe `KEY=VALUE` per le credenziali effettivamente usate dal
|
||||
`modelCatalog`. I nomi ammessi e il confine delle credenziali sono descritti nel file locale
|
||||
`deploy/secrets/README.md`. Non mettere token nelle URL, nel
|
||||
descriptor YAML, nel repository Git o nei comandi copiati nella shell.
|
||||
|
||||
Per accesso workspace SSH, predisporre anche la chiave privata e il file `known_hosts` indicati dal
|
||||
setup. Per accesso HTTPS, predisporre il credential file Git e l’eventuale CA. Entrambi devono
|
||||
restare protetti e fuori dal controllo versione.
|
||||
|
||||
Prima dell'avvio completare anche questi passaggi:
|
||||
|
||||
1. Aggiungere `THT_CATALOG_RUNTIME_PASSWORD_SOURCE` e `THT_CATALOG_MIGRATOR_PASSWORD_SOURCE` a
|
||||
`deploy/local/operator.env`, con gli stessi percorsi assoluti esportati sopra. Il setup non salva
|
||||
queste due variabili. Inserire i percorsi, non le password.
|
||||
2. Sostituire il `modelCatalog` generico nel descriptor con la configurazione provider/modelli
|
||||
approvata. I default generati non replicano il Mac esistente. Vedere [configurazione Pi/modelli](../general/pi-configuration.md)
|
||||
e l'esempio locale `deploy/psd/thothii-installation.yaml.example`.
|
||||
3. Inserire in `thothii.secrets` le chiavi referenziate da `authentication.apiKeyEnv`. I provider
|
||||
`pi_auth` richiedono credenziali valide nel file `PI_AUTH_FILE`; il template `{}` non autentica.
|
||||
4. Completare i file Git del workspace. SSH richiede una chiave deploy autorizzata e known-hosts
|
||||
verificato; HTTPS richiede il credential file e il bundle CA previsti dall'overlay. I template
|
||||
vuoti non consentono l'accesso al repository.
|
||||
|
||||
Dopo aver modificato la configurazione generata, non rilanciare setup: rifiuta file esistenti con
|
||||
contenuti diversi. Generare le proiezioni ed eseguire la migrazione esplicita indicata sotto.
|
||||
Impostare `THT_GIT_ACCESS=https` se scelto nel setup. Il blocco usa il nuovo descriptor `local`
|
||||
con il solo overlay Git; per installazioni personalizzate includere gli overlay aggiuntivi
|
||||
nello stesso ordine del descriptor.
|
||||
|
||||
```bash
|
||||
INSTALLATION="$(pwd -P)/deploy/local/thothii-installation.yaml"
|
||||
tht --installation "$INSTALLATION" installation generate
|
||||
THT_PROJECT="thothii-$(printf '%s' "$INSTALLATION" | shasum -a 256 | cut -c 1-12)"
|
||||
THT_GIT_ACCESS=ssh
|
||||
compose=(
|
||||
docker compose --project-name "$THT_PROJECT" --project-directory "$(pwd -P)"
|
||||
--env-file "$(pwd -P)/deploy/local/operator.env"
|
||||
-f compose.yaml -f deploy/compose.local.yaml
|
||||
-f "deploy/compose.git-$THT_GIT_ACCESS.yaml"
|
||||
-f deploy/local/generated/compose.models.yaml
|
||||
)
|
||||
"${compose[@]}" config --quiet
|
||||
"${compose[@]}" build core frontend
|
||||
"${compose[@]}" up -d catalog-db
|
||||
"${compose[@]}" run --rm catalog-migrate
|
||||
tht --installation "$INSTALLATION" start
|
||||
```
|
||||
|
||||
Fermarsi se un comando fallisce. Il nome progetto coincide con l'hash usato da `tht`, preservando
|
||||
l'identità dei volumi. `catalog-migrate` applica le migrazioni Catalog e Memory; `tht start` non
|
||||
lo esegue automaticamente. Il primo download del modello embedding può richiedere tempo.
|
||||
Usare questa sequenza legata all'installazione, non `run-stack.sh` con env/progetto diversi.
|
||||
|
||||
## 4. Verificare l’installazione
|
||||
|
||||
Il descriptor generato per l’ID predefinito è:
|
||||
|
||||
```sh
|
||||
INSTALLATION="$(pwd -P)/deploy/local/thothii-installation.yaml"
|
||||
test -f "$INSTALLATION"
|
||||
bash scripts/verify-standalone-install.sh "$INSTALLATION"
|
||||
```
|
||||
|
||||
Il verificatore è read-only: esegue `tht doctor --json` e `tht status`, senza ristartare lo stack,
|
||||
rigenerare la configurazione o stampare il contenuto dei segreti.
|
||||
|
||||
### Gate A — smoke di piattaforma, su tutti e tre i computer
|
||||
|
||||
Registrare per ogni macchina:
|
||||
|
||||
```sh
|
||||
uname -a
|
||||
docker version --format '{{.Server.Version}} {{.Server.Arch}}'
|
||||
tht version
|
||||
bash scripts/check-standalone-prerequisites.sh
|
||||
bash scripts/verify-standalone-install.sh "$INSTALLATION"
|
||||
```
|
||||
|
||||
Il gate passa quando il clone è integro, Docker e Compose sono raggiungibili, `tht doctor` è OK,
|
||||
lo stack è avviato e il frontend risponde sulla porta locale predefinita `http://127.0.0.1:8080`.
|
||||
Il doctor controlla anche workspace e Pi: registrare separatamente i loro errori senza attribuire
|
||||
ogni fallimento alla piattaforma. Verificare la disponibilità HTTP con:
|
||||
|
||||
```sh
|
||||
curl --fail --silent --show-error http://127.0.0.1:8080/health
|
||||
```
|
||||
|
||||
### Gate B — verifica funzionale
|
||||
|
||||
Eseguire almeno su una macchina con endpoint e credenziali disponibili:
|
||||
|
||||
Seguire prima [Workspace operations](../operations/workspaces.md) per importare/preparare il
|
||||
workspace e configurare Database e binding locale. Il clone sorgente non trasferisce catalogo,
|
||||
segreti o sessioni del Mac. Connettere l'eventuale VPN richiesta da DWH/modelli e verificare
|
||||
che i relativi nomi siano raggiungibili anche dai container.
|
||||
|
||||
1. aprire `http://127.0.0.1:8080`;
|
||||
2. autenticarsi con l’account locale configurato;
|
||||
3. verificare che il workspace configurato sia leggibile;
|
||||
4. avviare una domanda reale e completare i gate di revisione fino alla SQL finale;
|
||||
5. fermare e riavviare l’installazione, poi ripetere `verify-standalone-install.sh`.
|
||||
|
||||
Un fallimento del Gate B per DWH, provider LLM, Git workspace o credenziali non dimostra da solo un
|
||||
problema di portabilità Docker: registrare separatamente l’endpoint o il componente fallito.
|
||||
|
||||
## Ciclo di vita quotidiano
|
||||
|
||||
Usare il descriptor esplicito quando più installazioni possono essere scoperte:
|
||||
|
||||
```sh
|
||||
INSTALLATION="$PWD/deploy/local/thothii-installation.yaml"
|
||||
|
||||
tht --installation "$INSTALLATION" status
|
||||
tht --installation "$INSTALLATION" start
|
||||
tht --installation "$INSTALLATION" start --build
|
||||
tht --installation "$INSTALLATION" logs
|
||||
tht --installation "$INSTALLATION" doctor --json
|
||||
tht --installation "$INSTALLATION" stop
|
||||
```
|
||||
|
||||
`start --build` è necessario dopo una modifica al codice o per ricostruire le immagini dal clone
|
||||
corrente. `stop` conserva volumi, sessioni, catalogo, stato Pi, dati Qdrant e modello embedding.
|
||||
Non usare `docker compose down --volumes` durante una prova normale: è un’operazione distruttiva
|
||||
che cancella i dati locali.
|
||||
Per aggiornamenti che richiedono migrazioni, seguire il runbook della release prima dell'avvio.
|
||||
|
||||
## Diagnosi rapida
|
||||
|
||||
| Sintomo | Controllo |
|
||||
| --- | --- |
|
||||
| `Docker Engine is not reachable` | avviare Docker Desktop oppure il servizio Docker e ripetere `docker info` |
|
||||
| Windows vede Docker ma Bash fallisce | eseguire la guida dentro Ubuntu WSL2 e abilitare l’integrazione della distribuzione in Docker Desktop |
|
||||
| `tht: command not found` | aprire una nuova shell e verificare `command -v tht`; se necessario ripetere il bootstrap |
|
||||
| line ending o script non eseguibile | usare un clone nel filesystem WSL2/Linux e rieseguire `bash scripts/...` |
|
||||
| architettura non supportata | verificare `docker version --format '{{.Server.Arch}}'`; la prova richiede `amd64` o `arm64` |
|
||||
| descriptor o env file mancanti | usare il percorso `deploy/local/...` generato da `tht setup`, non un file copiato casualmente |
|
||||
| stack sano ma workflow fallisce | controllare separatamente URL, credential bundle, workspace Git e autenticazione |
|
||||
| dati apparentemente persi | verificare che non sia stato usato `down --volumes`; `stop` non rimuove i volumi |
|
||||
|
||||
## Checklist di accettazione
|
||||
|
||||
- [ ] Il clone proviene dal repository Gitea atteso e la revisione è stata annotata.
|
||||
- [ ] Docker Desktop/Engine e Compose v2 sono disponibili.
|
||||
- [ ] Il runtime restituisce un’architettura ammessa.
|
||||
- [ ] `tht` è stato costruito dal repository e risponde a `tht version`.
|
||||
- [ ] Il setup usa `profile: local`, `shell.mode: full` e `shell.defaultLocale: en`.
|
||||
- [ ] Descriptor, `operator.env`, autenticazione e segreti sono presenti solo in `deploy/local/`.
|
||||
- [ ] Nessun segreto compare in Git, URL, YAML pubblico o comandi registrati.
|
||||
- [ ] Gate A superato su macOS Apple Silicon, Windows WSL2/x64 e Linux x64.
|
||||
- [ ] Gate B eseguito almeno su una macchina con DWH e LLM disponibili.
|
||||
- [ ] Stop/start e verifica finale completati senza cancellare i volumi.
|
||||
|
||||
## Fuori perimetro di questa release
|
||||
|
||||
Restano attività successive:
|
||||
|
||||
- pubblicare immagini pre-costruite su Docker Hub;
|
||||
- ridurre ulteriormente i prompt tramite una configurazione non interattiva dedicata;
|
||||
- creare pacchetti DMG, MSI/EXE, AppImage o altri installer nativi;
|
||||
- fornire un runtime offline o un DWH/LLM locale incluso nell’applicazione.
|
||||
|
||||
## Documenti collegati
|
||||
|
||||
- [Install and first start](first-start.md)
|
||||
- [Shell and localization](../operations/shell-and-localization.md)
|
||||
- [Workspace operations](../operations/workspaces.md)
|
||||
- `deploy/secrets/README.md` (runtime secrets)
|
||||
@@ -1,5 +1,13 @@
|
||||
# Docker installation in the current operating contexts
|
||||
|
||||
Rendering and authentication are independent of these Docker contexts. Explicitly
|
||||
select full/en for the Mac, full/OIDC for an autonomous server, or
|
||||
embedded/upstream for Omics. The same frontend image supports both renderings;
|
||||
generated `config.js` and the host page decide the container, while the backend
|
||||
and trusted proxy decide identity. See [shell configuration and deploy](operations/shell-and-localization.md)
|
||||
and [server portal authentication](install/authentication-upstream.md). Do not
|
||||
apply the standalone server authentication projection to the Omics upstream path.
|
||||
|
||||
ThothII uses one Compose topology:
|
||||
|
||||
- `frontend`
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user