Compare commits

...
Author SHA1 Message Date
Codex c3c9f509bf test: prepare Chinook database for installation acceptance without Evidence 2026-09-28 18:25:33 +02:00
Codex 931ac2fad4 docs: record verified Docker Hub installation prerelease 2026-09-28 17:54:05 +02:00
Codex ec0421e9fd feat: publish verified installation images and native release bundles 2026-09-28 17:46:09 +02:00
Codex 55f3569e55 feat(cli): validate prerequisites and seal installation plans 2026-09-28 17:03:25 +02:00
Codex b9c3369e7b feat(cli): prepare and validate application documents offline 2026-09-28 16:33:07 +02:00
Codex 64e6b9664a feat(cli): prepare and validate workspace documents offline
Reuse the runtime catalog and workspace parsers in a standalone helper paired with tht. Add document templates, safe diagnostics, local Evidence checks, native bundle builds, shared CLI fixtures and IT/EN preparation guides. Record the approved document-first specification and ticket breakdown. Refs #43.
2026-09-28 15:35:25 +02:00
Codex 67ee52624c wip: guided standalone installation and workspace checks 2026-09-26 16:41:15 +02:00
Codex 0d2e573e0d fix(ui): reset session view on stop and exit 2026-09-26 16:40:37 +02:00
Codex bd416f7327 Fix new-question landing and question-language HITL
Publish documentation / publish (push) Successful in 34s
Reset the activity panel when starting a new question so the landing navigation is restored. Detect and persist the original question language, pass it through runtime and widget descriptors, and scope HITL controls to that language.

Validated with gate, session, backend and frontend tests, TypeScript checks, Ruff and strict docs build. Rebuilt and restarted local core/frontend; both healthy and serving HTTP successfully.
2026-09-21 19:47:22 +02:00
Codex 23e52c80de Record verified Qwen documentation publication
Publish documentation / publish (push) Successful in 23s
2026-09-21 16:27:10 +02:00
Codex 84084bba37 Fix Qwen session tool calls and expose thinking compatibility
Publish documentation / publish (push) Successful in 30s
2026-09-21 16:23:51 +02:00
pinoricci1956 efd7d788d9 correzione scroller verticale pagina di configurazione catalogo 2026-09-16 11:59:51 +02:00
Codex b1c510a097 fix(docs): preserve theme assets in deny-by-default publication
Publish documentation / publish (push) Successful in 36s
2026-09-16 09:36:30 +02:00
Codex 5f3680a0fb docs: record verified live manual publication
Publish documentation / publish (push) Successful in 28s
2026-09-15 14:39:46 +02:00
Codex 4ff91e8d6e docs: consolidate historical records and verify public manual publication
Publish documentation / publish (push) Successful in 29s
2026-09-15 14:37:29 +02:00
Codex 5f3a7f5975 docs: record stale public site publication blocker
Publish documentation / publish (push) Successful in 23s
2026-09-15 10:28:49 +02:00
Codex 043ffdfad6 docs: separate public manual from internal project documentation
Publish documentation / publish (push) Successful in 27s
2026-09-15 10:26:35 +02:00
188 changed files with 9663 additions and 7432 deletions
+4
View File
@@ -30,3 +30,7 @@ coverage/
data/ data/
sessions/ sessions/
workspace-registry/ workspace-registry/
.tht/
deploy/local/
+20 -4
View File
@@ -7,6 +7,11 @@ on:
paths: paths:
- "docs/**" - "docs/**"
- "mkdocs.yml" - "mkdocs.yml"
- "scripts/build-docs.sh"
- "scripts/verify-public-docs.py"
- "scripts/test-verify-public-docs.py"
- "scripts/verify-auth-docs.py"
- "scripts/test-verify-auth-docs.py"
- "docs/requirements.txt" - "docs/requirements.txt"
- ".gitea/workflows/publish-docs.yml" - ".gitea/workflows/publish-docs.yml"
workflow_dispatch: workflow_dispatch:
@@ -34,14 +39,25 @@ jobs:
with: with:
python-version: "3.x" python-version: "3.x"
cache: pip cache: pip
cache-dependency-path: docs/requirements.txt cache-dependency-path: docs/requirements.lock
- name: Install MkDocs dependencies - name: Install MkDocs dependencies
run: python -m pip install -r docs/requirements.txt run: python -m pip install -r docs/requirements.lock
- name: Test public documentation boundary
run: python scripts/test-verify-public-docs.py
- name: Test current authentication documentation
run: |
python scripts/verify-auth-docs.py auth
python scripts/verify-auth-docs.py dwh
python scripts/test-verify-auth-docs.py auth
python scripts/test-verify-auth-docs.py dwh
- name: Build documentation - name: Build documentation
# Some documented source files intentionally live outside docs/. run: |
run: mkdocs build mkdocs build --strict
python scripts/verify-public-docs.py
- name: Publish generated site to the pages branch - name: Publish generated site to the pages branch
working-directory: site working-directory: site
+11
View File
@@ -638,6 +638,17 @@ procedura non implica che DWH o provider LLM siano locali o disponibili offline.
installazione manuale: verifica dell'host, generazione della configurazione, predisposizione installazione manuale: verifica dell'host, generazione della configurazione, predisposizione
delle credenziali protette e avvio dei servizi. Non è un installer dell'applicazione. delle credenziali protette e avvio dei servizi. Non è un installer dell'applicazione.
**Installation preparation** — La predisposizione dei documenti che descrivono i workspace
e i parametri dell'installazione, prima di applicarli. Permette all'operatore di raccogliere
e correggere le informazioni senza avviare l'applicazione.
**Installation validation** — La verifica ripetibile della completezza e coerenza dei
documenti e delle precondizioni di un'installazione. Distingue ciò che è stato verificato
da ciò che richiede un'applicazione già avviata.
**Installation execution** — L'applicazione dei documenti verificati per predisporre e
avviare ThothII. Non raccoglie nuovi parametri dall'operatore durante l'esecuzione.
**Platform acceptance** — La verifica che una Manual standalone installation possa essere **Platform acceptance** — La verifica che una Manual standalone installation possa essere
predisposta e avviata su una specifica combinazione di sistema operativo, architettura e runtime, predisposta e avviata su una specifica combinazione di sistema operativo, architettura e runtime,
distinta dalla verifica funzionale del collegamento a DWH e provider LLM. distinta dalla verifica funzionale del collegamento a DWH e provider LLM.
+120 -704
View File
@@ -1,704 +1,120 @@
# ThothII — Project State # Project state
Last updated: 2026-09-14. Updated: 2026-09-28. This is a current snapshot, not a release diary. Stable commands
and invariants are in [AGENTS.md](AGENTS.md); prior snapshots remain in Git.
This file is the short operational snapshot. Stable commands and the architecture mental model
live in `AGENTS.md`; current design and runtime contracts live under `docs/architecture/`, ## Current contracts
`docs/contracts/`, `docs/adr/`, and `docs/evidence.md`. Superseded plans and reports are
available from Git history rather than duplicated in the working tree. - Document-first installation ticket #43 provides offline `tht workspace prepare`
and `tht workspace validate` through a native two-executable bundle. Build and
The guarded server migration from a legacy checkout to the schema-v2 installation, Gitea source, test instructions are in [the host CLI guide](tools/tht/README.md). Local validation
Authentik, internal catalog/embedding services, and the PSD workspace repository is documented in reuses runtime workspace/catalog parsers and explicitly defers runtime Evidence,
`docs/operations/server-upgrade-gitea-workspace-v2.md`. Treat its operator gates and rollback database binding and readiness checks. Ticket #44 adds `tht installation prepare`,
requirements as mandatory; do not replace the running server stack in place. explicit `installation credentials`, and `installation validate --workspaces PATH`
for protected application documents, canonical model settings and schema-v1
## Session review layout deployed — 2026-09-14 database bootstrap inputs. These commands do not start services or import Catalog
bindings. Ticket #45 adds host `installation preflight` and `installation plan`
Session-only dialogs now use the visible app bounds: artifact/column review grows with release/image checks, canonical external diagnostics and private input seals;
up to 80rem wide and the available height; short confirmations grow to 40rem. see [the preflight reference](docs/install/installation-preflight.md).
Existing primary actions appear above and below session forms and review content, Ticket #46 adds the maintainer release producer and the first public
with shared handlers, validation and pending state. Stop/delete focus Cancel; [Linux amd64 prerelease](https://git.tylconsulting.it/mptyl/ThothII/releases/tag/installation-v0.1.0-install-preview.1),
rename focuses the name field. Administration surfaces are unchanged. The activity with Docker Hub core/frontend images and a downloadable native operator bundle.
shortcut is relabelled To Administration / Vai all’Amministrazione, retaining its See [publication instructions](docs/install/publishing-images.md) and
workspace-management destination. 778 frontend tests, five responsive browser [verification evidence](docs/reports/2026-09-28-installation-prerelease.md).
scenarios, typecheck, translations and the production build passed. The owner Non-interactive execution (#47 onward), real-host acceptance and example
authorized deployment and the server frontend was recreated at 18:09 CEST with tag `b1723c34-session-dialogs-20260914`. Frontend is healthy, databases remain pending; the prerelease does not certify a complete installer.
Omics serves the new assets, and doctor passes 13/13 checks. Core and Omics web
were not restarted. See DESIGN.md for the layout contract and - React supports full/embedded rendering independently of local/OIDC/upstream auth,
`docs/reports/2026-09-14-session-dialogs-release.md` for provenance and rollback. with EN/IT UI and immutable session interaction language. See
[application shell](docs/architecture/application-shell.md) and
## Session composer and empty Memory fix deployed — 2026-09-14 [localization](docs/operations/shell-and-localization.md).
- PostgreSQL Metadata Catalog owns database identity, binding, schema, descriptions,
The embedded shell now caps its height at the portal mount height, keeping steering sensitivity and relationships for all core consumers. Workspace schema v4 contains
and Stop & save visible. Empty authoritative Memory archives return zero results identity and optional Evidence only. Installation schema v2 is the authored model
without requiring embedding/BM25; SQL-rule embedding is lazy and shared. The live catalog source. See [overview](docs/architecture/overview.md) and
empty PSD archive reproduces 503 with the current image and succeeds with the [model configuration](docs/general/pi-configuration.md).
candidate. 54 Memory tests, 90 frontend tests, five browser scenarios and both image - The harness owns workflow persistence; chat is not the durable session record.
builds passed. The owner authorized the restart and core/frontend were recreated on Memory uses PostgreSQL authority and derived Qdrant dense/BM25 search. Editable
2026-09-14 at 17:18 CEST with tags `49333a2d-session-memory-fix`, from code committed Evidence has local archive authority and manual consolidation. File save, search
as `d6cdffea`. Both are healthy; production Memory search returns `[]`, Omics serves activation and Git publication have distinct outcomes. See
the corrected CSS, native doctor passes 13/13 checks, and admissions are reopened. [Memory](docs/gestione-memory.md), [Evidence](docs/contracts/curated-evidence-v4.md)
Session inventory is preserved. Backups and rollback images are retained. Native and [consolidated release evidence](docs/reports/knowledge-archives-release.md).
CLI diagnostics must run as installation owner UID 10001 with access to Compose; - Reference preprocessing must not clear Memory. Use the installation-scoped
see `docs/reports/2026-09-14-session-layout-memory-fix.md` for exact commands and `tht --installation /absolute/path/thothii-installation.yaml workspace preprocess run`
the remaining interactive browser acceptance. and its [contract](docs/contracts/workspace-preprocessing-cli.md).
- DWH sessions are read-only. SSH tunnels support database-management diagnostics
## Full/embedded shell and bilingual interface and metadata synchronization, not NL→SQL session creation; use direct or REST
transport for sessions.
Full/embedded shell, EN/IT UI, immutable session interaction language, dark theme,
fullscreen and full-mode logout are implemented. The local Mac descriptor explicitly ## Installation and workspace boundaries
sets `shell.mode: full` and `shell.defaultLocale: en`; the native `tht` and local
core/frontend images were updated on 2026-09-13. Omics uses embedded with server-side Fresh standalone installations follow the manual terminal procedures in
identity verification and a replaceable presentation-only PortalAdapter; this does [Italian](docs/install/standalone-manual-it.md) or
not mean this branch was verified on the production server. Its changes are [English](docs/install/standalone-manual-en.md), without an installer or launcher.
in Omics commit `95154e1`; production deployment remains pending. The [Compose reference](docs/operations/compose-reference.md) is for maintainers,
See `docs/operations/shell-and-localization.md` for integration and installation not another quick start. Catalog and Memory migrations are explicit.
instructions and `docs/reports/2026-09-13-full-shell-implementation.md` for tests,
independent reviews, local browser checks and rollback details. Descriptors, authentication, provider credentials, certificates and runtime bindings
stay in protected installation-local paths. Do not copy secrets into examples or
### Documentation handoff before branch closure — 2026-09-13 workspace Git. Legacy runtime snapshots may use absolute paths and protected
`harness/.env`; do not silently relocate them.
Current rendering architecture is in `docs/architecture/application-shell.md`;
the exact portal identity/proxy contract is in `docs/install/authentication-upstream.md`. PSD authoring is separate at `/Users/mp/projects/tht-workspace-psd`. Its GitHub
`docs/operations/server-codex-handoff.md` is the current server delivery/deploy repository was copied to private Gitea
runbook, including Omics source integration from GitHub, configuration, tests and [workspace_psd](https://git.tylconsulting.it/mptyl/workspace_psd), preserving both
rollback. It supersedes the earlier Omics repository-relay instructions. The acceptance matrix is branches. It is a copy, not automatic synchronization. Running installations were
`docs/testing/authentication-manual-acceptance.md`. README, documentation navigation, not repointed to a different workspace remote.
user/installation/authentication guides and descriptor examples point to these
paths. Local examples explicitly use full/en; the projected server example is ## Recorded deployments and server authority
for standalone OIDC, not Omics upstream. No runtime configuration or deployment
was changed by that documentation pass. The 2026-09-14 delivery is prepared for Use the ordered [server handoff](docs/operations/server-codex-handoff.md) for
promotion to main; the actual merge is recorded in Git. PSD acceptance remains pending. coordinated ThothII/Omics upgrades. Omics integration uses GitHub
`Dallavilla-Tiziano/omics_portal`, with no Gitea relay prerequisite. Omics uses
### Navigation readiness and session accordions — 2026-09-13 embedded/upstream identity, not another ThothII OIDC login. Read
[upstream authentication](docs/install/authentication-upstream.md) before changes.
The Workspace navigation button now carries an accessible green/red readiness
dot instead of a separate text row. Green requires a selected workspace and a Last recorded application deliveries (not a fresh runtime attestation):
successful, current `ready` response; checking, unavailable and other states are
red, with the state exposed through the tooltip and accessible description. - [Session dialogs](docs/reports/2026-09-14-session-dialogs-release.md):
The backend readiness gate is unchanged. `b1723c34-session-dialogs-20260914`, frontend-only.
- [Session layout/Memory fix](docs/reports/2026-09-14-session-layout-memory-fix.md):
Active sessions (non-archived) and Archive both start collapsed. Their adjacent `49333a2d-session-memory-fix`, core/frontend.
headers are the only visible content below the scope tabs: no Sessions heading
or external selection toolbar. Each nonempty panel owns its Select all and Keep those reports and rollback instructions while operator gates remain open.
bulk-delete controls, scoped to that list and preserving the other's selection. A later deployment does not prove every earlier acceptance item passed.
As of 2026-09-14, only one section can be open at a time, and either can be
collapsed. Empty lists show only "No sessions yet." The open section uses the ## Remaining acceptance and design gates
remaining sidebar height, with scrolling content capped at `min(18rem, 35dvh)`. The mobile
navigation dialog also provides a bounded height. Keyboard controls and labels - The first installation acceptance uses an ad hoc
are retained. See `DESIGN.md` and `docs/guida-utente.md` for the UI contract. [Chinook workspace without Evidence](docs/testing/chinook-installation-smoke.md),
backed by a separate local PostgreSQL test container. Its fixture import,
Verification: 768 frontend unit tests, 20 browser scenarios (including 80 mocked read-only credentials and offline workspace validation were verified; full
sessions at 390/1280px), TypeScript and the production frontend build passed. installation/readiness and a human-reviewed question remain pending. Evidence
Only the Mac frontend was recreated; it is healthy at `127.0.0.1:8080`. acceptance and the three curated examples are separate later work.
Core, catalog, Qdrant and embedding containers were not changed. The prior - Fresh-machine Mac, Windows/WSL2 and Linux installation acceptance, including real
frontend image is retained as DWH/model endpoints, remains a separate operator exercise.
`thothii-frontend:before-single-session-accordion-20260914` for rollback. - Real IdP/portal login, logout, embedded interaction and PSD semantic acceptance
follow the [manual matrix](docs/testing/authentication-manual-acceptance.md) and
### Current Omics delivery and server handoff — 2026-09-14 delivery reports; synthetic tests do not close them.
- [Security hardening](docs/plans/2026-09-08-security-hardening-prd.md) is a draft.
The owner corrected the delivery requirement: Omics is obtained from GitHub, Revalidate SEC01–12 and obtain design approval before implementation or real
not relayed to another repository as part of this deployment. Any optional server/IdP/DWH mutation.
server-side repository copy is solely the owner's separate concern. This - Optional NER remains opt-in; labeled Italian quality, benchmark and licensing
supersedes the 2026-09-13 relay agreement, including historical delivery notes acceptance are not implied by document cleanup.
in the Omics branch. Do not make another remote publication a prerequisite. - Semantic aliases, value descriptions, synonyms/concepts, dialect and multi-schema
extensions remain explicit design work. Current sensitivity delivery follows the
GitHub branch `codex/thothii-embedded-shell` at Catalog contract; additional policies require their own acceptance.
`https://github.com/Dallavilla-Tiziano/omics_portal.git` was reverified at - Legacy database UI fallback (`?db-ui=legacy`, dev/staging) and prototype removal
`fca10901a73666ca257d8f4cc4b77066295c400a` (functional commit `95154e1`). remain subject to owner acceptance.
The server operator integrates it with the current code in the confirmed Omics
checkout, normally `/home/chirone/omics_portal`, preserving later server changes. ## Documentation maintenance
`docs/operations/server-codex-handoff.md` is the authoritative ordered procedure:
inventory, source verification, native CLI and installation projections, MkDocs publishes only 22 product/operator pages and five approved assets.
embedded/upstream auth, Omics templates/static assets/proxy, coordinated rollout, Architecture, contracts, ADRs, plans, research, tests and release evidence are
acceptance and rollback. Server deployment and real IdP acceptance remain pending. excluded from HTML and search. The repository itself is public: editorial exclusion
is not confidentiality.
## Current product shape
The [cleanup record](docs/maintenance/2026-09-15-documentation-cleanup.md) records
### Shared Memory/Evidence typography — 2026-09-13 retired sources and retained gates. Main contains source; Actions generates the
`pages` branch. The live site requires the separate explicit deployment described
Both detail readers now share Manrope and fixed reading roles: 24px card title, in [public manual publication](docs/operations/public-docs-publication.md).
20px section headings, 16px/1.65 narrative text, and 14px metadata/code. Authored
Markdown subheadings remain subordinate to field headings; inline/fenced code no
longer shrinks cumulatively. Full-width layout, paragraph separation and isolated
FAKE examples are retained. All 762 frontend tests and 18 browser scenarios pass,
including computed typography checks at 390/1280/2400px; typecheck, i18n and Docker
build pass. Only Mac frontend was recreated: `87fca0dc5e19`, image `19f1704b6bb2`,
healthy. Core and data services are unchanged. Rollback image:
`thothii-frontend:before-knowledge-typography-20260913`. See
`docs/reports/2026-09-13-knowledge-typography.md`. No PSD/Omics deployment.
### Knowledge reading and isolated fake Memory examples — 2026-09-13
Memory/Evidence details now use all available width, with display-only paragraph
splitting for long plain prose and preserved code/Markdown/source data. Evidence
provenance renders Markdown; copy controls are accessible two-sheet icons.
Memory's separate Formatting examples section contains four FAKE cards (one per
family), automatically expanded for an empty archive. These client-side examples
cannot be edited/saved/indexed and are never submitted to the model or Memory API.
No real archive content was changed. 762 tests, 18 browser scenarios, typecheck,
i18n and Docker build pass. Only the Mac frontend was recreated: `fc4286b5406d`,
image `cbe0fe0dce55`, healthy; other services unchanged. Rollback image:
`thothii-frontend:before-knowledge-reading-20260913`. Details in
`docs/reports/2026-09-13-knowledge-reading.md`; no PSD/Omics deployment.
### Unified Session entry and complete tab borders — 2026-09-13
The sidebar has one Session/Sessione button: return to the current unfinished
session (including pending creation) without resetting/reconnecting it; otherwise
prepare a new question with existing readiness and dirty-edit guards. Creation
still requires submitting a question. A cold document panel is not a running session.
Session tabs now have 11px horizontal/3px vertical padding, at least 38px height,
and matching 1px borders on every side (gray inactive, red active), with no shared
baseline or overlapping bottom border. This supersedes the earlier border removal.
757 frontend tests, 15 browser scenarios, typecheck, i18n and Docker build pass.
Mac frontend `2844778d311c`, image `d58dccfee3f9`, is healthy; only that service
was recreated. Core/data services are unchanged, with no Omics or server deploy.
Rollback image: `thothii-frontend:before-session-navigation-20260913`.
### Login copy and language-selector focus — 2026-09-13
Removed the redundant login eyebrow/icon and installation-account explanation;
the main sign-in heading remains. The full-header language select no longer
shows an outer focus ring after pointer interaction; keyboard focus remains
visible, including after returning with Tab. EN/IT login and both themes are
covered by 36 targeted tests and 14 browser scenarios; typecheck/build pass.
Only the Mac frontend was rebuilt/recreated: `204efd40d3eb`, image `7dd0752ff823`,
healthy. Other containers are unchanged. Rollback image:
`thothii-frontend:before-login-focus-20260913`. No production deployment.
### Full-header and layout refinements — 2026-09-13
The owner's four visual adjustments are implemented: full header uses Omics
`#CB333B` in both themes with a light complete wordmark/controls; Database status
has 16px clearance below its top divider; workspace/model/Done share a compact
desktop row; session-scope tabs have no bottom border. Embedded has no extra header.
Verified with 71 targeted frontend tests, 11 browser scenarios, typecheck and
Docker production build. The Mac frontend was recreated alone and is healthy
(`8a1608ad7fc6`, image `a2f489ebe9de`); Core/data-service containers are unchanged.
Full/en is retained. Rollback image: `thothii-frontend:before-header-layout-20260913`.
See `docs/reports/2026-09-13-header-layout-refinements.md` for validation and rollback.
### Visual review integrated with the full/embedded shell
At the owner's request, the seven commits through `a59624a6` from
`codex/ui-visual-review` are integrated with the shell/i18n work in this checkout,
`codex/prototype-administration-pages`. Both branches started at `2d1b714e`; the
first full-shell build omitted that lateral branch and regressed the installed UI.
The integrated source retains bundled Manrope, shared type roles, catalog/context
alignment, wordmark sizes and composer autosizing alongside full/embedded and EN/IT.
Local Docker was updated at 12:54 UTC on 2026-09-13: frontend image `605a6e6bba68`,
healthy; Core and all data-service containers were unchanged. Mac remains full/en.
The immediate pre-merge frontend is retained as
`thothii-frontend:before-visual-shell-merge-20260913`.
Verification: 755 frontend tests, 11 Playwright visual/interaction scenarios at five
widths, translation-catalog checks, typecheck and production build. See
`docs/reports/2026-09-13-visual-shell-integration.md` for delivery, provenance and
rollback details. The separate visual-review worktree and its rollback images are
retained. No merge to `main`, push or production deployment is part of this delivery.
Gitea #28–#31 follow-up is implemented in this checkout: Workspace's four tabs,
Database list-first entry without the preparation footer, full-height Pi instructions
with installation-host OS selection, and short Admin navigation labels. Core and
session behavior are retained. At the owner's request, these changes were rebuilt into
local Docker on 2026-09-12 at 18:31 UTC. Core/frontend are healthy and the UI at
`http://127.0.0.1:8080` serves the updated bundle. The host projection now supplies
`THT_HOST_PLATFORM=darwin`, so Pi selects macOS rather than the container's Linux OS.
Persistent dependency containers and volumes were unchanged. Rollback images are tagged
`thothii-core:before-admin-28-31-20260912` and
`thothii-frontend:before-admin-28-31-20260912`. See
`docs/reports/2026-09-12-admin-issues-28-31.md` for verification and deployment details.
The latest context-shelf A and five Administration pages are implemented locally.
At the owner's request, local Docker project `thothii-18998cca7b0a` was rebuilt and
its core/frontend recreated on 2026-09-12. The real UI at `http://127.0.0.1:8080`
serves shelf A; both services and their existing dependencies are healthy.
Runtime model projections were regenerated as schema v2 with the single
`zai/glm-5.3` interaction default. Persistent services/volumes were not recreated.
The local launcher `/private/tmp/thothii-memory-preview.sh` now adds
`/private/tmp/thothii-context-a.compose.yaml` last, building from this checkout
instead of the earlier Memory worktree. Previous images are retained under
`thothii-core:before-context-a-20260912` and
`thothii-frontend:before-context-a-20260912`. Remote server deployment remains pending.
Core/session behavior is retained; one independently remembered workspace/model
pair controls both Core and Admin. Unsaved Admin changes block navigation and
context changes; operation activity locks the selectors. See
`docs/reports/2026-09-12-context-shelf-a-implementation.md` for verification and the
remaining server/Omics integration gate. Prototype alternatives remain untouched.
ThothII is a human-in-the-loop datamart builder with three independently built layers:
```text
frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness → DWH (read-only)
```
The harness owns the deterministic eight-phase NL→SQL workflow and all session persistence.
The backend remains a process/RPC/SSE bridge for sessions and now also owns an isolated PostgreSQL
metadata catalog for administrative database configuration. The frontend renders the review gates
and keeps the live transcript in memory. See
`docs/architecture/components.md` for the detailed component and data-flow map.
## Memory M1–M3, Evidence E1–E3 and joint workflow repair X1 implemented
Browser authorization now derives permissions from validated session roles using the current
catalog. This fixes Memory/Evidence navigation remaining disabled for administrators whose
remembered login predates those permissions; refreshing the page loads the updated permissions.
The owner requested two independent administration projects: Memory management and Evidence
management. Their navigation entries must sit immediately after Database management, as peers;
neither page belongs to Database management. Both require complete browsing, filtering, and CRUD
without an active core session. Shared requirements and the two project briefs are linked from
`docs/plans/2026-09-08-memory-evidence-administration.md`. Memory administration is implemented;
Evidence administration is implemented through external file editing and explicit consolidation.
Evidence editing requires an explicit evolution of the current authoring/publication contract.
The M1 specification is at `docs/plans/2026-09-08-memory-m1-spec.md` and published as
[M1 — Archivio autorevole e amministrazione delle Memory Card](https://git.tylconsulting.it/mptyl/ThothII/issues/27)
with the `ready-for-agent` and `enhancement` labels.
It covers the authoritative store, administrative CRUD, projection recovery, and current
Memory/exemplar producer integration. Its accepted test boundaries are the public harness
service, Fastify APIs, and the AppShell page, joined by a focused real-stack browser path.
The owner confirmed those boundaries and authorized publication on 2026-09-08.
M1 is implemented locally: PostgreSQL authority in `thoth_memory`, versioned installation
migrations, admin CRUD for all four families, structured dependencies and links, explicit
projection recovery, verified recall and current workflow producers. The page requires no
active session or DWH binding. The existing `catalog-migrate` preparation service now runs
Memory migrations as well. Existing installations need that preparation before using M1;
the initial implementation did not deploy or migrate the owner's stacks.
The integrated browser check passed with real authentication, Fastify, ThtRunner, harness,
PostgreSQL and Qdrant; embeddings were deterministic and unrelated Pi/session activity used
test fixtures. See `docs/plans/2026-09-08-memory-m1-validation.md` for results and commands.
M2 is implemented locally: Memory dense/BM25 fusion, physical and business scope filters,
bounded outgoing-link expansion and joint ranking, all resolved against current PostgreSQL
authority. Migration `002_hybrid_projection.sql` makes old dense projections pending until
explicit retry/rebuild; Reference remains separate. The real retrieval check uses the configured
`qwen3-embedding:0.6b` model, a separate Ollama process with a read-only model-volume mount,
and isolated PostgreSQL/Qdrant resources. See `docs/plans/2026-09-09-memory-m2-validation.md`.
M3 is implemented: editable F8 summary grounded in effective approved decisions, explicit
updates with concurrent-edit protection, selected-card/link transactions and durable review
receipts. Finalization no longer saves exemplars implicitly. SQL rules and explained errors
are consulted in the existing F4/F6/F7 gates. Successful Catalog physical synchronization
performs dependency cleanup, preserving the original removals for recovery across restarts.
Migration `003_review_receipts.sql` is required. Validation includes a real GLM 5.3 generation
case against synthetic PostgreSQL data; see `docs/plans/2026-09-09-memory-m3-validation.md`.
Evidence administration and the joint X1 conflict-repair increment are now implemented.
The same local preview was subsequently rebuilt with M3 and migration 003 applied.
Core and frontend now include the final Memory review and physical dependency cleanup.
On 2026-09-09, at the owner's request, the local PSD Docker installation
`thothii-18998cca7b0a` was updated from this worktree. Core/frontend images were rebuilt,
Catalog and Memory migrations completed, and all five services became healthy. The UI is at
`http://127.0.0.1:8080`, using the existing local authentication and persistent volumes.
The `psd-clinical` authoritative Memory archive is initially empty (no legacy import).
The existing installation configuration remains in `/Users/mp/projects/ThothII/deploy/psd/`;
`/private/tmp/thothii-memory-preview.sh` invokes its Compose files with a final build-context
and migration-command override from this worktree. A future build from the main checkout
will use that checkout's code, so retain the worktree override until the changes are integrated.
Memory revision history and compatibility with existing development sessions are not requirements.
The agreed Memory scope includes reusable domain clarifications, SQL construction rules, solved
questions, and explained, approved mistakes to avoid. This extends the current runtime's
`concept_clarified`-only reusable Memory contract. The owner also approved an editable final summary
for proposed additions/updates and Memory consumption in the relevant existing review gates.
Further agreed behavior includes persistent corrections for Memory/Evidence conflicts through
explicit choices, deletion of Memory with invalid dependencies after successful physical schema
synchronization, and bounded functional/regression tests instead of a general quality benchmark.
In-house Memory evolution is approved, including hybrid Qdrant retrieval and explicit card links
traversed in core, without a dedicated graph database. Both capabilities belong to the current
scope. PostgreSQL is the agreed authority for Memory Cards, links, and schema dependencies;
Qdrant is a rebuildable index. Links are reviewed with cards and editable in Administration;
deleting a card removes its incident links while preserving the other cards. The accepted,
implemented architecture is recorded in
`docs/adr/0018-use-postgres-for-memory-and-qdrant-for-retrieval.md`.
For Evidence administration, Q12 rejects a separate editorial publication workflow.
The final R0 direction uses external editors and a manual consolidation command that
validates structure, reports required corrections, updates derived metadata, and
activates the local Evidence index. The operator then checks the diff and runs Git
commit/push manually. No watcher, automated Git, or web editor is required.
The owner's later simplification
instruction removes the requirement to support administrative edits while core work
is in progress. Deliberate corrections made by the workflow itself remain supported.
The context specialist writes Evidence drafts independently of the installation and
without PostgreSQL access; the system refines them and stores them locally for use
and maintenance. The revised Q11 recommendation separates external drafts from a
durable local canonical file archive, removing automatic commit/push from CRUD.
The owner has now accepted local files and requires discoverable, editable Markdown
for nontechnical domain specialists, with no JSONL management surface. Editors on
Mac/PC or vim/nano on the server are the chosen R0 editing surface. Evidence
management keeps browsing/filtering/detail and clearly identifies the persistent
working tree, each Markdown file's actual host path, and the manual commands.
E1 implements editable Curated Evidence v4, deterministic legacy conversion, persistent
local files and a consolidation API with immutable candidates, manual provenance,
deletion records and recoverable activation. E2 connects its installed command, runtime source
selection and administration page. The operator completes Git steps manually.
All 35 PSD units were converted on an isolated copy with identical IDs and typed content.
Tests exercise visible edits through normalization, indexing and recall with real Qdrant;
see `docs/plans/2026-09-09-evidence-e1-validation.md` and
`docs/contracts/curated-evidence-v4.md`. E2 is now running on the local Docker preview:
all 35 PSD units were converted and indexed through the installed command. The editable
host archive is `/Users/mp/projects/ThothII/deploy/psd/evidence-registry/repo/psd-clinical/evidence`.
The original registry-volume checkout and original author repository were retained.
The installation descriptor now includes `workspace-bindings.yaml` and `evidence-host.yaml`
so native maintenance uses the same volumes and checkout as the preview. Descriptor backup:
`/private/tmp/thothii-installation-before-e2.yaml`; previous native binary: `/private/tmp/tht-before-e2`.
The launcher remains `bash /private/tmp/thothii-memory-preview.sh`; its worktree build override
is still required until integration. Runtime lease filenames now include rendered bytes so
upgrading the Evidence renderer does not collide with old immutable configs; Catalog
input fingerprints and readiness are unchanged. See `docs/plans/2026-09-09-evidence-e2-validation.md`.
E3 adds explicit source acquisition/refinement and durable comparisons in Evidence management.
Local drafts go in `evidence/incoming/`; original local documents and configured HTTP/S3
sources are reacquired only by **Import or refresh sources** or installed
`tht workspace evidence refresh --workspace <id>`. Keep/replace decisions, including the
native `workspace evidence decide` command, activate through the existing archive/index
path and have durable retry state. Acquired source versions and remote provenance stay
local; old documentary lineage can coexist with current manual declarations.
The installed preview's 35 PSD sources were verified unchanged with no new proposals.
The previous native CLI is backed up at `/private/tmp/tht-before-e3`.
See `docs/plans/2026-09-09-evidence-e3-validation.md` for tests and validation limits.
X1 adds `reviewer_archive_repair`: closed alternatives with complete before/after content,
explicit rejection/reformulation, admin-only application, durable session receipts and
retry of saved but inactive corrections. The coordinator uses the canonical Memory and
Evidence services; phase approval remains separate. Migration `004_archive_repairs.sql`
is required. See `docs/contracts/archive-repair.md` for authorization and recovery limits.
X1 validation is recorded in `docs/plans/2026-09-09-archive-repair-x1-validation.md`:
both archives were corrected and retrieved through the actual CLI with real PostgreSQL
and Qdrant; desktop/mobile widget behavior and permission failures were verified separately.
The local preview images include X1 and migration 004 is applied. Follow-up technical
acceptance passed with configured GLM 5.3 generating closed conflict alternatives and
the chosen Memory correction persisted and retrieved. An authenticated browser test
also verified both administration pages, navigation order, Evidence filtering, workspace
404s, desktop/mobile layout, and Evidence survival after Memory deletion. All approved
technical increments/checks are complete. A real PSD domain-conflict session remains
the reviewer's semantic acceptance check; automated cases did not modify PSD knowledge.
The joint browser inspection also fixed mobile archive navigation: below 768px,
Memory and Evidence keep the full content width and open navigation in the shared
accessible dialog. Desktop retains its sidebar; the local frontend image includes this fix.
The owner accepted the remaining simplifications and requested explicit clarification
of the core format change and the simple terminal-based Git check. E1 must adapt
the Evidence parser, renderer, authoring, validation, and normalization; convert
existing files and reindex; and verify that visible edits reach core consumption.
Preserve the internal typed model where possible. This precedes the administrative
page and is not merely a presentation change. Human inspection uses normal Git
status and optional line-level diff commands; no custom diff viewer or mandatory
double review is required.
The suggestion of making PostgreSQL the Evidence authority was withdrawn after this
clarification; it was never implemented or accepted as a replacement for Q11.
Q13 keeps manual corrections active when updated sources contradict them, until an
administrator resolves the comparison; deleted Evidence must not be regenerated
automatically. Q14 allows direct manual creation and records a manual declaration
as the current source, preserving any original document as distinct provenance.
Q15 refreshes external sources only on explicit administrator request; normal saves
and lookups do not reacquire them. These decisions, now implemented through E1–E3, are recorded in
`docs/adr/0019-author-evidence-in-app-with-automatic-activation.md`.
The decision-by-decision review is recorded in
`docs/plans/2026-09-08-memory-evidence-simplification-review.md`; it distinguishes the
new constraints from the revised technical recommendations. It also specifies a
sequential save with minimal durable retry state and direct Memory cleanup after
successful schema synchronization, without new queues or event infrastructure.
The delivery order remains Memory, Evidence, and persistent conflict repair between
both modules. Local curated Evidence must survive preprocessing Clear and be backed
up as primary data; the Qdrant projection remains rebuildable.
No runtime gate change or administrative page is implemented yet.
## Evidence restructuring — accepted
The evidence restructuring and PSD migration completed real acceptance on 2026-08-25.
- The curated PSD revision contains 35 approved Evidence units and 60 review items.
- The PSD authoring repository publishes all 35 units using Curated unit schema v3. Its table-free
presentation uses hidden canonical metadata, wrapping Markdown scope lists, list-based enum
values, and collapsed technical provenance. Long domain rules now have a deterministic
human-readable presentation while retaining their exact canonical text for vector ingestion.
`tht evidence migrate <workspace-root>` performs the deterministic v1/v2 upgrade and older-v3
presentation rewrite without model calls. The structured-rule PSD rewrite is currently local and
pending commit/publication.
- The accepted snapshot is
`psd-clinical-675990d90eae51da6f2bd51b1ae2609f245772ef-snapshot`.
- The active generation is `gen:f968b3bd7a553dbfef3cf47093698f2bc7f95f11`.
- Retrieval acceptance reached 20/20 Hit@10.
- A real session, `20301df7-cad7-403d-a4c1-9f35c9d07b66`, completed F1–F8 with five
receipts, three CTEs, and a final result of 78 patients.
- The durable acceptance record is
`docs/testing/evidence/evidence-restructuring-psd-acceptance-2026-08-25.md`.
The canonical authoring, validation, publication, materialization, and preprocessing flow is
documented in `docs/evidence.md`. The governing contracts are
`docs/contracts/workspace-evidence-v3.md` and
`docs/contracts/workspace-preprocessing-cli.md`.
The incremental server procedure for the `260906-preprocessing-complete` release is
`docs/operations/server-handoff-260906-preprocessing-complete.md`.
## Workspace preprocessing and configuration
The native host CLI `tht` is the operator surface. Workspace preprocessing runs through:
```sh
tht --installation /absolute/path/thothii-installation.yaml \
workspace preprocess run --workspace <workspace-id>
```
This complete one-shot command uses the profile-gated `workspace-maintenance` service. Partial DWH,
schema, Evidence, and vector mutation commands are retired.
The right Administration sidebar invokes that same operation for the selected workspace. It shows
only current readiness or the latest bounded failure diagnostic; there is no preprocessing history.
Known non-ready state disables **Session** when it would start a new question,
while backend admission remains authoritative.
The same control exposes an inline-confirmed **Clear** action to remove replaceable reference
vectors, LSH, corpus, and checkpoints while preserving the separate Memory collection. The host CLI
equivalent is `workspace preprocess clear`.
Each workspace now uses `<workspace>-reference` for Schema, relationships, and Evidence and
`<workspace>-memory` for `memory` and `solved_question`. Clear and preprocessing own only the former.
LSH ownership additionally binds the Catalog database ID and Metadata Content Revision, so derived
values cannot be reused across database identities or Catalog revisions.
Workspace descriptors use schema v4 and contain only workspace identity and optional Evidence.
PostgreSQL Metadata Catalog owns database identity, binding, schema, descriptions, sensitivity, and
relationships; model, provider, embedding, and vector-store configuration is installation-owned. For
PSD, workspace content and runtime roots point to the
separate uncommitted repository `/Users/mp/projects/tht-workspace-psd`. Secrets remain outside
Git and are supplied only through installation-local protected files.
## Installation Model Catalog
`thothii-installation.yaml` schema version 2 is the only operator-authored source for session,
metadata-generation, and embedding models. The host `tht` lifecycle validates `modelCatalog` and
regenerates the backend catalog, Pi `models.json`/`settings.json`, and Compose override under the
installation-local `generated/` directory. Those projections are replaceable runtime adapters:
they are not edited, backed up, or treated as configuration.
Core and Administration now share one canonical `modelCatalog.defaults.interaction` per installation,
independent of workspace. Runtime catalog schema v2 contains only `defaultInteraction`; apply the host,
backend, and regenerated projections together. Equal legacy defaults normalize on read; conflicting
ones require an explicit operator choice. With Admin AI configured, selectable models are the
intersection of the Pi and LiteLLM adapters; Core-only installations remain supported without Admin AI.
The existing Core and Database controls share the operational selection. Explicit model choices are
remembered per authenticated user/application mount in this browser, not in installation settings.
Resume uses that selection/default while preserving the historical workspace/revision and manifest.
Every model must be manually exercised in both Core and Admin as documented in
`docs/general/pi-configuration.md`; validation is not a live model certification.
These changes and shelf A are now deployed to local Docker; remote deployment remains pending.
PSD DeepSeek Pro/Flash now share canonical `deepseek/...` identities across native Pi and LiteLLM,
using the existing `DEEPSEEK_API_KEY` bundle entry. Its value was confirmed identical to the working
Pi key without exposing it; no secret files were changed. The duplicate `deepseek-metadata` descriptor
is removed locally and the tracked example uses the shared provider. `secret_env` overrides legacy
Pi auth only inside temporary runtime snapshots and fails closed if the bundle key is missing.
The original auth store/history and `zai/glm-5.3` default are preserved. Local Docker projections
and core/frontend were updated together on 2026-09-12. Regenerate projections with
the matching release for the separate server deployment.
Provider authentication declares
one explicit mode (`secret_env`, `pi_auth`, or `none`); `secret_env` names a protected bundle key.
The backend settings store now owns only the selected workspace and thinking level. Existing v1
installations use the explicit catalog migration command; schema-v3 workspace descriptors are
converted deterministically in their curator-owned repository before commit. Strict runtime loading
does not silently infer or merge legacy sources. ADR 0013 and
`docs/plans/2026-09-02-installation-model-catalog.md` record the decision and implementation.
## Database management
The database, table, and authoritative physical-schema catalog slices are implemented. Database
Management now opens the Fleet Ledger presentation by default inside `AppShell`, lists every YAML
workspace, creates at most one PostgreSQL database configuration per workspace, edits direct
PostgreSQL, REST API, or SSH-tunnel installation bindings, replaces write-only encrypted secrets,
and tests supported connector bindings. The surface keeps one responsive AG Grid visible at a time:
databases lead to tables, tables lead to columns, and relationships are a sibling database view.
Parent navigation remains explicit through the breadcrumb and emphasized back control.
Selection-scoped operations use one action selector plus an explicit **Run** control; ineligible
actions remain visible with their disabled reason, while row-scoped actions stay in the pinned final
column. The KPI strip reads installation-wide or selected-database aggregates from
`GET /catalog/metrics`. Database configuration, metadata editors, synchronization history,
description history, and sensitive-field review/history use the production APIs in right-side
drawers rather than prototype fixtures; closing a history drawer does not stop its background run.
Sensitive-field review is now driven by the versioned local `sensitivity-v4` policy, not by a
catalog model. The backend reads selected source tables through read-only, database-specific
adapters and makes every `sensitive | non_sensitive` draft decision in the TypeScript
`SensitivityClassifier`. A single validated match protects the column. Tables up to 1,000 rows are
fully scanned; larger tables use breadth-first 300, 1,000, and text-only 3,000-value targets, with a
five-second limit per source query and no global request deadline. Source failures fail the run
instead of yielding `unknown`; coverage remains visible separately from the proposal. Draft
assessments remain transient until an administrator explicitly saves them. Optional GLiNER2
evidence is CPU-only, offline, opt-in, and never replaces the deterministic decision point; see
`docs/operations/sensitivity-analysis.md`. The earlier v1 PSD shadow comparison kept NER disabled by
default; see `docs/reports/2026-09-02-psd-sensitivity-shadow.md`. The v2 comparison completed all
2,275 columns: CPU NER added 18 sensitive proposals and increased warm runtime from 50.1 to 61.3
seconds; see `docs/reports/2026-09-03-psd-progressive-sensitivity-shadow.md`.
Version 4 excludes declared `bigint` primary-key columns and conventionally named `pk bigint`
columns before source inspection, reporting both as non-informative structural identifiers while
distinguishing declared constraints from inferred roles.
Physical membership, source
comments, column types/default/nullability/PK positions, and constraint-level ordered FK pairs are
projections of the external schema. They cannot be created, renamed, or structurally edited by
hand, but administrators can explicitly clear catalog tables, columns, or relationships without
touching the source database, binding, configuration, or secrets. Table deletion cascades through
columns and relationships; table-scoped relationship cleanup includes incoming and outgoing
relationships. Curated and generated descriptions are editable; generated descriptions start null
and Database Management can generate or consolidate them for selected tables, selected columns,
all targets, or only targets whose Generated Description is missing.
Relationship Management is now reachable directly from each configured Fleet database. One
Relationship Map shows read-only Physical Relationships together with Generated and Manual Logical
Relationships, with Active, Excluded, and All filters. Administrators can add a single-column
relationship, run deterministic name/PK/type inference, exclude or restore a logical relationship,
or delete it permanently. Exclusion retains a tombstone that a rebuild cannot reactivate; permanent
deletion allows a later rebuild to infer the same endpoints again. Inference uses no LLM, embedding,
or source values. It supports normalized table-qualified names, unique non-generic PK names,
composite-PK source columns, and the `*time_key -> dim_time.<single PK>` warehouse convention while
ignoring bare generic names. Explicit table/column metadata cleanup remains a destructive boundary: it removes
the attached logical relationships and exclusions and requires a full schema synchronization before
inference or runtime publication can continue.
The previous Database Management renderer remains a temporary comparison fallback for development
and staging only: `?db-ui=legacy` is honored in Vite development or when
`VITE_DB_MANAGEMENT_LEGACY=true`; it is not a production presentation. The standalone Fleet Ledger
prototype on port `5173` also remains temporary until owner acceptance of the integrated surface,
after which both migration aids can be removed.
Schema refresh is one durable asynchronous engine with database-table, selected-table-column,
relationship, and full-database actions. Database-level menus expose only the table, relationship,
and full scopes; selecting tables exposes column synchronization plus manual column and relationship cleanup for that subset. Database selections
also expose manual table and relationship cleanup. Cleanup selections are atomic and share the
one-active-operation-per-database exclusion with synchronization. Runs have leases and
restart recovery, atomic apply, destructive-diff confirmation with re-scan, cancellation before
apply, retained history, and a live SSE log with polling fallback. Null metadata renders blank
rather than as a placeholder.
Direct PostgreSQL and strict known-host-verified OpenSSH use `pg_catalog`. REST bindings use the
typed full-snapshot `POST /rpc/schema_snapshot` contract when available. Servers such as the
current PSD endpoint that exposes only `POST /rpc/run_query` use one catalog-owned read-only query
to return the exact same strict v1 snapshot in a single round trip. Both paths remain fail-closed:
an absent capability, query error, partial result, or invalid snapshot applies no catalog changes.
SSH is not yet enabled for NL→SQL session runtime.
The catalog runs in the internal `catalog-db` PostgreSQL service. Kysely migrations are an explicit
one-shot `catalog-migrate` operation; `scripts/run-stack.sh` runs it before local startup. Runtime
sessions consume an immutable Catalog JSON snapshot tied to the runtime-config lease. It contains
the tables, columns, effective descriptions, sensitivity flags, and active relationships used by
the harness; PostgreSQL is the exclusive runtime authority for database metadata. Authored
workspace YAML remains limited to workspace identity and optional Evidence configuration. The
accepted design is recorded in ADR 0016 and the contracts under `docs/contracts/`.
Semantic aliases, value descriptions, synonyms, and concepts remain deferred to their dedicated
slices.
AI Description Generation uses the catalog's human-owned Sensitive Data Flag. The flag defaults to
`false`, including for newly synchronized columns. An administrator may request a local sensitivity
analysis for one selected database, selected tables, or selected columns. One deterministic
TypeScript classifier combines metadata, bounded source-content rules, and optional CPU-only NER;
no generative model decides the result. Its `sensitive` or `non_sensitive` assessments remain an
unsaved draft until the human reviews and saves any chosen flag changes, including a downgrade to
non-sensitive. Coverage is reported separately; interrupted history may count unprocessed columns.
Each started analysis records a separate Sensitivity Analysis Run with aggregate counters and safe
ordered events. The progress drawer opens before the synchronous request completes, polls the run,
and displays sanitized source-scan and local-NER phase/batch activity while classification is in
progress. This operational history never stores per-column assessments, source values,
matched spans, prompts, or free-form diagnostics. Saving a sensitive decision persists a sanitized
Sensitivity Reason as column Catalog Metadata alongside the human-owned flag; clearing the flag
clears that reason. Reloading still discards an unsaved review draft.
For unprotected columns, up to five source rows and five representative non-null values may be sent
transiently to the configured model provider. Protected columns are omitted from source reads and
replaced in the prompt by deterministic plausible values derived only from their metadata. Existing
descriptions are not regenerated when a flag changes.
The accepted AI-description design is recorded in
`docs/plans/2026-08-28-ai-catalog-description-generation.md`, with the formal specification in the
adjacent `-spec.md` document and Gitea issue #4. Gitea issues #5–#11 deliver the implementation.
The runtime deliberately keeps ThothAI's simple operating model: one installation-wide sequential
run owned by the backend, one short-lived Python/LiteLLM completion helper per request, and
persistence limited to the run, its safe ordered text events, and each Generated Description as
soon as it succeeds. The helper performs at most one provider retry and never falls back to another
model. Stop terminates the current helper and retains prior results; three consecutive exhausted
technical batches fail the run. Startup marks stale queued/running work interrupted, and Unlock is
available only when no local start, worker, or helper is live. Runs remain inspectable through a
live SSE log with ordered polling fallback; there is no automatic resume or user-facing generation
CLI. ADRs 0009–0010 record the runtime and source-sampling decisions.
The Installation Model Catalog accepts the protected `DEEPSEEK_API_KEY` and `ZAI_API_KEY`
references for metadata-generation providers.
It also accepts a model with no secret reference only when its OpenAI-compatible endpoint is
explicit; this covers the VPN-only AritmoLab Qwen 3.6 server without creating a fake operator
credential. The Python client supplies only its fixed non-secret compatibility placeholder.
The AritmoLab entry also sets `disableThinking: true`, mapped to the endpoint's chat-template flag,
because its default reasoning prose would violate the worker's exact JSON response contract.
Logical relationship integration with core schema-linking is complete: session creation and resume
materialize the active physical/generated/manual map, retrieval-pack generation and Pi receive the
same runtime config, and snapshot validation fails closed on a declared missing, invalid, or orphaned
endpoint. Broader publication of other Catalog metadata to schema-linking remains a separate future
slice.
**Deferred follow-up — Sensitive Data Policy in schema-linking.** The policy is first delivered
and tested in catalog description generation. Its enforcement for core schema-linking remains
out of scope until the current tickets are closed and the owner has completed the acceptance test.
At that gate, resume the design: `tht` must receive a read-only projection of the current Sensitive
Data Flags and exclude values from columns marked sensitive from every LSH result before it is
given to Pi. Do not start this integration before the owner gives final approval after that test.
## Active deployment work and manual gates
### PSD server deployment program
The approved design and executable entry point are:
- `docs/plans/2026-08-20-psd-server-deployment-program-design.md`
- `docs/plans/2026-08-20-psd-server-deployment-program.md`
- `docs/plans/2026-08-20-psd-server-survey.md`
- `docs/plans/2026-08-20-psd-server-project-a-standalone.md`
- `docs/plans/2026-08-20-psd-server-project-b-authentik.md`
Last recorded state:
- survey: `SURVEY_NO_GO`;
- Project A: `BLOCKED_BY_SURVEY_AND_MUTATION_GATE`;
- Project B: `BLOCKED_BY_PROJECT_A_AND_PRE_B_GATE`.
The deployment is a clean replacement: legacy sessions, indexes, and application configuration
are not migration inputs. The existing stack remains intact until its documented mutation and
rollback gates are explicitly approved. Shared Omics/LocalLLM networks, ETL Evidence, DWH,
`dwh-auth`, Supabase, Authentik, Superset, and Aritmolab are outside cleanup scope.
Human acceptance guides and sanitized report templates live under `docs/testing/` and
`docs/testing/evidence/`. The remediation checklist is
`docs/operations/psd-server-survey-remediation-checklist.md`.
### Authentication
The local/OIDC authentication remediation passed its automated review on 2026-08-18. Release and
PSD mutation gates remain governed by:
- `docs/architecture/authentication.md`;
- `docs/plans/2026-08-18-thothii-authentication-acceptance-and-psd-deployment.md`;
- `docs/operations/psd-dwh-auth-rollout.md`;
- `docs/testing/authentication-manual-acceptance.md`.
Do not infer authorization for server, Nginx, Authentik, database, credential, or cutover changes
from an automated PASS.
## Verification status
- The P1.1 workspace-directory registry and P2–P6 preprocessing workstreams are implemented and
have automated coverage.
- Evidence restructuring has a real PSD acceptance PASS as recorded above.
- AI Description Generation has automated coverage across installation setup, model selection,
generation/consolidation scopes, bounded sampling, cancellation/recovery, history, SSE/polling,
and the LiteLLM helper boundary.
- L2 tests requiring real providers or remote databases remain opt-in.
- Server deployment, release, and owner-operated acceptance steps remain pending wherever the
referenced runbooks require explicit approval.
Run the layer-specific checks documented in `AGENTS.md`. For release-sensitive changes, also run
the repository contract scripts in `scripts/` and build the MkDocs site.
## Operational invariants
- `tht`'s `-c`/`--config` option follows the subcommand; it is not a global option.
- `--json` commands write pristine JSON to stdout.
- Persisted phase documents and the decision ledger are the source of session truth; chat is not.
- UI chrome is English; workspace document content retains the workspace language.
- The backend refuses resume for finalized or archived sessions.
- A resume must send `/riprendi-sessione <id>`; a new session must send `/nuova-domanda`.
- DWH access is read-only.
+21 -59
View File
@@ -25,69 +25,31 @@ For the clone-based manual standalone installation test on macOS, Windows, and L
[Italian procedure](docs/install/standalone-manual-it.md) or the [Italian procedure](docs/install/standalone-manual-it.md) or the
[English procedure](docs/install/standalone-manual-en.md). [English procedure](docs/install/standalone-manual-en.md).
## Docker Compose: local startup The [public manual](https://git.tylconsulting.it/thothii-docs/) covers the product,
installation, use and administration. Developer architecture, contracts, ADRs, tests,
plans and release records remain in this repository but are excluded from MkDocs
pages and search. This is an editorial boundary, not an access restriction on the
public repository. See the [documentation cleanup review](docs/maintenance/2026-09-15-documentation-cleanup.md)
for the executed consolidation and the inventory of historical sources retained in Git.
Requirements: Docker Engine with Compose v2. The mandatory stack is `frontend`, `core`, ## Docker Compose and installation
`catalog-db`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`. DWH and LLM remain external,
configurable endpoints—even when they are co-located with ThothII.
From a fresh clone, run these commands from the repository root: For a fresh installation, follow the guided terminal procedure in
[Italian](docs/install/standalone-manual-it.md) or
[English](docs/install/standalone-manual-en.md). The single
`tht setup --complete` command validates protected files, builds the images, runs
Catalog migration, starts the stack and imports the configured workspace repository.
There is no graphical installer or native launcher.
```sh The mandatory stack includes frontend, core, PostgreSQL catalog, Qdrant, Ollama
cp deploy/env/local.env.example deploy/env/local.env and the embedding initializer. DWH and LLM endpoints remain external dependencies.
# Copy docs/install/examples/thothii-installation.local.yaml to a protected operator path, Pi is included in the core image. Credentials and certificates belong in protected
# replace its placeholders, chmod it 600, and set that exact THT_INSTALLATION_CONFIG_SOURCE. installation-local files, never in the workspace repository.
# Edit deploy/env/local.env, including PI_AUTH_FILE, THT_SECRETS_FILE, and external endpoints.
./scripts/run-stack.sh
```
The low-level stack script builds the core, starts `catalog-db`, runs the explicit one-shot Kysely migrations, For developer topology, overlays and lifecycle details, see the internal
then runs the base+local stack in the foreground. Migrations never run implicitly in backend [Compose reference](docs/operations/compose-reference.md). Ordinary stop/down keeps
startup. The core image contains its Pi runtime; no host `pi` executable is used. For a server persistent data; removing volumes is destructive and is not an upgrade step.
installation, build the image, start the catalog, and run the same migration service before the Process health is distinct from external dependency checks performed by doctor.
application rollout:
```sh
cp deploy/env/server.env.example deploy/env/server.env
# Prepare a mode-600 thothii-installation.yaml from the server example and set its exact
# path as THT_INSTALLATION_CONFIG_SOURCE. Edit all remaining storage/secret/endpoint paths.
sudo scripts/prepare-server-pi-state.sh /srv/thothii/pi-state 10001 10001
docker compose --env-file deploy/env/server.env \
-f compose.yaml -f deploy/compose.server.yaml \
-f deploy/compose.session-server.yaml.example build core
docker compose --env-file deploy/env/server.env \
-f compose.yaml -f deploy/compose.server.yaml \
-f deploy/compose.session-server.yaml.example up -d catalog-db
docker compose --env-file deploy/env/server.env \
-f compose.yaml -f deploy/compose.server.yaml \
-f deploy/compose.session-server.yaml.example run --rm catalog-migrate
docker compose --env-file deploy/env/server.env \
-f compose.yaml -f deploy/compose.server.yaml \
-f deploy/compose.session-server.yaml.example up --build -d
```
The initializer is required for an empty or restored server Pi-state bind. It atomically creates
the three regular targets hidden below the writable parent bind; protected Pi auth and tracked
model/settings sources remain separate read-only mounts. See the server manual before substituting
a root other than `/srv/thothii/pi-state`.
Workspace descriptors come from the Git remote configured by `THT_WORKSPACE_GIT_REMOTE`; their
runtime endpoint and secret bindings remain installation-local. Open
<http://127.0.0.1:8080> (set `THOTH_HTTP_PORT` in `deploy/env/local.env` to choose another
loopback port).
Credentials and certificates are local protected files. Do not put them in environment examples,
workspace YAML, URLs, or Compose interpolation values.
Application state is split across the named `settings`, `pi-state`, `workspace-registry`,
`sessions`, `qdrant-data`, and `embedding-models` volumes. `docker compose down` keeps them.
`qdrant-data` is a derived but persistent index store; `embedding-models` is an Ollama model
cache for `qwen3-embedding:0.6b` with fixed `1024`-dimension embeddings. Only an explicit destructive command such as `docker compose
down --volumes` removes them.
The frontend depends on the core health check and proxies `/health` and `/api/*` to it. The
application health endpoint intentionally checks process readiness only; external dependency
diagnostics are exposed by `tht doctor` and do not prevent the UI from starting.
## Git-backed workspace repository ## Git-backed workspace repository
+609
View File
@@ -6,11 +6,13 @@
"": { "": {
"name": "thothii-backend", "name": "thothii-backend",
"dependencies": { "dependencies": {
"@aws-sdk/client-s3": "3.1141.0",
"@fastify/cookie": "11.1.2", "@fastify/cookie": "11.1.2",
"@fastify/cors": "^11.2.0", "@fastify/cors": "^11.2.0",
"@fastify/rate-limit": "11.2.0", "@fastify/rate-limit": "11.2.0",
"@types/pg": "^8.20.3", "@types/pg": "^8.20.3",
"fastify": "^5.0.0", "fastify": "^5.0.0",
"ipaddr.js": "2.4.0",
"kysely": "^0.29.5", "kysely": "^0.29.5",
"libphonenumber-js": "1.13.12", "libphonenumber-js": "1.13.12",
"openid-client": "6.8.5", "openid-client": "6.8.5",
@@ -23,11 +25,320 @@
"@testcontainers/postgresql": "^12.1.0", "@testcontainers/postgresql": "^12.1.0",
"@types/node": "24.13.3", "@types/node": "24.13.3",
"@types/validator": "13.15.10", "@types/validator": "13.15.10",
"bun": "1.4.2",
"tsx": "^4.19.0", "tsx": "^4.19.0",
"typescript": "^5.6.0", "typescript": "^5.6.0",
"vitest": "^2.1.0" "vitest": "^2.1.0"
} }
}, },
"node_modules/@aws-sdk/checksums": {
"version": "3.1001.1",
"resolved": "https://registry.npmjs.org/@aws-sdk/checksums/-/checksums-3.1001.1.tgz",
"integrity": "sha512-x12Q17KYlJAd3nKf8LV5LV0vt8sh8/6YfQLGPtrGnQf/tW4jqxPGq5GPpuVitpQYM3eUR4XB7CbxZf751NMbLw==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/client-s3": {
"version": "3.1141.0",
"resolved": "https://registry.npmjs.org/@aws-sdk/client-s3/-/client-s3-3.1141.0.tgz",
"integrity": "sha512-uOVH37xGLenAdJkCPCin/JJG2PgWrFcSsDnQ9+C9Zq8N9Oalo5ol4xmn5fG28iWAlA/b/9boQZgHbMh+UsIhcg==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/checksums": "^3.1001.1",
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/credential-provider-node": "^3.972.84",
"@aws-sdk/middleware-sdk-s3": "^3.972.77",
"@aws-sdk/signature-v4-multi-region": "^3.996.47",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/fetch-http-handler": "^5.8.0",
"@smithy/node-http-handler": "^4.12.1",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/core": {
"version": "3.978.1",
"resolved": "https://registry.npmjs.org/@aws-sdk/core/-/core-3.978.1.tgz",
"integrity": "sha512-LbY9aGsEiznDWmUc30Nwv3aIX/+dbwTx8KfS0yOC3NPYMO+O91e6jkT1azf34FwjOndq8/Q+RcVVZz5xnerwdg==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/types": "^3.974.6",
"@aws-sdk/xml-builder": "^3.972.41",
"@aws/lambda-invoke-store": "^0.3.0",
"@smithy/core": "^3.35.0",
"@smithy/signature-v4": "^5.7.3",
"@smithy/types": "^4.19.0",
"bowser": "^2.11.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/credential-provider-env": {
"version": "3.972.72",
"resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-env/-/credential-provider-env-3.972.72.tgz",
"integrity": "sha512-xTKO/FWJPozTIXbozVnVGoNBhaGba8TBcx+KyUjRVeOlXE+dUc7GTR1cLvu0uTdIdmemzaFbqqCshXeZA1fZew==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/credential-provider-http": {
"version": "3.972.74",
"resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-http/-/credential-provider-http-3.972.74.tgz",
"integrity": "sha512-u91E/hT8f4d1xy0Jl7VG4nVKJ3lxbrZkoBTeSVoJdWBiSEUMwMS/9+e0H/aJVQV//Lt5wuzP+E69v4aRSsNTmw==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/fetch-http-handler": "^5.8.0",
"@smithy/node-http-handler": "^4.12.1",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/credential-provider-ini": {
"version": "3.973.17",
"resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-ini/-/credential-provider-ini-3.973.17.tgz",
"integrity": "sha512-ged4KXdBkvIC81bLvNHHuQKdKak/VXhQTR1NWYTTqW0474nlmsxy9O/vlgTIohDDWH3xpBdtVMZRyjb+DnocDA==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/credential-provider-env": "^3.972.72",
"@aws-sdk/credential-provider-http": "^3.972.74",
"@aws-sdk/credential-provider-login": "^3.972.79",
"@aws-sdk/credential-provider-process": "^3.972.72",
"@aws-sdk/credential-provider-sso": "^3.973.16",
"@aws-sdk/credential-provider-web-identity": "^3.972.78",
"@aws-sdk/nested-clients": "^3.997.46",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/credential-provider-imds": "^4.5.2",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/credential-provider-login": {
"version": "3.972.79",
"resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-login/-/credential-provider-login-3.972.79.tgz",
"integrity": "sha512-L+Z85anONJd8MaiuraO4wRxATCdEejBZ3K3eymzWI5JPXa9sOS9CkIm72PBKqXKX+Z9p9NGMX5AIMXm0LEflgw==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/nested-clients": "^3.997.46",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/credential-provider-node": {
"version": "3.972.84",
"resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-node/-/credential-provider-node-3.972.84.tgz",
"integrity": "sha512-oHt854odINVwzwsh+c5x69j0ajm4DbqqqVJ+O1ECsCIZeMDAbzFpXItaqP7UZstJj/ATdTk/KFSH0LaNAgV+kA==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/credential-provider-env": "^3.972.72",
"@aws-sdk/credential-provider-http": "^3.972.74",
"@aws-sdk/credential-provider-ini": "^3.973.17",
"@aws-sdk/credential-provider-process": "^3.972.72",
"@aws-sdk/credential-provider-sso": "^3.973.16",
"@aws-sdk/credential-provider-web-identity": "^3.972.78",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/credential-provider-imds": "^4.5.2",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/credential-provider-process": {
"version": "3.972.72",
"resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-process/-/credential-provider-process-3.972.72.tgz",
"integrity": "sha512-rLIp2xbMjX/k9/od7APpqq1ZgXXnV0pOL1Th3ZsL8Wu0TRtBsDTVS8iPqcfRFcHakFxPvR04OSTv2ka2qOb/2A==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/credential-provider-sso": {
"version": "3.973.16",
"resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-sso/-/credential-provider-sso-3.973.16.tgz",
"integrity": "sha512-IGihaJfFZYacJJr/odqILCoK7W/mvrZ7cuK7ECn3sAu4vLC6u0V8bS7mCGbdugJ8Aum2tnvqmx0F2MRFp2rn9g==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/nested-clients": "^3.997.46",
"@aws-sdk/token-providers": "3.1138.0",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/credential-provider-web-identity": {
"version": "3.972.78",
"resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-web-identity/-/credential-provider-web-identity-3.972.78.tgz",
"integrity": "sha512-/y9WvNtlcPBGLR0qc1a+9J/xtYZfVczvLUOuXaVWylzttH7ewsxwHtjmiJSolNrVSDorIxHGHMU61CbonRkmwA==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/nested-clients": "^3.997.46",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/middleware-sdk-s3": {
"version": "3.972.77",
"resolved": "https://registry.npmjs.org/@aws-sdk/middleware-sdk-s3/-/middleware-sdk-s3-3.972.77.tgz",
"integrity": "sha512-E7W2UOeUoc+lg3uIfR/dM7ZwusHwhBQrKMnlkRv4EXRR+C0YtV1pg25xC7GdZIhXH+NAMgZPCbE7o5to2cjFiw==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/signature-v4-multi-region": "^3.996.47",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/nested-clients": {
"version": "3.997.46",
"resolved": "https://registry.npmjs.org/@aws-sdk/nested-clients/-/nested-clients-3.997.46.tgz",
"integrity": "sha512-oRxtBcka/JGHGs9l9p9IVajGoTP8vTPmoAzdHGy4Qcy9P5vPnDf6nhIeM/COQNY9k/OahImTRaLkHftoXvfcmQ==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/signature-v4-multi-region": "^3.996.47",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/fetch-http-handler": "^5.8.0",
"@smithy/node-http-handler": "^4.12.1",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/signature-v4-multi-region": {
"version": "3.996.47",
"resolved": "https://registry.npmjs.org/@aws-sdk/signature-v4-multi-region/-/signature-v4-multi-region-3.996.47.tgz",
"integrity": "sha512-Zk08macMvQTHzQJCLJVkOlviVoqwYMrpXv4lmLN7b7sAbiMoOK7Go0NYdR5UeF+MW8LIbRmwrNy9u/5VvX1U5g==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/types": "^3.974.6",
"@smithy/signature-v4": "^5.7.3",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/token-providers": {
"version": "3.1138.0",
"resolved": "https://registry.npmjs.org/@aws-sdk/token-providers/-/token-providers-3.1138.0.tgz",
"integrity": "sha512-GpyAr0DD63YOEmYFM6Df+gJuIgC92MMTiBK4FTKfxii5MJ9ge20epR7LyroulscYlG89J+ZB2ivFDPjvfQhzdw==",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/core": "^3.978.1",
"@aws-sdk/nested-clients": "^3.997.46",
"@aws-sdk/types": "^3.974.6",
"@smithy/core": "^3.35.0",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/types": {
"version": "3.974.6",
"resolved": "https://registry.npmjs.org/@aws-sdk/types/-/types-3.974.6.tgz",
"integrity": "sha512-v/clNZzZnDxGyvpHMOGpJKVXFAExJzUNAAjaWGdcx8QAcXLGwTaOkw33p5SHAi0YAioK32xB3hWwOekRVfmfKg==",
"license": "Apache-2.0",
"dependencies": {
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws-sdk/xml-builder": {
"version": "3.972.41",
"resolved": "https://registry.npmjs.org/@aws-sdk/xml-builder/-/xml-builder-3.972.41.tgz",
"integrity": "sha512-ctjVSyCMegrWfXlx6VqzSBFI6UqmQ5ZlnfMhdLIiWmhoH8UAQxSCP5N3OpG7X3k4LnS7ou74C4mt20+bfTW2aQ==",
"license": "Apache-2.0",
"dependencies": {
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/@aws/lambda-invoke-store": {
"version": "0.3.0",
"resolved": "https://registry.npmjs.org/@aws/lambda-invoke-store/-/lambda-invoke-store-0.3.0.tgz",
"integrity": "sha512-sl4Bm6yiMNYrZKkqqDFWN0UfnWhlS8ivKxrYl+6t0gCLrqr8y3B2IqZZbFRkfaVVp7C/baApyh71P+LeE1A2sQ==",
"license": "Apache-2.0",
"engines": {
"node": ">=18.0.0"
}
},
"node_modules/@balena/dockerignore": { "node_modules/@balena/dockerignore": {
"version": "1.0.2", "version": "1.0.2",
"resolved": "https://registry.npmjs.org/@balena/dockerignore/-/dockerignore-1.0.2.tgz", "resolved": "https://registry.npmjs.org/@balena/dockerignore/-/dockerignore-1.0.2.tgz",
@@ -802,6 +1113,174 @@
"node": ">=8" "node": ">=8"
} }
}, },
"node_modules/@oven/bun-darwin-aarch64": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-darwin-aarch64/-/bun-darwin-aarch64-1.4.2.tgz",
"integrity": "sha512-MXdZkP1featqxZ+/VTXWG1BVjM4OGBehVY2Q88EeUj/7L0UMeCGItmyPYTN+wxvlGJ6F66JEtzsw+GvQWewnag==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"darwin"
]
},
"node_modules/@oven/bun-darwin-x64": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-darwin-x64/-/bun-darwin-x64-1.4.2.tgz",
"integrity": "sha512-gZTxZuLjkUhAWjTETu3tw0WhsEdNkJ64daj60ybhPf835a2yollV3yTkK9JozvzKPx4TRFzLSl8C+U525pxVbw==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"darwin"
]
},
"node_modules/@oven/bun-freebsd-aarch64": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-freebsd-aarch64/-/bun-freebsd-aarch64-1.4.2.tgz",
"integrity": "sha512-SMNItMw1Z8QeeQVKnw8jA7xQNkeXdP+OPgin4Wi/QTx/B8RHHLnuZfqmFy7NtVeT2NF0kKYppW4WWd2CCYZjhQ==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"freebsd"
]
},
"node_modules/@oven/bun-freebsd-x64": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-freebsd-x64/-/bun-freebsd-x64-1.4.2.tgz",
"integrity": "sha512-THbPKXhO54N0DpFRKZNDZpQ7dpbX0bWASuARckAUS9wRtFIHsiY+uULXJvxJGo2YD1YewvXQ4G8Fj7XT5oBCiw==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"freebsd"
]
},
"node_modules/@oven/bun-linux-aarch64": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-linux-aarch64/-/bun-linux-aarch64-1.4.2.tgz",
"integrity": "sha512-3BBP9ovJ2RGHFH6Ae1CAtxNtG1+YY6GD6rmYbsUosoAk9+OEl6zeDQ/k4fBkc6dYOJCtWnx8hUxzNzQATSmvYQ==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/@oven/bun-linux-aarch64-android": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-linux-aarch64-android/-/bun-linux-aarch64-android-1.4.2.tgz",
"integrity": "sha512-3mZKO2rhsNgbAUtAHC1UKUlF2zTxFraDZT/Elv8wzyH0fJL9h+Iv3TgB9lO63w89PRn3eFe+NRA1bhVgikKNPQ==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"android"
]
},
"node_modules/@oven/bun-linux-aarch64-musl": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-linux-aarch64-musl/-/bun-linux-aarch64-musl-1.4.2.tgz",
"integrity": "sha512-+Sm6y+lSiSFBOtXmnekp5Q6n1tUKlyv71FCPWBc61Cgb14T5eBs8SN/nh4MUCOKzONkI3O+as3MGUgikS4aCBQ==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/@oven/bun-linux-x64": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-linux-x64/-/bun-linux-x64-1.4.2.tgz",
"integrity": "sha512-9/E/UXOTpSo3YsV5g+FhtTd/qTpiWoKuxS12cqtuYA1ssu9fRAoPQnipFgGyck3tWO63iUdxBiygq+kELFawng==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/@oven/bun-linux-x64-android": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-linux-x64-android/-/bun-linux-x64-android-1.4.2.tgz",
"integrity": "sha512-6HC5tzcC79113n2IHCTJMWv+HsQImv4ZFEK2XpYLxY6HbT8tM4cUM2Zv1bHZBQsS3jv/zYBamDJ1UX7If0d5tw==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"android"
]
},
"node_modules/@oven/bun-linux-x64-musl": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-linux-x64-musl/-/bun-linux-x64-musl-1.4.2.tgz",
"integrity": "sha512-vVTKUg1bnPhRP/Hp73jIVoFh2vPFNYEqYX0ERKfZBOQEEHitNAeukZzzuUDZS0SoDCIpuWUGSpd/CDMbjdR+Uw==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/@oven/bun-windows-aarch64": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-windows-aarch64/-/bun-windows-aarch64-1.4.2.tgz",
"integrity": "sha512-8EJ1ST7339WJE3poPW5nBgVW/lWf9HBz4W27ZUNhburKmcBLOByPyE6DP9fHD8FQGm5c+ilUN2hX1mrW0jxq9Q==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"win32"
]
},
"node_modules/@oven/bun-windows-x64": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/@oven/bun-windows-x64/-/bun-windows-x64-1.4.2.tgz",
"integrity": "sha512-+bN6OuVld/9diT/RLSXSW7JE6CvNE3gL9XsAEjULi1nUsXd6DNO6GuA9jNdNb3r8PdJFnYHr5aypNV1Oj3Rd9g==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"win32"
]
},
"node_modules/@pinojs/redact": { "node_modules/@pinojs/redact": {
"version": "0.4.0", "version": "0.4.0",
"resolved": "https://registry.npmjs.org/@pinojs/redact/-/redact-0.4.0.tgz", "resolved": "https://registry.npmjs.org/@pinojs/redact/-/redact-0.4.0.tgz",
@@ -1235,6 +1714,87 @@
"win32" "win32"
] ]
}, },
"node_modules/@smithy/core": {
"version": "3.35.0",
"resolved": "https://registry.npmjs.org/@smithy/core/-/core-3.35.0.tgz",
"integrity": "sha512-zRMhfkByhT2snNdr1si24vJitU6Cr9ix2MikUfWmkAgp4jrNP0GcKSP5YvwQ+TlI8AZXER5QOGJn3JsVtSD9/A==",
"license": "Apache-2.0",
"dependencies": {
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=18.0.0"
}
},
"node_modules/@smithy/credential-provider-imds": {
"version": "4.5.2",
"resolved": "https://registry.npmjs.org/@smithy/credential-provider-imds/-/credential-provider-imds-4.5.2.tgz",
"integrity": "sha512-A9uSdn72ozbRUSit0eib0TW7nXuNPlaeM0zcGkJ+nE6tFcSDbnmtwoxbTCFBukVQcszDAyvsd7+rTduPTXpygg==",
"license": "Apache-2.0",
"dependencies": {
"@smithy/core": "^3.33.2",
"@smithy/types": "^4.17.2",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=18.0.0"
}
},
"node_modules/@smithy/fetch-http-handler": {
"version": "5.8.0",
"resolved": "https://registry.npmjs.org/@smithy/fetch-http-handler/-/fetch-http-handler-5.8.0.tgz",
"integrity": "sha512-ycSJu3tFAQ4v04CBB0agqFMVsSQ1iG3yw+SpgxRqKfaURpQD4CZ8Wn0zPMmSnOuTpTh65Vz+EA0rMrw089wvkA==",
"license": "Apache-2.0",
"dependencies": {
"@smithy/core": "^3.33.3",
"@smithy/types": "^4.18.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=18.0.0"
}
},
"node_modules/@smithy/node-http-handler": {
"version": "4.12.1",
"resolved": "https://registry.npmjs.org/@smithy/node-http-handler/-/node-http-handler-4.12.1.tgz",
"integrity": "sha512-ThMkboGeONWXAelq9FvGsuJC4rOi+qyC4/zhUF58xYpxUg5sQKx2VXZYJmtNjr4dSuBJ1HeJXETQILCz3wOHvw==",
"license": "Apache-2.0",
"dependencies": {
"@smithy/core": "^3.33.3",
"@smithy/types": "^4.18.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=18.0.0"
}
},
"node_modules/@smithy/signature-v4": {
"version": "5.7.4",
"resolved": "https://registry.npmjs.org/@smithy/signature-v4/-/signature-v4-5.7.4.tgz",
"integrity": "sha512-tHy0K0VtqNd5Y7Y41h0a0Lhh0L1GzC08dTWg0F7vRJWFtTENg7IZikf3wQkanYIRdb7ngoIPMTmqgUi401fEeQ==",
"license": "Apache-2.0",
"dependencies": {
"@smithy/core": "^3.35.0",
"@smithy/types": "^4.19.0",
"tslib": "^2.6.2"
},
"engines": {
"node": ">=18.0.0"
}
},
"node_modules/@smithy/types": {
"version": "4.19.0",
"resolved": "https://registry.npmjs.org/@smithy/types/-/types-4.19.0.tgz",
"integrity": "sha512-r7jh49VJxGerfAcTQA6gXcKc+98zOp/tqRwzYjgOE+iSQsP6cEU1hq2QzbuipmP68QtYdY9wKEhiCQZIzHgZ4Q==",
"license": "Apache-2.0",
"dependencies": {
"tslib": "^2.6.2"
},
"engines": {
"node": ">=18.0.0"
}
},
"node_modules/@testcontainers/postgresql": { "node_modules/@testcontainers/postgresql": {
"version": "12.1.0", "version": "12.1.0",
"resolved": "https://registry.npmjs.org/@testcontainers/postgresql/-/postgresql-12.1.0.tgz", "resolved": "https://registry.npmjs.org/@testcontainers/postgresql/-/postgresql-12.1.0.tgz",
@@ -1821,6 +2381,12 @@
"node": ">= 6" "node": ">= 6"
} }
}, },
"node_modules/bowser": {
"version": "2.14.1",
"resolved": "https://registry.npmjs.org/bowser/-/bowser-2.14.1.tgz",
"integrity": "sha512-tzPjzCxygAKWFOJP011oxFHs57HzIhOEracIgAePE4pqB3LikALKnSzUyU4MGs9/iCEUuHlAJTjTc5M+u7YEGg==",
"license": "MIT"
},
"node_modules/brace-expansion": { "node_modules/brace-expansion": {
"version": "2.1.4", "version": "2.1.4",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.1.4.tgz", "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.1.4.tgz",
@@ -1876,6 +2442,43 @@
"node": ">=10.0.0" "node": ">=10.0.0"
} }
}, },
"node_modules/bun": {
"version": "1.4.2",
"resolved": "https://registry.npmjs.org/bun/-/bun-1.4.2.tgz",
"integrity": "sha512-TrSXo6HJfIEaczpb3kjX82I2pL47vK1QUNmHRCUdz9IzaOwa9lzOXSWwu2l18YHE3sNfGRapVLd4nNm+22vVVA==",
"cpu": [
"arm64",
"x64"
],
"dev": true,
"hasInstallScript": true,
"license": "MIT",
"os": [
"darwin",
"linux",
"android",
"freebsd",
"win32"
],
"bin": {
"bun": "bin/bun.exe",
"bunx": "bin/bunx.exe"
},
"optionalDependencies": {
"@oven/bun-darwin-aarch64": "1.4.2",
"@oven/bun-darwin-x64": "1.4.2",
"@oven/bun-freebsd-aarch64": "1.4.2",
"@oven/bun-freebsd-x64": "1.4.2",
"@oven/bun-linux-aarch64": "1.4.2",
"@oven/bun-linux-aarch64-android": "1.4.2",
"@oven/bun-linux-aarch64-musl": "1.4.2",
"@oven/bun-linux-x64": "1.4.2",
"@oven/bun-linux-x64-android": "1.4.2",
"@oven/bun-linux-x64-musl": "1.4.2",
"@oven/bun-windows-aarch64": "1.4.2",
"@oven/bun-windows-x64": "1.4.2"
}
},
"node_modules/byline": { "node_modules/byline": {
"version": "5.0.0", "version": "5.0.0",
"resolved": "https://registry.npmjs.org/byline/-/byline-5.0.0.tgz", "resolved": "https://registry.npmjs.org/byline/-/byline-5.0.0.tgz",
@@ -4114,6 +4717,12 @@
"node": ">=20" "node": ">=20"
} }
}, },
"node_modules/tslib": {
"version": "2.8.1",
"resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz",
"integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==",
"license": "0BSD"
},
"node_modules/tsx": { "node_modules/tsx": {
"version": "4.22.4", "version": "4.22.4",
"resolved": "https://registry.npmjs.org/tsx/-/tsx-4.22.4.tgz", "resolved": "https://registry.npmjs.org/tsx/-/tsx-4.22.4.tgz",
+4
View File
@@ -3,6 +3,7 @@
"private": true, "private": true,
"type": "module", "type": "module",
"scripts": { "scripts": {
"build:workspace-tools": "node scripts/build-workspace-tools.mjs",
"dev": "tsx watch src/server.ts", "dev": "tsx watch src/server.ts",
"prebuild": "node scripts/clean-dist.mjs", "prebuild": "node scripts/clean-dist.mjs",
"build": "tsc -p tsconfig.json", "build": "tsc -p tsconfig.json",
@@ -14,11 +15,13 @@
"test:schema-v3-verifier": "npm run test:schema-v4-verifier" "test:schema-v3-verifier": "npm run test:schema-v4-verifier"
}, },
"dependencies": { "dependencies": {
"@aws-sdk/client-s3": "3.1141.0",
"@fastify/cookie": "11.1.2", "@fastify/cookie": "11.1.2",
"@fastify/cors": "^11.2.0", "@fastify/cors": "^11.2.0",
"@fastify/rate-limit": "11.2.0", "@fastify/rate-limit": "11.2.0",
"@types/pg": "^8.20.3", "@types/pg": "^8.20.3",
"fastify": "^5.0.0", "fastify": "^5.0.0",
"ipaddr.js": "2.4.0",
"kysely": "^0.29.5", "kysely": "^0.29.5",
"libphonenumber-js": "1.13.12", "libphonenumber-js": "1.13.12",
"openid-client": "6.8.5", "openid-client": "6.8.5",
@@ -31,6 +34,7 @@
"@testcontainers/postgresql": "^12.1.0", "@testcontainers/postgresql": "^12.1.0",
"@types/node": "24.13.3", "@types/node": "24.13.3",
"@types/validator": "13.15.10", "@types/validator": "13.15.10",
"bun": "1.4.2",
"tsx": "^4.19.0", "tsx": "^4.19.0",
"typescript": "^5.6.0", "typescript": "^5.6.0",
"vitest": "^2.1.0" "vitest": "^2.1.0"
+49
View File
@@ -0,0 +1,49 @@
// Maintainer-only build. The resulting two-binary bundle needs no extra host runtime.
import { spawnSync } from "node:child_process";
import { createHash } from "node:crypto";
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { dirname, join, resolve } from "node:path";
import { fileURLToPath } from "node:url";
const repository = process.env.THT_BUILD_SOURCE_ROOT ? resolve(process.env.THT_BUILD_SOURCE_ROOT) : resolve(dirname(fileURLToPath(import.meta.url)), "../..");
const backend = join(repository, "backend");
const native = `${process.platform === "win32" ? "windows" : process.platform}-${process.arch === "x64" ? "amd64" : process.arch}`;
const targets = {
"windows-amd64": ["windows", "amd64", "bun-windows-x64"],
"darwin-amd64": ["darwin", "amd64", "bun-darwin-x64"],
"darwin-arm64": ["darwin", "arm64", "bun-darwin-arm64"],
"linux-amd64": ["linux", "amd64", "bun-linux-x64"],
"linux-arm64": ["linux", "arm64", "bun-linux-arm64"],
};
const requested = process.argv.slice(2);
const selected = requested.length === 1 && requested[0] === "--all" ? Object.keys(targets) : requested.length ? requested : [native];
if (selected.some((target) => !targets[target])) {
console.error(`Usage: npm run build:workspace-tools -- [${Object.keys(targets).join("|")}|--all]`);
process.exit(2);
}
function run(command, args, cwd = backend, env = process.env) {
const result = spawnSync(command, args, { cwd, env, stdio: "inherit" });
if (result.error || result.status !== 0) throw new Error(`Build failed: ${command}`);
}
const revision = spawnSync("git", ["rev-parse", "HEAD"], { cwd: repository, encoding: "utf8" });
if (revision.status !== 0) throw new Error("Cannot read build revision");
const commit = revision.stdout.trim();
const buildTime = process.env.THT_BUILD_TIME ?? new Date().toISOString();
const releaseVersion = process.env.THT_BUILD_VERSION ?? "0.0.0-dev";
if (!/^[0-9]+\.[0-9]+\.[0-9]+(?:-[A-Za-z0-9.-]+)?$/.test(releaseVersion) || !Number.isFinite(Date.parse(buildTime))) throw new Error("Invalid build identity");
const module = "github.com/aritmolab/thothii/tools/tht/internal/version";
const bunPackage = JSON.parse(readFileSync(join(backend, "node_modules", "bun", "package.json"), "utf8"));
const bun = join(backend, "node_modules", "bun", bunPackage.bin.bun);
for (const target of selected) {
const [os, arch, bunTarget] = targets[target];
const output = join(repository, "dist", "workspace-tools", target);
mkdirSync(output, { recursive: true });
const extension = os === "windows" ? ".exe" : "";
const names = [`tht${extension}`, `tht-workspace-documents${extension}`];
run(bun, ["build", "src/workspace-documents-cli.ts", "--compile", `--target=${bunTarget}`, "--outfile", join(output, names[1])]);
run("go", ["build", "-trimpath", "-ldflags", `-s -w -X ${module}.semanticVersion=${releaseVersion} -X ${module}.commit=${commit} -X ${module}.buildTime=${buildTime}`, "-o", join(output, names[0]), "./cmd/tht"], join(repository, "tools", "tht"), { ...process.env, CGO_ENABLED: "0", GOOS: os, GOARCH: arch });
const hashes = names.map((name) => `${createHash("sha256").update(readFileSync(join(output, name))).digest("hex")} ${name}\n`).join("");
writeFileSync(join(output, "SHA256SUMS"), hashes);
writeFileSync(join(output, "build.json"), JSON.stringify({ version: releaseVersion, commit, buildTime, target, bun: JSON.parse(readFileSync(join(backend, "package.json"), "utf8")).devDependencies.bun }, null, 2) + "\n");
console.log(output);
}
+195
View File
@@ -0,0 +1,195 @@
#!/usr/bin/env node
// Maintainer-only producer. Consumers download the resulting native bundle.
import { spawn } from "node:child_process";
import { mkdirSync, mkdtempSync, readFileSync, writeFileSync, existsSync, realpathSync, renameSync, rmSync, statSync, copyFileSync, chmodSync, openSync, closeSync, readdirSync } from "node:fs";
import { tmpdir } from "node:os";
import { basename, dirname, join, resolve } from "node:path";
import { pathToFileURL } from "node:url";
import { parse } from "yaml";
import { prepareBundle, sha256 } from "./release-bundle.mjs";
import { dockerCredentials, dockerHub } from "./release-registry.mjs";
import { giteaHosting } from "./release-hosting.mjs";
import { publishVerifiedRelease } from "./release-publication.mjs";
const repositoryRoot = resolve(import.meta.dirname, "../..");
export function optionsFromArgs(args) {
const values = {};
for (let i = 0; i < args.length; i += 2) {
if (!['--revision', '--version', '--namespace', '--platforms', '--output', '--repository'].includes(args[i]) || !args[i + 1] || values[args[i]]) throw new Error("Usage: --revision REF --version VERSION --namespace DOCKER_HUB_NAMESPACE --platforms linux/amd64 --output NEW_OR_MATCHING_DIRECTORY [--repository HTTPS_GITEA_REPO]");
values[args[i]] = args[i + 1];
}
if (!values['--revision'] || values['--revision'].startsWith('-') || !/^[0-9]+\.[0-9]+\.[0-9]+(?:-[A-Za-z0-9.-]+)?$/.test(values['--version'] ?? '') || !/^[a-z0-9][a-z0-9_-]{1,38}$/.test(values['--namespace'] ?? '') || !values['--output']) throw new Error("Supply an explicit source revision, semantic release version, Docker Hub namespace and output directory.");
const platforms = (values['--platforms'] ?? '').split(',');
if (!platforms.length || new Set(platforms).size !== platforms.length || platforms.some((p) => !['linux/amd64', 'linux/arm64'].includes(p))) throw new Error("Select explicit Linux image platforms; begin with linux/amd64 for Windows/WSL2 and Omarchy.");
const repository = values['--repository'] ?? 'https://git.tylconsulting.it/mptyl/ThothII';
const url = new URL(repository);
if (url.protocol !== 'https:' || url.username || url.password || url.search || url.hash || !/^\/[A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+$/.test(url.pathname)) throw new Error("Use a credential-free HTTPS Gitea owner/repository URL.");
return { revision: values['--revision'], version: values['--version'], namespace: values['--namespace'], platforms: platforms.sort(), output: resolve(values['--output']), repository };
}
export function commandRunner(logs) {
let sequence = 0;
return async (command, args, { cwd = repositoryRoot, input = '', env = process.env, quiet = false, timeout = 120_000 } = {}) => {
const log = join(logs, `${++sequence}-${basename(command)}.log`);
return new Promise((accept, reject) => {
if (process.platform === 'win32') { reject(new Error('Run the release producer on Linux, WSL2 or macOS.')); return; }
const child = spawn(command, args, { cwd, env, detached: true, stdio: ['pipe', 'pipe', 'pipe'] });
const stdout = [], stderr = []; let size = 0, overflow = false, settled = false, reapTimer;
const terminate = () => {
if (overflow) return;
overflow = true;
try { process.kill(-child.pid, 'SIGKILL'); } catch { child.kill('SIGKILL'); }
// An escaped descendant must never retain our pipes indefinitely.
reapTimer = setTimeout(() => { child.stdout.destroy(); child.stderr.destroy(); finish(-1); }, 500);
};
const timer = setTimeout(terminate, timeout);
const collect = (chunks) => (data) => { size += data.length; if (size > 64 * 2 ** 20) terminate(); else chunks.push(data); };
child.stdout.on('data', collect(stdout)); child.stderr.on('data', collect(stderr));
child.stdin.on('error', () => {}); child.stdin.end(input);
child.on('error', () => { settled = true; clearTimeout(timer); clearTimeout(reapTimer); reject(new Error(`Required maintainer command ${basename(command)} is unavailable.`)); });
function finish(code) {
if (settled) return;
settled = true;
clearTimeout(timer); clearTimeout(reapTimer);
if (!quiet) writeFileSync(log, Buffer.concat([...stdout, ...stderr]), { mode: 0o600 });
if (code !== 0 || overflow) reject(new Error(`${basename(command)} failed${quiet ? '.' : `; inspect private log ${log}`}`));
else accept(Buffer.concat(stdout).toString());
}
child.on('close', finish);
});
};
}
function acquireOutput(output) {
if (!existsSync(output)) mkdirSync(output, { mode: 0o700 });
if (realpathSync(output) !== output || !statSync(output).isDirectory() || (statSync(output).mode & 0o077)) throw new Error("Use a canonical owner-only output directory.");
if (!existsSync(join(output, 'publication-state.json')) && readdirSync(output).length) throw new Error("Choose an empty output directory or the matching previous publication directory.");
const lock = join(output, '.publisher.lock');
if (existsSync(lock)) {
const pid = Number(readFileSync(lock, 'utf8'));
if (!Number.isInteger(pid) || pid <= 0) throw new Error("Inspect the incomplete publisher lock before retrying.");
let alive = true;
try { process.kill(pid, 0); } catch (error) { if (error.code === 'ESRCH') alive = false; }
if (alive) throw new Error("Another publisher owns this output directory.");
rmSync(lock);
}
const fd = openSync(lock, 'wx', 0o600); writeFileSync(fd, String(process.pid)); closeSync(fd);
return () => rmSync(lock, { force: true });
}
export async function publishInstallation(options) {
const unlock = acquireOutput(options.output);
const logs = join(options.output, 'logs'); mkdirSync(logs, { recursive: true, mode: 0o700 });
const run = commandRunner(logs);
let worktree, temporary;
try {
const revision = (await run('git', ['rev-parse', '--verify', `${options.revision}^{commit}`], { quiet: true })).trim();
if (!/^[a-f0-9]{40}$/.test(revision)) throw new Error("Source revision is not a commit.");
const identity = { revision, version: options.version, namespace: options.namespace, platforms: options.platforms, repository: options.repository };
const statePath = join(options.output, 'publication-state.json');
let state = { identity };
if (existsSync(statePath)) {
state = JSON.parse(readFileSync(statePath, 'utf8'));
if (JSON.stringify(state.identity) !== JSON.stringify(identity)) throw new Error("Output directory belongs to a different release; choose a new directory.");
}
const save = () => { const temp = statePath + '.tmp'; writeFileSync(temp, JSON.stringify(state, null, 2) + '\n', { mode: 0o600 }); renameSync(temp, statePath); };
save();
console.log(`Release ${identity.version}: ${identity.namespace}, ${identity.platforms.join(',')}, source ${revision}`);
await run('docker', ['info', '--format', '{{.OSType}}']);
await run('docker', ['buildx', 'version']);
const registry = dockerHub(await dockerCredentials(run));
const hosting = await giteaHosting({ run, repository: options.repository, identity, body: `Installer prerelease for ${identity.platforms.join(', ')}.\n\nImages: docker.io/${identity.namespace}/thothii-core:${identity.version} and docker.io/${identity.namespace}/thothii-frontend:${identity.version}.\n\nDownload the native operator bundle and SHA256SUMS.txt below. This release supplies images, document validation and preflight; complete non-interactive setup and Windows/Omarchy/macOS acceptance are subsequent tickets. No example databases, credentials or user workspace data are included.` });
async function prepare() {
if (state.assets) {
const names = [...identity.platforms.map((platform) => `thothii-${identity.version}-${platform.replace('/', '-')}.tar.gz`), 'SHA256SUMS.txt'];
if (state.assets.length !== names.length) throw new Error("Cached artifact set is incomplete.");
for (const asset of state.assets) if (!names.includes(asset.name) || asset.path !== join(options.output, asset.name) || !existsSync(asset.path) || sha256(readFileSync(asset.path)) !== asset.sha256) throw new Error("Previously built release artifact changed; do not overwrite an immutable version.");
for (const [platform, images] of Object.entries(state.images)) for (const reference of Object.values(images)) {
const [repository, digest] = reference.replace(/^docker.io\//, '').split('@');
await registry.inspect(repository, digest, platform, { anonymous: true });
}
return state.assets;
}
temporary = realpathSync(mkdtempSync(join(tmpdir(), 'thothii-release-')));
worktree = join(temporary, 'source');
await run('git', ['worktree', 'add', '--detach', worktree, revision]);
const sourceCompose = parse(readFileSync(join(worktree, 'compose.yaml'), 'utf8'));
for (const role of ['core', 'frontend']) {
console.log(`Preparing public repository ${identity.namespace}/thothii-${role}`);
await registry.ensurePublic(identity.namespace, `thothii-${role}`);
let existing = null;
for (const platform of identity.platforms) {
const image = await registry.inspect(`${identity.namespace}/thothii-${role}`, identity.version, platform, { allowMissing: true });
if (image && (image.labels['org.opencontainers.image.revision'] !== revision || image.labels['org.opencontainers.image.version'] !== identity.version)) throw new Error("Image tag already belongs to another immutable build; select a new release version.");
existing = existing || image;
}
if (!existing) {
console.log(`Building and publishing ${role} (${identity.platforms.join(', ')})`);
await run('docker', ['buildx', 'build', '--platform', identity.platforms.join(','), '--file', `docker/${role}.Dockerfile`, '--tag', `docker.io/${identity.namespace}/thothii-${role}:${identity.version}`, '--build-arg', `IMAGE_VERSION=${identity.version}`, '--label', `org.opencontainers.image.revision=${revision}`, '--label', `org.opencontainers.image.source=${identity.repository}`, '--provenance=false', '--sbom=false', '--push', '.'], { cwd: worktree, timeout: 45 * 60_000 });
}
}
state.images = {};
for (const platform of identity.platforms) {
const images = {};
for (const role of ['core', 'frontend']) images[role] = (await registry.inspect(`${identity.namespace}/thothii-${role}`, identity.version, platform, { anonymous: true })).reference;
for (const [role, service] of [['catalog', 'catalog-db'], ['qdrant', 'qdrant'], ['embedding', 'embedding']]) {
const [named, digest] = sourceCompose.services[service].image.split('@');
let repository = named.replace(/:[^/:]+$/, '').replace(/^docker.io\//, '');
if (!repository.includes('/')) repository = 'library/' + repository;
images[role] = (await registry.inspect(repository, digest, platform, { anonymous: true })).reference;
}
state.images[platform] = images;
}
save();
console.log('Building native operator bundles from the selected source');
await run('npm', ['ci'], { cwd: join(worktree, 'backend'), timeout: 10 * 60_000 });
const sourceTime = (await run('git', ['show', '-s', '--format=%cI', revision], { quiet: true })).trim();
const targets = identity.platforms.map((platform) => platform.replace('/', '-'));
await run('node', [join(repositoryRoot, 'backend/scripts/build-workspace-tools.mjs'), ...targets], { cwd: join(worktree, 'backend'), env: { ...process.env, THT_BUILD_SOURCE_ROOT: worktree, THT_BUILD_VERSION: identity.version, THT_BUILD_TIME: sourceTime }, timeout: 10 * 60_000 });
const assets = [];
for (const platform of identity.platforms) {
const target = platform.replace('/', '-');
const name = `thothii-${identity.version}-${target}`;
const bundle = join(options.output, name);
if (existsSync(bundle)) rmSync(bundle, { recursive: true }); // owned staging, never an installed runtime
mkdirSync(bundle);
prepareBundle({ source: worktree, destination: bundle, platform, version: identity.version, revision, images: state.images[platform] });
mkdirSync(join(bundle, 'bin'));
for (const executable of ['tht', 'tht-workspace-documents']) {
copyFileSync(join(worktree, 'dist/workspace-tools', target, executable), join(bundle, 'bin', executable));
chmodSync(join(bundle, 'bin', executable), 0o755);
}
// Resolve source-independent resource/config shape without any operator credentials.
await run('docker', ['compose', '-f', join(bundle, 'compose.yaml'), '-f', join(bundle, 'deploy/compose.local.yaml'), 'config', '--no-interpolate', '--no-env-resolution', '--format', 'json']);
const archive = join(options.output, name + '.tar.gz');
await run('tar', ['-czf', archive, '-C', options.output, name]);
assets.push({ name: basename(archive), path: archive, bytes: statSync(archive).size, sha256: sha256(readFileSync(archive)) });
}
const sums = join(options.output, 'SHA256SUMS.txt');
writeFileSync(sums, assets.map((asset) => `${asset.sha256} ${asset.name}\n`).join(''));
assets.push({ name: 'SHA256SUMS.txt', path: sums, bytes: statSync(sums).size, sha256: sha256(readFileSync(sums)) });
// An empty Docker configuration proves the consumer can pull without publisher credentials.
const publicConfig = join(temporary, 'public-docker'); mkdirSync(publicConfig);
for (const [platform, images] of Object.entries(state.images)) {
for (const reference of Object.values(images)) {
console.log(`Verifying anonymous pull ${reference.split('@')[0]} (${platform})`);
await run('docker', ['--config', publicConfig, 'pull', '--platform', platform, reference], { timeout: 20 * 60_000 });
}
console.log(`Smoke checking published images (${platform})`);
await run('docker', ['run', '--rm', '--platform', platform, '--network', 'none', '--entrypoint', '/bin/sh', images.core, '-ec', 'test "$(pi --version)" = "$PI_VERSION"; tht --help >/dev/null; test -f /app/backend/dist/catalog/migrate.js; test -x /app/docker/workspace-maintenance-entrypoint.sh; test -f /app/docker/catalog-migrate.sh; test ! -e /run/secrets/thothii.secrets'], { timeout: 5 * 60_000 });
await run('docker', ['run', '--rm', '--platform', platform, '--network', 'none', '--entrypoint', '/usr/local/bin/frontend-config-smoke', images.frontend], { timeout: 60_000 });
}
state.assets = assets; save();
return assets;
}
const published = await publishVerifiedRelease({ hosting, prepare });
state.releaseURL = published.html_url; state.complete = true; save();
console.log(`Published and verified: ${published.html_url}`);
return published;
} finally {
if (worktree && existsSync(worktree)) await run('git', ['worktree', 'remove', '--force', worktree]).catch(() => {});
if (temporary && !existsSync(worktree ?? '')) rmSync(temporary, { recursive: true, force: true });
unlock();
}
}
if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1])).href) {
try { await publishInstallation(optionsFromArgs(process.argv.slice(2))); }
catch (error) { console.error(error instanceof SyntaxError ? 'Invalid release metadata; no secret values are printed.' : error.message); process.exitCode = 1; }
}
+48
View File
@@ -0,0 +1,48 @@
import { createHash } from "node:crypto";
import { mkdirSync, readFileSync, writeFileSync, lstatSync } from "node:fs";
import { dirname, join } from "node:path";
import { parse, stringify } from "yaml";
export const serviceRoles = Object.freeze({ core: "core", frontend: "frontend", "catalog-db": "catalog", "catalog-migrate": "core", "workspace-maintenance": "core", qdrant: "qdrant", embedding: "embedding", "embedding-model-init": "embedding" });
export const sha256 = (data) => createHash("sha256").update(data).digest("hex");
/** Only distribution assets enter the bundle: never a checkout, environment file or workspace. */
export function prepareBundle({ source, destination, platform, version, revision, images }) {
const compose = parse(readFileSync(join(source, "compose.yaml"), "utf8"));
if (Object.keys(compose.services).sort().join() !== Object.keys(serviceRoles).sort().join()) throw new Error("Release service contract changed; review packaging before publishing.");
for (const [name, role] of Object.entries(serviceRoles)) {
if (!/^docker\.io\/[a-z0-9][a-z0-9._/-]*@sha256:[a-f0-9]{64}$/.test(images[role] ?? "")) throw new Error("Release requires immutable Docker Hub image references.");
delete compose.services[name].build;
delete compose.services[name].pull_policy;
compose.services[name].image = images[role];
compose.services[name].platform = platform;
}
const files = {};
const write = (name, data) => {
mkdirSync(dirname(join(destination, name)), { recursive: true });
writeFileSync(join(destination, name), data);
files[name] = sha256(data);
};
write("compose.yaml", stringify(compose));
for (const name of ["deploy/compose.local.yaml", "deploy/compose.git-https.yaml", "deploy/compose.git-ssh.yaml", "docker/catalog-db-init.sql", "docker/embedding-model-init.sh"]) {
const path = join(source, name);
if (!lstatSync(path).isFile()) throw new Error("Release asset must be a regular tracked file.");
write(name, readFileSync(path));
}
// Any future source bind mount must be deliberately added to the asset allowlist.
for (const service of Object.values(compose.services)) {
for (const volume of service.volumes ?? []) {
const sourcePath = typeof volume === "string" ? volume.split(":")[0] : volume.type === "bind" ? volume.source : undefined;
if (sourcePath?.startsWith(".") && !files[sourcePath.replace(/^\.\//, "")]) throw new Error("Source bind mount is missing from the release bundle.");
}
}
const manifest = { schema_version: 1, version, revision, validator_protocol: 1,
requirements: { cpus: 2, memory_bytes: 4 * 2 ** 30, disk_bytes: 10 * 2 ** 30 },
components: ["pi", "catalog-migrations", "workspace-maintenance"],
images: Object.fromEntries(Object.entries(images).map(([role, reference]) => [role, { [platform]: reference }])),
files, compose: ["compose.yaml", "deploy/compose.local.yaml"] };
writeFileSync(join(destination, "release-manifest.json"), JSON.stringify(manifest, null, 2) + "\n");
writeFileSync(join(destination, "README.md"), `# ThothII ${version} — ${platform}\n\nSource / Sorgente: ${revision}\n\nThis prerelease provides images and document/preflight tools. Non-interactive execution and real-host acceptance are separate follow-up tickets; this is not a certified complete installation.\nQuesta prerelease fornisce immagini e strumenti di preparazione/preflight. Esecuzione non interattiva e collaudi reali sono incrementi successivi: non è ancora un'installazione completa certificata.\n\nUse bin/tht and its sibling bin/tht-workspace-documents together; no Node, Python, Bun or application checkout is required on the consumer host.\nWindows: use the Linux amd64 bundle inside Ubuntu WSL2, not a native Windows shell.\n\n1. bin/tht workspace prepare --directory NEW_WORKSPACE --id practice --name Practice\n2. bin/tht workspace validate --directory WORKSPACE\n3. bin/tht installation prepare --directory NEW_PRIVATE_INSTALLATION\n4. bin/tht installation preflight --directory INSTALLATION --release ABSOLUTE_RELEASE_DIR/release-manifest.json\n5. Complete the commented documents, generate technical credentials explicitly, then run installation validate and installation plan with --installation ABSOLUTE_INSTALLATION_FILE.\n\nImages are pinned by digest; runtime credentials and user workspaces are never bundled.\nConsult the accompanying IT/EN guides for prepared documents and mandatory runtime checks.\n`);
for (const [name, target] of [["standalone-manual-it.md", "GUIDE-IT.md"], ["standalone-manual-en.md", "GUIDE-EN.md"], ["installation-preflight.md", "PREFLIGHT.md"]]) writeFileSync(join(destination, target), readFileSync(join(source, "docs/install", name)));
return manifest;
}
+35
View File
@@ -0,0 +1,35 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, readFileSync, rmSync, existsSync } from "node:fs";
import { tmpdir } from "node:os";
import { resolve, join } from "node:path";
import { parse } from "yaml";
import { prepareBundle } from "./release-bundle.mjs";
test("consumer bundle pins every service and carries all source bind resources", () => {
const output = mkdtempSync(join(tmpdir(), "thoth-release-test-"));
const images = Object.fromEntries(["core", "frontend", "catalog", "qdrant", "embedding"].map((role) => [role, `docker.io/tylconsulting/${role}@sha256:${"a".repeat(64)}`]));
try {
prepareBundle({ source: resolve(import.meta.dirname, "../.."), destination: output, platform: "linux/amd64", version: "0.1.0-install-preview.1", revision: "b".repeat(40), images });
const manifest = JSON.parse(readFileSync(join(output, "release-manifest.json")));
const compose = parse(readFileSync(join(output, "compose.yaml"), "utf8"));
assert.equal(compose.services.core.image, images.core);
assert.equal(compose.services["catalog-migrate"].image, images.core);
assert.equal(compose.services["workspace-maintenance"].image, images.core);
assert.equal(compose.services["embedding-model-init"].image, images.embedding);
for (const service of Object.values(compose.services)) {
assert.equal(service.build, undefined);
assert.equal(service.pull_policy, undefined);
assert.equal(service.platform, "linux/amd64");
for (const volume of service.volumes ?? []) {
if (typeof volume === "string" && volume.startsWith("./")) assert.ok(existsSync(join(output, volume.split(":")[0])));
}
}
assert.equal(manifest.validator_protocol, 1);
assert.deepEqual(manifest.compose, ["compose.yaml", "deploy/compose.local.yaml"]);
assert.ok(manifest.files["docker/catalog-db-init.sql"]);
assert.ok(manifest.files["docker/embedding-model-init.sh"]);
assert.ok(!existsSync(join(output, "backend")));
assert.ok(!existsSync(join(output, "harness")));
} finally { rmSync(output, { recursive: true, force: true }); }
});
+69
View File
@@ -0,0 +1,69 @@
import { readFileSync } from "node:fs";
import { boundedFetch, readLimited } from "./release-registry.mjs";
import { sha256 } from "./release-bundle.mjs";
export async function giteaHosting({ run, repository, identity, body }) {
const remote = new URL(repository);
const credential = await run("git", ["credential", "fill"], { input: `protocol=https\nhost=${remote.host}\n\n`, quiet: true, env: { ...process.env, GIT_TERMINAL_PROMPT: "0" } });
const fields = Object.fromEntries(credential.trim().split("\n").map((line) => { const at = line.indexOf("="); return [line.slice(0, at), line.slice(at + 1)]; }));
if (!fields.username || !fields.password) throw new Error("Gitea publishing credentials are unavailable in the Git credential store.");
const authorization = `Basic ${Buffer.from(`${fields.username}:${fields.password}`).toString("base64")}`;
const api = `${remote.origin}/api/v1/repos${remote.pathname.replace(/\.git$/, "")}`;
const marker = `<!-- thothii-release:${JSON.stringify(identity)} -->`;
const tag = `installation-v${identity.version}`;
async function request(path, options = {}, allowMissing = false) {
const response = await boundedFetch(api + path, { ...options, headers: { Authorization: authorization, ...options.headers } }, 120_000);
if (allowMissing && response.status === 404) return null;
if (!response.ok) throw new Error(`Gitea release operation failed (HTTP ${response.status}).`);
const bytes = await readLimited(response, 4 * 2 ** 20);
try { return JSON.parse(bytes.toString()); } catch { throw new Error("Gitea returned invalid release metadata."); }
}
const json = (method, value) => ({ method, headers: { "Content-Type": "application/json" }, body: JSON.stringify(value) });
async function verifyTag(required = false) {
const existing = await request(`/tags/${encodeURIComponent(tag)}`, {}, true);
if ((!existing && required) || (existing && existing.commit?.sha !== identity.revision)) throw new Error("Release Git tag does not match the requested source revision.");
}
async function assets(release) { return request(`/releases/${release.id}/assets`); }
async function verifyAsset(asset, expected, publicRead) {
const url = new URL(asset.browser_download_url);
if (url.origin !== remote.origin) throw new Error("Unexpected release asset origin.");
const response = await boundedFetch(url, { headers: publicRead ? {} : { Authorization: authorization } }, 120_000);
if (!response.ok || sha256(await readLimited(response, expected.bytes + 1)) !== expected.sha256) throw new Error("Published release asset does not match its verified checksum.");
}
return {
async open() {
const info = await request("");
if (info.private || !info.permissions?.push) throw new Error("Release hosting must be a public Gitea repository with publication rights.");
await verifyTag();
const existing = await request(`/releases/tags/${encodeURIComponent(tag)}`, {}, true);
if (existing) {
if (!existing.body?.includes(marker)) throw new Error("Release version already belongs to another source or publication identity; it will not be overwritten.");
return existing;
}
return request("/releases", json("POST", { tag_name: tag, target_commitish: identity.revision, name: `ThothII ${identity.version}`, body: `${body}\n\n${marker}`, draft: true, prerelease: true }));
},
async upload(release, asset) {
const matching = (await assets(release)).filter((item) => item.name === asset.name);
if (matching.length > 1) throw new Error("Ambiguous release assets; no published files were replaced.");
if (matching.length === 1) { await verifyAsset(matching[0], asset, false); return; }
const data = readFileSync(asset.path);
if (sha256(data) !== asset.sha256) throw new Error("Local release asset changed before upload.");
const form = new FormData(); form.append("attachment", new Blob([data]), asset.name);
await request(`/releases/${release.id}/assets?name=${encodeURIComponent(asset.name)}`, { method: "POST", body: form });
},
async verify(release, expected, publicRead) {
await verifyTag(publicRead);
const uploaded = await assets(release);
if (uploaded.length !== expected.length) throw new Error("Release asset set is incomplete or contains unexpected files.");
for (const asset of expected) {
const matching = uploaded.filter((item) => item.name === asset.name);
if (matching.length !== 1) throw new Error("Release asset missing or ambiguous.");
await verifyAsset(matching[0], asset, publicRead);
}
},
async publish(release) {
await verifyTag();
return request(`/releases/${release.id}`, json("PATCH", { draft: false }));
},
};
}
+43
View File
@@ -0,0 +1,43 @@
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { giteaHosting } from './release-hosting.mjs';
test('release hosting refuses a pre-existing Git tag on another commit', async () => {
const fetch = globalThis.fetch;
const revision = 'a'.repeat(40);
let writes = 0;
globalThis.fetch = async (url, options) => {
if (options.method) { writes++; return Response.json({ id: 1, draft: true }); }
if (String(url).includes('/releases/tags/')) return new Response('', { status: 404 });
if (String(url).includes('/tags/installation-v')) return Response.json({ commit: { sha: 'b'.repeat(40) } });
return Response.json({ private: false, permissions: { push: true } });
};
try {
const hosting = await giteaHosting({ run: async () => 'username=test\npassword=test\n', repository: 'https://example.test/owner/repo', identity: { revision, version: '1.0.0' }, body: '' });
await assert.rejects(hosting.open(), /tag.*revision/i);
assert.equal(writes, 0);
} finally { globalThis.fetch = fetch; }
});
test('matching Git tag is accepted and verified again before publishing', async () => {
const fetch = globalThis.fetch;
const identity = { revision: 'a'.repeat(40), version: '1.0.0' };
let sha = identity.revision;
let writes = 0;
globalThis.fetch = async (url, options) => {
if (options.method) { writes++; return Response.json({ id: 1, draft: false }); }
if (String(url).includes('/releases/tags/')) return Response.json({ id: 1, draft: true, body: `<!-- thothii-release:${JSON.stringify(identity)} -->` });
if (String(url).includes('/tags/')) return Response.json({ commit: { sha } });
if (String(url).endsWith('/assets')) return Response.json([]);
return Response.json({ private: false, permissions: { push: true } });
};
try {
const hosting = await giteaHosting({ run: async () => 'username=test\npassword=test\n', repository: 'https://example.test/owner/repo', identity, body: '' });
const release = await hosting.open();
await hosting.verify(release, [], false);
sha = 'b'.repeat(40);
await assert.rejects(hosting.verify(release, [], false), /tag.*revision/i);
await assert.rejects(hosting.publish(release), /tag.*revision/i);
assert.equal(writes, 0);
} finally { globalThis.fetch = fetch; }
});
+18
View File
@@ -0,0 +1,18 @@
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { commandRunner } from './publish-installation.mjs';
test('publisher timeout terminates descendants that retain output pipes', async () => {
const logs = mkdtempSync(join(tmpdir(), 'release-process-test-'));
try {
const started = Date.now();
await assert.rejects(commandRunner(logs)(process.execPath, ['-e', `
require('child_process').spawn(process.execPath, ['-e', 'setTimeout(() => {}, 2500)'], {stdio: 'inherit'});
setTimeout(() => {}, 2500);
`], { timeout: 250, quiet: true }));
assert.ok(Date.now() - started < 1800, 'timeout must not wait for the descendant to exit naturally');
} finally { rmSync(logs, { recursive: true, force: true }); }
});
+14
View File
@@ -0,0 +1,14 @@
/** Drafts are the publication boundary. A partial build/upload is never a consumer release. */
export async function publishVerifiedRelease({ hosting, prepare }) {
const release = await hosting.open();
const assets = await prepare();
if (!release.draft) {
await hosting.verify(release, assets, true);
return release;
}
for (const asset of assets) await hosting.upload(release, asset);
await hosting.verify(release, assets, false);
const published = await hosting.publish(release);
await hosting.verify(published, assets, true);
return published;
}
@@ -0,0 +1,34 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { publishVerifiedRelease } from "./release-publication.mjs";
test("an interrupted preparation stays draft and retry publishes only after every artifact verifies", async () => {
const events = [];
let failing = true;
const hosting = {
open: async () => ({ id: 7, draft: true }),
upload: async (_draft, asset) => events.push(`upload:${asset.name}`),
verify: async (_draft, assets, publicRead) => events.push(`verify:${publicRead}:${assets.length}`),
publish: async () => { events.push("publish"); return { html_url: "https://example.test/release" }; },
};
const prepare = async () => {
if (failing) throw new Error("frontend build unavailable");
return [{ name: "bundle.tar.gz" }, { name: "SHA256SUMS" }];
};
await assert.rejects(publishVerifiedRelease({ hosting, prepare }), /frontend/);
assert.deepEqual(events, []);
failing = false;
await publishVerifiedRelease({ hosting, prepare });
assert.deepEqual(events, ["upload:bundle.tar.gz", "upload:SHA256SUMS", "verify:false:2", "publish", "verify:true:2"]);
});
test("a published version is verified without replacing any asset", async () => {
const events = [];
await publishVerifiedRelease({ prepare: async () => [{ name: "bundle.tar.gz" }], hosting: {
open: async () => ({ id: 7, draft: false }),
upload: async () => { throw new Error("overwrote a published version"); },
publish: async () => { throw new Error("republished a version"); },
verify: async (_release, _assets, publicRead) => events.push(publicRead),
} });
assert.deepEqual(events, [true]);
});
+86
View File
@@ -0,0 +1,86 @@
import { readFileSync } from "node:fs";
import { homedir } from "node:os";
import { join } from "node:path";
import { sha256 } from "./release-bundle.mjs";
const accept = "application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json";
export async function boundedFetch(url, options = {}, timeout = 30_000) {
try { return await fetch(url, { ...options, redirect: options.redirect ?? "error", signal: AbortSignal.timeout(timeout) }); }
catch { throw new Error("Release network request failed; retry with the same output directory."); }
}
export async function readLimited(response, maximum) {
const chunks = []; let size = 0;
for await (const chunk of response.body) { size += chunk.length; if (size > maximum) throw new Error("Release response exceeded its bound."); chunks.push(chunk); }
return Buffer.concat(chunks);
}
async function jsonResponse(response, description) {
if (!response.ok) throw new Error(`${description} refused (HTTP ${response.status}).`);
const bytes = await readLimited(response, 4 * 2 ** 20);
try { return { bytes, value: JSON.parse(bytes.toString()) }; }
catch { throw new Error("Release service returned invalid metadata."); }
}
export async function dockerCredentials(run) {
const config = JSON.parse(readFileSync(join(process.env.DOCKER_CONFIG || join(homedir(), ".docker"), "config.json"), "utf8"));
const server = "https://index.docker.io/v1/";
const helper = config.credHelpers?.[server] || config.credsStore;
if (!helper || !/^[A-Za-z0-9._-]+$/.test(helper)) throw new Error("Use docker login with an OS credential store before publishing.");
const auth = JSON.parse(await run(`docker-credential-${helper}`, ["get"], { input: server + "\n", quiet: true }));
if (!auth.Username || !auth.Secret) throw new Error("Docker Hub login is unavailable.");
return auth;
}
export function dockerHub(auth) {
async function token(repository, anonymous) {
const url = new URL("https://auth.docker.io/token");
url.searchParams.set("service", "registry.docker.io");
url.searchParams.set("scope", `repository:${repository}:pull`);
const response = await boundedFetch(url, { headers: anonymous ? {} : { Authorization: `Basic ${Buffer.from(`${auth.Username}:${auth.Secret}`).toString("base64")}` } });
return (await jsonResponse(response, "Registry authentication")).value.token;
}
async function registryJSON(repository, route, anonymous, allowMissing = false) {
const bearer = await token(repository, anonymous);
let response = await boundedFetch(`https://registry-1.docker.io/v2/${repository}/${route}`, { redirect: "manual", headers: { Accept: accept, Authorization: `Bearer ${bearer}` } });
if ([302, 307].includes(response.status) && route.startsWith("blobs/")) {
const target = new URL(response.headers.get("location"));
if (target.protocol !== "https:" || target.username || target.password) throw new Error("Invalid registry blob redirect.");
// Signed blob URLs are fetched without forwarding registry credentials.
response = await boundedFetch(target);
}
if (allowMissing && response.status === 404) return null;
const { bytes, value } = await jsonResponse(response, "Registry read");
const digest = `sha256:${sha256(bytes)}`;
const advertised = response.headers.get("docker-content-digest");
if (advertised && advertised !== digest) throw new Error("Registry content digest mismatch.");
return { value, digest };
}
return {
async ensurePublic(namespace, name) {
const response = await boundedFetch("https://hub.docker.com/v2/auth/token", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ identifier: auth.Username, secret: auth.Secret }) });
const bearer = (await jsonResponse(response, "Docker Hub authentication")).value.access_token;
const headers = { Authorization: `Bearer ${bearer}`, "Content-Type": "application/json" };
const path = `https://hub.docker.com/v2/namespaces/${namespace}/repositories`;
let existing = await boundedFetch(`${path}/${name}`, { headers });
if (existing.status === 404) {
existing = await boundedFetch(path, { method: "POST", headers, body: JSON.stringify({ namespace, name, registry: "docker.io", is_private: false, description: `ThothII ${name.endsWith("core") ? "core with embedded Pi" : "standalone frontend"}` }) });
}
const data = (await jsonResponse(existing, "Public repository preparation")).value;
if (data.is_private !== false) throw new Error("Selected Docker Hub repository is private; make this release repository public before retrying.");
},
async inspect(repository, reference, platform, { anonymous = false, allowMissing = false } = {}) {
let image = await registryJSON(repository, `manifests/${reference}`, anonymous, allowMissing);
if (!image) return null;
if (reference.startsWith("sha256:") && image.digest !== reference) throw new Error("Requested image digest does not match registry content.");
if (image.value.manifests) {
const match = image.value.manifests.find((entry) => `${entry.platform?.os}/${entry.platform?.architecture}` === platform);
if (!match || !/^sha256:[a-f0-9]{64}$/.test(match.digest)) throw new Error("Image does not contain the requested platform.");
image = await registryJSON(repository, `manifests/${match.digest}`, anonymous);
if (image.digest !== match.digest) throw new Error("Image index digest mismatch.");
}
if (reference.startsWith("sha256:") && !image.value.config) throw new Error("Image metadata is incomplete.");
const configDigest = image.value.config?.digest;
if (!/^sha256:[a-f0-9]{64}$/.test(configDigest ?? "")) throw new Error("Image configuration digest is invalid.");
const config = await registryJSON(repository, `blobs/${configDigest}`, anonymous);
if (config.digest !== configDigest || `${config.value.os}/${config.value.architecture}` !== platform) throw new Error("Image configuration or platform mismatch.");
return { reference: `docker.io/${repository}@${image.digest}`, labels: config.value.config?.Labels ?? {} };
},
};
}
+28
View File
@@ -0,0 +1,28 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { dockerHub } from "./release-registry.mjs";
import { sha256 } from "./release-bundle.mjs";
test("registry verifies pinned config/platform and does not forward auth to blob storage", async () => {
const original = globalThis.fetch;
const config = JSON.stringify({ os: "linux", architecture: "amd64", config: { Labels: { "org.opencontainers.image.revision": "b".repeat(40) } } });
const configDigest = "sha256:" + sha256(config);
const manifest = JSON.stringify({ schemaVersion: 2, config: { digest: configDigest } });
const imageDigest = "sha256:" + sha256(manifest);
const auth = [];
globalThis.fetch = async (target, options) => {
const url = new URL(target);
if (url.hostname === "auth.docker.io") return new Response(JSON.stringify({ token: "registry-token" }));
if (url.hostname === "blob.example.test") { auth.push(options.headers?.Authorization); return new Response(config); }
if (url.pathname.includes("/blobs/")) return new Response(null, { status: 307, headers: { location: "https://blob.example.test/config" } });
return new Response(manifest, { headers: { "docker-content-digest": imageDigest } });
};
try {
const registry = dockerHub({ Username: "publisher", Secret: "PRIVATE_TOKEN" });
const image = await registry.inspect("example/core", imageDigest, "linux/amd64", { anonymous: true });
assert.equal(image.reference, `docker.io/example/core@${imageDigest}`);
assert.deepEqual(auth, [undefined]);
await assert.rejects(registry.inspect("example/core", imageDigest, "linux/arm64"), /platform mismatch/);
await assert.rejects(registry.inspect("example/core", "sha256:" + "0".repeat(64), "linux/amd64"), /Requested image digest/);
} finally { globalThis.fetch = original; }
});
+21
View File
@@ -0,0 +1,21 @@
import { readFileSync, lstatSync } from "node:fs";
import { parseAllDocuments } from "yaml";
import { decode, DocumentError, runWorkspaceDocuments } from "../workspaces/documents.js";
import { validateDatabaseBootstrap } from "./bootstrap-documents.js";
/** Internal sibling protocol: only references cross back to Go, never secret contents. */
export function runBootstrapValidation(args: string[]): { status: number; output: string } {
try {
if (args.length !== 5 || args[0] !== "--directory" || args[2] !== "--bootstrap" || args[4] !== "--json") throw new Error("usage");
const checked = runWorkspaceDocuments(["validate", "--directory", args[1], "--json"]);
if (checked.status !== 0) return checked;
const info = lstatSync(args[3]);
if (!info.isFile() || info.isSymbolicLink() || info.size > 1024 * 1024) throw new Error("file");
const validated = decode(readFileSync(args[3], "utf8"), "database-bootstrap.yaml", (source) =>
validateDatabaseBootstrap(parseAllDocuments(source)[0].toJSON(), args[1]), "database bootstrap schema v1 and the Catalog binding contract");
return { status: 0, output: JSON.stringify({ schema_version: 1, ok: true, secret_files: validated.secretFiles, warnings: validated.warnings, issues: [] }) };
} catch (error) {
const issue = error instanceof DocumentError ? error.issue : { document: "database-bootstrap.yaml", field: "$", code: "bootstrap_invalid", correction: "Supply one complete Catalog database configuration per workspace in a readable local bootstrap document." };
return { status: 1, output: JSON.stringify({ schema_version: 1, ok: false, issues: [issue] }) };
}
}
@@ -0,0 +1,49 @@
import { readFileSync } from "node:fs";
import { join } from "node:path";
import { z } from "zod";
import { databaseConfigurationSchema } from "./configuration-schema.js";
import { parseWorkspaceCatalogYaml } from "../workspaces/catalog.js";
import { parseWorkspaceYaml } from "../workspaces/schema.js";
import { discoverWorkspaceSecretRequirements } from "../workspaces/secret-requirements.js";
const reference = z.string().min(1).max(4096);
const database = databaseConfigurationSchema.extend({
secretFiles: z.object({ password: reference.optional(), apiKey: reference.optional(), sshPrivateKey: reference.optional(), sshPrivateKeyPassphrase: reference.optional(), sshKnownHosts: reference.optional(), tlsCa: reference.optional() }).strict(),
evidenceSecretFiles: z.object({ "evidence.signed_urls": reference.optional(), "evidence.access_key": reference.optional(), "evidence.secret_key": reference.optional(), "evidence.session_token": reference.optional() }).strict().optional(),
});
const bootstrap = z.object({ schemaVersion: z.literal(1), databases: z.array(database).min(1).max(1000) }).strict();
export const parseDatabaseBootstrap = (value: unknown) => bootstrap.parse(value);
export interface BootstrapReference { field: string; path: string }
/** Offline bootstrap boundary: runtime Catalog owns the resulting bindings after import. */
export function validateDatabaseBootstrap(value: unknown, workspaceRoot: string): { secretFiles: BootstrapReference[]; warnings: string[] } {
const document = bootstrap.parse(value);
const catalog = parseWorkspaceCatalogYaml(readFileSync(join(workspaceRoot, "thoth-workspaces.yaml"), "utf8"));
const expected = new Set(catalog.workspaces.map((entry) => entry.id));
const seen = new Set<string>();
const secretFiles: BootstrapReference[] = [];
const warnings: string[] = [];
const issue = (path: (string | number)[], message: string): never => { throw new z.ZodError([{ code: "custom", path, message }]); };
document.databases.forEach((entry, index) => {
if (!expected.has(entry.workspaceId) || seen.has(entry.workspaceId)) issue(["databases", index, "workspaceId"], "Declare each catalog workspace exactly once.");
seen.add(entry.workspaceId);
const required = entry.binding.transport === "rest_api"
? entry.binding.restAuth === "none" ? [] : ["apiKey"] as const
: entry.binding.transport === "ssh_tunnel" ? ["password", "sshPrivateKey", "sshKnownHosts"] as const : ["password"] as const;
for (const name of required) {
if (!entry.secretFiles[name]) issue(["databases", index, "secretFiles", name], "Supply a protected file reference for this transport.");
}
if (entry.binding.transport === "ssh_tunnel") warnings.push(`databases.${index}:ssh_tunnel supports Catalog diagnostics, not NL-to-SQL sessions; choose direct or REST for practice.`);
for (const [name, path] of Object.entries(entry.secretFiles)) secretFiles.push({ field: `databases.${index}.secretFiles.${name}`, path });
const workspace = parseWorkspaceYaml(readFileSync(join(workspaceRoot, entry.workspaceId, "workspace.yaml"), "utf8"));
const requirements = discoverWorkspaceSecretRequirements(workspace, {});
for (const requirement of requirements.filter((item) => item.connector === "evidence" && item.required)) {
if (!(entry.evidenceSecretFiles as Record<string, string> | undefined)?.[requirement.id]) issue(["databases", index, "evidenceSecretFiles"], "Supply the configured Evidence authentication file references.");
}
for (const [name, path] of Object.entries(entry.evidenceSecretFiles ?? {})) secretFiles.push({ field: `databases.${index}.evidenceSecretFiles.${name}`, path });
});
if (seen.size !== expected.size) issue(["databases"], "Add a database binding for every catalog workspace.");
return { secretFiles, warnings };
}
@@ -0,0 +1,47 @@
import { readFileSync } from "node:fs";
import { join } from "node:path";
import { S3Client, ListObjectsV2Command } from "@aws-sdk/client-s3";
import { parseWorkspaceYaml } from "../workspaces/schema.js";
import { evidencePolicy } from "../workspaces/evidence/preprocessing.js";
import type { parseDatabaseBootstrap } from "./bootstrap-documents.js";
import { probePublicEvidenceUrl, publicEvidenceAgent } from "./evidence-probe-http.js";
type Entry = ReturnType<typeof parseDatabaseBootstrap>["databases"][number];
/** Read-only availability probes; domain correctness and materialization remain runtime gates. */
export async function probeEvidence(entry: Entry, root: string): Promise<void> {
const evidence = parseWorkspaceYaml(readFileSync(join(root, entry.workspaceId, "workspace.yaml"), "utf8")).evidence;
if (!evidence || evidence.source.type === "filesystem") return;
if (evidencePolicy(evidence)) throw new Error("Evidence egress policy refused");
const secret = (name: keyof NonNullable<Entry["evidenceSecretFiles"]>) => {
const path = entry.evidenceSecretFiles?.[name];
if (!path) throw new Error("Evidence credential missing");
return readFileSync(path, "utf8").trim();
};
const source = evidence.source;
if (source.type === "http") {
const urls: unknown = source.authentication === "signed_urls_file"
? JSON.parse(secret("evidence.signed_urls")) : source.uris;
if (!Array.isArray(urls) || urls.length !== source.uris.length || urls.length > 1000) throw new Error("Invalid signed URLs");
for (const [index, value] of urls.entries()) {
if (typeof value !== "string") throw new Error("Invalid signed URL");
const url = new URL(value);
const provenance = new URL(source.uris[index]);
// Signed queries may authorize the same identity, never a different host/path.
if (url.origin !== provenance.origin || url.pathname !== provenance.pathname || url.username || url.password || url.hash) throw new Error("Invalid signed URL identity");
await probePublicEvidenceUrl(url);
}
return;
}
// The shared runtime policy currently permits trusted AWS endpoints with explicit file credentials.
const location = new URL(source.uri);
const client = new S3Client({
region: source.region ?? "us-east-1", maxAttempts: 1,
requestHandler: { httpsAgent: publicEvidenceAgent(), connectionTimeout: 5_000, requestTimeout: 5_000 },
credentials: { accessKeyId: secret("evidence.access_key"), secretAccessKey: secret("evidence.secret_key"),
...(entry.evidenceSecretFiles?.["evidence.session_token"] ? { sessionToken: secret("evidence.session_token") } : {}) },
});
try {
await client.send(new ListObjectsV2Command({ Bucket: location.hostname, Prefix: decodeURIComponent(location.pathname.slice(1)), MaxKeys: 1 }), { abortSignal: AbortSignal.timeout(5_000) });
} finally { client.destroy(); }
}
+53
View File
@@ -0,0 +1,53 @@
import { readFileSync } from "node:fs";
import { parse } from "yaml";
import { parseDatabaseBootstrap } from "./bootstrap-documents.js";
import { runBootstrapValidation } from "./bootstrap-cli.js";
import { createConcreteDiagnosticAdapters, type DiagnosticAdapters } from "../workspaces/diagnostics.js";
import { probeEvidence } from "./bootstrap-evidence-probes.js";
interface ProbeCheck { id: string; outcome: "passed" | "error"; field: string; action: string }
/** Uses the same read-only, authenticated connector diagnostics as the Catalog. */
export async function probeBootstrapDependencies(value: unknown, adapters: DiagnosticAdapters = createConcreteDiagnosticAdapters(), workspaceRoot?: string) {
const document = parseDatabaseBootstrap(value);
const checks: ProbeCheck[] = [];
for (const [index, entry] of document.databases.entries()) {
let outcome: ProbeCheck["outcome"] = "passed";
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), 5_000);
try {
if (entry.binding.transport === "ssh_tunnel") throw new Error("session transport unavailable");
await adapters.probeConnector({
role: "dwh", transport: entry.binding.transport,
host: entry.binding.host, port: entry.binding.port, user: entry.binding.username,
baseUrl: entry.binding.baseUrl,
credentialFile: entry.binding.transport === "rest_api" ? entry.secretFiles.apiKey : entry.secretFiles.password,
tlsCaFile: entry.secretFiles.tlsCa, tlsServername: entry.binding.tlsServername,
resource: { database: entry.databaseName, schema: entry.schema },
timeoutMs: 5_000, signal: controller.signal,
diagnostic: { method: "GET", path: entry.binding.restPath ?? "/health", auth: entry.binding.restAuth ?? "bearer" },
});
} catch { outcome = "error"; } finally { clearTimeout(timer); }
checks.push({ id: `database-${index}`, outcome, field: `database-bootstrap.databases.${index}`,
action: entry.binding.transport === "ssh_tunnel"
? "Choose postgres_direct or rest_api for NL-to-SQL practice; SSH diagnostics alone cannot establish session readiness."
: "Require an authenticated read-only connection and access to the configured database/schema; correct endpoint, permissions or protected credentials." });
if (workspaceRoot) {
let evidenceOutcome: ProbeCheck["outcome"] = "passed";
try { await probeEvidence(entry, workspaceRoot); } catch { evidenceOutcome = "error"; }
checks.push({ id: `evidence-${index}`, outcome: evidenceOutcome, field: `workspaces.${index}.evidence`, action: "Require readable local Evidence or authenticated bounded HTTP/S3 access under the canonical egress policy; domain meaning is verified during practice." });
}
}
return { schema_version: 1, ok: checks.every((check) => check.outcome === "passed"), checks };
}
export async function runBootstrapProbes(args: string[]) {
const validation = runBootstrapValidation(args);
if (validation.status !== 0) return validation;
try {
const report = await probeBootstrapDependencies(parse(readFileSync(args[3], "utf8")), undefined, args[1]);
return { status: report.ok ? 0 : 1, output: JSON.stringify(report) };
} catch {
return { status: 1, output: JSON.stringify({ schema_version: 1, ok: false, checks: [{ id: "database-probes", outcome: "error", field: "database-bootstrap", action: "Revalidate prepared documents and protected credential references." }] }) };
}
}
@@ -0,0 +1,44 @@
import { z } from "zod";
import { parseCredentialFreeHttpUrl } from "../auth/url-policy.js";
import { DATABASE_TRANSPORTS } from "./types.js";
const workspaceIdSchema = z.string().regex(/^[a-z][a-z0-9-]{2,62}$/);
const identifier = z.string().trim().min(1).max(128).regex(/^[A-Za-z_][A-Za-z0-9_$-]*$/);
const nonEmpty = z.string().trim().min(1).max(512);
const port = z.number().int().min(1).max(65_535);
const optionalText = nonEmpty.optional();
const sshHost = z.string().trim().min(1).max(255).regex(/^[A-Za-z0-9_.:\[\]-]+$/).optional();
const sshUsername = z.string().trim().min(1).max(128).regex(/^[A-Za-z0-9._-]+$/).optional();
const bindingSchema = z.object({
transport: z.enum(DATABASE_TRANSPORTS),
host: optionalText,
port: port.optional(),
username: optionalText,
baseUrl: z.string().max(2048)
.refine((value) => parseCredentialFreeHttpUrl(value) !== undefined)
.optional(),
restPath: z.string().regex(/^\/(?!\/)[^?#\\\u0000-\u001f]*$/).max(512).optional(),
restAuth: z.enum(["none", "bearer", "x-api-key"]).optional(),
tlsServername: optionalText,
sshHost,
sshPort: port.optional(),
sshUsername,
sshTargetHost: sshHost,
sshTargetPort: port.optional(),
}).strict().superRefine((binding, context) => {
const required = binding.transport === "postgres_direct"
? ["host", "port", "username"] as const
: binding.transport === "rest_api"
? ["baseUrl", "restPath", "restAuth"] as const
: ["username", "sshHost", "sshPort", "sshUsername", "sshTargetHost", "sshTargetPort"] as const;
for (const field of required) {
if (binding[field] === undefined) context.addIssue({ code: "custom", path: [field], message: "Required" });
}
});
export const databaseConfigurationSchema = z.object({
workspaceId: workspaceIdSchema,
engine: z.literal("postgres"),
databaseName: identifier,
schema: identifier,
binding: bindingSchema,
}).strict();
@@ -0,0 +1,57 @@
import { lookup } from "node:dns/promises";
import { request as httpRequest } from "node:http";
import { request as httpsRequest, Agent } from "node:https";
import type { LookupFunction } from "node:net";
import ipaddr from "ipaddr.js";
const refused = () => new Error("Evidence network policy refused");
function normalizedPublicAddress(value: string): string {
const address = ipaddr.process(value);
if (address.range() !== "unicast") throw refused();
return address.toString();
}
/** Reject the entire DNS answer set, then pin the connection to that verified set. */
export async function resolvePublicEvidenceHost(hostname: string) {
const values = await lookup(hostname.replace(/^\[|\]$/g, ""), { all: true });
if (!values.length) throw refused();
values.forEach((value) => normalizedPublicAddress(value.address));
return values;
}
const publicLookup: LookupFunction = (hostname, options, callback) => {
void resolvePublicEvidenceHost(hostname).then((values) => {
if (options.all) callback(null, values);
else callback(null, values[0].address, values[0].family);
}, () => callback(refused(), "", 0));
};
// Node's direct agent does not inherit HTTP proxy environment or ambient credentials.
export const publicEvidenceAgent = () => new Agent({ lookup: publicLookup });
export async function probePublicEvidenceUrl(url: URL): Promise<void> {
if (!['http:', 'https:'].includes(url.protocol)) throw refused();
const signal = AbortSignal.timeout(5_000);
const values = await Promise.race([
resolvePublicEvidenceHost(url.hostname),
new Promise<never>((_, reject) => signal.addEventListener("abort", () => reject(refused()), { once: true })),
]);
signal.throwIfAborted();
const allowed = new Set(values.map((value) => normalizedPublicAddress(value.address)));
const pinned: LookupFunction = (_hostname, options, callback) => {
if (options.all) callback(null, values);
else callback(null, values[0].address, values[0].family);
};
await new Promise<void>((resolve, reject) => {
const request = (url.protocol === "https:" ? httpsRequest : httpRequest)(url, {
method: "GET", lookup: pinned, signal, agent: false,
}, (response) => {
try {
const peer = response.socket.remoteAddress;
if (!peer || !allowed.has(normalizedPublicAddress(peer)) || !response.statusCode || response.statusCode < 200 || response.statusCode >= 300) throw refused();
resolve();
} catch { reject(refused()); } finally { response.destroy(); }
});
request.on("error", () => reject(refused()));
request.end();
});
}
+4 -1
View File
@@ -41,8 +41,11 @@ const runtimeModelSchema = z.object({
supportsReasoningEffort: z.boolean(), supportsReasoningEffort: z.boolean(),
supportsStore: z.boolean(), supportsStore: z.boolean(),
maxTokensField: z.string().optional(), maxTokensField: z.string().optional(),
thinkingFormat: z.enum(["qwen", "qwen-chat-template"]).optional(),
}).strict().optional(), }).strict().optional(),
}).strict().optional(), }).strict().refine((session) => !session.compatibility?.thinkingFormat || session.reasoning, {
message: "thinkingFormat requires reasoning: true",
}).optional(),
metadataGeneration: z.object({ disableThinking: z.boolean() }).strict().optional(), metadataGeneration: z.object({ disableThinking: z.boolean() }).strict().optional(),
}).strict(); }).strict();
+121 -1
View File
@@ -16,12 +16,16 @@ import { loadSettings } from "./settings/settings-store.js";
import { ThtRunner, type SessionRow } from "./tht/tht-runner.js"; import { ThtRunner, type SessionRow } from "./tht/tht-runner.js";
import { WorkspaceRegistry } from "./workspaces/registry.js"; import { WorkspaceRegistry } from "./workspaces/registry.js";
import { WorkspaceSecretStore } from "./workspaces/secret-store.js"; import { WorkspaceSecretStore } from "./workspaces/secret-store.js";
import { createProductionWorkspaceDiagnoser } from "./workspaces/diagnostics.js";
import { resolveCatalogRuntimeBinding } from "./catalog/runtime-binding.js";
import { CatalogService } from "./catalog/service.js";
import { validateOperationalWorkspace } from "./workspaces/schema.js";
import { createCatalogRepository } from "./catalog/repository.js"; import { createCatalogRepository } from "./catalog/repository.js";
import type { CatalogRepository } from "./catalog/types.js"; import type { CatalogRepository } from "./catalog/types.js";
type OperatorAction = "maintenance-activate" | "maintenance-deactivate" | "maintenance-status" type OperatorAction = "maintenance-activate" | "maintenance-deactivate" | "maintenance-status"
| "session-inventory" | "workflow-doctor" | "workspace-integrity" | "session-inventory" | "workflow-doctor" | "workspace-integrity"
| "pi-test" | "effective-settings"; | "workspace-pull" | "workspace-test" | "pi-test" | "effective-settings";
const lifecyclePrincipal: PrincipalContext = { const lifecyclePrincipal: PrincipalContext = {
issuer: "tht-operator-command", issuer: "tht-operator-command",
@@ -119,6 +123,119 @@ async function workspaceIntegrity(config: AppConfig): Promise<{
return { ready: true, ...integrity }; return { ready: true, ...integrity };
} }
async function workspacePull(config: AppConfig): Promise<{
ready: boolean;
status: "succeeded" | "degraded";
branch: string;
head?: string;
degraded: boolean;
}> {
const status = await new WorkspaceRegistry(config.workspaceRegistry).pull();
return {
ready: !status.degraded,
status: status.degraded ? "degraded" : "succeeded",
branch: status.branch,
...(status.head ? { head: status.head } : {}),
degraded: status.degraded,
};
}
interface WorkspaceTestReport {
id: string;
status: "ready" | "failed";
database: "reachable" | "not_configured" | "failed";
diagnostics: string[];
}
async function workspaceTest(config: AppConfig): Promise<{
ready: boolean;
workspaces: WorkspaceTestReport[];
}> {
if (!config.catalogDatabase) throw new Error("Catalog database is not configured");
const registry = new WorkspaceRegistry(config.workspaceRegistry);
const revisions = await registry.list();
const repository = createCatalogRepository(config.catalogDatabase);
try {
const secretStore = new WorkspaceSecretStore({
root: config.workspaceSecretStoreRoot,
runtimeRoot: config.workspaceSecretRuntimeRoot,
installationId: config.workspaceRegistry.installationId,
});
const catalogService = new CatalogService(
repository,
registry,
secretStore,
config.workspaceRegistry.secretRoots,
config.workspaceDiagnosticTimeoutMs,
);
const diagnose = createProductionWorkspaceDiagnoser(config.workspaceDiagnosticTimeoutMs, undefined, {
internalQdrantUrl: config.internalQdrantUrl,
internalEmbeddingUrl: config.internalEmbeddingUrl,
internalEmbeddingId: config.internalEmbeddingId,
internalEmbeddingModel: config.internalEmbeddingModel,
internalEmbeddingDimensions: config.internalEmbeddingDimensions,
});
const databases = await repository.list();
const reports: WorkspaceTestReport[] = [];
for (const revision of revisions) {
const diagnostics: string[] = [];
let workspace: ReturnType<typeof validateOperationalWorkspace>;
try {
workspace = validateOperationalWorkspace((await registry.read(revision.id)).workspace);
} catch {
reports.push({ id: revision.id, status: "failed", database: "not_configured", diagnostics: ["workspace_invalid"] });
continue;
}
const database = databases.find((candidate) => candidate.workspaceId === revision.id);
if (!database) {
reports.push({ id: revision.id, status: "failed", database: "not_configured", diagnostics: ["database_binding_missing"] });
continue;
}
let tested;
try {
tested = await catalogService.test(database);
} catch {
reports.push({ id: revision.id, status: "failed", database: "failed", diagnostics: ["database_unreachable"] });
continue;
}
if (!tested || tested.connectionStatus !== "reachable") {
reports.push({ id: revision.id, status: "failed", database: "failed", diagnostics: ["database_unreachable"] });
continue;
}
let lease: ReturnType<typeof resolveCatalogRuntimeBinding>;
try {
lease = resolveCatalogRuntimeBinding({
workspace,
database: tested,
environment: process.env,
secretRoots: config.workspaceRegistry.secretRoots,
secretStore,
});
} catch {
reports.push({ id: revision.id, status: "failed", database: "reachable", diagnostics: ["binding_missing"] });
continue;
}
try {
// CatalogService.test() above is the authoritative database probe and records its
// outcome. The remaining diagnoser pass checks Evidence and internal semantic services;
// skipping its legacy DWH probe avoids requiring a second response-shape contract for a
// REST health endpoint.
const result = await diagnose(lease.workspace, lease.bindings, { writeProbe: false, skipDwh: true });
diagnostics.push(...result.diagnostics.map((diagnostic) => diagnostic.code));
const ready = result.activatable;
reports.push({ id: revision.id, status: ready ? "ready" : "failed", database: "reachable", diagnostics });
} catch {
reports.push({ id: revision.id, status: "failed", database: "reachable", diagnostics: ["connector_unavailable"] });
} finally {
lease.release();
}
}
return { ready: reports.length > 0 && reports.every((report) => report.status === "ready"), workspaces: reports };
} finally {
await repository.close?.();
}
}
export async function runOperatorAction( export async function runOperatorAction(
action: OperatorAction, action: OperatorAction,
config: AppConfig, config: AppConfig,
@@ -132,6 +249,8 @@ export async function runOperatorAction(
if (action === "session-inventory") return await sessionInventory(config); if (action === "session-inventory") return await sessionInventory(config);
if (action === "workflow-doctor") return await workflowDiagnostics(config); if (action === "workflow-doctor") return await workflowDiagnostics(config);
if (action === "workspace-integrity") return await workspaceIntegrity(config); if (action === "workspace-integrity") return await workspaceIntegrity(config);
if (action === "workspace-pull") return await workspacePull(config);
if (action === "workspace-test") return await workspaceTest(config);
const modelCatalog = loadRuntimeModelCatalog(config.modelCatalogFile); const modelCatalog = loadRuntimeModelCatalog(config.modelCatalogFile);
if (action === "effective-settings") { if (action === "effective-settings") {
return effectiveSettings(config, loadSettings(config), modelCatalog); return effectiveSettings(config, loadSettings(config), modelCatalog);
@@ -146,6 +265,7 @@ async function main(): Promise<void> {
if (!action || ![ if (!action || ![
"maintenance-activate", "maintenance-deactivate", "maintenance-status", "session-inventory", "maintenance-activate", "maintenance-deactivate", "maintenance-status", "session-inventory",
"workflow-doctor", "workspace-integrity", "pi-test", "effective-settings", "workflow-doctor", "workspace-integrity", "pi-test", "effective-settings",
"workspace-pull", "workspace-test",
].includes(action)) throw new Error("invalid operator action"); ].includes(action)) throw new Error("invalid operator action");
const result = await runOperatorAction(action, loadConfig(process.env)); const result = await runOperatorAction(action, loadConfig(process.env));
process.stdout.write(`${JSON.stringify(result)}\n`); process.stdout.write(`${JSON.stringify(result)}\n`);
+1 -42
View File
@@ -1,60 +1,19 @@
import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify"; import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
import { z } from "zod"; import { z } from "zod";
import { isPrincipalContext, requirePermission } from "../auth/authorization.js"; import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
import { parseCredentialFreeHttpUrl } from "../auth/url-policy.js"; import { databaseConfigurationSchema as configSchema } from "../catalog/configuration-schema.js";
import { CatalogService, type CatalogSecretName } from "../catalog/service.js"; import { CatalogService, type CatalogSecretName } from "../catalog/service.js";
import { WorkspaceRegistryError } from "../workspaces/git-repository.js"; import { WorkspaceRegistryError } from "../workspaces/git-repository.js";
import { import {
CatalogConflictError, CatalogConflictError,
CatalogOperationInProgressError, CatalogOperationInProgressError,
CatalogUnavailableError, CatalogUnavailableError,
DATABASE_TRANSPORTS,
type CatalogRepository, type CatalogRepository,
type DatabaseConfigurationInput, type DatabaseConfigurationInput,
} from "../catalog/types.js"; } from "../catalog/types.js";
import type { CatalogOperationCoordinator } from "../catalog/operation-coordinator.js"; import type { CatalogOperationCoordinator } from "../catalog/operation-coordinator.js";
const idSchema = z.uuid(); const idSchema = z.uuid();
const workspaceIdSchema = z.string().regex(/^[a-z][a-z0-9-]{2,62}$/);
const identifier = z.string().trim().min(1).max(128).regex(/^[A-Za-z_][A-Za-z0-9_$-]*$/);
const nonEmpty = z.string().trim().min(1).max(512);
const port = z.number().int().min(1).max(65_535);
const optionalText = nonEmpty.optional();
const sshHost = z.string().trim().min(1).max(255).regex(/^[A-Za-z0-9_.:\[\]-]+$/).optional();
const sshUsername = z.string().trim().min(1).max(128).regex(/^[A-Za-z0-9._-]+$/).optional();
const bindingSchema = z.object({
transport: z.enum(DATABASE_TRANSPORTS),
host: optionalText,
port: port.optional(),
username: optionalText,
baseUrl: z.string().max(2048)
.refine((value) => parseCredentialFreeHttpUrl(value) !== undefined)
.optional(),
restPath: z.string().regex(/^\/(?!\/)[^?#\\\u0000-\u001f]*$/).max(512).optional(),
restAuth: z.enum(["none", "bearer", "x-api-key"]).optional(),
tlsServername: optionalText,
sshHost,
sshPort: port.optional(),
sshUsername,
sshTargetHost: sshHost,
sshTargetPort: port.optional(),
}).strict().superRefine((binding, context) => {
const required = binding.transport === "postgres_direct"
? ["host", "port", "username"] as const
: binding.transport === "rest_api"
? ["baseUrl", "restPath", "restAuth"] as const
: ["username", "sshHost", "sshPort", "sshUsername", "sshTargetHost", "sshTargetPort"] as const;
for (const field of required) {
if (binding[field] === undefined) context.addIssue({ code: "custom", path: [field], message: "Required" });
}
});
const configSchema = z.object({
workspaceId: workspaceIdSchema,
engine: z.literal("postgres"),
databaseName: identifier,
schema: identifier,
binding: bindingSchema,
}).strict();
const updateSchema = configSchema.extend({ version: z.number().int().positive() }); const updateSchema = configSchema.extend({ version: z.number().int().positive() });
const secretNames = [ const secretNames = [
"password", "password",
+6 -3
View File
@@ -509,12 +509,15 @@ export function sessionRoutes(
// registry snapshot. The legacy fallback stays available for sessions created before // registry snapshot. The legacy fallback stays available for sessions created before
// the browser-local preference migration. // the browser-local preference migration.
let id: string; let id: string;
let sessionLanguage = language;
try { try {
({ id } = await runner.sessionNew({ const created = await runner.sessionNew({
question: b.question, name: b.name, workspaceConfigPath, question: b.question, name: b.name, workspaceConfigPath,
workspaceId, workspaceRevision, provider, model, thinking, workspaceId, workspaceRevision, provider, model, thinking,
interactionLanguage: language, interactionLanguage: language,
})); });
id = created.id;
sessionLanguage = created.interaction_language ?? language;
manifestPersisted = true; manifestPersisted = true;
if (revisionLease) { if (revisionLease) {
await revisionLease.markPersisted().catch((error: unknown) => { await revisionLease.markPersisted().catch((error: unknown) => {
@@ -527,7 +530,7 @@ export function sessionRoutes(
} catch { return storageFailure(reply); } } catch { return storageFailure(reply); }
const options = { const options = {
provider, model, thinking, provider, model, thinking,
interactionLanguage: language, interactionLanguage: sessionLanguage,
author: principal.displayName ?? principal.subject, author: principal.displayName ?? principal.subject,
principal, principal,
question: b.question, question: b.question,
+1 -1
View File
@@ -507,7 +507,7 @@ export class ThtRunner {
if (v) a.push(f, v); if (v) a.push(f, v);
} }
a.push("--json"); a.push("--json");
return this.json<{ id: string }>(a, o.workspaceConfigPath ?? o.workspace); return this.json<{ id: string; interaction_language?: string }>(a, o.workspaceConfigPath ?? o.workspace);
} }
/** Build and persist the deterministic F1 retrieval pack for a new session. */ /** Build and persist the deterministic F1 retrieval pack for a new session. */
+10
View File
@@ -0,0 +1,10 @@
/** Compiled with its runtime for the host CLI: no installation, Docker or host Node required. */
import { runWorkspaceDocuments } from "./workspaces/documents.js";
import { runBootstrapValidation } from "./catalog/bootstrap-cli.js";
import { runBootstrapProbes } from "./catalog/bootstrap-probes.js";
const args = process.argv.slice(2);
const result = args[0] === "probe" ? await runBootstrapProbes(args.slice(1))
: args[0] === "bootstrap" ? runBootstrapValidation(args.slice(1)) : runWorkspaceDocuments(args);
console.log(result.output);
process.exitCode = result.status;
+3 -3
View File
@@ -44,8 +44,8 @@ const catalogSchema = z.object({
}); });
}); });
function safeCatalogError(): Error { function safeCatalogError(cause?: unknown): Error {
return new Error("Workspace catalog is invalid"); return new Error("Workspace catalog is invalid", { cause });
} }
export function parseWorkspaceCatalogYaml(source: string): WorkspaceCatalog { export function parseWorkspaceCatalogYaml(source: string): WorkspaceCatalog {
@@ -57,7 +57,7 @@ export function parseWorkspaceCatalogYaml(source: string): WorkspaceCatalog {
return catalogSchema.parse(document.toJSON()) as WorkspaceCatalog; return catalogSchema.parse(document.toJSON()) as WorkspaceCatalog;
} catch (error) { } catch (error) {
if (error instanceof Error && error.message === "Workspace catalog is invalid") throw error; if (error instanceof Error && error.message === "Workspace catalog is invalid") throw error;
throw safeCatalogError(); throw safeCatalogError(error);
} }
} }
+222
View File
@@ -0,0 +1,222 @@
import { closeSync, lstatSync, mkdirSync, openSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { join, resolve } from "node:path";
import { parseAllDocuments, stringify } from "yaml";
import { ZodError } from "zod";
import { CATALOG_PATH, parseWorkspaceCatalogYaml, assertCatalogMatchesDescriptor } from "./catalog.js";
import { parseWorkspaceYaml, type WorkspaceDescriptor } from "./schema.js";
interface Issue { document: string; field: string; code: string; correction: string; line?: number }
interface Report {
schema_version: 1;
scope: "local-documents";
ok: boolean;
workspaces: { id: string; evidence: "absent" | "local-files" | "remote-deferred" }[];
issues: Issue[];
deferred_checks: string[];
}
const MAX_DOCUMENT_BYTES = 1024 * 1024;
const usage = "tht workspace prepare --directory NEW_PATH --id ID --name NAME [--language en|it] [--json]\ntht workspace validate --directory PATH [--json]";
export class DocumentError extends Error {
constructor(readonly issue: Issue) { super(issue.correction); }
}
function fail(document: string, field: string, code: string, correction: string): never {
throw new DocumentError({ document, field, code, correction });
}
/** Never include parser messages or submitted values: YAML and Zod errors can contain secrets. */
export function decode<T>(source: string, document: string, parser: (text: string) => T, contract = "workspace schema v4 or catalog schema v1"): T {
try {
const documents = parseAllDocuments(source, { uniqueKeys: true });
if (documents.length !== 1) fail(document, "$", "yaml_documents", "Keep exactly one YAML document in this file.");
const problem = [...documents[0].errors, ...documents[0].warnings][0];
if (problem) {
throw new DocumentError({ document, field: "$", code: "yaml_syntax", line: problem.linePos?.[0].line,
correction: "Correct YAML syntax, remove duplicate keys and unsupported tags at the indicated line." });
}
return parser(source);
} catch (error) {
if (error instanceof DocumentError) throw error;
const cause = error instanceof Error && error.cause instanceof ZodError ? error.cause : error;
if (cause instanceof ZodError) {
const issue = cause.issues[0];
// Strict schemas produce paths containing schema-defined keys and array indices only.
fail(document, issue.path.join(".") || "$", "schema_invalid",
issue.code === "unrecognized_keys" ? "Remove fields not defined by the current workspace/catalog contract."
: `Correct this field using ${contract}; check type, required value, uniqueness and allowed values.`);
}
fail(document, "$", "schema_invalid", "Use an authored workspace v4 descriptor; remove database configuration and keep it in the PostgreSQL Metadata Catalog.");
}
}
function stat(root: string, document: string, directory: boolean) {
let info;
try { info = lstatSync(join(root, document)); }
catch { fail(document, "$", "missing_reference", "Create the referenced local file or directory and grant read access."); }
if (info.isSymbolicLink() || (directory ? !info.isDirectory() : !info.isFile())) {
fail(document, "$", "unsafe_reference", "Use a regular local file or directory, without symbolic links or special files.");
}
return info;
}
function readDocument(root: string, document: string): string {
if (stat(root, document, false).size > MAX_DOCUMENT_BYTES) {
fail(document, "$", "document_too_large", "Keep YAML documents below the 1 MiB local validation limit.");
}
try { return readFileSync(join(root, document), "utf8"); }
catch { fail(document, "$", "unreadable", "Grant read access to this document and retry validation."); }
}
function directories(root: string, document: string) {
stat(root, document, true);
try { return readdirSync(join(root, document), { withFileTypes: true }).sort((a, b) => a.name.localeCompare(b.name)); }
catch { fail(document, "$", "unreadable", "Grant read and traversal access to this directory and retry validation."); }
}
function inspectEvidence(root: string, descriptor: WorkspaceDescriptor): "absent" | "local-files" | "remote-deferred" {
const evidence = descriptor.evidence;
if (!evidence) return "absent";
if (evidence.source.type !== "filesystem") return "remote-deferred";
const base = evidence.source.uri;
stat(root, base, true);
if (evidence.schema_version === 2) stat(root, `${base}/curated`, true);
const patterns = evidence.source.patterns ?? ["**/*.md"];
const literals = patterns.filter((pattern) => !/[?*\[]/.test(pattern));
for (const pattern of literals) {
const parts = pattern.split("/");
for (let index = 1; index < parts.length; index++) stat(root, `${base}/${parts.slice(0, index).join("/")}`, true);
stat(root, `${base}/${pattern}`, false);
}
// Inventory without following links; curated-unit interpretation remains a runtime check.
const pending = [base];
let count = 0;
while (pending.length) {
const current = pending.pop()!;
for (const entry of directories(root, current)) {
if (++count > 100_000) fail(base, "evidence.source", "inventory_limit", "Reduce the Evidence tree below 100,000 entries before local validation.");
const path = `${current}/${entry.name}`;
if (entry.isDirectory()) pending.push(path);
else {
const info = stat(root, path, false);
const relative = path.slice(base.length + 1);
// Only apply content checks to selections whose meaning is unambiguous locally.
// Arbitrary globs are expanded by the canonical Python adapter after startup.
const curated = evidence.schema_version === 2 && relative.startsWith("curated/") && relative.endsWith(".md");
const selected = curated || literals.includes(relative) || (patterns.includes("**/*.md") && relative.endsWith(".md"));
if (selected && info.size > evidence.source.max_bytes) fail(path, "evidence.source.max_bytes", "evidence_too_large", "Reduce the source file or increase the declared max_bytes limit deliberately.");
try { closeSync(openSync(join(root, path), "r")); }
catch { fail(path, "$", "unreadable", "Grant read access to this Evidence file and retry validation."); }
if (curated) {
const text = readDocument(root, path);
const frontmatter = text.match(/^---\r?\n([\s\S]*?)\r?\n---(?:\r?\n|$)/);
if (!frontmatter) fail(path, "$", "evidence_frontmatter", "Add a YAML frontmatter block delimited by --- to the curated Markdown unit; follow the Curated Evidence contract.");
decode(frontmatter[1], path, (source) => parseAllDocuments(source)[0].toJSON());
}
}
}
}
return "local-files";
}
function validate(root: string, report: Report): void {
directories(root, ".");
const catalog = decode(readDocument(root, CATALOG_PATH), CATALOG_PATH, parseWorkspaceCatalogYaml);
const ids = new Set(catalog.workspaces.map((entry) => entry.id));
for (const entry of directories(root, ".")) {
if (entry.name === ".git") continue;
if (entry.isSymbolicLink()) fail(".", "$", "unsafe_reference", "Replace repository-root symbolic links with regular files or directories.");
if (entry.isDirectory() && entry.name !== "workspace-docs" && !ids.has(entry.name)) {
fail(".", "workspaces", "unlisted_directory", "Every root directory except workspace-docs must match a catalog workspace id; remove or register the extra directory.");
}
if (entry.name === "workspace-docs") {
for (const docs of directories(root, "workspace-docs")) {
if (!ids.has(docs.name)) fail("workspace-docs", "$", "unlisted_documentation", "Keep documentation only for workspace ids listed in the catalog.");
for (const file of directories(root, `workspace-docs/${docs.name}`)) {
const path = `workspace-docs/${docs.name}/${file.name}`;
if (!["README.md", "contract.env.example"].includes(file.name)) fail(`workspace-docs/${docs.name}`, "$", "unsupported_documentation", "Keep only README.md and contract.env.example in workspace-docs/<id>. Place Evidence inside the workspace directory.");
stat(root, path, false);
}
}
}
}
for (const entry of catalog.workspaces) {
const document = `${entry.id}/workspace.yaml`;
try {
stat(root, entry.id, true);
const descriptor = decode(readDocument(root, document), document, parseWorkspaceYaml);
try { assertCatalogMatchesDescriptor(entry, descriptor); }
catch { fail(document, "workspace", "catalog_mismatch", "Make id, name and description identical in the catalog, descriptor and workspace directory name."); }
const evidence = inspectEvidence(root, descriptor);
report.workspaces.push({ id: entry.id, evidence });
if (evidence !== "absent") report.deferred_checks.push(`${entry.id}:evidence-source-selection`, `${entry.id}:evidence-content-provenance-and-indexing`);
if (evidence === "remote-deferred") report.deferred_checks.push(`${entry.id}:remote-evidence-access`);
} catch (error) {
if (error instanceof DocumentError) report.issues.push(error.issue);
else throw error;
}
}
}
function prepare(root: string, options: Map<string, string>): void {
const id = options.get("--id");
const name = options.get("--name");
const language = options.get("--language") ?? "en";
const catalog = stringify({ schema_version: 1, workspaces: [{ id, name }] });
const descriptor = stringify({ workspace: { schema_version: 4, id, name, language } });
decode(catalog, CATALOG_PATH, parseWorkspaceCatalogYaml);
decode(descriptor, "workspace.yaml", parseWorkspaceYaml);
try { mkdirSync(root); }
catch (error) {
if ((error as NodeJS.ErrnoException).code === "EEXIST") fail(".", "--directory", "destination_exists", "Choose a new directory; preparation never overwrites an existing directory or its documents.");
fail(".", "--directory", "destination_unavailable", "Create the parent directory and grant write access, then choose a new destination.");
}
try {
mkdirSync(join(root, id!));
mkdirSync(join(root, "workspace-docs", id!), { recursive: true });
writeFileSync(join(root, CATALOG_PATH), "# Index of workspace identities. Keep metadata identical to each descriptor.\n" + catalog, { flag: "wx" });
writeFileSync(join(root, id!, "workspace.yaml"), "# Authored workspace v4: optional Evidence; database binding belongs to the Metadata Catalog.\n" + descriptor, { flag: "wx" });
writeFileSync(join(root, "workspace-docs", id!, "README.md"),
"# Workspace documents / Documenti workspace\n\n" +
"EN: Edit thoth-workspaces.yaml and <id>/workspace.yaml together. Evidence is optional and absent by default. Add reviewed source material under <id>/evidence only when configured. Database connections and schema belong to the installation Metadata Catalog. No example databases are downloaded.\n\n" +
"IT: Modificare insieme thoth-workspaces.yaml e <id>/workspace.yaml. Le Evidence sono facoltative e inizialmente assenti. Inserire materiale verificato in <id>/evidence solo quando configurato. Connessioni e schema dei database appartengono al Metadata Catalog dell'installazione. Nessun database di esempio viene scaricato.\n\n" +
"Repeat / Ripetere: `tht workspace validate --directory <repository>`. Local success does not establish runtime readiness or semantic truth / Il successo locale non certifica readiness o verità semantica.\n", { flag: "wx" });
} catch {
rmSync(root, { recursive: true, force: true });
fail(".", "$", "prepare_failed", "Preparation could not write the documents; check disk space and permissions, then retry with a new directory.");
}
}
export function runWorkspaceDocuments(args: string[]): { status: number; output: string } {
const report: Report = { schema_version: 1, scope: "local-documents", ok: false, workspaces: [], issues: [], deferred_checks: ["catalog-database-binding", "database-connectivity", "runtime-preprocessing"] };
const json = args.includes("--json");
let status = 1;
try {
const [command, ...rest] = args;
if ((command === "prepare" || command === "validate") && rest.length === 1 && rest[0] === "--help") return { status: 0, output: usage };
const allowed = command === "prepare" ? ["--directory", "--id", "--name", "--language"] : command === "validate" ? ["--directory"] : [];
const options = new Map<string, string>();
const seen = new Set<string>();
for (let index = 0; index < rest.length; index++) {
const key = rest[index];
if (seen.has(key)) fail("CLI", "$", "usage", usage);
seen.add(key);
if (key === "--json") continue;
if (!allowed.includes(key) || !rest[index + 1] || rest[index + 1].startsWith("--")) fail("CLI", "$", "usage", usage);
options.set(key, rest[++index]);
}
if (!allowed.length || !options.has("--directory") || (command === "prepare" && (!options.has("--id") || !options.has("--name")))) fail("CLI", "$", "usage", usage);
const root = resolve(options.get("--directory")!);
if (command === "prepare") prepare(root, options);
validate(root, report);
report.ok = report.issues.length === 0;
status = report.ok ? 0 : 1;
} catch (error) {
report.issues.push(error instanceof DocumentError ? error.issue : { document: ".", field: "$", code: "io_error", correction: "Check local permissions and regular files, then retry; no services were started." });
if (report.issues[0].code === "usage") status = 2;
}
const output = json ? JSON.stringify(report) : [
report.ok ? "Workspace documents pass local validation." : "Workspace documents require corrections.",
...report.issues.map((issue) => `${issue.document}${issue.line ? `:${issue.line}` : ""} [${issue.field}] ${issue.code}: ${issue.correction}`),
...report.workspaces.map((entry) => `${entry.id}: Evidence ${entry.evidence}`),
`Deferred until runtime: ${report.deferred_checks.join(", ")}.`,
].join("\n");
return { status, output };
}
@@ -0,0 +1,49 @@
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { afterEach, expect, test } from "vitest";
import { validateDatabaseBootstrap } from "../src/catalog/bootstrap-documents.js";
import { runBootstrapValidation } from "../src/catalog/bootstrap-cli.js";
const roots: string[] = [];
afterEach(() => roots.splice(0).forEach((root) => rmSync(root, { recursive: true, force: true })));
function fixture() {
const root = mkdtempSync(join(tmpdir(), "bootstrap-documents-")); roots.push(root);
mkdirSync(join(root, "practice"));
writeFileSync(join(root, "thoth-workspaces.yaml"), "schema_version: 1\nworkspaces: [{id: practice, name: Practice}]\n");
writeFileSync(join(root, "practice/workspace.yaml"), "workspace: {schema_version: 4, id: practice, name: Practice, language: en}\n");
return root;
}
const entry = { workspaceId: "practice", engine: "postgres", databaseName: "practice", schema: "public", binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" }, secretFiles: { password: "/private/operator/db-password" } };
test("bootstrap reuses Catalog configuration and requires one complete binding per workspace", () => {
const root = fixture();
const valid = validateDatabaseBootstrap({ schemaVersion: 1, databases: [entry] }, root);
expect(valid.secretFiles).toContainEqual({ field: "databases.0.secretFiles.password", path: "/private/operator/db-password" });
for (const databases of [[], [entry, entry], [{ ...entry, workspaceId: "unknown" }], [{ ...entry, secretFiles: {} }], [{ ...entry, engine: "mysql" }], [{ ...entry, binding: { ...entry.binding, port: 70000 } }]]) {
expect(() => validateDatabaseBootstrap({ schemaVersion: 1, databases }, root)).toThrow();
}
});
test("REST authentication, SSH credentials and optional Evidence use their declared contracts", () => {
const root = fixture();
const rest = { ...entry, binding: { transport: "rest_api", baseUrl: "https://data.internal", restPath: "/query", restAuth: "none" }, secretFiles: {} };
expect(validateDatabaseBootstrap({ schemaVersion: 1, databases: [rest] }, root).secretFiles).toEqual([]);
expect(() => validateDatabaseBootstrap({ schemaVersion: 1, databases: [{ ...rest, binding: { ...rest.binding, restAuth: "bearer" } }] }, root)).toThrow();
const ssh = { ...entry, binding: { transport: "ssh_tunnel", username: "reader", sshHost: "bastion", sshPort: 22, sshUsername: "tunnel", sshTargetHost: "database", sshTargetPort: 5432 }, secretFiles: { password: "/password", sshPrivateKey: "/key", sshKnownHosts: "/hosts" } };
expect(validateDatabaseBootstrap({ schemaVersion: 1, databases: [ssh] }, root).warnings[0]).toContain("not NL-to-SQL");
writeFileSync(join(root, "practice/workspace.yaml"), "workspace: {schema_version: 4, id: practice, name: Practice, language: en}\nevidence:\n source:\n type: http\n uris: [https://docs.internal/manual.md]\n authentication: signed_urls_file\n");
expect(() => validateDatabaseBootstrap({ schemaVersion: 1, databases: [entry] }, root)).toThrow();
expect(validateDatabaseBootstrap({ schemaVersion: 1, databases: [{ ...entry, evidenceSecretFiles: { "evidence.signed_urls": "/urls.json" } }] }, root).secretFiles).toContainEqual({ field: "databases.0.evidenceSecretFiles.evidence.signed_urls", path: "/urls.json" });
});
test("bootstrap CLI never echoes arbitrary keys from submitted secret references", () => {
const root = fixture();
const path = join(root, "bootstrap.yaml");
for (const field of ["secretFiles", "evidenceSecretFiles"]) {
writeFileSync(path, JSON.stringify({ schemaVersion: 1, databases: [{ ...entry, [field]: { PRIVATE_CREDENTIAL_SENTINEL: "/path" } }] }));
const result = runBootstrapValidation(["--directory", root, "--bootstrap", path, "--json"]);
expect(result.status).toBe(1);
expect(result.output).not.toContain("PRIVATE_CREDENTIAL_SENTINEL");
}
});
@@ -0,0 +1,67 @@
import { spawnSync } from "node:child_process";
import { chmodSync, mkdtempSync, readFileSync, realpathSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { rootCertificates } from "node:tls";
import { afterEach, expect, test } from "vitest";
const binary = process.env.THT_INSTALLATION_TEST_CLI;
const roots: string[] = [];
afterEach(() => roots.splice(0).forEach((path) => rmSync(path, { recursive: true, force: true })));
function run(...args: string[]) {
return spawnSync(binary!, [...args, "--json"], { encoding: "utf8", env: { ...process.env, PATH: "" }, input: "" });
}
test.skipIf(!binary)("operator prepares, completes and repeatedly validates before any stack exists", () => {
const root = realpathSync(mkdtempSync(join(tmpdir(), "application-documents-"))); roots.push(root);
const workspace = join(root, "workspaces"), directory = join(root, "installation");
expect(run("workspace", "prepare", "--directory", workspace, "--id", "practice", "--name", "Practice").status).toBe(0);
expect(run("installation", "prepare", "--directory", directory).status).toBe(0);
const installation = join(directory, "thothii-installation.yaml");
const validate = () => run("--installation", installation, "installation", "validate", "--workspaces", workspace);
expect(validate().status).toBe(1); // Visible placeholders cannot be approved.
expect(run("installation", "credentials", "--directory", directory).status).toBe(0);
for (const name of ["thothii-installation.yaml", "operator.env", "database-bootstrap.yaml"]) {
const path = join(directory, name);
writeFileSync(path, readFileSync(path, "utf8").replaceAll("CHANGE_ME", "practice"));
}
for (const [name, contents] of Object.entries({ "secrets.env": "OPENAI_API_KEY=PRIVATE_PROVIDER_VALUE\n", "database-password": "PRIVATE_DATABASE_VALUE", "git-credentials": "", "git-ca.pem": rootCertificates[0] })) {
writeFileSync(join(directory, "secrets", name), contents, { mode: 0o600 });
}
const before = readFileSync(installation, "utf8");
for (let index = 0; index < 2; index++) {
const checked = validate();
expect(checked.stderr).toBe("");
expect(checked.stdout).not.toContain("PRIVATE_");
expect(checked.status, checked.stdout).toBe(0);
expect(JSON.parse(checked.stdout).deferred_checks).toContain("release-assets");
}
expect(readFileSync(installation, "utf8")).toBe(before);
writeFileSync(installation, before.replace("interaction: openai/gpt-4.1-mini", "interaction: openai/nonexistent"));
expect(validate().status).toBe(1);
writeFileSync(installation, before);
const authFile = join(directory, "secrets/pi-auth.json");
writeFileSync(authFile, "not-json");
expect(validate().status).toBe(1);
writeFileSync(installation, before.replace("{mode: secret_env, apiKeyEnv: OPENAI_API_KEY}", "{mode: pi_auth}"));
writeFileSync(authFile, JSON.stringify({ openai: { unrelated: true } }));
expect(validate().status).toBe(1);
writeFileSync(authFile, JSON.stringify({ " OpenAI ": { type: "api_key", key: "PRIVATE_PI_KEY" } }));
expect(validate().status).toBe(1);
writeFileSync(authFile, JSON.stringify({ openai: { type: "api_key", key: "PRIVATE_PI_KEY" } }));
expect(validate().status).toBe(0);
writeFileSync(installation, before);
writeFileSync(authFile, "{}");
const bootstrap = join(directory, "database-bootstrap.yaml");
const originalBootstrap = readFileSync(bootstrap, "utf8");
writeFileSync(join(workspace, "practice", "private-password"), "PRIVATE_DATABASE_VALUE", { mode: 0o600 });
writeFileSync(bootstrap, originalBootstrap.replace(join(directory, "secrets/database-password"), join(workspace, "practice/private-password")));
expect(validate().status).toBe(1);
writeFileSync(bootstrap, originalBootstrap);
chmodSync(join(directory, "secrets/database-password"), 0o644);
expect(validate().status).toBe(1);
chmodSync(join(directory, "secrets/database-password"), 0o600);
const env = join(directory, "operator.env");
writeFileSync(env, readFileSync(env, "utf8") + "THOTH_HTTP_PORT=8081\n");
expect(validate().status).toBe(1);
}, 15_000);
@@ -0,0 +1,43 @@
import { createServer } from "node:http";
import { mkdtempSync, writeFileSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { expect, it } from "vitest";
import { probeBootstrapDependencies } from "../src/catalog/bootstrap-probes.js";
import { probePublicEvidenceUrl } from "../src/catalog/evidence-probe-http.js";
it("refuses loopback literals and DNS answers before sending an Evidence GET", async () => {
for (const hostname of ["[::1]", "127.0.0.1", "localhost", "[::ffff:127.0.0.1]"]) {
await expect(probePublicEvidenceUrl(new URL(`http://${hostname}/private`))).rejects.toThrow("Evidence network policy refused");
}
});
it("authenticates a bounded read-only REST probe and blocks unavailable credentials/services", async () => {
const directory = mkdtempSync(join(tmpdir(), "tht-probe-"));
const secret = join(directory, "key");
writeFileSync(secret, "PRIVATE_SENTINEL", { mode: 0o600 });
let status = 200;
const requests: string[] = [];
const server = createServer((req, res) => {
requests.push(`${req.method} ${req.url}`);
res.writeHead(req.headers.authorization === "Bearer PRIVATE_SENTINEL" ? status : 401);
res.end("PRIVATE_SERVER_RESPONSE");
});
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
const address = server.address() as { port: number };
const document = { schemaVersion: 1, databases: [{ workspaceId: "demo", engine: "postgres", databaseName: "demo", schema: "public", binding: { transport: "rest_api", baseUrl: `http://127.0.0.1:${address.port}`, restPath: "/health", restAuth: "bearer" }, secretFiles: { apiKey: secret } }] };
try {
expect((await probeBootstrapDependencies(document)).ok).toBe(true);
status = 503;
const failed = await probeBootstrapDependencies(document);
expect(failed.ok).toBe(false);
expect(JSON.stringify(failed)).not.toContain("PRIVATE");
status = 200;
writeFileSync(secret, "rotated-but-invalid");
expect((await probeBootstrapDependencies(document)).ok).toBe(false);
expect(requests).toEqual(["GET /health", "GET /health", "GET /health"]);
} finally {
await new Promise<void>((resolve) => server.close(() => resolve()));
rmSync(directory, { recursive: true });
}
});
@@ -95,6 +95,35 @@ test("returns empty catalogs when no runtime projection is configured", () => {
expect(loadMetadataGenerationModels({}).catalog()).toEqual({ models: [], default: null }); expect(loadMetadataGenerationModels({}).catalog()).toEqual({ models: [], default: null });
}); });
test.each([
["qwen-chat-template", true, true],
["qwen", true, true],
["qwen-typo", true, false],
["qwen-chat-template", false, false],
])("validates Qwen thinking format %s with reasoning=%s", (thinkingFormat, reasoning, valid) => {
const { catalogFile } = runtimeCatalog({
defaultInteraction: "local/qwen",
models: [{
id: "local/qwen", provider: "local", model: "qwen", label: "Qwen",
upstreamModel: "qwen", endpoint: { baseUrl: "http://localhost:8000/v1" },
authentication: { mode: "none" }, sessionAdapter: { mode: "openai_compatible" },
session: {
reasoning, contextWindow: 32768, maxTokens: 8192,
compatibility: {
supportsDeveloperRole: false, supportsReasoningEffort: false,
supportsStore: false, maxTokensField: "max_tokens", thinkingFormat,
},
},
}],
});
if (valid) {
expect(loadRuntimeModelCatalog(catalogFile).sessionModels()[0].session?.compatibility)
.toMatchObject({ thinkingFormat });
} else {
expect(() => loadRuntimeModelCatalog(catalogFile)).toThrow("runtime model catalog is invalid");
}
});
test("rejects a drifted default and an unprotected projection", () => { test("rejects a drifted default and an unprotected projection", () => {
const drifted = runtimeCatalog({ defaultInteraction: "zai/missing" }); const drifted = runtimeCatalog({ defaultInteraction: "zai/missing" });
expect(() => loadRuntimeModelCatalog(drifted.catalogFile)).toThrow("interaction default is invalid"); expect(() => loadRuntimeModelCatalog(drifted.catalogFile)).toThrow("interaction default is invalid");
+14
View File
@@ -36,6 +36,10 @@ vi.mock("../src/tht/tht-runner.js", () => ({
vi.mock("../src/workspaces/registry.js", () => ({ vi.mock("../src/workspaces/registry.js", () => ({
WorkspaceRegistry: class { WorkspaceRegistry: class {
async pull() {
return { branch: "main", head: "a".repeat(40), ahead: 0, behind: 0, degraded: false };
}
async listRetainedSnapshots() { async listRetainedSnapshots() {
return [{ snapshotPath: "/data/workspace-registry/snapshots/revision/workspace.yaml" }]; return [{ snapshotPath: "/data/workspace-registry/snapshots/revision/workspace.yaml" }];
} }
@@ -98,3 +102,13 @@ test("workflow doctor gives schema-v4 runtime rendering a live Catalog repositor
expect(fakes.releaseWorkspaceRuntime).toHaveBeenCalledOnce(); expect(fakes.releaseWorkspaceRuntime).toHaveBeenCalledOnce();
expect(fakes.catalogRepository.close).toHaveBeenCalledOnce(); expect(fakes.catalogRepository.close).toHaveBeenCalledOnce();
}); });
test("workspace pull exposes only safe Git status", async () => {
await expect(runOperatorAction("workspace-pull", config)).resolves.toEqual({
ready: true,
status: "succeeded",
branch: "main",
head: "a".repeat(40),
degraded: false,
});
});
+27
View File
@@ -140,6 +140,33 @@ test("resume rejects browser interaction language overrides", async () => {
} finally { await app.close(); } } finally { await app.close(); }
}); });
test("new session starts Pi with the question language persisted by the harness", async () => {
let runtime: any;
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
thtRunner: {
sessionNew: async () => ({ id: "english-question", interaction_language: "en" }),
searchPack: async () => {},
},
readiness: { ensure: async () => ({ ok: true }) },
mgr: {
get: () => undefined,
createFor: (_id: string, options: any) => {
runtime = options;
return { bridge: { onClientEvent: () => {} } };
},
configure: async () => {}, start: () => {},
},
getSettings: () => ({ workspace: "default" }),
});
try {
const response = await app.inject({ method: "POST", url: "/sessions", payload: {
question: "How many patients were admitted last year?", interactionLanguage: "it",
} });
expect(response.statusCode).toBe(200);
expect(runtime.interactionLanguage).toBe("en");
} finally { await app.close(); }
});
test("resume pins legacy interaction language using the resolved harness workspace", async () => { test("resume pins legacy interaction language using the resolved harness workspace", async () => {
let pinnedWorkspace: string | undefined; let pinnedWorkspace: string | undefined;
let runtime: any; let runtime: any;
@@ -0,0 +1,119 @@
import { spawnSync } from "node:child_process";
import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync, symlinkSync } from "node:fs";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { afterEach, expect, test } from "vitest";
import { parseWorkspaceCatalogYaml } from "../src/workspaces/catalog.js";
import { parseWorkspaceYaml } from "../src/workspaces/schema.js";
const roots: string[] = [];
function directory() {
const root = mkdtempSync(join(tmpdir(), "tht-documents-"));
roots.push(root);
return join(root, "workspaces");
}
afterEach(() => roots.splice(0).forEach((root) => rmSync(root, { recursive: true, force: true })));
function cli(...args: string[]) {
const packaged = process.env.THT_WORKSPACE_TEST_CLI;
const result = spawnSync(packaged ?? process.execPath, [...(packaged ? ["workspace"] : ["--import", "tsx", resolve("src/workspace-documents-cli.ts")]), ...args, "--json"], {
encoding: "utf8", env: { ...process.env, PATH: "" },
});
return { ...result, report: result.stdout.trim() ? JSON.parse(result.stdout) : null };
}
const descriptor = "workspace:\n schema_version: 4\n id: practice\n name: Practice\n language: en\n";
const catalog = "schema_version: 1\nworkspaces:\n - id: practice\n name: Practice\n";
const corpus = [
{ name: "minimal", descriptor, catalog, valid: true },
{ name: "optional Evidence", descriptor: descriptor + "evidence:\n source:\n type: http\n uris: [https://example.com/manual.md]\n", catalog, valid: true },
{ name: "duplicate YAML key", descriptor: descriptor + " id: practice\n", catalog, valid: false },
{ name: "ambiguous document", descriptor: descriptor + "---\n" + descriptor, catalog, valid: false },
{ name: "unknown workspace field", descriptor: descriptor + " secret: VERY_SECRET_VALUE\n", catalog, valid: false },
{ name: "database binding in authored descriptor", descriptor: descriptor + "dwh: {password: VERY_SECRET_VALUE}\n", catalog, valid: false },
{ name: "duplicate catalog id", descriptor, catalog: catalog + " - id: practice\n name: Practice\n", valid: false },
{ name: "unknown catalog field", descriptor, catalog: catalog + "secret: VERY_SECRET_VALUE\n", valid: false },
{ name: "invalid catalog YAML", descriptor, catalog: catalog + "schema_version: 1\n", valid: false },
{ name: "unsafe Evidence URI", descriptor: descriptor + "evidence:\n source:\n type: http\n uris: [https://user:VERY_SECRET_VALUE@example.com/file]\n", catalog, valid: false },
];
test.each(corpus)("CLI and runtime agree: $name", (fixture) => {
const root = directory();
mkdirSync(join(root, "practice"), { recursive: true });
writeFileSync(join(root, "thoth-workspaces.yaml"), fixture.catalog);
writeFileSync(join(root, "practice/workspace.yaml"), fixture.descriptor);
let accepted = true;
try { parseWorkspaceCatalogYaml(fixture.catalog); parseWorkspaceYaml(fixture.descriptor); }
catch { accepted = false; }
expect(accepted).toBe(fixture.valid);
const checked = cli("validate", "--directory", root);
expect(checked.status).toBe(fixture.valid ? 0 : 1);
expect(checked.report.ok).toBe(accepted);
expect(checked.stdout + checked.stderr).not.toContain("VERY_SECRET_VALUE");
if (!fixture.valid) {
expect(checked.report.issues[0].document).not.toBe(".");
expect(checked.report.issues[0].correction.length).toBeGreaterThan(10);
}
});
test("prepare refuses existing directories and invalid options without changing documents", () => {
const root = directory();
expect(cli("prepare", "--directory", root, "--id", "practice", "--name", "Practice").status).toBe(0);
const before = readFileSync(join(root, "thoth-workspaces.yaml"), "utf8");
expect(cli("prepare", "--directory", root, "--id", "other", "--name", "Other").report.issues[0].code).toBe("destination_exists");
expect(readFileSync(join(root, "thoth-workspaces.yaml"), "utf8")).toBe(before);
expect(cli("validate", "--directory", root, "--typo", "VERY_SECRET_VALUE").status).toBe(2);
});
test("validation rejects directory mismatches, missing references and symlinks without following them", () => {
const root = directory();
cli("prepare", "--directory", root, "--id", "practice", "--name", "Practice");
mkdirSync(join(root, "unlisted"));
expect(cli("validate", "--directory", root).report.issues.some((i: {code: string}) => i.code === "unlisted_directory")).toBe(true);
rmSync(join(root, "unlisted"), { recursive: true });
writeFileSync(join(root, "practice/workspace.yaml"), descriptor.replace("name: Practice", "name: Different"));
expect(cli("validate", "--directory", root).report.issues[0].code).toBe("catalog_mismatch");
writeFileSync(join(root, "practice/workspace.yaml"), descriptor + "evidence:\n source:\n type: filesystem\n uri: practice/evidence\n");
expect(cli("validate", "--directory", root).report.issues[0].code).toBe("missing_reference");
symlinkSync(roots[roots.length - 1], join(root, "practice/evidence"), "dir");
expect(cli("validate", "--directory", root).report.issues[0].code).toBe("unsafe_reference");
});
test("local Evidence checks references and YAML frontmatter, and explicitly defers canonical content validation", () => {
const root = directory();
cli("prepare", "--directory", root, "--id", "practice", "--name", "Practice");
writeFileSync(join(root, "practice/workspace.yaml"), descriptor + "evidence:\n schema_version: 2\n source:\n type: filesystem\n uri: practice/evidence\n patterns: ['curated/**/*.md']\n");
mkdirSync(join(root, "practice/evidence/curated/domain"), { recursive: true });
const evidence = join(root, "practice/evidence/curated/domain/rule.md");
writeFileSync(evidence, "---\nschema_version: 4\nid: evidence:rule\nkind: domain\nlanguage: en\npurposes: [sql_generation]\n---\n# Rule\n\n## Rule\nUse the order number.\n");
const checked = cli("validate", "--directory", root);
expect(checked.status).toBe(0);
expect(checked.report.workspaces).toEqual([{ id: "practice", evidence: "local-files" }]);
expect(checked.report.deferred_checks).toContain("practice:evidence-content-provenance-and-indexing");
writeFileSync(evidence, "---\nid: first\nid: VERY_SECRET_VALUE\n---\n# Rule\n");
const invalid = cli("validate", "--directory", root);
expect(invalid.status).toBe(1);
expect(invalid.report.issues[0].document).toBe("practice/evidence/curated/domain/rule.md");
expect(invalid.stdout).not.toContain("VERY_SECRET_VALUE");
writeFileSync(join(root, "practice/workspace.yaml"), descriptor + "evidence:\n source:\n type: filesystem\n uri: practice/evidence\n patterns: [missing.md]\n");
expect(cli("validate", "--directory", root).report.issues[0].code).toBe("missing_reference");
});
test("prepare creates documents accepted by the runtime and validate is repeatable without installation or services", () => {
const root = directory();
const prepared = cli("prepare", "--directory", root, "--id", "practice", "--name", "Practice");
expect(prepared.stderr).toBe("");
expect(prepared.status).toBe(0);
expect(prepared.report.ok).toBe(true);
const catalog = readFileSync(join(root, "thoth-workspaces.yaml"), "utf8");
const descriptor = readFileSync(join(root, "practice/workspace.yaml"), "utf8");
expect(parseWorkspaceCatalogYaml(catalog).workspaces[0].id).toBe("practice");
expect(parseWorkspaceYaml(descriptor).evidence).toBeUndefined();
for (let index = 0; index < 2; index++) {
const checked = cli("validate", "--directory", root);
expect(checked.status).toBe(0);
expect(checked.report.workspaces).toEqual([{ id: "practice", evidence: "absent" }]);
}
expect(readFileSync(join(root, "thoth-workspaces.yaml"), "utf8")).toBe(catalog);
expect(readFileSync(join(root, "practice/workspace.yaml"), "utf8")).toBe(descriptor);
});
+10
View File
@@ -8,6 +8,11 @@ cp deploy/secrets/thothii.secrets.example deploy/secrets/thothii.secrets
chmod 600 deploy/secrets/thothii.secrets chmod 600 deploy/secrets/thothii.secrets
``` ```
For a normal local installation, `tht setup --complete` creates the active bundle at
`deploy/local/secrets/thothii.secrets` and creates the two Catalog password files beside it. The
generated `deploy/local/operator.env` contains only absolute paths to those files; never copy
secret values into `operator.env`.
The file uses strict `KEY=VALUE` lines (comments and blank lines are allowed). The supported The file uses strict `KEY=VALUE` lines (comments and blank lines are allowed). The supported
installation keys are `THT_MODEL_API_KEY`, `THT_DWH_API_KEY`, `THT_CA`, `THT_SSL_CA`, installation keys are `THT_MODEL_API_KEY`, `THT_DWH_API_KEY`, `THT_CA`, `THT_SSL_CA`,
`THT_OIDC_CLIENT_SECRET`, and `THT_AUTHENTIK_API_TOKEN`. Installation Model Catalog providers may `THT_OIDC_CLIENT_SECRET`, and `THT_AUTHENTIK_API_TOKEN`. Installation Model Catalog providers may
@@ -29,6 +34,11 @@ Session and metadata-generation runtimes read only the provider key named by
credential reference, not their execution lifecycle. Pi-owned authentication remains available only credential reference, not their execution lifecycle. Pi-owned authentication remains available only
to session-only built-in providers through `authentication.mode: pi_auth`. to session-only built-in providers through `authentication.mode: pi_auth`.
Workspace database credentials are intentionally not part of this global bundle. Configure each
workspace's database binding, password/token, tunnel key and CA in Database Management; the
installation stores those values in its encrypted workspace secret store. The workspace Git
repository may declare database identity and Evidence, but must never contain these credentials.
Do not add vector or embedding endpoint credentials to the bundle. Active operator manuals use Do not add vector or embedding endpoint credentials to the bundle. Active operator manuals use
internal Qdrant and Ollama services, so vector/embedding runtime endpoint secrets are not part of internal Qdrant and Ollama services, so vector/embedding runtime endpoint secrets are not part of
the supported installation contract. the supported installation contract.
+8 -3
View File
@@ -96,10 +96,15 @@ deve invece contenere `mode` e `defaultLocale`, altrimenti viene rifiutato.
## Lingua, continuità e dati ## Lingua, continuità e dati
La lingua UI traduce il testo dell'applicazione, non i contenuti di dominio. La lingua UI traduce il testo dell'applicazione, non i contenuti di dominio.
Alla creazione, la lingua UI viene acquisita come `interactionLanguage`; il Alla creazione, la lingua UI viene acquisita come `interactionLanguage` di ripiego.
manifest salva `interaction_language`, che governa domande e scelte del modello. Il CLI riconosce la lingua della domanda originale e salva `interaction_language`
nel manifest; usa il ripiego solo per input troppo brevi, ambigui o composti da codice.
Questa lingua governa domande, spiegazioni, scelte e controlli HITL. Il gate la
include nei descrittori e il frontend la applica al sottoalbero dei widget,
senza cambiare la lingua della navigazione.
Alla ripresa vale la lingua salvata, non l'ultima scelta dell'header. Per i manifest Alla ripresa vale la lingua salvata, non l'ultima scelta dell'header. Per i manifest
precedenti senza campo viene fissata la lingua workspace disponibile alla prima ripresa. precedenti senza campo viene riconosciuta e fissata la lingua della domanda,
con la lingua workspace disponibile alla prima ripresa come ripiego.
Il cambio lingua Omics invia il form Django e ricarica la pagina. ThothII conserva Il cambio lingua Omics invia il form Django e ricarica la pagina. ThothII conserva
solo l'ID della selezione in `sessionStorage`, separato per pathname, issuer e solo l'ID della selezione in `sessionStorage`, separato per pathname, issuer e
+19 -12
View File
@@ -81,20 +81,20 @@ The model proposes; a human reviewer decides at gates through widgets:
The frontend renders these widget descriptors (registry in `src/widgets/`). It rebuilds the live transcript in memory from the SSE stream (`src/store/sessionStore.ts`); it is **not persisted**. The frontend renders these widget descriptors (registry in `src/widgets/`). It rebuilds the live transcript in memory from the SSE stream (`src/store/sessionStore.ts`); it is **not persisted**.
## Curated and immutable Evidence ## Evidence sources, local authority and search projections
The workspace repository is the publication boundary. The curator prepares `evidence/source/`, The workspace repository publishes revision-pinned workspace identities and Evidence sources.
reviews units in `evidence/curated/`, validates them, and merges them. With `evidence.schema_version: 2`, An initialized editable Evidence archive is maintained locally through external editors and
the runtime materializes the full `evidence/` tree from the exact Git commit, but the renderer passes explicit consolidation; normal preprocessing or source refresh must not overwrite its manual
only `curated/**/*.md` from the immutable revision root to preprocessing. Sources, manifests, and corrections. File save, active search generation and a later human Git commit/push are separate
evaluation data remain available for traceability. The runtime never modifies, stages, commits, or outcomes. See [the current Evidence contract](../contracts/curated-evidence-v4.md) and
publishes the authoring repository. [source/publication boundaries](../contracts/workspace-evidence-v3.md).
Before indexing, the curated corpus from the pinned revision is validated. Each workspace has two Each workspace has separate Qdrant collections. `reference` contains Schema, relationships and
physical Qdrant collections with different lifecycles. `reference` contains Schema, relationships, Evidence and may be replaced or cleared by preprocessing. Memory uses its own dense/BM25
and Evidence and may be replaced or cleared by preprocessing; `memory` contains `memory` and projection, rebuilt from authoritative PostgreSQL cards rather than old vector payloads or
`solved_question` records and is never preprocessing output. Only the `reference` collection has the session artifacts. Both collections can contain sparse vectors; their ownership and cleanup
sparse `bm25` vector with `idf`, and only the Evidence stage writes sparse values. lifecycles remain separate. See [Memory](../gestione-memory.md).
The Administration control can clear the replaceable reference collection, LSH, corpus, and derived The Administration control can clear the replaceable reference collection, LSH, corpus, and derived
checkpoints. The operation preserves the memory collection and makes preprocessing required before checkpoints. The operation preserves the memory collection and makes preprocessing required before
@@ -115,6 +115,13 @@ the core can admit a new session.
## Runtime composition ## Runtime composition
PostgreSQL is the current database-metadata authority for all core consumers, not a deferred
catalog-to-core integration. The historical research on a separate catalog service and
`annotations.yaml` publication is superseded by ADRs 0004 and 0016. Preserve read-only DWH access,
installation-local bindings/secrets, revision-pinned workspace identity, and fail-closed readiness
when changing those boundaries. The exact snapshot interface is the
[Catalog Schema Snapshot contract](../contracts/catalog-schema-snapshot.md).
The local stack is started by `./scripts/run-stack.sh` after the installation descriptor and The local stack is started by `./scripts/run-stack.sh` after the installation descriptor and
`deploy/env/local.env` exist. `core` includes Pi. `catalog-db`, Qdrant, and the Ollama embedding `deploy/env/local.env` exist. `core` includes Pi. `catalog-db`, Qdrant, and the Ollama embedding
service are internal Compose services; only the DWH and model-provider endpoint remain external. service are internal Compose services; only the DWH and model-provider endpoint remain external.
Binary file not shown.

Before

Width:  |  Height:  |  Size: 132 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 131 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 189 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 186 KiB

@@ -1,32 +0,0 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>Archify automated browser evidence · thothii-core-sequence.html</title>
<style>
*{box-sizing:border-box}body{margin:0;padding:24px;background:#e9eef5;color:#172033;font:14px/1.5 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}header{max-width:1500px;margin:0 auto 18px}h1{margin:0 0 6px;font-size:20px}p{margin:0;color:#526176}.grid{max-width:1500px;margin:auto;display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:18px}figure{margin:0;padding:10px;background:white;border:1px solid #c9d4e3;border-radius:12px;box-shadow:0 10px 30px rgba(15,23,42,.08)}img{display:block;width:100%;height:auto;border:1px solid #e2e8f0}figcaption{padding:9px 4px 2px;color:#526176}@media(max-width:900px){.grid{grid-template-columns:1fr}}
</style>
</head>
<body>
<header><h1>Automated browser evidence</h1><p>thothii-core-sequence.html · visual-check containment pass · perceptual visual review pending</p></header>
<main class="grid">
<figure>
<img src="thothii-core-sequence.visual-check.1440x900.light.png" alt="light 1440 by 900">
<figcaption><strong>LIGHT</strong> · 1440×900 · containment pass</figcaption>
</figure>
<figure>
<img src="thothii-core-sequence.visual-check.1440x900.dark.png" alt="dark 1440 by 900">
<figcaption><strong>DARK</strong> · 1440×900 · containment pass</figcaption>
</figure>
<figure>
<img src="thothii-core-sequence.visual-check.2048x1320.light.png" alt="light 2048 by 1320">
<figcaption><strong>LIGHT</strong> · 2048×1320 · containment pass</figcaption>
</figure>
<figure>
<img src="thothii-core-sequence.visual-check.2048x1320.dark.png" alt="dark 2048 by 1320">
<figcaption><strong>DARK</strong> · 2048×1320 · containment pass</figcaption>
</figure>
</main>
</body>
</html>
@@ -1,548 +0,0 @@
{
"schemaVersion": 1,
"ok": true,
"command": "visual-check",
"evidenceKind": "automated-browser",
"status": "pass",
"visualReview": "pending",
"artifact": {
"path": "/Users/mp/projects/ThothII/docs/architecture/thothii-core-sequence.html",
"sha256": "b521eb942f7889cfc3a5e29010546ba59eeab1cf485c22c533d271ccef9a28d5",
"bytes": 716881
},
"state": {
"detail": "read",
"motion": "still"
},
"chrome": {
"status": "available",
"executable": "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"
},
"diagnostics": [],
"containment": {
"status": "pass",
"viewports": [
{
"width": 1440,
"height": 900,
"theme": "light",
"innerWidth": 1440,
"innerHeight": 900,
"scrollWidth": 1440,
"scrollHeight": 900,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1194,
"diagramWidth": 1164,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1600,
"height": 1000,
"theme": "light",
"innerWidth": 1600,
"innerHeight": 1000,
"scrollWidth": 1600,
"scrollHeight": 1000,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1242,
"diagramWidth": 1212,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1920,
"height": 1080,
"theme": "light",
"innerWidth": 1920,
"innerHeight": 1080,
"scrollWidth": 1920,
"scrollHeight": 1080,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1402,
"diagramWidth": 1372,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 2048,
"height": 1320,
"theme": "light",
"innerWidth": 2048,
"innerHeight": 1320,
"scrollWidth": 2048,
"scrollHeight": 1320,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1818,
"diagramWidth": 1768,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 41,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
}
]
},
"readability": {
"status": "pass",
"minimumProjectedNodeTextPx": 6,
"viewports": [
{
"width": 1440,
"height": 900,
"theme": "light",
"innerWidth": 1440,
"innerHeight": 900,
"scrollWidth": 1440,
"scrollHeight": 900,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1194,
"diagramWidth": 1164,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1600,
"height": 1000,
"theme": "light",
"innerWidth": 1600,
"innerHeight": 1000,
"scrollWidth": 1600,
"scrollHeight": 1000,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1242,
"diagramWidth": 1212,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1920,
"height": 1080,
"theme": "light",
"innerWidth": 1920,
"innerHeight": 1080,
"scrollWidth": 1920,
"scrollHeight": 1080,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1402,
"diagramWidth": 1372,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 2048,
"height": 1320,
"theme": "light",
"innerWidth": 2048,
"innerHeight": 1320,
"scrollWidth": 2048,
"scrollHeight": 1320,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1818,
"diagramWidth": 1768,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 41,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
}
]
},
"viewerChrome": {
"status": "pass",
"viewports": [
{
"width": 1440,
"height": 900,
"theme": "light",
"innerWidth": 1440,
"innerHeight": 900,
"scrollWidth": 1440,
"scrollHeight": 900,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1194,
"diagramWidth": 1164,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1600,
"height": 1000,
"theme": "light",
"innerWidth": 1600,
"innerHeight": 1000,
"scrollWidth": 1600,
"scrollHeight": 1000,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1242,
"diagramWidth": 1212,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1920,
"height": 1080,
"theme": "light",
"innerWidth": 1920,
"innerHeight": 1080,
"scrollWidth": 1920,
"scrollHeight": 1080,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1402,
"diagramWidth": 1372,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 2048,
"height": 1320,
"theme": "light",
"innerWidth": 2048,
"innerHeight": 1320,
"scrollWidth": 2048,
"scrollHeight": 1320,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1818,
"diagramWidth": 1768,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 41,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
}
]
},
"captures": {
"status": "pass",
"screenshots": [
{
"width": 1440,
"height": 900,
"theme": "light",
"innerWidth": 1440,
"innerHeight": 900,
"scrollWidth": 1440,
"scrollHeight": 900,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1194,
"diagramWidth": 1164,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light",
"file": "thothii-core-sequence.visual-check.1440x900.light.png"
},
{
"width": 1440,
"height": 900,
"theme": "dark",
"innerWidth": 1440,
"innerHeight": 900,
"scrollWidth": 1440,
"scrollHeight": 900,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1194,
"diagramWidth": 1164,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "dark",
"file": "thothii-core-sequence.visual-check.1440x900.dark.png"
},
{
"width": 2048,
"height": 1320,
"theme": "light",
"innerWidth": 2048,
"innerHeight": 1320,
"scrollWidth": 2048,
"scrollHeight": 1320,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1818,
"diagramWidth": 1768,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 41,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light",
"file": "thothii-core-sequence.visual-check.2048x1320.light.png"
},
{
"width": 2048,
"height": 1320,
"theme": "dark",
"innerWidth": 2048,
"innerHeight": 1320,
"scrollWidth": 2048,
"scrollHeight": 1320,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1818,
"diagramWidth": 1768,
"viewBoxWidth": 1080,
"minimumProjectedNodeTextPx": 7,
"minimumProjectedNodeText": "browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 41,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "dark",
"file": "thothii-core-sequence.visual-check.2048x1320.dark.png"
}
],
"contactSheet": "thothii-core-sequence.visual-check.html"
},
"sidecars": {
"receipt": "thothii-core-sequence.visual-check.json",
"contactSheet": "thothii-core-sequence.visual-check.html"
}
}
Binary file not shown.

Before

Width:  |  Height:  |  Size: 137 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 136 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 183 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 182 KiB

@@ -1,32 +0,0 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>Archify automated browser evidence · thothii-runtime.html</title>
<style>
*{box-sizing:border-box}body{margin:0;padding:24px;background:#e9eef5;color:#172033;font:14px/1.5 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}header{max-width:1500px;margin:0 auto 18px}h1{margin:0 0 6px;font-size:20px}p{margin:0;color:#526176}.grid{max-width:1500px;margin:auto;display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:18px}figure{margin:0;padding:10px;background:white;border:1px solid #c9d4e3;border-radius:12px;box-shadow:0 10px 30px rgba(15,23,42,.08)}img{display:block;width:100%;height:auto;border:1px solid #e2e8f0}figcaption{padding:9px 4px 2px;color:#526176}@media(max-width:900px){.grid{grid-template-columns:1fr}}
</style>
</head>
<body>
<header><h1>Automated browser evidence</h1><p>thothii-runtime.html · visual-check containment pass · perceptual visual review pending</p></header>
<main class="grid">
<figure>
<img src="thothii-runtime.visual-check.1440x900.light.png" alt="light 1440 by 900">
<figcaption><strong>LIGHT</strong> · 1440×900 · containment pass</figcaption>
</figure>
<figure>
<img src="thothii-runtime.visual-check.1440x900.dark.png" alt="dark 1440 by 900">
<figcaption><strong>DARK</strong> · 1440×900 · containment pass</figcaption>
</figure>
<figure>
<img src="thothii-runtime.visual-check.2048x1320.light.png" alt="light 2048 by 1320">
<figcaption><strong>LIGHT</strong> · 2048×1320 · containment pass</figcaption>
</figure>
<figure>
<img src="thothii-runtime.visual-check.2048x1320.dark.png" alt="dark 2048 by 1320">
<figcaption><strong>DARK</strong> · 2048×1320 · containment pass</figcaption>
</figure>
</main>
</body>
</html>
@@ -1,548 +0,0 @@
{
"schemaVersion": 1,
"ok": true,
"command": "visual-check",
"evidenceKind": "automated-browser",
"status": "pass",
"visualReview": "pending",
"artifact": {
"path": "/Users/mp/projects/ThothII/docs/architecture/thothii-runtime.html",
"sha256": "ad19195852c7c643228663e5bf7f3cb273afb0456fedbe295d4df24bc345b6c7",
"bytes": 724277
},
"state": {
"detail": "read",
"motion": "still"
},
"chrome": {
"status": "available",
"executable": "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"
},
"diagnostics": [],
"containment": {
"status": "pass",
"viewports": [
{
"width": 1440,
"height": 900,
"theme": "light",
"innerWidth": 1440,
"innerHeight": 900,
"scrollWidth": 1440,
"scrollHeight": 900,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 968,
"diagramWidth": 938,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 6.20735294117647,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1600,
"height": 1000,
"theme": "light",
"innerWidth": 1600,
"innerHeight": 1000,
"scrollWidth": 1600,
"scrollHeight": 1000,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1009,
"diagramWidth": 979,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 6.478676470588235,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1920,
"height": 1080,
"theme": "light",
"innerWidth": 1920,
"innerHeight": 1080,
"scrollWidth": 1920,
"scrollHeight": 1080,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1172,
"diagramWidth": 1142,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 7.557352941176471,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 2048,
"height": 1320,
"theme": "light",
"innerWidth": 2048,
"innerHeight": 1320,
"scrollWidth": 2048,
"scrollHeight": 1320,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1583,
"diagramWidth": 1533,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 9,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 41,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
}
]
},
"readability": {
"status": "pass",
"minimumProjectedNodeTextPx": 6,
"viewports": [
{
"width": 1440,
"height": 900,
"theme": "light",
"innerWidth": 1440,
"innerHeight": 900,
"scrollWidth": 1440,
"scrollHeight": 900,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 968,
"diagramWidth": 938,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 6.20735294117647,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1600,
"height": 1000,
"theme": "light",
"innerWidth": 1600,
"innerHeight": 1000,
"scrollWidth": 1600,
"scrollHeight": 1000,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1009,
"diagramWidth": 979,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 6.478676470588235,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1920,
"height": 1080,
"theme": "light",
"innerWidth": 1920,
"innerHeight": 1080,
"scrollWidth": 1920,
"scrollHeight": 1080,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1172,
"diagramWidth": 1142,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 7.557352941176471,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 2048,
"height": 1320,
"theme": "light",
"innerWidth": 2048,
"innerHeight": 1320,
"scrollWidth": 2048,
"scrollHeight": 1320,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1583,
"diagramWidth": 1533,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 9,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 41,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
}
]
},
"viewerChrome": {
"status": "pass",
"viewports": [
{
"width": 1440,
"height": 900,
"theme": "light",
"innerWidth": 1440,
"innerHeight": 900,
"scrollWidth": 1440,
"scrollHeight": 900,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 968,
"diagramWidth": 938,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 6.20735294117647,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1600,
"height": 1000,
"theme": "light",
"innerWidth": 1600,
"innerHeight": 1000,
"scrollWidth": 1600,
"scrollHeight": 1000,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1009,
"diagramWidth": 979,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 6.478676470588235,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 1920,
"height": 1080,
"theme": "light",
"innerWidth": 1920,
"innerHeight": 1080,
"scrollWidth": 1920,
"scrollHeight": 1080,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1172,
"diagramWidth": 1142,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 7.557352941176471,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
},
{
"width": 2048,
"height": 1320,
"theme": "light",
"innerWidth": 2048,
"innerHeight": 1320,
"scrollWidth": 2048,
"scrollHeight": 1320,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1583,
"diagramWidth": 1533,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 9,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 41,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light"
}
]
},
"captures": {
"status": "pass",
"screenshots": [
{
"width": 1440,
"height": 900,
"theme": "light",
"innerWidth": 1440,
"innerHeight": 900,
"scrollWidth": 1440,
"scrollHeight": 900,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 968,
"diagramWidth": 938,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 6.20735294117647,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light",
"file": "thothii-runtime.visual-check.1440x900.light.png"
},
{
"width": 1440,
"height": 900,
"theme": "dark",
"innerWidth": 1440,
"innerHeight": 900,
"scrollWidth": 1440,
"scrollHeight": 900,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 968,
"diagramWidth": 938,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 6.20735294117647,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 51,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "dark",
"file": "thothii-runtime.visual-check.1440x900.dark.png"
},
{
"width": 2048,
"height": 1320,
"theme": "light",
"innerWidth": 2048,
"innerHeight": 1320,
"scrollWidth": 2048,
"scrollHeight": 1320,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1583,
"diagramWidth": 1533,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 9,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 41,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "light",
"file": "thothii-runtime.visual-check.2048x1320.light.png"
},
{
"width": 2048,
"height": 1320,
"theme": "dark",
"innerWidth": 2048,
"innerHeight": 1320,
"scrollWidth": 2048,
"scrollHeight": 1320,
"overflowX": false,
"overflowY": false,
"ok": true,
"readerWidth": 1583,
"diagramWidth": 1533,
"viewBoxWidth": 1360,
"minimumProjectedNodeTextPx": 9,
"minimumProjectedNodeText": "Browser",
"minimumProjectedNodeTextDetail": "context",
"minimumRequiredNodeTextPx": 6,
"readabilityOk": true,
"hasLegend": true,
"hasNavigationDock": true,
"legendDockIntersectionArea": 0,
"dockStageIntersectionArea": 0,
"dockStageGap": 10.21875,
"requiredDockStageGap": 10,
"viewerChromeStageOk": true,
"viewerChromeReserve": 41,
"viewerChromeActive": true,
"viewerChromeOk": true,
"resolvedTheme": "dark",
"file": "thothii-runtime.visual-check.2048x1320.dark.png"
}
],
"contactSheet": "thothii-runtime.visual-check.html"
},
"sidecars": {
"receipt": "thothii-runtime.visual-check.json",
"contactSheet": "thothii-runtime.visual-check.html"
}
}
+2 -2
View File
@@ -244,5 +244,5 @@ The first installed consolidation performs this conversion automatically when th
legacy manifest is present, then validates and activates the result. Preserve the legacy manifest is present, then validates and activates the result. Preserve the
existing checkout in backups before upgrading. The E1 validation used an isolated existing checkout in backups before upgrading. The E1 validation used an isolated
copy; E2 also converted and indexed all 35 units on the running local preview. copy; E2 also converted and indexed all 35 units on the running local preview.
See the [E1 validation report](../plans/2026-09-09-evidence-e1-validation.md) and See the [E1 validation report](../reports/knowledge-archives-release.md) and
[E2 validation report](../plans/2026-09-09-evidence-e2-validation.md). [E2 validation report](../reports/knowledge-archives-release.md).
+6 -6
View File
@@ -4,7 +4,7 @@ This page describes the complete workspace Evidence lifecycle: where original ma
## Editable local Evidence ## Editable local Evidence
E1 adds [Curated Evidence v4 and a persistent local archive](contracts/curated-evidence-v4.md). E1 adds [Curated Evidence v4 and a persistent local archive](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/curated-evidence-v4.md).
The visible title and payload fields are authoritative Markdown. New manual units need no The visible title and payload fields are authoritative Markdown. New manual units need no
external source; consolidation records their curator and distinguishes later corrections external source; consolidation records their curator and distinguishes later corrections
from original documentary provenance. The core archive API creates immutable candidates from original documentary provenance. The core archive API creates immutable candidates
@@ -14,12 +14,12 @@ E2 adds **Administration → Evidence management**, actual host file paths, comp
browsing and filtering, and the installed `tht workspace evidence consolidate browsing and filtering, and the installed `tht workspace evidence consolidate
--workspace <id>` command. Edit files externally, consolidate to activate them, then --workspace <id>` command. Edit files externally, consolidate to activate them, then
review and run Git manually. Runtime consumes only the active local snapshot. review and run Git manually. Runtime consumes only the active local snapshot.
The [v4 contract](contracts/curated-evidence-v4.md#administration-and-installed-command) The [v4 contract](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/curated-evidence-v4.md#administration-and-installed-command)
describes host mounting, first conversion, failure recovery and Clear behavior. describes host mounting, first conversion, failure recovery and Clear behavior.
The local PSD preview has 35 converted and indexed units. E3 adds explicit import/refresh The local PSD preview has 35 converted and indexed units. E3 adds explicit import/refresh
from local drafts and configured HTTP/S3 sources, durable current/proposed comparisons, from local drafts and configured HTTP/S3 sources, durable current/proposed comparisons,
and keep/replace decisions with activation and retry. See and keep/replace decisions with activation and retry. See
[Import drafts and refresh sources](contracts/curated-evidence-v4.md#import-drafts-and-refresh-sources). [Import drafts and refresh sources](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/curated-evidence-v4.md#import-drafts-and-refresh-sources).
Workflow gate corrections remain the subsequent shared increment, X1. Workflow gate corrections remain the subsequent shared increment, X1.
## Existing repository publication path ## Existing repository publication path
@@ -92,7 +92,7 @@ Join orders using the order number, financial year and company.
Use `tht evidence migrate <workspace-root>` for deterministic legacy conversion. The Use `tht evidence migrate <workspace-root>` for deterministic legacy conversion. The
conversion preserves typed content and initializes an archive baseline; it does not conversion preserves typed content and initializes an archive baseline; it does not
activate the local corpus. See the [v4 contract](contracts/curated-evidence-v4.md) for activate the local corpus. See the [v4 contract](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/curated-evidence-v4.md) for
all eight kinds, provenance, file layout and the E1/E2 boundary. all eight kinds, provenance, file layout and the E1/E2 boundary.
## Legacy v3 representation ## Legacy v3 representation
@@ -262,5 +262,5 @@ Formulas use a format distinct from document Evidence. A formula proposed during
## Contract references ## Contract references
- [Workspace Evidence v3 contract](contracts/workspace-evidence-v3.md) - [Workspace Evidence v3 contract](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/workspace-evidence-v3.md)
- [Preprocessing CLI contract](contracts/workspace-preprocessing-cli.md) - [Preprocessing CLI contract](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/workspace-preprocessing-cli.md)
+45
View File
@@ -128,6 +128,51 @@ endpoint expects a model name different from the catalog key.
Provider integrations remain declarative. Do not register providers from Provider integrations remain declarative. Do not register providers from
`harness/.pi/extensions/`; those extensions implement the workflow and human gates only. `harness/.pi/extensions/`; those extensions implement the workflow and human gates only.
### Qwen 3.6 sessions and thinking controls
For Qwen served through a vLLM-compatible chat template, declare the following inside the
model's `session` block, alongside its context and output limits:
```yaml
reasoning: true
compatibility:
supportsDeveloperRole: false
supportsReasoningEffort: false
supportsStore: false
maxTokensField: max_tokens
thinkingFormat: qwen-chat-template
```
This makes Pi send `chat_template_kwargs.enable_thinking` from the selected thinking level,
with `preserve_thinking: true`. Choose **off** to explicitly disable thinking. The alternative
`thinkingFormat: qwen` is for endpoints expecting top-level `enable_thinking`. Both formats
require `reasoning: true`; declaring `reasoning: false` does not tell the server to disable
thinking. Omit `thinkingFormat` to preserve Pi's default behavior for other providers.
Regenerate projections with the updated host CLI and recreate the local core container after
rebuilding it. Do not add these fields directly to generated Pi files. These controls do not
force tool calls or certify the workflow; perform the operator verification above.
```sh
tht --installation /absolute/path/thothii-installation.yaml installation generate
```
Use the model identifier exposed by your endpoint, such as `qwen3.6-35b-a3b`, and limits
supported by that deployment. The thinking format configures the Pi session adapter;
metadata generation continues to use its separate LiteLLM settings.
If a session displays text such as `{"type":"bash","command":"tht session show … --json"}`
and never opens a review widget, that text is not an executed tool call. A verified cause
was the Evidence JSON extension being loaded into interactive sessions and forcing
`response_format: {type: "json_object"}`. Upgrade to the core image containing the fix:
the extension belongs in `.pi/evidence-extensions/` and is loaded explicitly only by
Evidence authoring. It must not also remain in the automatically loaded `.pi/extensions/`
directory. Regenerating model configuration alone does not remove an extension from an old image.
After upgrading, reload the browser and resume the session. Verify that Pi executes
`tht session show` and opens a review widget. This fix does not require changing the Qwen
server, forcing every turn to call a tool, or teaching the model to print tool-call JSON.
## Authentication ## Authentication
Every provider chooses one explicit mode: Every provider chooses one explicit mode:
+1 -1
View File
@@ -226,5 +226,5 @@ error. Preparation is repeatable and checks migration checksums.
See [the M1 specification](plans/2026-09-08-memory-m1-spec.md) and See [the M1 specification](plans/2026-09-08-memory-m1-spec.md) and
[ADR 0018](adr/0018-use-postgres-for-memory-and-qdrant-for-retrieval.md) for the [ADR 0018](adr/0018-use-postgres-for-memory-and-qdrant-for-retrieval.md) for the
approved scope and acceptance boundaries. approved scope and acceptance boundaries.
See [M2 implementation and validation](plans/2026-09-09-memory-m2-validation.md) See [M2 implementation and validation](reports/knowledge-archives-release.md)
for the retrieval checks and real embedding test command. for the retrieval checks and real embedding test command.
+1 -1
View File
@@ -90,4 +90,4 @@ happen from the workflow’s point of view.
session does not become Evidence automatically: a curator must review and publish it in Git. session does not become Evidence automatically: a curator must review and publish it in Git.
- For login and access recovery, use [local authentication](install/authentication-local.md) or - For login and access recovery, use [local authentication](install/authentication-local.md) or
[OIDC authentication](install/authentication-oidc.md) for full, or contact the [OIDC authentication](install/authentication-oidc.md) for full, or contact the
portal administrator for [embedded/upstream access](install/authentication-upstream.md). portal administrator for [embedded/upstream access](install/shell-and-language.md).
+24 -26
View File
@@ -1,35 +1,33 @@
# ThothII documentation # ThothII documentation
ThothII is a human-reviewed datamart builder. It turns a question into validated SQL through an ThothII turns a natural-language question into reviewed SQL. The model proposes;
eight-phase workflow: the model proposes; a reviewer makes the decisions that are persisted. a human reviewer decides which interpretations, sources and results to accept.
Start with the path that matches the work you need to do: This is the public product manual. Start with the task you need to perform:
| I need to… | Start here | | I need to… | Start here |
| --- | --- | | --- | --- |
| Install or operate one instance | [Install and first start](install/first-start.md) | | Understand the product and its boundaries | [What ThothII does](product-overview.md) |
| Clone and manually install on macOS, Windows, or Linux | [Italian procedure](install/standalone-manual-it.md) · [English procedure](install/standalone-manual-en.md) | | Install on Mac, Windows through WSL2, or Linux | [Italian procedure](install/standalone-manual-it.md) · [English procedure](install/standalone-manual-en.md) |
| Upgrade the server and integrate Omics Portal | [Codex server handoff](operations/server-codex-handoff.md) | | Choose display mode and language | [Display mode and language](install/shell-and-language.md) |
| Choose full/embedded and configure the shell | [Shell and localization](operations/shell-and-localization.md) | | Configure login | [Local accounts](install/authentication-local.md) · [OIDC](install/authentication-oidc.md) |
| Reuse an authenticated server portal without a second login | [Upstream authentication](install/authentication-upstream.md) | | Configure model providers | [Model configuration](general/pi-configuration.md) |
| Understand how the two renderings share the same application | [Rendering architecture](architecture/application-shell.md) | | Prepare a domain workspace | [Workspaces](operations/workspaces.md) |
| Validate login, logout and portal integration before release | [Acceptance matrix](testing/authentication-manual-acceptance.md) | | Configure databases and reviewed descriptions | [Database management](operations/database-management.md) |
| Add, update, or prepare a workspace | [Workspace operations](operations/workspaces.md) | | Ask a question and review SQL | [User guide](guida-utente.md) · [Workflow](skills.md) |
| Ask a question and review the SQL workflow | [User guide](guida-utente.md) | | Maintain domain knowledge | [Evidence](evidence.md) · [Memory](usage/memory.md) |
| Configure and refresh an authoritative database catalog | [Database management](operations/database-management.md) |
| Author and publish domain Evidence | [Evidence](evidence.md) |
| Understand boundaries and persistence | [Architecture overview](architecture/overview.md) |
## How the documentation is organised ## Scope of this manual
- **Install and operate** documents host-side setup, authentication, lifecycle, workspaces, and The manual covers the product, installation, use and administration. Architecture,
Pi administration. code contracts, design decisions, implementation plans, test reports and site-specific
- **Use ThothII** documents the two application paths: reviewed NL→SQL sessions and administrative deployment handoffs are developer/project material maintained in the repository;
database management. they are not pages of this site and are not included in its search index.
- **Architecture and contracts** explain why the system behaves as it does and define the
machine-facing boundaries. Consult them when integrating or changing an implementation; they
are not a substitute for an operator runbook.
The host-side `tht` command is the operator interface. The Python `tht` inside `harness/` is a The installation procedures distinguish checked documentation from platform and
different, internal workflow CLI invoked by the core. In examples, use an absolute installation functional tests that still require execution. A clone does not transfer another
descriptor path whenever discovery is not unambiguous. installation's credentials, data or network access.
The host-side `tht` command operates an installation. The Python workflow CLI inside
the runtime is a separate internal interface; do not substitute its commands for
the host installation procedure.
+1 -1
View File
@@ -3,7 +3,7 @@
Use local mode for a standalone PC or Mac, with `shell.mode: full` and Use local mode for a standalone PC or Mac, with `shell.mode: full` and
`shell.defaultLocale: en` in the installation descriptor. Presentation and `shell.defaultLocale: en` in the installation descriptor. Presentation and
authentication are independent: selecting full does not create accounts. Omics authentication are independent: selecting full does not create accounts. Omics
embedded instead uses the [upstream guide](authentication-upstream.md), not local users. embedded instead uses the [upstream guide](shell-and-language.md), not local users.
Configure local authentication through `tht`; passwords are entered at an Configure local authentication through `tht`; passwords are entered at an
echo-free prompt or read from a protected `--password-file`, never from a command argument. echo-free prompt or read from a protected `--password-file`, never from a command argument.
+3 -3
View File
@@ -2,7 +2,7 @@
Use this guide for **ThothII's own login**, normally `shell.mode: full` on an Use this guide for **ThothII's own login**, normally `shell.mode: full` on an
autonomous server. It is not the integration procedure for an already logged-in autonomous server. It is not the integration procedure for an already logged-in
Omics user. That deployment uses [embedded/upstream](authentication-upstream.md), Omics user. That deployment uses [embedded/upstream](shell-and-language.md),
even when Omics's identity provider is Authentik. even when Omics's identity provider is Authentik.
OIDC mode supports a standards-based provider. The browser flow is generic: Authorization Code, OIDC mode supports a standards-based provider. The browser flow is generic: Authorization Code,
@@ -104,7 +104,7 @@ Any authentication failure prevents activation according to the static or live s
relevant diagnostic surface. relevant diagnostic surface.
The complete closed diagnostic-code union and exact role-to-permission expansion are in the The complete closed diagnostic-code union and exact role-to-permission expansion are in the
[authentication architecture](../architecture/authentication.md). [authentication architecture](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/architecture/authentication.md).
## Browser login and logout ## Browser login and logout
@@ -114,4 +114,4 @@ password prompt, but this remains a distinct ThothII login/session, unlike Omics
upstream. Full's name menu logs out of ThothII only. It does not revoke the upstream. Full's name menu logs out of ThothII only. It does not revoke the
provider session or log out other applications, so a subsequent login can return provider session or log out other applications, so a subsequent login can return
immediately through SSO. No provider token is placed in the UI adapter or browser immediately through SSO. No provider token is placed in the UI adapter or browser
storage. See the [manual acceptance matrix](../testing/authentication-manual-acceptance.md). storage. See the [manual acceptance matrix](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/testing/authentication-manual-acceptance.md).
+1 -1
View File
@@ -8,7 +8,7 @@ For **embedded in Omics**, retain Omics's existing Authentik authentication and
configure ThothII as upstream. Omics verifies `datamart_builder.access` and configure ThothII as upstream. Omics verifies `datamart_builder.access` and
administrator status and the proxy supplies the identity; no additional ThothII administrator status and the proxy supplies the identity; no additional ThothII
OIDC client, login or local user is required for that path. Follow the OIDC client, login or local user is required for that path. Follow the
[portal integration guide](authentication-upstream.md). [portal integration guide](shell-and-language.md).
```mermaid ```mermaid
sequenceDiagram sequenceDiagram
+45 -80
View File
@@ -1,98 +1,63 @@
# Install and first start # Install and first start
For the ordered clone-to-start procedure on Mac, Windows WSL2 and Linux, use the Use the guided procedure for a fresh installation:
[Italian manual guide](standalone-manual-it.md) or [English manual guide](standalone-manual-en.md),
including protected credentials and the explicit initial catalog migration.
This is the supported local installation path. It creates an installation-local configuration and - [Italian guided installation](standalone-manual-it.md)
starts the Compose stack; it does not create a workspace repository or a database catalog entry. - [English guided installation](standalone-manual-en.md)
## Prerequisites and boundaries The procedure covers Windows through Ubuntu WSL2, macOS, and Linux including an Omarchy/Arch-like
host. It uses the application clone, one installation secret bundle, protected repository
credentials when needed, and a terminal command. No host Node.js, Python or Pi installation is
required.
Install Docker Engine with Compose v2, plus the host `tht` command. On macOS or Linux, install the ## What must be ready
host command from the repository with `./scripts/install-tht.sh`; Windows uses
`./scripts/install-tht.ps1`. The installer builds or verifies the native command and checks that
`tht` is resolvable on `PATH`.
The stack contains `frontend`, `core`, `catalog-db`, `qdrant`, `embedding`, and the one-shot You need Docker with Compose v2, Git, Bash, curl, OpenSSL and shasum. You also need access to the
`embedding-model-init` and `catalog-migrate` services. DWH and model-provider endpoints are workspace repository and the values supplied by its owner: repository URL/branch, DWH endpoint,
external installation settings. Pi runs inside `core`; do not install a host Pi executable for database/schema, transport, credentials or certificates, Evidence credentials when applicable,
the application runtime. and LLM provider/API-key information.
Secrets, certificates, Pi authentication, and workspace endpoint bindings are protected The workspace repository and the THothII application repository are different. A workspace
installation-local files. Never put them in a workspace descriptor, an env file intended for descriptor may declare Evidence, but database passwords and installation bindings are stored in
version control, a URL, or a command line. the installation Catalog, not in Git.
## Create the local installation ## One guided command
From the repository root, start the interactive setup and select the local profile: After cloning THothII, checking prerequisites and installing tht, run:
```sh ~~~
tht setup --profile local --shell-mode full --shell-default-locale en tht setup --complete --profile local --shell-mode full --shell-default-locale en
``` ~~~
It writes the selected non-secret descriptor below `deploy/<installation-id>/`, the associated The first run creates protected placeholders under deploy/local/secrets/. Fill the required
operator env file, and can create protected secret templates. Keep the descriptor path: pass it credential files and rerun the same command. The command validates the local files and paths,
to commands as `--installation /absolute/path/thothii-installation.yaml` when more than one renders Compose, builds the images, starts catalog-db, runs catalog-migrate, starts the full
installation can be discovered. stack, and pulls/activates the workspace repository. Evidence source files declared by the
workspace are imported during activation.
The explicit shell options are important: compatibility defaults without them For an already configured installation:
are embedded/en/omics-portal, which expects an Omics document. The Mac's standalone
installation must use full, with English as its initial locale. Existing browser
language preferences take precedence over that initial value. Full does not
configure authentication; setup separately bootstraps local login.
For a server portal, use the [embedded/upstream procedure](authentication-upstream.md) ~~~
instead of creating a second ThothII login. For a standalone server, use full
with [direct OIDC](authentication-oidc.md). Shell mode does not follow `profile`
automatically. See [configuration and regeneration](../operations/shell-and-localization.md)
before modifying an existing installation.
If the descriptor is prepared manually instead, begin with
[`thothii-installation.local.yaml`](examples/thothii-installation.local.yaml), set mode `0600` or
`0400`, and ensure `THT_INSTALLATION_CONFIG_SOURCE` in the selected env file points to that exact
file. Create the secret bundle from `deploy/secrets/thothii.secrets.example`, protect it, and set
the file locations and external endpoints in the env file. The required settings include:
- `PI_AUTH_FILE`, `THT_SECRETS_FILE`, and the catalog password source files;
- `THT_INSTALLATION_CONFIG_SOURCE` and the workspace Git remote/branch;
- the DWH and model-provider endpoints; and
- `THT_AUTH_CONFIG_ROOT` for local authentication or the OIDC configuration selected during setup.
For the supported secret names and the metadata-generation model credential boundary, see the
`deploy/secrets/README.md` file in the installation checkout. It is intentionally not published
as a documentation page because it describes a protected local-file contract.
## Start and verify
For the normal local path, use the launcher:
```sh
./scripts/run-stack.sh
```
It builds `core`, starts `catalog-db`, runs `catalog-migrate`, then keeps the base plus local
Compose profile in the foreground. Database migrations are deliberately not a hidden backend
startup action. Open `http://127.0.0.1:8080` unless `THOTH_HTTP_PORT` was changed.
In another terminal, verify the installation without changing it:
```sh
tht --installation /absolute/path/thothii-installation.yaml doctor --json
tht --installation /absolute/path/thothii-installation.yaml status tht --installation /absolute/path/thothii-installation.yaml status
``` tht --installation /absolute/path/thothii-installation.yaml doctor --json
tht --installation /absolute/path/thothii-installation.yaml workspace test --json
~~~
`/health` verifies application-process readiness; `tht doctor` is the diagnostic surface for doctor --json is the non-destructive general core test. workspace test also probes the configured
Compose, configuration, workspace, workflow, and Pi prerequisites. database, Evidence, Qdrant and embedding service for every active workspace. It requires the
workspace database to have been configured in Database Management first.
## Routine lifecycle and next steps ## Installer-only completion
Use `tht start [--build]`, `tht stop`, `tht logs`, and `tht doctor` rather than composing ad-hoc The installer must still decide which LLMs and API keys are approved, configure and test each
container commands. Named volumes retain settings, Pi state, workspace registry, sessions, workspace database, synchronize its schema, generate and consolidate descriptions, create Qdrant
Qdrant data, and embedding models across `docker compose down`; removing them requires the entries, review naming-based FK suggestions alongside schema FKs, and load the approved
explicit destructive `--volumes` form. relationships. The final declaration of completeness requires green doctor and workspace test
results plus one real natural-language question completed through final SQL.
After the stack is healthy, configure authentication if setup did not do so, then continue with Migrations are part of setup --complete. Do not mix this installation’s descriptor or volumes with
[Workspace operations](../operations/workspaces.md). For server profile, reverse proxy, backups, a different Compose environment. Preserve the descriptor, credentials, Catalog and persistent
and recovery, use the deployment program and its manual gates; the server profile is not a volumes; do not use docker compose down --volumes as a routine stop.
drop-in replacement for the local command above.
See the guides for the Windows/macOS/Linux prerequisite matrix, workspace repository explanation,
secret layout and Gate A/Gate B acceptance checks.
+91
View File
@@ -0,0 +1,91 @@
# Installation preflight and release manifest
The native operator protocol is version 1. `installation preflight` is the independent
step-3 host check. `installation plan` repeats document validation, performs step-5
live checks and writes an owner-only JSON plan plus a separate owner-only `.key` file.
Neither command creates containers, imports bindings, modifies a database or invokes
model generation. Exit codes are 0 (checks passed), 1 (blocking error), and 2 (usage).
JSON stdout contains only the report. Checks carry stable `id`, `outcome`, `field`
and `action`; outcomes are `passed`, `error`, `warning`, `deferred-to-runtime`.
## Release manifest schema 1
The manifest is a local JSON file shipped with the verified operator/release bundle.
Required keys:
| Key | Contract |
| --- | --- |
| `schema_version` | `1` |
| `version` | Semantic release version, optionally prerelease |
| `revision` | 40 lowercase hexadecimal Git commit characters |
| `validator_protocol` | `1`; incompatible consumers refuse the manifest |
| `requirements` | `cpus`, `memory_bytes`, `disk_bytes`; at least 2 CPUs, 4 GiB Docker memory and 10 GiB installation filesystem space |
| `components` | Includes `pi`, `catalog-migrations`, `workspace-maintenance` |
| `images` | Exactly `core`, `frontend`, `catalog`, `qdrant`, `embedding`; each maps `linux/amd64` and/or `linux/arm64` to a `docker.io/...@sha256:...` **single-platform image digest** |
| `files` | Relative packaged resource paths to SHA-256; no traversal, links or absolute paths; maximum 256 files, 32 MiB per resource |
| `compose` | Ordered relative Compose file paths present in `files` for this release configuration |
Include the selected `deploy/compose.git-https.yaml` or `deploy/compose.git-ssh.yaml`
transport overlay in `files`. Standard transport overlays are resolved from the
release while absent in the installation directory; existing authored overrides
remain input files. Compose must resolve all eight services: core, frontend,
catalog-db, catalog-migrate, workspace-maintenance, qdrant, embedding and
embedding-model-init. The two maintenance services share the core digest; embedding
initialization shares the embedding digest. Source builds and undeclared services
are rejected in this prebuilt path. The explicit source path is a separate ticket.
`docker manifest inspect --verbose` checks each selected immutable image and its
platform without pulling layers. `docker compose config --format json` checks the
effective service configuration. Compose receives only Docker connection/trust,
proxy and executable-discovery host variables; application parameters come from
the prepared environment file. Raw Docker output is never copied into reports.
Each Docker command has a 15-second bound. No release is currently certified merely
because controlled manifest tests pass: publication and real pull acceptance belong
to the publication/execution tickets.
## External checks and bounds
- Git HTTPS: authenticated `GET /info/refs?service=git-upload-pack`, configured CA,
no redirects, selected branch advertised, 1 MiB response and 5-second bound.
Git SSH uses its prepared key/known-hosts and `git-upload-pack --advertise-refs`
with the same response/time bounds; no checkout or push occurs.
- PostgreSQL: the existing Catalog diagnostic adapter authenticates and reads
`current_database()` plus schema `USAGE`; no user tables are modified.
- REST database transport: existing Catalog diagnostic `GET` with the configured
bearer/API-key header and status validation. SSH database bindings cannot pass
this NL-to-SQL installation plan because runtime sessions do not support them.
- Evidence: local paths were already validated. HTTP performs bounded GET requests
and cancels response bodies; signed URL identities must match authored provenance.
S3 performs one `ListObjectsV2` request with `MaxKeys: 1`, no retries, explicit
file credentials and the canonical Evidence egress policy. These metadata requests
can incur normal remote-service request charges; they do not run LLM generation.
Each external request has a 5-second bound; the native database/Evidence helper
has a 60-second aggregate bound. Correct unavailable services before repeating.
- Explicit model endpoints: DNS/TCP/TLS origin reachability, without generating
tokens. Built-in endpoint resolution, provider authentication and model smoke
operations use the bundled Pi SDK at runtime; the report never claims those
operations have already passed.
No unreachable configured external dependency is converted to a deferred success.
Runtime obligations have explicit identities: `container-network`,
`catalog-initialization`, `pi-operation`, `local-embedding`,
`workspace-preprocessing`, `workspace-readiness`. These must be discharged by the
execution/readiness tickets before final success. Host disk inspection cannot prove
Docker Desktop VM free storage; its separate storage warning remains explicit.
## Input identity and freshness
The plan records normalized installation configuration, release digests, validator
build identity, verified input paths, local workspace content and its Git HEAD when
available (otherwise a content snapshot). Files are bounded to 32 MiB each and
256 MiB total; workspace/auth trees to 10,000 entries, without links or special files.
Credential contents are never serialized. An HMAC covers the private inputs and
plan using a separate random 32-byte owner-only key; no public unkeyed secret hash
is generated. Credential rotation invalidates the plan. Added/deleted/changed
workspace files invalidate it as well. Changes during the checks abort publication.
Plan output never overwrites an existing plan or key.
Execution and resumption must call `VerifyPlanInputs` **and repeat live checks and
credential reads before mutations**. A valid seal alone does not certify current
network availability, Docker state or runtime readiness. Keep both plan files
private and outside the workspace repository; they are installation-local artifacts.
+104
View File
@@ -0,0 +1,104 @@
# Pubblicare immagini e pacchetto operatore / Publish images and operator bundle
## Italiano
Questo comando è riservato al manutentore. L'utente finale scarica il pacchetto e
le immagini già compilati. La prima prerelease riguarda **Linux amd64**, utilizzato
da Ubuntu WSL2 su Windows e successivamente da Omarchy; non certifica ancora il
percorso completo di installazione o i collaudi manuali.
Prerequisiti del manutentore: Git, Node 24/npm, Go indicato in `tools/tht/go.mod`,
Docker con Buildx e capacità di eseguire Linux amd64, `tar`. Bun viene installato
dal lock npm e serve soltanto per compilare il pacchetto. Il commit da pubblicare
deve essere già disponibile sul repository Gitea pubblico.
Eseguire il produttore su Linux, WSL2 o macOS; la shell Windows nativa non è supportata.
La prima [prerelease Linux amd64](https://git.tylconsulting.it/mptyl/ThothII/releases/tag/installation-v0.1.0-install-preview.1)
è disponibile: `0.1.0-install-preview.1`, con immagini pubbliche
[core](https://hub.docker.com/r/tylconsulting/thothii-core) e
[frontend](https://hub.docker.com/r/tylconsulting/thothii-frontend).
1. Eseguire `docker login` sul computer di pubblicazione con un account autorizzato
a creare repository pubblici e pubblicare immagini nel namespace scelto.
Usare un credential store Docker: il token non va passato sulla riga di comando,
scritto nel repository o copiato nel pacchetto.
2. Predisporre nel credential helper Git l'accesso al repository Gitea con diritto
di creare rilasci e allegati. Il comando riusa quelle credenziali senza stamparle.
3. Installare le dipendenze del produttore con `cd backend && npm ci`, poi eseguire:
```bash
node scripts/publish-installation.mjs \
--revision COMMIT_GIA_PUBBLICATO \
--version 0.1.0-install-preview.1 \
--namespace tylconsulting \
--platforms linux/amd64 \
--output /percorso/privato/rilascio-0.1.0-install-preview.1
```
La directory di output deve essere nuova o vuota e avere un genitore esistente.
Diventa privata e contiene stato di ripresa, log del produttore, archivi e checksum.
Non è una directory di installazione. Il produttore usa un worktree temporaneo al
commit richiesto, così modifiche locali, segreti e workspace non entrano nelle build.
Il comando prepara `tylconsulting/thothii-core` e `tylconsulting/thothii-frontend`
come repository pubblici, costruisce e pubblica le immagini versionate, risolve i
digest di tutte le immagini e crea gli archivi del comando nativo con Compose e
risorse di inizializzazione. Catalog migration e workspace maintenance usano lo
stesso digest core. PostgreSQL, Qdrant e Ollama restano immagini upstream.
Prima di rendere pubblico il rilascio Gitea, vengono verificati pull anonimi delle
immagini, smoke test senza rete di core/frontend, checksum e download degli allegati.
Una build o un upload incompleto lascia il rilascio **in bozza**. Per riprovare,
rieseguire lo stesso comando con gli stessi parametri e la stessa directory.
Immagini già presenti devono appartenere allo stesso commit/versione; allegati
esistenti devono avere lo stesso checksum. Il produttore non sostituisce versioni
pubblicate con contenuti diversi. Conservare la directory fino al completamento.
Il risultato pubblico comprende `thothii-VERSION-linux-amd64.tar.gz` e
`SHA256SUMS.txt`. L'archivio contiene `bin/tht`, il validatore affiancato,
`release-manifest.json`, Compose, SQL/script di inizializzazione e guide. Non
contiene credenziali, dati dei workspace o database di esempio. La verifica su
questa macchina di pubblicazione non sostituisce il successivo collaudo Windows.
## English
This is a maintainer command. Consumers download precompiled images and the native
operator bundle. The first prerelease targets **Linux amd64** for Ubuntu WSL2 and
later Omarchy; full installation and real-host acceptance remain separate work.
The maintainer needs Git, Node 24/npm, the Go toolchain from `tools/tht/go.mod`,
Docker Buildx with Linux amd64 execution support, and `tar`. The npm lock supplies
Bun for producer builds only. Push the selected source commit to the public Gitea
repository before publication. Use Docker's credential store for `docker login`
and Git's credential helper for Gitea release/attachment permissions. Never pass
tokens as command arguments or include them in a checkout or archive.
Run the producer on Linux, WSL2 or macOS, not a native Windows shell.
The first [Linux amd64 prerelease](https://git.tylconsulting.it/mptyl/ThothII/releases/tag/installation-v0.1.0-install-preview.1)
is available as `0.1.0-install-preview.1`, with public Docker Hub core/frontend images.
From `backend`, run `npm ci`, then the command above with an explicit revision,
version, namespace, platforms and a new private output directory. A temporary Git
worktree isolates the selected commit. The producer publishes core/frontend to
Docker Hub and retains PostgreSQL, Qdrant and Ollama upstream. All runtime and
maintenance services use resolved immutable platform digests.
The Gitea release stays a draft until images, anonymous pulls, network-isolated
smoke checks and uploaded bundle checksums pass. An interrupted run can be retried
with the same arguments and output directory. Existing images must match the
source/version, and existing attachments must match their checksum; published
versions are not overwritten. Producer logs and retry state stay local.
Download the matching `.tar.gz` and `SHA256SUMS.txt` from the public Gitea prerelease,
verify the archive checksum, then extract it. Keep the two executables together.
The bundle needs no application checkout, Node, Bun, Python or compiler on the
consumer host. It carries the manifest, Compose and initialization assets, but no
installation credentials, workspace data or example databases. Linux arm64 can be
selected explicitly for later release work; it does not imply macOS acceptance.
Developer regression checks:
```bash
cd backend
node --test scripts/release-*.test.mjs
```
+70
View File
@@ -0,0 +1,70 @@
# Display mode and language
ThothII can run with its own application header (**full**) or inside an integrated
portal (**embedded**). Display mode and authentication are separate choices.
| Installation | Display | Authentication |
| --- | --- | --- |
| Standalone local instance | `full` | Local ThothII account |
| Standalone server | `full` | Local accounts or configured OIDC provider |
| Integrated portal | `embedded` | Identity verified by the portal's trusted server proxy |
## Standalone setup
Follow the complete [Italian](standalone-manual-it.md) or
[English](standalone-manual-en.md) installation procedure. It explicitly selects
`--shell-mode full --shell-default-locale en` and separates configuration, credentials,
initial migrations and startup. Do not skip those steps by running setup alone.
The authored installation descriptor contains:
```yaml
shell:
mode: full
defaultLocale: en
```
Use `it` for an Italian initial interface. Existing browser language preferences can
override that initial value. Full mode remembers language and theme in the browser.
For an existing installation, preserve the current descriptor and edit only the intended
settings; do not rerun setup to overwrite it. With a current host `tht` binary:
```sh
tht --installation /absolute/path/thothii-installation.yaml installation generate
tht --installation /absolute/path/thothii-installation.yaml start
tht --installation /absolute/path/thothii-installation.yaml status
tht --installation /absolute/path/thothii-installation.yaml doctor --json
```
Generation updates derived configuration; it does not start services. `start` applies
the installation's normal lifecycle and is not guaranteed to touch only the frontend.
Follow the deployment's maintenance procedure and retain its network, authentication and
model settings. Do not edit generated files or remove persistent volumes.
## Authentication and embedded deployments
Full mode does not configure login by itself. See [local authentication](authentication-local.md)
or [OIDC](authentication-oidc.md), with [Authentik](authentik.md) as a provider option.
Embedded mode requires a compatible portal integration, not just a descriptor toggle.
The portal owns login/logout and supplies a server-verified identity. A presentation
adapter does not authenticate users. The core must not be reachable by a route that
bypasses the trusted proxy. Do not add a second OIDC login to an already authenticated
upstream deployment.
Portal implementation details belong to the
[developer integration reference in the repository](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/install/authentication-upstream.md),
not to the standalone installation procedure.
## Three different languages
- **Interface language** controls labels, forms and application messages.
- **Session interaction language** is captured when a session is created. Resuming it
retains that language even if the interface language changes later.
- **Workspace language** concerns domain content and retrieval; switching the interface
does not translate Evidence, SQL, identifiers or database values.
In embedded mode the interface follows the portal's language and theme. A portal language
change may reload the page. Saved session artifacts remain available, but resuming work
is explicit; a reload does not by itself request a new model generation.
+362 -245
View File
@@ -1,324 +1,441 @@
# Manual standalone installation # Guided standalone installation
[Versione italiana](standalone-manual-it.md) [Versione italiana](standalone-manual-it.md)
This is the verification procedure for preparing THothII as a standalone application in `full` This is the fresh-machine installation procedure for THothII. THothII receives a natural-language
mode on macOS, Windows, and Linux. question, queries an enterprise database read-only, and guides the user through SQL review. The
core, PostgreSQL catalog, Qdrant, embedding service, and Pi run in Docker; Node.js, Python, and Pi
are not required on the host.
In this document, “standalone” means that the user does not need to install Node.js, Python or Pi ## Prepare and validate workspace documents before starting the stack
on the host: the application services and local semantic
services run through Docker. DWH and LLM providers remain external endpoints configured by the
installation; this is not an offline package.
This path uses no graphical installer, native launcher, or Docker Hub image. It starts from a Gitea The first two steps of the new flow work without Docker, Node, Python, Pi or an
clone and uses explicit terminal commands. Publishing pre-built images is a later step. installation descriptor. Use the platform bundle with **both** `tht` and
`tht-workspace-documents` in the same directory (`.exe` on Windows). Put that directory
on `PATH`, or invoke the absolute executable path. Older packages containing only
`tht` do not provide this capability. Maintainers can currently build the bundle;
publishing assets and Docker Hub images belongs to a later delivery step. The rest
of this guide still describes the existing installation path.
## Verification matrix 1. Choose a new directory outside the application checkout, with an existing parent:
| System | Recommended terminal | Runtime | Test architecture | ```sh
| --- | --- | --- | --- | tht workspace prepare --directory ./my-workspaces --id practice --name "Practice" --language en
| macOS supported by the installed Docker Desktop version | Bash in Terminal | Docker Desktop | Apple Silicon (`arm64`) | ```
| Windows 11 | Ubuntu inside WSL2 | Docker Desktop with WSL2 integration | x64 (`amd64`) |
| Ubuntu Linux 22.04 or 24.04 | Bash | Docker Engine + Compose v2 | x64 (`amd64`) |
Intel macOS is excluded from the first verification campaign. ARM Linux may be tested when the This creates `thoth-workspaces.yaml`, `practice/workspace.yaml` and
machine’s Docker runtime reports `arm64`, but it is not part of the minimum matrix. `workspace-docs/practice/README.md`. Existing destinations, even empty ones, are
refused. No services start, Git is not initialized and no remote is contacted.
Example databases remain a deferred subproject. For a curator-supplied repository,
use a separate local copy and go straight to step 3; read access to the origin is
sufficient.
2. Edit the catalog (schema v1) and workspace descriptor (schema v4) at your own pace.
Keep `id`, `name` and optional `description` identical in both; the id must match
the directory name. Root directories must match catalog entries, except
`workspace-docs` and the local `.git` directory. Database connections, schema and
credentials belong to the installation Metadata Catalog. Evidence is optional
and initially absent.
3. Validate, correct the reported document/field, and repeat:
Documentation and Mac prerequisite checks have passed. Complete fresh installations on all three ```sh
systems remain pending; this matrix describes the tests to perform, not completed certification. tht workspace validate --directory ./my-workspaces
tht workspace validate --directory ./my-workspaces --json
```
## Before you start PowerShell uses the same arguments, for example
`C:\ThothII\bin\tht.exe workspace validate --directory C:\ThothII\my-workspaces`.
Validation changes no files. It rejects multiple/malformed YAML documents,
duplicate keys/ids, unknown fields, catalog/directory/descriptor mismatches,
missing local references and symbolic links. Fix the first error in each document
and repeat to reveal any subsequent errors.
You need: Evidence `absent` is valid. Filesystem checks cover directories, accessibility and
literal references; standard Markdown selections also check declared size limits.
Evidence v2 requires `curated/` and syntactically valid YAML frontmatter. Local limits
are 1 MiB per document read and 100,000 entries per Evidence tree. Arbitrary patterns,
the complete curated-unit contract, provenance, HTTP/S3 access and indexing remain
explicit runtime checks. See the [Evidence guide](../evidence.md).
- access to the THothII Gitea repository and the workspace Git repository; JSON includes `schema_version`, `scope: local-documents`, `ok`, `workspaces`, `issues`
- Git; and `deferred_checks`. Issues identify document, field, code, correction and YAML
- Docker Desktop on macOS and Windows, or Docker Engine with the Compose v2 plugin on Linux; line where available, without printing document values. Exit statuses: `0` local
- Bash, `curl`, OpenSSL and `shasum` (Ubuntu package: `libdigest-sha-perl`); success, `1` documents/access/bundle need correction, `2` invalid arguments. Local
- enough disk space to build the images and download the embedding model; success does not certify semantic truth, connectivity or readiness. The Git revision
- the DWH and LLM endpoints, plus the credentials required by the installation. activated later must contain the checked documents; this command does not publish
uncommitted files or empty directories.
On Linux, the current user must be able to run Docker. If the system requires `sudo`, add the user ## Prepare and validate application documents
to the Docker group according to local policy and open a new session before continuing.
On Windows, run every command in this guide from Ubuntu under WSL2. In Docker Desktop, enable WSL2 After validating workspaces, create a local directory **outside their repository**:
integration for that distribution. Clone the project inside the WSL2 Linux filesystem, for example
under `~/src`, rather than under `/mnt/c`: this avoids slow builds and path/line-ending issues. Pi
does not need to be installed on the host.
Check the runtime before or immediately after cloning:
```sh ```sh
docker version tht installation prepare --directory ./my-installation
docker compose version
docker version --format '{{.Server.Arch}}'
``` ```
The last command must return `amd64`, `x86_64`, `arm64`, or `aarch64`. This creates private, commented `thothii-installation.yaml`, `operator.env`,
`database-bootstrap.yaml` and `README.md`. The destination must be new and its parent
must exist. It starts no services and does not implicitly generate passwords.
## 1. Clone a project revision 1. Choose models and providers in the descriptor. The template proposes
`openai/gpt-4.1-mini` for interaction and `ollama/qwen3-embedding:0.6b` with 1024
dimensions for embedding. Edit these before setup. `modelCatalog.defaults.interaction`
must support sessions and metadata generation when the latter is configured.
The template omits optional metadata generation. See
[model configuration](../general/pi-configuration.md) for custom providers.
2. Replace the Git remote in both the descriptor and `operator.env`; keep branch and
transport consistent. Paths are absolute and machine-local. `operator.env` accepts
one literal `KEY=value` assignment per line, without duplicate keys or shell
interpolation. Credentials belong in referenced protected files.
3. Complete `database-bootstrap.yaml` with exactly one entry per workspace. A complete
direct-connection example is:
Use the project repository on Gitea: ```yaml
schemaVersion: 1
databases:
- workspaceId: practice
engine: postgres
databaseName: sales
schema: public
binding:
transport: postgres_direct
host: db.intranet
port: 5432
username: thoth_reader
secretFiles:
password: /private/path/my-installation/secrets/database-password
```
```sh Use a read-only DWH account. `rest_api` requires `baseUrl`, `restPath`, `restAuth`
(`none`, `bearer`, `x-api-key`) and `secretFiles.apiKey` when authenticated.
`ssh_tunnel` requires `username`, `sshHost`, `sshPort`, `sshUsername`,
`sshTargetHost`, `sshTargetPort` and files `password`, `sshPrivateKey`,
`sshKnownHosts`; it supports Catalog diagnostics, not NL-to-SQL sessions.
`tlsCa` and `sshPrivateKeyPassphrase` are optional. Signed HTTP Evidence needs
`evidenceSecretFiles` with key `evidence.signed_urls`; static S3 credentials need
`evidence.access_key`, `evidence.secret_key` and optional `evidence.session_token`.
All values are private file paths. Workspace descriptors remain schema v4;
this bootstrap input is not a second runtime Catalog.
4. Explicitly generate technical credentials in the standard layout:
```sh
tht installation credentials --directory ./my-installation
```
This creates separate random Catalog runtime/migrator and administrator passwords,
`auth/auth.yaml`, `auth/users.yaml`, a `secrets/secrets.env` template and
`secrets/pi-auth.json`. Existing files are retained; invalid ones stop the command.
The initial administrator is `admin`; its password stays in private
`secrets/admin-password` and is never printed. The default is local authentication
at `http://localhost:8080`: review and edit `auth/auth.yaml` before validation.
This increment does not validate offline OIDC bootstrap for the existing path.
5. Fill the provider key in `secrets/secrets.env` and create the DWH password file.
For Git HTTPS supply the referenced credentials and CA files; empty credentials
are allowed for a public remote, and the CA file must be available. For SSH supply
a key and known_hosts and select the matching descriptor override. `pi_auth`
providers require prepared Pi credentials. Keep every secret outside workspace
Git with installer-only access (0600 on Unix, equivalent Windows ACLs).
6. Validate and repeat after each correction:
```sh
tht --installation /absolute/path/my-installation/thothii-installation.yaml installation validate --workspaces /absolute/path/my-workspaces --json
```
The default bootstrap is beside the descriptor; `--bootstrap PATH` selects another.
Validation changes no documents, generates no projections, uses no network and
writes no database. It rejects placeholders, inconsistencies, missing/non-private
files and secrets inside workspace Git. Reports identify document, field and
correction without secret values. Exit statuses: 0 local success, 1 corrections
needed, 2 invalid arguments.
Standard release Compose assets may still be absent at this stage; custom overrides
must already exist. Release assets, external connectivity, Catalog import and runtime
readiness remain explicit deferred checks. Success prepares the next preflight;
it neither skips those checks nor establishes a completed installation.
## Check prerequisites and produce the plan
At step 3, before completing all application parameters, check the machine and
the private installation directory already prepared:
```bash
tht installation preflight --directory /path/installation --json
```
This requires a reachable Linux Docker daemon, Compose 2.24 or newer, at least
2 CPUs, 4 GiB allocated to Docker and 10 GiB free on the installation filesystem.
A release may require more resources. On Windows run the Linux executable in
Ubuntu WSL2 with Docker Desktop integration; Pi is bundled in the core image.
At step 5, after `installation validate`, select the published release manifest
with its downloaded bundle resources and produce a new plan:
```bash
tht --installation /path/installation/thothii-installation.yaml installation plan \
--workspaces /path/workspaces \
--release /path/release/release-manifest.json \
--output /path/installation/installation-plan.json --json
```
This repeats document checks, verifies image digests and Compose, Git, available
external databases and Evidence, then saves the plan and its separate private
`.key` file. It does not execute setup. Missing images and unavailable existing
dependencies block the plan. Actual Docker Hub publication remains the next ticket;
an invented manifest cannot bypass publication.
Correct `error` outcomes and read `warning` outcomes. `deferred-to-runtime` entries
are mandatory checks after startup, not readiness already achieved. After changing
documents or rotating credentials, produce a new plan; existing files are never
overwritten. Keep both plan files outside workspace Git. External probes perform
bounded database authentication/schema reads, Git/HTTP/S3 reads and explicit model
endpoint reachability checks. They invoke no LLM generation; HTTP/S3 requests may
incur ordinary service request charges. See the
[preflight reference](installation-preflight.md) for limits, the manifest
format and runtime obligations.
## Before you start: the two repositories
There are two separate repositories:
1. the application repository cloned by the user:
https://git.tylconsulting.it/mptyl/ThothII.git;
2. the workspace repository supplied by the curator/installer. It is not the THothII repository
and must not be cloned inside the application directory.
The workspace repository normally contains:
~~~
thoth-workspaces.yaml
<workspace-id>/workspace.yaml
<workspace-id>/evidence/** # when Evidence is declared
~~~
workspace.yaml contains workspace identity, language and optional Evidence source. By design it does
not contain database passwords. Database identity, transport (PostgreSQL, REST, or tunnel), user,
password, token and certificates are installation-local settings stored encrypted by the Catalog.
This prevents credentials from being committed to the workspace repository.
## 0. Machine prerequisites
### Windows
- Windows 10/11 with Docker Desktop running and the WSL2 backend enabled.
- Ubuntu in WSL2, with Docker Desktop integration enabled for that distribution.
- Git, Bash, curl, OpenSSL and shasum inside WSL2.
- Do not install Node.js, Python or Pi on the host for this procedure.
If WSL2 is not installed, use the company procedure or, in PowerShell:
~~~
wsl --install -d Ubuntu
~~~
Run all commands inside Ubuntu WSL2, in a Linux directory such as $HOME/src, not under /mnt/c.
scripts/install-tht.ps1 exists for advanced native PowerShell scenarios; use WSL2 for the
reproducible test.
### macOS
- Docker Desktop installed and running, with several GB free for images and the embedding model.
- Git, Bash, curl, OpenSSL and shasum.
- Intel and Apple Silicon Macs are supported when Docker Desktop supports the architecture
reported by the Docker server.
- Do not install Node.js, Python or Pi on the host for this procedure.
### Linux, including Omarchy
- Git, Bash, curl, OpenSSL and shasum.
- Docker Engine and the Docker Compose v2 plugin. On Omarchy, check first:
~~~
command -v docker
docker compose version
docker info
~~~
If Docker is missing, install Docker and Compose using the distribution-approved package procedure,
then start the service. On an Arch-like distribution the typical route is:
~~~
sudo pacman -S docker docker-compose
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"
~~~
After adding the group, open a new session and repeat docker info. Node.js, Python and Pi are not
needed on the host: they are in the Docker images.
On every system run:
~~~
bash scripts/check-standalone-prerequisites.sh
docker version --format '{{.Server.Arch}}'
~~~
The architecture must be amd64, x86_64, arm64, or aarch64. You also need access to the
application Gitea repository, the workspace repository URL/branch and credentials, container
reachability to DWH/LLM endpoints, and the credentials, tokens or certificates associated with
the databases.
## 1. What to clone
Clone only the application:
~~~
mkdir -p "$HOME/src" mkdir -p "$HOME/src"
cd "$HOME/src" cd "$HOME/src"
git clone https://git.tylconsulting.it/mptyl/ThothII.git git clone https://git.tylconsulting.it/mptyl/ThothII.git
cd ThothII cd ThothII
git rev-parse --short HEAD git rev-parse --short HEAD
``` ~~~
For an SSH clone, when the key is already authorized on Gitea: Record the revision. tht setup --complete downloads the workspace repository into a persistent
Docker volume using the URL, branch and transport supplied during setup.
```sh ## 2. Install the terminal command
git clone git@git.tylconsulting.it:mptyl/ThothII.git
```
Record the hash printed by `git rev-parse` for a repeatable test. In a later campaign, use the
maintainer-approved revision/tag rather than implicitly following a mutable `main` branch.
## 2. Check prerequisites and install the operator command
From the clone root: From the clone root:
```sh ~~~
bash scripts/check-standalone-prerequisites.sh bash scripts/check-standalone-prerequisites.sh
export PATH="$HOME/.local/bin:$PATH" mkdir -p "$HOME/.local/bin"
THT_INSTALL_DIRECTORY="$HOME/.local/bin" bash scripts/install-tht.sh THT_INSTALL_DIRECTORY="$HOME/.local/bin" bash scripts/install-tht.sh
export PATH="$HOME/.local/bin:$PATH"
tht version tht version
``` ~~~
`install-tht.sh` bootstraps only the native `tht` operator command; it does not install a desktop tht is the only native component to install. It builds the binary with Docker and orchestrates
version of THothII. It uses the repository’s Docker builder, installs the binary for the current Compose; it is not a second application runtime.
terminal environment, and installs it in the user directory. Persist `$HOME/.local/bin` in your
shell PATH for new terminals too. An existing `tht` in this directory will be updated.
On Windows, run these commands inside WSL2. The installed `tht` binary is the Linux binary inside ## 3. Prepare a few secrets and run complete setup
WSL2; the application runtime remains Docker Desktop. Do not use `scripts/install-tht.ps1` as the
primary path for this test.
## 3. Configure and start the local installation The first execution creates protected placeholders under deploy/local/secrets/ and stops if a
required credential is missing. Fill in the requested files and rerun the same command; compatible
configuration files are reused.
Run the remaining blocks in one Bash session from the physical clone root (`pwd -P`). ~~~
First create two distinct catalog passwords, preserving any existing files: tht setup --complete --profile local --shell-mode full --shell-default-locale en
~~~
```bash The setup asks only for information the computer cannot know:
umask 077
mkdir -p deploy/local/secrets
for name in catalog-runtime-password catalog-migrator-password; do
target="deploy/local/secrets/$name"
if [ ! -e "$target" ]; then
(set -C; openssl rand -hex 32 > "$target") || exit 1
fi
done
export THT_CATALOG_RUNTIME_PASSWORD_SOURCE="$(pwd -P)/deploy/local/secrets/catalog-runtime-password"
export THT_CATALOG_MIGRATOR_PASSWORD_SOURCE="$(pwd -P)/deploy/local/secrets/catalog-migrator-password"
```
Do not regenerate passwords for an initialized catalog. Configure without starting services: | Request | What to provide |
```sh
tht setup --profile local --shell-mode full --shell-default-locale en --configure-only
```
Answer the prompts as follows:
| Prompt | Value or rule |
| --- | --- | | --- | --- |
| Installation ID | `local`, unless one clone hosts multiple installations | | Workspace repository | Data/configuration repository URL, not ThothII.git |
| Deployment profile | `local` | | Branch | normally main |
| DWH API endpoint | An `http(s)` URL without user, password, query, or fragment; may be empty for a smoke-only test | | Access | ssh with key and known_hosts, or https with credential file and CA |
| LLM API endpoint | An `http(s)` URL without credentials; may be empty for a smoke-only test | | DWH/LLM URL | endpoint without a token in the URL |
| Workspace repository URL | The workspace repository URL, not the THothII source clone | | Local login | initial user and password requested by the prompt |
| Workspace branch | Normally `main` |
| Workspace access | `ssh` with a deploy key, or `https` with a protected credential file |
| File paths | Accept the default paths under `deploy/local/secrets/` for the first test |
| Secret templates | Answer `yes` when protected files do not exist yet |
| Authentication | Configure the local login required by the installation; never put passwords on a command line |
The generated configuration is local and ignored by Git: The setup generates random Catalog passwords and writes their paths, never their values, to
operator.env. It runs docker compose config, builds images, starts the Catalog, runs
catalog-migrate, starts the stack, and pulls the workspace repository. The pull also activates
declared Evidence; at minimum source files present in the workspace are materialized locally.
```text ### The file the user fills in
deploy/local/thothii-installation.yaml
deploy/local/operator.env
deploy/local/auth/
deploy/local/secrets/
```
Edit secrets only in protected local files; never commit them. `deploy/env/local.env.example` is a tracked reference; the The main file is:
generated path `deploy/local/operator.env` is the active path for this installation.
### Complete protected files ~~~
deploy/local/secrets/thothii.secrets
~~~
If setup created blank templates, enter the values with a local editor: Add only NAME=VALUE lines needed by modelCatalog and installation adapters, such as an LLM API key
(DEEPSEEK_API_KEY, OPENAI_API_KEY, or the key declared by the catalog) and, when applicable,
THT_DWH_API_KEY. Allowed names are documented in deploy/secrets/README.md. Never put tokens in
URLs, the repository, or copied shell commands.
```sh Two distinctions prevent common errors:
chmod 600 deploy/local/secrets/*
"${EDITOR:-vi}" deploy/local/secrets/thothii.secrets
```
The bundle must contain only `KEY=VALUE` lines for credentials actually used by `modelCatalog`. The - when the catalog uses pi_auth, the LLM token belongs in the Pi pi-auth.json file created by
allowed names and credential boundary are documented in the local file setup; {} is only a placeholder and does not enable a model;
`deploy/secrets/README.md`. Do not put tokens in URLs, the YAML - workspace database credentials (PostgreSQL password, REST API token, tunnel SSH key,
descriptor, the Git repository, or commands copied into the shell. known_hosts, CA) do not belong in the workspace repository. Enter them per workspace in
Database Management, which stores them encrypted in the Catalog. The workspace declares
database/schema and transport; the installer must obtain the actual values from the database owner.
For SSH workspace access, also provide the private key and `known_hosts` file requested by setup. A private workspace repository also needs the Git files required by its transport: an SSH key and
For HTTPS access, provide the Git credential file and any required CA. Both must remain protected known_hosts, or an HTTPS credential file and CA. These are transport files, not a second bundle to
and outside version control. commit. To minimize manual files, use SSH with an already-authorized deploy key.
Before starting, complete these additional configuration steps: ## 4. Automatic checks and terminal tests
1. Add `THT_CATALOG_RUNTIME_PASSWORD_SOURCE` and `THT_CATALOG_MIGRATOR_PASSWORD_SOURCE` to Setup verifies files, permissions, descriptor, Compose, Docker, authentication, services, Pi and
`deploy/local/operator.env`, with the same absolute paths exported above. Setup does not persist the workspace. After startup, run these commands at any time:
these two variables. Store paths, not passwords.
2. Replace the descriptor's generic `modelCatalog` with the approved provider/model configuration.
The generated defaults do not replicate the existing Mac. See [Pi/model configuration](../general/pi-configuration.md)
and the local example `deploy/psd/thothii-installation.yaml.example`.
3. Populate the keys referenced by `authentication.apiKeyEnv` in `thothii.secrets`. Providers using
`pi_auth` need valid credentials at `PI_AUTH_FILE`; the `{}` template is not authentication.
4. Complete the workspace Git files. SSH requires an authorized deploy key and verified known-hosts;
HTTPS requires the credential file and CA bundle expected by the overlay. Blank templates cannot
provide repository access.
After editing generated configuration, do not rerun setup: it rejects different existing content. ~~~
Generate the projections and run the explicit migration below. Use `THT_GIT_ACCESS=https` if that INSTALLATION="$PWD/deploy/local/thothii-installation.yaml"
was selected during setup. This block targets the fresh `local` descriptor with only the Git overlay; tht --installation "$INSTALLATION" doctor --json
custom installations must include their extra descriptor overlays in the same order. tht --installation "$INSTALLATION" workspace pull --json
tht --installation "$INSTALLATION" workspace test --json
~~~
```bash workspace test checks, for every active workspace, database binding and credentials, Evidence,
INSTALLATION="$(pwd -P)/deploy/local/thothii-installation.yaml" Qdrant, and the embedding service. It exits non-zero when the database binding is missing or a
tht --installation "$INSTALLATION" installation generate connection is unusable. Before running it, the installer must configure the database in Database
THT_PROJECT="thothii-$(printf '%s' "$INSTALLATION" | shasum -a 256 | cut -c 1-12)" Management: the workspace repository cannot contain the password by itself.
THT_GIT_ACCESS=ssh
compose=(
docker compose --project-name "$THT_PROJECT" --project-directory "$(pwd -P)"
--env-file "$(pwd -P)/deploy/local/operator.env"
-f compose.yaml -f deploy/compose.local.yaml
-f "deploy/compose.git-$THT_GIT_ACCESS.yaml"
-f deploy/local/generated/compose.models.yaml
)
"${compose[@]}" config --quiet
"${compose[@]}" build core frontend
"${compose[@]}" up -d catalog-db
"${compose[@]}" run --rm catalog-migrate
tht --installation "$INSTALLATION" start
```
Stop if a command fails. The project name matches the hash used by `tht`, preserving volume doctor --json is the repeatable, non-destructive core verification. The final functional test must
identity. `catalog-migrate` applies Catalog and Memory migrations; `tht start` does not run it also open http://127.0.0.1:8080, sign in, and complete a real question through final SQL.
automatically. Initial embedding-model download may take time. Use this installation-specific
sequence, not `run-stack.sh` with a different environment/project name.
## 4. Verify the installation ## Activities only the installer can complete
The descriptor generated for the default ID is: The procedure automates bootstrap, but it cannot invent enterprise decisions or authorizations.
The installer must complete and record:
```sh 1. usable LLMs, the modelCatalog, and linked API keys; then run tht pi test and tht doctor;
INSTALLATION="$(pwd -P)/deploy/local/thothii-installation.yaml" 2. database configuration, connection test, schema synchronization, and description generation;
test -f "$INSTALLATION" 3. human consolidation of generated descriptions;
bash scripts/verify-standalone-install.sh "$INSTALLATION" 4. Qdrant semantic entries through workspace preprocess run;
``` 5. naming-based FK suggestions as a complement to schema FKs, human review, and loading approved
relationships into Qdrant;
6. recurring tht doctor --json and tht workspace test --json checks;
7. one real question completed successfully without connection or model errors.
The verifier is read-only: it runs `tht doctor --json` and `tht status` without restarting the Configuration is complete only when all applicable activities are done, decisions are recorded, and
stack, regenerating configuration, or printing secret contents. the two terminal tests are green. The core is usable only after the real question, not merely
because the frontend answers /health.
### Gate A — platform smoke test on all three computers ## Gate A and Gate B
Record the following for each machine: ### Gate A — platform
```sh ~~~
uname -a uname -a
docker version --format '{{.Server.Version}} {{.Server.Arch}}' docker version --format '{{.Server.Version}} {{.Server.Arch}}'
tht version tht version
bash scripts/check-standalone-prerequisites.sh bash scripts/check-standalone-prerequisites.sh
bash scripts/verify-standalone-install.sh "$INSTALLATION" bash scripts/verify-standalone-install.sh "$INSTALLATION"
``` ~~~
The gate passes when the clone is intact, Docker and Compose are reachable, `tht doctor` is OK, the ### Gate B — usability
stack is running, and the frontend responds at the default local URL `http://127.0.0.1:8080`.
Doctor also checks workspace and Pi: record their failures separately rather than labeling every
failure as a platform problem. Check HTTP readiness with:
```sh ~~~
curl --fail --silent --show-error http://127.0.0.1:8080/health
```
### Gate B — functional verification
Run this on at least one machine with available endpoints and credentials:
First follow [Workspace operations](../operations/workspaces.md) to import/prepare the workspace
and configure the Database and local binding. The source clone does not transfer catalog data,
secrets or sessions from the Mac. Connect any VPN required by DWH/model endpoints and verify
that their names are reachable from containers too.
1. open `http://127.0.0.1:8080`;
2. sign in with the configured local account;
3. verify that the configured workspace is readable;
4. start a real question and complete the review gates through final SQL;
5. stop and restart the installation, then run `verify-standalone-install.sh` again.
A Gate B failure involving DWH, the LLM provider, workspace Git, or credentials does not by itself
prove a Docker portability problem: record the failed endpoint or component separately.
## Daily lifecycle
Use the explicit descriptor when more than one installation may be discoverable:
```sh
INSTALLATION="$PWD/deploy/local/thothii-installation.yaml"
tht --installation "$INSTALLATION" status
tht --installation "$INSTALLATION" start
tht --installation "$INSTALLATION" start --build
tht --installation "$INSTALLATION" logs
tht --installation "$INSTALLATION" doctor --json tht --installation "$INSTALLATION" doctor --json
tht --installation "$INSTALLATION" stop tht --installation "$INSTALLATION" workspace test --json
``` curl --fail --silent --show-error http://127.0.0.1:8080/health
~~~
Use `start --build` after source changes or to rebuild images from the current checkout. `stop` Then run a real question and stop/restart with tht stop and tht start. Do not use docker compose
preserves volumes, sessions, the catalog, Pi state, Qdrant data, and the embedding model. Do not down --volumes: it deletes the Catalog, sessions, Qdrant data and the embedding model.
use `docker compose down --volumes` during a normal test: it is destructive and removes local data.
For upgrades requiring migrations, follow the release runbook before starting the new application.
## Quick diagnosis ## Quick diagnosis
| Symptom | Check | | Symptom | Action |
| --- | --- | | --- | --- |
| `Docker Engine is not reachable` | start Docker Desktop or the Docker service and rerun `docker info` | | Docker Engine is not reachable | start Docker Desktop or systemctl and repeat docker info |
| Windows sees Docker but Bash fails | run the guide inside Ubuntu WSL2 and enable that distribution in Docker Desktop | | Omarchy cannot find docker | install Docker/Compose, enable the service and open a new session |
| `tht: command not found` | open a new shell and check `command -v tht`; rerun the bootstrap if needed | | Windows sees Docker but Bash fails | use Ubuntu WSL2 and enable its Docker Desktop integration |
| line-ending or executable-script errors | use a clone in the WSL2/Linux filesystem and rerun `bash scripts/...` | | workspace pull fails | check URL, branch, key/credential file and known_hosts from the container |
| unsupported architecture | check `docker version --format '{{.Server.Arch}}'`; the test requires `amd64` or `arm64` | | workspace test reports a missing binding | configure database, token/password and CA in Database Management |
| missing descriptor or env file | use `deploy/local/...` generated by `tht setup`, not an arbitrary copied file | | Pi is not ready | fill pi-auth.json or the key declared by modelCatalog, then run tht pi test |
| healthy stack but workflow failure | check external URLs, the credential bundle, workspace Git, and authentication separately |
| data appears missing | check that `down --volumes` was not used; `stop` does not remove volumes |
## Acceptance checklist
- [ ] The clone comes from the expected Gitea repository and the revision is recorded.
- [ ] Docker Desktop/Engine and Compose v2 are available.
- [ ] The runtime reports an allowed architecture.
- [ ] `tht` was built from the repository and responds to `tht version`.
- [ ] Setup uses `profile: local`, `shell.mode: full`, and `shell.defaultLocale: en`.
- [ ] The descriptor, `operator.env`, authentication, and secrets exist only under `deploy/local/`.
- [ ] No secret appears in Git, URLs, public YAML, or recorded commands.
- [ ] Gate A passes on Apple Silicon macOS, x64 Windows WSL2, and x64 Linux.
- [ ] Gate B runs on at least one machine with DWH and LLM available.
- [ ] Stop/start and final verification complete without deleting volumes.
## Out of scope for this release
The following remain future work:
- publishing pre-built images on Docker Hub;
- reducing prompts through a dedicated non-interactive configuration;
- creating DMG, MSI/EXE, AppImage, or other native installers;
- providing an offline runtime or bundling a local DWH/LLM into the application.
## Related documents ## Related documents
- [Install and first start](first-start.md) - [Install and first start](first-start.md)
- [Shell and localization](../operations/shell-and-localization.md)
- [Workspace operations](../operations/workspaces.md) - [Workspace operations](../operations/workspaces.md)
- `deploy/secrets/README.md` (runtime secrets) - [Database Management](../operations/database-management.md)
- [Model configuration](../general/pi-configuration.md)
- deploy/secrets/README.md
Publishing images on Docker Hub and native DMG/MSI/AppImage installers remain later work: this
procedure starts from the Gitea clone and does not require pre-published Docker Hub images.
+377 -252
View File
@@ -1,329 +1,454 @@
# Installazione manuale standalone # Installazione standalone guidata
[English version](standalone-manual-en.md) [English version](standalone-manual-en.md)
Questa è la procedura di prova per predisporre THothII come applicazione autonoma in modalità Questa è la procedura per installare THothII da zero. THothII riceve una domanda in linguaggio
`full` su macOS, Windows e Linux. naturale, interroga in sola lettura un database aziendale e accompagna l’utente nella revisione
della SQL risultante. Il core, il catalogo PostgreSQL, Qdrant, il servizio di embedding e Pi vengono
eseguiti in Docker; sul computer non servono Node.js, Python o Pi.
In questo documento “autonoma” significa che l’utente non deve installare Node.js, Python o Pi ## Preparazione e verifica dei workspace prima dello stack
sull'host: i servizi applicativi e i servizi semantici
locali vengono eseguiti con Docker. DWH e provider LLM restano endpoint esterni configurati
dall’installazione; questa procedura non è un pacchetto offline.
Il percorso non usa installer grafici, launcher nativi o immagini Docker Hub. Si parte da un clone I primi due passi del nuovo percorso funzionano senza Docker, Node, Python, Pi o
Gitea e si usano comandi espliciti da terminale. La pubblicazione di immagini pre-costruite è una un file di installazione. Usare il bundle della propria piattaforma con **entrambi**
fase successiva. gli eseguibili `tht` e `tht-workspace-documents` nella stessa cartella (`.exe` su
Windows). Aggiungere la cartella al `PATH`, oppure usare il percorso completo.
Il vecchio pacchetto con il solo `tht` non contiene questa capacità. Il bundle è
attualmente producibile dal manutentore; pubblicazione degli asset e immagini
Docker Hub appartengono a una fase successiva. Il resto della guida descrive ancora
il percorso di installazione esistente.
## Matrice di verifica 1. Scegliere una cartella nuova, esterna all'applicazione, con padre già esistente:
| Sistema | Terminale raccomandato | Runtime | Architettura della prova | ```sh
| --- | --- | --- | --- | tht workspace prepare --directory ./miei-workspace --id pratica --name "Pratica" --language it
| macOS supportato dalla versione Docker Desktop installata | Bash nel Terminale | Docker Desktop | Apple Silicon (`arm64`) | ```
| Windows 11 | Ubuntu dentro WSL2 | Docker Desktop con integrazione WSL2 | x64 (`amd64`) |
| Linux Ubuntu 22.04 o 24.04 | Bash | Docker Engine + Compose v2 | x64 (`amd64`) |
Intel macOS non fa parte della prima campagna di verifica. ARM Linux può essere provato quando il Sono creati `thoth-workspaces.yaml`, `pratica/workspace.yaml` e
runtime Docker della macchina restituisce `arm64`, ma non è un requisito della matrice minima. `workspace-docs/pratica/README.md`. Una destinazione esistente, anche vuota, viene
rifiutata. Nessun servizio viene avviato; Git non viene inizializzato, nessun remoto
viene contattato. I database di esempio restano un sottoprogetto differito. Per
un repository fornito dal curatore, usare una copia locale separata e passare al
punto 3: è sufficiente accesso in lettura all'origine.
2. Modificare con calma catalogo (schema v1) e descrittore workspace (schema v4).
Mantenere uguali `id`, `name` e l'eventuale `description` nei due file; l'id deve
coincidere con la cartella. Ogni cartella alla radice deve corrispondere a un
workspace elencato, salvo `workspace-docs` e la directory locale `.git`.
Connessioni, schema e credenziali dei database appartengono al Metadata Catalog
dell'installazione. Le Evidence sono facoltative e inizialmente assenti.
3. Verificare, correggere il documento/campo indicato e ripetere:
Le verifiche documentali e dei prerequisiti Mac sono passate. Le installazioni complete da zero ```sh
sui tre sistemi restano da eseguire: la matrice indica le prove previste, non una certificazione. tht workspace validate --directory ./miei-workspace
tht workspace validate --directory ./miei-workspace --json
```
## Cosa serve prima di iniziare Su PowerShell usare gli stessi argomenti, ad esempio
`C:\ThothII\bin\tht.exe workspace validate --directory C:\ThothII\miei-workspace`.
Il controllo non modifica file. Rifiuta YAML multipli o malformati, chiavi/id
duplicati, campi sconosciuti, incoerenze tra catalogo, cartelle e descrittori,
riferimenti locali mancanti e link simbolici. Correggere il primo errore del
documento e ripetere per vedere eventuali errori successivi.
Servono: Evidence `absent` è valido. Per filesystem si verificano directory, accessibilità e
riferimenti letterali; per le selezioni Markdown standard anche i limiti dichiarati.
Evidence v2 richiede `curated/` e frontmatter con sintassi YAML valida. I limiti locali
sono 1 MiB per documento letto e 100.000 elementi per albero Evidence. Pattern
arbitrari, contratto completo delle unità curate, provenienza, accesso HTTP/S3 e
indicizzazione restano controlli runtime espliciti. Vedere la [guida Evidence](../evidence.md).
- accesso al repository Gitea di THothII e al repository Git dei workspace; Il JSON espone `schema_version`, `scope: local-documents`, `ok`, `workspaces`, `issues`
- Git; e `deferred_checks`. I problemi riportano documento, campo, codice, correzione e,
- Docker Desktop su macOS e Windows, oppure Docker Engine con il plugin Compose v2 su Linux; quando disponibile, riga YAML, senza stampare i valori del documento. Exit status:
- Bash, `curl`, OpenSSL e `shasum` (su Ubuntu, pacchetto `libdigest-sha-perl`); `0` successo locale, `1` documenti/accesso/bundle da correggere, `2` argomenti errati.
- spazio disco sufficiente per compilare le immagini e scaricare il modello di embedding; Il successo locale non certifica verità semantica, connettività o readiness. La
- gli endpoint DWH e LLM, più le credenziali che l’installazione deve usare. revisione Git attivata in seguito deve contenere i documenti verificati; il comando
non pubblica file non committati o directory vuote.
Su Linux l’utente corrente deve poter eseguire Docker. Se il sistema richiede `sudo`, aggiungere ## Predisporre e verificare i documenti applicativi
l’utente al gruppo Docker secondo la policy locale e aprire una nuova sessione prima di continuare.
Su Windows usare Ubuntu in WSL2 per tutti i comandi di questa guida. In Docker Desktop attivare Dopo la verifica dei workspace, creare una cartella locale **esterna al loro repository**:
l’integrazione WSL2 per quella distribuzione. Clonare il progetto nel filesystem Linux di WSL2,
per esempio sotto `~/src`, e non sotto `/mnt/c`: si evitano rallentamenti e problemi di permessi o
line ending. Non è necessario installare Pi sull’host.
Verificare il runtime prima del clone o subito dopo:
```sh ```sh
docker version tht installation prepare --directory ./mia-installazione
docker compose version
docker version --format '{{.Server.Arch}}'
``` ```
L’ultima istruzione deve restituire `amd64`, `x86_64`, `arm64` o `aarch64`. Il comando crea file privati commentati: `thothii-installation.yaml`, `operator.env`,
`database-bootstrap.yaml` e `README.md`. La cartella deve essere nuova e il padre
deve esistere. Non avvia servizi e non genera implicitamente password.
## 1. Clonare una revisione del progetto 1. Nel descrittore, scegliere modelli e provider. Il template propone
`openai/gpt-4.1-mini` per l'interazione e `ollama/qwen3-embedding:0.6b` con 1024
dimensioni per l'embedding. Sono valori modificabili, non una selezione richiesta
durante il setup. `modelCatalog.defaults.interaction` deve essere utilizzabile
nelle sessioni e anche nella generazione metadati, se quest'ultima è configurata.
Il template omette la generazione metadati, che è facoltativa. Consultare la
[configurazione dei modelli](../general/pi-configuration.md) per provider personalizzati.
2. Sostituire il remoto Git sia nel descrittore sia in `operator.env`; mantenere
coerenti branch e trasporto. I percorsi sono assoluti e riferiti a questa macchina.
`operator.env` accetta una sola assegnazione letterale `KEY=value` per riga, senza
duplicati o interpolazioni shell. Le credenziali restano nei file referenziati.
3. Compilare `database-bootstrap.yaml`: una voce per ciascun workspace, senza
duplicati. Questo esempio mostra il contratto completo di un collegamento diretto:
Usare il repository di progetto su Gitea: ```yaml
schemaVersion: 1
databases:
- workspaceId: pratica
engine: postgres
databaseName: vendite
schema: public
binding:
transport: postgres_direct
host: db.intranet
port: 5432
username: thoth_reader
secretFiles:
password: /percorso/privato/mia-installazione/secrets/database-password
```
```sh Usare le credenziali di un utente DWH in sola lettura. Per `rest_api`, il binding
richiede `baseUrl`, `restPath` e `restAuth` (`none`, `bearer`, `x-api-key`); quando
serve autenticazione, aggiungere `secretFiles.apiKey`. `ssh_tunnel` richiede
`username`, `sshHost`, `sshPort`, `sshUsername`, `sshTargetHost`, `sshTargetPort` e
i file `password`, `sshPrivateKey`, `sshKnownHosts`; abilita diagnostica Catalog,
non sessioni NL→SQL. Sono facoltativi `tlsCa` e `sshPrivateKeyPassphrase`.
Le Evidence HTTP firmate richiedono `evidenceSecretFiles` con chiave
`evidence.signed_urls`; S3 con credenziali statiche richiede `evidence.access_key`
e `evidence.secret_key`, con `evidence.session_token` facoltativo. Tutti i valori
sono percorsi di file privati. I workspace rimangono nello schema v4: il bootstrap
è un input iniziale, non un secondo Catalog runtime.
4. Generare esplicitamente le credenziali tecniche nel layout standard:
```sh
tht installation credentials --directory ./mia-installazione
```
Sono creati password casuali separate per Catalog runtime/migrator e amministratore,
il relativo `auth/auth.yaml` con `auth/users.yaml`, un template `secrets/secrets.env`
e `secrets/pi-auth.json`. I file esistenti vengono conservati; se non validi, il
comando si ferma. L'amministratore iniziale è `admin`, la password è nel file
privato `secrets/admin-password` e non viene stampata. Il default è autenticazione
locale con URL pubblico `http://localhost:8080`: verificare e, se necessario,
modificare `auth/auth.yaml` prima del controllo. Questo incremento non valida il
bootstrap OIDC del percorso esistente.
5. Inserire la chiave del provider in `secrets/secrets.env` e creare il file della
password DWH. Per HTTPS Git fornire i file referenziati per credenziali e CA:
credenziali vuote sono ammesse per un remoto pubblico, la CA deve essere disponibile.
Per SSH fornire chiave e known_hosts, scegliendo il relativo override nel descrittore.
I provider `pi_auth` richiedono credenziali Pi già preparate. Conservare tutti
questi file fuori dal repository workspace e proteggere l'accesso al solo utente
installatore (0600 su Unix, ACL equivalenti su Windows). Non committarli.
6. Verificare e ripetere dopo ogni correzione:
```sh
tht --installation /percorso/assoluto/mia-installazione/thothii-installation.yaml installation validate --workspaces /percorso/assoluto/miei-workspace --json
```
Il bootstrap viene cercato accanto al descrittore; `--bootstrap PERCORSO` ne
seleziona uno diverso. Il controllo non cambia documenti, non genera proiezioni,
non usa la rete e non scrive database. Rifiuta placeholder, incoerenze, file
mancanti/non privati e segreti situati nel repository workspace. Il report indica
documento, campo e correzione senza valori riservati. Exit status: 0 successo
locale, 1 correzioni necessarie, 2 argomenti errati.
Gli asset Compose standard del rilascio possono ancora mancare in questa fase;
gli override personalizzati devono già esistere. Il report distingue i controlli
differiti: asset del rilascio, connettività esterna, import Catalog e readiness.
Un esito positivo prepara il successivo preflight: non autorizza a saltare tali
controlli e non equivale a un'installazione completata.
## Verificare le precondizioni e produrre il piano
Al passo 3, prima di completare tutti i parametri applicativi, controllare la
macchina e la directory privata già preparata:
```bash
tht installation preflight --directory /percorso/installazione --json
```
Servono Docker Linux raggiungibile, Compose 2.24 o successivo, almeno 2 CPU,
4 GiB assegnati a Docker e 10 GiB liberi nella directory di installazione. Il
rilascio può richiedere risorse maggiori. Su Windows eseguire il binario Linux
in Ubuntu WSL2 con integrazione Docker Desktop; Pi è incluso nell'immagine core.
Al passo 5, dopo `installation validate`, selezionare il manifest del rilascio
pubblicato, con le risorse del bundle già scaricate, e produrre un piano nuovo:
```bash
tht --installation /percorso/installazione/thothii-installation.yaml installation plan \
--workspaces /percorso/workspaces \
--release /percorso/rilascio/release-manifest.json \
--output /percorso/installazione/installation-plan.json --json
```
Il comando ripete i controlli dei documenti, verifica immagini/digest e Compose,
Git, database ed Evidence esterne disponibili, poi salva il piano e il suo file
privato `.key`. Non esegue il setup. Le immagini assenti e le dipendenze esterne
irraggiungibili bloccano il piano. Al momento la pubblicazione reale Docker Hub
è ancora il ticket successivo: un manifest inventato non permette di aggirarla.
Correggere gli esiti `error`; leggere gli `warning`. Gli esiti
`deferred-to-runtime` identificano controlli obbligatori dopo l'avvio, non una
readiness già ottenuta. Un piano valido non sostituisce questi gate. Dopo una
correzione o rotazione di credenziali produrre un nuovo piano; i file esistenti
non vengono sovrascritti. Conservare piano e chiave fuori dal Git dei workspace.
Le prove esterne sono letture limitate: autenticazione/schema del database,
letture Git e HTTP/S3, raggiungibilità degli endpoint modello espliciti. Nessuna
generazione LLM viene invocata; le richieste HTTP/S3 possono avere i normali costi
del servizio. Limiti, manifest e obblighi sono nel
[riferimento di preflight](installation-preflight.md).
## Prima di iniziare: i due repository
Servono due repository distinti:
1. il repository dell’applicazione, che l’utente clona:
https://git.tylconsulting.it/mptyl/ThothII.git;
2. il repository dei workspace, indicato dal curatore/installatore. Non è il repository di
THothII e non va clonato manualmente nella directory dell’applicazione.
Il repository workspace contiene il catalogo e una directory per ogni workspace, normalmente:
~~~
thoth-workspaces.yaml
<workspace-id>/workspace.yaml
<workspace-id>/evidence/** # se il workspace dichiara Evidence
~~~
Il file workspace.yaml descrive identità, lingua e, opzionalmente, la sorgente Evidence. Per scelta
architetturale non contiene password del database. L’identità del database, il trasporto
(PostgreSQL, REST o tunnel), username, password, token e certificati sono configurazione locale
dell’installazione, conservata cifrata dal Catalog. Questo evita di committare credenziali nel
repository workspace.
## 0. Prerequisiti della macchina
### Windows
- Windows 10/11 con Docker Desktop avviato e backend WSL2 abilitato.
- Ubuntu in WSL2, con integrazione Docker Desktop abilitata per quella distribuzione.
- Git, Bash, curl, OpenSSL e shasum nella distribuzione WSL2.
- Non installare Node.js, Python o Pi sull’host per questa procedura.
In PowerShell, se WSL2 non esiste ancora, usare la procedura aziendale oppure:
~~~
wsl --install -d Ubuntu
~~~
Eseguire poi tutti i comandi dentro Ubuntu WSL2, in una directory Linux come $HOME/src, non sotto
/mnt/c. Il percorso nativo scripts/install-tht.ps1 esiste per scenari PowerShell avanzati; per la
prova riproducibile usare WSL2.
### macOS
- Docker Desktop installato, avviato e con alcuni GB liberi per immagini e modello di embedding.
- Git, Bash, curl, OpenSSL e shasum.
- Sono supportati Mac Intel e Apple Silicon se Docker Desktop supporta l’architettura restituita
dal Docker server.
- Non installare Node.js, Python o Pi sull’host per questa procedura.
### Linux, incluso Omarchy
- Git, Bash, curl, OpenSSL e shasum.
- Docker Engine e il plugin Docker Compose v2. Su Omarchy verificare prima:
~~~
command -v docker
docker compose version
docker info
~~~
Se Docker manca, installare Docker e Compose con il gestore pacchetti/procedura approvata dalla
distribuzione, poi avviare il servizio. Su una distribuzione Arch-like il percorso tipico è:
~~~
sudo pacman -S docker docker-compose
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"
~~~
Dopo l’aggiunta al gruppo aprire una nuova sessione e ripetere docker info. Non installare Node.js,
Python o Pi sull’host: sono dentro le immagini Docker.
Su tutti i sistemi il controllo finale è:
~~~
bash scripts/check-standalone-prerequisites.sh
docker version --format '{{.Server.Arch}}'
~~~
L’architettura deve essere amd64, x86_64, arm64 o aarch64. Servono inoltre accesso al repository
Gitea dell’applicazione, URL/branch e credenziali del repository workspace, raggiungibilità dal
container degli endpoint DWH/LLM e le credenziali, token o certificati associati ai database.
## 1. Cosa clonare
Clonare solo l’applicazione:
~~~
mkdir -p "$HOME/src" mkdir -p "$HOME/src"
cd "$HOME/src" cd "$HOME/src"
git clone https://git.tylconsulting.it/mptyl/ThothII.git git clone https://git.tylconsulting.it/mptyl/ThothII.git
cd ThothII cd ThothII
git rev-parse --short HEAD git rev-parse --short HEAD
``` ~~~
Per un clone SSH usare, se la chiave è già autorizzata su Gitea: Annotare la revisione. Il repository workspace verrà scaricato da tht setup --complete dentro un
volume Docker persistente, usando URL, branch e trasporto indicati durante il setup.
```sh ## 2. Installare il comando terminale
git clone git@git.tylconsulting.it:mptyl/ThothII.git
```
Per una prova ripetibile annotare l’hash stampato da `git rev-parse`. In una campagna successiva
usare la revisione/tag approvata dal maintainer invece di seguire implicitamente una `main` che
può cambiare.
## 2. Verificare i prerequisiti e installare il comando operatore
Dal root del clone: Dal root del clone:
```sh ~~~
bash scripts/check-standalone-prerequisites.sh bash scripts/check-standalone-prerequisites.sh
export PATH="$HOME/.local/bin:$PATH" mkdir -p "$HOME/.local/bin"
THT_INSTALL_DIRECTORY="$HOME/.local/bin" bash scripts/install-tht.sh THT_INSTALL_DIRECTORY="$HOME/.local/bin" bash scripts/install-tht.sh
export PATH="$HOME/.local/bin:$PATH"
tht version tht version
``` ~~~
`install-tht.sh` è un bootstrap del solo comando operatore nativo `tht`; non installa una versione Il comando tht è l’unico componente nativo da installare. Costruisce il binario con Docker e
desktop di THothII. Usa il builder Docker del repository, installa il binario adatto all’ambiente orchestra Compose; non è un secondo runtime dell’applicazione.
del terminale e lo installa nella directory utente. Aggiungere `$HOME/.local/bin` al PATH della
shell anche per i terminali successivi. Un `tht` già presente in quella directory viene aggiornato.
Su Windows, eseguire questi comandi dentro WSL2. Il binario `tht` installato è quello Linux di WSL2; ## 3. Preparare pochi segreti e avviare il setup completo
il runtime dell’applicazione rimane Docker Desktop. Non usare `scripts/install-tht.ps1` come percorso
principale di questa prova.
## 3. Configurare e avviare l’installazione locale La prima esecuzione crea i placeholder protetti sotto deploy/local/secrets/ e si ferma se manca
una credenziale necessaria. Compilare i file indicati e rilanciare lo stesso comando: i file di
configurazione già compatibili vengono riutilizzati.
Eseguire i blocchi successivi nella stessa sessione Bash dal root fisico del clone (`pwd -P`). ~~~
Creare prima le due password distinte del catalogo, conservando eventuali file già esistenti: tht setup --complete --profile local --shell-mode full --shell-default-locale en
~~~
```bash Durante il setup servono solo le informazioni operative che il computer non può conoscere:
umask 077
mkdir -p deploy/local/secrets
for name in catalog-runtime-password catalog-migrator-password; do
target="deploy/local/secrets/$name"
if [ ! -e "$target" ]; then
(set -C; openssl rand -hex 32 > "$target") || exit 1
fi
done
export THT_CATALOG_RUNTIME_PASSWORD_SOURCE="$(pwd -P)/deploy/local/secrets/catalog-runtime-password"
export THT_CATALOG_MIGRATOR_PASSWORD_SOURCE="$(pwd -P)/deploy/local/secrets/catalog-migrator-password"
```
Non rigenerare le password di un catalogo già inizializzato. Configurare senza avviare i servizi: | Richiesta | Cosa inserire |
```sh
tht setup --profile local --shell-mode full --shell-default-locale en --configure-only
```
Rispondere ai prompt nel seguente modo:
| Prompt | Valore o regola |
| --- | --- | | --- | --- |
| Installation ID | `local`, salvo necessità di più installazioni nello stesso clone | | Repository workspace | URL del repository dati/configurazione, non ThothII.git |
| Deployment profile | `local` | | Branch | normalmente main |
| DWH API endpoint | URL `http(s)` senza user, password, query o fragment; può restare vuoto per il solo smoke test | | Accesso | ssh con chiave e known_hosts, oppure https con credential file e CA |
| LLM API endpoint | URL `http(s)` senza credenziali; può restare vuoto per il solo smoke test | | DWH/LLM URL | endpoint senza token nella URL |
| Workspace repository URL | URL del repository dei workspace, non il clone sorgente di THothII | | Login locale | utente e password iniziale richiesti dal prompt |
| Workspace branch | normalmente `main` |
| Workspace access | `ssh` se si usa una chiave deploy; altrimenti `https` con credential file protetto |
| Percorsi dei file | accettare i percorsi predefiniti sotto `deploy/local/secrets/` nella prima prova |
| Secret templates | rispondere `yes` quando i file protetti non esistono ancora |
| Autenticazione | configurare il login locale richiesto dall’installazione; non inserire password in una riga di comando |
La configurazione generata è locale e ignorata da Git: Il setup crea automaticamente le password casuali del Catalog e le scrive in operator.env come
percorsi, non come valori. Esegue docker compose config, costruisce le immagini, avvia il Catalog,
esegue catalog-migrate, avvia lo stack e importa il repository workspace. L’import attiva anche
l’Evidence dichiarata: almeno i file source presenti nel workspace vengono materializzati nel
registro locale.
```text ### Il file da compilare
deploy/local/thothii-installation.yaml
deploy/local/operator.env
deploy/local/auth/
deploy/local/secrets/
```
Modificare i segreti solo nei file locali protetti; non committarli. Il file `deploy/env/local.env.example` è un riferimento Il file principale è:
tracciato; il percorso generato da `tht setup`, `deploy/local/operator.env`, è quello da usare per
questa installazione.
### Completare i file protetti ~~~
deploy/local/secrets/thothii.secrets
~~~
Se il setup ha creato template vuoti, inserire i valori con un editor locale: Inserire solo righe NOME=VALORE necessarie al modelCatalog e agli adapter, per esempio una API key
LLM (DEEPSEEK_API_KEY, OPENAI_API_KEY o quella dichiarata dal catalogo) ed eventualmente
THT_DWH_API_KEY. I nomi ammessi sono documentati in deploy/secrets/README.md. Non mettere token
nelle URL, nel repository o nei comandi.
```sh Due precisazioni evitano gli errori più comuni:
chmod 600 deploy/local/secrets/*
"${EDITOR:-vi}" deploy/local/secrets/thothii.secrets
```
Il bundle deve contenere solo righe `KEY=VALUE` per le credenziali effettivamente usate dal - se il catalogo usa pi_auth, il token LLM va nel file Pi pi-auth.json creato dal setup; {} è solo
`modelCatalog`. I nomi ammessi e il confine delle credenziali sono descritti nel file locale un placeholder e non abilita alcun modello;
`deploy/secrets/README.md`. Non mettere token nelle URL, nel - le credenziali specifiche di un database workspace (password PostgreSQL, API token REST, chiave
descriptor YAML, nel repository Git o nei comandi copiati nella shell. SSH del tunnel, known_hosts, CA) non vanno nel repository workspace: si inseriscono per workspace
in Database Management, che le conserva nel Catalog cifrato. Il workspace indica database/schema
e trasporto; l’installatore deve ottenere dal proprietario il valore corretto.
Per accesso workspace SSH, predisporre anche la chiave privata e il file `known_hosts` indicati dal Per un repository workspace privato sono inoltre indispensabili i file Git richiesti dal trasporto:
setup. Per accesso HTTPS, predisporre il credential file Git e l’eventuale CA. Entrambi devono una chiave SSH e known_hosts, oppure credential file HTTPS e CA. Sono file di trasporto, non un
restare protetti e fuori dal controllo versione. secondo bundle da committare. Per ridurre i file da compilare, usare SSH con una deploy key già
autorizzata.
Prima dell'avvio completare anche questi passaggi: ## 4. Controlli automatici e test da terminale
1. Aggiungere `THT_CATALOG_RUNTIME_PASSWORD_SOURCE` e `THT_CATALOG_MIGRATOR_PASSWORD_SOURCE` a Il setup verifica file, permessi, descriptor, Compose, Docker, autenticazione, servizi, Pi e
`deploy/local/operator.env`, con gli stessi percorsi assoluti esportati sopra. Il setup non salva workspace. Dopo l’avvio usare questi comandi in qualunque momento:
queste due variabili. Inserire i percorsi, non le password.
2. Sostituire il `modelCatalog` generico nel descriptor con la configurazione provider/modelli
approvata. I default generati non replicano il Mac esistente. Vedere [configurazione Pi/modelli](../general/pi-configuration.md)
e l'esempio locale `deploy/psd/thothii-installation.yaml.example`.
3. Inserire in `thothii.secrets` le chiavi referenziate da `authentication.apiKeyEnv`. I provider
`pi_auth` richiedono credenziali valide nel file `PI_AUTH_FILE`; il template `{}` non autentica.
4. Completare i file Git del workspace. SSH richiede una chiave deploy autorizzata e known-hosts
verificato; HTTPS richiede il credential file e il bundle CA previsti dall'overlay. I template
vuoti non consentono l'accesso al repository.
Dopo aver modificato la configurazione generata, non rilanciare setup: rifiuta file esistenti con ~~~
contenuti diversi. Generare le proiezioni ed eseguire la migrazione esplicita indicata sotto. INSTALLATION="$PWD/deploy/local/thothii-installation.yaml"
Impostare `THT_GIT_ACCESS=https` se scelto nel setup. Il blocco usa il nuovo descriptor `local` tht --installation "$INSTALLATION" doctor --json
con il solo overlay Git; per installazioni personalizzate includere gli overlay aggiuntivi tht --installation "$INSTALLATION" workspace pull --json
nello stesso ordine del descriptor. tht --installation "$INSTALLATION" workspace test --json
~~~
```bash workspace test prova, per ogni workspace attivo, il binding del database e le credenziali, Evidence,
INSTALLATION="$(pwd -P)/deploy/local/thothii-installation.yaml" Qdrant e il servizio embedding. Restituisce exit code diverso da zero se manca il binding del
tht --installation "$INSTALLATION" installation generate database o una connessione non è utilizzabile. Prima di eseguirlo l’installatore deve aver
THT_PROJECT="thothii-$(printf '%s' "$INSTALLATION" | shasum -a 256 | cut -c 1-12)" configurato il database in Database Management: il workspace repository da solo non può contenere
THT_GIT_ACCESS=ssh la password.
compose=(
docker compose --project-name "$THT_PROJECT" --project-directory "$(pwd -P)"
--env-file "$(pwd -P)/deploy/local/operator.env"
-f compose.yaml -f deploy/compose.local.yaml
-f "deploy/compose.git-$THT_GIT_ACCESS.yaml"
-f deploy/local/generated/compose.models.yaml
)
"${compose[@]}" config --quiet
"${compose[@]}" build core frontend
"${compose[@]}" up -d catalog-db
"${compose[@]}" run --rm catalog-migrate
tht --installation "$INSTALLATION" start
```
Fermarsi se un comando fallisce. Il nome progetto coincide con l'hash usato da `tht`, preservando Per una verifica generale del core, doctor --json è il test ripetibile e non distruttivo. Il test
l'identità dei volumi. `catalog-migrate` applica le migrazioni Catalog e Memory; `tht start` non funzionale finale deve inoltre aprire http://127.0.0.1:8080, autenticarsi e completare una domanda
lo esegue automaticamente. Il primo download del modello embedding può richiedere tempo. reale fino alla SQL finale.
Usare questa sequenza legata all'installazione, non `run-stack.sh` con env/progetto diversi.
## 4. Verificare l’installazione ## Attività che può svolgere solo l’installatore
Il descriptor generato per l’ID predefinito è: La procedura automatizza il bootstrap, non può inventare decisioni o autorizzazioni aziendali.
L’installatore deve completare e registrare:
```sh 1. quali LLM sono utilizzabili, il relativo modelCatalog e le API key collegate; poi eseguire
INSTALLATION="$(pwd -P)/deploy/local/thothii-installation.yaml" tht pi test e tht doctor;
test -f "$INSTALLATION" 2. per ogni database: configurazione, test connessione, sincronizzazione dello schema e
bash scripts/verify-standalone-install.sh "$INSTALLATION" generazione delle descrizioni;
``` 3. consolidamento umano delle descrizioni generate;
4. generazione delle entry semantiche in Qdrant tramite workspace preprocess run;
5. generazione delle FK suggerite dal naming, come complemento alle FK lette dallo schema, revisione
umana delle proposte e caricamento delle relazioni approvate in Qdrant;
6. verifica periodica con tht doctor --json e tht workspace test --json;
7. una domanda reale completata con successo, senza errori di connessione o modello.
Il verificatore è read-only: esegue `tht doctor --json` e `tht status`, senza ristartare lo stack, La configurazione è dichiarata completa solo quando tutti i punti applicabili sono stati eseguiti,
rigenerare la configurazione o stampare il contenuto dei segreti. le decisioni sono state registrate e i due test terminali sono verdi. Il core è dichiarato usabile
solo dopo la domanda reale, non perché il frontend risponde a /health.
### Gate A — smoke di piattaforma, su tutti e tre i computer ## Gate A e Gate B
Registrare per ogni macchina: ### Gate A — piattaforma
```sh ~~~
uname -a uname -a
docker version --format '{{.Server.Version}} {{.Server.Arch}}' docker version --format '{{.Server.Version}} {{.Server.Arch}}'
tht version tht version
bash scripts/check-standalone-prerequisites.sh bash scripts/check-standalone-prerequisites.sh
bash scripts/verify-standalone-install.sh "$INSTALLATION" bash scripts/verify-standalone-install.sh "$INSTALLATION"
``` ~~~
Il gate passa quando il clone è integro, Docker e Compose sono raggiungibili, `tht doctor` è OK, ### Gate B — usabilità
lo stack è avviato e il frontend risponde sulla porta locale predefinita `http://127.0.0.1:8080`.
Il doctor controlla anche workspace e Pi: registrare separatamente i loro errori senza attribuire
ogni fallimento alla piattaforma. Verificare la disponibilità HTTP con:
```sh ~~~
curl --fail --silent --show-error http://127.0.0.1:8080/health
```
### Gate B — verifica funzionale
Eseguire almeno su una macchina con endpoint e credenziali disponibili:
Seguire prima [Workspace operations](../operations/workspaces.md) per importare/preparare il
workspace e configurare Database e binding locale. Il clone sorgente non trasferisce catalogo,
segreti o sessioni del Mac. Connettere l'eventuale VPN richiesta da DWH/modelli e verificare
che i relativi nomi siano raggiungibili anche dai container.
1. aprire `http://127.0.0.1:8080`;
2. autenticarsi con l’account locale configurato;
3. verificare che il workspace configurato sia leggibile;
4. avviare una domanda reale e completare i gate di revisione fino alla SQL finale;
5. fermare e riavviare l’installazione, poi ripetere `verify-standalone-install.sh`.
Un fallimento del Gate B per DWH, provider LLM, Git workspace o credenziali non dimostra da solo un
problema di portabilità Docker: registrare separatamente l’endpoint o il componente fallito.
## Ciclo di vita quotidiano
Usare il descriptor esplicito quando più installazioni possono essere scoperte:
```sh
INSTALLATION="$PWD/deploy/local/thothii-installation.yaml"
tht --installation "$INSTALLATION" status
tht --installation "$INSTALLATION" start
tht --installation "$INSTALLATION" start --build
tht --installation "$INSTALLATION" logs
tht --installation "$INSTALLATION" doctor --json tht --installation "$INSTALLATION" doctor --json
tht --installation "$INSTALLATION" stop tht --installation "$INSTALLATION" workspace test --json
``` curl --fail --silent --show-error http://127.0.0.1:8080/health
~~~
`start --build` è necessario dopo una modifica al codice o per ricostruire le immagini dal clone Poi eseguire una domanda reale e fermare/riavviare con tht stop e tht start. Non usare docker
corrente. `stop` conserva volumi, sessioni, catalogo, stato Pi, dati Qdrant e modello embedding. compose down --volumes: cancella Catalog, sessioni, Qdrant e il modello embedding.
Non usare `docker compose down --volumes` durante una prova normale: è un’operazione distruttiva
che cancella i dati locali.
Per aggiornamenti che richiedono migrazioni, seguire il runbook della release prima dell'avvio.
## Diagnosi rapida ## Diagnosi rapida
| Sintomo | Controllo | | Sintomo | Azione |
| --- | --- | | --- | --- |
| `Docker Engine is not reachable` | avviare Docker Desktop oppure il servizio Docker e ripetere `docker info` | | Docker Engine is not reachable | avviare Docker Desktop o systemctl e ripetere docker info |
| Windows vede Docker ma Bash fallisce | eseguire la guida dentro Ubuntu WSL2 e abilitare l’integrazione della distribuzione in Docker Desktop | | Omarchy non trova docker | installare Docker/Compose, abilitare il servizio e riaprire la sessione |
| `tht: command not found` | aprire una nuova shell e verificare `command -v tht`; se necessario ripetere il bootstrap | | Windows vede Docker ma Bash fallisce | usare Ubuntu WSL2 e abilitarne l’integrazione in Docker Desktop |
| line ending o script non eseguibile | usare un clone nel filesystem WSL2/Linux e rieseguire `bash scripts/...` | | pull workspace fallisce | controllare URL, branch, chiave/credential file e known_hosts dal container |
| architettura non supportata | verificare `docker version --format '{{.Server.Arch}}'`; la prova richiede `amd64` o `arm64` | | workspace test segnala binding mancante | configurare database, token/password e CA in Database Management |
| descriptor o env file mancanti | usare il percorso `deploy/local/...` generato da `tht setup`, non un file copiato casualmente | | Pi non è pronto | compilare pi-auth.json o la chiave dichiarata dal modelCatalog, poi eseguire tht pi test |
| stack sano ma workflow fallisce | controllare separatamente URL, credential bundle, workspace Git e autenticazione |
| dati apparentemente persi | verificare che non sia stato usato `down --volumes`; `stop` non rimuove i volumi |
## Checklist di accettazione
- [ ] Il clone proviene dal repository Gitea atteso e la revisione è stata annotata.
- [ ] Docker Desktop/Engine e Compose v2 sono disponibili.
- [ ] Il runtime restituisce un’architettura ammessa.
- [ ] `tht` è stato costruito dal repository e risponde a `tht version`.
- [ ] Il setup usa `profile: local`, `shell.mode: full` e `shell.defaultLocale: en`.
- [ ] Descriptor, `operator.env`, autenticazione e segreti sono presenti solo in `deploy/local/`.
- [ ] Nessun segreto compare in Git, URL, YAML pubblico o comandi registrati.
- [ ] Gate A superato su macOS Apple Silicon, Windows WSL2/x64 e Linux x64.
- [ ] Gate B eseguito almeno su una macchina con DWH e LLM disponibili.
- [ ] Stop/start e verifica finale completati senza cancellare i volumi.
## Fuori perimetro di questa release
Restano attività successive:
- pubblicare immagini pre-costruite su Docker Hub;
- ridurre ulteriormente i prompt tramite una configurazione non interattiva dedicata;
- creare pacchetti DMG, MSI/EXE, AppImage o altri installer nativi;
- fornire un runtime offline o un DWH/LLM locale incluso nell’applicazione.
## Documenti collegati ## Documenti collegati
- [Install and first start](first-start.md) - [Installazione e primo avvio](first-start.md)
- [Shell and localization](../operations/shell-and-localization.md) - [Operazioni sui workspace](../operations/workspaces.md)
- [Workspace operations](../operations/workspaces.md) - [Database Management](../operations/database-management.md)
- `deploy/secrets/README.md` (runtime secrets) - [Configurazione dei modelli](../general/pi-configuration.md)
- deploy/secrets/README.md
La pubblicazione di immagini su Docker Hub e gli installer nativi DMG/MSI/AppImage restano attività
successive: questa procedura parte dal clone Gitea e non richiede immagini Docker Hub pre-pubblicate.
-137
View File
@@ -1,137 +0,0 @@
# Docker installation in the current operating contexts
Rendering and authentication are independent of these Docker contexts. Explicitly
select full/en for the Mac, full/OIDC for an autonomous server, or
embedded/upstream for Omics. The same frontend image supports both renderings;
generated `config.js` and the host page decide the container, while the backend
and trusted proxy decide identity. See [shell configuration and deploy](operations/shell-and-localization.md)
and [server portal authentication](install/authentication-upstream.md). Do not
apply the standalone server authentication projection to the Omics upstream path.
ThothII uses one Compose topology:
- `frontend`
- `core`
- `catalog-db`
- `qdrant`
- `embedding`
- `embedding-model-init`
- `catalog-migrate` (one-shot)
Qdrant and Ollama embedding are required internal Compose services. Only DWH and the LLM remain
external. The fixed model is `qwen3-embedding:0.6b` with 1024 dimensions and cosine distance;
`embedding-model-init` prepares it before `core` starts.
```mermaid
flowchart TB
INSTALL["Installation descriptor"] --> CONTEXT{Context}
CONTEXT --> LOCAL["Local\nCompose local"]
CONTEXT --> SERVER["Server\nCompose server"]
CONTEXT --> SESSION["Session server\noperator services"]
CONTEXT --> AUTH["Auth runtime\nprojection services"]
LOCAL --> BUNDLE["Common secret bundle"]
SERVER --> BUNDLE
SESSION --> BUNDLE
AUTH --> BUNDLE
BUNDLE --> SERVICES["Frontend, core, catalog DB, vector, embedding"]
```
## Short ownership contract
| Componente | Ownership | Contratto operativo |
| --- | --- | --- |
| DWH | External | External endpoint configured by the installation. |
| LLM | External | Endpoint or policy outside the internal semantic infrastructure. |
| Qdrant | Internal | Required internal Compose service with persistent `qdrant-data` volume. |
| Ollama embedding | Internal | Required internal Compose service for `qwen3-embedding:0.6b`. |
## Local path: setup, migration, start
The normal local path is [Install and first start](install/first-start.md). It uses `tht setup`
to create the selected installation-local descriptor and runs the catalog migration explicitly
before application startup.
If an operator intentionally prepares the descriptor and protected files by hand, the equivalent
foreground launch is:
```sh
cp deploy/env/local.env.example deploy/env/local.env
cp deploy/secrets/thothii.secrets.example deploy/secrets/thothii.secrets
chmod 600 deploy/secrets/thothii.secrets
# Copy docs/install/examples/thothii-installation.local.yaml to a protected operator path,
# replace every placeholder, chmod it 600, then set that exact path as
# THT_INSTALLATION_CONFIG_SOURCE in deploy/env/local.env.
./scripts/run-stack.sh
```
Set these values in `deploy/env/local.env`:
- `PI_AUTH_FILE`
- `THT_SECRETS_FILE`
- `THT_INSTALLATION_CONFIG_SOURCE` (the exact protected host `thothii-installation.yaml`)
- `THT_WORKSPACE_GIT_REMOTE`
- DWH endpoint
- LLM endpoint
Do not put secrets in `.env`. Runtime secrets belong in the
`deploy/secrets/thothii.secrets` bundle.
## Secret bundle
The documented and supported bundle keys are:
```dotenv
THT_MODEL_API_KEY=...
THT_DWH_API_KEY=...
OPENAI_API_KEY=...
```
`THT_MODEL_API_KEY` is available only as an explicitly declared catalog bundle key. A
metadata-generation provider references one audited bundle name from `deploy/secrets/README.md`
through `modelCatalog.providers.<provider>.authentication.apiKeyEnv`.
`apiKeyEnv` may be omitted only for an explicit endpoint that accepts unauthenticated requests;
hosted/default endpoints remain keyed.
Provider/model/endpoint settings stay in the protected installation descriptor; raw keys do not.
Compose mounts exactly `THT_INSTALLATION_CONFIG_SOURCE` into `core` as a read-only config and sets
the backend-only runtime path `THT_INSTALLATION_CONFIG_FILE` to
`/run/thothii-installation/thothii-installation.yaml`. Do not set the runtime path in the host env.
Descriptor and bundle changes are loaded only after application restart.
A private PEM CA remains outside the bundle and must be mounted through a reviewed Compose override.
## Preprocessing
Preprocessing runs through the native host CLI and the installation descriptor:
```sh
tht --installation /percorso/assoluto/thothii-installation.yaml \
workspace preprocess run --workspace <workspace-id>
```
Per rimuovere soltanto gli indici e gli artifact ricostruibili, preservando Memory e domande risolte:
```sh
tht --installation /percorso/assoluto/thothii-installation.yaml \
workspace preprocess clear --workspace <workspace-id>
```
The CLI runs the profile-gated `workspace-maintenance` service. See the
[preprocessing contract](contracts/workspace-preprocessing-cli.md) and the
[Evidence guide](evidence.md) for details.
## Server
For server installations, use the server profile with the session overlay:
```sh
docker compose --env-file deploy/env/server.env \
-f compose.yaml -f deploy/compose.server.yaml \
-f deploy/compose.session-server.yaml.example up --build -d
```
Also see [Workspace operations](operations/workspaces.md). Server deployment, reverse-proxy,
backup, and recovery remain manual-gated operations; do not treat the local profile as a server
replacement.
@@ -0,0 +1,435 @@
# Revisione e bonifica della documentazione
Data: 15 settembre 2026. Baseline: `6a4634dcf1b1df4ad29510a4da371245abd5c666`.
Ambito: 121 Markdown e 23 altri file già presenti in `docs/`, navigazione, collegamenti
locali, build e verificatori documentali. Il documento conserva la proposta iniziale
e registra sotto l'esecuzione successivamente autorizzata dall'utente. È interno,
non una pagina del manuale pubblico.
## Bonifica eseguita dopo approvazione
Ritirati 35 file dal working tree: 23 Markdown e 12 artefatti visuali generati
(otto PNG, due HTML e due JSON). Recuperabili integralmente dalla revisione
`5f3a7f5975b96fae1e1cdd0da08b4d60d41064cc`, precedente a questo intervento,
mediante `git show REVISIONE:percorso`. Non eliminata né riscritta la storia Git.
- Dettagli Compose utili consolidati in [riferimento developer](../operations/compose-reference.md);
README rinvia alle sole due procedure manuali IT/EN per installazioni nuove.
- M1–M3, E1–E3, X1 e i piani Catalog/model già implementati consolidati nel
[record di rilascio](../reports/knowledge-archives-release.md). Test storici non
presentati come collaudi attuali; estensioni e accettazioni aperte conservate.
- Prompt security assorbito nel PRD, senza approvare il progetto di hardening.
- Snapshot di progetto riscritto; overview corretta su Catalog, Memory ed Evidence.
- Verificatori auth/DWH e fixture riallineati ai documenti attuali: controlli su
permessi, diagnostica, segreti, TLS, separazione dei servizi e link. Le simulazioni
del vecchio journal scanner, non più presente nei documenti, non sono conservate
come finto collaudo del runtime. Le suite applicative non vengono modificate.
- Rimandi Markdown verificati; contratti, ADR (anche superati), progetti con
decisioni ancora aperte e runbook/report con rollback o accettazione pendente
restano nel repository, esclusi dal sito. Non spostati gate in nuove issue né
dichiarati chiusi: lo snapshot corrente li rende espliciti. ADR 0019 conservato
senza dedurre dal solo nome che tutte le sue estensioni siano implementate.
- Pubblicazione effettiva distinta dal branch `pages`; procedura ripetibile nel
[runbook del manuale](../operations/public-docs-publication.md).
File ritirati (percorsi dalla radice):
- `docs/architecture/thothii-core-sequence.visual-check.1440x900.dark.png`
- `docs/architecture/thothii-core-sequence.visual-check.1440x900.light.png`
- `docs/architecture/thothii-core-sequence.visual-check.2048x1320.dark.png`
- `docs/architecture/thothii-core-sequence.visual-check.2048x1320.light.png`
- `docs/architecture/thothii-core-sequence.visual-check.html`
- `docs/architecture/thothii-core-sequence.visual-check.json`
- `docs/architecture/thothii-runtime.visual-check.1440x900.dark.png`
- `docs/architecture/thothii-runtime.visual-check.1440x900.light.png`
- `docs/architecture/thothii-runtime.visual-check.2048x1320.dark.png`
- `docs/architecture/thothii-runtime.visual-check.2048x1320.light.png`
- `docs/architecture/thothii-runtime.visual-check.html`
- `docs/architecture/thothii-runtime.visual-check.json`
- `docs/installazione-docker-4-contesti.md`
- `docs/plans/2026-08-26-metadata-catalog-from-thothai.md`
- `docs/plans/2026-08-28-ai-catalog-description-generation-spec.md`
- `docs/plans/2026-08-28-ai-catalog-description-generation.md`
- `docs/plans/2026-09-02-installation-model-catalog.md`
- `docs/plans/2026-09-08-memory-m1-validation.md`
- `docs/plans/2026-09-08-security-hardening-resume-prompt.md`
- `docs/plans/2026-09-09-archive-repair-x1-validation.md`
- `docs/plans/2026-09-09-evidence-e1-validation.md`
- `docs/plans/2026-09-09-evidence-e2-validation.md`
- `docs/plans/2026-09-09-evidence-e3-validation.md`
- `docs/plans/2026-09-09-memory-m2-validation.md`
- `docs/plans/2026-09-09-memory-m3-validation.md`
- `docs/research/2026-08-23-legacy-thothai-metadata-capabilities.md`
- `docs/research/2026-08-23-postgresql-catalog-deployment-constraints.md`
- `docs/research/2026-08-23-thothii-metadata-publication-qdrant-seams.md`
- `docs/research/2026-09-07-antigravity-gemini-flash-value.md`
- `docs/research/2026-09-07-deepseek-harness-terminal.md`
- `docs/research/2026-09-07-omp-codex-zcode.md`
- `docs/research/2026-09-07-pi-config-vs-oh-my-pi.md`
- `docs/research/2026-09-07-zai-coding-plan-cli-quota.md`
- `docs/research/2026-09-07-zed-acp-omp-pi.md`
- `docs/research/text-to-sql-products.md`
Le sezioni successive e l'inventario fotografano la revisione iniziale: la presente
sezione prevale per le azioni eseguite. Gli altri ritiri sono rinviati finché non
si chiudono i relativi gate; non si cancellano documenti solo perché datati.
## Esito e modifica applicata
Il sito confondeva quattro funzioni: manuale del prodotto, riferimento del codice,
progettazione e diario delle consegne. Il problema non era soltanto la lunghezza della nav:
le pagine non elencate potevano comunque essere generate e indicizzate.
La configurazione ora pubblica soltanto 20 pagine e cinque asset esplicitamente ammessi.
Il resto è escluso da HTML, file copiati e indice di ricerca. Le nuove pagine pubbliche
sono `product-overview.md`, `install/shell-and-language.md` e `usage/memory.md`;
i documenti tecnici da cui sono state ricavate restano intatti nel repository.
Home e `install/first-start.md` sono state riscritte per dare un ingresso univoco.
Non è stato cancellato alcun documento della baseline.
`exclude_docs` nega per default la pubblicazione: aggiungere un file a `docs/` non basta
più a metterlo online. La build e la pipeline verificano corrispondenza tra nav ed
eccezioni, pagine generate, asset e ricerca. I riferimenti tecnici ancora necessari
al lettore rinviano esplicitamente ai sorgenti su Gitea, non a pagine interne del sito.
**Fuori dal manuale non significa privato.** Il repository ThothII è pubblico: questi
file e la cronologia restano leggibili su Gitea. Per riservatezza reale occorrerebbe una
decisione separata su repository/accessi e sulla storia già pubblicata. Questa bonifica
non modifica autorizzazioni, altri repository o l'installazione applicativa.
## Tre destinazioni, con regole diverse
| Destinazione | Contenuto | Regola |
| --- | --- | --- |
| Manuale pubblico | Prodotto, uso, installazione, configurazione, amministrazione | Descrivere ciò che il lettore può fare oggi; distinguere prerequisiti e limiti verificati. |
| Riferimento developer | Architettura, contratti, ADR, test ripetibili, integrazione | Conservare nel repository, fuori da MkDocs; un'autorità per ciascun contratto. |
| Lavoro temporaneo o storico | Piani attuati, prompt di ripresa, survey, report di singola consegna | Estrarre decisioni, difetti aperti e rollback ancora necessari; poi ritirare dal working tree con Git come archivio. |
Un file datato non è automaticamente obsoleto. Un contratto con `v3` nel nome non è
automaticamente sostituito da un descriptor v4: sono versioni di oggetti diversi.
Un ADR superato conserva il motivo della decisione e il collegamento al successore.
## Rilievi prioritari, con motivazione
### 1. Percorsi di installazione concorrenti — corretto per il pubblico
`install/first-start.md` proponeva ancora setup senza `--configure-only` e avvio con
`run-stack.sh`, mentre le guide manuali separano segreti, migrazione e avvio per lo
stesso descriptor/progetto. Ora è un punto d'ingresso alle due procedure complete,
non una terza ricetta. `installazione-docker-4-contesti.md` rimane interno: consolidarne
i dettagli ancora esclusivi in una procedura developer di distribuzione, poi ritirarlo.
Anche il percorso rapido nel README va riallineato prima di eliminare quei dettagli.
### 2. Ricerca architetturale presentata come stato corrente — ritirare dopo estrazione
`research/2026-08-23-thothii-metadata-publication-qdrant-seams.md` dichiara ancora il
passaggio catalog-to-core «deferred» e descrive `annotations.yaml` come input corrente.
ADR 0016 e gli attuali contratti registrano invece PostgreSQL come autorità per i
consumatori del core. Non usarlo per implementare nuovi comportamenti.
`research/2026-08-23-postgresql-catalog-deployment-constraints.md` dichiara esplicitamente
di essere parzialmente superato da ADR 0004. L'inventario legacy ThothAI del 23 agosto
è esplicitamente storico. Estrarre soltanto vincoli non già presenti in ADR/contratti;
poi eliminare i tre file dal working tree, conservandoli nella storia Git.
### 3. Decisioni correnti distribuite tra troppi piani — consolidare prima di eliminare
Per Memory/Evidence convivono piani di famiglia, M1, amministrazione comune, revisione
di semplificazione e sette resoconti M1–M3/E1–E3/X1. Conservare gli invarianti nei
contratti e gli scenari ripetibili in `testing/`; trasferire i gate non chiusi in issue.
Solo dopo si possono ritirare piani e validazioni di incremento.
`adr/0019-author-evidence-in-app-with-automatic-activation.md` ha un nome che suggerisce
editor in-app e attivazione automatica, ma il titolo descrive editor esterni e
consolidamento manuale. Contiene anche «Implementation is pending»: non è un affidabile
indicatore dello stato di tutte le parti della release. Conservare numero/decisione,
correggere metadati e rimandi e distinguere eventuali estensioni ancora pendenti.
### 4. Report operativi e piani con stato non aggiornato — non cancellare in blocco
Il report `2026-09-12-ui-visual-review-delivery.md` apre con «non integrata in main»;
le integrazioni successive sono documentate altrove. I piani full-shell dicono ancora
«deploy server non eseguito», mentre esistono report di rilascio server del 14 settembre.
Ciò prova che lo stato è distribuito, non che tutti i collaudi siano conclusi.
Consolidare report UI del 12–13 settembre in un record di rilascio per versione.
Non ritirare i report del 14 settembre né i runbook di upgrade finché rollback,
accettazione interattiva e finestra di osservazione non risultano chiusi dall'operatore.
`server-handoff-260906-preprocessing-complete.md` va confrontato con il runbook corrente
`server-codex-handoff.md`: estrarre eventuali passaggi di preprocessing esclusivi prima
di ritirare la consegna datata.
### 5. Ricerca su strumenti personali estranea al manuale — candidata all'eliminazione
I sei confronti CLI/editor/provider del 7 settembre (Antigravity, DeepSeek/CyberArk,
OMP/Codex/ZCode, Pi/Oh My Pi, quota Z.ai e Zed/ACP) non spiegano un contratto di ThothII.
Prezzi, quote e confronti non sono stati riverificati in questa revisione. Proposta:
toglierli dal repository del prodotto, mantenendoli in Git o trasferendoli, su scelta
del proprietario, a una raccolta personale. Stessa destinazione proposta per
`research/text-to-sql-products.md`, che è una ricognizione di mercato.
Non applicare automaticamente questa regola alla ricerca ERD, relationship e
classificatore sensibile: prima verificare se documenta decisioni ancora aperte.
### 6. Controlli e snapshot già disallineati — debito da correggere
Due verificatori falliscono su file già assenti nella baseline, non per l'esclusione
da MkDocs introdotta qui:
- `scripts/auth-docs-smoke.sh`: manca `docs/install/local.md`.
- `scripts/verify-dwh-auth-docs.sh`: manca `docs/operations/psd-dwh-auth-rollout.md`.
Le relative suite di fixture conservano la vecchia struttura. Non ricreare documenti
obsoleti per far passare i test: aggiornare i controlli ai contratti attuali, mantenendo
le verifiche su segreti, modalità di autenticazione e TLS. Questo riallineamento è
proposto, non eseguito in questa bonifica editoriale.
`PROJECT_STATE.md` cita nove percorsi `docs/*.md` non più esistenti, fra cui il programma
server PSD del 20 agosto, l'accettazione Evidence del 25 agosto e il rollout dwh-auth.
Inoltre accumula consegne anziché restare uno snapshot breve. Riscriverlo come stato
attuale, gate aperti e rimandi esistenti. I normali link Markdown relativi dei documenti
esistenti non presentano target mancanti nella verifica eseguita: i riferimenti rotti
qui citati sono anche percorsi in backtick o imposti dagli script.
## Proposta operativa in ordine
0. **Collegare il sito servito al risultato della pipeline:** il controllo remoto descritto
sotto ha rilevato una copia statica vecchia. Prima di dichiarare la bonifica online,
aggiornare il percorso di pubblicazione sul server con accesso amministrativo verificato.
1. **Separazione pubblica:** applicata; niente cancellazioni di sorgenti.
2. **Riallineare le autorità:** snapshot, verificatori, ADR 0019, README e stato delle
consegne; spostare gate aperti in issue senza marcarli completati.
3. **Ritiro a basso rischio, previa approvazione:** confronti personali, ricerca già
esplicitamente superata, prompt di ripresa dopo assorbimento nel PRD, output visuali
rigenerabili. Controllare prima i riferimenti in tutto il repository.
4. **Consolidamento:** piani completati e report incrementali; non perdere criteri di
accettazione, problemi aperti, provenienza delle immagini e rollback ancora validi.
5. **Riorganizzazione eventuale:** `docs/` per il pubblico, `developer-docs/` per
architettura/contratti/ADR/testing, `project-notes/` per lavoro in corso. È un secondo
intervento: aggiornare insieme tutti i link in README, AGENTS, CONTEXT, script e codice.
Per ogni ritiro usare un commit dedicato con motivazione e documento sostitutivo.
Non creare una cartella `archive/` piena di copie obsolete: la cronologia Git è già
l'archivio, salvo una necessità operativa o di conservazione esplicita.
## Inventario e destinazione proposta
L'inventario seguente distingue l'azione proposta dalla sola esclusione già applicata
al sito. «Consolidare» e «ritirare» non indicano cancellazioni eseguite.
<!-- inventory:start -->
| File (relativo a `docs/`) | Destinazione | Azione proposta |
| --- | --- | --- |
| `adr/0001-postgres-metadata-catalog.md` | Developer | Conservare decisione architetturale. |
| `adr/0002-workspace-database-secret-references.md` | Developer | Conservare decisione architetturale. |
| `adr/0003-installation-local-database-bindings.md` | Developer | Conservare decisione architetturale. |
| `adr/0004-fastify-kysely-metadata-catalog.md` | Developer | Conservare decisione architetturale. |
| `adr/0005-hard-delete-catalog-tables-during-synchronization.md` | Developer | Conservare ADR superato e catena dei successori; non eliminarlo. |
| `adr/0006-separate-physical-and-logical-relationships.md` | Developer | Conservare decisione architetturale. |
| `adr/0007-durable-authoritative-schema-synchronization.md` | Developer | Conservare decisione architetturale. |
| `adr/0008-allow-manual-catalog-metadata-cleanup.md` | Developer | Conservare decisione architetturale. |
| `adr/0009-use-one-sequential-description-generation-run.md` | Developer | Conservare decisione architetturale. |
| `adr/0010-allow-bounded-real-source-samples-for-description-generation.md` | Developer | Conservare decisione architetturale. |
| `adr/0011-gate-source-samples-with-a-sensitive-data-flag.md` | Developer | Conservare decisione architetturale. |
| `adr/0012-use-the-catalog-as-the-logical-relationship-authority.md` | Developer | Conservare decisione architetturale. |
| `adr/0013-use-one-installation-model-catalog-with-runtime-projections.md` | Developer | Conservare decisione architetturale. |
| `adr/0014-assess-sensitive-columns-locally-from-source-content.md` | Developer | Conservare ADR superato e catena dei successori; non eliminarlo. |
| `adr/0015-use-progressive-sampling-for-sensitive-columns.md` | Developer | Conservare decisione architetturale. |
| `adr/0016-use-postgres-metadata-for-all-core-consumers.md` | Developer | Conservare decisione architetturale. |
| `adr/0017-separate-reference-vectors-from-runtime-memory.md` | Developer | Conservare decisione architetturale. |
| `adr/0018-use-postgres-for-memory-and-qdrant-for-retrieval.md` | Developer | Conservare decisione architetturale. |
| `adr/0019-author-evidence-in-app-with-automatic-activation.md` | Developer | Conservare; allineare nome/stato alla decisione manuale effettiva. |
| `adr/0020-unify-administration-pages-and-use-namespaced-routes.md` | Developer | Conservare decisione architetturale. |
| `adr/0021-separate-shell-modes-and-replaceable-portal-adapter.md` | Developer | Conservare decisione architetturale. |
| `adr/0022-separate-ui-locale-from-session-interaction-language.md` | Developer | Conservare decisione architetturale. |
| `agents/domain.md` | Developer | Conservare regole di contribuzione/agenti, fuori dal sito. |
| `agents/issue-tracker.md` | Developer | Conservare regole di contribuzione/agenti, fuori dal sito. |
| `agents/triage-labels.md` | Developer | Conservare regole di contribuzione/agenti, fuori dal sito. |
| `architecture/application-shell.md` | Developer | Conservare riferimento corrente; distinguere il prodotto dalla struttura del codice. |
| `architecture/authentication.md` | Developer | Conservare riferimento corrente; distinguere il prodotto dalla struttura del codice. |
| `architecture/components.md` | Developer | Conservare riferimento corrente; distinguere il prodotto dalla struttura del codice. |
| `architecture/overview.md` | Developer | Conservare riferimento corrente; distinguere il prodotto dalla struttura del codice. |
| `architecture/thothii-core-sequence.html` | Developer/build | Conservare sorgente/diagramma tecnico; non pubblicare nel manuale. |
| `architecture/thothii-core-sequence.visual-check.1440x900.dark.png` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `architecture/thothii-core-sequence.visual-check.1440x900.light.png` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `architecture/thothii-core-sequence.visual-check.2048x1320.dark.png` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `architecture/thothii-core-sequence.visual-check.2048x1320.light.png` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `architecture/thothii-core-sequence.visual-check.html` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `architecture/thothii-core-sequence.visual-check.json` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `architecture/thothii-core.sequence.json` | Developer/build | Conservare sorgente/diagramma tecnico; non pubblicare nel manuale. |
| `architecture/thothii-runtime.architecture.json` | Developer/build | Conservare sorgente/diagramma tecnico; non pubblicare nel manuale. |
| `architecture/thothii-runtime.html` | Developer/build | Conservare sorgente/diagramma tecnico; non pubblicare nel manuale. |
| `architecture/thothii-runtime.visual-check.1440x900.dark.png` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `architecture/thothii-runtime.visual-check.1440x900.light.png` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `architecture/thothii-runtime.visual-check.2048x1320.dark.png` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `architecture/thothii-runtime.visual-check.2048x1320.light.png` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `architecture/thothii-runtime.visual-check.html` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `architecture/thothii-runtime.visual-check.json` | Transitorio | Ritirare output QA rigenerabili dopo conferma che non siano baseline uniche. |
| `contracts/archive-repair.md` | Developer | Conservare contratto e versioni semantiche; aggiornare insieme a codice/test. |
| `contracts/catalog-schema-snapshot.md` | Developer | Conservare contratto e versioni semantiche; aggiornare insieme a codice/test. |
| `contracts/curated-evidence-v4.md` | Developer | Conservare contratto e versioni semantiche; aggiornare insieme a codice/test. |
| `contracts/portal-shell-adapter-v1.md` | Developer | Conservare contratto e versioni semantiche; aggiornare insieme a codice/test. |
| `contracts/tht-dwh.md` | Developer | Conservare contratto e versioni semantiche; aggiornare insieme a codice/test. |
| `contracts/workspace-evidence-v3.md` | Developer | Conservare contratto e versioni semantiche; aggiornare insieme a codice/test. |
| `contracts/workspace-preprocessing-cli.md` | Developer | Conservare contratto e versioni semantiche; aggiornare insieme a codice/test. |
| `disambiguazione-iniziale.md` | Developer | Conservare invarianti di gate/ledger; il percorso utente è in skills.md. |
| `evidence.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `general/pi-configuration.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `gestione-memory.md` | Developer | Conservare contratto di Memory; istruzioni pubbliche in usage/memory.md. |
| `guida-utente.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `index.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `install/authentication-local.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `install/authentication-oidc.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `install/authentication-upstream.md` | Developer/integratore | Conservare integrazione e confine di fiducia; guida pubblica separata. |
| `install/authentik.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `install/dwh-auth-client-enrollment.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `install/dwh-auth-server.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `install/dwh-auth-tls.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `install/examples/thothii-installation.local.yaml` | Asset pubblico | Conservare asset o esempio operativo. |
| `install/examples/thothii-installation.server.yaml` | Asset pubblico | Conservare asset o esempio operativo. |
| `install/examples/workspace-bindings.env.example` | Asset pubblico | Conservare asset o esempio operativo. |
| `install/first-start.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `install/standalone-manual-en.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `install/standalone-manual-it.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `installazione-docker-4-contesti.md` | Misto/transitorio | Estrarre dettagli server unici, poi ritirare la ricetta duplicata. |
| `javascripts/layout-init.js` | Asset pubblico | Conservare asset o esempio operativo. |
| `operations/database-management.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `operations/docker-refresh.md` | Operazioni interne | Conservare finché descrive il contesto locale attivo; poi consolidare il lifecycle. |
| `operations/sensitivity-analysis.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `operations/server-codex-handoff.md` | Operazioni interne | Conservare runbook finché migrazione e rollback sono operativamente necessari. |
| `operations/server-handoff-260906-preprocessing-complete.md` | Operazioni/transitorio | Confrontare con server-codex-handoff, assorbire passaggi unici, poi ritirare. |
| `operations/server-upgrade-gitea-workspace-v2.md` | Operazioni interne | Conservare runbook finché migrazione e rollback sono operativamente necessari. |
| `operations/shell-and-localization.md` | Developer/integratore | Conservare integrazione e confine di fiducia; guida pubblica separata. |
| `operations/workspaces.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `plans/2026-08-26-metadata-catalog-from-thothai.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-08-28-ai-catalog-description-generation-spec.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-08-28-ai-catalog-description-generation.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-02-installation-model-catalog.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-08-evidence-management.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-08-memory-evidence-administration.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-08-memory-evidence-simplification-review.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-08-memory-m1-spec.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-08-memory-m1-validation.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-08-memory-management.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-08-security-hardening-prd.md` | Progetto attivo | Conservare PRD aperto; risolvere decisioni/issue prima del ritiro. |
| `plans/2026-09-08-security-hardening-resume-prompt.md` | Transitorio | Ritirare dopo trasferimento di istruzioni e questioni aperte nel PRD/issue. |
| `plans/2026-09-09-archive-repair-x1-validation.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-09-evidence-e1-validation.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-09-evidence-e2-validation.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-09-evidence-e3-validation.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-09-memory-m2-validation.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-09-memory-m3-validation.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-10-administration-pages-spec.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-13-full-shell-and-portal-integration.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-13-full-shell-spec.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/2026-09-14-manual-standalone-installation.md` | Progetto attivo | Conservare fino alla verifica su tre piattaforme; poi assorbire gate e ritirare. |
| `plans/administration-pages/01-navigation.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/administration-pages/02-workspace-readiness.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/administration-pages/03-workbench-family.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `plans/administration-pages/04-embedded-acceptance.md` | Progetto/transitorio | Consolidare invarianti in contratti e gate in issue/test; poi ritirare piano o resoconto. |
| `reports/2026-09-02-psd-sensitivity-shadow.md` | Developer/benchmark | Conservare risultati aggregati come evidenza versionata, non garanzia corrente. |
| `reports/2026-09-02-sensitivity-ner-license-inventory.md` | Developer/release | Conservare provenienza licenze della release; rigenerare quando cambiano le dipendenze. |
| `reports/2026-09-03-psd-progressive-sensitivity-shadow.md` | Developer/benchmark | Conservare risultati aggregati come evidenza versionata, non garanzia corrente. |
| `reports/2026-09-12-admin-issues-28-31.md` | Progetto/transitorio | Consolidare in record di release; ritirare solo dopo trasferimento gate/rollback utili. |
| `reports/2026-09-12-context-shelf-a-implementation.md` | Progetto/transitorio | Consolidare in record di release; ritirare solo dopo trasferimento gate/rollback utili. |
| `reports/2026-09-12-ui-survey-and-graphic-revision-plan.md` | Progetto/transitorio | Consolidare in record di release; ritirare solo dopo trasferimento gate/rollback utili. |
| `reports/2026-09-12-ui-visual-review-delivery.md` | Progetto/transitorio | Consolidare in record di release; ritirare solo dopo trasferimento gate/rollback utili. |
| `reports/2026-09-12-unified-interaction-model.md` | Progetto/transitorio | Consolidare in record di release; ritirare solo dopo trasferimento gate/rollback utili. |
| `reports/2026-09-13-full-shell-implementation.md` | Progetto/transitorio | Consolidare in record di release; ritirare solo dopo trasferimento gate/rollback utili. |
| `reports/2026-09-13-header-layout-refinements.md` | Progetto/transitorio | Consolidare in record di release; ritirare solo dopo trasferimento gate/rollback utili. |
| `reports/2026-09-13-knowledge-reading.md` | Progetto/transitorio | Consolidare in record di release; ritirare solo dopo trasferimento gate/rollback utili. |
| `reports/2026-09-13-knowledge-typography.md` | Progetto/transitorio | Consolidare in record di release; ritirare solo dopo trasferimento gate/rollback utili. |
| `reports/2026-09-13-shell-simplification-review.md` | Progetto/transitorio | Consolidare in record di release; ritirare solo dopo trasferimento gate/rollback utili. |
| `reports/2026-09-13-visual-shell-integration.md` | Progetto/transitorio | Consolidare in record di release; ritirare solo dopo trasferimento gate/rollback utili. |
| `reports/2026-09-14-session-dialogs-release.md` | Operazioni/release | Conservare finché accettazione, osservazione e rollback non sono chiusi. |
| `reports/2026-09-14-session-layout-memory-fix.md` | Operazioni/release | Conservare finché accettazione, osservazione e rollback non sono chiusi. |
| `requirements.lock` | Developer/build | Conservare dipendenze e lock; esclusi dal sito. |
| `requirements.txt` | Developer/build | Conservare dipendenze e lock; esclusi dal sito. |
| `research/2026-08-23-legacy-thothai-metadata-capabilities.md` | Storico/superato | Estrarre vincoli non assorbiti da ADR/contratti, poi ritirare. |
| `research/2026-08-23-postgresql-catalog-deployment-constraints.md` | Storico/superato | Estrarre vincoli non assorbiti da ADR/contratti, poi ritirare. |
| `research/2026-08-23-thothii-metadata-publication-qdrant-seams.md` | Storico/superato | Estrarre vincoli non assorbiti da ADR/contratti, poi ritirare. |
| `research/2026-08-31-browser-erd-library-options.md` | Developer/ricerca | Verificare decisioni ancora aperte; assorbire conclusioni in issue/ADR, poi ritirare. |
| `research/2026-08-31-relationship-management-thothai-to-thothii.md` | Developer/ricerca | Verificare decisioni ancora aperte; assorbire conclusioni in issue/ADR, poi ritirare. |
| `research/2026-09-02-local-sensitive-column-classifier-libraries.md` | Developer/ricerca | Verificare decisioni ancora aperte; assorbire conclusioni in issue/ADR, poi ritirare. |
| `research/2026-09-07-antigravity-gemini-flash-value.md` | Fuori prodotto | Proposta di ritiro dal working tree; Git o raccolta personale su scelta del proprietario. |
| `research/2026-09-07-deepseek-harness-terminal.md` | Fuori prodotto | Proposta di ritiro dal working tree; Git o raccolta personale su scelta del proprietario. |
| `research/2026-09-07-omp-codex-zcode.md` | Fuori prodotto | Proposta di ritiro dal working tree; Git o raccolta personale su scelta del proprietario. |
| `research/2026-09-07-pi-config-vs-oh-my-pi.md` | Fuori prodotto | Proposta di ritiro dal working tree; Git o raccolta personale su scelta del proprietario. |
| `research/2026-09-07-zai-coding-plan-cli-quota.md` | Fuori prodotto | Proposta di ritiro dal working tree; Git o raccolta personale su scelta del proprietario. |
| `research/2026-09-07-zed-acp-omp-pi.md` | Fuori prodotto | Proposta di ritiro dal working tree; Git o raccolta personale su scelta del proprietario. |
| `research/text-to-sql-products.md` | Fuori prodotto | Proposta di ritiro dal working tree; Git o raccolta personale su scelta del proprietario. |
| `skills.md` | Pubblico | Conservare; istruzioni di prodotto/operatore. |
| `stylesheets/extra.css` | Asset pubblico | Conservare asset o esempio operativo. |
| `testing/2026-08-29-ai-catalog-description-generation-acceptance.md` | Developer/QA | Conservare criteri ripetibili e gate; separare risultati di singola release. |
| `testing/2026-08-31-metadata-privacy-description-test-plan.md` | Developer/QA | Conservare scenari; ridurre duplicazioni e distinguere test proposti da eseguiti. |
| `testing/authentication-manual-acceptance.md` | Developer/QA | Conservare criteri ripetibili e gate; separare risultati di singola release. |
| `testing/evidence-lifecycle-test-plan.md` | Developer/QA | Conservare criteri ripetibili e gate; separare risultati di singola release. |
<!-- inventory:end -->
## Verifica iniziale, prima dell'esecuzione dei ritiri
- Build MkDocs con `--strict` e dipendenze bloccate: superata.
- Verifica confine pubblico: 20 pagine, nessun file interno generato o indicizzato.
- Otto fixture positive/negative del confine pubblico: superate.
- Suite corrente installazione/workspace: superata.
- Due verificatori legacy: fallimenti preesistenti descritti sopra; non dichiarati verdi.
- Nessuna certificazione di nuova installazione o nuovo collaudo applicativo.
La pubblicazione remota va verificata separatamente dalla build locale: il ramo sorgente
`main`, il ramo generato `pages` e il sito servito non sono la stessa prova.
### Primo esito remoto del 15 settembre 2026 — diagnosi storica
Il commit `043ffdfa` è stato pubblicato su `main`. La
[pipeline 122](https://git.tylconsulting.it/mptyl/ThothII/actions/runs/122) è terminata
con successo e il ramo `pages` contiene 20 pagine pubbliche, senza pagine interne
nell'indice di ricerca verificato anonimamente.
L'URL `https://git.tylconsulting.it/thothii-docs/` serve però ancora la home precedente,
con `Last-Modified: Wed, 26 Aug 2026 09:00:07 GMT`; il suo indice di ricerca include
ancora architettura e contratti. La richiesta senza cache restituisce lo stesso risultato.
Nel repository la pipeline aggiorna soltanto il ramo `pages`: non è documentato il
collegamento alla directory effettivamente servita dal web server.
Il tentativo SSH in sola lettura con la configurazione locale non è proseguito:
manca una host key ED25519 conosciuta per `git.tylconsulting.it`. Non è stata disabilitata
la verifica dell'identità del server. Servono host/alias amministrativo verificato e
percorso o meccanismo di pubblicazione prima di intervenire sul sito effettivo.
Al momento di quella verifica la copia servita restava da aggiornare. L'accesso è
stato poi risolto usando l'alias SSH già configurato e fidato `contabo`; nessuna
host key è stata aggirata.
## Verifica della bonifica eseguita
- Build MkDocs strict e confine pubblico: superati, 20 pagine.
- Otto fixture del confine pubblico: superate.
- Contratti auth/DWH correnti e relative fixture di mutazione: superati.
- Suite installazione/workspace, Compose canonico e contratto comandi deployment: superate.
- Link Markdown locali di README, PROJECT_STATE e tutti i documenti: nessun target mancante.
- `git diff --check`: superato.
- `test-no-deployment-coupling.sh`: resta rosso su rilievi preesistenti relativi
a contenuti locali PSD e route Omics. Eseguita anche la versione dello script
precedente alla bonifica: stesso esito e identico insieme di rilievi, nessuno nuovo.
Non è un collaudo del sito e non è stato indebolito per nascondere i risultati.
- Nessuna nuova certificazione applicativa, modifica del runtime o chiusura di
collaudi reali. I gate elencati nello snapshot restano espliciti.
## Pubblicazione effettiva completata — 15 settembre 2026
- Sorgente: `4ff91e8d6e771545bc3d8e7a0ac0b027ef66ba14`, pubblicata su `main`.
- [Pipeline 124](https://git.tylconsulting.it/mptyl/ThothII/actions/runs/124):
completata con successo, inclusi i nuovi controlli auth/DWH e pubblicazione `pages`.
- Release servita: `/srv/thothii-docs/releases/20260915T123738Z-4ff91e8d`.
Il symlink `current` punta a questa release; ricreato soltanto `thothii-docs`.
- Configurazione nginx copiata identica; container healthy. Confronto degli ID
dei container prima/dopo: nessun altro container ricreato o fermato.
- Verifica anonima HTTP: home, ricerca e guide manuali IT/EN rispondono 200.
SHA-256 di home e indice remoto identici alla build locale; ricerca: 20 pagine,
nessuna interna.
- Campioni interni `architecture/overview/`, `plans/2026-09-08-memory-management/`
e `operations/compose-reference/`: HTTP 404.
- Release precedente `releases/20260826T090022Z-e910c7d` conservata per rollback;
nessuna vecchia release cancellata. Pubblicazioni future richiedono il passaggio
esplicito del runbook, non il solo push di `main`.
Il precedente blocco sulla copia statica obsoleta è quindi risolto.
+48
View File
@@ -0,0 +1,48 @@
# Compose reference for maintainers
This is an internal topology reference, not a second fresh-installation recipe. Operators
start with the [Italian](../install/standalone-manual-it.md) or
[English](../install/standalone-manual-en.md) manual. It replaces the duplicated four-context
Docker guide without changing the runtime.
The base topology contains frontend, core, catalog-db, qdrant, embedding, embedding-model-init,
catalog-migrate and profile-gated workspace-maintenance. Pi runs in core; DWH and generative
model endpoints remain installation settings. Reference preprocessing and Memory have distinct
lifecycles and collections. See [preprocessing](../contracts/workspace-preprocessing-cli.md).
## Configuration and migration boundaries
- Use one physical absolute installation descriptor path, its generated operator environment,
Compose project name, base/profile overlays, transport overlays and generated model overlay.
Mixing a generic `deploy/env/local.env` invocation with a native `tht` installation creates
a different stack; it is not an equivalent lifecycle command.
- `THT_INSTALLATION_CONFIG_SOURCE` identifies the protected host descriptor. The read-only
backend runtime mount is `/run/thothii-installation/thothii-installation.yaml`, exposed through
`THT_INSTALLATION_CONFIG_FILE`; the runtime path is not a host source path.
- Catalog runtime/migrator passwords and provider credentials are protected files. A provider's
`authentication.apiKeyEnv` names an allowed bundle entry; it is not a raw key. Private CAs
are separately mounted PEM files, not bundle values. See the repository's
`deploy/secrets/README.md` and the [model catalog guide](../general/pi-configuration.md).
- Generate projections after descriptor edits. Do not edit generated model/auth/frontend files.
Apply the installation's normal restart process when authored configuration changes.
- Explicitly start catalog-db and run catalog-migrate before application rollout on a fresh
database or after an approved schema update. That service runs Catalog and Memory migrations;
neither normal backend startup nor `tht start` implicitly performs them.
- Server deployments retain their reviewed session/auth/network/storage overlays. A writable
server Pi-state parent must be initialized with the regular targets expected by the read-only
nested mounts; use `scripts/prepare-server-pi-state.sh` with the installation's verified UID/GID.
## Deployment-specific authority
For the prepared generic server environment, the base/profile/session override
combination is `-f compose.yaml -f deploy/compose.server.yaml
-f deploy/compose.session-server.yaml.example`, with `--env-file` supplied before
the overrides. Add the reviewed transport/generated overlays for that installation;
this fragment alone is not a complete startup command.
The current [server handoff](server-codex-handoff.md) and
[legacy upgrade runbook](server-upgrade-gitea-workspace-v2.md) retain maintenance, backup and
rollback gates. Embedded/upstream identity is not standalone OIDC. Local/standalone setup
does not authorize replacing a running server stack, resetting volumes, or copying another
machine's descriptor. `scripts/run-stack.sh` remains a low-level path for an explicitly
prepared generic environment, not the public manual's default startup command.
+3 -3
View File
@@ -67,7 +67,7 @@ SSH uses a private key, optional key passphrase, mandatory `known_hosts`, and op
TLS CA/server name. REST prefers `POST /rpc/schema_snapshot`; when it is absent, the catalog may TLS CA/server name. REST prefers `POST /rpc/schema_snapshot`; when it is absent, the catalog may
use the same strict v1 snapshot through one read-only `POST /rpc/run_query`. An unavailable use the same strict v1 snapshot through one read-only `POST /rpc/run_query`. An unavailable
capability, malformed snapshot, or connector error applies no catalog changes. See the capability, malformed snapshot, or connector error applies no catalog changes. See the
[schema snapshot contract](../contracts/catalog-schema-snapshot.md). [schema snapshot contract](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/catalog-schema-snapshot.md).
## Synchronize authoritative schema metadata ## Synchronize authoritative schema metadata
@@ -176,6 +176,6 @@ atomic operation. It skips empty generated descriptions, reports aggregate copie
counts, and retains the generated text. Because this can replace reviewed descriptions, the counts, and retains the generated text. Because this can replace reviewed descriptions, the
interface requires explicit confirmation before applying it. interface requires explicit confirmation before applying it.
The decisions behind this surface are [ADRs 0001–0011](../adr/0001-postgres-metadata-catalog.md) The decisions behind this surface are [ADRs 0001–0011](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/adr/0001-postgres-metadata-catalog.md)
and the detailed acceptance record is and the detailed acceptance record is
[AI catalog description generation acceptance](../testing/2026-08-29-ai-catalog-description-generation-acceptance.md). [AI catalog description generation acceptance](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/testing/2026-08-29-ai-catalog-description-generation-acceptance.md).
@@ -0,0 +1,98 @@
# Publishing the public manual
This is an internal operator runbook. Publishing documentation must not recreate
the ThothII application, Gitea, proxy, DWH or other documentation containers.
## Three separate states
1. `main` contains reviewed source and the public allowlist in `mkdocs.yml`.
2. Gitea Actions builds and validates the generated `pages` branch. A successful
Actions run alone does **not** update the live site.
3. The live [manual](https://git.tylconsulting.it/thothii-docs/) is served by the
dedicated `thothii-docs` nginx container on the host reached through the existing
trusted SSH alias `contabo`. Its Compose project/service are `thothii-docs`/`docs`.
This procedure is manual. No timer, webhook, credential or automatic deployment
has been added. Use the operator's existing SSH authorization and known host key;
never disable host verification to make a publication work.
## Build and stage
From a clean ThothII `main` checkout already pushed to Gitea:
```sh
git status --short
git rev-parse HEAD
./scripts/build-docs.sh
bash scripts/auth-docs-smoke.sh
bash scripts/verify-dwh-auth-docs.sh
bash scripts/test-auth-docs-smoke.sh
bash scripts/test-verify-dwh-auth-docs.sh
```
The strict build also verifies the public boundary: exactly 20 navigation pages,
five approved source assets, no internal pages/assets, and matching search entries.
The locked Windmill theme requires a separate exact allowlist of 25 static assets;
MkDocs applies `exclude_docs` to those files too. Never replace the list with broad
directory exceptions. Sources under `docs/` may not shadow theme asset paths.
The verifier follows stylesheet/script/image references and CSS font references
from generated pages and fails when a local dependency is absent.
Choose a unique release identifier consisting of UTC timestamp and source SHA.
Record the current symlink and container identity before proceeding:
```sh
ssh -oBatchMode=yes -oStrictHostKeyChecking=yes contabo \
'readlink /srv/thothii-docs/current; docker inspect --format "{{.Id}} {{.State.Health.Status}}" thothii-docs'
```
On the server, create `/srv/thothii-docs/releases/RELEASE/site` only if RELEASE
does not exist. Copy the current `nginx.conf` unchanged into the new release.
Transfer the local built `site/` into that new empty directory with rsync over
the same verified SSH connection. Do not use `--delete` against `current`, and do
not edit routing, TLS, network or authentication configuration.
## Activate only this site
Replace RELEASE below with the validated identifier, not a user-supplied path.
Keep the previously recorded release for rollback.
```sh
cd /srv/thothii-docs
test -f releases/RELEASE/site/index.html
test -f releases/RELEASE/nginx.conf
docker exec thothii-docs nginx -t
ln -s releases/RELEASE current.next
mv -Tf current.next current
docker compose --project-name thothii-docs -f docker-compose.yml \
up -d --no-deps --force-recreate docs
```
Check that `current.next` does not already exist before creating it; stop if it
does, because another publication or recovery may be in progress. Changing the
symlink alone is insufficient: an existing Docker bind mount still resolves to
the old release. The service recreation above remounts the new directory. It may
briefly interrupt this manual only.
## Verify, record, or roll back
- Wait for `docker inspect` to report this container healthy; verify mounted
paths and nginx configuration. Stop after a bounded timeout (for example 60 s).
- Without login/cookies, require HTTP 200 for home, search and both
`install/standalone-manual-it/` and `install/standalone-manual-en/`.
- Compare live home and `search/search_index.json` checksums to the build. Search
must contain only the 20 approved pages, never plans, reports or architecture.
- Check the actual stylesheet and JavaScript URLs in both installation pages:
HTTP 200, CSS served as `text/css`, JavaScript with a valid script content type,
and fonts available. In a browser confirm the stylesheets load and the layout is
styled. HTML 200 alone is not enough to accept a publication.
- Require HTTP 404 for retired/internal paths, including `architecture/overview/`,
`plans/2026-09-08-memory-management/`, and `operations/compose-reference/`.
- Record source SHA, release ID, previous release and checks in the cleanup/release
record. Do not call the publication complete solely because `pages` was pushed.
If health or public verification fails, point `current` back to the recorded
previous release using a fresh temporary symlink and the same atomic rename, then
recreate **only** service `docs` with the same Compose command. Verify health and
old site availability. Do not delete either release during recovery. Historical
releases are retained; pruning them requires a separate retention decision.
+6 -11
View File
@@ -118,12 +118,12 @@ other CPU workloads.
Run the first evaluation in shadow mode: read the source with its existing read-only role, do not Run the first evaluation in shadow mode: read the source with its existing read-only role, do not
save proposed flags, and report only aggregate counts, rule IDs, coverage, and timings. Never copy save proposed flags, and report only aggregate counts, rule IDs, coverage, and timings. Never copy
matched values into test output. Use a separately approved, labeled Italian corpus to calculate matched values into test output. Use a separately approved, labeled Italian corpus to calculate
precision and recall; raw PSD values must remain inside the authorized environment. precision and recall; source values must remain inside the authorized environment.
Inside the configured core runtime, the non-mutating command is: Inside the configured core runtime, the non-mutating command is:
```bash ```bash
npm run sensitivity:shadow -- psd-clinical npm run sensitivity:shadow -- <workspace-id>
``` ```
It reads catalog metadata and source values but emits one aggregate JSON object with no database, It reads catalog metadata and source values but emits one aggregate JSON object with no database,
@@ -139,12 +139,7 @@ Enabling NER by default requires all of these gates:
If a gate fails, leave NER disabled. The deterministic policy remains available and produces the If a gate fails, leave NER disabled. The deterministic policy remains available and produces the
binary draft from its scan coverage; no content is sent to an internal or external LLM. binary draft from its scan coverage; no content is sent to an internal or external LLM.
The first aggregate PSD shadow comparison is recorded in Benchmarks from a particular installation are not a guarantee for another database or machine.
[`2026-09-02-psd-sensitivity-shadow.md`](../reports/2026-09-02-psd-sensitivity-shadow.md). On the NER remains opt-in until an approved evaluation establishes that additional findings justify
local CPU runner, NER found additional entities but reduced total coverage under the superseded their false-positive rate and operational cost. Keep benchmark and release records with the
global deadline. The v2 benchmark removed that confounder: CPU NER added 18 sensitive proposals and installation's technical evidence, separate from this operator procedure.
increased the warm analysis time from 50.1 to 61.3 seconds. It remains opt-in until a labeled Italian
evaluation establishes that the additional findings justify their false-positive rate and cost.
The deterministic progressive PSD run is recorded in
[`2026-09-03-psd-progressive-sensitivity-shadow.md`](../reports/2026-09-03-psd-progressive-sensitivity-shadow.md):
both deterministic and CPU-NER profiles assessed all 2,275 columns with zero `unknown` decisions.
@@ -722,6 +722,6 @@ la nuova installazione è ferma ma ispezionabile.
- [OIDC generico](../install/authentication-oidc.md) - [OIDC generico](../install/authentication-oidc.md)
- [Operazioni workspace](workspaces.md) - [Operazioni workspace](workspaces.md)
- [Configurazione dei modelli Pi](../general/pi-configuration.md) - [Configurazione dei modelli Pi](../general/pi-configuration.md)
- [Contesti Docker](../installazione-docker-4-contesti.md) - [Contesti Docker](compose-reference.md)
- [Database Management](database-management.md) - [Database Management](database-management.md)
- [Analisi locale della sensibilità](sensitivity-analysis.md) - [Analisi locale della sensibilità](sensitivity-analysis.md)
+11 -8
View File
@@ -54,7 +54,7 @@ Omics. Le preferenze non modificano il descrittore installato.
Per una nuova installazione autonoma, selezionare esplicitamente full: Per una nuova installazione autonoma, selezionare esplicitamente full:
```bash ```bash
tht setup --profile local --shell-mode full --shell-default-locale en tht setup --complete --profile local --shell-mode full --shell-default-locale en
``` ```
Il setup senza opzioni shell conserva per compatibilità il default embedded. Il setup senza opzioni shell conserva per compatibilità il default embedded.
@@ -184,19 +184,22 @@ Ci sono tre scelte distinte:
| Scelta | Dove viene conservata | Cosa influenza | | Scelta | Dove viene conservata | Cosa influenza |
| --- | --- | --- | | --- | --- | --- |
| Lingua UI | preferenza full o stato Omics | label, form, messaggi e controlli | | Lingua UI | preferenza full o stato Omics | navigazione, amministrazione e messaggi generali |
| Lingua di interazione | `interaction_language` nel manifest | nuove domande, spiegazioni e scelte del modello | | Lingua di interazione | `interaction_language` nel manifest | domande, spiegazioni, scelte e controlli HITL |
| Lingua workspace | configurazione del workspace | documenti, descrizioni e contenuti di dominio | | Lingua workspace | configurazione del workspace | documenti, descrizioni e contenuti di dominio |
La creazione web acquisisce la lingua UI prima delle operazioni asincrone e la La creazione web acquisisce la lingua UI prima delle operazioni asincrone e la
invia come `interactionLanguage`. Un cambio successivo non modifica quella invia come `interactionLanguage` di ripiego. Il CLI riconosce localmente la lingua
richiesta. La ripresa legge il manifest e non usa il locale del browser come della domanda originale e la fissa nel manifest; usa il ripiego per testo troppo
breve, ambiguo o composto soltanto da codice. Un cambio successivo non modifica
quella richiesta. Il gate trasmette la lingua nei widget: anche i controlli HITL
seguono la sessione, mentre la navigazione conserva la lingua UI.
La ripresa legge il manifest e non usa il locale del browser come
override. SQL, identificatori, valori e citazioni dei contenuti rimangono invariati. override. SQL, identificatori, valori e citazioni dei contenuti rimangono invariati.
Per sessioni precedenti senza `interaction_language`, la prima ripresa fissa la Per sessioni precedenti senza `interaction_language`, la prima ripresa fissa la
lingua del workspace in modo idempotente. Se il workspace era stato modificato lingua della domanda in modo idempotente, usando la lingua del workspace come
nel frattempo, non esiste una registrazione da cui ricostruire con certezza la ripiego. Le sessioni che hanno già una lingua fissata la conservano.
vecchia lingua: il criterio di compatibilità è quella disponibile alla ripresa.
Il cambio lingua di Omics ricarica la pagina. ThothII ricorda soltanto l'identificatore Il cambio lingua di Omics ricarica la pagina. ThothII ricorda soltanto l'identificatore
della sessione per utente e pagina, senza salvare una trascrizione nel browser. della sessione per utente e pagina, senza salvare una trascrizione nel browser.
+2 -2
View File
@@ -73,9 +73,9 @@ only that run's safe stage, error code, and finish time; use `docker compose log
corresponding service log. corresponding service log.
The contract gives exact validation, exit code, and JSON rules in The contract gives exact validation, exit code, and JSON rules in
[Workspace preprocessing CLI](../contracts/workspace-preprocessing-cli.md). For Evidence source [Workspace preprocessing CLI](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/workspace-preprocessing-cli.md). For Evidence source
forms and the schema-v4 descriptor contract, see forms and the schema-v4 descriptor contract, see
[Workspace Evidence v3](../contracts/workspace-evidence-v3.md). [Workspace Evidence v3](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/workspace-evidence-v3.md).
## Transport and revision rules ## Transport and revision rules
@@ -1,695 +0,0 @@
# Metadata Catalog di ThothII: ricognizione ThothAI e percorso incrementale
Data: 2026-08-26; aggiornato 2026-08-27
Stato: ricognizione e progettazione completate; navigazione, CRUD Workspace Database, Catalog
Table, Catalog Column, Catalog Relationship e sincronizzazione durevole dello schema implementati
il 2026-08-27. Generazione AI e integrazione con il workflow core restano negli step successivi.
## Obiettivo
ThothII deve introdurre un contesto amministrativo separato, il **Metadata Catalog**, per gestire
il database associato a ciascun workspace, la sua struttura fisica introspezionata e i metadati
semantici oggi rappresentati da `schema/annotations.yaml`.
Il programma procede per step indipendenti. Il primo step ha aggiunto l'accesso dalla sidebar; il
secondo ha sostituito la superficie vuota con il CRUD di configurazione, il PostgreSQL interno e i
test di connessione; gli step successivi hanno aggiunto navigazione gerarchica, colonne, relazioni
fisiche e sincronizzazione durevole dell'intero schema. Non introduce ancora generazione AI o
integrazione con il workflow core.
Questa analisi usa come riferimento il working tree legacy osservato in
`Thoth/ThothAI`. Non è stato verificato che quel contenuto corrisponda a una release o a un tag
canonico; i percorsi e i comportamenti descrivono il sorgente disponibile il 2026-08-26.
## Decisioni già confermate
1. Ogni Workspace Database appartiene a un solo workspace tramite un `workspace_id` obbligatorio e
univoco; un workspace può avere al massimo un Workspace Database. Poiché i workspace non sono
righe del catalogo PostgreSQL, l'associazione è un riferimento logico validato contro
`thoth-workspaces.yaml`, non una foreign key SQL.
2. Il CRUD non crea né rinomina workspace. Identità e lista ordinata dei workspace restano
autorevoli in `thoth-workspaces.yaml`; il catalogo conserva il loro identificatore stabile.
3. La struttura fisica viene acquisita interrogando il database esterno tramite i dati di
connessione registrati per il Workspace Database.
4. I contenuti semantici equivalenti a `annotations.yaml` vengono generati con l'AI e conservati nel
PostgreSQL interno.
5. Per PSD è prevista l'importazione delle annotations esistenti. Gli altri database partiranno
dalla struttura introspezionata e genereranno i metadati semantici da zero.
6. `annotations.yaml` sarà sostituito anche come input del core in uno step futuro. Il repository è
in fase di test e non è richiesta la conservazione delle sessioni esistenti durante il cutover.
7. La gestione catalogo resta una superficie separata dal processo NL→SQL. La futura integrazione
deve essere esplicita e non deve modificare fasi, gate o semantica del workflow.
8. Il link iniziale è visibile agli utenti con `workspace.manage`, usa stato React locale e non
introduce un router.
9. La pagina iniziale è vuota, segue il tema, nasconde l'intera colonna core e non interrompe una
sessione live. Le azioni di apertura, resume o creazione sessione riportano al core.
10. La compatibilità con il modello ThothAI è semantica, non una copia letterale: configurazione e
contenuti semantici sono campi relazionali mutabili, mentre identità e appartenenza della
struttura fisica derivano dall'introspezione; i segreti restano nel secret store e lo stato dei
job non viene mescolato ai dati amministrativi.
11. Il CRUD amministra il Metadata Catalog e non esegue DDL sul database esterno, che resta
read-only.
12. La prima versione supporta PostgreSQL; il confine di introspezione dovrà permettere di
aggiungere altri dialetti senza cambiare il modello del catalogo.
13. I segreti dei Workspace Database riusano il secret store cifrato di ThothII. Il catalogo
conserva riferimenti ai segreti e nessuna API, esportazione o log ne restituisce i valori.
14. La UI usa AG Grid Community per la lista master e un pannello React separato per il dettaglio;
non dipende dalle funzionalità master-detail di AG Grid Enterprise.
15. Un Workspace Database il cui `workspace_id` scompare dal catalogo YAML non viene cancellato
automaticamente: diventa orphaned e può soltanto essere recuperato, riassegnato o eliminato
esplicitamente da un amministratore.
16. La prima vertical slice gestisce configurazione del Workspace Database, riferimenti ai segreti,
test di connessione e stato. La seconda gestisce le Catalog Table: la collezione e i nomi sono
controllati dall'introspezione, mentre la descrizione curata è modificabile. Le slice successive
hanno aggiunto Catalog Column, Catalog Relationship e sincronizzazione durevole dello schema.
17. Il modello non conserva il `name` libero di ThothAI: nome e ID visualizzati appartengono al
workspace YAML, mentre `database_name` identifica il database PostgreSQL esterno.
18. Database management supporta i tre trasporti già riconosciuti da ThothII: `postgres_direct`,
`rest_api` e `ssh_tunnel`. PSD rimane un solo Workspace Database: usa la connessione diretta sul
server e l'endpoint REST in locale tramite una Database Binding specifica dell'installazione.
Questo supporto non abilita automaticamente `ssh_tunnel` nel runtime NL→SQL.
19. Una configurazione può essere salvata prima di una connessione riuscita. Il test separato
produce uno stato `untested`, `reachable` o `failed`; attivazione e introspezione richiedono uno
stato raggiungibile.
20. Il CRUD e il test di connessione richiedono `database.manage`; inserimento e sostituzione dei
segreti continuano a richiedere `workspace.secrets.manage`.
21. Il Workspace Database e il modo di raggiungerlo sono entità distinte. Ogni catalogo di
installazione conserva una sola Database Binding attiva per workspace: PSD usa `rest_api` in
locale e `postgres_direct` sul server senza duplicare il Workspace Database.
22. Nel modello finale il Metadata Catalog è autorevole per engine, `database_name`, schema,
capacità e binding. Lo YAML resta autorevole per identità e contenuti del workspace; i campi
DWH correnti saranno importati, confrontati e rimossi soltanto durante un cutover esplicito.
23. La lista master è l'unione fra workspace YAML e record del catalogo: mostra workspace
`unconfigured`, database configurati e record `orphaned`.
24. Ogni introspezione registra le capability disponibili. Una capability `unavailable` non viene
rappresentata come una collezione osservata ma vuota; REST può completare con successo anche
quando indici o enum non sono supportati.
25. Il Metadata Catalog non introduce snapshot, draft o pubblicazioni. Configurazione e contenuti
semantici, inclusi quelli futuri generati dall'AI, sono normali campi modificabili; la struttura
osservata cambia soltanto con una sincronizzazione esplicita.
26. Il normale Delete elimina realmente il Workspace Database, la Database Binding e i relativi
record catalogo e segreti. Non modifica il DWH esterno né il repository YAML; il workspace torna
visibile nella lista master come `unconfigured`.
27. La prima versione gestisce un solo schema obbligatorio per Workspace Database, identificato
dalla coppia `database_name + schema`; per PSD la coppia è `postgres + datawarehouse`.
28. I record mantengono soltanto `created_at`, `updated_at` e un contatore `version` per optimistic
concurrency. Non esistono storico delle revisioni, rollback o audit applicativo delle modifiche.
29. `workspace_databases` conserva soltanto UUID, `workspace_id` unique, engine, `database_name`,
schema, timestamp e version. Il nome visualizzato appartiene al workspace YAML.
30. Ogni Workspace Database ha al massimo una riga `database_bindings`. Una singola tabella usa
check constraint dipendenti da `transport` per i campi direct, REST e SSH; non esiste un flag
`active`, perché ciascuna installazione conserva una sola binding.
31. `rest_api` configura il Thoth REST Connector tipizzato: base URL, autenticazione e TLS sono dati
della binding, mentre path RPC e shape delle risposte appartengono al contratto applicativo e non
sono liberamente configurabili.
32. Il test connessione usa soltanto una configurazione già salvata ed è associato alla sua
`version`. Ogni modifica della binding o dei segreti invalida il risultato precedente e riporta
lo stato a `untested`.
33. Password, API key e chiavi sono write-only: l'API espone soltanto `configured`, un campo vuoto
conserva il valore esistente e la sostituzione è un'azione esplicita. Delete rimuove anche i
segreti associati.
34. La pagina usa AG Grid come master e un form React come detail, con sezioni Database, Connection
e TLS/SSH condizionali. Non esiste un'azione globale `Add database`: ogni riga `unconfigured`
offre `Configure catalog`, apre il form già vincolato a quello specifico workspace YAML e crea il
record soltanto al Save; `workspace_id` non è selezionabile né modificabile.
35. La grid mostra separatamente revisione/Evidence del workspace, binding runtime NL→SQL e
configurazione del Metadata Catalog, oltre a database, schema, endpoint e ultimo aggiornamento.
Su schermi piccoli il dettaglio occupa il pannello completo. Il cambio riga con modifiche non
salvate e Delete richiedono conferma, senza conferma testuale tipizzata.
36. La Database Binding conserva `connection_status`, `tested_version`, `last_tested_at`, un codice
errore e un messaggio breve sanificato. Non conserva stack trace, DSN, credenziali o output grezzo
del driver.
37. Le API vivono sotto `/api/catalog`: list/create di `/databases`, get/patch/delete di
`/databases/:id`, sostituzione dei segreti sotto `/databases/:id/secrets`, test connessione sotto
`/databases/:id/test` e list/patch/sync delle tabelle sotto `/databases/:id/tables`.
38. `GET /api/catalog/databases` restituisce l'intera master list unificata; AG Grid Community applica
client-side ricerca, filtri e ordinamento. La prima versione non introduce paginazione server o
funzionalità AG Grid Enterprise.
39. Il Metadata Catalog vive nello stesso processo Fastify come modulo isolato con repository,
service, route, diagnostica e readiness proprie. L'indisponibilità del catalogo non modifica
sessioni, SSE o health del core e non giustifica ancora un microservizio separato.
40. Il backend mantiene `pg@8.22.0` e aggiunge `kysely@0.29.5` per query e transazioni tipizzate. Le
migrazioni Kysely sono timestampate, compilate con il backend ed eseguite da un comando
`catalog:migrate` separato; l'applicazione non migra automaticamente il database all'avvio.
41. Lo stack aggiunge un servizio interno `catalog-db` con volume persistente, ruolo runtime DML,
ruolo migrator DDL e job one-shot `catalog-migrate`. Un catalogo indisponibile produce 503 sulle
sole route catalogo.
42. La prima vertical slice è amministrativa: scrive il catalogo ma non cambia ancora il runtime di
sessioni e workflow, che continua a usare YAML e binding correnti fino al cutover esplicito.
43. `Configure` precompila senza salvare engine, database e schema dal descriptor e i dati non
sensibili dalla binding effettiva. L'amministratore verifica, inserisce i segreti e salva; non
esiste importazione silenziosa.
44. Unit e route test usano un repository fake; una suite PostgreSQL Testcontainers separata verifica
migrazioni, constraint, transazioni, optimistic concurrency e cascade. SQLite ed emulatori non
sono sostituti ammessi per questi test.
45. La navigazione delle entità catalogo è gerarchica e senza scorciatoie globali: `Databases →
Database → Overview | Tables → Table`. Non esistono una voce globale Tables, un filtro globale
Database o una preselezione implicita; Columns continuerà sotto Table e Relationships sotto
Database.
46. Una Catalog Table conserva nome fisico, `source_comment`, descrizione curata nullable,
`generated_description` nullable per lo step AI futuro, version e timestamp. La UI mostra come
tre campi indipendenti senza fallback visivo: source comment read-only, generated description
modificabile e description modificabile. I valori null restano celle e controlli vuoti.
47. Le Catalog Table non possono essere aggiunte o rinominate manualmente. Un amministratore può
però ripulire esplicitamente le proiezioni nel Metadata Catalog senza modificare il database
esterno; `Sync tables` legge le tabelle PostgreSQL ordinarie e partizionate dello schema scelto,
mentre viste e materialized view sono escluse.
48. La sincronizzazione è esplicita. La scansione avviene fuori dalla transazione del catalogo; il
diff viene applicato atomicamente soltanto se la version del Workspace Database è ancora quella
sottoposta a scansione. Una scansione fallita non modifica il catalogo.
49. Tabelle nuove vengono create, i commenti sorgente vengono aggiornati e quelle non più osservate
vengono eliminate definitivamente. La rimozione di tabelle, colonne o relazioni richiede la
conferma dell'esatto piano distruttivo; se il secondo scan produce una fotografia differente,
l'applicazione richiede una nuova conferma.
50. Un rename fisico è intenzionalmente delete più create e perde i metadati curati. Le colonne e
relazioni dipendenti vengono eliminate in cascade insieme alla Catalog Table.
51. L'introspezione vive nel modulo catalogo Fastify dietro un adapter. PostgreSQL diretto e tunnel
SSH usano il catalogo `pg_catalog`; REST preferisce il contratto tipizzato
`POST /rpc/schema_snapshot` e, quando quell'RPC non è esposto, usa come fallback compatibile una
singola query read-only tramite `POST /rpc/run_query`. Entrambi i percorsi devono produrre la
stessa fotografia v1 stretta descritta in `docs/contracts/catalog-schema-snapshot.md`.
52. Test connessione e sincronizzazione sono serializzati per Workspace Database, hanno timeout e
richiedono che la binding nella version corrente abbia un test `reachable` prima di qualsiasi
Catalog Sync Run. La scansione asincrona ha un timeout separato, di default dieci minuti.
53. Il tunnel SSH usa OpenSSH in modalità stdio `-W`, chiave privata e passphrase opzionale dal
secret store, `known_hosts` obbligatorio, `StrictHostKeyChecking=yes`, agent e configurazione
globale disabilitati. Non è ammesso TOFU. TLS PostgreSQL con CA e server name resta verificato
anche attraverso il tunnel.
54. In questo slice `ssh_tunnel` è una binding supportata da Database management per Test connection
e Schema Sync. Il renderer e il runtime delle sessioni NL→SQL restano fuori scope e continuano a
rifiutarla finché non verrà deciso il relativo cutover.
55. I menu di azione a livello Workspace Database espongono separatamente `Synchronize tables`,
`Synchronize relationships` e `Synchronize all`. Su una selezione di
database lo scope scelto viene avviato per ogni database idoneo; non viene sostituito
implicitamente con una sincronizzazione completa.
56. Lo scope Columns è disponibile dalla grid Tables e limita la riconciliazione alle tabelle
selezionate; la pagina Columns non espone azioni di sincronizzazione. La grid Tables espone
`Synchronize columns` sulle tabelle selezionate.
## Correzione del modello mentale corrente
`schema/annotations.yaml` non contiene l'intero schema del database.
- `physical.yaml` è un artefatto derivato dall'introspezione. Contiene database, schema, timestamp,
tabelle, colonne, tipi, nullability, default, primary key, commenti sorgente, esempi, foreign key
fisiche e indici.
- `annotations.yaml` contiene metadati curati: descrizioni e concetti delle tabelle; descrizioni,
sinonimi, concetti, evidence, note e override `eligible` delle colonne; foreign key logiche.
- Il rendering M-Schema fonde questi due input. Le annotations prevalgono sui commenti sorgente e
le relazioni logiche vengono unite alle foreign key fisiche.
La sostituzione del solo file annotations non elimina automaticamente l'introspezione fisica. Il
nuovo catalogo dovrà conservare una distinzione esplicita fra fatti osservati nel database e
contenuto semantico modificabile.
## Architettura ThothII rilevante
### Autorità e revisionamento attuali
Il repository dei workspace contiene:
```text
thoth-workspaces.yaml
<workspace-id>/workspace.yaml
<workspace-id>/schema/annotations.yaml
<workspace-id>/evidence/**
```
Il backend legge descriptor e annotations allo stesso commit Git. Durante l'attivazione valida il
blob, lo copia atomicamente nello snapshot immutabile della revisione e registra commit, blob ID e
digest. Le nuove sessioni vengono legate a quella revisione; resume e SQL salvato riaprono lo stesso
snapshot.
Punti principali:
- `backend/src/workspaces/schema.ts`: descriptor v3 e singolo `dwh.database`/`dwh.schema`;
- `backend/src/workspaces/git-repository.ts`: lettura sicura del blob annotations al commit;
- `backend/src/workspaces/registry.ts`: validazione e attivazione atomica;
- `backend/src/workspaces/annotations-sync.ts`: materializzazione revision-qualified;
- `backend/src/workspaces/runtime-config-lease.ts`: binding dello snapshot al runtime;
- `harness/tht/mschema/models.py`: contratti `PhysicalSchema` e `Annotations`;
- `harness/tht/mschema/render.py`: fusione fisico/semantico;
- `harness/tht/cli/vector_cmd.py`: indicizzazione schema in Qdrant.
### Consumatori da preservare al cutover futuro
Le annotations incidono oggi su:
- override `eligible` prima del campionamento LSH;
- suggerimento, controllo e accettazione delle foreign key logiche;
- descrizioni, concetti e sinonimi dei record schema in Qdrant;
- retrieval delle tabelle e colonne candidate;
- rendering M-Schema usato dal gate F4 e dalla generazione SQL;
- digest della revisione accettata durante il preprocessing.
Il futuro cutover non potrà limitarsi a rimuovere il file: dovrà fornire al core lo stesso contenuto
effettivo, con un'identità coerente e test di equivalenza. Poiché non occorre preservare le sessioni
di test esistenti, non serve progettare compatibilità con i vecchi manifest, ma resta necessario
evitare letture parziali o semanticamente incoerenti.
## Inventario ThothAI
### Modelli legacy
I modelli sono definiti in `Thoth/ThothAI/backend/thoth_core/models.py`.
#### `SqlDb`
Campi di connessione osservati:
- `name`;
- `db_host`, `db_port`;
- `db_type`;
- `db_name`, `schema`;
- `user_name`, `password`;
- `db_mode`;
- configurazione SSH e Informix opzionale.
Il modello contiene anche scope, JSON dello scope, ERD, direttive, campi GDPR, collegamento a
`VectorDb` e numerosi campi di stato/task/log per lavori AI asincroni.
I tipi legacy dichiarati sono Informix, MariaDB, MySQL, Oracle, PostgreSQL, SQL Server e SQLite.
Questo elenco non costituisce automaticamente un requisito per ThothII: il core corrente supporta
PostgreSQL e l'estensione ad altri dialetti dovrà essere decisa separatamente.
#### `SqlTable`
- `name`;
- `description`;
- `generated_comment`;
- foreign key obbligatoria a `SqlDb`, con cancellazione cascade.
#### `SqlColumn`
- `original_column_name` e alias `column_name`;
- `data_format` normalizzato;
- `column_description`;
- `generated_comment`;
- `value_description`;
- stringhe denormalizzate `pk_field` e `fk_field`;
- foreign key obbligatoria a `SqlTable`, con cancellazione cascade.
#### `Relationship`
Contiene quattro foreign key obbligatorie:
- `source_table` e `source_column`;
- `target_table` e `target_column`.
Il form admin verifica che le tabelle appartengano allo stesso database e che ogni colonna
appartenga alla tabella selezionata. Il database non impone però gli stessi check.
#### `Workspace`
ThothAI usa `Workspace.sql_db` come foreign key nullable verso `SqlDb`: un workspace seleziona un
solo DB, mentre lo stesso DB può essere riusato da più workspace. ThothII adotterà invece una
relazione uno-a-uno: `workspace_id` deve essere unico nel catalogo.
### Lacune dei constraint legacy
Non risultano constraint database-level per:
- unicità del nome database nel workspace;
- unicità `(database, table name)`;
- unicità `(table, column name)`;
- unicità degli estremi di una relationship;
- appartenenza degli estremi della relationship allo stesso database;
- corrispondenza fra colonna e tabella dichiarata.
ThothII deve applicare queste invarianti sia nel database interno sia nel servizio applicativo. La
sola validazione del form non è sufficiente perché API, import e job la possono aggirare.
### Django Admin e UX da replicare concettualmente
ThothAI espone il CRUD tramite il Django Admin standard, registrato da
`backend/thoth_core/admin.py` e pubblicato su `/admin/`.
Capacità utili:
- lista database con ricerca per nome, host, tipo, database e schema;
- fieldset separati per identità, connessione, autenticazione, SSH e stato;
- lista tabelle filtrabile per database;
- lista colonne filtrabile in cascata per database e tabella;
- lista relazioni con estremi leggibili e filtri per database e tabelle;
- form relazione con dropdown dipendenti database → tabella → colonna;
- validazione degli estremi prima del salvataggio;
- azioni separate per test connessione, introspezione, import/export e generazione AI;
- azioni bulk sulle righe selezionate.
ThothII deve replicare i contratti di interazione e validazione, non il rendering server-side o i
template Django.
### Introspezione legacy
`Thoth/ThothAI/backend/thoth_core/dbmanagement.py` usa `thoth-dbmanager` per:
1. costruire l'adapter del dialetto;
2. acquisire tabelle;
3. acquisire e normalizzare colonne e tipi;
4. acquisire relazioni;
5. creare le eventuali colonne mancanti necessarie alle relazioni;
6. aggiornare i campi PK/FK denormalizzati.
Il comportamento è principalmente additivo: usa `get_or_create` o controlli `exists`, aggiorna
alcuni commenti, ma non riconcilia in modo completo rename, rimozioni o drift. Non va copiato così
com'è. Il processo ThothII implementato distingue scansione, differenze osservate e applicazione
della nuova snapshot.
### Generazione AI legacy
ThothAI dispone di azioni e workflow per:
- commenti delle tabelle;
- commenti delle colonne;
- scope del database;
- ERD Mermaid;
- documentazione del database;
- analisi GDPR.
Per il requisito attuale sono direttamente rilevanti descrizioni di tabelle e colonne, scope e
metadati semantici. ERD, documentazione aggregata e GDPR sono estensioni future, non prerequisiti
del CRUD iniziale.
La separazione `description`/`generated_comment` del legacy non offre versioning o approvazione
robusti. Nei passi successivi andrà deciso se l'output AI è una proposta revisionabile o diventa
immediatamente il valore editabile corrente.
### Import ed export legacy
ThothAI offre:
- CSV di database, tabelle, colonne e relazioni;
- export di struttura per workspace;
- import mediante `import_db_structure`;
- script SQL dei commenti per più dialetti;
- aggiornamento delle descrizioni colonna da CSV.
Il futuro import PSD dovrà leggere il contratto YAML corrente e convertirlo su chiavi naturali,
non riutilizzare gli ID numerici Django. Deve essere idempotente e produrre un report di elementi
creati, aggiornati, ignorati o non risolti.
## Comandi osservati in ThothAI
### Backend locale
Eseguiti da `Thoth/ThothAI/backend`:
```sh
uv sync
uv run python manage.py migrate
uv run python manage.py createsuperuser
uv run python manage.py runserver 8200
uv run pytest
```
Import catalogo legacy:
```sh
uv run python manage.py import_db_structure --source local
uv run python manage.py load_defaults --only-level 4 --source local
```
Test mirati rilevanti:
```sh
uv run pytest tests/test_relational_database_operations.py -v
uv run pytest tests/test_ssh_tunnel_configuration.py -v
```
### Stack Docker legacy
ThothAI dichiara `postgres:16-alpine` nel profilo `internal-db`, con volume persistente e
healthcheck `pg_isready`.
```sh
docker compose --profile internal-db up --build
```
Il wrapper legacy abilita lo stesso profilo quando `POSTGRES_INTERNAL=true`:
```sh
POSTGRES_INTERNAL=true ./docker-up.sh
```
Questi comandi documentano il riferimento osservato; non sono comandi di installazione per
ThothII.
## Cosa copiare in ThothII
### Parità necessaria
- gerarchia Workspace Database → Table → Column;
- relazione strutturale fra colonne sorgente e destinazione;
- navigazione e filtri dipendenti workspace/database/tabella;
- test di connessione separato dal salvataggio;
- introspezione esplicita e ripetibile;
- descrizioni generate dall'AI ma modificabili dall'utente;
- validazione cross-entity delle relazioni;
- azioni di import/export senza segreti;
- stato leggibile dei job lunghi;
- PostgreSQL interno persistente con migrazioni esplicite;
- test di CRUD, cardinalità, cascade/restrict, isolamento per workspace e idempotenza.
### Parità semantica con `annotations.yaml`
Il modello futuro deve poter rappresentare almeno:
- descrizione, concetti e note per tabella;
- descrizione, sinonimi, concetti, evidence, note ed `eligible` per colonna;
- foreign key logiche;
- distinzione fra commento fisico osservato e descrizione curata;
- provenienza del contenuto importato o generato.
L'eventuale esclusione di uno di questi campi deve essere una decisione esplicita perché cambia
rendering, retrieval, LSH o SQL generation.
### Vincoli minimi da progettare
- `workspace_id` obbligatorio e unico sul Workspace Database, con esistenza validata contro il
catalogo YAML dal servizio applicativo;
- nome tabella unico nel database e schema appropriato;
- nome colonna unico nella tabella;
- relationship unica secondo il modello, anche per chiavi composite;
- estremi della relationship nello stesso Workspace Database;
- appartenenza certa della colonna alla tabella;
- mutazioni aggregate transazionali;
- gestione esplicita di concorrenza fra CRUD e introspezione.
## Cosa non copiare
- Django, Django Admin, Django ORM, DRF, template admin e frontend Next;
- modello Workspace legacy e condivisione dello stesso DB fra più workspace;
- password o passphrase come normali campi testuali;
- password incluse in CSV o export completi;
- token SSO inseriti nella query string;
- migrazioni generate automaticamente all'avvio;
- validazioni presenti soltanto nel form;
- `pk_field` e `fk_field` testuali come fonte di verità;
- duplicazione di tabella e colonna negli estremi senza constraint coerenti;
- introspezione additiva che non segnala rename, delete o drift;
- azioni admin che possono mostrare successo dopo output AI non valido;
- dipendenza del workflow core dalla disponibilità della UI o del PostgreSQL amministrativo.
## Aspetti di sicurezza da non ereditare
L'export legacy della struttura include username e password in chiaro. Il modello conserva inoltre
password, passphrase SSH e altri segreti in `CharField`; non è stata trovata cifratura applicativa,
nonostante un testo admin affermi il contrario.
ThothII distingue i metadati di connessione dai riferimenti al secret store cifrato. In ogni caso:
- nessun endpoint o export deve restituire segreti;
- log ed errori devono sanificare DSN e credenziali;
- le credenziali di migrazione non devono essere disponibili al runtime CRUD;
- il catalogo non deve riusare credenziali del DWH, delle sessioni o di Qdrant;
- test connessione e introspezione devono usare timeout e privilegi read-only.
La binding REST corrente richiede una verifica prima del cutover: il renderer emette
`ssl_ca_file`, mentre il modello Python espone `ssl_ca`; il percorso della CA privata potrebbe quindi
non essere consumato. PSD richiede TLS con CA privata in locale, perciò questo disallineamento deve
essere corretto e coperto da un test end-to-end prima di affidare il profilo REST al catalogo.
## Percorso incrementale
### Step 1: accesso alla superficie vuota
Implementato in questo worktree:
- pulsante `Database management` nella sidebar destra;
- visibilità legata a `workspace.manage`;
- superficie centrale React separata e vuota;
- nessun router, endpoint, fetch o stato catalogo;
- sessione e SSE conservati in background;
- ritorno al core tramite creazione, apertura o resume di una sessione;
- test frontend dedicati.
Comandi di verifica:
```sh
cd frontend
npx vitest run src/shell/AppShell.database-management.test.tsx
npx vitest run src/shell/AppShell.new-session.test.tsx \
src/shell/AppShell.session-target.test.tsx \
src/shell/AppShell.session-mgmt.test.tsx
npx tsc -b
```
### Step 2: contratto di dominio e schema relazionale
Progettazione della vertical slice completata: Workspace Database, Database Binding, singolo schema,
riferimenti al secret store, optimistic concurrency e capability per trasporto hanno contratti
espliciti. Configurazione e contenuti semantici restano mutabili; la struttura fisica osservata è
sincronizzata e non modificabile manualmente.
### Step 3: PostgreSQL interno e migrazioni
PostgreSQL interno con volume e ruoli runtime/migrator separati. Il modulo catalogo usa Kysely sopra
il driver `pg`; le migrazioni compilate vengono applicate soltanto dal comando `catalog:migrate` e
mai allo startup Fastify. Health, readiness e diagnostica restano dedicate; l'indisponibilità del
catalogo non cambia `core /health` e non interrompe una sessione.
### Step 4: API CRUD
Contratti HTTP, autorizzazione, paginazione, filtri, errori, optimistic concurrency e transazioni.
Gli endpoint dovranno vivere sotto un namespace catalogo e non riutilizzare le route sessione.
### Step 5: UI CRUD
Workspace Database, Catalog Table, Catalog Column e Catalog Relationship sono implementati con
React/Vite e il design system ThothII.
La navigazione è gerarchica e locale al database (`Overview | Tables`), senza menu o filtri globali
per tipo di entità. La grid delle tabelle non offre Add o cancellazione della singola configurazione;
le selezioni espongono invece la pulizia esplicita dei metadati. Il dettaglio full-width mantiene
immutabili i fatti fisici e consente di modificare separatamente Description e Generated
Description. Colonne e relazioni seguono la stessa gerarchia: Columns appartiene al dettaglio
della tabella, Relationships al database. I valori descrittivi null sono mostrati come celle e
campi vuoti, senza fallback visivi o placeholder `Not set` che nascondano quale sorgente è
effettivamente valorizzata.
Le griglie che dispongono di azioni massive usano checkbox e una toolbar contestuale con conteggio,
menu `Actions` e cancellazione della selezione. La selezione identifica ID espliciti, può essere
accumulata attraverso i filtri e viene azzerata dopo successo, nuova sincronizzazione o uscita
dalla pagina; un'azione è all-or-nothing se un elemento non è idoneo. I menu a livello database
espongono gli scope fisici come azioni distinte: `Synchronize tables`, `Synchronize relationships`
e `Synchronize all`. La grid Tables espone invece `Synchronize columns` per le tabelle selezionate;
la pagina Columns non espone sincronizzazione. Le selezioni database aggiungono `Delete all tables` e
`Delete all relationships`; le selezioni tabelle aggiungono `Delete all columns` e `Delete all
relationships`. Queste operazioni sono atomiche, richiedono conferma e non modificano database
esterno, binding, configurazione o segreti. Test connection resta un'azione distinta; griglie senza
azioni non mostrano controlli di selezione inerti.
### Step 6: introspezione
Catalog Table, Catalog Column e Catalog Relationship sono implementate per PostgreSQL diretto,
Thoth REST Connector e tunnel SSH. La scansione read-only è separata dalla transazione; una
riconciliazione atomica crea, aggiorna i commenti sorgente ed elimina, dopo conferma, i fatti fisici
assenti senza rendere modificabile manualmente la struttura osservata. Gli scope autorevoli sono
Tables per database e Physical Relationships per database. Per Columns, `tableIds` vuoto include
tutte le Catalog Table correnti, mentre una lista di ID limita lo scope al sottoinsieme esplicito;
`Synchronize all` osserva tutti e tre gli scope in un unico snapshot e li riconcilia insieme. Tutti
gli scope sono eseguiti come Catalog Sync Run durevoli in background, non attraverso implementazioni
sincrone e asincrone separate. Un run che prevede cancellazioni conserva il diff, attende una
conferma esplicita e verifica nuovamente lo snapshot prima dell'applicazione; se la sorgente è
cambiata, invalida la conferma. Ogni applicazione è atomica e fail-closed: errori, timeout o
capability non disponibili non producono aggiornamenti parziali.
PK e FK devono essere visibili sulle Catalog Column senza duplicare le stringhe denormalizzate di
ThothAI. La posizione nella primary key è un fatto osservato della colonna; membership e conteggio
FK sono proiezioni derivate dalle Catalog Relationship e dalle loro coppie ordinate, aggiornate
nella stessa transazione di riconciliazione.
Ogni scope registra la versione della Database Binding osservata e l'istante dell'ultima
sincronizzazione. Una modifica della binding conserva il catalogo precedente ma lo marca stale;
solo un `Synchronize all` riuscito rende nuovamente corrente l'intero schema.
### Step 7: generazione AI dei metadati
Generated Description è una proposta distinta e modificabile: un revisore può correggerla prima
di consolidarla esplicitamente come Description. Lo slice AI dovrà decidere e implementare anche
alias semantici, descrizioni dei valori, sinonimi e concetti per tabelle e colonne, oltre alla
gestione esplicita di errori e output non validi. La generazione AI e l'azione di consolidamento non
appartengono allo slice di introspezione dello schema.
### Step 8: migrazione PSD
Import idempotente delle annotations PSD, riconciliazione contro la struttura introspezionata,
report degli orfani e confronto semantico con il rendering corrente. Gli altri workspace non
ricevono import legacy.
### Step 9: sostituzione dell'input core
Rimuovere la dipendenza da `annotations.yaml` soltanto dopo avere un contratto equivalente,
test di rendering/search/Qdrant e una policy di disponibilità. Le sessioni di test esistenti
possono essere eliminate, ma le nuove sessioni non devono osservare aggiornamenti parziali.
Questo cutover è esplicitamente rinviato fino al completamento del database dei metadati. Il primo
gate successivo obbligatorio sarà valutare l'integrazione del Catalog Schema Snapshot con il
workflow core e lo schema-linking corrente; il rinvio non autorizza a dimenticare o assorbire
implicitamente il lavoro in altri slice.
### Step 10: operazioni e accettazione
Backup/restore reale, diagnostica, metriche, permessi definitivi, hardening degli export e
test di failure isolation fra catalogo e workflow. I Catalog Sync Run hanno un solo job attivo per
Workspace Database, sono concorrenti fra database diversi e usano un lock persistente. Un pannello
operativo non modale rimane visibile durante la navigazione del database, mostra fasi, contatori,
tempo trascorso e log sanitizzato via SSE con polling di fallback, e offre Confirm, Cancel e Retry
quando consentiti. Un restart marca `interrupted` i run rimasti attivi; il retry crea un nuovo run.
Le modifiche ai metadati restano consentite durante la scansione e sono preservate dall'applicazione.
Il worker gira inizialmente nello stesso servizio Fastify ma dietro un'interfaccia estraibile, con
coda, lease e heartbeat persistiti nel catalog-db. I riepiloghi dei run non scadono; gli eventi
dettagliati sono conservati per 30 giorni, mentre snapshot e diff completi vengono eliminati dopo
la conclusione lasciando conteggi, decisioni e una sintesi sanitizzata dell'esito.
## Verifiche del core da conservare per il cutover
Comandi attuali rilevanti:
```sh
tht --installation <absolute>/thothii-installation.yaml workspace preprocess dwh \
--workspace <id> --json
tht --installation <absolute>/thothii-installation.yaml workspace schema suggest-fks \
--workspace <id> --json
tht --installation <absolute>/thothii-installation.yaml workspace schema check \
--workspace <id> --json
tht --installation <absolute>/thothii-installation.yaml workspace schema accept \
--workspace <id> --run <run-id> --yes --json
tht --installation <absolute>/thothii-installation.yaml workspace index-schema \
--workspace <id> --json
```
Suite che documentano il comportamento da preservare:
```sh
cd backend
npx vitest run test/workspaces-git-annotations.test.ts \
test/registry-annotations.test.ts \
test/annotations-sync.test.ts \
test/workspace-runtime-config-lease.test.ts \
test/workspace-preprocessing-service.test.ts
npx tsc --noEmit -p .
cd ../harness
.venv/bin/pytest -q \
tests/test_annotations_root.py \
tests/test_schema_fk_annotations.py \
tests/test_mschema_render.py \
tests/test_qdrant_cli_commands.py
```
Questi test non implicano che la futura implementazione debba continuare a usare file YAML.
Definiscono gli effetti semantici e le guardie da mantenere o sostituire consapevolmente.
## Decisioni rinviate
Le seguenti scelte non appartengono allo step 1:
- lifecycle dei riferimenti ai segreti durante sostituzione e cancellazione;
- criteri per aggiungere dialetti successivi a PostgreSQL;
- criteri per un'eventuale estensione futura a più schemi per database;
- lifecycle e gestione amministrativa delle future Logical Relationship;
- alias semantici, descrizioni dei valori, sinonimi e concetti prodotti o assistiti dall'AI;
- formato e momento del cutover dal file al database interno;
- permission definitiva separata da `workspace.manage`.
Ognuna sarà affrontata nel relativo step, senza anticipare scelte tecnologiche nel presente
documento.
@@ -1,251 +0,0 @@
# AI-generated descriptions for Catalog Tables and Catalog Columns
## Problem Statement
ThothII already stores a Generated Description separately from the curated Description for Catalog
Tables and Catalog Columns, but administrators cannot populate it with AI. ThothAI provides the
useful core workflow—generate table and column comments from schema context and small real-data
samples—but its execution, configuration, and interaction model cannot be copied directly into
ThothII.
Administrators need an asynchronous workflow integrated into Database Management. They must be
able to choose an installation-approved model, generate descriptions for selected or missing
targets, observe understandable progress, stop or recover a stuck operation, review generated
text, and explicitly consolidate it. The solution must retain ThothAI's practical simplicity and
must not introduce a general job platform, model gateway, distributed scheduler, or competing
user-facing CLI.
## Solution
Add Description Generation to Database Management as one installation-wide, sequential background
run owned by the Fastify backend. The browser starts a run and remains responsive while the backend
processes bounded requests one at a time. Each completion is delegated to a short-lived internal
Python helper using LiteLLM. Models, their default, and any API-key secret references are declared in
application setup YAML and are independent of both workspaces and Pi configuration.
Each valid result is written immediately to the target's Generated Description. A minimal run row
and ordered text events provide status, counters, history, and a live log. A stopped or crashed run
is not resumed automatically; completed results remain in place and Generate Missing supplies the
simple recovery path. An Unlock action marks a stale recorded run interrupted only when no helper
or backend generation loop is alive.
Prompts use catalog context and, when available, no more than five real rows and five representative
non-null examples. Samples are transient and never logged or persisted. A valid inability to infer
a description produces a standard application-localized value such as `Non generabile`; provider,
timeout, and response-validation failures remain technical errors.
Generated text remains separate from Description until an administrator uses the existing
checkbox selection and Actions control to consolidate it. Consolidation retains Generated
Description and never writes comments to the external Workspace Database.
## User Stories
1. As an installation operator, I want to declare the models allowed for metadata generation in setup YAML, so that model availability is controlled centrally.
2. As an installation operator, I want to declare one default metadata-generation model, so that administrators begin with a safe operational choice.
3. As an installation operator, I want each model to reference its own API-key secret, so that credentials are not stored in workspaces or browser-visible settings.
4. As an installation operator, I want metadata-generation models to remain independent of Pi models, so that changing this workflow cannot disrupt the core NL-to-SQL experience.
5. As an installation operator, I want invalid model setup to fail validation clearly, so that the application does not start with ambiguous provider behavior.
6. As a Catalog Administrator, I want generation controls to explain when no model is configured, so that I know why the action is unavailable.
7. As a Catalog Administrator, I want to select an approved model from a selector initialized to the setup default, so that I control which model performs the work.
8. As a Catalog Administrator, I want to generate descriptions for selected Catalog Tables, so that I can work on a focused part of the catalog.
9. As a Catalog Administrator, I want to generate descriptions for selected Catalog Columns, so that I can work on individual fields without regenerating a whole table.
10. As a Catalog Administrator, I want to generate all eligible descriptions for a Workspace Database, so that I can initialize a catalog in one operation.
11. As a Catalog Administrator, I want to generate only missing descriptions, so that I can continue interrupted work without replacing completed proposals.
12. As a Catalog Administrator, I want a full run to process Catalog Columns before their Catalog Tables, so that table descriptions can benefit from column descriptions.
13. As a Catalog Administrator, I want generation to run asynchronously after I start it, so that the browser remains usable and progress is not tied to one HTTP request.
14. As a Catalog Administrator, I want only one Description Generation Run active in the installation, so that provider traffic and operational behavior remain predictable.
15. As a Catalog Administrator, I want a second start attempt to return a clear conflict, so that I cannot accidentally overlap generation runs.
16. As a Catalog Administrator, I want synchronization, cleanup, consolidation, and edits for the target Workspace Database blocked during generation, so that the simple sequential run sees stable catalog state.
17. As a Catalog Administrator, I want to see the run's model, scope, status, counters, and timestamps, so that I understand what is happening.
18. As a Catalog Administrator, I want a chronological text log, so that I can follow completed targets and diagnose errors.
19. As a Catalog Administrator, I want live log updates with a polling fallback, so that temporary SSE problems do not hide run progress.
20. As a Catalog Administrator, I want completed and interrupted runs to remain inspectable, so that I can understand prior activity.
21. As a Catalog Administrator, I want to stop an active run, so that I can halt an incorrect or unexpectedly costly operation.
22. As a Catalog Administrator, I want stopping a run to terminate its current model helper and prevent later targets from starting, so that stop has prompt operational effect.
23. As a Catalog Administrator, I want valid results completed before a stop or failure to remain saved, so that useful work is not discarded.
24. As a Catalog Administrator, I want a run left active by a backend restart to become interrupted, so that the UI does not claim nonexistent work is still running.
25. As a Catalog Administrator, I want to unlock a stale active run when no generation process is alive, so that an erroneous recorded lock cannot block future work.
26. As a Catalog Administrator, I want Unlock rejected while a live generation process exists, so that recovery cannot create an overlapping run.
27. As a Catalog Administrator, I want Generate Missing to continue after interruption, so that recovery does not require a special resume mechanism.
28. As a Catalog Administrator, I want one retry for a transient model failure, so that a brief provider fault does not immediately lose a batch.
29. As a Catalog Administrator, I want the run to fail after three consecutive technical failures, so that a broken provider does not generate an unbounded stream of attempts.
30. As a Catalog Administrator, I want a successful request to reset the consecutive-failure count, so that isolated errors do not prematurely stop a useful run.
31. As a Catalog Administrator, I want a completed-with-errors result when isolated batches fail but the run reaches its end, so that partial problems remain visible.
32. As a Catalog Administrator, I want no automatic fallback to a different model, so that the selected model remains truthful and predictable.
33. As a Catalog Administrator, I want malformed or ambiguous model output rejected without writing it, so that descriptions cannot be assigned to the wrong target.
34. As a Catalog Administrator, I want an inability to infer a description represented by standard localized text, so that every valid outcome is understandable in the workspace language.
35. As a Catalog Administrator, I want technical failures kept distinct from non-generatable outcomes, so that provider problems are not mistaken for catalog knowledge.
36. As a Catalog Administrator, I want generated prose written in the workspace language, so that it matches the catalog's intended audience.
37. As a Catalog Administrator, I want generated text stored separately from curated Description, so that AI output remains a reviewable proposal.
38. As a Catalog Administrator, I want to edit a Generated Description manually, so that I can improve a proposal before consolidation.
39. As a Catalog Administrator, I want to select one or more tables or columns and run “Move generated description to Description” from the existing Actions control, so that review remains integrated into the current grids.
40. As a Catalog Administrator, I want consolidation to retain the Generated Description, so that I can still see the proposal from which the curated text was copied.
41. As a Catalog Administrator, I want selected records without a Generated Description skipped and reported, so that the bulk action does not erase curated text.
42. As a Catalog Administrator, I want consolidation and generation to modify only the Metadata Catalog, so that no external database comment is changed.
43. As a Catalog Administrator, I want prompts to use schema facts and existing catalog text, so that generated descriptions are grounded in available metadata.
44. As a Catalog Administrator, I want prompts to use at most five real source rows and five representative values when available, so that the model has useful examples without unbounded disclosure.
45. As a Catalog Administrator, I want to be warned that real source samples are sent to the selected provider, so that I can make an informed disclosure decision.
46. As a Catalog Administrator, I want sampled rows and values excluded from persistence and logs, so that operational history does not become a secondary data store.
47. As a security operator, I want API keys, prompts, samples, and complete provider payloads redacted from logs, so that diagnostics do not leak secrets or source data.
48. As a support operator, I want concise per-target and per-batch event messages, so that failures can be diagnosed without provider-specific internals.
49. As an authorized administrator, I want all generation, cancellation, unlock, and consolidation actions protected by database-management permission, so that ordinary users cannot mutate catalog metadata.
50. As an unauthorized user, I want generation controls hidden or disabled and API calls rejected, so that frontend visibility is not treated as authorization.
51. As an operator, I want setup changes to take effect after an application restart, so that configuration lifecycle remains simple and explicit.
52. As a product owner, I want the first release to avoid queues, parallel calls, distributed locks, and automatic resume, so that effort remains focused on generating and reviewing useful descriptions.
## Implementation Decisions
- The Fastify backend owns one installation-wide Description Generation Run and its sequential
processing loop. It does not delegate lifecycle ownership to Pi or Python.
- A Description Generation Run has one of `queued`, `running`, `completed`,
`completed_with_errors`, `cancelled`, `failed`, or `interrupted`. It stores the Workspace
Database, requested scope, selected model identifier, workspace language, progress counters,
timestamps, and an optional final error summary.
- Ordered Description Generation Events store timestamp, severity, and safe human-readable text.
When an event identifies a target, it uses the object type and qualified physical name, such as
`Column "patients.birth_date"` or `Table "patients"`; catalog UUIDs remain internal identifiers.
No durable per-target jobs, model invocation rows, prompt snapshots, sample snapshots, leases,
heartbeats, registry revisions, or provenance chains are introduced.
- Starting a run schedules an in-process background loop and returns the run immediately. The API
exposes start, run/history lookup, event listing and streaming, cancellation, stale-run unlock,
and the safe list of configured model choices. There are no retry-item or resume endpoints.
- One in-memory generation manager enforces the installation-wide active-run rule. The existing
Catalog Operation Coordinator reserves the target Workspace Database for the duration of the
run, without being generalized into a new operation framework.
- On backend startup, persisted `queued` or `running` Description Generation Runs become
`interrupted`. The application performs no automatic replay or resume.
- Unlock succeeds only when no live generation loop or helper child exists. It marks the stale run
interrupted and releases the local reservation; it is not a distributed lock recovery protocol.
- Each model completion uses a short-lived Python helper backed by LiteLLM. Structured input is
supplied over stdin, structured output alone is emitted on stdout, diagnostics use stderr, and
the helper can be terminated by cancellation.
- A completion request contains no more than ten targets. Requests run one at a time. The helper
performs at most one retry for a transient technical failure.
- Three consecutive model-request failures fail the run. A successful request resets that count.
Isolated exhausted failures may be logged and skipped, producing `completed_with_errors` if the
run later reaches its end.
- Every valid generated or non-generatable result is applied immediately to Generated Description.
Earlier writes are retained after cancellation, interruption, or later failure.
- A response must identify requested targets unambiguously and classify each returned result as
generated or non-generatable. Duplicate, unknown, missing, or malformed mappings cause a
technical request failure and no result from that ambiguous response is applied.
- The parser also tolerates one JSON object enclosed by one complete `json` code fence, because
some supported models add that formatting despite the prompt. Any prose outside the fence,
multiple payloads, or malformed/ambiguous mappings remain invalid.
- The application supplies localized standard non-generatable text. Provider wording is not used
as the standard value, and technical errors never write that value.
- A full-database run generates eligible Catalog Columns before Catalog Tables. Generate Missing
excludes targets whose Generated Description is already non-empty; all-generation may replace
existing generated proposals only after the initiating action makes that scope explicit.
- Model choices are declared under a metadata-generation section in installation setup YAML. Each
choice has a stable identifier, display label, LiteLLM provider/model settings, optional endpoint
settings, and an optional environment-secret reference for its API key. The reference may be
omitted only when an explicit endpoint is configured for unauthenticated access. One identifier
is the default.
- An explicit endpoint may set `disableThinking: true`; the helper translates it only to the
Qwen-compatible chat-template switch needed to keep the response within the strict JSON contract.
- Metadata-generation setup is separate from application settings for Pi and from workspace
`llm_policy`. Raw keys never enter setup YAML, the catalog database, API responses, process
arguments, or event text. Configuration reload is restart-only.
- If setup defines no usable model, the safe model-list response is empty and the UI disables
generation with an explanation. The backend still rejects direct generation attempts.
- Prompt construction treats schema names, comments, descriptions, and values as untrusted data.
It requests output in the workspace language and separates instructions from catalog content.
- A request may contain up to five real source rows and up to five representative distinct,
non-null values for relevant columns. Inputs are bounded before prompt construction and are not
persisted or logged.
- The UI discloses that real data can be sent to the selected provider. A future Sensitive Data
Policy will classify values and exclude or anonymize protected data; that policy is not silently
approximated in this slice.
- The generation UI reuses Database Management's table and column selections, model selector,
Actions control, run drawer conventions, SSE delivery, and polling fallback where practical.
Visual parity with Catalog Sync Run logs is not required.
- The consolidation action copies each selected, non-empty Generated Description into Description
in a catalog transaction, retains Generated Description, skips empty proposals, and reports
copied and skipped counts. It never writes to the external Workspace Database.
- Generation, cancellation, unlock, and consolidation require the existing database-management
permission and are validated by the backend independently of UI state.
- No user-facing generation CLI is added. The Python process is an internal completion adapter,
not an operator surface or a long-lived service.
## Testing Decisions
- Tests assert externally observable behavior rather than private loop structure, process timing,
or LiteLLM implementation details.
- The primary and highest test seam is the Fastify catalog API with a test PostgreSQL catalog and
an injected fake Model Completer. It verifies complete paths through authorization, run
persistence, sequential processing, event delivery, Generated Description updates, and final
status without contacting a real provider.
- API tests cover each generation scope, column-before-table order, the ten-target request bound,
model validation, one-active-run conflict, target-database exclusion, cancellation, startup
interruption, Unlock safeguards, Generate Missing, partial success, consecutive failure
handling, non-generatable localization, malformed responses, redacted events, and permissions.
- Catalog repository integration tests verify the migration, run and event ordering, active-run
constraint, immediate description writes, history queries, startup interruption, and bulk
consolidation behavior against PostgreSQL.
- The Python helper has a small black-box contract suite using a simulated LiteLLM adapter. It
verifies stdin/stdout framing, pristine stdout, stderr diagnostics, normalized success and
failure output, one transient retry, secret redaction, and termination behavior.
- Setup-validation tests cover duplicate model identifiers, missing or unknown defaults, malformed
provider settings, missing secret references, safe public model projection, and strict separation
from Pi and workspace model settings.
- Database Management tests use the existing browser-level component seam with MSW. They verify
model selection and default, selected/all/missing actions, disabled state without models, running
progress and logs, polling recovery, cancellation, Unlock visibility, terminal summaries,
generated-text refresh, and selected consolidation with copied/skipped counts.
- Existing Catalog Sync Run route, repository, SSE, and drawer tests are prior art for asynchronous
status and event behavior. Existing catalog table/column editing and Database Management tests
are prior art for optimistic catalog updates, permissions, selection, and action controls.
- One required manual acceptance gate, outside deterministic CI, uses the installation's configured
default model and a disposable PostgreSQL database containing only invented data. Its application
credentials are read-only. It generates Italian text for one Catalog Column and one Catalog
Table, verifies their Generated Description, inspects the safe activity log, confirms that no key
or sample value is exposed, and consolidates one selected result. If the configured secret is not
available, acceptance stops without exposing or requesting the key in conversation.
- Delivery includes a strict MkDocs build executed through repository-managed, reproducible
documentation dependencies rather than globally installed Python packages. A readable direct
dependency file is retained, a complete transitive lock is generated with `uv`, and one canonical
repository command performs the strict build from that lock.
- Successful real-provider acceptance is recorded in a short sanitized report under
`docs/testing/`. It identifies the model and checks performed but contains no credentials,
prompts, source samples, complete provider payloads, or generated database values.
- No tests are added for worker queues, parallel generation, distributed locking, multi-replica
recovery, automatic resume, cost accounting, or model fallback because those behaviors are out
of scope.
## Out of Scope
- Reusing Pi to execute Description Generation or changing Pi's model configuration.
- A shared Installation Model Registry, model gateway, long-lived Python sidecar, or provider
management platform.
- A user-facing generation CLI.
- Parallel model calls, worker queues, adaptive rate limiting, distributed locks, leases,
heartbeats, automatic resume, or multi-replica execution.
- Durable target jobs, invocation history, prompts, samples, token usage, cost accounting,
provenance chains, target snapshots, or advanced retention controls.
- Automatic retry or resume of individual targets beyond one technical helper retry and a new
Generate Missing run.
- Automatic fallback to a different model.
- Writing generated text into comments of the external Workspace Database.
- Generating logical relationships or other catalog metadata beyond Catalog Table and Catalog
Column descriptions.
- Implementing the Sensitive Data Policy. Its definition and exclusion/anonymization behavior are
a required follow-up improvement.
- Generalizing the log viewer across unrelated metadata operations. That broader concern remains
related to Gitea issue #2.
## Further Notes
- The design deliberately follows ThothAI's proven simple workflow while adapting it to ThothII's
asynchronous browser interaction, setup ownership, and existing Generated Description model.
- The source-sampling disclosure is a release requirement, not merely documentation for operators.
- `completed_with_errors` is reserved for a run that reaches the end after isolated technical
failures. Three consecutive failures end the run as `failed`.
- Successful values are their own recovery record: after interruption, Generate Missing naturally
skips them without needing replay state.
- The implementation is available without a feature flag once the catalog migration and valid
setup are present. With no configured model, the feature remains visibly unavailable rather than
partially initialized.
- The final manual gate is intentionally narrow: one real-provider run covers one Catalog Column
and one Catalog Table, generated Italian text, safe events, and one consolidation. Automated
tests remain the evidence for All, Missing, Stop, restart interruption, Unlock, and failure paths.
@@ -1,173 +0,0 @@
# AI catalog description generation
Status: simplified design, API, persistence, test seams, and delivery tickets accepted.
## Objective
Bring ThothAI's useful AI comment-generation workflow into the ThothII Metadata Catalog without
turning it into a general job platform. Administrators can generate editable descriptions for
catalog tables and columns, inspect progress, stop a run, recover a stale run, and explicitly copy
approved generated text into the curated Description field.
The implementation is UI/API only. There is no user-facing generation command.
## ThothAI behavior retained
- Generate descriptions for selected tables, selected columns, missing descriptions, or all
eligible targets.
- Generate columns before their containing table when running the full workflow, so table prompts
can benefit from the resulting column descriptions.
- Process bounded batches of at most ten targets, one model request at a time.
- Include schema context, existing catalog text, up to five real source rows, and up to five
representative non-null values when available.
- Keep generated text separate from the curated Description until an administrator consolidates
it.
- Use the existing table and column checkboxes plus the Actions selector to copy Generated
Description into Description for one or more selected records. The generated value is retained.
- Store a localized standard value such as `Non generabile` when a valid model response says that
a description cannot be inferred.
Unlike ThothAI, every generation action is asynchronous from the browser's perspective and exposes
a persistent, readable activity log.
## Minimal architecture
The Fastify backend owns the run lifecycle and sequential loop. It starts one short-lived Python
helper for each model completion. The helper uses LiteLLM, accepts structured input on stdin,
returns structured output on stdout, and writes diagnostics only to stderr.
This is preferred over reusing Pi. Pi remains the interactive NL-to-SQL orchestration surface,
whereas description generation is a bounded batch transformation with no conversational state or
human gate. A LiteLLM helper avoids inventing a Pi session protocol for a task that needs one
request and one structured response.
There is no Python daemon, model gateway, queue service, worker pool, or generation CLI. Python is
already a core implementation language in ThothII's harness and core image; this helper does not
introduce a new runtime family.
## Run lifecycle and exclusion
- At most one Description Generation Run may be queued or running in the installation.
- Start returns immediately after creating the run and scheduling the in-process backend loop.
- Requests are sequential; there is no parallel provider traffic.
- The target Workspace Database is reserved through the existing in-memory catalog-operation
coordinator. Synchronization, cleanup, consolidation, and direct catalog edits for that database
are rejected while generation is active.
- A second generation start is rejected with a conflict response.
- Stop terminates the current helper process, stops further targets, and marks the run cancelled.
- Backend startup marks any queued or running generation row interrupted. It does not resume work.
- Generate Missing is the normal manual continuation mechanism because successful values were
already saved.
- Unlock is available only when the backend has no live generation process; it marks a stale
recorded run interrupted and clears the local reservation.
This is intentionally a single-process policy. Multi-replica coordination is out of scope.
## Persistence
Persist only:
- a Description Generation Run with database, scope, selected model, language, status, counters,
timestamps, and an optional final error summary;
- ordered Description Generation Events containing timestamp, level, and human-readable text;
- each successful or non-generatable result directly in the target's Generated Description.
Do not add per-target job rows, invocation history, prompt or sample snapshots, provider cost
accounting, leases, heartbeats, registry revisions, or generated-description provenance. The event
log is operational evidence, not a replay mechanism.
## Model setup
Selectable models and their default belong to application setup YAML, not to a workspace. Each
entry supplies a stable display identifier, LiteLLM provider/model information, optional endpoint
settings, and—unless that explicit endpoint is unauthenticated—a reference to an installation
secret containing the API key. Keyless entries without an explicit endpoint are invalid. Raw keys must not be
stored in the YAML, database, frontend, events, or process arguments.
An explicit endpoint may opt into `disableThinking: true` when its Qwen-compatible chat template
would otherwise place reasoning text around the required JSON result.
This metadata-generation configuration is independent of the existing Pi provider/model settings
and workspace `llm_policy`. A setup change takes effect after application restart. If no model is
configured, generation controls are disabled with an explanatory message.
The browser receives only the selectable identifiers and labels. The selected value defaults to
the setup default and is validated again by the backend when a run starts.
## Prompt inputs and outputs
Targets are grouped in model requests of at most ten. Prompts distinguish instructions from
untrusted schema names, comments, descriptions, and sampled values. A response must map every
returned result to a requested target and classify it as generated or non-generatable. Missing,
duplicate, unknown, or malformed target results make that request a technical failure rather than
silently writing ambiguous text.
One complete `json` code fence around the object is tolerated for model compatibility; prose
outside it, multiple payloads, and ambiguous mappings are still rejected.
For a complete database run, eligible columns are processed before tables. A table request can use
the current Generated Description or Description of its columns. The output language is the
workspace language; the standard non-generatable text is localized by the application rather than
trusted to arbitrary model wording.
Up to five source rows and five representative examples may be sent to the provider and are never
persisted. Delivery must call out this disclosure. A follow-up Sensitive Data Policy will define
which values are excluded or anonymized.
## Errors, retry, and logs
The helper performs at most one retry for a transient technical provider failure. A final failed
request produces an error event and increments the consecutive-error count. The run stops as
failed after three consecutive technical failures; any successful request resets the count. There
is no automatic fallback to another model.
Valid non-generatable outcomes are results, not technical errors. Successful results from earlier
requests remain stored when a later request fails or the run is stopped.
The UI shows status, counters, selected model, start/end times, and a chronological text log. Live
delivery may reuse the existing SSE infrastructure with polling as fallback; exact visual parity
with synchronization logs is not required. Logs must not contain API keys, prompts, source sample
values, or full provider payloads.
## Explicitly deferred complexity
- shared model registry or cutover of Pi configuration;
- long-lived Python sidecar or internal HTTP model gateway;
- generic catalog-operation kernel;
- durable target items, invocation records, target snapshots, or provenance chains;
- distributed locks, leases, heartbeats, worker queues, automatic resume, or multi-replica support;
- parallel calls, adaptive rate limiting, cost estimation, advanced metrics, or model fallback;
- user-facing generation CLI;
- automatic writeback to comments in the external database;
- Sensitive Data Policy implementation, which remains a required improvement after this slice.
## Delivery tracking
The accepted specification is Gitea issue #4 and the implementation is split into issues #5–#11.
Each ticket is a bounded vertical slice with explicit Gitea dependencies. Implementation proceeds
from the unblocked frontier, using a fresh subagent context for each ticket; integration and final
verification remain centralized so later slices cannot silently reopen the deferred platform
features above.
Issue #4 remains open until delivery completes four final gates: the stale Compose service-set
contract is corrected in its own commit; documentation dependencies are repository-managed and a
strict MkDocs build passes; one narrow real-provider acceptance run succeeds against non-sensitive
test data; and a separate, non-blocking Sensitive Data Policy design ticket is linked as required
follow-up work.
The documentation toolchain retains a readable direct-dependency input, adds a complete lock
generated with `uv`, and exposes one canonical strict-build command. Real-provider acceptance uses
the installation's configured default model and a disposable PostgreSQL database seeded only with
invented values and accessed read-only by the application. A missing protected model secret stops
the gate without disclosing it. The successful gate is captured in a sanitized report under
`docs/testing/` without prompts, samples, full generated values, payloads, or credentials.
Delivery is organized as four reviewable commits: the stale Compose contract correction, the
reproducible documentation toolchain, the AI-description feature, and—only after acceptance—the
sanitized acceptance report. A failed real-provider gate does not invalidate already verified
commits, but issue #4 remains open and no acceptance report claims success. Application defects are
fixed and reverified; missing configuration or provider unavailability is recorded and retried.
After every gate passes, the existing `codex/db-management` branch is pushed to its configured
origin without introducing a new pull-request workflow, then issue #4 is closed with links to the
delivery evidence. The separate Sensitive Data Policy issue is created as non-blocking follow-up,
linked to #4, and labeled `enhancement` plus `ready-for-human` because its design requires a future
`grill-with-docs` before agent implementation.
@@ -1,248 +0,0 @@
# Installation Model Catalog
Status: implemented on 2026-09-02.
## Outcome
`thothii-installation.yaml` is the only operator-authored source for models used by interactive
sessions, metadata generation, and embedding. Runtime-specific files are deterministic projections,
not additional configuration sources. Workspace descriptors contain database and Evidence concerns
and no model, provider, allowlist, default, embedding, or vector-store configuration.
This design does not merge execution lifecycles. Pi continues to run interactive sessions, the
short-lived LiteLLM helper continues to perform metadata generation, and the internal Ollama service
continues to provide embeddings. They share model declaration, not execution machinery.
## Canonical installation shape
The following example covers all currently required cases: a Pi built-in model, an authenticated
custom endpoint, a keyless internal endpoint, metadata generation, and the single embedding model.
```yaml
schemaVersion: 2
profile: server
projectDirectory: /srv/thothii
envFile: /srv/thothii/operator.env
workspaceRepository:
remote: git@git.example.com:organization/workspaces.git
branch: main
access: ssh
modelCatalog:
defaults:
session: zai/glm-5.3
metadataGeneration: local-qwen/qwen3.6-35b-a3b
embedding:
id: ollama/qwen3-embedding:0.6b
dimensions: 1024
providers:
deepseek:
authentication:
mode: pi_auth
session:
mode: pi_builtin
models:
deepseek-v4-pro:
session: {}
deepseek-v4-flash:
session: {}
zai:
endpoint:
baseUrl: https://api.z.ai/api/coding/paas/v4
authentication:
mode: secret_env
apiKeyEnv: ZAI_API_KEY
session:
mode: openai_compatible
metadataGeneration:
litellmProvider: openai
models:
glm-5.3:
label: GLM-5.3
session:
reasoning: true
contextWindow: 200000
maxTokens: 131072
metadataGeneration: {}
local-qwen:
endpoint:
baseUrl: https://ml-aritmolab.policlinicosandonato.it/v1
authentication:
mode: none
session:
mode: openai_compatible
metadataGeneration:
litellmProvider: openai
models:
qwen3.6-35b-a3b:
label: Qwen3.6 35B A3B
session:
reasoning: false
contextWindow: 131072
maxTokens: 16384
compatibility:
supportsDeveloperRole: false
supportsReasoningEffort: false
supportsStore: false
maxTokensField: max_tokens
metadataGeneration:
disableThinking: true
authentication:
configDirectory: /srv/thothii/auth-canonical
runtimeProjection:
directory: /srv/thothii/auth-runtime
uid: 10001
gid: 10001
```
The catalog uses maps instead of repeated IDs. The canonical identity of a model is always derived
as `<provider-key>/<model-key>`. `upstreamModel` may be added to a model only when the endpoint uses
a different identifier. `label` is optional and falls back to the canonical identity.
Model eligibility is not repeated in an `usages` array. A `session` block makes the model eligible
for sessions; a `metadataGeneration` block makes it eligible for metadata generation. The embedding
is a single required installation value rather than a list plus default.
## Provider and authentication rules
A provider owns one endpoint, one authentication mode, and zero or one adapter for each runtime.
Model entries cannot override provider endpoint or credentials. If the same upstream service needs
different endpoints or credentials, the installation declares two provider identities.
Supported session modes are intentionally closed:
- `pi_builtin`: Pi already owns the model's technical descriptor; the model's `session` block is
empty and ThothII does not copy context-window or compatibility facts.
- `openai_compatible`: ThothII generates a Pi custom-provider descriptor; each session model supplies
the technical values required by Pi.
Metadata generation uses the provider-level `litellmProvider`. A model-level
`metadataGeneration.disableThinking: true` is permitted only for an explicit compatible endpoint.
There is no generic adapter or plugin abstraction in schema version 2.
Exactly one provider authentication mode is allowed:
- `secret_env` requires an approved API-key environment reference present in the protected secret
bundle. Secret values never enter YAML, generated files, logs, arguments, or API responses.
- `pi_auth` is valid only for session-only `pi_builtin` providers and resolves through Pi's protected
authentication projection.
- `none` is valid only for an explicit endpoint. Runtime projections may supply a fixed non-secret
compatibility placeholder when a client library requires a non-empty key.
## Defaults and selections
`defaults.session` and `embedding` are required. `defaults.metadataGeneration` is required exactly
when at least one model has a `metadataGeneration` block; metadata generation may otherwise be
absent and its UI controls are disabled.
`modelCatalog.defaults.session` is the only configured session-model default. `PI_PROVIDER`,
`PI_MODEL`, and provider/model fields in installation-default settings are removed. A user choice is
a Model Selection containing only the canonical model identity and runtime controls such as thinking
level. A session manifest pins the selected canonical identity.
Removing the currently selected model causes new-session selection to fall back to the catalog
default with an explicit administrative warning. An existing session is never silently moved to a
different model; resume fails with `model_unavailable` when its pinned identity can no longer be
resolved.
## Generated runtime projections
Before Compose starts, `tht` strictly validates schema version 2 and generates installation-local
artifacts below `deploy/<installation-id>/generated/`:
- a normalized catalog JSON consumed defensively by the backend;
- Pi `models.json` for custom providers;
- Pi `settings.json`, combining fixed product settings with the session-eligible canonical IDs;
- a Compose override that mounts the projections and supplies embedding identity and dimensions to
core, preprocessing, and `embedding-model-init`.
Generation is deterministic and published only after every candidate artifact validates. A failed
generation aborts start before Compose is invoked. `tht doctor` recomputes expected bytes and reports
differences; no digest manifest or separate apply command exists. When projection bytes change,
`tht start` recreates the affected services so they cannot continue with an older bind mount.
Pi-only restart, update, and rollback operations reject projection drift and direct the operator to
`tht start`, because applying only the core-facing files could leave embedding services stale.
Generated projections are not backed up. Restore validates the canonical installation descriptor,
regenerates every projection, and only then starts services. Base Compose files and `operator.env`
must contain no model identities, defaults, endpoints, or dimensions.
## Workspace schema v4
Workspace schema v4 removes both top-level `llm_policy` and `semantic_index`. The entire latter
block is redundant today: its engine and distance are product constants, its collection duplicates
the workspace ID, and its model and dimensions are installation facts.
The runtime derives:
- Qdrant collection identity from the workspace ID;
- engine and distance from the supported product contract;
- embedding identity and dimensions from the Installation Model Catalog.
The published index generation records the canonical embedding identity and dimensions that created
it. A mismatch makes the index explicitly incompatible and requires operator-triggered
preprocessing. No existing index is deleted or rebuilt automatically.
The v3-to-v4 workspace migration is deterministic: set `workspace.schema_version` to `4`, remove
`llm_policy`, and remove `semantic_index`. It does not alter database, Evidence, diagnostics, or
binding data.
## Installation migration
Legacy installation migration must inspect all three former sources:
1. `metadataGeneration` in `thothii-installation.yaml`;
2. `deploy/pi/models.json`;
3. `deploy/pi/settings.json`.
The migrator emits a version-2 candidate only when it can reconcile identities, endpoints,
credentials, and runtime-specific facts without guessing. Ambiguous aliases such as `glm-53`,
`zai/glm-5.3`, and `openai/glm-5.3` are not silently equated. A conflict produces a field-level
report and leaves every input unchanged for operator resolution.
After migration, the strict loader rejects `metadataGeneration`, workspace `llm_policy`, workspace
`semantic_index`, legacy Pi source files, unknown fields, duplicate YAML keys, invalid defaults, and
incompatible authentication/adapter combinations with an actionable `migration_required` or
validation error.
## Final simplicity audit
The accepted design removes every configuration duplication that can be removed without inference:
- one authored installation file instead of an installation block plus two Pi files;
- one canonical `provider/model` identity instead of display IDs and runtime IDs;
- per-use blocks instead of a duplicated usages list;
- one embedding entry instead of a selectable embedding catalog;
- one catalog session default instead of environment and settings defaults;
- no model or vector-store fields in workspace descriptors;
- provider-level credentials instead of per-model credentials;
- no generic runtime-plugin abstraction;
- no persisted digest, apply command, or backup of generated projections.
The remaining generated files are necessary boundary adapters, not configuration concepts. Making
the backend parse the authoring YAML independently would remove one file but restore two semantic
validators. Hard-coding embedding values in Compose would remove one projection but restore a model
source outside the catalog. Inferring authentication from missing fields would save one YAML key but
turn a safe explicit choice into ambiguity. These apparent simplifications are therefore rejected.
No further reduction was found that preserves one authority, strict validation, explicit security,
session determinism, and model-free workspaces.
## Implementation surface
Implementation must update the host `tht` installation loader, setup and lifecycle projection,
doctor, backup/restore, Compose mounts and embedding inputs, backend catalog/settings/session model
resolution, workspace schema and migration, runtime rendering and diagnostics, frontend workspace
drafts and model filtering, examples, fixtures, and documentation. Existing session manifests remain
readable and keep their pinned provider/model identity; only resume resolution changes to the new
catalog.
Implementation completed after explicit approval. The installation schema, deterministic runtime
projections, migration path, model-free workspace schema v4, backend consumers, operator UI,
fixtures, and documentation now enforce this contract.
+3 -3
View File
@@ -5,9 +5,9 @@ manuale con commit/push dell'operatore accettati per la release 0;
restanti semplificazioni confermate, con chiarimenti su impatto core e controllo Git; restanti semplificazioni confermate, con chiarimenti su impatto core e controllo Git;
E1, E2, E3 e il collegamento ai gate X1 implementati il 2026-09-09. E1, E2, E3 e il collegamento ai gate X1 implementati il 2026-09-09.
Il contratto X1 è in [Session corrections](../contracts/archive-repair.md). Risultati e limiti in Il contratto X1 è in [Session corrections](../contracts/archive-repair.md). Risultati e limiti in
[E1 — validazione](2026-09-09-evidence-e1-validation.md) e [E1 — validazione](../reports/knowledge-archives-release.md) e
[E2 — validazione](2026-09-09-evidence-e2-validation.md) e [E2 — validazione](../reports/knowledge-archives-release.md) e
[E3 — validazione](2026-09-09-evidence-e3-validation.md). [E3 — validazione](../reports/knowledge-archives-release.md).
La [revisione di semplicità](2026-09-08-memory-evidence-simplification-review.md) La [revisione di semplicità](2026-09-08-memory-evidence-simplification-review.md)
registra il riesame di Q1–Q15. Il flusso principale è draft scritta dallo specialista registra il riesame di Q1–Q15. Il flusso principale è draft scritta dallo specialista
@@ -2,7 +2,7 @@
Data del piano: 2026-09-08. Aggiornamento 2026-09-09: M1–M3, E1–E3 e X1 Data del piano: 2026-09-08. Aggiornamento 2026-09-09: M1–M3, E1–E3 e X1
implementati. Risultati e limiti della verifica finale sono raccolti nel implementati. Risultati e limiti della verifica finale sono raccolti nel
[rapporto X1](2026-09-09-archive-repair-x1-validation.md). [rapporto X1](../reports/knowledge-archives-release.md).
La [revisione di semplicità](2026-09-08-memory-evidence-simplification-review.md) La [revisione di semplicità](2026-09-08-memory-evidence-simplification-review.md)
riesamina tutte le decisioni Q1–Q15 alla luce degli ultimi chiarimenti del riesamina tutte le decisioni Q1–Q15 alla luce degli ultimi chiarimenti del
+1 -1
View File
@@ -6,7 +6,7 @@ confini di test confermati dal proprietario il 2026-09-08.
Primo incremento del progetto Memory management. Attua le decisioni già approvate Primo incremento del progetto Memory management. Attua le decisioni già approvate
nel piano del 2026-09-08 e nell'ADR 0018. Le scelte tecniche di dettaglio qui nel piano del 2026-09-08 e nell'ADR 0018. Le scelte tecniche di dettaglio qui
proposte derivano dalla ricognizione del runtime. Gli esiti dell'implementazione proposte derivano dalla ricognizione del runtime. Gli esiti dell'implementazione
sono riportati nel [rapporto di verifica](2026-09-08-memory-m1-validation.md). sono riportati nel [rapporto di verifica](../reports/knowledge-archives-release.md).
## Problem Statement ## Problem Statement
@@ -1,103 +0,0 @@
# M1 — Implementazione e verifica
Data: 2026-09-08. Implementazione locale della
[specifica approvata](2026-09-08-memory-m1-spec.md), associata all'
[issue 27](https://git.tylconsulting.it/mptyl/ThothII/issues/27).
## Risultato
La pagina **Memory management** è disponibile nell'Administration dopo Database
management. Gestisce le quattro famiglie di card, elenco completo, ricerca e filtri,
ordinamento, dettaglio, creazione, modifica, cancellazione, collegamenti e dipendenze.
Richiede un amministratore autenticato e una selezione esplicita del workspace;
non richiede una sessione o un database DWH configurato.
Il harness possiede l'archivio PostgreSQL `thoth_memory`. Card, collegamenti,
dipendenze e lavoro di propagazione sono salvati nella stessa transazione.
La pagina distingue salvataggio fallito e contenuto salvato con indice incompleto,
offrendo retry anche per le cancellazioni. Recall Memory ed exemplar verificano
esistenza, workspace e proiezione corrente nell'archivio prima di restituire contenuto.
Promozione, salvataggio singolo e finalizzazione corrente passano dal servizio
autorevole. Le ricevute della sorgente impediscono duplicati e ricreazione di card
cancellate. Reindicizzazione e preprocessing non importano vecchi payload o sessioni.
L'errore Memory non annulla una sessione già finalizzata; il gate segnala anche
una promozione salvata con indicizzazione incompleta.
Le migrazioni sono versionate, controllate tramite checksum e incluse nel wheel
e nell'immagine core. Il servizio di preparazione `catalog-migrate` le esegue dopo
quelle del Catalog. Il runtime assume il ruolo limitato `thoth_memory_runtime`,
con isolamento del workspace tramite RLS e senza privilegi DDL.
## Verifiche eseguite
| Confine | Esito |
| --- | --- |
| Harness, test senza L0/L2 | 1.134 passati; i 9 test dei percorsi portabili sono stati eseguiti separatamente e sono passati. |
| Servizio Memory, PostgreSQL e Qdrant reali | 17 passati, inclusi CLI, migrazioni, ruolo runtime, isolamento, transazioni, outage, retry, cancellazioni, cambio famiglia e rebuild. |
| Gate Pi | 190 passati, inclusi identità UUID e avviso dopo salvataggio con indice incompleto. |
| Backend | 1.345 passati nella suite completa, 40 esclusi dalle condizioni previste dai test; un test di autenticazione ha superato il timeout sotto carico. Il relativo file è stato rieseguito isolato: tutti i 17 test passati. |
| Frontend | 632 passati, inclusi ingresso dall'AppShell, form, filtri, collegamenti, dipendenze e retry delle cancellazioni. |
| Browser integrato | Passato: autenticazione amministratore, creazione, modifica, riavvio del backend, rilettura, cancellazione e assenza nel recall. |
| Build e tipi | Build backend e frontend, typecheck TypeScript e build documentale strict superati. |
| Lint e diff | Ruff sui file Python modificati e `git diff --check` superati. Il lint globale segnala tre rilievi in file non modificati, elencati sotto. |
Il browser utilizza autenticamente frontend, login locale, Fastify, ThtRunner,
CLI Python, PostgreSQL e Qdrant. Gli embedding sono deterministici e le attività
Pi/sessione estranee al percorso Memory usano le fixture esistenti. Non sono state
intercettate le API Memory. Sono stati usati container temporanei PostgreSQL 16 e
Qdrant 1.18.2, senza accesso a un DWH remoto o a un modello generativo.
Il test browser ha consentito di correggere etichette accessibili instabili nei
campi compilati e la sovrapposizione del pannello di recupero ai comandi del dettaglio.
La selezione del workspace e l'uscita dalla pagina sono bloccate durante le operazioni.
Il lint globale preesistente riguarda soltanto:
- ordinamento import in `harness/tests/test_effective_relationships.py`;
- ordinamento import in `harness/tests/test_p3_dwh_binding.py`;
- uso di `datetime.UTC` in `harness/tht/mschema/catalog_snapshot.py`.
## Riproduzione
Usare Node 24 e le dipendenze installate dei tre layer. Per eseguire il harness
in un ambiente con home non scrivibile si può impostare `THT_HOME` su una directory
di prova. I test dei percorsi portabili devono essere eseguiti senza questo override,
perché verificano deliberatamente la risoluzione dell'home e di `THT_DATA_ROOT`.
```sh
cd harness
THT_HOME=/private/tmp/thothii-m1-test-home .venv/bin/pytest -m 'not l0 and not l2' --ignore=tests/test_portable_paths.py -q
.venv/bin/pytest tests/test_portable_paths.py -q
.venv/bin/pytest tests/memory/test_administration.py -q
npm test
```
```sh
cd backend
npx vitest run
npx tsc --noEmit -p .
npm run build
```
```sh
cd frontend
npx vitest run
npx tsc -b
npm run build
THT_MEMORY_BROWSER_E2E=1 npx playwright test e2e/memory-real.spec.ts
```
Il percorso browser richiede Docker, Python del harness, Go per il bridge di
autenticazione e Chromium di Playwright. Avvia risorse isolate e le rimuove alla
fine. Su macOS il browser deve poter avviare i processi Chromium fuori dalle
restrizioni della sandbox. La build documentale si esegue dalla radice con
`./scripts/build-docs.sh`.
## Stato della consegna
Le modifiche sono nel worktree locale. Nessuno stack già attivo è stato aggiornato
e nessun dato esistente è stato migrato o eliminato. Prima di usare M1 su
un'installazione occorrono il nuovo core e la preparazione `catalog-migrate`.
M2 (retrieval ibrido ed espansione dei collegamenti), M3 (integrazione estesa nel
workflow) ed Evidence management restano incrementi successivi.
+1 -1
View File
@@ -2,7 +2,7 @@
Data del piano: 2026-09-08. Aggiornamento 2026-09-09: M1–M3 implementati, Data del piano: 2026-09-08. Aggiornamento 2026-09-09: M1–M3 implementati,
con integrazione X1 per le correzioni persistenti dei conflitti. Risultati e limiti con integrazione X1 per le correzioni persistenti dei conflitti. Risultati e limiti
sono raccolti nel [rapporto X1](2026-09-09-archive-repair-x1-validation.md). sono raccolti nel [rapporto X1](../reports/knowledge-archives-release.md).
La [revisione di semplicità](2026-09-08-memory-evidence-simplification-review.md) La [revisione di semplicità](2026-09-08-memory-evidence-simplification-review.md)
mantiene il perimetro Memory e precisa un salvataggio sequenziale e una pulizia mantiene il perimetro Memory e precisa un salvataggio sequenziale e una pulizia
@@ -1,9 +1,31 @@
# PRD — Security hardening per Docker personale e server multiutente # PRD — Security hardening per Docker personale e server multiutente
## Ripresa del lavoro e limiti di autorizzazione
Questo PRD resta una bozza, non una specifica di implementazione approvata. Il precedente
prompt di ripresa è stato consolidato qui il 15 settembre 2026. Riconfermare i rilievi
SEC-01–SEC-12 contro codice e dipendenze correnti, separando fatti, ipotesi, rischi del
profilo locale/server e problemi già risolti. Le vecchie survey non provano lo stato del server.
Usare inizialmente controlli in sola lettura e dati sintetici; accesso a server, IdP, DWH
o provider e relative mutazioni richiedono target e operazioni concordati.
Prima di implementare, il proprietario deve confermare profili di fiducia, isolamento
del runtime Pi, trattamento dei valori sensibili, revoca, retention, limiti di risorse,
priorità e criteri misurabili. I modelli visibili sul server sono un problema separato.
Registrare le decisioni in questo PRD; pubblicare spec e ticket Gitea soltanto dopo la
conferma del perimetro e della granularità. La bonifica documentale non autorizza il
codice di sicurezza, né il rollout di tutti i punti SEC.
L'implementazione futura richiede un worktree dedicato, test positivi/negativi ai confini
concordati, typecheck e revisione contro standard e specifica. Conservare gate manuali,
evidenze e rollback; un ticket completato non chiude automaticamente il PRD. Non copiare
segreti o dati operativi nei worktree. Usare le skill effettivamente disponibili per
chiarimento, diagnosi e revisione, senza assumere che i vecchi nomi dei comandi esistano.
**Stato:** bozza da validare con `grill-with-docs`; implementazione rinviata. **Stato:** bozza da validare con `grill-with-docs`; implementazione rinviata.
**Data:** 8 settembre 2026. **Data:** 8 settembre 2026.
**Owner delle decisioni:** il maintainer di ThothII. **Owner delle decisioni:** il maintainer di ThothII.
**Ripresa:** [prompt per la prossima sessione](2026-09-08-security-hardening-resume-prompt.md). **Ripresa:** seguire la sezione «Ripresa del lavoro e limiti di autorizzazione» sopra.
Questo documento conserva la survey di sicurezza discussa con il maintainer e propone requisiti, Questo documento conserva la survey di sicurezza discussa con il maintainer e propone requisiti,
priorità e criteri di accettazione. Non è una spec approvata, un penetration test, una certificazione priorità e criteri di accettazione. Non è una spec approvata, un penetration test, una certificazione
@@ -1,100 +0,0 @@
# Riprendere il PRD di sicurezza con le skill di Pocock
**Stato:** prompt conservato per uso futuro; nessuna esecuzione programmata.
**PRD:** [Security hardening per Docker personale e server multiutente](2026-09-08-security-hardening-prd.md).
Apri una sessione nella codebase ThothII e incolla il blocco seguente. Il nome corretto della
skill è `grill-with-docs`, che combina `grilling` e `domain-modeling`. Il prompt apre la fase di
chiarimento; il passaggio a issue, worktree e implementazione resta soggetto alle conferme indicate.
```text
Riprendiamo il lavoro di sicurezza rinviato l'8 settembre 2026.
Leggi docs/plans/2026-09-08-security-hardening-prd.md. È una bozza di PRD ricavata da una
survey storica, non una spec approvata né una prova della configurazione del server remoto.
Voglio preparare interventi proporzionati per Docker su Mac/PC personale e per un server
multiutente con autenticazione built-in oppure OIDC. Il problema separato dei modelli
visibili sul server è fuori perimetro.
Usa realmente le skill di Matt Pocock: leggi le istruzioni installate, dichiarando quali
applichi. Parti da ask-matt per verificare il percorso e da grill-with-docs per il lavoro
di design; quest'ultima richiede grilling e domain-modeling. Se una skill non è disponibile,
segnalalo e concorda il fallback, senza installarla o fingere di averla eseguita.
FASE 1 — Riconferma delle evidenze, senza modificare il runtime
1. Leggi AGENTS.md, PROJECT_STATE.md, CONTEXT.md, le istruzioni docs/agents/ su dominio,
issue tracker e label, gli ADR pertinenti e il PRD. Controlla HEAD, stato del worktree
e differenze dalla baseline della survey. Preserva tutte le modifiche preesistenti.
2. Riconferma i rilievi SEC-01…SEC-12 nel codice corrente. Separa fatti verificati,
ipotesi, rischi condizionati al profilo e problemi già risolti. Non trattare i vecchi
conteggi delle dipendenze come una scansione aggiornata.
3. Usa controlli locali read-only e dati sintetici. Per un difetto da riprodurre, usa
diagnosing-bugs con un segnale ripetibile sul comportamento effettivo; una diagnosi
non autorizza ancora il fix. Confronta fatti di librerie e advisory con fonti primarie
correnti quando necessario, senza inviare codice privato o segreti ai servizi di ricerca.
4. Non accedere o intervenire su server, IdP, DWH o provider reali senza aver concordato
target e operazioni. Non mostrare API key, cookie, password o campioni di dati reali.
Esito della fase: una matrice aggiornata che conserva gli ID dei rilievi, con evidenza,
profilo interessato e stato. Un fatto ancora non verificabile resta esplicitamente aperto.
FASE 2 — grill-with-docs, con me presente
5. Costruisci l'albero delle decisioni. Parti da profili di deploy, fiducia fra utenti e
condivisione dei dati; poi affronta isolamento di Pi, policy dei valori sensibili,
revoca, retention e limiti seguendo le dipendenze effettive.
6. A ogni round presenta soltanto le domande attualmente sbloccate, numerate, con la tua
raccomandazione e i trade-off. Attendi le mie risposte prima di assumere le decisioni
successive. Cerca autonomamente i fatti ricavabili dal repository; usa agenti di
ricerca mirati quando previsto dalla skill, senza delegare a loro le mie decisioni.
7. Aggiorna il PRD distinguendo proposte e decisioni confermate. Aggiorna CONTEXT.md solo
per termini realmente risolti. Proponi ADR soltanto per scelte difficili da invertire,
sorprendenti senza contesto e fondate su alternative reali: basta il formato minimo.
8. Concorda requisiti, priorità, rischi accettati e criteri misurabili, inclusi i tempi
di revoca e i limiti di risorse. Conferma con me i confini pubblici dei test prima
di scriverli. Mantieni espliciti i gate manuali già presenti in PROJECT_STATE.md.
Esito della fase: nessuna decisione bloccante lasciata implicitamente all'agente;
riepilogo e mia conferma della comprensione condivisa. Fino a quella conferma rimani
su analisi e documentazione: nessun cambiamento applicativo o di deployment.
FASE 3 — Spec e ticket, soltanto dopo mia conferma
9. Usa to-spec per sintetizzare le decisioni già prese, senza riaprire arbitrariamente
l'intervista. Chiedimi conferma della pubblicazione prima di creare la spec nel
tracker canonico Gitea indicato in docs/agents/issue-tracker.md, non nel mirror GitHub.
Collega la spec canonica dal PRD e rendi chiaro quale documento è la fonte aggiornata.
10. Usa to-tickets per proporre fette verticali verificabili autonomamente, dimensionate
per un contesto fresco. Collega ogni ticket ai requisiti e ai rilievi pertinenti,
indica i veri blocker e includi criteri positivi e negativi. Fai approvare granularità
e dipendenze prima di pubblicare. Solo i ticket approvati e completi ricevono
ready-for-agent; non rimetterli in triage e non chiudere automaticamente la spec padre.
11. Se una decisione richiede una prova eseguibile, proponi un prototype limitato a quella
domanda prima di fissare la spec. Usa wayfinder solo se il lavoro risulta realmente
troppo ampio e incerto per essere chiarito con grill-with-docs.
Esito della fase: spec approvata e ticket autosufficienti con dipendenze risolte o esplicite.
Chiedimi se autorizzo il primo ticket: l'approvazione del design non avvia da sola il codice.
FASE 4 — Implementazione futura autorizzata
12. Prima di modificare codice, concorda e crea un worktree dedicato, verificando percorso,
branch e commit base. Non riusare una directory occupata, non alterare il worktree
originario e non copiare automaticamente segreti o dati operativi. Assicurati che
PRD e prompt siano disponibili nel worktree attraverso un passaggio esplicito.
13. Esegui implement su un ticket sbloccato per volta, in un contesto fresco. Segui tdd
ai confini concordati: un test rosso sul comportamento, implementazione minima,
test verde. Esegui typecheck e test mirati durante il lavoro e le suite pertinenti
al termine; usa fixture locali per IdP, DWH e provider.
14. Esegui code-review sui due assi Standards e Spec, usando i due agenti previsti dalla
skill e una base Git fissata. Assicurati che il diff esaminato includa tutto il lavoro
del ticket, anche se ancora non committato; un diff vuoto non è una review superata.
Risolvi i rilievi e verifica di nuovo. Commit soltanto del lavoro pertinente nel
worktree autorizzato; push, merge e deploy richiedono un'autorizzazione distinta.
15. Consegna evidenze dei test, istruzioni di adozione e rollback, gate manuali pendenti
e rischi residui. Non dichiarare chiuso il PRD intero se è concluso soltanto un ticket
o se resta un'accettazione dell'owner.
Inizia dalla Fase 1, poi proponimi il primo round di grill-with-docs.
```
@@ -1,112 +0,0 @@
# X1 — validation of session archive corrections
Date: 2026-09-09. The joint Memory/Evidence repair increment is implemented.
The authoritative contract is [Session archive corrections](../contracts/archive-repair.md).
## Delivered behavior
The session gate shows complete before/after content for specific alternatives targeting
Memory or Evidence. The reviewer chooses one correction or rejects all proposals as
inadequate and requests reformulation. The resulting receipt survives interruption;
saved content and index activation are reported separately. Pending activation offers
retry of the same chosen operation. A subsequent curator change blocks replay.
Application requires an administrator in the harness and the responding browser
principal's archive-management permission. Cross-principal runtime responses cannot
misattribute the correction. A non-administrator can decline or continue the current
question without modifying shared archives. The gate does not advance a workflow phase.
Memory and Evidence remain separate domains. The integration coordinator reuses their
canonical persistence and activation operations. A session Evidence correction requires
a consolidated archive, preserves source lineage, and cannot publish unrelated external
edits. Existing administration, source import, dependency cleanup and final Memory review
remain available. No automatic Git commit or push was added.
## Verification
- Harness regression: **1,264 passed**, one skipped, five deselected. All **nine**
portable-path checks passed separately without `THT_HOME`. Python lint passed on changed modules.
- Backend: **1,366 passed**, 40 skipped. Tests include actual session-response routes
for both target archives, unauthorized response, malformed choice and runtime ownership.
- Frontend: complete suite **645 passed**; the final display adjustment passed all four
focused widget tests. Backend and frontend TypeScript checks passed.
- Pi extension: **199 passed**, including closed human choices, rejection, failure/retry,
forged selections and the updated public tool schema. The modular skill projection is
byte-identical to its updated approved template.
- PostgreSQL/Qdrant integration traverses the actual Python CLI for preparation,
application and recovery inspection. Corrected Memory and Evidence are retrieved from
real indexes, and Evidence activation preserves the Memory card. Session loading is a
controlled fixture and embeddings are deterministic; this is not an LLM quality test.
- Failure tests cover both targets, index outage, replay, later edits, workspace/session
isolation, changed session context, rejection, non-admin writes, and interruption after
the Evidence file write but before its saved receipt.
- Playwright desktop/mobile: **one passed**. The real widget renders both alternatives,
accepts an Evidence choice, displays pending activation and allows retry to active.
No page errors or mobile horizontal overflow. Screenshots are
`/private/tmp/thothii-x1-repair-desktop.png` and `/private/tmp/thothii-x1-repair-mobile.png`.
This browser fixture controls operation outcomes; persistent behavior is tested above.
- Strict MkDocs build and `git diff --check` passed.
## Local installation and reviewer acceptance
Core and frontend images were rebuilt from this worktree using the existing local
preview launcher. Migration `004_archive_repairs.sql` was applied to the existing
installation catalog. The new gate is available to session workflows; it is not an
always-visible administration panel. Existing PSD archive content was not changed by
the synthetic validation cases.
All five local services are healthy at `http://127.0.0.1:8080/`.
The technical increments and their planned checks are complete. The end-user acceptance
check remains a real session containing a meaningful domain conflict, with the reviewer
evaluating the proposed correction. Automated browser validation uses temporary accounts
and data, not the user's authenticated PSD session. Source import retains its E3
validation boundaries; no broader model-quality benchmark was added.
## Follow-up acceptance: configured model
The opt-in `test_real_model_proposes_a_reviewable_persistent_archive_correction`
passed with the installation's **zai/glm-5.3** model. Synthetic Memory asserted an
order-ID-only join; synthetic Evidence required the financial year too. The model
returned two schema-valid, specific alternatives with complete content and the exact
target revisions. The test reviewer selected Memory, persisted the correction through
the real coordinator and PostgreSQL, and retrieved the updated rule. Evidence stayed
unchanged. This test uses deterministic vectors and the configured completion helper;
it does not claim a full autonomous Pi session or human acceptance of PSD semantics.
The run log is `/private/tmp/x1-acceptance-model.log`. Reproduce with
`THT_MEMORY_L2_INSTALLATION=<installation.yaml>` and `THT_MEMORY_L2_CORE=<core-container>`
using `pytest -q -s -m l2 tests/memory/test_administration.py -k real_model_proposes`.
Credentials are resolved inside core and are not returned to the test runner.
## Follow-up acceptance: both administration pages
The opt-in `frontend/e2e/memory-real.spec.ts` passed through real authentication,
Fastify, ThtRunner, Python, isolated PostgreSQL and Qdrant. It verifies:
- Database management, Memory management and Evidence management appear as peers in
that order, with no active core session or DWH binding required.
- Memory creation, editing, persistence across backend restart, deletion and absence
from subsequent recall.
- Canonical Evidence remains intact after the Memory deletion. Its full rule is read
through the real Evidence administration worker; content filtering finds it and an
unmatched filter produces the empty state.
- Requests for an unregistered workspace return 404 for both archives.
- Desktop and mobile Evidence views render without horizontal document overflow.
On phones, both archive pages have at least 380px of usable width at a 390px viewport.
Navigation opens in the shared accessible dialog, closes with Escape or archive selection,
and returns focus to the trigger after Escape.
The temporary PostgreSQL readiness probe now waits for TCP, avoiding the image's
socket-only initialization server. The browser waits for Memory refresh to finish
before leaving its page, matching the existing navigation guard. Visual inspection
also exposed a real mobile layout issue: the fixed sidebar left only 134px for the
Evidence page. `ArchiveNavigation` now moves that sidebar into the shared dialog below
768px on Memory/Evidence pages. Desktop behavior is unchanged. The frontend image
was rebuilt for the local preview.
Run log: `/private/tmp/x1-acceptance-browser7.log` (**one passed**).
Screenshots: `/private/tmp/thothii-acceptance-evidence-desktop.png` and
`/private/tmp/thothii-acceptance-evidence-mobile.png`. Reproduce with
`THT_MEMORY_BROWSER_E2E=1 npx playwright test e2e/memory-real.spec.ts` from `frontend/`.
The fixture removes its temporary containers, accounts and checkout on completion.
The TypeScript check, Python lint, strict documentation build and diff check also pass.
@@ -1,67 +0,0 @@
# Evidence E1 — validation
Date: 2026-09-09. Scope: editable Curated Evidence v4 and the persistent local archive.
## Implemented behavior
- Parser, renderer, authoring output and normalization share the existing typed payloads.
Visible Markdown edits determine content for all eight kinds. Legacy v1–v3 conversion
is explicit and lossless, with errors for content that cannot be represented exactly.
- Manual declarations record the curator. A correction preserves the original document
as lineage, separately from the current declaration. No source hash is needed to
create a manual file.
- The local archive records baselines, immutable candidates, active revisions and
deletion/source suppression metadata. Unresolved review items and invalid edits block
consolidation. Missing archive directories are availability failures, not deletions.
- Activation failures preserve the previous active revision. Interrupted normalization
replays only unchanged input bytes; later operator edits survive recovery.
- Revision-checked correction methods reject stale workflow updates. Legacy preparation
and resolution cannot overwrite an initialized local archive; explicit import/refresh
integration is deferred to E3.
## Verification
The final harness suite excluding opt-in L0/L2 and portable-layout cases passed with
**1,180 tests** (58 deselected). All **9 portable-layout tests** passed separately with
`THT_HOME` unset. The dedicated real-Qdrant integration test passed, including the
optional 35-unit PSD probe. Ruff passed on the changed Evidence implementation and
tests, and the strict documentation build succeeded. No frontend or backend TypeScript
changes are part of E1.
The integration test uses an isolated Qdrant 1.18.2 container, the actual corpus
pipeline, semantic chunking, vector adapter and active Evidence searcher. Deterministic
three-dimensional embeddings isolate file/content correctness from model behavior.
It verifies that raw edits do not change recall, consolidation updates recalled content
and curator identity, a blocked candidate preserves prior recall, and deletions remove
recall. Existing schema and Memory records survive each operation.
All **35 PSD units** were copied from the owner's workspace into
`/private/tmp/thothii-e1-psd.bsW4cp`. Deterministic conversion preserved every ID, payload,
scope, provenance and review item. There were no unresolved review items. The optional
integration probe then indexed all 35 converted units and compared their complete ID
set to the original. It uses PSD's actual `max_chunk_chars: 5000`; a preliminary probe
at 4000 correctly blocked an oversized atomic unit.
Reproduce the isolated real-corpus probe after creating a converted workspace copy:
```sh
cd harness
THT_E1_PSD_COPY=/absolute/path/to/converted-copy \
.venv/bin/pytest -q -s tests/test_evidence_editable_integration.py
```
The environment variable is optional. Ordinary CI uses only synthetic Evidence. No
source refresh, external document download, DWH call or model request is involved.
## Delivery boundary
E1 is a core/library increment. E2 must add the installed manual consolidation command,
connect runtime source selection to the active local snapshot, and build administrative
list/filter/detail with real persistent host paths and manual Git instructions. E3
adds source acquisition and explicit refresh/conflict handling. X1 later wires deliberate
joint Memory/Evidence corrections into review gates.
The actual PSD Evidence checkout was not converted. The live Docker preview at
`http://127.0.0.1:8080` remains the previously deployed M3 stack, with no new Evidence
administration page. The corpus conversion and reindexing described here used copies
and disposable test resources.
@@ -1,86 +0,0 @@
# Evidence E2 — validation
Date: 2026-09-09. E2 is implemented locally and installed on the existing Docker preview.
E3 source imports/refresh and X1 deliberate Memory/Evidence workflow corrections remain open.
## Delivered behavior
- Independent **Administration → Evidence management**, after Memory, protected by
`evidence.manage`: complete typed content, provenance and original excerpts, review items,
pagination, search, kind/purpose/status and scope/source filters, sort, and refresh.
- Working-file states distinguish active, modified, new, removed, invalid, legacy and review
required. Detail shows the actual configured host path with copy controls. Instructions
cover external editing, all eight Markdown templates, consolidation and manual Git.
- Installed `tht workspace evidence consolidate --workspace <id> [--json]` uses a closed
maintenance envelope. First use converts legacy units. Validation, immutable candidates,
activation and retry run through the existing corpus pipeline without a DWH scan or Git.
- Runtime and ordinary preprocessing consume the active local snapshot. Unconsolidated
edits remain excluded. Catalog/Schema readiness is not advanced by this operation.
Clear preserves curated files, archive metadata and Memory; full preprocessing must
recreate the missing Reference/Schema derivations afterward.
- Immutable runtime lease filenames now identify rendered bytes as well as logical input
identity. This fixes upgrades colliding with old runtime files without changing Catalog
fingerprints or removing the checks against tampered files.
## Automated checks
The complete backend suite passed: **1,355 tests**, 40 skipped. The complete frontend
suite passed: **639 tests**. Both TypeScript checks passed. Native Go workspace operation
and CLI tests passed, including rejection of arbitrary consolidation flags. Ruff passed
for changed Python implementation and test files.
The harness run passed **1,248 tests**, with one skipped and five deselected. Its three
portable-path tests failed because that run deliberately set `THT_HOME` to the test
runtime; rerunning the portable tests with `THT_HOME` unset passed. The final focused
administration/path suite passed all 18 tests, including actionable migration errors and invalid
consolidation combinations rejected before cleanup or indexing.
The real-Qdrant integration test exercised the actual harness consolidation CLI with
deterministic embeddings: all 35 PSD units converted and indexed with stable identities;
active-only source selection; separate Schema and Memory canaries; Clear and rebuild
from the retained snapshot. Unit tests cover validation, saved-but-unindexed failure,
retry, browsing/filtering, no automatic Git, and no Catalog mutation from consolidation.
A temporary Git repository and bare local remote exercise the documented manual sequence:
edit, add and remove files, consolidate, inspect, stage the complete Evidence tree,
commit, push and clone. The clone retains changed content, additions, deletions, managed
metadata and an accessible active snapshot. No remote user repository was pushed.
## Installed preview
The existing Compose project is `thothii-18998cca7b0a`, at `http://127.0.0.1:8080`.
The persistent editable checkout is:
```text
/Users/mp/projects/ThothII/deploy/psd/evidence-registry/repo/psd-clinical/evidence
```
The original registry checkout was copied from its retained Docker volume. The original
author repository was not changed. Core and maintenance share a nested host bind for
`repo`; registry state/snapshots and all other existing data volumes were retained.
The installation descriptor includes the existing workspace bindings and the new
`evidence-host.yaml` override. The previous descriptor and native binary are backed up
at `/private/tmp/thothii-installation-before-e2.yaml` and `/private/tmp/tht-before-e2`.
The real installed command succeeded with **35 documents, 35 chunks, 35 changed, zero
removed**, using the configured embedding service and Qdrant. A second run succeeded
with **35 unchanged, zero changed**. Reading the actual archive from core returned
35 active units and no file errors. All five long-running services are healthy.
Only the Evidence stage ran. The strict documentation build and `git diff --check`
also passed. The stack launcher is `bash /private/tmp/thothii-memory-preview.sh`; keep its
worktree image-build override until this branch is integrated into the main checkout.
Browser verification reached the local login page. The saved administrator password
does not match the current account hash, so the authenticated visual check remains
manual. No account or password was modified. React interaction tests cover navigation,
detail, host paths, templates, filtering, pending activation and invalid files.
## Boundaries
There is no web content editor, watcher, automatic commit/push, or implicit source refresh.
The API exposes administration reads and consolidation; the archive's revision-checked
save/remove operations remain available for the later explicit workflow corrections.
These gates are not claimed as implemented by E2. Initialized local archives retain
structural/review checks but bypass the legacy fixed retrieval-evaluation fixture so
its old expected IDs cannot veto deliberate deletions. A general retrieval benchmark
is outside the agreed scope.
@@ -1,100 +0,0 @@
# Evidence E3 — validation
Date: 2026-09-09. Explicit source import/refresh and decisions are implemented. X1,
the integration of deliberate Memory/Evidence corrections into workflow gates, remains next.
## Delivered behavior
The independent Evidence page now offers **Sources and imports**. An operator copies
a specialist's draft into `evidence/incoming/`, then explicitly imports/refreshes.
Original local Markdown and configured HTTP/S3 sources use existing read-only adapters.
Acquisition retains raw bytes, source identity and versioned normalized documents.
The existing Pi authoring refiner prepares typed, editable v4 proposals.
Unchanged hashes skip refinement. All source acquisitions/refinements must succeed
before saving a new set of comparisons. Missing sources are recorded as unavailable,
never interpreted as permission to delete. No runtime lookup, ordinary consolidation
or preprocessing triggers remote refresh once the local archive is initialized.
The administrator sees current and proposed units, scope, content, excerpts, review
items and explicit retirement IDs. **Keep local Evidence** records the retained wording
as a manual declaration with original lineage. **Use proposed Evidence** adopts the
proposal and its source version. Both save and activate through the existing archive
and corpus pipeline; review items block adoption. Comparisons use optimistic checks
on affected file bytes. Interrupted decisions have a durable replay journal and retry
without reacquisition, while intervening external edits are preserved and reported.
Deleted IDs remain reserved. New model-generated identities from sources with curated
deletions are also conservatively suppressed; surviving IDs can still receive reviewed
updates. Deliberate new knowledge can be authored as a manual file. This mechanical
protection does not depend on the model detecting semantic duplication or contradictions.
Installed commands are `workspace evidence refresh` and `workspace evidence decide`,
alongside E2 consolidation. Decision envelopes carry a source identity, comparison
revision and keep/replace choice. Extra URLs, arbitrary paths, forged actors and unknown
fields are rejected at the public API/CLI boundary. HTTP requests bind the authenticated
curator. Source operations do not mutate Catalog readiness or run DWH/schema stages.
The Python source worker is internal; the workflow CLI's visible surface is preserved.
## Checks
- Complete backend suite: **1,359 passed**, 40 skipped. Complete frontend suite:
**641 passed**. Both TypeScript checks passed; native Go CLI/workspace tests passed.
- Harness regression run: **1,256 passed**, one skipped and five deselected, with
portable-path tests run separately without `THT_HOME`. All **24 focused import,
CLI-surface and portable-path checks** passed. These
cover import, unchanged refresh, access failure, missing source, manual correction,
keep/replace, deletion suppression, stale comparisons, failure/retry and interrupted
journal writes. Ruff passed on the changed Python implementation and tests.
- The real-Qdrant test traverses the actual harness source CLI with deterministic
refinement/embedding boundaries: import is absent from recall before a decision,
accepted content becomes searchable, refreshed proposals preserve active manual
corrections, replacement removes the former text, deletion remains absent after
another refresh, and unrelated Schema/Memory canaries survive.
- Source contract fixtures cover controlled HTTP and S3 identities, exact acquired
bytes, and acquisition call counts. Existing adapter tests retain transport/egress
coverage. The test does not claim to exercise a live S3 account.
- React interaction tests verify explicit refresh, comparison content, exact decisions,
saved-decision retry and failure feedback. Route tests cover admin authorization,
workspace isolation, strict inputs and principal attribution. Service tests verify
the trusted config file descriptor and absence of Catalog mutation.
## Local preview
Core/frontend were rebuilt for the existing `thothii-18998cca7b0a` stack. Its persistent
archive and data volumes are retained. The native `/usr/local/bin/tht` was updated;
the previous executable is at `/private/tmp/tht-before-e3`.
The installed refresh command ran against `psd-clinical` successfully: **35 unchanged
sources, zero changed, zero pending comparisons**. All 35 source hashes matched their
existing units, so this probe required no refinement and changed no active Evidence.
Source registry metadata was saved locally; no Git commit or push was performed.
A separate synthetic draft was passed to the configured Pi/model inside core. It
produced one domain proposal with one review item, which was not activated. The probe
exposed a deployment issue: Python wheel modules and Pi skills live in different
directories. The refiner now resolves resources through `THT_HARNESS_DIR`, with the
source-tree location as its development fallback; a regression test covers this layout.
The corrected installed worker was then exercised end to end in a temporary workspace
inside core, using the real configured Pi/model and a synthetic `incoming/orders.md`.
It returned success, one changed source, one comparison and one proposal with a review
item. The active snapshot remained absent. The temporary directory was removed on exit;
the probe did not open the DWH or activate an index. Strict docs build and
`git diff --check` also passed.
The in-app browser still showed the login page with the prior credential error.
Authenticated visual verification remains manual; no credentials were reset or retried.
## Operational instructions and boundaries
See [Import drafts and refresh sources](../contracts/curated-evidence-v4.md#import-drafts-and-refresh-sources)
for commands, local paths, review, retry and backup/Git requirements. Preserve the
complete Evidence tree, including source comparisons, journals and acquired versions.
Local activation and transfer to the remote Git repository remain separate operator steps.
No web content editor, automatic Git, background watcher, new job queue, general
retrieval benchmark or automatic source merge was introduced. Review/refinement is
sequential and bounded by per-source limits plus 200 documents/100 MiB per refresh.
New changed-source decisions and failure scenarios use isolated test data; live PSD
curated content was kept unchanged. X1 is not included in this increment.
@@ -1,96 +0,0 @@
# M2 — Ricerca ibrida e collegamenti
Data: 2026-09-09. Implementazione locale dell'incremento M2 del
[piano Memory approvato](2026-09-08-memory-management.md#piano-esecutivo).
## Risultato
La ricerca Memory ed exemplar usa embedding dense e BM25 in Qdrant. I filtri sono
applicati in entrambi i rami prima della selezione dei candidati. Il core espande
i collegamenti uscenti, deduplica, limita cicli e visite, riordina insieme i risultati
e restituisce il contenuto corrente verificato in PostgreSQL. Non usa un database
a grafo né una chiamata LLM per il riordinamento.
Il contesto fisico distingue database, schema, tabella e colonna e richiede che
corrispondano alla stessa dipendenza strutturata. Le card senza dipendenze valgono
per il workspace; un riferimento a un antenato fisico si applica ai suoi discendenti.
Ambito descrittivo e concetti possono restringere ulteriormente la ricerca tramite
`--filters`. La CLI impedisce di sostituire database/schema del runtime. Il dettaglio
dei limiti e della formula di ranking è nel [contratto operativo](../gestione-memory.md#hybrid-recall-and-links).
L'incremento conserva i confini dei gate correnti: F2 riceve chiarimenti di dominio,
gli exemplar restano consultativi. Un collegamento non autorizza a consumare una
famiglia diversa o una card fuori ambito. Il riepilogo finale, i nuovi gate e la
pulizia delle dipendenze dopo sincronizzazione fisica appartengono a M3.
## Transizione e recupero
La migrazione versionata `002_hybrid_projection.sql` aggiunge il formato delle
proiezioni. Le card M1 restano autorevoli e consultabili; le loro proiezioni dense
risultano pendenti e non possono alimentare il recall. Un retry esplicito o
`tht memory index -c <runtime.yaml>` costruisce dense e BM25 dal contenuto corrente.
Il formato della proiezione e la revisione della card sono verificati prima dell'uso.
Il salvataggio può aggiungere il vettore sparse mancante nella sola collezione
Memory. Non sostituisce configurazioni incompatibili e non modifica Reference.
Il rebuild esplicito ricrea anche una collezione Memory assente; il test elimina
la collezione, ricostruisce da PostgreSQL e verifica il ritorno della sola card conservata.
Fallimenti lasciano il lavoro di propagazione persistito e recuperabile. Nessuna
importazione da JSONL, sessioni storiche o vecchi payload Qdrant.
## Verifiche
| Controllo | Esito |
| --- | --- |
| Suite harness senza L0/L2, escluso il file dei percorsi portabili | 1.152 test passati nell'esecuzione finale. |
| Suite mirata Memory, adapter e CLI, con embedding reale | 78 test passati. |
| Verifica aggiuntiva del rebuild con collezione assente e adapter | 54 passati; il solo test del modello reale era escluso in questa riesecuzione. |
| API Fastify Memory | 13 test passati, compresa propagazione della lingua del workspace. |
| Browser amministrativo integrato | Passato: creazione, modifica, riavvio, rilettura, cancellazione e verifica del recall. |
| Wheel e casi CLI | 10 test passati; il wheel include entrambe le migrazioni Memory. |
| Build e controlli statici | Build/typecheck backend, Ruff sui file Python interessati e build documentale strict passati. |
- Test deterministici: collegamenti necessari, contenuto corrente, duplicati,
cicli, profondità e limiti, card mancanti o pendenti, rifiuti già registrati,
famiglie e ambiti esclusi, dipendenze omonime e filtri CLI vincolati al runtime.
- Adapter: stesso filtro nei prefetch dense/BM25 per Memory ed exemplar; restano
coperti i contratti Evidence esistenti.
- PostgreSQL/Qdrant: salvataggio, modifica, cambio famiglia, cancellazione,
ricostruzione, riferimento separato, isolamento, RLS, errori, retry e transizione
delle proiezioni M1 al formato ibrido.
- Recupero effettivo: client Ollama di produzione con il modello configurato
`qwen3-embedding:0.6b`, dimensione 1024, e Qdrant dell'immagine fissata in Compose.
La domanda sulla chiave commessa fra esercizi recupera la regola attesa e la
granularità collegata, escludendo un altro database, un altro ambito e dipendenze
che corrispondono soltanto combinando riferimenti distinti. Verifica separatamente
dense, BM25 e fusione, poi recall, cancellazione e rebuild.
Il test effettivo avvia un processo Ollama separato, montando il volume del modello
installato in sola lettura. PostgreSQL e Qdrant sono container temporanei dedicati,
eliminati a fine test. Non usa ID di risultati predisposti. Il browser amministrativo
usa invece embedding deterministici: verifica il collegamento fra UI, autenticazione,
Fastify, ThtRunner e persistenza, senza essere una misura di qualità del recupero.
Non è una valutazione generale della qualità semantica su un corpus di produzione;
verifica i casi di recupero richiesti da M2, con il percorso reale configurato.
## Riproduzione
Dalla directory `harness`, con Docker disponibile:
```sh
THT_HOME=/private/tmp/thothii-m2-test-home \
THT_MEMORY_TEST_OLLAMA_VOLUME=<volume-modelli-installazione> \
THT_MEMORY_TEST_MODEL=qwen3-embedding:0.6b \
THT_MEMORY_TEST_DIMENSIONS=1024 \
.venv/bin/pytest -q tests/memory/test_administration.py \
tests/memory/test_retrieval.py tests/memory/test_recall.py \
tests/test_qdrant_vector_store.py tests/test_solved_search_cli.py
```
Senza il volume esplicito, il solo test con modello reale viene escluso; gli altri
test restano eseguibili. Il volume deve contenere il modello indicato. Le immagini
Qdrant e Ollama del test sono lette da `compose.yaml`.
Consegna locale: non sono stati eseguiti deploy, migrazioni delle installazioni
attive o aggiornamenti remoti dell'issue tracker.
@@ -1,78 +0,0 @@
# Memory M3 — validation
Implemented on 2026-09-09 in the rapid-harbor worktree.
## Delivered behavior
- F8 presents an editable summary of proposed additions, explicit updates and links.
Only selected content is saved, including the optional solved-question exemplar.
- Proposals reference effective approved decisions. Exact existing content is reused.
Concurrent edits invalidate an update; manual identities and origins are preserved.
- Selected cards and links commit atomically. Durable receipts recover repeat delivery
and the gap before the session review marker. Finalization does not add Memory.
- F4/F6/F7 retrieve SQL rules and explained errors for the existing approval gates.
Retrieval is consultative and does not write an approval decision.
- Successful Catalog physical sync deletes cards with matching removed dependencies.
The same Catalog transaction marks pending Memory cleanup. Retry keeps the original
removals and does not rescan; deletion receipts and projection tombstones survive restarts.
- Migration 003 adds minimal review and physical-cleanup receipts.
## Executed checks
| Check | Result |
| --- | --- |
| Harness deterministic suite, excluding portable-path environment cases | 1,152 passed |
| Portable-path suite without the temporary THT_HOME override | 9 passed |
| Memory service/retrieval with isolated PostgreSQL and Qdrant | 41 passed, 1 optional real-embedding case skipped, 1 L2 case excluded |
| Pi gate suite | 195 passed |
| Backend full suite | 1,346 passed; one auth timing test exceeded 5 seconds under concurrent load |
| Isolated auth and Catalog route rerun | All 40 passed, including the timed-out case |
| Catalog PostgreSQL integration after adding atomic cleanup-marker coverage | All 5 passed |
| Frontend full suite | 635 passed |
| Chromium summary review, desktop and 390px mobile | Passed; no page errors or horizontal overflow |
| Configured real GLM 5.3 generation | Passed on synthetic PostgreSQL data |
| Backend/frontend production builds, modified Python lint, strict docs build | Passed |
The existing local Docker preview was rebuilt from this worktree, migration 003
was applied, and core/frontend were recreated with the existing persistent volumes.
The preview remains at `http://127.0.0.1:8080`.
The browser check uses the production widget in an isolated Vite fixture. It edits
the rule, declines the exemplar, submits only the selected card and checks responsive
layout. Gate tests separately verify request ordering through the production Pi
composition root; service and Catalog tests use real PostgreSQL. This is not a claim
of an automated complete live Pi conversation.
The L2 case retrieves an approved SQL rule, excludes a card bound to another database,
and asks the configured GLM 5.3 model to generate a query. Order IDs repeat between
financial years; the correct composite join returns 120 on the synthetic fixture.
The generated SQL is validated and executed in a read-only PostgreSQL transaction.
No real DWH rows are sent. Embeddings in this case are deterministic; the real
embedding/hybrid retrieval evidence remains documented in M2.
## Reproduction
From the harness:
```sh
THT_HOME=/private/tmp/thothii-m1-harness-home .venv/bin/pytest -q -m 'not l0 and not l2' --ignore=tests/test_portable_paths.py
.venv/bin/pytest -q tests/test_portable_paths.py
THT_HOME=/private/tmp/thothii-m1-harness-home .venv/bin/pytest -q tests/memory/test_administration.py tests/memory/test_retrieval.py -m 'not l2'
npm test
```
The optional generation case requires an installation YAML path and its running
core container. It resolves the model credential inside core without printing it:
```sh
THT_MEMORY_L2_INSTALLATION=<installation.yaml> THT_MEMORY_L2_CORE=<core-container> \
.venv/bin/pytest -q -s -m l2 tests/memory/test_administration.py -k real_model
```
From frontend: `npx playwright test e2e/memory-review.spec.ts`.
Screenshots are written to `/private/tmp/thothii-m3-summary-desktop.png` and
`/private/tmp/thothii-m3-summary-mobile.png`.
Evidence authoring and the joint X1 persistent Memory/Evidence conflict repair remain
outside M3. This increment does not infer knowledge from unexplained failures or
promise general improvements in SQL-generation accuracy.
@@ -3,6 +3,14 @@
**Stato:** design concordato con grill-with-docs il 2026-09-14. Questo worktree definisce e rende **Stato:** design concordato con grill-with-docs il 2026-09-14. Questo worktree definisce e rende
verificabile il percorso manuale; non introduce pacchetti nativi. verificabile il percorso manuale; non introduce pacchetti nativi.
**Aggiornamento:** il [PRD del 27 settembre](2026-09-27-guided-installation-prd.md)
rende le immagini precompilate su Docker Hub il percorso ordinario e include la
loro pubblicazione nel progetto. Il percorso con build da sorgente documentato qui
resta un'alternativa; il precedente rinvio di Docker Hub è superato.
La revisione del 28 settembre dello stesso PRD sostituisce inoltre il setup
interattivo con preparazione dei documenti, verifiche ripetibili ed esecuzione
senza richiesta di parametri.
## Obiettivo ## Obiettivo
Permettere di predisporre una copia di THothII su macOS, Windows e Linux partendo dal clone del Permettere di predisporre una copia di THothII su macOS, Windows e Linux partendo dal clone del
@@ -0,0 +1,386 @@
# PRD — Installazione di ThothII da documenti verificati
Creato: 2026-09-27. Revisione: 2026-09-28. Stato: requisiti e chiarimenti R1/R2
approvati; `grill-with-docs` e `/to-spec` conclusi. Specifica pubblicata su Gitea
con etichetta `ready-for-agent`; implementazione non iniziata.
Branch: `codex/guided-standalone-install`.
La [specifica derivata](2026-09-28-document-first-installation-spec.md) raccoglie
user story, decisioni implementative e collaudi. Il piano di test è stato confermato
dall'utente e la specifica è pubblicata nell'issue
[Spec: installazione da documenti verificati e distribuzione Docker Hub](https://git.tylconsulting.it/mptyl/ThothII/issues/42).
La [scomposizione approvata](2026-09-28-installation-ticket-breakdown.md) è pubblicata
nelle issue #43–#54 con dipendenze native. Il prossimo passaggio è `/implement`
sui ticket senza blocchi, inizialmente validazione workspace e rilascio Docker Hub.
Questo documento consolida le decisioni della
[ripresa del progetto](2026-09-27-guided-installation-resumption.md) e aggiorna il
[piano standalone del 14 settembre](2026-09-14-manual-standalone-installation.md)
per l'esperienza guidata. Conserva architettura, protezione dei segreti e contratti
del prodotto; modifica il percorso operativo e l'ordine dei collaudi. La successiva
precisazione dell'utente rende la distribuzione di immagini precompilate su Docker Hub
parte della prima versione del percorso ordinario, superando il precedente rinvio.
La revisione del 28 settembre sostituisce la raccolta interattiva dei parametri:
si preparano prima i documenti, li si verifica anche più volte, poi si esegue il setup.
Prevale sulle precedenti decisioni I1/I2 dove consentivano configurazioni obbligatorie
rinviate o richieste durante l'installazione.
## Obiettivo e destinatario
Una persona capace di installare Docker, clonare un repository e fornire le proprie
credenziali deve poter predisporre con calma i documenti necessari e rendere
utilizzabile almeno un workspace, senza conoscere l'architettura interna.
Template commentati, esempi compilati e documentazione passo per passo spiegano
cosa inserire nei file YAML e nei file protetti `.env` o equivalenti.
DWH e provider LLM possono essere esterni; non si promette un funzionamento offline.
La CLI offre preparazione dei template, verifiche ripetibili e applicazione dei
documenti già verificati. Il setup non raccoglie parametri, non apre questionari
e non completa silenziosamente documenti incompleti. L'utente corregge i documenti
prima di eseguirlo. Le pagine amministrative rimangono disponibili per l'uso e la
manutenzione successivi, senza diventare una scorciatoia per rinviare parametri
obbligatori dell'installazione.
Il percorso predefinito scarica da Docker Hub le immagini applicative già compilate:
il computer dell'utente svolge configurazione, inizializzazione dei servizi e dei
database, avvio e verifiche. L'installazione da sorgente resta disponibile come
scelta esplicita alternativa. Nessuna compilazione dell'applicazione, neppure
all'interno di un container locale, è richiesta dal percorso ordinario.
## Decisioni approvate
| Area | Comportamento richiesto |
| --- | --- |
| Distribuzione predefinita | Creare, collaudare e pubblicare su Docker Hub le immagini applicative precompilate; il setup le scarica e le avvia senza build locale. |
| Alternativa da sorgente | Conservare un percorso esplicito di build dai sorgenti, documentato e verificato, con configurazione e funzionalità equivalenti. |
| Due traguardi | Mostrare separatamente piattaforma installata e workspace pronto. |
| Preparazione anticipata | Repository workspace e documenti applicativi predisposti prima dell'esecuzione; template, esempi e guida alla compilazione. |
| Setup senza domande | Consuma documenti già verificati, mostra avanzamento ed errori e non chiede valori mancanti. |
| Modelli | Provider, modelli, usi, endpoint e riferimenti alle credenziali descritti nei documenti prima del setup; configurazioni di esempio supportate e commentate. |
| Embedding | Configurazione locale precompilata come percorso ordinario. |
| Ripresa | Correggere i documenti, ripetere le verifiche e riprendere l'esecuzione senza questionari né perdita del lavoro completato. |
| Workspace | Preparare prima repository e descriptor conformi; dichiarare separatamente i parametri di collegamento ai database secondo i contratti ThothII. |
| Contenuti | Riutilizzare descrizioni/Evidence curate; nessuna generazione AI implicita per completare un documento. Eventuali attività di curation restano esplicite. |
| Evidence | Opzionali nel contratto; se configurate devono essere valide e utilizzabili. |
| Collaudi | Prima Windows, poi Omarchy su PC Intel, infine macOS, in tre tappe distinte. |
## Rilascio e distribuzione delle immagini
La creazione e pubblicazione delle immagini appartengono al processo di rilascio
del progetto. Il sottoprogetto installazione comprende quindi anche una procedura
riproducibile per costruire, verificare e pubblicare queste immagini su Docker Hub;
non basta aggiungere un'opzione di pull senza fornire immagini utilizzabili.
**Stato iniziale dichiarato dall'utente il 28 settembre:** le immagini applicative
ThothII su Docker Hub non sono disponibili. Prima di collaudare l'installazione
precompilata su un PC Windows o Linux occorre produrle, pubblicarle e verificarne
il download. Questa dipendenza è bloccante per quel collaudo, non viene aggirata
usando immagini costruite soltanto nella cache della macchina di test.
Il progetto deve fornire un comando di produzione e pubblicazione, mantenuto nel
repository e documentato per il manutentore. Il comando riceve revisione/versione,
namespace Docker Hub e architetture, costruisce e verifica le immagini proprietarie,
le pubblica e produce il manifest di rilascio con digest e artefatti compatibili.
Il nome e la sintassi saranno fissati nella specifica; il comando non è ancora
implementato. Le credenziali di pubblicazione appartengono al manutentore e non
entrano nei documenti dell'utente che installerà ThothII.
La sequenza di rilascio è: produzione e controlli, pubblicazione su Docker Hub,
verifica del pull del rilascio pubblicato, quindi collaudo della procedura sui
sistemi destinatari. Questa fase del manutentore precede i sei passi dell'utente.
Il pacchetto distribuito deve comprendere tutto ciò che serve all'installazione:
immagini applicative, manifest Compose/configurazione di avvio, migrazioni e risorse
di inizializzazione, oltre al comando operatore o al suo bootstrap. Il numero delle
immagini segue i servizi dell'architettura corrente: non è richiesto accorpare tutto
in un singolo container. I servizi di terze parti mantengono immagini compatibili
con lo stack, senza ricostruirli sul computer dell'utente.
Nell'architettura corrente le immagini proprietarie da pubblicare sono `core` e
`frontend`; PostgreSQL, Qdrant e Ollama usano immagini upstream. Le attività
`catalog-migrate` e `workspace-maintenance` riusano la stessa immagine release di
`core`. Il pacchetto di installazione include anche le risorse oggi montate dal
checkout, fra cui `docker/catalog-db-init.sql` e `docker/embedding-model-init.sh`.
Il download del modello embedding, la creazione dei volumi/database e le migrazioni
restano attività di inizializzazione locale, distinte dalla compilazione.
Ogni rilascio identifica una versione coerente di immagini, CLI e configurazione;
il manifest registra riferimenti verificabili, inclusi i digest delle immagini.
Il setup riporta la versione installata e non combina automaticamente componenti
incompatibili tramite tag mobili. Nomi e namespace Docker Hub saranno definiti
nella specifica di pubblicazione; non sono presunti già esistenti.
Il percorso ordinario deve poter partire dal pacchetto di rilascio senza richiedere
il checkout dei sorgenti applicativi o strumenti di compilazione. Anche l'eventuale
CLI nativa deve essere distribuita già compilata per gli host supportati; nascondere
una compilazione di `tht` nel bootstrap non soddisfa il requisito.
L'installazione pubblica non richiede credenziali di pubblicazione Docker Hub.
Le immagini non includono credenziali dell'installazione, dati personali, workspace
dell'operatore o i database di esempio preinstallati. Configurazione e dati
persistenti vengono creati localmente nei percorsi e volumi dell'installazione.
Il supporto iniziale Windows/WSL2 e Omarchy richiede immagini Linux amd64; per la
tappa macOS Apple Silicon servono immagini Linux arm64 e un bootstrap host adeguato.
La pubblicazione dichiara solo le architetture effettivamente verificate, rispettando
l'ordine dei collaudi concordato.
La modalità sorgente costruisce gli stessi componenti a partire da una revisione
esplicita, documenta i prerequisiti aggiuntivi e usa gli stessi contratti di
configurazione, persistenza e migrazione. Un errore di download da Docker Hub non
deve attivarla automaticamente: l'utente può correggere il problema e riprovare,
oppure scegliere consapevolmente l'alternativa da sorgente.
## Percorso dell'utente
Prima dei sei passi sono disponibili guida, template e strumenti di verifica già
compilati. Git e gli strumenti minimi necessari per acquisire i documenti sono
esplicitati nella guida; la verifica completa dell'host rimane al passo 3. I primi
controlli documentali non devono richiedere l'avvio di ThothII o dei suoi container.
### 1. Preparazione del repository dei workspace
L'utente prepara una copia del repository predefinito con Financial, European
Football e F1, oppure un repository ad hoc a partire da un template documentato.
Il repository predefinito contiene definizioni, documentazione, Evidence e riferimenti
ai pacchetti dati; il clone non equivale ad aver già creato i database PostgreSQL.
La disponibilità effettiva del percorso predefinito dipende dal sottoprogetto esempi.
Si preservano le scelte già approvate sulla copia autonoma o sull'accesso in sola
lettura all'originale; workspace ed Evidence locali restano modificabili. La
preparazione non richiede diritti di push al repository pubblico e non sostituisce
un repository già configurato senza una scelta esplicita.
### 2. Verifica dei documenti dei workspace
Un comando dedicato verifica sintassi YAML, versione/schema ThothII, campi richiesti,
tipi, identificatori, unicità, corrispondenza fra catalogo e directory, descriptor
referenziati, percorsi e file Evidence dove configurati. La verifica è ripetibile
sui file locali prima che Docker o ThothII siano in esecuzione.
La verifica sostanziale copre ciò che è dimostrabile dai documenti: coerenza dei
riferimenti, esistenza e leggibilità dei contenuti, conformità delle Evidence e
assenza di contraddizioni rilevabili. Non pretende di certificare automaticamente
la verità delle regole di dominio o interrogare database non ancora creati.
Ogni errore identifica documento, campo e, quando disponibile, riga, con indicazione
della correzione. L'utente modifica i documenti e ripete il controllo. Le Evidence
restano opzionali: assenza dichiarata ed Evidence configurate ma invalide sono
condizioni diverse. I controlli incrociati che richiedono i parametri applicativi
vengono completati al passo 5.
### 3. Verifica delle precondizioni
Verificare sistema/architettura, Docker e Compose, accesso al daemon, risorse e spazio
richiesti, percorsi e permessi, porte previste, accesso ai servizi esterni e al registry
per quanto valutabile. Su Windows verificare Ubuntu WSL2 e integrazione Docker
Desktop. I requisiti dipendenti da valori scelti al passo 4 sono ricontrollati al 5.
Distinguere componenti necessari sull'host da componenti inclusi nelle immagini:
Pi appartiene a `core`, non è un prerequisito da installare separatamente sul PC.
Presenza e versione di Pi sono verificate nel rilascio; il funzionamento effettivo
nel container è verificato al passo 6. Lo stesso principio vale per le dipendenze
applicative già incluse. Go, Python, Node e compilatori non sono prerequisiti host
del percorso precompilato.
### 4. Preparazione dei parametri applicativi
L'utente compila i documenti locali usando template commentati ed esempi: descriptor
di installazione, configurazione dei modelli e riferimenti ai file protetti `.env`
o equivalenti. Sono espliciti campi obbligatori, opzionali, default e condizioni in
cui un parametro serve. La preparazione può svolgersi in più sessioni senza avviare
il setup o i servizi applicativi.
I documenti definiscono versione del rilascio, percorsi/volumi, profilo e accesso,
repository/workspace selezionati, provider/modelli per i rispettivi usi, endpoint,
embedding, parametri di collegamento ai database e riferimenti ai segreti. La
configurazione dei modelli deriva dall'Installation Model Catalog, senza un secondo
catalogo del setup. L'embedding locale ha un esempio precompilato.
Il descriptor workspace v4 continua a contenere identità ed Evidence: non vi si
inseriscono campi database o modelli estranei al contratto. I binding database sono
predisposti in un input locale separato, da specificare, e applicati al Metadata
Catalog durante il setup mediante i suoi servizi; non diventano una seconda autorità
runtime. La configurazione server/Omics rimane fuori dal perimetro.
I segreti non entrano nel repository pubblico, nei log o nei rapporti di verifica.
Per le credenziali tecniche interne un comando preparatorio può generare file
protetti prima delle verifiche, senza questionario né richiesta durante il setup.
Le selezioni degli esempi e le eventuali operazioni facoltative sono dichiarate
prima dell'esecuzione. Le pagine amministrative restano disponibili dopo l'avvio
per modifiche e curation, non per raccogliere valori obbligatori dimenticati.
### 5. Verifica dei parametri applicativi
Un comando ripetibile controlla completezza e correttezza dei documenti, sintassi
e compatibilità dei valori, riferimenti ai segreti, modelli/usi/default, collegamenti
workspace/database, configurazione Compose, disponibilità del rilascio nel registry
e compatibilità delle architetture. Completa i controlli incrociati dei passi 2 e 3.
Quando fattibile senza creare lo stack, verifica raggiungibilità, autenticazione
e compatibilità delle dipendenze esterne con operazioni circoscritte; la documentazione
spiega le prove effettuate, inclusi eventuali accessi a provider a consumo.
Non cambia dati applicativi né esegue migrazioni o importazioni.
Il rapporto distingue `superato`, `errore`, `avviso` e `non ancora verificabile`,
senza trasformare assenza di verifica in successo. I controlli che richiedono lo
stack sono elencati prima e obbligatori al passo 6, secondo R2 approvata.
Un errore formale o una configurazione obbligatoria mancante blocca l'esecuzione.
Il risultato identifica i documenti e la revisione del repository esaminati. Una
modifica successiva invalida i controlli dipendenti: il setup non deve applicare
file diversi da quelli verificati senza ricontrollarli. Segreti e loro valori
non vengono copiati nel rapporto.
### 6. Setup esecutivo
Il setup ricontrolla l'ammissibilità del piano verificato, scarica le immagini del
rilascio da Docker Hub, crea reti/volumi/container e inizializza i database applicativi.
Esegue migrazioni, applicazione della configurazione e dei binding, sincronizzazione
e preparazione necessaria dei workspace secondo i contratti esistenti. Quando
disponibili e selezionati, crea e carica i database di esempio dal relativo pacchetto.
Non pone domande su provider, modelli, percorsi, credenziali o altri parametri.
Se trova un valore mancante o incoerente, si ferma con una diagnosi e rimanda alla
correzione dei documenti e alla loro verifica; non apre un wizard di riparazione.
Le conferme dei contratti di dominio non vengono aggirate: la specifica deve
distinguere operazioni predisponibili nel piano da attività umane residue senza
reinserire la raccolta dei parametri durante l'installazione.
Esegue i controlli disponibili solo a runtime: salute dei servizi, Pi, modello
embedding effettivamente caricato, collegamenti dalla rete dei container, Catalog
e preparazione dei workspace. Mostra separatamente piattaforma avviata e workspace
pronto, più eventuali verifiche funzionali ancora da svolgere. Il collaudo completo
include una domanda reale con revisione umana, senza SQL target di benchmark.
## Documentazione di accompagnamento
Le guide italiana e inglese seguono esattamente i sei passi. Per ciascuno indicano
input, file da predisporre, template ed esempio compilato, comando di verifica,
esito atteso, errori comuni e passaggio successivo. Un elenco dei documenti e dei
segreti necessari consente di raccogliere le informazioni in anticipo.
Le istruzioni distinguono manutentore del rilascio e utente installatore, host e
container, controlli locali e runtime. Il percorso sorgente è esplicitamente
alternativo; un download fallito non ne provoca l'attivazione automatica.
## Avanzamento, errori e ripresa
La procedura conserva l'installazione di riferimento, gli input verificati, i
passaggi completati, l'ultimo errore e il prossimo passo, senza conservare segreti
nello stato di avanzamento. Alla ripresa verifica lo stato reale: una vecchia spunta
non prova che servizio, credenziale o indice siano ancora validi.
Correggere un endpoint o una credenziale non impone di rifare tutto il setup. La
specifica deve definire quali verifiche dipendenti vanno ripetute, preservando
configurazioni, workspace, sessioni e contenuti non coinvolti nella modifica.
Le modifiche manuali vengono riconosciute e spiegate, non cancellate implicitamente.
La ripresa riguarda le tappe della procedura: non promette una ripresa interna
di operazioni che non la supportano, come il preprocessing corrente. In questi
casi il passaggio viene rieseguito in modo coerente con il suo contratto.
Un fallimento riporta fase, causa comprensibile, correzione consigliata e modalità
di ripresa. Distinguere problemi dell'host, dei servizi locali e delle dipendenze
esterne. Un provider o DWH indisponibile non annulla una verifica valida della
piattaforma, ma impedisce di dichiarare il percorso complessivo pronto.
## Criteri di accettazione
| Scenario | Esito verificabile |
| --- | --- |
| Rilascio Docker Hub | Immagini costruite e pubblicate con versione, digest e architetture dichiarate; avvio verificato usando quanto effettivamente scaricato dal registry. |
| Installazione precompilata | Su host senza toolchain e senza sorgenti applicativi, il setup scarica le immagini e completa configurazione/avvio senza compilare né invocare build locali. |
| Risorse di avvio | Compose, migrazioni e inizializzazione non dipendono da file presenti soltanto in un checkout dei sorgenti. |
| Download fallito | Errore comprensibile e ripetibile; nessuna build da sorgente avviata implicitamente. |
| Modalità sorgente | Scelta esplicita ancora funzionante, con gli stessi contratti di configurazione e persistenza del rilascio precompilato. |
| Preparazione documentale | Template e guida consentono di predisporre tutti i YAML e file protetti necessari prima dell'esecuzione. |
| Ordine dei passi | Repository, verifica workspace, precondizioni, parametri applicativi, verifica parametri, setup esecutivo. |
| Verifica workspace senza stack | Il passo 2 funziona senza Docker attivo o applicazione installata e senza toolchain host aggiuntive. |
| Controlli ripetibili | Ripetere i controlli non crea container, non migra DB e non modifica i documenti dell'utente. |
| Completezza prima del setup | Un campo obbligatorio mancante blocca l'esecuzione, indicando file/campo e correzione; nessuna domanda interattiva. |
| Input modificati | I controlli dipendenti sono invalidati o ripetuti; nessuna applicazione di input diversi da quelli verificati. |
| Setup non interattivo | Con documenti completi il passo 6 termina senza richiedere input da terminale; un errore non apre un questionario. |
| Verifiche sostanziali | Il rapporto distingue prove eseguite da controlli non ancora possibili; non certifica verità di dominio o servizi non verificati. |
| Modello configurato | Credenziali/endpoint verificati e selezione valida per gli usi richiesti. |
| Credenziale errata | Diagnosi senza esporre il segreto, correzione nel file protetto e nuova verifica prima della ripresa. |
| Interruzione e riavvio | La procedura ricontrolla gli input e riprende dall'avanzamento verificato senza nuove domande sui parametri. |
| Contenuti esistenti | Descrizioni curate ed Evidence locali preservate; nessuna generazione o sovrascrittura silenziosa. |
| Evidence assenti | Nessun blocco dovuto alla sola assenza quando non sono configurate. |
| Workspace pronto | Connessione, schema e preparazione necessaria verificati; domanda reale completata con revisione umana. |
| Riesecuzione | Nessuna duplicazione, azzeramento di dati o perdita delle personalizzazioni. |
| Stop/start | Accesso e workspace utilizzabile persistono; i problemi esterni sono segnalati separatamente. |
| Segreti | Assenti da log, riepiloghi, file pubblici e stato di avanzamento. |
Il dettaglio dei controlli e i test automatici devono rispettare le API e i contratti
attuali; i test di portabilità e il collaudo con servizi reali restano distinti.
La prima tappa usa Windows x64/WSL2, la seconda Linux Omarchy x64, la terza macOS
Apple Silicon, coerentemente con le architetture già previste dal progetto.
Gli esiti di una tappa non valgono come collaudo delle successive.
## Sottoprogetto degli esempi
L'implementazione rimane rinviata, come confermato con R1. I requisiti sono conservati sul branch
`codex/benchmark-examples`, commit `08a5db55`. Il setup non offre come disponibili
CLI, database o modalità di copia del repository non ancora implementati.
Il percorso richiesto ora parte dal repository predefinito oppure da quello ad hoc.
Il primo collaudo utilizza un repository ad hoc; il percorso predefinito completo
arriverà con il sottoprogetto esempi. Quando disponibili e selezionati nei documenti,
creazione e caricamento dei database avvengono nell'installazione
dell'utente a partire dai pacchetti dati verificati, senza compilare l'applicazione
o incorporare quei database nelle immagini. Il percorso ad hoc può essere collaudato
con workspace/database disponibili. La disponibilità di immagini Docker Hub è
invece un prerequisito esplicito di ogni collaudo del percorso precompilato.
## Confini della specifica successiva
La verifica del codice ha individuato questi punti di integrazione concreti:
- `compose.yaml` costruisce oggi `core` e `frontend`; i servizi di manutenzione
usano l'immagine core locale. `setup --complete` esegue una build
(`tools/tht/internal/setup/run.go:77`) e `scripts/install-tht.sh:84` compila la CLI
tramite Docker se non riceve un artefatto già costruito. Il percorso ordinario
deve sostituire entrambi i comportamenti con artefatti del rilascio. La CI
`.github/workflows/container-multiarch.yml` verifica già entrambe le architetture
Linux; occorre aggiungere la pubblicazione Docker Hub e la verifica dei pacchetti
effettivamente distribuiti.
- I parser workspace (`backend/src/workspaces/catalog.ts:52`, `schema.ts:382`)
verificano i documenti senza dipendenze dal runtime, ma non sono oggi un comando
preinstallazione nativo. Devono essere resi disponibili negli strumenti
precompilati preservando gli stessi contratti, senza richiedere Node sul PC.
- La configurazione database è applicata al descriptor runtime dal Catalog
(`backend/src/catalog/runtime-binding.ts:33`). Serve un input locale preparatorio
e un percorso di applicazione al Catalog, senza cambiare il significato del
workspace v4 o creare due autorità persistenti per i binding.
- `config.Load` contiene verifiche riutilizzabili su YAML e file ambiente
(`tools/tht/internal/config/installation.go:100`); il `doctor` corrente combina
controlli documentali e runtime (`tools/tht/internal/doctor/report.go:97`).
La specifica deve separarli per rendere disponibili i gate dei passi 2, 3 e 5.
- L'ammissione della sessione controlla modello, preprocessing e servizi necessari
(`backend/src/routes/sessions.ts:438`); il preprocessing richiede un database
associato e una sincronizzazione corrente. Le descrizioni generate dall'AI
non costituiscono un requisito generale: possono essere già disponibili
descrizioni curate o commenti della sorgente. Riferimento:
[contratto di preprocessing](../contracts/workspace-preprocessing-cli.md).
La specifica tecnica dovrà definire build/pubblicazione Docker Hub, pacchetto di
rilascio e bootstrap precompilato, selezione esplicita del percorso sorgente,
template e documenti locali, validatori senza runtime, applicazione dei parametri
al Catalog, esecuzione senza questionari, stato/ripresa e verifiche dei due traguardi.
Nomi dei comandi di preparazione/verifica/pubblicazione e formato dello stato sono
dettagli da progettare, non funzionalità esistenti attestate da questo PRD.
## Chiarimenti approvati del 28 settembre
| ID | Scelta | Decisione approvata |
| --- | --- | --- |
| R1 | Disponibilità dei tre esempi al primo rilascio | Conservare il rinvio del sottoprogetto e collaudare prima il percorso con repository ad hoc; introdurre il percorso predefinito completo quando gli esempi sono disponibili. |
| R2 | Controlli impossibili prima della creazione dello stack | Bloccare gli errori rilevabili prima; elencare i controlli runtime non ancora eseguibili e renderli obbligatori al passo 6, senza contarli come superati preventivamente. |
Approvazione: «ok per le tue proposte. dopodichè procedi con to-spec».
Il principio dei sei passi resta invariato.
Non rientrano in questo lavoro un nuovo installer grafico, la riscrittura delle
superfici amministrative, il supporto multi-repository, la distribuzione dei
database di esempio o modifiche al deployment server/Omics.
@@ -0,0 +1,155 @@
# Ripresa dell'installazione guidata
Stato al 28 settembre: revisione documentale in sei passi richiesta dall'utente;
raccolta interattiva dei parametri durante il setup superata. R1/R2 confermate,
`grill-with-docs` e `/to-spec` conclusi, piano di test confermato dall'utente.
Il riferimento consolidato è il
[PRD dell'installazione guidata](2026-09-27-guided-installation-prd.md).
La [specifica derivata](2026-09-28-document-first-installation-spec.md) è pubblicata
su Gitea come
[Spec: installazione da documenti verificati e distribuzione Docker Hub](https://git.tylconsulting.it/mptyl/ThothII/issues/42),
con etichetta `ready-for-agent`. `/to-tickets` è completato: la
[scomposizione approvata](2026-09-28-installation-ticket-breakdown.md) collega le
issue #43–#54, con dipendenze native verificate e parent invariata.
Prossimo passo: `/implement` su #43 o #46, inizialmente senza blocchi.
Nessun codice applicativo è stato implementato in questi passaggi.
## Base di lavoro
Branch `codex/guided-standalone-install`, commit iniziale `67ee5262`.
Il branch conserva il setup guidato e i controlli workspace incompleti, separati dal
rilascio server. Il piano del 14 settembre descrive il precedente percorso manuale;
le decisioni qui registrate aggiornano l'obiettivo del lavoro ripreso.
## Decisioni confermate il 27 settembre 2026
- Destinatario: una persona capace di installare Docker, clonare un repository e
compilare i valori richiesti, senza conoscere l'architettura di ThothII.
- Risultato: almeno un workspace utilizzabile per una domanda reale, attraverso due
traguardi espliciti: piattaforma installata e workspace pronto. La procedura guida
le decisioni umane necessarie e può essere ripresa senza ricominciare.
- Collaudo in tre tappe distinte e ordinate: prima Windows, poi Linux Omarchy su PC
Intel, infine macOS. Non attribuire a una piattaforma gli esiti ottenuti su un'altra.
- Distribuzione ordinaria tramite immagini applicative precompilate pubblicate su
Docker Hub; build e pubblicazione fanno parte del progetto. L'installazione locale
configura e inizializza lo stack senza compilare; l'alternativa da sorgente resta
esplicitamente disponibile. Anche il comando host deve essere fornito precompilato
nel percorso ordinario. La creazione/caricamento degli esempi resta un'integrazione
prevista, con l'implementazione del relativo sottoprogetto ancora rinviata.
## Vincoli già documentati
Il precedente piano prevedeva clone Gitea e build locale; la decisione successiva
li mantiene come alternativa al percorso precompilato Docker Hub. Restano Docker
Compose e, su Windows, Ubuntu WSL2 con integrazione Docker Desktop.
Il workspace descriptor contiene identità ed
Evidence; configurazione database e metadati appartengono al Metadata Catalog.
L'Installation Model Catalog appartiene all'installazione.
## Revisione del 28 settembre — prevale sul percorso interattivo
La preparazione e le verifiche precedono l'esecuzione, nell'ordine richiesto:
1. Preparare il repository workspace predefinito con i tre esempi oppure uno ad hoc.
2. Verificare formalmente e, per quanto possibile, sostanzialmente i documenti workspace.
3. Verificare le precondizioni dell'host e dei componenti previsti.
4. Predisporre i documenti locali YAML/.env con i parametri dell'applicazione.
5. Verificarne completezza, correttezza e coerenza con i workspace.
6. Eseguire il setup dai documenti verificati, scaricando le immagini Docker Hub
e creando lo stack, senza domande sui parametri.
Template ed esempi commentati e guide IT/EN accompagnano ogni fase. Le verifiche
devono essere ripetibili prima di creare i container. Pi è incluso in `core`, non
deve essere installato sull'host; i suoi controlli runtime avvengono dopo l'avvio.
I binding database restano di competenza del Catalog, con un input locale
preparatorio distinto dai descriptor workspace v4.
L'utente conferma che le immagini Docker Hub oggi non esistono: occorre un comando
di produzione/pubblicazione per il manutentore, poi verificare il pull degli
artefatti pubblicati prima di collaudare l'installazione precompilata sui PC.
La preparazione del rilascio non è un compito dell'utente installatore.
R1/R2 confermate: esempi ancora rinviati e primo collaudo con repository ad hoc;
controlli runtime elencati prima ed eseguiti obbligatoriamente dopo la creazione
dello stack. Nessun controllo non eseguito conta come superato.
## Aspetti da tradurre nella specifica tecnica
- Quali template e documenti locali preparare, verificare e applicare, senza
raccogliere parametri durante l'esecuzione.
- Come distinguere validazione documentale, controlli preventivi esterni e controlli
runtime, mantenendo i contratti delle superfici amministrative esistenti.
- Criteri dettagliati di verifica, ripresa dopo errori e accettazione per ogni tappa.
## Sottoprogetto esempi: requisiti definiti, implementazione rinviata
Su richiesta dell'utente l'implementazione dei database di esempio viene rinviata;
la definizione dei requisiti dell'installazione prosegue indipendentemente.
Branch dedicato: `codex/benchmark-examples`, creato da `67ee5262`, con documentazione
consolidata nel commit `08a5db55`. Il PRD approvato è
`docs/plans/2026-09-27-example-databases-prd.md` su quel branch.
Il PRD conserva D1–D8: Financial, European Football e F1, tre workspace separati,
dati PostgreSQL e schema commentato, Evidence curate e domande di accompagnamento
senza SQL target, contenuti italiano/inglese. CLI scaricabile dal repository pubblico
degli esempi Gitea gestito da TYL Consulting, collegato dal repository pubblico
ThothII. Selezione di uno, due o tre esempi dopo il setup, oppure come ultimo passo
facoltativo dello stesso setup; pacchetti PostgreSQL già verificati ove redistribuibili.
La copia del repository potrà essere indipendente, senza storia e collegamenti Git
all'originale, oppure scaricata con accesso al repository pubblico in sola lettura.
Workspace ed Evidence locali restano modificabili. Preparazione e verifiche sono
responsabilità del progetto; all'utente vengono sottoposte solo ambiguità non
risolvibili dalle fonti. Chi installa dovrà trovare gli esempi pronti all'uso.
Il caricatore, il supporto alla cartella `examples/` e alla copia autonoma sono da
implementare. Finché il sottoprogetto è rinviato, il setup non deve offrirli come
funzionalità disponibili. Il requisito di un workspace utilizzabile resta valido:
per il collaudo si dovrà usare un workspace/database realmente disponibile.
## Verifiche da completare
Riesecuzione e fallimenti intermedi del setup; requisiti HTTPS per il repository
workspace; test dedicati dei nuovi comandi; distinzione fra stato della piattaforma
e Workspace Readiness; collaudo reale completo secondo l'ordine concordato.
## Riscontro sul setup corrente
- CLI guidata e pagine amministrative esistenti sono già il percorso previsto dal
piano del 14 settembre; non è richiesto un nuovo installer grafico.
- `setup --complete` prepara autenticazione e modelli, build, migrazioni Catalog,
avvio, workspace pull, test Pi e doctor. Non configura binding DB, sincronizzazione
dei metadati e preprocessing (`tools/tht/internal/setup/run.go:60`). Il messaggio
finale corrente «ThothII is ready» deve essere allineato al traguardo verificato.
- I modelli sono oggi preimpostati, senza selettore del provider nel Request
(`tools/tht/internal/setup/files.go:449`).
- La riesecuzione rifiuta file di configurazione esistenti con contenuto diverso;
non equivale ancora a una ripresa guidata delle tappe tecniche e umane
(`tools/tht/internal/setup/files.go:107`). La specifica deve prevedere stato delle
tappe e gestione delle correzioni, senza sovrascrivere personalizzazioni.
## Round installazione I1–I3 del 27 settembre — storico superato dove indicato
La revisione del 28 settembre sopra prevale su I1/I2: nessun rinvio di parametri
obbligatori al setup e nessuna raccolta tramite questionario. Il testo seguente
conserva il contesto della decisione precedente e non è il comportamento richiesto
per la nuova procedura. I3 resta valido per riuso dei contenuti e curation esplicita.
L'utente conferma «tutto come da te suggerito», dopo il chiarimento sulla CLI locale
interattiva: domande condizionate alle risposte, configurazioni precompilate,
credenziali protette, verifica delle connessioni, possibilità di rinviare una
configurazione e ripresa senza ricominciare. La CLI guida alle pagine amministrative
esistenti per il workspace e ne verifica il completamento.
| ID | Decisione | Scelta approvata |
| --- | --- | --- |
| I1 | Primo avvio senza repository/workspace disponibile | Consentire di completare il solo traguardo «piattaforma installata» e riprendere in seguito la configurazione del workspace. Il percorso complessivo resta incompleto fino al primo workspace utilizzabile e alla domanda reale. Nessuna dipendenza dalla futura disponibilità degli esempi. |
| I2 | Scelta dei modelli durante il setup | Selezione guidata di provider e modello tra configurazioni supportate/precompilate, chiedendo le credenziali necessarie; percorso avanzato per configurazioni personalizzate. Embedding locale preconfigurato come scelta iniziale. |
| I3 | Preparazione di descrizioni ed Evidence | Riutilizzare i contenuti già curati. Proporre la generazione AI delle descrizioni mancanti come scelta esplicita, con revisione umana, invece di avviarla automaticamente. Guidare alle pagine amministrative necessarie e registrare il punto di ripresa. |
Le Evidence sono opzionali nel contratto del workspace: non introdurre un obbligo
generale di crearle per completare l'installazione. La specifica deve rispettare
i controlli di Workspace Readiness su connessione, schema e indicizzazione,
distinguendo assenza lecita di Evidence da configurazione incompleta o incoerente.
Per le decisioni correnti e i criteri di accettazione fa fede il PRD revisionato
al 28 settembre, senza attestare che siano già implementati.
@@ -0,0 +1,387 @@
# Spec: installation from validated documents and Docker Hub releases
## Problem Statement
An operator who understands Docker and can edit a documented configuration should
not have to discover ThothII's architecture while answering an installation wizard.
The operator needs time to prepare workspace and application documents, validate
them repeatedly, and correct errors before applying changes to the machine.
The current setup combines document generation, local builds, startup and runtime
diagnostics. Its success message can precede an actually usable workspace. It does
not provide the complete preinstallation validation and resumable, non-interactive
execution required by this workflow.
The ordinary installation must consume published, prebuilt application images.
As reported by the project owner on 28 September 2026, ThothII images are not yet
available on Docker Hub. Publishing and verifying a real release is therefore a
prerequisite for testing the consumer installation, rather than a later enhancement.
## Solution
Deliver a documented six-step workflow, in this exact order:
1. Prepare a workspace repository: a project-specific repository initially, or the
default examples repository when that separately deferred project is available.
2. Validate workspace documents formally and substantively to the extent possible
from their contents and available references.
3. Verify host and distribution prerequisites.
4. Prepare application parameters in installation-local YAML and protected
environment/secret documents, using commented templates and complete examples.
5. Validate application parameters and their consistency with the selected workspaces.
6. Execute the validated installation without asking configuration questions: pull
the release images, create and initialize the stack, apply declared configuration,
and run the required runtime checks.
Before these consumer steps can be tested against Docker Hub, a maintainer command
must build, check and publish the release and verify that its artifacts can be pulled.
Local source builds remain an explicit alternative, with the same configuration and
persistence contracts. A failed pull never silently switches to source compilation.
Checks that cannot run before container or database creation are enumerated as
deferred obligations and must run after startup. Errors detectable beforehand block
execution. Platform acceptance, Workspace Readiness and functional acceptance are
reported separately. A real question with human review completes functional acceptance.
## User Stories
1. As an installer, I want a six-step guide, so that I can understand the whole process before changing my machine.
2. As an installer, I want commented templates and completed examples, so that I can prepare documents without knowing internal component names.
3. As an installer, I want to pause document preparation, so that I can obtain missing information without restarting an installer.
4. As an installer, I want to prepare a project-specific workspace repository, so that I can use my own database before the example databases are delivered.
5. As an installer, I want the future default repository to be clearly distinguished from available features, so that I am not directed to unavailable examples.
6. As an installer, I want read-only access to the source workspace repository, so that consuming a workspace does not require publication rights.
7. As a workspace author, I want local document validation before Docker starts, so that syntax and contract errors are inexpensive to correct.
8. As a workspace author, I want duplicate identifiers, unsupported fields and inconsistent references reported, so that formally valid YAML does not hide an invalid workspace.
9. As a workspace author, I want errors to identify the document and field, so that I know precisely what to edit.
10. As a workspace author, I want optional Evidence distinguished from invalid configured Evidence, so that an intentionally absent corpus does not block installation.
11. As a workspace author, I want semantic validation limits stated honestly, so that successful validation is not mistaken for certification of domain knowledge.
12. As an installer, I want host prerequisites checked explicitly, so that Docker, architecture, permissions and resource problems are detected before setup.
13. As a Windows installer, I want a verified WSL2 and Docker Desktop path, so that I can follow one supported installation procedure.
14. As an installer, I want Pi supplied with the application image, so that I do not have to install an unnecessary host dependency.
15. As an installer, I want application models, endpoints and credentials prepared before execution, so that I can consult colleagues or provider documentation at my own pace.
16. As an installer, I want one authored model catalog, so that provider and embedding configuration do not disagree across components.
17. As an installer, I want database bindings prepared locally and separately from workspace definitions, so that environment-specific details do not leak into shared workspace repositories.
18. As an installer, I want generated internal credentials prepared in protected files, so that I need not invent technical passwords during execution.
19. As an installer, I want my secrets excluded from logs and validation reports, so that diagnostic output is safe to inspect and share.
20. As an installer, I want repeatable application validation, so that I can correct configuration without creating containers or changing databases.
21. As an installer, I want available external connections checked before startup, so that preventable endpoint or authentication errors are found early.
22. As an installer, I want non-executable checks listed explicitly, so that I know what still needs to be proved after startup.
23. As an installer, I want validation tied to the documents and release I selected, so that execution cannot silently apply different inputs.
24. As an installer, I want setup to run with closed standard input, so that it cannot unexpectedly ask me for a parameter.
25. As an installer, I want missing values to stop setup with a useful diagnosis, so that I can correct the document and validate again.
26. As an installer, I want prebuilt images downloaded from Docker Hub, so that I need no application source checkout or compiler.
27. As an installer, I want the operator tool supplied precompiled, so that its bootstrap does not hide a local build.
28. As an installer, I want release components to be compatible and identifiable, so that my installation is reproducible.
29. As an installer, I want registry failures reported without automatic compilation, so that the selected installation mode remains predictable.
30. As an installer, I want configuration, credentials and data kept outside application images, so that my installation remains local and persistent.
31. As an installer, I want migrations and database binding initialization handled explicitly, so that a running container is not mistaken for an initialized application.
32. As an installer, I want existing curated descriptions and Evidence preserved, so that rerunning setup cannot overwrite human work.
33. As an installer, I want runtime checks performed from the actual container environment, so that host connectivity is not confused with application connectivity.
34. As an installer, I want completed and failed execution stages recorded, so that I can resume after interruption without duplicating data.
35. As an installer, I want configuration corrections to invalidate dependent checks, so that resuming does not trust stale results.
36. As an administrator, I want Catalog changes to retain their existing confirmation and concurrency rules, so that installation automation does not bypass domain safeguards.
37. As an installer, I want separate platform and workspace status, so that I know whether I can already ask a real question.
38. As an installer, I want stop/start behavior checked, so that a successful first run is not the only working state.
39. As a maintainer, I want an explicit release build-and-publish command, so that consumer installation can use actual Docker Hub artifacts.
40. As a maintainer, I want versioned images and release metadata, so that published artifacts can be traced to a source revision.
41. As a maintainer, I want publication credentials isolated from consumer configuration, so that installers need no write access to Docker Hub.
42. As a maintainer, I want the published artifacts tested by pulling them, so that an unpublished local image cannot satisfy release acceptance.
43. As a source-build user, I want an explicit supported build path, so that source customization remains possible.
44. As a maintainer, I want Windows, Omarchy and macOS acceptance recorded separately and in order, so that support claims reflect tests actually performed.
45. As an Italian or English reader, I want matching step-by-step guides, so that language choice does not change the installation contract.
## Implementation Decisions
### Boundaries and authority
- Keep the existing standalone architecture and operator CLI. The application runs
through Compose; the operator CLI owns preparation, validation and execution of
the installation, rather than introducing another installer or backend workflow.
- Preserve Workspace schema v4 and the workspace catalog as the authority for
workspace identity and optional Evidence. Formal validation uses the same rules
as runtime loading, including strict YAML interpretation, duplicate detection,
directory/index consistency and prohibited fields.
- Preserve the Installation Model Catalog as the sole authored source for provider,
model eligibility, defaults and embedding facts. Session and embedding configuration
must be complete before ordinary execution. Metadata generation remains optional
when its catalog configuration is omitted consistently.
- Preserve the PostgreSQL Metadata Catalog as the authority for Workspace Database
identity, schema, metadata and active Database Binding. Respect the existing
one-database-per-workspace boundary. Prepared binding documents are bootstrap
inputs, not a second runtime database catalog.
- Preserve installation-local secret storage, reference-based binding credentials,
editable Evidence authority, read-only DWH access and the separation of reference
preprocessing from Memory. Do not rewrite model projections or runtime snapshots
as independent authored configuration.
### Preparation documents and command surfaces
- Expose distinct operator operations for template preparation, workspace validation,
host checks, application validation, execution, status and resumption. Their exact
CLI spelling can be finalized with the implementation tickets; they must remain
separately invocable and scriptable. Setup execution is never a parameter wizard.
- Template preparation creates explicitly requested sample documents or protected
credential files without starting application services. It never silently replaces
existing user documents. Placeholders are visibly incomplete and cannot pass the
required-field checks.
- Supply a versioned, installation-local database bootstrap document describing the
workspace identity, engine, physical database/schema, transport and endpoint
configuration, and references to secrets. Validate against existing Catalog
capabilities and selected workspace identities. No credentials belong in the
shared workspace descriptors or public repository.
- Binding imports use authenticated, authorized Catalog services and their version
checks. On first execution they create the declared bindings and install secrets
through the existing store. On rerun, equivalent values are a no-op; conflicting
existing administrative changes stop with a reconciliation report. The bootstrap
input does not continuously overwrite a mutable Catalog.
- Initial local administrative authentication and its protected bootstrap material
are prepared before execution. Execution cannot depend on a person answering a
login wizard. Reuse supported operator authentication boundaries and required
permissions, without introducing a privileged unauthenticated installation API.
- Supply a precompiled validation capability with the operator distribution. The
workspace validation step must work without Docker, ThothII, Node or an application
source checkout. Reuse canonical validation rules; if a new packaging boundary is
necessary, prove equivalence with a shared set of valid and invalid documents.
### Validation contract
- Workspace validation covers syntax, strict schema, identity, catalog/descriptor
relationships and locally available Evidence references. It does not claim to
establish the truth of domain rules, database contents or services that do not exist.
- Host checks distinguish host prerequisites from bundled application dependencies.
Pi is checked as a release component and subsequently in the running core image,
never required as a separate host installation.
- Application validation checks required configuration, compatible release and host,
model usages/defaults, protected secret references, database transport capabilities,
workspace links, effective Compose configuration and accessible remote dependencies.
A transport that cannot serve NL-to-SQL sessions cannot pass workspace-readiness
validation merely because it can perform administrative diagnostics.
- Reports expose a stable outcome, check identifier, affected logical input/field,
explanation and next action. The public outcomes are passed, error, warning and
deferred-to-runtime. Human-readable output is accompanied by pristine structured
output for automation; exit status distinguishes success from blocking failure.
- Validation is read-only with respect to user documents, application state and
target databases. Explicit report output is allowed. Remote checks are bounded,
documented and non-mutating; any provider usage incurred by a configured smoke
test is disclosed before invocation, not requested interactively during setup.
- Missing or invalid required configuration, missing release artifacts, unsupported
architecture and failures of available required dependencies block execution.
Unreachable existing external services are errors, not automatically reclassified
as deferred. Only checks intrinsically dependent on the not-yet-created local
stack qualify for the accepted deferred category.
- Maintain an explicit obligation list for deferred checks: container-network
connectivity, Catalog initialization, Pi operation, local embedding availability,
preprocessing and relevant workspace runtime readiness. Each obligation has a
defined runtime check; there is no successful final state while a required
obligation remains unverified or failed.
- Bind the execution plan to normalized non-secret configuration, repository revision
and verified content, selected release digests and validator version. Re-read
protected credentials when checking or executing; do not expose their values or
unkeyed secret-derived fingerprints in reports. Re-run credential checks where
freshness cannot be established safely.
- Revalidate changed dependencies and live prerequisites at execution or resume.
A previously successful report is not blanket authorization to apply changed files
or evidence of current network availability.
### Release production and distribution
- Provide a maintainer command accepting source revision, release version, Docker Hub
namespace and target architectures. It performs preflight checks, reproducible
builds, artifact checks, publication and output of a coherent release manifest.
This command is a deliverable of this feature, not an undocumented manual prerequisite.
- Publish the existing core and frontend application images. Catalog migration and
workspace maintenance use the same released core image. Keep PostgreSQL, Qdrant
and Ollama as compatible upstream images; preserve the existing service boundaries.
- Package the operator executable, validation capability, Compose definitions,
initialization resources and migration support required by the release. No runtime
mount may require a resource that exists only in an application source checkout.
- Pin the release identity and resolved image digests. A published release manifest
binds compatible images and operator/configuration versions. Do not overwrite an
already published immutable release version or declare a partially published
image set installable. An interrupted publication can retry without advertising
an incomplete consumer release.
- Keep publishing credentials outside consumer bundles and logs. Public consumers
pull without publishing rights. Application images contain application software,
not installation secrets, user workspace data or prepopulated example databases.
- The ordinary bootstrap downloads a precompiled operator and release artifacts.
It must not compile the CLI through Docker as a hidden fallback. Windows WSL2
uses the Linux executable; macOS uses an appropriate host executable.
- Deliver and accept Linux amd64 images for Windows/WSL2 and Omarchy first. Add and
accept Linux arm64 for the macOS Apple Silicon stage. Multiarchitecture build
results do not by themselves prove host installation acceptance.
- A maintainer smoke test pulls the published artifacts by their release references.
Consumer acceptance runs must not succeed because of an unpushed locally built
image. Confirm core, frontend and maintenance references all resolve to the release.
- Retain explicit source mode with the same configuration, validation and persistence
rules. It is not the default and is never an automatic recovery action for a pull
failure. Registry recovery is a retry of the selected released artifacts.
### Non-interactive execution and recovery
- Execute only a complete, currently validated plan with no blocking errors. The
ordinary sequence pulls the release, prepares runtime projections and isolated
installation storage, starts required services, applies migrations, registers the
workspace source and imports the prepared Catalog bindings before dependent work.
- Apply configuration and migrations through existing service boundaries. Hold an
installation execution lock to prevent concurrent runs from racing over the same
state, containers or bootstrap imports.
- Reuse durable Catalog Sync Runs and their freshness/locking rules. Fresh additive
synchronization can proceed under the existing contract. A destructive diff or
another domain-required human decision stops at an explicit awaiting-review state;
the operator reviews through the existing administration surface and subsequently
resumes. This is a domain decision, not permission to collect missing setup parameters
or add an automatic confirmation bypass.
- Reuse existing curated metadata and Evidence. Do not generate AI descriptions or
new domain rules as an installation side effect. Optional generation remains a
separate explicit administrative action. Required preprocessing can use supported
source comments or curated descriptions without mandatory AI generation.
- Persist a bounded execution journal with installation identity, plan identity,
stage outcomes, released component versions, deferred-check outcomes and recovery
guidance. Do not persist secret values or raw exception output. Write progress
atomically and check actual state on resume.
- A repeated execution of the same completed plan must not recreate bindings,
duplicate data, clear Memory, overwrite Evidence or erase sessions. Reconcile
already completed stages with their actual persistent state before proceeding.
- After a document correction, revalidate and repeat only affected checks/stages.
Do not infer that an interrupted migration or import failed before inspecting its
durable result. Preprocessing, which has no internal resume contract, may need to
rerun as a whole; report that honestly.
- Never implement recovery by deleting all volumes or reverting user data. Report
the failing stage and safe next action. Failed or interrupted execution is not
advertised as a complete installation.
- Report platform state, Workspace Readiness and functional acceptance separately.
Runtime checks exercise the released Pi, embedding and actual container transport.
Final acceptance includes a real human-reviewed question and stop/start persistence.
The workflow's human review is not replaced by unattended benchmark evaluation.
### Documentation and delivery boundaries
- Keep Italian and English guides aligned with the six steps. Each step states its
inputs, documents, examples, verification operation, expected output and common
corrections. Provide an advance checklist of information and credentials to collect.
- Separate maintainer publication instructions, consumer prebuilt installation and
source-build instructions. State precisely which components are host prerequisites
and which are shipped inside the release.
- First acceptance uses an available project-specific repository and database.
The Financial, European Football and F1 project stays deferred. Do not expose its
unavailable loader, repository-copy support or data bundles as usable features.
- Preserve the future integration boundary: examples will be selected in documents
and loaded locally after application initialization, without rebuilding the
application or embedding the datasets into its Docker images.
## Testing Decisions
### Confirmed test boundaries
Use the public operator workflow as the principal test boundary: prepared documents
in, stable reports/exit statuses and observable installation outcomes out. Exercise
validation and execution through this boundary while replacing external command
execution and remote services with controllable test counterparts. Keep focused
contract tests at existing workspace parsing and Catalog boundaries where they
prevent divergent schemas or authority rules. Add real release and host acceptance
tests for behavior that simulated external services cannot establish.
The project owner confirmed this testing boundary on 28 September 2026, completing
the `/to-spec` checkpoint. The product decisions, six-step workflow and testing
scope are approved for specification publication and subsequent ticket decomposition.
### Existing testing practice to extend
- Operator setup tests already use temporary installation fixtures and a replaceable
command runner to cover sequencing, startup failures, error preservation and recovery
messages. Extend that boundary to document validation, prebuilt execution and resumption.
- Installation configuration tests cover strict schemas, model catalog rules and
incompatible existing files. Extend them with complete/incomplete preparation
documents, protected references and cross-document consistency.
- Workspace tests cover strict catalog/descriptor parsing, immutable Git revisions,
runtime handoff and secret handling. Reuse their document fixtures and validity
rules to demonstrate equivalence of the preinstallation validator.
- Catalog and preprocessing tests already cover durable runs, locks, binding freshness,
failure recording and preservation of authoritative state. Reuse these boundaries
to verify bootstrap import and resumption without bypassing the domain contracts.
- Existing multiarchitecture image checks provide a starting point for released
artifact verification. They do not replace pulling the published artifacts or
testing supported host environments.
### Required behavioral coverage
1. Validate workspace documents with Docker absent and no application runtime;
reject malformed YAML, duplicate keys/identifiers, unsupported fields, broken
references and invalid configured Evidence with actionable locations.
2. Repeated document/host/application checks do not create containers, change source
documents, import data or migrate databases. Only explicitly requested reports
may be written by validation.
3. Complete application documents pass; placeholders, missing required models,
invalid binding transport and unreadable secret references block execution.
4. Existing external-service failures remain errors. Checks genuinely dependent on
newly created local services are listed as deferred and cannot disappear from
final acceptance.
5. Execute with standard input closed. Valid inputs need no responses; missing
values yield an error without waiting for input or prompting for a replacement.
6. Changing documents, workspace revision or release after validation invalidates
dependent results. Credential changes are caught without leaking secret material.
7. A clean prebuilt consumer installation performs pulls and initialization, never
application or operator compilation; absent images fail without a source fallback.
8. Published manifests resolve all required images, platform variants and maintenance
components coherently. Simulated publication interruption does not advertise a
partial release; a real smoke test exercises artifacts pulled from Docker Hub.
9. Prepared database bindings become Catalog state once, use the existing protected
secret store and remain unchanged on equivalent reruns. Administrative divergence
is reported rather than silently overwritten.
10. Interruption after a durable operation but before journal completion resumes by
inspecting state, without duplicating that operation. Concurrent setup runs cannot
mutate the same installation simultaneously.
11. Destructive Catalog synchronization requires its existing review and fresh source
checks. The setup's non-interactive nature does not auto-approve a destructive diff.
12. Existing descriptions, Evidence, sessions and Memory survive validation, rerun,
configuration correction and stop/start. Optional AI generation is not triggered.
13. Runtime Pi, embedding and connectivity checks are executed from the installed
release, and platform success cannot mask failed Workspace Readiness.
14. Source mode remains functional and explicit with equivalent configuration
contracts. A source-mode pass cannot close prebuilt distribution acceptance.
15. Execute real consumer acceptance first on Windows x64/WSL2, then Omarchy x64,
then macOS Apple Silicon. Record each environment, release identity, stage
outcomes, a real reviewed question and a stop/start check separately.
Good tests assert observable contracts, preserved data and required side effects,
not private helper calls or incidental internal ordering. Use real isolated Catalog
instances where transaction and lock behavior matters. Mock external provider
failures for repeatable tests, while keeping actual DWH/model acceptance separate.
## Out of Scope
- Implementing or publishing the three example databases, their curated contents,
example CLI, auxiliary repository layout or autonomous repository-copy mode.
- A new graphical installer, a conversational parameter wizard, or collecting
required parameters in administration pages after an incomplete setup.
- Installing Pi, Python, Node or an application build toolchain on ordinary consumer hosts.
- Changes to server/Omics deployment, upstream authentication or the application's
existing human-in-the-loop workflow.
- Multiple workspace repositories per installation or multiple databases per workspace.
- Automatic release upgrades, destructive reset/uninstall, whole-volume rollback
or backup-policy redesign. Interrupted initial setup recovery remains in scope.
- Automatic semantic certification, generated Evidence without sources, benchmark
SQL targets or automated accuracy scoring.
- Publishing Docker images or executing installations as part of this specification
authoring task. These are implementation and release deliverables described above.
## Further Notes
The project owner approved the six-step revision and both final clarifications on
28 September 2026: examples stay deferred, and runtime-only checks are explicit
post-start obligations. Earlier interactive-wizard and incomplete-configuration
installation proposals are superseded where they conflict with this specification.
This specification follows the existing decisions on PostgreSQL metadata authority,
installation-local bindings, secret references, durable schema synchronization and
the Installation Model Catalog. It does not change those architectural authorities.
Publication of a usable Docker Hub release is a blocking dependency of consumer
prebuilt-installation acceptance. Availability of the deferred examples is not.
Docker Hub namespace, publishing credentials and concrete release versions are
maintainer release inputs, not values to invent or embed into user templates.
After publication to Gitea, `/to-tickets` will split this specification into small
end-to-end increments with explicit blockers. Implementation has not started in
this task, and publishing the specification does not attest that a release exists.
@@ -0,0 +1,420 @@
# Scomposizione della specifica di installazione
Data: 2026-09-28. Stato: scomposizione approvata dall'utente («approvo»);
pubblicazione `/to-tickets` completata. Verificati testi, etichetta `ready-for-agent`
e dipendenze native delle issue #43–#54; specifica parent invariata.
Parent: [Spec: installazione da documenti verificati e distribuzione Docker Hub](https://git.tylconsulting.it/mptyl/ThothII/issues/42).
Gli identificatori T01–T12 restano riferimenti della scomposizione; le issue reali
sono elencate sotto. Ogni issue usa `ready-for-agent` e dipendenze native Gitea.
La parent non viene modificata né chiusa.
## Issue pubblicate
Avanzamento locale, 2026-09-28: T01/#43 implementato sul branch
`codex/guided-standalone-install`. Disponibili `tht workspace prepare` e
`tht workspace validate`, con helper autonomo che riusa i parser runtime. Guide
IT/EN aggiornate; esempi e pubblicazione degli artefatti restano differiti ai ticket
previsti. Nessuna chiusura o modifica della parent effettuata.
Verifica: 14 test della CLI passano sia da sorgenti sia con il bundle nativo macOS
arm64 e `PATH` vuoto; artefatti Windows amd64/Linux amd64 cross-compilati, senza
attribuire loro un collaudo host. Backend su Node 24.16: 109 file passati, un file
saltato, 1.417 test passati e 40 saltati; typecheck e build rigorosa documentazione
passati. Suite Go: tutti i pacchetti passati, salvo un primo errore intermittente
nel test di concorrenza authstorage; quel test passa in tre ripetizioni e il pacchetto
completo passa nella verifica isolata. Le revisioni Standards e Spec non lasciano
finding aperti.
T02/#44 implementato sullo stesso branch: `tht installation prepare`, generazione
esplicita delle credenziali tecniche e `tht installation validate` preparano e
controllano documenti privati, modelli, autenticazione e bootstrap dei binding,
riusando gli schemi runtime senza avviare servizi. Guide IT/EN aggiornate.
Verifica backend completa su Node 24.16: 111 file passati, uno saltato, 1.420 test
passati e 40 saltati; le regressioni successive della revisione passano nella suite
mirata (quattro test, incluso il percorso con binari nativi e `PATH` vuoto).
Typecheck, build rigorosa documentazione e pacchetti Go passati; il pacchetto CLI
è stato ripetuto dopo la correzione rilevata in revisione. Nessun finding residuo
delle revisioni Standards/Spec.
T03/#45 implementato: `installation preflight` verifica l'host al passo 3 e
`installation plan` ripete i documenti, verifica rilascio/Compose e dipendenze
esterne, poi sigilla un piano privato legato agli input. Restano espliciti gli
obblighi runtime; nessun container viene creato. Il contratto del manifest è nel
[riferimento pubblico di preflight](../install/installation-preflight.md).
Verifica completa: tutti i pacchetti Go passati, inclusa l'integrazione con la
coppia nativa, Docker controllato e servizi Git HTTPS/database REST locali;
backend Node 24.16 con 112 file passati, uno saltato, 1.423 test passati e 40 saltati.
Typecheck e documentazione rigorosa passati. Bundle macOS arm64, Linux amd64 e
Windows amd64 ricompilati; solo macOS è stato eseguito qui, senza attribuire un
collaudo host alle compilazioni incrociate. Le revisioni Standards/Spec non lasciano
finding aperti dopo le regressioni su rete Evidence, collocazione del piano,
piattaforma Compose e comparsa di override locali. Nessuna pubblicazione reale
effettuata: il prossimo incremento è T04/#46, con namespace e accessi del manutentore.
| Ticket | Issue Gitea | Dipendenze dirette |
| --- | --- | --- |
| T01 | [Preparare e validare un repository workspace senza stack](https://git.tylconsulting.it/mptyl/ThothII/issues/43) | Nessuna |
| T02 | [Preparare e validare i documenti applicativi](https://git.tylconsulting.it/mptyl/ThothII/issues/44) | #43 |
| T03 | [Verificare precondizioni e produrre il piano eseguibile](https://git.tylconsulting.it/mptyl/ThothII/issues/45) | #44 |
| T04 | [Produrre e pubblicare un rilascio Docker Hub installabile](https://git.tylconsulting.it/mptyl/ThothII/issues/46) | Nessuna |
| T05 | [Installare la piattaforma dal rilascio senza domande](https://git.tylconsulting.it/mptyl/ThothII/issues/47) | #45, #46 |
| T06 | [Applicare i binding preparati al Catalog](https://git.tylconsulting.it/mptyl/ThothII/issues/48) | #47 |
| T07 | [Portare il workspace alla readiness con controlli runtime](https://git.tylconsulting.it/mptyl/ThothII/issues/49) | #48 |
| T08 | [Conservare il percorso esplicito da sorgente](https://git.tylconsulting.it/mptyl/ThothII/issues/50) | #47 |
| T09 | [Riprendere dopo correzioni e interruzioni senza perdere stato](https://git.tylconsulting.it/mptyl/ThothII/issues/51) | #49, #50 |
| T10 | [Collaudare l'installazione pubblicata su Windows/WSL2](https://git.tylconsulting.it/mptyl/ThothII/issues/52) | #51 |
| T11 | [Collaudare l'installazione su Omarchy](https://git.tylconsulting.it/mptyl/ThothII/issues/53) | #52 |
| T12 | [Pubblicare e collaudare il percorso macOS Apple Silicon](https://git.tylconsulting.it/mptyl/ThothII/issues/54) | #53 |
Ogni ticket comprende verifiche del comportamento e aggiornamenti pertinenti delle
guide IT/EN. Le dipendenze elencate sono dirette; non si ripetono quelle transitive.
Si riusano il runner dell'operatore e i servizi di dominio esistenti; gli adattamenti
necessari sono inclusi nella prima funzionalità che li usa. Non emerge una necessità
di refactoring trasversale da pubblicare come lavoro orizzontale separato.
| Ticket | Titolo | Bloccato da | Risultato dimostrabile |
| --- | --- | --- | --- |
| T01 | Preparare e validare un repository workspace senza stack | Nessuno | Template e controllo locale conformi ai contratti, senza Docker attivo. |
| T02 | Preparare e validare i documenti applicativi | T01 | Parametri, modelli e binding completi verificati senza avviare servizi. |
| T03 | Verificare precondizioni e produrre il piano eseguibile | T02 | Rapporto con errori bloccanti e obblighi runtime, legato agli input. |
| T04 | Produrre e pubblicare un rilascio Docker Hub installabile | Nessuno | Comando manutentore e pacchetto pubblico verificato tramite pull. |
| T05 | Installare la piattaforma dal rilascio senza domande | T03, T04 | Pull, inizializzazione e avvio da documenti, con stato e ripresa delle fasi. |
| T06 | Applicare i binding preparati al Catalog | T05 | Database dei workspace configurati senza questionari o duplicazioni. |
| T07 | Portare il workspace alla readiness con controlli runtime | T06 | Schema, preprocessing e servizi verificati senza aggirare la revisione umana. |
| T08 | Conservare il percorso esplicito da sorgente | T05 | Stessi input e contratti, con build scelta esplicitamente. |
| T09 | Riprendere dopo correzioni e interruzioni senza perdere stato | T07, T08 | Recupero dell'intero percorso, compresi binding, sync e preprocessing. |
| T10 | Collaudare l'installazione pubblicata su Windows/WSL2 | T09 | Prima accettazione reale senza sorgenti o compilatori. |
| T11 | Collaudare l'installazione su Omarchy | T10 | Seconda accettazione reale su Linux x64, distinta da Windows. |
| T12 | Pubblicare e collaudare il percorso macOS Apple Silicon | T11 | Terza accettazione con immagini arm64 e comando host compatibile. |
## T01 — Preparare e validare un repository workspace senza stack
### Parent
[Spec: installazione da documenti verificati e distribuzione Docker Hub](https://git.tylconsulting.it/mptyl/ThothII/issues/42).
### What to build
Un autore prepara un repository ad hoc usando un template documentato e verifica
i documenti localmente prima di installare ThothII. Il validatore è fornito come
capacità eseguibile senza Node, Docker attivo o checkout dei sorgenti applicativi.
Riusa i contratti del runtime, senza creare uno schema workspace alternativo.
### Acceptance criteria
- [ ] Il template distingue catalogo workspace, descriptor ed Evidence opzionali e non contiene funzionalità degli esempi ancora indisponibili.
- [ ] Preparare un template non avvia servizi, non sovrascrive documenti esistenti e non richiede accesso in scrittura al repository originale.
- [ ] Il controllo respinge YAML ambiguo o invalido, chiavi duplicate, identificatori duplicati, campi estranei, incoerenze catalogo/directory e riferimenti locali mancanti.
- [ ] Le Evidence configurate sono verificate per ciò che è controllabile localmente; assenza lecita e invalidità sono distinte, senza certificare il significato delle regole di dominio.
- [ ] Gli esiti identificano documento/campo e correzione; output strutturato e codici di uscita sono verificabili senza esporre segreti.
- [ ] Una raccolta condivisa di casi validi/invalidi prova equivalenza con i parser runtime, e il controllo passa senza Docker e senza runtime host aggiuntivi.
- [ ] Guide IT/EN mostrano preparazione, correzione e ripetizione del passo 2.
### Blocked by
None (can start immediately).
## T02 — Preparare e validare i documenti applicativi
### Parent
[Spec: installazione da documenti verificati e distribuzione Docker Hub](https://git.tylconsulting.it/mptyl/ThothII/issues/42).
### What to build
L'operatore compila template locali per installazione, modelli, binding database e
segreti, e ne verifica completezza e coerenza con i workspace già verificati.
Non viene avviata l'applicazione e nessun valore viene richiesto dal futuro setup.
### Acceptance criteria
- [ ] Template commentati ed esempi completi spiegano obblighi, default e riferimenti ai documenti protetti; i placeholder non superano la validazione.
- [ ] Modelli e embedding rispettano l'Installation Model Catalog; generazione metadati opzionale e default sono coerenti con i contratti esistenti.
- [ ] Un input bootstrap locale versionato descrive Workspace Database e Database Binding con riferimenti ai segreti; i descriptor workspace restano conformi allo schema v4.
- [ ] Validazione incrociata di workspace, binding, engine/trasporto, modelli, percorsi e file ambiente, senza migrare o interrogare in scrittura alcun database.
- [ ] La generazione esplicita delle credenziali tecniche produce file protetti prima del setup, senza sovrascritture o segreti nei log/rapporti.
- [ ] I test coprono input completi, mancanti, incompatibili e segreti illeggibili; i documenti dell'utente rimangono invariati durante le verifiche.
- [ ] Guide IT/EN consentono di raccogliere e preparare tutte le informazioni con calma prima dell'esecuzione.
### Blocked by
- T01 — Preparare e validare un repository workspace senza stack.
## T03 — Verificare precondizioni e produrre il piano eseguibile
### Parent
[Spec: installazione da documenti verificati e distribuzione Docker Hub](https://git.tylconsulting.it/mptyl/ThothII/issues/42).
### What to build
L'operatore verifica host e dipendenze esterne disponibili, quindi ottiene un piano
eseguibile riferito ai documenti e al rilascio scelti. Il rapporto distingue errori,
avvisi e controlli necessariamente rinviati al runtime, senza creare lo stack.
### Acceptance criteria
- [ ] I controlli host sono invocabili al passo 3; quelli dipendenti dai parametri finali sono completati o ripetuti al passo 5.
- [ ] Sono verificati Docker/Compose, architettura, WSL2 quando pertinente, percorsi/permessi, risorse e disponibilità del rilascio e dei suoi componenti nel registry.
- [ ] Le prove sulle dipendenze esterne disponibili sono circoscritte e documentate; un servizio esistente irraggiungibile non viene promosso a semplice controllo differito.
- [ ] Pi non è richiesto sull'host; le dipendenze incluse nelle immagini sono riconosciute nel manifest e associate a controlli runtime precisi.
- [ ] Il piano registra input non segreti, revisione/contenuti workspace, release e versione del validatore; nessun segreto o fingerprint pubblico non protetto di segreti.
- [ ] Ogni controllo differito ha un'identità e un'obbligazione runtime; valori obbligatori mancanti o immagini assenti bloccano il piano.
- [ ] Prove con manifest e servizi controllati coprono cambiamento degli input, credenziali, errori di rete e architetture; nessuna creazione di container o mutazione di dati.
- [ ] Guide IT/EN spiegano rapporto, errori e verifiche ancora da eseguire. La prova con il rilascio reale verrà completata dal ticket di esecuzione, dopo T04.
### Blocked by
- T02 — Preparare e validare i documenti applicativi.
## T04 — Produrre e pubblicare un rilascio Docker Hub installabile
### Parent
[Spec: installazione da documenti verificati e distribuzione Docker Hub](https://git.tylconsulting.it/mptyl/ThothII/issues/42).
### What to build
Il manutentore esegue un comando riproducibile che costruisce, verifica e pubblica
core/frontend su Docker Hub, insieme al pacchetto operatore compatibile, e dimostra
che il rilascio pubblicato è scaricabile. La pubblicazione delle immagini oggi
mancanti è un risultato concreto del ticket, non un prerequisito lasciato a mano.
### Acceptance criteria
- [ ] Il comando riceve revisione, versione, namespace e architetture e mantiene fuori da bundle/log le credenziali di pubblicazione.
- [ ] Pubblica core/frontend Linux amd64 per la prima tappa; Catalog migration e workspace maintenance risolvono alla stessa immagine core del rilascio.
- [ ] Il bundle contiene comando host precompilato, Compose, inizializzazione e risorse di migrazione, senza dipendenze da checkout sorgente durante l'avvio.
- [ ] Il processo può includere la capacità di validazione preinstallazione prodotta da T01 nelle revisioni che la contengono; non serve duplicarne l'implementazione per questo ticket.
- [ ] Il manifest lega versione/revisione e digest compatibili; una pubblicazione parziale non viene esposta come rilascio completo e una versione immutabile non viene sovrascritta.
- [ ] Viene pubblicato un rilascio reale e viene verificato il pull degli artefatti pubblicati, senza affidarsi a immagini presenti soltanto nella cache di build.
- [ ] Test automatici verificano orchestrazione, fallimenti e retry senza richiedere una pubblicazione reale a ogni test; la prova reale del ticket resta distinta e registrata.
- [ ] Documentazione manutentore IT/EN e istruzioni del bundle distinguono pubblicazione, consumo e futura estensione arm64. Namespace e accessi effettivi sono input del manutentore, non valori inventati.
### Blocked by
None (can start immediately).
## T05 — Installare la piattaforma dal rilascio senza domande
### Parent
[Spec: installazione da documenti verificati e distribuzione Docker Hub](https://git.tylconsulting.it/mptyl/ThothII/issues/42).
### What to build
L'operatore applica un piano verificato, scarica gli artefatti pubblicati e ottiene
una piattaforma inizializzata e accessibile senza compilazione o domande. Lo stato
registrato permette di ritentare le fasi di piattaforma interrotte; non viene ancora
dichiarato pronto un workspace privo delle successive verifiche Catalog.
### Acceptance criteria
- [ ] Avvio da bundle rilasciato e operatore precompilato, senza checkout applicativo o toolchain; la revisione del bundle include i validatori e i comandi effettivamente utilizzati.
- [ ] Il piano viene ricontrollato rispetto a input, release e prerequisiti vivi prima delle mutazioni; un piano mancante o incoerente viene rifiutato.
- [ ] Con standard input chiuso il setup esegue pull, configurazione runtime, reti/volumi/container, inizializzazione Catalog/Memory e migrazioni senza richiedere parametri.
- [ ] L'accesso amministrativo iniziale deriva da materiale protetto preparato prima; non si introduce un endpoint privilegiato senza autenticazione.
- [ ] Un lock impedisce esecuzioni concorrenti; il journal atomico registra le fasi senza segreti e consente di verificare lo stato reale prima di ripetere una fase interrotta.
- [ ] Errori di pull non causano build locali; errori di configurazione rimandano ai documenti e alla nuova verifica, senza prompt di riparazione.
- [ ] La piattaforma accessibile è distinta dalla Workspace Readiness ancora da verificare; nessun messaggio finale prematuro di piena utilizzabilità.
- [ ] Test del runner e prova con artefatti pubblicati coprono successo, stdin chiuso, interruzioni e ripetizione senza cancellare volumi o dati. Guide IT/EN documentano il risultato parziale corretto.
### Blocked by
- T03 — Verificare precondizioni e produrre il piano eseguibile.
- T04 — Produrre e pubblicare un rilascio Docker Hub installabile.
## T06 — Applicare i binding preparati al Catalog
### Parent
[Spec: installazione da documenti verificati e distribuzione Docker Hub](https://git.tylconsulting.it/mptyl/ThothII/issues/42).
### What to build
Il setup rende operativa la configurazione database predisposta nei documenti:
registra il repository, crea i Workspace Database e le Database Binding nel Catalog,
installa i riferimenti segreti e verifica le connessioni. Il risultato è un binding
utilizzabile senza una compilazione manuale dei parametri nell'interfaccia web.
### Acceptance criteria
- [ ] Identità e revisioni dei workspace corrispondono al piano; il consumo del repository non richiede push e non sostituisce implicitamente una sorgente esistente.
- [ ] Creazione e modifica dei binding utilizzano servizi autorizzati, controlli di versione e secret store esistenti; nessuna seconda autorità runtime nei documenti bootstrap.
- [ ] La stessa configurazione applicata due volte non duplica record, credenziali o binding.
- [ ] Una modifica amministrativa incompatibile produce un rapporto di riconciliazione invece di essere sovrascritta dai file preparatori.
- [ ] La connessione viene controllata dall'ambiente applicativo; un esito positivo ottenuto dall'host non basta a dichiararla utilizzabile dai container.
- [ ] Un'interruzione dopo il salvataggio ma prima dell'aggiornamento del journal viene riconosciuta alla ripresa, senza duplicazioni o perdita di segreti.
- [ ] Test di contratto e integrazione Catalog coprono autorizzazioni, concorrenza, versioni e rerun; guide IT/EN illustrano diagnosi e riconciliazione.
### Blocked by
- T05 — Installare la piattaforma dal rilascio senza domande.
## T07 — Portare il workspace alla readiness con controlli runtime
### Parent
[Spec: installazione da documenti verificati e distribuzione Docker Hub](https://git.tylconsulting.it/mptyl/ThothII/issues/42).
### What to build
Da un binding applicato, il percorso completa sincronizzazione dello schema e
preparazione necessaria e rende visibili i risultati dei controlli runtime.
Un workspace è pronto solo quando tutti i requisiti applicabili sono verificati;
le decisioni umane già previste dai contratti rimangono esplicite.
### Acceptance criteria
- [ ] La sincronizzazione usa i Catalog Sync Runs durabili con lock, freschezza e transazioni esistenti; non introduce una seconda implementazione.
- [ ] Diff distruttive fermano il percorso in attesa della revisione di dominio esistente; ripresa successiva senza auto-conferme né domande sui parametri di setup.
- [ ] Preprocessing e consolidamento riusano descrizioni/commenti ed Evidence curate; nessuna generazione AI implicita, nessuna cancellazione di Memory o sovrascrittura di curation.
- [ ] Assenza lecita di Evidence non blocca; Evidence configurate ma invalide e indici necessari non pronti restano blocchi reali.
- [ ] Pi, modello embedding, trasporto DWH e altri obblighi differiti sono eseguiti a runtime e rendicontati; nessun obbligo scompare o viene considerato superato senza prova.
- [ ] Stato piattaforma, Workspace Readiness e collaudo funzionale sono distinti; una domanda reale con revisione rimane la prova funzionale, senza SQL target.
- [ ] Test di servizio e integrazione dimostrano esiti, conservazione dati e ripresa dei run; guide IT/EN spiegano le eventuali revisioni umane residue.
### Blocked by
- T06 — Applicare i binding preparati al Catalog.
## T08 — Conservare il percorso esplicito da sorgente
### Parent
[Spec: installazione da documenti verificati e distribuzione Docker Hub](https://git.tylconsulting.it/mptyl/ThothII/issues/42).
### What to build
Un operatore sceglie esplicitamente la build da una revisione sorgente e usa gli
stessi documenti, validatori, identità d'installazione e servizi del percorso
precompilato. L'alternativa resta praticabile mentre il default diventa Docker Hub.
### Acceptance criteria
- [ ] Modalità sorgente e prerequisiti aggiuntivi sono espliciti; nessun errore del registry la attiva automaticamente.
- [ ] I componenti costruiti sono equivalenti nei contratti di configurazione, migrazione e persistenza; non esistono implementazioni parallele dei binding o della readiness.
- [ ] Il piano identifica modalità e revisione e invalida i controlli dipendenti quando cambiano.
- [ ] L'esecuzione rimane non interattiva e usa journal/lock comuni; i segreti non entrano nelle immagini di sviluppo.
- [ ] Una prova automatizzata dimostra build e avvio espliciti e il mancato fallback da pull; una prova sorgente non chiude l'accettazione del rilascio precompilato.
- [ ] Guide IT/EN separano il percorso avanzato da quello ordinario e rendono visibili i prerequisiti aggiuntivi.
### Blocked by
- T05 — Installare la piattaforma dal rilascio senza domande.
## T09 — Riprendere dopo correzioni e interruzioni senza perdere stato
### Parent
[Spec: installazione da documenti verificati e distribuzione Docker Hub](https://git.tylconsulting.it/mptyl/ThothII/issues/42).
### What to build
L'operatore corregge un endpoint, una credenziale o un altro documento dopo un errore
e riprende l'intero percorso con verifiche aggiornate. Questo ticket completa il
recupero fra stadi e modalità, oltre ai retry locali già consegnati dai singoli ticket.
### Acceptance criteria
- [ ] Cambiamenti ai documenti, ai contenuti/revisioni workspace o al rilascio invalidano le sole verifiche/fasi dipendenti; le altre vengono riconciliate con lo stato reale.
- [ ] La rotazione di una credenziale viene rilevata senza esporla o pubblicarne fingerprint non protetti; si ripetono le prove necessarie.
- [ ] Ripresa dopo interruzione nei confini fra pull, inizializzazione, importazione Catalog, sync e preprocessing non duplica operazioni già persistite.
- [ ] Il preprocessing interrotto è rieseguito secondo il contratto esistente, senza promettere resume interno; un run in attesa di decisione umana conserva tale stato.
- [ ] Interruzioni, errori e concorrenza non corrompono il journal né attivano reset di volumi; diagnosi e stato rimangono privi di segreti.
- [ ] Test del percorso pubblico, con guasti controllati e integrazione dove conta la persistenza, dimostrano conservazione di sessioni, Evidence, descrizioni e Memory in entrambe le modalità.
- [ ] Le guide IT/EN presentano scenari di correzione/ripresa senza suggerire la cancellazione dei dati come normale rimedio.
### Blocked by
- T07 — Portare il workspace alla readiness con controlli runtime.
- T08 — Conservare il percorso esplicito da sorgente.
## T10 — Collaudare l'installazione pubblicata su Windows/WSL2
### Parent
[Spec: installazione da documenti verificati e distribuzione Docker Hub](https://git.tylconsulting.it/mptyl/ThothII/issues/42).
### What to build
Dimostrare il percorso completo su un PC Windows x64 con Ubuntu WSL2 e Docker
Desktop usando il rilascio realmente pubblicato, repository ad hoc e documenti
predisposti. Il collaudo include le correzioni necessarie a rendere utilizzabile
la prima piattaforma e un rapporto riproducibile.
### Acceptance criteria
- [ ] Un rilascio della revisione integrata viene pubblicato tramite T04 e consumato tramite pull; le immagini costruite soltanto localmente non soddisfano la prova.
- [ ] Il consumer non dispone di sorgenti applicativi o toolchain necessarie a compilare; operatore e validatori sono quelli precompilati nel bundle.
- [ ] Tutti i sei passi sono percorsi nell'ordine documentato, con almeno una correzione documentale e una ripresa dopo errore, senza domande durante il setup.
- [ ] Primo workspace ad hoc realmente utilizzabile, una domanda con revisione umana e stop/start con stato preservato; nessun uso presunto degli esempi rinviati.
- [ ] Rapporto con host/runtime, revisione, digest e risultati distinti di piattaforma/workspace/funzione, senza segreti; problemi esterni non sono nascosti.
- [ ] Guide IT/EN sono verificate rispetto ai comandi e agli esiti reali; eventuale assenza di host o credenziali necessarie lascia il collaudo incompleto, non simulato come riuscito.
### Blocked by
- T09 — Riprendere dopo correzioni e interruzioni senza perdere stato.
## T11 — Collaudare l'installazione su Omarchy
### Parent
[Spec: installazione da documenti verificati e distribuzione Docker Hub](https://git.tylconsulting.it/mptyl/ThothII/issues/42).
### What to build
Dopo la tappa Windows, ripetere e rendere funzionante il percorso su Linux Omarchy
x64, producendo un'evidenza di accettazione propria e mantenendo il comportamento
documentale e non interattivo già consegnato.
### Acceptance criteria
- [ ] Rilascio pubblico compatibile scaricato da Docker Hub e comando host precompilato; nessuna compilazione nel percorso ordinario.
- [ ] Prerequisiti, permessi, percorsi e rete di Omarchy sono verificati su un host reale, senza trasferire automaticamente l'esito Windows.
- [ ] Sei passi, input invalido/corretto, ripresa, workspace ad hoc, domanda reale e stop/start superano il collaudo.
- [ ] Ogni correzione di portabilità include la relativa verifica e non introduce una divergenza dei contratti rispetto al percorso Windows.
- [ ] Rapporto separato con versioni/digest e guide IT/EN coerenti; senza un host disponibile il gate rimane aperto.
### Blocked by
- T10 — Collaudare l'installazione pubblicata su Windows/WSL2.
## T12 — Pubblicare e collaudare il percorso macOS Apple Silicon
### Parent
[Spec: installazione da documenti verificati e distribuzione Docker Hub](https://git.tylconsulting.it/mptyl/ThothII/issues/42).
### What to build
Dopo Omarchy, pubblicare e verificare il set di artefatti compatibile con macOS
Apple Silicon, incluse immagini Linux arm64 e comando nativo, e chiudere la terza
tappa di accettazione su un Mac reale.
### Acceptance criteria
- [ ] Il comando di rilascio pubblica immagini arm64 e bundle host compatibile, con manifest/digest coerenti e senza dichiarare supporto prima del collaudo.
- [ ] Il consumer usa il rilascio pubblico e non compila; gli script e le risorse di inizializzazione sono presenti nel bundle.
- [ ] Tutti i sei passi, correzione/ripresa, workspace ad hoc, domanda reale e stop/start sono verificati sul Mac.
- [ ] Le eventuali correzioni conservano compatibilità e contratti delle tappe precedenti; le prove multiarch di build non sostituiscono il collaudo host.
- [ ] Rapporto macOS separato e guide IT/EN finalizzate per le tre piattaforme; nessun risultato sintetico viene presentato come prova reale.
### Blocked by
- T11 — Collaudare l'installazione su Omarchy.
## Verifiche della scomposizione
- I primi ticket lavorabili sono T01 e T04.
- T03 usa manifest e servizi controllati per i propri contratti; non aspetta la
pubblicazione reale. T05 è il primo punto che richiede insieme validazione e
artefatti realmente pubblicati.
- T06 e T08 possono procedere in parallelo dopo T05. T09 riunisce i percorsi per
verificare correzioni e ripresa dell'intera installazione.
- I tre gate host sono sequenziali per scelta esplicita dell'utente, non per una
dipendenza architetturale inventata.
- Gli esempi restano esclusi. Il comando di pubblicazione Docker Hub e almeno una
pubblicazione reale sono inclusi, non demandati a un futuro progetto.
- Nessun aggiornamento o chiusura della parent è previsto dalla pubblicazione.

Some files were not shown because too many files have changed in this diff Show More