ThothII/docs/plans/2026-09-28-document-first-installation-spec.md

# Spec: installation from validated documents and Docker Hub releases

## Problem Statement

An operator who understands Docker and can edit a documented configuration should
not have to discover ThothII's architecture while answering an installation wizard.
The operator needs time to prepare workspace and application documents, validate
them repeatedly, and correct errors before applying changes to the machine.

The current setup combines document generation, local builds, startup and runtime
diagnostics. Its success message can precede an actually usable workspace. It does
not provide the complete preinstallation validation and resumable, non-interactive
execution required by this workflow.

The ordinary installation must consume published, prebuilt application images.
As reported by the project owner on 28 September 2026, ThothII images are not yet
available on Docker Hub. Publishing and verifying a real release is therefore a
prerequisite for testing the consumer installation, rather than a later enhancement.

## Solution

Deliver a documented six-step workflow, in this exact order:

1. Prepare a workspace repository: a project-specific repository initially, or the
   default examples repository when that separately deferred project is available.
2. Validate workspace documents formally and substantively to the extent possible
   from their contents and available references.
3. Verify host and distribution prerequisites.
4. Prepare application parameters in installation-local YAML and protected
   environment/secret documents, using commented templates and complete examples.
5. Validate application parameters and their consistency with the selected workspaces.
6. Execute the validated installation without asking configuration questions: pull
   the release images, create and initialize the stack, apply declared configuration,
   and run the required runtime checks.

Before these consumer steps can be tested against Docker Hub, a maintainer command
must build, check and publish the release and verify that its artifacts can be pulled.
Local source builds remain an explicit alternative, with the same configuration and
persistence contracts. A failed pull never silently switches to source compilation.

Checks that cannot run before container or database creation are enumerated as
deferred obligations and must run after startup. Errors detectable beforehand block
execution. Platform acceptance, Workspace Readiness and functional acceptance are
reported separately. A real question with human review completes functional acceptance.

## User Stories

1. As an installer, I want a six-step guide, so that I can understand the whole process before changing my machine.
2. As an installer, I want commented templates and completed examples, so that I can prepare documents without knowing internal component names.
3. As an installer, I want to pause document preparation, so that I can obtain missing information without restarting an installer.
4. As an installer, I want to prepare a project-specific workspace repository, so that I can use my own database before the example databases are delivered.
5. As an installer, I want the future default repository to be clearly distinguished from available features, so that I am not directed to unavailable examples.
6. As an installer, I want read-only access to the source workspace repository, so that consuming a workspace does not require publication rights.
7. As a workspace author, I want local document validation before Docker starts, so that syntax and contract errors are inexpensive to correct.
8. As a workspace author, I want duplicate identifiers, unsupported fields and inconsistent references reported, so that formally valid YAML does not hide an invalid workspace.
9. As a workspace author, I want errors to identify the document and field, so that I know precisely what to edit.
10. As a workspace author, I want optional Evidence distinguished from invalid configured Evidence, so that an intentionally absent corpus does not block installation.
11. As a workspace author, I want semantic validation limits stated honestly, so that successful validation is not mistaken for certification of domain knowledge.
12. As an installer, I want host prerequisites checked explicitly, so that Docker, architecture, permissions and resource problems are detected before setup.
13. As a Windows installer, I want a verified WSL2 and Docker Desktop path, so that I can follow one supported installation procedure.
14. As an installer, I want Pi supplied with the application image, so that I do not have to install an unnecessary host dependency.
15. As an installer, I want application models, endpoints and credentials prepared before execution, so that I can consult colleagues or provider documentation at my own pace.
16. As an installer, I want one authored model catalog, so that provider and embedding configuration do not disagree across components.
17. As an installer, I want database bindings prepared locally and separately from workspace definitions, so that environment-specific details do not leak into shared workspace repositories.
18. As an installer, I want generated internal credentials prepared in protected files, so that I need not invent technical passwords during execution.
19. As an installer, I want my secrets excluded from logs and validation reports, so that diagnostic output is safe to inspect and share.
20. As an installer, I want repeatable application validation, so that I can correct configuration without creating containers or changing databases.
21. As an installer, I want available external connections checked before startup, so that preventable endpoint or authentication errors are found early.
22. As an installer, I want non-executable checks listed explicitly, so that I know what still needs to be proved after startup.
23. As an installer, I want validation tied to the documents and release I selected, so that execution cannot silently apply different inputs.
24. As an installer, I want setup to run with closed standard input, so that it cannot unexpectedly ask me for a parameter.
25. As an installer, I want missing values to stop setup with a useful diagnosis, so that I can correct the document and validate again.
26. As an installer, I want prebuilt images downloaded from Docker Hub, so that I need no application source checkout or compiler.
27. As an installer, I want the operator tool supplied precompiled, so that its bootstrap does not hide a local build.
28. As an installer, I want release components to be compatible and identifiable, so that my installation is reproducible.
29. As an installer, I want registry failures reported without automatic compilation, so that the selected installation mode remains predictable.
30. As an installer, I want configuration, credentials and data kept outside application images, so that my installation remains local and persistent.
31. As an installer, I want migrations and database binding initialization handled explicitly, so that a running container is not mistaken for an initialized application.
32. As an installer, I want existing curated descriptions and Evidence preserved, so that rerunning setup cannot overwrite human work.
33. As an installer, I want runtime checks performed from the actual container environment, so that host connectivity is not confused with application connectivity.
34. As an installer, I want completed and failed execution stages recorded, so that I can resume after interruption without duplicating data.
35. As an installer, I want configuration corrections to invalidate dependent checks, so that resuming does not trust stale results.
36. As an administrator, I want Catalog changes to retain their existing confirmation and concurrency rules, so that installation automation does not bypass domain safeguards.
37. As an installer, I want separate platform and workspace status, so that I know whether I can already ask a real question.
38. As an installer, I want stop/start behavior checked, so that a successful first run is not the only working state.
39. As a maintainer, I want an explicit release build-and-publish command, so that consumer installation can use actual Docker Hub artifacts.
40. As a maintainer, I want versioned images and release metadata, so that published artifacts can be traced to a source revision.
41. As a maintainer, I want publication credentials isolated from consumer configuration, so that installers need no write access to Docker Hub.
42. As a maintainer, I want the published artifacts tested by pulling them, so that an unpublished local image cannot satisfy release acceptance.
43. As a source-build user, I want an explicit supported build path, so that source customization remains possible.
44. As a maintainer, I want Windows, Omarchy and macOS acceptance recorded separately and in order, so that support claims reflect tests actually performed.
45. As an Italian or English reader, I want matching step-by-step guides, so that language choice does not change the installation contract.

## Implementation Decisions

### Boundaries and authority

- Keep the existing standalone architecture and operator CLI. The application runs
  through Compose; the operator CLI owns preparation, validation and execution of
  the installation, rather than introducing another installer or backend workflow.
- Preserve Workspace schema v4 and the workspace catalog as the authority for
  workspace identity and optional Evidence. Formal validation uses the same rules
  as runtime loading, including strict YAML interpretation, duplicate detection,
  directory/index consistency and prohibited fields.
- Preserve the Installation Model Catalog as the sole authored source for provider,
  model eligibility, defaults and embedding facts. Session and embedding configuration
  must be complete before ordinary execution. Metadata generation remains optional
  when its catalog configuration is omitted consistently.
- Preserve the PostgreSQL Metadata Catalog as the authority for Workspace Database
  identity, schema, metadata and active Database Binding. Respect the existing
  one-database-per-workspace boundary. Prepared binding documents are bootstrap
  inputs, not a second runtime database catalog.
- Preserve installation-local secret storage, reference-based binding credentials,
  editable Evidence authority, read-only DWH access and the separation of reference
  preprocessing from Memory. Do not rewrite model projections or runtime snapshots
  as independent authored configuration.

### Preparation documents and command surfaces

- Expose distinct operator operations for template preparation, workspace validation,
  host checks, application validation, execution, status and resumption. Their exact
  CLI spelling can be finalized with the implementation tickets; they must remain
  separately invocable and scriptable. Setup execution is never a parameter wizard.
- Template preparation creates explicitly requested sample documents or protected
  credential files without starting application services. It never silently replaces
  existing user documents. Placeholders are visibly incomplete and cannot pass the
  required-field checks.
- Supply a versioned, installation-local database bootstrap document describing the
  workspace identity, engine, physical database/schema, transport and endpoint
  configuration, and references to secrets. Validate against existing Catalog
  capabilities and selected workspace identities. No credentials belong in the
  shared workspace descriptors or public repository.
- Binding imports use authenticated, authorized Catalog services and their version
  checks. On first execution they create the declared bindings and install secrets
  through the existing store. On rerun, equivalent values are a no-op; conflicting
  existing administrative changes stop with a reconciliation report. The bootstrap
  input does not continuously overwrite a mutable Catalog.
- Initial local administrative authentication and its protected bootstrap material
  are prepared before execution. Execution cannot depend on a person answering a
  login wizard. Reuse supported operator authentication boundaries and required
  permissions, without introducing a privileged unauthenticated installation API.
- Supply a precompiled validation capability with the operator distribution. The
  workspace validation step must work without Docker, ThothII, Node or an application
  source checkout. Reuse canonical validation rules; if a new packaging boundary is
  necessary, prove equivalence with a shared set of valid and invalid documents.

### Validation contract

- Workspace validation covers syntax, strict schema, identity, catalog/descriptor
  relationships and locally available Evidence references. It does not claim to
  establish the truth of domain rules, database contents or services that do not exist.
- Host checks distinguish host prerequisites from bundled application dependencies.
  Pi is checked as a release component and subsequently in the running core image,
  never required as a separate host installation.
- Application validation checks required configuration, compatible release and host,
  model usages/defaults, protected secret references, database transport capabilities,
  workspace links, effective Compose configuration and accessible remote dependencies.
  A transport that cannot serve NL-to-SQL sessions cannot pass workspace-readiness
  validation merely because it can perform administrative diagnostics.
- Reports expose a stable outcome, check identifier, affected logical input/field,
  explanation and next action. The public outcomes are passed, error, warning and
  deferred-to-runtime. Human-readable output is accompanied by pristine structured
  output for automation; exit status distinguishes success from blocking failure.
- Validation is read-only with respect to user documents, application state and
  target databases. Explicit report output is allowed. Remote checks are bounded,
  documented and non-mutating; any provider usage incurred by a configured smoke
  test is disclosed before invocation, not requested interactively during setup.
- Missing or invalid required configuration, missing release artifacts, unsupported
  architecture and failures of available required dependencies block execution.
  Unreachable existing external services are errors, not automatically reclassified
  as deferred. Only checks intrinsically dependent on the not-yet-created local
  stack qualify for the accepted deferred category.
- Maintain an explicit obligation list for deferred checks: container-network
  connectivity, Catalog initialization, Pi operation, local embedding availability,
  preprocessing and relevant workspace runtime readiness. Each obligation has a
  defined runtime check; there is no successful final state while a required
  obligation remains unverified or failed.
- Bind the execution plan to normalized non-secret configuration, repository revision
  and verified content, selected release digests and validator version. Re-read
  protected credentials when checking or executing; do not expose their values or
  unkeyed secret-derived fingerprints in reports. Re-run credential checks where
  freshness cannot be established safely.
- Revalidate changed dependencies and live prerequisites at execution or resume.
  A previously successful report is not blanket authorization to apply changed files
  or evidence of current network availability.

### Release production and distribution

- Provide a maintainer command accepting source revision, release version, Docker Hub
  namespace and target architectures. It performs preflight checks, reproducible
  builds, artifact checks, publication and output of a coherent release manifest.
  This command is a deliverable of this feature, not an undocumented manual prerequisite.
- Publish the existing core and frontend application images. Catalog migration and
  workspace maintenance use the same released core image. Keep PostgreSQL, Qdrant
  and Ollama as compatible upstream images; preserve the existing service boundaries.
- Package the operator executable, validation capability, Compose definitions,
  initialization resources and migration support required by the release. No runtime
  mount may require a resource that exists only in an application source checkout.
- Pin the release identity and resolved image digests. A published release manifest
  binds compatible images and operator/configuration versions. Do not overwrite an
  already published immutable release version or declare a partially published
  image set installable. An interrupted publication can retry without advertising
  an incomplete consumer release.
- Keep publishing credentials outside consumer bundles and logs. Public consumers
  pull without publishing rights. Application images contain application software,
  not installation secrets, user workspace data or prepopulated example databases.
- The ordinary bootstrap downloads a precompiled operator and release artifacts.
  It must not compile the CLI through Docker as a hidden fallback. Windows WSL2
  uses the Linux executable; macOS uses an appropriate host executable.
- Deliver and accept Linux amd64 images for Windows/WSL2 and Omarchy first. Add and
  accept Linux arm64 for the macOS Apple Silicon stage. Multiarchitecture build
  results do not by themselves prove host installation acceptance.
- A maintainer smoke test pulls the published artifacts by their release references.
  Consumer acceptance runs must not succeed because of an unpushed locally built
  image. Confirm core, frontend and maintenance references all resolve to the release.
- Retain explicit source mode with the same configuration, validation and persistence
  rules. It is not the default and is never an automatic recovery action for a pull
  failure. Registry recovery is a retry of the selected released artifacts.

### Non-interactive execution and recovery

- Execute only a complete, currently validated plan with no blocking errors. The
  ordinary sequence pulls the release, prepares runtime projections and isolated
  installation storage, starts required services, applies migrations, registers the
  workspace source and imports the prepared Catalog bindings before dependent work.
- Apply configuration and migrations through existing service boundaries. Hold an
  installation execution lock to prevent concurrent runs from racing over the same
  state, containers or bootstrap imports.
- Reuse durable Catalog Sync Runs and their freshness/locking rules. Fresh additive
  synchronization can proceed under the existing contract. A destructive diff or
  another domain-required human decision stops at an explicit awaiting-review state;
  the operator reviews through the existing administration surface and subsequently
  resumes. This is a domain decision, not permission to collect missing setup parameters
  or add an automatic confirmation bypass.
- Reuse existing curated metadata and Evidence. Do not generate AI descriptions or
  new domain rules as an installation side effect. Optional generation remains a
  separate explicit administrative action. Required preprocessing can use supported
  source comments or curated descriptions without mandatory AI generation.
- Persist a bounded execution journal with installation identity, plan identity,
  stage outcomes, released component versions, deferred-check outcomes and recovery
  guidance. Do not persist secret values or raw exception output. Write progress
  atomically and check actual state on resume.
- A repeated execution of the same completed plan must not recreate bindings,
  duplicate data, clear Memory, overwrite Evidence or erase sessions. Reconcile
  already completed stages with their actual persistent state before proceeding.
- After a document correction, revalidate and repeat only affected checks/stages.
  Do not infer that an interrupted migration or import failed before inspecting its
  durable result. Preprocessing, which has no internal resume contract, may need to
  rerun as a whole; report that honestly.
- Never implement recovery by deleting all volumes or reverting user data. Report
  the failing stage and safe next action. Failed or interrupted execution is not
  advertised as a complete installation.
- Report platform state, Workspace Readiness and functional acceptance separately.
  Runtime checks exercise the released Pi, embedding and actual container transport.
  Final acceptance includes a real human-reviewed question and stop/start persistence.
  The workflow's human review is not replaced by unattended benchmark evaluation.

### Documentation and delivery boundaries

- Keep Italian and English guides aligned with the six steps. Each step states its
  inputs, documents, examples, verification operation, expected output and common
  corrections. Provide an advance checklist of information and credentials to collect.
- Separate maintainer publication instructions, consumer prebuilt installation and
  source-build instructions. State precisely which components are host prerequisites
  and which are shipped inside the release.
- First acceptance uses an available project-specific repository and database.
  The Financial, European Football and F1 project stays deferred. Do not expose its
  unavailable loader, repository-copy support or data bundles as usable features.
- Preserve the future integration boundary: examples will be selected in documents
  and loaded locally after application initialization, without rebuilding the
  application or embedding the datasets into its Docker images.

## Testing Decisions

### Confirmed test boundaries

Use the public operator workflow as the principal test boundary: prepared documents
in, stable reports/exit statuses and observable installation outcomes out. Exercise
validation and execution through this boundary while replacing external command
execution and remote services with controllable test counterparts. Keep focused
contract tests at existing workspace parsing and Catalog boundaries where they
prevent divergent schemas or authority rules. Add real release and host acceptance
tests for behavior that simulated external services cannot establish.

The project owner confirmed this testing boundary on 28 September 2026, completing
the `/to-spec` checkpoint. The product decisions, six-step workflow and testing
scope are approved for specification publication and subsequent ticket decomposition.

### Existing testing practice to extend

- Operator setup tests already use temporary installation fixtures and a replaceable
  command runner to cover sequencing, startup failures, error preservation and recovery
  messages. Extend that boundary to document validation, prebuilt execution and resumption.
- Installation configuration tests cover strict schemas, model catalog rules and
  incompatible existing files. Extend them with complete/incomplete preparation
  documents, protected references and cross-document consistency.
- Workspace tests cover strict catalog/descriptor parsing, immutable Git revisions,
  runtime handoff and secret handling. Reuse their document fixtures and validity
  rules to demonstrate equivalence of the preinstallation validator.
- Catalog and preprocessing tests already cover durable runs, locks, binding freshness,
  failure recording and preservation of authoritative state. Reuse these boundaries
  to verify bootstrap import and resumption without bypassing the domain contracts.
- Existing multiarchitecture image checks provide a starting point for released
  artifact verification. They do not replace pulling the published artifacts or
  testing supported host environments.

### Required behavioral coverage

1. Validate workspace documents with Docker absent and no application runtime;
   reject malformed YAML, duplicate keys/identifiers, unsupported fields, broken
   references and invalid configured Evidence with actionable locations.
2. Repeated document/host/application checks do not create containers, change source
   documents, import data or migrate databases. Only explicitly requested reports
   may be written by validation.
3. Complete application documents pass; placeholders, missing required models,
   invalid binding transport and unreadable secret references block execution.
4. Existing external-service failures remain errors. Checks genuinely dependent on
   newly created local services are listed as deferred and cannot disappear from
   final acceptance.
5. Execute with standard input closed. Valid inputs need no responses; missing
   values yield an error without waiting for input or prompting for a replacement.
6. Changing documents, workspace revision or release after validation invalidates
   dependent results. Credential changes are caught without leaking secret material.
7. A clean prebuilt consumer installation performs pulls and initialization, never
   application or operator compilation; absent images fail without a source fallback.
8. Published manifests resolve all required images, platform variants and maintenance
   components coherently. Simulated publication interruption does not advertise a
   partial release; a real smoke test exercises artifacts pulled from Docker Hub.
9. Prepared database bindings become Catalog state once, use the existing protected
   secret store and remain unchanged on equivalent reruns. Administrative divergence
   is reported rather than silently overwritten.
10. Interruption after a durable operation but before journal completion resumes by
    inspecting state, without duplicating that operation. Concurrent setup runs cannot
    mutate the same installation simultaneously.
11. Destructive Catalog synchronization requires its existing review and fresh source
    checks. The setup's non-interactive nature does not auto-approve a destructive diff.
12. Existing descriptions, Evidence, sessions and Memory survive validation, rerun,
    configuration correction and stop/start. Optional AI generation is not triggered.
13. Runtime Pi, embedding and connectivity checks are executed from the installed
    release, and platform success cannot mask failed Workspace Readiness.
14. Source mode remains functional and explicit with equivalent configuration
    contracts. A source-mode pass cannot close prebuilt distribution acceptance.
15. Execute real consumer acceptance first on Windows x64/WSL2, then Omarchy x64,
    then macOS Apple Silicon. Record each environment, release identity, stage
    outcomes, a real reviewed question and a stop/start check separately.

Good tests assert observable contracts, preserved data and required side effects,
not private helper calls or incidental internal ordering. Use real isolated Catalog
instances where transaction and lock behavior matters. Mock external provider
failures for repeatable tests, while keeping actual DWH/model acceptance separate.

## Out of Scope

- Implementing or publishing the three example databases, their curated contents,
  example CLI, auxiliary repository layout or autonomous repository-copy mode.
- A new graphical installer, a conversational parameter wizard, or collecting
  required parameters in administration pages after an incomplete setup.
- Installing Pi, Python, Node or an application build toolchain on ordinary consumer hosts.
- Changes to server/Omics deployment, upstream authentication or the application's
  existing human-in-the-loop workflow.
- Multiple workspace repositories per installation or multiple databases per workspace.
- Automatic release upgrades, destructive reset/uninstall, whole-volume rollback
  or backup-policy redesign. Interrupted initial setup recovery remains in scope.
- Automatic semantic certification, generated Evidence without sources, benchmark
  SQL targets or automated accuracy scoring.
- Publishing Docker images or executing installations as part of this specification
  authoring task. These are implementation and release deliverables described above.

## Further Notes

The project owner approved the six-step revision and both final clarifications on
28 September 2026: examples stay deferred, and runtime-only checks are explicit
post-start obligations. Earlier interactive-wizard and incomplete-configuration
installation proposals are superseded where they conflict with this specification.

This specification follows the existing decisions on PostgreSQL metadata authority,
installation-local bindings, secret references, durable schema synchronization and
the Installation Model Catalog. It does not change those architectural authorities.

Publication of a usable Docker Hub release is a blocking dependency of consumer
prebuilt-installation acceptance. Availability of the deferred examples is not.
Docker Hub namespace, publishing credentials and concrete release versions are
maintainer release inputs, not values to invent or embed into user templates.

After publication to Gitea, `/to-tickets` will split this specification into small
end-to-end increments with explicit blockers. Implementation has not started in
this task, and publishing the specification does not attest that a release exists.