ThothII/docs/install/standalone-manual-en.md

# Guided standalone installation

[Versione italiana](standalone-manual-it.md)

This is the fresh-machine installation procedure for THothII. THothII receives a natural-language
question, queries an enterprise database read-only, and guides the user through SQL review. The
core, PostgreSQL catalog, Qdrant, embedding service, and Pi run in Docker; Node.js, Python, and Pi
are not required on the host.

## Prepare and validate workspace documents before starting the stack

The first two steps of the new flow work without Docker, Node, Python, Pi or an
installation descriptor. Use the platform bundle with **both** `tht` and
`tht-workspace-documents` in the same directory (`.exe` on Windows). Put that directory
on `PATH`, or invoke the absolute executable path. Older packages containing only
`tht` do not provide this capability. Maintainers can currently build the bundle;
publishing assets and Docker Hub images belongs to a later delivery step. The rest
of this guide still describes the existing installation path.

1. Choose a new directory outside the application checkout, with an existing parent:

   ```sh
   tht workspace prepare --directory ./my-workspaces --id practice --name "Practice" --language en
   ```

   This creates `thoth-workspaces.yaml`, `practice/workspace.yaml` and
   `workspace-docs/practice/README.md`. Existing destinations, even empty ones, are
   refused. No services start, Git is not initialized and no remote is contacted.
   Example databases remain a deferred subproject. For a curator-supplied repository,
   use a separate local copy and go straight to step 3; read access to the origin is
   sufficient.
2. Edit the catalog (schema v1) and workspace descriptor (schema v4) at your own pace.
   Keep `id`, `name` and optional `description` identical in both; the id must match
   the directory name. Root directories must match catalog entries, except
   `workspace-docs` and the local `.git` directory. Database connections, schema and
   credentials belong to the installation Metadata Catalog. Evidence is optional
   and initially absent.
3. Validate, correct the reported document/field, and repeat:

   ```sh
   tht workspace validate --directory ./my-workspaces
   tht workspace validate --directory ./my-workspaces --json
   ```

   PowerShell uses the same arguments, for example
   `C:\ThothII\bin\tht.exe workspace validate --directory C:\ThothII\my-workspaces`.
   Validation changes no files. It rejects multiple/malformed YAML documents,
   duplicate keys/ids, unknown fields, catalog/directory/descriptor mismatches,
   missing local references and symbolic links. Fix the first error in each document
   and repeat to reveal any subsequent errors.

Evidence `absent` is valid. Filesystem checks cover directories, accessibility and
literal references; standard Markdown selections also check declared size limits.
Evidence v2 requires `curated/` and syntactically valid YAML frontmatter. Local limits
are 1 MiB per document read and 100,000 entries per Evidence tree. Arbitrary patterns,
the complete curated-unit contract, provenance, HTTP/S3 access and indexing remain
explicit runtime checks. See the [Evidence guide](../evidence.md).

JSON includes `schema_version`, `scope: local-documents`, `ok`, `workspaces`, `issues`
and `deferred_checks`. Issues identify document, field, code, correction and YAML
line where available, without printing document values. Exit statuses: `0` local
success, `1` documents/access/bundle need correction, `2` invalid arguments. Local
success does not certify semantic truth, connectivity or readiness. The Git revision
activated later must contain the checked documents; this command does not publish
uncommitted files or empty directories.

## Prepare and validate application documents

After validating workspaces, create a local directory **outside their repository**:

```sh
tht installation prepare --directory ./my-installation
```

This creates private, commented `thothii-installation.yaml`, `operator.env`,
`database-bootstrap.yaml` and `README.md`. The destination must be new and its parent
must exist. It starts no services and does not implicitly generate passwords.

1. Choose models and providers in the descriptor. The template proposes
   `openai/gpt-4.1-mini` for interaction and `ollama/qwen3-embedding:0.6b` with 1024
   dimensions for embedding. Edit these before setup. `modelCatalog.defaults.interaction`
   must support sessions and metadata generation when the latter is configured.
   The template omits optional metadata generation. See
   [model configuration](../general/pi-configuration.md) for custom providers.
2. Replace the Git remote in both the descriptor and `operator.env`; keep branch and
   transport consistent. Paths are absolute and machine-local. `operator.env` accepts
   one literal `KEY=value` assignment per line, without duplicate keys or shell
   interpolation. Credentials belong in referenced protected files.
3. Complete `database-bootstrap.yaml` with exactly one entry per workspace. A complete
   direct-connection example is:

   ```yaml
   schemaVersion: 1
   databases:
     - workspaceId: practice
       engine: postgres
       databaseName: sales
       schema: public
       binding:
         transport: postgres_direct
         host: db.intranet
         port: 5432
         username: thoth_reader
       secretFiles:
         password: /private/path/my-installation/secrets/database-password
   ```

   Use a read-only DWH account. `rest_api` requires `baseUrl`, `restPath`, `restAuth`
   (`none`, `bearer`, `x-api-key`) and `secretFiles.apiKey` when authenticated.
   `ssh_tunnel` requires `username`, `sshHost`, `sshPort`, `sshUsername`,
   `sshTargetHost`, `sshTargetPort` and files `password`, `sshPrivateKey`,
   `sshKnownHosts`; it supports Catalog diagnostics, not NL-to-SQL sessions.
   `tlsCa` and `sshPrivateKeyPassphrase` are optional. Signed HTTP Evidence needs
   `evidenceSecretFiles` with key `evidence.signed_urls`; static S3 credentials need
   `evidence.access_key`, `evidence.secret_key` and optional `evidence.session_token`.
   All values are private file paths. Workspace descriptors remain schema v4;
   this bootstrap input is not a second runtime Catalog.
4. Explicitly generate technical credentials in the standard layout:

   ```sh
   tht installation credentials --directory ./my-installation
   ```

   This creates separate random Catalog runtime/migrator and administrator passwords,
   `auth/auth.yaml`, `auth/users.yaml`, a `secrets/secrets.env` template and
   `secrets/pi-auth.json`. Existing files are retained; invalid ones stop the command.
   The initial administrator is `admin`; its password stays in private
   `secrets/admin-password` and is never printed. The default is local authentication
   at `http://localhost:8080`: review and edit `auth/auth.yaml` before validation.
   This increment does not validate offline OIDC bootstrap for the existing path.
5. Fill the provider key in `secrets/secrets.env` and create the DWH password file.
   For Git HTTPS supply the referenced credentials and CA files; empty credentials
   are allowed for a public remote, and the CA file must be available. For SSH supply
   a key and known_hosts and select the matching descriptor override. `pi_auth`
   providers require prepared Pi credentials. Keep every secret outside workspace
   Git with installer-only access (0600 on Unix, equivalent Windows ACLs).
6. Validate and repeat after each correction:

   ```sh
   tht --installation /absolute/path/my-installation/thothii-installation.yaml installation validate --workspaces /absolute/path/my-workspaces --json
   ```

   The default bootstrap is beside the descriptor; `--bootstrap PATH` selects another.
   Validation changes no documents, generates no projections, uses no network and
   writes no database. It rejects placeholders, inconsistencies, missing/non-private
   files and secrets inside workspace Git. Reports identify document, field and
   correction without secret values. Exit statuses: 0 local success, 1 corrections
   needed, 2 invalid arguments.

Standard release Compose assets may still be absent at this stage; custom overrides
must already exist. Release assets, external connectivity, Catalog import and runtime
readiness remain explicit deferred checks. Success prepares the next preflight;
it neither skips those checks nor establishes a completed installation.

## Check prerequisites and produce the plan

At step 3, before completing all application parameters, check the machine and
the private installation directory already prepared:

```bash
tht installation preflight --directory /path/installation --json
```

This requires a reachable Linux Docker daemon, Compose 2.24 or newer, at least
2 CPUs, 4 GiB allocated to Docker and 10 GiB free on the installation filesystem.
A release may require more resources. On Windows run the Linux executable in
Ubuntu WSL2 with Docker Desktop integration; Pi is bundled in the core image.

At step 5, after `installation validate`, select the published release manifest
with its downloaded bundle resources and produce a new plan:

```bash
tht --installation /path/installation/thothii-installation.yaml installation plan \
  --workspaces /path/workspaces \
  --release /path/release/release-manifest.json \
  --output /path/installation/installation-plan.json --json
```

This repeats document checks, verifies image digests and Compose, Git, available
external databases and Evidence, then saves the plan and its separate private
`.key` file. It does not execute setup. Missing images and unavailable existing
dependencies block the plan. Actual Docker Hub publication remains the next ticket;
an invented manifest cannot bypass publication.

Correct `error` outcomes and read `warning` outcomes. `deferred-to-runtime` entries
are mandatory checks after startup, not readiness already achieved. After changing
documents or rotating credentials, produce a new plan; existing files are never
overwritten. Keep both plan files outside workspace Git. External probes perform
bounded database authentication/schema reads, Git/HTTP/S3 reads and explicit model
endpoint reachability checks. They invoke no LLM generation; HTTP/S3 requests may
incur ordinary service request charges. See the
[preflight reference](installation-preflight.md) for limits, the manifest
format and runtime obligations.

## Before you start: the two repositories

There are two separate repositories:

1. the application repository cloned by the user:
   https://git.tylconsulting.it/mptyl/ThothII.git;
2. the workspace repository supplied by the curator/installer. It is not the THothII repository
   and must not be cloned inside the application directory.

The workspace repository normally contains:

~~~
thoth-workspaces.yaml
<workspace-id>/workspace.yaml
<workspace-id>/evidence/**       # when Evidence is declared
~~~

workspace.yaml contains workspace identity, language and optional Evidence source. By design it does
not contain database passwords. Database identity, transport (PostgreSQL, REST, or tunnel), user,
password, token and certificates are installation-local settings stored encrypted by the Catalog.
This prevents credentials from being committed to the workspace repository.

## 0. Machine prerequisites

### Windows

- Windows 10/11 with Docker Desktop running and the WSL2 backend enabled.
- Ubuntu in WSL2, with Docker Desktop integration enabled for that distribution.
- Git, Bash, curl, OpenSSL and shasum inside WSL2.
- Do not install Node.js, Python or Pi on the host for this procedure.

If WSL2 is not installed, use the company procedure or, in PowerShell:

~~~
wsl --install -d Ubuntu
~~~

Run all commands inside Ubuntu WSL2, in a Linux directory such as $HOME/src, not under /mnt/c.
scripts/install-tht.ps1 exists for advanced native PowerShell scenarios; use WSL2 for the
reproducible test.

### macOS

- Docker Desktop installed and running, with several GB free for images and the embedding model.
- Git, Bash, curl, OpenSSL and shasum.
- Intel and Apple Silicon Macs are supported when Docker Desktop supports the architecture
  reported by the Docker server.
- Do not install Node.js, Python or Pi on the host for this procedure.

### Linux, including Omarchy

- Git, Bash, curl, OpenSSL and shasum.
- Docker Engine and the Docker Compose v2 plugin. On Omarchy, check first:

~~~
command -v docker
docker compose version
docker info
~~~

If Docker is missing, install Docker and Compose using the distribution-approved package procedure,
then start the service. On an Arch-like distribution the typical route is:

~~~
sudo pacman -S docker docker-compose
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"
~~~

After adding the group, open a new session and repeat docker info. Node.js, Python and Pi are not
needed on the host: they are in the Docker images.

On every system run:

~~~
bash scripts/check-standalone-prerequisites.sh
docker version --format '{{.Server.Arch}}'
~~~

The architecture must be amd64, x86_64, arm64, or aarch64. You also need access to the
application Gitea repository, the workspace repository URL/branch and credentials, container
reachability to DWH/LLM endpoints, and the credentials, tokens or certificates associated with
the databases.

## 1. What to clone

Clone only the application:

~~~
mkdir -p "$HOME/src"
cd "$HOME/src"
git clone https://git.tylconsulting.it/mptyl/ThothII.git
cd ThothII
git rev-parse --short HEAD
~~~

Record the revision. tht setup --complete downloads the workspace repository into a persistent
Docker volume using the URL, branch and transport supplied during setup.

## 2. Install the terminal command

From the clone root:

~~~
bash scripts/check-standalone-prerequisites.sh
mkdir -p "$HOME/.local/bin"
THT_INSTALL_DIRECTORY="$HOME/.local/bin" bash scripts/install-tht.sh
export PATH="$HOME/.local/bin:$PATH"
tht version
~~~

tht is the only native component to install. It builds the binary with Docker and orchestrates
Compose; it is not a second application runtime.

## 3. Prepare a few secrets and run complete setup

The first execution creates protected placeholders under deploy/local/secrets/ and stops if a
required credential is missing. Fill in the requested files and rerun the same command; compatible
configuration files are reused.

~~~
tht setup --complete --profile local --shell-mode full --shell-default-locale en
~~~

The setup asks only for information the computer cannot know:

| Request | What to provide |
| --- | --- |
| Workspace repository | Data/configuration repository URL, not ThothII.git |
| Branch | normally main |
| Access | ssh with key and known_hosts, or https with credential file and CA |
| DWH/LLM URL | endpoint without a token in the URL |
| Local login | initial user and password requested by the prompt |

The setup generates random Catalog passwords and writes their paths, never their values, to
operator.env. It runs docker compose config, builds images, starts the Catalog, runs
catalog-migrate, starts the stack, and pulls the workspace repository. The pull also activates
declared Evidence; at minimum source files present in the workspace are materialized locally.

### The file the user fills in

The main file is:

~~~
deploy/local/secrets/thothii.secrets
~~~

Add only NAME=VALUE lines needed by modelCatalog and installation adapters, such as an LLM API key
(DEEPSEEK_API_KEY, OPENAI_API_KEY, or the key declared by the catalog) and, when applicable,
THT_DWH_API_KEY. Allowed names are documented in deploy/secrets/README.md. Never put tokens in
URLs, the repository, or copied shell commands.

Two distinctions prevent common errors:

- when the catalog uses pi_auth, the LLM token belongs in the Pi pi-auth.json file created by
  setup; {} is only a placeholder and does not enable a model;
- workspace database credentials (PostgreSQL password, REST API token, tunnel SSH key,
  known_hosts, CA) do not belong in the workspace repository. Enter them per workspace in
  Database Management, which stores them encrypted in the Catalog. The workspace declares
  database/schema and transport; the installer must obtain the actual values from the database owner.

A private workspace repository also needs the Git files required by its transport: an SSH key and
known_hosts, or an HTTPS credential file and CA. These are transport files, not a second bundle to
commit. To minimize manual files, use SSH with an already-authorized deploy key.

## 4. Automatic checks and terminal tests

Setup verifies files, permissions, descriptor, Compose, Docker, authentication, services, Pi and
the workspace. After startup, run these commands at any time:

~~~
INSTALLATION="$PWD/deploy/local/thothii-installation.yaml"
tht --installation "$INSTALLATION" doctor --json
tht --installation "$INSTALLATION" workspace pull --json
tht --installation "$INSTALLATION" workspace test --json
~~~

workspace test checks, for every active workspace, database binding and credentials, Evidence,
Qdrant, and the embedding service. It exits non-zero when the database binding is missing or a
connection is unusable. Before running it, the installer must configure the database in Database
Management: the workspace repository cannot contain the password by itself.

doctor --json is the repeatable, non-destructive core verification. The final functional test must
also open http://127.0.0.1:8080, sign in, and complete a real question through final SQL.

## Activities only the installer can complete

The procedure automates bootstrap, but it cannot invent enterprise decisions or authorizations.
The installer must complete and record:

1. usable LLMs, the modelCatalog, and linked API keys; then run tht pi test and tht doctor;
2. database configuration, connection test, schema synchronization, and description generation;
3. human consolidation of generated descriptions;
4. Qdrant semantic entries through workspace preprocess run;
5. naming-based FK suggestions as a complement to schema FKs, human review, and loading approved
   relationships into Qdrant;
6. recurring tht doctor --json and tht workspace test --json checks;
7. one real question completed successfully without connection or model errors.

Configuration is complete only when all applicable activities are done, decisions are recorded, and
the two terminal tests are green. The core is usable only after the real question, not merely
because the frontend answers /health.

## Gate A and Gate B

### Gate A — platform

~~~
uname -a
docker version --format '{{.Server.Version}} {{.Server.Arch}}'
tht version
bash scripts/check-standalone-prerequisites.sh
bash scripts/verify-standalone-install.sh "$INSTALLATION"
~~~

### Gate B — usability

~~~
tht --installation "$INSTALLATION" doctor --json
tht --installation "$INSTALLATION" workspace test --json
curl --fail --silent --show-error http://127.0.0.1:8080/health
~~~

Then run a real question and stop/restart with tht stop and tht start. Do not use docker compose
down --volumes: it deletes the Catalog, sessions, Qdrant data and the embedding model.

## Quick diagnosis

| Symptom | Action |
| --- | --- |
| Docker Engine is not reachable | start Docker Desktop or systemctl and repeat docker info |
| Omarchy cannot find docker | install Docker/Compose, enable the service and open a new session |
| Windows sees Docker but Bash fails | use Ubuntu WSL2 and enable its Docker Desktop integration |
| workspace pull fails | check URL, branch, key/credential file and known_hosts from the container |
| workspace test reports a missing binding | configure database, token/password and CA in Database Management |
| Pi is not ready | fill pi-auth.json or the key declared by modelCatalog, then run tht pi test |

## Related documents

- [Install and first start](first-start.md)
- [Workspace operations](../operations/workspaces.md)
- [Database Management](../operations/database-management.md)
- [Model configuration](../general/pi-configuration.md)
- deploy/secrets/README.md

Publishing images on Docker Hub and native DMG/MSI/AppImage installers remain later work: this
procedure starts from the Gitea clone and does not require pre-published Docker Hub images.