mptyl/ThothII
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

wip: guided standalone installation and workspace checks

Browse Source
This commit is contained in:
Codex
2026-09-26 16:41:15 +02:00
parent 0d2e573e0d
commit 67ee52624c
15 changed files with 902 additions and 560 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/install/standalone-manual-en.md
+176 -244
View File
@@ -1,324 +1,256 @@
# Manual standalone installation
# Guided standalone installation
[Versione italiana](standalone-manual-it.md)
This is the verification procedure for preparing THothII as a standalone application in `full`
mode on macOS, Windows, and Linux.
This is the fresh-machine installation procedure for THothII. THothII receives a natural-language
question, queries an enterprise database read-only, and guides the user through SQL review. The
core, PostgreSQL catalog, Qdrant, embedding service, and Pi run in Docker; Node.js, Python, and Pi
are not required on the host.
In this document, “standalone” means that the user does not need to install Node.js, Python or Pi
on the host: the application services and local semantic
services run through Docker. DWH and LLM providers remain external endpoints configured by the
installation; this is not an offline package.
## Before you start: the two repositories
This path uses no graphical installer, native launcher, or Docker Hub image. It starts from a Gitea
clone and uses explicit terminal commands. Publishing pre-built images is a later step.
There are two separate repositories:
## Verification matrix
1. the application repository cloned by the user:
https://git.tylconsulting.it/mptyl/ThothII.git;
2. the workspace repository supplied by the curator/installer. It is not the THothII repository
and must not be cloned inside the application directory.
| System | Recommended terminal | Runtime | Test architecture |
| --- | --- | --- | --- |
| macOS supported by the installed Docker Desktop version | Bash in Terminal | Docker Desktop | Apple Silicon (`arm64`) |
| Windows 11 | Ubuntu inside WSL2 | Docker Desktop with WSL2 integration | x64 (`amd64`) |
| Ubuntu Linux 22.04 or 24.04 | Bash | Docker Engine + Compose v2 | x64 (`amd64`) |
The workspace repository normally contains:
Intel macOS is excluded from the first verification campaign. ARM Linux may be tested when the
machine’s Docker runtime reports `arm64`, but it is not part of the minimum matrix.
~~~
thoth-workspaces.yaml
<workspace-id>/workspace.yaml
<workspace-id>/evidence/** # when Evidence is declared
~~~
Documentation and Mac prerequisite checks have passed. Complete fresh installations on all three
systems remain pending; this matrix describes the tests to perform, not completed certification.
workspace.yaml contains workspace identity, language and optional Evidence source. By design it does
not contain database passwords. Database identity, transport (PostgreSQL, REST, or tunnel), user,
password, token and certificates are installation-local settings stored encrypted by the Catalog.
This prevents credentials from being committed to the workspace repository.
## Before you start
## 0. Machine prerequisites
You need:
### Windows
- access to the THothII Gitea repository and the workspace Git repository;
- Git;
- Docker Desktop on macOS and Windows, or Docker Engine with the Compose v2 plugin on Linux;
- Bash, `curl`, OpenSSL and `shasum` (Ubuntu package: `libdigest-sha-perl`);
- enough disk space to build the images and download the embedding model;
- the DWH and LLM endpoints, plus the credentials required by the installation.
- Windows 10/11 with Docker Desktop running and the WSL2 backend enabled.
- Ubuntu in WSL2, with Docker Desktop integration enabled for that distribution.
- Git, Bash, curl, OpenSSL and shasum inside WSL2.
- Do not install Node.js, Python or Pi on the host for this procedure.
On Linux, the current user must be able to run Docker. If the system requires `sudo`, add the user
to the Docker group according to local policy and open a new session before continuing.
If WSL2 is not installed, use the company procedure or, in PowerShell:
On Windows, run every command in this guide from Ubuntu under WSL2. In Docker Desktop, enable WSL2
integration for that distribution. Clone the project inside the WSL2 Linux filesystem, for example
under `~/src`, rather than under `/mnt/c`: this avoids slow builds and path/line-ending issues. Pi
does not need to be installed on the host.
~~~
wsl --install -d Ubuntu
~~~
Check the runtime before or immediately after cloning:
Run all commands inside Ubuntu WSL2, in a Linux directory such as $HOME/src, not under /mnt/c.
scripts/install-tht.ps1 exists for advanced native PowerShell scenarios; use WSL2 for the
reproducible test.
```sh
docker version
### macOS
- Docker Desktop installed and running, with several GB free for images and the embedding model.
- Git, Bash, curl, OpenSSL and shasum.
- Intel and Apple Silicon Macs are supported when Docker Desktop supports the architecture
reported by the Docker server.
- Do not install Node.js, Python or Pi on the host for this procedure.
### Linux, including Omarchy
- Git, Bash, curl, OpenSSL and shasum.
- Docker Engine and the Docker Compose v2 plugin. On Omarchy, check first:
~~~
command -v docker
docker compose version
docker info
~~~
If Docker is missing, install Docker and Compose using the distribution-approved package procedure,
then start the service. On an Arch-like distribution the typical route is:
~~~
sudo pacman -S docker docker-compose
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"
~~~
After adding the group, open a new session and repeat docker info. Node.js, Python and Pi are not
needed on the host: they are in the Docker images.
On every system run:
~~~
bash scripts/check-standalone-prerequisites.sh
docker version --format '{{.Server.Arch}}'
```
~~~
The last command must return `amd64`, `x86_64`, `arm64`, or `aarch64`.
The architecture must be amd64, x86_64, arm64, or aarch64. You also need access to the
application Gitea repository, the workspace repository URL/branch and credentials, container
reachability to DWH/LLM endpoints, and the credentials, tokens or certificates associated with
the databases.
## 1. Clone a project revision
## 1. What to clone
Use the project repository on Gitea:
Clone only the application:
```sh
~~~
mkdir -p "$HOME/src"
cd "$HOME/src"
git clone https://git.tylconsulting.it/mptyl/ThothII.git
cd ThothII
git rev-parse --short HEAD
```
~~~
For an SSH clone, when the key is already authorized on Gitea:
Record the revision. tht setup --complete downloads the workspace repository into a persistent
Docker volume using the URL, branch and transport supplied during setup.
```sh
git clone git@git.tylconsulting.it:mptyl/ThothII.git
```
Record the hash printed by `git rev-parse` for a repeatable test. In a later campaign, use the
maintainer-approved revision/tag rather than implicitly following a mutable `main` branch.
## 2. Check prerequisites and install the operator command
## 2. Install the terminal command
From the clone root:
```sh
~~~
bash scripts/check-standalone-prerequisites.sh
export PATH="$HOME/.local/bin:$PATH"
mkdir -p "$HOME/.local/bin"
THT_INSTALL_DIRECTORY="$HOME/.local/bin" bash scripts/install-tht.sh
export PATH="$HOME/.local/bin:$PATH"
tht version
```
~~~
`install-tht.sh` bootstraps only the native `tht` operator command; it does not install a desktop
version of THothII. It uses the repository’s Docker builder, installs the binary for the current
terminal environment, and installs it in the user directory. Persist `$HOME/.local/bin` in your
shell PATH for new terminals too. An existing `tht` in this directory will be updated.
tht is the only native component to install. It builds the binary with Docker and orchestrates
Compose; it is not a second application runtime.
On Windows, run these commands inside WSL2. The installed `tht` binary is the Linux binary inside
WSL2; the application runtime remains Docker Desktop. Do not use `scripts/install-tht.ps1` as the
primary path for this test.
## 3. Prepare a few secrets and run complete setup
## 3. Configure and start the local installation
The first execution creates protected placeholders under deploy/local/secrets/ and stops if a
required credential is missing. Fill in the requested files and rerun the same command; compatible
configuration files are reused.
Run the remaining blocks in one Bash session from the physical clone root (`pwd -P`).
First create two distinct catalog passwords, preserving any existing files:
~~~
tht setup --complete --profile local --shell-mode full --shell-default-locale en
~~~
```bash
umask 077
mkdir -p deploy/local/secrets
for name in catalog-runtime-password catalog-migrator-password; do
target="deploy/local/secrets/$name"
if [ ! -e "$target" ]; then
(set -C; openssl rand -hex 32 > "$target") || exit 1
fi
done
export THT_CATALOG_RUNTIME_PASSWORD_SOURCE="$(pwd -P)/deploy/local/secrets/catalog-runtime-password"
export THT_CATALOG_MIGRATOR_PASSWORD_SOURCE="$(pwd -P)/deploy/local/secrets/catalog-migrator-password"
```
The setup asks only for information the computer cannot know:
Do not regenerate passwords for an initialized catalog. Configure without starting services:
```sh
tht setup --profile local --shell-mode full --shell-default-locale en --configure-only
```
Answer the prompts as follows:
| Prompt | Value or rule |
| Request | What to provide |
| --- | --- |
| Installation ID | `local`, unless one clone hosts multiple installations |
| Deployment profile | `local` |
| DWH API endpoint | An `http(s)` URL without user, password, query, or fragment; may be empty for a smoke-only test |
| LLM API endpoint | An `http(s)` URL without credentials; may be empty for a smoke-only test |
| Workspace repository URL | The workspace repository URL, not the THothII source clone |
| Workspace branch | Normally `main` |
| Workspace access | `ssh` with a deploy key, or `https` with a protected credential file |
| File paths | Accept the default paths under `deploy/local/secrets/` for the first test |
| Secret templates | Answer `yes` when protected files do not exist yet |
| Authentication | Configure the local login required by the installation; never put passwords on a command line |
| Workspace repository | Data/configuration repository URL, not ThothII.git |
| Branch | normally main |
| Access | ssh with key and known_hosts, or https with credential file and CA |
| DWH/LLM URL | endpoint without a token in the URL |
| Local login | initial user and password requested by the prompt |
The generated configuration is local and ignored by Git:
The setup generates random Catalog passwords and writes their paths, never their values, to
operator.env. It runs docker compose config, builds images, starts the Catalog, runs
catalog-migrate, starts the stack, and pulls the workspace repository. The pull also activates
declared Evidence; at minimum source files present in the workspace are materialized locally.
```text
deploy/local/thothii-installation.yaml
deploy/local/operator.env
deploy/local/auth/
deploy/local/secrets/
```
### The file the user fills in
Edit secrets only in protected local files; never commit them. `deploy/env/local.env.example` is a tracked reference; the
generated path `deploy/local/operator.env` is the active path for this installation.
The main file is:
### Complete protected files
~~~
deploy/local/secrets/thothii.secrets
~~~
If setup created blank templates, enter the values with a local editor:
Add only NAME=VALUE lines needed by modelCatalog and installation adapters, such as an LLM API key
(DEEPSEEK_API_KEY, OPENAI_API_KEY, or the key declared by the catalog) and, when applicable,
THT_DWH_API_KEY. Allowed names are documented in deploy/secrets/README.md. Never put tokens in
URLs, the repository, or copied shell commands.
```sh
chmod 600 deploy/local/secrets/*
"${EDITOR:-vi}" deploy/local/secrets/thothii.secrets
```
Two distinctions prevent common errors:
The bundle must contain only `KEY=VALUE` lines for credentials actually used by `modelCatalog`. The
allowed names and credential boundary are documented in the local file
`deploy/secrets/README.md`. Do not put tokens in URLs, the YAML
descriptor, the Git repository, or commands copied into the shell.
- when the catalog uses pi_auth, the LLM token belongs in the Pi pi-auth.json file created by
setup; {} is only a placeholder and does not enable a model;
- workspace database credentials (PostgreSQL password, REST API token, tunnel SSH key,
known_hosts, CA) do not belong in the workspace repository. Enter them per workspace in
Database Management, which stores them encrypted in the Catalog. The workspace declares
database/schema and transport; the installer must obtain the actual values from the database owner.
For SSH workspace access, also provide the private key and `known_hosts` file requested by setup.
For HTTPS access, provide the Git credential file and any required CA. Both must remain protected
and outside version control.
A private workspace repository also needs the Git files required by its transport: an SSH key and
known_hosts, or an HTTPS credential file and CA. These are transport files, not a second bundle to
commit. To minimize manual files, use SSH with an already-authorized deploy key.
Before starting, complete these additional configuration steps:
## 4. Automatic checks and terminal tests
1. Add `THT_CATALOG_RUNTIME_PASSWORD_SOURCE` and `THT_CATALOG_MIGRATOR_PASSWORD_SOURCE` to
`deploy/local/operator.env`, with the same absolute paths exported above. Setup does not persist
these two variables. Store paths, not passwords.
2. Replace the descriptor's generic `modelCatalog` with the approved provider/model configuration.
The generated defaults do not replicate the existing Mac. See [Pi/model configuration](../general/pi-configuration.md)
and the local example `deploy/psd/thothii-installation.yaml.example`.
3. Populate the keys referenced by `authentication.apiKeyEnv` in `thothii.secrets`. Providers using
`pi_auth` need valid credentials at `PI_AUTH_FILE`; the `{}` template is not authentication.
4. Complete the workspace Git files. SSH requires an authorized deploy key and verified known-hosts;
HTTPS requires the credential file and CA bundle expected by the overlay. Blank templates cannot
provide repository access.
Setup verifies files, permissions, descriptor, Compose, Docker, authentication, services, Pi and
the workspace. After startup, run these commands at any time:
After editing generated configuration, do not rerun setup: it rejects different existing content.
Generate the projections and run the explicit migration below. Use `THT_GIT_ACCESS=https` if that
was selected during setup. This block targets the fresh `local` descriptor with only the Git overlay;
custom installations must include their extra descriptor overlays in the same order.
~~~
INSTALLATION="$PWD/deploy/local/thothii-installation.yaml"
tht --installation "$INSTALLATION" doctor --json
tht --installation "$INSTALLATION" workspace pull --json
tht --installation "$INSTALLATION" workspace test --json
~~~
```bash
INSTALLATION="$(pwd -P)/deploy/local/thothii-installation.yaml"
tht --installation "$INSTALLATION" installation generate
THT_PROJECT="thothii-$(printf '%s' "$INSTALLATION" | shasum -a 256 | cut -c 1-12)"
THT_GIT_ACCESS=ssh
compose=(
docker compose --project-name "$THT_PROJECT" --project-directory "$(pwd -P)"
--env-file "$(pwd -P)/deploy/local/operator.env"
-f compose.yaml -f deploy/compose.local.yaml
-f "deploy/compose.git-$THT_GIT_ACCESS.yaml"
-f deploy/local/generated/compose.models.yaml
)
"${compose[@]}" config --quiet
"${compose[@]}" build core frontend
"${compose[@]}" up -d catalog-db
"${compose[@]}" run --rm catalog-migrate
tht --installation "$INSTALLATION" start
```
workspace test checks, for every active workspace, database binding and credentials, Evidence,
Qdrant, and the embedding service. It exits non-zero when the database binding is missing or a
connection is unusable. Before running it, the installer must configure the database in Database
Management: the workspace repository cannot contain the password by itself.
Stop if a command fails. The project name matches the hash used by `tht`, preserving volume
identity. `catalog-migrate` applies Catalog and Memory migrations; `tht start` does not run it
automatically. Initial embedding-model download may take time. Use this installation-specific
sequence, not `run-stack.sh` with a different environment/project name.
doctor --json is the repeatable, non-destructive core verification. The final functional test must
also open http://127.0.0.1:8080, sign in, and complete a real question through final SQL.
## 4. Verify the installation
## Activities only the installer can complete
The descriptor generated for the default ID is:
The procedure automates bootstrap, but it cannot invent enterprise decisions or authorizations.
The installer must complete and record:
```sh
INSTALLATION="$(pwd -P)/deploy/local/thothii-installation.yaml"
test -f "$INSTALLATION"
bash scripts/verify-standalone-install.sh "$INSTALLATION"
```
1. usable LLMs, the modelCatalog, and linked API keys; then run tht pi test and tht doctor;
2. database configuration, connection test, schema synchronization, and description generation;
3. human consolidation of generated descriptions;
4. Qdrant semantic entries through workspace preprocess run;
5. naming-based FK suggestions as a complement to schema FKs, human review, and loading approved
relationships into Qdrant;
6. recurring tht doctor --json and tht workspace test --json checks;
7. one real question completed successfully without connection or model errors.
The verifier is read-only: it runs `tht doctor --json` and `tht status` without restarting the
stack, regenerating configuration, or printing secret contents.
Configuration is complete only when all applicable activities are done, decisions are recorded, and
the two terminal tests are green. The core is usable only after the real question, not merely
because the frontend answers /health.
### Gate A — platform smoke test on all three computers
## Gate A and Gate B
Record the following for each machine:
### Gate A — platform
```sh
~~~
uname -a
docker version --format '{{.Server.Version}} {{.Server.Arch}}'
tht version
bash scripts/check-standalone-prerequisites.sh
bash scripts/verify-standalone-install.sh "$INSTALLATION"
```
~~~
The gate passes when the clone is intact, Docker and Compose are reachable, `tht doctor` is OK, the
stack is running, and the frontend responds at the default local URL `http://127.0.0.1:8080`.
Doctor also checks workspace and Pi: record their failures separately rather than labeling every
failure as a platform problem. Check HTTP readiness with:
### Gate B — usability
```sh
curl --fail --silent --show-error http://127.0.0.1:8080/health
```
### Gate B — functional verification
Run this on at least one machine with available endpoints and credentials:
First follow [Workspace operations](../operations/workspaces.md) to import/prepare the workspace
and configure the Database and local binding. The source clone does not transfer catalog data,
secrets or sessions from the Mac. Connect any VPN required by DWH/model endpoints and verify
that their names are reachable from containers too.
1. open `http://127.0.0.1:8080`;
2. sign in with the configured local account;
3. verify that the configured workspace is readable;
4. start a real question and complete the review gates through final SQL;
5. stop and restart the installation, then run `verify-standalone-install.sh` again.
A Gate B failure involving DWH, the LLM provider, workspace Git, or credentials does not by itself
prove a Docker portability problem: record the failed endpoint or component separately.
## Daily lifecycle
Use the explicit descriptor when more than one installation may be discoverable:
```sh
INSTALLATION="$PWD/deploy/local/thothii-installation.yaml"
tht --installation "$INSTALLATION" status
tht --installation "$INSTALLATION" start
tht --installation "$INSTALLATION" start --build
tht --installation "$INSTALLATION" logs
~~~
tht --installation "$INSTALLATION" doctor --json
tht --installation "$INSTALLATION" stop
```
tht --installation "$INSTALLATION" workspace test --json
curl --fail --silent --show-error http://127.0.0.1:8080/health
~~~
Use `start --build` after source changes or to rebuild images from the current checkout. `stop`
preserves volumes, sessions, the catalog, Pi state, Qdrant data, and the embedding model. Do not
use `docker compose down --volumes` during a normal test: it is destructive and removes local data.
For upgrades requiring migrations, follow the release runbook before starting the new application.
Then run a real question and stop/restart with tht stop and tht start. Do not use docker compose
down --volumes: it deletes the Catalog, sessions, Qdrant data and the embedding model.
## Quick diagnosis
| Symptom | Check |
| Symptom | Action |
| --- | --- |
| `Docker Engine is not reachable` | start Docker Desktop or the Docker service and rerun `docker info` |
| Windows sees Docker but Bash fails | run the guide inside Ubuntu WSL2 and enable that distribution in Docker Desktop |
| `tht: command not found` | open a new shell and check `command -v tht`; rerun the bootstrap if needed |
| line-ending or executable-script errors | use a clone in the WSL2/Linux filesystem and rerun `bash scripts/...` |
| unsupported architecture | check `docker version --format '{{.Server.Arch}}'`; the test requires `amd64` or `arm64` |
| missing descriptor or env file | use `deploy/local/...` generated by `tht setup`, not an arbitrary copied file |
| healthy stack but workflow failure | check external URLs, the credential bundle, workspace Git, and authentication separately |
| data appears missing | check that `down --volumes` was not used; `stop` does not remove volumes |
## Acceptance checklist
- [ ] The clone comes from the expected Gitea repository and the revision is recorded.
- [ ] Docker Desktop/Engine and Compose v2 are available.
- [ ] The runtime reports an allowed architecture.
- [ ] `tht` was built from the repository and responds to `tht version`.
- [ ] Setup uses `profile: local`, `shell.mode: full`, and `shell.defaultLocale: en`.
- [ ] The descriptor, `operator.env`, authentication, and secrets exist only under `deploy/local/`.
- [ ] No secret appears in Git, URLs, public YAML, or recorded commands.
- [ ] Gate A passes on Apple Silicon macOS, x64 Windows WSL2, and x64 Linux.
- [ ] Gate B runs on at least one machine with DWH and LLM available.
- [ ] Stop/start and final verification complete without deleting volumes.
## Out of scope for this release
The following remain future work:
- publishing pre-built images on Docker Hub;
- reducing prompts through a dedicated non-interactive configuration;
- creating DMG, MSI/EXE, AppImage, or other native installers;
- providing an offline runtime or bundling a local DWH/LLM into the application.
| Docker Engine is not reachable | start Docker Desktop or systemctl and repeat docker info |
| Omarchy cannot find docker | install Docker/Compose, enable the service and open a new session |
| Windows sees Docker but Bash fails | use Ubuntu WSL2 and enable its Docker Desktop integration |
| workspace pull fails | check URL, branch, key/credential file and known_hosts from the container |
| workspace test reports a missing binding | configure database, token/password and CA in Database Management |
| Pi is not ready | fill pi-auth.json or the key declared by modelCatalog, then run tht pi test |
## Related documents
- [Install and first start](first-start.md)
- [Shell and localization](shell-and-language.md)
- [Workspace operations](../operations/workspaces.md)
- `deploy/secrets/README.md` (runtime secrets)
- [Database Management](../operations/database-management.md)
- [Model configuration](../general/pi-configuration.md)
- deploy/secrets/README.md
Publishing images on Docker Hub and native DMG/MSI/AppImage installers remain later work: this
procedure starts from the Gitea clone and does not require pre-published Docker Hub images.