ThothII/docs/install/local-workspace-registry.md

# Local workspace-registry installation (Mac and PC)

Complete the [local PC/Mac/Linux installation](local.md) first. This guide continues with the
Git-backed workspace source of truth, installation-local connector bindings, and diagnostics. Use
the [Pi management manual](pi-management.md) for provider configuration and image recovery.

This guide runs a single-user ThothII registry on Docker Desktop (macOS or Windows) or a local
Linux Docker Engine. It is intentionally loopback-only. Git is shared; the checkout, DWH
bindings, credentials, and session data are local, while internal Qdrant/Ollama ship in the
Compose stack. Never put credentials in workspace YAML, Git, browser drafts, diagnostics, or
`.env.example`.

## Architecture ownership contract

| Component | Ownership | Operator contract |
| --- | --- | --- |
| DWH | External | Installation-local endpoint/binding; never bundled into the Compose semantic stack. |
| LLM | External | Installation-local endpoint/policy choice outside the internal semantic services. |
| Qdrant | Internal | Mandatory private Compose semantic service; persistent `qdrant-data` volume. |
| Ollama embedding | Internal | Mandatory private Compose semantic service for `qwen3-embedding:0.6b`. |


## Host preprocessing (P2)

The installed native `thothctl` is the only host entrypoint for workspace preprocessing
(introspection+LSH, FK review, schema indexing, HTTP Evidence). Use
`thothctl --installation <thothii-installation.yaml> workspace <command> --workspace <id> [--json]`
per `docs/contracts/workspace-preprocessing-cli.md` and the P2 walkthrough in
`docs/testing/p2-p6-manual-verification.md`. Preprocessing runs through the profile-gated
`workspace-maintenance` Compose service; it never starts a backend/Pi/frontend listener and never
attaches Git credentials.


## Effective configuration and `.tht-dwh` (P3)

Prepared DWH generations are reusable and safe: `thothctl` and the application derive the same
canonical effective configuration and logical identity, so prepared work is reused when nothing
relevant changed and refused when the database/endpoint/identity changed. See
`docs/contracts/tht-dwh.md` for generations, `OWNER.json`, `ACTIVE`, fingerprints, migration and
recovery. A content-only or Evidence-only change never forces a full re-introspection.

## Prerequisites

- macOS: Docker Desktop, Git, and sufficient volume disk space. Git Credential Manager is useful
  for HTTPS sign-in.
- Windows: Docker Desktop with WSL2, Git for Windows, and the clone enabled in Docker file sharing.
  Use absolute paths/WSL paths; PowerShell uses `;` rather than `:` in `COMPOSE_FILE`.
- Linux PC: Docker Engine, Compose plugin, Git, and a user permitted to run Docker.
- Outbound access to the Git remote. A local installation needs no inbound firewall rule.

Keep the operator `.env` and `installation-secrets/` outside the workspace-registry Git checkout.
On macOS/Linux use mode `0600` for individual secret files. On Windows apply an ACL that grants
read access only to the Docker Desktop user. Do not use an empty file to silently bypass a selected
authentication method.

## Git remote: SSH and HTTPS

Create one private repository such as `thoth-workspaces.git`. It contains the curator-owned root
catalog, one directory per workspace, optional embedded Evidence trees, and generated public docs:

```text
registry.git/
├── thoth-workspaces.yaml
├── <workspace-id>/
│   ├── workspace.yaml
│   └── evidence/...
└── workspace-docs/
    └── <workspace-id>/{contract.env.example,README.md}
```

`thoth-workspaces.yaml` uses the `schema_version` value `1` and the ordered `workspaces` list of
`{id, name, description?}` entries. It is authoritative for workspace ID, name, description, and
display order; `<workspace-id>/workspace.yaml` must match that metadata exactly, while
`workspace-docs` remains the reserved top-level generated-docs directory.

For SSH, use a scoped deploy key, a verified `known_hosts` file, and strict host-key checking. For
HTTPS, use Git Credential Manager or a secret-manager-created credentials file. Mount a private
HTTPS CA as its own file. Do not disable host or certificate verification. The base Compose file
does not mount a Git credential: add exactly one optional `deploy/compose.git-ssh.yaml` or
`deploy/compose.git-https.yaml` override, so unused credential paths are never bind-mounted.

```dotenv
THT_WORKSPACE_GIT_REMOTE=ssh://git@git.example.invalid/platform/thoth-workspaces.git
THT_WORKSPACE_GIT_BRANCH=main
THT_WORKSPACE_INSTALLATION_ID=local-laptop
THT_SOURCE_ROOT=/absolute/path/to/ThothII
PI_AUTH_FILE=/absolute/path/installation-secrets/pi-auth.json
THT_SECRETS_FILE=/absolute/path/installation-secrets/thothii.secrets
THT_WORKSPACE_GIT_SSH_KEY_FILE=/absolute/path/installation-secrets/git-ssh-key
THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=/absolute/path/installation-secrets/git-known-hosts
THT_WORKSPACE_GIT_CA_FILE=/absolute/path/installation-secrets/git-ca.pem
```

For HTTPS set `THT_WORKSPACE_GIT_CREDENTIALS_FILE` instead of the SSH key/known-hosts pair. Remote
and branch are non-secret; every `*_FILE` is a local path whose content never enters Git or logs.

## Curator flow for shared-registry Evidence

Follow this order; the [canonical Evidence contract](../contracts/workspace-evidence-v3.md) defines
the source shapes and safety boundary.

1. Clone the one shared registry, or update the review clone with `git pull --ff-only`.
2. Keep `thoth-workspaces.yaml` curator-owned. It uses the `schema_version` value `1` and the ordered
   `workspaces` list of `{id, name, description?}` entries; it is authoritative for workspace ID,
   name, description, and display order.
3. For an existing workspace, edit `<id>/workspace.yaml` and any embedded `<id>/evidence/**`, then
   commit and push.
4. For a new workspace, add the catalog slot first. If you want ThothII to bootstrap the
   descriptor, leave `<id>/workspace.yaml` absent, commit and push, then pull that commit into the
   installation; the slot appears as `configuration_required`.
5. The API may create `<id>/workspace.yaml` only when the catalog slot already exists and no Git
   object exists at that path in the exact pulled base commit.
6. After bootstrap, existing descriptors change only through curator Git commit/push and
   installation pull. The API never writes `thoth-workspaces.yaml` or `<id>/evidence/**`.
7. Inspect `workspace-docs/<id>/contract.env.example` and `workspace-docs/<id>/README.md`.
8. Provision only the selected Evidence `*_FILE` files outside Git and strictly below a root in
   `THT_WORKSPACE_SECRET_ROOTS`; add matching host-only `*_SOURCE` paths for the generated
   connector override.
9. Render or acquire the runtime config, then run `tht config check -c <path>`.
10. Stop: P2/P6 later performs preprocessing and materialization.

For example, a signed-HTTP workspace and a different static-S3 workspace can use these host-only
connector sources; the values are paths, not file contents:

```dotenv
THT_WS_SIGNED_HTTP_EVIDENCE_SIGNED_URLS_SOURCE=/absolute/path/installation-secrets/signed-http-evidence-urls.json
THT_WS_STATIC_S3_EVIDENCE_ACCESS_KEY_SOURCE=/absolute/path/installation-secrets/static-s3-evidence-access-key
THT_WS_STATIC_S3_EVIDENCE_SECRET_KEY_SOURCE=/absolute/path/installation-secrets/static-s3-evidence-secret-key
THT_WS_STATIC_S3_EVIDENCE_SESSION_TOKEN_SOURCE=/absolute/path/installation-secrets/static-s3-evidence-session-token
```

The descriptor and declared filesystem root are validated at the same registry commit. The
browser shows a read-only Evidence summary, while exports omit Evidence bytes.

## Shared Git values, local bindings, and secret files

| Location | Contains | Never contains |
| --- | --- | --- |
| Git workspace repository | schema v3 YAML, generated binding names, LLM policy, and semantic-index identity | installation hostnames, keys, passwords, certificates, SSH keys |
| local `.env` | remote, branch, installation ID, `THT_WORKSPACE_BINDINGS_ENV_FILE`, and secret source paths | secret contents or `THT_WS_*` values |
| workspace bindings env file | only `THT_WS_*` transport, endpoint, user, and `/run/secrets/...` path bindings | secret contents or unrelated application settings |
| local secret directory | Git credentials/key, known hosts, CA, connector secret files | a copied registry checkout |
| Docker volumes | registry checkout/snapshots/state/locks and local data | host-only secret source files |

The persistent volume is `/data/workspace-registry`:

```text
repo/       # persistent Git checkout
snapshots/  # immutable validated revisions used by sessions
state/      # active revision and registry state
locks/      # short-lived registry synchronization locks
```

Installation variables are deterministic: `north-star-research` becomes `NORTH_STAR_RESEARCH`, and every name
is `THT_WS_<NAMESPACE>_<ROLE>_<SUFFIX>`. Copy
[the bindings env example](examples/workspace-bindings.env.example) to an untracked operator file
and set its absolute path as `THT_WORKSPACE_BINDINGS_ENV_FILE`. It is loaded only into `core`.
Credentials and certificates use `*_FILE` path variables that must point inside `/run/secrets`.

## Direct PostgreSQL, REST, and SSH tunnel bindings

Set only fields for the selected transport in the dedicated bindings env file. Canonical YAML keeps
database/schema shared in Git. Every `*_FILE=/run/secrets/<target>` binding needs one matching
host-only `*_SOURCE` path in operator `.env`. Generate the untracked connector override from those
two files during bootstrap; do not copy or maintain a workspace-specific Compose override.

```dotenv
# Direct PostgreSQL
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct
THT_WS_NORTH_STAR_RESEARCH_DWH_HOST=dwh.example.invalid
THT_WS_NORTH_STAR_RESEARCH_DWH_PORT=5432
THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
```

```dotenv
# REST; an API-key file is needed only for a declared bearer/x-api-key diagnostic.
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=rest_api
THT_WS_NORTH_STAR_RESEARCH_DWH_BASE_URL=https://dwh.example.invalid
THT_WS_NORTH_STAR_RESEARCH_DWH_API_KEY_FILE=/run/secrets/north-star-research-dwh-api-key
```

```dotenv
# SSH tunnel diagnostic only; runtime sessions are fail-closed in this release.
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=ssh_tunnel
THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_HOST=bastion.example.invalid
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PORT=22
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_USER=thoth_tunnel
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PRIVATE_KEY_FILE=/run/secrets/north-star-research-dwh-tunnel-key
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_KNOWN_HOSTS_FILE=/run/secrets/north-star-research-dwh-known-hosts
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_HOST=dwh.internal.example
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_PORT=5432
```

REST diagnostics reject a private per-request CA rather than weakening TLS; use runtime-trusted
HTTPS or verified direct/SSH native TLS. See the
[diagnostic protocol](../workspace-diagnostic-protocol.md).

An SSH connector can prove installation reachability, host-key verification, authentication, and
target identity, but it intentionally returns `workspace_not_activatable`; select direct or REST
before creating sessions. Git pull/push over SSH remains fully supported and is independent.

## Bootstrap, first pull, and diagnostics

Use the repository's canonical `compose.yaml` plus `deploy/compose.local.yaml`; they always start
`frontend`, `core`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`. This profile
is CPU-first. Add `THOTH_ENABLE_EMBEDDING_GPU=1` only on a Linux host that intentionally exposes a
supported GPU device to Docker. Qdrant is a derived but persistent index, while Ollama keeps a
local model cache for `qwen3-embedding:0.6b` (`1024` dimensions, cosine distance). Do not copy or
maintain a standalone application Compose file. Copy [the bindings env example](examples/workspace-bindings.env.example) into an
untracked operator directory and create a protected operator env file from
`deploy/env/local.env.example`. It must contain absolute `PI_AUTH_FILE`,
`THT_SECRETS_FILE`, `THT_WORKSPACE_BINDINGS_ENV_FILE`, and connector `*_SOURCE` paths.
The Pi auth JSON, runtime secret bundle, and each connector credential remain separate protected
host files and are mounted read-only; their contents never enter the operator env or rendered
Compose.

Select exactly one repository Git transport override, `deploy/compose.git-ssh.yaml` or
`deploy/compose.git-https.yaml`. A Compose env file is not a shell environment, so export only the
non-secret paths required by the maintenance commands. Generate the connector override and render
through the preflight wrapper, which rejects unsafe paths and combined SSH+HTTPS selection.
Record the selected Git and generated connector overrides in the operator
[`thothii-installation.yaml` example](examples/thothii-installation.local.yaml), using absolute
paths, so `thothctl` remains the ordinary lifecycle interface.

```sh
export THT_SOURCE_ROOT=/absolute/path/to/ThothII
export THT_OPERATOR_ENV=/absolute/path/to/operator/local.env
export THT_WORKSPACE_BINDINGS_ENV_FILE=/absolute/path/to/operator/workspace-bindings.env
export THT_CONNECTOR_OVERRIDE=/absolute/path/to/operator/connector-secrets.local.yaml
"$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" --bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" --operator-env "$THT_OPERATOR_ENV" --output "$THT_CONNECTOR_OVERRIDE"
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \
  -f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.local.yaml" \
  -f "$THT_SOURCE_ROOT/deploy/compose.git-ssh.yaml" -f "$THT_CONNECTOR_OVERRIDE" config --quiet
```

```sh
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \
  -f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.local.yaml" \
  -f "$THT_SOURCE_ROOT/deploy/compose.git-ssh.yaml" -f "$THT_CONNECTOR_OVERRIDE" up --build -d
curl --fail --silent http://127.0.0.1:8787/health
curl --fail --silent http://127.0.0.1:8787/workspace-registry/status
curl --fail --silent http://127.0.0.1:8787/workspaces
```

The first status request clones the registry, validates `thoth-workspaces.yaml`, every
catalog-listed descriptor, and any declared `<id>/evidence` tree, then atomically activates a
snapshot. A catalog-only slot with no descriptor reports `configuration_required` and is not
activatable. Use `POST /workspace-registry/pull` to fetch later curator revisions. Existing
curated descriptors stay read-only in the UI; only a missing descriptor may use the one-time
bootstrap create flow. Run workspace diagnostics only after required DWH bindings are mounted.
Schema-v3 diagnostics probe the internal Qdrant/Ollama services through backend config; ordinary
diagnostics are read-only.

<!-- workspace-descriptor-contract:start -->
Schema v3 is the only accepted workspace descriptor. Schema v1 and v2 workspace descriptors are
rejected before activation. Candidate snapshot validation makes bootstrap activation or a pull fail
atomically and leaves the prior active snapshot unchanged. There is no in-product migrator or
automatic conversion. The repository must already contain reviewed v3 descriptors. One workspace
owns one Qdrant collection; schema, Evidence, and Memory records share that collection and remain
isolated by payload `kind`.
<!-- workspace-descriptor-contract:end -->

## Semantic index ownership contract

| Scope | Ownership rule | Isolation rule |
| --- | --- | --- |
| Workspace semantic index | Each workspace reserves a single Qdrant collection. | Schema, Evidence, and Memory stay in that one collection and remain isolated by payload `kind`. |

If source material needs conversion, perform it outside ThothII in a separate reviewed process.
Commit only the resulting reviewed v3 descriptors. That external process must not import `${ENV}`
values, secret values, certificates, keys, or secret files into the repository.

## Publish, update, backup, outage recovery, and rollback

Existing curated workspaces are read-only in the browser. Use the Workspace
Management page to pull, inspect status, validate a workspace, test it on this installation, and
optionally create one bootstrap descriptor for a pulled `configuration_required` slot. After that
first descriptor exists, change it only through curator Git commit/push and installation pull;
never hand-edit the running `repo/` volume. Before upgrading, record registry status, stop Compose,
and take a timestamped ownership-preserving backup of both registry and local data volumes while
excluding `installation-secrets/`. Render Compose, rebuild, start, and check status before
resuming work. If you are upgrading an older P1 registry, apply the reviewed migration in
[`docs/migrations/p1-to-p1-1-registry-layout.md`](../migrations/p1-to-p1-1-registry-layout.md)
and upgrade ThothII only after that commit is pushed.

After a valid bootstrap, remote outage retains the last valid snapshot and reports `degraded: true`.
Pinned sessions continue. Repair network/authentication, pull, and confirm non-degraded status. To
undo a bad remote change, create a reviewed Git revert/release branch, advance the remote through
normal policy, pull, and confirm its new snapshot. Do not delete `snapshots/` as rollback.

## Troubleshooting

| Stable code | Meaning and safe action |
| --- | --- |
| `workspace_invalid` | Invalid descriptor/path/snapshot; restore a reviewed canonical revision. |
| `binding_missing` | A selected value or readable `*_FILE` is absent; fix the local binding/mount. |
| `workspace_not_activatable` | Diagnostics cannot activate the workspace; correct the selected transport. |
| `workspace_stale` | Checkout changed or is busy; stop competing pull or sync work. |
| `workspace_conflict` | Draft base differs from Git; pull, resolve the diff, validate, and retry after the curator pull/bootstrap flow. |
| `git_unavailable` | Remote, path, network, or lock unavailable; preserve the degraded valid snapshot. |
| `git_auth_failed` | Mounted SSH/HTTPS material rejected/unreadable; rotate or fix permissions without logging it. |
| `git_non_fast_forward` | Checkout diverged; reconcile through the registry workflow. |
| `git_push_rejected` | Remote policy rejected the change; review branch protection/hooks. |
| `connector_unavailable` | DNS/TLS/auth/resource identity diagnostic failed; inspect local bindings and egress. |
| `semantic_index_incompatible` | Collection/model/dimensions/distance differs from Git; perform an explicit index migration. |

On macOS, restart Docker Desktop if a named volume disappears. On Windows, check WSL2 and Docker
file sharing. A failed first bootstrap has no snapshot fallback: repair remote trust and retry;
never create an unreviewed local registry repository.