ThothII/docs/install/local-workspace-registry.md

# Local workspace-registry installation (Mac and PC)

Complete the [local PC/Mac/Linux installation](local.md) first. This guide continues with the
Git-backed workspace source of truth, installation-local connector bindings, and diagnostics. Use
the [Pi management manual](pi-management.md) for provider configuration and image recovery.

This guide runs a single-user ThothII registry on Docker Desktop (macOS or Windows) or a local
Linux Docker Engine. It is intentionally loopback-only. Git is shared; the checkout, DWH
bindings, credentials, and session data are local, while internal Qdrant/Ollama ship in the
Compose stack. Never put credentials in workspace YAML, Git, browser drafts, diagnostics, or
`.env.example`.

## Architecture ownership contract

| Component | Ownership | Operator contract |
| --- | --- | --- |
| DWH | External | Installation-local endpoint/binding; never bundled into the Compose semantic stack. |
| LLM | External | Installation-local endpoint/policy choice outside the internal semantic services. |
| Qdrant | Internal | Mandatory private Compose semantic service; persistent `qdrant-data` volume. |
| Ollama embedding | Internal | Mandatory private Compose semantic service for `qwen3-embedding:0.6b`. |

## Prerequisites

- macOS: Docker Desktop, Git, and sufficient volume disk space. Git Credential Manager is useful
  for HTTPS sign-in.
- Windows: Docker Desktop with WSL2, Git for Windows, and the clone enabled in Docker file sharing.
  Use absolute paths/WSL paths; PowerShell uses `;` rather than `:` in `COMPOSE_FILE`.
- Linux PC: Docker Engine, Compose plugin, Git, and a user permitted to run Docker.
- Outbound access to the Git remote. A local installation needs no inbound firewall rule.

Keep the operator `.env` and `installation-secrets/` outside the workspace-registry Git checkout.
On macOS/Linux use mode `0600` for individual secret files. On Windows apply an ACL that grants
read access only to the Docker Desktop user. Do not use an empty file to silently bypass a selected
authentication method.

## Git remote: SSH and HTTPS

Create one private repository such as `thoth-workspaces.git`. It contains canonical workspace
definitions, curated Evidence content, and generated artifacts:

```text
registry.git/
├── workspaces/
│   └── <workspace-id>.yaml
├── workspace-content/
│   └── <workspace-id>/evidence/...
└── workspace-docs/
    └── <workspace-id>/{contract.env.example,README.md}
```

For SSH, use a scoped deploy key, a verified `known_hosts` file, and strict host-key checking. For
HTTPS, use Git Credential Manager or a secret-manager-created credentials file. Mount a private
HTTPS CA as its own file. Do not disable host or certificate verification. The base Compose file
does not mount a Git credential: add exactly one optional `deploy/compose.git-ssh.yaml` or
`deploy/compose.git-https.yaml` override, so unused credential paths are never bind-mounted.

```dotenv
THT_WORKSPACE_GIT_REMOTE=ssh://git@git.example.invalid/platform/thoth-workspaces.git
THT_WORKSPACE_GIT_BRANCH=main
THT_WORKSPACE_INSTALLATION_ID=local-laptop
THT_SOURCE_ROOT=/absolute/path/to/ThothII
PI_AUTH_FILE=/absolute/path/installation-secrets/pi-auth.json
THT_SECRETS_FILE=/absolute/path/installation-secrets/thothii.secrets
THT_WORKSPACE_GIT_SSH_KEY_FILE=/absolute/path/installation-secrets/git-ssh-key
THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=/absolute/path/installation-secrets/git-known-hosts
THT_WORKSPACE_GIT_CA_FILE=/absolute/path/installation-secrets/git-ca.pem
```

For HTTPS set `THT_WORKSPACE_GIT_CREDENTIALS_FILE` instead of the SSH key/known-hosts pair. Remote
and branch are non-secret; every `*_FILE` is a local path whose content never enters Git or logs.

## Curator flow for shared-registry Evidence

Follow this order; the [canonical Evidence contract](../contracts/workspace-evidence-v3.md) defines
the source shapes and safety boundary.

1. Clone the one shared registry, or update the review clone with `git pull --ff-only`.
2. Add source bytes below `workspace-content/<id>/evidence`, then commit and push.
3. Validate and publish the descriptor against that base commit.
4. Inspect `workspace-docs/<id>/contract.env.example` and `workspace-docs/<id>/README.md`.
5. Provision only the selected Evidence `*_FILE` files outside Git and strictly below a root in `THT_WORKSPACE_SECRET_ROOTS`; add matching host-only `*_SOURCE` paths for the generated connector override.
6. Render or acquire the runtime config, then run `tht config check -c <path>`.
7. Stop: P2/P6 later performs preprocessing and materialization.

For example, a signed-HTTP workspace and a different static-S3 workspace can use these host-only
connector sources; the values are paths, not file contents:

```dotenv
THT_WS_SIGNED_HTTP_EVIDENCE_SIGNED_URLS_SOURCE=/absolute/path/installation-secrets/signed-http-evidence-urls.json
THT_WS_STATIC_S3_EVIDENCE_ACCESS_KEY_SOURCE=/absolute/path/installation-secrets/static-s3-evidence-access-key
THT_WS_STATIC_S3_EVIDENCE_SECRET_KEY_SOURCE=/absolute/path/installation-secrets/static-s3-evidence-secret-key
THT_WS_STATIC_S3_EVIDENCE_SESSION_TOKEN_SOURCE=/absolute/path/installation-secrets/static-s3-evidence-session-token
```

The descriptor and declared filesystem root are validated at the same registry commit. The
browser shows a read-only Evidence summary, while exports omit Evidence bytes.

## Shared Git values, local bindings, and secret files

| Location | Contains | Never contains |
| --- | --- | --- |
| Git workspace repository | schema v3 YAML, generated binding names, LLM policy, and semantic-index identity | installation hostnames, keys, passwords, certificates, SSH keys |
| local `.env` | remote, branch, installation ID, `THT_WORKSPACE_BINDINGS_ENV_FILE`, and secret source paths | secret contents or `THT_WS_*` values |
| workspace bindings env file | only `THT_WS_*` transport, endpoint, user, and `/run/secrets/...` path bindings | secret contents or unrelated application settings |
| local secret directory | Git credentials/key, known hosts, CA, connector secret files | a copied registry checkout |
| Docker volumes | registry checkout/snapshots/state/locks and local data | host-only secret source files |

The persistent volume is `/data/workspace-registry`:

```text
repo/       # persistent Git checkout
snapshots/  # immutable validated revisions used by sessions
state/      # active revision and registry state
locks/      # short-lived publish locks
```

Installation variables are deterministic: `north-star-research` becomes `NORTH_STAR_RESEARCH`, and every name
is `THT_WS_<NAMESPACE>_<ROLE>_<SUFFIX>`. Copy
[the bindings env example](examples/workspace-bindings.env.example) to an untracked operator file
and set its absolute path as `THT_WORKSPACE_BINDINGS_ENV_FILE`. It is loaded only into `core`.
Credentials and certificates use `*_FILE` path variables that must point inside `/run/secrets`.

## Direct PostgreSQL, REST, and SSH tunnel bindings

Set only fields for the selected transport in the dedicated bindings env file. Canonical YAML keeps
database/schema shared in Git. Every `*_FILE=/run/secrets/<target>` binding needs one matching
host-only `*_SOURCE` path in operator `.env`. Generate the untracked connector override from those
two files during bootstrap; do not copy or maintain a workspace-specific Compose override.

```dotenv
# Direct PostgreSQL
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct
THT_WS_NORTH_STAR_RESEARCH_DWH_HOST=dwh.example.invalid
THT_WS_NORTH_STAR_RESEARCH_DWH_PORT=5432
THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
```

```dotenv
# REST; an API-key file is needed only for a declared bearer/x-api-key diagnostic.
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=rest_api
THT_WS_NORTH_STAR_RESEARCH_DWH_BASE_URL=https://dwh.example.invalid
THT_WS_NORTH_STAR_RESEARCH_DWH_API_KEY_FILE=/run/secrets/north-star-research-dwh-api-key
```

```dotenv
# SSH tunnel diagnostic only; runtime sessions are fail-closed in this release.
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=ssh_tunnel
THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_HOST=bastion.example.invalid
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PORT=22
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_USER=thoth_tunnel
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PRIVATE_KEY_FILE=/run/secrets/north-star-research-dwh-tunnel-key
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_KNOWN_HOSTS_FILE=/run/secrets/north-star-research-dwh-known-hosts
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_HOST=dwh.internal.example
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_PORT=5432
```

REST diagnostics reject a private per-request CA rather than weakening TLS; use runtime-trusted
HTTPS or verified direct/SSH native TLS. See the
[diagnostic protocol](../workspace-diagnostic-protocol.md).

An SSH connector can prove installation reachability, host-key verification, authentication, and
target identity, but it intentionally returns `workspace_not_activatable`; select direct or REST
before creating sessions. Git pull/push over SSH remains fully supported and is independent.

## Bootstrap, first pull, and diagnostics

Use the repository's canonical `compose.yaml` plus `deploy/compose.local.yaml`; they always start
`frontend`, `core`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`. This profile
is CPU-first. Add `THOTH_ENABLE_EMBEDDING_GPU=1` only on a Linux host that intentionally exposes a
supported GPU device to Docker. Qdrant is a derived but persistent index, while Ollama keeps a
local model cache for `qwen3-embedding:0.6b` (`1024` dimensions, cosine distance). Do not copy or
maintain a standalone application Compose file. Copy [the bindings env example](examples/workspace-bindings.env.example) into an
untracked operator directory and create a protected operator env file from
`deploy/env/local.env.example`. It must contain absolute `PI_AUTH_FILE`,
`THT_SECRETS_FILE`, `THT_WORKSPACE_BINDINGS_ENV_FILE`, and connector `*_SOURCE` paths.
The Pi auth JSON, runtime secret bundle, and each connector credential remain separate protected
host files and are mounted read-only; their contents never enter the operator env or rendered
Compose.

Select exactly one repository Git transport override, `deploy/compose.git-ssh.yaml` or
`deploy/compose.git-https.yaml`. A Compose env file is not a shell environment, so export only the
non-secret paths required by the maintenance commands. Generate the connector override and render
through the preflight wrapper, which rejects unsafe paths and combined SSH+HTTPS selection.
Record the selected Git and generated connector overrides in the operator
[`thothii-installation.yaml` example](examples/thothii-installation.local.yaml), using absolute
paths, so `thothctl` remains the ordinary lifecycle interface.

```sh
export THT_SOURCE_ROOT=/absolute/path/to/ThothII
export THT_OPERATOR_ENV=/absolute/path/to/operator/local.env
export THT_WORKSPACE_BINDINGS_ENV_FILE=/absolute/path/to/operator/workspace-bindings.env
export THT_CONNECTOR_OVERRIDE=/absolute/path/to/operator/connector-secrets.local.yaml
"$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" --bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" --operator-env "$THT_OPERATOR_ENV" --output "$THT_CONNECTOR_OVERRIDE"
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \
  -f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.local.yaml" \
  -f "$THT_SOURCE_ROOT/deploy/compose.git-ssh.yaml" -f "$THT_CONNECTOR_OVERRIDE" config --quiet
```

```sh
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \
  -f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.local.yaml" \
  -f "$THT_SOURCE_ROOT/deploy/compose.git-ssh.yaml" -f "$THT_CONNECTOR_OVERRIDE" up --build -d
curl --fail --silent http://127.0.0.1:8787/health
curl --fail --silent http://127.0.0.1:8787/workspace-registry/status
curl --fail --silent http://127.0.0.1:8787/workspaces
```

The first status request clones, validates all descriptors, and atomically activates a snapshot.
Use `POST /workspace-registry/pull` to fetch later revisions. Run workspace diagnostics only after
required DWH bindings are mounted. Schema-v3 diagnostics probe the internal Qdrant/Ollama
services through backend config; ordinary diagnostics are read-only.

<!-- workspace-descriptor-contract:start -->
Schema v3 is the only accepted workspace descriptor. Schema v1 and v2 workspace descriptors are
rejected before activation. Candidate snapshot validation makes bootstrap activation or a pull fail
atomically and leaves the prior active snapshot unchanged. There is no in-product migrator or
automatic conversion. The repository must already contain reviewed v3 descriptors. One workspace
owns one Qdrant collection; schema, Evidence, and Memory records share that collection and remain
isolated by payload `kind`.
<!-- workspace-descriptor-contract:end -->

## Semantic index ownership contract

| Scope | Ownership rule | Isolation rule |
| --- | --- | --- |
| Workspace semantic index | Each workspace reserves a single Qdrant collection. | Schema, Evidence, and Memory stay in that one collection and remain isolated by payload `kind`. |

If source material needs conversion, perform it outside ThothII in a separate reviewed process.
Commit only the resulting reviewed v3 descriptors. That external process must not import `${ENV}`
values, secret values, certificates, keys, or secret files into the repository.

## Publish, update, backup, outage recovery, and rollback

Drafts live only in browser storage. Review the canonical field diff, validate/test locally, then
publish. If conflicted, pull first and create a new reviewed field-level draft; never hand-edit the
running `repo/` volume. Before upgrading, record registry status, stop Compose, and take a
timestamped ownership-preserving backup of both registry and local data volumes while excluding
`installation-secrets/`. Render Compose, rebuild, start, and check status before resuming work.

After a valid bootstrap, remote outage retains the last valid snapshot and reports `degraded: true`.
Pinned sessions continue. Repair network/authentication, pull, and confirm non-degraded status. To
undo a bad remote change, create a reviewed Git revert/release branch, advance the remote through
normal policy, pull, and confirm its new snapshot. Do not delete `snapshots/` as rollback.

## Troubleshooting

| Stable code | Meaning and safe action |
| --- | --- |
| `workspace_invalid` | Invalid descriptor/path/snapshot; restore a reviewed canonical revision. |
| `binding_missing` | A selected value or readable `*_FILE` is absent; fix the local binding/mount. |
| `workspace_not_activatable` | Diagnostics cannot activate the workspace; correct the selected transport. |
| `workspace_stale` | Checkout changed or is busy; stop competing pull/publish work. |
| `workspace_conflict` | Draft base differs from Git; pull, resolve the diff, validate, republish. |
| `git_unavailable` | Remote, path, network, or lock unavailable; preserve the degraded valid snapshot. |
| `git_auth_failed` | Mounted SSH/HTTPS material rejected/unreadable; rotate or fix permissions without logging it. |
| `git_non_fast_forward` | Checkout diverged; reconcile through the registry workflow. |
| `git_push_rejected` | Remote policy rejected the change; review branch protection/hooks. |
| `connector_unavailable` | DNS/TLS/auth/resource identity diagnostic failed; inspect local bindings and egress. |
| `semantic_index_incompatible` | Collection/model/dimensions/distance differs from Git; perform an explicit index migration. |

On macOS, restart Docker Desktop if a named volume disappears. On Windows, check WSL2 and Docker
file sharing. A failed first bootstrap has no snapshot fallback: repair remote trust and retry;
never create an unreviewed local registry repository.