diff --git a/README.md b/README.md index 0fe1840f..da6955fc 100644 --- a/README.md +++ b/README.md @@ -57,18 +57,37 @@ diagnostics are exposed by `tht doctor` and do not prevent the UI from starting. Workspace descriptors are shared through a validated Git repository while endpoint bindings and secret files remain installation-local. Use the [local Mac/PC installation manual](docs/install/local-workspace-registry.md) -for Docker Desktop or a local engine, and the [server installation manual](docs/install/server-workspace-registry.md) -for the Gitea, reverse-proxy, backup, upgrade, and recovery workflow. The isolated deployment -exercise is `./scripts/workspace-registry-smoke.sh`; both manuals are checked with -`./scripts/verify-workspace-install-docs.sh --profile local` or `--profile server`. +for Docker Desktop or a local engine, the [server installation manual](docs/install/server-workspace-registry.md) +for the Gitea, reverse-proxy, backup, upgrade, and recovery workflow, and the +[P1→P1.1 migration guide](docs/migrations/p1-to-p1-1-registry-layout.md) before upgrading an +older flat-layout registry. -The operator workflow is: update and review canonical YAML in the shared Git remote, **Pull latest -registry** from each ThothII installation, run **Validate workspace** and **Test on this -installation**, then select the workspace locally before creating sessions. Each new session pins -the Git revision it used; a later pull or publish cannot change a Resume. Snapshot cleanup retains -every revision referenced by an open, closed, or failed unarchived session. It reconciles from the +The curator-owned repository layout is: + +```text +thoth-workspaces.yaml +/workspace.yaml +/evidence/** +workspace-docs//{contract.env.example,README.md} +``` + +`thoth-workspaces.yaml` uses the `schema_version` value `1` and the ordered `workspaces` list of +`{id, name, description?}` entries. It is authoritative for workspace ID, name, description, and +display order. The API may create `/workspace.yaml` only when the catalog slot already exists +and the descriptor is absent. A pulled catalog-only slot reports `configuration_required`. After +bootstrap, existing descriptors remain curator-owned and change only through curator Git commit, +push, and installation pull. The API never writes `thoth-workspaces.yaml` or `/evidence/**`; +its generated docs live only at `workspace-docs//{contract.env.example,README.md}`. + +The operator workflow is: curate catalog/descriptor/Evidence changes in Git, commit and push, +**Pull latest registry** from each ThothII installation, run **Validate workspace** and **Test on +this installation**, then select the workspace locally before creating sessions. Each new session +pins the Git revision it used; a later pull cannot change a Resume. Snapshot cleanup retains every +revision referenced by an open, closed, or failed unarchived session. It reconciles from the single local installation list or from a server administrator's complete session list, never from -a remote user's partial list. +a remote user's partial list. The isolated deployment exercise is +`./scripts/workspace-registry-smoke.sh`; both manuals are checked with +`./scripts/verify-workspace-install-docs.sh --profile local` or `--profile server`. Schema v3 is the only accepted workspace descriptor. Schema v1 and v2 workspace descriptors are diff --git a/docs/contracts/workspace-evidence-v3.md b/docs/contracts/workspace-evidence-v3.md index 68e642cf..28b562e0 100644 --- a/docs/contracts/workspace-evidence-v3.md +++ b/docs/contracts/workspace-evidence-v3.md @@ -7,9 +7,9 @@ source variant and the policy reject unknown keys. ## Filesystem source -A filesystem source uses the exact URI `workspace-content//evidence`. `patterns` is -a nonempty list of unique, normalized relative POSIX globs. Its defaults are -`patterns: ["**/*.md"]` and `max_bytes: 10485760`. +A filesystem source uses the exact URI `/evidence`. `patterns` is a nonempty list of +unique, normalized relative POSIX globs. Its defaults are `patterns: ["**/*.md"]` and +`max_bytes: 10485760`. ### Example: filesystem @@ -17,7 +17,7 @@ a nonempty list of unique, normalized relative POSIX globs. Its defaults are evidence: source: type: filesystem - uri: workspace-content/example/evidence + uri: example/evidence patterns: - "**/*.md" max_bytes: 10485760 @@ -26,9 +26,9 @@ evidence: retain_published_generations: 3 ``` -Safe: `workspace-content/example/evidence`. Unsafe filesystem identities include `/srv/evidence`, -`workspace-content/another/evidence`, and `workspace-content/example/../another/evidence` because -absolute, cross-namespace, and traversal paths are not canonical. +Safe: `example/evidence`. Unsafe filesystem identities include `/srv/evidence`, +`another/evidence` and `example/evidence/../another` because cross-namespace and traversal +paths are not canonical. Legacy split-layout paths are rejected as noncanonical. ## HTTP source @@ -133,33 +133,46 @@ All workspace namespaces live in one Git repository: ```text registry.git/ -├── workspaces/ -│ ├── example.yaml -│ └── another.yaml -├── workspace-content/ -│ ├── example/evidence/... -│ └── another/evidence/... +├── thoth-workspaces.yaml +├── example/ +│ ├── workspace.yaml +│ └── evidence/... +├── another/ +│ └── workspace.yaml └── workspace-docs/ ├── example/{contract.env.example,README.md} └── another/{contract.env.example,README.md} ``` -Curators change only `workspace-content//evidence/**` through a normal clone. The API publishes -only `workspaces/.yaml` and -`workspace-docs//{contract.env.example,README.md}`. It never writes Evidence source bytes. +The curator-owned root catalog `thoth-workspaces.yaml` uses the `schema_version` value `1` and the ordered +`workspaces` list of `{id, name, description?}` entries. It is authoritative for workspace ID, +name, description, and display order. The descriptor at `/workspace.yaml` must match the +catalog metadata exactly. `workspace-docs` is the reserved top-level API directory and cannot be a +workspace ID. + +Catalog-only entries without `/workspace.yaml` are valid bootstrap slots and surface as +`configuration_required`. The API may create `/workspace.yaml` only when the catalog slot +already exists and no Git object exists at that path in the exact pulled base commit. After +bootstrap, existing descriptors change only through curator Git commit/push and installation pull. +The API never writes `thoth-workspaces.yaml` or `/evidence/**`. Generated docs stay outside the +workspace namespace at `workspace-docs//{contract.env.example,README.md}`. An explicit docs +synchronization may create a docs-only commit that changes only `workspace-docs/**` and preserves +the catalog, descriptor, and Evidence object IDs. ## Registry revision and phase ownership | Relationship | Contract | | --- | --- | -| Revision identity | The descriptor blob and filesystem Evidence root tree are checked at the same 40-hex Git commit. | -| Content-only revision | An Evidence-only commit changes authoritative `revision.commit` even when the descriptor blob is unchanged. | -| Browser | Create and edit flows preserve and show a read-only Evidence summary. | +| Revision identity | The catalog blob, descriptor blob, and filesystem Evidence root tree are checked at the same 40-hex Git commit. | +| Content-only revision | An Evidence-only commit changes authoritative `revision.commit` even when the catalog and descriptor blobs are unchanged. | +| Docs-only sync commit | A docs-only synchronization may advance `revision.commit`, change only `workspace-docs/**`, and preserve the catalog, descriptor, and Evidence object IDs. | +| Browser | Read-only curated workspaces preserve and show a read-only Evidence summary; only a catalog-only bootstrap slot may draft the first descriptor. | | Export | Export remains exactly manifest, descriptor, contract, and README; it excludes Evidence bytes. | -| P1 | Validates the lexical URI and proves the declared filesystem root object is a Git tree at that same commit; it does not recursively inspect nested symlinks. | +| P1.1 | Validates the lexical URI `/evidence` and proves the declared filesystem root object is a Git tree at that same commit; it does not recursively inspect nested symlinks. Evidence materialization stays out of scope for P1.1. | | P6 | Owns commit-addressed materialization, realpath and recursive containment, nested-symlink checks, and race checks. | -P1 performs no acquisition, extraction, preprocessing/indexing, embeddings, Qdrant writes, `ACTIVE` publication, retention, or GC. +P1.1 performs no acquisition, extraction, preprocessing/indexing, embeddings, Qdrant writes, +`ACTIVE` publication, retention, or GC. ## Operator validation diff --git a/docs/install/local-workspace-registry.md b/docs/install/local-workspace-registry.md index 9dca6257..13fc9c3b 100644 --- a/docs/install/local-workspace-registry.md +++ b/docs/install/local-workspace-registry.md @@ -35,19 +35,24 @@ authentication method. ## Git remote: SSH and HTTPS -Create one private repository such as `thoth-workspaces.git`. It contains canonical workspace -definitions, curated Evidence content, and generated artifacts: +Create one private repository such as `thoth-workspaces.git`. It contains the curator-owned root +catalog, one directory per workspace, optional embedded Evidence trees, and generated public docs: ```text registry.git/ -├── workspaces/ -│ └── .yaml -├── workspace-content/ -│ └── /evidence/... +├── thoth-workspaces.yaml +├── / +│ ├── workspace.yaml +│ └── evidence/... └── workspace-docs/ └── /{contract.env.example,README.md} ``` +`thoth-workspaces.yaml` uses the `schema_version` value `1` and the ordered `workspaces` list of +`{id, name, description?}` entries. It is authoritative for workspace ID, name, description, and +display order; `/workspace.yaml` must match that metadata exactly, while +`workspace-docs` remains the reserved top-level generated-docs directory. + For SSH, use a scoped deploy key, a verified `known_hosts` file, and strict host-key checking. For HTTPS, use Git Credential Manager or a secret-manager-created credentials file. Mount a private HTTPS CA as its own file. Do not disable host or certificate verification. The base Compose file @@ -75,12 +80,24 @@ Follow this order; the [canonical Evidence contract](../contracts/workspace-evid the source shapes and safety boundary. 1. Clone the one shared registry, or update the review clone with `git pull --ff-only`. -2. Add source bytes below `workspace-content//evidence`, then commit and push. -3. Validate and publish the descriptor against that base commit. -4. Inspect `workspace-docs//contract.env.example` and `workspace-docs//README.md`. -5. Provision only the selected Evidence `*_FILE` files outside Git and strictly below a root in `THT_WORKSPACE_SECRET_ROOTS`; add matching host-only `*_SOURCE` paths for the generated connector override. -6. Render or acquire the runtime config, then run `tht config check -c `. -7. Stop: P2/P6 later performs preprocessing and materialization. +2. Keep `thoth-workspaces.yaml` curator-owned. It uses the `schema_version` value `1` and the ordered + `workspaces` list of `{id, name, description?}` entries; it is authoritative for workspace ID, + name, description, and display order. +3. For an existing workspace, edit `/workspace.yaml` and any embedded `/evidence/**`, then + commit and push. +4. For a new workspace, add the catalog slot first. If you want ThothII to bootstrap the + descriptor, leave `/workspace.yaml` absent, commit and push, then pull that commit into the + installation; the slot appears as `configuration_required`. +5. The API may create `/workspace.yaml` only when the catalog slot already exists and no Git + object exists at that path in the exact pulled base commit. +6. After bootstrap, existing descriptors change only through curator Git commit/push and + installation pull. The API never writes `thoth-workspaces.yaml` or `/evidence/**`. +7. Inspect `workspace-docs//contract.env.example` and `workspace-docs//README.md`. +8. Provision only the selected Evidence `*_FILE` files outside Git and strictly below a root in + `THT_WORKSPACE_SECRET_ROOTS`; add matching host-only `*_SOURCE` paths for the generated + connector override. +9. Render or acquire the runtime config, then run `tht config check -c `. +10. Stop: P2/P6 later performs preprocessing and materialization. For example, a signed-HTTP workspace and a different static-S3 workspace can use these host-only connector sources; the values are paths, not file contents: @@ -111,7 +128,7 @@ The persistent volume is `/data/workspace-registry`: repo/ # persistent Git checkout snapshots/ # immutable validated revisions used by sessions state/ # active revision and registry state -locks/ # short-lived publish locks +locks/ # short-lived registry synchronization locks ``` Installation variables are deterministic: `north-star-research` becomes `NORTH_STAR_RESEARCH`, and every name @@ -208,10 +225,14 @@ curl --fail --silent http://127.0.0.1:8787/workspace-registry/status curl --fail --silent http://127.0.0.1:8787/workspaces ``` -The first status request clones, validates all descriptors, and atomically activates a snapshot. -Use `POST /workspace-registry/pull` to fetch later revisions. Run workspace diagnostics only after -required DWH bindings are mounted. Schema-v3 diagnostics probe the internal Qdrant/Ollama -services through backend config; ordinary diagnostics are read-only. +The first status request clones the registry, validates `thoth-workspaces.yaml`, every +catalog-listed descriptor, and any declared `/evidence` tree, then atomically activates a +snapshot. A catalog-only slot with no descriptor reports `configuration_required` and is not +activatable. Use `POST /workspace-registry/pull` to fetch later curator revisions. Existing +curated descriptors stay read-only in the UI; only a missing descriptor may use the one-time +bootstrap create flow. Run workspace diagnostics only after required DWH bindings are mounted. +Schema-v3 diagnostics probe the internal Qdrant/Ollama services through backend config; ordinary +diagnostics are read-only. Schema v3 is the only accepted workspace descriptor. Schema v1 and v2 workspace descriptors are @@ -234,11 +255,16 @@ values, secret values, certificates, keys, or secret files into the repository. ## Publish, update, backup, outage recovery, and rollback -Drafts live only in browser storage. Review the canonical field diff, validate/test locally, then -publish. If conflicted, pull first and create a new reviewed field-level draft; never hand-edit the -running `repo/` volume. Before upgrading, record registry status, stop Compose, and take a -timestamped ownership-preserving backup of both registry and local data volumes while excluding -`installation-secrets/`. Render Compose, rebuild, start, and check status before resuming work. +Existing curated workspaces are read-only in the browser. Use the Workspace +Management page to pull, inspect status, validate a workspace, test it on this installation, and +optionally create one bootstrap descriptor for a pulled `configuration_required` slot. After that +first descriptor exists, change it only through curator Git commit/push and installation pull; +never hand-edit the running `repo/` volume. Before upgrading, record registry status, stop Compose, +and take a timestamped ownership-preserving backup of both registry and local data volumes while +excluding `installation-secrets/`. Render Compose, rebuild, start, and check status before +resuming work. If you are upgrading an older P1 registry, apply the reviewed migration in +[`docs/migrations/p1-to-p1-1-registry-layout.md`](../migrations/p1-to-p1-1-registry-layout.md) +and upgrade ThothII only after that commit is pushed. After a valid bootstrap, remote outage retains the last valid snapshot and reports `degraded: true`. Pinned sessions continue. Repair network/authentication, pull, and confirm non-degraded status. To @@ -252,8 +278,8 @@ normal policy, pull, and confirm its new snapshot. Do not delete `snapshots/` as | `workspace_invalid` | Invalid descriptor/path/snapshot; restore a reviewed canonical revision. | | `binding_missing` | A selected value or readable `*_FILE` is absent; fix the local binding/mount. | | `workspace_not_activatable` | Diagnostics cannot activate the workspace; correct the selected transport. | -| `workspace_stale` | Checkout changed or is busy; stop competing pull/publish work. | -| `workspace_conflict` | Draft base differs from Git; pull, resolve the diff, validate, republish. | +| `workspace_stale` | Checkout changed or is busy; stop competing pull or sync work. | +| `workspace_conflict` | Draft base differs from Git; pull, resolve the diff, validate, and retry after the curator pull/bootstrap flow. | | `git_unavailable` | Remote, path, network, or lock unavailable; preserve the degraded valid snapshot. | | `git_auth_failed` | Mounted SSH/HTTPS material rejected/unreadable; rotate or fix permissions without logging it. | | `git_non_fast_forward` | Checkout diverged; reconcile through the registry workflow. | diff --git a/docs/install/server-workspace-registry.md b/docs/install/server-workspace-registry.md index 26f07bb4..4b11d96b 100644 --- a/docs/install/server-workspace-registry.md +++ b/docs/install/server-workspace-registry.md @@ -50,19 +50,24 @@ account Gitea administration, database-superuser rights, or a shell in the Git h Create a private Gitea (or compatible Git) repository such as `platform/thoth-workspaces`. Protect `main` according to the release policy and grant the ThothII publisher only the intended repository -scope. Commit canonical schema-v3 descriptors under `workspaces/.yaml`, curated Evidence -under `workspace-content//evidence/**`, and generated public artifacts only at -`workspace-docs//README.md` and `workspace-docs//contract.env.example`; do not commit -installation bindings or secret material. +scope. Commit the curator-owned root catalog `thoth-workspaces.yaml`, workspace descriptors under +`/workspace.yaml`, curated embedded Evidence under `/evidence/**`, and generated public +artifacts only at `workspace-docs//README.md` and +`workspace-docs//contract.env.example`; do not commit installation bindings or secret material. + +`thoth-workspaces.yaml` uses the `schema_version` value `1` and the ordered `workspaces` list of +`{id, name, description?}` entries. It is authoritative for workspace ID, name, description, and +display order. `/workspace.yaml` must match that metadata exactly, and `workspace-docs` +remains the reserved top-level generated-docs directory. For SSH, create a least-privilege deploy key, record Gitea's host key in managed known-hosts, and use `ssh://git@git.example.invalid/platform/thoth-workspaces.git`. For HTTPS, create a scoped machine credential in the secret manager and mount the Gitea/private CA separately. Never use a Gitea admin credential in the application. -Bootstrap an empty remote from a temporary review clone only after its canonical v3 descriptors -and generated public artifacts have been reviewed; commit and push `main`. The running server is -not a descriptor authoring or conversion environment. +Bootstrap an empty remote from a temporary review clone only after the catalog, descriptors, and +generated public artifacts have been reviewed; commit and push `main`. The running server is not a +descriptor authoring or conversion environment. ## Curator flow for shared-registry Evidence @@ -70,12 +75,24 @@ Follow this order; the [canonical Evidence contract](../contracts/workspace-evid the source shapes and safety boundary. 1. Clone the one shared registry, or update the review clone with `git pull --ff-only`. -2. Add source bytes below `workspace-content//evidence`, then commit and push. -3. Validate and publish the descriptor against that base commit. -4. Inspect `workspace-docs//contract.env.example` and `workspace-docs//README.md`. -5. Provision only the selected Evidence `*_FILE` files outside Git and strictly below a root in `THT_WORKSPACE_SECRET_ROOTS`; add matching host-only `*_SOURCE` paths for the generated connector override. -6. Render or acquire the runtime config, then run `tht config check -c `. -7. Stop: P2/P6 later performs preprocessing and materialization. +2. Keep `thoth-workspaces.yaml` curator-owned. It uses the `schema_version` value `1` and the ordered + `workspaces` list of `{id, name, description?}` entries; it is authoritative for workspace ID, + name, description, and display order. +3. For an existing workspace, edit `/workspace.yaml` and any embedded `/evidence/**`, then + commit and push. +4. For a new workspace, add the catalog slot first. If you want ThothII to bootstrap the + descriptor, leave `/workspace.yaml` absent, commit and push, then pull that commit into the + installation; the slot appears as `configuration_required`. +5. The API may create `/workspace.yaml` only when the catalog slot already exists and no Git + object exists at that path in the exact pulled base commit. +6. After bootstrap, existing descriptors change only through curator Git commit/push and + installation pull. The API never writes `thoth-workspaces.yaml` or `/evidence/**`. +7. Inspect `workspace-docs//contract.env.example` and `workspace-docs//README.md`. +8. Provision only the selected Evidence `*_FILE` files outside Git and strictly below a root in + `THT_WORKSPACE_SECRET_ROOTS`; add matching host-only `*_SOURCE` paths for the generated + connector override. +9. Render or acquire the runtime config, then run `tht config check -c `. +10. Stop: P2/P6 later performs preprocessing and materialization. For example, separate signed-HTTP and static-S3 workspaces can use these host-only connector source paths: @@ -241,22 +258,28 @@ without authenticating the request is not an identity boundary. `/health` is liveness. The authenticated Workspace Management page's registry status verifies branch/head/degraded state and the active validated snapshot; its workspace listing verifies -application access. A server with no active snapshot is not ready for workspace sessions even if -liveness succeeds. +application access. A catalog-only slot with no descriptor reports `configuration_required` and is +not ready for sessions until either the curator commits `/workspace.yaml` or the one-time +bootstrap create flow writes it. A server with no active snapshot is not ready for workspace +sessions even if liveness succeeds. ## Pull, publish, upgrade, backup, and recovery -Use the authenticated Workspace Management UI or `POST /workspace-registry/pull`. Drafts are -browser-local. Publish takes a canonical diff, validates before commit, and pushes under a registry -lock. On `workspace_conflict`, pull, resolve the reviewed field-level draft, validate/test, and -publish; never edit `repo/` inside a running volume. +Use the authenticated Workspace Management UI or `POST /workspace-registry/pull` to fetch later +curator revisions. Existing curated workspaces are read-only in the browser. Use the UI to inspect +status, validate a workspace, test it on this installation, and optionally create one bootstrap +descriptor for a pulled `configuration_required` slot. After that first descriptor exists, change +it only through curator Git commit/push and installation pull; never edit `repo/` inside a running +volume. For upgrades, record active status/head, finish active work, use the documented `thothctl pi update --drain` transaction when Pi/core changes, and take a stopped, filesystem-consistent backup of `/srv/thothii/workspace-registry` plus `/srv/thothii/data` and Pi state. Exclude `/srv/thothii/secrets` from the ordinary archive. Validate the descriptor with `thothctl update --check-only`, deploy the compatible image through `thothctl`, verify health/status, then resume -proxy traffic. +proxy traffic. If you are upgrading an older P1 registry, apply the reviewed migration in +[`docs/migrations/p1-to-p1-1-registry-layout.md`](../migrations/p1-to-p1-1-registry-layout.md) +and upgrade ThothII only after that commit is pushed. Schema v3 is the only accepted workspace descriptor. Schema v1 and v2 workspace descriptors are @@ -291,11 +314,11 @@ do not delete snapshots as a rollback shortcut. | `binding_missing` | Missing/invalid local value or readable `*_FILE`; correct mount and permissions. | | `workspace_not_activatable` | Bindings/diagnostics cannot activate; use sanitized fields to fix selected transport. | | `workspace_stale` | Checkout changed/locked; stop concurrent registry work, never force Git in the volume. | -| `workspace_conflict` | Draft base stale; pull, resolve, validate, republish. | +| `workspace_conflict` | Draft base stale; pull, resolve, validate, and retry after the curator pull/bootstrap flow. | | `git_unavailable` | Storage/remote/DNS/firewall/lock failed; preserve degraded active state while repairing it. | | `git_auth_failed` | SSH/HTTPS material rejected or unreadable; rotate/fix file without printing it. | | `git_non_fast_forward` | Checkout diverged; reconcile through registry workflow and branch policy. | -| `git_push_rejected` | Gitea policy rejected publish; review hooks/branch protection. | +| `git_push_rejected` | Gitea policy rejected the curator push or docs sync commit; review hooks/branch protection. | | `connector_unavailable` | DNS/TLS/auth/resource identity failed; check egress and local bindings. | | `semantic_index_incompatible` | Collection/model/dimensions/distance differs; perform explicit index migration. | diff --git a/docs/migrations/p1-to-p1-1-registry-layout.md b/docs/migrations/p1-to-p1-1-registry-layout.md new file mode 100644 index 00000000..75a272c3 --- /dev/null +++ b/docs/migrations/p1-to-p1-1-registry-layout.md @@ -0,0 +1,38 @@ +# P1 to P1.1 registry layout migration + +P1.1 is a repository-contract cutover. New ThothII builds reject the old flat layout and a +repository without `thoth-workspaces.yaml`, so migrate the registry in Git first and upgrade the +application only after that reviewed migration commit is pushed. + +## One reviewed migration commit + +Perform the layout move in a clean review clone and keep it in one reviewed Git commit: + +```sh +git mv workspaces/.yaml /workspace.yaml +git mv workspace-content//evidence /evidence +# create and review thoth-workspaces.yaml from descriptor metadata +``` + +For every workspace directory, preserve the existing descriptor bytes, move only the embedded +filesystem Evidence tree, and create `thoth-workspaces.yaml` with: + +- `schema_version: 1` +- the ordered `workspaces` list +- curator-owned `id`, `name`, and optional `description` copied from the reviewed descriptors + +Generated docs remain under `workspace-docs//`. Do not add an auto-migrator and do not let the +API rewrite the catalog or Evidence tree. + +## Cutover order + +1. Review the migration commit, including the new `thoth-workspaces.yaml` metadata. +2. Push that commit to the authoritative registry branch. +3. Upgrade ThothII only after that migration commit is pushed. +4. Pull the migrated registry into each installation before using workspace management. + +## Rollback + +Roll back the application revision and registry commit together. Do not point a P1.1 binary at the +old flat layout, and do not keep a migrated registry commit active while rolling the application +back to pre-P1.1 code.