docs: define the P1.1 registry layout and curator flow

This commit is contained in:
2026-08-11 15:22:57 +02:00
parent a071e0baff
commit c2f33e58d7
5 changed files with 196 additions and 77 deletions
+34 -21
View File
@@ -7,9 +7,9 @@ source variant and the policy reject unknown keys.
## Filesystem source
A filesystem source uses the exact URI `workspace-content/<workspace.id>/evidence`. `patterns` is
a nonempty list of unique, normalized relative POSIX globs. Its defaults are
`patterns: ["**/*.md"]` and `max_bytes: 10485760`.
A filesystem source uses the exact URI `<workspace.id>/evidence`. `patterns` is a nonempty list of
unique, normalized relative POSIX globs. Its defaults are `patterns: ["**/*.md"]` and
`max_bytes: 10485760`.
### Example: filesystem
@@ -17,7 +17,7 @@ a nonempty list of unique, normalized relative POSIX globs. Its defaults are
evidence:
source:
type: filesystem
uri: workspace-content/example/evidence
uri: example/evidence
patterns:
- "**/*.md"
max_bytes: 10485760
@@ -26,9 +26,9 @@ evidence:
retain_published_generations: 3
```
Safe: `workspace-content/example/evidence`. Unsafe filesystem identities include `/srv/evidence`,
`workspace-content/another/evidence`, and `workspace-content/example/../another/evidence` because
absolute, cross-namespace, and traversal paths are not canonical.
Safe: `example/evidence`. Unsafe filesystem identities include `/srv/evidence`,
`another/evidence` and `example/evidence/../another` because cross-namespace and traversal
paths are not canonical. Legacy split-layout paths are rejected as noncanonical.
## HTTP source
@@ -133,33 +133,46 @@ All workspace namespaces live in one Git repository:
```text
registry.git/
├── workspaces/
│ ├── example.yaml
│ └── another.yaml
├── workspace-content/
│ ├── example/evidence/...
│ └── another/evidence/...
├── thoth-workspaces.yaml
├── example/
│ ├── workspace.yaml
│ └── evidence/...
├── another/
│ └── workspace.yaml
└── workspace-docs/
├── example/{contract.env.example,README.md}
└── another/{contract.env.example,README.md}
```
Curators change only `workspace-content/<id>/evidence/**` through a normal clone. The API publishes
only `workspaces/<id>.yaml` and
`workspace-docs/<id>/{contract.env.example,README.md}`. It never writes Evidence source bytes.
The curator-owned root catalog `thoth-workspaces.yaml` uses the `schema_version` value `1` and the ordered
`workspaces` list of `{id, name, description?}` entries. It is authoritative for workspace ID,
name, description, and display order. The descriptor at `<id>/workspace.yaml` must match the
catalog metadata exactly. `workspace-docs` is the reserved top-level API directory and cannot be a
workspace ID.
Catalog-only entries without `<id>/workspace.yaml` are valid bootstrap slots and surface as
`configuration_required`. The API may create `<id>/workspace.yaml` only when the catalog slot
already exists and no Git object exists at that path in the exact pulled base commit. After
bootstrap, existing descriptors change only through curator Git commit/push and installation pull.
The API never writes `thoth-workspaces.yaml` or `<id>/evidence/**`. Generated docs stay outside the
workspace namespace at `workspace-docs/<id>/{contract.env.example,README.md}`. An explicit docs
synchronization may create a docs-only commit that changes only `workspace-docs/**` and preserves
the catalog, descriptor, and Evidence object IDs.
## Registry revision and phase ownership
| Relationship | Contract |
| --- | --- |
| Revision identity | The descriptor blob and filesystem Evidence root tree are checked at the same 40-hex Git commit. |
| Content-only revision | An Evidence-only commit changes authoritative `revision.commit` even when the descriptor blob is unchanged. |
| Browser | Create and edit flows preserve and show a read-only Evidence summary. |
| Revision identity | The catalog blob, descriptor blob, and filesystem Evidence root tree are checked at the same 40-hex Git commit. |
| Content-only revision | An Evidence-only commit changes authoritative `revision.commit` even when the catalog and descriptor blobs are unchanged. |
| Docs-only sync commit | A docs-only synchronization may advance `revision.commit`, change only `workspace-docs/**`, and preserve the catalog, descriptor, and Evidence object IDs. |
| Browser | Read-only curated workspaces preserve and show a read-only Evidence summary; only a catalog-only bootstrap slot may draft the first descriptor. |
| Export | Export remains exactly manifest, descriptor, contract, and README; it excludes Evidence bytes. |
| P1 | Validates the lexical URI and proves the declared filesystem root object is a Git tree at that same commit; it does not recursively inspect nested symlinks. |
| P1.1 | Validates the lexical URI `<id>/evidence` and proves the declared filesystem root object is a Git tree at that same commit; it does not recursively inspect nested symlinks. Evidence materialization stays out of scope for P1.1. |
| P6 | Owns commit-addressed materialization, realpath and recursive containment, nested-symlink checks, and race checks. |
P1 performs no acquisition, extraction, preprocessing/indexing, embeddings, Qdrant writes, `ACTIVE` publication, retention, or GC.
P1.1 performs no acquisition, extraction, preprocessing/indexing, embeddings, Qdrant writes,
`ACTIVE` publication, retention, or GC.
## Operator validation
+50 -24
View File
@@ -35,19 +35,24 @@ authentication method.
## Git remote: SSH and HTTPS
Create one private repository such as `thoth-workspaces.git`. It contains canonical workspace
definitions, curated Evidence content, and generated artifacts:
Create one private repository such as `thoth-workspaces.git`. It contains the curator-owned root
catalog, one directory per workspace, optional embedded Evidence trees, and generated public docs:
```text
registry.git/
├── workspaces/
│ └── <workspace-id>.yaml
├── workspace-content/
│ └── <workspace-id>/evidence/...
├── thoth-workspaces.yaml
├── <workspace-id>/
│ ├── workspace.yaml
│ └── evidence/...
└── workspace-docs/
└── <workspace-id>/{contract.env.example,README.md}
```
`thoth-workspaces.yaml` uses the `schema_version` value `1` and the ordered `workspaces` list of
`{id, name, description?}` entries. It is authoritative for workspace ID, name, description, and
display order; `<workspace-id>/workspace.yaml` must match that metadata exactly, while
`workspace-docs` remains the reserved top-level generated-docs directory.
For SSH, use a scoped deploy key, a verified `known_hosts` file, and strict host-key checking. For
HTTPS, use Git Credential Manager or a secret-manager-created credentials file. Mount a private
HTTPS CA as its own file. Do not disable host or certificate verification. The base Compose file
@@ -75,12 +80,24 @@ Follow this order; the [canonical Evidence contract](../contracts/workspace-evid
the source shapes and safety boundary.
1. Clone the one shared registry, or update the review clone with `git pull --ff-only`.
2. Add source bytes below `workspace-content/<id>/evidence`, then commit and push.
3. Validate and publish the descriptor against that base commit.
4. Inspect `workspace-docs/<id>/contract.env.example` and `workspace-docs/<id>/README.md`.
5. Provision only the selected Evidence `*_FILE` files outside Git and strictly below a root in `THT_WORKSPACE_SECRET_ROOTS`; add matching host-only `*_SOURCE` paths for the generated connector override.
6. Render or acquire the runtime config, then run `tht config check -c <path>`.
7. Stop: P2/P6 later performs preprocessing and materialization.
2. Keep `thoth-workspaces.yaml` curator-owned. It uses the `schema_version` value `1` and the ordered
`workspaces` list of `{id, name, description?}` entries; it is authoritative for workspace ID,
name, description, and display order.
3. For an existing workspace, edit `<id>/workspace.yaml` and any embedded `<id>/evidence/**`, then
commit and push.
4. For a new workspace, add the catalog slot first. If you want ThothII to bootstrap the
descriptor, leave `<id>/workspace.yaml` absent, commit and push, then pull that commit into the
installation; the slot appears as `configuration_required`.
5. The API may create `<id>/workspace.yaml` only when the catalog slot already exists and no Git
object exists at that path in the exact pulled base commit.
6. After bootstrap, existing descriptors change only through curator Git commit/push and
installation pull. The API never writes `thoth-workspaces.yaml` or `<id>/evidence/**`.
7. Inspect `workspace-docs/<id>/contract.env.example` and `workspace-docs/<id>/README.md`.
8. Provision only the selected Evidence `*_FILE` files outside Git and strictly below a root in
`THT_WORKSPACE_SECRET_ROOTS`; add matching host-only `*_SOURCE` paths for the generated
connector override.
9. Render or acquire the runtime config, then run `tht config check -c <path>`.
10. Stop: P2/P6 later performs preprocessing and materialization.
For example, a signed-HTTP workspace and a different static-S3 workspace can use these host-only
connector sources; the values are paths, not file contents:
@@ -111,7 +128,7 @@ The persistent volume is `/data/workspace-registry`:
repo/ # persistent Git checkout
snapshots/ # immutable validated revisions used by sessions
state/ # active revision and registry state
locks/ # short-lived publish locks
locks/ # short-lived registry synchronization locks
```
Installation variables are deterministic: `north-star-research` becomes `NORTH_STAR_RESEARCH`, and every name
@@ -208,10 +225,14 @@ curl --fail --silent http://127.0.0.1:8787/workspace-registry/status
curl --fail --silent http://127.0.0.1:8787/workspaces
```
The first status request clones, validates all descriptors, and atomically activates a snapshot.
Use `POST /workspace-registry/pull` to fetch later revisions. Run workspace diagnostics only after
required DWH bindings are mounted. Schema-v3 diagnostics probe the internal Qdrant/Ollama
services through backend config; ordinary diagnostics are read-only.
The first status request clones the registry, validates `thoth-workspaces.yaml`, every
catalog-listed descriptor, and any declared `<id>/evidence` tree, then atomically activates a
snapshot. A catalog-only slot with no descriptor reports `configuration_required` and is not
activatable. Use `POST /workspace-registry/pull` to fetch later curator revisions. Existing
curated descriptors stay read-only in the UI; only a missing descriptor may use the one-time
bootstrap create flow. Run workspace diagnostics only after required DWH bindings are mounted.
Schema-v3 diagnostics probe the internal Qdrant/Ollama services through backend config; ordinary
diagnostics are read-only.
<!-- workspace-descriptor-contract:start -->
Schema v3 is the only accepted workspace descriptor. Schema v1 and v2 workspace descriptors are
@@ -234,11 +255,16 @@ values, secret values, certificates, keys, or secret files into the repository.
## Publish, update, backup, outage recovery, and rollback
Drafts live only in browser storage. Review the canonical field diff, validate/test locally, then
publish. If conflicted, pull first and create a new reviewed field-level draft; never hand-edit the
running `repo/` volume. Before upgrading, record registry status, stop Compose, and take a
timestamped ownership-preserving backup of both registry and local data volumes while excluding
`installation-secrets/`. Render Compose, rebuild, start, and check status before resuming work.
Existing curated workspaces are read-only in the browser. Use the Workspace
Management page to pull, inspect status, validate a workspace, test it on this installation, and
optionally create one bootstrap descriptor for a pulled `configuration_required` slot. After that
first descriptor exists, change it only through curator Git commit/push and installation pull;
never hand-edit the running `repo/` volume. Before upgrading, record registry status, stop Compose,
and take a timestamped ownership-preserving backup of both registry and local data volumes while
excluding `installation-secrets/`. Render Compose, rebuild, start, and check status before
resuming work. If you are upgrading an older P1 registry, apply the reviewed migration in
[`docs/migrations/p1-to-p1-1-registry-layout.md`](../migrations/p1-to-p1-1-registry-layout.md)
and upgrade ThothII only after that commit is pushed.
After a valid bootstrap, remote outage retains the last valid snapshot and reports `degraded: true`.
Pinned sessions continue. Repair network/authentication, pull, and confirm non-degraded status. To
@@ -252,8 +278,8 @@ normal policy, pull, and confirm its new snapshot. Do not delete `snapshots/` as
| `workspace_invalid` | Invalid descriptor/path/snapshot; restore a reviewed canonical revision. |
| `binding_missing` | A selected value or readable `*_FILE` is absent; fix the local binding/mount. |
| `workspace_not_activatable` | Diagnostics cannot activate the workspace; correct the selected transport. |
| `workspace_stale` | Checkout changed or is busy; stop competing pull/publish work. |
| `workspace_conflict` | Draft base differs from Git; pull, resolve the diff, validate, republish. |
| `workspace_stale` | Checkout changed or is busy; stop competing pull or sync work. |
| `workspace_conflict` | Draft base differs from Git; pull, resolve the diff, validate, and retry after the curator pull/bootstrap flow. |
| `git_unavailable` | Remote, path, network, or lock unavailable; preserve the degraded valid snapshot. |
| `git_auth_failed` | Mounted SSH/HTTPS material rejected/unreadable; rotate or fix permissions without logging it. |
| `git_non_fast_forward` | Checkout diverged; reconcile through the registry workflow. |
+45 -22
View File
@@ -50,19 +50,24 @@ account Gitea administration, database-superuser rights, or a shell in the Git h
Create a private Gitea (or compatible Git) repository such as `platform/thoth-workspaces`. Protect
`main` according to the release policy and grant the ThothII publisher only the intended repository
scope. Commit canonical schema-v3 descriptors under `workspaces/<id>.yaml`, curated Evidence
under `workspace-content/<id>/evidence/**`, and generated public artifacts only at
`workspace-docs/<id>/README.md` and `workspace-docs/<id>/contract.env.example`; do not commit
installation bindings or secret material.
scope. Commit the curator-owned root catalog `thoth-workspaces.yaml`, workspace descriptors under
`<id>/workspace.yaml`, curated embedded Evidence under `<id>/evidence/**`, and generated public
artifacts only at `workspace-docs/<id>/README.md` and
`workspace-docs/<id>/contract.env.example`; do not commit installation bindings or secret material.
`thoth-workspaces.yaml` uses the `schema_version` value `1` and the ordered `workspaces` list of
`{id, name, description?}` entries. It is authoritative for workspace ID, name, description, and
display order. `<id>/workspace.yaml` must match that metadata exactly, and `workspace-docs`
remains the reserved top-level generated-docs directory.
For SSH, create a least-privilege deploy key, record Gitea's host key in managed known-hosts, and
use `ssh://git@git.example.invalid/platform/thoth-workspaces.git`. For HTTPS, create a scoped
machine credential in the secret manager and mount the Gitea/private CA separately. Never use a
Gitea admin credential in the application.
Bootstrap an empty remote from a temporary review clone only after its canonical v3 descriptors
and generated public artifacts have been reviewed; commit and push `main`. The running server is
not a descriptor authoring or conversion environment.
Bootstrap an empty remote from a temporary review clone only after the catalog, descriptors, and
generated public artifacts have been reviewed; commit and push `main`. The running server is not a
descriptor authoring or conversion environment.
## Curator flow for shared-registry Evidence
@@ -70,12 +75,24 @@ Follow this order; the [canonical Evidence contract](../contracts/workspace-evid
the source shapes and safety boundary.
1. Clone the one shared registry, or update the review clone with `git pull --ff-only`.
2. Add source bytes below `workspace-content/<id>/evidence`, then commit and push.
3. Validate and publish the descriptor against that base commit.
4. Inspect `workspace-docs/<id>/contract.env.example` and `workspace-docs/<id>/README.md`.
5. Provision only the selected Evidence `*_FILE` files outside Git and strictly below a root in `THT_WORKSPACE_SECRET_ROOTS`; add matching host-only `*_SOURCE` paths for the generated connector override.
6. Render or acquire the runtime config, then run `tht config check -c <path>`.
7. Stop: P2/P6 later performs preprocessing and materialization.
2. Keep `thoth-workspaces.yaml` curator-owned. It uses the `schema_version` value `1` and the ordered
`workspaces` list of `{id, name, description?}` entries; it is authoritative for workspace ID,
name, description, and display order.
3. For an existing workspace, edit `<id>/workspace.yaml` and any embedded `<id>/evidence/**`, then
commit and push.
4. For a new workspace, add the catalog slot first. If you want ThothII to bootstrap the
descriptor, leave `<id>/workspace.yaml` absent, commit and push, then pull that commit into the
installation; the slot appears as `configuration_required`.
5. The API may create `<id>/workspace.yaml` only when the catalog slot already exists and no Git
object exists at that path in the exact pulled base commit.
6. After bootstrap, existing descriptors change only through curator Git commit/push and
installation pull. The API never writes `thoth-workspaces.yaml` or `<id>/evidence/**`.
7. Inspect `workspace-docs/<id>/contract.env.example` and `workspace-docs/<id>/README.md`.
8. Provision only the selected Evidence `*_FILE` files outside Git and strictly below a root in
`THT_WORKSPACE_SECRET_ROOTS`; add matching host-only `*_SOURCE` paths for the generated
connector override.
9. Render or acquire the runtime config, then run `tht config check -c <path>`.
10. Stop: P2/P6 later performs preprocessing and materialization.
For example, separate signed-HTTP and static-S3 workspaces can use these host-only connector source
paths:
@@ -241,22 +258,28 @@ without authenticating the request is not an identity boundary.
`/health` is liveness. The authenticated Workspace Management page's registry status verifies
branch/head/degraded state and the active validated snapshot; its workspace listing verifies
application access. A server with no active snapshot is not ready for workspace sessions even if
liveness succeeds.
application access. A catalog-only slot with no descriptor reports `configuration_required` and is
not ready for sessions until either the curator commits `<id>/workspace.yaml` or the one-time
bootstrap create flow writes it. A server with no active snapshot is not ready for workspace
sessions even if liveness succeeds.
## Pull, publish, upgrade, backup, and recovery
Use the authenticated Workspace Management UI or `POST /workspace-registry/pull`. Drafts are
browser-local. Publish takes a canonical diff, validates before commit, and pushes under a registry
lock. On `workspace_conflict`, pull, resolve the reviewed field-level draft, validate/test, and
publish; never edit `repo/` inside a running volume.
Use the authenticated Workspace Management UI or `POST /workspace-registry/pull` to fetch later
curator revisions. Existing curated workspaces are read-only in the browser. Use the UI to inspect
status, validate a workspace, test it on this installation, and optionally create one bootstrap
descriptor for a pulled `configuration_required` slot. After that first descriptor exists, change
it only through curator Git commit/push and installation pull; never edit `repo/` inside a running
volume.
For upgrades, record active status/head, finish active work, use the documented `thothctl pi update
--drain` transaction when Pi/core changes, and take a stopped, filesystem-consistent backup of
`/srv/thothii/workspace-registry` plus `/srv/thothii/data` and Pi state. Exclude
`/srv/thothii/secrets` from the ordinary archive. Validate the descriptor with `thothctl update
--check-only`, deploy the compatible image through `thothctl`, verify health/status, then resume
proxy traffic.
proxy traffic. If you are upgrading an older P1 registry, apply the reviewed migration in
[`docs/migrations/p1-to-p1-1-registry-layout.md`](../migrations/p1-to-p1-1-registry-layout.md)
and upgrade ThothII only after that commit is pushed.
<!-- workspace-descriptor-contract:start -->
Schema v3 is the only accepted workspace descriptor. Schema v1 and v2 workspace descriptors are
@@ -291,11 +314,11 @@ do not delete snapshots as a rollback shortcut.
| `binding_missing` | Missing/invalid local value or readable `*_FILE`; correct mount and permissions. |
| `workspace_not_activatable` | Bindings/diagnostics cannot activate; use sanitized fields to fix selected transport. |
| `workspace_stale` | Checkout changed/locked; stop concurrent registry work, never force Git in the volume. |
| `workspace_conflict` | Draft base stale; pull, resolve, validate, republish. |
| `workspace_conflict` | Draft base stale; pull, resolve, validate, and retry after the curator pull/bootstrap flow. |
| `git_unavailable` | Storage/remote/DNS/firewall/lock failed; preserve degraded active state while repairing it. |
| `git_auth_failed` | SSH/HTTPS material rejected or unreadable; rotate/fix file without printing it. |
| `git_non_fast_forward` | Checkout diverged; reconcile through registry workflow and branch policy. |
| `git_push_rejected` | Gitea policy rejected publish; review hooks/branch protection. |
| `git_push_rejected` | Gitea policy rejected the curator push or docs sync commit; review hooks/branch protection. |
| `connector_unavailable` | DNS/TLS/auth/resource identity failed; check egress and local bindings. |
| `semantic_index_incompatible` | Collection/model/dimensions/distance differs; perform explicit index migration. |
@@ -0,0 +1,38 @@
# P1 to P1.1 registry layout migration
P1.1 is a repository-contract cutover. New ThothII builds reject the old flat layout and a
repository without `thoth-workspaces.yaml`, so migrate the registry in Git first and upgrade the
application only after that reviewed migration commit is pushed.
## One reviewed migration commit
Perform the layout move in a clean review clone and keep it in one reviewed Git commit:
```sh
git mv workspaces/<id>.yaml <id>/workspace.yaml
git mv workspace-content/<id>/evidence <id>/evidence
# create and review thoth-workspaces.yaml from descriptor metadata
```
For every workspace directory, preserve the existing descriptor bytes, move only the embedded
filesystem Evidence tree, and create `thoth-workspaces.yaml` with:
- `schema_version: 1`
- the ordered `workspaces` list
- curator-owned `id`, `name`, and optional `description` copied from the reviewed descriptors
Generated docs remain under `workspace-docs/<id>/`. Do not add an auto-migrator and do not let the
API rewrite the catalog or Evidence tree.
## Cutover order
1. Review the migration commit, including the new `thoth-workspaces.yaml` metadata.
2. Push that commit to the authoritative registry branch.
3. Upgrade ThothII only after that migration commit is pushed.
4. Pull the migrated registry into each installation before using workspace management.
## Rollback
Roll back the application revision and registry commit together. Do not point a P1.1 binary at the
old flat layout, and do not keep a migrated registry commit active while rolling the application
back to pre-P1.1 code.