mptyl/ThothII
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs: define the P1.1 registry layout and curator flow

Browse Source
This commit is contained in:
marcopan
2026-08-11 15:22:57 +02:00
parent a071e0baff
commit c2f33e58d7
5 changed files with 196 additions and 77 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/install/local-workspace-registry.md
+50 -24
View File
@@ -35,19 +35,24 @@ authentication method.
## Git remote: SSH and HTTPS
Create one private repository such as `thoth-workspaces.git`. It contains canonical workspace
definitions, curated Evidence content, and generated artifacts:
Create one private repository such as `thoth-workspaces.git`. It contains the curator-owned root
catalog, one directory per workspace, optional embedded Evidence trees, and generated public docs:
```text
registry.git/
├── workspaces/
│ └── <workspace-id>.yaml
├── workspace-content/
│ └── <workspace-id>/evidence/...
├── thoth-workspaces.yaml
├── <workspace-id>/
│ ├── workspace.yaml
│ └── evidence/...
└── workspace-docs/
└── <workspace-id>/{contract.env.example,README.md}
```
`thoth-workspaces.yaml` uses the `schema_version` value `1` and the ordered `workspaces` list of
`{id, name, description?}` entries. It is authoritative for workspace ID, name, description, and
display order; `<workspace-id>/workspace.yaml` must match that metadata exactly, while
`workspace-docs` remains the reserved top-level generated-docs directory.
For SSH, use a scoped deploy key, a verified `known_hosts` file, and strict host-key checking. For
HTTPS, use Git Credential Manager or a secret-manager-created credentials file. Mount a private
HTTPS CA as its own file. Do not disable host or certificate verification. The base Compose file
@@ -75,12 +80,24 @@ Follow this order; the [canonical Evidence contract](../contracts/workspace-evid
the source shapes and safety boundary.
1. Clone the one shared registry, or update the review clone with `git pull --ff-only`.
2. Add source bytes below `workspace-content/<id>/evidence`, then commit and push.
3. Validate and publish the descriptor against that base commit.
4. Inspect `workspace-docs/<id>/contract.env.example` and `workspace-docs/<id>/README.md`.
5. Provision only the selected Evidence `*_FILE` files outside Git and strictly below a root in `THT_WORKSPACE_SECRET_ROOTS`; add matching host-only `*_SOURCE` paths for the generated connector override.
6. Render or acquire the runtime config, then run `tht config check -c <path>`.
7. Stop: P2/P6 later performs preprocessing and materialization.
2. Keep `thoth-workspaces.yaml` curator-owned. It uses the `schema_version` value `1` and the ordered
`workspaces` list of `{id, name, description?}` entries; it is authoritative for workspace ID,
name, description, and display order.
3. For an existing workspace, edit `<id>/workspace.yaml` and any embedded `<id>/evidence/**`, then
commit and push.
4. For a new workspace, add the catalog slot first. If you want ThothII to bootstrap the
descriptor, leave `<id>/workspace.yaml` absent, commit and push, then pull that commit into the
installation; the slot appears as `configuration_required`.
5. The API may create `<id>/workspace.yaml` only when the catalog slot already exists and no Git
object exists at that path in the exact pulled base commit.
6. After bootstrap, existing descriptors change only through curator Git commit/push and
installation pull. The API never writes `thoth-workspaces.yaml` or `<id>/evidence/**`.
7. Inspect `workspace-docs/<id>/contract.env.example` and `workspace-docs/<id>/README.md`.
8. Provision only the selected Evidence `*_FILE` files outside Git and strictly below a root in
`THT_WORKSPACE_SECRET_ROOTS`; add matching host-only `*_SOURCE` paths for the generated
connector override.
9. Render or acquire the runtime config, then run `tht config check -c <path>`.
10. Stop: P2/P6 later performs preprocessing and materialization.
For example, a signed-HTTP workspace and a different static-S3 workspace can use these host-only
connector sources; the values are paths, not file contents:
@@ -111,7 +128,7 @@ The persistent volume is `/data/workspace-registry`:
repo/ # persistent Git checkout
snapshots/ # immutable validated revisions used by sessions
state/ # active revision and registry state
locks/ # short-lived publish locks
locks/ # short-lived registry synchronization locks
```
Installation variables are deterministic: `north-star-research` becomes `NORTH_STAR_RESEARCH`, and every name
@@ -208,10 +225,14 @@ curl --fail --silent http://127.0.0.1:8787/workspace-registry/status
curl --fail --silent http://127.0.0.1:8787/workspaces
```
The first status request clones, validates all descriptors, and atomically activates a snapshot.
Use `POST /workspace-registry/pull` to fetch later revisions. Run workspace diagnostics only after
required DWH bindings are mounted. Schema-v3 diagnostics probe the internal Qdrant/Ollama
services through backend config; ordinary diagnostics are read-only.
The first status request clones the registry, validates `thoth-workspaces.yaml`, every
catalog-listed descriptor, and any declared `<id>/evidence` tree, then atomically activates a
snapshot. A catalog-only slot with no descriptor reports `configuration_required` and is not
activatable. Use `POST /workspace-registry/pull` to fetch later curator revisions. Existing
curated descriptors stay read-only in the UI; only a missing descriptor may use the one-time
bootstrap create flow. Run workspace diagnostics only after required DWH bindings are mounted.
Schema-v3 diagnostics probe the internal Qdrant/Ollama services through backend config; ordinary
diagnostics are read-only.
<!-- workspace-descriptor-contract:start -->
Schema v3 is the only accepted workspace descriptor. Schema v1 and v2 workspace descriptors are
@@ -234,11 +255,16 @@ values, secret values, certificates, keys, or secret files into the repository.
## Publish, update, backup, outage recovery, and rollback
Drafts live only in browser storage. Review the canonical field diff, validate/test locally, then
publish. If conflicted, pull first and create a new reviewed field-level draft; never hand-edit the
running `repo/` volume. Before upgrading, record registry status, stop Compose, and take a
timestamped ownership-preserving backup of both registry and local data volumes while excluding
`installation-secrets/`. Render Compose, rebuild, start, and check status before resuming work.
Existing curated workspaces are read-only in the browser. Use the Workspace
Management page to pull, inspect status, validate a workspace, test it on this installation, and
optionally create one bootstrap descriptor for a pulled `configuration_required` slot. After that
first descriptor exists, change it only through curator Git commit/push and installation pull;
never hand-edit the running `repo/` volume. Before upgrading, record registry status, stop Compose,
and take a timestamped ownership-preserving backup of both registry and local data volumes while
excluding `installation-secrets/`. Render Compose, rebuild, start, and check status before
resuming work. If you are upgrading an older P1 registry, apply the reviewed migration in
[`docs/migrations/p1-to-p1-1-registry-layout.md`](../migrations/p1-to-p1-1-registry-layout.md)
and upgrade ThothII only after that commit is pushed.
After a valid bootstrap, remote outage retains the last valid snapshot and reports `degraded: true`.
Pinned sessions continue. Repair network/authentication, pull, and confirm non-degraded status. To
@@ -252,8 +278,8 @@ normal policy, pull, and confirm its new snapshot. Do not delete `snapshots/` as
| `workspace_invalid` | Invalid descriptor/path/snapshot; restore a reviewed canonical revision. |
| `binding_missing` | A selected value or readable `*_FILE` is absent; fix the local binding/mount. |
| `workspace_not_activatable` | Diagnostics cannot activate the workspace; correct the selected transport. |
| `workspace_stale` | Checkout changed or is busy; stop competing pull/publish work. |
| `workspace_conflict` | Draft base differs from Git; pull, resolve the diff, validate, republish. |
| `workspace_stale` | Checkout changed or is busy; stop competing pull or sync work. |
| `workspace_conflict` | Draft base differs from Git; pull, resolve the diff, validate, and retry after the curator pull/bootstrap flow. |
| `git_unavailable` | Remote, path, network, or lock unavailable; preserve the degraded valid snapshot. |
| `git_auth_failed` | Mounted SSH/HTTPS material rejected/unreadable; rotate or fix permissions without logging it. |
| `git_non_fast_forward` | Checkout diverged; reconcile through the registry workflow. |
docs/install/server-workspace-registry.md
+45 -22
View File
@@ -50,19 +50,24 @@ account Gitea administration, database-superuser rights, or a shell in the Git h
Create a private Gitea (or compatible Git) repository such as `platform/thoth-workspaces`. Protect
`main` according to the release policy and grant the ThothII publisher only the intended repository
scope. Commit canonical schema-v3 descriptors under `workspaces/<id>.yaml`, curated Evidence
under `workspace-content/<id>/evidence/**`, and generated public artifacts only at
`workspace-docs/<id>/README.md` and `workspace-docs/<id>/contract.env.example`; do not commit
installation bindings or secret material.
scope. Commit the curator-owned root catalog `thoth-workspaces.yaml`, workspace descriptors under
`<id>/workspace.yaml`, curated embedded Evidence under `<id>/evidence/**`, and generated public
artifacts only at `workspace-docs/<id>/README.md` and
`workspace-docs/<id>/contract.env.example`; do not commit installation bindings or secret material.
`thoth-workspaces.yaml` uses the `schema_version` value `1` and the ordered `workspaces` list of
`{id, name, description?}` entries. It is authoritative for workspace ID, name, description, and
display order. `<id>/workspace.yaml` must match that metadata exactly, and `workspace-docs`
remains the reserved top-level generated-docs directory.
For SSH, create a least-privilege deploy key, record Gitea's host key in managed known-hosts, and
use `ssh://git@git.example.invalid/platform/thoth-workspaces.git`. For HTTPS, create a scoped
machine credential in the secret manager and mount the Gitea/private CA separately. Never use a
Gitea admin credential in the application.
Bootstrap an empty remote from a temporary review clone only after its canonical v3 descriptors
and generated public artifacts have been reviewed; commit and push `main`. The running server is
not a descriptor authoring or conversion environment.
Bootstrap an empty remote from a temporary review clone only after the catalog, descriptors, and
generated public artifacts have been reviewed; commit and push `main`. The running server is not a
descriptor authoring or conversion environment.
## Curator flow for shared-registry Evidence
@@ -70,12 +75,24 @@ Follow this order; the [canonical Evidence contract](../contracts/workspace-evid
the source shapes and safety boundary.
1. Clone the one shared registry, or update the review clone with `git pull --ff-only`.
2. Add source bytes below `workspace-content/<id>/evidence`, then commit and push.
3. Validate and publish the descriptor against that base commit.
4. Inspect `workspace-docs/<id>/contract.env.example` and `workspace-docs/<id>/README.md`.
5. Provision only the selected Evidence `*_FILE` files outside Git and strictly below a root in `THT_WORKSPACE_SECRET_ROOTS`; add matching host-only `*_SOURCE` paths for the generated connector override.
6. Render or acquire the runtime config, then run `tht config check -c <path>`.
7. Stop: P2/P6 later performs preprocessing and materialization.
2. Keep `thoth-workspaces.yaml` curator-owned. It uses the `schema_version` value `1` and the ordered
`workspaces` list of `{id, name, description?}` entries; it is authoritative for workspace ID,
name, description, and display order.
3. For an existing workspace, edit `<id>/workspace.yaml` and any embedded `<id>/evidence/**`, then
commit and push.
4. For a new workspace, add the catalog slot first. If you want ThothII to bootstrap the
descriptor, leave `<id>/workspace.yaml` absent, commit and push, then pull that commit into the
installation; the slot appears as `configuration_required`.
5. The API may create `<id>/workspace.yaml` only when the catalog slot already exists and no Git
object exists at that path in the exact pulled base commit.
6. After bootstrap, existing descriptors change only through curator Git commit/push and
installation pull. The API never writes `thoth-workspaces.yaml` or `<id>/evidence/**`.
7. Inspect `workspace-docs/<id>/contract.env.example` and `workspace-docs/<id>/README.md`.
8. Provision only the selected Evidence `*_FILE` files outside Git and strictly below a root in
`THT_WORKSPACE_SECRET_ROOTS`; add matching host-only `*_SOURCE` paths for the generated
connector override.
9. Render or acquire the runtime config, then run `tht config check -c <path>`.
10. Stop: P2/P6 later performs preprocessing and materialization.
For example, separate signed-HTTP and static-S3 workspaces can use these host-only connector source
paths:
@@ -241,22 +258,28 @@ without authenticating the request is not an identity boundary.
`/health` is liveness. The authenticated Workspace Management page's registry status verifies
branch/head/degraded state and the active validated snapshot; its workspace listing verifies
application access. A server with no active snapshot is not ready for workspace sessions even if
liveness succeeds.
application access. A catalog-only slot with no descriptor reports `configuration_required` and is
not ready for sessions until either the curator commits `<id>/workspace.yaml` or the one-time
bootstrap create flow writes it. A server with no active snapshot is not ready for workspace
sessions even if liveness succeeds.
## Pull, publish, upgrade, backup, and recovery
Use the authenticated Workspace Management UI or `POST /workspace-registry/pull`. Drafts are
browser-local. Publish takes a canonical diff, validates before commit, and pushes under a registry
lock. On `workspace_conflict`, pull, resolve the reviewed field-level draft, validate/test, and
publish; never edit `repo/` inside a running volume.
Use the authenticated Workspace Management UI or `POST /workspace-registry/pull` to fetch later
curator revisions. Existing curated workspaces are read-only in the browser. Use the UI to inspect
status, validate a workspace, test it on this installation, and optionally create one bootstrap
descriptor for a pulled `configuration_required` slot. After that first descriptor exists, change
it only through curator Git commit/push and installation pull; never edit `repo/` inside a running
volume.
For upgrades, record active status/head, finish active work, use the documented `thothctl pi update
--drain` transaction when Pi/core changes, and take a stopped, filesystem-consistent backup of
`/srv/thothii/workspace-registry` plus `/srv/thothii/data` and Pi state. Exclude
`/srv/thothii/secrets` from the ordinary archive. Validate the descriptor with `thothctl update
--check-only`, deploy the compatible image through `thothctl`, verify health/status, then resume
proxy traffic.
proxy traffic. If you are upgrading an older P1 registry, apply the reviewed migration in
[`docs/migrations/p1-to-p1-1-registry-layout.md`](../migrations/p1-to-p1-1-registry-layout.md)
and upgrade ThothII only after that commit is pushed.
<!-- workspace-descriptor-contract:start -->
Schema v3 is the only accepted workspace descriptor. Schema v1 and v2 workspace descriptors are
@@ -291,11 +314,11 @@ do not delete snapshots as a rollback shortcut.
| `binding_missing` | Missing/invalid local value or readable `*_FILE`; correct mount and permissions. |
| `workspace_not_activatable` | Bindings/diagnostics cannot activate; use sanitized fields to fix selected transport. |
| `workspace_stale` | Checkout changed/locked; stop concurrent registry work, never force Git in the volume. |
| `workspace_conflict` | Draft base stale; pull, resolve, validate, republish. |
| `workspace_conflict` | Draft base stale; pull, resolve, validate, and retry after the curator pull/bootstrap flow. |
| `git_unavailable` | Storage/remote/DNS/firewall/lock failed; preserve degraded active state while repairing it. |
| `git_auth_failed` | SSH/HTTPS material rejected or unreadable; rotate/fix file without printing it. |
| `git_non_fast_forward` | Checkout diverged; reconcile through registry workflow and branch policy. |
| `git_push_rejected` | Gitea policy rejected publish; review hooks/branch protection. |
| `git_push_rejected` | Gitea policy rejected the curator push or docs sync commit; review hooks/branch protection. |
| `connector_unavailable` | DNS/TLS/auth/resource identity failed; check egress and local bindings. |
| `semantic_index_incompatible` | Collection/model/dimensions/distance differs; perform explicit index migration. |