From fef7a7614ee6bbb6ced4efc0d0991cb9833713df Mon Sep 17 00:00:00 2001 From: mptyl Date: Mon, 3 Aug 2026 20:41:57 +0200 Subject: [PATCH] docs: design git-backed workspace registry --- ...026-08-03-git-workspace-registry-design.md | 574 ++++++++++++++++++ 1 file changed, 574 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-03-git-workspace-registry-design.md diff --git a/docs/superpowers/specs/2026-08-03-git-workspace-registry-design.md b/docs/superpowers/specs/2026-08-03-git-workspace-registry-design.md new file mode 100644 index 00000000..38e05ee9 --- /dev/null +++ b/docs/superpowers/specs/2026-08-03-git-workspace-registry-design.md @@ -0,0 +1,574 @@ +# Git-backed Workspace Registry Design + +**Date:** 2026-08-03 +**Status:** Approved design +**Scope:** Portable workspace definition, CRUD UI, Git publication, local bindings, validation, and runtime revision pinning + +## 1. Purpose + +ThothII must manage workspace configuration as part of its own domain. A workspace must not depend on Chirone, Aritmolab, Supabase, or on the application that happens to provide a data warehouse, vector database, or embedding service. + +The same logical workspaces must be usable by: + +- the production ThothII server; +- a local ThothII installation running in Docker on macOS, Windows, or Linux; +- future ThothII installations connected to different data warehouses, vector stores, and model providers. + +A remote Git repository is the single source of truth. Each ThothII installation maintains a persistent local checkout, resolves installation-specific connectivity through local variables and secret files, and exchanges changes through pull and push. + +## 2. Goals + +- Provide a CRUD workspace page reachable from the right sidebar. +- Store canonical workspace definitions as versioned YAML in a generic Git repository. +- Support GitHub, Gitea, GitLab, and other standard Git servers without provider-specific APIs. +- Keep credentials and private keys outside the repository while defining their required variable names deterministically. +- Support direct database connections, REST access, and SSH-tunnelled connections. +- Treat vector collection and embedding configuration as one coherent semantic index. +- Keep user choices such as active workspace, LLM, and reasoning level local to the browser until an identity system exists. +- Validate free-form values formally, semantically, and—when local bindings exist—operationally. +- Pin every new session to an immutable workspace revision. +- Continue operating from the last valid snapshot when the Git remote is temporarily unavailable. +- Retain browser-mediated export/import as an offline fallback, not as the primary synchronization mechanism. + +## 3. Non-goals + +The first release will not provide: + +- embedded user authentication or per-person server profiles; +- automatic continuous synchronization; +- Git pull-request workflows; +- editing or storing secret values in the workspace UI; +- provider-specific GitHub or Gitea APIs; +- automatic conflict merging; +- a Supabase or SQLite source of truth; +- a user-selectable embedding model for a shared vector collection. + +## 4. Configuration layers + +ThothII separates configuration into three layers. + +### 4.1 Shared workspace definition + +The Git repository contains logical, shared configuration: + +- workspace identity and display metadata; +- DWH engine, logical database/schema, and supported access transports; +- vector-store type and collection; +- embedding provider contract, model, dimensions, and distance metric; +- LLM default and allowlist; +- language and supported workflow capabilities; +- the deterministic installation-variable contract. + +### 4.2 Installation bindings + +Each ThothII installation supplies operational values locally: + +- transport selected for each connector; +- host names, ports, base URLs, and tunnel targets; +- users and non-secret connection parameters; +- password, API-key, certificate, and private-key file paths; +- Git remote credentials, CA, and SSH known-hosts file; +- application data paths for sessions, artifacts, checkout, and snapshots. + +Installation bindings are excluded from the workspace repository. Absolute storage paths formerly represented by `roots` belong to this layer and are not shown in the ordinary workspace form. + +### 4.3 Browser preferences + +Until ThothII has a reliable user identity, these values are stored in browser-local storage: + +- active workspace; +- selected LLM provider/model; +- reasoning level; +- unfinished workspace drafts; +- visual preferences. + +These values are not included in Git or export bundles. The resolved workspace, model, and reasoning level are copied into each session manifest for reproducibility. + +## 5. Git repository contract + +The repository has this canonical layout: + +```text +thoth-workspaces.yaml +workspaces/ + .yaml +contracts/ + .env.example +docs/ + .md +``` + +`thoth-workspaces.yaml` declares the repository schema version. The YAML file is authoritative. The environment example and documentation are deterministic generated artifacts committed by the same publish operation. + +Workspace IDs must match: + +```text +^[a-z][a-z0-9-]{2,62}$ +``` + +The ID is an immutable technical identifier. Renaming the display label does not rename variables, files, or session references. Changing the ID is a migration operation outside ordinary edit mode. + +## 6. Workspace schema + +The initial canonical shape is: + +```yaml +workspace: + schema_version: 1 + id: psd-clinical + name: Policlinico San Donato + description: Clinical data warehouse workspace + language: it + +dwh: + engine: postgres + database: postgres + schema: datawarehouse + supported_transports: + - postgres_direct + - rest_api + - ssh_tunnel + +semantic_index: + vector_store: + engine: pgvector + collection: clinical_documents + dimensions: 768 + distance: cosine + supported_transports: + - pgvector_direct + - rest_api + - ssh_tunnel + embedding: + provider: ollama_compatible + model: nomic-embed-text-v2-moe + dimensions: 768 + +llm_policy: + default: zai/glm-5.2 + allowed: + - zai/glm-5.2 + - openai/gpt-5 +``` + +The exact machine schema is maintained by `WorkspaceSchema` and versioned with explicit migrations. Unknown keys are rejected by default so misspellings do not silently change runtime behavior. + +### 6.1 Semantic-index invariant + +`semantic_index` is atomic. The vector collection, vector dimensions, distance metric, embedding provider, embedding model, and embedding dimensions describe one index contract. + +The following are validation errors: + +- vector and embedding dimensions differ; +- the selected collection reports different dimensions or distance metric; +- the embedding endpoint does not expose the declared model; +- read and write bindings resolve to incompatible vector collections; +- indexing and retrieval resolve to different embedding contracts. + +Changing collection, embedding model, dimensions, or metric is presented as replacing or migrating the semantic index, not as an individual user preference. + +## 7. Deterministic installation-variable naming + +The environment namespace is derived from the immutable workspace ID: + +```text +psd-clinical -> PSD_CLINICAL +``` + +Every variable starts with `THT_WS__`. Connector roles and suffixes are defined by ThothII and cannot be invented in the form. + +### 7.1 DWH variables + +```dotenv +THT_WS_PSD_CLINICAL_DWH_TRANSPORT= +THT_WS_PSD_CLINICAL_DWH_HOST= +THT_WS_PSD_CLINICAL_DWH_PORT= +THT_WS_PSD_CLINICAL_DWH_BASE_URL= +THT_WS_PSD_CLINICAL_DWH_USER= +THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE= +THT_WS_PSD_CLINICAL_DWH_API_KEY_FILE= +THT_WS_PSD_CLINICAL_DWH_TLS_CA_FILE= +``` + +### 7.2 Vector-store variables + +```dotenv +THT_WS_PSD_CLINICAL_VECTOR_TRANSPORT= +THT_WS_PSD_CLINICAL_VECTOR_HOST= +THT_WS_PSD_CLINICAL_VECTOR_PORT= +THT_WS_PSD_CLINICAL_VECTOR_BASE_URL= +THT_WS_PSD_CLINICAL_VECTOR_USER= +THT_WS_PSD_CLINICAL_VECTOR_PASSWORD_FILE= +THT_WS_PSD_CLINICAL_VECTOR_API_KEY_FILE= +THT_WS_PSD_CLINICAL_VECTOR_TLS_CA_FILE= +``` + +The collection and dimensions remain in the canonical workspace because they define the shared semantic index. + +### 7.3 Embedding variables + +```dotenv +THT_WS_PSD_CLINICAL_EMBEDDING_BASE_URL= +THT_WS_PSD_CLINICAL_EMBEDDING_API_KEY_FILE= +THT_WS_PSD_CLINICAL_EMBEDDING_TLS_CA_FILE= +``` + +The embedding model and dimensions remain in the canonical workspace. + +### 7.4 SSH tunnel variables + +For any connector role `` that selects `ssh_tunnel`, ThothII requires: + +```dotenv +THT_WS_PSD_CLINICAL__SSH_HOST= +THT_WS_PSD_CLINICAL__SSH_PORT= +THT_WS_PSD_CLINICAL__SSH_USER= +THT_WS_PSD_CLINICAL__SSH_PRIVATE_KEY_FILE= +THT_WS_PSD_CLINICAL__SSH_KNOWN_HOSTS_FILE= +THT_WS_PSD_CLINICAL__SSH_TARGET_HOST= +THT_WS_PSD_CLINICAL__SSH_TARGET_PORT= +``` + +Secret values use `*_FILE` variables. The application reads the file at runtime and never serializes its content into API responses, logs, Git commits, diagnostics, or export bundles. + +The generated `.env.example`, generated workspace documentation, UI installation-requirements panel, and runtime validator are all derived from the same binding schema. + +## 8. Supported transports + +Transport behavior is encapsulated behind connector adapters. + +### 8.1 Direct + +Direct adapters connect to the configured host and port with the native protocol. PostgreSQL direct access supports TLS modes and CA files. Vector direct access uses the native vector-store protocol or database driver. + +### 8.2 REST API + +REST adapters use a base URL, an optional API-key file, TLS validation, and a documented capabilities endpoint. A REST adapter must expose enough metadata to validate schema or collection identity and semantic-index compatibility. + +### 8.3 SSH tunnel + +SSH adapters verify the remote host against an explicit known-hosts file, open a temporary local tunnel, and pass the resulting endpoint to the corresponding direct adapter. Host-key checking cannot be disabled by the form. + +Transport selection is installation-specific because a production server may connect directly while a laptop reaches the same logical resource through REST or SSH. + +## 9. Backend architecture + +### 9.1 `WorkspaceSchema` + +- Parses canonical YAML. +- Rejects unknown or malformed fields. +- Applies explicit schema migrations. +- Produces canonical serialization. +- Generates binding requirements and documentation. + +### 9.2 `GitWorkspaceRepository` + +- Owns the persistent checkout. +- Reports remote, branch, current commit, dirty state, and divergence. +- Performs fetch, fast-forward pull, diff, commit, and push using argument-safe process execution. +- Uses installation-mounted Git credentials and trust configuration. +- Never accepts repository paths or shell fragments from API requests. + +### 9.3 `WorkspaceRegistry` + +- Lists and reads workspaces from a validated repository revision. +- Creates, updates, duplicates, and deletes workspace documents. +- Enforces workspace IDs and revision preconditions. +- Coordinates publish under a repository lock. +- Materializes immutable validated snapshots. + +### 9.4 `BindingResolver` + +- Generates deterministic environment names. +- Determines required and conditional variables from the selected transports. +- Reads normal variables and secret files. +- Returns sanitized missing/invalid diagnostics without values. + +### 9.5 `WorkspaceDiagnostics` + +- Runs connector-specific operational checks. +- Verifies the semantic-index invariant against live capabilities. +- Separates errors from warnings and local non-activatability. +- Uses read-only probes by default. + +A vector write probe is an explicit action. It writes a uniquely named temporary record in a diagnostic namespace or transaction and removes it before returning. It is not part of ordinary save or publish. + +## 10. Persistent server and local layout + +Both production and local Docker deployments use: + +```text +/data/workspace-registry/ + repo/ # persistent Git checkout + snapshots/ # immutable validated revisions + state/ # active revision and repository metadata + locks/ # short-lived publish locks +``` + +The application image remains read-only. Git credentials, CA files, SSH keys, and known-hosts files are mounted under `/run/secrets` or another installation-controlled secret root. + +On startup: + +1. Clone the configured remote if no checkout exists. +2. Otherwise load the checkout and attempt fetch/pull. +3. Validate the complete candidate repository revision. +4. Atomically activate the new snapshot only if all workspace files and generated contracts are valid. +5. If the remote is unavailable or the candidate is invalid, retain the last valid snapshot and report degraded registry status. + +Database, vector, and embedding servers do not run the workspace manager. Only a ThothII installation needs outbound Git access. Gitea may be colocated with the production ThothII host. + +## 11. Git workflow and concurrency + +### 11.1 Browser drafts + +Drafts remain in browser-local storage and contain: + +- remote fingerprint; +- branch; +- workspace ID; +- base commit; +- base workspace blob checksum; +- form data and update timestamp. + +Drafts do not modify the shared checkout. + +### 11.2 Publish + +Publish is explicit and displays the canonical field-level diff. The backend then: + +1. Acquires the repository publish lock. +2. Fetches the remote branch. +3. Compares the submitted base commit and workspace checksum with the remote. +4. If only other workspaces changed, reapplies the draft on the new remote head. +5. If the same workspace changed, returns HTTP 409 with base, local, and remote field differences. +6. Validates the complete resulting repository. +7. Writes YAML and generated files atomically. +8. Creates a commit with the configured technical identity. +9. Pushes the configured branch. +10. Materializes and activates the validated snapshot. + +If a concurrent push wins after step 3, the backend fetches once more. It retries only when the target workspace is unchanged; otherwise it returns a conflict. + +Without embedded authentication, commits use a technical author such as `ThothII Workspace Manager` and include an installation-ID trailer. They do not claim a human identity. + +### 11.3 Pull + +Pull fetches the remote, requires fast-forward history, validates the complete candidate revision, and activates it atomically. Browser drafts whose base revision becomes stale remain available but are visibly marked as requiring reconciliation. + +### 11.4 Delete + +Delete creates a draft deletion and is published as a Git commit. A workspace referenced by an active runtime cannot be deleted. Historical session snapshots remain available, and Git history provides repository-level recovery. + +## 12. CRUD user experience + +The right sidebar exposes `Workspace management`, opening a dedicated page with a workspace list and editable detail area. + +The form sections are: + +1. General. +2. DWH. +3. Semantic index. +4. LLM policy. +5. Installation requirements. +6. Git status and history. + +Actions are: + +- New; +- Duplicate; +- Delete; +- Save draft; +- Discard changes; +- Pull; +- Publish; +- Export; +- Import; +- Test on this installation. + +Closed choices are used whenever the domain is enumerable: + +- database engine; +- transport type; +- vector-store engine; +- embedding provider; +- distance metric; +- TLS mode; +- language; +- LLM provider/model returned by Pi; +- embedding model returned by a reachable provider. + +Free text or numeric controls are used for names, descriptions, IDs, database/schema/collection identifiers, ports, dimensions, timeouts, and URLs. They display field-level constraints before submission and server validation errors after submission. + +The installation-requirements section shows the exact required, optional, and transport-conditional variable names. It never displays resolved secret values. + +## 13. Validation model + +### 13.1 Formal validation + +- YAML and schema version are valid. +- Required fields are present. +- Unknown fields are rejected. +- IDs and database identifiers match their allowed syntax. +- Ports are integers from 1 through 65535. +- Dimensions and timeouts are positive and within configured safety limits. +- URLs use supported schemes. +- enum values come from the closed schema lists. + +### 13.2 Static semantic validation + +- Selected transports are supported by the corresponding connector. +- Required fields for each transport can be derived unambiguously. +- Embedding and vector dimensions match. +- LLM default belongs to the allowlist. +- Duplicate workspace IDs and generated environment namespaces are rejected. +- Generated documentation exactly matches the binding contract. + +Save draft may retain incomplete local form state in the browser. Publish requires formal and static semantic validation to pass. + +### 13.3 Local operational validation + +- Required variables exist. +- Secret files are regular, readable files within approved secret roots. +- DNS, TCP, TLS, and authentication succeed. +- DWH database and schema exist and are readable. +- REST capabilities match the declared logical resource. +- SSH host verification and tunnel opening succeed. +- Vector collection, dimensions, metric, and read capability match. +- Embedding endpoint exposes the declared model and returns the expected dimensions for a controlled probe. + +A portable workspace can be valid but not activatable on a particular installation. Publish is allowed in that state; starting a new session on that installation is not. + +## 14. API contract + +```text +GET /workspace-registry/status +POST /workspace-registry/pull +GET /workspaces +GET /workspaces/:id +POST /workspaces/validate +POST /workspaces/:id/test +POST /workspaces/publish +GET /workspaces/:id/export +POST /workspaces/import +``` + +The status response contains sanitized remote identity, branch, active commit, divergence, last successful sync, last validation result, and degraded status. + +Validation accepts a structured workspace draft rather than arbitrary YAML text. Publish accepts create, update, or delete intent plus base revision metadata. No endpoint accepts a filesystem path. + +Operational errors use stable codes that distinguish: + +- invalid configuration; +- missing local binding; +- non-activatable workspace; +- stale revision; +- field conflict; +- Git remote unavailable; +- Git authentication failure; +- non-fast-forward history; +- push rejection; +- connector unavailable; +- semantic-index incompatibility. + +## 15. Offline export/import fallback + +Export returns `.thoth-workspace.zip` containing: + +```text +manifest.json +workspace.yaml +contract.env.example +README.md +``` + +The manifest contains bundle schema version, workspace ID, source commit, file checksums, and creation timestamp. It contains no secrets or browser preferences. + +Import uploads the bundle to the currently open ThothII installation. The backend validates archive size, entry count, entry names, checksums, schema, and semantics. A successful import returns a browser draft; it does not write or publish directly. + +The browser can therefore download from one ThothII installation and upload to another without direct server-to-server access. Git remains the authoritative synchronization mechanism. + +## 16. Session integration + +New-session creation sends the browser-selected workspace ID, LLM provider/model, and reasoning level. The backend: + +1. Resolves the active validated workspace snapshot. +2. Verifies local activatability. +3. Validates the LLM choice against workspace policy and Pi availability. +4. Starts the harness with the immutable snapshot path. +5. Persists workspace ID, workspace revision, provider, model, and reasoning level in the session manifest. + +Resume uses the persisted snapshot revision even after later pull or publish operations. Snapshot retention cannot remove revisions referenced by resumable sessions. + +Legacy sessions without workspace revision use the existing compatibility resolution and receive a visible legacy warning. New sessions always require a revision. + +## 17. Security constraints + +- No secret value appears in Git, generated documentation, API payloads, logs, diagnostics, browser storage, or export bundles. +- Secret references use approved `*_FILE` variables and approved secret roots. +- Workspace and archive names cannot influence filesystem paths. +- Import prevents zip-slip, symlinks, excessive file count, and excessive expanded size. +- Git commands receive fixed argument arrays; user input is never passed through a shell. +- Git SSH uses explicit known-hosts verification. +- REST and direct TLS validation cannot be disabled silently. +- Diagnostics sanitize provider errors before returning them to the browser. +- Production CORS remains same-origin; absence of embedded authentication does not imply cross-origin write access. +- The first release allows every user who can access the ThothII application to publish workspace changes. This limitation is documented until an authorization layer is introduced. + +## 18. Migration + +The migration path is: + +1. Introduce the versioned canonical schema and parser. +2. Convert existing `harness/workspaces` and deployment descriptors into repository fixtures. +3. Generate deterministic environment contracts and compare them with current Compose variables. +4. Configure the persistent registry volume and Git remote. +5. Import the current PSD workspace as the first canonical revision. +6. Keep legacy reads available during a bounded compatibility period. +7. Switch new sessions to validated snapshots and revision pinning. +8. Remove regex-based workspace metadata parsing after all active configurations use schema version 1. + +Migration never copies secret values into Git. Existing absolute roots become installation-level storage configuration. + +## 19. Documentation deliverables + +- Workspace schema reference with field descriptions and examples. +- Generated documentation for every workspace. +- Generated `.env.example` for every workspace. +- Docker Compose examples for server and local installations. +- Direct PostgreSQL, REST, and SSH-tunnel examples. +- Vector/embedding compatibility explanation. +- Git remote setup for HTTPS and SSH. +- Gitea deployment example. +- Pull, draft, publish, conflict, export, and import operator guide. +- Diagnostic command and error-code reference. +- Migration guide from current PSD configuration. + +## 20. Testing strategy + +- Unit tests for schema parsing, canonical serialization, migrations, and unknown-key rejection. +- Unit tests for deterministic environment naming and conditional binding requirements. +- Valid and invalid fixtures for direct, REST, and SSH transports. +- Semantic-index fixtures covering model, dimensions, metric, collection, and read/write mismatch. +- Temporary local Git remotes for clone, pull, publish, retry, divergence, and same-file conflicts. +- Atomic-write and lock tests. +- API tests for CRUD, stale revisions, stable error codes, and sanitized responses. +- Import tests for checksum failure, zip-slip, symlinks, archive limits, and malformed schemas. +- Frontend tests for closed choices, free-field errors, drafts, diffs, conflicts, and installation requirements. +- End-to-end tests using a local Git remote and simulated connectors. +- Deployment tests proving persistent checkout and last-valid-snapshot fallback across container replacement. +- Session tests proving revision pinning and resume after a newer workspace publish. + +## 21. Acceptance criteria + +The feature is complete when: + +- a workspace can be created, edited, duplicated, deleted, pulled, and published from the UI; +- two browsers cannot silently overwrite the same workspace revision; +- server and local Docker installations can consume the same Git repository; +- each installation can bind the same logical workspace through different transports; +- generated variable names and documentation are deterministic and tested; +- no secret value enters Git or an export bundle; +- vector collection and embedding compatibility is enforced; +- a remote outage leaves the last valid snapshot usable; +- every new session records and resumes with an immutable workspace revision; +- existing PSD configuration can be migrated without embedding PSD-specific behavior in the core schema.