docs: design git-backed workspace registry

2026-08-03 20:41:57 +02:00
parent 18233d59af
commit fef7a7614e
1 changed files with 574 additions and 0 deletions
@@ -0,0 +1,574 @@
+# Git-backed Workspace Registry Design
+
+**Date:** 2026-08-03  
+**Status:** Approved design  
+**Scope:** Portable workspace definition, CRUD UI, Git publication, local bindings, validation, and runtime revision pinning
+
+## 1. Purpose
+
+ThothII must manage workspace configuration as part of its own domain. A workspace must not depend on Chirone, Aritmolab, Supabase, or on the application that happens to provide a data warehouse, vector database, or embedding service.
+
+The same logical workspaces must be usable by:
+
+- the production ThothII server;
+- a local ThothII installation running in Docker on macOS, Windows, or Linux;
+- future ThothII installations connected to different data warehouses, vector stores, and model providers.
+
+A remote Git repository is the single source of truth. Each ThothII installation maintains a persistent local checkout, resolves installation-specific connectivity through local variables and secret files, and exchanges changes through pull and push.
+
+## 2. Goals
+
+- Provide a CRUD workspace page reachable from the right sidebar.
+- Store canonical workspace definitions as versioned YAML in a generic Git repository.
+- Support GitHub, Gitea, GitLab, and other standard Git servers without provider-specific APIs.
+- Keep credentials and private keys outside the repository while defining their required variable names deterministically.
+- Support direct database connections, REST access, and SSH-tunnelled connections.
+- Treat vector collection and embedding configuration as one coherent semantic index.
+- Keep user choices such as active workspace, LLM, and reasoning level local to the browser until an identity system exists.
+- Validate free-form values formally, semantically, and—when local bindings exist—operationally.
+- Pin every new session to an immutable workspace revision.
+- Continue operating from the last valid snapshot when the Git remote is temporarily unavailable.
+- Retain browser-mediated export/import as an offline fallback, not as the primary synchronization mechanism.
+
+## 3. Non-goals
+
+The first release will not provide:
+
+- embedded user authentication or per-person server profiles;
+- automatic continuous synchronization;
+- Git pull-request workflows;
+- editing or storing secret values in the workspace UI;
+- provider-specific GitHub or Gitea APIs;
+- automatic conflict merging;
+- a Supabase or SQLite source of truth;
+- a user-selectable embedding model for a shared vector collection.
+
+## 4. Configuration layers
+
+ThothII separates configuration into three layers.
+
+### 4.1 Shared workspace definition
+
+The Git repository contains logical, shared configuration:
+
+- workspace identity and display metadata;
+- DWH engine, logical database/schema, and supported access transports;
+- vector-store type and collection;
+- embedding provider contract, model, dimensions, and distance metric;
+- LLM default and allowlist;
+- language and supported workflow capabilities;
+- the deterministic installation-variable contract.
+
+### 4.2 Installation bindings
+
+Each ThothII installation supplies operational values locally:
+
+- transport selected for each connector;
+- host names, ports, base URLs, and tunnel targets;
+- users and non-secret connection parameters;
+- password, API-key, certificate, and private-key file paths;
+- Git remote credentials, CA, and SSH known-hosts file;
+- application data paths for sessions, artifacts, checkout, and snapshots.
+
+Installation bindings are excluded from the workspace repository. Absolute storage paths formerly represented by `roots` belong to this layer and are not shown in the ordinary workspace form.
+
+### 4.3 Browser preferences
+
+Until ThothII has a reliable user identity, these values are stored in browser-local storage:
+
+- active workspace;
+- selected LLM provider/model;
+- reasoning level;
+- unfinished workspace drafts;
+- visual preferences.
+
+These values are not included in Git or export bundles. The resolved workspace, model, and reasoning level are copied into each session manifest for reproducibility.
+
+## 5. Git repository contract
+
+The repository has this canonical layout:
+
+```text
+thoth-workspaces.yaml
+workspaces/
+  <workspace-id>.yaml
+contracts/
+  <workspace-id>.env.example
+docs/
+  <workspace-id>.md
+```
+
+`thoth-workspaces.yaml` declares the repository schema version. The YAML file is authoritative. The environment example and documentation are deterministic generated artifacts committed by the same publish operation.
+
+Workspace IDs must match:
+
+```text
+^[a-z][a-z0-9-]{2,62}$
+```
+
+The ID is an immutable technical identifier. Renaming the display label does not rename variables, files, or session references. Changing the ID is a migration operation outside ordinary edit mode.
+
+## 6. Workspace schema
+
+The initial canonical shape is:
+
+```yaml
+workspace:
+  schema_version: 1
+  id: psd-clinical
+  name: Policlinico San Donato
+  description: Clinical data warehouse workspace
+  language: it
+
+dwh:
+  engine: postgres
+  database: postgres
+  schema: datawarehouse
+  supported_transports:
+    - postgres_direct
+    - rest_api
+    - ssh_tunnel
+
+semantic_index:
+  vector_store:
+    engine: pgvector
+    collection: clinical_documents
+    dimensions: 768
+    distance: cosine
+    supported_transports:
+      - pgvector_direct
+      - rest_api
+      - ssh_tunnel
+  embedding:
+    provider: ollama_compatible
+    model: nomic-embed-text-v2-moe
+    dimensions: 768
+
+llm_policy:
+  default: zai/glm-5.2
+  allowed:
+    - zai/glm-5.2
+    - openai/gpt-5
+```
+
+The exact machine schema is maintained by `WorkspaceSchema` and versioned with explicit migrations. Unknown keys are rejected by default so misspellings do not silently change runtime behavior.
+
+### 6.1 Semantic-index invariant
+
+`semantic_index` is atomic. The vector collection, vector dimensions, distance metric, embedding provider, embedding model, and embedding dimensions describe one index contract.
+
+The following are validation errors:
+
+- vector and embedding dimensions differ;
+- the selected collection reports different dimensions or distance metric;
+- the embedding endpoint does not expose the declared model;
+- read and write bindings resolve to incompatible vector collections;
+- indexing and retrieval resolve to different embedding contracts.
+
+Changing collection, embedding model, dimensions, or metric is presented as replacing or migrating the semantic index, not as an individual user preference.
+
+## 7. Deterministic installation-variable naming
+
+The environment namespace is derived from the immutable workspace ID:
+
+```text
+psd-clinical -> PSD_CLINICAL
+```
+
+Every variable starts with `THT_WS_<NAMESPACE>_`. Connector roles and suffixes are defined by ThothII and cannot be invented in the form.
+
+### 7.1 DWH variables
+
+```dotenv
+THT_WS_PSD_CLINICAL_DWH_TRANSPORT=
+THT_WS_PSD_CLINICAL_DWH_HOST=
+THT_WS_PSD_CLINICAL_DWH_PORT=
+THT_WS_PSD_CLINICAL_DWH_BASE_URL=
+THT_WS_PSD_CLINICAL_DWH_USER=
+THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE=
+THT_WS_PSD_CLINICAL_DWH_API_KEY_FILE=
+THT_WS_PSD_CLINICAL_DWH_TLS_CA_FILE=
+```
+
+### 7.2 Vector-store variables
+
+```dotenv
+THT_WS_PSD_CLINICAL_VECTOR_TRANSPORT=
+THT_WS_PSD_CLINICAL_VECTOR_HOST=
+THT_WS_PSD_CLINICAL_VECTOR_PORT=
+THT_WS_PSD_CLINICAL_VECTOR_BASE_URL=
+THT_WS_PSD_CLINICAL_VECTOR_USER=
+THT_WS_PSD_CLINICAL_VECTOR_PASSWORD_FILE=
+THT_WS_PSD_CLINICAL_VECTOR_API_KEY_FILE=
+THT_WS_PSD_CLINICAL_VECTOR_TLS_CA_FILE=
+```
+
+The collection and dimensions remain in the canonical workspace because they define the shared semantic index.
+
+### 7.3 Embedding variables
+
+```dotenv
+THT_WS_PSD_CLINICAL_EMBEDDING_BASE_URL=
+THT_WS_PSD_CLINICAL_EMBEDDING_API_KEY_FILE=
+THT_WS_PSD_CLINICAL_EMBEDDING_TLS_CA_FILE=
+```
+
+The embedding model and dimensions remain in the canonical workspace.
+
+### 7.4 SSH tunnel variables
+
+For any connector role `<ROLE>` that selects `ssh_tunnel`, ThothII requires:
+
+```dotenv
+THT_WS_PSD_CLINICAL_<ROLE>_SSH_HOST=
+THT_WS_PSD_CLINICAL_<ROLE>_SSH_PORT=
+THT_WS_PSD_CLINICAL_<ROLE>_SSH_USER=
+THT_WS_PSD_CLINICAL_<ROLE>_SSH_PRIVATE_KEY_FILE=
+THT_WS_PSD_CLINICAL_<ROLE>_SSH_KNOWN_HOSTS_FILE=
+THT_WS_PSD_CLINICAL_<ROLE>_SSH_TARGET_HOST=
+THT_WS_PSD_CLINICAL_<ROLE>_SSH_TARGET_PORT=
+```
+
+Secret values use `*_FILE` variables. The application reads the file at runtime and never serializes its content into API responses, logs, Git commits, diagnostics, or export bundles.
+
+The generated `.env.example`, generated workspace documentation, UI installation-requirements panel, and runtime validator are all derived from the same binding schema.
+
+## 8. Supported transports
+
+Transport behavior is encapsulated behind connector adapters.
+
+### 8.1 Direct
+
+Direct adapters connect to the configured host and port with the native protocol. PostgreSQL direct access supports TLS modes and CA files. Vector direct access uses the native vector-store protocol or database driver.
+
+### 8.2 REST API
+
+REST adapters use a base URL, an optional API-key file, TLS validation, and a documented capabilities endpoint. A REST adapter must expose enough metadata to validate schema or collection identity and semantic-index compatibility.
+
+### 8.3 SSH tunnel
+
+SSH adapters verify the remote host against an explicit known-hosts file, open a temporary local tunnel, and pass the resulting endpoint to the corresponding direct adapter. Host-key checking cannot be disabled by the form.
+
+Transport selection is installation-specific because a production server may connect directly while a laptop reaches the same logical resource through REST or SSH.
+
+## 9. Backend architecture
+
+### 9.1 `WorkspaceSchema`
+
+- Parses canonical YAML.
+- Rejects unknown or malformed fields.
+- Applies explicit schema migrations.
+- Produces canonical serialization.
+- Generates binding requirements and documentation.
+
+### 9.2 `GitWorkspaceRepository`
+
+- Owns the persistent checkout.
+- Reports remote, branch, current commit, dirty state, and divergence.
+- Performs fetch, fast-forward pull, diff, commit, and push using argument-safe process execution.
+- Uses installation-mounted Git credentials and trust configuration.
+- Never accepts repository paths or shell fragments from API requests.
+
+### 9.3 `WorkspaceRegistry`
+
+- Lists and reads workspaces from a validated repository revision.
+- Creates, updates, duplicates, and deletes workspace documents.
+- Enforces workspace IDs and revision preconditions.
+- Coordinates publish under a repository lock.
+- Materializes immutable validated snapshots.
+
+### 9.4 `BindingResolver`
+
+- Generates deterministic environment names.
+- Determines required and conditional variables from the selected transports.
+- Reads normal variables and secret files.
+- Returns sanitized missing/invalid diagnostics without values.
+
+### 9.5 `WorkspaceDiagnostics`
+
+- Runs connector-specific operational checks.
+- Verifies the semantic-index invariant against live capabilities.
+- Separates errors from warnings and local non-activatability.
+- Uses read-only probes by default.
+
+A vector write probe is an explicit action. It writes a uniquely named temporary record in a diagnostic namespace or transaction and removes it before returning. It is not part of ordinary save or publish.
+
+## 10. Persistent server and local layout
+
+Both production and local Docker deployments use:
+
+```text
+/data/workspace-registry/
+  repo/            # persistent Git checkout
+  snapshots/       # immutable validated revisions
+  state/           # active revision and repository metadata
+  locks/           # short-lived publish locks
+```
+
+The application image remains read-only. Git credentials, CA files, SSH keys, and known-hosts files are mounted under `/run/secrets` or another installation-controlled secret root.
+
+On startup:
+
+1. Clone the configured remote if no checkout exists.
+2. Otherwise load the checkout and attempt fetch/pull.
+3. Validate the complete candidate repository revision.
+4. Atomically activate the new snapshot only if all workspace files and generated contracts are valid.
+5. If the remote is unavailable or the candidate is invalid, retain the last valid snapshot and report degraded registry status.
+
+Database, vector, and embedding servers do not run the workspace manager. Only a ThothII installation needs outbound Git access. Gitea may be colocated with the production ThothII host.
+
+## 11. Git workflow and concurrency
+
+### 11.1 Browser drafts
+
+Drafts remain in browser-local storage and contain:
+
+- remote fingerprint;
+- branch;
+- workspace ID;
+- base commit;
+- base workspace blob checksum;
+- form data and update timestamp.
+
+Drafts do not modify the shared checkout.
+
+### 11.2 Publish
+
+Publish is explicit and displays the canonical field-level diff. The backend then:
+
+1. Acquires the repository publish lock.
+2. Fetches the remote branch.
+3. Compares the submitted base commit and workspace checksum with the remote.
+4. If only other workspaces changed, reapplies the draft on the new remote head.
+5. If the same workspace changed, returns HTTP 409 with base, local, and remote field differences.
+6. Validates the complete resulting repository.
+7. Writes YAML and generated files atomically.
+8. Creates a commit with the configured technical identity.
+9. Pushes the configured branch.
+10. Materializes and activates the validated snapshot.
+
+If a concurrent push wins after step 3, the backend fetches once more. It retries only when the target workspace is unchanged; otherwise it returns a conflict.
+
+Without embedded authentication, commits use a technical author such as `ThothII Workspace Manager` and include an installation-ID trailer. They do not claim a human identity.
+
+### 11.3 Pull
+
+Pull fetches the remote, requires fast-forward history, validates the complete candidate revision, and activates it atomically. Browser drafts whose base revision becomes stale remain available but are visibly marked as requiring reconciliation.
+
+### 11.4 Delete
+
+Delete creates a draft deletion and is published as a Git commit. A workspace referenced by an active runtime cannot be deleted. Historical session snapshots remain available, and Git history provides repository-level recovery.
+
+## 12. CRUD user experience
+
+The right sidebar exposes `Workspace management`, opening a dedicated page with a workspace list and editable detail area.
+
+The form sections are:
+
+1. General.
+2. DWH.
+3. Semantic index.
+4. LLM policy.
+5. Installation requirements.
+6. Git status and history.
+
+Actions are:
+
+- New;
+- Duplicate;
+- Delete;
+- Save draft;
+- Discard changes;
+- Pull;
+- Publish;
+- Export;
+- Import;
+- Test on this installation.
+
+Closed choices are used whenever the domain is enumerable:
+
+- database engine;
+- transport type;
+- vector-store engine;
+- embedding provider;
+- distance metric;
+- TLS mode;
+- language;
+- LLM provider/model returned by Pi;
+- embedding model returned by a reachable provider.
+
+Free text or numeric controls are used for names, descriptions, IDs, database/schema/collection identifiers, ports, dimensions, timeouts, and URLs. They display field-level constraints before submission and server validation errors after submission.
+
+The installation-requirements section shows the exact required, optional, and transport-conditional variable names. It never displays resolved secret values.
+
+## 13. Validation model
+
+### 13.1 Formal validation
+
+- YAML and schema version are valid.
+- Required fields are present.
+- Unknown fields are rejected.
+- IDs and database identifiers match their allowed syntax.
+- Ports are integers from 1 through 65535.
+- Dimensions and timeouts are positive and within configured safety limits.
+- URLs use supported schemes.
+- enum values come from the closed schema lists.
+
+### 13.2 Static semantic validation
+
+- Selected transports are supported by the corresponding connector.
+- Required fields for each transport can be derived unambiguously.
+- Embedding and vector dimensions match.
+- LLM default belongs to the allowlist.
+- Duplicate workspace IDs and generated environment namespaces are rejected.
+- Generated documentation exactly matches the binding contract.
+
+Save draft may retain incomplete local form state in the browser. Publish requires formal and static semantic validation to pass.
+
+### 13.3 Local operational validation
+
+- Required variables exist.
+- Secret files are regular, readable files within approved secret roots.
+- DNS, TCP, TLS, and authentication succeed.
+- DWH database and schema exist and are readable.
+- REST capabilities match the declared logical resource.
+- SSH host verification and tunnel opening succeed.
+- Vector collection, dimensions, metric, and read capability match.
+- Embedding endpoint exposes the declared model and returns the expected dimensions for a controlled probe.
+
+A portable workspace can be valid but not activatable on a particular installation. Publish is allowed in that state; starting a new session on that installation is not.
+
+## 14. API contract
+
+```text
+GET    /workspace-registry/status
+POST   /workspace-registry/pull
+GET    /workspaces
+GET    /workspaces/:id
+POST   /workspaces/validate
+POST   /workspaces/:id/test
+POST   /workspaces/publish
+GET    /workspaces/:id/export
+POST   /workspaces/import
+```
+
+The status response contains sanitized remote identity, branch, active commit, divergence, last successful sync, last validation result, and degraded status.
+
+Validation accepts a structured workspace draft rather than arbitrary YAML text. Publish accepts create, update, or delete intent plus base revision metadata. No endpoint accepts a filesystem path.
+
+Operational errors use stable codes that distinguish:
+
+- invalid configuration;
+- missing local binding;
+- non-activatable workspace;
+- stale revision;
+- field conflict;
+- Git remote unavailable;
+- Git authentication failure;
+- non-fast-forward history;
+- push rejection;
+- connector unavailable;
+- semantic-index incompatibility.
+
+## 15. Offline export/import fallback
+
+Export returns `<workspace-id>.thoth-workspace.zip` containing:
+
+```text
+manifest.json
+workspace.yaml
+contract.env.example
+README.md
+```
+
+The manifest contains bundle schema version, workspace ID, source commit, file checksums, and creation timestamp. It contains no secrets or browser preferences.
+
+Import uploads the bundle to the currently open ThothII installation. The backend validates archive size, entry count, entry names, checksums, schema, and semantics. A successful import returns a browser draft; it does not write or publish directly.
+
+The browser can therefore download from one ThothII installation and upload to another without direct server-to-server access. Git remains the authoritative synchronization mechanism.
+
+## 16. Session integration
+
+New-session creation sends the browser-selected workspace ID, LLM provider/model, and reasoning level. The backend:
+
+1. Resolves the active validated workspace snapshot.
+2. Verifies local activatability.
+3. Validates the LLM choice against workspace policy and Pi availability.
+4. Starts the harness with the immutable snapshot path.
+5. Persists workspace ID, workspace revision, provider, model, and reasoning level in the session manifest.
+
+Resume uses the persisted snapshot revision even after later pull or publish operations. Snapshot retention cannot remove revisions referenced by resumable sessions.
+
+Legacy sessions without workspace revision use the existing compatibility resolution and receive a visible legacy warning. New sessions always require a revision.
+
+## 17. Security constraints
+
+- No secret value appears in Git, generated documentation, API payloads, logs, diagnostics, browser storage, or export bundles.
+- Secret references use approved `*_FILE` variables and approved secret roots.
+- Workspace and archive names cannot influence filesystem paths.
+- Import prevents zip-slip, symlinks, excessive file count, and excessive expanded size.
+- Git commands receive fixed argument arrays; user input is never passed through a shell.
+- Git SSH uses explicit known-hosts verification.
+- REST and direct TLS validation cannot be disabled silently.
+- Diagnostics sanitize provider errors before returning them to the browser.
+- Production CORS remains same-origin; absence of embedded authentication does not imply cross-origin write access.
+- The first release allows every user who can access the ThothII application to publish workspace changes. This limitation is documented until an authorization layer is introduced.
+
+## 18. Migration
+
+The migration path is:
+
+1. Introduce the versioned canonical schema and parser.
+2. Convert existing `harness/workspaces` and deployment descriptors into repository fixtures.
+3. Generate deterministic environment contracts and compare them with current Compose variables.
+4. Configure the persistent registry volume and Git remote.
+5. Import the current PSD workspace as the first canonical revision.
+6. Keep legacy reads available during a bounded compatibility period.
+7. Switch new sessions to validated snapshots and revision pinning.
+8. Remove regex-based workspace metadata parsing after all active configurations use schema version 1.
+
+Migration never copies secret values into Git. Existing absolute roots become installation-level storage configuration.
+
+## 19. Documentation deliverables
+
+- Workspace schema reference with field descriptions and examples.
+- Generated documentation for every workspace.
+- Generated `.env.example` for every workspace.
+- Docker Compose examples for server and local installations.
+- Direct PostgreSQL, REST, and SSH-tunnel examples.
+- Vector/embedding compatibility explanation.
+- Git remote setup for HTTPS and SSH.
+- Gitea deployment example.
+- Pull, draft, publish, conflict, export, and import operator guide.
+- Diagnostic command and error-code reference.
+- Migration guide from current PSD configuration.
+
+## 20. Testing strategy
+
+- Unit tests for schema parsing, canonical serialization, migrations, and unknown-key rejection.
+- Unit tests for deterministic environment naming and conditional binding requirements.
+- Valid and invalid fixtures for direct, REST, and SSH transports.
+- Semantic-index fixtures covering model, dimensions, metric, collection, and read/write mismatch.
+- Temporary local Git remotes for clone, pull, publish, retry, divergence, and same-file conflicts.
+- Atomic-write and lock tests.
+- API tests for CRUD, stale revisions, stable error codes, and sanitized responses.
+- Import tests for checksum failure, zip-slip, symlinks, archive limits, and malformed schemas.
+- Frontend tests for closed choices, free-field errors, drafts, diffs, conflicts, and installation requirements.
+- End-to-end tests using a local Git remote and simulated connectors.
+- Deployment tests proving persistent checkout and last-valid-snapshot fallback across container replacement.
+- Session tests proving revision pinning and resume after a newer workspace publish.
+
+## 21. Acceptance criteria
+
+The feature is complete when:
+
+- a workspace can be created, edited, duplicated, deleted, pulled, and published from the UI;
+- two browsers cannot silently overwrite the same workspace revision;
+- server and local Docker installations can consume the same Git repository;
+- each installation can bind the same logical workspace through different transports;
+- generated variable names and documentation are deterministic and tested;
+- no secret value enters Git or an export bundle;
+- vector collection and embedding compatibility is enforced;
+- a remote outage leaves the last valid snapshot usable;
+- every new session records and resumes with an immutable workspace revision;
+- existing PSD configuration can be migrated without embedding PSD-specific behavior in the core schema.