ThothII/docs/superpowers/specs/2026-08-03-git-workspace-registry-design.md

Git-backed Workspace Registry Design

Date: 2026-08-03
Status: Approved design
Scope: Portable workspace definition, CRUD UI, Git publication, local bindings, validation, and runtime revision pinning

1. Purpose

ThothII must manage workspace configuration as part of its own domain. A workspace must not depend on Chirone, Aritmolab, Supabase, or on the application that happens to provide a data warehouse, vector database, or embedding service.

The same logical workspaces must be usable by:

• the production ThothII server;
• a local ThothII installation running in Docker on macOS, Windows, or Linux;
• future ThothII installations connected to different data warehouses, vector stores, and model providers.

A remote Git repository is the single source of truth. Each ThothII installation maintains a persistent local checkout, resolves installation-specific connectivity through local variables and secret files, and exchanges changes through pull and push.

2. Goals

• Provide a CRUD workspace page reachable from the right sidebar.
• Store canonical workspace definitions as versioned YAML in a generic Git repository.
• Support GitHub, Gitea, GitLab, and other standard Git servers without provider-specific APIs.
• Keep credentials and private keys outside the repository while defining their required variable names deterministically.
• Support direct database connections, REST access, and SSH-tunnelled connections.
• Treat vector collection and embedding configuration as one coherent semantic index.
• Keep user choices such as active workspace, LLM, and reasoning level local to the browser until an identity system exists.
• Validate free-form values formally, semantically, and—when local bindings exist—operationally.
• Pin every new session to an immutable workspace revision.
• Continue operating from the last valid snapshot when the Git remote is temporarily unavailable.
• Retain browser-mediated export/import as an offline fallback, not as the primary synchronization mechanism.

3. Non-goals

The first release will not provide:

• embedded user authentication or per-person server profiles;
• automatic continuous synchronization;
• Git pull-request workflows;
• editing or storing secret values in the workspace UI;
• provider-specific GitHub or Gitea APIs;
• automatic conflict merging;
• a Supabase or SQLite source of truth;
• a user-selectable embedding model for a shared vector collection.

4. Configuration layers

ThothII separates configuration into three layers.

4.1 Shared workspace definition

The Git repository contains logical, shared configuration:

• workspace identity and display metadata;
• DWH engine, logical database/schema, and supported access transports;
• vector-store type and collection;
• embedding provider contract, model, dimensions, and distance metric;
• LLM default and allowlist;
• language and supported workflow capabilities;
• the deterministic installation-variable contract.

4.2 Installation bindings

Each ThothII installation supplies operational values locally:

• transport selected for each connector;
• host names, ports, base URLs, and tunnel targets;
• users and non-secret connection parameters;
• password, API-key, certificate, and private-key file paths;
• Git remote credentials, CA, and SSH known-hosts file;
• application data paths for sessions, artifacts, checkout, and snapshots.

Installation bindings are excluded from the workspace repository. Absolute storage paths formerly represented by roots belong to this layer and are not shown in the ordinary workspace form.

4.3 Browser preferences

Until ThothII has a reliable user identity, these values are stored in browser-local storage:

• active workspace;
• selected LLM provider/model;
• reasoning level;
• unfinished workspace drafts;
• visual preferences.

These values are not included in Git or export bundles. The resolved workspace, model, and reasoning level are copied into each session manifest for reproducibility.

5. Git repository contract

The repository has this canonical layout:

thoth-workspaces.yaml
workspaces/
  <workspace-id>.yaml
contracts/
  <workspace-id>.env.example
docs/
  <workspace-id>.md

thoth-workspaces.yaml declares the repository schema version. The YAML file is authoritative. The environment example and documentation are deterministic generated artifacts committed by the same publish operation.

Workspace IDs must match:

^[a-z][a-z0-9-]{2,62}$

The ID is an immutable technical identifier. Renaming the display label does not rename variables, files, or session references. Changing the ID is a migration operation outside ordinary edit mode.

6. Workspace schema

The initial canonical shape is:

workspace:
  schema_version: 2
  id: psd-clinical
  name: Policlinico San Donato
  description: Clinical data warehouse workspace
  language: it

dwh:
  engine: postgres
  database: postgres
  schema: datawarehouse
  supported_transports:
    - postgres_direct
    - rest_api
    - ssh_tunnel

semantic_index:
  vector_store:
    engine: pgvector
    database: postgres
    schema: vectors
    collection: clinical_documents
    dimensions: 768
    distance: cosine
    supported_transports:
      - pgvector_direct
      - rest_api
      - ssh_tunnel
  vector_writer: {} # optional; enables a distinct, locally bound reversible diagnostic writer
  embedding:
    provider: ollama_compatible
    model: nomic-embed-text-v2-moe
    dimensions: 768

diagnostics:
  dwh_rest:
    method: POST
    path: /rpc/ping
    auth: bearer
    response: { database: database, schema: schema }
  vector_rest:
    metadata:
      method: GET
      path: /vector/metadata
      auth: bearer
      response: { collection: collection, dimensions: dimensions, distance: distance }
    reversible_probe:
      method: POST
      path: /vector/diagnostic-probe
      auth: bearer
  embedding:
    method: GET
    path: /models
    auth: none
    response: { model: model, dimensions: dimensions }

llm_policy:
  default: zai/glm-5.2
  allowed:
    - zai/glm-5.2
    - openai/gpt-5

The exact machine schema is maintained by WorkspaceSchema and versioned with explicit migrations. Unknown keys are rejected by default so misspellings do not silently change runtime behavior.

6.1 Version migration and operational state

Schema version 2 makes semantic_index.vector_store.database and .schema mandatory. They identify the vector service independently of the DWH, even when both happen to use the same PostgreSQL instance.

Version 1 descriptors remain readable and listable so operators can discover legacy Git content. They are marked migration_required and may not generate installation bindings, runtime configuration, diagnostics, or publication artifacts. Migration is an explicit UI/transformer action that supplies the vector database/schema; it must never infer either value from the DWH. The resulting descriptor is written as schema version 2 and then passes normal operational validation.

The migration also preserves least privilege: vector_writer is optional and never inferred from the reader binding. A v2 descriptor without it is valid and operates reader-only. If it is declared, its local VECTOR_WRITER_API_KEY_FILE is distinct from the reader API-key file and is used only by the explicitly requested reversible writer diagnostic.

6.2 Semantic-index invariant

semantic_index is atomic. The vector collection, vector dimensions, distance metric, embedding provider, embedding model, and embedding dimensions describe one index contract.

The following are validation errors:

• vector and embedding dimensions differ;
• the selected collection reports different dimensions or distance metric;
• the embedding endpoint does not expose the declared model;
• read and write bindings resolve to incompatible vector collections;
• indexing and retrieval resolve to different embedding contracts.

Changing collection, embedding model, dimensions, or metric is presented as replacing or migrating the semantic index, not as an individual user preference.

6.3 Declared diagnostic protocol

Diagnostics are declarative and strict. dwh_rest declares the DWH ping method, origin-relative path, authentication mode, and JSON fields that must equal the canonical DWH database/schema. vector_rest.metadata does the same for collection, dimensions, and distance. embedding declares the model/dimensions response fields. Only GET and POST, none/bearer/x-api-key authentication, origin-relative paths without a query or fragment, and identifier-shaped response field names are accepted.

vector_rest.reversible_probe, when present, is POST-only. It is called with a generated diagnostic record create request and a matching remove request, with cleanup retried in finally. An upsert-only service cannot be declared as this probe. All ordinary diagnostics remain read-only. The complete request, response, timeout, reader-only fallback, SSH, and private-CA limitations are the operator contract in Workspace diagnostic protocol.

7. Deterministic installation-variable naming

The environment namespace is derived from the immutable workspace ID:

psd-clinical -> PSD_CLINICAL

Every variable starts with THT_WS_<NAMESPACE>_. Connector roles and suffixes are defined by ThothII and cannot be invented in the form.

7.1 DWH variables

THT_WS_PSD_CLINICAL_DWH_TRANSPORT=
THT_WS_PSD_CLINICAL_DWH_HOST=
THT_WS_PSD_CLINICAL_DWH_PORT=
THT_WS_PSD_CLINICAL_DWH_BASE_URL=
THT_WS_PSD_CLINICAL_DWH_USER=
THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE=
THT_WS_PSD_CLINICAL_DWH_API_KEY_FILE=
THT_WS_PSD_CLINICAL_DWH_TLS_CA_FILE=

7.2 Vector-store variables

THT_WS_PSD_CLINICAL_VECTOR_TRANSPORT=
THT_WS_PSD_CLINICAL_VECTOR_HOST=
THT_WS_PSD_CLINICAL_VECTOR_PORT=
THT_WS_PSD_CLINICAL_VECTOR_BASE_URL=
THT_WS_PSD_CLINICAL_VECTOR_USER=
THT_WS_PSD_CLINICAL_VECTOR_PASSWORD_FILE=
THT_WS_PSD_CLINICAL_VECTOR_API_KEY_FILE=
THT_WS_PSD_CLINICAL_VECTOR_TLS_CA_FILE=

The collection and dimensions remain in the canonical workspace because they define the shared semantic index.

7.3 Embedding variables

THT_WS_PSD_CLINICAL_EMBEDDING_BASE_URL=
THT_WS_PSD_CLINICAL_EMBEDDING_API_KEY_FILE=
THT_WS_PSD_CLINICAL_EMBEDDING_TLS_CA_FILE=

The embedding model and dimensions remain in the canonical workspace.

7.4 Optional vector-writer variable

Only a descriptor declaring semantic_index.vector_writer: {} generates this local secret-file binding. It is never generated for a reader-only workspace:

THT_WS_PSD_CLINICAL_VECTOR_WRITER_API_KEY_FILE=

The generated workspace documentation and .env.example must render this exact _FILE variable when the optional writer exists. The path must be distinct from THT_WS_PSD_CLINICAL_VECTOR_API_KEY_FILE; neither file's content is rendered.

7.5 SSH tunnel variables

For any connector role <ROLE> that selects ssh_tunnel, ThothII requires:

THT_WS_PSD_CLINICAL_<ROLE>_SSH_HOST=
THT_WS_PSD_CLINICAL_<ROLE>_SSH_PORT=
THT_WS_PSD_CLINICAL_<ROLE>_SSH_USER=
THT_WS_PSD_CLINICAL_<ROLE>_SSH_PRIVATE_KEY_FILE=
THT_WS_PSD_CLINICAL_<ROLE>_SSH_KNOWN_HOSTS_FILE=
THT_WS_PSD_CLINICAL_<ROLE>_SSH_TARGET_HOST=
THT_WS_PSD_CLINICAL_<ROLE>_SSH_TARGET_PORT=

Secret values use *_FILE variables. The application reads the file at runtime and never serializes its content into API responses, logs, Git commits, diagnostics, or export bundles.

The generated .env.example, generated workspace documentation, UI installation-requirements panel, and runtime validator are all derived from the same binding schema.

8. Supported transports

Transport behavior is encapsulated behind connector adapters.

8.1 Direct

Direct adapters connect to the configured host and port with the native protocol. PostgreSQL direct access supports TLS modes and CA files. Vector direct access uses the native vector-store protocol or database driver.

8.2 REST API

REST adapters use a base URL, an optional API-key file, TLS validation, and a documented capabilities endpoint. A REST adapter must expose enough metadata to validate schema or collection identity and semantic-index compatibility.

The present diagnostic adapter cannot load a private CA from a REST *_TLS_CA_FILE binding. It therefore refuses that diagnostic rather than weakening certificate verification. Operators must use a runtime-trusted HTTPS chain, direct/SSH transport with native PostgreSQL CA handling, or a trusted TLS-termination boundary.

8.3 SSH tunnel

SSH adapters verify the remote host against an explicit known-hosts file, open a temporary local tunnel, and pass the resulting endpoint to the corresponding direct adapter. Host-key checking cannot be disabled by the form.

Transport selection is installation-specific because a production server may connect directly while a laptop reaches the same logical resource through REST or SSH.

9. Backend architecture

9.1 WorkspaceSchema

• Parses canonical YAML.
• Rejects unknown or malformed fields.
• Applies explicit schema migrations.
• Produces canonical serialization.
• Generates binding requirements and documentation.

9.2 GitWorkspaceRepository

• Owns the persistent checkout.
• Reports remote, branch, current commit, dirty state, and divergence.
• Performs fetch, fast-forward pull, diff, commit, and push using argument-safe process execution.
• Uses installation-mounted Git credentials and trust configuration.
• Never accepts repository paths or shell fragments from API requests.

9.3 WorkspaceRegistry

• Lists and reads workspaces from a validated repository revision.
• Creates, updates, duplicates, and deletes workspace documents.
• Enforces workspace IDs and revision preconditions.
• Coordinates publish under a repository lock.
• Materializes immutable validated snapshots.

9.4 BindingResolver

• Generates deterministic environment names.
• Determines required and conditional variables from the selected transports.
• Reads normal variables and secret files.
• Returns sanitized missing/invalid diagnostics without values.

9.5 WorkspaceDiagnostics

• Runs connector-specific operational checks.
• Verifies the semantic-index invariant against live capabilities.
• Separates errors from warnings and local non-activatability.
• Uses read-only probes by default.

A vector write probe is an explicit action. It writes a uniquely named temporary record in a diagnostic namespace or transaction and removes it before returning. It is not part of ordinary save or publish.

When no writer descriptor or distinct local writer file is present, the same workspace remains reader-only and the write probe is omitted; no reader credential is repurposed for writing.

10. Persistent server and local layout

Both production and local Docker deployments use:

/data/workspace-registry/
  repo/            # persistent Git checkout
  snapshots/       # immutable validated revisions
  state/           # active revision and repository metadata
  locks/           # short-lived publish locks

The application image remains read-only. Git credentials, CA files, SSH keys, and known-hosts files are mounted under /run/secrets or another installation-controlled secret root.

On startup:

1. Clone the configured remote if no checkout exists.
2. Otherwise load the checkout and attempt fetch/pull.
3. Validate the complete candidate repository revision.
4. Atomically activate the new snapshot only if all workspace files and generated contracts are valid.
5. If the remote is unavailable or the candidate is invalid, retain the last valid snapshot and report degraded registry status.

Database, vector, and embedding servers do not run the workspace manager. Only a ThothII installation needs outbound Git access. Gitea may be colocated with the production ThothII host.

11. Git workflow and concurrency

11.1 Browser drafts

Drafts remain in browser-local storage and contain:

• remote fingerprint;
• branch;
• workspace ID;
• base commit;
• base workspace blob checksum;
• form data and update timestamp.

Drafts do not modify the shared checkout.

11.2 Publish

Publish is explicit and displays the canonical field-level diff. The backend then:

1. Acquires the repository publish lock.
2. Fetches the remote branch.
3. Compares the submitted base commit and workspace checksum with the remote.
4. If only other workspaces changed, reapplies the draft on the new remote head.
5. If the same workspace changed, returns HTTP 409 with base, local, and remote field differences.
6. Validates the complete resulting repository.
7. Writes YAML and generated files atomically.
8. Creates a commit with the configured technical identity.
9. Pushes the configured branch.
10. Materializes and activates the validated snapshot.

If a concurrent push wins after step 3, the backend fetches once more. It retries only when the target workspace is unchanged; otherwise it returns a conflict.

Without embedded authentication, commits use a technical author such as ThothII Workspace Manager and include an installation-ID trailer. They do not claim a human identity.

11.3 Pull

Pull fetches the remote, requires fast-forward history, validates the complete candidate revision, and activates it atomically. Browser drafts whose base revision becomes stale remain available but are visibly marked as requiring reconciliation.

11.4 Delete

Delete creates a draft deletion and is published as a Git commit. A workspace referenced by an active runtime cannot be deleted. Historical session snapshots remain available, and Git history provides repository-level recovery.

12. CRUD user experience

The right sidebar exposes Workspace management, opening a dedicated page with a workspace list and editable detail area.

The form sections are:

1. General.
2. DWH.
3. Semantic index.
4. LLM policy.
5. Installation requirements.
6. Git status and history.

Actions are:

• New;
• Duplicate;
• Delete;
• Save draft;
• Discard changes;
• Pull;
• Publish;
• Export;
• Import;
• Test on this installation.

Closed choices are used whenever the domain is enumerable:

• database engine;
• transport type;
• vector-store engine;
• embedding provider;
• distance metric;
• TLS mode;
• language;
• LLM provider/model returned by Pi;
• embedding model returned by a reachable provider.

Free text or numeric controls are used for names, descriptions, IDs, database/schema/collection identifiers, ports, dimensions, timeouts, and URLs. They display field-level constraints before submission and server validation errors after submission.

The installation-requirements section shows the exact required, optional, and transport-conditional variable names. It never displays resolved secret values.

13. Validation model

13.1 Formal validation

• YAML and schema version are valid.
• Required fields are present.
• Unknown fields are rejected.
• IDs and database identifiers match their allowed syntax.
• Ports are integers from 1 through 65535.
• Dimensions and timeouts are positive and within configured safety limits.
• URLs use supported schemes.
• enum values come from the closed schema lists.

13.2 Static semantic validation

• Selected transports are supported by the corresponding connector.
• Required fields for each transport can be derived unambiguously.
• Embedding and vector dimensions match.
• LLM default belongs to the allowlist.
• Duplicate workspace IDs and generated environment namespaces are rejected.
• Generated documentation exactly matches the binding contract.

Save draft may retain incomplete local form state in the browser. Publish requires formal and static semantic validation to pass.

13.3 Local operational validation

• Required variables exist.
• Secret files are regular, readable files within approved secret roots.
• DNS, TCP, TLS, and authentication succeed.
• DWH database and schema exist and are readable.
• REST capabilities match the declared logical resource.
• SSH host verification and tunnel opening succeed.
• Vector collection, dimensions, metric, and read capability match.
• Embedding endpoint exposes the declared model and returns the expected dimensions for a controlled probe.
• A requested writer probe has a declared reversible POST operation, distinct writer credential, and successful bounded cleanup; otherwise it is omitted without weakening reader validation.

A portable workspace can be valid but not activatable on a particular installation. Publish is allowed in that state; starting a new session on that installation is not.

14. API contract

GET    /workspace-registry/status
POST   /workspace-registry/pull
GET    /workspaces
GET    /workspaces/:id
POST   /workspaces/validate
POST   /workspaces/:id/test
POST   /workspaces/publish
GET    /workspaces/:id/export
POST   /workspaces/import

The status response contains sanitized remote identity, branch, active commit, divergence, last successful sync, last validation result, and degraded status.

Validation accepts a structured workspace draft rather than arbitrary YAML text. Publish accepts create, update, or delete intent plus base revision metadata. No endpoint accepts a filesystem path.

Operational errors use stable codes that distinguish:

• invalid configuration;
• missing local binding;
• non-activatable workspace;
• stale revision;
• field conflict;
• Git remote unavailable;
• Git authentication failure;
• non-fast-forward history;
• push rejection;
• connector unavailable;
• semantic-index incompatibility.

15. Offline export/import fallback

Export returns <workspace-id>.thoth-workspace.zip containing:

manifest.json
workspace.yaml
contract.env.example
README.md

The manifest contains bundle schema version, workspace ID, source commit, file checksums, and creation timestamp. It contains no secrets or browser preferences.

Import uploads the bundle to the currently open ThothII installation. The backend validates archive size, entry count, entry names, checksums, schema, and semantics. A successful import returns a browser draft; it does not write or publish directly.

The browser can therefore download from one ThothII installation and upload to another without direct server-to-server access. Git remains the authoritative synchronization mechanism.

16. Session integration

New-session creation sends the browser-selected workspace ID, LLM provider/model, and reasoning level. The backend:

1. Resolves the active validated workspace snapshot.
2. Verifies local activatability.
3. Validates the LLM choice against workspace policy and Pi availability.
4. Starts the harness with the immutable snapshot path.
5. Persists workspace ID, workspace revision, provider, model, and reasoning level in the session manifest.

Resume uses the persisted snapshot revision even after later pull or publish operations. Snapshot retention cannot remove revisions referenced by resumable sessions.

Legacy sessions without workspace revision use the existing compatibility resolution and receive a visible legacy warning. New sessions always require a revision.

17. Security constraints

• No secret value appears in Git, generated documentation, API payloads, logs, diagnostics, browser storage, or export bundles.
• Secret references use approved *_FILE variables and approved secret roots.
• Workspace and archive names cannot influence filesystem paths.
• Import prevents zip-slip, symlinks, excessive file count, and excessive expanded size.
• Git commands receive fixed argument arrays; user input is never passed through a shell.
• Git SSH uses explicit known-hosts verification.
• REST and direct TLS validation cannot be disabled silently.
• Diagnostics sanitize provider errors before returning them to the browser.
• Production CORS remains same-origin; absence of embedded authentication does not imply cross-origin write access.
• The first release allows every user who can access the ThothII application to publish workspace changes. This limitation is documented until an authorization layer is introduced.

18. Migration

The migration path is:

1. Introduce the versioned canonical schema and parser.
2. Convert existing harness/workspaces and deployment descriptors into repository fixtures.
3. Generate deterministic environment contracts and compare them with current Compose variables.
4. Configure the persistent registry volume and Git remote.
5. Import the current PSD workspace as the first canonical revision.
6. Keep legacy reads available during a bounded compatibility period.
7. Switch new sessions to validated snapshots and revision pinning.
8. Remove regex-based workspace metadata parsing after all active configurations use schema version 1.

Migration never copies secret values into Git. Existing absolute roots become installation-level storage configuration.

19. Documentation deliverables

• Workspace schema reference with field descriptions and examples.
• Generated documentation for every workspace.
• Generated .env.example for every workspace.
• A detailed local-installation manual for Docker Desktop on macOS and for a local Docker engine on PC. It must cover prerequisites, clone/checkout or remote bootstrap, local Git credentials, persistent volumes, installation bindings, secret files, Docker Compose startup, first pull, workspace diagnostics, local publish, update, backup, and rollback.
• A detailed server-installation manual. It must cover service account and filesystem ownership, persistent registry volume, remote Git and Gitea configuration, HTTPS/SSH Git credentials, CA and known-hosts mounts, secret-file layout and permissions, Compose deployment, first bootstrap, firewall and outbound Git requirements, same-origin reverse-proxy exposure, health/status verification, pull/publish operations, upgrade, backup, degraded-mode recovery, and rollback to a prior validated snapshot.
• The two manuals must distinguish values that are shared in Git from installation-local bindings and secret files. Both must include complete direct PostgreSQL, REST, and SSH-tunnel examples and a troubleshooting table keyed by the stable diagnostic error codes.
• Docker Compose examples for server and local installations, referenced by the corresponding manual and tested as runnable examples.
• Direct PostgreSQL, REST, and SSH-tunnel examples.
• Vector/embedding compatibility explanation.
• Git remote setup for HTTPS and SSH.
• Gitea deployment example.
• Pull, draft, publish, conflict, export, and import operator guide.
• Diagnostic command and error-code reference.
• Migration guide from current PSD configuration.

20. Testing strategy

• Unit tests for schema parsing, canonical serialization, migrations, and unknown-key rejection.
• Unit tests for deterministic environment naming and conditional binding requirements.
• Valid and invalid fixtures for direct, REST, and SSH transports.
• Semantic-index fixtures covering model, dimensions, metric, collection, and read/write mismatch.
• Temporary local Git remotes for clone, pull, publish, retry, divergence, and same-file conflicts.
• Atomic-write and lock tests.
• API tests for CRUD, stale revisions, stable error codes, and sanitized responses.
• Import tests for checksum failure, zip-slip, symlinks, archive limits, and malformed schemas.
• Frontend tests for closed choices, free-field errors, drafts, diffs, conflicts, and installation requirements.
• End-to-end tests using a local Git remote and simulated connectors.
• Deployment tests proving persistent checkout and last-valid-snapshot fallback across container replacement.
• Session tests proving revision pinning and resume after a newer workspace publish.
• Documentation verification that executes the manual's local and server Compose examples in isolated test fixtures, including initial bootstrap and recovery from an unavailable Git remote.

21. Acceptance criteria

The feature is complete when:

• a workspace can be created, edited, duplicated, deleted, pulled, and published from the UI;
• two browsers cannot silently overwrite the same workspace revision;
• server and local Docker installations can consume the same Git repository;
• each installation can bind the same logical workspace through different transports;
• generated variable names and documentation are deterministic and tested;
• no secret value enters Git or an export bundle;
• vector collection and embedding compatibility is enforced;
• a remote outage leaves the last valid snapshot usable;
• every new session records and resumes with an immutable workspace revision;
• detailed, tested installation manuals exist for local PC/Mac Docker deployments and for server deployments;
• existing PSD configuration can be migrated without embedding PSD-specific behavior in the core schema.