ThothII/docs/workspace-diagnostic-protocol.md

Workspace diagnostic protocol

This is the operator contract for testing a workspace on one ThothII installation. The Git-shared descriptor declares what can be checked; the installation supplies the selected transport and the local bindings. No secret value, certificate content, SSH key, or response body belongs in the descriptor, this document, a generated .env.example, or diagnostic output.

Scope and safety rules

• The descriptor is schema version 2. Its vector database and schema are required identity fields; they are not copied from the DWH, even when both services share PostgreSQL.
• Version 1 descriptors are readable only and have migration_required status. An explicit migration supplies semantic_index.vector_store.database and .schema, writes version 2, and must never infer either from dwh.
• Each diagnostic is bounded by the configured workspace diagnostic timeout. Redirects are rejected, response bodies stay inside the adapter, and browser-visible errors are limited to binding_missing, connector_unavailable, and semantic_index_incompatible.
• A REST path is descriptor-declared, origin-relative, starts with one /, and has no query or fragment. The client may use only the declared method, path, auth mode, and response-field names.
• auth: none sends no credential; auth: bearer reads a local file and sends Authorization: Bearer <file-content>; auth: x-api-key sends x-api-key: <file-content>. The resolver, rendered runtime endpoint, and diagnoser do not require or read an API-key file for an auth: none diagnostic. File content is never logged or returned.

Canonical descriptor additions

semantic_index:
  vector_store:
    engine: pgvector
    database: vector_database
    schema: vectors
    collection: clinical_documents
    dimensions: 768
    distance: cosine
    supported_transports: [pgvector_direct, rest_api, ssh_tunnel]
  vector_writer: {} # optional: declares a separately bound writer capability
  embedding:
    provider: ollama_compatible
    model: nomic-embed-text-v2-moe
    dimensions: 768

diagnostics:
  dwh_rest:
    method: POST
    path: /rpc/ping
    auth: bearer
    response: { database: database, schema: schema }
  vector_rest:
    metadata:
      method: GET
      path: /vector/metadata
      auth: bearer
      response: { collection: collection, dimensions: dimensions, distance: distance }
    reversible_probe:
      method: POST
      path: /vector/diagnostic-probe
      auth: bearer
      response: { operation: operation }
  embedding:
    method: GET
    path: /models
    auth: none
    response: { model: model, dimensions: dimensions }

diagnostics.dwh_rest requires DWH rest_api; diagnostics.vector_rest requires vector rest_api. reversible_probe is optional, but when present it must be authenticated POST and declare the response field that echoes the requested operation. Response-map values are JSON object field names, not values to be put in Git.

Installation-local variable contract

Replace <NAMESPACE> with the immutable workspace ID converted to upper case with hyphens changed to underscores. For example, psd-clinical becomes PSD_CLINICAL. Set only the variables for the selected transport. Every *_FILE value is an absolute path to a regular, readable file inside an approved local secret root; it is never the secret itself.

Connector and transport	Required local variables
DWH selection	THT_WS_<NAMESPACE>_DWH_TRANSPORT
DWH postgres_direct	THT_WS_<NAMESPACE>_DWH_HOST, THT_WS_<NAMESPACE>_DWH_PORT, THT_WS_<NAMESPACE>_DWH_USER, THT_WS_<NAMESPACE>_DWH_PASSWORD_FILE; optional THT_WS_<NAMESPACE>_DWH_TLS_CA_FILE
DWH rest_api	THT_WS_<NAMESPACE>_DWH_BASE_URL; THT_WS_<NAMESPACE>_DWH_API_KEY_FILE only for bearer/x-api-key; optional THT_WS_<NAMESPACE>_DWH_TLS_CA_FILE
DWH ssh_tunnel	THT_WS_<NAMESPACE>_DWH_USER, THT_WS_<NAMESPACE>_DWH_PASSWORD_FILE, THT_WS_<NAMESPACE>_DWH_SSH_HOST, THT_WS_<NAMESPACE>_DWH_SSH_PORT, THT_WS_<NAMESPACE>_DWH_SSH_USER, THT_WS_<NAMESPACE>_DWH_SSH_PRIVATE_KEY_FILE, THT_WS_<NAMESPACE>_DWH_SSH_KNOWN_HOSTS_FILE, THT_WS_<NAMESPACE>_DWH_SSH_TARGET_HOST, THT_WS_<NAMESPACE>_DWH_SSH_TARGET_PORT; optional THT_WS_<NAMESPACE>_DWH_TLS_CA_FILE
Vector selection	THT_WS_<NAMESPACE>_VECTOR_TRANSPORT
Vector pgvector_direct	THT_WS_<NAMESPACE>_VECTOR_HOST, THT_WS_<NAMESPACE>_VECTOR_PORT, THT_WS_<NAMESPACE>_VECTOR_USER, THT_WS_<NAMESPACE>_VECTOR_PASSWORD_FILE; optional THT_WS_<NAMESPACE>_VECTOR_TLS_CA_FILE
Vector rest_api	THT_WS_<NAMESPACE>_VECTOR_BASE_URL; THT_WS_<NAMESPACE>_VECTOR_API_KEY_FILE only for bearer/x-api-key; optional THT_WS_<NAMESPACE>_VECTOR_TLS_CA_FILE
Vector ssh_tunnel	THT_WS_<NAMESPACE>_VECTOR_USER, THT_WS_<NAMESPACE>_VECTOR_PASSWORD_FILE, THT_WS_<NAMESPACE>_VECTOR_SSH_HOST, THT_WS_<NAMESPACE>_VECTOR_SSH_PORT, THT_WS_<NAMESPACE>_VECTOR_SSH_USER, THT_WS_<NAMESPACE>_VECTOR_SSH_PRIVATE_KEY_FILE, THT_WS_<NAMESPACE>_VECTOR_SSH_KNOWN_HOSTS_FILE, THT_WS_<NAMESPACE>_VECTOR_SSH_TARGET_HOST, THT_WS_<NAMESPACE>_VECTOR_SSH_TARGET_PORT; optional THT_WS_<NAMESPACE>_VECTOR_TLS_CA_FILE
Optional vector writer	THT_WS_<NAMESPACE>_VECTOR_WRITER_API_KEY_FILE
Embedding service	THT_WS_<NAMESPACE>_EMBEDDING_BASE_URL; optional THT_WS_<NAMESPACE>_EMBEDDING_API_KEY_FILE, THT_WS_<NAMESPACE>_EMBEDDING_TLS_CA_FILE

For the example workspace, the optional writer name is exactly THT_WS_PSD_CLINICAL_VECTOR_WRITER_API_KEY_FILE. It must resolve to a different local file from THT_WS_PSD_CLINICAL_VECTOR_API_KEY_FILE; a reader key is never substituted for a writer key.

Private-CA REST limitation

The current REST diagnostic adapters use the platform fetch implementation and cannot load a per-request private CA. Therefore a REST diagnostic with any *_TLS_CA_FILE binding is refused rather than silently disabling certificate verification. Use an HTTPS endpoint trusted by the runtime trust store, use direct or SSH transport where the native PostgreSQL client can validate the local CA file, or arrange TLS termination at a trusted boundary. This limitation applies to DWH REST, vector metadata/write REST, and embedding REST diagnostics.

DWH diagnostic

For direct PostgreSQL and SSH-tunnelled PostgreSQL, the diagnostic connects with the declared dwh.database, checks TLS and authentication, then executes exactly:

SELECT current_database() AS database, current_schema() AS schema

Both returned values must equal the descriptor's DWH database and schema.

*_TLS_CA_FILE is optional for direct and SSH PostgreSQL diagnostics. When provided, it is used with certificate verification; when absent, the native client still requires a valid certificate chain from the runtime system trust store. Absence never disables TLS verification.

An SSH tunnel changes only the TCP peer to loopback. The forwarded PostgreSQL TLS connection sets its server name to <ROLE>_SSH_TARGET_HOST, so certificate hostname validation remains against the declared remote target rather than 127.0.0.1.

For REST, the descriptor above declares the exact ping:

POST <THT_WS_<NAMESPACE>_DWH_BASE_URL>/rpc/ping
Authorization: Bearer <content of DWH_API_KEY_FILE>

It has no request body. A 2xx response must be a JSON object whose declared database and schema fields equal dwh.database and dwh.schema. For the sample response map, that is:

{ "database": "warehouse", "schema": "datawarehouse" }

The values are illustrative resource identities, not credentials. A different response-field map is valid only when the descriptor declares it.

Vector metadata diagnostic

Direct and SSH vector checks connect to the vector database and schema, not the DWH identity. They inspect the declared collection's vector column and index and must find the exact collection, integer dimensions, and distance (cosine, l2, or inner_product) declared in semantic_index.vector_store.

Their optional *_TLS_CA_FILE follows the same verified private-CA-or-system-trust rule as the DWH diagnostic. For an SSH tunnel, their TLS server name is likewise the declared vector SSH_TARGET_HOST, not the loopback listener.

For REST, the exact descriptor-declared request is, for example:

GET <THT_WS_<NAMESPACE>_VECTOR_BASE_URL>/vector/metadata
Authorization: Bearer <content of VECTOR_API_KEY_FILE>

It has no request body. A 2xx JSON object must supply the declared collection, dimensions, and distance fields. All three values must exactly match the vector-store contract; an integer dimension is required. Metadata from a similarly named collection, a different metric, or a different dimension makes the semantic index incompatible.

Reversible vector writer probe

Ordinary validation is reader-only. A write probe runs only when all of the following are true:

1. The operator explicitly requests it.
2. The descriptor has semantic_index.vector_writer: {}.
3. The descriptor declares an authenticated diagnostics.vector_rest.reversible_probe with an operation response field.
4. The selected vector transport is rest_api.
5. THT_WS_<NAMESPACE>_VECTOR_WRITER_API_KEY_FILE exists locally and is distinct from the reader API-key file.

The probe uses the declared POST endpoint twice, with the same generated ID and the writer key:

{ "operation": "create", "id": "diagnostic:<random UUID>", "collection": "<declared collection>", "dimensions": 768 }

then:

{ "operation": "remove", "id": "diagnostic:<same UUID>", "collection": "<declared collection>" }

Both requests require a 2xx JSON response whose declared operation field equals the requested create or remove operation. Cleanup is attempted in finally, including after a write timeout or error. The endpoint must implement both operations as a bounded, reversible diagnostic operation; an upsert-only endpoint is prohibited. It must not retain, index, or expose diagnostic records. If the writer capability or its local binding is absent, validation remains reader-only and no write request is sent.

Embedding dimensions diagnostic

The embedding request is descriptor-declared, for example:

GET <THT_WS_<NAMESPACE>_EMBEDDING_BASE_URL>/models

It has no body and uses the declared authentication mode. A 2xx JSON object must contain the declared model field equal to semantic_index.embedding.model and a declared dimensions field that is an integer. That integer must equal both semantic_index.embedding.dimensions and semantic_index.vector_store.dimensions.

SSH host verification and tunnel lifecycle

For either DWH or vector ssh_tunnel, the known-hosts file is mandatory and is verified before a connection is accepted. The tunnel is a short-lived loopback forward for the diagnostic only. The effective OpenSSH constraints are:

-N -v
-o BatchMode=yes
-o ExitOnForwardFailure=yes
-o StrictHostKeyChecking=yes
-o UserKnownHostsFile=<ROLE>_SSH_KNOWN_HOSTS_FILE
-i <ROLE>_SSH_PRIVATE_KEY_FILE
-p <ROLE>_SSH_PORT
-L 127.0.0.1:<ephemeral-port>:<ROLE>_SSH_TARGET_HOST:<ROLE>_SSH_TARGET_PORT
<ROLE>_SSH_USER@<ROLE>_SSH_HOST

The local listener is 127.0.0.1 only. The process is terminated in cleanup after the direct probe, on timeout, or on failure. There is no accept-new mode, no disabled host-key checking, and no persistent forwarding.

In this release, ssh_tunnel is therefore a diagnostic-only connector transport. A successful probe is followed by workspace_not_activatable, and POST /sessions rejects the workspace before persisting a manifest or starting Pi. Use direct PostgreSQL/pgvector or REST for runtime sessions until the backend owns a tunnel for the full runtime lifecycle. This restriction does not apply to using SSH as the transport for the workspace Git remote.

Reader-only fallback

A workspace may be fully valid in Git but non-activatable locally when a required reader binding, secret file, host verification, TLS check, or declared diagnostic fails. That state does not alter the shared descriptor and does not permit a new session on that installation. It may still be published and activated elsewhere with valid local bindings. Missing optional writer capability is not a reader failure: it leaves the workspace in reader-only mode and suppresses the writer probe.