ThothII/docs/workspace-diagnostic-protocol.md

# Workspace diagnostic protocol

This is the operator contract for testing a workspace on one ThothII installation. The
Git-shared descriptor declares *what* can be checked; the installation supplies the selected
transport and the local bindings. No secret value, certificate content, SSH key, or response body
belongs in the descriptor, this document, a generated `.env.example`, or diagnostic output.

## Scope and safety rules

- The descriptor is schema version 2. Its vector `database` and `schema` are required identity
  fields; they are not copied from the DWH, even when both services share PostgreSQL.
- Version 1 descriptors are readable only and have `migration_required` status. An explicit
  migration supplies `semantic_index.vector_store.database` and `.schema`, writes version 2, and
  must never infer either from `dwh`.
- Each diagnostic is bounded by the configured workspace diagnostic timeout. Redirects are
  rejected, response bodies stay inside the adapter, and browser-visible errors are limited to
  `binding_missing`, `connector_unavailable`, and `semantic_index_incompatible`.
- A REST path is descriptor-declared, origin-relative, starts with one `/`, and has no query or
  fragment. The client may use only the declared method, path, auth mode, and response-field names.
- `auth: none` sends no credential; `auth: bearer` reads a local file and sends
  `Authorization: Bearer <file-content>`; `auth: x-api-key` sends `x-api-key: <file-content>`.
  The resolver, rendered runtime endpoint, and diagnoser do not require or read an API-key file
  for an `auth: none` diagnostic. File content is never logged or returned.

## Canonical descriptor additions

```yaml
semantic_index:
  vector_store:
    engine: pgvector
    database: vector_database
    schema: vectors
    collection: clinical_documents
    dimensions: 768
    distance: cosine
    supported_transports: [pgvector_direct, rest_api, ssh_tunnel]
  vector_writer: {} # optional: declares a separately bound writer capability
  embedding:
    provider: ollama_compatible
    model: nomic-embed-text-v2-moe
    dimensions: 768

diagnostics:
  dwh_rest:
    method: POST
    path: /rpc/ping
    auth: bearer
    response: { database: database, schema: schema }
  vector_rest:
    metadata:
      method: GET
      path: /vector/metadata
      auth: bearer
      response: { collection: collection, dimensions: dimensions, distance: distance }
    reversible_probe:
      method: POST
      path: /vector/diagnostic-probe
      auth: bearer
      response: { operation: operation }
  embedding:
    method: GET
    path: /models
    auth: none
    response: { model: model, dimensions: dimensions }
```

`diagnostics.dwh_rest` requires DWH `rest_api`; `diagnostics.vector_rest` requires vector
`rest_api`. `reversible_probe` is optional, but when present it must be authenticated `POST` and
declare the response field that echoes the requested `operation`. Response-map values are JSON
object field names, not values to be put in Git.

## Installation-local variable contract

Replace `<NAMESPACE>` with the immutable workspace ID converted to upper case with hyphens changed
to underscores. For example, `psd-clinical` becomes `PSD_CLINICAL`. Set only the variables for the
selected transport. Every `*_FILE` value is an absolute path to a regular, readable file inside an
approved local secret root; it is never the secret itself.

| Connector and transport | Required local variables |
| --- | --- |
| DWH selection | `THT_WS_<NAMESPACE>_DWH_TRANSPORT` |
| DWH `postgres_direct` | `THT_WS_<NAMESPACE>_DWH_HOST`, `THT_WS_<NAMESPACE>_DWH_PORT`, `THT_WS_<NAMESPACE>_DWH_USER`, `THT_WS_<NAMESPACE>_DWH_PASSWORD_FILE`; optional `THT_WS_<NAMESPACE>_DWH_TLS_CA_FILE` |
| DWH `rest_api` | `THT_WS_<NAMESPACE>_DWH_BASE_URL`; `THT_WS_<NAMESPACE>_DWH_API_KEY_FILE` only for `bearer`/`x-api-key`; optional `THT_WS_<NAMESPACE>_DWH_TLS_CA_FILE` |
| DWH `ssh_tunnel` | `THT_WS_<NAMESPACE>_DWH_USER`, `THT_WS_<NAMESPACE>_DWH_PASSWORD_FILE`, `THT_WS_<NAMESPACE>_DWH_SSH_HOST`, `THT_WS_<NAMESPACE>_DWH_SSH_PORT`, `THT_WS_<NAMESPACE>_DWH_SSH_USER`, `THT_WS_<NAMESPACE>_DWH_SSH_PRIVATE_KEY_FILE`, `THT_WS_<NAMESPACE>_DWH_SSH_KNOWN_HOSTS_FILE`, `THT_WS_<NAMESPACE>_DWH_SSH_TARGET_HOST`, `THT_WS_<NAMESPACE>_DWH_SSH_TARGET_PORT`; optional `THT_WS_<NAMESPACE>_DWH_TLS_CA_FILE` |
| Vector selection | `THT_WS_<NAMESPACE>_VECTOR_TRANSPORT` |
| Vector `pgvector_direct` | `THT_WS_<NAMESPACE>_VECTOR_HOST`, `THT_WS_<NAMESPACE>_VECTOR_PORT`, `THT_WS_<NAMESPACE>_VECTOR_USER`, `THT_WS_<NAMESPACE>_VECTOR_PASSWORD_FILE`; optional `THT_WS_<NAMESPACE>_VECTOR_TLS_CA_FILE` |
| Vector `rest_api` | `THT_WS_<NAMESPACE>_VECTOR_BASE_URL`; `THT_WS_<NAMESPACE>_VECTOR_API_KEY_FILE` only for `bearer`/`x-api-key`; optional `THT_WS_<NAMESPACE>_VECTOR_TLS_CA_FILE` |
| Vector `ssh_tunnel` | `THT_WS_<NAMESPACE>_VECTOR_USER`, `THT_WS_<NAMESPACE>_VECTOR_PASSWORD_FILE`, `THT_WS_<NAMESPACE>_VECTOR_SSH_HOST`, `THT_WS_<NAMESPACE>_VECTOR_SSH_PORT`, `THT_WS_<NAMESPACE>_VECTOR_SSH_USER`, `THT_WS_<NAMESPACE>_VECTOR_SSH_PRIVATE_KEY_FILE`, `THT_WS_<NAMESPACE>_VECTOR_SSH_KNOWN_HOSTS_FILE`, `THT_WS_<NAMESPACE>_VECTOR_SSH_TARGET_HOST`, `THT_WS_<NAMESPACE>_VECTOR_SSH_TARGET_PORT`; optional `THT_WS_<NAMESPACE>_VECTOR_TLS_CA_FILE` |
| Optional vector writer | `THT_WS_<NAMESPACE>_VECTOR_WRITER_API_KEY_FILE` |
| Embedding service | `THT_WS_<NAMESPACE>_EMBEDDING_BASE_URL`; optional `THT_WS_<NAMESPACE>_EMBEDDING_API_KEY_FILE`, `THT_WS_<NAMESPACE>_EMBEDDING_TLS_CA_FILE` |

For the example workspace, the optional writer name is exactly
`THT_WS_PSD_CLINICAL_VECTOR_WRITER_API_KEY_FILE`. It must resolve to a different local file from
`THT_WS_PSD_CLINICAL_VECTOR_API_KEY_FILE`; a reader key is never substituted for a writer key.

### Private-CA REST limitation

The current REST diagnostic adapters use the platform `fetch` implementation and cannot load a
per-request private CA. Therefore a REST diagnostic with any `*_TLS_CA_FILE` binding is refused
rather than silently disabling certificate verification. Use an HTTPS endpoint trusted by the
runtime trust store, use direct or SSH transport where the native PostgreSQL client can validate
the local CA file, or arrange TLS termination at a trusted boundary. This limitation applies to
DWH REST, vector metadata/write REST, and embedding REST diagnostics.

## DWH diagnostic

For direct PostgreSQL and SSH-tunnelled PostgreSQL, the diagnostic connects with the declared
`dwh.database`, checks TLS and authentication, then executes exactly:

```sql
SELECT current_database() AS database, current_schema() AS schema
```

Both returned values must equal the descriptor's DWH database and schema.

`*_TLS_CA_FILE` is optional for direct and SSH PostgreSQL diagnostics. When provided, it is used
with certificate verification; when absent, the native client still requires a valid certificate
chain from the runtime system trust store. Absence never disables TLS verification.

An SSH tunnel changes only the TCP peer to loopback. The forwarded PostgreSQL TLS connection sets
its server name to `<ROLE>_SSH_TARGET_HOST`, so certificate hostname validation remains against the
declared remote target rather than `127.0.0.1`.

For REST, the descriptor above declares the exact ping:

```text
POST <THT_WS_<NAMESPACE>_DWH_BASE_URL>/rpc/ping
Authorization: Bearer <content of DWH_API_KEY_FILE>
```

It has no request body. A 2xx response must be a JSON object whose declared `database` and
`schema` fields equal `dwh.database` and `dwh.schema`. For the sample response map, that is:

```json
{ "database": "warehouse", "schema": "datawarehouse" }
```

The values are illustrative resource identities, not credentials. A different response-field map
is valid only when the descriptor declares it.

## Vector metadata diagnostic

Direct and SSH vector checks connect to the *vector* `database` and `schema`, not the DWH
identity. They inspect the declared collection's vector column and index and must find the exact
collection, integer `dimensions`, and `distance` (`cosine`, `l2`, or `inner_product`) declared in
`semantic_index.vector_store`.

Their optional `*_TLS_CA_FILE` follows the same verified private-CA-or-system-trust rule as the
DWH diagnostic. For an SSH tunnel, their TLS server name is likewise the declared vector
`SSH_TARGET_HOST`, not the loopback listener.

For REST, the exact descriptor-declared request is, for example:

```text
GET <THT_WS_<NAMESPACE>_VECTOR_BASE_URL>/vector/metadata
Authorization: Bearer <content of VECTOR_API_KEY_FILE>
```

It has no request body. A 2xx JSON object must supply the declared `collection`, `dimensions`, and
`distance` fields. All three values must exactly match the vector-store contract; an integer
dimension is required. Metadata from a similarly named collection, a different metric, or a
different dimension makes the semantic index incompatible.

## Reversible vector writer probe

Ordinary validation is reader-only. A write probe runs only when all of the following are true:

1. The operator explicitly requests it.
2. The descriptor has `semantic_index.vector_writer: {}`.
3. The descriptor declares an authenticated `diagnostics.vector_rest.reversible_probe` with an
   `operation` response field.
4. The selected vector transport is `rest_api`.
5. `THT_WS_<NAMESPACE>_VECTOR_WRITER_API_KEY_FILE` exists locally and is distinct from the reader
   API-key file.

The probe uses the declared `POST` endpoint twice, with the same generated ID and the writer key:

```json
{ "operation": "create", "id": "diagnostic:<random UUID>", "collection": "<declared collection>", "dimensions": 768 }
```

then:

```json
{ "operation": "remove", "id": "diagnostic:<same UUID>", "collection": "<declared collection>" }
```

Both requests require a 2xx JSON response whose declared `operation` field equals the requested
`create` or `remove` operation. Cleanup is attempted in `finally`, including after a write timeout
or error. The endpoint must implement both operations as a bounded, reversible diagnostic
operation; an upsert-only endpoint is prohibited. It must not retain, index, or expose diagnostic
records. If the writer capability or its local binding is absent, validation remains reader-only
and no write request is sent.

## Embedding dimensions diagnostic

The embedding request is descriptor-declared, for example:

```text
GET <THT_WS_<NAMESPACE>_EMBEDDING_BASE_URL>/models
```

It has no body and uses the declared authentication mode. A 2xx JSON object must contain the
declared model field equal to `semantic_index.embedding.model` and a declared dimensions field that
is an integer. That integer must equal both `semantic_index.embedding.dimensions` and
`semantic_index.vector_store.dimensions`.

## SSH host verification and tunnel lifecycle

For either DWH or vector `ssh_tunnel`, the known-hosts file is mandatory and is verified before a
connection is accepted. The tunnel is a short-lived loopback forward for the diagnostic only. The
effective OpenSSH constraints are:

```text
-N -v
-o BatchMode=yes
-o ExitOnForwardFailure=yes
-o StrictHostKeyChecking=yes
-o UserKnownHostsFile=<ROLE>_SSH_KNOWN_HOSTS_FILE
-i <ROLE>_SSH_PRIVATE_KEY_FILE
-p <ROLE>_SSH_PORT
-L 127.0.0.1:<ephemeral-port>:<ROLE>_SSH_TARGET_HOST:<ROLE>_SSH_TARGET_PORT
<ROLE>_SSH_USER@<ROLE>_SSH_HOST
```

The local listener is `127.0.0.1` only. The process is terminated in cleanup after the direct
probe, on timeout, or on failure. There is no accept-new mode, no disabled host-key checking, and
no persistent forwarding.

In this release, `ssh_tunnel` is therefore a diagnostic-only connector transport. A successful
probe is followed by `workspace_not_activatable`, and `POST /sessions` rejects the workspace before
persisting a manifest or starting Pi. Use direct PostgreSQL/pgvector or REST for runtime sessions
until the backend owns a tunnel for the full runtime lifecycle. This restriction does not apply to
using SSH as the transport for the workspace Git remote.

## Reader-only fallback

A workspace may be fully valid in Git but non-activatable locally when a required reader binding,
secret file, host verification, TLS check, or declared diagnostic fails. That state does not alter
the shared descriptor and does not permit a new session on that installation. It may still be
published and activated elsewhere with valid local bindings. Missing optional writer capability is
not a reader failure: it leaves the workspace in reader-only mode and suppresses the writer probe.