docs: document internal semantic infrastructure
This commit is contained in:
@@ -16,7 +16,7 @@ decisione concept_clarified nel ledger della sessione
|
||||
F8: il reviewer decide se promuoverla
|
||||
│
|
||||
├── registro globale registry.jsonl
|
||||
└── indice semantico pgvector
|
||||
└── indice semantico Qdrant
|
||||
│
|
||||
▼
|
||||
F2 di una sessione futura
|
||||
@@ -33,7 +33,7 @@ Implementazione principale: [harness/tht/memory.py](../harness/tht/memory.py:14)
|
||||
| --- | --- | --- |
|
||||
| Ledger della sessione | `concept_clarified`, `memory_promoted`, `memory_promotion_declined` | Audit e stato della singola sessione |
|
||||
| Registro globale | Record `mem-XXXX` in `registry.jsonl` | Archivio canonico attuale delle memory |
|
||||
| Indice pgvector | Embedding e metadati derivati dal registro | Ricerca semantica |
|
||||
| Indice Qdrant | Embedding e metadati derivati dal registro | Ricerca semantica |
|
||||
|
||||
Il ledger contiene la provenienza e le decisioni umane. Il record globale contiene il testo riutilizzabile. L'indice vettoriale è una proiezione per la ricerca, non il posto in cui il workflow registra direttamente le decisioni.
|
||||
|
||||
@@ -114,9 +114,9 @@ Il registro attuale è:
|
||||
|
||||
La scrittura viene fatta tramite file temporaneo e `os.replace`, quindi la sostituzione del registro è atomica. L'idempotenza della promozione è basata sulla coppia `session_id + decision_seq`: la stessa decisione della stessa sessione non genera due record globali.
|
||||
|
||||
### pgvector
|
||||
### Qdrant
|
||||
|
||||
Dopo la promozione, `save-one` costruisce un solo `VectorRecord` e lo invia all'indice pgvector. Il testo indicizzato include:
|
||||
Dopo la promozione, `save-one` costruisce un solo `VectorRecord` e lo invia all'indice Qdrant. Il testo indicizzato include:
|
||||
|
||||
- tipo e soggetto;
|
||||
- dettaglio;
|
||||
@@ -124,13 +124,13 @@ Dopo la promozione, `save-one` costruisce un solo `VectorRecord` e lo invia all'
|
||||
- domanda di contesto;
|
||||
- eventuali concetti e mapping.
|
||||
|
||||
Il record vettoriale usa l'id `memory:mem-XXXX`, mentre i metadati conservano `subject`, `detail`, `rationale`, `tables` e `concepts`. L'hash SHA-256 del contenuto impedisce di ricalcolare embedding e upsert quando il testo non è cambiato.
|
||||
Il record vettoriale usa l'id `memory:mem-XXXX`, mentre i metadati conservano `subject`, `detail`, `rationale`, `tables`, `concepts` e il discriminante `kind`. L'hash SHA-256 del contenuto impedisce di ricalcolare embedding e upsert quando il testo non è cambiato.
|
||||
|
||||
Il comportamento è implementato in [harness/tht/memory.py](../harness/tht/memory.py:253) e [harness/tht/memory.py](../harness/tht/memory.py:305).
|
||||
|
||||
### Fonte canonica attuale
|
||||
|
||||
Oggi il registro JSONL è ancora la fonte canonica applicativa e pgvector è l'indice derivato. Il commento iniziale di [memory_cmd.py](../harness/tht/cli/memory_cmd.py:1) segnala un debito tecnico: l'architettura futura prevista sarebbe usare direttamente il vector DB come archivio unico, ma questa migrazione non è ancora completata.
|
||||
Oggi il registro JSONL è ancora la fonte canonica applicativa e Qdrant resta un indice derivato ma persistente. Il workflow non registra direttamente le decisioni nel vector DB: usa Qdrant come proiezione interrogabile del registro e del ledger effettivo.
|
||||
|
||||
## Riutilizzo in F2
|
||||
|
||||
@@ -209,10 +209,10 @@ Questo evita che una singola modifica al prompt o a un solo componente reintrodu
|
||||
Il salvataggio segue sostanzialmente questa sequenza:
|
||||
|
||||
```text
|
||||
registro JSONL → pgvector → marker memory_promoted nel ledger
|
||||
registro JSONL → Qdrant → marker memory_promoted nel ledger
|
||||
```
|
||||
|
||||
Se pgvector non è disponibile, il registro può contenere una memory non ancora ricercabile; il comando segnala che sarà necessario reindicizzare.
|
||||
Se Qdrant non è disponibile, il registro può contenere una memory non ancora ricercabile; il comando segnala che sarà necessario reindicizzare.
|
||||
|
||||
Se il marker del ledger fallisce dopo il salvataggio nel vector DB, la memory può risultare globalmente presente ma senza audit completo nella sessione. Il gate restituisce un comando di recupero manuale.
|
||||
|
||||
@@ -232,4 +232,4 @@ Vecchi record `table_promoted` o `table_excluded` possono ancora esistere in art
|
||||
|
||||
La gestione attuale è coerente con il requisito funzionale: una memory è una conoscenza concettuale riutilizzabile, non una scelta di schema-linking.
|
||||
|
||||
La parte più solida è la difesa multilivello del tipo `concept_clarified`. Il principale debito tecnico riguarda invece la convivenza del registro JSONL con pgvector e l'assenza di una transazione unica tra archivio globale, indice semantico e ledger della sessione.
|
||||
La parte più solida è la difesa multilivello del tipo `concept_clarified`. Il principale debito tecnico riguarda invece la convivenza del registro JSONL con Qdrant e l'assenza di una transazione unica tra archivio globale, indice semantico e ledger della sessione.
|
||||
|
||||
@@ -130,8 +130,11 @@ before creating sessions. Git pull/push over SSH remains fully supported and is
|
||||
## Bootstrap, first pull, and diagnostics
|
||||
|
||||
Use the repository's canonical `compose.yaml` plus `deploy/compose.local.yaml`; they always start
|
||||
the mandatory `frontend` and `core` services. Do not copy or maintain a standalone application
|
||||
Compose file. Copy [the bindings env example](examples/workspace-bindings.env.example) into an
|
||||
`frontend`, `core`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`. This profile
|
||||
is CPU-first. Add `THOTH_ENABLE_EMBEDDING_GPU=1` only on a Linux host that intentionally exposes a
|
||||
supported GPU device to Docker. Qdrant is a derived but persistent index, while Ollama keeps a
|
||||
local model cache for `qwen3-embedding:0.6b` (`1024` dimensions, cosine distance). Do not copy or
|
||||
maintain a standalone application Compose file. Copy [the bindings env example](examples/workspace-bindings.env.example) into an
|
||||
untracked operator directory and create a protected operator env file from
|
||||
`deploy/env/local.env.example`. It must contain absolute `PI_AUTH_FILE`,
|
||||
`THT_SECRETS_FILE`, `THT_WORKSPACE_BINDINGS_ENV_FILE`, and connector `*_SOURCE` paths.
|
||||
@@ -172,6 +175,10 @@ Use `POST /workspace-registry/pull` to fetch later revisions. Run workspace diag
|
||||
required DWH bindings are mounted. Schema-v3 diagnostics probe the internal Qdrant/Ollama
|
||||
services through backend config; ordinary diagnostics are read-only.
|
||||
|
||||
Schema-v3 is the only operational descriptor format. Schema-v1/v2 descriptors remain
|
||||
`migration_required` until an explicit reviewed migration writes schema version 3. One workspace owns one Qdrant collection; schema, Evidence, and Memory records share that collection and remain
|
||||
isolated by payload `kind`.
|
||||
|
||||
To migrate an existing legacy descriptor, create/clone an empty private remote, set the absolute
|
||||
`THT_SOURCE_ROOT`, transform with absolute paths, review the schema-v1 result, explicitly produce
|
||||
the reviewed schema-v3 contract, then commit/push. The transformer never imports `${ENV}` values
|
||||
|
||||
@@ -32,8 +32,9 @@ targets with runtime ownership without copying secret or tracked file contents i
|
||||
state. Rerun it after a restore and before Compose or `thothctl` startup; it is idempotent and does
|
||||
not overwrite existing targets.
|
||||
|
||||
Permit outbound TCP only to approved Git/Gitea, DWH, Qdrant, embedding, and bastion endpoints.
|
||||
Allow inbound traffic only from the reverse proxy/Docker network. Do not give the runtime service
|
||||
Permit outbound TCP only to approved Git/Gitea, DWH, LLM, and optional bastion endpoints.
|
||||
Qdrant and Ollama run inside the Compose stack. Allow inbound traffic only from the reverse
|
||||
proxy/Docker network. Do not give the runtime service
|
||||
account Gitea administration, database-superuser rights, or a shell in the Git host.
|
||||
|
||||
## Gitea and remote Git setup
|
||||
@@ -151,7 +152,11 @@ The Git registry itself may still use SSH normally.
|
||||
## Same-origin reverse proxy, bootstrap, and health
|
||||
|
||||
Use the repository's canonical `compose.yaml` plus `deploy/compose.server.yaml`; they always
|
||||
start the mandatory `frontend` and `core` services. Do not copy or maintain a standalone
|
||||
start `frontend`, `core`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`.
|
||||
Startup is CPU-first; use `THOTH_ENABLE_EMBEDDING_GPU=1` only when the server intentionally
|
||||
exposes a supported GPU device to Docker. Qdrant is a derived but persistent index, and the
|
||||
Ollama model cache persists the exact `qwen3-embedding:0.6b` model (`1024` dimensions, cosine
|
||||
distance) for offline reuse. Do not copy or maintain a standalone
|
||||
application Compose file. Copy `docs/install/examples/thothii-installation.server.yaml` to the
|
||||
protected operator directory and preserve its required session-server overlay, exactly one Git
|
||||
transport override, and generated connector-secret override.
|
||||
@@ -221,6 +226,10 @@ Its schema-v1 output is `migration_required`; explicitly supply collection ident
|
||||
and the reviewed v3 contract before commit. Never import `${ENV}` values or
|
||||
copy secret files.
|
||||
|
||||
Schema-v3 is the only operational descriptor contract. Schema-v1/v2 descriptors remain
|
||||
`migration_required` until an explicit reviewed migration writes version 3. One workspace owns one Qdrant collection; schema, Evidence, and Memory records share it and stay separated by payload
|
||||
`kind`.
|
||||
|
||||
After valid bootstrap, Git outage retains the active snapshot with `degraded: true`. Repair
|
||||
egress/DNS/CA/credentials, pull, and confirm healthy status. Roll back a bad descriptor through a
|
||||
reviewed Git revert/release branch, advance the remote through normal policy, pull it, and confirm
|
||||
@@ -248,3 +257,24 @@ revision and monitor status. If snapshots are missing or corrupt, stop the servi
|
||||
newest verified registry backup, start it privately, verify status, and then reopen proxy traffic.
|
||||
A first-bootstrap failure has no fallback: repair remote trust rather than creating an unreviewed
|
||||
runtime checkout.
|
||||
|
||||
## Qdrant backup/restore and cache recovery
|
||||
|
||||
Use the repository helpers for Qdrant backup/restore:
|
||||
|
||||
```sh
|
||||
./scripts/vector-backup.sh --project-name thothii --output /secure/backups/thoth-qdrant-2026-08-08.tar
|
||||
./scripts/vector-restore.sh --project-name thothii --input /secure/backups/thoth-qdrant-2026-08-08.tar --confirm-project thothii
|
||||
```
|
||||
|
||||
Qdrant backup/restore targets exactly one labeled `qdrant-data` volume for the named Compose
|
||||
project. Restore requires the exact repeated project confirmation, validates the archive before
|
||||
stopping `qdrant`, stages rollback content, and restores in place only for that project-scoped
|
||||
volume. It does not migrate schema-v1/v2 workspaces, rename collections, or resolve semantic-index
|
||||
incompatibilities.
|
||||
|
||||
The Ollama model cache is a recoverable local cache, not the canonical semantic source of truth.
|
||||
You may back up `embedding-models` for faster offline recovery, but a cache loss is recoverable by
|
||||
re-pulling `qwen3-embedding:0.6b` through `embedding-model-init`.
|
||||
|
||||
Only the Git remote, DWH, LLM, and optional bastion endpoints stay external.
|
||||
|
||||
@@ -8,8 +8,8 @@ ThothII usa una topologia Compose unica:
|
||||
- `embedding`
|
||||
- `embedding-model-init`
|
||||
|
||||
Qdrant e Ollama embedding sono servizi interni obbligatori del progetto Compose. Restano
|
||||
esterni solo DWH e LLM.
|
||||
Qdrant e Ollama embedding sono servizi interni obbligatori del progetto Compose. Restano esterni solo DWH e LLM. Il modello fissato è `qwen3-embedding:0.6b` con 1024 dimensioni e distanza
|
||||
coseno; `embedding-model-init` lo prepara prima dell'avvio di `core`.
|
||||
|
||||
## Comando standard locale
|
||||
|
||||
|
||||
@@ -1,44 +1,47 @@
|
||||
# Workspace diagnostic protocol
|
||||
|
||||
This is the operator contract for testing a workspace on one ThothII installation. The
|
||||
Git-shared descriptor declares *what* can be checked; the installation supplies the selected
|
||||
transport and the local bindings. No secret value, certificate content, SSH key, or response body
|
||||
belongs in the descriptor, this document, a generated `.env.example`, or diagnostic output.
|
||||
This is the operator contract for testing a workspace on one ThothII installation. The Git-shared
|
||||
descriptor declares what can be checked; the installation supplies only the selected DWH
|
||||
transport and local secret-file bindings. No secret value, certificate content, SSH key, or
|
||||
response body belongs in the descriptor, generated `.env.example` files, or diagnostic output.
|
||||
|
||||
## Scope and safety rules
|
||||
|
||||
- The descriptor is schema version 2. Its vector `database` and `schema` are required identity
|
||||
fields; they are not copied from the DWH, even when both services share PostgreSQL.
|
||||
- Version 1 descriptors are readable only and have `migration_required` status. An explicit
|
||||
migration supplies `semantic_index.vector_store.database` and `.schema`, writes version 2, and
|
||||
must never infer either from `dwh`.
|
||||
- Each diagnostic is bounded by the configured workspace diagnostic timeout. Redirects are
|
||||
rejected, response bodies stay inside the adapter, and browser-visible errors are limited to
|
||||
`binding_missing`, `connector_unavailable`, and `semantic_index_incompatible`.
|
||||
- A REST path is descriptor-declared, origin-relative, starts with one `/`, and has no query or
|
||||
fragment. The client may use only the declared method, path, auth mode, and response-field names.
|
||||
- `auth: none` sends no credential; `auth: bearer` reads a local file and sends
|
||||
`Authorization: Bearer <file-content>`; `auth: x-api-key` sends `x-api-key: <file-content>`.
|
||||
The resolver, rendered runtime endpoint, and diagnoser do not require or read an API-key file
|
||||
for an `auth: none` diagnostic. File content is never logged or returned.
|
||||
- The operational descriptor is schema version 3.
|
||||
- Schema-v1/v2 descriptors are readable only and remain `migration_required` until an explicit
|
||||
reviewed migration writes schema version 3.
|
||||
- One workspace owns one Qdrant collection.
|
||||
- Qdrant and Ollama are internal services. Operators do not bind external vector or embedding
|
||||
transports for active manuals or supported diagnostics.
|
||||
- Each diagnostic is bounded by the configured timeout. Redirects are rejected, response bodies
|
||||
stay inside the adapter, and browser-visible errors are limited to `binding_missing`,
|
||||
`connector_unavailable`, and `semantic_index_incompatible`.
|
||||
|
||||
## Canonical descriptor additions
|
||||
## Canonical descriptor contract
|
||||
|
||||
```yaml
|
||||
workspace:
|
||||
schema_version: 3
|
||||
id: psd-clinical
|
||||
name: PSD Clinical
|
||||
language: it
|
||||
|
||||
dwh:
|
||||
engine: postgres
|
||||
database: warehouse
|
||||
schema: datawarehouse
|
||||
supported_transports: [postgres_direct, rest_api, ssh_tunnel]
|
||||
|
||||
semantic_index:
|
||||
vector_store:
|
||||
engine: pgvector
|
||||
database: vector_database
|
||||
schema: vectors
|
||||
collection: clinical_documents
|
||||
dimensions: 768
|
||||
engine: qdrant
|
||||
collection: psd-clinical
|
||||
dimensions: 1024
|
||||
distance: cosine
|
||||
supported_transports: [pgvector_direct, rest_api, ssh_tunnel]
|
||||
vector_writer: {} # optional: declares a separately bound writer capability
|
||||
embedding:
|
||||
provider: ollama_compatible
|
||||
model: nomic-embed-text-v2-moe
|
||||
dimensions: 768
|
||||
provider: ollama_internal
|
||||
model: qwen3-embedding:0.6b
|
||||
dimensions: 1024
|
||||
|
||||
diagnostics:
|
||||
dwh_rest:
|
||||
@@ -46,35 +49,25 @@ diagnostics:
|
||||
path: /rpc/ping
|
||||
auth: bearer
|
||||
response: { database: database, schema: schema }
|
||||
vector_rest:
|
||||
metadata:
|
||||
method: GET
|
||||
path: /vector/metadata
|
||||
auth: bearer
|
||||
response: { collection: collection, dimensions: dimensions, distance: distance }
|
||||
reversible_probe:
|
||||
method: POST
|
||||
path: /vector/diagnostic-probe
|
||||
auth: bearer
|
||||
response: { operation: operation }
|
||||
embedding:
|
||||
method: GET
|
||||
path: /models
|
||||
auth: none
|
||||
response: { model: model, dimensions: dimensions }
|
||||
```
|
||||
|
||||
`diagnostics.dwh_rest` requires DWH `rest_api`; `diagnostics.vector_rest` requires vector
|
||||
`rest_api`. `reversible_probe` is optional, but when present it must be authenticated `POST` and
|
||||
declare the response field that echoes the requested `operation`. Response-map values are JSON
|
||||
object field names, not values to be put in Git.
|
||||
The semantic-index contract is fixed:
|
||||
|
||||
- `engine: qdrant`
|
||||
- collection name equals the workspace-owned portable identifier
|
||||
- `qwen3-embedding:0.6b`
|
||||
- `1024` dimensions
|
||||
- cosine distance
|
||||
|
||||
If any active collection reports a different model pairing, dimension, or distance, diagnostics
|
||||
must return `semantic_index_incompatible` rather than silently rewriting data.
|
||||
|
||||
## Installation-local variable contract
|
||||
|
||||
Replace `<NAMESPACE>` with the immutable workspace ID converted to upper case with hyphens changed
|
||||
to underscores. For example, `psd-clinical` becomes `PSD_CLINICAL`. Set only the variables for the
|
||||
selected transport. Every `*_FILE` value is an absolute path to a regular, readable file inside an
|
||||
approved local secret root; it is never the secret itself.
|
||||
selected DWH transport. Every `*_FILE` value is an absolute path to a regular, readable file
|
||||
inside an approved local secret root; it is never the secret itself.
|
||||
|
||||
| Connector and transport | Required local variables |
|
||||
| --- | --- |
|
||||
@@ -82,25 +75,9 @@ approved local secret root; it is never the secret itself.
|
||||
| DWH `postgres_direct` | `THT_WS_<NAMESPACE>_DWH_HOST`, `THT_WS_<NAMESPACE>_DWH_PORT`, `THT_WS_<NAMESPACE>_DWH_USER`, `THT_WS_<NAMESPACE>_DWH_PASSWORD_FILE`; optional `THT_WS_<NAMESPACE>_DWH_TLS_CA_FILE` |
|
||||
| DWH `rest_api` | `THT_WS_<NAMESPACE>_DWH_BASE_URL`; `THT_WS_<NAMESPACE>_DWH_API_KEY_FILE` only for `bearer`/`x-api-key`; optional `THT_WS_<NAMESPACE>_DWH_TLS_CA_FILE` |
|
||||
| DWH `ssh_tunnel` | `THT_WS_<NAMESPACE>_DWH_USER`, `THT_WS_<NAMESPACE>_DWH_PASSWORD_FILE`, `THT_WS_<NAMESPACE>_DWH_SSH_HOST`, `THT_WS_<NAMESPACE>_DWH_SSH_PORT`, `THT_WS_<NAMESPACE>_DWH_SSH_USER`, `THT_WS_<NAMESPACE>_DWH_SSH_PRIVATE_KEY_FILE`, `THT_WS_<NAMESPACE>_DWH_SSH_KNOWN_HOSTS_FILE`, `THT_WS_<NAMESPACE>_DWH_SSH_TARGET_HOST`, `THT_WS_<NAMESPACE>_DWH_SSH_TARGET_PORT`; optional `THT_WS_<NAMESPACE>_DWH_TLS_CA_FILE` |
|
||||
| Vector selection | `THT_WS_<NAMESPACE>_VECTOR_TRANSPORT` |
|
||||
| Vector `pgvector_direct` | `THT_WS_<NAMESPACE>_VECTOR_HOST`, `THT_WS_<NAMESPACE>_VECTOR_PORT`, `THT_WS_<NAMESPACE>_VECTOR_USER`, `THT_WS_<NAMESPACE>_VECTOR_PASSWORD_FILE`; optional `THT_WS_<NAMESPACE>_VECTOR_TLS_CA_FILE` |
|
||||
| Vector `rest_api` | `THT_WS_<NAMESPACE>_VECTOR_BASE_URL`; `THT_WS_<NAMESPACE>_VECTOR_API_KEY_FILE` only for `bearer`/`x-api-key`; optional `THT_WS_<NAMESPACE>_VECTOR_TLS_CA_FILE` |
|
||||
| Vector `ssh_tunnel` | `THT_WS_<NAMESPACE>_VECTOR_USER`, `THT_WS_<NAMESPACE>_VECTOR_PASSWORD_FILE`, `THT_WS_<NAMESPACE>_VECTOR_SSH_HOST`, `THT_WS_<NAMESPACE>_VECTOR_SSH_PORT`, `THT_WS_<NAMESPACE>_VECTOR_SSH_USER`, `THT_WS_<NAMESPACE>_VECTOR_SSH_PRIVATE_KEY_FILE`, `THT_WS_<NAMESPACE>_VECTOR_SSH_KNOWN_HOSTS_FILE`, `THT_WS_<NAMESPACE>_VECTOR_SSH_TARGET_HOST`, `THT_WS_<NAMESPACE>_VECTOR_SSH_TARGET_PORT`; optional `THT_WS_<NAMESPACE>_VECTOR_TLS_CA_FILE` |
|
||||
| Optional vector writer | `THT_WS_<NAMESPACE>_VECTOR_WRITER_API_KEY_FILE` |
|
||||
| Embedding service | `THT_WS_<NAMESPACE>_EMBEDDING_BASE_URL`; optional `THT_WS_<NAMESPACE>_EMBEDDING_API_KEY_FILE`, `THT_WS_<NAMESPACE>_EMBEDDING_TLS_CA_FILE` |
|
||||
|
||||
For the example workspace, the optional writer name is exactly
|
||||
`THT_WS_PSD_CLINICAL_VECTOR_WRITER_API_KEY_FILE`. It must resolve to a different local file from
|
||||
`THT_WS_PSD_CLINICAL_VECTOR_API_KEY_FILE`; a reader key is never substituted for a writer key.
|
||||
|
||||
### Private-CA REST limitation
|
||||
|
||||
The current REST diagnostic adapters use the platform `fetch` implementation and cannot load a
|
||||
per-request private CA. Therefore a REST diagnostic with any `*_TLS_CA_FILE` binding is refused
|
||||
rather than silently disabling certificate verification. Use an HTTPS endpoint trusted by the
|
||||
runtime trust store, use direct or SSH transport where the native PostgreSQL client can validate
|
||||
the local CA file, or arrange TLS termination at a trusted boundary. This limitation applies to
|
||||
DWH REST, vector metadata/write REST, and embedding REST diagnostics.
|
||||
There are no supported `THT_WS_<NAMESPACE>_VECTOR_*` or
|
||||
`THT_WS_<NAMESPACE>_EMBEDDING_*` installation bindings in the active operator contract.
|
||||
|
||||
## DWH diagnostic
|
||||
|
||||
@@ -113,15 +90,7 @@ SELECT current_database() AS database, current_schema() AS schema
|
||||
|
||||
Both returned values must equal the descriptor's DWH database and schema.
|
||||
|
||||
`*_TLS_CA_FILE` is optional for direct and SSH PostgreSQL diagnostics. When provided, it is used
|
||||
with certificate verification; when absent, the native client still requires a valid certificate
|
||||
chain from the runtime system trust store. Absence never disables TLS verification.
|
||||
|
||||
An SSH tunnel changes only the TCP peer to loopback. The forwarded PostgreSQL TLS connection sets
|
||||
its server name to `<ROLE>_SSH_TARGET_HOST`, so certificate hostname validation remains against the
|
||||
declared remote target rather than `127.0.0.1`.
|
||||
|
||||
For REST, the descriptor above declares the exact ping:
|
||||
For REST, the descriptor-declared request is for example:
|
||||
|
||||
```text
|
||||
POST <THT_WS_<NAMESPACE>_DWH_BASE_URL>/rpc/ping
|
||||
@@ -129,87 +98,27 @@ Authorization: Bearer <content of DWH_API_KEY_FILE>
|
||||
```
|
||||
|
||||
It has no request body. A 2xx response must be a JSON object whose declared `database` and
|
||||
`schema` fields equal `dwh.database` and `dwh.schema`. For the sample response map, that is:
|
||||
`schema` fields match the descriptor.
|
||||
|
||||
```json
|
||||
{ "database": "warehouse", "schema": "datawarehouse" }
|
||||
```
|
||||
## Internal semantic-service diagnostic
|
||||
|
||||
The values are illustrative resource identities, not credentials. A different response-field map
|
||||
is valid only when the descriptor declares it.
|
||||
Schema-v3 workspace diagnostics also verify the internal semantic infrastructure through backend
|
||||
configuration:
|
||||
|
||||
## Vector metadata diagnostic
|
||||
- Qdrant must be reachable at the installation-owned internal URL.
|
||||
- The workspace-owned collection must exist or be creatable with `1024` dimensions and cosine
|
||||
distance.
|
||||
- Ollama must provide `qwen3-embedding:0.6b`.
|
||||
- A bounded embed probe must return exactly `1024` dimensions.
|
||||
|
||||
Direct and SSH vector checks connect to the *vector* `database` and `schema`, not the DWH
|
||||
identity. They inspect the declared collection's vector column and index and must find the exact
|
||||
collection, integer `dimensions`, and `distance` (`cosine`, `l2`, or `inner_product`) declared in
|
||||
`semantic_index.vector_store`.
|
||||
|
||||
Their optional `*_TLS_CA_FILE` follows the same verified private-CA-or-system-trust rule as the
|
||||
DWH diagnostic. For an SSH tunnel, their TLS server name is likewise the declared vector
|
||||
`SSH_TARGET_HOST`, not the loopback listener.
|
||||
|
||||
For REST, the exact descriptor-declared request is, for example:
|
||||
|
||||
```text
|
||||
GET <THT_WS_<NAMESPACE>_VECTOR_BASE_URL>/vector/metadata
|
||||
Authorization: Bearer <content of VECTOR_API_KEY_FILE>
|
||||
```
|
||||
|
||||
It has no request body. A 2xx JSON object must supply the declared `collection`, `dimensions`, and
|
||||
`distance` fields. All three values must exactly match the vector-store contract; an integer
|
||||
dimension is required. Metadata from a similarly named collection, a different metric, or a
|
||||
different dimension makes the semantic index incompatible.
|
||||
|
||||
## Reversible vector writer probe
|
||||
|
||||
Ordinary validation is reader-only. A write probe runs only when all of the following are true:
|
||||
|
||||
1. The operator explicitly requests it.
|
||||
2. The descriptor has `semantic_index.vector_writer: {}`.
|
||||
3. The descriptor declares an authenticated `diagnostics.vector_rest.reversible_probe` with an
|
||||
`operation` response field.
|
||||
4. The selected vector transport is `rest_api`.
|
||||
5. `THT_WS_<NAMESPACE>_VECTOR_WRITER_API_KEY_FILE` exists locally and is distinct from the reader
|
||||
API-key file.
|
||||
|
||||
The probe uses the declared `POST` endpoint twice, with the same generated ID and the writer key:
|
||||
|
||||
```json
|
||||
{ "operation": "create", "id": "diagnostic:<random UUID>", "collection": "<declared collection>", "dimensions": 768 }
|
||||
```
|
||||
|
||||
then:
|
||||
|
||||
```json
|
||||
{ "operation": "remove", "id": "diagnostic:<same UUID>", "collection": "<declared collection>" }
|
||||
```
|
||||
|
||||
Both requests require a 2xx JSON response whose declared `operation` field equals the requested
|
||||
`create` or `remove` operation. Cleanup is attempted in `finally`, including after a write timeout
|
||||
or error. The endpoint must implement both operations as a bounded, reversible diagnostic
|
||||
operation; an upsert-only endpoint is prohibited. It must not retain, index, or expose diagnostic
|
||||
records. If the writer capability or its local binding is absent, validation remains reader-only
|
||||
and no write request is sent.
|
||||
|
||||
## Embedding dimensions diagnostic
|
||||
|
||||
The embedding request is descriptor-declared, for example:
|
||||
|
||||
```text
|
||||
GET <THT_WS_<NAMESPACE>_EMBEDDING_BASE_URL>/models
|
||||
```
|
||||
|
||||
It has no body and uses the declared authentication mode. A 2xx JSON object must contain the
|
||||
declared model field equal to `semantic_index.embedding.model` and a declared dimensions field that
|
||||
is an integer. That integer must equal both `semantic_index.embedding.dimensions` and
|
||||
`semantic_index.vector_store.dimensions`.
|
||||
These checks use the private Compose services and never require operator-supplied vector or
|
||||
embedding URLs, transports, or credentials.
|
||||
|
||||
## SSH host verification and tunnel lifecycle
|
||||
|
||||
For either DWH or vector `ssh_tunnel`, the known-hosts file is mandatory and is verified before a
|
||||
connection is accepted. The tunnel is a short-lived loopback forward for the diagnostic only. The
|
||||
effective OpenSSH constraints are:
|
||||
For DWH `ssh_tunnel`, the known-hosts file is mandatory and is verified before a connection is
|
||||
accepted. The tunnel is a short-lived loopback forward for the diagnostic only. The effective
|
||||
OpenSSH constraints are:
|
||||
|
||||
```text
|
||||
-N -v
|
||||
@@ -224,19 +133,16 @@ effective OpenSSH constraints are:
|
||||
```
|
||||
|
||||
The local listener is `127.0.0.1` only. The process is terminated in cleanup after the direct
|
||||
probe, on timeout, or on failure. There is no accept-new mode, no disabled host-key checking, and
|
||||
no persistent forwarding.
|
||||
probe, on timeout, or on failure.
|
||||
|
||||
In this release, `ssh_tunnel` is therefore a diagnostic-only connector transport. A successful
|
||||
probe is followed by `workspace_not_activatable`, and `POST /sessions` rejects the workspace before
|
||||
persisting a manifest or starting Pi. Use direct PostgreSQL/pgvector or REST for runtime sessions
|
||||
until the backend owns a tunnel for the full runtime lifecycle. This restriction does not apply to
|
||||
using SSH as the transport for the workspace Git remote.
|
||||
In this release, `ssh_tunnel` remains a diagnostic-only DWH transport. A successful probe is
|
||||
followed by `workspace_not_activatable`, and `POST /sessions` rejects the workspace before
|
||||
persisting a manifest or starting Pi. This restriction does not apply to SSH transport for the
|
||||
workspace Git remote.
|
||||
|
||||
## Reader-only fallback
|
||||
|
||||
A workspace may be fully valid in Git but non-activatable locally when a required reader binding,
|
||||
secret file, host verification, TLS check, or declared diagnostic fails. That state does not alter
|
||||
the shared descriptor and does not permit a new session on that installation. It may still be
|
||||
published and activated elsewhere with valid local bindings. Missing optional writer capability is
|
||||
not a reader failure: it leaves the workspace in reader-only mode and suppresses the writer probe.
|
||||
A workspace may be fully valid in Git but non-activatable locally when a required DWH binding,
|
||||
secret file, host verification, TLS check, or declared DWH diagnostic fails. That state does not
|
||||
alter the shared descriptor and does not permit a new session on that installation. It may still
|
||||
be published and activated elsewhere with valid local bindings.
|
||||
|
||||
Reference in New Issue
Block a user