fix: scope workspace registry smoke cleanup
This commit is contained in:
@@ -122,3 +122,54 @@ Final audit:
|
|||||||
- Broad harness Ruff remains existing unrelated debt: `Found 220 errors`.
|
- Broad harness Ruff remains existing unrelated debt: `Found 220 errors`.
|
||||||
- Final active-reference audit is not clean; it still finds legacy/negative-guard references outside explicit migration fixture files.
|
- Final active-reference audit is not clean; it still finds legacy/negative-guard references outside explicit migration fixture files.
|
||||||
- Ephemeral Task 13 core/frontend image IDs from `internal-semantic-smoke.sh`, `unified-deployment-smoke.sh`, `thothctl-update-smoke.sh`, and `server-deployment-smoke.sh` were removed by exact cleanup and were not emitted in stdout; pinned Qdrant/Ollama digests and the workspace-registry smoke image digest were captured.
|
- Ephemeral Task 13 core/frontend image IDs from `internal-semantic-smoke.sh`, `unified-deployment-smoke.sh`, `thothctl-update-smoke.sh`, and `server-deployment-smoke.sh` were removed by exact cleanup and were not emitted in stdout; pinned Qdrant/Ollama digests and the workspace-registry smoke image digest were captured.
|
||||||
|
|
||||||
|
## Fix Round 1 — reviewer findings
|
||||||
|
|
||||||
|
Status: DONE
|
||||||
|
|
||||||
|
Changes:
|
||||||
|
|
||||||
|
- `scripts/workspace-registry-smoke.sh` now derives the smoke image reference from the already unique Compose project instead of using the global tag `thothii-workspace-registry-smoke:local`.
|
||||||
|
- The workspace-registry cleanup helpers remove and verify only the exact per-run image reference, plus Compose resources labeled with the exact project.
|
||||||
|
- Added deterministic self-test coverage in `backend/test/workspaces-migrate-legacy.test.ts` via `WORKSPACE_REGISTRY_SMOKE_SELF_TEST=image-cleanup-identity`; it stubs Docker and fails if cleanup touches same-repository foreign tags such as `:local` or another project tag.
|
||||||
|
- Updated active harness/testing/PRD docs and Python comments that still described the current semantic store as pgvector/vectordb. Preserved schema-v1/v2 and harness legacy compatibility fixtures.
|
||||||
|
- Updated `PROJECT_STATE.md` with fix-round smoke evidence and a precise, non-overclaiming audit limitation.
|
||||||
|
|
||||||
|
Focused verification:
|
||||||
|
|
||||||
|
- `cd backend && npx vitest run test/workspaces-migrate-legacy.test.ts`
|
||||||
|
- Passed: `7 passed`.
|
||||||
|
- `cd harness && .venv/bin/pytest -q tests/test_memory_save_one.py tests/test_adapter_command_regressions.py tests/test_solved_search_cli.py tests/test_search_pack.py`
|
||||||
|
- Passed: `22 passed, 14 warnings`.
|
||||||
|
- `cd harness && .venv/bin/ruff check tht/memory.py tht/search/__init__.py tht/workspace.py tht/vectorstore/store.py tests/test_memory_save_one.py tests/test_adapter_command_regressions.py tests/test_solved_search_cli.py`
|
||||||
|
- Passed: `All checks passed!`
|
||||||
|
- `bash -n scripts/workspace-registry-smoke.sh && WORKSPACE_REGISTRY_SMOKE_SELF_TEST=image-cleanup-identity bash scripts/workspace-registry-smoke.sh`
|
||||||
|
- Passed: `workspace registry smoke image cleanup identity self-test passed`.
|
||||||
|
- `./scripts/test-no-deployment-coupling.sh`
|
||||||
|
- Passed: `no active retired deployment or external semantic coupling found.`
|
||||||
|
- `./scripts/verify-workspace-install-docs.sh --fixtures-only`
|
||||||
|
- Passed through `relative secret-source fixture rejected passed`.
|
||||||
|
- `cd backend && npx tsc --noEmit -p .`
|
||||||
|
- Passed with no output.
|
||||||
|
- `/usr/bin/time -p ./scripts/workspace-registry-smoke.sh`
|
||||||
|
- Passed: `workspace registry smoke passed`.
|
||||||
|
- Built exact per-run tag: `thothii-workspace-registry-smoke:thoth-workspace-registry-smoke-thoth-workspace-registry-smoke-10vi3a-19157`.
|
||||||
|
- Manifest list: `sha256:715b943057929418cad4aa71806d9edbaf823555d19bda6b875297617463fd4a`.
|
||||||
|
- Config: `sha256:a566521981e08958aae9a12bfc7803bb5f3f835536b4bb8c39df8fcf26063161`.
|
||||||
|
- Cleanup proof: `no compose containers, volumes, networks, or image remain for thoth-workspace-registry-smoke-thoth-workspace-registry-smoke-10vi3a-19157.`
|
||||||
|
- Duration: `real 42.06`.
|
||||||
|
|
||||||
|
Fix-round audit command:
|
||||||
|
|
||||||
|
- `rg -n "pgvector|local-vector|THT_VECTOR_|EMBEDDING_BASE_URL|openai_compatible|ollama_compatible" . --glob '!docs/plans/**' --glob '!docs/superpowers/**' --glob '!**/node_modules/**' --glob '!**/.venv/**' --glob '!**/.git/**'`
|
||||||
|
|
||||||
|
Categorized remaining hits:
|
||||||
|
|
||||||
|
- Backend legacy parser/migration compatibility, kept deliberately non-operational for schema-v1/v2 descriptors: `backend/src/workspaces/schema.ts`, `types.ts`, `migrate-legacy.ts`, `runtime-renderer.ts`, `bindings.ts`, `contracts.ts`, `diagnostics.ts`.
|
||||||
|
- Backend negative guards and legacy fixture tests: `backend/test/workspaces-schema.test.ts`, `workspaces-migrate-v2-qdrant.test.ts`, `workspace-registry.test.ts`, `workspace-runtime-renderer.test.ts`, `workspaces-bindings.test.ts`, `workspaces-contracts.test.ts`, `workspaces-diagnostics.test.ts`, `workspaces-git-repository.test.ts`, `routes-workspaces.test.ts`, `routes-sessions.test.ts`, `provider-credentials.test.ts`.
|
||||||
|
- Secret/env scrub guards for retired variables: `backend/src/config.ts`, `backend/src/config/secret-bundle.ts`, `backend/src/pi/provider-credentials.ts`, `scripts/compose-with-preflight.sh`, `scripts/test-external-compose-lifecycle.sh`.
|
||||||
|
- Deployment negative guards and fixture-scope tests: `scripts/test-no-deployment-coupling.sh`, `scripts/test-no-deployment-coupling-scope.sh`, `scripts/test-preprocess-compose-config.sh`, `scripts/test-verify-workspace-install-docs.sh`, `scripts/verify-workspace-install-docs.sh`, `scripts/vector-rotate-bootstrap-password.sh`.
|
||||||
|
- Harness legacy config compatibility and fixtures: `harness/tht/config.py`, `harness/tht/config_compat.py`, `harness/tests/test_config_resources.py`, `harness/tests/l2/test_session_ablazione.py`, `harness/workspaces/tht.example.yaml`, `harness/workspaces/tht-test.yaml`.
|
||||||
|
- Retained off-repository migration SQL fixtures: `harness/scripts/create_vector_reader_rpc.sql`, `harness/scripts/create_vector_writer_rpc.sql`.
|
||||||
|
- Historical/reference notes, not active operator contracts: `brain/codebase/datamart-builder-deployment-gotchas.md`, `PROJECT_STATE.md`.
|
||||||
|
- Gitignored task report self-reference: `.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-13-implementation.md`.
|
||||||
|
|||||||
+13
-8
@@ -37,23 +37,28 @@
|
|||||||
`verify-workspace-install-docs.sh --fixtures-only`.
|
`verify-workspace-install-docs.sh --fixtures-only`.
|
||||||
- **Task 13 Docker smoke evidence.** CPU semantic smoke passed in **217.34s** and proved
|
- **Task 13 Docker smoke evidence.** CPU semantic smoke passed in **217.34s** and proved
|
||||||
offline Qdrant/Ollama persistence plus exact cleanup. Workspace registry smoke passed in
|
offline Qdrant/Ollama persistence plus exact cleanup. Workspace registry smoke passed in
|
||||||
**9.93s** and now proves exact cleanup of compose containers, volumes, networks, and its
|
**42.06s** in fix round 1 with a per-run image tag derived from the unique Compose project,
|
||||||
smoke image. Unified deployment smoke passed in **125.57s**; update-only rollback smoke
|
and proves exact cleanup of compose containers, volumes, networks, and only that smoke image.
|
||||||
|
Unified deployment smoke passed in **125.57s**; update-only rollback smoke
|
||||||
passed in **85.40s**; Linux server deployment smoke passed in **55.99s**. The previously
|
passed in **85.40s**; Linux server deployment smoke passed in **55.99s**. The previously
|
||||||
observed `thothctl` rollback failure did not recur.
|
observed `thothctl` rollback failure did not recur.
|
||||||
- **Task 13 image and manual-gate notes.** Verified pinned runtime images:
|
- **Task 13 image and manual-gate notes.** Verified pinned runtime images:
|
||||||
`qdrant/qdrant:v1.18.2@sha256:75eab8c4ba42096724fdcfde8b4de0b5713d529dde32f285a1f86fdcb2c9e50c`
|
`qdrant/qdrant:v1.18.2@sha256:75eab8c4ba42096724fdcfde8b4de0b5713d529dde32f285a1f86fdcb2c9e50c`
|
||||||
and
|
and
|
||||||
`ollama/ollama:0.32.0@sha256:57f573b47f1f71ebb445789f279fe3e596a8beab182f7cf486db9205bad87c5a`.
|
`ollama/ollama:0.32.0@sha256:57f573b47f1f71ebb445789f279fe3e596a8beab182f7cf486db9205bad87c5a`.
|
||||||
The workspace-registry smoke built ephemeral manifest list
|
The workspace-registry smoke fix-round image used tag
|
||||||
`sha256:4d056bf2cb38d0e8ede91fbf121df1f9f18caee0d401581618ccef9ed8a55e73`
|
`thothii-workspace-registry-smoke:thoth-workspace-registry-smoke-thoth-workspace-registry-smoke-10vi3a-19157`,
|
||||||
and removed it during cleanup. Local GPU exposure (`THOTH_ENABLE_EMBEDDING_GPU=1`) and
|
built manifest list `sha256:715b943057929418cad4aa71806d9edbaf823555d19bda6b875297617463fd4a`
|
||||||
|
with config `sha256:a566521981e08958aae9a12bfc7803bb5f3f835536b4bb8c39df8fcf26063161`,
|
||||||
|
and removed that exact reference during cleanup. Local GPU exposure (`THOTH_ENABLE_EMBEDDING_GPU=1`) and
|
||||||
Windows Docker Desktop startup were not manually executed in this run.
|
Windows Docker Desktop startup were not manually executed in this run.
|
||||||
- **Task 13 known limitations.** Broad harness Ruff remains existing unrelated debt
|
- **Task 13 known limitations.** Broad harness Ruff remains existing unrelated debt
|
||||||
(**220 errors**); touched harness files were verified Ruff-clean. The final active-reference
|
(**220 errors**); touched harness files were verified Ruff-clean. The final active-reference
|
||||||
audit still reports legacy schema-v1/v2 parsing/migration, negative guards, and historical
|
audit remains non-empty only in categorized legacy parser/migration compatibility, legacy
|
||||||
notes containing `pgvector`, `THT_VECTOR_*`, `EMBEDDING_BASE_URL`, `openai_compatible`, or
|
descriptor/config fixtures, deterministic negative guards, retained off-repository migration
|
||||||
`ollama_compatible`; those hits were not all eliminated by this verification task.
|
SQL, L2 legacy fixtures, gitignored task notes, and historical reference notes. No active
|
||||||
|
schema-v3 operator manual or supported runtime deployment path retains external vector or
|
||||||
|
embedding endpoint coupling.
|
||||||
|
|
||||||
## Historical snapshots and archived reference notes
|
## Historical snapshots and archived reference notes
|
||||||
|
|
||||||
|
|||||||
@@ -1,3 +1,4 @@
|
|||||||
|
import { execFileSync } from "node:child_process";
|
||||||
import { existsSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
import { existsSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||||
import { mkdtemp } from "node:fs/promises";
|
import { mkdtemp } from "node:fs/promises";
|
||||||
import { tmpdir } from "node:os";
|
import { tmpdir } from "node:os";
|
||||||
@@ -100,3 +101,13 @@ test("declares a durable isolated registry volume and only read-only Git credent
|
|||||||
expect(smoke).toContain('"degraded":true');
|
expect(smoke).toContain('"degraded":true');
|
||||||
expect(smoke).toContain('core_remote="/fixtures/remote.git"');
|
expect(smoke).toContain('core_remote="/fixtures/remote.git"');
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test("workspace registry smoke image cleanup is scoped to the per-run image identity", () => {
|
||||||
|
const output = execFileSync("bash", ["scripts/workspace-registry-smoke.sh"], {
|
||||||
|
cwd: new URL("../..", import.meta.url),
|
||||||
|
env: { ...process.env, WORKSPACE_REGISTRY_SMOKE_SELF_TEST: "image-cleanup-identity" },
|
||||||
|
encoding: "utf8",
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(output).toContain("workspace registry smoke image cleanup identity self-test passed");
|
||||||
|
});
|
||||||
|
|||||||
+8
-5
@@ -22,16 +22,19 @@ The `tht` command is now on PATH. Node ≥ 20 is needed for the gate JS tests
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
cp .env.example .env
|
cp .env.example .env
|
||||||
# fill in: THT_PROFILE, THT_DB_*, THT_DWH_API_KEY, THT_VEC_API_KEY,
|
# fill in: THT_PROFILE, THT_DB_*, THT_DWH_API_KEY, THT_SSL_CA, ...
|
||||||
# THT_VEC_WRITE_API_KEY, THT_SSL_CA, THT_OLLAMA_URL, ...
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Keys are never logged; URLs are fine. Rotate any key that appeared in chat.
|
Keys are never logged; URLs are fine. Rotate any key that appeared in chat.
|
||||||
|
|
||||||
### `workspaces/<name>.yaml`
|
### `workspaces/<name>.yaml`
|
||||||
|
|
||||||
A workspace wires the relational DWH + the pgvector (dual-key) + embeddings + evidence.
|
A schema-v3 workspace wires the external relational DWH to one internal Qdrant collection
|
||||||
See `workspaces/tht.example.yaml`. `${THT_*}}` tokens expand from `.env`.
|
and the internal Ollama embedding model. Evidence paths remain workspace-local, while Qdrant
|
||||||
|
stores the derived semantic projection for schema, Evidence, Memory, and solved-question
|
||||||
|
records. The legacy files under `workspaces/` are retained as migration fixtures; new
|
||||||
|
operator-facing descriptors live in the workspace Git registry. `${THT_*}` tokens expand
|
||||||
|
from `.env`.
|
||||||
|
|
||||||
> **DB support (MVP):** the `direct` transport supports **PostgreSQL only** (psycopg2
|
> **DB support (MVP):** the `direct` transport supports **PostgreSQL only** (psycopg2
|
||||||
> driver, `pg_*` catalog introspection, postgres-dialect sqlcheck/EXPLAIN). The central
|
> driver, `pg_*` catalog introspection, postgres-dialect sqlcheck/EXPLAIN). The central
|
||||||
@@ -130,7 +133,7 @@ tht/ Python package (CLI + workflow + phase + decisions + db/res
|
|||||||
.pi/ Pi project (settings, prompts, themes, extensions/tht-gate.js + gate/)
|
.pi/ Pi project (settings, prompts, themes, extensions/tht-gate.js + gate/)
|
||||||
workflow.yaml single source of workflow truth (F2)
|
workflow.yaml single source of workflow truth (F2)
|
||||||
workspaces/ workspace YAML definitions (D3)
|
workspaces/ workspace YAML definitions (D3)
|
||||||
scripts/ reader/writer RPC SQL for pgvector (D11)
|
scripts/ retained legacy SQL fixtures and workspace utilities
|
||||||
tests/ L0 (testcontainers), L1 (logic + builders), L2 (real model + DB)
|
tests/ L0 (testcontainers), L1 (logic + builders), L2 (real model + DB)
|
||||||
docs/ testing guide + workflow editing
|
docs/ testing guide + workflow editing
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -59,20 +59,22 @@ the anti-bypass hooks. The glue depends on the Pi runtime (`pi.registerTool`,
|
|||||||
|
|
||||||
## L2 — real LLM + real remote DB, manual / pre-release (NOT automated)
|
## L2 — real LLM + real remote DB, manual / pre-release (NOT automated)
|
||||||
|
|
||||||
**What:** end-to-end sessions with GLM 5.2 + the real Chirone DWH + pgvector, reached
|
**What:** end-to-end sessions with GLM 5.2 + the real Chirone DWH, plus the internal
|
||||||
via REST over VPN. Plus the gate-glue validation (the part L1 cannot reach).
|
Qdrant/Ollama semantic services started by the ThothII stack. Plus the gate-glue
|
||||||
|
validation (the part L1 cannot reach).
|
||||||
|
|
||||||
**Dependencies (all required, skip cleanly if missing):**
|
**Dependencies (all required, skip cleanly if missing):**
|
||||||
- LLM: Pi configured locally with GLM 5.2.
|
- LLM: Pi configured locally with GLM 5.2.
|
||||||
- DB: the remote Supabase endpoints (DWH read-only + pgvector reader/writer), via VPN.
|
- DB: the remote DWH endpoint, via VPN when required.
|
||||||
- `harness/.env` populated with the API keys + CA path.
|
- ThothII stack: internal Qdrant and Ollama services reachable from `core`.
|
||||||
|
- `harness/.env` populated with the required DWH/model API keys + CA path.
|
||||||
|
|
||||||
**Coverage (honest):** validates the assumption L1 cannot — that GLM 5.2 produces tool
|
**Coverage (honest):** validates the assumption L1 cannot — that GLM 5.2 produces tool
|
||||||
calls the gate accepts, that the skill's prompts lead to the expected interaction shape,
|
calls the gate accepts, that the skill's prompts lead to the expected interaction shape,
|
||||||
that the gate glue handles real tool-call sequences (incl. Altro/Rifiuta/rollback),
|
that the gate glue handles real tool-call sequences (incl. Altro/Rifiuta/rollback),
|
||||||
that value grounding and formula approval surface correctly on the real schema, that
|
that value grounding and formula approval surface correctly on the real schema, that
|
||||||
`memory save-one` upserts to the real pgvector. **Closes the skill→LLM→gate loop AND
|
`memory save-one` upserts to the configured semantic store. **Closes the
|
||||||
exercises the gate glue.**
|
skill→LLM→gate loop AND exercises the gate glue.**
|
||||||
|
|
||||||
**Honest limitation:** L2 is non-deterministic (the model may behave differently across
|
**Honest limitation:** L2 is non-deterministic (the model may behave differently across
|
||||||
runs) and slow/costly. It is a **pre-release safety net, not a regression gate**.
|
runs) and slow/costly. It is a **pre-release safety net, not a regression gate**.
|
||||||
|
|||||||
@@ -62,8 +62,8 @@ def test_memory_command_writes_through_factory_vector_store(monkeypatch):
|
|||||||
captured = []
|
captured = []
|
||||||
original_upsert = store.upsert
|
original_upsert = store.upsert
|
||||||
store.upsert = lambda table, rows: captured.extend(rows) or original_upsert(table, rows)
|
store.upsert = lambda table, rows: captured.extend(rows) or original_upsert(table, rows)
|
||||||
# Server deployments write directly to pgvector and intentionally do not
|
# Legacy server deployments wrote directly through the factory and intentionally
|
||||||
# configure the workstation-only REST writer key.
|
# did not configure the workstation-only REST writer key.
|
||||||
cfg = SimpleNamespace(profile="server", embeddings=object(), vector_write_rest=None)
|
cfg = SimpleNamespace(profile="server", embeddings=object(), vector_write_rest=None)
|
||||||
manifest = SimpleNamespace(id="s1")
|
manifest = SimpleNamespace(id="s1")
|
||||||
snapshot = SimpleNamespace(manifest=manifest, decisions=[], artifacts={})
|
snapshot = SimpleNamespace(manifest=manifest, decisions=[], artifacts={})
|
||||||
|
|||||||
@@ -1,8 +1,8 @@
|
|||||||
"""L1: tht memory save-one -- targeted upsert via the writer key (spec D11).
|
"""L1: tht memory save-one -- targeted upsert via the writer key (spec D11).
|
||||||
|
|
||||||
The D11 deviation: instead of a full vectorstore resync (tht memory index / sync),
|
The D11 deviation: instead of a full vectorstore resync (tht memory index / sync),
|
||||||
a remote workstation with a writer key can push a SINGLE promoted decision to
|
the workflow can push a SINGLE promoted decision to the configured semantic store as
|
||||||
pgvector as a one-row upsert. This test pins the pure core of that behavior:
|
a one-row upsert. This test pins the pure core of that behavior:
|
||||||
- exactly one VectorRecord is built for the chosen decision_seq
|
- exactly one VectorRecord is built for the chosen decision_seq
|
||||||
- the writer.upsert_records is called once with a single row
|
- the writer.upsert_records is called once with a single row
|
||||||
- writer.sync is NEVER called (that is the full-resync path)
|
- writer.sync is NEVER called (that is the full-resync path)
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
"""L1: `tht memory solved-search` — degrado gentile e mapping dei risultati.
|
"""L1: `tht memory solved-search` — degrado gentile e mapping dei risultati.
|
||||||
|
|
||||||
SKILL.md prescrive solved-search in F4/F6/F7 di OGNI sessione: a vectordb
|
SKILL.md prescrive solved-search in F4/F6/F7 di OGNI sessione: se lo store
|
||||||
irraggiungibile (VPN giu', Ollama spento) il comando non deve morire con un
|
semantico è irraggiungibile (Qdrant/Ollama non disponibili) il comando non deve morire con un
|
||||||
traceback grezzo ma degradare a un avviso di una riga su stderr, con stdout
|
traceback grezzo ma degradare a un avviso di una riga su stderr, con stdout
|
||||||
puro (`[]` in modalita' --json) ed exit 0, cosi' il modello prosegue senza
|
puro (`[]` in modalita' --json) ed exit 0, cosi' il modello prosegue senza
|
||||||
exemplar. Il finalize-hook gestisce gia' lo stesso scenario in modo analogo.
|
exemplar. Il finalize-hook gestisce gia' lo stesso scenario in modo analogo.
|
||||||
|
|||||||
@@ -299,10 +299,10 @@ def memory_vector_record_for_decision(
|
|||||||
def save_one_memory(
|
def save_one_memory(
|
||||||
records: list[MemoryRecord], decision_seq: int, *, store, embedder
|
records: list[MemoryRecord], decision_seq: int, *, store, embedder
|
||||||
) -> int:
|
) -> int:
|
||||||
"""Targeted one-row upsert of a promoted decision to pgvector via the writer key
|
"""Targeted one-row upsert of a promoted decision to the configured semantic store
|
||||||
(spec D11). This is NOT a full vectorstore resync: it embeds and pushes a single
|
(spec D11). This is NOT a full vectorstore resync: it embeds and pushes a single
|
||||||
record, so a workstation with a writer key can publish one memory without
|
record, so a memory can be published without rebuilding the index.
|
||||||
rebuilding the index. Returns the upsert count (0 if no record matched or the
|
Returns the upsert count (0 if no record matched or the
|
||||||
record is already up to date).
|
record is already up to date).
|
||||||
|
|
||||||
Hash dedup client-side (spec §5.4): the SHA-256 of the content is compared with
|
Hash dedup client-side (spec §5.4): the SHA-256 of the content is compared with
|
||||||
|
|||||||
@@ -59,8 +59,8 @@ def aggregate_lsh_multi(hits: list[dict]) -> dict[str, list[dict]]:
|
|||||||
grouped: dict[str, list[dict]] = {}
|
grouped: dict[str, list[dict]] = {}
|
||||||
for (table, _), row in best.items():
|
for (table, _), row in best.items():
|
||||||
grouped.setdefault(table, []).append(row)
|
grouped.setdefault(table, []).append(row)
|
||||||
for table in grouped:
|
for rows in grouped.values():
|
||||||
grouped[table].sort(key=lambda r: r["score"], reverse=True)
|
rows.sort(key=lambda r: r["score"], reverse=True)
|
||||||
return grouped
|
return grouped
|
||||||
|
|
||||||
|
|
||||||
@@ -99,7 +99,7 @@ def combined_search(
|
|||||||
kinds: list[str] | None,
|
kinds: list[str] | None,
|
||||||
query_vec: list[float] | None = None,
|
query_vec: list[float] | None = None,
|
||||||
) -> list[SearchResult]:
|
) -> list[SearchResult]:
|
||||||
"""Fonde LSH (valori di campo) e pgvector con Reciprocal Rank Fusion.
|
"""Fonde LSH (valori di campo) e ricerca semantica con Reciprocal Rank Fusion.
|
||||||
|
|
||||||
`query_vec` permette di riusare un embedding gia' calcolato della stessa
|
`query_vec` permette di riusare un embedding gia' calcolato della stessa
|
||||||
keyword (es. `tht search pack`, che fa piu' ricerche sulla stessa domanda)."""
|
keyword (es. `tht search pack`, che fa piu' ricerche sulla stessa domanda)."""
|
||||||
|
|||||||
@@ -52,11 +52,13 @@ def hit_from_metadata(similarity: float, metadata: dict | None) -> VectorHit:
|
|||||||
|
|
||||||
|
|
||||||
class VectorStore:
|
class VectorStore:
|
||||||
"""Tabella pgvector table-scoped: scrittura diretta (loading) su una tabella dello schema
|
"""Legacy table-scoped vector store retained for compatibility fixtures.
|
||||||
`vectors`. La lettura via REST avviene su `search_similar`; questo store serve al loading e
|
|
||||||
alla lettura diretta (dev/test). Il contratto della tabella remota richiede `id` (BIGSERIAL),
|
New operational semantic storage is handled by the Qdrant adapter. This class preserves
|
||||||
`embedding vector(N)` e `metadata jsonb`; le colonne extra (`record_key`, `kind`,
|
the older SQL-table contract used by historical tests and migration checks: `id`
|
||||||
`content_hash`) servono solo al loader e non sono esposte dalla REST."""
|
(BIGSERIAL), `embedding vector(N)`, and `metadata jsonb`; the extra columns
|
||||||
|
(`record_key`, `kind`, `content_hash`) serve only the loader and are not exposed by REST.
|
||||||
|
"""
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
self, engine: Engine, schema: str = "vectors", table: str = "records", dim: int = 768
|
self, engine: Engine, schema: str = "vectors", table: str = "records", dim: int = 768
|
||||||
|
|||||||
@@ -4,8 +4,9 @@ Reads workspaces/<name>.yaml, expands ${VAR} from env, validates via the Config
|
|||||||
(ported from the reference implementation). Future migration to a DB store would replace only this module.
|
(ported from the reference implementation). Future migration to a DB store would replace only this module.
|
||||||
|
|
||||||
La struttura YAML rispecchia esattamente tht/config.py:
|
La struttura YAML rispecchia esattamente tht/config.py:
|
||||||
database + rest (DWH), vector_rest + vector_write_rest (pgvector, doppia key top-level),
|
database/rest o resources.dwh per il DWH, resources.vector/resources.embeddings
|
||||||
vector_db (loading diretto, server-only), embeddings, evidence, execution.
|
per Qdrant/Ollama interni, più evidence ed execution. I vecchi campi vector_db e
|
||||||
|
vector_rest restano solo per leggere fixture legacy durante la migrazione.
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
|||||||
+2
-2
@@ -23,10 +23,10 @@ L'aderenza al workflow delineato in ./ChironeWp3 deve essere stretta in quanto f
|
|||||||
L'applicazione, come già fa quella attualmente sviluppata, può contare su tre risorse disponibili collegandosi al server di produzione:
|
L'applicazione, come già fa quella attualmente sviluppata, può contare su tre risorse disponibili collegandosi al server di produzione:
|
||||||
|
|
||||||
- un Supabase contenente il datawarehouse per cui si vuole generare il SQL
|
- un Supabase contenente il datawarehouse per cui si vuole generare il SQL
|
||||||
- un pgvector, contenuto anch'esso nel Supabase, che contiene gli embeddings dei documenti che descrivono il datawarehouse
|
- un indice semantico interno a ThothII, basato su Qdrant nel Docker Compose applicativo, che contiene gli embeddings dei documenti che descrivono il datawarehouse, delle evidence e delle memory
|
||||||
- un LLM (qwen 3.6 - 35B) utilizzabile da Pi che gira sulle GPU del server di produzione, ed è quindi gratuito
|
- un LLM (qwen 3.6 - 35B) utilizzabile da Pi che gira sulle GPU del server di produzione, ed è quindi gratuito
|
||||||
|
|
||||||
all'interno del progetto ./ChironeWp3 vi sono già tutti gli elementi necessari per gestire la connessione col datawarehouse del policlinicosandonato, ma ThothII deve potersi interfacciare con qualunque database e con un pgvector locale nel caso non sia disponibile un pgvector remoto. Per cui deve essere previsto un insime di configurazioni destinate a implementare il concetto di workspace composto da db relazionale + pgvector (locale o remoto) su cui operare prevedendo diverse modalità di accesso (REST, tunnel ssh, accesso diretto) e diverse tipologie di db relazionale (posthres, sqlserver, mariadb ed informix innanzitutto)
|
all'interno del progetto ./ChironeWp3 vi sono già tutti gli elementi necessari per gestire la connessione col datawarehouse del policlinicosandonato, ma ThothII deve potersi interfacciare con qualunque database esterno mantenendo invece il vector DB e gli embeddings interni all'applicazione. Per cui deve essere previsto un insieme di configurazioni destinate a implementare il concetto di workspace composto da db relazionale esterno + collection Qdrant interna su cui operare prevedendo diverse modalità di accesso al DB (REST, tunnel ssh, accesso diretto) e diverse tipologie di db relazionale (postgres, sqlserver, mariadb ed informix innanzitutto)
|
||||||
|
|
||||||
Per quanto riguarda il collegamento ad un database qualunque trovi in ./Thoth/thoth_sqldb2 del codice a cui potersi ispirarsi per l'implementazione di un modulo di connessione a database generico.
|
Per quanto riguarda il collegamento ad un database qualunque trovi in ./Thoth/thoth_sqldb2 del codice a cui potersi ispirarsi per l'implementazione di un modulo di connessione a database generico.
|
||||||
|
|
||||||
|
|||||||
@@ -6,27 +6,87 @@ set -euo pipefail
|
|||||||
|
|
||||||
root="$(cd "$(dirname "$0")/.." && pwd -P)"
|
root="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
tmp="$(mktemp -d "${TMPDIR:-/tmp}/thoth-workspace-registry-smoke.XXXXXX")"
|
tmp="$(mktemp -d "${TMPDIR:-/tmp}/thoth-workspace-registry-smoke.XXXXXX")"
|
||||||
project="thoth-workspace-registry-smoke-$$"
|
tmp_slug="$(basename "$tmp" | tr '[:upper:]._' '[:lower:]--' | tr -cd 'a-z0-9-')"
|
||||||
image="thothii-workspace-registry-smoke:local"
|
project="thoth-workspace-registry-smoke-${tmp_slug}-$$"
|
||||||
|
image="thothii-workspace-registry-smoke:${project}"
|
||||||
remote="$tmp/remote.git"
|
remote="$tmp/remote.git"
|
||||||
seed="$tmp/seed"
|
seed="$tmp/seed"
|
||||||
branch="workspace-registry-smoke"
|
branch="workspace-registry-smoke"
|
||||||
core_remote="/fixtures/remote.git"
|
core_remote="/fixtures/remote.git"
|
||||||
|
|
||||||
cleanup() {
|
cleanup_smoke_image() {
|
||||||
local cleanup_status=$?
|
|
||||||
compose down --volumes --remove-orphans >/dev/null 2>&1 || true
|
|
||||||
docker image rm -f "$image" >/dev/null 2>&1 || true
|
docker image rm -f "$image" >/dev/null 2>&1 || true
|
||||||
if [[ "$cleanup_status" -eq 0 ]]; then
|
}
|
||||||
local leftovers
|
|
||||||
leftovers="$(
|
workspace_registry_smoke_leftovers() {
|
||||||
{
|
{
|
||||||
docker ps -a --filter "label=com.docker.compose.project=$project" -q
|
docker ps -a --filter "label=com.docker.compose.project=$project" -q
|
||||||
docker volume ls --filter "label=com.docker.compose.project=$project" -q
|
docker volume ls --filter "label=com.docker.compose.project=$project" -q
|
||||||
docker network ls --filter "label=com.docker.compose.project=$project" -q
|
docker network ls --filter "label=com.docker.compose.project=$project" -q
|
||||||
docker image ls -q "$image"
|
docker image inspect --format '{{.Id}}' "$image" 2>/dev/null || true
|
||||||
} | sed '/^$/d'
|
} | sed '/^$/d'
|
||||||
)"
|
}
|
||||||
|
|
||||||
|
workspace_registry_smoke_self_test_image_cleanup_identity() {
|
||||||
|
local calls exact_image foreign_project foreign_tag leftovers
|
||||||
|
calls="$(mktemp "${TMPDIR:-/tmp}/thoth-workspace-registry-smoke-image-contract.XXXXXX")"
|
||||||
|
project="thoth-workspace-registry-smoke-selftest-123"
|
||||||
|
exact_image="thothii-workspace-registry-smoke:${project}"
|
||||||
|
foreign_project="thothii-workspace-registry-smoke:thoth-workspace-registry-smoke-foreign-456"
|
||||||
|
foreign_tag="thothii-workspace-registry-smoke:local"
|
||||||
|
image="$exact_image"
|
||||||
|
|
||||||
|
docker() {
|
||||||
|
printf '%s\n' "docker $*" >>"$calls"
|
||||||
|
case "$1 $2" in
|
||||||
|
"image rm")
|
||||||
|
[[ "$3" == "-f" ]] || return 41
|
||||||
|
[[ "$4" == "$exact_image" ]] || return 42
|
||||||
|
return 0
|
||||||
|
;;
|
||||||
|
"image inspect")
|
||||||
|
[[ "$3" == "--format" ]] || return 43
|
||||||
|
[[ "$5" == "$exact_image" ]] || return 44
|
||||||
|
return 1
|
||||||
|
;;
|
||||||
|
"ps -a"|"volume ls"|"network ls")
|
||||||
|
[[ "$*" == *"label=com.docker.compose.project=$project"* ]] || return 45
|
||||||
|
return 0
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
return 46
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
cleanup_smoke_image
|
||||||
|
leftovers="$(workspace_registry_smoke_leftovers)"
|
||||||
|
[[ -z "$leftovers" ]] || {
|
||||||
|
echo "self-test observed leftovers for the per-run image" >&2
|
||||||
|
printf '%s\n' "$leftovers" >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
grep -Fq "docker image rm -f $exact_image" "$calls" \
|
||||||
|
|| { echo "self-test did not remove the exact per-run image reference" >&2; return 1; }
|
||||||
|
if grep -Fq "$foreign_project" "$calls" || grep -Fq "$foreign_tag" "$calls"; then
|
||||||
|
echo "self-test cleanup touched a foreign workspace-registry smoke image reference" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
echo "workspace registry smoke image cleanup identity self-test passed"
|
||||||
|
}
|
||||||
|
|
||||||
|
if [[ "${WORKSPACE_REGISTRY_SMOKE_SELF_TEST:-}" == "image-cleanup-identity" ]]; then
|
||||||
|
workspace_registry_smoke_self_test_image_cleanup_identity
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
local cleanup_status=$?
|
||||||
|
compose down --volumes --remove-orphans >/dev/null 2>&1 || true
|
||||||
|
cleanup_smoke_image
|
||||||
|
if [[ "$cleanup_status" -eq 0 ]]; then
|
||||||
|
local leftovers
|
||||||
|
leftovers="$(workspace_registry_smoke_leftovers)"
|
||||||
if [[ -n "$leftovers" ]]; then
|
if [[ -n "$leftovers" ]]; then
|
||||||
echo "workspace registry cleanup left owned Docker resources:" >&2
|
echo "workspace registry cleanup left owned Docker resources:" >&2
|
||||||
printf '%s\n' "$leftovers" >&2
|
printf '%s\n' "$leftovers" >&2
|
||||||
|
|||||||
Reference in New Issue
Block a user