docs(testing): record AI description acceptance
This commit is contained in:
@@ -0,0 +1,96 @@
|
|||||||
|
# AI Catalog Description Generation acceptance
|
||||||
|
|
||||||
|
Date: 2026-08-29
|
||||||
|
|
||||||
|
Feature commit: `376dd5a`
|
||||||
|
|
||||||
|
Result: **passed**
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This acceptance run covered the interactive generation and consolidation of Catalog Table and
|
||||||
|
Catalog Column descriptions. It used only an ephemeral PostgreSQL database and invented values; no
|
||||||
|
PSD database or other real business data was queried.
|
||||||
|
|
||||||
|
The installation-level model catalog contained these entries, with `glm-53` as the default:
|
||||||
|
|
||||||
|
| Selection ID | LiteLLM route | Authentication |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `deepseek-v4-pro` | `deepseek/deepseek-v4-pro` | Protected `DEEPSEEK_API_KEY` value already used by core |
|
||||||
|
| `glm-53` | `openai/glm-5.3` on the Z.AI coding endpoint | Protected `ZAI_API_KEY` value already used by core |
|
||||||
|
| `qwen-36` | `openai/qwen3.6-35b-a3b` on the VPN-only AritmoLab endpoint | No operator API key |
|
||||||
|
|
||||||
|
Qwen was configured with `disableThinking: true`. The helper translated this into the endpoint's
|
||||||
|
chat-template option so reasoning prose could not precede the required JSON result. LiteLLM's
|
||||||
|
OpenAI-compatible client still receives a fixed, non-secret compatibility placeholder; no Qwen
|
||||||
|
credential is configured or required.
|
||||||
|
|
||||||
|
Secret values were read only from the existing protected core authentication material and projected
|
||||||
|
into an ephemeral test bundle. No value, digest, prefix, or suffix is recorded here.
|
||||||
|
|
||||||
|
## Provider and database checks
|
||||||
|
|
||||||
|
- DeepSeek completed a real request successfully.
|
||||||
|
- GLM completed a real request successfully.
|
||||||
|
- Qwen's anonymous `/v1/models` endpoint exposed `qwen3.6-35b-a3b`, and a completion with thinking
|
||||||
|
disabled succeeded over the VPN.
|
||||||
|
- The disposable database used PostgreSQL `17.6-bookworm`, schema `acceptance`, and table
|
||||||
|
`prodotti_demo` with invented rows.
|
||||||
|
- The application role `ai_acceptance_reader` could execute `SELECT` but an `INSERT` failed with
|
||||||
|
PostgreSQL permission code `42501`, confirming the read-only boundary.
|
||||||
|
|
||||||
|
## End-to-end result
|
||||||
|
|
||||||
|
Four generation runs went through the application worker:
|
||||||
|
|
||||||
|
1. DeepSeek generated the Catalog Column description for `categoria`.
|
||||||
|
2. GLM generated the Catalog Column description for `prezzo`.
|
||||||
|
3. Qwen generated the Catalog Column description for `attivo`.
|
||||||
|
4. GLM generated the Catalog Table description for `prodotti_demo`.
|
||||||
|
|
||||||
|
Each run emitted exactly four safe activity events: queued, started, target generated, and
|
||||||
|
completed. The generated Italian descriptions were:
|
||||||
|
|
||||||
|
- `categoria`: “Categoria merceologica dimostrativa del prodotto.”
|
||||||
|
- `prezzo`: “Prezzo del prodotto, espresso come valore numerico con decimali (es. 10.00, 12.50);
|
||||||
|
campo obbligatorio, non ammette valori nulli.”
|
||||||
|
- `attivo`: “Indica se il prodotto è attivo.”
|
||||||
|
- `prodotti_demo`: “Tabella di prodotti dimostrativi per il collaudo: per ogni prodotto registra un
|
||||||
|
codice univoco inventato (chiave primaria testuale, es. ALFA-DEMO), la categoria merceologica, il
|
||||||
|
prezzo numerico con decimali (es. 10.00, 12.50) e un flag booleano di attivazione; tutti i quattro
|
||||||
|
campi sono obbligatori.”
|
||||||
|
|
||||||
|
The interactive consolidation action copied one generated description into `Description` and
|
||||||
|
reported `{copied: 1, skipped: 0}`.
|
||||||
|
|
||||||
|
## Data and log safety assertions
|
||||||
|
|
||||||
|
- Every provider request received bounded real source samples from the disposable table, within the
|
||||||
|
configured maximum of five rows and five representative values.
|
||||||
|
- Neither `sourceSample` nor `representativeValues` appeared in persisted catalog/run data.
|
||||||
|
- Protected credential values did not appear in worker logs, API responses, or persistence.
|
||||||
|
- Activity logs contained operational summaries only, without prompts, source rows, representative
|
||||||
|
values, or model responses.
|
||||||
|
- The generated table description echoed the invented sample `ALFA-DEMO`. This is acceptable for
|
||||||
|
this synthetic run but demonstrates why production policy must classify, exclude, or anonymize
|
||||||
|
sensitive values before model calls. That follow-up is tracked by
|
||||||
|
[issue #12](https://git.tylconsulting.it/mptyl/ThothII/issues/12).
|
||||||
|
|
||||||
|
## Regression gates
|
||||||
|
|
||||||
|
The following reproducible gates passed after the provider changes:
|
||||||
|
|
||||||
|
| Gate | Command | Result |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Backend tests | `cd backend && npm test` | 93 files passed, 1 skipped; 1,243 tests passed, 40 skipped |
|
||||||
|
| Backend typecheck | `cd backend && npx tsc --noEmit -p .` | Passed |
|
||||||
|
| Backend production build | `cd backend && npm run build` | Passed |
|
||||||
|
| Harness tests | `cd harness && .venv/bin/pytest -q` | 1,138 passed, 4 deselected, 53 warnings |
|
||||||
|
| Harness lint | `harness/.venv/bin/ruff check harness` | Passed |
|
||||||
|
| Go tests | `env GOCACHE=/tmp/thothii-go-cache go test -p 1 ./...` | All packages passed |
|
||||||
|
| Frontend tests | `cd frontend && npm test` | 63 files and 532 tests passed |
|
||||||
|
| Frontend production build | `cd frontend && npm run build` | Passed; 4,860 modules transformed |
|
||||||
|
| Documentation | `./scripts/build-docs.sh` | Strict MkDocs build passed |
|
||||||
|
|
||||||
|
The disposable database, temporary secret projection, and one-shot provider probe files were
|
||||||
|
removed after the successful run.
|
||||||
@@ -57,6 +57,7 @@ nav:
|
|||||||
- Architecture overview: architecture/overview.md
|
- Architecture overview: architecture/overview.md
|
||||||
- Components, modules, and flows: architecture/components.md
|
- Components, modules, and flows: architecture/components.md
|
||||||
- Evidence: evidence.md
|
- Evidence: evidence.md
|
||||||
|
- AI catalog description acceptance: testing/2026-08-29-ai-catalog-description-generation-acceptance.md
|
||||||
- Authentication: architecture/authentication.md
|
- Authentication: architecture/authentication.md
|
||||||
- Local authentication installation: install/authentication-local.md
|
- Local authentication installation: install/authentication-local.md
|
||||||
- Generic OIDC: install/authentication-oidc.md
|
- Generic OIDC: install/authentication-oidc.md
|
||||||
|
|||||||
Reference in New Issue
Block a user