Add full shell, replaceable Omics adapter and bilingual interaction

Implement approved specification #32 and tickets #33-#37. Keep host authentication server-verified and pin session interaction language. Compile scoped base selectors for browser compatibility and retain full gutters during CSS pruning.
This commit is contained in:
Codex
2026-09-13 14:26:39 +02:00
parent 2d1b714ebe
commit d8a29bfbdd
207 changed files with 8570 additions and 2164 deletions
+94
View File
@@ -0,0 +1,94 @@
# Native installation CLI
## Shell configuration
The schema-v2 `thothii-installation.yaml` accepts this optional section:
```yaml
shell:
mode: full
defaultLocale: en
```
Omitted shell settings resolve to `mode: embedded`, `defaultLocale: en`, and
`adapter: omics-portal`. Supported modes are `full` and `embedded`. The default
locale accepts well-formed BCP47 language tags, including tags without a UI catalog
(for example `fr-CA` or `sr-Latn-RS`). Syntax follows
[RFC 5646](https://www.rfc-editor.org/rfc/rfc5646.html#section-2.1), including private
use and grandfathered tags; duplicate variants and extension singletons are rejected.
The installation preserves the supplied tag; the frontend resolves available
translations and falls back to English. No catalog or language-registry allowlist
is imposed by the installer. The only supported adapter is `omics-portal`.
Invalid values and unknown descriptor fields are rejected. In full mode, a known
adapter setting is ignored and omitted from the resolved public configuration.
Shell selection is independent of `profile` and authentication.
New installations can select these values with `tht setup --shell-mode full
--shell-default-locale en`. The optional `--shell-adapter omics-portal` selects the
embedded adapter explicitly. Corresponding environment answers are
`THT_SETUP_SHELL_MODE`, `THT_SETUP_SHELL_DEFAULT_LOCALE`, and
`THT_SETUP_SHELL_ADAPTER`; explicit flags take precedence. Omitting all three
preserves the existing setup descriptor format and uses the defaults at load time.
Setup does not overwrite an existing descriptor with different settings.
## Public runtime projection
The installation generation workflow (`modelprojection.Generate`, used
by setup, lifecycle operations, and `installation generate`) publishes `generated/frontend/config.js` next
to the model artifacts, relative to the descriptor directory. It contains only:
```js
window.__THOTHII_CONFIG__ = {
"backendBaseUrl": "/api",
"shell": {
"mode": "embedded",
"defaultLocale": "en",
"adapter": "omics-portal"
}
};
```
For full mode, `shell.adapter` is absent. No secret values, model configuration,
authentication configuration, or host filesystem paths enter this JavaScript.
The standalone browser API address remains same-origin `/api`; it never uses
the private nginx upstream `http://core:8787`. The Omics document must continue
to supply its same-origin `/datamart-builder/api` override when mounting the
application under that prefix; selecting an adapter does not infer API routing.
The automatically included `generated/compose.models.yaml` adds a read-only file
bind from `generated/frontend/config.js` to `/usr/share/nginx/html/config.js` on
the frontend service. Missing sources fail instead of becoming directories
(`bind.create_host_path: false`). The frontend receives
`THT_FRONTEND_CONFIG_REVISION=sha256:<hash-of-config.js-bytes>` so Compose recreates
it when public settings change. Shell changes do not revise core's model settings.
Only the public file is mounted into the frontend, not the generated directory.
Publication retains the existing whole-generation replacement and rollback
behavior. Identical outputs keep file identities; projection drift checks include
the public file, Compose definition, and POSIX read permissions. Candidate directories
are explicitly set to `0755` and files to `0644`, independent of the host umask, so
nginx and core can read their file mounts under different UIDs. The enclosing private
installation directory is not made public. Existing permission drift is repaired by
publishing a new complete generation, using the same rollback path as content changes.
A running container requires the normal
Compose lifecycle step to pick up a changed file bind. No image rebuild is needed.
The generic image retains its empty fallback config, and nginx already serves
`/config.js` with `no-store, no-cache, must-revalidate`.
`docker/smoke/frontend-smoke.sh` checks both the generic image fallback and the
installed public projection. Its optional file argument permits the same check
against generated output in targeted host tests.
To generate files before rebuilding or launching containers, use the upgraded CLI:
```sh
/usr/local/bin/tht --installation /Users/mp/projects/ThothII/deploy/psd/thothii-installation.yaml installation generate
```
This command validates the installation and publishes the common runtime generation.
It invokes no Docker commands and does not modify the descriptor, authored model
catalog, environment, or authentication configuration. A shell-only descriptor change
preserves model projection contents, while the common generation may replace their
file identities. It is not a shell-only writer: model projections are always derived
from the current descriptor. Run the existing preview rebuild after this command to
replace containers with the upgraded images and updated projection mounts.
@@ -0,0 +1,77 @@
package main
import (
"bytes"
"context"
"os"
"path/filepath"
"testing"
"github.com/aritmolab/thothii/tools/tht/internal/config"
)
func TestInstallationGeneratePublishesShellWithoutDockerOrModelChanges(t *testing.T) {
fixture := newCLIFixture(t, "")
fixture.setEnvironment(t)
fixture.generateModelProjection(t)
descriptor, err := os.ReadFile(fixture.installationPath)
if err != nil {
t.Fatal(err)
}
descriptor = append(descriptor, []byte("shell:\n mode: full\n defaultLocale: fr-CA\n")...)
if err := os.WriteFile(fixture.installationPath, descriptor, 0o600); err != nil {
t.Fatal(err)
}
installation, err := config.Load(fixture.installationPath)
if err != nil {
t.Fatal(err)
}
preserved := make(map[string][]byte)
for _, path := range []string{fixture.installationPath, fixture.envFile, installation.GeneratedModelCatalogPath(), installation.GeneratedPiModelsPath(), installation.GeneratedPiSettingsPath()} {
preserved[path], err = os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
}
var stdout, stderr bytes.Buffer
args := []string{"--installation", fixture.installationPath, "installation", "generate"}
if code := run(context.Background(), args, &stdout, &stderr); code != 0 {
t.Fatalf("installation generate exit = %d: %s", code, stderr.String())
}
public, err := os.ReadFile(installation.GeneratedFrontendConfigPath())
if err != nil || !bytes.Contains(public, []byte(`"mode": "full"`)) || !bytes.Contains(public, []byte(`"defaultLocale": "fr-CA"`)) || bytes.Contains(public, []byte(`"adapter"`)) {
t.Fatalf("generated public shell = %s, %v", public, err)
}
for path, before := range preserved {
after, err := os.ReadFile(path)
if err != nil || !bytes.Equal(before, after) {
t.Fatalf("installation generate changed authored configuration or model projection %s: %v", path, err)
}
}
before, err := os.Stat(installation.GeneratedDirectory())
if err != nil {
t.Fatal(err)
}
if code := run(context.Background(), args, &stdout, &stderr); code != 0 {
t.Fatalf("repeated installation generate exit = %d: %s", code, stderr.String())
}
after, err := os.Stat(installation.GeneratedDirectory())
if err != nil || !os.SameFile(before, after) {
t.Fatalf("identical installation generate replaced the generation: %v", err)
}
if _, err := os.Stat(fixture.argsFile); !os.IsNotExist(err) {
t.Fatalf("installation generate invoked Docker: %v", err)
}
}
func TestInstallationGenerateRejectsArgumentsBeforePublication(t *testing.T) {
fixture := newCLIFixture(t, "")
fixture.setEnvironment(t)
var stdout, stderr bytes.Buffer
if code := run(context.Background(), []string{"--installation", fixture.installationPath, "installation", "generate", "--build"}, &stdout, &stderr); code != 2 {
t.Fatalf("exit = %d, want 2: %s", code, stderr.String())
}
if _, err := os.Stat(filepath.Join(fixture.root, "generated")); !os.IsNotExist(err) {
t.Fatalf("invalid arguments published projections: %v", err)
}
}
+38
View File
@@ -41,10 +41,13 @@ descriptor in the current project tree.
Commands:
setup [--configure-only] [--installation-id ID] [--profile local|server]
[--shell-mode full|embedded] [--shell-default-locale BCP47-TAG] [--shell-adapter omics-portal]
Create or validate the local non-secret installation configuration.
installation migrate --output PATH --session-default PROVIDER/MODEL
--embedding-id PROVIDER/MODEL --embedding-dimensions N
Create a review-only schema-v2 candidate from all three legacy model sources.
installation generate
Publish runtime projections from the descriptor without invoking Docker.
version [--json] Show the host CLI build identity.
auth configure --mode local|oidc ...
Configure local users or OIDC group mapping; see tht auth for exact options.
@@ -127,6 +130,9 @@ func run(ctx context.Context, args []string, stdout, stderr io.Writer) int {
return versionCommand(commandArgs, stdout, stderr)
}
if command == "installation" {
if len(commandArgs) > 0 && commandArgs[0] == "generate" {
return installationGenerationCommand(installationPath, commandArgs[1:], stdout, stderr)
}
return installationMigrationCommand(installationPath, commandArgs, stdout, stderr)
}
return commandUsageError(stderr, fmt.Sprintf("unknown command %q", command))
@@ -258,6 +264,32 @@ func isBootstrapCommand(command string) bool {
return command == "help" || command == "setup" || command == "version" || command == "installation"
}
func installationGenerationCommand(installationPath string, args []string, stdout, stderr io.Writer) int {
if len(args) != 0 {
return commandUsageError(stderr, "installation generate does not accept arguments")
}
workingDirectory, err := os.Getwd()
if err != nil {
return commandUsageError(stderr, "current directory is unavailable")
}
installationPath, err = config.Resolve(installationPath, os.Getenv, workingDirectory)
if err != nil {
fmt.Fprintf(stderr, "tht: %s\n", output.Sanitize(err.Error(), nil))
return 2
}
installation, err := config.Load(installationPath)
if err != nil {
fmt.Fprintf(stderr, "tht: %s\n", output.Sanitize(err.Error(), nil))
return 2
}
if err := modelprojection.Generate(installation); err != nil {
fmt.Fprintf(stderr, "tht: %s\n", output.Sanitize(err.Error(), nil))
return 2
}
fmt.Fprintf(stdout, "Runtime projections generated in %s.\n", installation.GeneratedDirectory())
return 0
}
func installationMigrationCommand(installationPath string, args []string, stdout, stderr io.Writer) int {
if installationPath == "" {
return commandUsageError(stderr, "installation migrate requires --installation with the legacy descriptor")
@@ -395,6 +427,12 @@ func parseSetupArgs(args []string) (setup.Request, error) {
target = &request.InstallationID
case "--profile":
target = &request.Profile
case "--shell-mode":
target = &request.Answers.ShellMode
case "--shell-default-locale":
target = &request.Answers.ShellDefaultLocale
case "--shell-adapter":
target = &request.Answers.ShellAdapter
case "--workspace-remote":
target = &request.Answers.WorkspaceRemote
case "--workspace-branch":
+4
View File
@@ -362,6 +362,7 @@ func TestParseSetupArgsAcceptsExplicitNonInteractiveAnswers(t *testing.T) {
"--workspace-remote", "https://git.example.invalid/workspaces.git", "--workspace-branch", "release",
"--workspace-access", "https", "--secrets-file", "/tmp/thothii.secrets", "--pi-auth-file", "/tmp/pi-auth.json",
"--git-credentials-file", "/tmp/git-credentials", "--git-ca-file", "/tmp/git-ca.pem",
"--shell-mode", "full", "--shell-default-locale", "it", "--shell-adapter", "omics-portal",
})
if err != nil {
t.Fatal(err)
@@ -372,6 +373,9 @@ func TestParseSetupArgsAcceptsExplicitNonInteractiveAnswers(t *testing.T) {
if request.Answers.WorkspaceBranch != "release" || request.Answers.GitCAFile != "/tmp/git-ca.pem" {
t.Fatalf("setup answers = %#v", request.Answers)
}
if request.Answers.ShellMode != "full" || request.Answers.ShellDefaultLocale != "it" || request.Answers.ShellAdapter != "omics-portal" {
t.Fatalf("setup shell answers = %#v", request.Answers)
}
}
func TestRunPiStatusUsesInstallationEnvironmentWithoutFlag(t *testing.T) {
+11 -1
View File
@@ -40,6 +40,7 @@ type descriptor struct {
WorkspaceRepository workspaceRepositoryDescriptor `yaml:"workspaceRepository"`
Authentication authenticationDescriptor `yaml:"authentication"`
ModelCatalog ModelCatalog `yaml:"modelCatalog"`
Shell Shell `yaml:"shell"`
Overrides []string `yaml:"overrides"`
}
@@ -91,6 +92,7 @@ type Installation struct {
WorkspaceRepository WorkspaceRepository
Authentication Authentication
ModelCatalog ModelCatalog
Shell Shell
Overrides []string
}
@@ -128,6 +130,9 @@ func Load(path string) (Installation, error) {
if err := raw.ModelCatalog.NormalizeDefaults(); err != nil {
return Installation{}, err
}
if err := raw.Shell.NormalizeDefaults(); err != nil {
return Installation{}, err
}
if raw.SchemaVersion != 2 {
return Installation{}, errors.New("migration_required: installation schemaVersion must be 2")
@@ -176,6 +181,7 @@ func Load(path string) (Installation, error) {
},
Authentication: authentication,
ModelCatalog: raw.ModelCatalog,
Shell: raw.Shell,
Overrides: make([]string, 0, len(raw.Overrides)),
}
values, err := installation.environmentValues()
@@ -435,7 +441,7 @@ func (i Installation) ComposeFiles() []string {
return files
}
// GeneratedDirectory contains disposable runtime adapters derived from modelCatalog.
// GeneratedDirectory contains disposable model and public frontend runtime projections.
func (i Installation) GeneratedDirectory() string {
return filepath.Join(filepath.Dir(i.Path), "generated")
}
@@ -452,6 +458,10 @@ func (i Installation) GeneratedPiSettingsPath() string {
return filepath.Join(i.GeneratedDirectory(), "pi", "settings.json")
}
func (i Installation) GeneratedFrontendConfigPath() string {
return filepath.Join(i.GeneratedDirectory(), "frontend", "config.js")
}
func (i Installation) ModelProjectionComposePath() string {
return filepath.Join(i.GeneratedDirectory(), "compose.models.yaml")
}
@@ -0,0 +1,113 @@
package config
import (
"os"
"strings"
"testing"
)
func TestLoadDefaultsLegacyInstallationToEmbeddedShell(t *testing.T) {
for _, profile := range []string{"local", "server"} {
t.Run(profile, func(t *testing.T) {
path, _, _, _ := writeInstallation(t, profile)
installation, err := Load(path)
if err != nil {
t.Fatal(err)
}
if got, want := installation.Shell, (Shell{Mode: "embedded", DefaultLocale: "en", Adapter: "omics-portal"}); got != want {
t.Fatalf("Shell = %#v, want %#v", got, want)
}
})
}
}
func TestLoadResolvesExplicitShell(t *testing.T) {
for _, test := range []struct {
name, yaml string
want Shell
}{
{"empty mapping", "shell: {}\n", Shell{Mode: "embedded", DefaultLocale: "en", Adapter: "omics-portal"}},
{"embedded Italian", "shell:\n mode: embedded\n defaultLocale: it\n", Shell{Mode: "embedded", DefaultLocale: "it", Adapter: "omics-portal"}},
{"full defaults", "shell:\n mode: full\n", Shell{Mode: "full", DefaultLocale: "en"}},
{"full ignores known adapter", "shell:\n mode: full\n defaultLocale: it\n adapter: omics-portal\n", Shell{Mode: "full", DefaultLocale: "it"}},
} {
t.Run(test.name, func(t *testing.T) {
path, _, _, _ := writeInstallation(t, "local")
appendShellDescriptor(t, path, test.yaml)
installation, err := Load(path)
if err != nil {
t.Fatal(err)
}
if installation.Shell != test.want {
t.Fatalf("Shell = %#v, want %#v", installation.Shell, test.want)
}
})
}
}
func TestLoadRejectsInvalidShell(t *testing.T) {
for _, test := range []struct{ name, yaml, errorPart string }{
{"unknown mode", "shell:\n mode: fullscreen\n", "shell.mode"},
{"mode whitespace", "shell:\n mode: ' full '\n", "shell.mode"},
{"malformed locale separator", "shell:\n defaultLocale: en_US\n", "shell.defaultLocale"},
{"malformed locale", "shell:\n defaultLocale: 'en;alert(1)'\n", "shell.defaultLocale"},
{"unknown adapter", "shell:\n adapter: other-portal\n", "shell.adapter"},
{"unknown adapter in full", "shell:\n mode: full\n adapter: other-portal\n", "shell.adapter"},
{"wrong shape", "shell: []\n", "cannot unmarshal"},
{"wrong field type", "shell:\n mode: [full]\n", "cannot unmarshal"},
{"unknown field", "shell:\n theme: light\n", "field theme not found"},
} {
t.Run(test.name, func(t *testing.T) {
path, _, _, _ := writeInstallation(t, "local")
appendShellDescriptor(t, path, test.yaml)
if _, err := Load(path); err == nil || !strings.Contains(err.Error(), test.errorPart) {
t.Fatalf("Load() error = %v, want %q", err, test.errorPart)
}
})
}
}
func TestLoadPreservesWellFormedShellLocalesWithoutCatalogLookup(t *testing.T) {
for _, locale := range []string{
"fr", "de-DE", "en-US", "sr-Latn-RS", "es-419", "zh-cmn-Hans-CN",
"sl-rozaj-biske", "de-CH-1901", "en-US-u-ca-gregory", "en-a-foo-b-bar-x-private",
"x-clinical", "en-x-a-a", "qaa-Qaaa-QM", "zzzz", "abcde", "EN-us",
"en-GB-oed", "i-klingon", "sgn-BE-FR", "zh-min-nan",
} {
t.Run(locale, func(t *testing.T) {
path, _, _, _ := writeInstallation(t, "local")
appendShellDescriptor(t, path, "shell:\n mode: full\n defaultLocale: "+locale+"\n")
installation, err := Load(path)
if err != nil || installation.Shell.DefaultLocale != locale {
t.Fatalf("Load() locale = %q, %v; want preserved %q", installation.Shell.DefaultLocale, err, locale)
}
})
}
}
func TestLoadRejectsMalformedShellLocales(t *testing.T) {
for _, locale := range []string{
"en_US", "en--US", "-en", "en-", "e", "123", "en-1234a5678", "en-12",
"en-US-Latn", "en-u", "en-x", "x", "a-foo", "i-unknown", "en-äbc", "en-Kaaa",
"en US", " en", "en;alert(1)", "sl-rozaj-ROZAJ", "en-a-foo-A-bar",
} {
t.Run(locale, func(t *testing.T) {
path, _, _, _ := writeInstallation(t, "local")
appendShellDescriptor(t, path, "shell:\n defaultLocale: '"+locale+"'\n")
if _, err := Load(path); err == nil || !strings.Contains(err.Error(), "shell.defaultLocale") {
t.Fatalf("Load() error = %v; want invalid locale %q rejected", err, locale)
}
})
}
}
func appendShellDescriptor(t *testing.T, path, shell string) {
t.Helper()
contents, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
if err := os.WriteFile(path, append(contents, []byte(shell)...), 0o600); err != nil {
t.Fatal(err)
}
}
+90
View File
@@ -0,0 +1,90 @@
package config
import (
"errors"
"regexp"
"strings"
)
// Shell is the public presentation configuration, independent of deployment topology and auth.
type Shell struct {
Mode string `yaml:"mode" json:"mode"`
DefaultLocale string `yaml:"defaultLocale" json:"defaultLocale"`
Adapter string `yaml:"adapter,omitempty" json:"adapter,omitempty"`
}
// NormalizeDefaults resolves omitted installation settings before projecting them to the browser.
func (s *Shell) NormalizeDefaults() error {
if s.Mode == "" {
s.Mode = "embedded"
}
if s.DefaultLocale == "" {
s.DefaultLocale = "en"
}
if s.Mode != "embedded" && s.Mode != "full" {
return errors.New("shell.mode must be full or embedded")
}
if !wellFormedLocale(s.DefaultLocale) {
return errors.New("shell.defaultLocale must be a well-formed BCP47 language tag")
}
if s.Adapter != "" && s.Adapter != "omics-portal" {
return errors.New("shell.adapter must be omics-portal")
}
if s.Mode == "full" {
s.Adapter = ""
return nil
}
if s.Adapter == "" {
s.Adapter = "omics-portal"
}
return nil
}
// RFC 5646 section 2.1 grammar. Capture variants and extensions to reject duplicates.
// Validate syntax without consulting a language registry or the frontend's available catalogs.
var localePattern = regexp.MustCompile(`^(?:` +
`(?:[a-z]{2,3}(?:-[a-z]{3}){0,3}|[a-z]{4}|[a-z]{5,8})` + // language / extlang
`(?:-[a-z]{4})?(?:-(?:[a-z]{2}|[0-9]{3}))?` + // script / region
`((?:-(?:[a-z0-9]{5,8}|[0-9][a-z0-9]{3}))*)` + // variants
`((?:-[0-9a-wy-z](?:-[a-z0-9]{2,8})+)*)` + // extensions
`(?:-x(?:-[a-z0-9]{1,8})+)?` + // private use after a language
`|x(?:-[a-z0-9]{1,8})+)$`) // private use alone
func wellFormedLocale(locale string) bool {
// BCP47 is ASCII. Check before case folding, which could otherwise map Unicode to ASCII.
for _, character := range locale {
if character > 127 {
return false
}
}
tag := strings.ToLower(locale)
// These grandfathered irregular tags do not match the normal language-tag grammar.
switch tag {
case "en-gb-oed", "i-ami", "i-bnn", "i-default", "i-enochian", "i-hak", "i-klingon",
"i-lux", "i-mingo", "i-navajo", "i-pwn", "i-tao", "i-tay", "i-tsu",
"sgn-be-fr", "sgn-be-nl", "sgn-ch-de":
return true
}
parts := localePattern.FindStringSubmatch(tag)
if parts == nil {
return false
}
variants := make(map[string]bool)
for _, variant := range strings.Split(strings.TrimPrefix(parts[1], "-"), "-") {
if variants[variant] {
return false
}
variants[variant] = true
}
singletons := make(map[string]bool)
for _, subtag := range strings.Split(parts[2], "-") {
if len(subtag) != 1 {
continue
}
if singletons[subtag] {
return false
}
singletons[subtag] = true
}
return true
}
@@ -1,4 +1,4 @@
// Package modelprojection renders disposable runtime adapters from the installation model catalog.
// Package modelprojection renders disposable model and public frontend installation projections.
package modelprojection
import (
@@ -16,10 +16,11 @@ import (
)
const (
CatalogFile = "catalog.json"
PiModelsFile = "pi/models.json"
PiSettingsFile = "pi/settings.json"
ComposeFile = "compose.models.yaml"
CatalogFile = "catalog.json"
PiModelsFile = "pi/models.json"
PiSettingsFile = "pi/settings.json"
ComposeFile = "compose.models.yaml"
FrontendConfigFile = "frontend/config.js"
)
var renameProjectionDirectory = os.Rename
@@ -77,6 +78,9 @@ func Render(installation config.Installation) (map[string][]byte, error) {
if err := installation.ModelCatalog.NormalizeDefaults(); err != nil {
return nil, err
}
if err := installation.Shell.NormalizeDefaults(); err != nil {
return nil, err
}
models := installation.ModelCatalog.RuntimeModels()
hasMetadata := false
for _, model := range models {
@@ -146,10 +150,21 @@ func Render(installation config.Installation) (map[string][]byte, error) {
return nil, fmt.Errorf("render Pi settings projection: %w", err)
}
fingerprint := sha256.Sum256(bytes.Join([][]byte{catalogBytes, piModelsBytes, piSettingsBytes}, nil))
composeBytes := renderCompose(installation, fmt.Sprintf("sha256:%x", fingerprint))
publicJSON, err := marshalJSON(struct {
BackendBaseURL string `json:"backendBaseUrl"`
Shell config.Shell `json:"shell"`
}{BackendBaseURL: "/api", Shell: installation.Shell})
if err != nil {
return nil, fmt.Errorf("render public frontend configuration: %w", err)
}
publicJS := append([]byte("window.__THOTHII_CONFIG__ = "), bytes.TrimSpace(publicJSON)...)
publicJS = append(publicJS, ';', '\n')
publicFingerprint := sha256.Sum256(publicJS)
composeBytes := renderCompose(installation, fmt.Sprintf("sha256:%x", fingerprint), fmt.Sprintf("sha256:%x", publicFingerprint))
return map[string][]byte{
CatalogFile: catalogBytes, PiModelsFile: piModelsBytes,
PiSettingsFile: piSettingsBytes, ComposeFile: composeBytes,
FrontendConfigFile: publicJS,
}, nil
}
@@ -210,10 +225,14 @@ func Generate(installation config.Installation) error {
}
func projectionMatches(directory string, artifacts map[string][]byte) bool {
if !projectionModeMatches(directory, 0o755) {
return false
}
for _, relative := range sortedArtifactPaths(artifacts) {
path := filepath.Join(directory, filepath.FromSlash(relative))
info, err := os.Lstat(path)
if err != nil || !info.Mode().IsRegular() {
if err != nil || !info.Mode().IsRegular() || !projectionModeMatches(path, 0o644) ||
!projectionModeMatches(filepath.Dir(path), 0o755) {
return false
}
actual, err := os.ReadFile(path)
@@ -224,6 +243,14 @@ func projectionMatches(directory string, artifacts map[string][]byte) bool {
return true
}
func projectionModeMatches(path string, mode os.FileMode) bool {
if runtime.GOOS == "windows" {
return true // POSIX mode bits do not represent Windows access controls.
}
info, err := os.Lstat(path)
return err == nil && info.Mode()&os.ModeSymlink == 0 && info.Mode().Perm() == mode
}
func writeProjectionCandidate(directory string, artifacts map[string][]byte) error {
if err := os.Chmod(directory, 0o755); err != nil {
return fmt.Errorf("protect model projection candidate: %w", err)
@@ -233,9 +260,16 @@ func writeProjectionCandidate(directory string, artifacts map[string][]byte) err
if err := os.MkdirAll(filepath.Dir(destination), 0o755); err != nil {
return fmt.Errorf("create model projection directory: %w", err)
}
if err := os.Chmod(filepath.Dir(destination), 0o755); err != nil {
return fmt.Errorf("set runtime projection directory permissions: %w", err)
}
if err := os.WriteFile(destination, artifacts[relative], 0o644); err != nil {
return fmt.Errorf("write model projection candidate: %w", err)
}
// Creation modes are filtered by the host umask; container readers have another UID.
if err := os.Chmod(destination, 0o644); err != nil {
return fmt.Errorf("set runtime projection file permissions: %w", err)
}
}
return nil
}
@@ -251,7 +285,7 @@ func absentTemporaryPath(parent string) (string, error) {
return path, nil
}
// Check returns relative artifact names whose current bytes differ from the catalog projection.
// Check returns relative artifact names whose bytes or POSIX modes differ from the projection.
func Check(installation config.Installation) ([]string, error) {
artifacts, err := Render(installation)
if err != nil {
@@ -259,8 +293,11 @@ func Check(installation config.Installation) ([]string, error) {
}
drift := make([]string, 0)
for _, relative := range sortedArtifactPaths(artifacts) {
actual, readErr := os.ReadFile(filepath.Join(installation.GeneratedDirectory(), filepath.FromSlash(relative)))
if readErr != nil || !bytes.Equal(actual, artifacts[relative]) {
path := filepath.Join(installation.GeneratedDirectory(), filepath.FromSlash(relative))
actual, readErr := os.ReadFile(path)
if readErr != nil || !bytes.Equal(actual, artifacts[relative]) ||
!projectionModeMatches(installation.GeneratedDirectory(), 0o755) ||
!projectionModeMatches(filepath.Dir(path), 0o755) || !projectionModeMatches(path, 0o644) {
drift = append(drift, relative)
}
}
@@ -275,9 +312,19 @@ func marshalJSON(value any) ([]byte, error) {
return append(contents, '\n'), nil
}
func renderCompose(installation config.Installation, fingerprint string) []byte {
func renderCompose(installation config.Installation, fingerprint, publicFingerprint string) []byte {
embedding := installation.ModelCatalog.Embedding
return []byte(fmt.Sprintf(`services:
frontend:
environment:
THT_FRONTEND_CONFIG_REVISION: %s
volumes:
- type: bind
source: %s
target: /usr/share/nginx/html/config.js
read_only: true
bind:
create_host_path: false
core:
environment:
THT_HOST_PLATFORM: %s
@@ -308,7 +355,8 @@ func renderCompose(installation config.Installation, fingerprint string) []byte
embedding-model-init:
environment:
OLLAMA_MODEL: %s
`, strconv.Quote(runtime.GOOS), strconv.Quote(fingerprint), strconv.Quote(installation.ModelCatalog.Defaults.Interaction),
`, strconv.Quote(publicFingerprint), strconv.Quote(installation.GeneratedFrontendConfigPath()),
strconv.Quote(runtime.GOOS), strconv.Quote(fingerprint), strconv.Quote(installation.ModelCatalog.Defaults.Interaction),
strconv.Quote(embedding.ID), strconv.Quote(embeddingModelName(embedding.ID)), strconv.Quote(strconv.Itoa(embedding.Dimensions)),
strconv.Quote(installation.GeneratedModelCatalogPath()), strconv.Quote(installation.GeneratedPiModelsPath()),
strconv.Quote(installation.GeneratedPiSettingsPath()), strconv.Quote(embedding.ID),
@@ -0,0 +1,88 @@
//go:build linux || darwin
package modelprojection
import (
"os"
"path/filepath"
"syscall"
"testing"
"github.com/aritmolab/thothii/tools/tht/internal/config"
)
func TestGenerateKeepsProjectionsContainerReadableWithRestrictiveUmask(t *testing.T) {
installation := projectionFixture(t)
parentBefore, err := os.Stat(filepath.Dir(installation.Path))
if err != nil {
t.Fatal(err)
}
// This test must remain sequential: umask is process-wide.
previous := syscall.Umask(0o077)
defer syscall.Umask(previous)
if err := Generate(installation); err != nil {
t.Fatal(err)
}
assertContainerProjectionModes(t, installation)
parentAfter, err := os.Stat(filepath.Dir(installation.Path))
if err != nil || parentAfter.Mode().Perm() != parentBefore.Mode().Perm() {
t.Fatalf("generation changed private installation directory permissions: %v", err)
}
}
func TestGenerateRepairsUnreadableProjectionAsANewGeneration(t *testing.T) {
installation := projectionFixture(t)
if err := Generate(installation); err != nil {
t.Fatal(err)
}
before, err := os.Stat(installation.GeneratedDirectory())
if err != nil {
t.Fatal(err)
}
for path, mode := range map[string]os.FileMode{
installation.GeneratedDirectory(): 0o700,
filepath.Dir(installation.GeneratedPiModelsPath()): 0o700,
installation.GeneratedFrontendConfigPath(): 0o600,
} {
if err := os.Chmod(path, mode); err != nil {
t.Fatal(err)
}
}
if drift, err := Check(installation); err != nil || len(drift) == 0 {
t.Fatalf("unreadable projection drift = %v, %v", drift, err)
}
if err := Generate(installation); err != nil {
t.Fatal(err)
}
assertContainerProjectionModes(t, installation)
after, err := os.Stat(installation.GeneratedDirectory())
if err != nil || os.SameFile(before, after) {
t.Fatalf("permissions were not repaired through candidate publication: %v", err)
}
if drift, err := Check(installation); err != nil || len(drift) != 0 {
t.Fatalf("repaired projection drift = %v, %v", drift, err)
}
}
func assertContainerProjectionModes(t *testing.T, installation config.Installation) {
t.Helper()
for path, mode := range map[string]os.FileMode{
installation.GeneratedDirectory(): 0o755,
filepath.Dir(installation.GeneratedFrontendConfigPath()): 0o755,
filepath.Dir(installation.GeneratedPiModelsPath()): 0o755,
installation.GeneratedFrontendConfigPath(): 0o644,
installation.GeneratedModelCatalogPath(): 0o644,
installation.GeneratedPiModelsPath(): 0o644,
installation.GeneratedPiSettingsPath(): 0o644,
installation.ModelProjectionComposePath(): 0o644,
} {
info, err := os.Stat(path)
if err != nil {
t.Error(err)
continue
}
if info.Mode().Perm() != mode {
t.Errorf("%s mode = %04o, want %04o for container readers", path, info.Mode().Perm(), mode)
}
}
}
@@ -24,8 +24,8 @@ func TestRenderProducesDeterministicCatalogPiAndComposeProjections(t *testing.T)
if err != nil {
t.Fatalf("Render() second error = %v", err)
}
if len(first) != 4 {
t.Fatalf("artifact count = %d, want 4", len(first))
if len(first) != 5 {
t.Fatalf("artifact count = %d, want 5", len(first))
}
for path, contents := range first {
if !bytes.Equal(contents, second[path]) {
@@ -133,6 +133,7 @@ func TestGenerateDoesNotReplaceUnchangedProjection(t *testing.T) {
installation.GeneratedModelCatalogPath(),
installation.GeneratedPiModelsPath(),
installation.GeneratedPiSettingsPath(),
installation.GeneratedFrontendConfigPath(),
installation.ModelProjectionComposePath(),
}
before := make(map[string]os.FileInfo, len(paths))
@@ -174,7 +175,7 @@ func TestGenerateRestoresWholePreviousGenerationWhenPublishFails(t *testing.T) {
t.Fatalf("Generate() initial error = %v", err)
}
before := make(map[string][]byte)
for _, path := range []string{CatalogFile, PiModelsFile, PiSettingsFile, ComposeFile} {
for _, path := range []string{CatalogFile, PiModelsFile, PiSettingsFile, ComposeFile, FrontendConfigFile} {
contents, err := os.ReadFile(filepath.Join(installation.GeneratedDirectory(), filepath.FromSlash(path)))
if err != nil {
t.Fatal(err)
@@ -0,0 +1,139 @@
package modelprojection
import (
"bytes"
"encoding/json"
"os"
"os/exec"
"path/filepath"
"reflect"
"runtime"
"strings"
"testing"
"github.com/aritmolab/thothii/tools/tht/internal/config"
"gopkg.in/yaml.v3"
)
func TestRenderPublicShellConfiguration(t *testing.T) {
for _, test := range []struct {
name string
shell config.Shell
want string
}{
{"legacy", config.Shell{}, `{"backendBaseUrl":"/api","shell":{"mode":"embedded","defaultLocale":"en","adapter":"omics-portal"}}`},
{"full", config.Shell{Mode: "full", Adapter: "omics-portal"}, `{"backendBaseUrl":"/api","shell":{"mode":"full","defaultLocale":"en"}}`},
{"Italian", config.Shell{Mode: "embedded", DefaultLocale: "it"}, `{"backendBaseUrl":"/api","shell":{"mode":"embedded","defaultLocale":"it","adapter":"omics-portal"}}`},
{"locale without catalog", config.Shell{Mode: "full", DefaultLocale: "fr-CA"}, `{"backendBaseUrl":"/api","shell":{"mode":"full","defaultLocale":"fr-CA"}}`},
} {
t.Run(test.name, func(t *testing.T) {
installation := projectionFixture(t)
installation.Shell = test.shell
files, err := Render(installation)
if err != nil {
t.Fatal(err)
}
// The entire public payload is an allowlist: no model endpoints, auth or host paths.
js := string(files["frontend/config.js"])
if !strings.HasPrefix(js, "window.__THOTHII_CONFIG__ = ") || !strings.HasSuffix(js, ";\n") {
t.Fatalf("invalid public config script: %q", js)
}
payload := strings.TrimSuffix(strings.TrimPrefix(js, "window.__THOTHII_CONFIG__ = "), ";\n")
var got, want any
if err := json.Unmarshal([]byte(payload), &got); err != nil {
t.Fatal(err)
}
if err := json.Unmarshal([]byte(test.want), &want); err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("public config = %s, want %s", payload, test.want)
}
var compose struct {
Services map[string]struct {
Volumes []struct {
Type string `yaml:"type"`
Source string `yaml:"source"`
Target string `yaml:"target"`
ReadOnly bool `yaml:"read_only"`
} `yaml:"volumes"`
} `yaml:"services"`
}
if err := yaml.Unmarshal(files[ComposeFile], &compose); err != nil {
t.Fatal(err)
}
mounts := compose.Services["frontend"].Volumes
if len(mounts) != 1 || mounts[0].Type != "bind" || !mounts[0].ReadOnly ||
mounts[0].Source != filepath.Join(installation.GeneratedDirectory(), "frontend", "config.js") ||
mounts[0].Target != "/usr/share/nginx/html/config.js" {
t.Fatalf("frontend mounts = %#v; want only public config.js, read-only", mounts)
}
})
}
}
func TestGenerateRefreshesPublicShellConfiguration(t *testing.T) {
installation := projectionFixture(t)
if err := Generate(installation); err != nil {
t.Fatal(err)
}
publicPath := filepath.Join(installation.GeneratedDirectory(), "frontend", "config.js")
before, err := os.ReadFile(publicPath)
if err != nil {
t.Fatal(err)
}
composeBefore, err := os.ReadFile(installation.ModelProjectionComposePath())
if err != nil {
t.Fatal(err)
}
installation.Shell = config.Shell{Mode: "full", DefaultLocale: "it"}
drift, err := Check(installation)
if err != nil || !reflect.DeepEqual(drift, []string{"compose.models.yaml", "frontend/config.js"}) {
t.Fatalf("shell drift = %v, %v", drift, err)
}
if err := Generate(installation); err != nil {
t.Fatal(err)
}
after, err := os.ReadFile(publicPath)
if err != nil || bytes.Equal(before, after) || !bytes.Contains(after, []byte(`"mode": "full"`)) || bytes.Contains(after, []byte(`"adapter"`)) {
t.Fatalf("updated public config = %s, %v", after, err)
}
composeAfter, err := os.ReadFile(installation.ModelProjectionComposePath())
if err != nil {
t.Fatal(err)
}
// A config revision changes Compose's service definition, ensuring active bind mounts refresh.
var oldCompose, newCompose map[string]map[string]map[string]any
if err := yaml.Unmarshal(composeBefore, &oldCompose); err != nil {
t.Fatal(err)
}
if err := yaml.Unmarshal(composeAfter, &newCompose); err != nil {
t.Fatal(err)
}
if reflect.DeepEqual(oldCompose["services"]["frontend"]["environment"], newCompose["services"]["frontend"]["environment"]) {
t.Fatal("shell change did not revise the frontend service")
}
if !reflect.DeepEqual(oldCompose["services"]["core"], newCompose["services"]["core"]) {
t.Fatal("shell-only change revised core configuration")
}
if drift, err := Check(installation); err != nil || len(drift) != 0 {
t.Fatalf("fresh generation drift = %v, %v", drift, err)
}
}
func TestGeneratedPublicShellPassesFrontendImageSmoke(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("frontend image smoke requires a POSIX host shell")
}
for _, shell := range []config.Shell{{}, {Mode: "full"}, {Mode: "full", DefaultLocale: "it"}, {Mode: "embedded", DefaultLocale: "it"}, {Mode: "full", DefaultLocale: "fr-CA"}, {Mode: "embedded", DefaultLocale: "zh-Hant-TW-u-nu-hanidec"}} {
installation := projectionFixture(t)
installation.Shell = shell
if err := Generate(installation); err != nil {
t.Fatal(err)
}
command := exec.Command("sh", "../../../../docker/smoke/frontend-smoke.sh", installation.GeneratedFrontendConfigPath())
if output, err := command.CombinedOutput(); err != nil {
t.Fatalf("frontend smoke for %#v: %v\n%s", shell, err, output)
}
}
}
+17
View File
@@ -36,6 +36,7 @@ var atomicWriteNewFile = writeNewFileAtomically
var effectiveUID = currentEffectiveUID
type answers struct {
shell config.Shell
installationID, profile string
workspaceRemote, workspaceBranch string
workspaceAccess string
@@ -65,6 +66,7 @@ type generatedDescriptor struct {
} `yaml:"runtimeProjection,omitempty"`
} `yaml:"authentication"`
ModelCatalog config.ModelCatalog `yaml:"modelCatalog"`
Shell config.Shell `yaml:"shell,omitempty"`
Overrides []string `yaml:"overrides"`
}
@@ -226,6 +228,11 @@ func collectAnswers(request Request, input io.Reader, output io.Writer, root str
func answersFromRequest(request Request) answers {
answer := request.Answers
return answers{
shell: config.Shell{
Mode: firstNonEmpty(answer.ShellMode, os.Getenv("THT_SETUP_SHELL_MODE")),
DefaultLocale: firstNonEmpty(answer.ShellDefaultLocale, os.Getenv("THT_SETUP_SHELL_DEFAULT_LOCALE")),
Adapter: firstNonEmpty(answer.ShellAdapter, os.Getenv("THT_SETUP_SHELL_ADAPTER")),
},
workspaceRemote: firstNonEmpty(answer.WorkspaceRemote, os.Getenv("THT_SETUP_WORKSPACE_REMOTE")),
workspaceBranch: firstNonEmpty(answer.WorkspaceBranch, os.Getenv("THT_SETUP_WORKSPACE_BRANCH")),
workspaceAccess: firstNonEmpty(answer.WorkspaceAccess, os.Getenv("THT_SETUP_WORKSPACE_ACCESS")),
@@ -272,6 +279,9 @@ func prompt(scanner *bufio.Scanner, output io.Writer, question, defaultValue str
}
func validateAnswers(value answers) error {
if err := value.shell.NormalizeDefaults(); err != nil {
return err
}
if !installationIDPattern.MatchString(value.installationID) {
return errors.New("installation ID must contain only letters, numbers, dashes, and underscores")
}
@@ -322,6 +332,13 @@ func installationDirectory(root, id string) (string, error) {
func render(root, descriptorPath string, value answers) ([]byte, []byte, error) {
descriptor := generatedDescriptor{SchemaVersion: 2, Profile: value.profile, ProjectDirectory: root, EnvFile: filepath.Join(filepath.Dir(descriptorPath), environmentName), ModelCatalog: defaultModelCatalog()}
// Preserve the legacy descriptor bytes when no shell setting was requested.
if value.shell != (config.Shell{}) {
if err := value.shell.NormalizeDefaults(); err != nil {
return nil, nil, err
}
descriptor.Shell = value.shell
}
descriptor.Workspace.Remote, descriptor.Workspace.Branch, descriptor.Workspace.Access = value.workspaceRemote, value.workspaceBranch, value.workspaceAccess
descriptor.Authentication.ConfigDirectory = filepath.Join(filepath.Dir(descriptorPath), "auth")
if value.profile == "server" {
@@ -0,0 +1,78 @@
package setup
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/aritmolab/thothii/tools/tht/internal/config"
)
func TestEnsureFilesProjectsShellAnswersThroughInstallationLoad(t *testing.T) {
for _, test := range []struct {
name string
answers Answers
want config.Shell
}{
{"defaults", Answers{}, config.Shell{Mode: "embedded", DefaultLocale: "en", Adapter: "omics-portal"}},
{"full", Answers{ShellMode: "full"}, config.Shell{Mode: "full", DefaultLocale: "en"}},
{"locale without UI catalog", Answers{ShellMode: "full", ShellDefaultLocale: "fr-CA"}, config.Shell{Mode: "full", DefaultLocale: "fr-CA"}},
{"embedded Italian", Answers{ShellMode: "embedded", ShellDefaultLocale: "it", ShellAdapter: "omics-portal"}, config.Shell{Mode: "embedded", DefaultLocale: "it", Adapter: "omics-portal"}},
} {
t.Run(test.name, func(t *testing.T) {
root := newProject(t, "shell setup")
setNonInteractiveAnswers(t, newExternalSecrets(t, root))
for _, key := range []string{"THT_SETUP_SHELL_MODE", "THT_SETUP_SHELL_DEFAULT_LOCALE", "THT_SETUP_SHELL_ADAPTER"} {
t.Setenv(key, "")
}
request := Request{ProjectRoot: root, InstallationID: "shell", Profile: "local", NonInteractive: true, Answers: test.answers}
result, err := EnsureFiles(request, strings.NewReader(""), ioDiscard{})
if err != nil {
t.Fatal(err)
}
installation, err := config.Load(result.DescriptorPath)
if err != nil || installation.Shell != test.want {
t.Fatalf("Load() shell = %#v, %v, want %#v", installation.Shell, err, test.want)
}
if repeated, err := EnsureFiles(request, strings.NewReader(""), ioDiscard{}); err != nil || len(repeated.Created) != 0 {
t.Fatalf("repeated setup = %#v, %v", repeated, err)
}
})
}
}
func TestEnsureFilesAcceptsShellEnvironmentAndExplicitPrecedence(t *testing.T) {
root := newProject(t, "shell environment")
setNonInteractiveAnswers(t, newExternalSecrets(t, root))
t.Setenv("THT_SETUP_SHELL_MODE", "embedded")
t.Setenv("THT_SETUP_SHELL_DEFAULT_LOCALE", "it")
t.Setenv("THT_SETUP_SHELL_ADAPTER", "omics-portal")
result, err := EnsureFiles(Request{
ProjectRoot: root, InstallationID: "shell", Profile: "local", NonInteractive: true,
Answers: Answers{ShellMode: "full"},
}, strings.NewReader(""), ioDiscard{})
if err != nil {
t.Fatal(err)
}
installation, err := config.Load(result.DescriptorPath)
if err != nil || installation.Shell != (config.Shell{Mode: "full", DefaultLocale: "it"}) {
t.Fatalf("Load() shell = %#v, %v", installation.Shell, err)
}
}
func TestEnsureFilesRejectsInvalidShellBeforeWritingInstallation(t *testing.T) {
for _, answers := range []Answers{{ShellMode: "fullscreen"}, {ShellDefaultLocale: "de_DE"}, {ShellAdapter: "unknown"}} {
root := newProject(t, "invalid shell")
setNonInteractiveAnswers(t, newExternalSecrets(t, root))
_, err := EnsureFiles(Request{
ProjectRoot: root, InstallationID: "shell", Profile: "local", NonInteractive: true, Answers: answers,
}, strings.NewReader(""), ioDiscard{})
if err == nil || !strings.Contains(err.Error(), "shell.") {
t.Fatalf("EnsureFiles() error = %v", err)
}
if _, err := os.Stat(filepath.Join(root, "deploy", "shell")); !os.IsNotExist(err) {
t.Fatalf("invalid shell created installation directory: %v", err)
}
}
}
+3
View File
@@ -15,6 +15,9 @@ type Request struct {
// Answers are optional explicit answers supplied by command flags. Empty values may be supplied
// through THT_SETUP_* environment variables or collected interactively.
type Answers struct {
ShellMode string
ShellDefaultLocale string
ShellAdapter string
WorkspaceRemote string
WorkspaceBranch string
WorkspaceAccess string