The Versioned Stores¶

Codaris reads five data files, the versioned stores, that are too large for the repository and change on their own schedule: the RAG store, the container index, the process registry, the tool parameters, and the test-asset cache. Users receive them inside the container image, so a deployment needs no access to the repository. The repository itself holds only backend/app/engine/stores_manifest.json, which names the release that holds the files and pins each one by sha256 and size.

What each store holds¶

How a deployment receives them¶

The image. The stores-cache stage of docker/all-in-one.Dockerfile downloads the five files named in the manifest, verifies each sha256, and the runtime image copies them into /app/app/engine/. A container therefore works offline from its first start, and doctor lists every store as ok. The release is public, so the stage needs no credentials; it reads an optional GitHub token from the BuildKit secret github_token to raise the API rate limit. The publish workflow passes its own token, and a local build can pass yours:

GH_TOKEN="$(gh auth token)" docker compose -f docker/docker-compose.unified.yml build

To confirm what an image carries:

docker run --rm --entrypoint python ghcr.io/primeomicx/codaris:latest -m app.cli stores check

A source checkout. ./bin/codaris stores fetch downloads the files next to the code in backend/app/engine/; with GH_TOKEN or GITHUB_TOKEN set, each asset is resolved through the GitHub API. codaris stores check verifies the local copies against the manifest.

Refreshing the stores¶

The refresh is a maintainer's task and ends with a new image, which is how the fresh knowledge reaches users. Run it from backend/ with the virtual environment active (the requirements include sentence-transformers, which the RAG ingest uses for embeddings), with network access, and with gh signed in. Start from the released files so each step extends them:

cd backend && source .venv/bin/activate
export GH_TOKEN="$(gh auth token)"
python -m app.cli stores fetch

1. Harvest nf-core/modules. Clone the repository, harvest it and build the corpus from the same checkout, and note its commit:

   git clone --depth 1 https://github.com/nf-core/modules /tmp/nfm
   git -C /tmp/nfm rev-parse HEAD                       # the commit these stores describe
   python -m app.engine.harvester --fresh --clone-dir /tmp/nfm
   python -m app.engine.nfcore_corpus /tmp/nfm -o /tmp/nfcore_corpus.json
   

   Use --fresh. A resumed harvest only adds modules it has not seen, so a module whose container, input or output changed upstream keeps its old row. --fresh rewrites every process_contracts row and every module_containers row the harvest wrote, and leaves the knowledge graph, the containers table and the presets alone. A module_containers row whose description starts with Harvested belongs to the harvest and is refreshed; any other row is a curated override (13 in 1.2.0) and is never touched. /tmp/nfcore_corpus.json is the module and subworkflow index that corpus_lookup reads; codaris builds its own copy into the state directory the first time it runs and never refreshes it, so the file published with a release is the way to carry a newer one. 2. Audit the pipeline catalogue. python scripts/audit_nfcore_pipelines.py && python scripts/classify_nfcore.py reads https://nf-co.re/pipelines.json and writes /tmp/nfcore_detailed_classification.json; the pipelines in categories A and B (released from 2022 onward) feed the RAG ingest. 3. Container tags. python scripts/ingest_containers.py && python scripts/setup_container_fts.py adds the builds that appeared on the Galaxy depot since the last run and rebuilds the FTS5 index; the curated aliases and the module_containers rows are kept. 4. RAG chunks. python scripts/ingest_nfcore.py && python scripts/ingest_deep_ast.py downloads each eligible pipeline, extracts the code blocks, embeds them, and inserts them. The first script skips pipelines already in the store, so delete app/engine/rag_store.db first for a full rebuild; the second replaces the deep-pattern rows (channel idioms, parameter schemas, resource closures) of its thirty flagship pipelines on every run. 5. Test-asset cache. When the cache tree changed, pack it again: tar czf app/engine/test_asset_cache.tar.gz -C "${CODARIS_HOME:-$HOME/.codaris}" test_asset_cache. 6. Publish the release. Choose the next stores version and upload the files that changed. A harvest changes three databases and produces the corpus; rag_store.db and the test-asset cache are uploaded only when they were rebuilt:

   gh release create stores-v1.2.0 \
     app/engine/codaris_registry.db app/engine/container_store.db app/engine/tool_parameters.db \
     /tmp/nfcore_corpus.json \
     --title "Codaris stores 1.2.0" --notes "nf-core/modules <commit>, harvested $(date +%Y-%m-%d)."
   

2. Pin the release in the manifest. refresh_manifest rewrites each refreshed file's sha256 and size from the file on disk, moves stores_version, and records the modules commit and the corpus:

   python -m app.engine.refresh_manifest --version 1.2.0 \
     --modules-sha <commit from step 1> --corpus /tmp/nfcore_corpus.json --components <count>
   python -m app.engine.refresh_manifest --version 1.2.0 --modules-sha <commit> --corpus /tmp/nfcore_corpus.json --components <count> --check
   

   A file the release did not re-upload (rag_store.db, the test-asset cache) gains "version": "1.1.1", the release that still holds it, so a fetch looks for it there and not under the new tag. upstream_modules_sha is the commit from step 1; the compiler passes it to nf-core modules install --sha, so a build installs the modules these stores describe and not whatever the local modules clone holds at its head (CODARIS_NFCORE_MODULES_SHA overrides it, and head lifts it). corpus records the corpus file's checksum and component count; it is published beside the databases but is not one of the files a fetch installs. python -m app.cli stores check reports every file as ok once the manifest matches. Commit the manifest and push; the verify workflow fetches the new release and runs the suite against it.

3. Publish the image. gh workflow run docker_publish.yml -f tag=v1.0.1 -f push_latest=true builds the image with the new stores. Deployments pick them up with docker compose -f docker/docker-compose.unified.yml pull && docker compose -f docker/docker-compose.unified.yml up -d; source checkouts run ./bin/codaris stores fetch.

The sha256 pins are what make a build reproducible: two builds of the same commit carry the same stores, and a refresh is always a deliberate release.

The scheduled harvest¶

.github/workflows/harvester_sync.yml runs every 48 hours and on demand. It fetches the released stores first (a checkout holds none of them, and without them the harvest would lack the knowledge graph, the containers table and the presets), clones nf-core/modules, runs harvester --fresh, builds the corpus, runs refresh_manifest (with --check), and uploads the result as the harvested-stores artifact: the three databases, nfcore_corpus.json, and a stores_manifest.json that already points at them. The job summary prints the gh release create line. The workflow holds a read-only token and publishes nothing; a maintainer uploads the files, commits the manifest, and publishes the image. Run it by hand with a stores_version to name the release; without one it proposes the next minor version.