Pipeline Sanitizer Agent¶

The Pipeline Sanitizer Agent is Codaris's automated hygiene and hardening engine for Nextflow DSL2 and nf-core pipelines. During interactive design, rapid prototyping, and automated repair loops, pipelines frequently accumulate host-specific file paths, debugging directives, unreferenced local modules, development credentials, and execution cache debris. The Sanitizer inspects, cleans, and seals the pipeline directory across six dimensions to ensure it is production-grade, secure, cloud-portable, and publication-ready.

Overview¶

When moving an exploratory workflow from a local workstation or development server to production HPC (SLURM, LSF, PBS) or Cloud environments (AWS Batch, Google Cloud Life Sciences, Azure Batch, Seqera Platform), hardcoded host artifacts cause silent execution failures, permission rejections, data leaks, and reproducibility breaks.

flowchart LR
    Dev["Development Pipeline<br/>• Absolute paths<br/>• Tokens & credentials<br/>• Debug .view() calls<br/>• Orphan modules<br/>• work/ cache debris"] --> Sanitize{"Codaris<br/>Pipeline Sanitizer"}
    Sanitize --> D1["1. Path Neutralization"]
    Sanitize --> D2["2. Credential & PII Scrubbing"]
    Sanitize --> D3["3. Debug Scaffolding Excision"]
    Sanitize --> D4["4. Dead Code Pruning"]
    Sanitize --> D5["5. Execution Debris Purge"]
    Sanitize --> D6["6. Publication Sealing"]
    D1 & D2 & D3 & D4 & D5 & D6 --> Prod["Production-Ready Pipeline<br/>• Cloud-portable paths<br/>• Secrets in environment<br/>• Clean stdout/stderr streams<br/>• Minimal module graph<br/>• Clean git status<br/>• Sealed provenance record"]

The Pipeline Sanitizer operates deterministically through syntax-tree and regex analysis, eliminating human oversight risks before pipeline release.

The Six Sanitization Dimensions¶

1. Path Neutralization¶

The Problem: Workstations commonly use absolute paths such as /Users/username/data/sample.fastq.gz or /home/developer/references/hg38.fa. When run on AWS Batch, Google Cloud, or a remote HPC cluster, these absolute paths fail because host directory hierarchies do not exist inside execution containers or remote worker nodes. Furthermore, hardcoded user paths leak internal infrastructure details and host usernames.

Sanitization Action: - Scans .nf workflows, .config configuration files, and samplesheet.csv metadata files for absolute paths (/Users/..., /home/..., /tmp/..., /private/tmp/..., /root/...). - Converts internal references relative to the pipeline directory using Nextflow's ${projectDir} anchor (for example, ${projectDir}/assets/samplesheet.csv). - Replaces absolute reference paths with standard parameterized bindings (such as params.fasta or params.input), allowing cloud bucket URIs (s3://..., gs://..., az://...) to be injected seamlessly at runtime. - Preserves standard system-level virtual filesystems (e.g., /dev/null, /dev/stdin).

// Before Sanitization
params.fasta = "/Users/arunbodd/genomes/hg38/Homo_sapiens.GRCh38.dna.primary_assembly.fa"
params.input = "/home/runner/work/pipeline/samplesheet.csv"

// After Sanitization
params.fasta = "${projectDir}/assets/reference.fa"
params.input = "${projectDir}/samplesheet.csv"

2. Credential & PII Scrubbing¶

The Problem: Developers often test external data pulls or cloud orchestrators by temporarily embedding API tokens, Seqera Platform (Tower) access tokens, AWS secret keys, or GitHub Personal Access Tokens (PATs) directly inside nextflow.config. Committing these files exposes critical cloud accounts to unauthorized access. Similarly, clinical or translational pipelines may accidentally commit patient IDs or Protected Health Information (PHI) within mock sample sheets.

Sanitization Action: - Detects high-entropy credential patterns and known token prefixes: - AWS Access Key IDs (AKIA[0-9A-Z]{16}) and associated secret keys. - Seqera Platform tokens (tower.accessToken = '...'). - Provider API keys (sk-..., ghp_..., github_pat_..., JWT Bearer tokens). - Explicit configuration entries for password, passwd, or db_password. - Migrates plaintext secrets to Nextflow's native secrets management syntax (secrets.MY_TOKEN) or runtime environment lookups (System.getenv('MY_TOKEN')). - Automatically replaces identifiable patient or donor keys in example sample sheets with anonymized, deterministic cohort identifiers (e.g., SAMPLE_001, SAMPLE_002).

// Before Sanitization
tower {
    accessToken = 'ey...my_secret_seqera_token...'
    enabled = true
}
aws {
    accessKey = 'AKIAIOSFODNN7EXAMPLE'
    secretKey = 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY'
}

// After Sanitization
tower {
    accessToken = secrets.TOWER_ACCESS_TOKEN ?: System.getenv('TOWER_ACCESS_TOKEN')
    enabled = true
}
aws {
    accessKey = System.getenv('AWS_ACCESS_KEY_ID')
    secretKey = secrets.AWS_SECRET_ACCESS_KEY ?: System.getenv('AWS_SECRET_ACCESS_KEY')
}

3. Debug Scaffolding Excision¶

The Problem: During workflow development, engineers routinely attach .view(), .dump(), or .println() operators to Nextflow channels to inspect tuple layouts and channel emissions. Similarly, process definitions may retain debug = true or echo = true. In production runs with thousands of samples, these directives flood standard output, generate hundreds of megabytes of log files, exhaust console buffers, and corrupt MultiQC parser outputs.

Sanitization Action: - Analyzes workflow files (main.nf, workflows/*.nf, subworkflows/*.nf) to identify channel debugging operators. - Excises .view(), .view { ... }, .dump(), and println calls while maintaining the syntactic continuity of channel chains. - Disables or removes legacy echo = true and debug = true process directives.

// Before Sanitization
FASTQC ( ch_samplesheet )
FASTQC.out.zip
    .view { meta, zip -> "Completed FastQC for: ${meta.id}" }
    .set { ch_fastqc_reports }

// After Sanitization
FASTQC ( ch_samplesheet )
FASTQC.out.zip
    .set { ch_fastqc_reports }

4. Dead Code & Orphan Asset Pruning¶

The Problem: Iterative development and automated LLM-assisted repair often create auxiliary modules under modules/local/ or utility scripts in bin/ that are replaced or abandoned in subsequent design turns. Leaving unused modules and scripts inflates repository size, causes confusion during maintenance, and triggers false positives in compliance linters.

Sanitization Action: - Parses all workflow files (main.nf, workflows/*.nf, subworkflows/*.nf) and constructs an invocation graph of all declared include blocks, process calls, and workflow definitions. - Inspects modules/local/ to identify local process modules that are never imported or invoked. - Inspects bin/ to identify standalone executable scripts (Python, Bash, R) that are never referenced inside process script: or shell: blocks. - Safely removes orphan local modules and unreferenced scripts (or reports them in dry-run mode).

5. Execution Debris Purging¶

The Problem: Running local compilation checks, readiness tests, and unit gates generates temporary Nextflow runtime artifacts: the heavy execution cache directory work/, database tracking directory .nextflow/, execution logs (.nextflow.log*), and task scratch scripts (.command.*). If committed to version control, repository sizes balloon rapidly, and internal host state leaks into Git history.

Sanitization Action: - Cleans and removes work/, .nextflow/, .nextflow.log*, and .command.* files. - Inspects .gitignore and automatically ensures all essential Nextflow execution exclusions are present: - work/ - .nextflow/ - .nextflow.log* - .nextflow* - results/

6. Publication Sealing¶

The Problem: Deploying pipelines to production or submitting them to scientific journals and institutional catalogs requires deterministic immutability. Floating container tags (such as :latest) can change without warning, and non-standard process resource directives cause task eviction on standardized HPC compute partitions.

Sanitization Action: - Standardizes resource directives across all process configurations to follow the canonical nf-core label hierarchy (process_single, process_low, process_medium, process_high). - Pins container definitions to immutable sha256 image digests or Seqera Wave frozen manifests rather than floating version tags. - Verifies and synchronizes parameter definitions between nextflow.config and nextflow_schema.json. - Calculates cryptographic SHA-256 digests across all pipeline code and configuration files, generating an immutable provenance record in .codaris/provenance.json.

CLI Usage¶

The Sanitizer is available via the codaris sanitize subcommand.

codaris sanitize <pipeline_dir> [options]

Options¶

Examples¶

1. Audit in Dry-Run Mode¶

Inspect a pipeline directory for path leaks, exposed credentials, and debug directives without making changes:

codaris sanitize ./pipelines/my-rnaseq --dry-run

Output:

Codaris Pipeline Sanitizer [DRY-RUN]
Target pipeline: /workspace/pipelines/my-rnaseq

Findings Summary:
  Paths normalized:    3
  Credentials secured: 1
  Debug code stripped: 2
  Dead code pruned:    1
  Execution debris:    0
  Gitignore updates:   1
  Total actions:       7

Files to modify: (4)
  - .gitignore
  - main.nf
  - nextflow.config
  - workflows/rnaseq.nf

Details (7 findings):
  [PATH] nextflow.config:14 — Normalized absolute host path '/Users/developer/data/hg38' to relative anchor
  [CREDENTIAL] nextflow.config:42 — Secured plaintext Tower accessToken into secrets.TOWER_ACCESS_TOKEN
  [DEBUG_CODE] workflows/rnaseq.nf:88 — Stripped '.view()' debug statement from channel pipeline
  [DEBUG_CODE] workflows/rnaseq.nf:104 — Stripped 'debug = true' directive from process configuration
  [DEAD_CODE] modules/local/unused_qc.nf — Orphan local module not referenced in any workflow
  [PATH] main.nf:22 — Normalized host path '/tmp/reads'
  [GITIGNORE] .gitignore — Added missing ignore entries: work/, .nextflow/, .nextflow.log*, results/

2. Apply Sanitization and Purge Execution Debris¶

Clean the pipeline in place, stripping debug statements and clearing cached execution artifacts:

codaris sanitize ./pipelines/my-rnaseq --purge-debris --strip-debug

3. Machine-Readable Output in CI/CD¶

Generate machine-readable JSON for integration into GitHub Actions, GitLab CI, or pre-commit hooks:

codaris sanitize ./pipelines/my-rnaseq --dry-run --json

{
  "pipeline_dir": "/workspace/pipelines/my-rnaseq",
  "dry_run": true,
  "ok": true,
  "counts": {
    "paths": 3,
    "credentials": 1,
    "debug_code": 2,
    "dead_code": 1,
    "execution_debris": 0,
    "gitignore": 1,
    "total": 7
  },
  "modified_files": [
    ".gitignore",
    "main.nf",
    "nextflow.config",
    "workflows/rnaseq.nf"
  ],
  "findings": [
    {
      "category": "path",
      "file_path": "nextflow.config",
      "line_number": 14,
      "message": "Normalized absolute host path '/Users/developer/data/hg38'",
      "original": "/Users/developer/data/hg38",
      "replacement": "${projectDir}/assets/hg38",
      "action_taken": "detected"
    }
  ],
  "exit_code": 0
}

Autopilot & Studio Integration¶

Autopilot Verification Loop¶

The Pipeline Sanitizer is integrated directly into the Codaris Autopilot driver loop (app.engine.autopilot). When Autopilot compiles, tests, and repairs a pipeline:

1. Compilation & Linting: Nextflow syntax passes the compile gate (nextflow -preview) and nf-core compliance linting.
2. Readiness & Execution: The test profile executes in a containerized environment to verify end-to-end biological and computational viability.
3. Pre-Publication Sanitization: Before the pipeline is marked as done and handed off to the user, Autopilot invokes PipelineSanitizer.sanitize_all().
4. Provenance Sealing: Once sanitized, the pipeline's final .codaris/provenance.json manifest is generated. Any unsanitized secret or lingering host path causes Autopilot to refuse final seal status.

Codaris Web Studio Integration¶

In the Codaris Web Studio interface (codaris web):

• Sanitize & Seal Action Card: The Pipeline Inspector surfaces a dedicated Sanitize & Seal action card next to the DAG and Compliance Scorecard tabs.
• Interactive Diff Preview: Users can inspect a side-by-side visual diff of all proposed modifications (neutralized paths, converted credentials, and stripped debug statements) prior to applying changes.
• Selective Toggles: Operators can selectively toggle whether to purge execution debris or keep debug scaffolding during internal iteration.
• One-Click Export: Once sanitized, the studio enables direct export to GitHub repositories, Seqera Platform workspaces, or tarball distribution packages.