Skip to content

HPC and Cloud Execution

A pipeline built by Codaris is an nf-core pipeline, so it runs wherever nf-core pipelines run. The portability comes from the nf-core template the build starts from; Codaris adds a self-contained test profile and leaves the executor choice to the configuration you launch with.

1. What the generated pipeline carries

  • Software profiles from the template: docker, singularity, apptainer, podman, conda, and the others nf-core ships, selected with -profile.
  • Resource labels in conf/base.config: process_single, process_low, process_medium, process_high, process_long, and process_high_memory, each with CPUs, memory, and time that scale with task.attempt. The template's retry strategy resubmits a task that exits with the standard resource codes (out of memory, time limit) at the next attempt. Caps are expressed with Nextflow's resourceLimits; the check_max helper of older templates is not used.
  • Institutional profiles through nf-core/configs: the template includes nfcore_custom.config, so -profile <institute> (for example -profile uppmax or -profile cheaha) applies that site's executor, queues, and container settings. Set NXF_OFFLINE=true or --custom_config_base for air-gapped hosts.
  • Two test profiles: conf/test.config from the template and conf/test_codaris.config, which points every input at fixtures under assets/test_fixtures/ and caps each task at 2 CPUs, 4 GB, and 1 hour:
process {
    resourceLimits = [
        cpus: 2,
        memory: '4.GB',
        time: '1.h'
    ]
}

Run it with nextflow run . -profile test_codaris,docker (add -stub-run for a seconds-long check without containers).

2. Slurm with Singularity or Apptainer

When you answered it on the intent card ("Executor: Slurm", "Container: Singularity"), or said it in the request, the build writes it for you: a slurm profile (also sge, lsf, pbs, pbspro) at the end of nextflow.config, the container engine's profile if the template lacks it, and these params, none required: --cluster_queue, --cluster_account, --cluster_options, --cluster_max_cpus, --cluster_max_memory, --cluster_max_time. The cluster limits replace the cap that conf/codaris_resources.config sets from the machine the pipeline was built on; the Docker test profile keeps it.

nextflow run . -profile slurm,singularity --cluster_queue general --cluster_account my-account \
  --cluster_max_memory 256.GB --cluster_max_time 72.h --input samplesheet.csv --outdir ./results

If the request also gave the size of the run (24 or more samples, or more than 10 M reads a sample), conf/base.config ends with a block marked Codaris scale: the labels' time grows with the read depth (and memory on the medium and high labels past 30 M reads), maxRetries is 3 so the template's retry of exits 137 and 140 has room to escalate, per-sample QC (FastQC) is right-sized, and each long step the plan installed (an aligner, a quantifier, MAGECK_COUNT, any process_medium or process_high module) has its own withName to tune. Delete the block to go back to the template's values. A small local run is left as the template made it.

If you have no card answer, launch with an institutional profile when your site has one in nf-core/configs. Otherwise supply a configuration file with -c:

// slurm.config (supplied by you)
process {
    executor       = 'slurm'
    queue          = 'general'
    clusterOptions = '--account=my-account'
    resourceLimits = [ cpus: 32, memory: '256.GB', time: '72.h' ]
}
singularity {
    enabled    = true
    autoMounts = true
    cacheDir   = "${System.getenv('HOME')}/.singularity/cache"
}

nextflow run . -profile singularity -c slurm.config \
  --input samplesheet.csv --outdir ./results

resourceLimits is where the cluster's ceilings go; the template's labels scale under them.

3. AWS Batch

// aws.config (supplied by you)
workDir = 's3://my-bio-bucket/work'
process {
    executor = 'awsbatch'
    queue    = 'arn:aws:batch:us-east-1:123456789012:job-queue/nextflow-queue'
}
aws {
    region = 'us-east-1'
    batch {
        cliPath = '/home/ec2-user/miniconda/bin/aws'
    }
}

nextflow run . -profile docker -c aws.config \
  --input s3://my-bio-bucket/inputs/samplesheet.csv \
  --outdir s3://my-bio-bucket/results/

4. Google Cloud Batch

// gcp.config (supplied by you)
workDir = 'gs://my-bucket/work'
process.executor = 'google-batch'
google {
    project  = 'my-genomics-project'
    location = 'us-central1'
}

nextflow run . -profile docker -c gcp.config \
  --input gs://my-bucket/samplesheet.csv \
  --outdir gs://my-bucket/results/

5. Where Codaris helps after the launch

  • Readiness check. In the studio, asking to run the test profile triggers a readiness check first: the host's CPUs and memory against the profile's limits, the container engine, and the sample sheet.
  • Log triage. analyze_run_log classifies a failed run (out of memory, time limit, missing command, missing file, container failure) and proposes the targeted change; resource failures are resumed with -resume after the fix.
  • Run tracking. Runs launched from the studio are recorded in runs.db under the state directory, with their status and the Nextflow report paths, and appear on the Runs page.

The compute itself happens where Nextflow puts it; Codaris does not need to be installed on the cluster or in the cloud account.