Nextflow Examples

This directory contains example Nextflow workflows and configuration files.

Files

simple_pipeline.nf - A basic ETL pipeline workflow demonstrating process chaining
nextflow.config - Configuration file with execution profiles and settings
README.md - This file

Setup

Install Nextflow:

# Using conda
conda install -c bioconda nextflow
   
# Or download directly
curl -s https://get.nextflow.io | bash
chmod +x nextflow
sudo mv nextflow /usr/local/bin/

Verify installation:
```
nextflow -version
```
Ensure Java is installed (Java 8+ required):
```
java -version
```

Running the Example

Basic Execution

Run the simple pipeline:

nextflow run simple_pipeline.nf --input data.txt

If no input file is specified, the workflow will create a sample data file automatically.

With Custom Parameters

nextflow run simple_pipeline.nf \
    --input mydata.txt \
    --output results \
    --work work_dir

Using Execution Profiles

The nextflow.config file includes several execution profiles:

Local execution (default):

nextflow run simple_pipeline.nf -profile local

SLURM cluster:

nextflow run simple_pipeline.nf -profile slurm

Docker containers:

nextflow run simple_pipeline.nf -profile docker

Apptainer containers (recommended):

nextflow run simple_pipeline.nf -profile apptainer

Singularity containers (legacy, still supported):

nextflow run simple_pipeline.nf -profile singularity

Understanding the Workflow

Process Structure

The simple_pipeline.nf workflow consists of three processes:

extract - Reads input data and extracts a sample
transform - Transforms the data (multiplies numbers by 2)
load - Generates summary statistics

Process Chaining

Processes are connected using channels:

extract(input_ch)
transform(extract.out.extracted)
load(transform.out.transformed)

Input/Output

Inputs are declared with the input: block
Outputs are declared with the output: block and emit: keyword
Data flows through channels between processes

Configuration

The nextflow.config file provides:

Execution profiles for different environments (local, SLURM, PBS, Docker, Apptainer/Singularity)
Resource defaults (CPU, memory, time limits)
Error handling settings
Reporting configuration (HTML reports, timeline, trace)

Customize the configuration by editing nextflow.config or override settings via command-line parameters.

Workflow Features

Resume Execution

If a workflow fails, you can resume it:

nextflow run simple_pipeline.nf -resume

This will skip successfully completed processes.

Viewing Results

Results are published to the output directory (default: results/). The workflow also generates:

HTML Report - results/report.html - Visual workflow execution report
Timeline - results/timeline.html - Timeline visualization
Trace - results/trace.txt - Detailed execution trace

Monitoring Execution

Nextflow provides real-time execution monitoring:

Progress information in the terminal
Execution logs in the work directory
HTML reports for detailed analysis

Advanced Usage

Creating Custom Profiles

Add custom profiles to nextflow.config:

profiles {
    myprofile {
        process.executor = "local"
        process.cpus = 8
        process.memory = "32 GB"
    }
}

Use with:

nextflow run simple_pipeline.nf -profile myprofile

Using Containers

Enable container execution:

# Docker
nextflow run simple_pipeline.nf -with-docker

# Apptainer (recommended)
nextflow run simple_pipeline.nf -with-apptainer

# Singularity (legacy, still supported)
nextflow run simple_pipeline.nf -with-singularity

Custom Work Directory

Specify a custom work directory:

nextflow run simple_pipeline.nf -w /path/to/work

Troubleshooting

Common Issues

Java not found: Install Java 8 or later
Permission denied: Ensure Nextflow is executable (chmod +x nextflow)
Process failures: Check logs in the work/ directory
Memory issues: Adjust process.memory in config or use -with-memory

Getting Help

Check Nextflow logs in the work directory
Review the HTML report for detailed error information
Consult the Nextflow Documentation

Additional Resources

For comprehensive documentation, tutorials, and additional resources, see the Nextflow documentation page.