Exhausting single element channels

Problem

Processes that accept more than one argument, will be executed as many times as the smaller number of elements in either channel, for example, if you have a channel with five elements and another with two elements, the process will be called twice. If you have a channel with a single element, it is not reused automatically but the process is executed just once. This can be surprising because nextflow in DSL2 copies channels as needed.

#!/usr/bin/env nextflow

nextflow.enable.dsl = 2

/*******************************************************************************
 * Define processes
 ******************************************************************************/

process ECHO {
    input:
    val(number)
    val(data)

    """
    echo ${number}-${data}
    """
}

/*******************************************************************************
 * Define main workflow
 ******************************************************************************/

workflow {
    ch_variadic = Channel.of(1..20)
    ch_singleton = Channel.of('arg')

    ECHO(ch_variadic, ch_singleton)
}

Run the above workflow with:

NXF_VER='21.10.6' nextflow run examples/singleton-channel/problem.nf

which gives the following output, the process is just called once.

executor >  local (1)
[b5/7c55cf] process > ECHO (1) [100%] 1 of 1 ✔

Solution

Channels can be turned into dataflow values which can never be exhausted and read an unlimited number of times. A simple way to do this is by applying the operator first.

#!/usr/bin/env nextflow

nextflow.enable.dsl = 2

/*******************************************************************************
 * Define processes
 ******************************************************************************/

process ECHO {
    input:
    val(number)
    val(data)

    """
    echo ${number}-${data}
    """
}

/*******************************************************************************
 * Define main workflow
 ******************************************************************************/

workflow {
    ch_variadic = Channel.of(1..20)
    ch_singleton = Channel.of('arg')

    ECHO(ch_variadic, ch_singleton.first())
}

You can see the difference by running the workflow.

NXF_VER='21.10.6' nextflow run examples/singleton-channel/solution.nf

executor >  local (20)
[02/fa180f] process > ECHO (20) [100%] 20 of 20 ✔

Please note the following admonition from the nextflow documentation:

Note

A dataflow value is implicitly created by a process when an input specifies a simple value in the from clause. Moreover, a dataflow value is also implicitly created as output for a process whose inputs are only dataflow values.

This means that a process that gets passed a value and, for example, downloads a file, implicitly has a dataflow value created for that file and it can be reused indefinitely.

Combinations

If you have multiple channels of different numbers of elements but more than one element such that a dataflow value is not an option, you can apply some transformations to achieve the correct outcome.

#!/usr/bin/env nextflow

nextflow.enable.dsl = 2

/*******************************************************************************
 * Define processes
 ******************************************************************************/

process ECHO {
    input:
    val(number)
    val(data)

    """
    echo ${number}-${data}
    """
}

/*******************************************************************************
 * Define main workflow
 ******************************************************************************/

workflow {
    ch_long = Channel.of(1..20)
    ch_short = Channel.of(31..33)

    ch_combined = ch_long
        .combine(ch_short)
        .multiMap {
            first: it[0]
            second: it[1]
        }

    ECHO(ch_combined.first, ch_combined.second)
}

In this workflow, combine does most of the work, as it combines each element from one channel with every element of the other channel. Afterwards, we use multiMap to split the pairs into two separate channels that we can pass to the process which expects two arguments.

You can see the difference by running the workflow.

NXF_VER='21.10.6' nextflow run examples/singleton-channel/combinations.nf

Combinatorics, wheee

executor >  local (60)
[09/71172f] process > ECHO (60) [100%] 60 of 60 ✔

With static typing, skip multiMap: pass the combined channel as the single channel argument and declare a tuple input. See the recommendation below.

Recommendation

Use static typing with Nextflow 26.10 or later. A typed process accepts at most one channel argument, and nextflow lint reports a call with more than one:

Error main.nf:18:5: Process `ECHO` was called with multiple channel arguments which can lead to non-deterministic behavior -- make sure that at most one argument is a channel and that all other arguments are dataflow values

Pass the other arguments as plain values or dataflow values instead:

ECHO(channel.fromList(1..20), 'arg')