Skip to content

Exhausting single element channels

Open in Gitpod

Problem

Processes that accept more than one argument, will be executed as many times as the smaller number of elements in either channel, for example, if you have a channel with five elements and another with two elements, the process will be called twice. If you have a channel with a single element, it is not reused automatically but the process is executed just once. This can be surprising because nextflow in DSL2 copies channels as needed.

problem.nf
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
#!/usr/bin/env nextflow

nextflow.enable.dsl = 2

/*******************************************************************************
 * Define processes
 ******************************************************************************/

process ECHO {
    input:
    val(number)
    val(data)

    """
    echo ${number}-${data}
    """
}

/*******************************************************************************
 * Define main workflow
 ******************************************************************************/

workflow {
    ch_variadic = Channel.of(1..20)
    ch_singleton = Channel.of('arg')

    ECHO(ch_variadic, ch_singleton)
}

Run the above workflow with:

NXF_VER='21.10.6' nextflow run examples/singleton-channel/problem.nf

which gives the following output, the process is just called once.

executor >  local (1)
[b5/7c55cf] process > ECHO (1) [100%] 1 of 1 ✔

Solution

Channels can be turned into value channels which can never be exhausted and read an unlimited number of times. A simple way to do this is by applying the operator first.

solution.nf
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
#!/usr/bin/env nextflow

nextflow.enable.dsl = 2

/*******************************************************************************
 * Define processes
 ******************************************************************************/

process ECHO {
    input:
    val(number)
    val(data)

    """
    echo ${number}-${data}
    """
}

/*******************************************************************************
 * Define main workflow
 ******************************************************************************/

workflow {
    ch_variadic = Channel.of(1..20)
    ch_singleton = Channel.of('arg')

    ECHO(ch_variadic, ch_singleton.first())
}

You can see the difference by running the workflow.

NXF_VER='21.10.6' nextflow run examples/singleton-channel/solution.nf

executor >  local (20)
[02/fa180f] process > ECHO (20) [100%] 20 of 20 ✔

Please note the following admonition from the nextflow documentation:

Note

A value channel is implicitly created by a process when an input specifies a simple value in the from clause. Moreover, a value channel is also implicitly created as output for a process whose inputs are only value channels.

This means that a process that gets passed a value and, for example, downloads a file, implicitly has a value channel created for that file and it can be reused indefinitely.

Combinations

If you have multiple channels of different numbers of elements but more than one element such that a value channel is not an option, you can apply some transformations to achieve the correct outcome.

combinations.nf
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
#!/usr/bin/env nextflow

nextflow.enable.dsl = 2

/*******************************************************************************
 * Define processes
 ******************************************************************************/

process ECHO {
    input:
    val(number)
    val(data)

    """
    echo ${number}-${data}
    """
}

/*******************************************************************************
 * Define main workflow
 ******************************************************************************/

workflow {
    ch_long = Channel.of(1..20)
    ch_short = Channel.of(31..33)

    ch_combined = ch_long
        .combine(ch_short)
        .multiMap {
            first: it[0]
            second: it[1]
        }

    ECHO(ch_combined.first, ch_combined.second)
}

In this workflow, combine does most of the work, as it combines each element from one channel with every element of the other channel. Afterwards, we use multiMap to split the pairs into two separate channels that we can pass to the process which expects two arguments.

You can see the difference by running the workflow.

NXF_VER='21.10.6' nextflow run examples/singleton-channel/combinations.nf

Combinatorics, wheee

executor >  local (60)
[09/71172f] process > ECHO (60) [100%] 60 of 60 ✔