Skip to content

Modifying mutable elements

Open in Gitpod

Problem

When working with nf-core modules, a ubiquitous pattern is to pass around sample metadata as a maps. Maps are mutable objects. In nextflow DSL2, you can reuse a channel. This creates a shallow copy of the channel's elements which can lead to surprising behavior in combination with mutable objects.

To see this in action run the first example.

echo.nf
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
process ECHO {
    input:
    val meta

    output:
    val meta

    """
    echo '${meta.id}'
    """
}
problem.nf
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
#!/usr/bin/env nextflow

nextflow.enable.dsl = 2

/*******************************************************************************
 * Import processes
 ******************************************************************************/

include {
    ECHO as ECHO1;
    ECHO as ECHO2;
} from './echo'

/*******************************************************************************
 * Define main workflow
 ******************************************************************************/

workflow {
    def ch_input = Channel.of([id: 'test1', idx: 1], [id: 'test2', idx: 2])

    ECHO1(ch_input).view()

    ECHO2(
        ch_input.map { meta ->
            meta.id = 'foo'
            meta
        }
    ).view()
}

Notice that we are modifying the id attribute of the maps.

NXF_VER='21.10.6' nextflow run examples/shallow-copy/problem.nf

executor >  local (4)
[10/bbc389] process > ECHO1 (2) [100%] 2 of 2 ✔
[fa/3f7585] process > ECHO2 (2) [100%] 2 of 2 ✔
[id:foo, idx:1]
[id:foo, idx:1]
[id:foo, idx:2]
[id:foo, idx:2]

The channel ch_input contains two elements which are both maps. The channel is passed to a process ECHO1 and then reused and modified before being passed to the process ECHO2. Intuitively, we would expect the reuse to copy the channel and thus be independent of the first use. However, due to nextflow being asynchronous and shallow copying the channels, we can see that all maps are modified.

Solution

In order to achieve the desired outcome of the reused channel being independent of the first use, we need to clone the mutable element. This then creates a shallow of the mutable element itself which can be modified independently.

problem.nf
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
#!/usr/bin/env nextflow

nextflow.enable.dsl = 2

/*******************************************************************************
 * Import processes
 ******************************************************************************/

include {
    ECHO as ECHO1;
    ECHO as ECHO2;
} from './echo'

/*******************************************************************************
 * Define main workflow
 ******************************************************************************/

workflow {
    def ch_input = Channel.of([id: 'test1', idx: 1], [id: 'test2', idx: 2])

    ECHO1(ch_input).view()

    ECHO2(
        ch_input.map { meta ->
            def copy = meta.clone()
            copy.id = 'foo'
            return copy
        }
    ).view()
}

To see the outcome, run the following:

NXF_VER='21.10.6' nextflow run examples/shallow-copy/solution.nf

executor >  local (4)
[0e/0b1e55] process > ECHO1 (1) [100%] 2 of 2 ✔
[f6/4e9656] process > ECHO2 (2) [100%] 2 of 2 ✔
[id:foo, idx:1]
[id:test2, idx:2]
[id:test1, idx:1]
[id:foo, idx:2]

In some cases where you have nested mutable objects you may have to create a deep copy.