# How to setup parallel Pipeline steps (1.0+)

**URL:** <https://drone.discourse.group/t/how-to-setup-parallel-pipeline-steps-1-0/10840>\
**Category:** Drone FAQ\
**Created:** [December 17, 2018, 9:17pm UTC](https://drone.discourse.group/t/how-to-setup-parallel-pipeline-steps-1-0/10840 "2018-12-17T21:17:52Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![bradrydzewski](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/bradrydzewski/32/3513_2.png) [@bradrydzewski](https://drone.discourse.group/u/bradrydzewski)\
**Post date:** [December 17, 2018, 9:17pm UTC](https://drone.discourse.group/t/how-to-setup-parallel-pipeline-steps-1-0/10840/1 "2018-12-17T21:17:52Z")

</div>

Pipeline steps are executed sequentially by default. You can optionally use the `depends_on` keywork to describe your build steps as a [directed acyclic graph](https://en.wikipedia.org/wiki/Directed_acyclic_graph). In the below example, steps `foo` and `bar` execute in parallel, and `baz` executes once both steps complete.

```plaintext
kind: pipeline
name: foo

steps:
- name: foo
  image: golang
  commands:
  - go build
  - go test

- name: bar
  image: node
  commands:
  - npm install
  - npm test

- name: baz
  image: plugins/slack
  settings:
    webhook:
      from_secret: webhook
  depends_on:
  - foo
  - bar

```

The above example is quite simple, but you could use this syntax to create very complex execution flows. The below diagram provides a visual representation of what a complex graph execution could look like:

![Topological_Ordering](https://canada1.discourse-cdn.com/flex003/uploads/testdrone/original/2X/1/114900a481e0cc2bb83c296f00f3b440e05d3995.png)

---

<div class="post-metadata">

**Author:** ![bradrydzewski](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/bradrydzewski/32/3513_2.png) [@bradrydzewski](https://drone.discourse.group/u/bradrydzewski)\
**Post date:** [December 17, 2018, 9:58pm UTC](https://drone.discourse.group/t/how-to-setup-parallel-pipeline-steps-1-0/10840/2 "2018-12-17T21:58:43Z")

</div>

If you were using the `group` keyword in previous versions of Drone, you should be able to emulate the same behavior with `depends_on`.

Example 0.8 yaml configuration using groups:

```nohighlight
pipeline:
  step1a:
    image: alpine
    group: group1
  step1b:
    image: alpine
    group: group1

  step2a:
    image: alpine
    group: group2
  step2b:
    image: alpine
    group: group2

  step3:
    image: alpine
    # no group

```

The above configuration file can be represented in 1.0 using the following syntax:

```nohighlight
kind: pipeline

steps:
- name: step1a
  image: alpine
- name: step1b
  image: alpine

- name: step2a
  image: alpine
  depends_on: [step1a, step1b]
- name: step2b
  image: alpine
  depends_on: [step1a, step1b]

- name: step3
  image: alpine
  depends_on: [step2a, step2b]

```

---

<div class="post-metadata">

**Author:** ![907th](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/907th/32/5972_2.png) [@907th](https://drone.discourse.group/u/907th)\
**Post date:** [December 19, 2018, 9:51am UTC](https://drone.discourse.group/t/how-to-setup-parallel-pipeline-steps-1-0/10840/3 "2018-12-19T09:51:49Z")

</div>

It seems like `depends_on` on 3rd task implicitly declares tasks 1 and 2 to be executed in parallel. Am I right? This implicitness is very confusing IMO.

---

<div class="post-metadata">

**Author:** ![shynome](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/shynome/32/3574_2.png) [@shynome](https://drone.discourse.group/u/shynome)\
**Post date:** [January 4, 2019, 4:02am UTC](https://drone.discourse.group/t/how-to-setup-parallel-pipeline-steps-1-0/10840/4 "2019-01-04T04:02:00Z")

</div>

if I want let all step run at the same time , I have to create a useless step with a depends\_on include a step name. it don’t seem good , any others better resolve way ?

---

<div class="post-metadata">

**Author:** ![techknowlogick](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/techknowlogick/32/4245_2.png) [@techknowlogick](https://drone.discourse.group/u/techknowlogick)\
**Post date:** [January 12, 2019, 2:31am UTC](https://drone.discourse.group/t/how-to-setup-parallel-pipeline-steps-1-0/10840/5 "2019-01-12T02:31:15Z")

</div>

@shynome

```auto
steps:
- name: one
 image: alpine
 depends_on: [clone]
- name: two
 image: alpine
 depends_on: [clone]
- name: three
 image: alpine
 depends_on: [clone] 

```

---

<div class="post-metadata">

**Author:** ![shynome](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/shynome/32/3574_2.png) [@shynome](https://drone.discourse.group/u/shynome)\
**Post date:** [January 12, 2019, 8:38am UTC](https://drone.discourse.group/t/how-to-setup-parallel-pipeline-steps-1-0/10840/6 "2019-01-12T08:38:54Z")

</div>

Thank you, it’s helpful

---

<div class="post-metadata">

**Author:** ![bradrydzewski](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/bradrydzewski/32/3513_2.png) [@bradrydzewski](https://drone.discourse.group/u/bradrydzewski)\
**Post date:** [January 13, 2019, 1:16am UTC](https://drone.discourse.group/t/how-to-setup-parallel-pipeline-steps-1-0/10840/8 "2019-01-13T01:16:08Z")

</div>

See [https://docs.drone.io/user-guide/pipeline/multi-machine/](https://docs.drone.io/user-guide/pipeline/multi-machine/)

---

<div class="post-metadata">

**Author:** ![vovimayhem](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/vovimayhem/32/4343_2.png) [@vovimayhem](https://drone.discourse.group/u/vovimayhem)\
**Post date:** [January 13, 2019, 1:16am UTC](https://drone.discourse.group/t/how-to-setup-parallel-pipeline-steps-1-0/10840/9 "2019-01-13T01:16:33Z")

</div>

Oh sorry! I’ve just read that too!

---

<div class="post-metadata">

**Author:** ![hughsw](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/hughsw/32/4591_2.png) [@hughsw](https://drone.discourse.group/u/hughsw)\
**Post date:** [May 3, 2019, 7:37pm UTC](https://drone.discourse.group/t/how-to-setup-parallel-pipeline-steps-1-0/10840/10 "2019-05-03T19:37:57Z")

</div>

AFAICT, and related to @907 concerns, as soon as you use `depends_on` in one place, everything that is declared later in the file without a `depends_on` runs in parallel. This means that you have to construct a DAG for all the steps that follow the first use of `depends_on`, even if you just want a few steps in the middle to run in parallel.

As it stands right now, to linearize the steps following some parallel piece, each step has to link to the previous step by name – definitely an anti-pattern smearing of concerns…

The generality of `depends_on` certainly has more power than `group`, but it is in no way a simple replacement for `group`.

At the risk of introducing unintended semantics, I could imagine that a `depends_on_previous: null` declaration could be used to turn off the parallelism for subsequent steps…

---

<div class="post-metadata">

**Author:** ![hughsw](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/hughsw/32/4591_2.png) [@hughsw](https://drone.discourse.group/u/hughsw)\
**Post date:** [May 3, 2019, 7:56pm UTC](https://drone.discourse.group/t/how-to-setup-parallel-pipeline-steps-1-0/10840/11 "2019-05-03T19:56:44Z")

</div>

I’ve now struggled to restore the parallel behavior we used to enjoy with `group` – PITA. E.g. the semantics of `depends_on` are not clear, e.g. when it refers to a build step that isn’t being run due to `when`… A step should not have to worry about other steps.

The more I think about this, the more I think that the most reasonable way to allow parallelism within a build would be to declare the DAG separately, giving each node in the DAG a name. A node would be a placeholder for steps to run in parallel. Each step would have to declare which node it belongs to. All steps in a node would run in parallel (so a node is like a named `group`), and the DAG of nodes would determine the constraints among the nodes of parallel steps. Empty nodes (due to `when`) would be epsilons.

---

<div class="post-metadata">

**Author:** ![bradrydzewski](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/bradrydzewski/32/3513_2.png) [@bradrydzewski](https://drone.discourse.group/u/bradrydzewski)\
**Post date:** [May 3, 2019, 9:14pm UTC](https://drone.discourse.group/t/how-to-setup-parallel-pipeline-steps-1-0/10840/12 "2019-05-03T21:14:15Z")

</div>

There is prior art for our design decisions, inspired by the google cloud builder syntax. I understand some people dislike depends\_on while others are happily using it. The point is moot since we are not making any breaking changes to the yaml now that 1.0 is released.

---

<div class="post-metadata">

**Author:** ![mvdan](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/mvdan/32/4188_2.png) [@mvdan](https://drone.discourse.group/u/mvdan)\
**Post date:** [May 17, 2019, 1:58pm UTC](https://drone.discourse.group/t/how-to-setup-parallel-pipeline-steps-1-0/10840/13 "2019-05-17T13:58:22Z")

</div>

@bradrydzewski mixing sequential and parallel steps is now a bit painful, with depends\_on. How about allowing something like:

```auto
depends_on: previous

```

Otherwise the list of steps is quite fragile, as each sequential step has to contain the name of the previous one.

---

<div class="post-metadata">

**Author:** ![mvdan](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/mvdan/32/4188_2.png) [@mvdan](https://drone.discourse.group/u/mvdan)\
**Post date:** [May 17, 2019, 2:02pm UTC](https://drone.discourse.group/t/how-to-setup-parallel-pipeline-steps-1-0/10840/14 "2019-05-17T14:02:56Z")

</div>

On second thought, depending on the previous step is sometimes wasteful on the overall run time. Forcing me to declare the actual dependency is fine, I guess.

---

<div class="post-metadata">

**Author:** ![annProg](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/annprog/32/4742_2.png) [@annProg](https://drone.discourse.group/u/annProg)\
**Post date:** [June 26, 2019, 4:08pm UTC](https://drone.discourse.group/t/how-to-setup-parallel-pipeline-steps-1-0/10840/15 "2019-06-26T16:08:39Z")

</div>

how to limit parallel max numbers?

---

<div class="post-metadata">

**Author:** ![hughsw](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/hughsw/32/4591_2.png) [@hughsw](https://drone.discourse.group/u/hughsw)\
**Post date:** [July 26, 2019, 9:00pm UTC](https://drone.discourse.group/t/how-to-setup-parallel-pipeline-steps-1-0/10840/16 "2019-07-26T21:00:38Z")

</div>

I’ve come around to enjoying what `depends_on:` enables, and I have implemented a complex DAG for our build/test/deploy/test setup.

I do have one request that relates to my initial (and continuing) difficulties. The problem occurs if a `when:` eliminates a job from the DAG. The missing job breaks the chain of dependencies, and downstream jobs start too soon (as if they have an epsilon to the start node).

Request:  
Could you please expand a node’s dependencies to the transitive closure of its parents **before** `when:` is used to eliminate arcs from the DAG. I believe this will result in far few surprises and lets dependencies be “local”, focussing on immediate predecessors. I am hard-pressed to think of working examples that will be broken by this change (but I’m sure a pathological one can be dreamed up) – anyone who’s run into this problem will have coded the transitive closure by hand already, as I have done.

---

<div class="post-metadata">

**Author:** ![bradrydzewski](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/bradrydzewski/32/3513_2.png) [@bradrydzewski](https://drone.discourse.group/u/bradrydzewski)\
**Post date:** [July 26, 2019, 9:06pm UTC](https://drone.discourse.group/t/how-to-setup-parallel-pipeline-steps-1-0/10840/17 "2019-07-26T21:06:41Z")

</div>

I cannot reproduce. Maybe you are running an older version? I have the below pipeline where the second step is skipped (does not match branch) and the third step depends on the second step:

```nohighlight
kind: pipeline
name: test

steps:
- name: foo
  image: alpine:3.8
  commands:
  - echo foo

- name: bar
  image: alpine:3.8
  commands:
  - echo bar
  depends_on: [foo]
  when:
    branch: [develop]

- name: baz
  image: alpine:3.8
  commands:
  - echo baz
  depends_on: [bar]

```

The scheduler handles the dependency correctly and runs the steps in the expected order, despite the step being skipped:

```auto
$ drone exec
[foo:0] + echo foo
[foo:1] foo
[baz:0] + echo baz
[baz:1] baz

```

If you look at the underlying implementation you will notice that Drone visits every single node in the graph. It does not alter the graph or break the chain in any way. Here is some pseudo code to illustrate.

```nohighlight
run := func(s *Step) error {
  if s.When(...) == true { // step bar will not match
    return s.Exec()
  }
  return nil
}

var d dag.Runner
d.AddVertex("foo", run)
d.AddVertex("bar", run) // added to graph despite being skipped
d.AddVertex("baz", run)

d.AddEdge("bar", "foo")
d.AddEdge("baz", "bar")

d.Run()

```

---

<div class="post-metadata">

**Author:** ![hughsw](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/hughsw/32/4591_2.png) [@hughsw](https://drone.discourse.group/u/hughsw)\
**Post date:** [July 26, 2019, 9:15pm UTC](https://drone.discourse.group/t/how-to-setup-parallel-pipeline-steps-1-0/10840/18 "2019-07-26T21:15:39Z")

</div>

The code was written for ver 1.1.0, and we are now using 1.2.1. I will look into whether the issue still exists.

---

<div class="post-metadata">

**Author:** ![bradrydzewski](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/bradrydzewski/32/3513_2.png) [@bradrydzewski](https://drone.discourse.group/u/bradrydzewski)\
**Post date:** [July 26, 2019, 9:23pm UTC](https://drone.discourse.group/t/how-to-setup-parallel-pipeline-steps-1-0/10840/19 "2019-07-26T21:23:44Z")

</div>

I recommend providing a _simplified_ example to reproduce the problem because I cannot reproduce and my time is scarce. If you can provide a simple way to reproduce I would be happy to dig deeper.

---

<div class="post-metadata">

**Author:** ![hughsw](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/hughsw/32/4591_2.png) [@hughsw](https://drone.discourse.group/u/hughsw)\
**Post date:** [July 27, 2019, 2:02am UTC](https://drone.discourse.group/t/how-to-setup-parallel-pipeline-steps-1-0/10840/20 "2019-07-27T02:02:58Z")

</div>

Well…, fabulous! Simple DAGs don’t have the problem, and, after removing all the hand-curated closures in our `.drone.yml`, our CICD jobs still work!

So thank you for getting it right!

DAGs from before and after removing the hand-curated (`when:`-aware) closures:

 ![image](https://canada1.discourse-cdn.com/flex003/uploads/testdrone/original/2X/a/a336c8ed4b0786fcd871a687fbb59c022a5fe832.png)

---

<div class="post-metadata">

**Author:** ![fhp](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/fhp/32/5851_2.png) [@fhp](https://drone.discourse.group/u/fhp)\
**Post date:** [February 20, 2020, 11:20am UTC](https://drone.discourse.group/t/how-to-setup-parallel-pipeline-steps-1-0/10840/21 "2020-02-20T11:20:19Z")

</div>

I would also like the option to limit parallelism.  
Right now I am building a mono-repo, that has 8 parallel build steps. However, as the drone worker handles 4 pipelines at once, using more than 2 processes means that the other workers on the same machine suffer from cpu starvation, sometimes leading to test timeouts.  
Having a ‘maxParallelism: 2’ option would help here, to limit the pipeline to execute at most 2 steps at once.

[Next page](https://drone.discourse.group/t/how-to-setup-parallel-pipeline-steps-1-0/10840.md?page=2)