Group not respected when used with matrix in .drone.yaml

Drone Support
1

Should group take higher precedence than matrix?

Drone file looks like this:

matrix:
  include:
    - ENV11: val11
      ENV21: val21
    - ENV12: val12
      ENV22: val22

pipeline:
  step1:
    group: group1
    ... 

  step2:
    group: group2
    ...

I see that step1 and step2 are executed in parallel instead of step1 run first and then step2.

2

Groups are limited to each matrix build, so this is an expected behavior.

3

@tboerger is correct, the group attribute will not control matrix concurrency. The matrix and parallel features are separate features, used to solve two different use cases. Let me provide a bit more detail to help better clarify –

The use case for matrix pipelines are testing multiple configurations in parallel. For example, you want to test your code using Go 1.5, Go 1.6, and Go 1.7. A matrix configuration essentially creates multiple copies of a pipeline, one for each permutation, and then executes each permutation in parallel. They execute completely independently of one another. This means there is no way for one matrix permutation (e.g. Go 1.5) to wait for or yield to another matrix permutation (e.g. Go 1.6).

The use case for parallel steps (groups) are when you want a single pipeline execution to fan out and then fan back in. For example, maybe you want your Go 1.7 unit tests to run in parallel so that your pipeline executes more quickly.

It is possible to use these features, matrix steps and parallel step execution, together. For example, imagine you wanted to test your codebase against Go 1.5, 1.6, and 1.7. That would be a matrix build. Now image you have a large test suite. You could parallelize your unit tests for each version of Go.

The yaml might look something like this:

clone:
  default:
    image: plugins/git

pipeline:
  test_unit:
    image: golang:${go_version}
    commands: 
      - go test ./unit/...
  test_integration:
    image: golang:${go_version}
    commands: 
      - go test ./integration/...
  notify:
    image: plugins/slack
    where:
      matrix:
        golang: 1.7

matrix:
  go_version:
    - 1.7
    - 1.8

So how will the above yaml execute? We can visualize what this would look like below:

05 PM
05 PM1312×900 18.7 KB

I hope that helps provide a bit more information about these two independent features, how they work, and how they can potentially be used together.

4

Thanks @tboerger and @bradrydzewski.

@bradrydzewski: Regarding your example. what would I add to the .drone.yaml if I wanted to execute a step just before Finish that runs after all previous tasks are complete? (in essence sequentially)

1 Like
5

@ptagr and @bradrydzewski would be interested on the same feature. We have some very IO-bound steps which we like to run on different agents using matrix builds and then only in the case they all succeed run certain steps after them. Could this be achieved with a custom plugin ?

6

@pyry there is really no way to enable this behavior, even with plugins.

Matrix builds are intended for testing different configurations (multiple versions of redis, node, etc). Using matrix builds to parallelize tasks across machines is more of a hack, since this is not the intended use case. There is an open issue for tracking proper multi-machine parallelism that you can subscribe to for updates at https://github.com/drone/drone/issues/1814