Distinct workspace for parallel build steps

Drone Support
1

There are steps in my pipeline that I’d like to run in parallel. The issue is that these steps cannot share the same workspace otherwise they will fail. Is there a way to run them in distinct workspace?

I used to have a solution with MATRIX but not all my steps can be run in parallel. So I had to disable some steps for some of the matrix combinations so that they don’t run multiple times.

2

There is no native drone solution or yaml syntax to handle this. So I would pose the following question: how would you solve this using bash and running parallel tasks on your laptop? Since drone steps are just executing bash commands, they are likely to be the same solution.

For example I could envision something like this:

pipeline:
  test_suite_a:
    image: node
    commands:
      - mkdir /tmp/a
      - cp $(pwd) /tmp/a
      - cd /tmp/a
      - npm run test-suite-a
  test_suite_b:
    image: node
    commands:
      - mkdir /tmp/b
      - cp $(pwd) /tmp/b
      - cd /tmp/b
      - npm run test-suite-b
  test_suite_c
    image: node
    commands:
      - mkdir /tmp/c
      - cp $(pwd) /tmp/c
      - cd /tmp/c
      - npm run test-suite-c
1 Like
3

I have follow-up question. To control the number of parallel build, should I use DRONE_MAX_PROCS, same as for MATRIX?

4

It is not possible to throttle parallelism at this time. If you define 10 parallel steps in a group, drone will fan-out and execute all 10 steps in parallel.

5

There seems to be a 30% slowndown moving from a solution based on matrices and one based on groups. Is this expected? Copying the directory doesn’t add much time. Most of the time is spent running the tests.

6

My bad. It was a misconfiguration.

7

When you configure parallel steps, Drone is basically executing multiple docker run commands and then waiting for them to complete using docker wait. It is quite simple and lightweight, so I would not expect anything about the parallel implementation (at a code level) to cause performance to degrade.

The only reason I can imagine matrix builds providing a performance increase is when your pipeline steps are resource intensive. Matrix builds can run on multiple machines, however, as parallel steps always run on the same machine. It is therefore possible for parallel steps to starve and / or compete for resources.

8

Has this changed at all in the last 3 years, is there any native Drone solution for this? I am running into the same issue - trying to run npm install in parallel steps, but of course this fails without some way to isolate the workspaces.

9

All pipeline steps share the same workspace. There is no way to have a distinct workspace for an individual step. If you need to execute tasks in parallel with distinct workspaces, you should instead define multiple pipelines in your yaml and distribute your work across these pipelines. Each pipeline has a distinct workspace. See these links:

1 Like
10

Thanks, that makes sense.