Debugging concurrent steps not executing concurrently

Drone Support
1

Hi,

We’re running into a bit of a confusing situation with our Drone builds not running concurrently for some of the steps (the ones that we’d actually care about running concurrently).

We have a pipeline that looks similar to this example:

workspace:
  base: /go
  path: src/<path>

pipeline:
  setup:
    image: <image>
    secrets:
      - GITHUB_SSH_PRIVATE_KEY
    commands:
      - echo "--> Setup..."

  test:
    group: pre-build
    image: <image>
    commands:
      - go test ./... -v

  vet:
    group: pre-build
    image: <image>
    commands:
      - go vet ./...

  service-1-binary:
    group: binaries
    image: <image>
    commands:
      - go build -o cmd/service-1/service-1 ./cmd/service-1

  service-1-image:
    group: docker-images
    image: plugins/docker
    registry: <registry>
    repo: <registry>/service-1
    tags:
      - ${DRONE_BUILD_NUMBER}
      - ${DRONE_COMMIT_SHA:0:8}
    dockerfile: cmd/service-1/Dockerfile
    context: cmd/service-1
    build_args:
      - BASE_IMAGE_REPO=<image>
      - BASE_IMAGE_TAG=<tag>
    custom_dns: <ip1>, <ip2>
    secrets:
      - DOCKER_USERNAME
      - DOCKER_PASSWORD
    debug: true

  service-2-binary:
    group: binaries
    image: <image>
    commands:
      - go build -o cmd/service-2/service-2 ./cmd/service-2

  service-2-image:
    group: docker-images
    image: plugins/docker
    registry: <registry>
    repo: <registry>/service-2
    tags:
      - ${DRONE_BUILD_NUMBER}
      - ${DRONE_COMMIT_SHA:0:8}
    dockerfile: cmd/service-2/Dockerfile
    context: cmd/service-2
    build_args:
      - BASE_IMAGE_REPO=<image>
      - BASE_IMAGE_TAG=<tag>
    custom_dns: <ip1>, <ip2>
    secrets:
      - DOCKER_USERNAME
      - DOCKER_PASSWORD
    debug: true

In the above, only the pre-build group run concurrently, but the binaries and docker-images groups do not run concurrently.

Ideally, we’d want the binaries group to run first, and then the docker-images group to run. Any idea why this is not happening? Does it have something to do with the ordering of the steps? Do you have to co-locate the groups, i.e binaries have to all be right after the other? Same with docker-images?

Thanks!
Or

2

Does this page help? https://docs.drone.io/user-guide/pipeline/multi-machine/#dependencies
I believe with 1 you need to declare multiple pipelines and use depends_on to specify execution order.

3

Thanks @devstar! That looks really interesting. However, what’s this mention of parallel builds on that page?

Please be warned that most projects do not require multi-machine pipelines. You should exhaust every other option, including parallel pipeline steps and more powerful machines.

I’m having a bit of trouble finding parallel pipeline steps in the 1.0.0 docs, is that something that was removed?

4

In the section about dependencies it states:

In the below example, the first two pipelines execute in parallel. On successful completion a third pipeline is executed that sends a Slack notification indicating the workflow is complete.

I’m assuming it means each declared pipeline will be executed concurrently only respecting the dependency specified.
So, steps are sequential, pipelines are parallel by default it seems.

5

So, steps are sequential, pipelines are parallel by default it seems.

Yup, I’m not disagreeing. I’m just trying to understand if the group functionality that existed in 0.8.0 still exists in 1.0.0 and if so - where is it documented. I don’t have much control over our Drone deployment, therefore I’m not even sure whether there are multiple Drone agents running on multiple machines. So ideally, if I could use group to cause multiple steps to run concurrently, that’d be the simplest solution for us at this moment.

6

I don’t think you can replicate the behaviour using groups in 1.0.0, someone more knowledgeable than me might correct me in case I’m wrong. However, you should be able to achieve the same results rewriting the .drone.yml for the new version, without changing the deployment.

---
kind: pipeline
name: test

steps:
- name: test
  image: golang
  commands:
  - go build
  - go test ./... -v

---
kind: pipeline
name: vet

steps:
- name: vet
  image: golang
  commands:
  - go vet ./...

---
kind: pipeline
name: service-1-binary

steps:
- name: binary 1
  image: plugins/docker
  registry: <registry>
  repo: <registry>/service-1
  commands:
    - go build -o cmd/service-1/service-1 ./cmd/service-1

depends_on:
- test
- vet

---
kind: pipeline
name: service-1-image

steps:
- name: image 1
  image: plugins/docker
  registry: <registry>
  repo: <registry>/service-1
  tags:
  - ${DRONE_BUILD_NUMBER}
  - ${DRONE_COMMIT_SHA:0:8}
  dockerfile: cmd/service-1/Dockerfile
  context: cmd/service-1
  build_args:
  - BASE_IMAGE_REPO=<image>
  - BASE_IMAGE_TAG=<tag>
  custom_dns: <ip1>, <ip2>
  secrets:
  - DOCKER_USERNAME
  - DOCKER_PASSWORD

depends_on:
- service-1-binary

I’m not sure how workspace was translated, I believe you need to share volumes if you want persistance between pipelines.

7

Sounds good. Thanks for the help @devstar. I’m going to hold-off for a bit and see if anyone else will comment on this to confirm it’s not possible with 1.0.0 and that the new format needs to be used (I mean the 1.0.0 docs do mention parallel pipeline steps, so I’d be curious to know if that was accidental or whether it really is part of 1.0.0).

8

1.0 supports concurrent steps. See How to setup parallel Pipeline steps (1.0+)

9

Thanks @bradrydzewski, I was able to migrate our Drone file to the 1.0.0 format and get the relevant steps to run concurrently (pretty happy - it cut our build from 10 minutes to about 3 minutes).

I’m sure you already know this - but it’d have been really helpful if this was already in the docs.

Thanks again and congrats on the 1.0.0 release!