Drone.io How to control multi-module build pipeline where modules are interdependent for CI

Drone Support
1

Hi guys,

I am working on my drone CI pipeline facing following issue:

My git repo looks like this:

  myrepo
     -> .drone.yml
     -> images/
           -> a1/
                 -> a1script
                 -> Dockerfile
           -> a2/
                 -> a2script
                 -> Dockerfile
           -> a3/
                 -> a3script
                 -> Dockerfile
            -> a-final/
                 -> Dockerfile

a-final depends on a3 to build and publish it’s image on our artifactory and same for others - a3 depends on a2 and a2 depends on a1.

Now, in my .drone.yml what I want is if a module has been changed in git repo - then only that module and all it’s dependent modules should be build and published (there is only 1 “step” for each module in my .drone.yml file to publish) one after another and not all of them. For example, If I push some code in a2, then only a2, a3 and a-final should be built and published in order a2, a3, a-final.

I tried doing that in my .drone.yml but can’t control the behavior even with using conditions like status:success OR environment variables.

This is how the first 2 modules (a1 and a2) of my .drone.yml looks like:

pipeline:

 # Get the latest git diff and see what directories changed.
check_clone:
  image: plugins/git
  commands:
    - git diff-tree --no-commit-id --name-only -r `git log | head -1 | awk '{print $2}'` > git_latest_delta
    - cat ./git_latest_delta
    - echo "Done!"

# Check if a1 is in the diff list - if not then exit 1 
a1_build:
  image: ubuntu:bionic
  commands:
     - ./is_build_required a1
    

 # Publish a1 on some condition - like if previous step is successful or if an env variable is set.
a1_publish:
  image: plugins/docker:17.05
  registry: docker.mycompany.com
  repo: docker.mycompany.com/app/a1
  dockerfile: images/a1/Dockerfile

  tags:
    - "${DRONE_COMMIT:0:8}"
  when:
    status: success
  username: myuser
  secrets:
    - source: artifactory_password
      target: plugin_password

a2_build:
  image: ubuntu:bionic
  commands:
    - echo $$DRONE_TAG
    - ./is_build_required a2
  when:
    status: [ success, failure ] 


 # Don't know how to get at this step as if a1 publish is skipped  then pipeline fails and the below step doesn't execute. :(

a2_publish:
  image: plugins/docker:17.05
  registry: docker.mycompany.com
  repo: docker.mycompany.com/app/a2
  dockerfile: images/a2/Dockerfile
  tags:
    - "${DRONE_COMMIT:0:8}"
  when:
    status: success
  username: svcpejkn001
  secrets:
    - source: artifactory_password
      target: plugin_password

Same for a3 and a-final. This how I need to control the pipeline build process.
I tried with custom environment variables but couldn’t control the “when:” field in step because it doesn’t accept our defined variables - it only goes with fixed things like status, branch, repo, etc.

So, please help me.

Thanks,
Neo

2

It sounds like you want to run certain pipeline steps based on whether or not files in subdirectories were changed. For this you should consider installing a plugin which can be used to add custom configuration options to Drone. Note that these plugins only work with Drone 1.3 or higher. Here are a few third party plugins that might be useful:

  1. https://github.com/microadam/drone-config-changeset-conditional
  2. https://github.com/bitsbeats/drone-tree-config

Please note these are third party plugins and are not maintained or supported by the Drone project. If you have any questions or need any help with a plugin you should create an issue in the plugin repository.

3

@bradrydzewski Thank you for the reply and suggesting the third-party plugins.
Unfortunately, I am using Drone 0.8 version.

  1. Isn’t there any other way I can achieve this? I was able to create global variables and was successful changing there values between the steps too. But, for publish steps, I couldn’t add them to “when” tag.

  2. May be is there something I can use specifically on publish sections in my above .drone.yml file where it will decide weather or not to run this publish step - without failing the entire pipeline OR any other solution if you have in mind.

  3. Also, as you have seen my git repo structure and dependencies above… I am open to a complete design change too for my git repo which can fit into Drone builds and I can achieve the same result.

So, please feel free to suggest anything that you know.
Again thank you so much for the help.

Thanks,
Neo

4

sorry, I am not aware of any way to solve this in 0.8, which is why we were excited to add this capability in 1.x

5

Hi bradrydzewski,

I want to implement same pipeline in drone 1.1.0 but when i used to deploy this plugin into my existing drone implementation. Its breaks up. As there is no proper docs regarding this. Can u please help me out in proper implementation.

I am using drone 1.1.0 on docker swarm cluster . I am trying to use drone-config-tree plugin but it doesn’t support my bitbucket repo.