Inject .drone.yml on the fly

Drone Support
1

Hi. Is there any way to inject .drone.yml once the repository is cloned? I wouldn’t like to have a .drone.yml per repository in my organization, but a centralized repo with all yamls defining project-specific pipelines. The reason for this is basically that we wouldn’t like to expose all the build steps and some specifics to people that have access to those repos (yeap, we can’t remove 'em — they should have access to the code, but not to the build pipeline).

What I thought in doing so far is creating a plugin that basically clones another repo, copy the specific .drone.yml for the project and replace with the current one that invokes the plugin. I’d still have an “empty” yml file in the project that would invoke this plugin, but not really sure that would work. I mean, it’s fine to have, but it’s kinda crappy 'cause ideally we wouldn’t reveal any of that.

Your thoughts?

2

What I thought in doing so far is creating a plugin that basically clones another repo, copy the specific .drone.yml for the project and replace with the current one that invokes the plugin.

This would not work because drone fetches the Yaml configuration file using the API before the repository is cloned. The Yaml is then compiled and all the build steps are pre-calculated. It would therefore not be possible to hot-swap a Yaml configuration on the fly.

I am not aware of any workarounds that would enable a central repository of shared Yaml files.

3

Hey Brad — so isn’t there any way to proxy that API request and serve the correct yml?

1 Like
4

so isn’t there any way to proxy that API request and serve the correct yml?

You could configure drone to communicate with github through a proxy that intercepts api calls and returns custom results when fetching the yaml file

-GITHUB_URL=https://github.com
+GITHUB_URL=https://proxy.github.mycompany.com

You would have to make sure everything proxies correctly (oauth, etc). So yes, something like this would work if implemented correctly.

5

Hey @bradrydzewski. I made it — it’s working! Thanks for the tip, however, I had to do HTTPS_PROXY and do some more “manual” stuff. Really simple setup, wrote a server in Go and coded some interceptions to change couple of behaviors as I needed.

1 Like
7

Could you share the detail to us? Looks like an useful feature.

8

Of course, @ozbillwang — I’ll check with the guys over here and I’ll post back. Thanks!

9

Amazing, How you did it? if it possible to share your code?

10

drone 1.0 has a dedicated endpoint that you can use to override how the yaml is fetched, which can be used to inject / override / modify the yaml. It is configured by passing the following environment variables to the server.

DRONE_YAML_ENDPOINT=http://...
DRONE_YAML_SECRET=...

The secret (above) is a shared secret that you provide to the server and the plugin, used to secure communications between the two.

We provide a starter project for creating yaml plugins, as well as an example plugin that is capable of processing a jsonnet file and returns a valid yaml back to drone:

3 Likes
11

Glad to know that @bradrydzewski :heart_eyes:—thanks for sharing.

I have one question though, is it possible that I render a different YAML based on whether it’s running a BUILD vs a DEPLOYMENT?

I’ve got a number of use cases for that :point_up_2:…say for example that when I build my repository it generates a Docker image that when we confirm the deployment we’ll push it to the registry.

12

I have one question though, is it possible that I render a different YAML based on whether it’s running a BUILD vs a DEPLOYMENT?

the only restriction is that the endpoint needs to return a valid yaml file. You can therefore implement whatever logic you want.

13

Hey @bradrydzewski I am trying to return a dynamic YAML pipeline file using the env DRONE_YAML_ENDPOINT as you mentioned.

Everything is looking fine in the process, but when the Drone hits my API that should return the YAML content, it shows the following error:

{"commit":"c4a8c2aaec7489db8515341110d0faa25f0fa7b0","error":"invalid character 'k' looking for beginning of value","event":"push","level":"warning","msg":"trigger: cannot find yaml","ref":"refs/heads/drone-test","repo":"Govlaunch/ui","time":"2019-02-18T02:13:09Z"}

I am trying returning a static yml content just for testing purposes by setting the header “Content-Type: text/yaml”

app.post("/", (req, res) => {
  res.set("Content-Type", "text/yaml");

  res.send(`kind: pipeline
  name: default
  
  steps:
  - name: frontend
    image: node
    commands:
    - npm install
    - npm test`);
});

Am I doing anything wrong?

14

The response body should be json, with {"data": "kind: pipeline\n..."}

15

Did this ever work? I’d like to avoid the go-based plugin route as my environment is Node.js

16

I’d like to avoid the go-based plugin route as my environment is Node.js

Plugins are the recommended approach to solving this problem. Plugins use REST / HTTP / JSON and can be written in any language, including in node.

17

I managed to write a simple Rest api in Node and it worked like a charm. I will look at integrating the plugin format that you recommend.

18

@bradrydzewski I would like to create a yaml file on the fly based on the repo by checking the files in the repo. if it has package.json add a npm build or if it has pom. add a maven build. How can I read the files using the plugin extension?

19

@vivekthangathurai you would have to create your own plugin extension [1][2] that reads files from github and then returns a yaml. To read files from github you can use the github API [3][4].

[1] https://docs.drone.io/extensions/configuration/
[2] https://docs.drone.io/extensions/configuration/#starter-project
[3] https://godoc.org/github.com/google/go-github/github#RepositoriesService.GetContents
[4] https://developer.github.com/v3/repos/contents/#get-contents

EDIT here is some pseudo code that shows how this could work, based on this sample extension.

	// creates a github client used to fetch the yaml.
	trans := oauth2.NewClient(ctx, oauth2.StaticTokenSource(
		&oauth2.Token{AccessToken: p.token},
	))
	client := github.NewClient(trans)

	// get the configuration file from the github
	// repository for the build ref.
	data, _, _, err := client.Repositories.GetContents(req.Repo.Namespace, req.Repo.Name, path, &github.RepositoryContentGetOptions{Ref: req.Build.After})
	if err == nil && data != nil {
		// get the file contents.
		content, err := data.GetContent()
		if err != nil {
			return nil, err
		}
		return &drone.Config{
			Data: content,
		}, nil
	}

	// if the configuration file does not exist, and a package.json
	// file exists return a default node pipeline
	data, _, _, err = client.Repositories.GetContents(req.Repo.Namespace, req.Repo.Name, "package.json", &github.RepositoryContentGetOptions{Ref: req.Build.After})
	if err == nil {
		return &drone.Config{
			Data: "..... your node yaml here ....",
		}, nil
	}

	// if the configuration file does not exist, and a pom.xml
	// file exists return a default maven pipeline
	data, _, _, err = client.Repositories.GetContents(req.Repo.Namespace, req.Repo.Name, "pom.xml", &github.RepositoryContentGetOptions{Ref: req.Build.After})
	if err == nil {
		return &drone.Config{
			Data: "..... your maven yaml here ....",
		}, nil
	}

	return nil, nil
20

Thanks. Much appreciated.