Drone fails to sync new GitHub repos via the API

[oszi]

Here’s my intended use case:

• Create a GitHub repository via the GitHub API
• Wait 30 seconds (retry in 5 minutes if the next step fails)
• Enable drone builds: POST /api/repos/github_org/repo_name
• Change the build settings: PATCH gated:true, trusted:true, private:true

The user that creates the repositories is an owner in the organization
and all of the drone API calls are made with this user’s token.

I always get the following results:

time="2017-09-20T08:31:01Z" level=debug msg="Cannot find repository myorg/myrepo. sql: no rows in result set"
time="2017-09-20T08:31:01Z" level=info ip=172.17.0.1 latency=3.432468ms method=POST path="/api/repos/myorg/myrepo" status=404 time="2017-09-20T08:31:01Z" user-agent=

I tried forcing a sync by calling /api/user/repos?all=true&flush=true before the POST request but that didn’t work. However, if I’m logged in with another account and I force sync through the web UI then it does work. Could be OAuth related?

[bradrydzewski]

Sorry, I am currently unable to repeat any issues with synchronization. I just created a new repository and then went into the user interface and clicked the “synchronize” button at which point it was visible in the repositories list.

If you continue to experience issues I recommend taking a look at the synchronization code and sending a patch https://github.com/drone/drone/blob/master/server/sync.go

[oszi]

That workflow works for me too.
Can you synchronize via the API using the bearer token?

[bradrydzewski]

Yes, the user interface uses the API. This means that any functions available to the user interface can be invoked directly using your personal account token.

[oszi]

So then the only difference is which user was making the API calls (my own account or the bot account that failed). As I said, the user that failed to synchronize is also an owner in the organization so I don’t see what could’ve gone wrong.

Is there some kind of periodic background sync that I could wait for?
I’ll also try a clean install tomorrow.

[bradrydzewski]

Here are my recommendations:

1. use the chrome developer tools network inspector to see how the API call is being invoked, and make sure you are invoking it correctly (correct path, method, etc)
2. check the database to see if the repository was added. `select * from repos where repo_full_name=‘foo/bar’
3. synchronize your personal account as well. It is not enough to synchronize with a bot because drone still needs to synchronize your account and synchronize your individual repository permissions.

Is there some kind of periodic background sync that I could wait for?

there is no background sync. When the user requests the repository list (via the UI or API) it will automatically synchronize if the previous sync was more than 72 hours ago.

[oszi]

The appeal of drone for us is the simple API. So I think when there is a POST request for a missing repository, drone should take care of synchronization automatically and not fail with a 404 response. For now, I solved this with multiple delayed retries - calling the refresh API endpoint each time before the POST request.

Also GitHub seems to have organization web hooks. Can’t we utilize that?
I can’t promise anything but I may be able to allocate some time to contribute work on this.
Organization wide repository settings would be pretty cool.

[bradrydzewski]

I recommend setting up a simple utility that accepts organization hooks and then invokes the drone endpoint to synchronize repositories.

• https://github.com/phayes/hookserve
• https://github.com/bketelsen/captainhook
• https://github.com/adnanh/webhook

For example you could setup a simple captain hook configuration:

echo <<EOF > /some/local/config
{
    "scripts": [
        {
            "command": "curl",
            "args": [
                "-x", 
                "POST", 
                "-H", 
                "Authorization: Bearer {token}", 
                "https://drone.company.com/api/user/repos?all=true&flush=true"
            ]
        }
    ]
}
EOF

and then run the docker image:

docker run -d -v /some/local/config:/config bketelsen/captainhook

The reason this is preferred (at this time) to enabling organization webhooks in drone core is that we would need to assess all providers (gitlab, gitea, gogs, bitbucket, stash, coding) and implement the functionality across the board, if available.

[oszi]

Oh, right, there are many providers. We talked about this on gitter once.
I’ll go with your recommendation then.

[Rajshr]

Hi oszi,

I am stuck with similar kind of issue.
Could you let me know how much delay did you add and exactly which API did you hit to synchronize the repos in drone

[coltenkrauter]

I am struggling to sync repos via the Drone API in order to setup Drone programmatically for the first time. Basically, I am using Ansible to setup Drone and when I run the Drone Docker image, I use this in order to create a user:

DRONE_USER_CREATE=username:coltenkrauter,machine:false,admin:true,token:4123a6c5986aeb0839b18a03ee3f6be1 (Don’t worry, this is a fake token)

Anyway, when I have Ansible try to sync repos with the following,

- name: Sync repositories
  uri:
    method: POST
    url: "https://drone.my_private_website.com/api/user/repos?all=true&flush=true"
    status_code:
      - 200
    headers:
      Authorization: Bearer {{ drone_token }}
  register: _sync_result    

- debug: msg={{ _sync_result }}

I simply get an internal server error 500 with message {“message”: “Bad credentials”}

But I don’t understand why I am getting a bad credentials error… This only happens when I try this particular POST… if I do a get on the same endpoint, it works fine… except it isn’t synched so I don’t get any GitHub repos back in the list.

Can someone please help? Thanks!

[bradrydzewski]

the reason is because the system cannot make any API calls to github on your behalf until you have logged-in and authorized access, which generates a github access token. Note that the token provided in DRONE_USER_CREATE is a drone API token, and not a github access token.

if you want to seed the database with a github token, we would need to update DRONE_USER_CREATE to support setting an access_token. You can find the relevant code at https://github.com/drone/drone/blob/master/cmd/drone-server/config/config.go#L545, if you are interested in sending a pull request.

[coltenkrauter]

Thanks for the speedy response. I had a hunch it was something like that. I’ll consider a PR.