Drone-migrate update-repos fails with "no such file or directory"

Drone Support
1

Hi there

I’m trying to migrate from Drone 0.8 to 1.2 using the drone/drone-migrate tool and in the last step I keep getting this error:

[ec2-user@ip-10-0-0-123 ~]$ ./drone-migrate.sh --debug update-repos
DEBU[0000] target database driver: mysql                
DEBU[0000] target database datasource: drone:...@tcp(drone....rds.amazonaws.com:3306)/drone?parseTime=true 
DEBU[0000] scm driver: stash                            
DEBU[0000] scm server: https://bitbucket.example.com/repos 
FATA[0000] open : no such file or directory             
[ec2-user@ip-10-0-0-123 ~]$

All the previous steps worked fine, only this one fails.

How can I fix it or work around it?

We’re using BitBucket Server 5.11 (i.e. self-hosted, not cloud).

Thanks!

2

did you provide the migration utility with the private key file? The particular step is the only step that uses the private key file, and open : no such file or directory tells me there was a problem reading it.

 SCM_DRIVER=stash 
 SCM_SERVER=https://stash.company.com
 STASH_CONSUMER_KEY=OauthKey
+STASH_PRIVATE_KEY_FILE=/path/to/private/key.pem

If you set the private key file path, and you are using the drone-migrate docker image, also make sure you mount the private key file (using a docker volume) otherwise it cannot be accessed from inside the container.

--volume=/path/to/private/key.pem:/path/to/private/key.pem
3

Good point, I put the key inline at first as I’ve taken it from the drone config DRONE_STASH_CONSUMER_RSA_STRING='-----BEGIN RSA PRIVATE KEY-----\nMIICXAIBAAK...'

Now I put the key in a file but it’s still the same. To make sure it’s correctly mounted in the container I start it from the built-in bash:

[laptop] ~ $ docker run -it --rm \
-e SOURCE_DATABASE_DRIVER=mysql \
-e TARGET_DATABASE_DRIVER=mysql \
-e 'SOURCE_DATABASE_DATASOURCE=drone:.../dronedev?parseTime=true' \
-e 'TARGET_DATABASE_DATASOURCE=drone:.../drone2?parseTime=true' \
-e DRONE_SERVER="http://drone-2.{example.com}" \
-e SCM_DRIVER=stash \
-e SCM_SERVER="https://bitbucket.{example.com}/repos" \
-e STASH_CONSUMER_KEY=... 
-e STASH_PRIVATE_KEY_FILE=/private-key-file.pem \
-v ${PWD}/private-key-file.pem:/private-key-file.pem \
--entrypoint /bin/sh drone/migrate

## Make sure the key is readable
[docker-migrate] ~ $ cat /private-key-file.pem
-----BEGIN RSA PRIVATE KEY-----
MIICXAIBAAK...
...
-----END RSA PRIVATE KEY-----

## Run drone-migrate
[docker-migrate] ~ $ /bin/drone-migrate --stash-private-key-file /private-key-file.pem \
                                      --debug update-repos 
DEBU[0000] target database driver: mysql                
DEBU[0000] target database datasource: drone:.../drone2?parseTime=true 
DEBU[0000] scm driver: stash                            
DEBU[0000] scm server: https://bitbucket..../repos 
FATA[0000] open : no such file or directory           
  
[docker-migrate] ~ $

What now?

BTW it would be great if it printed the filename it couldn’t open…

4

I think the next step would be to look at the code to triage further. If you see an issue please consider sending a pull request. Or a pull request to update the docs if they can be improved.

5

I managed to fix this issue - the code was using c.String("stash-private-key-file") which returned empty string. When changed to c.GlobalString(...) it works.
Here is the pull request: https://github.com/drone/drone-migrate/pull/14

However I’m still seeing some issues:

DEBU[0000] private key file: /home/ec2-user/private-key-file.pem
INFO[0000] updating repository metadata
ERRO[0000] failed to get remote repository   error="%!v(PANIC=runtime error: index out of range)" owner=xyz repo=CCC/ami-ec2
ERRO[0000] failed to get remote repository   error="Not Authorized" owner=abc repo=CCC/ami-ecs
...
INFO[0000] repository metadata update complete
6

Ok, fixed that PANIC too. Apparently all the URL paths in go-scm/scm/driver/stash/repo.go do not start with the leading / and that makes the requests fail. This helps:

index e4d4e37..3fb3852 100644
--- a/scm/driver/stash/repo.go
+++ b/scm/driver/stash/repo.go
@@ -99,7 +99,7 @@ type repositoryService struct {
 // Find returns the repository by name.
 func (s *repositoryService) Find(ctx context.Context, repo string) (*scm.Repository, *scm.Response, error) {
        namespace, name := scm.Split(repo)
-       path := fmt.Sprintf("rest/api/1.0/projects/%s/repos/%s", namespace, name)
+       path := fmt.Sprintf("/rest/api/1.0/projects/%s/repos/%s", namespace, name)
        out := new(repository)
        res, err := s.client.do(ctx, "GET", path, nil, out)
        return convertRepository(out), res, err

It must be done across all the Sprintf() calls in the file. Maybe there is a better way to do that. Perhaps check in scm/driver/stash/stash.go -> do() whether the path starts with / and if not prefix it?

I won’t make a pull request for this as I don’t know what’s the right way to fix it.

7

Hmm, no leading slash in the path is required for me to use Drone successfully with Bitbucket Server. Note that github, gogs, gitea, etc do not use a leading slash either [1].

Perhaps check in scm/driver/stash/stash.go → do() whether the path starts with / and if not prefix it?

The code accounts for a slash in the path [2][3] when the client is created. If the base path is empty or does not finish with a trailing slash one is added. The base path (trailing slash) is joined with the endpoint path (no leading slash).

I did notice your scm server address includes /repos in the path which seems abnormal. Perhaps this should be removed?

-https://bitbucket.{example.com}/repos
+https://bitbucket.{example.com}

[1] https://github.com/drone/go-scm/blob/master/scm/driver/github/repo.go#L66
[2] https://github.com/drone/go-scm/blob/master/scm/driver/stash/stash_test.go
[3] https://github.com/drone/go-scm/blob/master/scm/driver/stash/stash.go#L30