@bradrydzewski, thanks for helping us out.
Here are the details:
- Autoscaler configuration:
ExecStart=/bin/bash -c '/usr/local/bin/summon -p summon-aws-secrets -f /etc/drone/secrets.yml /usr/bin/docker run --name drone-autoscaler \
--rm \
-p 8080:8080 \
--env-file @SUMMONENVFILE \
--env-file /etc/drone/drone-agent-ami.env \
--volume=/var/lib/autoscaler:/data \
--env=DRONE_SERVER_HOST=drone.tools.prod.movio.co \
--env=DRONE_SERVER_PROTO=https \
--env=DRONE_POOL_MIN=3 \
--env=DRONE_POOL_MAX=12 \
--env=DRONE_AGENT_CONCURRENCY=2 \
--env=DRONE_INTERVAL=30s \
--env=DRONE_LOGS_DEBUG=true \
--env=DRONE_AMAZON_INSTANCE=m5d.xlarge \
--env=DRONE_AMAZON_TAGS="Name:drone-agent,Squad:devops,dibs/team:devops" \
--env=DRONE_AMAZON_REGION=ap-southeast-2 \
--env=DRONE_AMAZON_RETRIES=10 \
--env=DRONE_AMAZON_SUBNET_ID=subnet-0231e30d447dece79 \
--env=DRONE_AMAZON_SECURITY_GROUP=sg-00206da902bd4bd8c \
--env=DRONE_AMAZON_SSHKEY=movio-tools-ap-southeast-2.pem \
--env=DRONE_ENABLE_REAPER=true \
--env=DRONE_REAPER_ENABLED=true \
--env=DRONE_REAPER_INTERVAL=10m \
--env=DRONE_ENABLE_PINGER=true \
--env=DRONE_AMAZON_PRIVATE_IP=true \
--env=DRONE_AMAZON_IAM_PROFILE_ARN=arn:aws:iam::864091978270:instance-profile/drone-agent.tools.prod.movio.co \
--env=AWS_IAM=true \
--env=DRONE_SLACK_WEBHOOK=https://hooks.slack.com/services/*REDACTED*/*REDACTED*/*REDACTED* \
--env=DRONE_HTTP_HOST=drone.tools.prod.movio.co \
--env=DRONE_HTTP_PROTO=https \
--env=DRONE_AMAZON_VOLUME_SIZE=55 \
--env=DRONE_AGENT_ENVIRON=DRONE_LOGS_DEBUG=true,DRONE_LOGS_PRETTY=true,DRONE_LOGS_TRACE=true,DRONE_SECRET_PLUGIN_ENDPOINT=https://drone.tools.prod.movio.co,DRONE_SECRET_PLUGIN_TOKEN=*VERYSECRET*,DRONE_REGISTRY_ENDPOINT="https://drone.tools.prod.movio.co:3200",DRONE_REGISTRY_SECRET="*VERYSECRET*",DRONE_REGISTRY_VERIFY="false" \
--env=DRONE_DATABASE_DRIVER=mysql \
drone/autoscaler:1.4'
The same config was used with 1.6.
- Registry plugin:
We’re using https://github.com/davidbyttow/drone-ecr-registry-plugin, unfortunately, it’s not published in dockerhub, so we’re running it from our own ECR. Authentication on drone master is done via aws ecr authentication helper.
The code is EXACTLY the same as the code from the repository.
The config for the plugin:
ExecStart=/bin/bash -c '/usr/local/bin/summon -p summon-aws-secrets -f /etc/drone/secrets.yml /usr/bin/docker run --name=drone-plugin-ecr-registry \
--rm \
--volume=/var/run/docker.sock:/var/run/docker.sock \
--env-file @SUMMONENVFILE \
--publish=3200:3000 \
--env=PLUGIN_DEBUG=true \
--env=ECR_REGISTRY_LIST=191213556404.dkr.ecr.us-east-1.amazonaws.com,864091978270.dkr.ecr.ap-southeast-2.amazonaws.com,016843415717.dkr.ecr.us-east-1.amazonaws.com,649264355054.dkr.ecr.eu-central-1.amazonaws.com,650454640207.dkr.ecr.ap-southeast-2.amazonaws.com \
fa97908f1674'
# 864091978270.dkr.ecr.ap-southeast-2.amazonaws.com/drone-ecr-registry-plugin:latest'
- Registries from plugin list:
$ drone plugins registry list
https://191213556404.dkr.ecr.us-east-1.amazonaws.com
Username: AWS
Password: ಠ_ಠ
https://864091978270.dkr.ecr.ap-southeast-2.amazonaws.com
Username: AWS
Password: ಠ_ಠ
https://016843415717.dkr.ecr.us-east-1.amazonaws.com
Username: AWS
Password: ಠ_ಠ
https://649264355054.dkr.ecr.eu-central-1.amazonaws.com
Username: AWS
Password: ಠ_ಠ
https://650454640207.dkr.ecr.ap-southeast-2.amazonaws.com
Username: AWS
Password: ಠ_ಠ
{
"kind": "pipeline",
"name": "PR: Generate diff of actions",
"steps": [
{
"commands": [
"kubectl config set-cluster k8s.stage.movio.co --server=https://api.k8s.stage.movio.co",
"kubectl config set-credentials k8s.stage.movio.co-aws --exec-command=/bin/aws-iam-authenticator --exec-api-version=client.authentication.k8s.io/v1alpha1 --exec-arg=token --exec-arg=-i --exec-arg=k8s.stage.movio.co --exec-arg=-r --exec-arg=arn:aws:iam::225711623517:role/DevopsDroneRole",
"kubectl config set-context k8s.stage.movio.co --cluster=k8s.stage.movio.co --user=k8s.stage.movio.co-aws",
"kubectl config use-context k8s.stage.movio.co",
"kubectl config view",
"kubectl cluster-info",
"cd components && helmfile diff"
],
"depends_on": [
"clone"
],
"image": "864091978270.dkr.ecr.ap-southeast-2.amazonaws.com/devops-drone-images:helmfiles-plugin-latest",
"name": "PR: Generate diff of actions for stage cluster.",
"pull": "always"
} ],
"trigger": {
"event": [
"pull_request"
]
}
}
---
{
"kind": "pipeline",
"name": "Deploy changes",
"steps": [
{
"commands": [
"kubectl config set-cluster k8s.stage.movio.co --server=https://api.k8s.stage.movio.co",
"kubectl config set-credentials k8s.stage.movio.co-aws --exec-command=/bin/aws-iam-authenticator --exec-api-version=client.authentication.k8s.io/v1alpha1 --exec-arg=token --exec-arg=-i --exec-arg=k8s.stage.movio.co --exec-arg=-r --exec-arg=arn:aws:iam::225711623517:role/DevopsDroneRole",
"kubectl config set-context k8s.stage.movio.co --cluster=k8s.stage.movio.co --user=k8s.stage.movio.co-aws",
"kubectl config use-context k8s.stage.movio.co",
"kubectl config view",
"kubectl cluster-info",
"cd components && helmfile sync"
],
"depends_on": [
"clone"
],
"image": "864091978270.dkr.ecr.ap-southeast-2.amazonaws.com/devops-drone-images:helmfiles-plugin-latest",
"name": "Deploy helmfile changes to stage cluster.",
"pull": "always"
}
],
"trigger": {
"event": [
"promote"
],
"target": [
"production"
]
}
}
.
I’ve updated our autoscaler to use :latest runner, and it started failing again. Here’s the debug log
time="2020-01-31T08:44:57Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2020-01-31T08:44:57Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2020-01-31T08:45:21Z" level=debug msg="stage received" stage.id=22158 stage.name="Deploy changes" stage.number=1 thread=2
time="2020-01-31T08:45:21Z" level=debug msg="stage accepted" stage.id=22158 stage.name="Deploy changes" stage.number=1 thread=2
time="2020-01-31T08:45:23Z" level=debug msg="stage details fetched" build.id=13888 build.number=50 repo.id=232 repo.name=kops repo.namespace=movio stage.id=22158 stage.name="Deploy changes" stage.number=1 thread=2
time="2020-01-31T08:45:23Z" level=trace msg="registry: external: received credentials" address="https://191213556404.dkr.ecr.us-east-1.amazonaws.com" thread=2 username=AWS
time="2020-01-31T08:45:23Z" level=trace msg="registry: external: received credentials" address="https://864091978270.dkr.ecr.ap-southeast-2.amazonaws.com" thread=2 username=AWS
time="2020-01-31T08:45:23Z" level=trace msg="registry: external: received credentials" address="https://016843415717.dkr.ecr.us-east-1.amazonaws.com" thread=2 username=AWS
time="2020-01-31T08:45:23Z" level=trace msg="registry: external: received credentials" address="https://649264355054.dkr.ecr.eu-central-1.amazonaws.com" thread=2 username=AWS
time="2020-01-31T08:45:23Z" level=trace msg="registry: external: received credentials" address="https://650454640207.dkr.ecr.ap-southeast-2.amazonaws.com" thread=2 username=AWS
time="2020-01-31T08:45:23Z" level=debug msg="updated stage to running" build.id=13888 build.number=50 repo.id=232 repo.name=kops repo.namespace=movio stage.id=22158 stage.name="Deploy changes" stage.number=1 thread=2
time="2020-01-31T08:45:37Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2020-01-31T08:45:39Z" level=debug msg="updated stage to complete" build.id=13888 build.number=50 repo.id=232 repo.name=kops repo.namespace=movio stage.id=22158 stage.name="Deploy changes" stage.number=1 thread=2
time="2020-01-31T08:45:39Z" level=debug msg="poller: request stage from remote server" thread=2
time="2020-01-31T08:45:39Z" level=trace msg="http: context canceled"
time="2020-01-31T08:45:39Z" level=debug msg="done listening for cancellations" build.id=13888 build.number=50 repo.id=232 repo.name=kops repo.namespace=movio stage.id=22158 stage.name="Deploy changes" stage.number=1 thread=2
