Dockerhub rate-limit overview
Recently Docker announced that they started rate-limiting anonymous and free users.
The limit is 200 pulls per 6 hours for free users and 100 per 6 hours for anonymous users.
As a result, lots of cloud providers and other vendors started to take action. For example, if you are running in Google Cloud, you can use their repository mirror. Also they provide you with tooling to migrate your container images away from Dockerhub. Similarly, AWS also is working on their own solution. If you are running in the cloud, make sure you follow your cloud provider's advice.
Don't forget to take care of your application image builds. To clarify, many users use public docker images such as alpine for baking their application container images. So, make sure you review your CI pipelines and have a solid plan for that. Some of CI/CD cloud vendors provide a temporary solution to this problem. For example, CircleCI has negotiated with the Docker team to avoid rate limiting for now.
But what do you do if you run containers in your data-center?
The solution
The simplest way to solve this issue is to deploy your own Dockerhub mirror. Once you pull from Dockerhub, it will be stored in the mirror. Consequently, next time you try to pull the same image, you will get them from your local copy.
Multiple open-source container registries support this configuration:
Docker's registry, Harbor image registry. Additionally, Quay registry team is working towards it.
In this article, we will deploy the Dockerhub mirror using the Docker registry.
Deployment considerations
If you are running in Kubernetes, we typically recommend running mirror outside of the Kubernetes cluster. Typically you would deploy the registry mirror into a separate Virtual Machine.
Why?
Well, if you are going to deploy a registry mirror inside your Kubernetes cluster, you might have trouble.
For example, let's imagine you need to drain a node, which runs the registry mirror. While new pods are moving and your registry is down, you will pull directly from Dockerhub. And if you have many pods, you might hit Dockerhub's limit before you spin up your registry mirror.
Then you are in trouble. You are not able to spin up the registry pod because you can't pull from Dockerhub anymore. So you end up in a complete deadlock.
Okay, you have a new fresh Virtual Machine ready to rock-n-roll. Let's deploy the mirror!
Deployment
We are going to use Systemd to deploy Docker Registry.
Here is an example Systemd unit file:
[Unit]
Description=docker private registry service
After=docker.service
[Service]
ExecStartPre=-/usr/bin/docker kill registry
ExecStartPre=-/usr/bin/docker rm registry
ExecStart=/usr/bin/docker run --network host --name registry -v /etc/docker/registry/:/etc/docker/registry/ \
-v /data/:/data/ \
registry:2.7.1
ExecStop=-/usr/bin/docker stop registry
Restart=always
RestartSec=10s
[Install]
WantedBy=multi-user.target
Let's go over the configuration:
- We use /data mount on the host to store the images.
- Docker registry configuration is in the host's /etc/docker/registry/config.yml.
- If you are using any corporate firewall or HTTP proxy, don't forget to set HTTP_PROXY and HTTPS_PROXY environment variables. You can pass those via -e flag.
The annoying part is that we are using Dockerhub to pull the Registry image.
Currently, Docker doesn't provide the registry's binary files. So there is no simple way to run it via a Systemd unit without using Docker. Therefore we are forced to use the docker run, which counts as another pull from DockerHub's public registry. Make sure you are aware of this!
Now let's take a look at example registry configuration:
version: 0.1
log:
accesslog:
disabled: true
level: warn
fields:
service: registry
storage:
filesystem:
rootdirectory: /data/
http:
addr: :5000
debug:
addr: :5002
prometheus:
enabled: true
proxy:
remoteurl: https://registry-1.docker.io
username: username-here
password: password-here
You can read more about the configuration in official Docker registry documentation. For a simple use-case, this configuration should do the trick. In short, deploy it to your Virtual Machine and make sure you have enough space in /data.
Then you can start up the registry using: systemctl daemon-reload and systemctl restart registry-mirror.
You can check whether your registry mirror is running via systemcl status registry-mirror or docker ps commands.
Now it's time to configure Docker to use the new registry mirror.
Open up you /etc/docker/daemon.json. If it doesn't exists, then create it. Add the following configuration to it:
"registry-mirrors": [
"http://registry-mirror-host:5000"
],
Restart the docker daemon so that it picks up the new changes.
You can test the mirror by executing:
docker pull ubuntu
Then check the mirror's catalog using:
curl mirror-host:5000/_v2/catalog
You should see that it stores the ubuntu image in the mirror.
Conclusion
That's it for today. Hopefully, this simple deployment configuration will help you to avoid Dockerhub's rate limiting. By the way, don't forget to set up monitoring for your new Docker registry. You can do that using white-box and black-box monitoring techniques.
For black-box monitoring, continuously probe one of the registry's endpoints. It should be simple to do that as the registry uses HTTP to serve container images. For instance, probe /v2/_catalog endpoint. Probing the endpoint will imitate docker daemon behavior, and you will know that registry is up and working.
For white-box monitoring, you should ingest Prometheus metrics. The registry provides Prometheus metrics in:
http://registry-mirror-host:5002/metrics.
Typically you want to alert on errors.
By the way, we have already made the Docker registry monitoring package in PrometheusKube bundle.
Want to get it?