Elroi Marom April 11, 2022
11:45:46 AM
Elroi Marom Elroi MaromThis guide does not cover the basic topics covered in my previous blog post-Introducing Docker Datacenter.
In that post I answer the following questions:
You should:
For a quick primer into the world of dockers see this guide
Docker is constantly being updated. And although you are not required to always use the latest version, there are some things you should be aware of before reading this guide.
docker/somedocker some command it means that this is a ‘container command’ and should be called from a docker container using a format similar to docker run -it --rm docker/somedocker some commandBefore you begin your installation, it will be invaluable to familiarize yourself with the basic architecture of DDC.
DDC Consists essentially of 3 parts:
UCP Managers and DTR Managers both support High availability (HA) for 3+ nodes of each and nodes that run DTR can also run UCP Workers (but should not run UCP Managers).
For a more in depth view of DDC architecture, check out this Architecture and Best Practices guide.
Decide on a consistent naming scheme for your nodes in order to avoid confusion. For example, node1, node2, node3.
Next, choose the number of nodes to provision. I recommend that you start with at least 6 nodes.
Technically DDC can run on a minimum of 2 nodes, one for UCP and one for DTR, with UCP workers on both nodes.
However, If you intend to experiment with High Availability, you should provision at least 6 nodes. The reason why is explained in the next section.
If you are setting up a production environment and you know for a fact that you want HA enabled for both DTR and UCP, then you should use a minimum of 6 nodes ( 3 for UCP and 3 for DTR ) as UCP and DTR should not be hosted on the same node. The UCP workers, however, can technically be placed on any or all nodes.
For more information about High Availability in Docker Datacenter see the official guide for UCP and the guide for DTR on high availability as well as this blog post.
There is a particularly good explanation of how High Availability works within DDC in this guide.
For high availability, you can deploy DTR replicas on UCP worker nodes.
Beware when creating a Highly Available cluster. You must have an odd number of manager nodes higher than 1 for each service type in order to avoid a state known as split-brain. This means that for each service (UCP/DTR) that you wish to make highly available, you must have at least 3 manager nodes of that type.
To understand what split-brain is and how High Availability works in general within clusters, you can read about the Raft Consensus Algorithm.
Provision of the nodes for Docker Datacenter to be installed on
For the best performance ensure that all nodes (managers and workers) in the docker datacenter can talk to each other without a firewall.
To quote docker on its security reference page:
Networking can be an important part of a Docker EE deployment. The basic rule of thumb is not to have firewalls between the manager and worker nodes. When deploying to a cloud infrastructure, low latency is a must between nodes. Low latency ensures the databases are able to keep quorum. When a software, or hardware, firewall is deployed between the nodes the following ports need to be opened.
Setup Docker Storage
[ Optional ]
Setup storage to enhance the flexibility of your installation. This step is optional but recommended. On Linux machines Docker usually stores its data at
/var/lib/docker.The purpose of this step is to mount docker’s data folder onto external storage in order to better manage it.Devicemapper
The recommended way to setup flexible storage for docker containers, is using devicemapper.
To configure devicemapper, setup a virtual disk for every node, and then follow these instructions.
Additional Solutions
Although devicemapper is the recommended storage solution, depending on your infrastructure you may wish to implement other solutions.
One such alternative solution is the Docker vSphere Plugin. You can view other similar solutions by browsing the volume plugin section of the docker store.
Air-Gapped Installs
We did not find any way to download packages from the docker store in order to use store plugins on air-gapped nodes. So we used devicemapper.
However, devicemapper might not be accessible from all your air-gapped nodes. If this is the case you will need to download all the packages first on a machine with access using a download only utility such as
yum localupdatefor RedHat, and transfer the files afterward for installation.Post Configuration
Reboot your nodes after you configure devicemapper or similar drivers.
Setup DTR Image Storage
[ Optional ]
In addition to mounting the entire docker data directory onto a storage solution, if you plan to make your Docker Trusted Repository highly available by installing multiple DTR Masters, then you should mount the DTR image directory on a single mount-point shared between masters.
To quote Docker on the Configure DTR Image Storage page:
If your DTR deployment only has a single replica, you can continue using the local filesystem to store your Docker images. If your DTR deployment has multiple replicas, for high availability, you need to ensure all replicas are using the same storage backend. When a user pulls an image, the node serving the request needs to have access to that image.
DTR Supported Storage Systems:
- Local filesystem
- NFS
- Amazon S3 or compatible
- Google Cloud Storage
- Microsoft Azure Blob storage
- OpenStack Swift
For instructions on how to setup a DTR image mount see the above mentioned guide
Since NFS is a common option, here is a direct link to it’s tutorial.
Install Docker
After you have provisioned your nodes, install Docker EE on all of them.
Install Docker EE
See the tutorial for your specific Operating System
Air-Gapped Install
For nodes without access to the internet (air-gapped), you will need to download the packages and send them to the air-gapped nodes. This can be done one of the following ways:
- Download the packages manually with rsync or wget from a publicly available repository.
- Utilize the download-only feature of your package manager which is available on some systems.
For example: Redhat offers an easy way to download packages if you have access to a repository as described here.
Packages can be downloaded to a specific folder, gzipped, and then sent to the air-gapped nodes, where they can be installed manually.
You should take note of the order in which packages are installed using your package manager and use the same order when installing on your air-gapped nodes, in order to avoid dependency errors.
Linux Post-Install
If you are running Linux and this is your first time installing docker, you may wish to review the Docker linux-postinstall tips.
Enable Docker Service
Make sure the docker service is configured to automatically start on all nodes using the command
systemctl enable dockerStart Docker Service
Start the docker service on all nodes using
service docker startDownload UCP & DTR
[ Optional ]
If your nodes have access to the Internet you should skip this step, as the UCP and DTR images will be fetched automatically with the install command.
The Docker images for UCP and DTR can be pulled from the Internet, or from a locally available Docker Repository. However, if neither of these options are available to you, you will need to install them manually on each node which is to become a UCP or DTR Master.
Specific guides for Offline Install are available for DTR and UCP.
General Instructions
- Download the latest version of the DTR and UCP image
- Transfer the UCP or DTR image packages to the respective nodes that are to become UCP or DTR masters, using
scporrsyncfor example.- Load the UCP and DTR image packages on each UCP or DTR with the following commands:
docker load < ucp_images*.tgzanddocker load < dtr*.tgzUniversal Control Plane
UCP is the docker based application that is responsible for at least the following tasks among others:
- Controlling the dockers and applications across your cluster via the UCP web-dashboard.
- Handling login authentication for the UCP and DTR web-dashboard.
To quote the official UCP architecture guide.
Docker UCP leverages the clustering and orchestration functionality provided by Docker.
For a general understanding of UCP, read the UCP overview guide.
Install with Self-Signed TLS
If you do not have access to a CA or you are simply testing, you may opt to install UCP using insecure-tls mode
From the official guide:
1. Use ssh to log in to the host where you want to install UCP.
2. Run the following command:
# Install UCP
$ docker container run –rm -it –name ucp \
-v /var/run/docker.sock:/var/run/docker.sock \
docker/ucp:2.2.0 install \
–host-address <node-ip-address> \
–interactive
Replace 2.2.0 in the above command with whichever version of UCP you would like to install
If you are not sure which version of UCP you have installed, simply run docker images on the node in question. On Linux you can run docker images | grep ucp to filter the output.
[ Optional ]
You can view custom install options for your version of UCP by running: docker run --rm -it --name ucp docker/ucp install --help
For example, running this container command on ucp:2.2.0 will inform you that:
# UCP Install Help
This command initializes a new swarm, turns this node into a manager, and installs
Docker Universal Control Plane (UCP).
When installing UCP you can customize:
* The certificates used by the UCP web server. Create a volume
named ‘ucp-controller-server-certs’ and copy the ca.pem, cert.pem, and key.pem
files to the root directory. Then run the install command with the
‘–external-server-cert’ flag.
* The license used by UCP, by bind-mounting the file at
‘/config/docker_subscription.lic’ in the tool. E.g. -v /path/to/my/config/docker_subscription.lic:/config/docker_subscription.lic
or by specifying with ‘–license “$(cat license.lic)”
You can also visit the ucp install reference.
[ Optional ]
If you wish to use your own TLS certificates, you can specify them at install time with the above-mentioned commands. If you must automatically install TLS at this stage, you may find this post helpful for understanding how docker volumes are used to register TLS into UCP.
However, if you do not want to bother with it at this stage, you can update your TLS certificates later as described in the “Replace Self-signed TLS” section below.
Once you’ve successfully installed UCP, you will want to visit the dashboard to make sure that it works as expected.
Navigate to https://your-ucp-manager.your-domain.com and log in using the username and password that you chose during the installation of UCP
To quote the official guide:
1. Go to the Docker Store and buy a Docker EE subscription, or get a free trial license.
2. In your browser, navigate to the UCP web UI, log in with your administrator credentials and upload your license. Navigate to the Admin Settings page and in the left pane, click License.
3. Click Upload License and navigate to your license (.lic) file. When you’re finished selecting the license, UCP updates with the new settings.
[ Optional ]
Read the official guide for information about how to integrate UCP with LDAP
Now that your first UCP manager is up and running, you will usually want to register additional UCP agents with your UCP manager in order to handle higher loads.
Additionally, you may wish to increase the number of UCP managers/controllers in order to implement High Availability.
Follow the official guide for adding additional UCP nodes.
Nodes will automatically run the ucp-agent container and nodes that become managers will automatically run the ucp-controller container.
When adding nodes to the cluster it might prevent issues if you wait for one node registration to finish before continuing to add the next node
Docker Trusted Repository is the part of your installation that manages your docker images. UCP itself does not hold any images. Instead, it pulls from a remote image repository, which is where DTR comes in. Running DTR is like having your own private Docker Hub which your developers can push to and pull from.
DTR also includes a web dashboard accessible from your DTR node’s ip. The DTR dashboard’s authentication (login) is managed by the UCP node. So you should generally install UCP before attempting to install DTR
From the official DTR architecture guide:
Docker Trusted Registry (DTR) is a containerized application that runs on a Docker Universal Control Plane cluster.
For more information about what DTR is read the official guide.
Configure Storage
[ Optional ]
By default DTR uses the local filesystem of the node where it is running to store your Docker images. You can configure DTR to use an external storage backend, for improved performance or high availability.
See full guide on configuring DTR image storage.
Certain storage options are configurable during install time. For example to use an NFS directory at install time use the
--nfs-storage-urloption. Example:
# Install DTR with NFS
docker run -it –rm docker/dtr: install \
–nfs-storage-url <nfs-storage-url> \
<other options>
Follow the official install guide for more information about how to get started with DTR.
To install DTR you use the
docker/dtrimage. This image has commands to install, configure, and backup DTR.Run the following command to install DTR:
# Install DTR
$ docker run -it –rm \
docker/dtr:2.3.2 install \
–ucp-node <ucp-node-name> \
–ucp-insecure-tls
Where the
--ucp-nodeis the hostname of the UCP node where you want to deploy DTR.--ucp-insecure-tlstells the installer to trust the TLS certificates used by UCP.
Replace every occurrence of 2.3.2 with whatever version of DTR you are going to install
If you are not sure which version of DTR you have installed, simply run docker images on the node in question. On Linux, you can run docker images | grep dtr to filter the output.
Once you have finished installing DTR, you will want to visit the dashboard to make sure that it works as expected and configure it.
Be aware that UCP handles authentication for DTR, so if UCP goes down, so does DTR. Additionally, when you login you will temporarily be redirected to UCP in order to authenticate and then sent back to the DTR dashboard when authentication is complete.
Navigate to https://your-dtr-manager.your-domain.com and login using the username and password that you chose during the installation of UCP
By default, you don’t need to license your Docker Trusted Registry. When installing DTR, it automatically starts using the same license file used on your Docker Universal Control Plane cluster.
However, there are some situations when you have to manually license your DTR installation:
For more information visit the official guide.
[ Optional ]
To add replicas to a DTR cluster, use the docker/dtr join command:
# Replicate DTR
docker run -it –rm \
docker/dtr:2.3.2 join \
–ucp-node <ucp-node-name> \
–ucp-insecure-tls
Replace every occurrence of 2.3.2 with whatever version of DTR you are running
For more information on adding DTR Replicas read this guide.
DTR uses the self-signed certificate by default which is generally fine for testing.
However self-signed TLS is considered insecure and docker prevents access to insecure repositories by default. Therefore you will not be able to login to your brand new repository unless you configure the client ie. the computer connecting to DTR, such as UCP, or a developer who wants to access the repository.
Note: You can still login into the dashboard before you complete this step.
There are multiple ways to configure the docker-daemon. However, the preferred method is via the daemon.json configuration file which is often located at /etc/docker/daemon.json. If this file does not exist on your computer, simply create it.
Then edit your client’s daemon.json file to include the line "insecure-registries" : [ "dtr-node.mydomain.com", "dtr2-node.mydomain.com" ]
You will need to include a string in the array for every insecure remote repository you would like to reach.
For example, in a case where you have installed UCP and are now configuring an insecure DTR, you would create a file at /etc/docker/daemon.json on the UCP node that looks like this:
{
“insecure-registries” : [
“dtr-node.mydomain.com”, “dtr2-node.mydomain.com”
]
}
If daemon.json already exists and contains other entries, simply modify the root object {} by adding a , followed by an additional property.
For a full list of available options for daemon.json see the official documentation.
Read this guide for more information about using insecure Repositories in DDC.
The general syntax for logging into a docker repository such as DTR is with docker login dtr.example.org
To learn more about integrating your local computers and UCP with DTR read the official guide.
[ Optional ]
If you installed with Self-signed TLS (default configuration) and later decide to replace your TLS with your own TLS, you can do the following
In older versions of docker, you may be required to register your root CA with your Operating System.
sudo curl -k https://my-ucp-node.mysite.com/ca -o /tmp/my-ca.crt # Add Your CA
# Copy your CA to the crt file that represents your ucp address node
cp /tmp/my-ca.crt /etc/pki/ca-trust/source/anchors/my-ucp-node.mysite.com.crt
# Copy your CA to the crt file that represents your dtr address node
cp /tmp/my-ca.crt /etc/pki/ca-trust/source/anchors/my-dtr-node.mysite.com.crt
# Recognize your new records
sudo update-ca-trust
sudo service docker restartUCP: Configure UCP To Use Your Own TLS Certificates.
DTR: Use Your Own TLS Certificates
If you deployed Docker Trusted Registry, you’ll also need to reconfigure it to trust the new UCP TLS certificates. Learn how to configure DTR.
You may use a command similar to this docker run -it --rm docker/dtr:2.2.5 reconfigure --ucp-insecure-tls to reconfigure DTR to work with the new TLS.
UCP stores some DTR metadata, so when restoring, make sure you use the same version of the docker/dtr image that you’ve used to create the backup.
To quote the article on best practices and design considerations
UCP backup is done using the
docker/ucp backupcommand on a controller node. It stops the UCP containers on the node and perform a full backup of the configuration and state of UCP. Some of this information is sensitive, so you should use the--passphraseoption to encrypt the backup.
The UCP backup includes:
# Backup UCP
# Create a backup, encrypt it, and store it on /tmp/backup.tar
$ docker container run –log-driver none –rm -i –name ucp \
-v /var/run/docker.sock:/var/run/docker.sock \
docker/ucp:2.2.2 backup –interactive \
–passphrase “secret” > /tmp/backup.tar
# Decrypt the backup and list its contents
$ gpg –decrypt /tmp/backup.tar | tar –list
In this case, where we used a password to backup, we must restore it with a password as well.
# Restore UCP
$ docker container run –rm -i –name ucp \
-v /var/run/docker.sock:/var/run/docker.sock \
docker/ucp:2.2.2 restore –passphrase “secret” < /tmp/backup.tar
For more information about backup and restore functions for UCP visit the official docs.
Some DTR metadata is stored in UCP, so the backup command speaks with UCP
To create a backup of DTR you need to:
You should always create backups from the same DTR replica, to ensure a smoother restore.
A DTR backup includes Repository metadata, Access control to repos and images, Notary data, Scan Results, and Certificates and keys. Images themselves need to be backed up separately directly from storage.
For the complete guide to backing up and restoring DTR, visit the official docs.
You can force a UCP node to leave the swarm with docker swarm leave , but you should always leave by using the UCP dashboard or CLI.
If your swarm-mode cluster has lost quorum and the original set of managers is not recoverable, you can attempt to recover a single-manager cluster with docker swarm init --force-new-cluster --advertise-addr <ip:port> --listen-addr <ip:port>
Visit this guide to learn how to troubleshoot UCP and interpret its different states.
This guide was written after my many trials while installing DDC; so I tried to be as complete and accurate as possible. However, if you notice any errors or missing information, feel free to let me know in the comments.
And remember, Docker has great documentation and a great community. If the answer to your questions cannot be found in this guide or any of its linked resources, you can usually find a solution by either browsing the Docker Docs or posting to the Docker Forums.