John Vincent

Overview of Creating and Deploying a Multi-Container Docker Application to AWS

Overview of Creating and Deploying a Multi-Container Docker Application to AWS

by John Vincent

Posted on June 7, 2021

Let's discuss deploying TaskMuncher, a React and Node application, to a Multi-Container Docker Environment at AWS using Dockerhub and Travis CI

For extensive discussions regarding TaskMuncher, please see TaskMuncher Overview

Overview

It is apparent that this is an extremely difficult process with so many complex steps each requiring exact configuration. Further, each technology places its own architectural requirements upon the technologies around it so the architecture of the whole solution is extremely precise and devoid of options.

In software development it is often customary to architect and build from software to deployment. It is thus assumed the deployment process could be sufficiently customized to be able to handle any and all problems that appear as part of the deployment process.

My experience of these technologies is that this is a hazardous resolve. This approach allows for the resolution of architectural issues with a technology only to find architectural issues in the next technology that would cause a return to the drawing board and a considerable amount of rework.

Thus, for this topic, I have described the steps in what would appear to be reverse order. It is not customary but I believe this approach will allow for a better comprehension.

Configuring the Deployment Environment of AWS, Dockerhub and Travis-CI

The following illustrates the dependencies of the various components.

Deploy to AWS dependencies

To configure the above there are a series of tasks. They must be performed in the order shown.

So let's recap.

  • AWS Elastic Beanstalk Application requires the use of a Docker platform and must use Multi-Container Docker running on 64bit Amazon Linux.

  • Elastic Beanstalk doesn't know how to run containers and so it hands it off to Amazon Elastic Container Service (ECS).

  • ECS uses Dockerrun.aws.json, a task definition to run each container.

  • The multi-container Docker platform on Elastic Beanstalk does not support the building of custom Docker images during deployment. Thus custom Docker images are stored in a private Dockerhub repository.

  • AWS needs a .dockercfg file to uploaded to S3 bucket to allow AWS to authenticate with a Dockerhub private repository.

  • Dockerrun.aws.json requires configuration so it can access the .dockercfg file in the S3 bucket.

  • Travis needs to be configured with the AWS Access Keys allowing for access to AWS.

  • Travis needs to be configured with the Dockerhub id and password so it can push the custom images to a Dockerhub private repository.

  • To perform a build and deploy, Travis requires a .travis.yml file with specific instructions.

Thus it is clear there are a series of cascading requirements that dictates the terms of software development. There is more.

  • Deploying to AWS Beanstalk Multi-Container environment implies the existence of a single deployment thread, that being a single .travis.yml.

  • This suggests there is only one Github repository.

  • Running Docker requires a context of the current directory or a subdirectory of the directory in which Docker is running.

  • This requires the source code of the components of the application to be subdirectories of the Docker working directory.

Thus the deployment environment and processes dictate to software development that there shall be one and only one Github repository. This is clearly problematic.

It is necessary to first carefully determine the deployment architecture and then work backwards to determine how to architect the source code to allow for that deployment architecture.

Install Docker

For details, please see Install Docker and review Docker Basics

TaskMuncher AWS Production Docker Containers

TaskMuncher AWS Production Docker Containers

Note

  • Each of these components is a Docker Container.
  • Requests are routed to Nginx server listening on port 80.
  • Requests are proxied based on the URL of the request.
  • Production React are just static files. They need to be served by another Nginx server which is listening on port 8100.
  • Requests for APIs are routed to a Node/Express Server listening on port 3100.
  • Requests for MongoDB access are routed to a MongoDB Container which is listening on port 27017.
  • MongoClient Container seeds the database with the necessary startup data.

AWS offers an excellent MongoDB service. However, for this project I decided to deliver MongoDB as its own container.

TaskMuncher Development Docker Containers

TaskMuncher Development Docker Containers

TaskMuncher Production in Development Docker Containers

TaskMuncher Production in Development Docker Containers

Developer Requirements

Travis-CI Requirements Development Requirements

In summary, the required docker images are

  • client
  • server
  • mongodb
  • nginx
  • mongo-client

for each of the following environments

  • development
  • production in a development environment
  • production at AWS

and also

  • dockercfg

in a development environment. See Allow AWS to authenticate with a Dockerhub private repository for details.

Configuring the Docker Environment

Now

  • it is known what needs to be built
  • how it is to be configured
  • and what network configuration is required

This causes the requirement for the source code to be structured in a certain way.

I have chosen the following folder structure.

Folder Architecture

Configuring the Build Processes

The build requirements are so complicated a build system is a requirement.

I have chosen to use Makefiles. For details, please see Configuring Makefiles to build and run Docker Images using Docker, Docker Compose and Dockerfile

It is important to note that the Makefiles use Docker, Dockerfiles and Docker Compose to perform the various required tasks.

Notice that the builds use either

  • docker
  • docker-compose

Notice how the *-compose.mk Makefiles are really just simple patterns to

  • build
  • start and attach to containers for a service
  • stop containers, networks, volumes, and images created by up.

which, in the end, rely on a Dockerfile to 'configure' the image.

Notice how the build directives in dev.mk and devprod.mk also rely on Dockerfiles.

Dockerfiles

docker-compose.yml files are used for defining and running multi-container Docker applications.

Dockerfiles are simple text files that contain the commands to assemble an image that will be used to deploy containers.

The structure of the ./config folder is as follows.

Config Folder Architecture

The ./config folder shows a number of Dockerfiles.

The configuration of these Dockerfiles is discussed in Configuring Dockerfiles to build Docker Images.

Docker Development Environment

For details, please follow Creating and running a Docker Development Environment

Configuring and Using .env files

The multitude of requirements create the requirement there be many .env files. This requirement creates some challenges.

For details, please follow Configuring and Using .env files

Delete AWS Resources

At completion of the project it is desirable to delete any and all resources that were used for this project. For details, please see Delete all TaskMuncher AWS Resources

Configuration Values

The following are listed here for the sake of simplicity and to remove duplication. I consider it to be a good practice to put all configuration information in one place.

Please note that all of the below were deleted before this document was published.

  • Default VPC vpc-0dc1a7ee0c7bec811

  • Application: taskmuncher-docker

  • Environment: Taskmuncherdocker-env

  • Github repository: taskmuncher

  • Workspace: /Users/jv/Desktop/MyDevelopment/github/website/taskmuncher

  • Diagrams: /Users/jv/Desktop/MyDevelopment/github/website/taskmuncher/diagrams

  • S3 folder: taskmuncher-docker

  • S3 bucket name: elasticbeanstalk-us-east-1-971716655829

  • AWS user: taskmuncher-deployer

  • AWS_ACCESS_KEY:

  • AWS_SECRET_KEY:

  • AWS application url: http://taskmuncherdocker-env.eba-mv2hwnxx.us-east-1.elasticbeanstalk.com/

  • Dockerhub repository: johnvincentio/taskmuncher

References

AWS

Docker

MongoDB