Docker: From Development to Deployment with Django and Vue
Using Docker to isolate my environment has been a game-changer for the teams I've been a part of and me. It is a massive timesaver by installing Docker and then having new developers run docker compose up -d to get going. No more having long lists of things to install and then troubleshoot when they don't install cleanly to get going.
I know there are many different opinions and articles for how to get going with Django on Docker. I'm no expert on the best base image for different situations. I won't get into that. There is already plenty of discussion and debate in various corners of the web.
My plan for this post is to walk through one way to set up Docker for developing and deploying Django-based web applications that use a Vue frontend via container-based deployments and Github Actions automation.
Base Image
Instead of having a single image, I break things into a base image so that in Github Actions, I can leverage some caching. This base image only gets rebuilt if requirements change.
In this Dockerfile.base, I start with using a node image to install all the node dependencies
by copying in the files required for the install and then running yarn install.
Then I do the same for the python requirements, but those require a bit more setup. I set PYTHONFAULTHANDLER
to get a bit more traceback on faults. I also
set PYTHONUNBUFFERED so that output isn't buffered but sent immediately to console. Then, I add a user that will run the processes on the final container
because you don't want to run processes as root.
I then install a bunch of required pre-requisites, followed by the pip install.
I close up this base image build by copying in installed node_modules from the previous layer.
# Transient Node Build Image
FROM node:14.16-alpine
WORKDIR /app
COPY .npmrc package.json yarn.lock webpack.config.js .babelrc /app/
RUN yarn install
# Python Runtime Image
FROM python:3.9-alpine3.14
ENV PYTHONFAULTHANDLER=1 \
PYTHONUNBUFFERED=1
WORKDIR /app
# Install dependencies & setup user
RUN adduser -D appuser && \
apk add --no-cache --virtual .build-deps g++ gcc libffi-dev musl-dev libevent-dev openssl-dev python3-dev make && \
apk add --no-cache git postgresql-dev binutils && \
apk add --no-cache jpeg-dev zlib-dev freetype-dev lcms2-dev openjpeg-dev tiff-dev tk-dev tcl-dev && \
apk add --no-cache yarn
# Copy requirements
COPY requirements.txt .
RUN pip install -U pip && \
pip install --no-cache-dir -r requirements.txt && \
apk del --no-cache .build-deps
# Copy node_modules and yarn
COPY --from=0 /app/node_modules /app/node_modulesMain Image
In this main image, I'm pulling the latest copy of the base image that Github Actions built based on the Dockerfile.base config.
Specifying GIT_COMMIT, GIT_VERSION, and VERSION as ARG parameters allow those environment variables to be passed into this
image so I can reference as environment variables in processes that run on the container.
The rest of this is pretty simple. I copy in the code, set ownership over the copied in files to the user I use for
process execution, build the frontend bundle and then run gunicorn.
ARG BASE_TAG=latest
FROM ghcr.io/[repo]/[image-name]:$BASE_TAG
ARG GIT_COMMIT
ARG GIT_VERSION
ARG VERSION
WORKDIR /app
# Copy full source
COPY . .
RUN chown -R appuser:appuser /app
RUN yarn build
USER appuser
CMD gunicorn [your package].wsgi --reload --bind 0.0.0.0:$PORT --threads 4 --log-file -
EXPOSE 8000Docker Compose Configuration
For local development, I build up the services I need using Docker Compose to avoid having to install things like Redis and Postgres on my local machine. This makes it easy to try out different versions and keeps my laptop relatively clean.
The Postgres image will create the database defined by POSTGRES_DB on boot up if it doesn't already exist. I like to map a
local .data/ directory to the path that Postgres uses for its data rather than using docker volumes as it makes it easy for me
swap out different database states if switching between branches with migrations by just moving .data/ directories to
other temporary named folders between reboots.
The healthcheck sections for both Redis and Postgres mean that I can have the Django service wait to come online until those
required services are healthy and ready.
Even though the base image has the built frontend requirements, that's really for production deployments. I like to run the webpack dev server for development to get all the fun hot loading of CSS/JS changes. Since that runs on a different
port, I run it as a standalone service. For this, I can use a stock node image, map the local source directory into
the container and run yarn install and yarn start on bootup.
Finally, I startup the django container using the Dockerfile we've just created. I override default Django settings by defining env variables in the environment section. Here we are defining REDIS_URL and DATABASE_URL to use the docker compose services. The Django project's settings.py module reads these variables from the environment.
version: '3'networks: localdev:services: postgres: image: postgres:13-alpine container_name: postgres restart: unless-stopped environment: POSTGRES_DB: myapp POSTGRES_USER: myapp POSTGRES_PASSWORD: myapp volumes: - ./.data:/var/lib/postgresql/data/ - ./:/app ports: - "54321:5432" networks: - localdev healthcheck: test: ["CMD", "pg_isready", "-U", "myapp"] interval: 1s timeout: 3s retries: 30 redis: image: redis:4.0.14-alpine container_name: redis restart: unless-stopped networks: - localdev healthcheck: test: ["CMD", "redis-cli", "ping"] interval: 1s timeout: 3s retries: 30 npm: image: node:14.16-alpine container_name: frontend volumes: - ./:/app working_dir: /app networks: - localdev ports: - "8080:8080" command: /bin/sh -c 'yarn install && yarn start' django: image: django build: context: . dockerfile: Dockerfile container_name: django working_dir: /app volumes: - ./:/app ports: - "8000:8000" depends_on: postgres: condition: service_healthy redis: condition: service_healthy environment: DATABASE_URL: postgres://myapp:myapp@postgres/myapp REDIS_URL: redis://redis:6379 networks: - localdevGithub Actions
On Every Push
On every push, detected changes to requirements.txt or yarn.lock files trigger building a new base image.
The build-base-image job does the following:
- Sets up an output variable called
tagthat we'll store the image tag in - Checks out the code
- Computes the
tagoutput variable based on a hash of the dependencies files - Sets up the Docker Buildx action
- Authenticates with the Github container registry
- Sets up a cache for the docker layers
- Builds and pushes the base image to the registry
If the building and publishing of this base image is successful, then we run, in parallel,
the lint-and-test and build-and-push-images steps. We run these in parallel to save on some build time at the expense of potentially wasted compute time cost building images
when lint-and-test fails.
The lint-and-test step is what you'd expect from most CI setups:
- Setup services
- Check out code
- Install dependencies
- Run lints and tests
- Upload code coverage reports
The build-and-push-images step takes the current code and builds an image based off the
latest base image and pushes it to the container registry:
- Check out code
- Creates tags for the image
- Sets up the Docker Buildx action
- Authenticates with the Github container registry
- Sets up a cache for the docker layers
- Builds and pushes the production image to the registry
name: Test and Buildon: push: branches: "**" # All Branches tags-ignore: "**" # Releases also push tags (and so trigger both events). Don't double release these images.jobs: build-base-image: name: Build Base Image runs-on: ubuntu-latest outputs: tag: ${{ steps.base-tags.outputs.tag }} steps: - name: Checkout uses: actions/checkout@v2 - name: Prep Tag Data id: base-tags run: echo "::set-output name=tag::${{ hashFiles('**/requirements.txt', '**/yarn.lock') }}"; - name: Set up Docker Buildx uses: docker/setup-buildx-action@v1 - name: Login to GH Container Registry uses: docker/login-action@v1 with: registry: ghcr.io username: ${{ secrets.CR_UN }} password: ${{ secrets.CR_PAT }} - name: Cache Docker layers uses: actions/cache@v2 id: base-image-cache with: path: /tmp/.buildx-cache key: ${{ runner.os }}-buildx-${{ steps.base-tags.outputs.tag }}-v2 - name: Build and Push Base Image uses: docker/build-push-action@v2 if: steps.base-image-cache.outputs.cache-hit != 'true' with: context: . file: ./.docker/Dockerfile.base push: true # Tag :latest and a hash of requirements.txt + yarn.lock tags: | ghcr.io/${{ github.repository }}-base:${{ steps.base-tags.outputs.tag }} ghcr.io/${{ github.repository }}-base:latest cache-from: type=local,src=/tmp/.buildx-cache cache-to: type=local,dest=/tmp/.buildx-cache lint-and-test: name: Linting and Testing runs-on: ubuntu-18.04 needs: build-base-image services: image: postgres env: POSTGRES_DB: postgres POSTGRES_USER: postgres POSTGRES_PASSWORD: postgres ports: - 5432:5432 options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5 redis: image: redis ports: - 6379:6379 options: --entrypoint redis-server env: PYTHONDONTWRITEBYTECODE: 1 PYTHON_VERSION: 3.9 DATABASE_SSL: "off" DATABASE_URL: "postgresql://postgres:postgres@localhost:5432/postgres" steps: - name: Cancel Previous Runs uses: styfle/cancel-workflow-action@0.8.0 with: access_token: ${{ github.token }} - uses: actions/checkout@v2 - name: Setup Python uses: actions/setup-python@v1 if: steps.cache-venv.outputs.cache-hit == false with: python-version: ${{env.PYTHON_VERSION}} - name: Setup Python Cache uses: actions/cache@v2 id: cache-venv # name for referring later with: path: /opt/hostedtoolcache/Python/ key: v2-${{ runner.os }}-python-${{env.PYTHON_VERSION}}-venv-${{ hashFiles('**/requirements*.txt') }} restore-keys: | ${{ runner.os }}-venv- - name: Install Python Dependencies if: steps.cache-venv.outputs.cache-hit == false run: pip install -r requirements.txt - name: Print Versions run: pip freeze - name: Get Yarn Cache Directory Path id: yarn-cache-dir-path run: echo "::set-output name=dir::$(yarn cache dir)" - name: Cache Yarn uses: actions/cache@v2 id: yarn-cache # use this to check for `cache-hit` (`steps.yarn-cache.outputs.cache-hit != 'true'`) with: path: ${{ steps.yarn-cache-dir-path.outputs.dir }} key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }} restore-keys: | ${{ runner.os }}-yarn- - name: Install run: yarn --prefer-offline - name: Test Frontend run: yarn test - name: Build Frontend run: yarn build - name: Collect Static run: python manage.py collectstatic - name: Linting run: flake8 - name: Checking for Missing Migrations run: python manage.py makemigrations --check --dry-run - name: Running Python Tests run: coverage run -m pytest -vv --nomigrations - name: Uploading Coverage Report uses: codecov/codecov-action@v2 with: token: ${{ secrets.CODECOV_TOKEN }} files: ./coverage.xml fail_ci_if_error: true build-and-push-images: name: Build and Push Images needs: build-base-image runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v2 - name: Prep Tag Data id: tags run: | export REF=${GITHUB_REF#refs/*/}; export VERSION=${REF//[^[:alpha:][:digit:]\.\-\_]/}; echo "::set-output name=version::$VERSION"; echo "::set-output name=minor::${VERSION%.*}.x"; echo "::set-output name=major::${VERSION%.*.*}.x.x"; echo "::set-output name=sha7::${GITHUB_SHA::7}"; - name: Set up Docker Buildx uses: docker/setup-buildx-action@v1 - name: Login to GH Container Registry uses: docker/login-action@v1 with: registry: ghcr.io username: ${{ secrets.CR_UN }} password: ${{ secrets.CR_PAT }} - name: Cache Docker layers uses: actions/cache@v2 with: path: /tmp/.buildx-cache key: ${{ runner.os }}-buildx-${{ github.sha }} restore-keys: | ${{ runner.os }}-buildx- - name: Build and Push Dev Images uses: docker/build-push-action@v2 with: context: . file: ./.docker/Dockerfile push: true # Tag :branchname, and :commit-sha tags: | ghcr.io/${{ github.repository }}:${{ steps.tags.outputs.version }} ghcr.io/${{ github.repository }}:${{ steps.tags.outputs.sha7 }} build-args: | BASE_TAG=${{ needs.build-base-image.outputs.tag }} GIT_COMMIT=${{ github.sha }} GIT_VERSION=${{ github.ref }} VERSION=${{ steps.tags.outputs.sha7 }} cache-from: type=local,src=/tmp/.buildx-cache cache-to: type=local,dest=/tmp/.buildx-cache deploy: name: Deploy QA needs: [build-and-push-images, lint-and-test] # Only run on pushes to master if: ${{ github.event.ref == 'refs/heads/master' }} runs-on: ubuntu-latest env: HEROKU_API_KEY: ${{ secrets.HEROKU_API_KEY }} steps: - name: Checkout uses: actions/checkout@v2 - name: Set up Docker Buildx uses: docker/setup-buildx-action@v1 - name: Login to GH Container Registry uses: docker/login-action@v1 with: registry: ghcr.io username: ${{ secrets.CR_UN }} password: ${{ secrets.CR_PAT }} - name: Install Heroku run: curl https://cli-assets.heroku.com/install.sh | sh - name: Heroku Container Login run: heroku container:login - name: Push All Processes run: heroku container:push -R -a ${{ secrets.HEROKU_APP_NAME }} --arg IMAGE=master working-directory: ./deploy - name: Release run: heroku container:release --app ${{ secrets.HEROKU_APP_NAME }} web worker releaseA keen eye will notice on Push All Processes there is a working-directory specified. In this
./deploy path, we have the following Dockerfiles:
Dockerfile.releaseDockerfile.webDockerfile.worker
The heroku container:push -R command will use each Dockerfile.* definition to create containers for
each process/service we need. These are very simple Dockerfiles that all use the image we just pushed. They are used to execute different things.
The web one is the main one and is what runs the web
dynos. You might not need the worker, but I typically find it helpful to have a django-rq worker
available to offload tasks from the web request/response cycle. Finally, the release is for running
release scripts, most commonly a python manage.py migrate command to migrate the database on release.
Dockerfile.release:
ARG IMAGE=latest
FROM ghcr.io/[org]/[repo]:${IMAGE}
CMD python manage.py migrateDockerfile.web:
ARG IMAGE=latest
FROM ghcr.io/[org]/[repo]:${IMAGE}Dockerfile.worker:
ARG IMAGE=latest
FROM ghcr.io/[org]/[repo]:${IMAGE}
CMD python manage.py rqworker [queue1] [queue2]Additional Steps
So far, I've outlined a development to deployment pipeline that deploys code when it lands on master after lints and
tests pass. If you only need a single environment, then this pipeline will suffice. However, as your project grows
and you desire more rigor in the process, you may very well end up with a "staging" or "qa" environment where QA
teams can run integration and regression testing and product owners can do acceptance testing.
Once you arrive at this point, you can then leverage the Releases feature of Github to "promote" the image to your
production environment. The auto-deploys from master will continue to happen, but that environment now assumes
the role of "staging."
You then would create a new Heroku app and add the HEROKU_PROD_APP_NAME secret to your repository and add an
addition workflow to your .github folder.
Let's call this one prod.yml:
name: Promote Tag and Production Releaseon: release: types: [published]jobs: promote-tag: name: Promote Docker Tag runs-on: ubuntu-18.04 outputs: version: ${{ steps.tags.outputs.version }} steps: - name: Prep Tag Data id: tags run: | export REF=${GITHUB_REF#refs/*/}; export VERSION=${REF//[^[:alpha:][:digit:]\.\-\_]/}; echo "::set-output name=version::$VERSION"; echo "::set-output name=minor::${VERSION%.*}.x"; echo "::set-output name=major::${VERSION%.*.*}.x.x"; echo "::set-output name=sha7::${GITHUB_SHA::7}"; - name: Set up Docker Buildx uses: docker/setup-buildx-action@v1 - name: Login to GH Container Registry uses: docker/login-action@v1 with: registry: ghcr.io username: ${{ secrets.CR_UN }} password: ${{ secrets.CR_PAT }} - name: Pull Verified Image run: docker pull ghcr.io/${{ github.repository }}:${{ steps.tags.outputs.sha7 }} - name: Tag Verified Image run: | docker image tag ghcr.io/${{ github.repository }}:${{ steps.tags.outputs.sha7 }} ghcr.io/${{ github.repository }}:latest docker image tag ghcr.io/${{ github.repository }}:${{ steps.tags.outputs.sha7 }} ghcr.io/${{ github.repository }}:${{ steps.tags.outputs.version }} docker image tag ghcr.io/${{ github.repository }}:${{ steps.tags.outputs.sha7 }} ghcr.io/${{ github.repository }}:${{ steps.tags.outputs.minor }} docker image tag ghcr.io/${{ github.repository }}:${{ steps.tags.outputs.sha7 }} ghcr.io/${{ github.repository }}:${{ steps.tags.outputs.major }} - name: Push Promoted Image Tags run: docker image push --all-tags ghcr.io/${{ github.repository }} deploy: name: Deploy runs-on: ubuntu-latest env: VERSION: ${{ needs.promote-tag.outputs.version }} steps: - name: Setup Docker Buildx uses: docker/setup-buildx-action@v1 - name: Login to GH Container Registry uses: docker/login-action@v1 with: registry: ghcr.io username: ${{ secrets.GH_USERNAME }} password: ${{ secrets.GH_PAT }} - name: Install Heroku run: curl https://cli-assets.heroku.com/install.sh | sh - name: Heroku Container Login run: heroku container:login - name: Push All Processes run: heroku container:push -R -a ${{ secrets.HEROKU_PROD_APP_NAME }} --arg VERSION=$VERSION working-directory: ./deploy - name: Release run: heroku container:release -a ${{ secrets.HEROKU_PROD_APP_NAME }} web worker releaseAs you can tell, there is a lot of duplication in these Github Actions steps. The new Github Composite Actions feature should be able to tidy this up. I'll save that for a future post.