Patrick Altman
Docker: From Development to Deployment with Django and Vue

Docker: From Development to Deployment with Django and Vue

Using Docker to isolate my environment has been a game-changer for the teams I've been a part of and me. It is a massive timesaver by installing Docker and then having new developers run docker compose up -d to get going. No more having long lists of things to install and then troubleshoot when they don't install cleanly to get going.

I know there are many different opinions and articles for how to get going with Django on Docker. I'm no expert on the best base image for different situations. I won't get into that. There is already plenty of discussion and debate in various corners of the web.

My plan for this post is to walk through one way to set up Docker for developing and deploying Django-based web applications that use a Vue frontend via container-based deployments and Github Actions automation.

Base Image

Instead of having a single image, I break things into a base image so that in Github Actions, I can leverage some caching. This base image only gets rebuilt if requirements change.

In this Dockerfile.base, I start with using a node image to install all the node dependencies by copying in the files required for the install and then running yarn install.

Then I do the same for the python requirements, but those require a bit more setup. I set PYTHONFAULTHANDLER to get a bit more traceback on faults. I also set PYTHONUNBUFFERED so that output isn't buffered but sent immediately to console. Then, I add a user that will run the processes on the final container because you don't want to run processes as root. I then install a bunch of required pre-requisites, followed by the pip install.

I close up this base image build by copying in installed node_modules from the previous layer.

# Transient Node Build Image
FROM node:14.16-alpine
WORKDIR /app
COPY .npmrc package.json yarn.lock webpack.config.js .babelrc /app/
RUN yarn install

# Python Runtime Image
FROM python:3.9-alpine3.14

ENV PYTHONFAULTHANDLER=1 \
    PYTHONUNBUFFERED=1

WORKDIR /app

# Install dependencies & setup user
RUN adduser -D appuser && \
    apk add --no-cache --virtual .build-deps g++ gcc libffi-dev musl-dev libevent-dev openssl-dev python3-dev make && \
    apk add --no-cache git postgresql-dev binutils && \
    apk add --no-cache jpeg-dev zlib-dev freetype-dev lcms2-dev openjpeg-dev tiff-dev tk-dev tcl-dev && \
    apk add --no-cache yarn

# Copy requirements
COPY requirements.txt .

RUN pip install -U pip && \
    pip install --no-cache-dir -r requirements.txt && \
    apk del --no-cache .build-deps

# Copy node_modules and yarn
COPY --from=0 /app/node_modules /app/node_modules

Main Image

In this main image, I'm pulling the latest copy of the base image that Github Actions built based on the Dockerfile.base config.

Specifying GIT_COMMIT, GIT_VERSION, and VERSION as ARG parameters allow those environment variables to be passed into this image so I can reference as environment variables in processes that run on the container.

The rest of this is pretty simple. I copy in the code, set ownership over the copied in files to the user I use for process execution, build the frontend bundle and then run gunicorn.

ARG BASE_TAG=latest
FROM ghcr.io/[repo]/[image-name]:$BASE_TAG

ARG GIT_COMMIT
ARG GIT_VERSION
ARG VERSION

WORKDIR /app

# Copy full source
COPY . .

RUN chown -R appuser:appuser /app

RUN yarn build

USER appuser
CMD gunicorn [your package].wsgi --reload --bind 0.0.0.0:$PORT --threads 4 --log-file -
EXPOSE 8000

Docker Compose Configuration

For local development, I build up the services I need using Docker Compose to avoid having to install things like Redis and Postgres on my local machine. This makes it easy to try out different versions and keeps my laptop relatively clean.

The Postgres image will create the database defined by POSTGRES_DB on boot up if it doesn't already exist. I like to map a local .data/ directory to the path that Postgres uses for its data rather than using docker volumes as it makes it easy for me swap out different database states if switching between branches with migrations by just moving .data/ directories to other temporary named folders between reboots.

The healthcheck sections for both Redis and Postgres mean that I can have the Django service wait to come online until those required services are healthy and ready.

Even though the base image has the built frontend requirements, that's really for production deployments. I like to run the webpack dev server for development to get all the fun hot loading of CSS/JS changes. Since that runs on a different port, I run it as a standalone service. For this, I can use a stock node image, map the local source directory into the container and run yarn install and yarn start on bootup.

Finally, I startup the django container using the Dockerfile we've just created. I override default Django settings by defining env variables in the environment section. Here we are defining REDIS_URL and DATABASE_URL to use the docker compose services. The Django project's settings.py module reads these variables from the environment.

version: '3'

networks:
    localdev:

services:
    postgres:
        image: postgres:13-alpine
        container_name: postgres
        restart: unless-stopped
        environment:
            POSTGRES_DB: myapp
            POSTGRES_USER: myapp
            POSTGRES_PASSWORD: myapp
        volumes:
            - ./.data:/var/lib/postgresql/data/
            - ./:/app
        ports:
            - "54321:5432"
        networks:
            - localdev
        healthcheck:
            test: ["CMD", "pg_isready", "-U", "myapp"]
            interval: 1s
            timeout: 3s
            retries: 30
    redis:
        image: redis:4.0.14-alpine
        container_name: redis
        restart: unless-stopped
        networks:
            - localdev
        healthcheck:
            test: ["CMD", "redis-cli", "ping"]
            interval: 1s
            timeout: 3s
            retries: 30
    npm:
        image: node:14.16-alpine
        container_name: frontend
        volumes:
            - ./:/app
        working_dir: /app
        networks:
            - localdev
        ports:
            - "8080:8080"
        command: /bin/sh -c 'yarn install && yarn start'
    django:
        image: django
        build:
            context: .
            dockerfile: Dockerfile
        container_name: django
        working_dir: /app
        volumes:
            - ./:/app
        ports:
            - "8000:8000"
        depends_on:
            postgres:
                condition: service_healthy
            redis:
                condition: service_healthy
        environment:
            DATABASE_URL: postgres://myapp:myapp@postgres/myapp
            REDIS_URL: redis://redis:6379
        networks:
            - localdev

Github Actions

On Every Push

On every push, detected changes to requirements.txt or yarn.lock files trigger building a new base image.

The build-base-image job does the following:

  1. Sets up an output variable called tag that we'll store the image tag in
  2. Checks out the code
  3. Computes the tag output variable based on a hash of the dependencies files
  4. Sets up the Docker Buildx action
  5. Authenticates with the Github container registry
  6. Sets up a cache for the docker layers
  7. Builds and pushes the base image to the registry

If the building and publishing of this base image is successful, then we run, in parallel, the lint-and-test and build-and-push-images steps. We run these in parallel to save on some build time at the expense of potentially wasted compute time cost building images when lint-and-test fails.

The lint-and-test step is what you'd expect from most CI setups:

  1. Setup services
  2. Check out code
  3. Install dependencies
  4. Run lints and tests
  5. Upload code coverage reports

The build-and-push-images step takes the current code and builds an image based off the latest base image and pushes it to the container registry:

  1. Check out code
  2. Creates tags for the image
  3. Sets up the Docker Buildx action
  4. Authenticates with the Github container registry
  5. Sets up a cache for the docker layers
  6. Builds and pushes the production image to the registry
name: Test and Build
on:
  push:
    branches: "**" # All Branches
    tags-ignore: "**" # Releases also push tags (and so trigger both events). Don't double release these images.
jobs:
  build-base-image:
    name: Build Base Image
    runs-on: ubuntu-latest
    outputs:
      tag: ${{ steps.base-tags.outputs.tag }}
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Prep Tag Data
        id: base-tags
        run: echo "::set-output name=tag::${{ hashFiles('**/requirements.txt', '**/yarn.lock') }}";

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1

      - name: Login to GH Container Registry
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ secrets.CR_UN }}
          password: ${{ secrets.CR_PAT }}

      - name: Cache Docker layers
        uses: actions/cache@v2
        id: base-image-cache
        with:
          path: /tmp/.buildx-cache
          key: ${{ runner.os }}-buildx-${{ steps.base-tags.outputs.tag }}-v2

      - name: Build and Push Base Image
        uses: docker/build-push-action@v2
        if: steps.base-image-cache.outputs.cache-hit != 'true'
        with:
          context: .
          file: ./.docker/Dockerfile.base
          push: true
          # Tag :latest and a hash of requirements.txt + yarn.lock
          tags: |
            ghcr.io/${{ github.repository }}-base:${{ steps.base-tags.outputs.tag }}
            ghcr.io/${{ github.repository }}-base:latest
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,dest=/tmp/.buildx-cache

  lint-and-test:
    name: Linting and Testing
    runs-on: ubuntu-18.04
    needs: build-base-image
    services:
      image: postgres
        env:
          POSTGRES_DB: postgres
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
        ports:
          - 5432:5432
        options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
      redis:
        image: redis
        ports:
          - 6379:6379
        options: --entrypoint redis-server
    env:
      PYTHONDONTWRITEBYTECODE: 1
      PYTHON_VERSION: 3.9
      DATABASE_SSL: "off"
      DATABASE_URL: "postgresql://postgres:postgres@localhost:5432/postgres"

    steps:
      -
        name: Cancel Previous Runs
        uses: styfle/cancel-workflow-action@0.8.0
        with:
          access_token: ${{ github.token }}

      - uses: actions/checkout@v2

      - name: Setup Python
        uses: actions/setup-python@v1
        if: steps.cache-venv.outputs.cache-hit == false
        with:
          python-version: ${{env.PYTHON_VERSION}}

      - name: Setup Python Cache
        uses: actions/cache@v2
        id: cache-venv  # name for referring later
        with:
            path: /opt/hostedtoolcache/Python/
            key: v2-${{ runner.os }}-python-${{env.PYTHON_VERSION}}-venv-${{ hashFiles('**/requirements*.txt') }}
            restore-keys: |
                ${{ runner.os }}-venv-

      - name: Install Python Dependencies
        if: steps.cache-venv.outputs.cache-hit == false
        run: pip install -r requirements.txt

      - name: Print Versions
        run: pip freeze

      - name: Get Yarn Cache Directory Path
        id: yarn-cache-dir-path
        run: echo "::set-output name=dir::$(yarn cache dir)"

      - name: Cache Yarn
        uses: actions/cache@v2
        id: yarn-cache # use this to check for `cache-hit` (`steps.yarn-cache.outputs.cache-hit != 'true'`)
        with:
          path: ${{ steps.yarn-cache-dir-path.outputs.dir }}
          key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
          restore-keys: |
            ${{ runner.os }}-yarn-

      - name: Install
        run: yarn --prefer-offline

      - name: Test Frontend
        run: yarn test

      - name: Build Frontend
        run: yarn build

      - name: Collect Static
        run: python manage.py collectstatic

      - name: Linting
        run: flake8

      - name: Checking for Missing Migrations
        run: python manage.py makemigrations --check --dry-run

      - name: Running Python Tests
        run: coverage run -m pytest -vv --nomigrations

      - name: Uploading Coverage Report
        uses: codecov/codecov-action@v2
        with:
          token: ${{ secrets.CODECOV_TOKEN }}
          files: ./coverage.xml
          fail_ci_if_error: true

  build-and-push-images:
    name: Build and Push Images
    needs: build-base-image
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Prep Tag Data
        id: tags
        run: |
          export REF=${GITHUB_REF#refs/*/};
          export VERSION=${REF//[^[:alpha:][:digit:]\.\-\_]/};
          echo "::set-output name=version::$VERSION";
          echo "::set-output name=minor::${VERSION%.*}.x";
          echo "::set-output name=major::${VERSION%.*.*}.x.x";
          echo "::set-output name=sha7::${GITHUB_SHA::7}";

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1

      - name: Login to GH Container Registry
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ secrets.CR_UN }}
          password: ${{ secrets.CR_PAT }}

      - name: Cache Docker layers
        uses: actions/cache@v2
        with:
          path: /tmp/.buildx-cache
          key: ${{ runner.os }}-buildx-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-buildx-

      - name: Build and Push Dev Images
        uses: docker/build-push-action@v2
        with:
          context: .
          file: ./.docker/Dockerfile
          push: true
          # Tag :branchname, and :commit-sha
          tags: |
            ghcr.io/${{ github.repository }}:${{ steps.tags.outputs.version }}
            ghcr.io/${{ github.repository }}:${{ steps.tags.outputs.sha7 }}
          build-args: |
            BASE_TAG=${{ needs.build-base-image.outputs.tag }}
            GIT_COMMIT=${{ github.sha }}
            GIT_VERSION=${{ github.ref }}
            VERSION=${{ steps.tags.outputs.sha7 }}
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,dest=/tmp/.buildx-cache

  deploy:
    name: Deploy QA
    needs: [build-and-push-images, lint-and-test]
    # Only run on pushes to master
    if: ${{ github.event.ref == 'refs/heads/master' }}
    runs-on: ubuntu-latest
    env:
      HEROKU_API_KEY: ${{ secrets.HEROKU_API_KEY }}
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1

      - name: Login to GH Container Registry
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ secrets.CR_UN }}
          password: ${{ secrets.CR_PAT }}

      - name: Install Heroku
        run: curl https://cli-assets.heroku.com/install.sh | sh

      - name: Heroku Container Login
        run: heroku container:login

      - name: Push All Processes
        run: heroku container:push -R -a ${{ secrets.HEROKU_APP_NAME }} --arg IMAGE=master
        working-directory: ./deploy

      - name: Release
        run: heroku container:release --app ${{ secrets.HEROKU_APP_NAME }} web worker release

A keen eye will notice on Push All Processes there is a working-directory specified. In this ./deploy path, we have the following Dockerfiles:

  • Dockerfile.release
  • Dockerfile.web
  • Dockerfile.worker

The heroku container:push -R command will use each Dockerfile.* definition to create containers for each process/service we need. These are very simple Dockerfiles that all use the image we just pushed. They are used to execute different things.

The web one is the main one and is what runs the web dynos. You might not need the worker, but I typically find it helpful to have a django-rq worker available to offload tasks from the web request/response cycle. Finally, the release is for running release scripts, most commonly a python manage.py migrate command to migrate the database on release.

Dockerfile.release:

ARG IMAGE=latest
FROM ghcr.io/[org]/[repo]:${IMAGE}
CMD python manage.py migrate

Dockerfile.web:

ARG IMAGE=latest
FROM ghcr.io/[org]/[repo]:${IMAGE}

Dockerfile.worker:

ARG IMAGE=latest
FROM ghcr.io/[org]/[repo]:${IMAGE}
CMD python manage.py rqworker [queue1] [queue2]

Additional Steps

So far, I've outlined a development to deployment pipeline that deploys code when it lands on master after lints and tests pass. If you only need a single environment, then this pipeline will suffice. However, as your project grows and you desire more rigor in the process, you may very well end up with a "staging" or "qa" environment where QA teams can run integration and regression testing and product owners can do acceptance testing.

Once you arrive at this point, you can then leverage the Releases feature of Github to "promote" the image to your production environment. The auto-deploys from master will continue to happen, but that environment now assumes the role of "staging."

You then would create a new Heroku app and add the HEROKU_PROD_APP_NAME secret to your repository and add an addition workflow to your .github folder.

Let's call this one prod.yml:

name: Promote Tag and Production Release
on:
  release:
    types: [published]
jobs:
  promote-tag:
    name: Promote Docker Tag
    runs-on: ubuntu-18.04
    outputs:
      version: ${{ steps.tags.outputs.version }}
    steps:
      -
        name: Prep Tag Data
        id: tags
        run: |
          export REF=${GITHUB_REF#refs/*/};
          export VERSION=${REF//[^[:alpha:][:digit:]\.\-\_]/};
          echo "::set-output name=version::$VERSION";
          echo "::set-output name=minor::${VERSION%.*}.x";
          echo "::set-output name=major::${VERSION%.*.*}.x.x";
          echo "::set-output name=sha7::${GITHUB_SHA::7}";
      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      -
        name: Login to GH Container Registry
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ secrets.CR_UN }}
          password: ${{ secrets.CR_PAT }}
      -
        name: Pull Verified Image
        run: docker pull ghcr.io/${{ github.repository }}:${{ steps.tags.outputs.sha7 }}
      -
        name: Tag Verified Image
        run: |
          docker image tag ghcr.io/${{ github.repository }}:${{ steps.tags.outputs.sha7 }} ghcr.io/${{ github.repository }}:latest
          docker image tag ghcr.io/${{ github.repository }}:${{ steps.tags.outputs.sha7 }} ghcr.io/${{ github.repository }}:${{ steps.tags.outputs.version }}
          docker image tag ghcr.io/${{ github.repository }}:${{ steps.tags.outputs.sha7 }} ghcr.io/${{ github.repository }}:${{ steps.tags.outputs.minor }}
          docker image tag ghcr.io/${{ github.repository }}:${{ steps.tags.outputs.sha7 }} ghcr.io/${{ github.repository }}:${{ steps.tags.outputs.major }}
      -
        name: Push Promoted Image Tags
        run: docker image push --all-tags ghcr.io/${{ github.repository }}
  deploy:
    name: Deploy
    runs-on: ubuntu-latest
    env:
      VERSION: ${{ needs.promote-tag.outputs.version }}
    steps:
      - name: Setup Docker Buildx
        uses: docker/setup-buildx-action@v1

      - name: Login to GH Container Registry
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ secrets.GH_USERNAME }}
          password: ${{ secrets.GH_PAT }}

      - name: Install Heroku
        run: curl https://cli-assets.heroku.com/install.sh | sh

      - name: Heroku Container Login
        run: heroku container:login

      - name: Push All Processes
        run: heroku container:push -R -a ${{ secrets.HEROKU_PROD_APP_NAME }} --arg VERSION=$VERSION
        working-directory: ./deploy

      - name: Release
        run: heroku container:release -a ${{ secrets.HEROKU_PROD_APP_NAME }} web worker release

As you can tell, there is a lot of duplication in these Github Actions steps. The new Github Composite Actions feature should be able to tidy this up. I'll save that for a future post.

Would love to hear from you. You can find me on Twitter @paltman