← Thomas Lackemann

Continuous Integration using GitLab with Elastic Beanstalk

Posted on Mar 31, 2016 in

Every time I’m faced with the decision to choose an infrastructure for a side-project I always lean towards Elastic Beanstalk. For starters, the system is simple to setup, it’s relatively painless to deploy an application, and I have the peace of mind knowing that if I succumb to the “reddit hug of death” that my app can scale up automatically as necessary.

With all the pros of Elastic Beanstalk though, I fear there’s a few places where often I go wrong when it comes to deploying an application. I hope this article serves as a guide as well as a “lessons learned” for those who wish to setup their project on Elastic Beanstalk and deploy using an automated Continuous Integration system.

Disclaimer

This article is geared heavily for Node.js projects; however, the principles can be applied to any language using any CI with hopefully very small modifications.

Prerequisites

Before we start our adventure, I assume you have the following in place:

  1. GitLab Server with root access
  2. AWS ElasticBeanstalk instance

There are enough guides out there on how to spin up a new ElasticBeanstalk app so I don’t want to dive too much into the specifics. You can follow along with the article by simply creating a new application and deploying the Node.js sample app. Be sure to note your application’s name and environment name.

Setup

Before we can start writing code, we obviously need a repository in GitLab setup for our project.

  1. Create a new repository or navigate to an existing one
  2. Click Settings on the sidebar
  3. Click Runners on the sidebar
  4. Note the unique information found under the section “How to setup a new project specific runner”

Install GitLab Runner

One of the coolest parts about running your own private GitLab instance is you have absolute control over your integrations. I simply host a $20/mo instance on Digital Ocean and it’s been fantastic, especially with automated daily backups.

Plugs aside, having root access to GitLab is important as we’re going to need the right permissions to install gitlab-ci-multi-runner which is extremely easy to do.

# ssh into your gitlab server
ssh root@gitlab.example.com

# GitLab Runner is best used with Docker
curl -sSL https://get.docker.com/ | sh

# Add GitLab Runner packages (debian)
curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-ci-multi-runner/script.deb.sh | sudo bash

# Install GitLab Runner
apt-get install gitlab-ci-multi-runner

You can find more in-depth documentation about how to Install GitLab Runner for your server.

Register Runner

Now that the runner is installed, we need to register a new runner with our GitLab instance.

gitlab-ci-multi-runner register

Now simply follow the instructions.

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/ci )
# GitLab CI URL noted in step 4

Please enter the gitlab-ci token for this runner
# Project token noted in step 4

Please enter the gitlab-ci description for this runner
Builds, tests, and deploys nodejs-app to ElasticBeanstalk

Please enter the executor: shell, docker, docker-ssh, ssh?
docker

Please enter the Docker image (eg. ruby:2.1):
node:5.9.1

So, what did we just do? We registered a new runner with GitLab, we gave it a description of Builds, tests, and deploys nodejs-app to ElasticBeanstalk and we assigned it to a specific project given by the gitlab-ci token.

Now we’re ready to start writing our GitLab Runner configuration.

Configure GitLab Runner

If you’re starting with a blank repository, feel free to clone a boilerplate ES6 repository so we can dive right into the interesting bits.

$ git clone https://github.com/tlackemann/nodejs-es6-ci.git

Open or create .gitlab-ci.yml in your project root and change the values under variables to your actual AWS credentials. This file should be fairly well documented as to what our pipeline is going to look like.

# GitLab CI Docker Image
image: node:5.9.1

# Build - Build necessary JS files
# Test - Run tests
# Deploy - Deploy application to S3/ElasticBeanstalk
stages:
  - build
  - test
  - deploy

# Configuration
variables:
  AWS_ACCESS_KEY_ID: "" # Should have access to both S3/EB
  AWS_SECRET_ACCESS_KEY: ""
  AWS_DEFAULT_REGION: "us-east-1" # Or, wherever you are
  EB_APP_NAME: "" # ElasticBeanstalk Application Name
  EB_APP_ENV: "" # ElasticBeanstalk Application Environment
  S3_BUCKET: "" # S3 bucket for ElasticBeanstalk
  S3_KEY: "" # S3 folder to upload built app

# Job: Build
# Installs npm packages, transpiles ES6 -> ES5
# Passes node_modules/, dist/ onto next steps using artifacts
build:
  stage: build
  script:
    - npm install
    - ./bin/build
  artifacts:
    paths:
      - node_modules/
      - dist/
  tags:
    - nodejs-app

# Job: Test
# Run tests against our application
# If this fails, we do not deploy
test:
  stage: test
  script:
    - ./bin/test
  tags:
    - nodejs-app

# Job: Deploy
# Zips the contents of our built application, uploads to S3
# Configures a new EB version and switches to that version
deploy:
  stage: deploy
  script:
    - ./bin/deploy
  tags:
    - nodejs-app

This file defines a simple 3-step pipeline that will build, test, and finally deploy our application. If any steps fail, the rest will not run.

Pipeline Explained

This might seem like magic, and it partially is. A lot of the hard-work is hidden within a few choice bin/ files we defined under script for each of our jobs. While each bin/ file is relatively simple, there is one that should be carefully explained.

Let’s take a look at bin/deploy

#!/bin/bash

# Install AWS CLI
curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
python get-pip.py
pip install awscli --ignore-installed six

# Install "zip"
apt-get update
apt-get install -y zip

# Zip up everything with the exception of node_modules (including dist)
ts=`date +%s`
fn="$EB_APP_NAME-$ts.zip"
find ./ -path '*/node_modules/*' -prune -o -path '*/\.git*' -prune -o -type f -print | zip $fn -@
S3_KEY="$S3_KEY/$fn"
# Copy the app to S3
aws s3 cp $fn "s3://$S3_BUCKET/$S3_KEY"

# Create a new version in eb
echo "Creating ElasticBeanstalk Application Version ..."
aws elasticbeanstalk create-application-version \
  --application-name $EB_APP_NAME \
  --version-label "$EB_APP_NAME-$ts" \
  --description "$EB_APP_NAME-$ts" \
  --source-bundle S3Bucket="$S3_BUCKET",S3Key="$S3_KEY" --auto-create-application

# Update to that version
echo "Updating ElasticBeanstalk Application Version ..."
aws elasticbeanstalk update-environment \
  --application-name $EB_APP_NAME \
  --environment-name $EB_APP_ENV \
  --version-label "$EB_APP_NAME-$ts"

echo "Done! Deployed version $EB_APP_NAME-$ts"

There’s a lot going on in this file, so let’s break it down line-by-line.

Breaking Down Deployment

Since our GitLab Runner uses Docker and we’re configuring our pipeline to execute on a node:5.9.1 instance, we’ll need to install pip (a Python package manager) and awscli (AWS Command-Line-Interface) so that we can use it later in our script.

# Install AWS CLI
curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
python get-pip.py
pip install awscli --ignore-installed six

Next, we also need to install zip.

# Install "zip"
apt-get update
apt-get install -y zip

This is where things get interesting. By the time we hit the deploy job, we’ve already built our application by transpiling src/ into dist/ and we now need to upload this built application to S3.

First, we set two variables ts and fn which correspond to an epoch timestamp and the filename we’re going to zip and upload, respectively. Then, we zip up everything with the exception of node_modules/ (since that will be built on the ElasticBeanstalk instances)

# Zip up everything with the exception of node_modules (including dist)
ts=`date +%s`
fn="$EB_APP_NAME-$ts.zip"
find ./ -path '*/node_modules/*' -prune -o -path '*/\.git*' -prune -o -type f -print | zip $fn -@
S3_KEY="$S3_KEY/$fn"

Now, we copy the newly zipped file up to S3.

# Copy the app to S3
aws s3 cp $fn "s3://$S3_BUCKET/$S3_KEY"

With the application in S3, we’re now ready to create a new ElasticBeanstalk version.

# Create a new version in eb
echo "Creating ElasticBeanstalk Application Version ..."
aws elasticbeanstalk create-application-version \
  --application-name $EB_APP_NAME \
  --version-label "$EB_APP_NAME-$ts" \
  --description "$EB_APP_NAME-$ts" \
  --source-bundle S3Bucket="$S3_BUCKET",S3Key="$S3_KEY" --auto-create-application

Finally, we deploy the application by telling ElasticBeanstalk to update the new application version.

# Update to that version
echo "Updating ElasticBeanstalk Application Version ..."
aws elasticbeanstalk update-environment \
  --application-name $EB_APP_NAME \
  --environment-name $EB_APP_ENV \
  --version-label "$EB_APP_NAME-$ts"

Done!

Push Application

With our .gitlab-ci.yml configured and our bin/deploy script in place, we’re ready to push to GitLab and watch our runner go.

You can find your runner by navigating to your project repository and clicking Builds on the sidebar.

If all goes well, you should see your new application in ElasticBeanstalk shortly!

Lessons Learned

There are a few points of pain I want to acknowledge before wrapping this up.

ElasticBeanstalk makes launching Node.js dead simple but the actions that are run during a deployment are not all that obvious. For example, a new application version will run npm install but not npm run postinstall. While this may not seem like a big deal for small projects that may not benefit from a CI, this hinders your ability to build and compile necessary files, such as for PostCSS and ES6.

Another point of pain was the .ebignore file. This file is used in place of .gitignore so that your deployments do not include any bloat. A major lesson was that .gitignore !== .ebignore and they should both exist and act independently. In this article, we zipped and deployed our application after we built a dist/ folder. This folder is included in our .gitignore as we do not want to track transpiled code; however, we definitely need this folder for our application to run so we leave it out of our .ebignore file.

What We Didn’t Cover

The gitlab-ci.yml file is extremely extendible and we barely scratched the surface of what’s possible. A few notes:

  • It’s probably not a great idea to store your AWS credentials in public repository, so don’t do that.
  • In the interest of better security, variables should probably live in a secure place on an internal server where they could be downloaded as opposed to including them as environment exports.
  • This pipeline will run on all branches, on all commits - In a real-world setting this should be configured to only run on desired branches such as master.
  • We build our application in Docker but develop locally, ideally we would be developing using the same Docker image but that’s a topic for another day.

Conclusion

Getting a continuous integration system in place for some of my side-projects has been incredibly helpful as it allows me to move quick and not think about consequences when pushing. At the end of the day, if tests fail, or lint doesn’t check out, it won’t ship and that gives me peace of mind.

If you found this article helpful, let me know!

Cheers, Tom

About

Tom is the CEO (Customer Experience Overlord) of Unicorn Heart Club where he leads development for Astral Virtual TableTop. He's a homebrewer, hiker, and has an about page.

Read More

← Machine Learning for the Phillips Hue

Comments