Every time I'm faced with the decision to choose an infrastructure for a side-project I always lean towards Elastic Beanstalk. For starters, the system is simple to setup, it's relatively painless to deploy an application, and I have the peace of mind knowing that if I succumb to the "reddit hug of death" that my app can scale up automatically as necessary.
With all the pros of Elastic Beanstalk though, I fear there's a few places where often I go wrong when it comes to deploying an application. I hope this article serves as a guide as well as a "lessons learned" for those who wish to setup their project on Elastic Beanstalk and deploy using an automated Continuous Integration system.
This article is geared heavily for Node.js projects; however, the principles can be applied to any language using any CI with hopefully very small modifications.
Before we start our adventure, I assume you have the following in place:
There are enough guides out there on how to spin up a new ElasticBeanstalk app so I don't want to dive too much into the specifics. You can follow along with the article by simply creating a new application and deploying the Node.js sample app. Be sure to note your application's name and environment name.
Before we can start writing code, we obviously need a repository in GitLab setup for our project.
One of the coolest parts about running your own private GitLab instance is you have absolute control over your integrations. I simply host a $20/mo instance on Digital Ocean and it's been fantastic, especially with automated daily backups.
Plugs aside, having root access to GitLab is important as we're going to need the right permissions to install
gitlab-ci-multi-runner which is extremely easy to do.
# ssh into your gitlab server ssh firstname.lastname@example.org # GitLab Runner is best used with Docker curl -sSL https://get.docker.com/ | sh # Add GitLab Runner packages (debian) curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-ci-multi-runner/script.deb.sh | sudo bash # Install GitLab Runner apt-get install gitlab-ci-multi-runner
You can find more in-depth documentation about how to Install GitLab Runner for your server.
Now that the runner is installed, we need to register a new runner with our GitLab instance.
Now simply follow the instructions.
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/ci ) # GitLab CI URL noted in step 4 Please enter the gitlab-ci token for this runner # Project token noted in step 4 Please enter the gitlab-ci description for this runner Builds, tests, and deploys nodejs-app to ElasticBeanstalk Please enter the executor: shell, docker, docker-ssh, ssh? docker Please enter the Docker image (eg. ruby:2.1): node:5.9.1
So, what did we just do? We registered a new runner with GitLab, we gave it a description of
Builds, tests, and deploys nodejs-app to ElasticBeanstalk and we assigned it to a specific project given by the gitlab-ci token.
Now we're ready to start writing our GitLab Runner configuration.
If you're starting with a blank repository, feel free to clone a boilerplate ES6 repository so we can dive right into the interesting bits.
$ git clone https://github.com/tlackemann/nodejs-es6-ci.git
Open or create
.gitlab-ci.yml in your project root and change the values under
variables to your actual AWS credentials. This file should be fairly well documented as to what our pipeline is going to look like.
# GitLab CI Docker Image image: node:5.9.1 # Build - Build necessary JS files # Test - Run tests # Deploy - Deploy application to S3/ElasticBeanstalk stages: - build - test - deploy # Configuration variables: AWS_ACCESS_KEY_ID: "" # Should have access to both S3/EB AWS_SECRET_ACCESS_KEY: "" AWS_DEFAULT_REGION: "us-east-1" # Or, wherever you are EB_APP_NAME: "" # ElasticBeanstalk Application Name EB_APP_ENV: "" # ElasticBeanstalk Application Environment S3_BUCKET: "" # S3 bucket for ElasticBeanstalk S3_KEY: "" # S3 folder to upload built app # Job: Build # Installs npm packages, transpiles ES6 -> ES5 # Passes node_modules/, dist/ onto next steps using artifacts build: stage: build script: - npm install - ./bin/build artifacts: paths: - node_modules/ - dist/ tags: - nodejs-app # Job: Test # Run tests against our application # If this fails, we do not deploy test: stage: test script: - ./bin/test tags: - nodejs-app # Job: Deploy # Zips the contents of our built application, uploads to S3 # Configures a new EB version and switches to that version deploy: stage: deploy script: - ./bin/deploy tags: - nodejs-app
This file defines a simple 3-step pipeline that will build, test, and finally deploy our application. If any steps fail, the rest will not run.
This might seem like magic, and it partially is. A lot of the hard-work is hidden within a few choice
bin/ files we defined under
script for each of our jobs. While each
bin/ file is relatively simple, there is one that should be carefully explained.
Let's take a look at
# Install AWS CLI curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py" python get-pip.py pip install awscli --ignore-installed six # Install "zip" apt-get update apt-get install -y zip # Zip up everything with the exception of node_modules (including dist) ts=`date +%s` fn="$EB_APP_NAME-$ts.zip" find ./ -path '*/node_modules/*' -prune -o -path '*/\.git*' -prune -o -type f -print | zip $fn -@ S3_KEY="$S3_KEY/$fn" # Copy the app to S3 aws s3 cp $fn "s3://$S3_BUCKET/$S3_KEY" # Create a new version in eb echo "Creating ElasticBeanstalk Application Version ..." aws elasticbeanstalk create-application-version \ --application-name $EB_APP_NAME \ --version-label "$EB_APP_NAME-$ts" \ --description "$EB_APP_NAME-$ts" \ --source-bundle S3Bucket="$S3_BUCKET",S3Key="$S3_KEY" --auto-create-application # Update to that version echo "Updating ElasticBeanstalk Application Version ..." aws elasticbeanstalk update-environment \ --application-name $EB_APP_NAME \ --environment-name $EB_APP_ENV \ --version-label "$EB_APP_NAME-$ts" echo "Done! Deployed version $EB_APP_NAME-$ts"
There's a lot going on in this file, so let's break it down line-by-line.
Since our GitLab Runner uses Docker and we're configuring our pipeline to execute on a
node:5.9.1 instance, we'll need to install
pip (a Python package manager) and
awscli (AWS Command-Line-Interface) so that we can use it later in our script.
# Install AWS CLI curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py" python get-pip.py pip install awscli --ignore-installed six
Next, we also need to install
# Install "zip" apt-get update apt-get install -y zip
This is where things get interesting. By the time we hit the
deploy job, we've already built our application by transpiling
dist/ and we now need to upload this built application to S3.
First, we set two variables
fn which correspond to an epoch timestamp and the filename we're going to zip and upload, respectively. Then, we zip up everything with the exception of
node_modules/ (since that will be built on the ElasticBeanstalk instances)
# Zip up everything with the exception of node_modules (including dist) ts=`date +%s` fn="$EB_APP_NAME-$ts.zip" find ./ -path '*/node_modules/*' -prune -o -path '*/\.git*' -prune -o -type f -print | zip $fn -@ S3_KEY="$S3_KEY/$fn"
Now, we copy the newly zipped file up to S3.
# Copy the app to S3 aws s3 cp $fn "s3://$S3_BUCKET/$S3_KEY"
With the application in S3, we're now ready to create a new ElasticBeanstalk version.
# Create a new version in eb echo "Creating ElasticBeanstalk Application Version ..." aws elasticbeanstalk create-application-version \ --application-name $EB_APP_NAME \ --version-label "$EB_APP_NAME-$ts" \ --description "$EB_APP_NAME-$ts" \ --source-bundle S3Bucket="$S3_BUCKET",S3Key="$S3_KEY" --auto-create-application
Finally, we deploy the application by telling ElasticBeanstalk to update the new application version.
# Update to that version echo "Updating ElasticBeanstalk Application Version ..." aws elasticbeanstalk update-environment \ --application-name $EB_APP_NAME \ --environment-name $EB_APP_ENV \ --version-label "$EB_APP_NAME-$ts"
.gitlab-ci.yml configured and our
bin/deploy script in place, we're ready to push to GitLab and watch our runner go.
You can find your runner by navigating to your project repository and clicking Builds on the sidebar.
If all goes well, you should see your new application in ElasticBeanstalk shortly!
There are a few points of pain I want to acknowledge before wrapping this up.
ElasticBeanstalk makes launching Node.js dead simple but the actions that are run during a deployment are not all that obvious. For example, a new application version will run
npm install but not
npm run postinstall. While this may not seem like a big deal for small projects that may not benefit from a CI, this hinders your ability to build and compile necessary files, such as for PostCSS and ES6.
Another point of pain was the
.ebignore file. This file is used in place of
.gitignore so that your deployments do not include any bloat. A major lesson was that
.gitignore !== .ebignore and they should both exist and act independently. In this article, we zipped and deployed our application after we built a
dist/ folder. This folder is included in our
.gitignore as we do not want to track transpiled code; however, we definitely need this folder for our application to run so we leave it out of our
gitlab-ci.yml file is extremely extendible and we barely scratched the surface of what's possible. A few notes:
variablesshould probably live in a secure place on an internal server where they could be downloaded as opposed to including them as environment exports.
Getting a continuous integration system in place for some of my side-projects has been incredibly helpful as it allows me to move quick and not think about consequences when pushing. At the end of the day, if tests fail, or lint doesn't check out, it won't ship and that gives me peace of mind.
If you found this article helpful, let me know!