Deploying feature branches to have a review app

BigBinary has been working with Gumroad for a while. Following blog post has been posted with permission from Gumroad and we are very grateful to Sahil for allowing us to discuss the work in such an open environment.

Staging environment helps us in testing the code before pushing the code to production. However it becomes hard to manage the staging environment when more people work on different parts of the application. This can be solved by implementing a system where feature branch can have its own individual staging environment.

Heroku has Review Apps feature which can deploy different branches separately. Gumroad, doesn't use Heroku so we built a custom in-house solution.

The first step was to build the infrastructure. We created a new Auto Scaling Group, Application Load Balancer and route in AWS for the review apps. Load balancer and route are common for all review apps, but a new EC2 instance is created in the ASG when a new review app is commissioned.


The main challenge was to forward the incoming requests to the correct server running the review app. This was made possible using Lua in nginx and consul. When a review app is deployed, it writes its IP and port to consul along with the hostname. Each review app server runs an instance of OpenResty (Nginx + Lua modules) with the following configuration.

1server {
2  listen                   80;
3  server_name              _;
4  server_name_in_redirect  off;
5  port_in_redirect         off;
6
7  try_files $uri/index.html $uri $uri.html @app;
8
9  location @app {
10    set $upstream "";
11    rewrite_by_lua '
12      http   = require "socket.http"
13      json   = require "json"
14      base64 = require "base64"
15
16      -- read upstream from consul
17      host          = ngx.var.http_host
18      body, c, l, h = http.request("http://172.17.0.1:8500/v1/kv/" .. host)
19      data          = json.decode(body)
20      upstream      = base64.decode(data[1].Value)
21
22      ngx.var.upstream = upstream
23    ';
24
25    proxy_buffering   off;
26    proxy_set_header  Host $host;
27    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
28    proxy_redirect    off;
29    proxy_pass        http://$upstream;
30  }
31}
Copy

It forwards all incoming requests to the correct IP:PORT after looking up in consul with the hostname.

The next task was to build a system to deploy the review apps to this infrastructure. We were already using docker in both production and staging environments. We decided to extend it to deploy branches by building docker image for every branch with deploy- prefix in the branch name. When such a branch is pushed to GitHub, a CircleCI job is run to build a docker image with the code and all the necessary packages. This can be configured using a configuration template like this.

1jobs:
2  build_image:
3    <<: *defaults
4    parallelism: 2
5    steps:
6      - checkout
7      - setup_remote_docker:
8          version: 17.09.0-ce
9      - run:
10          command: |
11            ci_scripts/2.0/build_docker_image.sh
12          no_output_timeout: 20m
13
14workflows:
15  version: 2
16
17  web_app:
18    jobs:
19      - build_image:
20          filters:
21            branches:
22              only:
23                - /deploy-.*/
Copy

It also pushes static assets like JavaScript, CSS and images to an S3 bucket from where they are served directly through CDN. After building the docker image, another CircleCI job is run to do the following tasks.

• Create a new database in RDS and configure the required credentials.
• Scale up Review App's Auto Scaling Group by increasing the number of desired instances by 1.
• Run redis, database migration, seed-data population, unicorn and resque instances using nomad.

The ease of deploying a review app helped increase our productivity.