How to mitigate DDoS using Rack::Attack

Recently, we faced a DDoS attack in one of the clients' projects. There were many requests from different IPs to root and login paths, and we were running thrice the usual number of servers to keep the system alive.

We are using Cloudflare's HTTP proxy and it was doing a great job preventing malicious requests, but we wanted to check if we can avoid the loading/captcha pages which Cloudflare uses to filter requests. We came to a conclusion that we would be able to mitigate the ongoing attack if we could throttle requests by IP.

Cloudflare has an inbuilt Rate Limiting feature to throttle requests, but it would be a little expensive in our case since Cloudflare charges by the number of good requests and it was a high traffic website. On further analysis, we found that throttling at application level would be enough in that situation and the gem Rack::Attack helped us with that.

Rack::Attack is a Rack middleware from Kickstarter. It can be configured to throttle requests based on IP or any other parameter.

To use Rack::Attack, include the gem in Gemfile.

1gem "rack-attack"
Copy

After bundle install, configure the middleware in config/application.rb:

1config.middleware.use Rack::Attack
Copy

Now we can create the initializer config/initializers/rack_attack.rb to configure Rack::Attack.

By default, Rack::Attack uses Rails.cache to store requests information. In our case, we wanted a separate cache for Rack::Attack and it was configured as follows.

1redis_client = Redis.connect(url: ENV["REDIS_URL"])
2Rack::Attack.cache.store = Rack::Attack::StoreProxy::RedisStoreProxy.new(redis_client)
Copy

If the web server is behind a proxy like Cloudflare, we have to configure a method to fetch the correct remote_ip address. Otherwise, it would block based on proxy's IP address and would result in blocking a lot of legit requests.

1class Rack::Attack
2  class Request < ::Rack::Request
3    def remote_ip
4      # Cloudflare stores remote IP in CF_CONNECTING_IP header
5      @remote_ip ||= (env['HTTP_CF_CONNECTING_IP'] ||
6                      env['action_dispatch.remote_ip'] ||
7                      ip).to_s
8    end
9  end
10end
Copy

Requests can be throttled based on IP address or any other parameter. In the following example, we are setting a limit of 40rpm/IP for "/" path.

1class Rack::Attack
2  throttle("req/ip", :limit => 40, :period => 1.minute) do |req|
3    req.remote_ip if req.path == "/"
4  end
5end
Copy

The downside of this configuration is that after the 1 minute period, the attacker can launch another 40 requests/IP simultaneously and it would exert pressure on the servers. This can be solved using exponential backoff.

1class Rack::Attack
2  # Exponential backoff for all requests to "/" path
3  #
4  # Allows 240 requests/IP in ~8 minutes
5  #        480 requests/IP in ~1 hour
6  #        960 requests/IP in ~8 hours (~2,880 requests/day)
7  (3..5).each do |level|
8    throttle("req/ip/#{level}",
9               :limit => (30 * (2 ** level)),
10               :period => (0.9 * (8 ** level)).to_i.seconds) do |req|
11      req.remote_ip if req.path == "/"
12    end
13  end
14end
Copy

If we want to turn off throttling for some IPs (Eg.: Health check services), then those IPs can be safelisted.

1class Rack::Attack
2  class Request < ::Rack::Request
3    def allowed_ip?
4      allowed_ips = ["127.0.0.1", "::1"]
5      allowed_ips.include?(remote_ip)
6    end
7  end
8
9  safelist('allow from localhost') do |req|
10    req.allowed_ip?
11  end
12end
Copy

We can log blocked requests separately and this is helpful for analyzing the attack.

1ActiveSupport::Notifications.subscribe('rack.attack') do |name, start, finish, request_id, req|
2  if req.env["rack.attack.match_type"] == :throttle
3    request_headers = { "CF-RAY" => req.env["HTTP_CF_RAY"],
4                        "X-Amzn-Trace-Id" => req.env["HTTP_X_AMZN_TRACE_ID"] }
5
6    Rails.logger.info "[Rack::Attack][Blocked]" <<
7                      "remote_ip: \"#{req.remote_ip}\"," <<
8                      "path: \"#{req.path}\", " <<
9                      "headers: #{request_headers.inspect}"
10  end
11end
Copy

A sample initializer with these configurations can be downloaded from here.

Application will now throttle requests and will respond with HTTP 429 Too Many Requests response for the throttled requests.

We now block a lot of malicious requests using Rack::Attack. Here's a graph with % of blocked requests over a week.

EDIT: Updated the post to add more context to the situation.