Smart throttling and blocking of abusive clients using HAProxy
HAProxy‘s stick tables are not a new feature. In fact, they were introduced way back in 2010, and ever since they were handy for tracking the number of open connections, HTTP request rate, bytes in/out rate, etc. In most cases, we use stick tables for throttling abusive HTTP clients, but sometimes we receive a request from a customer that requires a bit of creative approach with stick-table usage.
The problem
One of our customers experienced issues with their API service. Certain API endpoints were being abused by both legitimate and malicious users in terms of high request rate. This had a negative impact on API performance and response time.
The customer wanted to throttle legitimate clients (known by their IP range) and temporarily block (1 hour) all other clients that exceed a reasonable HTTP request rate on specific URL routes.
The idea
Throttling abusive clients using HAProxy is easy thanks to the stick tables feature. We decided to identify unique clients by the hashed value of multiple concatenated parameters (client’s IP address, Host, and User-Agent HTTP request headers). The reason behind this decision was to identify unique clients that are accessing the API from the same public IP address.
As mentioned, certain legitimate clients, identified by their IP address, should be excluded from temporary deny logic, but they should receive “429 too many requests” HTTP response in case they exceed the defined HTTP request rate threshold.
On the other hand, IP addresses of all other clients that exceed the HTTP request rate threshold will be blocked for 1 hour.
The solution with HAProxy
To implement both tracking and a temporary deny feature, we had to use two stick tables. HAProxy allows only one stick table per frontend and backend. Thankfully, stick tables can be referenced across frontends and backends, so that limitation isn’t really an issue.
First, we created the stick table that’s used as a temporary deny list of abusive clients. The stick table will store only IPv4 addresses, and up to 100 000 entries with a lifetime of 1 hour (since the item was last created).
backend bk_stick_blocked
stick-table type ip size 100k expire 1h store gpc0
The other stick table is used for tracking the HTTP request rate and the number of concurrently open connections. The request rate is tracked within a 10-second sliding window, and the table accepts up to 100.000 entries, which are stored in binary format.
stick-table type binary len 64 size 100k store gpc0_rate(10s),conn_cur expire 4m
The reason for storing the entries in binary format is because we’re not tracking clients by their IP address, but by the custom tracking header (X-Concat) which contains base64 encoded hash of multiple values: client’s IP address, Host and, User-Agent HTTP request headers.
# identify unique clients based on temporary header http-request set-header X-SB-Track %[req.fhdr(Host)]_%[req.fhdr(X-Forwarded-For)]_%[req.fhdr(User-Agent)] # base64 encode hashed value in a tracking header http-request set-header X-Concat %[req.fhdr(X-SB-Track),base64] # remove temporary header http-request del-header X-SB-Track
With stick tables and custom tracking header now in place, it’s time to define a bunch of important ACLs that define which URLs are considered for request rate tracking, clients that have exceeded request rate threshold, number of concurrently open connections, trusted clients, etc.
# track HTTP request rate only on these URLs acl throttled_url path_beg -i /login/ acl throttled_url path_beg -i /register/ # IPs excluded from temporary deny feature acl throttle_exclude req.hdr_ip(X-Forwarded-For) -f /etc/haproxy/lists/throttle_exclude.lst # clients that were "seen" by HAProxy acl mark_seen sc0_inc_gpc0 gt 0 # clients that have exceeded HTTP request rate threshold acl fast_refresher sc0_gpc0_rate gt 10 # clients that have more than 20 concurrently open connections acl conn_limit sc0_conn_cur gt 20 # clients that have their IP listed in the temporary deny list acl ip_was_bad sc1_get_gpc0(bk_stick_blocked) gt 0 # ip_is_bad increments gpc0 counter every time it's evaluated acl ip_is_bad sc1_inc_gpc0(bk_stick_blocked) gt 0
And with the ACLs all sorted out, we can define which requests we want to track and where we want the entries to be stored. First, we’ll track only the requests for the URLs we considered for throttling. Tracking is done based on the X-Concat header.
# Track X-Concat header each time throttled_url is requested http-request track-sc0 hdr(X-Concat) if throttled_url
We then track requests for the throttled URLs, but we’ll store the client’s IP address (defined in the X-Forwarded-For request header) in a separate stick table (bk_stick_blocked). At this point, even though the client’s IP address is stored in the temporary deny list, it’s not actually denied.
The client’s IP will be denied only when the sticky counter (sc1) is greater than zero. And we increment the sticky counter only when the client exceeded the HTTP request rate.
# Track all requests for the throttled_url in a separate stick-table (bk_stick_blocked) http-request track-sc1 hdr_ip(X-Forwarded-For) table bk_stick_blocked if throttled_url # Increment the counter and therefore block the IP that was detected as a fast_refresher # IP is stored in stick-table bk_stick_blocked http-request track-sc1 hdr_ip(X-Forwarded-For) table bk_stick_blocked if fast_refresher ip_is_bad !throttle_exclude
Putting everything together
Let’s recap the whole story. If a client starts to request URLs that begin with a defined prefix, HAProxy will automatically track those requests.
If the client exceeds the threshold of 10 requests within a 10-second sliding window (an average of 1 req/s), HAProxy will increment the sticky counter and deny additional requests to those URLs for the next 1 hour.
If the client exceeds the request rate threshold, but its IP is listed on the throttle_exclude list, HAProxy will return “429 too many requests” response until the request rate falls to the acceptable level.
In addition, any client (trusted or not) will receive a 429 response code if the number of concurrently opened connections exceeds the threshold.
When put together, the HAProxy frontend and backend configuration looks like this:
backend bk_stick_blocked
# stick table for temporarily blocking fast refresher IPs (1h ban)
stick-table type ip size 100k expire 1h store gpc0
backend bk_429
errorfile 429 /etc/haproxy/errorfiles/429.http
http-request deny deny_status 429
frontend ft_api
bind *:80
bind *:443 ssl crt /etc/haproxy/certs.d/ no-sslv3 no-sslv3 no-tls-tickets no-tlsv10 no-tlsv11 alpn h2,http/1.1
# Track HTTP request rate only on these URLs
acl throttled_url path_beg -i /login/
acl throttled_url path_beg -i /register/
# IPs excluded from temporary deny feature
acl throttle_exclude req.hdr_ip(X-Forwarded-For) -f /etc/haproxy/lists/throttle_exclude.lst
# Identify unique clients based on temporary header
http-request set-header X-SB-Track %[req.fhdr(Host)]_%[req.fhdr(X-Forwarded-For)]_%[req.fhdr(User-Agent)]
# base64 encode temporary tracking header
http-request set-header X-Concat %[req.fhdr(X-SB-Track),base64]
# Remove temporary tracking header
http-request del-header X-SB-Track
# stick-table for tracking HTTP request rate and the number of concurrently open connections
# We track request rate within 10-second sliding window
stick-table type binary len 64 size 100k store gpc0_rate(10s),conn_cur expire 4m
# clients that were "seen" by HAProxy
acl mark_seen sc0_inc_gpc0 gt 0
# clients that have exceeded HTTP request rate threshold
acl fast_refresher sc0_gpc0_rate gt 10
# clients that have more than 20 concurrently open connections
acl conn_limit sc0_conn_cur gt 20
# ip_is_bad increments gpc0 counter every time it's evaluated
acl ip_is_bad sc1_inc_gpc0(bk_stick_blocked) gt 0
# Track X-Concat header each time throttled_url is requested
http-request track-sc0 hdr(X-Concat) if throttled_url
# Track all requests for the throttled_url in a separate stick-table (bk_stick_blocked)
http-request track-sc1 hdr_ip(X-Forwarded-For) table bk_stick_blocked if throttled_url
# Increment the counter and therefore block the IP that was detected as a fast_refresher
# IP is stored in stick-table bk_stick_blocked
http-request track-sc1 hdr_ip(X-Forwarded-For) table bk_stick_blocked if fast_refresher ip_is_bad !throttle_exclude
# Check if the client's IP is blocked
acl ip_was_bad sc1_get_gpc0(bk_stick_blocked) gt 0
# Deny access to blocked IP
http-request deny if ip_was_bad !throttle_exclude
# if the client has too many open connections, return 429 error
use_backend bk_429 if mark_seen conn_limit
# if the trusted client exceeded HTTP request rate, return 429 error
use_backend bk_429 if mark_seen fast_refresher
If you already use HAProxy and have a favourite feature of your own, let us know about it in the comments below! If you don’t, check out some of our managed services and we can help set it up for you.
