Greetings,
I've been trying to implement some changes to our config but it's not
working like I am expecting it to work. I'd really appreciate some
input on this to see what I am doing wrong here.
Oh and by the way, one thing I discovered last week is that if you are
behind a CDN like CloudFlare, they will sometimes pass v6 IPs in the
X-Forwarded-For header. If you're tracking this header using an ipv4
stick table, haproxy appears to convert this IP to 0.0.0.0 and funnels
all v6 requests into this one entry in the stick table. This was
matching some throtting ACLs and causing problems for me. What I did
was switch the stick table to ipv6, which appears to work fine for
both v4 and v6 IPs.
Basically what I've been wanting to do is increment gpc0 to 1 on
abusive IPs and sending them directly to the appropriate backend
depending on the type of abuse. I then want to stop tracking the IP
until it expires out of the stick table. In other words, if a user
hits a throttling threshold, I want them to be blocked outright for a
period of time. Example:
frontend http-in
acl kill sc0_inc_gpc0 gt 0
stick-table type ipv6 size 250k expire 1m store
http_err_rate(60s),http_req_rate(60s),conn_cur
tcp-request inspect-delay 10s
# Tracks the last IP in the X-Forwarded-For header if it's not in
either the whitelist or blacklist
# XXX the IP is still tracked and gpc0 increments even if "kill" is true!
tcp-request content track-sc0 req.hdr_ip(X-Forwarded-For) if
!whitelist_hdr !blacklist_hdr !kill
acl ease_up sc0_http_req_rate gt 1800
# There are other use_backend keywords above this line for
blacklisted IPs, etc.
# "kill" gets evaluated here as expected, incrementing gpc0 only
if the request rate is too high
use_backend ease-up if ease_up kill
default_backend servers
Using this config, haproxy increments the "kill" acl when the request
rate exceeds 1800 as expected. What I'm confused about is why gpc0
continues to increment and the expire time gets reset on subsequent
requests, even when "kill" evaluates to true.
It's as if haproxy tracks the IP before it evaluates the "kill" ACL,
so this ACL can NOT be used to stop tracking the IP.
I can kind of understand if this is the case (please correct me if I'm
wrong). I wondered if maybe incrementing gpc0 caused the IP to be
tracked and the expire timer to be reset, so the other thing I tried
was doing:
acl kill sc0_inc_gpc0 gt 0
acl really_kill sc0_get_gpc0 gt 0
# IP still continues to be tracked, even if "really_kill" evaluates to true!
tcp-request content track-sc0 req.hdr_ip(X-Forwarded-For) if
!whitelist_hdr !blacklist_hdr !really_kill
acl ease_up sc0_http_req_rate gt 1800
use_backend ease-up if ease_up really_kill
# Testing to see if evaluating "kill" causes the IP to be tracked
use_backend ease-up if ease_up kill
# other use_backend statements below this
The only change here from the above config is that gpc0 increments to
1 and that's it. The request is directed to the ease-up backend before
below use_backend statements evaluate "kill", which is what I'd
expect. What's confusing me is that, again, the IP continues to be
tracked even if "really_kill" is true.
So what appears to be the case is that certain ACLs can't be used to
disable tracking an IP. The blacklist/whitelist ACLs work just fine,
but anything incrementing or even accessing gpc0 does not work,
because it appears that the tracking happens before the evaluation of
the ACL. Please correct me if I'm wrong here.
I'm thinking the only way to do this is to somehow use sc0_clr_gpc0 in
a backend after the request rate (or whatever abuse parameter) drops
below a threshold. It doesn't appear that I will be able to just allow
the IP to expire out of the stick table using ACLs. If anyone has any
input on how to better structure my config please let me know!
Here is my haproxy info:
Running on Debian 7.8.
HA-Proxy version 1.5.12 2015/05/02
Copyright 2000-2015 Willy Tarreau <w@1wt.eu>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_STATIC_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.30 2012-02-04
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
--
Brendon Colby
Senior DevOps Engineer
Newgrounds.com
I've been trying to implement some changes to our config but it's not
working like I am expecting it to work. I'd really appreciate some
input on this to see what I am doing wrong here.
Oh and by the way, one thing I discovered last week is that if you are
behind a CDN like CloudFlare, they will sometimes pass v6 IPs in the
X-Forwarded-For header. If you're tracking this header using an ipv4
stick table, haproxy appears to convert this IP to 0.0.0.0 and funnels
all v6 requests into this one entry in the stick table. This was
matching some throtting ACLs and causing problems for me. What I did
was switch the stick table to ipv6, which appears to work fine for
both v4 and v6 IPs.
Basically what I've been wanting to do is increment gpc0 to 1 on
abusive IPs and sending them directly to the appropriate backend
depending on the type of abuse. I then want to stop tracking the IP
until it expires out of the stick table. In other words, if a user
hits a throttling threshold, I want them to be blocked outright for a
period of time. Example:
frontend http-in
acl kill sc0_inc_gpc0 gt 0
stick-table type ipv6 size 250k expire 1m store
http_err_rate(60s),http_req_rate(60s),conn_cur
tcp-request inspect-delay 10s
# Tracks the last IP in the X-Forwarded-For header if it's not in
either the whitelist or blacklist
# XXX the IP is still tracked and gpc0 increments even if "kill" is true!
tcp-request content track-sc0 req.hdr_ip(X-Forwarded-For) if
!whitelist_hdr !blacklist_hdr !kill
acl ease_up sc0_http_req_rate gt 1800
# There are other use_backend keywords above this line for
blacklisted IPs, etc.
# "kill" gets evaluated here as expected, incrementing gpc0 only
if the request rate is too high
use_backend ease-up if ease_up kill
default_backend servers
Using this config, haproxy increments the "kill" acl when the request
rate exceeds 1800 as expected. What I'm confused about is why gpc0
continues to increment and the expire time gets reset on subsequent
requests, even when "kill" evaluates to true.
It's as if haproxy tracks the IP before it evaluates the "kill" ACL,
so this ACL can NOT be used to stop tracking the IP.
I can kind of understand if this is the case (please correct me if I'm
wrong). I wondered if maybe incrementing gpc0 caused the IP to be
tracked and the expire timer to be reset, so the other thing I tried
was doing:
acl kill sc0_inc_gpc0 gt 0
acl really_kill sc0_get_gpc0 gt 0
# IP still continues to be tracked, even if "really_kill" evaluates to true!
tcp-request content track-sc0 req.hdr_ip(X-Forwarded-For) if
!whitelist_hdr !blacklist_hdr !really_kill
acl ease_up sc0_http_req_rate gt 1800
use_backend ease-up if ease_up really_kill
# Testing to see if evaluating "kill" causes the IP to be tracked
use_backend ease-up if ease_up kill
# other use_backend statements below this
The only change here from the above config is that gpc0 increments to
1 and that's it. The request is directed to the ease-up backend before
below use_backend statements evaluate "kill", which is what I'd
expect. What's confusing me is that, again, the IP continues to be
tracked even if "really_kill" is true.
So what appears to be the case is that certain ACLs can't be used to
disable tracking an IP. The blacklist/whitelist ACLs work just fine,
but anything incrementing or even accessing gpc0 does not work,
because it appears that the tracking happens before the evaluation of
the ACL. Please correct me if I'm wrong here.
I'm thinking the only way to do this is to somehow use sc0_clr_gpc0 in
a backend after the request rate (or whatever abuse parameter) drops
below a threshold. It doesn't appear that I will be able to just allow
the IP to expire out of the stick table using ACLs. If anyone has any
input on how to better structure my config please let me know!
Here is my haproxy info:
Running on Debian 7.8.
HA-Proxy version 1.5.12 2015/05/02
Copyright 2000-2015 Willy Tarreau <w@1wt.eu>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_STATIC_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.30 2012-02-04
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
--
Brendon Colby
Senior DevOps Engineer
Newgrounds.com