HAProxy makes backend unresponsive when handling multiple thousand connections per second (9 replies)

Hi everyone,

we got a problem recently which we can't explain to ourself. We got a
java application (Tomcat WAR-File) which has to handle several million
of requests per day and several thousand requests per second during peak
times. Due to this high amount we are splitting traffic using an ACL in
"booking traffic" and "availability traffic". Booking traffic is
negligible but the Availability traffic is load-balanced over several
application servers. The problem that occurs is that our external
partner "floods" the Availability-Frontend with several thousand
requests per second and the backend becomes unresponsive. If we redirect
them directly to our Tomcat-Instance via Firewall-Rules without passing
through HAProxy everything is fine. The Tomcat instances have
"/maxThreads=1024/" and "/acceptCount=500/" as their main connector
settings so this shouldn't interfere with the HAProxy configuration.

Our HAProxy configuration running on Solaris 11 64-bit:

HA-Proxy version 1.7.5 2017/04/03
Copyright 2000-2017 Willy Tarreau <willy@haproxy.org>

Build options :
TARGET = solaris
CPU = generic
CC = gcc
CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing
-Wdeclaration-after-statement -fomit-frame-pointer -DFD_SETSIZE=65536
-D_REENTRANT
OPTIONS = USE_TPROXY=1 USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8-T4mods
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Running on OpenSSL version : OpenSSL 1.0.2k 26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k 26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built without Lua support

Available polling systems :
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 2 (2 usable), will use poll.

Available filters :
[SPOE] spoe
[TRACE] trace
[COMP] compression
---
global
log 127.0.0.1:514 local0 debug
daemon
maxconn 50000
stats socket /opt/etc/haproxy/haproxy.sock mode 600 level admin
stats timeout 2m #Wait up to 2 minutes for input
tune.ssl.default-dh-param 2048
ulimit-n 20000


ssl-default-server-options no-sslv3 no-tls-tickets
ssl-default-bind-ciphers
EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:AES128+EECDH:AES128+EDH

defaults
log global
option tcplog
log-format %f\ %b/%s\ client_ip:%ci\ client_port:%cp\
SSL_version:%sslv\ SSL_cypher:%sslc\ %ts\ Tt:%Tt\ Tq:%Tq\ Tw:%Tw\
Tc:%Tc\ Tr:%Tr
mode http
timeout connect 5000
timeout check 5000
timeout client 30000
timeout server 30000
retries 3

frontend ndc
http-response set-header Strict-Transport-Security
max-age=31536000;\ includeSubdomains;\ preload
http-response set-header X-Content-Type-Options nosniff

bind *:443 ssl crt /opt/etc/haproxy/domain_com.pem force-tlsv12
no-sslv3
maxconn 20000

acl fare_availability path_beg /ndc/fare/v1/availability
acl flight_availability path_beg /ndc/flight/v1/availability
use_backend vakanz-backend if flight_availability or fare_availability
default_backend booking-backend

backend booking-backend
server 10.2.8.28 10.2.8.23:8443 check ssl verify none minconn 500
maxconn 500

backend vakanz-backend
server 10.2.8.28 10.2.8.28:8443 check ssl verify none minconn 500
maxconn 500
server 10.2.8.40 10.2.8.40:8443 check ssl verify none minconn 500
maxconn 500
server 10.2.8.41 10.2.8.41:8443 check ssl verify none minconn 500
maxconn 500

Hopefully somebody can shed some light if we got a bad configuration and
how we could troubleshoot this issue.

Thanks and regards,
Daniel