Hello,
it seems I've stumbled upon a regression in haproxy 1.5.16.
Short story: After upgrading 1.5.15 to 1.5.16 CPU usage increased
significantly (3-4 times)
Long story:
I have several haproxy servers running happily with 1.5.15 for quite
some time. Recently I've started upgrading them to 1.5.16.
Unfortunately, as soon as I start 1.5.16 the CPU usage increases 3-4
times. Some haproxy processes eat 100% CPU (I run with nbproc=4).
Apart from that... everything works just fine and I've not noticed any
problems.
I also have a standby server. I've tried simulating some traffic but I
was unable to reproduce the problem there. As soon as I direct
production traffic to the standby server it shows the same behavior
(increased CPU as compared to 1.5.15).
System is running on Debian Jessie with kernel 4.3.3 from backports
and hand compiled OpenSSL 1.0.2g. Haproxy is acting mostly as a HTTP
load balancer with SSL. There are also some plain TCP backend, but
with negligible traffic.
"Bad" haproxy:
HA-Proxy version 1.5.16 2016/03/14
Copyright 2000-2016 Willy Tarreau <willy@haproxy.org>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -g -O2 -fstack-protector-strong -Wformat
-Werror=format-security -D_FORTIFY_SOURCE=2
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_TFO=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016
Running on OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.35 2014-04-04
PCRE library supports JIT : yes
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
"Good" haproxy:
HA-Proxy version 1.5.15 2015/11/01
Copyright 2000-2015 Willy Tarreau <willy@haproxy.org>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -g -O2 -fstack-protector-strong -Wformat
-Werror=format-security -D_FORTIFY_SOURCE=2
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_TFO=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016
Running on OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.35 2014-04-04
PCRE library supports JIT : yes
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
I have also Certificate Transparency patch applied, backported from 1.6.
I've also captured 10000 strace line from each haproxy (I had to use
-s0 to hide any user data though) if it helps in any way. The only
suspicious thing I see with 1.5.16 are large (>50) number of
consecutive calls to epoll_wait. Is this normal?
I'm out of ideas how to debug this problem, so any suggestions are welcome.
--
Janusz Dziemidowicz
it seems I've stumbled upon a regression in haproxy 1.5.16.
Short story: After upgrading 1.5.15 to 1.5.16 CPU usage increased
significantly (3-4 times)
Long story:
I have several haproxy servers running happily with 1.5.15 for quite
some time. Recently I've started upgrading them to 1.5.16.
Unfortunately, as soon as I start 1.5.16 the CPU usage increases 3-4
times. Some haproxy processes eat 100% CPU (I run with nbproc=4).
Apart from that... everything works just fine and I've not noticed any
problems.
I also have a standby server. I've tried simulating some traffic but I
was unable to reproduce the problem there. As soon as I direct
production traffic to the standby server it shows the same behavior
(increased CPU as compared to 1.5.15).
System is running on Debian Jessie with kernel 4.3.3 from backports
and hand compiled OpenSSL 1.0.2g. Haproxy is acting mostly as a HTTP
load balancer with SSL. There are also some plain TCP backend, but
with negligible traffic.
"Bad" haproxy:
HA-Proxy version 1.5.16 2016/03/14
Copyright 2000-2016 Willy Tarreau <willy@haproxy.org>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -g -O2 -fstack-protector-strong -Wformat
-Werror=format-security -D_FORTIFY_SOURCE=2
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_TFO=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016
Running on OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.35 2014-04-04
PCRE library supports JIT : yes
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
"Good" haproxy:
HA-Proxy version 1.5.15 2015/11/01
Copyright 2000-2015 Willy Tarreau <willy@haproxy.org>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -g -O2 -fstack-protector-strong -Wformat
-Werror=format-security -D_FORTIFY_SOURCE=2
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_TFO=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016
Running on OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.35 2014-04-04
PCRE library supports JIT : yes
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
I have also Certificate Transparency patch applied, backported from 1.6.
I've also captured 10000 strace line from each haproxy (I had to use
-s0 to hide any user data though) if it helps in any way. The only
suspicious thing I see with 1.5.16 are large (>50) number of
consecutive calls to epoll_wait. Is this normal?
I'm out of ideas how to debug this problem, so any suggestions are welcome.
--
Janusz Dziemidowicz