Under heavy load haproxy returns 400 error
http://discourse.haproxy.org/t/400-error-on-cookie-string/868/13
OS Linux 3.10.0-514.2.2.el7.x86_64 #1 SMP Tue Dec 6 23:06:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
I've tried:
1. turn on accept-invalid-http-request. Some requests are now working some not. show errors see all of them
2. tune.bufsize 32768 maxrewrite 1024
3. tune.bufsize 32768
4. tune.bufsize 65536
problem still here
Tcpdump shows normal http requests(attached)
I've also tried to move to another server. Linux version is the same. Doesn't help
haproxy details
HA-Proxy version 1.7.1 2016/12/13
Copyright 2000-2016 Willy Tarreau <willy@haproxy.org>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -DTCP_USER_TIMEOUT=18
OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1 USE_PCRE_JIT=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.2j 26 Sep 2016
Running on OpenSSL version : OpenSSL 1.0.2j 26 Sep 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : yes
Built without Lua support
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[COMP] compression
[TRACE] trace
[SPOE] spoe
All discussion
http://discourse.haproxy.org/t/400-error-on-cookie-string/868
server ip address is 176.9.84.204. It's working but all traffic now goes to iis directly - 138.201.164.27
Example of "show errors":
https://gist.github.com/thyn/bf27e12dd27efb2173d3b625997b00de
Another example with tcpdump (attached as tcp6)
Total events captured on [22/Dec/2016:12:17:57.879] : 1
[22/Dec/2016:12:17:57.701] frontend http-in (#3): invalid request
backend <NONE> (#-1), server <NONE> (#-1), event #0
src 75.121.245.15:49372, session #535, session flags 0x00000080
HTTP msg state 26, msg flags 0x00000000, tx flags 0x00000000
HTTP chunk len 0 bytes, HTTP body len 0 bytes
buffer flags 0x00808002, out 0 bytes, total 2904 bytes
pending 2904 bytes, wrapping at 32768, error at position 1453:
00000 GET /?fm=1&qc_url=is-there-caffeine-in-chocolate&qc_type=question&s=1e
00070+ 4aa05d-f9a2-46f8-8d4f-a59bf9cdb3ad&answer=43049&abc_rd=&abc_s=sr7imz23
00140+ 8&shortener=sr7imz238&tref4=3048187&tref11=xmonetize&tref12=facebook-a
00210+ d&tref13=how-high-is-your-psychological-iq®date=2016-10-31&days=51&
00280+ date=2016-12-22%2005%3A51%3A41&shdate=2016-12-22&abc_pos=0 HTTP/1.1\r
00348+ \n
00349 Host: quizzclub.com\r\n
00370 Connection: keep-alive\r\n
00394 Upgrade-Insecure-Requests: 1\r\n
00424 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (K
00494+ HTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36\r\n
00547 X-Chrome-UMA-Enabled: 1\r\n
00572 X-Client-Data: CJW2yQEIorbJAQjDtskBCPucygEIqZ3KAQ==\r\n
00625 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/we
00695+ bp,*/*;q=0.8\r\n
00709 DNT: 1\r\n
00717 Accept-Encoding: gzip, deflate, sdch\r\n
00755 Accept-Language: en-US,en;q=0.8\r\n
00788 Cookie: 323d7f8f696ae3e7=53da53acf098e03e; fqz=93f4a264-2f28-477e-805d
00858+ -06e49d72ad1b; km_lv=x; __qca=P0-1248156878-1477951772464; _ym_uid=147
00928+ 7951774479571232; km_ai=6e5714c2-b69f-e611-80cd-3497f6004133; km_ni=6e
00998+ 5714c2-b69f-e611-80cd-3497f6004133; userid=559B4363-0913-D64C-49BB-829
01068+ E4AF73491; __lx205879_load_cnt=11; __lx205879_load_tmr=1480204024970;
01138+ __lx205879_load_tmr_pre=1480204025314; _pk_id.1.6d88=74cb3958dbecf4d3.
01208+ 1478470023.45.1481239615.1481239549.; mp_5c2767339ff8b27b54ee2f71c633b
01278+ d56_mixpanel=%7B%22distinct_id%22%3A%20%226e5714c2-b69f-e611-80cd-3497
01348+ f6004133%22%2C%22tref1%22%3A%20%22xmonetize%22%2C%22tref2%22%3A%20%22e
01418+ mailsmaster%22%2C%22tref3%22%3A%20%22Daily%20Quiz%20%7C%20Tests%20Clic
01488+ kers%200-14d%20Any%2024k%2Fd%20%7C%2011d%2B%20%7C%20Theme%20%7C%201%20
01558+ PM%20LiveIntent%22%2C%22tref4%22%3A%20%22NA%22%2C%22emaildomain%22%3A%
01628+ 20%22gmail.com%22%2C%22landing%22%3A%20%22default_emtest%22%2C%22regda
01698+ te%22%3A%20%222016-10-31%22%2C%22tref12%22%3A%20%22facebook-ad%22%2C%2
01768+ 2tref13%22%3A%20%22how-high-is-your-psychological-iq%22%2C%22tref14%22
01838+ %3A%20%22LL%22%2C%22tref15%22%3A%20%22personality%22%2C%22%24initial_r
01908+ eferrer%22%3A%20%22%24direct%22%2C%22%24initial_referring_domain%22%3A
01978+ %20%22%24direct%22%7D; crtg_rta=ad4g336280020%3D1%3Bad4g160600020%3D1%
02048+ 3B; .AspNet.ApplicationCookie=4QQ8dZ57Vp-tKy1kBXWSn68MSvGUAGmflloOXb3S
02118+ Im_9HtGkpzqiCGW1evbChiZ2qTYaelWEV2KKtgr2OAXEwsqeay5E8VnPczEFl4EBG4FIm9
02188+ 6OT2BzQAeY06wZJPe8R77hcnHexutoYlI9CcsXUGA_fiBElg38DWmZyPHIc-0bG-mKTTbI
02258+ DxyO9K3kzHU_LJhqyvmt3N8e7ewP_yI9uBBy2vSaqRPa5Tl1MDqKdT9usYgaPnWauThrsk
02328+ IHsIjFl9UPqFA3hkICauDOvzSd_USOf3mcZTXq5l4DLCT0HwYb5pq50KZUP0GPV6nKe2Nt
02398+ 3hvGojJL-9XriStCR8a5PyUXE4y4cGV7_Y5e6cXeWtaCn13HPLiVhLYa2b5aODKKG7ElLt
02468+ d1k3KXVxVZMtx2NC76Ppr6BBECUj0IqQVnmYbQIDKZ7csY5COQxMBHd7jg_6MBQNCS1aEx
02538+ tf46EDVbeqF7FKyU0qrxZm6VLi-yw7FMsDLcKc364BI5GObp2R2_A7TZK6qsqEBB2YS91m
02608+ -pIg; amplitude_idquizzclub.com=eyJkZXZpY2VJZCI6IjQ0ZmYxYzg5LThhZTQtND
02678+ BlOC1hNGE0LWRiZjAxNzdlNDI2NVIiLCJ1c2VySWQiOiI2ZTU3MTRjMi1iNjlmLWU2MTEt
02748+ ODBjZC0zNDk3ZjYwMDQxMzMiLCJvcHRPdXQiOmZhbHNlLCJzZXNzaW9uSWQiOjE0ODIxOD
02818+ A1MTEzNTMsImxhc3RFdmVudFRpbWUiOjE0ODIxODA1MTEzNjAsImV2ZW50SWQiOjIxLCJp
02888+ ZGVudGlmeUlkIjo2
-------- Конец пересылаемого сообщения --------
http://discourse.haproxy.org/t/400-error-on-cookie-string/868/13
OS Linux 3.10.0-514.2.2.el7.x86_64 #1 SMP Tue Dec 6 23:06:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
I've tried:
1. turn on accept-invalid-http-request. Some requests are now working some not. show errors see all of them
2. tune.bufsize 32768 maxrewrite 1024
3. tune.bufsize 32768
4. tune.bufsize 65536
problem still here
Tcpdump shows normal http requests(attached)
I've also tried to move to another server. Linux version is the same. Doesn't help
haproxy details
HA-Proxy version 1.7.1 2016/12/13
Copyright 2000-2016 Willy Tarreau <willy@haproxy.org>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -DTCP_USER_TIMEOUT=18
OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1 USE_PCRE_JIT=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.2j 26 Sep 2016
Running on OpenSSL version : OpenSSL 1.0.2j 26 Sep 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : yes
Built without Lua support
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[COMP] compression
[TRACE] trace
[SPOE] spoe
All discussion
http://discourse.haproxy.org/t/400-error-on-cookie-string/868
server ip address is 176.9.84.204. It's working but all traffic now goes to iis directly - 138.201.164.27
Example of "show errors":
https://gist.github.com/thyn/bf27e12dd27efb2173d3b625997b00de
Another example with tcpdump (attached as tcp6)
Total events captured on [22/Dec/2016:12:17:57.879] : 1
[22/Dec/2016:12:17:57.701] frontend http-in (#3): invalid request
backend <NONE> (#-1), server <NONE> (#-1), event #0
src 75.121.245.15:49372, session #535, session flags 0x00000080
HTTP msg state 26, msg flags 0x00000000, tx flags 0x00000000
HTTP chunk len 0 bytes, HTTP body len 0 bytes
buffer flags 0x00808002, out 0 bytes, total 2904 bytes
pending 2904 bytes, wrapping at 32768, error at position 1453:
00000 GET /?fm=1&qc_url=is-there-caffeine-in-chocolate&qc_type=question&s=1e
00070+ 4aa05d-f9a2-46f8-8d4f-a59bf9cdb3ad&answer=43049&abc_rd=&abc_s=sr7imz23
00140+ 8&shortener=sr7imz238&tref4=3048187&tref11=xmonetize&tref12=facebook-a
00210+ d&tref13=how-high-is-your-psychological-iq®date=2016-10-31&days=51&
00280+ date=2016-12-22%2005%3A51%3A41&shdate=2016-12-22&abc_pos=0 HTTP/1.1\r
00348+ \n
00349 Host: quizzclub.com\r\n
00370 Connection: keep-alive\r\n
00394 Upgrade-Insecure-Requests: 1\r\n
00424 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (K
00494+ HTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36\r\n
00547 X-Chrome-UMA-Enabled: 1\r\n
00572 X-Client-Data: CJW2yQEIorbJAQjDtskBCPucygEIqZ3KAQ==\r\n
00625 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/we
00695+ bp,*/*;q=0.8\r\n
00709 DNT: 1\r\n
00717 Accept-Encoding: gzip, deflate, sdch\r\n
00755 Accept-Language: en-US,en;q=0.8\r\n
00788 Cookie: 323d7f8f696ae3e7=53da53acf098e03e; fqz=93f4a264-2f28-477e-805d
00858+ -06e49d72ad1b; km_lv=x; __qca=P0-1248156878-1477951772464; _ym_uid=147
00928+ 7951774479571232; km_ai=6e5714c2-b69f-e611-80cd-3497f6004133; km_ni=6e
00998+ 5714c2-b69f-e611-80cd-3497f6004133; userid=559B4363-0913-D64C-49BB-829
01068+ E4AF73491; __lx205879_load_cnt=11; __lx205879_load_tmr=1480204024970;
01138+ __lx205879_load_tmr_pre=1480204025314; _pk_id.1.6d88=74cb3958dbecf4d3.
01208+ 1478470023.45.1481239615.1481239549.; mp_5c2767339ff8b27b54ee2f71c633b
01278+ d56_mixpanel=%7B%22distinct_id%22%3A%20%226e5714c2-b69f-e611-80cd-3497
01348+ f6004133%22%2C%22tref1%22%3A%20%22xmonetize%22%2C%22tref2%22%3A%20%22e
01418+ mailsmaster%22%2C%22tref3%22%3A%20%22Daily%20Quiz%20%7C%20Tests%20Clic
01488+ kers%200-14d%20Any%2024k%2Fd%20%7C%2011d%2B%20%7C%20Theme%20%7C%201%20
01558+ PM%20LiveIntent%22%2C%22tref4%22%3A%20%22NA%22%2C%22emaildomain%22%3A%
01628+ 20%22gmail.com%22%2C%22landing%22%3A%20%22default_emtest%22%2C%22regda
01698+ te%22%3A%20%222016-10-31%22%2C%22tref12%22%3A%20%22facebook-ad%22%2C%2
01768+ 2tref13%22%3A%20%22how-high-is-your-psychological-iq%22%2C%22tref14%22
01838+ %3A%20%22LL%22%2C%22tref15%22%3A%20%22personality%22%2C%22%24initial_r
01908+ eferrer%22%3A%20%22%24direct%22%2C%22%24initial_referring_domain%22%3A
01978+ %20%22%24direct%22%7D; crtg_rta=ad4g336280020%3D1%3Bad4g160600020%3D1%
02048+ 3B; .AspNet.ApplicationCookie=4QQ8dZ57Vp-tKy1kBXWSn68MSvGUAGmflloOXb3S
02118+ Im_9HtGkpzqiCGW1evbChiZ2qTYaelWEV2KKtgr2OAXEwsqeay5E8VnPczEFl4EBG4FIm9
02188+ 6OT2BzQAeY06wZJPe8R77hcnHexutoYlI9CcsXUGA_fiBElg38DWmZyPHIc-0bG-mKTTbI
02258+ DxyO9K3kzHU_LJhqyvmt3N8e7ewP_yI9uBBy2vSaqRPa5Tl1MDqKdT9usYgaPnWauThrsk
02328+ IHsIjFl9UPqFA3hkICauDOvzSd_USOf3mcZTXq5l4DLCT0HwYb5pq50KZUP0GPV6nKe2Nt
02398+ 3hvGojJL-9XriStCR8a5PyUXE4y4cGV7_Y5e6cXeWtaCn13HPLiVhLYa2b5aODKKG7ElLt
02468+ d1k3KXVxVZMtx2NC76Ppr6BBECUj0IqQVnmYbQIDKZ7csY5COQxMBHd7jg_6MBQNCS1aEx
02538+ tf46EDVbeqF7FKyU0qrxZm6VLi-yw7FMsDLcKc364BI5GObp2R2_A7TZK6qsqEBB2YS91m
02608+ -pIg; amplitude_idquizzclub.com=eyJkZXZpY2VJZCI6IjQ0ZmYxYzg5LThhZTQtND
02678+ BlOC1hNGE0LWRiZjAxNzdlNDI2NVIiLCJ1c2VySWQiOiI2ZTU3MTRjMi1iNjlmLWU2MTEt
02748+ ODBjZC0zNDk3ZjYwMDQxMzMiLCJvcHRPdXQiOmZhbHNlLCJzZXNzaW9uSWQiOjE0ODIxOD
02818+ A1MTEzNTMsImxhc3RFdmVudFRpbWUiOjE0ODIxODA1MTEzNjAsImV2ZW50SWQiOjIxLCJp
02888+ ZGVudGlmeUlkIjo2
-------- Конец пересылаемого сообщения --------