Hi All,
I'm using HAProxy 1.5.14 (the packaged version on CentOS 7.2) to front an IMAP proxy service, and I've noticed I'm getting quite a lot of connections in FIN_WAIT_2. For example, here are the totals for each state on my system right now:
$ netstat -an | awk '/tcp/ {print $6}' | sort | uniq -c
255 CLOSE_WAIT
1 CLOSING
1802 ESTABLISHED
82 FIN_WAIT1
514 FIN_WAIT2
8 LAST_ACK
11 LISTEN
2 SYN_RECV
315 TIME_WAIT
What seems to be happening is that the backend is closing the connection, leading HAProxy to close the connection to the client by sending a FIN. The client-side connection goes to FIN_WAIT_1, and then on receiving an ACK, to FIN_WAIT_2. It appears though, that some clients are not sending a FIN in return, resulting in the FIN_WAIT_2 connection hanging around until it times out.
I notice here that the connection takes 35 minutes to time out once entering FIN_WAIT_2, which is the value I'm setting for 'timeout tunnel'. I've tried setting 'timeout client-fin' to 30s to mitigate this issue, but it doesn't seem to have any effect. Can someone confirm whether timeout client-fin applies to FIN_WAIT_2, or if perhaps I'm not using the option correctly? I also wonder if the nolinger option might be effective in this case.
Also, in case it is relevant, I should point out that I am using the 'usesrc clientip' option on my backend servers.
I've included my configuration below.
Many thanks,
Richard
>>>>>>>>>>>>>>>>>>>>>
global
chroot /var/lib/haproxy
group root
log 127.0.0.1 local0 notice
log-send-hostname
maxconn 48000
pidfile /var/run/haproxy.pid
stats socket /var/lib/haproxy/stats
stats socket /var/lib/haproxy/stats.admin mode 600 level admin
tune.ssl.default-dh-param 2048
user root
defaults
log global
maxconn 8000
option redispatch
retries 3
stats enable
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout check 10s
listen hosting-imaps_proxy-external-vip
bind 10.0.1.1:993
mode tcp
acl pool_dead nbsrv lt 1
balance leastconn
maxconn 14000
option tcplog
option tcp-check
source 0.0.0.0 usesrc clientip
tcp-check expect rstring \*\ OK
tcp-request connection reject if pool_dead
timeout client-fin 30s
timeout tunnel 35m
server imap1 10.0.2.1:993 check port 3143
server imap2 10.0.2.2:993 check port 3143
_____________________________________________________________________________
This email has been filtered by SMX. For more info visit http://smxemail.com
_____________________________________________________________________________
I'm using HAProxy 1.5.14 (the packaged version on CentOS 7.2) to front an IMAP proxy service, and I've noticed I'm getting quite a lot of connections in FIN_WAIT_2. For example, here are the totals for each state on my system right now:
$ netstat -an | awk '/tcp/ {print $6}' | sort | uniq -c
255 CLOSE_WAIT
1 CLOSING
1802 ESTABLISHED
82 FIN_WAIT1
514 FIN_WAIT2
8 LAST_ACK
11 LISTEN
2 SYN_RECV
315 TIME_WAIT
What seems to be happening is that the backend is closing the connection, leading HAProxy to close the connection to the client by sending a FIN. The client-side connection goes to FIN_WAIT_1, and then on receiving an ACK, to FIN_WAIT_2. It appears though, that some clients are not sending a FIN in return, resulting in the FIN_WAIT_2 connection hanging around until it times out.
I notice here that the connection takes 35 minutes to time out once entering FIN_WAIT_2, which is the value I'm setting for 'timeout tunnel'. I've tried setting 'timeout client-fin' to 30s to mitigate this issue, but it doesn't seem to have any effect. Can someone confirm whether timeout client-fin applies to FIN_WAIT_2, or if perhaps I'm not using the option correctly? I also wonder if the nolinger option might be effective in this case.
Also, in case it is relevant, I should point out that I am using the 'usesrc clientip' option on my backend servers.
I've included my configuration below.
Many thanks,
Richard
>>>>>>>>>>>>>>>>>>>>>
global
chroot /var/lib/haproxy
group root
log 127.0.0.1 local0 notice
log-send-hostname
maxconn 48000
pidfile /var/run/haproxy.pid
stats socket /var/lib/haproxy/stats
stats socket /var/lib/haproxy/stats.admin mode 600 level admin
tune.ssl.default-dh-param 2048
user root
defaults
log global
maxconn 8000
option redispatch
retries 3
stats enable
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout check 10s
listen hosting-imaps_proxy-external-vip
bind 10.0.1.1:993
mode tcp
acl pool_dead nbsrv lt 1
balance leastconn
maxconn 14000
option tcplog
option tcp-check
source 0.0.0.0 usesrc clientip
tcp-check expect rstring \*\ OK
tcp-request connection reject if pool_dead
timeout client-fin 30s
timeout tunnel 35m
server imap1 10.0.2.1:993 check port 3143
server imap2 10.0.2.2:993 check port 3143
_____________________________________________________________________________
This email has been filtered by SMX. For more info visit http://smxemail.com
_____________________________________________________________________________