Keep-alive causing latency spike (2 replies)

Hey folks, I could use some help figuring this one out. My environment
looks like this:

(client) <-> (nginx) <-> (haproxy 1.5) <-> (backend server pools all with
(nginx -> phpfpm))

Where the client is a browser or bot, nginx is terminating http/https/spdy,
haproxy has the business and routing logic, and the backend pools are
running nginx+php-fpm.

Traditionally we have closed the connection after every request (a) between
the client and nginx (b) between nginx and haproxy and (c) between haproxy
and the backend servers.

Some of it was unintentional, some of it was intentionally working around
issues with php-fpm servicing multiple requests from the same connection
(which I assume is some sort of application programming issue).

So I made changes to enable keep-alive connections between the client and
nginx - no problems.

Then I made changes to enable keep-alive connections between nginx and
haproxy, and I've got problems. I'm seeing a 25% increase in latency, where
as I expected no change or a slight decrease. So either something is going
on that I don't understand or I'm not measuring the latency right and
haven't noticed before because all the connection closing hid the issue.

The way I am monitoring the request latency is by averaging the Tt field
from the haproxy logs by second.

My HAProxy config looks like this:

global
daemon
maxconn 81920

defaults
log global
timeout http-request 5s
timeout client 15s
timeout server 15s
timeout connect 4s
option forwardfor except 127.0.0.1
option httplog
option redispatch
option log-separate-errors
retries 2

frontend myrontend
bind 127.0.0.1:8080 defer-accept
backlog 65536
mode http
option http-keep-alive
log 127.0.0.1 local0
log 127.0.0.1 local1 err

default_backend mypool

backend mypool
mode http
option http-server-close
balance roundrobin


My NGinx config looks like this:

....
upstream haproxy {
server 127.0.0.1:8080;
keepalive 1; # also tried, 16, 32, 256, saw the same latency spike with
all
}
....
location / {
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_buffering off;
proxy_pass http://haproxy;
}


Any help or advice appreciated!