Multiple log entries with exact same Tq, Tc, Tr and Tt (no replies)

We run haproxy 1.5.14 on Ubuntu 14.04 in AWS. It load balances connections
from heroku to our backend, a neo4j cluster of multiple instances. It also
terminates SSL and handles auth. Connections to the backend are over
private network space within our AWS network and via unencrypted http.

During a recent event with heavy load, we saw log entries with the same
repeating pattern of multiple lines with the exact same Tq, Tc, Tr and Tt
values, but with different requests from different clients routed to
different backend pool members. We're not sure what's causing this or if it
relates to performance issues we saw during our events. Here's some log
lines with excerpted fields. We send a custom header with the query name
for metrics, which I've replaced with generic entries. I also replaced the
real client IP with 1.1.1.1, 2.2.2.2, etc.


Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37480
bs=db_production_read:production-04 hdrs="{query_1}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37481
bs=db_production_read:production-02 hdrs="{query_2}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37482
bs=db_production_read:production-03 hdrs="{query_3}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37484
bs=db_production_read:production-04 hdrs="{query_4}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37483
bs=db_production_read:production-02 hdrs="{query_5}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37485
bs=db_production_read:production-03 hdrs="{query_6}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37486
bs=db_production_read:production-05 hdrs="{query_7}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=2.2.2.2:37488
bs=db_production_read:production-03 hdrs="{query_8}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37487
bs=db_production_write:production-01 hdrs="{query_9}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37488
bs=db_production_read:production-04 hdrs="{query_10}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37489
bs=db_production_read:production-02 hdrs="{query_11}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37490
bs=db_production_read:production-03 hdrs="{query_12}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37491
bs=db_production_write:production-01 hdrs="{query_9}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37492
bs=db_production_read:production-04 hdrs="{query_13}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=3.3.3.3:53958
bs=db_production_read:production-02 hdrs="{query_10}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=3.3.3.3:53959
bs=db_production_read:production-03 hdrs="{query_11}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=3.3.3.3:53960
bs=db_production_read:production-04 hdrs="{query_12}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=2.2.2.2:37489
bs=db_production_write:production-01 hdrs="{query_9}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37493
bs=db_production_write:production-01 hdrs="{query_9}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=3.3.3.3:53961
bs=db_production_write:production-01 hdrs="{query_9}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=2.2.2.2:37490
bs=db_production_write:production-01 hdrs="{query_9}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=3.3.3.3:53962
bs=db_production_write:production-01 hdrs="{query_14}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=2.2.2.2:37491
bs=db_production_write:production-01 hdrs="{query_9}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=3.3.3.3:53963
bs=db_production_write:production-01 hdrs="{query_9}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37495
bs=db_production_write:production-01 hdrs="{query_9}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=3.3.3.3:53964
bs=db_production_write:production-01 hdrs="{query_9}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=3.3.3.3:53965
bs=db_production_read:production-02 hdrs="{query_13}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37496
bs=db_production_write:production-01 hdrs="{query_9}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37498
bs=db_production_read:production-04 hdrs="{query_15}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=2.2.2.2:37492
bs=db_production_read:production-02 hdrs="{query_14}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37499
bs=db_production_write:production-01 hdrs="{query_13}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=2.2.2.2:37493
bs=db_production_read:production-03 hdrs="{query_10}"
Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=2.2.2.2:37495
bs=db_production_read:production-02 hdrs="{query_12}"

The logs repeat like this with different sets of entries with the same Tq,
Tc, Tr and Tt for all lines. Is this a known phenomena? A bug in the
logging or code? Some kind of connection reuse we're not aware of?

Any assistance identifying this issue is greatly appreciated.


Thanks!



*DAVE STERN*

ENGINEERING


FIFTYTHREE, INC. http://www.fiftythree.com/

60 HUDSON ST #1810
NEW YORK, NY 10013 http://goo.gl/Rzk6lZ

@DAVESTERN1 https://twitter.com/davestern1