The cause for 504's (4 replies)

504's are killing us and we have no clue why we get them

Here's a sample log entry:

Jun 10 17:27:33 localhost haproxy[23508]: 10.126.160.11:37139 [10/Jun/2015:17:26:03.027] http-in resub-bb-default/njorch0pe16 30935/0/1/-1/90937 504 194 - - sH-- 16/14/0/0/0 0/0 {569760396|297|RESUB|EMAIL|0|9001|0|0|1.0|NJ|60} "POST /somepath HTTP/1.1"
This is not a case of the sever being unavailable or being too occupied. I profiled the server and is running at over 90% idle
In my service logs there are no entries corresponding to this request
leading me to believe that the request actually never made it to the server..

How can we troubleshoot this and find out why HaProxy decided to return a 504?

thanks
Jeff