Hi all,
We’re experiencing a problem with clients receiving 503 responses when
making a request to a backend with a Authorisation header set (basic
authentication) using HAProxy 1.7.2. We didn't experience this with HAProxy
1.6.11.
The HAProxy stats page shows that the backends in question have no downtime
(what I would usually expect to see in cases where 503 statuses are
returned). Only certain clients are affected by this behaviour - a number
have been completely unaffected. The behaviour we're seeing shares some
similarities to one that reported in 1.7.1 (
https://www.mail-archive.com/haproxy@formilux.org/msg24613.html) but we're
not seeing any useful output from show errors and the code is 503. We do
know that packet reordering/retranmission is happening in at least one case..
Our clients can retry these requests and occasionally get them to succeed.
Additionally, they found putting delays between the post of data and close
of the connection did lead to successes in making the connection. We're
sure HAProxy is the source of our client's 503 errors as the clients are
getting the text of our custom 503 page (rather than a 503 from the IIS
service behind).
For context - we have an ELB in TCP-only mode in front of the HAProxy (to
distribute connections across multiple HAProxy instances) if that's
relevant. We're using the latest HAProxy 1.7 Ubuntu PPA's (
https://launchpad.net/~vbernat/+archive/ubuntu/haproxy-1.7). Logging is
enabled, with the following format string (we capture the Host request
header with a len of 64 for +Q) -
log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\
%CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}r\ %ID
but we don't appear to be seeing the requests that are receiving 503
responses being logged. We do use 'option dontlognull' as these HAProxy
instances are behind ELB, and hence get checked for TCP liveness but as
these are valid HTTPS requests being returned 503 I would expect them to be
in our main haproxy log.
I'd really appreciate any input you could give on further triage steps
(especially being able to identify in log when clients are receiving 503
responses) or potential causes we can look at.
Thanks,
Jon
We’re experiencing a problem with clients receiving 503 responses when
making a request to a backend with a Authorisation header set (basic
authentication) using HAProxy 1.7.2. We didn't experience this with HAProxy
1.6.11.
The HAProxy stats page shows that the backends in question have no downtime
(what I would usually expect to see in cases where 503 statuses are
returned). Only certain clients are affected by this behaviour - a number
have been completely unaffected. The behaviour we're seeing shares some
similarities to one that reported in 1.7.1 (
https://www.mail-archive.com/haproxy@formilux.org/msg24613.html) but we're
not seeing any useful output from show errors and the code is 503. We do
know that packet reordering/retranmission is happening in at least one case..
Our clients can retry these requests and occasionally get them to succeed.
Additionally, they found putting delays between the post of data and close
of the connection did lead to successes in making the connection. We're
sure HAProxy is the source of our client's 503 errors as the clients are
getting the text of our custom 503 page (rather than a 503 from the IIS
service behind).
For context - we have an ELB in TCP-only mode in front of the HAProxy (to
distribute connections across multiple HAProxy instances) if that's
relevant. We're using the latest HAProxy 1.7 Ubuntu PPA's (
https://launchpad.net/~vbernat/+archive/ubuntu/haproxy-1.7). Logging is
enabled, with the following format string (we capture the Host request
header with a len of 64 for +Q) -
log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\
%CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}r\ %ID
but we don't appear to be seeing the requests that are receiving 503
responses being logged. We do use 'option dontlognull' as these HAProxy
instances are behind ELB, and hence get checked for TCP liveness but as
these are valid HTTPS requests being returned 503 I would expect them to be
in our main haproxy log.
I'd really appreciate any input you could give on further triage steps
(especially being able to identify in log when clients are receiving 503
responses) or potential causes we can look at.
Thanks,
Jon