1.7.5 503 Timeouts with SNI backend (2 replies)

We have the following backend configuration:

backend clientsite_ember
server cf foobar.cloudfront.net:443 ssl verify required sni str(
foobar.cloudfront.net) ca-file /etc/ssl/certs/ca-certificates.crt

This has been working great with 1.7.2 since February. I upgraded to 1.7.5
yesterday and today found that all requests through that backend were
returning 503. Testing the cloudfront url manually loaded the site.

Sample Logs:
May 18 10:13:47 ip-10-4-13-35 haproxy: <some_ip>:46924
[18/May/2017:17:13:32.237] http-in~ clientsite_ember/cf 0/0/-1/-1/14969 503
212 - - CC-- 10/10/1/1/1 0/0 {clientsite.com||Mozilla/5.0 (iPhone; CPU
iPhone OS 10_3_1 like Mac OS X) AppleWe} "GET /path1/?slide=1 HTTP/1.1"
May 18 10:13:54 ip-10-4-13-35 haproxy: <some_ip>:33235
[18/May/2017:17:13:22.354] http-in~ clientsite_ember/cf 0/30004/-1/-1/32296
503 212 - - CC-- 12/12/0/0/3 0/0 {clientsite.com||Mozilla/5.0 (Macintosh;
Intel Mac OS X 10_11_1) AppleWebKit/601.} "GET /path1/?slide=1 HTTP/1.1"
May 18 10:14:45 ip-10-4-13-35 haproxy: <some_ip>:9313
[18/May/2017:17:14:07.198] http-in~ clientsite_ember/cf 0/30003/-1/-1/38336
503 212 - - CC-- 13/13/0/0/3 0/0 {clientsite.com||Mozilla/5.0 (iPhone; CPU
iPhone OS 10_3_1 like Mac OS X) AppleWe} "GET /path1/?slide=1 HTTP/1.1"
May 18 10:15:30 ip-10-4-135-120 haproxy: <some_ip>:37948
[18/May/2017:17:14:59.850] http-in~ clientsite_ember/cf 0/30004/-1/-1/30400
503 212 - - CC-- 9/9/0/0/3 0/0 {clientsite.com||Mozilla/5.0 (Macintosh;
Intel Mac OS X 10_11_1) AppleWebKit/601.} "GET /path1/?slide=1 HTTP/1.1"
May 18 10:15:32 ip-10-4-69-34 haproxy: <some_ip>:38451
[18/May/2017:17:15:17.652] http-in~ clientsite_ember/cf 0/0/-1/-1/14714 503
212 - - CC-- 12/12/0/0/1 0/0 {clientsite.com||Mozilla/5.0 (iPhone; CPU
iPhone OS 10_3_1 like Mac OS X) AppleWe} "GET /path1/?slide=1 HTTP/1.1"
May 18 10:16:12 ip-10-4-135-120 haproxy: <some_ip>:52747
[18/May/2017:17:15:32.824] http-in~ clientsite_ember/cf 0/30004/-1/-1/40005
503 212 - - sC-- 12/12/0/0/3 0/0 {clientsite.com||Mozilla/5.0 (iPhone; CPU
iPhone OS 10_3_1 like Mac OS X) AppleWe} "GET /path1/?slide=1 HTTP/1.1"
May 18 10:17:45 ip-10-4-135-120 haproxy: <some_ip2>:60096
[18/May/2017:17:17:05.314] http-in~ clientsite_ember/cf 0/30005/-1/-1/40007
503 212 - - sC-- 9/9/0/0/3 0/0 {clientsite.com||Mozilla/5.0 (compatible;
YandexBot/3.0; +http://yandex.com/bots)} "GET /path2/ HTTP/1.1"
May 18 10:18:25 ip-10-4-69-34 haproxy: <some_ip2>:63513
[18/May/2017:17:17:45.827] http-in~ clientsite_ember/cf 0/30005/-1/-1/40006
503 212 - - sC-- 13/13/0/0/3 0/0 {clientsite.com||Mozilla/5.0 (compatible;
YandexBot/3.0; +http://yandex.com/bots)} "GET /path2/ HTTP/1.1"
May 18 10:18:27 ip-10-4-13-35 haproxy: <some_ip>:57858
[18/May/2017:17:18:15.384] http-in~ clientsite_ember/cf 0/0/-1/-1/11631 503
212 - - CC-- 15/15/1/1/1 0/0 {clientsite.com||Mozilla/5.0 (iPhone; CPU
iPhone OS 10_3_1 like Mac OS X) AppleWe} "GET /path1/?slide=1 HTTP/1.1"
May 18 10:18:34 ip-10-4-135-120 haproxy: <some_ip3>:55173
[18/May/2017:17:18:14.921] http-in~ clientsite_ember/cf 0/0/-1/-1/19973 503
212 - - CC-- 11/11/0/0/1 0/0 {clientsite.com||Mozilla/5.0 (compatible;
Cliqzbot/1.0; +http://cliqz.com/company} "GET /path5 HTTP/1.1"
May 18 10:18:49 ip-10-4-69-34 haproxy: <some_ip>:49219
[18/May/2017:17:18:34.138] http-in~ clientsite_ember/cf 0/0/-1/-1/15309 503
212 - - CC-- 16/16/0/0/1 0/0 {clientsite.com||Mozilla/5.0 (iPhone; CPU
iPhone OS 10_3_1 like Mac OS X) AppleWe} "GET /path1/?slide=1 HTTP/1.1"
May 18 10:18:55 ip-10-4-135-120 haproxy: <some_ip3>:58221
[18/May/2017:17:18:35.904] http-in~ clientsite_ember/cf 0/0/-1/-1/19988 503
212 - - CC-- 14/14/1/1/1 0/0 {clientsite.com||Mozilla/5.0 (compatible;
Cliqzbot/1.0; +http://cliqz.com/company} "GET /path5 HTTP/1.1"
May 18 10:19:06 ip-10-4-13-35 haproxy: <some_ip2>:36125
[18/May/2017:17:18:26.333] http-in~ clientsite_ember/cf 0/30005/-1/-1/40007
503 212 - - sC-- 19/19/0/0/3 0/0 {clientsite.com||Mozilla/5.0 (compatible;
YandexBot/3.0; +http://yandex.com/bots)} "GET /path2/ HTTP/1.1"
May 18 10:19:26 ip-10-4-135-120 haproxy: <some_ip4>:23388
[18/May/2017:17:18:47.167] http-in~ clientsite_ember/cf 0/30005/-1/-1/39090
503 212 - - CC-- 15/15/1/1/3 0/0 {clientsite.com||Mozilla/5.0 (Windows NT
6.2; Win64; x64) AppleWebKit/537.36 (KHT} "GET /path3 HTTP/1.1"
May 18 10:19:46 ip-10-4-135-120 haproxy: <some_ip2>:39212
[18/May/2017:17:19:06.835] http-in~ clientsite_ember/cf 0/30005/-1/-1/40006
503 212 - - sC-- 13/13/0/0/3 0/0 {clientsite.com||Mozilla/5.0 (compatible;
YandexBot/3.0; +http://yandex.com/bots)} "GET /path2/ HTTP/1.1"
May 18 10:19:47 ip-10-4-69-34 haproxy: <some_ip>:43670
[18/May/2017:17:19:38.573] http-in~ clientsite_ember/cf 0/0/-1/-1/9047 503
212 - - CC-- 18/18/0/0/0 0/0 {clientsite.com||Mozilla/5.0 (iPhone; CPU
iPhone OS 10_3_1 like Mac OS X) AppleWe} "GET /path1/?slide=1 HTTP/1.1"
May 18 10:19:55 ip-10-4-13-35 haproxy: <some_ip5>:20040
[18/May/2017:17:19:15.429] http-in~ clientsite_ember/cf 0/30004/-1/-1/40006
503 212 - - sC-- 18/18/1/1/3 0/0 {clientsite.com||Mozilla/5.0 (iPhone; CPU
iPhone OS 10_2_1 like Mac OS X) AppleWe} "GET /path4/?slide=1 HTTP/1.1"
May 18 10:20:06 ip-10-4-13-35 haproxy: <some_ip4>:48559
[18/May/2017:17:19:26.656] http-in~ clientsite_ember/cf 0/30005/-1/-1/40006
503 212 - - sC-- 16/16/1/1/3 0/0 {clientsite.com||Mozilla/5.0 (Windows NT
6.2; Win64; x64) AppleWebKit/537.36 (KHT} "GET /path3 HTTP/1.1"
May 18 10:20:16 ip-10-4-13-35 haproxy: <some_ip>:11792
[18/May/2017:17:19:55.785] http-in~ clientsite_ember/cf 0/0/-1/-1/20970 503
212 - - CC-- 15/15/0/0/2 0/0 {clientsite.com||Mozilla/5.0 (iPhone; CPU
iPhone OS 10_3_1 like Mac OS X) AppleWe} "GET /path1/?slide=5 HTTP/1.1"
May 18 10:20:17 ip-10-4-135-120 haproxy: <some_ip>:48952
[18/May/2017:17:19:47.829] http-in~ clientsite_ember/cf 0/0/-1/-1/29733 503
212 - - CC-- 13/13/0/0/2 0/0 {clientsite.com||Mozilla/5.0 (iPhone; CPU
iPhone OS 10_3_1 like Mac OS X) AppleWe} "GET /path1/?slide=1 HTTP/1.1"

I’ve never seen such high queue times. Working example for this
backend: 0/0/13/17/34

Replacing the haproxy containers (still 1.7.5) was enough to fix the issue.

Ryan