BUG: Seg fault when reloading from saved state after config change (no replies)

BUG:

Extra spaces inserted into the haproxy.cfg file cause haproxy reload with
saved state to seg fault haproxy.

WORKAROUND:

Remove the existing state file OR remove save state from config.

REPRODUCE:

- Enable save state across reloads
- Reload and save state file
- Insert extra space before "weight".
- Attempt to reload or validate the new config and haproxy will segv.

DESCRIPTION

Our haproxy config is comprised of a bunch of puppet magic. Someone made a
change that included an additional space in nearly every server line of our
config, which has almost 1300 servers:

- server foo3001.xx.box.net 10.16.26.2:8080 weight 16 check port 9086
inter 10000
- server foo3002.xx.box.net 10.16.26.3:8080 weight 16 check port 9086
inter 10000
- server foo3003.xx.box.net 10.16.26.4:8080 weight 32 check port 9086
inter 10000
- server foo3004.xx.box.net 10.16.26.5:8080 weight 16 check port 9086
inter 10000
+ server foo3001.xx.box.net 10.16.26.2:8080 weight 16 check port 9086
inter 10000
+ server foo3002.xx.box.net 10.16.26.3:8080 weight 16 check port 9086
inter 10000
+ server foo3003.xx.box.net 10.16.26.4:8080 weight 32 check port 9086
inter 10000
+ server foo3004.xx.box.net 10.16.26.5:8080 weight 16 check port 9086
inter 10000

We are testing save server state on some of some of our hosts. This change
causes reloads to fail with a segmentation fault.

If I remove the state saving stuff from the config, the segv goes away:

% diff ~/haproxy*cfg
15,16c15,16
< # stats socket /var/run/haproxy.sock mode 666 level admin
< # server-state-file /var/lib/haproxy/haproxy-server-state
---
> stats socket /var/run/haproxy.sock mode 666 level admin
> server-state-file /var/lib/haproxy/haproxy-server-state
20c20
< # load-server-state-from-file global
---
> load-server-state-from-file global
% /usr/sbin/haproxy -c -f ~/haproxy-test.cfg
Configuration file is valid
%

If I remove the state file, the config check is valid:

% rm /var/lib/haproxy/haproxy-server-state
% /usr/sbin/haproxy -c -f ~/haproxy.cfg
[WARNING] 149/180256 (140127) : stats socket will not work as expected in
multi-process mode (nbproc > 1), you should force process binding globally
using 'stats bind-process' or per socket using the 'process' attribute.
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
[WARNING] 149/180256 (140127) : Can't open server state file
'/var/lib/haproxy/haproxy-server-state': No such file or directory
Configuration file is valid
%


-------------------------------------------------------------------------------------------

From /etc/haproxy/haproxy.cfg:

global
stats socket /var/run/haproxy.sock mode 666 level admin
server-state-file /var/lib/haproxy/haproxy-server-state

defaults
log global
load-server-state-from-file global

-------------------------------------------------------------------------------------------

% /usr/sbin/haproxy -vv
HA-Proxy version 1.7.5 2017/04/03
Copyright 2000-2017 Willy Tarreau <willy@haproxy.org>

Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
OPTIONS = USE_LINUX_TPROXY=1 USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1
USE_OPENSSL=1 USE_PCRE=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.3
Running on zlib version : 1.2.3
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 7.8 2008-09-05
Running on PCRE version : 7.8 2008-09-05
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built without Lua support
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[TRACE] trace
[COMP] compression

%

% /usr/sbin/haproxy -c -V -f ~/haproxy.cfg
[WARNING] 149/175354 (114894) : stats socket will not work as expected in
multi-process mode (nbproc > 1), you should force process binding globally
using 'stats bind-process' or per socket using the 'process' attribute.
[WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13001.xx.box.net is
DOWN, changed from server-state after a reload. 230 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13002.xx.box.net is
DOWN, changed from server-state after a reload. 229 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13003.xx.box.net is
DOWN, changed from server-state after a reload. 228 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13004.xx.box.net is
DOWN, changed from server-state after a reload. 227 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13032.xx.box.net is
DOWN, changed from server-state after a reload. 226 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13083.xx.box.net is
DOWN, changed from server-state after a reload. 225 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13105.xx.box.net is
DOWN, changed from server-state after a reload. 224 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13122.xx.box.net is
DOWN, changed from server-state after a reload. 223 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13189.xx.box.net is
DOWN, changed from server-state after a reload. 222 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13214.xx.box.net is
DOWN, changed from server-state after a reload. 221 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13288.xx.box.net is
DOWN, changed from server-state after a reload. 220 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13319.xx.box.net is
DOWN, changed from server-state after a reload. 219 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server http_w_c1_6000/c13321.xx.box.net is
DOWN, changed from server-state after a reload. 218 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_d_6014/d13006.xx.box.net is
DOWN, changed from server-state after a reload. 30 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_d_6014/d13008.xx.box.net is
DOWN, changed from server-state after a reload. 29 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_d_6014/d13011.xx.box.net is
DOWN, changed from server-state after a reload. 28 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_d_6014/d13015.xx.box.net is
DOWN, changed from server-state after a reload. 27 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_d_6014/d13024.xx.box.net is
DOWN, changed from server-state after a reload. 26 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_d_6014/d13031.xx.box.net is
DOWN, changed from server-state after a reload. 25 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_d_6014/d13032.xx.box.net is
DOWN, changed from server-state after a reload. 24 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_d_6014/d13035.xx.box.net is
DOWN, changed from server-state after a reload. 23 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_ms_6005/c13001.xx.box.net is
DOWN, changed from server-state after a reload. 230 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_ms_6005/c13002.xx.box.net is
DOWN, changed from server-state after a reload. 229 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_ms_6005/c13003.xx.box.net is
DOWN, changed from server-state after a reload. 228 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_ms_6005/c13004.xx.box.net is
DOWN, changed from server-state after a reload. 227 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_ms_6005/c13032.xx.box.net is
DOWN, changed from server-state after a reload. 226 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_ms_6005/c13083.xx.box.net is
DOWN, changed from server-state after a reload. 225 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_ms_6005/c13105.xx.box.net is
DOWN, changed from server-state after a reload. 224 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_ms_6005/c13122.xx.box.net is
DOWN, changed from server-state after a reload. 223 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_ms_6005/c13189.xx.box.net is
DOWN, changed from server-state after a reload. 222 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_ms_6005/c13214.xx.box.net is
DOWN, changed from server-state after a reload. 221 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_ms_6005/c13288.xx.box.net is
DOWN, changed from server-state after a reload. 220 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_ms_6005/c13319.xx.box.net is
DOWN, changed from server-state after a reload. 219 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_ms_6005/c13321.xx.box.net is
DOWN, changed from server-state after a reload. 218 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_u_u_6003/u3050.xx.box.net is
DOWN, changed from server-state after a reload. 113 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_u_u_6003/u3053.xx.box.net is
DOWN, changed from server-state after a reload. 112 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_u_u_6003/u3063.xx.box.net is
DOWN, changed from server-state after a reload. 111 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_u_u_6003/u3070.xx.box.net is
DOWN, changed from server-state after a reload. 110 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_u_u_6003/u3100.xx.box.net is
DOWN, changed from server-state after a reload. 109 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_u_u_6003/u3120.xx.box.net is
DOWN, changed from server-state after a reload. 108 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : Server https_u_u_6003/u3121.xx.box.net is
DOWN, changed from server-state after a reload. 107 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : In backend 'https_u_u_6003' (id: 17):
server name mismatch: from server state file: 'u3126.xx.box.net', from
running config 'u3127.xx.box.net'
[WARNING] 149/175354 (114894) : Server https_u_u_6003/u3127.xx.box.net is
DOWN, changed from server-state after a reload. 106 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 149/175354 (114894) : In backend 'https_u_u_6003' (id: '17'):
server ID mismatch: from server state file: '98', from running config 97
Segmentation fault
%