Hi,
HA-Proxy version 1.5.14
Ubuntu 15.10
I have had a look in the archives for this problem and it appears a few
times, but I have an additional worry:
When we have a number of sequential reloads happening, some old processes
are left behind.
This is ok if we have old connections that persist, process should die
after the all drain, this is fine.
Multiple processes listening on the same socket is something that happens
by design too, for more seamless reloads without interruption to services,
this is fine.
However, we are seeing old processes listening for new connections for a
long time, and accepting new connections with stale proxy config.
We had an outage for a few seconds, service discovery pulled servers from
the haproxy backends config and reloaded, and added them back a few seconds
later on recovery and reloaded again.
We continued to respond with 503 for a lot of requests because the old
process with no backends stayed around, still accepting new connections
even though the new process was well established.
I killed the old process and life returned to normal, but this is not the
first time this has happened. I have seen old processes hang around for
days and still accepting new connections (thankfully with a working backend
still intact)
Presumably this is not the intended behaviour of a reload - the listening
should stop soon after the new one starts.
Any advice before I start down the rather hacky route of scripts that seek
and destroy these processes?
Le meas,
Larry
--
*Confidentiality Notice:* - This message and any attached files may
contain information that is confidential and/or subject of legal privilege
intended only for the use by the intended recipient. If you are not the
intended recipient or the person responsible for delivering the message to
the intended recipient, be advised that you have received this message in
error and that any dissemination, copying or use of this message or
attachment is strictly forbidden, as is the disclosure of the information
therein. If you have received this message in error please notify the
sender immediately and delete the message.
HA-Proxy version 1.5.14
Ubuntu 15.10
I have had a look in the archives for this problem and it appears a few
times, but I have an additional worry:
When we have a number of sequential reloads happening, some old processes
are left behind.
This is ok if we have old connections that persist, process should die
after the all drain, this is fine.
Multiple processes listening on the same socket is something that happens
by design too, for more seamless reloads without interruption to services,
this is fine.
However, we are seeing old processes listening for new connections for a
long time, and accepting new connections with stale proxy config.
We had an outage for a few seconds, service discovery pulled servers from
the haproxy backends config and reloaded, and added them back a few seconds
later on recovery and reloaded again.
We continued to respond with 503 for a lot of requests because the old
process with no backends stayed around, still accepting new connections
even though the new process was well established.
I killed the old process and life returned to normal, but this is not the
first time this has happened. I have seen old processes hang around for
days and still accepting new connections (thankfully with a working backend
still intact)
Presumably this is not the intended behaviour of a reload - the listening
should stop soon after the new one starts.
Any advice before I start down the rather hacky route of scripts that seek
and destroy these processes?
Le meas,
Larry
--
*Confidentiality Notice:* - This message and any attached files may
contain information that is confidential and/or subject of legal privilege
intended only for the use by the intended recipient. If you are not the
intended recipient or the person responsible for delivering the message to
the intended recipient, be advised that you have received this message in
error and that any dissemination, copying or use of this message or
attachment is strictly forbidden, as is the disclosure of the information
therein. If you have received this message in error please notify the
sender immediately and delete the message.