Hi all,
the impatient readers among you will have noticed that it's been almost 3
weeks since I sent the e-mail announcing the imminent release of 1.6-dev2.
That end of merge window has been a nightmare and is not finished, but I
thought it would be wise to issue dev2 anyway so that people can test the
stuff that has been merged anyway. Lesson learned, for 1.7 we'll have a
much shorter merge window so that people don't have enough time to push
that much stuff at the last minute :-)
To be honnest, I'm far from being satisfied with this version. It's as huge
as dev1 (344 commits) despite some things still being pending. Also noticed
quite a number of areas that need to be fixed / cleaned up etc. So at least
the feature freeze is a good thing.
Reading the changelog since 1.6-dev1, in no particular order, I've found :
- DNS-based server name resolution : haproxy is now able to periodically
ask a set of resolvers for the IP address of some servers and to update
them without restarting. This will make life much easier for people
running in AWS where IP address change randomly. Some more stuff was
planned for this such as marking the server as unresolvable if resolving
fails, but we found that people would probably like to have a configurable
behaviour. Feedback on this is desired and will drive the next steps.
- peers protocol v2 : haproxy 1.6 and 1.5 will not be able to synchronize
their stick tables but on the other hand the new protocol is much better
and more extensible. First it uses a single connection regardless of the
number of tables to synchronize. Second it will support synchronizing
much more than just stick tables. For now it replicates all stick-tables
contents (including gpc, etc...). This allows reloads to keep entries,
rates, etc... as well as to pass them to a backup node in case of a
switchover. It's very likely that during 1.7 development we'll further
extend the amount of information that can be exchanged.
- peers support nbproc > 1 as long as they're referenced by a single process,
and peers sections can be disabled (useful for debugging).
- config : removed a few deprecated keywords (eg: "reqsetbe"). I wanted to
remove "block" as well, and appsession. On the first one I'm not sure,
on the second one only Aleks (the author of the feature) provided some
feedback and agreed it was probably time for it to go. Expect that we'd
get rid of them soon if nobody objects.
- pattern cache : a small lru cache applies to pattern matching when it
runs from a list (eg: case insensitive string match, regex, etc). This
can significantly speed up host header matching or regex matching
against a huge list.
- support for stateless zip compression with libslz : this doesn't waste
memory anymore and compresses about 3 times faster than zlib, at a lower
compression ratio.
- support for session/transaction/request/response variables : using the
"set-var" action in {tcp,http}-{request-response} rulesets, it's possible
to assign the result of a sample expression to a variable allocated on the
fly and which lasts for all the session, the transaction or just the
ephemeral processing being done on the request or response. This makes
it possible to keep copies of certain request information and reuse them
in the response for example. Some work is still pending on this part,
in particular the ability to use variables with in all arithmetic
converters which currently only take a constant.
- support for declared captures : sometimes it's desired to capture in
the backend or response path but that was not possible since only the
frontend can assign a capture slot. The solution consists in making
it possible to declare a capture slot in the frontend for later use.
- servers: in addition to DNS, it's possible to change a server's IP address
from the CLI.
- ssl: it's now possible to forge SSL certs on the fly. That's convenient
when haproxy has to be deployed in front of proxies which already work
like this.
- device identification : two companies, 51Degrees and DeviceAtlas,
provided patches to add support for their respective libs. We're
starting to see some demand for such features due to the abundance
of smartphones, tablets and I don't-know-what, and both libs come
with a free device database, so it seems to be the right timing.
The README was updated for both, there you'll find how to build with
either solution (or both, I checked and they don't break each other).
It would be interesting to get feedback on these features, especially
from people who already have access to the full databases and who see
a benefit in moving this processing to haproxy instead of having one
different implementation per application server. More information is
available below for each of them respectively :
https://deviceatlas.com/deviceatlas-haproxy-module
https://github.com/51Degreesmobi/51Degrees-C
- ssl: default DH param groups were replaced with custom ones in order
to limit the exposure in case of a targetted attack.
- config: support for quotes (nor more backslashes needed before spaces),
and stricter control of argument counts so that people who write invalid
configs where words were silently ignored don't get trapped anymore. The
long-deprecated syntax consisting in putting the ip:port on the "listen"
line has now been removed as well since it didn't support any bind option
and used to regularly confuse users.
- config: environment variables can be used everywhere inside double-quotes,
not just in listening addresses.
- stats: the CSV dump now knows how to properly quote strings containing
commas or quotes. This will make it possible to start adding many counters
there (those who are only present in the HTML dump for now).
- http-response now supports "redirect" rules. That's sometimes useful to
replace a 500 server error with a nice page.
- config: duplicated backend names or server names are now completely
detected and better reported so that it's easy to know what needs to be
fixed.
- multiple redispatches are now possible on configurable retry intervals
when connection fails to a server.
- url_param() and body_param() can check for multiple (or any) parameter.
That can be used as a preliminary cleanup for certain invalid requests.
- TLS key loading from file and update on the CLI : this will save some
reloads for some users and provide better security to SSL users.
- "option http-buffer-request" allows request processing to be deferred
until the request body is received, thus it's possible to look up a
routing key in a POST body (eg: user id).
- "option http-ignore-probes" to silent 400/408 on preconnect, and to
avoid counting errors in this case.
- support for HTTP/0.9 is now disabled by default. It's totally useless
and can lead to some security issues by making it easier to forge
requests from foreign protocols. In addition, some extra cleanups to
comply with RFC7230 were applied. "RTSP" is now allowed as a protocol
name for those who want to load-balance RTSP farms (parses like HTTP
for basic needs).
- lua: implemented a simple memory allocator which makes it possible to
limit memory usage.
- lots of internal changes (applets now run independantly from streams,
sample fetch API changed, etc...).
I couldn't complete the response processing changes that I had to interrupt
3 weeks ago to review patches. So most likely this will be postponed to 1.7.
We still have a huge amount of work to do to clean what we have. For example
session variables are still attached to the stream while they need to move
to the session (and the internal variables API must already change for this).
The stick-tables still use old types and we could simplify their code by
moving that to the common sample types (and remove a conversion stage).
We still have pending the patch to retrieve/restore server states across
reloads. It needs more work to improve lookups to better resist to config
changes (otherwise why would people restart?). We realized that the notion
of "state" differs depending on the use case. Some will want to keep only
the up/down status. Others might want to keep the dynamic weights and
anything that was updated on the CLI, while others would probably prefer
to ensure the CLI is dropped upon reloads since the CLI is here to adjust
what can be done without restarting, etc. I hope to be able to merge that
soon so that we can get some feedback about it. It definitely is useful
but we don't know clearly where we want to go with this.
As indicated 3 weeks ago, future changes should have a limited impact
on code stability (unless they fix bugs of course), and on configuration
so that early adopters can quickly update when they face a bug that is
fixed. If you're developing something great and intrusive, please keep
it for when 1.7 opens.
I was told that current version could fail to build on OpenBSD, but there's
a patch floating around for this so hopefully this will be resolved soon.
Last point, very recently I got a request from someone who desired a bit
more signatures in the release process. I don't want to make the whole
workflow a pain, but at least now I've switched to signed tags, which is
easy to do and happens only once in a while.
I'm not appending the changelog, it's too large and boring, really.
Usual URLs below :
Site index : http://www.haproxy.org/
Sources : http://www.haproxy.org/download/1.6/src/devel/
Git repository : http://git.haproxy.org/git/haproxy.git/
Git Web browsing : http://git.haproxy.org/?p=haproxy.git
Changelog : http://www.haproxy.org/download/1.6/src/CHANGELOG
Cyril's HTML doc : http://cbonte.github.com/haproxy-dconv/configuration-1.6.html
Regards,
Willy
the impatient readers among you will have noticed that it's been almost 3
weeks since I sent the e-mail announcing the imminent release of 1.6-dev2.
That end of merge window has been a nightmare and is not finished, but I
thought it would be wise to issue dev2 anyway so that people can test the
stuff that has been merged anyway. Lesson learned, for 1.7 we'll have a
much shorter merge window so that people don't have enough time to push
that much stuff at the last minute :-)
To be honnest, I'm far from being satisfied with this version. It's as huge
as dev1 (344 commits) despite some things still being pending. Also noticed
quite a number of areas that need to be fixed / cleaned up etc. So at least
the feature freeze is a good thing.
Reading the changelog since 1.6-dev1, in no particular order, I've found :
- DNS-based server name resolution : haproxy is now able to periodically
ask a set of resolvers for the IP address of some servers and to update
them without restarting. This will make life much easier for people
running in AWS where IP address change randomly. Some more stuff was
planned for this such as marking the server as unresolvable if resolving
fails, but we found that people would probably like to have a configurable
behaviour. Feedback on this is desired and will drive the next steps.
- peers protocol v2 : haproxy 1.6 and 1.5 will not be able to synchronize
their stick tables but on the other hand the new protocol is much better
and more extensible. First it uses a single connection regardless of the
number of tables to synchronize. Second it will support synchronizing
much more than just stick tables. For now it replicates all stick-tables
contents (including gpc, etc...). This allows reloads to keep entries,
rates, etc... as well as to pass them to a backup node in case of a
switchover. It's very likely that during 1.7 development we'll further
extend the amount of information that can be exchanged.
- peers support nbproc > 1 as long as they're referenced by a single process,
and peers sections can be disabled (useful for debugging).
- config : removed a few deprecated keywords (eg: "reqsetbe"). I wanted to
remove "block" as well, and appsession. On the first one I'm not sure,
on the second one only Aleks (the author of the feature) provided some
feedback and agreed it was probably time for it to go. Expect that we'd
get rid of them soon if nobody objects.
- pattern cache : a small lru cache applies to pattern matching when it
runs from a list (eg: case insensitive string match, regex, etc). This
can significantly speed up host header matching or regex matching
against a huge list.
- support for stateless zip compression with libslz : this doesn't waste
memory anymore and compresses about 3 times faster than zlib, at a lower
compression ratio.
- support for session/transaction/request/response variables : using the
"set-var" action in {tcp,http}-{request-response} rulesets, it's possible
to assign the result of a sample expression to a variable allocated on the
fly and which lasts for all the session, the transaction or just the
ephemeral processing being done on the request or response. This makes
it possible to keep copies of certain request information and reuse them
in the response for example. Some work is still pending on this part,
in particular the ability to use variables with in all arithmetic
converters which currently only take a constant.
- support for declared captures : sometimes it's desired to capture in
the backend or response path but that was not possible since only the
frontend can assign a capture slot. The solution consists in making
it possible to declare a capture slot in the frontend for later use.
- servers: in addition to DNS, it's possible to change a server's IP address
from the CLI.
- ssl: it's now possible to forge SSL certs on the fly. That's convenient
when haproxy has to be deployed in front of proxies which already work
like this.
- device identification : two companies, 51Degrees and DeviceAtlas,
provided patches to add support for their respective libs. We're
starting to see some demand for such features due to the abundance
of smartphones, tablets and I don't-know-what, and both libs come
with a free device database, so it seems to be the right timing.
The README was updated for both, there you'll find how to build with
either solution (or both, I checked and they don't break each other).
It would be interesting to get feedback on these features, especially
from people who already have access to the full databases and who see
a benefit in moving this processing to haproxy instead of having one
different implementation per application server. More information is
available below for each of them respectively :
https://deviceatlas.com/deviceatlas-haproxy-module
https://github.com/51Degreesmobi/51Degrees-C
- ssl: default DH param groups were replaced with custom ones in order
to limit the exposure in case of a targetted attack.
- config: support for quotes (nor more backslashes needed before spaces),
and stricter control of argument counts so that people who write invalid
configs where words were silently ignored don't get trapped anymore. The
long-deprecated syntax consisting in putting the ip:port on the "listen"
line has now been removed as well since it didn't support any bind option
and used to regularly confuse users.
- config: environment variables can be used everywhere inside double-quotes,
not just in listening addresses.
- stats: the CSV dump now knows how to properly quote strings containing
commas or quotes. This will make it possible to start adding many counters
there (those who are only present in the HTML dump for now).
- http-response now supports "redirect" rules. That's sometimes useful to
replace a 500 server error with a nice page.
- config: duplicated backend names or server names are now completely
detected and better reported so that it's easy to know what needs to be
fixed.
- multiple redispatches are now possible on configurable retry intervals
when connection fails to a server.
- url_param() and body_param() can check for multiple (or any) parameter.
That can be used as a preliminary cleanup for certain invalid requests.
- TLS key loading from file and update on the CLI : this will save some
reloads for some users and provide better security to SSL users.
- "option http-buffer-request" allows request processing to be deferred
until the request body is received, thus it's possible to look up a
routing key in a POST body (eg: user id).
- "option http-ignore-probes" to silent 400/408 on preconnect, and to
avoid counting errors in this case.
- support for HTTP/0.9 is now disabled by default. It's totally useless
and can lead to some security issues by making it easier to forge
requests from foreign protocols. In addition, some extra cleanups to
comply with RFC7230 were applied. "RTSP" is now allowed as a protocol
name for those who want to load-balance RTSP farms (parses like HTTP
for basic needs).
- lua: implemented a simple memory allocator which makes it possible to
limit memory usage.
- lots of internal changes (applets now run independantly from streams,
sample fetch API changed, etc...).
I couldn't complete the response processing changes that I had to interrupt
3 weeks ago to review patches. So most likely this will be postponed to 1.7.
We still have a huge amount of work to do to clean what we have. For example
session variables are still attached to the stream while they need to move
to the session (and the internal variables API must already change for this).
The stick-tables still use old types and we could simplify their code by
moving that to the common sample types (and remove a conversion stage).
We still have pending the patch to retrieve/restore server states across
reloads. It needs more work to improve lookups to better resist to config
changes (otherwise why would people restart?). We realized that the notion
of "state" differs depending on the use case. Some will want to keep only
the up/down status. Others might want to keep the dynamic weights and
anything that was updated on the CLI, while others would probably prefer
to ensure the CLI is dropped upon reloads since the CLI is here to adjust
what can be done without restarting, etc. I hope to be able to merge that
soon so that we can get some feedback about it. It definitely is useful
but we don't know clearly where we want to go with this.
As indicated 3 weeks ago, future changes should have a limited impact
on code stability (unless they fix bugs of course), and on configuration
so that early adopters can quickly update when they face a bug that is
fixed. If you're developing something great and intrusive, please keep
it for when 1.7 opens.
I was told that current version could fail to build on OpenBSD, but there's
a patch floating around for this so hopefully this will be resolved soon.
Last point, very recently I got a request from someone who desired a bit
more signatures in the release process. I don't want to make the whole
workflow a pain, but at least now I've switched to signed tags, which is
easy to do and happens only once in a while.
I'm not appending the changelog, it's too large and boring, really.
Usual URLs below :
Site index : http://www.haproxy.org/
Sources : http://www.haproxy.org/download/1.6/src/devel/
Git repository : http://git.haproxy.org/git/haproxy.git/
Git Web browsing : http://git.haproxy.org/?p=haproxy.git
Changelog : http://www.haproxy.org/download/1.6/src/CHANGELOG
Cyril's HTML doc : http://cbonte.github.com/haproxy-dconv/configuration-1.6.html
Regards,
Willy