gpc0_rate computing incorrectly with peer replication turned in [in 1.6.3] (3 replies)

We use a gpc0 counter for rate-limiting certain requests in our
application. It was working fine with 1.5.14, but as soon as I upgraded to
1.6.3, we started seeing the gpc0_rate value go crazy – it's currently
showing values in the hundreds of thousands when the underlying gpc0
counter has *never* been incremented (and has a value of 0). This only
occurs if I have peer replication turned on for the relevant stick table --
disabling peer replication and resetting the table puts everything back to
normal.

Some examples from the `show table` command on the relevant table (with
sensitive data x'd out):

0x305ef9c: key=Bearer\ xxxxxxxxxxxxxxxxxxxx use=0 exp=294451 gpc0=0
gpc0_rate(300000)=2281 http_req_rate(10000)=7
0x306058c: key=Bearer\ xxxxxxxxxxxxxxxxxxxx use=0 exp=299766 gpc0=0
gpc0_rate(300000)=2737 http_req_rate(10000)=18
0x30615cc: key=Bearer\ xxxxxxxxxxxxxxxxxxxx use=1 exp=298819 gpc0=0
gpc0_rate(300000)=3285 http_req_rate(10000)=2
0x306101c: key=Bearer\ xxxxxxxxxxxxxxxxxxxx use=0 exp=296676 gpc0=0
gpc0_rate(300000)=3959 http_req_rate(10000)=1
0x305edfc: key=Bearer\ xxxxxxxxxxxxxxxxxxxx use=0 exp=258288 gpc0=0
gpc0_rate(300000)=5190 http_req_rate(10000)=0
0x305df5c: key=Bearer\ xxxxxxxxxxxxxxxxxxxx use=0 exp=254686 gpc0=0
gpc0_rate(300000)=5936 http_req_rate(10000)=0
0x321a2dc: key=Bearer\ xxxxxxxxxxxxxxxxxxxx use=0 exp=136673 gpc0=0
gpc0_rate(300000)=22826 http_req_rate(10000)=0
0x3061aac: key=Bearer\ xxxxxxxxxxxxxxxxxxxx use=0 exp=299854 gpc0=0
gpc0_rate(300000)=261368 http_req_rate(10000)=2
0x305fafc: key=Bearer\ xxxxxxxxxxxxxxxxxxxx use=0 exp=299041 gpc0=0
gpc0_rate(300000)=262366 http_req_rate(10000)=1
0x306183c: key=Bearer\ xxxxxxxxxxxxxxxxxxxx use=0 exp=226854 gpc0=0
gpc0_rate(300000)=299375 http_req_rate(10000)=0
0x3060cdc: key=Bearer\ xxxxxxxxxxxxxxxxxxxx use=0 exp=281845 gpc0=0
gpc0_rate(300000)=489641 http_req_rate(10000)=0


​And the relevant config snipped (again, somewhat trimmed for this
audience):

frontend https_easypost_com
mode http

# [... lots of config snipped here ...]

# rate-limiting by auth code
tcp-request inspect-delay 4s

tcp-request content track-sc0 req.hdr(Authorization)

stick-table type string len 32 size 512 expire 5m store
gpc0,gpc0_rate(5m),http_req_rate(10s) peers lbsj

# Limit: 200 r/s/client
acl high_rate sc0_http_req_rate gt 2000
acl force_rate_limit req.hdr(X-Force-Status) eq rate_limit
acl force_flooding req.hdr(X-Force-Status) eq flooding

acl punish sc0_inc_gpc0 ge 0
acl is_403 status 403
acl is_402 status 402

# allow no more than 10 401's or 402's every 5 minutes from a given
client
acl flooding4XXs sc0_gpc0_rate ge 10

http-response allow if is_402 punish
http-response allow if is_403 punish

use_backend four_twenty_nine_bad_status if flooding4XXs or
force_flooding

default_backend easypost_be


​The rate-limiting based on the http_req_rate counter seems fine (and none
of the values in the table look nuts).

I've been trying to reproduce this in our development environment with no
luck (even when running with peer replication enabled)​, but within seconds
of issuing a `clear table` in production, there are gpc0_rate values in the
hundreds of thousands.

​Has anyone seen any weird behavior like this?​

--
James Brown
Engineer