Hi,
I have some linux boxes with very old kernels. Unfortunately, I cannot
upgrade them due to the fact that they work very stable. for
example,their uptime is already some
years, which is not true speaking about modern kernels.
But there is one problem: HAPproxy hangs when I turn on SSL options.
# haproxy -v
HA-Proxy version 1.5.4 2014/09/02
My config:
global
tune.ssl.default-dh-param 2048
ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK
frontend https-in
bind 111.222.111.222:443 ssl strict-sni no-sslv3 crt-list /etc/haproxy_aux2_pools/crt.list
errorfile 408 /dev/null
option http-keep-alive
option http-server-close
http-request add-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https
use_backend apache_aux2_workers
# ps -o s,pid,start,comm -C haproxy_aux2_pools
S PID STARTED COMMAND
D 472 Apr 07 haproxy_aux2_po
D 725 Apr 07 haproxy_aux2_po
D 1185 Apr 07 haproxy_aux2_po
D 1706 Apr 07 haproxy_aux2_po
D 2168 Apr 07 haproxy_aux2_po
D 2749 Apr 07 haproxy_aux2_po
D 2996 Apr 07 haproxy_aux2_po
D 3620 Apr 07 haproxy_aux2_po
D 3960 Apr 07 haproxy_aux2_po
and kernel trace:
Apr 7 17:40:23 l4 kernel: Unable to handle kernel paging request at fffffffffffffff4 RIP:
Apr 7 17:40:23 l4 kernel: [<ffffffff8047f770>] dma_unpin_iovec_pages+0x10/0x80
Apr 7 17:40:23 l4 kernel: PGD 203067 PUD 204067 PMD 0
Apr 7 17:40:23 l4 kernel: Oops: 0000 [1] SMP
Apr 7 17:40:23 l4 kernel: CPU 0
Apr 7 17:40:23 l4 kernel: Pid: 17747, comm: haproxy_aux2_po Not tainted 2.6.24-1gb-1 #4
Apr 7 17:40:23 l4 kernel: RIP: 0010:[<ffffffff8047f770>] [<ffffffff8047f770>] dma_unpin_iovec_pages+0x10/0x80
Apr 7 17:40:23 l4 kernel: RSP: 0018:ffff8101164dbbb8 EFLAGS: 00010282
Apr 7 17:40:23 l4 kernel: RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000
Apr 7 17:40:23 l4 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: fffffffffffffff4
Apr 7 17:40:23 l4 kernel: RBP: ffff8102acf5c6b0 R08: 0000000000000040 R09: 0000000000000000
Apr 7 17:40:23 l4 kernel: R10: ffffffff80629900 R11: ffffffff80398920 R12: ffff8102acf5c600
Apr 7 17:40:23 l4 kernel: R13: ffff8102acf5c6b0 R14: fffffffffffffff4 R15: 000000007fffffff
Apr 7 17:40:23 l4 kernel: FS: 00002b5d03469b20(0000) GS:ffffffff8062f000(0000) knlGS:0000000000000000
Apr 7 17:40:23 l4 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 7 17:40:23 l4 kernel: CR2: fffffffffffffff4 CR3: 00000001c50f2000 CR4: 00000000000006e0
Apr 7 17:40:23 l4 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 7 17:40:23 l4 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Apr 7 17:40:23 l4 kernel: Process haproxy_aux2_po (pid: 17747, threadinfo ffff8101164da000, task ffff8101b733a000)
Apr 7 17:40:23 l4 kernel: Stack: 0000000000000000 ffff8102acf5c6b0 ffff8102acf5c600 ffff8102acf5c6b0
Apr 7 17:40:23 l4 kernel: ffff8102acf5c9dc ffffffff804e2f11 ffff810010535900 ffffffff804e1a53
Apr 7 17:40:23 l4 kernel: 0000000000000000 0000402000000000 ffff8101164dbee8 0000000007524a80
Apr 7 17:40:23 l4 kernel: Call Trace:
Apr 7 17:40:23 l4 kernel: Call Trace:
Apr 7 17:40:23 l4 kernel: [<ffffffff804e2f11>] tcp_recvmsg+0x581/0xcd0
Apr 7 17:40:23 l4 kernel: [<ffffffff804e1a53>] tcp_sendmsg+0x593/0xc30
Apr 7 17:40:23 l4 kernel: [<ffffffff8052d719>] _spin_lock_bh+0x9/0x20
Apr 7 17:40:23 l4 kernel: [<ffffffff804885c3>] release_sock+0x13/0xb0
Apr 7 17:40:23 l4 kernel: [<ffffffff80487e30>] sock_common_recvmsg+0x30/0x50
Apr 7 17:40:23 l4 kernel: [<ffffffff804860ca>] sock_recvmsg+0x14a/0x160
Apr 7 17:40:23 l4 kernel: [<ffffffff8025e0ae>] filemap_fault+0x21e/0x420
Apr 7 17:40:23 l4 kernel: [<ffffffff80247440>] autoremove_wake_function+0x0/0x30
Apr 7 17:40:23 l4 kernel: [<ffffffff80269665>] __do_fault+0x1e5/0x460
Apr 7 17:40:23 l4 kernel: [<ffffffff8026b22f>] handle_mm_fault+0x1af/0x7c0
Apr 7 17:40:23 l4 kernel: [<ffffffff8048728e>] sys_recvfrom+0xfe/0x1a0
Apr 7 17:40:23 l4 kernel: [<ffffffff8021f450>] do_page_fault+0x1e0/0x830
Apr 7 17:40:23 l4 kernel: [<ffffffff8026f851>] vma_merge+0x161/0x1f0
Apr 7 17:40:23 l4 kernel: [<ffffffff8020c21e>] system_call+0x7e/0x83
Apr 7 17:40:23 l4 kernel:
Apr 7 17:40:23 l4 kernel:
Apr 7 17:40:23 l4 kernel: Code: 8b 37 85 f6 7e 51 48 8d 6f 08 45 31 ed 0f 1f 00 8b 4d 08 85
Apr 7 17:40:23 l4 kernel: RIP [<ffffffff8047f770>] dma_unpin_iovec_pages+0x10/0x80
Apr 7 17:40:23 l4 kernel: RSP <ffff8101164dbbb8>
Apr 7 17:40:23 l4 kernel: CR2: fffffffffffffff4
Apr 7 17:40:23 l4 kernel: ---[ end trace e1ec26f01a394080 ]---
Can it be fixed in haproxy? Or it can only be solved by kernel updating?
Thanks for help.
I have some linux boxes with very old kernels. Unfortunately, I cannot
upgrade them due to the fact that they work very stable. for
example,their uptime is already some
years, which is not true speaking about modern kernels.
But there is one problem: HAPproxy hangs when I turn on SSL options.
# haproxy -v
HA-Proxy version 1.5.4 2014/09/02
My config:
global
tune.ssl.default-dh-param 2048
ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK
frontend https-in
bind 111.222.111.222:443 ssl strict-sni no-sslv3 crt-list /etc/haproxy_aux2_pools/crt.list
errorfile 408 /dev/null
option http-keep-alive
option http-server-close
http-request add-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https
use_backend apache_aux2_workers
# ps -o s,pid,start,comm -C haproxy_aux2_pools
S PID STARTED COMMAND
D 472 Apr 07 haproxy_aux2_po
D 725 Apr 07 haproxy_aux2_po
D 1185 Apr 07 haproxy_aux2_po
D 1706 Apr 07 haproxy_aux2_po
D 2168 Apr 07 haproxy_aux2_po
D 2749 Apr 07 haproxy_aux2_po
D 2996 Apr 07 haproxy_aux2_po
D 3620 Apr 07 haproxy_aux2_po
D 3960 Apr 07 haproxy_aux2_po
and kernel trace:
Apr 7 17:40:23 l4 kernel: Unable to handle kernel paging request at fffffffffffffff4 RIP:
Apr 7 17:40:23 l4 kernel: [<ffffffff8047f770>] dma_unpin_iovec_pages+0x10/0x80
Apr 7 17:40:23 l4 kernel: PGD 203067 PUD 204067 PMD 0
Apr 7 17:40:23 l4 kernel: Oops: 0000 [1] SMP
Apr 7 17:40:23 l4 kernel: CPU 0
Apr 7 17:40:23 l4 kernel: Pid: 17747, comm: haproxy_aux2_po Not tainted 2.6.24-1gb-1 #4
Apr 7 17:40:23 l4 kernel: RIP: 0010:[<ffffffff8047f770>] [<ffffffff8047f770>] dma_unpin_iovec_pages+0x10/0x80
Apr 7 17:40:23 l4 kernel: RSP: 0018:ffff8101164dbbb8 EFLAGS: 00010282
Apr 7 17:40:23 l4 kernel: RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000
Apr 7 17:40:23 l4 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: fffffffffffffff4
Apr 7 17:40:23 l4 kernel: RBP: ffff8102acf5c6b0 R08: 0000000000000040 R09: 0000000000000000
Apr 7 17:40:23 l4 kernel: R10: ffffffff80629900 R11: ffffffff80398920 R12: ffff8102acf5c600
Apr 7 17:40:23 l4 kernel: R13: ffff8102acf5c6b0 R14: fffffffffffffff4 R15: 000000007fffffff
Apr 7 17:40:23 l4 kernel: FS: 00002b5d03469b20(0000) GS:ffffffff8062f000(0000) knlGS:0000000000000000
Apr 7 17:40:23 l4 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 7 17:40:23 l4 kernel: CR2: fffffffffffffff4 CR3: 00000001c50f2000 CR4: 00000000000006e0
Apr 7 17:40:23 l4 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 7 17:40:23 l4 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Apr 7 17:40:23 l4 kernel: Process haproxy_aux2_po (pid: 17747, threadinfo ffff8101164da000, task ffff8101b733a000)
Apr 7 17:40:23 l4 kernel: Stack: 0000000000000000 ffff8102acf5c6b0 ffff8102acf5c600 ffff8102acf5c6b0
Apr 7 17:40:23 l4 kernel: ffff8102acf5c9dc ffffffff804e2f11 ffff810010535900 ffffffff804e1a53
Apr 7 17:40:23 l4 kernel: 0000000000000000 0000402000000000 ffff8101164dbee8 0000000007524a80
Apr 7 17:40:23 l4 kernel: Call Trace:
Apr 7 17:40:23 l4 kernel: Call Trace:
Apr 7 17:40:23 l4 kernel: [<ffffffff804e2f11>] tcp_recvmsg+0x581/0xcd0
Apr 7 17:40:23 l4 kernel: [<ffffffff804e1a53>] tcp_sendmsg+0x593/0xc30
Apr 7 17:40:23 l4 kernel: [<ffffffff8052d719>] _spin_lock_bh+0x9/0x20
Apr 7 17:40:23 l4 kernel: [<ffffffff804885c3>] release_sock+0x13/0xb0
Apr 7 17:40:23 l4 kernel: [<ffffffff80487e30>] sock_common_recvmsg+0x30/0x50
Apr 7 17:40:23 l4 kernel: [<ffffffff804860ca>] sock_recvmsg+0x14a/0x160
Apr 7 17:40:23 l4 kernel: [<ffffffff8025e0ae>] filemap_fault+0x21e/0x420
Apr 7 17:40:23 l4 kernel: [<ffffffff80247440>] autoremove_wake_function+0x0/0x30
Apr 7 17:40:23 l4 kernel: [<ffffffff80269665>] __do_fault+0x1e5/0x460
Apr 7 17:40:23 l4 kernel: [<ffffffff8026b22f>] handle_mm_fault+0x1af/0x7c0
Apr 7 17:40:23 l4 kernel: [<ffffffff8048728e>] sys_recvfrom+0xfe/0x1a0
Apr 7 17:40:23 l4 kernel: [<ffffffff8021f450>] do_page_fault+0x1e0/0x830
Apr 7 17:40:23 l4 kernel: [<ffffffff8026f851>] vma_merge+0x161/0x1f0
Apr 7 17:40:23 l4 kernel: [<ffffffff8020c21e>] system_call+0x7e/0x83
Apr 7 17:40:23 l4 kernel:
Apr 7 17:40:23 l4 kernel:
Apr 7 17:40:23 l4 kernel: Code: 8b 37 85 f6 7e 51 48 8d 6f 08 45 31 ed 0f 1f 00 8b 4d 08 85
Apr 7 17:40:23 l4 kernel: RIP [<ffffffff8047f770>] dma_unpin_iovec_pages+0x10/0x80
Apr 7 17:40:23 l4 kernel: RSP <ffff8101164dbbb8>
Apr 7 17:40:23 l4 kernel: CR2: fffffffffffffff4
Apr 7 17:40:23 l4 kernel: ---[ end trace e1ec26f01a394080 ]---
Can it be fixed in haproxy? Or it can only be solved by kernel updating?
Thanks for help.