3) Backend server B - 10.0.138.59
- on this backend, we are dropping packets from reverse proxy server:
# iptables -A INPUT -s 10.0.137.13 -j DROP
- perl script - /var/www/cgi-bin/test.cgi
#!/usr/bin/perl
print "Content-type: text/plain\r\n";
print "\r\n";
read (STDIN, $data, $ENV{'CONTENT_LENGTH'});
- when reverse proxy selects backend server B, it fails to connect to it and then fail over to backend server A. But when I collect network traffic on backend server B, invalid Content-Length is set. Network data in case of no fail-over:
HTTP/1.1 200 OK
Date: Tue, 03 Mar 2020 12:24:18 GMT
Server: Apache/2.4.41 (Fedora)
Connection: close
Content-Type: text/plain; charset=UTF-8
"
I don't know mod_proxy very well, but I did some debugging. When reverse proxy server tries to contact backend server B, it executes ap_proxy_http_prefetch function, reads data from r->input_filters, sets temp_brigade, then stores temp_brigade into input_brigade and leaves function. Then on fail-over, it executes ap_proxy_http_prefetch function again, but no data are read and C-L is set to 0.
1) run, without fail-over (backend server A is contacted first):
- in ap_proxy_http_prefetch function is called just once:
- input_brigade:
0 | HEAP (0x7fffc80012c8) | 5 | 0x7fffc8001188 | [abcd~] | 1
1 | EOS (0x7fffc8001e08) | 0 | 0x00000000 | | n/a
- cl_val is set to 5, old_cl_val is set to "5"
2) run, with fail-over (backend server B is contacted first, then it fails-over on backend server A)
- first call of ap_proxy_http_prefetch function is same as in case of run without fail-over
- second ap_proxy_http_prefetch call:
- no data are read by ap_get_brigade(r->input_filters, temp_brigade,...
- input brigade contains just EOS
- cal_val is equal to 0, old_cl_val is set to "5", but it is afterwards overwritten:
ap_proxy_http_prefetch (url=<optimized out>, uri=0x7fffc8010738, req=0x7fffc80106a8) at mod_proxy_http.c:743
743 if (req->old_cl_val || req->old_te_val || bytes_read) {
(gdb) n
744 req->old_cl_val = apr_off_t_toa(r->pool, bytes_read);
backtrace of second call:
(gdb) bt
#0 ap_proxy_http_prefetch (url=<optimized out>, uri=0x7fffc8010738, req=0x7fffc80106a8) at mod_proxy_http.c:747
#1 proxy_http_handler (r=0x7fffc8013820, worker=<optimized out>, conf=<optimized out>, url=0x7fffc8010678 "http://10.0.138.138/cgi-bin/test.cgi", proxyname=<optimized out>,
proxyport=<optimized out>) at mod_proxy_http.c:2063
#2 0x00007ffff74d1a2b in proxy_run_scheme_handler (r=r@entry=0x7fffc8013820, worker=0x55555568e8b8, conf=conf@entry=0x555555656560,
url=0x7fffc8010678 "http://10.0.138.138/cgi-bin/test.cgi", proxyhost=proxyhost@entry=0x0, proxyport=proxyport@entry=0) at mod_proxy.c:3088
#3 0x00007ffff74d27ee in proxy_handler (r=0x7fffc8013820) at mod_proxy.c:1257
#4 0x000055555556db98 in ap_run_handler (r=r@entry=0x7fffc8013820) at config.c:170
#5 0x000055555556e146 in ap_invoke_handler (r=r@entry=0x7fffc8013820) at config.c:444
#6 0x00005555555a9e23 in ap_process_async_request (r=0x7fffc8013820) at http_request.c:453
#7 0x00005555555a61f3 in ap_process_http_async_connection (c=0x7fffb8001738) at http_core.c:158
#8 ap_process_http_connection (c=0x7fffb8001738) at http_core.c:252
#9 0x000055555557f048 in ap_run_process_connection (c=c@entry=0x7fffb8001738) at connection.c:42
#10 0x00007ffff74f10e9 in process_socket (thd=<optimized out>, p=<optimized out>, sock=<optimized out>, cs=0x7fffb8001690, my_child_num=<optimized out>, my_thread_num=<optimized out>)
at event.c:1050
#11 0x00007ffff74f1a87 in worker_thread (thd=0x5555557384f8, dummy=<optimized out>) at event.c:2084
#12 0x00007ffff7d854c0 in start_thread () from /lib64/libpthread.so.0
#13 0x00007ffff7cad163 in clone () from /lib64/libc.so.6
Hi all,
I'm experiencing one more issue (I'm not sure, if it is related to this bug or not).
There is following configuration:
1) reverse proxy server with configuration: 10.0.138. 138:80 ping=5 10.0.138. 59:80 ping=5
ProxyPass /sample/ balancer://cluster/
<Proxy balancer://cluster>
BalancerMember http://
BalancerMember http://
</Proxy>
2) Backend server A - 10.0.138.138 cgi-bin/ test.cgi LENGTH' });
- perl script - /var/www/
#!/usr/bin/perl
print "Content-type: text/plain\r\n";
print "\r\n";
read (STDIN, $data, $ENV{'CONTENT_
3) Backend server B - 10.0.138.59 cgi-bin/ test.cgi LENGTH' });
- on this backend, we are dropping packets from reverse proxy server:
# iptables -A INPUT -s 10.0.137.13 -j DROP
- perl script - /var/www/
#!/usr/bin/perl
print "Content-type: text/plain\r\n";
print "\r\n";
read (STDIN, $data, $ENV{'CONTENT_
4) client 10.0.137. 13/sample/ cgi-bin/ test.cgi
# echo "abcd" > /tmp/data
# curl -X POST --data-binary '@/tmp/data' http://
- when reverse proxy selects backend server B, it fails to connect to it and then fail over to backend server A. But when I collect network traffic on backend server B, invalid Content-Length is set. Network data in case of no fail-over:
"POST /cgi-bin/test.cgi HTTP/1.1 x-www-form- urlencoded 3eff:fe62: 246
Host: 10.0.138.138
User-Agent: curl/7.65.3
Accept: */*
Content-Type: application/
Expect: 100-Continue
X-Forwarded-For: 10.0.136.27
X-Forwarded-Host: 10.0.137.13
X-Forwarded-Server: fe80::f816:
Content-Length: 5
Connection: Keep-Alive
HTTP/1.1 100 Continue
abcd
HTTP/1.1 200 OK
Date: Tue, 03 Mar 2020 12:23:15 GMT
Server: Apache/2.4.41 (Fedora)
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/plain; charset=UTF-8
"
vs. network data when there was a fail-over:
"POST /cgi-bin/test.cgi HTTP/1.1 x-www-form- urlencoded 3eff:fe62: 246
Host: 10.0.138.138
User-Agent: curl/7.65.3
Accept: */*
Content-Type: application/
Expect: 100-Continue
X-Forwarded-For: 10.0.136.27
X-Forwarded-Host: 10.0.137.13
X-Forwarded-Server: fe80::f816:
Content-Length: 0
Connection: Keep-Alive
HTTP/1.1 200 OK
Date: Tue, 03 Mar 2020 12:24:18 GMT
Server: Apache/2.4.41 (Fedora)
Connection: close
Content-Type: text/plain; charset=UTF-8
"
I don't know mod_proxy very well, but I did some debugging. When reverse proxy server tries to contact backend server B, it executes ap_proxy_ http_prefetch function, reads data from r->input_filters, sets temp_brigade, then stores temp_brigade into input_brigade and leaves function. Then on fail-over, it executes ap_proxy_ http_prefetch function again, but no data are read and C-L is set to 0.
1) run, without fail-over (backend server A is contacted first): http_prefetch function is called just once:
- in ap_proxy_
- input_brigade:
0 | HEAP (0x7fffc80012c8) | 5 | 0x7fffc8001188 | [abcd~] | 1
1 | EOS (0x7fffc8001e08) | 0 | 0x00000000 | | n/a
- cl_val is set to 5, old_cl_val is set to "5"
2) run, with fail-over (backend server B is contacted first, then it fails-over on backend server A) http_prefetch function is same as in case of run without fail-over http_prefetch call: brigade( r->input_ filters, temp_brigade,... http_prefetch (url=<optimized out>, uri=0x7fffc8010738, req=0x7fffc80106a8) at mod_proxy_ http.c: 743 t_toa(r- >pool, bytes_read);
- first call of ap_proxy_
- second ap_proxy_
- no data are read by ap_get_
- input brigade contains just EOS
- cal_val is equal to 0, old_cl_val is set to "5", but it is afterwards overwritten:
ap_proxy_
743 if (req->old_cl_val || req->old_te_val || bytes_read) {
(gdb) n
744 req->old_cl_val = apr_off_
backtrace of second call: http_prefetch (url=<optimized out>, uri=0x7fffc8010738, req=0x7fffc80106a8) at mod_proxy_ http.c: 747 10.0.138. 138/cgi- bin/test. cgi", proxyname= <optimized out>, <optimized out>) at mod_proxy_ http.c: 2063 scheme_ handler (r=r@entry= 0x7fffc8013820, worker= 0x55555568e8b8, conf=conf@ entry=0x5555556 56560, 0x7fffc8010678 "http:// 10.0.138. 138/cgi- bin/test. cgi", proxyhost= proxyhost@ entry=0x0, proxyport= proxyport@ entry=0) at mod_proxy.c:3088 0x7fffc8013820) at config.c:170 0x7fffc8013820) at config.c:444 async_request (r=0x7fffc8013820) at http_request.c:453 http_async_ connection (c=0x7fffb8001738) at http_core.c:158 http_connection (c=0x7fffb8001738) at http_core.c:252 process_ connection (c=c@entry= 0x7fffb8001738) at connection.c:42 num=<optimized out>, my_thread_ num=<optimized out>) 84f8, dummy=<optimized out>) at event.c:2084 libpthread. so.0
(gdb) bt
#0 ap_proxy_
#1 proxy_http_handler (r=0x7fffc8013820, worker=<optimized out>, conf=<optimized out>, url=0x7fffc8010678 "http://
proxyport=
#2 0x00007ffff74d1a2b in proxy_run_
url=
#3 0x00007ffff74d27ee in proxy_handler (r=0x7fffc8013820) at mod_proxy.c:1257
#4 0x000055555556db98 in ap_run_handler (r=r@entry=
#5 0x000055555556e146 in ap_invoke_handler (r=r@entry=
#6 0x00005555555a9e23 in ap_process_
#7 0x00005555555a61f3 in ap_process_
#8 ap_process_
#9 0x000055555557f048 in ap_run_
#10 0x00007ffff74f10e9 in process_socket (thd=<optimized out>, p=<optimized out>, sock=<optimized out>, cs=0x7fffb8001690, my_child_
at event.c:1050
#11 0x00007ffff74f1a87 in worker_thread (thd=0x55555573
#12 0x00007ffff7d854c0 in start_thread () from /lib64/
#13 0x00007ffff7cad163 in clone () from /lib64/libc.so.6