Comment 55 for bug 1641238

Hi all,

I'm experiencing one more issue (I'm not sure, if it is related to this bug or not).

There is following configuration:

1) reverse proxy server with configuration:
ProxyPass /sample/ balancer://cluster/
<Proxy balancer://cluster>
  BalancerMember http://10.0.138.138:80 ping=5
  BalancerMember http://10.0.138.59:80 ping=5
</Proxy>

2) Backend server A - 10.0.138.138
- perl script - /var/www/cgi-bin/test.cgi
#!/usr/bin/perl
print "Content-type: text/plain\r\n";
print "\r\n";
read (STDIN, $data, $ENV{'CONTENT_LENGTH'});

3) Backend server B - 10.0.138.59
- on this backend, we are dropping packets from reverse proxy server:
  # iptables -A INPUT -s 10.0.137.13 -j DROP
- perl script - /var/www/cgi-bin/test.cgi
#!/usr/bin/perl
print "Content-type: text/plain\r\n";
print "\r\n";
read (STDIN, $data, $ENV{'CONTENT_LENGTH'});

4) client
# echo "abcd" > /tmp/data
# curl -X POST --data-binary '@/tmp/data' http://10.0.137.13/sample/cgi-bin/test.cgi

- when reverse proxy selects backend server B, it fails to connect to it and then fail over to backend server A. But when I collect network traffic on backend server B, invalid Content-Length is set. Network data in case of no fail-over:

"POST /cgi-bin/test.cgi HTTP/1.1
Host: 10.0.138.138
User-Agent: curl/7.65.3
Accept: */*
Content-Type: application/x-www-form-urlencoded
Expect: 100-Continue
X-Forwarded-For: 10.0.136.27
X-Forwarded-Host: 10.0.137.13
X-Forwarded-Server: fe80::f816:3eff:fe62:246
Content-Length: 5
Connection: Keep-Alive

HTTP/1.1 100 Continue

abcd
HTTP/1.1 200 OK
Date: Tue, 03 Mar 2020 12:23:15 GMT
Server: Apache/2.4.41 (Fedora)
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/plain; charset=UTF-8
"

vs. network data when there was a fail-over:

"POST /cgi-bin/test.cgi HTTP/1.1
Host: 10.0.138.138
User-Agent: curl/7.65.3
Accept: */*
Content-Type: application/x-www-form-urlencoded
Expect: 100-Continue
X-Forwarded-For: 10.0.136.27
X-Forwarded-Host: 10.0.137.13
X-Forwarded-Server: fe80::f816:3eff:fe62:246
Content-Length: 0
Connection: Keep-Alive

HTTP/1.1 200 OK
Date: Tue, 03 Mar 2020 12:24:18 GMT
Server: Apache/2.4.41 (Fedora)
Connection: close
Content-Type: text/plain; charset=UTF-8
"

I don't know mod_proxy very well, but I did some debugging. When reverse proxy server tries to contact backend server B, it executes ap_proxy_http_prefetch function, reads data from r->input_filters, sets temp_brigade, then stores temp_brigade into input_brigade and leaves function. Then on fail-over, it executes ap_proxy_http_prefetch function again, but no data are read and C-L is set to 0.

1) run, without fail-over (backend server A is contacted first):
- in ap_proxy_http_prefetch function is called just once:
   - input_brigade:
    0 | HEAP (0x7fffc80012c8) | 5 | 0x7fffc8001188 | [abcd~] | 1
    1 | EOS (0x7fffc8001e08) | 0 | 0x00000000 | | n/a
   - cl_val is set to 5, old_cl_val is set to "5"

2) run, with fail-over (backend server B is contacted first, then it fails-over on backend server A)
- first call of ap_proxy_http_prefetch function is same as in case of run without fail-over
- second ap_proxy_http_prefetch call:
   - no data are read by ap_get_brigade(r->input_filters, temp_brigade,...
   - input brigade contains just EOS
   - cal_val is equal to 0, old_cl_val is set to "5", but it is afterwards overwritten:
ap_proxy_http_prefetch (url=<optimized out>, uri=0x7fffc8010738, req=0x7fffc80106a8) at mod_proxy_http.c:743
743 if (req->old_cl_val || req->old_te_val || bytes_read) {
(gdb) n
744 req->old_cl_val = apr_off_t_toa(r->pool, bytes_read);

backtrace of second call:
(gdb) bt
#0 ap_proxy_http_prefetch (url=<optimized out>, uri=0x7fffc8010738, req=0x7fffc80106a8) at mod_proxy_http.c:747
#1 proxy_http_handler (r=0x7fffc8013820, worker=<optimized out>, conf=<optimized out>, url=0x7fffc8010678 "http://10.0.138.138/cgi-bin/test.cgi", proxyname=<optimized out>,
    proxyport=<optimized out>) at mod_proxy_http.c:2063
#2 0x00007ffff74d1a2b in proxy_run_scheme_handler (r=r@entry=0x7fffc8013820, worker=0x55555568e8b8, conf=conf@entry=0x555555656560,
    url=0x7fffc8010678 "http://10.0.138.138/cgi-bin/test.cgi", proxyhost=proxyhost@entry=0x0, proxyport=proxyport@entry=0) at mod_proxy.c:3088
#3 0x00007ffff74d27ee in proxy_handler (r=0x7fffc8013820) at mod_proxy.c:1257
#4 0x000055555556db98 in ap_run_handler (r=r@entry=0x7fffc8013820) at config.c:170
#5 0x000055555556e146 in ap_invoke_handler (r=r@entry=0x7fffc8013820) at config.c:444
#6 0x00005555555a9e23 in ap_process_async_request (r=0x7fffc8013820) at http_request.c:453
#7 0x00005555555a61f3 in ap_process_http_async_connection (c=0x7fffb8001738) at http_core.c:158
#8 ap_process_http_connection (c=0x7fffb8001738) at http_core.c:252
#9 0x000055555557f048 in ap_run_process_connection (c=c@entry=0x7fffb8001738) at connection.c:42
#10 0x00007ffff74f10e9 in process_socket (thd=<optimized out>, p=<optimized out>, sock=<optimized out>, cs=0x7fffb8001690, my_child_num=<optimized out>, my_thread_num=<optimized out>)
    at event.c:1050
#11 0x00007ffff74f1a87 in worker_thread (thd=0x5555557384f8, dummy=<optimized out>) at event.c:2084
#12 0x00007ffff7d854c0 in start_thread () from /lib64/libpthread.so.0
#13 0x00007ffff7cad163 in clone () from /lib64/libc.so.6