Proxy setup masks client IP needed by osrf-http-translator

Bug #1684970 reported by Bill Erickson on 2017-04-20
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenSRF
Medium
Unassigned
3.0
Medium
Unassigned
3.1
Medium
Unassigned

Bug Description

OpenSRF 2.5

Using an nginx proxy (and likely haproxy, though unconfirmed) results in the osrf-http-translator module seeing the client IP address as that of the proxy instead of the true client. This makes it possible, though still very unlikely given that jabber IDs are randomized, for 2 clients to attempt to talk to the same opensrf worker process.

What's worse, though, the IP address reported by Apache for the connecting proxy (confirmed with nginx) toggles between the IPv4 address and the IPv6 address, preventing the translator from maintaining long-lived transactions (e.g. evergreen pcrud transactions), since the IP of the client must be consistent with every call within the transaction.

These may not be identical problems, but they have identical solutions, which is to extract and use the client IP address from the headers provided by the proxy and use those as the client IP address.

To confirm the bug in Evergreen land, configure the proxy for use, then try to create an Acquisitions provider. For me, it fails every time. Regardless of whether if fails, though, it will report 127.0.0.1 (or similar) as the client IP in the activity logs, even for remote clients (which most are).

Note this does not affect WebSockets or the JSON Gateway -- only the translator.

One solution that I have confirmed works in Apache 2.4:

1. apt-get install libapache2-mod-rpaf

2. Add to Apache configuration:

<IfModule mod_rpaf.c>
    RPAFenable On
    RPAFsethostname On
    # Extract IP's from any proxy in 127.* -- change to taste.
    RPAFproxy_ips 127.0.0.0/8
</IfModule>

I also tried mod_remoteip, but could not get it to work -- it may not be has comprehensive as mod_rpaf. Specifically, we need the value applied to (in C) request_rec->connection->client_ip (or ->remote_id for Apache 2.2).

Alternatively, we could teach the translator to read the headers (or a custom header) directly.

Ben Shum (bshum) wrote :

Confirming this issue since I just hit it with the hard due date editor. Also noting that berick's proposed additions worked for me on Ubuntu 16.04 system.

Changed in opensrf:
status: New → Confirmed
importance: Undecided → Medium
milestone: none → 2.5.1
Jeff Davis (jdavis-sitka) wrote :

I tested with Ubuntu 14.04, Apache 2.4, nginx, and EG 2.12. I was able to add both an acq provider and a hard due date without the fix, but the IP address shows as 127.0.0.1 in translator logs. After applying Bill's fix, logs show the correct client IP.

Bill Erickson (berick) wrote :

I figured out the IPv6 oddness. Using 'localhost' as the destination in the NGINX configuration, coupled with a "::1 localhost" alias in /etc/hosts, resulted in requests being made to both 127.0.0.0 and ::1, intermittently. Configuring NGINX to proxy to 127.0.0.1 instead of localhost fixes it. This also affects bug #1691473.

So in addition to the above config changes, I propose we change the default NGINX config to proxy to 127.0.0.1 instead of localhost.

Unknown if this affects HAProxy.

Mike Rylander (mrylander) wrote :

As a data point, FWIW, mod_rpaf is available for apache 2.2 on debian-based distros. This should work there as well.

One drawback is that mod_rpaf takes the last X-Forwared-For header value seen by the backend, meaning that if you connect directly to apache and bypass the proxy, you'll be seen as coming from the most recently proxied IP. So, we may want to document that with a "don't do this" note, if we settle on mod_rpaf as the prescribed solution.

Also, thanks, Bill!

Dan Scott (denials) wrote :

The default mods-available/rpaf.conf in Ubuntu 16.04 has:

    RPAFproxy_ips 127.0.0.1 ::1

Wouldn't that proxy "localhost" when reached from either IPv4 or IPv6, thus addressing the concern in #3?

Dan Scott (denials) wrote :

FWIW on our production OpenSRF 2.5.0 / EG 2.12.x proxied-via-nginx system, I was able to create an acq provider with just the default rpaf.conf on Ubuntu 16.04. Not sure if we're still open to issues with pcrud though.

Dan Scott (denials) wrote :

Upping max_stanza_size to 2000000 (easier than rebuilding & reinstalling OpenSRF and Evergreen on our production server) resolved our issues with using the "Import via Z39.50" interface, which would hang when retrieving a full set of 25 results.

Dan Scott (denials) wrote :

Argh, wrong bug for that last comment - sorry.

Bill Erickson (berick) wrote :

Good point, Dan, with RPAF enabled (handling 127.0.0.1 and ::1), #3 should be a non-issue, since the translator is seeing the proxied IP and not the IP of the proxy.

I have reverted my nginx config to use localhost, using only the mods-available/rpaf.conf config, and I have created several acq providers without issue.

Presumably we don't want to require RPAF, so is this essentially a documentation bug?

Changed in opensrf:
milestone: 2.5.1 → 2.5.2
Galen Charlton (gmc) on 2017-09-15
Changed in opensrf:
milestone: 2.5.2 → 2.5.3
Jeff Godin (jgodin) wrote :

mod_rpaf upstream seems to be defunct, so we might in the future need a different option.

mod_remoteip is included in Apache 2.4, and has received at least some changes since this bug was opened.

Bill, do you have a concise test case for surfacing the issue you had with mod_remoteip in your previous testing in this bug's description?

Bill Erickson (berick) wrote :

Recapping from IRC discussion...

When I first tested mod_remoteip, it failed to update the value in the C code for request_rec->connection->client_ip. This is where the osrf http translator finds the IP address of the client. Long story short, it fails to fix the problem.

A test case is to use a stateful pcrud call (e.g. create an ACQ provider) via the translator through a proxy. Every client will be treated as coming from 127.0.0.1 (or similar) and this will be noted in the logs.

Jeff Godin (jgodin) wrote :

In a proxy/load balancer scenario, our expectation that client_ip contains an IP other than that of the proxy or load balanacer seems to be contrary to the Apache 2.4 API.

From http://httpd.apache.org/docs/2.4/developer/new_api_2_4.html

 * When you require the IP address of the user agent, which might be connected directly to the server, or might optionally be separated from the server by a transparent load balancer or proxy, use request_rec->useragent_ip and request_rec->useragent_addr.

 * When you require the IP address of the client that is connected directly to the server, which might be the useragent or might be the load balancer or proxy itself, use conn_rec->client_ip and conn_rec->client_addr.

Bill Erickson (berick) on 2018-08-27
Changed in opensrf:
assignee: nobody → Bill Erickson (berick)
Bill Erickson (berick) wrote :

Patch to call useragent_ip with optional configs and docs for using mod_remoteip instead of mod_rpaf:

http://git.evergreen-ils.org/?p=working/OpenSRF.git;a=shortlog;h=refs/heads/user/berick/lp1684970-translator-useragent-ip

tags: added: pullrequest
Bill Erickson (berick) wrote :

Oh, and thanks Jeff!

Changed in opensrf:
assignee: Bill Erickson (berick) → nobody
Galen Charlton (gmc) wrote :

Pushed to master, rel_3_0, and rel_2_5. Thanks, Bill!

Changed in opensrf:
status: Confirmed → Fix Committed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers