Add option to hide ipaddress in neutron logs

[prashkre]

As some might know, EU has released a new law, that gets in action on May25th 2018, that prohibits misuse of any personal data corresponding to a natural person. Any information that can directly or indirectly lead to tracking of a natural person can be captured, stored or processed only with the consent of the natural person. IP Address is categorized as one such data. It can be debated if ipaddress can be classified as personal data or not but that would be beyond the scope of this defect.

The below log statements from neutron log displays the fixed ips associated with the VMs provisioned. The VMs provisioned from an Cloud platform like OpenStack could host someone's website and thus could be used to identify a natural person. Having said that, this information(logged ip) could be very useful from a serviceability perspective. So, the question is is it possible to add a mechanism such that we are able to configure whether this information should be logged or not?

2018-05-10 03:50:01.157 18683 INFO neutron.wsgi [req-b7b52f32-bbde-41b2-b882-707d63729256 a1e569eb16f0ec710b82314e31af4f8cfb1eedc3f0fb38554186e08717c21f0c 676da0962c9e48c687312f1a023af9ca - 96c9c4469e0b499e8c14043aa093b5bd 96c9c4469e0b499e8c14043aa093b5bd] 10.253.234.23,127.0.0.1 "GET /v2.0/floatingips?fixed_ip_address=162.42.34.10&port_id=41a1dff7-d5f5-43eb-a911-a594c4576f6a HTTP/1.1" status: 200 len: 217 time: 0.0247259

2018-05-10 03:50:02.049 18683 INFO neutron.wsgi [req-879d19ac-7b6f-4af0-b8b0-48d259f20ae7 a1e569eb16f0ec710b82314e31af4f8cfb1eedc3f0fb38554186e08717c21f0c 676da0962c9e48c687312f1a023af9ca - 96c9c4469e0b499e8c14043aa093b5bd 96c9c4469e0b499e8c14043aa093b5bd] 10.253.234.23,127.0.0.1 "GET /v2.0/floatingips?fixed_ip_address=162.42.34.10&port_id=41a1dff7-d5f5-43eb-a911-a594c4576f6a HTTP/1.1" status: 200 len: 217 time: 0.0201731

[Slawek Kaplonski (slaweq)]

I'm not a layer and I don't know details of GDPR compliance but the same IP address is stored in Neutron's (and Nova as well probably) database. Do You want to remove it from there also? It's IMO not possible :)

[prashkre (prashkre)]

We don't have much concern about storing IP address in database, major concern is about IP address in logs because when customers face some issues, without having notice that their IP addresses exits in logs, they share the logs for debugging, this should be not allowed as per GDPR.

[Brian Haley (brian-haley)]

I am not a lawyer either, but if we're talking about redacting all IP(v6) addresses from the logs this should probably be done in oslo.log since there are other services that are probably logging them as well.

I guess it comes down to what IP address is considered personal data - is it just a floating IP?

The downside is obvious - it could be very hard, in some cases impossible, to debug a problem when we can't trace a series of events if the IP address is the key.

[Assaf Muller (amuller)]

We must double and triple check if this is the right thing to do - Because as Brian said the cost is that it will often make troubleshooting impossible, and users wouldn't like that either.

[Divya K Konoor (dikonoor)]

This defect is not about removing ip logging from neutron logs. It is about putting a mechanism in place such that any deployment that does not want to log ip (due to compliance requirements) should have a way to achieve that.

[Brian Haley (brian-haley)]

Divya, well the title of the bug is "Skip logging of ipaddress in neutron logs", if it needs to be more generic that should be changed. As I mentioned, if the intent is to have a way to not log IPs in any Openstack service, then that needs to be done in the logging library (IMHO), else some will be missed. Otherwise it will be up to the developers to decide what IPs should be logged versus what shouldn't.

I am also curious if the onus resides on the Openstack services to do this - a cloud provider way want to log everything, and only redact things when the logs need to be shared with others. Not being aware of the policy here it's unclear if that meets the requirements.

[Divya K Konoor (dikonoor)]

Brian, I have updated the title of the bug. https://en.wikipedia.org/wiki/General_Data_Protection_Regulation should provide some general information about this new law. As per this, ipaddress comes under personal data (because in certain cases ipaddress in itself or with the help of additional data could help in identifying a person) and the law focuses on protecting personal data of EU citizens and preventing misuse of it - violation could lead to 20 million pound or 4% of the annual global turnover.

Any personal data that's obtained from a citizen should be with the consent of that person and should follow certain regulation. Having said this, logging ipaddress in logs in itself is not a violation. It should be done for a legitimate reason (serviceability and debug in the case of logs). It should be stored only for as long as it's needed. Logs with ipaddress should be captured with the consent of the person; it should be secured etc;

[Brian Haley (brian-haley)]

Divya, your last paragraph, in short, says the entity storing the log, not the logging itself, is the crux of the problem. The policy and control of that information are outside the scope of neutron. So it's still unclear if there is work for us to do here.

I think we should keep the bug open until there is a clear decision on exactly what changes, if any, Openstack services need to do.

[Brian Haley (brian-haley)]

I'm going to close this for a couple of reasons.

1) It doesn't seem like this is specific to neutron, so it's not the correct place to make such a change.

2) The neutron code has no concept of a particular person, everything is just a project ID, which isn't an identifiable piece of information without another mapping, for example, from Keystone.

3) At the end of the day, an operation such as filtering IP addresses from logs seems like something for an operator, and something they would have to do for a lot more than just this.

If there is a clearer document on what should be done here please update this bug with more information.