[vpnaas]UnicodeEncodeError occurs when using chinese charactors in vpn connection

[siyingchun]

In our Newton test environment, we found VPN connection can not be created successfully when you inputting Chinese characters which is our national language as its name, or when you use Chinese characters as its PSK, an unexpected and terrible result will happen，for instance, VPN peers can access each other by using PSKs which include different Chinese characters.

BTW, errors as below:
[vpnaas]UnicodeEncodeError: 'ascii' codec can't encode characters in position 20-21: ordinal not in range(128) occur when add IKE Policy with chinese charactors in vpn connection

22016-12-13 11:22:43.824 587926 WARNING neutron.agent.linux.iptables_manager [req-c888cc5c-0cc9-4070-85ad-514b0a552285 ebfa941f10994c859ad61ce074ea6f4a 69db65f43832456581518c876bd94706 - - -] Duplicate iptables rule detected. This may indicate a bug in the the iptables rule generation code. Line: -A neutron-vpn-agen-POSTROUTING -s 192.168.10.0/24 -d 192.168.2.0/24 -m policy --dir out --pol ipsec -j ACCEPT
2016-12-13 11:22:43.825 587926 WARNING neutron.agent.linux.iptables_manager [req-c888cc5c-0cc9-4070-85ad-514b0a552285 ebfa941f10994c859ad61ce074ea6f4a 69db65f43832456581518c876bd94706 - - -] Duplicate iptables rule detected. This may indicate a bug in the the iptables rule generation code. Line: -A neutron-vpn-agen-POSTROUTING -s 192.168.10.0/24 -d 192.168.0.0/24 -m policy --dir out --pol ipsec -j ACCEPT
2016-12-13 11:22:43.826 587926 WARNING neutron.agent.linux.iptables_manager [req-c888cc5c-0cc9-4070-85ad-514b0a552285 ebfa941f10994c859ad61ce074ea6f4a 69db65f43832456581518c876bd94706 - - -] Duplicate iptables rule detected. This may indicate a bug in the the iptables rule generation code. Line: -A neutron-vpn-agen-POSTROUTING -s 192.168.10.0/24 -d 192.168.2.0/24 -m policy --dir out --pol ipsec -j ACCEPT
2016-12-13 11:22:43.826 587926 WARNING neutron.agent.linux.iptables_manager [req-c888cc5c-0cc9-4070-85ad-514b0a552285 ebfa941f10994c859ad61ce074ea6f4a 69db65f43832456581518c876bd94706 - - -] Duplicate iptables rule detected. This may indicate a bug in the the iptables rule generation code. Line: -A neutron-vpn-agen-POSTROUTING -s 192.168.10.0/24 -d 192.168.0.0/24 -m policy --dir out --pol ipsec -j ACCEPT
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher [req-c888cc5c-0cc9-4070-85ad-514b0a552285 ebfa941f10994c859ad61ce074ea6f4a 69db65f43832456581518c876bd94706 - - -] Exception during message handling: 'ascii' codec can't encode characters in position 20-21: ordinal not in range(128)
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 669, in vpnservice_updated
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher self.sync(context, [router] if router else [])
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 823, in sync
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher self._sync_vpn_processes(vpnservices, sync_router_ids)
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 847, in _sync_vpn_processes
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher process.update()
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 238, in update
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher self.enable()
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 256, in enable
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher self.ensure_configs()
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py", line 61, in ensure_configs
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher super(LibreSwanProcess, self).ensure_configs()
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 353, in ensure_configs
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher self.vpnservice)
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 184, in ensure_config_file
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher utils.replace_file(config_file_name, config_str)
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/debtcollector/removals.py", line 242, in wrapper
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 177, in replace_file
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/socket.py", line 316, in write
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher data = str(data) # XXX Should really reject non-string non-buffers
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher UnicodeEncodeError: 'ascii' codec can't encode characters in position 20-21: ordinal not in range(128)
2016-12-13 11:22:43.885 587926 ERROR oslo_messaging.rpc.dispatcher

* Precondition:
   You have a large scale environment or a small test one which includes vpnaas.

* Step-by-step:
   1. Go to horizon > switch project to Winters
   2. Create vpnservices with Chinese characters
   3. Create IPSec and IKE policy
   4. create VPN connection with vpnservice(e.g. Chinese characters)
   5. Check VPN status

* Expect result:
   vpn connection can be created successfully without errors

* Actual result:
   errors can be found in /var/log/neutron/vpn-agent.log

* Version:
   Openstack Newton, deployed with Fuel 10.0
   Ubuntu Ubuntu 16.04.1 LTS, running kernel 4.4.0-57-generic
   Neutron version 5.1.0
   VPN 7.0.0

[Akihiro Motoki (amotoki)]

The patch is proposed at https://review.openstack.org/#/c/384215/ . This touched neutron.common.utils.replace_file to encode the file path. I am not sure this is the right approach.

I wonder why the configuration file path generated by neutron-vpnaas is affected by a parameter value with multi-byte character. It is worth investigated in neutron-vpnaas code first.

[YAMAMOTO Takashi (yamamoto)]

Akihiro,

is it the right url?

[siyingchun (wintersi)]

@Akihiro,

As you mentioned, my patch may be proposed at https://review.openstack.org/#/c/384215/ , but when I check it clearly, I find this patch just use the utils.replace_file from neutron_lib. And my patch wants to deal with the UnicodeEncodeError in VPN session name and conf file.

And I have modified as you suggested approach in http://docs.openstack.org/infra/manual/developers.html .

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/532475

[Cao Xuan Hoang (hoangcx)]

I have reproduced the usecase that reported mentioned above but there is no problem with steps. See [1] [2] for more detail.

But I and Hunt Xu realized that it will cause problem with vpnservice's name and PSKs. And It may be a mistaken by reporter.

@siyingchun: Could you please confirm?

[1] https://prnt.sc/ijtgwz
[2] https://prnt.sc/ijticw

[Cao Xuan Hoang (hoangcx)]

I just update a small nit in the steps to reproduce this bug.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/532475
Committed: https://git.openstack.org/cgit/openstack/neutron-vpnaas/commit/?id=2ec34202fd679d9ab3963b0ad5f83b0400a0b247
Submitter: Zuul
Branch: master

commit 2ec34202fd679d9ab3963b0ad5f83b0400a0b247
Author: Hunt Xu <email address hidden>
Date: Wed Jan 10 17:54:43 2018 +0800

    Avoid using non-ASCII characters when generating config files

    The name of a VPN service and the PSK of an IPsec site connection may
    contain non-ASCII characters. Outputing plain texts of these contents
    may lead to UnicodeEncodeError.

    As *swan can support base64 encoded PSKs. With this commit, we
      1. use VPN service id instead of the name in configuration files, and
      2. encode IPsec site connection PSK with base64
    to make sure that generated configuration files will only contain ASCII
    characters.

    Closes-Bug: #1652909

    Change-Id: Ie7edf080fc44537a74c57262bd9943c5e4337428

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron-vpnaas 13.0.0.0b3 development milestone.