utf-8 characters are not supported by opencontrail for resource names

[Numan Siddique]

"neutron net-create netéù" is returning "Internal Server Error".
neutron ml2 plugin supports this.

This needs to be supported in opencontrail

The error is mainly seen in Apache thrift (fastbinart.encode_binary()) as it is using the default system encoding to encode.

The issue is not seen when the default encoding is changed to 'utf-8'.

[Numan Siddique (numansiddique)]

Below is the exception message

127.0.0.1 - - [2014-11-24 06:08:03] "GET /project/5ee0a1e3-903f-4d2b-adb4-41833ecfc3b8?exclude_back_refs=True&exclude_children=True HTTP/1.1" 200 904 0.124964
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/bottle.py", line 862, in _handle
    return route.call(**args)
  File "/usr/local/lib/python2.7/dist-packages/bottle.py", line 1729, in wrapper
    rv = callback(*a, **ka)
  File "/usr/local/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 475, in handler_trap_exception
    response = handler(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_api_server_gen.py", line 17070, in virtual_networks_http_post
    (ok, result) = self._post_common(request, 'virtual-network', obj_dict)
  File "/usr/local/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 1375, in _http_post_common
    obj_type, obj_dict['fq_name'])
  File "/usr/local/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 1571, in fq_name_to_uuid
    obj_uuid = self._cassandra_db.fq_name_to_uuid(obj_type, fq_name)
  File "/usr/local/lib/python2.7/dist-packages/cfgm_common/vnc_cassandra.py", line 220, in fq_name_to_uuid
    col_infos = list(col_info_iter)
  File "/usr/local/lib/python2.7/dist-packages/pycassa/columnfamily.py", line 558, in xget
    list_cosc = self.pool.execute('get_slice', packed_key, cp, sp, rcl)
  File "/usr/local/lib/python2.7/dist-packages/pycassa/pool.py", line 577, in execute
    return getattr(conn, f)(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pycassa/pool.py", line 127, in new_f
    result = f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pycassa/cassandra/Cassandra.py", line 579, in get_slice
    self.send_get_slice(key, column_parent, predicate, consistency_level)
  File "/usr/local/lib/python2.7/dist-packages/pycassa/cassandra/Cassandra.py", line 589, in send_get_slice
    args.write(self._oprot)
  File "/usr/local/lib/python2.7/dist-packages/pycassa/cassandra/Cassandra.py", line 3322, in write
    oprot.trans.write(fastbinary.encode_binary(self, (self.__class__, self.thrift_spec)))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 24: ordinal not in range(128)
127.0.0.1 - - [2014-11-24 06:08:03] "POST /virtual-networks HTTP/1.1" 500 156 0.274577
<class 'cfgm_common.exceptions.HttpError'>
Python 2.7.6: /usr/bin/python

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/5977
Committed: http://github.org/Juniper/contrail-controller/commit/75ca50998304a3de5679d17df73d1b04c8c34e76
Submitter: Zuul
Branch: master

commit 75ca50998304a3de5679d17df73d1b04c8c34e76
Author: Hampapur Ajay <email address hidden>
Date: Tue Dec 30 13:17:34 2014 -0800

Add support for names with non-ascii (utf-8) characters.

Encode utf-8 chars and store in obj_fq_name_table column names
since comparator type cannot be changed on already existing CF.

Also set validator type (column values) to utf8 (on startup to handle
upgrade) to handle contents (e.g. description etc.) having non-ascii
characters.

Change-Id: I9238838baba908784c7e8c8a13abc062c576cba2
Partial-Bug: #1396894

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/6018
Committed: http://github.org/Juniper/contrail-generateDS/commit/23e0ab3f040da6158500d329dbf1b75b3a4df572
Submitter: Zuul
Branch: master

commit 23e0ab3f040da6158500d329dbf1b75b3a4df572
Author: Hampapur Ajay <email address hidden>
Date: Mon Jan 5 09:35:21 2015 -0800

Add support for names with non-ascii (utf-8) characters.

Encode utf-8 chars and store in obj_fq_name_table column names
since comparator type cannot be changed on already existing CF.

Change-Id: Ia20fe2064060901dcf9cbfbf269e9681db7429c5
Closes-Bug: #1396894

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/6417
Committed: http://github.org/Juniper/contrail-controller/commit/2ab3e08484096388b98846322456073495f7213b
Submitter: Zuul
Branch: R1.10

commit 2ab3e08484096388b98846322456073495f7213b
Author: Numan Siddique <email address hidden>
Date: Wed Jan 21 12:02:38 2015 +0530

Add support for names with non-ascii (utf-8) characters.

Encode utf-8 chars and store in obj_fq_name_table column names
since comparator type cannot be changed on already existing CF.

Also set validator type (column values) to utf8 (on startup to handle
upgrade) to handle contents (e.g. description etc.) having non-ascii
characters.

(cherry pick of I9238838baba908784c7e8c8a13abc062c576cba2)

Change-Id: I7d911ba3981dba7eee25d036ea3a96d09d062c27
Partial-Bug: #1396894

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/7316
Committed: http://github.org/Juniper/contrail-controller/commit/c15936714f9b66b50d9af19af987db1e4ceb37ab
Submitter: Zuul
Branch: R2.1

commit c15936714f9b66b50d9af19af987db1e4ceb37ab
Author: Numan Siddique <email address hidden>
Date: Wed Feb 11 20:27:22 2015 +0530

Enable utf-8 support in schema transformer

To support utf-8 characters in resource name,
the default encoding needs to be changed to 'utf-8'
for schema transformer as well.

Change-Id: Iebf3344820519ef2ca3e8f5a72fa697ebc40e982
Partial-Bug: #1396894

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/6709
Committed: http://github.org/Juniper/contrail-controller/commit/0b30de4b163e9642f1bfa59ac07607c966b4a7de
Submitter: Zuul
Branch: master

commit 0b30de4b163e9642f1bfa59ac07607c966b4a7de
Author: Numan Siddique <email address hidden>
Date: Wed Feb 11 20:27:22 2015 +0530

Enable utf-8 support in schema transformer

To support utf-8 characters in resource name,
the default encoding needs to be changed to 'utf-8'
for schema transformer as well.

Change-Id: Iebf3344820519ef2ca3e8f5a72fa697ebc40e982
Partial-Bug: #1396894

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/7312
Committed: http://github.org/Juniper/contrail-controller/commit/3231cb79941b3fb4ec9ae24e980d5f90c849bdba
Submitter: Zuul
Branch: R2.0

commit 3231cb79941b3fb4ec9ae24e980d5f90c849bdba
Author: Numan Siddique <email address hidden>
Date: Wed Feb 11 20:27:22 2015 +0530

Enable utf-8 support in schema transformer

To support utf-8 characters in resource name,
the default encoding needs to be changed to 'utf-8'
for schema transformer as well.

Change-Id: Iebf3344820519ef2ca3e8f5a72fa697ebc40e982
Partial-Bug: #1396894