bump cliff version now that cliff-tablib has been removed

[Loic Dachary]

If cliff-tablib is installed, the output of openstack server show -f json is an array such as:

   [{"Field": "OS-DCF:diskConfig", "Value": "MANUAL"}, {"Field": "OS-EXT-AZ:availability_zone", "Value": "nova"}, ...

if cliff-tablib is not installed, the output of openstack server show -f json is dictionary such as:

  {
  "OS-EXT-STS:task_state": null,
  "addresses": "Ext-Net=167.114.238.68",
  "image": "teuthology-ubuntu-14.04 (4300a7ca-4fbd-4b34-a8d5-5a4ebf204df5)",
  "OS-EXT-STS:vm_state": "active",

this variance makes it difficult for scripts to use the output consistently as they have to detect the type of the output and use different codepath to find the value of a given field. A few weeks ago, cliff-tablib was an indirect dependency of pip install python-openstack client there was a single output format, the array format. Then the indirect dependency was dropped and cliff-tablib may or may not be installed which introduces the problem.

This variance can easily be resolved by adding cliff-tablib as a direct dependency of python-openstackclient.

[Steve Martinelli (stevemar)]

as of 1.8.0 we no longer install cliff-tablib (https://github.com/openstack/python-openstackclient/commit/b60a9492a31bb1130338176e4b90a7572c10edc3)

it seems unmaintained and any functionality we were using has been moved to cliff (https://github.com/openstack/cliff/commit/4e1470d93e101d1390b39b93dec569c7c8966b83)

i think the right move here is to bump the requirements to 1.15.0 so we are definitely using the bits from cliff and not cliff-tablib

[Steve Martinelli (stevemar)]

refer to https://review.openstack.org/#/c/250916/

[Loic Dachary (dachary)]

Even with cliff 1.15.0, the presence or absence of the cliff-tablib module changes the output.

(virtualenv)loic@fold:~/software/ceph/teuthology$ openstack server show teuthology -f json
{
  "OS-EXT-STS:task_state": null,
  "addresses": "Ext-Net=167.114.248.8",
  "image": "teuthology-ubuntu-14.04 (4300a7ca-4fbd-4b34-a8d5-5a4ebf204df5)",
  "OS-EXT-STS:vm_state": "active",
  "OS-SRV-USG:launched_at": "2015-11-27T09:11:22.000000",
  "flavor": "eg-30 (3c1d6170-0097-4b5c-a3b3-adff1b7a86e0)",
  "id": "affbd6b5-f41e-49cc-b145-a7ebcac145f8",
  "security_groups": [
    {
      "name": "teuthology"
    }
  ],
  "user_id": "291dde1633154837be2693c6ffa6315c",
  "OS-DCF:diskConfig": "MANUAL",
  "accessIPv4": "",
  "accessIPv6": "",
  "progress": 0,
  "OS-EXT-STS:power_state": 1,
  "project_id": "131b886b156a4f84b5f41baf2fbe646c",
  "config_drive": "",
  "status": "ACTIVE",
  "updated": "2015-11-27T09:11:22Z",
  "hostId": "fcb073c0ed699b624eb5c843408b5689428f0250c5b1cf67f1636b5a",
  "OS-SRV-USG:terminated_at": null,
  "key_name": "myself",
  "properties": "",
  "OS-EXT-AZ:availability_zone": "nova",
  "name": "teuthology",
  "created": "2015-11-27T09:10:42Z",
  "os-extended-volumes:volumes_attached": []
}(virtualenv)loic@fold:~/software/ceph/teuthology$ pip list cliff
*** output flushed ***
(virtualenv)loic@fold:~/software/ceph/teuthology$ pip list installed | grep cliff
cliff (1.15.0)
(virtualenv)loic@fold:~/software/ceph/teuthology$ pip install cliff-tablib
Collecting cliff-tablib
Installing collected packages: cliff-tablib
Successfully installed cliff-tablib-1.1
(virtualenv)loic@fold:~/software/ceph/teuthology$ openstack server show teuthology -f json
[{"Field": "OS-DCF:diskConfig", "Value": "MANUAL"}, {"Field": "OS-EXT-AZ:availability_zone", "Value": "nova"}, {"Field": "OS-EXT-STS:power_state", "Value": 1}, {"Field": "OS-EXT-STS:task_state", "Value": null}, {"Field": "OS-EXT-STS:vm_state", "Value": "active"}, {"Field": "OS-SRV-USG:launched_at", "Value": "2015-11-27T09:11:22.000000"}, {"Field": "OS-SRV-USG:terminated_at", "Value": null}, {"Field": "accessIPv4", "Value": ""}, {"Field": "accessIPv6", "Value": ""}, {"Field": "addresses", "Value": "Ext-Net=167.114.248.8"}, {"Field": "config_drive", "Value": ""}, {"Field": "created", "Value": "2015-11-27T09:10:42Z"}, {"Field": "flavor", "Value": "eg-30 (3c1d6170-0097-4b5c-a3b3-adff1b7a86e0)"}, {"Field": "hostId", "Value": "fcb073c0ed699b624eb5c843408b5689428f0250c5b1cf67f1636b5a"}, {"Field": "id", "Value": "affbd6b5-f41e-49cc-b145-a7ebcac145f8"}, {"Field": "image", "Value": "teuthology-ubuntu-14.04 (4300a7ca-4fbd-4b34-a8d5-5a4ebf204df5)"}, {"Field": "key_name", "Value": "myself"}, {"Field": "name", "Value": "teuthology"}, {"Field": "os-extended-volumes:volumes_attached", "Value": []}, {"Field": "progress", "Value": 0}, {"Field": "project_id", "Value": "131b886b156a4f84b5f41baf2fbe646c"}, {"Field": "properties", "Value": ""}, {"Field": "security_groups", "Value": [{"name": "teuthology"}]}, {"Field": "status", "Value": "ACTIVE"}, {"Field": "updated", "Value": "2015-11-27T09:11:22Z"}, {"Field": "user_id", "Value": "291dde1633154837be2693c6ffa6315c"}]

[Steve Martinelli (stevemar)]

@loic, right, on old environments that have cliff-tablib installed we won't be able to fix. but going forward, OSC should only depend on cliff not cliff-tablib. i doubt we can do much about existing installations with tablib installed

[Loic Dachary (dachary)]

@stevemar this is not limited to old environment. If cliff-tablib is installed by the user, because she/he wants this module for some reason or as a dependency of another a module entirely unrelated to python-openstackclient, the output of python-openstackclient will unexpectedly change and no longer be parseable by scripts that assume the output of python-openstackclient is stable.

Do you think it is unreasonable to expect that the output of python-openstackclient is stable ? Maybe scripts parsing the output of python-openstackclient should all have a wrapper to detect if --format json returns an array of Field/Value or a dictionary and convert accordingly.

[Steve Martinelli (stevemar)]

maybe i'm missing something, but i didn't think OSC would be picking up any traces of cliff-tablib any longer.

adding dhellmann, maybe he knows whats going on

[Steve Martinelli (stevemar)]

do i need to have metadata or something? i'm not seeing it on osc-master and cliff 1.15.0

$ pip freeze | grep cliff
cliff==1.15.0

$ openstack user show admin -f json
{
  "username": "admin",
  "enabled": true,
  "id": "afb907505fdb411ebf9d7a8b20cf74c5",
  "name": "admin"
}

$ pip install cliff-tablib
Collecting cliff-tablib
Requirement already satisfied (use --upgrade to upgrade): cliff in /usr/local/lib/python2.7/dist-packages (from cliff-tablib)
Requirement already satisfied (use --upgrade to upgrade): tablib in /usr/local/lib/python2.7/dist-packages (from cliff-tablib)
Requirement already satisfied (use --upgrade to upgrade): cmd2>=0.6.7 in /usr/local/lib/python2.7/dist-packages (from cliff->cliff-tablib)
Requirement already satisfied (use --upgrade to upgrade): argparse in /usr/local/lib/python2.7/dist-packages (from cliff->cliff-tablib)
Requirement already satisfied (use --upgrade to upgrade): pyparsing>=2.0.1 in /usr/local/lib/python2.7/dist-packages (from cliff->cliff-tablib)
Requirement already satisfied (use --upgrade to upgrade): stevedore>=1.5.0 in /usr/local/lib/python2.7/dist-packages (from cliff->cliff-tablib)
Requirement already satisfied (use --upgrade to upgrade): unicodecsv>=0.8.0 in /usr/local/lib/python2.7/dist-packages (from cliff->cliff-tablib)
Requirement already satisfied (use --upgrade to upgrade): six>=1.9.0 in /usr/local/lib/python2.7/dist-packages (from cliff->cliff-tablib)
Requirement already satisfied (use --upgrade to upgrade): PyYAML>=3.1.0 in /usr/local/lib/python2.7/dist-packages (from cliff->cliff-tablib)
Requirement already satisfied (use --upgrade to upgrade): pbr<2.0,>=1.4 in /usr/local/lib/python2.7/dist-packages (from cliff->cliff-tablib)
Requirement already satisfied (use --upgrade to upgrade): PrettyTable<0.8,>=0.7 in /usr/local/lib/python2.7/dist-packages (from cliff->cliff-tablib)
Installing collected packages: cliff-tablib
Successfully installed cliff-tablib-1.1

$ pip freeze | grep cliff
cliff==1.15.0
cliff-tablib==1.1

$ openstack user show admin -f json
{
  "username": "admin",
  "enabled": true,
  "id": "afb907505fdb411ebf9d7a8b20cf74c5",
  "name": "admin"
}

[Loic Dachary (dachary)]

Here are the steps to reproduce:

* provision a fresh Ubuntu 14.04
* sudo apt-get install git
* git clone http://github.com/ceph/teuthology
* cd teuthology
* ./bootstrap install
* source virtualenv/bin/activate
* source openrc.sh # your own openrc.sh ;-)
* openstack server show -f json a server

After running the steps above, the list of python modules installed is:

ansible (1.9.2)
appdirs (1.4.0)
apt-xapian-index (0.45)
Babel (2.1.1)
beanstalkc (0.4.0)
boto (2.38.0)
bunch (1.0.1)
cffi (1.3.1)
chardet (2.0.1)
Cheetah (2.4.4)
cliff (1.15.0)
cliff-tablib (1.1)
cloud-init (0.7.5)
cmd2 (0.6.8)
colorama (0.2.5)
configobj (5.0.6)
configparser (3.3.0.post2)
cryptography (1.1.1)
debtcollector (1.0.0)
docopt (0.6.2)
ecdsa (0.13)
enum34 (1.0.4)
functools32 (3.2.3.post2)
gevent (1.0.2)
greenlet (0.4.9)
html5lib (0.999)
httplib2 (0.9.2)
idna (2.0)
ipaddress (1.0.15)
iso8601 (0.1.11)
Jinja2 (2.8)
jsonpatch (1.12)
jsonpointer (1.10)
jsonschema (2.5.1)
keystoneauth1 (1.2.0)
Landscape-Client (14.12)
libvirt-python (1.2.2)
MarkupSafe (0.23)
meld3 (0.6.10)
monotonic (0.4)
msgpack-python (0.4.6)
MySQL-python (1.2.3)
ndg-httpsclient (0.4.0)
netaddr (0.7.18)
netifaces (0.10.4)
oauth (1.0.1)
os-client-config (1.12.0)
oslo.config (3.0.0)
oslo.i18n (3.0.0)
oslo.serialization (2.0.0)
oslo.utils (3.0.0)
PAM (0.4.2)
paramiko (1.16.0)
pbr (1.8.1)
pexpect (4.0.1)
pip (7.1.2)
prettytable (0.7.2)
psutil (3.3.0)
ptyprocess (0.5)
py (1.4.30)
pyasn1 (0.1.9)
pycparser (2.14)
pycrypto (2.6.1)
pycurl (7.19.3)
pyOpenSSL (0.15.1)
pyparsing (2.0.6)
pyserial (2.6)
pytest (2.8.3)
python-apt (0.9.3.5ubuntu1)
python-cinderclient (1.4.0)
python-debian (0.1.21-nmu2ubuntu2)
python-glanceclient (1.2.0)
python-keystoneclient (1.8.1)
python-neutronclient (3.1.0)
python-novaclient (2.35.0)
python-openstackclient (1.9.0)
pytz (2015.7)
PyYAML (3.11)
raven (5.8.1)
requests (2.8.1)
requestsexceptions (1.1.1)
setuptools (18.5)
simplejson (3.8.1)
six (1.10.0)
ssh-import-id (3.21)
stevedore (1.9.0)
supervisor (3.0b2)
tablib (0.10.0)
teuthology (0.1.0, /home/ubuntu/teuthology)
Twisted-Core (13.2.0)
Twisted-Names (13.2.0)
Twisted-Web (13.2.0)
unicodecsv (0.14.1)
urllib3 (1.7.1)
virtualenv (1.11.4)
warlock (1.2.0)
web.py (0.37)
wheel (0.24.0)
wrapt (1.10.5)
zope.interface (4.0.5)

[Loic Dachary (dachary)]

Although it is probably obvious, here are the steps after the install:

(virtualenv)ubuntu@try:~$ openstack server show -f json target-do-not
[{"Field": "OS-DCF:diskConfig", "Value": "MANUAL"}, {"Field": "OS-EXT-AZ:availability_zone", "Value": "nova"}, {"Field": "OS-EXT-STS:power_state", "Value": 1}, {"Field": "OS-EXT-STS:task_state", "Value": null}, {"Field": "OS-EXT-STS:vm_state", "Value": "active"}, {"Field": "OS-SRV-USG:launched_at", "Value": "2015-11-27T00:17:36.000000"}, {"Field": "OS-SRV-USG:terminated_at", "Value": null}, {"Field": "accessIPv4", "Value": ""}, {"Field": "accessIPv6", "Value": ""}, {"Field": "addresses", "Value": "Ext-Net=149.202.160.204"}, {"Field": "config_drive", "Value": ""}, {"Field": "created", "Value": "2015-11-27T00:16:06Z"}, {"Field": "flavor", "Value": "vps-ssd-1 (98c1e679-5f2c-4069-b4da-4a4f7179b758)"}, {"Field": "hostId", "Value": "4b1f4250c088e508e4358bcbaaecf280c3e851b9367f0d1d701f8683"}, {"Field": "id", "Value": "e4e2f397-cc51-4e70-b9db-760ba70c9236"}, {"Field": "image", "Value": "teuthology-ubuntu-14.04 (c1fe268d-d0e9-48af-9482-7303d49cfce4)"}, {"Field": "key_name", "Value": "loic"}, {"Field": "name", "Value": "target-do-not"}, {"Field": "os-extended-volumes:volumes_attached", "Value": []}, {"Field": "progress", "Value": 0}, {"Field": "project_id", "Value": "62cf1be03cec403c8ed8e64df55732ea"}, {"Field": "properties", "Value": ""}, {"Field": "security_groups", "Value": [{"name": "default"}]}, {"Field": "status", "Value": "ACTIVE"}, {"Field": "updated", "Value": "2015-11-27T00:17:36Z"}, {"Field": "user_id", "Value": "3a075820e5d24fda96cd340b87fd94e9"}]
(virtualenv)ubuntu@try:~$ pip uninstall cliff-tablib
Uninstalling cliff-tablib-1.1:
  /home/ubuntu/teuthology/virtualenv/lib/python2.7/site-packages/cliff_tablib-1.1.dist-info/DESCRIPTION.rst
  /home/ubuntu/teuthology/virtualenv/lib/python2.7/site-packages/cliff_tablib-1.1.dist-info/METADATA
  /home/ubuntu/teuthology/virtualenv/lib/python2.7/site-packages/cliff_tablib-1.1.dist-info/RECORD
  /home/ubuntu/teuthology/virtualenv/lib/python2.7/site-packages/cliff_tablib-1.1.dist-info/WHEEL
  /home/ubuntu/teuthology/virtualenv/lib/python2.7/site-packages/cliff_tablib-1.1.dist-info/entry_points.txt
  /home/ubuntu/teuthology/virtualenv/lib/python2.7/site-packages/cliff_tablib-1.1.dist-info/metadata.json
  /home/ubuntu/teuthology/virtualenv/lib/python2.7/site-packages/cliff_tablib-1.1.dist-info/namespace_packages.txt
  /home/ubuntu/teuthology/virtualenv/lib/python2.7/site-packages/cliff_tablib-1.1.dist-info/top_level.txt
  /home/ubuntu/teuthology/virtualenv/lib/python2.7/site-packages/clifftablib/__init__.py
  /home/ubuntu/teuthology/virtualenv/lib/python2.7/site-packages/clifftablib/__init__.pyc
  /home/ubuntu/teuthology/virtualenv/lib/python2.7/site-packages/clifftablib/formatters.py
  /home/ubuntu/teuthology/virtualenv/lib/python2.7/site-packages/clifftablib/formatters.pyc
Proceed (y/n)? y
  Successfully uninstalled cliff-tablib-1.1
(virtualenv)ubuntu@try:~$ openstack server show -f json target-do-not
{
  "OS-EXT-STS:task_state": null,
  "addresses": "Ext-Net=149.202.160.204",
  "image": "teuthology-ubuntu-14.04 (c1fe268d-d0e9-48af-9482-7303d49cfce4)",
  "OS-EX...

[Manavalan Krishnan (manavalan-krishnan-s)]

It looks we hit similar issue when we run the openstack teuthology as per instruction given at: https://github.com/dachary/teuthology/tree/openstack/#openstack-backend. Discussed with the original bug reporter Loic Dachary.

I get the following error when i run teuthology-openstack script.

2015-12-05 14:57:58,203.203 DEBUG:teuthology.misc:}
Traceback (most recent call last):
  File "./virtualenv/bin/teuthology-openstack", line 9, in <module>
    load_entry_point('teuthology==0.1.0', 'console_scripts', 'teuthology-openstack')()
  File "/schooner/backup/mana/openstack/teuthology/scripts/openstack.py", line 7, in main
    sys.exit(teuthology.openstack.main(parse_args(argv), argv))
  File "/schooner/backup/mana/openstack/teuthology/teuthology/openstack/__init__.py", line 777, in main
    return TeuthologyOpenStack(ctx, teuth_config, argv).main()
  File "/schooner/backup/mana/openstack/teuthology/teuthology/openstack/__init__.py", line 446, in main
    self.setup()
  File "/schooner/backup/mana/openstack/teuthology/teuthology/openstack/__init__.py", line 532, in setup
    self.create_cluster()
  File "/schooner/backup/mana/openstack/teuthology/teuthology/openstack/__init__.py", line 762, in create_cluster
    self.instance = OpenStackInstance(self.args.name)
  File "/schooner/backup/mana/openstack/teuthology/teuthology/openstack/__init__.py", line 52, in __init__
    self.set_info()
  File "/schooner/backup/mana/openstack/teuthology/teuthology/openstack/__init__.py", line 61, in set_info
    lambda p: (p['Field'].lower(), p['Value']), info))
  File "/schooner/backup/mana/openstack/teuthology/teuthology/openstack/__init__.py", line 61, in <lambda>
    lambda p: (p['Field'].lower(), p['Value']), info))
TypeError: string indices must be integers

Output of "pip list installed" and "type python" given below.

(virtualenv)root@lab143:/schooner/backup/mana/openstack/teuthology# pip list installed
ansible (1.9.2)
appdirs (1.4.0)
apt-xapian-index (0.45)
Babel (2.1.1)
beanstalkc (0.4.0)
blinker (1.3)
boto (2.38.0)
bunch (1.0.1)
cffi (1.3.1)
chardet (2.0.1)
cliff (1.15.0)
cliff-tablib (1.1)
cmd2 (0.6.8)
colorama (0.2.5)
configobj (5.0.6)
configparser (3.3.0.post2)
cryptography (1.1.1)
debtcollector (1.0.0)
docopt (0.6.2)
docutils (0.11)
ecdsa (0.13)
enum34 (1.1.1)
functools32 (3.2.3.post2)
gevent (1.0.2)
greenlet (0.4.9)
html5lib (0.999)
httplib2 (0.9.2)
idna (2.0)
ipaddress (1.0.15)
iso8601 (0.1.11)
itsdangerous (0.22)
Jinja2 (2.8)
jsonpatch (1.12)
jsonpointer (1.10)
jsonschema (2.5.1)
keystoneauth1 (2.0.0)
Landscape-Client (14.12)
libvirt-python (1.2.2)
MarkupSafe (0.23)
monotonic (0.4)
msgpack-python (0.4.6)
MySQL-python (1.2.3)
ndg-httpsclient (0.4.0)
netaddr (0.7.18)
netifaces (0.10.4)
nose (1.3.1)
os-client-config (1.12.0)
oslo.config (3.0.0)
oslo.i18n (3.0.0)
oslo.serialization (2.0.0)
oslo.utils (3.0.0)
PAM (0.4.2)
paramiko (1.16.0)
pbr (1.8.1)
pexpect (4.0.1)
Pillow (2.3.0)
pip (7.1.2)
prettytable (0.7.2)
psutil (3.3.0)
ptyprocess (0.5)
py (1.4.31)
pyasn1 (0.1.9)
pycparser (2.14)
pycrypto (2.6.1)
Pygments (1.6)
pyinotify (0.9.4)
pyOpenSSL (0.15.1)
pyparsing (2.0.6)
pyserial (2.6)
pytest (2.8.3)
python-apt (0.9.3.5ubuntu1)
python-cindercl...

[Loic Dachary (dachary)]

@manavalan-krishnan-s you actually have the opposite problem, that is you get a dictionnary for some reason. It would be helpful if you could explain how you made the installation, starting from a brand new version of your operating system (in a virtual machine ideally) so that it can be repeated.

The deeper problem is that the python-openstackclient may change for reasons that are very difficult to prevent or diagnose.

[Loic Dachary (dachary)]

s/the python-openstackclient may/the python-openstackclient output may/

[Steve Martinelli (stevemar)]

i'm at my wits end here guys, i've tried to reproduce this three times and i can't:

$ pip freeze | grep cliff
cliff==1.15.0
cliff-tablib==1.1

$ openstack server show my_serv --format json
{
  "OS-EXT-STS:task_state": null,
  "addresses": "private=10.0.0.2",
  "image": "cirros-0.3.4-x86_64-uec (a5aba569-5c8c-4e53-b989-d5bdfc70001c)",
  "OS-EXT-STS:vm_state": "active",
  "OS-EXT-SRV-ATTR:instance_name": "instance-00000001",
  "OS-SRV-USG:launched_at": "2015-12-10T23:31:47.000000",
  "flavor": "m1.tiny (1)",
  "id": "e2ebabea-c0ac-403b-b427-152f4daa4952",
  "security_groups": [
    {
      "name": "default"
    }
  ],
  "user_id": "37f71514c0e94219a31bdf467fdce420",
  "OS-DCF:diskConfig": "MANUAL",
  "accessIPv4": "",
  "accessIPv6": "",
  "progress": 0,
  "OS-EXT-STS:power_state": 1,
  "OS-EXT-AZ:availability_zone": "nova",
  "config_drive": "True",
  "status": "ACTIVE",
  "updated": "2015-12-10T23:31:48Z",
  "hostId": "97803136f965cded313cd56f9bfe80173e1e7964f0e5be17587cbac2",
  "OS-EXT-SRV-ATTR:host": "ubuntu",
  "OS-SRV-USG:terminated_at": null,
  "key_name": null,
  "properties": "key1='value1'",
  "project_id": "72ec36d95f854264a0eac32939d22aa6",
  "OS-EXT-SRV-ATTR:hypervisor_hostname": "ubuntu",
  "name": "my_serv",
  "created": "2015-12-10T23:31:26Z",
  "os-extended-volumes:volumes_attached": []
}

Is modifiying teuthology-openstack to remove cliff-tablib out of the question? Looks like it's requirements here: https://github.com/ceph/teuthology/blob/master/setup.py#L61

In all seriousness, the cliff-tablib library is unmaintained (it is failing to build on pypi: https://pypi.python.org/pypi/cliff-tablib/1.1) and there hasn't been a commit since Jan 2014 (https://github.com/dreamhost/cliff-tablib/commits/master). So I don't really buy the argument that someone is using it for non-openstack purposes, since it was developed for openstack purposes.

[Steve Martinelli (stevemar)]

I've moved it to 'opinion', and unassigned myself, i have other things to work on. If someone else wants to pick it up, or can reproduce the error, they can do so. But as it stands, I can't. Since there is some desire from the original reporter to keep it alive, I won't mark it as incomplete for now.

[Loic Dachary (dachary)]

> If someone else wants to pick it up, or can reproduce the error, they can do so. But as it stands, I can't.

Can you please confirm you followed the instructions to reproduce and found a different behavior than what is reported ?

* provision a fresh Ubuntu 14.04
* sudo apt-get install git
* git clone http://github.com/ceph/teuthology
* cd teuthology
* ./bootstrap install
* source virtualenv/bin/activate
* source openrc.sh # your own openrc.sh ;-)
* openstack server show -f json a server

[Loic Dachary (dachary)]

> I've moved it to 'opinion', and unassigned myself, i have other things to work on.

I'd like to point out that this phrase makes me feel like I've wasted your valuable time. It is a strong incentive to not report bugs should I find one. I assume this is written out of frustration.

[Steve Martinelli (stevemar)]

> I assume this is written out of frustration.

It was.

> It is a strong incentive to not report bugs should I find one.

Not my intention, we actually have a pretty good track record for resolving bugs, quickly too.

I'll follow the instructions, but at what point does it become an OSC bug vs a bug against teuthology?

[Steve Martinelli (stevemar)]

FWIW, this code looks hella suspicious: https://github.com/ceph/teuthology/blob/master/teuthology/openstack/__init__.py#L56-L62

    def set_info(self):
        try:
            info = json.loads(
                misc.sh("openstack server show -f json " + self.name_or_id))
            self.info = dict(map(
                lambda p: (p['Field'].lower(), p['Value']), info))
        except CalledProcessError:
            self.info = None

[Steve Martinelli (stevemar)]

SO... when I don't use a virtualenv, things are fine:

stevemar@ubuntu:~/teuthology$ openstack server show my_serv --format json
{
  "OS-EXT-STS:task_state": null,
  "addresses": "private=10.0.0.2",
  "image": "cirros-0.3.4-x86_64-uec (d75ab611-8af6-41df-88e2-210526ea1c2b)",
  "OS-EXT-STS:vm_state": "active",
  "OS-EXT-SRV-ATTR:instance_name": "instance-00000001",
  "OS-SRV-USG:launched_at": "2015-12-11T02:56:33.000000",
  "flavor": "m1.tiny (1)",
  "id": "b88bb13d-193a-4809-955c-f828dd69e784",
  "security_groups": [
    {
      "name": "default"
    }
  ],
  "user_id": "736e8e5dcb0441748eb91378225c9aa4",
  "OS-DCF:diskConfig": "MANUAL",
  "accessIPv4": "",
  "accessIPv6": "",
  "progress": 0,
  "OS-EXT-STS:power_state": 1,
  "OS-EXT-AZ:availability_zone": "nova",
  "config_drive": "True",
  "status": "ACTIVE",
  "updated": "2015-12-11T02:56:33Z",
  "hostId": "3c5806fe3f28c64aa63cbbdf023a3790e7d5100c7e396d3516bedebb",
  "OS-EXT-SRV-ATTR:host": "ubuntu",
  "OS-SRV-USG:terminated_at": null,
  "key_name": null,
  "properties": "",
  "project_id": "62d97cd7b4674113a3fe1bf2bfdf4702",
  "OS-EXT-SRV-ATTR:hypervisor_hostname": "ubuntu",
  "name": "my_serv",
  "created": "2015-12-11T02:56:12Z",
  "os-extended-volumes:volumes_attached": []
}

stevemar@ubuntu:~/teuthology$ pip list installed | grep cliff
cliff (1.15.0)
cliff-tablib (1.1)

stevemar@ubuntu:~/teuthology$ pip list installed | grep client
ndg-httpsclient (0.4.0)
os-client-config (1.12.0)
piston-mini-client (0.7.5)
python-cinderclient (1.5.0)
python-glanceclient (1.2.0)
python-keystoneclient (2.0.0)
python-neutronclient (3.1.0)
python-novaclient (2.35.0)
python-openstackclient (2.0.0)

[Steve Martinelli (stevemar)]

when I use a virtual environment, i get:

stevemar@ubuntu:~/teuthology$ source virtualenv/bin/activate
(virtualenv)stevemar@ubuntu:~/teuthology$ openstack server show my_serv --format json
[{"Field": "OS-DCF:diskConfig", "Value": "MANUAL"}, {"Field": "OS-EXT-AZ:availability_zone", "Value": "nova"}, {"Field": "OS-EXT-SRV-ATTR:host", "Value": "ubuntu"}, {"Field": "OS-EXT-SRV-ATTR:hypervisor_hostname", "Value": "ubuntu"}, {"Field": "OS-EXT-SRV-ATTR:instance_name", "Value": "instance-00000001"}, {"Field": "OS-EXT-STS:power_state", "Value": 1}, {"Field": "OS-EXT-STS:task_state", "Value": null}, {"Field": "OS-EXT-STS:vm_state", "Value": "active"}, {"Field": "OS-SRV-USG:launched_at", "Value": "2015-12-11T02:56:33.000000"}, {"Field": "OS-SRV-USG:terminated_at", "Value": null}, {"Field": "accessIPv4", "Value": ""}, {"Field": "accessIPv6", "Value": ""}, {"Field": "addresses", "Value": "private=10.0.0.2"}, {"Field": "config_drive", "Value": "True"}, {"Field": "created", "Value": "2015-12-11T02:56:12Z"}, {"Field": "flavor", "Value": "m1.tiny (1)"}, {"Field": "hostId", "Value": "3c5806fe3f28c64aa63cbbdf023a3790e7d5100c7e396d3516bedebb"}, {"Field": "id", "Value": "b88bb13d-193a-4809-955c-f828dd69e784"}, {"Field": "image", "Value": "cirros-0.3.4-x86_64-uec (d75ab611-8af6-41df-88e2-210526ea1c2b)"}, {"Field": "key_name", "Value": null}, {"Field": "name", "Value": "my_serv"}, {"Field": "os-extended-volumes:volumes_attached", "Value": []}, {"Field": "progress", "Value": 0}, {"Field": "project_id", "Value": "62d97cd7b4674113a3fe1bf2bfdf4702"}, {"Field": "properties", "Value": ""}, {"Field": "security_groups", "Value": [{"name": "default"}]}, {"Field": "status", "Value": "ACTIVE"}, {"Field": "updated", "Value": "2015-12-11T02:56:33Z"}, {"Field": "user_id", "Value": "736e8e5dcb0441748eb91378225c9aa4"}]

(virtualenv)stevemar@ubuntu:~/teuthology$ pip list installed | grep cliff
cliff (1.15.0)
cliff-tablib (1.1)

(virtualenv)stevemar@ubuntu:~/teuthology$ pip list installed | grep client
ndg-httpsclient (0.4.0)
os-client-config (1.12.0)
piston-mini-client (0.7.5)
python-cinderclient (1.5.0)
python-glanceclient (1.2.0)
python-keystoneclient (2.0.0)
python-neutronclient (3.1.0)
python-novaclient (2.35.0)
python-openstackclient (2.0.0)

[Steve Martinelli (stevemar)]

#19 and #20 were both run on a VM with the steps to reproduce the issue

[Steve Martinelli (stevemar)]

Same thing with user show, so it's not just server show.

(virtualenv)stevemar@ubuntu:~/teuthology$ openstack user show admin --format json
[{"Field": "enabled", "Value": true}, {"Field": "id", "Value": "736e8e5dcb0441748eb91378225c9aa4"}, {"Field": "name", "Value": "admin"}, {"Field": "username", "Value": "admin"}]

(virtualenv)stevemar@ubuntu:~/teuthology$ deactivate
stevemar@ubuntu:~/teuthology$ openstack user show admin --format json
{
  "username": "admin",
  "enabled": true,
  "id": "736e8e5dcb0441748eb91378225c9aa4",
  "name": "admin"
}

Can you confirm you are seeing the same behavior?

[Steve Martinelli (stevemar)]

Using a non-teuthology virtual environment, i get normal looking results:

stevemar@ubuntu:~/teuthology$ virtualenv virt1

stevemar@ubuntu:~/teuthology$ source virt1/bin/activate

(virt1)stevemar@ubuntu:~/teuthology$ pip install python-openstackclien

(virt1)stevemar@ubuntu:~/teuthology$ env | grep OS
OS_PASSWORD=openstack
OS_AUTH_URL=http://172.16.240.145:5000/v2.0
OS_USERNAME=admin
OS_TENANT_NAME=admin

(virt1)stevemar@ubuntu:~/teuthology$ openstack server show my_serv --format json
{
  "OS-EXT-STS:task_state": null,
  "addresses": "private=10.0.0.2",
  "image": "cirros-0.3.4-x86_64-uec (d75ab611-8af6-41df-88e2-210526ea1c2b)",
  "OS-EXT-STS:vm_state": "active",
  "OS-EXT-SRV-ATTR:instance_name": "instance-00000001",
  "OS-SRV-USG:launched_at": "2015-12-11T02:56:33.000000",
  "flavor": "m1.tiny (1)",
  "id": "b88bb13d-193a-4809-955c-f828dd69e784",
  "security_groups": [
    {
      "name": "default"
    }
  ],
  "user_id": "736e8e5dcb0441748eb91378225c9aa4",
  "OS-DCF:diskConfig": "MANUAL",
  "accessIPv4": "",
  "accessIPv6": "",
  "progress": 0,
  "OS-EXT-STS:power_state": 1,
  "OS-EXT-AZ:availability_zone": "nova",
  "config_drive": "True",
  "status": "ACTIVE",
  "updated": "2015-12-11T02:56:33Z",
  "hostId": "3c5806fe3f28c64aa63cbbdf023a3790e7d5100c7e396d3516bedebb",
  "OS-EXT-SRV-ATTR:host": "ubuntu",
  "OS-SRV-USG:terminated_at": null,
  "key_name": null,
  "properties": "",
  "project_id": "62d97cd7b4674113a3fe1bf2bfdf4702",
  "OS-EXT-SRV-ATTR:hypervisor_hostname": "ubuntu",
  "name": "my_serv",
  "created": "2015-12-11T02:56:12Z",
  "os-extended-volumes:volumes_attached": []
}

[Steve Martinelli (stevemar)]

I get normal output with:

  - vanilla OSC installed as a system wide package.
  - a vanilla virtualenv (created with `virtualenv virt`), then OSC installed.
  - a bunch of installed stuff n' things from tuethology and not activating the virualenv provided by tuethology.

I get incorrect output with:

  - activating the virtualenv provided by the tuethology scripts.

[Steve Martinelli (stevemar)]

Set the status to 'incomplete', not really sure how you want me to proceed. I don't intend to start digging into tuethology code.

[Loic Dachary (dachary)]

@stevamar here are the minimal steps to reproduce (without teuthology):

provision a fresh Ubuntu 14.04
source openrc.sh
sudo apt-get update
sudo apt-get install -y python-virtualenv python-dev
virtualenv virtualenv
source virtualenv/bin/activate
pip install 'cliff-tablib' 'python-openstackclient'
openstack server show myserver -f json

[Steve Martinelli (stevemar)]

So, I was able to reproduce this on a fresh system with the steps written in #26 - hurray! virtual environments be damned, it happens when installed globally, too. So we can cross that off our list of possible reasons, along with tuethology affecting it.

I checked on my dev environment, where the error *never* occurs (for whatever reason), I installed and uninstalled cliff and cliff-tablib a bunch of times - no change.

On my new fresh box the results were more interesting. If i uninstalled cliff-tablib, then the output was nicely formatted in the new format, as expected. If I re-installed cliff-tablib, it was the old output (Field / Value in the JSON) that's been reported.

Looking at cliff and cliff-tablib, I see that they share similar entry points:

cliff-tablib
---------------
    entry_points={
        'cliff.formatter.list': [...],
        'cliff.formatter.show': [
            'yaml = clifftablib.formatters:YamlFormatter',
            'html = clifftablib.formatters:HtmlFormatter',
            'json = clifftablib.formatters:JsonFormatter',
            ],
        },

cliff
------
cliff.formatter.show =
    table = cliff.formatters.table:TableFormatter
    shell = cliff.formatters.shell:ShellFormatter
    value = cliff.formatters.value:ValueFormatter
    yaml = cliff.formatters.yaml_format:YAMLFormatter
    json = cliff.formatters.json_format:JSONFormatter

Maybe it's the way the entry points are processed? Depending on what package was installed first then that's what it'll use going forward? I don't know. I checked stevedore, and it was at the same level on both machines. I tried re-installing a bunch of times and the behavior persisted.

Short of renaming the entry points, which I'm not even sure would fix the problem, and may cause more problems, I don't know of an easy solution for this issue. Uninstalling cliff-tablib should resolve the issue on any environment. But as @Loic said, it's possible someone is using this for other purposes. My counter to that is there hasn't been a meaningful commit to that repo in over 2.5 years (https://github.com/dreamhost/cliff-tablib/commits/master), and the functionality has been superseded by cliff.

I'm going to move this to new so it shows up in OSC's open bug list.

[Loic Dachary (dachary)]

In light of this, I'll update teuthology to not install cliff-tablib and throw with a message pointing to this bug recommending a) to add a comment so you get a sense of how often (hopefully never) it happens, b) uninstall cliff-tablib. With luck this bug can be closed in a few months because the problem is so rare that nobody complains.

Thanks for investigating the problem :-)

[Steve Martinelli (stevemar)]

Circling back to this one, we managed to talk to the author of cliff-tablib and he released a new version (2.0.0: https://pypi.python.org/pypi/cliff-tablib) that removed the json and yaml formatters. Installing the latest one should no longer create issues for openstackclient.

[Loic Dachary (dachary)]

Thanks for the update !

[Steve Martinelli (stevemar)]

So I think we're good on this one. All new OSC installs should be pulling in a new cliff and no longer depending on cliff-tablib. Marking this as Fix Released, let me know if this is still an issue!

[Loic Dachary (dachary)]

Absolutely :-) Thanks for following up !