Bad wheel with lxml 3.7.0

[John L. Villalovos]

CI Jobs are failing across OpenStack with an error like this:

http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_787

2016-12-22 22:25:08.714 | 2016-12-22 22:25:08.714 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Granted admin on admin to user admin.
2016-12-22 22:25:08.731 | 2016-12-22 22:25:08.730 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created region RegionOne
2016-12-22 22:25:08.763 | 2016-12-22 22:25:08.763 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created admin endpoint http://15.184.67.84/identity_admin
2016-12-22 22:25:08.775 | 2016-12-22 22:25:08.774 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created internal endpoint http://15.184.67.84/identity
2016-12-22 22:25:08.787 | 2016-12-22 22:25:08.787 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created public endpoint http://15.184.67.84/identity
2016-12-22 22:25:08.793 | 2016-12-22 22:25:08.793 31714 INFO keystone.assignment.core [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Creating the default role 9fe2ff9ee4384b1894a90878d3e92bab because it does not exist.
2016-12-22 22:25:08.923 | + ./stack.sh:main:1065 : create_keystone_accounts
2016-12-22 22:25:08.927 | + lib/keystone:create_keystone_accounts:371 : local admin_project
2016-12-22 22:25:08.931 | + lib/keystone:create_keystone_accounts:372 : openstack project show admin -f value -c id
2016-12-22 22:25:10.787 | /opt/stack/new/devstack/lib/keystone: line 366: 31727 Illegal instruction openstack project show "admin" -f value -c id
2016-12-22 22:25:10.791 | + lib/keystone:create_keystone_accounts:372 : true
2016-12-22 22:25:10.796 | ++ lib/keystone:create_keystone_accounts:373 : openstack project show admin -f value -c id
2016-12-22 22:25:12.589 | + lib/keystone:create_keystone_accounts:373 : admin_project=
2016-12-22 22:25:12.593 | + lib/keystone:create_keystone_accounts:1 : exit_trap
2016-12-22 22:25:12.596 | + ./stack.sh:exit_trap:487 : local r=132
2016-12-22 22:25:12.602 | ++ ./stack.sh:exit_trap:488 : jobs -p
2016-12-22 22:25:12.606 | + ./stack.sh:exit_trap:488 : jobs=31048
2016-12-22 22:25:12.610 | + ./stack.sh:exit_trap:491 : [[ -n 31048 ]]
2016-12-22 22:25:12.614 | + ./stack.sh:exit_trap:491 : [[ -n /opt/stack/new/devstacklog.txt.2016-12-22-221622 ]]
2016-12-22 22:25:12.617 | + ./stack.sh:exit_trap:491 : [[ True == \T\r\u\e ]]
2016-12-22 22:25:12.621 | + ./stack.sh:exit_trap:492 : echo 'exit_trap: cleaning up child processes'
2016-12-22 22:25:12.621 | exit_trap: cleaning up child processes
2016-12-22 22:25:12.625 | + ./stack.sh:exit_trap:493 : kill 31048
2016-12-22 22:25:12.625 | ./stack.sh: line 493: kill: (31048) - No such process

[Matt Riedemann (mriedem)]

The key part of that trace is this:

2016-12-22 22:25:10.787 | /opt/stack/new/devstack/lib/keystone: line 366: 31727 Illegal instruction openstack project show "admin" -f value -c id

[Matt Riedemann (mriedem)]

It could be this:

https://review.openstack.org/#/c/413771/

[John L. Villalovos (happycamp)]

Logstash, can take a long time for data to populate in browser and also might need to change the date range to see anything.

http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22lib%2Fkeystone:create_keystone_accounts:1%20%20:%20%20%20exit_trap%5C%22

[Matt Riedemann (mriedem)]

It's not the osc update in comment 2. It's also hitting unit test jobs:

http://logs.openstack.org/05/378105/14/check/gate-cinder-python34-db/01d4149/console.html#_2016-12-22_22_18_35_037491

So likely something with infra.

[Matt Riedemann (mriedem)]

http://logs.openstack.org/41/414341/1/check/gate-tempest-dsvm-platform-ubuntu-trusty/349e2a0/logs/syslog.txt.gz#_Dec_22_22_30_54

Dec 22 22:30:54 ubuntu-trusty-rax-iad-6364931 kernel: [ 1058.679590] traps: openstack[25170] trap invalid opcode ip:7fced9fd1bee sp:7ffd5f3de398 error:0 in etree.so[7fced9db3000+4b7000]

[John L. Villalovos (happycamp)]

Seems like the error is in etree.so:
http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/syslog.txt.gz#_Dec_22_22_25_10

Same process ID as above:
Dec 22 22:25:10 ubuntu-xenial-infracloud-vanilla-6365055 kernel: traps: openstack[31727] trap invalid opcode ip:7f78c2b6ebee sp:7ffddd576298 error:0 in etree.so[7f78c2950000+4b7000]

[John L. Villalovos (happycamp)]

johnsonm discovered new release of lxml today:

https://pypi.python.org/pypi/lxml/3.7.0

[Matt Riedemann (mriedem)]

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22trap%20invalid%20opcode%5C%22%20AND%20message%3A%5C%22in%20etree.so%5C%22&from=7d

[Matt Riedemann (mriedem)]

upstream lxml bug: https://bugs.launchpad.net/lxml/+bug/1652191

[Matt Riedemann (mriedem)]

https://bugs.launchpad.net/lxml/+bug/1652191 is fixed so we can close this bug.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/requirements 1.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/requirements ocata-eol release.