nova network-vif-plugged event fails with 400 from n-api
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Undecided
|
Unassigned |
Bug Description
Running Tempest in our internal CI we hit 3 test failures out of 1,971 tests, one was:
tempest.
This is running x86_64 RHEL 6.5 with the nova libvirt driver and neutron openvswitch.
We have neutron.conf configured for nova events:
[root@rhel62 ~]# cat /etc/neutron/
# being used in conjunction with nova security groups
# allowed_
nova_url = http://
nova_region_name = RegionOne
nova_admin_username = nova
nova_admin_
nova_admin_password = nova
nova_admin_auth_url = http://
For one failure I'm seeing this in the neutron server log:
2014-03-16 05:57:36.619 8462 ERROR neutron.
2014-03-16 05:57:36.619 8462 TRACE neutron.
2014-03-16 05:57:36.619 8462 TRACE neutron.
2014-03-16 05:57:36.619 8462 TRACE neutron.
2014-03-16 05:57:36.619 8462 TRACE neutron.
2014-03-16 05:57:36.619 8462 TRACE neutron.
2014-03-16 05:57:36.619 8462 TRACE neutron.
2014-03-16 05:57:36.619 8462 TRACE neutron.
2014-03-16 05:57:36.619 8462 TRACE neutron.
2014-03-16 05:57:36.619 8462 TRACE neutron.
2014-03-16 05:57:36.619 8462 TRACE neutron.
2014-03-16 05:57:36.619 8462 TRACE neutron.
2014-03-16 05:57:36.619 8462 TRACE neutron.
2014-03-16 05:57:36.619 8462 TRACE neutron.
2014-03-16 05:57:36.619 8462 TRACE neutron.
2014-03-16 05:57:36.619 8462 TRACE neutron.
2014-03-16 05:57:36.619 8462 TRACE neutron.
2014-03-16 05:57:36.619 8462 TRACE neutron.
I correlated that server uuid to the nova-api log error here:
2014-03-16 05:57:27.770 12207 INFO nova.osapi_
2014-03-16 05:57:36.615 12204 ERROR nova.api.
2014-03-16 05:57:36.615 12204 TRACE nova.api.
2014-03-16 05:57:36.615 12204 TRACE nova.api.
2014-03-16 05:57:36.615 12204 TRACE nova.api.
2014-03-16 05:57:36.615 12204 TRACE nova.api.
2014-03-16 05:57:36.615 12204 TRACE nova.api.
2014-03-16 05:57:36.615 12204 TRACE nova.api.
2014-03-16 05:57:36.615 12204 TRACE nova.api.
2014-03-16 05:57:36.616 12204 INFO nova.osapi_
Talking with Dan Smith in IRC it sounds like maybe neutron isn't putting the nova tenant ID into the request for some reason?
(4:15:57 PM) dansmith: mriedem: I think neutron sometimes gets the nova url from the context you pass it, so maybe it's related to that
...
(4:17:20 PM) dansmith: mriedem: right, but arosen did some work to have neutron determine the proper nova endpoint to call back to based on the context, in the case you've got two novas sharing a neutron
This is intermittent, but I do see the neutron server error showing up in community logstash, but only 7% failures:
I'm not seeing the same error in logstash in nova-api logs, but that was only going back the last 48 hours.
Michael Kazakov (gnomino) wrote : | #2 |
I have same bug and instances fails to run:
2014-04-07 14:16:08.829 28157 WARNING nova.virt.
2014-04-07 14:16:09.817 28157 INFO nova.virt.
2014-04-07 14:16:09.818 28157 INFO nova.virt.
2014-04-07 14:16:09.912 28157 ERROR nova.compute.
2014-04-07 14:16:09.912 28157 TRACE nova.compute.
2014-04-07 14:16:09.912 28157 TRACE nova.compute.
2014-04-07 14:16:09.912 28157 TRACE nova.compute.
2014-04-07 14:16:09.912 28157 TRACE nova.compute.
2014-04-07 14:16:09.912 28157 TRACE nova.compute.
2014-04-07 14:16:09.912 28157 TRACE nova.compute.
2014-04-07 14:16:09.912 28157 TRACE nova.compute.
2014-04-07 14:16:09.912 28157 TRACE nova.compute.
2014-04-07 14:16:09.912 28157 TRACE nova.compute.
Michael Kazakov (gnomino) wrote : | #3 |
DEBUG nova.api.
lugged", "server_uuid": "c7a6f6bf-
......
TypeError: multi() got an unexpected keyword argument 'body'
Matt Riedemann (mriedem) wrote : | #4 |
@gnomino - are you running with the latest nova code on milestone-proposed? I haven't seen this in our internal CI since it was reported.
Michael Kazakov (gnomino) wrote : | #5 |
I am running latest nova code in ubuntu 14.04 repo:
ii nova-api 1:2014.
ii nova-cert 1:2014.
ii nova-common 1:2014.
ii nova-conductor 1:2014.
ii nova-consoleauth 1:2014.
ii nova-novncproxy 1:2014.
ii nova-scheduler 1:2014.
ii python-nova 1:2014.
ii python-novaclient 1:2.17.0-0ubuntu1 all client library for OpenStack Compute API
I have same bug too. I confirmed nova, neutron RC1.
Michael Kazakov (gnomino) wrote : | #7 |
Still present after nova updates to version 1:2014.
Matt Riedemann (mriedem) wrote : | #8 |
Is everyone hitting this intermittently or are you able to reproduce easily?
Matt Riedemann (mriedem) wrote : | #9 |
Also, what version of novaclient/
Michael Kazakov (gnomino) wrote : | #10 |
On every instance creation. Creation fails with error.
Michael Kazakov (gnomino) wrote : | #11 |
Ubuntu 14.04
python-novaclient 1:2.17.0-0ubuntu1
python-
python-
Matt Riedemann (mriedem) wrote : | #12 |
Michael, just to confirm, you're getting this on every boot request?
"BadRequest: The server could not comply with the request since it is either malformed or otherwise incorrect. (HTTP 400)"
Are you using devstack?
Michael Kazakov (gnomino) wrote : | #13 |
Yes. on every boot request. I don't use devstack
Arun Thulasi (arunuke) wrote : | #14 |
I am running into the same issue as well (RDO/Fedora 20/Icehouse). I am unable to delete older instances that have been marked as "ERROR" as well (ie, nova delete <instance name> does not remove them).
Matt Riedemann (mriedem) wrote : | #15 |
Arun, you're looking for bug 1299139 for the problem with not being able to delete instances in error state. You have to restart the compute service to clean those up unfortunately.
Matt Riedemann (mriedem) wrote : | #16 |
Michael/Arun, what does this look like?
cat /etc/neutron/
Need to make sure you have the nova config options set correctly in neutron.conf for callbacks.
Arun Thulasi (arunuke) wrote : | #17 |
Matt's message #15 just needed a "elementary, my dear watson" to be complete. A nova-compute restart fixed the issue. Will rebuild the environment and try and get an answer for message #16 later tonight.
From what I remember, I have not specified any nova specific option in my neutron.conf file on the cloud or network controller nodes since I am not using the ML2 plugin. The custom set values are for neutron plugins, keystone credentials and databases and that could very well be the issue.
Do I need to set nova callbacks even if I am not using ML2, but using only openvswitch?
I'm seeing this on every boot as well. ubuntu 14.04 same client version as Michael above. None of my nova.conf's (ie compute, controller or network) have any uncommented nova options. I see things in there like
# nova_url = http://
which if that's a default could be "bad", right?
what nova options should be enabled and on which types of hosts?
Matt Riedemann (mriedem) wrote : | #19 |
This is going to be a problem if you're using ML2 or not, this is something that needs to be configured in neutron.conf for boots to work with nova if you're using neutron at all. See the devstack patch upstream to configure this:
https:/
If you're using RDO you need to get a bug report opened against RDO. I know there was a patch upstream in the chef cookbooks on stackforge, I'd have to dig that up now. But basically anyone writing their own install scripts is going to have to adjust for this change like devstack did above.
Arun Thulasi (arunuke) wrote : | #20 |
Enabling "debug" for neutron on the cloud controller node shows an attempt to communicate with 127.0.0.1 for notifications if it's not specifically set to the cloud controller's IP as the newer version of the guide recommends. However, I was under the assumption that it needs to be set only if neutron.conf sets nova notification changes to "True". I explicitly set both notifications to False on all my nodes' neutron.conf files, and yet I hit a failure when I try to boot with no new notifications on server.log for neutron on the cloud controller.
As a next step, I will try and set all the nova parameters within neutron.conf on all the nodes to see if that helps.
The install guide only documents these steps for the ML2 plugin. That might the source of some confusion. I'm testing the steps in that section for OVS and if all good will log a bug / fix for the guides.
when setting this on the controller
notify_
notify_
nova_url = http://
nova_admin_username = nova
nova_admin_
nova_admin_password = novapass
nova_admin_auth_url = http://
i still see errors. should it be set on controllers, network and compute?
Michael Kazakov (gnomino) wrote : | #23 |
My nova settings in neutron.conf is:
notify_
notify_
nova_url = http://
nova_region_name = RegionOne
nova_admin_username = nova
nova_admin_
nova_admin_password = openstack
nova_admin_auth_url = http://
I am using OVS via ml2 core plugin. I have tried to use OVS neuton core plugin but received the same error.
I've set all the nova values in neutron.conf on all node types as:
notify_
notify_
nova_url = http://
nova_region_name = regionOne
nova_admin_username = nova
nova_admin_
nova_admin_password = novapass
nova_admin_auth_url = http://
using the "services" tenant as per the install guide notes for ml2.
I'm using OVS NOT ml2.
It still fails with:
2014-04-11 14:39:09.258 928 ERROR neutron.
2014-04-11 14:39:09.258 928 TRACE neutron.
2014-04-11 14:39:09.258 928 TRACE neutron.
2014-04-11 14:39:09.258 928 TRACE neutron.
2014-04-11 14:39:09.258 928 TRACE neutron.
2014-04-11 14:39:09.258 928 TRACE neutron.
2014-04-11 14:39:09.258 928 TRACE neutron.
2014-04-11 14:39:09.258 928 TRACE neutron.
2014-04-11 14:39:09.258 928 TRACE neutron.
2014-04-11 14:39:09.258 928 TRACE neutron.
2014-04-11 14:39:09.258 928 TRACE neutron.
2014-04-11 14:39:09.258 928 TRACE neutron.
2014-04-11 14:39:09.258 928 TRACE neutron.
2014-04-11 14:39:09.258 928 TRACE neutron.
2014-04-11 14:39:09.258 928 TRACE neutron.
2014-04-11 14:39:09.258 928 TRACE neutron.
2014-04-11 14:39:09.258 928 TRACE neutron.
2014-04-11 14:39:09.258 928 TRACE neutron.
Sorry - corresponding api log output is:
2014-04-11 14:39:09.256 1391 ERROR nova.api.
2014-04-11 14:39:09.256 1391 TRACE nova.api.
2014-04-11 14:39:09.256 1391 TRACE nova.api.
2014-04-11 14:39:09.256 1391 TRACE nova.api.
2014-04-11 14:39:09.256 1391 TRACE nova.api.
2014-04-11 14:39:09.256 1391 TRACE nova.api.
2014-04-11 14:39:09.256 1391 TRACE nova.api.
2014-04-11 14:39:09.256 1391 TRACE nova.api.
Henry Gessau (gessau) wrote : | #26 |
Check if specifying the API version works.
See https:/
Thiago Martins (martinx) wrote : | #27 |
Henry,
Now it is working!
I replaced this line:
nova_url = http://
For this one:
nova_url = http://
Now it works!
First Instance created using IceHouse! YAY! :-P
I'm using ML2 with Flat network, Ubuntu 14.04 upgraded today.
Michael Kazakov (gnomino) wrote : | #28 |
After changing nova_url to http://
VirtualInterfac
I think that this problem is solved. A problem with running instances rooted in something else.
Thiago Martins (martinx) wrote : | #29 |
Off-Topic:
Next problem: https:/
Michael Kazakov (gnomino) wrote : | #30 |
Instances runs well, many thanks!
me too. looks better with /v2/ and me properly setting nova values in neutron configs on controller.
thanks for the help!
Matt Riedemann (mriedem) wrote : | #32 |
If you're using puppet, track bug 1306694.
Arun Thulasi (arunuke) wrote : | #33 |
After adding the nova notification changes to all the neutron.conf files (controller, network and compute) I am able to successfully boot instances as well - thanks to all the folks who helped.
I am still having some connectivity issues where dhcp requests from the compute nodes are not reaching the network node. I see my bootp requests coming out of br-eth1 on the compute node, but I don't see them coming back up on the br-eth1 node on the network node. Will keep this list posted on what I find out.
Matt Riedemann (mriedem) wrote : | #34 |
For other network issues, make sure you're not hitting bug 1284718 or bug 1302796.
Arun Thulasi (arunuke) wrote : | #35 |
Thanks for the pointers Matt. I am afraid I am not seeing either of these issues. The trouble I am having is that packets are not going out of the compute host. I see the packets on br-eth1 of the compute node, but not on br-eth1 of the network node.
When I try to ping a VM from compute host 'A' to another VM on compute host 'B", I see the ICMP packets on br-eth1 for host-A, but no on host "B".
I am still using VLAN (not GRE) and OpenVSwitch (not ML2) on Icehouse/Fedora20. The guide clearly mentions that ML2 is tested while OpenVSwitch is not, so I am not sure if it's related to that.
Matt Riedemann (mriedem) wrote : | #36 |
Arun, sounds like a neutron issue, there was a change related to vif and firewalls on icehouse milestone-proposed, maybe you're having issues there? I thought neutron was tested with openvswitch ML2 mechanism driver in the gate?
Matt Riedemann (mriedem) wrote : | #37 |
Arun, bug 1297469 is the vif and firewall related issue I was thinking of.
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit c09a14089a5ca7c
Author: Dan Prince <email address hidden>
Date: Thu Apr 10 12:40:13 2014 -0400
Make default nova_url use a version
The default nova_url for neutron is missing an API
version number. This can cause requests to fail
because the Nova /versions API cannot respond
to Neutron notification requests.
It seems reasonable for the default value to
at least have a chance at being correct so
this patch upgrades the default Nova API url to
use the Nova 'v2' API.
Related-bug: #1298640
Change-Id: Ib1449de84fbc01
Arun Thulasi (arunuke) wrote : | #39 |
Matt - thanks again. A quick update from me. I was able to resolve one issue when I noticed my physical interface (eth1, on top of which i have my bridge br-eth1) was marked as down. I was able to address that and now I see packets moving between hosts.
On system boot, dhcp is able to assign an address and it is configured as expected on the compute VMs. Checking the interface status using ovs-ofctl helped me in finding that the link was down.
However, when I try to ping the address from my network node, it does not get past int-br-eth1 on the compute host. I will keep this list posted on what I find.
Related fix proposed to branch: stable/icehouse
Review: https:/
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/icehouse
commit d1ab56d8c4e19c2
Author: Dan Prince <email address hidden>
Date: Thu Apr 10 12:40:13 2014 -0400
Make default nova_url use a version
The default nova_url for neutron is missing an API
version number. This can cause requests to fail
because the Nova /versions API cannot respond
to Neutron notification requests.
It seems reasonable for the default value to
at least have a chance at being correct so
this patch upgrades the default Nova API url to
use the Nova 'v2' API.
Related-bug: #1298640
Change-Id: Ib1449de84fbc01
(cherry picked from commit c09a14089a5ca7c
tags: | added: in-stable-icehouse |
tags: | removed: in-stable-icehouse |
Matt Riedemann (mriedem) wrote : | #42 |
Looks like bug 1306727 has a fix for the "TypeError: multi() got an unexpected keyword argument 'body'" issue, so between that and the config issues for neutron/nova events, I think we can probably just close this out as fixed by other patches.
no longer affects: | neutron |
hugo deng (dengqm) wrote : | #43 |
hi everybody,when i use icehouse i got the same error:
HTTP/1.1 400 Bad Request; content: [{"badRequest": {"message": "The server could not comply with the request since it is either malformed or otherwise incorrect.", "code": 400}}]
and my configuration like this:
nova_url = http://
nova_admin_username = nova
nova_admin_
nova_admin_password = 111111
nova_admin_auth_url = http://
the "nova_url" config is right,why got this error?and what i need do?
Baohua Yang (yangbaohua) wrote : | #44 |
The same error happens with devstack + stable/juno branch.
The nova_url is correct, while the vm cannot spawn out successfully.
Oh and this is nova/neutron code from master as of 3/26, and novaclient is 2.17 and neutronclient is 2.3.4.