Neutron DHCP agent failed to survive AMQP failover: UX after reboot of controller and compute instance can not get an address from DHCP

[Łukasz Oleś]

Steps to reproduce:

- Deploy Fuel ISO using VirtualBox
- Cteate env using default settings with one Controller and one Compute
- login to Horizon and create one instance and assign it net04 network
- lunch an instance
- instance should get ip from 192.168.111.0 network
- reboot Controller and Compute

After reboot the instance is started but it can not get the addres. New instances also starts but do not get address from DHCP.

{
   "build_id":"2015-08-10_19-24-50",
   "build_number":"157",
   "release_versions":{
      "2015.1.0-7.0":{
         "VERSION":{
            "build_id":"2015-08-10_19-24-50",
            "build_number":"157",
            "api":"1.0",
            "fuel-library_sha":"c4b2b9a1b464d7f538635cbfc9304a714ededc41",
            "nailgun_sha":"9a6ac9b08733ded67bc65345a37787886ff4249d",
            "feature_groups":[
               "mirantis"
            ],
            "fuel-nailgun-agent_sha":"e01693992d7a0304d926b922b43f3b747c35964c",
            "openstack_version":"2015.1.0-7.0",
            "fuel-agent_sha":"57145b1d8804389304cd04322ba0fb3dc9d30327",
            "production":"docker",
            "python-fuelclient_sha":"ec5c02b3848fe1d15e5b50b323c3dda030f8e1ac",
            "astute_sha":"e1d3a435e5df5b40cbfb1a3acf80b4176d15a2dc",
            "fuel-ostf_sha":"c7f745431aa3c147f2491c865e029e0ffea91c47",
            "release":"7.0",
            "fuelmain_sha":"826387d1a05a40e4849433598442024bb1e3b7c7"
         }
      }
   },
   "auth_required":true,
   "api":"1.0",
   "fuel-library_sha":"c4b2b9a1b464d7f538635cbfc9304a714ededc41",
   "nailgun_sha":"9a6ac9b08733ded67bc65345a37787886ff4249d",
   "feature_groups":[
      "mirantis"
   ],
   "fuel-nailgun-agent_sha":"e01693992d7a0304d926b922b43f3b747c35964c",
   "openstack_version":"2015.1.0-7.0",
   "fuel-agent_sha":"57145b1d8804389304cd04322ba0fb3dc9d30327",
   "production":"docker",
   "python-fuelclient_sha":"ec5c02b3848fe1d15e5b50b323c3dda030f8e1ac",
   "astute_sha":"e1d3a435e5df5b40cbfb1a3acf80b4176d15a2dc",
   "fuel-ostf_sha":"c7f745431aa3c147f2491c865e029e0ffea91c47",
   "release":"7.0",
   "fuelmain_sha":"826387d1a05a40e4849433598442024bb1e3b7c7"
}

[Bogdan Dobrelya (bogdando)]

The repro steps seem doesn't match the logs, there was a "time jump":
2015-08-12T23:22:57.773286+00:00 info: INFO: p_rabbitmq-server: get_monitor(): get_monitor function ready to return 8
2015-08-13T06:32:43.689554+00:00 info: INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0

Please elaborate which exactly actions were taken?

[Bogdan Dobrelya (bogdando)]

The RC is DHCP agent's Oslo messaging code failed to survive AMQP failover: http://pastebin.com/9i0FH6Rv

[Bogdan Dobrelya (bogdando)]

This is a dup of Oslo bug https://bugs.launchpad.net/oslo.messaging/+bug/1338732

[Davanum Srinivas (DIMS) (dims-v)]

Please reply to Bogdan's query in comment #2

[Łukasz Oleś (loles)]

Time jump has nothing to do with this bug. I installed Openstack on one day. Hibernated my laptop and on second day I started the tests.

[Davanum Srinivas (DIMS) (dims-v)]

How many times has this happened? Is this recreate able?

[Łukasz Oleś (loles)]

For me it happens every time after I reboot controller and compute nodes.

[Davanum Srinivas (DIMS) (dims-v)]

@loles, thanks for the info. Since this is a virtualbox environment, i am not sure if this merits a "High" Importance. If you do get a chance, please do a fresh install from the latest ISO, try doing what you are doing immediately after the install is over (and NOT hibernate the laptop etc). if that works and the problems start happening after hibernate, that would give us some data. Also, we would not expect production installations to hibernate either :)

[Dmitry Mescheryakov (dmitrymex)]

Lukasz, please move the bug to new state in case it is reproducible without hibernation. I am afraid that OpenStack is too imperfect yet for us to support hibernation of the host OS :-)

[Vitaly Sedelnik (vsedelnik)]

This issue happens in VB env only after hibernation. Changing the importance to Medium, Won't Fix for 7.0, Confirmed for 8.0

[Dmitry Mescheryakov (dmitrymex)]

Somehow the issue became confirmed again. Moving it back to incomplete. Łukasz, please move the bug to new state in case it is reproducible without hibernation.

[Maciej Relewicz (rlu)]

After talk to Lukasz, I closed the bug.

[Dina Belova (dbelova)]

Moving to Invalid after more than a month in the incomplete state.