Override quantum services autostart

[Vladimir Kuklin]

Currently we have quantum agents autostart override in Ubuntu only for HA mode. We need to fix it to add override file for upstart and remove it after.

The discovery of error was the following:
deploy was successful, even VMs started and get IP. However, metadata service was inaccessible. When we logged in to the controller host, there was no l3-agent process running and no namespace qrouter-XXX.

RabbitMQ had following errors in logs:
=ERROR REPORT==== 3-Dec-2013::21:00:19 ===
closing AMQP connection <0.1239.0> (127.0.0.1:60060 -> 127.0.0.1:5672):
{channel0_error,starting,
                {amqp_error,access_refused,
                            "AMQPLAIN login refused: user 'guest' - invalid credentials",
                            'connection.start_ok'}}

When l3-agent was restored, metadata service became accessible but returning 401 Authorized error. Restart of metadata fixed the issue.

So the issue happens because puppet changes config after service starts with old config (service starts automatically by upstart).

[Mike Scherbakov (mihgen)]

This is ISO #89, simple Ubuntu deployment with Neutron VLAN mode.

[Mike Scherbakov (mihgen)]

Proposed fix: https://github.com/Mirantis/fuel/pull/904 - needs testing.

[Dmitry Borodaenko (angdraug)]

Fix confirmed: Ubuntu HA environment with Neutron VLAN network segmentation deployed successfully with ISO #90 and the above fix applied, there's no errors in RabbitMQ logs.

[Dmitry Borodaenko (angdraug)]

Correction: even though there's no errors in RabbitMQ, l3-agent isn't running and there's no qrouter-* netns. Even though one of the symptoms was fixed, the base problem is still there.

[Mike Scherbakov (mihgen)]

In my case of simple deployment with Quantum in VLAN mode - l3agent started:
root@node-4:~# ps aux|grep l3
quantum 15356 0.4 1.9 103752 29920 ? Ss 06:33 0:00 python /usr/bin/quantum-l3-agent --config-file=/etc/quantum/quantum.conf --config-file=/etc/quantum/l3_agent.ini --log-file=/var/log/quantum/l3-agent.log
root 18333 0.0 0.0 9384 928 pts/3 S+ 06:36 0:00 grep --color=auto l3
root@node-4:~# ip netns list
qrouter-81c21b95-6e5c-4f4b-a76d-31e889e8e170
qdhcp-a1573797-7fc8-4ad9-9640-e1b493c0be64

No errors in l3agent logs found. Instance started and obtained metadata information.

Patch was merged, but the issue needs further investigation as it was not fixed for Dmitry and xenolog also has some concerns about the fix.

[Vladimir Kuklin (vkuklin)]

Dmitry, it really looks like you are mistaking. According to the logs L3-agent was started on the first controller, then it moved to the 3rd controller.

[Mike Scherbakov (mihgen)]

It moves in case of failure, is not it? Why did it move to 3rd controller in Dmitry's env?

[Vladimir Kuklin (vkuklin)]

It is default behaviour. Pacemaker does this for load balancing as soon as
it sees other candidates for service placement. And it does not break
things.
04 дек. 2013 г. 13:25 пользователь "Mike Scherbakov" <
<email address hidden>> написал:

> It moves in case of failure, is not it? Why did it move to 3rd
> controller in Dmitry's env?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1257504
>
> Title:
> Override quantum services autostart
>
> Status in Fuel: OpenStack installer that works:
> Triaged
>
> Bug description:
> Currently we have quantum agents autostart override in Ubuntu only for
> HA mode. We need to fix it to add override file for upstart and remove
> it after.
>
> The discovery of error was the following:
> deploy was successful, even VMs started and get IP. However, metadata
> service was inaccessible. When we logged in to the controller host, there
> was no l3-agent process running and no namespace qrouter-XXX.
>
> RabbitMQ had following errors in logs:
> =ERROR REPORT==== 3-Dec-2013::21:00:19 ===
> closing AMQP connection <0.1239.0> (127.0.0.1:60060 -> 127.0.0.1:5672):
> {channel0_error,starting,
> {amqp_error,access_refused,
> "AMQPLAIN login refused: user 'guest' -
> invalid credentials",
> 'connection.start_ok'}}
>
> When l3-agent was restored, metadata service became accessible but
> returning 401 Authorized error. Restart of metadata fixed the issue.
>
> So the issue happens because puppet changes config after service
> starts with old config (service starts automatically by upstart).
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fuel/+bug/1257504/+subscriptions
>

[Mike Scherbakov (mihgen)]

I tested new patchset from Andrey https://github.com/Mirantis/fuel/pull/905, - deployed 3 environments with one controller in each. Verified that namespaces created and l3 agent running.