Document Nova v2 Cells and Placement API

[Brian Moss]

During an Installation Guide CentOS install, attempting to launch an instance hits this issue:

https://bugzilla.redhat.com/show_bug.cgi?id=1405098

It appears that we need to document how to add a cell manually in the Install Guide.

[Alexandra Settle (alexandra-settle)]

bmoss - perhaps it's best to contact Stephen Gordon on this one?

[Alexandra Settle (alexandra-settle)]

Stephen - are you able to help here?

[Stephen Gordon (sgordon)]

Hi Alex,

As a result of the issues uncovered in developing/testing TripleO for Ocata (as touched on in that and other RHBZ entries) there were both some improvements to the Cells V2 stuff in Nova itself and Dan Smith and Melanie Witt provided some additional documentation in the devref. This should give you some good source material:

https://docs.openstack.org/developer/nova/cells.html#setup-of-cells-v2

The other, unrelated, issue you will want to watch out for if it's not already tracked is that Nova in Ocata also requires that a new Nova service be setup, the Placement API:

https://docs.openstack.org/developer/nova/placement.html#deployment

Thanks,

Steve

[Brian Moss (bmoss)]

Thanks Stephen. Just found the placement issue in my test deployment.

Alex, I'll take this on and track both issues in this bug.

[Brian Moss (bmoss)]

Ok, I've made progress. I believe I've got most of the steps for Cells and the Placement API sorted for the controller node. On Monday I'll configure the Compute node and test everything.

[Brian Moss (bmoss)]

Nearly there. Having trouble with Placement permissions. Shall continue tomorrow.

[Darren Chan (dazzachan)]

Added Amy to the bug report.

[Amy Marrich (amy-marrich)]

So here's where I'm at on this is I can at least get past being stuck on BUILD(that's when I found out about placement and cells), but I get No Valid Hosts with the current install of placement and cells. I'm trying another install currently doing the placement and cells at the same time as nova/nova-api in case there's an issue of doing it after everything else. I'll be happy to share what I've got so far to the docs list or here if it'll be helpful. Also on IRC as spotz

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/438328

[Brian Moss (bmoss)]

So I tossed up a WIP patch to show you what I threw together for testing. Things seemed to work except that the compute node was getting permission errors (access forbidden) when trying to access the Placement database.

https://review.openstack.org/#/c/438328

From what I've read, the order of things is important, but I'm not sure how important.

[Amy Marrich (amy-marrich)]

Brian,

I've heard especially the DB steps are important and I think I might have more steps then you as well. Let me know if you want me to comment on the patch or mail my steps to you. I have yet to get an instance to boot, so definitely not saying what I have is right.

Amy

[Brian Moss (bmoss)]

Whatever you think is going to be easiest. Feel free to push another patchset to that review if you'd like. Or just start another review entirely. I only put that one up to show you what I'd tried so far.

[Brian Moss (bmoss)]

Oh, I see you've already made some comments on the patch. I'll incorporate those and we can continue iterating there.

[Amy Marrich (amy-marrich)]

I'm still getting No Valid Hosts and not sure what's going on at this point.

On Mon, Feb 27, 2017 at 5:11 PM, Brian Moss <email address hidden> wrote:

> Oh, I see you've already made some comments on the patch. I'll
> incorporate those and we can continue iterating there.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1663485
>
> Title:
> Document Nova v2 Cells and Placement API
>
> Status in openstack-manuals:
> In Progress
>
> Bug description:
> During an Installation Guide CentOS install, attempting to launch an
> instance hits this issue:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1405098
>
> It appears that we need to document how to add a cell manually in the
> Install Guide.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/openstack-manuals/+bug/1663485/+subscriptions
>

[Georgios Dimitrakakis (giorgis-r)]

Hi Brian,

so if someone has followed the guide without your additions can he now do the extra steps as you have provide them in the https://review.openstack.org/#/c/438328 and have a working environment?

Is something else missing?

Best regards,

George

[Roger Luethi (rl-o)]

I implemented the changes in https://review.openstack.org/#/c/438328/3/ for the training-labs scripts (on Ubuntu).

I had to make some changes (/etc/httpd/httpd.conf is probably RDO, on Ubuntu it's /etc/apache2/apache2.conf) and I may well have made mistakes. Still, for what it is worth, here is what I got when trying to launch an instance:

- nova-conductor.log: "No valid host was found."
- nova-api.log: "Flavor m1.nano could not be found" (the flavor exists)

After about ten attempts to create an instance:
- nova-api.log: "Quota exceeded for instances" (no instances in server list)

[Georgios Dimitrakakis (giorgis-r)]

I was able to launch a Cirros image but I am not able to launch a CentOS and I am getting a "No valid host was found" error. Is that possible?

I mean is it possible to be able to launch some instances while not some others due to this bug?

Regards,

George

[Georgios Dimitrakakis (giorgis-r)]

I forgot to mention that if someone would like to review the steps followed until I was able to launch a Cirros intance I would be more than happy to post them.

Best,

George

[Thiago Martins (martinx)]

I'm testing the reviewing / testing procedure as we speak! On Ubuntu 16.04
with Ocata UCA.

On 6 March 2017 at 17:21, Georgios Dimitrakakis <email address hidden>
wrote:

> I forgot to mention that if someone would like to review the steps
> followed until I was able to launch a Cirros intance I would be more
> than happy to post them.
>
> Best,
>
> George
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1669279).
> https://bugs.launchpad.net/bugs/1663485
>
> Title:
> Document Nova v2 Cells and Placement API
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/openstack-manuals/+bug/1663485/+subscriptions
>

[Georgios Dimitrakakis (giorgis-r)]

All,
I am able to launch Cirros, CentOS and Debian instances by carefully following the changes that are under review here.
My previous problem was due to not sufficient disk space on the compute node and had nothing to do with this bug!

Best,

George

[Thiago Martins (martinx)]

New instructions looks good!

I found it weird that we have to run "su -s /bin/sh -c "nova-manage db
sync" nova" twice but, that's okay...

Also, another problem, there is no mention to the following nova.conf
config group:

---
[scheduler]
discover_hosts_in_cells_interval = 300
---

This is very important! This way, there is no need to run:

su -s /bin/sh -c "nova-manage cell_v2 discover_hosts" nova

...every time you add an hypervisor, which is a bad idea anyway, right?

On 8 March 2017 at 11:44, Georgios Dimitrakakis <email address hidden>
wrote:

> All,
> I am able to launch Cirros, CentOS and Debian instances by carefully
> following the changes that are under review here.
> My previous problem was due to not sufficient disk space on the compute
> node and had nothing to do with this bug!
>
> Best,
>
> George
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1669279).
> https://bugs.launchpad.net/bugs/1663485
>
> Title:
> Document Nova v2 Cells and Placement API
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/openstack-manuals/+bug/1663485/+subscriptions
>

[Brian Moss (bmoss)]

Thanks everyone for trying this out! We're still working on improving https://review.openstack.org/#/c/438328. Currently, I can't get an instance running with an RDO deployment, but I think we're getting closer to the correct procedures.

[Amy Marrich (amy-marrich)]

Apparently one of the issues we're seeing during testing is a bug, this is the No Quota issue

https://bugs.launchpad.net/nova/+bug/1670627

[Roger Luethi (rl-o)]

Regarding nova-api.log: "Flavor m1.nano could not be found"

This error can be ignored.

This exception is helpfully thrown if you refer to the flavor by name instead of by its id. That was already the case in Newton and the instances launched anyway.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/438328
Committed: https://git.openstack.org/cgit/openstack/openstack-manuals/commit/?id=d6a9c104454fb5628a05df1a9342f0e27e785195
Submitter: Jenkins
Branch: master

commit d6a9c104454fb5628a05df1a9342f0e27e785195
Author: Brian Moss <email address hidden>
Date: Tue Feb 21 16:05:31 2017 +1000

    [Install] Document Nova cells and placement API

    With the Ocata release, Nova requires a basic cell_v2
    setup and the Placement API service.

    Change-Id: I3ad1a53ea072f79cbc356248b0b800ddaabd5038
    Closes-Bug: #1663485
    Closes-Bug: #1669163

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-manuals 15.0.0 release.

[Thiago Martins (martinx)]

The following entry in nova.conf is missing from the fix:

---
[scheduler]
discover_hosts_in_cells_interval = 300
---

[Brian Moss (bmoss)]

Thiago, since we instruct the user to run `nova manage cell_v2 discover_hosts` after configuring the compute node, do you think we need to change the default -1 interval?

[Roger Luethi (rl-o)]

Since the people who tested the Ocata install-guide are conveniently still subscribed to this bug, let me ask here if anyone tested the Dashboard on Ubuntu. Because it seems that the Ubuntu package has the secret_key (/var/lib/openstack-dashboard/secret_key) readable only for root which breaks the Dashboard.

[Darren Chan (dazzachan)]

Hi Roger,

I have been testing the Ocata install-guide and have a similar issue with no access to Dashboard.

The reported error /var/log/apache2/error.log:

[Tue Mar 28 06:20:24.264312 2017] [wsgi:error] [pid 25585:tid 140667617994496] [remote 10.1.11.11:194] mod_wsgi (pid=25585): Target WSGI script '/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.w$
[Tue Mar 28 06:20:24.264358 2017] [wsgi:error] [pid 25585:tid 140667617994496] [remote 10.1.11.11:194] mod_wsgi (pid=25585): Exception occurred processing WSGI script '/usr/share/openstack-dashboard/openstack_$
[Tue Mar 28 06:20:24.264383 2017] [wsgi:error] [pid 25585:tid 140667617994496] [remote 10.1.11.11:194] Traceback (most recent call last):
[Tue Mar 28 06:20:24.264406 2017] [wsgi:error] [pid 25585:tid 140667617994496] [remote 10.1.11.11:194] File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi", line 16, in <module>
[Tue Mar 28 06:20:24.264462 2017] [wsgi:error] [pid 25585:tid 140667617994496] [remote 10.1.11.11:194] application = get_wsgi_application()
[Tue Mar 28 06:20:24.264475 2017] [wsgi:error] [pid 25585:tid 140667617994496] [remote 10.1.11.11:194] File "/usr/lib/python2.7/dist-packages/django/core/wsgi.py", line 14, in get_wsgi_application
[Tue Mar 28 06:20:24.264512 2017] [wsgi:error] [pid 25585:tid 140667617994496] [remote 10.1.11.11:194] django.setup()
[Tue Mar 28 06:20:24.264524 2017] [wsgi:error] [pid 25585:tid 140667617994496] [remote 10.1.11.11:194] File "/usr/lib/python2.7/dist-packages/django/__init__.py", line 17, in setup
[Tue Mar 28 06:20:24.264559 2017] [wsgi:error] [pid 25585:tid 140667617994496] [remote 10.1.11.11:194] configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
[Tue Mar 28 06:20:24.264572 2017] [wsgi:error] [pid 25585:tid 140667617994496] [remote 10.1.11.11:194] File "/usr/lib/python2.7/dist-packages/django/conf/__init__.py", line 48, in __getattr__
[Tue Mar 28 06:20:24.264647 2017] [wsgi:error] [pid 25585:tid 140667617994496] [remote 10.1.11.11:194] self._setup(name)
[Tue Mar 28 06:20:24.264660 2017] [wsgi:error] [pid 25585:tid 140667617994496] [remote 10.1.11.11:194] File "/usr/lib/python2.7/dist-packages/django/conf/__init__.py", line 44, in _setup
[Tue Mar 28 06:20:24.264678 2017] [wsgi:error] [pid 25585:tid 140667617994496] [remote 10.1.11.11:194] self._wrapped = Settings(settings_module)
[Tue Mar 28 06:20:24.264688 2017] [wsgi:error] [pid 25585:tid 140667617994496] [remote 10.1.11.11:194] File "/usr/lib/python2.7/dist-packages/django/conf/__init__.py", line 92, in __init__
[Tue Mar 28 06:20:24.264702 2017] [wsgi:error] [pid 25585:tid 140667617994496] [remote 10.1.11.11:194] mod = importlib.import_module(self.SETTINGS_MODULE)
[Tue Mar 28 06:20:24.264715 2017] [wsgi:error] [pid 25585:tid 140667617994496] [remote 10.1.11.11:194] File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
[Tue Mar 28 06:20:24.264754 2017] [wsgi:error] [pid 25585:tid 140667617994496] [remote 10.1.11.11:194] __import__(name)
[Tue Mar 28 06:20:24.264767 2017] [wsgi:error] [pid 25585:tid 140667617994496] [remote 10.1.11.11:194] File "/usr/share/openstack-dashboard/openstack_d...

[Roger Luethi (rl-o)]

The temporary workaround I am currently using is here:

https://review.openstack.org/#/c/443778/13/labs/osbash/scripts/ubuntu/setup_horizon.sh

I changed the owner for secret_key and the containing directory to www-data:www-data. It used to be horizon:horizon in Newton, but for Canonical's Ocata packages, that is no longer the case (and changing the owner to horizon does not help, either).

[Thiago Martins (martinx)]

Hi Brian, Hmmm... I'm not sure about a good value for it. Maybe 5~10 min?

I think that at least, worth informing that, instead of running an extra
command every time you add a node, it can be just something automatic, if
I'm understand it correctly.

On 28 March 2017 at 01:16, Brian Moss <email address hidden> wrote:

> Thiago, since we instruct the user to run `nova manage cell_v2
> discover_hosts` after configuring the compute node, do you think we need
> to change the default -1 interval?
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1669279).
> https://bugs.launchpad.net/bugs/1663485
>
> Title:
> Document Nova v2 Cells and Placement API
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/openstack-manuals/+bug/1663485/+subscriptions
>

[Thiago Martins (martinx)]

Yes Roger,

I'm seeing this problem as well... To fix it, I'm running (via Ansible):

- file: dest=/var/lib/openstack-dashboard/secret_key owner=www-data
group=www-data

Looks like a package bug.

On 28 March 2017 at 01:52, Roger Luethi <email address hidden> wrote:

> Since the people who tested the Ocata install-guide are conveniently
> still subscribed to this bug, let me ask here if anyone tested the
> Dashboard on Ubuntu. Because it seems that the Ubuntu package has the
> secret_key (/var/lib/openstack-dashboard/secret_key) readable only for
> root which breaks the Dashboard.
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1669279).
> https://bugs.launchpad.net/bugs/1663485
>
> Title:
> Document Nova v2 Cells and Placement API
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/openstack-manuals/+bug/1663485/+subscriptions
>

[Darren Chan (dazzachan)]

Thanks Roger, the temporary workaround fixed my problem. I'll update the install-guide to include the workaround.

[Brian Moss (bmoss)]

Hi Thiago, good idea. I think it is useful to have the user run the command manually the first time so they understand the process. No reason they can't have it running automatically after that though. I'll add a note to that section.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/453003

[Chris Suttles (killface007)]

Another +1 for the secret key problem. I also work around it with install automation.

-Chris

> On Apr 3, 2017, at 5:52 PM, Brian Moss <email address hidden> wrote:
>
> Hi Thiago, good idea. I think it is useful to have the user run the
> command manually the first time so they understand the process. No
> reason they can't have it running automatically after that though. I'll
> add a note to that section.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1663485
>
> Title:
> Document Nova v2 Cells and Placement API
>
> Status in openstack-manuals:
> Fix Released
>
> Bug description:
> During an Installation Guide CentOS install, attempting to launch an
> instance hits this issue:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1405098
>
> It appears that we need to document how to add a cell manually in the
> Install Guide.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/openstack-manuals/+bug/1663485/+subscriptions

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/453004

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/453003
Committed: https://git.openstack.org/cgit/openstack/openstack-manuals/commit/?id=815691352a47fd9688d58c38583180d3516a93d9
Submitter: Jenkins
Branch: master

commit 815691352a47fd9688d58c38583180d3516a93d9
Author: Brian Moss <email address hidden>
Date: Tue Apr 4 11:23:21 2017 +1000

    [Install-Guide] Note discover_hosts_in_cells_interval

    Note the configuration setting in nova.conf that enables
    automatic host discovery at a set interval.

    Change-Id: I46ca0a3f4abc9edd83cd0e07e86d9a599af3d51c
    Backport: ocata
    Related-Bug: #1663485

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/453004
Committed: https://git.openstack.org/cgit/openstack/openstack-manuals/commit/?id=8c32fc2917e4e855daa9f239985c60eb874c3939
Submitter: Jenkins
Branch: stable/ocata

commit 8c32fc2917e4e855daa9f239985c60eb874c3939
Author: Brian Moss <email address hidden>
Date: Tue Apr 4 11:23:21 2017 +1000

    [Install-Guide] Note discover_hosts_in_cells_interval

    Note the configuration setting in nova.conf that enables
    automatic host discovery at a set interval.

    Change-Id: I46ca0a3f4abc9edd83cd0e07e86d9a599af3d51c
    Related-Bug: #1663485