Document migration path from Folsom to Grizzly

[Tom Fifield]

We will need a similar migration document to http://docs.openstack.org/diablo/openstack-compute/admin/content/migrating-from-cactus-to-diablo.html for going from Folsom to Grizzly.

Of particular help will be sections on:

nova-network --> Quantum
nova-volume --> Cinder

[Tom Fifield (fifieldt)]

This patch https://review.openstack.org/20244 is relevant for libvirt networking

[Tom Fifield (fifieldt)]

From https://review.openstack.org/#/c/21327/

Changes the relationship between users and projects.
There is no more direct membership in projects. Instead,
all membership is now done via roles.
A default role has been created called _member_ with a uuid (both
configurable) that will be added in place of the group membership
for databse upgrades.

[Tom Fifield (fifieldt)]

https://review.openstack.org/#/c/24251/

authtoken changes

[dan wendlandt (danwent)]

adding quantum tag, as there are several quantum items that must be documented as well. at the least:

- running db migration command
- starting a new metadata agent (and lbaas agent, if needed).
- changes to the nova.conf to work with new metdata agent.
- changes to l3-conf for metadata?

[Stephen Gordon (sgordon)]

Adding Nova tag, users will need want to install and start an instance of the new nova-conductor service.

[Stephen Gordon (sgordon)]

Keystone appears to default to PKI rather than UUID tokens in Grizzly so users must either:

1) Manually set token_format = UUID to revert to the old behaviour.

2) Run the keystone-manage pki_setup operation to create the PKI infrastructure files.

[Tom Fifield (fifieldt)]

The below is from NeCTAR's upgrade this week. All of the smarts are in puppet (https://github.com/NeCTAR-RC/), but the general flow might help in solving this:

=Nova=
     API: Stop nova-cells
    CC: stop nova-cells; stop nova-scheduler; stop nova-consoleauth; stop nova-novncproxy
    NODES: stop nova-compute; stop nova-network; stop nova-api-metadata
    Upgrade RabbitMQ servers (see section below)
    CC: puppet agent -t
    CC: Take a backup of nova DB
    CC: apt-get upgrade
    CC: stop nova-cells; stop nova-scheduler; stop nova-consoleauth; stop nova-novncproxy
    CC: /etc/init.d/memcached restart
    CC: nova-manage db sync
    CC: start nova-scheduler; start nova-consoleauth; start nova-novncproxy
    NODES: puppet agent -t
    NODES: apt-get upgrade
    CC: start nova-cells

Post upgrade:
    Need to add availability_zone metadata to your production aggregate.

=RabbitMQ=
     Rabbit moving to version 3.1.3. Need to stop all servers in the cluster to upgrade.

= Swift =
* Upgrade swift nodes first, then proxys

==== Prepare ====
* Stop puppet everywhere
* Backup your ring files
* Stop all background processes on your storage nodes (everything except (container|object|account)-server)
* Set puppet to noop the service resource
* Change openstack version to grizzly
==== Upgrade storage nodes ====
One by one do:
* puppet agent -t
* apt-get upgrade
* reboot
==== Upgrade proxy servers ====
One by one do:
* puppet agent -t
* apt-get upgrade
* reboot
==== Finish off ====
* Set puppet to no longer noop the service resource
* Start puppet everywhere

= glance-api =
* puppet agent -t
* apt-get upgrade

= Known issues =
==== nova-compute won't start with: KeyError: 'instance_type_memory_mb' ====
* This could be due to a stale VM in shutoff state. The log should have the instance uuid. Check in the DB to see if it's deleted. If so you should be able to do a virsh undefine <id>

[Tom Fifield (fifieldt)]

ops guide chapter: http://docs.openstack.org/trunk/openstack-ops/content/maintenance.html#upgrades

[Anne Gentle (annegentle)]

Hi All,

I recently did a Folsom to Grizzly upgrade on a production OpenStack environment and wanted to post the process and some notes. I understand that Havana is coming out in a few weeks, so this might be old news... I hope to do Grizzly to Havana within 3 months of Havana's release.

tl;dr: Short of a few minor manual changes, doing a straight "apt-get install" provided a clean and safe upgrade. The instances themselves saw no downtime.

I definitely didn't see the issues that Jon ran into (whose posts I had starred in my inbox as warnings :)

Environment:

Ubuntu 12.04 and OpenStack Folsom with two distinct regions and a cloud controller in each region. The controller runs all services but nova-compute. There's a NetApp providing NFS for shared instance storage as well as iSCSI for block storage.

nova-network is being used (and is still being used).

Swift is used, but it was upgraded to Grizzly months ago to utilize the new quota feature.

Everything is strictly controlled by Puppet.

Prep:

I ran through this process several times in a sandbox environment. I highly recommend doing this. Even if you don't have the budget for several servers, virtualize your cloud controller and a few compute nodes with the same configuration.

With Puppet, I use hiera to split my environment data into "sandbox" and "production" data sets. This way, the same Puppet config is used in both environments, but the proper environment data (subnets, passwords etc) is swapped out.

Cloud Controller Upgrade:

1. Turn off Puppet

2. Backup the databases

3. Remove the Folsom cloud repo, install the Grizzly repo, apt-get update

4. apt-get dist-upgrade and say No to apt wanting to install new config files. The new files will be saved with a .dpkg-dist extension.

5. cp /etc/keystone/keystone.conf{,.orig}
6. cp /etc/keystone/keystone.conf.dpkg-dist /etc/keystone/keystone.conf
7. Manually edit /etc/keystone/keystone.conf with the relevant options. There's a significant different between the Folsom and Grizzly conf files that will prevent Keystone from running correctly before Puppet is turned back on.

8. Repeat for /etc/cinder/api-paste.ini.

All other Folsom-based configs were able to be used to get the services running with Grizzly before Puppet jumped back in to make any Grizzly-specific changes.

9. Run db migrations on all services:

keystone-manage db_sync
glance-manage db_sync
nova-manage db sync
cinder-manage db sync

10. Install nova-conductor manually

11. Manually patch keystone. See this thread for details:

http://www.gossamer-threads.com/lists/openstack/dev/30683

12. Verify all services are back up. nova-network needed to be restarted once all other nova services were running... happened in both regions. No big deal.

13. Install the latest Puppet Keystone module because older versions are not compatible with Grizzly and newer versions are not compatible with Folsom.

14. Test Puppet: puppet agent --verbose --onetime --no-daemonize --noop

15. Turn on Puppet

Compute Node Upgrade:

1. Turn off Puppet

2. Remove Folsom repo, add Grizzly repo, apt-get update

3. apt-get install <long list of packages>

apt-get dist-upgrade was not possible becau...

[Stephen Gordon (sgordon)]

Refer bug #1199773 for future work on this topic.