Hi All,

I recently did a Folsom to Grizzly upgrade on a production OpenStack environment and wanted to post the process and some notes. I understand that Havana is coming out in a few weeks, so this might be old news... I hope to do Grizzly to Havana within 3 months of Havana's release.

tl;dr: Short of a few minor manual changes, doing a straight "apt-get install" provided a clean and safe upgrade. The instances themselves saw no downtime. 

I definitely didn't see the issues that Jon ran into (whose posts I had starred in my inbox as warnings :)

Environment:

Ubuntu 12.04 and OpenStack Folsom with two distinct regions and a cloud controller in each region. The controller runs all services but nova-compute. There's a NetApp providing NFS for shared instance storage as well as iSCSI for block storage.  

nova-network is being used (and is still being used). 

Swift is used, but it was upgraded to Grizzly months ago to utilize the new quota feature.

Everything is strictly controlled by Puppet.

Prep:

I ran through this process several times in a sandbox environment. I highly recommend doing this. Even if you don't have the budget for several servers, virtualize your cloud controller and a few compute nodes with the same configuration. 

With Puppet, I use hiera to split my environment data into "sandbox" and "production" data sets. This way, the same Puppet config is used in both environments, but the proper environment data (subnets, passwords etc) is swapped out.

Cloud Controller Upgrade:

1. Turn off Puppet

2. Backup the databases

3. Remove the Folsom cloud repo, install the Grizzly repo, apt-get update

4. apt-get dist-upgrade and say No to apt wanting to install new config files. The new files will be saved with a .dpkg-dist extension.

5. cp /etc/keystone/keystone.conf{,.orig}
6. cp /etc/keystone/keystone.conf.dpkg-dist /etc/keystone/keystone.conf
7. Manually edit /etc/keystone/keystone.conf with the relevant options. There's a significant different between the Folsom and Grizzly conf files that will prevent Keystone from running correctly before Puppet is turned back on.

8. Repeat for /etc/cinder/api-paste.ini. 

All other Folsom-based configs were able to be used to get the services running with Grizzly before Puppet jumped back in to make any Grizzly-specific changes.

9. Run db migrations on all services:

keystone-manage db_sync
glance-manage db_sync
nova-manage db sync
cinder-manage db sync

10. Install nova-conductor manually

11. Manually patch keystone. See this thread for details:

http://www.gossamer-threads.com/lists/openstack/dev/30683

12. Verify all services are back up. nova-network needed to be restarted once all other nova services were running... happened in both regions. No big deal.

13. Install the latest Puppet Keystone module because older versions are not compatible with Grizzly and newer versions are not compatible with Folsom.

14. Test Puppet: puppet agent --verbose --onetime --no-daemonize --noop

15. Turn on Puppet

Compute Node Upgrade:

1. Turn off Puppet

2. Remove Folsom repo, add Grizzly repo, apt-get update

3. apt-get install <long list of packages>

apt-get dist-upgrade was not possible because there was an updated version of open-iscsi available. Upgrading the package would cause all current iSCSI connections to disconnect effectively disconnecting all volumes. Not cool. I think the only way around this would be to evacuate compute nodes and install the update.

4. Test and turn on Puppet

Final Notes:

Why not have Puppet running during the upgrade or let Puppet do the upgrade? I probably could have done this but I preferred to "personally" oversee the upgrade myself.

I saw a recent thread on the -dev list about providing reverse database migrations for the possibility of having to roll back an upgrade. A comment was made (I forget by whom) asking if this was really needed and if an admin would simply perform a database restore instead. I fully agree with that statement. If I had to roll back, I would uninstall the Grizzly packages, install the Folsom packages, and then import my latest database backup.

I hope someone finds this useful. Please let me know if you'd like further details.
Thanks,
Joe



-- 
Joe Topjian
Systems Architect
Cybera Inc.

www.cybera.ca

Cybera is a not-for-profit organization that works to spur and support innovation, for the economic benefit of Alberta, through the use of cyberinfrastructure.