important servers should have backups

[James E. Blair]

cloud image snapshots aren't as reliable as one would hope. We should have a backup system in place to do off-site backups.

Consider obnam: http://lwn.net/Articles/499846/

Ideally, we should backup cross-provider (rax <-> hp), or at least to different data centers.

[James E. Blair (corvus)]

Monty, I think this is waiting on you to work out the best way to backup mysql dbs.

[Patrick Crews (patrick-crews)]

Here is a script I wrote that can backup a mysql database via innobackupex and upload it to a specified swift container.
It assumes there is a pw file and that the swift container already exists...might be useful for this

python backup_and_store_directory.py --os_username=USER --os_password=PASSWORD --os_tenant_name=USER-tenant1 --swift_store_name='mahtester' --backup_name='db_backup_stage' --pass_file=pw_file.dat --db_backup_path=/var/lib/mysql/ --os_auth_url=https://region-a.geo-1.identity.hpcloudsvc.com:35357/v2.0/tokens

Creating encrypted backup up MySQL database via innobackupex...
Backup name: db_backup_stage-20130423-21-27-05.tar.gz.des3
Command: innobackupex --stream=tar /var/lib/mysql/ | gzip - | openssl des3 -salt -kfile pw_file.dat > db_backup_stage-20130423-21-27-05.tar.gz.des3
Status: 0
Output:

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona Ireland Ltd 2009-2012. All Rights Reserved.

innobackupex: Backup created in directory '/var/lib/mysql'
innobackupex: MySQL binlog position: filename '', position
innobackupex: You must use -i (--ignore-zeros) option for extraction of the tar stream.
130423 21:27:29 innobackupex: completed OK!
Swift store result:

201
--------------------------------------------------------------------------------
Listing all objects in store...
db_backup_stage-20130423-21-27-05.tar.gz.des3

Removing backup file: db_backup_stage-20130423-21-27-05.tar.gz.des3

[Patrick Crews (patrick-crews)]

updated backup + store script. cleanups of encryption + uploads to swift

[Clark Boylan (cboylan)]

Status report:

There appear to be two major pieces to this bug. 1. Backup MySQL server contents (output with mysqldump) offsite 2. Backup actual server filesystems offsite.

MySQL backups have been sorted with a puppet manifest that runs mysqldump against the local mysqld keeping a 30 day rotation of these dumps. The offsite backup system, bup, then performs offsite backups of server filesystems including the mysqldumps.

This is in place on wiki.o.o, jenkins-dev.o.o, and review-dev.o.o. Backups will be added for review.o.o, jenkins.o.o, and zuul.o.o shortly.

[Clark Boylan (cboylan)]

The decision was made not to backup the jenkins masters and zuul.o.o as they can be rebuilt from scratch easily without the need of backups. review.o.o and etherpad.o.o are now being backed up to the rackspace backup host. The last remaining host that needs backups is the puppetmaster, fungi has thoughts on how to do this.

We should also work on getting the second backup host running in HP.

[Jeremy Stanley (fungi)]

Basically I think we backup the puppetmaster with bup just like any other server, with the following caveats:

exclude places where sensitive plaintext credentials reside (/etc/puppet/heiradata, /root/ci-launch, /etc/ssl/private, what have you) and then whip up a little cron job to encrypt copies of these to multiple root admin OpenPGP keys when they change and stash the results in a place bup will back up. This doesn't take advantage of the gitishness of bup's protocol for those particular files because they're represented by one or more encrypted blobs which change in their entirety, but they will be smal and this provides a minimum divergence from our normal backup and recovery strategy.

[Clark Boylan (cboylan)]

Should also bup eavesdrop to backup the IRC and meeting logs.

[Tom Fifield (fifieldt)]

Have all the important servers got backups now?