There is not enough place for a backup-restore scenario on CI

[Alexandr Kostrikov]

Summary:
The backup is stored at /var partition and it is filled:
/dev/mapper/os-var 17G 17G 0 100% /var
/dev/mapper/os-varlog 25G 174M 24G 1% /var/log

Expected result:
Master node is backed up and restored.

Actual result:
https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.backup_restore_master/125/console
2016-02-03 00:54:37,113 - ERROR utils.py:421 -- Unexpected exit_code returned: actual 1, expected 0. Command: 'dockerctl backup' Details: {'host': '10.109.25.2', 'command': 'dockerctl backup', 'exit_code': 1, 'stderr': ['time="2016-02-03T00:52:47Z" level=debug msg="[stream] End of stdout" \n', "tar: Removing leading `/' from member names\n", "tar: Removing leading `/' from member names\n", "tar: Removing leading `/' from hard link targets\n", 'tar: /var/backup/fuel/backup_2016-02-03_0049/fuel_backup_2016-02-03_0049.tar: Cannot write: No space left on device\n', 'tar: Error is not recoverable: exiting now\n', 'Backup failed!\n'], 'stdout': ['Reading container data...\n', 'Committing fuel-core-8.0-mcollective...\n', '8227314d877bb938eedcdbf079cf8dc19cb60af715c4464543358687f1a25d6b\n', 'Committing fuel-core-8.0-cobbler...\n', '2f039c9f67c86104449c71273cbc0bd19f574658ef11a6e544cd97b40636b229\n', 'Committing fuel-core-8.0-nginx...\n', '7bba643a53a58f1dea0a5edce5e6924238b724dd85f703200c4581214f534425\n', 'Committing fuel-core-8.0-ostf...\n', 'c4a0b3aabb64449de0ba5179637a31d6cde824ba2b2401ad0467d39ea7987c6f\n', 'Committing fuel-core-8.0-nailgun...\n', '63e1961d40b1da5c18d1bda990b5182453ea20ac63fb1b451d1cfd09635a15f1\n', 'Committing fuel-core-8.0-rsyslog...\n', 'a712441bb0fe9a1b59b046dfa0b8fea274fd1d614b6e87cf559a6131d218b984\n', 'Committing fuel-core-8.0-astute...\n', 'd6bc53dc5dc1b77e2823d2b8834debc77ab39b4997bd423c70881801bc738962\n', 'Committing fuel-core-8.0-rsync...\n', '744c13be4e3ed6491328457613625e789b60f61a99f5f5c505f59cadaf1bd7d0\n', 'Committing fuel-core-8.0-keystone...\n', '8ac45bf8cb13535e87d63f61f359f5ff2a19e5789b217944ddd681e56457eced\n', 'Committing fuel-core-8.0-rabbitmq...\n', 'bb242e971f099a37a95780e6f956c167066e628a8319ce58c09939dc97d60b43\n', 'Committing fuel-core-8.0-postgres...\n', '307cc1077dc9814a570fda52c774801eb7660a4166551ba850562da8ce3ac1f1\n', 'Saving containers to combined archive...\n', 'Cleaning up temporary images...\n', 'Untagged: fuel/postgres_8.0_2016-02-03_0049:latest\n', 'Deleted: 307cc1077dc9814a570fda52c774801eb7660a4166551ba850562da8ce3ac1f1\n', 'Untagged: fuel/rabbitmq_8.0_2016-02-03_0049:latest\n', 'Deleted: bb242e971f099a37a95780e6f956c167066e628a8319ce58c09939dc97d60b43\n', 'Untagged: fuel/keystone_8.0_2016-02-03_0049:latest\n', 'Deleted: 8ac45bf8cb13535e87d63f61f359f5ff2a19e5789b217944ddd681e56457eced\n', 'Untagged: fuel/rsync_8.0_2016-02-03_0049:latest\n', 'Deleted: 744c13be4e3ed6491328457613625e789b60f61a99f5f5c505f59cadaf1bd7d0\n', 'Untagged: fuel/astute_8.0_2016-02-03_0049:latest\n', 'Deleted: d6bc53dc5dc1b77e2823d2b8834debc77ab39b4997bd423c70881801bc738962\n', 'Untagged: fuel/rsyslog_8.0_2016-02-03_0049:latest\n', 'Deleted: a712441bb0fe9a1b59b046dfa0b8fea274fd1d614b6e87cf559a6131d218b984\n', 'Untagged: fuel/nailgun_8.0_2016-02-03_0049:latest\n', 'Deleted: 63e1961d40b1da5c18d1bda990b5182453ea20ac63fb1b451d1cfd09635a15f1\n', 'Untagged: fuel/ostf_8.0_2016-02-03_0049:latest\n', 'Deleted: c4a0b3aabb64449de0ba5179637a31d6cde824ba2b2401ad0467d39ea7987c6f\n', 'Untagged: fuel/nginx_8.0_2016-02-03_0049:latest\n', 'Deleted: 7bba643a53a58f1dea0a5edce5e6924238b724dd85f703200c4581214f534425\n', 'Untagged: fuel/cobbler_8.0_2016-02-03_0049:latest\n', 'Deleted: 2f039c9f67c86104449c71273cbc0bd19f574658ef11a6e544cd97b40636b229\n', 'Untagged: fuel/mcollective_8.0_2016-02-03_0049:latest\n', 'Deleted: 8227314d877bb938eedcdbf079cf8dc19cb60af715c4464543358687f1a25d6b\n', 'Pausing containers...\n', '68c9d7e22f66\n', 'db30054b02a0\n', 'f6d1305bb5a8\n', '7e586ec0d960\n', 'c53f216c52c7\n', 'a141f015c57d\n', 'bdaf1d04c3b1\n', '9c532d5abcbf\n', '03198e169ffa\n', 'aafe09c56c37\n', '14f78ec0f0fc\n', 'Archiving system folders\n', 'Unpausing containers...\n', '68c9d7e22f66\n', 'db30054b02a0\n', 'f6d1305bb5a8\n', '7e586ec0d960\n', 'c53f216c52c7\n', 'a141f015c57d\n', 'bdaf1d04c3b1\n', '9c532d5abcbf\n', '03198e169ffa\n', 'aafe09c56c37\n', '14f78ec0f0fc\n', 'Backing up PostgreSQL database to /var/backup/fuel/backup_2016-02-03_0049/postgres_backup.sql...\n', 'Compressing archives...\n']}

Proposed fix - create admin node with larger disk.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: master
Change author: Alexandr Kostrikov <email address hidden>
Review: https://review.fuel-infra.org/16652

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/16652
Submitter: Aleksandra Fedorova <email address hidden>
Branch: master

Commit: d291be4ab39d33605730805cf82206623b60a980
Author: Alexandr Kostrikov <email address hidden>
Date: Wed Feb 3 11:38:16 2016

Add additional space to backup_restore_master

There is lack of space to backup and restore master
on swarm.

Change-Id: Ied1cb1dcdb672a0e08f802cf5c2393d615b5d01a
Closes-bug: #1541339

[Alexandr Kostrikov (akostrikov-mirantis)]

After maxing to 100 ADMIN_NODE_DISK we have enough space for backup (it is close to previous available 17G)
[root@nailgun ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-root 9.5G 1.4G 7.7G 16% /
devtmpfs 1.4G 0 1.4G 0% /dev
tmpfs 1.4G 0 1.4G 0% /dev/shm
tmpfs 1.4G 26M 1.4G 2% /run
tmpfs 1.4G 0 1.4G 0% /sys/fs/cgroup
/dev/vda3 197M 108M 90M 55% /boot
/dev/vda2 200M 0 200M 0% /boot/efi
/dev/mapper/os-var 27G 17G 8.2G 68% /var
/dev/mapper/os-varlog 40G 194M 38G 1% /var/log