dockerctl backup report success incorrectly

[Francois Deppierraz]

Running dockerctl backup on a machine with a limited amount of memory can fail, however it incorrectly reports "Backup complete". This is misleading because the backup file is probably either truncated or corrupted.

[root@fuel tmp]# dockerctl backup
DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the ongoing version of python-fuelclient.
Reading container data...
Committing fuel-core-6.1-nginx...
605e3bdfeec9f0d89246617b03d5500c19cc7655e69e1ddbfc393328d1b09c19
[...]
Saving fuel-core-6.0-postgres...
Archiving system folders
tar: Removing leading `/' from member names
Compressing archives...
Failed to malloc ckbuf in hash_search
Cannot allocate memory
Fatal error - exiting
Cleaning up...
Backup complete. File is available at /var/backup/fuel/backup_2015-03-05_2303/fuel_backup_2015-03-05_2303.tar.lrz
[root@fuel tmp]#

[Matthew Mosesohn (raytrac3r)]

Bogdan, what's the spec of your VM you used to generate this bug? I just want to know so I can reproduce. I can add a check, but there needs to be a way to back up even if there is little memory.

[Francois Deppierraz (francois-ctrlaltdel)]

Hi Matthew, the physical server that was used to generate this bug has only 2 GB of RAM and 4 GB of swap.

[Matthew Mosesohn (raytrac3r)]

Oh, sorry. My mistake. I misread who opened the bug.

Francois, it looks like you're backing up a 6.1 environment that was upgrade from 6.0. Is that right? What was the initial version of Fuel installed on your host?

[Francois Deppierraz (francois-ctrlaltdel)]

Yes, that's right. The server was initially installed from the MirantisOpenStack-6.0.iso image and then upgraded with fuel-community-6.1-upgrade-89-2015-02-20_05-43-26.tar.lrz.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/172000

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/176779

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/172000
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=c2dfb512e58cb6793d29a989a3380dc1d844be08
Submitter: Jenkins
Branch: master

commit c2dfb512e58cb6793d29a989a3380dc1d844be08
Author: Matthew Mosesohn <email address hidden>
Date: Thu Apr 9 13:34:58 2015 +0300

    Fail backup on docker backup/restore error

    Change-Id: I2be9d281f9a22bf1fed8dc9739d8fc2c68cbcc0b
    Closes-Bug: #1428982

[Alexander Arzhanov (aarzhanov)]

Verified on iso #466:

release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 0bd72c72369e743376864e8e8dabfe873d40450a
      build_id: 2015-05-25_20-55-26
      build_number: '466'
      feature_groups:
      - mirantis
      fuel-library_sha: d7128c27a1b76f4813f3697609f82875c68e85ed
      fuel-ostf_sha: 87819878bc0ca572900e1f6933d9b99e666d6f62
      fuelmain_sha: 5c8ebddf64ea93000af2de3ccdb4aa8bb766ce93
      nailgun_sha: 61ef0edfbfe0c457265a62f0eab05af634ec3b91
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: e19f1b65792f84c4a18b5a9473f85ef3ba172fce
      release: '6.1'

[root@nailgun ~]# dockerctl backup --full
DEPRECATION WARNING: /etc/fuel/client/config.yaml exists and will be used as the source for settings. This behavior is deprecated. Please specify the path to your custom settings file in the FUELCLIENT_CUSTOM_SETTINGS environment variable.
Reading container data...
Committing fuel-core-6.1-mcollective...
FATA[0018] Error response from daemon: ApplyLayer fork/exec /usr/bin/docker: cannot allocate memory
Backup failed!

[Baboune (seyvet)]

FYI same bug on 6.0:

dockerctl backup /mnt/fractal/openstack/fuel-backup/kds-cmc-fuel-01/
rm: cannot remove `/mnt/fractal/openstack/fuel-backup/kds-cmc-fuel-01': Permission denied
Reading container data...
Committing fuel-core-6.0-mcollective...
f2a58b285ddd41835415516dbe3a7b18eb6f09904378917427e35022a7ca78df
Saving fuel-core-6.0-mcollective...
Committing fuel-core-6.0-cobbler...
2f524910844be692234a3a264d4cab5e50eda2d2c6ffbd3aac4db1866d220e33
Saving fuel-core-6.0-cobbler...
Committing fuel-core-6.0-nginx...
00422359ff2417ce01e6c5267dbdbb81f875e87eb735a3ff368b40478874c823
Saving fuel-core-6.0-nginx...
Committing fuel-core-6.0-ostf...
7f58540314d0f5bba7d04f0dafcf95c3c077c1e99221a131cb4eea508c4c48cf
Saving fuel-core-6.0-ostf...
Committing fuel-core-6.0-nailgun...
735ed5e8a3a9b34d03c16931302dd493c76789c0d86eadf124113f1421d75700
Saving fuel-core-6.0-nailgun...
Committing fuel-core-6.0-rsyslog...
745a7f3d9c850d706cb024495225ef3ef4d5e90b409cb87835670935e8237d01
Saving fuel-core-6.0-rsyslog...
Committing fuel-core-6.0-astute...
cef747cda2f431ea6576f21df2b43945d32c3bae669d8e47230a6671451c325d
Saving fuel-core-6.0-astute...
Committing fuel-core-6.0-rsync...
d63cd43be5bde535d59982e7adff44287aaa967427b9727e0736f49efa38a48e
Saving fuel-core-6.0-rsync...
Committing fuel-core-6.0-keystone...
11fc92ab19d42fe9d3e082c4fc23fd7e14fb7bc2909119e93de3d8acc2eee1ee
Saving fuel-core-6.0-keystone...
Committing fuel-core-6.0-rabbitmq...
c6f5a3d5c07b93c18505935fc3e1aae5d80febe5690e5a579a9b82e755907e14
Saving fuel-core-6.0-rabbitmq...
Committing fuel-core-6.0-postgres...
19d28801b4318e73b2d62eb322e148e917abdd58b58835255b70d74b37b4dfd3
Saving fuel-core-6.0-postgres...
Archiving system folders
tar: Removing leading `/' from member names
Compressing archives...
Failed to malloc ckbuf in hash_search
Cannot allocate memory
Fatal error - exiting
Cleaning up...
Backup complete. File is available at /mnt/fractal/openstack/fuel-backup/kds-cmc-fuel-01//fuel_backup_2015-06-09_0710.tar.lrz

Mem: 2053632k total
Swap: 4128764k total

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.0"
  api: "1.0"
  build_number: "58"
  build_id: "2014-12-26_14-25-46"
  astute_sha: "16b252d93be6aaa73030b8100cf8c5ca6a970a91"
  fuellib_sha: "fde8ba5e11a1acaf819d402c645c731af450aff0"
  ostf_sha: "a9afb68710d809570460c29d6c3293219d3624d4"
  nailgun_sha: "5f91157daa6798ff522ca9f6d34e7e135f150a90"
  fuelmain_sha: "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8"

[Baboune (seyvet)]

Tried suggested fix, got correct error message. The lrz was however still there at the end and since it is corrupted it might be better to delete it.

Good that the tar is not deleted.