[Backup/Restore] Restore doesn't restore containers and system dirs correctly

[Tatyanka]

Steps to reproduce:
1. Create some file into /etc/fuel directory
2. Run backup of master node (dockerctl backup)
3. rm -rf /etc/puppet
4. Run restore dockerctl restore /var/backup/path_to_backup

Expected Result:
/etc/puppet contains restored manifests, /etc/fuel contains created before backup file

Actual result:
There is no created file in /etc/fuel directory and manifests were not restored

Versions:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2-6.1"
  api: "1.0"
  build_number: "323"
  build_id: "2015-04-20_07-52-58"
  nailgun_sha: "5a4556ea0cf943edffa33788994b7fb7abe150b3"
  python-fuelclient_sha: "b4f1ddffd5263489090b65e662173e9e11aafd94"
  astute_sha: "bf1751a4fe0d912325e3b4af629126a59c1b2b51"
  fuellib_sha: "36f30ae7f19092a61eebb0522ca20d27468b4cbf"
  ostf_sha: "4bab9b975ace8d9a305d6e0f112b734de587f847"
  fuelmain_sha: "991fa8feca4afcf05ce85a1e7ebd7c7503b3f212"

Restore process info :
http://paste.openstack.org/show/205741/

[Matthew Mosesohn (raytrac3r)]

Tatyanka, your backup didn't succeed. How big is the file size?

Can you possibly reproduce the full backup/restore with bash -x?
bash -x dockerctl backup
bash -x dockerctl restore /var/backup/fuel/.....

I think these files just didn't get backed up. I reproduced locally by removing /etc/puppet/manifests and /etc/puppet/modules and it restored them okay. And I see /etc/fuel in the system-dirs.tar file.

[Matthew Mosesohn (raytrac3r)]

Tatyana, can you reproduce this issue and give me access to the backup file? Preferrably a diagnostic snapshot as well.

[Tatyanka (tatyana-leontovich)]

Matthew, you can find a lot of snapshots on our CI, like example here http://jenkins-product.srt.mirantis.net:8080/view/6.1_swarm/job/6.1.system_test.centos.known_issues/100/ , we have failed test here with absent file after restore in /etc/fuel. In my case it was revert of those failures and just attempt to check is only one of directory not restored or both from specified.

[Tatyanka (tatyana-leontovich)]

But I am pretty sure that this issue is related to the available space in /var and false positive message from dockerctl backup command (and as result corrupted archives that are not executable)

BTW seems on 359 iso you've already fixed this with check of available space in /var before backup

[Stanislaw Bogatkin (sbogatkin)]

I faced with another problem on restore:
[root@nailgun ~]# dockerctl restore /var/backup/fuel/backup_2015-05-15_1550/fuel_backup_2015-05-15_1550.tar.lrz --full
DEPRECATION WARNING: /etc/fuel/client/config.yaml exists and will be used as the source for settings. This behavior is deprecated. Please specify the path to your custom settings file in the FUELCLIENT_CUSTOM_SETT
INGS environment variable.
Shut down
Stopping and destroying existing containers...
fuel-core-6.1-nginx
fuel-core-6.1-rabbitmq
fuel-core-6.1-astute
fuel-core-6.1-rsync
fuel-core-6.1-keystone
fuel-core-6.1-postgres
fuel-core-6.1-rsyslog
fuel-core-6.1-nailgun
fuel-core-6.1-cobbler
fuel-core-6.1-ostf
fuel-core-6.1-mcollective
fuel-core-6.1-nginx
fuel-core-6.1-rabbitmq
fuel-core-6.1-astute
fuel-core-6.1-rsync
fuel-core-6.1-keystone
fuel-core-6.1-postgres
fuel-core-6.1-rsyslog
fuel-core-6.1-nailgun
fuel-core-6.1-cobbler
fuel-core-6.1-ostf
fuel-core-6.1-mcollective
Output filename is: /var/backup/fuel/restore-2015-05-15_1550//fuel_backup.tar
Decompressing...
100% 11.03 / 11.03 GB
Average DeCompression Speed: 46.465MB/s
Output filename is: /var/backup/fuel/restore-2015-05-15_1550//fuel_backup.tar: [OK] - 11840245760 bytes
Total time: 00:04:02.62
Loading /var/backup/fuel/restore-2015-05-15_1550//fuel__astute_6.1_2015-05-15_1550.tar...
DEBU[0027] [stream] End of stdout
Loading /var/backup/fuel/restore-2015-05-15_1550//fuel__cobbler_6.1_2015-05-15_1550.tar...
DEBU[0026] [stream] End of stdout
Loading /var/backup/fuel/restore-2015-05-15_1550//fuel__keystone_6.1_2015-05-15_1550.tar...
DEBU[0041] [stream] End of stdout
Loading /var/backup/fuel/restore-2015-05-15_1550//fuel__mcollective_6.1_2015-05-15_1550.tar...
DEBU[0018] [stream] End of stdout
Loading /var/backup/fuel/restore-2015-05-15_1550//fuel__nailgun_6.1_2015-05-15_1550.tar...
DEBU[0030] [stream] End of stdout
Loading /var/backup/fuel/restore-2015-05-15_1550//fuel__nginx_6.1_2015-05-15_1550.tar...
DEBU[0014] [stream] End of stdout
Loading /var/backup/fuel/restore-2015-05-15_1550//fuel__ostf_6.1_2015-05-15_1550.tar...
DEBU[0023] [stream] End of stdout
Loading /var/backup/fuel/restore-2015-05-15_1550//fuel__postgres_6.1_2015-05-15_1550.tar...
DEBU[0013] [stream] End of stdout
Loading /var/backup/fuel/restore-2015-05-15_1550//fuel__rabbitmq_6.1_2015-05-15_1550.tar...
DEBU[0013] [stream] End of stdout
Loading /var/backup/fuel/restore-2015-05-15_1550//fuel__rsync_6.1_2015-05-15_1550.tar...
DEBU[0018] [stream] End of stdout
Loading /var/backup/fuel/restore-2015-05-15_1550//fuel__rsyslog_6.1_2015-05-15_1550.tar...
DEBU[0013] [stream] End of stdout
Loading /var/backup/fuel/restore-2015-05-15_1550//system-dirs.tar...
FATA[0000] Error response from daemon: Conflict: Tag latest is already set to image 79706617128fcfb31bbdf246a80444ce9352da03a050490b59cd46f894c...

[Stanislaw Bogatkin (sbogatkin)]

So, system dirs actually not restored successfully and containers restoring has problems too.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/183936

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/183936
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=dac6847cfc1883a0f15cb717f91239c7142fa11b
Submitter: Jenkins
Branch: master

commit dac6847cfc1883a0f15cb717f91239c7142fa11b
Author: Matthew Mosesohn <email address hidden>
Date: Sun May 17 20:46:51 2015 +0300

    Force tagging on docker restore

    Docker refuses to allow latest tags to be written
    if there is already a newer latest tag without
    -f option.

    Change-Id: I72e9e3addff6f2be5b66dc18d3162c19045c4e47
    Closes-Bug: #1448211

[Stanislaw Bogatkin (sbogatkin)]

Now it fails with another error:

<output omitted>
Starting containers...
03e0c416b1a1ba816f530b078b5f7f84f264e52796f3db09ce27e8a1e1773a7f
DEBU[0000] End of CmdRun(), Waiting for hijack to finish.
a25caf8f484bd48964eaaaa9ba651e2f9eac782224bf0d9033b6dcde0eecb150
DEBU[0000] End of CmdRun(), Waiting for hijack to finish.
e42e6b7e846f341700ac1a716a0bceb66910ca822ee85b86d12ae9a0d01e7956
DEBU[0000] End of CmdRun(), Waiting for hijack to finish.
9aac48e3119682fc1755a65507e354f0df6df4d21bc08714ba68f9c3268790bf
DEBU[0001] End of CmdRun(), Waiting for hijack to finish.
85a2e969352df4ed78ca1127f1f1c3e7c6503c8ea97f3f05770da29d00351cf4
DEBU[0001] End of CmdRun(), Waiting for hijack to finish.
1fdcbcd564c10a74d028538988c99933393a4eb4e5a56e3e116ed3b79ed5cc63
DEBU[0000] End of CmdRun(), Waiting for hijack to finish.
f4206682c3afde463498ce5a85cd999194db0214ba08a2804d4de427ca1111b7
DEBU[0000] End of CmdRun(), Waiting for hijack to finish.
409816f7ea5613c7a799c63f2e2fddf131fae9165d17e7f6b832368fc3133950
DEBU[0000] End of CmdRun(), Waiting for hijack to finish.
56436e4b717b71f7d9b932675887f02db815ab8a4fb6afafeed58f1ce8c69d9a
DEBU[0000] End of CmdRun(), Waiting for hijack to finish.
452bf9862a9df53919698007a3fe368f0720c2a73022c9ca553606e609a2ffa5
DEBU[0000] End of CmdRun(), Waiting for hijack to finish.
3e7c9b2d57f3ca733dad3b1192a826f7ff5d9994928f2207f3a57ce0717d0496
DEBU[0000] End of CmdRun(), Waiting for hijack to finish.
Starting supervisord:
docker-astute STARTING
docker-cobbler STARTING
docker-keystone STARTING
docker-mcollective STARTING
docker-nailgun STARTING
docker-nginx STARTING
docker-ostf STARTING
docker-postgres STARTING
docker-rabbitmq STARTING
docker-rsync STARTING
docker-rsyslog STARTING
checking container postgres
checking with command "shell_container postgres PGPASSWORD=ZRcPmOUW /usr/bin/psql -h 10.109.0.2 -U "nailgun" "nailgun" -c '\copyright' 2>&1 1>/dev/null"
psql: could not connect to server: Connection refused
        Is the server running on host "10.109.0.2" and accepting
        TCP/IP connections on port 5432?
Restore failed!

[Stanislaw Bogatkin (sbogatkin)]

Seems that postgres container needs a bigger amount of time to start than others. If I check container right after restoring, it doesn't checked successfully. But after about 30 seconds it works ok.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/184090

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/184298

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/184090
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=6b7fefcc2bea077bb00d5300316d090856938227
Submitter: Jenkins
Branch: master

commit 6b7fefcc2bea077bb00d5300316d090856938227
Author: Stanislaw Bogatkin <email address hidden>
Date: Mon May 18 19:19:25 2015 +0300

    Add timeout to restore

    On restore containers need to apply all settings. We need
    to wait when it happens.

    Change-Id: Iaf159736d68774bd8b785af7078d4e9f17b1a30c
    Closes-Bug: #1448211

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/184298
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=02393e503db32ee3c790e249df11c967dfcd1af4
Submitter: Jenkins
Branch: master

commit 02393e503db32ee3c790e249df11c967dfcd1af4
Author: Matthew Mosesohn <email address hidden>
Date: Tue May 19 21:19:49 2015 +0400

    Fix disk space and error handling in dockerctl restore

    Reduced free disk space for stateful restore to 2gb
    (still 11gb for full restore). Nonzero command exit
    codes only apply to the beginning of restore, but
    container startup now is handled without "set -e".

    Change-Id: Ifc0109e55adc7fbf30b51f5cfb259ea2ae5bf585
    Partial-Bug: #1448211
    Closes-Bug: #1455488

[Tatyanka (tatyana-leontovich)]

verified 525 iso