System Restore fails on attempt to apply banner customization

[mhg]

Brief Description
-----------------
CLI 'sudo configure_controller --restore-system', one of the steps for System Restore failed with Traceback and error messages. Further investigation found that one expected file was missing: /usr/sbin/apply_banner_customization.

Severity
--------
Critical

Steps to Reproduce
------------------
1) Do system backup following the documented procedures:
   -backup system data using CLI:
             sudo config_controller --backup-system <backup_package_name>
   -backup glance images using CLI for each image:
             sudo image-backup export <image_uuid>
   -backup cinder volumes using CLI for each volumes:
             cinder export <volume_uuid> (for "available" volumes)
             cinder snapshot-create --force True --name <snp_name> <volume_uuid> (for "in-use" volumes)
             cinder snapshot-export <snp_id> (for "in-use" volumes)

     -save all files generated under /opt/backup

2) wipe disks and reinstall same version of system on controller-0 (e.g. using same ISO)
3) restore system using CLI:
     sudo config_controller --restore-system <system-backup-package_file>

Expected Behavior
------------------
The CLI is expected to run to completion without any error and.

Actual Behavior
----------------
The CLI failed after 11 out of 25 steps (44%), Traceback was diplayed:
Step 11 of 25 [################### ] [44%]Traceback (most recent call last):
  File "/usr/bin/config_controller", line 9, in <module>
    load_entry_point('controllerconfig==1.0.0', 'console_scripts', 'config_controller')()
  File "/usr/lib64/python2.7/site-packages/controllerconfig/systemconfig.py", line 476, in main
    backup_restore.restore_system(backup_name)
  File "/usr/lib64/python2.7/site-packages/controllerconfig/backup_restore.py", line 1757, in restore_system
    utils.apply_banner_customization()
  File "/usr/lib64/python2.7/site-packages/controllerconfig/utils.py", line 821, in apply_banner_customization
    stdout=blog, stderr=blog)
  File "/usr/lib64/python2.7/subprocess.py", line 537, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib64/python2.7/subprocess.py", line 524, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Further investigation found that the file '/usr/sbin/apply_banner_customization' does not exist, which failed the config_controller.

Reproducibility
---------------
Reproducible
Only ran once, but it's confirmed the package containing the file was not in the build.

System Configuration
--------------------
Any system types

Branch/Pull Time/Commit
-----------------------
StarlingX master as of 2018-08-29_20-18-00

Timestamp/Logs
2018-08-31 21:13:20,512

--------------
[2018-08-31 21:13:20,512] 824 INFO MainThread telnet.exec_cmd:: executing: echo "Li69nux*" | sudo -S config_controller --restore-system /home/wrsroot/titanium_backup_.ceph20180831-153713_2018-08-29_20-18-00_yow-cgcs-ironpass-20_27_system.tgz
[2018-08-31 21:29:43,431] 840 DEBUG MainThread telnet.exec_cmd:: index:2, output:echo "Li69nux*" | sudo -S config_controller --restore-sy
stem /home/wrsroot/titanium_backup_.ceph20180831-153713_2018-08-29_20-18-00_yow-
cgcs-ironpass-20_27_system.tgz

Restoring system (this will take several minutes):

Step 1 of 25 [# ] [4%]
Step 2 of 25 [### ] [8%]
Step 3 of 25 [##### ] [12%]
Step 4 of 25 [####### ] [16%]
Step 5 of 25 [######### ] [20%]
Step 6 of 25 [########## ] [24%]
Step 7 of 25 [############ ] [28%]
Step 8 of 25 [############## ] [32%]
Step 9 of 25 [################ ] [36%]
Step 10 of 25 [################## ] [40%]
Step 11 of 25 [################### ] [44%]Traceback (most recent call last):
  File "/usr/bin/config_controller", line 9, in <module>
    load_entry_point('controllerconfig==1.0.0', 'console_scripts', 'config_controller')()
  File "/usr/lib64/python2.7/site-packages/controllerconfig/systemconfig.py", line 476, in main
    backup_restore.restore_system(backup_name)
  File "/usr/lib64/python2.7/site-packages/controllerconfig/backup_restore.py", line 1757, in restore_system
    utils.apply_banner_customization()
  File "/usr/lib64/python2.7/site-packages/controllerconfig/utils.py", line 821, in apply_banner_customization
    stdout=blog, stderr=blog)
  File "/usr/lib64/python2.7/subprocess.py", line 537, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib64/python2.7/subprocess.py", line 524, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

[Ghada Khalil (gkhalil)]

Targeted for stx.2018.10 - this breaks starlingx backup & restore

[Ghada Khalil (gkhalil)]

The plan is to restore the missing package: update-motd

[Michel Thebeau [WIND] (mthebeau)]

"update-motd" is missing from "centos_iso_image.inc". After adding it into the load, testing shows:
 - the original /etc/motd from "setup" rpm package will be clobbered (replaced with empty file) by update-motd cron job at the top of the hour
 - the code is absent from config_controller to apply banner customization during config_controller (per stx-integ/utilities/branding/sources/login-banner/README

[Michel Thebeau [WIND] (mthebeau)]

Testing includes:
 - config_controller with/without banner customization
 - customization after config_controller

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/603819

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/603820

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/603821

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/603819
Committed: https://git.openstack.org/cgit/openstack/stx-integ/commit/?id=5de6b8a50b8a3c5b381ccd771f3ab5350d4eebc2
Submitter: Zuul
Branch: master

commit 5de6b8a50b8a3c5b381ccd771f3ab5350d4eebc2
Author: Michel Thebeau <email address hidden>
Date: Mon Sep 17 11:33:26 2018 -0400

    add update-motd to build/iso

    Partial-Bug: 1790863

    Change-Id: I26db48ca182824ff443eceead45305ad9a57cce1
    Signed-off-by: Michel Thebeau <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/603820
Committed: https://git.openstack.org/cgit/openstack/stx-integ/commit/?id=685baaed339567728a1ef4827dba0a7f99c2da67
Submitter: Zuul
Branch: master

commit 685baaed339567728a1ef4827dba0a7f99c2da67
Author: Michel Thebeau <email address hidden>
Date: Tue Sep 18 15:15:12 2018 -0400

    update-motd: add motd.head to /etc

    The cron job for update-motd will run at the top of the hour and clobber
    an original /etc/motd file provided by 'setup' RPM.

    Consider that update-motd is intended to replace the motd provided by
    setup, and provide that same content in the update-motd package. In
    this way, the original content from setup RPM will be displayed until
    the cron job for update-motd runs. After that, unless the user has
    changed to the motd, the same content will be presented.

    Partial-Bug: 1790863

    Change-Id: I2b66ce7475525920c52e129102fa859fb29e0eed
    Signed-off-by: Michel Thebeau <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/603821
Committed: https://git.openstack.org/cgit/openstack/stx-config/commit/?id=001e02ab2755cd3aa19887a5b349b8e957fdaaa1
Submitter: Zuul
Branch: master

commit 001e02ab2755cd3aa19887a5b349b8e957fdaaa1
Author: Michel Thebeau <email address hidden>
Date: Wed Sep 19 09:17:23 2018 -0400

    controllerconfig: apply the login banner, motd customization

    When the config is persisted during config_controller, the horizon
    branding is placed into platform directory. This is an appropriate
    time to also place the login banner and motd customization into platform
    directory.

    Refer also to the README in stx-integ, which lists the feature:
      utilities/branding/sources/login-banner/README

    Closes-Bug: 1790863

    Change-Id: Ia29bc8e56e07294e717819db5978199aec81e5b8
    Signed-off-by: Michel Thebeau <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/604202
Committed: https://git.openstack.org/cgit/openstack/stx-integ/commit/?id=19a59679e14c1eed83ec60a7015fbd2436a23f85
Submitter: Zuul
Branch: master

commit 19a59679e14c1eed83ec60a7015fbd2436a23f85
Author: Michel Thebeau <email address hidden>
Date: Thu Sep 20 12:52:33 2018 -0400

    update-motd: fix file permissions for /etc/motd.head

    Commit 685baaed added text file /etc/motd.head with execute
    permission.

    Partial-Bug: 1790863

    Change-Id: I1104a5d7320f5fc32801c1129df589ce32fdf656
    Signed-off-by: Michel Thebeau <email address hidden>