OpenStack os-horizon-install failled at task "Ensure static files are collected and compressed"

[Daniel Pham]

Hello,

I used OpenStack-Ansible to deploy OpenStack on 3 nodes : 1 infra, 1 storage, 1 compute

I want to migrate from my current Rocky-em to stable/stein so I checkout to openstack-ansible stable/stein and run the script run-upgrade.sh

During the migration I encounter a fatal error who shut down the playbook setup-openstack

The task is "Ensure static files are collected and compressed" in the os-horizon-install.yml

You can find here the log of
$ openstack-ansible playbook/os-horizon-install.yml -vvv

TASK [os_horizon : Ensure static files are collected and compressed] ************************************************************************************************************************************************************************
task path: /etc/ansible/roles/os_horizon/tasks/main.yml:95
<infra1_horizon_container-cccecb28> The "physical_host" variable of "infra1" has been found to have a corresponding host entry in inventory.
<infra1_horizon_container-cccecb28> The "physical_host" variable of "infra1" terminates at "172.20.39.118" using the host variable "ansible_host".
container_name: "infra1_horizon_container-cccecb28"
physical_host: "infra1"
Container confirmed
Container type "lxc"
Using module file /opt/ansible-runtime/local/lib/python2.7/site-packages/ansible/modules/utilities/logic/async_status.py
container_name: "infra1_horizon_container-cccecb28"
physical_host: "infra1"
Container confirmed
Container type "lxc"
<172.20.39.118> ESTABLISH SSH CONNECTION FOR USER: root
<172.20.39.118> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=5 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o ServerAliveInterval=64 -o ServerAliveCountMax=1024 -o Compression=no -o TCPKeepAlive=yes -o VerifyHostKeyDNS=no -o ForwardX11=no -o ForwardAgent=yes -T -o ControlPath=/root/.ansible/cp/abfe0c8479 172.20.39.118 'lxc-attach --clear-env --name infra1_horizon_container-cccecb28 -- su - root -c '"'"'/bin/sh -c '"'"'"'"'"'"'"'"'sudo -H -S -n -u horizon /bin/sh -c '"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-qkvrxwcnrfpdckjkjelwrysbmpozgzqx; /usr/bin/python'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' && sleep 0'"'"'"'"'"'"'"'"''"'"''
Escalation succeeded
<172.20.39.118> (0, '\n{"changed": true, "end": "2021-03-03 08:43:23.442025", "ansible_job_id": "145211007.3212", "stdout": "\\n0 static files copied to \'/openstack/venvs/horizon-19.1.8.dev1/lib/python2.7/dist-packages/static\', 2133 unmodified.", "cmd": ["/openstack/venvs/horizon-19.1.8.dev1/bin/horizon-manage.py", "collectstatic", "--noinput"], "rc": 0, "start": "2021-03-03 08:43:15.739421", "finished": 1, "stderr": "WARNING:root:Skipping openstack_dashboard.local.enabled._1699_orchestration_settings because it doesn\'t have DASHBOARD, PANEL, PANEL_GROUP, or FEATURE defined.", "delta": "0:00:07.702604", "invocation": {"module_args": {"creates": null, "executable": null, "_uses_shell": false, "_raw_params": "/openstack/venvs/horizon-19.1.8.dev1/bin/horizon-manage.py collectstatic --noinput", "removes": null, "argv": null, "warn": true, "chdir": null, "stdin": null}}}\n', 'mesg: ttyname failed: Inappropriate ioctl for device\n')
changed: [infra1_horizon_container-cccecb28] => (item={'_ansible_parsed': True, '_ansible_item_result': True, '_ansible_item_label': u'/openstack/venvs/horizon-19.1.8.dev1/bin/horizon-manage.py collectstatic --noinput', u'ansible_job_id': u'145211007.3212', 'item': u'/openstack/venvs/horizon-19.1.8.dev1/bin/horizon-manage.py collectstatic --noinput', '_ansible_notify': [u'Restart apache2'], 'changed': False, u'started': 1, 'failed': False, u'finished': 0, u'results_file': u'/var/lib/horizon/.ansible_async/145211007.3212', '_ansible_ignore_errors': None, '_ansible_no_log': False}) => {
    "ansible_job_id": "145211007.3212",
    "attempts": 1,
    "changed": true,
    "cmd": [
        "/openstack/venvs/horizon-19.1.8.dev1/bin/horizon-manage.py",
        "collectstatic",
        "--noinput"
    ],
    "delta": "0:00:07.702604",
    "end": "2021-03-03 08:43:23.442025",
    "finished": 1,
    "invocation": {
        "module_args": {
            "_raw_params": "/openstack/venvs/horizon-19.1.8.dev1/bin/horizon-manage.py collectstatic --noinput",
            "_uses_shell": false,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "item": {
        "ansible_job_id": "145211007.3212",
        "changed": false,
        "failed": false,
        "finished": 0,
        "item": "/openstack/venvs/horizon-19.1.8.dev1/bin/horizon-manage.py collectstatic --noinput",
        "results_file": "/var/lib/horizon/.ansible_async/145211007.3212",
        "started": 1
    },
    "rc": 0,
    "start": "2021-03-03 08:43:15.739421",
    "stderr": "WARNING:root:Skipping openstack_dashboard.local.enabled._1699_orchestration_settings because it doesn't have DASHBOARD, PANEL, PANEL_GROUP, or FEATURE defined.",
    "stderr_lines": [
        "WARNING:root:Skipping openstack_dashboard.local.enabled._1699_orchestration_settings because it doesn't have DASHBOARD, PANEL, PANEL_GROUP, or FEATURE defined."
    ],
    "stdout": "\n0 static files copied to '/openstack/venvs/horizon-19.1.8.dev1/lib/python2.7/dist-packages/static', 2133 unmodified.",
    "stdout_lines": [
        "",
        "0 static files copied to '/openstack/venvs/horizon-19.1.8.dev1/lib/python2.7/dist-packages/static', 2133 unmodified."
    ]
}
container_name: "infra1_horizon_container-cccecb28"
physical_host: "infra1"
Container confirmed
Container type "lxc"
Using module file /opt/ansible-runtime/local/lib/python2.7/site-packages/ansible/modules/utilities/logic/async_status.py
container_name: "infra1_horizon_container-cccecb28"
physical_host: "infra1"
Container confirmed
Container type "lxc"
<172.20.39.118> ESTABLISH SSH CONNECTION FOR USER: root
<172.20.39.118> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=5 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o ServerAliveInterval=64 -o ServerAliveCountMax=1024 -o Compression=no -o TCPKeepAlive=yes -o VerifyHostKeyDNS=no -o ForwardX11=no -o ForwardAgent=yes -T -o ControlPath=/root/.ansible/cp/abfe0c8479 172.20.39.118 'lxc-attach --clear-env --name infra1_horizon_container-cccecb28 -- su - root -c '"'"'/bin/sh -c '"'"'"'"'"'"'"'"'sudo -H -S -n -u horizon /bin/sh -c '"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-xduotkbjbdhioazbkvwrphsywkomggst; /usr/bin/python'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' && sleep 0'"'"'"'"'"'"'"'"''"'"''
Escalation succeeded
<172.20.39.118> (0, '\n{"started": 1, "invocation": {"module_args": {"jid": "409383834513.3265", "mode": "status"}}, "finished": 0, "ansible_job_id": "409383834513.3265"}\n', 'mesg: ttyname failed: Inappropriate ioctl for device\n')
FAILED - RETRYING: Ensure static files are collected and compressed (300 retries left).Result was: {
    "ansible_job_id": "409383834513.3265",
    "attempts": 1,
    "changed": false,
    "finished": 0,
    "invocation": {
        "module_args": {
            "jid": "409383834513.3265",
            "mode": "status"
        }
    },
    "retries": 301,
    "started": 1
}

300 retry and then :

Using module file /opt/ansible-runtime/local/lib/python2.7/site-packages/ansible/modules/utilities/logic/async_status.py
container_name: "infra1_horizon_container-cccecb28"
physical_host: "infra1"
Container confirmed
Container type "lxc"
<172.20.39.118> ESTABLISH SSH CONNECTION FOR USER: root
<172.20.39.118> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=5 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o ServerAliveInterval=64 -o ServerAliveCountMax=1024 -o Compression=no -o TCPKeepAlive=yes -o VerifyHostKeyDNS=no -o ForwardX11=no -o ForwardAgent=yes -T -o ControlPath=/root/.ansible/cp/abfe0c8479 172.20.39.118 'lxc-attach --clear-env --name infra1_horizon_container-cccecb28 -- su - root -c '"'"'/bin/sh -c '"'"'"'"'"'"'"'"'sudo -H -S -n -u horizon /bin/sh -c '"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-axbodqkjqberyiytacddbwlwoghnmbwh; /usr/bin/python'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' && sleep 0'"'"'"'"'"'"'"'"''"'"''
Escalation succeeded
<172.20.39.118> (0, '\n{"started": 1, "invocation": {"module_args": {"jid": "409383834513.3265", "mode": "status"}}, "finished": 0, "ansible_job_id": "409383834513.3265"}\n', 'mesg: ttyname failed: Inappropriate ioctl for device\n')
failed: [infra1_horizon_container-cccecb28] (item={'_ansible_parsed': True, '_ansible_item_result': True, '_ansible_item_label': u'/openstack/venvs/horizon-19.1.8.dev1/bin/horizon-manage.py compress --force', u'ansible_job_id': u'409383834513.3265', 'item': u'/openstack/venvs/horizon-19.1.8.dev1/bin/horizon-manage.py compress --force', '_ansible_notify': [u'Restart apache2'], 'changed': False, u'started': 1, 'failed': False, u'finished': 0, u'results_file': u'/var/lib/horizon/.ansible_async/409383834513.3265', '_ansible_ignore_errors': None, '_ansible_no_log': False}) => {
    "ansible_job_id": "409383834513.3265",
    "attempts": 300,
    "changed": false,
    "finished": 0,
    "invocation": {
        "module_args": {
            "jid": "409383834513.3265",
            "mode": "status"
        }
    },
    "item": {
        "ansible_job_id": "409383834513.3265",
        "changed": false,
        "failed": false,
        "finished": 0,
        "item": "/openstack/venvs/horizon-19.1.8.dev1/bin/horizon-manage.py compress --force",
        "results_file": "/var/lib/horizon/.ansible_async/409383834513.3265",
        "started": 1
    },
    "started": 1
}
Read vars_file 'defaults/repo_packages/openstack_services.yml'
skipping vars_file 'defaults/{{ install_method }}_install.yml' due to an undefined variable

At the end of the script the horizon dashboard didn't have any CSS and several resources was not found, I ran the "/openstack/venvs/horizon-19.1.8.dev1/bin/horizon-manage.py compress --force" manually into my horizon container (into the infra node) and it ends well. Horizon was slightly prettier but not fully functional.

I destroy the horizon-container with the
$ openstack-ansible playbooks/lxc-container-destroy.yml --limit infra-horizon-container
Recreate it
$ openstack-ansible playbooks/lxc-container-create.yml --limit infra-horizon-container
$ openstack-ansible playbooks/os-horizon-install.yml

But the error persists, the task "Ensure static files are collected and compressed" keeps failing.

Thank you in advance for any hint or workaround

[Daniel Pham (shin-ichi)]

Hello again,

I re-ran the following commands manually inside the horizon container :

  - sudo -u horizon -s
  - /openstack/venvs/horizon-19.1.7.dev1/bin/horizon-manage.py collectstatic --noinput

Despite the warning (WARNING:root:Skipping openstack_dashboard.local.enabled._1699_orchestration_settings because it doesn't have DASHBOARD, PANEL, PANEL_GROUP, or FEATURE defined.) I ran

  - /openstack/venvs/horizon-19.1.7.dev1/bin/horizon-manage.py compress --force

After 20 minutes the compression ends well

I update the configuration file of Apache

  - sed -i.old 's/18.1.20/19.1.7/' /etc/apache2/sites-enabled/openstack-dashboard.conf
  - service apache2 reload
  - service apache2 restart

And the inconsistency of Horizon dashboard disappeared.

To complete the run-upgrade.sh script I by pass the Task "Ensure static files are collected and compressed'.

Summary the Task compress works fine but the ansible job never change the file to something like this:
cat /var/lib/horizon/.ansible_async/<job_id>
{"started": 1, "finished": 1, "ansible_job_id": <job_id>}

[Jonathan Rosser (jrosser)]

Hi Daniel,

I think there is a patch for this issue here and i've created a backport all the way back to stein. https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/779471

It's merged in master and victoria branches but no earlier. Could you check if this patch fixes the issue for you?

Thanks,
Jonathan.

[Daniel Pham (shin-ichi)]

Hi Jonathan,

I applied the patch, but unfortunately the tasks remains in error state.

I supervise the processes "/openstack/venvs/horizon-19.1.7/bin/horizon-manage.py compress --force" I think this command did not end well when it is performed by the playbook.
Indeed the process ends in 10 minutes while it ends in 20 minutes when I ran it manually.

So the tasks "Ensure compress statics files are collected" failed, despite the patch.

Step to reproduce :
  - Deploy OpenStack with Openstack Ansible version stein-em

Like I said earlier, some manuals manipulations can by pass this error, so this issue have not an high priority. Moreover this issue affect an old version of OpenStack Ansible.
Feel free to closed the topic.

Thank you for your time,
Daniel.