Tempest: cirros-0.5.2 image kernel panic

Bug #1926734 reported by Amol Kahat
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned

Bug Description

Description
============
Cirros-0.5.2 image failing with kernel panic

Actual Result
=============
{0} tempest.scenario.test_snapshot_pattern.TestSnapshotPattern.test_snapshot_pattern [348.660322s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 113, in _get_ssh_connection
        sock=proxy_chan)
      File "/usr/lib/python3.6/site-packages/paramiko/client.py", line 349, in connect
        retry_on_signal(lambda: sock.connect(addr))
      File "/usr/lib/python3.6/site-packages/paramiko/util.py", line 283, in retry_on_signal
        return function()
      File "/usr/lib/python3.6/site-packages/paramiko/client.py", line 349, in <lambda>
        retry_on_signal(lambda: sock.connect(addr))
    socket.timeout: timed out

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/tempest/common/utils/__init__.py", line 90, in wrapper
        return f(*func_args, **func_kwargs)
      File "/usr/lib/python3.6/site-packages/tempest/scenario/test_snapshot_pattern.py", line 63, in test_snapshot_pattern
        server=server)
      File "/usr/lib/python3.6/site-packages/tempest/scenario/manager.py", line 907, in create_timestamp
        username=username)
      File "/usr/lib/python3.6/site-packages/tempest/scenario/manager.py", line 638, in get_remote_client
        linux_client.validate_authentication()
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 59, in wrapper
        six.reraise(*original_exception)
      File "/usr/local/lib/python3.6/site-packages/six.py", line 703, in reraise
        raise value
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 32, in wrapper
        return function(self, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 115, in validate_authentication
        self.ssh_client.test_connection_auth()
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 217, in test_connection_auth
        connection = self._get_ssh_connection()
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 129, in _get_ssh_connection
        password=self.password)
    tempest.lib.exceptions.SSHTimeout: Connection to the 192.168.24.114 via SSH timed out.
    User: cirros, Password: None
.
.
.
    2021-04-30 01:35:20,090 464853 INFO [tempest.lib.common.ssh] Creating ssh connection to '192.168.24.114:22' as 'cirros' with public key authentication
    2021-04-30 01:36:20,150 464853 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.114 (timed out). Number attempts: 1. Retry after 2 seconds.
    2021-04-30 01:37:22,691 464853 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.114 (timed out). Number attempts: 2. Retry after 3 seconds.
    2021-04-30 01:38:26,252 464853 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.114 (timed out). Number attempts: 3. Retry after 4 seconds.
    2021-04-30 01:39:30,819 464853 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.114 (timed out). Number attempts: 4. Retry after 5 seconds.
    2021-04-30 01:40:36,372 464853 ERROR [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.114 after 4 attempts. Proxy client: no proxy client
    2021-04-30 01:40:36.372 464853 ERROR tempest.lib.common.ssh Traceback (most recent call last):
    2021-04-30 01:40:36.372 464853 ERROR tempest.lib.common.ssh File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 113, in _get_ssh_connection
    2021-04-30 01:40:36.372 464853 ERROR tempest.lib.common.ssh sock=proxy_chan)
    2021-04-30 01:40:36.372 464853 ERROR tempest.lib.common.ssh File "/usr/lib/python3.6/site-packages/paramiko/client.py", line 349, in connect
    2021-04-30 01:40:36.372 464853 ERROR tempest.lib.common.ssh retry_on_signal(lambda: sock.connect(addr))
    2021-04-30 01:40:36.372 464853 ERROR tempest.lib.common.ssh File "/usr/lib/python3.6/site-packages/paramiko/util.py", line 283, in retry_on_signal
    2021-04-30 01:40:36.372 464853 ERROR tempest.lib.common.ssh return function()
    2021-04-30 01:40:36.372 464853 ERROR tempest.lib.common.ssh File "/usr/lib/python3.6/site-packages/paramiko/client.py", line 349, in <lambda>
    2021-04-30 01:40:36.372 464853 ERROR tempest.lib.common.ssh retry_on_signal(lambda: sock.connect(addr))
    2021-04-30 01:40:36.372 464853 ERROR tempest.lib.common.ssh socket.timeout: timed out
    2021-04-30 01:40:36.372 464853 ERROR tempest.lib.common.ssh

.
.
    [ 4.159768] prepare_namespace+0x139/0x18e
    [ 4.159913] kernel_init_freeable+0x245/0x26d
    [ 4.160023] ? rest_init+0xb0/0xb0
    [ 4.160125] kernel_init+0xe/0x110
    [ 4.160215] ret_from_fork+0x35/0x40
    [ 4.161021] Kernel Offset: 0x34600000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
    [ 4.161588] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---

Logs
====
- https://logserver.rdoproject.org/periodic/review.rdoproject.org/rdoinfo/master/cloudsig-victoria-release-centos-8-scenario001-standalone/fa891a7/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz
- https://logserver.rdoproject.org/periodic/review.rdoproject.org/rdoinfo/master/cloudsig-victoria-release-centos-8-scenario001-standalone/c9c9d4a/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

Additional Info
===============
Cirros-0.5.1 BZ: https://bugs.launchpad.net/tripleo/+bug/1885279
Patch Fix: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/738025

Revision history for this message
Amol Kahat (amolkahat) wrote :
Revision history for this message
wes hayutin (weshayutin) wrote :

Do not revert back to cirros 4.0 here. Let's see how often this happens and perhaps skiplist the tests that hit this often.

Revision history for this message
wes hayutin (weshayutin) wrote :
Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.