Tests don't run correctly with a non-local lxd default-remote

Bug #1771583 reported by Dan Watkins
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init
Expired
Medium
Unassigned

Bug Description

I run a lxd server in a cloud instance, and have that configured as my default remote. The cloud-init test framework still launches its first container ("cloud-test-ubuntu-bionic-image-modification-oot8q3h1p4jup1jb8ce") locally, but then fails to use it properly. When I change my lxd configuration to use my local host as default-remote, the tests run correctly.

---

$ python3 -m tests.cloud_tests run --os-name bionic --verbose
2018-05-16 13:56:34,694 - tests.cloud_tests - DEBUG - running with args: Namespace(data_dir=None, deb=None, feature_override={}, os_name=['bionic'], platform=['lxd'], ppa=None, preserve_data=False, quiet=False, repo=None, result=None, rpm=None, script=None, subcmd='run', test_config=['/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/bugs/lp1511485.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/bugs/lp1611074.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/bugs/lp1628337.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/examples/add_apt_repositories.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/examples/alter_completion_message.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/examples/configure_instance_trusted_ca_certificates.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/examples/configure_instances_ssh_keys.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/examples/including_user_groups.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/examples/install_arbitrary_packages.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/examples/install_run_chef_recipes.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/examples/run_apt_upgrade.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/examples/run_commands.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/examples/run_commands_first_boot.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/examples/setup_run_puppet.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/examples/writing_out_arbitrary_files.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/main/command_output_simple.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/apt_configure_conf.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/apt_configure_disable_suites.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/apt_configure_primary.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/apt_configure_proxy.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/apt_configure_security.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/apt_configure_sources_key.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/apt_configure_sources_keyserver.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/apt_configure_sources_list.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/apt_configure_sources_ppa.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/apt_pipelining_disable.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/apt_pipelining_os.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/bootcmd.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/byobu.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/ca_certs.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/debug_disable.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/debug_enable.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/final_message.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/keys_to_console.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/landscape.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/locale.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/lxd_bridge.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/lxd_dir.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/ntp.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/ntp_chrony.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/ntp_pools.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/ntp_servers.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/ntp_timesyncd.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/package_update_upgrade_install.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/runcmd.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/salt_minion.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/seed_random_command.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/seed_random_data.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/set_hostname.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/set_hostname_fqdn.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/set_password.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/set_password_expire.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/set_password_list.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/set_password_list_string.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/snap.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/snappy.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/ssh_auth_key_fingerprints_disable.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/ssh_auth_key_fingerprints_enable.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/ssh_import_id.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/ssh_keys_generate.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/ssh_keys_provided.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/timezone.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/user_groups.yaml', '/home/daniel/dev/cloud-init/tests/cloud_tests/testcases/modules/write_files.yaml'], upgrade=False, upgrade_full=False, verbose=True)
2018-05-16 13:56:34,695 - tests.cloud_tests - DEBUG - using tmpdir: /tmp/cloud_test_data_erivf9yz
2018-05-16 13:56:34,702 - tests.cloud_tests - DEBUG - platform config: {'enabled': True, 'get_image_timeout': 300, 'create_instance_timeout': 60, 'private_key': 'cloud_init_rsa', 'public_key': 'cloud_init_rsa.pub', 'template_overrides': {'/var/lib/cloud/seed/nocloud-net/meta-data': {'when': ['create', 'copy'], 'template': 'cloud-init-meta.tpl'}, '/var/lib/cloud/seed/nocloud-net/network-config': {'when': ['create', 'copy'], 'template': 'cloud-init-network.tpl'}, '/var/lib/cloud/seed/nocloud-net/user-data': {'when': ['create', 'copy'], 'template': 'cloud-init-user.tpl', 'properties': {'default': '#cloud-config\n{}\n'}}, '/var/lib/cloud/seed/nocloud-net/vendor-data': {'when': ['create', 'copy'], 'template': 'cloud-init-vendor.tpl', 'properties': {'default': '#cloud-config\n{}\n'}}}, 'template_files': {'cloud-init-meta.tpl': '#cloud-config\ninstance-id: {{ container.name }}\nlocal-hostname: {{ container.name }}\n{{ config_get("user.meta-data", "") }}\n', 'cloud-init-network.tpl': '{% if config_get("user.network-config", "") == "" %}version: 1\nconfig:\n - type: physical\n name: eth0\n subnets:\n - type: {% if config_get("user.network_mode", "") == "link-local" %}manual{% else %}dhcp{% endif %}\n control: auto{% else %}{{ config_get("user.network-config", "") }}{% endif %}\n', 'cloud-init-user.tpl': '{{ config_get("user.user-data", properties.default) }}\n', 'cloud-init-vendor.tpl': '{{ config_get("user.vendor-data", properties.default) }}\n'}, 'data_dir': '/tmp/cloud_test_data_erivf9yz'}
2018-05-16 13:56:34,702 - tests.cloud_tests - INFO - setting up platform: lxd
2018-05-16 13:56:35,781 - tests.cloud_tests - DEBUG - os config: {'enabled': True, 'boot_timeout': 120, 'boot_clean_script': '#!/bin/bash\nrm -rf /var/log/cloud-init.log /var/log/cloud-init-output.log \\\n /var/lib/cloud/ /run/cloud-init/ /var/log/syslog\n', 'system_ready_script': "# permit running or degraded state as both indicate complete boot\n[ $(systemctl is-system-running) = 'running' -o\n $(systemctl is-system-running) = 'degraded' ]\n", 'cloud_init_ready_script': "[ -f '/run/cloud-init/result.json' ]\n", 'feature_groups': ['base', 'debian_base', 'ubuntu_specific'], 'features': {'apt': True, 'byobu': True, 'landscape': True, 'lxd': True, 'ppa': True, 'rpm': None, 'snap': True, 'hostname': True, 'apt_src_cont': True, 'apt_hist_fmt': True, 'daylight_time': True, 'apt_up_out': True, 'engb_locale': True, 'locale_gen': True, 'no_ntpdate': True, 'no_file_fmt_e': True, 'ppa_file_name': True, 'sshd': True, 'ssh_key_fmt': True, 'syslog': True, 'ubuntu_ntp': True, 'ubuntu_repos': True, 'ubuntu_user': True, 'lsb_release': True, 'sudo': True}, 'mirror_url': 'https://cloud-images.ubuntu.com/daily', 'mirror_dir': '/srv/citest/images', 'keyring': '/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg', 'version': 18.04, 'sstreams_server': 'https://cloud-images.ubuntu.com/daily', 'cache_base_image': True, 'override_templates': False, 'setup_overrides': None, 'release': 'bionic', 'os': 'ubuntu', 'alias': 'bionic', 'arch': 'amd64'}
2018-05-16 13:56:35,781 - tests.cloud_tests - INFO - acquiring image for os: bionic
2018-05-16 13:56:35,859 - tests.cloud_tests - DEBUG - updating args for setup with: None
2018-05-16 13:56:35,860 - tests.cloud_tests - INFO - setting up ubuntu-bionic
2018-05-16 13:56:35,898 - tests.cloud_tests - DEBUG - creating snapshot for bionic
2018-05-16 13:56:46,114 - tests.cloud_tests - DEBUG - no console-support: LXD server does not support console api extension
2018-05-16 13:56:46,115 - tests.cloud_tests - DEBUG - Set console log method to logfile-tmp
2018-05-16 13:56:46,940 - tests.cloud_tests - DEBUG - executing command: sh -c 'i=0; while [ $i -lt 120 ] && i=$(($i+1)); do [ $(systemctl is-system-running) = '"'"'running'"'"' -o $(systemctl is-system-running) = '"'"'degraded'"'"' ] && exit 0; sleep 1; done; exit 1'
2018-05-16 13:56:49,133 - tests.cloud_tests - ERROR - stage: set up and collect data for os: bionic encountered error: timeout: after 120s system not started
2018-05-16 13:56:49,135 - tests.cloud_tests - ERROR - traceback:
  File "/home/daniel/dev/cloud-init/tests/cloud_tests/stage.py", line 88, in run_stage
    (call_res, call_failed) = call()
  File "/home/daniel/dev/cloud-init/tests/cloud_tests/collect.py", line 128, in collect_snapshot
    with component as snapshot:
  File "/home/daniel/dev/cloud-init/tests/cloud_tests/stage.py", line 21, in __enter__
    self.instance = self.get_func()
  File "/home/daniel/dev/cloud-init/tests/cloud_tests/platforms/__init__.py", line 36, in get_snapshot
    return image.snapshot()
  File "/home/daniel/dev/cloud-init/tests/cloud_tests/platforms/lxd/image.py", line 176, in snapshot
    container=self._instance.name, image_desc=str(self),
  File "/home/daniel/dev/cloud-init/tests/cloud_tests/platforms/lxd/image.py", line 63, in _instance
    self._img_instance.start()
  File "/home/daniel/dev/cloud-init/tests/cloud_tests/platforms/lxd/instance.py", line 178, in start
    self._wait_for_system(wait_for_cloud_init)
  File "/home/daniel/dev/cloud-init/tests/cloud_tests/platforms/instances.py", line 142, in _wait_for_system
    raise OSError('timeout: after {}s system not started'.format(time))

2018-05-16 13:56:58,441 - tests.cloud_tests - DEBUG - collect stages: {'name': 'collect data', 'time': 23.746460676193237, 'errors': [], 'stages': [{'name': 'collect for platform: lxd', 'time': 22.691640615463257, 'errors': [], 'stages': [{'name': 'set up and collect data for os: bionic', 'time': 13.275653839111328, 'errors': ['timeout: after 120s system not started'], 'stages': [{'name': 'set up for ubuntu-bionic', 'time': 9.5367431640625e-07, 'errors': [], 'stages': [], 'success': True}], 'success': False}], 'success': False}], 'success': False}
2018-05-16 13:56:58,442 - tests.cloud_tests - DEBUG - found test data: {}

Joshua Powers (powersj)
Changed in cloud-init:
status: New → Confirmed
assignee: nobody → Joshua Powers (powersj)
importance: Undecided → Medium
Joshua Powers (powersj)
Changed in cloud-init:
assignee: Joshua Powers (powersj) → nobody
Revision history for this message
James Falcon (falcojr) wrote :
Changed in cloud-init:
status: Confirmed → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.