R3.0 build 60 Kilo :glance takes 210s for create image of ubuntu image

[sundarkh]

R3.0 build 60 Kilo :glance takes 210s for create image of ubuntu image

Sanity Test case TestECMPSanityIPv6.test_ecmp_svc_in_network_with_3_instance fails with

Traceback (most recent call last):
  File "tcutils/wrappers.py", line 73, in wrapper
    result = function(self, *args, **kwargs)
  File "scripts/ecmp/test_ecmp.py", line 98, in test_ecmp_svc_in_network_with_3_instance
    si_count=1, svc_scaling=True, max_inst=3, svc_mode=svc_mode)
  File "common/servicechain/firewall/verify.py", line 299, in verify_svc_in_network_datapath
    right_vn=self.vn2_fixture.vn_fq_name, svc_mode=svc_mode, flavor=flavor, static_route=static_route, ordered_interfaces=ordered_interfaces, svc_img_name=svc_img_name, project=self.inputs.project_name, st_version=st_version)
  File "common/servicechain/config.py", line 75, in config_st_si
    if_list=if_list, svc_mode=svc_mode, svc_scaling=svc_scaling, flavor=flavor, ordered_interfaces=ordered_interfaces, version=st_version))
  File "/contrail-test/fixtures/svc_template_fixture.py", line 23, in __init__
    self.nova_h.get_image(self.image_name)
  File "/contrail-test/fixtures/nova_test.py", line 137, in get_image
    self._install_image(image_name=image_name)
  File "/contrail-test/fixtures/nova_test.py", line 235, in _install_image
    return self.copy_and_glance(build_path, image_name, image, params, image_type)
  File "/contrail-test/fixtures/nova_test.py", line 321, in copy_and_glance
    self.execute_cmd_with_proxy(cmd)
  File "/contrail-test/fixtures/nova_test.py", line 290, in execute_cmd_with_proxy
    sudo(cmd)
  File "/usr/local/lib/python2.7/dist-packages/fabric/network.py", line 677, in host_prompting_wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/fabric/operations.py", line 1146, in sudo
    capture_buffer_size=capture_buffer_size,
  File "/usr/local/lib/python2.7/dist-packages/fabric/operations.py", line 931, in _run_command
    capture_buffer_size=capture_buffer_size)
  File "/usr/local/lib/python2.7/dist-packages/fabric/operations.py", line 813, in _execute
    worker.raise_if_needed()
  File "/usr/local/lib/python2.7/dist-packages/fabric/thread_handling.py", line 12, in wrapper
    callable(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/fabric/io.py", line 32, in output_loop
    OutputLooper(*args, **kwargs).loop()
  File "/usr/local/lib/python2.7/dist-packages/fabric/io.py", line 87, in loop
    raise CommandTimeout(timeout=self.timeout)
CommandTimeout: Command failed to finish in 120 seconds

while using the following command

glance --os-username TestECMPSanityIPv6 --os-password TestECMPSanityIPv6 --os-tenant-name ctest-TestECMPSanityIPv6-00283178 --os-auth-url http://10.204.217.176:35357/v2.0 --os-region-name RegionOne image-create --name "ubuntu-in-net" --is-public True --container-format bare --disk-format vmdk --property vmware_disktype="sparse" --property vmware_adaptertype="ide" --file /tmp/ubuntu-in-net.vmdk

While trying manually , the above command took 210s to complete,
--------------------------------------+---------------+-------------+------------------+------------+--------+
| ID | Name | Disk Format | Container Format | Size | Status |
+--------------------------------------+---------------+-------------+------------------+------------+--------+
| 450eebca-0909-4fe3-8bb1-45130eecf452 | ubuntu | vmdk | bare | 304953856 | active |
| 3aa89932-06cf-43e3-a4dc-4f16f4cd0816 | ubuntu-in-net | vmdk | bare | 1363214336 | active |

Seen with nodes, provisioned using fab/SM

/tmp/ubuntu-in-net.vmdk is 1.3 G

[sundarkh (sundar-kh)]

ubuntu-dhcpdns-server.vmdk.gz, this image used in TestEvpnCasesVxlan.test_with_vxlan_encap_to_verify_l2_vm_file_trf_by_scp test case also faces the same issue

[Abhay Joshi (abhayj)]

Ashish,

Need someone in contrail to look into this one. Can you please reassign as appropriate?

Thanks,

Abhay

[sundarkh (sundar-kh)]

test_intf_mirroring_1 test case also failing with glance image ubuntu-traffic.vmdk.gz',

Traceback (most recent call last):
  File "tcutils/wrappers.py", line 73, in wrapper
    result = function(self, *args, **kwargs)
  File "scripts/intf_mirror/test_intf_mirror.py", line 30, in test_intf_mirroring_1
    return self.verify_intf_mirroring_1()
  File "common/intf_mirroring/verify.py", line 36, in verify_intf_mirroring_1
    return self.verify_intf_mirroring(src_compute, dst_compute, analyzer_compute)
  File "common/intf_mirroring/verify.py", line 257, in verify_intf_mirroring
    vn_fix=self.vn1_fixture, vm_name=self.vm1_name_vn1, node_name=src_compute, image_name=image_name)
  File "common/servicechain/config.py", line 169, in config_vm
    vn_obj=vn_fix.obj, vm_name=vm_name, node_name=node_name, image_name=image_name, flavor=flavor, count=count))
  File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 679, in useFixture
    reraise(*exc_info)
  File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 666, in useFixture
    fixture.setUp()
  File "/contrail-test/fixtures/vm_test.py", line 161, in setUp
    self.create()
  File "/contrail-test/fixtures/vm_test.py", line 198, in create
    fixed_ips=self.fixed_ips)
  File "/contrail-test/fixtures/openstack.py", line 78, in create_vm
    return self.nova_h.create_vm(vm_name=vm_name, image_name=image_name, vn_ids=vn_ids, **kwargs)
  File "/contrail-test/fixtures/nova_test.py", line 456, in create_vm
    image = self.get_image(image_name=image_name)
  File "/contrail-test/fixtures/nova_test.py", line 137, in get_image
    self._install_image(image_name=image_name)
  File "/contrail-test/fixtures/nova_test.py", line 235, in _install_image
    return self.copy_and_glance(build_path, image_name, image, params, image_type)
  File "/contrail-test/fixtures/nova_test.py", line 321, in copy_and_glance
    self.execute_cmd_with_proxy(cmd)
  File "/contrail-test/fixtures/nova_test.py", line 290, in execute_cmd_with_proxy
    sudo(cmd)
  File "/usr/local/lib/python2.7/dist-packages/fabric/network.py", line 677, in host_prompting_wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/fabric/operations.py", line 1146, in sudo
    capture_buffer_size=capture_buffer_size,
  File "/usr/local/lib/python2.7/dist-packages/fabric/operations.py", line 931, in _run_command
    capture_buffer_size=capture_buffer_size)
  File "/usr/local/lib/python2.7/dist-packages/fabric/operations.py", line 813, in _execute
    worker.raise_if_needed()
  File "/usr/local/lib/python2.7/dist-packages/fabric/thread_handling.py", line 12, in wrapper
    callable(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/fabric/io.py", line 32, in output_loop
    OutputLooper(*args, **kwargs).loop()
  File "/usr/local/lib/python2.7/dist-packages/fabric/io.py", line 87, in loop
    raise CommandTimeout(timeout=self.timeout)
CommandTimeout: Command failed to finish in 120 seconds

[Atul Moghe (moghea)]

where is a image ubuntu-in-net.vmdk, on /cs-shared/images/ we have ubuntu-in-net.img which is of similar size as mentioned in the bug. On single node setup with default tenant and user, glance could create image in less than 2 sec.

Need to know the following:
1. vmdk image in question?
2. need to have a setup in problem state, let me know when you recreate, I can work with you.

[sundarkh (sundar-kh)]

Setup Provided with issue reproduced

[Atul Moghe (moghea)]

There are potentially two issues with this setup.

1. Discrepancy in keystone_authtoken section in glance-api.conf and glance-regisry.conf:
This setup is provisioned with SM and keystone_authtoken in glance-api and glance-registry have discrepancy as compared to fab provisioning.
In case of ha setup, identity_uri should be with address:port of internal_vip:5000, but on this setup it is local_host:35357
There is another parameter got added in this section (auth_uri), which is pointing to proper port (5000) but node ip address not to the internal_vip.
When auth_uri is present, glance will ignore auth_port, auth_protocol and auth_host (some of those are correct in conf files). Fab does not write to auth_uri but individual components (auth_host, auth_port and auth_protocol). wrong configuration may not cause any issue as long as glance and keystone are on same node.

2. Nfs mount issue on this setup
Most of the time, glance stuck in saving the image. Images are stored at /var/lib/glance/images/ which was mounted on other node, could be node might be unresponsive. Delay is image creation is mainly due to glance stuck in saving.. we need to investigate setup (node with ip address 10.204.217.61. Reassigning bug to QA

[Jeba Paulaiyan (jebap)]

Issue #1 in comment #6:

SM bug, will file one for post 3.0.3.0 fix.

Issue #2 in comment #6:

Could be due to slow NFS mount due to system running with full RAM capacity. Will investigate and close the bug.

[Jeba Paulaiyan (jebap)]

Sundar, can you update from latest test run? If it is not happening in latest images/newer testbeds, this can be closed.