glance fails to import image as image has been saved on a diffrent host to the one where API request has been sent.

Bug #1982587 reported by Mirek
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
kolla-ansible
New
Undecided
Unassigned

Bug Description

**Environment**:
* OS ubunut 20.04
* Kolla-Ansible version Xena
* Docker image Install type source
* Docker image distribution: ubuntu
* Are you using official images: Yes

As in summary, I have 3x node HA cluster. HAProxy is in the default load balancing algorithm, I haven't seen it with small images, but with big images, I've tried 20GB and 50GB win10 images. The image has been saved in glance staging directory on one node, but then the request has been sent to another glance container on another host, and I can see the following ERROR in the logs. The image is correctly transferred and I can see it in another host but glance won't attempt to re-run the importing stage and probably won't know anyway where the file is, has 33% chance of getting it right. When I've added hash-type consistent to glance-api.cfg in haproxy it's all working as expected.

```

Task 'api_image_import-rbd-ImportToStore-16b6e222-e061-4a4e-b8f4-a420db0baf25' (f8f7f290-ced5-45a7-8b33-881958f6d416) transitioned into state 'FAILURE' from state 'RUNNING'
5 predecessors (most recent first):
  Flow 'api_image_import-rbd'
  |__Atom 'api_image_import-ConfigureStaging-16b6e222-e061-4a4e-b8f4-a420db0baf25' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {}, 'provides': 'file:///var/lib/glance/staging/20131196-75fe-4f9f-8417-21a3218a5b30'}
     |__Atom 'api_image_import-ImageLock-16b6e222-e061-4a4e-b8f4-a420db0baf25' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {}, 'provides': None}
        |__Atom 'api_image_import_retry' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {}, 'provides': [(None, {})]}
           |__Flow 'api_image_import': FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/glance/staging/20131196-75fe-4f9f-8417-21a3218a5b30'
2022-07-22 11:45:06.540 974 ERROR glance.async_.taskflow_executor Traceback (most recent call last):
2022-07-22 11:45:06.540 974 ERROR glance.async_.taskflow_executor File "/var/lib/kolla/venv/lib/python3.8/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2022-07-22 11:45:06.540 974 ERROR glance.async_.taskflow_executor result = task.execute(**arguments)
2022-07-22 11:45:06.540 974 ERROR glance.async_.taskflow_executor File "/var/lib/kolla/venv/lib/python3.8/site-packages/glance/async_/flows/api_image_import.py", line 596, in execute
2022-07-22 11:45:06.540 974 ERROR glance.async_.taskflow_executor self._execute(action, file_path)
2022-07-22 11:45:06.540 974 ERROR glance.async_.taskflow_executor File "/var/lib/kolla/venv/lib/python3.8/site-packages/glance/async_/flows/api_image_import.py", line 605, in _execute
2022-07-22 11:45:06.540 974 ERROR glance.async_.taskflow_executor action.set_image_data(file_path or self.uri,
2022-07-22 11:45:06.540 974 ERROR glance.async_.taskflow_executor File "/var/lib/kolla/venv/lib/python3.8/site-packages/glance/async_/flows/api_image_import.py", line 298, in set_image_data
2022-07-22 11:45:06.540 974 ERROR glance.async_.taskflow_executor return image_import.set_image_data(self._image, uri, task_id,
2022-07-22 11:45:06.540 974 ERROR glance.async_.taskflow_executor File "/var/lib/kolla/venv/lib/python3.8/site-packages/glance/common/scripts/image_import/main.py", line 158, in set_image_data
2022-07-22 11:45:06.540 974 ERROR glance.async_.taskflow_executor LOG.info(_LI("Task %(task_id)s: Could not import image file"
2022-07-22 11:45:06.540 974 ERROR glance.async_.taskflow_executor File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-07-22 11:45:06.540 974 ERROR glance.async_.taskflow_executor self.force_reraise()
2022-07-22 11:45:06.540 974 ERROR glance.async_.taskflow_executor File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-07-22 11:45:06.540 974 ERROR glance.async_.taskflow_executor raise self.value
2022-07-22 11:45:06.540 974 ERROR glance.async_.taskflow_executor File "/var/lib/kolla/venv/lib/python3.8/site-packages/glance/common/scripts/image_import/main.py", line 146, in set_image_data
2022-07-22 11:45:06.540 974 ERROR glance.async_.taskflow_executor data_iter = script_utils.get_image_data_iter(uri)
2022-07-22 11:45:06.540 974 ERROR glance.async_.taskflow_executor File "/var/lib/kolla/venv/lib/python3.8/site-packages/glance/common/scripts/utils.py", line 140, in get_image_data_iter
2022-07-22 11:45:06.540 974 ERROR glance.async_.taskflow_executor return open(uri, "rb")
2022-07-22 11:45:06.540 974 ERROR glance.async_.taskflow_executor FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/glance/staging/20131196-75fe-4f9f-8417-21a3218a5b30'
2022-07-22 11:45:06.540 974 ERROR glance.async_.taskflow_executor
```

```
ubuntu@infra-az1-1:~$ sudo cat /etc/kolla/haproxy/services.d/glance-api.cfg
frontend glance_api_front
    mode http
    http-request del-header X-Forwarded-Proto
    timeout client 6h
    option httplog
    option forwardfor
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    bind 172.16.71.20:9292 ssl crt /etc/haproxy/haproxy-internal.pem
    default_backend glance_api_back

backend glance_api_back
    mode http
    timeout server 6h
    hash-type consistent
    server infra-az1-1 172.16.71.4:9292 check inter 2000 rise 2 fall 5
    server infra-az2-1 172.16.71.5:9292 check inter 2000 rise 2 fall 5
    server infra-az3-1 172.16.71.6:9292 check inter 2000 rise 2 fall 5

frontend glance_api_external_front
    mode http
    http-request del-header X-Forwarded-Proto
    timeout client 6h
    option httplog
    option forwardfor
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    bind 172.16.72.20:9292 ssl crt /etc/haproxy/haproxy.pem
    default_backend glance_api_external_back

backend glance_api_external_back
    mode http
    timeout server 6h
    hash-type consistent
    server infra-az1-1 172.16.71.4:9292 check inter 2000 rise 2 fall 5
    server infra-az2-1 172.16.71.5:9292 check inter 2000 rise 2 fall 5
    server infra-az3-1 172.16.71.6:9292 check inter 2000 rise 2 fall 5

```

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.