| 2020-05-27 07:35:04 |
Taihsiang Ho |
bug |
|
|
added bug |
| 2020-05-27 07:37:30 |
Taihsiang Ho |
attachment added |
|
var-log-nova-kreiken-200526.tar.gz https://bugs.launchpad.net/openstack-bundles/+bug/1880828/+attachment/5377260/+files/var-log-nova-kreiken-200526.tar.gz |
|
| 2020-05-27 07:40:57 |
Taihsiang Ho |
description |
bundle: openstack-base-bionic-train https://github.com/openstack-charmers/openstack-bundles/blob/master/development/openstack-base-bionic-train/bundle.yaml
hardware: 2 d05 and 2 d06 (the log of the compute node is from one of the d06)
When trying to create new instances on the deployed openstack, the instance is always in the status of "spawning"
[Steps to Reproduce]
1. Deploy with the above bundle and hardware by following the instruction of https://jaas.ai/openstack-base/bundle/67
2. Wait about 1.5 until the deployment is ready. By ready it means every unit shows its message as "ready" e.g. https://paste.ubuntu.com/p/k48YVnPyVZ/
3. Follow the instruction of https://jaas.ai/openstack-base/bundle/67 until the step of "openstack server create" to create new instance. This step is also summarized in details in this gist code snippet https://gist.github.com/tai271828/b0c00a611e703046dd52da12a66226b0#file-02-basic-test-just-deployed-sh
[Expected Behavior]
An instance is created a few second later
[Actual Behavior]
The status of the instance is always (> 20 minutes) "spawning"
[Additional Information]
1. By waiting for 0.5 ~ 1 hr first, and then restart the nova-compute by "sudo service nova-compute restart", may make the creation behavior back to normal. You may refer to this snippet to see this workaround https://gist.github.com/tai271828/b0c00a611e703046dd52da12a66226b0#file-03-nova-compute-debugging-sh
2. By applying the above workaround and then check log from the corresponding nova-compute node, you could find lines like:
818 2020-05-26 02:33:17.845 417168 INFO nova.compute.claims [req-9b570186-d136-4df5-8b90-6dd3e8b67f3e 91f35a24c0b1465fbd147d70e439646e e16f574f6448430aac827c968e5145be - e61235e9fe7f4 d0284f035c7588dd2b9 e61235e9fe7f4d0284f035c7588dd2b9] [instance: 54e49495-c1c2-4567-8493-2c412d676e5c] Claim successful on node kreiken.maas
819 2020-05-26 02:33:18.657 417168 INFO nova.virt.libvirt.driver [req-9b570186-d136-4df5-8b90-6dd3e8b67f3e 91f35a24c0b1465fbd147d70e439646e e16f574f6448430aac827c968e5145be - e61235e9 fe7f4d0284f035c7588dd2b9 e61235e9fe7f4d0284f035c7588dd2b9] [instance: 54e49495-c1c2-4567-8493-2c412d676e5c] Creating image
820 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [req-318355e9-213c-466b-bb9a-f8170ee7ff95 91f35a24c0b1465fbd147d70e439646e e16f574f6448430aac827c968e5145be - e61235e9fe7 f4d0284f035c7588dd2b9 e61235e9fe7f4d0284f035c7588dd2b9] [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Instance failed to spawn: nova.exception.ImageUnacceptable: Image e9149aae -246a-4eb9-963e-d7f25286ba7a is unacceptable: Unable to convert image to raw: Image /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.part is unacceptable: Un able to convert image to raw: Unexpected error while running command.
821 Command: qemu-img convert -t none -O raw -f qcow2 /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.part /var/lib/nova/instances/_base/9b8156fbecaa194804a6372 26c8ffded93a57489.converted
822 Exit code: -15
823 Stdout: ''
824 Stderr: ''
825 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Traceback (most recent call last):
826 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] File "/usr/lib/python3/dist-packages/nova/virt/images.py", line 128, i n _convert_image
827 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] compress)
828 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] File "/usr/lib/python3/dist-packages/nova/privsep/qemu.py", line 73, i n unprivileged_convert_image
829 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] processutils.execute(*cmd)
830 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] File "/usr/lib/python3/dist-packages/oslo_concurrency/processutils.py" , line 424, in execute
831 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] cmd=sanitized_cmd)
832 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error wh ile running command.
833 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Command: qemu-img convert -t none -O raw -f qcow2 /var/lib/nova/instance s/_base/9b8156fbecaa194804a637226c8ffded93a57489.part /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.converted
834 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Exit code: -15
835 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Stdout: ''
836 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Stderr: '' |
bundle: openstack-base-bionic-train https://github.com/openstack-charmers/openstack-bundles/blob/master/development/openstack-base-bionic-train/bundle.yaml
hardware: 2 d05 and 2 d06 (the log of the compute node is from one of the d06)
When trying to create new instances on the deployed openstack, the instance is always in the status of "spawning"
[Steps to Reproduce]
1. Deploy with the above bundle and hardware by following the instruction of https://jaas.ai/openstack-base/bundle/67
2. Wait about 1.5 until the deployment is ready. By ready it means every unit shows its message as "ready" e.g. https://paste.ubuntu.com/p/k48YVnPyVZ/
3. Follow the instruction of https://jaas.ai/openstack-base/bundle/67 until the step of "openstack server create" to create new instance. This step is also summarized in details in this gist code snippet https://gist.github.com/tai271828/b0c00a611e703046dd52da12a66226b0#file-02-basic-test-just-deployed-sh
[Expected Behavior]
An instance is created a few second later
[Actual Behavior]
The status of the instance is always (> 20 minutes) "spawning"
[Additional Information]
1. By waiting for 0.5 ~ 1 hr first, and then restart the nova-compute by "sudo service nova-compute restart", may make the creation behavior back to normal. You may refer to this snippet to see this workaround https://gist.github.com/tai271828/b0c00a611e703046dd52da12a66226b0#file-03-nova-compute-debugging-sh
2. By applying the above workaround and then check log from the corresponding nova-compute node, you could find lines like:
818 2020-05-26 02:33:17.845 417168 INFO nova.compute.claims [req-9b570186-d136-4df5-8b90-6dd3e8b67f3e 91f35a24c0b1465fbd147d70e439646e e16f574f6448430aac827c968e5145be - e61235e9fe7f4 d0284f035c7588dd2b9 e61235e9fe7f4d0284f035c7588dd2b9] [instance: 54e49495-c1c2-4567-8493-2c412d676e5c] Claim successful on node kreiken.maas
819 2020-05-26 02:33:18.657 417168 INFO nova.virt.libvirt.driver [req-9b570186-d136-4df5-8b90-6dd3e8b67f3e 91f35a24c0b1465fbd147d70e439646e e16f574f6448430aac827c968e5145be - e61235e9 fe7f4d0284f035c7588dd2b9 e61235e9fe7f4d0284f035c7588dd2b9] [instance: 54e49495-c1c2-4567-8493-2c412d676e5c] Creating image
820 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [req-318355e9-213c-466b-bb9a-f8170ee7ff95 91f35a24c0b1465fbd147d70e439646e e16f574f6448430aac827c968e5145be - e61235e9fe7 f4d0284f035c7588dd2b9 e61235e9fe7f4d0284f035c7588dd2b9] [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Instance failed to spawn: nova.exception.ImageUnacceptable: Image e9149aae -246a-4eb9-963e-d7f25286ba7a is unacceptable: Unable to convert image to raw: Image /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.part is unacceptable: Un able to convert image to raw: Unexpected error while running command.
821 Command: qemu-img convert -t none -O raw -f qcow2 /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.part /var/lib/nova/instances/_base/9b8156fbecaa194804a6372 26c8ffded93a57489.converted
822 Exit code: -15
823 Stdout: ''
824 Stderr: ''
825 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Traceback (most recent call last):
826 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] File "/usr/lib/python3/dist-packages/nova/virt/images.py", line 128, i n _convert_image
827 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] compress)
828 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] File "/usr/lib/python3/dist-packages/nova/privsep/qemu.py", line 73, i n unprivileged_convert_image
829 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] processutils.execute(*cmd)
830 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] File "/usr/lib/python3/dist-packages/oslo_concurrency/processutils.py" , line 424, in execute
831 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] cmd=sanitized_cmd)
832 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error wh ile running command.
833 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Command: qemu-img convert -t none -O raw -f qcow2 /var/lib/nova/instance s/_base/9b8156fbecaa194804a637226c8ffded93a57489.part /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.converted
834 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Exit code: -15
835 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Stdout: ''
836 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Stderr: ''
3. Before restarting any nova-compute service manually, if you try to issue this command on the compute node, the command will also never end:
qemu-img convert -t none -O raw -f qcow2 /var/lib/nova/instance s/_base/9b8156fbecaa194804a637226c8ffded93a57489.part /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.converted |
|
| 2020-05-27 07:41:34 |
Taihsiang Ho |
summary |
nova-compute Traceback for "qemu-img convert" |
New instance is always in "spawning" status |
|
| 2020-05-27 08:35:13 |
Taihsiang Ho |
bug task added |
|
charm-nova-compute |
|
| 2020-05-27 08:35:45 |
Taihsiang Ho |
bug task added |
|
nova |
|
| 2020-05-27 15:12:32 |
Balazs Gibizer |
nova: status |
New |
Incomplete |
|
| 2020-05-28 17:10:37 |
Andrew Cloke |
bug |
|
|
added subscriber Andrew Cloke |
| 2020-05-29 09:48:49 |
Taihsiang Ho |
attachment added |
|
var-log-nova-kreiken-200529-enable-debug.tar.gz https://bugs.launchpad.net/openstack-bundles/+bug/1880828/+attachment/5378326/+files/var-log-nova-kreiken-200529-enable-debug.tar.gz |
|
| 2020-05-29 09:49:11 |
Taihsiang Ho |
nova: status |
Incomplete |
New |
|
| 2020-05-29 09:56:46 |
Taihsiang Ho |
description |
bundle: openstack-base-bionic-train https://github.com/openstack-charmers/openstack-bundles/blob/master/development/openstack-base-bionic-train/bundle.yaml
hardware: 2 d05 and 2 d06 (the log of the compute node is from one of the d06)
When trying to create new instances on the deployed openstack, the instance is always in the status of "spawning"
[Steps to Reproduce]
1. Deploy with the above bundle and hardware by following the instruction of https://jaas.ai/openstack-base/bundle/67
2. Wait about 1.5 until the deployment is ready. By ready it means every unit shows its message as "ready" e.g. https://paste.ubuntu.com/p/k48YVnPyVZ/
3. Follow the instruction of https://jaas.ai/openstack-base/bundle/67 until the step of "openstack server create" to create new instance. This step is also summarized in details in this gist code snippet https://gist.github.com/tai271828/b0c00a611e703046dd52da12a66226b0#file-02-basic-test-just-deployed-sh
[Expected Behavior]
An instance is created a few second later
[Actual Behavior]
The status of the instance is always (> 20 minutes) "spawning"
[Additional Information]
1. By waiting for 0.5 ~ 1 hr first, and then restart the nova-compute by "sudo service nova-compute restart", may make the creation behavior back to normal. You may refer to this snippet to see this workaround https://gist.github.com/tai271828/b0c00a611e703046dd52da12a66226b0#file-03-nova-compute-debugging-sh
2. By applying the above workaround and then check log from the corresponding nova-compute node, you could find lines like:
818 2020-05-26 02:33:17.845 417168 INFO nova.compute.claims [req-9b570186-d136-4df5-8b90-6dd3e8b67f3e 91f35a24c0b1465fbd147d70e439646e e16f574f6448430aac827c968e5145be - e61235e9fe7f4 d0284f035c7588dd2b9 e61235e9fe7f4d0284f035c7588dd2b9] [instance: 54e49495-c1c2-4567-8493-2c412d676e5c] Claim successful on node kreiken.maas
819 2020-05-26 02:33:18.657 417168 INFO nova.virt.libvirt.driver [req-9b570186-d136-4df5-8b90-6dd3e8b67f3e 91f35a24c0b1465fbd147d70e439646e e16f574f6448430aac827c968e5145be - e61235e9 fe7f4d0284f035c7588dd2b9 e61235e9fe7f4d0284f035c7588dd2b9] [instance: 54e49495-c1c2-4567-8493-2c412d676e5c] Creating image
820 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [req-318355e9-213c-466b-bb9a-f8170ee7ff95 91f35a24c0b1465fbd147d70e439646e e16f574f6448430aac827c968e5145be - e61235e9fe7 f4d0284f035c7588dd2b9 e61235e9fe7f4d0284f035c7588dd2b9] [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Instance failed to spawn: nova.exception.ImageUnacceptable: Image e9149aae -246a-4eb9-963e-d7f25286ba7a is unacceptable: Unable to convert image to raw: Image /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.part is unacceptable: Un able to convert image to raw: Unexpected error while running command.
821 Command: qemu-img convert -t none -O raw -f qcow2 /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.part /var/lib/nova/instances/_base/9b8156fbecaa194804a6372 26c8ffded93a57489.converted
822 Exit code: -15
823 Stdout: ''
824 Stderr: ''
825 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Traceback (most recent call last):
826 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] File "/usr/lib/python3/dist-packages/nova/virt/images.py", line 128, i n _convert_image
827 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] compress)
828 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] File "/usr/lib/python3/dist-packages/nova/privsep/qemu.py", line 73, i n unprivileged_convert_image
829 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] processutils.execute(*cmd)
830 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] File "/usr/lib/python3/dist-packages/oslo_concurrency/processutils.py" , line 424, in execute
831 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] cmd=sanitized_cmd)
832 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error wh ile running command.
833 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Command: qemu-img convert -t none -O raw -f qcow2 /var/lib/nova/instance s/_base/9b8156fbecaa194804a637226c8ffded93a57489.part /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.converted
834 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Exit code: -15
835 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Stdout: ''
836 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Stderr: ''
3. Before restarting any nova-compute service manually, if you try to issue this command on the compute node, the command will also never end:
qemu-img convert -t none -O raw -f qcow2 /var/lib/nova/instance s/_base/9b8156fbecaa194804a637226c8ffded93a57489.part /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.converted |
bundle: openstack-base-bionic-train https://github.com/openstack-charmers/openstack-bundles/blob/master/development/openstack-base-bionic-train/bundle.yaml
hardware: 2 d05 and 2 d06 (the log of the compute node is from one of the d06. Please note they are arm64 arch.)
When trying to create new instances on the deployed openstack, the instance is always in the status of "spawning"
[Steps to Reproduce]
1. Deploy with the above bundle and hardware by following the instruction of https://jaas.ai/openstack-base/bundle/67
2. Wait about 1.5 until the deployment is ready. By ready it means every unit shows its message as "ready" e.g. https://paste.ubuntu.com/p/k48YVnPyVZ/
3. Follow the instruction of https://jaas.ai/openstack-base/bundle/67 until the step of "openstack server create" to create new instance. This step is also summarized in details in this gist code snippet https://gist.github.com/tai271828/b0c00a611e703046dd52da12a66226b0#file-02-basic-test-just-deployed-sh
[Expected Behavior]
An instance is created a few second later
[Actual Behavior]
The status of the instance is always (> 20 minutes) "spawning"
[Additional Information]
1. By waiting for 0.5 ~ 1 hr first, and then restart the nova-compute by "sudo service nova-compute restart", may make the creation behavior back to normal. You may refer to this snippet to see this workaround https://gist.github.com/tai271828/b0c00a611e703046dd52da12a66226b0#file-03-nova-compute-debugging-sh
2. By applying the above workaround and then check log from the corresponding nova-compute node, you could find lines like:
818 2020-05-26 02:33:17.845 417168 INFO nova.compute.claims [req-9b570186-d136-4df5-8b90-6dd3e8b67f3e 91f35a24c0b1465fbd147d70e439646e e16f574f6448430aac827c968e5145be - e61235e9fe7f4 d0284f035c7588dd2b9 e61235e9fe7f4d0284f035c7588dd2b9] [instance: 54e49495-c1c2-4567-8493-2c412d676e5c] Claim successful on node kreiken.maas
819 2020-05-26 02:33:18.657 417168 INFO nova.virt.libvirt.driver [req-9b570186-d136-4df5-8b90-6dd3e8b67f3e 91f35a24c0b1465fbd147d70e439646e e16f574f6448430aac827c968e5145be - e61235e9 fe7f4d0284f035c7588dd2b9 e61235e9fe7f4d0284f035c7588dd2b9] [instance: 54e49495-c1c2-4567-8493-2c412d676e5c] Creating image
820 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [req-318355e9-213c-466b-bb9a-f8170ee7ff95 91f35a24c0b1465fbd147d70e439646e e16f574f6448430aac827c968e5145be - e61235e9fe7 f4d0284f035c7588dd2b9 e61235e9fe7f4d0284f035c7588dd2b9] [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Instance failed to spawn: nova.exception.ImageUnacceptable: Image e9149aae -246a-4eb9-963e-d7f25286ba7a is unacceptable: Unable to convert image to raw: Image /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.part is unacceptable: Un able to convert image to raw: Unexpected error while running command.
821 Command: qemu-img convert -t none -O raw -f qcow2 /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.part /var/lib/nova/instances/_base/9b8156fbecaa194804a6372 26c8ffded93a57489.converted
822 Exit code: -15
823 Stdout: ''
824 Stderr: ''
825 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Traceback (most recent call last):
826 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] File "/usr/lib/python3/dist-packages/nova/virt/images.py", line 128, i n _convert_image
827 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] compress)
828 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] File "/usr/lib/python3/dist-packages/nova/privsep/qemu.py", line 73, i n unprivileged_convert_image
829 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] processutils.execute(*cmd)
830 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] File "/usr/lib/python3/dist-packages/oslo_concurrency/processutils.py" , line 424, in execute
831 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] cmd=sanitized_cmd)
832 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error wh ile running command.
833 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Command: qemu-img convert -t none -O raw -f qcow2 /var/lib/nova/instance s/_base/9b8156fbecaa194804a637226c8ffded93a57489.part /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.converted
834 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Exit code: -15
835 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Stdout: ''
836 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Stderr: ''
3. Before restarting any nova-compute service manually, if you try to issue this command on the compute node, the command will also never end:
qemu-img convert -t none -O raw -f qcow2 /var/lib/nova/instance s/_base/9b8156fbecaa194804a637226c8ffded93a57489.part /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.converted |
|
| 2020-05-29 10:35:17 |
Balazs Gibizer |
nova: status |
New |
Incomplete |
|
| 2020-05-29 13:29:37 |
Taihsiang Ho |
attachment added |
|
var-log-nova-kreiken-200529-enable-debug.2.tar.gz https://bugs.launchpad.net/openstack-bundles/+bug/1880828/+attachment/5378402/+files/var-log-nova-kreiken-200529-enable-debug.2.tar.gz |
|
| 2020-06-03 02:34:50 |
Taihsiang Ho |
attachment removed |
var-log-nova-kreiken-200529-enable-debug.tar.gz https://bugs.launchpad.net/openstack-bundles/+bug/1880828/+attachment/5378326/+files/var-log-nova-kreiken-200529-enable-debug.tar.gz |
|
|
| 2020-06-03 02:43:24 |
Taihsiang Ho |
nova: status |
Incomplete |
New |
|
| 2020-07-13 07:12:49 |
Taihsiang Ho |
description |
bundle: openstack-base-bionic-train https://github.com/openstack-charmers/openstack-bundles/blob/master/development/openstack-base-bionic-train/bundle.yaml
hardware: 2 d05 and 2 d06 (the log of the compute node is from one of the d06. Please note they are arm64 arch.)
When trying to create new instances on the deployed openstack, the instance is always in the status of "spawning"
[Steps to Reproduce]
1. Deploy with the above bundle and hardware by following the instruction of https://jaas.ai/openstack-base/bundle/67
2. Wait about 1.5 until the deployment is ready. By ready it means every unit shows its message as "ready" e.g. https://paste.ubuntu.com/p/k48YVnPyVZ/
3. Follow the instruction of https://jaas.ai/openstack-base/bundle/67 until the step of "openstack server create" to create new instance. This step is also summarized in details in this gist code snippet https://gist.github.com/tai271828/b0c00a611e703046dd52da12a66226b0#file-02-basic-test-just-deployed-sh
[Expected Behavior]
An instance is created a few second later
[Actual Behavior]
The status of the instance is always (> 20 minutes) "spawning"
[Additional Information]
1. By waiting for 0.5 ~ 1 hr first, and then restart the nova-compute by "sudo service nova-compute restart", may make the creation behavior back to normal. You may refer to this snippet to see this workaround https://gist.github.com/tai271828/b0c00a611e703046dd52da12a66226b0#file-03-nova-compute-debugging-sh
2. By applying the above workaround and then check log from the corresponding nova-compute node, you could find lines like:
818 2020-05-26 02:33:17.845 417168 INFO nova.compute.claims [req-9b570186-d136-4df5-8b90-6dd3e8b67f3e 91f35a24c0b1465fbd147d70e439646e e16f574f6448430aac827c968e5145be - e61235e9fe7f4 d0284f035c7588dd2b9 e61235e9fe7f4d0284f035c7588dd2b9] [instance: 54e49495-c1c2-4567-8493-2c412d676e5c] Claim successful on node kreiken.maas
819 2020-05-26 02:33:18.657 417168 INFO nova.virt.libvirt.driver [req-9b570186-d136-4df5-8b90-6dd3e8b67f3e 91f35a24c0b1465fbd147d70e439646e e16f574f6448430aac827c968e5145be - e61235e9 fe7f4d0284f035c7588dd2b9 e61235e9fe7f4d0284f035c7588dd2b9] [instance: 54e49495-c1c2-4567-8493-2c412d676e5c] Creating image
820 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [req-318355e9-213c-466b-bb9a-f8170ee7ff95 91f35a24c0b1465fbd147d70e439646e e16f574f6448430aac827c968e5145be - e61235e9fe7 f4d0284f035c7588dd2b9 e61235e9fe7f4d0284f035c7588dd2b9] [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Instance failed to spawn: nova.exception.ImageUnacceptable: Image e9149aae -246a-4eb9-963e-d7f25286ba7a is unacceptable: Unable to convert image to raw: Image /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.part is unacceptable: Un able to convert image to raw: Unexpected error while running command.
821 Command: qemu-img convert -t none -O raw -f qcow2 /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.part /var/lib/nova/instances/_base/9b8156fbecaa194804a6372 26c8ffded93a57489.converted
822 Exit code: -15
823 Stdout: ''
824 Stderr: ''
825 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Traceback (most recent call last):
826 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] File "/usr/lib/python3/dist-packages/nova/virt/images.py", line 128, i n _convert_image
827 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] compress)
828 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] File "/usr/lib/python3/dist-packages/nova/privsep/qemu.py", line 73, i n unprivileged_convert_image
829 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] processutils.execute(*cmd)
830 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] File "/usr/lib/python3/dist-packages/oslo_concurrency/processutils.py" , line 424, in execute
831 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] cmd=sanitized_cmd)
832 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error wh ile running command.
833 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Command: qemu-img convert -t none -O raw -f qcow2 /var/lib/nova/instance s/_base/9b8156fbecaa194804a637226c8ffded93a57489.part /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.converted
834 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Exit code: -15
835 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Stdout: ''
836 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Stderr: ''
3. Before restarting any nova-compute service manually, if you try to issue this command on the compute node, the command will also never end:
qemu-img convert -t none -O raw -f qcow2 /var/lib/nova/instance s/_base/9b8156fbecaa194804a637226c8ffded93a57489.part /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.converted |
bundle: openstack-base-bionic-train https://github.com/openstack-charmers/openstack-bundles/blob/master/development/openstack-base-bionic-train/bundle.yaml
hardware: 2 d05 and 2 d06 (the log of the compute node is from one of the d06. Please note they are arm64 arch.)
When trying to create new instances on the deployed openstack, the instance is always in the status of "spawning"
[Steps to Reproduce]
1. Deploy with the above bundle and hardware by following the instruction of https://jaas.ai/openstack-base/bundle/67
2. Wait about 1.5 until the deployment is ready. By ready it means every unit shows its message as "ready" e.g. https://paste.ubuntu.com/p/k48YVnPyVZ/
3. Follow the instruction of https://jaas.ai/openstack-base/bundle/67 until the step of "openstack server create" to create new instance. This step is also summarized in details in this gist code snippet https://gist.github.com/tai271828/b0c00a611e703046dd52da12a66226b0#file-02-basic-test-just-deployed-sh
[Expected Behavior]
An instance is created a few seconds later
[Actual Behavior]
The status of the instance is always (> 20 minutes) "spawning"
[Additional Information]
1. [workaround] Use `ps aux | grep qemu-img` to check if a qemu-img image converting process exists or not. The process should complete within ~20 sec. If the process exists for more than 1 minutes, use `pkill -f qemu-img` to terminate the process and re-create instances again.
The image converting process looks like this one:
```
qemu-img convert -t none -O raw -f qcow2 /var/lib/nova/instance s/_base/9b8156fbecaa194804a637226c8ffded93a57489.part /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.converted
```
2. By investing in more details, this issue is a coupled issue of 1) nova should timeout instance process (comment#21) 2) qemu does not terminate the process to convert the image successfully (comment#20) |
|
| 2020-07-13 07:12:58 |
Taihsiang Ho |
tags |
|
tai-radar |
|
| 2020-09-16 12:47:04 |
Balazs Gibizer |
nova: status |
New |
Triaged |
|
| 2020-09-16 12:47:08 |
Balazs Gibizer |
nova: importance |
Undecided |
Wishlist |
|
| 2022-05-13 22:45:12 |
Billy Olsen |
charm-nova-compute: status |
New |
Invalid |
|
| 2022-05-13 22:45:22 |
Billy Olsen |
openstack-bundles: status |
New |
Invalid |
|
