Comment 2 for bug 1700843

Revision history for this message
Danil Akhmetov (dinobot) wrote :

Denis, sure.

I faced this issue on one of controllers nodes in MOS 7 (Kilo) environment with Cinder providing 2 backends: netapp and SolidFire. This issue has been successfully resolved.

According to cinder-backup debug log, the problem looked like unexpectedly interrupted HTTP session while cinder-backup getting next chunk's data from swift-object-server via swift-proxy.

I found bunch of issues on swift's side, but I was surprised that I run into original problem again after I resolved problems of swift's side.

Then I found out that the problem is connected only to one of the Cinder backends, and after we cleaned up stuck SolidFire mounts the problem gone.

Finally, it seems to me like cinder-backup multithreading consistency and logging issue: when iSCSI operations failed during the volume restore, process ended up in urllib3 library exception due to terminated http which is very unclear and may send engineer who troubleshooting it on a wrong path.