[Stable/newton] socket closed due to time out error

Bug #1645172 reported by TommyLike
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
oslo.privsep
Fix Released
Undecided
TommyLike

Bug Description

This bug is already reported [1] and fixed in oslo.privsep 1.14.0, however we published 1.13.0 in newton and the socket keeps breaking down with official Newton release, we need backport this bug into stable/newton release and bump release version (please check the original bug for more logs):

```
2016-11-17 17:48:10.457 9744 WARNING oslo.service.loopingcall [req-ab26d5c1-6028-4d62-a910-534b19c97a83 cee7ffcdb7744ace82be607171c7f5e7 15711de32e454b1b8924ddd9aa0d56af - default default]
Function 'cinder.service.Service.report_state' run outlasted interval by 130.60 sec
2016-11-17 17:48:10.461 9744 DEBUG cinder.backup.chunkeddriver [req-ab26d5c1-6028-4d62-a910-534b19c97a83 cee7ffcdb7744ace82be607171c7f5e7 15711de32e454b1b8924ddd9aa0d56af - default default]
backup 449236af-56f2-4c9d-abc2-9a9cf29dd7bd finished. _finalize_backup /usr/lib/python2.7/dist-packages/cinder/backup/chunkeddriver.py:386
2016-11-17 17:48:10.485 9744 DEBUG oslo_concurrency.processutils [req-ab26d5c1-6028-4d62-a910-534b19c97a83 cee7ffcdb7744ace82be607171c7f5e7 15711de32e454b1b8924ddd9aa0d56af - default default]
Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/disk/by-path/ip-111.111.101.38:3260-iscsi-iqn.2006-08.com.huawei:oceanstor:21003400a30d844d::20000:111.111.101.38-lun-3 execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:344
2016-11-17 17:48:11.066 9744 INFO oslo.messaging._drivers.impl_rabbit [req-ab26d5c1-6028-4d62-a910-534b19c97a83 cee7ffcdb7744ace82be607171c7f5e7 15711de32e454b1b8924ddd9aa0d56af - default default]
[d2925c37-5172-4fcf-9279-8ecc0ba1bc3b] Reconnected to AMQP server on controller:5672 via [amqp] clientwith port 43606.
2016-11-17 17:48:13.800 9744 DEBUG oslo_concurrency.processutils [req-ab26d5c1-6028-4d62-a910-534b19c97a83 cee7ffcdb7744ace82be607171c7f5e7 15711de32e454b1b8924ddd9aa0d56af - default default]
CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/disk/by-path/ip-111.111.101.38:3260-iscsi-iqn.2006-08.com.huawei:oceanstor:21003400a30d844d::20000:111.111.101.38-lun-3" returned: 0 in 3.315s execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:374
2016-11-17 17:48:13.802 9744 DEBUG os_brick.initiator.connectors.iscsi [req-ab26d5c1-6028-4d62-a910-534b19c97a83 cee7ffcdb7744ace82be607171c7f5e7 15711de32e454b1b8924ddd9aa0d56af - default default]
==> disconnect_volume: call {'args': (<os_brick.initiator.connectors.iscsi.ISCSIConnector object at 0x7f7378df33d0>, {'target_discovered': False, 'target_iqn': u'iqn.2006-08.com.huawei:oceanstor:21003400a30d844d::20000:111.111.101.38', 'target_portal': u'111.111.101.38:3260', 'volume_id': '1c6a86bd-8728-4d8c-896d-8867f6b35973', 'target_lun': 3, 'map_info': {'lun_id': u'63', 'view_id': u'9', 'aval_luns': u'[0,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287,288,289,290,291,292,293,294,295,296,297,298,299,300,301,302,303,304,305,306,307,308,309,310,311,312,313,314,315,316,317,318,319,320,321,322,323,324,325,326,327,328,329,330,331,332,333,334,335,336,337,338,339,340,341,342,343,344,345,346,347,348,349,350,351,352,353,354,355,356,357,358,359,360,361,362,363,364,365,366,367,368,369,370,371,372,373,374,375,376,377,378,379,380,381,382,383,384,385,386,387,388,389,390,391,392,393,394,395,396,397,398,399,400,401,402,403,404,405,406,407,408,409,410,411,412,413,414,415,416,417,418,419,420,421,422,423,424,425,426,427,428,429,430,431,432,433,434,435,436,437,438,439,440,441,442,443,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,461,462,463,464,465,466,467,468,469,470,471,472,473,474,475,476,477,478,479,480,481,482,483,484,485,486,487,488,489,490,491,492,493,494,495,496,497,498,499,500,501,502,503,504,505,506,507,508,509,510,511]'}}, {'path': u'/dev/disk/by-path/ip-111.111.101.38:3260-iscsi-iqn.2006-08.com.huawei:oceanstor:21003400a30d844d::20000:111.111.101.38-lun-3', 'scsi_wwn': '363400a31000d844d021b7c720000003f', 'type': 'block'}), 'kwargs': {}} trace_logging_wrapper /usr/lib/python2.7/dist-packages/os_brick/utils.py:141
2016-11-17 17:48:13.806 9744 DEBUG oslo_concurrency.lockutils [req-ab26d5c1-6028-4d62-a910-534b19c97a83 cee7ffcdb7744ace82be607171c7f5e7 15711de32e454b1b8924ddd9aa0d56af - default default]
Lock "connect_volume" acquired by "os_brick.initiator.connectors.iscsi.disconnect_volume" :: waited 0.000s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:273
2016-11-17 17:48:13.808 9744 DEBUG os_brick.initiator.linuxscsi [req-ab26d5c1-6028-4d62-a910-534b19c97a83 cee7ffcdb7744ace82be607171c7f5e7 15711de32e454b1b8924ddd9aa0d56af - default default]
Flushing IO for device /dev/sdd flush_device_io /usr/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py:148
2016-11-17 17:48:55.202 9744 INFO oslo.messaging._drivers.impl_rabbit [req-ab26d5c1-6028-4d62-a910-534b19c97a83 cee7ffcdb7744ace82be607171c7f5e7 15711de32e454b1b8924ddd9aa0d56af - default default]
A recoverable connection/channel error occurred, trying to reconnect: Socket closed

[1] https://bugs.launchpad.net/oslo.privsep/+bug/1623286
```

TommyLike (hu-husheng)
Changed in oslo.privsep:
assignee: nobody → TommyLike (hu-husheng)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/oslo.privsep 1.13.1

This issue was fixed in the openstack/oslo.privsep 1.13.1 release.

Ben Nemec (bnemec)
Changed in oslo.privsep:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.