Tapan, R2.20 Build 10 already had fix for 1430091. Does it mean 1430091 is not resolved yet or 1453369 is a new issue.
I see similar crash on 2.20 build 30 as well.
Could you pls check ?
Vedu
On 5/11/15, 9:09 PM, "Tapan Karwa" <email address hidden> wrote:
>*** This bug is a duplicate of bug 1430091 *** > https://bugs.launchpad.net/bugs/1430091 > >Dup of 1430091 > >** This bug has been marked a duplicate of bug 1430091 > control-node assertion in IFMapExporter::StateUpdateOnDequeue on >deleting logical interfaces > >-- >You received this bug notification because you are a member of Contrail >Systems engineering, which is subscribed to Juniper Openstack. >Matching subscriptions: Juniper Openstack subscription >https://bugs.launchpad.net/bugs/1453369 > >Title: > [Build R2.20 10 Juno] TOR Scale:Control node crash @ > IFMapExporter::StateUpdateOnDequeue > >Status in Juniper Openstack distribution: > New >Status in Juniper Openstack r2.20 series: > New >Status in Juniper Openstack trunk series: > New > >Bug description: > > Trigger > ------------- > System had around 16K Vmi. Deleting all of them in one go with UI. > > Backtrace > ---------------- > (gdb) bt > #0 0x00007f547dd66cc9 in __GI_raise (sig=sig@entry=6) at >../nptl/sysdeps/unix/sysv/linux/raise.c:56 > #1 0x00007f547dd6a0d8 in __GI_abort () at abort.c:89 > #2 0x00007f547dd5fb86 in __assert_fail_base (fmt=0x7f547deb0830 >"%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", >assertion=assertion@entry=0xadebe1 "state->advertised().empty()", > file=file@entry=0xadec78 "controller/src/ifmap/ifmap_exporter.cc", >line=line@entry=548, > function=function@entry=0xadf000 ><IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, BitSet const&, >bool)::__PRETTY_FUNCTION__> "void >IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, const BitSet&, bool)") >at assert.c:92 > #3 0x00007f547dd5fc32 in __GI___assert_fail (assertion=0xadebe1 >"state->advertised().empty()", file=0xadec78 >"controller/src/ifmap/ifmap_exporter.cc", line=548, > function=0xadf000 ><IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, BitSet const&, >bool)::__PRETTY_FUNCTION__> "void >IFMapExporter::StateUpdateOnDequeue(IFMapUpdate*, const BitSet&, bool)") >at assert.c:101 > #4 0x000000000045df76 in IFMapExporter::StateUpdateOnDequeue >(this=0x2d92150, update=update@entry=0x7f545688de70, dequeue_set=..., >is_delete=<optimized out>) > at controller/src/ifmap/ifmap_exporter.cc:548 > #5 0x000000000048add2 in IFMapUpdateSender::ProcessUpdate >(this=this@entry=0x2d93600, update=update@entry=0x7f545688de70, >base_send_set=...) > at controller/src/ifmap/ifmap_update_sender.cc:225 > #6 0x000000000048b314 in IFMapUpdateSender::Send (this=0x2d93600, >imarker=<optimized out>) at >controller/src/ifmap/ifmap_update_sender.cc:184 > #7 0x000000000048badb in IFMapUpdateSender::SendTask::Run >(this=0x2dcf520) at controller/src/ifmap/ifmap_update_sender.cc:41 > #8 0x0000000000ab2ad0 in TaskImpl::execute (this=0x7f54774c2040) at >controller/src/base/task.cc:232 > #9 0x00007f547eb3db3a in ?? () from /usr/lib/libtbb.so.2 > #10 0x00007f547eb39816 in ?? () from /usr/lib/libtbb.so.2 > #11 0x00007f547eb38f4b in ?? () from /usr/lib/libtbb.so.2 > #12 0x00007f547eb350ff in ?? () from /usr/lib/libtbb.so.2 > #13 0x00007f547eb352f9 in ?? () from /usr/lib/libtbb.so.2 > #14 0x00007f547ed59182 in start_thread (arg=0x7f544dbf6700) at >pthread_create.c:312 > #15 0x00007f547de2a47d in clone () at >../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > (gdb) quit > > > Contrail-control log during crash > ------------------------------------------------- > 2015-05-09 Sat 14:54:14:617.374 IST nodei6 [Thread 140000083560192, >Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer >default-domain:default-project:ip-fabric:__default__:nodei6:default-domain >:default-project:ip-fabric:__default__:walsh SEND Notification with Code >2 and SubCode 7 ( OPEN Message Error:Unsupported Capability ) >controller/src/bgp/bgp_session.cc 96 > 2015-05-09 Sat 14:54:35:205.862 IST nodei6 [Thread 140000045774592, >Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: Event: Tcp >Connection Closed peer ip: 192.168.22.4 ( nodei9-1 ) >controller/src/xmpp/xmpp_state_machine.cc 1322 > 2015-05-09 Sat 14:54:42:809.939 IST nodei6 [Thread 140000710457088, >Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: PassiveOpen >in state: Idle peer ip: 192.168.22.4 ( ) >controller/src/xmpp/xmpp_state_machine.cc 1335 > 2015-05-09 Sat 14:54:50:564.235 IST nodei6 [Thread 140000718853888, >Pid 18669]: BGP [SYS_WARN]: BgpPeerMessageLog: BGP Peer >default-domain:default-project:ip-fabric:__default__:nodei6:default-domain >:default-project:ip-fabric:__default__:walsh RECV Unsupported Capability: >MpExtension (1) controller/src/bgp/bgp_proto.cc 112 > 2015-05-09 Sat 14:54:50:564.660 IST nodei6 [Thread 140000718853888, >Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer >default-domain:default-project:ip-fabric:__default__:nodei6:default-domain >:default-project:ip-fabric:__default__:walsh RECV Notification with Code >6 and SubCode 5 ( Cease:Connection is rejected by the peer ) >controller/src/bgp/state_machine.cc 307 > 2015-05-09 Sat 14:54:50:564.968 IST nodei6 [Thread 140000718853888, >Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer >default-domain:default-project:ip-fabric:__default__:nodei6:default-domain >:default-project:ip-fabric:__default__:walsh SEND Notification with Code >2 and SubCode 7 ( OPEN Message Error:Unsupported Capability ) >controller/src/bgp/bgp_session.cc 96 > 2015-05-09 Sat 14:55:29:444.147 IST nodei6 [Thread 140000096155392, >Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: Event: Tcp >Connection Closed peer ip: 192.168.22.5 ( nodei10 ) >controller/src/xmpp/xmpp_state_machine.cc 1322 > 2015-05-09 Sat 14:55:31:681.300 IST nodei6 [Thread 140000731449088, >Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer >default-domain:default-project:ip-fabric:__default__:nodei6:default-domain >:default-project:ip-fabric:__default__:nodei8 RECV Notification with Code >6 and SubCode 3 ( Cease:Administrator has unconfigured the peer ) >controller/src/bgp/state_machine.cc 307 > 2015-05-09 Sat 14:55:39:548.516 IST nodei6 [Thread 140000752441088, >Pid 18669]: BGP [SYS_WARN]: BgpPeerMessageLog: BGP Peer >default-domain:default-project:ip-fabric:__default__:nodei6:default-domain >:default-project:ip-fabric:__default__:walsh RECV Unsupported Capability: >MpExtension (1) controller/src/bgp/bgp_proto.cc 112 > 2015-05-09 Sat 14:55:39:548.843 IST nodei6 [Thread 140000752441088, >Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer >default-domain:default-project:ip-fabric:__default__:nodei6:default-domain >:default-project:ip-fabric:__default__:walsh RECV Notification with Code >6 and SubCode 5 ( Cease:Connection is rejected by the peer ) >controller/src/bgp/state_machine.cc 307 > 2015-05-09 Sat 14:55:39:549.138 IST nodei6 [Thread 140000752441088, >Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotificationLog: Bgp Peer >default-domain:default-project:ip-fabric:__default__:nodei6:default-domain >:default-project:ip-fabric:__default__:walsh SEND Notification with Code >2 and SubCode 7 ( OPEN Message Error:Unsupported Capability ) >controller/src/bgp/bgp_session.cc 96 > 2015-05-09 Sat 14:55:39:597.924 IST nodei6 [Thread 140000714655488, >Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: PassiveOpen >in state: Idle peer ip: 192.168.22.5 ( ) >controller/src/xmpp/xmpp_state_machine.cc 1335 > 2015-05-09 Sat 14:55:40:694.229 IST nodei6 [Thread 140000054171392, >Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: Event: Tcp >Connection Closed peer ip: 192.168.22.4 ( nodei9 ) >controller/src/xmpp/xmpp_state_machine.cc 1322 > 2015-05-09 Sat 14:55:46:241.071 IST nodei6 [Thread 140000037377792, >Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: Event: Tcp >Connection Closed peer ip: 192.168.22.4 ( nodei9-1 ) >controller/src/xmpp/xmpp_state_machine.cc 1322 > 2015-05-09 Sat 14:56:05:916.364 IST nodei6 [Thread 140012986161088, >Pid 31844]: SANDESH: Logging: DISABLED -> ENABLED > 2015-05-09 Sat 14:56:05:916.563 IST nodei6 [Thread 140012986161088, >Pid 31844]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_NOTICE ] >log4level: [ TRACE ] -> [ WARN ] > 2015-05-09 Sat 14:56:05:917.715 IST nodei6 [Thread 140012986161088, >Pid 31844]: Starting Bgp Server at port 179 > 2015-05-09 Sat 14:56:05:930.579 IST nodei6 [Thread 140012986161088, >Pid 31844]: SANDESH: No Client: 1431163565930447 >SandeshModuleClientTrace: data= [ name = >nodei6:Control:contrail-control:0 client_info= [ status = Idle >successful_connections = 0 pid = 31844 http_port = 8083 start_time = >1431163565930223 collector_name = primary = 0.0.0.0:0 secondary = >0.0.0.0:0 rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 >blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = >errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 >blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = >errors = 0 ] ] ] > > > > > Testbed > -------------- > > env.roledefs = { > 'all': [host1, host2, host3, host4, host5, host6], > 'cfgm': [host1, host2, host3], > 'openstack': [host1, host2, host3], > 'webui': [host2], > 'control': [host1, host3], > 'compute': [host4, host5, host6], > 'tsn': [host4, host5], > 'toragent': [host4, host5], > 'collector': [host1, host3], > 'database': [host1, host2, host3], > 'build': [host_build], > } > > env.hostnames = { > 'all': ['nodei6', 'nodei7', 'nodei8', 'nodei9', 'nodei10', >'nodei19'] > } > >To manage notifications about this bug go to: >https://bugs.launchpad.net/juniperopenstack/+bug/1453369/+subscriptions
Tapan,
R2.20 Build 10 already had fix for 1430091.
Does it mean 1430091 is not resolved yet or 1453369 is a new issue.
I see similar crash on 2.20 build 30 as well.
Could you pls check ?
Vedu
On 5/11/15, 9:09 PM, "Tapan Karwa" <email address hidden> wrote:
>*** This bug is a duplicate of bug 1430091 *** /bugs.launchpad .net/bugs/ 1430091 :StateUpdateOnD equeue on /bugs.launchpad .net/bugs/ 1453369 :StateUpdateOnD equeue sysdeps/ unix/sysv/ linux/raise. c:56 assertion@ entry=0xadebe1 "state- >advertised( ).empty( )", entry=0xadec78 "controller/ src/ifmap/ ifmap_exporter. cc", entry=548, function@ entry=0xadf000 ::StateUpdateOn Dequeue( IFMapUpdate* , BitSet const&, :__PRETTY_ FUNCTION_ _> "void :StateUpdateOnD equeue( IFMapUpdate* , const BitSet&, bool)") >advertised( ).empty( )", file=0xadec78 src/ifmap/ ifmap_exporter. cc", line=548, ::StateUpdateOn Dequeue( IFMapUpdate* , BitSet const&, :__PRETTY_ FUNCTION_ _> "void :StateUpdateOnD equeue( IFMapUpdate* , const BitSet&, bool)") :StateUpdateOnD equeue update@ entry=0x7f54568 8de70, dequeue_set=..., <optimized out>) src/ifmap/ ifmap_exporter. cc:548 er::ProcessUpda te this@entry= 0x2d93600, update= update@ entry=0x7f54568 8de70, src/ifmap/ ifmap_update_ sender. cc:225 er::Send (this=0x2d93600, src/ifmap/ ifmap_update_ sender. cc:184 er::SendTask: :Run src/ifmap/ ifmap_update_ sender. cc:41 c2040) at src/base/ task.cc: 232 libtbb. so.2 libtbb. so.2 libtbb. so.2 libtbb. so.2 libtbb. so.2 6700) at create. c:312 unix/sysv/ linux/x86_ 64/clone. S:111 ------- ------- ------- ------- ------- ------- tionLog: Bgp Peer domain: default- project: ip-fabric: __default_ _:nodei6: default- domain project: ip-fabric: __default_ _:walsh SEND Notification with Code src/bgp/ bgp_session. cc 96 src/xmpp/ xmpp_state_ machine. cc 1322 src/xmpp/ xmpp_state_ machine. cc 1335 domain: default- project: ip-fabric: __default_ _:nodei6: default- domain project: ip-fabric: __default_ _:walsh RECV Unsupported Capability: src/bgp/ bgp_proto. cc 112 tionLog: Bgp Peer domain: default- project: ip-fabric: __default_ _:nodei6: default- domain project: ip-fabric: __default_ _:walsh RECV Notification with Code src/bgp/ state_machine. cc 307 tionLog: Bgp Peer domain: default- project: ip-fabric: __default_ _:nodei6: default- domain project: ip-fabric: __default_ _:walsh SEND Notification with Code src/bgp/ bgp_session. cc 96 src/xmpp/ xmpp_state_ machine. cc 1322 tionLog: Bgp Peer domain: default- project: ip-fabric: __default_ _:nodei6: default- domain project: ip-fabric: __default_ _:nodei8 RECV Notification with Code src/bgp/ state_machine. cc 307 domain: default- project: ip-fabric: __default_ _:nodei6: default- domain project: ip-fabric: __default_ _:walsh RECV Unsupported Capability: src/bgp/ bgp_proto. cc 112 tionLog: Bgp Peer domain: default- project: ip-fabric: __default_ _:nodei6: default- domain project: ip-fabric: __default_ _:walsh RECV Notification with Code src/bgp/ state_machine. cc 307 tionLog: Bgp Peer domain: default- project: ip-fabric: __default_ _:nodei6: default- domain project: ip-fabric: __default_ _:walsh SEND Notification with Code src/bgp/ bgp_session. cc 96 src/xmpp/ xmpp_state_ machine. cc 1335 src/xmpp/ xmpp_state_ machine. cc 1322 src/xmpp/ xmpp_state_ machine. cc 1322 lientTrace: data= [ name = Control: contrail- control: 0 client_info= [ status = Idle connections = 0 pid = 31844 http_port = 8083 start_time = blocked_ duration = blocked_ duration = /bugs.launchpad .net/juniperope nstack/ +bug/1453369/ +subscriptions
> https:/
>
>Dup of 1430091
>
>** This bug has been marked a duplicate of bug 1430091
> control-node assertion in IFMapExporter:
>deleting logical interfaces
>
>--
>You received this bug notification because you are a member of Contrail
>Systems engineering, which is subscribed to Juniper Openstack.
>Matching subscriptions: Juniper Openstack subscription
>https:/
>
>Title:
> [Build R2.20 10 Juno] TOR Scale:Control node crash @
> IFMapExporter:
>
>Status in Juniper Openstack distribution:
> New
>Status in Juniper Openstack r2.20 series:
> New
>Status in Juniper Openstack trunk series:
> New
>
>Bug description:
>
> Trigger
> -------------
> System had around 16K Vmi. Deleting all of them in one go with UI.
>
> Backtrace
> ----------------
> (gdb) bt
> #0 0x00007f547dd66cc9 in __GI_raise (sig=sig@entry=6) at
>../nptl/
> #1 0x00007f547dd6a0d8 in __GI_abort () at abort.c:89
> #2 0x00007f547dd5fb86 in __assert_fail_base (fmt=0x7f547deb0830
>"%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
>assertion=
> file=file@
>line=line@
> function=
><IFMapExporter
>bool):
>IFMapExporter:
>at assert.c:92
> #3 0x00007f547dd5fc32 in __GI___assert_fail (assertion=0xadebe1
>"state-
>"controller/
> function=0xadf000
><IFMapExporter
>bool):
>IFMapExporter:
>at assert.c:101
> #4 0x000000000045df76 in IFMapExporter:
>(this=0x2d92150, update=
>is_delete=
> at controller/
> #5 0x000000000048add2 in IFMapUpdateSend
>(this=
>base_send_set=...)
> at controller/
> #6 0x000000000048b314 in IFMapUpdateSend
>imarker=<optimized out>) at
>controller/
> #7 0x000000000048badb in IFMapUpdateSend
>(this=0x2dcf520) at controller/
> #8 0x0000000000ab2ad0 in TaskImpl::execute (this=0x7f54774
>controller/
> #9 0x00007f547eb3db3a in ?? () from /usr/lib/
> #10 0x00007f547eb39816 in ?? () from /usr/lib/
> #11 0x00007f547eb38f4b in ?? () from /usr/lib/
> #12 0x00007f547eb350ff in ?? () from /usr/lib/
> #13 0x00007f547eb352f9 in ?? () from /usr/lib/
> #14 0x00007f547ed59182 in start_thread (arg=0x7f544dbf
>pthread_
> #15 0x00007f547de2a47d in clone () at
>../sysdeps/
> (gdb) quit
>
>
> Contrail-control log during crash
> -------
> 2015-05-09 Sat 14:54:14:617.374 IST nodei6 [Thread 140000083560192,
>Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotifica
>default-
>:default-
>2 and SubCode 7 ( OPEN Message Error:Unsupported Capability )
>controller/
> 2015-05-09 Sat 14:54:35:205.862 IST nodei6 [Thread 140000045774592,
>Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: Event: Tcp
>Connection Closed peer ip: 192.168.22.4 ( nodei9-1 )
>controller/
> 2015-05-09 Sat 14:54:42:809.939 IST nodei6 [Thread 140000710457088,
>Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: PassiveOpen
>in state: Idle peer ip: 192.168.22.4 ( )
>controller/
> 2015-05-09 Sat 14:54:50:564.235 IST nodei6 [Thread 140000718853888,
>Pid 18669]: BGP [SYS_WARN]: BgpPeerMessageLog: BGP Peer
>default-
>:default-
>MpExtension (1) controller/
> 2015-05-09 Sat 14:54:50:564.660 IST nodei6 [Thread 140000718853888,
>Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotifica
>default-
>:default-
>6 and SubCode 5 ( Cease:Connection is rejected by the peer )
>controller/
> 2015-05-09 Sat 14:54:50:564.968 IST nodei6 [Thread 140000718853888,
>Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotifica
>default-
>:default-
>2 and SubCode 7 ( OPEN Message Error:Unsupported Capability )
>controller/
> 2015-05-09 Sat 14:55:29:444.147 IST nodei6 [Thread 140000096155392,
>Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: Event: Tcp
>Connection Closed peer ip: 192.168.22.5 ( nodei10 )
>controller/
> 2015-05-09 Sat 14:55:31:681.300 IST nodei6 [Thread 140000731449088,
>Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotifica
>default-
>:default-
>6 and SubCode 3 ( Cease:Administrator has unconfigured the peer )
>controller/
> 2015-05-09 Sat 14:55:39:548.516 IST nodei6 [Thread 140000752441088,
>Pid 18669]: BGP [SYS_WARN]: BgpPeerMessageLog: BGP Peer
>default-
>:default-
>MpExtension (1) controller/
> 2015-05-09 Sat 14:55:39:548.843 IST nodei6 [Thread 140000752441088,
>Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotifica
>default-
>:default-
>6 and SubCode 5 ( Cease:Connection is rejected by the peer )
>controller/
> 2015-05-09 Sat 14:55:39:549.138 IST nodei6 [Thread 140000752441088,
>Pid 18669]: BGP [SYS_NOTICE]: BgpPeerNotifica
>default-
>:default-
>2 and SubCode 7 ( OPEN Message Error:Unsupported Capability )
>controller/
> 2015-05-09 Sat 14:55:39:597.924 IST nodei6 [Thread 140000714655488,
>Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: PassiveOpen
>in state: Idle peer ip: 192.168.22.5 ( )
>controller/
> 2015-05-09 Sat 14:55:40:694.229 IST nodei6 [Thread 140000054171392,
>Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: Event: Tcp
>Connection Closed peer ip: 192.168.22.4 ( nodei9 )
>controller/
> 2015-05-09 Sat 14:55:46:241.071 IST nodei6 [Thread 140000037377792,
>Pid 18669]: XMPP [SYS_NOTICE]: XmppEventLog: Mode Server: Event: Tcp
>Connection Closed peer ip: 192.168.22.4 ( nodei9-1 )
>controller/
> 2015-05-09 Sat 14:56:05:916.364 IST nodei6 [Thread 140012986161088,
>Pid 31844]: SANDESH: Logging: DISABLED -> ENABLED
> 2015-05-09 Sat 14:56:05:916.563 IST nodei6 [Thread 140012986161088,
>Pid 31844]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_NOTICE ]
>log4level: [ TRACE ] -> [ WARN ]
> 2015-05-09 Sat 14:56:05:917.715 IST nodei6 [Thread 140012986161088,
>Pid 31844]: Starting Bgp Server at port 179
> 2015-05-09 Sat 14:56:05:930.579 IST nodei6 [Thread 140012986161088,
>Pid 31844]: SANDESH: No Client: 1431163565930447
>SandeshModuleC
>nodei6:
>successful_
>1431163565930223 collector_name = primary = 0.0.0.0:0 secondary =
>0.0.0.0:0 rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0
>blocked_duration = 00:00:00 blocked_count = 0 average_
>errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0
>blocked_duration = 00:00:00 blocked_count = 0 average_
>errors = 0 ] ] ]
>
>
>
>
> Testbed
> --------------
>
> env.roledefs = {
> 'all': [host1, host2, host3, host4, host5, host6],
> 'cfgm': [host1, host2, host3],
> 'openstack': [host1, host2, host3],
> 'webui': [host2],
> 'control': [host1, host3],
> 'compute': [host4, host5, host6],
> 'tsn': [host4, host5],
> 'toragent': [host4, host5],
> 'collector': [host1, host3],
> 'database': [host1, host2, host3],
> 'build': [host_build],
> }
>
> env.hostnames = {
> 'all': ['nodei6', 'nodei7', 'nodei8', 'nodei9', 'nodei10',
>'nodei19']
> }
>
>To manage notifications about this bug go to:
>https:/