Juniper Openstack

vdns: agent to dns server comm broken after lab restart

Bug #1478733 reported by Senthilnathan Murugappan on 2015-07-27

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Juniper Openstack	Fix Released	Medium	Nipa	Juniper Openstack r2.22

Bug Description

Observed that dns is broken after dead office (all the controllers were down due to power strip failure) recovery.
Seems agent is connected to the wrong controller after restart.

Setup Details:
https://10.84.5.100:8143/#p=setting_sm_clusters&q[cluster_id]=cluster_scale
creds: (admin/contrail123)

Please find the sandesh traces below:
traces:
2015-07-24 07:00:59.702 DnsBindTrace: Vm Interface added : TestPublicTenant-VN02f6Eca-VM9826Af5 controller/src/vnsw/agent/services/dns_proto.cc 126
2015-07-24 07:00:59.718 DnsBindTrace: Ipam Notify : default-domain:default-project:default-network-ipam controller/src/vnsw/agent/services/dns_proto.cc 226
2015-07-24 07:00:59.718 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:TestPublicTenant:TestPublicTenant-VN02f6Eca IPv4 : 124.195.8.12 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:00:59.718 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:TestPublicTenant:TestPublicTenant-VN02f6Eca IPv4 : 124.195.8.12 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:00:59.719 DnsBindTrace: Vn Notify : default-domain:TestPublicTenant:TestPublicTenant-VN02f6Eca controller/src/vnsw/agent/services/dns_proto.cc 172
2015-07-24 07:00:59.719 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:TestPublicTenant:TestPublicTenant-VN02f6Eca IPv4 : 124.195.8.12 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:00:59.720 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:TestPublicTenant:TestPublicTenant-VN02f6Eca IPv4 : 124.195.8.12 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:01:00.021 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:TestPublicTenant:TestPublicTenant-VN02f6Eca IPv4 : 124.195.8.12 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:01:04.495 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 61040 IN/A/ntp.ubuntu.com/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:01:04.495 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 6412 IN/AAAA/ntp.ubuntu.com/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:01:04.496 DnsBindTrace: Query successful : xid = 61040 IN/A/ntp.ubuntu.com/91.189.94.4; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:01:04.498 DnsBindError: Query failed : Non-existent domain xid = 6412 IN/AAAA/ntp.ubuntu.com/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:01:05.331 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 31221 IN/A/changelogs.ubuntu.com/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:01:05.331 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 6461 IN/AAAA/changelogs.ubuntu.com/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:01:05.332 DnsBindTrace: Query successful : xid = 31221 IN/A/changelogs.ubuntu.com/91.189.95.36; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:01:05.332 DnsBindTrace: Query successful : xid = 6461 IN/AAAA/changelogs.ubuntu.com/2001:67c:1560:8008::10; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:01:09.181 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 25756 IN/A/does-not-exist.example.com/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:01:09.181 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 61437 IN/AAAA/does-not-exist.example.com/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:01:09.281 DnsBindError: Query failed : Non-existent domain xid = 25756 IN/A/does-not-exist.example.com/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:01:09.282 DnsBindError: Query failed : Non-existent domain xid = 61437 IN/AAAA/does-not-exist.example.com/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:01:09.283 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 46782 IN/A/example.invalid/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:01:09.283 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 38026 IN/AAAA/example.invalid/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:01:09.285 DnsBindError: Query failed : Non-existent domain xid = 46782 IN/A/example.invalid/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:01:09.286 DnsBindError: Query failed : Non-existent domain xid = 38026 IN/AAAA/example.invalid/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:01:09.287 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 44517 IN/A/Ga3PvuNuPY5gjParaQoZCT0W5VNGZekq/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:01:09.287 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 55463 IN/AAAA/Ga3PvuNuPY5gjParaQoZCT0W5VNGZekq/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:01:09.379 DnsBindError: Query failed : Non-existent domain xid = 44517 IN/A/Ga3PvuNuPY5gjParaQoZCT0W5VNGZekq/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:01:09.380 DnsBindError: Query failed : Non-existent domain xid = 55463 IN/AAAA/Ga3PvuNuPY5gjParaQoZCT0W5VNGZekq/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:01:09.381 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 4870 IN/A/metadata.google.internal/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:01:09.381 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 25662 IN/AAAA/metadata.google.internal/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:01:09.383 DnsBindError: Query failed : Non-existent domain xid = 4870 IN/A/metadata.google.internal/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:01:09.384 DnsBindError: Query failed : Non-existent domain xid = 25662 IN/AAAA/metadata.google.internal/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:01:11.249 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 6301 IN/A/testpublictenant-vn02f6eca-vm9826af5/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:01:11.400 DnsBindError: Query failed : Non-existent domain xid = 6301 IN/A/testpublictenant-vn02f6eca-vm9826af5/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:01:11.410 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 4785 IN/A/testpublictenant-vn02f6eca-vm9826af5/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:01:11.412 DnsBindError: Query failed : Non-existent domain xid = 4785 IN/A/testpublictenant-vn02f6eca-vm9826af5/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:01:11.468 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 57558 IN/A/entropy.ubuntu.com/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:01:11.468 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 58753 IN/AAAA/entropy.ubuntu.com/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:01:11.469 DnsBindTrace: Query successful : xid = 57558 IN/A/entropy.ubuntu.com/91.189.94.53; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:01:11.470 DnsBindError: Query failed : Non-existent domain xid = 58753 IN/AAAA/entropy.ubuntu.com/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:01:16.253 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 2199 IN/A/ntp.ubuntu.com/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:01:16.253 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 16463 IN/AAAA/ntp.ubuntu.com/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:01:16.253 DnsBindTrace: Query successful : xid = 2199 IN/A/ntp.ubuntu.com/91.189.89.199; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:01:16.255 DnsBindError: Query failed : Non-existent domain xid = 16463 IN/AAAA/ntp.ubuntu.com/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:01:19.742 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 45719 IN/A/entropy.ubuntu.com/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:01:19.742 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 36439 IN/AAAA/entropy.ubuntu.com/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:01:19.743 DnsBindTrace: Query successful : xid = 45719 IN/A/entropy.ubuntu.com/91.189.94.50; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:01:19.744 DnsBindError: Query failed : Non-existent domain xid = 36439 IN/AAAA/entropy.ubuntu.com/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:01:36.011 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 60911 IN/A/ntp.ubuntu.com/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:01:36.011 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 12392 IN/AAAA/ntp.ubuntu.com/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:01:36.012 DnsBindTrace: Query successful : xid = 60911 IN/A/ntp.ubuntu.com/91.189.89.199; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:01:36.013 DnsBindError: Query failed : Non-existent domain xid = 12392 IN/AAAA/ntp.ubuntu.com/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:05:18.506 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 12113 controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:18.506 DnsBindTrace: Query successful : xid = 12113 IN/PTR/254.169.254.169.in-addr.arpa/metadata; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:05:18.507 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 48881 controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:18.507 DnsBindTrace: Query successful : xid = 48881 IN/A/metadata/169.254.169.254; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:05:18.534 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 40322 controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:18.534 DnsBindTrace: Query successful : xid = 40322 IN/A/metadata/169.254.169.254; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:05:18.534 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 37249 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:18.536 DnsBindError: Query failed : Non-existent domain xid = 37249 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:05:18.537 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 29698 controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:18.537 DnsBindTrace: Query successful : xid = 29698 IN/A/metadata/169.254.169.254; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:05:18.537 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 27600 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:18.538 DnsBindError: Query failed : Non-existent domain xid = 27600 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:05:18.540 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 45121 controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:18.540 DnsBindTrace: Query successful : xid = 45121 IN/A/metadata/169.254.169.254; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:05:18.540 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 1900 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:18.541 DnsBindError: Query failed : Non-existent domain xid = 1900 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:05:18.761 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 13429 controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:18.761 DnsBindTrace: Query successful : xid = 13429 IN/A/metadata/169.254.169.254; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:05:18.761 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 30258 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:18.762 DnsBindError: Query failed : Non-existent domain xid = 30258 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:05:18.765 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 13172 controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:18.765 DnsBindTrace: Query successful : xid = 13172 IN/A/metadata/169.254.169.254; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:05:18.765 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 9484 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:18.766 DnsBindError: Query failed : Non-existent domain xid = 9484 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:05:18.783 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 24725 controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:18.783 DnsBindTrace: Query successful : xid = 24725 IN/A/metadata/169.254.169.254; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:05:18.783 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 26132 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:18.785 DnsBindError: Query failed : Non-existent domain xid = 26132 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:05:18.786 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 60324 controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:18.786 DnsBindTrace: Query successful : xid = 60324 IN/A/metadata/169.254.169.254; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:05:18.786 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 33483 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:18.787 DnsBindError: Query failed : Non-existent domain xid = 33483 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:05:19.966 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 12428 controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:19.966 DnsBindTrace: Query successful : xid = 12428 IN/PTR/254.169.254.169.in-addr.arpa/metadata; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:05:19.967 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 22752 controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:19.967 DnsBindTrace: Query successful : xid = 22752 IN/A/metadata/169.254.169.254; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:05:19.979 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 55839 controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:19.979 DnsBindTrace: Query successful : xid = 55839 IN/A/metadata/169.254.169.254; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:05:19.979 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 28488 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:19.982 DnsBindError: Query failed : Non-existent domain xid = 28488 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:05:19.983 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 50626 controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:19.984 DnsBindTrace: Query successful : xid = 50626 IN/A/metadata/169.254.169.254; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:05:19.984 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 12739 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:19.985 DnsBindError: Query failed : Non-existent domain xid = 12739 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:05:19.987 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 58945 controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:19.987 DnsBindTrace: Query successful : xid = 58945 IN/A/metadata/169.254.169.254; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:05:19.987 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 46182 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:19.989 DnsBindError: Query failed : Non-existent domain xid = 46182 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:05:20.188 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 55073 controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:20.188 DnsBindTrace: Query successful : xid = 55073 IN/A/metadata/169.254.169.254; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:05:20.188 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 3768 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:20.190 DnsBindError: Query failed : Non-existent domain xid = 3768 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:05:20.192 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 50970 controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:20.192 DnsBindTrace: Query successful : xid = 50970 IN/A/metadata/169.254.169.254; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:05:20.192 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 45325 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:20.193 DnsBindError: Query failed : Non-existent domain xid = 45325 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:05:20.216 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 15277 IN/A/testpublictenant-vn02f6eca-vm9826af5/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:20.216 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 47619 IN/AAAA/testpublictenant-vn02f6eca-vm9826af5/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:05:20.217 DnsBindError: Query failed : Non-existent domain xid = 15277 IN/A/testpublictenant-vn02f6eca-vm9826af5/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:05:20.218 DnsBindError: Query failed : Non-existent domain xid = 47619 IN/AAAA/testpublictenant-vn02f6eca-vm9826af5/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:05:50.753 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:admin:TestPublic IPv4 : 1.1.1.5 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:05:50.755 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:TestPublicTenant:TestPublicTenant-VN02f6Eca IPv4 : 124.195.8.12 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:05:50.755 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:admin:TestPublic IPv4 : 1.1.1.5 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:05:50.757 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:admin:TestPublic IPv4 : 1.1.1.5 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:05:50.758 DnsBindTrace: Vn Notify : default-domain:admin:TestPublic controller/src/vnsw/agent/services/dns_proto.cc 172
2015-07-24 07:05:50.758 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:admin:TestPublic IPv4 : 1.1.1.5 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:05:50.759 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:TestPublicTenant:TestPublicTenant-VN02f6Eca IPv4 : 124.195.8.12 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:05:50.759 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:admin:TestPublic IPv4 : 1.1.1.5 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:07:20.320 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 1941 controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:07:20.320 DnsBindTrace: Query successful : xid = 1941 IN/A/metadata/169.254.169.254; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:07:20.321 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 8430 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:07:20.322 DnsBindError: Query failed : Non-existent domain xid = 8430 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:07:20.324 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 11797 controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:07:20.324 DnsBindTrace: Query successful : xid = 11797 IN/A/metadata/169.254.169.254; controller/src/vnsw/agent/services/dns_handler.cc 279
2015-07-24 07:07:20.324 DnsBindTrace: Default DNS query received; interface = TestPublicTenant-VN02f6Eca-VM9826Af5 xid = 20758 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 195
2015-07-24 07:07:20.325 DnsBindError: Query failed : Non-existent domain xid = 20758 IN/AAAA/metadata/; controller/src/vnsw/agent/services/dns_handler.cc 275
2015-07-24 07:27:21.103 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:admin:TestPublic IPv4 : 1.1.1.5 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:27:21.104 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:TestPublicTenant:TestPublicTenant-VN02f6Eca IPv4 : 124.195.8.12 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:27:21.104 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:admin:TestPublic IPv4 : 1.1.1.5 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:27:21.572 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:admin:TestPublic IPv4 : 1.1.1.5 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:27:21.573 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:TestPublicTenant:TestPublicTenant-VN02f6Eca IPv4 : 124.195.8.12 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:27:21.573 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:admin:TestPublic IPv4 : 1.1.1.5 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:30:57.515 DnsBindTrace: Vm Interface added : Project0-VN04E0BB2-VM9fEBf8b controller/src/vnsw/agent/services/dns_proto.cc 126
2015-07-24 07:30:57.532 DnsBindTrace: Ipam Notify : default-domain:Project0:Project0-ipame7C44B controller/src/vnsw/agent/services/dns_proto.cc 226
2015-07-24 07:30:57.532 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:TestPublicTenant:TestPublicTenant-VN02f6Eca IPv4 : 124.195.8.12 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:30:57.532 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:Project0:Project0-VN04E0BB2 IPv4 : 187.139.77.12 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:30:57.532 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:admin:TestPublic IPv4 : 1.1.1.5 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:30:57.532 DnsBindTrace: Vdns Notify : default-domain:vDNSe51DBC controller/src/vnsw/agent/services/dns_proto.cc 231
2015-07-24 07:30:57.532 DnsBindTrace: VDNS modify for VM <All Vms old VDNS : default-domain:vDNSe51DBC new VDNS : default-domain:vDNSe51DBC ttl : 100 floating : no controller/src/vnsw/agent/services/dns_proto.cc 502
2015-07-24 07:30:57.532 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:TestPublicTenant:TestPublicTenant-VN02f6Eca IPv4 : 124.195.8.12 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:30:57.532 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:Project0:Project0-VN04E0BB2 IPv4 : 187.139.77.12 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:30:57.532 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:admin:TestPublic IPv4 : 1.1.1.5 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:30:57.532 DnsBindTrace: Vdns Notify : default-domain:vDNSe51DBC controller/src/vnsw/agent/services/dns_proto.cc 231
2015-07-24 07:30:57.532 DnsBindTrace: VDNS modify for VM <All Vms old VDNS : default-domain:vDNSe51DBC new VDNS : default-domain:vDNSe51DBC ttl : 100 floating : no controller/src/vnsw/agent/services/dns_proto.cc 502
2015-07-24 07:30:57.532 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:TestPublicTenant:TestPublicTenant-VN02f6Eca IPv4 : 124.195.8.12 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:30:57.532 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:Project0:Project0-VN04E0BB2 IPv4 : 187.139.77.12 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:30:57.532 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:admin:TestPublic IPv4 : 1.1.1.5 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:30:57.532 DnsBindTrace: Unable to retrieve VDNS data; VN : default-domain:Project0:Project0-VN04E0BB2 IPv4 : 187.139.77.12 IPv6 : :: controller/src/vnsw/agent/services/dns_proto.cc 522
2015-07-24 07:30:57.533 DnsBindTrace: Vn Notify : default-domain:Project0:Project0-VN04E0BB2 controller/src/vnsw/agent/services/dns_proto.cc 172
2015-07-24 07:30:57.533 DnsBindTrace: DNS update sent for : IN/A/Project0-VN04E0BB2-VM9fEBf8b/187.139.77.12; IPv6 : IN///; VDNS : default-domain:vDNSe51DBC Zone : juniper.net controller/src/vnsw/agent/services/dns_proto.cc 363
2015-07-24 07:30:57.533 DnsBindTrace: DNS update sent for : IN/PTR/12.77.139.187.in-addr.arpa/Project0-VN04E0BB2-VM9fEBf8b; VDNS : default-domain:vDNSe51DBC Zone : 77.139.187.in-addr.arpa controller/src/vnsw/agent/services/dns_proto.cc 388
2015-07-24 07:31:02.360 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:02.360 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:03.112 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:03.112 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:06.925 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:06.925 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:07.364 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:07.365 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:08.118 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:08.118 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:11.930 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:11.930 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:12.370 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:12.370 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:13.124 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:13.124 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:16.936 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:16.936 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:17.375 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:17.375 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:18.128 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:18.129 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:21.940 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:21.940 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:26.946 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:26.946 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:31.951 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:31.951 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:34.130 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:34.130 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:36.956 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:36.956 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:39.134 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:39.134 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:41.961 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:41.961 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:44.139 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:44.139 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:46.967 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:46.967 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:49.144 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:49.144 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93
2015-07-24 07:31:51.971 DnsBindError: Invalid server index: 4294967295; controller/src/dns/bind/bind_resolver.cc 93

Tags:

Revision history for this message

Senthilnathan Murugappan (msenthil) wrote on 2015-07-28:

The computes of interest would be 10.84.21.12, 10.84.21.14

Revision history for this message

Nipa (nipak) wrote on 2015-07-28:

Issue 1:

Agent is not able to contact any named server to resolve DNS queries. Last DOWN control-node resulted in resetting the server-index. For R2.0 and R2.1, we need reset the named server index when connection to control-node is broken, only if the named is the currently elected named.

For R2.2 and mainline, the changes are irrelavant as we send named queries to all known named servers in the list.
For 2.2 and mainline, bug#1466683 adds enchanced feature that makes the bug irrevalant.
======================================================================================================
Issue 2:

1) Orignal list in the server-list of agent [Server A -UP, Server B - UP]
2) After sometime connection to both xmpp-servers pulled down [Server A - Down, Server B - Down]
3) Discovery sends a new list of servers as the control-node's come up and register service
[Server B, Server C]
4) Applying the servers and removing stale server results in only [Server B] connection up.

For R2.2 and mainline, we will honor the discovery list without checking if current connections are DOWN.
Fro R2.0 and R2.1, we will need to test this case and resolve.

Changed in juniperopenstack:
assignee:	nobody → Nipa (nipak)
importance:	Undecided → Medium

Revision history for this message

Nipa (nipak) wrote on 2015-10-02:

Fix commmited in R2.20 and trunk, please retest

Changed in juniperopenstack:
milestone:	none → r2.22
status:	New → Fix Committed

tikitavi (rtikitavi) on 2017-11-08

Changed in juniperopenstack:
status:	Fix Committed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.