[rhel-queens-5.0-214] DnsBindError: Query failed due to wrong host name in the dns record

[Ankit Jain]

This issue seen on RHOSP13 setup.

Test case : scripts.vdns.test_vdns_basic.TestvDNSBasic0.test_vdns_ping_same_vn

Steps as per the test case:

        Test:- Test vdns functionality. On VM launch agent should dynamically update dns records to dns agent
            1. Create vDNS server
            2. Create IPAM using above vDNS data
            3. Create VN using above IPAM and launch 2 VM's within it
            4. Ping between these 2 VM's using dns name
            5. Try to delete vDNS server which has IPAM back-reference[Negative case]
            6. Add CNAME VDNS record for vm1-test and verify we able to ping by alias name
        Pass criteria: Step 4,5 and 6 should pass

Issue: Not able to resolve due to DnsBindError: Query failed : Non-existent domain

From the VM, ping fails when vDNS is configured:

ping: unknown host ctest-vm1-test-76226215v <---Host name not getting resolved

ubuntu@ctest-vm1-test-76226215:~$ ping ctest-vm1-test-76226215
ping: unknown host ctest-vm1-test-76226215
ubuntu@ctest-vm1-test-76226215:~$

In the dns record, the host name is "instance-000002fc" for the vm with the host name "ctest-vm1-test-76226215 as shown below, the same causing the failure.

nslookup for the vm passes:

ubuntu@ctest-vm1-test-76226215:~$ nslookup instance-000002fc
Server: 10.10.10.2
Address: 10.10.10.2#53

Name: instance-000002fc.juniper.net <------wrong host name getting resolved
Address: 10.10.10.3

ubuntu@ctest-vm1-test-76226215:~$

agent introspect logs:

DnsBindTrace: DNS query sent to named server : 10.0.0.19; xid =5059 IN/A/ctest-vm1-test-76226215.juniper.net/; controller/src/vnsw/agent/services/dns_handler.cc 424</element><element>1534835659223983 DnsBindTrace: DNS query sent to named server : 10.0.0.17; xid =5060 IN/A/ctest-vm1-test-76226215.juniper.net/; controller/src/vnsw/agent/services/dns_handler.cc 424</element><element>1534835659224005 DnsBindTrace: DNS query sent to named server : 10.0.0.18; xid =5061 IN/A/ctest-vm1-test-76226215.juniper.net/; controller/src/vnsw/agent/services/dns_handler.cc 424</element><element>1534835659224332 DnsBindError: Unable to find Ipam data; interface = vhost0 controller/src/vnsw/agent/services/dns_handler.cc 169</element><element>1534835659224866 DnsBindError: Invalid or Response ignored xid 216 received from DNS server - dropping controller/src/vnsw/agent/services/dns_handler.cc 623</element><element>1534835659224883 DnsBindError: Query failed : Non-existent domain xid = 5059 controller/src/vnsw/agent/services/dns_handler.cc 568</element><element>1534835659225461 DnsBindError: Query failed : Non-existent domain xid = 5061 controller/src/vnsw/agent/services/dns_handler.cc 568</element><element>1534835659226057 DnsBindError: Query failed : Non-existent domain xid = 5060 controller/src/vnsw/agent/services/dns_handler.cc 568</element><element>1534835659226089 DnsBindTrace: Send invalid BIND response: xid = 5060 controller/src/vnsw/agent/services/dns_handler.cc 640</element><element>1534835659226689 DnsBindTrace: DNS query sent to named server : 10.0.0.19; xid =5062 IN/A/ctest-vm1-test-76226215.juniper.net/; controller/src/vnsw/agent/services/dns_handler.cc 424</element><element>1534835659226714 DnsBindTrace: DNS query sent to named server : 10.0.0.17; xid =5063 IN/A/ctest-vm1-test-76226215.juniper.net/; controller/src/vnsw/agent/services/dns_handler.cc 424</element><element>1534835659226733 DnsBindTrace: DNS query sent to named server : 10.0.0.18; xid =5064 IN/A/ctest-vm1-test-76226215.juniper.net/; controller/src/vnsw/agent/services/dns_handler.cc 424</element><element>1534835659227223 DnsBindError: Unable to find Ipam data; interface = vhost0 controller/src/vnsw/agent/services/dns_handler.cc 169</element><element>1534835659227779 DnsBindError: Invalid or Response ignored xid 212 received from DNS server - dropping controller/src/vnsw/agent/services/dns_handler.cc 623</element><element>1534835659227797 DnsBindError: Query failed : Non-existent domain xid = 5063 controller/src/vnsw/agent/services/dns_handler.cc 568</element><element>1534835659227806 DnsBindError: Query failed : Non-existent domain xid = 5062 controller/src/vnsw/agent/services/dns_handler.cc 568</element><element>1534835659227813 DnsBindError: Query failed : Non-existent domain xid = 5064 controller/src/vnsw/agent/services/dns_handler.cc 568</element><element>1534835659227850 DnsBindTrace: Send invalid BIND response: xid = 5064 controller/src/vnsw/agent/services/dns_handler.cc 640</element></list></traces><more type="bool" identifier="0">false</more></SandeshTraceTextResponse>[heat-admin@overcloud-contraildpdk-0 ~]$

[root@overcloud-contrailcontroller-0 heat-admin]# tcpdump -nni any port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
09:52:42.952386 IP 10.0.0.15.8098 > 10.0.0.19.53: 5069+ [1au] A? ctest-vm1-test-76226215.juniper.net. (110)
09:52:42.952963 IP 10.0.0.19.53 > 10.0.0.15.8098: 5069 NXDomain* 0/1/0 (113)
09:52:42.954728 IP 10.0.0.15.8098 > 10.0.0.19.53: 5072+ [1au] A? ctest-vm1-test-76226215.juniper.net. (110)
09:52:42.955071 IP 10.0.0.19.53 > 10.0.0.15.8098: 5072 NXDomain* 0/1/0 (113)

(control-dns)[root@overcloud-contrailcontroller-0 /etc/contrail/dns]$ cat default-domain-ctest-vdns1-74973401.juniper.net.zone
$ORIGIN .
$TTL 100
juniper.net IN SOA contrail-ns.juniper.net contrail-mx.juniper.net (
                                      54
                                      10800
                                      900
                                      604800
                                      86400
                                      )
                                   NS contrail-ns.juniper.net
$ORIGIN juniper.net
contrail-ns IN A 10.0.0.19
(control-dns)[root@overcloud-contrailcontroller-0 /etc/contrail/dns]$

(control-dns)[root@overcloud-contrailcontroller-1 /etc/contrail/dns]$ cat default-domain-ctest-vdns1-74973401.juniper.net.zone
$ORIGIN .
$TTL 100 ; 1 minute 40 seconds
juniper.net IN SOA contrail-ns.juniper.net. contrail-mx.juniper.net. (
    55 ; serial
    10800 ; refresh (3 hours)
    900 ; retry (15 minutes)
    604800 ; expire (1 week)
    86400 ; minimum (1 day)
    )
   NS contrail-ns.juniper.net.
$ORIGIN juniper.net.
contrail-ns A 10.0.0.17
instance-000002fc A 10.10.10.3 <-----wrong host name present in the record

(control-dns)[root@overcloud-contrailcontroller-2 /etc/contrail/dns]$ cat default-domain-ctest-vdns1-74973401.juniper.net.zone
$ORIGIN .
$TTL 100
juniper.net IN SOA contrail-ns.juniper.net contrail-mx.juniper.net (
                                      54
                                      10800
                                      900
                                      604800
                                      86400
                                      )
                                   NS contrail-ns.juniper.net
$ORIGIN juniper.net
contrail-ns IN A 10.0.0.18
(control-dns)[root@overcloud-contrailcontroller-2 /etc/contrail/dns]$

[Kumar Harsh (hkumar)]

Please provide logs for

contrail-named.log
contrail-vrouter-agent.log

Need setup in failed state .

[alok kumar (kalok)]

The issue is not reproducible even on the same build, so closing the bug.

[alok kumar (kalok)]

the same issue is seen again in dpdk RHOSP13 setup and test is in pdb mode and setup is available to debug it further.

(control-dns)[root@overcloud-contrailcontroller-0 /etc/contrail/dns]$ ls -ltr
total 88
-rwxr-xr-x. 1 contrail contrail 5723 Oct 9 02:10 applynamedconfig.py
-rw-r--r--. 1 contrail contrail 26019 Oct 9 02:10 COPYRIGHT
-rw-r--r--. 2 contrail contrail 4470 Oct 9 02:33 applynamedconfig.pyo
-rw-r--r--. 2 contrail contrail 4470 Oct 9 02:33 applynamedconfig.pyc
-rw-r--r--. 1 contrail contrail 868 Oct 14 15:51 contrail-named-base.conf
-rw-r--r--. 1 contrail contrail 179 Oct 14 15:51 contrail-rndc.conf
-rw-------. 1 contrail contrail 102 Oct 14 15:51 session.key
-rw-r--r--. 1 contrail contrail 1654 Oct 15 15:33 contrail-named.conf
-rw-r--r--. 1 contrail contrail 6 Oct 15 15:33 contrail-named.pid
-rw-r--r--. 1 contrail contrail 767 Oct 15 15:34 default-domain-ctest-vdns1-59027086.juniper.net.zone.jnl
-rw-r--r--. 1 contrail contrail 808 Oct 15 15:34 default-domain-ctest-vdns1-59027086.10.10.10.in-addr.arpa.zone.jnl
-rw-r--r--. 1 contrail contrail 388 Oct 15 15:46 default-domain-ctest-vdns1-59027086.juniper.net.zone
-rw-r--r--. 1 contrail contrail 390 Oct 15 15:49 default-domain-ctest-vdns1-59027086.10.10.10.in-addr.arpa.zone

(control-dns)[root@overcloud-contrailcontroller-0 /etc/contrail/dns]$ cat default-domain-ctest-vdns1-59027086.juniper.net.zone
$ORIGIN .
$TTL 100 ; 1 minute 40 seconds
juniper.net IN SOA contrail-ns.juniper.net. contrail-mx.juniper.net. (
                                55 ; serial
                                10800 ; refresh (3 hours)
                                900 ; retry (15 minutes)
                                604800 ; expire (1 week)
                                86400 ; minimum (1 day)
                                )
                        NS contrail-ns.juniper.net.
$ORIGIN juniper.net.
contrail-ns A 10.0.0.16
instance-0000031f A 10.10.10.3

(control-dns)[root@overcloud-contrailcontroller-0 /etc/contrail/dns]$ cat default-domain-ctest-vdns1-59027086.10.10.10.in-addr.arpa.zone
$ORIGIN .
$TTL 100 ; 1 minute 40 seconds
10.10.10.in-addr.arpa IN SOA contrail-ns.juniper.net. contrail-mx.juniper.net. (
                                55 ; serial
                                10800 ; refresh (3 hours)
                                900 ; retry (15 minutes)
                                604800 ; expire (1 week)
                                86400 ; minimum (1 day)
                                )
                        NS contrail-ns.juniper.net.
$ORIGIN 10.10.10.in-addr.arpa.
3 PTR instance-0000031f.juniper.net.

VM name: ctest-vm1-test-37955079 to be resolved in test.

nslookup query:

ubuntu@ctest-vm1-test-37955079:~$ nslookup ctest-vm1-test-37955079
Server: 10.10.10.2
Address: 10.10.10.2#53

** server can't find ctest-vm1-test-37955079: NXDOMAIN

ubuntu@ctest-vm1-test-37955079:~$ nslookup instance-0000031f <--------- this name is resolved and same name is seen in dns record however it sho...

[Kumar Harsh (hkumar)]

Agent has not received correct config for the virtual machine with ip 10.10.10.3

vm name in interface list is shown as:

"vm_name": {
"_type": "string",
"_identifier": "12",
"__text": "instance-0000031f"},

Incorrect vm name is send to contrail-dns and added to zone file:

&lt;/iq&gt; $ controller/src/xmpp/xmpp_connection.cc 563</element><element>1539617668264480 XmppRxStream: Received xmpp message from: 10.0.0.12 Port 36483 Size: 412 Packet: &lt;?xml version="1.0"?&gt;
&lt;iq type="set" from="overcloud-contraildpdk-1/dns" <email address hidden>/dns-peer" id="2075"&gt;
&lt;dns transid="2075"&gt;
&lt;update&gt;
&lt;virtual-dns&gt;default-domain:ctest-vdns1-59027086&lt;/virtual-dns&gt;
&lt;zone&gt;juniper.net&lt;/zone&gt;
&lt;entry&gt;
&lt;class&gt;1&lt;/class&gt;
&lt;type&gt;1&lt;/type&gt;
&lt;name&gt;instance-0000031f&lt;/name&gt;
&lt;data&gt;10.10.10.3&lt;/data&gt;
&lt;ttl&gt;100&lt;/ttl&gt;
&lt;priority&gt;0&lt;/priority&gt;
&lt;/entry&gt;
&lt;/update&gt;
&lt;/dns&gt;
&lt;/iq&gt; $ controller/src/xmpp/xmpp_connection.cc 563</element><element>1539617668265070 XmppRxStream: Received xmpp message from: 10.0.0.12 Port 36483 Size: 436 Packet: &lt;?xml version="1.0"?&gt;
&lt;iq type="set" from="overcloud-contraildpdk-1/dns" <email address hidden>/dns-peer" id="2077"&gt;
&lt;dns transid="2077"&gt;

Wrong vm name results in nslookup failure.

Debugging further to why agent has not received correct config for virtual machine (10.10.10.3)

libvirt log :

[root@overcloud-contraildpdk-1 qemu]# cat instance-0000031f.log
2018-10-15 15:34:28.433+0000: starting up libvirt version: 3.9.0, package: 14.el7_5.8 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2018-09-04-07:27:24, x86-019.build.eng.bos.redhat.com), qemu version: 2.10.0(qemu-kvm-rhev-2.10.0-21.el7_5.6), hostname: overcloud-contraildpdk-1.localdomain
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name guest=instance-0000031f,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-116-instance-0000031f/master-key.aes -machine pc-i440fx-rhel7.5.0,accel=kvm,usb=off,dump-guest-core=off -cpu IvyBridge,ss=on,pcid=on,hypervisor=on,arat=on,tsc_adjust=on,xsaveopt=on,pdpe1gb=on -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages2M/libvirt/qemu/116-instance-0000031f,share=yes,size=536870912,host-nodes=0,policy=bind -numa node,nodeid=0,cpus=0,memdev=ram-node0 -uuid 83f08b7f-f6b3-44bb-9142-85dbb1abc884 -smbios 'type=1,manufacturer=Red Hat,product=OpenStack Compute,version=0.0.1-3.d7864fbgit.el7ost,serial=00000000-0000-0000-0000-002590e77e82,uuid=83f08b7f-f6b3-44bb-9142-85dbb1abc884,family=Virtual Machine' -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-116-instance-0000031f/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/...

[Kumar Harsh (hkumar)]

vif is not added by contrail-vrouter-dpdk needs to be looked by dpdk side .

contrail/agent/contrail-vrouter-dpdk.log:2018-10-17 16:55:43,984 VROUTER: Adding vif 4 (gen. 19) virtual device tap234a83e3-ca
contrail/agent/contrail-vrouter-dpdk.log:2018-10-17 16:55:43,985 UVHOST: Adding vif 4 virtual device tap234a83e3-ca
contrail/agent/contrail-vrouter-dpdk.log:2018-10-17 16:55:43,985 UVHOST: vif 4 socket tap234a83e3-ca FD is 84
contrail/agent/contrail-vrouter-dpdk.log:2018-10-17 16:55:43,985 UVHOST: error connecting uvhost socket FD 84 to /var/run/vrouter/uvh_vif_tap234a83e3-ca: No such file or directory (2)
contrail/agent/contrail-vrouter-dpdk.log:2018-10-17 16:55:48,988 UVHOST: Client _tap234a83e3-ca: handling message 1

[Sudheendra Rao (sudheendra-k)]

Snippet from Jayaganesh, hence closing the bug (This TC is not valid for DPDK deployment)
-------- Original message --------
From: Jeya Ganesh Babu Jegatheesan <email address hidden>
Date: 18/10/2018 11:37 am (GMT+05:30)
To: Sudheendra Rao <email address hidden>, Jeba Paulaiyan <email address hidden>, Sachchidanand Vaidya <email address hidden>, Sivakumar Ganapathy <email address hidden>
Cc: Alok Kumar <email address hidden>
Subject: Re: https://bugs.launchpad.net/juniperopenstack/+bug/1788146

This seems to be the default behavior in case of dpdk. Was this a newly added testcase using dpdk? I believe this should have been the case starting from 3.x when we started using the vif plugin model for dpdk. The name is provided by nova to os_vif. Unlike the kernel mode vrouter for which the vrouter-port-control is called directly from the nova, for dpdk it’s called from the vif plugin. The plugin doesn’t have access to all the data structures that nova has access to. The ones that are sent from nova to vif only has the “virsh instance name” rather than the nova instance name. I don’t think it can be fixed in our code, fix will only be possible if nova is modified.

[Jeya ganesh babu J (jjeya)]

The bug is valid as it works for kernel. Moving to wont fix as we dont have a way to fix it because of the limitation in the nova/vif-plugin.

[Sergey Kreys (skreys)]

This issue is reproducible with Train and Ussuri versions of OpenStack.
If 5.x is officially supposed to work with either of them, then I suggest to reopen this bug.
The following commit is likely to fix it and can be cherry-picked to 5.1: https://review.opencontrail.org/c/tungstenfabric/tf-nova-vif-driver/+/60254