[library] DNS settings are not applied to instances

[Vadim Rovachev]

{"build_id": "2014-03-17_01-18-16", "mirantis": "yes", "build_number": "248", "nailgun_sha": "f58aad317829112913f364347b14f1f0518ad371", "ostf_sha": "dc54d99ddff2f497b131ad1a42362515f2a61afa", "fuelmain_sha": "16637e2ea0ae6fe9a773aceb9d76c6e3a75f6c3b", "astute_sha": "f15f5615249c59c826ea05d26707f062c88db32a", "release": "4.1", "fuellib_sha": "73313007c0914e602246ea41fa5e8ca2dfead9f8"}

Steps to reproduce.
1. Deploy KVM with MasterFuel ISO.
2. Go to Fuel WebUI
3. Deploy OpenStack with parameters:
CentOS 6.4 distributive,
KVM hipervizor,
NovaNetwork,
Install Savanna.
{Controller (KVM)} + {Compute-Ceph OSD (Hardware).}
4. Start Instance(CentOS 6.4)
5. Establish ssh connect with instance.
6.
--------------------------------------------------------------------------
[root@test-2 ~]# ping -c 3 ya.ru
ping: unknown host ya.ru
[root@test-2 ~]# ping -c 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=48 time=38.8 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=48 time=38.5 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=48 time=36.9 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2040ms
rtt min/avg/max/mdev = 36.967/38.118/38.832/0.852 ms
--------------------------------------------------------------------------
7. Start Instance(Ubuntu 12.04)
8. Establish ssh connect with instance.
9.
--------------------------------------------------------------------------
ubuntu@test-1:~$ ping -c 3 ya.ru
ping: unknown host ya.ru
ubuntu@test-1:~$ ping -c 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_req=1 ttl=48 time=36.9 ms
64 bytes from 8.8.8.8: icmp_req=2 ttl=48 time=39.5 ms
64 bytes from 8.8.8.8: icmp_req=3 ttl=48 time=36.7 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 36.796/37.761/39.508/1.237 ms
--------------------------------------------------------------------------
resolv.conf in instances:
--------------------------------------------------------------------------
ubuntu@test-1:~$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 10.0.0.3
search novalocal
--------------------------------------------------------------------------
[root@test-2 ~]# cat /etc/resolv.conf
; generated by /sbin/dhclient-script
search novalocal
nameserver 10.0.0.3
--------------------------------------------------------------------------
IP 10.20.0.3 - Controller node IP
resolv.conf in controller node:
--------------------------------------------------------------------------
[root@node-3 ~]# cat /etc/resolv.conf
# Generated by NetworkManager
domain domain.tld
search domain.tld
nameserver 10.20.0.2
--------------------------------------------------------------------------
IP 10.20.0.2 - Fuel Master node IP
resolv.conf in Fuel Master node:
--------------------------------------------------------------------------
[root@fuel ~]# cat /etc/resolv.conf
nameserver 127.0.0.1
--------------------------------------------------------------------------
[root@fuel etc]# cat /etc/astute.yaml
HOSTNAME: fuel
DNS_DOMAIN: domain.tld
DNS_SEARCH: domain.tld
DNS_UPSTREAM: 8.8.8.8
NTP1: 0.pool.ntp.org
NTP2: 1.pool.ntp.org
NTP3: 2.pool.ntp.org
ADMIN_NETWORK:
  dhcp_pool_start: 10.20.0.128
  static_pool_start: 10.20.0.3
  netmask: 255.255.255.0
  dhcp_pool_end: 10.20.0.254
  interface: eth0
  static_pool_end: 10.20.0.127
  ipaddress: 10.20.0.2

[Vadim Rovachev (vrovachev)]

If change nameserver in controller and conpute nodes in resolv.conf file from 10.20.0.2 to 8.8.8.8, DNS settings are applied

[Vladimir Kuklin (vkuklin)]

This is how nova-network works. You have compute nodes resolving using master node which does not have access to the outer world. This is because you did not specify external DNS servers for node deployment.

[Vadim Rovachev (vrovachev)]

{"build_id": "2014-04-30_10-31-20", "mirantis": "yes", "build_number": "170", "ostf_sha": "134765fcb5a07dce0cd1bb399b2290c988c3c63b", "nailgun_sha": "fb676be97794d4b63b8d179e2a8518ad70448f48", "production": "docker", "api": "1.0", "fuelmain_sha": "44954d8ffd74d27219434ec5b676874504b6bf76", "astute_sha": "3cffebde1e5452f5dbf8f744c6525fc36c7afbf3", "release": "5.0", "fuellib_sha": "fd547e5008957183200608124c677c096bf099f9"}

I really did not specify external DNS servers for node deployment.
But default DNS servers 8.8.4.4 and 8.8.8.8.

in fuel_master node:
vim /etc/astute.yaml:
HOSTNAME: fuel
DNS_DOMAIN: domain.tld
DNS_SEARCH: domain.tld
DNS_UPSTREAM: 8.8.8.8
NTP1: 0.pool.ntp.org
NTP2: 1.pool.ntp.org
NTP3: 2.pool.ntp.org
ADMIN_NETWORK:
  dhcp_pool_start: 10.20.0.128
  netmask: 255.255.255.0
  static_pool_start: 10.20.0.3
  mac: 52:54:00:95:3b:a4
  dhcp_pool_end: 10.20.0.254
  interface: eth0
  static_pool_end: 10.20.0.127
  ipaddress: 10.20.0.2

But I create instace and go to instance console:

.# ping -c 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=46 time=68.6 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=46 time=70.5 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=46 time=70.4 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2073ms

# ping -c 3 ya.ru
ping: unknown host ya.ru

# cat /etc/resolv.conf
; generated by /sbin/dhclient-script
search novalocal
nameserver 10.0.0.3

[Bogdan Dobrelya (bogdando)]

need to check if 53 tcp & udp was enabled for security group

[Bogdan Dobrelya (bogdando)]

and please, while performing some name resolving actions, provide "tcpdump -nnXSs 0 -i br-fw-admin dst port 53" for the interface with 10.0.0.3 IP on it (master node side) as well

[Bogdan Dobrelya (bogdando)]

{"build_id": "2014-06-16_00-31-15", "mirantis": "yes", "build_number": "255", "ostf_sha": "67b61ed3788297fa5d985afec32498d8c0f812db", "nailgun_sha": "984aa7a86487f1488c2f83c052904abd9f589b7f", "production": "docker", "api": "1.0", "fuelmain_sha": "6f355160366475d52050d7898a1080a95ecb9cbf", "astute_sha": "17b1afa5f0dc8f4fca5ed4eb03ec566fbfb5ed19", "release": "5.1", "fuellib_sha": "99d74172887ab81d38132655d6e5d180e8726437"}

Cannot reproduce this:
# ping -c 3 ya.ru
ping: unknown host ya.ru

I have correct dns conversations instead:
$ ping -c 3 ya.ru
PING ya.ru (213.180.204.3): 56 data bytes

and tcpdump at compute node (has no access to internet)
[root@node-2 ~]# tcpdump -i br100 -p icmp or -p udp dst port 53 or src port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br100, link-type EN10MB (Ethernet), capture size 65535 bytes
07:08:26.015612 IP 10.0.0.2 > 10.0.0.3: ICMP 10.0.0.2 udp port bootpc unreachable, length 346
07:08:59.080128 IP 10.0.0.2.33428 > 10.0.0.3.domain: 2+ A? ya.ru. (23)
07:08:59.080279 IP 10.0.0.3.domain > 10.0.0.2.33428: 2 3/0/0 A 93.158.134.3, A 213.180.193.3, A 213.180.204.3 (71)
07:08:59.085743 IP 10.0.0.2 > www.yandex.ru: ICMP echo request, id 40193, seq 0, length 64
07:09:00.089635 IP 10.0.0.2 > www.yandex.ru: ICMP echo request, id 40193, seq 1, length 64
07:09:01.092449 IP 10.0.0.2 > www.yandex.ru: ICMP echo request, id 40193, seq 2, length 64
07:09:02.086338 IP 172.16.0.3 > 10.0.0.2: ICMP host www.yandex.ru unreachable, length 92
07:09:02.086338 IP 172.16.0.3 > 10.0.0.2: ICMP host www.yandex.ru unreachable, length 92
07:09:02.086338 IP 172.16.0.3 > 10.0.0.2: ICMP host www.yandex.ru unreachable, length 92

Note that host www.yandex.ru unreachable due to I use tagged interfaces and my compute node cannot route traffic to the Internet. That fact is not related to the subject (DNS) and could be safely ignored for this issue.

[Bogdan Dobrelya (bogdando)]

ps.
$ cat /etc/resolv.conf
search novalocal
nameserver 10.0.0.3

and 10.0.0.3 is an IP of br100 int of compute node hosting the instance, so I believe this is just OK.
12: br100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether fe:16:3e:21:61:f4 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.3/24 brd 10.0.0.255 scope global br100

[Vadim Rovachev (vrovachev)]

I use Fuel Master node without internet access (It's normal configuration, internet access for Fuel Master node is not necessarily).
Fuel Master(10.20.0.2) is specified on nodes as nameserver, but Fuel Master node without internet access!!! That is why instances don't have DNS resolution.

In instance:

ubuntu@test:~$ cat /etc/resolv.conf
nameserver 10.0.0.3
search novalocal

ubuntu@test:~$ ping ya.ru
ping: unknown host ya.ru

ubuntu@test:~$ ping -c 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_req=1 ttl=47 time=37.2 ms
64 bytes from 8.8.8.8: icmp_req=2 ttl=47 time=39.1 ms
64 bytes from 8.8.8.8: icmp_req=3 ttl=47 time=39.0 ms
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 37.208/38.476/39.167/0.912 ms

In compute:
[root@node-7 ~]# tcpdump -i br100 -p icmp or -p udp dst port 53 or src port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br100, link-type EN10MB (Ethernet), capture size 65535 bytes
16:23:55.339663 IP 10.0.0.4 > 10.0.0.3: ICMP 10.0.0.4 udp port bootpc unreachable, length 346
16:24:55.399246 IP 10.0.0.4 > 10.0.0.3: ICMP 10.0.0.4 udp port bootpc unreachable, length 346
16:25:55.461568 IP 10.0.0.4 > 10.0.0.3: ICMP 10.0.0.4 udp port bootpc unreachable, length 346
16:26:55.523499 IP 10.0.0.4 > 10.0.0.3: ICMP 10.0.0.4 udp port bootpc unreachable, length 346
16:27:55.585654 IP 10.0.0.4 > 10.0.0.3: ICMP 10.0.0.4 udp port bootpc unreachable, length 346
16:28:55.648477 IP 10.0.0.4 > 10.0.0.3: ICMP 10.0.0.4 udp port bootpc unreachable, length 346
16:29:55.711998 IP 10.0.0.4 > 10.0.0.3: ICMP 10.0.0.4 udp port bootpc unreachable, length 346
16:30:55.773465 IP 10.0.0.4 > 10.0.0.3: ICMP 10.0.0.4 udp port bootpc unreachable, length 346
16:31:55.835013 IP 10.0.0.4 > 10.0.0.3: ICMP 10.0.0.4 udp port bootpc unreachable, length 346
...