So I tried to replicate the scenario in the bug description, however, I was not able to reproduce the issue and the UDP traffic got to a backend successfully (see below). The created amphora has the right set of packages and configuration generated by the amphora-agent. The scenario I tested with bionic-train (ML2/OVS): external machine -> provider network -> qrouter ns (non-distributed, non-L3HA router) -> DNAT from LB FIP to LB VIP -> amphora with keepalived+LVS -> backend member IP -> bind9 It would be good to have more information on the networking setup used when the issue was encountered. Specifically, on whether LBs were attached directly to a provider network or via a router and using a FIP (if so, whether that router was distributed, which would be a problem, or L3HA/legacy). Likewise, it would be good to know whether OVN or ML2/OVS was used. https://paste.ubuntu.com/p/BmcVxjMcxs/ (same as below for readability) # deployed via `tox -e func-target bionic-train-ha` ubuntu@dmitriis-bastion:~$ openstack loadbalancer create --name testlb --vip-network-id private_lb_fip_network +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | admin_state_up | True | | availability_zone | | | created_at | 2021-04-06T21:51:05 | | description | | | flavor_id | None | | id | 4273ab40-b760-43b5-9a40-19f7490e3bbd | | listeners | | | name | testlb | | operating_status | OFFLINE | | pools | | | project_id | bd5208e7ae354f339567d71e49cf544b | | provider | amphora | | provisioning_status | PENDING_CREATE | | updated_at | None | | vip_address | 10.42.0.216 | | vip_network_id | 06ab6573-55df-43ad-ace3-0a4afc1964b9 | | vip_port_id | 7453d6da-f0f0-4f1a-bc5f-6258e40a2182 | | vip_qos_policy_id | None | | vip_subnet_id | 4d011e74-cbda-42ab-b58c-7a5aef4cb5fd | +---------------------+--------------------------------------+ ubuntu@dmitriis-bastion:~$ openstack loadbalancer listener create --name test-listener --protocol UDP --protocol-port 53 testlb +-----------------------------+--------------------------------------+ | Field | Value | +-----------------------------+--------------------------------------+ | admin_state_up | True | | connection_limit | -1 | | created_at | 2021-04-06T21:51:58 | | default_pool_id | None | | default_tls_container_ref | None | | description | | | id | 74fc4e7e-9866-4858-b1de-6410383b7b75 | | insert_headers | None | | l7policies | | | loadbalancers | 4273ab40-b760-43b5-9a40-19f7490e3bbd | | name | | | operating_status | OFFLINE | | project_id | bd5208e7ae354f339567d71e49cf544b | | protocol | UDP | | protocol_port | 53 | | provisioning_status | PENDING_CREATE | | sni_container_refs | [] | | timeout_client_data | 50000 | | timeout_member_connect | 5000 | | timeout_member_data | 50000 | | timeout_tcp_inspect | 0 | | updated_at | None | | client_ca_tls_container_ref | None | | client_authentication | NONE | | client_crl_container_ref | None | | allowed_cidrs | None | | tls_ciphers | | | tls_versions | | | alpn_protocols | | +-----------------------------+--------------------------------------+ ubuntu@dmitriis-bastion:~$ openstack loadbalancer pool create --protocol UDP --lb-algorithm ROUND_ROBIN --loadbalancer testlb --name test-pool +----------------------+--------------------------------------+ | Field | Value | +----------------------+--------------------------------------+ | admin_state_up | True | | created_at | 2021-04-06T21:52:53 | | description | | | healthmonitor_id | | | id | 1353a352-49bd-4bab-8ee4-61fe4f70586a | | lb_algorithm | ROUND_ROBIN | | listeners | | | loadbalancers | 4273ab40-b760-43b5-9a40-19f7490e3bbd | | members | | | name | test-pool | | operating_status | OFFLINE | | project_id | bd5208e7ae354f339567d71e49cf544b | | protocol | UDP | | provisioning_status | PENDING_CREATE | | session_persistence | None | | updated_at | None | | tls_container_ref | None | | ca_tls_container_ref | None | | crl_container_ref | None | | tls_enabled | False | | tls_ciphers | | | tls_versions | | +----------------------+--------------------------------------+ ubuntu@dmitriis-bastion:~$ openstack loadbalancer listener set --default-pool test-pool test-listener ubuntu@dmitriis-bastion:~$ openstack loadbalancer list +--------------------------------------+--------+----------------------------------+-------------+---------------------+------------------+----------+ | id | name | project_id | vip_address | provisioning_status | operating_status | provider | +--------------------------------------+--------+----------------------------------+-------------+---------------------+------------------+----------+ | 4273ab40-b760-43b5-9a40-19f7490e3bbd | testlb | bd5208e7ae354f339567d71e49cf544b | 10.42.0.216 | ACTIVE | OFFLINE | amphora | +--------------------------------------+--------+----------------------------------+-------------+---------------------+------------------+----------+ ubuntu@dmitriis-bastion:~$ openstack server create --network private --key-name zaza --image focal --flavor m1.medium testvm +-------------------------------------+----------------------------------------------+ | Field | Value | +-------------------------------------+----------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | | | OS-EXT-SRV-ATTR:host | None | | OS-EXT-SRV-ATTR:hypervisor_hostname | None | | OS-EXT-SRV-ATTR:instance_name | | | OS-EXT-STS:power_state | NOSTATE | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | None | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | | | adminPass | Z4shpEHGkmEj | | config_drive | | | created | 2021-04-06T21:59:48Z | | flavor | m1.medium (3) | | hostId | | | id | 6f3a6a64-ad85-4d68-ac6b-f77af043693b | | image | focal (515ad94c-692f-44e7-8c04-03f036a5902c) | | key_name | zaza | | name | testvm | | progress | 0 | | project_id | bd5208e7ae354f339567d71e49cf544b | | properties | | | security_groups | name='default' | | status | BUILD | | updated | 2021-04-06T21:59:48Z | | user_id | 10c90c3cc2474bfaa45be492f50351ad | | volumes_attached | | +-------------------------------------+----------------------------------------------+ ubuntu@dmitriis-bastion:~$ openstack server list +--------------------------------------+--------+--------+-----------------------+-------+-----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+--------+--------+-----------------------+-------+-----------+ | 6f3a6a64-ad85-4d68-ac6b-f77af043693b | testvm | ACTIVE | private=192.168.0.199 | focal | m1.medium | +--------------------------------------+--------+--------+-----------------------+-------+-----------+ ubuntu@dmitriis-bastion:~$ openstack loadbalancer member create --name dns-server --subnet-id=private_subnet --disable-backup --address 192.168.0.199 --protocol-port 53 test-pool +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | address | 192.168.0.199 | | admin_state_up | True | | created_at | 2021-04-06T22:03:23 | | id | 9a5a2a3d-5214-4566-a09f-f6711435ce61 | | name | dns-server | | operating_status | NO_MONITOR | | project_id | bd5208e7ae354f339567d71e49cf544b | | protocol_port | 53 | | provisioning_status | PENDING_CREATE | | subnet_id | ab85c199-9938-40d3-aa5c-5ea12f241632 | | updated_at | None | | weight | 1 | | monitor_port | None | | monitor_address | None | | backup | False | +---------------------+--------------------------------------+ ubuntu@dmitriis-bastion:~$ openstack loadbalancer member list test-pool +--------------------------------------+------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ | id | name | project_id | provisioning_status | address | protocol_port | operating_status | weight | +--------------------------------------+------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ | 9a5a2a3d-5214-4566-a09f-f6711435ce61 | dns-server | bd5208e7ae354f339567d71e49cf544b | ACTIVE | 192.168.0.199 | 53 | NO_MONITOR | 1 | +--------------------------------------+------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ ubuntu@dmitriis-bastion:~$ openstack loadbalancer amphora list +--------------------------------------+--------------------------------------+-----------+--------+-----------------------------------------+-------------+ | id | loadbalancer_id | status | role | lb_network_ip | ha_ip | +--------------------------------------+--------------------------------------+-----------+--------+-----------------------------------------+-------------+ | 5185e364-c3a6-4126-bebc-6fafea0fb807 | 4273ab40-b760-43b5-9a40-19f7490e3bbd | ALLOCATED | MASTER | fc00:7449:7270:3ff8:f816:3eff:fe59:c494 | 10.42.0.216 | | fb8dacc0-8c7b-4fd3-af96-663d887026cb | 4273ab40-b760-43b5-9a40-19f7490e3bbd | ALLOCATED | BACKUP | fc00:7449:7270:3ff8:f816:3eff:fee9:dcaa | 10.42.0.216 | | 0667974b-d597-4b5f-ab1a-ce60a0ad60fb | None | READY | None | fc00:7449:7270:3ff8:f816:3eff:feca:d30a | None | | f7577d41-a670-4c3f-8782-9ff339c8e8cd | None | READY | None | fc00:7449:7270:3ff8:f816:3eff:fe1b:621b | None | +--------------------------------------+--------------------------------------+-----------+--------+-----------------------------------------+-------------+ ubuntu@dmitriis-bastion:~$ openstack port list | grep 10.42.0.216 | 7453d6da-f0f0-4f1a-bc5f-6258e40a2182 | octavia-lb-4273ab40-b760-43b5-9a40-19f7490e3bbd | fa:16:3e:31:68:b8 | ip_address='10.42.0.216', subnet_id='4d011e74-cbda-42ab-b58c-7a5aef4cb5fd' | DOWN | ubuntu@dmitriis-bastion:~$ juju ssh octavia/0 ubuntu@juju-d9bf09-zaza-3b752ba3303f-10:~$ # cat > ~/.ssh/id_rsa # ubuntu@juju-d9bf09-zaza-3b752ba3303f-10:~$ ssh fc00:7449:7270:3ff8:f816:3eff:fe59:c494 # The VIP is present on the amphora interface ubuntu@amphora-5185e364-c3a6-4126-bebc-6fafea0fb807:~$ sudo ip netns exec amphora-haproxy ip a s 1: lo: mtu 65536 qdisc noop state DOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 3: eth1: mtu 1458 qdisc fq state UP group default qlen 1000 link/ether fa:16:3e:48:58:38 brd ff:ff:ff:ff:ff:ff inet 10.42.0.76/24 brd 10.42.0.255 scope global eth1 valid_lft forever preferred_lft forever inet 10.42.0.216/32 scope global eth1 valid_lft forever preferred_lft forever 4: eth2: mtu 1458 qdisc fq state UP group default qlen 1000 link/ether fa:16:3e:48:49:a5 brd ff:ff:ff:ff:ff:ff inet 192.168.0.116/24 brd 192.168.0.255 scope global eth2 valid_lft forever preferred_lft forever # Network connectivity to the backend is present. ubuntu@amphora-5185e364-c3a6-4126-bebc-6fafea0fb807:~$ sudo ip netns exec amphora-haproxy ping 192.168.0.199 sudo: unable to resolve host amphora-5185e364-c3a6-4126-bebc-6fafea0fb807 PING 192.168.0.199 (192.168.0.199) 56(84) bytes of data. 64 bytes from 192.168.0.199: icmp_seq=1 ttl=64 time=5.30 ms ^C --- 192.168.0.199 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.302/5.302/5.302/0.000 ms ubuntu@amphora-5185e364-c3a6-4126-bebc-6fafea0fb807:~$ dpkg -l | grep ipvs ii ipvsadm 1:1.28-3ubuntu0.18.04.1 amd64 Linux Virtual Server support programs ubuntu@amphora-5185e364-c3a6-4126-bebc-6fafea0fb807:~$ dpkg -l | grep keepalived ii keepalived 1:1.3.9-1ubuntu0.18.04.2 amd64 Failover and monitoring daemon for LVS clusters ubuntu@amphora-5185e364-c3a6-4126-bebc-6fafea0fb807:~$ pgrep -af keep 1805 /usr/sbin/keepalived --log-facility=1 -f /var/lib/octavia/vrrp/octavia-keepalived.conf -p /var/lib/octavia/vrrp/octavia-keepalived.pid 1806 /usr/sbin/keepalived --log-facility=1 -f /var/lib/octavia/vrrp/octavia-keepalived.conf -p /var/lib/octavia/vrrp/octavia-keepalived.pid 1807 /usr/sbin/keepalived --log-facility=1 -f /var/lib/octavia/vrrp/octavia-keepalived.conf -p /var/lib/octavia/vrrp/octavia-keepalived.pid 2986 /usr/sbin/keepalived --log-facility=1 -f /var/lib/octavia/lvs/octavia-keepalivedlvs-74fc4e7e-9866-4858-b1de-6410383b7b75.conf -p /var/lib/octavia/lvs/octavia-keepalivedlvs-74fc4e7e-9866-4858-b1de-6410383b7b75.pid -r /var/lib/octavia/lvs/octavia-keepalivedlvs-74fc4e7e-9866-4858-b1de-6410383b7b75.vrrp.pid -c /var/lib/octavia/lvs/octavia-keepalivedlvs-74fc4e7e-9866-4858-b1de-6410383b7b75.check.pid 2987 /usr/sbin/keepalived --log-facility=1 -f /var/lib/octavia/lvs/octavia-keepalivedlvs-74fc4e7e-9866-4858-b1de-6410383b7b75.conf -p /var/lib/octavia/lvs/octavia-keepalivedlvs-74fc4e7e-9866-4858-b1de-6410383b7b75.pid -r /var/lib/octavia/lvs/octavia-keepalivedlvs-74fc4e7e-9866-4858-b1de-6410383b7b75.vrrp.pid -c /var/lib/octavia/lvs/octavia-keepalivedlvs-74fc4e7e-9866-4858-b1de-6410383b7b75.check.pid 2988 /usr/sbin/keepalived --log-facility=1 -f /var/lib/octavia/lvs/octavia-keepalivedlvs-74fc4e7e-9866-4858-b1de-6410383b7b75.conf -p /var/lib/octavia/lvs/octavia-keepalivedlvs-74fc4e7e-9866-4858-b1de-6410383b7b75.pid -r /var/lib/octavia/lvs/octavia-keepalivedlvs-74fc4e7e-9866-4858-b1de-6410383b7b75.vrrp.pid -c /var/lib/octavia/lvs/octavia-keepalivedlvs-74fc4e7e-9866-4858-b1de-6410383b7b75.check.pid root@amphora-5185e364-c3a6-4126-bebc-6fafea0fb807:~# cat /var/lib/octavia/lvs/octavia-keepalivedlvs-74fc4e7e-9866-4858-b1de-6410383b7b75.conf # Configuration for Loadbalancer 4273ab40-b760-43b5-9a40-19f7490e3bbd # Configuration for Listener 74fc4e7e-9866-4858-b1de-6410383b7b75 net_namespace amphora-haproxy virtual_server 10.42.0.216 53 { lb_algo rr lb_kind NAT protocol UDP # Configuration for Pool 1353a352-49bd-4bab-8ee4-61fe4f70586a # Configuration for Member 9a5a2a3d-5214-4566-a09f-f6711435ce61 real_server 192.168.0.199 53 { weight 1 } # the ipvs config is there: root@amphora-5185e364-c3a6-4126-bebc-6fafea0fb807:~# ip netns exec amphora-haproxy ipvsadm --list IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn UDP 10.42.0.216:domain rr -> 192.168.0.199:domain Masq 1 0 0 ============================================== openstack floating ip create ext_net +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | created_at | 2021-04-06T22:44:31Z | | description | | | dns_domain | None | | dns_name | None | | fixed_ip_address | None | | floating_ip_address | 10.5.150.230 | | floating_network_id | d1839b2d-8a13-4bb2-b939-2903599f32ec | | id | b84be861-5d3a-46f5-8198-9203d57c5726 | | name | 10.5.150.230 | | port_details | None | | port_id | None | | project_id | bd5208e7ae354f339567d71e49cf544b | | qos_policy_id | None | | revision_number | 0 | | router_id | None | | status | DOWN | | subnet_id | None | | tags | [] | | updated_at | 2021-04-06T22:44:31Z | +---------------------+--------------------------------------+ openstack server add floating ip testvm 10.5.150.230 # ssh-ed into the backend VM and installed bind9 on it ssh -i ~/.ssh/id_rsa_zaza ubuntu@10.5.150.230 ubuntu@testvm:~$ sudo apt update && sudo apt install -yqq bind9 The following additional packages will be installed: bind9-utils dns-root-data python3-ply # ... ubuntu@testvm:~$ ss -ul 'sport = 53' State Recv-Q Send-Q Local Address:Port Peer Address:Port Process UNCONN 0 0 192.168.0.199:domain 0.0.0.0:* UNCONN 0 0 192.168.0.199:domain 0.0.0.0:* UNCONN 0 0 127.0.0.1:domain 0.0.0.0:* UNCONN 0 0 127.0.0.1:domain 0.0.0.0:* UNCONN 0 0 127.0.0.53%lo:domain 0.0.0.0:* UNCONN 0 0 [::1]:domain [::]:* UNCONN 0 0 [::1]:domain [::]:* UNCONN 0 0 [fe80::f816:3eff:feb0:9df7]%ens2:domain [::]:* UNCONN 0 0 [fe80::f816:3eff:feb0:9df7]%ens2:domain [::]:* # A local test to verify bind9 is working: ubuntu@testvm:~$ nslookup 127.0.0.1 192.168.0.199 1.0.0.127.in-addr.arpa name = localhost. # in parallel to the above ubuntu@testvm:~$ sudo tcpdump -n -i any dst port 53 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes 22:53:51.892763 IP 192.168.0.199.58537 > 192.168.0.199.53: 31056+ PTR? 1.0.0.127.in-addr.arpa. (40) ============================================== # add a floating IP to the LB VIP openstack floating ip set --port octavia-lb-4273ab40-b760-43b5-9a40-19f749 # check the security group rules for the project ubuntu@dmitriis-bastion:~$ openstack security group rule list fb2f1280-8f00-44b9-9665-cb78e811a687 +--------------------------------------+-------------+-----------+-----------+------------+--------------------------------------+ | ID | IP Protocol | Ethertype | IP Range | Port Range | Remote Security Group | +--------------------------------------+-------------+-----------+-----------+------------+--------------------------------------+ | 1ee04be6-32ac-4ecf-83a7-0b3f29d99a7d | None | IPv4 | 0.0.0.0/0 | | None | | 571fc7a4-e5c5-4464-b858-7fcbaad0aded | None | IPv4 | 0.0.0.0/0 | | fb2f1280-8f00-44b9-9665-cb78e811a687 | | 7ac0689f-9a1a-4a7c-bcbd-bbe67cea4d57 | tcp | IPv4 | 0.0.0.0/0 | 22:22 | None | | 8bce04ed-cbe3-437e-90e4-65d72c85cf15 | None | IPv6 | ::/0 | | fb2f1280-8f00-44b9-9665-cb78e811a687 | | c39592c2-da03-41e3-baf2-be613ef5c7c8 | tcp | IPv4 | 0.0.0.0/0 | 80:80 | None | | f2aa6066-6ccb-46d7-8f63-a6f772aac72e | None | IPv6 | ::/0 | | None | | f3fa9d2a-2d0d-4fe5-af8a-6f5163522709 | icmp | IPv4 | 0.0.0.0/0 | | None | +--------------------------------------+-------------+-----------+-----------+------------+--------------------------------------+ ubuntu@dmitriis-bastion:~$ openstack security group rule create --ingress --protocol udp --description allow-dns --dst-port 53:53 fb2f1280-8f00-44b9-9665-cb78e811a687 +-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | created_at | 2021-04-06T23:27:46Z | | description | allow-dns | | direction | ingress | | ether_type | IPv4 | | id | b2736861-4961-4c61-93d4-82b6337acf65 | | name | None | | port_range_max | 53 | | port_range_min | 53 | | project_id | bd5208e7ae354f339567d71e49cf544b | | protocol | udp | | remote_group_id | None | | remote_ip_prefix | 0.0.0.0/0 | | revision_number | 0 | | security_group_id | fb2f1280-8f00-44b9-9665-cb78e811a687 | | tags | [] | | updated_at | 2021-04-06T23:27:46Z | +-------------------+--------------------------------------+ ubuntu@dmitriis-bastion:~$ openstack security group rule list fb2f1280-8f00-44b9-9665-cb78e811a687 +--------------------------------------+-------------+-----------+-----------+------------+--------------------------------------+ | ID | IP Protocol | Ethertype | IP Range | Port Range | Remote Security Group | +--------------------------------------+-------------+-----------+-----------+------------+--------------------------------------+ | 1ee04be6-32ac-4ecf-83a7-0b3f29d99a7d | None | IPv4 | 0.0.0.0/0 | | None | | 571fc7a4-e5c5-4464-b858-7fcbaad0aded | None | IPv4 | 0.0.0.0/0 | | fb2f1280-8f00-44b9-9665-cb78e811a687 | | 7ac0689f-9a1a-4a7c-bcbd-bbe67cea4d57 | tcp | IPv4 | 0.0.0.0/0 | 22:22 | None | | 8bce04ed-cbe3-437e-90e4-65d72c85cf15 | None | IPv6 | ::/0 | | fb2f1280-8f00-44b9-9665-cb78e811a687 | | b2736861-4961-4c61-93d4-82b6337acf65 | udp | IPv4 | 0.0.0.0/0 | 53:53 | None | | c39592c2-da03-41e3-baf2-be613ef5c7c8 | tcp | IPv4 | 0.0.0.0/0 | 80:80 | None | | f2aa6066-6ccb-46d7-8f63-a6f772aac72e | None | IPv6 | ::/0 | | None | | f3fa9d2a-2d0d-4fe5-af8a-6f5163522709 | icmp | IPv4 | 0.0.0.0/0 | | None | +--------------------------------------+-------------+-----------+-----------+------------+--------------------------------------+ # try to resolve a name from the outside host via the FIP backed by a VIP ubuntu@dmitriis-bastion:~$ nslookup 127.0.0.1 10.5.150.10 1.0.0.127.in-addr.arpa name = localhost. # the traffic gets past the qrouter namespace (the router is non-l3ha and not distributed) to the vip on the LB network ubuntu@juju-d9bf09-zaza-3b752ba3303f-9:~$ sudo ip netns exec qrouter-4f5edf0f-999a-45bb-aaae-66de6fbd549c tcpdump -n -i any port 53 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes 23:30:44.129252 IP 10.5.0.5.41532 > 10.5.150.10.53: 58315+ PTR? 1.0.0.127.in-addr.arpa. (40) 23:30:44.129334 IP 10.5.0.5.41532 > 10.42.0.216.53: 58315+ PTR? 1.0.0.127.in-addr.arpa. (40) 23:30:44.134402 IP 10.42.0.216.53 > 10.5.0.5.41532: 58315* 1/0/0 PTR localhost. (63) 23:30:44.134431 IP 10.5.150.10.53 > 10.5.0.5.41532: 58315* 1/0/0 PTR localhost. (63) 4 packets captured 4 packets received by filter 0 packets dropped by kernel # also appears at the amphora and gets forwarded to the backend ubuntu@amphora-5185e364-c3a6-4126-bebc-6fafea0fb807:~$ sudo ip netns exec amphora-haproxy tcpdump -i any port 53 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes 23:30:43.696524 IP 10.5.0.5.41532 > 10.42.0.216.domain: 58315+ PTR? 1.0.0.127.in-addr.arpa. (40) 23:30:43.696633 IP 192.168.0.116.41532 > 192.168.0.199.domain: 58315+ PTR? 1.0.0.127.in-addr.arpa. (40) 23:30:43.699456 IP 192.168.0.199.domain > 192.168.0.116.41532: 58315* 1/0/0 PTR localhost. (63) 23:30:43.699479 IP 10.42.0.216.domain > 10.5.0.5.41532: 58315* 1/0/0 PTR localhost. (63) 4 packets captured 4 packets received by filter 0 packets dropped by kernel # Ends up at the backend VM ubuntu@testvm:~$ sudo tcpdump -n -i any port 53 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes 23:30:44.136723 IP 192.168.0.116.41532 > 192.168.0.199.53: 58315+ PTR? 1.0.0.127.in-addr.arpa. (40) 23:30:44.136949 IP 192.168.0.199.53 > 192.168.0.116.41532: 58315* 1/0/0 PTR localhost. (63)