kerberos setup fails, with broken krb5.conf

[Leo Richard Comerford]

I am attempting to install Samba 4 for use as a primary domain controller using version 4.0.0~alpha18.dfsg1-4 of samba4 on a new install of Ubuntu Server 12.04 beta 2 for x86-64. Installing the samba4 package, or running /usr/share/samba/setup/provision at any other time, does not configure Keberos for samba4, but this message is printed to standard output:

A Kerberos configuration suitable for Samba 4 has been generated at /var/lib/samba/private/krb5.conf

The official Samba 4 HOWTO http://wiki.samba.org/index.php/Samba4/HOWTO suggests copying this file to /etc/krb.conf to configure Kerberos - it seems that this should be /etc/krb5.conf on Ubuntu 12.04. Unfortunately, the whole /var/lib/samba/private/krb5.conf produced by running

sudo /usr/share/samba/setup/provision --realm=irishtown.localonly.rvcomerford.ie --domain=IRISHTOWN --adminpass=[the password] --server-role=dc --host-ip=10.37.55.20

on my system is this:

[libdefaults]
        default_realm = IRISHTOWN.LOCALONLY.RVCOMERFORD.IE
        dns_lookup_realm = false
        dns_lookup_kdc = true

Old posts on the samba mailing list seem to suggest that this file is incomplete http://lists.samba.org/archive/samba/2010-September/158087.html as well incorrect in other ways (dns_lookup_realm should be set to true?) http://lists.samba.org/archive/samba/2010-September/158088.html . And indeed using kinit to test Kerberos, as suggested in the HOWTO, produces a failure when this is the /etc/krb5.conf :

leo@blackbox:~$ kinit -V <email address hidden>
Using default cache: /tmp/krb5cc_1000
Using principal: <email address hidden>
kinit: Cannot contact any KDC for realm 'IRISHTOWN.LOCALONLY.RVCOMERFORD.IE' while getting initial credentials

(The kinit is the version from the krb5-clients package.) I tried producing a localised version of the krb.conf at http://lists.samba.org/archive/samba/2010-September/158087.html by hand, putting in the DC's hostname 'blackbox' for 'pdc' and 'irishtown.localonly.rvcomerford.ie' for 'example.com'. This ended up as

[libdefaults]
        default_realm = IRISHTOWN.LOCALONLY.RVCOMERFORD.IE
        dns_lookup_realm = true
        dns_lookup_kdc = true
        ticket_lifetime = 24h
        forwardable = yes

[realms]
        IRISHTOWN.LOCALONLY.RVCOMERFORD.IE = {
                kdc = blackbox.irishtown.localonly.rvcomerford.ie:88
                admin_server = blackbox.irishtown.localonly.rvcomerford.ie:749
                default_domain = irishtown.localonly.rvcomerford.ie
        }

[domain_realm]
        .irishtown.localonly.rvcomerford.ie = IRISHTOWN.LOCALONLY.RVCOMERFORD.IE
        irishtown.localonly.rvcomerford.ie = IRISHTOWN.LOCALONLY.RVCOMERFORD.IE

Unfortunately, making this /etc/krb5.conf and rebooting the server did not change the behaviour of kinit, which produces exactly the same result when tested in the new setup.

My general network configuration seems to be working. DNS is set up, and the test host commands specified in the HOWTO all seem to work. There doesn't appear to be any firewalling going on. kinit doesn't seem to be generating any messages on /var/log/syslog . There *are* also problems which show up when using smbclient to test the samba4 installation, as suggested in the HOWTO: I have described these in bug 976137 https://bugs.launchpad.net/ubuntu/+source/samba4/+bug/976137 .

[Jelmer Vernooij (jelmer)]

The default configuration should be sufficient if you have DNS set up properly.

Are you using krb5-user or heimdal-clients ?

[Jelmer Vernooij (jelmer)]

Am 08/04/12 19:12, schrieb Leo Richard Comerford:
> I'm using krb5-user. I've restored the default krb5.conf and then
> rebooted, but the situation is as before:
>
> leo@blackbox:~$ ls -l /etc/krb5.conf
> -rw-r--r-- 1 root root 115 Apr 8 16:54 /etc/krb5.conf
> leo@blackbox:~$ cat /etc/krb5.conf
> [libdefaults]
> default_realm = IRISHTOWN.LOCALONLY.RVCOMERFORD.IE
> dns_lookup_realm = false
> dns_lookup_kdc = true
> leo@blackbox:~$ host -t SRV _ldap._tcp.irishtown.localonly.rvcomerford.ie
> _ldap._tcp.irishtown.localonly.rvcomerford.ie has SRV record 0 100 389 blackbox.irishtown.localonly.rvcomerford.ie.
> leo@blackbox:~$ host -t SRV _kerberos._udp.irishtown.localonly.rvcomerford.ie
> _kerberos._udp.irishtown.localonly.rvcomerford.ie has SRV record 0 100 88 blackbox.irishtown.localonly.rvcomerford.ie.
> leo@blackbox:~$ host -t A blackbox.irishtown.localonly.rvcomerford.ie
> blackbox.irishtown.localonly.rvcomerford.ie has address 10.37.55.20
> leo@blackbox:~$ sudo kinit -V <email address hidden>
> Using default cache: /tmp/krb5cc_0
> Using principal: <email address hidden>
> kinit: Cannot contact any KDC for realm 'IRISHTOWN.LOCALONLY.RVCOMERFORD.IE' while getting initial credentials
>

What about:

host -t TXT _kerberos.IRISHTOWN.LOCALONLY.RVCOMERFORD.IE

You might be able to see what is going wrong with wireshark - what sort
of requests are being made?

Cheers,

Jelmer

[Leo Richard Comerford (lrc1)]

That fails:

leo@blackbox:~$ host -t TXT _kerberos.IRISHTOWN.LOCALONLY.RVCOMERFORD.IE
Host _kerberos.IRISHTOWN.LOCALONLY.RVCOMERFORD.IE not found: 3(NXDOMAIN)

I've attached a pcap file for the loopback interface on blackbox. While the capture was running I did

sudo kinit -V <email address hidden>

(and then, just for completeness, the host commands too). I also have an eth0 capture from the same period but there's evidently nothing relevant on it (except maybe for some failed PTR requests showing that there's no reverse DNS for 10.37.55.20).

[Leo Richard Comerford (lrc1)]

I attempted to join a Windows 7 SP1 computer on the local network to the domain known (locally) as irishtown.localonly.rvcomerford.ie , in the manner shown in the first http://www.samba.org/tridge/Samba4Demo/s4demo1.ogv of the Samba4 video demonstrations http://wiki.samba.org/index.php/Samba4/videos . Ping requests, DNS requests, and SSH connections all go from this machine (currently assigned 10.37.55.20 by DHCP) to the server machine without problems, whether the server is identified as 10.37.55.20 or as blackbox.irishtown.localonly.rvcomerford.ie . But attemping to join the AD domain fails: instead of the "Computer Name/Domain Changes" username/password dialog (as seen at 3:41 on the video), an error dialog comes up. The error dialog's detail message is:

Note: This information is intended for a network administrator. If you are not your network's administrator, notify the administrator that you received this information, which has been recorded in the file C:\Windows\debug\dcdiag.txt.

DNS was successfully queried for the service location (SRV) resource record used to locate a domain controller for domain "irishtown.localonly.rvcomerford.ie":

The query was for the SRV record for _ldap._tcp.dc._msdcs.irishtown.localonly.rvcomerford.ie

The following domain controllers were identified by the query:
blackbox.irishtown.localonly.rvcomerford.ie

However no domain controllers could be contacted.

Common causes of this error include:

- Host (A) or (AAAA) records that map the names of the domain controllers to their IP addresses are missing or contain incorrect addresses.

- Domain controllers registered in DNS are not connected to the network or are not running.

I have attached an edited pcap file for the server's eth0 which (hopefully) shows the relevant packets sent during the failed domain change. (I have full pcaps from the server and a client-side netmon capture if anyone wants to see them.) It seems that after finishing with DNS, the client attempted to "LDAP ping" 10.37.55.20 on UDP port 389 but received a "Destination unreachable (Port unreachable)" response. And indeed it seems that there is nothing listening on UDP port 389 on the server, even though there are several samba processes running:

leo@blackbox:~$ sudo nc -vvvzu 10.37.55.20 389
leo@blackbox:~$ sudo lsof | grep TCP
sshd 1076 root 3r IPv4 8802 0t0 TCP *:ssh (LISTEN)
sshd 1076 root 4u IPv6 8805 0t0 TCP *:ssh (LISTEN)
named 1213 bind 20u IPv6 9260 0t0 TCP *:domain (LISTEN)
named 1213 bind 21u IPv4 9272 0t0 TCP localhost:domain (LISTEN)
named 1213 bind 22u IPv4 9274 0t0 TCP blackbox:domain (LISTEN)
named 1213 bind 23u IPv4 9388 0t0 TCP localhost:953 (LISTEN)
named 1213 bind 24u IPv6 9390 0t0 TCP ip6-localhost:953 (LISTEN)
dnsmasq 1293 libvirt-dnsmasq 7u IPv4 7046 0t0 TCP 192.168.122.1:domain (LISTEN)
samba ...

[Leo Richard Comerford (lrc1)]

Sorry,

> this machine (currently assigned 10.37.55.20 by DHCP)

should read

> this machine (currently assigned 10.37.55.26 by DHCP)

[Jelmer Vernooij (jelmer)]

It seems like the issue is mostly in your DNS setup. kinit can't find the realm in DNS.

That's either a bug in the dlz module or a problem in your setup.

[Leo Richard Comerford (lrc1)]

Jelmer,

Have you seen the libpcap files which I have attached? In particular, the first eight packets of the lo capture made while kinit was run, which I attached to https://bugs.launchpad.net/ubuntu/+source/samba4/+bug/976138/comments/4 comment #4?

  1 0.000000 127.0.0.1 -> 127.0.0.1 DNS 109 Standard query SRV _kerberos._udp.IRISHTOWN.LOCALONLY.RVCOMERFORD.IE
  2 0.018521 127.0.0.1 -> 127.0.0.1 DNS 202 Standard query response SRV 0 100 88 blackbox.irishtown.localonly.rvcomerford.ie
  3 0.018804 127.0.0.1 -> 127.0.0.1 DNS 109 Standard query SRV _kerberos._tcp.IRISHTOWN.LOCALONLY.RVCOMERFORD.IE
  4 0.026521 127.0.0.1 -> 127.0.0.1 DNS 202 Standard query response SRV 0 100 88 blackbox.irishtown.localonly.rvcomerford.ie
  5 0.026815 127.0.0.1 -> 127.0.0.1 DNS 103 Standard query A blackbox.irishtown.localonly.rvcomerford.ie
  6 0.035305 127.0.0.1 -> 127.0.0.1 DNS 133 Standard query response A 10.37.55.20
  7 0.035411 10.37.55.20 -> 10.37.55.20 KRB5 274 AS-REQ
  8 0.035427 10.37.55.20 -> 10.37.55.20 ICMP 302 Destination unreachable (Port unreachable)

As you would expect, packets 1-6 are all to and from port 53 UDP, while 7 and 8 are to and from port 88 UDP. I can promise you that no DNS requests go out over eth0 or virbr0 as a result of

sudo kinit -V <email address hidden>

 being run locally on blackbox. Bearing that in mind, and looking at the sequence of packets above, I have to admit that I can see only four possible explanations:

0) kinit is privately cacheing some bad DNS information from the past somewhere rather than making new DNS requests. This seems unlikely to me for a number of reasons: just for one thing, purging and then reinstalling the krb5-user package doesn't seem to change what DNS requests kinit chooses to make at all.

1) kinit itself is misbehaving somehow. This also seems unlikely to me.

2) kinit is getting wrong information from the DNS server it is querying, the one at 127.0.0.1 , port 53 UDP. As best I can tell, the information above makes it clear that kinit is not trying to make DNS requests to any other address or port, and that it is not having any difficulty making DNS requests to and getting valid DNS responses from 127.0.0.1:53 UDP. So - to the best of my little understanding - the only likely way in which my DNS setup could be causing kinit to go wrong here is by causing the 127.0.0.1:53 UDP server to give incorrect information.

But this is an almost pristine install of 12.04b2 Server. The only choices made during install which would be likely to affect DNS setup were giving some static IP settings - address 10.37.55.20, netmask 255.255.255.0, gateway 10.37.55.1, nameserver 10.37.55.20 - and choosing the package tasks "Virtual Machine host" and (of course) "DNS server" (as well as "OpenSSH server"). After install, the only changes which I made to BIND settings were to insert

include "/var/lib/samba/private/named.conf";

into /etc/bind/named.conf , and

tkey-gssapi-keytab "/usr/local/samba/private/dns.keytab";

inside the options { [...] } brackets in /etc/bind/named.conf.options, more or less exactly as instructed by the Samb...

[Leo Richard Comerford (lrc1)]

> After install, the only changes which I made to BIND settings were to insert
> include "/var/lib/samba/private/named.conf";
> into /etc/bind/named.conf , and
> tkey-gssapi-keytab "/usr/local/samba/private/dns.keytab";
> inside the options { [...] } brackets in /etc/bind/named.conf.options

Of course there were also the changes to /etc/apparmor.d/usr.sbin.named as described in Bug #975973 . There aren't any apparmor errors in /var/log/syslog when kinit is run, so I don't think that apparmor is involved here.

Having noticed that I had put

include "/var/lib/samba/private/named.conf";

into /etc/bind/named.conf rather than /etc/bind/named.conf.local by mistake, I switched it round just in case that might make a difference: it didn't.

[Leo Richard Comerford (lrc1)]

In comment #8,

> except for the bind4 package and its dependencies of course

should read "except for the samba4 package and its dependencies of course"

[Jelmer Vernooij (jelmer)]

On Thu, Apr 12, 2012 at 10:34:25PM -0000, Leo Richard Comerford wrote:
> Jelmer,
>
> Have you seen the libpcap files which I have attached? In particular,
> the first eight packets of the lo capture made while kinit was run,
> which I attached to
> https://bugs.launchpad.net/ubuntu/+source/samba4/+bug/976138/comments/4
> comment #4?
>
> 1 0.000000 127.0.0.1 -> 127.0.0.1 DNS 109 Standard query SRV _kerberos._udp.IRISHTOWN.LOCALONLY.RVCOMERFORD.IE
> 2 0.018521 127.0.0.1 -> 127.0.0.1 DNS 202 Standard query response SRV 0 100 88 blackbox.irishtown.localonly.rvcomerford.ie
> 3 0.018804 127.0.0.1 -> 127.0.0.1 DNS 109 Standard query SRV _kerberos._tcp.IRISHTOWN.LOCALONLY.RVCOMERFORD.IE
> 4 0.026521 127.0.0.1 -> 127.0.0.1 DNS 202 Standard query response SRV 0 100 88 blackbox.irishtown.localonly.rvcomerford.ie
> 5 0.026815 127.0.0.1 -> 127.0.0.1 DNS 103 Standard query A blackbox.irishtown.localonly.rvcomerford.ie
> 6 0.035305 127.0.0.1 -> 127.0.0.1 DNS 133 Standard query response A 10.37.55.20
> 7 0.035411 10.37.55.20 -> 10.37.55.20 KRB5 274 AS-REQ
> 8 0.035427 10.37.55.20 -> 10.37.55.20 ICMP 302 Destination unreachable (Port unreachable)

Further down though, it looks for TXT field _kerberos.DOMAIN and
fails. Perhaps that won't happen if the KRB5 request here succeeded.

Is 10.37.55.20 the wrong IP address? Is that not the host on which
Samba is installed?

This still very much seems like a configuration issue to me.

Cheers,

Jelmer

[Leo Richard Comerford (lrc1)]

> Is 10.37.55.20 the wrong IP address? Is that not the host on which Samba is installed?

It absolutely is: samba4, bind and kinit are all running on the same server, whose IP address for eth0 is 10.37.55.20 . Again, during installation the server was given the static IP settings address 10.37.55.20, netmask 255.255.255.0, gateway 10.37.55.1, nameserver 10.37.55.20 . You can see the resulting ifconfig -a output in comment #2. I can't see any problems in connecting from the server to itself as 10.37.55.20, or indeed as blackbox.irishtown.localonly.rvcomerford.ie - ping, ssh and DNS requests all work fine. (They all work fine from other machines on the local network, too.) Also, the ICMP error message comes from source 10.37.55.20, it's a 'port unreachable' rather than 'host unreachable' message, and in fact lsof confirms (see comment #5) that there is no process listening on 10.37.55.20:88 UDP.

Speaking of which, I have discovered that starting the server with the Ethernet cable disconnected causes it to start with samba processes listening on port 88 UDP (UDP Kerberos) and port 389 UDP (UDP LDAP):

leo@blackbox:~$ sudo lsof | grep UDP
samba 1031 root 22u IPv4 8563 0t0 UDP *:netbios-ns
samba 1031 root 23u IPv4 8564 0t0 UDP *:netbios-dgm
samba 1031 root 24u IPv4 8565 0t0 UDP 10.37.55.255:netbios-ns
samba 1031 root 25u IPv4 8566 0t0 UDP blackbox:netbios-ns
samba 1031 root 26u IPv4 8567 0t0 UDP 10.37.55.255:netbios-dgm
samba 1031 root 27u IPv4 8568 0t0 UDP blackbox:netbios-dgm
samba 1035 root 22u IPv4 8591 0t0 UDP *:ldap
samba 1035 root 23u IPv4 8592 0t0 UDP blackbox:ldap
samba 1036 root 23u IPv4 8600 0t0 UDP *:kerberos
samba 1036 root 25u IPv4 8602 0t0 UDP *:kpasswd
samba 1036 root 26u IPv4 8603 0t0 UDP blackbox:kerberos
samba 1036 root 27u IPv4 8604 0t0 UDP blackbox:kpasswd
named 1188 bind 512u IPv6 8730 0t0 UDP *:domain
named 1188 bind 513u IPv4 8735 0t0 UDP localhost:domain
named 1188 bind 514u IPv4 8737 0t0 UDP blackbox:domain
named 1188 bind 515u IPv4 10205 0t0 UDP 192.168.122.1:domain
dnsmasq 1313 libvirt-dnsmasq 5u IPv4 10229 0t0 UDP *:bootps
dnsmasq 1313 libvirt-dnsmasq 6u IPv4 10234 0t0 UDP 192.168.122.1:domain
leo@blackbox:~$ sudo lsof | grep TCP
samba 1029 root 22u IPv4 8536 0t0 TCP *:microsoft-ds (LISTEN)
samba 1029 root 23u IPv4 8537 0t0 TC...

[Jelmer Vernooij (jelmer)]

According to your lsof output there does seem to be a process listening on that port:

samba 1036 root 26u IPv4 8603 0t0 UDP blackbox:kerberos

Am I missing something?

[Leo Richard Comerford (lrc1)]

> Am I missing something?

Indeed.

> Speaking of which, I have discovered that starting the server with the Ethernet cable disconnected causes it to start with samba processes listening on port 88 UDP (UDP Kerberos) and port 389 UDP (UDP LDAP):

When the server is started with the network cable pulled out then there is a samba process listening on port 88 UDP. When the server is started with the network cable inserted then there is no process listening on port 88 UDP - see the lsof in comment #5.

[Jelmer Vernooij (jelmer)]

Is there nothing listening on *any* port 88/UDP ? or is it just on a different IP?

Is there any output in the log file?

[Leo Richard Comerford (lrc1)]

> Is there nothing listening on *any* port 88/UDP ? or is it just on a different IP?

Nothing on port 88/UDP for any IP, if I'm understanding lsof correctly. Likewise nothing on any port 389/UDP:

leo@blackbox:~$ sudo lsof | grep UDP
named 1114 bind 512u IPv6 8584 0t0 UDP *:domain
named 1114 bind 513u IPv4 8589 0t0 UDP localhost:domain
named 1114 bind 514u IPv4 8591 0t0 UDP blackbox:domain
named 1114 bind 515u IPv4 9501 0t0 UDP 192.168.122.1:domain
samba 1214 root 22u IPv4 8899 0t0 UDP *:netbios-ns
samba 1214 root 23u IPv4 8900 0t0 UDP *:netbios-dgm
samba 1214 root 24u IPv4 8901 0t0 UDP 10.37.55.255:netbios-ns
samba 1214 root 25u IPv4 8902 0t0 UDP blackbox:netbios-ns
samba 1214 root 26u IPv4 8903 0t0 UDP 10.37.55.255:netbios-dgm
samba 1214 root 27u IPv4 8904 0t0 UDP blackbox:netbios-dgm
dnsmasq 1243 libvirt-dnsmasq 5u IPv4 8936 0t0 UDP *:bootps
dnsmasq 1243 libvirt-dnsmasq 6u IPv4 8942 0t0 UDP 192.168.122.1:domain
leo@blackbox:~$

This is the same as the sudo lsof | grep UDP output provided in comment #5 - except that for some reason these IP/port combinations now no longer get (specific) listeners:

samba 1298 root 24u IPv4 7075 0t0 UDP 192.168.122.255:netbios-ns
samba 1298 root 25u IPv4 7076 0t0 UDP 192.168.122.1:netbios-ns
samba 1298 root 26u IPv4 7077 0t0 UDP 192.168.122.255:netbios-dgm
samba 1298 root 27u IPv4 7078 0t0 UDP 192.168.122.1:netbios-dgm

Again, this is when the machine has been started with the Ethernet cable plugged in: the lsof results when the system has been started with the cable plugged *out* are in comment #13.

(Also compare the sudo lsof | grep *TCP* output after a cable-in boot (comment #5) and a cable-out boot (comment #13): there are also TCP ports without listeners in the cable-in case.)

> Is there any output in the log file?

Nothing is written to /var/log/samba/log.samba when kinit is run, and nothing relevant is written to /var/log/syslog either. This is true regardless of whether the machine was last started with the Ethernet cable plugged out (and thus has samba processes listening on 88/UDP) or not. There *are* system-startup-time error messages in /var/log/samba/log.samba , and they are different depending on whether the Ethernet cable was plugged in or not. I'll upload those next, along with the corresponding /var/log/syslog entries.

[Launchpad Janitor (janitor)]

[Expired for samba4 (Ubuntu) because there has been no activity for 60 days.]