fbsd8_stable nfsv3 sys=krb5 issue

[Rick Macklem]

> Hi all,
> 
> I re-decided to move my nfs server from solaris to fbsd. So I am using
> test machines to see if it works. I have my kerberos realm configured,
> and seems to work fine, both nfsserver and nfsclient have their host
> and
> nfs keytabs stored in /etc/krb5.keytab files, and I am following the
> configuration instructions from
> http://code.google.com/p/macnfsv4/wiki/FreeBSD8KerberizedNFSSetup that
> rick was kind enough to write (thanx rick!). Before analyzing my
> problem
> and configuration steps further, let me state the reason for this
> email:
> I am not able to access an nfsv3 mounted filesystem when mounted with
> sys=krb5 (or krb5i, krb5p whatsoever) by following rick's
> instructions,
> whereas in the past I had no such problem.
> 
> To be more specific:
> Last time I was playing with the configuration most things worked fine
> (Feb 2010), but now things seem a bit different, and I am not sure
> whether I have forgotten something fundamental on my configuration or
> something has changed since I updated both my machines (client and
> server) to the latest sources:
> 
> nfs-server:
> # uname -a
> FreeBSD fbsdclient.ee.auth.gr 8.1-STABLE FreeBSD 8.1-STABLE #1: Wed
> Sep
> 15 17:07:13 EEST 2010
> root at fbsdclient.ee.auth.gr:/usr/obj/usr/src/sys/SERVER i386
> 
> nfs-client:
> # uname -a
> FreeBSD filesrv.ee.auth.gr 8.1-STABLE FreeBSD 8.1-STABLE #0: Fri Sep
> 10
> 13:08:06 EEST 2010
> root at filesrv.ee.auth.gr:/usr/obj/usr/src/sys/CLIENT amd64
> 
> I have my two usual test-users on my test-machines, mamalos and
> testakis, who both exist as kerberos principals too; their uids and
> gids
> are the same on all machines. I am able to kinit to any of them on my
> machines and acquire a valid kerberos ticket, which makes me assume
> that
> kdc runs nicely.
> 
> fbsd-client's /etc/rc.conf reads:
> 
> rpcbind_enable="YES"
> mountd_enable="YES"
> mountd_flags="-e"
> nfs_server_enable="YES"
> nfs_client_enable="YES"
> nfsv4_server_enable="YES"
> nfsuserd_enable="YES"
> gssd_enable="YES"
> 
> and fbsd-server's /etc/rc.conf reads:
> rpcbind_enable="YES"
> nfs_client_flags="-n 4"
> rpc_statd_enable="YES"
> rpc_lockd_enable="YES"
> #nfsd_flags="-e"
> gssd_enable="YES"
> nfsuserd_enable="YES"
> nfsclient_enable="YES"
> # nfs server
> nfs_server_enable="YES"
> mountd_enable="YES"
> #mountd_flags="-e"
> 
> 
> Don't get confused that both machines have nfsd enabled (the client is
> used as an experimental nfsv4 server too), and I think that this
> should
> not be an issue with regard to my problem (on the other hand, nobody
> knows...).
> 
> the server's kernel-config reads:
> 
> options KGSSAPI
> device crypto
> options NFSCL
> 
> and the client's kernel-config reads:
> 
> options NFSD #(don't forget that the client works as an nfsv4
> server too)
> options KGSSAPI
> device crypto
> 
> Lastly, the server's /etc/exports reads:
> /exports -alldirs -sec=krb5
> 
> on the server:
> # ls -la /exports
> total 10
> drwxr-xr-x 5 root wheel - 512 17 Feb 2010 ./
> drwxr-xr-x 22 root wheel - 512 15 Sep 19:33 ../
> drwxr-xr-x 3 root wheel - 512 5 Feb 2010 m/
> drwxr-xr-x 2 mamalos wheel - 512 16 Sep 15:43 mamalos/
> drwx------ 2 testakis wheel - 512 4 Feb 2010 testakis/
> 
> 
> 
> on the client:
> # klist
> klist: No ticket file: /tmp/krb5cc_0
> # mount mount_nfs -onfsv3,sec=krb5 server:/exports /mnt
> # mount
> /dev/da0s1a on / (ufs, local, soft-updates)
> devfs on /dev (devfs, local, multilabel)
> server:/exports on /mnt (nfs)
> # ls -la /mnt
> total 0
> ls: /mnt: Permission denied
> # exit
> $ id
> uid=1001(mamalos) gid=1001(mamalos) groups=1001(mamalos),0(wheel)
> $ klist
> klist: No ticket file: /tmp/krb5cc_1001
> $ ls -la /mnt
> total 0
> ls: /mnt: Permission denied
> $ kinit mamalos
> mamalos at EXAMPLE's Password:
> $ klist
> Credentials cache: FILE:/tmp/krb5cc_1001
> Principal: mamalos at EXAMPLE
> 
> Issued Expires Principal
> Sep 16 16:26:49 Sep 17 02:26:49 krbtgt/EXAMPLE at EXAMPLE
> $ ls -la /mnt
> total 0
> ls: /mnt: Permission denied
> ...
> (dooea?!?!?!!?)
> ...
> $ klist
> Credentials cache: FILE:/tmp/krb5cc_1001
> Principal: mamalos at EXAMPLE
> 
> Issued Expires Principal
> Sep 16 16:26:49 Sep 17 02:26:49 krbtgt/EXAMPLE at EXAMPLE
> Sep 16 16:27:51 Sep 17 02:26:49 nfs/server at EXAMPLE
> 

Normally the server will have a keytab entry for its
fully qualified domain name like:
nfs/fbsdclient.ee.auth.gr at EXAMPLE

and if that is the case, the client is expected to use
fbsdclient.ee.auth.gr as the server's name in the mount
and not "server" (which I assume is an alias for the above).

I'm definitely no kerberos wizard, but I'd guess that's where
the problem is?
(try "kinit -k nfs/fbsdclient.ee.auth.gr at EXAMPLE" on the server,
to see that the keytab works.)
The fully qualified domain name is used so that the keytab can't
be moved to a different client and made to work easily, although
a keytab entry is obviously weaker that a password based ccache
entry.

Beyond that, I'd suggest that you look in your KDC's logs to see
what it thinks is going on when you try to access the mount point.

rick
ps: Btw, I haven't forgotten about the issue w.r.t. kdestroy not
    invalidating the handle in the client so that access continues
    to work until the handle times out, but I haven't gotten around
    to fixing it.