NFSv4 Kerberos mount from Linux
Karli Sjöberg
karli at inparadise.se
Thu Oct 11 11:52:05 UTC 2018
On Thu, 2018-10-11 at 10:36 +0200, Peter Eriksson wrote:
> Just a few comments (random brain-dump) in case someone else is
> having problems with NFS & Kerberos.
"Just a few comments", hot dog, that was a great write-up! It would be
great, I think, to have this on the FreeBSD wiki or in the Handbook
even, because I agree not many people use Kerberized NFS since it´s
such a hassle and not that much info about it.
/K
>
>
> We’ve been using NFSv4 with Kerberos from Linux clients here for many
> years (with Solaris-based NFS servers and MIT Kerberos) and lately
> using FreeBSD as the NFS server OS (in an Microsoft AD Kerberos
> environment).
>
> There are a few differences in behaviour (from a Solaris
> perspective), mainly regarding the “pseudo” NFSv4 filesystem but not
> something that can’t be handled. In the process of moving to FreeBSD
> based servers I’ve also been testing lots of different clients in
> order to make sure everything works as expected. Here are some notes:
>
> General stuff:
>
> 1. Have a _kerberos.my.zone.com DNS TXT record containing the
> Kerberos realm (nice to have)
> 2. Have a _nfsv4domain.my.zone.com <http://nfsv4domain.my.zone.com/>
> DNS TXT record containing the NFSv4 “domain” (not all clients use it,
> but it’s nice to have)
>
>
> * FreeBSD server (with ZFS filesystems), things below /export is NFS-
> shared as (for example) server:/staff/user1
>
> 1. /etc/exports (we _only_ allow sec=krb<various flavours> for home
> directories):
> V4: /export -sec=krb5:krb5i:krb5p
>
> Or (on a server also serving some (read-only) sec=sys filesystems
> below /exports)
> V4: /export -sec=krb5:krb5i:krb5p:sys
>
>
> 2. /etc/zfs/exports (generated from “set sharenfs=xxx” on the ZFS
> filesystems)
> Home-server:
> /export/staff -sec=krb5:krb5i:krb5p
> /export/staff/user1 -sec=krb5:krb5i:krb5p
> /export/staff/user2 -sec=krb5:krb5i:krb5p
> /export/students -sec=krb5:krb5i:krb5p
> /export/students/user3 -sec=krb5:krb5i:krb5p
>
> Software-server:
> /export/db/icons -sec=sys -ro
> /export/old/ifm/solaris-x86 -sec=krb5:krb5i:krb5p:sys -ro
> /export/old/ifm/windows-86 -sec=krb5:krb5i:krb5p:sys -ro
> /export/software -sec=krb5:krb5i:krb5p -ro
>
>
> 3. Make sure you have host/FQDN at REALM and nfs/FQDN at REALM Kerberos
> principals in /etc/krb5.keytab and that FQDN is listed in /etc/hosts
> like:
>
> 1.2.3.4 foo.bar.com <http://foo.bar.com/> foo
>
> 4. rc.conf (we have a lot of users in our AD so we have to use a
> large number for usermax, replace “liu.se <http://liu.se/>” with your
> NFSv4 “domain” for nfsuserd_flags)
>
> gssd_enable="YES"
> nfs_server_enable="YES"
> nfsv4_server_enable="YES"
> nfscbd_enable="YES"
> mountd_enable="YES"
> nfsuserd_enable="YES"
> nfsuserd_flags="-manage-gids -domain liu.se -usertimeout 10 -usermax
> 100000 16"
>
> 5. Make sure you use NTPD so the clock is correct.
>
>
> * All clients (Solaris 10, OmniOS, MacOS 10.12-10.14, FreeBSD 11.0-
> 11.2, CentOS 7, Debian 9, Ubuntu 17-18 tested):
>
> 1. Make sure FQDN is in /etc/hosts
>
> 2. Make sure you use NTPD so the clock is correct.
>
> 3. Have a “host/FQDN at REALM” Kerberos host principal in
> /etc/krb5.keytab (nfs or root is not needed for NFS-mounting to work)
>
> 4. We use a fairly default /etc/krb5.conf, sort of like:
>
> > [libdefaults]
> > default_realm = REALM
> > dns_lookup_realm = true
> >
> > ticket_lifetime = 24h
> > renew_lifetime = 7d
> > forwardable = true
> >
> > default_ccache_name = KEYRING:persistent:%{uid}
>
> KEYRING probably only works on Linux and there are some problems with
> KEYRING in Debian & Ubuntu since not everything in it supports it due
> to them using Heimdal instead of MIT (like for smbclient) but it
> mostly works. Works fine in CentOS 7 though - in general CentOS 7
> feels more “enterprise”-ready than Debian & Ubuntu. The old classic
> FILE-ccaches should work fine though.
>
> For mounting we use the automounter and a "executable map” (perl
> script) that looks up records in DNS (Hesiod-style) since the built-
> in Hesiod support in most automounters is a bit.. lacking. Works
> quite well. You can find the scripts we use here:
>
> http://www.grebo.net/~peter/nfs <
> http://www.grebo.net/~peter/nfs>
>
> (The dns-update scripts use data from an SQL database so probably
> isn’t directly usable to anybody else. We use the same SQL database
> to populate a locally developed BerkeleyDB-based NSS-database on each
> FreeBSD server in order to speed things up since AD/LDAP-looks with
> ~90k users and silly amounts of AD groups takes forever, even with
> cacheing).
>
> Some Linux-specific stuff:
>
> Packages needed:
>
> CentOS:
> - nfs-utils
> - libnfsidmap
> - nfs4-acl-tools
> - autofs
>
> Debian:
> - keyutils
> - nfs-kernel-server # rpc.idmapd needs this due to a bug in Debian
>
> Ubuntu:
> - keyutils
>
> Other nice-to have packages:
> - hesiod
> - autofs-hesiod
>
> Some settings to check for:
>
> /etc/default/nfs-common:
> NEED_IDMAPD=yes
> NEED_GSSD=yes
>
> /etc/idmapd.conf (replace “liu.se <http://liu.se/>” with your NFSv4
> “domain”):
> Domain=liu.se
>
> /etc/request-key.d/id_resolver.conf (should be there already if
> using a modern Linux and you’ve added the packages above):
> create id_resolver * * /usr/sbin/nfsidmap %k
> %d
>
>
> MacOS:
>
> Basically require the latest - 10.14 (Mojave) - for things to work
> smoothly. In theory 10.12 & 10.13 should work but there is some bug
> in them that causes the OS to panic when you try to use NFS &
> Kerberos. 10.11 and earlier doesn’t support good enough encryption
> for us… But with 10.14 you just need to get a Kerberos ticket and
> then you can mount things just fine.
>
> /etc/nfs.conf should contain (replace “liu.se <http://liu.se/>” with
> your NFSv4 “domain”):
> nfs.client.default_nfs4domain=liu.se <http://liu.se/>
>
>
>
> (There are a lot of problems you can run into with Microsofts AD
> implementation of Kerberos too that we’ve had to be fighting with,
> but that’s a whole other topic)
>
> - Peter
>
>
> > On 10 Oct 2018, at 23:47, Rick Macklem <rmacklem at uoguelph.ca>
> > wrote:
> >
> > Felix Winterhalter wrote:
> > On 10/4/18 5:21 PM, Rick Macklem wrote:
> > [stuff snipped]
> > > > > I am now trying to mount this directory as root first without
> > > > > having to
> > > > > deal with user keytabs or tickets.
> > > > >
> > > > > This works fine with -sec=sys and nfsv4.1 and nfsv3 and
> > > > > -sec=krb5p.
> > > > > This does not however work with nfsv4 and krb5p or any other
> > > > > krb5 flavor.
> > > >
> > > > Sorry, I'm not sure what you are saying here. Is it
> > > > 1 - no version of NFS works for krb5p or
> > > > 2 - NFSv4.1 works for krb5p, but NFSv4.0 does not or
> > > > 3 - only nfsv3 works for krb5p
> >
> > [snipped lots of text]
> > >
> > > #3 is indeed what was happening. I could mount with krb5p for
> > > nfsv3
> > > (which I was not aware was even doable) however nfsv4 would
> > > stubbornly
> > > refuse to do any mounting.
> >
> > Yes, RPCSEC_GSS was done by Sun for NFSv3 and it was a good fit,
> > since NFSv3
> > does not have any server state to maintain. As such, all RPCs are
> > atomic operations
> > done by users (which for Kerberized mounts must have a TGT in a
> > credential cache).
> >
> > NFSv4 wasn't really a good fit for the model, because the NFSv4
> > server maintains
> > lock state (NFSv4 Opens are a form of lock used by Windows at file
> > open time).
> > There are "state maintenance" operations that must be done by the
> > user doing
> > the mount (usually root), where they typically don't have a TGT in
> > a credential
> > cache.
> > --> The ugly solution for this is typically a host-based client
> > credential in a keytab
> > on the client. (Usually a principal like "
> > root/client-host.domain at REALM" or
> > "host/client-host.domain at REALM" or "
> > nfs/client-host.domain at REALM"
> > in the default keytab on the client.)
> >
> > > I have now after a lot of try and error figured out what I need
> > > to do in
> > > order to make it work.
> > >
> > > To start with I have kerberos credentials with both host/ and
> > > nfs/ on
> > > both client and server. Mounting nfsv4 shares with krb5p from a
> > > linux
> > > server has also worked in this context.
> >
> > Yes, I'm assuming that satisfied the host-based client credential
> > as I described
> > above.
> >
> > > I leave you to judge whether what I found out is intended
> > > behaviour or
> > > if something weird is going on.
> >
> > Yes, sounds like intended behaviour, since the client must have a
> > Kerberos
> > credential to use for the "state maintenance" operations that are
> > not done on
> > behalf of a user.
> >
> > > My exports file originally looked something like this:
> > >
> > > /nfsTests/ /nfsTests/testexport /nfsTests/otherexport
> > > -maproot=root
> > > -sec=krb5p clients
> > >
> > > V4: /nfsTests -sec=krb5p clients
> > >
> > > Which allowed me to do nfsv3 krb5p mounts but not nfsv4 krb5p
> > > mounts.
> > >
> > > Changing the exports file to this:
> > >
> > > /nfsTests/ /nfsTests/testexport /nfsTests/otherexport
> > > -maproot=root
> > > -sec=krb5p clients
> > >
> > > V4: /nfsTests -sec=krb5p,krb5i clients
> >
> > This suggests that there is a bug in the client, where it uses
> > krb5i instead of krb5p
> > at some point in the mounting process. (I have also seen cases
> > where the client
> > erroneously falls back on using sys at some point in the mounting
> > process.)
> > (You did mention before you were using the Linux client. If you are
> > using a FreeBSD
> > client, I would be interested in looking at this.)
> >
> > > Allows nfsv4 krb5p mounts to work for some reason I do not
> > > understand.
> > > Not setting the -sec option on the V4 line apparently defaults to
> > > -sec=sys and doesn't allow any krb5 mounts. I'm not sure that
> > > this is a
> > > good default as I wasn't even aware that the -sec option needed
> > > to be
> > > set on this line.
> >
> > In FreeBSD, defaults are meant to maintain backwards compatibility.
> > This means that
> > AUTH_SYS should work by default. Also, AUTH_SYS is what 99.9999% of
> > FreeBSD
> > NFS mounts still use, from what I've seen.)
> >
> > > I've got packet traces of the nfsv3 krb5 and krb5i mounts and
> > > I'll make
> > > traces of the two nfsv4 mount attempts and send them to you if
> > > you're
> > > interested. I'm still not sure what exactly is happening here.
> >
> > The successful one for NFSv4 might be interesting. If you look at
> > it in
> > wireshark, I suspect you'd find somewhere during the mount that it
> > did RPCs which were not krb5p and that would show why the addition
> > of krb5i made it work.
> >
> > I did suggest you start with -sec=sys:krb5:krb5i:krb5p and, once
> > that works,
> > remove the security flavours one at a time until the mount doesn't
> > work.
> > (Then you capture packets for the minimal case that does work and
> > look at
> > what security flavours the client is using for all RPCs done during
> > the mount.)
> >
> > You now know why almost no one uses Kerberized NFSv4 mounts.
> > Unfortunately, the NFSv4 working group has never gotten around to
> > a better solution. Discussion of a host based encryption technique
> > using
> > something like SSL has happened, but no one has gone beyond that.
> >
> > rick
> > _______________________________________________
> > freebsd-fs at freebsd.org <mailto:freebsd-fs at freebsd.org> mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs <
> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs>
> > To unsubscribe, send any mail to "
> > freebsd-fs-unsubscribe at freebsd.org <mailto:
> > freebsd-fs-unsubscribe at freebsd.org>"
>
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20181011/ed4e3680/attachment.sig>
More information about the freebsd-fs
mailing list