NFSv4 Kerberos mount from Linux
Peter Eriksson
peter at ifm.liu.se
Thu Oct 11 09:08:12 UTC 2018
Just a few comments (random brain-dump) in case someone else is having problems with NFS & Kerberos.
We’ve been using NFSv4 with Kerberos from Linux clients here for many years (with Solaris-based NFS servers and MIT Kerberos) and lately using FreeBSD as the NFS server OS (in an Microsoft AD Kerberos environment).
There are a few differences in behaviour (from a Solaris perspective), mainly regarding the “pseudo” NFSv4 filesystem but not something that can’t be handled. In the process of moving to FreeBSD based servers I’ve also been testing lots of different clients in order to make sure everything works as expected. Here are some notes:
General stuff:
1. Have a _kerberos.my.zone.com DNS TXT record containing the Kerberos realm (nice to have)
2. Have a _nfsv4domain.my.zone.com <http://nfsv4domain.my.zone.com/> DNS TXT record containing the NFSv4 “domain” (not all clients use it, but it’s nice to have)
* FreeBSD server (with ZFS filesystems), things below /export is NFS-shared as (for example) server:/staff/user1
1. /etc/exports (we _only_ allow sec=krb<various flavours> for home directories):
V4: /export -sec=krb5:krb5i:krb5p
Or (on a server also serving some (read-only) sec=sys filesystems below /exports)
V4: /export -sec=krb5:krb5i:krb5p:sys
2. /etc/zfs/exports (generated from “set sharenfs=xxx” on the ZFS filesystems)
Home-server:
/export/staff -sec=krb5:krb5i:krb5p
/export/staff/user1 -sec=krb5:krb5i:krb5p
/export/staff/user2 -sec=krb5:krb5i:krb5p
/export/students -sec=krb5:krb5i:krb5p
/export/students/user3 -sec=krb5:krb5i:krb5p
Software-server:
/export/db/icons -sec=sys -ro
/export/old/ifm/solaris-x86 -sec=krb5:krb5i:krb5p:sys -ro
/export/old/ifm/windows-86 -sec=krb5:krb5i:krb5p:sys -ro
/export/software -sec=krb5:krb5i:krb5p -ro
3. Make sure you have host/FQDN at REALM and nfs/FQDN at REALM Kerberos principals in /etc/krb5.keytab and that FQDN is listed in /etc/hosts like:
1.2.3.4 foo.bar.com <http://foo.bar.com/> foo
4. rc.conf (we have a lot of users in our AD so we have to use a large number for usermax, replace “liu.se <http://liu.se/>” with your NFSv4 “domain” for nfsuserd_flags)
gssd_enable="YES"
nfs_server_enable="YES"
nfsv4_server_enable="YES"
nfscbd_enable="YES"
mountd_enable="YES"
nfsuserd_enable="YES"
nfsuserd_flags="-manage-gids -domain liu.se -usertimeout 10 -usermax 100000 16"
5. Make sure you use NTPD so the clock is correct.
* All clients (Solaris 10, OmniOS, MacOS 10.12-10.14, FreeBSD 11.0-11.2, CentOS 7, Debian 9, Ubuntu 17-18 tested):
1. Make sure FQDN is in /etc/hosts
2. Make sure you use NTPD so the clock is correct.
3. Have a “host/FQDN at REALM” Kerberos host principal in /etc/krb5.keytab (nfs or root is not needed for NFS-mounting to work)
4. We use a fairly default /etc/krb5.conf, sort of like:
> [libdefaults]
> default_realm = REALM
> dns_lookup_realm = true
>
> ticket_lifetime = 24h
> renew_lifetime = 7d
> forwardable = true
>
> default_ccache_name = KEYRING:persistent:%{uid}
KEYRING probably only works on Linux and there are some problems with KEYRING in Debian & Ubuntu since not everything in it supports it due to them using Heimdal instead of MIT (like for smbclient) but it mostly works. Works fine in CentOS 7 though - in general CentOS 7 feels more “enterprise”-ready than Debian & Ubuntu. The old classic FILE-ccaches should work fine though.
For mounting we use the automounter and a "executable map” (perl script) that looks up records in DNS (Hesiod-style) since the built-in Hesiod support in most automounters is a bit.. lacking. Works quite well. You can find the scripts we use here:
http://www.grebo.net/~peter/nfs <http://www.grebo.net/~peter/nfs>
(The dns-update scripts use data from an SQL database so probably isn’t directly usable to anybody else. We use the same SQL database to populate a locally developed BerkeleyDB-based NSS-database on each FreeBSD server in order to speed things up since AD/LDAP-looks with ~90k users and silly amounts of AD groups takes forever, even with cacheing).
Some Linux-specific stuff:
Packages needed:
CentOS:
- nfs-utils
- libnfsidmap
- nfs4-acl-tools
- autofs
Debian:
- keyutils
- nfs-kernel-server # rpc.idmapd needs this due to a bug in Debian
Ubuntu:
- keyutils
Other nice-to have packages:
- hesiod
- autofs-hesiod
Some settings to check for:
/etc/default/nfs-common:
NEED_IDMAPD=yes
NEED_GSSD=yes
/etc/idmapd.conf (replace “liu.se <http://liu.se/>” with your NFSv4 “domain”):
Domain=liu.se
/etc/request-key.d/id_resolver.conf (should be there already if using a modern Linux and you’ve added the packages above):
create id_resolver * * /usr/sbin/nfsidmap %k %d
MacOS:
Basically require the latest - 10.14 (Mojave) - for things to work smoothly. In theory 10.12 & 10.13 should work but there is some bug in them that causes the OS to panic when you try to use NFS & Kerberos. 10.11 and earlier doesn’t support good enough encryption for us… But with 10.14 you just need to get a Kerberos ticket and then you can mount things just fine.
/etc/nfs.conf should contain (replace “liu.se <http://liu.se/>” with your NFSv4 “domain”):
nfs.client.default_nfs4domain=liu.se <http://liu.se/>
(There are a lot of problems you can run into with Microsofts AD implementation of Kerberos too that we’ve had to be fighting with, but that’s a whole other topic)
- Peter
> On 10 Oct 2018, at 23:47, Rick Macklem <rmacklem at uoguelph.ca> wrote:
>
> Felix Winterhalter wrote:
> On 10/4/18 5:21 PM, Rick Macklem wrote:
> [stuff snipped]
>>>> I am now trying to mount this directory as root first without having to
>>>> deal with user keytabs or tickets.
>>>>
>>>> This works fine with -sec=sys and nfsv4.1 and nfsv3 and -sec=krb5p.
>>>> This does not however work with nfsv4 and krb5p or any other krb5 flavor.
>>> Sorry, I'm not sure what you are saying here. Is it
>>> 1 - no version of NFS works for krb5p or
>>> 2 - NFSv4.1 works for krb5p, but NFSv4.0 does not or
>>> 3 - only nfsv3 works for krb5p
> [snipped lots of text]
>>
>> #3 is indeed what was happening. I could mount with krb5p for nfsv3
>> (which I was not aware was even doable) however nfsv4 would stubbornly
>> refuse to do any mounting.
> Yes, RPCSEC_GSS was done by Sun for NFSv3 and it was a good fit, since NFSv3
> does not have any server state to maintain. As such, all RPCs are atomic operations
> done by users (which for Kerberized mounts must have a TGT in a credential cache).
>
> NFSv4 wasn't really a good fit for the model, because the NFSv4 server maintains
> lock state (NFSv4 Opens are a form of lock used by Windows at file open time).
> There are "state maintenance" operations that must be done by the user doing
> the mount (usually root), where they typically don't have a TGT in a credential
> cache.
> --> The ugly solution for this is typically a host-based client credential in a keytab
> on the client. (Usually a principal like "root/client-host.domain at REALM" or
> "host/client-host.domain at REALM" or "nfs/client-host.domain at REALM"
> in the default keytab on the client.)
>
>> I have now after a lot of try and error figured out what I need to do in
>> order to make it work.
>>
>> To start with I have kerberos credentials with both host/ and nfs/ on
>> both client and server. Mounting nfsv4 shares with krb5p from a linux
>> server has also worked in this context.
> Yes, I'm assuming that satisfied the host-based client credential as I described
> above.
>
>> I leave you to judge whether what I found out is intended behaviour or
>> if something weird is going on.
> Yes, sounds like intended behaviour, since the client must have a Kerberos
> credential to use for the "state maintenance" operations that are not done on
> behalf of a user.
>
>> My exports file originally looked something like this:
>>
>> /nfsTests/ /nfsTests/testexport /nfsTests/otherexport -maproot=root
>> -sec=krb5p clients
>>
>> V4: /nfsTests -sec=krb5p clients
>>
>> Which allowed me to do nfsv3 krb5p mounts but not nfsv4 krb5p mounts.
>>
>> Changing the exports file to this:
>>
>> /nfsTests/ /nfsTests/testexport /nfsTests/otherexport -maproot=root
>> -sec=krb5p clients
>>
>> V4: /nfsTests -sec=krb5p,krb5i clients
> This suggests that there is a bug in the client, where it uses krb5i instead of krb5p
> at some point in the mounting process. (I have also seen cases where the client
> erroneously falls back on using sys at some point in the mounting process.)
> (You did mention before you were using the Linux client. If you are using a FreeBSD
> client, I would be interested in looking at this.)
>
>> Allows nfsv4 krb5p mounts to work for some reason I do not understand.
>> Not setting the -sec option on the V4 line apparently defaults to
>> -sec=sys and doesn't allow any krb5 mounts. I'm not sure that this is a
>> good default as I wasn't even aware that the -sec option needed to be
>> set on this line.
> In FreeBSD, defaults are meant to maintain backwards compatibility. This means that
> AUTH_SYS should work by default. Also, AUTH_SYS is what 99.9999% of FreeBSD
> NFS mounts still use, from what I've seen.)
>
>> I've got packet traces of the nfsv3 krb5 and krb5i mounts and I'll make
>> traces of the two nfsv4 mount attempts and send them to you if you're
>> interested. I'm still not sure what exactly is happening here.
> The successful one for NFSv4 might be interesting. If you look at it in
> wireshark, I suspect you'd find somewhere during the mount that it
> did RPCs which were not krb5p and that would show why the addition
> of krb5i made it work.
>
> I did suggest you start with -sec=sys:krb5:krb5i:krb5p and, once that works,
> remove the security flavours one at a time until the mount doesn't work.
> (Then you capture packets for the minimal case that does work and look at
> what security flavours the client is using for all RPCs done during the mount.)
>
> You now know why almost no one uses Kerberized NFSv4 mounts.
> Unfortunately, the NFSv4 working group has never gotten around to
> a better solution. Discussion of a host based encryption technique using
> something like SSL has happened, but no one has gone beyond that.
>
> rick
> _______________________________________________
> freebsd-fs at freebsd.org <mailto:freebsd-fs at freebsd.org> mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org <mailto:freebsd-fs-unsubscribe at freebsd.org>"
More information about the freebsd-fs
mailing list