misc/181834: amd mounting NFS directories can drive a dead-lock

Julien Charbon jcharbon at verisign.com
Thu Sep 5 09:50:00 UTC 2013

>Number:         181834
>Category:       misc
>Synopsis:       amd mounting NFS directories can drive a dead-lock
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Sep 05 09:50:00 UTC 2013
>Originator:     Julien Charbon
>Release:        FreeBSD 8.4 RELEASE
FreeBSD atlas4 8.4-RELEASE-p3 FreeBSD 8.4-RELEASE-p3 #0 r+efa3d77-dirty: Wed Sep  4 20:41:53 UTC 2013     root at atlas4:/app/jcharbon/git/freebsd-vrsn/sys/GENERIC  amd64
Short Summary:

On FreeBSD 8.4, the amd auto-mounter daemon can drive a machine dead-lock when mounting NFS directories.

Long Summary:

If amd daemon starts, the machine appears dead-locked right after:

- ssh connection are stalled
- virtual consoles are stalled
- serial console is stalled

However machine still replies to ping, thus the kernel is still alive.  Launching DDB kernel debugger via hardware NMI button during dead-lock gave us this status:

syslogd and devd are waiting for the Giant kernel lock:

db> show allchains
chain 1:
   thread 100171 (pid 885, syslogd) blocked on lock 0xffffffff80e2e100 
(sleep mutex) "Giant"
   thread 100148 (pid 1120, amd) running on CPU 9
chain 2:
   thread 100205 (pid 742, devd) blocked on lock 0xffffffff80e2e100 
(sleep mutex) "Giant"
   thread 100148 (pid 1120, amd) running on CPU 9

 which is owned by amd deamon

db> show lock Giant
   class: sleep mutex
   name: Giant
   flags: {DEF, RECURSE}
   owner: 0xffffff003ef0f8e0 (tid 100148, pid 1120, "amd")
   recursed: 1
   An other backstrace with the witness kernel (kernel-witness):

  by the way this amd thread also owns other kernel mutexes:

db> show alllocks
Process 1120 (amd) thread 0xffffff003ef0f8e0 (100148)
exclusive rw udpinp (udpinp) r = 0 (0xffffff007b39fa60) locked @ 
exclusive rw udp (udp) r = 0 (0xffffffff80ff4d28) locked @ 
exclusive lockmgr nfs (nfs) r = 0 (0xffffff007b1bd7e8) locked @ 
exclusive sleep mutex Giant (Giant) r = 1 (0xffffffff80e2e100) locked @ 

 Next we launch DDB directly from kernel NFS code which gave us as backstrace:

Tracing pid 1142 tid 100301 td 0xffffff00379098e0
kvprintf() at kvprintf+0x17a
nfs_msg() at nfs_msg+0x52
nfs_feedback() at nfs_feedback+0x105
clnt_reconnect_call() at clnt_reconnect_call+0x19b
nfs_request() at nfs_request+0x1e5
nfs_getattr() at nfs_getattr+0x2bc
mountnfs() at mountnfs+0x330
nfs_mount() at nfs_mount+0xe3f
vfs_donmount() at vfs_donmount+0xcde
kernel_mount() at kernel_mount+0xa1
nfs_cmount() at nfs_cmount+0x5a
mount() at mount+0x1ea
amd64_syscall() at amd64_syscall+0xf9
Xfast_syscall() at Xfast_syscall+0xfc
--- syscall (21, FreeBSD ELF64, mount), rip = 0x8007cea4c, rsp = 
0x7fffffffdd88, rbp = 0x2 ---
db> next
db> next
db> next

  And many 'next' debugger commands later:  We are back in the same place in 

 At that point, the mountnfs() call will just loop infinitely in nfs_request() function and never releases kernel Giant and (nfs) kernel mutexes.
Using these amd's files: /etc/amd.conf:

$ cat /etc/amd.conf
browsable_dirs = no
map_type = file
mount_type = nfs
search_path = /etc
auto_dir = /.amd
cache_duration = 30
log_file = syslog:daemon
log_options = fatal,error
print_pid = yes
pid_file = /var/run/amd.pid
restart_mounts = yes
selectors_in_defaults = no

map_name = /etc/home.map

and /etc/home.map:

$ cat /etc/home.map
/defaults type:=nfs;opts:=tcp,intr,nosuid;rhost:=
* rfs:=/dev/${key};fs:=${autodir}/nfs/home/${key}

Just a:

# /etc/rc.d/amd onestart

will drive the dead-lock.


More information about the freebsd-bugs mailing list