[Bug 250770] AWS EC2 system freezes up possibly associated with NFS (EFS)

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=250770

--- Comment #2 from Gunther Schadow <raj at gusw.net> ---
Before anyone spends time on this, I am still trying some options.

For now I have made 3 modification in my NFS mount options:

  1. timeo=150 down from 600 - that alone didn't help
  2. oneopenown - that alone didn't help
  3. minorversion=1 added, the manual isn't clear if it is a requirement for
oneopenown or if it implicitly sets minorversion=1, but since I added that, I
have not frozen up for almost 6 hours. That's new.

I also enabled crude NFS logging with 

  sysctl vfs.nfs.debuglevel=4

And this is what I see in the /var/log/messages:

Nov  1 08:59:23 freebsd su[926]: ec2-user to root on /dev/pts/1
Nov  1 10:28:20 freebsd su[1271]: ec2-user to root on /dev/pts/2
Nov  1 10:28:38 freebsd kernel: clnt call=0
Nov  1 10:28:43 freebsd syslogd: last message repeated 8 times
Nov  1 10:28:43 freebsd kernel: readrpc: aft doiods=5
Nov  1 10:28:43 freebsd kernel: clnt call=0
Nov  1 10:29:14 freebsd syslogd: last message repeated 1125 times
Nov  1 10:31:20 freebsd syslogd: last message repeated 63 times
Nov  1 10:36:05 freebsd syslogd: last message repeated 368 times
Nov  1 10:36:05 freebsd kernel: readrpc: aft doiods=5
Nov  1 10:36:05 freebsd kernel: clnt call=0
...
Nov  1 13:25:37 freebsd kernel: clnt call=0
Nov  1 13:25:37 freebsd kernel: readrpc: aft doiods=5
Nov  1 13:25:37 freebsd kernel: clnt call=0
Nov  1 13:25:37 freebsd syslogd: last message repeated 6 times
Nov  1 13:25:37 freebsd kernel: readrpc: aft doiods=5
Nov  1 13:25:37 freebsd kernel: clnt call=0
Nov  1 13:25:37 freebsd kernel: readrpc: aft doiods=5
Nov  1 13:25:37 freebsd kernel: clnt call=0
Nov  1 13:25:37 freebsd kernel: readrpc: aft doiods=5
Nov  1 13:25:37 freebsd kernel: clnt call=0
Nov  1 13:25:37 freebsd syslogd: last message repeated 6 times
Nov  1 13:25:37 freebsd kernel: readrpc: aft doiods=5
Nov  1 13:25:37 freebsd kernel: clnt call=0
Nov  1 13:25:37 freebsd syslogd: last message repeated 6 times
Nov  1 13:25:37 freebsd kernel: readrpc: aft doiods=5
Nov  1 13:25:37 freebsd kernel: clnt call=0
Nov  1 13:25:37 freebsd kernel: readrpc: aft doiods=5
Nov  1 13:25:37 freebsd kernel: clnt call=0
Nov  1 13:25:56 freebsd syslogd: last message repeated 53 times
Nov  1 13:25:56 freebsd kernel: readrpc: aft doiods=5
Nov  1 13:25:56 freebsd kernel: clnt call=0
Nov  1 13:26:27 freebsd syslogd: last message repeated 80 times
Nov  1 13:26:55 freebsd syslogd: last message repeated 204 times
Nov  1 13:26:55 freebsd kernel: readrpc: aft doiods=5
Nov  1 13:26:55 freebsd kernel: clnt call=0
Nov  1 13:27:36 freebsd syslogd: last message repeated 58 times
Nov  1 13:29:26 freebsd syslogd: last message repeated 87 times
Nov  1 13:30:47 freebsd syslogd: last message repeated 78 times
Nov  1 13:30:47 freebsd kernel: readrpc: aft doiods=5
Nov  1 13:30:47 freebsd kernel: clnt call=0
Nov  1 13:31:21 freebsd syslogd: last message repeated 26 times
Nov  1 13:33:00 freebsd syslogd: last message repeated 58 times
Nov  1 13:43:21 freebsd syslogd: last message repeated 119 times
Nov  1 13:53:06 freebsd syslogd: last message repeated 277 times
Nov  1 14:03:00 freebsd syslogd: last message repeated 263 times
Nov  1 14:13:22 freebsd syslogd: last message repeated 216 times
Nov  1 14:23:06 freebsd syslogd: last message repeated 5110 times
Nov  1 14:33:00 freebsd syslogd: last message repeated 491 times
Nov  1 14:43:22 freebsd syslogd: last message repeated 120 times

nothing really meaningful to me. But I hope that if it does lock up, I will see
something peculiar. 

On the other hand, now on c5.large with:
 1. minorversion=1
 2. sysctl vfs.nfs.debuglevel=4

I have not locked up again. If I can make it a full day, I will take back these
3 changes one by one, first the debug logging, then the minorversion=1, to see
when the problem recurs.

-- 
You are receiving this mail because:
You are the assignee for the bug.