post ino64: lockd no runs?

[Xin LI]

On Mon, Jun 12, 2017 at 10:14 AM, John Baldwin <jhb at freebsd.org> wrote:
> On Sunday, June 11, 2017 11:12:25 AM David Wolfskill wrote:
>> On Sun, Jun 04, 2017 at 08:57:44AM -0400, Michael Butler wrote:
>> > It seems that {rpc.}lockd no longer runs after the ino64 changes on any
>> > of my systems after a full rebuild of src and ports. No log entries
>> > offer any insight as to why :-(
>> >
>> >     imb
>>
>> I don't tend to use NFS on my systems that are running head, so I
>> haven't had occasion to test this as stated.
>>
>> However, I just completed my weekly update of the "prooduction" systems
>> here at home, running stable/11.  And I find that lockd seems to be ...
>> claiming that all is well, but declining to run (for long).
>>
>> To the best of my knowledge, that was not the case until this last
>> update, which was from:
>>
>> FreeBSD albert.catwhisker.org 11.1-PRERELEASE FreeBSD 11.1-PRERELEASE #316  r319566M/319569:1100514: Sun Jun  4 03:54:41 PDT 2017     root at freebeast.catwhisker.org:/common/S1/obj/usr/src/sys/ALBERT  amd64
>>
>> to
>>
>> FreeBSD albert.catwhisker.org 11.1-BETA1 FreeBSD 11.1-BETA1 #322  r319823M/319823:1100514: Sun Jun 11 03:56:10 PDT 2017     root at freebeast.catwhisker.org:/common/S1/obj/usr/src/sys/ALBERT  amd64
>>
>> The "glaringly obvious" symptom in my case is that I am now unable
>> to (directly) save an email message from within mutt(1) by appending
>> it to an NFS-resident file.  (Saving it to a local file, then using
>> cat(1) to append that to the NFS- resident file & removing the local
>> copy works....)
>>
>> After a few variations on a theme of:
>>
>> albert(11.1)[5] sudo service lockd restart
>> lockd not running?
>> Starting lockd.
>> albert(11.1)[6] echo $?
>> 0
>> albert(11.1)[7] service lockd status
>> lockd is not running.
>>
>> I finally(!) thought to ask ktrace what's going on (as tailing
>> /var/log/messages was completely unproductive, even after enabling
>> rc_debug).
>>
>> So I tried: "sudo ktrace -di service lockd restart"; upon exanimation of
>> the output of kdump(1), I see that the trace ends with:
>>
>>   ...
>>   2811 rpc.lockd NAMI  "/var/run/logpriv"
>>   2786 sh       CALL  read(0xa,0x627fc0,0x400)
>>   2786 sh       GIO   fd 10 read 0 bytes
>>        ""
>>   2811 rpc.lockd RET   connect 0
>>   2786 sh       RET   read 0
>>   2811 rpc.lockd CALL  sendto(0x3,0x7fffffffe2c0,0x27,0,0,0)
>>   2786 sh       CALL  exit(0)
>>   2811 rpc.lockd GIO   fd 3 wrote 39 bytes
>>        "<30>Jun 11 15:43:10 rpc.lockd: Starting"
>>   2811 rpc.lockd RET   sendto 39/0x27
>>   2811 rpc.lockd CALL  sigaction(SIGALRM,0x7fffffffec20,0)
>>   2811 rpc.lockd RET   sigaction 0
>>   2811 rpc.lockd CALL  nlm_syscall(0,0x1e,0x4,0x801015040)
>>   2811 rpc.lockd RET   nlm_syscall -1 errno 14 Bad address
>
> This is a really good clue.  nlm_syscall is dying with EFAULT.  The last
> argument is a pointer to an array of char * pointers, and the only way
> I can see it dying is if it fails to copyin() one of the strings pointed
> to by those pointers.  You could try running rpc.lockd under gdb from
> ports and setting a breakpoint on 'nlm_syscall' and then printing out
> 'addr_count' and 'p addrs@(addr_count * 2)'.

Yes, I found that the kernel was trying to copyin() from NULL, and
then found that corresponds to 'uaddr'.  After some tracing I found
that the tightened condition for taddr2uaddr have enforced (correctly)
buffer length passed from caller, which was not set correctly since ~9
years ago (r177633, which sets the size to sizeof(pointer)) but never
gets noticed because there is no check on that, so the solution seems
to be to correctly set the length values to (allocated size), and that
have fixed the issue for me.

The code could use some cleanups and I plan to do it at some later time.

> Unfortunately I'm not able to reproduce the failure on a test machine
> I have running head post-ino64.

This should have been fixed by r319852 in -HEAD (
https://svnweb.freebsd.org/base?view=revision&revision=319852 ), and
I'll MFC the change after 3 days' settle  assuming there is no
objections, as this is a regression.

Cheers,