From nobody Mon May 31 14:46:25 2021 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id F0B91D7AC14 for ; Mon, 31 May 2021 14:46:28 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-lf1-x130.google.com (mail-lf1-x130.google.com [IPv6:2a00:1450:4864:20::130]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Ftym41qv0z3Jrq; Mon, 31 May 2021 14:46:28 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: by mail-lf1-x130.google.com with SMTP id j10so17154869lfb.12; Mon, 31 May 2021 07:46:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=dFin8jdXmSk3anxzUzM3A87VSEUsMEIO+vvobdAoyB8=; b=R5IPwgvrxiG/fP8ivWk71Dz0ugLFkeBrlfhNoQwf9jehB6uPhJ//4bQHLB7RvK4pRk Q013hb8OxOMAsz/5R2yVPYjcEy8SrlHeA1tsQ3rEtJ4u7PeopBBTvC+KPqyfPBSmHzdO ASo7wePO9jTXfbbFqMD5ekZeIYHFIxoEgkl4IdbSA7EdgWRZE2vFjnKnCfTKZddFhVsL HF5/ihb4df0QGG9JOcp7p6gVtDj79rSFwkanXyHC+bO1r7qstmEj4zJdtuK+t8n5FwHx df5iBP5nYB04c12GaBhD3EomQqWgZHK+Ms6VomSHyW6vGLwD52Ak2pdwKpgONjl1NawW E/0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=dFin8jdXmSk3anxzUzM3A87VSEUsMEIO+vvobdAoyB8=; b=PAcNazf5v1S1HaS6oMidOJtZ2G1k+Wm1jW3kcy/wLiZG22VCYYfE1xV9/GuL+BBn/1 safo89tVVHd7CCH9fD1ZIeCVp8PR36rThufSE9v631EToPUz/TjLRZxd7K4ubSlPCvOL 8EZqDOvVZBP79DkW/v3mppyc78BikcAm+AKFZI8RDRRLM3AnjvU5+OhqmYawmAluIaCo TVRBdqtnVec1eHyWkMPvQnHfGYXo4LM2jjO8xb0jfO88Eo2rkg/crFEFZkowgIHkLUqS +fECef4j4cQEmLDKaCyfd1ngIqC7MIaQRaVM9G0W6MIY/QlAAH1fiGIy0YMV66P66bls Aryg== X-Gm-Message-State: AOAM531ReDV8/l1m5vFlAbdZWjgcRcyfQtK+1Ln/7pL6YvMfl441H1Mg 0XL5Iv1BGj1X3UTPBUjYd4I+OOKL9bOMimnoj+oB8Qd4 X-Google-Smtp-Source: ABdhPJw/qTjxtvL5Vua+dWlO6Ka//JeTlmSwL53Js1GDCa0ndXMi4W78hhKXr/YdR+On8fnTTD6CHAhslskJmQQr0qc= X-Received: by 2002:a19:da12:: with SMTP id r18mr12181845lfg.549.1622472386556; Mon, 31 May 2021 07:46:26 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Received: by 2002:a2e:98c4:0:0:0:0:0 with HTTP; Mon, 31 May 2021 07:46:25 -0700 (PDT) In-Reply-To: References: From: Mateusz Guzik Date: Mon, 31 May 2021 16:46:25 +0200 Message-ID: Subject: Re: Panics in recent NFS server To: Dimitry Andric Cc: FreeBSD Current , Rick Macklem Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4Ftym41qv0z3Jrq X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=R5IPwgvr; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2a00:1450:4864:20::130 as permitted sender) smtp.mailfrom=mjguzik@gmail.com X-Spamd-Result: default: False [-2.11 / 15.00]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::130:from]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_SPAM_SHORT(0.89)[0.886]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::130:from:127.0.2.255]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::130:from]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-current]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: N On 5/31/21, Mateusz Guzik wrote: > It's probably my commit d81aefa8b7dd8cbeffeda541fca9962802404983 , > I'll look at this later. Well let me rephrase. While the panic was added in said commit, I suspect the bug is on nfs side -- it has its own namei variant which I suspect is managing ni_pathlen in a manner different than the original, it just happens to not panic on kernels prior to the above change. > > On 5/31/21, Dimitry Andric wrote: >> Hi, >> >> I recently upgraded a -CURRENT NFS server from 2021-05-12 to today >> (2021-05-31), and when the first NFS client attempted to connect, I got >> this >> panic: >> >> panic: lookup: expected nul at 0xfffff800104b3002; string [dim] >> >> cpuid = 0 >> time = 1622463863 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >> 0xfffffe00747e89b0 >> vpanic() at vpanic+0x187/frame 0xfffffe00747e8a10 >> panic() at panic+0x43/frame 0xfffffe00747e8a70 >> lookup() at lookup+0xad2/frame 0xfffffe00747e8b10 >> nfsvno_namei() at nfsvno_namei+0x1a4/frame 0xfffffe00747e8bc0 >> nfsrvd_lookup() at nfsrvd_lookup+0x191/frame 0xfffffe00747e8eb0 >> nfsrvd_dorpc() at nfsrvd_dorpc+0xfab/frame 0xfffffe00747e90c0 >> nfssvc_program() at nfssvc_program+0x604/frame 0xfffffe00747e92a0 >> svc_run_internal() at svc_run_internal+0xa72/frame 0xfffffe00747e93d0 >> svc_run() at svc_run+0x250/frame 0xfffffe00747e9430 >> nfsrvd_nfsd() at nfsrvd_nfsd+0x33c/frame 0xfffffe00747e9590 >> nfssvc_nfsd() at nfssvc_nfsd+0x473/frame 0xfffffe00747e9aa0 >> sys_nfssvc() at sys_nfssvc+0xc7/frame 0xfffffe00747e9ac0 >> amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe00747e9bf0 >> fast_syscall_common() at fast_syscall_common+0xf8/frame >> 0xfffffe00747e9bf0 >> --- syscall (155, FreeBSD ELF64, sys_nfssvc), rip = 0x8011aa59a, rsp = >> 0x7fffffffe4e8, rbp = 0x7fffffffe780 --- >> KDB: enter: panic >> >> __curthread () >> at /share/dim/src/freebsd/src-dim/sys/amd64/include/pcpu_aux.h:55 >> 55 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu, >> (kgdb) #0 __curthread () >> at /share/dim/src/freebsd/src-dim/sys/amd64/include/pcpu_aux.h:55 >> #1 doadump (textdump=textdump@entry=0) >> at /share/dim/src/freebsd/src-dim/sys/kern/kern_shutdown.c:399 >> #2 0xffffffff804cca5a in db_dump (dummy=, >> dummy2=, dummy3=, dummy4=) >> at /share/dim/src/freebsd/src-dim/sys/ddb/db_command.c:575 >> #3 0xffffffff804cc912 in db_command (last_cmdp=, >> cmd_table=, dopager=dopager@entry=1) >> at /share/dim/src/freebsd/src-dim/sys/ddb/db_command.c:482 >> #4 0xffffffff804cc58d in db_command_loop () >> at /share/dim/src/freebsd/src-dim/sys/ddb/db_command.c:535 >> #5 0xffffffff804cfd06 in db_trap (type=, code=> out>) >> at /share/dim/src/freebsd/src-dim/sys/ddb/db_main.c:270 >> #6 0xffffffff80c69f17 in kdb_trap (type=type@entry=3, code=code@entry=0, >> tf=tf@entry=0xfffffe00747e88e0) >> at /share/dim/src/freebsd/src-dim/sys/kern/subr_kdb.c:727 >> #7 0xffffffff810d7aee in trap (frame=0xfffffe00747e88e0) >> at /share/dim/src/freebsd/src-dim/sys/amd64/amd64/trap.c:576 >> #8 >> #9 kdb_enter (why=0xffffffff812d3d27 "panic", msg=) >> at /share/dim/src/freebsd/src-dim/sys/kern/subr_kdb.c:506 >> #10 0xffffffff80c1d248 in vpanic ( >> fmt=0xffffffff8129dfef "%s: expected nul at %p; string [%s]\n", >> ap=, ap@entry=0xfffffe00747e8a50) >> at /share/dim/src/freebsd/src-dim/sys/kern/kern_shutdown.c:907 >> #11 0xffffffff80c1cfd3 in panic ( >> fmt=0xffffffff81e9b9c8 "=\t)\201\377\377\377\377") >> at /share/dim/src/freebsd/src-dim/sys/kern/kern_shutdown.c:843 >> #12 0xffffffff80cfa992 in lookup (ndp=ndp@entry=0xfffffe00747e8d90) >> at /share/dim/src/freebsd/src-dim/sys/kern/vfs_lookup.c:919 >> #13 0xffffffff80b33f84 in nfsvno_namei (nd=nd@entry=0xfffffe00747e9100, >> ndp=ndp@entry=0xfffffe00747e8d90, dp=, >> dp@entry=0xfffff80010544380, islocked=, >> islocked@entry=0, >> exp=exp@entry=0xfffffe00747e8fd8, p=p@entry=0xfffffe00bbfa3000, >> retdirp=0xfffffe00747e8e78) >> at /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdport.c:597 >> #14 0xffffffff80b266a1 in nfsrvd_lookup (nd=0xfffffe00747e9100, >> isdgram=, dp=0xfffff80010544380, >> vpp=0xfffffe00747e9010, >> fhp=0xfffffe00747e9074, exp=0xfffffe00747e8fd8) >> at /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdserv.c:607 >> #15 0xffffffff80b1073b in nfsrvd_compound (nd=0xfffffe00747e9100, >> isdgram=0, >> tag=0xf , taglen=6, >> minorvers=4294967294) >> at >> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdsocket.c:1098 >> #16 nfsrvd_dorpc (nd=nd@entry=0xfffffe00747e9100, >> isdgram=isdgram@entry=0, >> tag=0xf , taglen=6, >> minorvers=4294967294) >> at >> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdsocket.c:626 >> #17 0xffffffff80b24c44 in nfs_proc (nd=0xfffffe00747e9100, >> xid=, xprt=0xfffff80003a14e00, rpp=) >> at /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdkrpc.c:402 >> #18 nfssvc_program (rqst=0xfffff80010455800, xprt=0xfffff80003a14e00) >> at /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdkrpc.c:288 >> #19 0xffffffff80edead2 in svc_executereq (rqstp=0xfffff80010455800) >> at /share/dim/src/freebsd/src-dim/sys/rpc/svc.c:1037 >> #20 svc_run_internal (grp=, grp@entry=0xfffff800100e6100, >> ismaster=ismaster@entry=1) >> at /share/dim/src/freebsd/src-dim/sys/rpc/svc.c:1313 >> #21 0xffffffff80eddf80 in svc_run (pool=) >> at /share/dim/src/freebsd/src-dim/sys/rpc/svc.c:1392 >> #22 0xffffffff80b251ec in nfsrvd_nfsd (td=, >> td@entry=0xfffffe00bbfa3000, args=args@entry=0xfffffe00747e9660) >> at /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdkrpc.c:561 >> #23 0xffffffff80b3ec93 in nfssvc_nfsd (td=0xfffffe00bbfa3000, >> uap=) >> at >> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdport.c:3714 >> #24 0xffffffff80e6f647 in sys_nfssvc (td=0xfffffe00bbfa3000, >> uap=0xfffffe00bbfa33e8) >> at /share/dim/src/freebsd/src-dim/sys/nfs/nfs_nfssvc.c:111 >> #25 0xffffffff810d891e in syscallenter (td=) >> at >> /share/dim/src/freebsd/src-dim/sys/amd64/amd64/../../kern/subr_syscall.c:189 >> #26 amd64_syscall (td=0xfffffe00bbfa3000, traced=0) >> at /share/dim/src/freebsd/src-dim/sys/amd64/amd64/trap.c:1156 >> #27 >> #28 0x00000008011aa59a in ?? () >> >> Is anybody seeing this too? :) >> >> I can probably bisect, but it'll take quite a while. >> >> -Dimitry >> >> > > > -- > Mateusz Guzik > -- Mateusz Guzik