From jhs at berklix.com Wed Jul 1 01:09:59 2009 From: jhs at berklix.com (Julian H. Stacey) Date: Wed Jul 1 01:10:05 2009 Subject: umount -f implementation In-Reply-To: Your message "Tue, 30 Jun 2009 14:58:39 PDT." <200906302158.n5ULwdxk002480@chez.mckusick.com> Message-ID: <200907010048.n610mPem058027@fire.js.berklix.net> Kirk McKusick wrote: > forced unmounts. The gentle force (-f) and the brute force (-F) > unmount. -F Would also be nice for devd.conf detach, for when people forget & pull a USB stick without unmounting first. Better a corrupt stick than a crashed OS. Cheers, Julian -- Julian Stacey: BSD Unix Linux C Sys. Eng. Consultant Munich http://berklix.com Mail in plain ASCII text, HTML & Base64 are spam. http://asciiribbon.org From dfilter at FreeBSD.ORG Wed Jul 1 12:50:06 2009 From: dfilter at FreeBSD.ORG (dfilter service) Date: Wed Jul 1 12:50:13 2009 Subject: kern/135412: commit references a PR Message-ID: <200907011250.n61Co51P034885@freefall.freebsd.org> The following reply was made to PR kern/135412; it has been noted by GNATS. From: dfilter@FreeBSD.ORG (dfilter service) To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/135412: commit references a PR Date: Wed, 1 Jul 2009 12:44:37 +0000 (UTC) Author: avg Date: Wed Jul 1 12:44:23 2009 New Revision: 195236 URL: http://svn.freebsd.org/changeset/base/195236 Log: MFC 185586 (kan): Change nfsserver slightly so that it does not trip over the timestamp validation code on ZFS. This should fix O_CREAT|O_EXCL open on NFS where a server is 64-bit with v13 ZFS code. PR: kern/135412 Pointed out by: Jaakko Heinonen Tested by: Jaakko Heinonen, Danny Braniss Modified: stable/7/sys/ (props changed) stable/7/sys/contrib/pf/ (props changed) stable/7/sys/nfsserver/nfs_serv.c Modified: stable/7/sys/nfsserver/nfs_serv.c ============================================================================== --- stable/7/sys/nfsserver/nfs_serv.c Wed Jul 1 12:36:10 2009 (r195235) +++ stable/7/sys/nfsserver/nfs_serv.c Wed Jul 1 12:44:23 2009 (r195236) @@ -1656,13 +1656,12 @@ nfsrv_create(struct nfsrv_descript *nfsd caddr_t bpos; int error = 0, rdev, len, tsize, dirfor_ret = 1, diraft_ret = 1; int v3 = (nfsd->nd_flag & ND_NFSV3), how, exclusive_flag = 0; - caddr_t cp; struct mbuf *mb, *mreq; struct vnode *dirp = NULL; nfsfh_t nfh; fhandle_t *fhp; u_quad_t tempsize; - u_char cverf[NFSX_V3CREATEVERF]; + struct timespec cverf; struct mount *mp = NULL; int tvfslocked; int vfslocked; @@ -1741,8 +1740,11 @@ nfsrv_create(struct nfsrv_descript *nfsd nfsm_srvsattr(vap); break; case NFSV3CREATE_EXCLUSIVE: - cp = nfsm_dissect_nonblock(caddr_t, NFSX_V3CREATEVERF); - bcopy(cp, cverf, NFSX_V3CREATEVERF); + tl = nfsm_dissect_nonblock(u_int32_t *, + NFSX_V3CREATEVERF); + /* Unique bytes, endianness is not important. */ + cverf.tv_sec = tl[0]; + cverf.tv_nsec = tl[1]; exclusive_flag = 1; break; }; @@ -1788,8 +1790,7 @@ nfsrv_create(struct nfsrv_descript *nfsd if (exclusive_flag) { exclusive_flag = 0; VATTR_NULL(vap); - bcopy(cverf, (caddr_t)&vap->va_atime, - NFSX_V3CREATEVERF); + vap->va_atime = cverf; error = VOP_SETATTR(nd.ni_vp, vap, cred, td); } @@ -1873,7 +1874,7 @@ nfsrv_create(struct nfsrv_descript *nfsd } if (v3) { if (exclusive_flag && !error && - bcmp(cverf, (caddr_t)&vap->va_atime, NFSX_V3CREATEVERF)) + bcmp(&cverf, &vap->va_atime, sizeof (cverf))) error = EEXIST; if (dirp == nd.ni_dvp) diraft_ret = VOP_GETATTR(dirp, &diraft, cred, td); _______________________________________________ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org" From rmacklem at uoguelph.ca Wed Jul 1 17:27:25 2009 From: rmacklem at uoguelph.ca (Rick Macklem) Date: Wed Jul 1 17:27:32 2009 Subject: umount -f implementation In-Reply-To: <200907010048.n610mPem058027@fire.js.berklix.net> References: <200907010048.n610mPem058027@fire.js.berklix.net> Message-ID: On Wed, 1 Jul 2009, Julian H. Stacey wrote: > Kirk McKusick wrote: >> forced unmounts. The gentle force (-f) and the brute force (-F) >> unmount. > > -F Would also be nice for devd.conf detach, for when people > forget & pull a USB stick without unmounting first. > Better a corrupt stick than a crashed OS. > All I'll add is, for the experimental nfs client, if both semantics are desired (and, imho they are), there will need to be separate flags to indicate whether or not to terminate RPCs in progress. So, it seems that there is interest in a separate "umount -F" to handle the case of failed storage (disk subsystem, NAS server down,...). Is there anyone who is opposed to my pursuing this after FreeBSD-CURRENT branches from 8? (I can do the experimental nfs client + some testing. Hopefully others can help with the generic VFS issues and other file systems.) rick, who obviously doesn't have as good a memory as Kirk's:-) ps: Unfortunately Solaris uses "-F" for something entirely different, so feel free to suggest other flag values if you think that is a concern. From linimon at FreeBSD.org Thu Jul 2 10:41:41 2009 From: linimon at FreeBSD.org (linimon@FreeBSD.org) Date: Thu Jul 2 10:41:53 2009 Subject: kern/136218: [zfs] Exported ZFS pools can't be imported into (Open)Solaris Message-ID: <200907021041.n62AfegV089453@freefall.freebsd.org> Old Synopsis: Exported ZFS pools can't be imported into (Open)Solaris New Synopsis: [zfs] Exported ZFS pools can't be imported into (Open)Solaris Responsible-Changed-From-To: freebsd-amd64->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Thu Jul 2 10:41:22 UTC 2009 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=136218 From rmacklem at uoguelph.ca Thu Jul 2 15:25:50 2009 From: rmacklem at uoguelph.ca (Rick Macklem) Date: Thu Jul 2 15:25:57 2009 Subject: documenting setup of Kerberized NFS Message-ID: I'm not even sure if this is the right place to ask the question, but... I'm wondering what would be an appropriate place to document how to set up/use Kerberized NFS? I'm far from a Kerberos wizard, but I have picked up some tricks that might be useful to others. At this point, I think it would be too informal for something like a man page. Maybe a wiki or similar? (Since wiki.freebsd.org is for developers and not users, it seems that isn't the right place.) Maybe I could start with just a brain dump posting to this list? (Basically anywhere that the search engines can find, would be a start.) Thanks in advance for any suggestions, rick From DS at praxisvermittlung24.de Fri Jul 3 12:29:50 2009 From: DS at praxisvermittlung24.de (Daniel Seuffert) Date: Fri Jul 3 12:30:02 2009 Subject: Subject: documenting setup of Kerberized NFS In-Reply-To: <20090703120023.7E40B1065713@hub.freebsd.org> References: <20090703120023.7E40B1065713@hub.freebsd.org> Message-ID: <4A4DF6D8.4040304@praxisvermittlung24.de> > From: Rick Macklem > Subject: documenting setup of Kerberized NFS > To: freebsd-fs@freebsd.org > Message-ID: > Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed > > I'm not even sure if this is the right place to ask the question, but... > > I'm wondering what would be an appropriate place to document how to > set up/use Kerberized NFS? I'm far from a Kerberos wizard, but I have > picked up some tricks that might be useful to others. At this point, > I think it would be too informal for something like a man page. > Maybe a wiki or similar? (Since wiki.freebsd.org is for developers and > not users, it seems that isn't the right place.) > > Maybe I could start with just a brain dump posting to this list? > (Basically anywhere that the search engines can find, would be a start.) > > Thanks in advance for any suggestions, rick Hi Rick, I suggest using a public wiki and adding it to http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/network-nfs.html once it is considered mature enough. Maybe freebsd-doc would be a better place to ask for feedback. Best regards, Daniel From rmacklem at uoguelph.ca Fri Jul 3 16:29:33 2009 From: rmacklem at uoguelph.ca (Rick Macklem) Date: Fri Jul 3 16:29:39 2009 Subject: Kerberized NFS doc (raw text before I get it on a wiki) Message-ID: Ok, here's the raw text that I plan on putting on a wiki as a starting point for documenting how to use set up Kerberized NFS on FreeBSD8. Any comments w.r.t. it are welcome. I'll probably put it: http://code.google.com/p/macnfsv4 under the wiki tab in a day or so. Hopefully it is some use, rick --- first draft of doc --- Setting up Kerberized NFS (more correctly: NFS using RPCSEC_GSS authentication via the kerberos5 mechanism) on FreeBSD8/CURRENT: Before we get started, here are a few preliminary things you need to know: (Lets assume the default Realm is CIS.UOGUELPH.CA with systems called nfs-client.cis.uoguelph.ca and nfs-server.cis.uoguelph.ca, plus a user called ricktst that exists in the password database on all systems with uid == 502.) In the GSSAPI there are two kinds of principal names, User and Host_Based. A User principal name normally refers to a user and looks like: ricktst <-- which becomes ricktst@CIS.UOGUELPH.CA in KerberosV It normally has credentials in a credentials cache file (/tmp/krb5cc_502) created by kinit or during the user's login process. A Host_Based principal name also has the fully qualified host domain name in it (so it can only be used on that host) and looks like: nfs@nfs-server.cis.uoguelph.ca <-- which becomes nfs/nfs-server.cis.uoguelph.ca@CIS.UOGUELPH.CA in KerberosV and normally has credentials in a keytab file created on the KDC and transferred to the system via some secure means. (Since this principal will only work on the system called nfs-server.cis.uoguelph.ca, the damage done if the keytab file is compromised on nfs-server.cis.uoguelph.ca is is limited to that system, which has already been compromised anyhow.) These principals have the advantage that their credentials can always be refreshed until a new keytab entry is generated for the principal on the KDC. For RPCSEC_GSS, there are 3 types of service which, for KerberosV are: krb5 - Use KerberosV for user authentication, but only protect the RPC header from compromise. krb5i - Use KerberosV for user authentication, but also use excrypted checksums on the RPC data to protect against "man in the middle" attacks involving replacement of the RPC data. krb5p - Use KerberosV for user authentication, but also encrypt the RPC data, so that it isn't on the wire in clear text. Setup: 1 - The client and server system(s) will need to be running kernels built with the following in their kernel config(5) files: options KGSSAPI device crypto 2 - KerberosV will need to be configured on all the systems, with them all in the same default REALM (I believe cross-REALM authentication isn't going to work well, since the uid<->user-principal-name translations won't work out). This can be confirmed via kinit working ok on them. 3 - All systems will need to be running the gssd daemon. The daemon can be started at boot time by setting: gssd_enable="YES" in the /etc/rc.conf file. 4 - The server(s) will need a host based entry for "nfs" in their default keytab file (see above). 5 - The server(s) will need "-sec" options added to the lines in /etc/exports for the file systems being exported for Kerberized NFS access. If AUTH_SYS access is to be allowed as well, "sys" must be specified too. For example: /exp -sec=sys:krb5:krb5i:krb5p nfs-client.cis.uoguelph.ca /exp -sec=krb5i:krb5p would allow nfs-client.cis.uoguelph.ca to use any authentication, but other clients would be required to use krb5i or krb5p. For NFSv4, the "V4:" line(s) will have to have the "-sec" option added, as well. This controls what authentication methods can be used for the system related operations that do not have any associated file handle. It might look like: V4: / -sec=sys:krb5:krb5i:krb5p (See "man exports" and "man nfsv4" for more info.) 5 - In the client, for NFSv3, the mount can usually be done by "root" using AUTH_SYS, since that is what most mountd servers handle. However, accesses to the file system may require that the users have valid TGTs in their credential cache file. For example (on the client): # mount -t nfs -o nfsv3,sec=krb5 nfs-server.cis.uoguelph.ca:/exp /mnt - where "root" does not have a TGT Then "ricktst" can: % kinit - enter KerberosV password when prompted % cd /mnt - and use the file system until the TGT expired (Exactly what is allowed is determined by the NFS server, as above.) For NFSv4, the situation is somewhat different, since there is no mountd protocol and there are system operations related to Opens and Locks that need to keep working until the file system is dismounted. There are basically two ways to do this (with a third variant): (A) - A user with a valid TGT may do the mount. For this case, the file system should be dismounted before the TGT expires. To do this: # sysctl vfs.usermounts=1 <-- done by root - Then, as the user logged in with a valid TGT % mount -t nfs -o nfsv4,sec=krb5i nfs-server.cis.uoguelph.ca:/exp mydir - where "mydir" is owned by the user - This will generate a warning about the mount table not being updated, but that isn't a serious problem imho. There are open source programs like "krenew" that can be used to help ensure that the TGT doesn't expire before dismounting one of these mounts. This is the only variant supported by FreeBSD8 "out of the box". (The entire mount path must be allowed to use the specified authentication flavour for the mount to work. In particular, Netapp filers tend to only allow AUTH_SYS (aka "sys") for the root directory by default and this must be changed for the mount to work. The FreeBSD8 NFSv4 client does not know how to switch authentication flavours dynamically, based on the server returning NFS4ERR_WRONGSEC (err# 10016).) (B) - Use a host based principal name in the default keytab file on the client, to allow the mount to be done by "root". This mount should continue to work, since the credentials for the host based principal in the default keytab file can continue to be refreshed. To do this form of NFSv4 mount, the kernel on the client will have to be patched with ftp://ftp.cis.uoguelph.ca/pub/nfsv4/freebsd-rpcsec.patch (available anonymous ftpable). It will also need an entry like: nfs/nfs-client.cis.uoguelph.ca@CIS.UOGUELPH.CA in the client machine's default keytab file. (For the client, the first component doesn't need to be "nfs". For example, Solaris10 typically uses "root".) Then you must set the sysctl variable vfs.rpcsec.keytab_enctype to the numeric value for the encryption type used when creating the keytab entry on the KDC. (The patch is not in FreeBSD8 partly because it requires this pesky business of setting the encryption type. I haven't figured out a way to make it work without doing gss_krb5_set_allowable_enctypes() calls to every client side credential acquired.) The numeric values for the encryption types look like: #define ETYPE_DES_CBC_CRC 1 and can be found in sys/kgssapi/krb5/kcrypto.h. (This must be done before the gssd daemon is started on the system.) Then, the mount command looks like: # mount -t nfs -o nfsv4,sec=krb5,gssname=nfs nfs-server.cis.uoguelph.ca:/exp /mnt Now, the system operations will be performed using the host based credential, but other operations will be performed using credentials for the appropriate user, such as ricktst, via their TGT. (This is the style used by the Solaris10 and Linux clients.) (C) - Is a variant of (B), where the "allgssname" mount option is specified, which means that all accesses to the file system use the host based principal name in the default keytab file. This case may be useful when the client is actually some kind of system that runs batch processes, where the user's are not normally logged with valid TGTs when running them. The mount command would look like: # mount -t nfs -o nfsv4,sec=krb5i,gssname=root,allgssname nfs-server.cis.uoguelph.ca:/exp /mnt Here are some gotchas to be aware of: - The time of day clocks for all systems must be synchronized to within the clock skew specified in /etc/krb5.conf. - KerberosV principal names are case sensitive, although DNS names are not. The simplest way to avoid grief is to use all lower case characters in your DNS host names and all upper case characters for your KerberosV REALM name. - The host name resolver functions must return the fully qualified host name i.e. nfs-server.cis.uoguelph.ca and not nfs-server. If you are using /etc/hosts, put the fully qualified name first, like: 131.104.48.99 nfs-server.cis.uoguelph.ca nfs-server - What to do w.r.t. ticket encryption types is beyond my limited KerberosV expertise, however if you stick with des-cbc-crc initially and get that working, you can try others later. I have been told that some Netapp filers only handle des-cbc-crc. (When creating keytab file entries on the KDC, be careful to specify the chosen encryption type and then set that type as the first/only type for the "default-etypes" entry in /etc/krb5.conf.) Also, if you are using the host based principal patch, be sure to set the encryption type used for the keytab entry, as above. If using dec-cbc-crc: # sysctl vfs.rpcsec.keytab_enctype=1 Good luck with it, rick From jflemer at alum.rpi.edu Sat Jul 4 03:00:19 2009 From: jflemer at alum.rpi.edu (James E. Flemer) Date: Sat Jul 4 03:00:26 2009 Subject: kern/124621: [ext3] [patch] Cannot mount ext2fs partition Message-ID: <200907040300.n6430JTf068924@freefall.freebsd.org> The following reply was made to PR kern/124621; it has been noted by GNATS. From: "James E. Flemer" To: paulf@free.fr, bug-followup@FreeBSD.org Cc: Subject: Re: kern/124621: [ext3] [patch] Cannot mount ext2fs partition Date: Fri, 3 Jul 2009 19:23:53 -0700 Environment: FreeBSD cage.local 7.1-STABLE FreeBSD 7.1-STABLE #1: Thu Jan 8 20:10:28 PST 2009 root@cage.local:/mnt/space/usr/obj.i386/mnt/space/usr/src/sys/CAGE7-SMP i386 Just another data point, the patch (http://pflog.net/~floyd/ext2fs.diff, md5:3dd5125eeb591e9c53930beb216d523e) fixes mounting a ext3 partition from an Ubuntu 9.4 install. Confirmed with tune2fs that the partition has 256-byte inode size. Tested by applying patch and rebuilding in /usr/src/sys/modules/ext2fs. From jilles at FreeBSD.org Sat Jul 4 21:26:51 2009 From: jilles at FreeBSD.org (jilles@FreeBSD.org) Date: Sat Jul 4 21:26:58 2009 Subject: kern/135412: [zfs] [nfs] zfs(v13)+nfs and open(..., O_WRONLY|O_CREAT|O_EXCL, ...) returns io error Message-ID: <200907042126.n64LQoBC025291@freefall.freebsd.org> Synopsis: [zfs] [nfs] zfs(v13)+nfs and open(..., O_WRONLY|O_CREAT|O_EXCL, ...) returns io error State-Changed-From-To: open->closed State-Changed-By: jilles State-Changed-When: Sat Jul 4 21:26:50 UTC 2009 State-Changed-Why: Fix committed, no other branches affected. http://www.freebsd.org/cgi/query-pr.cgi?pr=135412 From dan.naumov at gmail.com Sat Jul 4 21:56:22 2009 From: dan.naumov at gmail.com (Dan Naumov) Date: Sat Jul 4 21:56:28 2009 Subject: ZFS and df weirdness In-Reply-To: References: Message-ID: Hello list. I have a single 2tb disk used on a 7.2-release/amd64 system with a small part of it given to UFS and most of the disk given to a single "simple" zfs pool with several filesystems without redundancy. I've noticed a really weird thing regarding what "df" reports regarding the "total space" of one of my filesystems: atom# zpool list NAME ? ? ? ? ? ? ? ? ? ?SIZE ? ?USED ? AVAIL ? ?CAP ?HEALTH ? ? ALTROOT tank ? ? ? ? ? ? ? ? ? 1.80T ? ?294G ? 1.51T ? ?15% ?ONLINE ? ? - atom# zfs list NAME ? ? ? ? ? ? ?USED ?AVAIL ?REFER ?MOUNTPOINT tank ? ? ? ? ? ? ?294G ?1.48T ? ?18K ?none tank/DATA ? ? ? ? 292G ?1.48T ? 292G ?/DATA tank/home ? ? ? ? 216K ?1.48T ? ?21K ?/home tank/home/jago ? ?132K ?1.48T ? 132K ?/home/jago tank/home/karni ? ?62K ?1.48T ? ?62K ?/home/karni tank/usr ? ? ? ? 1.33G ?1.48T ? ?18K ?none tank/usr/local ? ?455M ?1.48T ? 455M ?/usr/local tank/usr/obj ? ? ? 18K ?1.48T ? ?18K ?/usr/obj tank/usr/ports ? ?412M ?1.48T ? 412M ?/usr/ports tank/usr/src ? ? ?495M ?1.48T ? 495M ?/usr/src tank/var ? ? ? ? ?320K ?1.48T ? ?18K ?none tank/var/log ? ? ?302K ?1.48T ? 302K ?/var/log atom# df Filesystem ? ? ?1K-blocks ? ? ? Used ? ? ?Avail Capacity ?Mounted on /dev/ad12s1a ? ? ?16244334 ? 1032310 ? 13912478 ? ? 7% ? ?/ devfs ? ? ? ? ? ? ? ? ? ?1 ? ? ? ? 1 ? ? ? ? ?0 ? 100% ? ?/dev linprocfs ? ? ? ? ? ? ? ?4 ? ? ? ? 4 ? ? ? ? ?0 ? 100% ? ?/usr/compat/linux/proc tank/DATA ? ? ? 1897835904 306397056 1591438848 ? ?16% ? ?/DATA tank/home ? ? ? 1591438848 ? ? ? ? 0 1591438848 ? ? 0% ? ?/home tank/home/jago ?1591438976 ? ? ? 128 1591438848 ? ? 0% ? ?/home/jago tank/home/karni 1591438848 ? ? ? ? 0 1591438848 ? ? 0% ? ?/home/karni tank/usr/local ?1591905024 ? ?466176 1591438848 ? ? 0% ? ?/usr/local tank/usr/obj ? ?1591438848 ? ? ? ? 0 1591438848 ? ? 0% ? ?/usr/obj tank/usr/ports ?1591860864 ? ?422016 1591438848 ? ? 0% ? ?/usr/ports tank/usr/src ? ?1591945600 ? ?506752 1591438848 ? ? 0% ? ?/usr/src tank/var/log ? ?1591439104 ? ? ? 256 1591438848 ? ? 0% ? ?/var/log atom# df -h Filesystem ? ? ? ? Size ? ?Used ? Avail Capacity ?Mounted on /dev/ad12s1a ? ? ? ?15G ? ?1.0G ? ? 13G ? ? 7% ? ?/ devfs ? ? ? ? ? ? ?1.0K ? ?1.0K ? ? ?0B ? 100% ? ?/dev linprocfs ? ? ? ? ?4.0K ? ?4.0K ? ? ?0B ? 100% ? ?/usr/compat/linux/proc tank/DATA ? ? ? ? ?1.8T ? ?292G ? ?1.5T ? ?16% ? ?/DATA tank/home ? ? ? ? ?1.5T ? ? ?0B ? ?1.5T ? ? 0% ? ?/home tank/home/jago ? ? 1.5T ? ?128K ? ?1.5T ? ? 0% ? ?/home/jago tank/home/karni ? ?1.5T ? ? ?0B ? ?1.5T ? ? 0% ? ?/home/karni tank/usr/local ? ? 1.5T ? ?455M ? ?1.5T ? ? 0% ? ?/usr/local tank/usr/obj ? ? ? 1.5T ? ? ?0B ? ?1.5T ? ? 0% ? ?/usr/obj tank/usr/ports ? ? 1.5T ? ?412M ? ?1.5T ? ? 0% ? ?/usr/ports tank/usr/src ? ? ? 1.5T ? ?495M ? ?1.5T ? ? 0% ? ?/usr/src tank/var/log ? ? ? 1.5T ? ?256K ? ?1.5T ? ? 0% ? ?/var/log Considering that every single filesystem is part of the exact same pool, with no custom options whatsoever used during filesystem creation (except for mountpoints), why is the size of tank/DATA 1.8T while the others are 1.5T? - Sincerely, Dan Naumov From jilles at FreeBSD.org Sat Jul 4 22:12:24 2009 From: jilles at FreeBSD.org (jilles@FreeBSD.org) Date: Sat Jul 4 22:12:30 2009 Subject: kern/104133: [ext2fs] EXT2FS module corrupts EXT2/3 filesystems Message-ID: <200907042212.n64MCNx9062206@freefall.freebsd.org> Synopsis: [ext2fs] EXT2FS module corrupts EXT2/3 filesystems State-Changed-From-To: feedback->open State-Changed-By: jilles State-Changed-When: Sat Jul 4 22:12:13 UTC 2009 State-Changed-Why: Requested feedback has been received. http://www.freebsd.org/cgi/query-pr.cgi?pr=104133 From gldisater at gmail.com Sun Jul 5 17:07:29 2009 From: gldisater at gmail.com (Jeremy Faulkner) Date: Sun Jul 5 17:07:36 2009 Subject: ZFS and df weirdness In-Reply-To: References: Message-ID: <4A50A02B.5060402@gmail.com> Dan Naumov wrote: > Hello list. > > I have a single 2tb disk used on a 7.2-release/amd64 system with a > small part of it given to UFS and most of the disk given to a single > "simple" zfs pool with several filesystems without redundancy. I've > noticed a really weird thing regarding what "df" reports regarding the > "total space" of one of my filesystems: > > atom# zpool list > NAME SIZE USED AVAIL CAP HEALTH ALTROOT > tank 1.80T 294G 1.51T 15% ONLINE - > > atom# zfs list > NAME USED AVAIL REFER MOUNTPOINT > tank 294G 1.48T 18K none > tank/DATA 292G 1.48T 292G /DATA > tank/home 216K 1.48T 21K /home > tank/home/jago 132K 1.48T 132K /home/jago > tank/home/karni 62K 1.48T 62K /home/karni > tank/usr 1.33G 1.48T 18K none > tank/usr/local 455M 1.48T 455M /usr/local > tank/usr/obj 18K 1.48T 18K /usr/obj > tank/usr/ports 412M 1.48T 412M /usr/ports > tank/usr/src 495M 1.48T 495M /usr/src > tank/var 320K 1.48T 18K none > tank/var/log 302K 1.48T 302K /var/log > > atom# df > Filesystem 1K-blocks Used Avail Capacity Mounted on > /dev/ad12s1a 16244334 1032310 13912478 7% / > devfs 1 1 0 100% /dev > linprocfs 4 4 0 100% /usr/compat/linux/proc > tank/DATA 1897835904 306397056 1591438848 16% /DATA > tank/home 1591438848 0 1591438848 0% /home > tank/home/jago 1591438976 128 1591438848 0% /home/jago > tank/home/karni 1591438848 0 1591438848 0% /home/karni > tank/usr/local 1591905024 466176 1591438848 0% /usr/local > tank/usr/obj 1591438848 0 1591438848 0% /usr/obj > tank/usr/ports 1591860864 422016 1591438848 0% /usr/ports > tank/usr/src 1591945600 506752 1591438848 0% /usr/src > tank/var/log 1591439104 256 1591438848 0% /var/log > > atom# df -h > Filesystem Size Used Avail Capacity Mounted on > /dev/ad12s1a 15G 1.0G 13G 7% / > devfs 1.0K 1.0K 0B 100% /dev > linprocfs 4.0K 4.0K 0B 100% /usr/compat/linux/proc > tank/DATA 1.8T 292G 1.5T 16% /DATA > tank/home 1.5T 0B 1.5T 0% /home > tank/home/jago 1.5T 128K 1.5T 0% /home/jago > tank/home/karni 1.5T 0B 1.5T 0% /home/karni > tank/usr/local 1.5T 455M 1.5T 0% /usr/local > tank/usr/obj 1.5T 0B 1.5T 0% /usr/obj > tank/usr/ports 1.5T 412M 1.5T 0% /usr/ports > tank/usr/src 1.5T 495M 1.5T 0% /usr/src > tank/var/log 1.5T 256K 1.5T 0% /var/log > > Considering that every single filesystem is part of the exact same > pool, with no custom options whatsoever used during filesystem > creation (except for mountpoints), why is the size of tank/DATA 1.8T > while the others are 1.5T? > > > - Sincerely, > Dan Naumov Because 292G is already written to it with another 1.5T available for use. All your other filesystems are less than 0.5G so they don't impact the rounding o fthe Size field. -- Jeremy Faulkner From bugmaster at FreeBSD.org Mon Jul 6 11:06:57 2009 From: bugmaster at FreeBSD.org (FreeBSD bugmaster) Date: Mon Jul 6 11:08:02 2009 Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org Message-ID: <200907061106.n66B6uRG010746@freefall.freebsd.org> Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/136218 fs [zfs] Exported ZFS pools can't be imported into (Open) o kern/135594 fs [zfs] Single dataset unresponsive with Samba o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135480 fs [zfs] panic: lock &arg.lock already initialized o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o bin/135314 fs [zfs] assertion failed for zdb(8) usage o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot f kern/134496 fs [zfs] [panic] ZFS pool export occasionally causes a ke o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133980 fs [panic] [ffs] panic: ffs_valloc: dup alloc o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis o kern/133614 fs [smbfs] [panic] panic: ffs_truncate: read-only filesys o kern/133373 fs [zfs] umass attachment causes ZFS checksum errors, dat o kern/133174 fs [msdosfs] [patch] msdosfs must support utf-encoded int f kern/133150 fs [zfs] Page fault with ZFS on 7.1-RELEASE/amd64 while w o kern/133134 fs [zfs] Missing ZFS zpool labels f kern/133020 fs [zfs] [panic] inappropriate panic caused by zfs. Pani o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132597 fs [tmpfs] [panic] tmpfs-related panic while interrupting o kern/132551 fs [zfs] ZFS locks up on extattr_list_link syscall o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132337 fs [zfs] [panic] kernel panic in zfs_fuid_create_cred o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes f kern/132068 fs [zfs] page fault when using ZFS over NFS on 7.1-RELEAS o kern/131995 fs [nfs] Failure to mount NFSv4 server o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/131086 fs [ext2fs] [patch] mkfs.ext2 creates rotten partition o kern/130979 fs [smbfs] [panic] boot/kernel/smbfs.ko o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130229 fs [iconv] usermount fails on fs that need iconv o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/129148 fs [zfs] [panic] panic on concurrent writing & rollback o kern/129059 fs [zfs] [patch] ZFS bootloader whitelistable via WITHOUT f kern/128829 fs smbd(8) causes periodic panic on 7-RELEASE o kern/128633 fs [zfs] [lor] lock order reversal in zfs o kern/128514 fs [zfs] [mpt] problems with ZFS and LSILogic SAS/SATA Ad f kern/128173 fs [ext2fs] ls gives "Input/output error" on mounted ext3 o kern/127659 fs [tmpfs] tmpfs memory leak o kern/127492 fs [zfs] System hang on ZFS input-output o kern/127420 fs [gjournal] [panic] Journal overflow on gmirrored gjour o kern/127213 fs [tmpfs] sendfile on tmpfs data corruption o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS o kern/125644 fs [zfs] [panic] zfs unfixable fs errors caused panic whe f kern/125536 fs [ext2fs] ext 2 mounts cleanly but fails on commands li o kern/125149 fs [nfs] [panic] changing into .zfs dir from nfs client c f kern/124621 fs [ext3] [patch] Cannot mount ext2fs partition f bin/124424 fs [zfs] zfs(8): zfs list -r shows strange snapshots' siz o kern/123939 fs [msdosfs] corrupts new files o kern/122888 fs [zfs] zfs hang w/ prefetch on, zil off while running t o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o kern/122173 fs [zfs] [panic] Kernel Panic if attempting to replace a o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o kern/122047 fs [ext2fs] [patch] incorrect handling of UF_IMMUTABLE / o kern/122038 fs [tmpfs] [panic] tmpfs: panic: tmpfs_alloc_vp: type 0xc o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121779 fs [ufs] snapinfo(8) (and related tools?) only work for t o kern/121770 fs [zfs] ZFS on i386, large file or heavy I/O leads to ke o bin/121366 fs [zfs] [patch] Automatic disk scrubbing from periodic(8 o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha f kern/120991 fs [panic] [fs] [snapshot] System crashes when manipulati o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F o bin/120288 fs zfs(8): "zfs share -a" does not send SIGHUP to mountd f kern/119735 fs [zfs] geli + ZFS + samba starting on boot panics 7.0-B o kern/118912 fs [2tb] disk sizing/geometry problem with large array o misc/118855 fs [zfs] ZFS-related commands are nonfunctional in fixit o kern/118713 fs [minidump] [patch] Display media size required for a k o kern/118320 fs [zfs] [patch] NFS SETATTR sometimes fails to set file o bin/118249 fs mv(1): moving a directory changes its mtime o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117314 fs [ntfs] Long-filename only NTFS fs'es cause kernel pani o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o kern/116913 fs [ffs] [panic] ffs_blkfree: freeing free block p kern/116608 fs [msdosfs] [patch] msdosfs fails to check mount options o kern/116583 fs [ffs] [hang] System freezes for short time when using o kern/116170 fs [panic] Kernel panic when mounting /tmp o kern/115645 fs [snapshots] [panic] lockmgr: thread 0xc4c00d80, not ex o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o kern/113180 fs [zfs] Setting ZFS nfsshare property does not cause inh o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o kern/109024 fs [msdosfs] mount_msdosfs: msdosfs_iconv: Operation not o kern/109010 fs [msdosfs] can't mv directory within fat32 file system o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106030 fs [ufs] [panic] panic in ufs from geom when a dead disk o kern/105093 fs [ext2fs] [patch] ext2fs on read-only media cannot be m o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [iso9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna f kern/91568 fs [ufs] [panic] writing to UFS/softupdates DVD media in o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/89991 fs [ufs] softupdates with mount -ur causes fs UNREFS o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88266 fs [smbfs] smbfs does not implement UIO_NOCOPY and sendfi o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o kern/85326 fs [smbfs] [panic] saving a file via samba to an overquot o kern/84589 fs [2TB] 5.4-STABLE unresponsive during background fsck 2 o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o kern/77826 fs [ext2fs] ext2fs usb filesystem will not mount RW o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/51583 fs [nullfs] [patch] allow to work with devices and socket o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 143 problems total. From denismacpherson at gmail.com Mon Jul 6 18:21:07 2009 From: denismacpherson at gmail.com (Denis MacPherson) Date: Mon Jul 6 18:21:18 2009 Subject: ZFS invalid vdev configuration Message-ID: <7c82f7860907061056gb7b38e4s52b15ab0e664ffc6@mail.gmail.com> Having a issue with a raidz1 setup, I was testing some things out on a live cd and made a change in the bios for the sata to change it from AHCI to IDE, then before I booted back into FreeBSD I had changed this back so FreeBSD never saw any of these changes. When I booted back up, I was getting insufficient replicas corrupted data message. I tried to export the pool as I found some info on this helping some people but now I am unable to re-import the pool back. Below I have attached some of the outputs of various commands. Any help would be much appreciated. -------------------------------------------------------------------------------------------------------------------------------------- command: zpool import pool: datapool id: 5998882629718828483 state: FAULTED action: The pool cannot be imported due to damaged devices or data. config: datapool UNAVAIL insufficient replicas raidz1 UNAVAIL corrupted data ad14 ONLINE ad12 ONLINE ad10 ONLINE ad8 ONLINE ad16 ONLINE ad18 ONLINE command: zpool import datapool cannot import 'datapool': invalid vdev configuration command: dmesg ZFS filesystem version 6 ZFS storage pool version 6 acd0: DVDR at ata0-slave UDMA33 ad4: 152627MB at ata2-master SATA150 ad6: 953868MB at ata3-master SATA300 ad8: 953869MB at ata4-master SATA300 GEOM_LABEL: Label for provider ad4p1 is msdosfs/EFI. GEOM_LABEL: Label for provider ad4s1a is ufsid/493ee78d1bd00753. ad10: 953869MB at ata5-master SATA300 GEOM_LABEL: Label for provider ad6s1 is ext2fs/1.39-Aug092008. ad12: 953869MB at ata6-master SATA300 ad14: 953869MB at ata7-master SATA300 ad16: 953868MB at ata8-master SATA300 ad18: 953869MB at ata9-master SATA300 Trying to mount root from zfs:tank/root GEOM_LABEL: Label msdosfs/EFI removed. GEOM_LABEL: Label ufsid/493ee78d1bd00753 removed. GEOM_LABEL: Label for provider ad4s1a is ufsid/493ee78d1bd00753. GEOM_LABEL: Label ufsid/493ee78d1bd00753 removed. GEOM_LABEL: Label ext2fs/1.39-Aug092008 removed. command: zdb -l /dev/ad8 -------------------------------------------- LABEL 0 -------------------------------------------- version=6 name='datapool' state=0 txg=2498378 pool_guid=5998882629718828483 hostid=2846502798 hostname='' top_guid=2074816204479013297 guid=8852512481608149738 vdev_tree type='raidz' id=0 guid=2074816204479013297 nparity=1 metaslab_array=14 metaslab_shift=35 ashift=9 asize=6001199677440 children[0] type='disk' id=0 guid=11429030338875577754 path='/dev/ad14' devid='ad:WD-WCAU46148536' whole_disk=0 DTL=23 children[1] type='disk' id=1 guid=15556804307381672763 path='/dev/ad12' devid='ad:WD-WCAU44223223' whole_disk=0 DTL=22 children[2] type='disk' id=2 guid=11466465050580886753 path='/dev/ad10' devid='ad:WD-WCAU44223259' whole_disk=0 DTL=21 children[3] type='disk' id=3 guid=8852512481608149738 path='/dev/ad8' devid='ad:WD-WCAU44026431' whole_disk=0 DTL=20 children[4] type='disk' id=4 guid=12199743530994947046 path='/dev/ad16' devid='ad:WD-WCAU46152226' whole_disk=0 DTL=19 children[5] type='disk' id=5 guid=9333427139064108901 path='/dev/ad18' devid='ad:WD-WCAU46165741' whole_disk=0 DTL=18 -------------------------------------------- LABEL 1 -------------------------------------------- version=6 name='datapool' state=0 txg=2498378 pool_guid=5998882629718828483 hostid=2846502798 hostname='' top_guid=2074816204479013297 guid=8852512481608149738 vdev_tree type='raidz' id=0 guid=2074816204479013297 nparity=1 metaslab_array=14 metaslab_shift=35 ashift=9 asize=6001199677440 children[0] type='disk' id=0 guid=11429030338875577754 path='/dev/ad14' devid='ad:WD-WCAU46148536' whole_disk=0 DTL=23 children[1] type='disk' id=1 guid=15556804307381672763 path='/dev/ad12' devid='ad:WD-WCAU44223223' whole_disk=0 DTL=22 children[2] type='disk' id=2 guid=11466465050580886753 path='/dev/ad10' devid='ad:WD-WCAU44223259' whole_disk=0 DTL=21 children[3] type='disk' id=3 guid=8852512481608149738 path='/dev/ad8' devid='ad:WD-WCAU44026431' whole_disk=0 DTL=20 children[4] type='disk' id=4 guid=12199743530994947046 path='/dev/ad16' devid='ad:WD-WCAU46152226' whole_disk=0 DTL=19 children[5] type='disk' id=5 guid=9333427139064108901 path='/dev/ad18' devid='ad:WD-WCAU46165741' whole_disk=0 DTL=18 -------------------------------------------- LABEL 2 -------------------------------------------- version=6 name='datapool' state=0 txg=2498378 pool_guid=5998882629718828483 hostid=2846502798 hostname='' top_guid=2074816204479013297 guid=8852512481608149738 vdev_tree type='raidz' id=0 guid=2074816204479013297 nparity=1 metaslab_array=14 metaslab_shift=35 ashift=9 asize=6001199677440 children[0] type='disk' id=0 guid=11429030338875577754 path='/dev/ad14' devid='ad:WD-WCAU46148536' whole_disk=0 DTL=23 children[1] type='disk' id=1 guid=15556804307381672763 path='/dev/ad12' devid='ad:WD-WCAU44223223' whole_disk=0 DTL=22 children[2] type='disk' id=2 guid=11466465050580886753 path='/dev/ad10' devid='ad:WD-WCAU44223259' whole_disk=0 DTL=21 children[3] type='disk' id=3 guid=8852512481608149738 path='/dev/ad8' devid='ad:WD-WCAU44026431' whole_disk=0 DTL=20 children[4] type='disk' id=4 guid=12199743530994947046 path='/dev/ad16' devid='ad:WD-WCAU46152226' whole_disk=0 DTL=19 children[5] type='disk' id=5 guid=9333427139064108901 path='/dev/ad18' devid='ad:WD-WCAU46165741' whole_disk=0 DTL=18 -------------------------------------------- LABEL 3 -------------------------------------------- version=6 name='datapool' state=0 txg=2498378 pool_guid=5998882629718828483 hostid=2846502798 hostname='' top_guid=2074816204479013297 guid=8852512481608149738 vdev_tree type='raidz' id=0 guid=2074816204479013297 nparity=1 metaslab_array=14 metaslab_shift=35 ashift=9 asize=6001199677440 children[0] type='disk' id=0 guid=11429030338875577754 path='/dev/ad14' devid='ad:WD-WCAU46148536' whole_disk=0 DTL=23 children[1] type='disk' id=1 guid=15556804307381672763 path='/dev/ad12' devid='ad:WD-WCAU44223223' whole_disk=0 DTL=22 children[2] type='disk' id=2 guid=11466465050580886753 path='/dev/ad10' devid='ad:WD-WCAU44223259' whole_disk=0 DTL=21 children[3] type='disk' id=3 guid=8852512481608149738 path='/dev/ad8' devid='ad:WD-WCAU44026431' whole_disk=0 DTL=20 children[4] type='disk' id=4 guid=12199743530994947046 path='/dev/ad16' devid='ad:WD-WCAU46152226' whole_disk=0 DTL=19 children[5] type='disk' id=5 guid=9333427139064108901 path='/dev/ad18' devid='ad:WD-WCAU46165741' whole_disk=0 DTL=18 From dan.naumov at gmail.com Tue Jul 7 00:45:41 2009 From: dan.naumov at gmail.com (Dan Naumov) Date: Tue Jul 7 00:45:50 2009 Subject: ZFS: swap on a ZVOL Message-ID: Hello list. As far as I know, using swap on top of a "non-trivial" filesystem like ZFS is considered "unsupported", but it does in fact work. You can create a ZVOL of arbitrary size (say, 4G) and then do the following: zfs set org.freebsd:swap=on pool/swapvolname to have /etc/rc.d/zfs enable swap on said ZVOL on every boot. You can also do this in an ugly way: put swapon /dev/zvol// into your /etc/rc.local (without having to pass the "org.freebsd:swap=on" option to the ZVOL). Now the question remains, what kind of issues are expected to arise when using swap on a ZVOL and is there any work going to in order to resolve them? One of the issues mentioned is that ZVOL swap cannot handle kernel dumps and another, more serious potential issue is a race condition where "more swap is needed to swap". Assuming I have a machine with 2gb ram, if I use a 4gb ZVOL swap, am I likely to run into any serious issues assuming that under normal operation, the system uses from none to very little swap? - Sincerely, Dan Naumov From numisemis at yahoo.com Tue Jul 7 07:21:55 2009 From: numisemis at yahoo.com (=?utf-8?B?xaBpbXVuIE1pa2VjaW4=?=) Date: Tue Jul 7 07:22:01 2009 Subject: ZFS: swap on a ZVOL Message-ID: <354003.91539.qm@web37307.mail.mud.yahoo.com> Dan Naumov wrote: As far as I know, using swap on top of a "non-trivial" filesystem like ZFS is considered "unsupported", but it does in fact work. You can create a ZVOL of arbitrary size (say, 4G) and then do the following: zfs set org.freebsd:swap=on pool/swapvolname to have /etc/rc.d/zfs enable swap on said ZVOL on every boot. You can also do this in an ugly way: put swapon /dev/zvol// into your /etc/rc.local (without having to pass the "org.freebsd:swap=on" option to the ZVOL). Now the question remains, what kind of issues are expected to arise when using swap on a ZVOL and is there any work going to in order to resolve them? One of the issues mentioned is that ZVOL swap cannot handle kernel dumps and another, more serious potential issue is a race condition where "more swap is needed to swap". Assuming I have a machine with 2gb ram, if I use a 4gb ZVOL swap, am I likely to run into any serious issues assuming that under normal operation, the system uses from none to very little swap? AFAIK, it was said that race condition you mentioned also exists in OpenSolaris (that was back in the ZFS v6 days). But, AFAIK new versions of OpenSolaris do use swap on ZFS volume as by default (correct me if I'm wrong). Somebody more knowledgeable should answer this, but this made me thinking that maybe, just maybe that race condition was solved in some ZFS version >v6. From Frank.Batschulat at Sun.COM Tue Jul 7 08:00:05 2009 From: Frank.Batschulat at Sun.COM (Frank Batschulat (Home)) Date: Tue Jul 7 08:00:12 2009 Subject: ZFS: swap on a ZVOL In-Reply-To: <354003.91539.qm@web37307.mail.mud.yahoo.com> References: <354003.91539.qm@web37307.mail.mud.yahoo.com> Message-ID: On Tue, 07 Jul 2009 09:21:54 +0200, ?imun Mikecin wrote: > > Dan Naumov wrote: > As far as I know, using swap on top of a "non-trivial" filesystem like > ZFS is considered "unsupported", but it does in fact work. You can > create a ZVOL of arbitrary size (say, 4G) and then do the following: > zfs set org.freebsd:swap=on pool/swapvolname > to have /etc/rc.d/zfs enable swap on said ZVOL on every boot. You can > also do this in an ugly way: put swapon > /dev/zvol// into your /etc/rc.local (without > having to pass the "org.freebsd:swap=on" option to the ZVOL). > > Now the question remains, what kind of issues are expected to arise > when using swap on a ZVOL and is there any work going to in order to > resolve them? One of the issues mentioned is that ZVOL swap cannot > handle kernel dumps and another, more serious potential issue is a > race condition where "more swap is needed to swap". Assuming I have a > machine with 2gb ram, if I use a 4gb ZVOL swap, am I likely to run > into any serious issues assuming that under normal operation, the > system uses from none to very little swap? > > AFAIK, it was said that race condition you mentioned also exists in OpenSolaris (that was back in the ZFS v6 days). > But, AFAIK new versions of OpenSolaris do use swap on ZFS volume as by default (correct me if I'm wrong). > Somebody more knowledgeable should answer this, but this made me thinking that maybe, just maybe that race condition was solved in some ZFS version >v6. Right, we do use a zvol as swap device, and we also do use a dedicated zvol as the dump device for the kernel crash dump. http://www.opensolaris.org/os/community/zfs/boot/zfsbootFAQ/#zfsswap http://www.solarisinternals.com/wiki/index.php/ZFS_Troubleshooting_Guide#ZFS_Swap_and_Dump_Devices http://docs.sun.com/app/docs/doc/819-5461/zfsboot-1?a=view http://www.opensolaris.org/os/community/zfs/boot/zfsboottalk.0910.pdf http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=5008936 was part of the ZFS boot project: http://www.opensolaris.org/os/community/zfs/boot/ afai remember neither the dump nor swap part itself required changes to the ondisk format, zpool version 6 just covered storing some new pool properties like 'bootfs' -> http://www.opensolaris.org/os/community/zfs/version/6/ hth frankB From linimon at FreeBSD.org Wed Jul 8 19:05:02 2009 From: linimon at FreeBSD.org (linimon@FreeBSD.org) Date: Wed Jul 8 19:05:08 2009 Subject: kern/136470: [nfs] Cannot mount / in read-only, over NFS Message-ID: <200907081905.n68J5167098114@freefall.freebsd.org> Old Synopsis: Cannot mount / in read-only, over NFS New Synopsis: [nfs] Cannot mount / in read-only, over NFS Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Wed Jul 8 19:04:49 UTC 2009 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=136470 From james-freebsd-fs2 at jrv.org Wed Jul 8 21:48:48 2009 From: james-freebsd-fs2 at jrv.org (James R. Van Artsdalen) Date: Wed Jul 8 21:48:55 2009 Subject: ZFS: File name too long Message-ID: <4A55142B.8020800@jrv.org> svn rev 192136 (-CURRENT on May 14, 2009) /Volumes/incoming/Personal# ls -l '.zfs/snapshot/syssnap-1247029203.2009-07-08.00:00:03.189-27-3' ls: .zfs/snapshot/syssnap-1247029203.2009-07-08.00:00:03.189-27-3: File name too long /Volumes/incoming/Personal# /Volumes/incoming/Personal# ls -l '/root/.zfs/snapshot/syssnap-1247029203.2009-07-08.00:00:03.189-27-3'/.login -rw-r--r-- 1 root wheel 290 Feb 27 07:42 /root/.zfs/snapshot/syssnap-1247029203.2009-07-08.00:00:03.189-27-3/.login /Volumes/incoming/Personal# From serenity at exscape.org Thu Jul 9 09:02:26 2009 From: serenity at exscape.org (Thomas Backman) Date: Thu Jul 9 09:02:39 2009 Subject: "New" ZFS crash on FS (pool?) unmount/export In-Reply-To: <72163521-40BF-4764-8B74-5446A88DFBF8@exscape.org> References: <72163521-40BF-4764-8B74-5446A88DFBF8@exscape.org> Message-ID: <766FFF07-181A-4180-B020-AA3EE46CF6F8@exscape.org> On Jun 20, 2009, at 09:11, Thomas Backman wrote: > I just ran into this tonight. Not sure exactly what triggered it - > the box stopped responding to pings at 02:07AM and it has a cron > backup job using zfs send/recv at 02:00, so I'm guessing it's > related, even though the backup probably should have finished before > then... Hmm. Anyway. > > r194478. > > kernel trap 12 with interrupts disabled > > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0x288 > fault code = supervisor read data, page not present > instruction pointer = 0x20:0xffffffff805a4989 > stack pointer = 0x28:0xffffff803e8b57e0 > frame pointer = 0x28:0xffffff803e8b5840 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = resume, IOPL = 0 > current process = 57514 (zpool) > panic: from debugger > cpuid = 0 > Uptime: 10h22m13s > Physical memory: 2027 MB > > (kgdb) bt > #0 doadump () at pcpu.h:223 > #1 0xffffffff8059c409 in boot (howto=260) at /usr/src/sys/kern/ > kern_shutdown.c:419 > #2 0xffffffff8059c85c in panic (fmt=Variable "fmt" is not available. > ) at /usr/src/sys/kern/kern_shutdown.c:575 > #3 0xffffffff801f1377 in db_panic (addr=Variable "addr" is not > available. > ) at /usr/src/sys/ddb/db_command.c:478 > #4 0xffffffff801f1781 in db_command (last_cmdp=0xffffffff80c38620, > cmd_table=Variable "cmd_table" is not available. > ) at /usr/src/sys/ddb/db_command.c:445 > #5 0xffffffff801f19d0 in db_command_loop () at /usr/src/sys/ddb/ > db_command.c:498 > #6 0xffffffff801f3969 in db_trap (type=Variable "type" is not > available. > ) at /usr/src/sys/ddb/db_main.c:229 > #7 0xffffffff805ce465 in kdb_trap (type=12, code=0, > tf=0xffffff803e8b5730) at /usr/src/sys/kern/subr_kdb.c:534 > #8 0xffffffff8088715d in trap_fatal (frame=0xffffff803e8b5730, > eva=Variable "eva" is not available. > ) at /usr/src/sys/amd64/amd64/trap.c:847 > #9 0xffffffff80887fb2 in trap (frame=0xffffff803e8b5730) at /usr/ > src/sys/amd64/amd64/trap.c:345 > #10 0xffffffff8086e007 in calltrap () at /usr/src/sys/amd64/amd64/ > exception.S:223 > #11 0xffffffff805a4989 in _sx_xlock_hard (sx=0xffffff0043557d50, > tid=18446742975830720512, opts=Variable "opts" is not available. > ) > at /usr/src/sys/kern/kern_sx.c:575 > #12 0xffffffff805a52fe in _sx_xlock (sx=Variable "sx" is not > available. > ) at sx.h:155 > #13 0xffffffff80fe2995 in zfs_freebsd_reclaim () from /boot/kernel/ > zfs.ko > #14 0xffffffff808cefca in VOP_RECLAIM_APV (vop=0xffffff0043557d38, > a=0xffffff0043557d50) at vnode_if.c:1926 > #15 0xffffffff80626f6e in vgonel (vp=0xffffff00437a7938) at > vnode_if.h:830 > #16 0xffffffff8062b528 in vflush (mp=0xffffff0060f2a000, rootrefs=0, > flags=0, td=0xffffff0061528000) > at /usr/src/sys/kern/vfs_subr.c:2450 > #17 0xffffffff80fdd3a8 in zfs_umount () from /boot/kernel/zfs.ko > #18 0xffffffff8062420a in dounmount (mp=0xffffff0060f2a000, > flags=1626513408, td=Variable "td" is not available. > ) > at /usr/src/sys/kern/vfs_mount.c:1287 > #19 0xffffffff80624975 in unmount (td=0xffffff0061528000, > uap=0xffffff803e8b5c00) > at /usr/src/sys/kern/vfs_mount.c:1172 > #20 0xffffffff8088783f in syscall (frame=0xffffff803e8b5c90) at /usr/ > src/sys/amd64/amd64/trap.c:984 > #21 0xffffffff8086e290 in Xfast_syscall () at /usr/src/sys/amd64/ > amd64/exception.S:364 > #22 0x000000080104e49c in ?? () > Previous frame inner to this frame (corrupt stack?) I might have hit the same thing again... only it didn't *crash* this time, just freeze! I got a "GEOM_GATE: Device ggateX destroyed" on my console, and it stopped responding to pings, keyboard input, etc. (NOTE: The GEOM_GATE message MAY have been an old one. I *think* it was from the night before but can't be sure...) It obviously happened while running my ugly-hack backup script this time too, since it stopped responding to pings ~02:02AM with the script running at 02:00. I'm not sure where it crashed, since snapshots were NOT taken on the "local box". It usually crashes during export, long after taking the snapshots. BTW, current system is BETA1 r195422M (dtrace timestamp patch + libzfs_sendrecv.c patch ( http://lists.freebsd.org/pipermail/freebsd-current/2009-May/006814.html ). Here's the script in its relevant entirety (I removed the "initial backup" part since it never runs using cron anyway). I realize it's an ugly hack (and that my bash-fu could be stronger), but what the heck. Essentially, it creates a GEOM provider of a file, containing a zpool, imports the pool and creates a clone to it, and then exports the pool. The export appears to be what causes all the trouble - usually, not this time around. Every time it crashes it seems to be during or very soon after the export - only this time it didn't even take the snapshots. #!/bin/bash PATH="$PATH:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/ sbin" function die() { echo "$@" 2>&1 zpool export slave 2>&1 > /dev/null ggatel destroy -u 666 2>&1 > /dev/null exit 1 } function mount_unmount { if [ -z "$1" ]; then die 'Invalid argument given to mount_unmount' elif [[ "$1" == "mount" ]]; then zpool list | grep -q slave if [ "$?" = "0" ]; then echo Already mounted return 0 fi echo Creating ggate device ggatel create -u 666 /mnt/backup/chaos/slavefile || die 'Unable to create GEOM provider from file' echo 'Sleeping for 5 seconds...' sleep 5 echo Importing pool zpool import -R /slave slave || die 'Unable to import slave pool' elif [[ "$1" == "unmount" ]]; then echo Exporting pool zpool export slave || die 'Unable to export slave pool' ggatel destroy -u 666 || die 'Unable to destroy GEOM provider' fi } f [ ! -z "$1" ]; then case $1 in mount) mount_unmount mount; exit 0;; unmount) mount_unmount unmount; exit 0;; initial) initial; exit 0;; backup) ;; *) help;; esac else help fi if [ ! -f "/mnt/backup/chaos/slavefile" ]; then echo 'Backup error! slavefile does not exist!' | mail -s "Backup error" root echo 'Slavefile does not exist!' exit 1 fi mount_unmount mount CURR=$(date +"backup-%F-%H%M") echo Taking snapshots zfs snapshot -r tank@$CURR || die 'Unable to create $CURR snapshot' echo Starting backup... LAST=$(cat /root/.last-backup) zfs send -R -I $LAST tank@$CURR | zfs recv -Fvd slave echo $CURR > /root/.last-backup mount_unmount unmount echo Running rsync rsync -av --delete /bootdir/boot exscape::backup-freebsd/chaos rsync -av --delete /root exscape::backup-freebsd/chaos rsync -av --delete ~serenity exscape::backup-freebsd/chaos echo 'All done!' ------------------- So, in *normal* cases, everything runs just fine. This is the case perhaps 90% of the time. In normal *crashes*, it hangs on export with the above backtrace. This time, all I know is that it crashes soon after starting the backup... during import, perhaps? Regards, Thomas From serenity at exscape.org Thu Jul 9 12:36:28 2009 From: serenity at exscape.org (Thomas Backman) Date: Thu Jul 9 12:36:35 2009 Subject: "New" ZFS crash on FS (pool?) unmount/export In-Reply-To: <766FFF07-181A-4180-B020-AA3EE46CF6F8@exscape.org> References: <72163521-40BF-4764-8B74-5446A88DFBF8@exscape.org> <766FFF07-181A-4180-B020-AA3EE46CF6F8@exscape.org> Message-ID: <9FAC783B-5709-460B-B6DA-364DCD0DE8DA@exscape.org> On Jul 9, 2009, at 11:01, Thomas Backman wrote: > On Jun 20, 2009, at 09:11, Thomas Backman wrote: > >> I just ran into this tonight. Not sure exactly what triggered it - >> the box stopped responding to pings at 02:07AM and it has a cron >> backup job using zfs send/recv at 02:00, so I'm guessing it's >> related, even though the backup probably should have finished >> before then... Hmm. Anyway. >> >> r194478. >> >> kernel trap 12 with interrupts disabled >> >> Fatal trap 12: page fault while in kernel mode >> cpuid = 0; apic id = 00 >> fault virtual address = 0x288 >> fault code = supervisor read data, page not present >> instruction pointer = 0x20:0xffffffff805a4989 >> stack pointer = 0x28:0xffffff803e8b57e0 >> frame pointer = 0x28:0xffffff803e8b5840 >> code segment = base 0x0, limit 0xfffff, type 0x1b >> = DPL 0, pres 1, long 1, def32 0, gran 1 >> processor eflags = resume, IOPL = 0 >> current process = 57514 (zpool) >> panic: from debugger >> cpuid = 0 >> Uptime: 10h22m13s >> Physical memory: 2027 MB >> >> (kgdb) bt >> #0 doadump () at pcpu.h:223 >> #1 0xffffffff8059c409 in boot (howto=260) at /usr/src/sys/kern/ >> kern_shutdown.c:419 >> #2 0xffffffff8059c85c in panic (fmt=Variable "fmt" is not available. >> ) at /usr/src/sys/kern/kern_shutdown.c:575 >> #3 0xffffffff801f1377 in db_panic (addr=Variable "addr" is not >> available. >> ) at /usr/src/sys/ddb/db_command.c:478 >> #4 0xffffffff801f1781 in db_command (last_cmdp=0xffffffff80c38620, >> cmd_table=Variable "cmd_table" is not available. >> ) at /usr/src/sys/ddb/db_command.c:445 >> #5 0xffffffff801f19d0 in db_command_loop () at /usr/src/sys/ddb/ >> db_command.c:498 >> #6 0xffffffff801f3969 in db_trap (type=Variable "type" is not >> available. >> ) at /usr/src/sys/ddb/db_main.c:229 >> #7 0xffffffff805ce465 in kdb_trap (type=12, code=0, >> tf=0xffffff803e8b5730) at /usr/src/sys/kern/subr_kdb.c:534 >> #8 0xffffffff8088715d in trap_fatal (frame=0xffffff803e8b5730, >> eva=Variable "eva" is not available. >> ) at /usr/src/sys/amd64/amd64/trap.c:847 >> #9 0xffffffff80887fb2 in trap (frame=0xffffff803e8b5730) at /usr/ >> src/sys/amd64/amd64/trap.c:345 >> #10 0xffffffff8086e007 in calltrap () at /usr/src/sys/amd64/amd64/ >> exception.S:223 >> #11 0xffffffff805a4989 in _sx_xlock_hard (sx=0xffffff0043557d50, >> tid=18446742975830720512, opts=Variable "opts" is not available. >> ) >> at /usr/src/sys/kern/kern_sx.c:575 >> #12 0xffffffff805a52fe in _sx_xlock (sx=Variable "sx" is not >> available. >> ) at sx.h:155 >> #13 0xffffffff80fe2995 in zfs_freebsd_reclaim () from /boot/kernel/ >> zfs.ko >> #14 0xffffffff808cefca in VOP_RECLAIM_APV (vop=0xffffff0043557d38, >> a=0xffffff0043557d50) at vnode_if.c:1926 >> #15 0xffffffff80626f6e in vgonel (vp=0xffffff00437a7938) at >> vnode_if.h:830 >> #16 0xffffffff8062b528 in vflush (mp=0xffffff0060f2a000, >> rootrefs=0, flags=0, td=0xffffff0061528000) >> at /usr/src/sys/kern/vfs_subr.c:2450 >> #17 0xffffffff80fdd3a8 in zfs_umount () from /boot/kernel/zfs.ko >> #18 0xffffffff8062420a in dounmount (mp=0xffffff0060f2a000, >> flags=1626513408, td=Variable "td" is not available. >> ) >> at /usr/src/sys/kern/vfs_mount.c:1287 >> #19 0xffffffff80624975 in unmount (td=0xffffff0061528000, >> uap=0xffffff803e8b5c00) >> at /usr/src/sys/kern/vfs_mount.c:1172 >> #20 0xffffffff8088783f in syscall (frame=0xffffff803e8b5c90) at / >> usr/src/sys/amd64/amd64/trap.c:984 >> #21 0xffffffff8086e290 in Xfast_syscall () at /usr/src/sys/amd64/ >> amd64/exception.S:364 >> #22 0x000000080104e49c in ?? () >> Previous frame inner to this frame (corrupt stack?) > > > Here's the script in its relevant entirety [...] > > #!/bin/bash > > PATH="$PATH:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/ > sbin" > > function die() { > echo "$@" 2>&1 > zpool export slave 2>&1 > /dev/null > ggatel destroy -u 666 2>&1 > /dev/null > exit 1 > } > > function mount_unmount { > if [ -z "$1" ]; then > die 'Invalid argument given to mount_unmount' > elif [[ "$1" == "mount" ]]; then > zpool list | grep -q slave > if [ "$?" = "0" ]; then > echo Already mounted > return 0 > fi > echo Creating ggate device > ggatel create -u 666 /mnt/backup/chaos/slavefile || die 'Unable to > create GEOM provider from file' > echo 'Sleeping for 5 seconds...' > sleep 5 > echo Importing pool > zpool import -R /slave slave || die 'Unable to import slave pool' > elif [[ "$1" == "unmount" ]]; then > echo Exporting pool > zpool export slave || die 'Unable to export slave pool' > ggatel destroy -u 666 || die 'Unable to destroy GEOM provider' > fi > } > > f [ ! -z "$1" ]; then > case $1 in > mount) mount_unmount mount; exit 0;; > unmount) mount_unmount unmount; exit 0;; > initial) initial; exit 0;; > backup) ;; > *) help;; > esac > else > help > fi > > if [ ! -f "/mnt/backup/chaos/slavefile" ]; then > echo 'Backup error! slavefile does not exist!' | mail -s "Backup > error" root > echo 'Slavefile does not exist!' > exit 1 > fi > > mount_unmount mount > > CURR=$(date +"backup-%F-%H%M") > > echo Taking snapshots > zfs snapshot -r tank@$CURR || die 'Unable to create $CURR snapshot' > > echo Starting backup... > LAST=$(cat /root/.last-backup) > zfs send -R -I $LAST tank@$CURR | zfs recv -Fvd slave > > echo $CURR > /root/.last-backup > > mount_unmount unmount > > echo Running rsync > rsync -av --delete /bootdir/boot exscape::backup-freebsd/chaos > rsync -av --delete /root exscape::backup-freebsd/chaos > rsync -av --delete ~serenity exscape::backup-freebsd/chaos > > echo 'All done!' > > ------------------- Sorry for the monologue, but I ran in to this again, this time with a panic again. Similar but not identical to the old one. OK, so I figured I would update my "untouched" src clone (used to save bandwidth from the FBSD SVN server when I feel I need to start with a *clean* source tree), now that there have been quite a few changes since that revision. I pretty much did this (if other shells are different, !$ in bash is the last argument to the previous command.) 1) Clean up /usr/src from "my" stuff 2) svn update 3) svn diff and svn status, to make sure it was clean 4) zfs promote tank/usr/src ## usr/src was a clone of the untouched, read-only fs "tank/usr/src_r194478-UNTOUCHED" 5) zfs destroy -r tank/usr/src_r194478-UNTOUCHED ## The old one, obviously 6) zfs snapshot tank/usr/src@r195488_UNTOUCHED 7) zfs clone !$ tank/usr/src_r195488-UNTOUCHED 8) zfs promote !$ 9) zfs set readonly=on !$ 10) And, in case it may matter, I slightly modified the contents of / usr/src afterwards (applied two patches that aren't merged into HEAD (yet?)). ... I THINK that's it. Since my bash_history got killed in the panic, I may be slighty wrong. In any case, usr/src is now a clone of the readonly UNTOUCHED fs: [root@chaos ~]# zfs get origin tank/usr/src NAME PROPERTY VALUE SOURCE tank/usr/src origin tank/usr/ src_r195488_UNTOUCHED@r195488_UNTOUCHED - I then ran the backup script just posted in this thread: [root@chaos ~]# bash backup.sh backup Creating ggate device Sleeping for 5 seconds... Importing pool Taking snapshots Starting backup... attempting destroy slave/usr/src_r194478-UNTOUCHED@backup-20090709-1250 success attempting destroy slave/usr/src_r194478-UNTOUCHED@r194478-UNTOUCHED failed - trying rename to slave/usr/src_r194478-UNTOUCHED@recv-38883-1 failed (2) - will try again on next pass attempting destroy slave/usr/src_r194478-UNTOUCHED@backup-20090709-1235 success attempting destroy slave/usr/src_r194478-UNTOUCHED failed - trying rename to slave/recv-38883-2 failed (2) - will try again on next pass promoting slave/usr/src another pass: attempting destroy slave/usr/src_r194478-UNTOUCHED success attempting destroy slave/usr/src@r194478-UNTOUCHED success attempting rename slave/usr/src to slave/usr/src_r195488_UNTOUCHED success receiving incremental stream of tank@backup-20090709-1328 into slave@backup-20090709-1328 received 312B stream in 1 seconds (312B/sec) receiving incremental stream of tank/tmp@backup-20090709-1328 into slave/tmp@backup-20090709-1328 received 119KB stream in 1 seconds (119KB/sec) receiving incremental stream of tank/var@backup-20090709-1328 into slave/var@backup-20090709-1328 received 211KB stream in 1 seconds (211KB/sec) receiving incremental stream of tank/var/log@backup-20090709-1328 into slave/var/log@backup-20090709-1328 received 468KB stream in 1 seconds (468KB/sec) receiving incremental stream of tank/var/crash@backup-20090709-1328 into slave/var/crash@backup-20090709-1328 received 312B stream in 1 seconds (312B/sec) receiving incremental stream of tank/root@backup-20090709-1328 into slave/root@backup-20090709-1328 received 156KB stream in 1 seconds (156KB/sec) receiving incremental stream of tank/usr@backup-20090709-1328 into slave/usr@backup-20090709-1328 received 302KB stream in 1 seconds (302KB/sec) receiving incremental stream of tank/usr/obj@backup-20090709-1328 into slave/usr/obj@backup-20090709-1328 received 8.52MB stream in 8 seconds (1.07MB/sec) receiving incremental stream of tank/usr/ src_r195488_UNTOUCHED@r195488_UNTOUCHED into slave/usr/ src_r195488_UNTOUCHED@r195488_UNTOUCHED received 112MB stream in 43 seconds (2.60MB/sec) receiving incremental stream of tank/usr/ src_r195488_UNTOUCHED@backup-20090709-1328 into slave/usr/ src_r195488_UNTOUCHED@backup-20090709-1328 received 312B stream in 1 seconds (312B/sec) receiving incremental stream of tank/usr/ports@backup-20090709-1328 into slave/usr/ports@backup-20090709-1328 received 312B stream in 1 seconds (312B/sec) receiving incremental stream of tank/usr/ports/ distfiles@backup-20090709-1328 into slave/usr/ports/ distfiles@backup-20090709-1328 received 312B stream in 1 seconds (312B/sec) found clone origin slave/usr/src_r195488_UNTOUCHED@r195488_UNTOUCHED receiving incremental stream of tank/usr/src@backup-20090709-1328 into slave/usr/src@backup-20090709-1328 received 216KB stream in 1 seconds (216KB/sec) local fs slave/usr/src does not have fromsnap (backup-20090709-1250 in stream); must have been deleted locally; ignoring Exporting pool Read from remote host 192.168.1.10: Operation timed out ... and the DDB output (first part copy/paste from kgdb, second part handwritten since the kgdb BT was totally broken, as I expected.) Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x70 fault code = supervisor write data, page not present instruction pointer = 0x20:0xffffffff8036e855 stack pointer = 0x28:0xffffff803ea637d0 frame pointer = 0x28:0xffffff803ea637e0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 38905 (zpool) _sx_slock() dmu_buf_update_user()+0x47 zfs_znode_dmu_fini() zfs_freebsd_reclaim() VOP_RECLAIM_APV() vgonel() vflush() zfs_umount() dounmount() unmount() syscall() Xfast_syscall() BTW, what's with the "local fs slave/usr/src does not have fromsnap (backup-20090709-1250 in stream); must have been deleted locally; ignoring" part? This is what I get when I try an incremental backup now: [root@chaos ~]# bash backup.sh backup Creating ggate device Sleeping for 5 seconds... Importing pool Taking snapshots Starting backup... local fs slave/usr/src does not have fromsnap (backup-20090709-1250 in stream); must have been deleted locally; ignoring receiving incremental stream of tank@backup-20090709-1328 into slave@backup-20090709-1328 snap slave@backup-20090709-1328 already exists; ignoring local fs slave/usr/src does not have fromsnap (backup-20090709-1250 in stream); must have been deleted locally; ignoring warning: cannot send 'tank/tmp@backup-20090709-1328': Broken pipe warning: cannot send 'tank/tmp@backup-20090709-1406': Broken pipe Exporting pool Running rsync ... rsync runs, no panic, but no ZFS backup, either. Guess it's time for another "initial" backup, i.e. start all over with 0 snapshots... The initial backup worked just fine, it found the clone/origin etc. and made it work. Stripped from comments and echo statements, the function is simply: function initial { for BACK in $(zfs list -t snapshot -H -r tank | awk '{print $1}'); do zfs destroy $BACK; done zpool destroy slave 2>/dev/null; sleep 3; ggatel destroy -u 666 2>/dev/null; sleep 3 # Clean up if needed ggatel create -u 666 /mnt/backup/chaos/slavefile; sleep 3 zpool create -f -R /slave slave ggate666 && NOW=$(date +"backup-%Y %m%d-%H%M") || die 'Unable to create pool' zfs snapshot -r tank@$NOW || die 'Unable to snapshot' zfs send -R tank@$NOW | zfs recv -vFd slave mount_unmount unmount echo $NOW > /root/.last-backup } After that, incrementals are fine again. Regards, Thomas From spartak at valerich.com Thu Jul 9 14:50:04 2009 From: spartak at valerich.com (Spartak Radchenko) Date: Thu Jul 9 14:50:10 2009 Subject: kern/127420: [gjournal] [panic] Journal overflow on gmirrored gjournal Message-ID: <200907091450.n69Eo3bp009383@freefall.freebsd.org> The following reply was made to PR kern/127420; it has been noted by GNATS. From: Spartak Radchenko To: bug-followup@FreeBSD.org, ruben@verweg.com Cc: Subject: Re: kern/127420: [gjournal] [panic] Journal overflow on gmirrored gjournal Date: Thu, 09 Jul 2009 18:22:55 +0400 I have the same problem. FreeBSD 7.2-RELEASE amd64, gjournal on gmirrored volume (local drive + geom_gate mirrored). I am trying to make something like a HA cluster using freevrrpd, ggate, gmirror and gjournal. It generally works, but every time a server with ggated running goes down (I use hardware reset for testing) first ggate0 device is removed from gmirrored volume on master as it should, next master panics with "gjournal overflow" message. From mailing at gaturkey.com Thu Jul 9 16:37:38 2009 From: mailing at gaturkey.com (Global Access Travel) Date: Thu Jul 9 16:37:45 2009 Subject: Fam Trip to TURKEY for $999 (Refundable) Message-ID: <5549a4f4b9c86258f818ed1e1b583f84@localhost.localdomain> [http://www.turkeycallingus.com/] Exclusive Boutique Enterprise Turkey FAM ISTANBUL - CAPPADOCIA - KONYA - ANTALYA - PAMUKKALE - KUSADASI 9 Nights / 11 Days $999 ? 5 Continents ? 150 Countries Worldwide ? 100.000 Hotels ? Instant Confirmation [http://www.turkeycallingus.com] [http://www.turkeycallingus.com/turkey-fam/TurkeyFam.htm] [http://www.turkeycallingus.com/turkey-fam/TurkeyFamItinerary.htm] [http://www.turkeycallingus.com/turkey-fam/TurkeyFamRates.htm] [http://www.turkeycallingus.com/turkey-fam/TurkeyFamServices.htm] [http://www.turkeycallingus.com/turkey-fam/TurkeyFamHotels.htm] Global Access proudly presents the biggest FAM Trip of the year, teaming with Turkish Airlines and Turkish Ministry of Tourism and Culture. As the host of ASTA IDE 2010 and European Capital of Culture 2010, Turkey is likely to be the one of the most popular destinations in 2010. Those who act early and get to know this beautiful country better will be able to give a better insight to their clients and secure more bookings. Our specially selected travel agents will stay in best hotels in each town, be escorted by professional, top tour guides, taste exceptionally good examples of Turkish Cuisine, and get to know Turkey in elegant way. Join us for a luxury FAM adventure and be our special guest in our beautiful country! COMBINE WITH World Travel Market! One of the biggest travel shows of Europe and the world, WTM, will be held in London between 9-12 November 2009. Combine your London trip with Turkey and benefit from great agent rates to see one of the most popular tourist destinations from USA and Canada. WE WILL REFUND YOUR MONEY BACK ! Upon booking your 20th passenger on a Global Access Travel Service, we will refund you the whole tour price that you?ve paid for the FAM Trip. If you book 20 or more people on a Global Access Travel Service before the FAM Trip starts, then you will travel for free! About Us Global Access Travel (GA) was founded in Turkey by a group of tourism professionals and marketing experts who recognized the needs to offer online services for accommodations, car rentals, and other travel related services to travel agencies. Through its sophisticated online reservation services, GA offers more than 100,000 hotels, motels, resorts, clubs and apartments all around the world. Other services of GA include car rentals, transfers, special tours, luxury services, city breaks, flight tickets and other services such as tailor made tour packages, exhibition organizations, incentives and other travel related services around the globe at competitive rates. [http://www.TurkeyCallingus.com] www.TurkeyCalling.us [http://www.turkeycallingus.com/turkey-calling-contact-us.htm] Global Access Travel Tel: +90 212 258 58 29 Fax: +90 212 258 34 47 E-mail : [mailto:incoming@gaturkey.com] incoming@gaturkey.com Website: [http://www.turkeycallingus.com/] www.TurkeyCalling.Us This message was sent by: FamTrit turkey, N?zhetiye Cad., istanbul, besiktas 34357, Turkey To be removed click here: http://app.icontact.com/icp/mmail-mprofile.pl?r=47129454&l=82243&s=5T3C&m=578549&c=305227 Forward to a friend: http://app.icontact.com/icp/sub/forward?m=578549&s=47129454&c=5T3C&cid=305227 From linimon at FreeBSD.org Fri Jul 10 03:54:07 2009 From: linimon at FreeBSD.org (linimon@FreeBSD.org) Date: Fri Jul 10 03:54:19 2009 Subject: kern/136601: [zfs] tar: Couldn't list extended attributes: Read-only file system Message-ID: <200907100354.n6A3s69v029575@freefall.freebsd.org> Old Synopsis: tar: Couldn't list extended attributes: Read-only file system New Synopsis: [zfs] tar: Couldn't list extended attributes: Read-only file system Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Fri Jul 10 03:53:05 UTC 2009 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=136601 From serenity at exscape.org Fri Jul 10 19:01:49 2009 From: serenity at exscape.org (Thomas Backman) Date: Fri Jul 10 19:01:56 2009 Subject: Reproducible ZFS panic, w/ script (Was: "New" ZFS crash on FS (pool?) unmount/export) In-Reply-To: <72163521-40BF-4764-8B74-5446A88DFBF8@exscape.org> References: <72163521-40BF-4764-8B74-5446A88DFBF8@exscape.org> Message-ID: On Jun 20, 2009, at 09:11, Thomas Backman wrote: > I just ran into this tonight. Not sure exactly what triggered it - > the box stopped responding to pings at 02:07AM and it has a cron > backup job using zfs send/recv at 02:00, so I'm guessing it's > related, even though the backup probably should have finished before > then... Hmm. Anyway. > > r194478. > > kernel trap 12 with interrupts disabled > > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0x288 > fault code = supervisor read data, page not present > instruction pointer = 0x20:0xffffffff805a4989 > stack pointer = 0x28:0xffffff803e8b57e0 > frame pointer = 0x28:0xffffff803e8b5840 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = resume, IOPL = 0 > current process = 57514 (zpool) > panic: from debugger > cpuid = 0 > Uptime: 10h22m13s > Physical memory: 2027 MB > > (kgdb) bt > ... > #9 0xffffffff80887fb2 in trap (frame=0xffffff803e8b5730) at /usr/ > src/sys/amd64/amd64/trap.c:345 > #10 0xffffffff8086e007 in calltrap () at /usr/src/sys/amd64/amd64/ > exception.S:223 > #11 0xffffffff805a4989 in _sx_xlock_hard (sx=0xffffff0043557d50, > tid=18446742975830720512, opts=Variable "opts" is not available. > ) > at /usr/src/sys/kern/kern_sx.c:575 > #12 0xffffffff805a52fe in _sx_xlock (sx=Variable "sx" is not > available. > ) at sx.h:155 > #13 0xffffffff80fe2995 in zfs_freebsd_reclaim () from /boot/kernel/ > zfs.ko > #14 0xffffffff808cefca in VOP_RECLAIM_APV (vop=0xffffff0043557d38, > a=0xffffff0043557d50) at vnode_if.c:1926 > #15 0xffffffff80626f6e in vgonel (vp=0xffffff00437a7938) at > vnode_if.h:830 > #16 0xffffffff8062b528 in vflush (mp=0xffffff0060f2a000, rootrefs=0, > flags=0, td=0xffffff0061528000) > at /usr/src/sys/kern/vfs_subr.c:2450 > #17 0xffffffff80fdd3a8 in zfs_umount () from /boot/kernel/zfs.ko > #18 0xffffffff8062420a in dounmount (mp=0xffffff0060f2a000, > flags=1626513408, td=Variable "td" is not available. > ) > at /usr/src/sys/kern/vfs_mount.c:1287 > #19 0xffffffff80624975 in unmount (td=0xffffff0061528000, > uap=0xffffff803e8b5c00) > at /usr/src/sys/kern/vfs_mount.c:1172 > #20 0xffffffff8088783f in syscall (frame=0xffffff803e8b5c90) at /usr/ > src/sys/amd64/amd64/trap.c:984 > #21 0xffffffff8086e290 in Xfast_syscall () at /usr/src/sys/amd64/ > amd64/exception.S:364 > #22 0x000000080104e49c in ?? () > Previous frame inner to this frame (corrupt stack?) > > BTW, I got a (one) "force unmount is experimental" on the console. > On regular shutdown I usually get one per filesystem, it seems (at > least 10) and this pool should contain exactly as many filesystems > as the root pool since it's a copy of it. On running the backup > script manually post-crash, though, I didn't get any. OK, I've finally written a script that reproduces this panic for me every time (6-7 tries in a row should be good enough, plus one on another box). It would be great to have a few testers - and if you do test it, PLEASE report your results here - positive or negative! The main aim is, of course, to provide ZFS devs with their own core dumps, DDB consoles and whatnot to possibly resolve this issue. It requires: * bash (or something compatible, for the script itself) * a box you're willing to crash. ;) * ~200MB free on your /root/ partition (or just edit the paths at the top of the ugly hack of a script.) If you use ggatel with silly high numbers (1482 and 1675 - chosen since they're unlikely to be used), again, edit at the top. * The libzfs_sendrecv.c patch - see http://lists.freebsd.org/pipermail/freebsd-current/2009-May/006814.html or fetch the patch from my server (if it's down, it's for less than 2 minutes - my modem restarts every 181 minutes atm...): http://exscape.org/temp/libzfs_sendrecv.patch Here's a oneliner to fetch, patch, compile and install: cd /usr/src && fetch http://exscape.org/temp/libzfs_sendrecv.patch && patch -p0 < libzfs_sendrecv.patch && cd /usr/src/cddl/lib/libzfs && make && make install No reboot required (nor do you need a reboot to revert, see the end of the mail). PLEASE NOTE that without this patch, you'll just get a segfault and then an infinite loop of backups (since send/recv doesn't work in HEAD since at least february, I'm guessing since ZFS v13). Too bad that the patch is needed, since I suspect quite a few testers will bail out there... If not (great!), here's the script to wreck havoc: http://exscape.org/temp/zfs_clone_panic.sh After fetching the above, just run the script like "bash zfs_clone_panic.sh crash" and you should have a panic in about 20 seconds. The other possiblee arguments are mostly there because I'm too lazy to clean it up now that it "works". Back to the panic: The problem appears to be related to clones somehow - the first two times I ran in to this panic (in real use) was when messing with clone/ promote... so that's what this script does. Here's the backtrace it produces, and some info (show lockedvnods, if that helps at all): kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x288 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff8036df39 stack pointer = 0x28:0xffffff803ea6d7e0 frame pointer = 0x28:0xffffff803ea6d840 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = resume, IOPL = 0 current process = 16367 (zpool) 0xffffff000510fce8: tag zfs, type VDIR usecount 1, writecount 0, refcount 1 mountedhere 0xffffff0002cd8bc0 flags () lock type zfs: EXCL by thread 0xffffff0024c5d000 (pid 16367) 0xffffff00050dc3b0: tag zfs, type VDIR usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VI_DOOMED) VI_LOCKed lock type zfs: EXCL by thread 0xffffff0024c5d000 (pid 16367) panic: from debugger ... ... boot, panic, trap etc ... #10 0xffffffff805d36a7 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:223 #11 0xffffffff8036df39 in _sx_xlock_hard (sx=0xffffff005d11a8e8, tid=18446742974814867456, opts=Variable "opts" is not available. ) at /usr/src/sys/kern/kern_sx.c:575 #12 0xffffffff8036e8ae in _sx_xlock (sx=Variable "sx" is not available. ) at sx.h:155 #13 0xffffffff80b889e5 in zfs_freebsd_reclaim () from /boot/kernel/ zfs.ko #14 0xffffffff8062447a in VOP_RECLAIM_APV (vop=0xffffff005d11a8d0, a=0xffffff005d11a8e8) at vnode_if.c:1926 #15 0xffffffff803f3b0e in vgonel (vp=0xffffff00050dc3b0) at vnode_if.h: 830 #16 0xffffffff803f80c8 in vflush (mp=0xffffff0002cd8bc0, rootrefs=0, flags=0, td=0xffffff0024c5d000) at /usr/src/sys/kern/vfs_subr.c:2449 #17 0xffffffff80b833d8 in zfs_umount () from /boot/kernel/zfs.ko #18 0xffffffff803f0d3a in dounmount (mp=0xffffff0002cd8bc0, flags=47025088, td=Variable "td" is not available. ) at /usr/src/sys/kern/vfs_mount.c:1289 #19 0xffffffff803f1568 in unmount (td=0xffffff0024c5d000, uap=0xffffff803ea6dc00) at /usr/src/sys/kern/vfs_mount.c:1174 #20 0xffffffff805ed4cf in syscall (frame=0xffffff803ea6dc90) at /usr/src/sys/amd64/amd64/trap.c:984 #21 0xffffffff805d3930 in Xfast_syscall () at /usr/src/sys/amd64/ amd64/exception.S:364 #22 0x000000080104e9ac in ?? () Previous frame inner to this frame (corrupt stack?) Regards, Thomas PS. A oneliner to get back to the non-patched state, if you're using SVN (if not, sorry): cd /usr/src && svn revert cddl/contrib/opensolaris/lib/libzfs/common/ libzfs_sendrecv.c && cd /usr/src/cddl/lib/libzfs && make && make install DS. Hope this helps track this down, as I spent quite a while on finding the root cause (the clones, in one way or another), writing the script, this mail, etc. From serenity at exscape.org Fri Jul 10 19:25:42 2009 From: serenity at exscape.org (Thomas Backman) Date: Fri Jul 10 19:25:54 2009 Subject: Reproducible ZFS panic, w/ script (Was: "New" ZFS crash on FS (pool?) unmount/export) In-Reply-To: References: <72163521-40BF-4764-8B74-5446A88DFBF8@exscape.org> Message-ID: <45291598-D091-4E90-B968-22E59BEB3846@exscape.org> On Jul 10, 2009, at 21:01, Thomas Backman wrote: > OK, I've finally written a script that reproduces this panic for me > every time (6-7 tries in a row should be good enough, plus one on > another box). It would be great to have a few testers - and if you > do test it, PLEASE report your results here - positive or negative! > The main aim is, of course, to provide ZFS devs with their own core > dumps, DDB consoles and whatnot to possibly resolve this issue. > [...] > Back to the panic: > The problem appears to be related to clones somehow - the first two > times I ran in to this panic (in real use) was when messing with > clone/promote... so that's what this script does. Damnit. Very sorry for the noise, but I just noticed that it IS NOT related to the clones. It crashes with the clone/promote lines (#77-78) commented out, too... Now I'm stumped as to where the issue is, I hope the previous mail can help track it down. A very similar setup runs every night, and it "only" crashes about one time in 10 or so. This crashes *every* time and I don't really see the difference between the commands the scripts run. Oh well... Regards, Thomas From kmacy at freebsd.org Fri Jul 10 19:27:46 2009 From: kmacy at freebsd.org (Kip Macy) Date: Fri Jul 10 19:27:57 2009 Subject: Reproducible ZFS panic, w/ script (Was: "New" ZFS crash on FS (pool?) unmount/export) In-Reply-To: <45291598-D091-4E90-B968-22E59BEB3846@exscape.org> References: <72163521-40BF-4764-8B74-5446A88DFBF8@exscape.org> <45291598-D091-4E90-B968-22E59BEB3846@exscape.org> Message-ID: <3c1674c90907101227ueab78eem6f8c5c7fdf0337cc@mail.gmail.com> "zfs export" does a forced unmount. We may not be properly handling dangling references. -Kip On Fri, Jul 10, 2009 at 12:25 PM, Thomas Backman wrote: > On Jul 10, 2009, at 21:01, Thomas Backman wrote: >> >> OK, I've finally written a script that reproduces this panic for me every >> time (6-7 tries in a row should be good enough, plus one on another box). It >> would be great to have a few testers - and if you do test it, PLEASE report >> your results here - positive or negative! >> The main aim is, of course, to provide ZFS devs with their own core dumps, >> DDB consoles and whatnot to possibly resolve this issue. >> [...] >> Back to the panic: >> The problem appears to be related to clones somehow - the first two times >> I ran in to this panic (in real use) was when messing with clone/promote... >> so that's what this script does. > > Damnit. Very sorry for the noise, but I just noticed that it IS NOT related > to the clones. It crashes with the clone/promote lines (#77-78) commented > out, too... > Now I'm stumped as to where the issue is, I hope the previous mail can help > track it down. > A very similar setup runs every night, and it "only" crashes about one time > in 10 or so. This crashes *every* time and I don't really see the difference > between the commands the scripts run. Oh well... > > Regards, > Thomas > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > -- When bad men combine, the good must associate; else they will fall one by one, an unpitied sacrifice in a contemptible struggle. Edmund Burke From serenity at exscape.org Fri Jul 10 20:01:30 2009 From: serenity at exscape.org (Thomas Backman) Date: Fri Jul 10 20:01:43 2009 Subject: Reproducible ZFS panic, w/ script (Was: "New" ZFS crash on FS (pool?) unmount/export) In-Reply-To: <3c1674c90907101227ueab78eem6f8c5c7fdf0337cc@mail.gmail.com> References: <72163521-40BF-4764-8B74-5446A88DFBF8@exscape.org> <45291598-D091-4E90-B968-22E59BEB3846@exscape.org> <3c1674c90907101227ueab78eem6f8c5c7fdf0337cc@mail.gmail.com> Message-ID: On Jul 10, 2009, at 21:27, Kip Macy wrote: > "zfs export" does a forced unmount. We may not be properly handling > dangling references. > > -Kip > > On Fri, Jul 10, 2009 at 12:25 PM, Thomas > Backman wrote: >> On Jul 10, 2009, at 21:01, Thomas Backman wrote: >> ... Just one more thing to add for me today: the crash always happens when exporting the slave. Constant send/recv loops multiple times a second, no sweat. import/export of both pools multiple times a second, without any send/recv in between them, no sweat. Combined, however, it panics on "zpool export crashtestslave". (I verified this twice, once by changing stress() to simply run loads of incremental backups for a for a few minutes, break, and export the pools manually. Both times, the master pool was no problem, and it immediately panics on exporting the slave.) Regards, Thomas From swell.k at gmail.com Sat Jul 11 02:30:03 2009 From: swell.k at gmail.com (Anonymous) Date: Sat Jul 11 02:30:10 2009 Subject: kern/129148: [zfs] [panic] panic on concurrent writing & rollback Message-ID: <200907110230.n6B2U3w8090380@freefall.freebsd.org> The following reply was made to PR kern/129148; it has been noted by GNATS. From: Anonymous To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/129148: [zfs] [panic] panic on concurrent writing & rollback Date: Sat, 11 Jul 2009 06:28:03 +0400 This panic makes rollback feature really unusable. I usually do: # cd /usr/src (svn checkout sources) # zfs snapshot q/usr/src@blah # zcat ~/some_big_patch.bz2 | patch -Efsp0 -F0 ... # zfs rollback q/usr/src@blah # zcat ~/another_big_patch.bz2 | patch -Efsp0 -F0 BANG! panics here Here is recent one for FreeBSD 8.0-BETA1 #0: Sat Jul 4 03:55:14 UTC 2009 root@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC i386 --- panic begins here --- $ qemu -no-kqemu -echr 3 -nographic \ -hda /dev/zvol/h/home/luser/freebsd-i386 \ -hdb /dev/zvol/h/home/luser/freebsd-i386-zpool [...] # zpool create q ad1 # sh crash.sh cannot open 'q/test': dataset does not exist load: 0.90 cmd: sh 66 [runnable] 2.53r 0.28u 1.92s 29% 1808k Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x4c fault code = supervisor read, page not present instruction pointer = 0x20:0xc087c9c3 stack pointer = 0x28:0xc89f5790 frame pointer = 0x28:0xc89f57b0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, IOPL = 0 current process = 66 (sh) [thread pid 66 tid 100041 ] Stopped at _sx_xlock+0x43: movl 0x10(%ebx),%eax db> show all locks Process 66 (sh) thread 0xc238db40 (100041) exclusive lockmgr zfs (zfs) r = 0 (0xc2613270) locked @ /usr/src/sys/kern/vfs_subr.c:880 exclusive lockmgr zfs (zfs) r = 0 (0xc2613594) locked @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:856 db> show lockedvnods Locked vnodes 0xc261353c: tag zfs, type VDIR usecount 2, writecount 0, refcount 2 mountedhere 0 flags (VV_ROOT) lock type zfs: EXCL by thread 0xc238db40 (pid 66) 0xc2613218: tag zfs, type VREG usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VI_DOOMED) lock type zfs: EXCL by thread 0xc238db40 (pid 66) db> show all pcpu Current CPU: 0 cpuid = 0 dynamic pcpu = 0x6aed54 curthread = 0xc238db40: pid 66 "sh" curpcb = 0xc89f5d90 fpcurthread = 0xc238db40: pid 66 "sh" idlethread = 0xc2156b40: pid 11 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: db> bt Tracing pid 66 tid 100041 td 0xc238db40 _sx_xlock(3c,0,c24ba14d,70f,c2617ae0,...) at _sx_xlock+0x43 dmu_buf_update_user(0,c2617ae0,0,0,0,...) at dmu_buf_update_user+0x35 zfs_znode_dmu_fini(c2617ae0,c24c2fed,1114,110b,c26d5000,...) at zfs_znode_dmu_fini+0x43 zfs_freebsd_reclaim(c89f5858,1,0,c2613218,c89f587c,...) at zfs_freebsd_reclaim+0xc0 VOP_RECLAIM_APV(c24c65a0,c89f5858,0,0,c261328c,...) at VOP_RECLAIM_APV+0xa5 vgonel(c261328c,0,c0c6567e,386,0,...) at vgonel+0x1a4 vnlru_free(c0f16ef0,0,c0c6567e,3a1,c2da47ac,...) at vnlru_free+0x2d5 getnewvnode(c24c0cfc,c237778c,c24c65a0,c89f58fc,c25baa80,...) at getnewvnode+0x4a zfs_znode_cache_constructor(c25bbe00,2,c24c1357,2fd,c2db8880,...) at zfs_znode_cache_constructor+0x2e zfs_znode_alloc(c26d5498,0,c24c1357,2fd,c89f5978,...) at zfs_znode_alloc+0x35 zfs_mknode(c2617bc8,c89f5a60,c2d97000,c2152080,0,...) at zfs_mknode+0x286 zfs_freebsd_create(c89f5ac8,c89f5ae0,0,0,c89f5ba8,...) at zfs_freebsd_create+0x722 VOP_CREATE_APV(c24c65a0,c89f5ac8,c89f5bd4,c89f5a60,0,...) at VOP_CREATE_APV+0xa5 vn_open_cred(c89f5ba8,c89f5c5c,1a4,0,c2152080,...) at vn_open_cred+0x200 vn_open(c89f5ba8,c89f5c5c,1a4,c238f310,14,...) at vn_open+0x3b kern_openat(c238db40,ffffff9c,28304378,0,602,...) at kern_openat+0x118 kern_open(c238db40,28304378,0,601,1b6,...) at kern_open+0x35 open(c238db40,c89f5cf8,c,c0c5ee4b,c0d3c0ac,...) at open+0x30 syscall(c89f5d38) at syscall+0x2a3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (5, FreeBSD ELF32, open), eip = 0x281d8fe3, esp = 0xbfbfe6cc, ebp = 0xbfbfe768 --- db> ps pid ppid pgrp uid state wmesg wchan cmd 69 0 0 0 SL tq->tq_d 0xc257141c [zil_clean] 68 67 60 0 S+ tx->tx_s 0xc27599e0 zfs 67 60 60 0 S+ wait 0xc25de7f8 sh 66 60 60 0 R+ CPU 0 sh 60 19 60 0 S+ wait 0xc25ded48 sh 57 0 0 0 SL tq->tq_d 0xc25715a4 [zil_clean] 54 0 0 0 SL zio->io_ 0xc265ab94 [txg_thread_enter] 53 0 0 0 SL tx->tx_q 0xc27599e8 [txg_thread_enter] 52 0 0 0 RL [vdev:worker ad1] 51 0 0 0 SL tq->tq_d 0xc2571668 [spa_zio] 50 0 0 0 SL tq->tq_d 0xc257172c [spa_zio] 49 0 0 0 SL tq->tq_d 0xc25717f0 [spa_zio] 48 0 0 0 SL tq->tq_d 0xc25718b4 [spa_zio] 47 0 0 0 SL tq->tq_d 0xc2571978 [spa_zio] 46 0 0 0 SL tq->tq_d 0xc2571a3c [spa_zio] 45 0 0 0 SL tq->tq_d 0xc2571b00 [spa_zio] 44 0 0 0 SL tq->tq_d 0xc2571bc4 [spa_zio] 43 0 0 0 SL tq->tq_d 0xc2571bc4 [spa_zio] 42 0 0 0 SL tq->tq_d 0xc2571bc4 [spa_zio] 41 0 0 0 SL tq->tq_d 0xc2571bc4 [spa_zio] 40 0 0 0 SL tq->tq_d 0xc2571bc4 [spa_zio] 39 0 0 0 SL tq->tq_d 0xc2571bc4 [spa_zio] 38 0 0 0 SL tq->tq_d 0xc2571bc4 [spa_zio] 37 0 0 0 SL tq->tq_d 0xc2571bc4 [spa_zio] 36 0 0 0 SL tq->tq_d 0xc2571c88 [spa_zio] 35 0 0 0 SL tq->tq_d 0xc2571c88 [spa_zio] 34 0 0 0 SL tq->tq_d 0xc2571c88 [spa_zio] 33 0 0 0 SL tq->tq_d 0xc2571c88 [spa_zio] 32 0 0 0 SL tq->tq_d 0xc2571c88 [spa_zio] 31 0 0 0 SL tq->tq_d 0xc2571c88 [spa_zio] 30 0 0 0 SL tq->tq_d 0xc2571c88 [spa_zio] 29 0 0 0 SL tq->tq_d 0xc2571c88 [spa_zio] 28 0 0 0 SL tq->tq_d 0xc2571d4c [spa_zio] 27 0 0 0 SL tq->tq_d 0xc2571e10 [spa_zio] 26 0 0 0 SL tq->tq_d 0xc2571ed4 [spa_zio] 25 0 0 0 RL [l2arc_feed_thread] 24 0 0 0 RL [arc_reclaim_thread] 23 0 0 0 RL [vaclean] 22 0 0 0 SL tq->tq_d 0xc2572048 [system_taskq] 19 1 19 0 Ss+ wait 0xc2376550 sh 18 0 0 0 SL flowclea 0xc0daa4a4 [flowcleaner] 17 0 0 0 RL [softdepflush] 16 0 0 0 RL [vnlru] 15 0 0 0 RL [syncer] 14 0 0 0 RL [bufdaemon] 9 0 0 0 SL pgzero 0xc0f23314 [pagezero] 8 0 0 0 SL psleep 0xc0f22f3c [vmdaemon] 7 0 0 0 RL [pagedaemon] 6 0 0 0 SL waiting_ 0xc0f18d5c [sctp_iterator] 5 0 0 0 SL ccb_scan 0xc0d76fd4 [xpt_thrd] 13 0 0 0 SL - 0xc0daa4a4 [yarrow] 4 0 0 0 SL - 0xc0da8264 [g_down] 3 0 0 0 SL - 0xc0da8260 [g_up] 2 0 0 0 SL - 0xc0da8258 [g_event] 12 0 0 0 WL (threaded) intr 100029 I [swi0: uart] 100028 I [irq7: ppc0] 100027 I [irq12: psm0] 100026 I [irq1: atkbd0] 100025 I [irq11: ed0] 100024 I [irq15: ata1] 100023 I [irq14: ata0] 100022 I [irq9: acpi0] 100021 I [swi6: task queue] 100020 I [swi6: Giant taskq] 100018 I [swi5: +] 100013 I [swi2: cambio] 100006 I [swi3: vm] 100005 I [swi4: clock] 100004 I [swi1: netisr 0] 11 0 0 0 RL [idle: cpu0] 1 0 1 0 SLs wait 0xc2154d48 [init] 10 0 0 0 SL audit_wo 0xc0f21f80 [audit] 0 0 0 0 SLs (threaded) kernel 100019 D - 0xc2220d40 [thread taskq] 100017 D - 0xc2221100 [kqueue taskq] 100016 D - 0xc2221140 [acpi_task_2] 100015 D - 0xc2221140 [acpi_task_1] 100014 D - 0xc2221140 [acpi_task_0] 100010 D - 0xc2138940 [firmware taskq] 100000 D sched 0xc0da8320 [swapper] --- panic ends here --- From serenity at exscape.org Sat Jul 11 14:08:50 2009 From: serenity at exscape.org (Thomas Backman) Date: Sat Jul 11 14:08:56 2009 Subject: Reproducible ZFS panic, w/ script (Was: "New" ZFS crash on FS (pool?) unmount/export) In-Reply-To: <3c1674c90907101227ueab78eem6f8c5c7fdf0337cc@mail.gmail.com> References: <72163521-40BF-4764-8B74-5446A88DFBF8@exscape.org> <45291598-D091-4E90-B968-22E59BEB3846@exscape.org> <3c1674c90907101227ueab78eem6f8c5c7fdf0337cc@mail.gmail.com> Message-ID: On Jul 10, 2009, at 21:27, Kip Macy wrote: > "zfs export" does a forced unmount. We may not be properly handling > dangling references. > > -Kip A bit more digging: [root@chaos ~]# bash zfs_crash.sh initial [root@chaos ~]# bash zfs_crash.sh stress ## with the unmount part (line 107) **commented out** I then let the above run for say 20 seconds to create a bunch of snapshots (ignoring errors; in my own script I added a random number to the snapshot name to avoid collisions), and then: [root@chaos ~]# zpool export crashtestmaster [root@chaos ~]# zfs list NAME USED AVAIL REFER MOUNTPOINT crashtestslave 20.3M 40.7M 20K /crashtestslave/ crashtestslave crashtestslave/test_cloned 19.8M 40.7M 19.8M /crashtestslave/ crashtestslave/test_cloned crashtestslave/test_orig 0 40.7M 19.8M /crashtestslave/ crashtestslave/test_orig tank 5.67G 59.3G 18K none tank/root 616M 59.3G 224M / tank/... [root@chaos ~]# zfs unmount crashtestslave/test_orig [root@chaos ~]# zfs unmount crashtestslave/test_cloned [root@chaos ~]# zfs unmount crashtestslave ... panic here. Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0xc fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff803a5682 stack pointer = 0x28:0xffffff803ea09980 frame pointer = 0x28:0xffffff803ea099b0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = resume, IOPL = 0 current process = 5099 (zfs) 0xffffff002ac4a938: tag zfs, type VDIR usecount 1, writecount 0, refcount 1 mountedhere 0xffffff00068be8d0 flags () lock type zfs: EXCL by thread 0xffffff0006f13390 (pid 5099) BT: ... #9 0xffffffff805edc42 in trap (frame=0xffffff803ea098d0) at /usr/src/ sys/amd64/amd64/trap.c:345 #10 0xffffffff805d36a7 in calltrap () at /usr/src/sys/amd64/amd64/ exception.S:223 #11 0xffffffff803a5682 in propagate_priority (td=0xffffff0027174ab0) at /usr/src/sys/kern/subr_turnstile.c:194 #12 0xffffffff803a64ec in turnstile_wait (ts=Variable "ts" is not available. ) at /usr/src/sys/kern/subr_turnstile.c:738 #13 0xffffffff80355101 in _mtx_lock_sleep (m=0xffffff002ca6d9f8, tid=18446742974314394512, opts=Variable "opts" is not available. ) at /usr/src/sys/kern/kern_mutex.c:447 #14 0xffffffff803f7893 in vfs_msync (mp=0xffffff00068be8d0, flags=1) at /usr/src/sys/kern/vfs_subr.c:3179 #15 0xffffffff803f0c7e in dounmount (mp=0xffffff00068be8d0, flags=0, td=Variable "td" is not available. ) at /usr/src/sys/kern/vfs_mount.c:1263 #16 0xffffffff803f1568 in unmount (td=0xffffff0006f13390, uap=0xffffff803ea09c00) at /usr/src/sys/kern/vfs_mount.c:1174 #17 0xffffffff805ed4cf in syscall (frame=0xffffff803ea09c90) at /usr/ src/sys/amd64/amd64/trap.c:984 #18 0xffffffff805d3930 in Xfast_syscall () at /usr/src/sys/amd64/amd64/ exception.S:364 #19 0x0000000800f4b9ac in ?? () Previous frame inner to this frame (corrupt stack?) NOT the same backtrace as before (nothing after dounmount() is the same as the zpool export panic), and this time from zfs unmount, not zpool export. I tried it again, and got another backtrace(!) - it "ends" (or begins, depending on your view) with propagate_priority(), turnstile_wait() and _mtx_lock_sleep() in both cases, though. Here's the second, which happened while doing the same as above - initial, stress and then manually zfs unmount the them. "zfs unmount crashtestslave" (the root fs) is what panics yet again: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0xc fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff803aa722 stack pointer = 0x28:0xffffff8000025a60 frame pointer = 0x28:0xffffff8000025a90 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = resume, IOPL = 0 current process = 12 (swi4: clock) ... #8 0xffffffff805f1fcd in trap_fatal (frame=0xffffff80000259b0, eva=Variable "eva" is not available. ) at /usr/src/sys/amd64/amd64/trap.c:847 #9 0xffffffff805f2e22 in trap (frame=0xffffff80000259b0) at /usr/src/ sys/amd64/amd64/trap.c:345 #10 0xffffffff805d87c7 in calltrap () at /usr/src/sys/amd64/amd64/ exception.S:224 #11 0xffffffff803aa722 in propagate_priority (td=0xffffff00296ce390) at /usr/src/sys/kern/subr_turnstile.c:194 #12 0xffffffff803ab58c in turnstile_wait (ts=Variable "ts" is not available. ) at /usr/src/sys/kern/subr_turnstile.c:738 #13 0xffffffff8035a1c1 in _mtx_lock_sleep (m=0xffffffff808a1de0, tid=18446742974234830624, opts=Variable "opts" is not available. ) at /usr/src/sys/kern/kern_mutex.c:447 #14 0xffffffff8037ea92 in softclock (arg=Variable "arg" is not available. ) at /usr/src/sys/kern/kern_timeout.c:376 #15 0xffffffff803417b0 in intr_event_execute_handlers (p=Variable "p" is not available. ) at /usr/src/sys/kern/kern_intr.c:1165 #16 0xffffffff80342d1e in ithread_loop (arg=0xffffff000231e6a0) at / usr/src/sys/kern/kern_intr.c:1178 #17 0xffffffff8033ebb8 in fork_exit (callout=0xffffffff80342c90 , arg=0xffffff000231e6a0, frame=0xffffff8000025c80) at /usr/src/sys/kern/kern_fork.c:842 #18 0xffffffff805d8c9e in fork_trampoline () at /usr/src/sys/amd64/ amd64/exception.S:561 #19 0x0000000000000000 in ?? () #20 0x0000000000000000 in ?? () #21 0x0000000000000001 in ?? () #22 0x0000000000000000 in ?? () #23 0x0000000000000000 in ?? () #24 0x0000000000000000 in ?? () #25 0x0000000000000000 in ?? () Note that the active process is *not* zfs this time. Regards, Thomas From noackjr at alumni.rice.edu Sun Jul 12 07:06:15 2009 From: noackjr at alumni.rice.edu (Jonathan Noack) Date: Sun Jul 12 07:06:22 2009 Subject: Booting from ZFS raidz In-Reply-To: References: <9461581F-F354-486D-961D-3FD5B1EF007C@rabson.org> <20090201072432.GA25276@server.vk2pj.dyndns.org> <246ecf0c87f944d70c5562eeed4165c9@mail.rabson.org> <9cc826f0720e1624489dd6e6d384babc.squirrel@www.noacks.org> Message-ID: On Sun, June 14, 2009 00:09, Jonathan Noack wrote: > On Fri, May 15, 2009 19:07, Jonathan Noack wrote: >> On Thu, May 14, 2009 10:25, Doug Rabson wrote: >>> I fixed a bug in the patch. Try this version: >>> http://people.freebsd.org/~dfr/raidzboot-14052009.diff >> >> I know the bug fix was for booting from degraded pools, but I can at >> least >> give you a "no regression" report. I just set up a new amd64 box and >> was >> able to boot from a raidz1 pool using your latest patch. >> >> Getting this working from scratch was tedious but not too complicated. >> I >> followed lulf's instructions >> (http://blogs.freebsdish.org/lulf/2008/12/16/setting-up-a-zfs-only-system/) >> using the May snapshot fixit CD. Only differences were that I set up >> all >> 4 disks with gpart (identically), created a raidz1 pool, and used a >> patched gptzfsboot that I cross-compiled on my 7.2 i386 box for the >> bootcode (applied to all 4 disks). >> >> If only I had remembered to patch my /usr/src tree before rebuilding >> world >> and rebooting... *sigh* Once more unto the fixit breach... :) > > This (and the committed version) had been working fine for me on my stock > amd64 CURRENT system until I rebuilt world/kernel on 5/30 and rebooted. I > get the following error on boot (hand transcribed so hopefully I didn't > screw it up): > ************************************************************ > ZFS: i/o error - all block copies unavailable > ZFS: can't read object set for dataset lld > Can't find root filesystem - giving up > ZFS: unexpected object set type lld > ZFS: unexpected object set type lld > > FreeBSD/i386 boot > Default: tank:/boot/kernel/kernerl > boot: > ZFS: unexpected object set type lld > > FreeBSD/i386 boot > Default: tank:/boot/kernel/kernel > boot: > ************************************************************ > > The previously working world/kernel was from 5/26. I haven't had much > time to troubleshoot until today. I can use the fixit CD to access the > ZFS pool with no issues; the problem appears to just be the boot code. I > cross-built a fresh world on my i386 system today, reinstalled everything > in /boot, reinstalled gptzfsboot, and still got the same results. > > What steps should I take to troubleshoot and resolve this? Anyone know how I should proceed? I'm really frustrated that I haven't been able to get this working. Thanks, -Jon From kmacy at freebsd.org Sun Jul 12 21:27:04 2009 From: kmacy at freebsd.org (Kip Macy) Date: Sun Jul 12 21:27:16 2009 Subject: Reproducible ZFS panic, w/ script (Was: "New" ZFS crash on FS (pool?) unmount/export) In-Reply-To: References: <72163521-40BF-4764-8B74-5446A88DFBF8@exscape.org> <45291598-D091-4E90-B968-22E59BEB3846@exscape.org> <3c1674c90907101227ueab78eem6f8c5c7fdf0337cc@mail.gmail.com> Message-ID: <3c1674c90907121427u5206d398pfcde8bb1cfb15f93@mail.gmail.com> This is caused by the timeout thread trying to execute a freed callout. It is likely just another instance of poor cleanup. -Kip > > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address ? = 0xc > fault code ? ? ? ? ? ? ?= supervisor read data, page not present > instruction pointer ? ? = 0x20:0xffffffff803aa722 > stack pointer ? ? ? ? ? = 0x28:0xffffff8000025a60 > frame pointer ? ? ? ? ? = 0x28:0xffffff8000025a90 > code segment ? ? ? ? ? ?= base 0x0, limit 0xfffff, type 0x1b > ? ? ? ? ? ? ? ? ? ? ? ?= DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags ? ? ? ?= resume, IOPL = 0 > current process ? ? ? ? = 12 (swi4: clock) > > ... > #8 ?0xffffffff805f1fcd in trap_fatal (frame=0xffffff80000259b0, eva=Variable > "eva" is not available. > ) at /usr/src/sys/amd64/amd64/trap.c:847 > #9 ?0xffffffff805f2e22 in trap (frame=0xffffff80000259b0) at > /usr/src/sys/amd64/amd64/trap.c:345 > #10 0xffffffff805d87c7 in calltrap () at > /usr/src/sys/amd64/amd64/exception.S:224 > #11 0xffffffff803aa722 in propagate_priority (td=0xffffff00296ce390) at > /usr/src/sys/kern/subr_turnstile.c:194 > #12 0xffffffff803ab58c in turnstile_wait (ts=Variable "ts" is not available. > ) at /usr/src/sys/kern/subr_turnstile.c:738 > #13 0xffffffff8035a1c1 in _mtx_lock_sleep (m=0xffffffff808a1de0, > tid=18446742974234830624, opts=Variable "opts" is not available. > ) > ? ?at /usr/src/sys/kern/kern_mutex.c:447 > #14 0xffffffff8037ea92 in softclock (arg=Variable "arg" is not available. > ) at /usr/src/sys/kern/kern_timeout.c:376 > #15 0xffffffff803417b0 in intr_event_execute_handlers (p=Variable "p" is not > available. > ) at /usr/src/sys/kern/kern_intr.c:1165 > #16 0xffffffff80342d1e in ithread_loop (arg=0xffffff000231e6a0) at > /usr/src/sys/kern/kern_intr.c:1178 > #17 0xffffffff8033ebb8 in fork_exit (callout=0xffffffff80342c90 > , arg=0xffffff000231e6a0, > ? ?frame=0xffffff8000025c80) at /usr/src/sys/kern/kern_fork.c:842 > #18 0xffffffff805d8c9e in fork_trampoline () at > /usr/src/sys/amd64/amd64/exception.S:561 > #19 0x0000000000000000 in ?? () > #20 0x0000000000000000 in ?? () > #21 0x0000000000000001 in ?? () > #22 0x0000000000000000 in ?? () > #23 0x0000000000000000 in ?? () > #24 0x0000000000000000 in ?? () > #25 0x0000000000000000 in ?? () > > > Note that the active process is *not* zfs this time. > > Regards, > Thomas > -- When bad men combine, the good must associate; else they will fall one by one, an unpitied sacrifice in a contemptible struggle. Edmund Burke From bugmaster at FreeBSD.org Mon Jul 13 11:06:54 2009 From: bugmaster at FreeBSD.org (FreeBSD bugmaster) Date: Mon Jul 13 11:08:09 2009 Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org Message-ID: <200907131106.n6DB6rOY040579@freefall.freebsd.org> Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/136601 fs [zfs] tar: Couldn't list extended attributes: Read-onl o kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/136218 fs [zfs] Exported ZFS pools can't be imported into (Open) o kern/135594 fs [zfs] Single dataset unresponsive with Samba o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135480 fs [zfs] panic: lock &arg.lock already initialized o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o bin/135314 fs [zfs] assertion failed for zdb(8) usage o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot f kern/134496 fs [zfs] [panic] ZFS pool export occasionally causes a ke o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133980 fs [panic] [ffs] panic: ffs_valloc: dup alloc o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis o kern/133614 fs [smbfs] [panic] panic: ffs_truncate: read-only filesys o kern/133373 fs [zfs] umass attachment causes ZFS checksum errors, dat o kern/133174 fs [msdosfs] [patch] msdosfs must support utf-encoded int f kern/133150 fs [zfs] Page fault with ZFS on 7.1-RELEASE/amd64 while w o kern/133134 fs [zfs] Missing ZFS zpool labels f kern/133020 fs [zfs] [panic] inappropriate panic caused by zfs. Pani o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132597 fs [tmpfs] [panic] tmpfs-related panic while interrupting o kern/132551 fs [zfs] ZFS locks up on extattr_list_link syscall o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132337 fs [zfs] [panic] kernel panic in zfs_fuid_create_cred o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes f kern/132068 fs [zfs] page fault when using ZFS over NFS on 7.1-RELEAS o kern/131995 fs [nfs] Failure to mount NFSv4 server o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/131086 fs [ext2fs] [patch] mkfs.ext2 creates rotten partition o kern/130979 fs [smbfs] [panic] boot/kernel/smbfs.ko o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130229 fs [iconv] usermount fails on fs that need iconv o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/129148 fs [zfs] [panic] panic on concurrent writing & rollback o kern/129059 fs [zfs] [patch] ZFS bootloader whitelistable via WITHOUT f kern/128829 fs smbd(8) causes periodic panic on 7-RELEASE o kern/128633 fs [zfs] [lor] lock order reversal in zfs o kern/128514 fs [zfs] [mpt] problems with ZFS and LSILogic SAS/SATA Ad f kern/128173 fs [ext2fs] ls gives "Input/output error" on mounted ext3 o kern/127659 fs [tmpfs] tmpfs memory leak o kern/127492 fs [zfs] System hang on ZFS input-output o kern/127420 fs [gjournal] [panic] Journal overflow on gmirrored gjour o kern/127213 fs [tmpfs] sendfile on tmpfs data corruption o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS o kern/125644 fs [zfs] [panic] zfs unfixable fs errors caused panic whe f kern/125536 fs [ext2fs] ext 2 mounts cleanly but fails on commands li o kern/125149 fs [nfs] [panic] changing into .zfs dir from nfs client c f kern/124621 fs [ext3] [patch] Cannot mount ext2fs partition f bin/124424 fs [zfs] zfs(8): zfs list -r shows strange snapshots' siz o kern/123939 fs [msdosfs] corrupts new files o kern/122888 fs [zfs] zfs hang w/ prefetch on, zil off while running t o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o kern/122173 fs [zfs] [panic] Kernel Panic if attempting to replace a o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o kern/122047 fs [ext2fs] [patch] incorrect handling of UF_IMMUTABLE / o kern/122038 fs [tmpfs] [panic] tmpfs: panic: tmpfs_alloc_vp: type 0xc o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121779 fs [ufs] snapinfo(8) (and related tools?) only work for t o kern/121770 fs [zfs] ZFS on i386, large file or heavy I/O leads to ke o bin/121366 fs [zfs] [patch] Automatic disk scrubbing from periodic(8 o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha f kern/120991 fs [panic] [fs] [snapshot] System crashes when manipulati o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F o bin/120288 fs zfs(8): "zfs share -a" does not send SIGHUP to mountd f kern/119735 fs [zfs] geli + ZFS + samba starting on boot panics 7.0-B o kern/118912 fs [2tb] disk sizing/geometry problem with large array o misc/118855 fs [zfs] ZFS-related commands are nonfunctional in fixit o kern/118713 fs [minidump] [patch] Display media size required for a k o kern/118320 fs [zfs] [patch] NFS SETATTR sometimes fails to set file o bin/118249 fs mv(1): moving a directory changes its mtime o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117314 fs [ntfs] Long-filename only NTFS fs'es cause kernel pani o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o kern/116913 fs [ffs] [panic] ffs_blkfree: freeing free block p kern/116608 fs [msdosfs] [patch] msdosfs fails to check mount options o kern/116583 fs [ffs] [hang] System freezes for short time when using o kern/116170 fs [panic] Kernel panic when mounting /tmp o kern/115645 fs [snapshots] [panic] lockmgr: thread 0xc4c00d80, not ex o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o kern/113180 fs [zfs] Setting ZFS nfsshare property does not cause inh o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o kern/109024 fs [msdosfs] mount_msdosfs: msdosfs_iconv: Operation not o kern/109010 fs [msdosfs] can't mv directory within fat32 file system o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106030 fs [ufs] [panic] panic in ufs from geom when a dead disk o kern/105093 fs [ext2fs] [patch] ext2fs on read-only media cannot be m o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [iso9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna f kern/91568 fs [ufs] [panic] writing to UFS/softupdates DVD media in o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/89991 fs [ufs] softupdates with mount -ur causes fs UNREFS o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88266 fs [smbfs] smbfs does not implement UIO_NOCOPY and sendfi o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o kern/85326 fs [smbfs] [panic] saving a file via samba to an overquot o kern/84589 fs [2TB] 5.4-STABLE unresponsive during background fsck 2 o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o kern/77826 fs [ext2fs] ext2fs usb filesystem will not mount RW o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/51583 fs [nullfs] [patch] allow to work with devices and socket o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 145 problems total. From jh at saunalahti.fi Tue Jul 14 18:25:25 2009 From: jh at saunalahti.fi (Jaakko Heinonen) Date: Tue Jul 14 18:25:31 2009 Subject: VOP_WRITE & read-only file system In-Reply-To: <20090619094316.GA805@a91-153-125-115.elisa-laajakaista.fi> References: <20090527150258.GA3666@a91-153-125-115.elisa-laajakaista.fi> <20090619094316.GA805@a91-153-125-115.elisa-laajakaista.fi> Message-ID: <20090714182522.GA40040@a91-153-125-115.elisa-laajakaista.fi> On 2009-06-19, Jaakko Heinonen wrote: > > I found a few ways to get VOP_WRITE called for a read-only system. > > > > 2) ktrace(2) - start ktracing a process - remount file-system as > > read-only > > ktrace too has a problem with writecount. ktrace uses vn_open() to > open the trace file but immediately after that it calls vn_close() > which decreases the writecount. I wrote a couple of patches for ktrace(2). > - Fiddle with writecount. IMHO it wouldn't fix the real bug (write > after vn_close()). http://www.saunalahti.fi/~jh3/patches/ktrace-writecount.diff The patch just increases v_writecount for trace file vnodes. It's quite straightforward and fixes the problem at hand but writes will still happen after vn_close()/VOP_CLOSE(). > - Keep track of vnodes which are used for tracing and have reference > count for them. http://www.saunalahti.fi/~jh3/patches/ktrace-vn_close.diff The patch uses a list to keep track of trace files. A file is closed when reference count hits zero. I am slightly concerned about scalability of the list approach but otherwise this should be more complete fix. -- Jaakko From tim.cleaver at gmail.com Wed Jul 15 00:40:03 2009 From: tim.cleaver at gmail.com (Timothy Cleaver) Date: Wed Jul 15 00:40:15 2009 Subject: kern/132145: [panic] File System Hard Crashes Message-ID: <200907150040.n6F0e291035940@freefall.freebsd.org> The following reply was made to PR kern/132145; it has been noted by GNATS. From: Timothy Cleaver To: bug-followup@FreeBSD.org, timcleaver@hotmail.com Cc: Subject: Re: kern/132145: [panic] File System Hard Crashes Date: Wed, 15 Jul 2009 10:10:57 +1000 Just to let you know, this is an ongoing problem for me. Although I haven't had a request for more information, I will supply some in the hope it sparks some interest. I am at a loss as to how to fix it. It has persisted with the move to 7.2-STABLE. Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/ad0s1f 177G 56G 107G 35% 771547 23251491 3% /usr ******* Working on device /dev/ad0 ******* parameters extracted from in-core disklabel are: cylinders=387621 heads=16 sectors/track=63 (1008 blks/cyl) Figures below won't work with BIOS for partitions not in cyl 1 parameters to be used for BIOS calculations are: cylinders=387621 heads=16 sectors/track=63 (1008 blks/cyl) Media sector size is 512 Warning: BIOS sector numbering starts with sector 1 Information from DOS bootblock is: The data for partition 1 is: sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD) start 63, size 390716802 (190779 Meg), flag 80 (active) beg: cyl 0/ head 1/ sector 1; end: cyl 1023/ head 254/ sector 63 The data for partition 2 is: The data for partition 3 is: The data for partition 4 is: Dump header from device /dev/ad0s1b Architecture: i386 Architecture Version: 2 Dump Length: 133369856B (127 MB) Blocksize: 512 Dumptime: Tue May 26 20:41:37 2009 Hostname: pc078650.workstation.griffith.edu.au Magic: FreeBSD Kernel Dump Version String: FreeBSD 7.2-RELEASE #0: Fri May 22 14:09:08 EST 2009 root@pc078650.workstation.griffith.edu.au:/usr/obj/usr/src/sys/KERNCONF Panic String: ffs_blkfree: freeing free block Dump Parity: 668566608 Bounds: 0 Dump Status: good Dump header from device /dev/ad0s1b Architecture: i386 Architecture Version: 2 Dump Length: 140001280B (133 MB) Blocksize: 512 Dumptime: Sun Jul 12 18:41:30 2009 Hostname: pc078650.workstation.griffith.edu.au Magic: FreeBSD Kernel Dump Version String: FreeBSD 7.2-RELEASE #1: Wed Jun 3 11:36:04 EST 2009 root@pc078650.workstation.griffith.edu.au:/usr/obj/usr/src/sys/KERNCONF Panic String: ffs_valloc: dup alloc Dump Parity: 2816038494 Bounds: 1 Dump Status: good GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... Unread portion of the kernel message buffer: mode = 040755, inum = 6101401, fs = /usr panic: ffs_valloc: dup alloc cpuid = 0 Uptime: 1d4h34m8s Physical memory: 494 MB Dumping 133 MB: 118 102 86 70 54 38 22 6 Reading symbols from /boot/kernel/sound.ko...Reading symbols from /boot/kernel/sound.ko.symbols...done. done. Loaded symbols for /boot/kernel/sound.ko Reading symbols from /boot/kernel/snd_ich.ko...Reading symbols from /boot/kernel/snd_ich.ko.symbols...done. done. Loaded symbols for /boot/kernel/snd_ich.ko Reading symbols from /boot/kernel/i915.ko...Reading symbols from /boot/kernel/i915.ko.symbols...done. done. Loaded symbols for /boot/kernel/i915.ko Reading symbols from /boot/kernel/drm.ko...Reading symbols from /boot/kernel/drm.ko.symbols...done. done. Loaded symbols for /boot/kernel/drm.ko Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/kernel/acpi.ko.symbols...done. done. Loaded symbols for /boot/kernel/acpi.ko Reading symbols from /boot/kernel/linprocfs.ko...Reading symbols from /boot/kernel/linprocfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/linprocfs.ko Reading symbols from /boot/kernel/linux.ko...Reading symbols from /boot/kernel/linux.ko.symbols...done. done. Loaded symbols for /boot/kernel/linux.ko Reading symbols from /boot/kernel/logo_saver.ko...Reading symbols from /boot/kernel/logo_saver.ko.symbols...done. done. Loaded symbols for /boot/kernel/logo_saver.ko #0 doadump () at pcpu.h:196 196 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); (kgdb) bt #0 doadump () at pcpu.h:196 #1 0xc057832c in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418 #2 0xc05785d9 in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:574 #3 0xc0747b02 in ffs_valloc (pvp=0xc40df228, mode=33188, cred=0xc4888500, vpp=0xd66f5914) at /usr/src/sys/ufs/ffs/ffs_alloc.c:987 #4 0xc07769c5 in ufs_makeinode (mode=33188, dvp=0xc40df228, vpp=0xd66f5b90, cnp=0xd66f5ba4) at /usr/src/sys/ufs/ufs/ufs_vnops.c:2252 #5 0xc07770e0 in ufs_create (ap=0xd66f5a88) at /usr/src/sys/ufs/ufs/ufs_vnops.c:193 #6 0xc07f0ef2 in VOP_CREATE_APV (vop=0xc0872420, a=0xd66f5a88) at vnode_if.c:206 #7 0xc0606210 in vn_open_cred (ndp=0xd66f5b7c, flagp=0xd66f5c78, cmode=) at vnode_if.h:112 #8 0xc06065e3 in vn_open (ndp=0xd66f5b7c, flagp=0xd66f5c78, cmode=420, fp=0xc357c0e4) at /usr/src/sys/kern/vfs_vnops.c:94 #9 0xc0603d13 in kern_open (td=0xc35bf8c0, path=0x811fbf0
, pathseg=UIO_USERSPACE, flags=1538, mode=438) at /usr/src/sys/kern/vfs_syscalls.c:1042 #10 0xc0604280 in open (td=0xc35bf8c0, uap=0xd66f5cfc) at /usr/src/sys/kern/vfs_syscalls.c:1009 #11 0xc07daff5 in syscall (frame=0xd66f5d38) at /usr/src/sys/i386/i386/trap.c:1090 #12 0xc07bfdc0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255 #13 0x00000033 in ?? () (kgdb) q Dump header from device /dev/ad0s1b Architecture: i386 Architecture Version: 2 Dump Length: 149962752B (143 MB) Blocksize: 512 Dumptime: Tue Jul 14 20:40:28 2009 Hostname: pc078650.workstation.griffith.edu.au Magic: FreeBSD Kernel Dump Version String: FreeBSD 7.2-RELEASE #1: Wed Jun 3 11:36:04 EST 2009 root@pc078650.workstation.griffith.edu.au:/usr/obj/usr/src/sys/KERNCONF Panic String: ffs_valloc: dup alloc Dump Parity: 2165898078 Bounds: 2 Dump Status: good GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... Unread portion of the kernel message buffer: mode = 0100644, inum = 2639003, fs = /usr panic: ffs_valloc: dup alloc cpuid = 0 Uptime: 2d1h44m31s Physical memory: 494 MB Dumping 143 MB: 128 112 96 80 64 48 32 16 Reading symbols from /boot/kernel/sound.ko...Reading symbols from /boot/kernel/sound.ko.symbols...done. done. Loaded symbols for /boot/kernel/sound.ko Reading symbols from /boot/kernel/snd_ich.ko...Reading symbols from /boot/kernel/snd_ich.ko.symbols...done. done. Loaded symbols for /boot/kernel/snd_ich.ko Reading symbols from /boot/kernel/i915.ko...Reading symbols from /boot/kernel/i915.ko.symbols...done. done. Loaded symbols for /boot/kernel/i915.ko Reading symbols from /boot/kernel/drm.ko...Reading symbols from /boot/kernel/drm.ko.symbols...done. done. Loaded symbols for /boot/kernel/drm.ko Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/kernel/acpi.ko.symbols...done. done. Loaded symbols for /boot/kernel/acpi.ko Reading symbols from /boot/kernel/linprocfs.ko...Reading symbols from /boot/kernel/linprocfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/linprocfs.ko Reading symbols from /boot/kernel/linux.ko...Reading symbols from /boot/kernel/linux.ko.symbols...done. done. Loaded symbols for /boot/kernel/linux.ko Reading symbols from /boot/kernel/logo_saver.ko...Reading symbols from /boot/kernel/logo_saver.ko.symbols...done. done. Loaded symbols for /boot/kernel/logo_saver.ko #0 doadump () at pcpu.h:196 196 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); (kgdb) bt #0 doadump () at pcpu.h:196 #1 0xc057832c in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418 #2 0xc05785d9 in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:574 #3 0xc0747b02 in ffs_valloc (pvp=0xc4fa033c, mode=33261, cred=0xc33a5900, vpp=0xd673d914) at /usr/src/sys/ufs/ffs/ffs_alloc.c:987 #4 0xc07769c5 in ufs_makeinode (mode=33261, dvp=0xc4fa033c, vpp=0xd673db90, cnp=0xd673dba4) at /usr/src/sys/ufs/ufs/ufs_vnops.c:2252 #5 0xc07770e0 in ufs_create (ap=0xd673da88) at /usr/src/sys/ufs/ufs/ufs_vnops.c:193 #6 0xc07f0ef2 in VOP_CREATE_APV (vop=0xc0872420, a=0xd673da88) at vnode_if.c:206 #7 0xc0606210 in vn_open_cred (ndp=0xd673db7c, flagp=0xd673dc78, cmode=) at vnode_if.h:112 #8 0xc06065e3 in vn_open (ndp=0xd673db7c, flagp=0xd673dc78, cmode=493, fp=0xc351fe40) at /usr/src/sys/kern/vfs_vnops.c:94 #9 0xc0603d13 in kern_open (td=0xc5252af0, path=0x33e0f0c0
, pathseg=UIO_USERSPACE, flags=2562, mode=493) at /usr/src/sys/kern/vfs_syscalls.c:1042 #10 0xc0604280 in open (td=0xc5252af0, uap=0xd673dcfc) at /usr/src/sys/kern/vfs_syscalls.c:1009 #11 0xc07daff5 in syscall (frame=0xd673dd38) at /usr/src/sys/i386/i386/trap.c:1090 #12 0xc07bfdc0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255 #13 0x00000033 in ?? () (kgdb) q If there is any additional information I can supply to aid in the resolution of this issue, don't hesitate to ask. From stb at lassitu.de Thu Jul 16 12:26:04 2009 From: stb at lassitu.de (Stefan Bethke) Date: Thu Jul 16 12:26:11 2009 Subject: Booting from ZFS raidz In-Reply-To: References: <9461581F-F354-486D-961D-3FD5B1EF007C@rabson.org> <20090201072432.GA25276@server.vk2pj.dyndns.org> <246ecf0c87f944d70c5562eeed4165c9@mail.rabson.org> <9cc826f0720e1624489dd6e6d384babc.squirrel@www.noacks.org> Message-ID: <67089745-3E3B-45CF-B4E8-91F3B52C8A4C@lassitu.de> Am 12.07.2009 um 09:06 schrieb Jonathan Noack: > Anyone know how I should proceed? I'm really frustrated that I > haven't > been able to get this working. Me too. I just decided to revisit this, and unfortunatly, I'm still stuck with the loader having trouble reading blocks off the raidz1 volume. I've updated the creation script (see below). This is on r195709, with LOADER_ZFS_SUPPORT?=YES in /etc/make.conf. Boot output, transcribed by hand: error 4 lba 4290782259 error 4 lba 4290782258 error 4 lba 4290782259 error 4 lba 4290782258 error 4 lba 4290782257 error 4 lba 4290782257 ... error 4 lba 4292652987 error 4 lba 4292652986 error 4 lba 4292652987 BTX loader 1.00 BTX version is 1.02 Consoles: internal video/keyboard BIOS drive A: is disk0 BIOS drive C: is disk1 BIOS drive D: is disk2 BIOS drive E: is disk3 BIOS 638kB/2095040kB available memory FreeBSD/i386 bootstrap loader, Revision 1.1 (root@freebsd-current-lassitu.de, Wed Jul 15 19:31.22 CEST 2009) Loading /boot/defaults/loader.conf ZFS: i/o error - all block copies unavailable ZFS: i/o error - all block copies unavailable ... (about 10 more of these) Unable to load a kernel! can't load 'kernel' Then the prompt. My setup script: # cat prepare-zfs-boot #!/bin/sh idx=0 label() { dd if=/dev/zero of=/dev/$1 bs=1m count=10 sz="$(diskinfo /dev/$1 | awk '{print $4}')" echo $sz gpart destroy $1 gpart create -s gpt $1 gpart add -b 34 -s 128 -t freebsd-boot -l "boot${idx}" $1 gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 $1 gpart add -b 512 -s $(($sz-1024)) -t freebsd-zfs -l "zfs${idx}" $1 gpart list $1 idx="$((idx+1))" } set -x zpool destroy zroot label da1 label da2 label da3 zpool create zroot raidz /dev/gpt/zfs[0-2] zpool set bootfs=zroot zroot zfs create -V 1gb zroot/swap zfs set org.freebsd:swap=on zroot/swap zfs set checksum=off zpool/swap cd /usr/src && make installworld installkernel distribution DESTDIR=/ zroot cp /boot/loader.conf /zroot/boot/loader.conf echo 'zfs_load="YES"' >>/zroot/boot/loader.conf echo 'vfs.root.mountfrom="zfs:zroot"' >>/zroot/boot/loader.conf zpool export zroot zpool import zroot cp /boot/zfs/zpool.cache /zroot/boot/zfs/ zfs set mountpoint=legacy zroot # eof -- Stefan Bethke Fon +49 151 14070811 From stb at lassitu.de Thu Jul 16 12:28:39 2009 From: stb at lassitu.de (Stefan Bethke) Date: Thu Jul 16 12:28:53 2009 Subject: Booting from ZFS raidz In-Reply-To: <67089745-3E3B-45CF-B4E8-91F3B52C8A4C@lassitu.de> References: <9461581F-F354-486D-961D-3FD5B1EF007C@rabson.org> <20090201072432.GA25276@server.vk2pj.dyndns.org> <246ecf0c87f944d70c5562eeed4165c9@mail.rabson.org> <9cc826f0720e1624489dd6e6d384babc.squirrel@www.noacks.org> <67089745-3E3B-45CF-B4E8-91F3B52C8A4C@lassitu.de> Message-ID: <3F75A414-1D5F-4692-B3B1-41EBCC1A22D2@lassitu.de> Am 16.07.2009 um 14:06 schrieb Stefan Bethke: > Am 12.07.2009 um 09:06 schrieb Jonathan Noack: > >> Anyone know how I should proceed? I'm really frustrated that I >> haven't >> been able to get this working. > > Me too. I just decided to revisit this, and unfortunatly, I'm still > stuck with the loader having trouble reading blocks off the raidz1 > volume. I've updated the creation script (see below). This is on > r195709, with LOADER_ZFS_SUPPORT?=YES in /etc/make.conf. I just checked that my basic procedure is sane by creating a single vdev pool (zpool create /dev/gpt/zfs0), and that is booting and working just fine. Stefan -- Stefan Bethke Fon +49 151 14070811 From arnaud.houdelette at tzim.net Fri Jul 17 15:24:50 2009 From: arnaud.houdelette at tzim.net (Arnaud Houdelette) Date: Fri Jul 17 15:24:56 2009 Subject: Booting from ZFS raidz In-Reply-To: <3F75A414-1D5F-4692-B3B1-41EBCC1A22D2@lassitu.de> References: <9461581F-F354-486D-961D-3FD5B1EF007C@rabson.org> <20090201072432.GA25276@server.vk2pj.dyndns.org> <246ecf0c87f944d70c5562eeed4165c9@mail.rabson.org> <9cc826f0720e1624489dd6e6d384babc.squirrel@www.noacks.org> <67089745-3E3B-45CF-B4E8-91F3B52C8A4C@lassitu.de> <3F75A414-1D5F-4692-B3B1-41EBCC1A22D2@lassitu.de> Message-ID: <4A6097BA.8030305@tzim.net> Stefan Bethke a ?crit : > Am 16.07.2009 um 14:06 schrieb Stefan Bethke: > >> Am 12.07.2009 um 09:06 schrieb Jonathan Noack: >> >>> Anyone know how I should proceed? I'm really frustrated that I haven't >>> been able to get this working. >> >> Me too. I just decided to revisit this, and unfortunatly, I'm still >> stuck with the loader having trouble reading blocks off the raidz1 >> volume. I've updated the creation script (see below). This is on >> r195709, with LOADER_ZFS_SUPPORT?=YES in /etc/make.conf. > > I just checked that my basic procedure is sane by creating a single > vdev pool (zpool create /dev/gpt/zfs0), and that is booting and > working just fine. > > > Stefan > I got the same kind of errors with 8.0-BETA1 (rebuilt loader with ZFS_SUPPORT) with raidz on a VirtualBox. I however got some hints : Tried with single disk on pool : works like a charm. Tried with a mirror : the loader only 'sees' the first disk. Could that be the source of our problems ? BTX loader only shows drives A: and C: lsdev -v gives : cd devices: disk devices: disk0: BIOS drive A: disk1: BIOS drive C: disk1p1: FreeBSD boot 64KB disk1p2: FreeBSD ZFS 19GB pxe devices: zfs devices: pool: zroot config: NAME STATE zroot DEGRADED mirror DEGRADED gpt/zfs1 ONLINE gpt/zfs2 OFFLINE As the mirror is degraded but readable, the loader is able to boot. Could it be a BIOS or boot0 problem ? Arnaud From seyes463 at gmail.com Fri Jul 17 15:50:31 2009 From: seyes463 at gmail.com (Snakey Eyes) Date: Fri Jul 17 15:51:02 2009 Subject: kern/118107: [ntfs] [panic] Kernel panic when accessing a file at NTFS file system [regression] Message-ID: <1d8e056d0907170819k368dbb4s34445441fd56d74d@mail.gmail.com> I'm running into this bug and would like to help out with testing/resolving. Is this the appropriate place to start? From stb at lassitu.de Fri Jul 17 16:22:16 2009 From: stb at lassitu.de (Stefan Bethke) Date: Fri Jul 17 16:22:23 2009 Subject: Booting from ZFS raidz In-Reply-To: <4A6097BA.8030305@tzim.net> References: <9461581F-F354-486D-961D-3FD5B1EF007C@rabson.org> <20090201072432.GA25276@server.vk2pj.dyndns.org> <246ecf0c87f944d70c5562eeed4165c9@mail.rabson.org> <9cc826f0720e1624489dd6e6d384babc.squirrel@www.noacks.org> <67089745-3E3B-45CF-B4E8-91F3B52C8A4C@lassitu.de> <3F75A414-1D5F-4692-B3B1-41EBCC1A22D2@lassitu.de> <4A6097BA.8030305@tzim.net> Message-ID: Am 17.07.2009 um 17:24 schrieb Arnaud Houdelette: > Stefan Bethke a ?crit : >> Am 16.07.2009 um 14:06 schrieb Stefan Bethke: >> >>> Am 12.07.2009 um 09:06 schrieb Jonathan Noack: >>> >>>> Anyone know how I should proceed? I'm really frustrated that I >>>> haven't >>>> been able to get this working. >>> >>> Me too. I just decided to revisit this, and unfortunatly, I'm >>> still stuck with the loader having trouble reading blocks off the >>> raidz1 volume. I've updated the creation script (see below). >>> This is on r195709, with LOADER_ZFS_SUPPORT?=YES in /etc/make.conf. >> >> I just checked that my basic procedure is sane by creating a single >> vdev pool (zpool create /dev/gpt/zfs0), and that is booting and >> working just fine. >> >> >> Stefan >> > I got the same kind of errors with 8.0-BETA1 (rebuilt loader with > ZFS_SUPPORT) with raidz on a VirtualBox. > I however got some hints : > Tried with single disk on pool : works like a charm. > Tried with a mirror : the loader only 'sees' the first disk. Could > that be the source of our problems ? > > BTX loader only shows drives A: and C: > > lsdev -v gives : > cd devices: > disk devices: > disk0: BIOS drive A: > disk1: BIOS drive C: > disk1p1: FreeBSD boot 64KB > disk1p2: FreeBSD ZFS 19GB > pxe devices: > zfs devices: > pool: zroot > config: > > NAME STATE > zroot DEGRADED > mirror DEGRADED > gpt/zfs1 ONLINE > gpt/zfs2 OFFLINE > > As the mirror is degraded but readable, the loader is able to boot. > Could it be a BIOS or boot0 problem ? In VMware Fusion, it appears the loader sees all disks (with my UFS disk attached, and with ZFS on raw devices: OK lsdev -v cd devices: disk devices: disk0: BIOS drive A: disk1: BIOS drive C: disk1s1a: FFS 30GB (2097215 - 67103505) disk1s1b: swap 1024MB (63 - 2097215) disk2: BIOS drive D: disk2: FFS bad disklabel disk3: BIOS drive E: disk3: FFS bad disklabel disk4: BIOS drive F: disk4: FFS bad disklabel pxe devices: zfs devices: pool: zroot config: NAME STATE zroot ONLINE raidz1 ONLINE da1 ONLINE da2 ONLINE da3 ONLINE OK When detaching the first disk, so that only the three ZFS disks are available, I get the same output (but loader error on trying to read the kernel, as posted previously). Stefan -- Stefan Bethke Fon +49 151 14070811 From jgfredo at gmail.com Fri Jul 17 16:31:05 2009 From: jgfredo at gmail.com (Fred J.G.) Date: Fri Jul 17 16:31:12 2009 Subject: FreeBSD, SSD's and partition alignment Message-ID: <0584EF0B-B93A-4765-9510-0EA06B9D4BAE@gmail.com> Dear All, In reference to this issue about SSD's and partition alignment at http://kerneltrap.org/mailarchive/freebsd-fs/2009/1/5/4601614 I have been struggling a few hours with installing FreeBSD on a SSD attached to a VIA-EPIA Mini ITX board with the VIA PLE133 Chipset on it. I have choosen FreeBSD specifically because it appeared already running as os with samba on this commercial NAS-device based on the VIA EPIA board with many interesting neglected features. The 5400rpm internal drive is retired from today. After getting a lot of inode errors when trying to install FreeBSD 7.2 on a OCZ PATA 120GB SSD attached to IDE1 interface of the board I ended up to follow up the Geometry reported by the BIOS C/H/S = 61865/16/255 . Surprisingly or maybe not surprisingly to readers here no inode errors anymore during a full installation! I do have some concerns though regarding wearing and performance ( degradation)such as: - The default newfs args -b 16384 -f 2048 will this be in harmony? - Currently 1GB memory onboard and default 2GB swap is defined on the SSD. - etc. out of my current perception... Further it is a great learning project since this board is more than 6 years old but reasonable in power consumption. the ultimate goal is to create a thinfat client annex voip videophone multimedia box which will be able to connect to a virtualized featherweight virtual machine somewhere. if kerneloptimalisation is required for this than it be so. looking forward to any constructive input. cheers, Fred From acc at hexadecagram.org Sun Jul 19 09:21:25 2009 From: acc at hexadecagram.org (Anthony Chavez) Date: Sun Jul 19 09:21:33 2009 Subject: Re-starting a gjournal provider Message-ID: <4A62E0CE.1000508@hexadecagram.org> Hello freebsd-fs, I'm trying to get gjournal working on a "removable" hard disk. I use the term loosely, because I'm using a very simple eSATA enclosure: an AMS Venus DS5 [1]. If I swap out disks, atacontrol cap ad0 seems sufficient enough to detect the new drive: the reported device model, serial number, firmware revision, and CHS values change as one would expect. My interpretation of [2] section 5.3 and gjournal(8) is that the following sequence of commands should ensure me that all write buffers have been flushed and bring the system to a point where it is safe to remove a disk. sync; sync; sync gjournal sync umount /dev/ad0s1.journal gjournal stop ad0s1.journal However, once they are executed, /dev/ad0s1.journal disappears and when I swap out the disk it doesn't come back. The only way I've found to bring it back is atacontrol detach ata0; atacontrol attach ata0, which doesn't seem like a wise thing to do if I have another device on the same channel. My question is, do I need to issue gjournal stop before I swap disks? And if so, is there any way that I can avoid the atacontrol detach/attach cycle that would need to take place before any mount is attempted so that /dev/ad0s1.journal appears (if in the drive inserted at the time does in fact utilize gjournal; I may want to experiment with having disks with either gjournal or soft updates)? And while I'm on the subject, are the (gjournal) syncs commands preceeding umount absolutely necessary in the case of removable media? [1] http://www.american-media.com/product/external/ds/ds.html [2] http://www.freebsd.org/doc/en/articles/gjournal-desktop/article.html -- Anthony Chavez http://hexadecagram.org/ mailto:acc@hexadecagram.org xmpp:acc@hexadecagram.org -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20090719/3b18b36e/signature.pgp From acc at hexadecagram.org Sun Jul 19 10:03:01 2009 From: acc at hexadecagram.org (Anthony Chavez) Date: Sun Jul 19 10:03:08 2009 Subject: Re-starting a gjournal provider In-Reply-To: <4A62E0CE.1000508@hexadecagram.org> References: <4A62E0CE.1000508@hexadecagram.org> Message-ID: <4A62EF41.50702@hexadecagram.org> I'm moving this discussion to freebsd-questions, which is probably a more suitable forum (if not freebsd-geom). I apologize for the intrusion and the subsequent crosspost. Anthony Chavez wrote: > Hello freebsd-fs, > > I'm trying to get gjournal working on a "removable" hard disk. I use > the term loosely, because I'm using a very simple eSATA enclosure: an > AMS Venus DS5 [1]. > > If I swap out disks, atacontrol cap ad0 seems sufficient enough to > detect the new drive: the reported device model, serial number, firmware > revision, and CHS values change as one would expect. > > My interpretation of [2] section 5.3 and gjournal(8) is that the > following sequence of commands should ensure me that all write buffers > have been flushed and bring the system to a point where it is safe to > remove a disk. > > sync; sync; sync > gjournal sync > umount /dev/ad0s1.journal > gjournal stop ad0s1.journal > > However, once they are executed, /dev/ad0s1.journal disappears and when > I swap out the disk it doesn't come back. The only way I've found to > bring it back is atacontrol detach ata0; atacontrol attach ata0, which > doesn't seem like a wise thing to do if I have another device on the > same channel. > > My question is, do I need to issue gjournal stop before I swap disks? > And if so, is there any way that I can avoid the atacontrol > detach/attach cycle that would need to take place before any mount is > attempted so that /dev/ad0s1.journal appears (if in the drive inserted > at the time does in fact utilize gjournal; I may want to experiment with > having disks with either gjournal or soft updates)? > > And while I'm on the subject, are the (gjournal) syncs commands > preceeding umount absolutely necessary in the case of removable media? > > [1] http://www.american-media.com/product/external/ds/ds.html > > [2] http://www.freebsd.org/doc/en/articles/gjournal-desktop/article.html -- Anthony Chavez http://hexadecagram.org/ mailto:acc@hexadecagram.org xmpp:acc@hexadecagram.org -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20090719/36041a09/signature.pgp From bugmaster at FreeBSD.org Mon Jul 20 11:06:54 2009 From: bugmaster at FreeBSD.org (FreeBSD bugmaster) Date: Mon Jul 20 11:08:05 2009 Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org Message-ID: <200907201106.n6KB6rTg002254@freefall.freebsd.org> Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/136601 fs [zfs] tar: Couldn't list extended attributes: Read-onl o kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/136218 fs [zfs] Exported ZFS pools can't be imported into (Open) o kern/135594 fs [zfs] Single dataset unresponsive with Samba o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135480 fs [zfs] panic: lock &arg.lock already initialized o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o bin/135314 fs [zfs] assertion failed for zdb(8) usage o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot f kern/134496 fs [zfs] [panic] ZFS pool export occasionally causes a ke o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133980 fs [panic] [ffs] panic: ffs_valloc: dup alloc o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis o kern/133614 fs [smbfs] [panic] panic: ffs_truncate: read-only filesys o kern/133373 fs [zfs] umass attachment causes ZFS checksum errors, dat o kern/133174 fs [msdosfs] [patch] msdosfs must support utf-encoded int f kern/133150 fs [zfs] Page fault with ZFS on 7.1-RELEASE/amd64 while w o kern/133134 fs [zfs] Missing ZFS zpool labels f kern/133020 fs [zfs] [panic] inappropriate panic caused by zfs. Pani o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132597 fs [tmpfs] [panic] tmpfs-related panic while interrupting o kern/132551 fs [zfs] ZFS locks up on extattr_list_link syscall o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132337 fs [zfs] [panic] kernel panic in zfs_fuid_create_cred o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes f kern/132068 fs [zfs] page fault when using ZFS over NFS on 7.1-RELEAS o kern/131995 fs [nfs] Failure to mount NFSv4 server o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/131086 fs [ext2fs] [patch] mkfs.ext2 creates rotten partition o kern/130979 fs [smbfs] [panic] boot/kernel/smbfs.ko o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130229 fs [iconv] usermount fails on fs that need iconv o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/129148 fs [zfs] [panic] panic on concurrent writing & rollback o kern/129059 fs [zfs] [patch] ZFS bootloader whitelistable via WITHOUT f kern/128829 fs smbd(8) causes periodic panic on 7-RELEASE o kern/128633 fs [zfs] [lor] lock order reversal in zfs o kern/128514 fs [zfs] [mpt] problems with ZFS and LSILogic SAS/SATA Ad f kern/128173 fs [ext2fs] ls gives "Input/output error" on mounted ext3 o kern/127659 fs [tmpfs] tmpfs memory leak o kern/127492 fs [zfs] System hang on ZFS input-output o kern/127420 fs [gjournal] [panic] Journal overflow on gmirrored gjour o kern/127213 fs [tmpfs] sendfile on tmpfs data corruption o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS o kern/125644 fs [zfs] [panic] zfs unfixable fs errors caused panic whe f kern/125536 fs [ext2fs] ext 2 mounts cleanly but fails on commands li o kern/125149 fs [nfs] [panic] changing into .zfs dir from nfs client c f kern/124621 fs [ext3] [patch] Cannot mount ext2fs partition f bin/124424 fs [zfs] zfs(8): zfs list -r shows strange snapshots' siz o kern/123939 fs [msdosfs] corrupts new files o kern/122888 fs [zfs] zfs hang w/ prefetch on, zil off while running t o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o kern/122173 fs [zfs] [panic] Kernel Panic if attempting to replace a o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o kern/122047 fs [ext2fs] [patch] incorrect handling of UF_IMMUTABLE / o kern/122038 fs [tmpfs] [panic] tmpfs: panic: tmpfs_alloc_vp: type 0xc o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121779 fs [ufs] snapinfo(8) (and related tools?) only work for t o kern/121770 fs [zfs] ZFS on i386, large file or heavy I/O leads to ke o bin/121366 fs [zfs] [patch] Automatic disk scrubbing from periodic(8 o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha f kern/120991 fs [panic] [fs] [snapshot] System crashes when manipulati o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F o bin/120288 fs zfs(8): "zfs share -a" does not send SIGHUP to mountd f kern/119735 fs [zfs] geli + ZFS + samba starting on boot panics 7.0-B o kern/118912 fs [2tb] disk sizing/geometry problem with large array o misc/118855 fs [zfs] ZFS-related commands are nonfunctional in fixit o kern/118713 fs [minidump] [patch] Display media size required for a k o kern/118320 fs [zfs] [patch] NFS SETATTR sometimes fails to set file o bin/118249 fs mv(1): moving a directory changes its mtime o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117314 fs [ntfs] Long-filename only NTFS fs'es cause kernel pani o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o kern/116913 fs [ffs] [panic] ffs_blkfree: freeing free block p kern/116608 fs [msdosfs] [patch] msdosfs fails to check mount options o kern/116583 fs [ffs] [hang] System freezes for short time when using o kern/116170 fs [panic] Kernel panic when mounting /tmp o kern/115645 fs [snapshots] [panic] lockmgr: thread 0xc4c00d80, not ex o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o kern/113180 fs [zfs] Setting ZFS nfsshare property does not cause inh o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o kern/109024 fs [msdosfs] mount_msdosfs: msdosfs_iconv: Operation not o kern/109010 fs [msdosfs] can't mv directory within fat32 file system o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106030 fs [ufs] [panic] panic in ufs from geom when a dead disk o kern/105093 fs [ext2fs] [patch] ext2fs on read-only media cannot be m o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [iso9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna f kern/91568 fs [ufs] [panic] writing to UFS/softupdates DVD media in o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/89991 fs [ufs] softupdates with mount -ur causes fs UNREFS o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88266 fs [smbfs] smbfs does not implement UIO_NOCOPY and sendfi o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o kern/85326 fs [smbfs] [panic] saving a file via samba to an overquot o kern/84589 fs [2TB] 5.4-STABLE unresponsive during background fsck 2 o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o kern/77826 fs [ext2fs] ext2fs usb filesystem will not mount RW o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/51583 fs [nullfs] [patch] allow to work with devices and socket o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 145 problems total. From spawk at acm.poly.edu Mon Jul 20 13:41:16 2009 From: spawk at acm.poly.edu (Boris Kochergin) Date: Mon Jul 20 13:41:22 2009 Subject: geom_mirror/UFS weirdness with 7.2-STABLE Message-ID: <4A646DA8.2050201@acm.poly.edu> Ahoy. I noticed some very odd things in my file server's kernel buffer this morning (there were actually a ton of these--this is a snippet): Jul 20 05:54:10 exodus smartd[763]: Device: /dev/ad1, FAILED SMART self-check. BACK UP DATA NOW! Jul 20 05:57:57 exodus kernel: g_vfs_done():mirror/boots1[READ(offset=-4569735194538825728, length=16384)]error = 5 Jul 20 05:57:57 exodus kernel: bad block 8806809555123731765, ino 4430620 Jul 20 05:57:57 exodus kernel: pid 35 (softdepflush), uid 0 inumber 4430620 on /: bad block Jul 20 05:57:57 exodus kernel: g_vfs_done():mirror/boots1[READ(offset=-7379534036451358720, length=16384)]error = 5 Jul 20 05:57:57 exodus kernel: bad block 5076457091570183473, ino 4430620 Jul 20 05:57:57 exodus kernel: pid 35 (softdepflush), uid 0 inumber 4430620 on /: bad block Jul 20 05:57:57 exodus kernel: g_vfs_done():mirror/boots1[READ(offset=8010486245121781760, length=16384)]error = 5 Jul 20 05:57:57 exodus kernel: bad block 1859394416713520222, ino 4430620 Jul 20 05:57:57 exodus kernel: pid 35 (softdepflush), uid 0 inumber 4430620 on /: bad block Jul 20 05:57:57 exodus kernel: bad block 8993345361309361786, ino 4430620 Jul 20 05:57:57 exodus kernel: pid 35 (softdepflush), uid 0 inumber 4430620 on /: bad block Jul 20 05:57:57 exodus kernel: bad block 5442079378433802039, ino 4430620 Jul 20 05:57:57 exodus kernel: pid 35 (softdepflush), uid 0 inumber 4430620 on /: bad block Jul 20 05:57:57 exodus kernel: bad block -386473693423142096, ino 4430620 Jul 20 05:57:57 exodus kernel: pid 35 (softdepflush), uid 0 inumber 4430620 on /: bad block Jul 20 05:57:57 exodus kernel: bad block -6538821972600747, ino 4430620 Jul 20 05:57:57 exodus kernel: pid 35 (softdepflush), uid 0 inumber 4430620 on /: bad block Jul 20 05:57:57 exodus kernel: bad block 7280706165078778427, ino 4430620 Jul 20 05:57:57 exodus kernel: pid 35 (softdepflush), uid 0 inumber 4430620 on /: bad block Jul 20 05:57:57 exodus kernel: bad block 2788813269397304649, ino 4430620 Jul 20 05:57:57 exodus kernel: pid 35 (softdepflush), uid 0 inumber 4430620 on /: bad block Jul 20 05:57:57 exodus kernel: bad block 8324768912580846323, ino 4430620 Jul 20 05:57:57 exodus kernel: pid 35 (softdepflush), uid 0 inumber 4430620 on /: bad block Jul 20 05:57:57 exodus kernel: bad block 8358350883763232034, ino 4430620 Jul 20 05:57:57 exodus kernel: pid 35 (softdepflush), uid 0 inumber 4430620 on /: bad block The filesystem lives atop a two-disk geom_mirror: # gmirror status Name Status Components mirror/boot COMPLETE ad0 ad1 ...and the output of "df" on it also shows something odd: # df / Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/mirror/boots1 37846636 -4058799239201906816 4058799239236725722 -11656883301279% / The system is a: # uname -a FreeBSD exodus.poly.edu 7.2-STABLE FreeBSD 7.2-STABLE #3: Sat Jul 11 16:22:02 EDT 2009 root@exodus.poly.edu:/usr/obj/usr/src/sys/EXODUS amd64 Regarding smartd yelling at me about /dev/ad1, it's been doing that for long while before this. There is one sector on the drive that cannot be read, but the disk has otherwise been fine for months. My experience with geom_mirror has been that it disconnects members from an array if they experience I/O errors, so this seems to be something different. Any clues? -Boris From ivoras at freebsd.org Mon Jul 20 21:32:30 2009 From: ivoras at freebsd.org (Ivan Voras) Date: Mon Jul 20 21:32:37 2009 Subject: geom_mirror/UFS weirdness with 7.2-STABLE In-Reply-To: <4A646DA8.2050201@acm.poly.edu> References: <4A646DA8.2050201@acm.poly.edu> Message-ID: Boris Kochergin wrote: > Ahoy. I noticed some very odd things in my file server's kernel buffer > this morning (there were actually a ton of these--this is a snippet): > > Jul 20 05:54:10 exodus smartd[763]: Device: /dev/ad1, FAILED SMART > self-check. BACK UP DATA NOW! > Jul 20 05:57:57 exodus kernel: > g_vfs_done():mirror/boots1[READ(offset=-4569735194538825728, > length=16384)]error = 5 > Jul 20 05:57:57 exodus kernel: bad block 8806809555123731765, ino 4430620 > Jul 20 05:57:57 exodus kernel: pid 35 (softdepflush), uid 0 inumber > 4430620 on /: bad block > > # df / > Filesystem 1K-blocks Used Avail > Capacity Mounted on > /dev/mirror/boots1 37846636 -4058799239201906816 4058799239236725722 > -11656883301279% / > > The system is a: > > # uname -a > FreeBSD exodus.poly.edu 7.2-STABLE FreeBSD 7.2-STABLE #3: Sat Jul 11 > 16:22:02 EDT 2009 root@exodus.poly.edu:/usr/obj/usr/src/sys/EXODUS > amd64 > > Regarding smartd yelling at me about /dev/ad1, it's been doing that for > long while before this. There is one sector on the drive that cannot be > read, but the disk has otherwise been fine for months. My experience > with geom_mirror has been that it disconnects members from an array if > they experience I/O errors, so this seems to be something different. Any > clues? It looks like the drive returned corrupted data without returning an error - which is strange, but not impossible. You are probably seeing numbers like -4058799239201906816 because some metadata is corrupted. If so, you should immediately disconnect the problematic drive so that the errorneous data isn't picked up and written to the good drive. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 259 bytes Desc: OpenPGP digital signature Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20090720/2720dffc/signature.pgp From linimon at FreeBSD.org Tue Jul 21 04:00:36 2009 From: linimon at FreeBSD.org (linimon@FreeBSD.org) Date: Tue Jul 21 04:00:43 2009 Subject: kern/136873: [ntfs] Missing directories/files on NTFS volume Message-ID: <200907210400.n6L40ZOE093783@freefall.freebsd.org> Old Synopsis: Missing directories/files on NTFS volume New Synopsis: [ntfs] Missing directories/files on NTFS volume Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Tue Jul 21 04:00:21 UTC 2009 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=136873 From trasz at FreeBSD.org Wed Jul 22 07:43:12 2009 From: trasz at FreeBSD.org (trasz@FreeBSD.org) Date: Wed Jul 22 07:43:19 2009 Subject: kern/136601: [zfs] tar: Couldn't list extended attributes: Read-only file system Message-ID: <200907220743.n6M7hBlw049326@freefall.freebsd.org> Synopsis: [zfs] tar: Couldn't list extended attributes: Read-only file system Responsible-Changed-From-To: freebsd-fs->trasz Responsible-Changed-By: trasz Responsible-Changed-When: Wed Jul 22 07:43:11 UTC 2009 Responsible-Changed-Why: I'll take it. http://www.freebsd.org/cgi/query-pr.cgi?pr=136601 From rick-freebsd2008 at kiwi-computer.com Wed Jul 22 22:10:48 2009 From: rick-freebsd2008 at kiwi-computer.com (Rick C. Petty) Date: Wed Jul 22 22:10:54 2009 Subject: UFS label limitations In-Reply-To: <20081216201046.GA34809@keira.kiwi-computer.com> References: <20081213173902.GA96883@keira.kiwi-computer.com> <20081213183058.GA20992@a91-153-125-115.elisa-laajakaista.fi> <20081213192320.GA97766@keira.kiwi-computer.com> <86y6yh5pz0.fsf@ds4.des.no> <20081215234809.GA24403@keira.kiwi-computer.com> <8663lk5ju7.fsf@ds4.des.no> <20081216201046.GA34809@keira.kiwi-computer.com> Message-ID: <20090722221046.GA62007@keira.kiwi-computer.com> I posted this patch back in December, which allows some separator characters to be used in UFS labels. I've attached the patch below. des@ suggested using strspn(3) instead, so I've attached that version also. Is someone willing to commit this into 8.0? It's rather innocuous. -- Rick C. Petty ~~~ ~~~ original patch ~~~ --- src/sbin/newfs/newfs.c.orig 2007-03-02 14:07:59.000000000 -0600 +++ src/sbin/newfs/newfs.c 2008-12-15 17:29:26.000000000 -0600 @@ -168,11 +168,15 @@ case 'L': volumelabel = optarg; i = -1; - while (isalnum(volumelabel[++i])); - if (volumelabel[i] != '\0') { - errx(1, "bad volume label. Valid characters are alphanumerics."); - } - if (strlen(volumelabel) >= MAXVOLLEN) { + while ((ch = volumelabel[++i]) != '\0') + if (ch != '-' && ch != '.' && ch != '_' && + (ch < '0' || ch > '9') && + (ch < 'A' || ch > 'Z') && + (ch < 'a' || ch > 'z')) + errx(1, + "bad volume label. Valid characters are " + "[0-9A-Za-z._-]."); + if (i >= MAXVOLLEN) { errx(1, "bad volume label. Length is longer than %d.", MAXVOLLEN); } --- src/sbin/tunefs/tunefs.c.orig 2008-02-26 14:25:35.000000000 -0600 +++ src/sbin/tunefs/tunefs.c 2008-12-15 17:27:58.000000000 -0600 @@ -153,13 +153,16 @@ name = "volume label"; Lvalue = optarg; i = -1; - while (isalnum(Lvalue[++i])); - if (Lvalue[i] != '\0') { + while ((ch = Lvalue[++i]) != '\0') + if (ch != '-' && ch != '.' && ch != '_' && + (ch < '0' || ch > '9') && + (ch < 'A' || ch > 'Z') && + (ch < 'a' || ch > 'z')) errx(10, - "bad %s. Valid characters are alphanumerics.", + "bad %s. Valid characters are " + "[0-9A-Za-z._-].", name); - } - if (strlen(Lvalue) >= MAXVOLLEN) { + if (i >= MAXVOLLEN) { errx(10, "bad %s. Length is longer than %d.", name, MAXVOLLEN - 1); } ~~~ ~~~ using strspn instead ~~~ --- src/sbin/newfs/newfs.c.orig 2008-05-03 23:51:38.000000000 -0500 +++ src/sbin/newfs/newfs.c 2009-07-22 16:58:48.000000000 -0500 @@ -167,10 +167,10 @@ break; case 'L': volumelabel = optarg; - i = -1; - while (isalnum(volumelabel[++i])); + i = strspn(volumelabel, + "-.0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz"); if (volumelabel[i] != '\0') { - errx(1, "bad volume label. Valid characters are alphanumerics."); + errx(1, "bad volume label. Valid characters are [0-9A-Za-z._-]."); } if (strlen(volumelabel) >= MAXVOLLEN) { errx(1, "bad volume label. Length is longer than %d.", --- src/sbin/tunefs/tunefs.c.orig 2008-05-03 23:51:52.000000000 -0500 +++ src/sbin/tunefs/tunefs.c 2009-07-22 17:01:23.000000000 -0500 @@ -152,11 +152,11 @@ found_arg = 1; name = "volume label"; Lvalue = optarg; - i = -1; - while (isalnum(Lvalue[++i])); + i = strspn(Lvalue, + "-.0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz"); if (Lvalue[i] != '\0') { errx(10, - "bad %s. Valid characters are alphanumerics.", + "bad %s. Valid characters are [0-9A-Za-z._-].", name); } if (strlen(Lvalue) >= MAXVOLLEN) { From serenity at exscape.org Thu Jul 23 08:22:41 2009 From: serenity at exscape.org (Thomas Backman) Date: Thu Jul 23 08:22:54 2009 Subject: kern/132337: [zfs] [panic] kernel panic in zfs_fuid_create_cred Message-ID: Hey all, Unfortunately, this PR appears to still be unfixed. Is anyone looking at this? I tried to set up a diskless client over NFS yesterday, running ZFS on /, and ran in to this panic (exact same backtrace as the PR (see bottom the this mail)) a lot of times. In the end, I created a ZVOL with UFS on it, and the panics disappeared as expected - since it no longer has to work directly with files on a ZFS *filesystem*. It seems to be pretty easy to trigger, as I couldn't finish the boot process on the diskless client more than once until I went with UFS. Since then I've gotten 0 panics. FWIW, my /etc/exports (I know now R/W NFS root isn't a good idea, but it might be relevant): /diskless -maproot=0 -alldirs 192.168.1.6 Regards, Thomas Backtrace: panic: zfs_fuid_create_cred cpuid = x KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a panic() at panic+0x182 zfs_fuid_create_cred() at zfs_fuid_create_cred+0x56 zfs_perm_init() at zfs_perm_init+0x84 zfs_mknode() at zfs_mknode+0x24e zfs_freebsd_create() at zfs_freebsd_create+0x617 VOP_CREATE_APV() at VOP_CREATE_APV+0xb3 nfsrv_create() at nfsrv_create+0x909 nfssvc_program() at nfssvc_program+0x1a1 svc_run_internal() at svc_run_internal+0x62b svc_thread_start() at svc_thread_start+0xb fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe From serenity at exscape.org Thu Jul 23 08:30:05 2009 From: serenity at exscape.org (Thomas Backman) Date: Thu Jul 23 08:30:15 2009 Subject: kern/132337: [zfs] [panic] kernel panic in zfs_fuid_create_cred Message-ID: <200907230830.n6N8U54k000218@freefall.freebsd.org> The following reply was made to PR kern/132337; it has been noted by GNATS. From: Thomas Backman To: bug-followup@FreeBSD.org, freebsd@r.zeeb.org Cc: freebsd-fs@freebsd.org, FreeBSD current Subject: Re: kern/132337: [zfs] [panic] kernel panic in zfs_fuid_create_cred Date: Thu, 23 Jul 2009 10:22:13 +0200 Hey all, Unfortunately, this PR appears to still be unfixed. Is anyone looking at this? I tried to set up a diskless client over NFS yesterday, running ZFS on /, and ran in to this panic (exact same backtrace as the PR (see bottom the this mail)) a lot of times. In the end, I created a ZVOL with UFS on it, and the panics disappeared as expected - since it no longer has to work directly with files on a ZFS *filesystem*. It seems to be pretty easy to trigger, as I couldn't finish the boot process on the diskless client more than once until I went with UFS. Since then I've gotten 0 panics. FWIW, my /etc/exports (I know now R/W NFS root isn't a good idea, but it might be relevant): /diskless -maproot=0 -alldirs 192.168.1.6 Regards, Thomas Backtrace: panic: zfs_fuid_create_cred cpuid = x KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a panic() at panic+0x182 zfs_fuid_create_cred() at zfs_fuid_create_cred+0x56 zfs_perm_init() at zfs_perm_init+0x84 zfs_mknode() at zfs_mknode+0x24e zfs_freebsd_create() at zfs_freebsd_create+0x617 VOP_CREATE_APV() at VOP_CREATE_APV+0xb3 nfsrv_create() at nfsrv_create+0x909 nfssvc_program() at nfssvc_program+0x1a1 svc_run_internal() at svc_run_internal+0x62b svc_thread_start() at svc_thread_start+0xb fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe From jh at saunalahti.fi Thu Jul 23 09:30:04 2009 From: jh at saunalahti.fi (Jaakko Heinonen) Date: Thu Jul 23 09:30:10 2009 Subject: kern/132337: [zfs] [panic] kernel panic in zfs_fuid_create_cred Message-ID: <200907230930.n6N9U4YZ046915@freefall.freebsd.org> The following reply was made to PR kern/132337; it has been noted by GNATS. From: Jaakko Heinonen To: Thomas Backman Cc: bug-followup@FreeBSD.org, freebsd@r.zeeb.org, FreeBSD current Subject: Re: kern/132337: [zfs] [panic] kernel panic in zfs_fuid_create_cred Date: Thu, 23 Jul 2009 12:24:36 +0300 Hi, On 2009-07-23, Thomas Backman wrote: > Unfortunately, this PR appears to still be unfixed. Is anyone looking > at this? > panic: zfs_fuid_create_cred This PR is a duplicate of kern/133020. There is a workaround fix in pjd's perforce branch but apparently it was never committed to svn. See: http://p4db.freebsd.org/changeView.cgi?CH=159874 http://people.freebsd.org/~pjd/patches/zfs_znode.h.patch -- Jaakko From serenity at exscape.org Thu Jul 23 13:20:04 2009 From: serenity at exscape.org (Thomas Backman) Date: Thu Jul 23 13:20:09 2009 Subject: kern/132337: [zfs] [panic] kernel panic in zfs_fuid_create_cred Message-ID: <200907231320.n6NDK2M3033925@freefall.freebsd.org> The following reply was made to PR kern/132337; it has been noted by GNATS. From: Thomas Backman To: Michael Reifenberger Cc: Jaakko Heinonen , FreeBSD current , bug-followup@FreeBSD.org, freebsd@r.zeeb.org Subject: Re: kern/132337: [zfs] [panic] kernel panic in zfs_fuid_create_cred Date: Thu, 23 Jul 2009 15:13:51 +0200 On Jul 23, 2009, at 15:06, Michael Reifenberger wrote: > On Thu, 23 Jul 2009, Jaakko Heinonen wrote: > ... >> This PR is a duplicate of kern/133020. There is a workaround fix in >> pjd's perforce branch but apparently it was never committed to svn. >> >> See: >> >> http://p4db.freebsd.org/changeView.cgi?CH=159874 >> http://people.freebsd.org/~pjd/patches/zfs_znode.h.patch >> > I'm using this patch since march without problems. I've only used it for hours but it seems to work here too, no longer using ZVOLs. Would be nice to have in -RELEASE :) Jaakko: It's the other way around. 132337 < 133020. This bug was filed Mar 05 2009, while 133020 was filed Mar 24 2009. Regards, Thomas From mike at reifenberger.com Thu Jul 23 13:30:05 2009 From: mike at reifenberger.com (Michael Reifenberger) Date: Thu Jul 23 13:30:13 2009 Subject: kern/132337: [zfs] [panic] kernel panic in zfs_fuid_create_cred Message-ID: <200907231330.n6NDU53H041116@freefall.freebsd.org> The following reply was made to PR kern/132337; it has been noted by GNATS. From: Michael Reifenberger To: Jaakko Heinonen Cc: Thomas Backman , FreeBSD current , bug-followup@FreeBSD.org, freebsd@r.zeeb.org Subject: Re: kern/132337: [zfs] [panic] kernel panic in zfs_fuid_create_cred Date: Thu, 23 Jul 2009 15:06:52 +0200 (CEST) On Thu, 23 Jul 2009, Jaakko Heinonen wrote: ... > This PR is a duplicate of kern/133020. There is a workaround fix in > pjd's perforce branch but apparently it was never committed to svn. > > See: > > http://p4db.freebsd.org/changeView.cgi?CH=159874 > http://people.freebsd.org/~pjd/patches/zfs_znode.h.patch > I'm using this patch since march without problems. Bye/2 --- Michael Reifenberger Michael@Reifenberger.com http://www.Reifenberger.com From samankaya at netscape.net Sat Jul 25 01:15:32 2009 From: samankaya at netscape.net (samankaya@netscape.net) Date: Sat Jul 25 01:15:42 2009 Subject: Partitioning disks on PPC for FreeBSD, installer crashes Message-ID: <8CBDADB1950718E-16E8-7AF@webmail-dx04.sysops.aol.com> Hi, this is a strange issue but I'm trying to install FreeBSD onto my PowerMac G4 Quicksilver 733MHz machine. I got as far as the partitioner in the standard install but couldn't figure out how to add slices as there is no option to create a disk label for it. Each time I try to use (C)reate a slice I get a message on the screen saying that I have to do this on a Master Partition.... How do I create this?? I tried dropping into a shell do use fdisk on it but it wouldn't allow me to do so using the holographic shell via the installer menu. Can anyone help give me an idea of what to do?? I used a Debian Linux boot cd which works fine and is able to detect the disk and partition it. I also created the needed 8MB yaboot partition in order to perhaps detect the disk but no such luck. The CD is iso1 of FreeBSD 7.1 I hope I don't have to go back to Debian as I really want to learn BSD!! Regards, Kaya From linimon at FreeBSD.org Sat Jul 25 01:18:47 2009 From: linimon at FreeBSD.org (linimon@FreeBSD.org) Date: Sat Jul 25 01:18:58 2009 Subject: kern/137034: [ufs] [lor] lock-order reversal near vfs_bio.c, ufs_dirhash.c Message-ID: <200907250118.n6P1Ik43026998@freefall.freebsd.org> Old Synopsis: lock-order reversal near vfs_bio.c, ufs_dirhash.c New Synopsis: [ufs] [lor] lock-order reversal near vfs_bio.c, ufs_dirhash.c Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Sat Jul 25 01:17:44 UTC 2009 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=137034 From linimon at FreeBSD.org Sat Jul 25 01:27:37 2009 From: linimon at FreeBSD.org (linimon@FreeBSD.org) Date: Sat Jul 25 01:27:48 2009 Subject: kern/136942: [zfs] zvol resize not reflected until reboot Message-ID: <200907250127.n6P1RZxe034035@freefall.freebsd.org> Synopsis: [zfs] zvol resize not reflected until reboot Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Sat Jul 25 01:27:14 UTC 2009 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=136942 From linimon at FreeBSD.org Sat Jul 25 01:28:33 2009 From: linimon at FreeBSD.org (linimon@FreeBSD.org) Date: Sat Jul 25 01:28:39 2009 Subject: kern/136944: [ffs] [lor] bufwait/snaplk (fsync) Message-ID: <200907250128.n6P1SW7V034138@freefall.freebsd.org> Old Synopsis: [lor] bufwait/snaplk (fsync) New Synopsis: [ffs] [lor] bufwait/snaplk (fsync) Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Sat Jul 25 01:28:12 UTC 2009 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=136944 From linimon at FreeBSD.org Sat Jul 25 01:28:58 2009 From: linimon at FreeBSD.org (linimon@FreeBSD.org) Date: Sat Jul 25 01:29:03 2009 Subject: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) Message-ID: <200907250128.n6P1SukA034186@freefall.freebsd.org> Old Synopsis: [lor] filedesc structure/ufs (poll) New Synopsis: [ufs] [lor] filedesc structure/ufs (poll) Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Sat Jul 25 01:28:44 UTC 2009 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=136945 From linimon at FreeBSD.org Sat Jul 25 01:29:34 2009 From: linimon at FreeBSD.org (linimon@FreeBSD.org) Date: Sat Jul 25 01:29:41 2009 Subject: kern/136968: [ufs] [lor] ufs/bufwait/ufs (open) Message-ID: <200907250129.n6P1TXqf034237@freefall.freebsd.org> Old Synopsis: [lor] ufs/bufwait/ufs (open) New Synopsis: [ufs] [lor] ufs/bufwait/ufs (open) Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Sat Jul 25 01:29:20 UTC 2009 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=136968 From linimon at FreeBSD.org Sat Jul 25 02:33:13 2009 From: linimon at FreeBSD.org (linimon@FreeBSD.org) Date: Sat Jul 25 02:33:24 2009 Subject: kern/136865: [nfs] [patch] NFS exports atomic and on-the-fly atomic updates Message-ID: <200907250233.n6P2XBC3093162@freefall.freebsd.org> Old Synopsis: NFS exports atomic and on-the-fly atomic updates New Synopsis: [nfs] [patch] NFS exports atomic and on-the-fly atomic updates Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Sat Jul 25 02:32:38 UTC 2009 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=136865 From samankaya at netscape.net Sat Jul 25 02:36:18 2009 From: samankaya at netscape.net (samankaya@netscape.net) Date: Sat Jul 25 02:36:25 2009 Subject: Partitioning disks on PPC for FreeBSD, installer crashes In-Reply-To: <8CBDADB1950718E-16E8-7AF@webmail-dx04.sysops.aol.com> References: <8CBDADB1950718E-16E8-7AF@webmail-dx04.sysops.aol.com> Message-ID: <8CBDAE7B6DF4496-16E8-A34@webmail-dx04.sysops.aol.com> Ok, I managed to use the fixit part of the install menu to get to activate the holographic shell then by pressing alt F4 I was able to open up the shell prompt on tty4. So far good news :-) Using a bit of help by this: http://unix.derkeiler.com/Newsgroups/comp.unix.bsd.freebsd.misc/2004-10/0920.html I was able to figure out that echo * lists the contents of each cd'ed directory. Therefor I ran: cd /bin echo * this revealed the available commands to me as /bin and /sbin where both the same. I then did: cd /dev echo * found out that my hard drive was located at ad0 with a slice on ad0s1 then ran: newfs ad0 while I was in the /dev dir and it managed to print out all the sectors on the disk but I still haven't been able to work out how to create a new filesystem using and label using this command syntax! -----Original Message----- From: samankaya@netscape.net To: freebsd-fs@freebsd.org Sent: Sat, Jul 25, 2009 4:05 am Subject: Partitioning disks on PPC for FreeBSD, installer crashes Hi, this is a strange issue but I'm trying to install FreeBSD onto my PowerMac G4 Quicksilver 733MHz machine. I got as far as the partitioner in the standard install but couldn't figure out how to add slices as there is no option to create a disk label for it. Each time I try to use (C)reate a slice I get a message on the screen saying that I have to do this on a Master Partition.... How do I create this?? I tried dropping into a shell do use fdisk on it but it wouldn't allow me to do so using the holographic shell via the installer menu. Can anyone help give me an idea of what to do?? I used a Debian Linux boot cd which works fine and is able to detect the disk and partition it. I also created the needed 8MB yaboot partition in order to perhaps detect the disk but no such luck. The CD is iso1 of FreeBSD 7.1 I hope I don't have to go back to Debian as I really want to learn BSD!! Regards, Kaya A Good Credit Score is 700 or Above. See yours in just 2 easy steps! From snb at FreeBSD.org Sat Jul 25 10:20:03 2009 From: snb at FreeBSD.org (Nick Barkas) Date: Sat Jul 25 10:20:10 2009 Subject: kern/137034: [ufs] [lor] lock-order reversal near vfs_bio.c, ufs_dirhash.c Message-ID: <200907251020.n6PAK3hr088021@freefall.freebsd.org> The following reply was made to PR kern/137034; it has been noted by GNATS. From: Nick Barkas To: bug-followup@FreeBSD.org, royce@tycho.org Cc: Subject: Re: kern/137034: [ufs] [lor] lock-order reversal near vfs_bio.c, ufs_dirhash.c Date: Sat, 25 Jul 2009 12:11:42 +0200 This should be harmless. Take a look at http://sources.zabbadoz.net/freebsd/lor/261.html Have you had any problems since the LOR message came up? From tonyt at logyst.com Sat Jul 25 11:00:08 2009 From: tonyt at logyst.com (Tony Theodore) Date: Sat Jul 25 11:00:15 2009 Subject: Partitioning disks on PPC for FreeBSD, installer crashes In-Reply-To: <8CBDADB1950718E-16E8-7AF@webmail-dx04.sysops.aol.com> References: <8CBDADB1950718E-16E8-7AF@webmail-dx04.sysops.aol.com> Message-ID: <22166b750907250326w4d490c76k70f9f43b0c3b51e2@mail.gmail.com> > > Each time I try to use (C)reate a slice I get a message on the screen saying that I have to do this on a Master Partition.... > I think there are several partitioning schemes: Apple (open firmware), GPT (EFI), and MBR (BIOS). FreeBSD supports either MBR or GPT, but I'm not sure if sysinstall will recognise anything other than MBR. Have you tried the "use entire disk" option or do you need to keep other partitions? Otherwise, I'd try using Apple's disk utility, or something like gparted, to convert the partition scheme first. Tony From samankaya at netscape.net Sat Jul 25 12:03:22 2009 From: samankaya at netscape.net (samankaya@netscape.net) Date: Sat Jul 25 12:03:30 2009 Subject: Partitioning disks on PPC for FreeBSD, installer crashes In-Reply-To: <22166b750907250326w4d490c76k70f9f43b0c3b51e2@mail.gmail.com> References: <8CBDADB1950718E-16E8-7AF@webmail-dx04.sysops.aol.com> <22166b750907250326w4d490c76k70f9f43b0c3b51e2@mail.gmail.com> Message-ID: <8CBDB36F2A2368E-1950-2250@webmail-me04.sysops.aol.com> Many thanks Tony, I have ripped this disk out of an x86 so there were originally ext3 and NTFS partitions on it which I deleted from the x86 itself! The only thing I didn't delete was the filesystem which remained on the disk. Recently I stuck in a Debian installer and created one yaboot partition of 8MB and the rest as ext3. If I partition from Apples partitioner all I will get is HFS and HFS+ options and I am assuming here that BSD will not detect those either?? There unfortunately is no option for selecting the entire disk. In fact under the installer I don't see the disk at all which is strange. I do via tty4 and the holographic terminal though. Back when I started with Sun SPARC machines a few months ago, I figured out that Linux wouldn't see the disk or Solaris wouldn't see the disk depending on the label whether it was Sun or alternate. I am not sure if it is something similar here! I mean is there a way to partition using the Hg terminal at all??? As I stated I have discovered the newfs command but it is highly limited in what it can do and I have no idea how to use it either properly. What should be my next move?? I can happily try Apple's partitioner but I only have OS9 OS 10.1 and 10.2 neither of these guys allow for BSD partition tables if i recall correctly.... Kaya -----Original Message----- From: Tony Theodore To: samankaya@netscape.net Cc: freebsd-fs@freebsd.org Sent: Sat, Jul 25, 2009 1:26 pm Subject: Re: Partitioning disks on PPC for FreeBSD, installer crashes > > Each time I try to use (C)reate a slice I get a message on the screen saying that I have to do this on a Master Partition.... > I think there are several partitioning schemes: Apple (open firmware), GPT (EFI), and MBR (BIOS). FreeBSD supports either MBR or GPT, but I'm not sure if sysinstall will recognise anything other than MBR. Have you tried the "use entire disk" option or do you need to keep other partitions? Otherwise, I'd try using Apple's disk utility, or something like gparted, to convert the partition scheme first. Tony _______________________________________________ freebsd-fs@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From royce at tycho.org Sat Jul 25 15:10:03 2009 From: royce at tycho.org (Royce Williams) Date: Sat Jul 25 15:10:10 2009 Subject: kern/137034: [ufs] [lor] lock-order reversal near vfs_bio.c, ufs_dirhash.c Message-ID: <200907251510.n6PFA3t3015327@freefall.freebsd.org> The following reply was made to PR kern/137034; it has been noted by GNATS. From: Royce Williams To: Nick Barkas Cc: bug-followup@FreeBSD.org Subject: Re: kern/137034: [ufs] [lor] lock-order reversal near vfs_bio.c, ufs_dirhash.c Date: Sat, 25 Jul 2009 07:02:58 -0800 Nick Barkas wrote, on 7/25/2009 2:11 AM: > This should be harmless. Take a look at > http://sources.zabbadoz.net/freebsd/lor/261.html > > Have you had any problems since the LOR message came up? None; if harmless, then please close this PR. I will try to do a better job of detecting duplicate issue reports in the future. Sorry for the trouble. Royce From tonyt at logyst.com Sat Jul 25 16:50:08 2009 From: tonyt at logyst.com (Tony Theodore) Date: Sat Jul 25 16:50:15 2009 Subject: Partitioning disks on PPC for FreeBSD, installer crashes In-Reply-To: <8CBDB36F2A2368E-1950-2250@webmail-me04.sysops.aol.com> References: <8CBDADB1950718E-16E8-7AF@webmail-dx04.sysops.aol.com> <22166b750907250326w4d490c76k70f9f43b0c3b51e2@mail.gmail.com> <8CBDB36F2A2368E-1950-2250@webmail-me04.sysops.aol.com> Message-ID: <22166b750907250950g741b3e2dqce1c046f8f5e351c@mail.gmail.com> 2009/7/25 : > Many thanks Tony, > > I have ripped this disk out of an x86 so there were originally ext3 and NTFS > partitions on it which I deleted from the x86 itself! The only thing I > didn't delete was the filesystem which remained on the disk. > > Recently I stuck in a Debian installer and created one yaboot partition of > 8MB and the rest as ext3. If I partition from Apples partitioner all I will > get is HFS and HFS+ options and I am assuming here that BSD will not detect > those either?? > > There unfortunately is no option for selecting the entire disk. In fact > under the installer I don't see the disk at all which is strange. I do via > tty4 and the holographic terminal though. You would have seen the disk at the point you were pressing "C" to create a slice. It's the "A" option on this screen. > Back when I started with Sun SPARC machines a few months ago, I figured out > that Linux wouldn't see the disk or Solaris wouldn't see the disk depending > on the label whether it was Sun or alternate. I am not sure if it is > something similar here! It could be, Sun use EFI, BIOS, and Open Firmware, and I think they all read partition tables differently. > I mean is there a way to partition using the Hg terminal at all??? As I > stated I have discovered the newfs command but it is highly limited in what > it can do and I have no idea how to use it either properly. The commands you want are fdisk, bsdlabel, and newfs. If you're new to FreeBSD, the man pages and handbook will take you very far. I'd stick to the installer though, as you'll also have to create /etc/fstab among other things that I wouldn't know where to start. You could try this (from http://www.freebsd.org/cgi/man.cgi?query=bsdlabel&manpath=FreeBSD+7.1-RELEASE) dd if=/dev/zero of=/dev/da0 bs=512 count=32 fdisk -BI da0 dd if=/dev/zero of=/dev/da0s1 bs=512 count=32 bsdlabel -w -B da0s1 Then relaunch the installer to see if it's recognised. > What should be my next move?? I can happily try Apple's partitioner but I > only have OS9 OS 10.1 and 10.2 neither of these guys allow for BSD partition > tables if i recall correctly.... See if the Apple partitioner can create an MBR partition scheme. If not, find a live cd with gparted. I normally use sysresccd.org, but the ppc is very old. Otherwise, freebsd-questions might be a better place to ask, as this isn't really related to filesystems. Regards, Tony From snb at FreeBSD.org Sun Jul 26 11:49:54 2009 From: snb at FreeBSD.org (snb@FreeBSD.org) Date: Sun Jul 26 11:50:01 2009 Subject: kern/137034: [ufs] [lor] lock-order reversal near vfs_bio.c, ufs_dirhash.c Message-ID: <200907261149.n6QBnrBd007482@freefall.freebsd.org> Synopsis: [ufs] [lor] lock-order reversal near vfs_bio.c, ufs_dirhash.c State-Changed-From-To: open->closed State-Changed-By: snb State-Changed-When: Sön 26 Jul 2009 11:39:52 UTC State-Changed-Why: This LOR cannot result in deadlock: http://sources.zabbadoz.net/freebsd/lor/261.html http://www.freebsd.org/cgi/query-pr.cgi?pr=137034 From linimon at FreeBSD.org Mon Jul 27 04:43:47 2009 From: linimon at FreeBSD.org (linimon@FreeBSD.org) Date: Mon Jul 27 04:43:54 2009 Subject: kern/137037: [zfs] [hang] zfs rollback on root causes FreeBSD to freeze in few seconds Message-ID: <200907270443.n6R4hkDc086872@freefall.freebsd.org> Old Synopsis: zfs rolback on root causes FreeBSD to freez in few seconds New Synopsis: [zfs] [hang] zfs rollback on root causes FreeBSD to freeze in few seconds Responsible-Changed-From-To: freebsd-i386->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Mon Jul 27 04:43:14 UTC 2009 Responsible-Changed-Why: reclassify. http://www.freebsd.org/cgi/query-pr.cgi?pr=137037 From lists at jpru.de Mon Jul 27 07:43:48 2009 From: lists at jpru.de (Juergen Unger) Date: Mon Jul 27 07:43:55 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode Message-ID: <20090727072503.GA52309@jpru.ffm.jpru.de> Hi, I have one box where I am doing an zfs-receive job every five minutes for each of eleven zvols. Beside this no other service runs on this box. The system is an current checked out Jul 25 09:11 CET compiled with the following options: > include GENERIC > options KVA_PAGES=512 > options KDB > options DDB uname -a: > FreeBSD testbox 8.0-BETA2 FreeBSD 8.0-BETA2 #0: Sat Jul 25 21:43:42 CEST 2009 root@testbox:/usr/obj/usr/src/sys/ZFS-DEBUG i386 loader.conf: > geom_mirror_load="YES" > vm.kmem_size="1536M" > vm.kmem_size_max="1536M" > vfs.zfs.arc_max="100M" > vfs.zfs.prefetch_disable=1 This runs quite well for a few hours but after max 20 to 30 hours I get this error: > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0x4c > fault code = supervisor read, page not present > instruction pointer = 0x20:0x80883d93 > stack pointer = 0x28:0xfcd29b74 > frame pointer = 0x28:0xfcd29b94 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, def32 1, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 36 (vnlru) > [thread pid 36 tid 100062 ] > Stopped at _sx_xlock+0x43: movl 0x10(%ebx),%eax > db> bt > Tracing pid 36 tid 100062 td 0x87166480 > _sx_xlock(3c,0,874aa28d,70f,8ae9a9f8,...) at _sx_xlock+0x43 > dmu_buf_update_user(0,8ae9a9f8,0,0,0,...) at dmu_buf_update_user+0x35 > zfs_znode_dmu_fini(8ae9a9f8,874b312d,1114,110b,879ab000,...) at zfs_znode_dmu_f3 > zfs_freebsd_reclaim(fcd29c3c,1,0,8ec63754,fcd29c60,...) at zfs_freebsd_reclaim+0 > VOP_RECLAIM_APV(874b65a0,fcd29c3c,0,0,8ec637c8,...) at VOP_RECLAIM_APV+0xa5 > vgonel(8ec637c8,0,80c77037,386,0,...) at vgonel+0x1a4 > vnlru_free(80f2a0f0,0,80c77037,300,3e8,...) at vnlru_free+0x2d5 > vnlru_proc(0,fcd29d38,80c652bc,33e,871932a8,...) at vnlru_proc+0x80 > fork_exit(8090d960,0,fcd29d38) at fork_exit+0xb8 > fork_trampoline() at fork_trampoline+0x8 > --- trap 0, eip = 0, esp = 0xfcd29d70, ebp = 0 --- > db> any suggestions ? Juergen -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20090727/c42b10f7/attachment.pgp From bugmaster at FreeBSD.org Mon Jul 27 11:06:53 2009 From: bugmaster at FreeBSD.org (FreeBSD bugmaster) Date: Mon Jul 27 11:08:10 2009 Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org Message-ID: <200907271106.n6RB6qSn018914@freefall.freebsd.org> Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/137037 fs [zfs] [hang] zfs rollback on root causes FreeBSD to fr o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136942 fs [zfs] zvol resize not reflected until reboot o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic o kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/136218 fs [zfs] Exported ZFS pools can't be imported into (Open) o kern/135594 fs [zfs] Single dataset unresponsive with Samba o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135480 fs [zfs] panic: lock &arg.lock already initialized o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o bin/135314 fs [zfs] assertion failed for zdb(8) usage o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot f kern/134496 fs [zfs] [panic] ZFS pool export occasionally causes a ke o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133980 fs [panic] [ffs] panic: ffs_valloc: dup alloc o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis o kern/133614 fs [smbfs] [panic] panic: ffs_truncate: read-only filesys o kern/133373 fs [zfs] umass attachment causes ZFS checksum errors, dat o kern/133174 fs [msdosfs] [patch] msdosfs must support utf-encoded int f kern/133150 fs [zfs] Page fault with ZFS on 7.1-RELEASE/amd64 while w o kern/133134 fs [zfs] Missing ZFS zpool labels f kern/133020 fs [zfs] [panic] inappropriate panic caused by zfs. Pani o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132597 fs [tmpfs] [panic] tmpfs-related panic while interrupting o kern/132551 fs [zfs] ZFS locks up on extattr_list_link syscall o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132337 fs [zfs] [panic] kernel panic in zfs_fuid_create_cred o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes f kern/132068 fs [zfs] page fault when using ZFS over NFS on 7.1-RELEAS o kern/131995 fs [nfs] Failure to mount NFSv4 server o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/131086 fs [ext2fs] [patch] mkfs.ext2 creates rotten partition o kern/130979 fs [smbfs] [panic] boot/kernel/smbfs.ko o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130229 fs [iconv] usermount fails on fs that need iconv o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/129148 fs [zfs] [panic] panic on concurrent writing & rollback o kern/129059 fs [zfs] [patch] ZFS bootloader whitelistable via WITHOUT f kern/128829 fs smbd(8) causes periodic panic on 7-RELEASE o kern/128633 fs [zfs] [lor] lock order reversal in zfs o kern/128514 fs [zfs] [mpt] problems with ZFS and LSILogic SAS/SATA Ad f kern/128173 fs [ext2fs] ls gives "Input/output error" on mounted ext3 o kern/127659 fs [tmpfs] tmpfs memory leak o kern/127492 fs [zfs] System hang on ZFS input-output o kern/127420 fs [gjournal] [panic] Journal overflow on gmirrored gjour o kern/127213 fs [tmpfs] sendfile on tmpfs data corruption o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS o kern/125644 fs [zfs] [panic] zfs unfixable fs errors caused panic whe f kern/125536 fs [ext2fs] ext 2 mounts cleanly but fails on commands li o kern/125149 fs [nfs] [panic] changing into .zfs dir from nfs client c f kern/124621 fs [ext3] [patch] Cannot mount ext2fs partition f bin/124424 fs [zfs] zfs(8): zfs list -r shows strange snapshots' siz o kern/123939 fs [msdosfs] corrupts new files o kern/122888 fs [zfs] zfs hang w/ prefetch on, zil off while running t o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o kern/122173 fs [zfs] [panic] Kernel Panic if attempting to replace a o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o kern/122047 fs [ext2fs] [patch] incorrect handling of UF_IMMUTABLE / o kern/122038 fs [tmpfs] [panic] tmpfs: panic: tmpfs_alloc_vp: type 0xc o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121779 fs [ufs] snapinfo(8) (and related tools?) only work for t o kern/121770 fs [zfs] ZFS on i386, large file or heavy I/O leads to ke o bin/121366 fs [zfs] [patch] Automatic disk scrubbing from periodic(8 o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha f kern/120991 fs [panic] [fs] [snapshot] System crashes when manipulati o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F o bin/120288 fs zfs(8): "zfs share -a" does not send SIGHUP to mountd f kern/119735 fs [zfs] geli + ZFS + samba starting on boot panics 7.0-B o kern/118912 fs [2tb] disk sizing/geometry problem with large array o misc/118855 fs [zfs] ZFS-related commands are nonfunctional in fixit o kern/118713 fs [minidump] [patch] Display media size required for a k o kern/118320 fs [zfs] [patch] NFS SETATTR sometimes fails to set file o bin/118249 fs mv(1): moving a directory changes its mtime o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117314 fs [ntfs] Long-filename only NTFS fs'es cause kernel pani o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o kern/116913 fs [ffs] [panic] ffs_blkfree: freeing free block p kern/116608 fs [msdosfs] [patch] msdosfs fails to check mount options o kern/116583 fs [ffs] [hang] System freezes for short time when using o kern/116170 fs [panic] Kernel panic when mounting /tmp o kern/115645 fs [snapshots] [panic] lockmgr: thread 0xc4c00d80, not ex o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o kern/113180 fs [zfs] Setting ZFS nfsshare property does not cause inh o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o kern/109024 fs [msdosfs] mount_msdosfs: msdosfs_iconv: Operation not o kern/109010 fs [msdosfs] can't mv directory within fat32 file system o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106030 fs [ufs] [panic] panic in ufs from geom when a dead disk o kern/105093 fs [ext2fs] [patch] ext2fs on read-only media cannot be m o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [iso9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna f kern/91568 fs [ufs] [panic] writing to UFS/softupdates DVD media in o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/89991 fs [ufs] softupdates with mount -ur causes fs UNREFS o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88266 fs [smbfs] smbfs does not implement UIO_NOCOPY and sendfi o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o kern/85326 fs [smbfs] [panic] saving a file via samba to an overquot o kern/84589 fs [2TB] 5.4-STABLE unresponsive during background fsck 2 o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o kern/77826 fs [ext2fs] ext2fs usb filesystem will not mount RW o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/51583 fs [nullfs] [patch] allow to work with devices and socket o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 151 problems total. From jhb at FreeBSD.org Mon Jul 27 13:40:03 2009 From: jhb at FreeBSD.org (John Baldwin) Date: Mon Jul 27 13:40:09 2009 Subject: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) Message-ID: <200907271340.n6RDe3YR042554@freefall.freebsd.org> The following reply was made to PR kern/136945; it has been noted by GNATS. From: John Baldwin To: bug-followup@freebsd.org, rene@freebsd.org Cc: Subject: Re: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) Date: Mon, 27 Jul 2009 08:32:26 -0400 I would actually expect this to be the correct order for these two locks. Can you capture the output of the 'debug.witness.fullgraph' sysctl to a file? -- John Baldwin From rene at freebsd.org Mon Jul 27 14:00:06 2009 From: rene at freebsd.org (Rene Ladan) Date: Mon Jul 27 14:00:11 2009 Subject: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) Message-ID: <200907271400.n6RE05Rv056472@freefall.freebsd.org> The following reply was made to PR kern/136945; it has been noted by GNATS. From: Rene Ladan To: John Baldwin Cc: bug-followup@freebsd.org Subject: Re: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) Date: Mon, 27 Jul 2009 15:51:15 +0200 2009/7/27 John Baldwin : > I would actually expect this to be the correct order for these two locks.= =A0Can > you capture the output of the 'debug.witness.fullgraph' sysctl to a file? > Yes, see attachment. I'm still running the same 8.0-BETA2. From pjd at FreeBSD.org Mon Jul 27 14:53:26 2009 From: pjd at FreeBSD.org (pjd@FreeBSD.org) Date: Mon Jul 27 14:53:43 2009 Subject: kern/132337: [zfs] [panic] kernel panic in zfs_fuid_create_cred Message-ID: <200907271453.n6RErNs2001742@freefall.freebsd.org> Synopsis: [zfs] [panic] kernel panic in zfs_fuid_create_cred State-Changed-From-To: open->patched State-Changed-By: pjd State-Changed-When: pon 27 lip 14:52:49 2009 UTC State-Changed-Why: Fix committed to HEAD. Responsible-Changed-From-To: freebsd-fs->pjd Responsible-Changed-By: pjd Responsible-Changed-When: pon 27 lip 14:52:49 2009 UTC Responsible-Changed-Why: I'll take this one. http://www.freebsd.org/cgi/query-pr.cgi?pr=132337 From pjd at FreeBSD.org Mon Jul 27 14:55:00 2009 From: pjd at FreeBSD.org (pjd@FreeBSD.org) Date: Mon Jul 27 14:55:07 2009 Subject: kern/133020: [zfs] [panic] inappropriate panic caused by zfs. Panic: zfs_fuid_create Message-ID: <200907271454.n6REsxM1001826@freefall.freebsd.org> Synopsis: [zfs] [panic] inappropriate panic caused by zfs. Panic: zfs_fuid_create State-Changed-From-To: feedback->closed State-Changed-By: pjd State-Changed-When: pon 27 lip 14:53:56 2009 UTC State-Changed-Why: Duplicate of 132337, which is now fixed in HEAD. Responsible-Changed-From-To: freebsd-fs->pjd Responsible-Changed-By: pjd Responsible-Changed-When: pon 27 lip 14:53:56 2009 UTC Responsible-Changed-Why: I'll take this one. http://www.freebsd.org/cgi/query-pr.cgi?pr=133020 From ohartman at mail.zedat.fu-berlin.de Mon Jul 27 20:15:30 2009 From: ohartman at mail.zedat.fu-berlin.de (O. Hartmann) Date: Mon Jul 27 20:15:36 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <20090727072503.GA52309@jpru.ffm.jpru.de> References: <20090727072503.GA52309@jpru.ffm.jpru.de> Message-ID: <4A6E06E6.9030300@mail.zedat.fu-berlin.de> Juergen Unger wrote: > Hi, > > I have one box where I am doing an zfs-receive job every > five minutes for each of eleven zvols. Beside this no > other service runs on this box. > The system is an current checked out Jul 25 09:11 CET > compiled with the following options: > >> include GENERIC >> options KVA_PAGES=512 >> options KDB >> options DDB > > uname -a: > >> FreeBSD testbox 8.0-BETA2 FreeBSD 8.0-BETA2 #0: Sat Jul 25 21:43:42 CEST 2009 root@testbox:/usr/obj/usr/src/sys/ZFS-DEBUG i386 > > loader.conf: > >> geom_mirror_load="YES" >> vm.kmem_size="1536M" >> vm.kmem_size_max="1536M" >> vfs.zfs.arc_max="100M" >> vfs.zfs.prefetch_disable=1 > > This runs quite well for a few hours but after max 20 to 30 > hours I get this error: > >> Fatal trap 12: page fault while in kernel mode >> cpuid = 0; apic id = 00 >> fault virtual address = 0x4c >> fault code = supervisor read, page not present >> instruction pointer = 0x20:0x80883d93 >> stack pointer = 0x28:0xfcd29b74 >> frame pointer = 0x28:0xfcd29b94 >> code segment = base 0x0, limit 0xfffff, type 0x1b >> = DPL 0, pres 1, def32 1, gran 1 >> processor eflags = interrupt enabled, resume, IOPL = 0 >> current process = 36 (vnlru) >> [thread pid 36 tid 100062 ] >> Stopped at _sx_xlock+0x43: movl 0x10(%ebx),%eax >> db> bt >> Tracing pid 36 tid 100062 td 0x87166480 >> _sx_xlock(3c,0,874aa28d,70f,8ae9a9f8,...) at _sx_xlock+0x43 >> dmu_buf_update_user(0,8ae9a9f8,0,0,0,...) at dmu_buf_update_user+0x35 >> zfs_znode_dmu_fini(8ae9a9f8,874b312d,1114,110b,879ab000,...) at zfs_znode_dmu_f3 >> zfs_freebsd_reclaim(fcd29c3c,1,0,8ec63754,fcd29c60,...) at zfs_freebsd_reclaim+0 >> VOP_RECLAIM_APV(874b65a0,fcd29c3c,0,0,8ec637c8,...) at VOP_RECLAIM_APV+0xa5 >> vgonel(8ec637c8,0,80c77037,386,0,...) at vgonel+0x1a4 >> vnlru_free(80f2a0f0,0,80c77037,300,3e8,...) at vnlru_free+0x2d5 >> vnlru_proc(0,fcd29d38,80c652bc,33e,871932a8,...) at vnlru_proc+0x80 >> fork_exit(8090d960,0,fcd29d38) at fork_exit+0xb8 >> fork_trampoline() at fork_trampoline+0x8 >> --- trap 0, eip = 0, esp = 0xfcd29d70, ebp = 0 --- >> db> > > any suggestions ? > > Juergen > I see a similar problem on two SMP boxes (is your SMP?), but in my case, it seems not to be ZFS related although I also use ZFS as /home filesystem From lists at jpru.de Mon Jul 27 21:33:58 2009 From: lists at jpru.de (Juergen Unger) Date: Mon Jul 27 21:34:10 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <4A6E06E6.9030300@mail.zedat.fu-berlin.de> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> Message-ID: <20090727213355.GA37551@jpru.ffm.jpru.de> Hi, On Mon, Jul 27, 2009 at 09:58:30PM +0200, O. Hartmann wrote: [...] > >> db> bt > >> Tracing pid 36 tid 100062 td 0x87166480 > >> _sx_xlock(3c,0,874aa28d,70f,8ae9a9f8,...) at _sx_xlock+0x43 > >> dmu_buf_update_user(0,8ae9a9f8,0,0,0,...) at dmu_buf_update_user+0x35 > >> zfs_znode_dmu_fini(8ae9a9f8,874b312d,1114,110b,879ab000,...) at zfs_znode_dmu_f3 > >> zfs_freebsd_reclaim(fcd29c3c,1,0,8ec63754,fcd29c60,...) at zfs_freebsd_reclaim+0 > >> VOP_RECLAIM_APV(874b65a0,fcd29c3c,0,0,8ec637c8,...) at VOP_RECLAIM_APV+0xa5 > >> vgonel(8ec637c8,0,80c77037,386,0,...) at vgonel+0x1a4 > >> vnlru_free(80f2a0f0,0,80c77037,300,3e8,...) at vnlru_free+0x2d5 > >> vnlru_proc(0,fcd29d38,80c652bc,33e,871932a8,...) at vnlru_proc+0x80 > >> fork_exit(8090d960,0,fcd29d38) at fork_exit+0xb8 > >> fork_trampoline() at fork_trampoline+0x8 > >> --- trap 0, eip = 0, esp = 0xfcd29d70, ebp = 0 --- > >> db> > > I see a similar problem on two SMP boxes (is your SMP?), but in my case, > it seems not to be ZFS related although I also use ZFS as /home filesystem no real SMP, its only an old P4 with hyperthreading: > CPU: Intel(R) Pentium(R) 4 CPU 3.20GHz (3192.02-MHz 686-class CPU) > Origin = "GenuineIntel" Id = 0xf41 Stepping = 1 > Features=0xbfebfbff > Features2=0x441d > AMD Features=0x100000 > TSC: P-state invariant > real memory = 4294967296 (4096 MB) > avail memory = 3392716800 (3235 MB) > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs > FreeBSD/SMP: 1 package(s) x 1 core(s) x 2 HTT threads Juergen -- ENOSIG From bo.coopci at gmail.com Tue Jul 28 06:43:09 2009 From: bo.coopci at gmail.com (cooper) Date: Tue Jul 28 06:43:16 2009 Subject: About RAIDZ geometry Message-ID: <4A6E970A.1060503@gmail.com> Hi folks, Now that ZFS has been ported to FreeBSD, I think it's proper to ask these questions in this mailing list. Is there any detailed documentation about RAIDZ geometry? I has been wandering how zfs remembers which disk sectors are for data and which ones are for parity. They claimed "Every block is its own RAID-Z stripe" does that mean zfs writes exactly one block in every vdev_raidz_io_start if the io type of the zio is write? And also claimed "You have to traverse the filesystem metadata to determine the RAID-Z geometry" where is this metadata stored? From avg at icyb.net.ua Tue Jul 28 10:03:07 2009 From: avg at icyb.net.ua (Andriy Gapon) Date: Tue Jul 28 10:03:54 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <4A6E06E6.9030300@mail.zedat.fu-berlin.de> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> Message-ID: <4A6EC9E2.5070200@icyb.net.ua> on 27/07/2009 22:58 O. Hartmann said the following: > Juergen Unger wrote: [snip] >>> _sx_xlock(3c,0,874aa28d,70f,8ae9a9f8,...) at _sx_xlock+0x43 >>> dmu_buf_update_user(0,8ae9a9f8,0,0,0,...) at dmu_buf_update_user+0x35 >>> zfs_znode_dmu_fini(8ae9a9f8,874b312d,1114,110b,879ab000,...) at zfs_znode_dmu_f3 >>> zfs_freebsd_reclaim(fcd29c3c,1,0,8ec63754,fcd29c60,...) at zfs_freebsd_reclaim+0 >>> VOP_RECLAIM_APV(874b65a0,fcd29c3c,0,0,8ec637c8,...) at VOP_RECLAIM_APV+0xa5 >>> vgonel(8ec637c8,0,80c77037,386,0,...) at vgonel+0x1a4 >>> vnlru_free(80f2a0f0,0,80c77037,300,3e8,...) at vnlru_free+0x2d5 >>> vnlru_proc(0,fcd29d38,80c652bc,33e,871932a8,...) at vnlru_proc+0x80 >>> fork_exit(8090d960,0,fcd29d38) at fork_exit+0xb8 >>> fork_trampoline() at fork_trampoline+0x8 [snip] > > I see a similar problem on two SMP boxes (is your SMP?), but in my case, > it seems not to be ZFS related although I also use ZFS as /home filesystem In this case this does seem to be caused by ZFS. >From the backtrace we see that _sx_xlock() is called on bogus struct sx pointer (0x3c) and this is caused by dmu_buf_update_user() called with NULL first argument (dmu_buf_t). Which means that znode_t z_dbuf was NULL - this could have been caught by ASSERT in zfs_znode_dmu_fini if it were enabled. If you have the crash dump, then it would be interesting to examine znode_t structure ('zp' argument) in zfs_znode_dmu_fini. P.S. I see that zfs_inactive checks for z_dbuf being NULL and there is the following comment: /* * The fs has been unmounted, or we did a * suspend/resume and this file no longer exists. */ Maybe zfs_freebsd_reclaim should do the same? P.P.S. I am not a VFS or ZFS expert. -- Andriy Gapon From jhb at freebsd.org Tue Jul 28 13:50:45 2009 From: jhb at freebsd.org (John Baldwin) Date: Tue Jul 28 13:50:52 2009 Subject: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) In-Reply-To: <200907271400.n6RE05Rv056472@freefall.freebsd.org> References: <200907271400.n6RE05Rv056472@freefall.freebsd.org> Message-ID: <200907280941.32840.jhb@freebsd.org> On Monday 27 July 2009 10:00:05 am Rene Ladan wrote: > The following reply was made to PR kern/136945; it has been noted by GNATS. > > From: Rene Ladan > To: John Baldwin > Cc: bug-followup@freebsd.org > Subject: Re: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) > Date: Mon, 27 Jul 2009 15:51:15 +0200 > > 2009/7/27 John Baldwin : > > I would actually expect this to be the correct order for these two locks.= > =A0Can > > you capture the output of the 'debug.witness.fullgraph' sysctl to a file? > > > Yes, see attachment. I'm still running the same 8.0-BETA2. Hmm, the attachment was eaten by a grue, can you post the file somewhere? -- John Baldwin From r.c.ladan at gmail.com Tue Jul 28 14:35:17 2009 From: r.c.ladan at gmail.com (Rene Ladan) Date: Tue Jul 28 14:35:28 2009 Subject: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) In-Reply-To: <200907280941.32840.jhb@freebsd.org> References: <200907271400.n6RE05Rv056472@freefall.freebsd.org> <200907280941.32840.jhb@freebsd.org> Message-ID: 2009/7/28 John Baldwin : > On Monday 27 July 2009 10:00:05 am Rene Ladan wrote: >> The following reply was made to PR kern/136945; it has been noted by GNATS. >> >> From: Rene Ladan >> To: John Baldwin >> Cc: bug-followup@freebsd.org >> Subject: Re: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) >> Date: Mon, 27 Jul 2009 15:51:15 +0200 >> >> ?2009/7/27 John Baldwin : >> ?> I would actually expect this to be the correct order for these two > locks.= >> ? =A0Can >> ?> you capture the output of the 'debug.witness.fullgraph' sysctl to a file? >> ?> >> ?Yes, see attachment. ?I'm still running the same 8.0-BETA2. > > Hmm, the attachment was eaten by a grue, can you post the file somewhere? > Yes, see ftp://rene-ladan.nl/pub/freebsd/kern_136945.txt Ren? From jhb at freebsd.org Tue Jul 28 14:38:54 2009 From: jhb at freebsd.org (John Baldwin) Date: Tue Jul 28 14:39:01 2009 Subject: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) In-Reply-To: References: <200907271400.n6RE05Rv056472@freefall.freebsd.org> <200907280941.32840.jhb@freebsd.org> Message-ID: <200907281038.30277.jhb@freebsd.org> On Tuesday 28 July 2009 10:03:40 am Rene Ladan wrote: > 2009/7/28 John Baldwin : > > On Monday 27 July 2009 10:00:05 am Rene Ladan wrote: > >> The following reply was made to PR kern/136945; it has been noted by GNATS. > >> > >> From: Rene Ladan > >> To: John Baldwin > >> Cc: bug-followup@freebsd.org > >> Subject: Re: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) > >> Date: Mon, 27 Jul 2009 15:51:15 +0200 > >> > >> ?2009/7/27 John Baldwin : > >> ?> I would actually expect this to be the correct order for these two > > locks.= > >> ? =A0Can > >> ?> you capture the output of the 'debug.witness.fullgraph' sysctl to a file? > >> ?> > >> ?Yes, see attachment. ?I'm still running the same 8.0-BETA2. > > > > Hmm, the attachment was eaten by a grue, can you post the file somewhere? > > > Yes, see ftp://rene-ladan.nl/pub/freebsd/kern_136945.txt Ok, it looks like it did encounter a UFS -> filedesc order at some point. Can you patch sys/kern/subr_witness.c to add a section to the order_lists[] array after the 'ZFS locking list' and before the spin locks list that looks like this: { "filedesc structure", &lock_class_sx }, { "ufs", &lock_class_lockmgr}, { NULL, NULL }, -- John Baldwin From james-freebsd-fs2 at jrv.org Wed Jul 29 05:49:12 2009 From: james-freebsd-fs2 at jrv.org (James R. Van Artsdalen) Date: Wed Jul 29 05:49:18 2009 Subject: [ZFS] umount at reboot crashes Message-ID: <4A6FE2D6.7050300@jrv.org> FreeBSD bigback.housenet.jrv 8.0-BETA2 FreeBSD 8.0-BETA2 #0 r195757M: Mon Jul 20 10:27:28 CDT 2009 james@bigback.housenet.jrv:/usr/obj/usr/src/sys/BIGTEX amd64 I have a system that almost always crashes whenever it receives a ZFS replication package ("zfs recv") that either deleted or renames filesystems, both operations requiring unmounting. Sometimes it crashes later in the "zfs recv", sometimes not until I reboot that system. The sx_xlock() in frame 10 seems a common theme in these crashes. The dump is available. #0 doadump () at pcpu.h:223 223 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump () at pcpu.h:223 #1 0xffffffff801dfdec in db_fncall (dummy1=Variable "dummy1" is not available. ) at /usr/src/sys/ddb/db_command.c:548 #2 0xffffffff801e0121 in db_command (last_cmdp=0xffffffff80bbd9e0, cmd_table=Variable "cmd_table" is not available. ) at /usr/src/sys/ddb/db_command.c:445 #3 0xffffffff801e0370 in db_command_loop () at /usr/src/sys/ddb/db_command.c:498 #4 0xffffffff801e2349 in db_trap (type=Variable "type" is not available. ) at /usr/src/sys/ddb/db_main.c:229 #5 0xffffffff805bab85 in kdb_trap (type=12, code=0, tf=0xffffff810f20a690) at /usr/src/sys/kern/subr_kdb.c:534 #6 0xffffffff8083cf7d in trap_fatal (frame=0xffffff810f20a690, eva=Variable "eva" is not available. ) at /usr/src/sys/amd64/amd64/trap.c:847 #7 0xffffffff8083d2ed in trap_pfault (frame=0xffffff810f20a690, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:768 #8 0xffffffff8083dce3 in trap (frame=0xffffff810f20a690) at /usr/src/sys/amd64/amd64/trap.c:494 #9 0xffffffff80823883 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:224 #10 0xffffffff80592e4a in _sx_xlock (sx=0x58, opts=0, file=0xffffffff810b4d68 "/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c", line=1807) at /usr/src/sys/kern/kern_sx.c:284 #11 0xffffffff80ffa9d7 in dmu_buf_update_user (db_fake=0x0, old_user_ptr=0xffffff0148924468, user_ptr=0x0, user_data_ptr_ptr=0x0, evict_func=0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1807 #12 0xffffffff810401e8 in zfs_znode_dmu_fini (zp=0xffffff0148924468) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:551 #13 0xffffffff8105fcee in zfs_freebsd_reclaim (ap=Variable "ap" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:4381 #14 0xffffffff8061ae05 in vgonel (vp=0xffffff014893f3b0) at vnode_if.h:830 #15 0xffffffff8061e975 in vflush (mp=0xffffff01468355e0, rootrefs=0, flags=0, td=0xffffff01468a3000) at /usr/src/sys/kern/vfs_subr.c:2449 #16 0xffffffff8105a598 in zfs_umount (vfsp=0xffffff01468355e0, fflag=524288) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:996 #17 0xffffffff80616336 in dounmount (mp=0xffffff01468355e0, flags=524288, td=Variable "td" is not available. ) at /usr/src/sys/kern/vfs_mount.c:1289 #18 0xffffffff8061be54 in vfs_unmountall () at /usr/src/sys/kern/vfs_subr.c:3141 #19 0xffffffff8058b58f in boot (howto=0) at /usr/src/sys/kern/kern_shutdown.c:401 #20 0xffffffff8058b8b8 in reboot (td=Variable "td" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:173 #21 0xffffffff8083d4af in syscall (frame=0xffffff810f20ac80) at /usr/src/sys/amd64/amd64/trap.c:984 #22 0xffffffff80823b61 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:373 #23 0x000000080078f96c in ?? () Previous frame inner to this frame (corrupt stack?) From pjd at FreeBSD.org Wed Jul 29 08:40:48 2009 From: pjd at FreeBSD.org (Pawel Jakub Dawidek) Date: Wed Jul 29 08:40:55 2009 Subject: About RAIDZ geometry In-Reply-To: <4A6E970A.1060503@gmail.com> References: <4A6E970A.1060503@gmail.com> Message-ID: <20090729081900.GC1586@garage.freebsd.pl> On Tue, Jul 28, 2009 at 02:13:30PM +0800, cooper wrote: > Hi folks, > > Now that ZFS has been ported to FreeBSD, I think it's proper to ask > these questions in this mailing list. > > Is there any detailed documentation about RAIDZ geometry? I has been > wandering how zfs remembers which disk sectors are for data and which > ones are for parity. > > They claimed "Every block is its own RAID-Z stripe" does that mean zfs > writes exactly one block in every vdev_raidz_io_start if the io type of > the zio is write? > > And also claimed "You have to traverse the filesystem metadata to > determine the RAID-Z geometry" where is this metadata stored? If you have file system separated from volume manager doing RAID, the volume manager has no knowledge where the live data is, etc. Imagine a ZFS 5-disks RAIDZ1 configuration: disk0 disk1 disk2 disk3 disk4 ZFS' file system level wants to store two sectors of data so it sends 1kB of data to lower ZFS layers. You then get something like this: disk0 disk1 disk2 disk3 disk4 data0 data1 parity01 Now ZFS wants to store one sector and you get this: disk0 disk1 disk2 disk3 disk4 data0 data1 parity01 data2 parity2 ZFS can do that, because he knows where and how to read the data back (its a file system and volume manager). ZFS will never try to read three sectors as one block of data. This is impossible to do when file system is separated from volume manager. -- Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20090729/80e60ce0/attachment.pgp From pjd at FreeBSD.org Wed Jul 29 08:47:18 2009 From: pjd at FreeBSD.org (Pawel Jakub Dawidek) Date: Wed Jul 29 08:47:25 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <4A6EC9E2.5070200@icyb.net.ua> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> Message-ID: <20090729084723.GD1586@garage.freebsd.pl> On Tue, Jul 28, 2009 at 12:50:26PM +0300, Andriy Gapon wrote: > on 27/07/2009 22:58 O. Hartmann said the following: > > Juergen Unger wrote: > [snip] > >>> _sx_xlock(3c,0,874aa28d,70f,8ae9a9f8,...) at _sx_xlock+0x43 > >>> dmu_buf_update_user(0,8ae9a9f8,0,0,0,...) at dmu_buf_update_user+0x35 > >>> zfs_znode_dmu_fini(8ae9a9f8,874b312d,1114,110b,879ab000,...) at zfs_znode_dmu_f3 > >>> zfs_freebsd_reclaim(fcd29c3c,1,0,8ec63754,fcd29c60,...) at zfs_freebsd_reclaim+0 > >>> VOP_RECLAIM_APV(874b65a0,fcd29c3c,0,0,8ec637c8,...) at VOP_RECLAIM_APV+0xa5 > >>> vgonel(8ec637c8,0,80c77037,386,0,...) at vgonel+0x1a4 > >>> vnlru_free(80f2a0f0,0,80c77037,300,3e8,...) at vnlru_free+0x2d5 > >>> vnlru_proc(0,fcd29d38,80c652bc,33e,871932a8,...) at vnlru_proc+0x80 > >>> fork_exit(8090d960,0,fcd29d38) at fork_exit+0xb8 > >>> fork_trampoline() at fork_trampoline+0x8 > [snip] > > > > I see a similar problem on two SMP boxes (is your SMP?), but in my case, > > it seems not to be ZFS related although I also use ZFS as /home filesystem > > In this case this does seem to be caused by ZFS. > >From the backtrace we see that _sx_xlock() is called on bogus struct sx pointer > (0x3c) and this is caused by dmu_buf_update_user() called with NULL first argument > (dmu_buf_t). Which means that znode_t z_dbuf was NULL - this could have been > caught by ASSERT in zfs_znode_dmu_fini if it were enabled. > > If you have the crash dump, then it would be interesting to examine znode_t > structure ('zp' argument) in zfs_znode_dmu_fini. > > P.S. I see that zfs_inactive checks for z_dbuf being NULL and there is the > following comment: > /* > * The fs has been unmounted, or we did a > * suspend/resume and this file no longer exists. > */ > Maybe zfs_freebsd_reclaim should do the same? Yes, you might be right. Could you guys, who can reproduce it, try this patch: http://people.freebsd.org/~pjd/patches/zfs_vnops.c.2.patch -- Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20090729/defb6791/attachment.pgp From zlo at zlo.nu Wed Jul 29 09:20:03 2009 From: zlo at zlo.nu (Marc Olzheim) Date: Wed Jul 29 09:20:10 2009 Subject: kern/127213: [tmpfs] sendfile on tmpfs data corruption Message-ID: <200907290920.n6T9K2TZ025796@freefall.freebsd.org> The following reply was made to PR kern/127213; it has been noted by GNATS. From: Marc Olzheim To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/127213: [tmpfs] sendfile on tmpfs data corruption Date: Wed, 29 Jul 2009 10:51:40 +0200 This is not just a bug, but a security hole. What you are seeing is the content of other VM pages. This has exactly the same symptoms as http://security.freebsd.org/advisories/FreeBSD-SA-05:02.sendfile.asc In short: do not use tmpfs unless you trust all users on the system with root access. From rene at freebsd.org Wed Jul 29 09:52:26 2009 From: rene at freebsd.org (Rene Ladan) Date: Wed Jul 29 09:52:33 2009 Subject: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) In-Reply-To: <200907281038.30277.jhb@freebsd.org> References: <200907271400.n6RE05Rv056472@freefall.freebsd.org> <200907280941.32840.jhb@freebsd.org> <200907281038.30277.jhb@freebsd.org> Message-ID: 2009/7/28 John Baldwin : > On Tuesday 28 July 2009 10:03:40 am Rene Ladan wrote: >> 2009/7/28 John Baldwin : >> > On Monday 27 July 2009 10:00:05 am Rene Ladan wrote: >> >> The following reply was made to PR kern/136945; it has been noted by > GNATS. >> >> >> >> From: Rene Ladan >> >> To: John Baldwin >> >> Cc: bug-followup@freebsd.org >> >> Subject: Re: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) >> >> Date: Mon, 27 Jul 2009 15:51:15 +0200 >> >> >> >> ?2009/7/27 John Baldwin : >> >> ?> I would actually expect this to be the correct order for these two >> > locks.= >> >> ? =A0Can >> >> ?> you capture the output of the 'debug.witness.fullgraph' sysctl to a > file? >> >> ?> >> >> ?Yes, see attachment. ?I'm still running the same 8.0-BETA2. >> > >> > Hmm, the attachment was eaten by a grue, can you post the file somewhere? >> > >> Yes, see ftp://rene-ladan.nl/pub/freebsd/kern_136945.txt > > Ok, it looks like it did encounter a UFS -> filedesc order at some point. ?Can > you patch sys/kern/subr_witness.c to add a section to the order_lists[] array > after the 'ZFS locking list' and before the spin locks list that looks like > this: > > ? ? ? ?{ "filedesc structure", &lock_class_sx }, > ? ? ? ?{ "ufs", &lock_class_lockmgr}, > ? ? ? ?{ NULL, NULL }, > The LOR seems to be gone, previously it showed up only once right after booting the system. But now a new LOR (according to the LOR page) seems pop up: Trying to mount root from ufs:/dev/ad0s1a lock order reversal: 1st 0xffffff0002a4ad80 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1465 2nd 0xffffff0002b29a48 filedesc structure (filedesc structure) @ /usr/src/sys/kern/kern_descrip.c:2478 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x49 witness_checkorder() at witness_checkorder+0x7ea _sx_xlock() at _sx_xlock+0x44 mountcheckdirs() at mountcheckdirs+0x80 vfs_donmount() at vfs_donmount+0xfbf kernel_mount() at kernel_mount+0xa1 vfs_mountroot_try() at vfs_mountroot_try+0x177 vfs_mountroot() at vfs_mountroot+0x47d start_init() at start_init+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800001ad30, rbp = 0 --- The output of `df' and `mount' looks ok. Ren? From serenity at exscape.org Wed Jul 29 10:32:30 2009 From: serenity at exscape.org (Thomas Backman) Date: Wed Jul 29 10:32:43 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <20090729084723.GD1586@garage.freebsd.pl> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> Message-ID: On Jul 29, 2009, at 10:47, Pawel Jakub Dawidek wrote: > On Tue, Jul 28, 2009 at 12:50:26PM +0300, Andriy Gapon wrote: >> on 27/07/2009 22:58 O. Hartmann said the following: >>> Juergen Unger wrote: >> [snip] >>>>> _sx_xlock(3c,0,874aa28d,70f,8ae9a9f8,...) at _sx_xlock+0x43 >>>>> dmu_buf_update_user(0,8ae9a9f8,0,0,0,...) at dmu_buf_update_user >>>>> +0x35 >>>>> zfs_znode_dmu_fini(8ae9a9f8,874b312d,1114,110b,879ab000,...) at >>>>> zfs_znode_dmu_f3 >>>>> zfs_freebsd_reclaim(fcd29c3c,1,0,8ec63754,fcd29c60,...) at >>>>> zfs_freebsd_reclaim+0 >>>>> VOP_RECLAIM_APV(874b65a0,fcd29c3c,0,0,8ec637c8,...) at >>>>> VOP_RECLAIM_APV+0xa5 >>>>> vgonel(8ec637c8,0,80c77037,386,0,...) at vgonel+0x1a4 >>>>> vnlru_free(80f2a0f0,0,80c77037,300,3e8,...) at vnlru_free+0x2d5 >>>>> vnlru_proc(0,fcd29d38,80c652bc,33e,871932a8,...) at vnlru_proc >>>>> +0x80 >>>>> fork_exit(8090d960,0,fcd29d38) at fork_exit+0xb8 >>>>> fork_trampoline() at fork_trampoline+0x8 >> [snip] >>> >>> I see a similar problem on two SMP boxes (is your SMP?), but in my >>> case, >>> it seems not to be ZFS related although I also use ZFS as /home >>> filesystem >> >> In this case this does seem to be caused by ZFS. >>> From the backtrace we see that _sx_xlock() is called on bogus >>> struct sx pointer >> (0x3c) and this is caused by dmu_buf_update_user() called with NULL >> first argument >> (dmu_buf_t). Which means that znode_t z_dbuf was NULL - this could >> have been >> caught by ASSERT in zfs_znode_dmu_fini if it were enabled. >> >> If you have the crash dump, then it would be interesting to examine >> znode_t >> structure ('zp' argument) in zfs_znode_dmu_fini. >> >> P.S. I see that zfs_inactive checks for z_dbuf being NULL and there >> is the >> following comment: >> /* >> * The fs has been unmounted, or we did a >> * suspend/resume and this file no longer exists. >> */ >> Maybe zfs_freebsd_reclaim should do the same? > > Yes, you might be right. > > Could you guys, who can reproduce it, try this patch: > > http://people.freebsd.org/~pjd/patches/zfs_vnops.c.2.patch OFF TOPIC: Due to similarities in the backtrace between this and a panic I've been seeing on exporting after zfs recv (see http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009105.html and also http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009174.html for a panics-every-time script) I tried this patch. Unfortunately, I still get the same panic (from vgonel() and up, it's the same, except for my typo in the linked email.) Just thought I should point it out. Except for temporary storage problems when moving my data to ZFS, this panic is the last hurdle in not using FreeBSD for me. :/ Regards, Thomas From avg at icyb.net.ua Wed Jul 29 11:21:31 2009 From: avg at icyb.net.ua (Andriy Gapon) Date: Wed Jul 29 11:21:44 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> Message-ID: <4A7030B6.8010205@icyb.net.ua> on 29/07/2009 13:32 Thomas Backman said the following: > OFF TOPIC: > Due to similarities in the backtrace between this and a panic I've been > seeing on exporting after zfs recv (see > http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009105.html and > also > http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009174.html for > a panics-every-time script) I tried this patch. Unfortunately, I still > get the same panic (from vgonel() and up, it's the same, except for my > typo in the linked email.) Your panics are superficially similar but seem to be different. But it is hard to tell as function argument values are not available in your backtraces for the interesting calls. One difference that I see is that your panics happen one level below _sx_xlock, in sx_xlock_hard and sx argument value appears to be far from NULL (0xffffff0043557d50) - in the panic that started this thread it was near NULL. Another difference is that you panics do not involve zfs_znode_dmu_fini and mu_buf_update_user, in your case sx_xlock is called directly from zfs_freebsd_reclaim. So it must a problem with a different lock. -- Andriy Gapon From avg at icyb.net.ua Wed Jul 29 11:42:21 2009 From: avg at icyb.net.ua (Andriy Gapon) Date: Wed Jul 29 11:42:34 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <4A7030B6.8010205@icyb.net.ua> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> Message-ID: <4A703598.8080809@icyb.net.ua> on 29/07/2009 14:21 Andriy Gapon said the following: > Your panics are superficially similar but seem to be different. > But it is hard to tell as function argument values are not available in your > backtraces for the interesting calls. > One difference that I see is that your panics happen one level below _sx_xlock, in > sx_xlock_hard and sx argument value appears to be far from NULL > (0xffffff0043557d50) - in the panic that started this thread it was near NULL. > Another difference is that you panics do not involve zfs_znode_dmu_fini and > mu_buf_update_user, in your case sx_xlock is called directly from > zfs_freebsd_reclaim. So it must a problem with a different lock. BTW, have you tried to reproduce the problem with INVARIANTS enabled? Do you have crashdumps with debugging symbols? -- Andriy Gapon From avg at icyb.net.ua Wed Jul 29 12:25:24 2009 From: avg at icyb.net.ua (Andriy Gapon) Date: Wed Jul 29 12:25:30 2009 Subject: [ZFS] umount at reboot crashes In-Reply-To: <4A6FE2D6.7050300@jrv.org> References: <4A6FE2D6.7050300@jrv.org> Message-ID: <4A703FAF.1090109@icyb.net.ua> on 29/07/2009 08:49 James R. Van Artsdalen said the following: > FreeBSD bigback.housenet.jrv 8.0-BETA2 FreeBSD 8.0-BETA2 #0 r195757M: > Mon Jul 20 10:27:28 CDT 2009 > james@bigback.housenet.jrv:/usr/obj/usr/src/sys/BIGTEX amd64 > > I have a system that almost always crashes whenever it receives a ZFS > replication package ("zfs recv") that either deleted or renames > filesystems, both operations requiring unmounting. Sometimes it crashes > later in the "zfs recv", sometimes not until I reboot that system. > > The sx_xlock() in frame 10 seems a common theme in these crashes. Could you please try a patch posted by Pawel in a parallel thread with subject "zfs: Fatal trap 12: page fault while in kernel mode"? Please followup with your results and CC Pawel. Thank you! -- Andriy Gapon From serenity at exscape.org Wed Jul 29 12:46:55 2009 From: serenity at exscape.org (Thomas Backman) Date: Wed Jul 29 12:47:07 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <4A7030B6.8010205@icyb.net.ua> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> Message-ID: <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> On Jul 29, 2009, at 13:21, Andriy Gapon wrote: > on 29/07/2009 13:32 Thomas Backman said the following: >> OFF TOPIC: >> Due to similarities in the backtrace between this and a panic I've >> been >> seeing on exporting after zfs recv (see >> http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009105.html >> and >> also >> http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009174.html >> for >> a panics-every-time script) I tried this patch. Unfortunately, I >> still >> get the same panic (from vgonel() and up, it's the same, except for >> my >> typo in the linked email.) > > Your panics are superficially similar but seem to be different. > But it is hard to tell as function argument values are not available > in your > backtraces for the interesting calls. > One difference that I see is that your panics happen one level below > _sx_xlock, in > sx_xlock_hard and sx argument value appears to be far from NULL > (0xffffff0043557d50) - in the panic that started this thread it was > near NULL. > Another difference is that you panics do not involve > zfs_znode_dmu_fini and > mu_buf_update_user, in your case sx_xlock is called directly from > zfs_freebsd_reclaim. So it must a problem with a different lock. > > -- > Andriy Gapon The DDB output from one panic does involve zfs_znode_dmu_fini and dmu_buf_update_user: _sx_xlock() dmu_buf_update_user()+0x47 zfs_znode_dmu_fini() zfs_freebsd_reclaim() VOP_RECLAIM_APV() vgonel() vflush() zfs_umount() dounmount() unmount() syscall() Xfast_syscall() (Sorry if the formatting got screwed up above.) > BTW, have you tried to reproduce the problem with INVARIANTS enabled? > Do you have crashdumps with debugging symbols? I tried again with INVARIANTS, but see no difference in the panic, the DDB bt or the KGDB bt. What does invariants really do? (Not sure how to use it to my advantage here :) Re: debugging symbols; isn't that the default? I do have a .symbols file for all the files in /boot/kernel, but that's all I know to be honest. Regards, Thomas From avg at icyb.net.ua Wed Jul 29 13:03:46 2009 From: avg at icyb.net.ua (Andriy Gapon) Date: Wed Jul 29 13:04:17 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> Message-ID: <4A7048A9.4020507@icyb.net.ua> on 29/07/2009 15:45 Thomas Backman said the following: > On Jul 29, 2009, at 13:21, Andriy Gapon wrote: > >> on 29/07/2009 13:32 Thomas Backman said the following: >>> OFF TOPIC: >>> Due to similarities in the backtrace between this and a panic I've been >>> seeing on exporting after zfs recv (see >>> http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009105.html and >>> >>> also >>> http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009174.html for >>> [snip] > The DDB output from one panic does involve zfs_znode_dmu_fini and > dmu_buf_update_user: > _sx_xlock() > dmu_buf_update_user()+0x47 > zfs_znode_dmu_fini() > zfs_freebsd_reclaim() > VOP_RECLAIM_APV() > vgonel() > vflush() > zfs_umount() > dounmount() > unmount() > syscall() > Xfast_syscall() > (Sorry if the formatting got screwed up above.) Hmm, then you experienced two different kinds of panics. To quote the link you posted earlier: [1] http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009174.html ... #10 0xffffffff8086e007 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:223 #11 0xffffffff805a4989 in _sx_xlock_hard (sx=0xffffff0043557d50, tid=18446742975830720512, opts=Variable "opts" is not available.) at /usr/src/sys/kern/kern_sx.c:575 #12 0xffffffff805a52fe in _sx_xlock (sx=Variable "sx" is not available.) at sx.h:155 #13 0xffffffff80fe2995 in zfs_freebsd_reclaim () from /boot/kernel/zfs.ko ... So now that you said that the patch didn't fix the problem for you, could you please clarify what panic you do see after applying it? >> BTW, have you tried to reproduce the problem with INVARIANTS enabled? >> Do you have crashdumps with debugging symbols? > > I tried again with INVARIANTS, but see no difference in the panic, the > DDB bt or the KGDB bt. What does invariants really do? (Not sure how to > use it to my advantage here :) INVARIANTS enables KASSERTs in various parts of code which can help to catch bugs earlier that may result in cryptic panics afterwards. > Re: debugging symbols; isn't that the default? I do have a .symbols file > for all the files in /boot/kernel, but that's all I know to be honest. Ok, then if you get a crash dump, you are able use kgdb and it will be able to produce line numbers and will allow to examine variables. P.S. sorry if I miss context of your previous reports. -- Andriy Gapon From serenity at exscape.org Wed Jul 29 13:24:33 2009 From: serenity at exscape.org (Thomas Backman) Date: Wed Jul 29 13:24:40 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <4A7048A9.4020507@icyb.net.ua> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> Message-ID: <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> On Jul 29, 2009, at 15:03, Andriy Gapon wrote: > on 29/07/2009 15:45 Thomas Backman said the following: >> On Jul 29, 2009, at 13:21, Andriy Gapon wrote: >> >>> on 29/07/2009 13:32 Thomas Backman said the following: >>>> OFF TOPIC: >>>> Due to similarities in the backtrace between this and a panic >>>> I've been >>>> seeing on exporting after zfs recv (see >>>> http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009105.html >>>> and >>>> >>>> also >>>> http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009174.html >>>> for >>>> > [snip] >> The DDB output from one panic does involve zfs_znode_dmu_fini and >> dmu_buf_update_user: >> _sx_xlock() >> dmu_buf_update_user()+0x47 >> zfs_znode_dmu_fini() >> zfs_freebsd_reclaim() >> VOP_RECLAIM_APV() >> vgonel() >> vflush() >> zfs_umount() >> dounmount() >> unmount() >> syscall() >> Xfast_syscall() >> (Sorry if the formatting got screwed up above.) > > Hmm, then you experienced two different kinds of panics. > To quote the link you posted earlier: > [1] http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009174.html > ... > #10 0xffffffff8086e007 in calltrap () at /usr/src/sys/amd64/amd64/ > exception.S:223 > #11 0xffffffff805a4989 in _sx_xlock_hard (sx=0xffffff0043557d50, > tid=18446742975830720512, opts=Variable "opts" is not available.) at > /usr/src/sys/kern/kern_sx.c:575 > #12 0xffffffff805a52fe in _sx_xlock (sx=Variable "sx" is not > available.) at sx.h:155 > #13 0xffffffff80fe2995 in zfs_freebsd_reclaim () from /boot/kernel/ > zfs.ko > ... > > So now that you said that the patch didn't fix the problem for you, > could you > please clarify what panic you do see after applying it? > [...] > > Ok, then if you get a crash dump, you are able use kgdb and it will > be able to > produce line numbers and will allow to examine variables. > > P.S. sorry if I miss context of your previous reports. > > -- > Andriy Gapon Hmm, you are indeed right, it's not the same panic. The backtrace I got just now with INVARIANTS is the one you quoted above. I still get the "_sx_xlock (sx=Variable "sx" is not available.)" and "_sx_xlock_hard (sx=0xffffff00090d5018, ..., opts=Variable "opts" is not available.)" though. Am I missing some option (I've got GENERIC, minus WITNESS plus DTRACE, now that INVARIANTS is back in place), or does this "just happen"? Here's the "full" backtrace (minus the panic(), trap() etc.): #10 0xffffffff8057dfe7 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:224 #11 0xffffffff80342b99 in _sx_xlock_hard (sx=0xffffff00090d5018, tid=18446742974952890368, opts=Variable "opts" is not available. ) at /usr/src/sys/kern/kern_sx.c:575 #12 0xffffffff8034350e in _sx_xlock (sx=Variable "sx" is not available. ) at sx.h:155 #13 0xffffffff80af7596 in zfs_freebsd_reclaim () from /boot/kernel/ zfs.ko #14 0xffffffff805c5c2a in VOP_RECLAIM_APV (vop=0xffffff00090d5018, a=0xffffff00090d5000) at vnode_if.c:1926 #15 0xffffffff803c839e in vgonel (vp=0xffffff0009252588) at vnode_if.h: 830 #16 0xffffffff803cc958 in vflush (mp=0xffffff0002cd7bc0, rootrefs=0, flags=0, td=0xffffff002cffe000) at /usr/src/sys/kern/vfs_subr.c:2449 #17 0xffffffff80af2038 in zfs_umount () from /boot/kernel/zfs.ko #18 0xffffffff803c55ca in dounmount (mp=0xffffff0002cd7bc0, flags=47020992, td=Variable "td" is not available. ) at /usr/src/sys/kern/vfs_mount.c:1289 #19 0xffffffff803c5df8 in unmount (td=0xffffff002cffe000, uap=0xffffff803e98bbf0) at /usr/src/sys/kern/vfs_mount.c:1174 #20 0xffffffff805980bf in syscall (frame=0xffffff803e98bc80) at /usr/src/sys/amd64/amd64/trap.c:984 #21 0xffffffff8057e2c1 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:373 #22 0x000000080104e9ec in ?? () Previous frame inner to this frame (corrupt stack?) This happens on "zpool export" on the receiving pool (never on the sending pool) when running the script in the posts above. (Which, I realize, few people will have run.) I also get another panic when manually doing zfs unmount on the root FS on the pool, rather than exporting it: http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009209.html Now we're drifting way off topic, though. Sorry. Regards, Thomas From avg at icyb.net.ua Wed Jul 29 13:46:07 2009 From: avg at icyb.net.ua (Andriy Gapon) Date: Wed Jul 29 13:46:19 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> Message-ID: <4A705299.8060504@icyb.net.ua> on 29/07/2009 16:24 Thomas Backman said the following: > Hmm, you are indeed right, it's not the same panic. The backtrace I got > just now with INVARIANTS is the one you quoted above. > I still get the "_sx_xlock (sx=Variable "sx" is not available.)" and > "_sx_xlock_hard (sx=0xffffff00090d5018, ..., opts=Variable "opts" is not > available.)" though. > Am I missing some option (I've got GENERIC, minus WITNESS plus DTRACE, > now that INVARIANTS is back in place), or does this "just happen"? Not sure what this question is about. What option, what "this" :-) > Here's the "full" backtrace (minus the panic(), trap() etc.): > > #10 0xffffffff8057dfe7 in calltrap () > at /usr/src/sys/amd64/amd64/exception.S:224 > #11 0xffffffff80342b99 in _sx_xlock_hard (sx=0xffffff00090d5018, > tid=18446742974952890368, opts=Variable "opts" is not available. > ) at /usr/src/sys/kern/kern_sx.c:575 > #12 0xffffffff8034350e in _sx_xlock (sx=Variable "sx" is not available. > ) at sx.h:155 > #13 0xffffffff80af7596 in zfs_freebsd_reclaim () from /boot/kernel/zfs.ko > #14 0xffffffff805c5c2a in VOP_RECLAIM_APV (vop=0xffffff00090d5018, > a=0xffffff00090d5000) at vnode_if.c:1926 > #15 0xffffffff803c839e in vgonel (vp=0xffffff0009252588) at vnode_if.h:830 > #16 0xffffffff803cc958 in vflush (mp=0xffffff0002cd7bc0, rootrefs=0, > flags=0, > td=0xffffff002cffe000) at /usr/src/sys/kern/vfs_subr.c:2449 > #17 0xffffffff80af2038 in zfs_umount () from /boot/kernel/zfs.ko > #18 0xffffffff803c55ca in dounmount (mp=0xffffff0002cd7bc0, flags=47020992, > td=Variable "td" is not available. > ) at /usr/src/sys/kern/vfs_mount.c:1289 > #19 0xffffffff803c5df8 in unmount (td=0xffffff002cffe000, > uap=0xffffff803e98bbf0) at /usr/src/sys/kern/vfs_mount.c:1174 > #20 0xffffffff805980bf in syscall (frame=0xffffff803e98bc80) > at /usr/src/sys/amd64/amd64/trap.c:984 > #21 0xffffffff8057e2c1 in Xfast_syscall () > at /usr/src/sys/amd64/amd64/exception.S:373 > #22 0x000000080104e9ec in ?? () > Previous frame inner to this frame (corrupt stack?) Looks like your zfs module is built without debugging symbols? Maybe because was it built/rebuilt individually, not as part of kernel build? It would be useful to get line number in frame 13 and examine sx object in frame 11, esp. sx_lock field. -- Andriy Gapon From serenity at exscape.org Wed Jul 29 13:52:45 2009 From: serenity at exscape.org (Thomas Backman) Date: Wed Jul 29 13:52:58 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <4A705299.8060504@icyb.net.ua> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> Message-ID: On Jul 29, 2009, at 15:46, Andriy Gapon wrote: > on 29/07/2009 16:24 Thomas Backman said the following: >> Hmm, you are indeed right, it's not the same panic. The backtrace I >> got >> just now with INVARIANTS is the one you quoted above. >> I still get the "_sx_xlock (sx=Variable "sx" is not available.)" and >> "_sx_xlock_hard (sx=0xffffff00090d5018, ..., opts=Variable "opts" >> is not >> available.)" though. >> Am I missing some option (I've got GENERIC, minus WITNESS plus >> DTRACE, >> now that INVARIANTS is back in place), or does this "just happen"? > > Not sure what this question is about. What option, what "this" :-) > >> Here's the "full" backtrace (minus the panic(), trap() etc.): >> >> #10 0xffffffff8057dfe7 in calltrap () >> at /usr/src/sys/amd64/amd64/exception.S:224 >> #11 0xffffffff80342b99 in _sx_xlock_hard (sx=0xffffff00090d5018, >> tid=18446742974952890368, opts=Variable "opts" is not available. >> ) at /usr/src/sys/kern/kern_sx.c:575 >> #12 0xffffffff8034350e in _sx_xlock (sx=Variable "sx" is not >> available. >> ) at sx.h:155 >> #13 0xffffffff80af7596 in zfs_freebsd_reclaim () from /boot/kernel/ >> zfs.ko >> #14 0xffffffff805c5c2a in VOP_RECLAIM_APV (vop=0xffffff00090d5018, >> a=0xffffff00090d5000) at vnode_if.c:1926 >> #15 0xffffffff803c839e in vgonel (vp=0xffffff0009252588) at >> vnode_if.h:830 >> #16 0xffffffff803cc958 in vflush (mp=0xffffff0002cd7bc0, rootrefs=0, >> flags=0, >> td=0xffffff002cffe000) at /usr/src/sys/kern/vfs_subr.c:2449 >> #17 0xffffffff80af2038 in zfs_umount () from /boot/kernel/zfs.ko >> #18 0xffffffff803c55ca in dounmount (mp=0xffffff0002cd7bc0, >> flags=47020992, >> td=Variable "td" is not available. >> ) at /usr/src/sys/kern/vfs_mount.c:1289 >> #19 0xffffffff803c5df8 in unmount (td=0xffffff002cffe000, >> uap=0xffffff803e98bbf0) at /usr/src/sys/kern/vfs_mount.c:1174 >> #20 0xffffffff805980bf in syscall (frame=0xffffff803e98bc80) >> at /usr/src/sys/amd64/amd64/trap.c:984 >> #21 0xffffffff8057e2c1 in Xfast_syscall () >> at /usr/src/sys/amd64/amd64/exception.S:373 >> #22 0x000000080104e9ec in ?? () >> Previous frame inner to this frame (corrupt stack?) > > Looks like your zfs module is built without debugging symbols? > Maybe because was it built/rebuilt individually, not as part of > kernel build? > > It would be useful to get line number in frame 13 and examine sx > object in frame > 11, esp. sx_lock field. > > -- > Andriy Gapon The "this" (above) was referring to variable values not being available in a vmcore. :) The zfs module appears to be built with symbols, and the symbols appear to be loaded in kgdb: Reading symbols from /boot/kernel/zfs.ko...Reading symbols from / bootdir/boot/kernel/zfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/zfs.ko Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /bootdir/boot/kernel/opensolaris.ko.symbols...done. done. Loaded symbols for /boot/kernel/opensolaris.ko I didn't build the module(s) individually, either; in the previous cases, it was a clean buildworld/buildkernel (even with rm -rf /usr/ obj/* beforehand), and in this case "just" a buildkernel (no manual cleaning, but no -DNO_CLEAN either). Regards, Thomas From avg at icyb.net.ua Wed Jul 29 13:55:50 2009 From: avg at icyb.net.ua (Andriy Gapon) Date: Wed Jul 29 13:56:02 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> Message-ID: <4A7054E1.5060402@icyb.net.ua> on 29/07/2009 16:52 Thomas Backman said the following: >> >> It would be useful to get line number in frame 13 and examine sx >> object in frame >> 11, esp. sx_lock field. >> >> -- >> Andriy Gapon > The "this" (above) was referring to variable values not being available > in a vmcore. :) > > The zfs module appears to be built with symbols, and the symbols appear > to be loaded in kgdb: > > Reading symbols from /boot/kernel/zfs.ko...Reading symbols from > /bootdir/boot/kernel/zfs.ko.symbols...done. done. > Loaded symbols for /boot/kernel/zfs.ko > Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from > /bootdir/boot/kernel/opensolaris.ko.symbols...done. done. > Loaded symbols for /boot/kernel/opensolaris.ko > > I didn't build the module(s) individually, either; in the previous > cases, it was a clean buildworld/buildkernel (even with rm -rf > /usr/obj/* beforehand), and in this case "just" a buildkernel (no manual > cleaning, but no -DNO_CLEAN either). Got it. No idea unfortunately :( Could you still please examine sx in frame 11? -- Andriy Gapon From pjd at FreeBSD.org Wed Jul 29 14:04:22 2009 From: pjd at FreeBSD.org (Pawel Jakub Dawidek) Date: Wed Jul 29 14:04:29 2009 Subject: Re-starting a gjournal provider In-Reply-To: <4A62E0CE.1000508@hexadecagram.org> References: <4A62E0CE.1000508@hexadecagram.org> Message-ID: <20090729140436.GG1586@garage.freebsd.pl> On Sun, Jul 19, 2009 at 03:01:02AM -0600, Anthony Chavez wrote: > Hello freebsd-fs, > > I'm trying to get gjournal working on a "removable" hard disk. I use > the term loosely, because I'm using a very simple eSATA enclosure: an > AMS Venus DS5 [1]. > > If I swap out disks, atacontrol cap ad0 seems sufficient enough to > detect the new drive: the reported device model, serial number, firmware > revision, and CHS values change as one would expect. > > My interpretation of [2] section 5.3 and gjournal(8) is that the > following sequence of commands should ensure me that all write buffers > have been flushed and bring the system to a point where it is safe to > remove a disk. > > sync; sync; sync > gjournal sync > umount /dev/ad0s1.journal > gjournal stop ad0s1.journal You should first unmount and then call 'gjournal sync'. > However, once they are executed, /dev/ad0s1.journal disappears and when > I swap out the disk it doesn't come back. The only way I've found to > bring it back is atacontrol detach ata0; atacontrol attach ata0, which > doesn't seem like a wise thing to do if I have another device on the > same channel. It doesn't come back because something (ATA layer?) doesn't properly remove ad0 provider. When you remove the disk, /dev/ad0 should disappear and reappear once you insert it again. You can still do this trick after you insert the disk again so the GEOM can schedule retaste: # true > /dev/ad0 > My question is, do I need to issue gjournal stop before I swap disks? > And if so, is there any way that I can avoid the atacontrol > detach/attach cycle that would need to take place before any mount is > attempted so that /dev/ad0s1.journal appears (if in the drive inserted > at the time does in fact utilize gjournal; I may want to experiment with > having disks with either gjournal or soft updates)? > > And while I'm on the subject, are the (gjournal) syncs commands > preceeding umount absolutely necessary in the case of removable media? 'gjournal sync' should follow unmount, not the other way around. And its better to do it, but 'gjournal stop' should do the same. -- Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20090729/af00d705/attachment.pgp From jhb at freebsd.org Wed Jul 29 14:08:17 2009 From: jhb at freebsd.org (John Baldwin) Date: Wed Jul 29 14:08:33 2009 Subject: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) In-Reply-To: References: <200907271400.n6RE05Rv056472@freefall.freebsd.org> <200907281038.30277.jhb@freebsd.org> Message-ID: <200907290742.20838.jhb@freebsd.org> On Wednesday 29 July 2009 5:52:24 am Rene Ladan wrote: > 2009/7/28 John Baldwin : > > On Tuesday 28 July 2009 10:03:40 am Rene Ladan wrote: > >> 2009/7/28 John Baldwin : > >> > On Monday 27 July 2009 10:00:05 am Rene Ladan wrote: > >> >> The following reply was made to PR kern/136945; it has been noted by > > GNATS. > >> >> > >> >> From: Rene Ladan > >> >> To: John Baldwin > >> >> Cc: bug-followup@freebsd.org > >> >> Subject: Re: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) > >> >> Date: Mon, 27 Jul 2009 15:51:15 +0200 > >> >> > >> >> ?2009/7/27 John Baldwin : > >> >> ?> I would actually expect this to be the correct order for these two > >> > locks.= > >> >> ? =A0Can > >> >> ?> you capture the output of the 'debug.witness.fullgraph' sysctl to a > > file? > >> >> ?> > >> >> ?Yes, see attachment. ?I'm still running the same 8.0-BETA2. > >> > > >> > Hmm, the attachment was eaten by a grue, can you post the file somewhere? > >> > > >> Yes, see ftp://rene-ladan.nl/pub/freebsd/kern_136945.txt > > > > Ok, it looks like it did encounter a UFS -> filedesc order at some point. ?Can > > you patch sys/kern/subr_witness.c to add a section to the order_lists[] array > > after the 'ZFS locking list' and before the spin locks list that looks like > > this: > > > > ? ? ? ?{ "filedesc structure", &lock_class_sx }, > > ? ? ? ?{ "ufs", &lock_class_lockmgr}, > > ? ? ? ?{ NULL, NULL }, > > > The LOR seems to be gone, previously it showed up only once right > after booting the system. > > But now a new LOR (according to the LOR page) seems pop up: > Trying to mount root from ufs:/dev/ad0s1a > lock order reversal: > 1st 0xffffff0002a4ad80 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1465 > 2nd 0xffffff0002b29a48 filedesc structure (filedesc structure) @ > /usr/src/sys/kern/kern_descrip.c:2478 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > _witness_debugger() at _witness_debugger+0x49 > witness_checkorder() at witness_checkorder+0x7ea > _sx_xlock() at _sx_xlock+0x44 > mountcheckdirs() at mountcheckdirs+0x80 > vfs_donmount() at vfs_donmount+0xfbf > kernel_mount() at kernel_mount+0xa1 > vfs_mountroot_try() at vfs_mountroot_try+0x177 > vfs_mountroot() at vfs_mountroot+0x47d > start_init() at start_init+0x62 > fork_exit() at fork_exit+0x12a > fork_trampoline() at fork_trampoline+0xe > --- trap 0, rip = 0, rsp = 0xffffff800001ad30, rbp = 0 --- > > The output of `df' and `mount' looks ok. Yes, this is the "real" LOR as "filedesc" -> "ufs" in the poll() case should be the normal order. I believe this should fix it. mountcheckdirs() doesn't need the vnodes locked, it just needs the caller to hold references on them so they aren't recycled: --- //depot/projects/smpng/sys/kern/vfs_mount.c#96 +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c @@ -1069,9 +1069,10 @@ vfs_event_signal(NULL, VQ_MOUNT, 0); if (VFS_ROOT(mp, LK_EXCLUSIVE, &newdp)) panic("mount: lost mount"); + VOP_UNLOCK(newdp, 0); + VOP_UNLOCK(vp, 0); mountcheckdirs(vp, newdp); - vput(newdp); - VOP_UNLOCK(vp, 0); + vrele(newdp); if ((mp->mnt_flag & MNT_RDONLY) == 0) error = vfs_allocate_syncvnode(mp); vfs_unbusy(mp); -- John Baldwin From serenity at exscape.org Wed Jul 29 14:11:08 2009 From: serenity at exscape.org (Thomas Backman) Date: Wed Jul 29 14:11:15 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <4A7054E1.5060402@icyb.net.ua> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> Message-ID: <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> On Jul 29, 2009, at 15:55, Andriy Gapon wrote: > on 29/07/2009 16:52 Thomas Backman said the following: >>> >>> It would be useful to get line number in frame 13 and examine sx >>> object in frame >>> 11, esp. sx_lock field. >>> >>> -- >>> Andriy Gapon >> The "this" (above) was referring to variable values not being >> available >> in a vmcore. :) >> >> The zfs module appears to be built with symbols, and the symbols >> appear >> to be loaded in kgdb: >> >> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from >> /bootdir/boot/kernel/zfs.ko.symbols...done. done. >> Loaded symbols for /boot/kernel/zfs.ko >> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols >> from >> /bootdir/boot/kernel/opensolaris.ko.symbols...done. done. >> Loaded symbols for /boot/kernel/opensolaris.ko >> >> I didn't build the module(s) individually, either; in the previous >> cases, it was a clean buildworld/buildkernel (even with rm -rf >> /usr/obj/* beforehand), and in this case "just" a buildkernel (no >> manual >> cleaning, but no -DNO_CLEAN either). > > Got it. No idea unfortunately :( > Could you still please examine sx in frame 11? Maybe... ;) If this isn't right, just tell me how: (kgdb) fr 11 #11 0xffffffff80342b99 in _sx_xlock_hard (sx=0xffffff00090d5018, tid=18446742974952890368, opts=Variable "opts" is not available. ) at /usr/src/sys/kern/kern_sx.c:575 575 owner = (struct thread *)SX_OWNER(x); (kgdb) list 570 * chain lock. If so, drop the sleep queue lock and try 571 * again. 572 */ 573 if (!(x & SX_LOCK_SHARED) && 574 (sx->lock_object.lo_flags & SX_NOADAPTIVE) == 0) { 575 owner = (struct thread *)SX_OWNER(x); 576 if (TD_IS_RUNNING(owner)) { 577 sleepq_release(&sx- >lock_object); 578 continue; 579 } (kgdb) p sx $3 = (struct sx *) 0xffffff00090d5018 (kgdb) x/x sx 0xffffff00090d5018: 0xffffffff80b5634c (kgdb) p *sx $8 = {lock_object = {lo_name = 0xffffffff80b5634c "zp->z_lock", lo_flags = 40894464 [0x2700000, btw], lo_data = 0, lo_witness = 0x0}, sx_lock = 6} ... as you might notice, I'm mostly clueless as to what I'm doing here. :o Hope that helps (a bit), though. Regards, Thomas From avg at icyb.net.ua Wed Jul 29 14:36:06 2009 From: avg at icyb.net.ua (Andriy Gapon) Date: Wed Jul 29 14:36:27 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> Message-ID: <4A705E50.8070307@icyb.net.ua> on 29/07/2009 17:10 Thomas Backman said the following: [snip] > (kgdb) fr 11 [snip] > (kgdb) p *sx > $8 = {lock_object = {lo_name = 0xffffffff80b5634c "zp->z_lock", lo_flags > = 40894464 [0x2700000, btw], lo_data = 0, lo_witness = 0x0}, > sx_lock = 6} > > ... as you might notice, I'm mostly clueless as to what I'm doing here. :o > Hope that helps (a bit), though. Yes, it does and a lot. sx_lock = 6 means that this sx lock is destroyed: #define SX_LOCK_DESTROYED \ (SX_LOCK_SHARED_WAITERS | SX_LOCK_EXCLUSIVE_WAITERS) And lo_name tells that this is zp->z_lock. This lock is destroyed in zfs_znode_cache_destructor. Not enough knowledge for me to proceed further. -- Andriy Gapon From rene at freebsd.org Wed Jul 29 15:20:24 2009 From: rene at freebsd.org (Rene Ladan) Date: Wed Jul 29 15:20:30 2009 Subject: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) In-Reply-To: <200907290742.20838.jhb@freebsd.org> References: <200907271400.n6RE05Rv056472@freefall.freebsd.org> <200907281038.30277.jhb@freebsd.org> <200907290742.20838.jhb@freebsd.org> Message-ID: 2009/7/29 John Baldwin : > On Wednesday 29 July 2009 5:52:24 am Rene Ladan wrote: >> 2009/7/28 John Baldwin : >> > On Tuesday 28 July 2009 10:03:40 am Rene Ladan wrote: >> >> 2009/7/28 John Baldwin : >> >> > On Monday 27 July 2009 10:00:05 am Rene Ladan wrote: >> >> >> The following reply was made to PR kern/136945; it has been noted by >> > GNATS. >> >> >> >> >> >> From: Rene Ladan >> >> >> To: John Baldwin >> >> >> Cc: bug-followup@freebsd.org >> >> >> Subject: Re: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) >> >> >> Date: Mon, 27 Jul 2009 15:51:15 +0200 >> >> >> >> >> >> ?2009/7/27 John Baldwin : >> >> >> ?> I would actually expect this to be the correct order for these two >> >> > locks.= >> >> >> ? =A0Can >> >> >> ?> you capture the output of the 'debug.witness.fullgraph' sysctl to a >> > file? >> >> >> ?> >> >> >> ?Yes, see attachment. ?I'm still running the same 8.0-BETA2. >> >> > >> >> > Hmm, the attachment was eaten by a grue, can you post the file > somewhere? >> >> > >> >> Yes, see ftp://rene-ladan.nl/pub/freebsd/kern_136945.txt >> > >> > Ok, it looks like it did encounter a UFS -> filedesc order at some > point. ?Can >> > you patch sys/kern/subr_witness.c to add a section to the order_lists[] > array >> > after the 'ZFS locking list' and before the spin locks list that looks > like >> > this: >> > >> > ? ? ? ?{ "filedesc structure", &lock_class_sx }, >> > ? ? ? ?{ "ufs", &lock_class_lockmgr}, >> > ? ? ? ?{ NULL, NULL }, >> > >> The LOR seems to be gone, previously it showed up only once right >> after booting the system. >> >> But now a new LOR (according to the LOR page) seems pop up: >> Trying to mount root from ufs:/dev/ad0s1a >> lock order reversal: >> ?1st 0xffffff0002a4ad80 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1465 >> ?2nd 0xffffff0002b29a48 filedesc structure (filedesc structure) @ >> /usr/src/sys/kern/kern_descrip.c:2478 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a >> _witness_debugger() at _witness_debugger+0x49 >> witness_checkorder() at witness_checkorder+0x7ea >> _sx_xlock() at _sx_xlock+0x44 >> mountcheckdirs() at mountcheckdirs+0x80 >> vfs_donmount() at vfs_donmount+0xfbf >> kernel_mount() at kernel_mount+0xa1 >> vfs_mountroot_try() at vfs_mountroot_try+0x177 >> vfs_mountroot() at vfs_mountroot+0x47d >> start_init() at start_init+0x62 >> fork_exit() at fork_exit+0x12a >> fork_trampoline() at fork_trampoline+0xe >> --- trap 0, rip = 0, rsp = 0xffffff800001ad30, rbp = 0 --- >> >> The output of `df' and `mount' looks ok. > > Yes, this is the "real" LOR as "filedesc" -> "ufs" in the poll() case should > be the normal order. ?I believe this should fix it. ?mountcheckdirs() doesn't > need the vnodes locked, it just needs the caller to hold references on them > so they aren't recycled: > > --- //depot/projects/smpng/sys/kern/vfs_mount.c#96 > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c > @@ -1069,9 +1069,10 @@ > ? ? ? ? ? ? ? ?vfs_event_signal(NULL, VQ_MOUNT, 0); > ? ? ? ? ? ? ? ?if (VFS_ROOT(mp, LK_EXCLUSIVE, &newdp)) > ? ? ? ? ? ? ? ? ? ? ? ?panic("mount: lost mount"); > + ? ? ? ? ? ? ? VOP_UNLOCK(newdp, 0); > + ? ? ? ? ? ? ? VOP_UNLOCK(vp, 0); > ? ? ? ? ? ? ? ?mountcheckdirs(vp, newdp); > - ? ? ? ? ? ? ? vput(newdp); > - ? ? ? ? ? ? ? VOP_UNLOCK(vp, 0); > + ? ? ? ? ? ? ? vrele(newdp); > ? ? ? ? ? ? ? ?if ((mp->mnt_flag & MNT_RDONLY) == 0) > ? ? ? ? ? ? ? ? ? ? ? ?error = vfs_allocate_syncvnode(mp); > ? ? ? ? ? ? ? ?vfs_unbusy(mp); > The LOR is still present, but at a different place without the mountcheckdirs() call (not on the LOR page either) : Trying to mount root from ufs:/dev/ad0s1a lock order reversal: 1st 0xffffff0002a4ad80 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2083 2nd 0xffffff000233f048 filedesc structure (filedesc structure) @ /usr/src/sys/kern/vfs_mount.c:1485 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x49 witness_checkorder() at witness_checkorder+0x7ea _sx_xlock() at _sx_xlock+0x44 set_rootvnode() at set_rootvnode+0x57 vfs_mountroot_try() at vfs_mountroot_try+0x371 vfs_mountroot() at vfs_mountroot+0x47d start_init() at start_init+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800001ad30, rbp = 0 --- Ren? From jhb at freebsd.org Wed Jul 29 15:35:24 2009 From: jhb at freebsd.org (John Baldwin) Date: Wed Jul 29 15:35:31 2009 Subject: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) In-Reply-To: References: <200907271400.n6RE05Rv056472@freefall.freebsd.org> <200907290742.20838.jhb@freebsd.org> Message-ID: <200907291135.17569.jhb@freebsd.org> On Wednesday 29 July 2009 11:20:21 am Rene Ladan wrote: > 2009/7/29 John Baldwin : > > On Wednesday 29 July 2009 5:52:24 am Rene Ladan wrote: > >> 2009/7/28 John Baldwin : > >> > On Tuesday 28 July 2009 10:03:40 am Rene Ladan wrote: > >> >> 2009/7/28 John Baldwin : > >> >> > On Monday 27 July 2009 10:00:05 am Rene Ladan wrote: > >> >> >> The following reply was made to PR kern/136945; it has been noted by > >> > GNATS. > >> >> >> > >> >> >> From: Rene Ladan > >> >> >> To: John Baldwin > >> >> >> Cc: bug-followup@freebsd.org > >> >> >> Subject: Re: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) > >> >> >> Date: Mon, 27 Jul 2009 15:51:15 +0200 > >> >> >> > >> >> >> ?2009/7/27 John Baldwin : > >> >> >> ?> I would actually expect this to be the correct order for these two > >> >> > locks.= > >> >> >> ? =A0Can > >> >> >> ?> you capture the output of the 'debug.witness.fullgraph' sysctl to a > >> > file? > >> >> >> ?> > >> >> >> ?Yes, see attachment. ?I'm still running the same 8.0-BETA2. > >> >> > > >> >> > Hmm, the attachment was eaten by a grue, can you post the file > > somewhere? > >> >> > > >> >> Yes, see ftp://rene-ladan.nl/pub/freebsd/kern_136945.txt > >> > > >> > Ok, it looks like it did encounter a UFS -> filedesc order at some > > point. ?Can > >> > you patch sys/kern/subr_witness.c to add a section to the order_lists[] > > array > >> > after the 'ZFS locking list' and before the spin locks list that looks > > like > >> > this: > >> > > >> > ? ? ? ?{ "filedesc structure", &lock_class_sx }, > >> > ? ? ? ?{ "ufs", &lock_class_lockmgr}, > >> > ? ? ? ?{ NULL, NULL }, > >> > > >> The LOR seems to be gone, previously it showed up only once right > >> after booting the system. > >> > >> But now a new LOR (according to the LOR page) seems pop up: > >> Trying to mount root from ufs:/dev/ad0s1a > >> lock order reversal: > >> ?1st 0xffffff0002a4ad80 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1465 > >> ?2nd 0xffffff0002b29a48 filedesc structure (filedesc structure) @ > >> /usr/src/sys/kern/kern_descrip.c:2478 > >> KDB: stack backtrace: > >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > >> _witness_debugger() at _witness_debugger+0x49 > >> witness_checkorder() at witness_checkorder+0x7ea > >> _sx_xlock() at _sx_xlock+0x44 > >> mountcheckdirs() at mountcheckdirs+0x80 > >> vfs_donmount() at vfs_donmount+0xfbf > >> kernel_mount() at kernel_mount+0xa1 > >> vfs_mountroot_try() at vfs_mountroot_try+0x177 > >> vfs_mountroot() at vfs_mountroot+0x47d > >> start_init() at start_init+0x62 > >> fork_exit() at fork_exit+0x12a > >> fork_trampoline() at fork_trampoline+0xe > >> --- trap 0, rip = 0, rsp = 0xffffff800001ad30, rbp = 0 --- > >> > >> The output of `df' and `mount' looks ok. > > > > Yes, this is the "real" LOR as "filedesc" -> "ufs" in the poll() case should > > be the normal order. ?I believe this should fix it. ?mountcheckdirs() doesn't > > need the vnodes locked, it just needs the caller to hold references on them > > so they aren't recycled: > > > > --- //depot/projects/smpng/sys/kern/vfs_mount.c#96 > > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c > > @@ -1069,9 +1069,10 @@ > > ? ? ? ? ? ? ? ?vfs_event_signal(NULL, VQ_MOUNT, 0); > > ? ? ? ? ? ? ? ?if (VFS_ROOT(mp, LK_EXCLUSIVE, &newdp)) > > ? ? ? ? ? ? ? ? ? ? ? ?panic("mount: lost mount"); > > + ? ? ? ? ? ? ? VOP_UNLOCK(newdp, 0); > > + ? ? ? ? ? ? ? VOP_UNLOCK(vp, 0); > > ? ? ? ? ? ? ? ?mountcheckdirs(vp, newdp); > > - ? ? ? ? ? ? ? vput(newdp); > > - ? ? ? ? ? ? ? VOP_UNLOCK(vp, 0); > > + ? ? ? ? ? ? ? vrele(newdp); > > ? ? ? ? ? ? ? ?if ((mp->mnt_flag & MNT_RDONLY) == 0) > > ? ? ? ? ? ? ? ? ? ? ? ?error = vfs_allocate_syncvnode(mp); > > ? ? ? ? ? ? ? ?vfs_unbusy(mp); > > > The LOR is still present, but at a different place without the > mountcheckdirs() call (not on the LOR page either) : Ok, try this patch as well: --- //depot/projects/smpng/sys/kern/vfs_mount.c#97 +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c @@ -1481,6 +1481,8 @@ if (VFS_ROOT(TAILQ_FIRST(&mountlist), LK_EXCLUSIVE, &rootvnode)) panic("Cannot find root vnode"); + VOP_UNLOCK(rootvnode, 0); + p = curthread->td_proc; FILEDESC_XLOCK(p->p_fd); @@ -1496,8 +1498,6 @@ FILEDESC_XUNLOCK(p->p_fd); - VOP_UNLOCK(rootvnode, 0); - EVENTHANDLER_INVOKE(mountroot); } -- John Baldwin From serenity at exscape.org Wed Jul 29 16:11:23 2009 From: serenity at exscape.org (Thomas Backman) Date: Wed Jul 29 16:11:35 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <4A70728C.7020004@freebsd.org> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> Message-ID: <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> On Jul 29, 2009, at 18:02, Andriy Gapon wrote: > on 29/07/2009 17:36 Andriy Gapon said the following: >> on 29/07/2009 17:10 Thomas Backman said the following: >> [snip] >>> (kgdb) fr 11 >> [snip] >>> (kgdb) p *sx >>> $8 = {lock_object = {lo_name = 0xffffffff80b5634c "zp->z_lock", >>> lo_flags >>> = 40894464 [0x2700000, btw], lo_data = 0, lo_witness = 0x0}, >>> sx_lock = 6} >>> >>> ... as you might notice, I'm mostly clueless as to what I'm doing >>> here. :o >>> Hope that helps (a bit), though. >> >> Yes, it does and a lot. >> sx_lock = 6 means that this sx lock is destroyed: >> #define >> SX_LOCK_DESTROYED \ >> (SX_LOCK_SHARED_WAITERS | SX_LOCK_EXCLUSIVE_WAITERS) >> >> And lo_name tells that this is zp->z_lock. >> This lock is destroyed in zfs_znode_cache_destructor. >> Not enough knowledge for me to proceed further. > > So I guess that this is a case when zfs_znode_delete() was called on > znode that > was still referenced from some vnode. When the vnode gets reclaimed > we get this > problem. > Could you please examine vp in frame 15 or 16? > > -- > Andriy Gapon Sure. Lots of info in that one: (kgdb) fr 15 #15 0xffffffff803c839e in vgonel (vp=0xffffff0009252588) at vnode_if.h: 830 830 in vnode_if.h (kgdb) p *vp $3 = {v_type = VDIR, v_tag = 0xffffffff80b56347 "zfs", v_op = 0xffffffff80b5af00, v_data = 0xffffff00090d5000, v_mount = 0xffffff0002cd7bc0, v_nmntvnodes = {tqe_next = 0xffffff00090f5000, tqe_prev = 0xffffff0009252960}, v_un = {vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0, vu_fifoinfo = 0x0, vu_yield = 0}, v_hashlist = { le_next = 0x0, le_prev = 0x0}, v_hash = 0, v_cache_src = {lh_first = 0x0}, v_cache_dst = {tqh_first = 0x0, tqh_last = 0xffffff00092525e8}, v_cache_dd = 0x0, v_cstart = 0, v_lasta = 0, v_lastw = 0, v_clen = 0, v_lock = { lock_object = {lo_name = 0xffffffff80b56347 "zfs", lo_flags = 91947008, lo_data = 0, lo_witness = 0x0}, lk_lock = 18446742974952890368, lk_timo = 51, lk_pri = 80}, v_interlock = {lock_object = { lo_name = 0xffffffff806126d9 "vnode interlock", lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 4}, v_vnlock = 0xffffff0009252620, v_holdcnt = 1, v_usecount = 0, v_iflag = 128, v_vflag = 0, v_writecount = 0, v_freelist = {tqe_next = 0xffffff00090c3760, tqe_prev = 0xffffff002c0bfc18}, v_bufobj = { bo_mtx = {lock_object = {lo_name = 0xffffffff806126e9 "bufobj interlock", lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 4}, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0xffffff00092526c0}, bv_root = 0x0, bv_cnt = 0}, bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xffffff00092526e0}, bv_root = 0x0, bv_cnt = 0}, bo_numoutput = 0, bo_flag = 0, bo_ops = 0xffffffff8079afa0, bo_bsize = 131072, bo_object = 0x0, bo_synclist = {le_next = 0x0, le_prev = 0x0}, bo_private = 0xffffff0009252588, __bo_vnode = 0xffffff0009252588}, v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0} Regards, Thomas From avg at freebsd.org Wed Jul 29 16:13:09 2009 From: avg at freebsd.org (Andriy Gapon) Date: Wed Jul 29 16:13:15 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <4A705E50.8070307@icyb.net.ua> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> Message-ID: <4A70728C.7020004@freebsd.org> on 29/07/2009 17:36 Andriy Gapon said the following: > on 29/07/2009 17:10 Thomas Backman said the following: > [snip] >> (kgdb) fr 11 > [snip] >> (kgdb) p *sx >> $8 = {lock_object = {lo_name = 0xffffffff80b5634c "zp->z_lock", lo_flags >> = 40894464 [0x2700000, btw], lo_data = 0, lo_witness = 0x0}, >> sx_lock = 6} >> >> ... as you might notice, I'm mostly clueless as to what I'm doing here. :o >> Hope that helps (a bit), though. > > Yes, it does and a lot. > sx_lock = 6 means that this sx lock is destroyed: > #define SX_LOCK_DESTROYED \ > (SX_LOCK_SHARED_WAITERS | SX_LOCK_EXCLUSIVE_WAITERS) > > And lo_name tells that this is zp->z_lock. > This lock is destroyed in zfs_znode_cache_destructor. > Not enough knowledge for me to proceed further. So I guess that this is a case when zfs_znode_delete() was called on znode that was still referenced from some vnode. When the vnode gets reclaimed we get this problem. Could you please examine vp in frame 15 or 16? -- Andriy Gapon From serenity at exscape.org Wed Jul 29 17:14:10 2009 From: serenity at exscape.org (Thomas Backman) Date: Wed Jul 29 17:14:17 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <4A705E50.8070307@icyb.net.ua> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> Message-ID: <6C8097A7-1383-42C0-9A87-34C5065CA453@exscape.org> On Jul 29, 2009, at 16:36, Andriy Gapon wrote: > on 29/07/2009 17:10 Thomas Backman said the following: > [snip] >> (kgdb) fr 11 > [snip] >> (kgdb) p *sx >> $8 = {lock_object = {lo_name = 0xffffffff80b5634c "zp->z_lock", >> lo_flags >> = 40894464 [0x2700000, btw], lo_data = 0, lo_witness = 0x0}, >> sx_lock = 6} >> >> ... as you might notice, I'm mostly clueless as to what I'm doing >> here. :o >> Hope that helps (a bit), though. > > Yes, it does and a lot. > sx_lock = 6 means that this sx lock is destroyed: > #define > SX_LOCK_DESTROYED \ > (SX_LOCK_SHARED_WAITERS | SX_LOCK_EXCLUSIVE_WAITERS) > > And lo_name tells that this is zp->z_lock. > This lock is destroyed in zfs_znode_cache_destructor. > Not enough knowledge for me to proceed further. Also, FWIW: Without "options SMP", "zpool" simply goes into an uninterruptible sleep (state D+) on export. kill -9 has no effect, and the backup process just hangs. The rest of the system works great, but... yeah. (The block that causes the panic is wrapped by #ifdef ADAPTIVE_SX, which isn't defined without smp AFAIK.) Regards, Thomas From avg at freebsd.org Wed Jul 29 17:18:18 2009 From: avg at freebsd.org (Andriy Gapon) Date: Wed Jul 29 17:18:35 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> Message-ID: <4A708455.5070304@freebsd.org> Thanks a lot again! Could you please try the following change? In sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c, in function zfs_inactive() insert the following line: vrecycle(vp, curthread); before the following line: zfs_znode_free(zp); This is in "if (zp->z_dbuf == NULL)" branch. I hope that this should work in concert with the patch that Pawel has posted. P.S. Also Pawel has told me that adding 'CFLAGS+=-DDEBUG=1' to sys/modules/zfs/Makefile should enable additional debugging checks (ASSERTs) in ZFS code. -- Andriy Gapon From serenity at exscape.org Wed Jul 29 18:06:18 2009 From: serenity at exscape.org (Thomas Backman) Date: Wed Jul 29 18:06:24 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <4A708455.5070304@freebsd.org> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> <4A708455.5070304@freebsd.org> Message-ID: <86983A55-E5C4-4C04-A4C7-0AE9A9EE37A3@exscape.org> On Jul 29, 2009, at 19:18, Andriy Gapon wrote: > > Thanks a lot again! > > Could you please try the following change? > In sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c, in > function > zfs_inactive() insert the following line: > vrecycle(vp, curthread); > before the following line: > zfs_znode_free(zp); > > This is in "if (zp->z_dbuf == NULL)" branch. > > I hope that this should work in concert with the patch that Pawel > has posted. > > P.S. > Also Pawel has told me that adding 'CFLAGS+=-DDEBUG=1' to sys/ > modules/zfs/Makefile > should enable additional debugging checks (ASSERTs) in ZFS code. > > -- > Andriy Gapon Thanks for your work :) However, bad news: it didn't help. It *might* have gotten us further, though, because the DDB backtrace now looks like this: _sx_xlock_hard() _sx_xlock() zfs_znode_free() zfs_freebsd_inactive() VOP_INACTIVE_APV() vinactive() vput() dounmount() unmount() syscall() XFast_syscall() KGDB: Unread portion of the kernel message buffer: kernel trap 9 with interrupts disabled Fatal trap 9: general protection fault while in kernel mode cpuid = 0; apic id = 00 instruction pointer = 0x20:0xffffffff80342b99 stack pointer = 0x28:0xffffff803e9b7910 frame pointer = 0x28:0xffffff803e9b7970 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = resume, IOPL = 0 current process = 1398 (zpool) panic: from debugger cpuid = 0 KDB: stack backtrace: Uptime: 1m28s Physical memory: 2030 MB Dumping 1405 MB: ... Reading symbols: ... #9 0xffffffff805986aa in trap (frame=0xffffff803e9b7860) at /usr/src/ sys/amd64/amd64/trap.c:639 #10 0xffffffff8057dfe7 in calltrap () at /usr/src/sys/amd64/amd64/ exception.S:224 #11 0xffffffff80342b99 in _sx_xlock_hard (sx=0xffffff0071019181, tid=18446742976093954048, opts=Variable "opts" is not available. ) at /usr/src/sys/kern/kern_sx.c:575 #12 0xffffffff8034350e in _sx_xlock (sx=Variable "sx" is not available. ) at sx.h:155 #13 0xffffffff80ad6be7 in zfs_znode_free () from /boot/kernel/zfs.ko #14 0xffffffff80b5af20 in ?? () #15 0xffffff803e9b79f0 in ?? () #16 0xffffff0071032000 in ?? () #17 0xffffff803e9b79c0 in ?? () #18 0xffffffff80af719a in zfs_freebsd_inactive () from /boot/kernel/ zfs.ko #19 0xffffffff805c5b5a in VOP_INACTIVE_APV (vop=0xffffff0071101a48, a=0xffffff0071019181) at vnode_if.c:1863 #20 0xffffffff803c6aaa in vinactive (vp=0xffffff0071290938, td=0xffffff0071019001) at vnode_if.h:807 #21 0xffffffff803cbf26 in vput (vp=0xffffff0071290938) at /usr/src/sys/ kern/vfs_subr.c:2257 #22 0xffffffff803c57ef in dounmount (mp=0xffffff0002b9e8d0, flags=0, td=Variable "td" is not available. ) at /usr/src/sys/kern/vfs_mount.c:1333 #23 0xffffffff803c5df8 in unmount (td=0xffffff0071032000, uap=0xffffff803e9b7bf0) at /usr/src/sys/kern/vfs_mount.c:1174 #24 0xffffffff805980bf in syscall (frame=0xffffff803e9b7c80) at /usr/ src/sys/amd64/amd64/trap.c:984 #25 0xffffffff8057e2c1 in Xfast_syscall () at /usr/src/sys/amd64/amd64/ exception.S:373 (kgdb) fr 22 #22 0xffffffff803c57ef in dounmount (mp=0xffffff0002b9e8d0, flags=0, td=Variable "td" is not available. ) at /usr/src/sys/kern/vfs_mount.c:1333 1333 vput(coveredvp); (kgdb) p *mp $1 = {mnt_mtx = {lock_object = {lo_name = 0xffffffff80611acd "struct mount mtx", lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 4}, mnt_gen = 2, mnt_list = {tqe_next = 0x0, tqe_prev = 0xffffff0002c71be8}, mnt_op = 0xffffffff80b5ae80, mnt_vfc = 0xffffffff80b5ae20, mnt_vnodecovered = 0xffffff0071290938, mnt_syncer = 0x0, mnt_ref = 0, mnt_nvnodelist = {tqh_first = 0x0, tqh_last = 0xffffff0002b9e930}, mnt_nvnodelistsize = 0, mnt_writeopcount = 0, mnt_kern_flag = 1627390088, mnt_flag = 4096, mnt_xflag = 0, mnt_noasync = 0, mnt_opt = 0xffffff0002f666f0, mnt_optnew = 0x0, mnt_maxsymlinklen = 0, mnt_stat = {f_version = 537068824, f_type = 4, f_flags = 4096, f_bsize = 131072, f_iosize = 131072, f_blocks = 486, f_bfree = 328, f_bavail = 328, f_files = 334, f_ffree = 328, f_syncwrites = 0, f_asyncwrites = 0, f_syncreads = 0, f_asyncreads = 0, f_spare = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, f_namemax = 255, f_owner = 0, f_fsid = {val = {1968303680, -171280380}}, f_charspare = '\0' , f_fstypename = "zfs", '\0' , f_mntfromname = "crashtestslave/test_orig", '\0' , f_mntonname = "/crashtestslave/crashtestslave/test_orig", '\0' }, mnt_cred = 0xffffff0002f0b700, mnt_data = 0xffffff002489e000, mnt_time = 0, mnt_iosize_max = 65536, mnt_export = 0x0, mnt_label = 0x0, mnt_hashseed = 1597825977, mnt_lockref = 0, mnt_secondary_writes = 0, mnt_secondary_accwrites = 0, mnt_susp_owner = 0x0, mnt_gjprovider = 0x0, mnt_explock = {lock_object = { lo_name = 0xffffffff80611ade "explock", lo_flags = 91422720, lo_data = 0, lo_witness = 0x0}, lk_lock = 1, lk_timo = 0, lk_pri = 80}} Worth noting above: it's NOT the "pool root FS" that's being unmounted here. The panic can also be triggered on "zfs unmount crashtestslave/ test_orig" (i.e. not the root FS which was the only that panicked with zfs unmount, as opposed to zpool export, before). (kgdb) fr 21 #21 0xffffffff803cbf26 in vput (vp=0xffffff0071290938) at /usr/src/sys/ kern/vfs_subr.c:2257 2257 vinactive(vp, td); (kgdb) p *vp $3 = {v_type = VBAD, v_tag = 0xffffffff80600ff6 "none", v_op = 0xffffffff80779700, v_data = 0x0, v_mount = 0x0, v_nmntvnodes = {tqe_next = 0x0, tqe_prev = 0xffffff0071290b38}, v_un = {vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0, vu_fifoinfo = 0x0, vu_yield = 0}, v_hashlist = {le_next = 0x0, le_prev = 0x0}, v_hash = 0, v_cache_src = {lh_first = 0x0}, v_cache_dst = {tqh_first = 0x0, tqh_last = 0xffffff0071290998}, v_cache_dd = 0x0, v_cstart = 0, v_lasta = 0, v_lastw = 0, v_clen = 0, v_lock = {lock_object = {lo_name = 0xffffffff80b56367 "zfs", lo_flags = 91947008, lo_data = 0, lo_witness = 0x0}, lk_lock = 18446742976093954048, lk_timo = 51, lk_pri = 80}, v_interlock = {lock_object = {lo_name = 0xffffffff806126d9 "vnode interlock", lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 4}, v_vnlock = 0xffffff00712909d0, v_holdcnt = 1, v_usecount = 0, v_iflag = 2176, v_vflag = 0, v_writecount = 0, v_freelist = {tqe_next = 0x0, tqe_prev = 0xffffff0002cecc18}, v_bufobj = {bo_mtx = {lock_object = {lo_name = 0xffffffff806126e9 "bufobj interlock", lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 4}, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0xffffff0071290a70}, bv_root = 0x0, bv_cnt = 0}, bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xffffff0071290a90}, bv_root = 0x0, bv_cnt = 0}, bo_numoutput = 0, bo_flag = 0, bo_ops = 0xffffffff8079afa0, bo_bsize = 131072, bo_object = 0x0, bo_synclist = {le_next = 0x0, le_prev = 0x0}, bo_private = 0xffffff0071290938, __bo_vnode = 0xffffff0071290938}, v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0} (kgdb) fr 11 #11 0xffffffff80342b99 in _sx_xlock_hard (sx=0xffffff0071019181, tid=18446742976093954048, opts=Variable "opts" is not available. ) at /usr/src/sys/kern/kern_sx.c:575 575 owner = (struct thread *)SX_OWNER(x); (kgdb) p *sx $4 = {lock_object = {lo_name = 0xffffffff80b571
, lo_flags = 160000, lo_data = 0, lo_witness = 0x100000000000000}, sx_lock = 16717361816799281152} Regards, Thomas From serenity at exscape.org Wed Jul 29 20:15:15 2009 From: serenity at exscape.org (Thomas Backman) Date: Wed Jul 29 20:15:27 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <4A708455.5070304@freebsd.org> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> <4A708455.5070304@freebsd.org> Message-ID: <16B40A2B-A1B5-4528-8721-6D352E7D5419@exscape.org> On Jul 29, 2009, at 19:18, Andriy Gapon wrote: > > Thanks a lot again! > > Could you please try the following change? > In sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c, in > function > zfs_inactive() insert the following line: > vrecycle(vp, curthread); > before the following line: > zfs_znode_free(zp); > > This is in "if (zp->z_dbuf == NULL)" branch. > > I hope that this should work in concert with the patch that Pawel > has posted. > > P.S. > Also Pawel has told me that adding 'CFLAGS+=-DDEBUG=1' to sys/ > modules/zfs/Makefile > should enable additional debugging checks (ASSERTs) in ZFS code. > > -- > Andriy Gapon Better backtraces: Without your vrecycle() addition, and with the -DDEBUG=1 one (note to self: core.txt.32): Unread portion of the kernel message buffer: panic: solaris assert: ((zp)->z_vnode) == ((void *)0), file: /usr/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ zfs_znode.c, line: 1043 cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a panic() at panic+0x182 zfs_znode_free() at zfs_znode_free+0xef zfs_freebsd_inactive() at zfs_freebsd_inactive+0x1a VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x4a vinactive() at vinactive+0x6a vput() at vput+0x1c6 dounmount() at dounmount+0x4af unmount() at unmount+0x3c8 syscall() at syscall+0x28f Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (22, FreeBSD ELF64, unmount), rip = 0x80104e9ec, rsp = 0x7fffffffaa98, rbp = 0x801223300 --- KDB: enter: panic panic: from debugger cpuid = 0 Uptime: 1m5s Physical memory: 2034 MB Dumping 1405 MB: ... #11 0xffffffff8033a9cb in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:558 #12 0xffffffff80aed21f in zfs_znode_free () from /boot/kernel/zfs.ko #13 0xffffffff80b10a9a in zfs_freebsd_inactive () from /boot/kernel/ zfs.ko #14 0xffffffff805c5b5a in VOP_INACTIVE_APV (vop=0xffffffff80b88220, a=0xffffff00401b9a48) at vnode_if.c:1863 #15 0xffffffff803c6aaa in vinactive (vp=0xffffff004038c3b0, td=0xffffff0040031000) at vnode_if.h:807 #16 0xffffffff803cbf26 in vput (vp=0xffffff004038c3b0) at /usr/src/sys/kern/vfs_subr.c:2257 #17 0xffffffff803c57ef in dounmount (mp=0xffffff0001cea8d0, flags=0, td=Variable "td" is not available. ) at /usr/src/sys/kern/vfs_mount.c:1333 #18 0xffffffff803c5df8 in unmount (td=0xffffff0013adfab0, uap=0xffffff803ead0bf0) at /usr/src/sys/kern/vfs_mount.c:1174 #19 0xffffffff805980bf in syscall (frame=0xffffff803ead0c80) at /usr/src/sys/amd64/amd64/trap.c:984 #20 0xffffffff8057e2c1 in Xfast_syscall () at /usr/src/sys/amd64/ amd64/exception.S:373 #21 0x000000080104e9ec in ?? () Previous frame inner to this frame (corrupt stack?) --------------------------- WITH the vrecycle() and -DDEBUG=1: kernel trap 9 with interrupts disabled Fatal trap 9: general protection fault while in kernel mode cpuid = 0; apic id = 00 instruction pointer = 0x20:0xffffffff80342b99 stack pointer = 0x28:0xffffff803eaf8910 frame pointer = 0x28:0xffffff803eaf8970 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = resume, IOPL = 0 current process = 1414 (zpool) panic: from debugger cpuid = 0 KDB: stack backtrace: Uptime: 1m16s Physical memory: 2034 MB Dumping 1407 MB: ... #9 0xffffffff805986aa in trap (frame=0xffffff803eaf8860) at /usr/src/ sys/amd64/amd64/trap.c:639 #10 0xffffffff8057dfe7 in calltrap () at /usr/src/sys/amd64/amd64/ exception.S:224 #11 0xffffffff80342b99 in _sx_xlock_hard (sx=0xffffff0044136251, tid=18446742975340199936, opts=Variable "opts" is not available. ) at /usr/src/sys/kern/kern_sx.c:575 #12 0xffffffff8034350e in _sx_xlock (sx=Variable "sx" is not available. ) at sx.h:155 #13 0xffffffff80aed172 in zfs_znode_free () from /boot/kernel/zfs.ko #14 0xffffffff80b10a8a in zfs_freebsd_inactive () from /boot/kernel/ zfs.ko #15 0xffffffff805c5b5a in VOP_INACTIVE_APV (vop=0xffffff0044136251, a=0xffffff0015b9cd38) at vnode_if.c:1863 #16 0xffffffff803c6aaa in vinactive (vp=0xffffff00443dc588, td=0xffffff0044136001) at vnode_if.h:807 #17 0xffffffff803cbf26 in vput (vp=0xffffff00443dc588) at /usr/src/sys/ kern/vfs_subr.c:2257 #18 0xffffffff803c57ef in dounmount (mp=0xffffff0001cc38d0, flags=0, td=Variable "td" is not available. ) at /usr/src/sys/kern/vfs_mount.c:1333 #19 0xffffffff803c5df8 in unmount (td=0xffffff004415c000, uap=0xffffff803eaf8bf0) at /usr/src/sys/kern/vfs_mount.c:1174 #20 0xffffffff805980bf in syscall (frame=0xffffff803eaf8c80) at /usr/ src/sys/amd64/amd64/trap.c:984 #21 0xffffffff8057e2c1 in Xfast_syscall () at /usr/src/sys/amd64/amd64/ exception.S:373 #22 0x000000080104e9ec in ?? () Previous frame inner to this frame (corrupt stack?) Time to sleep. I only have the kernel.debug for the latter panic, by the way, but at a quick glance they appear to be the same except for the panic line...? Regards, Thomas From pjd at FreeBSD.org Wed Jul 29 21:17:47 2009 From: pjd at FreeBSD.org (Pawel Jakub Dawidek) Date: Wed Jul 29 21:18:00 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <16B40A2B-A1B5-4528-8721-6D352E7D5419@exscape.org> References: <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> <4A708455.5070304@freebsd.org> <16B40A2B-A1B5-4528-8721-6D352E7D5419@exscape.org> Message-ID: <20090729211803.GA2130@garage.freebsd.pl> On Wed, Jul 29, 2009 at 10:15:06PM +0200, Thomas Backman wrote: > On Jul 29, 2009, at 19:18, Andriy Gapon wrote: > > > > >Thanks a lot again! > > > >Could you please try the following change? > >In sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c, in > >function > >zfs_inactive() insert the following line: > > vrecycle(vp, curthread); > >before the following line: > > zfs_znode_free(zp); > > > >This is in "if (zp->z_dbuf == NULL)" branch. > > > >I hope that this should work in concert with the patch that Pawel > >has posted. > > > >P.S. > >Also Pawel has told me that adding 'CFLAGS+=-DDEBUG=1' to sys/ > >modules/zfs/Makefile > >should enable additional debugging checks (ASSERTs) in ZFS code. > > > >-- > >Andriy Gapon > Better backtraces: > > Without your vrecycle() addition, and with the -DDEBUG=1 one (note to > self: core.txt.32): > > Unread portion of the kernel message buffer: > panic: solaris assert: ((zp)->z_vnode) == ((void *)0), file: /usr/src/ > sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ > zfs_znode.c, line: 1043 Modify zfs_inactive() 'zp->z_dbuf == NULL' case to look like this: if (zp->z_dbuf == NULL) { /* * The fs has been unmounted, or we did a * suspend/resume and this file no longer exists. */ VI_LOCK(vp); vp->v_count = 0; /* count arrives as 1 */ vp->v_data = NULL; VI_UNLOCK(vp); rw_exit(&zfsvfs->z_teardown_inactive_lock); ZTOV(zp) = NULL; vrecycle(vp, curthread); zfs_znode_free(zp); return; } -- Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20090729/5c0424ab/attachment.pgp From mdounin at mdounin.ru Thu Jul 30 01:57:49 2009 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu Jul 30 01:58:02 2009 Subject: another zfs panic Message-ID: <20090730013857.GB8794@mdounin.ru> Hello! Here is zfs panic I'm able to reproduce by running an scp from remote machine to zfs volume and 3 parallel untars of ports tree in cycle. Not sure that everything is required, but the above workload triggers panic in several hours. This is on fresh current with GENERIC kernel: panic: sx_xlock() of destroyed sx @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_rlock.c:535 cpuid = 6 KDB: enter: panic [thread pid 36 tid 100071 ] Stopped at kdb_enter+0x3d: movq $0,0x68a040(%rip) db> bt Tracing pid 36 tid 100071 td 0xffffff00040f3720 kdb_enter() at kdb_enter+0x3d panic() at panic+0x17b _sx_xlock() at _sx_xlock+0xfc zfs_range_unlock() at zfs_range_unlock+0x38 zfs_get_data() at zfs_get_data+0xc1 zil_commit() at zil_commit+0x532 zfs_sync() at zfs_sync+0xa6 sync_fsync() at sync_fsync+0x13a sync_vnode() at sync_vnode+0x157 sched_sync() at sched_sync+0x1d1 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80e7ee3d30, rbp = 0 --- Machine is otherwise idle. The only zfs-related tuning applied is compression=gzip-9. Please let me know if you want me to test some patches. Maxim Dounin From sfourman at gmail.com Thu Jul 30 02:38:10 2009 From: sfourman at gmail.com (Sam Fourman Jr.) Date: Thu Jul 30 02:38:16 2009 Subject: another zfs panic In-Reply-To: <20090730013857.GB8794@mdounin.ru> References: <20090730013857.GB8794@mdounin.ru> Message-ID: <11167f520907291913i2718f784hf3d468284383eab1@mail.gmail.com> On Wed, Jul 29, 2009 at 8:38 PM, Maxim Dounin wrote: > Hello! > > Here is zfs panic I'm able to reproduce by running an scp from > remote machine to zfs volume and 3 parallel untars of ports tree > in cycle. ?Not sure that everything is required, but the above > workload triggers panic in several hours. > > This is on fresh current with GENERIC kernel: > > panic: sx_xlock() of destroyed sx @ > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_rlock.c:535 > cpuid = 6 > KDB: enter: panic > [thread pid 36 tid 100071 ] > Stopped at ? ? ?kdb_enter+0x3d: movq ? ?$0,0x68a040(%rip) > db> bt > Tracing pid 36 tid 100071 td 0xffffff00040f3720 > kdb_enter() at kdb_enter+0x3d > panic() at panic+0x17b > _sx_xlock() at _sx_xlock+0xfc > zfs_range_unlock() at zfs_range_unlock+0x38 > zfs_get_data() at zfs_get_data+0xc1 > zil_commit() at zil_commit+0x532 > zfs_sync() at zfs_sync+0xa6 > sync_fsync() at sync_fsync+0x13a > sync_vnode() at sync_vnode+0x157 > sched_sync() at sched_sync+0x1d1 > fork_exit() at fork_exit+0x12a > fork_trampoline() at fork_trampoline+0xe > --- trap 0, rip = 0, rsp = 0xffffff80e7ee3d30, rbp = 0 --- > > Machine is otherwise idle. ?The only zfs-related tuning applied is > compression=gzip-9. > > Please let me know if you want me to test some patches. > > Maxim Dounin > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" what is the output of uname -a what is the contents of /boot/loader.conf Sam Fourman Jr From mdounin at mdounin.ru Thu Jul 30 02:51:49 2009 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu Jul 30 02:52:01 2009 Subject: another zfs panic In-Reply-To: <11167f520907291913i2718f784hf3d468284383eab1@mail.gmail.com> References: <20090730013857.GB8794@mdounin.ru> <11167f520907291913i2718f784hf3d468284383eab1@mail.gmail.com> Message-ID: <20090730025147.GC8794@mdounin.ru> Hello! On Wed, Jul 29, 2009 at 09:13:41PM -0500, Sam Fourman Jr. wrote: > On Wed, Jul 29, 2009 at 8:38 PM, Maxim Dounin wrote: > > Hello! > > > > Here is zfs panic I'm able to reproduce by running an scp from > > remote machine to zfs volume and 3 parallel untars of ports tree > > in cycle. ?Not sure that everything is required, but the above > > workload triggers panic in several hours. > > > > This is on fresh current with GENERIC kernel: > > > > panic: sx_xlock() of destroyed sx @ > > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_rlock.c:535 [...] > what is the output of uname -a > what is the contents of /boot/loader.conf $ uname -a FreeBSD x0040.mgmt.vega.ru 8.0-BETA2 FreeBSD 8.0-BETA2 #0: Wed Jul 29 12:46:06 UTC 2009 root@x0040.mgmt.vega.ru:/usr/obj/usr/src/sys/GENERIC amd64 $ cat /boot/loader.conf beastie_disable="YES" geom_mirror_load="YES" hint.uart.0.flags="0" hint.uart.1.flags="0x10" Maxim Dounin From ludwigp at chip-web.com Thu Jul 30 05:39:02 2009 From: ludwigp at chip-web.com (Ludwig Pummer) Date: Thu Jul 30 05:39:09 2009 Subject: ZFS raidz1 pool unavailable from losing 1 device Message-ID: <4A712290.9030308@chip-web.com> Hello, I found myself with a 4-drive raidz1 pool that was put into the UNAVAIL state ("insufficient replicas") when 3 drives shows ONLINE and 1 showed UNAVAIL. Can anyone suggest how I can get out of this pickle? Here's the full backstory: My system is 7.2-STABLE Jul 27, amd64, 4GB memory, just upgraded from 6.4-STABLE from last year. I just set up a ZFS raidz volume to replace a graid5 volume I had been using. I had it successfully set up using partitions across 4 disks, ad{6,8,10,12}s1e. Then I wanted to expand the raidz volume by merging the space from the adjacent disk partition. I thought I could just fail out the partition device in ZFS, edit the bsdlabel, and re-add the larger partition, ZFS would resilver, repeat until done. That's when I found out that ZFS doesn't let you fail out a device in a raidz volume. No big deal, I thought, I'll just go to single user mode and mess with the partition when ZFS isn't looking. When it comes back up it should notice that one of the device is gone, I can do a 'zfs replace' and continue my plan. Well, after rebooting to single user mode, combining partitions ad12s1d and ad12s1e (removed the d partiton), "zfs volinit", then "zpool status" just hung (Ctrl-C didn't kill it, so I rebooted). I thought this was a bit odd so I thought perhaps ZFS is confused by the ZFS metadata left on ad12s1e, so I blanked it out with "dd". That didn't help. I changed the name of the partition to ad12s1d thinking perhaps that would help. After that, "zfs volinit; zfs mount -a; zpool status" showed my raidz pool UNAVAIL with the message "insufficient replicas", ad{6,8,10}s1e ONLINE, and ad12s1e UNAVAIL "cannot open", and a more detailed message pointing me to http://www.sun.com/msg/ZFS-8000-3C. I tried doing a "zpool replace storage ad12s1e ad12s1d" but it refused, saying my zpool ("storage") was unavailable. Ditto for pretty much every zpool command I tried. "zpool clear" gave me a "permission denied" error. After some more searching of forums/mailing lists, I ran across one that suggested exporting & importing the zpool. I'm afraid that didn't fix my problem. The export worked, but now I cannot import the volume again ("cannot import 'storage': pool may be in use from other system", or with -f, "cannot import 'storage': one or more devices is currently unavailable"). Help! From pjd at FreeBSD.org Thu Jul 30 05:49:44 2009 From: pjd at FreeBSD.org (Pawel Jakub Dawidek) Date: Thu Jul 30 05:49:51 2009 Subject: another zfs panic In-Reply-To: <20090730013857.GB8794@mdounin.ru> References: <20090730013857.GB8794@mdounin.ru> Message-ID: <20090730055001.GB2130@garage.freebsd.pl> On Thu, Jul 30, 2009 at 05:38:57AM +0400, Maxim Dounin wrote: > Hello! > > Here is zfs panic I'm able to reproduce by running an scp from > remote machine to zfs volume and 3 parallel untars of ports tree > in cycle. Not sure that everything is required, but the above > workload triggers panic in several hours. > > This is on fresh current with GENERIC kernel: > > panic: sx_xlock() of destroyed sx @ > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_rlock.c:535 > cpuid = 6 > KDB: enter: panic > [thread pid 36 tid 100071 ] > Stopped at kdb_enter+0x3d: movq $0,0x68a040(%rip) > db> bt > Tracing pid 36 tid 100071 td 0xffffff00040f3720 > kdb_enter() at kdb_enter+0x3d > panic() at panic+0x17b > _sx_xlock() at _sx_xlock+0xfc > zfs_range_unlock() at zfs_range_unlock+0x38 > zfs_get_data() at zfs_get_data+0xc1 > zil_commit() at zil_commit+0x532 > zfs_sync() at zfs_sync+0xa6 > sync_fsync() at sync_fsync+0x13a > sync_vnode() at sync_vnode+0x157 > sched_sync() at sched_sync+0x1d1 > fork_exit() at fork_exit+0x12a > fork_trampoline() at fork_trampoline+0xe > --- trap 0, rip = 0, rsp = 0xffffff80e7ee3d30, rbp = 0 --- > > Machine is otherwise idle. The only zfs-related tuning applied is > compression=gzip-9. > > Please let me know if you want me to test some patches. The kernel syncer tries to sync vnode which has its znode already destroyed. There is one place (that we know of) where vrecycle() is missing. Could you try this patch: http://people.freebsd.org/~pjd/patches/zfs_vnops.c.2.patch -- Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20090730/f12bee63/attachment-0001.pgp From serenity at exscape.org Thu Jul 30 07:05:02 2009 From: serenity at exscape.org (Thomas Backman) Date: Thu Jul 30 07:05:09 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <20090729211803.GA2130@garage.freebsd.pl> References: <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> <4A708455.5070304@freebsd.org> <16B40A2B-A1B5-4528-8721-6D352E7D5419@exscape.org> <20090729211803.GA2130@garage.freebsd.pl> Message-ID: <4FD5D430-9847-4333-AF47-00DE735E0E25@exscape.org> On Jul 29, 2009, at 23:18, Pawel Jakub Dawidek wrote: > On Wed, Jul 29, 2009 at 10:15:06PM +0200, Thomas Backman wrote: >> On Jul 29, 2009, at 19:18, Andriy Gapon wrote: >> >>> >>> Thanks a lot again! >>> >>> Could you please try the following change? >>> In sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c, in >>> function >>> zfs_inactive() insert the following line: >>> vrecycle(vp, curthread); >>> before the following line: >>> zfs_znode_free(zp); >>> >>> This is in "if (zp->z_dbuf == NULL)" branch. >>> >>> I hope that this should work in concert with the patch that Pawel >>> has posted. >>> >>> P.S. >>> Also Pawel has told me that adding 'CFLAGS+=-DDEBUG=1' to sys/ >>> modules/zfs/Makefile >>> should enable additional debugging checks (ASSERTs) in ZFS code. >>> >>> -- >>> Andriy Gapon >> Better backtraces: >> >> Without your vrecycle() addition, and with the -DDEBUG=1 one (note to >> self: core.txt.32): >> >> Unread portion of the kernel message buffer: >> panic: solaris assert: ((zp)->z_vnode) == ((void *)0), file: /usr/ >> src/ >> sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ >> zfs_znode.c, line: 1043 > > Modify zfs_inactive() 'zp->z_dbuf == NULL' case to look like this: > > if (zp->z_dbuf == NULL) { > /* > * The fs has been unmounted, or we did a > * suspend/resume and this file no longer exists. > */ > VI_LOCK(vp); > vp->v_count = 0; /* count arrives as 1 */ > vp->v_data = NULL; > VI_UNLOCK(vp); > rw_exit(&zfsvfs->z_teardown_inactive_lock); > ZTOV(zp) = NULL; > vrecycle(vp, curthread); > zfs_znode_free(zp); > return; > } New code, new panic. :( Same place as before, on exporting. panic: solaris assert: zp != ((void *)0), file: /usr/src/sys/modules/ zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c, line: 4357 cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a panic() at panic+0x182 zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x1f2 VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0x4a vgonel() at vgonel+0x12e vrecycle() at vrecycle+0x7d zfs_inactive() at zfs_inactive+0x1aa zfs_freebsd_inactive() at zfs_freebsd_inactive+0x1a VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x4a vinactive() at vinactive+0x6a vput() at vput+0x1c6 dounmount() at dounmount+0x4af unmount() at unmount+0x3c8 syscall() at syscall+0x28f Xfast_syscall() at Xfast_syscall+0xe1--- syscall (22, FreeBSD ELF64, unmount), rip = 0x80104e9ec, rsp = 0x7fffffffaa98, rbp = 0x801223300 --- KDB: enter: panic [lockedvnods] 0xffffff000bf8f3b0: tag zfs, type VDIR usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VI_DOOMED|VI_DOINGINACT) lock type zfs: EXCL by thread 0xffffff00450b0390 (pid 1400)panic: from debuggercpuid = 0 Uptime: 1m34s Physical memory: 2030 MB Dumping 1407 MB: ... #11 0xffffffff8033a9cb in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:558#12 0xffffffff80b110c2 in zfs_freebsd_reclaim () from /boot/kernel/zfs.ko #13 0xffffffff805c5c2a in VOP_RECLAIM_APV (vop=0x0, a=0xffffff803e9578f0) at vnode_if.c:1926 #14 0xffffffff803c839e in vgonel (vp=0xffffff000bf8f3b0) at vnode_if.h: 830 #15 0xffffffff803ca7ad in vrecycle (vp=0xffffff000bf8f3b0, td=Variable "td" is not available. ) at /usr/src/sys/kern/vfs_subr.c:2504 #16 0xffffffff80b109ea in zfs_inactive () from /boot/kernel/zfs.ko #17 0xffffffff80b88220 in ?? () #18 0xffffff803e9579f0 in ?? () #19 0xffffff00450b0390 in ?? () #20 0x0000000000000000 in ?? () #21 0xffffff803e957a40 in ?? () #22 0xffffff803e9579c0 in ?? () #23 0xffffffff80b10a9a in zfs_freebsd_inactive () from /boot/kernel/ zfs.ko #24 0xffffffff805c5b5a in VOP_INACTIVE_APV (vop=0xffffff000bf8f470, a=0xffffff0045146a48) at vnode_if.c:1863 #25 0xffffffff803c6aaa in vinactive (vp=0xffffff000bf8f3b0, td=0xffffff000bf8f3b0) at vnode_if.h:807 #26 0xffffffff803cbf26 in vput (vp=0xffffff000bf8f3b0) at /usr/src/ sys/kern/vfs_subr.c:2257 #27 0xffffffff803c57ef in dounmount (mp=0xffffff0002d0e8d0, flags=0, td=Variable "td" is not available. ) #28 0xffffffff803c5df8 in unmount (td=0xffffff00450b0390, uap=0xffffff803e957bf0) at /usr/src/sys/kern/vfs_mount.c:1174#29 0xffffffff805980bf in syscall (frame=0xffffff803e957c80) at /usr/src/sys/amd64/amd64/trap.c:984 #30 0xffffffff8057e2c1 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:373#31 0x000000080104e9ec in ?? ()Previous frame inner to this frame (corrupt stack?) BTW, here's my svn diff output (in /usr/src; one irrelevant patch not shown; I used your previous zfs_vnops.2.c patch and then replaced the if block as above): Index: sys/modules/zfs/Makefile =================================================================== --- sys/modules/zfs/Makefile (revision 195910) +++ sys/modules/zfs/Makefile (working copy) @@ -97,3 +97,4 @@ CWARNFLAGS+=-Wno-inline CWARNFLAGS+=-Wno-switch CWARNFLAGS+=-Wno-pointer-arith +CFLAGS+=-DDEBUG=1 Index: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c =================================================================== --- sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c (revision 195910) +++ sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c (working copy) @@ -3709,11 +3709,13 @@ * The fs has been unmounted, or we did a * suspend/resume and this file no longer exists. */ - mutex_enter(&zp->z_lock); VI_LOCK(vp); vp->v_count = 0; /* count arrives as 1 */ - mutex_exit(&zp->z_lock); + vp->v_data = NULL; + VI_UNLOCK(vp); rw_exit(&zfsvfs->z_teardown_inactive_lock); + ZTOV(zp) = NULL; + vrecycle(vp, curthread); zfs_znode_free(zp); return; } @@ -4351,7 +4353,6 @@ { vnode_t *vp = ap->a_vp; znode_t *zp = VTOZ(vp); - zfsvfs_t *zfsvfs; ASSERT(zp != NULL); @@ -4361,13 +4362,18 @@ vnode_destroy_vobject(vp); mutex_enter(&zp->z_lock); - ASSERT(zp->z_phys); + ASSERT(zp->z_phys != NULL); ZTOV(zp) = NULL; - if (!zp->z_unlinked) { + mutex_exit(&zp->z_lock); + + if (zp->z_unlinked) + ; /* Do nothing. */ + else if (zp->z_dbuf == NULL) + zfs_znode_free(zp); + else /* if (!zp->z_unlinked && zp->z_dbuf != NULL) */ { + zfsvfs_t *zfsvfs = zp->z_zfsvfs; int locked; - zfsvfs = zp->z_zfsvfs; - mutex_exit(&zp->z_lock); locked = MUTEX_HELD(ZFS_OBJ_MUTEX(zfsvfs, zp- >z_id)) ? 2 : ZFS_OBJ_HOLD_TRYENTER(zfsvfs, zp->z_id); if (locked == 0) { @@ -4383,8 +4389,6 @@ ZFS_OBJ_HOLD_EXIT(zfsvfs, zp->z_id); zfs_znode_free(zp); } - } else { - mutex_exit(&zp->z_lock); } VI_LOCK(vp); vp->v_data = NULL; Should I revert to the svn state and then change the if clause as above, or is this correct? Regards, Thomas From numisemis at yahoo.com Thu Jul 30 07:21:55 2009 From: numisemis at yahoo.com (Simun Mikecin) Date: Thu Jul 30 07:22:01 2009 Subject: ZFS raidz1 pool unavailable from losing 1 device In-Reply-To: <4A712290.9030308@chip-web.com> References: <4A712290.9030308@chip-web.com> Message-ID: <46899.11156.qm@web37301.mail.mud.yahoo.com> Ludwin Pummer wrote: > My system is 7.2-STABLE Jul 27, amd64, 4GB memory, just upgraded from 6.4-STABLE > from last year. I just set up a ZFS raidz volume to replace a graid5 volume I > had been using. I had it successfully set up using partitions across 4 disks, > ad{6,8,10,12}s1e. Then I wanted to expand the raidz volume by merging the space > from the adjacent disk partition. I thought I could just fail out the partition > device in ZFS, edit the bsdlabel, and re-add the larger partition, ZFS would > resilver, repeat until done. That's when I found out that ZFS doesn't let you > fail out a device in a raidz volume. No big deal, I thought, I'll just go to > single user mode and mess with the partition when ZFS isn't looking. When it > comes back up it should notice that one of the device is gone, I can do a 'zfs > replace' and continue my plan. > > Well, after rebooting to single user mode, combining partitions ad12s1d and > ad12s1e (removed the d partiton), "zfs volinit", then "zpool status" just hung > (Ctrl-C didn't kill it, so I rebooted). I thought this was a bit odd so I > thought perhaps ZFS is confused by the ZFS metadata left on ad12s1e, so I > blanked it out with "dd". That didn't help. I changed the name of the partition > to ad12s1d thinking perhaps that would help. After that, "zfs volinit; zfs mount > -a; zpool status" showed my raidz pool UNAVAIL with the message "insufficient > replicas", ad{6,8,10}s1e ONLINE, and ad12s1e UNAVAIL "cannot open", and a more > detailed message pointing me to http://www.sun.com/msg/ZFS-8000-3C. I tried > doing a "zpool replace storage ad12s1e ad12s1d" but it refused, saying my zpool > ("storage") was unavailable. Ditto for pretty much every zpool command I tried. > "zpool clear" gave me a "permission denied" error. Was your pool imported while you were repartitioning in single user mode? From ludwigp at chip-web.com Thu Jul 30 07:34:37 2009 From: ludwigp at chip-web.com (Ludwig Pummer) Date: Thu Jul 30 07:34:44 2009 Subject: ZFS raidz1 pool unavailable from losing 1 device In-Reply-To: <46899.11156.qm@web37301.mail.mud.yahoo.com> References: <4A712290.9030308@chip-web.com> <46899.11156.qm@web37301.mail.mud.yahoo.com> Message-ID: <4A714B03.6050704@chip-web.com> Simun Mikecin wrote: > Ludwin Pummer wrote: > > >> My system is 7.2-STABLE Jul 27, amd64, 4GB memory, just upgraded from 6.4-STABLE >> from last year. I just set up a ZFS raidz volume to replace a graid5 volume I >> had been using. I had it successfully set up using partitions across 4 disks, >> ad{6,8,10,12}s1e. Then I wanted to expand the raidz volume by merging the space >> from the adjacent disk partition. I thought I could just fail out the partition >> device in ZFS, edit the bsdlabel, and re-add the larger partition, ZFS would >> resilver, repeat until done. That's when I found out that ZFS doesn't let you >> fail out a device in a raidz volume. No big deal, I thought, I'll just go to >> single user mode and mess with the partition when ZFS isn't looking. When it >> comes back up it should notice that one of the device is gone, I can do a 'zfs >> replace' and continue my plan. >> >> Well, after rebooting to single user mode, combining partitions ad12s1d and >> ad12s1e (removed the d partiton), "zfs volinit", then "zpool status" just hung >> (Ctrl-C didn't kill it, so I rebooted). I thought this was a bit odd so I >> thought perhaps ZFS is confused by the ZFS metadata left on ad12s1e, so I >> blanked it out with "dd". That didn't help. I changed the name of the partition >> to ad12s1d thinking perhaps that would help. After that, "zfs volinit; zfs mount >> -a; zpool status" showed my raidz pool UNAVAIL with the message "insufficient >> replicas", ad{6,8,10}s1e ONLINE, and ad12s1e UNAVAIL "cannot open", and a more >> detailed message pointing me to http://www.sun.com/msg/ZFS-8000-3C. I tried >> doing a "zpool replace storage ad12s1e ad12s1d" but it refused, saying my zpool >> ("storage") was unavailable. Ditto for pretty much every zpool command I tried. >> "zpool clear" gave me a "permission denied" error. >> > > Was your pool imported while you were repartitioning in single user mode? > > Yes, I guess you could say it was. ZFS wasn't loaded while I was doing the repartitioning, though. --Ludwig From rene at freebsd.org Thu Jul 30 09:05:35 2009 From: rene at freebsd.org (Rene Ladan) Date: Thu Jul 30 09:05:42 2009 Subject: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) In-Reply-To: <200907291135.17569.jhb@freebsd.org> References: <200907271400.n6RE05Rv056472@freefall.freebsd.org> <200907290742.20838.jhb@freebsd.org> <200907291135.17569.jhb@freebsd.org> Message-ID: 2009/7/29 John Baldwin : > On Wednesday 29 July 2009 11:20:21 am Rene Ladan wrote: >> 2009/7/29 John Baldwin : >> > On Wednesday 29 July 2009 5:52:24 am Rene Ladan wrote: >> >> 2009/7/28 John Baldwin : >> >> > On Tuesday 28 July 2009 10:03:40 am Rene Ladan wrote: >> >> >> 2009/7/28 John Baldwin : >> >> >> > On Monday 27 July 2009 10:00:05 am Rene Ladan wrote: >> >> >> >> The following reply was made to PR kern/136945; it has been noted > by >> >> > GNATS. >> >> >> >> >> >> >> >> From: Rene Ladan >> >> >> >> To: John Baldwin >> >> >> >> Cc: bug-followup@freebsd.org >> >> >> >> Subject: Re: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) >> >> >> >> Date: Mon, 27 Jul 2009 15:51:15 +0200 >> >> >> >> >> >> >> >> ?2009/7/27 John Baldwin : >> >> >> >> ?> I would actually expect this to be the correct order for these > two >> >> >> > locks.= >> >> >> >> ? =A0Can >> >> >> >> ?> you capture the output of the 'debug.witness.fullgraph' sysctl > to a >> >> > file? >> >> >> >> ?> >> >> >> >> ?Yes, see attachment. ?I'm still running the same 8.0-BETA2. >> >> >> > >> >> >> > Hmm, the attachment was eaten by a grue, can you post the file >> > somewhere? >> >> >> > >> >> >> Yes, see ftp://rene-ladan.nl/pub/freebsd/kern_136945.txt >> >> > >> >> > Ok, it looks like it did encounter a UFS -> filedesc order at some >> > point. ?Can >> >> > you patch sys/kern/subr_witness.c to add a section to the order_lists[] >> > array >> >> > after the 'ZFS locking list' and before the spin locks list that looks >> > like >> >> > this: >> >> > >> >> > ? ? ? ?{ "filedesc structure", &lock_class_sx }, >> >> > ? ? ? ?{ "ufs", &lock_class_lockmgr}, >> >> > ? ? ? ?{ NULL, NULL }, >> >> > >> >> The LOR seems to be gone, previously it showed up only once right >> >> after booting the system. >> >> >> >> But now a new LOR (according to the LOR page) seems pop up: >> >> Trying to mount root from ufs:/dev/ad0s1a >> >> lock order reversal: >> >> ?1st 0xffffff0002a4ad80 ufs (ufs) > @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1465 >> >> ?2nd 0xffffff0002b29a48 filedesc structure (filedesc structure) @ >> >> /usr/src/sys/kern/kern_descrip.c:2478 >> >> KDB: stack backtrace: >> >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a >> >> _witness_debugger() at _witness_debugger+0x49 >> >> witness_checkorder() at witness_checkorder+0x7ea >> >> _sx_xlock() at _sx_xlock+0x44 >> >> mountcheckdirs() at mountcheckdirs+0x80 >> >> vfs_donmount() at vfs_donmount+0xfbf >> >> kernel_mount() at kernel_mount+0xa1 >> >> vfs_mountroot_try() at vfs_mountroot_try+0x177 >> >> vfs_mountroot() at vfs_mountroot+0x47d >> >> start_init() at start_init+0x62 >> >> fork_exit() at fork_exit+0x12a >> >> fork_trampoline() at fork_trampoline+0xe >> >> --- trap 0, rip = 0, rsp = 0xffffff800001ad30, rbp = 0 --- >> >> >> >> The output of `df' and `mount' looks ok. >> > >> > Yes, this is the "real" LOR as "filedesc" -> "ufs" in the poll() case > should >> > be the normal order. ?I believe this should fix it. ?mountcheckdirs() > doesn't >> > need the vnodes locked, it just needs the caller to hold references on > them >> > so they aren't recycled: >> > >> > --- //depot/projects/smpng/sys/kern/vfs_mount.c#96 >> > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c >> > @@ -1069,9 +1069,10 @@ >> > ? ? ? ? ? ? ? ?vfs_event_signal(NULL, VQ_MOUNT, 0); >> > ? ? ? ? ? ? ? ?if (VFS_ROOT(mp, LK_EXCLUSIVE, &newdp)) >> > ? ? ? ? ? ? ? ? ? ? ? ?panic("mount: lost mount"); >> > + ? ? ? ? ? ? ? VOP_UNLOCK(newdp, 0); >> > + ? ? ? ? ? ? ? VOP_UNLOCK(vp, 0); >> > ? ? ? ? ? ? ? ?mountcheckdirs(vp, newdp); >> > - ? ? ? ? ? ? ? vput(newdp); >> > - ? ? ? ? ? ? ? VOP_UNLOCK(vp, 0); >> > + ? ? ? ? ? ? ? vrele(newdp); >> > ? ? ? ? ? ? ? ?if ((mp->mnt_flag & MNT_RDONLY) == 0) >> > ? ? ? ? ? ? ? ? ? ? ? ?error = vfs_allocate_syncvnode(mp); >> > ? ? ? ? ? ? ? ?vfs_unbusy(mp); >> > >> The LOR is still present, but at a different place without the >> mountcheckdirs() call (not on the LOR page either) : > > Ok, try this patch as well: > > --- //depot/projects/smpng/sys/kern/vfs_mount.c#97 > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c > @@ -1481,6 +1481,8 @@ > ? ? ? ?if (VFS_ROOT(TAILQ_FIRST(&mountlist), LK_EXCLUSIVE, &rootvnode)) > ? ? ? ? ? ? ? ?panic("Cannot find root vnode"); > > + ? ? ? VOP_UNLOCK(rootvnode, 0); > + > ? ? ? ?p = curthread->td_proc; > ? ? ? ?FILEDESC_XLOCK(p->p_fd); > > @@ -1496,8 +1498,6 @@ > > ? ? ? ?FILEDESC_XUNLOCK(p->p_fd); > > - ? ? ? VOP_UNLOCK(rootvnode, 0); > - > ? ? ? ?EVENTHANDLER_INVOKE(mountroot); > ?} > Still no luck, I now get a LOR that is similar to LOR 281 right after booting: lock order reversal: 1st 0xffffff0002c2c7f8 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2083 2nd 0xffffff0002b2a248 filedesc structure (filedesc structure) @ /usr/src/sys/kern/vfs_syscalls.c:3776 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x49 witness_checkorder() at witness_checkorder+0x7ea _sx_slock() at _sx_slock+0x44 kern_mkdirat() at kern_mkdirat+0x201 syscall() at syscall+0x1af Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (136, FreeBSD ELF64, mkdir), rip = 0x800729dac, rsp = 0x7fffffffec88, rbp = 0x7fffffffef66 --- Ren? From kostikbel at gmail.com Thu Jul 30 09:25:16 2009 From: kostikbel at gmail.com (Kostik Belousov) Date: Thu Jul 30 09:25:23 2009 Subject: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) In-Reply-To: References: <200907271400.n6RE05Rv056472@freefall.freebsd.org> <200907290742.20838.jhb@freebsd.org> <200907291135.17569.jhb@freebsd.org> Message-ID: <20090730092507.GF1884@deviant.kiev.zoral.com.ua> On Thu, Jul 30, 2009 at 11:05:32AM +0200, Rene Ladan wrote: > 2009/7/29 John Baldwin : > > On Wednesday 29 July 2009 11:20:21 am Rene Ladan wrote: > >> 2009/7/29 John Baldwin : > >> > On Wednesday 29 July 2009 5:52:24 am Rene Ladan wrote: > >> >> 2009/7/28 John Baldwin : > >> >> > On Tuesday 28 July 2009 10:03:40 am Rene Ladan wrote: > >> >> >> 2009/7/28 John Baldwin : > >> >> >> > On Monday 27 July 2009 10:00:05 am Rene Ladan wrote: > >> >> >> >> The following reply was made to PR kern/136945; it has been noted > > by > >> >> > GNATS. > >> >> >> >> > >> >> >> >> From: Rene Ladan > >> >> >> >> To: John Baldwin > >> >> >> >> Cc: bug-followup@freebsd.org > >> >> >> >> Subject: Re: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) > >> >> >> >> Date: Mon, 27 Jul 2009 15:51:15 +0200 > >> >> >> >> > >> >> >> >> ?2009/7/27 John Baldwin : > >> >> >> >> ?> I would actually expect this to be the correct order for these > > two > >> >> >> > locks.= > >> >> >> >> ? =A0Can > >> >> >> >> ?> you capture the output of the 'debug.witness.fullgraph' sysctl > > to a > >> >> > file? > >> >> >> >> ?> > >> >> >> >> ?Yes, see attachment. ?I'm still running the same 8.0-BETA2. > >> >> >> > > >> >> >> > Hmm, the attachment was eaten by a grue, can you post the file > >> > somewhere? > >> >> >> > > >> >> >> Yes, see ftp://rene-ladan.nl/pub/freebsd/kern_136945.txt > >> >> > > >> >> > Ok, it looks like it did encounter a UFS -> filedesc order at some > >> > point. ?Can > >> >> > you patch sys/kern/subr_witness.c to add a section to the order_lists[] > >> > array > >> >> > after the 'ZFS locking list' and before the spin locks list that looks > >> > like > >> >> > this: > >> >> > > >> >> > ? ? ? ?{ "filedesc structure", &lock_class_sx }, > >> >> > ? ? ? ?{ "ufs", &lock_class_lockmgr}, > >> >> > ? ? ? ?{ NULL, NULL }, > >> >> > > >> >> The LOR seems to be gone, previously it showed up only once right > >> >> after booting the system. > >> >> > >> >> But now a new LOR (according to the LOR page) seems pop up: > >> >> Trying to mount root from ufs:/dev/ad0s1a > >> >> lock order reversal: > >> >> ?1st 0xffffff0002a4ad80 ufs (ufs) > > @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1465 > >> >> ?2nd 0xffffff0002b29a48 filedesc structure (filedesc structure) @ > >> >> /usr/src/sys/kern/kern_descrip.c:2478 > >> >> KDB: stack backtrace: > >> >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > >> >> _witness_debugger() at _witness_debugger+0x49 > >> >> witness_checkorder() at witness_checkorder+0x7ea > >> >> _sx_xlock() at _sx_xlock+0x44 > >> >> mountcheckdirs() at mountcheckdirs+0x80 > >> >> vfs_donmount() at vfs_donmount+0xfbf > >> >> kernel_mount() at kernel_mount+0xa1 > >> >> vfs_mountroot_try() at vfs_mountroot_try+0x177 > >> >> vfs_mountroot() at vfs_mountroot+0x47d > >> >> start_init() at start_init+0x62 > >> >> fork_exit() at fork_exit+0x12a > >> >> fork_trampoline() at fork_trampoline+0xe > >> >> --- trap 0, rip = 0, rsp = 0xffffff800001ad30, rbp = 0 --- > >> >> > >> >> The output of `df' and `mount' looks ok. > >> > > >> > Yes, this is the "real" LOR as "filedesc" -> "ufs" in the poll() case > > should > >> > be the normal order. ?I believe this should fix it. ?mountcheckdirs() > > doesn't > >> > need the vnodes locked, it just needs the caller to hold references on > > them > >> > so they aren't recycled: > >> > > >> > --- //depot/projects/smpng/sys/kern/vfs_mount.c#96 > >> > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c > >> > @@ -1069,9 +1069,10 @@ > >> > ? ? ? ? ? ? ? ?vfs_event_signal(NULL, VQ_MOUNT, 0); > >> > ? ? ? ? ? ? ? ?if (VFS_ROOT(mp, LK_EXCLUSIVE, &newdp)) > >> > ? ? ? ? ? ? ? ? ? ? ? ?panic("mount: lost mount"); > >> > + ? ? ? ? ? ? ? VOP_UNLOCK(newdp, 0); > >> > + ? ? ? ? ? ? ? VOP_UNLOCK(vp, 0); > >> > ? ? ? ? ? ? ? ?mountcheckdirs(vp, newdp); > >> > - ? ? ? ? ? ? ? vput(newdp); > >> > - ? ? ? ? ? ? ? VOP_UNLOCK(vp, 0); > >> > + ? ? ? ? ? ? ? vrele(newdp); > >> > ? ? ? ? ? ? ? ?if ((mp->mnt_flag & MNT_RDONLY) == 0) > >> > ? ? ? ? ? ? ? ? ? ? ? ?error = vfs_allocate_syncvnode(mp); > >> > ? ? ? ? ? ? ? ?vfs_unbusy(mp); > >> > > >> The LOR is still present, but at a different place without the > >> mountcheckdirs() call (not on the LOR page either) : > > > > Ok, try this patch as well: > > > > --- //depot/projects/smpng/sys/kern/vfs_mount.c#97 > > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c > > @@ -1481,6 +1481,8 @@ > > ? ? ? ?if (VFS_ROOT(TAILQ_FIRST(&mountlist), LK_EXCLUSIVE, &rootvnode)) > > ? ? ? ? ? ? ? ?panic("Cannot find root vnode"); > > > > + ? ? ? VOP_UNLOCK(rootvnode, 0); > > + > > ? ? ? ?p = curthread->td_proc; > > ? ? ? ?FILEDESC_XLOCK(p->p_fd); > > > > @@ -1496,8 +1498,6 @@ > > > > ? ? ? ?FILEDESC_XUNLOCK(p->p_fd); > > > > - ? ? ? VOP_UNLOCK(rootvnode, 0); > > - > > ? ? ? ?EVENTHANDLER_INVOKE(mountroot); > > ?} > > > > Still no luck, I now get a LOR that is similar to LOR 281 right after booting: > > lock order reversal: > 1st 0xffffff0002c2c7f8 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2083 > 2nd 0xffffff0002b2a248 filedesc structure (filedesc structure) @ > /usr/src/sys/kern/vfs_syscalls.c:3776 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > _witness_debugger() at _witness_debugger+0x49 > witness_checkorder() at witness_checkorder+0x7ea > _sx_slock() at _sx_slock+0x44 > kern_mkdirat() at kern_mkdirat+0x201 > syscall() at syscall+0x1af > Xfast_syscall() at Xfast_syscall+0xe1 > --- syscall (136, FreeBSD ELF64, mkdir), rip = 0x800729dac, rsp = > 0x7fffffffec88, rbp = 0x7fffffffef66 --- Remove the FILEDESC_SLOCK()/FILEDESC_SUNLOCK() calls from kern_mkdirat(). -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20090730/69c94e42/attachment.pgp From avg at freebsd.org Thu Jul 30 12:11:52 2009 From: avg at freebsd.org (Andriy Gapon) Date: Thu Jul 30 12:12:03 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <86983A55-E5C4-4C04-A4C7-0AE9A9EE37A3@exscape.org> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> <4A708455.5070304@freebsd.org> <86983A55-E5C4-4C04-A4C7-0AE9A9EE37A3@exscape.org> Message-ID: <4A718E03.6030909@freebsd.org> on 29/07/2009 21:04 Thomas Backman said the following: > Thanks for your work :) > However, bad news: it didn't help. It *might* have gotten us further, > though, because the DDB backtrace now looks like this: > > _sx_xlock_hard() > _sx_xlock() > zfs_znode_free() > zfs_freebsd_inactive() > VOP_INACTIVE_APV() > vinactive() > vput() > dounmount() > unmount() > syscall() > XFast_syscall() > Oh my bad. I missed the fact that recycle would do zfs_znode_free, so it seems like zfs_znode_free was called twice on the same znode. Could you please try replacing zfs_znode_free(zp); with vrecycle(vp, curthread); in the same block (instead of adding the latter before the former). Sorry, if this looks like shooting in the dark - because this is what it is. I am not familiar with the code and it's hard to follow all possibilities without good understanding. -- Andriy Gapon From serenity at exscape.org Thu Jul 30 12:51:49 2009 From: serenity at exscape.org (Thomas Backman) Date: Thu Jul 30 12:51:56 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <4A718E03.6030909@freebsd.org> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> <4A708455.5070304@freebsd.org> <86983A55-E5C4-4C04-A4C7-0AE9A9EE37A3@exscape.org> <4A718E03.6030909@freebsd.org> Message-ID: <71A038EC-02B1-4606-96C2-5E84BE80F005@exscape.org> On Jul 30, 2009, at 14:11, Andriy Gapon wrote: > on 29/07/2009 21:04 Thomas Backman said the following: >> Thanks for your work :) >> However, bad news: it didn't help. It *might* have gotten us further, >> though, because the DDB backtrace now looks like this: >> >> _sx_xlock_hard() >> _sx_xlock() >> zfs_znode_free() >> zfs_freebsd_inactive() >> VOP_INACTIVE_APV() >> vinactive() >> vput() >> dounmount() >> unmount() >> syscall() >> XFast_syscall() >> > > Oh my bad. I missed the fact that recycle would do zfs_znode_free, > so it seems > like zfs_znode_free was called twice on the same znode. > Could you please try replacing > zfs_znode_free(zp); > with > vrecycle(vp, curthread); > in the same block (instead of adding the latter before the former). > Sorry, if this looks like shooting in the dark - because this is > what it is. I am > not familiar with the code and it's hard to follow all possibilities > without good > understanding. New panic. :( Damnit! I think I'm using svn + http://people.freebsd.org/~pjd/patches/zfs_vnops.c.2.patch + your change, now... Unread portion of the kernel message buffer: GEOM_GATE: Device ggate1482 destroyed. panic: solaris assert: zp != ((void *)0), file: /usr/src/sys/modules/ zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c, line: 4359 cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a panic() at panic+0x182 zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x244 VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0x4a vgonel() at vgonel+0x12e vrecycle() at vrecycle+0x7d zfs_freebsd_inactive() at zfs_freebsd_inactive+0x1a VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x4a vinactive() at vinactive+0x6a vput() at vput+0x1c6 dounmount() at dounmount+0x4af unmount() at unmount+0x3c8 syscall() at syscall+0x28f Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (22, FreeBSD ELF64, unmount), rip = 0x80104e9ec, rsp = 0x7fffffffaa98, rbp = 0x801223300 --- KDB: enter: panic 0xffffff00452971d8: tag zfs, type VDIR usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VI_DOOMED|VI_DOINGINACT) lock type zfs: EXCL by thread 0xffffff0019ff6000 (pid 1425) panic: from debugger ... #11 0xffffffff8033a9cb in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:558 #12 0xffffffff80b11124 in zfs_freebsd_reclaim () from /boot/kernel/ zfs.ko #13 0xffffffff805c5c2a in VOP_RECLAIM_APV (vop=0x0, a=0xffffff803eaf8930) at vnode_if.c:1926 #14 0xffffffff803c839e in vgonel (vp=0xffffff00452971d8) at vnode_if.h: 830 #15 0xffffffff803ca7ad in vrecycle (vp=0xffffff00452971d8, td=Variable "td" is not available. ) at /usr/src/sys/kern/vfs_subr.c:2504 #16 0xffffffff80b10aaa in zfs_freebsd_inactive () from /boot/kernel/ zfs.ko #17 0xffffffff805c5b5a in VOP_INACTIVE_APV (vop=0xffffffff80b882a0, a=0xffffff803eaf89f0) at vnode_if.c:1863 #18 0xffffffff803c6aaa in vinactive (vp=0xffffff00452971d8, td=0xffffff0019ff6000) at vnode_if.h:807 #19 0xffffffff803cbf26 in vput (vp=0xffffff00452971d8) at /usr/src/sys/kern/vfs_subr.c:2257 #20 0xffffffff803c57ef in dounmount (mp=0xffffff0001d058d0, flags=0, td=Variable "td" is not available. ) at /usr/src/sys/kern/vfs_mount.c:1333 #21 0xffffffff803c5df8 in unmount (td=0xffffff0019ff6000, uap=0xffffff803eaf8bf0) at /usr/src/sys/kern/vfs_mount.c:1174 #22 0xffffffff805980bf in syscall (frame=0xffffff803eaf8c80) at /usr/src/sys/amd64/amd64/trap.c:984 #23 0xffffffff8057e2c1 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:373 #24 0x000000080104e9ec in ?? () FWIW: Line 4359 (panic line): zfs_freebsd_reclaim(ap) ... { vnode_t *vp = ap->a_vp; znode_t *zp = VTOZ(vp); ASSERT(ap != NULL); // added by me ASSERT(vp != NULL); // added by me >>> ASSERT(zp != NULL); // line 4359 --------------- zfs_inactive(vnode_t *vp, cred_t *cr, caller_context_t *ct) { znode_t *zp = VTOZ(vp); zfsvfs_t *zfsvfs = zp->z_zfsvfs; int error; rw_enter(&zfsvfs->z_teardown_inactive_lock, RW_READER); if (zp->z_dbuf == NULL) { /* * The fs has been unmounted, or we did a * suspend/resume and this file no longer exists. */ VI_LOCK(vp); vp->v_count = 0; /* count arrives as 1 */ vp->v_data = NULL; VI_UNLOCK(vp); rw_exit(&zfsvfs->z_teardown_inactive_lock); ZTOV(zp) = NULL; vrecycle(vp, curthread); // zfs_znode_free(zp); return; } Regards, Thomas PS. ... and thanks again for working to solve this. :) From mdounin at mdounin.ru Thu Jul 30 12:55:45 2009 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu Jul 30 12:56:29 2009 Subject: another zfs panic In-Reply-To: <20090730055001.GB2130@garage.freebsd.pl> References: <20090730013857.GB8794@mdounin.ru> <20090730055001.GB2130@garage.freebsd.pl> Message-ID: <20090730125542.GI8794@mdounin.ru> Hello! On Thu, Jul 30, 2009 at 07:50:01AM +0200, Pawel Jakub Dawidek wrote: > On Thu, Jul 30, 2009 at 05:38:57AM +0400, Maxim Dounin wrote: > > Hello! > > > > Here is zfs panic I'm able to reproduce by running an scp from > > remote machine to zfs volume and 3 parallel untars of ports tree > > in cycle. Not sure that everything is required, but the above > > workload triggers panic in several hours. > > > > This is on fresh current with GENERIC kernel: > > > > panic: sx_xlock() of destroyed sx @ > > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_rlock.c:535 > > cpuid = 6 > > KDB: enter: panic > > [thread pid 36 tid 100071 ] > > Stopped at kdb_enter+0x3d: movq $0,0x68a040(%rip) > > db> bt > > Tracing pid 36 tid 100071 td 0xffffff00040f3720 > > kdb_enter() at kdb_enter+0x3d > > panic() at panic+0x17b > > _sx_xlock() at _sx_xlock+0xfc > > zfs_range_unlock() at zfs_range_unlock+0x38 > > zfs_get_data() at zfs_get_data+0xc1 > > zil_commit() at zil_commit+0x532 > > zfs_sync() at zfs_sync+0xa6 > > sync_fsync() at sync_fsync+0x13a > > sync_vnode() at sync_vnode+0x157 > > sched_sync() at sched_sync+0x1d1 > > fork_exit() at fork_exit+0x12a > > fork_trampoline() at fork_trampoline+0xe > > --- trap 0, rip = 0, rsp = 0xffffff80e7ee3d30, rbp = 0 --- > > > > Machine is otherwise idle. The only zfs-related tuning applied is > > compression=gzip-9. > > > > Please let me know if you want me to test some patches. > > The kernel syncer tries to sync vnode which has its znode already > destroyed. There is one place (that we know of) where vrecycle() is > missing. Could you try this patch: > > http://people.freebsd.org/~pjd/patches/zfs_vnops.c.2.patch Still here with patch applied: panic: sx_xlock() of destroyed sx @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_rlock.c:535 cpuid = 3 KDB: enter: panic [thread pid 37 tid 100072 ] Stopped at kdb_enter+0x3d: movq $0,0x68a040(%rip) db> bt Tracing pid 37 tid 100072 td 0xffffff00040f3390 kdb_enter() at kdb_enter+0x3d panic() at panic+0x17b _sx_xlock() at _sx_xlock+0xfc zfs_range_unlock() at zfs_range_unlock+0x38 zfs_get_data() at zfs_get_data+0xc1 zil_commit() at zil_commit+0x532 zfs_sync() at zfs_sync+0xa6 sync_fsync() at sync_fsync+0x13a sync_vnode() at sync_vnode+0x157 sched_sync() at sched_sync+0x1d1 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80e7ee8d30, rbp = 0 --- Maxim Dounin From rene at freebsd.org Thu Jul 30 12:55:51 2009 From: rene at freebsd.org (Rene Ladan) Date: Thu Jul 30 12:56:29 2009 Subject: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) In-Reply-To: <20090730092507.GF1884@deviant.kiev.zoral.com.ua> References: <200907271400.n6RE05Rv056472@freefall.freebsd.org> <200907290742.20838.jhb@freebsd.org> <200907291135.17569.jhb@freebsd.org> <20090730092507.GF1884@deviant.kiev.zoral.com.ua> Message-ID: 2009/7/30 Kostik Belousov : > On Thu, Jul 30, 2009 at 11:05:32AM +0200, Rene Ladan wrote: >> 2009/7/29 John Baldwin : >> > On Wednesday 29 July 2009 11:20:21 am Rene Ladan wrote: >> >> 2009/7/29 John Baldwin : >> >> > On Wednesday 29 July 2009 5:52:24 am Rene Ladan wrote: >> >> >> 2009/7/28 John Baldwin : >> >> >> > On Tuesday 28 July 2009 10:03:40 am Rene Ladan wrote: >> >> >> >> 2009/7/28 John Baldwin : >> >> >> >> > On Monday 27 July 2009 10:00:05 am Rene Ladan wrote: >> >> >> >> >> The following reply was made to PR kern/136945; it has been noted >> > by >> >> >> > GNATS. >> >> >> >> >> >> >> >> >> >> From: Rene Ladan >> >> >> >> >> To: John Baldwin >> >> >> >> >> Cc: bug-followup@freebsd.org >> >> >> >> >> Subject: Re: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) >> >> >> >> >> Date: Mon, 27 Jul 2009 15:51:15 +0200 >> >> >> >> >> >> >> >> >> >> ?2009/7/27 John Baldwin : >> >> >> >> >> ?> I would actually expect this to be the correct order for these >> > two >> >> >> >> > locks.= >> >> >> >> >> ? =A0Can >> >> >> >> >> ?> you capture the output of the 'debug.witness.fullgraph' sysctl >> > to a >> >> >> > file? >> >> >> >> >> ?> >> >> >> >> >> ?Yes, see attachment. ?I'm still running the same 8.0-BETA2. >> >> >> >> > >> >> >> >> > Hmm, the attachment was eaten by a grue, can you post the file >> >> > somewhere? >> >> >> >> > >> >> >> >> Yes, see ftp://rene-ladan.nl/pub/freebsd/kern_136945.txt >> >> >> > >> >> >> > Ok, it looks like it did encounter a UFS -> filedesc order at some >> >> > point. ?Can >> >> >> > you patch sys/kern/subr_witness.c to add a section to the order_lists[] >> >> > array >> >> >> > after the 'ZFS locking list' and before the spin locks list that looks >> >> > like >> >> >> > this: >> >> >> > >> >> >> > ? ? ? ?{ "filedesc structure", &lock_class_sx }, >> >> >> > ? ? ? ?{ "ufs", &lock_class_lockmgr}, >> >> >> > ? ? ? ?{ NULL, NULL }, >> >> >> > >> >> >> The LOR seems to be gone, previously it showed up only once right >> >> >> after booting the system. >> >> >> >> >> >> But now a new LOR (according to the LOR page) seems pop up: >> >> >> Trying to mount root from ufs:/dev/ad0s1a >> >> >> lock order reversal: >> >> >> ?1st 0xffffff0002a4ad80 ufs (ufs) >> > @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1465 >> >> >> ?2nd 0xffffff0002b29a48 filedesc structure (filedesc structure) @ >> >> >> /usr/src/sys/kern/kern_descrip.c:2478 >> >> >> KDB: stack backtrace: >> >> >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a >> >> >> _witness_debugger() at _witness_debugger+0x49 >> >> >> witness_checkorder() at witness_checkorder+0x7ea >> >> >> _sx_xlock() at _sx_xlock+0x44 >> >> >> mountcheckdirs() at mountcheckdirs+0x80 >> >> >> vfs_donmount() at vfs_donmount+0xfbf >> >> >> kernel_mount() at kernel_mount+0xa1 >> >> >> vfs_mountroot_try() at vfs_mountroot_try+0x177 >> >> >> vfs_mountroot() at vfs_mountroot+0x47d >> >> >> start_init() at start_init+0x62 >> >> >> fork_exit() at fork_exit+0x12a >> >> >> fork_trampoline() at fork_trampoline+0xe >> >> >> --- trap 0, rip = 0, rsp = 0xffffff800001ad30, rbp = 0 --- >> >> >> >> >> >> The output of `df' and `mount' looks ok. >> >> > >> >> > Yes, this is the "real" LOR as "filedesc" -> "ufs" in the poll() case >> > should >> >> > be the normal order. ?I believe this should fix it. ?mountcheckdirs() >> > doesn't >> >> > need the vnodes locked, it just needs the caller to hold references on >> > them >> >> > so they aren't recycled: >> >> > >> >> > --- //depot/projects/smpng/sys/kern/vfs_mount.c#96 >> >> > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c >> >> > @@ -1069,9 +1069,10 @@ >> >> > ? ? ? ? ? ? ? ?vfs_event_signal(NULL, VQ_MOUNT, 0); >> >> > ? ? ? ? ? ? ? ?if (VFS_ROOT(mp, LK_EXCLUSIVE, &newdp)) >> >> > ? ? ? ? ? ? ? ? ? ? ? ?panic("mount: lost mount"); >> >> > + ? ? ? ? ? ? ? VOP_UNLOCK(newdp, 0); >> >> > + ? ? ? ? ? ? ? VOP_UNLOCK(vp, 0); >> >> > ? ? ? ? ? ? ? ?mountcheckdirs(vp, newdp); >> >> > - ? ? ? ? ? ? ? vput(newdp); >> >> > - ? ? ? ? ? ? ? VOP_UNLOCK(vp, 0); >> >> > + ? ? ? ? ? ? ? vrele(newdp); >> >> > ? ? ? ? ? ? ? ?if ((mp->mnt_flag & MNT_RDONLY) == 0) >> >> > ? ? ? ? ? ? ? ? ? ? ? ?error = vfs_allocate_syncvnode(mp); >> >> > ? ? ? ? ? ? ? ?vfs_unbusy(mp); >> >> > >> >> The LOR is still present, but at a different place without the >> >> mountcheckdirs() call (not on the LOR page either) : >> > >> > Ok, try this patch as well: >> > >> > --- //depot/projects/smpng/sys/kern/vfs_mount.c#97 >> > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c >> > @@ -1481,6 +1481,8 @@ >> > ? ? ? ?if (VFS_ROOT(TAILQ_FIRST(&mountlist), LK_EXCLUSIVE, &rootvnode)) >> > ? ? ? ? ? ? ? ?panic("Cannot find root vnode"); >> > >> > + ? ? ? VOP_UNLOCK(rootvnode, 0); >> > + >> > ? ? ? ?p = curthread->td_proc; >> > ? ? ? ?FILEDESC_XLOCK(p->p_fd); >> > >> > @@ -1496,8 +1498,6 @@ >> > >> > ? ? ? ?FILEDESC_XUNLOCK(p->p_fd); >> > >> > - ? ? ? VOP_UNLOCK(rootvnode, 0); >> > - >> > ? ? ? ?EVENTHANDLER_INVOKE(mountroot); >> > ?} >> > >> >> Still no luck, I now get a LOR that is similar to LOR 281 right after booting: >> >> lock order reversal: >> ?1st 0xffffff0002c2c7f8 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2083 >> ?2nd 0xffffff0002b2a248 filedesc structure (filedesc structure) @ >> /usr/src/sys/kern/vfs_syscalls.c:3776 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a >> _witness_debugger() at _witness_debugger+0x49 >> witness_checkorder() at witness_checkorder+0x7ea >> _sx_slock() at _sx_slock+0x44 >> kern_mkdirat() at kern_mkdirat+0x201 >> syscall() at syscall+0x1af >> Xfast_syscall() at Xfast_syscall+0xe1 >> --- syscall (136, FreeBSD ELF64, mkdir), rip = 0x800729dac, rsp = >> 0x7fffffffec88, rbp = 0x7fffffffef66 --- > > Remove the FILEDESC_SLOCK()/FILEDESC_SUNLOCK() calls from kern_mkdirat(). > I removed the two lines at sys/kern/vfs_syscalls.c (3776 and 3778), but there still seem to be some LORs (attached). The two LORs about the reboot call are from before Kostiks patch. -------------- next part -------------- FreeBSD 8.0-BETA2 #2: Thu Jul 30 09:xx:xx CEST 2009 lock order reversal: (#276) lock order reversal: 1st 0xffffff0002b50270 ufs (ufs) @ /usr/src/sys/kern/vfs_mount.c:1200 2nd 0xffffffff80bdcca0 allproc (allproc) @ /usr/src/sys/kern/kern_descrip.c:2473 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x49 witness_checkorder() at witness_checkorder+0x7ea _sx_slock() at _sx_slock+0x44 mountcheckdirs() at mountcheckdirs+0x3f dounmount() at dounmount+0x477 vfs_unmountall() at vfs_unmountall+0x54 boot() at boot+0x818 mkdumpheader() at mkdumpheader syscall() at syscall+0x1af Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (55, FreeBSD ELF64, reboot), rip = 0x80078f85c, rsp = 0x7fffffffeab8, rbp = 0 --- FreeBSD 8.0-BETA2 #3: Thu Jul 30 13:29:46 CEST 2009 lock order reversal: 1st 0xffffff00510a5d80 ufs (ufs) @ /usr/src/sys/kern/kern_exec.c:570 2nd 0xffffff0002dfe248 filedesc structure (filedesc structure) @ /usr/src/sys/kern/kern_descrip.c:1864 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x49 witness_checkorder() at witness_checkorder+0x7ea _sx_xlock() at _sx_xlock+0x44 setugidsafety() at setugidsafety+0x40 kern_execve() at kern_execve+0xf22 execve() at execve+0x38 syscall() at syscall+0x1af Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (59, FreeBSD ELF64, execve), rip = 0x8007c3d0c, rsp = 0x7fffffffec48, rbp = 0x7fffffffed50 --- lock order reversal: (like #261) 1st 0xffffff8029512438 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2558 2nd 0xffffff0002c44400 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:285 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x49 witness_checkorder() at witness_checkorder+0x7ea _sx_xlock() at _sx_xlock+0x44 ufsdirhash_acquire() at ufsdirhash_acquire+0x29 ufsdirhash_move() at ufsdirhash_move+0x19 ufs_direnter() at ufs_direnter+0x4a9 ufs_makeinode() at ufs_makeinode+0x2a7 VOP_CREATE_APV() at VOP_CREATE_APV+0x8d vn_open_cred() at vn_open_cred+0x406 kern_openat() at kern_openat+0x163 syscall() at syscall+0x1af Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (5, FreeBSD ELF64, open), rip = 0x8009dadec, rsp = 0x7fffffffe5c8, rbp = 0x1b6 --- From avg at freebsd.org Thu Jul 30 13:14:17 2009 From: avg at freebsd.org (Andriy Gapon) Date: Thu Jul 30 13:14:23 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <71A038EC-02B1-4606-96C2-5E84BE80F005@exscape.org> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> <4A708455.5070304@freebsd.org> <86983A55-E5C4-4C04-A4C7-0AE9A9EE37A3@exscape.org> <4A718E03.6030909@freebsd.org> <71A038EC-02B1-4606-96C2-5E84BE80F005@exscape.org> Message-ID: <4A719CA4.4060400@freebsd.org> Thomas, I wasn't clear - please make sure that you have original zfs_inactive (without the changes that Pawel proposed) with the only change zfs_znode_free -> vrecycle. I.e.: if (zp->z_dbuf == NULL) { /* * The fs has been unmounted, or we did a * suspend/resume and this file no longer exists. */ mutex_enter(&zp->z_lock); VI_LOCK(vp); vp->v_count = 0; /* count arrives as 1 */ mutex_exit(&zp->z_lock); rw_exit(&zfsvfs->z_teardown_inactive_lock); vrecycle(vp, curthread); return; } I believe that the latest panic is a direct result of ZTOV(zp) = NULL line introduced in zfs_vnops.c.2.patch. reclaim function should stay patched with Pawel's patch. -- Andriy Gapon From kostikbel at gmail.com Thu Jul 30 13:21:28 2009 From: kostikbel at gmail.com (Kostik Belousov) Date: Thu Jul 30 13:21:34 2009 Subject: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) In-Reply-To: References: <200907271400.n6RE05Rv056472@freefall.freebsd.org> <200907290742.20838.jhb@freebsd.org> <200907291135.17569.jhb@freebsd.org> <20090730092507.GF1884@deviant.kiev.zoral.com.ua> Message-ID: <20090730132121.GH1884@deviant.kiev.zoral.com.ua> On Thu, Jul 30, 2009 at 02:55:48PM +0200, Rene Ladan wrote: > > FreeBSD 8.0-BETA2 #3: Thu Jul 30 13:29:46 CEST 2009 > > lock order reversal: > 1st 0xffffff00510a5d80 ufs (ufs) @ /usr/src/sys/kern/kern_exec.c:570 > 2nd 0xffffff0002dfe248 filedesc structure (filedesc structure) @ /usr/src/sys/kern/kern_descrip.c:1864 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > _witness_debugger() at _witness_debugger+0x49 > witness_checkorder() at witness_checkorder+0x7ea > _sx_xlock() at _sx_xlock+0x44 > setugidsafety() at setugidsafety+0x40 > kern_execve() at kern_execve+0xf22 > execve() at execve+0x38 > syscall() at syscall+0x1af > Xfast_syscall() at Xfast_syscall+0xe1 > --- syscall (59, FreeBSD ELF64, execve), rip = 0x8007c3d0c, rsp = 0x7fffffffec48, rbp = 0x7fffffffed50 --- For this one, please replace the order of lines 676 and 677 in sys/kern/kern_exec.c, that is make it be VOP_UNLOCK(imgp->vp, 0); setugidsafety(td); instead of VOP_UNLOCK(imgp->vp, 0); setugidsafety(td); -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20090730/f55be3d3/attachment.pgp From vinnix.bsd at gmail.com Thu Jul 30 13:21:52 2009 From: vinnix.bsd at gmail.com (Vinicius Abrahao) Date: Thu Jul 30 13:21:58 2009 Subject: Can'not mount "sysid 15 (0x0f),(Extended DOS (LBA))" at da0s2 Message-ID: <1e31c7980907300554k46ab5a70mf0ed36f6ab7f2cd9@mail.gmail.com> Hello friends, I was create a new "FAT32" partition with Acronis Disk Director[1] at the same place where in the past I have a UFS partition. Now when I try to mount this new partiton I get this error: # mount /dev/da0s2 /mnt/usb1 mount: /dev/da0s2 : Invalid argument # mount -t msdosfs /dev/da0s2 /mnt/usb2 mount_msdosfs: /dev/da0s2: Invalid argument I wonder that the "strange thing" is the sysid 15 for this partition, no the sysid 12 found in another usb disk that I have here. Could you help me with this trouble? Thanks so much, Vinnix [1]: http://www.acronis.com/homecomputing/products/diskdirector/ [2]: # fdisk da0 ******* Working on device /dev/da0 ******* parameters extracted from in-core disklabel are: cylinders=9729 heads=255 sectors/track=63 (16065 blks/cyl) Figures below won't work with BIOS for partitions not in cyl 1 parameters to be used for BIOS calculations are: cylinders=9729 heads=255 sectors/track=63 (16065 blks/cyl) Media sector size is 512 Warning: BIOS sector numbering starts with sector 1 Information from DOS bootblock is: The data for partition 1 is: sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD) start 63, size 41929587 (20473 Meg), flag 80 (active) beg: cyl 0/ head 1/ sector 1; end: cyl 1023/ head 254/ sector 63 The data for partition 2 is: sysid 15 (0x0f),(Extended DOS (LBA)) start 41929650, size 114366735 (55843 Meg), flag 0 beg: cyl 1023/ head 0/ sector 1; end: cyl 1023/ head 254/ sector 63 The data for partition 3 is: The data for partition 4 is: [3]: # file -s /dev/da* /dev/da0: x86 boot sector; partition 1: ID=0xa5, active, starthead 1, startsector 63, 41929587 sectors; partition 2: ID=0xf, starthead 0, startsector 41929650, 114366735 sectors, code offset 0x31 /dev/da0s1: Unix Fast File system [v2] (little-endian) last mounted on /u02, last written at Wed Jul 1 19:12:35 2009, clean flag 1, readonly flag 0, number of blocks 10482396, number of data blocks 10150835, number of cylinder groups 112, block size 16384, fragment size 2048, average file size 16384, average number of files in dir 64, pending blocks to free 0, pending inodes to free 0, system-wide uuid 0, minimum percentage of free blocks 8, TIME optimization /dev/da0s1a: Unix Fast File system [v2] (little-endian) last mounted on /, last written at Tue Jul 28 13:06:51 2009, clean flag 1, readonly flag 0, number of blocks 10482392, number of data blocks 10150831, number of cylinder groups 112, block size 16384, fragment size 2048, average file size 16384, average number of files in dir 64, pending blocks to free 0, pending inodes to free 0, system-wide uuid 0, minimum percentage of free blocks 8, TIME optimization /dev/da0s2: x86 boot sector; partition 1: ID=0xb, starthead 1, startsector 63, 114366672 sectors, extended partition table (last)\011, code offset 0x0, BSD disklabel [4]: # uname -a FreeBSD vinnix.corp.triarius.com.br 8.0-BETA2 FreeBSD 8.0-BETA2 #9: Tue Jul 21 21:28:19 BRT 2009 root@vinnix.corp.triarius.com.br:/usr/obj/usr/src/sys/VINNIX amd64 From spambox at haruhiism.net Thu Jul 30 13:25:21 2009 From: spambox at haruhiism.net (Kamigishi Rei) Date: Thu Jul 30 13:25:28 2009 Subject: Can'not mount "sysid 15 (0x0f),(Extended DOS (LBA))" at da0s2 In-Reply-To: <1e31c7980907300554k46ab5a70mf0ed36f6ab7f2cd9@mail.gmail.com> References: <1e31c7980907300554k46ab5a70mf0ed36f6ab7f2cd9@mail.gmail.com> Message-ID: <4A719B34.3000203@haruhiism.net> Vinicius Abrahao wrote: > I was create a new "FAT32" partition with Acronis Disk Director[1] at > the same place > where in the past I have a UFS partition. > Now when I try to mount this new partiton I get this error: > > # mount /dev/da0s2 /mnt/usb1 > mount: /dev/da0s2 : Invalid argument > > # mount -t msdosfs /dev/da0s2 /mnt/usb2 > mount_msdosfs: /dev/da0s2: Invalid argument > > The data for partition 2 is: > sysid 15 (0x0f),(Extended DOS (LBA)) da0s2 is an Extended DOS partition. It has no VFAT tables whatsoever, it's just a reference to the logical drive table later. Try using da0s5, for example, or just check the list of da0s* devices available. -- Kamigishi Rei KREI-RIPE From serenity at exscape.org Thu Jul 30 13:32:05 2009 From: serenity at exscape.org (Thomas Backman) Date: Thu Jul 30 13:32:56 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <4A719CA4.4060400@freebsd.org> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> <4A708455.5070304@freebsd.org> <86983A55-E5C4-4C04-A4C7-0AE9A9EE37A3@exscape.org> <4A718E03.6030909@freebsd.org> <71A038EC-02B1-4606-96C2-5E84BE80F005@exscape.org> <4A719CA4.4060400@freebsd.org> Message-ID: <19347561-3CE6-40B3-930A-EB9925D3AFD1@exscape.org> On Jul 30, 2009, at 15:14, Andriy Gapon wrote: > Thomas, > > I wasn't clear - please make sure that you have original > zfs_inactive (without the > changes that Pawel proposed) with the only change zfs_znode_free -> > vrecycle. > I.e.: > if (zp->z_dbuf == NULL) { > /* > * The fs has been unmounted, or we did a > * suspend/resume and this file no longer exists. > */ > mutex_enter(&zp->z_lock); > VI_LOCK(vp); > vp->v_count = 0; /* count arrives as 1 */ > mutex_exit(&zp->z_lock); > rw_exit(&zfsvfs->z_teardown_inactive_lock); > vrecycle(vp, curthread); > return; > } > > I believe that the latest panic is a direct result of ZTOV(zp) = > NULL line > introduced in zfs_vnops.c.2.patch. > > reclaim function should stay patched with Pawel's patch. Hey, it works!!! :) For the first time ever, my now mislabeled "clone_crash.sh" doesn't panic! A quick test of my ordinary, actually-used backup script also worked fine! For the record, here's the diff I got: http://exscape.org/temp/zfs_vnops.c.patch Thanks a lot! Hope to see this tested further (I'll do some more testing for sure) so that we can consider it a stable change. Regards, Thomas From avg at freebsd.org Thu Jul 30 13:35:17 2009 From: avg at freebsd.org (Andriy Gapon) Date: Thu Jul 30 13:35:24 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <19347561-3CE6-40B3-930A-EB9925D3AFD1@exscape.org> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> <4A708455.5070304@freebsd.org> <86983A55-E5C4-4C04-A4C7-0AE9A9EE37A3@exscape.org> <4A718E03.6030909@freebsd.org> <71A038EC-02B1-4606-96C2-5E84BE80F005@exscape.org> <4A719CA4.4060400@freebsd.org> <19347561-3CE6-40B3-930A-EB9925D3AFD1@exscape.org> Message-ID: <4A71A191.1080307@freebsd.org> on 30/07/2009 16:31 Thomas Backman said the following: > Hey, it works!!! :) > For the first time ever, my now mislabeled "clone_crash.sh" doesn't > panic! A quick test of my ordinary, actually-used backup script also > worked fine! > > For the record, here's the diff I got: > http://exscape.org/temp/zfs_vnops.c.patch > > Thanks a lot! Hope to see this tested further (I'll do some more testing > for sure) so that we can consider it a stable change. Very good! Thank you for all the testing and debugging feedback! And for your patience and persistence too :-) -- Andriy Gapon From jhb at freebsd.org Thu Jul 30 13:52:12 2009 From: jhb at freebsd.org (John Baldwin) Date: Thu Jul 30 13:52:25 2009 Subject: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) In-Reply-To: <20090730092507.GF1884@deviant.kiev.zoral.com.ua> References: <200907271400.n6RE05Rv056472@freefall.freebsd.org> <20090730092507.GF1884@deviant.kiev.zoral.com.ua> Message-ID: <200907300845.27663.jhb@freebsd.org> On Thursday 30 July 2009 5:25:07 am Kostik Belousov wrote: > On Thu, Jul 30, 2009 at 11:05:32AM +0200, Rene Ladan wrote: > > 2009/7/29 John Baldwin : > > > On Wednesday 29 July 2009 11:20:21 am Rene Ladan wrote: > > >> 2009/7/29 John Baldwin : > > >> > On Wednesday 29 July 2009 5:52:24 am Rene Ladan wrote: > > >> >> 2009/7/28 John Baldwin : > > >> >> > On Tuesday 28 July 2009 10:03:40 am Rene Ladan wrote: > > >> >> >> 2009/7/28 John Baldwin : > > >> >> >> > On Monday 27 July 2009 10:00:05 am Rene Ladan wrote: > > >> >> >> >> The following reply was made to PR kern/136945; it has been noted > > > by > > >> >> > GNATS. > > >> >> >> >> > > >> >> >> >> From: Rene Ladan > > >> >> >> >> To: John Baldwin > > >> >> >> >> Cc: bug-followup@freebsd.org > > >> >> >> >> Subject: Re: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) > > >> >> >> >> Date: Mon, 27 Jul 2009 15:51:15 +0200 > > >> >> >> >> > > >> >> >> >> ?2009/7/27 John Baldwin : > > >> >> >> >> ?> I would actually expect this to be the correct order for these > > > two > > >> >> >> > locks.= > > >> >> >> >> ? =A0Can > > >> >> >> >> ?> you capture the output of the 'debug.witness.fullgraph' sysctl > > > to a > > >> >> > file? > > >> >> >> >> ?> > > >> >> >> >> ?Yes, see attachment. ?I'm still running the same 8.0-BETA2. > > >> >> >> > > > >> >> >> > Hmm, the attachment was eaten by a grue, can you post the file > > >> > somewhere? > > >> >> >> > > > >> >> >> Yes, see ftp://rene-ladan.nl/pub/freebsd/kern_136945.txt > > >> >> > > > >> >> > Ok, it looks like it did encounter a UFS -> filedesc order at some > > >> > point. ?Can > > >> >> > you patch sys/kern/subr_witness.c to add a section to the order_lists[] > > >> > array > > >> >> > after the 'ZFS locking list' and before the spin locks list that looks > > >> > like > > >> >> > this: > > >> >> > > > >> >> > ? ? ? ?{ "filedesc structure", &lock_class_sx }, > > >> >> > ? ? ? ?{ "ufs", &lock_class_lockmgr}, > > >> >> > ? ? ? ?{ NULL, NULL }, > > >> >> > > > >> >> The LOR seems to be gone, previously it showed up only once right > > >> >> after booting the system. > > >> >> > > >> >> But now a new LOR (according to the LOR page) seems pop up: > > >> >> Trying to mount root from ufs:/dev/ad0s1a > > >> >> lock order reversal: > > >> >> ?1st 0xffffff0002a4ad80 ufs (ufs) > > > @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1465 > > >> >> ?2nd 0xffffff0002b29a48 filedesc structure (filedesc structure) @ > > >> >> /usr/src/sys/kern/kern_descrip.c:2478 > > >> >> KDB: stack backtrace: > > >> >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > > >> >> _witness_debugger() at _witness_debugger+0x49 > > >> >> witness_checkorder() at witness_checkorder+0x7ea > > >> >> _sx_xlock() at _sx_xlock+0x44 > > >> >> mountcheckdirs() at mountcheckdirs+0x80 > > >> >> vfs_donmount() at vfs_donmount+0xfbf > > >> >> kernel_mount() at kernel_mount+0xa1 > > >> >> vfs_mountroot_try() at vfs_mountroot_try+0x177 > > >> >> vfs_mountroot() at vfs_mountroot+0x47d > > >> >> start_init() at start_init+0x62 > > >> >> fork_exit() at fork_exit+0x12a > > >> >> fork_trampoline() at fork_trampoline+0xe > > >> >> --- trap 0, rip = 0, rsp = 0xffffff800001ad30, rbp = 0 --- > > >> >> > > >> >> The output of `df' and `mount' looks ok. > > >> > > > >> > Yes, this is the "real" LOR as "filedesc" -> "ufs" in the poll() case > > > should > > >> > be the normal order. ?I believe this should fix it. ?mountcheckdirs() > > > doesn't > > >> > need the vnodes locked, it just needs the caller to hold references on > > > them > > >> > so they aren't recycled: > > >> > > > >> > --- //depot/projects/smpng/sys/kern/vfs_mount.c#96 > > >> > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c > > >> > @@ -1069,9 +1069,10 @@ > > >> > ? ? ? ? ? ? ? ?vfs_event_signal(NULL, VQ_MOUNT, 0); > > >> > ? ? ? ? ? ? ? ?if (VFS_ROOT(mp, LK_EXCLUSIVE, &newdp)) > > >> > ? ? ? ? ? ? ? ? ? ? ? ?panic("mount: lost mount"); > > >> > + ? ? ? ? ? ? ? VOP_UNLOCK(newdp, 0); > > >> > + ? ? ? ? ? ? ? VOP_UNLOCK(vp, 0); > > >> > ? ? ? ? ? ? ? ?mountcheckdirs(vp, newdp); > > >> > - ? ? ? ? ? ? ? vput(newdp); > > >> > - ? ? ? ? ? ? ? VOP_UNLOCK(vp, 0); > > >> > + ? ? ? ? ? ? ? vrele(newdp); > > >> > ? ? ? ? ? ? ? ?if ((mp->mnt_flag & MNT_RDONLY) == 0) > > >> > ? ? ? ? ? ? ? ? ? ? ? ?error = vfs_allocate_syncvnode(mp); > > >> > ? ? ? ? ? ? ? ?vfs_unbusy(mp); > > >> > > > >> The LOR is still present, but at a different place without the > > >> mountcheckdirs() call (not on the LOR page either) : > > > > > > Ok, try this patch as well: > > > > > > --- //depot/projects/smpng/sys/kern/vfs_mount.c#97 > > > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c > > > @@ -1481,6 +1481,8 @@ > > > ? ? ? ?if (VFS_ROOT(TAILQ_FIRST(&mountlist), LK_EXCLUSIVE, &rootvnode)) > > > ? ? ? ? ? ? ? ?panic("Cannot find root vnode"); > > > > > > + ? ? ? VOP_UNLOCK(rootvnode, 0); > > > + > > > ? ? ? ?p = curthread->td_proc; > > > ? ? ? ?FILEDESC_XLOCK(p->p_fd); > > > > > > @@ -1496,8 +1498,6 @@ > > > > > > ? ? ? ?FILEDESC_XUNLOCK(p->p_fd); > > > > > > - ? ? ? VOP_UNLOCK(rootvnode, 0); > > > - > > > ? ? ? ?EVENTHANDLER_INVOKE(mountroot); > > > ?} > > > > > > > Still no luck, I now get a LOR that is similar to LOR 281 right after booting: > > > > lock order reversal: > > 1st 0xffffff0002c2c7f8 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2083 > > 2nd 0xffffff0002b2a248 filedesc structure (filedesc structure) @ > > /usr/src/sys/kern/vfs_syscalls.c:3776 > > KDB: stack backtrace: > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > > _witness_debugger() at _witness_debugger+0x49 > > witness_checkorder() at witness_checkorder+0x7ea > > _sx_slock() at _sx_slock+0x44 > > kern_mkdirat() at kern_mkdirat+0x201 > > syscall() at syscall+0x1af > > Xfast_syscall() at Xfast_syscall+0xe1 > > --- syscall (136, FreeBSD ELF64, mkdir), rip = 0x800729dac, rsp = > > 0x7fffffffec88, rbp = 0x7fffffffef66 --- > > Remove the FILEDESC_SLOCK()/FILEDESC_SUNLOCK() calls from kern_mkdirat(). Several other system calls have the same LOR and need the same fix. I've consolidated all the fixes so far into http://www.FreeBSD.org/~jhb/patches/vnode_filedesc.patch -- John Baldwin From jhb at freebsd.org Thu Jul 30 13:52:12 2009 From: jhb at freebsd.org (John Baldwin) Date: Thu Jul 30 13:52:25 2009 Subject: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) In-Reply-To: References: <200907271400.n6RE05Rv056472@freefall.freebsd.org> <20090730092507.GF1884@deviant.kiev.zoral.com.ua> Message-ID: <200907300951.44364.jhb@freebsd.org> On Thursday 30 July 2009 8:55:48 am Rene Ladan wrote: > 2009/7/30 Kostik Belousov : > > On Thu, Jul 30, 2009 at 11:05:32AM +0200, Rene Ladan wrote: > >> 2009/7/29 John Baldwin : > >> > On Wednesday 29 July 2009 11:20:21 am Rene Ladan wrote: > >> >> 2009/7/29 John Baldwin : > >> >> > On Wednesday 29 July 2009 5:52:24 am Rene Ladan wrote: > >> >> >> 2009/7/28 John Baldwin : > >> >> >> > On Tuesday 28 July 2009 10:03:40 am Rene Ladan wrote: > >> >> >> >> 2009/7/28 John Baldwin : > >> >> >> >> > On Monday 27 July 2009 10:00:05 am Rene Ladan wrote: > >> >> >> >> >> The following reply was made to PR kern/136945; it has been noted > >> > by > >> >> >> > GNATS. > >> >> >> >> >> > >> >> >> >> >> From: Rene Ladan > >> >> >> >> >> To: John Baldwin > >> >> >> >> >> Cc: bug-followup@freebsd.org > >> >> >> >> >> Subject: Re: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) > >> >> >> >> >> Date: Mon, 27 Jul 2009 15:51:15 +0200 > >> >> >> >> >> > >> >> >> >> >> ?2009/7/27 John Baldwin : > >> >> >> >> >> ?> I would actually expect this to be the correct order for these > >> > two > >> >> >> >> > locks.= > >> >> >> >> >> ? =A0Can > >> >> >> >> >> ?> you capture the output of the 'debug.witness.fullgraph' sysctl > >> > to a > >> >> >> > file? > >> >> >> >> >> ?> > >> >> >> >> >> ?Yes, see attachment. ?I'm still running the same 8.0-BETA2. > >> >> >> >> > > >> >> >> >> > Hmm, the attachment was eaten by a grue, can you post the file > >> >> > somewhere? > >> >> >> >> > > >> >> >> >> Yes, see ftp://rene-ladan.nl/pub/freebsd/kern_136945.txt > >> >> >> > > >> >> >> > Ok, it looks like it did encounter a UFS -> filedesc order at some > >> >> > point. ?Can > >> >> >> > you patch sys/kern/subr_witness.c to add a section to the order_lists[] > >> >> > array > >> >> >> > after the 'ZFS locking list' and before the spin locks list that looks > >> >> > like > >> >> >> > this: > >> >> >> > > >> >> >> > ? ? ? ?{ "filedesc structure", &lock_class_sx }, > >> >> >> > ? ? ? ?{ "ufs", &lock_class_lockmgr}, > >> >> >> > ? ? ? ?{ NULL, NULL }, > >> >> >> > > >> >> >> The LOR seems to be gone, previously it showed up only once right > >> >> >> after booting the system. > >> >> >> > >> >> >> But now a new LOR (according to the LOR page) seems pop up: > >> >> >> Trying to mount root from ufs:/dev/ad0s1a > >> >> >> lock order reversal: > >> >> >> ?1st 0xffffff0002a4ad80 ufs (ufs) > >> > @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1465 > >> >> >> ?2nd 0xffffff0002b29a48 filedesc structure (filedesc structure) @ > >> >> >> /usr/src/sys/kern/kern_descrip.c:2478 > >> >> >> KDB: stack backtrace: > >> >> >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > >> >> >> _witness_debugger() at _witness_debugger+0x49 > >> >> >> witness_checkorder() at witness_checkorder+0x7ea > >> >> >> _sx_xlock() at _sx_xlock+0x44 > >> >> >> mountcheckdirs() at mountcheckdirs+0x80 > >> >> >> vfs_donmount() at vfs_donmount+0xfbf > >> >> >> kernel_mount() at kernel_mount+0xa1 > >> >> >> vfs_mountroot_try() at vfs_mountroot_try+0x177 > >> >> >> vfs_mountroot() at vfs_mountroot+0x47d > >> >> >> start_init() at start_init+0x62 > >> >> >> fork_exit() at fork_exit+0x12a > >> >> >> fork_trampoline() at fork_trampoline+0xe > >> >> >> --- trap 0, rip = 0, rsp = 0xffffff800001ad30, rbp = 0 --- > >> >> >> > >> >> >> The output of `df' and `mount' looks ok. > >> >> > > >> >> > Yes, this is the "real" LOR as "filedesc" -> "ufs" in the poll() case > >> > should > >> >> > be the normal order. ?I believe this should fix it. ?mountcheckdirs() > >> > doesn't > >> >> > need the vnodes locked, it just needs the caller to hold references on > >> > them > >> >> > so they aren't recycled: > >> >> > > >> >> > --- //depot/projects/smpng/sys/kern/vfs_mount.c#96 > >> >> > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c > >> >> > @@ -1069,9 +1069,10 @@ > >> >> > ? ? ? ? ? ? ? ?vfs_event_signal(NULL, VQ_MOUNT, 0); > >> >> > ? ? ? ? ? ? ? ?if (VFS_ROOT(mp, LK_EXCLUSIVE, &newdp)) > >> >> > ? ? ? ? ? ? ? ? ? ? ? ?panic("mount: lost mount"); > >> >> > + ? ? ? ? ? ? ? VOP_UNLOCK(newdp, 0); > >> >> > + ? ? ? ? ? ? ? VOP_UNLOCK(vp, 0); > >> >> > ? ? ? ? ? ? ? ?mountcheckdirs(vp, newdp); > >> >> > - ? ? ? ? ? ? ? vput(newdp); > >> >> > - ? ? ? ? ? ? ? VOP_UNLOCK(vp, 0); > >> >> > + ? ? ? ? ? ? ? vrele(newdp); > >> >> > ? ? ? ? ? ? ? ?if ((mp->mnt_flag & MNT_RDONLY) == 0) > >> >> > ? ? ? ? ? ? ? ? ? ? ? ?error = vfs_allocate_syncvnode(mp); > >> >> > ? ? ? ? ? ? ? ?vfs_unbusy(mp); > >> >> > > >> >> The LOR is still present, but at a different place without the > >> >> mountcheckdirs() call (not on the LOR page either) : > >> > > >> > Ok, try this patch as well: > >> > > >> > --- //depot/projects/smpng/sys/kern/vfs_mount.c#97 > >> > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c > >> > @@ -1481,6 +1481,8 @@ > >> > ? ? ? ?if (VFS_ROOT(TAILQ_FIRST(&mountlist), LK_EXCLUSIVE, &rootvnode)) > >> > ? ? ? ? ? ? ? ?panic("Cannot find root vnode"); > >> > > >> > + ? ? ? VOP_UNLOCK(rootvnode, 0); > >> > + > >> > ? ? ? ?p = curthread->td_proc; > >> > ? ? ? ?FILEDESC_XLOCK(p->p_fd); > >> > > >> > @@ -1496,8 +1498,6 @@ > >> > > >> > ? ? ? ?FILEDESC_XUNLOCK(p->p_fd); > >> > > >> > - ? ? ? VOP_UNLOCK(rootvnode, 0); > >> > - > >> > ? ? ? ?EVENTHANDLER_INVOKE(mountroot); > >> > ?} > >> > > >> > >> Still no luck, I now get a LOR that is similar to LOR 281 right after booting: > >> > >> lock order reversal: > >> ?1st 0xffffff0002c2c7f8 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2083 > >> ?2nd 0xffffff0002b2a248 filedesc structure (filedesc structure) @ > >> /usr/src/sys/kern/vfs_syscalls.c:3776 > >> KDB: stack backtrace: > >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > >> _witness_debugger() at _witness_debugger+0x49 > >> witness_checkorder() at witness_checkorder+0x7ea > >> _sx_slock() at _sx_slock+0x44 > >> kern_mkdirat() at kern_mkdirat+0x201 > >> syscall() at syscall+0x1af > >> Xfast_syscall() at Xfast_syscall+0xe1 > >> --- syscall (136, FreeBSD ELF64, mkdir), rip = 0x800729dac, rsp = > >> 0x7fffffffec88, rbp = 0x7fffffffef66 --- > > > > Remove the FILEDESC_SLOCK()/FILEDESC_SUNLOCK() calls from kern_mkdirat(). > > > I removed the two lines at sys/kern/vfs_syscalls.c (3776 and 3778), > but there still seem to be some LORs > (attached). The two LORs about the reboot call are from before Kostiks patch. Kostik has already suggested a fix for the second one. The first one is a bit harder to fix. :-/ The third one is a false LOR that you can ignore. -- John Baldwin From avg at freebsd.org Thu Jul 30 14:24:47 2009 From: avg at freebsd.org (Andriy Gapon) Date: Thu Jul 30 14:24:53 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <19347561-3CE6-40B3-930A-EB9925D3AFD1@exscape.org> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> <4A708455.5070304@freebsd.org> <86983A55-E5C4-4C04-A4C7-0AE9A9EE37A3@exscape.org> <4A718E03.6030909@freebsd.org> <71A038EC-02B1-4606-96C2-5E84BE80F005@exscape.org> <4A719CA4.4060400@freebsd.org> <19347561-3CE6-40B3-930A-EB9925D3AFD1@exscape.org> Message-ID: <4A71AD29.10705@freebsd.org> Could you please add DEBUG_VFS_LOCKS to kernel config and check that we haven't broke VFS locking with the patch? Thank you again! -- Andriy Gapon From rene at freebsd.org Thu Jul 30 14:38:29 2009 From: rene at freebsd.org (Rene Ladan) Date: Thu Jul 30 14:38:36 2009 Subject: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) In-Reply-To: <20090730132121.GH1884@deviant.kiev.zoral.com.ua> References: <200907271400.n6RE05Rv056472@freefall.freebsd.org> <200907290742.20838.jhb@freebsd.org> <200907291135.17569.jhb@freebsd.org> <20090730092507.GF1884@deviant.kiev.zoral.com.ua> <20090730132121.GH1884@deviant.kiev.zoral.com.ua> Message-ID: 2009/7/30 Kostik Belousov : > On Thu, Jul 30, 2009 at 02:55:48PM +0200, Rene Ladan wrote: >> >> FreeBSD 8.0-BETA2 #3: Thu Jul 30 13:29:46 CEST 2009 >> >> lock order reversal: >> ?1st 0xffffff00510a5d80 ufs (ufs) @ /usr/src/sys/kern/kern_exec.c:570 >> ?2nd 0xffffff0002dfe248 filedesc structure (filedesc structure) @ /usr/src/sys/kern/kern_descrip.c:1864 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a >> _witness_debugger() at _witness_debugger+0x49 >> witness_checkorder() at witness_checkorder+0x7ea >> _sx_xlock() at _sx_xlock+0x44 >> setugidsafety() at setugidsafety+0x40 >> kern_execve() at kern_execve+0xf22 >> execve() at execve+0x38 >> syscall() at syscall+0x1af >> Xfast_syscall() at Xfast_syscall+0xe1 >> --- syscall (59, FreeBSD ELF64, execve), rip = 0x8007c3d0c, rsp = 0x7fffffffec48, rbp = 0x7fffffffed50 --- > > For this one, please replace the order of lines 676 and 677 in > sys/kern/kern_exec.c, that is make it be > ? ? ? ? ? ? ? ?VOP_UNLOCK(imgp->vp, 0); > ? ? ? ? ? ? ? ?setugidsafety(td); > instead of > ? ? ? ? ? ? ? ?VOP_UNLOCK(imgp->vp, 0); > ? ? ? ? ? ? ? ?setugidsafety(td); > This patch seems to solve the LORs I got right after boot (only one about wpi is left which I already reported). Ren? From serenity at exscape.org Thu Jul 30 14:39:52 2009 From: serenity at exscape.org (Thomas Backman) Date: Thu Jul 30 14:39:58 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <4A71AD29.10705@freebsd.org> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> <4A708455.5070304@freebsd.org> <86983A55-E5C4-4C04-A4C7-0AE9A9EE37A3@exscape.org> <4A718E03.6030909@freebsd.org> <71A038EC-02B1-4606-96C2-5E84BE80F005@exscape.org> <4A719CA4.4060400@freebsd.org> <19347561-3CE6-40B3-930A-EB9925D3AFD1@exscape.org> <4A71AD29.10705@freebsd.org> Message-ID: <7544AED1-1216-4A24-B287-F54117641F76@exscape.org> On Jul 30, 2009, at 16:24, Andriy Gapon wrote: > > Could you please add DEBUG_VFS_LOCKS to kernel config and check that > we haven't > broke VFS locking with the patch? > Thank you again! > > -- > Andriy Gapon Hey, thank *you* :) Currently recompiling the kernel, I'll have a look later. What do I do, though? Just keep an eye on the console, or something more involved? (Or, since the handbook mentions lockedvnods in ddb: when should I check lockedvnods?) BTW: Could you (or anyone else with knowledge in these areas) have a look at the libzfs_sendrecv patch? Final piece of the puzzle as far as all the panics (well, core dump in this case) I've ran in to is concerned. http://lists.freebsd.org/pipermail/freebsd-current/2009-May/006814.html Or, in patch form (I think the intendation screws the patch up as linked there): http://exscape.org/temp/libzfs_sendrecv.patch Appears to be a pretty simple patch. I've tried writing a test case, but it's a bit of work to make it create separate pools, etc, so I'd rather skip that if possible. Without the patch, I can't get send -R - I (recursive + auto-incremental, i.e. you can do -I snap1 tank@snap4 instead of -i snap1 -i snap2 ...) to work without core dumping on the recv (sending to a file works just fine, but when receiving from the file, it core dumps; of course, the same is true for a pipe). Regards, Thomas From avg at freebsd.org Thu Jul 30 14:46:22 2009 From: avg at freebsd.org (Andriy Gapon) Date: Thu Jul 30 14:46:34 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <7544AED1-1216-4A24-B287-F54117641F76@exscape.org> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> <4A708455.5070304@freebsd.org> <86983A55-E5C4-4C04-A4C7-0AE9A9EE37A3@exscape.org> <4A718E03.6030909@freebsd.org> <71A038EC-02B1-4606-96C2-5E84BE80F005@exscape.org> <4A719CA4.4060400@freebsd.org> <19347561-3CE6-40B3-930A-EB9925D3AFD1@exscape.org> <4A71AD29.10705@freebsd.org> <7544AED1-1216-4A24-B287-F54117641F76@exscape.org> Message-ID: <4A71B239.8060007@freebsd.org> on 30/07/2009 17:39 Thomas Backman said the following: > On Jul 30, 2009, at 16:24, Andriy Gapon wrote: >> >> Could you please add DEBUG_VFS_LOCKS to kernel config and check that >> we haven't >> broke VFS locking with the patch? >> Thank you again! >> >> -- >> Andriy Gapon > Hey, thank *you* :) > Currently recompiling the kernel, I'll have a look later. What do I do, > though? Just keep an eye on the console, or something more involved? > (Or, since the handbook mentions lockedvnods in ddb: when should I check > lockedvnods?) I think you should get a panic if anything goes wrong. And I think you would get one :-) Next thing to try after that is the updated patch from Pawel: http://people.freebsd.org/~pjd/patches/zfs_vnops.c.2.patch -- Andriy Gapon From avg at freebsd.org Thu Jul 30 14:49:03 2009 From: avg at freebsd.org (Andriy Gapon) Date: Thu Jul 30 14:49:15 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <7544AED1-1216-4A24-B287-F54117641F76@exscape.org> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> <4A708455.5070304@freebsd.org> <86983A55-E5C4-4C04-A4C7-0AE9A9EE37A3@exscape.org> <4A718E03.6030909@freebsd.org> <71A038EC-02B1-4606-96C2-5E84BE80F005@exscape.org> <4A719CA4.4060400@freebsd.org> <19347561-3CE6-40B3-930A-EB9925D3AFD1@exscape.org> <4A71AD29.10705@freebsd.org> <7544AED1-1216-4A24-B287-F54117641F76@exscape.org> Message-ID: <4A71B2DA.9060902@freebsd.org> on 30/07/2009 17:39 Thomas Backman said the following: > Or, in patch form (I think the intendation screws the patch up as linked > there): > http://exscape.org/temp/libzfs_sendrecv.patch One comment on the patch - I personally don't like bit-wise xor in a logical expression. But if otherwise the expression would be huge and ugly, then OK. -- Andriy Gapon From serenity at exscape.org Thu Jul 30 15:28:34 2009 From: serenity at exscape.org (Thomas Backman) Date: Thu Jul 30 15:28:51 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <4A71B239.8060007@freebsd.org> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> <4A708455.5070304@freebsd.org> <86983A55-E5C4-4C04-A4C7-0AE9A9EE37A3@exscape.org> <4A718E03.6030909@freebsd.org> <71A038EC-02B1-4606-96C2-5E84BE80F005@exscape.org> <4A719CA4.4060400@freebsd.org> <19347561-3CE6-40B3-930A-EB9925D3AFD1@exscape.org> <4A71AD29.10705@freebsd.org> <7544AED1-1216-4A24-B287-F54117641F76@exscape.org> <4 A71B239.8060007@freebsd.org> Message-ID: <3AA3C1CB-CEF7-46CC-A9C7-1648093D679E@exscape.org> On Jul 30, 2009, at 16:46, Andriy Gapon wrote: > on 30/07/2009 17:39 Thomas Backman said the following: >> On Jul 30, 2009, at 16:24, Andriy Gapon wrote: >>> >>> Could you please add DEBUG_VFS_LOCKS to kernel config and check that >>> we haven't >>> broke VFS locking with the patch? >>> Thank you again! >>> >>> -- >>> Andriy Gapon >> Hey, thank *you* :) >> Currently recompiling the kernel, I'll have a look later. What do I >> do, >> though? Just keep an eye on the console, or something more involved? >> (Or, since the handbook mentions lockedvnods in ddb: when should I >> check >> lockedvnods?) > > I think you should get a panic if anything goes wrong. > And I think you would get one :-) > > Next thing to try after that is the updated patch from Pawel: > http://people.freebsd.org/~pjd/patches/zfs_vnops.c.2.patch Well, damnit! KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a vfs_badlock() at vfs_badlock+0x95 VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0x4a vgonel() at vgonel+0x14d vrecycle() at vrecycle+0x8b zfs_freebsd_inactive() at zfs_freebsd_inactive+0x1a VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x6c vinactive() at vinactive+0x85 vput() at vput+0x1d8 dounmount() at dounmount+0x4af unmount() at unmount+0x3c8 syscall() at syscall+0x28f Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (22, FreeBSD ELF64, unmount), rip = 0x80104e9ec, rsp = 0x7fffffffaa98, rbp = 0x801223300 --- VOP_RECLAIM: 0xffffff007b0ff1d8 interlock is locked but should not be KDB: enter: lock violation panic: from debugger #9 0xffffffff8057eda7 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:224 #10 0xffffffff8036c8ad in kdb_enter (why=0xffffffff80613fd5 "vfslock", msg=0xa
) at cpufunc.h:63 #11 0xffffffff805c82fa in VOP_RECLAIM_APV (vop=0xffffffff80b8b220, a=0xffffff803ea09930) at vnode_if.c:1923 #12 0xffffffff803cbd5d in vgonel (vp=0xffffff007b0ff1d8) at vnode_if.h: 830 #13 0xffffffff803cc10b in vrecycle (vp=0xffffff007b0ff1d8, td=Variable "td" is not available. ) at /usr/src/sys/kern/vfs_subr.c:2504 #14 0xffffffff80b13a9a in zfs_freebsd_inactive () from /boot/kernel/ zfs.ko #15 0xffffffff805c842c in VOP_INACTIVE_APV (vop=0xffffffff80b8b220, a=0xffffff803ea099f0) at vnode_if.c:1863 #16 0xffffffff803cb435 in vinactive (vp=0xffffff007b0ff1d8, td=0xffffff007b2cd720) at vnode_if.h:807 #17 0xffffffff803cc788 in vput (vp=0xffffff007b0ff1d8) at /usr/src/sys/kern/vfs_subr.c:2257 #18 0xffffffff803c5a4f in dounmount (mp=0xffffff0002c438d0, flags=0, td=Variable "td" is not available. ) at /usr/src/sys/kern/vfs_mount.c:1333 #19 0xffffffff803c6058 in unmount (td=0xffffff007b2cd720, uap=0xffffff803ea09bf0) at /usr/src/sys/kern/vfs_mount.c:1174 #20 0xffffffff80598e7f in syscall (frame=0xffffff803ea09c80) at /usr/src/sys/amd64/amd64/trap.c:984 #21 0xffffffff8057f081 in Xfast_syscall () Happened at (or very close to!) the place of the original panic. :/ Regards, Thomas PS. I'll test Pawel's patch sometime after dinner. ;) From avg at freebsd.org Thu Jul 30 15:40:13 2009 From: avg at freebsd.org (Andriy Gapon) Date: Thu Jul 30 15:40:27 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <3AA3C1CB-CEF7-46CC-A9C7-1648093D679E@exscape.org> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> <4A708455.5070304@freebsd.org> <86983A55-E5C4-4C04-A4C7-0AE9A9EE37A3@exscape.org> <4A718E03.6030909@freebsd.org> <71A038EC-02B1-4606-96C2-5E84BE80F005@exscape.org> <4A719CA4.4060400@freebsd.org> <19347561-3CE6-40B3-930A-EB9925D3AFD1@exscape.org> <4A71AD29.10705@freebsd.org> <7544AED1-1216-4A24-B287-F54117641F76@exscape.org> <4 A71B239.8060007@freebsd.org> <3AA3C1CB-CEF7-46CC-A9C7-1648093D679E@exsca! pe.org> Message-ID: <4A71BED8.7050300@freebsd.org> on 30/07/2009 18:25 Thomas Backman said the following: > PS. I'll test Pawel's patch sometime after dinner. ;) I believe that you should get a perfect result with it. -- Andriy Gapon From serenity at exscape.org Thu Jul 30 16:41:44 2009 From: serenity at exscape.org (Thomas Backman) Date: Thu Jul 30 16:41:51 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <4A71BED8.7050300@freebsd.org> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> <4A708455.5070304@freebsd.org> <86983A55-E5C4-4C04-A4C7-0AE9A9EE37A3@exscape.org> <4A718E03.6030909@freebsd.org> <71A038EC-02B1-4606-96C2-5E84BE80F005@exscape.org> <4A719CA4.4060400@freebsd.org> <19347561-3CE6-40B3-930A-EB9925D3AFD1@exscape.org> <4A71AD29.10705@freebsd.org> <7544AED1-1216-4A24-B287-F54117641F76@exscape.org> <4 A71B239.8060007@freebsd.org> <3AA3C1CB-CEF7-46CC-A9C7-1648093D679E@exsca! pe.org> <4A71BED8.7050300@freebsd.org> Message-ID: On Jul 30, 2009, at 17:40, Andriy Gapon wrote: > on 30/07/2009 18:25 Thomas Backman said the following: >> PS. I'll test Pawel's patch sometime after dinner. ;) > > I believe that you should get a perfect result with it. > > -- > Andriy Gapon If I dare say it, you were right! I've been testing for about half an hour or so (probably a bit more) now. Still using DEBUG_VFS_LOCKS, and I've tried the test case several times, ran an initial backup (i.e. destroy target pool and send|recv the entire pool) and a few incrementals. Rebooted, tried it again. No panic, no problems! :) Let's hope it stays this way. So, in short: With that patch (copied here just in case: http://exscape.org/temp/zfs_vnops.working.patch ) and the libzfs patch linked previously, it appears zfs send/recv works plain fine. I have yet to try it with clone/promote and stuff, but since that gave the same panic that this solved, I'm hoping there will be no problems with that anymore. Regards, Thomas From serenity at exscape.org Thu Jul 30 18:29:38 2009 From: serenity at exscape.org (Thomas Backman) Date: Thu Jul 30 18:29:46 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: References: <20090727072503.GA52309@jpru.ffm.jpru.de> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> <4A708455.5070304@freebsd.org> <86983A55-E5C4-4C04-A4C7-0AE9A9EE37A3@exscape.org> <4A718E03.6030909@freebsd.org> <71A038EC-02B1-4606-96C2-5E84BE80F005@exscape.org> <4A719CA4.4060400@freebsd.org> <19347561-3CE6-40B3-930A-EB9925D3AFD1@exscape.org> <4A71AD29.10705@freebsd.org> <7544AED1-1216-4A24-B287-F54117641F76@exscape.org> <4 A71B239.8060007@freebsd.org> <3AA3C1CB-CEF7-46CC-A9C7-1648093D679E@exsca! pe.org> <4A71BED8.7050300@freebsd.org> Message-ID: On Jul 30, 2009, at 18:41, Thomas Backman wrote: > On Jul 30, 2009, at 17:40, Andriy Gapon wrote: >> on 30/07/2009 18:25 Thomas Backman said the following: >>> PS. I'll test Pawel's patch sometime after dinner. ;) >> >> I believe that you should get a perfect result with it. >> >> -- Andriy Gapon > If I dare say it, you were right! I've been testing for about half > an hour or so (probably a bit more) now. > Still using DEBUG_VFS_LOCKS, and I've tried the test case several > times, ran an initial backup (i.e. destroy target pool and send|recv > the entire pool) and a few incrementals. Rebooted, tried it again. > No panic, no problems! :) > Let's hope it stays this way. > > So, in short: With that patch (copied here just in case: http://exscape.org/temp/zfs_vnops.working.patch > ) and the libzfs patch linked previously, it appears zfs send/recv > works plain fine. I have yet to try it with clone/promote and stuff, > but since that gave the same panic that this solved, I'm hoping > there will be no problems with that anymore. Arrrgh! I guess I spoke too soon after all... new panic yet again. :( *sigh* It feels as if this will never become stable right now. (Maybe that's because I've spent all day and most of yesterday too on this ;) Steps and panic info: (Prior to this, I tried a simple zfs promote on one of my clones, and then reverted it by promoting the other FS again, with no problems on running the backup script.) [root@chaos ~]# zfs destroy -r tank/testfs [root@chaos ~]# bash backup.sh backup (all output is from zfs, on zfs send -R -I old tank@new | zfs recv - Fvd slave) attempting destroy slave/testfs@backup-20090730-2009 success attempting destroy slave/testfs@backup-20090730-1823 success attempting destroy slave/testfs@backup-20090730-1801 success attempting destroy slave/testfs@backup-20090730-2011 success attempting destroy slave/testfs@backup-20090730-1827 success attempting destroy slave/testfs success receiving incremental stream of tank@backup-20090730-2012 into slave@backup-20090730-2012 received 312B stream in 1 seconds (312B/sec) receiving incremental stream of tank/tmp@backup-20090730-2012 into slave/tmp@backup-20090730-2012 received 312B stream in 1 seconds (312B/sec) receiving incremental stream of tank/var@backup-20090730-2012 into slave/var@backup-20090730-2012 received 32.6KB stream in 1 seconds (32.6KB/sec) receiving incremental stream of tank/var/log@backup-20090730-2012 into slave/var/log@backup-20090730-2012 received 298KB stream in 1 seconds (298KB/sec) receiving incremental stream of tank/var/crash@backup-20090730-2012 into slave/var/crash@backup-20090730-2012 received 312B stream in 1 seconds (312B/sec) receiving incremental stream of tank/root@backup-20090730-2012 into slave/root@backup-20090730-2012 [... panic here ...] Unread portion of the kernel message buffer:panic: solaris assert: ((zp)->z_vnode)->v_usecount > 0, file: /usr/src/sys/modules/zfs/../../ cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c, line: 920 cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a panic() at panic+0x182 zfsvfs_teardown() at zfsvfs_teardown+0x24d zfs_suspend_fs() at zfs_suspend_fs+0x2b zfs_ioc_recv() at zfs_ioc_recv+0x28b zfsdev_ioctl() at zfsdev_ioctl+0x8a devfs_ioctl_f() at devfs_ioctl_f+0x77 kern_ioctl() at kern_ioctl+0xf6 ioctl() at ioctl+0xfd syscall() at syscall+0x28f Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe5f7c, rsp = 0x7fffffff8ef8, rbp = 0x7fffffff9c30 --- KDB: enter: panic panic: from debugger #9 0xffffffff8057eda7 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:224 #10 0xffffffff8036c8ad in kdb_enter (why=0xffffffff80609c44 "panic", msg=0xa
) at cpufunc.h:63 #11 0xffffffff8033abcb in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:558#12 0xffffffff80b0ec5d in zfsvfs_teardown () from /boot/kernel/zfs.ko#13 0x0000000000100000 in ?? () #14 0xffffff001bff0250 in ?? () #15 0xffffff001bff0000 in ?? () #16 0xffffff0008004000 in ?? () #17 0xffffff803e9747a0 in ?? () #18 0xffffff803e9747d0 in ?? () #19 0xffffff803e974770 in ?? () #20 0xffffff803e974740 in ?? () #21 0xffffffff80b0ecab in zfs_suspend_fs () from /boot/kernel/zfs.ko Previous frame inner to this frame (corrupt stack?) Unfortunately, I'm not sure I can reproduce this reliably, since it worked a bunch of times both before and after my previous mail. Oh, and I'm still using -DDEBUG=1 and DEBUG_VFS_LOCKS... If this isn't a new panic because of the changes, perhaps it was triggered now and never before because of the -DDEBUG? Regards, Thomas From avg at freebsd.org Thu Jul 30 18:42:20 2009 From: avg at freebsd.org (Andriy Gapon) Date: Thu Jul 30 18:42:32 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> <4A708455.5070304@freebsd.org> <86983A55-E5C4-4C04-A4C7-0AE9A9EE37A3@exscape.org> <4A718E03.6030909@freebsd.org> <71A038EC-02B1-4606-96C2-5E84BE80F005@exscape.org> <4A719CA4.4060400@freebsd.org> <19347561-3CE6-40B3-930A-EB9925D3AFD1@exscape.org> <4A71AD29.10705@freebsd.org> <7544AED1-1216-4A24-B287-F54117641F76@exscape.org> <4 A71B239.8060007@freebsd.org> <3AA3C1CB-CEF7-46CC-A9C7-1648093D679E@exsca! pe.org> <4A71BED8.7050300@freebsd.org> Message-ID: <4A71E986.9010800@freebsd.org> on 30/07/2009 21:29 Thomas Backman said the following: > > Unfortunately, I'm not sure I can reproduce this reliably, since it > worked a bunch of times both before and after my previous mail. > > Oh, and I'm still using -DDEBUG=1 and DEBUG_VFS_LOCKS... If this isn't a > new panic because of the changes, perhaps it was triggered now and never > before because of the -DDEBUG? Thomas, I am going on vacation, so no help from me for the next two weeks. Yes, if you get a panic in ASSERT in zfs code, then it's caught because of DEBUG=1. I can't say if you would get a different panic further on if DEBUG weren't enabled. Maybe Pawel can say if this ASSERT is a correct one (I've seen in the past some incorrect asserts, but not in ZFS code). -- Andriy Gapon From acc at hexadecagram.org Fri Jul 31 06:06:14 2009 From: acc at hexadecagram.org (Anthony Chavez) Date: Fri Jul 31 06:06:21 2009 Subject: Re-starting a gjournal provider In-Reply-To: <20090729140436.GG1586@garage.freebsd.pl> References: <4A62E0CE.1000508@hexadecagram.org> <20090729140436.GG1586@garage.freebsd.pl> Message-ID: <4A7289B9.2060907@hexadecagram.org> Thanks very much for responding, Pawel. I'm moving this discussion to freebsd-geom, which is where I probably should have posted in the first place. Lack of sleep and coffee on Sunday morning were partly to blame, I'm sure. ;-) Pawel Jakub Dawidek wrote: > On Sun, Jul 19, 2009 at 03:01:02AM -0600, Anthony Chavez wrote: >> Hello freebsd-fs, >> >> I'm trying to get gjournal working on a "removable" hard disk. I use >> the term loosely, because I'm using a very simple eSATA enclosure: an >> AMS Venus DS5 [1]. >> >> If I swap out disks, atacontrol cap ad0 seems sufficient enough to >> detect the new drive: the reported device model, serial number, firmware >> revision, and CHS values change as one would expect. >> >> My interpretation of [2] section 5.3 and gjournal(8) is that the >> following sequence of commands should ensure me that all write buffers >> have been flushed and bring the system to a point where it is safe to >> remove a disk. >> >> sync; sync; sync >> gjournal sync >> umount /dev/ad0s1.journal >> gjournal stop ad0s1.journal > > You should first unmount and then call 'gjournal sync'. Thank you for clarifying that. You mention this again later on in your response, and I respond below. >> However, once they are executed, /dev/ad0s1.journal disappears and when >> I swap out the disk it doesn't come back. The only way I've found to >> bring it back is atacontrol detach ata0; atacontrol attach ata0, which >> doesn't seem like a wise thing to do if I have another device on the >> same channel. > > It doesn't come back because something (ATA layer?) doesn't properly > remove ad0 provider. When you remove the disk, /dev/ad0 should disappear > and reappear once you insert it again. > > You can still do this trick after you insert the disk again so the GEOM > can schedule retaste: > > # true > /dev/ad0 Thank you for informing me of that trick. I tried using it after "gjournal stop" but unfortunately, nothing changed. My terminology might have been a bit off in my initial post (gjournal is still a bit new to me). So I will attempt to clarify a bit more. Here is an example of a typical session (the only difference this time being "gjournal sync" following umount as you prescribed). % sudo atacontrol info ata0 Master: ad0 SATA revision 1.x Slave: no device present % ls /dev/ad0* /dev/ad0 /dev/ad0s1 /dev/ad0s1.journal % mount | grep ad0s1 /dev/ad0s1.journal on /mnt/ad0s1 (ufs, local, gjournal) % ( subsh> set -o errexit subsh> sync subsh> sync subsh> sync subsh> sudo umount /dev/ad0s1.journal subsh> gjournal sync subsh> sudo gjournal stop ad0s1.journal subsh> ) % ls /dev/ad0* /dev/ad0 /dev/ad0s1 % sudo true \> /dev/ad0; ls /dev/ad0* /dev/ad0 /dev/ad0s1 % sudo true \> /dev/ad0s1; ls /dev/ad0* /dev/ad0 /dev/ad0s1 % sudo atacontrol detach ata0 && sudo atacontrol attach ata0 Master: ad0 SATA revision 1.x Slave: no device present % ls /dev/ad0* /dev/ad0 /dev/ad0s1 /dev/ad0s1.journal Here are the points to note. 1) When I physically remove a drive from the enclosure, /dev/ad0 does not disappear. /dev/ad0 *always* exists until I "atacontrol detach." Even when the device is powered off, /dev/ad0 continues to exist. 2) /dev/ad0s1.journal disappears when I "gjournal stop." /dev/ad0s1.journal is the device that, AFAIK, will only come back after "atacontrol detach ata0; atacontrol attach ata0". 3) When I swap drives, "atacontrol cap ad0" will produce a report for the newly-inserted drive. If I attempt to "atacontrol info ata0" before issuing that command, it continues to display the drive model and firmware revision from the drive that was previously inserted. However, "atacontrol cap" does not appear to provoke the return of /dev/ad0s1.journal. >> My question is, do I need to issue gjournal stop before I swap disks? >> And if so, is there any way that I can avoid the atacontrol >> detach/attach cycle that would need to take place before any mount is >> attempted so that /dev/ad0s1.journal appears (if in the drive inserted >> at the time does in fact utilize gjournal; I may want to experiment with >> having disks with either gjournal or soft updates)? This paragraph (above) and the one that that proceeded it in my original post contains the following 2 questions that remain unanswered (I've added another which was implied previously at best). 1) Is "atacontrol detach ata0 && atacontrol attach ata0" in fact a safe operation to perform in any circumstance? My better judgment has me thinking that the answer to this question is almost certainly "no." However, I am hypothesizing that it would safe enough if all devices on ata0 are properly unmounted first, but if I can avoid that, I will. It feels clumsy and seems to defeat the purpose of hot-swapping. 2) Is it *necessary* to "gjournal stop" before hot-swapping? In such a scenario, I would opt to simply "umount; gjournal sync," swap disks, and then "atacontrol cap ad0; mount" (or even just "mount"). It seems quite likely, however, that all drives that undergo this treatment would be *required* to have gjournal labels since /dev/ad0s1.journal would never disappear (although I've yet to actually test that). 3) If the answer to question 2 is "yes," then how can I handle the case of inserting a drive that does *not* have a gjournal label? >> And while I'm on the subject, are the (gjournal) syncs commands >> preceeding umount absolutely necessary in the case of removable media? > > 'gjournal sync' should follow unmount, not the other way around. And its > better to do it, but 'gjournal stop' should do the same. If that is indeed the case then the article I referenced as [2], "Implementing UFS Journaling on a Desktop PC," should be updated to reflect that ordering (section 5.3 prescribes a "umount" followed by "gjournal sync"). I'm submitting a PR that addresses this. In any case, the question I was asking here is actually twofold: 1) Is it really necessary to perform 3 "sync" commands before "umount"? Line 94 of src/sbin/umount/umount.c,v 1.45.20.1 has me thinking that the answer is "no," since it calls sync() itself, albeit only once. I got the idea for executing "sync" three times from /etc/rc.suspend. 2) Is it necessary to "gjournal sync" if I'm going to "gjournal stop" anyway? (You answered this one already.) Thank you for the assistance. -- Anthony Chavez http://hexadecagram.org/ mailto:acc@hexadecagram.org xmpp:acc@hexadecagram.org -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20090731/a76e024a/signature-0001.pgp From pjd at FreeBSD.org Fri Jul 31 06:49:32 2009 From: pjd at FreeBSD.org (Pawel Jakub Dawidek) Date: Fri Jul 31 06:49:39 2009 Subject: Re-starting a gjournal provider In-Reply-To: <4A7289B9.2060907@hexadecagram.org> References: <4A62E0CE.1000508@hexadecagram.org> <20090729140436.GG1586@garage.freebsd.pl> <4A7289B9.2060907@hexadecagram.org> Message-ID: <20090731064948.GG1584@garage.freebsd.pl> On Fri, Jul 31, 2009 at 12:05:45AM -0600, Anthony Chavez wrote: > > It doesn't come back because something (ATA layer?) doesn't properly > > remove ad0 provider. When you remove the disk, /dev/ad0 should disappear > > and reappear once you insert it again. > > > > You can still do this trick after you insert the disk again so the GEOM > > can schedule retaste: > > > > # true > /dev/ad0 > > Thank you for informing me of that trick. I tried using it after > "gjournal stop" but unfortunately, nothing changed. This is because it should be /dev/ad0s1 and not /dev/ad0. Try with /dev/ad0s1. > Here are the points to note. > > 1) When I physically remove a drive from the enclosure, /dev/ad0 does > not disappear. /dev/ad0 *always* exists until I "atacontrol detach." > Even when the device is powered off, /dev/ad0 continues to exist. This might be three things: 1. Your enclosure/controller doesn't report back about disk being removed. 2. Your enclosure does report back, but ATA ignores such report. This will be a bug in ATA. 3. Your controller doesn't support hot-swap or it supports warm-swap, which means you have to detach it by hand before removing it. > 2) /dev/ad0s1.journal disappears when I "gjournal stop." > /dev/ad0s1.journal is the device that, AFAIK, will only come back after > "atacontrol detach ata0; atacontrol attach ata0". It should also get back after 'true > /dev/ad0s1'. What this command do is to open provider for writing (it doesn't write anything). In GEOM it will trigger spoil event and then, once command completes, it will trigger retaste event. This mean that GEOM will inform gjournal to check /dev/ad0s1 again and this will allow gjournal to find its metadata and create /dev/ad0s1.journal once again. One more test would be in place. If you could try the command below before removing disk and after inserting different disk: # diskinfo -v /dev/ad0 If it shows exactly the same in two cases, it means that it is not aware that disk was replaced and detach/attach cycle is needed. > 1) Is "atacontrol detach ata0 && atacontrol attach ata0" in fact a safe > operation to perform in any circumstance? > > My better judgment has me thinking that the answer to this question is > almost certainly "no." However, I am hypothesizing that it would safe > enough if all devices on ata0 are properly unmounted first, but if I can > avoid that, I will. It feels clumsy and seems to defeat the purpose of > hot-swapping. It should be safe, but there were plenty of bugs related to disappearing disk from under mount file system, etc. If nothing is mounted you should be fine (if there are no ATA bugs in this area). But for full hot-swap the disk controller should discover disk being removed and ATA code should remove it from /dev/. > 2) Is it *necessary* to "gjournal stop" before hot-swapping? > > In such a scenario, I would opt to simply "umount; gjournal sync," swap > disks, and then "atacontrol cap ad0; mount" (or even just "mount"). It > seems quite likely, however, that all drives that undergo this treatment > would be *required* to have gjournal labels since /dev/ad0s1.journal > would never disappear (although I've yet to actually test that). I'd go with 'umount; gjournal stop' and drop 'gjournal sync'. Controler should inform ATA that disk is gone. ATA should inform GEOM that ad0 is gone. If that would be the case, simple 'umount; gjournal sync' will be enough. But because it isn't the case, you have to stop gjournal and detach ad0. > 3) If the answer to question 2 is "yes," then how can I handle the case > of inserting a drive that does *not* have a gjournal label? There's nothing special here. Let's see how diskinfo test will go first. > 1) Is it really necessary to perform 3 "sync" commands before "umount"? > > Line 94 of src/sbin/umount/umount.c,v 1.45.20.1 has me thinking that the > answer is "no," since it calls sync() itself, albeit only once. I got > the idea for executing "sync" three times from /etc/rc.suspend. The idea is that unmount should take case of syncing data. There should be not need for even one sync. It is called "just in case". > 2) Is it necessary to "gjournal sync" if I'm going to "gjournal stop" > anyway? (You answered this one already.) No, stop should be sufficient. -- Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20090731/415d0c4d/attachment.pgp From serenity at exscape.org Fri Jul 31 09:05:27 2009 From: serenity at exscape.org (Thomas Backman) Date: Fri Jul 31 09:05:39 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: References: <20090727072503.GA52309@jpru.ffm.jpru.de> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> <4A708455.5070304@freebsd.org> <86983A55-E5C4-4C04-A4C7-0AE9A9EE37A3@exscape.org> <4A718E03.6030909@freebsd.org> <71A038EC-02B1-4606-96C2-5E84BE80F005@exscape.org> <4A719CA4.4060400@freebsd.org> <19347561-3CE6-40B3-930A-EB9925D3AFD1@exscape.org> <4A71AD29.10705@freebsd.org> <7544AED1-1216-4A24-B287-F54117641F76@exscape.org> <4 A71B239.8060007@freebsd.org> <3AA3C1CB-CEF7-46CC-A9C7-1648093D679E@exsca! pe.org> <4A71BED8.7050300@freebsd.org> Message-ID: On Jul 30, 2009, at 20:29, Thomas Backman wrote: > On Jul 30, 2009, at 18:41, Thomas Backman wrote: > >> On Jul 30, 2009, at 17:40, Andriy Gapon wrote: >>> on 30/07/2009 18:25 Thomas Backman said the following: >>>> PS. I'll test Pawel's patch sometime after dinner. ;) >>> >>> I believe that you should get a perfect result with it. >>> >>> -- Andriy Gapon >> If I dare say it, you were right! I've been testing for about half >> an hour or so (probably a bit more) now. >> Still using DEBUG_VFS_LOCKS, and I've tried the test case several >> times, ran an initial backup (i.e. destroy target pool and send| >> recv the entire pool) and a few incrementals. Rebooted, tried it >> again. No panic, no problems! :) >> Let's hope it stays this way. >> >> So, in short: With that patch (copied here just in case: http://exscape.org/temp/zfs_vnops.working.patch >> ) and the libzfs patch linked previously, it appears zfs send/recv >> works plain fine. I have yet to try it with clone/promote and >> stuff, but since that gave the same panic that this solved, I'm >> hoping there will be no problems with that anymore. > > Arrrgh! > I guess I spoke too soon after all... new panic yet again. :( > *sigh* It feels as if this will never become stable right now. > (Maybe that's because I've spent all day and most of yesterday too > on this ;) > > Steps and panic info: > > (Prior to this, I tried a simple zfs promote on one of my clones, > and then reverted it by promoting the other FS again, with no > problems on running the backup script.) > > [root@chaos ~]# zfs destroy -r tank/testfs > [root@chaos ~]# bash backup.sh backup > (all output is from zfs, on zfs send -R -I old tank@new | zfs recv - > Fvd slave) > > attempting destroy slave/testfs@backup-20090730-2009 > success > attempting destroy slave/testfs@backup-20090730-1823 > success > attempting destroy slave/testfs@backup-20090730-1801 > success > attempting destroy slave/testfs@backup-20090730-2011 > success > attempting destroy slave/testfs@backup-20090730-1827 > success > attempting destroy slave/testfs > success > receiving incremental stream of tank@backup-20090730-2012 into > slave@backup-20090730-2012 > received 312B stream in 1 seconds (312B/sec) > receiving incremental stream of tank/tmp@backup-20090730-2012 into > slave/tmp@backup-20090730-2012 > received 312B stream in 1 seconds (312B/sec) > receiving incremental stream of tank/var@backup-20090730-2012 into > slave/var@backup-20090730-2012 > received 32.6KB stream in 1 seconds (32.6KB/sec) > receiving incremental stream of tank/var/log@backup-20090730-2012 > into slave/var/log@backup-20090730-2012 > received 298KB stream in 1 seconds (298KB/sec) > receiving incremental stream of tank/var/crash@backup-20090730-2012 > into slave/var/crash@backup-20090730-2012 > received 312B stream in 1 seconds (312B/sec) > receiving incremental stream of tank/root@backup-20090730-2012 into > slave/root@backup-20090730-2012 > [... panic here ...] > > Unread portion of the kernel message buffer:panic: solaris assert: > ((zp)->z_vnode)->v_usecount > 0, file: /usr/src/sys/modules/ > zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c, > line: 920 > cpuid = 0 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > panic() at panic+0x182 > zfsvfs_teardown() at zfsvfs_teardown+0x24d > zfs_suspend_fs() at zfs_suspend_fs+0x2b > zfs_ioc_recv() at zfs_ioc_recv+0x28b > zfsdev_ioctl() at zfsdev_ioctl+0x8a > devfs_ioctl_f() at devfs_ioctl_f+0x77 > kern_ioctl() at kern_ioctl+0xf6 > ioctl() at ioctl+0xfd > syscall() at syscall+0x28f > Xfast_syscall() at Xfast_syscall+0xe1 > --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe5f7c, rsp = > 0x7fffffff8ef8, rbp = 0x7fffffff9c30 --- > KDB: enter: panic > panic: from debugger > > #9 0xffffffff8057eda7 in calltrap () > at /usr/src/sys/amd64/amd64/exception.S:224 > #10 0xffffffff8036c8ad in kdb_enter (why=0xffffffff80609c44 "panic", > msg=0xa
) at cpufunc.h:63 > #11 0xffffffff8033abcb in panic (fmt=Variable "fmt" is not available. > ) at /usr/src/sys/kern/kern_shutdown.c:558#12 0xffffffff80b0ec5d > in zfsvfs_teardown () from /boot/kernel/zfs.ko#13 0x0000000000100000 > in ?? () > #14 0xffffff001bff0250 in ?? () > #15 0xffffff001bff0000 in ?? () > #16 0xffffff0008004000 in ?? () > #17 0xffffff803e9747a0 in ?? () > #18 0xffffff803e9747d0 in ?? () > #19 0xffffff803e974770 in ?? () > #20 0xffffff803e974740 in ?? () > #21 0xffffffff80b0ecab in zfs_suspend_fs () from /boot/kernel/zfs.ko > Previous frame inner to this frame (corrupt stack?) > > Unfortunately, I'm not sure I can reproduce this reliably, since it > worked a bunch of times both before and after my previous mail. > > Oh, and I'm still using -DDEBUG=1 and DEBUG_VFS_LOCKS... If this > isn't a new panic because of the changes, perhaps it was triggered > now and never before because of the -DDEBUG? > > Regards, > Thomas I'm able to reliably reproduce this panic, by having zfs recv destroy a filesystem on the receiving end. 1) Use DDEBUG=1, I guess 2) Create a FS on the source pool you don't care about: zfs create -o mountpoint=/testfs source/testfs 3) Clone a pool to another: zfs snapshot -r source@snap && zfs send -R source@snap | zfs recv -Fvd target 4) zfs destroy -r source/testfs 4) zfs snapshot -r source@snap2 && zfs send -R -I snap source@snap2 | zfs recv -Fvd target 5) ^ Panic while receiving the FS the destroyed one is mounted under. In my case, this was tank/root three times out of three; I then tried creating testfs under /tmp (tank/tmp/testfs), *mounting* it under /usr/ testfs, and it panics on receiving tank/usr: attempting destroy slave/tmp/testfs@backup-20090731-1100 success attempting destroy slave/tmp/testfs@backup-20090731-1036 success attempting destroy slave/tmp/testfs success ... receiving incremental stream of tank/tmp@backup-20090731-1101 into slave/tmp@backup-20090731-1101 received 312B stream in 1 seconds (312B/sec) receiving incremental stream of tank/root@backup-20090731-1101 into slave/root@backup-20090731-1101 received 58.3KB stream in 1 seconds (58.3KB/sec) receiving incremental stream of tank/usr@backup-20090731-1101 into slave/usr@backup-20090731-1101 ... panic here, no more output Same backtrace/assert as above. Regards, Thomas From james-freebsd-fs2 at jrv.org Fri Jul 31 11:45:15 2009 From: james-freebsd-fs2 at jrv.org (James R. Van Artsdalen) Date: Fri Jul 31 11:45:22 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <4A71B2DA.9060902@freebsd.org> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> <4A708455.5070304@freebsd.org> <86983A55-E5C4-4C04-A4C7-0AE9A9EE37A3@exscape.org> <4A718E03.6030909@freebsd.org> <71A038EC-02B1-4606-96C2-5E84BE80F005@exscape.org> <4A719CA4.4060400@freebsd.org> <19347561-3CE6-40B3-930A-EB9925D3AFD1@exscape.org> <4A71AD29.10705@freebsd.org> <7544AED1-1216-4A24-B287-F54117641F76@exscape.org> <4A71B2DA.9060902@freebsd.org> Message-ID: <4A72D946.4090401@jrv.org> Andriy Gapon wrote: > on 30/07/2009 17:39 Thomas Backman said the following: > >> Or, in patch form (I think the intendation screws the patch up as linked >> there): >> http://exscape.org/temp/libzfs_sendrecv.patch >> > > One comment on the patch - I personally don't like bit-wise xor in a logical > expression. But if otherwise the expression would be huge and ugly, then OK. > If you're going to code an XOR, use an XOR. Don' make the reader untangle code to figure out that that some other code is really just an XOR. However I think I was trying to handle two cases that can't happen: the top filesystem cannot be renamed to somewhere else in the pool, and no other filesystem can be renamed to the root. So the new version of the patch below needs no XOR. Without this or something like it you can't replicate an entire pool, i.e. zfs send -R -I @yesterday pool@today | ssh backup zfs recv -vF -d pool dumps core from the strccmp(0, 0) in the original code below. Index: cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c =================================================================== --- cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c (revision 192136) +++ cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c (working copy) @@ -1126,7 +1126,7 @@ uint64_t originguid = 0; uint64_t stream_originguid = 0; uint64_t parent_fromsnap_guid, stream_parent_fromsnap_guid; - char *fsname, *stream_fsname; + char *fsname, *stream_fsname, *p1, *p2; nextfselem = nvlist_next_nvpair(local_nv, fselem); @@ -1295,10 +1295,11 @@ "parentfromsnap", &stream_parent_fromsnap_guid)); /* check for rename */ + p1 = strrchr(fsname, '/'); + p2 = strrchr(stream_fsname, '/'); if ((stream_parent_fromsnap_guid != 0 && stream_parent_fromsnap_guid != parent_fromsnap_guid) || - strcmp(strrchr(fsname, '/'), - strrchr(stream_fsname, '/')) != 0) { + (p1 != NULL && p2 != NULL && strcmp (p1, p2) != 0)) { nvlist_t *parent; char tryname[ZFS_MAXNAMELEN]; @@ -1317,7 +1318,7 @@ VERIFY(0 == nvlist_lookup_string(parent, "name", &pname)); (void) snprintf(tryname, sizeof (tryname), - "%s%s", pname, strrchr(stream_fsname, '/')); + "%s%s", pname, p2 ? p2 : ""); } else { tryname[0] = '\0'; if (flags.verbose) { From serenity at exscape.org Fri Jul 31 12:27:52 2009 From: serenity at exscape.org (Thomas Backman) Date: Fri Jul 31 12:28:04 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: <4A72D946.4090401@jrv.org> References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> <4A708455.5070304@freebsd.org> <86983A55-E5C4-4C04-A4C7-0AE9A9EE37A3@exscape.org> <4A718E03.6030909@freebsd.org> <71A038EC-02B1-4606-96C2-5E84BE80F005@exscape.org> <4A719CA4.4060400@freebsd.org> <19347561-3CE6-40B3-930A-EB9925D3AFD1@exscape.org> <4A71AD29.10705@freebsd.org> <7544AED1-1216-4A24-B287-F54117641F76@exscape.org> <4A71B2DA.9060902@freebsd.org> <4A72D946.4090 401@jrv.org> Message-ID: <76338BEC-9B85-4AD1-B04B-850486866F3B@exscape.org> On Jul 31, 2009, at 13:45, James R. Van Artsdalen wrote: > Andriy Gapon wrote: >> on 30/07/2009 17:39 Thomas Backman said the following: >> >>> Or, in patch form (I think the intendation screws the patch up as >>> linked >>> there): >>> http://exscape.org/temp/libzfs_sendrecv.patch >>> >> >> One comment on the patch - I personally don't like bit-wise xor in >> a logical >> expression. But if otherwise the expression would be huge and ugly, >> then OK. >> > > If you're going to code an XOR, use an XOR. > Don' make the reader untangle code to figure out that that some other > code is really just an XOR. > > However I think I was trying to handle two cases that can't happen: > the > top filesystem cannot be renamed to somewhere else in the pool, and no > other filesystem can be renamed to the root. So the new version of > the > patch below needs no XOR. > > Without this or something like it you can't replicate an entire > pool, i.e. > > zfs send -R -I @yesterday pool@today | ssh backup zfs recv -vF -d > pool > > dumps core from the strccmp(0, 0) in the original code below. > > > Index: cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c > =================================================================== > ... Nice job, thanks :) Just wanted to chime in and say that your new patch seems to work just as well as the previous one. I hope you don't mind me hosting this too (I had to apply it manually thanks to spacing... I think it's my mail client not being very nice at retaining tabs/spaces)... Straight from svn diff: http://exscape.org/temp/libzfs_sendrecv.new.patch BTW (maybe not on topic for this mail, but for this thread), I've created a test case to reproduce the new panic (every time). It happens with -DDEBUG=1, after destroying a filesystem and then doing an incremental backup. Currently recompiling world/kernel on a second box to reproduce before I post that. Regards, Thomas From bf1783 at googlemail.com Fri Jul 31 13:51:46 2009 From: bf1783 at googlemail.com (b. f.) Date: Fri Jul 31 13:51:53 2009 Subject: UFS2 tuning for heterogeneous 4TB file system In-Reply-To: <26ddd1750907260719x761a1c94r27c572ab1ff6a582@mail.gmail.com> References: <26ddd1750907260719x761a1c94r27c572ab1ff6a582@mail.gmail.com> Message-ID: On 7/26/09, Maxim Khitrov wrote: > On Sun, Jul 26, 2009 at 3:56 AM, b. f. wrote: >>>The file system in question will not have a common file size (which is >>>what, as I understand, bytes per inode should be tuned for). There >>>will be many small files (< 10 KB) and many large ones (> 500 MB). A >>>similar, in terms of content, 2TB ntfs file system on another server >>>has an average file size of about 26 MB with 59,246 files. >> >> Ordinarily, it may have a large variation in file sizes, but can you >> intervene, and segregate large and small files in separate >> filesystems, so that you can optimize the settings for each >> independently? > > That's a good idea, but the problem is that this raid array will grow > in the future as I add additional drives. As far as I know, a > partition can be expanded using growfs, but it cannot be moved to a > higher address (with any "standard" tools). So if I create two > separate partitions for different file types, the first partition will > have to remain a fixed size. That would be problematic, since I cannot > easily predict how much space it would need initially and for all > future purposes (enough to store all the files, yet not waste space > that could otherwise be used for the second partition). > Perhaps gconcat(8), gmirror(8), or vinum(4) will solve your problem here. I think there are other tools as well. >>>Ideally, I would prefer that small files do not waste more than 4 KB >>>of space, which is what you have with ntfs. At the same time, having >>>fsck running for days after an unclean shutdown is also not a good >>>option (I always disable background checking). From what I've gathered >>>so far, the two requirements are at the opposite ends in terms of file >>>system optimization. >> >> I gather you are trying to be conservative, but have you considered >> using gjournal(8)? At least for the filesystems with many small >> files? In that way, you could safely avoid the need for most if not >> all use of fsck(8), and, as an adjunct benefit, you would be able to >> operate on the small files more quickly: >> >> http://lists.freebsd.org/pipermail/freebsd-current/2006-June/064043.html >> http://www.freebsd.org/doc/en_US.ISO8859-1/articles/gjournal-desktop/article.html >> >> gjournal has a lower overhead than ZFS, and has proven to be fairly >> reliable. Also, you can always unhook it and revert to plain UFS >> mounts easily. >> >> b. >> > > Just fairly reliable? :) > Well, I'm not going to promise the sun, the moon, and the stars. It has worked for me (better than softupdates, I might add) under my more modest workloads. > I've done a bit of reading on gjournal and the main thing that's > preventing me from using it is the recency of implementation. I've had > a number of FreeBSD servers go down in the past due to power outages > and SoftUpdates with foreground fsck have never failed me. I have > never had a corrupt ufs2 partition, which is not something I can say > about a few linux servers with ext3. > > Have there been any serious studies into how gjournal and SU deal with > power outages? By that I mean taking two identical machines, issuing > write operations, yanking the power cords, and then watching both > systems recover? I'm sure that gjournal will take less time to reboot, > but if this experiment is repeated a few hundred times I wonder what > the corruption statistics would be. Is there ever a case, for > instance, when the journal itself becomes corrupt because the power > was pulled in the middle of a metadata flush? > I'm not aware of any such tests, but I wouldn't be surprised if pjd@ or someone else who was interested in using gjournal(8) in a demanding environment had made some. I'll cc freebsd-fs@, because some of them may not monitor freebsd-questions. Perhaps someone there has some advice. You might also try asking on freebsd-geom@. Regards, b. > Basically, I have no experience with gjournal, poor experience with > other journaled file systems, and no real comparison between > reliability characteristics of gjournal and SoftUpdates, which have > served me very well in the past. > > - Max > From jensrasmus at gmail.com Fri Jul 31 14:53:39 2009 From: jensrasmus at gmail.com (Jens Rasmus Liland) Date: Fri Jul 31 14:53:58 2009 Subject: How do I mount an external ntfs formatted harddisk manually and through /etc/fstab? Message-ID: <63e02e980907310725t2b38d1d3iff66aca3948ac8dd@mail.gmail.com> Hi, How do I mount an NTFS formatted external harddisk plugged into the computer using a usb cable? And what do i write in the /etc/fstab after being able to successfully mount it manually? I have some blurry understanding after reading a bit in handbook that the harddisk's NTFS partition is at /dev/da0s1 by default. I have installed ntfs-3g from ports. /Rasmus From serenity at exscape.org Fri Jul 31 17:10:37 2009 From: serenity at exscape.org (Thomas Backman) Date: Fri Jul 31 17:10:44 2009 Subject: zfs: Fatal trap 12: page fault while in kernel mode In-Reply-To: References: <20090727072503.GA52309@jpru.ffm.jpru.de> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> <4A708455.5070304@freebsd.org> <86983A55-E5C4-4C04-A4C7-0AE9A9EE37A3@exscape.org> <4A718E03.6030909@freebsd.org> <71A038EC-02B1-4606-96C2-5E84BE80F005@exscape.org> <4A719CA4.4060400@freebsd.org> <19347561-3CE6-40B3-930A-EB9925D3AFD1@exscape.org> <4A71AD29.10705@freebsd.org> <7544AED1-1216-4A24-B287-F54117641F76@exscape.org> <4 A71B239.8060007@freebsd.org> <3AA3C1CB-CEF7-46CC-A9C7-1648093D679E@exsca! pe.org> <4A71BED8.7050300@freebsd.org> Message-ID: On Jul 30, 2009, at 20:29, Thomas Backman wrote: > On Jul 30, 2009, at 18:41, Thomas Backman wrote: > >> On Jul 30, 2009, at 17:40, Andriy Gapon wrote: >>> on 30/07/2009 18:25 Thomas Backman said the following: >>>> PS. I'll test Pawel's patch sometime after dinner. ;) >>> >>> I believe that you should get a perfect result with it. >>> >>> -- Andriy Gapon >> If I dare say it, you were right! I've been testing for about half >> an hour or so (probably a bit more) now. >> Still using DEBUG_VFS_LOCKS, and I've tried the test case several >> times, ran an initial backup (i.e. destroy target pool and send| >> recv the entire pool) and a few incrementals. Rebooted, tried it >> again. No panic, no problems! :) >> Let's hope it stays this way. >> >> So, in short: With that patch (copied here just in case: http://exscape.org/temp/zfs_vnops.working.patch >> ) and the libzfs patch linked previously, it appears zfs send/recv >> works plain fine. I have yet to try it with clone/promote and >> stuff, but since that gave the same panic that this solved, I'm >> hoping there will be no problems with that anymore. > > Arrrgh! > I guess I spoke too soon after all... new panic yet again. :( > *sigh* It feels as if this will never become stable right now. > (Maybe that's because I've spent all day and most of yesterday too > on this ;) > > [... same panic as I'm posting in the reply below snipped ...] > > Unfortunately, I'm not sure I can reproduce this reliably, since it > worked a bunch of times both before and after my previous mail. > > Oh, and I'm still using -DDEBUG=1 and DEBUG_VFS_LOCKS... If this > isn't a new panic because of the changes, perhaps it was triggered > now and never before because of the -DDEBUG? OK, I created a "test case" that triggers this panic for me every time, and reproduced it on another machine, so it should, uh, "work" for anyone reading this as well. Here are my patches, and the script used to reproduce the panic: (This assumes that you've got a clean SVN/cvsup source tree. If you have any of the patches mentioned below, remove them from the .patch first.) http://exscape.org/temp/zfs_destroy_panic_patches.patch (contains: James R. Van Artsdalen's libzfs_sendrecv patch that makes it not coredump(...), activating ZFS debugging (-DDEBUG=1), and Pawel's zfs_vnops.c patch.) http://exscape.org/temp/zfs_destroy_panic.sh (needs bash and 200MB free on your /root/-containing FS, unless you change the variables at the top; usage: "bash ...sh crash") You'll need to rebuild zfs.ko and libzfs, and if you use zfs.ko already, of course, reboot. (The libzfs patch can be installed and used without rebooting.) 1) cd /usr/src; fetch http://exscape.org/temp/zfs_destroy_panic_patches.patch && patch < zfs_destroy_panic_patches.patch 2) cd /usr/src/cddl/lib/libzfs/ ; make && make install 3) cd /usr/src/sys/modules/zfs ; make && make install 3b) (reboot, or kldload zfs) 4) fetch http://exscape.org/temp/zfs_destroy_panic.sh && bash zfs_destroy_panic.sh crash My output (snipped for brevity, most is useless stuff from dd, etc.): (I prepended a >> to output written by my script; the rest is from zfs. This isn't in the script itself.) >> Creating pools >> Creating filesystems >> Creating snapshot(s) >> Doing initial clone to slave pool receiving full stream of crashtestmaster@backup-20090731-185218 into crashtestslave@backup-20090731-185218 received 15.0KB stream in 1 seconds (15.0KB/sec) receiving full stream of crashtestmaster/ testroot@backup-20090731-185218 into crashtestslave/ testroot@backup-20090731-185218 received 15.0KB stream in 1 seconds (15.0KB/sec) receiving full stream of crashtestmaster/testroot/ testfs@backup-20090731-185218 into crashtestslave/testroot/ testfs@backup-20090731-185218 received 1.02MB stream in 1 seconds (1.02MB/sec) >> Initial step done! >> Destroying testfs >> Taking snapshots >> Starting backup... sending from @backup-20090731-185218 to crashtestmaster@backup-20090731-185226-11214-7776 sending from @backup-20090731-185218 to crashtestmaster/ testroot@backup-20090731-185226-11214-7776 attempting destroy crashtestslave/testroot/testfs@backup-20090731-185218 success attempting destroy crashtestslave/testroot/testfs success receiving incremental stream of crashtestmaster@backup-20090731-185226-11214-7776 into crashtestslave@backup-20090731-185226-11214-7776 received 312B stream in 1 seconds (312B/sec) receiving incremental stream of crashtestmaster/ testroot@backup-20090731-185226-11214-7776 into crashtestslave/ testroot@backup-20090731-185226-11214-7776 [... panic, no no more output ...] DDB info, etc (from the original box; not the same run as above, but the same panic, so...): Unread portion of the kernel message buffer: panic: solaris assert: ((zp)->z_vnode)->v_usecount > 0, file: /usr/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ zfs_vfsops.c, line: 920 cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a panic() at panic+0x182 zfsvfs_teardown() at zfsvfs_teardown+0x24d zfs_suspend_fs() at zfs_suspend_fs+0x2b zfs_ioc_recv() at zfs_ioc_recv+0x28b zfsdev_ioctl() at zfsdev_ioctl+0x8a devfs_ioctl_f() at devfs_ioctl_f+0x77 kern_ioctl() at kern_ioctl+0xf6 ioctl() at ioctl+0xfd syscall() at syscall+0x28f Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe5f7c, rsp = 0x7fffffff8ee8, rbp = 0x7fffffff9c20 --- KDB: enter: panic panic: from debugger cpuid = 0 Uptime: 25m47s Physical memory: 2030 MB Dumping 1663 MB: ... #11 0xffffffff8033abcb in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:558 #12 0xffffffff80b0ec5d in zfsvfs_teardown () from /boot/kernel/zfs.ko #13 0x0000000000100000 in ?? () #14 0xffffff0048a7e250 in ?? () #15 0xffffff0048a7e000 in ?? () #16 0xffffff00063c0000 in ?? () #17 0xffffff803e8f27a0 in ?? () #18 0xffffff803e8f27d0 in ?? () #19 0xffffff803e8f2770 in ?? () #20 0xffffff803e8f2740 in ?? () #21 0xffffffff80b0ecab in zfs_suspend_fs () from /boot/kernel/zfs.ko Previous frame inner to this frame (corrupt stack?) I commented out -DDEBUG=1 and rebuilt+installed just the zfs module, and the panic appears to be gone. With DEBUG, it panicked every time (and I tried it at least 4-5 times). Without, it has worked flawlessly three times in a row, as has my regular backup. So, the big, TL;DR question is: is the ASSERT() unnecessary, as Andriy proposed it *might* be, or is this a real issue that actually needs fixing? It doesn't feel right to just ignore a potential bug by ignoring a failed assertion... Pawel? Regards, Thomas