From danny at cs.huji.ac.il Wed Oct 1 12:04:21 2008 From: danny at cs.huji.ac.il (Danny Braniss) Date: Wed Oct 1 12:04:29 2008 Subject: svn rev. number Message-ID: Hi, Now that freebsd is under svn, I decided to try what I failed with cvs, and actually using svn/svk/svnsync I have a mirror and a local branch in sync! Since the date reported by uname is not that relevant, is it possible to add the svn-revision # ala build-...? This could make finding problems easier, instead of kernel from 'date' one could say date/revision... just a thought. danny From rpaulo at freebsd.org Wed Oct 1 12:15:54 2008 From: rpaulo at freebsd.org (Rui Paulo) Date: Wed Oct 1 12:16:27 2008 Subject: svn rev. number In-Reply-To: References: Message-ID: On 1 Oct 2008, at 13:04, Danny Braniss wrote: > Hi, > Now that freebsd is under svn, I decided to try what I failed > with cvs, and actually using svn/svk/svnsync I have a mirror and a > local branch > in sync! > Since the date reported by uname is not that relevant, is it > possible to add > the svn-revision # ala build-...? This could make finding problems > easier, > instead of kernel from 'date' one could say date/revision... just a > thought. That was already done weeks ago: rpaulo@alpha ~ % uname -v FreeBSD 8.0-CURRENT #2 r182964: Sat Sep 13 17:32:57 WEST 2008 rpaulo@alpha.local :/home/rpaulo/freebsd/obj/home/rpaulo/freebsd/base/head/sys/ALPHA `ident /boot/kernel/kernel' also works. Regards, -- Rui Paulo -- Rui Paulo From max at love2party.net Wed Oct 1 12:21:16 2008 From: max at love2party.net (Max Laier) Date: Wed Oct 1 12:21:24 2008 Subject: svn rev. number In-Reply-To: References: Message-ID: <200810011421.13411.max@love2party.net> On Wednesday 01 October 2008 14:04:20 Danny Braniss wrote: > Hi, > Now that freebsd is under svn, I decided to try what I failed > with cvs, and actually using svn/svk/svnsync I have a mirror and a local > branch in sync! > Since the date reported by uname is not that relevant, is it possible to > add the svn-revision # ala build-...? This could make finding problems > easier, instead of kernel from 'date' one could say date/revision... just a > thought. We are doing that - for quite some time now. (see r179637 & r179655). The requirements are that: 1) svnversion is executable in /bin, /usr/bin or /usr/local/bin 2) there is a .svn directory in your SRCDIR If that's the case, newvers.sh will add the output of it to uname: "FreeBSD fbsd8 8.0-CURRENT FreeBSD 8.0-CURRENT #4 r180876:183019M:..." -- /"\ Best regards, | mlaier@freebsd.org \ / Max Laier | ICQ #67774661 X http://pf4freebsd.love2party.net/ | mlaier@EFnet / \ ASCII Ribbon Campaign | Against HTML Mail and News From danny at cs.huji.ac.il Wed Oct 1 12:37:15 2008 From: danny at cs.huji.ac.il (Danny Braniss) Date: Wed Oct 1 12:37:22 2008 Subject: svn rev. number In-Reply-To: Your message of Wed, 1 Oct 2008 14:21:13 +0200 . Message-ID: > On Wednesday 01 October 2008 14:04:20 Danny Braniss wrote: > > Hi, > > Now that freebsd is under svn, I decided to try what I failed > > with cvs, and actually using svn/svk/svnsync I have a mirror and a local > > branch in sync! > > Since the date reported by uname is not that relevant, is it possible to > > add the svn-revision # ala build-...? This could make finding problems > > easier, instead of kernel from 'date' one could say date/revision... just a > > thought. > > We are doing that - for quite some time now. (see r179637 & r179655). The > requirements are that: > great! > 1) svnversion is executable in /bin, /usr/bin or /usr/local/bin > 2) there is a .svn directory in your SRCDIR > small catch, i'm using svk, but i guess I'll look into current's newvers.sh. and running -stable. > If that's the case, newvers.sh will add the output of it to uname: > > "FreeBSD fbsd8 8.0-CURRENT FreeBSD 8.0-CURRENT #4 r180876:183019M:..." From bagavathykumar.m at hcl.in Wed Oct 1 12:50:30 2008 From: bagavathykumar.m at hcl.in (Bagavathy Kumar Mahendran ) Date: Wed Oct 1 13:03:47 2008 Subject: FW: i386/127710: My driver PCI probe is not called for my correspondingdevice ID and Vendor ID Message-ID: <68C9F31EF19DB6448F515EF294028FDEE999AB@chn-hclt-evs05.HCLT.CORP.HCL.IN> Dear All, Iam writing a new driver for a SAS/SATA Controller having a Class ID -0x01 Sub Class - 0x07 Programming Interface - 0x00 Hence instead of my probe function the Static build Card Bus Driver cbb is attaching just by simply checking sub class 0x07 and programming interface 0x00.hence my probe gets failed. Kindly help me in resolving this .what I thought is to add the card bus driver a checking of CLASS ID in its pci probe function. Thanks With Regards, Bagavathy kumar .M -----Original Message----- From: Remko Lodder [mailto:remko@FreeBSD.org] Sent: Wednesday, October 01, 2008 5:48 PM To: Bagavathy Kumar Mahendran Cc: freebsd-gnats-submit@FreeBSD.org Subject: Re: i386/127710: My driver PCI probe is not called for my correspondingdevice ID and Vendor ID Bagavathy kumar . M wrote: Please submit this question to the hackers@FreeBSD.org mailinglist. This is not a PR (yet), you might have done something wrong to your code or are missing logic(s) which are needed in that case. I will close the ticket because of that. Thanks for taking the time to report this and for using FreeBSD, it's appreciated! -- /"\ Best regards, | remko@FreeBSD.org \ / Remko Lodder | remko@EFnet X http://www.evilcoder.org/ | / \ ASCII Ribbon Campaign | Against HTML Mail and News DISCLAIMER: ----------------------------------------------------------------------------------------------------------------------- The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or HCL or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any mail and attachments please check them for viruses and defect. ----------------------------------------------------------------------------------------------------------------------- From bagavathykumar.m at hcl.in Wed Oct 1 12:56:36 2008 From: bagavathykumar.m at hcl.in (Bagavathy Kumar Mahendran ) Date: Wed Oct 1 13:28:21 2008 Subject: i386/127710: My driver PCI probe is not called for my correspondingdevice ID and Vendor ID In-Reply-To: <48E36A7C.2040505@FreeBSD.org> References: <200809291002.m8TA2E95065005@www.freebsd.org> <48E36A7C.2040505@FreeBSD.org> Message-ID: <68C9F31EF19DB6448F515EF294028FDEE99980@chn-hclt-evs05.HCLT.CORP.HCL.IN> My PCI Express card has Class ID is 0x1 Sub Class ID is 0x7 Programming Interface is 0x0 How a card bus driver can attach my PCI Express Card. The Card Bus driver gets attached with out checking the Class ID. How he can take Sub Class ID & Programming Interface alone for Attaching it. Thanks for your replay. regards, Bagavathy kumar.M -----Original Message----- From: Remko Lodder [mailto:remko@FreeBSD.org] Sent: Wednesday, October 01, 2008 5:48 PM To: Bagavathy Kumar Mahendran Cc: freebsd-gnats-submit@FreeBSD.org Subject: Re: i386/127710: My driver PCI probe is not called for my correspondingdevice ID and Vendor ID Bagavathy kumar . M wrote: Please submit this question to the hackers@FreeBSD.org mailinglist. This is not a PR (yet), you might have done something wrong to your code or are missing logic(s) which are needed in that case. I will close the ticket because of that. Thanks for taking the time to report this and for using FreeBSD, it's appreciated! -- /"\ Best regards, | remko@FreeBSD.org \ / Remko Lodder | remko@EFnet X http://www.evilcoder.org/ | / \ ASCII Ribbon Campaign | Against HTML Mail and News DISCLAIMER: ----------------------------------------------------------------------------------------------------------------------- The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or HCL or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any mail and attachments please check them for viruses and defect. ----------------------------------------------------------------------------------------------------------------------- From jhb at freebsd.org Wed Oct 1 19:19:24 2008 From: jhb at freebsd.org (John Baldwin) Date: Wed Oct 1 19:19:32 2008 Subject: FW: i386/127710: My driver PCI probe is not called for my correspondingdevice ID and Vendor ID In-Reply-To: <68C9F31EF19DB6448F515EF294028FDEE999AB@chn-hclt-evs05.HCLT.CORP.HCL.IN> References: <68C9F31EF19DB6448F515EF294028FDEE999AB@chn-hclt-evs05.HCLT.CORP.HCL.IN> Message-ID: <200810011127.14593.jhb@freebsd.org> On Wednesday 01 October 2008 08:50:15 am Bagavathy Kumar Mahendran wrote: > > Dear All, > Iam writing a new driver for a SAS/SATA Controller having a > Class ID -0x01 > Sub Class - 0x07 > Programming Interface - 0x00 > > Hence instead of my probe function the Static build Card Bus Driver cbb > is attaching just by simply checking sub class 0x07 and programming > interface 0x00.hence my probe gets failed. Kindly help me in resolving > this .what I thought is to add the card bus driver a checking of CLASS > ID in its pci probe function. The pccbb driver returns BUS_PROBE_DEFAULT (it should probably return GENERIC in the case where it matches only on class codes). Your driver just needs to return a numerically higher value (but still < 0) to claim the device. You can probably use BUS_PROBE_VENDOR or BUS_PROBE_DEFAULT + 1. -- John Baldwin From artis.caune at gmail.com Wed Oct 1 20:03:15 2008 From: artis.caune at gmail.com (Artis Caune) Date: Wed Oct 1 20:03:22 2008 Subject: svn rev. number In-Reply-To: References: Message-ID: <9e20d71e0810011231t424ccce1q8b6dc02807572fc4@mail.gmail.com> > small catch, i'm using svk, but i guess I'll look into current's newvers.sh. > and running -stable. > >> If that's the case, newvers.sh will add the output of it to uname: >> >> "FreeBSD fbsd8 8.0-CURRENT FreeBSD 8.0-CURRENT #4 r180876:183019M:..." How about $FreeBSD$ svn tag. Some binaries include rcsids in them. Shell scripts and config files in /etc will be with empt revision string?! You should really use subversion-freebsd. -- regards, Artis Caune <----. CCNA <----|==================== <----' didii FreeBSD From jhb at freebsd.org Wed Oct 1 21:18:29 2008 From: jhb at freebsd.org (John Baldwin) Date: Wed Oct 1 21:18:36 2008 Subject: Major SMP problems with lstat/namei In-Reply-To: References: <8185F68B-C443-4891-BEC2-5E3D453DDC93@wheelhouse.org> <200809241212.09920.jhb@freebsd.org> Message-ID: <200810011717.35873.jhb@freebsd.org> On Thursday 25 September 2008 07:00:04 pm Jeff Wheelhouse wrote: > > On Sep 24, 2008, at 12:12 PM, John Baldwin wrote: > > Shared lookups only work on the NFS client in 6.x. I'm about to > > turn them on > > for UFS in HEAD (8.x) and will backport the needed fixes to 7.x > > after 7.1 > > (too risky to merge to 7.x this close to a release). > > OK, given all the patches you referenced, I did make a decent effort > at backporting to 7.0. It sounds like you missed some of the dirhash changes somehow, as dirhash no longer has any lockmgr stuff in it (and only ever did in HEAD). I've generated a patch though using svn. You can grab it from http://www.FreeBSD.org/~jhb/patches/ufs_lookup7.patch Note that you will have to set vfs.lookup_shared=1 to enable shared locks (either loader tunable or sysctl). Also, I found a few other changes I had missed earlier that needed to be included. -- John Baldwin From ravi.murty at intel.com Wed Oct 1 23:44:24 2008 From: ravi.murty at intel.com (Murty, Ravi) Date: Wed Oct 1 23:44:31 2008 Subject: Sched_ule.c - 8.0 Message-ID: Hello All, I was browsing the ULE 8.0 scheduler code and happen to find something interesting. This might be intentional; since I don't think it is that big a deal and is certainly not a bug. In the implementation of sched_affinity - which from what I understand gets called when the cpuset mask for a thread or a process is setup and threads need to potentially migrated. The code is pretty straightforward and one of the checks it does is if (!TD_IS_RUNNING(td)) return; I initially read this to mean, if the thread isn't running, it's probably inhibited and that's okay because when it wanders into sched_add eventually and since its cpuset mask is setup, it'll make its way to the runq of one of the "legal" cpus. However the very next thought I had was this thread could be on a runq right now and the macro will return the fact that the thread isn't running. In such a case we would probably end up running on the wrong CPU for a while before realizing that we aren't allowed to do so. Thanks Ravi From eitanadlerlist at gmail.com Thu Oct 2 03:15:00 2008 From: eitanadlerlist at gmail.com (Eitan Adler) Date: Thu Oct 2 03:15:07 2008 Subject: SSH Brute Force attempts In-Reply-To: <48E16E93.3090601@gmail.com> References: <48E16E93.3090601@gmail.com> Message-ID: <48E4368E.4020404@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Rich Healey wrote: > Recently I'm getting a lot of brute force attempts on my server, in the > past I've used various tips and tricks with linux boxes but many of them > were fairly linux specific. > > What do you BSD guys use for this purpose? > > If this belongs on -security let me know and I'll ask over there. > > Cheers > > > Rich _______________________________________________ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" Personally I find that changing the port to anything other than 22 stops a lot of the skiddie brute force attacks. Thats not to say you shouldn't use something else as well - but it is something. P.S. Can someone please let me know if the mailing list got this message - some of my messages have not been getting replies for some reason. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkjkNo4ACgkQtl8kq+nCzNGUKQCeN/bmzWuYr+Xc8i/iXufayd3l LWYAnRcdVWTQe4t/EXDxBYpC+QlEO1CD =4rKm -----END PGP SIGNATURE----- From ancelgray at yahoo.com Thu Oct 2 02:32:12 2008 From: ancelgray at yahoo.com (ancelgray@yahoo.com) Date: Thu Oct 2 03:49:21 2008 Subject: Hardware support for AMD Geode CS5536 audio? In-Reply-To: <21541490.01216080059414.JavaMail.root@wombat.diezmil.com> Message-ID: <18326726.31222938365567.JavaMail.root@wombat.diezmil.com> To All, I finally have a working CS5536 audio driver for FreeBSD 6.2. The filename is snd_amd5536.ko and I will be making it available shortly. I have been using an ALIX-1C motherboard for my testing. Anyone have other hardware that they want to test it on? Another question. How does one get the FreeBSD people to put this driver into their OS? Andrew Gray ancelgray "A T" y a h o o "D O T" c o m -- This message was sent on behalf of ancelgray@yahoo.com at openSubscriber.com http://www.opensubscriber.com/message/freebsd-hackers@freebsd.org/9623264.html From ancelgray at yahoo.com Thu Oct 2 02:32:12 2008 From: ancelgray at yahoo.com (ancelgray@yahoo.com) Date: Thu Oct 2 03:49:29 2008 Subject: Hardware support for AMD Geode CS5536 audio? In-Reply-To: <21541490.01216080059414.JavaMail.root@wombat.diezmil.com> Message-ID: <32225808.21222938312495.JavaMail.root@wombat.diezmil.com> To All, I finally have a working CS5536 audio driver for FreeBSD 6.2. The filename is snd_amd5536.ko and I will be making it available shortly. I have been using an ALIX-1C motherboard for my testing. Anyone have other hardware that they want to test it on? Another question. How does one get the FreeBSD people to put this driver into their OS? Andrew Gray ancelgray "A T" y a h o o "D O T" c o m -- This message was sent on behalf of ancelgray@yahoo.com at openSubscriber.com http://www.opensubscriber.com/message/freebsd-hackers@freebsd.org/9623264.html From yuri at rawbw.com Thu Oct 2 07:27:14 2008 From: yuri at rawbw.com (Yuri) Date: Thu Oct 2 07:27:44 2008 Subject: valgrind on FreeBSD 7 In-Reply-To: <4803C3B4.4000405@delphij.net> References: <200803172156.37407.modelnine@modelnine.org> <20080317214510.G89676@odysseus.silby.com> <20080413103351.GA1382@dose.local.invalid> <4803C3B4.4000405@delphij.net> Message-ID: <48E474C7.8050507@rawbw.com> Xin LI wrote: > Simon Barner wrote: >> Mike Silbersack wrote: >>> On Mon, 17 Mar 2008, Heiko Wundram wrote: >>> >>> Here's a tarball of what's in perforce right now. I tried it a >>> little bit, and it seemed to work for me. Make sure to install the >>> kernel module! >>> >>> http://www.silby.com/valgrind_freebsd_3.tar.gz >>> >>> But don't send me questions about it - I'm not an expert on it, I'm >>> just the guy who grabbed it from perforce and found that it seems to >>> work. :) >> >> Could you please provide me with the details, so I can update my >> (horribly outdated :( valgrind ports? > > It was available from p4 at: > > //depot/projects/valgrind/... > > Note that this version does not work on architectures other than i386. > > Cheers, Any developments in Valgrind/Callgrind on FreeBSD? Any hope to get this version into ports and to merge FreeBSD support up into Valgrind project? Yuri From asmodai at in-nomine.org Thu Oct 2 07:34:57 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Thu Oct 2 07:35:05 2008 Subject: valgrind on FreeBSD 7 In-Reply-To: <48E474C7.8050507@rawbw.com> References: <200803172156.37407.modelnine@modelnine.org> <20080317214510.G89676@odysseus.silby.com> <20080413103351.GA1382@dose.local.invalid> <4803C3B4.4000405@delphij.net> <48E474C7.8050507@rawbw.com> Message-ID: <20081002073454.GA30869@nexus.in-nomine.org> -On [20081002 09:28], Yuri (yuri@rawbw.com) wrote: >Any developments in Valgrind/Callgrind on FreeBSD? I have been working on/off on it. I am trying to find my work in progress sources, but I think they got lost when a hard disk died. This is all I have found: http://www.in-nomine.org/~asmodai/valgrind/valgrind-trunk-for-freebsd.diff -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B Experience keeps a dear school, yet Fools will learn in no other. From joel at FreeBSD.org Thu Oct 2 08:13:25 2008 From: joel at FreeBSD.org (Joel Dahl) Date: Thu Oct 2 08:13:56 2008 Subject: Hardware support for AMD Geode CS5536 audio? In-Reply-To: <32225808.21222938312495.JavaMail.root@wombat.diezmil.com> References: <32225808.21222938312495.JavaMail.root@wombat.diezmil.com> Message-ID: <48E47B3C.60104@FreeBSD.org> ancelgray@yahoo.com skrev: > Another question. How does one get the FreeBSD people to put this > driver into their OS? Make the source code available somewhere and hopefully someone (usually ariff@ or mav@) with knowledge about the sound subsystem will review it. Just make sure that you follow our usual style(9) guidelines etc etc. -- Joel From Alexander at Leidinger.net Thu Oct 2 07:26:18 2008 From: Alexander at Leidinger.net (Alexander Leidinger) Date: Thu Oct 2 12:00:40 2008 Subject: Hardware support for AMD Geode CS5536 audio? In-Reply-To: <32225808.21222938312495.JavaMail.root@wombat.diezmil.com> References: <32225808.21222938312495.JavaMail.root@wombat.diezmil.com> Message-ID: <20081002090911.396629rcs0kdofwg@webmail.leidinger.net> Quoting ancelgray@yahoo.com (from Thu, 2 Oct 2008 05:05:12 -0400): > To All, > > I finally have a working CS5536 audio driver for FreeBSD 6.2. The > filename is snd_amd5536.ko and I will be making it available > shortly. I have been using an ALIX-1C motherboard for my testing. > Anyone have other hardware that they want to test it on? > > Another question. How does one get the FreeBSD people to put this > driver into their OS? You have to generate a patch for 8-current and send it to multimedia@FreeBSD.org for review. There may be some improvement suggestions or bug reports then. Fix the issues and generate a new patch. When there are no issues anymore, send a PR (problem report) with your patch. If you are lucky, a committer will take care of the PR (or he may already approach you before you send the PR). Bye, Alexander. -- It has long been an axiom of mine that the little things are infinitely the most important. -- Sir Arthur Conan Doyle, "A Case of Identity" http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 From alec-keyword-freebsd.befd64 at SetFilePointer.com Thu Oct 2 14:27:18 2008 From: alec-keyword-freebsd.befd64 at SetFilePointer.com (Alec Kloss) Date: Thu Oct 2 14:50:43 2008 Subject: Hardware support for AMD Geode CS5536 audio? In-Reply-To: <32225808.21222938312495.JavaMail.root@wombat.diezmil.com> References: <21541490.01216080059414.JavaMail.root@wombat.diezmil.com> <32225808.21222938312495.JavaMail.root@wombat.diezmil.com> Message-ID: <20081002140035.GM23927@hamlet.SetFilePointer.com> On 2008-10-02 05:05, ancelgray@yahoo.com wrote: > To All, > > I finally have a working CS5536 audio driver for FreeBSD 6.2. The filename is snd_amd5536.ko and I will be making it available shortly. I have been using an ALIX-1C motherboard for my testing. Anyone have other hardware that they want to test it on? [chop] I've got hardware I'd like to test on, but it's either running 8-CURRENT or 7-STABLE, not 6.x. Once source code is available, I'd be happy to take a shot at porting it up to the newer codebase. -- Alec Kloss alec@SetFilePointer.com IM: angryspamhater@yahoo.com PGP key at http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xA241980E "No Bunny!" -- Simon, http://wiki.adultswim.com/xwiki/bin/Frisky+Dingo/Simon -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081002/67ece8d6/attachment.pgp From imp at bsdimp.com Fri Oct 3 05:33:42 2008 From: imp at bsdimp.com (M. Warner Losh) Date: Fri Oct 3 05:33:50 2008 Subject: i386/127710: My driver PCI probe is not called for my correspondingdevice ID and Vendor ID In-Reply-To: <68C9F31EF19DB6448F515EF294028FDEE99BCE@chn-hclt-evs05.HCLT.CORP.HCL.IN> References: <68C9F31EF19DB6448F515EF294028FDEE999AB@chn-hclt-evs05.HCLT.CORP.HCL.IN> <200810011127.14593.jhb@freebsd.org> <68C9F31EF19DB6448F515EF294028FDEE99BCE@chn-hclt-evs05.HCLT.CORP.HCL.IN> Message-ID: <20081002.233328.-432820840.imp@bsdimp.com> In message: <68C9F31EF19DB6448F515EF294028FDEE99BCE@chn-hclt-evs05.HCLT.CORP.HCL.IN> "Bagavathy Kumar Mahendran " writes: : : Dear Baldwin, : Thanks for your support .but my pci probe function is not : getting called for my device id and vendor id. Because pccbb driver : already sets the device_set_desc as PCI-CardBus Bridge. So is there any : other option for me to make my_pciprobe function to be called for my : corresponding device id and vendor id. That's not why your probe isn't called. Setting a description is standard behavior for the probe routine. Are you sure that the device probe routine is getting called at all for any device? Have you tried just leaving cbb out of the kernel? I recently fixed the original problem in cbb (the fact it doesn't check the bridge type too), maybe you could try to pick up that fix as well? Warner : Thanks, : : Regards, : Bagavathy kumar .M : : : : -----Original Message----- : From: John Baldwin [mailto:jhb@freebsd.org] : Sent: Wednesday, October 01, 2008 8:57 PM : To: freebsd-hackers@freebsd.org : Cc: Bagavathy Kumar Mahendran ; Warner Losh : Subject: Re: FW: i386/127710: My driver PCI probe is not called for my : correspondingdevice ID and Vendor ID : : On Wednesday 01 October 2008 08:50:15 am Bagavathy Kumar Mahendran : wrote: : > : > Dear All, : > Iam writing a new driver for a SAS/SATA Controller having : a : > Class ID -0x01 : > Sub Class - 0x07 : > Programming Interface - 0x00 : > : > Hence instead of my probe function the Static build Card Bus Driver : cbb : > is attaching just by simply checking sub class 0x07 and programming : > interface 0x00.hence my probe gets failed. Kindly help me in resolving : > this .what I thought is to add the card bus driver a checking of CLASS : > ID in its pci probe function. : : The pccbb driver returns BUS_PROBE_DEFAULT (it should probably return : GENERIC : in the case where it matches only on class codes). Your driver just : needs to : return a numerically higher value (but still < 0) to claim the device. : You : can probably use BUS_PROBE_VENDOR or BUS_PROBE_DEFAULT + 1. : : -- : John Baldwin : : DISCLAIMER: : ----------------------------------------------------------------------------------------------------------------------- : : The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. : It shall not attach any liability on the originator or HCL or its affiliates. Any views or opinions presented in : this email are solely those of the author and may not necessarily reflect the opinions of HCL or its affiliates. : Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of : this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have : received this email in error please delete it and notify the sender immediately. Before opening any mail and : attachments please check them for viruses and defect. : : ----------------------------------------------------------------------------------------------------------------------- : : From danny at cs.huji.ac.il Fri Oct 3 08:00:45 2008 From: danny at cs.huji.ac.il (Danny Braniss) Date: Fri Oct 3 08:00:58 2008 Subject: bad NFS/UDP performance In-Reply-To: References: <20080926081806.GA19055@icarus.home.lan> <20080926095230.GA20789@icarus.home.lan> Message-ID: > > it more difficult than I expected. > > for one, the kernel date was missleading, the actual source update is the key, so > > the window of changes is now 28/July to 19/August. I have the diffs, but nothing > > yet seems relevant. > > > > on the other hand, I tried NFS/TCP, and there things seem ok, ie the 'good' and the 'bad' > > give the same throughput, which seem to point to UDP changes ... > > Can you post the network-numbers? so I ran some more test, these are for writes IO: server is a NetApp: kernel from 18/08/08 00:00:0 : /----- UDP ----//---- TCP -------/ 1*512 38528 0.19s 83.50MB 0.20s 80.82MB/s 2*512 19264 0.21s 76.83MB 0.21s 77.57MB/s 4*512 9632 0.19s 85.51MB 0.22s 73.13MB/s 8*512 4816 0.19s 83.76MB 0.21s 75.84MB/s 16*512 2408 0.19s 83.99MB 0.21s 77.18MB/s 32*512 1204 0.19s 84.45MB 0.22s 71.79MB/s 64*512 602 0.20s 79.98MB 0.20s 78.44MB/s 128*512 301 0.18s 86.51MB 0.22s 71.53MB/s 256*512 150 0.19s 82.83MB 0.20s 78.86MB/s 512*512 75 0.19s 82.77MB 0.21s 76.39MB/s 1024*512 37 0.19s 85.62MB 0.21s 76.64MB/s 2048*512 18 0.21s 77.72MB 0.20s 80.30MB/s 4096*512 9 0.26s 61.06MB 0.30s 53.79MB/s 8192*512 4 0.83s 19.20MB 0.41s 39.12MB/s 16384*512 2 0.84s 19.01MB 0.41s 39.03MB/s 32768*512 1 0.82s 19.59MB 0.39s 40.89MB/s kernel from 19/08/08 00:00:00: 1*512 38528 0.45s 35.59MB 0.20s 81.43MB/s 2*512 19264 0.45s 35.56MB 0.20s 79.24MB/s 4*512 9632 0.49s 32.66MB 0.22s 73.72MB/s 8*512 4816 0.47s 34.06MB 0.21s 75.52MB/s 16*512 2408 0.53s 30.16MB 0.22s 72.58MB/s 32*512 1204 0.31s 51.68MB 0.40s 40.14MB/s 64*512 602 0.43s 37.23MB 0.25s 63.57MB/s 128*512 301 0.51s 31.39MB 0.26s 62.70MB/s 256*512 150 0.47s 34.02MB 0.23s 69.06MB/s 512*512 75 0.47s 34.01MB 0.23s 70.52MB/s 1024*512 37 0.53s 30.12MB 0.22s 73.01MB/s 2048*512 18 0.55s 29.07MB 0.23s 70.64MB/s 4096*512 9 0.46s 34.69MB 0.21s 75.92MB/s 8192*512 4 0.81s 19.66MB 0.43s 36.89MB/s 16384*512 2 0.80s 19.99MB 0.40s 40.29MB/s 32768*512 1 1.11s 14.41MB 0.38s 42.56MB/s From rwatson at FreeBSD.org Fri Oct 3 08:15:02 2008 From: rwatson at FreeBSD.org (Robert Watson) Date: Fri Oct 3 08:15:14 2008 Subject: bad NFS/UDP performance In-Reply-To: References: <20080926081806.GA19055@icarus.home.lan> <20080926095230.GA20789@icarus.home.lan> Message-ID: On Fri, 3 Oct 2008, Danny Braniss wrote: >>> it more difficult than I expected. >>> for one, the kernel date was missleading, the actual source update is the key, so >>> the window of changes is now 28/July to 19/August. I have the diffs, but nothing >>> yet seems relevant. >>> >>> on the other hand, I tried NFS/TCP, and there things seem ok, ie the >>> 'good' and the 'bad' give the same throughput, which seem to point to UDP >>> changes ... >> >> Can you post the network-numbers? > so I ran some more test, these are for writes IO: OK, so it looks like this was almost certainly the rwlock change. What happens if you pretty much universally substitute the following in udp_usrreq.c: Currently Change to --------- --------- INP_RLOCK INP_WLOCK INP_RUNLOCK INP_WUNLOCK INP_RLOCK_ASSERT INP_WLOCK_ASSERT Robert N M Watson Computer Laboratory University of Cambridge > > server is a NetApp: > > kernel from 18/08/08 00:00:0 : > /----- UDP ----//---- TCP -------/ > 1*512 38528 0.19s 83.50MB 0.20s 80.82MB/s > 2*512 19264 0.21s 76.83MB 0.21s 77.57MB/s > 4*512 9632 0.19s 85.51MB 0.22s 73.13MB/s > 8*512 4816 0.19s 83.76MB 0.21s 75.84MB/s > 16*512 2408 0.19s 83.99MB 0.21s 77.18MB/s > 32*512 1204 0.19s 84.45MB 0.22s 71.79MB/s > 64*512 602 0.20s 79.98MB 0.20s 78.44MB/s > 128*512 301 0.18s 86.51MB 0.22s 71.53MB/s > 256*512 150 0.19s 82.83MB 0.20s 78.86MB/s > 512*512 75 0.19s 82.77MB 0.21s 76.39MB/s > 1024*512 37 0.19s 85.62MB 0.21s 76.64MB/s > 2048*512 18 0.21s 77.72MB 0.20s 80.30MB/s > 4096*512 9 0.26s 61.06MB 0.30s 53.79MB/s > 8192*512 4 0.83s 19.20MB 0.41s 39.12MB/s > 16384*512 2 0.84s 19.01MB 0.41s 39.03MB/s > 32768*512 1 0.82s 19.59MB 0.39s 40.89MB/s > > kernel from 19/08/08 00:00:00: > 1*512 38528 0.45s 35.59MB 0.20s 81.43MB/s > 2*512 19264 0.45s 35.56MB 0.20s 79.24MB/s > 4*512 9632 0.49s 32.66MB 0.22s 73.72MB/s > 8*512 4816 0.47s 34.06MB 0.21s 75.52MB/s > 16*512 2408 0.53s 30.16MB 0.22s 72.58MB/s > 32*512 1204 0.31s 51.68MB 0.40s 40.14MB/s > 64*512 602 0.43s 37.23MB 0.25s 63.57MB/s > 128*512 301 0.51s 31.39MB 0.26s 62.70MB/s > 256*512 150 0.47s 34.02MB 0.23s 69.06MB/s > 512*512 75 0.47s 34.01MB 0.23s 70.52MB/s > 1024*512 37 0.53s 30.12MB 0.22s 73.01MB/s > 2048*512 18 0.55s 29.07MB 0.23s 70.64MB/s > 4096*512 9 0.46s 34.69MB 0.21s 75.92MB/s > 8192*512 4 0.81s 19.66MB 0.43s 36.89MB/s > 16384*512 2 0.80s 19.99MB 0.40s 40.29MB/s > 32768*512 1 1.11s 14.41MB 0.38s 42.56MB/s > > > > > From danny at cs.huji.ac.il Fri Oct 3 09:02:46 2008 From: danny at cs.huji.ac.il (Danny Braniss) Date: Fri Oct 3 09:02:53 2008 Subject: bad NFS/UDP performance In-Reply-To: References: <20080926081806.GA19055@icarus.home.lan> <20080926095230.GA20789@icarus.home.lan> Message-ID: > > On Fri, 3 Oct 2008, Danny Braniss wrote: > > >>> it more difficult than I expected. > >>> for one, the kernel date was missleading, the actual source update is the key, so > >>> the window of changes is now 28/July to 19/August. I have the diffs, but nothing > >>> yet seems relevant. > >>> > >>> on the other hand, I tried NFS/TCP, and there things seem ok, ie the > >>> 'good' and the 'bad' give the same throughput, which seem to point to UDP > >>> changes ... > >> > >> Can you post the network-numbers? > > so I ran some more test, these are for writes IO: > > OK, so it looks like this was almost certainly the rwlock change. What > happens if you pretty much universally substitute the following in > udp_usrreq.c: > > Currently Change to > --------- --------- > INP_RLOCK INP_WLOCK > INP_RUNLOCK INP_WUNLOCK > INP_RLOCK_ASSERT INP_WLOCK_ASSERT > I guess you were almost certainly correct :-) I did the global subst. on the udp_usrreq.c from 19/08, __FBSDID("$FreeBSD: src/sys/netinet/udp_usrreq.c,v 1.218.2.3 2008/08/18 23:00:41 bz Exp $"); and now udp is fine again! danny > Robert N M Watson > Computer Laboratory > University of Cambridge > > > > > server is a NetApp: > > > > kernel from 18/08/08 00:00:0 : > > /----- UDP ----//---- TCP -------/ > > 1*512 38528 0.19s 83.50MB 0.20s 80.82MB/s > > 2*512 19264 0.21s 76.83MB 0.21s 77.57MB/s > > 4*512 9632 0.19s 85.51MB 0.22s 73.13MB/s > > 8*512 4816 0.19s 83.76MB 0.21s 75.84MB/s > > 16*512 2408 0.19s 83.99MB 0.21s 77.18MB/s > > 32*512 1204 0.19s 84.45MB 0.22s 71.79MB/s > > 64*512 602 0.20s 79.98MB 0.20s 78.44MB/s > > 128*512 301 0.18s 86.51MB 0.22s 71.53MB/s > > 256*512 150 0.19s 82.83MB 0.20s 78.86MB/s > > 512*512 75 0.19s 82.77MB 0.21s 76.39MB/s > > 1024*512 37 0.19s 85.62MB 0.21s 76.64MB/s > > 2048*512 18 0.21s 77.72MB 0.20s 80.30MB/s > > 4096*512 9 0.26s 61.06MB 0.30s 53.79MB/s > > 8192*512 4 0.83s 19.20MB 0.41s 39.12MB/s > > 16384*512 2 0.84s 19.01MB 0.41s 39.03MB/s > > 32768*512 1 0.82s 19.59MB 0.39s 40.89MB/s > > > > kernel from 19/08/08 00:00:00: > > 1*512 38528 0.45s 35.59MB 0.20s 81.43MB/s > > 2*512 19264 0.45s 35.56MB 0.20s 79.24MB/s > > 4*512 9632 0.49s 32.66MB 0.22s 73.72MB/s > > 8*512 4816 0.47s 34.06MB 0.21s 75.52MB/s > > 16*512 2408 0.53s 30.16MB 0.22s 72.58MB/s > > 32*512 1204 0.31s 51.68MB 0.40s 40.14MB/s > > 64*512 602 0.43s 37.23MB 0.25s 63.57MB/s > > 128*512 301 0.51s 31.39MB 0.26s 62.70MB/s > > 256*512 150 0.47s 34.02MB 0.23s 69.06MB/s > > 512*512 75 0.47s 34.01MB 0.23s 70.52MB/s > > 1024*512 37 0.53s 30.12MB 0.22s 73.01MB/s > > 2048*512 18 0.55s 29.07MB 0.23s 70.64MB/s > > 4096*512 9 0.46s 34.69MB 0.21s 75.92MB/s > > 8192*512 4 0.81s 19.66MB 0.43s 36.89MB/s > > 16384*512 2 0.80s 19.99MB 0.40s 40.29MB/s > > 32768*512 1 1.11s 14.41MB 0.38s 42.56MB/s > > > > > > > > > > From rwatson at FreeBSD.org Fri Oct 3 09:06:17 2008 From: rwatson at FreeBSD.org (Robert Watson) Date: Fri Oct 3 09:06:24 2008 Subject: bad NFS/UDP performance In-Reply-To: References: <20080926081806.GA19055@icarus.home.lan> <20080926095230.GA20789@icarus.home.lan> Message-ID: On Fri, 3 Oct 2008, Danny Braniss wrote: >> OK, so it looks like this was almost certainly the rwlock change. What >> happens if you pretty much universally substitute the following in >> udp_usrreq.c: >> >> Currently Change to >> --------- --------- >> INP_RLOCK INP_WLOCK >> INP_RUNLOCK INP_WUNLOCK >> INP_RLOCK_ASSERT INP_WLOCK_ASSERT > > I guess you were almost certainly correct :-) I did the global subst. on the > udp_usrreq.c from 19/08, __FBSDID("$FreeBSD: src/sys/netinet/udp_usrreq.c,v > 1.218.2.3 2008/08/18 23:00:41 bz Exp $"); and now udp is fine again! OK. This is a change I'd rather not back out since it significantly improves performance for many other UDP workloads, so we need to figure out why it's hurting us so much here so that we know if there are reasonable alternatives. Would it be possible for you to do a run of the workload with both kernels using LOCK_PROFILING around the benchmark, and then we can compare lock contention in the two cases? What we often find is that relieving contention at one point causes new contention at another point, and if the primitive used at that point handles contention less well for whatever reason, performance can be reduced rather than improved. So maybe we're looking at an issue in the dispatched UDP code from so_upcall? Another less satisfying (and fundamentally more difficult) answer might be "something to do with the scheduler", but a bit more analysis may shed some light. Robert N M Watson Computer Laboratory University of Cambridge > > danny > > >> Robert N M Watson >> Computer Laboratory >> University of Cambridge >> >>> >>> server is a NetApp: >>> >>> kernel from 18/08/08 00:00:0 : >>> /----- UDP ----//---- TCP -------/ >>> 1*512 38528 0.19s 83.50MB 0.20s 80.82MB/s >>> 2*512 19264 0.21s 76.83MB 0.21s 77.57MB/s >>> 4*512 9632 0.19s 85.51MB 0.22s 73.13MB/s >>> 8*512 4816 0.19s 83.76MB 0.21s 75.84MB/s >>> 16*512 2408 0.19s 83.99MB 0.21s 77.18MB/s >>> 32*512 1204 0.19s 84.45MB 0.22s 71.79MB/s >>> 64*512 602 0.20s 79.98MB 0.20s 78.44MB/s >>> 128*512 301 0.18s 86.51MB 0.22s 71.53MB/s >>> 256*512 150 0.19s 82.83MB 0.20s 78.86MB/s >>> 512*512 75 0.19s 82.77MB 0.21s 76.39MB/s >>> 1024*512 37 0.19s 85.62MB 0.21s 76.64MB/s >>> 2048*512 18 0.21s 77.72MB 0.20s 80.30MB/s >>> 4096*512 9 0.26s 61.06MB 0.30s 53.79MB/s >>> 8192*512 4 0.83s 19.20MB 0.41s 39.12MB/s >>> 16384*512 2 0.84s 19.01MB 0.41s 39.03MB/s >>> 32768*512 1 0.82s 19.59MB 0.39s 40.89MB/s >>> >>> kernel from 19/08/08 00:00:00: >>> 1*512 38528 0.45s 35.59MB 0.20s 81.43MB/s >>> 2*512 19264 0.45s 35.56MB 0.20s 79.24MB/s >>> 4*512 9632 0.49s 32.66MB 0.22s 73.72MB/s >>> 8*512 4816 0.47s 34.06MB 0.21s 75.52MB/s >>> 16*512 2408 0.53s 30.16MB 0.22s 72.58MB/s >>> 32*512 1204 0.31s 51.68MB 0.40s 40.14MB/s >>> 64*512 602 0.43s 37.23MB 0.25s 63.57MB/s >>> 128*512 301 0.51s 31.39MB 0.26s 62.70MB/s >>> 256*512 150 0.47s 34.02MB 0.23s 69.06MB/s >>> 512*512 75 0.47s 34.01MB 0.23s 70.52MB/s >>> 1024*512 37 0.53s 30.12MB 0.22s 73.01MB/s >>> 2048*512 18 0.55s 29.07MB 0.23s 70.64MB/s >>> 4096*512 9 0.46s 34.69MB 0.21s 75.92MB/s >>> 8192*512 4 0.81s 19.66MB 0.43s 36.89MB/s >>> 16384*512 2 0.80s 19.99MB 0.40s 40.29MB/s >>> 32768*512 1 1.11s 14.41MB 0.38s 42.56MB/s >>> >>> >>> >>> >>> > > > From danny at cs.huji.ac.il Fri Oct 3 09:17:46 2008 From: danny at cs.huji.ac.il (Danny Braniss) Date: Fri Oct 3 09:17:59 2008 Subject: bad NFS/UDP performance In-Reply-To: References: <20080926081806.GA19055@icarus.home.lan> <20080926095230.GA20789@icarus.home.lan> Message-ID: > > On Fri, 3 Oct 2008, Danny Braniss wrote: > > >> OK, so it looks like this was almost certainly the rwlock change. What > >> happens if you pretty much universally substitute the following in > >> udp_usrreq.c: > >> > >> Currently Change to > >> --------- --------- > >> INP_RLOCK INP_WLOCK > >> INP_RUNLOCK INP_WUNLOCK > >> INP_RLOCK_ASSERT INP_WLOCK_ASSERT > > > > I guess you were almost certainly correct :-) I did the global subst. on the > > udp_usrreq.c from 19/08, __FBSDID("$FreeBSD: src/sys/netinet/udp_usrreq.c,v > > 1.218.2.3 2008/08/18 23:00:41 bz Exp $"); and now udp is fine again! > > OK. This is a change I'd rather not back out since it significantly improves > performance for many other UDP workloads, so we need to figure out why it's > hurting us so much here so that we know if there are reasonable alternatives. > > Would it be possible for you to do a run of the workload with both kernels > using LOCK_PROFILING around the benchmark, and then we can compare lock > contention in the two cases? What we often find is that relieving contention > at one point causes new contention at another point, and if the primitive used > at that point handles contention less well for whatever reason, performance > can be reduced rather than improved. So maybe we're looking at an issue in > the dispatched UDP code from so_upcall? Another less satisfying (and > fundamentally more difficult) answer might be "something to do with the > scheduler", but a bit more analysis may shed some light. gladly, but have no idea how to do LOCK_PROFILING, so some pointers would be helpfull. as a side note, many years ago I checked out NFS/TCP and it was really bad, I even remember NetApp telling us to drop TCP, but now, things look rather better. Wonder what caused it. danny From danny at cs.huji.ac.il Fri Oct 3 09:21:10 2008 From: danny at cs.huji.ac.il (Danny Braniss) Date: Fri Oct 3 09:21:23 2008 Subject: bad NFS/UDP performance In-Reply-To: References: <20080926081806.GA19055@icarus.home.lan> <20080926095230.GA20789@icarus.home.lan> Message-ID: forget it about LOCK_PROFILING, I'm RTFM now :-) though some hints on values might be helpful. have a nice weekend, danny From rwatson at FreeBSD.org Fri Oct 3 09:25:24 2008 From: rwatson at FreeBSD.org (Robert Watson) Date: Fri Oct 3 09:25:37 2008 Subject: bad NFS/UDP performance In-Reply-To: References: <20080926081806.GA19055@icarus.home.lan> <20080926095230.GA20789@icarus.home.lan> Message-ID: On Fri, 3 Oct 2008, Danny Braniss wrote: > gladly, but have no idea how to do LOCK_PROFILING, so some pointers would be > helpfull. The LOCK_PROFILING(9) man page isn't a bad starting point -- I find that the defaults work fine most of the time, so just use them. Turn the enable syscl on just before you begin a run, and turn it off immediately afterwards. Make sure to reset between reruns (rebooting to a new kernel is fine too!). > as a side note, many years ago I checked out NFS/TCP and it was really bad, > I even remember NetApp telling us to drop TCP, but now, things look rather > better. Wonder what caused it. Well, the virtues of TCP become more apparent with higher network speeds, as the logic to fill pipes using TCP, manage flow control, etc, is a lot more sophisticated than what's in the RPC code for using UDP. The downsides to UDP are also becoming more apparent: as network speeds go up, fragmented UDP risks IP ID collisions which could lead to data corruption, or at the very least, dropped packets. We have changed the default for NFSv3 mounts to TCP in 8.x, and talked about doing it for 7.1; unfortunately the timing wasn't quite right, so it most likely will appear in 7.2. Robert N M Watson Computer Laboratory University of Cambridge From bagavathykumar.m at hcl.in Fri Oct 3 04:56:40 2008 From: bagavathykumar.m at hcl.in (Bagavathy Kumar Mahendran ) Date: Fri Oct 3 11:14:19 2008 Subject: FW: i386/127710: My driver PCI probe is not called for my correspondingdevice ID and Vendor ID In-Reply-To: <200810011127.14593.jhb@freebsd.org> References: <68C9F31EF19DB6448F515EF294028FDEE999AB@chn-hclt-evs05.HCLT.CORP.HCL.IN> <200810011127.14593.jhb@freebsd.org> Message-ID: <68C9F31EF19DB6448F515EF294028FDEE99BCE@chn-hclt-evs05.HCLT.CORP.HCL.IN> Dear Baldwin, Thanks for your support .but my pci probe function is not getting called for my device id and vendor id. Because pccbb driver already sets the device_set_desc as PCI-CardBus Bridge. So is there any other option for me to make my_pciprobe function to be called for my corresponding device id and vendor id. Thanks, Regards, Bagavathy kumar .M -----Original Message----- From: John Baldwin [mailto:jhb@freebsd.org] Sent: Wednesday, October 01, 2008 8:57 PM To: freebsd-hackers@freebsd.org Cc: Bagavathy Kumar Mahendran ; Warner Losh Subject: Re: FW: i386/127710: My driver PCI probe is not called for my correspondingdevice ID and Vendor ID On Wednesday 01 October 2008 08:50:15 am Bagavathy Kumar Mahendran wrote: > > Dear All, > Iam writing a new driver for a SAS/SATA Controller having a > Class ID -0x01 > Sub Class - 0x07 > Programming Interface - 0x00 > > Hence instead of my probe function the Static build Card Bus Driver cbb > is attaching just by simply checking sub class 0x07 and programming > interface 0x00.hence my probe gets failed. Kindly help me in resolving > this .what I thought is to add the card bus driver a checking of CLASS > ID in its pci probe function. The pccbb driver returns BUS_PROBE_DEFAULT (it should probably return GENERIC in the case where it matches only on class codes). Your driver just needs to return a numerically higher value (but still < 0) to claim the device. You can probably use BUS_PROBE_VENDOR or BUS_PROBE_DEFAULT + 1. -- John Baldwin DISCLAIMER: ----------------------------------------------------------------------------------------------------------------------- The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or HCL or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any mail and attachments please check them for viruses and defect. ----------------------------------------------------------------------------------------------------------------------- From bagavathykumar.m at hcl.in Fri Oct 3 05:42:19 2008 From: bagavathykumar.m at hcl.in (Bagavathy Kumar Mahendran ) Date: Fri Oct 3 11:24:24 2008 Subject: i386/127710: My driver PCI probe is not called for mycorrespondingdevice ID and Vendor ID In-Reply-To: <20081002.233328.-432820840.imp@bsdimp.com> References: <68C9F31EF19DB6448F515EF294028FDEE999AB@chn-hclt-evs05.HCLT.CORP .HCL.IN><200810011127.14593.jhb@freebsd.org><68C9F31EF19DB6448F515EF294028F DEE99BCE@chn-hclt-evs05.HCLT.CORP.HCL.IN> <20081002.233328.-432820840.imp@bsdimp.com> Message-ID: <68C9F31EF19DB6448F515EF294028FDEE99C6D@chn-hclt-evs05.HCLT.CORP.HCL.IN> Yes . my pci_probe is called for all the other bridge interfaces but its not called for my PCI Card(Device ID and Vendor ID) instead of that Card bus pci Probe is called while loading my driver as kldload. I removed cbb driver out of kernel and tried. It works perfectly for me. I think Class Type of the bridge has to be checked during the probe of cbb-driver. Please provide me the recent fix of cbb driver so that I can use it. Thanks With Regards, Bagavathy kumar .M -----Original Message----- From: M. Warner Losh [mailto:imp@bsdimp.com] Sent: Friday, October 03, 2008 11:03 AM To: Bagavathy Kumar Mahendran Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org Subject: Re: i386/127710: My driver PCI probe is not called for mycorrespondingdevice ID and Vendor ID In message: <68C9F31EF19DB6448F515EF294028FDEE99BCE@chn-hclt-evs05.HCLT.CORP.HCL.IN> "Bagavathy Kumar Mahendran " writes: : : Dear Baldwin, : Thanks for your support .but my pci probe function is not : getting called for my device id and vendor id. Because pccbb driver : already sets the device_set_desc as PCI-CardBus Bridge. So is there any : other option for me to make my_pciprobe function to be called for my : corresponding device id and vendor id. That's not why your probe isn't called. Setting a description is standard behavior for the probe routine. Are you sure that the device probe routine is getting called at all for any device? Have you tried just leaving cbb out of the kernel? I recently fixed the original problem in cbb (the fact it doesn't check the bridge type too), maybe you could try to pick up that fix as well? Warner : Thanks, : : Regards, : Bagavathy kumar .M : : : : -----Original Message----- : From: John Baldwin [mailto:jhb@freebsd.org] : Sent: Wednesday, October 01, 2008 8:57 PM : To: freebsd-hackers@freebsd.org : Cc: Bagavathy Kumar Mahendran ; Warner Losh : Subject: Re: FW: i386/127710: My driver PCI probe is not called for my : correspondingdevice ID and Vendor ID : : On Wednesday 01 October 2008 08:50:15 am Bagavathy Kumar Mahendran : wrote: : > : > Dear All, : > Iam writing a new driver for a SAS/SATA Controller having : a : > Class ID -0x01 : > Sub Class - 0x07 : > Programming Interface - 0x00 : > : > Hence instead of my probe function the Static build Card Bus Driver : cbb : > is attaching just by simply checking sub class 0x07 and programming : > interface 0x00.hence my probe gets failed. Kindly help me in resolving : > this .what I thought is to add the card bus driver a checking of CLASS : > ID in its pci probe function. : : The pccbb driver returns BUS_PROBE_DEFAULT (it should probably return : GENERIC : in the case where it matches only on class codes). Your driver just : needs to : return a numerically higher value (but still < 0) to claim the device. : You : can probably use BUS_PROBE_VENDOR or BUS_PROBE_DEFAULT + 1. : : -- : John Baldwin : : DISCLAIMER: : ------------------------------------------------------------------------ ----------------------------------------------- : : The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. : It shall not attach any liability on the originator or HCL or its affiliates. Any views or opinions presented in : this email are solely those of the author and may not necessarily reflect the opinions of HCL or its affiliates. : Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of : this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have : received this email in error please delete it and notify the sender immediately. Before opening any mail and : attachments please check them for viruses and defect. : : ------------------------------------------------------------------------ ----------------------------------------------- : : From mboxindia at gmail.com Fri Oct 3 11:48:00 2008 From: mboxindia at gmail.com (Srinivas) Date: Fri Oct 3 11:48:07 2008 Subject: options in configuration file Message-ID: Hello, May be these are beginner questions ... could you plz answer the following questions? I think "options" line and "device" line are in the configuration file, in order to support those features and devices. I see that sed script will parse that file. Could you plz let me know what will be done in this phase and how these lines will be transferred into gcc define directives(in the case of options) and inclusion of source files for compilation(in case of device). I have seen lint in some Makefiles, but dont know why it was used. Why is lint used? The "device" line adds device support to the kernel. What exactly does this mean. A more basic question is: how the devices are detected initially by the FreeBSD with the aid of hardware and bios? I think this is a broad topic. Could you plz provide a link if there is any info, you know, in the net? Thanks, Srinivas From danny at cs.huji.ac.il Fri Oct 3 13:26:54 2008 From: danny at cs.huji.ac.il (Danny Braniss) Date: Fri Oct 3 13:27:02 2008 Subject: bad NFS/UDP performance In-Reply-To: References: <20080926081806.GA19055@icarus.home.lan> <20080926095230.GA20789@icarus.home.lan> Message-ID: > > On Fri, 3 Oct 2008, Danny Braniss wrote: > > > gladly, but have no idea how to do LOCK_PROFILING, so some pointers would be > > helpfull. > > The LOCK_PROFILING(9) man page isn't a bad starting point -- I find that the > defaults work fine most of the time, so just use them. Turn the enable syscl > on just before you begin a run, and turn it off immediately afterwards. Make > sure to reset between reruns (rebooting to a new kernel is fine too!). > in ftp://ftp.cs.huji.ac.il/users/danny/lock.prof there 3 files: 7.1-100 host connected at 100 running -prerelease 7.1-1000 same but connected at 1000 7.0-1000 -stable with your 'patch' at 100 my benchmark didn't suffer from the profiling, average was about 9. at 1000 the benchmark got realy hit, average was around 12 for the patched, and 4 for the unpatched (less than at 100). danny From watson at FreeBSD.org Fri Oct 3 18:55:15 2008 From: watson at FreeBSD.org (Robert Watson) Date: Fri Oct 3 18:55:29 2008 Subject: bad NFS/UDP performance In-Reply-To: References: <20080926081806.GA19055@icarus.home.lan> <20080926095230.GA20789@icarus.home.lan> Message-ID: On Fri, 3 Oct 2008, Danny Braniss wrote: >> On Fri, 3 Oct 2008, Danny Braniss wrote: >> >>> gladly, but have no idea how to do LOCK_PROFILING, so some pointers would >>> be helpfull. >> >> The LOCK_PROFILING(9) man page isn't a bad starting point -- I find that >> the defaults work fine most of the time, so just use them. Turn the enable >> syscl on just before you begin a run, and turn it off immediately >> afterwards. Make sure to reset between reruns (rebooting to a new kernel >> is fine too!). > > in ftp://ftp.cs.huji.ac.il/users/danny/lock.prof > there 3 files: > 7.1-100 host connected at 100 running -prerelease > 7.1-1000 same but connected at 1000 > 7.0-1000 -stable with your 'patch' > at 100 my benchmark didn't suffer from the profiling, average was about 9. > at 1000 the benchmark got realy hit, average was around 12 for the patched, > and 4 for the unpatched (less than at 100). Interesting. A bit of post-processing: robert@fledge:/tmp> cat 7.1-1000 | awk -F' ' '{print $3" "$9}' | sort -n | tail -10 2413283 /r+d/7/sys/kern/kern_mutex.c:141 2470096 /r+d/7/sys/nfsclient/nfs_socket.c:1218 2676282 /r+d/7/sys/net/route.c:293 2754866 /r+d/7/sys/kern/vfs_bio.c:1468 3196298 /r+d/7/sys/nfsclient/nfs_bio.c:1664 3318742 /r+d/7/sys/net/route.c:1584 3711139 /r+d/7/sys/dev/bge/if_bge.c:3287 3753518 /r+d/7/sys/net/if_ethersubr.c:405 3961312 /r+d/7/sys/nfsclient/nfs_subs.c:1066 10688531 /r+d/7/sys/dev/bge/if_bge.c:3726 robert@fledge:/tmp> cat 7.0-1000 | awk -F' ' '{print $3" "$9}' | sort -n | tail -10 468631 /r+d/hunt/src/sys/nfsclient/nfs_nfsiod.c:286 501989 /r+d/hunt/src/sys/nfsclient/nfs_vnops.c:1148 631587 /r+d/hunt/src/sys/nfsclient/nfs_socket.c:1198 701155 /r+d/hunt/src/sys/nfsclient/nfs_socket.c:1258 718211 /r+d/hunt/src/sys/kern/kern_mutex.c:141 1118711 /r+d/hunt/src/sys/nfsclient/nfs_bio.c:1664 1169125 /r+d/hunt/src/sys/nfsclient/nfs_subs.c:1066 1222867 /r+d/hunt/src/sys/kern/vfs_bio.c:1468 3876072 /r+d/hunt/src/sys/netinet/udp_usrreq.c:545 5198927 /r+d/hunt/src/sys/netinet/udp_usrreq.c:864 The first set above is with the unmodified 7-STABLE tree, the second with a reversion of read locking on the UDP inpcb. The big blinking sign of interest is that the bge interface lock is massively contended in the first set of output, and basically doesn't appear in the second. There are various reasons bge could stand out quite so much -- one possibly is that previously, the udp lock serialized all access to the interface from the send code, preventing the send and receive paths from contending. A few things to try: - Let's look compare the context switch rates on the two benchmarks. Could you run vmstat and look at the cpu cs line during the benchmarks and see how similar the two are as the benchmarks run? You'll want to run it with vmstat -w 1 and collect several samples per benchmark, since we're really interested in the distribution rather than an individual sample. - Is there any chance you could drop an if_em card into the same box and run the identical benchmarks with and without LOCK_PROFILING to see whether it behaves differently than bge when the patch is applied? if_em's interrupt handling is quite different, and may significantly affect lock use, and hence contention. Thanks, Robert N M Watson Computer Laboratory University of Cambridge From chuckr at telenix.org Fri Oct 3 19:33:06 2008 From: chuckr at telenix.org (Chuck Robey) Date: Fri Oct 3 19:33:13 2008 Subject: UCLogic tablets Message-ID: <48E67299.5080703@telenix.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I've just finished the first version of a Xorg driver for the UCLogic family of graphic tablets. It may well work for other tablets, if I could see what folks have, so I could program the names in. Anyhow, the UCLogic tablets are *very* widely OEMed, so I couldn't even begin to guess the tablet name you have, but if it USB-probes as having a Vendor name of UCLogic, then I think this might work for you. Anyone who might want to try it, you might mail me, I'd kinda like to beta-test it before I generally release it. I also did up a manual page. Write me if you're interested. Thanks -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkjmcpkACgkQz62J6PPcoOnd1QCdHnPhB31IIyTnlk6Ii8E3eEh/ oV4AoJEn4+jA/wJjw9o+5OGsQeohALYS =RmnH -----END PGP SIGNATURE----- From chuckr at telenix.org Fri Oct 3 19:34:39 2008 From: chuckr at telenix.org (Chuck Robey) Date: Fri Oct 3 19:34:47 2008 Subject: UCLogic tablets In-Reply-To: <48E67299.5080703@telenix.org> References: <48E67299.5080703@telenix.org> Message-ID: <48E672F7.4040103@telenix.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Chuck Robey wrote: > I've just finished the first version of a Xorg driver for the UCLogic family of > graphic tablets. It may well work for other tablets, if I could see what folks > have, so I could program the names in. > > Anyhow, the UCLogic tablets are *very* widely OEMed, so I couldn't even begin to > guess the tablet name you have, but if it USB-probes as having a Vendor name of > UCLogic, then I think this might work for you. Anyone who might want to try it, > you might mail me, I'd kinda like to beta-test it before I generally release it. > > I also did up a manual page. Write me if you're interested. > > Thanks I forgot to add, the reason for posting here is because I wrote it specifically (at first) for FreeBSD. _______________________________________________ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkjmcvcACgkQz62J6PPcoOmRRQCfed2tnNxlN6zI+2me/ubbDRpJ jzoAnRlCdVJW/4LsYB+uMhlAEie52FNs =u4bq -----END PGP SIGNATURE----- From freebsd-hackers at wheelhouse.org Fri Oct 3 21:11:11 2008 From: freebsd-hackers at wheelhouse.org (Jeff Wheelhouse) Date: Fri Oct 3 21:11:18 2008 Subject: Major SMP problems with lstat/namei In-Reply-To: <200810011717.35873.jhb@freebsd.org> References: <8185F68B-C443-4891-BEC2-5E3D453DDC93@wheelhouse.org> <200809241212.09920.jhb@freebsd.org> <200810011717.35873.jhb@freebsd.org> Message-ID: <85727D2B-5ABD-4A23-B1B4-80B00407D0FC@wheelhouse.org> On Oct 1, 2008, at 5:17 PM, John Baldwin wrote: > It sounds like you missed some of the dirhash changes somehow, as > dirhash no > longer has any lockmgr stuff in it (and only ever did in HEAD). I've > generated a patch though using svn. You can grab it from > http://www.FreeBSD.org/~jhb/patches/ufs_lookup7.patch Note that you > will > have to set vfs.lookup_shared=1 to enable shared locks (either > loader tunable > or sysctl). We lost our test server due to production needs, but should have it back tomorrow. In the mean time, I've downloaded the patch; it applied cleanly and I'm building the resulting kernel now. I'll report back once we get the chance to test it. Thanks, Jeff From danny at cs.huji.ac.il Sat Oct 4 06:40:48 2008 From: danny at cs.huji.ac.il (Danny Braniss) Date: Sat Oct 4 06:41:01 2008 Subject: bad NFS/UDP performance In-Reply-To: References: <20080926081806.GA19055@icarus.home.lan> <20080926095230.GA20789@icarus.home.lan> Message-ID: > > On Fri, 3 Oct 2008, Danny Braniss wrote: > > >> On Fri, 3 Oct 2008, Danny Braniss wrote: > >> > >>> gladly, but have no idea how to do LOCK_PROFILING, so some pointers would > >>> be helpfull. > >> > >> The LOCK_PROFILING(9) man page isn't a bad starting point -- I find that > >> the defaults work fine most of the time, so just use them. Turn the enable > >> syscl on just before you begin a run, and turn it off immediately > >> afterwards. Make sure to reset between reruns (rebooting to a new kernel > >> is fine too!). > > > > in ftp://ftp.cs.huji.ac.il/users/danny/lock.prof > > there 3 files: > > 7.1-100 host connected at 100 running -prerelease > > 7.1-1000 same but connected at 1000 > > 7.0-1000 -stable with your 'patch' > > at 100 my benchmark didn't suffer from the profiling, average was about 9. > > at 1000 the benchmark got realy hit, average was around 12 for the patched, > > and 4 for the unpatched (less than at 100). > > Interesting. A bit of post-processing: > > robert@fledge:/tmp> cat 7.1-1000 | awk -F' ' '{print $3" "$9}' | sort -n | > tail -10 > 2413283 /r+d/7/sys/kern/kern_mutex.c:141 > 2470096 /r+d/7/sys/nfsclient/nfs_socket.c:1218 > 2676282 /r+d/7/sys/net/route.c:293 > 2754866 /r+d/7/sys/kern/vfs_bio.c:1468 > 3196298 /r+d/7/sys/nfsclient/nfs_bio.c:1664 > 3318742 /r+d/7/sys/net/route.c:1584 > 3711139 /r+d/7/sys/dev/bge/if_bge.c:3287 > 3753518 /r+d/7/sys/net/if_ethersubr.c:405 > 3961312 /r+d/7/sys/nfsclient/nfs_subs.c:1066 > 10688531 /r+d/7/sys/dev/bge/if_bge.c:3726 > robert@fledge:/tmp> cat 7.0-1000 | awk -F' ' '{print $3" "$9}' | sort -n | > tail -10 > 468631 /r+d/hunt/src/sys/nfsclient/nfs_nfsiod.c:286 > 501989 /r+d/hunt/src/sys/nfsclient/nfs_vnops.c:1148 > 631587 /r+d/hunt/src/sys/nfsclient/nfs_socket.c:1198 > 701155 /r+d/hunt/src/sys/nfsclient/nfs_socket.c:1258 > 718211 /r+d/hunt/src/sys/kern/kern_mutex.c:141 > 1118711 /r+d/hunt/src/sys/nfsclient/nfs_bio.c:1664 > 1169125 /r+d/hunt/src/sys/nfsclient/nfs_subs.c:1066 > 1222867 /r+d/hunt/src/sys/kern/vfs_bio.c:1468 > 3876072 /r+d/hunt/src/sys/netinet/udp_usrreq.c:545 > 5198927 /r+d/hunt/src/sys/netinet/udp_usrreq.c:864 > > The first set above is with the unmodified 7-STABLE tree, the second with a > reversion of read locking on the UDP inpcb. The big blinking sign of interest > is that the bge interface lock is massively contended in the first set of > output, and basically doesn't appear in the second. There are various reasons > bge could stand out quite so much -- one possibly is that previously, the udp > lock serialized all access to the interface from the send code, preventing the > send and receive paths from contending. > > A few things to try: > > - Let's look compare the context switch rates on the two benchmarks. Could > you run vmstat and look at the cpu cs line during the benchmarks and see how > similar the two are as the benchmarks run? You'll want to run it with > vmstat -w 1 and collect several samples per benchmark, since we're really > interested in the distribution rather than an individual sample. > > - Is there any chance you could drop an if_em card into the same box and run > the identical benchmarks with and without LOCK_PROFILING to see whether it > behaves differently than bge when the patch is applied? if_em's interrupt > handling is quite different, and may significantly affect lock use, and > hence contention. at the moment, the best I can do is run it on a different hardware that has if_em, the results are in ftp://ftp.cs.huji.ac.il/users/danny/lock.prof/7.1-1000.em the benchmark ran better with the Intel NIC, averaged UDP 54MB/s, TCP 53MB/s (I get the same numbers with an older kernel). danny From chuckr at telenix.org Sun Oct 5 18:39:15 2008 From: chuckr at telenix.org (Chuck Robey) Date: Sun Oct 5 18:39:23 2008 Subject: touch screen recommendation? Message-ID: <48E90904.4020007@telenix.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I was wondering if anyone here had a recommendation for a touch screen, specifically to run on FreeBSD? Any user report? Thanks -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkjpCQQACgkQz62J6PPcoOl5MwCeKJAkbeQ/+GGpWX/b2CAV/F8J c1sAn0d7tXct21/SfdAYY2otfvFVuqK2 =XMw8 -----END PGP SIGNATURE----- From rea-fbsd at codelabs.ru Sun Oct 5 19:03:14 2008 From: rea-fbsd at codelabs.ru (Eygene Ryabinkin) Date: Sun Oct 5 19:03:22 2008 Subject: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages In-Reply-To: <4bESZpNwE3z/DdlE2fwK/BXzQSo@2MQ0uKCiT7mdMUuLeUzs8Nv3ToQ> References: <48DE5CC0.9000708@localhost.inse.ru> <48DF6735.4030906@quip.cz> <4bESZpNwE3z/DdlE2fwK/BXzQSo@2MQ0uKCiT7mdMUuLeUzs8Nv3ToQ> Message-ID: Skipped content of type multipart/mixed-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081005/00b09b4a/attachment.pgp From rea-fbsd at codelabs.ru Sun Oct 5 19:16:13 2008 From: rea-fbsd at codelabs.ru (Eygene Ryabinkin) Date: Sun Oct 5 19:16:25 2008 Subject: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages In-Reply-To: References: <48DE5CC0.9000708@localhost.inse.ru> <48DF6735.4030906@quip.cz> <4bESZpNwE3z/DdlE2fwK/BXzQSo@2MQ0uKCiT7mdMUuLeUzs8Nv3ToQ> Message-ID: Sun, Oct 05, 2008 at 11:03:17PM +0400, Eygene Ryabinkin wrote: > I had also changed the output format for pkg_audit, so I am attaching > another version of the second patch for the pkg_install bundle. One neat about new pkg_audit utility: if you already have the build directory for pkg_install in the /usr/obj, you should create subdirectory for the pkg_audit, ----- mkdir /usr/obj/usr/src/usr.sbin/pkg_install/audit ----- or completely remove /usr/obj/usr/src/usr.sbin/pkg_install World build should do it automatically, at least it worked for me. -- Eygene _ ___ _.--. # \`.|\..----...-'` `-._.-'_.-'` # Remember that it is hard / ' ` , __.--' # to read the on-line manual )/' _/ \ `-_, / # while single-stepping the kernel. `-'" `"\_ ,_.-;_.-\_ ', fsc/as # _.-'_./ {_.' ; / # -- FreeBSD Developers handbook {_.-``-' {_/ # -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081005/e2f0cfd4/attachment.pgp From rwatson at FreeBSD.org Sun Oct 5 22:21:06 2008 From: rwatson at FreeBSD.org (Robert Watson) Date: Sun Oct 5 22:21:19 2008 Subject: bad NFS/UDP performance In-Reply-To: References: <20080926081806.GA19055@icarus.home.lan> <20080926095230.GA20789@icarus.home.lan> Message-ID: On Sat, 4 Oct 2008, Danny Braniss wrote: > at the moment, the best I can do is run it on a different hardware that has > if_em, the results are in > ftp://ftp.cs.huji.ac.il/users/danny/lock.prof/7.1-1000.em the > benchmark ran better with the Intel NIC, averaged UDP 54MB/s, TCP 53MB/s (I > get the same numbers with an older kernel). Dear Danny: Unfortunately, I was left slightly unclear on the comparison you are making above. Could you confirm whether or not, with if_em, you see a performance regression using UDP NFS between 7.0-RELEASE and the most recent 7.1-STABLE, and if you do, whether or not the RLOCK->WLOCK change has any effect on performance? It would be nice to know on the same hardware but at least with different hardware we get a sense of whether or not this might affect other systems or whether it's limited to a narrower set of configurations. Thanks, Robert N M Watson Computer Laboratory University of Cambridge From brd at FreeBSD.org Mon Oct 6 03:10:18 2008 From: brd at FreeBSD.org (Brad Davis) Date: Mon Oct 6 03:10:30 2008 Subject: FreeBSD Status Reports due October 19th, 2008 Message-ID: <20081006031017.GA43283@valentine.liquidneon.com> Hi Everyone, It is that time again. We would like to remind everybody who has exciting news to share to write a report about their project. This is a good way to improve exposure of your work, receive feedback and help. We are looking forward to your reports. As always you can either use the template or the CGI generator and mail the output to monthly@ by Sunday October 19th, 2008. http://www.freebsd.org/news/status/ http://www.freebsd.org/cgi/monthly.cgi http://www.freebsd.org/news/status/report-sample.xml Regards, Brad Davis From rea-fbsd at codelabs.ru Mon Oct 6 05:23:40 2008 From: rea-fbsd at codelabs.ru (Eygene Ryabinkin) Date: Mon Oct 6 05:23:47 2008 Subject: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages In-Reply-To: <48E94281.8010300@quip.cz> References: <48DE5CC0.9000708@localhost.inse.ru> <48DF6735.4030906@quip.cz> <4bESZpNwE3z/DdlE2fwK/BXzQSo@2MQ0uKCiT7mdMUuLeUzs8Nv3ToQ> <48E94281.8010300@quip.cz> Message-ID: Miroslav, good day. Mon, Oct 06, 2008 at 12:41:05AM +0200, Miroslav Lachman wrote: > I am busy these days, but it is nice to read about your progress. I hope > I will get some time to test all of these large patches in a few days > and I will report back my experiences! Fine, thank you! I am re-CC'ing bug-followup@ to track this letter, since it contains some useful information that should go into GNATS. > One note before tests... do -n flag always download new INDEX file, or > is it possible to use one already existing in /usr/ports? Currently, it is downloads bzipped INDEX file to /var/db/portaudit every time, but it uses mirror mode, so if remote file hadn't changed at all, all network expences are just the HTTP's HEAD request and reply. I can add another variable to the portaudit to force the usage of the existing INDEX file, if it is needed. By the way, how are you keeping your INDEX file up to date (your proposed usage of 'pkg_version -I' implies that you're always rely on it)? I am just curious -- my INDEX files are almost always stay unupdated, even if I am using portupgrade. And there can be another way if one keeps ports tree updated: utility can use 'make' to determine the version that is currently available on the examined host. But downloading the INDEX file from the central server seemed to be the best way, since it almost always gives one the latest port versions, so I had implemented this in a first place. Don't know, however, how the badly the load to the central HTTP server will be raised. I am using just two first fields from the INDEX file, so I can use such a stripped file. For me, the reduction was about 6x: SIZE(INDEX-7.bz2) = 1126189, SIZE(INDEX-7.stripped.bz2) = 184345. I am CC'ing the portmgr team. Guys, could you quickly glance over these patches and determine if they are useful to the project in large? If yes, then may be such a stripped INDEX can be created on the FreeBSD servers (via cut -f1-2 -d'|' INDEX-N)? Thanks! -- Eygene _ ___ _.--. # \`.|\..----...-'` `-._.-'_.-'` # Remember that it is hard / ' ` , __.--' # to read the on-line manual )/' _/ \ `-_, / # while single-stepping the kernel. `-'" `"\_ ,_.-;_.-\_ ', fsc/as # _.-'_./ {_.' ; / # -- FreeBSD Developers handbook {_.-``-' {_/ # -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081006/46aa518a/attachment.pgp From imp at bsdimp.com Mon Oct 6 06:58:08 2008 From: imp at bsdimp.com (Warner Losh) Date: Mon Oct 6 06:58:15 2008 Subject: i386/127710: My driver PCI probe is not called for mycorrespondingdevice ID and Vendor ID In-Reply-To: <68C9F31EF19DB6448F515EF294028FDEE9A4A5@chn-hclt-evs05.HCLT.CORP.HCL.IN> References: <68C9F31EF19DB6448F515EF294028F DEE99BCE@chn-hclt-evs05.HCLT.CORP.HCL.IN> <20081002.233328.-432820840.imp@bsdimp.com> <68C9F31EF19DB6448F515EF294028FDEE9A4A5@chn-hclt-evs05.HCLT.CORP.HCL.IN> Message-ID: <20081006.005657.71122961.imp@bsdimp.com> > Thanks for your support. my probe is getting called for all > the bridges not for my pci device. so please provide the fix . > > OR > Is any other way available for making my driver to override the probe of > cbb driver for my corresponding device (With out changing cbb driver). If your probe returns a higher number that's negative, it will. Unless cbb is returning 0, your probe routine will get called. Make sure it isn't. Code inspection suggests that it isn't. Warner > With regards, > Bagavathy kumar .M > > -----Original Message----- > From: M. Warner Losh [mailto:imp@bsdimp.com] > Sent: Friday, October 03, 2008 11:03 AM > To: Bagavathy Kumar Mahendran > Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org > Subject: Re: i386/127710: My driver PCI probe is not called for > mycorrespondingdevice ID and Vendor ID > > In message: > <68C9F31EF19DB6448F515EF294028FDEE99BCE@chn-hclt-evs05.HCLT.CORP.HCL.IN> > "Bagavathy Kumar Mahendran " > writes: > : > : Dear Baldwin, > : Thanks for your support .but my pci probe function is > not > : getting called for my device id and vendor id. Because pccbb driver > : already sets the device_set_desc as PCI-CardBus Bridge. So is there > any > : other option for me to make my_pciprobe function to be called for my > : corresponding device id and vendor id. > > That's not why your probe isn't called. Setting a description is > standard behavior for the probe routine. Are you sure that the device > probe routine is getting called at all for any device? Have you tried > just leaving cbb out of the kernel? I recently fixed the original > problem in cbb (the fact it doesn't check the bridge type too), maybe > you could try to pick up that fix as well? > > Warner > > > : Thanks, > : > : Regards, > : Bagavathy kumar .M > : > : > : > : -----Original Message----- > : From: John Baldwin [mailto:jhb@freebsd.org] > : Sent: Wednesday, October 01, 2008 8:57 PM > : To: freebsd-hackers@freebsd.org > : Cc: Bagavathy Kumar Mahendran ; Warner Losh > : Subject: Re: FW: i386/127710: My driver PCI probe is not called for my > : correspondingdevice ID and Vendor ID > : > : On Wednesday 01 October 2008 08:50:15 am Bagavathy Kumar Mahendran > : wrote: > : > > : > Dear All, > : > Iam writing a new driver for a SAS/SATA Controller > having > : a > : > Class ID -0x01 > : > Sub Class - 0x07 > : > Programming Interface - 0x00 > : > > : > Hence instead of my probe function the Static build Card Bus Driver > : cbb > : > is attaching just by simply checking sub class 0x07 and programming > : > interface 0x00.hence my probe gets failed. Kindly help me in > resolving > : > this .what I thought is to add the card bus driver a checking of > CLASS > : > ID in its pci probe function. > : > : The pccbb driver returns BUS_PROBE_DEFAULT (it should probably return > : GENERIC > : in the case where it matches only on class codes). Your driver just > : needs to > : return a numerically higher value (but still < 0) to claim the device. > : You > : can probably use BUS_PROBE_VENDOR or BUS_PROBE_DEFAULT + 1. > : > : -- > : John Baldwin > : > : DISCLAIMER: > : > ------------------------------------------------------------------------ > ----------------------------------------------- > : > : The contents of this e-mail and any attachment(s) are confidential and > intended for the named recipient(s) only. > : It shall not attach any liability on the originator or HCL or its > affiliates. Any views or opinions presented in > : this email are solely those of the author and may not necessarily > reflect the opinions of HCL or its affiliates. > : Any form of reproduction, dissemination, copying, disclosure, > modification, distribution and / or publication of > : this message without the prior written consent of the author of this > e-mail is strictly prohibited. If you have > : received this email in error please delete it and notify the sender > immediately. Before opening any mail and > : attachments please check them for viruses and defect. > : > : > ------------------------------------------------------------------------ > ----------------------------------------------- > : > : > > From imp at bsdimp.com Mon Oct 6 07:28:53 2008 From: imp at bsdimp.com (Warner Losh) Date: Mon Oct 6 07:28:59 2008 Subject: i386/127710: My driver PCI probe is not called formycorrespondingdevice ID and Vendor ID In-Reply-To: <68C9F31EF19DB6448F515EF294028FDEE9A624@chn-hclt-evs05.HCLT.CORP.HCL.IN> References: <68C9F31EF19DB6448F515EF 294028FDEE9A4A5@chn-hclt-evs05.HCLT.CORP.HCL.IN> <20081006.005657.71122961.imp@bsdimp.com> <68C9F31EF19DB6448F515EF294028FDEE9A624@chn-hclt-evs05.HCLT.CORP.HCL.IN> Message-ID: <20081006.012619.78741615.imp@bsdimp.com> From: "Bagavathy Kumar Mahendran " Subject: RE: i386/127710: My driver PCI probe is not called formycorrespondingdevice ID and Vendor ID Date: Mon, 6 Oct 2008 12:36:44 +0530 > Dear Warner, > > My probe is getting called for the parent bridge devices > .but > Not for my pci Card. I have tested this by printing the Device ID and > Vendor ID of the corresponding device_t in my probe. > > You are trying to say even cbb probes for my device and return > BUS_PROBE_DEFAULT still my probe function will be called. Just clarify > it. > But my testing seems that my probe is not called for my pci device Can you send me the DRIVER_MODULE line in your driver? Warner > Bagavathy kumar .M > > > -----Original Message----- > From: Warner Losh [mailto:imp@bsdimp.com] > Sent: Monday, October 06, 2008 12:27 PM > To: Bagavathy Kumar Mahendran > Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org > Subject: Re: i386/127710: My driver PCI probe is not called > formycorrespondingdevice ID and Vendor ID > > > Thanks for your support. my probe is getting called for > all > > the bridges not for my pci device. so please provide the fix . > > > > OR > > Is any other way available for making my driver to override the probe > of > > cbb driver for my corresponding device (With out changing cbb driver). > > If your probe returns a higher number that's negative, it will. > Unless cbb is returning 0, your probe routine will get called. Make > sure it isn't. Code inspection suggests that it isn't. > > Warner > > > > > With regards, > > Bagavathy kumar .M > > > > -----Original Message----- > > From: M. Warner Losh [mailto:imp@bsdimp.com] > > Sent: Friday, October 03, 2008 11:03 AM > > To: Bagavathy Kumar Mahendran > > Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org > > Subject: Re: i386/127710: My driver PCI probe is not called for > > mycorrespondingdevice ID and Vendor ID > > > > In message: > > > <68C9F31EF19DB6448F515EF294028FDEE99BCE@chn-hclt-evs05.HCLT.CORP.HCL.IN> > > "Bagavathy Kumar Mahendran " > > writes: > > : > > : Dear Baldwin, > > : Thanks for your support .but my pci probe function is > > not > > : getting called for my device id and vendor id. Because pccbb driver > > : already sets the device_set_desc as PCI-CardBus Bridge. So is there > > any > > : other option for me to make my_pciprobe function to be called for my > > : corresponding device id and vendor id. > > > > That's not why your probe isn't called. Setting a description is > > standard behavior for the probe routine. Are you sure that the device > > probe routine is getting called at all for any device? Have you tried > > just leaving cbb out of the kernel? I recently fixed the original > > problem in cbb (the fact it doesn't check the bridge type too), maybe > > you could try to pick up that fix as well? > > > > Warner > > > > > > : Thanks, > > : > > : Regards, > > : Bagavathy kumar .M > > : > > : > > : > > : -----Original Message----- > > : From: John Baldwin [mailto:jhb@freebsd.org] > > : Sent: Wednesday, October 01, 2008 8:57 PM > > : To: freebsd-hackers@freebsd.org > > : Cc: Bagavathy Kumar Mahendran ; Warner Losh > > : Subject: Re: FW: i386/127710: My driver PCI probe is not called for > my > > : correspondingdevice ID and Vendor ID > > : > > : On Wednesday 01 October 2008 08:50:15 am Bagavathy Kumar Mahendran > > : wrote: > > : > > > : > Dear All, > > : > Iam writing a new driver for a SAS/SATA Controller > > having > > : a > > : > Class ID -0x01 > > : > Sub Class - 0x07 > > : > Programming Interface - 0x00 > > : > > > : > Hence instead of my probe function the Static build Card Bus > Driver > > : cbb > > : > is attaching just by simply checking sub class 0x07 and > programming > > : > interface 0x00.hence my probe gets failed. Kindly help me in > > resolving > > : > this .what I thought is to add the card bus driver a checking of > > CLASS > > : > ID in its pci probe function. > > : > > : The pccbb driver returns BUS_PROBE_DEFAULT (it should probably > return > > : GENERIC > > : in the case where it matches only on class codes). Your driver just > > : needs to > > : return a numerically higher value (but still < 0) to claim the > device. > > : You > > : can probably use BUS_PROBE_VENDOR or BUS_PROBE_DEFAULT + 1. > > : > > : -- > > : John Baldwin > > : > > : DISCLAIMER: > > : > > > ------------------------------------------------------------------------ > > ----------------------------------------------- > > : > > : The contents of this e-mail and any attachment(s) are confidential > and > > intended for the named recipient(s) only. > > : It shall not attach any liability on the originator or HCL or its > > affiliates. Any views or opinions presented in > > : this email are solely those of the author and may not necessarily > > reflect the opinions of HCL or its affiliates. > > : Any form of reproduction, dissemination, copying, disclosure, > > modification, distribution and / or publication of > > : this message without the prior written consent of the author of this > > e-mail is strictly prohibited. If you have > > : received this email in error please delete it and notify the sender > > immediately. Before opening any mail and > > : attachments please check them for viruses and defect. > > : > > : > > > ------------------------------------------------------------------------ > > ----------------------------------------------- > > : > > : > > > > > > From fbsd.hackers at rachie.is-a-geek.net Mon Oct 6 09:43:27 2008 From: fbsd.hackers at rachie.is-a-geek.net (Mel) Date: Mon Oct 6 09:43:34 2008 Subject: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages In-Reply-To: References: <48E94281.8010300@quip.cz> Message-ID: <200810061124.55209.fbsd.hackers@rachie.is-a-geek.net> Hello, On Monday 06 October 2008 07:23:37 Eygene Ryabinkin wrote: > But downloading the INDEX file from the central server seemed to be the > best way, since it almost always gives one the latest port versions, so > I had implemented this in a first place. I've been following this, but I don't agree that (port|pkg_)audit should do this, from the very perspective you're writing this program from: On Sunday 28 September 2008 11:49:18 Eygene Ryabinkin wrote: > 4. I feel that it is Unix-way to do the things: create small utilities > that do their (small) job in a proper fashion. Instead, it can provide installed-pkgnamepkgorigin output. Then, any utility can check whether a new version is available, using what ever source it finds relevant. For example, it is completely irrelevant if a new version is available on the FreeBSD servers, when your machine uses a buildserver in a local network. For those machines it's relevant whether their build server has a new version and one can automatically upgrade if one so desires. Similarly, if your /usr/ports is ahead of the FreeBSD's INDEX.bz2, you're again reporting false information. It's also quite trivial to provide this availibility information in a daily security script, for the "majority of cases" and it's better to have tunables like _use_remote_portindex, _use_portsdir=/bigdisk/usr/ports in a script. -- Mel From rea-fbsd at codelabs.ru Mon Oct 6 10:28:52 2008 From: rea-fbsd at codelabs.ru (Eygene Ryabinkin) Date: Mon Oct 6 10:28:59 2008 Subject: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages In-Reply-To: <200810061124.55209.fbsd.hackers@rachie.is-a-geek.net> References: <48E94281.8010300@quip.cz> <200810061124.55209.fbsd.hackers@rachie.is-a-geek.net> Message-ID: Mel, good day. Mon, Oct 06, 2008 at 11:24:54AM +0200, Mel wrote: > On Monday 06 October 2008 07:23:37 Eygene Ryabinkin wrote: > > But downloading the INDEX file from the central server seemed to be the > > best way, since it almost always gives one the latest port versions, so > > I had implemented this in a first place. > > I've been following this, but I don't agree that (port|pkg_)audit should do > this, from the very perspective you're writing this program from: The download is done not by the portaudit itself, but by the helper script, portaudit-checknew. > On Sunday 28 September 2008 11:49:18 Eygene Ryabinkin wrote: > > 4. I feel that it is Unix-way to do the things: create small utilities > > that do their (small) job in a proper fashion. > > Instead, it can provide installed-pkgnamepkgorigin output. Then, > any utility can check whether a new version is available, using what ever > source it finds relevant. > > For example, it is completely irrelevant if a new version is available on the > FreeBSD servers, when your machine uses a buildserver in a local network. For > those machines it's relevant whether their build server has a new version and > one can automatically upgrade if one so desires. > Similarly, if your /usr/ports is ahead of the FreeBSD's INDEX.bz2, you're > again reporting false information. I hear you, but it seems to me that I should just equip portaudit-checknew with the other sources of a new ports information and provide tunables for their location (on-disk path, URL, etc). I am planning to do this, but first I want to know if these patches will be viable for the project: feeding these into the /dev/null or just using them locally, but equipping with a lot of functionality, is not what I really want ;)) > It's also quite trivial to provide this availibility information in a daily > security script, for the "majority of cases" Didn't get it, sorry. Could you, please, elaborate a bit? > and it's better to have tunables > like _use_remote_portindex, _use_portsdir=/bigdisk/usr/ports in a script. Yes, it was what I had talked about above in this mail. Thanks for the input! -- Eygene _ ___ _.--. # \`.|\..----...-'` `-._.-'_.-'` # Remember that it is hard / ' ` , __.--' # to read the on-line manual )/' _/ \ `-_, / # while single-stepping the kernel. `-'" `"\_ ,_.-;_.-\_ ', fsc/as # _.-'_./ {_.' ; / # -- FreeBSD Developers handbook {_.-``-' {_/ # -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081006/1c364cd2/attachment.pgp From rea-fbsd at codelabs.ru Mon Oct 6 10:30:32 2008 From: rea-fbsd at codelabs.ru (Eygene Ryabinkin) Date: Mon Oct 6 10:30:39 2008 Subject: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages In-Reply-To: <48E9D382.4000001@quip.cz> References: <48DE5CC0.9000708@localhost.inse.ru> <48DF6735.4030906@quip.cz> <4bESZpNwE3z/DdlE2fwK/BXzQSo@2MQ0uKCiT7mdMUuLeUzs8Nv3ToQ> <48E94281.8010300@quip.cz> <48E9D382.4000001@quip.cz> Message-ID: Miroslav, Mon, Oct 06, 2008 at 10:59:46AM +0200, Miroslav Lachman wrote: > I have '/usr/sbin/portsnap cron' and '/usr/sbin/portsnap -I update' in > my crontab, so I get INDEX updated every night before nightly security > e-mail is generated. Ah, I see. Thanks! > > But downloading the INDEX file from the central server seemed to be the > > best way, since it almost always gives one the latest port versions, so > > I had implemented this in a first place. > > My previous question was not against your solution, it seems useful to > have really actual data from the fresh INDEX. It was just a question > "how it is done". Maybe someone will be happier to use the existing > INDEX because of traffic on some GPRS internet connection or because of > the own INDEX creation. (it is not my case, I have all machines as the > servers with enough connectivity) ;) OK, fine. I will implement the usage of the local INDEX file in some days. -- Eygene _ ___ _.--. # \`.|\..----...-'` `-._.-'_.-'` # Remember that it is hard / ' ` , __.--' # to read the on-line manual )/' _/ \ `-_, / # while single-stepping the kernel. `-'" `"\_ ,_.-;_.-\_ ', fsc/as # _.-'_./ {_.' ; / # -- FreeBSD Developers handbook {_.-``-' {_/ # -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081006/a0751395/attachment.pgp From fbsd.hackers at rachie.is-a-geek.net Mon Oct 6 11:07:55 2008 From: fbsd.hackers at rachie.is-a-geek.net (Mel) Date: Mon Oct 6 11:09:43 2008 Subject: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages In-Reply-To: References: <200810061124.55209.fbsd.hackers@rachie.is-a-geek.net> Message-ID: <200810061307.51977.fbsd.hackers@rachie.is-a-geek.net> On Monday 06 October 2008 12:28:48 Eygene Ryabinkin wrote: > Mel, good day. > > Mon, Oct 06, 2008 at 11:24:54AM +0200, Mel wrote: > > On Monday 06 October 2008 07:23:37 Eygene Ryabinkin wrote: > > > But downloading the INDEX file from the central server seemed to be the > > > best way, since it almost always gives one the latest port versions, so > > > I had implemented this in a first place. > > > > I've been following this, but I don't agree that (port|pkg_)audit should > > do this, from the very perspective you're writing this program from: > > The download is done not by the portaudit itself, but by the helper > script, portaudit-checknew. > > > On Sunday 28 September 2008 11:49:18 Eygene Ryabinkin wrote: > > > 4. I feel that it is Unix-way to do the things: create small utilities > > > that do their (small) job in a proper fashion. > > > > Instead, it can provide installed-pkgnamepkgorigin output. > > Then, any utility can check whether a new version is available, using > > what ever source it finds relevant. > > > > For example, it is completely irrelevant if a new version is available on > > the FreeBSD servers, when your machine uses a buildserver in a local > > network. For those machines it's relevant whether their build server has > > a new version and one can automatically upgrade if one so desires. > > Similarly, if your /usr/ports is ahead of the FreeBSD's INDEX.bz2, you're > > again reporting false information. > > I hear you, but it seems to me that I should just equip > portaudit-checknew with the other sources of a new ports information and > provide tunables for their location (on-disk path, URL, etc). I am > planning to do this, but first I want to know if these patches will be > viable for the project: feeding these into the /dev/null or just using > them locally, but equipping with a lot of functionality, is not what I > really want ;)) > > > It's also quite trivial to provide this availibility information in a > > daily security script, for the "majority of cases" > > Didn't get it, sorry. Could you, please, elaborate a bit? Once you have the origin of the port, you can: - make -C $PORTSDIR/$origin -V PKGNAME - get the matching origin(s) out of ${INDEXDIR}/${INDEXFILE} - get the matching origin(s) out of a downloaded INDEX.bz2 This covers the majority of cases. What portaudit lacks, is providing the origin along with the installed package name in easily parseable format. So, a central server wanting to query all the machines for vulnerable packages, now has to do an extra step of going into $PKG_DBDIR/$pkgname/+CONTENTS and getting the @comment ORIGIN: line, while (port|pkg_)audit has just been there. This would be something I'd expect: ssh clientmachine "/usr/sbin/pkg_audit -l" foo-1.2,3:misc/foo bar-4.5_6:devel/bar ... -- Mel From bagavathykumar.m at hcl.in Mon Oct 6 05:18:36 2008 From: bagavathykumar.m at hcl.in (Bagavathy Kumar Mahendran ) Date: Mon Oct 6 11:17:15 2008 Subject: i386/127710: My driver PCI probe is not called for mycorrespondingdevice ID and Vendor ID In-Reply-To: <20081002.233328.-432820840.imp@bsdimp.com> References: <68C9F31EF19DB6448F515EF294028FDEE999AB@chn-hclt-evs05.HCLT.CORP .HCL.IN><200810011127.14593.jhb@freebsd.org><68C9F31EF19DB6448F515EF294028F DEE99BCE@chn-hclt-evs05.HCLT.CORP.HCL.IN> <20081002.233328.-432820840.imp@bsdimp.com> Message-ID: <68C9F31EF19DB6448F515EF294028FDEE9A4A5@chn-hclt-evs05.HCLT.CORP.HCL.IN> Dear Warner, Thanks for your support. my probe is getting called for all the bridges not for my pci device. so please provide the fix . OR Is any other way available for making my driver to override the probe of cbb driver for my corresponding device (With out changing cbb driver). With regards, Bagavathy kumar .M -----Original Message----- From: M. Warner Losh [mailto:imp@bsdimp.com] Sent: Friday, October 03, 2008 11:03 AM To: Bagavathy Kumar Mahendran Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org Subject: Re: i386/127710: My driver PCI probe is not called for mycorrespondingdevice ID and Vendor ID In message: <68C9F31EF19DB6448F515EF294028FDEE99BCE@chn-hclt-evs05.HCLT.CORP.HCL.IN> "Bagavathy Kumar Mahendran " writes: : : Dear Baldwin, : Thanks for your support .but my pci probe function is not : getting called for my device id and vendor id. Because pccbb driver : already sets the device_set_desc as PCI-CardBus Bridge. So is there any : other option for me to make my_pciprobe function to be called for my : corresponding device id and vendor id. That's not why your probe isn't called. Setting a description is standard behavior for the probe routine. Are you sure that the device probe routine is getting called at all for any device? Have you tried just leaving cbb out of the kernel? I recently fixed the original problem in cbb (the fact it doesn't check the bridge type too), maybe you could try to pick up that fix as well? Warner : Thanks, : : Regards, : Bagavathy kumar .M : : : : -----Original Message----- : From: John Baldwin [mailto:jhb@freebsd.org] : Sent: Wednesday, October 01, 2008 8:57 PM : To: freebsd-hackers@freebsd.org : Cc: Bagavathy Kumar Mahendran ; Warner Losh : Subject: Re: FW: i386/127710: My driver PCI probe is not called for my : correspondingdevice ID and Vendor ID : : On Wednesday 01 October 2008 08:50:15 am Bagavathy Kumar Mahendran : wrote: : > : > Dear All, : > Iam writing a new driver for a SAS/SATA Controller having : a : > Class ID -0x01 : > Sub Class - 0x07 : > Programming Interface - 0x00 : > : > Hence instead of my probe function the Static build Card Bus Driver : cbb : > is attaching just by simply checking sub class 0x07 and programming : > interface 0x00.hence my probe gets failed. Kindly help me in resolving : > this .what I thought is to add the card bus driver a checking of CLASS : > ID in its pci probe function. : : The pccbb driver returns BUS_PROBE_DEFAULT (it should probably return : GENERIC : in the case where it matches only on class codes). Your driver just : needs to : return a numerically higher value (but still < 0) to claim the device. : You : can probably use BUS_PROBE_VENDOR or BUS_PROBE_DEFAULT + 1. : : -- : John Baldwin : : DISCLAIMER: : ------------------------------------------------------------------------ ----------------------------------------------- : : The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. : It shall not attach any liability on the originator or HCL or its affiliates. Any views or opinions presented in : this email are solely those of the author and may not necessarily reflect the opinions of HCL or its affiliates. : Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of : this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have : received this email in error please delete it and notify the sender immediately. Before opening any mail and : attachments please check them for viruses and defect. : : ------------------------------------------------------------------------ ----------------------------------------------- : : From bagavathykumar.m at hcl.in Mon Oct 6 07:06:43 2008 From: bagavathykumar.m at hcl.in (Bagavathy Kumar Mahendran ) Date: Mon Oct 6 11:23:32 2008 Subject: i386/127710: My driver PCI probe is not called formycorrespondingdevice ID and Vendor ID In-Reply-To: <20081006.005657.71122961.imp@bsdimp.com> References: <68C9F31EF19DB6448F515EF294028FDEE99BCE@chn-hclt-evs05.HCLT.CORP .HCL.IN><20081002.233328.-432820840.imp@bsdimp.com><68C9F31EF19DB6448F515EF 294028FDEE9A4A5@chn-hclt-evs05.HCLT.CORP.HCL.IN> <20081006.005657.71122961.imp@bsdimp.com> Message-ID: <68C9F31EF19DB6448F515EF294028FDEE9A624@chn-hclt-evs05.HCLT.CORP.HCL.IN> Dear Warner, My probe is getting called for the parent bridge devices .but Not for my pci Card. I have tested this by printing the Device ID and Vendor ID of the corresponding device_t in my probe. You are trying to say even cbb probes for my device and return BUS_PROBE_DEFAULT still my probe function will be called. Just clarify it. But my testing seems that my probe is not called for my pci device Bagavathy kumar .M -----Original Message----- From: Warner Losh [mailto:imp@bsdimp.com] Sent: Monday, October 06, 2008 12:27 PM To: Bagavathy Kumar Mahendran Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org Subject: Re: i386/127710: My driver PCI probe is not called formycorrespondingdevice ID and Vendor ID > Thanks for your support. my probe is getting called for all > the bridges not for my pci device. so please provide the fix . > > OR > Is any other way available for making my driver to override the probe of > cbb driver for my corresponding device (With out changing cbb driver). If your probe returns a higher number that's negative, it will. Unless cbb is returning 0, your probe routine will get called. Make sure it isn't. Code inspection suggests that it isn't. Warner > With regards, > Bagavathy kumar .M > > -----Original Message----- > From: M. Warner Losh [mailto:imp@bsdimp.com] > Sent: Friday, October 03, 2008 11:03 AM > To: Bagavathy Kumar Mahendran > Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org > Subject: Re: i386/127710: My driver PCI probe is not called for > mycorrespondingdevice ID and Vendor ID > > In message: > <68C9F31EF19DB6448F515EF294028FDEE99BCE@chn-hclt-evs05.HCLT.CORP.HCL.IN> > "Bagavathy Kumar Mahendran " > writes: > : > : Dear Baldwin, > : Thanks for your support .but my pci probe function is > not > : getting called for my device id and vendor id. Because pccbb driver > : already sets the device_set_desc as PCI-CardBus Bridge. So is there > any > : other option for me to make my_pciprobe function to be called for my > : corresponding device id and vendor id. > > That's not why your probe isn't called. Setting a description is > standard behavior for the probe routine. Are you sure that the device > probe routine is getting called at all for any device? Have you tried > just leaving cbb out of the kernel? I recently fixed the original > problem in cbb (the fact it doesn't check the bridge type too), maybe > you could try to pick up that fix as well? > > Warner > > > : Thanks, > : > : Regards, > : Bagavathy kumar .M > : > : > : > : -----Original Message----- > : From: John Baldwin [mailto:jhb@freebsd.org] > : Sent: Wednesday, October 01, 2008 8:57 PM > : To: freebsd-hackers@freebsd.org > : Cc: Bagavathy Kumar Mahendran ; Warner Losh > : Subject: Re: FW: i386/127710: My driver PCI probe is not called for my > : correspondingdevice ID and Vendor ID > : > : On Wednesday 01 October 2008 08:50:15 am Bagavathy Kumar Mahendran > : wrote: > : > > : > Dear All, > : > Iam writing a new driver for a SAS/SATA Controller > having > : a > : > Class ID -0x01 > : > Sub Class - 0x07 > : > Programming Interface - 0x00 > : > > : > Hence instead of my probe function the Static build Card Bus Driver > : cbb > : > is attaching just by simply checking sub class 0x07 and programming > : > interface 0x00.hence my probe gets failed. Kindly help me in > resolving > : > this .what I thought is to add the card bus driver a checking of > CLASS > : > ID in its pci probe function. > : > : The pccbb driver returns BUS_PROBE_DEFAULT (it should probably return > : GENERIC > : in the case where it matches only on class codes). Your driver just > : needs to > : return a numerically higher value (but still < 0) to claim the device. > : You > : can probably use BUS_PROBE_VENDOR or BUS_PROBE_DEFAULT + 1. > : > : -- > : John Baldwin > : > : DISCLAIMER: > : > ------------------------------------------------------------------------ > ----------------------------------------------- > : > : The contents of this e-mail and any attachment(s) are confidential and > intended for the named recipient(s) only. > : It shall not attach any liability on the originator or HCL or its > affiliates. Any views or opinions presented in > : this email are solely those of the author and may not necessarily > reflect the opinions of HCL or its affiliates. > : Any form of reproduction, dissemination, copying, disclosure, > modification, distribution and / or publication of > : this message without the prior written consent of the author of this > e-mail is strictly prohibited. If you have > : received this email in error please delete it and notify the sender > immediately. Before opening any mail and > : attachments please check them for viruses and defect. > : > : > ------------------------------------------------------------------------ > ----------------------------------------------- > : > : > > From bagavathykumar.m at hcl.in Mon Oct 6 07:37:11 2008 From: bagavathykumar.m at hcl.in (Bagavathy Kumar Mahendran ) Date: Mon Oct 6 11:23:52 2008 Subject: i386/127710: My driver PCI probe is not calledformycorrespondingdevice ID and Vendor ID In-Reply-To: <20081006.012619.78741615.imp@bsdimp.com> References: <68C9F31EF19DB6448F515EF294028FDEE9A4A5@chn-hclt-evs05.HCLT.CORP .HCL.IN><20081006.005657.71122961.imp@bsdimp.com><68C9F31EF19DB6448F515EF29 4028FDEE9A624@chn-hclt-evs05.HCLT.CORP.HCL.IN> <20081006.012619.78741615.imp@bsdimp.com> Message-ID: <68C9F31EF19DB6448F515EF294028FDEE9A695@chn-hclt-evs05.HCLT.CORP.HCL.IN> static int My_probe(device_t dev) { uint32_t progif; uint32_t subclass; uint32_t class; device_printf(dev, " Probe\n Vendor ID : 0x%x\n Device ID : 0x%x\n",pci_get_vendor(dev), pci_get_device(dev)); class = pci_get_class(dev); subclass = pci_get_subclass(dev); progif = pci_get_progif(dev); if (class == 0x1 && subclass == 0x07 && progif == 0x00) { printf("probe successful!\n"); device_set_desc(dev, "My_Probe"); return (BUS_PROBE_DEFAULT); } return (ENXIO); } Here above I have attached my pci probe. The vendor ID and Device ID of all the above PCI bridges are printing But instead of My_probe function for my PCI Card the probe of the cbb driver is called. This is the exact problem but when I remove the cbb driver from the kernel My_probe function is called for my PCI Card too. Regards, Bagavathy kumar .M -----Original Message----- From: Warner Losh [mailto:imp@bsdimp.com] Sent: Monday, October 06, 2008 12:56 PM To: Bagavathy Kumar Mahendran Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org Subject: Re: i386/127710: My driver PCI probe is not calledformycorrespondingdevice ID and Vendor ID From: "Bagavathy Kumar Mahendran " Subject: RE: i386/127710: My driver PCI probe is not called formycorrespondingdevice ID and Vendor ID Date: Mon, 6 Oct 2008 12:36:44 +0530 > Dear Warner, > > My probe is getting called for the parent bridge devices > .but > Not for my pci Card. I have tested this by printing the Device ID and > Vendor ID of the corresponding device_t in my probe. > > You are trying to say even cbb probes for my device and return > BUS_PROBE_DEFAULT still my probe function will be called. Just clarify > it. > But my testing seems that my probe is not called for my pci device Can you send me the DRIVER_MODULE line in your driver? Warner > Bagavathy kumar .M > > > -----Original Message----- > From: Warner Losh [mailto:imp@bsdimp.com] > Sent: Monday, October 06, 2008 12:27 PM > To: Bagavathy Kumar Mahendran > Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org > Subject: Re: i386/127710: My driver PCI probe is not called > formycorrespondingdevice ID and Vendor ID > > > Thanks for your support. my probe is getting called for > all > > the bridges not for my pci device. so please provide the fix . > > > > OR > > Is any other way available for making my driver to override the probe > of > > cbb driver for my corresponding device (With out changing cbb driver). > > If your probe returns a higher number that's negative, it will. > Unless cbb is returning 0, your probe routine will get called. Make > sure it isn't. Code inspection suggests that it isn't. > > Warner > > > > > With regards, > > Bagavathy kumar .M > > > > -----Original Message----- > > From: M. Warner Losh [mailto:imp@bsdimp.com] > > Sent: Friday, October 03, 2008 11:03 AM > > To: Bagavathy Kumar Mahendran > > Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org > > Subject: Re: i386/127710: My driver PCI probe is not called for > > mycorrespondingdevice ID and Vendor ID > > > > In message: > > > <68C9F31EF19DB6448F515EF294028FDEE99BCE@chn-hclt-evs05.HCLT.CORP.HCL.IN> > > "Bagavathy Kumar Mahendran " > > writes: > > : > > : Dear Baldwin, > > : Thanks for your support .but my pci probe function is > > not > > : getting called for my device id and vendor id. Because pccbb driver > > : already sets the device_set_desc as PCI-CardBus Bridge. So is there > > any > > : other option for me to make my_pciprobe function to be called for my > > : corresponding device id and vendor id. > > > > That's not why your probe isn't called. Setting a description is > > standard behavior for the probe routine. Are you sure that the device > > probe routine is getting called at all for any device? Have you tried > > just leaving cbb out of the kernel? I recently fixed the original > > problem in cbb (the fact it doesn't check the bridge type too), maybe > > you could try to pick up that fix as well? > > > > Warner > > > > > > : Thanks, > > : > > : Regards, > > : Bagavathy kumar .M > > : > > : > > : > > : -----Original Message----- > > : From: John Baldwin [mailto:jhb@freebsd.org] > > : Sent: Wednesday, October 01, 2008 8:57 PM > > : To: freebsd-hackers@freebsd.org > > : Cc: Bagavathy Kumar Mahendran ; Warner Losh > > : Subject: Re: FW: i386/127710: My driver PCI probe is not called for > my > > : correspondingdevice ID and Vendor ID > > : > > : On Wednesday 01 October 2008 08:50:15 am Bagavathy Kumar Mahendran > > : wrote: > > : > > > : > Dear All, > > : > Iam writing a new driver for a SAS/SATA Controller > > having > > : a > > : > Class ID -0x01 > > : > Sub Class - 0x07 > > : > Programming Interface - 0x00 > > : > > > : > Hence instead of my probe function the Static build Card Bus > Driver > > : cbb > > : > is attaching just by simply checking sub class 0x07 and > programming > > : > interface 0x00.hence my probe gets failed. Kindly help me in > > resolving > > : > this .what I thought is to add the card bus driver a checking of > > CLASS > > : > ID in its pci probe function. > > : > > : The pccbb driver returns BUS_PROBE_DEFAULT (it should probably > return > > : GENERIC > > : in the case where it matches only on class codes). Your driver just > > : needs to > > : return a numerically higher value (but still < 0) to claim the > device. > > : You > > : can probably use BUS_PROBE_VENDOR or BUS_PROBE_DEFAULT + 1. > > : > > : -- > > : John Baldwin > > : > > : DISCLAIMER: > > : > > > ------------------------------------------------------------------------ > > ----------------------------------------------- > > : > > : The contents of this e-mail and any attachment(s) are confidential > and > > intended for the named recipient(s) only. > > : It shall not attach any liability on the originator or HCL or its > > affiliates. Any views or opinions presented in > > : this email are solely those of the author and may not necessarily > > reflect the opinions of HCL or its affiliates. > > : Any form of reproduction, dissemination, copying, disclosure, > > modification, distribution and / or publication of > > : this message without the prior written consent of the author of this > > e-mail is strictly prohibited. If you have > > : received this email in error please delete it and notify the sender > > immediately. Before opening any mail and > > : attachments please check them for viruses and defect. > > : > > : > > > ------------------------------------------------------------------------ > > ----------------------------------------------- > > : > > : > > > > > > From bagavathykumar.m at hcl.in Mon Oct 6 07:40:58 2008 From: bagavathykumar.m at hcl.in (Bagavathy Kumar Mahendran ) Date: Mon Oct 6 11:24:09 2008 Subject: i386/127710: My driver PCI probe is not calledformycorrespondingdevice ID and Vendor ID In-Reply-To: <20081006.012619.78741615.imp@bsdimp.com> References: <68C9F31EF19DB6448F515EF294028FDEE9A4A5@chn-hclt-evs05.HCLT.CORP .HCL.IN><20081006.005657.71122961.imp@bsdimp.com><68C9F31EF19DB6448F515EF29 4028FDEE9A624@chn-hclt-evs05.HCLT.CORP.HCL.IN> <20081006.012619.78741615.imp@bsdimp.com> Message-ID: <68C9F31EF19DB6448F515EF294028FDEE9A6A6@chn-hclt-evs05.HCLT.CORP.HCL.IN> static devclass_t My_devclass; DRIVER_MODULE(My, pci, My_driver, My_devclass, 0, 0); With Regards, Bagavathy kumar .M -----Original Message----- From: Warner Losh [mailto:imp@bsdimp.com] Sent: Monday, October 06, 2008 12:56 PM To: Bagavathy Kumar Mahendran Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org Subject: Re: i386/127710: My driver PCI probe is not calledformycorrespondingdevice ID and Vendor ID From: "Bagavathy Kumar Mahendran " Subject: RE: i386/127710: My driver PCI probe is not called formycorrespondingdevice ID and Vendor ID Date: Mon, 6 Oct 2008 12:36:44 +0530 > Dear Warner, > > My probe is getting called for the parent bridge devices > .but > Not for my pci Card. I have tested this by printing the Device ID and > Vendor ID of the corresponding device_t in my probe. > > You are trying to say even cbb probes for my device and return > BUS_PROBE_DEFAULT still my probe function will be called. Just clarify > it. > But my testing seems that my probe is not called for my pci device Can you send me the DRIVER_MODULE line in your driver? Warner > Bagavathy kumar .M > > > -----Original Message----- > From: Warner Losh [mailto:imp@bsdimp.com] > Sent: Monday, October 06, 2008 12:27 PM > To: Bagavathy Kumar Mahendran > Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org > Subject: Re: i386/127710: My driver PCI probe is not called > formycorrespondingdevice ID and Vendor ID > > > Thanks for your support. my probe is getting called for > all > > the bridges not for my pci device. so please provide the fix . > > > > OR > > Is any other way available for making my driver to override the probe > of > > cbb driver for my corresponding device (With out changing cbb driver). > > If your probe returns a higher number that's negative, it will. > Unless cbb is returning 0, your probe routine will get called. Make > sure it isn't. Code inspection suggests that it isn't. > > Warner > > > > > With regards, > > Bagavathy kumar .M > > > > -----Original Message----- > > From: M. Warner Losh [mailto:imp@bsdimp.com] > > Sent: Friday, October 03, 2008 11:03 AM > > To: Bagavathy Kumar Mahendran > > Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org > > Subject: Re: i386/127710: My driver PCI probe is not called for > > mycorrespondingdevice ID and Vendor ID > > > > In message: > > > <68C9F31EF19DB6448F515EF294028FDEE99BCE@chn-hclt-evs05.HCLT.CORP.HCL.IN> > > "Bagavathy Kumar Mahendran " > > writes: > > : > > : Dear Baldwin, > > : Thanks for your support .but my pci probe function is > > not > > : getting called for my device id and vendor id. Because pccbb driver > > : already sets the device_set_desc as PCI-CardBus Bridge. So is there > > any > > : other option for me to make my_pciprobe function to be called for my > > : corresponding device id and vendor id. > > > > That's not why your probe isn't called. Setting a description is > > standard behavior for the probe routine. Are you sure that the device > > probe routine is getting called at all for any device? Have you tried > > just leaving cbb out of the kernel? I recently fixed the original > > problem in cbb (the fact it doesn't check the bridge type too), maybe > > you could try to pick up that fix as well? > > > > Warner > > > > > > : Thanks, > > : > > : Regards, > > : Bagavathy kumar .M > > : > > : > > : > > : -----Original Message----- > > : From: John Baldwin [mailto:jhb@freebsd.org] > > : Sent: Wednesday, October 01, 2008 8:57 PM > > : To: freebsd-hackers@freebsd.org > > : Cc: Bagavathy Kumar Mahendran ; Warner Losh > > : Subject: Re: FW: i386/127710: My driver PCI probe is not called for > my > > : correspondingdevice ID and Vendor ID > > : > > : On Wednesday 01 October 2008 08:50:15 am Bagavathy Kumar Mahendran > > : wrote: > > : > > > : > Dear All, > > : > Iam writing a new driver for a SAS/SATA Controller > > having > > : a > > : > Class ID -0x01 > > : > Sub Class - 0x07 > > : > Programming Interface - 0x00 > > : > > > : > Hence instead of my probe function the Static build Card Bus > Driver > > : cbb > > : > is attaching just by simply checking sub class 0x07 and > programming > > : > interface 0x00.hence my probe gets failed. Kindly help me in > > resolving > > : > this .what I thought is to add the card bus driver a checking of > > CLASS > > : > ID in its pci probe function. > > : > > : The pccbb driver returns BUS_PROBE_DEFAULT (it should probably > return > > : GENERIC > > : in the case where it matches only on class codes). Your driver just > > : needs to > > : return a numerically higher value (but still < 0) to claim the > device. > > : You > > : can probably use BUS_PROBE_VENDOR or BUS_PROBE_DEFAULT + 1. > > : > > : -- > > : John Baldwin > > : > > : DISCLAIMER: > > : > > > ------------------------------------------------------------------------ > > ----------------------------------------------- > > : > > : The contents of this e-mail and any attachment(s) are confidential > and > > intended for the named recipient(s) only. > > : It shall not attach any liability on the originator or HCL or its > > affiliates. Any views or opinions presented in > > : this email are solely those of the author and may not necessarily > > reflect the opinions of HCL or its affiliates. > > : Any form of reproduction, dissemination, copying, disclosure, > > modification, distribution and / or publication of > > : this message without the prior written consent of the author of this > > e-mail is strictly prohibited. If you have > > : received this email in error please delete it and notify the sender > > immediately. Before opening any mail and > > : attachments please check them for viruses and defect. > > : > > : > > > ------------------------------------------------------------------------ > > ----------------------------------------------- > > : > > : > > > > > > From 000.fbsd at quip.cz Mon Oct 6 08:59:17 2008 From: 000.fbsd at quip.cz (Miroslav Lachman) Date: Mon Oct 6 11:24:17 2008 Subject: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages In-Reply-To: References: <48DE5CC0.9000708@localhost.inse.ru> <48DF6735.4030906@quip.cz> <4bESZpNwE3z/DdlE2fwK/BXzQSo@2MQ0uKCiT7mdMUuLeUzs8Nv3ToQ> <48E94281.8010300@quip.cz> Message-ID: <48E9D382.4000001@quip.cz> Eygene Ryabinkin wrote: > Miroslav, good day. > > Mon, Oct 06, 2008 at 12:41:05AM +0200, Miroslav Lachman wrote: > >>I am busy these days, but it is nice to read about your progress. I hope >>I will get some time to test all of these large patches in a few days >>and I will report back my experiences! > > > Fine, thank you! I am re-CC'ing bug-followup@ to track this letter, > since it contains some useful information that should go into GNATS. > > >>One note before tests... do -n flag always download new INDEX file, or >>is it possible to use one already existing in /usr/ports? > > > Currently, it is downloads bzipped INDEX file to /var/db/portaudit every > time, but it uses mirror mode, so if remote file hadn't changed at all, > all network expences are just the HTTP's HEAD request and reply. > > I can add another variable to the portaudit to force the usage of the > existing INDEX file, if it is needed. By the way, how are you keeping > your INDEX file up to date (your proposed usage of 'pkg_version -I' > implies that you're always rely on it)? I am just curious -- my INDEX > files are almost always stay unupdated, even if I am using portupgrade. I have '/usr/sbin/portsnap cron' and '/usr/sbin/portsnap -I update' in my crontab, so I get INDEX updated every night before nightly security e-mail is generated. > And there can be another way if one keeps ports tree updated: utility > can use 'make' to determine the version that is currently available on > the examined host. > > But downloading the INDEX file from the central server seemed to be the > best way, since it almost always gives one the latest port versions, so > I had implemented this in a first place. My previous question was not against your solution, it seems useful to have really actual data from the fresh INDEX. It was just a question "how it is done". Maybe someone will be happier to use the existing INDEX because of traffic on some GPRS internet connection or because of the own INDEX creation. (it is not my case, I have all machines as the servers with enough connectivity) ;) > Don't know, however, how the badly the load to the central HTTP server > will be raised. I am using just two first fields from the INDEX file, > so I can use such a stripped file. For me, the reduction was about > 6x: SIZE(INDEX-7.bz2) = 1126189, SIZE(INDEX-7.stripped.bz2) = 184345. > > I am CC'ing the portmgr team. Guys, could you quickly glance over these > patches and determine if they are useful to the project in large? If > yes, then may be such a stripped INDEX can be created on the FreeBSD > servers (via cut -f1-2 -d'|' INDEX-N)? > > Thanks! From yurtesen at ispro.net Mon Oct 6 10:25:15 2008 From: yurtesen at ispro.net (Evren Yurtesen) Date: Mon Oct 6 11:24:31 2008 Subject: continuous backup solution for FreeBSD Message-ID: <48E9E1BB.6020908@ispro.net> Hello, As far as I can see, there is no continuous backup solution for FreeBSD at the moment. I talked with R1Soft and they seem to not be able to support FreeBSD and need help. Does anybody have free time and skills to give a hand? Please see: http://forum.r1soft.com/showpost.php?p=3414&postcount=9 I know people who want to switch to using FreeBSD but cant do that because their backup solution (R1Soft) does not support FreeBSD :( Thanks, Evren From rea-fbsd at codelabs.ru Mon Oct 6 12:22:19 2008 From: rea-fbsd at codelabs.ru (Eygene Ryabinkin) Date: Mon Oct 6 12:22:25 2008 Subject: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages In-Reply-To: <200810061307.51977.fbsd.hackers@rachie.is-a-geek.net> References: <200810061124.55209.fbsd.hackers@rachie.is-a-geek.net> <200810061307.51977.fbsd.hackers@rachie.is-a-geek.net> Message-ID: Mel, Mon, Oct 06, 2008 at 01:07:51PM +0200, Mel wrote: > On Monday 06 October 2008 12:28:48 Eygene Ryabinkin wrote: > Once you have the origin of the port, you can: > - make -C $PORTSDIR/$origin -V PKGNAME > - get the matching origin(s) out of ${INDEXDIR}/${INDEXFILE} > - get the matching origin(s) out of a downloaded INDEX.bz2 > > This covers the majority of cases. > > What portaudit lacks, is providing the origin along with the installed package > name in easily parseable format. So, a central server wanting to query all > the machines for vulnerable packages, now has to do an extra step of going > into $PKG_DBDIR/$pkgname/+CONTENTS and getting the @comment ORIGIN: line, > while (port|pkg_)audit has just been there. > > This would be something I'd expect: > ssh clientmachine "/usr/sbin/pkg_audit -l" > foo-1.2,3:misc/foo > bar-4.5_6:devel/bar > ... OK, got it. There is one neat: pkg_audit should be feeded with the contents of the auditfile and the latter is located in the tar archive. So, if you wouldn't mind about the following sequence ----- tar xf /var/db/portaudit/auditfile.tbz pkg_audit < auditfile | portaudit-checknew -o | cut -d '|' -f1,4,5 ----- then I can add the flag '-o' to the portaudit-checknew: it will additionally output the port origin along with the new version. Is that what you meant? -- Eygene _ ___ _.--. # \`.|\..----...-'` `-._.-'_.-'` # Remember that it is hard / ' ` , __.--' # to read the on-line manual )/' _/ \ `-_, / # while single-stepping the kernel. `-'" `"\_ ,_.-;_.-\_ ', fsc/as # _.-'_./ {_.' ; / # -- FreeBSD Developers handbook {_.-``-' {_/ # -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081006/0bdf4547/attachment.pgp From rb at gid.co.uk Mon Oct 6 12:29:37 2008 From: rb at gid.co.uk (Bob Bishop) Date: Mon Oct 6 12:29:44 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <48E9E1BB.6020908@ispro.net> References: <48E9E1BB.6020908@ispro.net> Message-ID: <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> Hi, On 6 Oct 2008, at 11:00, Evren Yurtesen wrote: > As far as I can see, there is no continuous backup solution for > FreeBSD at the moment. I talked with R1Soft and they seem to not be > able to support FreeBSD and need help. > > Does anybody have free time and skills to give a hand? Please see: > http://forum.r1soft.com/showpost.php?p=3414&postcount=9 Should be possible to do this with a geom(4) class? > I know people who want to switch to using FreeBSD but cant do that > because their backup solution (R1Soft) does not support FreeBSD :( > > Thanks, > Evren > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org > " > -- Bob Bishop rb@gid.co.uk From fbsd.hackers at rachie.is-a-geek.net Mon Oct 6 12:40:51 2008 From: fbsd.hackers at rachie.is-a-geek.net (Mel) Date: Mon Oct 6 12:41:03 2008 Subject: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages In-Reply-To: References: <200810061307.51977.fbsd.hackers@rachie.is-a-geek.net> Message-ID: <200810061440.49113.fbsd.hackers@rachie.is-a-geek.net> On Monday 06 October 2008 14:22:13 Eygene Ryabinkin wrote: > Mel, > > Mon, Oct 06, 2008 at 01:07:51PM +0200, Mel wrote: > > On Monday 06 October 2008 12:28:48 Eygene Ryabinkin wrote: > > Once you have the origin of the port, you can: > > - make -C $PORTSDIR/$origin -V PKGNAME > > - get the matching origin(s) out of ${INDEXDIR}/${INDEXFILE} > > - get the matching origin(s) out of a downloaded INDEX.bz2 > > > > This covers the majority of cases. > > > > What portaudit lacks, is providing the origin along with the installed > > package name in easily parseable format. So, a central server wanting to > > query all the machines for vulnerable packages, now has to do an extra > > step of going into $PKG_DBDIR/$pkgname/+CONTENTS and getting the @comment > > ORIGIN: line, while (port|pkg_)audit has just been there. > > > > This would be something I'd expect: > > ssh clientmachine "/usr/sbin/pkg_audit -l" > > foo-1.2,3:misc/foo > > bar-4.5_6:devel/bar > > ... > > OK, got it. There is one neat: pkg_audit should be feeded with the > contents of the auditfile and the latter is located in the tar archive. > So, if you wouldn't mind about the following sequence > ----- > tar xf /var/db/portaudit/auditfile.tbz > pkg_audit < auditfile | portaudit-checknew -o | cut -d '|' -f1,4,5 > ----- > then I can add the flag '-o' to the portaudit-checknew: it will > additionally output the port origin along with the new version. > > Is that what you meant? What I meant is the '-o' flag in pkg_audit, so I can figure out myself whether it's new or not and my buildserver can prioritize it's builds based on vulnerable packages it's clients have installed. The origin is the unique key that identifies any port, so that's vital information in a pipeline. -- Mel From rea-fbsd at codelabs.ru Mon Oct 6 13:08:53 2008 From: rea-fbsd at codelabs.ru (Eygene Ryabinkin) Date: Mon Oct 6 13:09:05 2008 Subject: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages In-Reply-To: <200810061440.49113.fbsd.hackers@rachie.is-a-geek.net> References: <200810061307.51977.fbsd.hackers@rachie.is-a-geek.net> <200810061440.49113.fbsd.hackers@rachie.is-a-geek.net> Message-ID: Mel, Mon, Oct 06, 2008 at 02:40:48PM +0200, Mel wrote: > What I meant is the '-o' flag in pkg_audit, so I can figure out myself whether > it's new or not and my buildserver can prioritize it's builds based on > vulnerable packages it's clients have installed. The origin is the unique key > that identifies any port, so that's vital information in a pipeline. Ah, OK: no problems, will do it. -- Eygene _ ___ _.--. # \`.|\..----...-'` `-._.-'_.-'` # Remember that it is hard / ' ` , __.--' # to read the on-line manual )/' _/ \ `-_, / # while single-stepping the kernel. `-'" `"\_ ,_.-;_.-\_ ', fsc/as # _.-'_./ {_.' ; / # -- FreeBSD Developers handbook {_.-``-' {_/ # -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081006/4a233c67/attachment.pgp From yurtesen at ispro.net Mon Oct 6 14:33:15 2008 From: yurtesen at ispro.net (Evren Yurtesen) Date: Mon Oct 6 15:05:07 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> Message-ID: <48EA21AE.80607@ispro.net> Bob Bishop wrote: >> Does anybody have free time and skills to give a hand? Please see: >> http://forum.r1soft.com/showpost.php?p=3414&postcount=9 > > Should be possible to do this with a geom(4) class? > I am not saying it is impossible. They just need somebody to put them to right track I guess. I personally cant do that. It would be nice if somebody who has knowledge in this area contacts r1soft. At the very least r1soft seems to be willing to communicate on this issue. Continuous backups as well as bare-metal-restore seem to be a key feature for many hosters. FreeBSD is loosing users because of this issue. Thanks, Evren From dudu at dudu.ro Mon Oct 6 15:36:06 2008 From: dudu at dudu.ro (Vlad GALU) Date: Mon Oct 6 15:36:13 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <48EA21AE.80607@ispro.net> References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> Message-ID: On Mon, Oct 6, 2008 at 5:33 PM, Evren Yurtesen wrote: > Bob Bishop wrote: > >>> Does anybody have free time and skills to give a hand? Please see: >>> http://forum.r1soft.com/showpost.php?p=3414&postcount=9 >> >> Should be possible to do this with a geom(4) class? >> > > I am not saying it is impossible. They just need somebody to put them to > right track I guess. I personally cant do that. It would be nice if somebody > who has knowledge in this area contacts r1soft. At the very least r1soft > seems to be willing to communicate on this issue. > > Continuous backups as well as bare-metal-restore seem to be a key feature > for many hosters. FreeBSD is loosing users because of this issue. gmirror+ggate come to mind as a nifty solution ... > > Thanks, > Evren > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > -- ~/.signature: no such file or directory From danny at cs.huji.ac.il Mon Oct 6 15:46:14 2008 From: danny at cs.huji.ac.il (Danny Braniss) Date: Mon Oct 6 15:46:28 2008 Subject: bad NFS/UDP performance In-Reply-To: References: <20080926081806.GA19055@icarus.home.lan> <20080926095230.GA20789@icarus.home.lan> Message-ID: > > On Sat, 4 Oct 2008, Danny Braniss wrote: > > > at the moment, the best I can do is run it on a different hardware that has > > if_em, the results are in > > ftp://ftp.cs.huji.ac.il/users/danny/lock.prof/7.1-1000.em the > > benchmark ran better with the Intel NIC, averaged UDP 54MB/s, TCP 53MB/s (I > > get the same numbers with an older kernel). > > Dear Danny: > > Unfortunately, I was left slightly unclear on the comparison you are making > above. Could you confirm whether or not, with if_em, you see a performance > regression using UDP NFS between 7.0-RELEASE and the most recent 7.1-STABLE, > and if you do, whether or not the RLOCK->WLOCK change has any effect on > performance? It would be nice to know on the same hardware but at least with > different hardware we get a sense of whether or not this might affect other > systems or whether it's limited to a narrower set of configurations. > > Thanks, 7.1-1000.em vanilla 7.1 1 x Intel Core Duo 7.1-1000.x2200.em vanilla 7.1 2 x Dual-Core AMD Opteron 7.0-1000.x2200.em 7.0 + RLOCK->WLOCK the plot thickens. I put an em card in, and the throughput is almost the same than with the bge. all the tests were done on the same host, a Sun x2200/amd/2cpux2core except for the one over the weekend that is a intel Core Duo, and not the same if_em card, sorry about that but one has PCI X, the other PCI Express :-(. what is becoming obvious is that NFS/UDP is very temperamental/sensitive :-) danny From zbeeble at gmail.com Mon Oct 6 16:23:02 2008 From: zbeeble at gmail.com (Zaphod Beeblebrox) Date: Mon Oct 6 16:23:09 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <48EA21AE.80607@ispro.net> References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> Message-ID: <5f67a8c40810060852k4c51c8far511891c4b135a1e2@mail.gmail.com> On Mon, Oct 6, 2008 at 10:33 AM, Evren Yurtesen wrote: [regarding r1soft.com, ...] > I am not saying it is impossible. They just need somebody to put them to > right track I guess. I personally cant do that. It would be nice if somebody > who has knowledge in this area contacts r1soft. At the very least r1soft > seems to be willing to communicate on this issue. > > Continuous backups as well as bare-metal-restore seem to be a key feature > for many hosters. FreeBSD is loosing users because of this issue. Actually, having looked at the site, the hammer filesystem and it's replication strategy seem to be the most applicable technology (but then you wouldn't even need these guys --- you'd be doing it yourself). Like anything, though, live applications will require special treatment. Keeping a live filesystem replicated does in no way guarentee that your database (for instance) will be sane at any particular moment. It sounds like these guys have made allowances for MySQL (they specifically mention it), but this won't help the PostgreSQL users, etc. I've spent a lot of time thinking about redundancy and I've come to one inescapable conclusion: That the further up the stack you design for redundancy, the cheaper and easier it becomes. Most databases have replication strategies of one type or another that don't require exotic hosting solutions to work. The most fundamental example I can think of to show this principle, however, is the fact that if the HTTP standard required web browsers to try all A records (instead of randomly choosing one), web site redundancy would be amazingly simple to achieve. Consider that most other protocols right down to telnet do this, but web browsers don't. As a complete aside, if you have both AAAA and A records for your website, you have a form of poor-man's redundancy available to you --- with the caveat that it only works for people with both IPv6 and IPv4 connectivity. Browsers will try AAAA followed by A if the former doesn't respond. From volker at vwsoft.com Mon Oct 6 18:05:38 2008 From: volker at vwsoft.com (Volker) Date: Mon Oct 6 18:05:47 2008 Subject: touch screen recommendation? In-Reply-To: <48E90904.4020007@telenix.org> References: <48E90904.4020007@telenix.org> Message-ID: <48EA5357.8050503@vwsoft.com> On 12/23/-58 20:59, Chuck Robey wrote: > I was wondering if anyone here had a recommendation for a touch screen, > specifically to run on FreeBSD? Any user report? I am wondering how this belongs to hackers@? Now the real answer: It depends. Touchscreen != Touchscreen, as there are different technologies available (better ask a sales person for your application). If you just want to get a "click" from a screen device, you can use any. First, I'm not aware of any working USB based devices working under FreeBSD (except mine, not yet released to public). There are other devices with RS-232 connectors for which you may find X.Org drivers in the port tree. For the usb devices, I've got working drivers for GeneralTouch ST-601U (buggy firmware, dumb tech support, not my preference, but it's working) and @-Touch SAWUSB. Next week I should have a working driver for Zytronic devices and one or two others. All for usb devices, everything for the BSDs only but not yet released. From volker at vwsoft.com Mon Oct 6 18:52:21 2008 From: volker at vwsoft.com (Volker) Date: Mon Oct 6 18:52:29 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <48E9E1BB.6020908@ispro.net> References: <48E9E1BB.6020908@ispro.net> Message-ID: <48EA56BB.6040702@vwsoft.com> On 12/23/-58 20:59, Evren Yurtesen wrote: > Hello, > > As far as I can see, there is no continuous backup solution for FreeBSD > at the moment. I talked with R1Soft and they seem to not be able to > support FreeBSD and need help. > > Does anybody have free time and skills to give a hand? Please see: > http://forum.r1soft.com/showpost.php?p=3414&postcount=9 Quoting the thread: > and they are usually very busy... and they like to work for hire. Wrong term: Even open source folks need something for living. That's too cheap and I don't suggest supporting a company like this, as they're expecting somebody else to do their job and make profit of it. They're free to use a system like FreeBSD for their products, they're free to modify and redistribute it but they need to do their job and don't expect anybody else to code for them for free. Next quote of thread: > real world competency writing block device drivers for FreeBSD Let me check that out... %ls -l /dev | grep ^b % Well, I guess, nobody is able to do so and it's noted in the handbook or in GNN's "The Design and Implementation of the FreeBSD operating system". A good read. > I know people who want to switch to using FreeBSD but cant do that > because their backup solution (R1Soft) does not support FreeBSD :( What's wrong with solutions like Bacula? It's working, cross platform, well supported. And now, go ahead and flame me for not doing work for free for other peoples profit but I'm pretty sure there won't be much guys wanting to do that. Volker From bakul at bitblocks.com Mon Oct 6 19:07:13 2008 From: bakul at bitblocks.com (Bakul Shah) Date: Mon Oct 6 19:07:21 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: Your message of "Mon, 06 Oct 2008 18:09:06 +0300." References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> Message-ID: <20081006184934.04A645B4C@mail.bitblocks.com> On Mon, 06 Oct 2008 18:09:06 +0300 "Vlad GALU" wrote: > On Mon, Oct 6, 2008 at 5:33 PM, Evren Yurtesen wrote: > > Bob Bishop wrote: > > > >>> Does anybody have free time and skills to give a hand? Please see: > >>> http://forum.r1soft.com/showpost.php?p=3414&postcount=9 > >> > >> Should be possible to do this with a geom(4) class? > >> > > > > I am not saying it is impossible. They just need somebody to put them to > > right track I guess. I personally cant do that. It would be nice if somebod > y > > who has knowledge in this area contacts r1soft. At the very least r1soft > > seems to be willing to communicate on this issue. > > > > Continuous backups as well as bare-metal-restore seem to be a key feature > > for many hosters. FreeBSD is loosing users because of this issue. > > gmirror+ggate come to mind as a nifty solution ... My guess is these guys do something simpler like keep keep track of changed blocks since the last backup and periodically dump those blocks to a server. This is good enough for backups (but not mirroring) and it has low memory overhead (1 or 2 bits per block), lower network overhead than remote mirroring (you send a block at most once every sync interval), and a tiny loss of performance (over no backups). May be someone ought to do a garchive device! From yurtesen at ispro.net Mon Oct 6 19:28:37 2008 From: yurtesen at ispro.net (Evren Yurtesen) Date: Mon Oct 6 19:34:18 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> Message-ID: <48EA66DB.8020903@ispro.net> Vlad GALU wrote: > On Mon, Oct 6, 2008 at 5:33 PM, Evren Yurtesen wrote: >> Bob Bishop wrote: >> >>>> Does anybody have free time and skills to give a hand? Please see: >>>> http://forum.r1soft.com/showpost.php?p=3414&postcount=9 >>> Should be possible to do this with a geom(4) class? >>> >> I am not saying it is impossible. They just need somebody to put them to >> right track I guess. I personally cant do that. It would be nice if somebody >> who has knowledge in this area contacts r1soft. At the very least r1soft >> seems to be willing to communicate on this issue. >> >> Continuous backups as well as bare-metal-restore seem to be a key feature >> for many hosters. FreeBSD is loosing users because of this issue. > > gmirror+ggate come to mind as a nifty solution ... That allows mirroring however not a very practical solution. You would need to mirror every drive in another machine and you cant restore the drives into a earlier time. I am talking about a real backup solution where the backup agent collecting the information about written sectors and sending to backup server. You can then restore a file which existed in the box 1 hour ago if you need to. For example think about a situation where a server is processing important data which shouldnt get lost. A software failure wipes out the hard drive. You would loose all the data in the mirror as well. Also the setup of such mirroring system would be rather complicated. In addition to that, the mirroring does not support for example restore of mysql databases in table level. Think about a customer who wiped out his database accidentally. All the data would be gone in the mirror as well. With near continuous backup you can restore the data to just moments before the deletion process. Traditional backup systems at best daily backups, even if you could restore the data, the data could be up to 1 day old. More on that subject, r1soft supports multiple hosting control panel softwares. For example H-Sphere ( http://www.parallels.com/hsphere/ ) etc. through plugins which allow hosting customers to restore their own data easily. Something impossible with gmirror+ggate combination (since it does not actually backup the data and only mirror it) and not even practical if it was possible, if you have thousands of users. Actually, I am not saying that anybody should be doing about this and neither am I an r1soft advocate. I am just pointing out that there is a company out there which can provide a valuable software tool and they need somebody to put them into right direction only. If you know some FreeBSD developers who know the disk subsystem and can help R1Soft then you can perhaps forward this information to them. This not a feature or help request and I just am mentioning this. However you can imagine that a company which is giving online services on a serious scale probably would be interested in a CDP solution. It is currently impossible with FreeBSD. This makes using FreeBSD servers on critical applications a little bit insecure data protectionwise thus often people prefer Linux which is supported by such continuous data protection type backup solutions. (not that I am advocating FreeBSD or Linux here but this is the situation). Thanks, Evrne From yurtesen at ispro.net Mon Oct 6 19:38:40 2008 From: yurtesen at ispro.net (Evren Yurtesen) Date: Mon Oct 6 19:52:18 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <5f67a8c40810060852k4c51c8far511891c4b135a1e2@mail.gmail.com> References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> <5f67a8c40810060852k4c51c8far511891c4b135a1e2@mail.gmail.com> Message-ID: <48EA6939.6090405@ispro.net> Zaphod Beeblebrox wrote: > On Mon, Oct 6, 2008 at 10:33 AM, Evren Yurtesen > wrote: > > [regarding r1soft.com , ...] > > > I am not saying it is impossible. They just need somebody to put > them to right track I guess. I personally cant do that. It would be > nice if somebody who has knowledge in this area contacts r1soft. At > the very least r1soft seems to be willing to communicate on this issue. > > Continuous backups as well as bare-metal-restore seem to be a key > feature for many hosters. FreeBSD is loosing users because of this > issue. > > > Actually, having looked at the site, the hammer filesystem and it's > replication strategy seem to be the most applicable technology (but then > you wouldn't even need these guys --- you'd be doing it yourself). Like > anything, though, live applications will require special treatment. > Keeping a live filesystem replicated does in no way guarentee that your > database (for instance) will be sane at any particular moment. It > sounds like these guys have made allowances for MySQL (they specifically > mention it), but this won't help the PostgreSQL users, etc. I think you didnt get the point here. Replication or mirroring != backup. You cant return back to how things were 1 hour ago. Also they support postgresql as well (while its usage is way smaller than mysql) http://www.r1soft.com/CDP_db_postgreSQL.html In any case, the product guarantees that it can return your databases to any point in the time. Do you see what you are missing? :) > I've spent a lot of time thinking about redundancy and I've come to one > inescapable conclusion: That the further up the stack you design for > redundancy, the cheaper and easier it becomes. Most databases have > replication strategies of one type or another that don't require exotic > hosting solutions to work. The idea/problem is not redundancy here, it is data protection. Thanks, Evren From rea-fbsd at codelabs.ru Mon Oct 6 20:31:02 2008 From: rea-fbsd at codelabs.ru (Eygene Ryabinkin) Date: Mon Oct 6 20:31:15 2008 Subject: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages In-Reply-To: References: <48DE5CC0.9000708@localhost.inse.ru> <48DF6735.4030906@quip.cz> <4bESZpNwE3z/DdlE2fwK/BXzQSo@2MQ0uKCiT7mdMUuLeUzs8Nv3ToQ> <48E94281.8010300@quip.cz> <48E9D382.4000001@quip.cz> Message-ID: Mon, Oct 06, 2008 at 02:30:29PM +0400, Eygene Ryabinkin wrote: > OK, fine. I will implement the usage of the local INDEX file in some > days. OK, I had implemented both '-o' option to pkg_audit and the usage of the local INDEX file. I had reworked pkg_audit and portaudit a bit further, mostly fixing some issues (both mine and existing). Here we go. Patches for pkg_install that adds pkg_audit: http://codelabs.ru/fbsd/patches/portaudit/0001-Add-functions-for-traversing-package-database-and-ma.patch http://codelabs.ru/fbsd/patches/portaudit/0002-Add-function-match_get_pkgorigin.patch http://codelabs.ru/fbsd/patches/portaudit/0003-New-utility-pkg_audit.patch http://codelabs.ru/fbsd/patches/portaudit/0004-pkg_audit-add-option-to-print-origins.patch Mega-patch for pkg_install: http://codelabs.ru/fbsd/patches/portaudit/pkg_install-megapatch-pkg_audit.diff Patches for portaudit: http://codelabs.ru/fbsd/patches/portaudit/0001-Avoid-usage-of-global-variables-N-in-the-print_affe.patch http://codelabs.ru/fbsd/patches/portaudit/0002-Separate-vulnerable-ports-search-from-the-formatter.patch http://codelabs.ru/fbsd/patches/portaudit/0003-Use-pkg_audit-utility-if-it-is-available.patch http://codelabs.ru/fbsd/patches/portaudit/0004-Implement-checking-for-a-new-package-versions.patch Mega-patch for portaudit: http://codelabs.ru/fbsd/patches/portaudit/portaudit-megapatch_pkg_audit-and-checknew.diff Opinions are welcome! -- Eygene From rea-fbsd at codelabs.ru Mon Oct 6 20:36:19 2008 From: rea-fbsd at codelabs.ru (Eygene Ryabinkin) Date: Mon Oct 6 20:36:28 2008 Subject: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages In-Reply-To: References: <48DE5CC0.9000708@localhost.inse.ru> <48DF6735.4030906@quip.cz> <4bESZpNwE3z/DdlE2fwK/BXzQSo@2MQ0uKCiT7mdMUuLeUzs8Nv3ToQ> <48E94281.8010300@quip.cz> <48E9D382.4000001@quip.cz> Message-ID: Forgot to say: Tue, Oct 07, 2008 at 12:30:58AM +0400, Eygene Ryabinkin wrote: > OK, I had implemented both '-o' option to pkg_audit and the usage of the > local INDEX file. The latter can be activated by writing something like ----- portaudit_pkg_index="file:///usr/ports/INDEX-%d" ----- to the /usr/local/etc/portaudit.conf. -- Eygene From hartzell at alerce.com Mon Oct 6 21:43:56 2008 From: hartzell at alerce.com (George Hartzell) Date: Mon Oct 6 22:06:33 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <20081006184934.04A645B4C@mail.bitblocks.com> References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> <20081006184934.04A645B4C@mail.bitblocks.com> Message-ID: <18666.33296.607120.889620@almost.alerce.com> Bakul Shah writes: > On Mon, 06 Oct 2008 18:09:06 +0300 "Vlad GALU" wrote: > > On Mon, Oct 6, 2008 at 5:33 PM, Evren Yurtesen wrote: > > > Bob Bishop wrote: > > > > > >>> Does anybody have free time and skills to give a hand? Please see: > > >>> http://forum.r1soft.com/showpost.php?p=3414&postcount=9 > > >> > > >> Should be possible to do this with a geom(4) class? > > >> > > > > > > I am not saying it is impossible. They just need somebody to put them to > > > right track I guess. I personally cant do that. It would be nice if somebod > > y > > > who has knowledge in this area contacts r1soft. At the very least r1soft > > > seems to be willing to communicate on this issue. > > > > > > Continuous backups as well as bare-metal-restore seem to be a key feature > > > for many hosters. FreeBSD is loosing users because of this issue. > > > > gmirror+ggate come to mind as a nifty solution ... > > My guess is these guys do something simpler like keep keep > track of changed blocks since the last backup and > periodically dump those blocks to a server. This is good > enough for backups (but not mirroring) and it has low memory > overhead (1 or 2 bits per block), lower network overhead than > remote mirroring (you send a block at most once every sync > interval), and a tiny loss of performance (over no backups). > May be someone ought to do a garchive device! There were a couple of threads about using kqueue or other FreeBSD tools to build something like Mac OS X's Time Machine. R1soft's software sounds very similar. The conclusion seemed to be that it'd be doable. Here's a pointer to the start of one of the threads. http://lists.freebsd.org/pipermail/freebsd-hackers/2008-June/024730.html g. From yurtesen at ispro.net Mon Oct 6 21:56:59 2008 From: yurtesen at ispro.net (Evren Yurtesen) Date: Mon Oct 6 22:06:45 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <48EA3451.7040801@miralink.com> References: <48E9E1BB.6020908@ispro.net> <48EA3451.7040801@miralink.com> Message-ID: <48EA83CE.4060702@ispro.net> Sean Bruno wrote: > Evren Yurtesen wrote: >> Hello, >> >> As far as I can see, there is no continuous backup solution for >> FreeBSD at the moment. I talked with R1Soft and they seem to not be >> able to support FreeBSD and need help. >> >> Does anybody have free time and skills to give a hand? Please see: >> http://forum.r1soft.com/showpost.php?p=3414&postcount=9 >> >> I know people who want to switch to using FreeBSD but cant do that >> because their backup solution (R1Soft) does not support FreeBSD :( >> >> Thanks, >> Evren >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to >> "freebsd-hackers-unsubscribe@freebsd.org" > Hmmm....Well, my company (MiraLink) makes a BSD based appliance that > mirrors block level changes between two units. > > It's commercial, not BSD licensed, but it's something. > > www.miralink.com > > sean > > Hello Sean, Thanks for the link. The miralink products seem to be doing data replication similar to gmirror+ggate? This wont protect against accidental deletion of data etc. The near continuous backup solutions do not mirror/replicate the data. Think about it as continuously taking backup. You can return back to 10 minutes or 1 hour before and restore old data. In either case, I wasnt trying to start a debate between backup solutions etc. here. I simply wanted to ask for help if somebody who has in depth knowledge about disk subsystems of FreeBSD and can give some tips to r1soft so FreeBSD could be supported also. As you can imagine, it is not only important that data can be restored when a box hardware failure etc. it is also important that data can be restored if deleted by accidents etc. While traditional backup programs provide this functionality, you cant really go back to 10 min or 1h ago, often they take daily backups and have to scan whole filesystem for changed files every time the backup is taken which stresses out the systems. Thanks, Evren From yurtesen at ispro.net Mon Oct 6 22:01:38 2008 From: yurtesen at ispro.net (Evren Yurtesen) Date: Mon Oct 6 22:07:02 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <48EA56BB.6040702@vwsoft.com> References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> Message-ID: <48EA8B3A.3090609@ispro.net> Volker wrote: > On 12/23/-58 20:59, Evren Yurtesen wrote: >> Hello, >> >> As far as I can see, there is no continuous backup solution for FreeBSD >> at the moment. I talked with R1Soft and they seem to not be able to >> support FreeBSD and need help. >> >> Does anybody have free time and skills to give a hand? Please see: >> http://forum.r1soft.com/showpost.php?p=3414&postcount=9 > > Quoting the thread: >> and they are usually very busy... and they like to work for hire. > > Wrong term: Even open source folks need something for living. That's too > cheap and I don't suggest supporting a company like this, as they're > expecting somebody else to do their job and make profit of it. > > They're free to use a system like FreeBSD for their products, they're > free to modify and redistribute it but they need to do their job and > don't expect anybody else to code for them for free. I agree, perhaps whoever can help than can ask for money for the job done and I am sure they would pay reasonably since this is a commercial company. But as far as I can see people here do not even know the difference between near continous backup and mirroring. I just wanted to inform that there is such solutions nowadays available but FreeBSD users are not able to take advantage of them and the company who is making the product is interested in supporting FreeBSD but perhaps somebody who has experience can give some hints to them. They actually do not think that it is an easy job to adapt their software to support FreeBSD even. See this post: http://forum.r1soft.com/showpost.php?p=4224&postcount=3 > Next quote of thread: >> real world competency writing block device drivers for FreeBSD > > Let me check that out... > %ls -l /dev | grep ^b > % > > Well, I guess, nobody is able to do so and it's noted in the handbook or > in GNN's "The Design and Implementation of the FreeBSD operating > system". A good read. You can go ahead and explain it to r1soft or any other software company which makes near continuous backup solutions. So maybe they can find this information and improve their product. So far at least this company thinks that this is impossible to do with FreeBSD as far as I can tell. >> I know people who want to switch to using FreeBSD but cant do that >> because their backup solution (R1Soft) does not support FreeBSD :( > > What's wrong with solutions like Bacula? It's working, cross platform, > well supported. Do you know what is near continuous backup solution or ever heard of such technologies? I dont see how you can compare this with Bacula. They are not the same thing really. I can tell you one thing which is wrong with Bacula, it scans the whole filesystem everytime it takes a backup even incremental backups (while this disk io loading operation is unnecessary with CDP), and Bacula cant restore your data to what there was 10 minutes ago at any time of the day. > And now, go ahead and flame me for not doing work for free for other > peoples profit but I'm pretty sure there won't be much guys wanting to > do that. There is nothing to flame or blame really. I can only blame my own stupidity. I posted this information here in hopes that somebody capable can give some hints to r1soft or to another near continuous backup solution so we could all benefit. But what do I get? All I get is information about solutions which mirror / replicate the data or suggestions of standard backup programs. I am not even surprised that FreeBSD is not supported. FreeBSD users do not even know the difference between a CDP solution or mirroring / replication or traditional backup... Go figure... If you dont know what is CDP then please read and learn (there are links to wikipedia articles in this address also) http://www.r1soft.com/CDP.html I never told that anybody should make the software ready for selling by another company for free, neither the post in the thread is asking for somebody to write a driver for free. The company obviously understands that such job requires an expert who works for money. They probably wouldnt mind if somebody wrote it for free to them (who wouldnt) but it doesnt say that they wouldnt pay. If you can do the job, please contact the company and give your price. If they tell that they want it done for free, then come and complain that they want it done for free. I am sorry if I was a little bit out of line here but a simple question became an unnecessary debate really. It was more or less a yes or no question :) Thanks, Evren From zbeeble at gmail.com Mon Oct 6 22:08:30 2008 From: zbeeble at gmail.com (Zaphod Beeblebrox) Date: Mon Oct 6 22:08:37 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <48EA6939.6090405@ispro.net> References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> <5f67a8c40810060852k4c51c8far511891c4b135a1e2@mail.gmail.com> <48EA6939.6090405@ispro.net> Message-ID: <5f67a8c40810061508t300e77c7n8c1439462622a71c@mail.gmail.com> On Mon, Oct 6, 2008 at 3:38 PM, Evren Yurtesen wrote: > Zaphod Beeblebrox wrote: > >> On Mon, Oct 6, 2008 at 10:33 AM, Evren Yurtesen > yurtesen@ispro.net>> wrote: >> >> [regarding r1soft.com , ...] >> >> I am not saying it is impossible. They just need somebody to put >> them to right track I guess. I personally cant do that. It would be >> nice if somebody who has knowledge in this area contacts r1soft. At >> the very least r1soft seems to be willing to communicate on this issue. >> >> Continuous backups as well as bare-metal-restore seem to be a key >> feature for many hosters. FreeBSD is loosing users because of this >> issue. >> >> >> Actually, having looked at the site, the hammer filesystem and it's >> replication strategy seem to be the most applicable technology (but then you >> wouldn't even need these guys --- you'd be doing it yourself). Like >> anything, though, live applications will require special treatment. Keeping >> a live filesystem replicated does in no way guarentee that your database >> (for instance) will be sane at any particular moment. It sounds like these >> guys have made allowances for MySQL (they specifically mention it), but this >> won't help the PostgreSQL users, etc. >> > > I think you didnt get the point here. Replication or mirroring != backup. > You cant return back to how things were 1 hour ago. Actually, right back at you. You didn't fathom the meaning in my statement. While your post was vague, I read the company's website to determine the featureset they were claiming (although I missed their postgres support --- I only read the "features" page). NetBSD's hammer filesystem achieves replication at the filesystem layer (which will do infinitely better at this problem than a block-only driver) by maintaining a history of what's happened and being able to "select" (as in the database term) changes to the filesystem that have occurred since the last batch of blocks were shiped out to replication. This gives you both fine grained recovery (basically changes to files are kept until your "rules" define they should be freed) and replication and fine grained recovery on the other side of replication. In fact, Hammer delivers what they claim in a far more sophisticated way. (but it's NetBSD, not FreeBSD, unless someone decides to port it) > Also they support postgresql as well (while its usage is way smaller than > mysql) > http://www.r1soft.com/CDP_db_postgreSQL.html > > In any case, the product guarantees that it can return your databases to > any point in the time. Do you see what you are missing? :) > Well... "I" am not missing it. I have that without making my filesystem jump through an enormous hoop. But designing "my" application correctly, I have that feature for far less effort. (that was the other point of my post) > I've spent a lot of time thinking about redundancy and I've come to one >> inescapable conclusion: That the further up the stack you design for >> redundancy, the cheaper and easier it becomes. Most databases have >> replication strategies of one type or another that don't require exotic >> hosting solutions to work. >> > > The idea/problem is not redundancy here, it is data protection. Well... no... you don't need fine grained filesystem history for data integrity (unless you let loose a bunch of summer students armed with the ability to RM or your application is faulty (in which case, your filesystem won't protect you). As I said, This can all be achieved with other far simpler solutions. However, if your dev team isn't smart enough (or don't care for some reason), then you can take advantage of their product and pay their price. From mwm-keyword-freebsdhackers2.e313df at mired.org Mon Oct 6 22:28:32 2008 From: mwm-keyword-freebsdhackers2.e313df at mired.org (Mike Meyer) Date: Mon Oct 6 22:28:40 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <18666.33296.607120.889620@almost.alerce.com> References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> <20081006184934.04A645B4C@mail.bitblocks.com> <18666.33296.607120.889620@almost.alerce.com> Message-ID: <20081006182558.708e4094@bhuda.mired.org> On Mon, 6 Oct 2008 14:24:32 -0700 George Hartzell wrote: > There were a couple of threads about using kqueue or other FreeBSD > tools to build something like Mac OS X's Time Machine. R1soft's > software sounds very similar. Time machine doesn't do continuous backups, it does them once an hour or so. People have built similar systems on top of rsync; I did it on top of zfs (turned out to be to fragile, though). You then just need a spiffy GUI for wondering through the backups. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From alancyang at gmail.com Mon Oct 6 23:09:41 2008 From: alancyang at gmail.com (alan yang) Date: Mon Oct 6 23:09:48 2008 Subject: kgdb debugging Message-ID: <290865fd0810061544ubbe92fdsf75501bb729da3f0@mail.gmail.com> hi, there, wonder people can shed some lights on remote debugging. i have freebsd7 configured with option DDB / KDB / GDB but after entering the db on the target system the command gdb gives "the remote GDB backend could not be selected". i browsed through the mailing list, and do find 1 similar post but without answer. thanks in advance & apology if i overlooked things ... cheers, alan From yurtesen at ispro.net Mon Oct 6 22:40:33 2008 From: yurtesen at ispro.net (Evren Yurtesen) Date: Mon Oct 6 23:15:30 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <5f67a8c40810061508t300e77c7n8c1439462622a71c@mail.gmail.com> References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> <5f67a8c40810060852k4c51c8far511891c4b135a1e2@mail.gmail.com> <48EA6939.6090405@ispro.net> <5f67a8c40810061508t300e77c7n8c1439462622a71c@mail.gmail.com> Message-ID: <48EA9459.2000807@ispro.net> Zaphod Beeblebrox wrote: > Actually, right back at you. You didn't fathom the meaning in my > statement. While your post was vague, I read the company's website to I am sorry what was vague since I wrote 'continuous backup' in my post? The whole idea is such a basic idea that if you put this to google you can get wikipedia entry about it in first 3 results (I see in 2nd). Maybe you didnt know anything about it makes it vague to you. The message I sent was quite clear and plain. > determine the featureset they were claiming (although I missed their > postgres support --- I only read the "features" page). NetBSD's hammer > filesystem achieves replication at the filesystem layer (which will do > infinitely better at this problem than a block-only driver) by > maintaining a history of what's happened and being able to "select" (as > in the database term) changes to the filesystem that have occurred since > the last batch of blocks were shiped out to replication. This gives you > both fine grained recovery (basically changes to files are kept until > your "rules" define they should be freed) and replication and fine > grained recovery on the other side of replication. In fact, Hammer > delivers what they claim in a far more sophisticated way. You might want to read this page too: http://www.r1soft.com/CDP.html > (but it's NetBSD, not FreeBSD, unless someone decides to port it) While Hammer might be doing a similar job, it is a filesystem not a backup application and it wont replace backups. It doesnt just store the data in a backup server. While eventually it might become a backup solution, it will take years before that can happen. Even then, people will not just switch their current filesystems overnight. The CDP backup softwares are available today, it just needs a sort of driver to function in current system. > Also they support postgresql as well (while its usage is way smaller > than mysql) > http://www.r1soft.com/CDP_db_postgreSQL.html > > In any case, the product guarantees that it can return your > databases to any point in the time. Do you see what you are missing? :) > > > Well... "I" am not missing it. I have that without making my filesystem > jump through an enormous hoop. But designing "my" application > correctly, I have that feature for far less effort. (that was the other > point of my post) Can you just explain how can one currently do that in FreeBSD? Is it as easy as in Linux with CDP backup softwares? such as installing a program and done? > > > I've spent a lot of time thinking about redundancy and I've come > to one inescapable conclusion: That the further up the stack you > design for redundancy, the cheaper and easier it becomes. Most > databases have replication strategies of one type or another > that don't require exotic hosting solutions to work. > > > The idea/problem is not redundancy here, it is data protection. > > > Well... no... you don't need fine grained filesystem history for data > integrity (unless you let loose a bunch of summer students armed with > the ability to RM or your application is faulty (in which case, your > filesystem won't protect you). As I said, This can all be achieved with > other far simpler solutions. However, if your dev team isn't smart > enough (or don't care for some reason), then you can take advantage of > their product and pay their price. There is no perfect system. This is exactly why people take backups. If what you said was applicable then there wouldnt be any need for backup software. People would just make sure that they dont loose their data. In addition to this, I have no control of if my customer will delete his/her data accidentally. I cant make a system which does not allow customers to delete data. I have given an example of a simple solution that Linux users can utilize (which obviously we also can utilize if we put our heads into it and give some directions as r1soft is willing to support FreeBSD). While you are first saying that these can be achieved with far simpler solutions, at the same time you are saying that the dev team must be smart enough. My solution is simple enough to write here. You install Linux on all machines and then CDP backup server on the backup server and CDP agent on the client machines, is yours simpler? Then explain how we can create similar sort of data protection? Thanks, Evren From gahr at FreeBSD.org Mon Oct 6 23:26:52 2008 From: gahr at FreeBSD.org (Pietro Cerutti) Date: Mon Oct 6 23:27:24 2008 Subject: kgdb debugging In-Reply-To: <290865fd0810061544ubbe92fdsf75501bb729da3f0@mail.gmail.com> References: <290865fd0810061544ubbe92fdsf75501bb729da3f0@mail.gmail.com> Message-ID: <48EA9E95.80105@FreeBSD.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 alan yang wrote: | hi, there, | | wonder people can shed some lights on remote debugging. i have | freebsd7 configured with option DDB / KDB / GDB but after entering the | db on the target system the command gdb gives "the remote GDB backend | could not be selected". | | i browsed through the mailing list, and do find 1 similar post but | without answer. | | thanks in advance & apology if i overlooked things ... I suggest the following tutorial: http://www.lemis.com/grog/Papers/Debug-tutorial/tutorial.pdf Have fun :) | | cheers, | alan | _______________________________________________ | freebsd-hackers@freebsd.org mailing list | http://lists.freebsd.org/mailman/listinfo/freebsd-hackers | To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" - -- Pietro Cerutti gahr@FreeBSD.org PGP Public Key: http://gahr.ch/pgp -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEAREKAAYFAkjqnpQACgkQwMJqmJVx945RSwCgoDb0JTr8LSFDB1vpAbGUjb76 ZH0An19HpFVJJTUB5/XnyZc0pIDzgxc3 =6Pdm -----END PGP SIGNATURE----- From zbeeble at gmail.com Tue Oct 7 00:07:27 2008 From: zbeeble at gmail.com (Zaphod Beeblebrox) Date: Tue Oct 7 00:07:34 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <48EA9459.2000807@ispro.net> References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> <5f67a8c40810060852k4c51c8far511891c4b135a1e2@mail.gmail.com> <48EA6939.6090405@ispro.net> <5f67a8c40810061508t300e77c7n8c1439462622a71c@mail.gmail.com> <48EA9459.2000807@ispro.net> Message-ID: <5f67a8c40810061707m33a52547idae13e2bb9eb2f9a@mail.gmail.com> On Mon, Oct 6, 2008 at 6:42 PM, Evren Yurtesen wrote: > Zaphod Beeblebrox wrote: > >> Actually, right back at you. You didn't fathom the meaning in my >> statement. While your post was vague, I read the company's website to >> > > While Hammer might be doing a similar job, it is a filesystem not a backup > application and it wont replace backups. It doesnt just store the data in a > backup server. While eventually it might become a backup solution, it will > take years before that can happen. Even then, people will not just switch > their current filesystems overnight. The CDP backup softwares are available > today, it just needs a sort of driver to function in current system. > >From my reading, Hammer is much more than a filesystem, but then you probably havn't read about it yet. By my reading, Hammer hits all their feature points and does it better _because_ it's a filesystem. Well... "I" am not missing it. I have that without making my filesystem > jump through an enormous hoop. But designing "my" application correctly, I > have that feature for far less effort. (that was the other point of my > post) > Can you just explain how can one currently do that in FreeBSD? Is it as easy > as in Linux with CDP backup softwares? such as installing a program and > done? > It's relatively simple. Database replication solves the data backup problem (I don't store application data outside the database). Database replication for both MySQL and PostgreSQL is relatively straight forward. As for the configuration of code and servers --- that is taken care of with configuration management (it's really a bigger issue than just backing up a filesystem) and installing a new server to take a place in the cluster is a straight forward checkout from the CM system. For things I really care about staying up, add VRRP and an application design that is fault tolerant. This actually works rather well if you do your research. Database replication is possible at all kinds of different link speeds and distances. Database replication also allows you to control your data better --- you know more about your data than a block replicator would. It means that your backup is already live and it means that, with the right scripts, invoking a backup on primary failure is simple. Database replication on some databases even allows you to preserve transactions --- which is important in some cases. > There is no perfect system. This is exactly why people take backups. If > what you said was applicable then there wouldnt be any need for backup > software. People would just make sure that they dont loose their data. In > addition to this, I have no control of if my customer will delete his/her > data accidentally. I cant make a system which does not allow customers to > delete data. > > I have given an example of a simple solution that Linux users can utilize > (which obviously we also can utilize if we put our heads into it and give > some directions as r1soft is willing to support FreeBSD). While you are > first saying that these can be achieved with far simpler solutions, at the > same time you are saying that the dev team must be smart enough. > > My solution is simple enough to write here. You install Linux on all > machines and then CDP backup server on the backup server and CDP agent on > the client machines, is yours simpler? Then explain how we can create > similar sort of data protection? Well... no. Backup software and strategies are just one available tool for risk mitigation. To know what tools you require, you must define your risks. Then with your list of risks you look at the cost of each tool and find the toolset that suits you. By the responses in this thread, it seems like the set of FreeBSD developers and the set of people who desire this solution are disjoint. Actually... as some obligatory positive content, the time travel features of Hammer should be straightforward to implement in ZFS... are ZFS modules supported on FreeBSD yet? It would seem to be a logical module. From hartzell at alerce.com Mon Oct 6 23:57:16 2008 From: hartzell at alerce.com (George Hartzell) Date: Tue Oct 7 02:17:21 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <20081006182558.708e4094@bhuda.mired.org> References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> <20081006184934.04A645B4C@mail.bitblocks.com> <18666.33296.607120.889620@almost.alerce.com> <20081006182558.708e4094@bhuda.mired.org> Message-ID: <18666.40675.636312.893786@almost.alerce.com> Mike Meyer writes: > On Mon, 6 Oct 2008 14:24:32 -0700 > George Hartzell wrote: > > There were a couple of threads about using kqueue or other FreeBSD > > tools to build something like Mac OS X's Time Machine. R1soft's > > software sounds very similar. > > Time machine doesn't do continuous backups, it does them once an hour > or so. People have built similar systems on top of rsync; I did it on > top of zfs (turned out to be to fragile, though). You then just need a > spiffy GUI for wondering through the backups. On the other hand Time Machine does take advantage of a kernel based mechanism that watches file activity and does its best to take advantage of that information to avoid scanning the filesystem when it does a backup. That's the context of the message thread that I pointed to (again, for completeness) http://lists.freebsd.org/pipermail/freebsd-hackers/2008-June/024730.html The thread seemed relevant given the context of backup systems that watch filesystem io. g. From des at des.no Tue Oct 7 10:12:51 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Tue Oct 7 10:13:02 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <48EA8B3A.3090609@ispro.net> (Evren Yurtesen's message of "Tue, 07 Oct 2008 01:03:38 +0300") References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> Message-ID: <861vysiv9i.fsf@ds4.des.no> Evren Yurtesen writes: > They actually do not think that it is an easy job to adapt their > software to support FreeBSD even. See this post: > http://forum.r1soft.com/showpost.php?p=4224&postcount=3 All this shows is that they don't know anything about FreeBSD at all (plus they need a refresher course in OS design; Linux is also a monolithic kernel) What really annoys me with this thread is that nobody has provided any information at all that would allow someone to understand what needs to be done and estimate how hard it would be. DES -- Dag-Erling Sm?rgrav - des@des.no From yurtesen at ispro.net Tue Oct 7 08:11:21 2008 From: yurtesen at ispro.net (Evren Yurtesen) Date: Tue Oct 7 11:33:09 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <5f67a8c40810061707m33a52547idae13e2bb9eb2f9a@mail.gmail.com> References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> <5f67a8c40810060852k4c51c8far511891c4b135a1e2@mail.gmail.com> <48EA6939.6090405@ispro.net> <5f67a8c40810061508t300e77c7n8c1439462622a71c@mail.gmail.com> <48EA9459.2000807@ispro.net> <5f67a8c40810061707m33a52547idae13e2bb9eb2f9a@mail.gmail.com> Message-ID: <48EB1A1C.7020701@ispro.net> I think here might be a misunderstanding. I was talking about a reliable backup solution whereas you guys are all the time talking about mirroring and replication type solutions. Since you cant be thinking that mirroring and replication can replace backup, there must be a misunderstanding? Zaphod Beeblebrox wrote: > > From my reading, Hammer is much more than a filesystem, but then you > probably havn't read about it yet. By my reading, Hammer hits all their > feature points and does it better _because_ it's a filesystem. I glanced through these actually: http://www.dragonflybsd.org/hammer/ I didnt see anywhere that it will replace backup programs? > It's relatively simple. Database replication solves the data backup > problem (I don't store application data outside the database). Database > replication for both MySQL and PostgreSQL is relatively straight > forward. As for the configuration of code and servers --- that is taken > care of with configuration management (it's really a bigger issue than > just backing up a filesystem) and installing a new server to take a > place in the cluster is a straight forward checkout from the CM system. > For things I really care about staying up, add VRRP and an application > design that is fault tolerant. Have you ever tried to restore data from MySQL replication logs? :) Even if you use binary logging, when you want to go back in time. You will need to first restore the whole database first from normal backups then replay the logs until the point that you wanna be. There is no simple way to go back in time. That is of course you have backups. If you dont have backups because you think replication is a backup solution, you would be screwed. Totally more complicated that clicking from the web to select data and time and table and restore! Also, you are thinking about a very small sized system. While replication might work if you are relatively small sized company (like 1-2 servers). If you have many independent servers with different databases inside you just cant use it. Even if you could replicate multiple boxes into one box, there would eventually be problems such as same named databases etc. and even then, you cant just easily restore the data if the user deletes all his data in his tables. Also that is not practical for users at all. For example I cant give an option to the user to restore his data by himself. While that is possible with most backup software easily. About VRRP etc. I already told that I am not talking about redundancy here. You are talking about totally different things. I need data protection. > This actually works rather well if you do your research. Database > replication is possible at all kinds of different link speeds and > distances. Database replication also allows you to control your data > better --- you know more about your data than a block replicator would. > It means that your backup is already live and it means that, with the > right scripts, invoking a backup on primary failure is simple. Database > replication on some databases even allows you to preserve transactions > --- which is important in some cases. And how do you propose that I restore a table in the database to of 1h before status? like you can do with a data backup solution? You are talking about a spare server solution not backup solution. Replication IS NOT backup. If you look at articles and information about database replication, almost all mention that it DOES NOT replace backups. > Well... no. Backup software and strategies are just one available tool > for risk mitigation. To know what tools you require, you must define > your risks. Then with your list of risks you look at the cost of each > tool and find the toolset that suits you. By the responses in this > thread, it seems like the set of FreeBSD developers and the set of > people who desire this solution are disjoint. Right, I just cant use the tool I require. There is no way to take near continuous backups of FreeBSD filesystems. > Actually... as some obligatory positive content, the time travel > features of Hammer should be straightforward to implement in ZFS... are > ZFS modules supported on FreeBSD yet? It would seem to be a logical module. Those features work within the filesystem, you there is no mention of mirroring to a remote hard drive, if you could mirror to a remote hard drive, you couldnt easily mirror 10 machines into 1 remote hard drive, even if you could do that, it would require total disk space of all those 10 drives to exist in that 1 backup server. Even if you could overcome that, this would mean that all FreeBSD users wanting to take advantage of such system to convert their filesystems. Even if it was easy enough, there is no GUI tools to allow users exist in those systems to restore files by themselves easily. As you can see, none of the solutions you are suggesting is anywhere near simple solutions which can replace near continous backup solutions. Thanks, Evren From Ramachandran.Sathyanarayanan at aricent.com Tue Oct 7 10:59:17 2008 From: Ramachandran.Sathyanarayanan at aricent.com (Ramachandran Sathyanarayanan) Date: Tue Oct 7 11:33:40 2008 Subject: sleep is not working to avoid starvation among threads of same priority Message-ID: Hi I hope this is the right mailer for this question. pls let us know your inputs on this. Thanks Ram ________________________________ From: Rajeshwar Patil Sent: Tuesday, October 07, 2008 3:36 PM To: freebsd-questions@freebsd.org Cc: Ramachandran Sathyanarayanan; Rajeshwar Patil Subject: sleep is not working to avoid starvation among threads of same priority Hi, I have two threads of same priority lets say thread A and thread B, thread A is starving since thread B has got a for loop of size 100k. So, to avoid starvation I am processing 1000 iterations of for loop(that of thread B) in one batch and giving sleep of 1ms, but still thread A is starving. If I increase the batch size used in for loop from 1k to 10k then starvation of thread A is little less whereas it should be more as Im increasing processing time of thread B. I tried various combinations of sleeps and batches, but I couldnt solve the starvation of thread A. Is the sleep right solution? I will be grateful if someone could answer on this. Thanks Rajeshwar ________________________________ "DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error,please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus." From Ramachandran.Sathyanarayanan at aricent.com Tue Oct 7 11:58:54 2008 From: Ramachandran.Sathyanarayanan at aricent.com (Ramachandran Sathyanarayanan) Date: Tue Oct 7 11:59:00 2008 Subject: sleep is not working to avoid starvation among threads of same priority Message-ID: Hi I hope this is the right mailer for this question. pls let us know your inputs on this. Thanks Ram ________________________________ From: Rajeshwar Patil Sent: Tuesday, October 07, 2008 3:36 PM To: freebsd-questions@freebsd.org Cc: Ramachandran Sathyanarayanan; Rajeshwar Patil Subject: sleep is not working to avoid starvation among threads of same priority Hi, I have two threads of same priority lets say thread A and thread B, thread A is starving since thread B has got a for loop of size 100k. So, to avoid starvation I am processing 1000 iterations of for loop(that of thread B) in one batch and giving sleep of 1ms, but still thread A is starving. If I increase the batch size used in for loop from 1k to 10k then starvation of thread A is little less whereas it should be more as Im increasing processing time of thread B. I tried various combinations of sleeps and batches, but I couldnt solve the starvation of thread A. Is the sleep right solution? I will be grateful if someone could answer on this. Thanks Rajeshwar ________________________________ "DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error,please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus." From jille at quis.cx Tue Oct 7 12:20:07 2008 From: jille at quis.cx (Jille Timmermans) Date: Tue Oct 7 12:20:34 2008 Subject: sleep is not working to avoid starvation among threads of same priority In-Reply-To: References: Message-ID: <48EB5036.80209@quis.cx> Hello, Take a look at pthread_yield(): DESCRIPTION The pthread_yield() forces the running thread to relinquish the processor until it again becomes the head of its thread list. (Note that it is not portable) And if you want to use sleep, I found out that using sleep with more ms does 'yield' the thread/process. iirc, 20ms was enough to stop it from using 100% CPU. -- Jille Ramachandran Sathyanarayanan wrote: > Hi > > I hope this is the right mailer for this question. pls let us know your inputs on this. > > Thanks > > Ram > > ________________________________ > From: Rajeshwar Patil > Sent: Tuesday, October 07, 2008 3:36 PM > To: freebsd-questions@freebsd.org > Cc: Ramachandran Sathyanarayanan; Rajeshwar Patil > Subject: sleep is not working to avoid starvation among threads of same priority > > Hi, > > I have two threads of same priority lets say thread A and thread B, thread A is starving since thread B has got a for loop of size 100k. So, to avoid starvation I am processing 1000 iterations of for loop(that of thread B) in one batch and giving sleep of 1ms, but still thread A is starving. If I increase the batch size used in for loop from 1k to 10k then starvation of thread A is little less whereas it should be more as Im increasing processing time of thread B. I tried various combinations of sleeps and batches, but I couldnt solve the starvation of thread A. > > Is the sleep right solution? > > I will be grateful if someone could answer on this. > > Thanks > Rajeshwar > > > > > > > ________________________________ > "DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error,please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus." > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From shaun at FreeBSD.org Tue Oct 7 16:25:22 2008 From: shaun at FreeBSD.org (Shaun Amott) Date: Tue Oct 7 16:25:27 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <48EA83CE.4060702@ispro.net> References: <48E9E1BB.6020908@ispro.net> <48EA3451.7040801@miralink.com> <48EA83CE.4060702@ispro.net> Message-ID: <20081007155059.GA20615@charon.picobyte.net> On Tue, Oct 07, 2008 at 12:31:58AM +0300, Evren Yurtesen wrote: > > so FreeBSD could be supported also. As you can imagine, it is not only > important that data can be restored when a box hardware failure etc. it is > also important that data can be restored if deleted by accidents etc. While > traditional backup programs provide this functionality, you cant really go > back to 10 min or 1h ago, often they take daily backups and have to scan > whole filesystem for changed files every time the backup is taken which > stresses out the systems. > This can (more or less) be achieved with snapshots: you can cheaply maintain old versions of the file system, and mount an old snapshot at any time. Hourly is about as fine-grained as you can expect though. -- Shaun Amott // PGP: 0x6B387A9A "A foolish consistency is the hobgoblin of little minds." - Ralph Waldo Emerson From zbeeble at gmail.com Tue Oct 7 16:25:52 2008 From: zbeeble at gmail.com (Zaphod Beeblebrox) Date: Tue Oct 7 16:26:01 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <48EB1A1C.7020701@ispro.net> References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> <5f67a8c40810060852k4c51c8far511891c4b135a1e2@mail.gmail.com> <48EA6939.6090405@ispro.net> <5f67a8c40810061508t300e77c7n8c1439462622a71c@mail.gmail.com> <48EA9459.2000807@ispro.net> <5f67a8c40810061707m33a52547idae13e2bb9eb2f9a@mail.gmail.com> <48EB1A1C.7020701@ispro.net> Message-ID: <5f67a8c40810070925k109e3510y18fa64861ef06f03@mail.gmail.com> On Tue, Oct 7, 2008 at 4:13 AM, Evren Yurtesen wrote: > > Zaphod Beeblebrox wrote: > > >> From my reading, Hammer is much more than a filesystem, but then you >> probably havn't read about it yet. By my reading, Hammer hits all their >> feature points and does it better _because_ it's a filesystem. >> > > I glanced through these actually: > http://www.dragonflybsd.org/hammer/ > I didnt see anywhere that it will replace backup programs? > You need to read the PDF on that page. On the first page of the PDF the 4th point is "queue-less incremental mirroring" If you read the PDF to determine what this phrase means, you'll find it describes the filesystem as a database indexed by a B+ tree of radix 64. It mentions that you can easily select all changes newer than a certain time. The problem with backup solutions that run on raw blocks is that you need meta-data to queue up the blocks that have changed faster than you can ship them out. Hammer solves this by allowing you to select changes based on when you shipped out the last batch of changes. The granularity of this will be dependent on how fast you can do this. > > It's relatively simple. Database replication solves the data backup >> problem (I don't store application data outside the database). Database >> replication for both MySQL and PostgreSQL is relatively straight forward. >> As for the configuration of code and servers --- that is taken care of with >> configuration management (it's really a bigger issue than just backing up a >> filesystem) and installing a new server to take a place in the cluster is a >> straight forward checkout from the CM system. For things I really care >> about staying up, add VRRP and an application design that is fault tolerant. >> > > Have you ever tried to restore data from MySQL replication logs? :) Even if > you use binary logging, when you want to go back in time. You will need to > first restore the whole database first from normal backups then replay the > logs until the point that you wanna be. There is no simple way to go back in > time. That is of course you have backups. If you dont have backups because > you think replication is a backup solution, you would be screwed. Totally > more complicated that clicking from the web to select data and time and > table and restore! Largely why I use PostgreSQL. > Also, you are thinking about a very small sized system. While replication > might work if you are relatively small sized company (like 1-2 servers). If > you have many independent servers with different databases inside you just > cant use it. Even if you could replicate multiple boxes into one box, there > would eventually be problems such as same named databases etc. and even > then, you cant just easily restore the data if the user deletes all his data > in his tables. Again, this is application design. I know of rather large organizations that use PostgreSQL replication to deliver very serious and onerous SLAs. Several of them have sponsored the Slovney (sp? russian for elephan --- the PostgreSQL multi-master replication thing). > Also that is not practical for users at all. For example I cant give an > option to the user to restore his data by himself. While that is possible > with most backup software easily. Well... if we're talking about a random samba servers for windoze users, ZFS is the tech you want. Users can retrieve their own snapshots, etc. ZFS was designed with generalized fileserver duties in mind. > And how do you propose that I restore a table in the database to of 1h > before status? like you can do with a data backup solution? You are talking > about a spare server solution not backup solution. Replication IS NOT > backup. If you look at articles and information about database replication, > almost all mention that it DOES NOT replace backups. > I use differential backups to give old database restore functionality. I use replication to give me a live alternative database for failures (different risks, different solutions). > Well... no. Backup software and strategies are just one available tool >> for risk mitigation. To know what tools you require, you must define your >> risks. Then with your list of risks you look at the cost of each tool and >> find the toolset that suits you. By the responses in this thread, it seems >> like the set of FreeBSD developers and the set of people who desire this >> solution are disjoint. >> > > Right, I just cant use the tool I require. There is no way to take near > continuous backups of FreeBSD filesystems. > You're free to pay anyone you'd like to implement this solution. I'd be happy to work for hire. I can provide a quote if you're serious about it. But if I was doing it for free, I'd want it to be fun and interesting. I'd be imlementing a server as well as a client. In fact, if I were doing this as my own open source project, I'd be looking at a better way of achieving it. FreeBSD is, after all, about doing things right. But --- as I said --- regardless of what I'd do if I were doing it for free, I will achieve the goals set out for me if I'm paid. From zbeeble at gmail.com Tue Oct 7 16:37:39 2008 From: zbeeble at gmail.com (Zaphod Beeblebrox) Date: Tue Oct 7 16:37:46 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <861vysiv9i.fsf@ds4.des.no> References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> Message-ID: <5f67a8c40810070937r5ba89773ncee407ace25fa0dd@mail.gmail.com> I wanted to respond to DES' email separately --- because he's right. On Tue, Oct 7, 2008 at 5:56 AM, Dag-Erling Sm?rgrav wrote: > Evren Yurtesen writes: > > They actually do not think that it is an easy job to adapt their > > software to support FreeBSD even. See this post: > > http://forum.r1soft.com/showpost.php?p=4224&postcount=3 > > All this shows is that they don't know anything about FreeBSD at all > (plus they need a refresher course in OS design; Linux is also a > monolithic kernel) A very succinct way of making a similar poing :) > What really annoys me with this thread is that nobody has provided any > information at all that would allow someone to understand what needs to > be done and estimate how hard it would be. Well... I hinted that a hammer port would be sufficient (although they need to finish their replication design) and I hinted that the hammer approach may be graftable to ZFS. Both reasonably large effort-wise (but probably within the scope of a single developer with sufficient time). My primary concern about hammer is it's floating history stuff. It seems to me that legal compliance might have some things to say about it. It seems (from the document --- I havn't read the source) that the tunability of the "real deletion" of data are "goals" not absolutes. This is a concern. But as filesystems _are_ databases and as they grow database technology (like transactions and B+tree indexes), we should look to database systems for some of the solutions (ie: problems already solved). Replication replacing "mirroring" is just one of them. Doing this at the block level was suggested by someone earlier (BTW). Suggesting that a geom node could store a bit or two per bock (marking it "dirty" prehaps) and shipping off the blocks. That only solves the replication. You'd need something like a transaction ID per block stored on the backup server to enable time travel. Probably want a B+tree index there too. If you're not careful, you might find implementing the filesystem solution much easier. You could, however, implement this with hooks for gjournal --- such that the filesystems you backup are always sane. From bakul at bitblocks.com Tue Oct 7 16:40:16 2008 From: bakul at bitblocks.com (Bakul Shah) Date: Tue Oct 7 16:40:23 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: Your message of "Tue, 07 Oct 2008 11:56:09 +0200." <861vysiv9i.fsf@ds4.des.no> References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> Message-ID: <20081007164015.2E3085B29@mail.bitblocks.com> On Tue, 07 Oct 2008 11:56:09 +0200 =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= wrote: > Evren Yurtesen writes: > > They actually do not think that it is an easy job to adapt their > > software to support FreeBSD even. See this post: > > http://forum.r1soft.com/showpost.php?p=3D4224&postcount=3D3 > > All this shows is that they don't know anything about FreeBSD at all > (plus they need a refresher course in OS design; Linux is also a > monolithic kernel) > > What really annoys me with this thread is that nobody has provided any > information at all that would allow someone to understand what needs to > be done and estimate how hard it would be. >From their http://forum.r1soft.com/CDP.html page: R1Soft's Continuous Data Protection solution is a ==>>near-Continuous Backups system capable of providing <<== hundreds of recovery points per day scheduled as little as 5 or 10 minutes apart. ... CDP Server works by reading your hard disk volumes at the sector level, bypassing the file system for the ultimate in performance and recovery. This disk sector synchronization is performed while the server is online and provides no interruption to other I/O requests even on a busy server. Clearly "near-Continuous" is *not* the same as "continuous" but never mind -- truthiness in business is so last century! But this could be the cause of some confusion. What they do is backups, not mirroring. A remote mirror would essentially require a continuous "backup" -- every disk write must be sent right away but in pure mirroring there is no access previous snapshots. In a true backup solution you can restore disk state to some number of previous backup points, regardless of whether you have *online* access to them. My guess is they have a driver that keeps track of disk writes. Something like set bit N of a bitmap when sector N is to be written. Then once every 10 minutes (or whatever snapshot interval you have selected) a client app scans the bitmap and sends these sectors to the backup server. If they did *just* this, there'd be consistency issues -- between the time a snapshot is taken and some sector N is actually backed up, there may be new writes to N by the OS. To deal with this, the new write must be delayed until N has been backed up. Another alternative is to "slide" forward the snapshot point. That is, if the snapshot was taken at time T1 and the backup finished by T2, and there were conflicting writes during [T1..T2), backup these writes as well and slide forward this snapshot time from T1 to T2. Repeat until there are no conflicting writes. This latter method won't block any writes from the OS. So my guess is they need an interface where they get notified for every disk write and optionally a way to delay a write. [To respond to an earlier point raised by others, I believe OS X Time Machine does a filesystem level backup and not at the disk level.] From zbeeble at gmail.com Tue Oct 7 16:42:35 2008 From: zbeeble at gmail.com (Zaphod Beeblebrox) Date: Tue Oct 7 16:42:42 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <20081007155059.GA20615@charon.picobyte.net> References: <48E9E1BB.6020908@ispro.net> <48EA3451.7040801@miralink.com> <48EA83CE.4060702@ispro.net> <20081007155059.GA20615@charon.picobyte.net> Message-ID: <5f67a8c40810070942k317d55f4i5d46b1409ef34034@mail.gmail.com> On Tue, Oct 7, 2008 at 11:50 AM, Shaun Amott wrote: > On Tue, Oct 07, 2008 at 12:31:58AM +0300, Evren Yurtesen wrote: > > > > so FreeBSD could be supported also. As you can imagine, it is not only > > important that data can be restored when a box hardware failure etc. it > is > > also important that data can be restored if deleted by accidents etc. > While > > traditional backup programs provide this functionality, you cant really > go > > back to 10 min or 1h ago, often they take daily backups and have to scan > > whole filesystem for changed files every time the backup is taken which > > stresses out the systems. > > > > This can (more or less) be achieved with snapshots: you can cheaply > maintain old versions of the file system, and mount an old snapshot at > any time. Hourly is about as fine-grained as you can expect though. > Unfortunately, as mentioned on another recent thread, FreeBSD UFS snapshots are quite fragile. The last time I tried this type-of-thing, it rather quickly resulted in filesystem corruption on moderately busy filesystems. Also, the cost of UFS snapshots (in time, while locking the filesystem) are unacceptable in practice. Now... ZFS seems to do this well enough. I use snapshots and send snapshots to my backup server --- it all seems to work OK. The idea that snapshots have infinite granularity and are automatic (introduced to me by the Hammer filesystem document) has merit, though --- and this is certainly core to the featurelist of the company that started this thread . From alancyang at gmail.com Tue Oct 7 17:53:23 2008 From: alancyang at gmail.com (alan yang) Date: Tue Oct 7 17:53:37 2008 Subject: kgdb debugging In-Reply-To: <290865fd0810061712sfdf5a0p4d3954773ee27a3d@mail.gmail.com> References: <290865fd0810061544ubbe92fdsf75501bb729da3f0@mail.gmail.com> <48EA9E95.80105@FreeBSD.org> <290865fd0810061712sfdf5a0p4d3954773ee27a3d@mail.gmail.com> Message-ID: <290865fd0810071053k6da13d96j391ade1a30599fa1@mail.gmail.com> Could people shed some light how to get remote debugging going, must be something that i overlooked, really appreciate. Two FreeBSD7 systems, target and development, connected with null modem cable on each's COM1. step 1) - rebuild kernel with following options: options DDB options KDB options GDB makeoptions DEBUG=-g step 2) - from development system cd to /usr/src/sys/i386/comiple/MYKERNEL kgdb -r /dev/cuad0 kernel.debug it displays the following: Switching to remote protocol Ignoring packet error, continuing ......... Couldn't establish connection to remote target Malformed response to offset query, timeout step 3) - from targetsystem: 1. Ctrl + Alt + Esc to go into db 2. from db> type gdb 3. it displays: "The remote GDB backend could not be selected" On Mon, Oct 6, 2008 at 5:12 PM, alan yang wrote: > the problem is, when entering gdb from db as described in the following: > -- > Enter ing gdb from ddb > In FreeBSD you can build a kernel with support for both ddb and gdb. > You can then > change backwards and forwards between them. For example, if you're in > ddb, you can > go to gdb like this: > db> gdb > Next trap will enter GDB remote protocol mode > db> si step a single instruction to reenter ddb > -- > after typing gdb command, it says: "The remote GDB backend could not > be selected" > that i am not sure what this indicates; looking at subr_kdb.c, wonder > maybe kdb_dbbe_set is not set? seems kdb_dbbe_set is not referenced > anywhere, not sure how to get it right. > > thanks for shed some light ... > > On Mon, Oct 6, 2008 at 4:26 PM, Pietro Cerutti wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA512 >> >> alan yang wrote: >> | hi, there, >> | >> | wonder people can shed some lights on remote debugging. i have >> | freebsd7 configured with option DDB / KDB / GDB but after entering the >> | db on the target system the command gdb gives "the remote GDB backend >> | could not be selected". >> | >> | i browsed through the mailing list, and do find 1 similar post but >> | without answer. >> | >> | thanks in advance & apology if i overlooked things ... >> >> I suggest the following tutorial: >> http://www.lemis.com/grog/Papers/Debug-tutorial/tutorial.pdf >> >> Have fun :) >> >> | >> | cheers, >> | alan >> | _______________________________________________ >> | freebsd-hackers@freebsd.org mailing list >> | http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> | To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >> >> >> - -- >> Pietro Cerutti >> gahr@FreeBSD.org >> >> PGP Public Key: >> http://gahr.ch/pgp >> >> -----BEGIN PGP SIGNATURE----- >> Version: GnuPG v2.0.9 (FreeBSD) >> >> iEYEAREKAAYFAkjqnpQACgkQwMJqmJVx945RSwCgoDb0JTr8LSFDB1vpAbGUjb76 >> ZH0An19HpFVJJTUB5/XnyZc0pIDzgxc3 >> =6Pdm >> -----END PGP SIGNATURE----- >> > From nparhar at gmail.com Tue Oct 7 18:12:08 2008 From: nparhar at gmail.com (Navdeep Parhar) Date: Tue Oct 7 18:12:15 2008 Subject: kgdb debugging In-Reply-To: <290865fd0810071053k6da13d96j391ade1a30599fa1@mail.gmail.com> References: <290865fd0810061544ubbe92fdsf75501bb729da3f0@mail.gmail.com> <48EA9E95.80105@FreeBSD.org> <290865fd0810061712sfdf5a0p4d3954773ee27a3d@mail.gmail.com> <290865fd0810071053k6da13d96j391ade1a30599fa1@mail.gmail.com> Message-ID: On Tue, Oct 7, 2008 at 10:53 AM, alan yang wrote: > Could people shed some light how to get remote debugging going, must > be something that i overlooked, really appreciate. > Do you have the right flags for sio or uart in /boot/device.hints? I have this (for a recent HEAD):: hint.uart.0.flags="0x90" You are probably using sio as it is a FreeBSD7 system. From the man page of sio (the part that talks about flags):: ... 0x00080 use this port for remote kernel debugging ... Make sure you have this bit set. Regards, Navdeep > Two FreeBSD7 systems, target and development, connected with null > modem cable on each's COM1. > > step 1) > - rebuild kernel with following options: > options DDB > options KDB > options GDB > > makeoptions DEBUG=-g > > step 2) > - from development system > cd to /usr/src/sys/i386/comiple/MYKERNEL > kgdb -r /dev/cuad0 kernel.debug > > it displays the following: > Switching to remote protocol > Ignoring packet error, continuing > ......... > Couldn't establish connection to remote target > Malformed response to offset query, timeout > > step 3) > - from targetsystem: > 1. Ctrl + Alt + Esc to go into db > 2. from db> type gdb > 3. it displays: "The remote GDB backend could not be selected" > > > > On Mon, Oct 6, 2008 at 5:12 PM, alan yang wrote: >> the problem is, when entering gdb from db as described in the following: >> -- >> Enter ing gdb from ddb >> In FreeBSD you can build a kernel with support for both ddb and gdb. >> You can then >> change backwards and forwards between them. For example, if you're in >> ddb, you can >> go to gdb like this: >> db> gdb >> Next trap will enter GDB remote protocol mode >> db> si step a single instruction to reenter ddb >> -- >> after typing gdb command, it says: "The remote GDB backend could not >> be selected" >> that i am not sure what this indicates; looking at subr_kdb.c, wonder >> maybe kdb_dbbe_set is not set? seems kdb_dbbe_set is not referenced >> anywhere, not sure how to get it right. >> >> thanks for shed some light ... >> >> On Mon, Oct 6, 2008 at 4:26 PM, Pietro Cerutti wrote: >>> -----BEGIN PGP SIGNED MESSAGE----- >>> Hash: SHA512 >>> >>> alan yang wrote: >>> | hi, there, >>> | >>> | wonder people can shed some lights on remote debugging. i have >>> | freebsd7 configured with option DDB / KDB / GDB but after entering the >>> | db on the target system the command gdb gives "the remote GDB backend >>> | could not be selected". >>> | >>> | i browsed through the mailing list, and do find 1 similar post but >>> | without answer. >>> | >>> | thanks in advance & apology if i overlooked things ... >>> >>> I suggest the following tutorial: >>> http://www.lemis.com/grog/Papers/Debug-tutorial/tutorial.pdf >>> >>> Have fun :) >>> >>> | >>> | cheers, >>> | alan >>> | _______________________________________________ >>> | freebsd-hackers@freebsd.org mailing list >>> | http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >>> | To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >>> >>> >>> - -- >>> Pietro Cerutti >>> gahr@FreeBSD.org >>> >>> PGP Public Key: >>> http://gahr.ch/pgp >>> >>> -----BEGIN PGP SIGNATURE----- >>> Version: GnuPG v2.0.9 (FreeBSD) >>> >>> iEYEAREKAAYFAkjqnpQACgkQwMJqmJVx945RSwCgoDb0JTr8LSFDB1vpAbGUjb76 >>> ZH0An19HpFVJJTUB5/XnyZc0pIDzgxc3 >>> =6Pdm >>> -----END PGP SIGNATURE----- >>> >> > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From julian at elischer.org Tue Oct 7 18:14:13 2008 From: julian at elischer.org (Julian Elischer) Date: Tue Oct 7 18:14:26 2008 Subject: kgdb debugging In-Reply-To: <290865fd0810071053k6da13d96j391ade1a30599fa1@mail.gmail.com> References: <290865fd0810061544ubbe92fdsf75501bb729da3f0@mail.gmail.com> <48EA9E95.80105@FreeBSD.org> <290865fd0810061712sfdf5a0p4d3954773ee27a3d@mail.gmail.com> <290865fd0810071053k6da13d96j391ade1a30599fa1@mail.gmail.com> Message-ID: <48EBA6F2.2090606@elischer.org> alan yang wrote: > Could people shed some light how to get remote debugging going, must > be something that i overlooked, really appreciate. > > Two FreeBSD7 systems, target and development, connected with null > modem cable on each's COM1. > > step 1) > - rebuild kernel with following options: > options DDB > options KDB > options GDB > > makeoptions DEBUG=-g add hints.dev.uart.0.flags=0xc0 (or whatever it is) (see man uart or man sio) to /boot/device.hints > > step 2) > - from development system > cd to /usr/src/sys/i386/comiple/MYKERNEL > kgdb -r /dev/cuad0 kernel.debug > > it displays the following: > Switching to remote protocol > Ignoring packet error, continuing > ......... > Couldn't establish connection to remote target > Malformed response to offset query, timeout > > step 3) > - from targetsystem: > 1. Ctrl + Alt + Esc to go into db > 2. from db> type gdb > 3. it displays: "The remote GDB backend could not be selected" > > > > On Mon, Oct 6, 2008 at 5:12 PM, alan yang wrote: >> the problem is, when entering gdb from db as described in the following: >> -- >> Enter ing gdb from ddb >> In FreeBSD you can build a kernel with support for both ddb and gdb. >> You can then >> change backwards and forwards between them. For example, if you're in >> ddb, you can >> go to gdb like this: >> db> gdb >> Next trap will enter GDB remote protocol mode >> db> si step a single instruction to reenter ddb >> -- >> after typing gdb command, it says: "The remote GDB backend could not >> be selected" >> that i am not sure what this indicates; looking at subr_kdb.c, wonder >> maybe kdb_dbbe_set is not set? seems kdb_dbbe_set is not referenced >> anywhere, not sure how to get it right. >> >> thanks for shed some light ... >> >> On Mon, Oct 6, 2008 at 4:26 PM, Pietro Cerutti wrote: >>> -----BEGIN PGP SIGNED MESSAGE----- >>> Hash: SHA512 >>> >>> alan yang wrote: >>> | hi, there, >>> | >>> | wonder people can shed some lights on remote debugging. i have >>> | freebsd7 configured with option DDB / KDB / GDB but after entering the >>> | db on the target system the command gdb gives "the remote GDB backend >>> | could not be selected". >>> | >>> | i browsed through the mailing list, and do find 1 similar post but >>> | without answer. >>> | >>> | thanks in advance & apology if i overlooked things ... >>> >>> I suggest the following tutorial: >>> http://www.lemis.com/grog/Papers/Debug-tutorial/tutorial.pdf >>> >>> Have fun :) >>> >>> | >>> | cheers, >>> | alan >>> | _______________________________________________ >>> | freebsd-hackers@freebsd.org mailing list >>> | http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >>> | To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >>> >>> >>> - -- >>> Pietro Cerutti >>> gahr@FreeBSD.org >>> >>> PGP Public Key: >>> http://gahr.ch/pgp >>> >>> -----BEGIN PGP SIGNATURE----- >>> Version: GnuPG v2.0.9 (FreeBSD) >>> >>> iEYEAREKAAYFAkjqnpQACgkQwMJqmJVx945RSwCgoDb0JTr8LSFDB1vpAbGUjb76 >>> ZH0An19HpFVJJTUB5/XnyZc0pIDzgxc3 >>> =6Pdm >>> -----END PGP SIGNATURE----- >>> > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From tummala.ushasri at gmail.com Wed Oct 8 03:07:40 2008 From: tummala.ushasri at gmail.com (ushasri tummala) Date: Wed Oct 8 03:07:47 2008 Subject: What is the time between 2 mi_switches in freebsd. Message-ID: <53fa490b0810071947j23fc0f72n5360b6f174ddc96d@mail.gmail.com> What is the time between 2 mi_switches in freebsd?In which variable is this information stored? Could you plz help me out with this Thank You, Usha. From des at des.no Wed Oct 8 08:14:59 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed Oct 8 08:15:06 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <5f67a8c40810070937r5ba89773ncee407ace25fa0dd@mail.gmail.com> (Zaphod Beeblebrox's message of "Tue, 7 Oct 2008 12:37:37 -0400") References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <5f67a8c40810070937r5ba89773ncee407ace25fa0dd@mail.gmail.com> Message-ID: <86iqs3sdtp.fsf@ds4.des.no> "Zaphod Beeblebrox" writes: > "Dag-Erling Sm?rgrav" writes: > > What really annoys me with this thread is that nobody has provided > > any information at all that would allow someone to understand what > > needs to be done and estimate how hard it would be. > Well... I hinted that a hammer port would be sufficient (although they > need to finish their replication design) and I hinted that the hammer > approach may be graftable to ZFS. Both reasonably large effort-wise > (but probably within the scope of a single developer with sufficient > time). No... you're so far off the mark it's not even funny, especially when it's been repeatedly pointed out to you. This is not a file system, it's a backup system. It's not designed to survive a disk crash or an accidental file deletion, it's designed to survive a direct missile strike on your colo center. To quote Wikipedia, "CDP is a service that captures changes to data to a separate storage location" - emphasis on "separate". DES -- Dag-Erling Sm?rgrav - des@des.no From des at des.no Wed Oct 8 08:18:48 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed Oct 8 08:18:59 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <20081007164015.2E3085B29@mail.bitblocks.com> (Bakul Shah's message of "Tue, 07 Oct 2008 09:40:15 -0700") References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <20081007164015.2E3085B29@mail.bitblocks.com> Message-ID: <86ej2rsdnc.fsf@ds4.des.no> Bakul Shah writes: > "Dag-Erling Sm?rgrav" writes: > > What really annoys me with this thread is that nobody has provided any > > information at all that would allow someone to understand what needs to > > be done and estimate how hard it would be. > From their http://forum.r1soft.com/CDP.html page: [...] You completely missed the mark. I know what R1Soft's product is. What I want to know is what needs to be done to port it to FreeBSD. DES -- Dag-Erling Sm?rgrav - des@des.no From rea-fbsd at codelabs.ru Wed Oct 8 08:28:31 2008 From: rea-fbsd at codelabs.ru (Eygene Ryabinkin) Date: Wed Oct 8 08:28:39 2008 Subject: HEADS UP: GCC 4.2.0 is coming In-Reply-To: <48EC67FF.3030900@zedat.fu-berlin.de> References: <20070518192007.6e6a91e1@kan.dnsalias.net> <20070519022016.2b4a6bda@kan.dnsalias.net> <48EC67FF.3030900@zedat.fu-berlin.de> Message-ID: Good day. Wed, Oct 08, 2008 at 07:57:51AM +0000, O. Hartmann wrote: > Alexander Kabaev wrote: > > On Fri, 18 May 2007 19:20:07 -0400 > > Alexander Kabaev wrote: > > > >> HEADS UP: I will start importing GCC 4.2.0 bits in about one hour and > >> plan to finish in a couple of hours after that. > >> > >> The src/ tree will be utterly broken meanwhile. I'll send an 'all > >> clear' message when done. > > > > Done. > > > > Just for those who aren't on the cutting edge: why gcc 4.2.0 and not > 4.2.1 as it is used in 7.X? It was the old message that slipped to the list due to some misconfiguration of the mailing list software. Look at its date -- it's more than one year old. -- Eygene _ ___ _.--. # \`.|\..----...-'` `-._.-'_.-'` # Remember that it is hard / ' ` , __.--' # to read the on-line manual )/' _/ \ `-_, / # while single-stepping the kernel. `-'" `"\_ ,_.-;_.-\_ ', fsc/as # _.-'_./ {_.' ; / # -- FreeBSD Developers handbook {_.-``-' {_/ # -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081008/95084fed/attachment.pgp From ertr1013 at student.uu.se Wed Oct 8 08:40:10 2008 From: ertr1013 at student.uu.se (Erik Trulsson) Date: Wed Oct 8 08:40:20 2008 Subject: HEADS UP: GCC 4.2.0 is coming In-Reply-To: <48EC67FF.3030900@zedat.fu-berlin.de> References: <20070518192007.6e6a91e1@kan.dnsalias.net> <20070519022016.2b4a6bda@kan.dnsalias.net> <48EC67FF.3030900@zedat.fu-berlin.de> Message-ID: <20081008082449.GA3741@owl.midgard.homeip.net> On Wed, Oct 08, 2008 at 07:57:51AM +0000, O. Hartmann wrote: > Alexander Kabaev wrote: > > On Fri, 18 May 2007 19:20:07 -0400 > > Alexander Kabaev wrote: > > > >> HEADS UP: I will start importing GCC 4.2.0 bits in about one hour and > >> plan to finish in a couple of hours after that. > >> > >> The src/ tree will be utterly broken meanwhile. I'll send an 'all > >> clear' message when done. > > > > Done. > > > > Just for those who aren't on the cutting edge: why gcc 4.2.0 and not > 4.2.1 as it is used in 7.X? Take a close look at the date of the message you were replying to. It is somewhat old. (There seems to have been a mail-server hiccup somewhere.) -- Erik Trulsson ertr1013@student.uu.se From olli at lurza.secnetix.de Wed Oct 8 11:20:05 2008 From: olli at lurza.secnetix.de (Oliver Fromme) Date: Wed Oct 8 11:20:12 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <86iqs3sdtp.fsf@ds4.des.no> Message-ID: <200810081120.m98BK2fV043545@lurza.secnetix.de> Dag-Erling Sm?rgrav wrote: > "Zaphod Beeblebrox" writes: > > "Dag-Erling Sm?rgrav" writes: > > > What really annoys me with this thread is that nobody has provided > > > any information at all that would allow someone to understand what > > > needs to be done and estimate how hard it would be. > > Well... I hinted that a hammer port would be sufficient (although they > > need to finish their replication design) and I hinted that the hammer > > approach may be graftable to ZFS. Both reasonably large effort-wise > > (but probably within the scope of a single developer with sufficient > > time). > > No... you're so far off the mark it's not even funny, especially when > it's been repeatedly pointed out to you. This is not a file system, > it's a backup system. It's not designed to survive a disk crash or an > accidental file deletion, it's designed to survive a direct missile > strike on your colo center. > > To quote Wikipedia, "CDP is a service that captures changes to data to a > separate storage location" - emphasis on "separate". FWIW, the HAMMER file system _does_ support replication to remote targets (thus "separate"). Unfortunately they call this feature "mirroring", which is misleading at best. It's really rather a replication mechanism, much like the binlog of MySQL. It can be used for various purposes, including live mirroring, delayed mirroring, archiving, backup and point-in-time recovery. Well, of course, all of that doesn't help us at all because HAMMER doesn't exist on FreeBSD. However, ZFS does exist on FreeBSD, and I think it wouldn't be impossible to add similar features to ZFS. Another possibility would be to extend gjournal by adding time stamps to journal transactions and a possibility to feed the journal to a pipe, socket or whatever. And of course a client-side implementation that does something useful with the journal stream. This might even be a good SoC project. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Gesch?ftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M?n- chen, HRB 125758, Gesch?ftsf?hrer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "File names are infinite in length, where infinity is set to 255 characters." -- Peter Collinson, "The Unix File System" From ohartman at zedat.fu-berlin.de Wed Oct 8 08:18:35 2008 From: ohartman at zedat.fu-berlin.de (O. Hartmann) Date: Wed Oct 8 11:29:09 2008 Subject: HEADS UP: GCC 4.2.0 is coming In-Reply-To: <20070519022016.2b4a6bda@kan.dnsalias.net> References: <20070518192007.6e6a91e1@kan.dnsalias.net> <20070519022016.2b4a6bda@kan.dnsalias.net> Message-ID: <48EC67FF.3030900@zedat.fu-berlin.de> Alexander Kabaev wrote: > On Fri, 18 May 2007 19:20:07 -0400 > Alexander Kabaev wrote: > >> HEADS UP: I will start importing GCC 4.2.0 bits in about one hour and >> plan to finish in a couple of hours after that. >> >> The src/ tree will be utterly broken meanwhile. I'll send an 'all >> clear' message when done. > > Done. > Just for those who aren't on the cutting edge: why gcc 4.2.0 and not 4.2.1 as it is used in 7.X? Regards, O. From peter at wemm.org Wed Oct 8 09:01:57 2008 From: peter at wemm.org (Peter Wemm) Date: Wed Oct 8 11:29:23 2008 Subject: TIME WARP! Re: HEADS UP: GCC 4.2.0 is coming Message-ID: On Wed, Oct 8, 2008 at 12:57 AM, O. Hartmann wrote: > Alexander Kabaev wrote: >> >> On Fri, 18 May 2007 19:20:07 -0400 >> Alexander Kabaev wrote: >> >>> HEADS UP: I will start importing GCC 4.2.0 bits in about one hour and >>> plan to finish in a couple of hours after that. >>> >>> The src/ tree will be utterly broken meanwhile. I'll send an 'all >>> clear' message when done. >> >> Done. >> > > Just for those who aren't on the cutting edge: why gcc 4.2.0 and not 4.2.1 > as it is used in 7.X? > > Regards, > O. Sorry about that. I accidently revived a bunch of stuck email messages from our mailing list processing system. These messages from 2007 came back to life somehow. (Hint: Mailman's 'unshunt' command doesn't give a usage message) -- Peter Wemm - peter@wemm.org; peter@FreeBSD.org; peter@yahoo-inc.com; KI6FJV "All of this is for nothing if we don't go to the stars" - JMS/B5 "If Java had true garbage collection, most programs would delete themselves upon execution." -- Robert Sewell From vova at fbsd.ru Wed Oct 8 09:17:37 2008 From: vova at fbsd.ru (Vladimir Grebenschikov) Date: Wed Oct 8 11:29:34 2008 Subject: HEADS UP: GCC 4.2.0 is coming In-Reply-To: <48EC67FF.3030900@zedat.fu-berlin.de> References: <20070518192007.6e6a91e1@kan.dnsalias.net> <20070519022016.2b4a6bda@kan.dnsalias.net> <48EC67FF.3030900@zedat.fu-berlin.de> Message-ID: <1223455199.1864.16.camel@localhost> On Wed, 2008-10-08 at 07:57 +0000, O. Hartmann wrote: > Alexander Kabaev wrote: > > On Fri, 18 May 2007 19:20:07 -0400 -------^^^^^^^^^^^^^^^^^ > > Alexander Kabaev wrote: > > > >> HEADS UP: I will start importing GCC 4.2.0 bits in about one hour and > >> plan to finish in a couple of hours after that. > >> > >> The src/ tree will be utterly broken meanwhile. I'll send an 'all > >> clear' message when done. > > > > Done. > > > > Just for those who aren't on the cutting edge: why gcc 4.2.0 and not > 4.2.1 as it is used in 7.X? Looks like very outdated messages were delivered from spool. > Regards, > O. -- Vladimir B. Grebenschikov vova@fbsd.ru From c.kworr at gmail.com Wed Oct 8 13:50:04 2008 From: c.kworr at gmail.com (Volodymyr Kostyrko) Date: Wed Oct 8 13:50:11 2008 Subject: llvm/clang early test Message-ID: Hi all. Does anyone tried to build world with clang (devel/llvm-devel)? I just have tested clang on some code from our tree, gzip and bzip2 for example. Well... it works. Gzip compiled with clang become faster, bzip2 don't... Right now I'm playing with world making it compile with clang. If anyone doing the same thing we can share some thoughts and patches... -- Sphinx of black quartz judge my vow. From outbackdingo at gmail.com Wed Oct 8 13:59:39 2008 From: outbackdingo at gmail.com (Outback Dingo) Date: Wed Oct 8 13:59:46 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <200810081120.m98BK2fV043545@lurza.secnetix.de> References: <86iqs3sdtp.fsf@ds4.des.no> <200810081120.m98BK2fV043545@lurza.secnetix.de> Message-ID: <5635aa0d0810080631s41a43444hbfd1eed11f48f6c2@mail.gmail.com> one answer... www.bakbone.com On Wed, Oct 8, 2008 at 6:20 PM, Oliver Fromme wrote: > Dag-Erling Sm?rgrav wrote: > > "Zaphod Beeblebrox" writes: > > > "Dag-Erling Sm?rgrav" writes: > > > > What really annoys me with this thread is that nobody has provided > > > > any information at all that would allow someone to understand what > > > > needs to be done and estimate how hard it would be. > > > Well... I hinted that a hammer port would be sufficient (although they > > > need to finish their replication design) and I hinted that the hammer > > > approach may be graftable to ZFS. Both reasonably large effort-wise > > > (but probably within the scope of a single developer with sufficient > > > time). > > > > No... you're so far off the mark it's not even funny, especially when > > it's been repeatedly pointed out to you. This is not a file system, > > it's a backup system. It's not designed to survive a disk crash or an > > accidental file deletion, it's designed to survive a direct missile > > strike on your colo center. > > > > To quote Wikipedia, "CDP is a service that captures changes to data to a > > separate storage location" - emphasis on "separate". > > FWIW, the HAMMER file system _does_ support replication to > remote targets (thus "separate"). Unfortunately they call > this feature "mirroring", which is misleading at best. > It's really rather a replication mechanism, much like the > binlog of MySQL. It can be used for various purposes, > including live mirroring, delayed mirroring, archiving, > backup and point-in-time recovery. > > Well, of course, all of that doesn't help us at all because > HAMMER doesn't exist on FreeBSD. > > However, ZFS does exist on FreeBSD, and I think it wouldn't > be impossible to add similar features to ZFS. > > Another possibility would be to extend gjournal by adding > time stamps to journal transactions and a possibility to > feed the journal to a pipe, socket or whatever. And of > course a client-side implementation that does something > useful with the journal stream. This might even be a good > SoC project. > > Best regards > Oliver > > -- > Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. > Handelsregister: Registergericht Muenchen, HRA 74606, Gesch?ftsfuehrung: > secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M?n- > chen, HRB 125758, Gesch?ftsf?hrer: Maik Bachmann, Olaf Erb, Ralf Gebhart > > FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd > > "File names are infinite in length, where infinity is set to 255 > characters." > -- Peter Collinson, "The Unix File System" > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From rdivacky at freebsd.org Wed Oct 8 14:09:55 2008 From: rdivacky at freebsd.org (Roman Divacky) Date: Wed Oct 8 14:10:02 2008 Subject: llvm/clang early test In-Reply-To: References: Message-ID: <20081008135339.GA66730@freebsd.org> On Wed, Oct 08, 2008 at 04:29:46PM +0300, Volodymyr Kostyrko wrote: > Hi all. > > Does anyone tried to build world with clang (devel/llvm-devel)? I just > have tested clang on some code from our tree, gzip and bzip2 for > example. Well... it works. Gzip compiled with clang become faster, bzip2 > don't... Right now I'm playing with world making it compile with clang. > If anyone doing the same thing we can share some thoughts and patches... check this patch: www.vlakno.cz/~rdivacky/clang.patch I am playing with it as well :) From des at des.no Wed Oct 8 16:45:48 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed Oct 8 16:45:58 2008 Subject: What is the time between 2 mi_switches in freebsd. In-Reply-To: <53fa490b0810071947j23fc0f72n5360b6f174ddc96d@mail.gmail.com> (ushasri tummala's message of "Tue, 7 Oct 2008 22:47:16 -0400") References: <53fa490b0810071947j23fc0f72n5360b6f174ddc96d@mail.gmail.com> Message-ID: <86ljwzqblx.fsf@ds4.des.no> "ushasri tummala" writes: > What is the time between 2 mi_switches in freebsd? "it depends" I recommend _The Design and Implementation of the FreeBSD Operating Systems_ (http://www.amazon.com/dp/0201702452) or any good textbook on operating system design (e.g. http://www.amazon.com/dp/0136006639) DES -- Dag-Erling Sm?rgrav - des@des.no From tummala.ushasri at gmail.com Wed Oct 8 17:14:50 2008 From: tummala.ushasri at gmail.com (ushasri tummala) Date: Wed Oct 8 17:14:57 2008 Subject: What is the time between 2 mi_switches in freebsd. In-Reply-To: <86ljwzqblx.fsf@ds4.des.no> References: <53fa490b0810071947j23fc0f72n5360b6f174ddc96d@mail.gmail.com> <86ljwzqblx.fsf@ds4.des.no> Message-ID: <53fa490b0810081014k7c6c3ddfs8f43526da72d881@mail.gmail.com> Thank You Des for your reply. I am using Design and Implementation of the FreeBSD Operating Systems. I just want to know how its(time between 2 mi_switch()) calculated and in which variable is it stored in the code.(FreeBSD 5.2 release) This is not addressed in text book. Thank You, Usha. On Wed, Oct 8, 2008 at 12:45 PM, Dag-Erling Sm?rgrav wrote: > "ushasri tummala" writes: > > What is the time between 2 mi_switches in freebsd? > > "it depends" > > I recommend _The Design and Implementation of the FreeBSD Operating > Systems_ (http://www.amazon.com/dp/0201702452) or any good textbook on > operating system design (e.g. http://www.amazon.com/dp/0136006639) > > DES > -- > Dag-Erling Sm?rgrav - des@des.no > From zbeeble at gmail.com Wed Oct 8 17:15:26 2008 From: zbeeble at gmail.com (Zaphod Beeblebrox) Date: Wed Oct 8 17:15:33 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <86iqs3sdtp.fsf@ds4.des.no> References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <5f67a8c40810070937r5ba89773ncee407ace25fa0dd@mail.gmail.com> <86iqs3sdtp.fsf@ds4.des.no> Message-ID: <5f67a8c40810081015p2c14e38evbeed0a97242a7c4a@mail.gmail.com> On Wed, Oct 8, 2008 at 4:14 AM, Dag-Erling Sm?rgrav wrote: > "Zaphod Beeblebrox" writes: > > "Dag-Erling Sm?rgrav" writes: > > > What really annoys me with this thread is that nobody has provided > > > any information at all that would allow someone to understand what > > > needs to be done and estimate how hard it would be. > > Well... I hinted that a hammer port would be sufficient (although they > > need to finish their replication design) and I hinted that the hammer > > approach may be graftable to ZFS. Both reasonably large effort-wise > > (but probably within the scope of a single developer with sufficient > > time). > > No... you're so far off the mark it's not even funny, especially when > it's been repeatedly pointed out to you. This is not a file system, > it's a backup system. It's not designed to survive a disk crash or an > accidental file deletion, it's designed to survive a direct missile > strike on your colo center. > > To quote Wikipedia, "CDP is a service that captures changes to data to a > separate storage location" - emphasis on "separate". Wow... thanks for the flame, but there's no reason that the device that is receiving the hammer replication couldn't be on the other side of the globe and there's no reason it couldn't be considered a backup. Part of the advantage of the structure that allows you to efficiently select for new changes allows you to do the same kind of *backup* as they claim. From zbeeble at gmail.com Wed Oct 8 17:19:27 2008 From: zbeeble at gmail.com (Zaphod Beeblebrox) Date: Wed Oct 8 17:19:34 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <200810081120.m98BK2fV043545@lurza.secnetix.de> References: <86iqs3sdtp.fsf@ds4.des.no> <200810081120.m98BK2fV043545@lurza.secnetix.de> Message-ID: <5f67a8c40810081019w79e0bb42i49c4da623b6e08ab@mail.gmail.com> On Wed, Oct 8, 2008 at 7:20 AM, Oliver Fromme wrote: > Dag-Erling Sm?rgrav wrote: > > FWIW, the HAMMER file system _does_ support replication to > remote targets (thus "separate"). Unfortunately they call > this feature "mirroring", which is misleading at best. > It's really rather a replication mechanism, much like the > binlog of MySQL. It can be used for various purposes, > including live mirroring, delayed mirroring, archiving, > backup and point-in-time recovery [thank-you for repeating that, BTW] > However, ZFS does exist on FreeBSD, and I think it wouldn't > be impossible to add similar features to ZFS. Possibly even as a ZFS module? This might be something better addressed at the ZFS project level --- but the next question is: does FreeBSD support ZFS modules? > Another possibility would be to extend gjournal by adding > time stamps to journal transactions and a possibility to > feed the journal to a pipe, socket or whatever. And of > course a client-side implementation that does something > useful with the journal stream. This might even be a good > SoC project. Now this interests me. Firstly, I thought that gjournal might only be responsible for the meta-data (but I'm happy to be wrong on this point). Secondly, is it a) sufficient and b) efficient to attempt to time-travel UFS with the gjournal log? From asmodai at in-nomine.org Wed Oct 8 17:49:59 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Wed Oct 8 17:50:07 2008 Subject: What is the time between 2 mi_switches in freebsd. In-Reply-To: <53fa490b0810081014k7c6c3ddfs8f43526da72d881@mail.gmail.com> References: <53fa490b0810071947j23fc0f72n5360b6f174ddc96d@mail.gmail.com> <86ljwzqblx.fsf@ds4.des.no> <53fa490b0810081014k7c6c3ddfs8f43526da72d881@mail.gmail.com> Message-ID: <20081008174956.GA98121@nexus.in-nomine.org> -On [20081008 19:15], ushasri tummala (tummala.ushasri@gmail.com) wrote: >I just want to know how its(time between 2 mi_switch()) calculated and in >which variable is it stored in the code.(FreeBSD 5.2 release) >This is not addressed in text book. What Dag-Erling meant to say, and if I recall correctly, a switch() is highly dependent on your hardware. So the time taken for a specific machine can be vastly different from another machine. -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B Lead us to the place, guide us with your grace, to a place where we'll be safe... From yurtesen at ispro.net Wed Oct 8 17:31:28 2008 From: yurtesen at ispro.net (Evren Yurtesen) Date: Wed Oct 8 18:00:08 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <20081007155059.GA20615@charon.picobyte.net> References: <48E9E1BB.6020908@ispro.net> <48EA3451.7040801@miralink.com> <48EA83CE.4060702@ispro.net> <20081007155059.GA20615@charon.picobyte.net> Message-ID: <48ECE382.4060907@ispro.net> Shaun Amott wrote: > On Tue, Oct 07, 2008 at 12:31:58AM +0300, Evren Yurtesen wrote: >> so FreeBSD could be supported also. As you can imagine, it is not only >> important that data can be restored when a box hardware failure etc. it is >> also important that data can be restored if deleted by accidents etc. While >> traditional backup programs provide this functionality, you cant really go >> back to 10 min or 1h ago, often they take daily backups and have to scan >> whole filesystem for changed files every time the backup is taken which >> stresses out the systems. >> > > This can (more or less) be achieved with snapshots: you can cheaply > maintain old versions of the file system, and mount an old snapshot at > any time. Hourly is about as fine-grained as you can expect though. > The documentation says one cant do more than 20 snapshots. http://www.freebsd.org/doc/en/books/handbook/snapshots.html Although 20 could be enough combined with a normal backup program. As far as I understand creating snapshots will consume disk space, and freeze the disk writes for a certain amount of time, every time (I read 5 seconds for 8GB system http://www.wave2.org/2007/10/08/mysql-snapshots-on-freebsd/ ) snapshot is created. The snapshots are stored in the local filesystem and it would require manually transferring the data to a remote machine. More importantly, as far as I understand, if the hard drive totally fails, there would be no way to restore a snapshot anymore unless we have a dump of the whole filesystem and first restore it and make sure everything is exactly at the right blocks in the drive. No? Although this probably could be worked out. In my opinion it requires a lot of work, Bbt thanks for the advice. Just that I would rather pay a small amount of fee and use Linux and use a continous backup software which works as easy as install and run. Which also provides utilities for easily recovering files or the whole filesystem or disk. Thanks again for pointing out snapshots. It is more or less suitable :) Thanks, Evren From yurtesen at ispro.net Wed Oct 8 17:43:29 2008 From: yurtesen at ispro.net (Evren Yurtesen) Date: Wed Oct 8 18:05:50 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <5635aa0d0810080631s41a43444hbfd1eed11f48f6c2@mail.gmail.com> References: <86iqs3sdtp.fsf@ds4.des.no> <200810081120.m98BK2fV043545@lurza.secnetix.de> <5635aa0d0810080631s41a43444hbfd1eed11f48f6c2@mail.gmail.com> Message-ID: <48ECF144.305@ispro.net> Outback Dingo wrote: > one answer... www.bakbone.com > Unfortunately if you check their compatibility matrix you can see that I have to use Linux to be able to do CDP :) http://www.bakbone.com/docs/NetVault_Backup_Supported_Platforms_October_2008.pdf or am I reading this wrong? Thanks, Evren From yurtesen at ispro.net Wed Oct 8 18:01:05 2008 From: yurtesen at ispro.net (Evren Yurtesen) Date: Wed Oct 8 18:06:25 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <5f67a8c40810081015p2c14e38evbeed0a97242a7c4a@mail.gmail.com> References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <5f67a8c40810070937r5ba89773ncee407ace25fa0dd@mail.gmail.com> <86iqs3sdtp.fsf@ds4.des.no> <5f67a8c40810081015p2c14e38evbeed0a97242a7c4a@mail.gmail.com> Message-ID: <48ECF564.7000204@ispro.net> Zaphod Beeblebrox wrote: > On Wed, Oct 8, 2008 at 4:14 AM, Dag-Erling Sm?rgrav > wrote: > > "Zaphod Beeblebrox" > > writes: > > "Dag-Erling Sm?rgrav" > writes: > > > What really annoys me with this thread is that nobody has provided > > > any information at all that would allow someone to understand what > > > needs to be done and estimate how hard it would be. > > Well... I hinted that a hammer port would be sufficient (although > they > > need to finish their replication design) and I hinted that the hammer > > approach may be graftable to ZFS. Both reasonably large effort-wise > > (but probably within the scope of a single developer with sufficient > > time). > > No... you're so far off the mark it's not even funny, especially when > it's been repeatedly pointed out to you. This is not a file system, > it's a backup system. It's not designed to survive a disk crash or an > accidental file deletion, it's designed to survive a direct missile > strike on your colo center. > > To quote Wikipedia, "CDP is a service that captures changes to data to a > separate storage location" - emphasis on "separate". > > > Wow... thanks for the flame, but there's no reason that the device that > is receiving the hammer replication couldn't be on the other side of the > globe and there's no reason it couldn't be considered a backup. Part of > the advantage of the structure that allows you to efficiently select for > new changes allows you to do the same kind of *backup* as they claim. > Wouldnt that device need to keep the whole filesystem? Like if you have 10 machines with 10x 1GB drives (lets say each used about 250gb), you will need 10TB disk space in the backup server? From neldredge at math.ucsd.edu Wed Oct 8 18:34:14 2008 From: neldredge at math.ucsd.edu (Nate Eldredge) Date: Wed Oct 8 18:34:20 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <48ECE382.4060907@ispro.net> References: <48E9E1BB.6020908@ispro.net> <48EA3451.7040801@miralink.com> <48EA83CE.4060702@ispro.net> <20081007155059.GA20615@charon.picobyte.net> <48ECE382.4060907@ispro.net> Message-ID: On Wed, 8 Oct 2008, Evren Yurtesen wrote: > Thanks again for pointing out snapshots. It is more or less suitable :) I'll just warn you that if you're planning to use snapshots for your own purposes, to first do an extensive stress test on a non-critical machine with backed up data. I've had a lot of problems with snapshots occasionally causing deadlocks which hang the machine. This was under 6.x but I had the same problem under many previous versions, so I don't necessarily expect that it's fixed. Also, while it's never happened to me, I've heard other people report data corruption. -- Nate Eldredge neldredge@math.ucsd.edu From bakul at bitblocks.com Wed Oct 8 19:03:55 2008 From: bakul at bitblocks.com (Bakul Shah) Date: Wed Oct 8 19:04:01 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: Your message of "Wed, 08 Oct 2008 10:18:47 +0200." <86ej2rsdnc.fsf@ds4.des.no> References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <20081007164015.2E3085B29@mail.bitblocks.com> <86ej2rsdnc.fsf@ds4.des.no> Message-ID: <20081008190354.264C25B4C@mail.bitblocks.com> On Wed, 08 Oct 2008 10:18:47 +0200 =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= wrote: > Bakul Shah writes: > > "Dag-Erling Sm=C3=B8rgrav" writes: > > > What really annoys me with this thread is that nobody has provided any > > > information at all that would allow someone to understand what needs to > > > be done and estimate how hard it would be. > > From their http://forum.r1soft.com/CDP.html page: [...] > > You completely missed the mark. I know what R1Soft's product is. What > I want to know is what needs to be done to port it to FreeBSD. Sorry, my mindreader is broken. If this is what you really wanted to know, why not ask R1Soft? -hackers is not going to shed any light on the specifics of R1Soft's product. I replied to say that implementing a similar solution is not hard (I am sure you knew that too but I wasn't responding just to you). It may even be worth doing. From jhb at freebsd.org Wed Oct 8 19:11:11 2008 From: jhb at freebsd.org (John Baldwin) Date: Wed Oct 8 19:11:25 2008 Subject: What is the time between 2 mi_switches in freebsd. In-Reply-To: <53fa490b0810071947j23fc0f72n5360b6f174ddc96d@mail.gmail.com> References: <53fa490b0810071947j23fc0f72n5360b6f174ddc96d@mail.gmail.com> Message-ID: <200810081406.14916.jhb@freebsd.org> On Tuesday 07 October 2008 10:47:16 pm ushasri tummala wrote: > What is the time between 2 mi_switches in freebsd?In which variable is this > information stored? > > Could you plz help me out with this There isn't a fixed timeslice due to preemption for interrupts, etc. However, the default time slice is available as the kern.sched.quantum sysctl: % sysctl -d kern.sched.quantum kern.sched.quantum: Roundrobin scheduling quantum in microseconds In the kernel it is stored as a static (private) variable in the scheduler implementation. -- John Baldwin From zbeeble at gmail.com Wed Oct 8 19:35:14 2008 From: zbeeble at gmail.com (Zaphod Beeblebrox) Date: Wed Oct 8 19:35:22 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <48ECF564.7000204@ispro.net> References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <5f67a8c40810070937r5ba89773ncee407ace25fa0dd@mail.gmail.com> <86iqs3sdtp.fsf@ds4.des.no> <5f67a8c40810081015p2c14e38evbeed0a97242a7c4a@mail.gmail.com> <48ECF564.7000204@ispro.net> Message-ID: <5f67a8c40810081235k227dc870tce5fcbcbca61d3c1@mail.gmail.com> On Wed, Oct 8, 2008 at 2:01 PM, Evren Yurtesen wrote: > Zaphod Beeblebrox wrote: > >> >> Wow... thanks for the flame, but there's no reason that the device that >> is receiving the hammer replication couldn't be on the other side of the >> globe and there's no reason it couldn't be considered a backup. Part of the >> advantage of the structure that allows you to efficiently select for new >> changes allows you to do the same kind of *backup* as they claim. >> >> > Wouldnt that device need to keep the whole filesystem? Like if you have 10 > machines with 10x 1GB drives (lets say each used about 250gb), you will need > 10TB disk space in the backup server? > Urm... I think everything we've been discussing here backs up the whole filesystem (it would be near impossible for a block-oriented system to do elsewise). I suppose you could do something with the archive bit or dump bits with a filesystem based backup. But anyways... in a filesystem based replication system, you'd need enough space to store the data and the history of the data. The sum of the history of the data could even exceed the size of the sum of the input disks. It could also be much smaller. It really depends on how much you change. From des at des.no Wed Oct 8 19:46:14 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed Oct 8 19:46:22 2008 Subject: What is the time between 2 mi_switches in freebsd. In-Reply-To: <20081008174956.GA98121@nexus.in-nomine.org> (Jeroen Ruigrok van der Werven's message of "Wed, 8 Oct 2008 19:49:56 +0200") References: <53fa490b0810071947j23fc0f72n5360b6f174ddc96d@mail.gmail.com> <86ljwzqblx.fsf@ds4.des.no> <53fa490b0810081014k7c6c3ddfs8f43526da72d881@mail.gmail.com> <20081008174956.GA98121@nexus.in-nomine.org> Message-ID: <86zlleq397.fsf@ds4.des.no> Jeroen Ruigrok van der Werven writes: > -On [20081008 19:15], ushasri tummala (tummala.ushasri@gmail.com) wrote: > > I just want to know how its(time between 2 mi_switch()) calculated > > and in which variable is it stored in the code.(FreeBSD 5.2 release) > > This is not addressed in text book. > What Dag-Erling meant to say, and if I recall correctly, a switch() is > highly dependent on your hardware. So the time taken for a specific > machine can be vastly different from another machine. No, no, no. Assuming the question is really "what is the time between two task switches", A task switch can happen for one of many reasons: - first, and simplest, the current task has used up its quantum; - the current task is waiting for an external event (I/O, a mutex, a timeout, etc.) - the current task has terminated; - something happened to make a higher-priority task runnable; - ... The closest you can get to a hard answer is if you consider only the first of the above, in which case the answer is 1/hz second, where "hz" is literally a kernel variable named hz. Its default value is 1,000 on amd64, i386, ia64 and sparc64, and 100 on all other platforms. (actually, it's more complicated than that, because this default value is a preprocessor macro called HZ which you can redefine in your kernel config, and you can also set hz at boot time using the kern.hz loader tunable) A nice little project for someone with a lot of time on their hands would be to make the FreeBSD kernel tickless, thus (to oversimplify) eliminating hz. DES -- Dag-Erling Sm?rgrav - des@des.no From des at des.no Wed Oct 8 19:47:41 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed Oct 8 19:47:47 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <48ECE92D.7050904@ispro.net> (Evren Yurtesen's message of "Wed, 08 Oct 2008 20:09:01 +0300") References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <20081007164015.2E3085B29@mail.bitblocks.com> <86ej2rsdnc.fsf@ds4.des.no> <48ECE92D.7050904@ispro.net> Message-ID: <86vdw2q36s.fsf@ds4.des.no> Evren Yurtesen writes: > It would perhaps be too much to ask but can you drop a line to them > and ask or is it ok if I give your email address to them and they can > contact you? Feel free. DES -- Dag-Erling Sm?rgrav - des@des.no From des at des.no Wed Oct 8 20:03:39 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed Oct 8 20:03:46 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <48ECF564.7000204@ispro.net> (Evren Yurtesen's message of "Wed, 08 Oct 2008 21:01:08 +0300") References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <5f67a8c40810070937r5ba89773ncee407ace25fa0dd@mail.gmail.com> <86iqs3sdtp.fsf@ds4.des.no> <5f67a8c40810081015p2c14e38evbeed0a97242a7c4a@mail.gmail.com> <48ECF564.7000204@ispro.net> Message-ID: <86r66qq2g5.fsf@ds4.des.no> Evren Yurtesen writes: > Wouldnt that device need to keep the whole filesystem? Like if you > have 10 machines with 10x 1GB drives (lets say each used about 250gb), > you will need 10TB disk space in the backup server? Yes and no and yes... It stores *changes* to the file systems, so how much space it needs depends on how full your file systems are, how often and how much you write to them, etc. Also, at every recovery point, you can discard all but the last change since the previous recovery point for every changed block. FWIW, the exact same answer applies to pretty much any backup solution that supports incremental backups. DES -- Dag-Erling Sm?rgrav - des@des.no From des at des.no Wed Oct 8 20:19:50 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed Oct 8 20:19:56 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <20081008190354.264C25B4C@mail.bitblocks.com> (Bakul Shah's message of "Wed, 08 Oct 2008 12:03:54 -0700") References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <20081007164015.2E3085B29@mail.bitblocks.com> <86ej2rsdnc.fsf@ds4.des.no> <20081008190354.264C25B4C@mail.bitblocks.com> Message-ID: <86prma3km3.fsf@ds4.des.no> Bakul Shah writes: > "Dag-Erling Sm?rgrav" writes: > > Bakul Shah writes: > > > "Dag-Erling Sm?rgrav" writes: > > > > What really annoys me with this thread is that nobody has > > > > provided any information at all that would allow someone to > > > > understand what needs to be done and estimate how hard it would > > > > be. > > > From their http://forum.r1soft.com/CDP.html page: [...] > > You completely missed the mark. I know what R1Soft's product is. > > What I want to know is what needs to be done to port it to FreeBSD. > Sorry, my mindreader is broken. If this is what you really wanted to > know, why not ask R1Soft? -hackers is not going to shed any light on > the specifics of R1Soft's product. I replied to say that implementing > a similar solution is not hard (I am sure you knew that too but I > wasn't responding just to you). It may even be worth doing. I didn't actually ask a question, and I don't mind that you don't have the answer. What I do mind is that you interpreted my statement of frustration as a question, and provided a completely irrelevant answer. You don't need to read minds to understand this, just English. If you take a step back and go through and read the entire thread again from the start, though, I think you will understand my frustration. Evren asked a question which everybody else is doing their best not to answer in as many words as possible; and when I try to answer, I find out that Evren doesn't really know what the question is. This is where - in an ideal world - somebody at R1Soft would jump in and start asking the right questions... DES -- Dag-Erling Sm?rgrav - des@des.no From ivoras at freebsd.org Wed Oct 8 20:20:19 2008 From: ivoras at freebsd.org (Ivan Voras) Date: Wed Oct 8 20:20:26 2008 Subject: strftime's %c warning? Message-ID: I'm trying to use the %c formatter in strftime(3), documented as: " %c is replaced by national representation of time and date. " ... which looks useful, except that in code in which WFORMAT is defined as "1" I get this error: str.c: In function 'ltime': str.c:141: warning: '%c' yields only last 2 digits of year in some locales on non-BSD systems *** Error code 1 Since the code I'm developing is definitely BSD-only (patch to pkg_* infrastructure), should I: a) stop using locale-based %c and choose my own date/time format b) remove WFORMAT from the Makefile? The same warning/error is generated by %x and %X, and %+ described in the strftime man page isn't recognized. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 258 bytes Desc: OpenPGP digital signature Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081008/da1c7a32/signature.pgp From des at des.no Wed Oct 8 20:20:54 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed Oct 8 20:21:00 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <5f67a8c40810081019w79e0bb42i49c4da623b6e08ab@mail.gmail.com> (Zaphod Beeblebrox's message of "Wed, 8 Oct 2008 13:19:25 -0400") References: <86iqs3sdtp.fsf@ds4.des.no> <200810081120.m98BK2fV043545@lurza.secnetix.de> <5f67a8c40810081019w79e0bb42i49c4da623b6e08ab@mail.gmail.com> Message-ID: <86ljwy3kka.fsf@ds4.des.no> "Zaphod Beeblebrox" writes: >> Dag-Erling Sm?rgrav wrote: >> >> FWIW, the HAMMER file system _does_ support replication [...] No, actually, I didn't write that. You need to learn to quote properly. DES -- Dag-Erling Sm?rgrav - des@des.no From des at des.no Wed Oct 8 20:23:45 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed Oct 8 20:23:52 2008 Subject: What is the time between 2 mi_switches in freebsd. In-Reply-To: <200810081406.14916.jhb@freebsd.org> (John Baldwin's message of "Wed, 8 Oct 2008 14:06:14 -0400") References: <53fa490b0810071947j23fc0f72n5360b6f174ddc96d@mail.gmail.com> <200810081406.14916.jhb@freebsd.org> Message-ID: <86hc7m3kfk.fsf@ds4.des.no> John Baldwin writes: > There isn't a fixed timeslice due to preemption for interrupts, etc. However, > the default time slice is available as the kern.sched.quantum sysctl: > > % sysctl -d kern.sched.quantum > kern.sched.quantum: Roundrobin scheduling quantum in microseconds des@ds4 ~% uname -a FreeBSD ds4.des.no 8.0-CURRENT FreeBSD 8.0-CURRENT #48 r183493M: Tue Sep 30 23:35:48 CEST 2008 des@ds4.des.no:/usr/obj/usr/src/sys/ds4 amd64 des@ds4 ~% sysctl kern.sched.quantum sysctl: unknown oid 'kern.sched.quantum' DES -- Dag-Erling Sm?rgrav - des@des.no From jhb at freebsd.org Wed Oct 8 21:02:23 2008 From: jhb at freebsd.org (John Baldwin) Date: Wed Oct 8 21:02:31 2008 Subject: What is the time between 2 mi_switches in freebsd. In-Reply-To: <86hc7m3kfk.fsf@ds4.des.no> References: <53fa490b0810071947j23fc0f72n5360b6f174ddc96d@mail.gmail.com> <200810081406.14916.jhb@freebsd.org> <86hc7m3kfk.fsf@ds4.des.no> Message-ID: <200810081701.45199.jhb@freebsd.org> On Wednesday 08 October 2008 04:23:43 pm Dag-Erling Sm?rgrav wrote: > John Baldwin writes: > > There isn't a fixed timeslice due to preemption for interrupts, etc. However, > > the default time slice is available as the kern.sched.quantum sysctl: > > > > % sysctl -d kern.sched.quantum > > kern.sched.quantum: Roundrobin scheduling quantum in microseconds > > des@ds4 ~% uname -a > FreeBSD ds4.des.no 8.0-CURRENT FreeBSD 8.0-CURRENT #48 r183493M: Tue Sep 30 23:35:48 CEST 2008 des@ds4.des.no:/usr/obj/usr/src/sys/ds4 amd64 > des@ds4 ~% sysctl kern.sched.quantum > sysctl: unknown oid 'kern.sched.quantum' Apparently the quantum isn't exposed anywhere for ULE. -- John Baldwin From bakul at bitblocks.com Wed Oct 8 21:30:55 2008 From: bakul at bitblocks.com (Bakul Shah) Date: Wed Oct 8 21:31:02 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: Your message of "Wed, 08 Oct 2008 22:19:48 +0200." <86prma3km3.fsf@ds4.des.no> References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <20081007164015.2E3085B29@mail.bitblocks.com> <86ej2rsdnc.fsf@ds4.des.no> <20081008190354.264C25B4C@mail.bitblocks.com> <86prma3km3.fsf@ds4.des.no> Message-ID: <20081008213054.5AF645B4C@mail.bitblocks.com> On Wed, 08 Oct 2008 22:19:48 +0200 =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= wrote: > Bakul Shah writes: > > "Dag-Erling Sm=C3=B8rgrav" writes: > > > Bakul Shah writes: > > > > "Dag-Erling Sm=C3=B8rgrav" writes: > > > > > What really annoys me with this thread is that nobody has > > > > > provided any information at all that would allow someone to > > > > > understand what needs to be done and estimate how hard it would > > > > > be. > > > > From their http://forum.r1soft.com/CDP.html page: [...] > > > You completely missed the mark. I know what R1Soft's product is. > > > What I want to know is what needs to be done to port it to FreeBSD. > > Sorry, my mindreader is broken. If this is what you really wanted to > > know, why not ask R1Soft? -hackers is not going to shed any light on > > the specifics of R1Soft's product. I replied to say that implementing > > a similar solution is not hard (I am sure you knew that too but I > > wasn't responding just to you). It may even be worth doing. > > I didn't actually ask a question, and I don't mind that you don't have > the answer. What I do mind is that you interpreted my statement of > frustration as a question, and provided a completely irrelevant answer. > You don't need to read minds to understand this, just English. Interpreting an expression of frustration as a request for a solution is a common engineering trait:-) I can see you may not prefer my interpretation but I can't understand why you "mind" it. But so be it. I do not wish to annoy you. > If you take a step back and go through and read the entire thread again > from the start, though, I think you will understand my frustration. I understand your frustration but I chose to instead focus on the technical part. I too can get frustrated in similar situations but every time that happens I can trace it back to my own stress. I can't really control what others say so the way I deal with it is to ignore it or joke about it. I do try to clear misunderstandings but people don't always understand my point of view! As for feeling frustrated, I now view that as a warning signal. > Evren asked a question which everybody else is doing their best not to > answer in as many words as possible; and when I try to answer, I find > out that Evren doesn't really know what the question is. Actually that was clear in his very first email. > This is where - in an ideal world - somebody at R1Soft would jump in and > start asking the right questions... Don't bet on it. My musings even makes me wonder if they do it right. From bakul at bitblocks.com Wed Oct 8 21:33:22 2008 From: bakul at bitblocks.com (Bakul Shah) Date: Wed Oct 8 21:33:29 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: Your message of "Wed, 08 Oct 2008 14:30:54 PDT." Message-ID: <20081008213321.AEC065B4C@mail.bitblocks.com> Sorry about that. Didn't mean to continue this discussion in -hackers but forgot to remove the cc list. From yurtesen at ispro.net Wed Oct 8 23:28:20 2008 From: yurtesen at ispro.net (yurtesen@ispro.net) Date: Thu Oct 9 00:00:03 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <5f67a8c40810081235k227dc870tce5fcbcbca61d3c1@mail.gmail.com> References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <5f67a8c40810070937r5ba89773ncee407ace25fa0dd@mail.gmail.com> <86iqs3sdtp.fsf@ds4.des.no> <5f67a8c40810081015p2c14e38evbeed0a97242a7c4a@mail.gmail.com> <48ECF564.7000204@ispro.net> <5f67a8c40810081235k227dc870tce5fcbcbca61d3c1@mail.gmail.com> Message-ID: <20081009022833.73943nda9muz0qo0@mail.ispro.net> Quoting "Zaphod Beeblebrox" : > On Wed, Oct 8, 2008 at 2:01 PM, Evren Yurtesen wrote: > >> Zaphod Beeblebrox wrote: >> >>> >>> Wow... thanks for the flame, but there's no reason that the device that >>> is receiving the hammer replication couldn't be on the other side of the >>> globe and there's no reason it couldn't be considered a backup. >>> Part of the >>> advantage of the structure that allows you to efficiently select for new >>> changes allows you to do the same kind of *backup* as they claim. >>> >>> >> Wouldnt that device need to keep the whole filesystem? Like if you have 10 >> machines with 10x 1GB drives (lets say each used about 250gb), you will need >> 10TB disk space in the backup server? >> > > > Urm... I think everything we've been discussing here backs up the whole > filesystem (it would be near impossible for a block-oriented system to do > elsewise). I suppose you could do something with the archive bit or dump > bits with a filesystem based backup. > > But anyways... in a filesystem based replication system, you'd need enough > space to store the data and the history of the data. The sum of the history > of the data could even exceed the size of the sum of the input disks. It > could also be much smaller. It really depends on how much you change. > The CDP backup solutions can manage that, even better you dont have to change to a totally new filesystem to use them. I really do not want to have a large argument about this but the only reason we cant do this on FreeBSD is because the companies who write CDP type backup solutions either do not know much about FreeBSD or they just cant find anybody who they can hire to do the job on FreeBSD. So that is why I posted this information to this list. If there is anybody who is capable of porting such software to FreeBSD then they can, 1- probably make money by doing the job, 2- if they have very little free time and not interested in making money then they can perhaps contact some of these companies (for example r1soft seems to be interested in supporting FreeBSD) and give them some hints and ideas on how this can be done on FreeBSD and where to find example codes and more information etc. Thanks, Evren From yurtesen at ispro.net Thu Oct 9 00:16:37 2008 From: yurtesen at ispro.net (yurtesen@ispro.net) Date: Thu Oct 9 00:23:57 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <86prma3km3.fsf@ds4.des.no> References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <20081007164015.2E3085B29@mail.bitblocks.com> <86ej2rsdnc.fsf@ds4.des.no> <20081008190354.264C25B4C@mail.bitblocks.com> <86prma3km3.fsf@ds4.des.no> Message-ID: <20081009031653.19603w0x984w8l6o@mail.ispro.net> Quoting "Dag-Erling Sm?rgrav" : > If you take a step back and go through and read the entire thread again > from the start, though, I think you will understand my frustration. > Evren asked a question which everybody else is doing their best not to > answer in as many words as possible; and when I try to answer, I find > out that Evren doesn't really know what the question is. > > This is where - in an ideal world - somebody at R1Soft would jump in and > start asking the right questions... Actually, I was expecting that somebody would contact to R1Soft and offer them help for $$$. But I have given your contact information to them now. So they can forward their questions to you and probably you can let them know who knows what and who they can contact with more information or ask $$$ from them to do the job :) Thanks, Evren From scf at FreeBSD.org Thu Oct 9 03:22:22 2008 From: scf at FreeBSD.org (Sean C. Farley) Date: Thu Oct 9 03:22:28 2008 Subject: strftime's %c warning? In-Reply-To: References: Message-ID: On Wed, 8 Oct 2008, Ivan Voras wrote: > I'm trying to use the %c formatter in strftime(3), documented as: > > " > %c is replaced by national representation of time and date. > " > > ... which looks useful, except that in code in which WFORMAT is defined > as "1" I get this error: > > str.c: In function 'ltime': > str.c:141: warning: '%c' yields only last 2 digits of year in some > locales on non-BSD systems > *** Error code 1 > > Since the code I'm developing is definitely BSD-only (patch to pkg_* > infrastructure), should I: > > a) stop using locale-based %c and choose my own date/time format > b) remove WFORMAT from the Makefile? > > The same warning/error is generated by %x and %X, and %+ described in > the strftime man page isn't recognized. You are hitting a gcc builtin. Have you tried adding -fno-builtin-strftime? Sean -- scf@FreeBSD.org From brucem at mail.cruzio.com Thu Oct 9 03:32:25 2008 From: brucem at mail.cruzio.com (Bruce R. Montague) Date: Thu Oct 9 03:32:32 2008 Subject: continuous backup solution for FreeBSD Message-ID: <200810080444.m984ireq000400@mail.cruzio.com> Hi, re Continuous Data Protection (an enterprise market niche, today), a recent paper that provides some background and has basic references might be: http://www.usenix.org/event/fast08/tech/verma.html "SWEEPER: An Efficient Disaster Recovery Point Identification Mechanism", FAST '08, Akshat Verma, IBM India Research; Kaladhar Voruganti, Network Appliance; Ramani Routray, IBM Almaden Research; Rohit Jain, Yahoo India. -bruce From ivoras at freebsd.org Thu Oct 9 07:17:07 2008 From: ivoras at freebsd.org (Ivan Voras) Date: Thu Oct 9 07:17:15 2008 Subject: strftime's %c warning? In-Reply-To: References: Message-ID: <9bbcef730810082349t733a0295hf9227d35534fdfcb@mail.gmail.com> 2008/10/9 Sean C. Farley : > On Wed, 8 Oct 2008, Ivan Voras wrote: > >> str.c: In function 'ltime': >> str.c:141: warning: '%c' yields only last 2 digits of year in some >> locales on non-BSD systems >> *** Error code 1 >> >> Since the code I'm developing is definitely BSD-only (patch to pkg_* >> infrastructure), should I: >> >> a) stop using locale-based %c and choose my own date/time format >> b) remove WFORMAT from the Makefile? >> >> The same warning/error is generated by %x and %X, and %+ described in >> the strftime man page isn't recognized. > > You are hitting a gcc builtin. Have you tried adding > -fno-builtin-strftime? The code im working on is part of FreeBSD and uses FreeBSD's infrastructure for building. I can avoid the warning by defining WFORMAT to be 0 but I don't know how to add -fno-builtin-strftime except directly into CFLAGS. I don't think the original author (John Hubbard) is around anymore so the question is - should I just remove WFPORMAT from this Makefile? From marcolz at stack.nl Thu Oct 9 09:16:51 2008 From: marcolz at stack.nl (Marc Olzheim) Date: Thu Oct 9 09:17:02 2008 Subject: strftime's %c warning? In-Reply-To: References: Message-ID: <20081009085409.GA11669@zlo.nu> On Wed, Oct 08, 2008 at 10:20:00PM +0200, Ivan Voras wrote: > I'm trying to use the %c formatter in strftime(3), documented as: > > " > %c is replaced by national representation of time and date. > " > > ... which looks useful, except that in code in which WFORMAT is defined > as "1" I get this error: > > str.c: In function 'ltime': > str.c:141: warning: '%c' yields only last 2 digits of year in some > locales on non-BSD systems > *** Error code 1 > > Since the code I'm developing is definitely BSD-only (patch to pkg_* > infrastructure), should I: > > a) stop using locale-based %c and choose my own date/time format > b) remove WFORMAT from the Makefile? > > The same warning/error is generated by %x and %X, and %+ described in > the strftime man page isn't recognized. Yes. In FreeBSD 4.x, gcc was patched not to warn about these, so compiling with -Werror was possible even when using this. Since FreeBSD 5.x, this patch wasn't applied anymore. If you only want it to compile on your own system with -Werror, you can use this patch (and probably adapt it to %c, %x and %X as well): http://www.stack.nl/~marcolz/FreeBSD/gcc-strftime-format-freebsd7.patch.txt Marc -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081009/3458e1c9/attachment.pgp From sigtrm at gmail.com Thu Oct 9 12:29:43 2008 From: sigtrm at gmail.com (Lukasz Jaroszewski) Date: Thu Oct 9 12:29:50 2008 Subject: Sockstress Message-ID: Hi, I am wondering about sockstres informations recently published. I cant really figure what new they could found. Do we have anything to worry about? ;-) http://searchsecurity.techtarget.com/news/article/0,289142,sid14_gci1332898,00.html ``(...)Sockstress computes and stores so-called client-side SYN cookies and enables Lee and Louis to specify a destination port and IP address. The method allows them to complete the TCP handshake without having to store any values, which takes time and resources. "We can then say that we want to establish X number of TCP connections on that address and that we want to use this attack type, and it does it," Lee said.(...)'' ``(...)Lee said that when and _if_ specific vendors develop workarounds for the issues, they will release details of those issues.(...)'' Was FreeBSD team contacted? ;) -- Regards/Pozdrawiam LVJ -------------------------------------------------------------------------------------------------- They must find it difficult, those that take authority as truth, instead of truth as the authority From olli at lurza.secnetix.de Thu Oct 9 13:38:35 2008 From: olli at lurza.secnetix.de (Oliver Fromme) Date: Thu Oct 9 13:38:49 2008 Subject: Sockstress In-Reply-To: Message-ID: <200810091338.m99DcW3a006320@lurza.secnetix.de> This is the wrong mailing list, you should send this to the -security list. By the way, this kind of attack isn't really new (as far as I can tell from the few information that have been made public so far). One way to mitigate it is to limit the number of open connections per remote IP address; you can easily do that with PF or IPFW ("limit" option). Best regards Oliver Lukasz Jaroszewski wrote: > Hi, > I am wondering about sockstres informations recently published. I cant > really figure what new they could found. Do we have anything to worry about? > ;-) > > http://searchsecurity.techtarget.com/news/article/0,289142,sid14_gci1332898,00.html > > ``(...)Sockstress computes and stores so-called client-side SYN cookies and > enables Lee and Louis to specify a destination port and IP address. The > method allows them to complete the TCP handshake without having to store any > values, which takes time and resources. "We can then say that we want to > establish X number of TCP connections on that address and that we want to > use this attack type, and it does it," Lee said.(...)'' > > ``(...)Lee said that when and _if_ specific vendors develop workarounds for > the issues, they will release details of those issues.(...)'' > > Was FreeBSD team contacted? ;) > -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Gesch?ftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M?n- chen, HRB 125758, Gesch?ftsf?hrer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "Unix gives you just enough rope to hang yourself -- and then a couple of more feet, just to be sure." -- Eric Allman From olli at lurza.secnetix.de Thu Oct 9 14:11:05 2008 From: olli at lurza.secnetix.de (Oliver Fromme) Date: Thu Oct 9 14:11:19 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <48ECE382.4060907@ispro.net> Message-ID: <200810091411.m99EB0Vo007538@lurza.secnetix.de> Evren Yurtesen wrote: > Shaun Amott wrote: > > On Tue, Oct 07, 2008 at 12:31:58AM +0300, Evren Yurtesen wrote: > > > so FreeBSD could be supported also. As you can imagine, it is not only > > > important that data can be restored when a box hardware failure etc. it is > > > also important that data can be restored if deleted by accidents etc. While > > > traditional backup programs provide this functionality, you cant really go > > > back to 10 min or 1h ago, often they take daily backups and have to scan > > > whole filesystem for changed files every time the backup is taken which > > > stresses out the systems. > > > > This can (more or less) be achieved with snapshots: you can cheaply > > maintain old versions of the file system, and mount an old snapshot at > > any time. Hourly is about as fine-grained as you can expect though. > > The documentation says one cant do more than 20 snapshots. > http://www.freebsd.org/doc/en/books/handbook/snapshots.html I wouldn't use UFS snapshots for this purpose. They have a few well-known disadvantages. However, ZFS snapshots should work very well for this. They're not limited to 20, and you can create them very quickly and with low overhead. You could create a new snapshot every 5 minutes if you want. Then you can use the "zfs send" command to produce a stream that contains the incremental differences between the snapshot and the previous snapshot, i.e. the stream represents the changes to the filesystem within the past five minutes (or whatever snapshot interval you choose). You can store that stream in a file, on backup medium, or send it with ssh to a different continent. Every once in a while you should make a full backup from a snapshot, of course. Basically it works like any other incremental backup mechanism, except that you can make the time interval between incremental backups very small (like five minutes in the above example), so you get a nearly continous backup solution. By the way, if you accidentally deleted something, you can of course simply copy it back from a snapshot. You can even revert the whole file system to a previous snapshot using the "zfs rollback" command. This is like going back in time. There is no need to touch your backups for that. These features are readily available right now on FreeBSD. You don't have to code anything. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Gesch?ftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M?n- chen, HRB 125758, Gesch?ftsf?hrer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things." -- Doug Gwyn From olli at lurza.secnetix.de Thu Oct 9 14:21:16 2008 From: olli at lurza.secnetix.de (Oliver Fromme) Date: Thu Oct 9 14:21:22 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <5f67a8c40810081019w79e0bb42i49c4da623b6e08ab@mail.gmail.com> Message-ID: <200810091421.m99ELEcm007901@lurza.secnetix.de> Zaphod Beeblebrox wrote: > Oliver Fromme wrote: > > However, ZFS does exist on FreeBSD, and I think it wouldn't > > be impossible to add similar features to ZFS. > > Possibly even as a ZFS module? This might be something better addressed at > the ZFS project level --- but the next question is: does FreeBSD support ZFS > modules? I'm sorry I don't know. But also see my other reply regarding ZFS snapshots ans "zfs send". > > Another possibility would be to extend gjournal by adding > > time stamps to journal transactions and a possibility to > > feed the journal to a pipe, socket or whatever. And of > > course a client-side implementation that does something > > useful with the journal stream. This might even be a good > > SoC project. > > Now this interests me. Firstly, I thought that gjournal might only be > responsible for the meta-data (but I'm happy to be wrong on this point). Nope, gjournal handles _all_ data, i.e. meta data and file contents. > Secondly, is it a) sufficient and b) efficient to attempt to time-travel UFS > with the gjournal log? I almost don't dare to mention DragonFly BSD again, but they do have a UFS journaling implementation that does exactly that: http://leaf.dragonflybsd.org/cgi/web-man?command=mountctl http://leaf.dragonflybsd.org/cgi/web-man?command=jscan However, I think the implementation was abandoned because it was obsoleted by the development of the HAMMER file system. But the basic functionality is there and works. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Gesch?ftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M?n- chen, HRB 125758, Gesch?ftsf?hrer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "IRIX is about as stable as a one-legged drunk with hypothermia in a four-hundred mile per hour wind, balancing on a banana peel on a greased cookie sheet -- when someone throws him an elephant with bad breath and a worse temper." -- Ralf Hildebrandt From alancyang at gmail.com Thu Oct 9 15:46:33 2008 From: alancyang at gmail.com (alan yang) Date: Thu Oct 9 15:46:47 2008 Subject: setkey panic freebsd7 Message-ID: <290865fd0810090846y57bbdc1fs3db5c5334fe80c09@mail.gmail.com> i wonder people ran into similar issue on setkey with freebsd7 that panic at ~/crypto/sha1.c:263 within sha1_result() digest[0] = ctxt->h.b8[3]; digest[1] = ctxt->h.b8[2]; on the following sadb add with setkey: add 192.168.0.101 192.168.0.110 esp-old 0x10001 -m any -E des-cbc "12345678" -A keyed-sha1 "12345678123456781234" thanks in advance on any hints. From wjw at digiware.nl Thu Oct 9 17:00:00 2008 From: wjw at digiware.nl (Willem Jan Withagen) Date: Thu Oct 9 17:02:56 2008 Subject: request for testers - xen support for domU in head In-Reply-To: <3c1674c90808221503v5ee48f05td71f70f152e71ef8@mail.gmail.com> References: <3c1674c90808221503v5ee48f05td71f70f152e71ef8@mail.gmail.com> Message-ID: <48EE340B.9010507@IMAP> Kip Macy wrote: > Basic Xen support for 32-bit in PAE mode is in CVS. Please see the > wiki for general information: > > http://wiki.freebsd.org/FreeBSD/Xen > > Please be forewarned that I am not claiming that this is > production-ready. There are many known limitations. If you would like > to take it for a test drive and report bugs please give it a spin. Well it took me some figuring out, before I found this message. Getting it to run with a current-kernel is a lot simpeler. So what I've done is: take your mdroot-disk and your config build the current XEN kernel So I'm running a current-kernel with 7.0 world, I guess. That boots just fine after I changed the disk to xbd1a... So as a nice stresstest I thought: lets mount /usr/src8/src and try to build world. :) Well that is killed in the first serious work with sig 11. Which I at first expected to be due the small memsize. But even upping this on to 256Mb, the compiler still crashes. Also the system suffers from rather seriously long freezes. It can sit there for like > 30 secs. But then on the other hand: It is getting real close to being usefull. --WjW Underlying stuff: dual opteron 245 with 2gb RAM running Ubuntu Hardy en Xen 3.2 PAE --WjW From dillon at apollo.backplane.com Thu Oct 9 19:40:40 2008 From: dillon at apollo.backplane.com (Matthew Dillon) Date: Thu Oct 9 19:40:47 2008 Subject: continuous backup solution for FreeBSD References: <200810091421.m99ELEcm007901@lurza.secnetix.de> Message-ID: <200810091940.m99Jed1t041572@apollo.backplane.com> Go here: http://forums.mysql.com/read.php?28,197644,197644 There are a ton of ways to maintain mysql backups without having to replay the entire log. Google some keywords. With regards to solutions based on filesystem snapshots, such as partial log replaying (snapshot + new-log, then replay from snapshot after crash), HAMMER and ZFS are your only real choices in the BSD world. UFS snapshots have all sorts of issues that make them unsuitable. Block level snapshots are unreliable. When interacting with a high level program for crash recovery purposes you really want to use a filesystem's native snapshot capability (e.g. HAMMER or ZFS. UFS's isn't good enough). You do not need to explicitly backup the filesystem to other media. That is, you of course still make such backups, but they would only be used in case of a catastrophic loss of the production filesystem, not for simple crash recovery which can be done from the same-media snapshots. -- In anycase, HAMMER has two native backup solutions. Both are easy to use. (1) Just use cpdup. This works nicely as the source filesystem, if HAMMER, can be snapshotted, and then used as a stable cpdup source. cpdup to the target and let HAMMER manage the history for you. No hardlink trick needed or anything like that. Use HAMMER's snapshots on the target to determine what history you want to retain. Advantages: Extremely convenient, target storage is reasonably efficient. Great if sources are mixed filesystem types. Disadvantages: Runs cpdup so you are scanning the directory hierarchy on both. Inode numbers not retained, so not suitable for fallback. (this sort of methodology can be used w/ ZFS too). (2) For HAMMER-to-HAMMER you can use HAMMER's native mirroring. Simply create a PFS (pseudo hammer filesystem) slave on the target, which takes 2 seconds, then use the 'hammer mirror-copy' or 'hammer mirror-stream' directives (which can take remote host specifications and will run ssh) to mirror from source to target. HAMMER's mirroring is incremental from the protocol right down to the media accesses. No pre-scan occurs is needed. Advantages: Provides bandwidth-controlled incremental streaming, near-realtime operation (30-60 second lag though finer-grained operation is possible). Extremely robust (can be ^C'd and restarted, or left offline for long periods of time, etc). Mirrors all fine-grained history on source and can be re-pruned on the target to the desired interval. Has little effect on production machines (no queues means backups can't feed-back and effect the performance of the production box). Mirrors inode numbers etc. Mirroring target can become a new master in a pinch (but can't be made a slave again, sorry). Should serve the same NFS fsid, etc. Disadvantages: Must mirror the entire PFS, only works between HAMMER filesystems. Near-real-time (30-60 second lag) is not real time, but it's probably close enough. (ZFS has a way to do something similar but I do not know what the various advantages or disadvantages of using the feature are). DragonFly also has real-time journaling at the VFS layer, which is not HAMMER-specific, but it requires queuing and isn't really suitable on a production environment for off-machine copying because the queue can fill up and block the filesystem. HAMMER's mirroring is far more robust. -Matt From jgrosch at MooseRiver.com Thu Oct 9 20:44:44 2008 From: jgrosch at MooseRiver.com (Josef Grosch) Date: Thu Oct 9 20:44:51 2008 Subject: FreeBSD support for HP DL180/G5 Message-ID: <20081009202521.GA57222@mooseriver.com> Does anyone have experience running FreeBSD 6.x and 7.x on an HP DL180/G5? The company I work for is looking to get a number of these to be put in production. Your general impressions would be a good start. Josef -- Josef Grosch | Another day closer to a | FreeBSD 6.3 jgrosch@MooseRiver.com | Micro$oft free world | Berkeley, Ca. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 155 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081009/f3e96321/attachment.pgp From vanhu at FreeBSD.org Thu Oct 9 21:16:20 2008 From: vanhu at FreeBSD.org (VANHULLEBUS Yvan) Date: Thu Oct 9 21:16:32 2008 Subject: setkey panic freebsd7 In-Reply-To: <290865fd0810090846y57bbdc1fs3db5c5334fe80c09@mail.gmail.com> References: <290865fd0810090846y57bbdc1fs3db5c5334fe80c09@mail.gmail.com> Message-ID: <20081009205636.GA3002@zeninc.net> Hi. On Thu, Oct 09, 2008 at 08:46:32AM -0700, alan yang wrote: > i wonder people ran into similar issue on setkey with freebsd7 that > panic at ~/crypto/sha1.c:263 within sha1_result() > digest[0] = ctxt->h.b8[3]; digest[1] = ctxt->h.b8[2]; > > on the following sadb add with setkey: > add 192.168.0.101 192.168.0.110 esp-old 0x10001 -m any -E des-cbc > "12345678" -A keyed-sha1 "12345678123456781234" > > thanks in advance on any hints. I guess most people just don't use static SAs anymore :-) Can you reproduce the bug ? Are you using /sbin/setkey (provided by FreeBSD), /usr/local/sbin/setkey (provided by ipsec-tools), or does it crash with both ? If you can reproduce it, please fill in a PR, Bjoern or I will take it. Anyways, I'll have a look asap at that part of the code, to see if I can find "something". Any extra information on how to reproduce the bug is welcome ! :-) Yvan. From subhro.kar at gmail.com Thu Oct 9 21:48:54 2008 From: subhro.kar at gmail.com (Subhro) Date: Thu Oct 9 22:05:28 2008 Subject: FreeBSD support for HP DL180/G5 In-Reply-To: <20081009202521.GA57222@mooseriver.com> References: <20081009202521.GA57222@mooseriver.com> Message-ID: HP produces pretty good boxes and historically I have been able to get them working without any troubles. However I would say DL180 is a pretty non customizable box. The hardware works perfectly with FreeBSD 7.0. I didnt try it with 6.3, so cant comment on that. However I would say DL380 is a better off. The main advantage of DL3xx boxes are there is a lot of room to play with add-on cards. Also not all the latest and greatest processors are available with DL1xx family of servers. Also make sure that you go for an external RAID controller like 3ware or Areca. I prefer Areca more :-D. The HP RAID controller cant take the beating I give to it. Thanks Subhro On Fri, Oct 10, 2008 at 1:55 AM, Josef Grosch wrote: > > Does anyone have experience running FreeBSD 6.x and 7.x on an HP DL180/G5? > The company I work for is looking to get a number of these to be put in > production. Your general impressions would be a good start. > > > > Josef > > -- > Josef Grosch | Another day closer to a | FreeBSD 6.3 > jgrosch@MooseRiver.com | Micro$oft free world | Berkeley, Ca. > From zbeeble at gmail.com Thu Oct 9 22:35:16 2008 From: zbeeble at gmail.com (Zaphod Beeblebrox) Date: Thu Oct 9 22:35:23 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <200810091940.m99Jed1t041572@apollo.backplane.com> References: <200810091421.m99ELEcm007901@lurza.secnetix.de> <200810091940.m99Jed1t041572@apollo.backplane.com> Message-ID: <5f67a8c40810091300v37f089a8pa8828f7fba5e9c@mail.gmail.com> On Thu, Oct 9, 2008 at 3:40 PM, Matthew Dillon wrote: > (ZFS has a way to do something similar but I do not know what the > various advantages or disadvantages of using the feature are). The only current way to do this on ZFS is to snapshot (very cheap) and stream the differences between the current snapshot and the previous snapshot to the remote host. The remote host can store the flat files or store the filesystem (that is: the streams can feed into new snapshots of the filesystem on the remote machine). Like Hammer, this gives history on both the local machine and the remote machine and is amazingly efficient. Unlike hammer, the process is not automated by the filesystem. You need a script that does "zfs snaphot..." followed by "zfs send ... | ssh zfs receive ..." --- such that each individual backup is a job rather than the connection approach you discribe for Hammer. Unlike Hammer, ZFS doesn't, by default, keep all history. I was speculating earlier that this might be possible to make as a ZFS module, though. From yurtesen at ispro.net Thu Oct 9 23:34:12 2008 From: yurtesen at ispro.net (yurtesen@ispro.net) Date: Thu Oct 9 23:45:32 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <200810091411.m99EB0Vo007538@lurza.secnetix.de> References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> Message-ID: <20081010023428.87556dt18ejyzf48@mail.ispro.net> Quoting "Oliver Fromme" : > These features are readily available right now on FreeBSD. > You don't have to code anything. Well with 2 downsides, The fact that I still would need to take full backups once in a while if I do this and Linux users do not have to because the CDP software on Linux does not need to do this. The software expires the old data automatically and you only need a full backup at first run only. The bigger problem is that I have to convert all my filesystems to ZFS. Can one convert UFS2 to ZFS easily even? I dont wanna spend a large part of the year doing such job while Linux users can just do this on 'any' filesystem they use. How am I suppose to compete with companies which use Linux otherwise if I am doing this sort of tasks all the time? Thanks for all the advices but my original question was if somebody can give inside information to a company(for example r1soft) which is writing CDP backup solutions so they could implement such solution on FreeBSD also. Do you know such person? I am not really looking for alternatives because there is none. You cant just expect commercial companies to convert to a new filesystem to add a feature which other OSes manage without going to such measures. Can you imagine the monetary cost if all FreeBSD users had to convert to ZFS (or another filesystem) to take near cdp level backups? This simply would make people think 'I wish I used Linux from the beginning'. Thanks, Evren From mwm-keyword-freebsdhackers2.e313df at mired.org Fri Oct 10 00:09:16 2008 From: mwm-keyword-freebsdhackers2.e313df at mired.org (Mike Meyer) Date: Fri Oct 10 00:09:31 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <20081010023428.87556dt18ejyzf48@mail.ispro.net> References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> Message-ID: <20081009200641.60d0b236@bhuda.mired.org> On Fri, 10 Oct 2008 02:34:28 +0300 yurtesen@ispro.net wrote: > Quoting "Oliver Fromme" : > > > These features are readily available right now on FreeBSD. > > You don't have to code anything. > > Well with 2 downsides, Once you actually try and implement these solutions, you'll see that your "downsides" are largely figments of your imagination. > The fact that I still would need to take full backups once in a while Every time you started backing up a new file system. This is a requirement of all backup solutions. After that, never again. > The bigger problem is that I have to convert all my filesystems to > ZFS. Can one convert UFS2 to ZFS easily even? I didn't have any trouble. And once you do it, you have advantages that the poor schmucks using Linux can only dream about: like self-healing file systems that are so cheap and easy to create that it makes sense to put each application or jail on it's own file system, one that's tuned for the application. The ability to set up raid and mirror devices without ever having to deal with LLVM (worth the cost of entry all by itself), not having to worry about allocating partitions, and - well, the list just goes on and on. Having converted to ZFS on my FreeBSD boxes, the only thing I feel for my clients still using Linux file systems is pity. > this on 'any' filesystem they use I seriously doubt that it supports things like GMailFS. > How am I suppose to compete with > companies which use Linux otherwise if I am doing this sort of tasks > all the time? Well, once you've done the conversion, by outproducing them while they waste time dealing with LLVM, partitions, and other such crap that ZFS frees you from. > Thanks for all the advices but my original question was if somebody > can give inside information to a company(for example r1soft) which is > writing CDP backup solutions so they could implement such solution on > FreeBSD also. Do you know such person? The only "inside information" here is held by the company (for example rlsoft) providing the CDP software. On the FreeBSD side, the source and documentation are all freely available to anyone who wants to look at it. But it doesn't matter how well you know FreeBSD, you aren't going to get anywhere unless you also you know what the software from that company needs in order to operate. If said company wanted to hire someone to either write this or to help get their people started working with FreeBSD, then the thing to do is send mail to jobs@freebsd.org announcing the position. If they aren't interested in hiring someone, but hope to get it done for free, then they should set up a web page providing the technical details that someone who knows FreeBSD (or is willing to learn it) needs to do the job. If all you want is a CDP solution and you don't care where it comes from - well, you pretty much get the same two choices. It's an interesting enough problem that you might get someone to build it for free, but don't expect it to use proprietary software from some company that already provides such a thing for other systems. Nor - if you don't provide any incentive for meeting your requirements - should you expect it to actually meet them. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From dillon at apollo.backplane.com Fri Oct 10 01:14:23 2008 From: dillon at apollo.backplane.com (Matthew Dillon) Date: Fri Oct 10 01:14:30 2008 Subject: continuous backup solution for FreeBSD References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> Message-ID: <200810100114.m9A1ELOG043983@apollo.backplane.com> :The fact that I still would need to take full backups once in a while =20 :if I do this and Linux users do not have to because the CDP software =20 :on Linux does not need to do this. The software expires the old data =20 :automatically and you only need a full backup at first run only. You have to do this anyway. Nobody in their right mind trusts a single storage solution for all of their backups. CDP is a brute-force block-level continuous backup mechanism and while it works, to a point, it also has all the drawbacks that ANY block level backup system has... it is discontinuous from the high level filesystem structure and while you can pretty much guarantee that it is possible to recover from a disaster eventually, the filesystem you choose to run on top of that block device still matters a lot. On top of that being a client-server solution CDP is going to have some significant bottlenecks too. Even more telling is the fact that block level storage solutions tend to migrate corruption instead of detect it early. Big oopses can wind up stealthily winding their way through all your backups when you use a block-level solution. I stopped using block-level solutions almost 10 years ago... that's how little I trust them. Only a 'modern' linux filesystem (ext3 or the upcoming ext4, reiser, xfs, and few others), or something like ZFS or HAMMER, really have the ability to reliably recover from a point-in-time block level snapshot. Filesystems such as ZFS and HAMMER also give you insanely good snapshotting solutions that are far, FAR more flexible then what CDP gives you. You can upgrade between EXT filesystems without having to copy, but if you decided the best filesystem for you was one of the other many Linux filesystems, such as Reiser4 or XFS, and you were running EXT3, then you would have to copy. There are massive lists of pluses and minuses to each of the linux filesystem choices. Data expiration is a non-issue. You have to think about it either way. You have to test that the backup system actually works. You have to carefully control the backup policy and in particular not allow heavy disk I/O (such as a hacker DD'ing to your filesystem for 24 hours) to blow out your ability to recover the system. It requires time and effort no matter how well automated it is. :The bigger problem is that I have to convert all my filesystems to =20 :ZFS. Can one convert UFS2 to ZFS easily even? I dont wanna spend a =20 :large part of the year doing such job while Linux users can just do =20 :this on 'any' filesystem they use. How am I suppose to compete with =20 :companies which use Linux otherwise if I am doing this sort of tasks =20 :all the time? I converted our main developer machine from UFS to HAMMER in about 12 hours. 99% of that time simply waiting for the cpdup to finish copying a few hundred gigabytes from the old filesystem to the new. I think we're talking a few days here... time enough to learn how the filesystem works and play with it, make a few test copies (as you would with ANY new system that you did not have previous experienced with), and then do it for real. Linux users cannot just do this with a flip of a switch either. The new filesystem has to be constructed and the old data copied over. The old data set has to be retained, you cannot convert in-place, it's just too dangerous. It takes a certain amount of time to copy the data no matter what OS you are running, based on the amount of data you have. Some filesystem transitions such as going from ext2 to ext3 or ext3 to ext4 (if I remember right) are forwards compatible and do not require copying, but that sort of transition is NOT to a new filesystem, it's to a newer version of the same filesystem. With storage capacities increasing so quickly and mandating the replacement of whole disk subsystems (for running and electricity cost more then anything else, and more convenience, and less maintainance) it is a small convenience at best if you are going to copy the data anyway. Frankly, even if I were upgrading from ext2 to ext3 the last thing I would do is run it in-place. There's just too high a chance of software bugs creeping in. I would want a fresh ext3 and I would want my old ext2 data sitting on a shelf for a few days in case something were to go terribly wrong. I'd copy there too, just for safety. What is a few extra hours compared to blowing up the life-blood of your company? :I am not really looking for alternatives because there is none. You =20 :cant just expect commercial companies to convert to a new filesystem =20 :to add a feature which other OSes manage without going to such =20 :measures. Can you imagine the monetary cost if all FreeBSD users had =20 :to convert to ZFS (or another filesystem) to take near cdp level =20 :backups? This simply would make people think 'I wish I used Linux from =20 :the beginning'. : :Thanks, :Evren Commercial companies like to farm things out and they certainly like turnkey solutions. There are many more linux companies then BSD companies which offer turnkey solutions for various narrow bands of problems. I'm not knocking it, it's how the world works. Linux has a great deal of momentum despite the fracturing of the distributions. Linux has many other things going for it including a far better package updating scheme then any of the BSDs, but there are also downsides such as that huge ssh public key mess that was basically one programmer committing a change to a piece of code he was clueless about. The BSDs don't have the vendor support that linux has. There are many reasons for this, and it's really too bad that it turned out that way, but if you are a sysop in a company that doesn't have infinite $$ to spend, and you can't afford the high costs of turnkey solutions, then rolling your own on the core platforms (Linux or BSD) require only having a good head on your shoulders to turn into a success. It will take about the same amount of time either way. You might be pleasantly surprised! I know a lot of people who tear their hair out on Linux (but unfortunately the same people mostly think BSD's upsides don't make up for its downsides, sigh). But be careful not to confuse turnkey solutions with flexibility. Turnkey solutions... things like CDP, are narrowly focused and tend to be very inflexible when it comes to integration beyond their basic design. If something breaks in the middle of that black box you will be in 'houston we have a problem! mode' and it won't be fixed quickly. These sorts of companies expect to be paid, most of them with ongoing support contracts. Turnkey solutions are not going to be inexpensive. Even for open-source software, if you intend to manage both sides of the turnkey equation yourself you are essentially going to be devoting the same amount of time to it as someone rolling their own solution based on lower level (but still substantial) building blocks, such as the native features of ZFS. I can give you nightmare stories about turnkey backup solutions which either fail unexpectedly and f*ck up a company, or worse: Become obsolete and the programmers have so little visibility into the turnkey 'solution' they are unable to upgrade it. Lots of people have the latter problem. Turnkey can be good if carefully managed, and a nightmare if not. -Matt Matthew Dillon From alancyang at gmail.com Fri Oct 10 03:19:01 2008 From: alancyang at gmail.com (alan yang) Date: Fri Oct 10 03:19:08 2008 Subject: setkey panic freebsd7 In-Reply-To: <20081009205636.GA3002@zeninc.net> References: <290865fd0810090846y57bbdc1fs3db5c5334fe80c09@mail.gmail.com> <20081009205636.GA3002@zeninc.net> Message-ID: <290865fd0810092018v59aeecc1rfc38790c0fde2f5e@mail.gmail.com> that single line of adding SA in a setkey.conf file with /sbin/setkey -f setkey.conf would fail 100% from all my try. /usr/local/sbin/setkey just tried, failed also. 'fill in PR' haven't done that before, could you please advise. thanks for looking into! On Thu, Oct 9, 2008 at 1:56 PM, VANHULLEBUS Yvan wrote: > Hi. > > > On Thu, Oct 09, 2008 at 08:46:32AM -0700, alan yang wrote: >> i wonder people ran into similar issue on setkey with freebsd7 that >> panic at ~/crypto/sha1.c:263 within sha1_result() >> digest[0] = ctxt->h.b8[3]; digest[1] = ctxt->h.b8[2]; >> >> on the following sadb add with setkey: >> add 192.168.0.101 192.168.0.110 esp-old 0x10001 -m any -E des-cbc >> "12345678" -A keyed-sha1 "12345678123456781234" >> >> thanks in advance on any hints. > > I guess most people just don't use static SAs anymore :-) > > Can you reproduce the bug ? > Are you using /sbin/setkey (provided by FreeBSD), > /usr/local/sbin/setkey (provided by ipsec-tools), or does it crash > with both ? > > > If you can reproduce it, please fill in a PR, Bjoern or I will take > it. > > Anyways, I'll have a look asap at that part of the code, to see if I > can find "something". > > Any extra information on how to reproduce the bug is welcome ! :-) > > > Yvan. > From delphij at delphij.net Fri Oct 10 09:20:27 2008 From: delphij at delphij.net (Xin LI) Date: Fri Oct 10 09:20:34 2008 Subject: Debugging modify-after-free issue Message-ID: <48EF1E51.4050205@delphij.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, Recently I had some crashes and other issues on my laptop and I found that, among some bugs I already caught and fixed on my local tree, there is still something like this: Oct 10 01:35:53 delta kernel: Memory modified after free 0xffffff00a2ef9600(256) val=6438300 @ 0xffffff00a2ef9608 Is there any way to track this down (looks like to be wpi(4) related but still keep trying)? I can not seem to reliably trigger it so it would be helpful if such thing would trigger a panic. Cheers, -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkjvHlAACgkQi+vbBBjt66B2YQCeMXQwUWJOOm9PdCiFnsQD7Qba exkAnAz/4VRm+ZrU83XudWeo6lLTlkHO =N2pO -----END PGP SIGNATURE----- From yurtesen at ispro.net Fri Oct 10 12:53:45 2008 From: yurtesen at ispro.net (Evren Yurtesen) Date: Fri Oct 10 12:56:23 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <20081009200641.60d0b236@bhuda.mired.org> References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> Message-ID: <48EF5052.2000707@ispro.net> Mike Meyer wrote: > On Fri, 10 Oct 2008 02:34:28 +0300 > yurtesen@ispro.net wrote: > >> Quoting "Oliver Fromme" : >> >>> These features are readily available right now on FreeBSD. >>> You don't have to code anything. >> Well with 2 downsides, > > Once you actually try and implement these solutions, you'll see that > your "downsides" are largely figments of your imagination. So if it is my imagination, how can I actually convert UFS to ZFS easily? Everybody seems to say that this is easy and that is easy. When you look at this from a single point then you might be right. But next thing in my agenda is to provide restore services to hosting customers. Now when I use a commercial solution like r1soft backup, I can just install the plugin for the control panel software (like cPanel or H-Sphere). If it is imagination and things are so easy, I can give you $500(r1soft pricing) for every 5servers I have, and you can give the same features that r1soft gives, deal? I have no doubt that similar solution can be achieved on FreeBSD. If you sit and think for a second, the problem is the amount of time, resources and knowledge it requires to do the same solution in FreeBSD is way higher than if one was using Linux. > I seriously doubt that it supports things like GMailFS. You are right, my mistake. But it supports the most commonly used Linux filesystems, still better than changing to a brand new filesystem. But sorry for the mistake. Thanks, Evren From koitsu at FreeBSD.org Fri Oct 10 14:42:33 2008 From: koitsu at FreeBSD.org (Jeremy Chadwick) Date: Fri Oct 10 14:42:42 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <48EF5052.2000707@ispro.net> References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> Message-ID: <20081010144111.GA34609@icarus.home.lan> On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: > Mike Meyer wrote: >> On Fri, 10 Oct 2008 02:34:28 +0300 >> yurtesen@ispro.net wrote: >> >>> Quoting "Oliver Fromme" : >>> >>>> These features are readily available right now on FreeBSD. >>>> You don't have to code anything. >>> Well with 2 downsides, >> >> Once you actually try and implement these solutions, you'll see that >> your "downsides" are largely figments of your imagination. > > So if it is my imagination, how can I actually convert UFS to ZFS > easily? Everybody seems to say that this is easy and that is easy. It's not that easy. I really don't know why people are telling you it is. Converting some filesystems are easier than others; /home (if you create one) for example is generally easy: 1) ZFS fs is called foo/home, mounted as /mnt 2) fstat, ensure nothing is using /home -- if something is, shut it down or kill it 3) rsync or cpdup /home files to /mnt 4) umount /home 5) zfs set mountpoint=/home foo/home 6) Restart said processes or daemons "See! It's like I said! EASY!" You can do this with /var as well. Now try /usr. Hope you've got /rescue available, because once /usr/lib and /usr/libexec disappear, you're in trouble. Good luck doing this in multi-user, too. And finally, the root fs. Whoever says "this is easy" is kidding themselves; it's a pain. You get to make a new filesystem called /boot, and have all sorts of fun. It's really not a snap-fingers-voila thing, and I will gladly argue with anyone who thinks otherwise. Is it do-able though? Yes. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From mwm-keyword-freebsdhackers2.e313df at mired.org Fri Oct 10 15:32:27 2008 From: mwm-keyword-freebsdhackers2.e313df at mired.org (Mike Meyer) Date: Fri Oct 10 15:32:34 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <20081010144111.GA34609@icarus.home.lan> References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> <20081010144111.GA34609@icarus.home.lan> Message-ID: <20081010112952.52b8209b@bhuda.mired.org> On Fri, 10 Oct 2008 07:41:11 -0700 Jeremy Chadwick wrote: > On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: > > Mike Meyer wrote: > >> On Fri, 10 Oct 2008 02:34:28 +0300 > >> yurtesen@ispro.net wrote: > >> > >>> Quoting "Oliver Fromme" : > >>> > >>>> These features are readily available right now on FreeBSD. > >>>> You don't have to code anything. > >>> Well with 2 downsides, > >> > >> Once you actually try and implement these solutions, you'll see that > >> your "downsides" are largely figments of your imagination. > > > > So if it is my imagination, how can I actually convert UFS to ZFS > > easily? Everybody seems to say that this is easy and that is easy. > > It's not that easy. I really don't know why people are telling you it > is. Maybe because it is? Of course, it *does* require a little prior planning, but anyone with more than a few months experience as a sysadmin should be able to deal with it without to much hassle. > Converting some filesystems are easier than others; /home (if you > create one) for example is generally easy: > > 1) ZFS fs is called foo/home, mounted as /mnt > 2) fstat, ensure nothing is using /home -- if something is, shut it > down or kill it > 3) rsync or cpdup /home files to /mnt > 4) umount /home > 5) zfs set mountpoint=/home foo/home > 6) Restart said processes or daemons > > "See! It's like I said! EASY!" You can do this with /var as well. Yup. Of course, if you've done it that way, you're not thinking ahead, because: > Now try /usr. Hope you've got /rescue available, because once /usr/lib > and /usr/libexec disappear, you're in trouble. Good luck doing this in > multi-user, too. Oops. You F'ed up. If you'd done a little planning, you would have realized that / and /usr would be a bit of extra trouble, and planned accordingly. > And finally, the root fs. Whoever says "this is easy" is kidding > themselves; it's a pain. Um, no, it wasn't. Of course, I've been doing this long enough to have a system set up to make this kind of thing easy. My system disk is on a mirror, and I do system upgrades by breaking the mirror and upgrading one disk, making everything work, then putting the mirror back together. And moving to zfs on root is a lot like a system upgrade: 1) Break the mirror (mirrors actually, as I mirrored file systems). 2) Repartition the unused drive into /boot, swap & data 3) Build zfs & /boot according to the instructions on ZFSOnRoot wiki, just copying /boot and / at this point. 4) Boot the zfs disk in single user mode. 5) If 4 fails, boot back to the ufs disk so you're operational while you contemplate what went wrong, then repeat step 3. Otherwise, go on to step 6. 6) Create zfs file systems as appropriate (given that zfs file systems are cheap, and have lots of cool features that ufs file systems don't have, you probably want to create more than you had before, doing thing like putting SQL serves on their own file system with appropriate blocking, etc, but you'll want to have figured all this out before starting step 1). 7) Copy data from the ufs file systems to their new homes, not forgetting to take them out of /etc/fstab. 8) Reboot on the zfs disk. 9) Test until you're happy that everything is working properly, and be prepared to reboot on the ufs disk if something is broken. 10) Reformat the ufs disk to match the zfs one. Gmirror /boot, add the data partition to the zfs pool so it's mirrored, and you should have already been using swap. This is 10 steps to your "easy" 6, but two of the extra steps are testing you didn't include, and 1 of the steps is a failure recovery step that shouldn't be necessary. So - one more step than your easy process. Yeah, this isn't something you do on a whim. On the other hand, it's not something that any competent sysadmin would consider a pain. For a good senior admin, it's a lot easier than doing an OS upgrade from source, which should be the next step up from trivial. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From koitsu at FreeBSD.org Fri Oct 10 15:42:53 2008 From: koitsu at FreeBSD.org (Jeremy Chadwick) Date: Fri Oct 10 15:42:59 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <20081010112952.52b8209b@bhuda.mired.org> References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> <20081010144111.GA34609@icarus.home.lan> <20081010112952.52b8209b@bhuda.mired.org> Message-ID: <20081010154249.GA35859@icarus.home.lan> On Fri, Oct 10, 2008 at 11:29:52AM -0400, Mike Meyer wrote: > On Fri, 10 Oct 2008 07:41:11 -0700 > Jeremy Chadwick wrote: > > > On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: > > > Mike Meyer wrote: > > >> On Fri, 10 Oct 2008 02:34:28 +0300 > > >> yurtesen@ispro.net wrote: > > >> > > >>> Quoting "Oliver Fromme" : > > >>> > > >>>> These features are readily available right now on FreeBSD. > > >>>> You don't have to code anything. > > >>> Well with 2 downsides, > > >> > > >> Once you actually try and implement these solutions, you'll see that > > >> your "downsides" are largely figments of your imagination. > > > > > > So if it is my imagination, how can I actually convert UFS to ZFS > > > easily? Everybody seems to say that this is easy and that is easy. > > > > It's not that easy. I really don't know why people are telling you it > > is. > > Maybe because it is? Of course, it *does* require a little prior > planning, but anyone with more than a few months experience as a > sysadmin should be able to deal with it without to much hassle. > > > Converting some filesystems are easier than others; /home (if you > > create one) for example is generally easy: > > > > 1) ZFS fs is called foo/home, mounted as /mnt > > 2) fstat, ensure nothing is using /home -- if something is, shut it > > down or kill it > > 3) rsync or cpdup /home files to /mnt > > 4) umount /home > > 5) zfs set mountpoint=/home foo/home > > 6) Restart said processes or daemons > > > > "See! It's like I said! EASY!" You can do this with /var as well. > > Yup. Of course, if you've done it that way, you're not thinking ahead, > because: > > > Now try /usr. Hope you've got /rescue available, because once /usr/lib > > and /usr/libexec disappear, you're in trouble. Good luck doing this in > > multi-user, too. > > Oops. You F'ed up. If you'd done a little planning, you would have > realized that / and /usr would be a bit of extra trouble, and planned > accordingly. > > > And finally, the root fs. Whoever says "this is easy" is kidding > > themselves; it's a pain. > > Um, no, it wasn't. Of course, I've been doing this long enough to have > a system set up to make this kind of thing easy. My system disk is on > a mirror, and I do system upgrades by breaking the mirror and > upgrading one disk, making everything work, then putting the mirror > back together. And moving to zfs on root is a lot like a system > upgrade: > > 1) Break the mirror (mirrors actually, as I mirrored file systems). > 2) Repartition the unused drive into /boot, swap & data > 3) Build zfs & /boot according to the instructions on ZFSOnRoot > wiki, just copying /boot and / at this point. > 4) Boot the zfs disk in single user mode. > 5) If 4 fails, boot back to the ufs disk so you're operational while > you contemplate what went wrong, then repeat step 3. Otherwise, go > on to step 6. > 6) Create zfs file systems as appropriate (given that zfs file > systems are cheap, and have lots of cool features that ufs > file systems don't have, you probably want to create more than > you had before, doing thing like putting SQL serves on their > own file system with appropriate blocking, etc, but you'll want to > have figured all this out before starting step 1). > 7) Copy data from the ufs file systems to their new homes, > not forgetting to take them out of /etc/fstab. > 8) Reboot on the zfs disk. > 9) Test until you're happy that everything is working properly, > and be prepared to reboot on the ufs disk if something is broken. > 10) Reformat the ufs disk to match the zfs one. Gmirror /boot, > add the data partition to the zfs pool so it's mirrored, and > you should have already been using swap. > > This is 10 steps to your "easy" 6, but two of the extra steps are > testing you didn't include, and 1 of the steps is a failure recovery > step that shouldn't be necessary. So - one more step than your easy > process. Of course, the part you seem to be (intentionally?) forgetting: most people are not using gmirror. There is no 2nd disk. They have one disk with a series of UFS2 filesystems, and they want to upgrade. That's how I read Evren's "how do I do this? You say it's easy..." comment, and I think his viewpoint is very reasonable. > Yeah, this isn't something you do on a whim. On the other hand, it's > not something that any competent sysadmin would consider a pain. For a > good senior admin, it's a lot easier than doing an OS upgrade from > source, which should be the next step up from trivial. I guess you have a very different definition of "easy". :-) The above procedure, in no way shape or form, will be classified as "easy" by the user (or even junior sysadmin) community, I can assure you of that. I'll also throw this in the mix: the fact that we are *expecting* users to know how to do this is unreasonable. It's even *more* rude to expect that mid-level or senior SAs have to do it "the hard way". Why? I'll explain: I'm an SA of 16+ years. I'm quite familiar with PBR/MBR, general disk partitioning, sectors vs. blocks, slices, filesystems, and whatever else. You want me to do it by hand, say, with bsdlabel -e? Fine, I will -- but I will not be happy about it. I have the knowledge, I know how to do it, so why must the process continue to be a PITA and waste my time? This is exactly why I hate things like the OpenBSD installer: do not make me do it the hard way. We're living in 2008, not 1989. Evolve. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From mwm-keyword-freebsdhackers2.e313df at mired.org Fri Oct 10 15:59:36 2008 From: mwm-keyword-freebsdhackers2.e313df at mired.org (Mike Meyer) Date: Fri Oct 10 15:59:42 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <48EF79FE.9010302@ispro.net> References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> <20081010144111.GA34609@icarus.home.lan> <48EF79FE.9010302@ispro.net> Message-ID: <20081010115702.05440c6f@bhuda.mired.org> On Fri, 10 Oct 2008 18:51:26 +0300 Evren Yurtesen wrote: > The original question (which was lost) was if somebody who has technical > knowledge and coding skills who can put r1soft into the right track so > their software can support FreeBSD. Because r1soft is interested in > supporting FreeBSD. I answered that, and you ignored it: If rlsoft is interested in hiring someone for this, they should sedn a message to freebsd-jobs. If they aren't interested in hiring someone, they need to release the technical information that would be required to do it, and then provide pointers to that information to the FreeBSD community (i.e. - by posting it here, rather than just posting trolls). http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From yurtesen at ispro.net Fri Oct 10 15:51:22 2008 From: yurtesen at ispro.net (Evren Yurtesen) Date: Fri Oct 10 16:03:51 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <20081010144111.GA34609@icarus.home.lan> References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> <20081010144111.GA34609@icarus.home.lan> Message-ID: <48EF79FE.9010302@ispro.net> I just wanted to say thanks to all the replies to this thread. It has been insightful even though the suggestions I have received were not really answers to what I asked. I dont see any reason why we should continue to argue about if this can be done using ZFS or Hammer or any other filesystem. The fact is that it can be done eventually with something else if one has the will, time and money to put into it. One can even write his own filesystem code for it eh? The original question (which was lost) was if somebody who has technical knowledge and coding skills who can put r1soft into the right track so their software can support FreeBSD. Because r1soft is interested in supporting FreeBSD. Most of you probably know that FreeBSD support in products of commercial companies are scarce already. This is causing reductions in user base and popularity, although I am not saying that the more users are the better or I am not saying that popularity is everything. But you have to be able to combine one issue with another and see what the results might be in the long run. For example, wouldnt it be good for FreeBSD itself if Oracle supported FreeBSD? While I cant make Oracle to support FreeBSD, there is a company out there with a popular and quality product which is interested in supporting FreeBSD in their products. So why not take advantage of the situation? While I didnt think about some details when I sent the first post, some people thought that I would want somebody to code for r1soft for free so they can sell the software. No I do not and did not expect anybody to do anything for free. So, to summarize, if there is anybody who wants to help the 'actual' issue here, either by getting payed or just giving free hints and tips to r1soft then please contact me (I can forward your contact info) or with r1soft. Dag-Erling Sm?rgrav already told me that I can send his info and I have sent his contact info to r1soft for this issue. I hope we can now stop arguing about all the other things :) It wasnt my intention to start such arguments. Thanks, Evren From mwm-keyword-freebsdhackers2.e313df at mired.org Fri Oct 10 16:25:02 2008 From: mwm-keyword-freebsdhackers2.e313df at mired.org (Mike Meyer) Date: Fri Oct 10 16:25:09 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <20081010154249.GA35859@icarus.home.lan> References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> <20081010144111.GA34609@icarus.home.lan> <20081010112952.52b8209b@bhuda.mired.org> <20081010154249.GA35859@icarus.home.lan> Message-ID: <20081010122228.355c2c3e@bhuda.mired.org> On Fri, 10 Oct 2008 08:42:49 -0700 Jeremy Chadwick wrote: > On Fri, Oct 10, 2008 at 11:29:52AM -0400, Mike Meyer wrote: > > On Fri, 10 Oct 2008 07:41:11 -0700 > > Jeremy Chadwick wrote: > > > > > On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: > > > > Mike Meyer wrote: > > > >> On Fri, 10 Oct 2008 02:34:28 +0300 > > > >> yurtesen@ispro.net wrote: > > > >> > > > >>> Quoting "Oliver Fromme" : > > > >>> > > > >>>> These features are readily available right now on FreeBSD. > > > >>>> You don't have to code anything. > > > >>> Well with 2 downsides, > > > >> > > > >> Once you actually try and implement these solutions, you'll see that > > > >> your "downsides" are largely figments of your imagination. > > > > > > > > So if it is my imagination, how can I actually convert UFS to ZFS > > > > easily? Everybody seems to say that this is easy and that is easy. > > > > > > It's not that easy. I really don't know why people are telling you it > > > is. > > > > Maybe because it is? Of course, it *does* require a little prior > > planning, but anyone with more than a few months experience as a > > sysadmin should be able to deal with it without to much hassle. > > > > > Converting some filesystems are easier than others; /home (if you > > > create one) for example is generally easy: > > > > > > 1) ZFS fs is called foo/home, mounted as /mnt > > > 2) fstat, ensure nothing is using /home -- if something is, shut it > > > down or kill it > > > 3) rsync or cpdup /home files to /mnt > > > 4) umount /home > > > 5) zfs set mountpoint=/home foo/home > > > 6) Restart said processes or daemons > > > > > > "See! It's like I said! EASY!" You can do this with /var as well. > > > > Yup. Of course, if you've done it that way, you're not thinking ahead, > > because: > > > > > Now try /usr. Hope you've got /rescue available, because once /usr/lib > > > and /usr/libexec disappear, you're in trouble. Good luck doing this in > > > multi-user, too. > > > > Oops. You F'ed up. If you'd done a little planning, you would have > > realized that / and /usr would be a bit of extra trouble, and planned > > accordingly. > > > > > And finally, the root fs. Whoever says "this is easy" is kidding > > > themselves; it's a pain. > > > > Um, no, it wasn't. Of course, I've been doing this long enough to have > > a system set up to make this kind of thing easy. My system disk is on > > a mirror, and I do system upgrades by breaking the mirror and > > upgrading one disk, making everything work, then putting the mirror > > back together. And moving to zfs on root is a lot like a system > > upgrade: > > > > 1) Break the mirror (mirrors actually, as I mirrored file systems). > > 2) Repartition the unused drive into /boot, swap & data > > 3) Build zfs & /boot according to the instructions on ZFSOnRoot > > wiki, just copying /boot and / at this point. > > 4) Boot the zfs disk in single user mode. > > 5) If 4 fails, boot back to the ufs disk so you're operational while > > you contemplate what went wrong, then repeat step 3. Otherwise, go > > on to step 6. > > 6) Create zfs file systems as appropriate (given that zfs file > > systems are cheap, and have lots of cool features that ufs > > file systems don't have, you probably want to create more than > > you had before, doing thing like putting SQL serves on their > > own file system with appropriate blocking, etc, but you'll want to > > have figured all this out before starting step 1). > > 7) Copy data from the ufs file systems to their new homes, > > not forgetting to take them out of /etc/fstab. > > 8) Reboot on the zfs disk. > > 9) Test until you're happy that everything is working properly, > > and be prepared to reboot on the ufs disk if something is broken. > > 10) Reformat the ufs disk to match the zfs one. Gmirror /boot, > > add the data partition to the zfs pool so it's mirrored, and > > you should have already been using swap. > > > > This is 10 steps to your "easy" 6, but two of the extra steps are > > testing you didn't include, and 1 of the steps is a failure recovery > > step that shouldn't be necessary. So - one more step than your easy > > process. > > Of course, the part you seem to be (intentionally?) forgetting: most > people are not using gmirror. There is no 2nd disk. They have one disk > with a series of UFS2 filesystems, and they want to upgrade. That's how > I read Evren's "how do I do this? You say it's easy..." comment, and I > think his viewpoint is very reasonable. Granted, most people don't think about system upgrades when they build a system, so they wind up having to do extra work. In particular, Evren is talking about spending thousands of dollars on proprietary software, not to mention the cost of the server that all this data is going to flow to, for a backup solution. Compared to that, the cost of a few spare disks and the work to install them are trivial. > > Yeah, this isn't something you do on a whim. On the other hand, it's > > not something that any competent sysadmin would consider a pain. For a > > good senior admin, it's a lot easier than doing an OS upgrade from > > source, which should be the next step up from trivial. > I guess you have a very different definition of "easy". :-) Given that mine is based on years of working with the kinds of backup solutions that Evren is asking for: ones that an enterprise deploys for backing up a data center, the answer may well be "yes". > The above procedure, in no way shape or form, will be classified as > "easy" by the user (or even junior sysadmin) community, I can assure you > of that. I never said it would be easy for a user. Then again, your average user doesn't do backups, and wouldn't know a continuous backup solution from a credit default swap. We're not talking about ghosting a disk partition for a backup, we're talking about enterprise-level backup solutions for data centers. People deploying those kinds of solutions tend to have multiple senior sysadmins around. I wouldn't expect a junior admin to call it easy. At least, not the first two or three times. If they still have problems with it after that, they should find a new career path, as they aren't ever going to advance beyond junior. > I'll also throw this in the mix: the fact that we are *expecting* users > to know how to do this is unreasonable. It's even *more* rude to expect Um, is anyone expecting users to do this? I'm not. ZFS is still marked as "experimental" in FreeBSD. That means that, among other things, it's not really well-supported by the installer, etc. Nuts, as of January of this year, there wasn't an operating system on the planet that would install and boot from ZFS. I'm willing to jump through some hoops to get ZFS's advantages. Those happen to include some things that go a long way to solving Zefren's problems, so it was suggested as the basis for such (not by me, mind you). Having done the conversion, and found it easy, I responded when he asked how hard it was. But I'd never recommend this for your average user - which pretty much excludes anyone contemplating continuous backup solutions. > that mid-level or senior SAs have to do it "the hard way". Why? I'll > explain: > > I'm an SA of 16+ years. I'm quite familiar with PBR/MBR, general disk > partitioning, sectors vs. blocks, slices, filesystems, and whatever > else. You want me to do it by hand, say, with bsdlabel -e? Fine, I > will -- but I will not be happy about it. I have the knowledge, I > know how to do it, so why must the process continue to be a PITA and > waste my time? Did I ever mention bsdlabel? But in any case, ZFS makes pretty much *all* that crap obsolete. You still have to deal with getting a boot loader installed, but after that, you never have to worry about partitioning, blocks, sectors, or slices again - until you go to an operating system that doesn't have ZFS. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From amdmi3 at amdmi3.ru Fri Oct 10 18:28:55 2008 From: amdmi3 at amdmi3.ru (Dmitry Marakasov) Date: Fri Oct 10 18:29:03 2008 Subject: VirtualBox looks for FreeBSD developer In-Reply-To: <20080919082719.GH676@e.0x20.net> References: <48C52718.5080807@sun.com> <48C8F051.7060107@sun.com> <20080919082719.GH676@e.0x20.net> Message-ID: <20081010174204.GB90757@hades.panopticon> Hi! Little time ago I was misleaded by the certain people and got an idea that VirtualBox actually works on FreeBSD, so I've made a draft port for it. It doesn't actually work, but since I've spent several hours hacking it and made bunch of (likely) useful patches, here it is, feel free to use it for any purpose. I hope someone of kernel hackers will make it work actually ;) Summary: - installation doesn't work - only 7.0 (7.x maybe) / i386 is supported - VM doesn't run - You'll be able to run QT4 GUI and some tests if you're lucky :) See README.txt inside for additional details. http://amdmi3.ru/files/virtualbox-port.tar.gz -- Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D amdmi3@amdmi3.ru ..: jabber: amdmi3@jabber.ru http://www.amdmi3.ru From alancyang at gmail.com Fri Oct 10 20:25:04 2008 From: alancyang at gmail.com (alan yang) Date: Fri Oct 10 20:25:10 2008 Subject: setkey panic freebsd7 In-Reply-To: <290865fd0810090846y57bbdc1fs3db5c5334fe80c09@mail.gmail.com> References: <290865fd0810090846y57bbdc1fs3db5c5334fe80c09@mail.gmail.com> Message-ID: <290865fd0810101325kbd4caber6b132253784523a5@mail.gmail.com> sorry, /usr/local/sbin/setkey failed on parsing that specific add, not panic. no specific info, just say parse failed. maybe something is not supported ...? On Thu, Oct 9, 2008 at 8:46 AM, alan yang wrote: > i wonder people ran into similar issue on setkey with freebsd7 that > panic at ~/crypto/sha1.c:263 within sha1_result() > digest[0] = ctxt->h.b8[3]; digest[1] = ctxt->h.b8[2]; > > on the following sadb add with setkey: > add 192.168.0.101 192.168.0.110 esp-old 0x10001 -m any -E des-cbc > "12345678" -A keyed-sha1 "12345678123456781234" > > thanks in advance on any hints. > From sgk at troutmask.apl.washington.edu Fri Oct 10 21:30:54 2008 From: sgk at troutmask.apl.washington.edu (Steve Kargl) Date: Fri Oct 10 22:02:55 2008 Subject: HPC with ULE vs 4BSD Message-ID: <20081010213042.GA96822@troutmask.apl.washington.edu> Yes, this is a long email. In working with a colleague to diagnosis poor performance of his MPI code, we've discovered that ULE is drastically inferior to 4BSD in utilizing a system with 2 physical cpus (opteron) and a total of 8 cores. We have observed this problem with the Open MPI implementation of MPI and with the MPICH2 implementation. Note, I am using the exact same hardware and FreeBSD-current code dated Sep 22, 2008. The only difference in the kernel config file is whether ULE or 4BSD is used. Using the following command, % time /OpenMPI/mpiexec -machinefile mf -n 8 ./Test_mpi |& tee sgk.log we have ULE --> 546.99 real 0.02 user 0.03 sys 4BSD -> 218.96 real 0.03 user 0.02 sys where the machinefile simply tells Open MPI to launch 8 jobs on the local node. Test_mpi uses MPI's scatter, gather, and all_to_all functions to transmit various arrays between the 8 jobs. To get meaningful numbers, a number of iterations are done in a tight loop. With ULE, a snapshot of top(1) shows last pid: 33765; load averages: 7.98, 7.51, 5.63 up 10+03:20:30 13:13:56 43 processes: 9 running, 34 sleeping CPU: 68.6% user, 0.0% nice, 18.9% system, 0.0% interrupt, 12.5% idle Mem: 296M Active, 20M Inact, 192M Wired, 1112K Cache, 132M Buf, 31G Free Swap: 4096M Total, 4096M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU COMMAND 33743 kargl 1 118 0 300M 22788K CPU7 7 4:48 100.00% Test_mpi 33747 kargl 1 118 0 300M 22820K CPU3 3 4:43 100.00% Test_mpi 33742 kargl 1 118 0 300M 22692K CPU5 5 4:42 100.00% Test_mpi 33744 kargl 1 117 0 300M 22752K CPU6 6 4:29 100.00% Test_mpi 33748 kargl 1 117 0 300M 22768K CPU2 2 4:31 96.39% Test_mpi 33741 kargl 1 112 0 299M 43628K CPU1 1 4:40 80.08% Test_mpi 33745 kargl 1 113 0 300M 44272K RUN 0 4:27 76.17% Test_mpi 33746 kargl 1 109 0 300M 22740K RUN 0 4:25 57.86% Test_mpi 33749 kargl 1 44 0 8196K 2280K CPU4 4 0:00 0.20% top while with 4BSD, a snapshot of top(1) shows last pid: 1019; load averages: 7.24, 3.05, 1.25 up 0+00:04:40 13:27:09 43 processes: 9 running, 34 sleeping CPU: 45.4% user, 0.0% nice, 54.5% system, 0.1% interrupt, 0.0% idle Mem: 329M Active, 33M Inact, 107M Wired, 104K Cache, 14M Buf, 31G Free Swap: 4096M Total, 4096M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU COMMAND 1012 kargl 1 126 0 300M 44744K CPU6 6 2:16 99.07% Test_mpi 1016 kargl 1 126 0 314M 59256K RUN 4 2:16 99.02% Test_mpi 1011 kargl 1 126 0 300M 44652K CPU5 5 2:16 99.02% Test_mpi 1013 kargl 1 126 0 300M 44680K CPU2 2 2:16 99.02% Test_mpi 1010 kargl 1 126 0 300M 44740K CPU7 7 2:16 99.02% Test_mpi 1009 kargl 1 126 0 299M 43884K CPU0 0 2:16 98.97% Test_mpi 1014 kargl 1 126 0 300M 44664K CPU1 1 2:16 98.97% Test_mpi 1015 kargl 1 126 0 300M 44620K CPU3 3 2:16 98.93% Test_mpi 989 kargl 1 96 0 8196K 2460K CPU4 4 0:00 0.10% top Notice the interesting, or even perhaps odd, scheduling with ULE that results in a 20 second gap between the "fastest" job (4:48) and the "slowest" (4:25). With ULE, 2 Test_mpi jobs are always scheduled on the same core while one core remains idle. Also, note the difference in the reported load averages. Various stats are generated by and collected from executing the MPI program With ULE, the numbers are Procs Array size Kb Iters Function Bandwidth(Mbs) Time(s) 8 800000 3125 100 scatter 12.58386 0.24251367 8 800000 3125 100 all_to_all 17.24503 0.17696444 8 800000 3125 100 gather 14.82058 0.20591355 8 1600000 6250 100 scatter 28.25922 0.21598316 8 1600000 6250 100 all_to_all 1985.74915 0.00307366 8 1600000 6250 100 gather 30.42038 0.20063902 8 2400000 9375 100 scatter 44.65615 0.20501709 8 2400000 9375 100 all_to_all 16.09386 0.56886748 8 2400000 9375 100 gather 44.38801 0.20625555 8 3200000 12500 100 scatter 60.04160 0.20330956 8 3200000 12500 100 all_to_all 2157.10010 0.00565900 8 3200000 12500 100 gather 59.72242 0.20439614 8 4000000 15625 100 scatter 86.65769 0.17608117 8 4000000 15625 100 all_to_all 2081.25195 0.00733154 8 4000000 15625 100 gather 27.47257 0.55541896 8 4800000 18750 100 scatter 33.02306 0.55447768 8 4800000 18750 100 all_to_all 200.09908 0.09150740 8 4800000 18750 100 gather 91.08742 0.20102168 8 5600000 21875 100 scatter 109.82005 0.19452098 8 5600000 21875 100 all_to_all 76.87574 0.27788095 8 5600000 21875 100 gather 41.67106 0.51264128 8 6400000 25000 100 scatter 26.92482 0.90674917 8 6400000 25000 100 all_to_all 64.74528 0.37707868 8 6400000 25000 100 gather 41.29724 0.59117904 and with 4BSD, the numbers are Procs Array size Kb Iters Function Bandwidth(Mbs) Time(s) 8 800000 3125 100 scatter 21.33697 0.14302677 8 800000 3125 100 all_to_all 3941.39624 0.00077428 8 800000 3125 100 gather 24.75520 0.12327747 8 1600000 6250 100 scatter 45.20134 0.13502954 8 1600000 6250 100 all_to_all 1987.94348 0.00307027 8 1600000 6250 100 gather 42.02498 0.14523541 8 2400000 9375 100 scatter 63.03553 0.14523989 8 2400000 9375 100 all_to_all 2015.19580 0.00454312 8 2400000 9375 100 gather 66.72807 0.13720272 8 3200000 12500 100 scatter 91.90541 0.13282169 8 3200000 12500 100 all_to_all 2029.62622 0.00601442 8 3200000 12500 100 gather 87.99693 0.13872112 8 4000000 15625 100 scatter 107.48991 0.14195556 8 4000000 15625 100 all_to_all 1970.66907 0.00774295 8 4000000 15625 100 gather 110.70226 0.13783630 8 4800000 18750 100 scatter 140.39014 0.13042616 8 4800000 18750 100 all_to_all 2401.80054 0.00762367 8 4800000 18750 100 gather 134.60948 0.13602717 8 5600000 21875 100 scatter 152.31958 0.14024661 8 5600000 21875 100 all_to_all 2379.12207 0.00897907 8 5600000 21875 100 gather 154.60051 0.13817745 8 6400000 25000 100 scatter 190.03561 0.12847099 8 6400000 25000 100 all_to_all 2661.36963 0.00917350 8 6400000 25000 100 gather 183.08250 0.13335006 Noting that all communication is over the memory bus, a comparison of the Bandwidth columns suggests that ULE is causing the MPI jobs to stall waiting for data. This has potentially serious negative impact on clusters used for HPC. -- Steve From koitsu at FreeBSD.org Fri Oct 10 22:29:07 2008 From: koitsu at FreeBSD.org (Jeremy Chadwick) Date: Fri Oct 10 22:29:14 2008 Subject: HPC with ULE vs 4BSD In-Reply-To: <20081010213042.GA96822@troutmask.apl.washington.edu> References: <20081010213042.GA96822@troutmask.apl.washington.edu> Message-ID: <20081010222904.GA44873@icarus.home.lan> On Fri, Oct 10, 2008 at 02:30:42PM -0700, Steve Kargl wrote: > Yes, this is a long email. > > In working with a colleague to diagnosis poor performance of > his MPI code, we've discovered that ULE is drastically inferior > to 4BSD in utilizing a system with 2 physical cpus (opteron) and > a total of 8 cores. We have observed this problem with the Open MPI > implementation of MPI and with the MPICH2 implementation. > > Note, I am using the exact same hardware and FreeBSD-current > code dated Sep 22, 2008. The only difference in the kernel > config file is whether ULE or 4BSD is used. > > Using the following command, > > % time /OpenMPI/mpiexec -machinefile mf -n 8 ./Test_mpi |& tee sgk.log > > we have > > ULE --> 546.99 real 0.02 user 0.03 sys > 4BSD -> 218.96 real 0.03 user 0.02 sys > > where the machinefile simply tells Open MPI to launch 8 jobs on the > local node. Test_mpi uses MPI's scatter, gather, and all_to_all > functions to transmit various arrays between the 8 jobs. To get > meaningful numbers, a number of iterations are done in a tight loop. > > With ULE, a snapshot of top(1) shows > > last pid: 33765; load averages: 7.98, 7.51, 5.63 up 10+03:20:30 13:13:56 > 43 processes: 9 running, 34 sleeping > CPU: 68.6% user, 0.0% nice, 18.9% system, 0.0% interrupt, 12.5% idle > Mem: 296M Active, 20M Inact, 192M Wired, 1112K Cache, 132M Buf, 31G Free > Swap: 4096M Total, 4096M Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU COMMAND > 33743 kargl 1 118 0 300M 22788K CPU7 7 4:48 100.00% Test_mpi > 33747 kargl 1 118 0 300M 22820K CPU3 3 4:43 100.00% Test_mpi > 33742 kargl 1 118 0 300M 22692K CPU5 5 4:42 100.00% Test_mpi > 33744 kargl 1 117 0 300M 22752K CPU6 6 4:29 100.00% Test_mpi > 33748 kargl 1 117 0 300M 22768K CPU2 2 4:31 96.39% Test_mpi > 33741 kargl 1 112 0 299M 43628K CPU1 1 4:40 80.08% Test_mpi > 33745 kargl 1 113 0 300M 44272K RUN 0 4:27 76.17% Test_mpi > 33746 kargl 1 109 0 300M 22740K RUN 0 4:25 57.86% Test_mpi > 33749 kargl 1 44 0 8196K 2280K CPU4 4 0:00 0.20% top > > while with 4BSD, a snapshot of top(1) shows > > last pid: 1019; load averages: 7.24, 3.05, 1.25 up 0+00:04:40 13:27:09 > 43 processes: 9 running, 34 sleeping > CPU: 45.4% user, 0.0% nice, 54.5% system, 0.1% interrupt, 0.0% idle > Mem: 329M Active, 33M Inact, 107M Wired, 104K Cache, 14M Buf, 31G Free > Swap: 4096M Total, 4096M Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU COMMAND > 1012 kargl 1 126 0 300M 44744K CPU6 6 2:16 99.07% Test_mpi > 1016 kargl 1 126 0 314M 59256K RUN 4 2:16 99.02% Test_mpi > 1011 kargl 1 126 0 300M 44652K CPU5 5 2:16 99.02% Test_mpi > 1013 kargl 1 126 0 300M 44680K CPU2 2 2:16 99.02% Test_mpi > 1010 kargl 1 126 0 300M 44740K CPU7 7 2:16 99.02% Test_mpi > 1009 kargl 1 126 0 299M 43884K CPU0 0 2:16 98.97% Test_mpi > 1014 kargl 1 126 0 300M 44664K CPU1 1 2:16 98.97% Test_mpi > 1015 kargl 1 126 0 300M 44620K CPU3 3 2:16 98.93% Test_mpi > 989 kargl 1 96 0 8196K 2460K CPU4 4 0:00 0.10% top > > Notice the interesting, or even perhaps odd, scheduling with ULE that results > in a 20 second gap between the "fastest" job (4:48) and the "slowest" (4:25). > With ULE, 2 Test_mpi jobs are always scheduled on the same core while one > core remains idle. Also, note the difference in the reported load averages. > > Various stats are generated by and collected from executing the MPI program > With ULE, the numbers are > > Procs Array size Kb Iters Function Bandwidth(Mbs) Time(s) > 8 800000 3125 100 scatter 12.58386 0.24251367 > 8 800000 3125 100 all_to_all 17.24503 0.17696444 > 8 800000 3125 100 gather 14.82058 0.20591355 > > 8 1600000 6250 100 scatter 28.25922 0.21598316 > 8 1600000 6250 100 all_to_all 1985.74915 0.00307366 > 8 1600000 6250 100 gather 30.42038 0.20063902 > > 8 2400000 9375 100 scatter 44.65615 0.20501709 > 8 2400000 9375 100 all_to_all 16.09386 0.56886748 > 8 2400000 9375 100 gather 44.38801 0.20625555 > > 8 3200000 12500 100 scatter 60.04160 0.20330956 > 8 3200000 12500 100 all_to_all 2157.10010 0.00565900 > 8 3200000 12500 100 gather 59.72242 0.20439614 > > 8 4000000 15625 100 scatter 86.65769 0.17608117 > 8 4000000 15625 100 all_to_all 2081.25195 0.00733154 > 8 4000000 15625 100 gather 27.47257 0.55541896 > > 8 4800000 18750 100 scatter 33.02306 0.55447768 > 8 4800000 18750 100 all_to_all 200.09908 0.09150740 > 8 4800000 18750 100 gather 91.08742 0.20102168 > > 8 5600000 21875 100 scatter 109.82005 0.19452098 > 8 5600000 21875 100 all_to_all 76.87574 0.27788095 > 8 5600000 21875 100 gather 41.67106 0.51264128 > > 8 6400000 25000 100 scatter 26.92482 0.90674917 > 8 6400000 25000 100 all_to_all 64.74528 0.37707868 > 8 6400000 25000 100 gather 41.29724 0.59117904 > > and with 4BSD, the numbers are > > Procs Array size Kb Iters Function Bandwidth(Mbs) Time(s) > 8 800000 3125 100 scatter 21.33697 0.14302677 > 8 800000 3125 100 all_to_all 3941.39624 0.00077428 > 8 800000 3125 100 gather 24.75520 0.12327747 > > 8 1600000 6250 100 scatter 45.20134 0.13502954 > 8 1600000 6250 100 all_to_all 1987.94348 0.00307027 > 8 1600000 6250 100 gather 42.02498 0.14523541 > > 8 2400000 9375 100 scatter 63.03553 0.14523989 > 8 2400000 9375 100 all_to_all 2015.19580 0.00454312 > 8 2400000 9375 100 gather 66.72807 0.13720272 > > 8 3200000 12500 100 scatter 91.90541 0.13282169 > 8 3200000 12500 100 all_to_all 2029.62622 0.00601442 > 8 3200000 12500 100 gather 87.99693 0.13872112 > > 8 4000000 15625 100 scatter 107.48991 0.14195556 > 8 4000000 15625 100 all_to_all 1970.66907 0.00774295 > 8 4000000 15625 100 gather 110.70226 0.13783630 > > 8 4800000 18750 100 scatter 140.39014 0.13042616 > 8 4800000 18750 100 all_to_all 2401.80054 0.00762367 > 8 4800000 18750 100 gather 134.60948 0.13602717 > > 8 5600000 21875 100 scatter 152.31958 0.14024661 > 8 5600000 21875 100 all_to_all 2379.12207 0.00897907 > 8 5600000 21875 100 gather 154.60051 0.13817745 > > 8 6400000 25000 100 scatter 190.03561 0.12847099 > 8 6400000 25000 100 all_to_all 2661.36963 0.00917350 > 8 6400000 25000 100 gather 183.08250 0.13335006 > > Noting that all communication is over the memory bus, a comparison of > the Bandwidth columns suggests that ULE is causing the MPI jobs to stall > waiting for data. This has potentially serious negative impact on > clusters used for HPC. What surprises me is that you didn't CC the individual who wrote ULE: Jeff Roberson. :-) I've CC'd him here. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From danny at cs.huji.ac.il Sat Oct 11 10:35:18 2008 From: danny at cs.huji.ac.il (Danny Braniss) Date: Sat Oct 11 10:35:26 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <20081010122228.355c2c3e@bhuda.mired.org> References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> <20081010144111.GA34609@icarus.home.lan> <20081010112952.52b8209b@bhuda.mired.org> <20081010154249.GA35859@icarus.home.lan> <20081010122228.355c2c3e@bhuda.mired.org> Message-ID: > On Fri, 10 Oct 2008 08:42:49 -0700 > Jeremy Chadwick wrote: > > > On Fri, Oct 10, 2008 at 11:29:52AM -0400, Mike Meyer wrote: > > > On Fri, 10 Oct 2008 07:41:11 -0700 > > > Jeremy Chadwick wrote: > > > > > > > On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: > > > > > Mike Meyer wrote: > > > > >> On Fri, 10 Oct 2008 02:34:28 +0300 > > > > >> yurtesen@ispro.net wrote: > > > > >> > > > > >>> Quoting "Oliver Fromme" : > > > > >>> > > > > >>>> These features are readily available right now on FreeBSD. > > > > >>>> You don't have to code anything. > > > > >>> Well with 2 downsides, > > > > >> > > > > >> Once you actually try and implement these solutions, you'll see that > > > > >> your "downsides" are largely figments of your imagination. > > > > > > > > > > So if it is my imagination, how can I actually convert UFS to ZFS > > > > > easily? Everybody seems to say that this is easy and that is easy. > > > > > > > > It's not that easy. I really don't know why people are telling you it > > > > is. > > > > > > Maybe because it is? Of course, it *does* require a little prior > > > planning, but anyone with more than a few months experience as a > > > sysadmin should be able to deal with it without to much hassle. > > > > > > > Converting some filesystems are easier than others; /home (if you > > > > create one) for example is generally easy: > > > > > > > > 1) ZFS fs is called foo/home, mounted as /mnt > > > > 2) fstat, ensure nothing is using /home -- if something is, shut it > > > > down or kill it > > > > 3) rsync or cpdup /home files to /mnt > > > > 4) umount /home > > > > 5) zfs set mountpoint=/home foo/home > > > > 6) Restart said processes or daemons > > > > > > > > "See! It's like I said! EASY!" You can do this with /var as well. > > > > > > Yup. Of course, if you've done it that way, you're not thinking ahead, > > > because: > > > > > > > Now try /usr. Hope you've got /rescue available, because once /usr/lib > > > > and /usr/libexec disappear, you're in trouble. Good luck doing this in > > > > multi-user, too. > > > > > > Oops. You F'ed up. If you'd done a little planning, you would have > > > realized that / and /usr would be a bit of extra trouble, and planned > > > accordingly. > > > > > > > And finally, the root fs. Whoever says "this is easy" is kidding > > > > themselves; it's a pain. > > > > > > Um, no, it wasn't. Of course, I've been doing this long enough to have > > > a system set up to make this kind of thing easy. My system disk is on > > > a mirror, and I do system upgrades by breaking the mirror and > > > upgrading one disk, making everything work, then putting the mirror > > > back together. And moving to zfs on root is a lot like a system > > > upgrade: > > > > > > 1) Break the mirror (mirrors actually, as I mirrored file systems). > > > 2) Repartition the unused drive into /boot, swap & data > > > 3) Build zfs & /boot according to the instructions on ZFSOnRoot > > > wiki, just copying /boot and / at this point. > > > 4) Boot the zfs disk in single user mode. > > > 5) If 4 fails, boot back to the ufs disk so you're operational while > > > you contemplate what went wrong, then repeat step 3. Otherwise, go > > > on to step 6. > > > 6) Create zfs file systems as appropriate (given that zfs file > > > systems are cheap, and have lots of cool features that ufs > > > file systems don't have, you probably want to create more than > > > you had before, doing thing like putting SQL serves on their > > > own file system with appropriate blocking, etc, but you'll want to > > > have figured all this out before starting step 1). > > > 7) Copy data from the ufs file systems to their new homes, > > > not forgetting to take them out of /etc/fstab. > > > 8) Reboot on the zfs disk. > > > 9) Test until you're happy that everything is working properly, > > > and be prepared to reboot on the ufs disk if something is broken. > > > 10) Reformat the ufs disk to match the zfs one. Gmirror /boot, > > > add the data partition to the zfs pool so it's mirrored, and > > > you should have already been using swap. > > > > > > This is 10 steps to your "easy" 6, but two of the extra steps are > > > testing you didn't include, and 1 of the steps is a failure recovery > > > step that shouldn't be necessary. So - one more step than your easy > > > process. > > > > Of course, the part you seem to be (intentionally?) forgetting: most > > people are not using gmirror. There is no 2nd disk. They have one disk > > with a series of UFS2 filesystems, and they want to upgrade. That's how > > I read Evren's "how do I do this? You say it's easy..." comment, and I > > think his viewpoint is very reasonable. > > Granted, most people don't think about system upgrades when they build > a system, so they wind up having to do extra work. In particular, > Evren is talking about spending thousands of dollars on proprietary > software, not to mention the cost of the server that all this data is > going to flow to, for a backup solution. Compared to that, the cost of > a few spare disks and the work to install them are trivial. > > > > Yeah, this isn't something you do on a whim. On the other hand, it's > > > not something that any competent sysadmin would consider a pain. For a > > > good senior admin, it's a lot easier than doing an OS upgrade from > > > source, which should be the next step up from trivial. > > I guess you have a very different definition of "easy". :-) > > Given that mine is based on years of working with the kinds of backup > solutions that Evren is asking for: ones that an enterprise deploys > for backing up a data center, the answer may well be "yes". > > > The above procedure, in no way shape or form, will be classified as > > "easy" by the user (or even junior sysadmin) community, I can assure you > > of that. > > I never said it would be easy for a user. Then again, your average > user doesn't do backups, and wouldn't know a continuous backup > solution from a credit default swap. We're not talking about ghosting > a disk partition for a backup, we're talking about enterprise-level > backup solutions for data centers. People deploying those kinds of > solutions tend to have multiple senior sysadmins around. > > I wouldn't expect a junior admin to call it easy. At least, not the > first two or three times. If they still have problems with it after > that, they should find a new career path, as they aren't ever going to > advance beyond junior. > > > I'll also throw this in the mix: the fact that we are *expecting* users > > to know how to do this is unreasonable. It's even *more* rude to expect > > Um, is anyone expecting users to do this? I'm not. ZFS is still marked > as "experimental" in FreeBSD. That means that, among other things, > it's not really well-supported by the installer, etc. Nuts, as of > January of this year, there wasn't an operating system on the planet > that would install and boot from ZFS. > > I'm willing to jump through some hoops to get ZFS's advantages. Those > happen to include some things that go a long way to solving Zefren's > problems, so it was suggested as the basis for such (not by me, mind > you). Having done the conversion, and found it easy, I responded when > he asked how hard it was. > > But I'd never recommend this for your average user - which pretty much > excludes anyone contemplating continuous backup solutions. > > > that mid-level or senior SAs have to do > it "the hard way". Why? I'll > > explain: > > > > I'm an SA of 16+ years. I'm quite familiar with PBR/MBR, general disk > > partitioning, sectors vs. blocks, slices, filesystems, and whatever > > else. You want me to do it by hand, say, with bsdlabel -e? Fine, I > > will -- but I will not be happy about it. I have the knowledge, I > > know how to do it, so why must the process continue to be a PITA and > > waste my time? > > Did I ever mention bsdlabel? But in any case, ZFS makes pretty much > *all* that crap obsolete. You still have to deal with getting a boot > loader installed, but after that, you never have to worry about > partitioning, blocks, sectors, or slices again - until you go to an > operating system that doesn't have ZFS. so can Freebsd boot off a ZFS root? in stable? current? ... danny From koitsu at FreeBSD.org Sat Oct 11 10:44:11 2008 From: koitsu at FreeBSD.org (Jeremy Chadwick) Date: Sat Oct 11 10:44:19 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> <20081010144111.GA34609@icarus.home.lan> <20081010112952.52b8209b@bhuda.mired.org> <20081010154249.GA35859@icarus.home.lan> <20081010122228.355c2c3e@bhuda.mired.org> Message-ID: <20081011104409.GA58698@icarus.home.lan> On Sat, Oct 11, 2008 at 12:35:16PM +0200, Danny Braniss wrote: > > On Fri, 10 Oct 2008 08:42:49 -0700 > > Jeremy Chadwick wrote: > > > > > On Fri, Oct 10, 2008 at 11:29:52AM -0400, Mike Meyer wrote: > > > > On Fri, 10 Oct 2008 07:41:11 -0700 > > > > Jeremy Chadwick wrote: > > > > > > > > > On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: > > > > > > Mike Meyer wrote: > > > > > >> On Fri, 10 Oct 2008 02:34:28 +0300 > > > > > >> yurtesen@ispro.net wrote: > > > > > >> > > > > > >>> Quoting "Oliver Fromme" : > > > > > >>> > > > > > >>>> These features are readily available right now on FreeBSD. > > > > > >>>> You don't have to code anything. > > > > > >>> Well with 2 downsides, > > > > > >> > > > > > >> Once you actually try and implement these solutions, you'll see that > > > > > >> your "downsides" are largely figments of your imagination. > > > > > > > > > > > > So if it is my imagination, how can I actually convert UFS to ZFS > > > > > > easily? Everybody seems to say that this is easy and that is easy. > > > > > > > > > > It's not that easy. I really don't know why people are telling you it > > > > > is. > > > > > > > > Maybe because it is? Of course, it *does* require a little prior > > > > planning, but anyone with more than a few months experience as a > > > > sysadmin should be able to deal with it without to much hassle. > > > > > > > > > Converting some filesystems are easier than others; /home (if you > > > > > create one) for example is generally easy: > > > > > > > > > > 1) ZFS fs is called foo/home, mounted as /mnt > > > > > 2) fstat, ensure nothing is using /home -- if something is, shut it > > > > > down or kill it > > > > > 3) rsync or cpdup /home files to /mnt > > > > > 4) umount /home > > > > > 5) zfs set mountpoint=/home foo/home > > > > > 6) Restart said processes or daemons > > > > > > > > > > "See! It's like I said! EASY!" You can do this with /var as well. > > > > > > > > Yup. Of course, if you've done it that way, you're not thinking ahead, > > > > because: > > > > > > > > > Now try /usr. Hope you've got /rescue available, because once /usr/lib > > > > > and /usr/libexec disappear, you're in trouble. Good luck doing this in > > > > > multi-user, too. > > > > > > > > Oops. You F'ed up. If you'd done a little planning, you would have > > > > realized that / and /usr would be a bit of extra trouble, and planned > > > > accordingly. > > > > > > > > > And finally, the root fs. Whoever says "this is easy" is kidding > > > > > themselves; it's a pain. > > > > > > > > Um, no, it wasn't. Of course, I've been doing this long enough to have > > > > a system set up to make this kind of thing easy. My system disk is on > > > > a mirror, and I do system upgrades by breaking the mirror and > > > > upgrading one disk, making everything work, then putting the mirror > > > > back together. And moving to zfs on root is a lot like a system > > > > upgrade: > > > > > > > > 1) Break the mirror (mirrors actually, as I mirrored file systems). > > > > 2) Repartition the unused drive into /boot, swap & data > > > > 3) Build zfs & /boot according to the instructions on ZFSOnRoot > > > > wiki, just copying /boot and / at this point. > > > > 4) Boot the zfs disk in single user mode. > > > > 5) If 4 fails, boot back to the ufs disk so you're operational while > > > > you contemplate what went wrong, then repeat step 3. Otherwise, go > > > > on to step 6. > > > > 6) Create zfs file systems as appropriate (given that zfs file > > > > systems are cheap, and have lots of cool features that ufs > > > > file systems don't have, you probably want to create more than > > > > you had before, doing thing like putting SQL serves on their > > > > own file system with appropriate blocking, etc, but you'll want to > > > > have figured all this out before starting step 1). > > > > 7) Copy data from the ufs file systems to their new homes, > > > > not forgetting to take them out of /etc/fstab. > > > > 8) Reboot on the zfs disk. > > > > 9) Test until you're happy that everything is working properly, > > > > and be prepared to reboot on the ufs disk if something is broken. > > > > 10) Reformat the ufs disk to match the zfs one. Gmirror /boot, > > > > add the data partition to the zfs pool so it's mirrored, and > > > > you should have already been using swap. > > > > > > > > This is 10 steps to your "easy" 6, but two of the extra steps are > > > > testing you didn't include, and 1 of the steps is a failure recovery > > > > step that shouldn't be necessary. So - one more step than your easy > > > > process. > > > > > > Of course, the part you seem to be (intentionally?) forgetting: most > > > people are not using gmirror. There is no 2nd disk. They have one disk > > > with a series of UFS2 filesystems, and they want to upgrade. That's how > > > I read Evren's "how do I do this? You say it's easy..." comment, and I > > > think his viewpoint is very reasonable. > > > > Granted, most people don't think about system upgrades when they build > > a system, so they wind up having to do extra work. In particular, > > Evren is talking about spending thousands of dollars on proprietary > > software, not to mention the cost of the server that all this data is > > going to flow to, for a backup solution. Compared to that, the cost of > > a few spare disks and the work to install them are trivial. > > > > > > Yeah, this isn't something you do on a whim. On the other hand, it's > > > > not something that any competent sysadmin would consider a pain. For a > > > > good senior admin, it's a lot easier than doing an OS upgrade from > > > > source, which should be the next step up from trivial. > > > I guess you have a very different definition of "easy". :-) > > > > Given that mine is based on years of working with the kinds of backup > > solutions that Evren is asking for: ones that an enterprise deploys > > for backing up a data center, the answer may well be "yes". > > > > > The above procedure, in no way shape or form, will be classified as > > > "easy" by the user (or even junior sysadmin) community, I can assure you > > > of that. > > > > I never said it would be easy for a user. Then again, your average > > user doesn't do backups, and wouldn't know a continuous backup > > solution from a credit default swap. We're not talking about ghosting > > a disk partition for a backup, we're talking about enterprise-level > > backup solutions for data centers. People deploying those kinds of > > solutions tend to have multiple senior sysadmins around. > > > > I wouldn't expect a junior admin to call it easy. At least, not the > > first two or three times. If they still have problems with it after > > that, they should find a new career path, as they aren't ever going to > > advance beyond junior. > > > > > I'll also throw this in the mix: the fact that we are *expecting* users > > > to know how to do this is unreasonable. It's even *more* rude to expect > > > > Um, is anyone expecting users to do this? I'm not. ZFS is still marked > > as "experimental" in FreeBSD. That means that, among other things, > > it's not really well-supported by the installer, etc. Nuts, as of > > January of this year, there wasn't an operating system on the planet > > that would install and boot from ZFS. > > > > I'm willing to jump through some hoops to get ZFS's advantages. Those > > happen to include some things that go a long way to solving Zefren's > > problems, so it was suggested as the basis for such (not by me, mind > > you). Having done the conversion, and found it easy, I responded when > > he asked how hard it was. > > > > But I'd never recommend this for your average user - which pretty much > > excludes anyone contemplating continuous backup solutions. > > > > > that mid-level or senior SAs have to do > > it "the hard way". Why? I'll > > > explain: > > > > > > I'm an SA of 16+ years. I'm quite familiar with PBR/MBR, general disk > > > partitioning, sectors vs. blocks, slices, filesystems, and whatever > > > else. You want me to do it by hand, say, with bsdlabel -e? Fine, I > > > will -- but I will not be happy about it. I have the knowledge, I > > > know how to do it, so why must the process continue to be a PITA and > > > waste my time? > > > > Did I ever mention bsdlabel? But in any case, ZFS makes pretty much > > *all* that crap obsolete. You still have to deal with getting a boot > > loader installed, but after that, you never have to worry about > > partitioning, blocks, sectors, or slices again - until you go to an > > operating system that doesn't have ZFS. > > so can Freebsd boot off a ZFS root? in stable? current? ... boot0 doesn't apply here; it cares about what's at sector 0 on the disk, not filesystems. boot2/loader does not speak ZFS -- this is why you need the /boot UFS2 partition. This is an annoyance. For the final "stage/step", vfs.root.mountfrom="zfs:mypool/root" in loader.conf will cause FreeBSD to mount the root filesystem from ZFS. This works fine. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From danny at cs.huji.ac.il Sat Oct 11 11:07:46 2008 From: danny at cs.huji.ac.il (Danny Braniss) Date: Sat Oct 11 11:07:53 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <20081011104409.GA58698@icarus.home.lan> References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> <20081010144111.GA34609@icarus.home.lan> <20081010112952.52b8209b@bhuda.mired.org> <20081010154249.GA35859@icarus.home.lan> <20081010122228.355c2c3e@bhuda.mired.org> <20081011104409.GA58698@icarus.home.lan> Message-ID: > On Sat, Oct 11, 2008 at 12:35:16PM +0200, Danny Braniss wrote: > > > On Fri, 10 Oct 2008 08:42:49 -0700 > > > Jeremy Chadwick wrote: > > > > > > > On Fri, Oct 10, 2008 at 11:29:52AM -0400, Mike Meyer wrote: > > > > > On Fri, 10 Oct 2008 07:41:11 -0700 > > > > > Jeremy Chadwick wrote: > > > > > > > > > > > On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: > > > > > > > Mike Meyer wrote: > > > > > > >> On Fri, 10 Oct 2008 02:34:28 +0300 > > > > > > >> yurtesen@ispro.net wrote: > > > > > > >> > > > > > > >>> Quoting "Oliver Fromme" : > > > > > > >>> > > > > > > >>>> These features are readily available right now on FreeBSD. > > > > > > >>>> You don't have to code anything. > > > > > > >>> Well with 2 downsides, > > > > > > >> > > > > > > >> Once you actually try and implement these solutions, you'll see that > > > > > > >> your "downsides" are largely figments of your imagination. > > > > > > > > > > > > > > So if it is my imagination, how can I actually convert UFS to ZFS > > > > > > > easily? Everybody seems to say that this is easy and that is easy. > > > > > > > > > > > > It's not that easy. I really don't know why people are telling you it > > > > > > is. > > > > > > > > > > Maybe because it is? Of course, it *does* require a little prior > > > > > planning, but anyone with more than a few months experience as a > > > > > sysadmin should be able to deal with it without to much hassle. > > > > > > > > > > > Converting some filesystems are easier than others; /home (if you > > > > > > create one) for example is generally easy: > > > > > > > > > > > > 1) ZFS fs is called foo/home, mounted as /mnt > > > > > > 2) fstat, ensure nothing is using /home -- if something is, shut it > > > > > > down or kill it > > > > > > 3) rsync or cpdup /home files to /mnt > > > > > > 4) umount /home > > > > > > 5) zfs set mountpoint=/home foo/home > > > > > > 6) Restart said processes or daemons > > > > > > > > > > > > "See! It's like I said! EASY!" You can do this with /var as well. > > > > > > > > > > Yup. Of course, if you've done it that way, you're not thinking ahead, > > > > > because: > > > > > > > > > > > Now try /usr. Hope you've got /rescue available, because once /usr/lib > > > > > > and /usr/libexec disappear, you're in trouble. Good luck doing this in > > > > > > multi-user, too. > > > > > > > > > > Oops. You F'ed up. If you'd done a little planning, you would have > > > > > realized that / and /usr would be a bit of extra trouble, and planned > > > > > accordingly. > > > > > > > > > > > And finally, the root fs. Whoever says "this is easy" is kidding > > > > > > themselves; it's a pain. > > > > > > > > > > Um, no, it wasn't. Of course, I've been doing this long enough to have > > > > > a system set up to make this kind of thing easy. My system disk is on > > > > > a mirror, and I do system upgrades by breaking the mirror and > > > > > upgrading one disk, making everything work, then putting the mirror > > > > > back together. And moving to zfs on root is a lot like a system > > > > > upgrade: > > > > > > > > > > 1) Break the mirror (mirrors actually, as I mirrored file systems). > > > > > 2) Repartition the unused drive into /boot, swap & data > > > > > 3) Build zfs & /boot according to the instructions on ZFSOnRoot > > > > > wiki, just copying /boot and / at this point. > > > > > 4) Boot the zfs disk in single user mode. > > > > > 5) If 4 fails, boot back to the ufs disk so you're operational while > > > > > you contemplate what went wrong, then repeat step 3. Otherwise, go > > > > > on to step 6. > > > > > 6) Create zfs file systems as appropriate (given that zfs file > > > > > systems are cheap, and have lots of cool features that ufs > > > > > file systems don't have, you probably want to create more than > > > > > you had before, doing thing like putting SQL serves on their > > > > > own file system with appropriate blocking, etc, but you'll want to > > > > > have figured all this out before starting step 1). > > > > > 7) Copy data from the ufs file systems to their new homes, > > > > > not forgetting to take them out of /etc/fstab. > > > > > 8) Reboot on the zfs disk. > > > > > 9) Test until you're happy that everything is working properly, > > > > > and be prepared to reboot on the ufs disk if something is broken. > > > > > 10) Reformat the ufs disk to match the zfs one. Gmirror /boot, > > > > > add the data partition to the zfs pool so it's mirrored, and > > > > > you should have already been using swap. > > > > > > > > > > This is 10 steps to your "easy" 6, but two of the extra steps are > > > > > testing you didn't include, and 1 of the steps is a failure recovery > > > > > step that shouldn't be necessary. So - one more step than your easy > > > > > process. > > > > > > > > Of course, the part you seem to be (intentionally?) forgetting: most > > > > people are not using gmirror. There is no 2nd disk. They have one disk > > > > with a series of UFS2 filesystems, and they want to upgrade. That's how > > > > I read Evren's "how do I do this? You say it's easy..." comment, and I > > > > think his viewpoint is very reasonable. > > > > > > Granted, most people don't think about system upgrades when they build > > > a system, so they wind up having to do extra work. In particular, > > > Evren is talking about spending thousands of dollars on proprietary > > > software, not to mention the cost of the server that all this data is > > > going to flow to, for a backup solution. Compared to that, the cost of > > > a few spare disks and the work to install them are trivial. > > > > > > > > Yeah, this isn't something you do on a whim. On the other hand, it's > > > > > not something that any competent sysadmin would consider a pain. For a > > > > > good senior admin, it's a lot easier than doing an OS upgrade from > > > > > source, which should be the next step up from trivial. > > > > I guess you have a very different definition of "easy". :-) > > > > > > Given that mine is based on years of working with the kinds of backup > > > solutions that Evren is asking for: ones that an enterprise deploys > > > for backing up a data center, the answer may well be "yes". > > > > > > > The above procedure, in no way shape or form, will be classified as > > > > "easy" by the user (or even junior sysadmin) community, I can assure you > > > > of that. > > > > > > I never said it would be easy for a user. Then again, your average > > > user doesn't do backups, and wouldn't know a continuous backup > > > solution from a credit default swap. We're not talking about ghosting > > > a disk partition for a backup, we're talking about enterprise-level > > > backup solutions for data centers. People deploying those kinds of > > > solutions tend to have multiple senior sysadmins around. > > > > > > I wouldn't expect a junior admin to call it easy. At least, not the > > > first two or three times. If they still have problems with it after > > > that, they should find a new career path, as they aren't ever going to > > > advance beyond junior. > > > > > > > I'll also throw this in the mix: the fact that we are *expecting* users > > > > to know how to do this is unreasonable. It's even *more* rude to expect > > > > > > Um, is anyone expecting users to do this? I'm not. ZFS is still marked > > > as "experimental" in FreeBSD. That means that, among other things, > > > it's not really well-supported by the installer, etc. Nuts, as of > > > January of this year, there wasn't an operating system on the planet > > > that would install and boot from ZFS. > > > > > > I'm willing to jump through some hoops to get ZFS's advantages. Those > > > happen to include some things that go a long way to solving Zefren's > > > problems, so it was suggested as the basis for such (not by me, mind > > > you). Having done the conversion, and found it easy, I responded when > > > he asked how hard it was. > > > > > > But I'd never recommend this for your average user - which pretty much > > > excludes anyone contemplating continuous backup solutions. > > > > > > > that mid-level or senior SAs have to do > > > it "the hard way". Why? I'll > > > > explain: > > > > > > > > I'm an SA of 16+ years. I'm quite familiar with PBR/MBR, general disk > > > > partitioning, sectors vs. blocks, slices, filesystems, and whatever > > > > else. You want me to do it by hand, say, with bsdlabel -e? Fine, I > > > > will -- but I will not be happy about it. I have the knowledge, I > > > > know how to do it, so why must the process continue to be a PITA and > > > > waste my time? > > > > > > Did I ever mention bsdlabel? But in any case, ZFS makes pretty much > > > *all* that crap obsolete. You still have to deal with getting a boot > > > loader installed, but after that, you never have to worry about > > > partitioning, blocks, sectors, or slices again - until you go to an > > > operating system that doesn't have ZFS. > > > > so can Freebsd boot off a ZFS root? in stable? current? ... > > boot0 doesn't apply here; it cares about what's at sector 0 on the > disk, not filesystems. > > boot2/loader does not speak ZFS -- this is why you need the /boot UFS2 > partition. This is an annoyance. > > For the final "stage/step", vfs.root.mountfrom="zfs:mypool/root" in > loader.conf will cause FreeBSD to mount the root filesystem from ZFS. > This works fine. so the answer is: yes, if you have only one disk. no, if you have ZFS over many disks because I see no advantage in the springboard solution where ZFS is used to cover several disks. I'm asking, because I want to deploy some zfs fileservers soon, and so far the solution is either PXE boot, or keep one disk UFS (or boot off a USB) Today's /(root+usr) is somewhere between .5 to 1Gb(kernel+debug+src), and is readonly, so having 1 disk UFS seems to be a pitty. danny From koitsu at FreeBSD.org Sat Oct 11 11:24:34 2008 From: koitsu at FreeBSD.org (Jeremy Chadwick) Date: Sat Oct 11 11:24:41 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: References: <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> <20081010144111.GA34609@icarus.home.lan> <20081010112952.52b8209b@bhuda.mired.org> <20081010154249.GA35859@icarus.home.lan> <20081010122228.355c2c3e@bhuda.mired.org> <20081011104409.GA58698@icarus.home.lan> Message-ID: <20081011112431.GA60924@icarus.home.lan> On Sat, Oct 11, 2008 at 01:07:44PM +0200, Danny Braniss wrote: > > On Sat, Oct 11, 2008 at 12:35:16PM +0200, Danny Braniss wrote: > > > > On Fri, 10 Oct 2008 08:42:49 -0700 > > > > Jeremy Chadwick wrote: > > > > > > > > > On Fri, Oct 10, 2008 at 11:29:52AM -0400, Mike Meyer wrote: > > > > > > On Fri, 10 Oct 2008 07:41:11 -0700 > > > > > > Jeremy Chadwick wrote: > > > > > > > > > > > > > On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: > > > > > > > > Mike Meyer wrote: > > > > > > > >> On Fri, 10 Oct 2008 02:34:28 +0300 > > > > > > > >> yurtesen@ispro.net wrote: > > > > > > > >> > > > > > > > >>> Quoting "Oliver Fromme" : > > > > > > > >>> > > > > > > > >>>> These features are readily available right now on FreeBSD. > > > > > > > >>>> You don't have to code anything. > > > > > > > >>> Well with 2 downsides, > > > > > > > >> > > > > > > > >> Once you actually try and implement these solutions, you'll see that > > > > > > > >> your "downsides" are largely figments of your imagination. > > > > > > > > > > > > > > > > So if it is my imagination, how can I actually convert UFS to ZFS > > > > > > > > easily? Everybody seems to say that this is easy and that is easy. > > > > > > > > > > > > > > It's not that easy. I really don't know why people are telling you it > > > > > > > is. > > > > > > > > > > > > Maybe because it is? Of course, it *does* require a little prior > > > > > > planning, but anyone with more than a few months experience as a > > > > > > sysadmin should be able to deal with it without to much hassle. > > > > > > > > > > > > > Converting some filesystems are easier than others; /home (if you > > > > > > > create one) for example is generally easy: > > > > > > > > > > > > > > 1) ZFS fs is called foo/home, mounted as /mnt > > > > > > > 2) fstat, ensure nothing is using /home -- if something is, shut it > > > > > > > down or kill it > > > > > > > 3) rsync or cpdup /home files to /mnt > > > > > > > 4) umount /home > > > > > > > 5) zfs set mountpoint=/home foo/home > > > > > > > 6) Restart said processes or daemons > > > > > > > > > > > > > > "See! It's like I said! EASY!" You can do this with /var as well. > > > > > > > > > > > > Yup. Of course, if you've done it that way, you're not thinking ahead, > > > > > > because: > > > > > > > > > > > > > Now try /usr. Hope you've got /rescue available, because once /usr/lib > > > > > > > and /usr/libexec disappear, you're in trouble. Good luck doing this in > > > > > > > multi-user, too. > > > > > > > > > > > > Oops. You F'ed up. If you'd done a little planning, you would have > > > > > > realized that / and /usr would be a bit of extra trouble, and planned > > > > > > accordingly. > > > > > > > > > > > > > And finally, the root fs. Whoever says "this is easy" is kidding > > > > > > > themselves; it's a pain. > > > > > > > > > > > > Um, no, it wasn't. Of course, I've been doing this long enough to have > > > > > > a system set up to make this kind of thing easy. My system disk is on > > > > > > a mirror, and I do system upgrades by breaking the mirror and > > > > > > upgrading one disk, making everything work, then putting the mirror > > > > > > back together. And moving to zfs on root is a lot like a system > > > > > > upgrade: > > > > > > > > > > > > 1) Break the mirror (mirrors actually, as I mirrored file systems). > > > > > > 2) Repartition the unused drive into /boot, swap & data > > > > > > 3) Build zfs & /boot according to the instructions on ZFSOnRoot > > > > > > wiki, just copying /boot and / at this point. > > > > > > 4) Boot the zfs disk in single user mode. > > > > > > 5) If 4 fails, boot back to the ufs disk so you're operational while > > > > > > you contemplate what went wrong, then repeat step 3. Otherwise, go > > > > > > on to step 6. > > > > > > 6) Create zfs file systems as appropriate (given that zfs file > > > > > > systems are cheap, and have lots of cool features that ufs > > > > > > file systems don't have, you probably want to create more than > > > > > > you had before, doing thing like putting SQL serves on their > > > > > > own file system with appropriate blocking, etc, but you'll want to > > > > > > have figured all this out before starting step 1). > > > > > > 7) Copy data from the ufs file systems to their new homes, > > > > > > not forgetting to take them out of /etc/fstab. > > > > > > 8) Reboot on the zfs disk. > > > > > > 9) Test until you're happy that everything is working properly, > > > > > > and be prepared to reboot on the ufs disk if something is broken. > > > > > > 10) Reformat the ufs disk to match the zfs one. Gmirror /boot, > > > > > > add the data partition to the zfs pool so it's mirrored, and > > > > > > you should have already been using swap. > > > > > > > > > > > > This is 10 steps to your "easy" 6, but two of the extra steps are > > > > > > testing you didn't include, and 1 of the steps is a failure recovery > > > > > > step that shouldn't be necessary. So - one more step than your easy > > > > > > process. > > > > > > > > > > Of course, the part you seem to be (intentionally?) forgetting: most > > > > > people are not using gmirror. There is no 2nd disk. They have one disk > > > > > with a series of UFS2 filesystems, and they want to upgrade. That's how > > > > > I read Evren's "how do I do this? You say it's easy..." comment, and I > > > > > think his viewpoint is very reasonable. > > > > > > > > Granted, most people don't think about system upgrades when they build > > > > a system, so they wind up having to do extra work. In particular, > > > > Evren is talking about spending thousands of dollars on proprietary > > > > software, not to mention the cost of the server that all this data is > > > > going to flow to, for a backup solution. Compared to that, the cost of > > > > a few spare disks and the work to install them are trivial. > > > > > > > > > > Yeah, this isn't something you do on a whim. On the other hand, it's > > > > > > not something that any competent sysadmin would consider a pain. For a > > > > > > good senior admin, it's a lot easier than doing an OS upgrade from > > > > > > source, which should be the next step up from trivial. > > > > > I guess you have a very different definition of "easy". :-) > > > > > > > > Given that mine is based on years of working with the kinds of backup > > > > solutions that Evren is asking for: ones that an enterprise deploys > > > > for backing up a data center, the answer may well be "yes". > > > > > > > > > The above procedure, in no way shape or form, will be classified as > > > > > "easy" by the user (or even junior sysadmin) community, I can assure you > > > > > of that. > > > > > > > > I never said it would be easy for a user. Then again, your average > > > > user doesn't do backups, and wouldn't know a continuous backup > > > > solution from a credit default swap. We're not talking about ghosting > > > > a disk partition for a backup, we're talking about enterprise-level > > > > backup solutions for data centers. People deploying those kinds of > > > > solutions tend to have multiple senior sysadmins around. > > > > > > > > I wouldn't expect a junior admin to call it easy. At least, not the > > > > first two or three times. If they still have problems with it after > > > > that, they should find a new career path, as they aren't ever going to > > > > advance beyond junior. > > > > > > > > > I'll also throw this in the mix: the fact that we are *expecting* users > > > > > to know how to do this is unreasonable. It's even *more* rude to expect > > > > > > > > Um, is anyone expecting users to do this? I'm not. ZFS is still marked > > > > as "experimental" in FreeBSD. That means that, among other things, > > > > it's not really well-supported by the installer, etc. Nuts, as of > > > > January of this year, there wasn't an operating system on the planet > > > > that would install and boot from ZFS. > > > > > > > > I'm willing to jump through some hoops to get ZFS's advantages. Those > > > > happen to include some things that go a long way to solving Zefren's > > > > problems, so it was suggested as the basis for such (not by me, mind > > > > you). Having done the conversion, and found it easy, I responded when > > > > he asked how hard it was. > > > > > > > > But I'd never recommend this for your average user - which pretty much > > > > excludes anyone contemplating continuous backup solutions. > > > > > > > > > that mid-level or senior SAs have to do > > > > it "the hard way". Why? I'll > > > > > explain: > > > > > > > > > > I'm an SA of 16+ years. I'm quite familiar with PBR/MBR, general disk > > > > > partitioning, sectors vs. blocks, slices, filesystems, and whatever > > > > > else. You want me to do it by hand, say, with bsdlabel -e? Fine, I > > > > > will -- but I will not be happy about it. I have the knowledge, I > > > > > know how to do it, so why must the process continue to be a PITA and > > > > > waste my time? > > > > > > > > Did I ever mention bsdlabel? But in any case, ZFS makes pretty much > > > > *all* that crap obsolete. You still have to deal with getting a boot > > > > loader installed, but after that, you never have to worry about > > > > partitioning, blocks, sectors, or slices again - until you go to an > > > > operating system that doesn't have ZFS. > > > > > > so can Freebsd boot off a ZFS root? in stable? current? ... > > > > boot0 doesn't apply here; it cares about what's at sector 0 on the > > disk, not filesystems. > > > > boot2/loader does not speak ZFS -- this is why you need the /boot UFS2 > > partition. This is an annoyance. > > > > For the final "stage/step", vfs.root.mountfrom="zfs:mypool/root" in > > loader.conf will cause FreeBSD to mount the root filesystem from ZFS. > > This works fine. > > so the answer is: > yes, if you have only one disk. > no, if you have ZFS over many disks > > because I see no advantage in the springboard solution where ZFS is used to > cover several disks. > > I'm asking, because I want to deploy some zfs fileservers soon, and so > far the solution is either PXE boot, or keep one disk UFS (or boot off a USB) > Today's /(root+usr) is somewhere between .5 to 1Gb(kernel+debug+src), > and is readonly, so having 1 disk UFS seems to be a pitty. Hold on a minute. "One disk" has nothing to do with the filesystem. You asked if FreeBSD could boot off of a specific filesystem, and I answered that -- I didn't state anything about disk counts. Now you're changing the focus. :-) I'm pretty sure FreeBSD can boot off of gmirror setups (see above, boot2/loader should work off of gmirror), which means >1 disk. You do not have to gmirror the entire disk, you can gmirror just a slice (AFAIK). I think (hope?) you can use the "remaining" (e.g. non-UFS/non-gmirror) part of the 2nd disk for ZFS as well, otherwise the space would go to waste. The "Root on ZFS configuration" FreeBSD ZFS Wiki page seems to imply you can. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From edwin at mavetju.org Sat Oct 11 12:02:51 2008 From: edwin at mavetju.org (Edwin Groothuis) Date: Sat Oct 11 12:03:15 2008 Subject: VirtualBox looks for FreeBSD developer In-Reply-To: <20081010174204.GB90757@hades.panopticon> References: <48C52718.5080807@sun.com> <48C8F051.7060107@sun.com> <20080919082719.GH676@e.0x20.net> <20081010174204.GB90757@hades.panopticon> Message-ID: <20081011114557.GA71472@mavetju.org> On Fri, Oct 10, 2008 at 09:42:04PM +0400, Dmitry Marakasov wrote: > Little time ago I was misleaded by the certain people and got an > idea that VirtualBox actually works on FreeBSD, so I've made a draft > port for it. It doesn't actually work, but since I've spent several > hours hacking it and made bunch of (likely) useful patches, here > it is, feel free to use it for any purpose. I hope someone of kernel > hackers will make it work actually ;) Have a talk with bms@ about it, he had some interesting working code too. Edwin -- Edwin Groothuis Website: http://www.mavetju.org/ edwin@mavetju.org Weblog: http://www.mavetju.org/weblog/ From dfr at rabson.org Sat Oct 11 12:08:02 2008 From: dfr at rabson.org (Doug Rabson) Date: Sat Oct 11 12:08:10 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> <20081010144111.GA34609@icarus.home.lan> <20081010112952.52b8209b@bhuda.mired.org> <20081010154249.GA35859@icarus.home.lan> <20081010122228.355c2c3e@bhuda.mired.org> <20081011104409.GA58698@icarus.home.lan> Message-ID: <12DB2D13-1F6F-4CA5-B583-CCDD678419FF@rabson.org> On 11 Oct 2008, at 12:07, Danny Braniss wrote: >> On Sat, Oct 11, 2008 at 12:35:16PM +0200, Danny Braniss wrote: >>>> On Fri, 10 Oct 2008 08:42:49 -0700 >>>> Jeremy Chadwick wrote: >>>> >>>>> On Fri, Oct 10, 2008 at 11:29:52AM -0400, Mike Meyer wrote: >>>>>> On Fri, 10 Oct 2008 07:41:11 -0700 >>>>>> Jeremy Chadwick wrote: >>>>>> >>>>>>> On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: >>>>>>>> Mike Meyer wrote: >>>>>>>>> On Fri, 10 Oct 2008 02:34:28 +0300 >>>>>>>>> yurtesen@ispro.net wrote: >>>>>>>>> >>>>>>>>>> Quoting "Oliver Fromme" : >>>>>>>>>> >>>>>>>>>>> These features are readily available right now on FreeBSD. >>>>>>>>>>> You don't have to code anything. >>>>>>>>>> Well with 2 downsides, >>>>>>>>> >>>>>>>>> Once you actually try and implement these solutions, you'll >>>>>>>>> see that >>>>>>>>> your "downsides" are largely figments of your imagination. >>>>>>>> >>>>>>>> So if it is my imagination, how can I actually convert UFS to >>>>>>>> ZFS >>>>>>>> easily? Everybody seems to say that this is easy and that is >>>>>>>> easy. >>>>>>> >>>>>>> It's not that easy. I really don't know why people are >>>>>>> telling you it >>>>>>> is. >>>>>> >>>>>> Maybe because it is? Of course, it *does* require a little prior >>>>>> planning, but anyone with more than a few months experience as a >>>>>> sysadmin should be able to deal with it without to much hassle. >>>>>> >>>>>>> Converting some filesystems are easier than others; /home (if >>>>>>> you >>>>>>> create one) for example is generally easy: >>>>>>> >>>>>>> 1) ZFS fs is called foo/home, mounted as /mnt >>>>>>> 2) fstat, ensure nothing is using /home -- if something is, >>>>>>> shut it >>>>>>> down or kill it >>>>>>> 3) rsync or cpdup /home files to /mnt >>>>>>> 4) umount /home >>>>>>> 5) zfs set mountpoint=/home foo/home >>>>>>> 6) Restart said processes or daemons >>>>>>> >>>>>>> "See! It's like I said! EASY!" You can do this with /var as >>>>>>> well. >>>>>> >>>>>> Yup. Of course, if you've done it that way, you're not thinking >>>>>> ahead, >>>>>> because: >>>>>> >>>>>>> Now try /usr. Hope you've got /rescue available, because >>>>>>> once /usr/lib >>>>>>> and /usr/libexec disappear, you're in trouble. Good luck >>>>>>> doing this in >>>>>>> multi-user, too. >>>>>> >>>>>> Oops. You F'ed up. If you'd done a little planning, you would >>>>>> have >>>>>> realized that / and /usr would be a bit of extra trouble, and >>>>>> planned >>>>>> accordingly. >>>>>> >>>>>>> And finally, the root fs. Whoever says "this is easy" is >>>>>>> kidding >>>>>>> themselves; it's a pain. >>>>>> >>>>>> Um, no, it wasn't. Of course, I've been doing this long enough >>>>>> to have >>>>>> a system set up to make this kind of thing easy. My system disk >>>>>> is on >>>>>> a mirror, and I do system upgrades by breaking the mirror and >>>>>> upgrading one disk, making everything work, then putting the >>>>>> mirror >>>>>> back together. And moving to zfs on root is a lot like a system >>>>>> upgrade: >>>>>> >>>>>> 1) Break the mirror (mirrors actually, as I mirrored file >>>>>> systems). >>>>>> 2) Repartition the unused drive into /boot, swap & data >>>>>> 3) Build zfs & /boot according to the instructions on ZFSOnRoot >>>>>> wiki, just copying /boot and / at this point. >>>>>> 4) Boot the zfs disk in single user mode. >>>>>> 5) If 4 fails, boot back to the ufs disk so you're operational >>>>>> while >>>>>> you contemplate what went wrong, then repeat step 3. >>>>>> Otherwise, go >>>>>> on to step 6. >>>>>> 6) Create zfs file systems as appropriate (given that zfs file >>>>>> systems are cheap, and have lots of cool features that ufs >>>>>> file systems don't have, you probably want to create more than >>>>>> you had before, doing thing like putting SQL serves on their >>>>>> own file system with appropriate blocking, etc, but you'll >>>>>> want to >>>>>> have figured all this out before starting step 1). >>>>>> 7) Copy data from the ufs file systems to their new homes, >>>>>> not forgetting to take them out of /etc/fstab. >>>>>> 8) Reboot on the zfs disk. >>>>>> 9) Test until you're happy that everything is working properly, >>>>>> and be prepared to reboot on the ufs disk if something is >>>>>> broken. >>>>>> 10) Reformat the ufs disk to match the zfs one. Gmirror /boot, >>>>>> add the data partition to the zfs pool so it's mirrored, and >>>>>> you should have already been using swap. >>>>>> >>>>>> This is 10 steps to your "easy" 6, but two of the extra steps are >>>>>> testing you didn't include, and 1 of the steps is a failure >>>>>> recovery >>>>>> step that shouldn't be necessary. So - one more step than your >>>>>> easy >>>>>> process. >>>>> >>>>> Of course, the part you seem to be (intentionally?) forgetting: >>>>> most >>>>> people are not using gmirror. There is no 2nd disk. They have >>>>> one disk >>>>> with a series of UFS2 filesystems, and they want to upgrade. >>>>> That's how >>>>> I read Evren's "how do I do this? You say it's easy..." comment, >>>>> and I >>>>> think his viewpoint is very reasonable. >>>> >>>> Granted, most people don't think about system upgrades when they >>>> build >>>> a system, so they wind up having to do extra work. In particular, >>>> Evren is talking about spending thousands of dollars on proprietary >>>> software, not to mention the cost of the server that all this >>>> data is >>>> going to flow to, for a backup solution. Compared to that, the >>>> cost of >>>> a few spare disks and the work to install them are trivial. >>>> >>>>>> Yeah, this isn't something you do on a whim. On the other hand, >>>>>> it's >>>>>> not something that any competent sysadmin would consider a >>>>>> pain. For a >>>>>> good senior admin, it's a lot easier than doing an OS upgrade >>>>>> from >>>>>> source, which should be the next step up from trivial. >>>>> I guess you have a very different definition of "easy". :-) >>>> >>>> Given that mine is based on years of working with the kinds of >>>> backup >>>> solutions that Evren is asking for: ones that an enterprise deploys >>>> for backing up a data center, the answer may well be "yes". >>>> >>>>> The above procedure, in no way shape or form, will be classified >>>>> as >>>>> "easy" by the user (or even junior sysadmin) community, I can >>>>> assure you >>>>> of that. >>>> >>>> I never said it would be easy for a user. Then again, your average >>>> user doesn't do backups, and wouldn't know a continuous backup >>>> solution from a credit default swap. We're not talking about >>>> ghosting >>>> a disk partition for a backup, we're talking about enterprise-level >>>> backup solutions for data centers. People deploying those kinds of >>>> solutions tend to have multiple senior sysadmins around. >>>> >>>> I wouldn't expect a junior admin to call it easy. At least, not the >>>> first two or three times. If they still have problems with it after >>>> that, they should find a new career path, as they aren't ever >>>> going to >>>> advance beyond junior. >>>> >>>>> I'll also throw this in the mix: the fact that we are >>>>> *expecting* users >>>>> to know how to do this is unreasonable. It's even *more* rude >>>>> to expect >>>> >>>> Um, is anyone expecting users to do this? I'm not. ZFS is still >>>> marked >>>> as "experimental" in FreeBSD. That means that, among other things, >>>> it's not really well-supported by the installer, etc. Nuts, as of >>>> January of this year, there wasn't an operating system on the >>>> planet >>>> that would install and boot from ZFS. >>>> >>>> I'm willing to jump through some hoops to get ZFS's advantages. >>>> Those >>>> happen to include some things that go a long way to solving >>>> Zefren's >>>> problems, so it was suggested as the basis for such (not by me, >>>> mind >>>> you). Having done the conversion, and found it easy, I responded >>>> when >>>> he asked how hard it was. >>>> >>>> But I'd never recommend this for your average user - which pretty >>>> much >>>> excludes anyone contemplating continuous backup solutions. >>>> >>>>> that mid-level or senior SAs have to do >>>> it "the hard way". Why? I'll >>>>> explain: >>>>> >>>>> I'm an SA of 16+ years. I'm quite familiar with PBR/MBR, >>>>> general disk >>>>> partitioning, sectors vs. blocks, slices, filesystems, and >>>>> whatever >>>>> else. You want me to do it by hand, say, with bsdlabel -e? >>>>> Fine, I >>>>> will -- but I will not be happy about it. I have the knowledge, I >>>>> know how to do it, so why must the process continue to be a PITA >>>>> and >>>>> waste my time? >>>> >>>> Did I ever mention bsdlabel? But in any case, ZFS makes pretty much >>>> *all* that crap obsolete. You still have to deal with getting a >>>> boot >>>> loader installed, but after that, you never have to worry about >>>> partitioning, blocks, sectors, or slices again - until you go to an >>>> operating system that doesn't have ZFS. >>> >>> so can Freebsd boot off a ZFS root? in stable? current? ... >> >> boot0 doesn't apply here; it cares about what's at sector 0 on the >> disk, not filesystems. >> >> boot2/loader does not speak ZFS -- this is why you need the /boot >> UFS2 >> partition. This is an annoyance. >> >> For the final "stage/step", vfs.root.mountfrom="zfs:mypool/root" in >> loader.conf will cause FreeBSD to mount the root filesystem from ZFS. >> This works fine. > > so the answer is: > yes, if you have only one disk. > no, if you have ZFS over many disks > > because I see no advantage in the springboard solution where ZFS is > used to > cover several disks. > > I'm asking, because I want to deploy some zfs fileservers soon, and so > far the solution is either PXE boot, or keep one disk UFS (or boot > off a USB) > Today's /(root+usr) is somewhere between .5 to 1Gb(kernel+debug+src), > and is readonly, so having 1 disk UFS seems to be a pitty. ZFS boot is coming. Currently its part of pjd's perforce branch and supports disks, mirrors and collections of disks or mirrors with the only restriction being that enough drives in the pool must be accessible from the bios (i.e. at least one element of a mirror must be seen by the bios). Currently we don't support booting from raidz or raidz2 pools but there is no fundamental reason why that can't be added - someone just needs to implement the code to understand the raidz layout. From dillon at apollo.backplane.com Sat Oct 11 12:30:35 2008 From: dillon at apollo.backplane.com (Matthew Dillon) Date: Sat Oct 11 12:30:42 2008 Subject: continuous backup solution for FreeBSD References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> <20081010144111.GA34609@icarus.home.lan> <20081010112952.52b8209b@bhuda.mired.org> <20081010154249.GA35859@icarus.home.lan> <20081010122228.355c2c3e@bhuda.mired.org> <20081011104409.GA58698@icarus.home.lan> Message-ID: <200810111230.m9BCUMxp057391@apollo.backplane.com> :> boot2/loader does not speak ZFS -- this is why you need the /boot UFS2 :> partition. This is an annoyance. :> :> For the final "stage/step", vfs.root.mountfrom="zfs:mypool/root" in :> loader.conf will cause FreeBSD to mount the root filesystem from ZFS. :> This works fine. : :so the answer is: : yes, if you have only one disk. : no, if you have ZFS over many disks : :because I see no advantage in the springboard solution where ZFS is used to :cover several disks. : :I'm asking, because I want to deploy some zfs fileservers soon, and so :far the solution is either PXE boot, or keep one disk UFS (or boot off a USB) :Today's /(root+usr) is somewhere between .5 to 1Gb(kernel+debug+src), :and is readonly, so having 1 disk UFS seems to be a pitty. : :danny I think it is is perfectly acceptable to have a /boot + ZFS style solution, where /boot is a small ~256M UFS filesystem that the system actually boots from (containing only the kernel, modules, loader.conf, etc), and ZFS is the root filesystem. In a running system /boot would be mounted under the ZFS root. All I needed was a line in /boot/loader.conf to tell the kernel where the root FS was. In my case, I pointed it at HAMMER. vfs.root.mountfrom="hammer:ad0s1d" This gives you the flexibility of being able to have as complex a root FS as you want. Filesystem 1K-blocks Used Avail Capacity Mounted on HAMMER_ROOT 36388864 9789440 26599424 27% / /dev/ad0s1a 257998 100074 137286 42% /boot /pfs/@@-1:00001 36388864 9789440 26599424 27% /usr /pfs/@@-1:00003 36388864 9789440 26599424 27% /var /pfs/@@-1:00006 36388864 9789440 26599424 27% /tmp /pfs/@@-1:00007 36388864 9789440 26599424 27% /home /pfs/@@-1:00005 36388864 9789440 26599424 27% /var/tmp /pfs/@@-1:00002 36388864 9789440 26599424 27% /usr/obj /pfs/@@-1:00004 36388864 9789440 26599424 27% /var/crash procfs 4 4 0 100% /proc The /boot is small enough that it can be dealt with numerous ways, including simple duplication if you have multiple disks (have a adXs1a on two drives). And if you were really that worried you could put /boot on a SSD. Frankly, anything that has approximately the same MTBF as the motherboard itself is suitable, there's really no point trying to make /boot disk-redundant when the motherboard and memory aren't redundant. If you have more then one HD connected to the system, and you want boot redundancy, then you also likely have the $$ to purchase a tiny SSD for your /boot. The big problem trying to boot from a completely generic FS setup is that it tends to severely limit your options. You might want to have more flexibility in your root filesystem that you could otherwise accomodate if /boot were integrated into it. -Matt Matthew Dillon From danny at cs.huji.ac.il Sat Oct 11 13:28:04 2008 From: danny at cs.huji.ac.il (Danny Braniss) Date: Sat Oct 11 13:28:11 2008 Subject: ZFS boot Message-ID: > > > > so can Freebsd boot off a ZFS root? in stable? current? ... > > > > > > boot0 doesn't apply here; it cares about what's at sector 0 on the > > > disk, not filesystems. > > > > > > boot2/loader does not speak ZFS -- this is why you need the /boot UFS2 > > > partition. This is an annoyance. > > > > > > For the final "stage/step", vfs.root.mountfrom="zfs:mypool/root" in > > > loader.conf will cause FreeBSD to mount the root filesystem from ZFS. > > > This works fine. > > > > so the answer is: > > yes, if you have only one disk. > > no, if you have ZFS over many disks > > > > because I see no advantage in the springboard solution where ZFS is used to > > cover several disks. > > > > I'm asking, because I want to deploy some zfs fileservers soon, and so > > far the solution is either PXE boot, or keep one disk UFS (or boot off a USB) > > Today's /(root+usr) is somewhere between .5 to 1Gb(kernel+debug+src), > > and is readonly, so having 1 disk UFS seems to be a pitty. > > Hold on a minute. "One disk" has nothing to do with the filesystem. > You asked if FreeBSD could boot off of a specific filesystem, and I > answered that -- I didn't state anything about disk counts. Now you're > changing the focus. :-) > not intentionaly, but once you mention boot0/2, bsdlabel, slice/partition ... /OT Initially, I was not thrilled with ZFS, but once you cross the few hundred gigabyte filesystems UFS is impractical, and though old sysadmins will have to be re-educated (zfs catch-all-commands, instead of mount/fsck/export/blah...) it is the (only?) way for the new-terrabyte-world. So having bitten the bullet, and doing some experiments i'm stuck as to what to do with / OT/ > I'm pretty sure FreeBSD can boot off of gmirror setups (see above, > boot2/loader should work off of gmirror), which means >1 disk. You > do not have to gmirror the entire disk, you can gmirror just a slice > (AFAIK). but gmirror is not ZFS, and yes it can, why not. > > I think (hope?) you can use the "remaining" (e.g. non-UFS/non-gmirror) > part of the 2nd disk for ZFS as well, otherwise the space would go > to waste. The "Root on ZFS configuration" FreeBSD ZFS Wiki page > seems to imply you can. The idea is to used the 'free/remaining' as part of the BIG ZFS 'array' [ED note : I've highjacked the 'Re: continuous backup solution for FreeBSD'] To Matt: since 'small' nowadays is big enough to hold /, what advantages are there in having root split up? also, having this split personality, what if the disk goes? the hammer/zfs is probably raided ... [btw, having a small-boot-partition brings back bad memories: the first thing I did on a new Compact was boot it diskless, repartition the disk, newfs, and I could no longer boot it :-) - part of the BIOS was there] To Doug: > ZFS boot is coming. great! any time estimate?, just curious, no preasure :-) some food for thought: In the past raid 5 would reduce the throughput conciderably, though nowadays it's hardly notisable, so I guess my reluctance to having a swap partition raided is gone. danny From dillon at apollo.backplane.com Sat Oct 11 18:10:34 2008 From: dillon at apollo.backplane.com (Matthew Dillon) Date: Sat Oct 11 18:10:41 2008 Subject: ZFS boot References: Message-ID: <200810111810.m9BIAGPw059975@apollo.backplane.com> :To Matt: : since 'small' nowadays is big enough to hold /, what advantages are there :in having root split up? :also, having this split personality, what if the disk goes? the hammer/zfs :is probably raided ... You mean /boot + root , or do you mean /root vs /usr vs /home? I'll answer both. With regards to /boot + root. A small separate /boot partition (256m) allows your root filesystem to use an arbitrarily complex topology. e.g. multiple geom layers, weird zfs setups, etc. So you get flexibility that you would otherwise not have if you went with a directly-bootable ZFS root. /boot can be as complex as boot2 allows. There's nothing preventing it from being RAIDed if boot2 supported that, and there's nothing preventing it (once you had ZFS boot capabilities) from being ZFS using a topology supported by boot2. Having a sparate /boot allows your filesystem root to use topologues boot2 would otherwise not support. With regards to the traditional BSD partitioning scheme, having a separate /usr, /home, /tmp, etc... there's no reason to do that stuff any more with ZFS (or HAMMER). You just need one, and can break it down into separate management domains within the filesystem (e.g. HAMMER PFS's). That's a generic statement of course, there will always be situations where you might want to partition things out separately. Most linux dists don't bother with multiple partitions any more. They just have '/' and maybe a small boot partition, and that's it. -Matt Matthew Dillon From fjwcash at gmail.com Sat Oct 11 20:44:40 2008 From: fjwcash at gmail.com (Freddie Cash) Date: Sat Oct 11 20:44:47 2008 Subject: ZFS boot In-Reply-To: <200810111810.m9BIAGPw059975@apollo.backplane.com> References: <200810111810.m9BIAGPw059975@apollo.backplane.com> Message-ID: On 10/11/08, Matthew Dillon wrote: > With regards to the traditional BSD partitioning scheme, having a > separate /usr, /home, /tmp, etc... there's no reason to do that stuff > any more with ZFS (or HAMMER). As separate partitions, no. As separate filesystems, definitely. While HAMMER PFSes may not support these things yet, ZFS allows you to tailor each filesystem to its purpose. For example, you can enable compression on /usr/ports, but have a separate /usr/ports/distfilles and /usr/ports/work that aren't compressed. Or /usr/src compressed and /usr/obj not. Have a small record (block) size for /usr/src, but a larger one for /home. Give each user a separate filesystem for their /home/, with separate snapshot policies, quotas, and reservations (initial filesystem size). Creating new filesystems with ZFS is as simple as "zfs create -o mountpoint=/wherever pool/fsname". If you put a little time into planning the hierarchy/structure, you can take advantage off the properties inheritance features of ZFS as well. > You just need one, and can break it > down into separate management domains within the filesystem > (e.g. HAMMER PFS's). Similar kind of idea. > Most linux dists don't bother with multiple partitions any more. > They just have '/' and maybe a small boot partition, and that's it. Heh, that's more proof of the difficulties inherent with old-school disk partitioning, compared to pooled storage setups, than an endorsement of using a single partition/filesystem. :) -- Freddie Cash fjwcash@gmail.com From neldredge at math.ucsd.edu Sat Oct 11 20:53:36 2008 From: neldredge at math.ucsd.edu (Nate Eldredge) Date: Sat Oct 11 20:53:43 2008 Subject: ZFS boot In-Reply-To: References: <200810111810.m9BIAGPw059975@apollo.backplane.com> Message-ID: On Sat, 11 Oct 2008, Freddie Cash wrote: > On 10/11/08, Matthew Dillon wrote: >> With regards to the traditional BSD partitioning scheme, having a >> separate /usr, /home, /tmp, etc... there's no reason to do that stuff >> any more with ZFS (or HAMMER). > > As separate partitions, no. As separate filesystems, definitely. > > While HAMMER PFSes may not support these things yet, ZFS allows you to > tailor each filesystem to its purpose. For example, you can enable > compression on /usr/ports, but have a separate /usr/ports/distfilles > and /usr/ports/work that aren't compressed. Or /usr/src compressed > and /usr/obj not. Have a small record (block) size for /usr/src, but > a larger one for /home. Give each user a separate filesystem for > their /home/, with separate snapshot policies, quotas, and > reservations (initial filesystem size). All this about ZFS sounds great, and I'd like to try it out, but some of the bugs, etc, listed at http://wiki.freebsd.org/ZFSKnownProblems are rather alarming. Even on a personal machine, I don't want these features at the cost of an unstable system. Is that list still current? FWIW, my system is amd64 with 1 G of memory, which the page implies is insufficient. Is it really? -- Nate Eldredge neldredge@math.ucsd.edu From fjwcash at gmail.com Sat Oct 11 20:54:27 2008 From: fjwcash at gmail.com (Freddie Cash) Date: Sat Oct 11 20:54:33 2008 Subject: ZFS boot In-Reply-To: References: Message-ID: On 10/11/08, Danny Braniss wrote: >> > I'm asking, because I want to deploy some zfs fileservers soon, and so >> > far the solution is either PXE boot, or keep one disk UFS (or boot off a >> > USB) For the servers we're deploying FreeBSD+ZFS on, mainly large backup systems with 24 drives, we're putting / onto either CompactFlash (using IDE adapters) or USB sticks (using internal connectors), using gmirror to provide fail-over for /. That way, we can boot off UFS, have full access to single-user mode and /rescue, and use every bit of each disk for ZFS. Works quite nicely. >> > Today's /(root+usr) is somewhere between .5 to 1Gb(kernel+debug+src), >> > and is readonly, so having 1 disk UFS seems to be a pitty. / by itself (no /usr, /home, /tmp, or /var) is under 300 MB on our systems (FreeBSD 7-STABLE from August, amd64). Definitely not worth dedicating an entire 500 GB drive to, or even a single slice or partition to. By putting / onto separate media (like CF, USB, whatever), you can dedicate all your harddrive space to ZFS. > /OT > Initially, I was not thrilled with ZFS, but once you cross the Once you start using ZFS features, especially snapshots, it's really hard to move to non-pooled-storage setups. Even LVM on Linux becomes hard to work with. There's just no easier way to work with multi-TB storage setups using 10+ drives. Even for smaller systems with only 3 drives, it's so much nicer working with pooled storage systems like ZFS. My home server uses a 2 GB USB stick for / with 3x 120 GB drives for ZFS, with separate filesystems for /usr, /usr/ports, /usr/src, /usr/obj, /usr/local, /home, /var, and /tmp. No fussing around with partition sizes ahead of time is probably the single greatest feature, with instant/unlimited snapshots a very close second. >> I think (hope?) you can use the "remaining" (e.g. non-UFS/non-gmirror) >> part of the 2nd disk for ZFS as well, otherwise the space would go >> to waste. The "Root on ZFS configuration" FreeBSD ZFS Wiki page >> seems to imply you can. I did this for awhile. 3x 120 GB drives configured as: 10 GB slice for / 2 GB slice for swap 108 GB slice to ZFS The first slice was configured as a 3-way gmirror, and the last slice was configured as a raidz pool. But performance wasn't that great. Moved / to a USB stick, and dedicated the entire drives to the zpool, and things have been a lot smoother. -- Freddie Cash fjwcash@gmail.com From delphij at delphij.net Sat Oct 11 21:40:57 2008 From: delphij at delphij.net (Xin LI) Date: Sat Oct 11 21:41:04 2008 Subject: ZFS boot In-Reply-To: <200810111810.m9BIAGPw059975@apollo.backplane.com> References: <200810111810.m9BIAGPw059975@apollo.backplane.com> Message-ID: <48F11D5E.7090108@delphij.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, Matt, Matthew Dillon wrote: [...] > /boot can be as complex as boot2 allows. There's nothing preventing > it from being RAIDed if boot2 supported that, and there's nothing > preventing it (once you had ZFS boot capabilities) from being ZFS > using a topology supported by boot2. Having a sparate /boot allows > your filesystem root to use topologues boot2 would otherwise not > support. I believe that it's a good idea to separate / from the zpool for other file systems, or even use a UFS /. My experience with ZFS on my laptop shows that disk failures can be more easily fixed if there are some utilities available in the UFS /, even when ZFS is used as /. Another issue with a ZFS / is that the snapshot rollback feature generally does not work on / since it needs the mountpoint to be unmounted. One thing that I found very useful is the new GPT boot feature on 8.0, which also works on older BIOS because the protected MBR would deal with the bootstrap to the actual GPT boot. Now we have a 15-block sized gptboot that can boot FreeBSD from UFS, however this 'boot' can be in virtually any size that the BIOS supports, so we can embed more logic there. Cheers, -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkjxHV0ACgkQi+vbBBjt66CpXgCfWstsxNc3B4xOzNTxz9/kdl3Y /WYAnjqiV5H8xQYxGgZTnwWieuG6ZZij =LH+x -----END PGP SIGNATURE----- From ken at mthelicon.com Sat Oct 11 22:30:57 2008 From: ken at mthelicon.com (Pegasus Mc Cleaft) Date: Sat Oct 11 22:31:04 2008 Subject: ZFS boot In-Reply-To: References: Message-ID: <200810112330.53214.ken@mthelicon.com> On Saturday 11 October 2008 21:53:35 Nate Eldredge wrote: > On Sat, 11 Oct 2008, Freddie Cash wrote: > > On 10/11/08, Matthew Dillon wrote: > >> With regards to the traditional BSD partitioning scheme, having a > >> separate /usr, /home, /tmp, etc... there's no reason to do that > >> stuff any more with ZFS (or HAMMER). > > > > As separate partitions, no. As separate filesystems, definitely. > > > > While HAMMER PFSes may not support these things yet, ZFS allows you to > > tailor each filesystem to its purpose. For example, you can enable > > compression on /usr/ports, but have a separate /usr/ports/distfilles > > and /usr/ports/work that aren't compressed. Or /usr/src compressed > > and /usr/obj not. Have a small record (block) size for /usr/src, but > > a larger one for /home. Give each user a separate filesystem for > > their /home/, with separate snapshot policies, quotas, and > > reservations (initial filesystem size). > > All this about ZFS sounds great, and I'd like to try it out, but some of > the bugs, etc, listed at http://wiki.freebsd.org/ZFSKnownProblems are > rather alarming. Even on a personal machine, I don't want these features > at the cost of an unstable system. Is that list still current? I dont know if that list is completely accurate any more, but I can tell you from my own personal experience with ZFS that it has been quite good. I have two servers (one is my test-bed at home) and the other is a production server running mostly mysql at work and I have never experienced the dead-locking problem. > > FWIW, my system is amd64 with 1 G of memory, which the page implies is > insufficient. Is it really? This may be purely subjective, as I have never bench marked the speeds, but when I was first testing zfs on a i386 machine with 1gig ram, I thought the performance was mediocre. However, when I loaded the system on a quad core - core2 with 8 gigs ram, I was quite impressed. I put localized changes in my /boot/loader.conf to give the kernel more breathing room and disabled the prefetch for zfs. #more loader.conf vm.kmem_size_max="1073741824" vm.kmem_size="1073741824" vfs.zfs.prefetch_disable=1 The best advice I can give is for you to find an old machine and test-bed zfs for yourself. I personally have been pleased with it and It has saved my machines data 4 times already (dieing hardware, unexpected power bounces, etc) As a side note, my production machine boots off a dedicated UFS drive (where I also have a slice for the swap). /usr, /var, /var/db and /usr/home are zfs. My test server at home only has /usr/home as zfs. I found it easier for me, when I kill the home machine to just do a reload/rebuild of the OS, rebuild the applications, and rechown/grp the home directories. Peg From neldredge at math.ucsd.edu Sat Oct 11 23:21:57 2008 From: neldredge at math.ucsd.edu (Nate Eldredge) Date: Sat Oct 11 23:22:03 2008 Subject: ZFS boot In-Reply-To: <200810112330.53214.ken@mthelicon.com> References: <200810112330.53214.ken@mthelicon.com> Message-ID: On Sat, 11 Oct 2008, Pegasus Mc Cleaft wrote: >> FWIW, my system is amd64 with 1 G of memory, which the page implies is >> insufficient. Is it really? > > This may be purely subjective, as I have never bench marked the speeds, but > when I was first testing zfs on a i386 machine with 1gig ram, I thought the > performance was mediocre. However, when I loaded the system on a quad core - > core2 with 8 gigs ram, I was quite impressed. I put localized changes in my > /boot/loader.conf to give the kernel more breathing room and disabled the > prefetch for zfs. > > #more loader.conf > vm.kmem_size_max="1073741824" > vm.kmem_size="1073741824" > vfs.zfs.prefetch_disable=1 I was somewhat confused by the suggestions on the wiki. Do the kmem_size sysctls affect the allocation of *memory* or of *address space*? It seems a bit much to reserve 1 G of memory solely for the use of the kernel, expecially in my case when that's all I have :) But on amd64, it's welcome to have terabytes of address space if it will help. > The best advice I can give is for you to find an old machine and test-bed zfs > for yourself. I personally have been pleased with it and It has saved my > machines data 4 times already (dieing hardware, unexpected power bounces, etc) Sure, but if my "new" machine isn't studly enough to run it, there's no hope for an old machine. So I'm trying to figure out what I actually need. -- Nate Eldredge neldredge@math.ucsd.edu From yanefbsd at gmail.com Sun Oct 12 00:02:29 2008 From: yanefbsd at gmail.com (Garrett Cooper) Date: Sun Oct 12 00:02:36 2008 Subject: TIME WARP! Re: HEADS UP: GCC 4.2.0 is coming In-Reply-To: References: Message-ID: <7d6fde3d0810111641s673bfc0aiac7735f517b73dd9@mail.gmail.com> On Wed, Oct 8, 2008 at 1:36 AM, Peter Wemm wrote: > On Wed, Oct 8, 2008 at 12:57 AM, O. Hartmann > wrote: >> Alexander Kabaev wrote: >>> >>> On Fri, 18 May 2007 19:20:07 -0400 >>> Alexander Kabaev wrote: >>> >>>> HEADS UP: I will start importing GCC 4.2.0 bits in about one hour and >>>> plan to finish in a couple of hours after that. >>>> >>>> The src/ tree will be utterly broken meanwhile. I'll send an 'all >>>> clear' message when done. >>> >>> Done. >>> >> >> Just for those who aren't on the cutting edge: why gcc 4.2.0 and not 4.2.1 >> as it is used in 7.X? >> >> Regards, >> O. > > Sorry about that. I accidently revived a bunch of stuck email > messages from our mailing list processing system. These messages from > 2007 came back to life somehow. > > (Hint: Mailman's 'unshunt' command doesn't give a usage message) Apparently you mastered what Robert Zemekis was trying to do back in 1985 XD. -Garrett From yuri at rawbw.com Sun Oct 12 00:41:11 2008 From: yuri at rawbw.com (Yuri) Date: Sun Oct 12 00:41:17 2008 Subject: Is it possible to recover from SEGV? Message-ID: <48F147A5.1040107@rawbw.com> Let's say I have signal(3) handler set. And I know exactly what instruction caused SEGV and why. Is there a way to access from signal handler CPU registers as they were before signal, modify some of them, clear the signal and continue from the instruction that caused SEGV initially? I see that if signal handler doesn't terminate the process signal is being generated again and again. I understand it the way that the faulty instruction is being rerun if signal handler didn't terminate the process. rusage.ru_nsignals is also being incremented every time signal handler is being called. Yuri PS: Of course I understand why SEGVs happen in general. I am trying to understand if it's possible to use SEGV beyond the way it's commonly used. From neldredge at math.ucsd.edu Sun Oct 12 00:51:45 2008 From: neldredge at math.ucsd.edu (Nate Eldredge) Date: Sun Oct 12 00:51:52 2008 Subject: Is it possible to recover from SEGV? In-Reply-To: <48F147A5.1040107@rawbw.com> References: <48F147A5.1040107@rawbw.com> Message-ID: On Sat, 11 Oct 2008, Yuri wrote: > Let's say I have signal(3) handler set. > And I know exactly what instruction caused SEGV and why. > > Is there a way to access from signal handler CPU registers as they > were before signal, modify some of them, clear the signal and > continue from the instruction that caused SEGV initially? Absolutely. Declare your signal handler as void handler(int sig, int code, struct sigcontext *scp); You will need to cast the pointer passed to signal(3). struct sigcontext is defined in I believe. struct sigcontext contains the CPU registers as they were when the faulting instruction began to execute. You can modify them and then return from the signal handler. The program will resume the faulting instruction with the new registers. You can also alter the copy of the instruction pointer in the struct sigcontext if you want it to resume somewhere else. There is also a libsigsegv which looks like it wraps some of this process in a less machine-specific way. Out of curiosity, what are you looking to achieve with this? And what architecture are you on? -- Nate Eldredge neldredge@math.ucsd.edu From mwm-keyword-freebsdhackers2.e313df at mired.org Sun Oct 12 01:31:03 2008 From: mwm-keyword-freebsdhackers2.e313df at mired.org (Mike Meyer) Date: Sun Oct 12 01:31:11 2008 Subject: continuous backup solution for FreeBSD In-Reply-To: <20081011112431.GA60924@icarus.home.lan> References: <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> <20081010144111.GA34609@icarus.home.lan> <20081010112952.52b8209b@bhuda.mired.org> <20081010154249.GA35859@icarus.home.lan> <20081010122228.355c2c3e@bhuda.mired.org> <20081011104409.GA58698@icarus.home.lan> <20081011112431.GA60924@icarus.home.lan> Message-ID: <20081011212829.57b889d7@bhuda.mired.org> On Sat, 11 Oct 2008 04:24:31 -0700 Jeremy Chadwick wrote: > > I'm asking, because I want to deploy some zfs fileservers soon, and so > > far the solution is either PXE boot, or keep one disk UFS (or boot off a USB) > > Today's /(root+usr) is somewhere between .5 to 1Gb(kernel+debug+src), > > and is readonly, so having 1 disk UFS seems to be a pitty. > > Hold on a minute. "One disk" has nothing to do with the filesystem. > You asked if FreeBSD could boot off of a specific filesystem, and I > answered that -- I didn't state anything about disk counts. Now you're > changing the focus. :-) > > I'm pretty sure FreeBSD can boot off of gmirror setups (see above, > boot2/loader should work off of gmirror), which means >1 disk. You > do not have to gmirror the entire disk, you can gmirror just a slice > (AFAIK). > > I think (hope?) you can use the "remaining" (e.g. non-UFS/non-gmirror) > part of the 2nd disk for ZFS as well, otherwise the space would go > to waste. The "Root on ZFS configuration" FreeBSD ZFS Wiki page > seems to imply you can. You mean like this: bhuda% gmirror status Name Status Components mirror/boot COMPLETE ad0s1a ad1s1a bhuda% zpool status pool: internal state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM internal ONLINE 0 0 0 mirror ONLINE 0 0 0 ad0s1d ONLINE 0 0 0 ad1s1d ONLINE 0 0 0 errors: No known data errors Yes, I don't get the benefits of having /boot on a zfs partition, but I do get the benefits of having it on a mirror: automatic duplication, reads from either device, and I can use either device stand-alone if I break the mirror. Note that FreeBSD booting from a gmirror'ed partition/disk can't boot from the gmirror device - boot doesn't understand gmirror. It can, however, boot from any of the devices participating in the mirror. The mirror device appears after the kernel is loaded. Given that I have to have a separate boot partition, having swap partitions on the drives is a win compared to swapping to a zvol. I'm going to investigate putting /boot on an SSD of some kind so that ZFS can have the entire disk. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From mwm-keyword-freebsdhackers2.e313df at mired.org Sun Oct 12 01:34:23 2008 From: mwm-keyword-freebsdhackers2.e313df at mired.org (Mike Meyer) Date: Sun Oct 12 01:34:30 2008 Subject: ZFS boot In-Reply-To: References: <200810111810.m9BIAGPw059975@apollo.backplane.com> Message-ID: <20081011213149.19385b8d@bhuda.mired.org> On Sat, 11 Oct 2008 20:37:10 +0000 "Freddie Cash" wrote: > > Most linux dists don't bother with multiple partitions any more. > > They just have '/' and maybe a small boot partition, and that's it. > > Heh, that's more proof of the difficulties inherent with old-school > disk partitioning, compared to pooled storage setups, than an > endorsement of using a single partition/filesystem. :) I think it's more likely that, given you know absolutely nothing about what the system is going to be used for, you don't know enough to set up the partitions intelligently, so one partitions makes as much sense as anything else. That's one of the best thing about pooled storage: you can create new file systems for new usages without having to repartition your disk subsystem. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From dfr at rabson.org Sun Oct 12 07:44:21 2008 From: dfr at rabson.org (Doug Rabson) Date: Sun Oct 12 07:44:27 2008 Subject: ZFS boot In-Reply-To: References: Message-ID: On 11 Oct 2008, at 14:28, Danny Braniss wrote: > > > To Doug: > > ZFS boot is coming. > great! any time estimate?, just curious, no preasure :-) Its part of pjd's current big ZFS patch which brings us more or less up-to-date with Solaris. I'm not the best person to ask when that will be ready but I would expect to see it in current later this year. I hope to have some time soon to work on some outstanding issues with the boot code, so I may be able to add raidz and raidz2 support before its committed to current. From koitsu at FreeBSD.org Sun Oct 12 08:43:38 2008 From: koitsu at FreeBSD.org (Jeremy Chadwick) Date: Sun Oct 12 08:43:45 2008 Subject: ZFS boot In-Reply-To: References: <200810112330.53214.ken@mthelicon.com> Message-ID: <20081012084336.GA84786@icarus.home.lan> On Sat, Oct 11, 2008 at 04:21:55PM -0700, Nate Eldredge wrote: > On Sat, 11 Oct 2008, Pegasus Mc Cleaft wrote: > >>> FWIW, my system is amd64 with 1 G of memory, which the page implies is >>> insufficient. Is it really? >> >> This may be purely subjective, as I have never bench marked the speeds, but >> when I was first testing zfs on a i386 machine with 1gig ram, I thought the >> performance was mediocre. However, when I loaded the system on a quad core - >> core2 with 8 gigs ram, I was quite impressed. I put localized changes in my >> /boot/loader.conf to give the kernel more breathing room and disabled the >> prefetch for zfs. >> >> #more loader.conf >> vm.kmem_size_max="1073741824" >> vm.kmem_size="1073741824" >> vfs.zfs.prefetch_disable=1 > > I was somewhat confused by the suggestions on the wiki. Do the kmem_size > sysctls affect the allocation of *memory* or of *address space*? The Wiki is somewhat vague and doesn't give you all the knowledge you need. The kmem_* sysctls do not define pre-allocated amounts. They define the amount of memory which can be used by the kernel for allocation. I strongly advocate tuning two other sysctls, which can help greatly in ensuring no system lock-ups and no kmem exhaustion panics: vfs.zfs.arc_min vfs.zfs.arc_max The following values are what I use, but others have reported better performance with arc_max set to 128M: vfs.zfs.arc_min="16M" vfs.zfs.arc_max="64M" > It seems a bit much to reserve 1 G of memory solely for the use of the > kernel, expecially in my case when that's all I have :) But on amd64, > it's welcome to have terabytes of address space if it will help. ZFS is a memory hog, period. That's just the nature of the beast. You probably should not be using it on a system with 1GB. I'll remind you that memory right now is *incredibly* cheap; you can get 4GB of brand-name lifetime-warranty RAM for around US$40-50. Secondly, with regards to amd64: RELENG_6 and RELENG_7 amd64 cannot handle more than 2GB of kmem. Yes, you read that correct; it's not a typo. It's an implementation issue which cannot be easily solved on those releases. CURRENT can address up to 512GB. I've fully documented this on my Wiki, see section Kernel. http://wiki.freebsd.org/JeremyChadwick/Commonly_reported_issues -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From romain at blogreen.org Sun Oct 12 09:48:11 2008 From: romain at blogreen.org (Romain =?iso-8859-1?Q?Tarti=E8re?=) Date: Sun Oct 12 09:48:18 2008 Subject: Is it possible to recover from SEGV? In-Reply-To: <48F147A5.1040107@rawbw.com> References: <48F147A5.1040107@rawbw.com> Message-ID: <20081012092937.GA53444@marvin.blogreen.org> Hi Yuri, On Sat, Oct 11, 2008 at 05:41:09PM -0700, Yuri wrote: > Is there a way to access from signal handler CPU registers as they > were before signal, modify some of them, clear the signal and > continue from the instruction that caused SEGV initially? Maybe you can have a look to the development version of the Enlightenment window manager [1]. It catches segfaults and displayed a window to ask the user what to do (continue, abort). I experienced a few crashes where this helpful window was triggered a dozen times, I asked the window manager to continue and could save my work before everything crashed. First moves in the svn repository [2] (basically grep SEGV): | ./src/bin/e_exec.c:456: else if (cfdata->event.exit_signal == SIGSEGV) | ./src/bin/e_desklock.c:689: sigaction(SIGSEGV, &action, NULL); | ./src/bin/e_object.c:153: sigaction(SIGSEGV, &act, &oact); | ./src/bin/e_object.c:158: sigaction(SIGSEGV, &oact, NULL); | ./src/bin/e_object.c:168: sigaction(SIGSEGV, &oact, NULL); | ./src/bin/e_signals.c:28: e_alert_show("This is very bad. Enlightenment SEGV'd.\n" | ./src/bin/e_signals.c:48: e_alert_show("This is very bad. Enlightenment SEGV'd.\n" | ./src/bin/e_main.c:99: sigaction(SIGSEGV, &action, NULL); | ./src/bin/e_main.c:322:// FIXME: SEGV's on shutdown if fm2 windows up - disable for now. Hope that helps! Romain References: 1. http://enlightenment.org/ 2. http://svn.enlightenment.org/svn/e/trunk/e/ -- Romain Tarti?re http://romain.blogreen.org/ pgp: 8DAB A124 0DA4 7024 F82A E748 D8E9 A33F FF56 FF43 (ID: 0xFF56FF43) (plain text =non-HTML= PGP/GPG encrypted/signed e-mail much appreciated) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081012/446db475/attachment.pgp From talon at lpthe.jussieu.fr Sun Oct 12 12:47:29 2008 From: talon at lpthe.jussieu.fr (Michel Talon) Date: Sun Oct 12 13:40:31 2008 Subject: ZFS boot Message-ID: <20081012122204.GA85909@lpthe.jussieu.fr> > > It seems a bit much to reserve 1 G of memory solely for the use of the > > kernel, expecially in my case when that's all I have :) But on amd64, > > it's welcome to have terabytes of address space if it will help. > > ZFS is a memory hog, period. That's just the nature of the beast. > You probably should not be using it on a system with 1GB. I'll remind > you that memory right now is *incredibly* cheap; you can get 4GB of > brand-name lifetime-warranty RAM for around US$40-50. I am running FreeBSD-7 with the following: m.kmem_size="512M" vm.kmem_size_max="512M" vfs.zfs.arc_max="256M" vfs.zfs.arc_min="32M" vfs.zfs.prefetch_disable="1" on a i386 machine with 1 Gig memory (now upgraded to 1.5 Gig) and i have not experienced, up to now, any lockup problem. Apparently the kernel effectively uses around 500M memory, indeed. Performance seems reasonable to me, not worse than with UFS. -- Michel TALON From jhb at freebsd.org Mon Oct 13 16:03:21 2008 From: jhb at freebsd.org (John Baldwin) Date: Mon Oct 13 16:03:28 2008 Subject: What is the time between 2 mi_switches in freebsd. In-Reply-To: <86zlleq397.fsf@ds4.des.no> References: <53fa490b0810071947j23fc0f72n5360b6f174ddc96d@mail.gmail.com> <20081008174956.GA98121@nexus.in-nomine.org> <86zlleq397.fsf@ds4.des.no> Message-ID: <200810131051.07174.jhb@freebsd.org> On Wednesday 08 October 2008 03:46:12 pm Dag-Erling Sm?rgrav wrote: > Jeroen Ruigrok van der Werven writes: > > -On [20081008 19:15], ushasri tummala (tummala.ushasri@gmail.com) wrote: > > > I just want to know how its(time between 2 mi_switch()) calculated > > > and in which variable is it stored in the code.(FreeBSD 5.2 release) > > > This is not addressed in text book. > > What Dag-Erling meant to say, and if I recall correctly, a switch() is > > highly dependent on your hardware. So the time taken for a specific > > machine can be vastly different from another machine. > > No, no, no. > > Assuming the question is really "what is the time between two task > switches", > > A task switch can happen for one of many reasons: > > - first, and simplest, the current task has used up its quantum; > > - the current task is waiting for an external event (I/O, a mutex, a > timeout, etc.) > > - the current task has terminated; > > - something happened to make a higher-priority task runnable; > > - ... > > The closest you can get to a hard answer is if you consider only the > first of the above, in which case the answer is 1/hz second, where "hz" > is literally a kernel variable named hz. Its default value is 1,000 on > amd64, i386, ia64 and sparc64, and 100 on all other platforms. Actually, hz isn't the quantum. sched_tick() is called 'hz' times per second, but the scheduler is free to implement its own quantum. The default quantum for 4BSD is actually hz / 10 for example: static int sched_quantum; /* Roundrobin scheduling quantum in ticks. */ #define SCHED_QUANTUM (hz / 10) /* Default sched quantum */ I'm not sure what ULE's quantum is or how it is computed. -- John Baldwin From mboxindia at gmail.com Tue Oct 14 18:32:49 2008 From: mboxindia at gmail.com (Srinivas) Date: Tue Oct 14 18:32:56 2008 Subject: [Doubt] Can a PCI device communicate with another PCI or other device? Message-ID: Hello, I have a small doubt. Suppose I have a PCI card with a general purpose CPU on it. Could it be able to communicate with another PCI device or ISA device(lets say IDE hard disk)? Thanks, Srinivas From ken at mthelicon.com Tue Oct 14 19:39:43 2008 From: ken at mthelicon.com (Pegasus Mc Cleaft) Date: Tue Oct 14 19:39:49 2008 Subject: [Doubt] Can a PCI device communicate with another PCI or other device? In-Reply-To: References: Message-ID: <200810142039.39287.ken@mthelicon.com> On Tuesday 14 October 2008 19:32:47 Srinivas wrote: > Hello, > > I have a small doubt. > > Suppose I have a PCI card with a general purpose CPU on it. Could it be > able to communicate with another PCI device or ISA device(lets say IDE hard > disk)? Hi Srinivas, Others may have a different opinion on this, and I am curious to see there input to this question as well. I tried doing something like this years ago with the Blackfin processor, but I found it to be not a good idea. While I could DMA transfer memory from the DSP into the physical memory of another device (In my case, the HD controller) I couldn't take control of IRQ?s. Other potential problems arose that turned me off to the whole idea (Data concurrency between the DSP->HD and the host->HD). Even if you got past these problems you would still face having to deal with any number of filing systems formats. I found it much easier to make a device driver and service/daemon on the host machine that proxi-requested things for the DSP board and burped it into ram that the DSP could then grab and process (or dump to disk from, in the other direction). While the speed may not be brilliant, it made things a lot easier for me to manage on a software level. In later revisions, I re-spun the PCB so it had an IDE controller local to the DSP and stuck an PLX-Tech PCI to PCI bridge between the host and the DSP so I could manage 2 small memory windows between the two (One for API commands, and the other data) with a single IRQ back to the host. This allowed me to have a high-speed IDE port local to the DSP where it was needed, and a slower link back to the host CPU for pulling video files, etc. Hope this helps, Peg From alancyang at gmail.com Wed Oct 15 00:47:47 2008 From: alancyang at gmail.com (alan yang) Date: Wed Oct 15 00:47:54 2008 Subject: tracing pf code Message-ID: <290865fd0810141747l39b80e2ao329c8212061a67c1@mail.gmail.com> hello, for pf port on freebsd, i would like to trace the packet flow, looking at from ether_input -> etiher_demux -> ip_input -> tcp_input where / how pf handles / process the packet. can people shed some lights where to start. really appreciate. From max at love2party.net Wed Oct 15 01:02:06 2008 From: max at love2party.net (Max Laier) Date: Wed Oct 15 01:02:14 2008 Subject: tracing pf code In-Reply-To: <290865fd0810141747l39b80e2ao329c8212061a67c1@mail.gmail.com> References: <290865fd0810141747l39b80e2ao329c8212061a67c1@mail.gmail.com> Message-ID: <200810150302.03949.max@love2party.net> On Wednesday 15 October 2008 02:47:46 alan yang wrote: > hello, > > for pf port on freebsd, i would like to trace the packet flow, looking > at from ether_input -> etiher_demux -> ip_input -> tcp_input where / > how pf handles / process the packet. > > can people shed some lights where to start. really appreciate. ps hooks into the pfil(9) hook point in ip[6]_{in,out}put(). Look for calls to "pfil_run_hooks" in the code. From there the call proceeds to the hook functions defined in pf_ioctl.c pf_check_{in,out}[6]. The processing inside pf is best understood by looking at the following chart: http://homepage.mac.com/quension/pf/flow.png Is this the information you are looking for? -- /"\ Best regards, | mlaier@freebsd.org \ / Max Laier | ICQ #67774661 X http://pf4freebsd.love2party.net/ | mlaier@EFnet / \ ASCII Ribbon Campaign | Against HTML Mail and News From alancyang at gmail.com Wed Oct 15 02:41:39 2008 From: alancyang at gmail.com (alan yang) Date: Wed Oct 15 02:42:10 2008 Subject: tracing pf code In-Reply-To: <200810150302.03949.max@love2party.net> References: <290865fd0810141747l39b80e2ao329c8212061a67c1@mail.gmail.com> <200810150302.03949.max@love2party.net> Message-ID: <290865fd0810141941l7c63a8e6l1c9c4839518c9ac8@mail.gmail.com> yes, exact. thanks a lot! On Tue, Oct 14, 2008 at 6:02 PM, Max Laier wrote: > On Wednesday 15 October 2008 02:47:46 alan yang wrote: >> hello, >> >> for pf port on freebsd, i would like to trace the packet flow, looking >> at from ether_input -> etiher_demux -> ip_input -> tcp_input where / >> how pf handles / process the packet. >> >> can people shed some lights where to start. really appreciate. > > ps hooks into the pfil(9) hook point in ip[6]_{in,out}put(). Look for calls > to "pfil_run_hooks" in the code. From there the call proceeds to the hook > functions defined in pf_ioctl.c pf_check_{in,out}[6]. > > The processing inside pf is best understood by looking at the following chart: > http://homepage.mac.com/quension/pf/flow.png > > Is this the information you are looking for? > > -- > /"\ Best regards, | mlaier@freebsd.org > \ / Max Laier | ICQ #67774661 > X http://pf4freebsd.love2party.net/ | mlaier@EFnet > / \ ASCII Ribbon Campaign | Against HTML Mail and News > From m.jakeman at lancaster.ac.uk Wed Oct 15 13:07:53 2008 From: m.jakeman at lancaster.ac.uk (Matthew Jakeman) Date: Wed Oct 15 13:08:01 2008 Subject: Call function on sysctl value change Message-ID: <200810151343.28136.m.jakeman@lancaster.ac.uk> Hi all, I was wondering if it is possible to call a function when a sysctl value is changed. I have added a few sysctl int variables to the kernel and for some of these i only want certain values to be acceptable as input depending on some conditions. I would like to be able to call a function if possible, to validate the value entered via the sysctl command. Thanks in advance Matt From otaviof at gmail.com Wed Oct 15 13:41:29 2008 From: otaviof at gmail.com (=?UTF-8?Q?Ot=C3=A1vio_Fernandes?=) Date: Wed Oct 15 13:41:36 2008 Subject: Invitation to connect on LinkedIn Message-ID: <1062810545.2134127.1224076387167.JavaMail.app@esv4-com11.prod> LinkedIn ------------ FreeBSD, I'd like to add you to my professional network on LinkedIn. - Ot?vio Learn more: https://www.linkedin.com/e/isd/380376003/wEFbdMkx/ ------------------------------------------ What is LinkedIn and why should you join? http://learn.linkedin.com/what-is-linkedin/ ------ (c) 2008, LinkedIn Corporation From des at des.no Wed Oct 15 14:03:05 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed Oct 15 14:03:11 2008 Subject: Call function on sysctl value change In-Reply-To: <200810151343.28136.m.jakeman@lancaster.ac.uk> (Matthew Jakeman's message of "Wed, 15 Oct 2008 13:43:28 +0100") References: <200810151343.28136.m.jakeman@lancaster.ac.uk> Message-ID: <8663nu0xd4.fsf@ds4.des.no> Matthew Jakeman writes: > I was wondering if it is possible to call a function when a sysctl value is > changed. grep -r SYSCTL_ADD_PROC /usr/src/sys DES -- Dag-Erling Sm?rgrav - des@des.no From des at des.no Wed Oct 15 14:03:46 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed Oct 15 14:03:52 2008 Subject: Call function on sysctl value change In-Reply-To: <8663nu0xd4.fsf@ds4.des.no> ("Dag-Erling =?utf-8?Q?Sm=C3=B8rg?= =?utf-8?Q?rav=22's?= message of "Wed, 15 Oct 2008 16:03:03 +0200") References: <200810151343.28136.m.jakeman@lancaster.ac.uk> <8663nu0xd4.fsf@ds4.des.no> Message-ID: <861vyi0xby.fsf@ds4.des.no> Dag-Erling Sm?rgrav writes: > Matthew Jakeman writes: > > I was wondering if it is possible to call a function when a sysctl value is > > changed. > grep -r SYSCTL_ADD_PROC /usr/src/sys even better: 'man SYSCTL_ADD_PROC' will answer all your questions. DES -- Dag-Erling Sm?rgrav - des@des.no From olli at lurza.secnetix.de Wed Oct 15 14:17:33 2008 From: olli at lurza.secnetix.de (Oliver Fromme) Date: Wed Oct 15 14:17:40 2008 Subject: Call function on sysctl value change In-Reply-To: <200810151343.28136.m.jakeman@lancaster.ac.uk> Message-ID: <200810151417.m9FEHUEn037917@lurza.secnetix.de> Matthew Jakeman wrote: > I was wondering if it is possible to call a function when a sysctl value is > changed. I have added a few sysctl int variables to the kernel and for some > of these i only want certain values to be acceptable as input depending on > some conditions. I would like to be able to call a function if possible, to > validate the value entered via the sysctl command. Yes, you can do this with a "PROC" type sysctl. For example, look at sysctl_hlt_cpus() in sys/i386/i386/mp_machdep.c. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Gesch?ftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M?n- chen, HRB 125758, Gesch?ftsf?hrer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "Whatever happened to the days when hacking started at the cerebral cortex, and not at the keyboard?" -- Sid on userfriendly.org by Illiad, 2007-06-20 From m.jakeman at lancaster.ac.uk Wed Oct 15 14:32:57 2008 From: m.jakeman at lancaster.ac.uk (Matthew Jakeman) Date: Wed Oct 15 14:33:04 2008 Subject: Call function on sysctl value change In-Reply-To: <861vyi0xby.fsf@ds4.des.no> References: <200810151343.28136.m.jakeman@lancaster.ac.uk> <8663nu0xd4.fsf@ds4.des.no> <861vyi0xby.fsf@ds4.des.no> Message-ID: <200810151509.45363.m.jakeman@lancaster.ac.uk> On Wednesday 15 October 2008, Dag-Erling Sm?rgrav wrote: > Dag-Erling Sm?rgrav writes: > > Matthew Jakeman writes: > > > I was wondering if it is possible to call a function when a sysctl > > > value is changed. > > > > grep -r SYSCTL_ADD_PROC /usr/src/sys > > even better: 'man SYSCTL_ADD_PROC' will answer all your questions. > > DES Thanks for that. It looks like just the job. Now exactly sure how I missed it in the first place but thanks... From que_deseja at hotmail.com Thu Oct 16 05:29:49 2008 From: que_deseja at hotmail.com (Desmond Chapman) Date: Thu Oct 16 05:29:56 2008 Subject: need help with vbox Message-ID: It's dependent upon kbuild. Since the developers have no intention of fixing the issue, I would like a tutorial on converting the kmk file to a normal Makefile. _________________________________________________________________ See how Windows connects the people, information, and fun that are part of your life. http://clk.atdmt.com/MRT/go/msnnkwxp1020093175mrt/direct/01/ From ivoras at freebsd.org Thu Oct 16 09:11:41 2008 From: ivoras at freebsd.org (Ivan Voras) Date: Thu Oct 16 09:11:48 2008 Subject: need help with vbox In-Reply-To: References: Message-ID: Desmond Chapman wrote: > It's dependent upon kbuild. Since the developers have no intention of fixing the issue, I would like a tutorial on converting the kmk file to a normal Makefile. What is kmk? Google only shows it's used with VirtualBox and nowhere else. If it's something the authors of VirtualBox created, you'll have to ask them. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081016/1bb575e5/signature.pgp From bruce at cran.org.uk Thu Oct 16 09:27:45 2008 From: bruce at cran.org.uk (Bruce Cran) Date: Thu Oct 16 09:27:53 2008 Subject: need help with vbox In-Reply-To: References: Message-ID: <48F70904.8000702@cran.org.uk> Ivan Voras wrote: > Desmond Chapman wrote: > >> It's dependent upon kbuild. Since the developers have no intention of fixing the issue, I would like a tutorial on converting the kmk file to a normal Makefile. >> > > What is kmk? Google only shows it's used with VirtualBox and nowhere > else. If it's something the authors of VirtualBox created, you'll have > to ask them. > Apparently it's the tool used to build kbuild (http://svn.netlabs.org/kbuild/wiki/kmk, http://kbuild.sourceforge.net/) projects. It seems it's what the Linux kernel build system uses. -- Bruce From ivoras at freebsd.org Thu Oct 16 09:32:51 2008 From: ivoras at freebsd.org (Ivan Voras) Date: Thu Oct 16 09:32:57 2008 Subject: need help with vbox In-Reply-To: <48F70904.8000702@cran.org.uk> References: <48F70904.8000702@cran.org.uk> Message-ID: Bruce Cran wrote: > Ivan Voras wrote: >> Desmond Chapman wrote: >> >>> It's dependent upon kbuild. Since the developers have no intention of >>> fixing the issue, I would like a tutorial on converting the kmk file >>> to a normal Makefile. >>> >> >> What is kmk? Google only shows it's used with VirtualBox and nowhere >> else. If it's something the authors of VirtualBox created, you'll have >> to ask them. >> > > Apparently it's the tool used to build kbuild > (http://svn.netlabs.org/kbuild/wiki/kmk, http://kbuild.sourceforge.net/) > projects. It seems it's what the Linux kernel build system uses. Well, this seems to sum it up: """ Why not use vanilla GNU make? There are several reasons: ... # Finally, because we can. :-) """ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081016/ebd0f1b0/signature.pgp From que_deseja at hotmail.com Thu Oct 16 13:47:19 2008 From: que_deseja at hotmail.com (Desmond Chapman) Date: Thu Oct 16 13:47:26 2008 Subject: need help with vbox In-Reply-To: References: Message-ID: ---------------------------------------- > To: freebsd-hackers@freebsd.org > From: ivoras@freebsd.org > Date: Thu, 16 Oct 2008 11:11:37 +0200 > Subject: Re: need help with vbox > > Desmond Chapman wrote: >> It's dependent upon kbuild. Since the developers have no intention of fixing the issue, I would like a tutorial on converting the kmk file to a normal Makefile. > > What is kmk? Google only shows it's used with VirtualBox and nowhere > else. If it's something the authors of VirtualBox created, you'll have > to ask them. > That's the problem. I started maintaining the kBuild port which is also part of virtualbox and makes the kmk type files. Since kBuild is marked as broken, and for good reason. Anyway, thanks for the response. _________________________________________________________________ Want to read Hotmail messages in Outlook? The Wordsmiths show you how. http://windowslive.com/connect/post/wedowindowslive.spaces.live.com-Blog-cns!20EE04FBC541789!167.entry?ocid=TXT_TAGLM_WL_hotmail_092008 From que_deseja at hotmail.com Thu Oct 16 13:54:42 2008 From: que_deseja at hotmail.com (Desmond Chapman) Date: Thu Oct 16 13:54:48 2008 Subject: need help with vbox In-Reply-To: References: <48F70904.8000702@cran.org.uk> Message-ID: Gentleman, I agree with both of you. Thanks for everything. ---------------------------------------- > To: freebsd-hackers@freebsd.org > From: ivoras@freebsd.org > Date: Thu, 16 Oct 2008 11:32:48 +0200 > Subject: Re: need help with vbox > > Bruce Cran wrote: >> Ivan Voras wrote: >>> Desmond Chapman wrote: >>> >>>> It's dependent upon kbuild. Since the developers have no intention of >>>> fixing the issue, I would like a tutorial on converting the kmk file >>>> to a normal Makefile. >>>> >>> >>> What is kmk? Google only shows it's used with VirtualBox and nowhere >>> else. If it's something the authors of VirtualBox created, you'll have >>> to ask them. >>> >> >> Apparently it's the tool used to build kbuild >> (http://svn.netlabs.org/kbuild/wiki/kmk, http://kbuild.sourceforge.net/) >> projects. It seems it's what the Linux kernel build system uses. > > Well, this seems to sum it up: > > """ > Why not use vanilla GNU make? There are several reasons: > ... > # Finally, because we can. :-) > """ > _________________________________________________________________ When your life is on the go?take your life with you. http://clk.atdmt.com/MRT/go/115298558/direct/01/ From mboxindia at gmail.com Thu Oct 16 14:43:08 2008 From: mboxindia at gmail.com (Srinivas) Date: Thu Oct 16 14:43:15 2008 Subject: [Info required] PC Architecture Message-ID: Hello, I have a theoretical understanding of the PC architecture and the details but have no idea of how things go under the hood(for a real computer). I think it would be very useful for me(as well as beginners) to know how things work real-time. Even though it is not a correct mailing list, I am posting this because you guys are the real hackers and know a lot about how things work inside. I would like to know about the following. Plz add anything which you think will be helpful for the beginners. 1. Schematic diagrams of motherboards 2. How a bios detects various kinds of buses 3. How bios and os detects the different devices present in the system and what are their capabilities 4. How interrupts are routed inside 5. Groups where we can communicate related information I searched a lot in the internet but I am not able to find a good place or info that gives the detailed information. Thanks, Srinivas From asmodai at in-nomine.org Thu Oct 16 14:48:17 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Thu Oct 16 14:48:24 2008 Subject: [Info required] PC Architecture In-Reply-To: References: Message-ID: <20081016144813.GY98121@nexus.in-nomine.org> -On [20081016 16:43], Srinivas (mboxindia@gmail.com) wrote: >I have a theoretical understanding of the PC architecture and the >details but have no idea of how things go under the hood(for a real >computer). http://www.amazon.com/dp/0123706068/ - Computer Organization and Design: The Hardware/Software Interface by Patterson and Hennessy http://www.amazon.com/dp/0131485210/ - Structured Computer Organization by Tanenbaum That should answer most, if not all, of your questions on that subject. -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B Seek not death in the error of your life: and pull not upon yourselves destruction with the works of your hands... From ivoras at freebsd.org Thu Oct 16 15:11:41 2008 From: ivoras at freebsd.org (Ivan Voras) Date: Thu Oct 16 15:11:48 2008 Subject: need help with vbox In-Reply-To: References: <48F70904.8000702@cran.org.uk> Message-ID: Desmond Chapman wrote: > Gentleman, I agree with both of you. > Thanks for everything. Sorry if it seemed terse - I wasn't trying to discourage you. Translating from one Makefile type into another is similar to translating from one programming language to another - you need someone who knows both languages very well. I don't know how many people know both kmk and FreeBSD make well enough to help you (for that matter, both systems have people dedicated to makefiles since they can be very complex, so the majority of developers don't know the full magic behind the makefiles). Since you probably can't use much of the Linux source for your FreeBSD kernel module (you'll probably have to do most of it from scratch), you'd do much better to abandon kmk and write your FreeBSD makefiles with FreeBSD make. You'll have to write new makefiles anyway. On the other hand (I'm not a makefile expert), browsing through http://svn.netlabs.org/kbuild/wiki/kmk it looks like most "new" features are present in FreeBSD's make, though in a different form (and were probably implemented ages ago so they just went ahead and reinvented the wheel again). For example: # Explicit multi-target rules, i.e. explicit make rules that output more than one file. make(1): "Dependency lines consist of one or more targets, an operator... " # Prepend assignment operator I think you can do this with regular variable expansion. # The special .NOTPARALLEL goal has been extended... The .NOTPARALLEL goal exists, but it looks like it's not "extended". Anyway it doesn't matter. # It has some extra predefined variables: You'll have to simulate those with regular variables. # It has a few new builtin functions... FreeBSD's make doesn't have many builtin functions but arithmetic operations work by default (".if $a < 10"). There are no binary operators. Some string functions are present as operators (like "O - Order every word in the variable alphabetically"). You can simulate many functions and operators by invoking shell scripts. # A bunch of builtin utilities which will be invoked without spawning new process or shell. Most of these are taken from BSD. (cp, echo, cat, append...) Though it says they came from BSD, I can't find anything about builtin utilities in make(1). Just use regular shell utilities. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 258 bytes Desc: OpenPGP digital signature Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081016/2f4b2f07/signature.pgp From ivoras at freebsd.org Thu Oct 16 15:45:18 2008 From: ivoras at freebsd.org (Ivan Voras) Date: Thu Oct 16 15:45:25 2008 Subject: [Info required] PC Architecture In-Reply-To: References: Message-ID: Srinivas wrote: > Hello, > > I have a theoretical understanding of the PC architecture and the > details but have no idea of how things go under the hood(for a real > computer). I think it would be very useful for me(as well as > beginners) to know how things work real-time. Even though it is not a > correct mailing list, I am posting this because you guys are the real > hackers and know a lot about how things work inside. As Jeroen said, to get any kind of useful knowledge (e.g. something you can do real work with) you should read the books on the topics. These topics are, from one side, very complex, and from the other, very useless unless you're going to program device drivers and low level kernel facilities. They are rarely (if ever) needed even for hard-core sysadmin work and even most embedded work. If you just want a birds-eye overview, I'll try to say something about the topics (and wait for laughter from the really knowledgeable people :) ). > I would like to know about the following. Plz add anything which you > think will be helpful for the beginners. > 1. Schematic diagrams of motherboards This won't help you in any way except if you need to design motherboards, in which case you need a multi-year education. Any high level overview will be equally good, but the story usually goes "You have this thing called the CPU, right, and you have these other things called memory, the north bridge, the south bridge, the PCI slots. They are all connected with various buses (and I mean really various). The north bridge has the memory controller and some control of the low-level CPU functions. The south bridge has the PCI controller and various other integrated controllers like IDE, COM, sometimes sound, network & graphics". Install "dmidecode" from ports/sysutils and if you're lucky enough to have a good motherboard, you'll see what's connected to what (and how many volts are used by it). > 2. How a bios detects various kinds of buses It usually asks the bus controllers what they know. It knows about the low-level bus controllers from its own embedded data. The BIOS, of course, knows how to communicate with the devices it contacts directly, because it's created for the purpose of supporting them. Note that the "BIOS" is just a software program, executed on the CPU as any other, when the machine boots. Once a modern operating system is loaded, everything is done by its device drivers and the BIOS is almost never accessed after that. > 3. How bios and os detects the different devices present in the system > and what are their capabilities It asks every device found on a bus, found by a bus controller, what is it and what it does. There are really few capabilities that the devices themselves have to say about themselves at this point - most of it is low level stuff like electrical capabilities, what connectors and interrupts they have. Even though devices have broad "groups" of functionalities (like "Ethernet controller") they can report back, the OS and the BIOS cannot really do anything with them - this is where drivers come along. When a new device is found, internal tables are searched to discover if there's a driver that can handle the device (devices are identified by numbers; run "pciconf -lv" for examples). If there is, the driver is left to do with the device as it sees fit. > 4. How interrupts are routed inside Some of the chips or parts of the north bridge chipset are "interrupt controllers", which are programmed by the BIOS and the OS to map device interrupts (i.e. "pci bus X, device Y, function Z") to CPU/software interrupts (like "irq 18"). These interrupts are then handled by specific drivers (run "vmstat -i" for a table which interrupts are used by what driver). > 5. Groups where we can communicate related information I don't know any. Device driver writers or hardware designers would be my best guess. This particular group is read by FreeBSD developers, which are mostly software developers - you'll probably want to seek direct answers someplace that's specialized for the topics. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 258 bytes Desc: OpenPGP digital signature Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081016/48921742/signature.pgp From mezz7 at cox.net Thu Oct 16 15:51:46 2008 From: mezz7 at cox.net (Jeremy Messenger) Date: Thu Oct 16 15:51:52 2008 Subject: need help with vbox In-Reply-To: References: Message-ID: On Thu, 16 Oct 2008 05:17:47 -0000, Desmond Chapman wrote: > > It's dependent upon kbuild. Since the developers have no intention of > fixing the issue, I would like a tutorial on converting the kmk file to > a normal Makefile. I think you are barking at the wrong tree. :-) I don't think the issue is in kBuild. It looks like an issue is in devel/kbuild/Makefile in the do-install target part. http://pointyhat.freebsd.org/errorlogs/sparc64-errorlogs/e.6.20080731104323/kBuild-0.1.3.log ---------------------------------------- (cd ${WRKSRC}/out/freebsd.${MACHINE_ARCH}/release${PREFIX}/bin && ${COPYTREE_BIN} \* ${PREFIX}/bin) ---------------------------------------- ---------------------------------------- # make -V COPYTREE_BIN /bin/sh -c '(/usr/bin/find -d $0 $2 | /usr/bin/cpio -dumpl $1 >/dev/null 2>&1) && /usr/sbin/chown -R root:wheel $1 && /usr/bin/find $1 -type d -exec chmod 755 {} \; && /usr/bin/find $1 -type f -exec chmod 555 {} \;' -- ---------------------------------------- So.. See that $1, it is ${PREFIX}/bin. It's a bug. The COPYTREE_BIN can't have ${PREFIX}/bin. I suggest you to not use COPYTREE_BIN, so do the different method should solve kbuild ports problem. I personal haven't use COPYTREE_* before, so possible misuse COPYTREE_BIN or just can't have ${PREFIX}/bin (uncheck in bsd.port.mk/document). BTW: Add CC'ing to freebsd-ports@ to make its search useful. Cheers, Mezz -- mezz7@cox.net - mezz@FreeBSD.org FreeBSD GNOME Team http://www.FreeBSD.org/gnome/ - gnome@FreeBSD.org From chrisc at vmunix.com Thu Oct 16 16:36:49 2008 From: chrisc at vmunix.com (Chris Coleman) Date: Thu Oct 16 16:36:56 2008 Subject: smbfs hang Message-ID: <8f6775e00810160913y1f30e936ic9c514547ef0827@mail.gmail.com> I'm doing a large transfer from an SMB mounted drive, about 2TB of files. After about 250G, it hanging. Of course any process that tries to access that drive hangs as well. Is there anyway to get those processes killed off and remount the drive without rebooting? I haven't had much luck so far. They don't seem to want to die. -- Chris Coleman -- http://songnumbers.com From attilio at freebsd.org Thu Oct 16 17:17:31 2008 From: attilio at freebsd.org (Attilio Rao) Date: Thu Oct 16 17:18:04 2008 Subject: smbfs hang In-Reply-To: <8f6775e00810160913y1f30e936ic9c514547ef0827@mail.gmail.com> References: <8f6775e00810160913y1f30e936ic9c514547ef0827@mail.gmail.com> Message-ID: <3bbf2fe10810160947y23fa19d3td51b37c9823147a0@mail.gmail.com> 2008/10/16, Chris Coleman : > I'm doing a large transfer from an SMB mounted drive, about 2TB of > files. After about 250G, it hanging. Of course any process that > tries to access that drive hangs as well. > > Is there anyway to get those processes killed off and remount the > drive without rebooting? I haven't had much luck so far. They don't > seem to want to die. Hello Chris, are you equipped with a debugging kernel? Are you interested in helping fixing this bug? Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein From mboxindia at gmail.com Thu Oct 16 18:30:02 2008 From: mboxindia at gmail.com (Srinivas) Date: Thu Oct 16 18:30:10 2008 Subject: [Doubt] Can a PCI device communicate with another PCI or other device? In-Reply-To: <20081015160522.04A824FEE1E@xroff.net> References: <20081015160522.04A824FEE1E@xroff.net> Message-ID: I think a PCI device can communicate with another PCI device directly without the intervention of the CPU. Excerpt from "PCI Express System Architecture" ... PCI Transaction Model - Peer-to-Peer A Peer-to-peer transaction shown as Transaction 3 in Figure 1-5 on page 20 is the direct transfer of data between two PCI devices. A master that wishes to initiate a transaction, arbitrates, wins ownership of the bus and starts a transaction. A target PCI device that recognizes the address claims the bus cycle. For a write bus cycle, data is moved from master to target. For a read bus cycle, data is moved from target to master. .... --S On Wed, Oct 15, 2008 at 9:35 PM, Eduardo Morras wrote: > At 20:32 14/10/2008, you wrote: >> >> Hello, >> >> I have a small doubt. >> >> Suppose I have a PCI card with a general purpose CPU on it. Could it be >> able >> to communicate with another PCI device or ISA device(lets say IDE hard >> disk)? > > You can't do it directly. You must pass through the OS driver that controls > your card. You pass to your driver the data and it send data to other > driver, hard disk, etc.. Note that in some OSs your driver can't pass that > info from one driver to another and need an app that binds your driver with > the other driver. > > >> Thanks, >> Srinivas >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > From steve at Watt.COM Thu Oct 16 23:14:50 2008 From: steve at Watt.COM (Steve Watt) Date: Thu Oct 16 23:14:57 2008 Subject: [Doubt] Can a PCI device communicate with another PCI or other In-Reply-To: References: <20081015160522.04A824FEE1E@xroff.net> Message-ID: <200810162248.m9GMmocw007400@wattres.watt.com> In mboxindia@gmail.com wrote: >I think a PCI device can communicate with another PCI device directly >without the intervention of the CPU. Absolutely. Happens a fair amount in embedded networking systems (among others). The host is responsible for enumeration and resource assignment, but once that's complete, any device on the bus can see all others. Whether that's useful depends rather heavily on the devices on the bus, obviously. TANSTAAFL applies, though, in that multiple initiators must be careful not to step on each others' accesses. -- Steve Watt KD6GGD PP-ASEL-IA ICBM: 121W 56' 57.5" / 37N 20' 15.3" Internet: steve @ Watt.COM Whois: SW32-ARIN Free time? There's no such thing. It just comes in varying prices... From chuckr at telenix.org Thu Oct 16 23:27:53 2008 From: chuckr at telenix.org (Chuck Robey) Date: Thu Oct 16 23:27:59 2008 Subject: indicating a debug image Message-ID: <48F7CD5E.8000905@telenix.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I was wondering, for FreeBSD images, is there a symbol that one could look for, to indicate if image had debug symbols? I know you could destroy that by just stripping, I just wanted to know if there is any way to definitely tell, short of firing up gdb and looking for info. Thanks -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkj3zV4ACgkQz62J6PPcoOn7FQCfSmBRRDLHJE71x/86IF+ELGKi beEAoJLb/2SpNhg9YGmaioGkSCcySVlw =4AbD -----END PGP SIGNATURE----- From neldredge at math.ucsd.edu Fri Oct 17 00:23:48 2008 From: neldredge at math.ucsd.edu (Nate Eldredge) Date: Fri Oct 17 00:23:54 2008 Subject: indicating a debug image In-Reply-To: <48F7CD5E.8000905@telenix.org> References: <48F7CD5E.8000905@telenix.org> Message-ID: On Thu, 16 Oct 2008, Chuck Robey wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > I was wondering, for FreeBSD images, is there a symbol that one could look for, > to indicate if image had debug symbols? I know you could destroy that by just > stripping, I just wanted to know if there is any way to definitely tell, short > of firing up gdb and looking for info. There's really three possibilities: 1. Image has no symbols 2. Image has only non-debug symbols (e.g. global functions and variables) 3. Image has debug symbols (e.g. line numbers, local variables) strip(1) or gcc -s produces #1. gcc without -g produces #2. gcc -g produces #3. You can distinguish #1 because 'nm image' will give no output. nm and objdump don't appear able to distinguish #2 and #3, but readelf -w will give a bunch of output for #3 and none for #2. Does that help? -- Nate Eldredge neldredge@math.ucsd.edu From kmacy at freebsd.org Fri Oct 17 05:53:09 2008 From: kmacy at freebsd.org (Kip Macy) Date: Fri Oct 17 05:53:16 2008 Subject: new mailing list for xen Message-ID: <3c1674c90810162224s4d56ffdcnf6c461d1e28e2384@mail.gmail.com> I've been getting a lot of recurring questions about the status of Xen support in FreeBSD and Xen configuration issues - the answers to which are changing frequently enough that simply adding a FAQ wouldn't make sense. I expect that initially the mailing list will be the "Dailykip", consisting of regular updates about enhancements combined with a steady influx of bug reports and configuration questions. If you're planning on testing out FreeBSD on Xen, please subscribe to freebsd-xen. Cheers, Kip From xfb52 at dial.pipex.com Fri Oct 17 10:26:21 2008 From: xfb52 at dial.pipex.com (Alex Zbyslaw) Date: Fri Oct 17 10:26:33 2008 Subject: smbfs hang In-Reply-To: <8f6775e00810160913y1f30e936ic9c514547ef0827@mail.gmail.com> References: <8f6775e00810160913y1f30e936ic9c514547ef0827@mail.gmail.com> Message-ID: <48F8615A.5070103@dial.pipex.com> Chris Coleman wrote: > I'm doing a large transfer from an SMB mounted drive, about 2TB of > files. After about 250G, it hanging. Of course any process that > tries to access that drive hangs as well. > > Is there anyway to get those processes killed off and remount the > drive without rebooting? I haven't had much luck so far. They don't > seem to want to die. > > I'm pretty sure that your answer is going to be "no". Most of my experience with SMBFS comes in the opposite direction - trying to write large files to an SMB-mounted filesystem and that was so problematic that we switched to NFS. That's a bit more work to set up, but well worth it, imho. See http://technet.microsoft.com/en-us/library/bb463212.aspx So far, fingers cross and touch wood, with Windows virus checking turned off on the remote drive, writing large files has been problem free and tests reading those files have had no problems either. Only talking in the 40Gb range though. hth, --Alex PS The problems we experienced didn't include wedging, iirc. Files would fail to rename, but if you waited 30 seconds and looked for your target file, it would, hey presto, have appeared. But writes also timed out, taking, in this case, a backup job with it. From ip at doom.ctinet.ru Fri Oct 17 10:03:05 2008 From: ip at doom.ctinet.ru (Igor Pokrovsky) Date: Fri Oct 17 11:22:48 2008 Subject: lock test from sh script Message-ID: <20081017094820.GA2013@doom.ctinet.ru> Hi all! I need to check if file is locked or not (with flock) from a shell script. I remember there was something but cannot recall what exactly. And if possible I do not want to write my own test utility even it is several lines in length) Thanks, -ip From des at des.no Fri Oct 17 11:56:56 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Fri Oct 17 11:57:10 2008 Subject: lock test from sh script In-Reply-To: <20081017094820.GA2013@doom.ctinet.ru> (Igor Pokrovsky's message of "Fri, 17 Oct 2008 13:48:20 +0400") References: <20081017094820.GA2013@doom.ctinet.ru> Message-ID: <86abd35ta1.fsf@ds4.des.no> Igor Pokrovsky writes: > I need to check if file is locked or not (with flock) from a shell > script. I remember there was something but cannot recall what exactly. > And if possible I do not want to write my own test utility even it > is several lines in length) lockf -t0 true DES -- Dag-Erling Sm?rgrav - des@des.no From pdegoeje at service2media.com Fri Oct 17 11:53:21 2008 From: pdegoeje at service2media.com (Pieter de Goeje) Date: Fri Oct 17 11:57:57 2008 Subject: lock test from sh script In-Reply-To: <20081017094820.GA2013@doom.ctinet.ru> References: <20081017094820.GA2013@doom.ctinet.ru> Message-ID: <200810171352.46688.pdegoeje@service2media.com> On Friday 17 October 2008, Igor Pokrovsky wrote: > Hi all! > > I need to check if file is locked or not (with flock) from a shell > script. I remember there was something but cannot recall what exactly. > And if possible I do not want to write my own test utility even it > is several lines in length) > > Thanks, > -ip Perhaps you mean lockf(3) ? -- Pieter de Goeje From doconnor at gsoft.com.au Fri Oct 17 12:16:33 2008 From: doconnor at gsoft.com.au (Daniel O'Connor) Date: Fri Oct 17 12:16:45 2008 Subject: lock test from sh script In-Reply-To: <20081017094820.GA2013@doom.ctinet.ru> References: <20081017094820.GA2013@doom.ctinet.ru> Message-ID: <200810172231.05324.doconnor@gsoft.com.au> On Friday 17 October 2008 20:18:20 Igor Pokrovsky wrote: > I need to check if file is locked or not (with flock) from a shell > script. I remember there was something but cannot recall what exactly. > And if possible I do not want to write my own test utility even it > is several lines in length) lockf -s -t 0 /path/to/file /bin/echo -n if [ $? -eq 75 ]; then echo file is locked exit fi -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081017/e4da5d15/attachment.pgp From doconnor at gsoft.com.au Fri Oct 17 12:16:33 2008 From: doconnor at gsoft.com.au (Daniel O'Connor) Date: Fri Oct 17 12:16:46 2008 Subject: lock test from sh script In-Reply-To: <20081017094820.GA2013@doom.ctinet.ru> References: <20081017094820.GA2013@doom.ctinet.ru> Message-ID: <200810172231.05324.doconnor@gsoft.com.au> On Friday 17 October 2008 20:18:20 Igor Pokrovsky wrote: > I need to check if file is locked or not (with flock) from a shell > script. I remember there was something but cannot recall what exactly. > And if possible I do not want to write my own test utility even it > is several lines in length) lockf -s -t 0 /path/to/file /bin/echo -n if [ $? -eq 75 ]; then echo file is locked exit fi -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081017/e4da5d15/attachment-0001.pgp From daniel at dgnetwork.com.br Fri Oct 17 17:56:53 2008 From: daniel at dgnetwork.com.br (=?ISO-8859-1?Q?Daniel_Dias_Gon=E7alves?=) Date: Fri Oct 17 17:57:25 2008 Subject: FreeBSD and ISCSI, Strange Problem In-Reply-To: References: Message-ID: <48F8D1C3.1080607@dgnetwork.com.br> Thanks Danny, The patch work fine, but i have new problem: Have two servers 7.0R with iscsi-2.1. They are mounted the same directory way iSCSI. When I create an archive inside of this directory in the server A, the server B don't show the archive, if in the server B to unmount and mount the directory again, the archive created in the server A appears It. Where it is the problem? The storage is Dell MD3000i. Thanks, Daniel Danny Braniss escreveu: >> the problem is probably that iscsi is deadlocked, so fetch >> ftp://ftp/users/danny/freebsd/iscsi-2.1.tar.gzs;/ftp/;/&.cs.huji.ac.il; >> > ftp://ftp.cs.huji.ac.il/users/danny/freebsd/iscsi-2.1.tar.gz > > >> Danny, >> >> You typed the ftp wrong. >> >> hi Daniel, >> > > oh well, it was before coffee :-) > > >> Obrigado, thanks !! =) >> >> Daniel >> >> >> > > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > > From subbsd at gmail.com Fri Oct 17 20:43:03 2008 From: subbsd at gmail.com (Oleg) Date: Fri Oct 17 20:43:11 2008 Subject: how determine all possible drivers for my system Message-ID: <200810180021.28327.subbsd@gmail.com> Hello maillist of my favorite Wortstation&Server OS May i asking: how i can determine all drivers of existing devices, but who is not present in GENERIC kernel. For definition of all devices useful to me I have written the following a little stupid script: ---------- #!/bin/sh MODLIST=`find /boot/kernel -type f -name *.ko -print` LOGFILE="/var/tmp/kldlog.txt" # ignore for kldload module IGNORE_MODLIST="/boot/kernel/kernel.ko /boot/kernel/tom.ko /boot/kernel/iw_cxgb.ko" # (comment: tom.ko and iw_cwbg.ko after loading make "kernel panic" on my system) REPORTS="/var/tmp/reports.txt" FINAL_FILTER="(already exist)|(File exist)|(Unsupported file type)|\ (already present)|(failed to register)|(Exec format error)|\ (symbol [aA-zZ]++ undefin)|(Weak symbols not supported)|\ (already registered)|(unable to register)" # not know another easy way for fopen("/dev/klog") /etc/rc.d/syslogd stop echo "kern.* ${LOGFILE}" > /var/tmp/syslogd.conf syslogd -f /var/tmp/syslogd.conf # truncate -s0 ${LOGFILE} for MOD in ${MODLIST}; do ignore=0 for IGNORE_MOD in $IGNORE_MODLIST; do if [ "${IGNORE_MOD}" = "${MOD}" ]; then ignore=1 fi done if [ $ignore -eq 1 ]; then echo "Skipping ${MOD}..." >> ${LOGFILE} else echo "Try Loading ${MOD}..." >> ${LOGFILE} sync; kldload ${MOD} >> ${LOGFILE} 2>&1 fi done /etc/rc.d/syslogd start egrep -Eiv "\"${FINAL_FILTER}\"" ${LOGFILE} |tee ${REPORTS} --------- Result working off that scripts is the file reports.txt: { Oct 17 23:51:11 kernel: acpi_aiboost0: on acpi0 Oct 17 23:51:11 kernel: ppc0: cannot reserve I/O port range Oct 17 23:51:12 kernel: acd0: FAILURE - INQUIRY ILLEGAL REQUEST asc=0x24 ascq=0x00 sks=0x48 0x00 0x01 Oct 17 23:51:12 kernel: cd0 at ata3 bus 0 target 0 lun 0 Oct 17 23:51:12 kernel: cd0: Removable CD-ROM SCSI-0 device Oct 17 23:51:12 kernel: cd0: 3.300MB/s transfers Oct 17 23:51:12 kernel: cd0: Attempt to query device size failed: NOT READY, Medium not present Oct 17 23:51:12 kernel: cryptosoft0: on motherboard Oct 17 23:51:13 kernel: This module (opensolaris) contains code covered by the Oct 17 23:51:13 kernel: Common Development and Distribution License (CDDL) Oct 17 23:51:13 kernel: see http://opensolaris.org/os/licensing/opensolaris_license/ Oct 17 23:51:13 kernel: This module (opensolaris) contains code covered by the Oct 17 23:51:13 kernel: Common Development and Distribution License (CDDL) Oct 17 23:51:13 kernel: see http://opensolaris.org/os/licensing/opensolaris_license/ Oct 17 23:51:13 kernel: KLD dtrace.ko: depends on cyclic - not available Oct 17 23:51:13 kernel: KLD dtmalloc.ko: depends on dtrace - not available Oct 17 23:51:13 kernel: This module (opensolaris) contains code covered by the Oct 17 23:51:13 kernel: Common Development and Distribution License (CDDL) Oct 17 23:51:13 kernel: see http://opensolaris.org/os/licensing/opensolaris_license/ Oct 17 23:51:13 kernel: KLD dtrace.ko: depends on cyclic - not available Oct 17 23:51:13 kernel: This module (opensolaris) contains code covered by the Oct 17 23:51:13 kernel: Common Development and Distribution License (CDDL) Oct 17 23:51:13 kernel: see http://opensolaris.org/os/licensing/opensolaris_license/ Oct 17 23:51:13 kernel: KLD profile.ko: depends on cyclic - not available Oct 17 23:51:13 kernel: KLD dtraceall.ko: depends on profile - not available Oct 17 23:51:13 kernel: This module (opensolaris) contains code covered by the Oct 17 23:51:13 kernel: Common Development and Distribution License (CDDL) Oct 17 23:51:13 kernel: see http://opensolaris.org/os/licensing/opensolaris_license/ Oct 17 23:51:13 kernel: KLD profile.ko: depends on cyclic - not available Oct 17 23:51:13 kernel: KLD dtraceall.ko: depends on profile - not available Oct 17 23:51:13 kernel: KLD dtrace_test.ko: depends on dtraceall - not available Oct 17 23:51:13 kernel: This module (opensolaris) contains code covered by the Oct 17 23:51:13 kernel: Common Development and Distribution License (CDDL) Oct 17 23:51:13 kernel: see http://opensolaris.org/os/licensing/opensolaris_license/ Oct 17 23:51:13 kernel: KLD profile.ko: depends on cyclic - not available Oct 17 23:51:13 kernel: This module (opensolaris) contains code covered by the Oct 17 23:51:13 kernel: Common Development and Distribution License (CDDL) Oct 17 23:51:13 kernel: see http://opensolaris.org/os/licensing/opensolaris_license/ Oct 17 23:51:13 kernel: KLD dtrace.ko: depends on cyclic - not available Oct 17 23:51:13 kernel: KLD prototype.ko: depends on dtrace - not available Oct 17 23:51:13 kernel: This module (opensolaris) contains code covered by the Oct 17 23:51:13 kernel: Common Development and Distribution License (CDDL) Oct 17 23:51:13 kernel: see http://opensolaris.org/os/licensing/opensolaris_license/ Oct 17 23:51:13 kernel: KLD dtrace.ko: depends on cyclic - not available Oct 17 23:51:13 kernel: KLD sdt.ko: depends on dtrace - not available Oct 17 23:51:13 kernel: This module (opensolaris) contains code covered by the Oct 17 23:51:13 kernel: Common Development and Distribution License (CDDL) Oct 17 23:51:13 kernel: see http://opensolaris.org/os/licensing/opensolaris_license/ Oct 17 23:51:13 kernel: KLD dtrace.ko: depends on cyclic - not available Oct 17 23:51:13 kernel: KLD systrace.ko: depends on dtrace - not available Oct 17 23:51:13 kernel: This module (opensolaris) contains code covered by the Oct 17 23:51:13 kernel: Common Development and Distribution License (CDDL) Oct 17 23:51:13 kernel: see http://opensolaris.org/os/licensing/opensolaris_license/ Oct 17 23:51:13 kernel: KLD dtrace.ko: depends on cyclic - not available Oct 17 23:51:13 kernel: KLD fbt.ko: depends on dtrace - not available Oct 17 23:51:13 kernel: KLD if_en.ko: depends on utopia - not available Oct 17 23:51:13 kernel: KLD if_fatm.ko: depends on utopia - not available Oct 17 23:51:15 kernel: KLD if_hatm.ko: depends on atm - not available Oct 17 23:51:15 kernel: module_register_init: MOD_LOAD (hwpmc, 0xffffffff804f2440, 0xffffffff80fffce0) error 78 Oct 17 23:51:15 kernel: ichsmb0: port 0x2900-0x293f,0x2d00-0x2d3f,0x2e00-0x2e3f irq 21 at device 10.1 on pci0 Oct 17 23:51:15 kernel: ichsmb0: [ITHREAD] Oct 17 23:51:15 kernel: smbus0: on ichsmb0 Oct 17 23:51:16 kernel: smb0: on smbus0 Oct 17 23:51:16 kernel: ichwd module loaded Oct 17 23:51:16 kernel: ppc0: cannot reserve I/O port range Oct 17 23:51:16 kernel: nfe1f0: Ethernet address: 00:22:15:20:66:1b Oct 17 23:51:16 kernel: nfe1f1: Ethernet address: 00:22:15:20:66:1b Oct 17 23:51:16 kernel: nfe1f2: Ethernet address: 00:22:15:20:66:1b Oct 17 23:51:16 kernel: nfe1f3: Ethernet address: 00:22:15:20:66:1b Oct 17 23:51:16 kernel: nfe0f0: Ethernet address: 00:22:15:20:63:bc Oct 17 23:51:16 kernel: nfe0f1: Ethernet address: 00:22:15:20:63:bc Oct 17 23:51:16 kernel: nfe0f2: Ethernet address: 00:22:15:20:63:bc Oct 17 23:51:16 kernel: nfe0f3: Ethernet address: 00:22:15:20:63:bc Oct 17 23:51:16 kernel: IP Filter: v4.1.28 initialized. Default = pass all, Logging = enabled Oct 17 23:51:16 kernel: ppc0: cannot reserve I/O port range Oct 17 23:51:17 kernel: module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff810a7000, 0) error 1 Oct 17 23:51:17 kernel: module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff810db000, 0) error 1 Oct 17 23:51:17 kernel: module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff8110d000, 0) error 1 Oct 17 23:51:17 kernel: registered firmware set Oct 17 23:51:17 kernel: registered firmware set Oct 17 23:51:17 kernel: registered firmware set Oct 17 23:51:17 kernel: registered firmware set Oct 17 23:51:17 kernel: registered firmware set Oct 17 23:51:17 kernel: registered firmware set Oct 17 23:51:17 kernel: registered firmware set Oct 17 23:51:17 kernel: registered firmware set Oct 17 23:51:17 kernel: registered firmware set Oct 17 23:51:17 kernel: registered firmware set Oct 17 23:51:17 kernel: registered firmware set Oct 17 23:51:17 kernel: unable to register firmware 'isp_1040' Oct 17 23:51:17 kernel: unable to register firmware 'isp_1040_it' Oct 17 23:51:17 kernel: unable to register firmware 'isp_1080' Oct 17 23:51:17 kernel: unable to register firmware 'isp_1080_it' Oct 17 23:51:17 kernel: unable to register firmware 'isp_12160' Oct 17 23:51:17 kernel: unable to register firmware 'isp_12160_it' Oct 17 23:51:17 kernel: unable to register firmware 'isp_2100' Oct 17 23:51:17 kernel: unable to register firmware 'isp_2200' Oct 17 23:51:17 kernel: unable to register firmware 'isp_2300' Oct 17 23:51:17 kernel: unable to register firmware 'isp_2322' Oct 17 23:51:17 kernel: unable to register firmware 'isp_2400' Oct 17 23:51:17 kernel: module_register_init: MOD_LOAD (iwnfw_fw, 0xffffffff812d4000, 0) error 1 Oct 17 23:51:17 kernel: ppc0: cannot reserve I/O port range Oct 17 23:51:18 kernel: KLD mac_biba.ko: depends on kernel_mac_support - not available Oct 17 23:51:18 kernel: KLD mac_bsdextended.ko: depends on kernel_mac_support - not available Oct 17 23:51:18 kernel: KLD mac_ifoff.ko: depends on kernel_mac_support - not available Oct 17 23:51:18 kernel: KLD mac_lomac.ko: depends on kernel_mac_support - not available Oct 17 23:51:18 kernel: KLD mac_mls.ko: depends on kernel_mac_support - not available Oct 17 23:51:18 kernel: KLD mac_none.ko: depends on kernel_mac_support - not available Oct 17 23:51:18 kernel: KLD mac_partition.ko: depends on kernel_mac_support - not available Oct 17 23:51:18 kernel: KLD mac_portacl.ko: depends on kernel_mac_support - not available Oct 17 23:51:18 kernel: KLD mac_seeotheruids.ko: depends on kernel_mac_support - not available Oct 17 23:51:18 kernel: KLD mac_stub.ko: depends on kernel_mac_support - not available Oct 17 23:51:18 kernel: KLD mac_test.ko: depends on kernel_mac_support - not available Oct 17 23:51:18 kernel: KLD if_malo.ko: depends on malofw_fw - not available Oct 17 23:51:18 kernel: ppc0: cannot reserve I/O port range Oct 17 23:51:19 kernel: WARNING: attempt to net_add_domain(bluetooth) after domainfinalize() Oct 17 23:51:19 kernel: WARNING: attempt to net_add_domain(netgraph) after domainfinalize() Oct 17 23:51:20 kernel: This module (opensolaris) contains code covered by the Oct 17 23:51:20 kernel: Common Development and Distribution License (CDDL) Oct 17 23:51:20 kernel: see http://opensolaris.org/os/licensing/opensolaris_license/ Oct 17 23:51:20 kernel: KLD if_patm.ko: depends on atm - not available Oct 17 23:51:20 kernel: ppc0: cannot reserve I/O port range Oct 17 23:51:21 kernel: netsmb_dev: loaded Oct 17 23:51:21 kernel: ppc0: cannot reserve I/O port range Oct 17 23:51:22 last message repeated 2 times Oct 17 23:51:22 kernel: hdac0: mem 0xf9ef0000-0xf9ef3fff irq 21 at device 15.1 on pci0 Oct 17 23:51:22 kernel: hdac0: Oct 17 23:51:22 kernel: hdac0: [ITHREAD] Oct 17 23:51:22 kernel: hdac0: Oct 17 23:51:22 kernel: pcm0: on hdac0 Oct 17 23:51:22 kernel: pcm1: on hdac0 Oct 17 23:51:22 kernel: pcm2: on hdac0 Oct 17 23:51:24 kernel: ppc0: cannot reserve I/O port range Oct 17 23:51:24 last message repeated 2 times Oct 17 23:51:25 kernel: dragon_saver: the console does not support M_VGA_CG320 Oct 17 23:51:25 kernel: module_register_init: MOD_LOAD (dragon_saver, 0xffffffff81637140, 0) error 19 Oct 17 23:51:25 kernel: fire_saver: the console does not support M_VGA_CG320 Oct 17 23:51:25 kernel: module_register_init: MOD_LOAD (fire_saver, 0xffffffff81639000, 0) error 19 Oct 17 23:51:25 kernel: logo_saver: the console does not support M_VGA_CG320 Oct 17 23:51:25 kernel: module_register_init: MOD_LOAD (logo_saver, 0xffffffff8163b010, 0) error 19 Oct 17 23:51:25 kernel: rain_saver: the console does not support M_VGA_CG320 Oct 17 23:51:25 kernel: module_register_init: MOD_LOAD (rain_saver, 0xffffffff8163e010, 0) error 19 Oct 17 23:51:25 kernel: warp_saver: the console does not support M_VGA_CG320 Oct 17 23:51:25 kernel: module_register_init: MOD_LOAD (warp_saver, 0xffffffff81641270, 0) error 19 Oct 17 23:51:26 kernel: KLD if_upgt.ko: depends on upgtfw_fw - not available Oct 17 23:51:26 kernel: wlan: mac acl policy registered } from which I get following useful and working devices for generate my custom kernels: -- device ichwd device smb device smbus device ichsmb device cd device acpi_aiboost device crypto device cryptodev -- My question: what the "official" FreeBSD-way for the approach similar operations? From dnelson at allantgroup.com Fri Oct 17 21:37:34 2008 From: dnelson at allantgroup.com (Dan Nelson) Date: Fri Oct 17 21:37:41 2008 Subject: FreeBSD and ISCSI, Strange Problem In-Reply-To: <48F8D1C3.1080607@dgnetwork.com.br> References: <48F8D1C3.1080607@dgnetwork.com.br> Message-ID: <20081017211028.GK99270@dan.emsphone.com> In the last episode (Oct 17), Daniel Dias Gon?alves said: > Thanks Danny, > > The patch work fine, but i have new problem: > > Have two servers 7.0R with iscsi-2.1. > They are mounted the same directory way iSCSI. When I create an archive > inside of this directory in the server A, the server B don't show the > archive, if in the server B to unmount and mount the directory again, > the archive created in the server A appears It. > > Where it is the problem? You can't mount the same UFS filesystem on two servers at once. You'll end up with a horribly-corrupted filesystem, since neither one is aware of changes the other makes. You would need a shared-storage cluster filessytem to be able to do that (or mount the volume read-only on both servers). Mount the filesystem on one server only, then access it via NFS from the other. -- Dan Nelson dnelson@allantgroup.com From fjwcash at gmail.com Fri Oct 17 22:16:40 2008 From: fjwcash at gmail.com (Freddie Cash) Date: Fri Oct 17 22:16:48 2008 Subject: how determine all possible drivers for my system In-Reply-To: <200810180021.28327.subbsd@gmail.com> References: <200810180021.28327.subbsd@gmail.com> Message-ID: <200810171450.43590.fjwcash@gmail.com> On October 17, 2008 01:21 pm Oleg wrote: > Hello maillist of my favorite Wortstation&Server OS > > May i asking: how i can determine all drivers of existing devices, but > who is not present in GENERIC kernel. > My question: what the "official" FreeBSD-way for the approach similar > operations? I tend to use a more trial-and-error method. Basically, it's run "pciconf -vl" to get a listing of all hardware in the system, and then do searches in man pages, NOTES, and google to find what drivers to test for each device listed as "noneX@" (where X is a number). Most of the time, you can guess what the driver will be called based on the type off device listed in pciconf. -- Freddie Cash fjwcash@gmail.com From alfred at freebsd.org Sat Oct 18 01:07:20 2008 From: alfred at freebsd.org (Alfred Perlstein) Date: Sat Oct 18 01:07:32 2008 Subject: [Info required] PC Architecture In-Reply-To: <20081016144813.GY98121@nexus.in-nomine.org> References: <20081016144813.GY98121@nexus.in-nomine.org> Message-ID: <20081018004900.GB5651@elvis.mu.org> * Jeroen Ruigrok van der Werven [081016 08:06] wrote: > -On [20081016 16:43], Srinivas (mboxindia@gmail.com) wrote: > >I have a theoretical understanding of the PC architecture and the > >details but have no idea of how things go under the hood(for a real > >computer). > > http://www.amazon.com/dp/0123706068/ - Computer Organization and Design: The > Hardware/Software Interface by Patterson and Hennessy > > http://www.amazon.com/dp/0131485210/ - Structured Computer Organization by > Tanenbaum > > That should answer most, if not all, of your questions on that subject. I also REALLY like: The 8088 Project Book http://www.amazon.com/8088-Project-Book-Robert-Grossblatt/dp/0830631712 Although this isn't a real PC, it will give a nice start in case more technical stuff it too much too soon. From max at love2party.net Sat Oct 18 18:18:17 2008 From: max at love2party.net (Max Laier) Date: Sat Oct 18 18:18:25 2008 Subject: conf/128030: [request] Isn't it time to enable IPsec in GENERIC? In-Reply-To: <48FA1756.1080708@freebsd.org> References: <200810181655.m9IGtxWk089117@freefall.freebsd.org> <48FA1756.1080708@freebsd.org> Message-ID: <200810182018.13757.max@love2party.net> On Saturday 18 October 2008 19:05:26 Sam Leffler wrote: > gavin@freebsd.org wrote: > > Synopsis: [request] Isn't it time to enable IPsec in GENERIC? > > > > Responsible-Changed-From-To: freebsd-bugs->freebsd-net > > Responsible-Changed-By: gavin > > Responsible-Changed-When: Sat Oct 18 16:55:14 UTC 2008 > > Responsible-Changed-Why: > > Over to maintainer(s) for consideration > > > > http://www.freebsd.org/cgi/query-pr.cgi?pr=128030 > > Last I checked IPSEC added noticeable overhead. Before anyone does this > you need to measure the cost of having it enabled but not used. It should be possible to turn IPSEC into a module - maybe only loadable on boot to avoid locking issues. This would reduce the overhead to a handful of function pointer checks that should not impact performance (thanks to modern branch prediction and cache sizes). This would have to be measured as well, of course. Maybe this should go to the project page? It's a good junior kernel hacker project, I believe. -- /"\ Best regards, | mlaier@freebsd.org \ / Max Laier | ICQ #67774661 X http://pf4freebsd.love2party.net/ | mlaier@EFnet / \ ASCII Ribbon Campaign | Against HTML Mail and News From sam at freebsd.org Sat Oct 18 20:43:43 2008 From: sam at freebsd.org (Sam Leffler) Date: Sat Oct 18 20:43:49 2008 Subject: conf/128030: [request] Isn't it time to enable IPsec in GENERIC? In-Reply-To: <200810182018.13757.max@love2party.net> References: <200810181655.m9IGtxWk089117@freefall.freebsd.org> <48FA1756.1080708@freebsd.org> <200810182018.13757.max@love2party.net> Message-ID: <48FA4633.9090500@freebsd.org> Max Laier wrote: > On Saturday 18 October 2008 19:05:26 Sam Leffler wrote: > >> gavin@freebsd.org wrote: >> >>> Synopsis: [request] Isn't it time to enable IPsec in GENERIC? >>> >>> Responsible-Changed-From-To: freebsd-bugs->freebsd-net >>> Responsible-Changed-By: gavin >>> Responsible-Changed-When: Sat Oct 18 16:55:14 UTC 2008 >>> Responsible-Changed-Why: >>> Over to maintainer(s) for consideration >>> >>> http://www.freebsd.org/cgi/query-pr.cgi?pr=128030 >>> >> Last I checked IPSEC added noticeable overhead. Before anyone does this >> you need to measure the cost of having it enabled but not used. >> > > It should be possible to turn IPSEC into a module - maybe only loadable on > boot to avoid locking issues. This would reduce the overhead to a handful of > function pointer checks that should not impact performance (thanks to modern > branch prediction and cache sizes). This would have to be measured as well, > of course. Maybe this should go to the project page? It's a good junior > kernel hacker project, I believe. > > I believe the most important issue are the SADB checks in the tx path. It used to be possible to do them cheaply by checking a single ptr value but now it's much more expensive. My memory is hazy as it's been a while. Sam From ivoras at freebsd.org Sat Oct 18 21:17:19 2008 From: ivoras at freebsd.org (Ivan Voras) Date: Sat Oct 18 21:17:26 2008 Subject: Pipes, cat buffer size Message-ID: Hi, I'm working on a program that's intended to be used as a "filter", as in "something | myprogram > file". I'm trying it with cat and I'm seeing my read()s return small blocks, 64 kB in size. I suppose this is because cat writes in 64 kB blocks. So: a) Is there a way to programatically, per-process, set the pipe buffer size? The program in question is a compressor and it's particularly inefficient when given small blocks and I'm wondering if the system can buffer enough data for it. b) Is there any objection to the following patch to cat: Index: cat.c =================================================================== --- cat.c (revision 184033) +++ cat.c (working copy) @@ -247,7 +247,16 @@ if (buf == NULL) { if (fstat(wfd, &sbuf)) err(1, "%s", filename); - bsize = MAX(sbuf.st_blksize, 1024); + if (S_ISREG(sbuf.st_mode)) { + /* If there's plenty of RAM, use a 1 MB + * copy buffer, else use a 128 kB buffer */ + if (sysconf(_SC_PHYS_PAGES) > 32768) + bsize = 1*1024*1024; + else + bsize = 128*1024; + } else + bsize = MAX(sbuf.st_blksize, + (blksize_t)sysconf(_SC_PAGESIZE)); if ((buf = malloc(bsize)) == NULL) err(1, "buffer"); } ? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 258 bytes Desc: OpenPGP digital signature Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081018/79155b4e/signature.pgp From dnelson at allantgroup.com Sat Oct 18 21:35:08 2008 From: dnelson at allantgroup.com (Dan Nelson) Date: Sat Oct 18 21:35:14 2008 Subject: Pipes, cat buffer size In-Reply-To: References: Message-ID: <20081018213502.GL99270@dan.emsphone.com> In the last episode (Oct 18), Ivan Voras said: > I'm working on a program that's intended to be used as a "filter", as > in "something | myprogram > file". I'm trying it with cat and I'm > seeing my read()s return small blocks, 64 kB in size. I suppose this > is because cat writes in 64 kB blocks. So: > > a) Is there a way to programatically, per-process, set the pipe buffer > size? The program in question is a compressor and it's particularly > inefficient when given small blocks and I'm wondering if the system can > buffer enough data for it. Why not keep reading until you reach your desired compression block size? Bzip2's default blocksize is 900k, for example. > b) Is there any objection to the following patch to cat: It might be simpler to just use "dd if=myfile obs=1m" instead of patching cat. -- Dan Nelson dnelson@allantgroup.com From ivoras at freebsd.org Sat Oct 18 22:05:23 2008 From: ivoras at freebsd.org (Ivan Voras) Date: Sat Oct 18 22:05:29 2008 Subject: Pipes, cat buffer size In-Reply-To: <20081018213502.GL99270@dan.emsphone.com> References: <20081018213502.GL99270@dan.emsphone.com> Message-ID: Dan Nelson wrote: > In the last episode (Oct 18), Ivan Voras said: >> I'm working on a program that's intended to be used as a "filter", as >> in "something | myprogram > file". I'm trying it with cat and I'm >> seeing my read()s return small blocks, 64 kB in size. I suppose this >> is because cat writes in 64 kB blocks. So: >> >> a) Is there a way to programatically, per-process, set the pipe buffer >> size? The program in question is a compressor and it's particularly >> inefficient when given small blocks and I'm wondering if the system can >> buffer enough data for it. > > Why not keep reading until you reach your desired compression block > size? Bzip2's default blocksize is 900k, for example. Of course. But that's not the point :) From what I see (didn't look at the code), Linux for example does some kind of internal buffering that decouples how the reader and the writer interact. I think that with FreeBSD's current behaviour the writer could write 1-byte buffers and the reader will be forced to read each byte individually. I don't know if there's some ulterior reason for this. >> b) Is there any objection to the following patch to cat: > > It might be simpler to just use "dd if=myfile obs=1m" instead of > patching cat. I believe patching cat to bring its block size into the century of the fruitbat has its own benefits. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 258 bytes Desc: OpenPGP digital signature Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081018/e23ea5db/signature.pgp From dnelson at allantgroup.com Sat Oct 18 23:12:04 2008 From: dnelson at allantgroup.com (Dan Nelson) Date: Sat Oct 18 23:12:10 2008 Subject: Pipes, cat buffer size In-Reply-To: References: <20081018213502.GL99270@dan.emsphone.com> Message-ID: <20081018231201.GM99270@dan.emsphone.com> In the last episode (Oct 19), Ivan Voras said: > Dan Nelson wrote: > > In the last episode (Oct 18), Ivan Voras said: > >> I'm working on a program that's intended to be used as a "filter", > >> as in "something | myprogram > file". I'm trying it with cat and > >> I'm seeing my read()s return small blocks, 64 kB in size. I > >> suppose this is because cat writes in 64 kB blocks. So: > >> > >> a) Is there a way to programatically, per-process, set the pipe buffer > >> size? The program in question is a compressor and it's particularly > >> inefficient when given small blocks and I'm wondering if the system can > >> buffer enough data for it. > > > > Why not keep reading until you reach your desired compression block > > size? Bzip2's default blocksize is 900k, for example. > > Of course. But that's not the point :) From what I see (didn't look at > the code), Linux for example does some kind of internal buffering that > decouples how the reader and the writer interact. I think that with > FreeBSD's current behaviour the writer could write 1-byte buffers and > the reader will be forced to read each byte individually. I don't know > if there's some ulterior reason for this. No; take a look at /sys/kern/sys_pipe.c . Depending on how much data is in the pipe, it switches between async in-kernel buffering (<8192 bytes), and direct page wiring between sender and receiver (basically zero-copy). > >> b) Is there any objection to the following patch to cat: > > > > It might be simpler to just use "dd if=myfile obs=1m" instead of > > patching cat. > > I believe patching cat to bring its block size into the century of the > fruitbat has its own benefits. -- Dan Nelson dnelson@allantgroup.com From richard.robert at gmail.com Sun Oct 19 00:03:04 2008 From: richard.robert at gmail.com (Robert Richards) Date: Sun Oct 19 00:03:12 2008 Subject: gprocessor Error Message-ID: <43c4e0a0810181635w58e2c4a6tf8d1e9e6563ede7f@mail.gmail.com> Hi: I have been trying to upgrade to the latest gnash on my W/S running: FreeBSD 6.3-RELEASE-p5 #3. First, I upgraded the kernel and base to the latest patch level, then I executed a portupgrade -R gnash, and after all the pre-requisite packages were upgraded successfully gnash itself began to compile. Then this: =========================== ../libcore/.libs/libgnashcore.so: undefined reference to `std::basic_ostream >& std::basic_ostream >::_M_insert(bool)' /usr/ports/graphics/gnash/work/gnash-0.8.4/libbase/.libs/libgnashbase.so: undefined reference to `std::basic_istream >& std::basic_istream >::_M_extract(unsigned int&)' collect2: ld returned 1 exit status gmake[2]: *** [gprocessor] Error 1 gmake[2]: Leaving directory `/usr/ports/graphics/gnash/work/gnash-0.8.4/utilities' gmake[1]: *** [all-recursive] Error 1 gmake[1]: Leaving directory `/usr/ports/graphics/gnash/work/gnash-0.8.4' gmake: *** [all] Error 2 *** Error code 2 Stop in /usr/ports/graphics/gnash. *** Error code 1 Stop in /usr/ports/graphics/gnash. ** Command failed [exit code 1]: /usr/bin/script -qa /tmp/portupgrade.43253.1 env UPGRADE_TOOL=portupgrade UPGRADE_PORT=gnash-0.8.2_2 UPGRADE_PORT_VER=0.8.2_2 make ** Fix the problem and try again. ** Listing the failed packages (-:ignored / *:skipped / !:failed) ! graphics/gnash (gnash-0.8.2_2) (new compiler error) =========================== What do I need to do to get this accomplished? I am struggling with limited flash support, and have been doing fairly well with gnash/firefox. I would like to upgrade my gnash-0.8.2_2 to the latest version gnash-0.8.4. and possibly get a bit more functionality. Oh, one more possible clue. The first time I attempted this, I issued a portupgrade gnash without recursion. At about the same point of failure, the system powered down. I mean I saw an LD error, and then POOF, laptop went power-down! I didn't believe that was the cause, so after cleaning the file systems, I tried again, and again it powered down. So I pretty much upgraded everything and after the -R attempt, gnash simply failed. TIA Bob From ivoras at freebsd.org Sun Oct 19 00:19:29 2008 From: ivoras at freebsd.org (Ivan Voras) Date: Sun Oct 19 00:19:36 2008 Subject: Pipes, cat buffer size In-Reply-To: <20081018231201.GM99270@dan.emsphone.com> References: <20081018213502.GL99270@dan.emsphone.com> <20081018231201.GM99270@dan.emsphone.com> Message-ID: <9bbcef730810181719x4387a14yec74bdb6893d1a2a@mail.gmail.com> 2008/10/19 Dan Nelson : > In the last episode (Oct 19), Ivan Voras said: >> Of course. But that's not the point :) From what I see (didn't look at >> the code), Linux for example does some kind of internal buffering that >> decouples how the reader and the writer interact. I think that with >> FreeBSD's current behaviour the writer could write 1-byte buffers and >> the reader will be forced to read each byte individually. I don't know >> if there's some ulterior reason for this. > > No; take a look at /sys/kern/sys_pipe.c . Depending on how much data > is in the pipe, it switches between async in-kernel buffering (<8192 > bytes), and direct page wiring between sender and receiver (basically > zero-copy). Ok, maybe it's just not behaving as I thought it should. See this test program: ---- #include #include #include #define BSIZE (1024*1024) void main() { int r; char buf[BSIZE]; while (1) { r = read(0, buf, BSIZE); fprintf(stderr, "read %d bytes\n", r); if (r <= 0) break; } } ---- and this command line: > dd bs=1 if=/dev/zero| ./reader The output of this on RELENG_7 is: read 8764 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes ... The first value puzzles me - so it actually is doing some kind of buffering. Linux isn't actually much better, but the intention is there: $ dd if=/dev/zero bs=1 | ./bla read 1 bytes read 38 bytes read 8 bytes read 2 bytes read 2 bytes read 2 bytes read 2 bytes read 4 bytes read 3 bytes read 2 bytes read 2 bytes read 2 bytes read 2 bytes read 2 bytes read 2 bytes read 3 bytes read 3 bytes read 112 bytes read 2 bytes read 2 bytes ... Maybe FreeBSD switches between the writer and the reader too soon so the buffer doesn't get filled? Using cat (which started all this), FreeBSD consistently processes 4096 byte buffers, while Linux's sizes are all over the place - from 4 kB to 1 MB, randomly fluctuating. My goal would be (if it's possible - it might not be) to maximize coalescing in an environment where the reader does something with the data (e.g. compression) so there should be a reasonable amount of backlogged input data. But if it works in general, it may simply be that it isn't really applicable to my purpose (and I should modify the reader to read multiple blocks). Though it won't help me, I still think that modifying cat is worth it :) From dnelson at allantgroup.com Sun Oct 19 00:50:24 2008 From: dnelson at allantgroup.com (Dan Nelson) Date: Sun Oct 19 00:50:31 2008 Subject: Pipes, cat buffer size In-Reply-To: <9bbcef730810181719x4387a14yec74bdb6893d1a2a@mail.gmail.com> References: <20081018213502.GL99270@dan.emsphone.com> <20081018231201.GM99270@dan.emsphone.com> <9bbcef730810181719x4387a14yec74bdb6893d1a2a@mail.gmail.com> Message-ID: <20081019005021.GN99270@dan.emsphone.com> In the last episode (Oct 19), Ivan Voras said: > 2008/10/19 Dan Nelson : > > In the last episode (Oct 19), Ivan Voras said: > >> Of course. But that's not the point :) From what I see (didn't > >> look at the code), Linux for example does some kind of internal > >> buffering that decouples how the reader and the writer interact. I > >> think that with FreeBSD's current behaviour the writer could write > >> 1-byte buffers and the reader will be forced to read each byte > >> individually. I don't know if there's some ulterior reason for > >> this. > > > > No; take a look at /sys/kern/sys_pipe.c . Depending on how much > > data is in the pipe, it switches between async in-kernel buffering > > (<8192 bytes), and direct page wiring between sender and receiver > > (basically zero-copy). > > Ok, maybe it's just not behaving as I thought it should. See this > test program: [ program that prints the amount of data in each read() ] > and this command line: > > > dd bs=1 if=/dev/zero| ./reader > > The output of this on RELENG_7 is: > > read 8764 bytes > read 1 bytes [..] > read 1 bytes > read 1 bytes > ... > > The first value puzzles me - so it actually is doing some kind of > buffering. Linux isn't actually much better, but the intention is > there: > > $ dd if=/dev/zero bs=1 | ./bla > read 1 bytes > read 38 bytes > read 8 bytes > read 2 bytes [..] > read 2 bytes > read 3 bytes > read 3 bytes > read 112 bytes > read 2 bytes > read 2 bytes > ... > > Maybe FreeBSD switches between the writer and the reader too soon so > the buffer doesn't get filled? If your reader isn't doing any real work between reads, it is always reading, so the pipe will never fill up. The delay in FreeBSD was probably due to the shell spawning the writer first, so it buffered up 8k of data before the reader was ready. After that, the reader was able to pull data as fast as the writer pushed. > Using cat (which started all this), FreeBSD consistently processes > 4096 byte buffers, while Linux's sizes are all over the place - from > 4 kB to 1 MB, randomly fluctuating. My goal would be (if it's > possible - it might not be) to maximize coalescing in an environment > where the reader does something with the data (e.g. compression) so > there should be a reasonable amount of backlogged input data. Remember that increasing coelescing also increases latency and decreases the parallelism between reader and writer (since if you coalesce you cause the reader to wait for data that's already been writen, in the hopes that the writer will write again soon). > But if it works in general, it may simply be that it isn't really > applicable to my purpose (and I should modify the reader to read > multiple blocks). That's my suggestion, yes. That way your program would also work when passed data from an internet socket (where you will get varying read() sizes too). It wouldn't add more than 10 lines to wrap your read in a loop that exits when your preferred size has been reached. > Though it won't help me, I still think that modifying cat is worth it :) -- Dan Nelson dnelson@allantgroup.com From kientzle at freebsd.org Sun Oct 19 01:05:53 2008 From: kientzle at freebsd.org (Tim Kientzle) Date: Sun Oct 19 01:05:58 2008 Subject: Pipes, cat buffer size In-Reply-To: <9bbcef730810181719x4387a14yec74bdb6893d1a2a@mail.gmail.com> References: <20081018213502.GL99270@dan.emsphone.com> <20081018231201.GM99270@dan.emsphone.com> <9bbcef730810181719x4387a14yec74bdb6893d1a2a@mail.gmail.com> Message-ID: <48FA821A.8070200@freebsd.org> >>>... the writer could write 1-byte buffers and >>>the reader will be forced to read each byte individually. >> >>No; take a look at /sys/kern/sys_pipe.c . Depending on how much data >>is in the pipe, it switches between async in-kernel buffering (<8192 >>bytes), and direct page wiring between sender and receiver (basically >>zero-copy). > > Ok, maybe it's just not behaving as I thought it should. See this test program: However, when I add "sleep(1)" to your test program, I see the following output: $ dd bs=1 if=/dev/zero | ./reader read 2282 bytes read 65536 bytes read 65536 bytes read 65536 bytes read 65536 bytes read 65536 bytes read 65536 bytes read 65536 bytes read 65536 bytes read 65536 bytes read 65536 bytes In your original example, because your sample reader does nothing, it sees each write separately. If the reader was actually doing some work then the pipe would buffer up data while your reader was busy. This looks like exactly the right behavior: The reader will only block if there is no data in the pipe at all; the writer will only block if it gets "too far ahead" of the reader. Except in those cases, each program gets to do I/O as fast as it can. If your program needs larger blocks, it should keep reading until it gets enough data. (The -B option of GNU tar is an example of this sort of behavior.) Tim From fbsd.hackers at rachie.is-a-geek.net Mon Oct 20 14:13:28 2008 From: fbsd.hackers at rachie.is-a-geek.net (Mel) Date: Mon Oct 20 14:13:35 2008 Subject: Pipes, cat buffer size In-Reply-To: <20081019005021.GN99270@dan.emsphone.com> References: <9bbcef730810181719x4387a14yec74bdb6893d1a2a@mail.gmail.com> <20081019005021.GN99270@dan.emsphone.com> Message-ID: <200810201613.25864.fbsd.hackers@rachie.is-a-geek.net> On Sunday 19 October 2008 02:50:22 Dan Nelson wrote: > > But if it works in general, it may simply be that it isn't really > > applicable to my purpose (and I should modify the reader to read > > multiple blocks). > > That's my suggestion, yes. That way your program would also work when > passed data from an internet socket (where you will get varying read() > sizes too). It wouldn't add more than 10 lines to wrap your read in a > loop that exits when your preferred size has been reached. Since you mention a socket, would this patch be a good idea and use kqueue to read from the pipe? I would think that having the kernel fill the buffer, rather then a busy loop kernel/userland would improve speed, but I'm not too familiar with the code to know if this causes any problems. diff -u -r1.191.2.3 sys_pipe.c --- sys_pipe.c 6 Jun 2008 12:17:28 -0000 1.191.2.3 +++ sys_pipe.c 20 Oct 2008 14:04:18 -0000 @@ -1594,7 +1609,10 @@ PIPE_UNLOCK(rpipe); return (1); } - ret = kn->kn_data > 0; + if ( kn->kn_sfflags & NOTE_LOWAT) + ret = kn->kn_data >= kn->kn_sdata; + else + ret = kn->kn_data > 0; PIPE_UNLOCK(rpipe); return ret; } -- Mel From nkoch at demig.de Tue Oct 21 08:13:50 2008 From: nkoch at demig.de (Norbert Koch) Date: Tue Oct 21 08:14:02 2008 Subject: 'libc_r: enter/leave_cancellation_point() Message-ID: <48FD88B1.3050106@demig.de> Hello, I was just inspecting libc_r for trying to understand some things and found this: <------------------------- --- src/lib/libc_r/uthread/uthread_cond.c 2002/05/24 04:32:28 1.33 +++ src/lib/libc_r/uthread/uthread_cond.c 2002/11/13 18:13:26 1.34 ... int -_pthread_cond_signal(pthread_cond_t * cond) +__pthread_cond_timedwait(pthread_cond_t *cond, pthread_mutex_t *mutex, + const struct timespec *abstime) +{ + int ret; + + _thread_enter_cancellation_point(); + ret = _pthread_cond_timedwait(cond, mutex, abstime); + _thread_enter_cancellation_point(); + return (ret); +} ----------------------------> Shouldn't that be _thread_leave_cancellation_point() after calling _pthread_cond_timedwait() ? What effect should I see if this is wrong? Best regards, Norbert Koch From alfred at freebsd.org Tue Oct 21 21:04:45 2008 From: alfred at freebsd.org (Alfred Perlstein) Date: Tue Oct 21 21:04:56 2008 Subject: 'libc_r: enter/leave_cancellation_point() In-Reply-To: <48FD88B1.3050106@demig.de> References: <48FD88B1.3050106@demig.de> Message-ID: <20081021210444.GB22503@elvis.mu.org> Hey Norbert, this is probably a bug, but might not be addressed because libc_r is not really supported any longer. Someone may pick it up, but I'm uncertain of that. * Norbert Koch [081021 01:32] wrote: > Hello, > > I was just inspecting libc_r for trying to understand > some things and found this: > > <------------------------- > > --- src/lib/libc_r/uthread/uthread_cond.c 2002/05/24 04:32:28 1.33 > +++ src/lib/libc_r/uthread/uthread_cond.c 2002/11/13 18:13:26 1.34 > > ... > > int > -_pthread_cond_signal(pthread_cond_t * cond) > +__pthread_cond_timedwait(pthread_cond_t *cond, pthread_mutex_t *mutex, > + const struct timespec *abstime) > +{ > + int ret; > + > + _thread_enter_cancellation_point(); > + ret = _pthread_cond_timedwait(cond, mutex, abstime); > + _thread_enter_cancellation_point(); > + return (ret); > +} > > > ----------------------------> > > Shouldn't that be _thread_leave_cancellation_point() after > calling _pthread_cond_timedwait() ? > What effect should I see if this is wrong? > > Best regards, > > Norbert Koch > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" -- - Alfred Perlstein From gamato at users.sf.net Tue Oct 21 22:25:11 2008 From: gamato at users.sf.net (martinko) Date: Tue Oct 21 22:25:18 2008 Subject: Laptop suggestions? In-Reply-To: References: <1216910072.2251.8.camel@jill.exit.com> Message-ID: Matt Olander wrote: > On Jul 25, 2008, at 3:23 PM, Jeremy Messenger wrote: > >> On Thu, 24 Jul 2008 09:34:32 -0500, Frank Mayhar wrote: >> >>> My old Dell Inspiron 5160 has developed problems that I can't fix, sigh, >>> so it's time to replace it. I'm hoping for some good suggestions from >>> this list (cc'd to hackers for the exposure, I know everyone doesn't >>> read -mobile). >>> >>> My criteria: >>> * 3D acceleration. >>> * MiniPCI wireless (don't care which card, I'll replace it >>> anyway). >>> * At least 15" screen. >>> * Decent power consumption. >>> * Plays well with FreeBSD 7-stable. >>> >>> Nice to have: >>> * Dual core. >>> * >4GB memory. >>> * Working suspend/hibernate mode (and no, I'm not holding my >>> breath). >>> >>> So, suggestions? BTW, if I get a decent response I'll summarize it for >>> the list, along with the one I chose and my experience after >>> ordering/installing it. >> >> Maybe you can wait for this: >> >> http://www.ixsystems.com/products/bsd-laptop.html > > Hi everyone! I actually had our prototype of this laptop up at the OSCON > show in Portland and it was pretty well received. > Everything works for the most part although we're still tweaking some > things for ACPI. > > I'll have one at the FreeBSD booth at LinuxWorld in San Francisco next > week, August 5-7. We'll announce as soon as this thing is 100% and we're > comfortable bringing the product line up as an item that we're > comfortable supporting long term. Most likely, available to the general > public in September. > > best, > -matt > Hi, I have always thought that Fn key in left most bottom corner of the keyboard is, especially for programmers, a very bad idea. :-( Otherwise it looks very promising although DVI or HDMI video output would be very welcome these days as would be built-in Bluetooth. (Btw thanks for RS232!:)) Cheers, Martin From des at des.no Wed Oct 22 11:06:31 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed Oct 22 11:06:38 2008 Subject: Laptop suggestions? In-Reply-To: (martinko's message of "Tue, 21 Oct 2008 23:49:09 +0200") References: <1216910072.2251.8.camel@jill.exit.com> Message-ID: <86fxmox51m.fsf@ds4.des.no> martinko writes: > I have always thought that Fn key in left most bottom corner of the > keyboard is, especially for programmers, a very bad idea. :-( Seconded. Worse still, on my Lenovo T60, if the Fn key is held down longer than a fraction of a second, it generates an input event which just happens to correspond to Gnome's default key binding for the "next track" function in media players... DES -- Dag-Erling Sm?rgrav - des@des.no From kline at thought.org Wed Oct 22 17:57:40 2008 From: kline at thought.org (Gary Kline) Date: Wed Oct 22 19:43:03 2008 Subject: Laptop suggestions? In-Reply-To: <86fxmox51m.fsf@ds4.des.no> References: <1216910072.2251.8.camel@jill.exit.com> <86fxmox51m.fsf@ds4.des.no> Message-ID: <20081022173634.GA57706@thought.org> On Wed, Oct 22, 2008 at 01:06:29PM +0200, Dag-Erling Sm?rgrav wrote: > martinko writes: > > I have always thought that Fn key in left most bottom corner of the > > keyboard is, especially for programmers, a very bad idea. :-( > > Seconded. Worse still, on my Lenovo T60, if the Fn key is held down > longer than a fraction of a second, it generates an input event which > just happens to correspond to Gnome's default key binding for the "next > track" function in media players... > I've seen that Fn key, but don't know what it is for. What? you press it, then follow with the integers [ 1, 2, 3 ... ]? At any rate, maybe you can remap the key with ~/.xmodmaprc. -g > DES > -- > Dag-Erling Sm?rgrav - des@des.no > _______________________________________________ > freebsd-mobile@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-mobile > To unsubscribe, send any mail to "freebsd-mobile-unsubscribe@freebsd.org" -- Gary Kline kline@thought.org http://www.thought.org Public Service Unix http://jottings.thought.org http://transfinite.thought.org From neldredge at math.ucsd.edu Wed Oct 22 20:06:36 2008 From: neldredge at math.ucsd.edu (Nate Eldredge) Date: Wed Oct 22 20:06:43 2008 Subject: Laptop suggestions? In-Reply-To: <20081022173634.GA57706@thought.org> References: <1216910072.2251.8.camel@jill.exit.com> <86fxmox51m.fsf@ds4.des.no> <20081022173634.GA57706@thought.org> Message-ID: On Wed, 22 Oct 2008, Gary Kline wrote: > On Wed, Oct 22, 2008 at 01:06:29PM +0200, Dag-Erling Sm?rgrav wrote: >> martinko writes: >>> I have always thought that Fn key in left most bottom corner of the >>> keyboard is, especially for programmers, a very bad idea. :-( >> >> Seconded. Worse still, on my Lenovo T60, if the Fn key is held down >> longer than a fraction of a second, it generates an input event which >> just happens to correspond to Gnome's default key binding for the "next >> track" function in media players... >> > > > I've seen that Fn key, but don't know what it is for. What? you press > it, then follow with the integers [ 1, 2, 3 ... ]? At any rate, maybe > you can remap the key with ~/.xmodmaprc. Fn is usually used on laptop keyboards to allow two logical keys to share a single physical key. For example, see the keyboard pictured at http://www.notebookreview.com/assets/3415.jpg . On the extreme lower right is a key with "->" in white and "End" in blue. Pressing it by itself sends the keycode corresponding to an ordinary keyboard's "->" key. Holding Fn and pressing that key sends the keycode corresponding to an ordinary keyboard's "End" key. On many keyboards, pressing Fn by itself sends no keycode at all, so it cannot be remapped. It is also sometimes used to control hardware features which on a desktop machine might have a different interface. For instance, on the laptop pictured, holding Fn and pressing F6 would increase the screen brightness, probably without sending a keycode. A desktop machine would probably have a button on the monitor itself to do this. -- Nate Eldredge neldredge@math.ucsd.edu From koitsu at FreeBSD.org Wed Oct 22 20:31:46 2008 From: koitsu at FreeBSD.org (Jeremy Chadwick) Date: Wed Oct 22 20:31:53 2008 Subject: Laptop suggestions? In-Reply-To: References: <1216910072.2251.8.camel@jill.exit.com> <86fxmox51m.fsf@ds4.des.no> <20081022173634.GA57706@thought.org> Message-ID: <20081022203143.GA67740@icarus.home.lan> On Wed, Oct 22, 2008 at 01:06:20PM -0700, Nate Eldredge wrote: > On Wed, 22 Oct 2008, Gary Kline wrote: > >> On Wed, Oct 22, 2008 at 01:06:29PM +0200, Dag-Erling Sm?rgrav wrote: >>> martinko writes: >>>> I have always thought that Fn key in left most bottom corner of the >>>> keyboard is, especially for programmers, a very bad idea. :-( >>> >>> Seconded. Worse still, on my Lenovo T60, if the Fn key is held down >>> longer than a fraction of a second, it generates an input event which >>> just happens to correspond to Gnome's default key binding for the "next >>> track" function in media players... >> >> I've seen that Fn key, but don't know what it is for. What? you press >> it, then follow with the integers [ 1, 2, 3 ... ]? At any rate, maybe >> you can remap the key with ~/.xmodmaprc. > > Fn is usually used on laptop keyboards to allow two logical keys to share > a single physical key. For example, see the keyboard pictured at > http://www.notebookreview.com/assets/3415.jpg . On the extreme lower > right is a key with "->" in white and "End" in blue. Pressing it by > itself sends the keycode corresponding to an ordinary keyboard's "->" > key. Holding Fn and pressing that key sends the keycode corresponding to > an ordinary keyboard's "End" key. On many keyboards, pressing Fn by > itself sends no keycode at all, so it cannot be remapped. > > It is also sometimes used to control hardware features which on a desktop > machine might have a different interface. For instance, on the laptop > pictured, holding Fn and pressing F6 would increase the screen > brightness, probably without sending a keycode. A desktop machine would > probably have a button on the monitor itself to do this. I always figured "Fn" was a good name for the key, given that it resembles the expletive that comes forth from my mouth when intending to hit Control. http://www.notebookreview.com/assets/9328.jpg ;-) -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From kline at thought.org Wed Oct 22 22:59:51 2008 From: kline at thought.org (Gary Kline) Date: Wed Oct 22 23:20:15 2008 Subject: Laptop suggestions? In-Reply-To: <20081022203143.GA67740@icarus.home.lan> References: <1216910072.2251.8.camel@jill.exit.com> <86fxmox51m.fsf@ds4.des.no> <20081022173634.GA57706@thought.org> <20081022203143.GA67740@icarus.home.lan> Message-ID: <1224716380.58305.44.camel@tao.thought.org> On Wed, 2008-10-22 at 13:31 -0700, Jeremy Chadwick wrote: > On Wed, Oct 22, 2008 at 01:06:20PM -0700, Nate Eldredge wrote: > > On Wed, 22 Oct 2008, Gary Kline wrote: > > > >> On Wed, Oct 22, 2008 at 01:06:29PM +0200, Dag-Erling Sm?rgrav wrote: > >>> martinko writes: > >>>> I have always thought that Fn key in left most bottom corner of the > >>>> keyboard is, especially for programmers, a very bad idea. :-( > >>> > >>> Seconded. Worse still, on my Lenovo T60, if the Fn key is held down > >>> longer than a fraction of a second, it generates an input event which > >>> just happens to correspond to Gnome's default key binding for the "next > >>> track" function in media players... > >> > >> I've seen that Fn key, but don't know what it is for. What? you press > >> it, then follow with the integers [ 1, 2, 3 ... ]? At any rate, maybe > >> you can remap the key with ~/.xmodmaprc. > > > > Fn is usually used on laptop keyboards to allow two logical keys to share > > a single physical key. For example, see the keyboard pictured at > > http://www.notebookreview.com/assets/3415.jpg . On the extreme lower > > right is a key with "->" in white and "End" in blue. Pressing it by > > itself sends the keycode corresponding to an ordinary keyboard's "->" > > key. Holding Fn and pressing that key sends the keycode corresponding to > > an ordinary keyboard's "End" key. On many keyboards, pressing Fn by > > itself sends no keycode at all, so it cannot be remapped. > > > > It is also sometimes used to control hardware features which on a desktop > > machine might have a different interface. For instance, on the laptop > > pictured, holding Fn and pressing F6 would increase the screen > > brightness, probably without sending a keycode. A desktop machine would > > probably have a button on the monitor itself to do this. Thanks for clearing up a back-of-mind mystery since I bought my 600E in 2003; I kept hitting the "Fn" for the ^ key, and *nothing happened* so I had to re-type the control sequence. It is an ill-planned layout and I'm sure that 'BM has heard about it from us hacker types. --Why this is the best list in the (known) universe. Seriously. > > I always figured "Fn" was a good name for the key, given that it > resembles the expletive that comes forth from my mouth when intending to > hit Control. That ain't that much of a joke, Jeremy. unless I'm at my desk with wrist-rest I can barely reach the back keys. [shoulder problems]. So far I've invented around 7--maybe 8--new profanities. BTW, if that jpeg is a Lenovo, is that a scratch-and-sniff pad below the mouse buttons? (The TPad's *did* need a redesign, but for me, the trakmouse/trakstick/<> was perfect. My left paw went right there.) ...FWIW, I just bought a G41 (3.06GHz) pre-Lenovo. gary > > http://www.notebookreview.com/assets/9328.jpg > > ;-) > From des at des.no Thu Oct 23 11:41:36 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Thu Oct 23 11:41:48 2008 Subject: Laptop suggestions? In-Reply-To: <20081022173634.GA57706@thought.org> (Gary Kline's message of "Wed, 22 Oct 2008 10:36:34 -0700") References: <1216910072.2251.8.camel@jill.exit.com> <86fxmox51m.fsf@ds4.des.no> <20081022173634.GA57706@thought.org> Message-ID: <86fxmn35e9.fsf@ds4.des.no> Gary Kline writes: > I've seen that Fn key, but don't know what it is for. What? you press > it, then follow with the integers [ 1, 2, 3 ... ]? At any rate, maybe > you can remap the key with ~/.xmodmaprc. They're used to access keys which won't physically fit on a laptop keyboard, such as the numeric keypad, NumLock, ScrollLock etc., and (along with function keys) to control hardware-specific functions like switching between internal and external display, turning bluetooth and wlan on and off, adjusting the backlight brightness, etc. DES -- Dag-Erling Sm?rgrav - des@des.no From des at des.no Thu Oct 23 13:23:36 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Thu Oct 23 13:23:48 2008 Subject: Laptop suggestions? In-Reply-To: <20081023235822.I4254@sola.nimnet.asn.au> (Ian Smith's message of "Fri, 24 Oct 2008 00:08:59 +1100 (EST)") References: <1216910072.2251.8.camel@jill.exit.com> <86fxmox51m.fsf@ds4.des.no> <20081022173634.GA57706@thought.org> <86fxmn35e9.fsf@ds4.des.no> <20081023235822.I4254@sola.nimnet.asn.au> Message-ID: <86od1b1m3s.fsf@ds4.des.no> Ian Smith writes: > Re your original issue, can you get any mileage out of using acpi_ibm, > devd and this post and/or the other one it references: The laptop in question does not run FreeBSD. I gave up running FreeBSD on any sort of desktop or laptop computer years ago. DES -- Dag-Erling Sm?rgrav - des@des.no From ravi.murty at intel.com Thu Oct 23 18:21:57 2008 From: ravi.murty at intel.com (Murty, Ravi) Date: Thu Oct 23 18:22:04 2008 Subject: sched_thread_priority in ULE 8.0 Message-ID: <6D5D25EA3941074EB7734E51B166870401DDFA1D@orsmsx506.amr.corp.intel.com> Hello All, This is something I've been trying to figure out in the last couple of hours, but can't seem to understand. Sched_thread_priority() updates a threads priority to "prio". If the thread is on the RUNQ, we have to pull it out and put it back at a different spot on the same queue. However, if the thread is currently running, I am missing something. If it is running shouldn't tdq->tdq_lowpri basically be same as the threads current priority (aka oldpri).. I can't seem to figure out why we need the else if check and a call to tdq_lowpri. If the thread is running and we've boosted the thread's priority, then simply changing tdq_lowpri should do it right? I've included part of the sched_thread_priority code below. Thanks Ravi The code is as follows: ... If (prio < tdq->tdq_lowpri) tdq->tdq_lowpri else if (tdq->tdq_lowpri == oldpri) tdq_setlowpri(tdq, td); From pisymbol at gmail.com Fri Oct 24 00:52:08 2008 From: pisymbol at gmail.com (Alexander Sack) Date: Fri Oct 24 00:52:15 2008 Subject: Why does adding /usr/lib32 to LD_LIBRARY_PATH break 64-bit binaries? Message-ID: <3c0b01820810231731s1b4d4659j7d1df8bf4abb229c@mail.gmail.com> Hello: I have some weird behavior I'm trying to figure out and was wondering if someone can point me in the right direction. I'm running a FreeBSD 6.1-RELEASE-amd64 machine. If I add /usr/lib32 to my LD_LIBRARY_PATH it breaks all of my binaries on my 64-bit machine. For example: [root@hagen ~]# file /bin/ls /bin/ls: ELF 64-bit LSB executable, AMD x86-64, version 1 (FreeBSD), dynamically linked (uses shared libs), stripped [root@hagen ~]# ldd /bin/ls /bin/ls: libutil.so.5 => /lib/libutil.so.5 (0x800630000) libncurses.so.6 => /lib/libncurses.so.6 (0x80073d000) libc.so.6 => /lib/libc.so.6 (0x800896000) [root@hagen ~]# ls -l /libexec/ total 306 -r-xr-xr-x 1 root wheel 163864 Aug 21 2007 ld-elf.so.1 -r-xr-xr-x 1 root wheel 146420 Aug 21 2007 ld-elf32.so.1 [root@hagen ~]# export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/bin:/usr/lib:/usr/lib32:/usr/lib64 [root@hagen ~]# ls /libexec/ld-elf.so.1: /usr/lib32/libutil.so.5: unsupported file layout [root@hagen ~]# unset LD_LIBRARY_PATH [root@hagen ~]# ls (normal ls output) [root@hagen ~]# ls -l /usr/lib/libut* -r--r--r-- 1 root wheel 100518 Aug 21 2007 /usr/lib/libutil.a lrwxrwxrwx 1 root wheel 17 Sep 11 11:44 /usr/lib/libutil.so -> /lib/libutil.so.5 -r--r--r-- 1 root wheel 103846 Aug 21 2007 /usr/lib/libutil_p.a [root@hagen ~]# file /lib/libutil.so.5 /lib/libutil.so.5: ELF 64-bit LSB shared object, AMD x86-64, version 1 (FreeBSD), stripped I would ASSUME that rtld would look at my LD_LIBRARY_PATH and use /usr/lib to find libraries, not /usr/lib32. Why does it insist on picking /usr/lib32 when "/bin/ls" is CLEARLY a 64-bit binary? This doesn't make complete sense to me just yet. Someone I'm sure is going "don't do that" and I agree. The issue is I'm porting a library/framework (boost) and it creates a runtime LD_LIBRARY_PATH for its gcc toolchain with the above which breaks the build ROYALLY on FreeBSD 64-bit machine. I'm trying to come up with the right heuristic here. Any help would be much appreciated! Thanks! -aps From des at des.no Fri Oct 24 01:05:39 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Fri Oct 24 01:05:46 2008 Subject: Why does adding /usr/lib32 to LD_LIBRARY_PATH break 64-bit binaries? In-Reply-To: <3c0b01820810231731s1b4d4659j7d1df8bf4abb229c@mail.gmail.com> (Alexander Sack's message of "Thu, 23 Oct 2008 20:31:05 -0400") References: <3c0b01820810231731s1b4d4659j7d1df8bf4abb229c@mail.gmail.com> Message-ID: <86hc72x0nx.fsf@ds4.des.no> "Alexander Sack" writes: > I have some weird behavior I'm trying to figure out and was wondering > if someone can point me in the right direction. I'm running a FreeBSD > 6.1-RELEASE-amd64 machine. If I add /usr/lib32 to my LD_LIBRARY_PATH > it breaks all of my binaries on my 64-bit machine. > [...] > LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/bin:/usr/lib:/usr/lib32:/usr/lib64 I'm surprised you have /usr/bin in there... > I would ASSUME that rtld would look at my LD_LIBRARY_PATH and use > /usr/lib to find libraries, not /usr/lib32. Why does it insist on > picking /usr/lib32 when "/bin/ls" is CLEARLY a 64-bit binary? This > doesn't make complete sense to me just yet. If you look at the rtld(1) man page, there are a number of environment variables you can set to debug the loader. I'm not sure how helpful they are, though. > Someone I'm sure is going "don't do that" and I agree. Well, yeah, but it should (at the very least) fail in a more graceful manner. DES -- Dag-Erling Sm?rgrav - des@des.no From des at des.no Fri Oct 24 01:23:10 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Fri Oct 24 01:23:16 2008 Subject: Why does adding /usr/lib32 to LD_LIBRARY_PATH break 64-bit binaries? In-Reply-To: <86hc72x0nx.fsf@ds4.des.no> ("Dag-Erling =?utf-8?Q?Sm=C3=B8rg?= =?utf-8?Q?rav=22's?= message of "Fri, 24 Oct 2008 03:05:38 +0200") References: <3c0b01820810231731s1b4d4659j7d1df8bf4abb229c@mail.gmail.com> <86hc72x0nx.fsf@ds4.des.no> Message-ID: <86d4hqwzur.fsf@ds4.des.no> Dag-Erling Sm?rgrav writes: > If you look at the rtld(1) man page, there are a number of environment > variables you can set to debug the loader. I'm not sure how helpful > they are, though. You can rebuild rtld(1) with debugging enabled: % cd /usr/src/libexec/rtld-elf % make clean % make DEBUG_FLAGS=-DDEBUG % make install % echo $LD_LIBRARY_PATH /home/des/lib:/opt/varnish/lib:/usr/local/lib % LD_DEBUG=1 /usr/bin/true /libexec/ld-elf.so.1 is initialized, base address = 0x800500000 RTLD dynamic = 0x8006305b0 RTLD pltgot = 0x0 processing main program's program header Filling in DT_DEBUG entry lm_init("(null)") loading LD_PRELOAD libraries loading needed objects Searching for "libc.so.7" Trying "/home/des/lib/libc.so.7" Trying "/opt/varnish/lib/libc.so.7" Trying "/usr/local/lib/libc.so.7" Trying "/lib/libc.so.7" loading "/lib/libc.so.7" Ignoring d_tag 1879048185 = 0x6ffffff9 0x80063b000 .. 0x80085efff: /lib/libc.so.7 checking for required versions initializing initial thread local storage relocating "/usr/bin/true" relocating "/lib/libc.so.7" doing copy relocations initializing key program variables "__progname": *0x5005e8 <-- 0x7fffffffebc1 "environ": *0x500878 <-- 0x7fffffffe9a8 initializing thread locks calling init function for /lib/libc.so.7 at 0x800664da8 "__sysctl" in "libc.so.7" ==> 0x80071ae00 in "libc.so.7" reloc_jmpslot: *0x800845c78 = 0x80071ae00 transferring control to program entry point = 0x400420 "atexit" in "true" ==> 0x8006fac3e in "libc.so.7" reloc_jmpslot: *0x500868 = 0x8006fac3e "exit" in "true" ==> 0x8006af118 in "libc.so.7" reloc_jmpslot: *0x500860 = 0x8006af118 "__cxa_finalize" in "libc.so.7" ==> 0x8006fa940 in "libc.so.7" reloc_jmpslot: *0x800846140 = 0x8006fa940 rtld_exit() calling fini function for /lib/libc.so.7 at 0x80071ae60 "_exit" in "libc.so.7" ==> 0x8006cfff0 in "libc.so.7" reloc_jmpslot: *0x8008471d8 = 0x8006cfff0 DES -- Dag-Erling Sm?rgrav - des@des.no From pisymbol at gmail.com Fri Oct 24 01:48:48 2008 From: pisymbol at gmail.com (Alexander Sack) Date: Fri Oct 24 01:49:01 2008 Subject: Why does adding /usr/lib32 to LD_LIBRARY_PATH break 64-bit binaries? In-Reply-To: <86d4hqwzur.fsf@ds4.des.no> References: <3c0b01820810231731s1b4d4659j7d1df8bf4abb229c@mail.gmail.com> <86hc72x0nx.fsf@ds4.des.no> <86d4hqwzur.fsf@ds4.des.no> Message-ID: <3c0b01820810231848r3e3e297cl3dc9bf1d0edcd588@mail.gmail.com> On Thu, Oct 23, 2008 at 9:23 PM, Dag-Erling Sm?rgrav wrote: > Dag-Erling Sm?rgrav writes: >> If you look at the rtld(1) man page, there are a number of environment >> variables you can set to debug the loader. I'm not sure how helpful >> they are, though. > > You can rebuild rtld(1) with debugging enabled: > > % cd /usr/src/libexec/rtld-elf > % make clean > % make DEBUG_FLAGS=-DDEBUG > % make install > % echo $LD_LIBRARY_PATH > /home/des/lib:/opt/varnish/lib:/usr/local/lib > % LD_DEBUG=1 /usr/bin/true > /libexec/ld-elf.so.1 is initialized, base address = 0x800500000 > RTLD dynamic = 0x8006305b0 > RTLD pltgot = 0x0 > processing main program's program header > Filling in DT_DEBUG entry > lm_init("(null)") > loading LD_PRELOAD libraries > loading needed objects > Searching for "libc.so.7" > Trying "/home/des/lib/libc.so.7" > Trying "/opt/varnish/lib/libc.so.7" > Trying "/usr/local/lib/libc.so.7" > Trying "/lib/libc.so.7" > loading "/lib/libc.so.7" > Ignoring d_tag 1879048185 = 0x6ffffff9 > 0x80063b000 .. 0x80085efff: /lib/libc.so.7 > checking for required versions > initializing initial thread local storage > relocating "/usr/bin/true" > relocating "/lib/libc.so.7" > doing copy relocations > initializing key program variables > "__progname": *0x5005e8 <-- 0x7fffffffebc1 > "environ": *0x500878 <-- 0x7fffffffe9a8 > initializing thread locks > calling init function for /lib/libc.so.7 at 0x800664da8 > "__sysctl" in "libc.so.7" ==> 0x80071ae00 in "libc.so.7" > reloc_jmpslot: *0x800845c78 = 0x80071ae00 > transferring control to program entry point = 0x400420 > "atexit" in "true" ==> 0x8006fac3e in "libc.so.7" > reloc_jmpslot: *0x500868 = 0x8006fac3e > "exit" in "true" ==> 0x8006af118 in "libc.so.7" > reloc_jmpslot: *0x500860 = 0x8006af118 > "__cxa_finalize" in "libc.so.7" ==> 0x8006fa940 in "libc.so.7" > reloc_jmpslot: *0x800846140 = 0x8006fa940 > rtld_exit() > calling fini function for /lib/libc.so.7 at 0x80071ae60 > "_exit" in "libc.so.7" ==> 0x8006cfff0 in "libc.so.7" > reloc_jmpslot: *0x8008471d8 = 0x8006cfff0 > > DES > -- > Dag-Erling Sm?rgrav - des@des.no Thanks, comments most appreciated. Damn, I was looking for someone to go "a ha, you can't do this because...." Alright, let me see why rtld on 6.1-amd64 is picking up /usr/lib32 stuff for a native 64-bit binary via debugging techniques. This seems very very wrong to me. I mean if /usr/lib is in my LD_LIBRARY_PATH and it comes before /usr/lib the /usr/lib32 *should* be innocuous, right? Feel free to use that last statement on my epitaph! :D -aps From pisymbol at gmail.com Fri Oct 24 02:10:43 2008 From: pisymbol at gmail.com (Alexander Sack) Date: Fri Oct 24 02:10:54 2008 Subject: Why does adding /usr/lib32 to LD_LIBRARY_PATH break 64-bit binaries? In-Reply-To: <3c0b01820810231848r3e3e297cl3dc9bf1d0edcd588@mail.gmail.com> References: <3c0b01820810231731s1b4d4659j7d1df8bf4abb229c@mail.gmail.com> <86hc72x0nx.fsf@ds4.des.no> <86d4hqwzur.fsf@ds4.des.no> <3c0b01820810231848r3e3e297cl3dc9bf1d0edcd588@mail.gmail.com> Message-ID: <3c0b01820810231910u59f97cecg8b01f391a9b353f3@mail.gmail.com> Alright, well I found some weirdness: [root@hagen ~]# export LD_LIBRARY_PATH=/usr/bin:/usr/lib:/usr/lib32:/usr/lib64 [root@hagen ~]# LD_DEBUG=1 ls /libexec/ld-elf.so.1 is initialized, base address = 0x800506000 RTLD dynamic = 0x80062ad78 RTLD pltgot = 0x0 processing main program's program header Filling in DT_DEBUG entry lm_init("(null)") loading LD_PRELOAD libraries loading needed objects Searching for "libutil.so.5" Trying "/usr/bin/libutil.so.5" Trying "/usr/lib/libutil.so.5" Trying "/usr/lib32/libutil.so.5" loading "/usr/lib32/libutil.so.5" /libexec/ld-elf.so.1: /usr/lib32/libutil.so.5: unsupported file layout That's because libutil.so.5 does not exist in /usr/lib only in /lib. The /usr/lib directory has: [root@hagen ~]# ls -l /usr/lib/libutil* -r--r--r-- 1 root wheel 100518 Aug 21 2007 /usr/lib/libutil.a lrwxrwxrwx 1 root wheel 17 Sep 11 11:44 /usr/lib/libutil.so -> /lib/libutil.so.5 -r--r--r-- 1 root wheel 103846 Aug 21 2007 /usr/lib/libutil_p.a So rtld is looking for major number 5 of libutil, without the standard /lib in my LD_LIBRARY_PATH it searches /usr/lib, doesn't find it but: [root@hagen ~]# ls -l /usr/lib32/libutil* -r--r--r-- 1 root wheel 65274 Aug 21 2007 /usr/lib32/libutil.a lrwxrwxrwx 1 root wheel 12 Sep 11 11:45 /usr/lib32/libutil.so -> libutil.so.5 -r--r--r-- 1 root wheel 46872 Aug 21 2007 /usr/lib32/libutil.so.5 -r--r--r-- 1 root wheel 66918 Aug 21 2007 /usr/lib32/libutil_p.a And whalah, I'm broke since there is a libutil.so.5 in there. So my question to anyone out there, WHY does /usr/lib32 contain major numbers but /usr/lib does not? This seems like a bug to me (FreeBSD 7.0-RELEASE is the same) or at least a dubious design decision. Thanks! -aps From pisymbol at gmail.com Fri Oct 24 02:31:42 2008 From: pisymbol at gmail.com (Alexander Sack) Date: Fri Oct 24 02:31:49 2008 Subject: Why does adding /usr/lib32 to LD_LIBRARY_PATH break 64-bit binaries? In-Reply-To: <20081023220949.7f304bbb@kan.dnsalias.net> References: <3c0b01820810231731s1b4d4659j7d1df8bf4abb229c@mail.gmail.com> <86hc72x0nx.fsf@ds4.des.no> <86d4hqwzur.fsf@ds4.des.no> <3c0b01820810231848r3e3e297cl3dc9bf1d0edcd588@mail.gmail.com> <20081023220949.7f304bbb@kan.dnsalias.net> Message-ID: <3c0b01820810231931p2acbf426m7f1b94b73b466e5d@mail.gmail.com> On Thu, Oct 23, 2008 at 10:09 PM, Alexander Kabaev wrote: > On Thu, 23 Oct 2008 21:48:47 -0400 > "Alexander Sack" wrote: > >> Thanks, comments most appreciated. Damn, I was looking for someone to >> go "a ha, you can't do this because...." Alright, let me see why rtld >> on 6.1-amd64 is picking up /usr/lib32 stuff for a native 64-bit binary >> via debugging techniques. This seems very very wrong to me. I mean if >> /usr/lib is in my LD_LIBRARY_PATH and it comes before /usr/lib the >> /usr/lib32 *should* be innocuous, right? >> >> Feel free to use that last statement on my epitaph! :D >> > LD_LIBRARY_PATH is for native 64bit rtld. If you want a specific path > added for use by 32-bit ld-elf.so.1 only, use LD_32_LIBRARY_PATH. > > Said that, your problem is likely caused by the fact that there is > no /lib32, only /usr/lib32. So if 64-bit library lives in /lib, > your LD_LIBRARY_PATH will cause loader to find its 32-bit equivalent > in /usr/lib32 first. > > Try LD_LIBRARY_PATH=/lib:/usr/lib:/usr/lib32:/usr/lib64 for better > results. Yes I figured that out on my own but my question still exists, why isn't /usr/lib similar in format to /usr/lib32 though with respect to major numbers? Actually now that I re-read your paragraph I suppose this isn't such a bad idea but for some reason I believe that if you have /usr/lib before /usr/lib32 it should *just* work. -aps From kabaev at gmail.com Fri Oct 24 02:41:42 2008 From: kabaev at gmail.com (Alexander Kabaev) Date: Fri Oct 24 02:41:49 2008 Subject: Why does adding /usr/lib32 to LD_LIBRARY_PATH break 64-bit binaries? In-Reply-To: <3c0b01820810231848r3e3e297cl3dc9bf1d0edcd588@mail.gmail.com> References: <3c0b01820810231731s1b4d4659j7d1df8bf4abb229c@mail.gmail.com> <86hc72x0nx.fsf@ds4.des.no> <86d4hqwzur.fsf@ds4.des.no> <3c0b01820810231848r3e3e297cl3dc9bf1d0edcd588@mail.gmail.com> Message-ID: <20081023220949.7f304bbb@kan.dnsalias.net> On Thu, 23 Oct 2008 21:48:47 -0400 "Alexander Sack" wrote: > Thanks, comments most appreciated. Damn, I was looking for someone to > go "a ha, you can't do this because...." Alright, let me see why rtld > on 6.1-amd64 is picking up /usr/lib32 stuff for a native 64-bit binary > via debugging techniques. This seems very very wrong to me. I mean if > /usr/lib is in my LD_LIBRARY_PATH and it comes before /usr/lib the > /usr/lib32 *should* be innocuous, right? > > Feel free to use that last statement on my epitaph! :D > LD_LIBRARY_PATH is for native 64bit rtld. If you want a specific path added for use by 32-bit ld-elf.so.1 only, use LD_32_LIBRARY_PATH. Said that, your problem is likely caused by the fact that there is no /lib32, only /usr/lib32. So if 64-bit library lives in /lib, your LD_LIBRARY_PATH will cause loader to find its 32-bit equivalent in /usr/lib32 first. Try LD_LIBRARY_PATH=/lib:/usr/lib:/usr/lib32:/usr/lib64 for better results. -- Alexander Kabaev -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081024/dafd6909/signature.pgp From neldredge at math.ucsd.edu Fri Oct 24 02:49:03 2008 From: neldredge at math.ucsd.edu (Nate Eldredge) Date: Fri Oct 24 02:49:10 2008 Subject: Why does adding /usr/lib32 to LD_LIBRARY_PATH break 64-bit binaries? In-Reply-To: <3c0b01820810231910u59f97cecg8b01f391a9b353f3@mail.gmail.com> References: <3c0b01820810231731s1b4d4659j7d1df8bf4abb229c@mail.gmail.com> <86hc72x0nx.fsf@ds4.des.no> <86d4hqwzur.fsf@ds4.des.no> <3c0b01820810231848r3e3e297cl3dc9bf1d0edcd588@mail.gmail.com> <3c0b01820810231910u59f97cecg8b01f391a9b353f3@mail.gmail.com> Message-ID: On Thu, 23 Oct 2008, Alexander Sack wrote: > Alright, well I found some weirdness: > > [root@hagen ~]# export LD_LIBRARY_PATH=/usr/bin:/usr/lib:/usr/lib32:/usr/lib64 > [root@hagen ~]# LD_DEBUG=1 ls > /libexec/ld-elf.so.1 is initialized, base address = 0x800506000 > RTLD dynamic = 0x80062ad78 > RTLD pltgot = 0x0 > processing main program's program header > Filling in DT_DEBUG entry > lm_init("(null)") > loading LD_PRELOAD libraries > loading needed objects > Searching for "libutil.so.5" > Trying "/usr/bin/libutil.so.5" > Trying "/usr/lib/libutil.so.5" > Trying "/usr/lib32/libutil.so.5" > loading "/usr/lib32/libutil.so.5" > /libexec/ld-elf.so.1: /usr/lib32/libutil.so.5: unsupported file layout > > That's because libutil.so.5 does not exist in /usr/lib only in /lib. > The /usr/lib directory has: > > [root@hagen ~]# ls -l /usr/lib/libutil* > -r--r--r-- 1 root wheel 100518 Aug 21 2007 /usr/lib/libutil.a > lrwxrwxrwx 1 root wheel 17 Sep 11 11:44 /usr/lib/libutil.so -> > /lib/libutil.so.5 > -r--r--r-- 1 root wheel 103846 Aug 21 2007 /usr/lib/libutil_p.a > > So rtld is looking for major number 5 of libutil, without the standard > /lib in my LD_LIBRARY_PATH it searches /usr/lib, doesn't find it but: > > [root@hagen ~]# ls -l /usr/lib32/libutil* > -r--r--r-- 1 root wheel 65274 Aug 21 2007 /usr/lib32/libutil.a > lrwxrwxrwx 1 root wheel 12 Sep 11 11:45 /usr/lib32/libutil.so -> > libutil.so.5 > -r--r--r-- 1 root wheel 46872 Aug 21 2007 /usr/lib32/libutil.so.5 > -r--r--r-- 1 root wheel 66918 Aug 21 2007 /usr/lib32/libutil_p.a > > And whalah, I'm broke since there is a libutil.so.5 in there. > > So my question to anyone out there, WHY does /usr/lib32 contain major > numbers but /usr/lib does not? This seems like a bug to me (FreeBSD > 7.0-RELEASE is the same) or at least a dubious design decision. I think the distinction is this. rtld is looking for libutil.so.5 (with version number). This file has to be in /lib, in the root filesystem, so that programs can run before /usr is mounted. libutil.so on the other hand is not searched for by rtld, but by ld (driven by cc), when the program is built. /usr/lib is the traditional place for it to search; I'm not sure if it searches /lib at all. In the case of static libraries, /usr/lib is certainly the right place for libutil.a to go, so having libutil.so there makes sense in my mind. I think your best bet is to dig into whatever is setting LD_LIBRARY_PATH and get it set correctly. Remove /usr/lib32 or at least ensure that /lib is searched first. Trying to change rtld's behavior is not the right approach, IMHO. -- Nate Eldredge neldredge@math.ucsd.edu From dnelson at allantgroup.com Fri Oct 24 03:07:41 2008 From: dnelson at allantgroup.com (Dan Nelson) Date: Fri Oct 24 03:07:48 2008 Subject: Why does adding /usr/lib32 to LD_LIBRARY_PATH break 64-bit binaries? In-Reply-To: <3c0b01820810231931p2acbf426m7f1b94b73b466e5d@mail.gmail.com> References: <3c0b01820810231731s1b4d4659j7d1df8bf4abb229c@mail.gmail.com> <86hc72x0nx.fsf@ds4.des.no> <86d4hqwzur.fsf@ds4.des.no> <3c0b01820810231848r3e3e297cl3dc9bf1d0edcd588@mail.gmail.com> <20081023220949.7f304bbb@kan.dnsalias.net> <3c0b01820810231931p2acbf426m7f1b94b73b466e5d@mail.gmail.com> Message-ID: <20081024030354.GA41283@dan.emsphone.com> In the last episode (Oct 23), Alexander Sack said: > On Thu, Oct 23, 2008 at 10:09 PM, Alexander Kabaev wrote: > > LD_LIBRARY_PATH is for native 64bit rtld. If you want a specific > > path added for use by 32-bit ld-elf.so.1 only, use > > LD_32_LIBRARY_PATH. > > > > Said that, your problem is likely caused by the fact that there is > > no /lib32, only /usr/lib32. So if 64-bit library lives in /lib, > > your LD_LIBRARY_PATH will cause loader to find its 32-bit > > equivalent in /usr/lib32 first. > > > > Try LD_LIBRARY_PATH=/lib:/usr/lib:/usr/lib32:/usr/lib64 for better > > results. > > Yes I figured that out on my own but my question still exists, why > isn't /usr/lib similar in format to /usr/lib32 though with respect to > major numbers? Ever since the switch from static to dynamic-linked /bin and /sbin, some shared libraries are needed during the boot process. Those libraries live in /lib, and since there are no 32-bit binaries required to boot a 64-bit system, there is no need for a /lib32. -- Dan Nelson dnelson@allantgroup.com From danny at cs.huji.ac.il Fri Oct 24 09:05:35 2008 From: danny at cs.huji.ac.il (Danny Braniss) Date: Fri Oct 24 09:05:41 2008 Subject: zfs & waiting on zio->io_cv Message-ID: there is a big delay (probably more than 1 sec.) when doing simple tasks on this zfs, like ls(1), or 'zfs list', long enough to hit ^T and get the same [zio->io_cv)], any hints? store-01# zfs list (hitting ^T)load: 0.00 cmd: zfs 88376 [zio->io_cv)] 0.00u 0.00s 0% 1672k (hitting ^T)load: 0.00 cmd: zfs 88376 [zio->io_cv)] 0.00u 0.00s 0% 1684k NAME USED AVAIL REFER MOUNTPOINT h 472G 11.2T 23K /h h/home 466G 11.2T 466G /h/home h/home@23-10-08 54K - 466G - h/root 18K 11.2T 18K /h/root h/src 18K 11.2T 18K /h/src h/system 5.64G 11.2T 5.64G /h/system cheers, danny From wojtek at wojtek.tensor.gdynia.pl Fri Oct 24 08:43:14 2008 From: wojtek at wojtek.tensor.gdynia.pl (Wojciech Puchar) Date: Fri Oct 24 11:19:49 2008 Subject: Why does adding /usr/lib32 to LD_LIBRARY_PATH break 64-bit binaries? In-Reply-To: <3c0b01820810231731s1b4d4659j7d1df8bf4abb229c@mail.gmail.com> References: <3c0b01820810231731s1b4d4659j7d1df8bf4abb229c@mail.gmail.com> Message-ID: <20081024104232.X21603@wojtek.tensor.gdynia.pl> > 6.1-RELEASE-amd64 machine. If I add /usr/lib32 to my LD_LIBRARY_PATH > it breaks all of my binaries on my 64-bit machine. what do you expect else? this will make system trying to bind 32-bit libs to 64-bit program. it can't work From peterjeremy at optushome.com.au Fri Oct 24 12:51:06 2008 From: peterjeremy at optushome.com.au (Peter Jeremy) Date: Fri Oct 24 12:51:14 2008 Subject: Why does adding /usr/lib32 to LD_LIBRARY_PATH break 64-bit binaries? In-Reply-To: <20081024104232.X21603@wojtek.tensor.gdynia.pl> References: <3c0b01820810231731s1b4d4659j7d1df8bf4abb229c@mail.gmail.com> <20081024104232.X21603@wojtek.tensor.gdynia.pl> Message-ID: <20081024125059.GE1137@server.vk2pj.dyndns.org> On 2008-Oct-24 10:43:04 +0200, Wojciech Puchar wrote: >> 6.1-RELEASE-amd64 machine. If I add /usr/lib32 to my LD_LIBRARY_PATH >> it breaks all of my binaries on my 64-bit machine. > >what do you expect else? Well, the rtld should be smart enough to recognize 32-bit .so's and skip them when binding a 64-bit executable. Whilst having /usr/lib32 in LD_LIBRARY_PATH doesn't make sense from a solely FreeBSD perspective, I have done similar things when writing cross-platform scripts (to avoid having to use platform-dependent code). >this will make system trying to bind 32-bit libs to 64-bit program. it >can't work rtld shouldn't attempt to bind 32-bit libs to 64-bit programs. -- Peter Jeremy Please excuse any delays as the result of my ISP's inability to implement an MTA that is either RFC2821-compliant or matches their claimed behaviour. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081024/d72ea2cd/attachment.pgp From george+freebsd at m5p.com Fri Oct 24 14:05:27 2008 From: george+freebsd at m5p.com (george+freebsd@m5p.com) Date: Fri Oct 24 14:05:34 2008 Subject: Severe DNS Problems, 6.2-RELEASE, BIND 9.5.2 Message-ID: <200810241404.m9OE4oYl026020@m5p.com> I'm having severe DNS problems. I'm running 6.2-RELEASE, and I upgraded to the bind9 port (after cvsup) on July 14. Starting yesterday morning, DNS became very, very slow. If I repeated a "dig" command three or four times, I could get an answer after 20-30 seconds. This morning I cvsupped again and installed the bind95 port. Still very, very slow. I will probably shift my server to a FreeBSD 7.0 system this weekend, but I would like very much to understand what's going on. -- George Mitchell From dnelson at allantgroup.com Fri Oct 24 15:09:19 2008 From: dnelson at allantgroup.com (Dan Nelson) Date: Fri Oct 24 15:09:26 2008 Subject: zfs & waiting on zio->io_cv In-Reply-To: References: Message-ID: <20081024150916.GB41283@dan.emsphone.com> In the last episode (Oct 24), Danny Braniss said: > there is a big delay (probably more than 1 sec.) when doing simple tasks > on this zfs, like ls(1), or 'zfs list', long enough to hit ^T > and get the same [zio->io_cv)], any hints? > > store-01# zfs list > (hitting ^T)load: 0.00 cmd: zfs 88376 [zio->io_cv)] 0.00u 0.00s 0% 1672k > (hitting ^T)load: 0.00 cmd: zfs 88376 [zio->io_cv)] 0.00u 0.00s 0% 1684k > NAME USED AVAIL REFER MOUNTPOINT > h 472G 11.2T 23K /h > h/home 466G 11.2T 466G /h/home > h/home@23-10-08 54K - 466G - > h/root 18K 11.2T 18K /h/root > h/src 18K 11.2T 18K /h/src > h/system 5.64G 11.2T 5.64G /h/system That's sort of the equivalent to waiting in "biord" on a UFS filesystem, I think. ZFS is just waiting for the disk to return a block. If you happen to do something during the window where ZFS is commiting its transaction group, it has to wait until the sync finishes. If some other process is doing a lot of writes, or you only have one disk in your zpool, or your pool is close to full, it may take a couple seconds to sync. There's a couple of things you can try to improve interactive performance. Raising zfs's arc_max is the easiest to do, and will let ZFS cache more stuff, increasing the likelyhood that an "ls" will be able to read from cache instead of having to go to disk. Setting it at 1/4 your physical RAM is probably as high as you can go without causing panics. Raising txg_time ( in /sys/cddl/.../zfs/txg.c ) from 5 to say 30 will tell zfs to sync less often, which can be a win if you don't actually do that much writing. With a single spindle, it may take a substantial fraction of a second just to sync a tiny txg due to the number of copies of metadata ZFS writes for redundancy. If you do a lot of writing, lowering zfs_vdev_max_pending ( in /sys/cddl/.../zfs/vdev_queue.c ) from 35 down to 16 or less will reduce the number of simultaneous I/Os ZFS will try to send to each disk, which will let your reads compete a little better with other I/O. On ATA or SATA disks, you might want to set it to 2. -- Dan Nelson dnelson@allantgroup.com From thierry at herbelot.com Fri Oct 24 16:36:59 2008 From: thierry at herbelot.com (Thierry Herbelot) Date: Fri Oct 24 16:42:49 2008 Subject: question about sb->st_blksize in src/sys/kern/vfs_vnops.c Message-ID: <200810241818.37262.thierry@herbelot.com> Hello, the [SUBJ] file contains the following extract (around line 705) : * Default to PAGE_SIZE after much discussion. * XXX: min(PAGE_SIZE, vp->v_bufobj.bo_bsize) may be more correct. */ sb->st_blksize = PAGE_SIZE; which arrived around four years ago, with revision 1.211 (see http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/vfs_vnops.c.diff?r1=1.210;r2=1.211;f=h) the net effect of this change is to decrease the block buffer size used in libc/stdio from 16 kbytes (derived from the underlying ufs partition) to PAGE_SIZE ==4 kbytes (fixed value), and consequently the I/O bandwidth is lowered (this is on a slow Flash). I have patched the kernel with a larger, fixed value (simply 4*PAGE_SIZE, to revert to the block size previoulsly used), and the kernel and world seem to be running fine. Seeing the XXX coment above, I'm a bit worried about keeping this new st_blksize value. are there any drawbacks with running with this bigger buffer size value ? TfH From m.seaman at infracaninophile.co.uk Fri Oct 24 17:05:37 2008 From: m.seaman at infracaninophile.co.uk (Matthew Seaman) Date: Fri Oct 24 17:05:44 2008 Subject: Severe DNS Problems, 6.2-RELEASE, BIND 9.5.2 In-Reply-To: <200810241404.m9OE4oYl026020@m5p.com> References: <200810241404.m9OE4oYl026020@m5p.com> Message-ID: <4902004F.4010002@infracaninophile.co.uk> george+freebsd@m5p.com wrote: > I'm having severe DNS problems. I'm running 6.2-RELEASE, and I upgraded > to the bind9 port (after cvsup) on July 14. Starting yesterday morning, > DNS became very, very slow. If I repeated a "dig" command three or four > times, I could get an answer after 20-30 seconds. This morning I cvsupped > again and installed the bind95 port. Still very, very slow. I will > probably shift my server to a FreeBSD 7.0 system this weekend, but I > would like very much to understand what's going on. Did you configure DLV (DNSSEC Look-aside Validation)? If so, you were probably bitten by the ISC key timing out. Key roll-over was scheduled for the month leading up to Tuesday 21st. Get the new key from: https://secure.isc.org/ops/dlv/index.php#dlv_key Cheers, Matthew -- Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate Kent, CT11 9PW -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 258 bytes Desc: OpenPGP digital signature Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081024/103a41b1/signature.pgp From martin.laabs at mailbox.tu-dresden.de Fri Oct 24 17:11:28 2008 From: martin.laabs at mailbox.tu-dresden.de (Martin Laabs) Date: Fri Oct 24 17:11:34 2008 Subject: libusb for linux-emulation Message-ID: Hi, I'd like to run a program in linux-emulation (closed source ...) which uses the libusb. Is the structure (ioctls on devices and so on) sufficient compatible to use a linux libusb? Or is there any way to build a simple wrapper library to use the nativ libusb? (IMHO this is not possible.) Do you have any other idea? Thank you, Martin L. From hselasky at c2i.net Fri Oct 24 18:49:29 2008 From: hselasky at c2i.net (Hans Petter Selasky) Date: Fri Oct 24 18:49:37 2008 Subject: libusb for linux-emulation In-Reply-To: References: Message-ID: <200810241951.32135.hselasky@c2i.net> On Friday 24 October 2008, Martin Laabs wrote: > Hi, > > I'd like to run a program in linux-emulation (closed source ...) > which uses the libusb. Is the structure (ioctls on devices and so on) > sufficient compatible to use a linux libusb? > Or is there any way to build a simple wrapper library to use the > nativ libusb? (IMHO this is not possible.) > Do you have any other idea? > No, you cannot use the linux libusb on FreeBSD. You need to use the FreeBSD compiled libusb. The USB kernel interfaces are quite different. --HPS From martin.laabs at mailbox.tu-dresden.de Fri Oct 24 18:53:31 2008 From: martin.laabs at mailbox.tu-dresden.de (Martin Laabs) Date: Fri Oct 24 18:53:37 2008 Subject: libusb for linux-emulation In-Reply-To: <200810241951.32135.hselasky@c2i.net> References: <200810241951.32135.hselasky@c2i.net> Message-ID: Hi, On Fri, 24 Oct 2008 19:51:31 +0200, Hans Petter Selasky wrote: > > No, you cannot use the linux libusb on FreeBSD. You need to use the FreeBSD > compiled libusb. The USB kernel interfaces are quite different. OK - I see. I'm just trying to build a "hermaphrodite" library. Compile with linux but using the BSD ioctls. Is there a crosscompiler to compile linux binarys from freebsd? This would make the job much easier. Thank you, Martin L. From hselasky at c2i.net Fri Oct 24 19:25:35 2008 From: hselasky at c2i.net (Hans Petter Selasky) Date: Fri Oct 24 19:25:41 2008 Subject: libusb for linux-emulation In-Reply-To: References: <200810241951.32135.hselasky@c2i.net> Message-ID: <200810242127.37997.hselasky@c2i.net> Hi Martin, Why can't you use and install: /usr/ports/devel/libusb ? --HPS On Friday 24 October 2008, Martin Laabs wrote: > Hi, > > On Fri, 24 Oct 2008 19:51:31 +0200, Hans Petter Selasky wrote: > > No, you cannot use the linux libusb on FreeBSD. You need to use the > > FreeBSD compiled libusb. The USB kernel interfaces are quite different. > > OK - I see. I'm just trying to build a "hermaphrodite" library. Compile > with linux but using the BSD ioctls. > Is there a crosscompiler to compile linux binarys from freebsd? This would > make the job much easier. > From stevefranks at ieee.org Fri Oct 24 22:34:32 2008 From: stevefranks at ieee.org (Steve Franks) Date: Fri Oct 24 22:34:39 2008 Subject: neophyte: tcsetattr() gives 22 error in i386, not in amd64? Message-ID: <539c60b90810241534l6bedc5e3s1c2e3162c2a7ff38@mail.gmail.com> Hi, I'm getting a 22 errno from tcsetattr() on 7-STABLE i386 in code which was working under 7-STABLE amd64. Serial device is a ucom (silabs cp2103). Permissions on /dev/cuaU0 look fine. Cutecom/Minicom appears to open the port without error... Ideas? Thanks, Steve if(cfsetspeed(&IspEnvironment->newtio,(speed_t) strtol(IspEnvironment->baud_rate,NULL,10))) { DebugPrintf(1, "baudrate %s not supported\n", IspEnvironment->baud_rate); exit(3); }; switch (atol(IspEnvironment->baud_rate)) { case 1152000: NEWTERMIOS_SETBAUDARTE(B1152000); break; default: { DebugPrintf(1, "unknown baudrate %s\n", IspEnvironment->baud_rate); exit(3); } } IspEnvironment->newtio.c_iflag = IGNPAR | IGNBRK | IXON | IXOFF; IspEnvironment->newtio.c_oflag = 0; /* set input mode (non-canonical, no echo,...) */ IspEnvironment->newtio.c_lflag = 0; cfmakeraw(&IspEnvironment->newtio); IspEnvironment->newtio.c_cc[VTIME] = 1; /* inter-character timer used */ IspEnvironment->newtio.c_cc[VMIN] = 0; /* blocking read until 0 chars received */ tcflush(IspEnvironment->fdCom, TCIFLUSH); int err = tcsetattr(IspEnvironment->fdCom, TCSANOW, &IspEnvironment->newtio); From neldredge at math.ucsd.edu Fri Oct 24 23:30:36 2008 From: neldredge at math.ucsd.edu (Nate Eldredge) Date: Fri Oct 24 23:30:43 2008 Subject: neophyte: tcsetattr() gives 22 error in i386, not in amd64? In-Reply-To: <539c60b90810241534l6bedc5e3s1c2e3162c2a7ff38@mail.gmail.com> References: <539c60b90810241534l6bedc5e3s1c2e3162c2a7ff38@mail.gmail.com> Message-ID: On Fri, 24 Oct 2008, Steve Franks wrote: > Hi, > > I'm getting a 22 errno from tcsetattr() on 7-STABLE i386 in code which > was working under 7-STABLE amd64. Serial device is a ucom (silabs > cp2103). Permissions on /dev/cuaU0 look fine. Cutecom/Minicom > appears to open the port without error... I don't see anything obviously wrong, but I'd bet a bug related to 32/64-bit types. Can you post a complete piece of code that can be compiled and run and demonstrates the problem? Also, try compiling with -Wall -W and investigate any warnings that are produced. By the way, errno 22 is EINVAL, "Invalid argument". perror() is your friend. [snip code] -- Nate Eldredge neldredge@math.ucsd.edu From george+freebsd at m5p.com Fri Oct 24 23:33:44 2008 From: george+freebsd at m5p.com (george+freebsd@m5p.com) Date: Fri Oct 24 23:33:51 2008 Subject: Severe DNS Problems, 6.2-RELEASE, BIND 9.5.2 Message-ID: <200810242332.m9ONWT8S032380@m5p.com> > From: Matthew Seaman > george+freebsd@m5p.com wrote: > > I'm having severe DNS problems. I'm running 6.2-RELEASE, and I upgrade= > d > > to the bind9 port (after cvsup) on July 14. Starting yesterday morning= > , > > DNS became very, very slow. If I repeated a "dig" command three or fou= > r > > times, I could get an answer after 20-30 seconds. This morning I cvsup= > ped > > again and installed the bind95 port. Still very, very slow. I will > > probably shift my server to a FreeBSD 7.0 system this weekend, but I > > would like very much to understand what's going on. > > Did you configure DLV (DNSSEC Look-aside Validation)? If so, you were=20 > probably bitten by the ISC key timing out. Key roll-over was scheduled=20 > for the month leading up to Tuesday 21st. > > Get the new key from: https://secure.isc.org/ops/dlv/index.php#dlv_key > > Cheers, > > Matthew No, I'm not using DLV, but thanks for the hint anyway. > From: Mike Meyer > X-Spam-Score: 0 () > X-Scanned-By: MIMEDefang 2.57 on 10.100.0.247 > X-Greylist: Delayed for 00:52:50 by milter-greylist-2.0.2 (mailhost.m5p.com [10.100.0.247]); Fri, 24 Oct 2008 13:41:31 -0400 (EDT) > Status: R > > On Fri, 24 Oct 2008 10:04:50 -0400 (EDT) > george+freebsd@m5p.com wrote: > > > I'm having severe DNS problems. I'm running 6.2-RELEASE, and I upgraded > > to the bind9 port (after cvsup) on July 14. Starting yesterday morning, > > DNS became very, very slow. If I repeated a "dig" command three or four > > times, I could get an answer after 20-30 seconds. This morning I cvsupped > > again and installed the bind95 port. Still very, very slow. I will > > probably shift my server to a FreeBSD 7.0 system this weekend, but I > > would like very much to understand what's going on. > > Could this be a downstream server timing out? > > References: <200810242127.37997.hselasky@c2i.net> Message-ID: <200810251003.12882.doconnor@gsoft.com.au> On Saturday 25 October 2008 05:57:36 Hans Petter Selasky wrote: > Why can't you use and install: > > /usr/ports/devel/libusb > > ? Because that gives you a FreeBSD libusb and he needs to have a Linux program talk to a USB device. I'd like this too but it's beyond by ken :( (It would be handy for Xilinx Webpack FPGA programming tools) -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081024/61ab0fa6/attachment.pgp From doconnor at gsoft.com.au Fri Oct 24 23:50:40 2008 From: doconnor at gsoft.com.au (Daniel O'Connor) Date: Fri Oct 24 23:50:56 2008 Subject: Why does adding /usr/lib32 to LD_LIBRARY_PATH break 64-bit binaries? In-Reply-To: <20081024125059.GE1137@server.vk2pj.dyndns.org> References: <3c0b01820810231731s1b4d4659j7d1df8bf4abb229c@mail.gmail.com> <20081024104232.X21603@wojtek.tensor.gdynia.pl> <20081024125059.GE1137@server.vk2pj.dyndns.org> Message-ID: <200810250958.15130.doconnor@gsoft.com.au> On Friday 24 October 2008 23:20:59 Peter Jeremy wrote: > >this will make system trying to bind 32-bit libs to 64-bit program. it > >can't work > > rtld shouldn't attempt to bind 32-bit libs to 64-bit programs. The same problem happens with the Linux run time linker - it merrily tries to link FreeBSD libraries to Linux binaries with predictable results.. One trick I use for that is to put a symlink in /compat/linux in the place the problematic FreeBSD library is.. That said it would be really nice if it ignored incompatible libraries :) -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081024/d73d9494/attachment.pgp From danny at cs.huji.ac.il Sat Oct 25 07:48:17 2008 From: danny at cs.huji.ac.il (Danny Braniss) Date: Sat Oct 25 07:48:25 2008 Subject: zfs & waiting on zio->io_cv In-Reply-To: <20081024150916.GB41283@dan.emsphone.com> References: <20081024150916.GB41283@dan.emsphone.com> Message-ID: > In the last episode (Oct 24), Danny Braniss said: > > there is a big delay (probably more than 1 sec.) when doing simple tasks > > on this zfs, like ls(1), or 'zfs list', long enough to hit ^T > > and get the same [zio->io_cv)], any hints? > > > > store-01# zfs list > > (hitting ^T)load: 0.00 cmd: zfs 88376 [zio->io_cv)] 0.00u 0.00s 0% 1672k > > (hitting ^T)load: 0.00 cmd: zfs 88376 [zio->io_cv)] 0.00u 0.00s 0% 1684k > > NAME USED AVAIL REFER MOUNTPOINT > > h 472G 11.2T 23K /h > > h/home 466G 11.2T 466G /h/home > > h/home@23-10-08 54K - 466G - > > h/root 18K 11.2T 18K /h/root > > h/src 18K 11.2T 18K /h/src > > h/system 5.64G 11.2T 5.64G /h/system > > That's sort of the equivalent to waiting in "biord" on a UFS > filesystem, I think. ZFS is just waiting for the disk to return a > block. If you happen to do something during the window where ZFS is > commiting its transaction group, it has to wait until the sync > finishes. If some other process is doing a lot of writes, or you only > have one disk in your zpool, or your pool is close to full, it may take > a couple seconds to sync. > > There's a couple of things you can try to improve interactive > performance. Raising zfs's arc_max is the easiest to do, and will let > ZFS cache more stuff, increasing the likelyhood that an "ls" will be > able to read from cache instead of having to go to disk. Setting it at > 1/4 your physical RAM is probably as high as you can go without causing > panics. > > Raising txg_time ( in /sys/cddl/.../zfs/txg.c ) from 5 to > say 30 will tell zfs to sync less often, which can be a win if you > don't actually do that much writing. With a single spindle, it may > take a substantial fraction of a second just to sync a tiny txg due to > the number of copies of metadata ZFS writes for redundancy. > > If you do a lot of writing, lowering zfs_vdev_max_pending ( in > /sys/cddl/.../zfs/vdev_queue.c ) from 35 down to 16 or less will reduce > the number of simultaneous I/Os ZFS will try to send to each disk, > which will let your reads compete a little better with other I/O. On > ATA or SATA disks, you might want to set it to 2. > ok, forgot to mention a small detail, the machine is a cuad core, with 8gb of main memory, the disks are 14x1tb connected via a perc/raid5 tests show that disk access is quiet fast, over 200Mg/s. the 'delays' are seen when the machine is totaly idle. (it's not production yet) and been up for some time. btw, I can't reproduce the 'delay', so I think it has to do with caching. I guess this beast needs some tunning, are there any tools out there to monitor/tune ZFS? thanks, danny > -- > Dan Nelson > dnelson@allantgroup.com From en0f at bokey.mine.nu Sat Oct 25 07:54:32 2008 From: en0f at bokey.mine.nu (en0f) Date: Sat Oct 25 07:54:38 2008 Subject: neophyte: tcsetattr() gives 22 error in i386, not in amd64? In-Reply-To: References: <539c60b90810241534l6bedc5e3s1c2e3162c2a7ff38@mail.gmail.com> Message-ID: <4902CC86.8030408@bokey.mine.nu> Nate Eldredge wrote: > On Fri, 24 Oct 2008, Steve Franks wrote: > >> Hi, >> >> I'm getting a 22 errno from tcsetattr() on 7-STABLE i386 in code which >> was working under 7-STABLE amd64. Serial device is a ucom (silabs >> cp2103). Permissions on /dev/cuaU0 look fine. Cutecom/Minicom >> appears to open the port without error... > > I don't see anything obviously wrong, but I'd bet a bug related to > 32/64-bit types. Can you post a complete piece of code that can be > compiled and run and demonstrates the problem? Also, try compiling with > -Wall -W and investigate any warnings that are produced. > > By the way, errno 22 is EINVAL, "Invalid argument". perror() is your > friend. Strange freebsd doesnt document error numbers. On POSIX, errno 22 is EINVAL as well (documented in errno(3)). Is this applicable to freebsd? -- en0f From koitsu at FreeBSD.org Sat Oct 25 07:56:50 2008 From: koitsu at FreeBSD.org (Jeremy Chadwick) Date: Sat Oct 25 07:56:57 2008 Subject: neophyte: tcsetattr() gives 22 error in i386, not in amd64? In-Reply-To: <4902CC86.8030408@bokey.mine.nu> References: <539c60b90810241534l6bedc5e3s1c2e3162c2a7ff38@mail.gmail.com> <4902CC86.8030408@bokey.mine.nu> Message-ID: <20081025075648.GB55339@icarus.home.lan> On Sat, Oct 25, 2008 at 06:06:38PM +1030, en0f wrote: > Nate Eldredge wrote: > > On Fri, 24 Oct 2008, Steve Franks wrote: > > > >> Hi, > >> > >> I'm getting a 22 errno from tcsetattr() on 7-STABLE i386 in code which > >> was working under 7-STABLE amd64. Serial device is a ucom (silabs > >> cp2103). Permissions on /dev/cuaU0 look fine. Cutecom/Minicom > >> appears to open the port without error... > > > > I don't see anything obviously wrong, but I'd bet a bug related to > > 32/64-bit types. Can you post a complete piece of code that can be > > compiled and run and demonstrates the problem? Also, try compiling with > > -Wall -W and investigate any warnings that are produced. > > > > By the way, errno 22 is EINVAL, "Invalid argument". perror() is your > > friend. > > Strange freebsd doesnt document error numbers. On POSIX, errno 22 is > EINVAL as well (documented in errno(3)). Is this applicable to freebsd? /usr/include/errno.h isn't documentation of error numbers? -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From en0f at bokey.mine.nu Sat Oct 25 08:00:00 2008 From: en0f at bokey.mine.nu (en0f) Date: Sat Oct 25 08:00:06 2008 Subject: neophyte: tcsetattr() gives 22 error in i386, not in amd64? In-Reply-To: <20081025075648.GB55339@icarus.home.lan> References: <539c60b90810241534l6bedc5e3s1c2e3162c2a7ff38@mail.gmail.com> <4902CC86.8030408@bokey.mine.nu> <20081025075648.GB55339@icarus.home.lan> Message-ID: <4902D1FA.3070003@bokey.mine.nu> Jeremy Chadwick wrote: > On Sat, Oct 25, 2008 at 06:06:38PM +1030, en0f wrote: >> Nate Eldredge wrote: >>> On Fri, 24 Oct 2008, Steve Franks wrote: >>> >>>> Hi, >>>> >>>> I'm getting a 22 errno from tcsetattr() on 7-STABLE i386 in code which >>>> was working under 7-STABLE amd64. Serial device is a ucom (silabs >>>> cp2103). Permissions on /dev/cuaU0 look fine. Cutecom/Minicom >>>> appears to open the port without error... >>> I don't see anything obviously wrong, but I'd bet a bug related to >>> 32/64-bit types. Can you post a complete piece of code that can be >>> compiled and run and demonstrates the problem? Also, try compiling with >>> -Wall -W and investigate any warnings that are produced. >>> >>> By the way, errno 22 is EINVAL, "Invalid argument". perror() is your >>> friend. >> Strange freebsd doesnt document error numbers. On POSIX, errno 22 is >> EINVAL as well (documented in errno(3)). Is this applicable to freebsd? > > /usr/include/errno.h isn't documentation of error numbers? Gahhhhh! But Jeremy, I dont have magic brains to work me way out of source code :) -- en0f From koitsu at FreeBSD.org Sat Oct 25 08:02:01 2008 From: koitsu at FreeBSD.org (Jeremy Chadwick) Date: Sat Oct 25 08:02:13 2008 Subject: zfs & waiting on zio->io_cv In-Reply-To: References: <20081024150916.GB41283@dan.emsphone.com> Message-ID: <20081025075543.GA55339@icarus.home.lan> On Sat, Oct 25, 2008 at 09:48:15AM +0200, Danny Braniss wrote: > > In the last episode (Oct 24), Danny Braniss said: > > > there is a big delay (probably more than 1 sec.) when doing simple tasks > > > on this zfs, like ls(1), or 'zfs list', long enough to hit ^T > > > and get the same [zio->io_cv)], any hints? > > > > > > store-01# zfs list > > > (hitting ^T)load: 0.00 cmd: zfs 88376 [zio->io_cv)] 0.00u 0.00s 0% 1672k > > > (hitting ^T)load: 0.00 cmd: zfs 88376 [zio->io_cv)] 0.00u 0.00s 0% 1684k > > > NAME USED AVAIL REFER MOUNTPOINT > > > h 472G 11.2T 23K /h > > > h/home 466G 11.2T 466G /h/home > > > h/home@23-10-08 54K - 466G - > > > h/root 18K 11.2T 18K /h/root > > > h/src 18K 11.2T 18K /h/src > > > h/system 5.64G 11.2T 5.64G /h/system > > > > That's sort of the equivalent to waiting in "biord" on a UFS > > filesystem, I think. ZFS is just waiting for the disk to return a > > block. If you happen to do something during the window where ZFS is > > commiting its transaction group, it has to wait until the sync > > finishes. If some other process is doing a lot of writes, or you only > > have one disk in your zpool, or your pool is close to full, it may take > > a couple seconds to sync. > > > > There's a couple of things you can try to improve interactive > > performance. Raising zfs's arc_max is the easiest to do, and will let > > ZFS cache more stuff, increasing the likelyhood that an "ls" will be > > able to read from cache instead of having to go to disk. Setting it at > > 1/4 your physical RAM is probably as high as you can go without causing > > panics. > > > > Raising txg_time ( in /sys/cddl/.../zfs/txg.c ) from 5 to > > say 30 will tell zfs to sync less often, which can be a win if you > > don't actually do that much writing. With a single spindle, it may > > take a substantial fraction of a second just to sync a tiny txg due to > > the number of copies of metadata ZFS writes for redundancy. > > > > If you do a lot of writing, lowering zfs_vdev_max_pending ( in > > /sys/cddl/.../zfs/vdev_queue.c ) from 35 down to 16 or less will reduce > > the number of simultaneous I/Os ZFS will try to send to each disk, > > which will let your reads compete a little better with other I/O. On > > ATA or SATA disks, you might want to set it to 2. > > > ok, forgot to mention a small detail, the machine is a cuad core, with 8gb > of main memory, the disks are 14x1tb connected via a perc/raid5 > tests show that disk access is quiet fast, over 200Mg/s. > > the 'delays' are seen when the machine is totaly idle. (it's not production > yet) > and been up for some time. btw, I can't reproduce the 'delay', so I think > it has to do with caching. > > I guess this beast needs some tunning, are there any tools out there > to monitor/tune ZFS? Monitor ZFS: sysctl Tune ZFS: vi /boot/loader.conf or sysctl I'm not sure what you're looking for. :-) -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From koitsu at FreeBSD.org Sat Oct 25 08:13:07 2008 From: koitsu at FreeBSD.org (Jeremy Chadwick) Date: Sat Oct 25 08:13:14 2008 Subject: neophyte: tcsetattr() gives 22 error in i386, not in amd64? In-Reply-To: <4902D1FA.3070003@bokey.mine.nu> References: <539c60b90810241534l6bedc5e3s1c2e3162c2a7ff38@mail.gmail.com> <4902CC86.8030408@bokey.mine.nu> <20081025075648.GB55339@icarus.home.lan> <4902D1FA.3070003@bokey.mine.nu> Message-ID: <20081025081305.GA55683@icarus.home.lan> On Sat, Oct 25, 2008 at 06:29:54PM +1030, en0f wrote: > Jeremy Chadwick wrote: > > On Sat, Oct 25, 2008 at 06:06:38PM +1030, en0f wrote: > >> Nate Eldredge wrote: > >>> On Fri, 24 Oct 2008, Steve Franks wrote: > >>> > >>>> Hi, > >>>> > >>>> I'm getting a 22 errno from tcsetattr() on 7-STABLE i386 in code which > >>>> was working under 7-STABLE amd64. Serial device is a ucom (silabs > >>>> cp2103). Permissions on /dev/cuaU0 look fine. Cutecom/Minicom > >>>> appears to open the port without error... > >>> I don't see anything obviously wrong, but I'd bet a bug related to > >>> 32/64-bit types. Can you post a complete piece of code that can be > >>> compiled and run and demonstrates the problem? Also, try compiling with > >>> -Wall -W and investigate any warnings that are produced. > >>> > >>> By the way, errno 22 is EINVAL, "Invalid argument". perror() is your > >>> friend. > >> Strange freebsd doesnt document error numbers. On POSIX, errno 22 is > >> EINVAL as well (documented in errno(3)). Is this applicable to freebsd? > > > > /usr/include/errno.h isn't documentation of error numbers? > > Gahhhhh! But Jeremy, I dont have magic brains to work me way out of > source code :) You're confusing me. :P The errors in errno.h are commented, and it's quite readable. Of course, it matches what's in errno(3). If you're wanting to track down how/why tcsetattr(3) results in EINVAL, using truss or ktrace might come in handy. Otherwise, you literally will have to throw some debugging code into the ucom(4) driver to try and figure out what function is kicking out code 22. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From danny at cs.huji.ac.il Sat Oct 25 09:18:38 2008 From: danny at cs.huji.ac.il (Danny Braniss) Date: Sat Oct 25 09:18:45 2008 Subject: zfs & waiting on zio->io_cv In-Reply-To: <20081025075543.GA55339@icarus.home.lan> References: <20081024150916.GB41283@dan.emsphone.com> <20081025075543.GA55339@icarus.home.lan> Message-ID: [...] > > I guess this beast needs some tunning, are there any tools out there > > to monitor/tune ZFS? > > Monitor ZFS: sysctl > Tune ZFS: vi /boot/loader.conf or sysctl > > I'm not sure what you're looking for. :-) the magic wand :-) /OT there's an old saying, numbers don't lie, only _____. [you can fill the _____ with whatever, I like 'salesperson'] update: ____ should be wall-street-analysts /OT/ there are too many tunables, and I really don't want to spend all my waking hours tunning, but once in a while, I do check performance, and running top, iostat, nfsstat, etc, sometimes show a 'congestion'. the problem I encounter very often, is that what is a great solution for one, might be disastrous for another. ZFS is the new kid in town, and so far I am quiet happy - have 2 in production, and the new one will be very soon, it's being used to train/learn/play but will go online very soon, then it will be hammered by several projects, and it would-be-nice to have some tool to watch performance and be able to do some tunning, to quote tunefs: 'You can tune a file system, but you cannot tune a fish.' babbling on a rainy day, danny From njm at njm.f2s.com Sat Oct 25 09:31:37 2008 From: njm at njm.f2s.com (N.J. Mann) Date: Sat Oct 25 09:31:44 2008 Subject: neophyte: tcsetattr() gives 22 error in i386, not in amd64? In-Reply-To: <20081025081305.GA55683@icarus.home.lan> References: <539c60b90810241534l6bedc5e3s1c2e3162c2a7ff38@mail.gmail.com> <4902CC86.8030408@bokey.mine.nu> <20081025075648.GB55339@icarus.home.lan> <4902D1FA.3070003@bokey.mine.nu> <20081025081305.GA55683@icarus.home.lan> Message-ID: <20081025092157.GB86967@oberon.njm.f2s.com> In message <20081025081305.GA55683@icarus.home.lan>, Jeremy Chadwick (koitsu@FreeBSD.org) wrote: > On Sat, Oct 25, 2008 at 06:29:54PM +1030, en0f wrote: > > Jeremy Chadwick wrote: > > > On Sat, Oct 25, 2008 at 06:06:38PM +1030, en0f wrote: > > >> Nate Eldredge wrote: > > >>> On Fri, 24 Oct 2008, Steve Franks wrote: > > >>> > > >>>> Hi, > > >>>> > > >>>> I'm getting a 22 errno from tcsetattr() on 7-STABLE i386 in code which > > >>>> was working under 7-STABLE amd64. Serial device is a ucom (silabs > > >>>> cp2103). Permissions on /dev/cuaU0 look fine. Cutecom/Minicom > > >>>> appears to open the port without error... > > >>> I don't see anything obviously wrong, but I'd bet a bug related to > > >>> 32/64-bit types. Can you post a complete piece of code that can be > > >>> compiled and run and demonstrates the problem? Also, try compiling with > > >>> -Wall -W and investigate any warnings that are produced. > > >>> > > >>> By the way, errno 22 is EINVAL, "Invalid argument". perror() is your > > >>> friend. > > >> Strange freebsd doesnt document error numbers. On POSIX, errno 22 is > > >> EINVAL as well (documented in errno(3)). Is this applicable to freebsd? > > > > > > /usr/include/errno.h isn't documentation of error numbers? > > > > Gahhhhh! But Jeremy, I dont have magic brains to work me way out of > > source code :) > > You're confusing me. :P The errors in errno.h are commented, and it's > quite readable. Of course, it matches what's in errno(3). Just in case someone actually goes looking for errno(3)... It is actually errno(2), but I'm sure you knew that. :-) Cheers, Nick. -- From njm at njm.f2s.com Sat Oct 25 09:46:43 2008 From: njm at njm.f2s.com (N.J. Mann) Date: Sat Oct 25 09:46:56 2008 Subject: neophyte: tcsetattr() gives 22 error in i386, not in amd64? In-Reply-To: <4902CC86.8030408@bokey.mine.nu> References: <539c60b90810241534l6bedc5e3s1c2e3162c2a7ff38@mail.gmail.com> <4902CC86.8030408@bokey.mine.nu> Message-ID: <20081025091721.GA86967@oberon.njm.f2s.com> In message <4902CC86.8030408@bokey.mine.nu>, en0f (en0f@bokey.mine.nu) wrote: > Nate Eldredge wrote: > > On Fri, 24 Oct 2008, Steve Franks wrote: > > > >> Hi, > >> > >> I'm getting a 22 errno from tcsetattr() on 7-STABLE i386 in code which > >> was working under 7-STABLE amd64. Serial device is a ucom (silabs > >> cp2103). Permissions on /dev/cuaU0 look fine. Cutecom/Minicom > >> appears to open the port without error... > > > > I don't see anything obviously wrong, but I'd bet a bug related to > > 32/64-bit types. Can you post a complete piece of code that can be > > compiled and run and demonstrates the problem? Also, try compiling with > > -Wall -W and investigate any warnings that are produced. > > > > By the way, errno 22 is EINVAL, "Invalid argument". perror() is your > > friend. > > Strange freebsd doesnt document error numbers. man errno Cheers, Nick. -- From pisymbol at gmail.com Sat Oct 25 12:49:20 2008 From: pisymbol at gmail.com (Alexander Sack) Date: Sat Oct 25 12:49:26 2008 Subject: Why does adding /usr/lib32 to LD_LIBRARY_PATH break 64-bit binaries? In-Reply-To: <200810250958.15130.doconnor@gsoft.com.au> References: <3c0b01820810231731s1b4d4659j7d1df8bf4abb229c@mail.gmail.com> <20081024104232.X21603@wojtek.tensor.gdynia.pl> <20081024125059.GE1137@server.vk2pj.dyndns.org> <200810250958.15130.doconnor@gsoft.com.au> Message-ID: <3c0b01820810250549r6c1f5614i27709c09d73a2018@mail.gmail.com> On Fri, Oct 24, 2008 at 7:28 PM, Daniel O'Connor wrote: > On Friday 24 October 2008 23:20:59 Peter Jeremy wrote: >> >this will make system trying to bind 32-bit libs to 64-bit program. it >> >can't work >> >> rtld shouldn't attempt to bind 32-bit libs to 64-bit programs. > > The same problem happens with the Linux run time linker - it merrily tries to > link FreeBSD libraries to Linux binaries with predictable results.. > > One trick I use for that is to put a symlink in /compat/linux in the place the > problematic FreeBSD library is.. > > That said it would be really nice if it ignored incompatible libraries :) Yea. Also you go tot realize that if it didn't pick up /usr/lib32 shared objects then /lib would be searched as a last resort I believe since its the default path. As a result, things again would have just worked. Is this a bug or not in FreeBSD's rtld? -aps From kabaev at gmail.com Sat Oct 25 13:57:15 2008 From: kabaev at gmail.com (Alexander Kabaev) Date: Sat Oct 25 13:57:22 2008 Subject: Why does adding /usr/lib32 to LD_LIBRARY_PATH break 64-bit binaries? In-Reply-To: <3c0b01820810250549r6c1f5614i27709c09d73a2018@mail.gmail.com> References: <3c0b01820810231731s1b4d4659j7d1df8bf4abb229c@mail.gmail.com> <20081024104232.X21603@wojtek.tensor.gdynia.pl> <20081024125059.GE1137@server.vk2pj.dyndns.org> <200810250958.15130.doconnor@gsoft.com.au> <3c0b01820810250549r6c1f5614i27709c09d73a2018@mail.gmail.com> Message-ID: <20081025095707.5226d663@kan.dnsalias.net> On Sat, 25 Oct 2008 08:49:19 -0400 "Alexander Sack" wrote: > > Is this a bug or not in FreeBSD's rtld? > > -aps It is not. In case it was not clear before, I maintain that you _ask_ rtld for wrong behaviour and you get back what you asked for, down to the letter. 'Tasting' libraries just because someone somewhere want to screw up their configuration does not seem right to me at all. -- Alexander Kabaev -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081025/da9b8e5f/signature.pgp From en0f at bokey.mine.nu Sat Oct 25 15:00:38 2008 From: en0f at bokey.mine.nu (en0f) Date: Sat Oct 25 15:00:45 2008 Subject: neophyte: tcsetattr() gives 22 error in i386, not in amd64? In-Reply-To: <20081025092157.GB86967@oberon.njm.f2s.com> References: <539c60b90810241534l6bedc5e3s1c2e3162c2a7ff38@mail.gmail.com> <4902CC86.8030408@bokey.mine.nu> <20081025075648.GB55339@icarus.home.lan> <4902D1FA.3070003@bokey.mine.nu> <20081025081305.GA55683@icarus.home.lan> <20081025092157.GB86967@oberon.njm.f2s.com> Message-ID: <49033485.5000105@bokey.mine.nu> N.J. Mann wrote: > In message <20081025081305.GA55683@icarus.home.lan>, > Jeremy Chadwick (koitsu@FreeBSD.org) wrote: >> On Sat, Oct 25, 2008 at 06:29:54PM +1030, en0f wrote: >>> Jeremy Chadwick wrote: >>>> On Sat, Oct 25, 2008 at 06:06:38PM +1030, en0f wrote: >>>>> Nate Eldredge wrote: >>>>>> On Fri, 24 Oct 2008, Steve Franks wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I'm getting a 22 errno from tcsetattr() on 7-STABLE i386 in code which >>>>>>> was working under 7-STABLE amd64. Serial device is a ucom (silabs >>>>>>> cp2103). Permissions on /dev/cuaU0 look fine. Cutecom/Minicom >>>>>>> appears to open the port without error... >>>>>> I don't see anything obviously wrong, but I'd bet a bug related to >>>>>> 32/64-bit types. Can you post a complete piece of code that can be >>>>>> compiled and run and demonstrates the problem? Also, try compiling with >>>>>> -Wall -W and investigate any warnings that are produced. >>>>>> >>>>>> By the way, errno 22 is EINVAL, "Invalid argument". perror() is your >>>>>> friend. >>>>> Strange freebsd doesnt document error numbers. On POSIX, errno 22 is >>>>> EINVAL as well (documented in errno(3)). Is this applicable to freebsd? >>>> /usr/include/errno.h isn't documentation of error numbers? >>> Gahhhhh! But Jeremy, I dont have magic brains to work me way out of >>> source code :) >> You're confusing me. :P The errors in errno.h are commented, and it's >> quite readable. Of course, it matches what's in errno(3). > > Just in case someone actually goes looking for errno(3)... It is > actually errno(2), but I'm sure you knew that. :-) Was a good discussion :) Jeremy, not everyone's going to madly grep the fbsd source are they? :P Nick, I was actually reading some non-authoritative manpage online[1] so did not find one so got quite a bit excited! Found it now ;) Thanks for the pointer. I mainly subscribe to fbsd-hackers because of the pool of knowledge people have & share on the list. :) [1] !http://www.freebsd.org/cgi/man.cgi -- en0f From thierry.herbelot at laposte.net Sat Oct 25 15:05:45 2008 From: thierry.herbelot at laposte.net (Thierry Herbelot) Date: Sat Oct 25 15:05:53 2008 Subject: question about sb->st_blksize in src/sys/kern/vfs_vnops.c In-Reply-To: <20081025203549.C76165@delplex.bde.org> References: <200810241818.37262.thierry@herbelot.com> <20081025203549.C76165@delplex.bde.org> Message-ID: <200810251639.17586.thierry.herbelot@laposte.net> Le Saturday 25 October 2008, Bruce Evans a ?crit : > On Fri, 24 Oct 2008, Thierry Herbelot wrote: > > the [SUBJ] file contains the following extract (around line 705) : > > > > * Default to PAGE_SIZE after much discussion. > > * XXX: min(PAGE_SIZE, vp->v_bufobj.bo_bsize) may be more correct. > > */ > > > > sb->st_blksize = PAGE_SIZE; > > > > which arrived around four years ago, with revision 1.211 (see > > http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/vfs_vnops.c.diff?r1=1. > >210;r2=1.211;f=h) > > Indeed, this was completely broken long ago (in 1.211). Before then, and > after 1.128, some cases worked as intended if not perfectly: > - regular files: file systems still set va_blksize to their idea of the > best i/o size (normally to the file system block size, which is > normally larger than PAGE_SIZE and probably better in all cases) and > this was used here. However, for regular files, the fs block size > and the application's i/o size are almost irrelevant in most cases > due to vfs clustering. Most large i/o's are done physically with > the cluster size (which due to a related bug suite ends up being > hard-coded to MAXPHYS (128K) at a minor cost when this is different > from the best size). > - disk files: non-broken device drivers set si_iosize_best to their idea > of the best i/o size (normally to the max i/o size, which is normally > better than PAGE_SIZE) and this was used here. The bogus default > of BLKDEV_IOSIZE was used for broken drivers (this is bogus because it > was for the buffer cache implementation for block devices which no > longer exist and was too small for them anyway). > - non-disk character-special files: the default of PAGE_SIZE was used. > The comment about defaulting to PAGE_SIZE was added in 1.128 and is > mainly for this case. Now the comment is nonsense since the value is > fixed, not a default. > - other file types (fifos, pipes, sockets, ...): these got the default of > PAGE_SIZE too. > > In rev.1.1, st_blksize was set to va_blksize in all cases. So file systems > were supposed to set va_blksize reasonably in all cases, but this is not > easy and they did nothing good except for regular files. agreed, anyway the comment by phk about using ioctl(DIOCGSECTORSIZE) applies. > > Versions between 1.2 and 1.127 did weird things like defaulting to DFLTPHYS > (64K) for most cdevs but using a small size like BLKDEV_IOSIZE (2K) for > disks. This gave nonsense like 64K buffers for slow tty devices (keyboards) > and 2K buffers for fast disks. At least for programs that trust st_blksize > o be reasonable. Fortunately, st_blsize is rarely used... > > > the net effect of this change is to decrease the block buffer size used > > in libc/stdio from 16 kbytes (derived from the underlying ufs partition) > > to PAGE_SIZE ==4 kbytes (fixed value), and consequently the I/O bandwidth > > is lowered (this is on a slow Flash). > > ... except it is used by stdio. (Another mess here is that stdio mostly > doesn't use its own BUFSIZ. It trusts st_blksize if fstat() to determine This is indeed what I saw, meandering between the libc and the vfs part of the kernel. In fact, I was essentially wondering if st_blksize was used *elsewhere*, and bumping the value could break some memory allocation ... > st_blksize works. Of course, the existence of BUFSIZ is a related > historical mistake -- no fixed size can work best for all cases. But > when BUFSIZ is used, it is an even worse default than PAGE_SIZE.) (as it is even smaller ?) > > It's interesting that you can see the difference. Clustering is especially > good for hiding slowness on slow devices. Maybe you are using a > configuration that makes clustering ineffective. Mounting the file system > with -o sync or equivalently, doing a sync after every (too-small) write > would do it. Otherwise, writes are normally delated until the next cluster > boundary. My use case is for small (buffered) writes to a file between 4 kbytes and 16 16 kbytes. For example, writing a 16-kbyte file with a st_blksize of 4k is twice as slow as with 16k (220 ms compared to 110). The penalty is less for 8k-byte (105 ms vs 66). > > > I have patched the kernel with a larger, fixed value (simply 4*PAGE_SIZE, > > to revert to the block size previoulsly used), and the kernel and world > > seem to be running fine. > > > > Seeing the XXX coment above, I'm a bit worried about keeping this new > > st_blksize value. > > > > are there any drawbacks with running with this bigger buffer size value ? > > Mostly it doesn't matter, since buffering (clustering) hides the > differences. (as seen before, mostly) > Without clustering, 16K is a much better default for disks > than 4K, though not as good as the non-default va_blksize for regular > files. Newer disks might prefer 32K or 64k, but then the fs block size > should also be increased from 16K. Otherwise, increasing the block size > usually reduces performance, by thrashing caches or increasing latencies. > With modern cache sizes and disk speeds, you won't see these effects for a > block size of 64K, so defaulting to 64K would be reasonable for disks. It > would be silly for keyboards, but with modern memory sizes you would notice > this even less than when it was that in old versions. OK, thanks for the answer : I will submit the change to more stress tests and hope to shake it all before putting it to production. TfH > > Bruce From george+freebsd at m5p.com Sat Oct 25 16:39:19 2008 From: george+freebsd at m5p.com (george+freebsd@m5p.com) Date: Sat Oct 25 16:39:27 2008 Subject: Severe DNS Problems, 6.2-RELEASE, BIND 9.5.2 Message-ID: <200810251639.m9PGdCsY082807@m5p.com> Stupid configuration error on my part. I still had my old ISP's name servers configured in my named.conf forwarders statement, and they have apparently been responding to me for eighteen months -- until two days ago. Sorry for the noise! -- George Mitchell From pisymbol at gmail.com Sat Oct 25 17:10:54 2008 From: pisymbol at gmail.com (Alexander Sack) Date: Sat Oct 25 17:11:01 2008 Subject: Why does adding /usr/lib32 to LD_LIBRARY_PATH break 64-bit binaries? In-Reply-To: <20081025095707.5226d663@kan.dnsalias.net> References: <3c0b01820810231731s1b4d4659j7d1df8bf4abb229c@mail.gmail.com> <20081024104232.X21603@wojtek.tensor.gdynia.pl> <20081024125059.GE1137@server.vk2pj.dyndns.org> <200810250958.15130.doconnor@gsoft.com.au> <3c0b01820810250549r6c1f5614i27709c09d73a2018@mail.gmail.com> <20081025095707.5226d663@kan.dnsalias.net> Message-ID: <3c0b01820810251010n17ba274dsf0a543b8287e8e65@mail.gmail.com> On Sat, Oct 25, 2008 at 9:57 AM, Alexander Kabaev wrote: > On Sat, 25 Oct 2008 08:49:19 -0400 > "Alexander Sack" wrote: > >> >> Is this a bug or not in FreeBSD's rtld? >> >> -aps > > It is not. In case it was not clear before, I maintain that you _ask_ > rtld for wrong behaviour and you get back what you asked for, down to > the letter. 'Tasting' libraries just because someone somewhere want to > screw up their configuration does not seem right to me at all. I maintain that rtld should not load 32-bit libraries for a 64-bit binary. That is WRONG anyway you look at it. And again, if it checked the arch type and skipped libutil.so.5 in /usr/lib32 it would fall back to checking /lib and things would work. Moreover, if /usr/lib had major number links just like /usr/lib32 has, this would again have worked without issue. I believe this will be fixed on the other side of the fence (not setting LD_LIBRARY_PATH to include /usr/lib32 to begin wtih) but still, my point still stands. -aps From kabaev at gmail.com Sat Oct 25 18:55:53 2008 From: kabaev at gmail.com (Alexander Kabaev) Date: Sat Oct 25 18:56:00 2008 Subject: Why does adding /usr/lib32 to LD_LIBRARY_PATH break 64-bit binaries? In-Reply-To: <3c0b01820810251010n17ba274dsf0a543b8287e8e65@mail.gmail.com> References: <3c0b01820810231731s1b4d4659j7d1df8bf4abb229c@mail.gmail.com> <20081024104232.X21603@wojtek.tensor.gdynia.pl> <20081024125059.GE1137@server.vk2pj.dyndns.org> <200810250958.15130.doconnor@gsoft.com.au> <3c0b01820810250549r6c1f5614i27709c09d73a2018@mail.gmail.com> <20081025095707.5226d663@kan.dnsalias.net> <3c0b01820810251010n17ba274dsf0a543b8287e8e65@mail.gmail.com> Message-ID: <20081025145545.52db2d8a@kan.dnsalias.net> On Sat, 25 Oct 2008 13:10:53 -0400 "Alexander Sack" wrote: > On Sat, Oct 25, 2008 at 9:57 AM, Alexander Kabaev > wrote: > > On Sat, 25 Oct 2008 08:49:19 -0400 > > "Alexander Sack" wrote: > > > >> > >> Is this a bug or not in FreeBSD's rtld? > >> > >> -aps > > > > It is not. In case it was not clear before, I maintain that you > > _ask_ rtld for wrong behaviour and you get back what you asked for, > > down to the letter. 'Tasting' libraries just because someone > > somewhere want to screw up their configuration does not seem right > > to me at all. > > I maintain that rtld should not load 32-bit libraries for a 64-bit > binary. That is WRONG anyway you look at it. And again, if it checked > the arch type and skipped libutil.so.5 in /usr/lib32 it would fall > back to checking /lib and things would work. Moreover, if /usr/lib > had major number links just like /usr/lib32 has, this would again have > worked without issue. > > I believe this will be fixed on the other side of the fence (not > setting LD_LIBRARY_PATH to include /usr/lib32 to begin wtih) but > still, my point still stands. > > -aps It doesn't. Stop feeding 32 bit libraries and it won't try to load them. It is as simple as that. For complex scenarious we do provide LD_32_ family of environment variables and if you refuse using them and insist on sticking with clearly broken configuration, it your problem, not rtld's. -- Alexander Kabaev -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081025/3848eb05/signature.pgp From brde at optusnet.com.au Sat Oct 25 19:46:24 2008 From: brde at optusnet.com.au (Bruce Evans) Date: Sat Oct 25 19:57:07 2008 Subject: question about sb->st_blksize in src/sys/kern/vfs_vnops.c In-Reply-To: <200810241818.37262.thierry@herbelot.com> References: <200810241818.37262.thierry@herbelot.com> Message-ID: <20081025203549.C76165@delplex.bde.org> On Fri, 24 Oct 2008, Thierry Herbelot wrote: > the [SUBJ] file contains the following extract (around line 705) : > > * Default to PAGE_SIZE after much discussion. > * XXX: min(PAGE_SIZE, vp->v_bufobj.bo_bsize) may be more correct. > */ > > sb->st_blksize = PAGE_SIZE; > > which arrived around four years ago, with revision 1.211 (see > http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/vfs_vnops.c.diff?r1=1.210;r2=1.211;f=h) Indeed, this was completely broken long ago (in 1.211). Before then, and after 1.128, some cases worked as intended if not perfectly: - regular files: file systems still set va_blksize to their idea of the best i/o size (normally to the file system block size, which is normally larger than PAGE_SIZE and probably better in all cases) and this was used here. However, for regular files, the fs block size and the application's i/o size are almost irrelevant in most cases due to vfs clustering. Most large i/o's are done physically with the cluster size (which due to a related bug suite ends up being hard-coded to MAXPHYS (128K) at a minor cost when this is different from the best size). - disk files: non-broken device drivers set si_iosize_best to their idea of the best i/o size (normally to the max i/o size, which is normally better than PAGE_SIZE) and this was used here. The bogus default of BLKDEV_IOSIZE was used for broken drivers (this is bogus because it was for the buffer cache implementation for block devices which no longer exist and was too small for them anyway). - non-disk character-special files: the default of PAGE_SIZE was used. The comment about defaulting to PAGE_SIZE was added in 1.128 and is mainly for this case. Now the comment is nonsense since the value is fixed, not a default. - other file types (fifos, pipes, sockets, ...): these got the default of PAGE_SIZE too. In rev.1.1, st_blksize was set to va_blksize in all cases. So file systems were supposed to set va_blksize reasonably in all cases, but this is not easy and they did nothing good except for regular files. Versions between 1.2 and 1.127 did weird things like defaulting to DFLTPHYS (64K) for most cdevs but using a small size like BLKDEV_IOSIZE (2K) for disks. This gave nonsense like 64K buffers for slow tty devices (keyboards) and 2K buffers for fast disks. At least for programs that trust st_blksize o be reasonable. Fortunately, st_blsize is rarely used... > the net effect of this change is to decrease the block buffer size used in > libc/stdio from 16 kbytes (derived from the underlying ufs partition) to > PAGE_SIZE ==4 kbytes (fixed value), and consequently the I/O bandwidth is > lowered (this is on a slow Flash). ... except it is used by stdio. (Another mess here is that stdio mostly doesn't use its own BUFSIZ. It trusts st_blksize if fstat() to determine st_blksize works. Of course, the existence of BUFSIZ is a related historical mistake -- no fixed size can work best for all cases. But when BUFSIZ is used, it is an even worse default than PAGE_SIZE.) It's interesting that you can see the difference. Clustering is especially good for hiding slowness on slow devices. Maybe you are using a configuration that makes clustering ineffective. Mounting the file system with -o sync or equivalently, doing a sync after every (too-small) write would do it. Otherwise, writes are normally delated until the next cluster boundary. > I have patched the kernel with a larger, fixed value (simply 4*PAGE_SIZE, to > revert to the block size previoulsly used), and the kernel and world seem to > be running fine. > > Seeing the XXX coment above, I'm a bit worried about keeping this new > st_blksize value. > > are there any drawbacks with running with this bigger buffer size value ? Mostly it doesn't matter, since buffering (clustering) hides the differences. Without clustering, 16K is a much better default for disks than 4K, though not as good as the non-default va_blksize for regular files. Newer disks might prefer 32K or 64k, but then the fs block size should also be increased from 16K. Otherwise, increasing the block size usually reduces performance, by thrashing caches or increasing latencies. With modern cache sizes and disk speeds, you won't see these effects for a block size of 64K, so defaulting to 64K would be reasonable for disks. It would be silly for keyboards, but with modern memory sizes you would notice this even less than when it was that in old versions. Bruce From novel at FreeBSD.org Sun Oct 26 06:27:02 2008 From: novel at FreeBSD.org (Roman Bogorodskiy) Date: Sun Oct 26 06:27:37 2008 Subject: problems obtaining kernel dump Message-ID: <20081026060906.GA66894@underworld.novel.ru> Hello, I'm having a problem obtaining kernel dump. The box has 512Mb of RAM. In rc.conf I have: dumpdev="/dev/ad4s1b" dumpdir="/var/crash" swapinfo -h gives the following: Device 1K-blocks Used Avail Capacity /dev/ad4s1b 1048576 0B 1.0G 0% /var/crash directory exists, and the root partition where it is placed has enough space as well: /dev/ad4s1a 989M 350M 560M 38% / The box runs fresh FreeBSD/i386 -CURRENT. So, I do swapoff and then perform actions to reproduce the crash and it breaks me into ddb prompt. I do: call doadump continue it reboots, and when I run "savecore -v /var/crash /dev/ad4s1b" it prints: unable to open bounds file, using 0 checking for kernel dump on device /dev/ad4s1b mediasize = 1073741824 sectorsize = 512 magic mismatch on last dump header on /dev/ad4s1b savecore: no dumps found The same happens when I do 'panic' instead of 'call doadump' like handbook suggests. What am I doing wrong? I googled for similar problems, found some mail threads but didn't find meaningful advises though. Roman Bogorodskiy From martin.laabs at mailbox.tu-dresden.de Sun Oct 26 10:49:08 2008 From: martin.laabs at mailbox.tu-dresden.de (Martin Laabs) Date: Sun Oct 26 10:49:16 2008 Subject: linux-libusb done Message-ID: Hi, I've build succesfully an androgynous libusb. This is a libusb in linux binary format with freebsd-like usb-access. (I use it to run a jtag-debugging utility which is supplied as a linux binary only.) Unfortunately I had to patch the linux.ko module to support the ioctl calls out of the linux application. I can supply the libusb.so, a patch for the source of libusb as well as the linux_ioctl.c on request. Unfotunately I did not save the original linux_ioctl.c so I can not publish a patch yet. (I'll do a cvsup these days ..) Anyone who wants to maintain such a port? Greetings, Martin From rdivacky at freebsd.org Sun Oct 26 10:54:02 2008 From: rdivacky at freebsd.org (Roman Divacky) Date: Sun Oct 26 10:54:14 2008 Subject: linux-libusb done In-Reply-To: References: Message-ID: <20081026105252.GA45809@freebsd.org> On Sun, Oct 26, 2008 at 11:49:02AM +0100, Martin Laabs wrote: > Hi, > > I've build succesfully an androgynous libusb. This is a libusb in > linux binary format with freebsd-like usb-access. (I use it to > run a jtag-debugging utility which is supplied as a linux > binary only.) > Unfortunately I had to patch the linux.ko module to support the > ioctl calls out of the linux application. > I can supply the libusb.so, a patch for the source of libusb > as well as the linux_ioctl.c on request. > Unfotunately I did not save the original linux_ioctl.c so I can not > publish a patch yet. (I'll do a cvsup these days ..) > > Anyone who wants to maintain such a port? great work martin! please show us the linuxulator patch and I'll see what I can do about it... roman From ed at 80386.nl Sun Oct 26 15:27:11 2008 From: ed at 80386.nl (Ed Schouten) Date: Sun Oct 26 15:27:51 2008 Subject: [Testers wanted] /dev/console cleanups Message-ID: <20081026152707.GJ6808@hoeg.nl> Hello everyone, Most of you probably already know that I've been very busy improving our kernel's TTY implementation. I've committed the new MPSAFE TTY layer back in August. So far most of the things seem to work properly as far as I can see. There are always some small bugs, but I'm confident we'll get them fixed before 8.0. One of the things that I dislike about the code we have right now, is the way /dev/console is implemented. There is a small amount of complexity there, which is mainly because of the fact that our console code actually works on two different levels: - We've got kernel messages that are printed using very low-level routines and communicate directly with the drivers. - We've got user messages that are printed through /dev/console, which actually work on the TTY level, but make use of a similar device selection as the first set of routines. In an attempt to make /dev/console MPSAFE, I moved /dev/console into the TTY layer itself, which makes it a lot more simple than it is now. Well, to keep a long story short, it would be wonderful if some people could test the latest MPSAFE TTY patchset, just to make sure it doesn't wreck people's setups after I commit this. So just apply the patch and see if you can still boot your system, go into single user and multi user mode, use conscontrol(8), etc. I've stored the latest MPSAFE TTY patchsets at the usual location. Make sure you download the latest version. http://people.FreeBSD.org/~ed/mpsafetty/ The patchset also includes some other nice things, like some manual pages (not finished) and a port of snp(4) to the new TTY layer (also not finished). Thank you for your attention! -- Ed Schouten WWW: http://80386.nl/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081026/3d8c4748/attachment.pgp From olli at lurza.secnetix.de Mon Oct 27 14:11:13 2008 From: olli at lurza.secnetix.de (Oliver Fromme) Date: Mon Oct 27 14:11:20 2008 Subject: Why does adding /usr/lib32 to LD_LIBRARY_PATH break 64-bit ?binaries? In-Reply-To: Message-ID: <200810271411.m9REB6te015188@lurza.secnetix.de> Daniel O'Connor wrote: > On Friday 24 October 2008 23:20:59 Peter Jeremy wrote: > > > this will make system trying to bind 32-bit libs to 64-bit program. it > > > can't work > > > > rtld shouldn't attempt to bind 32-bit libs to 64-bit programs. > > The same problem happens with the Linux run time linker - it merrily tries to > link FreeBSD libraries to Linux binaries with predictable results.. You *can* link Linux libraries with FreeBSD binaries (and vice versa), if the library does not perform any syscalls, e.g. it is a pure computation library or similar. > That said it would be really nice if it ignored incompatible libraries :) No. Please don't put such pseudo-cleverness into rtld. It wouldn't be an improvement, in fact it might break some working configurations. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Gesch?ftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M?n- chen, HRB 125758, Gesch?ftsf?hrer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "That's what I love about GUIs: They make simple tasks easier, and complex tasks impossible." -- John William Chambless From olli at lurza.secnetix.de Mon Oct 27 14:20:11 2008 From: olli at lurza.secnetix.de (Oliver Fromme) Date: Mon Oct 27 14:20:18 2008 Subject: libusb for linux-emulation In-Reply-To: Message-ID: <200810271420.m9REK9iF015611@lurza.secnetix.de> Martin Laabs wrote: > Hans Petter Selasky wrote: > > No, you cannot use the linux libusb on FreeBSD. You need to use the FreeBSD > > compiled libusb. The USB kernel interfaces are quite different. > > OK - I see. I'm just trying to build a "hermaphrodite" library. Compile with > linux but using the BSD ioctls. > Is there a crosscompiler to compile linux binarys from freebsd? This would > make the job much easier. Yes, I think you can use ports/devel/cross-gcc. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Gesch?ftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M?n- chen, HRB 125758, Gesch?ftsf?hrer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "Documentation is like sex; when it's good, it's very, very good, and when it's bad, it's better than nothing." -- Dick Brandon From imp at bsdimp.com Mon Oct 27 15:01:44 2008 From: imp at bsdimp.com (M. Warner Losh) Date: Mon Oct 27 15:01:51 2008 Subject: libusb for linux-emulation In-Reply-To: <200810271420.m9REK9iF015611@lurza.secnetix.de> References: <200810271420.m9REK9iF015611@lurza.secnetix.de> Message-ID: <20081027.090139.-1699739227.imp@bsdimp.com> In message: <200810271420.m9REK9iF015611@lurza.secnetix.de> Oliver Fromme writes: : Martin Laabs wrote: : > Hans Petter Selasky wrote: : > > No, you cannot use the linux libusb on FreeBSD. You need to use the FreeBSD : > > compiled libusb. The USB kernel interfaces are quite different. : > : > OK - I see. I'm just trying to build a "hermaphrodite" library. Compile with : > linux but using the BSD ioctls. : > Is there a crosscompiler to compile linux binarys from freebsd? This would : > make the job much easier. : : Yes, I think you can use ports/devel/cross-gcc. I've definitely got to get my cross tools changes into head. Warner From imp at bsdimp.com Mon Oct 27 15:01:55 2008 From: imp at bsdimp.com (M. Warner Losh) Date: Mon Oct 27 15:02:02 2008 Subject: Why does adding /usr/lib32 to LD_LIBRARY_PATH break 64-bit ?binaries? In-Reply-To: <200810271411.m9REB6te015188@lurza.secnetix.de> References: <200810271411.m9REB6te015188@lurza.secnetix.de> Message-ID: <20081027.090116.-1827344390.imp@bsdimp.com> In message: <200810271411.m9REB6te015188@lurza.secnetix.de> Oliver Fromme writes: : Daniel O'Connor wrote: : > On Friday 24 October 2008 23:20:59 Peter Jeremy wrote: : > > > this will make system trying to bind 32-bit libs to 64-bit program. it : > > > can't work : > > : > > rtld shouldn't attempt to bind 32-bit libs to 64-bit programs. : > : > The same problem happens with the Linux run time linker - it merrily tries to : > link FreeBSD libraries to Linux binaries with predictable results.. : : You *can* link Linux libraries with FreeBSD binaries (and : vice versa), if the library does not perform any syscalls, : e.g. it is a pure computation library or similar. : : > That said it would be really nice if it ignored incompatible libraries :) : : No. Please don't put such pseudo-cleverness into rtld. : It wouldn't be an improvement, in fact it might break some : working configurations. Yes. I have a bunch of printer drivers that I've used that link in linux shared libraries... They are in ports... Warner From mboxindia at gmail.com Mon Oct 27 22:22:37 2008 From: mboxindia at gmail.com (Srinivas) Date: Mon Oct 27 22:22:44 2008 Subject: Usage of "files" for config Message-ID: Hello, I am a beginner of freebsd kernel. I have some doubts regarding the Makefile generation using "files" by config. Could you plz answer the following doubts: I would like to know the usage of files and files.[arch] in sys/conf. Basically, I didnt get the advantage of having a common file for compilation(like files) rather than an individual Makefile in each subdirectory. I have read makefile(of mkmakefile.c in config). What is the usage of "standard", "optional" and "mandatory" and why it is followed by device. What are .m files? What are they used for? Why are some of the rules in the generated makefile *.ln like scsi_all.ln? What is ${NORMAL_LINT} and ${NORMAL_C} in the generated makefile mean? Thanks, Srinivas From doconnor at gsoft.com.au Mon Oct 27 23:59:44 2008 From: doconnor at gsoft.com.au (Daniel O'Connor) Date: Mon Oct 27 23:59:58 2008 Subject: Why does adding /usr/lib32 to LD_LIBRARY_PATH break 64-bit ?binaries? In-Reply-To: <20081027.090116.-1827344390.imp@bsdimp.com> References: <200810271411.m9REB6te015188@lurza.secnetix.de> <20081027.090116.-1827344390.imp@bsdimp.com> Message-ID: <200810281018.18786.doconnor@gsoft.com.au> On Tuesday 28 October 2008 01:31:16 M. Warner Losh wrote: > In message: <200810271411.m9REB6te015188@lurza.secnetix.de> > > Oliver Fromme writes: > : Daniel O'Connor wrote: > : > On Friday 24 October 2008 23:20:59 Peter Jeremy wrote: > : > > > this will make system trying to bind 32-bit libs to 64-bit > : > > > program. it can't work > : > > > : > > rtld shouldn't attempt to bind 32-bit libs to 64-bit programs. > : > > : > The same problem happens with the Linux run time linker - it merrily > : > tries to link FreeBSD libraries to Linux binaries with predictable > : > results.. > : > : You *can* link Linux libraries with FreeBSD binaries (and > : vice versa), if the library does not perform any syscalls, > : e.g. it is a pure computation library or similar. > : > : > That said it would be really nice if it ignored incompatible libraries > : > :) > : > : No. Please don't put such pseudo-cleverness into rtld. > : It wouldn't be an improvement, in fact it might break some > : working configurations. > > Yes. I have a bunch of printer drivers that I've used that link in > linux shared libraries... They are in ports... Good point.. The problem is really the Linux linker - it will find a FreeBSD library and try and use it ahead of a Linux one later in the search path - this prevents stuff working :) I have this exact problem with libfontconfig and Xilinx ISE. Perhaps instead of ignore, use last.. But then it doesn't really matter for the FreeBSD linker - I imagine I would have to convince Linux folks it's a good idea. -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081027/59dbe18f/attachment.pgp From unixmania at gmail.com Tue Oct 28 02:58:42 2008 From: unixmania at gmail.com (Carlos A. M. dos Santos) Date: Tue Oct 28 02:58:48 2008 Subject: [Testers wanted] /dev/console cleanups In-Reply-To: <20081026152707.GJ6808@hoeg.nl> References: <20081026152707.GJ6808@hoeg.nl> Message-ID: On Sun, Oct 26, 2008 at 1:27 PM, Ed Schouten wrote: > Hello everyone, > > Most of you probably already know that I've been very busy improving our > kernel's TTY implementation. I've committed the new MPSAFE TTY layer > back in August. So far most of the things seem to work properly as far > as I can see. There are always some small bugs, but I'm confident we'll > get them fixed before 8.0. > > One of the things that I dislike about the code we have right now, is > the way /dev/console is implemented. There is a small amount of > complexity there, which is mainly because of the fact that our console > code actually works on two different levels: > > - We've got kernel messages that are printed using very low-level > routines and communicate directly with the drivers. > > - We've got user messages that are printed through /dev/console, which > actually work on the TTY level, but make use of a similar device > selection as the first set of routines. > > In an attempt to make /dev/console MPSAFE, I moved /dev/console into the > TTY layer itself, which makes it a lot more simple than it is now. > > Well, to keep a long story short, it would be wonderful if some people > could test the latest MPSAFE TTY patchset, just to make sure it doesn't > wreck people's setups after I commit this. So just apply the patch and > see if you can still boot your system, go into single user and multi > user mode, use conscontrol(8), etc. > > I've stored the latest MPSAFE TTY patchsets at the usual location. Make > sure you download the latest version. > > http://people.FreeBSD.org/~ed/mpsafetty/ > > The patchset also includes some other nice things, like some manual > pages (not finished) and a port of snp(4) to the new TTY layer (also not > finished). > > Thank you for your attention! The patched source builds and installs flawlessy. However I observed something that seems to be a regression. If I run either xconsole or xterm -C I only see kernel messages, even though my X startup (via XDM) changes the owner of /dev/console to the logged-in user. I mean, if I do some timg like "echo OK > /dev/console", the message is echoed on /dev/ttyv0, not by xconsole This is the same problem reported by Jeff Blank on RELENG_7: http://lists.freebsd.org/pipermail/freebsd-stable/2008-September/044949.html http://lists.freebsd.org/pipermail/freebsd-stable/2008-October/045885.html -- cd /usr/ports/sysutils/life make clean From rea-fbsd at codelabs.ru Tue Oct 28 07:46:35 2008 From: rea-fbsd at codelabs.ru (Eygene Ryabinkin) Date: Tue Oct 28 07:46:43 2008 Subject: Usage of "files" for config In-Reply-To: References: Message-ID: Srinivas, good day. Tue, Oct 28, 2008 at 03:52:35AM +0530, Srinivas wrote: > I would like to know the usage of files and files.[arch] in sys/conf. > Basically, I didnt get the advantage of having a common file for > compilation(like files) rather than an individual Makefile in each > subdirectory. 'files' and 'files.$ARCH' are the input directives for the config(8) utility. Makefile is produced with the help of these files. The rationale for having 'files' and 'files.$ARCH' is simple: there are platform-specific directives and common directives. > > I have read makefile(of mkmakefile.c in config). What is the usage of > "standard", "optional" and "mandatory" and why it is followed by > device. Read comments from 'mkmakefile.c': ----- /* * If an entry is marked "mandatory", config will abort if it's * not called by a configuration line in the config file. Apart * from this, the device is handled like one marked "optional". */ ----- > What are .m files? What are they used for? They define module interfaces and are processed by the AWK script /sys/tools/makeobjops.awk. The output will be source and header files, named .c and .h. You can do 'ls *_if.[ch]' in the kernel build directory and examine some files to get a hint on what's going on. M-files are processed with the help of /sys/conf/kern.post.mk and /sys/conf/kmod.mk. > Why are some of the rules in the generated makefile *.ln like scsi_all.ln? These files are lint(1)'ed: see /sys/conf/kern.post.mk, search for LNFILES. > What is ${NORMAL_LINT} and ${NORMAL_C} in the generated makefile mean? 'make -V NORMAL_LINT' and 'make -V NORMAL_C' invoked from the kernel compilation directory should tell you about the values of these variables. They are defined by /sys/conf/kern.pre.mk, so you can examine it as well. -- Eygene _ ___ _.--. # \`.|\..----...-'` `-._.-'_.-'` # Remember that it is hard / ' ` , __.--' # to read the on-line manual )/' _/ \ `-_, / # while single-stepping the kernel. `-'" `"\_ ,_.-;_.-\_ ', fsc/as # _.-'_./ {_.' ; / # -- FreeBSD Developers handbook {_.-``-' {_/ # -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081028/07f1362a/attachment.pgp From ed at 80386.nl Tue Oct 28 08:11:56 2008 From: ed at 80386.nl (Ed Schouten) Date: Tue Oct 28 08:12:03 2008 Subject: [Testers wanted] /dev/console cleanups In-Reply-To: References: Message-ID: <20081028081154.GQ6808@hoeg.nl> Hello Carlos, * Carlos A. M. dos Santos wrote: > The patched source builds and installs flawlessy. However I observed > something that seems to be a regression. If I run either xconsole or > xterm -C I only see kernel messages, even though my X startup (via > XDM) changes the owner of /dev/console to the logged-in user. I mean, > if I do some timg like "echo OK > /dev/console", the message is echoed > on /dev/ttyv0, not by xconsole > > This is the same problem reported by Jeff Blank on RELENG_7: > > http://lists.freebsd.org/pipermail/freebsd-stable/2008-September/044949.html > http://lists.freebsd.org/pipermail/freebsd-stable/2008-October/045885.html It's nice to hear that the patch didn't break anything on your system. I hope to receive more reviews, but I think I'll just commit it this weekend (with small modifications). About the /dev/console issues: Robert Watson and I discussed this some time ago on IRC and what I did in HEAD (not RELENG_7) was that I changed TIOCCONS not to take a look at the permissions of /dev/console, but we changed it to use priv_check(). This means that right now you can only call TIOCCONS as root. I can't really understand why the problem exists on RELENG_7. About making xconsole setuid: I've read the messages you mentioned, but I think we could just alter console to call TIOCCONS and just drop privileges. An even better solution would be to just get rid of TIOCCONS and invent a better solution to capture syslog messages. I can't really understand why we want to abuse TTY's to do this. So I can't say we're working on this, but at least I can confirm the issue. -- Ed Schouten WWW: http://80386.nl/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081028/52ea00d1/attachment.pgp From mboxindia at gmail.com Tue Oct 28 09:31:38 2008 From: mboxindia at gmail.com (Srinivas) Date: Tue Oct 28 09:31:45 2008 Subject: Usage of "files" for config In-Reply-To: References: Message-ID: Eygene, Your reply is very helpful. Thank you very much. On Tue, Oct 28, 2008 at 1:16 PM, Eygene Ryabinkin wrote: >> I would like to know the usage of files and files.[arch] in sys/conf. >> Basically, I didnt get the advantage of having a common file for >> compilation(like files) rather than an individual Makefile in each >> subdirectory. > > 'files' and 'files.$ARCH' are the input directives for the config(8) > utility. Makefile is produced with the help of these files. The > rationale for having 'files' and 'files.$ARCH' is simple: there are > platform-specific directives and common directives. Still, I didnt get the purpose of having a common "files" file for the kernel to generate Makefile. I am trying to understand the advantage of this approach with the conventional way of having a makefile for each sub-directory(device or module) and recurse from top of kernel with a configuration file dictating what features need to be included in the kernel. Thanks, Srinivas From jhb at freebsd.org Tue Oct 28 15:31:05 2008 From: jhb at freebsd.org (John Baldwin) Date: Tue Oct 28 15:31:16 2008 Subject: Usage of "files" for config In-Reply-To: References: Message-ID: <200810281054.07065.jhb@freebsd.org> On Tuesday 28 October 2008 05:31:36 am Srinivas wrote: > Eygene, Your reply is very helpful. Thank you very much. > > On Tue, Oct 28, 2008 at 1:16 PM, Eygene Ryabinkin wrote: > >> I would like to know the usage of files and files.[arch] in sys/conf. > >> Basically, I didnt get the advantage of having a common file for > >> compilation(like files) rather than an individual Makefile in each > >> subdirectory. > > > > 'files' and 'files.$ARCH' are the input directives for the config(8) > > utility. Makefile is produced with the help of these files. The > > rationale for having 'files' and 'files.$ARCH' is simple: there are > > platform-specific directives and common directives. > > Still, I didnt get the purpose of having a common "files" file for the > kernel to generate Makefile. > > I am trying to understand the advantage of this approach with the > conventional way of having a makefile for each sub-directory(device or > module) and recurse from top of kernel with a configuration file > dictating what features need to be included in the kernel. The usage of config goes back to BSD itself prior to FreeBSD for one. However, I find the 'files' format a lot easier to parse and work with then the mess of .ifdef's, etc. that would end up in 'kern/Makefile' for example. -- John Baldwin From Alexander at Leidinger.net Tue Oct 28 18:13:47 2008 From: Alexander at Leidinger.net (Alexander Leidinger) Date: Tue Oct 28 20:43:42 2008 Subject: RTLD changes for non-native system (was: Re: Why does adding /usr/lib32 to LD_LIBRARY_PATH break 64-bit ?binaries?) In-Reply-To: <200810281018.18786.doconnor@gsoft.com.au> References: <200810271411.m9REB6te015188@lurza.secnetix.de> <20081027.090116.-1827344390.imp@bsdimp.com> <200810281018.18786.doconnor@gsoft.com.au> Message-ID: <20081028191335.26152gicr4ni54ys@webmail.leidinger.net> Quoting Daniel O'Connor (from Tue, 28 Oct 2008 10:18:10 +1030): > On Tuesday 28 October 2008 01:31:16 M. Warner Losh wrote: >> In message: <200810271411.m9REB6te015188@lurza.secnetix.de> >> >> Oliver Fromme writes: >> : Daniel O'Connor wrote: >> : > On Friday 24 October 2008 23:20:59 Peter Jeremy wrote: >> : > > > this will make system trying to bind 32-bit libs to 64-bit >> : > > > program. it can't work >> : > > >> : > > rtld shouldn't attempt to bind 32-bit libs to 64-bit programs. >> : > >> : > The same problem happens with the Linux run time linker - it merrily >> : > tries to link FreeBSD libraries to Linux binaries with predictable >> : > results.. >> : >> : You *can* link Linux libraries with FreeBSD binaries (and >> : vice versa), if the library does not perform any syscalls, >> : e.g. it is a pure computation library or similar. >> : >> : > That said it would be really nice if it ignored incompatible libraries >> : > :) >> : >> : No. Please don't put such pseudo-cleverness into rtld. >> : It wouldn't be an improvement, in fact it might break some >> : working configurations. >> >> Yes. I have a bunch of printer drivers that I've used that link in >> linux shared libraries... They are in ports... > > Good point.. > The problem is really the Linux linker - it will find a FreeBSD library and > try and use it ahead of a Linux one later in the search path - this prevents > stuff working :) > > I have this exact problem with libfontconfig and Xilinx ISE. > > Perhaps instead of ignore, use last.. But then it doesn't really matter for > the FreeBSD linker - I imagine I would have to convince Linux folks it's a > good idea. Please ignore for a moment that we are not talking about changing the FreeBSD RTLD anymore: Would it make sense (for us and/or for GNU) to first search for libs for the current system and if none are found to try the others? Bye, Alexander. -- There is hardly a thing in the world that some man can not make a little worse and sell a little cheaper. http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 From jrytoung at gmail.com Wed Oct 29 21:08:54 2008 From: jrytoung at gmail.com (Jerry Toung) Date: Wed Oct 29 21:09:02 2008 Subject: crash at in_pcb.c Message-ID: <86068e730810291345r738242b0lb8130bf6bd011015@mail.gmail.com> Hello List, I can realiably reproduce this crash. We have a deamon that accept several connections per sec. We use iperf and Microsoft Web application stress 1.0 to push traffic to the FreeBSD box. Without further delay, the crash dump is below. I've been troubleshooting, but I am no longer sure if this is a race condition or a stack corruption. The socket pointer between frame 12 and 11 is different. This is on 6.2, but the code for 7.0 is identical, so I think it still applies. Any hint, patching or troubleshooting this is appreciated. Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x2aef0210 fault code = supervisor read, page not present instruction pointer = 0x20:0xc0769098 stack pointer = 0x28:0xef781bc0 frame pointer = 0x28:0xef781bd0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 1166 (ndaemon) trap number = 12 panic: page fault cpuid = 0 Uptime: 8h32m25s Dumping 3325 MB (3 chunks) #0 doadump () at pcpu.h:165 165 pcpu.h: No such file or directory. in pcpu.h (kgdb) l *0xc0769098 0xc0769098 is in in_pcblookup_local (/usr/src/sys/netinet/in_pcb.c:923). 918 /usr/src/sys/netinet/in_pcb.c: No such file or directory. in /usr/src/sys/netinet/in_pcb.c (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc06c2812 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:412 #2 0xc06c2bbd in panic (fmt=0xc0940872 "%s") at /usr/src/sys/kern/kern_shutdown.c:573 #3 0xc08f3e4e in trap_fatal (frame=0xef781b80, eva=720306704) at /usr/src/sys/i386/i386/trap.c:838 #4 0xc08f3b57 in trap_pfault (frame=0xef781b80, usermode=0, eva=720306704) at /usr/src/sys/i386/i386/trap.c:745 #5 0xc08f3745 in trap (frame= {tf_fs = -277348344, tf_es = 40, tf_ds = -913309656, tf_edi = 6, tf_esi = 0, tf_ebp = -277341232, tf_isp = -277341268, tf_ebx = -1062683820, tf_edx = 720306704, tf_ecx = 14063, tf_eax = 720306704, tf_trapno = 12, tf_err = 0, tf_eip = -1065971560, tf_cs = 32, tf_eflags = 66050, tf_esp = 0, tf_ss = -1062683820}) at /usr/src/sys/i386/i386/trap.c:435 #6 0xc08dddba in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #7 0xc0769098 in in_pcblookup_local (pcbinfo=0x2aef0210, laddr={s_addr = 0}, lport_arg=720306704, wild_okay=1) at /usr/src/sys/netinet/in_pcb.c:923 #8 0xc0768452 in in_pcbbind_setup (inp=0xc97150b4, nam=0x36ef, laddrp=0xc97150ec, lportp=0xc97150ce, cred=0xc8726780) at /usr/src/sys/netinet/in_pcb.c:464 #9 0xc0767f56 in in_pcbbind (inp=0xc97150b4, nam=0x2aef0210, cred=0xc8726780) at /usr/src/sys/netinet/in_pcb.c:240 #10 0xc077f272 in tcp_connect (tp=0xc9897000, nam=0xc98a1ba0, td=0xc990e180) at /usr/src/sys/netinet/tcp_usrreq.c:864 #11 0xc077e141 in tcp_usr_connect (so=0xc9897000, nam=0xc98a1ba0, td=0xc990e180) at /usr/src/sys/netinet/tcp_usrreq.c:369 #12 0xc06fec4e in soconnect (so=0xc97b39bc, nam=0xc98a1ba0, td=0xc990e180) at /usr/src/sys/kern/uipc_socket.c:558 #13 0xc07046a8 in kern_connect (td=0xc990e180, fd=89, sa=0xc98a1ba0) at /usr/src/sys/kern/uipc_syscalls.c:536 #14 0xc070460f in connect (td=0xc990e180, uap=0xef781d04) at /usr/src/sys/kern/uipc_syscalls.c:505 #15 0xc08f4193 in syscall (frame= {tf_fs = 135725115, tf_es = 59, tf_ds = -1088487365, tf_edi = 135745024, tf_esi = -1089511444, tf_ebp = -1089514536, tf_isp = -277340828, tf_ebx = 671753396, tf_edx = 0, tf_ecx = 135524256, tf_eax = 98, tf_trapno = 0, tf_err = 2, tf_eip = 674451435, tf_cs = 51, tf_eflags = 642, tf_esp = -1089514580, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:984 #16 0xc08dde0f in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200 #17 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 7 #7 0xc0769098 in in_pcblookup_local (pcbinfo=0x2aef0210, laddr={s_addr = 0}, lport_arg=720306704, wild_okay=1) at /usr/src/sys/netinet/in_pcb.c:923 923 in /usr/src/sys/netinet/in_pcb.c (kgdb) i loc phd = (struct inpcbport *) 0x2aef0210 tmphd = (struct inpcbport *) 0x2aef0210 match = (struct inpcb *) 0x0 inp = (struct inpcb *) 0x2aef0210 tmpinp = (struct inpcb *) 0x2aef0210 matchwild = 6 wildcard = -1062683820 lport = 14063 (kgdb) p phd $1 = (struct inpcbport *) 0x2aef0210 (kgdb) p phd->phd_port Cannot access memory at address 0x2aef021c (kgdb) f 12 #12 0xc06fec4e in soconnect (so=0xc97b39bc, nam=0xc98a1ba0, td=0xc990e180) at /usr/src/sys/kern/uipc_socket.c:558 558 /usr/src/sys/kern/uipc_socket.c: No such file or directory. in /usr/src/sys/kern/uipc_socket.c (kgdb) p so $2 = (struct socket *) 0xc97b39bc (kgdb) p nam $3 = (struct sockaddr *) 0xc98a1ba0 (kgdb) p td $4 = (struct thread *) 0xc990e180 (kgdb) l 553 in /usr/src/sys/kern/uipc_socket.c (kgdb) f 11 #11 0xc077e141 in tcp_usr_connect (so=0xc9897000, nam=0xc98a1ba0, td=0xc990e180) at /usr/src/sys/netinet/tcp_usrreq.c:369 369 /usr/src/sys/netinet/tcp_usrreq.c: No such file or directory. in /usr/src/sys/netinet/tcp_usrreq.c (kgdb) From jrytoung at gmail.com Wed Oct 29 21:48:11 2008 From: jrytoung at gmail.com (Jerry Toung) Date: Wed Oct 29 21:48:17 2008 Subject: crash at in_pcb.c In-Reply-To: <3c1674c90810291437n3f0d5132t52bc2fa4f4e1b9d0@mail.gmail.com> References: <86068e730810291345r738242b0lb8130bf6bd011015@mail.gmail.com> <3c1674c90810291437n3f0d5132t52bc2fa4f4e1b9d0@mail.gmail.com> Message-ID: <86068e730810291448t342ffaa6xe2abbf6804f29c5b@mail.gmail.com> On Wed, Oct 29, 2008 at 2:37 PM, Kip Macy wrote: > The code in 7.0 is actually locked quite differently. Could you please > try and reproduce on 7.0 and RELENG_7? > > ok. I'll keep you posted. Jerry From kmacy at freebsd.org Wed Oct 29 22:07:05 2008 From: kmacy at freebsd.org (Kip Macy) Date: Wed Oct 29 22:07:36 2008 Subject: crash at in_pcb.c In-Reply-To: <86068e730810291345r738242b0lb8130bf6bd011015@mail.gmail.com> References: <86068e730810291345r738242b0lb8130bf6bd011015@mail.gmail.com> Message-ID: <3c1674c90810291437n3f0d5132t52bc2fa4f4e1b9d0@mail.gmail.com> The code in 7.0 is actually locked quite differently. Could you please try and reproduce on 7.0 and RELENG_7? Thanks, Kip On Wed, Oct 29, 2008 at 8:45 PM, Jerry Toung wrote: > Hello List, > I can realiably reproduce this crash. We have a deamon that accept several > connections > per sec. We use iperf and Microsoft Web application stress 1.0 to push > traffic to the FreeBSD box. > Without further delay, the crash dump is below. I've been troubleshooting, > but I am no longer sure > if this is a race condition or a stack corruption. The socket pointer > between frame 12 and 11 is different. > This is on 6.2, but the code for 7.0 is identical, so I think it still > applies. > > Any hint, patching or troubleshooting this is appreciated. > > Unread portion of the kernel message buffer: > > > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0x2aef0210 > fault code = supervisor read, page not present > instruction pointer = 0x20:0xc0769098 > stack pointer = 0x28:0xef781bc0 > frame pointer = 0x28:0xef781bd0 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, def32 1, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 1166 (ndaemon) > trap number = 12 > panic: page fault > cpuid = 0 > Uptime: 8h32m25s > Dumping 3325 MB (3 chunks) > #0 doadump () at pcpu.h:165 > 165 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) l *0xc0769098 > 0xc0769098 is in in_pcblookup_local (/usr/src/sys/netinet/in_pcb.c:923). > 918 /usr/src/sys/netinet/in_pcb.c: No such file or directory. > in /usr/src/sys/netinet/in_pcb.c > (kgdb) bt > #0 doadump () at pcpu.h:165 > #1 0xc06c2812 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:412 > #2 0xc06c2bbd in panic (fmt=0xc0940872 "%s") at > /usr/src/sys/kern/kern_shutdown.c:573 > #3 0xc08f3e4e in trap_fatal (frame=0xef781b80, eva=720306704) at > /usr/src/sys/i386/i386/trap.c:838 > #4 0xc08f3b57 in trap_pfault (frame=0xef781b80, usermode=0, eva=720306704) > at /usr/src/sys/i386/i386/trap.c:745 > #5 0xc08f3745 in trap (frame= > {tf_fs = -277348344, tf_es = 40, tf_ds = -913309656, tf_edi = 6, > tf_esi = 0, tf_ebp = -277341232, tf_isp = -277341268, tf_ebx = -1062683820, > tf_edx = 720306704, tf_ecx = 14063, tf_eax = 720306704, tf_trapno = 12, > tf_err = 0, tf_eip = -1065971560, tf_cs = 32, tf_eflags = 66050, tf_esp = 0, > tf_ss = -1062683820}) at /usr/src/sys/i386/i386/trap.c:435 > #6 0xc08dddba in calltrap () at /usr/src/sys/i386/i386/exception.s:139 > #7 0xc0769098 in in_pcblookup_local (pcbinfo=0x2aef0210, laddr={s_addr = > 0}, lport_arg=720306704, wild_okay=1) > at /usr/src/sys/netinet/in_pcb.c:923 > #8 0xc0768452 in in_pcbbind_setup (inp=0xc97150b4, nam=0x36ef, > laddrp=0xc97150ec, lportp=0xc97150ce, cred=0xc8726780) > at /usr/src/sys/netinet/in_pcb.c:464 > #9 0xc0767f56 in in_pcbbind (inp=0xc97150b4, nam=0x2aef0210, > cred=0xc8726780) at /usr/src/sys/netinet/in_pcb.c:240 > #10 0xc077f272 in tcp_connect (tp=0xc9897000, nam=0xc98a1ba0, td=0xc990e180) > at /usr/src/sys/netinet/tcp_usrreq.c:864 > #11 0xc077e141 in tcp_usr_connect (so=0xc9897000, nam=0xc98a1ba0, > td=0xc990e180) > at /usr/src/sys/netinet/tcp_usrreq.c:369 > #12 0xc06fec4e in soconnect (so=0xc97b39bc, nam=0xc98a1ba0, td=0xc990e180) > at /usr/src/sys/kern/uipc_socket.c:558 > #13 0xc07046a8 in kern_connect (td=0xc990e180, fd=89, sa=0xc98a1ba0) at > /usr/src/sys/kern/uipc_syscalls.c:536 > #14 0xc070460f in connect (td=0xc990e180, uap=0xef781d04) at > /usr/src/sys/kern/uipc_syscalls.c:505 > #15 0xc08f4193 in syscall (frame= > {tf_fs = 135725115, tf_es = 59, tf_ds = -1088487365, tf_edi = > 135745024, tf_esi = -1089511444, tf_ebp = -1089514536, tf_isp = -277340828, > tf_ebx = 671753396, tf_edx = 0, tf_ecx = 135524256, tf_eax = 98, tf_trapno = > 0, tf_err = 2, tf_eip = 674451435, tf_cs = 51, tf_eflags = 642, tf_esp = > -1089514580, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:984 > #16 0xc08dde0f in Xint0x80_syscall () at > /usr/src/sys/i386/i386/exception.s:200 > #17 0x00000033 in ?? () > Previous frame inner to this frame (corrupt stack?) > (kgdb) f 7 > #7 0xc0769098 in in_pcblookup_local (pcbinfo=0x2aef0210, laddr={s_addr = > 0}, lport_arg=720306704, wild_okay=1) > at /usr/src/sys/netinet/in_pcb.c:923 > 923 in /usr/src/sys/netinet/in_pcb.c > (kgdb) i loc > phd = (struct inpcbport *) 0x2aef0210 > tmphd = (struct inpcbport *) 0x2aef0210 > match = (struct inpcb *) 0x0 > inp = (struct inpcb *) 0x2aef0210 > tmpinp = (struct inpcb *) 0x2aef0210 > matchwild = 6 > wildcard = -1062683820 > lport = 14063 > (kgdb) p phd > $1 = (struct inpcbport *) 0x2aef0210 > (kgdb) p phd->phd_port > Cannot access memory at address 0x2aef021c > > (kgdb) f 12 > #12 0xc06fec4e in soconnect (so=0xc97b39bc, nam=0xc98a1ba0, td=0xc990e180) > at /usr/src/sys/kern/uipc_socket.c:558 > 558 /usr/src/sys/kern/uipc_socket.c: No such file or directory. > in /usr/src/sys/kern/uipc_socket.c > (kgdb) p so > $2 = (struct socket *) 0xc97b39bc > (kgdb) p nam > $3 = (struct sockaddr *) 0xc98a1ba0 > (kgdb) p td > $4 = (struct thread *) 0xc990e180 > (kgdb) l > 553 in /usr/src/sys/kern/uipc_socket.c > (kgdb) f 11 > #11 0xc077e141 in tcp_usr_connect (so=0xc9897000, nam=0xc98a1ba0, > td=0xc990e180) > at /usr/src/sys/netinet/tcp_usrreq.c:369 > 369 /usr/src/sys/netinet/tcp_usrreq.c: No such file or directory. > in /usr/src/sys/netinet/tcp_usrreq.c > (kgdb) > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From stevefranks at ieee.org Wed Oct 29 22:30:49 2008 From: stevefranks at ieee.org (Steve Franks) Date: Wed Oct 29 22:30:57 2008 Subject: neophyte: tcsetattr() gives 22 error in i386, not in amd64? In-Reply-To: <20081025081305.GA55683@icarus.home.lan> References: <539c60b90810241534l6bedc5e3s1c2e3162c2a7ff38@mail.gmail.com> <4902CC86.8030408@bokey.mine.nu> <20081025075648.GB55339@icarus.home.lan> <4902D1FA.3070003@bokey.mine.nu> <20081025081305.GA55683@icarus.home.lan> Message-ID: <539c60b90810291530v54712qe801c094ebcdecd1@mail.gmail.com> >> >>>> Hi, >> >>>> >> >>>> I'm getting a 22 errno from tcsetattr() on 7-STABLE i386 in code which >> >>>> was working under 7-STABLE amd64. Serial device is a ucom (silabs >> >>>> cp2103). Permissions on /dev/cuaU0 look fine. Cutecom/Minicom >> >>>> appears to open the port without error... >> >>> I don't see anything obviously wrong, but I'd bet a bug related to >> >>> 32/64-bit types. Can you post a complete piece of code that can be >> >>> compiled and run and demonstrates the problem? Also, try compiling with >> >>> -Wall -W and investigate any warnings that are produced. >> >>> >> >>> By the way, errno 22 is EINVAL, "Invalid argument". perror() is your >> >>> friend. >> >> Strange freebsd doesnt document error numbers. On POSIX, errno 22 is >> >> EINVAL as well (documented in errno(3)). Is this applicable to freebsd? >> > >> > /usr/include/errno.h isn't documentation of error numbers? >> > If you're wanting to track down how/why tcsetattr(3) results in EINVAL, > using truss or ktrace might come in handy. Otherwise, you literally > will have to throw some debugging code into the ucom(4) driver to > try and figure out what function is kicking out code 22. Wow! truss is quite handy. I've located the problem, and am posting it for posterity: Someone was memset()'ing the termios struct to zero's, then setting the baudrate (setcfspeed) and a couple other things. Apparently this was not a canonical set of required members of the struct, because adding a tcgetattr(f, termio) right after the memset apparently pre-populated the thing correctly and now it works fine... Thanks for the leg up, Jeremy. Best, Steve From pluknet at gmail.com Wed Oct 29 22:56:44 2008 From: pluknet at gmail.com (pluknet) Date: Wed Oct 29 22:56:51 2008 Subject: crash at in_pcb.c In-Reply-To: <86068e730810291345r738242b0lb8130bf6bd011015@mail.gmail.com> References: <86068e730810291345r738242b0lb8130bf6bd011015@mail.gmail.com> Message-ID: 2008/10/29 Jerry Toung : > Hello List, > I can realiably reproduce this crash. We have a deamon that accept several > connections > per sec. We use iperf and Microsoft Web application stress 1.0 to push > traffic to the FreeBSD box. > Without further delay, the crash dump is below. I've been troubleshooting, > but I am no longer sure > if this is a race condition or a stack corruption. The socket pointer > between frame 12 and 11 is different. > This is on 6.2, but the code for 7.0 is identical, so I think it still > applies. > > Any hint, patching or troubleshooting this is appreciated. > > Unread portion of the kernel message buffer: > > > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0x2aef0210 > fault code = supervisor read, page not present > instruction pointer = 0x20:0xc0769098 > stack pointer = 0x28:0xef781bc0 > frame pointer = 0x28:0xef781bd0 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, def32 1, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 1166 (ndaemon) > trap number = 12 > panic: page fault > cpuid = 0 > Uptime: 8h32m25s > Dumping 3325 MB (3 chunks) > #0 doadump () at pcpu.h:165 > 165 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) l *0xc0769098 > 0xc0769098 is in in_pcblookup_local (/usr/src/sys/netinet/in_pcb.c:923). > 918 /usr/src/sys/netinet/in_pcb.c: No such file or directory. > in /usr/src/sys/netinet/in_pcb.c > (kgdb) bt > #0 doadump () at pcpu.h:165 > #1 0xc06c2812 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:412 > #2 0xc06c2bbd in panic (fmt=0xc0940872 "%s") at > /usr/src/sys/kern/kern_shutdown.c:573 > #3 0xc08f3e4e in trap_fatal (frame=0xef781b80, eva=720306704) at > /usr/src/sys/i386/i386/trap.c:838 > #4 0xc08f3b57 in trap_pfault (frame=0xef781b80, usermode=0, eva=720306704) > at /usr/src/sys/i386/i386/trap.c:745 > #5 0xc08f3745 in trap (frame= > {tf_fs = -277348344, tf_es = 40, tf_ds = -913309656, tf_edi = 6, > tf_esi = 0, tf_ebp = -277341232, tf_isp = -277341268, tf_ebx = -1062683820, > tf_edx = 720306704, tf_ecx = 14063, tf_eax = 720306704, tf_trapno = 12, > tf_err = 0, tf_eip = -1065971560, tf_cs = 32, tf_eflags = 66050, tf_esp = 0, > tf_ss = -1062683820}) at /usr/src/sys/i386/i386/trap.c:435 > #6 0xc08dddba in calltrap () at /usr/src/sys/i386/i386/exception.s:139 > #7 0xc0769098 in in_pcblookup_local (pcbinfo=0x2aef0210, laddr={s_addr = > 0}, lport_arg=720306704, wild_okay=1) > at /usr/src/sys/netinet/in_pcb.c:923 > #8 0xc0768452 in in_pcbbind_setup (inp=0xc97150b4, nam=0x36ef, > laddrp=0xc97150ec, lportp=0xc97150ce, cred=0xc8726780) > at /usr/src/sys/netinet/in_pcb.c:464 > #9 0xc0767f56 in in_pcbbind (inp=0xc97150b4, nam=0x2aef0210, > cred=0xc8726780) at /usr/src/sys/netinet/in_pcb.c:240 > #10 0xc077f272 in tcp_connect (tp=0xc9897000, nam=0xc98a1ba0, td=0xc990e180) > at /usr/src/sys/netinet/tcp_usrreq.c:864 > #11 0xc077e141 in tcp_usr_connect (so=0xc9897000, nam=0xc98a1ba0, > td=0xc990e180) > at /usr/src/sys/netinet/tcp_usrreq.c:369 > #12 0xc06fec4e in soconnect (so=0xc97b39bc, nam=0xc98a1ba0, td=0xc990e180) > at /usr/src/sys/kern/uipc_socket.c:558 > #13 0xc07046a8 in kern_connect (td=0xc990e180, fd=89, sa=0xc98a1ba0) at > /usr/src/sys/kern/uipc_syscalls.c:536 > #14 0xc070460f in connect (td=0xc990e180, uap=0xef781d04) at > /usr/src/sys/kern/uipc_syscalls.c:505 > #15 0xc08f4193 in syscall (frame= > {tf_fs = 135725115, tf_es = 59, tf_ds = -1088487365, tf_edi = > 135745024, tf_esi = -1089511444, tf_ebp = -1089514536, tf_isp = -277340828, > tf_ebx = 671753396, tf_edx = 0, tf_ecx = 135524256, tf_eax = 98, tf_trapno = > 0, tf_err = 2, tf_eip = 674451435, tf_cs = 51, tf_eflags = 642, tf_esp = > -1089514580, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:984 > #16 0xc08dde0f in Xint0x80_syscall () at > /usr/src/sys/i386/i386/exception.s:200 > #17 0x00000033 in ?? () > Previous frame inner to this frame (corrupt stack?) > (kgdb) f 7 > #7 0xc0769098 in in_pcblookup_local (pcbinfo=0x2aef0210, laddr={s_addr = > 0}, lport_arg=720306704, wild_okay=1) > at /usr/src/sys/netinet/in_pcb.c:923 > 923 in /usr/src/sys/netinet/in_pcb.c > (kgdb) i loc > phd = (struct inpcbport *) 0x2aef0210 > tmphd = (struct inpcbport *) 0x2aef0210 > match = (struct inpcb *) 0x0 > inp = (struct inpcb *) 0x2aef0210 > tmpinp = (struct inpcb *) 0x2aef0210 > matchwild = 6 > wildcard = -1062683820 > lport = 14063 > (kgdb) p phd > $1 = (struct inpcbport *) 0x2aef0210 > (kgdb) p phd->phd_port > Cannot access memory at address 0x2aef021c > > (kgdb) f 12 > #12 0xc06fec4e in soconnect (so=0xc97b39bc, nam=0xc98a1ba0, td=0xc990e180) > at /usr/src/sys/kern/uipc_socket.c:558 > 558 /usr/src/sys/kern/uipc_socket.c: No such file or directory. > in /usr/src/sys/kern/uipc_socket.c > (kgdb) p so > $2 = (struct socket *) 0xc97b39bc > (kgdb) p nam > $3 = (struct sockaddr *) 0xc98a1ba0 > (kgdb) p td > $4 = (struct thread *) 0xc990e180 > (kgdb) l > 553 in /usr/src/sys/kern/uipc_socket.c > (kgdb) f 11 > #11 0xc077e141 in tcp_usr_connect (so=0xc9897000, nam=0xc98a1ba0, > td=0xc990e180) > at /usr/src/sys/netinet/tcp_usrreq.c:369 > 369 /usr/src/sys/netinet/tcp_usrreq.c: No such file or directory. > in /usr/src/sys/netinet/tcp_usrreq.c > (kgdb) Could you please get the following from kgdb? f 7 p *inp p *inp->inp_laddr P.S. It's definitely 7.0 backtrace (or close to).. 6.2 has different line numbers. -- wbr, pluknet From des at des.no Thu Oct 30 11:21:18 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Thu Oct 30 11:21:24 2008 Subject: Usage of "files" for config In-Reply-To: (Srinivas's message of "Tue, 28 Oct 2008 03:52:35 +0530") References: Message-ID: <8663naib10.fsf@ds4.des.no> Srinivas writes: > I would like to know the usage of files and files.[arch] in sys/conf. "files" and "options" are used to determine which source files should be included and which preprocessor macros should be defined according to the "device" and "option" lines in the kernel config. However, as a "beginner of freebsd kernel", I would recommend that you spend your time reading (and writing or modifying) source code, not trying to understand the Makefiles. DES -- Dag-Erling Sm?rgrav - des@des.no From guru at unixarea.de Thu Oct 30 12:32:31 2008 From: guru at unixarea.de (Matthias Apitz) Date: Thu Oct 30 12:32:38 2008 Subject: 7.0R && amd does not unmount USB-based UFS Message-ID: <20081030122023.GA25223@rebelion.Sisis.de> Hello, I'm using the amd to mount CDFS and MSDOSFS on USB without any kind of problems for years; now I formated a new USB key with UFS, added the config in amd's map and encounter that it does not unmount the file system after the configured time of 20 secs: any idea? thanks in advance; matthias details: # cat /etc/amdmaps/amd.ufs ufs type:=ufs;fs:=/mnt/ufs;dev:=/dev/da0s1a;opts:=rw # tail -f /var/log/amd Oct 30 12:06:04 rebelion amd[23331]/info: /f: disabling nfs congestion window Oct 30 12:06:04 rebelion amd[23328]/info: initializing amd.conf map /etc/amdmaps/amd.usb of type file Oct 30 12:06:04 rebelion amd[23328]/info: first time load of map /etc/amdmaps/amd.usb succeeded Oct 30 12:06:04 rebelion amd[23328]/info: /etc/amdmaps/amd.usb mounted fstype toplvl on /u Oct 30 12:06:04 rebelion amd[23328]/info: initializing amd.conf map /etc/amdmaps/amd.cdrom of type file Oct 30 12:06:04 rebelion amd[23328]/info: first time load of map /etc/amdmaps/amd.cdrom succeeded Oct 30 12:06:04 rebelion amd[23328]/info: /etc/amdmaps/amd.cdrom mounted fstype toplvl on /a Oct 30 12:06:04 rebelion amd[23328]/info: initializing amd.conf map /etc/amdmaps/amd.ufs of type file Oct 30 12:06:04 rebelion amd[23328]/info: first time load of map /etc/amdmaps/amd.ufs succeeded Oct 30 12:06:04 rebelion amd[23328]/info: /etc/amdmaps/amd.ufs mounted fstype toplvl on /f entering the dir /f/ufs mounts the FS: Oct 30 12:08:02 rebelion amd[23328]/map: Trying mount of /dev/da0s1a on /f/usf fstype ufs Oct 30 12:08:02 rebelion amd[23328]/info: /dev/da0s1a mounted fstype ufs on /mnt/ufs here another example of entering /a/cdrom which gets unmounted after 20 secs: Oct 30 12:54:49 rebelion amd[23328]/map: Trying mount of /dev/acd0 on /a/cdrom fstype cdfs Oct 30 12:54:50 rebelion amd[23328]/error: /cdrom: mount: Input/output error Oct 30 12:54:50 rebelion amd[23328]/error: mount_cdfs: Input/output error Oct 30 12:54:59 rebelion amd[23328]/map: Trying mount of /dev/acd0 on /a/cdrom fstype cdfs Oct 30 12:55:00 rebelion amd[23328]/info: /dev/acd0 mounted fstype cdfs on /cdrom Oct 30 12:55:20 rebelion amd[23328]/info: /dev/acd0 unmounted fstype cdfs from /cdrom the UFS stays mounted forever: $ mount /dev/ad4s1a on / (ufs, local) devfs on /dev (devfs, local) /dev/ad4s1e on /tmp (ufs, local, soft-updates) /dev/ad4s1f on /usr (ufs, local, soft-updates) /dev/ad4s1d on /var (ufs, local, soft-updates) pid23328@rebelion:/u on /u (nfs) pid23328@rebelion:/a on /a (nfs) pid23328@rebelion:/f on /f (nfs) /dev/da0s1a on /mnt/ufs (ufs, local) a umount by hand helps: # umount /mnt/ufs # mount /dev/ad4s1a on / (ufs, local) devfs on /dev (devfs, local) /dev/ad4s1e on /tmp (ufs, local, soft-updates) /dev/ad4s1f on /usr (ufs, local, soft-updates) /dev/ad4s1d on /var (ufs, local, soft-updates) pid23328@rebelion:/u on /u (nfs) pid23328@rebelion:/a on /a (nfs) pid23328@rebelion:/f on /f (nfs) -- Matthias Apitz Manager Technical Support - OCLC GmbH Gruenwalder Weg 28g - 82041 Oberhaching - Germany t +49-89-61308 351 - f +49-89-61308 399 - m +49-170-4527211 e - w http://www.oclc.org/ http://www.UnixArea.de/ b http://gurucubano.blogspot.com/ A computer is like an air conditioner, it stops working when you open Windows Una computadora es como aire acondicionado, deja de funcionar si abres Windows From aniketpansare at gmail.com Thu Oct 30 13:22:23 2008 From: aniketpansare at gmail.com (aniket pansare) Date: Thu Oct 30 13:22:30 2008 Subject: process hibernation and process descriptor table Message-ID: <35a750510810300602w6ed15a83m6780284601deed7b@mail.gmail.com> hi guys i am doing a project on process hibernation. i am new to linux and i want u to tell me how can i print the contents of a process descriptor table. i had a look at the softwares like cryopid and BLCR but i am not able to get it at this stage. Any suggestions about how i should go about the project. Aniket Pansare From des at des.no Thu Oct 30 08:38:21 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Thu Oct 30 08:38:28 2008 Subject: process hibernation and process descriptor table In-Reply-To: <35a750510810300602w6ed15a83m6780284601deed7b@mail.gmail.com> (aniket pansare's message of "Thu, 30 Oct 2008 18:32:12 +0530") References: <35a750510810300602w6ed15a83m6780284601deed7b@mail.gmail.com> Message-ID: <86skqegkk4.fsf@ds4.des.no> "aniket pansare" writes: > i am new to linux and i want u to tell me how can i print the contents of a > process descriptor table. You should probably ask some Linux people. DES -- Dag-Erling Sm?rgrav - des@des.no From koitsu at FreeBSD.org Thu Oct 30 08:47:14 2008 From: koitsu at FreeBSD.org (Jeremy Chadwick) Date: Thu Oct 30 08:47:20 2008 Subject: open(2) and O_NOATIME Message-ID: <20081030154711.GA8416@icarus.home.lan> I've recently been reading about Linux's O_NOATIME flag to open(2), and I'm curious why we haven't implemented this. There seem to be a lot of good reasons to implement such a thing. Chances are it's due to lack of time/interest, which is expected, but I was wondering if there were other reasons. I realise mount's noatime trumps this, but there are lots of scenarios where atime is desired as a default, but disabled in specific cases. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From jrytoung at gmail.com Thu Oct 30 08:47:20 2008 From: jrytoung at gmail.com (Jerry Toung) Date: Thu Oct 30 08:47:29 2008 Subject: crash at in_pcb.c In-Reply-To: References: <86068e730810291345r738242b0lb8130bf6bd011015@mail.gmail.com> Message-ID: <86068e730810300847j64cafdeqab8b3b6a5f59b949@mail.gmail.com> On Wed, Oct 29, 2008 at 3:32 PM, pluknet wrote: > > Could you please get the following from kgdb? > f 7 > p *inp > p *inp->inp_laddr > > P.S. It's definitely 7.0 backtrace (or close to).. 6.2 has different > line numbers. > kgdb) f 7 #7 0xc0769098 in in_pcblookup_local (pcbinfo=0x2aef0210, laddr={s_addr = 0}, lport_arg=720306704, wild_okay=1) at /usr/src/sys/netinet/in_pcb.c:923 923 /usr/src/sys/netinet/in_pcb.c: No such file or directory. in /usr/src/sys/netinet/in_pcb.c (kgdb) p *inp Cannot access memory at address 0x2aef0210 (kgdb) p *inp->inp_laddr There is no member named inp_laddr. (kgdb) From toyj at union.edu Thu Oct 30 08:52:20 2008 From: toyj at union.edu (jT) Date: Thu Oct 30 08:52:28 2008 Subject: process hibernation and process descriptor table In-Reply-To: <35a750510810300602w6ed15a83m6780284601deed7b@mail.gmail.com> References: <35a750510810300602w6ed15a83m6780284601deed7b@mail.gmail.com> Message-ID: <9f8af95f0810300825g5b0fae82u85002e56434b3edd@mail.gmail.com> Hello, This list is unrelated to linux, it was designed for technical discussion concerning the FreeBSD operating system: www.freebsd.org. If you are looking for linux help I recommend you check kernel.org for documentation and online forums. On 10/30/08, aniket pansare wrote: > hi guys > i am doing a project on process hibernation. > i am new to linux and i want u to tell me how can i print the contents of a > process descriptor table. > > i had a look at the softwares like cryopid and BLCR but i am not able to get > it at this stage. > > Any suggestions about how i should go about the project. > > > > Aniket Pansare > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > -- /jT http://git.zen-sources.org/?p=kernel/zenmm.git;a=summary From avg at icyb.net.ua Thu Oct 30 09:33:42 2008 From: avg at icyb.net.ua (Andriy Gapon) Date: Thu Oct 30 09:33:50 2008 Subject: memtest86+ can not link: binutils issue? Message-ID: <4909DC03.1080901@icyb.net.ua> 0. FreeBSD 7.1-PRERELEASE r184195 i386 $ ld -v GNU ld version 2.15 [FreeBSD] 2004-05-23 1. obtain and extract http://www.memtest.org/download/2.01/memtest86+-2.01.bin.gz 2. run gmake: $ gmake gcc -E -traditional head.S -o head.s as -32 -o head.o head.s gcc -c -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin -ffreestanding -fPIC -fno-strict-aliasing reloc.c gcc -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin -ffreestanding -fPIC -c -o main.o main.c gcc -c -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin -ffreestanding test.c gcc -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin -ffreestanding -fPIC -c -o init.o init.c gcc -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin -ffreestanding -fPIC -c -o lib.o lib.c gcc -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin -ffreestanding -fPIC -c -o patn.o patn.c gcc -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin -ffreestanding -fPIC -c -o screen_buffer.o screen_buffer.c gcc -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin -ffreestanding -fPIC -c -o config.o config.c gcc -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin -ffreestanding -fPIC -c -o linuxbios.o linuxbios.c gcc -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin -ffreestanding -fPIC -c -o memsize.o memsize.c gcc -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin -ffreestanding -fPIC -c -o pci.o pci.c gcc -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin -ffreestanding -fPIC -c -o controller.o controller.c gcc -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin -ffreestanding -fPIC -c -o random.o random.c gcc -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin -ffreestanding -fPIC -c -o extra.o extra.c gcc -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin -ffreestanding -fPIC -c -o spd.o spd.c gcc -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin -ffreestanding -fPIC -c -o error.o error.c gcc -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin -ffreestanding -fPIC -c -o dmi.o dmi.c ld --warn-constructors --warn-common -static -T memtest_shared.lds \ -o memtest_shared head.o reloc.o main.o test.o init.o lib.o patn.o screen_buffer.o config.o linuxbios.o memsize.o pci.o controller.o random.o extra.o spd.o error.o dmi.o && \ ld -shared -Bsymbolic -T memtest_shared.lds -o memtest_shared head.o reloc.o main.o test.o init.o lib.o patn.o screen_buffer.o config.o linuxbios.o memsize.o pci.o controller.o random.o extra.o spd.o error.o dmi.o head.o(.text+0x7): In function `startup_32': : undefined reference to `_GLOBAL_OFFSET_TABLE_' Segmentation fault (core dumped) gmake: *** [memtest_shared] Error 139 Not only linking fails, but ld even crashes. Things are more complicated than usual because of the custom linker script memtest_shared.lds. The same compiles/links nicely on Fedora 9. $ ld -v GNU ld version 2.18.50.0.6-5.fc9 20080403 Can anybody suggest anything about this problem? If somebody is working on newer version of binuitls for FreeBSD I can help as a tester. -- Andriy Gapon From peterjeremy at optushome.com.au Thu Oct 30 11:46:32 2008 From: peterjeremy at optushome.com.au (Peter Jeremy) Date: Thu Oct 30 11:46:39 2008 Subject: memtest86+ can not link: binutils issue? In-Reply-To: <4909DC03.1080901@icyb.net.ua> References: <4909DC03.1080901@icyb.net.ua> Message-ID: <20081030184625.GA99398@server.vk2pj.dyndns.org> On 2008-Oct-30 18:08:35 +0200, Andriy Gapon wrote: >1. obtain and extract >http://www.memtest.org/download/2.01/memtest86+-2.01.bin.gz This is a compressed bootable image and can't be compiled. Possibly you mean http://www.memtest.org/download/2.01/memtest86+-2.01.tar.gz >2. run gmake: >$ gmake >gcc -E -traditional head.S -o head.s >as -32 -o head.o head.s >gcc -c -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin >-ffreestanding -fPIC -fno-strict-aliasing reloc.c >gcc -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin >-ffreestanding -fPIC -c -o main.o main.c >gcc -c -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin >-ffreestanding test.c Blows up at this point for me: gcc -c -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin -ffreestanding test.c test.c:14:20: error: sys/io.h: No such file or directory test.c: In function 'beep': test.c:1410: warning: implicit declaration of function 'outb_p' test.c:1410: warning: implicit declaration of function 'inb_p' test.c:1417: warning: implicit declaration of function 'outb' gmake: *** [test.o] Error 1 I can't find in CVS or any declarations for outb_p or inb_p in my source tree. >ld --warn-constructors --warn-common -static -T memtest_shared.lds \ > -o memtest_shared head.o reloc.o main.o test.o init.o lib.o >patn.o screen_buffer.o config.o linuxbios.o memsize.o pci.o controller.o >random.o extra.o spd.o error.o dmi.o && \ > ld -shared -Bsymbolic -T memtest_shared.lds -o memtest_shared >head.o reloc.o main.o test.o init.o lib.o patn.o screen_buffer.o >config.o linuxbios.o memsize.o pci.o controller.o random.o extra.o spd.o >error.o dmi.o >head.o(.text+0x7): In function `startup_32': >: undefined reference to `_GLOBAL_OFFSET_TABLE_' >Segmentation fault (core dumped) >gmake: *** [memtest_shared] Error 139 I can't help here. _GLOBAL_OFFSET_TABLE_ is related to the binutils PIC support and it appears that the linker doesn't like the code (in head.S) is explicitly referencing it. >Not only linking fails, but ld even crashes. I agree this shouldn't happen. >Can anybody suggest anything about this problem? It looks like stand-alone PIC code on FreeBSD needs some different incantations to Linux. My understanding is that several of the i386 bootstraps are relocatable so you might like to peruse the code in /usr/src/sys/boot/i386 for ideas. -- Peter Jeremy Please excuse any delays as the result of my ISP's inability to implement an MTA that is either RFC2821-compliant or matches their claimed behaviour. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081030/2903f135/attachment.pgp From delphij at delphij.net Thu Oct 30 19:16:57 2008 From: delphij at delphij.net (Xin LI) Date: Thu Oct 30 19:17:04 2008 Subject: open(2) and O_NOATIME In-Reply-To: <20081030154711.GA8416@icarus.home.lan> References: <20081030154711.GA8416@icarus.home.lan> Message-ID: <490A6A8A.7080504@delphij.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jeremy Chadwick wrote: > I've recently been reading about Linux's O_NOATIME flag to open(2), and > I'm curious why we haven't implemented this. There seem to be a lot of > good reasons to implement such a thing. > > Chances are it's due to lack of time/interest, which is expected, but I > was wondering if there were other reasons. > > I realise mount's noatime trumps this, but there are lots of scenarios > where atime is desired as a default, but disabled in specific cases. Em... Allowing administrators to disable NOATIME would be a good thing, but wouldn't allowing arbitrary program to decide whether atime should be changed, be a serious security disaster? Disclaimer: I'm not a big atime fan myself, actually I disable atime on a lot of my servers for performance reasons :) Cheers, - -- Xin LI http://www.delphij.net/ FreeBSD - The Power to Serve! -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkkKaooACgkQi+vbBBjt66CImQCgj51GGHXFaGhsFk4fAAWhmfV5 +s4An2Hn2TCVhqXEpzEL3xNwxy6YE84M =n7f/ -----END PGP SIGNATURE----- From joerg at britannica.bec.de Thu Oct 30 19:40:37 2008 From: joerg at britannica.bec.de (Joerg Sonnenberger) Date: Thu Oct 30 19:40:44 2008 Subject: open(2) and O_NOATIME In-Reply-To: <490A6A8A.7080504@delphij.net> References: <20081030154711.GA8416@icarus.home.lan> <490A6A8A.7080504@delphij.net> Message-ID: <20081031021823.GA575@britannica.bec.de> On Thu, Oct 30, 2008 at 07:16:42PM -0700, Xin LI wrote: > Em... Allowing administrators to disable NOATIME would be a good thing, > but wouldn't allowing arbitrary program to decide whether atime should > be changed, be a serious security disaster? Think of backup programs. Joerg From koitsu at FreeBSD.org Thu Oct 30 19:47:55 2008 From: koitsu at FreeBSD.org (Jeremy Chadwick) Date: Thu Oct 30 19:48:03 2008 Subject: open(2) and O_NOATIME In-Reply-To: <490A6A8A.7080504@delphij.net> References: <20081030154711.GA8416@icarus.home.lan> <490A6A8A.7080504@delphij.net> Message-ID: <20081031024748.GA20319@icarus.home.lan> On Thu, Oct 30, 2008 at 07:16:42PM -0700, Xin LI wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Jeremy Chadwick wrote: > > I've recently been reading about Linux's O_NOATIME flag to open(2), and > > I'm curious why we haven't implemented this. There seem to be a lot of > > good reasons to implement such a thing. > > > > Chances are it's due to lack of time/interest, which is expected, but I > > was wondering if there were other reasons. > > > > I realise mount's noatime trumps this, but there are lots of scenarios > > where atime is desired as a default, but disabled in specific cases. > > Em... Allowing administrators to disable NOATIME would be a good thing, > but wouldn't allowing arbitrary program to decide whether atime should > be changed, be a serious security disaster? How? There's only one condition I can think of: where a system administrator is, for some reason, relying upon atimes as a form of proof of something bad happening (which is a horrible concept in general, being as the amount of false positives seen would be tremendous; using atime as a security auditing method is stupid). If that's what you were referring to, then possibly making O_NOATIME only to root would be a suitable compromise. > Disclaimer: I'm not a big atime fan myself, actually I disable atime on > a lot of my servers for performance reasons :) I can't disable atime on any systems I maintain, because they all provide access to classic UNIX mbox spools where atime is used to determine if new mail has arrived. The instant filesystem-level backups run, atime is lost, and users have no way of knowing if they have new mail or not. Switching to Maildir is an option, but the performance hit of readdir() + stat() on thousands of files is tremendous (which is why mail clients like mutt have features like "header caching" via Oracle/Sleepycat DB). Anyway, I just was reading about it and realise that a lot of backup solutions out there can make use of O_NOATIME if available, which it isn't on FreeBSD. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From cayson.z at gmail.com Fri Oct 31 02:54:10 2008 From: cayson.z at gmail.com (Boern) Date: Fri Oct 31 02:54:19 2008 Subject: Is there virtualbox ports? Message-ID: Hi,all: I am ready to intall sun xVM VirtualBox on my FreeBSD7.0,but have no found in the ports,anybody can help me? -- Boern Parx From unixmania at gmail.com Fri Oct 31 03:54:43 2008 From: unixmania at gmail.com (Carlos A. M. dos Santos) Date: Fri Oct 31 03:54:51 2008 Subject: Is there virtualbox ports? In-Reply-To: References: Message-ID: On Fri, Oct 31, 2008 at 7:23 AM, Boern wrote: > Hi,all: > I am ready to intall sun xVM VirtualBox on my FreeBSD7.0,but have no > found in the ports,anybody can help me? There is no port. VirtualBox depends on a kernel module that was not ported to FreeBSD yet, among other things. There was some discussion on the -virtualization mailing list. Take a look at the "VirtualBox looks for FreeBSD developer" thread in http://lists.freebsd.org/pipermail/freebsd-virtualization/2008-September/thread.html http://lists.freebsd.org/pipermail/freebsd-virtualization/2008-October/thread.html -- cd /usr/ports/sysutils/life make clean From mozolevsky at gmail.com Fri Oct 31 01:04:49 2008 From: mozolevsky at gmail.com (Igor Mozolevsky) Date: Fri Oct 31 04:23:51 2008 Subject: open(2) and O_NOATIME In-Reply-To: <20081031024748.GA20319@icarus.home.lan> References: <20081030154711.GA8416@icarus.home.lan> <490A6A8A.7080504@delphij.net> <20081031024748.GA20319@icarus.home.lan> Message-ID: 2008/10/31 Jeremy Chadwick : > ... If that's what you were referring to, then possibly making O_NOATIME > only to root would be a suitable compromise. And no systems are compromised with rootkits?.. Igor :-) From lars.engels at 0x20.net Fri Oct 31 03:20:27 2008 From: lars.engels at 0x20.net (Lars Engels) Date: Fri Oct 31 04:24:40 2008 Subject: Is there virtualbox ports? In-Reply-To: References: Message-ID: <20081031112025.c2hg8pb1usg0ocg4@0x20.net> Quoting Boern : > Hi,all: > I am ready to intall sun xVM VirtualBox on my FreeBSD7.0,but have no > found in the ports,anybody can help me? > VirtualBox doesn't work on FreeBSD... -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: Digitale PGP-Unterschrift Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081031/f37f0b30/attachment.pgp From avg at icyb.net.ua Fri Oct 31 06:18:59 2008 From: avg at icyb.net.ua (Andriy Gapon) Date: Fri Oct 31 06:19:06 2008 Subject: memtest86+ can not link: binutils issue? In-Reply-To: <20081030184625.GA99398@server.vk2pj.dyndns.org> References: <4909DC03.1080901@icyb.net.ua> <20081030184625.GA99398@server.vk2pj.dyndns.org> Message-ID: <490B05BA.9090306@icyb.net.ua> on 30/10/2008 20:46 Peter Jeremy said the following: > On 2008-Oct-30 18:08:35 +0200, Andriy Gapon wrote: >> 1. obtain and extract >> http://www.memtest.org/download/2.01/memtest86+-2.01.bin.gz > > This is a compressed bootable image and can't be compiled. Possibly > you mean http://www.memtest.org/download/2.01/memtest86+-2.01.tar.gz Sorry - yes, this was it. >> 2. run gmake: >> $ gmake >> gcc -E -traditional head.S -o head.s >> as -32 -o head.o head.s >> gcc -c -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin >> -ffreestanding -fPIC -fno-strict-aliasing reloc.c >> gcc -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin >> -ffreestanding -fPIC -c -o main.o main.c >> gcc -c -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin >> -ffreestanding test.c > > Blows up at this point for me: > gcc -c -Wall -march=i486 -m32 -Os -fomit-frame-pointer -fno-builtin -ffreestanding test.c > test.c:14:20: error: sys/io.h: No such file or directory > test.c: In function 'beep': > test.c:1410: warning: implicit declaration of function 'outb_p' > test.c:1410: warning: implicit declaration of function 'inb_p' > test.c:1417: warning: implicit declaration of function 'outb' > gmake: *** [test.o] Error 1 > > I can't find in CVS or any declarations for outb_p or inb_p > in my source tree. Sorry again - I patched this file to remove inclusion of this linux-specific file and instead include machine/cpufunc.h, also I changed outb_p => outb, inb_p => inb and swapped parameters of outb-s. These are typical linuxisms. >> ld --warn-constructors --warn-common -static -T memtest_shared.lds \ >> -o memtest_shared head.o reloc.o main.o test.o init.o lib.o >> patn.o screen_buffer.o config.o linuxbios.o memsize.o pci.o controller.o >> random.o extra.o spd.o error.o dmi.o && \ >> ld -shared -Bsymbolic -T memtest_shared.lds -o memtest_shared >> head.o reloc.o main.o test.o init.o lib.o patn.o screen_buffer.o >> config.o linuxbios.o memsize.o pci.o controller.o random.o extra.o spd.o >> error.o dmi.o >> head.o(.text+0x7): In function `startup_32': >> : undefined reference to `_GLOBAL_OFFSET_TABLE_' >> Segmentation fault (core dumped) >> gmake: *** [memtest_shared] Error 139 > > I can't help here. _GLOBAL_OFFSET_TABLE_ is related to the binutils > PIC support and it appears that the linker doesn't like the code (in > head.S) is explicitly referencing it. > >> Not only linking fails, but ld even crashes. > > I agree this shouldn't happen. > >> Can anybody suggest anything about this problem? > > It looks like stand-alone PIC code on FreeBSD needs some different > incantations to Linux. My understanding is that several of the > i386 bootstraps are relocatable so you might like to peruse the > code in /usr/src/sys/boot/i386 for ideas. I wonder if this is something about out port of binutils or is it an issue in older version of binutils. I'll try to look at the boot code, thank you for the hint. -- Andriy Gapon From fb-hackers at psconsult.nl Fri Oct 31 06:48:50 2008 From: fb-hackers at psconsult.nl (Paul Schenkeveld) Date: Fri Oct 31 06:48:59 2008 Subject: open(2) and O_NOATIME In-Reply-To: References: <20081030154711.GA8416@icarus.home.lan> <490A6A8A.7080504@delphij.net> <20081031024748.GA20319@icarus.home.lan> Message-ID: <20081031134842.GA15218@psconsult.nl> On Fri, Oct 31, 2008 at 08:04:48AM +0000, Igor Mozolevsky wrote: > 2008/10/31 Jeremy Chadwick : > > > ... If that's what you were referring to, then possibly making O_NOATIME > > only to root would be a suitable compromise. > > And no systems are compromised with rootkits?.. utimes(2) allows non-root users to (re)set atime provided they own the file or have write permission. Having O_NOATIME follow the same rules would not break any assumed security any further than utimes(2) already does but greatfully benefit all kind of backup programs. So I'd be more than happy to see O_NOATIME be implemented as I'm currently experimenting with backups to detachable harddisks using rsync and not having a way to reset atime is my one big reason for not deploying this kind of backups with more servers. If you wonder why I'm using rsyng instead of dump or tar, here are two reasons: first the detachable disks are much slower than the systems disks so rsync saves a lot of time and secondly a file-by-file-only-if-changed scheme allows me to efficiently use snapshots on the backup medium. Patching rsync to implement the kind of reset atime as i.e. cpio does looks far more complex than adding O_NOATIME to rsync. My $0.02 Regards, Paul Schenkeveld From jilles at stack.nl Fri Oct 31 08:51:20 2008 From: jilles at stack.nl (Jilles Tjoelker) Date: Fri Oct 31 08:51:27 2008 Subject: open(2) and O_NOATIME In-Reply-To: <20081031134842.GA15218@psconsult.nl> References: <20081030154711.GA8416@icarus.home.lan> <490A6A8A.7080504@delphij.net> <20081031024748.GA20319@icarus.home.lan> <20081031134842.GA15218@psconsult.nl> Message-ID: <20081031155117.GA55445@stack.nl> On Fri, Oct 31, 2008 at 02:48:42PM +0100, Paul Schenkeveld wrote: > utimes(2) allows non-root users to (re)set atime provided they own the > file or have write permission. Having O_NOATIME follow the same rules > would not break any assumed security any further than utimes(2) already > does but greatfully benefit all kind of backup programs. This is not entirely correct. utimes(2) with NULL timestamps (reset atime and mtime to current time) is allowed to root, owner or with write permission, but utimes(2) with given timestamps is only allowed to root and owner. O_NOATIME seems equivalent to the latter, and in fact this is the case in Linux (if someone else than root or the owner tries to open a file with O_NOATIME, they get EPERM). There's only a small detail missing: any utimes(2) call updates the ctime, so you can see "something" happened to the file. Linux's O_NOATIME does not update any times at all (this speeds up things). Anyway, O_NOATIME (only for root/owner) seems a useful feature. -- Jilles Tjoelker From thierry.herbelot at free.fr Fri Oct 31 10:14:48 2008 From: thierry.herbelot at free.fr (Thierry Herbelot) Date: Fri Oct 31 10:27:53 2008 Subject: strange behaviour with /sbin/init and serial console Message-ID: <200810311746.23743.thierry.herbelot@free.fr> Hello, with the following patch on /sbin/init, I have two different behaviours depending on the console type (on a i386/32 PC) : - on a video console, I see the expected two messages, - on a serial console, the messages are not displayed (init silently finishes its job and gets to start /etc/rc and everything) I assume that the writev system call is implemented in src/sys/kern/tty_cons.c::cnwrite(), but I could not parse the code to find an explanation. any taker ? TfH PS : this is initially for a RELENG_6 machine, but the code is quite similar under RELENG_7 or Current --- usr/src/sbin/init/init.c.ori 2008-10-31 14:20:48.294794898 +0100 +++ usr/src/sbin/init/init.c 2008-10-31 14:12:16.168062031 +0100 @@ -44,6 +44,8 @@ "$FreeBSD: src/sbin/init/init.c,v 1.60.2.2 2006/07/08 15:34:27 kib Exp $"; #endif /* not lint */ +#include + #include #include #include @@ -239,6 +241,23 @@ */ openlog("init", LOG_CONS|LOG_ODELAY, LOG_AUTH); + warning("warning after openlog"); +{ +int fd; + if ((fd = open("/dev/console", O_WRONLY|O_NONBLOCK, 0)) >= 0) { + struct iovec iov[2]; + struct iovec *v = iov; + + v->iov_base = (void *)"iov direct write test"; + v->iov_len = 21; + ++v; + v->iov_base = (void *)"\r\n"; + v->iov_len = 2; + (void)writev(fd, iov, 2); + (void)close(fd); + } + +} /* * Create an initial session. */ From neldredge at math.ucsd.edu Fri Oct 31 10:32:50 2008 From: neldredge at math.ucsd.edu (Nate Eldredge) Date: Fri Oct 31 10:32:56 2008 Subject: memtest86+ can not link: binutils issue? In-Reply-To: <490B05BA.9090306@icyb.net.ua> References: <4909DC03.1080901@icyb.net.ua> <20081030184625.GA99398@server.vk2pj.dyndns.org> <490B05BA.9090306@icyb.net.ua> Message-ID: On Fri, 31 Oct 2008, Andriy Gapon wrote: > on 30/10/2008 20:46 Peter Jeremy said the following: >> On 2008-Oct-30 18:08:35 +0200, Andriy Gapon wrote: >>> ld --warn-constructors --warn-common -static -T memtest_shared.lds \ >>> -o memtest_shared head.o reloc.o main.o test.o init.o lib.o >>> patn.o screen_buffer.o config.o linuxbios.o memsize.o pci.o controller.o >>> random.o extra.o spd.o error.o dmi.o && \ >>> ld -shared -Bsymbolic -T memtest_shared.lds -o memtest_shared >>> head.o reloc.o main.o test.o init.o lib.o patn.o screen_buffer.o >>> config.o linuxbios.o memsize.o pci.o controller.o random.o extra.o spd.o >>> error.o dmi.o >>> head.o(.text+0x7): In function `startup_32': >>> : undefined reference to `_GLOBAL_OFFSET_TABLE_' >>> Segmentation fault (core dumped) >>> gmake: *** [memtest_shared] Error 139 >> >> I can't help here. _GLOBAL_OFFSET_TABLE_ is related to the binutils >> PIC support and it appears that the linker doesn't like the code (in >> head.S) is explicitly referencing it. >> >>> Not only linking fails, but ld even crashes. >> >> I agree this shouldn't happen. >> >>> Can anybody suggest anything about this problem? >> >> It looks like stand-alone PIC code on FreeBSD needs some different >> incantations to Linux. My understanding is that several of the >> i386 bootstraps are relocatable so you might like to peruse the >> code in /usr/src/sys/boot/i386 for ideas. > > I wonder if this is something about out port of binutils or is it an > issue in older version of binutils. > I'll try to look at the boot code, thank you for the hint. FreeBSD's version of binutils is quite old. I've definitely found bugs in it which are fixed in GNU's current version. So you might try building the official GNU binutils and see if that works any better. I don't know if it will fix your error but maybe it at least won't crash. ld crashing is definitely a bug, and it would be nice if you could file a PR, including the object files. If the GNU version doesn't crash that would be useful information for the PR also, as it might encourage Them to consider importing a newer version. -- Nate Eldredge neldredge@math.ucsd.edu From koitsu at FreeBSD.org Fri Oct 31 10:42:40 2008 From: koitsu at FreeBSD.org (Jeremy Chadwick) Date: Fri Oct 31 10:42:47 2008 Subject: strange behaviour with /sbin/init and serial console In-Reply-To: <200810311746.23743.thierry.herbelot@free.fr> References: <200810311746.23743.thierry.herbelot@free.fr> Message-ID: <20081031173224.GA37636@icarus.home.lan> On Fri, Oct 31, 2008 at 05:46:23PM +0100, Thierry Herbelot wrote: > with the following patch on /sbin/init, I have two different behaviours > depending on the console type (on a i386/32 PC) : > - on a video console, I see the expected two messages, > - on a serial console, the messages are not displayed (init silently finishes > its job and gets to start /etc/rc and everything) I thought this was normal behaviour on FreeBSD, but it's very likely I'm misunderstanding. The charts in Section 27.6.4 describe what "level" of logging is shown where and at what stage, depending upon which boot flags and device settings you use: http://www.freebsd.org/doc/en/books/handbook/serialconsole-setup.html -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From thierry.herbelot at laposte.net Fri Oct 31 10:47:41 2008 From: thierry.herbelot at laposte.net (Thierry Herbelot) Date: Fri Oct 31 10:47:49 2008 Subject: strange behaviour with /sbin/init and serial console In-Reply-To: <20081031173224.GA37636@icarus.home.lan> References: <200810311746.23743.thierry.herbelot@free.fr> <20081031173224.GA37636@icarus.home.lan> Message-ID: <200810311847.31723.thierry.herbelot@laposte.net> Le Friday 31 October 2008, Jeremy Chadwick a ?crit : > On Fri, Oct 31, 2008 at 05:46:23PM +0100, Thierry Herbelot wrote: > > with the following patch on /sbin/init, I have two different behaviours > > depending on the console type (on a i386/32 PC) : > > - on a video console, I see the expected two messages, > > - on a serial console, the messages are not displayed (init silently > > finishes its job and gets to start /etc/rc and everything) > > I thought this was normal behaviour on FreeBSD, but it's very likely I'm > misunderstanding. The charts in Section 27.6.4 describe what "level" of > logging is shown where and at what stage, depending upon which boot > flags and device settings you use: > > http://www.freebsd.org/doc/en/books/handbook/serialconsole-setup.html Hello, I had not taken the time to read this link as thouroughly as should have been. nevertheless, I think the config is right, as the serial console is selected with "-h" in /boot.config (from memory, the machine is at work ...) and all *other* expected messages from the kernel ("dmesg") and the rc scripts are correctly displayed on respectively the serial and video console. what struck me is that, from all the startup messages, just the messages from /sbin/init are displayed only on the video console TfH From delphij at delphij.net Fri Oct 31 10:59:28 2008 From: delphij at delphij.net (Xin LI) Date: Fri Oct 31 10:59:35 2008 Subject: open(2) and O_NOATIME In-Reply-To: <20081031134842.GA15218@psconsult.nl> References: <20081030154711.GA8416@icarus.home.lan> <490A6A8A.7080504@delphij.net> <20081031024748.GA20319@icarus.home.lan> <20081031134842.GA15218@psconsult.nl> Message-ID: <490B4771.9040709@delphij.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Paul Schenkeveld wrote: [...] > utimes(2) allows non-root users to (re)set atime provided they own the > file or have write permission. Having O_NOATIME follow the same rules > would not break any assumed security any further than utimes(2) already > does but greatfully benefit all kind of backup programs. Yes this makes sense I think. Cheers, - -- Xin LI http://www.delphij.net/ FreeBSD - The Power to Serve! -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkkLR3AACgkQi+vbBBjt66BPhACfcZf6JcH0RmTpbpZHVXjdrJTq f7oAoLqQwb2UkFGrDDTy7//Ril2JWmA4 =y1zY -----END PGP SIGNATURE----- From stevefranks at ieee.org Fri Oct 31 11:23:54 2008 From: stevefranks at ieee.org (Steve Franks) Date: Fri Oct 31 11:24:01 2008 Subject: includes, configure, /usr/lib vs. /usr/local/lib, and linux coders Message-ID: <539c60b90810311123w2aa94b8akcd0a5d0fe791885a@mail.gmail.com> I believe someone has told me on this list that the proper way to compile a linux program is to run configure --includedir=/usr/local/include --libdir=/usr/local/lib. Is that correct? I've got a bunch of linux weenies trying to tell me their code isn't broken because I'm supposed to have headers where theirs are. They state that includedir is used by *their* project to locate it's *own* headers, so they never bothered to wire it up in Makefile.in....it gets ignored entirely. I figured I'd better know what I'm talking about before I tell someone they are 'wrong'. Especially as it's usually me ;) Steve From ed at 80386.nl Fri Oct 31 12:06:15 2008 From: ed at 80386.nl (Ed Schouten) Date: Fri Oct 31 12:06:23 2008 Subject: strange behaviour with /sbin/init and serial console In-Reply-To: <200810311746.23743.thierry.herbelot@free.fr> References: <200810311746.23743.thierry.herbelot@free.fr> Message-ID: <20081031190614.GQ1165@hoeg.nl> Hello Theirry, * Thierry Herbelot wrote: > with the following patch on /sbin/init, I have two different behaviours > depending on the console type (on a i386/32 PC) : > - on a video console, I see the expected two messages, > - on a serial console, the messages are not displayed (init silently finishes > its job and gets to start /etc/rc and everything) > > I assume that the writev system call is implemented in > src/sys/kern/tty_cons.c::cnwrite(), but I could not parse the code to find an > explanation. > > any taker ? > > TfH > > PS : this is initially for a RELENG_6 machine, but the code is quite similar > under RELENG_7 or Current Any data written to /dev/console is not multiplexed to all console devices, but only the first active device in the list. The reason behind this, is because it adds a real lot of complexity to the console code, especially related to polling and reading on /dev/console. This weekend I'm going to commit a replacement implementation of /dev/console, which also has this restriction. -- Ed Schouten WWW: http://80386.nl/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081031/d38bde51/attachment.pgp From stevefranks at ieee.org Fri Oct 31 12:30:48 2008 From: stevefranks at ieee.org (Steve Franks) Date: Fri Oct 31 12:31:07 2008 Subject: includes, configure, /usr/lib vs. /usr/local/lib, and linux coders In-Reply-To: References: <539c60b90810311123w2aa94b8akcd0a5d0fe791885a@mail.gmail.com> Message-ID: <539c60b90810311230i11460966la7ff35b0093642ec@mail.gmail.com> On Fri, Oct 31, 2008 at 12:11 PM, Nate Eldredge wrote: > On Fri, 31 Oct 2008, Steve Franks wrote: > >> I believe someone has told me on this list that the proper way to >> compile a linux program is to run configure >> --includedir=/usr/local/include --libdir=/usr/local/lib. > > Nitpick: not specifically a Linux program, but a program using a configure > script generated by GNU's autoconf system. Programs using autoconf may > run on systems other than Linux (usually do, in fact, since the point of > autoconf is portability), and many Linux programs don't use autoconf. > > And I'll assume that by 'linux' you actually mean FreeBSD, in order for this > to be on-topic for this list :) > >> Is that >> correct? I've got a bunch of linux weenies trying to tell me their >> code isn't broken because I'm supposed to have headers where theirs >> are. > > I don't understand this sentence. > >> They state that includedir is used by *their* project to locate >> it's *own* headers, so they never bothered to wire it up in >> Makefile.in....it gets ignored entirely. >> >> I figured I'd better know what I'm talking about before I tell someone >> they are 'wrong'. Especially as it's usually me ;) > > It looks to me like both of you are wrong. :) Looking at the configure that > comes with wget-1.11.2, running configure --help says > > Fine tuning of the installation directories: > ... > --libdir=DIR object code libraries [EPREFIX/lib] > --includedir=DIR C header files [PREFIX/include] > > So it looks like --libdir is supposed to specify where libraries are to be > *installed*, not where they are to be searched for. > > Further up in the help we have > > --prefix=PREFIX install architecture-independent files in PREFIX > [/usr/local] > --exec-prefix=EPREFIX install architecture-dependent files in EPREFIX > [PREFIX] > > So libraries would already be installed in /usr/local/lib by default, unless > you used a --prefix or --exec-prefix option to override that. Similarly for > include files. > > If you need the program being built to search for libraries or include files > in a certain place (such as /usr/local/include and /usr/local/lib, which are > not searched by default on FreeBSD), AFAIK the right way to do it is to set > the LIBRARY_PATH and C_INCLUDE_PATH (or CPLUS_INCLUDE_PATH) environment > variables. > > -- > > Nate Eldredge > neldredge@math.ucsd.edu > Let's backup. What's the 'right' way to get a bloody linux program that expects all it's headers in /usr/include to compile on freebsd where all the headers are in /usr/local/include? That's all I'm really asking. Specifically, it's looking for libusb & libftdi. If I just type gmake, it can't find it, but if I manually edit the Makefiles to add -I/usr/local/include, it can. Obviously, manually editing the makefiles is *not* the right way to fix it (plus it's driving me crazy). Steve Steve From peterjeremy at optushome.com.au Fri Oct 31 12:41:30 2008 From: peterjeremy at optushome.com.au (Peter Jeremy) Date: Fri Oct 31 12:41:39 2008 Subject: includes, configure, /usr/lib vs. /usr/local/lib, and linux coders In-Reply-To: <539c60b90810311123w2aa94b8akcd0a5d0fe791885a@mail.gmail.com> References: <539c60b90810311123w2aa94b8akcd0a5d0fe791885a@mail.gmail.com> Message-ID: <20081031194127.GD99398@server.vk2pj.dyndns.org> On 2008-Oct-31 11:23:53 -0700, Steve Franks wrote: >I believe someone has told me on this list that the proper way to >compile a linux program is to run configure >--includedir=/usr/local/include --libdir=/usr/local/lib. Is that >correct? Yes. The FreeBSD toolchain does not automatically include /usr/local/... though the Linux one does. > I've got a bunch of linux weenies trying to tell me their >code isn't broken because I'm supposed to have headers where theirs >are. A very blinkered PoV... There are a whole pile of reasons why you might use/want a different layout. > They state that includedir is used by *their* project to locate >it's *own* headers, so they never bothered to wire it up in >Makefile.in....it gets ignored entirely. Then their project is broken. These options are intended for use by someone who is building the project. They are not for use by the project itself. -- Peter Jeremy Please excuse any delays as the result of my ISP's inability to implement an MTA that is either RFC2821-compliant or matches their claimed behaviour. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081031/3deb6540/attachment.pgp From neldredge at math.ucsd.edu Fri Oct 31 12:46:42 2008 From: neldredge at math.ucsd.edu (Nate Eldredge) Date: Fri Oct 31 12:46:51 2008 Subject: includes, configure, /usr/lib vs. /usr/local/lib, and linux coders In-Reply-To: <539c60b90810311230i11460966la7ff35b0093642ec@mail.gmail.com> References: <539c60b90810311123w2aa94b8akcd0a5d0fe791885a@mail.gmail.com> <539c60b90810311230i11460966la7ff35b0093642ec@mail.gmail.com> Message-ID: On Fri, 31 Oct 2008, Steve Franks wrote: > Let's backup. What's the 'right' way to get a bloody linux program > that expects all it's headers in /usr/include to compile on freebsd > where all the headers are in /usr/local/include? That's all I'm > really asking. Specifically, it's looking for libusb & libftdi. If I > just type gmake, it can't find it, but if I manually edit the > Makefiles to add -I/usr/local/include, it can. Obviously, manually > editing the makefiles is *not* the right way to fix it (plus it's > driving me crazy). C_INCLUDE_PATH=$C_INCLUDE_PATH:/usr/local/include LIBRARY_PATH=$LIBRARY_PATH:/usr/local/lib export C_INCLUDE_PATH LIBRARY_PATH ./configure gmake Adjust as appropriate if using csh. Personally, I set those environment variables in my .profile. By the way, I think you're being a little unfair to blame this on Linux programs or programmers. Normally it's the user's responsibility to ensure that their compiler searches for include files, etc, in the appropriate place. Many Linux distributions put everything in /usr/include, which is searched by default. FreeBSD puts stuff from ports in /usr/local/include which isn't searched by default. I find that behavior inconvenient, which is why I set those environment variables, so I don't have to think about it. -- Nate Eldredge neldredge@math.ucsd.edu From ed at 80386.nl Fri Oct 31 12:56:35 2008 From: ed at 80386.nl (Ed Schouten) Date: Fri Oct 31 12:56:41 2008 Subject: strange behaviour with /sbin/init and serial console In-Reply-To: <490B5C42.10200@samsco.org> References: <200810311746.23743.thierry.herbelot@free.fr> <20081031190614.GQ1165@hoeg.nl> <490B5C42.10200@samsco.org> Message-ID: <20081031195632.GR1165@hoeg.nl> * Scott Long wrote: > The multiplexed console feature is one thing that linux got right. In a > corporate setting, you really need both a serial console and a video > console in order to effectively manage the machines, as you want to be > able to access them both remotely and locally. While it might be hard > to build multiplexing into the console driver, do you think it would be > possible to layer a multiplexer on top of it, similar to how the kbdmux > driver works? I'm not sure at which level we should implement this. I mainly wrote the new /dev/console implementation, because it is a lot more simple than the existing one and removes ugly hacks from the TTY code (like recursive locking, etc). Maybe if I can find some more time I'll look into it more closely, but my todo list is very long right now. ;-) -- Ed Schouten WWW: http://80386.nl/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081031/c012678f/attachment.pgp From scottl at samsco.org Fri Oct 31 12:42:55 2008 From: scottl at samsco.org (Scott Long) Date: Fri Oct 31 13:20:06 2008 Subject: strange behaviour with /sbin/init and serial console In-Reply-To: <20081031190614.GQ1165@hoeg.nl> References: <200810311746.23743.thierry.herbelot@free.fr> <20081031190614.GQ1165@hoeg.nl> Message-ID: <490B5C42.10200@samsco.org> Ed Schouten wrote: > Hello Theirry, > > * Thierry Herbelot wrote: >> with the following patch on /sbin/init, I have two different behaviours >> depending on the console type (on a i386/32 PC) : >> - on a video console, I see the expected two messages, >> - on a serial console, the messages are not displayed (init silently finishes >> its job and gets to start /etc/rc and everything) >> >> I assume that the writev system call is implemented in >> src/sys/kern/tty_cons.c::cnwrite(), but I could not parse the code to find an >> explanation. >> >> any taker ? >> >> TfH >> >> PS : this is initially for a RELENG_6 machine, but the code is quite similar >> under RELENG_7 or Current > > Any data written to /dev/console is not multiplexed to all console > devices, but only the first active device in the list. The reason behind > this, is because it adds a real lot of complexity to the console code, > especially related to polling and reading on /dev/console. > > This weekend I'm going to commit a replacement implementation of > /dev/console, which also has this restriction. > The multiplexed console feature is one thing that linux got right. In a corporate setting, you really need both a serial console and a video console in order to effectively manage the machines, as you want to be able to access them both remotely and locally. While it might be hard to build multiplexing into the console driver, do you think it would be possible to layer a multiplexer on top of it, similar to how the kbdmux driver works? Scott From deischen at freebsd.org Fri Oct 31 13:21:47 2008 From: deischen at freebsd.org (Daniel Eischen) Date: Fri Oct 31 13:21:54 2008 Subject: includes, configure, /usr/lib vs. /usr/local/lib, and linux coders In-Reply-To: References: <539c60b90810311123w2aa94b8akcd0a5d0fe791885a@mail.gmail.com> <539c60b90810311230i11460966la7ff35b0093642ec@mail.gmail.com> Message-ID: On Fri, 31 Oct 2008, Nate Eldredge wrote: > On Fri, 31 Oct 2008, Steve Franks wrote: > >> Let's backup. What's the 'right' way to get a bloody linux program >> that expects all it's headers in /usr/include to compile on freebsd >> where all the headers are in /usr/local/include? That's all I'm >> really asking. Specifically, it's looking for libusb & libftdi. If I >> just type gmake, it can't find it, but if I manually edit the >> Makefiles to add -I/usr/local/include, it can. Obviously, manually >> editing the makefiles is *not* the right way to fix it (plus it's >> driving me crazy). > > C_INCLUDE_PATH=$C_INCLUDE_PATH:/usr/local/include > LIBRARY_PATH=$LIBRARY_PATH:/usr/local/lib > export C_INCLUDE_PATH LIBRARY_PATH > ./configure > gmake > > Adjust as appropriate if using csh. > > Personally, I set those environment variables in my .profile. > > By the way, I think you're being a little unfair to blame this on Linux > programs or programmers. Normally it's the user's responsibility to ensure > that their compiler searches for include files, etc, in the appropriate > place. Many Linux distributions put everything in /usr/include, which is > searched by default. FreeBSD puts stuff from ports in /usr/local/include > which isn't searched by default. I find that behavior inconvenient, which is > why I set those environment variables, so I don't have to think about it. I don't really care who's to blame (I'd guess I'd blame both the Linux distros and the Linux application developers), but the move to put everything in /usr/include and /usr/lib annoys the heck out of me. It blurs the line between the base OS and installed 3rd party software. Perhaps that's because Linux is really just a kernel, and to the distributors - most, if not all, of their software is 3rd-party. It's really nice to be able to install 3rd-party software so that it doesn't affect the base OS. On FreeBSD, it's easy enough just to 'rm -rf /usr/local' and start fresh without having to worry about screwing up the base OS. -- DE From tijl at ulyssis.org Fri Oct 31 13:32:25 2008 From: tijl at ulyssis.org (Tijl Coosemans) Date: Fri Oct 31 13:32:33 2008 Subject: includes, configure, /usr/lib vs. /usr/local/lib, and linux coders In-Reply-To: <539c60b90810311230i11460966la7ff35b0093642ec@mail.gmail.com> References: <539c60b90810311123w2aa94b8akcd0a5d0fe791885a@mail.gmail.com> <539c60b90810311230i11460966la7ff35b0093642ec@mail.gmail.com> Message-ID: <200810312101.22605.tijl@ulyssis.org> On Friday 31 October 2008 20:30:46 Steve Franks wrote: > Let's backup. What's the 'right' way to get a bloody linux program > that expects all it's headers in /usr/include to compile on freebsd > where all the headers are in /usr/local/include? That's all I'm > really asking. Specifically, it's looking for libusb & libftdi. If > I just type gmake, it can't find it, but if I manually edit the > Makefiles to add -I/usr/local/include, it can. Obviously, manually > editing the makefiles is *not* the right way to fix it (plus it's > driving me crazy). ./configure CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/lib" They should consider using pkg-config in their configure script to locate libusb and libftdi. From koitsu at FreeBSD.org Fri Oct 31 14:09:07 2008 From: koitsu at FreeBSD.org (Jeremy Chadwick) Date: Fri Oct 31 14:09:14 2008 Subject: strange behaviour with /sbin/init and serial console In-Reply-To: <490B5C42.10200@samsco.org> References: <200810311746.23743.thierry.herbelot@free.fr> <20081031190614.GQ1165@hoeg.nl> <490B5C42.10200@samsco.org> Message-ID: <20081031205305.GA41320@icarus.home.lan> On Fri, Oct 31, 2008 at 01:28:02PM -0600, Scott Long wrote: > Ed Schouten wrote: >> Hello Theirry, >> >> * Thierry Herbelot wrote: >>> with the following patch on /sbin/init, I have two different >>> behaviours depending on the console type (on a i386/32 PC) : >>> - on a video console, I see the expected two messages, >>> - on a serial console, the messages are not displayed (init silently >>> finishes its job and gets to start /etc/rc and everything) >>> >>> I assume that the writev system call is implemented in >>> src/sys/kern/tty_cons.c::cnwrite(), but I could not parse the code to >>> find an explanation. >>> >>> any taker ? >>> >>> TfH >>> >>> PS : this is initially for a RELENG_6 machine, but the code is quite >>> similar under RELENG_7 or Current >> >> Any data written to /dev/console is not multiplexed to all console >> devices, but only the first active device in the list. The reason behind >> this, is because it adds a real lot of complexity to the console code, >> especially related to polling and reading on /dev/console. >> >> This weekend I'm going to commit a replacement implementation of >> /dev/console, which also has this restriction. >> > > The multiplexed console feature is one thing that linux got right. In a > corporate setting, you really need both a serial console and a video > console in order to effectively manage the machines, as you want to be > able to access them both remotely and locally. I know this comment isn't much help, but, I am in full agreement with Scott. FreeBSD's lack of *true* multi (or even dual) console during all stages is a big disappointment to server administrators. The common reaction is: "What do you mean I can only get some messages on serial or some messages on VGA?! That's retarded!" I believe DragonFly has addressed this (offering a true dual console mechanism), and if I remember correctly, Matt Dillon discussed the code changes in great detail, citing a large amount of re-engineering required to accomplish it. > While it might be hard to build multiplexing into the console driver, > do you think it would be possible to layer a multiplexer on top of it, > similar to how the kbdmux driver works? Let's make sure that we don't implement it identically though, as there are many of us who have major problems with kbdmux (reports of LORs, and even more reports of incredibly slow keyboard input when a USB keyboard is used; workarounds are either disabling atkbd/atkbdc entirely, or disabling kbdmux entirely. In my case, I found the latter to be preferable). :-) -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |