From david.kwan at isilon.com Tue Jul 1 15:50:28 2008 From: david.kwan at isilon.com (David Kwan) Date: Tue Jul 1 15:52:18 2008 Subject: TCP stack in FreeBSD poor performance in 100MB to Gigabit environment. Message-ID: I have a few questions regarding the TCP: I have a situation with clients on a 100MB network connecting to servers on a Gigabit network where the client read speeds are very slow from the FreeBSD server and fast from the Linux server. Write speeds from the clients to both servers are fast. (Clients on the gigabit network work fine with blazing read and write speeds). The network traces shows congestion packets for both servers when doing reads from the clients (dup acks and retransmissions), but the Linux server seem to handle the congestion better. ECN is not enabled on the network and I don't see any congestion windowing or clients window changing. The 100/1G switch is dropping packets. I double checked the network configuration and also swapped swithports for the servers to use the others to make sure the switch configuration are the same and the Linux always does better than FreeBSD. Assuming that the network configuration is a constant for all clients and servers (speed, duplex, and etc...), the only variable is the servers themselves (Linux and FreeBSD). I have tried a couple of FreeBSD machines with 6.1 and 7.0 and they show the same problem, with no luck matching the speed and network utilization of Linux (2 years old). The read speed test I'm referring is doing transferring of a 100MB file (cifs, nfs, and ftp), and the Linux server does it constantly in around 10 sec (line speed) with a constant network utilization chart, while the FreeBSD servers are magnitudes slower with erratic network utilization chart. I've attempted to tweak some network sysctl options on the FreeBSD, and the only ones that helped were disabling TSO and inflight; which leads me to think that the inter-packet gap was slightly increased to partially relieve congestion on the switch; not a long term solution. My questions are: 1. Have you heard of this problem before with 100MB clients to Gigabit servers? 2. Are you aware of any Linux fix/patch the TCP stack to better handling congestion than FreeBSD? I'm looking to address this issue in the FreeBSD, but wondering if the Linux stack did something special that can help with the FreeBSD performance. David K. From hwahing at smartteam.net Wed Jul 2 03:09:55 2008 From: hwahing at smartteam.net (Hwa Hing) Date: Wed Jul 2 03:10:00 2008 Subject: TCP stack in FreeBSD poor performance in 100MB to Gigabit environment. In-Reply-To: References: Message-ID: <486AE678.8050206@smartteam.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I also had such problem with FreeBSD-7 Stable-SMP with 4 port Intel 1GBit NIC em Driver and with Packet Filter, ALTQ enabled. I tested with iperf from network a to network b. I found freebsd droping packets and the transfer speed in routed mode is only 2 mbit/s compare to linux router which able to forward the traffic at almost full speed. David Kwan wrote: > > > I have a few questions regarding the TCP: > > > > I have a situation with clients on a 100MB network connecting to servers > on a Gigabit network where the client read speeds are very slow from the > FreeBSD server and fast from the Linux server. Write speeds from the > clients to both servers are fast. (Clients on the gigabit network work > fine with blazing read and write speeds). The network traces shows > congestion packets for both servers when doing reads from the clients > (dup acks and retransmissions), but the Linux server seem to handle the > congestion better. ECN is not enabled on the network and I don't see > any congestion windowing or clients window changing. The 100/1G switch > is dropping packets. I double checked the network configuration and > also swapped swithports for the servers to use the others to make sure > the switch configuration are the same and the Linux always does better > than FreeBSD. Assuming that the network configuration is a constant for > all clients and servers (speed, duplex, and etc...), the only variable > is the servers themselves (Linux and FreeBSD). I have tried a couple > of FreeBSD machines with 6.1 and 7.0 and they show the same problem, > with no luck matching the speed and network utilization of Linux (2 > years old). The read speed test I'm referring is doing transferring of > a 100MB file (cifs, nfs, and ftp), and the Linux server does it > constantly in around 10 sec (line speed) with a constant network > utilization chart, while the FreeBSD servers are magnitudes slower with > erratic network utilization chart. I've attempted to tweak some network > sysctl options on the FreeBSD, and the only ones that helped were > disabling TSO and inflight; which leads me to think that the > inter-packet gap was slightly increased to partially relieve congestion > on the switch; not a long term solution. > > > > My questions are: > > 1. Have you heard of this problem before with 100MB clients to > Gigabit servers? > > 2. Are you aware of any Linux fix/patch the TCP stack to better > handling congestion than FreeBSD? I'm looking to address this issue in > the FreeBSD, but wondering if the Linux stack did something special that > can help with the FreeBSD performance. > > > > David K. > > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkhq5ngACgkQn94C8abtJZqjGQCgguGT6NG4LknjD3El+dnamlBv Z1cAniBSYCJ7n2wi/MpWgRcdVhBWcMVT =6jzX -----END PGP SIGNATURE----- From todor at fbi.ie Thu Jul 3 11:12:23 2008 From: todor at fbi.ie (Ing. Todor Colakov) Date: Thu Jul 3 11:12:30 2008 Subject: how to measure performance Message-ID: Hello, I'm trying to measure performance of my servers. I would like to have (at least rough) estimation, how much of punishment it can take, so I've crawled web for some howtos.. i've founded some, it gave me some start, but what i've miss in all of them was simple guide in the style... If you are setting new server, you are interested at THIS, THIS and THIS (say how much packets incomming packets can it handle at one time) because of THAT, THAT and THAT (because if you plan to put it into enviroment of some magnitudes higher it wont do) and you can meassure it with tool A B C ( dunno :( ) Can somebody help me compile this list? I would like address specially experienced admins. I would like to have this list mainly for general server (without specialization), mailserver, webserver, DNS and NFS server - because it address services the most of newbiee admins work with. Thank to all Todor From kris at FreeBSD.org Thu Jul 3 13:11:34 2008 From: kris at FreeBSD.org (Kris Kennaway) Date: Thu Jul 3 13:11:36 2008 Subject: how to measure performance In-Reply-To: References: Message-ID: <486CCFEF.4080000@FreeBSD.org> Ing. Todor Colakov wrote: > Hello, > > I'm trying to measure performance of my servers. I would like to have (at least rough) estimation, how much of punishment it can take, so I've crawled web for some howtos.. i've founded some, it gave me some start, but what i've miss in all of them was simple guide in the style... > > If you are setting new server, you are interested at THIS, THIS and THIS (say how much packets incomming packets can it handle at one time) because of THAT, THAT and THAT (because if you plan to put it into enviroment of some magnitudes higher it wont do) and you can meassure it with tool A B C ( dunno :( ) > > Can somebody help me compile this list? I would like address specially experienced admins. I would like to have this list mainly for general server (without specialization), mailserver, webserver, DNS and NFS server - because it address services the most of newbiee admins work with. > Thank to all > Todor Perhaps the answer is because there is no single thing as "performance". Rather, "performance" can mean many things depending on what your requirements are. What do you use your server for? Kris From todor at fbi.ie Fri Jul 4 12:58:55 2008 From: todor at fbi.ie (Ing. Todor Colakov) Date: Fri Jul 4 12:59:01 2008 Subject: performance meassuring Message-ID: I have mailserver, webserver, DNS and NFS server. I know there is no specific performance value, because of that I wanted make separated lists for those basic types (Maybe my english wasn't descriptive enough for that :( ). And also I wanted know why that or other value is important for the concrete type of server... I know it all stands on the knowledges of HW architectures, kernel, FS and alike... but I don't have these knowdleges (yet) and I need to meassure server. Because I need to have feedback, when changing setting of server. Todor P.S. Can the number of packets per second be the optimistic assumption of raw network processing power of the server? From tevans.uk at googlemail.com Fri Jul 4 15:00:02 2008 From: tevans.uk at googlemail.com (Tom Evans) Date: Fri Jul 4 15:00:07 2008 Subject: performance meassuring In-Reply-To: References: Message-ID: <1215181994.35536.58.camel@localhost> On Fri, 2008-07-04 at 04:58 -0800, Ing. Todor Colakov wrote: > I have mailserver, webserver, DNS and NFS server. I know there is no specific > performance value, because of that I wanted make separated lists for those basic types > (Maybe my english wasn't descriptive enough for that :( ). And also I wanted know why that > or other value is important for the concrete type of server... I know it all stands on the > knowledges of HW architectures, kernel, FS and alike... but I don't have these knowdleges > (yet) and I need to meassure server. Because I need to have feedback, when changing > setting of server. > Todor > P.S. Can the number of packets per second be the optimistic assumption of raw network > processing power of the server? PPS of what? If the server is a forwarding router, then PPS of 64 byte packets forwarded would be a good metric. If its a database server, then PPS is largely irrelevant. Generally speaking, work out what you want to change, find a good metric to measure the performance change (Eg, change filesystem/tuning options, run bonnie++), measure before and after. Even then, its not that simple, as bonnie++ doesnt accurately represent my FS-related workloads. The ideal way is to find a way of measuring $APPLICATION, since that is what the server is for. Tom -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: This is a digitally signed message part Url : http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20080704/a8d5bb29/attachment.pgp From rabing at omc.net Fri Jul 4 16:53:13 2008 From: rabing at omc.net (Lutz Rabing) Date: Fri Jul 4 17:20:02 2008 Subject: low sysbench scores on 8 core server Message-ID: <486E4C20.7050102@omc.net> hi, I did some testing an a supermicro 2 x 4 core xeon server under 64bit "7.0-STABLE #1: Fri Jul 4". when we first tested the system under load (2000 apache threads) the system performed bad compare to other dual core systems under the same workload. the 8 core system had 0% idle time and almost 100% system load. I could not find out what the load was. disk IO was close to zero during that time. because of that I checked the sysbench results with this test: sysbench --test=oltp --mysql-socket=/tmp/mysql.sock --mysql-user=root \ --max-requests=0 --max-time=60 --oltp-read -only=on --num-threads=$1 run here are the sysbench results: - 1 thread -------------------------------------------------------- OLTP test statistics: queries performed: read: 354774 write: 0 other: 50682 total: 405456 transactions: 25341 (422.34 per sec.) deadlocks: 0 (0.00 per sec.) read/write requests: 354774 (5912.82 per sec.) other operations: 50682 (844.69 per sec.) - 8 threads ------------------------------------------------------- OLTP test statistics: queries performed: read: 145166 write: 0 other: 20738 total: 165904 transactions: 10369 (172.75 per sec.) deadlocks: 0 (0.00 per sec.) read/write requests: 145166 (2418.54 per sec.) other operations: 20738 (345.51 per sec.) -------------------------------------------------------------------- during the 8 thread test "top" looks like this: ----------------------------------------------------------------------------- last pid: 30104; load averages: 2.50, 0.65, 0.52 up 0+01:49:44 17:52:14 69 processes: 7 running, 62 sleeping CPU: 5.6% user, 0.0% nice, 64.0% system, 0.0% interrupt, 30.4% idle Mem: 44M Active, 296M Inact, 277M Wired, 84K Cache, 214M Buf, 15G Free Swap: 16G Total, 16G Free ----------------------------------------------------------------------------- software versions used: - mysql-server-5.1.25 - sysbench-0.4.8 any ideas what I should look for or what to change? I could supply more info if needed. dmesg appended. thanks, lutz ================================================================================================== FreeBSD 7.0-STABLE #1: Fri Jul 4 01:41:13 CEST 2008 root@www5.4eins.de:/usr/obj/usr/src/sys/OMC Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz (2333.35-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6f7 Stepping = 7 Features=0xbfebfbff Features2=0x4e3bd AMD Features=0x20100800 AMD Features2=0x1 Cores per package: 4 usable memory = 17166041088 (16370 MB) avail memory = 16626790400 (15856 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard kbd1 at kbdmux0 ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 2.0 on pci0 pci1: on pcib1 pcib2: irq 16 at device 0.0 on pci1 pci2: on pcib2 pcib3: irq 16 at device 0.0 on pci2 pci3: on pcib3 pcib4: at device 2.0 on pci2 pci4: on pcib4 em0: port 0x2000-0x201f mem 0xda000000-0xda01ffff irq 18 at device 0.0 on pci4 em0: Using MSI interrupt em0: [FILTER] em0: Ethernet address: 00:30:48:79:cb:ba em1: port 0x2020-0x203f mem 0xda020000-0xda03ffff irq 19 at device 0.1 on pci4 em1: Using MSI interrupt em1: [FILTER] em1: Ethernet address: 00:30:48:79:cb:bb pcib5: at device 0.3 on pci1 pci5: on pcib5 pcib6: at device 4.0 on pci0 pci6: on pcib6 pcib7: at device 0.0 on pci6 pci7: on pcib7 pcib8: at device 0.2 on pci6 pci8: on pcib8 pcib9: at device 6.0 on pci0 pci9: on pcib9 pcib10: at device 0.0 on pci9 pci10: on pcib10 3ware device driver for 9000 series storage controllers, version: 3.70.05.001 twa0: <3ware 9000 series Storage Controller> port 0x3000-0x303f mem 0xd8000000-0xd9ffffff,0xda300000-0xda300fff irq 18 at device 1.0 on pci10 twa0: [ITHREAD] twa0: INFO: (0x04: 0x0053): Battery capacity test is overdue: twa0: INFO: (0x15: 0x1300): Controller details:: Model 9550SXU-4LP, 4 ports, Firmware FE9X 3.04.00.005, BIOS BE9X 3.04.00.002 pcib11: at device 0.2 on pci9 pci11: on pcib11 pci0: at device 8.0 (no driver attached) uhci0: port 0x1800-0x181f irq 17 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb0: on uhci0 usb0: USB revision 1.0 uhub0: on usb0 uhub0: 2 ports with 2 removable, self powered uhci1: port 0x1820-0x183f irq 19 at device 29.1 on pci0 uhci1: [GIANT-LOCKED] uhci1: [ITHREAD] usb1: on uhci1 usb1: USB revision 1.0 uhub1: on usb1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0x1840-0x185f irq 18 at device 29.2 on pci0 uhci2: [GIANT-LOCKED] uhci2: [ITHREAD] usb2: on uhci2 usb2: USB revision 1.0 uhub2: on usb2 uhub2: 2 ports with 2 removable, self powered ehci0: mem 0xda600400-0xda6007ff irq 17 at device 29.7 on pci0 ehci0: [GIANT-LOCKED] ehci0: [ITHREAD] usb3: EHCI version 1.0 usb3: companion controllers, 2 ports each: usb0 usb1 usb2 usb3: on ehci0 usb3: USB revision 2.0 uhub3: on usb3 uhub3: 6 ports with 6 removable, self powered pcib12: at device 30.0 on pci0 pci12: on pcib12 vgapci0: port 0x4000-0x40ff mem 0xd0000000-0xd7ffffff,0xda200000-0xda20ffff irq 18 at device 1.0 on pci12 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1860-0x186f at device 31.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] atapci1: port 0x18a0-0x18a7,0x1874-0x1877,0x1878-0x187f,0x1870-0x1873,0x1880-0x189f mem 0xda600800-0xda600bff irq 19 at device 31.2 on pci0 atapci1: [ITHREAD] atapci1: AHCI Version 01.10 controller with 6 ports detected ata2: on atapci1 ata2: [ITHREAD] ata3: on atapci1 ata3: [ITHREAD] ata4: on atapci1 ata4: [ITHREAD] ata5: on atapci1 ata5: [ITHREAD] ata6: on atapci1 ata6: [ITHREAD] ata7: on atapci1 ata7: [ITHREAD] pci0: at device 31.3 (no driver attached) cpu0: on acpi0 est0: on cpu0 p4tcc0: on cpu0 cpu1: on acpi0 est1: on cpu1 p4tcc1: on cpu1 cpu2: on acpi0 est2: on cpu2 p4tcc2: on cpu2 cpu3: on acpi0 est3: on cpu3 p4tcc3: on cpu3 cpu4: on acpi0 est4: on cpu4 p4tcc4: on cpu4 cpu5: on acpi0 est5: on cpu5 p4tcc5: on cpu5 cpu6: on acpi0 est6: on cpu6 p4tcc6: on cpu6 cpu7: on acpi0 est7: on cpu7 p4tcc7: on cpu7 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio0: [FILTER] sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A sio1: [FILTER] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 acpi_button0: on acpi0 orm0: at iomem 0xc0000-0xcafff,0xcb000-0xcc7ff on isa0 ppc0: cannot reserve I/O port range sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based forwarding disabled, default to accept, logging limited to 100 packets/entry by default acd0: CDRW at ata0-slave UDMA33 SMP: AP CPU #2 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #5 Launched! da0 at twa0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-3 device da0: 100.000MB/s transfers da0: 1430490MB (2929643520 512 byte sectors: 255H 63S/T 182361C) Trying to mount root from ufs:/dev/da0s1a From kris at FreeBSD.org Fri Jul 4 18:25:05 2008 From: kris at FreeBSD.org (Kris Kennaway) Date: Fri Jul 4 18:25:11 2008 Subject: low sysbench scores on 8 core server In-Reply-To: <486E4C20.7050102@omc.net> References: <486E4C20.7050102@omc.net> Message-ID: <486E6B00.10208@FreeBSD.org> Lutz Rabing wrote: > hi, > > I did some testing an a supermicro 2 x 4 core xeon server under 64bit > "7.0-STABLE #1: Fri Jul 4". when we first tested the system under load > (2000 apache threads) the system performed bad compare to other dual > core systems under the same workload. > > the 8 core system had 0% idle time and almost 100% system load. I could > not find out what the load was. disk IO was close to zero during that time. > > because of that I checked the sysbench results with this test: > > sysbench --test=oltp --mysql-socket=/tmp/mysql.sock --mysql-user=root \ > --max-requests=0 --max-time=60 --oltp-read -only=on --num-threads=$1 run See my tuning notes at: http://people.freebsd.org/~kris/scaling/mysql.html > software versions used: > - mysql-server-5.1.25 In my tests mysql 5.1 has much worse performance than 5.0. Kris From kris at FreeBSD.org Fri Jul 4 18:30:37 2008 From: kris at FreeBSD.org (Kris Kennaway) Date: Fri Jul 4 18:30:43 2008 Subject: TCP stack in FreeBSD poor performance in 100MB to Gigabit environment. In-Reply-To: <486AE678.8050206@smartteam.net> References: <486AE678.8050206@smartteam.net> Message-ID: <486E6C4B.6060902@FreeBSD.org> Hwa Hing wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > > I also had such problem with FreeBSD-7 Stable-SMP with 4 port Intel > 1GBit NIC em Driver and with Packet Filter, ALTQ enabled. But it doesn't sound like you are testing the same thing at all. > I tested with iperf from network a to network b. I found freebsd droping > packets and the transfer speed in routed mode is only 2 mbit/s compare > to linux router which able to forward the traffic at almost full speed. Check that the software isn't making invalid Linux-specific assumptions about e.g. socket buffer sizes. Kris From kris at FreeBSD.org Fri Jul 4 18:31:49 2008 From: kris at FreeBSD.org (Kris Kennaway) Date: Fri Jul 4 18:31:56 2008 Subject: TCP stack in FreeBSD poor performance in 100MB to Gigabit environment. In-Reply-To: References: Message-ID: <486E6C95.1020509@FreeBSD.org> David Kwan wrote: > > > I have a few questions regarding the TCP: > > > > I have a situation with clients on a 100MB network connecting to servers > on a Gigabit network where the client read speeds are very slow from the > FreeBSD server and fast from the Linux server. Write speeds from the > clients to both servers are fast. (Clients on the gigabit network work > fine with blazing read and write speeds). The network traces shows > congestion packets for both servers when doing reads from the clients > (dup acks and retransmissions), but the Linux server seem to handle the > congestion better. ECN is not enabled on the network and I don't see > any congestion windowing or clients window changing. The 100/1G switch > is dropping packets. I double checked the network configuration and > also swapped swithports for the servers to use the others to make sure > the switch configuration are the same and the Linux always does better > than FreeBSD. Assuming that the network configuration is a constant for > all clients and servers (speed, duplex, and etc...), the only variable > is the servers themselves (Linux and FreeBSD). I have tried a couple > of FreeBSD machines with 6.1 and 7.0 and they show the same problem, > with no luck matching the speed and network utilization of Linux (2 > years old). The read speed test I'm referring is doing transferring of > a 100MB file (cifs, nfs, and ftp), and the Linux server does it > constantly in around 10 sec (line speed) with a constant network > utilization chart, while the FreeBSD servers are magnitudes slower with > erratic network utilization chart. I've attempted to tweak some network > sysctl options on the FreeBSD, and the only ones that helped were > disabling TSO and inflight; which leads me to think that the > inter-packet gap was slightly increased to partially relieve congestion > on the switch; not a long term solution. > > > > My questions are: > > 1. Have you heard of this problem before with 100MB clients to > Gigabit servers? The key point seems to be that the switch is dropping packets. If you have packet loss then TCP is not going to perform well. Kris From kris at FreeBSD.org Mon Jul 7 09:43:20 2008 From: kris at FreeBSD.org (Kris Kennaway) Date: Mon Jul 7 09:43:26 2008 Subject: low sysbench scores on 8 core server In-Reply-To: <4871D784.1020809@omc.net> References: <486E4C20.7050102@omc.net> <486E6B00.10208@FreeBSD.org> <4871D784.1020809@omc.net> Message-ID: <4871E537.7060504@FreeBSD.org> Lutz Rabing wrote: > Kris Kennaway schrieb: >> Lutz Rabing wrote: >>> hi, >>> >>> I did some testing an a supermicro 2 x 4 core xeon server under 64bit >>> "7.0-STABLE #1: Fri Jul 4". when we first tested the system under load >>> (2000 apache threads) the system performed bad compare to other dual >>> core systems under the same workload. >>> >>> the 8 core system had 0% idle time and almost 100% system load. I could >>> not find out what the load was. disk IO was close to zero during that >>> time. >>> >>> because of that I checked the sysbench results with this test: >>> >>> sysbench --test=oltp --mysql-socket=/tmp/mysql.sock --mysql-user=root \ >>> --max-requests=0 --max-time=60 --oltp-read -only=on --num-threads=$1 run >> See my tuning notes at: >> >> http://people.freebsd.org/~kris/scaling/mysql.html >> >>> software versions used: >>> - mysql-server-5.1.25 >> In my tests mysql 5.1 has much worse performance than 5.0. >> >> Kris >> >> >> . >> > > > hi kris, > > thanks for your response. going back to mysql-5.0.51a the 8core server > performed as expected. (sysbench oltp: 4335 with 8 threads) Good to know. Let's hope mysql get their act together with the next release ;-) Kris From rabing at omc.net Mon Jul 7 08:45:46 2008 From: rabing at omc.net (Lutz Rabing) Date: Mon Jul 7 11:25:06 2008 Subject: low sysbench scores on 8 core server In-Reply-To: <486E6B00.10208@FreeBSD.org> References: <486E4C20.7050102@omc.net> <486E6B00.10208@FreeBSD.org> Message-ID: <4871D784.1020809@omc.net> Kris Kennaway schrieb: > Lutz Rabing wrote: >> hi, >> >> I did some testing an a supermicro 2 x 4 core xeon server under 64bit >> "7.0-STABLE #1: Fri Jul 4". when we first tested the system under load >> (2000 apache threads) the system performed bad compare to other dual >> core systems under the same workload. >> >> the 8 core system had 0% idle time and almost 100% system load. I could >> not find out what the load was. disk IO was close to zero during that >> time. >> >> because of that I checked the sysbench results with this test: >> >> sysbench --test=oltp --mysql-socket=/tmp/mysql.sock --mysql-user=root \ >> --max-requests=0 --max-time=60 --oltp-read -only=on --num-threads=$1 run > > See my tuning notes at: > > http://people.freebsd.org/~kris/scaling/mysql.html > >> software versions used: >> - mysql-server-5.1.25 > > In my tests mysql 5.1 has much worse performance than 5.0. > > Kris > > > . > hi kris, thanks for your response. going back to mysql-5.0.51a the 8core server performed as expected. (sysbench oltp: 4335 with 8 threads) thanks, lutz From rahulone at gmail.com Tue Jul 15 20:56:15 2008 From: rahulone at gmail.com (Rahul) Date: Tue Jul 15 20:56:22 2008 Subject: Real Insight on Performance Message-ID: <4804a6670807151328y2cf30363x9b808912286ea1f5@mail.gmail.com> Hi all, I am looking to install Unix (and/or like) system for server to run some highly computational and multi-threaded applications. My background being pretty much all Windows, I am a novice to this field. I have grasped some basics by reading some material on web and how-to books but event after extensive digging around on web for real performance numbers on various operating systems, I still haven't found anything useful. Most of the data I found were basically comparisons of operating systems running MySQL or PostgreSQL to see how many connections or simple look up queries they each can server per second sort of things. But nothing that would point to underlying operations like threading, cache-ing, time slicing, I/O, etc. Now, I must admit most of the material showed Linux having upper hand. But I am not convinced FreeBSD would be behind in almost all performance benchmarks from always hearing the legendary performance and stability characteristics of FreeBSD. Can you please shed some light on what I really should be looking for in FreeBSD to optimize it to it's best performance? Am I expecting something that is just purely not BSD's priority or philosophy, per se? Is there some material I can look over? (note: I've gone through the 7.0 Preview and Tuning documentation on FreeBSD's site already.) Thanks. From julian at elischer.org Tue Jul 15 21:17:56 2008 From: julian at elischer.org (Julian Elischer) Date: Tue Jul 15 21:18:05 2008 Subject: Real Insight on Performance In-Reply-To: <4804a6670807151328y2cf30363x9b808912286ea1f5@mail.gmail.com> References: <4804a6670807151328y2cf30363x9b808912286ea1f5@mail.gmail.com> Message-ID: <487D1069.3030801@elischer.org> Rahul wrote: > Hi all, > > I am looking to install Unix (and/or like) system for server to run > some highly computational and multi-threaded applications. My > background being pretty much all Windows, I am a novice to this field. > I have grasped some basics by reading some material on web and how-to > books but event after extensive digging around on web for real > performance numbers on various operating systems, I still haven't > found anything useful. Most of the data I found were basically > comparisons of operating systems running MySQL or PostgreSQL to see > how many connections or simple look up queries they each can server > per second sort of things. But nothing that would point to underlying > operations like threading, cache-ing, time slicing, I/O, etc. > > Now, I must admit most of the material showed Linux having upper hand. > But I am not convinced FreeBSD would be behind in almost all > performance benchmarks from always hearing the legendary performance > and stability characteristics of FreeBSD. most Linux documents will show Linux having hte upper hand of course.. For computational stuff in my experience (Image analysis software using Feedback networks with a mix of floating and fixed point work) there was not much to choose between the various OS's because the limiting factor tends to simply be the hardware calculating throughput. If you have networking or IO as part of the equation then of course it's different. One deal that we can offer you that Linux won't is that if you are prepared to work with us, we can help you find any bottlenecks that are hitting you.. (the advantage of working with a smaller group :-) > > Can you please shed some light on what I really should be looking for > in FreeBSD to optimize it to it's best performance? Am I expecting > something that is just purely not BSD's priority or philosophy, per > se? Is there some material I can look over? (note: I've gone through > the 7.0 Preview and Tuning documentation on FreeBSD's site already.) you need to tell us more about your workload. > > > > Thanks. > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" From rahulone at gmail.com Wed Jul 16 00:17:47 2008 From: rahulone at gmail.com (Rahul) Date: Wed Jul 16 00:17:53 2008 Subject: Real Insight on Performance In-Reply-To: <487D1069.3030801@elischer.org> References: <4804a6670807151328y2cf30363x9b808912286ea1f5@mail.gmail.com> <487D1069.3030801@elischer.org> Message-ID: <4804a6670807151717r54f2889fq6c9c70afebee413d@mail.gmail.com> Thanks for the prompt response. Believe me I am fan of FreeBSD already. Your group size and discipline is what has attracted me to FreeBSD and the great respect for such work preceding it! It's just that when it comes to work/business I have to be objective. Hence, the extensive research I am doing to find the right OS for "my purpose". I know hardware has a lot to do with floating point calculations, integer math, pointers, and data caching. I don't have any utility at the moment I can run to simply measure performance. And I don't think there will be only one utility running on there all the time. For example: I would like to run a high performance web server that can handle up to 200 connections per second and serve them with great speed. That's where the concern for multi-threaded support. Depending on the request, I may have to load (and possibly unload) dynamic modules to perform calculations, and if need be, fetch data from either DB or flat file. It could involve connecting to a process on another box to get request specific command strings. This process could run for almost 20 hours straight and OS still has to be able to keep in shape. Shewww... so there is multiple parts to this. I don't want to throw hardware as the solution for anything if it could be resolved by choosing the OS and tuning it. Because hardware only solves problem temporarily, it does not give you a true measure of your capability and thus renders any prediction of linear scaling impossible. So on and so forth. Thanks again. On Tue, Jul 15, 2008 at 5:02 PM, Julian Elischer wrote: > Rahul wrote: >> >> Hi all, >> >> I am looking to install Unix (and/or like) system for server to run >> some highly computational and multi-threaded applications. My >> background being pretty much all Windows, I am a novice to this field. >> I have grasped some basics by reading some material on web and how-to >> books but event after extensive digging around on web for real >> performance numbers on various operating systems, I still haven't >> found anything useful. Most of the data I found were basically >> comparisons of operating systems running MySQL or PostgreSQL to see >> how many connections or simple look up queries they each can server >> per second sort of things. But nothing that would point to underlying >> operations like threading, cache-ing, time slicing, I/O, etc. >> >> Now, I must admit most of the material showed Linux having upper hand. >> But I am not convinced FreeBSD would be behind in almost all >> performance benchmarks from always hearing the legendary performance >> and stability characteristics of FreeBSD. > > most Linux documents will show Linux having hte upper hand of course.. > For computational stuff in my experience (Image analysis software > using Feedback networks with a mix of floating and fixed point work) > there was not much to choose between the various OS's because the > limiting factor tends to simply be the hardware calculating throughput. > If you have networking or IO as part of the equation then > of course it's different. > > One deal that we can offer you that Linux won't is that if you are prepared > to work with us, we can help you find any bottlenecks > that are hitting you.. > (the advantage of working with a smaller group :-) > > >> >> Can you please shed some light on what I really should be looking for >> in FreeBSD to optimize it to it's best performance? Am I expecting >> something that is just purely not BSD's priority or philosophy, per >> se? Is there some material I can look over? (note: I've gone through >> the 7.0 Preview and Tuning documentation on FreeBSD's site already.) > > you need to tell us more about your workload. > >> >> >> >> Thanks. >> _______________________________________________ >> freebsd-performance@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >> To unsubscribe, send any mail to >> "freebsd-performance-unsubscribe@freebsd.org" > > From Alexander at Leidinger.net Wed Jul 16 06:21:54 2008 From: Alexander at Leidinger.net (Alexander Leidinger) Date: Wed Jul 16 06:22:10 2008 Subject: Real Insight on Performance In-Reply-To: <4804a6670807151717r54f2889fq6c9c70afebee413d@mail.gmail.com> References: <4804a6670807151328y2cf30363x9b808912286ea1f5@mail.gmail.com> <487D1069.3030801@elischer.org> <4804a6670807151717r54f2889fq6c9c70afebee413d@mail.gmail.com> Message-ID: <20080716080532.18862vjyr477xls0@webmail.leidinger.net> Quoting Rahul (from Tue, 15 Jul 2008 20:17:46 -0400): > For example: I would like to run a high performance web server that > can handle up to 200 connections per second and serve them with great > speed. That's where the concern for multi-threaded support. Depending Serving 200 connections per second for static data is not hard. And you don't need multi-threading for that. Multiple processes is actually better than multi-threading in this regard, as you don't have to do locking of filedescriptors in the the webserver over multiple threads. The difference between linux and FreeBSD should be not big for single CPU/core systems, but if you increase the number of CPUs/cores this may be different. See the postgresql (it uses processes, not threads like mysql) graphs in http://people.freebsd.org/~kris/scaling/7.0%20and%20beyond.pdf to get an idea about the scaling of more or less independent processes in FreeBSD. > on the request, I may have to load (and possibly unload) dynamic > modules to perform calculations, and if need be, fetch data from > either DB or flat file. It could involve connecting to a process on As soon as you have calculations and/or DB accesses involved, it mostly depends upon the DB optimizations ("good" tables, indexes, data volume, queries, good concurrency of the DB, ...) and the computation, not on the OS. So without any specific workload, we can not really give recommendations (besides giving FreeBSD a try and working with us if there's problem). > another box to get request specific command strings. This process > could run for almost 20 hours straight and OS still has to be able to > keep in shape. The OS doesn't care about how long a process runs. But if you talk about good responsiveness of the OS while a process uses a lot of memory and CPU, FreeBSD will handle it good (and from what I heard and seen better than Linux, but I don't have numbers at hand). Bye, Alexander. -- Leela: Well, someone's in a good mode. http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 From rahulone at gmail.com Wed Jul 16 10:30:16 2008 From: rahulone at gmail.com (Rahul) Date: Wed Jul 16 10:30:23 2008 Subject: Real Insight on Performance In-Reply-To: <20080716080532.18862vjyr477xls0@webmail.leidinger.net> References: <4804a6670807151328y2cf30363x9b808912286ea1f5@mail.gmail.com> <487D1069.3030801@elischer.org> <4804a6670807151717r54f2889fq6c9c70afebee413d@mail.gmail.com> <20080716080532.18862vjyr477xls0@webmail.leidinger.net> Message-ID: <4804a6670807160330n7d0265ads547297e863dcd347@mail.gmail.com> Thanks Alexander, I need multi-threaded support because I will need to share data between threads. I am not just serving static pages, I am actually performing data manipulation using data that comes in the request as well as what's in memory. The memory could grow to at least 4 GB. Actually, I would like to use at least 10GB for that process (shouldn't be a problem with 16GB RAM on 64Bit OS with Intel Quad Core - Q6600 or Q9450). Although, I do prefer to launch a thread and keep it around for when there is work to do than launching a new process for every request because launching a process is more expensive to begin with and if it has to fetch data from parent process' memory somehow, it only makes things worse. I have used MySQL in the past and it has been able to handle large volume of data without complains (I am talking about 20K rows per second with 25 fields per row). So I was hoping to use MySQL this time around. I guess I should have been more clear in my question, by querying DB and reading flat files, I was meaning to ask about I/O storage and retrieval mechanism, are there any optimizations in FreeBSD regarding I/O? By hinting at possible run time, I meant it would be running 20 hours under heavy work load. So it could possibly get far more than 200 requests per second. At peak time it could get around 2500 per second. So it would be important for OS to not crash or show flaky behavior with memory management with so much memory to maintain throughout the day. Sorry if I seem to be neat-picking but I would like to pick the right OS and stick to as the demand grows and I add more servers. I would like them all to be running same OS as hardware is likely to change more frequently. Thanks. On Wed, Jul 16, 2008 at 2:05 AM, Alexander Leidinger wrote: > Quoting Rahul (from Tue, 15 Jul 2008 20:17:46 -0400): > >> For example: I would like to run a high performance web server that >> can handle up to 200 connections per second and serve them with great >> speed. That's where the concern for multi-threaded support. Depending > > Serving 200 connections per second for static data is not hard. And you > don't need multi-threading for that. Multiple processes is actually better > than multi-threading in this regard, as you don't have to do locking of > filedescriptors in the the webserver over multiple threads. The difference > between linux and FreeBSD should be not big for single CPU/core systems, but > if you increase the number of CPUs/cores this may be different. See the > postgresql (it uses processes, not threads like mysql) graphs in > http://people.freebsd.org/~kris/scaling/7.0%20and%20beyond.pdf to get an > idea about the scaling of more or less independent processes in FreeBSD. > >> on the request, I may have to load (and possibly unload) dynamic >> modules to perform calculations, and if need be, fetch data from >> either DB or flat file. It could involve connecting to a process on > > As soon as you have calculations and/or DB accesses involved, it mostly > depends upon the DB optimizations ("good" tables, indexes, data volume, > queries, good concurrency of the DB, ...) and the computation, not on the > OS. So without any specific workload, we can not really give recommendations > (besides giving FreeBSD a try and working with us if there's problem). > >> another box to get request specific command strings. This process >> could run for almost 20 hours straight and OS still has to be able to >> keep in shape. > > The OS doesn't care about how long a process runs. But if you talk about > good responsiveness of the OS while a process uses a lot of memory and CPU, > FreeBSD will handle it good (and from what I heard and seen better than > Linux, but I don't have numbers at hand). > > Bye, > Alexander. > > -- > Leela: Well, someone's in a good mode. > > http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 > http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 > From astrange at ithinksw.com Thu Jul 17 02:09:53 2008 From: astrange at ithinksw.com (Alexander Strange) Date: Thu Jul 17 02:10:00 2008 Subject: Large number of http connections immediately dropped Message-ID: We're running a rather high-load webserver using FreeBSD 7-RELEASE/ amd64/nginx on an Intel em gigabit connection. Performance is good for our current bandwidth use (about 20Mbit and ~2000 connections/sec at the moment), but a large number of HTTP requests are being immediately dropped before getting to nginx. I see complaints about this with earlier versions of FreeBSD - http://forum.lighttpd.net/topic/171 - but no solutions. Does anyone know what could be the problem, or anything we could do about it? There are several other servers running earlier FreeBSDs on i386 which don't seem to have this problem, but I still haven't ruled out upstream hardware problems or Sandvine yet. On the server: -nginx's error log is full of "accept() failed (53: Software caused connection abort)", sometimes printing three or four at the same time. -messages is full of: Limiting open port RST response from 441 to 200 packets/sec Limiting open port RST response from 488 to 200 packets/sec Limiting open port RST response from 399 to 200 packets/sec Limiting open port RST response from 434 to 200 packets/sec Limiting open port RST response from 308 to 200 packets/sec I'm not sure if that's related or not. -sysctl.conf: net.inet.tcp.tso=1 kern.ipc.somaxconn=10240 kern.ipc.nmbclusters=65536 net.inet.tcp.sendspace=65536 net.inet.tcp.recvspace=65536 net.inet.tcp.rfc1323=1 kern.ipc.maxsockbuf=262144 net.inet.tcp.blackhole=2 net.inet.udp.blackhole=1 net.inet.tcp.msl=7500 net.inet.icmp.icmplim=400 net.inet.tcp.drop_synfin=1 net.inet.tcp.icmp_may_rst=0 net.inet.tcp.fast_finwait2_recycle=1 -netstat -m: 4677/6603/11280 mbufs in use (current/cache/total) 1017/2643/3660/65536 mbuf clusters in use (current/cache/total/max) 1017/1961 mbuf+clusters out of packet secondary zone in use (current/ cache) 9/514/523/12800 4k (page size) jumbo clusters in use (current/cache/ total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 3239K/8992K/12232K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 9204 requests for I/O initiated by sendfile 0 calls to protocol drain routines nginx is not running any accept filters. Locally, after sending an HTTP request, I get a normal connection close, then one RST with sequence 1, then another (possibly more than one) RST with sequence 2. I can post a tcpdump sequence if necessary, after I sanitize some cookies away. From leccine at gmail.com Thu Jul 17 21:04:01 2008 From: leccine at gmail.com (Istvan Szukacs) Date: Thu Jul 17 21:04:07 2008 Subject: Large number of http connections immediately dropped In-Reply-To: References: Message-ID: <487FAF68.6040200@gmail.com> Hi! Something to read: http://people.freebsd.org/~hmp/utilities/satbl/sysctl-net.html I have these in the sysctl.conf kern.ipc.somaxconn=4096 net.inet.tcp.recvspace=78840 net.inet.tcp.sendspace=78840 kern.ipc.shmmax=67108864 kern.ipc.shmmni=200 kern.ipc.shmseg=128 kern.ipc.semmni=70 net.local.stream.sendspace=82320 net.local.stream.recvspace=82320 net.inet.tcp.local_slowstart_flightsize=10 net.inet.tcp.nolocaltimewait=1 net.inet.tcp.hostcache.expire=3900 and the loader.conf kern.maxusers=512 kern.ipc.nmbclusters=32768 kern.ipc.maxsockets=81920 kern.ipc.maxsockbuf=1048576 net.inet.tcp.tcbhashsize=4096 net.inet.tcp.hostcache.hashsize=1024 Regards, Istvan Alexander Strange wrote: > We're running a rather high-load webserver using FreeBSD > 7-RELEASE/amd64/nginx on an Intel em gigabit connection. > Performance is good for our current bandwidth use (about 20Mbit and > ~2000 connections/sec at the moment), but a large number of HTTP > requests are being immediately dropped before getting to nginx. I see > complaints about this with earlier versions of FreeBSD - > http://forum.lighttpd.net/topic/171 - but no solutions. Does anyone > know what could be the problem, or anything we could do about it? > > There are several other servers running earlier FreeBSDs on i386 which > don't seem to have this problem, but I still haven't ruled out > upstream hardware problems or Sandvine yet. > > On the server: > -nginx's error log is full of "accept() failed (53: Software caused > connection abort)", sometimes printing three or four at the same time. > > -messages is full of: > Limiting open port RST response from 441 to 200 packets/sec > Limiting open port RST response from 488 to 200 packets/sec > Limiting open port RST response from 399 to 200 packets/sec > Limiting open port RST response from 434 to 200 packets/sec > Limiting open port RST response from 308 to 200 packets/sec > I'm not sure if that's related or not. > > -sysctl.conf: > > net.inet.tcp.tso=1 > kern.ipc.somaxconn=10240 > kern.ipc.nmbclusters=65536 > net.inet.tcp.sendspace=65536 > net.inet.tcp.recvspace=65536 > net.inet.tcp.rfc1323=1 > kern.ipc.maxsockbuf=262144 > net.inet.tcp.blackhole=2 > net.inet.udp.blackhole=1 > net.inet.tcp.msl=7500 > net.inet.icmp.icmplim=400 > net.inet.tcp.drop_synfin=1 > net.inet.tcp.icmp_may_rst=0 > net.inet.tcp.fast_finwait2_recycle=1 > > -netstat -m: > 4677/6603/11280 mbufs in use (current/cache/total) > 1017/2643/3660/65536 mbuf clusters in use (current/cache/total/max) > 1017/1961 mbuf+clusters out of packet secondary zone in use > (current/cache) > 9/514/523/12800 4k (page size) jumbo clusters in use > (current/cache/total/max) > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 3239K/8992K/12232K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 9204 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > > nginx is not running any accept filters. > > Locally, after sending an HTTP request, I get a normal connection > close, then one RST with sequence 1, then another (possibly more than > one) RST with sequence 2. I can post a tcpdump sequence if necessary, > after I sanitize some cookies away. > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org" From ivoras at freebsd.org Sun Jul 20 18:05:06 2008 From: ivoras at freebsd.org (Ivan Voras) Date: Sun Jul 20 18:05:37 2008 Subject: Large number of http connections immediately dropped In-Reply-To: References: Message-ID: Alexander Strange wrote: > We're running a rather high-load webserver using FreeBSD > 7-RELEASE/amd64/nginx on an Intel em gigabit connection. > Performance is good for our current bandwidth use (about 20Mbit and > ~2000 connections/sec at the moment), but a large number of HTTP > requests are being immediately dropped before getting to nginx. I see > complaints about this with earlier versions of FreeBSD - > http://forum.lighttpd.net/topic/171 - but no solutions. Does anyone know > what could be the problem, or anything we could do about it? > > There are several other servers running earlier FreeBSDs on i386 which > don't seem to have this problem, but I still haven't ruled out upstream > hardware problems or Sandvine yet. > > On the server: > -nginx's error log is full of "accept() failed (53: Software caused > connection abort)", sometimes printing three or four at the same time. > > -messages is full of: > Limiting open port RST response from 441 to 200 packets/sec > Limiting open port RST response from 488 to 200 packets/sec > Limiting open port RST response from 399 to 200 packets/sec > Limiting open port RST response from 434 to 200 packets/sec > Limiting open port RST response from 308 to 200 packets/sec > I'm not sure if that's related or not. It's almost certainly related - in addition to other suggested tuning by Istvan, set net.inet.icmp.icmplim sysctl to something high - for example 2000 in your case. Actually, in your sysctl.conf it's set to 400 - you do know you have to run "/etc/rc.d/sysctl restart" to reaload sysctl.conf? From astrange at ithinksw.com Sun Jul 20 18:25:41 2008 From: astrange at ithinksw.com (Alexander Strange) Date: Sun Jul 20 18:25:48 2008 Subject: Large number of http connections immediately dropped In-Reply-To: <31AFE70B-CE45-42DE-97C7-AFF96383C6E2@chittenden.org> References: <31AFE70B-CE45-42DE-97C7-AFF96383C6E2@chittenden.org> Message-ID: On Jul 17, 2008, at 12:44 PM, Sean Chittenden wrote: >> -messages is full of: >> Limiting open port RST response from 441 to 200 packets/sec >> Limiting open port RST response from 488 to 200 packets/sec >> Limiting open port RST response from 399 to 200 packets/sec >> Limiting open port RST response from 434 to 200 packets/sec >> Limiting open port RST response from 308 to 200 packets/sec >> I'm not sure if that's related or not. > > Likely not, but you want to set net.inet.icmp.icmplim=2000 or > something much higher. ICMP is a good thing and an important part > of TCP. For that much traffic, you need more ICMP packets. > net.inet.tcp.recvspace seems high, you probably only want it to be > 4096 or maybe double that.... unless your traffic is all HTTP > posts. Why don't you want to run with accept filters? Any > firewalls or rate filters in the way? -sc The httpready filter was just off for debugging (in case it solved our problem) - it didn't seem to affect it, so it's back on now. There are a lot of large HTTP posts happening, and we don't seem to be low on memory, so recvspace should be ok. somaxconn is also much higher than necessary, though, so maybe that could be a problem. Anyway, raising icmplim has emptied the system log, but there are still several errors per minute. I don't think any of the netstat -s counters are going up at the same rate, but I'll keep looking at those. And there's no firewalls or packet shapers in front of it. From ivoras at freebsd.org Mon Jul 21 19:53:41 2008 From: ivoras at freebsd.org (Ivan Voras) Date: Mon Jul 21 19:53:48 2008 Subject: Large number of http connections immediately dropped In-Reply-To: References: <31AFE70B-CE45-42DE-97C7-AFF96383C6E2@chittenden.org> Message-ID: Alexander Strange wrote: > And there's no firewalls or packet shapers in front of it. How about on it? Do you run ipfw? From freebsd at sopwith.solgatos.com Wed Jul 23 19:08:12 2008 From: freebsd at sopwith.solgatos.com (Dieter) Date: Wed Jul 23 19:08:23 2008 Subject: disk i/o unfairness with multiple processes Message-ID: <200807231843.SAA09833@sopwith.solgatos.com> I've been seeing unfairness in disk i/o when multiple processes compete for resources. While some unfairness can be tolerated in order to gain overall efficiency, (e.g. avoiding long seeks) there is a limit. I've seen this with various scenarios, with 6.0, 6.2 and 7.0. Here is a simple test case which demonstrates the problem, and should be easy for others to duplicate. AMD64 2 GiB memory 7200 rpm SATA connected to nforce4-ultra FreeBSD 7.0 FFS, soft-updates $ time man de > /dev/null real 0m0.013s user 0m0.011s sys 0m0.001s $ cat 9_GB_file 9_GB_file 9_GB_file 9_GB_file > /dev/null & [1] 84904 $ time man de > /dev/null [1]+ Done cat 9_GB_file 9_GB_file 9_GB_file 9_GB_file > /dev/null real 9m20.508s user 0m1.053s sys 0m44.091s $ systat -vmstat reports that cat is reading at 50-60 MB/s, which is reasonable for this disk. The 9_GB_file and /usr are both on the same disk. Accessing different disks is more likely to give the expected performance. I suspect that some scenarios bottleneck in memory. I certainly expect man to take longer if it is competing for disk i/o, but 9 minutes seems a bit much. The user and sys times are also up significantly, which seems odd? From astrange at ithinksw.com Wed Jul 30 05:38:55 2008 From: astrange at ithinksw.com (Alexander Strange) Date: Wed Jul 30 05:39:02 2008 Subject: Large number of http connections immediately dropped In-Reply-To: References: <31AFE70B-CE45-42DE-97C7-AFF96383C6E2@chittenden.org> Message-ID: On Jul 21, 2008, at 3:53 PM, Ivan Voras wrote: > Alexander Strange wrote: > >> And there's no firewalls or packet shapers in front of it. > > How about on it? Do you run ipfw? No, I wouldn't answer a question so specifically like that. We didn't see this problem after recompiling without SMP support and waiting for a day or two, but that immediately brought the load average up to around 50 and made it much slower, so that's clearly not a solution. It also really doesn't make me look forward to debugging it... (Disabling net.isr.direct and some other things didn't seem to have any effect) From rwatson at FreeBSD.org Thu Jul 31 21:30:28 2008 From: rwatson at FreeBSD.org (Robert Watson) Date: Thu Jul 31 21:30:35 2008 Subject: Large number of http connections immediately dropped In-Reply-To: References: <31AFE70B-CE45-42DE-97C7-AFF96383C6E2@chittenden.org> Message-ID: <20080731220701.Y22038@fledge.watson.org> On Wed, 30 Jul 2008, Alexander Strange wrote: > On Jul 21, 2008, at 3:53 PM, Ivan Voras wrote: > >> Alexander Strange wrote: >> >>> And there's no firewalls or packet shapers in front of it. >> >> How about on it? Do you run ipfw? > > No, I wouldn't answer a question so specifically like that. > > We didn't see this problem after recompiling without SMP support and waiting > for a day or two, but that immediately brought the load average up to around > 50 and made it much slower, so that's clearly not a solution. It also really > doesn't make me look forward to debugging it... > > (Disabling net.isr.direct and some other things didn't seem to have any > effect) Turning off SMP is probably slowing the transaction rate down sufficiently that you're not seeing the problem. The reason to ask the firewall question (ipfw, pf, etc) is that as the rate of TCP connections goes up, and if there are a small number of addresses involved, the reuse rate for TCP/IP port/address tuples becomes very high, which can cause connections to reuse tuples too quickly. Sometimes firewalls are more sensitive to this than the stack -- especially if those firewalls are doing things like randomizing port numbers, TCP sequence numbers, etc, so in the past there have been reports (and bug fixes) along those lines. I may have missed you answering this already, but are there a large number of remote endpoints (unique IP addresses) or a small one? Such problems have come up in the past especially when there is a load balancer or proxy in front, as that reduces what starts out as a large number of hosts to a very small number (exactly one). Robert N M Watson Computer Laboratory University of Cambridge