Little research how rm -rf and tar kill server
Artem Kuchin
artem at artem.ru
Mon Mar 30 15:45:40 UTC 2015
30.03.2015 18:05, Mark Felder пишет:
>
>
> On Sat, Mar 28, 2015, at 15:22, Artem Kuchin wrote:
>>
>> So, questions and thoughts: 1) Why i had no problem such this in
>> fbsd 9? I think the reason for the problem is in kernel, not in
>> hardware or mariadb+nginx because server load did not increase at
>> all, even decreases a little.
>
> This is only anecdotal until you run the exact same OS version and
> load on the old hardware which had a different motherboard and
> (probably) disk controller. Can you provide us with any further
> hardware details between the two systems?
My point was that the change what causes it was in freebsd kernel or ufs
driver or sata
driver or somewhere there, not the hardware.I even left the Kernel
config file the same.
I cannot provide exact hardware and dmesg for the old server (it is down
and not available), but it was
Intel Xeon E3-1245 /Quad Core/
16 GB DDR3 ECC RAM
2x 3 TB SATA III Enterprise HDD (seagate ST33000650NS)
New server
kernel: Copyright (c) 1992-2015 The FreeBSD Project.
kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992,
1993, 1994
kernel: The Regents of the University of California. All rights reserved.
kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
kernel: FreeBSD 10.1-STABLE #0 r279278: Wed Feb 25 15:17:48 MSK 2015
kernel: FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032)
20140512
kernel: CPU: Intel(R) Xeon(R) CPU E3-1270 v3 @ 3.50GHz (3500.07-MHz
K8-class CPU)
kernel: Origin="GenuineIntel" Id=0x306c3 Family=0x6 Model=0x3c Stepping=3
kernel:
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
kernel:
Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,<b11>,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AE
kernel: AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
kernel: AMD Features2=0x21<LAHF,ABM>
kernel: Structured Extended
Features=0x2fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM>
kernel: XSAVE Features=0x1<XSAVEOPT>
kernel: VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
kernel: TSC: P-state invariant, performance statistics
kernel: real memory = 34359738368 (32768 MB)
kernel: avail memory = 33271001088 (31729 MB)
kernel: Event timer "LAPIC" quality 600
kernel: ACPI APIC Table: <SUPERM SMCI--MB>
kernel: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
kernel: FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 SMT threads
kernel: cpu0 (BSP): APIC ID: 0
kernel: cpu1 (AP): APIC ID: 1
kernel: cpu2 (AP): APIC ID: 2
kernel: cpu3 (AP): APIC ID: 3
kernel: cpu4 (AP): APIC ID: 4
kernel: cpu5 (AP): APIC ID: 5
kernel: cpu6 (AP): APIC ID: 6
kernel: cpu7 (AP): APIC ID: 7
kernel: ioapic0 <Version 2.0> irqs 0-23 on motherboard
kernel: random: <Software, Yarrow> initialized
kernel: module_register_init: MOD_LOAD (vesa, 0xffffffff80b6a090, 0)
error 19
kernel: kbd1 at kbdmux0
kernel: module_register_init: MOD_LOAD (accf_data, 0xffffffff807e9af0,
0xffffffff8155d0c8) error 17
kernel: acpi0: <SUPERM SMCI--MB> on motherboard
kernel: acpi0: Power Button (fixed)
kernel: cpu0: <ACPI CPU> on acpi0
kernel: cpu1: <ACPI CPU> on acpi0
kernel: cpu2: <ACPI CPU> on acpi0
kernel: cpu3: <ACPI CPU> on acpi0
kernel: cpu4: <ACPI CPU> on acpi0
kernel: cpu5: <ACPI CPU> on acpi0
kernel: cpu6: <ACPI CPU> on acpi0
kernel: cpu7: <ACPI CPU> on acpi0
kernel: hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff
on acpi0
kernel: Timecounter "HPET" frequency 14318180 Hz quality 950
kernel: Event timer "HPET" frequency 14318180 Hz quality 550
kernel: atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
kernel: Event timer "RTC" frequency 32768 Hz quality 0
kernel: attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
kernel: Timecounter "i8254" frequency 1193182 Hz quality 0
kernel: Event timer "i8254" frequency 1193182 Hz quality 100
kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
kernel: acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on
acpi0
kernel: pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
kernel: pci0: <ACPI PCI bus> on pcib0
kernel: xhci0: <Intel Lynx Point USB 3.0 controller> mem
0xf7220000-0xf722ffff irq 16 at device 20.0 on pci0
kernel: xhci0: 32 bytes context size, 64-bit DMA
kernel: xhci0: Port routing mask set to 0xffffffff
kernel: usbus0 on xhci0
kernel: em0: <Intel(R) PRO/1000 Network Connection 7.4.2> port
0xf020-0xf03f mem 0xf7200000-0xf721ffff,0xf7235000-0xf7235fff irq 20 at
device
25.0 on pci0
kernel: em0: Using an MSI interrupt
kernel: em0: Ethernet address: 00:25:90:47:42:57
kernel: ehci0: <Intel Lynx Point USB 2.0 controller USB-B> mem
0xf7234000-0xf72343ff irq 16 at device 26.0 on pci0
kernel: usbus1: EHCI version 1.0
kernel: usbus1 on ehci0
kernel: pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
kernel: pci1: <ACPI PCI bus> on pcib1
kernel: pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci1
kernel: pci2: <ACPI PCI bus> on pcib2
kernel: vgapci0: <VGA-compatible display> port 0xe000-0xe07f mem
0xf6000000-0xf6ffffff,0xf7000000-0xf701ffff irq 16 at device 0.0 on pci2
kernel: vgapci0: Boot video device
kernel: pcib3: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0
kernel: pci3: <ACPI PCI bus> on pcib3
kernel: igb0: <Intel(R) PRO/1000 Network Connection version - 2.4.0>
port 0xd000-0xd01f mem 0xf7100000-0xf717ffff,0xf7180000-0xf7183fff irq 17 at
device 0.0 on pci3
kernel: igb0: Using MSIX interrupts with 5 vectors
kernel: igb0: Ethernet address: 00:25:90:47:42:56
kernel: igb0: Bound queue 0 to cpu 0
kernel: igb0: Bound queue 1 to cpu 1
kernel: igb0: Bound queue 2 to cpu 2
kernel: igb0: Bound queue 3 to cpu 3
kernel: ehci1: <Intel Lynx Point USB 2.0 controller USB-A> mem
0xf7233000-0xf72333ff irq 23 at device 29.0 on pci0
kernel: usbus2: EHCI version 1.0
kernel: usbus2 on ehci1
kernel: isab0: <PCI-ISA bridge> at device 31.0 on pci0
kernel: isa0: <ISA bus> on isab0
kernel: ahci0: <Intel Lynx Point AHCI SATA controller> port
0xf070-0xf077,0xf060-0xf063,0xf050-0xf057,0xf040-0xf043,0xf000-0xf01f mem
0xf7232000-0xf72327ff irq 19 at device 31.2 on pci0
kernel: ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
kernel: ahcich0: <AHCI channel> at channel 0 on ahci0
kernel: ahcich1: <AHCI channel> at channel 1 on ahci0
kernel: ahcich2: <AHCI channel> at channel 2 on ahci0
kernel: ahcich3: <AHCI channel> at channel 3 on ahci0
kernel: ahcich4: <AHCI channel> at channel 4 on ahci0
kernel: ahcich5: <AHCI channel> at channel 5 on ahci0
kernel: ahciem0: <AHCI enclosure management bridge> on ahci0
kernel: acpi_button0: <Sleep Button> on acpi0
kernel: acpi_button1: <Power Button> on acpi0
kernel: acpi_tz0: <Thermal Zone> on acpi0
kernel: acpi_tz1: <Thermal Zone> on acpi0
kernel: uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10
on acpi0
kernel: uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
kernel: orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff
on isa0
kernel: sc0: <System console> at flags 0x100 on isa0
kernel: sc0: CGA <16 virtual consoles, flags=0x300>
kernel: vga0: <Generic ISA VGA> at port 0x3d0-0x3db iomem
0xb8000-0xbffff on isa0
kernel: ppc0: cannot reserve I/O port range
kernel: est0: <Enhanced SpeedStep Frequency Control> on cpu0
kernel: est1: <Enhanced SpeedStep Frequency Control> on cpu1
kernel: est2: <Enhanced SpeedStep Frequency Control> on cpu2
kernel: est3: <Enhanced SpeedStep Frequency Control> on cpu3
kernel: est4: <Enhanced SpeedStep Frequency Control> on cpu4
kernel: est5: <Enhanced SpeedStep Frequency Control> on cpu5
kernel: est6: <Enhanced SpeedStep Frequency Control> on cpu6
kernel: est7: <Enhanced SpeedStep Frequency Control> on cpu7
kernel: random: unblocking device.
kernel: usbus0: 5.0Gbps Super Speed USB v3.0
kernel: Timecounters tick every 1.000 msec
kernel: ipfw2 (+ipv6) initialized, divert enabled, nat loadable, default
to accept, logging disabled
kernel: DUMMYNET 0 with IPv6 initialized (100409)
kernel: load_dn_sched dn_sched FIFO loaded
kernel: load_dn_sched dn_sched PRIO loaded
kernel: load_dn_sched dn_sched QFQ loaded
kernel: load_dn_sched dn_sched RR loaded
kernel: load_dn_sched dn_sched WF2Q+ loaded
kernel: usbus1: 480Mbps High Speed USB v2.0
kernel: usbus2: 480Mbps High Speed USB v2.0
kernel: ugen1.1: <Intel> at usbus1
kernel: uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1>
on usbus1
kernel: ugen0.1: <0x8086> at usbus0
kernel: ugen2.1: <Intel> at usbus2
kernel: uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1>
on usbus2
kernel: uhub2: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1>
on usbus0
kernel: ses0 at ahciem0 bus 0 scbus6 target 0 lun 0
kernel: ses0: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device
kernel: ses0: SEMB SES Device
kernel: ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
kernel: ada0: <TOSHIBA DT01ACA300 MX6OABB0> ATA-8 SATA 3.x device
kernel: ada0: Serial Number 64O4281GS
kernel: ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
kernel: ada0: Command Queueing enabled
kernel: ada0: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
kernel: ada0: Previously was known as ad4
kernel: ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
kernel: ada1: <TOSHIBA DT01ACA300 MX6OABB0> ATA-8 SATA 3.x device
kernel: ada1: Serial Number 64O43WZGS
kernel: ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
kernel: ada1: Command Queueing enabled
kernel: ada1: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
kernel: ada1: Previously was known as ad6
kernel: SMP: AP CPU #1 Launched!
kernel: SMP: AP CPU #4 Launched!
kernel: SMP: AP CPU #2 Launched!
kernel: SMP: AP CPU #3 Launched!
kernel: SMP: AP CPU #6 Launched!
kernel: SMP: AP CPU #7 Launched!
kernel: SMP: AP CPU #5 Launched!
kernel: Timecounter "TSC-low" frequency 1750036200 Hz quality 1000
kernel: GEOM_MIRROR: Device mirror/boot launched (2/2).
kernel: GEOM_MIRROR: Device mirror/swap launched (2/2).
kernel: GEOM_MIRROR: Device mirror/root launched (2/2).
kernel: Root mount waiting for: usbus2 usbus1 usbus0
kernel: uhub2: 17 ports with 17 removable, self powered
kernel: uhub0: 2 ports with 2 removable, self powered
kernel: uhub1: 2 ports with 2 removable, self powered
kernel: Root mount waiting for: usbus2 usbus1 usbus0
kernel: ugen0.2: <vendor 0x0000> at usbus0
kernel: uhub3: <vendor 0x0000 product 0x0001, class 9/0, rev 2.00/0.00,
addr 1> on usbus0
kernel: uhub3: 4 ports with 3 removable, self powered
kernel: ugen1.2: <vendor 0x8087> at usbus1
kernel: uhub4: <vendor 0x8087 product 0x8008, class 9/0, rev 2.00/0.05,
addr 2> on usbus1
kernel: ugen2.2: <vendor 0x8087> at usbus2
kernel: uhub5: <vendor 0x8087 product 0x8000, class 9/0, rev 2.00/0.05,
addr 2> on usbus2
kernel: uhub4: 4 ports with 4 removable, self powered
kernel: uhub5: 6 ports with 6 removable, self powered
kernel: ugen0.3: <vendor 0x0557> at usbus0
kernel: ukbd0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00,
addr 2> on usbus0
kernel: kbd0 at ukbd0
kernel: ums0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00,
addr 2> on usbus0
kernel: ums0: 3 buttons and [Z] coordinates ID=0
kernel: Trying to mount root from ufs:/dev/mirror/root
[rw,async,groupquota,noatime]...
>
>
>> 2) I consider it a sever bug, because even normal used (and i have
>> plenty of them using ssh) can eventually do rm -rf and kill all
>> sites. Which means there are must be some way to limit io usage per
>> user
>>
>
> ZFS has far superior disk IO scheduling. You may have more success
> with that. Unfortunately I'm not aware of a way to limit user/process
> disk IO on FreeBSD.
I hesitatated migrating to ZFS because "don't fix it if it ain't broken".
And it was not broken.. until now. But now i cannot migrate to ZFS -
server is too busy
and actually i am afraid something bad will come up with ZFS eventually
too.
Well, if not limit, then maybe just monitor and kill user processes
that create problems.
But the problem is how to determine which processes overuse the io
system. I have
very little clue how to collect such information.
Artem
More information about the freebsd-fs
mailing list