Show stopper for large disks with 5.4-RELEASE

Pierre DAVID Pierre.David at crc.u-strasbg.fr
Wed Jun 8 16:06:07 GMT 2005


On Wed, Jun 08, 2005 at 08:52:39PM +0800, Xin LI wrote:
> 
> Would you please provide a bit more of information so we can investigate
> what was happening, like:
> 
> 	- first few lines dumpfs(8) output from your storage filesystem
> 	- df -i on your storag filesystem
> 	- dmesg.boot from your /var/run
> 
> I think the delay is too long.  Taking a snapshot on a large volume
> is slow, but should not be that slow :-)  We run similiar mail server
> at company, but with ~10x of users and 2x storage (divided into two
> RAID groups).  Hope I would be able to provide some help.
> 

The server with the MSA 500 is not available at this time (there is
another OS on it, sigh...).

We reproduced the problem with the planned backup server, with ~2TB
on a DAS storage (HP MSA 20).

    seth ~ # newfs -O 2 -U /dev/da1
    ...
    seth ~ # mount /dev/da1 /mail
    seth ~ # df -g
    Filesystem  1G-blocks Used Avail Capacity  Mounted on
    /dev/da0s1a         0    0     0    65%    /
    devfs               0    0     0   100%    /dev
    /dev/da0s1g         9    0     8     0%    /local
    /dev/da0s1f         0    0     0     0%    /tmp
    /dev/da0s1d         7    0     6     9%    /usr
    /dev/da0s1e         9    0     8     0%    /var
    /dev/da1         1983    1  1823     0%    /mail

And the problem is worse:

    seth ~ # time mksnap_ffs /mail /mail/.snap/snap1

    real    69m18.083s
    user    0m0.000s
    sys     0m21.121s

First lines of "dumpfs /dev/da1" are:
	magic	19540119 (UFS2)	time	Wed Jun  8 17:03:14 2005
	superblock location	65536	id	[ 42a6f77a a2574d5f ]
	ncg	11413	size	1073741823	blocks	1039959213
	bsize	16384	shift	14	mask	0xffffc000
	fsize	2048	shift	11	mask	0xfffff800
	frag	8	shift	3	fsbtodb	2
	minfree	8%	optim	time	symlinklen 120
	maxbsize 16384	maxbpg	2048	maxcontig 8	contigsumsize 8
	nbfree	129917894	ndir	2	nifree	268798970	nffree	27
	bpg	11761	fpg	94088	ipg	23552
	nindir	2048	inopb	64	maxfilesize	140806241583103
	sbsize	2048	cgsize	16384	csaddr	3000	cssize	184320
	sblkno	40	cblkno	48	iblkno	56	dblkno	3000
	cgrotor	8470	fmod	0	ronly	0	clean	0
	avgfpdir 64	avgfilesize 16384
	flags	soft-updates 
	fsmnt	/mail
	volname		swuid	0

The complete "dumpfs /dev/da1" output is available on:
	ftp://ftp8.fr.freebsd.org/pub/tmp/dumpfs


Here is the "df -i" output:

	seth mail # df -i
	Filesystem  1K-blocks     Used      Avail Capacity iused     ifree %iused 
	  Mounted on
	/dev/da0s1a    1012974  607370     324568    65%    1399    139911    1%   /
	devfs                1       1          0   100%       0         0  100% 
	  /dev
	/dev/da0s1g   10063838    4142    9254590     0%       6   1318904    0% 
	  /local
	/dev/da0s1f    1012974      14     931924     0%       8    141302    0% 
	  /tmp
	/dev/da0s1d    8122126  682654    6789702     9%  116342    943496   11% 
	  /usr
	/dev/da0s1e   10154158   25732    9316094     0%     258   1318652    0% 
	  /var
	/dev/da1    2079918426 1232068 1912292884     0%       4 268798970    0% 
      /mail


The dmesg.boot is attached to this mail.

Just in case, we also included the kernel configuration file to
this mail.


Thanks in advance for your help,


Philippe Pegon & Pierre David
-------------- next part --------------
Copyright (c) 1992-2005 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD 5.4-STABLE #2: Thu May 19 21:43:04 CEST 2005
    root at crc.u-strasbg.fr:/usr/obj/usr/src/sys/SETH
ACPI APIC Table: <COMPAQ 00000083>
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2790.96-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf29  Stepping = 9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Hyperthreading: 2 logical CPUs
real memory  = 1073717248 (1023 MB)
avail memory = 1041145856 (992 MB)
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-15 on motherboard
ioapic1 <Version 1.1> irqs 16-31 on motherboard
ioapic2 <Version 1.1> irqs 32-47 on motherboard
ioapic3 <Version 1.1> irqs 48-63 on motherboard
npx0: <math processor> on motherboard
npx0: INT 16 interface
acpi0: <COMPAQ P29> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 1000
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x920-0x923 on acpi0
cpu0: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <display, VGA> at device 3.0 (no driver attached)
pci0: <base peripheral> at device 4.0 (no driver attached)
pci0: <base peripheral> at device 4.2 (no driver attached)
isab0: <PCI-ISA bridge> at device 15.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <ServerWorks CSB5 UDMA100 controller> port 0x2000-0x200f,0x376,0x170-0x177,0x3f6,0x1f0-0x1f7 at device 15.1 on pci0
ata0: channel #0 on atapci0
ata1: channel #1 on atapci0
ohci0: <OHCI (generic) USB controller> mem 0xf5ef0000-0xf5ef0fff irq 7 at device 15.2 on pci0
usb0: OHCI version 1.0, legacy support
usb0: SMM does not respond, resetting
usb0: <OHCI (generic) USB controller> on ohci0
usb0: USB revision 1.0
uhub0: (0x1166) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 4 ports with 4 removable, self powered
pcib1: <ACPI Host-PCI bridge> on acpi0
pci1: <ACPI PCI bus> on pcib1
ciss0: <Compaq Smart Array 5i> port 0x3000-0x30ff mem 0xf7bf0000-0xf7bf3fff,0xf7cc0000-0xf7cfffff irq 30 at device 3.0 on pci1
pcib2: <ACPI Host-PCI bridge> on acpi0
pci2: <ACPI PCI bus> on pcib2
bge0: <Broadcom BCM5703 Gigabit Ethernet, ASIC rev. 0x1002> mem 0xf7df0000-0xf7dfffff irq 29 at device 1.0 on pci2
miibus0: <MII bus> on bge0
brgphy0: <BCM5703 10/100/1000baseTX PHY> on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto
bge0: Ethernet address: 00:11:0a:9a:ce:7d
bge1: <Broadcom BCM5703 Gigabit Ethernet, ASIC rev. 0x1002> mem 0xf7de0000-0xf7deffff irq 31 at device 2.0 on pci2
miibus1: <MII bus> on bge1
brgphy1: <BCM5703 10/100/1000baseTX PHY> on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto
bge1: Ethernet address: 00:11:0a:9a:ce:7c
pcib3: <ACPI Host-PCI bridge> on acpi0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI Host-PCI bridge> on acpi0
pci6: <ACPI PCI bus> on pcib4
ciss1: <HP Smart Array 642> port 0x4000-0x40ff mem 0xf7f80000-0xf7fbffff,0xf7ff0000-0xf7ff1fff irq 24 at device 1.0 on pci6
pci6: <base peripheral, PCI hot-plug controller> at device 30.0 (no driver attached)
acpi_tz0: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x64,0x60 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: model IntelliMouse, device ID 3
sio0: <Standard PC COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
fdc0: <floppy drive controller (FDE)> port 0x3f2-0x3f5 irq 6 drq 2 on acpi0
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
orm0: <ISA Option ROMs> at iomem 0xee000-0xeffff,0xd0000-0xd17ff,0xcc000-0xcffff,0xc8000-0xcbfff,0xc0000-0xc7fff on isa0
pmtimer0 on isa0
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 2790962048 Hz quality 800
Timecounters tick every 1.000 msec
acd0: CDROM <COMPAQ CD-ROM SN-124/N104> at ata0-master PIO4
da1 at ciss1 bus 1 target 0 lun 0
da1: <COMPAQ RAID ADGVOLUME OK> Fixed Direct Access SCSI-5 device 
da1: 135.168MB/s transfers
da1: 2097151MB (4294967295 512 byte sectors: 255H 32S/T 65535C)
da2 at ciss1 bus 1 target 1 lun 0
da2: <COMPAQ RAID ADGVOLUME OK> Fixed Direct Access SCSI-5 device 
da2: 135.168MB/s transfers
da2: 287277MB (588344688 512 byte sectors: 255H 32S/T 65535C)
da0 at ciss0 bus 0 target 0 lun 0
da0: <COMPAQ RAID 1  VOLUME OK> Fixed Direct Access SCSI-0 device 
da0: 135.168MB/s transfers
da0: 34727MB (71122560 512 byte sectors: 255H 32S/T 8716C)
Mounting root from ufs:/dev/da0s1a
-------------- next part --------------
#
# SETH
#

machine		i386
#cpu		I486_CPU
#cpu		I586_CPU
cpu		I686_CPU
ident		SETH

# To statically compile in device wiring instead of /boot/device.hints
#hints		"GENERIC.hints"		# Default places to look for devices.

options 	SCHED_4BSD		# 4BSD scheduler
options 	INET			# InterNETworking
options 	INET6			# IPv6 communications protocols
options 	FFS			# Berkeley Fast Filesystem
options 	SOFTUPDATES		# Enable FFS soft updates support
options 	UFS_ACL			# Support for access control lists
options 	UFS_DIRHASH		# Improve performance on big directories
options 	MD_ROOT			# MD is a potential root device
options 	NFSCLIENT		# Network Filesystem Client
options 	NFSSERVER		# Network Filesystem Server
options 	NFS_ROOT		# NFS usable as /, requires NFSCLIENT
options 	MSDOSFS			# MSDOS Filesystem
options 	CD9660			# ISO 9660 Filesystem
options 	PROCFS			# Process filesystem (requires PSEUDOFS)
options 	PSEUDOFS		# Pseudo-filesystem framework
options 	GEOM_GPT		# GUID Partition Tables.
options 	COMPAT_43		# Compatible with BSD 4.3 [KEEP THIS!]
options 	COMPAT_FREEBSD4		# Compatible with FreeBSD4
options 	SCSI_DELAY=15000	# Delay (in ms) before probing SCSI
options 	KTRACE			# ktrace(1) support
options 	SYSVSHM			# SYSV-style shared memory
options 	SYSVMSG			# SYSV-style message queues
options 	SYSVSEM			# SYSV-style semaphores
options 	_KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options 	KBD_INSTALL_CDEV	# install a CDEV entry in /dev
options 	AHC_REG_PRETTY_PRINT	# Print register bitfields in debug
					# output.  Adds ~128k to driver.
options 	AHD_REG_PRETTY_PRINT	# Print register bitfields in debug
					# output.  Adds ~215k to driver.
options 	ADAPTIVE_GIANT		# Giant mutex is adaptive.

device		apic		# I/O APIC

# Bus support.  Do not remove isa, even if you have no isa slots
device		isa
device		eisa
device		pci

# Floppy drives
device		fdc

# ATA and ATAPI devices
device		ata
device		atadisk		# ATA disk drives
device		ataraid		# ATA RAID drives
device		atapicd		# ATAPI CDROM drives
device		atapifd		# ATAPI floppy drives
device		atapist		# ATAPI tape drives
options 	ATA_STATIC_ID	# Static device numbering

# SCSI Controllers
device		ahb		# EISA AHA1742 family
device		ahc		# AHA2940 and onboard AIC7xxx devices
device		ahd		# AHA39320/29320 and onboard AIC79xx devices
device		amd		# AMD 53C974 (Tekram DC-390(T))
device		isp		# Qlogic family
device		mpt		# LSI-Logic MPT-Fusion
#device		ncr		# NCR/Symbios Logic
device		sym		# NCR/Symbios Logic (newer chipsets + those of `ncr')
device		trm		# Tekram DC395U/UW/F DC315U adapters

device		adv		# Advansys SCSI adapters
device		adw		# Advansys wide SCSI adapters
device		aha		# Adaptec 154x SCSI adapters
device		aic		# Adaptec 15[012]x SCSI adapters, AIC-6[23]60.
device		bt		# Buslogic/Mylex MultiMaster SCSI adapters

device		ncv		# NCR 53C500
device		nsp		# Workbit Ninja SCSI-3
device		stg		# TMC 18C30/18C50

# SCSI peripherals
device		scbus		# SCSI bus (required for SCSI)
device		ch		# SCSI media changers
device		da		# Direct Access (disks)
device		sa		# Sequential Access (tape etc)
device		cd		# CD
device		pass		# Passthrough device (direct SCSI access)
device		ses		# SCSI Environmental Services (and SAF-TE)

# RAID controllers interfaced to the SCSI subsystem
device		amr		# AMI MegaRAID
device		asr		# DPT SmartRAID V, VI and Adaptec SCSI RAID
device		ciss		# Compaq Smart RAID 5*
device		dpt		# DPT Smartcache III, IV - See NOTES for options
device		hptmv		# Highpoint RocketRAID 182x
device		iir		# Intel Integrated RAID
device		ips		# IBM (Adaptec) ServeRAID
device		mly		# Mylex AcceleRAID/eXtremeRAID
device		twa		# 3ware 9000 series PATA/SATA RAID

# RAID controllers
device		aac		# Adaptec FSA RAID
device		aacp		# SCSI passthrough for aac (requires CAM)
device		ida		# Compaq Smart RAID
device		mlx		# Mylex DAC960 family
device		pst		# Promise Supertrak SX6000
device		twe		# 3ware ATA RAID

# atkbdc0 controls both the keyboard and the PS/2 mouse
device		atkbdc		# AT keyboard controller
device		atkbd		# AT keyboard
device		psm		# PS/2 mouse

device		vga		# VGA video card driver

device		splash		# Splash screen and screen saver support

# syscons is the default console driver, resembling an SCO console
device		sc

# Enable this for the pcvt (VT220 compatible) console driver
#device		vt
#options 	XSERVER		# support for X server on a vt console
#options 	FAT_CURSOR	# start with block cursor

device		agp		# support several AGP chipsets

# Floating point support - do not disable.
device		npx

# Power management support (see NOTES for more options)
#device		apm
# Add suspend/resume support for the i8254.
device		pmtimer

# PCCARD (PCMCIA) support
# PCMCIA and cardbus bridge support
device		cbb		# cardbus (yenta) bridge
device		pccard		# PC Card (16-bit) bus
device		cardbus		# CardBus (32-bit) bus

# Serial (COM) ports
device		sio		# 8250, 16[45]50 based serial ports

# Parallel port
device		ppc
device		ppbus		# Parallel port bus (required)
device		lpt		# Printer
device		plip		# TCP/IP over parallel
device		ppi		# Parallel port interface device
#device		vpo		# Requires scbus and da

# If you've got a "dumb" serial or parallel PCI card that is
# supported by the puc(4) glue driver, uncomment the following
# line to enable it (connects to the sio and/or ppc drivers):
#device         puc

# PCI Ethernet NICs.
device		de		# DEC/Intel DC21x4x (``Tulip'')
device		em		# Intel PRO/1000 adapter Gigabit Ethernet Card
device		ixgb		# Intel PRO/10GbE Ethernet Card
device		txp		# 3Com 3cR990 (``Typhoon'')
device		vx		# 3Com 3c590, 3c595 (``Vortex'')

# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device		miibus		# MII bus support
device		bfe		# Broadcom BCM440x 10/100 Ethernet
device		bge		# Broadcom BCM570xx Gigabit Ethernet
device		dc		# DEC/Intel 21143 and various workalikes
device		fxp		# Intel EtherExpress PRO/100B (82557, 82558)
device		lge		# Level 1 LXT1001 gigabit ethernet
device		nge		# NatSemi DP83820 gigabit ethernet
device		pcn		# AMD Am79C97x PCI 10/100 (precedence over 'lnc')
device		re		# RealTek 8139C+/8169/8169S/8110S
device		rl		# RealTek 8129/8139
device		sf		# Adaptec AIC-6915 (``Starfire'')
device		sis		# Silicon Integrated Systems SiS 900/SiS 7016
device		sk		# SysKonnect SK-984x & SK-982x gigabit Ethernet
device		ste		# Sundance ST201 (D-Link DFE-550TX)
device		ti		# Alteon Networks Tigon I/II gigabit Ethernet
device		tl		# Texas Instruments ThunderLAN
device		tx		# SMC EtherPower II (83c170 ``EPIC'')
device		vge		# VIA VT612x gigabit ethernet
device		vr		# VIA Rhine, Rhine II
device		wb		# Winbond W89C840F
device		xl		# 3Com 3c90x (``Boomerang'', ``Cyclone'')

# ISA Ethernet NICs.  pccard NICs included.
device		cs		# Crystal Semiconductor CS89x0 NIC
# 'device ed' requires 'device miibus'
device		ed		# NE[12]000, SMC Ultra, 3c503, DS8390 cards
device		ex		# Intel EtherExpress Pro/10 and Pro/10+
device		ep		# Etherlink III based cards
device		fe		# Fujitsu MB8696x based cards
device		ie		# EtherExpress 8/16, 3C507, StarLAN 10 etc.
device		lnc		# NE2100, NE32-VL Lance Ethernet cards
device		sn		# SMC's 9000 series of Ethernet chips
device		xe		# Xircom pccard Ethernet

# ISA devices that use the old ISA shims
#device		le

# Wireless NIC cards
device		wlan		# 802.11 support
device		an		# Aironet 4500/4800 802.11 wireless NICs.
device		awi		# BayStack 660 and others
device		wi		# WaveLAN/Intersil/Symbol 802.11 wireless NICs.
#device		wl		# Older non 802.11 Wavelan wireless NIC.

# Pseudo devices.
device		loop		# Network loopback
device		mem		# Memory and kernel memory devices
device		io		# I/O device
device		random		# Entropy device
device		ether		# Ethernet support
device		sl		# Kernel SLIP
device		ppp		# Kernel PPP
device		tun		# Packet tunnel.
device		pty		# Pseudo-ttys (telnet etc)
device		md		# Memory "disks"
device		gif		# IPv6 and IPv4 tunneling
device		faith		# IPv6-to-IPv4 relaying (translation)

# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
device		bpf		# Berkeley packet filter

# USB support
device		uhci		# UHCI PCI->USB interface
device		ohci		# OHCI PCI->USB interface
device		usb		# USB Bus (required)
#device		udbp		# USB Double Bulk Pipe devices
device		ugen		# Generic
device		uhid		# "Human Interface Devices"
device		ukbd		# Keyboard
device		ulpt		# Printer
device		umass		# Disks/Mass storage - Requires scbus and da
device		ums		# Mouse
device		urio		# Diamond Rio 500 MP3 player
device		uscanner	# Scanners
# USB Ethernet, requires mii
device		aue		# ADMtek USB Ethernet
device		axe		# ASIX Electronics USB Ethernet
device		cue		# CATC USB Ethernet
device		kue		# Kawasaki LSI USB Ethernet
device		rue		# RealTek RTL8150 USB Ethernet

# FireWire support
device		firewire	# FireWire bus code
device		sbp		# SCSI over FireWire (Requires scbus and da)
device		fwe		# Ethernet over FireWire (non-standard!)

# altq(9). Enable the base part of the hooks with the ALTQ option.
# Individual disciplines must be built into the base system and can not be
# loaded as modules at this point. In order to build a SMP kernel you must
# also have the ALTQ_NOPCC option.
options         ALTQ
options         ALTQ_CBQ        # Class Bases Queueing
options         ALTQ_RED        # Random Early Drop
options         ALTQ_RIO        # RED In/Out
options         ALTQ_HFSC       # Hierarchical Packet Scheduler
options         ALTQ_CDNR       # Traffic conditioner
options         ALTQ_PRIQ       # Priority Queueing
options         ALTQ_NOPCC      # Required for SMP build
#options         ALTQ_DEBUG

# Change syscons colors
options         SC_KERNEL_CONS_ATTR="(FG_RED|BG_BLACK)"
options         SC_KERNEL_CONS_REV_ATTR="(FG_BLACK|BG_RED)"

# ajout mano
device		vlan
device		pf
device		pflog
device		pfsync
device		carp
options		HZ=1000

# debug
#options        KDB                     # Enable kernel debugger support.
#options        DDB                     # Support DDB.
#options        GDB                     # Support remote GDB.
#options        INVARIANTS              # Enable calls of extra sanity checking
#options        INVARIANT_SUPPORT       # Extra sanity checks of internal structures, required by INVARIANTS
#options        WITNESS                 # Enable checks to detect deadlocks and cycles
#options        WITNESS_SKIPSPIN        # Don't run witness on spinlocks for speed


More information about the freebsd-stable mailing list