memstick.img is bloated with 7% 2K blocks of nulls

[Julian H. Stacey]

Bruce Cran wrote:
> On Sat, 12 Feb 2011 01:54:58 +0100
> "Julian H. Stacey" <jhs at berklix.com> wrote:
>
> > 	 -O filesystem-type
> >              Use 1 to specify that a UFS1 format file system be
> > built; use 2 to specify that a UFS2 format file system be built.  The
> > default format is UFS2.
> > If anyone fancies looking deeper, please do :-)
>
> I checked with dumpfs that memstick.img is UFS1.
>
> Also, mounting /dev/md0 confuses the kernel into ultimately panic'ing,
> since /dev/md0a is the proper slice.

I'm suprised it mounts. Not suprised it crashes.

> For the mfsroot.gz file from the
> CD ISOs:
>
> # mdconfig -a -f mfsroot
> md0
> # mount /dev/md0a /mnt
> # ls /mnt
> ls: /mnt: Bad file descriptor
> # cd /mnt
> cd: /mnt: Not a directory
> # vim /mnt
>
> panic: ffs_read: type 0

It's not so obvious but in man mdconfig, there's no "[]" around "-t
type", I read that as "-t something" is mandatory, (though it starts
without for you ... & I suspect -t default is malloc, though manual
doesnt say that, but look what manual says re. malloc ... panic ).

I use "-t vnode", which doesn't crash here on 8.1-RELEASE i686:
	mdconfig -a -t vnode -f FreeBSD-8.2-RC3-amd64-memstick.img
	mount /dev/md0a /mnt
	cd /mnt
	tar cf - . | ( cd /pri/FreeBSD/releases/amd64/ISO-IMAGES/8.2\
		/FreeBSD-8.2-RC3-amd64-memstick && tar xf - )
	dumpfs /dev/md0a # UFS1
	mdconfig -a -t vnode -f FreeBSD-8.2-RC3-amd64-livefs.iso
	mount -t cd9660 /dev/md1 /mnt1
	cd /mnt1
	tar cf - . | ( cd /pri/FreeBSD/releases/amd64/ISO-IMAGES/8.2\
		/FreeBSD-8.2-RC3-amd64-livefs && tar xf - )
	mdconfig -a -t vnode -f FreeBSD-8.2-RC3-amd64-disc1.iso
	xs mount -t cd9660 /dev/md2 /mnt2
	cd /mnt2
	tar cf - . | ( cd /pri/FreeBSD/releases/amd64/ISO-IMAGES/8.2\
		/FreeBSD-8.2-RC3-amd64-disc1 && tar xf - )
	cd / ; umount /mnt2 ; umount /mnt1 ; umount /mnt
	mdconfig -d -u 2 ; mdconfig -d -u 1 ; mdconfig -d -u 0

Analysing sizes:
	ls -l
		 723834880 FreeBSD-8.2-RC3-amd64-disc1.iso
		 348796928 FreeBSD-8.2-RC3-amd64-livefs.iso
		1087467520 FreeBSD-8.2-RC3-amd64-memstick.img
	cd /pri/FreeBSD/releases/amd64/ISO-IMAGES/8.2 ; du -s -k /mnt* *
		988708  /mnt	# memstick.img
		985807  /mnt1	# livefs.iso
		705444  /mnt2	# disc1.iso
		999340  FreeBSD-8.2-RC3-amd64-memstick
		1023032 FreeBSD-8.2-RC3-amd64-livefs
		711640  FreeBSD-8.2-RC3-amd64-disc1
	df /mnt
		Filesystem    Size    Used   Avail Capacity  Mounted on
		/dev/md0a     1.1G    1.0G    -14M   101%    /mnt
	umount /mnt ; tunefs -m 0 /dev/md0a
		tunefs: minimum percentage of free space changes from 8% to 0%
		tunefs: should optimize for space with minfree < 8%
	df /dev/md0a
		Filesystem           Size    Used   Avail Capacity  Mounted on
		/dev/md0a            1.1G    1.0G     73M    93%
	cd /pri/FreeBSD/releases/amd64/ISO-IMAGES/8.2 ; /bin/pwd
	mount | grep /usra
		/dev/ad4s4e on /usra (ufs, NFS exported, local, soft-updates)
	dumpfs /dev/ad4s4e | head -1	# UFS2

	Note du on UFS1 stick (988708) is near the same as tree
	copied by tar to UFS2 hard disc (999340), so its not the
	differerence between UFS1 & UFS2 that wastes the space
	(though good guess & thanks for suggesting it pushed me to
	investigate more :-)

We are distributing 7% null space:
	Mainly because the file system is too big. We probably don't
	need 7% free space on stick just to install or repair,
	though that might be useful (if slow) for demos )

	+ Also each file will have unused bytes at end, (but so will 
	  livefs.iso have padding per file & not near so bloated,
	  albeit probably some other fragment size.  Per my first post:
	> >	2K blocks of nulls.
	> >	FreeBSD-8.2-RC3-i386-disc1.iso: 		   565
	> >	FreeBSD-8.2-RC3-i386-livefs.iso:		   905
	> >	FreeBSD-8.2-RC3-i386-memstick.img:		 34521

As a sample test of what makefs produces:
	makefs /tmp/bla /usr/share
		Extent size set to 8192 block size 8192, fragment size 1024
	du -s -k  /usr/share	#     54 108   
	ls -l /tmp/bla		# 54 927 360 
	df /tmp/bla
		Filesystem     Size    Used   Avail Capacity  Mounted on
		/dev/ad0s2e    520M     55M    423M    12%    /tmp
		# PS Note df lies 520M - some bug as not a node ?
	mdconfig -a -t vnode -f /tmp/bla	# md2
	df /dev/md2*
		Filesystem        1K-blocks      Used    Avail Capacity  
		/dev/md2              51175     48723    -1642   103%    
	tunefs -m 2 /dev/md2
		minimum percentage of free space changes from 8% to 2%
	df /dev/md2*
		Filesystem        1K-blocks      Used    Avail Capacity  
		/dev/md2              51175     48723     1429    97%    
	Conclusion: makefs leaves 2+3=5% here.

memstick.img is created by /usr/src/release/scripts/make-memstick.sh
which (as above test) at present calls makefs without any tuning parameters
Maybe t should have some parameters ? eg:
	makefs -m 1030000000 -b 1% ${tempfile} ${1}

Julian
-- 
Julian Stacey, BSD Unix Linux C Sys Eng Consultants Munich http://berklix.com
 Mail plain text;  Not quoted-printable, Not HTML, Not base 64.
 Reply below text sections not at top, to avoid breaking cumulative context.