FreeBSD-10.3-RELEASE-i386-memstick.img installer changes contents of USB flash drive

[Ian Smith]

On Sat, 1 Oct 2016 11:39:35 -0600, Warren Block wrote:
 > On Sun, 2 Oct 2016, Ian Smith wrote:
 > 
 > > > A secondary copy of the partition table is saved at the end of a
 > > > GPT-partitioned drive. This is a problem for GPT when a binary image is
 > > > written to a device that is not exactly the same size as the image. That
 > > > is pretty much always the case with installer images.
 > > 
 > > Indeed.  MBR/BSD images don't have this problem because all metadata is
 > > at the beginning of and within the image in the case of UFS superblocks.
 > > So an image can be copied to a device like /dev/da0, in the case of the
 > > hitherto dedicated drive /dev/da0a, or as easily to a slice like da0s1
 > > where it becomes da0s1a, as long as there's enough room on the memstick.
 > > 
 > > A GPT image seems more of a whole-disk image rather than (what equally
 > > can serve as) a single partition image.  It must have a primary GPT (all
 > > 34 sectors?) but do you know if these images already contain the full 34
 > > sector secondary GPT?  i.e, I don't see why only the last sector of the
 > > copied image file was changed, with no update to any of the prior 33?
 > 
 > Most of those 34 blocks (not a fixed size, by the way) contain partition
 > information.  But there are global flags and things like the PMBR which could
 > change.  Without knowing exactly what that byte does, it's a guess.

Ok.  I've been using the Wikipedia page as reference, as I tend to find 
diagrams more useful: https://en.wikipedia.org/wiki/GUID_Partition_Table

I find it slightly weird that the secondary GPT (there) uses LBA-33 for 
the first four partitions; if it was my design (cough) I'd have reversed 
that also, and used LBA-2 for the first four, etc.  But then, I've never 
been enamoured of this (problematic) end-of-physical-disk positioning.

 > Note that we don't know when that change is happening.  It could be the
 > machine checking the secondary GPT on the memory stick that is helpfully and
 > silently "fixing" something when it recognizes the stick, before it is read.

As David mentioned, doing so silently would be less than helpful (POLA!)

 > It could be malicious or buggy firmware in any of several places, including
 > the UEFI or BIOS or the memory stick micontroller itself.  The point is that
 > we don't know that it is the FreeBSD installer making that change.

Right, so I guess we need more people to test this on their machines, 
but I'm not in a position to do so lately .. maybe David's PR will help.

And Perry's issue is also outstanding, and rather inexplicable so far.

 > I remember a few years back that someone helped me understand what was going
 > on, but forget the details (and, unfortunately, who it was).

Many of us could relate to that :)  "I thought I met a man .. who said 
he knew a man .. who knew what was going on .."  David Crosby - IIRC!

 > > Personally I'm wanting to get to a memstick with 2 or 3 bootable slices
 > > for installers (amd64 and i386) - as least the amd64 one from dvd1 with
 > > packages - plus a mountable r/w data partition.  Using only native tools
 > > currently implies using boot0, thus MBR/BSD schema.  Converting a new
 > > GPT-scheme memstick image to ONE slice on my new 16GB memstick is what I
 > > need to figure out, possibly by means of using a tar pipeline similar to
 > > dvd1_to_memstick.sh.  I hope to get time to look at 11.0 make release
 > > memstick scripts, but I'd appreciate any clues / ideas meanwhile ..
 > 
 > I'd think you could just dd(1) the contents of the single installer disk
 > partition into a big-enough MBR partition.  Repeat for each image, then use
 > boot0cfg.

Sure, for old-style raw BSD images, but that won't work for images with: 
1) PMBR, 2) primary GPT, 3) efifat partition, 4) gptboot partition then 
5) the UFS partition - which is all we want to copy to a bootable BSD 
slice - then 6) 1M(??) swap.  We need to use /boot/boot bootblocks for 
each slice; still, it may be easy to locate and dd just that partition 
out of the mdconfig'd image - or mount it and copy; some tests needed.

 > It really is a shame we still require separate installers for amd64 and i386,
 > a single installer can install either.

It's got a lot simpler on 11 src/release, using mkimg(1) rather than 
scripting gpart steps as before, but both memstick and iso still differ 
between i385 and amd64.  Perhaps the EFI issues are the main difference?

root at x200:/usr/head/src/release # diff -u i386/make-memstick.sh amd64/make-memstick.sh
--- i386/make-memstick.sh       2016-05-22 06:27:55.712694000 +1000
+++ amd64/make-memstick.sh      2016-05-22 06:27:55.713721000 +1000
@@ -7,7 +7,7 @@
 #
 # Usage: make-memstick.sh <directory tree> <image filename>
 #
-# $FreeBSD: head/release/i386/make-memstick.sh 293188 2016-01-05 03:20:45Z gjb $
+# $FreeBSD: head/release/amd64/make-memstick.sh 293188 2016-01-05 03:20:45Z gjb $

  [.. clipped and reformatted for clarity on the list ..]

-mkimg -s gpt -b ${1}/boot/pmbr \
 -p freebsd-boot:=${1}/boot/gptboot -p freebsd-ufs:=${2}.part \
 -p freebsd-swap::1M -o ${2}

+mkimg -s gpt -b ${1}/boot/pmbr -p efi:=${1}/boot/boot1.efifat \
 -p freebsd-boot:=${1}/boot/gptboot -p freebsd-ufs:=${2}.part \
 -p freebsd-swap::1M -o ${2}

cheers, Ian