FreeBSD 10.0-STABLE on EdgeRouter Lite notes/issues

[Stacey Son]

Hi Sean:

Some answers to your questions are inline below...

On Aug 24, 2014, at 4:23 PM, Sean Hamilton <seanhamilton at gmail.com> wrote:

> Running 10.0-STABLE r270425 on an EdgeRouter Lite.
> 
> "makefs -B big -o version=2" seems to produce a broken image which
> cannot be mounted read/write. See misc/162503, bin/188762. FS
> performance on USB flash is awful without softupdates, so UFS2 is
> effectively mandatory. I built a rescue-only mfsroot and newfs'd and
> untar'd the real system from there. Flash performance still isn't
> great, but it is tolerable. Mounting noatime helps. Were it not for
> the makefs bug, this process would be much easier.
> 
> Attempting to makefs directly onto the USB flash is painfully slow --
> writing a file, then dd'ing that to the device is much faster. I
> assume makefs does a lot of small writes. Can the OS cache and combine
> these? It would also be nice if makefs could detect the size of the
> target device and set the FS size and number of inodes appropriately.
> Of course at that point it's basically newfs.


The way I do it is create a FS only the size that is needed and 'dd' to USB then 'growfs' later. 

Nathan Dorfman's script makes it easy to make images...
  
   http://rtfm.net/FreeBSD/ERL/mkerlimage

You may have also have seen his webpage on the ERL:

   http://rtfm.net/FreeBSD/ERL
   
> Also for some reason, ERL u-boot's tftpboot seems to refuse to talk to
> FreeBSD's tftpd, so I had to copy my rescue-only image to the flash. I
> didn't investigate this closely, could be operator error.
> 
> Any suggestions for VFS tuning on ERL? Ideally the OS could
> aggressively cache filesystem writes, then lazily write them out
> asynchronously. Would softupdate journaling be of any benefit?
> 
> gpart insists on aligning MBR partitions to cylinder boundaries, and
> seemingly ignores -a. Is there some way to prevent this? If not, can
> it be fooled by overriding the disk geometry? I'm not sure how much
> this matters on USB flash devices, but it matters a lot on SSDs. I
> wound up partitioning with GNU parted, since it does what it's told.
> 
> I would prefer to use GPT, but it seems like ERL u-boot doesn't
> support it. Another option might be to use GPT, create a freebsd-boot
> partition, dd the kernel directly there, then just load the sectors
> directly with u-boot, bypassing the filesystem entirely.

Yeah, the ERL u-boot seems kind of simple.  You may have noticed that 'mkerlimage' puts /boot in a FAT partition.  Once the kernel is loaded you can do about any partitioning scheme you want.  IDK.   It would be nice if the ERL had a second USB or some onboard flash that could be used or something.

> Ports are totally unbuildable on ERL. This seems to be a toolchain
> problem. ports-mgmt/pkg (which everything depends on) fails to build
> with:
> 
> libtool: install: ranlib
> /usr/ports/ports-mgmt/pkg/work/stage/usr/local/lib/libpkg_static.a
> ranlib: fatal: Invalid filename
> 
> I imagine this relates to the recent freebsd-mips posts about strip
> corrupting static binaries.

Yes, this is most likely the 'strip' bug.  See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191587 for the details.  The workaround is to not strip static libraries.

FYI, you can find some ports I have built for FreeBSD/mips64 (using qemu user-mode) at:

   http://www.cl.cam.ac.uk/research/security/ctsrd/mips64-packages/

Sean Bruno may even have more some place.

> tcpdump is broken, as the packet lengths it reports are incorrect, and
> it eventually segfaults. I thought this might be an issue with the
> octe driver, but it seems even the loopback device is afflicted, so
> perhaps it's an endianness issue?

You should file a bugzilla bug report especially if you have a packet trace that can be used to easily reproduce the problem.

> Regarding https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=177876 --
> is this patch still a good idea? Or has it been merged?

I have been using it for some time.  If you plan on mounting and using NFS volumes or anything else the might take a lot of kernel thread stack space then it is a good idea.

A better version of this patch (that moves the mips dependent code out of the vm layer) has been merged into a MIPS64 FreeBSD project and that code will be upstreamed sometime in the near future.   You can find the changes here:

https://github.com/CTSRD-CHERI/cheribsd/commit/b39bec2cefe36293ffb2322969c9f7c9cae8c1a2

https://github.com/CTSRD-CHERI/cheribsd/commit/04ea99d04f6d5a595be759d34e84e7fd9cea8166

https://github.com/CTSRD-CHERI/cheribsd/commit/ae4f8d84bc30ffe2b2698534a2e6ab64480e8432

For now, I would just used the kernel patch at:

https://people.freebsd.org/~sson/mips/kstack/kstack_large_page.diff

> Is there an estimate as to how many people are running FreeBSD on ERL?
> Many thanks in advance for suggestions on any of the above!

Good question.   I know of three or so.

-stacey.