FreeBSD 10.0-STABLE on EdgeRouter Lite notes/issues

Thu Aug 28 20:06:35 UTC 2014

On Aug 28, 2014, at 12:59 PM, Stacey Son <sson at FreeBSD.org> wrote:

> Hi Sean:
> 
> Some answers to your questions are inline below...
> 
> On Aug 24, 2014, at 4:23 PM, Sean Hamilton <seanhamilton at gmail.com> wrote:
> 
>> Running 10.0-STABLE r270425 on an EdgeRouter Lite.
>> 
>> "makefs -B big -o version=2" seems to produce a broken image which
>> cannot be mounted read/write. See misc/162503, bin/188762. FS
>> performance on USB flash is awful without softupdates, so UFS2 is
>> effectively mandatory. I built a rescue-only mfsroot and newfs'd and
>> untar'd the real system from there. Flash performance still isn't
>> great, but it is tolerable. Mounting noatime helps. Were it not for
>> the makefs bug, this process would be much easier.
>> 
>> Attempting to makefs directly onto the USB flash is painfully slow --
>> writing a file, then dd'ing that to the device is much faster. I
>> assume makefs does a lot of small writes. Can the OS cache and combine
>> these? It would also be nice if makefs could detect the size of the
>> target device and set the FS size and number of inodes appropriately.
>> Of course at that point it's basically newfs.
> 
> 
> The way I do it is create a FS only the size that is needed and 'dd' to USB then 'growfs' later. 
> 
> Nathan Dorfman's script makes it easy to make images...
> 
>   http://rtfm.net/FreeBSD/ERL/mkerlimage
> 
> You may have also have seen his webpage on the ERL:
> 
>   http://rtfm.net/FreeBSD/ERL
> 
>> Also for some reason, ERL u-boot's tftpboot seems to refuse to talk to
>> FreeBSD's tftpd, so I had to copy my rescue-only image to the flash. I
>> didn't investigate this closely, could be operator error.
>> 
>> Any suggestions for VFS tuning on ERL? Ideally the OS could
>> aggressively cache filesystem writes, then lazily write them out
>> asynchronously. Would softupdate journaling be of any benefit?
>> 
>> gpart insists on aligning MBR partitions to cylinder boundaries, and
>> seemingly ignores -a. Is there some way to prevent this? If not, can
>> it be fooled by overriding the disk geometry? I'm not sure how much
>> this matters on USB flash devices, but it matters a lot on SSDs. I
>> wound up partitioning with GNU parted, since it does what it's told.
>> 
>> I would prefer to use GPT, but it seems like ERL u-boot doesn't
>> support it. Another option might be to use GPT, create a freebsd-boot
>> partition, dd the kernel directly there, then just load the sectors
>> directly with u-boot, bypassing the filesystem entirely.
> 
> Yeah, the ERL u-boot seems kind of simple.  You may have noticed that 'mkerlimage' puts /boot in a FAT partition.  Once the kernel is loaded you can do about any partitioning scheme you want.  IDK.   It would be nice if the ERL had a second USB or some onboard flash that could be used or something.
> 
>> Ports are totally unbuildable on ERL. This seems to be a toolchain
>> problem. ports-mgmt/pkg (which everything depends on) fails to build
>> with:
>> 
>> libtool: install: ranlib
>> /usr/ports/ports-mgmt/pkg/work/stage/usr/local/lib/libpkg_static.a
>> ranlib: fatal: Invalid filename
>> 
>> I imagine this relates to the recent freebsd-mips posts about strip
>> corrupting static binaries.
> 
> Yes, this is most likely the 'strip' bug.  See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191587 for the details.  The workaround is to not strip static libraries.
> 
> FYI, you can find some ports I have built for FreeBSD/mips64 (using qemu user-mode) at:
> 
>   http://www.cl.cam.ac.uk/research/security/ctsrd/mips64-packages/
> 
> Sean Bruno may even have more some place.
> 
>> tcpdump is broken, as the packet lengths it reports are incorrect, and
>> it eventually segfaults. I thought this might be an issue with the
>> octe driver, but it seems even the loopback device is afflicted, so
>> perhaps it's an endianness issue?
> 
> You should file a bugzilla bug report especially if you have a packet trace that can be used to easily reproduce the problem.
> 
>> Regarding https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=177876 --
>> is this patch still a good idea? Or has it been merged?
> 
> I have been using it for some time.  If you plan on mounting and using NFS volumes or anything else the might take a lot of kernel thread stack space then it is a good idea.
> 
> A better version of this patch (that moves the mips dependent code out of the vm layer) has been merged into a MIPS64 FreeBSD project and that code will be upstreamed sometime in the near future.   You can find the changes here:
> 
> https://github.com/CTSRD-CHERI/cheribsd/commit/b39bec2cefe36293ffb2322969c9f7c9cae8c1a2
> 
> https://github.com/CTSRD-CHERI/cheribsd/commit/04ea99d04f6d5a595be759d34e84e7fd9cea8166

I don’t suppose this could be done w/o adding #ifdef __mips__ to vm_glue? No need to block upstreaming on this, but it seems this could be a MD routine instead...

> https://github.com/CTSRD-CHERI/cheribsd/commit/ae4f8d84bc30ffe2b2698534a2e6ab64480e8432

On the whole, I really like this, but was hoping it would be cleaner. There’s a crying need for it…

Warner

> For now, I would just used the kernel patch at:
> 
> https://people.freebsd.org/~sson/mips/kstack/kstack_large_page.diff
> 
>> Is there an estimate as to how many people are running FreeBSD on ERL?
>> Many thanks in advance for suggestions on any of the above!
> 
> Good question.   I know of three or so.
> 
> -stacey.
> 
> _______________________________________________
> freebsd-mips at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-mips
> To unsubscribe, send any mail to "freebsd-mips-unsubscribe at freebsd.org"

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.freebsd.org/pipermail/freebsd-mips/attachments/20140828/969cdf95/attachment.sig>