Re: -current buildworld stoppage

On Sun, Oct 11, 2026 at 08:33:49PM -0700, Mark Millard wrote:
> On 10/11/26 19:13, bob prohaska wrote:
> > On Sun, Oct 11, 2026 at 11:47:21AM -0700, Mark Millard wrote:
> >> On 10/11/26 08:27, bob prohaska wrote:
> >>> On Sat, Oct 10, 2026 at 03:08:18PM -0700, Mark Millard wrote:
> >>>>
> >>>> I'm not suggesting drive or cable failures, though I can not rule them
> >>>> out. If you had such, testing the an alternate same type of drive setup
> >>>> that would use the same drivers would be a cross check on such by seeing
> >>>> if only the one device gets failures. (Not that you have the means of
> >>>> doing such testing.)
> >>>
> >>> It isn't obvious how to swap drives without also disturbing cables, somewhat
> >>> invalidating the test. 
> >>
> >> No testing is even less information. In general any physical testing
> >> involves multiple subsystems that might be contributing. Testing is
> >> difficult but provides more information.
> >>
> >>> In the past at least a few problems, usually on boot,
> >>> have been seemingly cleared by re-seating USB connectors or microSD cards.
> >>>
> >>> In the last couple of days a string of lines resembling
> >>> find: input/output error <pathname> 
> >>> have been showing up in the daily security email.
> >>
> >>
> >> That means EIO (errno 5), if what I read is correct.
> >>
> >> This suggests possibilities:
> >>
> >> ) physical drive failures
> >> ) file system corruptions
> >> ) connection problems
> >>
> > 
> > At this point re-seating the connectors looks like a
> > reasonable thing to try.  
> >> (No network drive is involved as I understand.)
> > 
> > That is correct.
> > 
> >>
> >>
> >> Did you check you kernel messages in /var/log/messages* for the right
> >> time frames? (Recent live contexts you could use dmsg -a or the like.)
> >> Even without the detailed time frames for the specific find reports,
> >> finding any appropriate errors over the days is relevant to report.
> >>
> > 
> > The machine has been running installworld all afternoon and dmesg is
> > reporting a steady stream of errors but it isn't giving up, for example
> 
> So likely/possibly there is extensive corruption.
> 
> > 
> > /usr: inode 84028739: check-hash failed
> > /usr: inode 84028739: check-hash failed
> > /usr: inode 84028739: check-hash failed
> > /usr: inode 84028739: check-hash failed
> > /usr: inode 84028744: check-hash failed
> > /usr: inode 84028744: check-hash failed
> > /usr: inode 84028744: check-hash failed
> > /usr: inode 84028744: check-hash failed
> > /usr: bad dir ino 90930262 at offset 0: mangled entry
> > /usr: bad dir ino 90930262 at offset 0: mangled entry
> > /usr: bad dir ino 90930284 at offset 0: mangled entry
> > /usr: bad dir ino 90930284 at offset 512: mangled entry
> > /usr: bad dir ino 90930284 at offset 0: mangled entry
> 
> How many distinct nodes for each type of message? Just the ones shown? Lots?
> 
> I will note that you have reported both EIO errors and corrupted data
> overall. The corrupted data is from widely distinct places, including
> internal metadata and user file data content.
> 
> Are you using SU or SU+J?

Apologies, but I don't understand the question...
 
> With all the failure information, if the data is important then
> duplicating it elsewhere seems to be what is appropriate. Is there
> anything that you need to try to preserve? Or is loosing arbitrary parts
> of the content okay (even loss of all of it)?

The whole system is expendable in practical terms. I was curious to see
how long it could be kept going, maybe time is up. One of my Pi2 systems
has been running since 2020, though not with the same level of intensity.  

> 
> The EIO may be unlikely to be a software problem. The rest is unclear.
> 
> > 
> > Am I correct in guessing it's succeeding after some
> > amount of re-trying?  
> 
> No. Retries are for the media controller reporting problems at that
> level.  Note that at that level merely having bad data compared to what
> was written originally need not be reported as an error at all.
> 
> "check-hash failed" is reporting internal/metadata data is not as
> expected.Only if that is because a check-hash was simply not written to
> match the data would saving a recalculation of it give the correct result.
> 
> As I understand, fsck might not do much about "mangled entry". If I
> understand right, fixes to "mangled entry" tend to move files to
> lost+found .
> 
> I'll note that just forcing the check-hash to match the data leaves the
> data with an unknown status for it is garbage. You have already seen
> corruptions that were obvious.
> 
> How much can you validate in order to depend on it vs. how much is
> messed up?
>

On my own, very little, 

> I do not see how you can only use a context this messed up to diagnose
> itself.
> 
> Unlike fsck, fsdb is not automatic but seems to likely be needed, not
> that I have ever used it.
> 
> >> Do you have smartmontools installed so that you could do a:
> >> smartctl -a /dev/... or the like?
> >>
> >>
> > 
> > I just checked and apparently don't, though I think the
> > drive does support smartmontools. I'll try compiling it
> > after installworld finishes, assuming it does finish 8-) 
> > 
> >>> The ssh 
> >>> window used didn't preserve the lines but I could try to
> >>> save them if it's of interest to anybody. In the meantime
> >>> the machine's been running a -j4 buildworld without disaster,
> >>> no errors on the serial console either.
> 
> Your serial console is not setup to get kernel messages?
It does show things like login failures, maybe not everything.

I've placed a sample of security email at
http://www.zefox.net/~fbsd/rpi2/crashes/_-current_buildworld_stoppage/pelorus_security_email
as it's the most complete report generated recently. It follows a cycle of
git pull, system build and install, along with an update to poudriere but no ports git pull.
That makes a large chunk of the warnings normal, somewhat more look like trouble.

I just noticed a small mismatch between kernel and world:
# uname -apKU
FreeBSD pelorus.zefox.org 16.0-CURRENT FreeBSD 16.0-CURRENT #71 main-n289010-f02f93e79147-dirty: Thu Sep 10 07:55:05 PDT 2026     bob@pelorus.zefox.org:/usr/obj/usr/src/arm.armv7/sys/GENERIC arm armv7 1600025 1600027
If that looks significant perhaps the machine should be rebooted to 
bring them into sync.

Thanks for writing!

bob prohaska