Re: -current buildworld stoppage
- In reply to: bob prohaska : "Re: -current buildworld stoppage"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Mon, 12 Oct 2026 15:09:59 UTC
On a somewhat new tack:
I just noticed a core file, apparently from gpart:
-rw------- 1 root wheel 3182592 Feb 21 2026 gpart.core
I don't remember specifics, but I did try to adjust swap space
around that time. Is it plausible that an ungraceful exit by gpart
might result in the sort of mischief showing up now?
At present, gpart show reports
# gpart show da0
=> 63 1953525105 da0 MBR (932G)
63 1985 - free - (993K)
2048 102400 1 fat32lba [active] (50M)
104448 1953420720 2 freebsd (931G)
# gpart show da0s2
=> 0 1953420720 da0s2 BSD (931G)
0 128 - free - (64K)
128 10381184 1 freebsd-ufs (4.9G)
10381312 3686400 2 freebsd-swap (1.8G)
14067712 4505600 - free - (2.1G)
18573312 1934847408 4 freebsd-ufs (923G)
IIRC I was trying to reduce swap space from 2GB (which
produces a warning) to slightly less. I didn't notice a
crash, and the excess swap warnings went away.
bob prohaska
On Mon, Oct 12, 2026 at 07:45:33AM -0700, bob prohaska wrote:
> On Sun, Oct 11, 2026 at 08:33:49PM -0700, Mark Millard wrote:
> > On 10/11/26 19:13, bob prohaska wrote:
> > > On Sun, Oct 11, 2026 at 11:47:21AM -0700, Mark Millard wrote:
> > >> On 10/11/26 08:27, bob prohaska wrote:
> > >>> On Sat, Oct 10, 2026 at 03:08:18PM -0700, Mark Millard wrote:
> > >>>>
> > >>>> I'm not suggesting drive or cable failures, though I can not rule them
> > >>>> out. If you had such, testing the an alternate same type of drive setup
> > >>>> that would use the same drivers would be a cross check on such by seeing
> > >>>> if only the one device gets failures. (Not that you have the means of
> > >>>> doing such testing.)
> > >>>
> > >>> It isn't obvious how to swap drives without also disturbing cables, somewhat
> > >>> invalidating the test.
> > >>
> > >> No testing is even less information. In general any physical testing
> > >> involves multiple subsystems that might be contributing. Testing is
> > >> difficult but provides more information.
> > >>
> > >>> In the past at least a few problems, usually on boot,
> > >>> have been seemingly cleared by re-seating USB connectors or microSD cards.
> > >>>
> > >>> In the last couple of days a string of lines resembling
> > >>> find: input/output error <pathname>
> > >>> have been showing up in the daily security email.
> > >>
> > >>
> > >> That means EIO (errno 5), if what I read is correct.
> > >>
> > >> This suggests possibilities:
> > >>
> > >> ) physical drive failures
> > >> ) file system corruptions
> > >> ) connection problems
> > >>
> > >
> > > At this point re-seating the connectors looks like a
> > > reasonable thing to try.
> > >> (No network drive is involved as I understand.)
> > >
> > > That is correct.
> > >
> > >>
> > >>
> > >> Did you check you kernel messages in /var/log/messages* for the right
> > >> time frames? (Recent live contexts you could use dmsg -a or the like.)
> > >> Even without the detailed time frames for the specific find reports,
> > >> finding any appropriate errors over the days is relevant to report.
> > >>
> > >
> > > The machine has been running installworld all afternoon and dmesg is
> > > reporting a steady stream of errors but it isn't giving up, for example
> >
> > So likely/possibly there is extensive corruption.
> >
> > >
> > > /usr: inode 84028739: check-hash failed
> > > /usr: inode 84028739: check-hash failed
> > > /usr: inode 84028739: check-hash failed
> > > /usr: inode 84028739: check-hash failed
> > > /usr: inode 84028744: check-hash failed
> > > /usr: inode 84028744: check-hash failed
> > > /usr: inode 84028744: check-hash failed
> > > /usr: inode 84028744: check-hash failed
> > > /usr: bad dir ino 90930262 at offset 0: mangled entry
> > > /usr: bad dir ino 90930262 at offset 0: mangled entry
> > > /usr: bad dir ino 90930284 at offset 0: mangled entry
> > > /usr: bad dir ino 90930284 at offset 512: mangled entry
> > > /usr: bad dir ino 90930284 at offset 0: mangled entry
> >
> > How many distinct nodes for each type of message? Just the ones shown? Lots?
> >
> > I will note that you have reported both EIO errors and corrupted data
> > overall. The corrupted data is from widely distinct places, including
> > internal metadata and user file data content.
> >
> > Are you using SU or SU+J?
>
> Apologies, but I don't understand the question...
>
> > With all the failure information, if the data is important then
> > duplicating it elsewhere seems to be what is appropriate. Is there
> > anything that you need to try to preserve? Or is loosing arbitrary parts
> > of the content okay (even loss of all of it)?
>
> The whole system is expendable in practical terms. I was curious to see
> how long it could be kept going, maybe time is up. One of my Pi2 systems
> has been running since 2020, though not with the same level of intensity.
>
> >
> > The EIO may be unlikely to be a software problem. The rest is unclear.
> >
> > >
> > > Am I correct in guessing it's succeeding after some
> > > amount of re-trying?
> >
> > No. Retries are for the media controller reporting problems at that
> > level. Note that at that level merely having bad data compared to what
> > was written originally need not be reported as an error at all.
> >
> > "check-hash failed" is reporting internal/metadata data is not as
> > expected.Only if that is because a check-hash was simply not written to
> > match the data would saving a recalculation of it give the correct result.
> >
> > As I understand, fsck might not do much about "mangled entry". If I
> > understand right, fixes to "mangled entry" tend to move files to
> > lost+found .
> >
> > I'll note that just forcing the check-hash to match the data leaves the
> > data with an unknown status for it is garbage. You have already seen
> > corruptions that were obvious.
> >
> > How much can you validate in order to depend on it vs. how much is
> > messed up?
> >
>
> On my own, very little,
>
> > I do not see how you can only use a context this messed up to diagnose
> > itself.
> >
> > Unlike fsck, fsdb is not automatic but seems to likely be needed, not
> > that I have ever used it.
> >
> > >> Do you have smartmontools installed so that you could do a:
> > >> smartctl -a /dev/... or the like?
> > >>
> > >>
> > >
> > > I just checked and apparently don't, though I think the
> > > drive does support smartmontools. I'll try compiling it
> > > after installworld finishes, assuming it does finish 8-)
> > >
> > >>> The ssh
> > >>> window used didn't preserve the lines but I could try to
> > >>> save them if it's of interest to anybody. In the meantime
> > >>> the machine's been running a -j4 buildworld without disaster,
> > >>> no errors on the serial console either.
> >
> > Your serial console is not setup to get kernel messages?
> It does show things like login failures, maybe not everything.
>
> I've placed a sample of security email at
> http://www.zefox.net/~fbsd/rpi2/crashes/_-current_buildworld_stoppage/pelorus_security_email
> as it's the most complete report generated recently. It follows a cycle of
> git pull, system build and install, along with an update to poudriere but no ports git pull.
> That makes a large chunk of the warnings normal, somewhat more look like trouble.
>
> I just noticed a small mismatch between kernel and world:
> # uname -apKU
> FreeBSD pelorus.zefox.org 16.0-CURRENT FreeBSD 16.0-CURRENT #71 main-n289010-f02f93e79147-dirty: Thu Sep 10 07:55:05 PDT 2026 bob@pelorus.zefox.org:/usr/obj/usr/src/arm.armv7/sys/GENERIC arm armv7 1600025 1600027
> If that looks significant perhaps the machine should be rebooted to
> bring them into sync.
>
> Thanks for writing!
>
> bob prohaska
>
>
>
>