Re: -current buildworld stoppage

On 10/12/26 07:45, bob prohaska wrote:
> On Sun, Oct 11, 2026 at 08:33:49PM -0700, Mark Millard wrote:
>> On 10/11/26 19:13, bob prohaska wrote:
>>> On Sun, Oct 11, 2026 at 11:47:21AM -0700, Mark Millard wrote:
>>>> On 10/11/26 08:27, bob prohaska wrote:
>>>>> On Sat, Oct 10, 2026 at 03:08:18PM -0700, Mark Millard wrote:
>>>>>>
>>>>>> I'm not suggesting drive or cable failures, though I can not rule them
>>>>>> out. If you had such, testing the an alternate same type of drive setup
>>>>>> that would use the same drivers would be a cross check on such by seeing
>>>>>> if only the one device gets failures. (Not that you have the means of
>>>>>> doing such testing.)
>>>>>
>>>>> It isn't obvious how to swap drives without also disturbing cables, somewhat
>>>>> invalidating the test. 
>>>>
>>>> No testing is even less information. In general any physical testing
>>>> involves multiple subsystems that might be contributing. Testing is
>>>> difficult but provides more information.
>>>>
>>>>> In the past at least a few problems, usually on boot,
>>>>> have been seemingly cleared by re-seating USB connectors or microSD cards.
>>>>>
>>>>> In the last couple of days a string of lines resembling
>>>>> find: input/output error <pathname> 
>>>>> have been showing up in the daily security email.
>>>>
>>>>
>>>> That means EIO (errno 5), if what I read is correct.
>>>>
>>>> This suggests possibilities:
>>>>
>>>> ) physical drive failures
>>>> ) file system corruptions
>>>> ) connection problems
>>>>
>>>
>>> At this point re-seating the connectors looks like a
>>> reasonable thing to try.  
>>>> (No network drive is involved as I understand.)
>>>
>>> That is correct.
>>>
>>>>
>>>>
>>>> Did you check you kernel messages in /var/log/messages* for the right
>>>> time frames? (Recent live contexts you could use dmsg -a or the like.)
>>>> Even without the detailed time frames for the specific find reports,
>>>> finding any appropriate errors over the days is relevant to report.
>>>>
>>>
>>> The machine has been running installworld all afternoon and dmesg is
>>> reporting a steady stream of errors but it isn't giving up, for example
>>
>> So likely/possibly there is extensive corruption.
>>
>>>
>>> /usr: inode 84028739: check-hash failed
>>> /usr: inode 84028739: check-hash failed
>>> /usr: inode 84028739: check-hash failed
>>> /usr: inode 84028739: check-hash failed
>>> /usr: inode 84028744: check-hash failed
>>> /usr: inode 84028744: check-hash failed
>>> /usr: inode 84028744: check-hash failed
>>> /usr: inode 84028744: check-hash failed
>>> /usr: bad dir ino 90930262 at offset 0: mangled entry
>>> /usr: bad dir ino 90930262 at offset 0: mangled entry
>>> /usr: bad dir ino 90930284 at offset 0: mangled entry
>>> /usr: bad dir ino 90930284 at offset 512: mangled entry
>>> /usr: bad dir ino 90930284 at offset 0: mangled entry
>>
>> How many distinct nodes for each type of message? Just the ones shown? Lots?
>>
>> I will note that you have reported both EIO errors and corrupted data
>> overall. The corrupted data is from widely distinct places, including
>> internal metadata and user file data content.
>>
>> Are you using SU or SU+J?
> 
> Apologies, but I don't understand the question...

using "man tunefs" output about Soft Updates (SU) and Journaling for it
(SU+J):

     -n enable | disable
             Turn on/off soft updates.  Soft updates are incompatible with
             GEOM journaling.


     -j enable | disable
             Turn on/off soft updates journaling.  Enabling soft updates
             journaling implies enabling soft updates and is
incompatible with
             GEOM journaling.

             Enabling journaling reduces the time spent by fsck_ffs(8)
             cleaning up a filesystem after a crash to a few seconds from
             minutes to hours.  Without journaling, the time to recover
after
             a crash is a function of the number of files in the filesystem
             and the size of the filesystem.  With journaling, the time to
             recover after a crash is a function of the amount of
activity in
             the filesystem in the minute before the crash.  Journaled
             recovery time is usually only a few seconds and never exceeds a
             minute.

             The drawback to using journaling is that the writes to its log
             adds an extra write load to the media containing the
filesystem.
             Thus a write-intensive workload will have reduced
throughput on a
             filesystem running with journaling.

             Like all journaling filesystems, the journal recovery will only
             fix issues known to the journal.  Specifically if a media error
             occurs, the journal will not know about it and hence will
not fix
             it.  Thus when using journaling, it is still necessary to run a
             full fsck every few months or after a filesystem panic to check
             for and fix any errors brought on by media failure.  A full
fsck
             can be done by running a background fsck on a live
filesystem or
             by running with the -f flag on an unmounted filesystem.  When
             running fsck_ffs(8) in background on a live filesystem the
             filesystem performance will be about half of normal during the
             time that the background fsck_ffs(8) is running.  Running a
full
             fsck on a UFS filesystem is the equivalent of running a
scrub on
             a ZFS filesystem.

>  
>> With all the failure information, if the data is important then
>> duplicating it elsewhere seems to be what is appropriate. Is there
>> anything that you need to try to preserve? Or is loosing arbitrary parts
>> of the content okay (even loss of all of it)?
> 
> The whole system is expendable in practical terms. I was curious to see
> how long it could be kept going, maybe time is up. One of my Pi2 systems
> has been running since 2020, though not with the same level of intensity.  

Then: Might the best experiment be to start over from scratch and see if
the problem reappears?

Part of that might be reinitializing the SSD such that it no longer
suffers from the lack of trim activity. (FreeBSD does not trim over USB
as far as I know.)

> 
>>
>> The EIO may be unlikely to be a software problem. The rest is unclear.
>>
>>>
>>> Am I correct in guessing it's succeeding after some
>>> amount of re-trying?  
>>
>> No. Retries are for the media controller reporting problems at that
>> level.  Note that at that level merely having bad data compared to what
>> was written originally need not be reported as an error at all.
>>
>> "check-hash failed" is reporting internal/metadata data is not as
>> expected.Only if that is because a check-hash was simply not written to
>> match the data would saving a recalculation of it give the correct result.
>>
>> As I understand, fsck might not do much about "mangled entry". If I
>> understand right, fixes to "mangled entry" tend to move files to
>> lost+found .
>>
>> I'll note that just forcing the check-hash to match the data leaves the
>> data with an unknown status for it is garbage. You have already seen
>> corruptions that were obvious.
>>
>> How much can you validate in order to depend on it vs. how much is
>> messed up?
>>
> 
> On my own, very little, 
> 
>> I do not see how you can only use a context this messed up to diagnose
>> itself.
>>
>> Unlike fsck, fsdb is not automatic but seems to likely be needed, not
>> that I have ever used it.
>>
>>>> Do you have smartmontools installed so that you could do a:
>>>> smartctl -a /dev/... or the like?
>>>>
>>>>
>>>
>>> I just checked and apparently don't, though I think the
>>> drive does support smartmontools. I'll try compiling it
>>> after installworld finishes, assuming it does finish 8-) 
>>>
>>>>> The ssh 
>>>>> window used didn't preserve the lines but I could try to
>>>>> save them if it's of interest to anybody. In the meantime
>>>>> the machine's been running a -j4 buildworld without disaster,
>>>>> no errors on the serial console either.
>>
>> Your serial console is not setup to get kernel messages?
> It does show things like login failures, maybe not everything.
> 
> I've placed a sample of security email at
> http://www.zefox.net/~fbsd/rpi2/crashes/_-current_buildworld_stoppage/pelorus_security_email
> as it's the most complete report generated recently. It follows a cycle of
> git pull, system build and install, along with an update to poudriere but no ports git pull.
> That makes a large chunk of the warnings normal, somewhat more look like trouble.

The most significant thing I see is repeatability:

Checking setuid files and devices:
find: /usr/ports/devel/py-anyio3: Input/output error
. . .
find: /usr/src/contrib/llvm-project/llvm/lib/Target/AVR: Input/output error

Checking negative group permissions:
find: /usr/ports/devel/py-anyio3: Input/output error
. . .
find: /usr/src/contrib/llvm-project/llvm/lib/Target/AVR: Input/output error

Those lists exactly agree. Most of the messages look to be EIO related
messages indicating low level I/O failures. Some are "Invalid argument",
for which it is not clear to me just what leads to the message.

> 
> I just noticed a small mismatch between kernel and world:
> # uname -apKU
> FreeBSD pelorus.zefox.org 16.0-CURRENT FreeBSD 16.0-CURRENT #71 main-n289010-f02f93e79147-dirty: Thu Sep 10 07:55:05 PDT 2026     bob@pelorus.zefox.org:/usr/obj/usr/src/arm.armv7/sys/GENERIC arm armv7 1600025 1600027
> If that looks significant perhaps the machine should be rebooted to 
> bring them into sync.

The FreeBSD kernel bias/design-goal is for newer kernels to support
older worlds (going back some distance). The other way around is not
guaranteed, even though the official package builders usually have older
kernel and newer poudriere jail worlds. (poudriere jails do no not run
as much software as the boot world does.)

> 
> Thanks for writing!
> 
> bob prohaska



-- 
===
Mark Millard
marklmi at yahoo.com