Re: -current buildworld stoppage

On 10/11/26 19:13, bob prohaska wrote:
> On Sun, Oct 11, 2026 at 11:47:21AM -0700, Mark Millard wrote:
>> On 10/11/26 08:27, bob prohaska wrote:
>>> On Sat, Oct 10, 2026 at 03:08:18PM -0700, Mark Millard wrote:
>>>>
>>>> I'm not suggesting drive or cable failures, though I can not rule them
>>>> out. If you had such, testing the an alternate same type of drive setup
>>>> that would use the same drivers would be a cross check on such by seeing
>>>> if only the one device gets failures. (Not that you have the means of
>>>> doing such testing.)
>>>
>>> It isn't obvious how to swap drives without also disturbing cables, somewhat
>>> invalidating the test. 
>>
>> No testing is even less information. In general any physical testing
>> involves multiple subsystems that might be contributing. Testing is
>> difficult but provides more information.
>>
>>> In the past at least a few problems, usually on boot,
>>> have been seemingly cleared by re-seating USB connectors or microSD cards.
>>>
>>> In the last couple of days a string of lines resembling
>>> find: input/output error <pathname> 
>>> have been showing up in the daily security email.
>>
>>
>> That means EIO (errno 5), if what I read is correct.
>>
>> This suggests possibilities:
>>
>> ) physical drive failures
>> ) file system corruptions
>> ) connection problems
>>
> 
> At this point re-seating the connectors looks like a
> reasonable thing to try.  
>> (No network drive is involved as I understand.)
> 
> That is correct.
> 
>>
>>
>> Did you check you kernel messages in /var/log/messages* for the right
>> time frames? (Recent live contexts you could use dmsg -a or the like.)
>> Even without the detailed time frames for the specific find reports,
>> finding any appropriate errors over the days is relevant to report.
>>
> 
> The machine has been running installworld all afternoon and dmesg is
> reporting a steady stream of errors but it isn't giving up, for example

So likely/possibly there is extensive corruption.

> 
> /usr: inode 84028739: check-hash failed
> /usr: inode 84028739: check-hash failed
> /usr: inode 84028739: check-hash failed
> /usr: inode 84028739: check-hash failed
> /usr: inode 84028744: check-hash failed
> /usr: inode 84028744: check-hash failed
> /usr: inode 84028744: check-hash failed
> /usr: inode 84028744: check-hash failed
> /usr: bad dir ino 90930262 at offset 0: mangled entry
> /usr: bad dir ino 90930262 at offset 0: mangled entry
> /usr: bad dir ino 90930284 at offset 0: mangled entry
> /usr: bad dir ino 90930284 at offset 512: mangled entry
> /usr: bad dir ino 90930284 at offset 0: mangled entry

How many distinct nodes for each type of message? Just the ones shown? Lots?

I will note that you have reported both EIO errors and corrupted data
overall. The corrupted data is from widely distinct places, including
internal metadata and user file data content.

Are you using SU or SU+J?

With all the failure information, if the data is important then
duplicating it elsewhere seems to be what is appropriate. Is there
anything that you need to try to preserve? Or is loosing arbitrary parts
of the content okay (even loss of all of it)?

The EIO may be unlikely to be a software problem. The rest is unclear.

> 
> Am I correct in guessing it's succeeding after some
> amount of re-trying?  

No. Retries are for the media controller reporting problems at that
level.  Note that at that level merely having bad data compared to what
was written originally need not be reported as an error at all.

"check-hash failed" is reporting internal/metadata data is not as
expected.Only if that is because a check-hash was simply not written to
match the data would saving a recalculation of it give the correct result.

As I understand, fsck might not do much about "mangled entry". If I
understand right, fixes to "mangled entry" tend to move files to
lost+found .

I'll note that just forcing the check-hash to match the data leaves the
data with an unknown status for it is garbage. You have already seen
corruptions that were obvious.

How much can you validate in order to depend on it vs. how much is
messed up?

I do not see how you can only use a context this messed up to diagnose
itself.

Unlike fsck, fsdb is not automatic but seems to likely be needed, not
that I have ever used it.

>> Do you have smartmontools installed so that you could do a:
>> smartctl -a /dev/... or the like?
>>
>>
> 
> I just checked and apparently don't, though I think the
> drive does support smartmontools. I'll try compiling it
> after installworld finishes, assuming it does finish 8-) 
> 
>>> The ssh 
>>> window used didn't preserve the lines but I could try to
>>> save them if it's of interest to anybody. In the meantime
>>> the machine's been running a -j4 buildworld without disaster,
>>> no errors on the serial console either.

Your serial console is not setup to get kernel messages?

>>>
>>> Thanks for writing, I'll keep poking at the problem(s).
>>>
>>> bob prohaska
>>
>>
>> I will note that trying to repair a failing storage subsystem can make
>> things worse, such as repairing via fsk use. Generally, if there is
>> evidence of failures suggesting drive replacements, the point is
>> normally to duplicate to some other media anything that should be
>> preserved first, instead of disturbing it first. Repair attempts come
>> after preserving what one can that one wants preserved.
>>
> 
> Fortunately this is a test machine so nothing to lose. I did have hopes
> that it might be backed down to a stable release and used as a nameserver,
> that plan may have to wait.  
>> For boot media, this normally means no longer booting the media and/oir
>> having the media as strictly read-only.
> 
> The usual boot failures seem to be warm (shutdown -r), power cycling
> always seems to boot hands-off. When boot fails, it's usually a hang
> in uboot with loss of console control.  
> 
> It looks like the install has finished successfully. What's your guess
> for the first thing to try? I think mine would be to shutdown to power
> off, reset the USB connectors and attempt a restart. If something else
> comes to your mind please let me know!
> 
> Thanks for writing,
> 
> bob prohaska
>   
> 
> 
> 


-- 
===
Mark Millard
marklmi at yahoo.com