Verry serious problem with ZFS & 12.0

[David Christensen]

On 8/28/19 3:45 PM, Albert Shih wrote:
> Hi
> 
> After update 4 servers from 11.2 to 12.0 without any problem, wait few
> weeks to see if everything work well, and it did. I just upgrade my mail
> server.
> 
> During the upgrade I also upgrade all firmware for the hardware.
> 
> And now I got a very serious issue with my server.
> 
> Configuration :
> 
>    Dell PowerEdge R740Xd with H730P, 192 Go Ram, 2 SAS mechanical disk for the system,
>    2 SSD (in a zfs pool) for the mail index (cyrus), and 28 mechanical disk
>    (in a second zfs pool) for the mailbox.
> 
> The problem:
> 
>    After running few days the zfs pool with the 2 SSD are not responding.
> 
>    The system are perfectly working.
> 
>    The second zpool (mechanical disk) are perfectly working.
>    
>    I got zero log, zero message in the console or in dmesg.
> 
>    The arc_size are correct, it's around 70-75 %.
> 
>    The moment the zfs pool become not responding are random, not related to
>    any activity (human or cron).
> 
>    The only option I pass for the kernel related to ZFS are vfs.zfs.min_auto_ashift=12 and
>    vfs.zfs.prefetch_disable=1. Without the second one the system no
>    responding (under 11.2) when the server send (through zfs send) the data to another
>    server.
> 
>    After the first problem I make a zfs upgrade, thinking maybe that's the
>    problem so I'm not sure I can downgrade to 11.2 (and 11.2 are EOL)
> 
> In your opinion :
> 
>    1/ What should I do to try to find the problem ?
> 
>    2/ Do you think that's a hardware/firmware problem or FreeBSD problem,
>    the point is the second zpool are working perfectly so I'm thinking at
>    some firmware/hardware/compatibility problem.

Please run the following commands in a terminal and post the actual 
prompt, the actual command entered, and the actual output produced by 
the command (redact as required).  Substitute "SSDPOOL", below, with the 
actual name of your SSD pool.  Substitute MAILD with the actual name of 
your mail service:

     freebsd-version

     uname -a

     pkg version | grep MAILD

     zpool list

     zpool status SSDPOOL

     zfs list -d 1 SSDPOOL

     top -S -d3

     top -mio -d3


Are mail services still operational without the mail index?


Can you configure your mail service to not use a mail index?


Can you put two fast drives into the mail server, create a mirror, and 
put the mail index there?


David