ZFS l2arc broken in 10.3

[Andriy Gapon]

On 12/10/2016 23:18, Peter wrote:
> Details:
> After upgrading 2 machines from 9.3 to 10.3-STABLE, on one of them the
> l2arc stays empty (capacity alloc = 0), although it is online and gets
> accessed. It did work well on 9.3.
> 
> I did the following tests:
>  * Create a zpool on a stick, with two volumes: one filesystem and one
>    cache. The cache stays with alloc=0.
>    Export it and move it into the other machine. The cache immediately
>    fills.
>    Move it back, the cache stays with alloc=0.
>    -> this rules out all zpool/zfs get/set options, as they should
>       walk with the pool.
>  * Boot the GENERIC kernel. l2arc stays with alloc=0.
>    -> this rules out all my nonstandard kernel options.
>  * Boot in single user mode. l2arc stays with alloc=0.
>    -> this rules out all /etc/* config files.
>  * Delete the zpool.cache and reimport pools. l2arc stays with alloc=0.
>  * Copy the /boot/loader.conf settings to the other machine. The l2arc
>    still works there.
> 
> I could not think of any remaining place where this could come from,
> except the kernel code itself.
> From there, I found these counters nicely incrementing each second:
>   kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 50758
>   kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 27121
>   kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 40589375488
> But also this counter incrementing:
>   kstat.zfs.misc.arcstats.l2_write_full: 14604
> 
> Then with some printf in the code I saw these values provided:
>          buf_sz = hdr->b_size;
>     align = (size_t)1 << dev->l2ad_vdev->vdev_ashift;
>         buf_a_sz = P2ROUNDUP(buf_sz, align);
>     if ((write_asize + buf_a_sz) > target_sz) {
>            full = B_TRUE;
>            mutex_exit(hash_lock);
>        ARCSTAT_BUMP(arcstat_l2_write_full);
>        break;
>     }
> 
> buf_sz =    1536
> align =        512
> buf_a_sz =    18446744069414585856
> write_asize =    0
> target_sz =    16777216
> 
> where buf_a_sz is obviousely off by (2^64 - 2^32).
> 
> Maybe this is an effect of crosscompiling i386 on amd64.

Yes, the problem is specific to 32-bit platforms where size_t is 32-bit.

> But anyway, as long as
> i386 is still supported, it should not happen.

Certainly.

> Now, my real concern is: if this really obvious ... made it undetected until
> 10.3, how many other missing typecasts are still in the code??

No need to be dramatic here.  That particular piece code is very new.
I committed it to head in April (r297848), MFC-ed even later.
Apparently no one else who uses 32-bit systems and has L2ARC configured had a
chance to run into the bug.

Thank you very much for discovering and analyzing the bug and providing a fix
for it!


-- 
Andriy Gapon