[Bug 226064] Observing panic at ctl_free_beio+0x80 while running traffic on 100 iSCSI LUNs

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=226064

            Bug ID: 226064
           Summary: Observing panic at ctl_free_beio+0x80 while running
                    traffic on 100 iSCSI LUNs
           Product: Base System
           Version: CURRENT
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: freebsd-bugs at FreeBSD.org
          Reporter: manishk at chelsio.com
                CC: mav at FreeBSD.org

Created attachment 190824
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=190824&action=edit
core text file

Observing panic while running traffic on 100 iSCSI LUNs (used zvols as LUNs) on
the target machine. The panic stack trace:  

panic: Memory modified after free 0xfffff8016fadcb40(32) val=0 @
0xfffff8016fadcb40

cpuid = 0
time = 1518462716
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2c/frame 0xfffffe0065865370
kdb_backtrace() at kdb_backtrace+0x53/frame 0xfffffe0065865440
vpanic() at vpanic+0x268/frame 0xfffffe0065865510
kproc_shutdown() at kproc_shutdown/frame 0xfffffe0065865570
trash_ctor() at trash_ctor+0x6b/frame 0xfffffe00658655b0
uma_zalloc_arg() at uma_zalloc_arg+0x284/frame 0xfffffe00658656f0
bucket_alloc() at bucket_alloc+0x102/frame 0xfffffe0065865730
uma_zfree_arg() at uma_zfree_arg+0x647/frame 0xfffffe0065865820
uma_zfree() at uma_zfree+0x29/frame 0xfffffe0065865850
ctl_free_beio() at ctl_free_beio+0x80/frame 0xfffffe0065865880
ctl_complete_beio() at ctl_complete_beio+0x48/frame 0xfffffe00658658a0
ctl_be_block_dispatch_zvol() at ctl_be_block_dispatch_zvol+0x664/frame
0xfffffe00658659b0
ctl_be_block_worker() at ctl_be_block_worker+0x1fb/frame 0xfffffe0065865a40
taskqueue_run_locked() at taskqueue_run_locked+0x246/frame 0xfffffe0065865ab0
taskqueue_thread_loop() at taskqueue_thread_loop+0xc2/frame 0xfffffe0065865b30
fork_exit() at fork_exit+0x145/frame 0xfffffe0065865bb0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0065865bb0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic


-------------------------------------
Steps to recreate the issue:
-------------------------------------
The target machine exposes 100 zvols as iSCSI LUNs. The Initiator discovers
these LUNS, creates file system on them, and runs traffic (iozone). The target
machine panics after few hours.

OS: FreeBSD 12.0-CURRENT (svn r329019)
I'm using GENERIC config file, with "nooptions       VIMAGE"

==>Start the Target (FreeBSD)
    1. Create 100 zvols
      # zpool create iscsi <slice/partition>
      # for ((i=0;i<100;i++)); do zfs create -V1G -o volmode=dev iscsi/d$i;
done

    2. configure the network interface to be used (Target_IP_Address)

    2. /etc/ctl.conf file:
         portal-group pg0 {
            discovery-auth-group no-authentication
            listen <Target_IP_Address>
         }
         target iqn.2016-11.com.xyz.abc:0 {
            auth-group no-authentication
            portal-group pg0
                     lun 0 {
                              path /dev/zvol/iscsi/d0
                              size 1G
                              option vendor "foo"
                              option product "bar"
                              option revision "d0"
                     }
         }
         .
         .

        100 targets with 1 lun each

    3. start the target daemon 
        # service ctld onestart

==>Initiator (Linux)
    1. login/discover the target 
        # iscsiadm -m discovery -t st -p <Target_IP_Address> -l
    2. List all Luns 
        # lsscsi        
    3. Format and create file system ONLY on all discovered luns (TAKE CARE TO
NOT TOUCH OTHER LUNS). 
        # mkfs.ext3 /dev/sdxx      // for all discovered LUNs
    4. mount all the devices formatted in the above step to different mount
points
        # mount /dev/sdxx /mnt/iscsiyy    // for all the luns formatted in the
above step
    5. Run traffic on all luns using iozone
        # cd /mnt/iscsiyy
        # iozone -a -I -+d -g 512m &


==> Target machine panics after 3 to 10 hours.

-- 
You are receiving this mail because:
You are the assignee for the bug.