kern/179932: [ciss] ciss i/o stall problem with HP Bl Gen8 (and HP Bl Gen7 Storage Blade)

Philipp Maechler philipp.maechler at hostpoint.ch
Wed Jun 26 11:20:01 UTC 2013


The following reply was made to PR kern/179932; it has been noted by GNATS.

From: Philipp Maechler <philipp.maechler at hostpoint.ch>
To: sbruno at freebsd.org
Cc: bug-followup at FreeBSD.org, Sean Bruno <sean_bruno at yahoo.com>
Subject: Re: kern/179932: [ciss] ciss i/o stall problem with HP Bl Gen8 (and
 HP Bl Gen7   Storage Blade)
Date: Wed, 26 Jun 2013 13:12:40 +0200

 This is a multi-part message in MIME format.
 --------------070005030706090006050108
 Content-Type: text/plain; charset=ISO-8859-1
 Content-Transfer-Encoding: 7bit
 
 hi Sean
 
 At the moment we are running the serial console at 9600 baud; we have to
 tune this to 115200 baud and also then an alltrace will take "forever"
 because of hundreds of apache threads; because this machines are
 productive and non-redundant we have to figure out a procedure how we
 can handle to do this long-running alltrace without interrupting the
 services for such a time.
 
 As an attachement, I can provide you an old debugger session with a
 (part) of a ps and some backtraces. It is from one of our HP
 Gen7-StorageBlade-Systems running on 9.1-RELEASE or higher, hanging
 earlier this year with the same problem.
 
 If you find there enough information to say "alltrace is not needed",
 we'd be happy. But until then we will search for possibilities to do
 such an alltrace...
 
 The break possibility we already enabled by setting the sysctl
   debug.kdb.break_to_debugger=1
 
 As soon as you send us the mentioned patch we will begin to test it and
 introduce it to the affected systems.
 
 Philipp
 
 p.s. please note; even there are a lot of processes in some zfs states,
 we also tried to do dd to some raw partitions on disk and were not
 successful, so we think it's not a zfs problem. In gstat we saw the wait
 queue's stall at specific values and nothing going on anymore.
 
 --------------070005030706090006050108
 Content-Type: text/plain; charset=UTF-8;
  name="20130626_olddebugsession.txt"
 Content-Transfer-Encoding: 7bit
 Content-Disposition: attachment;
  filename="20130626_olddebugsession.txt"
 
 Notes:
 * the ps is not complete, we think there is a part missing; but it gives you an overview
 * are you looking for specific processes or threads? please tell us!
 
 </>hpiLO-> vsp      
 
 Virtual Serial Port Active: COM1
 
 Starting virtual serial port.
 Press 'ESC (' to return to the CLI Session.
 
 
 Tracing pid 12 tid 100043 td 0xfffffe0019ce0000
 kdb_enter() at kdb_enter+0x3b
 kdb_break() at kdb_break+0x27
 scgetc() at scgetc+0x361
 sckbdevent() at sckbdevent+0xed
 kbdmux_intr() at kbdmux_intr+0x43
 kbdmux_kbd_intr() at kbdmux_kbd_intr+0x20
 taskqueue_run_locked() at taskqueue_run_locked+0x85
 taskqueue_run() at taskqueue_run+0x3a
 intr_event_execute_handlers() at intr_event_execute_handlers+0x104
 ithread_loop() at ithread_loop+0xa6
 fork_exit() at fork_exit+0x11f
 fork_trampoline() at fork_trampoline+0xe
 --- trap 0, rip = 0, rsp = 0xffffff80003ffcf0, rbp = 0 ---
 db> ps
   pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
 87125 87123 87125     0  Ds+     zfs      0xfffffe05c0c80db8 bash
 87123  1683 87123     0  Ss      select   0xfffffe0a5d29a8c0 sshd
 87122 87120 87122     0  Ds+     zfs      0xfffffe05c0c80db8 bash
 87120  1683 87120     0  Ss      select   0xfffffe0a5d4603c0 sshd
 86929 86927 86929     0  Ds      zio->io_ 0xfffffe02a13b4d70 atrun
 86928 86926 86928     2  Ds      zio->io_ 0xfffffe08a73d5a00 sh
 86927  1687  1687     0  S       piperd   0xfffffe02f036a5b0 cron
 86926  1687  1687     0  S       piperd   0xfffffe050c0cb5b0 cron
 86738 86736 86738     0  Ds+     zio->io_ 0xfffffe03f1d81690 bash
 86736  1683 86736     0  Ss      select   0xfffffe0b7f10b7c0 sshd
 86729 86728 86729   181  Ds      zfs      0xfffffe026a16d458 python
 86728 86726 86726   181  S       select   0xfffffe02996c7a40 sshd
 86726  1683 86726     0  Ss      sbwait   0xfffffe03a1d6ee8c sshd
 86725 86724 86725   181  Ds      zfs      0xfffffe026a16d458 python
 86724 86722 86722   181  S       select   0xfffffe0aa6b7e840 sshd
 86722  1683 86722     0  Ss      sbwait   0xfffffe0c4f90ae8c sshd
 86721 86719 86718   181  S       piperd   0xfffffe064c9d8b60 wc
 86720 86719 86718   181  D       spa_name 0xffffffff817f7d00 zpool
 86719 86718 86718   181  S       wait     0xfffffe0e9fe53000 bash
 86718 86717 86718   181  Ss      piperd   0xfffffe0604ecd2d8 bash
 86717 86715 86715   181  S       select   0xfffffe03761605c0 sshd
 86715  1683 86715     0  Ss      sbwait   0xfffffe0d3d8a83ec sshd
 86535 86534 86535   181  Ds      zfs      0xfffffe026a16d458 python
 86534 86528 86528   181  S       select   0xfffffe081b6a1040 sshd
 86533 86532 86533   181  Ds      zfs      0xfffffe026a16d458 python
 86532 86526 86526   181  S       select   0xfffffe02815b85c0 sshd
 86531 86530 86531   181  Ds      zfs      0xfffffe026a16d458 python
 86530 86524 86524   181  S       select   0xfffffe048360e7c0 sshd
 86528  1683 86528     0  Ss      sbwait   0xfffffe035cbb0e8c sshd
 86526  1683 86526     0  Ss      sbwait   0xfffffe006acb03ec sshd
 86524  1683 86524     0  Ss      sbwait   0xfffffe0030653694 sshd
 86328 86327 86328   181  Ds      zfs      0xfffffe026a16d458 python
 86327 86323 86323   181  S       select   0xfffffe0a8855a4c0 sshd
 86325 86324 86325   181  Ds      zfs      0xfffffe026a16d458 python
 86324 86315 86315   181  S       select   0xfffffe09ed0b7cc0 sshd
 86323  1683 86323     0  Ss      sbwait   0xfffffe07afe3d144 sshd
 86322 86321 86322   181  Ds      zfs      0xfffffe026a16d458 python
 86321 86311 86311   181  S       select   0xfffffe0e3265b840 sshd
 86320 86319 86320   181  Ds      zfs      0xfffffe026a16d458 python
 86319 86304 86304   181  S       select   0xfffffe088678d2c0 sshd
 86318 86317 86318   181  Ds      zfs      0xfffffe026a16d458 python
 86317 86303 86303   181  S       select   0xfffffe02418f3640 sshd
 86315  1683 86315     0  Ss      sbwait   0xfffffe02a72e23ec sshd
 86313 86312 86313   181  Ds      zfs      0xfffffe026a16d458 python
 86312 86297 86297   181  S       select   0xfffffe009dcc10c0 sshd
 86311  1683 86311     0  Ss      sbwait   0xfffffe013d385144 sshd
 86310 86309 86310   181  Ds      zfs      0xfffffe026a16d458 python
 86309 86301 86301   181  S       select   0xfffffe01fd72d3c0 sshd
 86308 86307 86308   181  Ds      zfs      0xfffffe026a16d458 python
 86307 86298 86298   181  S       select   0xfffffe0483fa1a40 sshd
 86304  1683 86304     0  Ss      sbwait   0xfffffe02544f5144 sshd
 86303  1683 86303     0  Ss      sbwait   0xfffffe0ddf800694 sshd
 86301  1683 86301     0  Ss      sbwait   0xfffffe0b4ef6b144 sshd
 86298  1683 86298     0  Ss      sbwait   0xfffffe0beda78e8c sshd
 86297  1683 86297     0  Ss      sbwait   0xfffffe0a36823144 sshd
 86292 86291 86292   181  Ds      zfs      0xfffffe026a16d458 python
 86291 86285 86285   181  S       select   0xfffffe0d0524b640 sshd
 86290 86289 86290   181  Ds      zfs      0xfffffe026a16d458 python
 86289 86283 86283   181  S       select   0xfffffe0f480e1cc0 sshd
 86288 86287 86288   181  Ds      zio->io_ 0xfffffe039861f690 python
 86287 86281 86281   181  S       select   0xfffffe04cf9c80c0 sshd
 86285  1683 86285     0  Ss      sbwait   0xfffffe03fe528694 sshd
 86283  1683 86283     0  Ss      sbwait   0xfffffe0311e9593c sshd
 86281  1683 86281     0  Ss      sbwait   0xfffffe0d3d8a8be4 sshd
 86279  1668  1668     0  D       zio->io_ 0xfffffe028063ad70 exim-4.80.1-0
 86091 86089 86088   181  S       piperd   0xfffffe06da459b60 wc
 86090 86089 86088   181  D       spa_name 0xffffffff817f7d00 zpool
 86089 86088 86088   181  S       wait     0xfffffe0a88c16000 bash
 86088 86087 86088   181  Ss      piperd   0xfffffe050c0cb888 bash
 86087 86085 86085   181  S       select   0xfffffe0bb51f8cc0 sshd
 86085  1683 86085     0  Ss      sbwait   0xfffffe02ab86e3ec sshd
 86080 86078 83614     0  S       piperd   0xfffffe0e58ef85b0 grep
 86079 86078 83614     0  D       spa_name 0xffffffff817f7d00 zfs
 86078 83628 83614     0  S       wait     0xfffffe0a99074000 bash
 85728 83566 49671     0  D       tx->tx_s 0xfffffe00209eee20 zfs
 85332 85321 85332     0  Ss+     ttyin    0xfffffe0524a144a8 bash
 85321  1683 85321     0  Ss      select   0xfffffe0b122beb40 sshd
 83628 83621 83614     0  S       piperd   0xfffffe013ff055b0 bash
 83621 83614 83614     0  S       wait     0xfffffe0ef81f74a0 bash
 83614 83610 83614     0  Ss      piperd   0xfffffe0ad77c7000 sh
 83610  1687  1687     0  S       piperd   0xfffffe03d14ffb60 cron
 83566 49854 49671     0  S       wait     0xfffffe0eae6ab000 bash
 49854 49685 49671     0  S       wait     0xfffffe0d80e8b4a0 bash
 49685 49671 49671     0  S       wait     0xfffffe0caf48d940 bash
 49671 49665 49671     0  Ss      piperd   0xfffffe0cd7726b60 sh
 49665  1687  1687     0  S       piperd   0xfffffe0b5d9482d8 cron
 34311 34306 34311     0  Ss+     ttyin    0xfffffe002c9388a8 bash
 34306  1683 34306     0  Ss      select   0xfffffe079c3c0d40 sshd
 24912 24564 24912     0  Ss+     ttyin    0xfffffe03e79d2ca8 bash
 24564  1683 24564     0  Ss      select   0xfffffe0f6a802540 sshd
 64399 63540 34618    88  S       (threaded)                  mysqld
 105150                   S       uwait    0xfffffe022c939900 mysqld
 105149                   S       uwait    0xfffffe0603a95280 mysqld
 105148                   S       uwait    0xfffffe0e6d323300 mysqld
 105147                   S       uwait    0xfffffe0f11b16680 mysqld
 105146                   S       sigwait  0xfffffe086602d000 mysqld
 105145                   S       uwait    0xfffffe03c8eff880 mysqld
 105144                   S       uwait    0xfffffe07370e5100 mysqld
 105143                   S       uwait    0xfffffe0ef7fa9b00 mysqld
 105142                   S       uwait    0xfffffe02d415ba00 mysqld
 105140                   S       uwait    0xfffffe0e58e9a100 mysqld
 105139                   S       uwait    0xfffffe06c9cfd000 mysqld
 105138                   S       uwait    0xfffffe01f5eb7e00 mysqld
 105137                   S       uwait    0xfffffe0de3773100 mysqld
 105136                   S       uwait    0xfffffe077c241900 mysqld
 105135                   S       uwait    0xfffffe03dae38800 mysqld
 105134                   S       uwait    0xfffffe09069d7400 mysqld
 105133                   S       uwait    0xfffffe045cc16980 mysqld
 105132                   S       uwait    0xfffffe083cacc000 mysqld
 105131                   S       uwait    0xfffffe02ece6fa80 mysqld
 102539                   S       select   0xfffffe099d9a6040 mysqld
 63540     1 34618     0  S       wait     0xfffffe004fa1d940 sh
 47863     1 47863     0  Ss+     ttyin    0xfffffe001e47e8a8 getty
 11143     1 11143    88  DsJ     (threaded)                  mysqld
 105622                   S       uwait    0xfffffe028377db00 mysqld
 100617                   S       uwait    0xfffffe0d3a2d7080 mysqld
 100557                   S       sbwait   0xfffffe00ae8d5144 mysqld
 105432                   S       sbwait   0xfffffe0466736694 mysqld
 105431                   S       sbwait   0xfffffe05950073ec mysqld
 105401                   S       sbwait   0xfffffe069c2dbe8c mysqld
 105365                   D       zio->io_ 0xfffffe0cbafe1a00 mysqld
 104814                   S       uwait    0xfffffe08b0699180 mysqld
 104813                   S       uwait    0xfffffe0b4ee56900 mysqld
 104812                   S       sbwait   0xfffffe017e6fd144 mysqld
 104811                   S       sbwait   0xfffffe094050e694 mysqld
 104810                   S       sbwait   0xfffffe004e27abe4 mysqld
 104773                   S       sbwait   0xfffffe06b343e144 mysqld
 104772                   S       sbwait   0xfffffe059500be8c mysqld
 104771                   S       sbwait   0xfffffe06a97ba93c mysqld
 104766                   S       sbwait   0xfffffe059500ae8c mysqld
 104765                   S       uwait    0xfffffe052bc88c00 mysqld
 104764                   S       sbwait   0xfffffe02ad61a694 mysqld
 104763                   S       sbwait   0xfffffe01fab54694 mysqld
 104762                   S       sbwait   0xfffffe0912318694 mysqld
 104759                   S       sbwait   0xfffffe06b343ebe4 mysqld
 104756                   S       uwait    0xfffffe084b1e4400 mysqld
 104738                   S       umtxn    0xfffffe07bbb1b700 mysqld
 103653                   S       uwait    0xfffffe0393be2280 mysqld
 103643                   S       uwait    0xfffffe0b4e03a600 mysqld
 103633                   S       sbwait   0xfffffe04b04b693c mysqld
 103534                   D       zio->io_ 0xfffffe0d5d939690 mysqld
 103217                   S       sbwait   0xfffffe0e588afe8c mysqld
 103216                   S       sbwait   0xfffffe077e407be4 mysqld
 103128                   S       uwait    0xfffffe043b164c00 mysqld
 103126                   S       uwait    0xfffffe0426a71b00 mysqld
 103125                   S       sbwait   0xfffffe006a0f63ec mysqld
 103105                   S       sbwait   0xfffffe02ad61a93c mysqld
 102497                   D       zio->io_ 0xfffffe0b5de39320 mysqld
 102496                   S       sbwait   0xfffffe006acb0e8c mysqld
 102487                   S       sbwait   0xfffffe077e4063ec mysqld
 102475                   S       sbwait   0xfffffe040f987e8c mysqld
 102427                   S       sbwait   0xfffffe00a35643ec mysqld
 102394                   S       uwait    0xfffffe09ed0c0e00 mysqld
 102360                   S       sbwait   0xfffffe0beda7b694 mysqld
 102357                   S       sbwait   0xfffffe0e6673ee8c mysqld
 102356                   S       sigwait  0xfffffe0b7e213000 mysqld
 102355                   D       zio->io_ 0xfffffe03c317cd70 mysqld
 102354                   S       uwait    0xfffffe0bad460880 mysqld
 102353                   D       zio->io_ 0xfffffe0d09658690 mysqld
 102352                   S       uwait    0xfffffe0bad379b00 mysqld
 102350                   S       uwait    0xfffffe079c3b5680 mysqld
 102349                   S       uwait    0xfffffe09ed530400 mysqld
 102348                   S       uwait    0xfffffe09edae6100 mysqld
 102347                   S       uwait    0xfffffe0c4f119580 mysqld
 102346                   S       uwait    0xfffffe0de1a35780 mysqld
 102345                   S       uwait    0xfffffe0c4f11a900 mysqld
 102344                   S       uwait    0xfffffe0c4f116c00 mysqld
 102343                   S       uwait    0xfffffe079c2f1480 mysqld
 102342                   S       uwait    0xfffffe09ed935d80 mysqld
 102341                   S       uwait    0xfffffe079c1e6880 mysqld
 102299                   S       select   0xfffffe0b1cec4140 mysqld
  4970  4968  4970     0  Ss+     ttyin    0xfffffe02b83830a8 bash
  4968  1683  4968     0  Ss      select   0xfffffe0bad48b840 sshd
  2777  2183  2777     0  S+      kqread   0xfffffe027cd92900 tail
  2183  2181  2183     0  Ss+     wait     0xfffffe002c5a1940 bash
  2181  1683  2181     0  Ss      select   0xfffffe01e42f4140 sshd
  2180     1     1     0  S       ttydcd   0xfffffe001e4758e8 getty
  2178     1  2178     0  Ss+     ttyin    0xfffffe001e47bca8 getty
  2177     1  2177     0  Ss+     ttyin    0xfffffe001e47b4a8 getty
  2176     1  2176     0  Ss+     ttyin    0xfffffe001e47b0a8 getty
  2175     1  2175     0  Ss+     ttyin    0xfffffe001e47a8a8 getty
  2174     1  2174     0  Ss+     ttyin    0xfffffe001e47a4a8 getty
  2173     1  2173     0  Ss+     ttyin    0xfffffe001e4768a8 getty
  2172     1  2172     0  Ss+     ttyin    0xfffffe001e476ca8 getty
  2171     1  2171     0  DLs+    zio->io_ 0xfffffe0325905d70 login
  2129     1  2129    88  DsJ     (threaded)                  mysqld
 108446                   S       umtxn    0xfffffe0d99f8c380 mysqld
 108436                   S       umtxn    0xfffffe05f7e9cb00 mysqld
 108431                   S       umtxn    0xfffffe0a23934200 mysqld
 108413                   S       umtxn    0xfffffe09e6cedb00 mysqld
 108397                   S       umtxn    0xfffffe02d4458080 mysqld
 108389                   S       umtxn    0xfffffe0ab413ea00 mysqld
 108352                   S       umtxn    0xfffffe07612efe00 mysqld
 108350                   S       umtxn    0xfffffe0ddf748180 mysqld
 108332                   S       umtxn    0xfffffe0d89b09100 mysqld
 108326                   S       umtxn    0xfffffe0254472880 mysqld
 108322                   S       umtxn    0xfffffe09dd542300 mysqld
 108318                   S       umtxn    0xfffffe09371be200 mysqld
 108316                   S       umtxn    0xfffffe09371be380 mysqld
 108311                   S       umtxn    0xfffffe0e95eff700 mysqld
 108294                   S       umtxn    0xfffffe0a886e0680 mysqld
 108263                   S       umtxn    0xfffffe0ee0175780 mysqld
 108253                   S       umtxn    0xfffffe0ee0175a80 mysqld
 108239                   S       umtxn    0xfffffe0573530a80 mysqld
 108236                   S       umtxn    0xfffffe0573530d80 mysqld
 108234                   S       umtxn    0xfffffe096063cc80 mysqld
 108214                   S       umtxn    0xfffffe07afb6ac80 mysqld
 108213                   S       umtxn    0xfffffe018c8dbc00 mysqld
 108204                   S       umtxn    0xfffffe0ec8cffc80 mysqld
 108192                   S       umtxn    0xfffffe0aec5fd500 mysqld
 108167                   S       umtxn    0xfffffe020ce9a600 mysqld
 108164                   S       umtxn    0xfffffe094f908e00 mysqld
 108145                   S       umtxn    0xfffffe01361ccc80 mysqld
 108113                   S       umtxn    0xfffffe03b7288200 mysqld
 108104                   S       umtxn    0xfffffe0543900500 mysqld
 108086                   S       umtxn    0xfffffe0575f95a80 mysqld
 108077                   S       umtxn    0xfffffe02f9862c80 mysqld
 108069                   S       umtxn    0xfffffe0f6b7d3480 mysqld
 108055                   S       umtxn    0xfffffe0967e77300 mysqld
 108054                   S       umtxn    0xfffffe0d397a8080 mysqld
 108050                   S       umtxn    0xfffffe013f554980 mysqld
 108038                   S       umtxn    0xfffffe01248d0780 mysqld
 108035                   S       umtxn    0xfffffe06e288f100 mysqld
 108025                   S       umtxn    0xfffffe0299997880 mysqld
 108022                   S       umtxn    0xfffffe0444f7ed80 mysqld
 107991                   S       umtxn    0xfffffe0b0769bb00 mysqld
 107969                   S       umtxn    0xfffffe01bf108480 mysqld
 107944                   S       umtxn    0xfffffe0a230e2b80 mysqld
 107916                   S       umtxn    0xfffffe017e2b4c80 mysqld
 107884                   S       umtxn    0xfffffe017e2b4a80 mysqld
 107883                   S       umtxn    0xfffffe0418fc7480 mysqld
 107871                   S       umtxn    0xfffffe03acb51b00 mysqld
 107855                   S       umtxn    0xfffffe08ed201c80 mysqld
 107824                   S       umtxn    0xfffffe0aece27d00 mysqld
 107820                   S       umtxn    0xfffffe0bb1988c00 mysqld
 107809                   S       umtxn    0xfffffe0de2182a00 mysqld
 107794                   S       umtxn    0xfffffe06ad810380 mysqld
 107787                   S       umtxn    0xfffffe0f56112100 mysqld
 107781                   S       umtxn    0xfffffe0f56112380 mysqld
 107736                   S       umtxn    0xfffffe05a29c6100 mysqld
 107717                   S       umtxn    0xfffffe0c179dd980 mysqld
 107716                   S       umtxn    0xfffffe0650e87a00 mysqld
 107709                   S       umtxn    0xfffffe0650e87c80 mysqld
 107690                   S       umtxn    0xfffffe0ed40b6880 mysqld
 107674                   S       umtxn    0xfffffe02c3e7bc80 mysqld
 107664                   S       umtxn    0xfffffe02c3e7b080 mysqld
 107645                   S       umtxn    0xfffffe04ff252e00 mysqld
 107641                   S       umtxn    0xfffffe01d77a3800 mysqld
 107640                   S       umtxn    0xfffffe04bd459400 mysqld
 107629                   S       umtxn    0xfffffe02cfeba700 mysqld
 107600                   S       umtxn    0xfffffe03406e8980 mysqld
 107598                   S       umtxn    0xfffffe0d6cdd5c00 mysqld
 107581                   S       umtxn    0xfffffe03406e8880 mysqld
 107580                   S       umtxn    0xfffffe0b1cee4d80 mysqld
 107569                   S       umtxn    0xfffffe09e6c14580 mysqld
 107567                   S       umtxn    0xfffffe08b2a70500 mysqld
 107563                   S       umtxn    0xfffffe03df90e100 mysqld
 107562                   S       umtxn    0xfffffe0cdbee6580 mysqld
 107556                   S       umtxn    0xfffffe0bbab0d200 mysqld
 107527                   S       umtxn    0xfffffe0fb315cd00 mysqld
 107526                   S       umtxn    0xfffffe057338e580 mysqld
 107525                   S       umtxn    0xfffffe0bbab0db00 mysqld
 107517                   S       umtxn    0xfffffe01db5b7700 mysqld
 107516                   S       umtxn    0xfffffe01db5b7900 mysqld
 107507                   S       umtxn    0xfffffe0e7bc38c80 mysqld
 107472                   S       umtxn    0xfffffe0e7bc38a00 mysqld
 107460                   S       umtxn    0xfffffe02f05c9000 mysqld
 107449                   S       umtxn    0xfffffe02cbde6d00 mysqld
 107443                   S       umtxn    0xfffffe0e64653d80 mysqld
 107422                   S       umtxn    0xfffffe0dc6963700 mysqld
 107420                   S       umtxn    0xfffffe06d8f20680 mysqld
 107418                   S       umtxn    0xfffffe02a812e480 mysqld
 107417                   S       umtxn    0xfffffe035e4cdb00 mysqld
 107403                   S       umtxn    0xfffffe016ca31300 mysqld
 107399                   S       umtxn    0xfffffe0d73b0bb00 mysqld
 107382                   S       umtxn    0xfffffe03d4208980 mysqld
 107352                   S       umtxn    0xfffffe0eb2423d00 mysqld
 107340                   S       umtxn    0xfffffe05c0dc6280 mysqld
 107337                   S       umtxn    0xfffffe0b0f27c600 mysqld
 107333                   S       umtxn    0xfffffe0a519d8700 mysqld
 107312                   S       umtxn    0xfffffe0c5451da80 mysqld
 107308                   S       umtxn    0xfffffe0f0df60b00 mysqld
 107305                   S       umtxn    0xfffffe06487eb980 mysqld
 107303                   S       umtxn    0xfffffe0a3df35980 mysqld
 107299                   S       umtxn    0xfffffe0f58fea700 mysqld
 107291                   S       umtxn    0xfffffe0a5f6b0300 mysqld
 107278                   S       umtxn    0xfffffe020095be00 mysqld
 107277                   S       umtxn    0xfffffe03783a2200 mysqld
 107264                   S       umtxn    0xfffffe0595e24d00 mysqld
 107260                   S       umtxn    0xfffffe0d989d6400 mysqld
 107255                   S       umtxn    0xfffffe0c25a91380 mysqld
 107254                   S       umtxn    0xfffffe0dfc812080 mysqld
 107253                   S       umtxn    0xfffffe08ed8a6a80 mysqld
 107251                   S       umtxn    0xfffffe039ee19700 mysqld
 107249                   S       umtxn    0xfffffe039ee19980 mysqld
 107221                   S       umtxn    0xfffffe0d807c2e00 mysqld
 107215                   S       umtxn    0xfffffe0a4d784380 mysqld
 107207                   S       umtxn    0xfffffe06e90da980 mysqld
 107204                   S       umtxn    0xfffffe0cc3992780 mysqld
 107200                   S       umtxn    0xfffffe0512754d80 mysqld
 107190                   S       umtxn    0xfffffe0d7a47b100 mysqld
 107173                   S       umtxn    0xfffffe03a16c5c80 mysqld
 107171                   S       umtxn    0xfffffe0404607c00 mysqld
 107170                   S       umtxn    0xfffffe0cc8c81980 mysqld
 107162                   S       umtxn    0xfffffe0ad8a51580 mysqld
 107157                   S       umtxn    0xfffffe0631cc2380 mysqld
 107152                   S       umtxn    0xfffffe0de3bd9500 mysqld
 107129                   S       umtxn    0xfffffe0e32d8d600 mysqld
 107121                   S       umtxn    0xfffffe0e32d8dc00 mysqld
 107120                   S       umtxn    0xfffffe012738b780 mysqld
 107110                   S       umtxn    0xfffffe0e589c6880 mysqld
 107107                   S       umtxn    0xfffffe088617f880 mysqld
 107101                   S       umtxn    0xfffffe0937ba2c80 mysqld
 107098                   S       umtxn    0xfffffe022e48bd80 mysqld
 107085                   S       umtxn    0xfffffe0886039980 mysqld
 107084                   S       umtxn    0xfffffe094f042080 mysqld
 107069                   S       umtxn    0xfffffe03df31b180 mysqld
 107045                   S       umtxn    0xfffffe023a19a780 mysqld
 107036                   S       umtxn    0xfffffe07a002ed80 mysqld
 107035                   S       umtxn    0xfffffe021bcee280 mysqld
 107022                   S       umtxn    0xfffffe0a51abcc80 mysqld
 107007                   S       umtxn    0xfffffe0da93f9400 mysqld
 107006                   S       umtxn    0xfffffe0687561300 mysqld
 106997                   S       umtxn    0xfffffe0b76c23280 mysqld
 106995                   S       umtxn    0xfffffe0faf70b800 mysqld
 106985                   S       umtxn    0xfffffe04ff405680 mysqld
 106980                   S       umtxn    0xfffffe0b7f5ba680 mysqld
 106973                   S       umtxn    0xfffffe0393be2c00 mysqld
 106965                   S       umtxn    0xfffffe03f50fcc00 mysqld
 106962                   S       umtxn    0xfffffe0e8034bb80 mysqld
 106933                   S       umtxn    0xfffffe03b4bb0680 mysqld
 106932                   S       umtxn    0xfffffe0dc1ac0680 mysqld
 106916                   S       umtxn    0xfffffe0abcaab980 mysqld
 106883                   S       umtxn    0xfffffe043db24300 mysqld
 106876                   S       umtxn    0xfffffe072773be00 mysqld
 106871                   S       umtxn    0xfffffe0e1d447100 mysqld
 106863                   S       umtxn    0xfffffe07a052b780 mysqld
 106847                   S       umtxn    0xfffffe0ad78da800 mysqld
 106844                   S       umtxn    0xfffffe0445acfc00 mysqld
 106842                   S       umtxn    0xfffffe025bba2e00 mysqld
 106841                   S       umtxn    0xfffffe036539c280 mysqld
 106810                   S       umtxn    0xfffffe058719b800 mysqld
 106786                   S       umtxn    0xfffffe0e32703880 mysqld
 106781                   S       umtxn    0xfffffe03dfed7c00 mysqld
 106779                   S       umtxn    0xfffffe0c686f7a00 mysqld
 106769                   S       umtxn    0xfffffe01bf889c00 mysqld
 106760                   S       umtxn    0xfffffe07da3bb800 mysqld
 106754                   S       umtxn    0xfffffe044e35a800 mysqld
 106744                   S       umtxn    0xfffffe03d6e88900 mysqld
 106714                   S       umtxn    0xfffffe075f052180 mysqld
 106710                   S       umtxn    0xfffffe0d80cd7a80 mysqld
 106692                   S       umtxn    0xfffffe03df96a500 mysqld
 106690                   S       umtxn    0xfffffe04bd2fcc80 mysqld
 106673                   S       umtxn    0xfffffe0cc75d1300 mysqld
 106656                   S       umtxn    0xfffffe03b58d0180 mysqld
 106650                   S       umtxn    0xfffffe0516d16480 mysqld
 106647                   S       umtxn    0xfffffe056ea70d80 mysqld
 106643                   S       umtxn    0xfffffe0f8a322b00 mysqld
 106641                   S       umtxn    0xfffffe0d9834e480 mysqld
 106633                   S       umtxn    0xfffffe02306f7900 mysqld
 106604                   S       umtxn    0xfffffe034dc04000 mysqld
 106583                   S       umtxn    0xfffffe0d7aad5880 mysqld
 106579                   S       umtxn    0xfffffe0abc16a600 mysqld
 106561                   S       umtxn    0xfffffe03d6188680 mysqld
 106537                   S       umtxn    0xfffffe01afebe000 mysqld
 106521                   S       umtxn    0xfffffe0360772700 mysqld
 106516                   S       umtxn    0xfffffe025a7a4900 mysqld
 106506                   S       umtxn    0xfffffe03423fb480 mysqld
 106495                   S       umtxn    0xfffffe06ea2d2280 mysqld
 106488                   S       umtxn    0xfffffe0a2f142d00 mysqld
 106487                   S       umtxn    0xfffffe02f0a14c80 mysqld
 106479                   S       umtxn    0xfffffe0ddf63c780 mysqld
 106438                   S       umtxn    0xfffffe07be277100 mysqld
 106433                   S       umtxn    0xfffffe037a37cc00 mysqld
 106428                   S       umtxn    0xfffffe0d0eb3eb80 mysqld
 106397                   S       umtxn    0xfffffe0d7a07c200 mysqld
 106390                   S       umtxn    0xfffffe094f76d600 mysqld
 106379                   S       umtxn    0xfffffe014423ab00 mysqld
 106377                   S       umtxn    0xfffffe0721e68000 mysqld
 106376                   S       umtxn    0xfffffe0d47d1a400 mysqld
 106364                   S       umtxn    0xfffffe07a07d2980 mysqld
 106348                   S       umtxn    0xfffffe0293963b00 mysqld
 106311                   S       umtxn    0xfffffe03264fa080 mysqld
 106302                   S       umtxn    0xfffffe02306f6780 mysqld
 106274                   S       umtxn    0xfffffe0f60f58200 mysqld
 106272                   S       umtxn    0xfffffe084f6f9100 mysqld
 106250                   S       umtxn    0xfffffe09304f3300 mysqld
 106242                   S       umtxn    0xfffffe005fa65180 mysqld
 106241                   S       umtxn    0xfffffe009d3aea00 mysqld
 106236                   S       umtxn    0xfffffe0fb3ffbc80 mysqld
 106235                   S       umtxn    0xfffffe058d8a7480 mysqld
 106218                   S       umtxn    0xfffffe0aefd36180 mysqld
 106191                   S       umtxn    0xfffffe009d683880 mysqld
 106187                   S       umtxn    0xfffffe024d76dc80 mysqld
 106186                   S       umtxn    0xfffffe09bda65400 mysqld
 106185                   S       umtxn    0xfffffe0ae627d780 mysqld
 106175                   S       umtxn    0xfffffe0432633600 mysqld
 106144                   S       umtxn    0xfffffe048381ae00 mysqld
 106143                   S       umtxn    0xfffffe09bfb5c980 mysqld
 db> bt 86738
 Tracing pid 86738 tid 102823 td 0xfffffe0cc7ecc470
 sched_switch() at sched_switch+0x13d
 mi_switch() at mi_switch+0x196
 sleepq_wait() at sleepq_wait+0x42
 _cv_wait() at _cv_wait+0x121
 zio_wait() at zio_wait+0x61
 dmu_buf_hold_array_by_dnode() at dmu_buf_hold_array_by_dnode+0x22b
 dmu_buf_hold_array() at dmu_buf_hold_array+0x67
 dmu_read_uio() at dmu_read_uio+0x3f
 zfs_freebsd_read() at zfs_freebsd_read+0x3e3
 VOP_READ_APV() at VOP_READ_APV+0x44
 vn_read() at vn_read+0x316
 dofileread() at dofileread+0xa1
 kern_readv() at kern_readv+0x6c
 sys_read() at sys_read+0x64
 amd64_syscall() at amd64_syscall+0x5ea
 Xfast_syscall() at Xfast_syscall+0xf7
 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x4eeafc, rsp = 0x7fffffffd888, rbp = 0x3 ---
 db> bt 86279
 Tracing pid 86279 tid 104231 td 0xfffffe0360fd3470
 sched_switch() at sched_switch+0x13d
 mi_switch() at mi_switch+0x196
 sleepq_wait() at sleepq_wait+0x42
 _cv_wait() at _cv_wait+0x121
 zio_wait() at zio_wait+0x61
 dmu_tx_count_write() at dmu_tx_count_write+0x34b
 dmu_tx_hold_write() at dmu_tx_hold_write+0x4a
 zfs_freebsd_write() at zfs_freebsd_write+0x43b
 VOP_WRITE_APV() at VOP_WRITE_APV+0xc6
 vn_write() at vn_write+0x38c
 dofilewrite() at dofilewrite+0x8b
 kern_writev() at kern_writev+0x6c
 sys_write() at sys_write+0x64
 amd64_syscall() at amd64_syscall+0x5ea
 Xfast_syscall() at Xfast_syscall+0xf7
 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x802aa0b2c, rsp = 0x7fffffffc488, rbp = 0x7fffffffc4b0 ---
 db> bt 102355
 Tracing pid 11143 tid 102355 td 0xfffffe0c8214a8e0
 sched_switch() at sched_switch+0x13d
 mi_switch() at mi_switch+0x196
 sleepq_wait() at sleepq_wait+0x42
 _cv_wait() at _cv_wait+0x121
 zio_wait() at zio_wait+0x61
 zil_commit() at zil_commit+0x3e5
 zfs_freebsd_fsync() at zfs_freebsd_fsync+0xdc
 VOP_FSYNC_APV() at VOP_FSYNC_APV+0x4a
 sys_fsync() at sys_fsync+0x187
 amd64_syscall() at amd64_syscall+0x5ea
 Xfast_syscall() at Xfast_syscall+0xf7
 --- syscall (95, FreeBSD ELF64, sys_fsync), rip = 0x80237848c, rsp = 0x7ffffdfefd98, rbp = 0x7ffffdfefdb0 ---
 db> bt 86292
 Tracing pid 86292 tid 101376 td 0xfffffe006152c000
 sched_switch() at sched_switch+0x13d
 mi_switch() at mi_switch+0x196
 sleepq_wait() at sleepq_wait+0x42
 __lockmgr_args() at __lockmgr_args+0x610
 vop_stdlock() at vop_stdlock+0x39
 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x52
 _vn_lock() at _vn_lock+0x47
 vget() at vget+0x70
 cache_lookup_times() at cache_lookup_times+0x5f5
 vfs_cache_lookup() at vfs_cache_lookup+0xc8
 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x4c
 lookup() at lookup+0x424
 namei() at namei+0x535
 vn_open_cred() at vn_open_cred+0x3cb
 kern_openat() at kern_openat+0x1f9
 amd64_syscall() at amd64_syscall+0x5ea
 Xfast_syscall() at Xfast_syscall+0xf7
 --- syscall (5, FreeBSD ELF64, sys_open), rip = 0x8010a00ac, rsp = 0x7fffffffbd28, rbp = 0x7fffffffbdc0 ---
 db> bt 87125
 Tracing pid 87125 tid 103485 td 0xfffffe0cb6330000
 sched_switch() at sched_switch+0x13d
 mi_switch() at mi_switch+0x196
 sleepq_wait() at sleepq_wait+0x42
 __lockmgr_args() at __lockmgr_args+0x610
 vop_stdlock() at vop_stdlock+0x39
 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x52
 _vn_lock() at _vn_lock+0x47
 vget() at vget+0x70
 cache_lookup_times() at cache_lookup_times+0x5f5
 vfs_cache_lookup() at vfs_cache_lookup+0xc8
 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x4c
 lookup() at lookup+0x424
 namei() at namei+0x535
 vn_open_cred() at vn_open_cred+0x3cb
 kern_openat() at kern_openat+0x1f9
 amd64_syscall() at amd64_syscall+0x5ea
 Xfast_syscall() at Xfast_syscall+0xf7
 --- syscall (5, FreeBSD ELF64, sys_open), rip = 0x4e1a1c, rsp = 0x7fffffffd8b8, rbp = 0 ---
 db> bt 86928
 Tracing pid 86928 tid 104507 td 0xfffffe0e2ffdb000
 sched_switch() at sched_switch+0x13d
 mi_switch() at mi_switch+0x196
 sleepq_wait() at sleepq_wait+0x42
 _cv_wait() at _cv_wait+0x121
 zio_wait() at zio_wait+0x61
 dbuf_read() at dbuf_read+0x5e5
 dmu_buf_hold() at dmu_buf_hold+0xe0
 zap_lockdir() at zap_lockdir+0x58
 zap_lookup_norm() at zap_lookup_norm+0x45
 zap_lookup() at zap_lookup+0x2e
 zfs_dirent_lock() at zfs_dirent_lock+0x4ff
 zfs_dirlook() at zfs_dirlook+0x69
 zfs_lookup() at zfs_lookup+0x26b
 zfs_freebsd_lookup() at zfs_freebsd_lookup+0x81
 VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0x44
 vfs_cache_lookup() at vfs_cache_lookup+0xf8
 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x4c
 lookup() at lookup+0x424
 namei() at namei+0x535
 kern_statat_vnhook() at kern_statat_vnhook+0xb6
 kern_statat() at kern_statat+0x15
 sys_stat() at sys_stat+0x2a
 amd64_syscall() at amd64_syscall+0x5ea
 Xfast_syscall() at Xfast_syscall+0xf7
 --- syscall (188, FreeBSD ELF64, sys_stat), rip = 0x800da971c, rsp = 0x7fffffffd6c8, rbp = 0x7fffffffd7f0 ---
 db> bt 86720
 Tracing pid 86720 tid 104539 td 0xfffffe0ad6cce8e0
 sched_switch() at sched_switch+0x13d
 mi_switch() at mi_switch+0x196
 sleepq_wait() at sleepq_wait+0x42
 _sx_xlock_hard() at _sx_xlock_hard+0x4f1
 _sx_xlock() at _sx_xlock+0x75
 spa_all_configs() at spa_all_configs+0x5e
 zfs_ioc_pool_configs() at zfs_ioc_pool_configs+0x29
 zfsdev_ioctl() at zfsdev_ioctl+0xe6
 devfs_ioctl_f() at devfs_ioctl_f+0x7b
 kern_ioctl() at kern_ioctl+0x115
 sys_ioctl() at sys_ioctl+0xfd
 amd64_syscall() at amd64_syscall+0x5ea
 Xfast_syscall() at Xfast_syscall+0xf7
 --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x801df0aec, rsp = 0x7fffffff83b8, rbp = 0x7fffffff8400 ---
 db> bt 86079 
 Tracing pid 86079 tid 104370 td 0xfffffe06f80cd8e0
 sched_switch() at sched_switch+0x13d
 mi_switch() at mi_switch+0x196
 sleepq_wait() at sleepq_wait+0x42
 _sx_xlock_hard() at _sx_xlock_hard+0x4f1
 _sx_xlock() at _sx_xlock+0x75
 spa_open_common() at spa_open_common+0x7a
 pool_status_check() at pool_status_check+0x19
 zfsdev_ioctl() at zfsdev_ioctl+0x223
 devfs_ioctl_f() at devfs_ioctl_f+0x7b
 kern_ioctl() at kern_ioctl+0x115
 sys_ioctl() at sys_ioctl+0xfd
 amd64_syscall() at amd64_syscall+0x5ea
 Xfast_syscall() at Xfast_syscall+0xf7
 --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x801befaec, rsp = 0x7fffffff8ed8, rbp = 0x7fffffff8f20 ---
 db> bt 85728
 Tracing pid 85728 tid 105053 td 0xfffffe0294bc5470
 sched_switch() at sched_switch+0x13d
 mi_switch() at mi_switch+0x196
 sleepq_wait() at sleepq_wait+0x42
 _cv_wait() at _cv_wait+0x121
 txg_wait_synced() at txg_wait_synced+0x85
 dsl_sync_task_group_wait() at dsl_sync_task_group_wait+0x128
 dsl_dataset_user_release() at dsl_dataset_user_release+0x1c9
 dsl_dataset_user_release_tmp() at dsl_dataset_user_release_tmp+0xee
 dsl_dataset_user_release_onexit() at dsl_dataset_user_release_onexit+0x21
 zfs_onexit_destroy() at zfs_onexit_destroy+0x56
 zfsdev_close() at zfsdev_close+0x4f
 devfs_destroy_cdevpriv() at devfs_destroy_cdevpriv+0x69
 devfs_close_f() at devfs_close_f+0x54
 _fdrop() at _fdrop+0x23
 closef() at closef+0x52
 kern_close() at kern_close+0x172
 amd64_syscall() at amd64_syscall+0x5ea
 Xfast_syscall() at Xfast_syscall+0xf7
 --- syscall (6, FreeBSD ELF64, sys_close), rip = 0x801befb0c, rsp = 0x7fffffffc5a8, rbp = 0x7fffffffcff0 ---
 
 --------------070005030706090006050108--


More information about the freebsd-scsi mailing list