KDB: debugger backends: ddb KDB: current backend: ddb ---<>--- Copyright (c) 1992-2020 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 13.0-CURRENT #2 r367253M: Sun Nov 1 19:54:47 CET 2020 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-rc2-0-g77d76b71d7d) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. VT(vga): resolution 640x480 CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz 686-class CPU) Origin="AuthenticAMD" Id=0x100f23 Family=0x10 Model=0x2 Stepping=3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff SVM: NP,NAsids=64 TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 3522457600 (3359 MB) Event timer "LAPIC" quality 100 ACPI APIC Table: <060508 APIC1048> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) random: unblocking device. ioapic0 irqs 0-23 Launching APs: 2 3 1 Timecounter "TSC" frequency 1800019412 Hz quality 800 random: entropy device external interface WARNING: Device "kbd" is Giant locked and may be deleted before FreeBSD 13.0. kbd1 at kbdmux0 [ath_hal] loaded nexus0 vtvga0: cryptosoft0: acpi0: <060508 RSDT1048> acpi0: Power Button (fixed) cpu0: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 vgapci0: port 0xc000-0xc0ff mem 0xf0000000-0xf7ffffff,0xfe9f0000-0xfe9fffff,0xfe800000-0xfe8fffff irq 18 at device 5.0 on pci1 vgapci0: Boot video device hdac0: mem 0xfe9e8000-0xfe9ebfff irq 19 at device 5.2 on pci1 pcib2: at device 7.0 on pci0 pci2: on pcib2 re0: port 0xd800-0xd8ff mem 0xfeaff000-0xfeafffff irq 19 at device 0.0 on pci2 re0: Using 1 MSI message re0: Chip rev. 0x38000000 re0: MAC rev. 0x00000000 miibus0: on re0 rgephy0: PHY 1 on miibus0 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re0: Using defaults for TSO: 65518/35/2048 re0: Ethernet address: 00:21:85:15:33:7d ahci0: port 0xb000-0xb007,0xa000-0xa003,0x9000-0x9007,0x8000-0x8003,0x7000-0x700f mem 0xfe7ff800-0xfe7ffbff irq 22 at device 18.0 on pci0 ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier supported ahci0: quirks=0x7000 ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ohci0: mem 0xfe7fe000-0xfe7fefff irq 16 at device 19.0 on pci0 usbus0 on ohci0 usbus0: 12Mbps Full Speed USB v1.0 ohci1: mem 0xfe7fd000-0xfe7fdfff irq 17 at device 19.1 on pci0 usbus1 on ohci1 usbus1: 12Mbps Full Speed USB v1.0 ohci2: mem 0xfe7fc000-0xfe7fcfff irq 18 at device 19.2 on pci0 usbus2 on ohci2 usbus2: 12Mbps Full Speed USB v1.0 ohci3: mem 0xfe7fb000-0xfe7fbfff irq 17 at device 19.3 on pci0 usbus3 on ohci3 usbus3: 12Mbps Full Speed USB v1.0 ohci4: mem 0xfe7fa000-0xfe7fafff irq 18 at device 19.4 on pci0 usbus4 on ohci4 usbus4: 12Mbps Full Speed USB v1.0 ehci0: mem 0xfe7ff000-0xfe7ff0ff irq 19 at device 19.5 on pci0 ehci0: AMD SB600/700 quirk applied usbus5: EHCI version 1.0 usbus5 on ehci0 usbus5: 480Mbps High Speed USB v2.0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0 ata0: at channel 0 on atapci0 hdac1: mem 0xfe7f4000-0xfe7f7fff irq 16 at device 20.2 on pci0 isab0: at device 20.3 on pci0 isa0: on isab0 pcib3: at device 20.4 on pci0 pci3: on pcib3 pci3: at device 6.0 (no driver attached) acpi_button0: on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (115200,n,8,1) fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] ppc0: parallel port not found. hwpstate0: on cpu0 NULL mp in getnewvnode(9), tag crossmp Timecounters tick every 1.000 msec ugen1.1: at usbus1 ugen3.1: at usbus3 uhub0ugen2.1: at usbus2 ugen0.1: at usbus0 on usbus1 uhub1 on usbus2 uhub0: on usbus1 uhub1: on usbus2 uhub2 on usbus0 uhub2: on usbus0 uhub3 on usbus3 uhub3: on usbus3 ugen5.1: at usbus5 hdacc0: at cad 0 on hdac0 hdaa0: at nid 1 on hdacc0 pcm0: at nid 3 on hdaa0 hdacc1: at cad 3 on hdac1 hdaa1: at nid 1 on hdacc1 pcm1: at nid 20,22,21,23 and 24,26 on hdaa1 pcm2: at nid 27 and 25 on hdaa1 pcm3: at nid 30 on hdaa1 Trying to mount root from ufs:/dev/ada0s1a [rw]... Root mount waiting for: CAMWARNING: WITNESS option enabled, expect reduced performance. usbus0WARNING: DIAGNOSTIC option enabled, expect reduced performance. usbus1ugen4.1: at usbus4 usbus2uhub4 usbus3 on usbus5 usbus4 usbus5 uhub5 on usbus4 Expensive timeout(9) function: 0xa3eea0(0x18010000) 0.009124064 s uhub4: on usbus5 (aprobe0:ahcich0:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00 uhub5: on usbus4 (aprobe0:ahcich0:0:15:0): CAM status: Command timeout (aprobe0:ahcich0:0:15:0): Error 5, Retries exhausted (aprobe0:ahcich0:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00 uhub0: 2 ports with 2 removable, self powered (aprobe0:ahcich0:0:15:0): CAM status: Command timeout uhub1: 2 ports with 2 removable, self powered (aprobe0:ahcich0:0:15:0): Error 5, Retries exhausted uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ATA8-ACS SATA 2.x device ada0: Serial Number 9VMJ99G5 ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 476940MB (976773168 512 byte sectors) uhub5: 2 ports with 2 removable, self powered Root mount waiting for: usbus5 Root mount waiting for: usbus5 Root mount waiting for: usbus5 Root mount waiting for: usbus5 uhub4: 10 ports with 10 removable, self powered mountroot: waiting for device /dev/ada0s1a... WARNING: / was not properly dismounted Dual Console: Serial Primary, Video Secondary Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. warning: total configured swap (4194304 pages) exceeds maximum recommended amount (2009952 pages). warning: increase kern.maxswzone or reduce amount of swap. Starting file system checks: /dev/ada0s1a: 7922 files, 403732 used, 609283 free (2115 frags, 75896 blocks, 0.2% fragmentation) /dev/ada0s1f: 18337 files, 236728 used, 776287 free (12007 frags, 95535 blocks, 1.2% fragmentation) /dev/ada0s1d: 555674 files, 9129899 used, 1024300 free (363772 frags, 82566 blocks, 3.6% fragmentation) ** SU+J Recovering /dev/ada0s1e ** Reading 33554432 byte journal from inode 6. ** Building recovery table. ** Resolving unreferenced inode list. ** Processing journal entries. ** 10 journal records in 2560 bytes for 12.50% utilization ** Freed 0 inodes (0 dirs) 0 blocks, and 0 frags. ***** FILE SYSTEM MARKED CLEAN ***** /dev/label/tmp: 727479 files, 81272566 used, 37041415 free (26023 frags, 4626924 blocks, 0.0% fragmentation) Mounting local filesystems:. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/arduino/lib /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/e2fsprogs /usr/local/lib/gcc7 /usr/local/lib/gcc8 /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/perl5/5.28/mach/CORE /usr/local/lib/pth /usr/local/lib/qt4 /usr/local/lib/qt5 /usr/local/llvm37/lib /usr/local/llvm39/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout ldconfig: aout support is deprecated Setting hostname: x4.osted.lan. Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy: . lo0: link state changed to UP re0: link state changed to DOWN Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=8209b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 media: Ethernet autoselect (none) status: no carrier nd6 options=29 Starting devd. add host 127.0.0.1: gateway lo0 fib 0: route already in table add net default: gateway 192.168.1.254 add host ::1: gateway lo0 fib 0: route already in table add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Creating and/or trimming log files. Starting syslogd. Starting rpcbind. NFS access cache time=60 Starting watchdogd. No core dumps found. Clearing /tmp (X related). Starting nfsuserd. Starting mountd. Starting nfsd. Starting statd. Starting lockd. Recovering vi editor sessions:. Updating motd:. Mounting late filesystems:. Starting ntpd. Starting powerd. Starting smartd. Nov 2 18:35:57 x4 smartd[639]: Device: /dev/ada0, WARNING: A firmware update for this drive may be available, Nov 2 18:35:57 x4 smartd[639]: see the following Seagate web pages: Nov 2 18:35:57 x4 smartd[639]: http://knowledge.seagate.com/articles/en_US/FAQ/207931en Nov 2 18:35:57 x4 smartd[639]: http://knowledge.seagate.com/articles/en_US/FAQ/213891en Configuring vt: keymap blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting sendmail_submit. Starting sendmail_msp_queue. Starting cron. Local package initialization: backup. Starting nfscbd. Starting default mousedmoused: unable to open /dev/psm0: No such file or directory . Starting inetd. Mon Nov 2 18:36 FreeBSD/i386 (x4.osted.lan) (ttyu0) login: Nov 2 18:38:42 x4 su[945]: pho to root on /dev/pts/1 20201102 18:39:08 all (1/240): mkfifo4.sh stress2: freework leak: 20/50. stress2: newblk leak: 49134/49183. stress2: pts leak: 1/3 stress2: indirdep leak: 26/27 20201102 18:46:31 all (2/240): crossmp4.sh lock order reversal: 1st 0x2ad19a84 syncer (syncer, lockmgr) @ kern/vfs_subr.c:2522 2nd 0x2a7cd794 ufs (ufs, lockmgr) @ kern/vfs_subr.c:3131 lock order ufs -> syncer established at: #0 0x1046ea5 at witness_checkorder+0x3c5 #1 0xfb98f2 at lockmgr_lock_flags+0x132 #2 0x109af44 at vop_stdlock+0x24 #3 0x13e7828 at VOP_LOCK1_APV+0x38 #4 0x10c381f at _vn_lock+0x4f #5 0x10ad9c7 at vput_final+0x107 #6 0x10a5ab5 at dounmount+0x3f5 #7 0x10a565d at kern_unmount+0x2dd #8 0x10a5373 at sys_unmount+0x13 #9 0x13d1798 at syscall+0x1b8 #10 0xffc033f9 at null_bug_bypass+0xd39fd045 lock order syncer -> ufs attempted at: #0 0x1047530 at witness_checkorder+0xa50 #1 0xfb98f2 at lockmgr_lock_flags+0x132 #2 0x12682b6 at ffs_lock+0x76 #3 0x13e7828 at VOP_LOCK1_APV+0x38 #4 0x10c381f at _vn_lock+0x4f #5 0x10ad9c7 at vput_final+0x107 #6 0x10ad39b at vget_finish+0x8b #7 0x10ad300 at vget+0x50 #8 0x12666cb at ffs_sync+0x4fb #9 0x10b79c1 at sync_fsync+0xe1 #10 0x13e669d at VOP_FSYNC_APV+0x4d #11 0x10b509c at sched_sync+0x3dc #12 0xfa73e8 at fork_exit+0x68 #13 0xffc0340e at null_bug_bypass+0xd39fd05a Nov 2 18:53:10 x4 kernel: pid 50737 (rw), uid 1004 inumber 321153 on /mnt: filesystem full lock order reversal: 1st 0x2045161c ufs (ufs, lockmgr) @ kern/vfs_mount.c:1711 2nd 0x289e061c devfs (devfs, lockmgr) @ kern/vfs_subr.c:3131 lock order devfs -> ufs established at: #0 0x1046ea5 at witness_checkorder+0x3c5 #1 0xfb98f2 at lockmgr_lock_flags+0x132 #2 0x12682b6 at ffs_lock+0x76 #3 0x13e7828 at VOP_LOCK1_APV+0x38 #4 0x10c381f at _vn_lock+0x4f #5 0x10a4d45 at vfs_domount+0xf75 #6 0x10a31db at vfs_donmount+0x75b #7 0x10a7037 at kernel_mount+0x57 #8 0x10a9292 at parse_mount+0x452 #9 0x10a7a6e at vfs_mountroot+0x4ce #10 0xf81e3b at start_init+0x1b #11 0xfa73e8 at fork_exit+0x68 #12 0xffc0340e at null_bug_bypass+0xd39fd05a lock order ufs -> devfs attempted at: #0 0x1047530 at witness_checkorder+0xa50 #1 0xfb98f2 at lockmgr_lock_flags+0x132 #2 0x109c6fe at vop_lock+0x5e #3 0x13e7828 at VOP_LOCK1_APV+0x38 #4 0x10c381f at _vn_lock+0x4f #5 0x10ad9c7 at vput_final+0x107 #6 0x1265e9b at ffs_unmount+0x39b #7 0x10a5ac3 at dounmount+0x403 #8 0x10a565d at kern_unmount+0x2dd #9 0x10a5373 at sys_unmount+0x13 #10 0x13d1798 at syscall+0x1b8 #11 0xffc033f9 at null_bug_bypass+0xd39fd045 20201102 18:55:16 all (3/240): fsck4.sh stress2: freework leak: 10/17. stress2: newblk leak: 11086/11092. stress2: mount leak: 1/98 stress2: indirdep leak: 28/28 20201102 18:56:07 all (4/240): procfs.sh stress2: mount leak: 1/99 20201102 18:57:04 all (5/240): sched.sh stress2: newblk leak: 965/971. stress2: indirdep leak: 3/3 20201102 19:04:55 all (6/240): pthread3.sh 20201102 19:14:54 all (7/240): pread.sh Nov 2 19:15:17 x4 mountd[565]: can't export /mnt lock order reversal: (sleepable after non-sleepable) 1st 0x9ae8e04 seqflq (seqflq, sleep mutex) @ dev/sound/midi/sequencer.c:757 2nd 0x1bb2548 midistat lock (midistat lock, sx) @ dev/sound/midi/midi.c:1481 lock order seqflq -> midistat lock attempted at: #0 0x1047530 at witness_checkorder+0xa50 #1 0xfee39d at _sx_xlock+0x4d #2 0xdc488a at midimapper_open+0x1a #3 0xdc71dd at mseq_open+0xdd #4 0xeaa275 at devfs_open+0x115 #5 0x13e500d at VOP_OPEN_APV+0x4d #6 0x10c35cf at vn_open_vnode+0x16f #7 0x10c2ff5 at vn_open_cred+0x395 #8 0x10c2c50 at vn_open+0x20 #9 0x10ba3ba at kern_openat+0x2fa #10 0x10ba5cf at sys_openat+0x2f #11 0x13d1798 at syscall+0x1b8 #12 0xffc033f9 at fdhashtbl+0xd39f7a99 Nov 2 19:15:38 x4 syslogd: last message repeated 2 times stress2: mount leak: 3/102 20201102 19:15:59 all (8/240): churn.sh stress2: pts leak: 1/4 20201102 19:26:05 all (9/240): inversion.sh 20201102 19:36:12 all (10/240): poll2.sh 20201102 19:46:18 all (11/240): callout_reset_on2.sh 20201102 19:56:26 all (12/240): core5.sh stress2: freework leak: 6/16. stress2: newblk leak: 26/32. stress2: mount leak: 1/103 stress2: indirdep leak: 1/1 20201102 20:06:36 all (13/240): badcode2.sh Nov 2 20:06:53 x4 kernel: pid 35547 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 20:07:03 x4 kernel: pid 35580 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 20:07:14 x4 kernel: pid 35571 (swap), jid 0, uid 1004, was killed: out of swap space lock order reversal: 1st 0x1df3c00 cpslk (cpslk, sx) @ i386/i386/copyout.c:121 2nd 0x18d3e21c bufwait (bufwait, lockmgr) @ vm/vm_pager.c:391 lock order bufwait -> cpslk established at: #0 0x1046ea5 at witness_checkorder+0x3c5 #1 0xfee39d at _sx_xlock+0x4d #2 0x13aa87c at cp_slow0+0x12c #3 0x13aaba5 at copyout+0xa5 #4 0x1040d8c at uiomove_faultflag+0x13c #5 0x1040c43 at uiomove+0x13 #6 0x10c47e3 at vn_io_fault_uiomove+0x33 #7 0xef2439 at ncl_bioread+0xab9 #8 0xee7b40 at nfs_readdir+0x190 #9 0x109cbd7 at vop_sigdefer+0x27 #10 0xee3030 at nfs_vnodeops_bypass+0x10 #11 0x13e717d at VOP_READDIR_APV+0x4d #12 0x10c03e2 at kern_getdirentries+0x1b2 #13 0x10c05f5 at sys_getdirentries+0x25 #14 0x13d1798 at syscall+0x1b8 #15 0xffc033f9 at fdhashtbl+0xd39f7a99 lock order cpslk -> bufwait attempted at: #0 0x1047530 at witness_checkorder+0xa50 #1 0xfb9f5a at lockmgr_xlock_hard+0x4a #2 0xfbacaf at __lockmgr_args+0x6cf #3 0x12add19 at pbuf_ctor+0x79 #4 0x128984c at item_ctor+0x37c #5 0x1282e29 at uma_zalloc_arg+0x119 #6 0x127f4ce at swap_pager_getpages_locked+0x2ee #7 0x127afe5 at swap_pager_getpages+0x35 #8 0x12adf44 at vm_pager_get_pages+0x44 #9 0x128adf9 at vm_fault+0x589 #10 0x128a752 at vm_fault_trap+0x52 #11 0x13d1301 at trap_pfault+0x161 #12 0x13d08b0 at trap+0x3a0 #13 0xffc0319f at fdhashtbl+0xd39f783f #14 0x28 at ll+0x7 Nov 2 20:11:08 x4 kernel: pid 48836 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 20:11:19 x4 kernel: pid 48811 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 20:13:13 x4 kernel: pid 50249 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 20:13:25 x4 kernel: pid 50251 (swap), jid 0, uid 1004, was killed: out of swap space stress2: mount leak: 1/104 20201102 20:17:14 all (14/240): dangling.sh stress2: mount leak: 1/105 20201102 20:27:33 all (15/240): ext2fs2.sh stress2: mount leak: 1/106 20201102 20:37:50 all (16/240): msdos.sh stress2: mount leak: 1/107 20201102 20:48:02 all (17/240): quota1.sh Nov 2 20:48:29 x4 kernel: pid 9625 (rw), uid 0 inumber 15894 on /mnt: filesystem full Nov 2 20:48:31 x4 kernel: pid 9633 (creat), uid 0 inumber 5 on /mnt: out of inodes Nov 2 20:48:42 x4 syslogd: last message repeated 1 times Nov 2 20:48:57 x4 kernel: pid 9628 (mkdir), uid 0 inumber 107 on /mnt: out of inodes Nov 2 20:49:17 x4 kernel: pid 9633 (creat), uid 0 inumber 5 on /mnt: out of inodes Nov 2 20:49:25 x4 syslogd: last message repeated 1 times Nov 2 20:49:31 x4 kernel: pid 9636 (creat), uid 0 inumber 7 on /mnt: out of inodes Nov 2 20:49:57 x4 kernel: pid 9633 (creat), uid 0 inumber 5 on /mnt: out of inodes Nov 2 20:50:02 x4 kernel: pid 9631 (mkdir), uid 0 inumber 12259 on /mnt: out of inodes Nov 2 20:52:52 x4 kernel: pid 30726 (rw), uid 0 inumber 13189 on /mnt: filesystem full Nov 2 20:54:15 x4 kernel: pid 41525 (mkdir), uid 0 inumber 12405 on /mnt: out of inodes Nov 2 20:54:35 x4 kernel: pid 41535 (mkdir), uid 0 inumber 12287 on /mnt: out of inodes Nov 2 20:55:55 x4 kernel: pid 41530 (mkdir), uid 0 inumber 8922 on /mnt: out of inodes Nov 2 20:56:15 x4 kernel: pid 51433 (rw), uid 0 inumber 42 on /mnt: filesystem full Nov 2 20:56:58 x4 kernel: pid 51437 (rw), uid 0 inumber 30 on /mnt: out of inodes stress2: freework leak: 11/20. stress2: mount leak: 1/108 20201102 20:58:16 all (18/240): mmap3.sh stress2: newblk leak: 594/600. stress2: indirdep leak: 25/25 20201102 21:08:33 all (19/240): select.sh 20201102 21:19:00 all (20/240): tmpfs3.sh Nov 2 21:19:19 x4 kernel: pid 72705 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 21:19:39 x4 kernel: pid 72686 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 21:20:19 x4 kernel: pid 72665 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 21:21:25 x4 kernel: pid 75432 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 21:21:50 x4 kernel: pid 75433 (swap), jid 0, uid 0, was killed: out of swap space stress2: mount leak: 1/109 20201102 21:29:32 all (21/240): fsgs.sh 20201102 21:30:45 all (22/240): mkfifo3.sh 20201102 21:31:52 all (23/240): sem_wait.sh 20201102 21:32:58 all (24/240): sigstop2.sh 20201102 21:34:04 all (25/240): socketpair3.sh 20201102 21:35:10 all (26/240): pts.sh Nov 2 21:35:30 x4 kernel: pid 68491 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 21:35:51 x4 kernel: pid 68479 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 21:37:42 x4 kernel: pid 71796 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 21:43:48 x4 kernel: pid 83916 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 21:44:45 x4 kernel: pid 83906 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 21:44:47 x4 kernel: pid 83922 (swap), jid 0, uid 0, was killed: out of swap space stress2: pts leak: 63/67 20201102 21:45:44 all (27/240): sendfile8.sh lock order reversal: 1st 0x2445ef20 so_snd_sx (so_snd_sx, sx) @ kern/uipc_sockbuf.c:464 2nd 0x25d0761c ufs (ufs, lockmgr) @ kern/kern_sendfile.c:850 lock order so_snd_sx -> ufs attempted at: #0 0x1047530 at witness_checkorder+0xa50 #1 0xfb9827 at lockmgr_lock_flags+0x67 #2 0x12682b6 at ffs_lock+0x76 #3 0x13e7828 at VOP_LOCK1_APV+0x38 #4 0x10c381f at _vn_lock+0x4f #5 0xfe1f11 at vn_sendfile+0x7c1 #6 0xfe3f15 at sendfile+0x125 #7 0xfe3de0 at sys_sendfile+0x10 #8 0x13d1798 at syscall+0x1b8 #9 0xffc033f9 at fdhashtbl+0xd39f7a99 stress2: mount leak: 1/110 20201102 21:46:52 all (28/240): udp.sh 20201102 21:47:00 all (29/240): sendfile7.sh Nov 2 21:47:17 x4 kernel: pid 94538 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 21:49:44 x4 kernel: pid 95688 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 21:49:55 x4 kernel: pid 95684 (swap), jid 0, uid 0, was killed: out of swap space lock order reversal: 1st 0x238b8d74 tmpfs (tmpfs, lockmgr) @ kern/kern_sendfile.c:850 2nd 0x18d3e91c bufwait (bufwait, lockmgr) @ vm/vm_pager.c:391 lock order tmpfs -> bufwait attempted at: #0 0x1047530 at witness_checkorder+0xa50 #1 0xfb9f5a at lockmgr_xlock_hard+0x4a #2 0xfbacaf at __lockmgr_args+0x6cf #3 0x12add19 at pbuf_ctor+0x79 #4 0x128984c at item_ctor+0x37c #5 0x1282e29 at uma_zalloc_arg+0x119 #6 0x127f4ce at swap_pager_getpages_locked+0x2ee #7 0x127b026 at swap_pager_getpages_async+0x36 #8 0xfe2876 at vn_sendfile+0x1126 #9 0xfe3f15 at sendfile+0x125 #10 0xfe3de0 at sys_sendfile+0x10 #11 0x13d1798 at syscall+0x1b8 #12 0xffc033f9 at fdhashtbl+0xd39f7a99 Nov 2 21:51:23 x4 kernel: pid 96834 (sendfile5), jid 0, uid 0, was killed: out of swap space Nov 2 21:51:35 x4 kernel: pid 96812 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 21:51:54 x4 kernel: pid 96831 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 21:52:05 x4 kernel: pid 96828 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 21:53:39 x4 kernel: pid 97946 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 21:56:39 x4 kernel: pid 98203 (sendfile5), jid 0, uid 0, was killed: out of swap space Nov 2 21:57:00 x4 kernel: pid 213 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 21:57:47 x4 kernel: pid 312 (sendfile5), jid 0, uid 0, was killed: out of swap space stress2: mount leak: 3/113 20201102 21:59:47 all (30/240): badcode.sh Nov 2 22:02:47 x4 kernel: pid 14211 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 22:02:57 x4 kernel: pid 14228 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 22:04:17 x4 kernel: pid 17469 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 22:04:26 x4 kernel: pid 17497 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 22:04:37 x4 kernel: pid 17511 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 22:06:33 x4 kernel: pid 18840 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 22:06:45 x4 kernel: pid 18891 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 22:06:56 x4 kernel: pid 18885 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 22:07:48 x4 kernel: pid 18888 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 22:07:54 x4 kernel: pid 18895 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 22:08:05 x4 kernel: pid 18887 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 22:08:07 x4 kernel: pid 18892 (swap), jid 0, uid 1004, was killed: out of swap space stress2: mount leak: 1/114 20201102 22:10:46 all (31/240): md3.sh Nov 2 22:13:13 x4 kernel: pid 24770 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:13:23 x4 kernel: pid 24767 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:13:31 x4 kernel: pid 24798 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:13:31 x4 kernel: pid 24765 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:13:41 x4 kernel: pid 24772 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:15:32 x4 kernel: pid 27085 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:15:42 x4 kernel: pid 27081 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:15:53 x4 kernel: pid 27093 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:19:49 x4 kernel: pid 30739 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:19:59 x4 kernel: pid 30768 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:20:10 x4 kernel: pid 30733 (swap), jid 0, uid 0, was killed: out of swap space swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio Nov 2 22:20:20 x4 kernel: pid 30737 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:20:31 x4 kernel: pid 30765 (swap), jid 0, uid 0, was killed: out of swap space stress2: mount leak: 1/115 20201102 22:21:42 all (32/240): datamove2.sh Nov 2 22:22:19 x4 kernel: pid 33458 (swap), jid 0, uid 0, was killed: out of swap space Expensive timeout(9) function: 0xe21220(0x18014c78) 5.103594393 s Nov 2 22:22:31 x4 kernel: pid 33400 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:22:40 x4 kernel: pid 33460 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:22:50 x4 kernel: pid 33422 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:23:35 x4 kernel: pid 33473 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:24:29 x4 kernel: pid 33448 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:24:48 x4 kernel: pid 33483 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:25:04 x4 kernel: pid 33441 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:25:18 x4 kernel: pid 33477 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:25:57 x4 kernel: pid 33492 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:27:47 x4 kernel: pid 35703 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:28:08 x4 kernel: pid 35669 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:28:46 x4 kernel: pid 35678 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:29:11 x4 kernel: pid 35667 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:29:51 x4 kernel: pid 35652 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:29:53 x4 kernel: pid 35700 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:30:30 x4 kernel: pid 35661 (swap), jid 0, uid 0, was killed: out of swap space stress2: freework leak: 18/26. stress2: newblk leak: 2/8. 20201102 22:32:29 all (33/240): fdgrowtable.sh Nov 2 22:33:51 x4 kernel: pid 39011 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:33:55 x4 kernel: pid 39020 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:33:55 x4 kernel: pid 39015 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:33:55 x4 kernel: pid 39008 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:34:56 x4 kernel: pid 40608 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:35:09 x4 kernel: pid 40609 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:36:58 x4 kernel: pid 42034 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:37:08 x4 kernel: pid 42041 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:39:10 x4 kernel: pid 43616 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:39:26 x4 kernel: pid 43655 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:41:20 x4 kernel: pid 45058 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:41:29 x4 kernel: pid 45046 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:41:54 x4 kernel: pid 45080 (swap), jid 0, uid 0, was killed: out of swap space 20201102 22:43:13 all (34/240): tmpfs.sh Nov 2 22:43:57 x4 kernel: pid 46593 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:44:05 x4 kernel: pid 46595 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:45:54 x4 kernel: pid 47829 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:46:05 x4 kernel: pid 47852 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:46:15 x4 kernel: pid 47839 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:46:25 x4 kernel: pid 47834 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:48:05 x4 kernel: pid 49094 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:48:15 x4 kernel: pid 49163 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:48:28 x4 kernel: pid 49107 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:48:39 x4 kernel: pid 49114 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:48:50 x4 kernel: pid 49158 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:49:01 x4 kernel: pid 49159 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:52:38 x4 kernel: pid 52015 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:52:48 x4 kernel: pid 52013 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:53:00 x4 kernel: pid 52021 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:53:12 x4 kernel: pid 52033 (swap), jid 0, uid 0, was killed: out of swap space Nov 2 22:53:35 x4 kernel: pid 52030 (swap), jid 0, uid 0, was killed: out of swap space stress2: mount leak: 1/116 20201102 22:54:04 all (35/240): fsck.sh stress2: freework leak: 10/18. stress2: newblk leak: 45/51. stress2: mount leak: 1/117 stress2: indirdep leak: 1/1 20201102 22:55:12 all (36/240): rename7.sh 20201102 22:56:28 all (37/240): ext2fs.sh Nov 2 22:56:50 x4 kernel: pid 70319 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 22:57:09 x4 kernel: pid 70294 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:00:59 x4 kernel: pid 72844 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:01:09 x4 kernel: pid 72859 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:01:26 x4 kernel: pid 72838 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:03:05 x4 kernel: pid 75526 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:03:12 x4 kernel: pid 75560 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:03:25 x4 kernel: pid 75559 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:03:30 x4 kernel: pid 75541 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:03:30 x4 kernel: pid 75540 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:03:30 x4 kernel: pid 75539 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:05:09 x4 kernel: pid 76732 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:05:20 x4 kernel: pid 76690 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:05:30 x4 kernel: pid 76725 (swap), jid 0, uid 1004, was killed: out of swap space stress2: mount leak: 1/118 20201102 23:07:10 all (38/240): snap5-1.sh lock order reversal: 1st 0xb816f04 bufwait (bufwait, lockmgr) @ ufs/ffs/ffs_vnops.c:309 2nd 0x27ee131c snaplk (snaplk, lockmgr) @ ufs/ffs/ffs_snapshot.c:2333 lock order snaplk -> bufwait established at: #0 0x1046ea5 at witness_checkorder+0x3c5 #1 0xfb9f5a at lockmgr_xlock_hard+0x4a #2 0xfbacaf at __lockmgr_args+0x6cf #3 0x108a4d0 at getnewbuf+0x4d0 #4 0x108787d at getblkx+0x60d #5 0x1087085 at breadn_flags+0x35 #6 0x1239713 at ffs_getcg+0x103 #7 0x1238d9c at ffs_blkfree_cg+0x10c #8 0x12358af at ffs_blkfree+0x9f #9 0x12447eb at mapacct_ufs2+0x8b #10 0x1243ec1 at expunge_ufs2+0x3e1 #11 0x1241c72 at ffs_snapshot+0x1272 #12 0x12648d1 at ffs_mount+0xb91 #13 0x10a4504 at vfs_domount+0x734 #14 0x10a31db at vfs_donmount+0x75b #15 0x10a2a4d at sys_nmount+0x5d #16 0x13d1798 at syscall+0x1b8 #17 0xffc033f9 at __stop_set_sysinit_set+0xd39f4989 lock order bufwait -> snaplk attempted at: #0 0x1047530 at witness_checkorder+0xa50 #1 0xfb9f5a at lockmgr_xlock_hard+0x4a #2 0xfbacaf at __lockmgr_args+0x6cf #3 0x124696f at ffs_copyonwrite+0x18f #4 0x1267179 at ffs_geom_strategy+0xa9 #5 0x1278a37 at ufs_strategy+0x87 #6 0x13e7c08 at VOP_STRATEGY_APV+0x58 #7 0x1085b8f at bufstrategy+0x3f #8 0x1085a3b at bufwrite+0x1fb #9 0x1268fbe at ffs_syncvnode+0x2ce #10 0x1268167 at ffs_fsync+0x17 #11 0x13e669d at VOP_FSYNC_APV+0x4d #12 0x127682f at ufs_remove+0xbf #13 0x13e6825 at VOP_REMOVE_APV+0x75 #14 0x10bbc79 at kern_funlinkat+0x339 #15 0x10bb92f at sys_unlink+0x1f #16 0x13d1798 at syscall+0x1b8 #17 0xffc033f9 at __stop_set_sysinit_set+0xd39f4989 lock order reversal: 1st 0x27ee131c snaplk (snaplk, lockmgr) @ ufs/ufs/ufs_vnops.c:1031 2nd 0x2338390c ufs (ufs, lockmgr) @ ufs/ffs/ffs_snapshot.c:1656 lock order ufs -> snaplk established at: #0 0x1046ea5 at witness_checkorder+0x3c5 #1 0xfb9827 at lockmgr_lock_flags+0x67 #2 0x12682b6 at ffs_lock+0x76 #3 0x13e7828 at VOP_LOCK1_APV+0x38 #4 0x10c381f at _vn_lock+0x4f #5 0x10ad37a at vget_finish+0x6a #6 0x109eb85 at vfs_hash_get+0x95 #7 0x126346e at ffs_vgetf+0x3e #8 0x1263429 at ffs_vget+0x19 #9 0x126fa18 at ufs_lookup_ino+0xba8 #10 0x126ee66 at ufs_lookup+0x16 #11 0x13e49dd at VOP_CACHEDLOOKUP_APV+0x4d #12 0x1092db8 at vfs_cache_lookup+0x98 #13 0x13e485d at VOP_LOOKUP_APV+0x4d #14 0x10a1020 at lookup+0x580 #15 0x10a049f at namei+0x6df #16 0x10c2f7e at vn_open_cred+0x31e #17 0x10c2c50 at vn_open+0x20 lock order snaplk -> ufs attempted at: #0 0x1047530 at witness_checkorder+0xa50 #1 0xfb9f5a at lockmgr_xlock_hard+0x4a #2 0xfbacaf at __lockmgr_args+0x6cf #3 0x1244d46 at ffs_snapremove+0xc6 #4 0x123ec5e at ffs_truncate+0x29e #5 0x126ebcb at ufs_inactive+0x18b #6 0x13e7440 at VOP_INACTIVE_APV+0x50 #7 0x10ae2fd at vinactivef+0xed #8 0x10adb39 at vput_final+0x279 #9 0x10adbdb at vput+0x4b #10 0x10bbcd3 at kern_funlinkat+0x393 #11 0x10bb92f at sys_unlink+0x1f #12 0x13d1798 at syscall+0x1b8 #13 0xffc033f9 at __stop_set_sysinit_set+0xd39f4989 stress2: freework leak: 11/19. stress2: mount leak: 8/126 20201102 23:17:27 all (39/240): graid1.sh Nov 2 23:17:46 x4 kernel: pid 33996 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:17:58 x4 kernel: pid 33995 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:18:08 x4 kernel: pid 33994 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:19:50 x4 kernel: pid 36175 (swap), jid 0, uid 1004, was killed: out of swap space swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio Nov 2 23:20:00 x4 kernel: pid 36184 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:20:11 x4 kernel: pid 36225 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:20:25 x4 kernel: pid 36181 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:24:04 x4 kernel: pid 41277 (swap), jid 0, uid 1004, was killed: out of swap space swap_pager: cannot allocate bio swap_pager: cannot allocate bio Nov 2 23:24:16 x4 kernel: pid 41325 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:26:11 x4 kernel: pid 42374 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:26:21 x4 kernel: pid 42364 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:26:32 x4 kernel: pid 42372 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:26:44 x4 kernel: pid 42335 (swap), jid 0, uid 1004, was killed: out of swap space stress2: mount leak: 1/127 20201102 23:28:23 all (40/240): sysctl3.sh Nov 2 23:28:53 x4 kernel: pid 44475 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:32:08 x4 kernel: pid 51961 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:32:22 x4 kernel: pid 51984 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:32:27 x4 kernel: pid 51979 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:32:49 x4 kernel: pid 51966 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:33:19 x4 kernel: pid 51981 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:34:27 x4 kernel: pid 52686 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:36:49 x4 kernel: pid 54331 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:37:04 x4 kernel: pid 54409 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:37:14 x4 kernel: pid 54362 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:37:29 x4 kernel: pid 54359 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:37:43 x4 kernel: pid 54353 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:38:25 x4 kernel: pid 54367 (swap), jid 0, uid 1004, was killed: out of swap space 20201102 23:40:34 all (41/240): nullfs26.sh lock order reversal: 1st 0x2900203c tmpfs (tmpfs, lockmgr) @ kern/vfs_subr.c:2914 2nd 0x2527d4a4 nullfs (nullfs, lockmgr) @ /usr/src/sys/fs/nullfs/null_vnops.c:838 lock order tmpfs -> nullfs attempted at: #0 0x1047530 at witness_checkorder+0xa50 #1 0xfb9f5a at lockmgr_xlock_hard+0x4a #2 0xfbacaf at __lockmgr_args+0x6cf #3 0x2c20494c at null_reclaim+0x5c #4 0x13e76d0 at VOP_RECLAIM_APV+0x50 #5 0x10af0a9 at vgonel+0x389 #6 0x10af682 at vrecycle+0x52 #7 0x2c20488f at null_inactive+0x4f #8 0x13e7440 at VOP_INACTIVE_APV+0x50 #9 0x10ae2fd at vinactivef+0xed #10 0x10adb39 at vput_final+0x279 #11 0x10adbdb at vput+0x4b #12 0x10bff54 at kern_frmdirat+0x304 #13 0x10bfc4b at sys_rmdir+0x1b #14 0x13d1798 at syscall+0x1b8 #15 0xffc033f9 at __stop_set_sysinit_set+0xd39f4989 Nov 2 23:43:00 x4 kernel: pid 62631 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:43:10 x4 kernel: pid 62663 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:43:23 x4 kernel: pid 62680 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:44:19 x4 kernel: pid 62650 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:44:55 x4 kernel: pid 62661 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:45:07 x4 kernel: pid 62652 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:45:08 x4 kernel: pid 62675 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:45:08 x4 kernel: pid 62659 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:45:08 x4 kernel: pid 62653 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:45:08 x4 kernel: pid 62646 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:45:08 x4 kernel: pid 62671 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:45:31 x4 kernel: pid 63933 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:45:41 x4 kernel: pid 63972 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:45:51 x4 kernel: pid 63953 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:46:01 x4 kernel: pid 63944 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:46:12 x4 kernel: pid 63949 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:46:23 x4 kernel: pid 63941 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:46:33 x4 kernel: pid 63969 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:46:43 x4 kernel: pid 63964 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:47:48 x4 kernel: pid 65144 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:47:59 x4 kernel: pid 65155 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:48:09 x4 kernel: pid 65163 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:48:19 x4 kernel: pid 65186 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:48:31 x4 kernel: pid 65238 (swap), jid 0, uid 1004, was killed: out of swap space stress2: mount leak: 1/128 20201102 23:51:39 all (42/240): pathconf.sh 20201102 23:51:45 all (43/240): rename15.sh Nov 2 23:52:17 x4 kernel: pid 69225 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:54:21 x4 kernel: pid 70797 (swap), jid 0, uid 1004, was killed: out of swap space swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio Nov 2 23:56:55 x4 kernel: pid 74271 (swap), jid 0, uid 1004, was killed: out of swap space Nov 2 23:57:05 x4 kernel: pid 74273 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:02:00 x4 kernel: pid 79026 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:02:11 x4 kernel: pid 79028 (swap), jid 0, uid 1004, was killed: out of swap space 20201103 00:03:40 all (44/240): mmap21.sh stress2: freework leak: 8/17. stress2: newblk leak: 19795/19802. stress2: indirdep leak: 12/12 20201103 00:04:58 all (45/240): tmpfs21.sh Nov 3 00:05:16 x4 kernel: pid 88562 (swap), jid 0, uid 0, was killed: out of swap space Nov 3 00:05:26 x4 kernel: pid 88555 (swap), jid 0, uid 0, was killed: out of swap space Nov 3 00:05:39 x4 kernel: pid 88549 (swap), jid 0, uid 0, was killed: out of swap space stress2: mount leak: 13/141 20201103 00:06:05 all (46/240): cluster.sh stress2: freework leak: 6/13. stress2: newblk leak: 4153/4159. stress2: indirdep leak: 18/18 20201103 00:17:27 all (47/240): overcommit.sh Nov 3 00:17:57 x4 kernel: pid 43594 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:18:08 x4 kernel: pid 43595 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:18:18 x4 kernel: pid 43672 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:18:28 x4 kernel: pid 43634 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:21:56 x4 kernel: pid 45906 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:22:06 x4 kernel: pid 45910 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:22:16 x4 kernel: pid 45902 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:26:05 x4 kernel: pid 52077 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:26:16 x4 kernel: pid 52126 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:26:26 x4 kernel: pid 52078 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:26:37 x4 kernel: pid 52130 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:26:47 x4 kernel: pid 52190 (swap), jid 0, uid 1004, was killed: out of swap space stress2: mount leak: 1/142 20201103 00:28:08 all (48/240): extattr2.sh Nov 3 00:28:25 x4 kernel: pid 53910 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:28:37 x4 kernel: pid 53920 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:28:57 x4 kernel: pid 53925 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:29:10 x4 kernel: pid 53935 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:30:43 x4 kernel: pid 56124 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:30:53 x4 kernel: pid 56101 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:31:04 x4 kernel: pid 56098 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:31:14 x4 kernel: pid 56102 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:33:11 x4 kernel: pid 59132 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:33:21 x4 kernel: pid 59230 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:33:31 x4 kernel: pid 59222 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:33:41 x4 kernel: pid 59129 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:33:51 x4 kernel: pid 59224 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:34:11 x4 kernel: pid 59227 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:35:40 x4 kernel: pid 60453 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:35:53 x4 kernel: pid 60449 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:36:03 x4 kernel: pid 60445 (swap), jid 0, uid 1004, was killed: out of swap space stress2: mount leak: 1/143 20201103 00:39:53 all (49/240): crossmp8.sh Nov 3 00:39:59 x4 mountd[565]: can't change attributes for /: MNT_DEFEXPORTED already set for mount 0x18ccc080 20201103 00:41:02 all (50/240): mmap2.sh stress2: freework leak: 49/55. stress2: newblk leak: 98/104. stress2: indirdep leak: 52/52 20201103 00:52:00 all (51/240): pfl2.sh swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio Nov 3 00:52:25 x4 kernel: pid 28555 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:52:36 x4 kernel: pid 28546 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:52:46 x4 kernel: pid 28539 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:52:56 x4 kernel: pid 28538 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:53:07 x4 kernel: pid 28533 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:53:17 x4 kernel: pid 28545 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:56:37 x4 kernel: pid 30300 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:57:00 x4 kernel: pid 30317 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:57:10 x4 kernel: pid 30312 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:57:21 x4 kernel: pid 30310 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:57:32 x4 kernel: pid 30309 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:57:54 x4 kernel: pid 30305 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:59:19 x4 kernel: pid 31445 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:59:29 x4 kernel: pid 31456 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:59:41 x4 kernel: pid 31462 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 00:59:51 x4 kernel: pid 31459 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 01:01:47 x4 kernel: pid 32270 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 01:01:57 x4 kernel: pid 32265 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 01:02:08 x4 kernel: pid 32269 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 01:03:36 x4 kernel: pid 32273 (swap), jid 0, uid 1004, was killed: out of swap space 20201103 01:03:59 all (52/240): split.sh stress2: freework leak: 11/18. 20201103 01:26:10 all (53/240): tcp3.sh 20201103 01:27:26 all (54/240): vmstat.sh 20201103 01:28:41 all (55/240): nullfs5.sh Nov 3 01:37:34 x4 kernel: pid 84969 (swap), jid 0, uid 0, was killed: out of swap space Nov 3 01:37:47 x4 kernel: pid 84945 (swap), jid 0, uid 0, was killed: out of swap space Nov 3 01:38:02 x4 kernel: pid 84947 (swap), jid 0, uid 0, was killed: out of swap space Nov 3 01:38:14 x4 kernel: pid 84968 (swap), jid 0, uid 0, was killed: out of swap space stress2: freework leak: 1/8. stress2: newblk leak: 115279/115290. stress2: indirdep leak: 71/71 20201103 01:42:19 all (56/240): marcus7.sh stress2: mount leak: 1/144 20201103 01:54:36 all (57/240): vm_lowmem.sh Nov 3 01:54:51 x4 kernel: pid 53624 (sort), jid 0, uid 0, was killed: out of swap space Nov 3 01:55:01 x4 kernel: pid 53626 (sort), jid 0, uid 0, was killed: out of swap space Nov 3 01:55:11 x4 kernel: pid 53623 (sort), jid 0, uid 0, was killed: out of swap space 20201103 01:55:52 all (58/240): umountf.sh stress2: freework leak: 11/18. stress2: newblk leak: 7032/7039. stress2: mount leak: 1/145 stress2: indirdep leak: 5/5 20201103 01:56:31 all (59/240): suj17.sh stress2: mount leak: 1/146 20201103 02:09:30 all (60/240): readdir.sh Nov 3 02:13:49 x4 mountd[565]: can't export /mnt stress2: mount leak: 4/150 20201103 02:21:52 all (61/240): truncate.sh stress2: freework leak: 40103/40111. stress2: newblk leak: 40104/40110. 20201103 02:32:43 all (62/240): execve.sh 20201103 02:33:45 all (63/240): f_offset.sh 20201103 02:35:07 all (64/240): mmap5.sh 20201103 02:36:51 all (65/240): newfs.sh Nov 3 02:47:01 x4 kernel: pid 41646 (creat), uid 1004 inumber 12295 on /mnt: out of inodes Nov 3 02:47:36 x4 kernel: pid 44669 (creat), uid 1004 inumber 8200 on /mnt: out of inodes Nov 3 02:47:47 x4 kernel: pid 45326 (creat), uid 1004 inumber 2060 on /mnt: out of inodes Nov 3 02:47:48 x4 kernel: pid 45324 (creat), uid 1004 inumber 2064 on /mnt: out of inodes Nov 3 02:51:30 x4 kernel: pid 63604 (mkdir), uid 1004 inumber 12074 on /mnt: out of inodes stress2: mount leak: 80/230 20201103 02:51:51 all (66/240): sendfile3.sh stress2: freework leak: 249/257. stress2: newblk leak: 78189/78196. stress2: indirdep leak: 72/72 20201103 02:53:20 all (67/240): kinfo2.sh 20201103 03:08:27 all (68/240): suj14.sh Nov 3 03:12:17 x4 kernel: pid 12426 (cp), uid 0 inumber 245131 on /mnt: filesystem full Nov 3 03:12:18 x4 kernel: pid 12426 (cp), uid 0 inumber 245149 on /mnt: filesystem full Nov 3 03:12:19 x4 kernel: pid 12426 (cp), uid 0 inumber 245166 on /mnt: filesystem full Nov 3 03:12:20 x4 kernel: pid 13531 (cp), uid 0 inumber 112640 on /mnt: filesystem full Nov 3 03:12:21 x4 kernel: pid 13547 (cp), uid 0 inumber 112639 on /mnt: filesystem full Nov 3 03:12:22 x4 kernel: pid 13563 (cp), uid 0 inumber 112641 on /mnt: filesystem full Nov 3 03:12:23 x4 kernel: pid 13580 (mkdir), uid 0 inumber 112639 on /mnt: filesystem full Nov 3 03:12:24 x4 kernel: pid 13597 (cp), uid 0 inumber 112641 on /mnt: filesystem full stress2: mount leak: 1/231 20201103 03:12:29 all (69/240): suj6.sh stress2: mount leak: 1/232 20201103 03:15:17 all (70/240): md.sh Nov 3 03:15:22 x4 kernel: pid 30494 (rw), uid 0 inumber 138 on /mnt: filesystem full Nov 3 03:15:23 x4 kernel: pid 30504 (rw), uid 0 inumber 130 on /mnt: filesystem full Nov 3 03:15:26 x4 kernel: pid 30490 (rw), uid 0 inumber 6 on /mnt: filesystem full Nov 3 03:15:27 x4 kernel: pid 30493 (rw), uid 0 inumber 129 on /mnt: filesystem full Nov 3 03:15:31 x4 kernel: pid 30502 (rw), uid 0 inumber 130 on /mnt: filesystem full Nov 3 03:15:36 x4 kernel: pid 31558 (rw), uid 0 inumber 142 on /mnt: filesystem full Nov 3 03:15:38 x4 kernel: pid 31557 (rw), uid 0 inumber 142 on /mnt: filesystem full Nov 3 03:15:40 x4 kernel: pid 31566 (rw), uid 0 inumber 140 on /mnt: filesystem full Nov 3 03:15:42 x4 kernel: pid 31553 (rw), uid 0 inumber 5 on /mnt: filesystem full Nov 3 03:17:33 x4 kernel: pid 40919 (rw), uid 0 inumber 138 on /mnt: filesystem full Nov 3 03:17:34 x4 kernel: pid 40924 (rw), uid 0 inumber 138 on /mnt: filesystem full Nov 3 03:17:37 x4 kernel: pid 40914 (rw), uid 0 inumber 6 on /mnt: filesystem full Nov 3 03:17:38 x4 kernel: pid 40917 (rw), uid 0 inumber 6 on /mnt: filesystem full Nov 3 03:17:40 x4 kernel: pid 40915 (rw), uid 0 inumber 131 on /mnt: filesystem full Nov 3 03:17:41 x4 kernel: pid 40916 (rw), uid 0 inumber 131 on /mnt: filesystem full Nov 3 03:17:43 x4 kernel: pid 41662 (rw), uid 0 inumber 136 on /mnt: filesystem full Nov 3 03:17:46 x4 kernel: pid 41663 (rw), uid 0 inumber 7 on /mnt: filesystem full Nov 3 03:17:47 x4 kernel: pid 41664 (rw), uid 0 inumber 135 on /mnt: filesystem full Nov 3 03:17:49 x4 kernel: pid 41668 (rw), uid 0 inumber 130 on /mnt: filesystem full Nov 3 03:17:51 x4 kernel: pid 41667 (rw), uid 0 inumber 129 on /mnt: filesystem full Nov 3 03:17:53 x4 kernel: pid 42804 (rw), uid 0 inumber 141 on /mnt: filesystem full Nov 3 03:17:56 x4 kernel: pid 42795 (rw), uid 0 inumber 12 on /mnt: filesystem full Nov 3 03:17:58 x4 kernel: pid 42810 (rw), uid 0 inumber 131 on /mnt: filesystem full Nov 3 03:18:00 x4 kernel: pid 42793 (rw), uid 0 inumber 132 on /mnt: filesystem full Nov 3 03:18:01 x4 kernel: pid 42802 (rw), uid 0 inumber 5 on /mnt: filesystem full Nov 3 03:19:54 x4 kernel: pid 53932 (rw), uid 0 inumber 133 on /mnt: filesystem full Nov 3 03:19:57 x4 kernel: pid 53933 (rw), uid 0 inumber 131 on /mnt: filesystem full Nov 3 03:19:59 x4 kernel: pid 53931 (rw), uid 0 inumber 5 on /mnt: filesystem full Nov 3 03:20:08 x4 kernel: pid 53941 (rw), uid 0 inumber 130 on /mnt: filesystem full Nov 3 03:20:10 x4 kernel: pid 55054 (rw), uid 0 inumber 130 on /mnt: filesystem full Nov 3 03:20:17 x4 kernel: pid 55055 (rw), uid 0 inumber 5 on /mnt: filesystem full Nov 3 03:20:21 x4 kernel: pid 56188 (rw), uid 0 inumber 143 on /mnt: filesystem full Nov 3 03:20:24 x4 kernel: pid 56175 (rw), uid 0 inumber 138 on /mnt: filesystem full Nov 3 03:20:26 x4 kernel: pid 56185 (rw), uid 0 inumber 131 on /mnt: filesystem full Nov 3 03:20:28 x4 kernel: pid 56170 (rw), uid 0 inumber 132 on /mnt: filesystem full Nov 3 03:22:19 x4 kernel: pid 65656 (rw), uid 0 inumber 5 on /mnt: filesystem full Nov 3 03:22:24 x4 kernel: pid 65654 (rw), uid 0 inumber 132 on /mnt: filesystem full Nov 3 03:22:26 x4 kernel: pid 65652 (rw), uid 0 inumber 131 on /mnt: filesystem full Nov 3 03:22:27 x4 kernel: pid 65655 (rw), uid 0 inumber 129 on /mnt: filesystem full Nov 3 03:22:30 x4 kernel: pid 66208 (rw), uid 0 inumber 8 on /mnt: filesystem full Nov 3 03:22:32 x4 kernel: pid 66202 (rw), uid 0 inumber 135 on /mnt: filesystem full Nov 3 03:22:34 x4 kernel: pid 66196 (rw), uid 0 inumber 131 on /mnt: filesystem full Nov 3 03:22:35 x4 kernel: pid 66200 (rw), uid 0 inumber 5 on /mnt: filesystem full Nov 3 03:22:36 x4 kernel: pid 66204 (rw), uid 0 inumber 129 on /mnt: filesystem full Nov 3 03:22:38 x4 kernel: pid 66197 (rw), uid 0 inumber 4 on /mnt: filesystem full Nov 3 03:22:39 x4 kernel: pid 67343 (rw), uid 0 inumber 140 on /mnt: filesystem full Nov 3 03:22:42 x4 kernel: pid 67340 (rw), uid 0 inumber 11 on /mnt: filesystem full Nov 3 03:22:44 x4 kernel: pid 67335 (rw), uid 0 inumber 129 on /mnt: filesystem full Nov 3 03:22:45 x4 kernel: pid 67339 (rw), uid 0 inumber 7 on /mnt: filesystem full Nov 3 03:22:47 x4 kernel: pid 67336 (rw), uid 0 inumber 6 on /mnt: filesystem full Nov 3 03:24:41 x4 kernel: pid 78445 (rw), uid 0 inumber 6 on /mnt: filesystem full Nov 3 03:24:44 x4 kernel: pid 78440 (rw), uid 0 inumber 135 on /mnt: filesystem full Nov 3 03:24:45 x4 kernel: pid 78444 (rw), uid 0 inumber 129 on /mnt: filesystem full Nov 3 03:24:47 x4 kernel: pid 78441 (rw), uid 0 inumber 129 on /mnt: filesystem full Nov 3 03:24:48 x4 kernel: pid 78443 (rw), uid 0 inumber 4 on /mnt: filesystem full stress2: mount leak: 1/233 20201103 03:24:52 all (71/240): quota12.sh Nov 3 03:25:26 x4 kernel: pid 79886 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 03:25:37 x4 kernel: pid 79882 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 03:25:50 x4 kernel: pid 79872 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 03:27:38 x4 kernel: pid 82282 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 03:27:49 x4 kernel: pid 82278 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 03:27:59 x4 kernel: pid 82272 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 03:28:09 x4 kernel: pid 82302 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 03:28:21 x4 kernel: pid 82309 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 03:30:38 x4 kernel: pid 87726 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 03:30:48 x4 kernel: pid 87718 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 03:31:05 x4 kernel: pid 87779 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 03:33:14 x4 kernel: pid 92693 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 03:35:21 x4 kernel: pid 94784 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 03:35:32 x4 kernel: pid 94777 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 03:35:44 x4 kernel: pid 94767 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 03:37:48 x4 kernel: pid 96418 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 03:39:45 x4 kernel: pid 97813 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 03:39:55 x4 kernel: pid 97812 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 03:40:06 x4 kernel: pid 97711 (swap), jid 0, uid 1004, was killed: out of swap space lock order reversal: 1st 0x23dd684c FFS (FFS Lock, sleep mutex) @ ufs/ufs/ufs_quota.c:759 2nd 0x29469040 struct mount mtx (struct mount mtx, sleep mutex) @ ufs/ufs/ufs_quota.c:765 lock order struct mount mtx -> FFS Lock established at: #0 0x1046ea5 at witness_checkorder+0x3c5 #1 0xfc4b86 at __mtx_lock_flags+0x76 #2 0x1247e8c at softdep_flushfiles+0x35c #3 0x1265b5c at ffs_unmount+0x5c #4 0x10a5ac3 at dounmount+0x403 #5 0x10a565d at kern_unmount+0x2dd #6 0x10a5373 at sys_unmount+0x13 #7 0x13d1798 at syscall+0x1b8 #8 0xffc033f9 at __stop_set_sysinit_set+0xd39f4989 lock order FFS Lock -> struct mount mtx attempted at: #0 0x1047530 at witness_checkorder+0xa50 #1 0xfc4b86 at __mtx_lock_flags+0x76 #2 0x127332e at quotaoff_inchange+0x9e #3 0x1273403 at quotaoff+0x93 #4 0x127488b at ufs_quotactl+0xcb #5 0x10b8161 at sys_quotactl+0x151 #6 0x13d1798 at syscall+0x1b8 #7 0xffc033f9 at __stop_set_sysinit_set+0xd39f4989 stress2: mount leak: 1/234 20201103 03:42:00 all (72/240): linger.sh stress2: mount leak: 1/235 20201103 03:58:52 all (73/240): umountf2.sh stress2: freework leak: 342/350. stress2: newblk leak: 87638/87644. stress2: mount leak: 1/236 stress2: indirdep leak: 45/45 20201103 04:10:52 all (74/240): fs.sh Nov 3 04:14:10 x4 kernel: pid 59655 (rw), uid 1004 inumber 1542 on /mnt: filesystem full Nov 3 04:14:14 x4 kernel: pid 59640 (rw), uid 1004 inumber 1549 on /mnt: filesystem full Nov 3 04:17:18 x4 kernel: pid 64380 (rw), uid 1004 inumber 13 on /mnt: filesystem full Nov 3 04:20:21 x4 kernel: pid 71277 (rw), uid 1004 inumber 886 on /mnt: filesystem full Nov 3 04:20:23 x4 kernel: pid 71268 (mkdir), uid 1004 inumber 2555 on /mnt: out of inodes Nov 3 04:20:24 x4 kernel: pid 71267 (creat), uid 1004 inumber 4 on /mnt: out of inodes Nov 3 04:20:25 x4 kernel: pid 71273 (creat), uid 1004 inumber 1921 on /mnt: out of inodes Nov 3 04:20:27 x4 kernel: pid 71289 (mkdir), uid 1004 inumber 2023 on /mnt: out of inodes Nov 3 04:20:28 x4 kernel: pid 71285 (mkdir), uid 1004 inumber 660 on /mnt: out of inodes Nov 3 04:20:30 x4 kernel: pid 71279 (creat), uid 1004 inumber 651 on /mnt: out of inodes Nov 3 04:20:31 x4 kernel: pid 71267 (creat), uid 1004 inumber 4 on /mnt: out of inodes Nov 3 04:20:32 x4 syslogd: last message repeated 1 times Nov 3 04:20:34 x4 kernel: pid 71274 (mkdir), uid 1004 inumber 115 on /mnt: filesystem full Nov 3 04:20:38 x4 kernel: pid 71281 (mkdir), uid 1004 inumber 861 on /mnt: out of inodes Nov 3 04:20:42 x4 kernel: pid 71266 (rw), uid 1004 inumber 870 on /mnt: filesystem full Nov 3 04:20:48 x4 kernel: pid 71275 (rw), uid 1004 inumber 2483 on /mnt: filesystem full Nov 3 04:21:04 x4 kernel: pid 71278 (rw), uid 1004 inumber 1002 on /mnt: filesystem full Nov 3 04:21:07 x4 kernel: pid 71279 (creat), uid 1004 inumber 651 on /mnt: out of inodes Nov 3 04:21:22 x4 kernel: pid 75806 (rw), uid 1004 inumber 1397 on /mnt: filesystem full Nov 3 04:21:26 x4 kernel: pid 75793 (creat), uid 1004 inumber 11 on /mnt: out of inodes Nov 3 04:21:28 x4 kernel: pid 75788 (creat), uid 1004 inumber 1281 on /mnt: out of inodes Nov 3 04:21:29 x4 kernel: pid 75795 (creat), uid 1004 inumber 5 on /mnt: out of inodes Nov 3 04:21:30 x4 kernel: pid 75790 (creat), uid 1004 inumber 1280 on /mnt: out of inodes Nov 3 04:21:32 x4 kernel: pid 75789 (creat), uid 1004 inumber 1924 on /mnt: out of inodes Nov 3 04:21:33 x4 syslogd: last message repeated 1 times Nov 3 04:21:34 x4 kernel: pid 75791 (creat), uid 1004 inumber 1922 on /mnt: out of inodes Nov 3 04:21:35 x4 kernel: pid 75789 (creat), uid 1004 inumber 1924 on /mnt: out of inodes Nov 3 04:21:36 x4 kernel: pid 75788 (creat), uid 1004 inumber 1281 on /mnt: out of inodes Nov 3 04:21:37 x4 kernel: pid 75786 (creat), uid 1004 inumber 1921 on /mnt: out of inodes Nov 3 04:21:38 x4 kernel: pid 75792 (creat), uid 1004 inumber 1923 on /mnt: out of inodes Nov 3 04:21:40 x4 kernel: pid 75786 (creat), uid 1004 inumber 1921 on /mnt: out of inodes Nov 3 04:21:41 x4 kernel: pid 75793 (creat), uid 1004 inumber 11 on /mnt: out of inodes Nov 3 04:21:43 x4 syslogd: last message repeated 2 times Nov 3 04:21:44 x4 kernel: pid 75786 (creat), uid 1004 inumber 1921 on /mnt: out of inodes Nov 3 04:21:46 x4 kernel: pid 75790 (creat), uid 1004 inumber 1280 on /mnt: out of inodes Nov 3 04:21:47 x4 kernel: pid 75788 (creat), uid 1004 inumber 1281 on /mnt: out of inodes Nov 3 04:21:48 x4 kernel: pid 75793 (creat), uid 1004 inumber 11 on /mnt: out of inodes Nov 3 04:21:49 x4 kernel: pid 75792 (creat), uid 1004 inumber 1923 on /mnt: out of inodes Nov 3 04:21:50 x4 kernel: pid 75790 (creat), uid 1004 inumber 1280 on /mnt: out of inodes Nov 3 04:21:51 x4 kernel: pid 75786 (creat), uid 1004 inumber 1921 on /mnt: out of inodes Nov 3 04:21:52 x4 syslogd: last message repeated 1 times Nov 3 04:21:53 x4 kernel: pid 75792 (creat), uid 1004 inumber 1923 on /mnt: out of inodes Nov 3 04:21:54 x4 kernel: pid 75795 (creat), uid 1004 inumber 5 on /mnt: out of inodes Nov 3 04:21:55 x4 kernel: pid 75793 (creat), uid 1004 inumber 11 on /mnt: out of inodes Nov 3 04:21:56 x4 kernel: pid 75794 (creat), uid 1004 inumber 642 on /mnt: out of inodes Nov 3 04:21:58 x4 kernel: pid 75791 (creat), uid 1004 inumber 1922 on /mnt: out of inodes Nov 3 04:21:59 x4 syslogd: last message repeated 1 times Nov 3 04:22:00 x4 kernel: pid 75786 (creat), uid 1004 inumber 1921 on /mnt: out of inodes Nov 3 04:22:01 x4 kernel: pid 75793 (creat), uid 1004 inumber 11 on /mnt: out of inodes Nov 3 04:22:02 x4 kernel: pid 75790 (creat), uid 1004 inumber 1280 on /mnt: out of inodes Nov 3 04:22:03 x4 syslogd: last message repeated 1 times Nov 3 04:22:04 x4 kernel: pid 75791 (creat), uid 1004 inumber 1922 on /mnt: out of inodes Nov 3 04:22:05 x4 kernel: pid 75788 (creat), uid 1004 inumber 1281 on /mnt: out of inodes Nov 3 04:22:06 x4 kernel: pid 75794 (creat), uid 1004 inumber 642 on /mnt: out of inodes Nov 3 04:22:07 x4 kernel: pid 75789 (creat), uid 1004 inumber 1924 on /mnt: out of inodes Nov 3 04:22:08 x4 kernel: pid 75786 (creat), uid 1004 inumber 1921 on /mnt: out of inodes Nov 3 04:22:10 x4 kernel: pid 75794 (creat), uid 1004 inumber 642 on /mnt: out of inodes Nov 3 04:22:11 x4 kernel: pid 75790 (creat), uid 1004 inumber 1280 on /mnt: out of inodes Nov 3 04:22:12 x4 syslogd: last message repeated 1 times Nov 3 04:22:13 x4 kernel: pid 75791 (creat), uid 1004 inumber 1922 on /mnt: out of inodes Nov 3 04:22:14 x4 kernel: pid 75792 (creat), uid 1004 inumber 1923 on /mnt: out of inodes Nov 3 04:22:15 x4 kernel: pid 75788 (creat), uid 1004 inumber 1281 on /mnt: out of inodes Nov 3 04:22:16 x4 kernel: pid 75795 (creat), uid 1004 inumber 5 on /mnt: out of inodes Nov 3 04:22:19 x4 kernel: pid 79241 (rw), uid 1004 inumber 781 on /mnt: filesystem full Nov 3 04:22:26 x4 kernel: pid 79228 (creat), uid 1004 inumber 1287 on /mnt: out of inodes Nov 3 04:22:27 x4 kernel: pid 79227 (creat), uid 1004 inumber 1284 on /mnt: out of inodes Nov 3 04:22:28 x4 kernel: pid 79232 (creat), uid 1004 inumber 4 on /mnt: out of inodes Nov 3 04:22:29 x4 kernel: pid 79235 (creat), uid 1004 inumber 1923 on /mnt: out of inodes Nov 3 04:22:30 x4 kernel: pid 79230 (creat), uid 1004 inumber 1922 on /mnt: out of inodes Nov 3 04:22:32 x4 kernel: pid 79220 (creat), uid 1004 inumber 1282 on /mnt: out of inodes Nov 3 04:22:33 x4 kernel: pid 79224 (creat), uid 1004 inumber 1285 on /mnt: out of inodes Nov 3 04:22:35 x4 kernel: pid 79233 (creat), uid 1004 inumber 1924 on /mnt: out of inodes Nov 3 04:22:36 x4 syslogd: last message repeated 1 times Nov 3 04:22:37 x4 kernel: pid 79234 (creat), uid 1004 inumber 641 on /mnt: out of inodes Nov 3 04:22:38 x4 kernel: pid 79224 (creat), uid 1004 inumber 1285 on /mnt: out of inodes Nov 3 04:22:39 x4 kernel: pid 79229 (creat), uid 1004 inumber 1921 on /mnt: out of inodes Nov 3 04:22:40 x4 kernel: pid 79236 (creat), uid 1004 inumber 642 on /mnt: out of inodes Nov 3 04:22:42 x4 kernel: pid 79229 (creat), uid 1004 inumber 1921 on /mnt: out of inodes Nov 3 04:22:42 x4 kernel: pid 79236 (creat), uid 1004 inumber 642 on /mnt: out of inodes Nov 3 04:22:45 x4 syslogd: last message repeated 2 times Nov 3 04:22:46 x4 kernel: pid 79237 (creat), uid 1004 inumber 5 on /mnt: out of inodes Nov 3 04:22:47 x4 kernel: pid 79229 (creat), uid 1004 inumber 1921 on /mnt: out of inodes Nov 3 04:22:48 x4 kernel: pid 79227 (creat), uid 1004 inumber 1284 on /mnt: out of inodes Nov 3 04:22:49 x4 kernel: pid 79234 (creat), uid 1004 inumber 641 on /mnt: out of inodes Nov 3 04:22:51 x4 kernel: pid 79228 (creat), uid 1004 inumber 1287 on /mnt: out of inodes Nov 3 04:22:52 x4 kernel: pid 79227 (creat), uid 1004 inumber 1284 on /mnt: out of inodes Nov 3 04:22:53 x4 kernel: pid 79232 (creat), uid 1004 inumber 4 on /mnt: out of inodes Nov 3 04:22:54 x4 kernel: pid 79224 (creat), uid 1004 inumber 1285 on /mnt: out of inodes Nov 3 04:22:55 x4 kernel: pid 79220 (creat), uid 1004 inumber 1282 on /mnt: out of inodes Nov 3 04:22:56 x4 kernel: pid 79237 (creat), uid 1004 inumber 5 on /mnt: out of inodes Nov 3 04:22:57 x4 kernel: pid 79225 (creat), uid 1004 inumber 640 on /mnt: out of inodes Nov 3 04:22:59 x4 kernel: pid 79237 (creat), uid 1004 inumber 5 on /mnt: out of inodes Nov 3 04:23:00 x4 kernel: pid 79232 (creat), uid 1004 inumber 4 on /mnt: out of inodes Nov 3 04:23:01 x4 syslogd: last message repeated 1 times Nov 3 04:23:02 x4 kernel: pid 79228 (creat), uid 1004 inumber 1287 on /mnt: out of inodes Nov 3 04:23:03 x4 kernel: pid 79237 (creat), uid 1004 inumber 5 on /mnt: out of inodes Nov 3 04:23:04 x4 kernel: pid 79225 (creat), uid 1004 inumber 640 on /mnt: out of inodes Nov 3 04:23:05 x4 kernel: pid 79233 (creat), uid 1004 inumber 1924 on /mnt: out of inodes Nov 3 04:23:06 x4 kernel: pid 79229 (creat), uid 1004 inumber 1921 on /mnt: out of inodes Nov 3 04:23:07 x4 kernel: pid 79228 (creat), uid 1004 inumber 1287 on /mnt: out of inodes Nov 3 04:23:09 x4 kernel: pid 79226 (creat), uid 1004 inumber 1286 on /mnt: out of inodes Nov 3 04:23:10 x4 kernel: pid 79230 (creat), uid 1004 inumber 1922 on /mnt: out of inodes Nov 3 04:23:11 x4 kernel: pid 79237 (creat), uid 1004 inumber 5 on /mnt: out of inodes Nov 3 04:23:13 x4 kernel: pid 79226 (creat), uid 1004 inumber 1286 on /mnt: out of inodes Nov 3 04:23:14 x4 syslogd: last message repeated 1 times Nov 3 04:23:15 x4 kernel: pid 79237 (creat), uid 1004 inumber 5 on /mnt: out of inodes Nov 3 04:23:16 x4 kernel: pid 79224 (creat), uid 1004 inumber 1285 on /mnt: out of inodes Nov 3 04:23:51 x4 kernel: pid 83760 (rw), uid 1004 inumber 2307 on /mnt: filesystem full Nov 3 04:24:02 x4 kernel: pid 83773 (rw), uid 1004 inumber 1928 on /mnt: out of inodes Nov 3 04:24:38 x4 kernel: pid 86980 (rw), uid 1004 inumber 923 on /mnt: filesystem full Nov 3 04:24:40 x4 kernel: pid 86978 (rw), uid 1004 inumber 1927 on /mnt: filesystem full Nov 3 04:24:41 x4 kernel: pid 86967 (rw), uid 1004 inumber 2110 on /mnt: filesystem full Nov 3 04:25:11 x4 kernel: pid 86987 (rw), uid 1004 inumber 2083 on /mnt: filesystem full Nov 3 04:25:28 x4 kernel: pid 90560 (rw), uid 1004 inumber 746 on /mnt: filesystem full Nov 3 04:25:30 x4 kernel: pid 90565 (mkdir), uid 1004 inumber 946 on /mnt: filesystem full Nov 3 04:25:46 x4 kernel: pid 90559 (rw), uid 1004 inumber 1836 on /mnt: filesystem full Nov 3 04:26:31 x4 kernel: pid 96159 (rw), uid 1004 inumber 1922 on /mnt: filesystem full Nov 3 04:27:17 x4 kernel: pid 96161 (rw), uid 1004 inumber 1923 on /mnt: filesystem full stress2: mount leak: 8/244 20201103 04:27:24 all (75/240): nullfs22.sh stress2: newblk leak: 53948/84310. stress2: mount leak: 1/245 stress2: indirdep leak: 27/44 20201103 04:30:04 all (76/240): sendfile5.sh stress2: newblk leak: 106986/200563. stress2: mount leak: 3/248 stress2: indirdep leak: 62/111 20201103 04:36:24 all (77/240): tmpfs11.sh Nov 3 04:39:50 x4 kernel: pid 48907 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 04:40:00 x4 kernel: pid 48920 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 04:40:10 x4 kernel: pid 48917 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 04:40:22 x4 kernel: pid 48913 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 04:40:32 x4 kernel: pid 48924 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 04:40:43 x4 kernel: pid 48905 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 04:41:41 x4 kernel: pid 29166 (pkg), jid 0, uid 0, was killed: out of swap space Nov 3 04:41:51 x4 kernel: pid 50186 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 04:42:01 x4 kernel: pid 50194 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 04:42:11 x4 kernel: pid 50189 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 04:42:22 x4 kernel: pid 50198 (swap), jid 0, uid 1004, was killed: out of swap space swap_pager: indefinite wait buffer: bufobj: 0, blkno: 347647, size: 4096 Nov 3 04:48:29 x4 kernel: pid 53492 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 04:48:41 x4 kernel: pid 53489 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 04:48:52 x4 kernel: pid 53487 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 04:49:04 x4 kernel: pid 53486 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 04:52:17 x4 kernel: pid 55478 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 04:52:27 x4 kernel: pid 55476 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 04:52:37 x4 kernel: pid 55473 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 04:52:47 x4 kernel: pid 55472 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 04:52:57 x4 kernel: pid 55469 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 04:53:08 x4 kernel: pid 55467 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 04:53:13 x4 kernel: pid 55474 (swap), jid 0, uid 1004, was killed: out of swap space Nov 3 04:53:23 x4 kernel: pid 55452 (swap), jid 0, uid 1004, was killed: out of swap space stress2: mount leak: 1/249 20201103 04:53:47 all (78/240): altbufferflushes.sh Nov 3 04:59:05 x4 kernel: pid 82678 (dd), uid 0 inumber 6195815 on /var: filesystem full Nov 3 04:59:07 x4 kernel: pid 84907 (dd), uid 0 inumber 6196000 on /var: filesystem full Nov 3 05:03:27 x4 kernel: pid 4990 (dd), uid 0 inumber 6195619 on /var: filesystem full Nov 3 05:03:29 x4 kernel: pid 8337 (dd), uid 0 inumber 6195815 on /var: filesystem full Nov 3 05:10:48 x4 kernel: pid 44996 (dd), uid 0 inumber 6195619 on /var: filesystem full Nov 3 05:10:50 x4 kernel: pid 47231 (dd), uid 0 inumber 6196000 on /var: filesystem full panic: bio_completed can't be less than 0 cpuid = 3 time = 1604376664 KDB: stack backtrace: db_trace_self_wrapper(e67ffc,1a5e5d0,0,3ae043e4,bd3b20,...) at db_trace_self_wrapper+0x2a/frame 0x3ae043b8 kdb_backtrace(7,2b3b668,2b31fb0,ffffffff,261e8cf0,...) at kdb_backtrace+0x2e/frame 0x3ae04418 vpanic(146f152,3ae04458,3ae04458,3ae04488,f39752,...) at vpanic+0x11f/frame 0x3ae04438 panic(146f152,0,18d7ef80,fc50a7,18d7ae00,...) at panic+0x14/frame 0x3ae0444c g_io_deliver(261e8cf0,0) at g_io_deliver+0x322/frame 0x3ae04488 g_disk_done(24029678,24029678) at g_disk_done+0xf0/frame 0x3ae044bc cam_iosched_queue_work(18813700,24029678) at cam_iosched_queue_work+0xcb/frame 0x3ae044d8 adastrategy(24029678,0,114,2c4,b9aa800,...) at adastrategy+0x50/frame 0x3ae044f4 g_disk_start(261e8cf0,18d7ef80,18d7ae00,f38877,b9aa800,...) at g_disk_start+0x4de/frame 0x3ae04550 g_io_request(261e8cf0,18d7ef80,33901fec,2b3b598,3ae045f8,...) at g_io_request+0x2aa/frame 0x3ae04578 g_part_start(339019b4,18d88dc0,9ac4900,f38877,b9aa800,...) at g_part_start+0x26f/frame 0x3ae045e8 g_io_request(339019b4,18d88dc0,1d8fb8c,1,41000000,...) at g_io_request+0x2aa/frame 0x3ae04610 g_part_start(2d39fbdc,1,18d8ed00,18d94b80,30ebd600,...) at g_part_start+0x26f/frame 0x3ae04680 g_io_request(2d39fbdc,b8e0400,b8e0400) at g_io_request+0x2aa/frame 0x3ae046a8 g_vfs_strategy(20051828,30ebd600) at g_vfs_strategy+0xe8/frame 0x3ae046c8 softdep_request_cleanup(20165000,2a343758,29a3e000,4) at softdep_request_cleanup+0x2b7/frame 0x3ae04720 ffs_alloc(2aafb4fc,0,0,6a4908,0,1000,2050000,29a3e000,3ae047d8) at ffs_alloc+0x2ad/frame 0x3ae0476c ffs_balloc_ufs2(2a343758,0,0,1000,29a3e000,2050000,3ae04900) at ffs_balloc_ufs2+0xd8b/frame 0x3ae048a0 ffs_write(3ae04a08) at ffs_write+0x216/frame 0x3ae04924 VOP_WRITE_APV(1aabe80,3ae04a08) at VOP_WRITE_APV+0xdf/frame 0x3ae049e0 vn_write(2a29ec30,3ae04bd0,2881e500,1,2af64700) at vn_write+0x305/frame 0x3ae04a3c vn_io_fault_doio(2af64700) at vn_io_fault_doio+0x3a/frame 0x3ae04a84 vn_io_fault1(2af64700) at vn_io_fault1+0x164/frame 0x3ae04b34 vn_io_fault(2a29ec30,3ae04bd0,2881e500,0,2af64700) at vn_io_fault+0x1c4/frame 0x3ae04b70 dofilewrite(2a29ec30,3ae04bd0,ffffffff,ffffffff,0) at dofilewrite+0x71/frame 0x3ae04ba4 sys_write(2af64700,2af64998,2af64700,28409000,2af64700,...) at sys_write+0x78/frame 0x3ae04c08 syscall(3ae04ce8,3b,3b,3b,1000,...) at syscall+0x1b8/frame 0x3ae04cdc Xint0x80_syscall() at 0xffc033f9/frame 0x3ae04cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- KDB: enter: panic [ thread pid 48350 tid 100686 ] Stopped at kdb_enter+0x35: movl $0,kdb_why db> run pho db:0:pho> set $lines 20000 db:0:pho> run pho1 db:1:pho1> bt Tracing pid 48350 tid 100686 td 0x2af64700 kdb_enter(14df98a,14df98a) at kdb_enter+0x35/frame 0x3ae04410 vpanic(146f152,3ae04458,3ae04458,3ae04488,f39752,...) at vpanic+0x137/frame 0x3ae04438 panic(146f152,0,18d7ef80,fc50a7,18d7ae00,...) at panic+0x14/frame 0x3ae0444c g_io_deliver(261e8cf0,0) at g_io_deliver+0x322/frame 0x3ae04488 g_disk_done(24029678,24029678) at g_disk_done+0xf0/frame 0x3ae044bc cam_iosched_queue_work(18813700,24029678) at cam_iosched_queue_work+0xcb/frame 0x3ae044d8 adastrategy(24029678,0,114,2c4,b9aa800,...) at adastrategy+0x50/frame 0x3ae044f4 g_disk_start(261e8cf0,18d7ef80,18d7ae00,f38877,b9aa800,...) at g_disk_start+0x4de/frame 0x3ae04550 g_io_request(261e8cf0,18d7ef80,33901fec,2b3b598,3ae045f8,...) at g_io_request+0x2aa/frame 0x3ae04578 g_part_start(339019b4,18d88dc0,9ac4900,f38877,b9aa800,...) at g_part_start+0x26f/frame 0x3ae045e8 g_io_request(339019b4,18d88dc0,1d8fb8c,1,41000000,...) at g_io_request+0x2aa/frame 0x3ae04610 g_part_start(2d39fbdc,1,18d8ed00,18d94b80,30ebd600,...) at g_part_start+0x26f/frame 0x3ae04680 g_io_request(2d39fbdc,b8e0400,b8e0400) at g_io_request+0x2aa/frame 0x3ae046a8 g_vfs_strategy(20051828,30ebd600) at g_vfs_strategy+0xe8/frame 0x3ae046c8 softdep_request_cleanup(20165000,2a343758,29a3e000,4) at softdep_request_cleanup+0x2b7/frame 0x3ae04720 ffs_alloc(2aafb4fc,0,0,6a4908,0,1000,2050000,29a3e000,3ae047d8) at ffs_alloc+0x2ad/frame 0x3ae0476c ffs_balloc_ufs2(2a343758,0,0,1000,29a3e000,2050000,3ae04900) at ffs_balloc_ufs2+0xd8b/frame 0x3ae048a0 ffs_write(3ae04a08) at ffs_write+0x216/frame 0x3ae04924 VOP_WRITE_APV(1aabe80,3ae04a08) at VOP_WRITE_APV+0xdf/frame 0x3ae049e0 vn_write(2a29ec30,3ae04bd0,2881e500,1,2af64700) at vn_write+0x305/frame 0x3ae04a3c vn_io_fault_doio(2af64700) at vn_io_fault_doio+0x3a/frame 0x3ae04a84 vn_io_fault1(2af64700) at vn_io_fault1+0x164/frame 0x3ae04b34 vn_io_fault(2a29ec30,3ae04bd0,2881e500,0,2af64700) at vn_io_fault+0x1c4/frame 0x3ae04b70 dofilewrite(2a29ec30,3ae04bd0,ffffffff,ffffffff,0) at dofilewrite+0x71/frame 0x3ae04ba4 sys_write(2af64700,2af64998,2af64700,28409000,2af64700,...) at sys_write+0x78/frame 0x3ae04c08 syscall(3ae04ce8,3b,3b,3b,1000,...) at syscall+0x1b8/frame 0x3ae04cdc Xint0x80_syscall() at 0xffc033f9/frame 0x3ae04cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- db:1:pho1> show allpcpu Current CPU: 3 cpuid = 0 dynamic pcpu = 0x835dc0 curthread = 0xb9c3500: pid 11 tid 100003 critnest 1 "idle: cpu0" curpcb = 0x12b21d40 fpcurthread = none idlethread = 0xb9c3500: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 trampstk = 0xffc07ff0 kesp0 = 0x12b21d30 common_tssp = 0xffc01260 tlb gen = 321743883 curvnet = 0 spin locks held: cpuid = 1 dynamic pcpu = 0x9d86dc0 curthread = 0xb9c3180: pid 11 tid 100004 critnest 1 "idle: cpu1" curpcb = 0x12b26d40 fpcurthread = none idlethread = 0xb9c3180: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 trampstk = 0xffc0aff0 kesp0 = 0x12b26d30 common_tssp = 0xffc012c8 tlb gen = 321743893 curvnet = 0 spin locks held: cpuid = 2 dynamic pcpu = 0x9d8cdc0 curthread = 0xb9c2e00: pid 11 tid 100005 critnest 1 "idle: cpu2" curpcb = 0x12b2bd40 fpcurthread = none idlethread = 0xb9c2e00: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 trampstk = 0xffc0bff0 kesp0 = 0x12b2bd30 common_tssp = 0xffc01330 tlb gen = 321743893 curvnet = 0 spin locks held: cpuid = 3 dynamic pcpu = 0x9d92dc0 curthread = 0x2af64700: pid 48350 tid 100686 critnest 1 "dd" curpcb = 0x3ae04d40 fpcurthread = 0x2af64700: pid 48350 "dd" idlethread = 0xb9c2a80: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 trampstk = 0xffc0dff0 kesp0 = 0x3ae04d30 common_tssp = 0xffc01398 tlb gen = 321743893 curvnet = 0 spin locks held: db:1:pho1> show alllocks Process 48350 (dd) thread 0x2af64700 (100686) exclusive sleep mutex CAM device lock (CAM device lock) r = 0 (0x1801c45c) locked @ cam/ata/ata_da.c:1069 exclusive lockmgr ufs (ufs) r = 0 (0x2a343794) locked @ kern/vfs_vnops.c:1076 Process 47234 (dd) thread 0x2f318380 (100989) exclusive lockmgr ufs (ufs) r = 0 (0x2bc23bfc) locked @ kern/vfs_vnops.c:1076 Process 22 (bufdaemon) thread 0x2004f500 (100117) exclusive lockmgr bufwait (bufwait) r = 0 (0xb768524) locked @ kern/vfs_bio.c:1662 exclusive lockmgr bufwait (bufwait) r = 0 (0xb70fe80) locked @ kern/vfs_bio.c:1662 db:1:pho1> show lockedvnods Locked vnodes vnode 0x2a343758: type VREG usecount 1, writecount 1, refcount 1 seqc users 0 hold count flags () flags (VIRF_PGREAD|VMP_LAZYLIST) v_object 0x2a89c148 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type ufs: EXCL by thread 0x2af64700 (pid 48350, dd, tid 100686) #0 0xfb9952 at lockmgr_lock_flags+0x192 #1 0x12682b6 at ffs_lock+0x76 #2 0x13e7828 at VOP_LOCK1_APV+0x38 #3 0x10c381f at _vn_lock+0x4f #4 0x10c7ada at vn_write+0x1da #5 0x10c748a at vn_io_fault_doio+0x3a #6 0x10c3f84 at vn_io_fault1+0x164 #7 0x10c1db4 at vn_io_fault+0x1c4 #8 0x104b571 at dofilewrite+0x71 #9 0x104b188 at sys_write+0x78 #10 0x13d1798 at syscall+0x1b8 #11 0xffc033f9 at __stop_set_sysinit_set+0xd39f4989 nlink=1, effnlink=1, size=0, extsize 0 generation=19a19984, uid=2, gid=5, flags=0x1 ino 1743010, on dev ada0s1e vnode 0x2bc23bc0: type VREG usecount 1, writecount 1, refcount 5257 seqc users 0 hold count flags () flags (VIRF_PGREAD|VMP_LAZYLIST) v_object 0x2de4c668 ref 0 pages 53872 cleanbuf 5247 dirtybuf 8 lock type ufs: EXCL by thread 0x2f318380 (pid 47234, dd, tid 100989) #0 0xfb9952 at lockmgr_lock_flags+0x192 #1 0x12682b6 at ffs_lock+0x76 #2 0x13e7828 at VOP_LOCK1_APV+0x38 #3 0x10c381f at _vn_lock+0x4f #4 0x10c7ada at vn_write+0x1da #5 0x10c748a at vn_io_fault_doio+0x3a #6 0x10c3f84 at vn_io_fault1+0x164 #7 0x10c1db4 at vn_io_fault+0x1c4 #8 0x104b571 at dofilewrite+0x71 #9 0x104b188 at sys_write+0x78 #10 0x13d1798 at syscall+0x1b8 #11 0xffc033f9 at __stop_set_sysinit_set+0xd39f4989 nlink=1, effnlink=1, size=220528640, extsize 0 generation=37b8aec8, uid=0, gid=0, flags=0x0 ino 6196001, on dev ada0s1e db:1:pho1> show allchains chain 1: thread 102298 (pid 48347, sh) is sleeping on 0x36b736dc "wait" chain 2: thread 101241 (pid 48346, cron) is sleeping on 0x2eefc848 "piperd" chain 3: thread 101664 (pid 48345, sleep) is sleeping on 0x1d5f8cf "nanslp" chain 4: thread 102153 (pid 47235, egrep) is sleeping on 0x2a2b84f8 "piperd" chain 5: thread 100989 (pid 47234, dd) is sleeping on 0x1dcfefc "wdrain" chain 6: thread 101630 (pid 57173, tee) is sleeping on 0x23445350 "piperd" chain 7: thread 101625 (pid 57172, sh) is sleeping on 0x2e2ac6dc "wait" chain 8: thread 100253 (pid 57171, sh) is sleeping on 0x284099cc "wait" chain 9: thread 100229 (pid 46027, bash) is sleeping on 0x9bf69a4 "select" chain 10: thread 100249 (pid 46018, sshd) is sleeping on 0xb9d38a4 "select" chain 11: thread 100347 (pid 45996, sshd) is sleeping on 0x1fc0f124 "select" chain 12: thread 100285 (pid 6156, tail) is sleeping on 0x9bf68e4 "select" chain 13: thread 100291 (pid 6153, tail) is sleeping on 0x28989c00 "kqread" chain 14: thread 100306 (pid 5106, bash) is sleeping on 0x28409344 "wait" chain 15: thread 100292 (pid 5105, sshd) is sleeping on 0x188265e4 "select" chain 16: thread 100300 (pid 5103, sshd) is sleeping on 0x9bf6864 "select" chain 17: thread 100213 (pid 1825, sh) is sleeping on 0x27edc4f8 "piperd" chain 18: thread 100222 (pid 1823, awk) is sleeping on 0x27ece000 "piperd" chain 19: thread 100218 (pid 1822, sh) is sleeping on 0x27ec86dc "wait" chain 20: thread 100219 (pid 1801, sh) is sleeping on 0x22eba344 "wait" chain 21: thread 100204 (pid 1060, top) is sleeping on 0xb9d3aa4 "select" chain 22: thread 100209 (pid 984, sh) is sleeping on 0x244b8054 "wait" chain 23: thread 100108 (pid 947, bash) is sleeping on 0x1fc2c398 "wait" chain 24: thread 100198 (pid 945, csh) is sleeping on 0x22e67a30 "pause" chain 25: thread 100124 (pid 944, su) is sleeping on 0x20427a20 "wait" chain 26: thread 100202 (pid 923, bash) is sleeping on 0x244b8a20 "wait" chain 27: thread 100173 (pid 922, sshd) is sleeping on 0x18826824 "select" chain 28: thread 100179 (pid 920, sshd) is sleeping on 0x18826964 "select" chain 29: thread 100199 (pid 909, bash) is sleeping on 0x22e67688 "wait" chain 30: thread 100197 (pid 908, sshd) is sleeping on 0xb9d3ae4 "select" chain 31: thread 100175 (pid 906, sshd) is sleeping on 0x1fc0fd64 "select" chain 32: thread 100169 (pid 790, getty) is sleeping on 0x1833cc74 "ttyin" chain 33: thread 100196 (pid 789, getty) is sleeping on 0x18c1e874 "ttyin" chain 34: thread 100195 (pid 788, getty) is sleeping on 0x18c1ec74 "ttyin" chain 35: thread 100194 (pid 787, getty) is sleeping on 0x18b7a074 "ttyin" chain 36: thread 100193 (pid 786, getty) is sleeping on 0x18b7a474 "ttyin" chain 37: thread 100192 (pid 785, getty) is sleeping on 0x18b7a874 "ttyin" chain 38: thread 100190 (pid 784, getty) is sleeping on 0x18b7ac74 "ttyin" chain 39: thread 100120 (pid 783, getty) is sleeping on 0x1833c074 "ttyin" chain 40: thread 100106 (pid 782, getty) is sleeping on 0x1833c474 "ttyin" chain 41: thread 100178 (pid 762, inetd) is sleeping on 0x18826924 "select" chain 42: thread 100109 (pid 728, nfscbd: master) is sleeping on 0x22c64950 "rpcsvc" chain 43: thread 100182 (pid 728, nfscbd: service) is sleeping on 0x9bd62d0 "rpcsvc" chain 44: thread 100183 (pid 728, nfscbd: service) is sleeping on 0x9bd6250 "rpcsvc" chain 45: thread 100184 (pid 728, nfscbd: service) is sleeping on 0x22c648d0 "rpcsvc" chain 46: thread 100171 (pid 727, nfscbd) is sleeping on 0x22c8b30c "accept" chain 47: thread 100176 (pid 691, cron) is sleeping on 0x1d5f8cc "nanslp" chain 48: thread 100126 (pid 687, sendmail) is sleeping on 0x9b08d74 "pause" chain 49: thread 100127 (pid 684, sendmail) is sleeping on 0x1fc0ff24 "select" chain 50: thread 100174 (pid 681, sshd) is sleeping on 0xb9d3ca4 "select" chain 51: thread 100119 (pid 641, smartd) is sleeping on 0x1d5f8ce "nanslp" chain 52: thread 100112 (pid 617, powerd) is sleeping on 0x9bf6ce4 "select" chain 53: thread 100168 (pid 614, ntpd) is sleeping on 0x1fc0fde4 "select" chain 54: thread 100134 (pid 578, NLM: master) is sleeping on 0x22c64b50 "rpcsvc" chain 55: thread 100167 (pid 575, rpc.statd) is sleeping on 0x188269e4 "select" chain 56: thread 100135 (pid 574, nfsd: master) is sleeping on 0x188722d0 "rpcsvc" chain 57: thread 100136 (pid 574, nfsd: service) is sleeping on 0x1891aad0 "rpcsvc" chain 58: thread 100137 (pid 574, nfsd: service) is sleeping on 0x1835d150 "rpcsvc" chain 59: thread 100138 (pid 574, nfsd: service) is sleeping on 0x1891aa50 "rpcsvc" chain 60: thread 100139 (pid 574, nfsd: service) is sleeping on 0x1891a9d0 "rpcsvc" chain 61: thread 100140 (pid 574, nfsd: service) is sleeping on 0x1835d0d0 "rpcsvc" chain 62: thread 100141 (pid 574, nfsd: service) is sleeping on 0x1891a950 "rpcsvc" chain 63: thread 100142 (pid 574, nfsd: service) is sleeping on 0x1891a8d0 "rpcsvc" chain 64: thread 100143 (pid 574, nfsd: service) is sleeping on 0x1835d050 "rpcsvc" chain 65: thread 100144 (pid 574, nfsd: service) is sleeping on 0x1891a850 "rpcsvc" chain 66: thread 100145 (pid 574, nfsd: service) is sleeping on 0x1891a7d0 "rpcsvc" chain 67: thread 100146 (pid 574, nfsd: service) is sleeping on 0x22c64f50 "rpcsvc" chain 68: thread 100147 (pid 574, nfsd: service) is sleeping on 0x1891a750 "rpcsvc" chain 69: thread 100148 (pid 574, nfsd: service) is sleeping on 0x1891a6d0 "rpcsvc" chain 70: thread 100149 (pid 574, nfsd: service) is sleeping on 0x22c64ed0 "rpcsvc" chain 71: thread 100150 (pid 574, nfsd: service) is sleeping on 0x1891a650 "rpcsvc" chain 72: thread 100151 (pid 574, nfsd: service) is sleeping on 0x22c64e50 "rpcsvc" chain 73: thread 100152 (pid 574, nfsd: service) is sleeping on 0x1891a5d0 "rpcsvc" chain 74: thread 100153 (pid 574, nfsd: service) is sleeping on 0x1891a550 "rpcsvc" chain 75: thread 100154 (pid 574, nfsd: service) is sleeping on 0x22c64dd0 "rpcsvc" chain 76: thread 100155 (pid 574, nfsd: service) is sleeping on 0x1891a4d0 "rpcsvc" chain 77: thread 100156 (pid 574, nfsd: service) is sleeping on 0x1891a450 "rpcsvc" chain 78: thread 100157 (pid 574, nfsd: service) is sleeping on 0x22c64d50 "rpcsvc" chain 79: thread 100158 (pid 574, nfsd: service) is sleeping on 0x1891a3d0 "rpcsvc" chain 80: thread 100159 (pid 574, nfsd: service) is sleeping on 0x22c64cd0 "rpcsvc" chain 81: thread 100160 (pid 574, nfsd: service) is sleeping on 0x1891a350 "rpcsvc" chain 82: thread 100161 (pid 574, nfsd: service) is sleeping on 0x22c64c50 "rpcsvc" chain 83: thread 100162 (pid 574, nfsd: service) is sleeping on 0x1891a2d0 "rpcsvc" chain 84: thread 100163 (pid 574, nfsd: service) is sleeping on 0x1891a250 "rpcsvc" chain 85: thread 100164 (pid 574, nfsd: service) is sleeping on 0x22c64bd0 "rpcsvc" chain 86: thread 100165 (pid 574, nfsd: service) is sleeping on 0x1891a1d0 "rpcsvc" chain 87: thread 100166 (pid 574, nfsd: service) is sleeping on 0x1891a150 "rpcsvc" chain 88: thread 100125 (pid 571, nfsd) is sleeping on 0xb9d3d24 "select" chain 89: thread 100123 (pid 565, mountd) is sleeping on 0x9bf6ca4 "select" chain 90: thread 100133 (pid 559, nfsuserd) is sleeping on 0x18826a24 "select" chain 91: thread 100132 (pid 557, nfsuserd) is sleeping on 0x1fc0fe64 "select" chain 92: thread 100131 (pid 556, nfsuserd) is sleeping on 0x18826a64 "select" chain 93: thread 100130 (pid 555, nfsuserd) is sleeping on 0x1fc0fea4 "select" chain 94: thread 100129 (pid 554, nfsuserd) is sleeping on 0x204273fc "pause" chain 95: thread 100121 (pid 511, watchdogd) is sleeping on 0x1d5f8cc "nanslp" chain 96: thread 100107 (pid 422, rpcbind) is sleeping on 0x18826ae4 "select" chain 97: thread 100111 (pid 413, syslogd) is sleeping on 0xb9d3da4 "select" chain 98: thread 100122 (pid 341, devd) is sleeping on 0xb9d3d64 "select" chain 99: thread 100104 (pid 24, syncer) is sleeping on 0x1dd07b8 "syncer" chain 100: thread 100103 (pid 23, vnlru) is sleeping on 0x182cb9cc "vlruwt" chain 101: thread 100099 (pid 22, bufdaemon) is sleeping on 0x1dcff04 "qsleep" chain 102: thread 100100 (pid 22, bufspacedaemon-0) is sleeping on 0x1a82340 "-" chain 103: thread 100114 (pid 22, /home worker) is sleeping on 0x9bd8684 "sdflush" chain 104: thread 100115 (pid 22, /tmp worker) is sleeping on 0x9bd8484 "sdflush" chain 105: thread 100116 (pid 22, /usr worker) is sleeping on 0x9bd8284 "sdflush" chain 106: thread 100117 (pid 22, /var worker) is sleeping on 0xb7684c8 "biord" chain 107: thread 100098 (pid 21, vmdaemon) is sleeping on 0x1de7270 "psleep" chain 108: thread 100097 (pid 20, dom0) is sleeping on 0x1de62bc "psleep" chain 109: thread 100101 (pid 20, laundry: dom0) is sleeping on 0x1de62c8 "launds" chain 110: thread 100102 (pid 20, uma) is sleeping on 0x1284ce0 "umarcl" chain 111: thread 100091 (pid 19, rand_harvestq) is sleeping on 0x1bad37c "-" chain 112: thread 100090 (pid 18, soaiod4) is sleeping on 0x1dcfa68 "-" chain 113: thread 100089 (pid 17, soaiod3) is sleeping on 0x1dcfa68 "-" chain 114: thread 100088 (pid 16, soaiod2) is sleeping on 0x1dcfa68 "-" chain 115: thread 100087 (pid 9, soaiod1) is sleeping on 0x1dcfa68 "-" chain 116: thread 100084 (pid 8, fdc0) is sleeping on 0x9bf743c "-" chain 117: thread 100049 (pid 15, usbus0) is sleeping on 0x18019aac "-" chain 118: thread 100050 (pid 15, usbus0) is sleeping on 0x18019adc "-" chain 119: thread 100051 (pid 15, usbus0) is sleeping on 0x18019b0c "-" chain 120: thread 100052 (pid 15, usbus0) is sleeping on 0x18019b3c "-" chain 121: thread 100053 (pid 15, usbus0) is sleeping on 0x18019b6c "-" chain 122: thread 100055 (pid 15, usbus1) is sleeping on 0x182d4aac "-" chain 123: thread 100056 (pid 15, usbus1) is sleeping on 0x182d4adc "-" chain 124: thread 100057 (pid 15, usbus1) is sleeping on 0x182d4b0c "-" chain 125: thread 100058 (pid 15, usbus1) is sleeping on 0x182d4b3c "-" chain 126: thread 100059 (pid 15, usbus1) is sleeping on 0x182d4b6c "-" chain 127: thread 100061 (pid 15, usbus2) is sleeping on 0x182deaac "-" chain 128: thread 100062 (pid 15, usbus2) is sleeping on 0x182deadc "-" chain 129: thread 100063 (pid 15, usbus2) is sleeping on 0x182deb0c "-" chain 130: thread 100064 (pid 15, usbus2) is sleeping on 0x182deb3c "-" chain 131: thread 100065 (pid 15, usbus2) is sleeping on 0x182deb6c "-" chain 132: thread 100066 (pid 15, usbus3) is sleeping on 0x182e6aac "-" chain 133: thread 100067 (pid 15, usbus3) is sleeping on 0x182e6adc "-" chain 134: thread 100068 (pid 15, usbus3) is sleeping on 0x182e6b0c "-" chain 135: thread 100069 (pid 15, usbus3) is sleeping on 0x182e6b3c "-" chain 136: thread 100070 (pid 15, usbus3) is sleeping on 0x182e6b6c "-" chain 137: thread 100071 (pid 15, usbus4) is sleeping on 0x182eeaac "-" chain 138: thread 100072 (pid 15, usbus4) is sleeping on 0x182eeadc "-" chain 139: thread 100073 (pid 15, usbus4) is sleeping on 0x182eeb0c "-" chain 140: thread 100074 (pid 15, usbus4) is sleeping on 0x182eeb3c "-" chain 141: thread 100075 (pid 15, usbus4) is sleeping on 0x182eeb6c "-" chain 142: thread 100077 (pid 15, usbus5) is sleeping on 0x18014c84 "-" chain 143: thread 100078 (pid 15, usbus5) is sleeping on 0x18014cb4 "-" chain 144: thread 100079 (pid 15, usbus5) is sleeping on 0x18014ce4 "-" chain 145: thread 100080 (pid 15, usbus5) is sleeping on 0x18014d14 "-" chain 146: thread 100081 (pid 15, usbus5) is sleeping on 0x18014d44 "-" chain 147: thread 100044 (pid 7, doneq0) is sleeping on 0x1ae46c0 "-" chain 148: thread 100096 (pid 7, scanner) is sleeping on 0x1ae45ec "-" chain 149: thread 100043 (pid 6, crypto returns 3) is sleeping on 0xb9f89c4 "crypto_ret_wait" chain 150: thread 100042 (pid 5, crypto returns 2) is sleeping on 0xb9f898c "crypto_ret_wait" chain 151: thread 100041 (pid 4, crypto returns 1) is sleeping on 0xb9f8954 "crypto_ret_wait" chain 152: thread 100040 (pid 3, crypto returns 0) is sleeping on 0xb9f891c "crypto_ret_wait" chain 153: thread 100039 (pid 2, crypto) is sleeping on 0x1de4a20 "crypto_wait" chain 154: thread 100034 (pid 14, sequencer 00) is sleeping on 0x9ae8e4c "seqstate" chain 155: thread 100031 (pid 13, g_event) is sleeping on 0x1d4c288 "-" chain 156: thread 100032 (pid 13, g_up) is sleeping on 0x1d4c28c "-" chain 157: thread 100033 (pid 13, g_down) is sleeping on 0x1d4c290 "-" chain 158: thread 100002 (pid 1, init) is sleeping on 0xb9c16dc "wait" chain 159: thread 100001 (pid 10, audit) is sleeping on 0x1de4d0c "audit_worker_cv" chain 160: thread 100000 (pid 0, swapper) is sleeping on 0x1d4c75c "swapin" chain 161: thread 100007 (pid 0, if_io_tqg_0) is sleeping on 0xb9caa00 "-" chain 162: thread 100008 (pid 0, if_io_tqg_1) is sleeping on 0xb9ca800 "-" chain 163: thread 100009 (pid 0, if_io_tqg_2) is sleeping on 0xb9ca600 "-" chain 164: thread 100010 (pid 0, if_io_tqg_3) is sleeping on 0xb9ca400 "-" chain 165: thread 100011 (pid 0, if_config_tqg_0) is sleeping on 0xb9ca200 "-" chain 166: thread 100012 (pid 0, softirq_0) is sleeping on 0xb9ca000 "-" chain 167: thread 100013 (pid 0, softirq_1) is sleeping on 0xb9f0d00 "-" chain 168: thread 100014 (pid 0, softirq_2) is sleeping on 0xb9f0b00 "-" chain 169: thread 100015 (pid 0, softirq_3) is sleeping on 0xb9f0900 "-" chain 170: thread 100017 (pid 0, in6m_free taskq) is sleeping on 0xb9f0500 "-" chain 171: thread 100018 (pid 0, thread taskq) is sleeping on 0xb9f0300 "-" chain 172: thread 100020 (pid 0, aiod_kick taskq) is sleeping on 0xb9f4e00 "-" chain 173: thread 100021 (pid 0, kqueue_ctx taskq) is sleeping on 0xb9f4c00 "-" chain 174: thread 100023 (pid 0, inm_free taskq) is sleeping on 0xb9f4800 "-" chain 175: thread 100030 (pid 0, firmware taskq) is sleeping on 0xb9f4000 "-" chain 176: thread 100035 (pid 0, crypto_0) is sleeping on 0xb9ff900 "-" chain 177: thread 100036 (pid 0, crypto_1) is sleeping on 0xb9ff900 "-" chain 178: thread 100037 (pid 0, crypto_2) is sleeping on 0xb9ff900 "-" chain 179: thread 100038 (pid 0, crypto_3) is sleeping on 0xb9ff900 "-" chain 180: thread 100086 (pid 0, mca taskq) is sleeping on 0x18315800 "-" chain 181: thread 100092 (pid 0, acpi_task_0) is sleeping on 0x187ff100 "-" chain 182: thread 100093 (pid 0, acpi_task_1) is sleeping on 0x187ff100 "-" chain 183: thread 100094 (pid 0, acpi_task_2) is sleeping on 0x187ff100 "-" chain 184: thread 100095 (pid 0, CAM taskq) is sleeping on 0x9b0a500 "-" db:1:pho1> show mount 0x18ccc080 /dev/ada0s1a on / (ufs) 0x18cca040 devfs on /dev (devfs) 0x1fe1a040 /dev/ada0s1f on /home (ufs) 0x18cc9580 /dev/label/tmp on /tmp (ufs) 0x18cc8ac0 /dev/ada0s1d on /usr (ufs) 0x18cc8000 /dev/ada0s1e on /var (ufs) 0x1fe1c080 procfs on /proc (procfs) More info: show mount db:1:pho1> show bufqueues bqempty: 58 Buf domain 0 freebufs 366 lofreebufs 311 hifreebufs 466 bufspace 88582144 maxbufspace 94683136 hibufspace 94027776 lobufspace 89326372 bufspacethresh 91677074 numdirtybuffers 129 lodirtybuffers 732 hidirtybuffers 1464 dirtybufthresh 10 cleanq count 5150 (84336640) dirtyq count 217 (3540992) wakeup 0 lim 28 CPU 3, 11, 3, 14, Locked buffers: 103 space 1687552 Total buffers: 5779 space 88582144 db:0:pho> ps pid ppid pgrp uid state wmesg wchan cmd 48350 48347 48347 2 R CPU 3 dd 48347 48346 48347 2 Ss wait 0x36b736dc sh 48346 691 691 0 S piperd 0x2eefc848 cron 48345 1822 1801 0 S nanslp 0x1d5f8cf sleep 47235 57172 984 0 S+ piperd 0x2a2b84f8 egrep 47234 57172 984 0 D+ wdrain 0x1dcfefc dd 57173 984 984 0 S+ piperd 0x23445350 tee 57172 57171 984 0 S+ wait 0x2e2ac6dc sh 57171 984 984 0 S+ wait 0x284099cc sh 46027 46018 46027 1001 Ss+ select 0x9bf69a4 bash 46018 45996 45996 1001 S select 0xb9d38a4 sshd 45996 681 45996 0 Ss select 0x1fc0f124 sshd 6156 1 6153 1001 S+ select 0x9bf68e4 tail 6153 5106 6153 1001 S+ kqread 0x28989c00 tail 5106 5105 5106 1001 SWs+ wait 0x28409344 bash 5105 5103 5103 1001 S select 0x188265e4 sshd 5103 681 5103 0 Ss select 0x9bf6864 sshd 1825 1801 1801 0 S piperd 0x27edc4f8 sh 1823 1801 1801 0 S piperd 0x27ece000 awk 1822 1801 1801 0 S wait 0x27ec86dc sh 1801 1 1801 0 SWs wait 0x22eba344 sh 1060 909 1060 1001 S+ select 0xb9d3aa4 top 984 947 984 0 S+ wait 0x244b8054 sh 947 945 947 0 SW+ wait 0x1fc2c398 bash 945 944 945 0 SW+ pause 0x22e67a30 csh 944 923 944 1001 SW+ wait 0x20427a20 su 923 922 923 1001 SWs+ wait 0x244b8a20 bash 922 920 920 1001 S select 0x18826824 sshd 920 681 920 0 Ss select 0x18826964 sshd 909 908 909 1001 SWs+ wait 0x22e67688 bash 908 906 906 1001 S select 0xb9d3ae4 sshd 906 681 906 0 Ss select 0x1fc0fd64 sshd 790 1 790 0 Ss+ ttyin 0x1833cc74 getty 789 1 789 0 Ss+ ttyin 0x18c1e874 getty 788 1 788 0 Ss+ ttyin 0x18c1ec74 getty 787 1 787 0 Ss+ ttyin 0x18b7a074 getty 786 1 786 0 Ss+ ttyin 0x18b7a474 getty 785 1 785 0 Ss+ ttyin 0x18b7a874 getty 784 1 784 0 Ss+ ttyin 0x18b7ac74 getty 783 1 783 0 Ss+ ttyin 0x1833c074 getty 782 1 782 0 Ss+ ttyin 0x1833c474 getty 762 1 762 0 Ss select 0x18826924 inetd 728 727 727 0 S (threaded) nfscbd 100109 S rpcsvc 0x22c64950 nfscbd: master 100182 S rpcsvc 0x9bd62d0 nfscbd: service 100183 S rpcsvc 0x9bd6250 nfscbd: service 100184 S rpcsvc 0x22c648d0 nfscbd: service 727 1 727 0 SWs accept 0x22c8b30c nfscbd 691 1 691 0 Ss nanslp 0x1d5f8cc cron 687 1 687 25 Ss pause 0x9b08d74 sendmail 684 1 684 0 Ss select 0x1fc0ff24 sendmail 681 1 681 0 Ss select 0xb9d3ca4 sshd 641 1 640 0 S nanslp 0x1d5f8ce smartd 617 1 617 0 Ss select 0x9bf6ce4 powerd 614 1 614 0 Ss (threaded) ntpd 100168 S select 0x1fc0fde4 ntpd 578 1 578 0 Ss rpcsvc 0x22c64b50 NLM: master 575 1 575 0 Ss select 0x188269e4 rpc.statd 574 571 571 0 S (threaded) nfsd 100135 S rpcsvc 0x188722d0 nfsd: master 100136 S rpcsvc 0x1891aad0 nfsd: service 100137 S rpcsvc 0x1835d150 nfsd: service 100138 S rpcsvc 0x1891aa50 nfsd: service 100139 S rpcsvc 0x1891a9d0 nfsd: service 100140 S rpcsvc 0x1835d0d0 nfsd: service 100141 S rpcsvc 0x1891a950 nfsd: service 100142 S rpcsvc 0x1891a8d0 nfsd: service 100143 S rpcsvc 0x1835d050 nfsd: service 100144 S rpcsvc 0x1891a850 nfsd: service 100145 S rpcsvc 0x1891a7d0 nfsd: service 100146 S rpcsvc 0x22c64f50 nfsd: service 100147 S rpcsvc 0x1891a750 nfsd: service 100148 S rpcsvc 0x1891a6d0 nfsd: service 100149 S rpcsvc 0x22c64ed0 nfsd: service 100150 S rpcsvc 0x1891a650 nfsd: service 100151 S rpcsvc 0x22c64e50 nfsd: service 100152 S rpcsvc 0x1891a5d0 nfsd: service 100153 S rpcsvc 0x1891a550 nfsd: service 100154 S rpcsvc 0x22c64dd0 nfsd: service 100155 S rpcsvc 0x1891a4d0 nfsd: service 100156 S rpcsvc 0x1891a450 nfsd: service 100157 S rpcsvc 0x22c64d50 nfsd: service 100158 S rpcsvc 0x1891a3d0 nfsd: service 100159 S rpcsvc 0x22c64cd0 nfsd: service 100160 S rpcsvc 0x1891a350 nfsd: service 100161 S rpcsvc 0x22c64c50 nfsd: service 100162 S rpcsvc 0x1891a2d0 nfsd: service 100163 S rpcsvc 0x1891a250 nfsd: service 100164 S rpcsvc 0x22c64bd0 nfsd: service 100165 S rpcsvc 0x1891a1d0 nfsd: service 100166 S rpcsvc 0x1891a150 nfsd: service 571 1 571 0 Ss select 0xb9d3d24 nfsd 565 1 565 0 Ss select 0x9bf6ca4 mountd 559 554 554 0 S select 0x18826a24 nfsuserd 557 554 554 0 S select 0x1fc0fe64 nfsuserd 556 554 554 0 S select 0x18826a64 nfsuserd 555 554 554 0 S select 0x1fc0fea4 nfsuserd 554 1 554 0 SWs pause 0x204273fc nfsuserd 511 1 511 0 Ss nanslp 0x1d5f8cc watchdogd 422 1 422 0 Ss select 0x18826ae4 rpcbind 413 1 413 0 Ss select 0xb9d3da4 syslogd 341 1 341 0 Ss select 0xb9d3d64 devd 24 0 0 0 DL syncer 0x1dd07b8 [syncer] 23 0 0 0 DL vlruwt 0x182cb9cc [vnlru] 22 0 0 0 DL (threaded) [bufdaemon] 100099 D qsleep 0x1dcff04 [bufdaemon] 100100 D - 0x1a82340 [bufspacedaemon-0] 100114 D sdflush 0x9bd8684 [/home worker] 100115 D sdflush 0x9bd8484 [/tmp worker] 100116 D sdflush 0x9bd8284 [/usr worker] 100117 D biord 0xb7684c8 [/var worker] 21 0 0 0 DL psleep 0x1de7270 [vmdaemon] 20 0 0 0 DL (threaded) [pagedaemon] 100097 D psleep 0x1de62bc [dom0] 100101 D launds 0x1de62c8 [laundry: dom0] 100102 D umarcl 0x1284ce0 [uma] 19 0 0 0 DL - 0x1bad37c [rand_harvestq] 18 0 0 0 DL - 0x1dcfa68 [soaiod4] 17 0 0 0 DL - 0x1dcfa68 [soaiod3] 16 0 0 0 DL - 0x1dcfa68 [soaiod2] 9 0 0 0 DL - 0x1dcfa68 [soaiod1] 8 0 0 0 DL - 0x9bf743c [fdc0] 15 0 0 0 DL (threaded) [usb] 100049 D - 0x18019aac [usbus0] 100050 D - 0x18019adc [usbus0] 100051 D - 0x18019b0c [usbus0] 100052 D - 0x18019b3c [usbus0] 100053 D - 0x18019b6c [usbus0] 100055 D - 0x182d4aac [usbus1] 100056 D - 0x182d4adc [usbus1] 100057 D - 0x182d4b0c [usbus1] 100058 D - 0x182d4b3c [usbus1] 100059 D - 0x182d4b6c [usbus1] 100061 D - 0x182deaac [usbus2] 100062 D - 0x182deadc [usbus2] 100063 D - 0x182deb0c [usbus2] 100064 D - 0x182deb3c [usbus2] 100065 D - 0x182deb6c [usbus2] 100066 D - 0x182e6aac [usbus3] 100067 D - 0x182e6adc [usbus3] 100068 D - 0x182e6b0c [usbus3] 100069 D - 0x182e6b3c [usbus3] 100070 D - 0x182e6b6c [usbus3] 100071 D - 0x182eeaac [usbus4] 100072 D - 0x182eeadc [usbus4] 100073 D - 0x182eeb0c [usbus4] 100074 D - 0x182eeb3c [usbus4] 100075 D - 0x182eeb6c [usbus4] 100077 D - 0x18014c84 [usbus5] 100078 D - 0x18014cb4 [usbus5] 100079 D - 0x18014ce4 [usbus5] 100080 D - 0x18014d14 [usbus5] 100081 D - 0x18014d44 [usbus5] 7 0 0 0 DL (threaded) [cam] 100044 D - 0x1ae46c0 [doneq0] 100096 D - 0x1ae45ec [scanner] 6 0 0 0 DL crypto_ 0xb9f89c4 [crypto returns 3] 5 0 0 0 DL crypto_ 0xb9f898c [crypto returns 2] 4 0 0 0 DL crypto_ 0xb9f8954 [crypto returns 1] 3 0 0 0 DL crypto_ 0xb9f891c [crypto returns 0] 2 0 0 0 DL crypto_ 0x1de4a20 [crypto] 14 0 0 0 DL seqstat 0x9ae8e4c [sequencer 00] 13 0 0 0 DL (threaded) [geom] 100031 D - 0x1d4c288 [g_event] 100032 D - 0x1d4c28c [g_up] 100033 D - 0x1d4c290 [g_down] 12 0 0 0 WL (threaded) [intr] 100016 I [swi6: Giant taskq] 100019 I [swi5: fast taskq] 100022 I [swi6: task queue] 100024 I [swi4: clock (0)] 100025 I [swi4: clock (1)] 100026 I [swi4: clock (2)] 100027 I [swi4: clock (3)] 100028 I [swi1: netisr 0] 100029 I [swi3: vm] 100045 I [irq24: hdac0] 100046 I [irq25: re0] 100047 I [irq22: ahci0] 100048 I [irq16: hdac1 ohci0] 100054 I [irq17: ohci1 ohci3] 100060 I [irq18: ohci2 ohci4] 100076 I [irq19: ehci0] 100082 I [irq14: ata0] 100083 I [swi0: uart] 100085 I [irq1: atkbd0] 11 0 0 0 RL (threaded) [idle] 100003 Run CPU 0 [idle: cpu0] 100004 Run CPU 1 [idle: cpu1] 100005 Run CPU 2 [idle: cpu2] 100006 CanRun [idle: cpu3] 1 0 1 0 SLs wait 0xb9c16dc [init] 10 0 0 0 DL audit_w 0x1de4d0c [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D swapin 0x1d4c75c [swapper] 100007 D - 0xb9caa00 [if_io_tqg_0] 100008 D - 0xb9ca800 [if_io_tqg_1] 100009 D - 0xb9ca600 [if_io_tqg_2] 100010 D - 0xb9ca400 [if_io_tqg_3] 100011 D - 0xb9ca200 [if_config_tqg_0] 100012 D - 0xb9ca000 [softirq_0] 100013 D - 0xb9f0d00 [softirq_1] 100014 D - 0xb9f0b00 [softirq_2] 100015 D - 0xb9f0900 [softirq_3] 100017 D - 0xb9f0500 [in6m_free taskq] 100018 D - 0xb9f0300 [thread taskq] 100020 D - 0xb9f4e00 [aiod_kick taskq] 100021 D - 0xb9f4c00 [kqueue_ctx taskq] 100023 D - 0xb9f4800 [inm_free taskq] 100030 D - 0xb9f4000 [firmware taskq] 100035 D - 0xb9ff900 [crypto_0] 100036 D - 0xb9ff900 [crypto_1] 100037 D - 0xb9ff900 [crypto_2] 100038 D - 0xb9ff900 [crypto_3] 100086 D - 0x18315800 [mca taskq] 100092 D - 0x187ff100 [acpi_task_0] 100093 D - 0x187ff100 [acpi_task_1] 100094 D - 0x187ff100 [acpi_task_2] 100095 D - 0x9b0a500 [CAM taskq] db:0:pho> show freepages DOMAIN: 0 FREE LIST 0: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 09 (002048K) | 000000 08 (001024K) | 000000 07 (000512K) | 000000 06 (000256K) | 000000 05 (000128K) | 000000 04 (000064K) | 000000 03 (000032K) | 000000 02 (000016K) | 000000 01 (000008K) | 000000 00 (000004K) | 000000 FREE LIST 1: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 09 (002048K) | 000000 08 (001024K) | 000009 07 (000512K) | 000262 06 (000256K) | 001749 05 (000128K) | 004477 04 (000064K) | 009575 03 (000032K) | 019055 02 (000016K) | 024840 01 (000008K) | 022440 00 (000004K) | 000000 FREE LIST 2: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 09 (002048K) | 000000 08 (001024K) | 000000 07 (000512K) | 000000 06 (000256K) | 000000 05 (000128K) | 000002 04 (000064K) | 000002 03 (000032K) | 000002 02 (000016K) | 000005 01 (000008K) | 000005 00 (000004K) | 000003 db:0:pho> acttrace Tracing command dd pid 48350 tid 100686 td 0x2af64700 (CPU 3) kdb_enter(14df98a,14df98a) at kdb_enter+0x35/frame 0x3ae04410 vpanic(146f152,3ae04458,3ae04458,3ae04488,f39752,...) at vpanic+0x137/frame 0x3ae04438 panic(146f152,0,18d7ef80,fc50a7,18d7ae00,...) at panic+0x14/frame 0x3ae0444c g_io_deliver(261e8cf0,0) at g_io_deliver+0x322/frame 0x3ae04488 g_disk_done(24029678,24029678) at g_disk_done+0xf0/frame 0x3ae044bc cam_iosched_queue_work(18813700,24029678) at cam_iosched_queue_work+0xcb/frame 0x3ae044d8 adastrategy(24029678,0,114,2c4,b9aa800,...) at adastrategy+0x50/frame 0x3ae044f4 g_disk_start(261e8cf0,18d7ef80,18d7ae00,f38877,b9aa800,...) at g_disk_start+0x4de/frame 0x3ae04550 g_io_request(261e8cf0,18d7ef80,33901fec,2b3b598,3ae045f8,...) at g_io_request+0x2aa/frame 0x3ae04578 g_part_start(339019b4,18d88dc0,9ac4900,f38877,b9aa800,...) at g_part_start+0x26f/frame 0x3ae045e8 g_io_request(339019b4,18d88dc0,1d8fb8c,1,41000000,...) at g_io_request+0x2aa/frame 0x3ae04610 g_part_start(2d39fbdc,1,18d8ed00,18d94b80,30ebd600,...) at g_part_start+0x26f/frame 0x3ae04680 g_io_request(2d39fbdc,b8e0400,b8e0400) at g_io_request+0x2aa/frame 0x3ae046a8 g_vfs_strategy(20051828,30ebd600) at g_vfs_strategy+0xe8/frame 0x3ae046c8 softdep_request_cleanup(20165000,2a343758,29a3e000,4) at softdep_request_cleanup+0x2b7/frame 0x3ae04720 ffs_alloc(2aafb4fc,0,0,6a4908,0,1000,2050000,29a3e000,3ae047d8) at ffs_alloc+0x2ad/frame 0x3ae0476c ffs_balloc_ufs2(2a343758,0,0,1000,29a3e000,2050000,3ae04900) at ffs_balloc_ufs2+0xd8b/frame 0x3ae048a0 ffs_write(3ae04a08) at ffs_write+0x216/frame 0x3ae04924 VOP_WRITE_APV(1aabe80,3ae04a08) at VOP_WRITE_APV+0xdf/frame 0x3ae049e0 vn_write(2a29ec30,3ae04bd0,2881e500,1,2af64700) at vn_write+0x305/frame 0x3ae04a3c vn_io_fault_doio(2af64700) at vn_io_fault_doio+0x3a/frame 0x3ae04a84 vn_io_fault1(2af64700) at vn_io_fault1+0x164/frame 0x3ae04b34 vn_io_fault(2a29ec30,3ae04bd0,2881e500,0,2af64700) at vn_io_fault+0x1c4/frame 0x3ae04b70 dofilewrite(2a29ec30,3ae04bd0,ffffffff,ffffffff,0) at dofilewrite+0x71/frame 0x3ae04ba4 sys_write(2af64700,2af64998,2af64700,28409000,2af64700,...) at sys_write+0x78/frame 0x3ae04c08 syscall(3ae04ce8,3b,3b,3b,1000,...) at syscall+0x1b8/frame 0x3ae04cdc Xint0x80_syscall() at 0xffc033f9/frame 0x3ae04cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- Tracing command idle pid 11 tid 100003 td 0xb9c3500 (CPU 0) cpustop_handler(12b21bc4,12b21bb8,13d05cb,8,14a9f5c,...) at cpustop_handler+0x22/frame 0x12b21aec ipi_nmi_handler(8,14a9f5c,a40,b9f2a80,b9c1398,...) at ipi_nmi_handler+0x30/frame 0x12b21af8 trap(12b21bc4,8,28,28,9bd8e00,...) at trap+0xbb/frame 0x12b21bb8 calltrap() at 0xffc0319f/frame 0x12b21bb8 --- trap 0x13, eip = 0x1384635, esp = 0x12b21c04, ebp = 0x12b21c04 --- acpi_cpu_c1(9717,66c4262f,9717,c9dcc63a,3f68,...) at acpi_cpu_c1+0x5/frame 0x12b21c04 acpi_cpu_idle(7b243f8,0) at acpi_cpu_idle+0x22f/frame 0x12b21c34 cpu_idle_acpi(7b243f8,0) at cpu_idle_acpi+0x35/frame 0x12b21c48 cpu_idle(0) at cpu_idle+0xa3/frame 0x12b21c64 sched_idletd(0,12b21ce8) at sched_idletd+0x33e/frame 0x12b21cb8 fork_exit(10117d0,0,12b21ce8,0,0,...) at fork_exit+0x68/frame 0x12b21cd4 fork_trampoline() at 0xffc0340e/frame 0x12b21cd4 --- kthread start Tracing command idle pid 11 tid 100004 td 0xb9c3180 (CPU 1) cpustop_handler(12b26bc4,12b26bb8,13d05cb,a4c795a,fc53e3,...) at cpustop_handler+0x22/frame 0x12b26aec ipi_nmi_handler(a4c795a,fc53e3,0,8,b9c1398,...) at ipi_nmi_handler+0x30/frame 0x12b26af8 trap(12b26bc4,8,28,28,9bd8c00,...) at trap+0xbb/frame 0x12b26bb8 calltrap() at 0xffc0319f/frame 0x12b26bb8 --- trap 0x13, eip = 0x1384635, esp = 0x12b26c04, ebp = 0x12b26c04 --- acpi_cpu_c1(9717,66c0ed4e,9717,c9db7ace,3f68,...) at acpi_cpu_c1+0x5/frame 0x12b26c04 acpi_cpu_idle(316fdade,0) at acpi_cpu_idle+0x22f/frame 0x12b26c34 cpu_idle_acpi(316fdade,0) at cpu_idle_acpi+0x35/frame 0x12b26c48 cpu_idle(0) at cpu_idle+0xa3/frame 0x12b26c64 sched_idletd(0,12b26ce8) at sched_idletd+0x33e/frame 0x12b26cb8 fork_exit(10117d0,0,12b26ce8,0,0,...) at fork_exit+0x68/frame 0x12b26cd4 fork_trampoline() at 0xffc0340e/frame 0x12b26cd4 --- kthread start Tracing command idle pid 11 tid 100005 td 0xb9c2e00 (CPU 2) cpustop_handler(12b2bbc4,12b2bbb8,13d05cb,a4c795a,fc53e3,...) at cpustop_handler+0x22/frame 0x12b2baec ipi_nmi_handler(a4c795a,fc53e3,0,8,b9c1398,...) at ipi_nmi_handler+0x30/frame 0x12b2baf8 trap(12b2bbc4,8,28,28,9bd8a00,...) at trap+0xbb/frame 0x12b2bbb8 calltrap() at 0xffc0319f/frame 0x12b2bbb8 --- trap 0x13, eip = 0x1384635, esp = 0x12b2bc04, ebp = 0x12b2bc04 --- acpi_cpu_c1(9717,66c38b85,9717,c9dc83e5,3f68,...) at acpi_cpu_c1+0x5/frame 0x12b2bc04 acpi_cpu_idle(38802e09,0) at acpi_cpu_idle+0x22f/frame 0x12b2bc34 cpu_idle_acpi(38802e09,0) at cpu_idle_acpi+0x35/frame 0x12b2bc48 cpu_idle(0) at cpu_idle+0xa3/frame 0x12b2bc64 sched_idletd(0,12b2bce8) at sched_idletd+0x33e/frame 0x12b2bcb8 fork_exit(10117d0,0,12b2bce8,0,0,...) at fork_exit+0x68/frame 0x12b2bcd4 fork_trampoline() at 0xffc0340e/frame 0x12b2bcd4 --- kthread start db:0:pho> alltrace Tracing command dd pid 48350 tid 100686 td 0x2af64700 (CPU 3) kdb_enter(14df98a,14df98a) at kdb_enter+0x35/frame 0x3ae04410 vpanic(146f152,3ae04458,3ae04458,3ae04488,f39752,...) at vpanic+0x137/frame 0x3ae04438 panic(146f152,0,18d7ef80,fc50a7,18d7ae00,...) at panic+0x14/frame 0x3ae0444c g_io_deliver(261e8cf0,0) at g_io_deliver+0x322/frame 0x3ae04488 g_disk_done(24029678,24029678) at g_disk_done+0xf0/frame 0x3ae044bc cam_iosched_queue_work(18813700,24029678) at cam_iosched_queue_work+0xcb/frame 0x3ae044d8 adastrategy(24029678,0,114,2c4,b9aa800,...) at adastrategy+0x50/frame 0x3ae044f4 g_disk_start(261e8cf0,18d7ef80,18d7ae00,f38877,b9aa800,...) at g_disk_start+0x4de/frame 0x3ae04550 g_io_request(261e8cf0,18d7ef80,33901fec,2b3b598,3ae045f8,...) at g_io_request+0x2aa/frame 0x3ae04578 g_part_start(339019b4,18d88dc0,9ac4900,f38877,b9aa800,...) at g_part_start+0x26f/frame 0x3ae045e8 g_io_request(339019b4,18d88dc0,1d8fb8c,1,41000000,...) at g_io_request+0x2aa/frame 0x3ae04610 g_part_start(2d39fbdc,1,18d8ed00,18d94b80,30ebd600,...) at g_part_start+0x26f/frame 0x3ae04680 g_io_request(2d39fbdc,b8e0400,b8e0400) at g_io_request+0x2aa/frame 0x3ae046a8 g_vfs_strategy(20051828,30ebd600) at g_vfs_strategy+0xe8/frame 0x3ae046c8 softdep_request_cleanup(20165000,2a343758,29a3e000,4) at softdep_request_cleanup+0x2b7/frame 0x3ae04720 ffs_alloc(2aafb4fc,0,0,6a4908,0,1000,2050000,29a3e000,3ae047d8) at ffs_alloc+0x2ad/frame 0x3ae0476c ffs_balloc_ufs2(2a343758,0,0,1000,29a3e000,2050000,3ae04900) at ffs_balloc_ufs2+0xd8b/frame 0x3ae048a0 ffs_write(3ae04a08) at ffs_write+0x216/frame 0x3ae04924 VOP_WRITE_APV(1aabe80,3ae04a08) at VOP_WRITE_APV+0xdf/frame 0x3ae049e0 vn_write(2a29ec30,3ae04bd0,2881e500,1,2af64700) at vn_write+0x305/frame 0x3ae04a3c vn_io_fault_doio(2af64700) at vn_io_fault_doio+0x3a/frame 0x3ae04a84 vn_io_fault1(2af64700) at vn_io_fault1+0x164/frame 0x3ae04b34 vn_io_fault(2a29ec30,3ae04bd0,2881e500,0,2af64700) at vn_io_fault+0x1c4/frame 0x3ae04b70 dofilewrite(2a29ec30,3ae04bd0,ffffffff,ffffffff,0) at dofilewrite+0x71/frame 0x3ae04ba4 sys_write(2af64700,2af64998,2af64700,28409000,2af64700,...) at sys_write+0x78/frame 0x3ae04c08 syscall(3ae04ce8,3b,3b,3b,1000,...) at syscall+0x1b8/frame 0x3ae04cdc Xint0x80_syscall() at 0xffc033f9/frame 0x3ae04cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- Tracing command sh pid 48347 tid 102298 td 0x2f4aa880 sched_switch(2f4aa880,104) at sched_switch+0x4e9/frame 0x3713d9c8 mi_switch(104) at mi_switch+0x1c4/frame 0x3713d9fc sleepq_switch(3a80,14e4b25,6c,0,1d74c54,...) at sleepq_switch+0xf7/frame 0x3713da1c sleepq_catch_signals(3713da8c,feff3c,36b736dc,6c,200246,...) at sleepq_catch_signals+0x278/frame 0x3713da44 sleepq_wait_sig(36b736dc,6c) at sleepq_wait_sig+0xe/frame 0x3713da4c _sleep(36b736dc,36b73784,36c,15016eb,0,0,0,0,100) at _sleep+0x27c/frame 0x3713da8c kern_wait6(2f4aa880,7,0,0,3713db14,30,0,0) at kern_wait6+0x3b8/frame 0x3713dae8 sys_wait4(2f4aa880,2f4aab18,2f4aa880,36b736dc,2f4aa880,...) at sys_wait4+0x74/frame 0x3713dc08 syscall(3713dce8,3b,3b,3b,ffbfeb4c,...) at syscall+0x1b8/frame 0x3713dcdc Xint0x80_syscall() at 0xffc033f9/frame 0x3713dcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe75c --- Tracing command cron pid 48346 tid 101241 td 0x27ed8c00 sched_switch(27ed8c00,104) at sched_switch+0x4e9/frame 0x2c0fba70 mi_switch(104) at mi_switch+0x1c4/frame 0x2c0fbaa4 sleepq_switch(2000,14e4b25,5c,0,1d731d4,...) at sleepq_switch+0xf7/frame 0x2c0fbac4 sleepq_catch_signals(2c0fbb34,feff3c,2eefc848,5c,2eefc848,...) at sleepq_catch_signals+0x278/frame 0x2c0fbaec sleepq_wait_sig(2eefc848,5c) at sleepq_wait_sig+0xe/frame 0x2c0fbaf4 _sleep(2eefc848,2eefc9d8,15c,14e4cd5,0,0,0,0,100) at _sleep+0x27c/frame 0x2c0fbb34 pipe_read(2b781b04,2c0fbbd0,22e6be00,0,27ed8c00) at pipe_read+0x452/frame 0x2c0fbb74 dofileread(2b781b04,2c0fbbd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x2c0fbba4 sys_read(27ed8c00,27ed8e98,27ed8c00,2b838344,27ed8c00,...) at sys_read+0x78/frame 0x2c0fbc08 syscall(2c0fbce8,3b,3b,3b,20650b2c,...) at syscall+0x1b8/frame 0x2c0fbcdc Xint0x80_syscall() at 0xffc033f9/frame 0x2c0fbcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d98, esp = 0xffc0bfe8, ebp = 0xffbfecbc --- Tracing command sleep pid 48345 tid 101664 td 0x28f59000 sched_switch(28f59000,104) at sched_switch+0x4e9/frame 0x34236a70 mi_switch(104) at mi_switch+0x1c4/frame 0x34236aa8 sleepq_switch(dc0,0,6c,0,1d71f94,...) at sleepq_switch+0xf7/frame 0x34236ac8 sleepq_catch_signals(14e4b25,183,28f59000,100,1d5f8cf,...) at sleepq_catch_signals+0x278/frame 0x34236af0 sleepq_timedwait_sig(1d5f8cf,6c,34236b50,fc50a7,25fe8ac8,...) at sleepq_timedwait_sig+0x17/frame 0x34236b1c _sleep(1d5f8cf,0,16c,14f7fc3,58e61b6e,971c,a0000000,0,200) at _sleep+0x21a/frame 0x34236b5c kern_clock_nanosleep(28f59000,0,0,34236be0,34236be8) at kern_clock_nanosleep+0x1b7/frame 0x34236bc4 sys_nanosleep(28f59000,28f59298,28f59000,25fe8a20,28f59000,...) at sys_nanosleep+0x37/frame 0x34236c08 syscall(34236ce8,3b,3b,3b,ffbfe760,...) at syscall+0x1b8/frame 0x34236cdc Xint0x80_syscall() at 0xffc033f9/frame 0x34236cdc --- syscall (560, FreeBSD ELF32, sys_kevent), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe878 --- Tracing command egrep pid 47235 tid 102153 td 0x2506a380 sched_switch(2506a380,104) at sched_switch+0x4e9/frame 0x36c64a70 mi_switch(104) at mi_switch+0x1c4/frame 0x36c64aa4 sleepq_switch(1f00,14e4b25,5c,0,1d730d4,...) at sleepq_switch+0xf7/frame 0x36c64ac4 sleepq_catch_signals(36c64b34,feff3c,2a2b84f8,5c,2a2b84f8,...) at sleepq_catch_signals+0x278/frame 0x36c64aec sleepq_wait_sig(2a2b84f8,5c) at sleepq_wait_sig+0xe/frame 0x36c64af4 _sleep(2a2b84f8,2a2b8688,15c,14e4cd5,0,0,0,0,100) at _sleep+0x27c/frame 0x36c64b34 pipe_read(27efb21c,36c64bd0,22c69200,0,2506a380) at pipe_read+0x452/frame 0x36c64b74 dofileread(27efb21c,36c64bd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x36c64ba4 sys_read(2506a380,2506a618,2506a380,326bb000,2506a380,...) at sys_read+0x78/frame 0x36c64c08 syscall(36c64ce8,3b,3b,3b,417f80,...) at syscall+0x1b8/frame 0x36c64cdc Xint0x80_syscall() at 0xffc033f9/frame 0x36c64cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d98, esp = 0xffc0bfe8, ebp = 0xffbfecbc --- Tracing command dd pid 47234 tid 100989 td 0x2f318380 sched_switch(2f318380,104) at sched_switch+0x4e9/frame 0x34f646f0 mi_switch(104) at mi_switch+0x1c4/frame 0x34f64728 sleepq_switch(1dcfefc,54) at sleepq_switch+0xf7/frame 0x34f64748 _sleep(1dcfefc,1a81000,54,14061ca,0,0,0,0,100) at _sleep+0x2f3/frame 0x34f64788 bufwrite(18d41c40) at bufwrite+0x29a/frame 0x34f647c8 cluster_wbuild(2bc23bc0,4000,348c,0,8,8) at cluster_wbuild+0x682/frame 0x34f64818 cluster_write(2bc23bc0,b762de0,d250000,0,7f,8) at cluster_write+0x184/frame 0x34f648a4 ffs_write(34f64a08) at ffs_write+0x458/frame 0x34f64924 VOP_WRITE_APV(1aabe80,34f64a08) at VOP_WRITE_APV+0xdf/frame 0x34f649e0 vn_write(2b8264ec,34f64bd0,2720bb00,1,2f318380) at vn_write+0x305/frame 0x34f64a3c vn_io_fault_doio(2f318380) at vn_io_fault_doio+0x3a/frame 0x34f64a84 vn_io_fault1(2f318380) at vn_io_fault1+0x164/frame 0x34f64b34 vn_io_fault(2b8264ec,34f64bd0,2720bb00,0,2f318380) at vn_io_fault+0x1c4/frame 0x34f64b70 dofilewrite(2b8264ec,34f64bd0,ffffffff,ffffffff,0) at dofilewrite+0x71/frame 0x34f64ba4 sys_write(2f318380,2f318618,2f318380,27ee0398,a4c795a,...) at sys_write+0x78/frame 0x34f64c08 syscall(34f64ce8,3b,3b,3b,100000,...) at syscall+0x1b8/frame 0x34f64cdc Xint0x80_syscall() at 0xffc033f9/frame 0x34f64cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe75c --- Tracing command tee pid 57173 tid 101630 td 0x2ea51180 sched_switch(2ea51180,104) at sched_switch+0x4e9/frame 0x33feca70 mi_switch(104) at mi_switch+0x1c4/frame 0x33fecaa4 sleepq_switch(c0,14e4b25,5c,0,1d71294,...) at sleepq_switch+0xf7/frame 0x33fecac4 sleepq_catch_signals(33fecb34,feff3c,23445350,5c,23445350,...) at sleepq_catch_signals+0x278/frame 0x33fecaec sleepq_wait_sig(23445350,5c) at sleepq_wait_sig+0xe/frame 0x33fecaf4 _sleep(23445350,234454e0,15c,14e4cd5,0,0,0,0,100) at _sleep+0x27c/frame 0x33fecb34 pipe_read(2b832564,33fecbd0,2bbefc00,0,2ea51180) at pipe_read+0x452/frame 0x33fecb74 dofileread(2b832564,33fecbd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x33fecba4 sys_read(2ea51180,2ea51418,2ea51180,2e2ab688,a4c795a,...) at sys_read+0x78/frame 0x33fecc08 syscall(33fecce8,3b,3b,3b,2062d016,...) at syscall+0x1b8/frame 0x33feccdc Xint0x80_syscall() at 0xffc033f9/frame 0x33feccdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d98, esp = 0xffc0bfe8, ebp = 0xffbfecbc --- Tracing command sh pid 57172 tid 101625 td 0x29133380 sched_switch(29133380,104) at sched_switch+0x4e9/frame 0x33fcb9c8 mi_switch(104) at mi_switch+0x1c4/frame 0x33fcb9fc sleepq_switch(680,14e4b25,6c,0,1d71854,...) at sleepq_switch+0xf7/frame 0x33fcba1c sleepq_catch_signals(33fcba8c,feff3c,2e2ac6dc,6c,200246,...) at sleepq_catch_signals+0x278/frame 0x33fcba44 sleepq_wait_sig(2e2ac6dc,6c) at sleepq_wait_sig+0xe/frame 0x33fcba4c _sleep(2e2ac6dc,2e2ac784,36c,15016eb,0,0,0,0,100) at _sleep+0x27c/frame 0x33fcba8c kern_wait6(29133380,7,0,0,33fcbb14,30,0,0) at kern_wait6+0x3b8/frame 0x33fcbae8 sys_wait4(29133380,29133618,29133380,2e2ac6dc,29133380,...) at sys_wait4+0x74/frame 0x33fcbc08 syscall(33fcbce8,3b,3b,3b,ffbfe4f4,...) at syscall+0x1b8/frame 0x33fcbcdc Xint0x80_syscall() at 0xffc033f9/frame 0x33fcbcdc --- syscall (560, FreeBSD ELF32, sys_kevent), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe878 --- Tracing command sh pid 57171 tid 100253 td 0x2af48180 sched_switch(2af48180,104) at sched_switch+0x4e9/frame 0x3aadc9c8 mi_switch(104) at mi_switch+0x1c4/frame 0x3aadc9fc sleepq_switch(1540,14e4b25,6c,0,1d72714,...) at sleepq_switch+0xf7/frame 0x3aadca1c sleepq_catch_signals(3aadca8c,feff3c,284099cc,6c,200246,...) at sleepq_catch_signals+0x278/frame 0x3aadca44 sleepq_wait_sig(284099cc,6c) at sleepq_wait_sig+0xe/frame 0x3aadca4c _sleep(284099cc,28409a74,36c,15016eb,0,0,0,0,100) at _sleep+0x27c/frame 0x3aadca8c kern_wait6(2af48180,7,0,0,3aadcb14,30,0,0) at kern_wait6+0x3b8/frame 0x3aadcae8 sys_wait4(2af48180,2af48418,2af48180,284099cc,2af48180,...) at sys_wait4+0x74/frame 0x3aadcc08 syscall(3aadcce8,3b,3b,3b,ffbfe534,...) at syscall+0x1b8/frame 0x3aadccdc Xint0x80_syscall() at 0xffc033f9/frame 0x3aadccdc --- syscall (560, FreeBSD ELF32, sys_kevent), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe878 --- Tracing command bash pid 46027 tid 100229 td 0x27ed7380 sched_switch(27ed7380,104) at sched_switch+0x4e9/frame 0x2798b938 mi_switch(104) at mi_switch+0x1c4/frame 0x2798b970 sleepq_switch(3340,27ed7380,0,0,1d74514,...) at sleepq_switch+0xf7/frame 0x2798b990 sleepq_catch_signals(2798b9f8,f845b8,9bf69a4,0,13fc2f9,...) at sleepq_catch_signals+0x278/frame 0x2798b9b8 sleepq_wait_sig(9bf69a4,0) at sleepq_wait_sig+0xe/frame 0x2798b9c0 _cv_wait_sig(9bf69a4,9bf6990) at _cv_wait_sig+0x138/frame 0x2798b9f8 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x2798ba14 kern_select(27ed7380,1,ffbfe018,0,0,0,20) at kern_select+0x8e5/frame 0x2798bb9c sys_pselect(27ed7380,27ed7618,27ed7380,27edf688,27ed7380,...) at sys_pselect+0x10c/frame 0x2798bc08 syscall(2798bce8,3b,3b,3b,ffbfe018,...) at syscall+0x1b8/frame 0x2798bcdc Xint0x80_syscall() at 0xffc033f9/frame 0x2798bcdc --- syscall (560, FreeBSD ELF32, sys_kevent), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe878 --- Tracing command sshd pid 46018 tid 100249 td 0x22c6a380 sched_switch(22c6a380,104) at sched_switch+0x4e9/frame 0x28251970 mi_switch(104) at mi_switch+0x1c4/frame 0x282519a8 sleepq_switch(2700,22c6a380,0,0,1d738d4,...) at sleepq_switch+0xf7/frame 0x282519c8 sleepq_catch_signals(28251a30,f845b8,b9d38a4,0,13fc2f9,...) at sleepq_catch_signals+0x278/frame 0x282519f0 sleepq_wait_sig(b9d38a4,0) at sleepq_wait_sig+0xe/frame 0x282519f8 _cv_wait_sig(b9d38a4,b9d3890) at _cv_wait_sig+0x138/frame 0x28251a30 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x28251a4c kern_select(22c6a380,10,2047c270,2047c278,0,0,20) at kern_select+0x8e5/frame 0x28251bd4 sys_select(22c6a380,22c6a618,22c6a380,27f86054,22c6a380,...) at sys_select+0x41/frame 0x28251c08 syscall(28251ce8,3b,3b,3b,20482000,...) at syscall+0x1b8/frame 0x28251cdc Xint0x80_syscall() at 0xffc033f9/frame 0x28251cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe75c --- Tracing command sshd pid 45996 tid 100347 td 0x27f65a80 sched_switch(27f65a80,104) at sched_switch+0x4e9/frame 0x2a5309a0 mi_switch(104) at mi_switch+0x1c4/frame 0x2a5309d8 sleepq_switch(3540,27f65a80,0,0,1d74714,...) at sleepq_switch+0xf7/frame 0x2a5309f8 sleepq_catch_signals(2a530a60,f845b8,1fc0f124,0,13fc2f9,...) at sleepq_catch_signals+0x278/frame 0x2a530a20 sleepq_wait_sig(1fc0f124,0) at sleepq_wait_sig+0xe/frame 0x2a530a28 _cv_wait_sig(1fc0f124,1fc0f110) at _cv_wait_sig+0x138/frame 0x2a530a60 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x2a530a7c kern_poll(27f65a80,ffbfe7a0,1,0,0) at kern_poll+0x4a4/frame 0x2a530bd4 sys_poll(27f65a80,27f65d18,27f65a80,2b838d10,a4c795a,...) at sys_poll+0x50/frame 0x2a530c08 syscall(2a530ce8,3b,3b,3b,20482000,...) at syscall+0x1b8/frame 0x2a530cdc Xint0x80_syscall() at 0xffc033f9/frame 0x2a530cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- Tracing command tail pid 6156 tid 100285 td 0x28cbb500 sched_switch(28cbb500,104) at sched_switch+0x4e9/frame 0x28305970 mi_switch(104) at mi_switch+0x1c4/frame 0x283059a8 sleepq_switch(2300,28cbb500,0,0,1d734d4,...) at sleepq_switch+0xf7/frame 0x283059c8 sleepq_catch_signals(28305a30,f845b8,9bf68e4,0,13fc2f9,...) at sleepq_catch_signals+0x278/frame 0x283059f0 sleepq_wait_sig(9bf68e4,0) at sleepq_wait_sig+0xe/frame 0x283059f8 _cv_wait_sig(9bf68e4,9bf68d0) at _cv_wait_sig+0x138/frame 0x28305a30 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x28305a4c kern_select(28cbb500,7,ffbfe780,0,0,0,20) at kern_select+0x8e5/frame 0x28305bd4 sys_select(28cbb500,28cbb798,28cbb500,27fed344,a4c795a,...) at sys_select+0x41/frame 0x28305c08 syscall(28305ce8,3b,3b,3b,7,...) at syscall+0x1b8/frame 0x28305cdc Xint0x80_syscall() at 0xffc033f9/frame 0x28305cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe75c --- Tracing command tail pid 6153 tid 100291 td 0x28cba000 sched_switch(28cba000,104) at sched_switch+0x4e9/frame 0x283237c8 mi_switch(104) at mi_switch+0x1c4/frame 0x28323800 sleepq_switch(2700,0,68,0,1d738d4,...) at sleepq_switch+0xf7/frame 0x28323820 sleepq_catch_signals(14e4b25,183,28cba000,100,28989c00,...) at sleepq_catch_signals+0x278/frame 0x28323848 sleepq_timedwait_sig(28989c00,68,150be91,a4c795a,14da9a5,...) at sleepq_timedwait_sig+0x17/frame 0x28323874 _sleep(28989c00,28989c00,168,14b39c0,64ac4588,9718,10000000,0,200) at _sleep+0x21a/frame 0x283238b4 kqueue_kevent(0,1,28323be0,28323bb0) at kqueue_kevent+0xa0f/frame 0x28323b14 kern_kevent_fp(28cba000,27efb870,0,1,28323be0,28323bb0) at kern_kevent_fp+0x71/frame 0x28323b38 kern_kevent(28cba000,8,0,1,28323be0,28323bb0) at kern_kevent+0x73/frame 0x28323b88 kern_kevent_generic(28323be0,1501664) at kern_kevent_generic+0x5c/frame 0x28323bd0 sys_kevent(28cba000,28cba298,28cba000,28cc9d10,b8565b8,...) at sys_kevent+0x64/frame 0x28323c08 syscall(28323ce8,3b,3b,3b,20675288,...) at syscall+0x1b8/frame 0x28323cdc Xint0x80_syscall() at 0xffc033f9/frame 0x28323cdc --- syscall (560, FreeBSD ELF32, sys_kevent), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe878 --- Tracing command bash pid 5106 tid 100306 td 0x28d91880 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1a5e62c,1a5e644,3ae03f14,103d33c,1a5e644,...) at db_trace_self_wrapper+0x2a/frame 0x3ae03ed0 kdb_backtrace(3ae03ff8,13d05e1,1430c59,0,3ae04030,...) at kdb_backtrace+0x2e/frame 0x3ae03f30 kdb_reenter(1430c59,0,3ae04030,3ae0400c,28409000,...) at kdb_reenter+0x29/frame 0x3ae03f38 trap(3ae04004,8,28,28,ffffffff,...) at trap+0xd1/frame 0x3ae03ff8 calltrap() at 0xffc0319f/frame 0x3ae03ff8 --- trap 0xc, eip = 0x13ac85a, esp = 0x3ae04044, ebp = 0x3ae04050 --- db_trace_thread(28d91880,ffffffff) at db_trace_thread+0x2a/frame 0x3ae04050 _db_stack_trace_all(1028685,0,ffffffff,3ae040dc) at _db_stack_trace_all+0x93/frame 0x3ae040b0 db_command(1ae7672) at db_command+0x26a/frame 0x3ae04164 db_script_exec(bd2daf,0,3ae0420c,3ae041f0) at db_script_exec+0x1a6/frame 0x3ae041c4 db_command(3ae042c8,a06536,0,0,2af64700,...) at db_command+0x26a/frame 0x3ae04278 db_command_loop(0,0,2af64700,3ae04280,3ae042c8,...) at db_command_loop+0x3c/frame 0x3ae04280 db_trap(3,0) at db_trap+0xf6/frame 0x3ae042c8 kdb_trap(3,0,3ae043cc) at kdb_trap+0x109/frame 0x3ae042f4 trap(3ae043cc,8,28,28,100,...) at trap+0x382/frame 0x3ae043c0 calltrap() at 0xffc0319f/frame 0x3ae043c0 --- trap 0x3, eip = 0x1028685, esp = 0x3ae0440c, ebp = 0x3ae04410 --- kdb_enter(14df98a,14df98a) at kdb_enter+0x35/frame 0x3ae04410 vpanic(146f152,3ae04458,3ae04458,3ae04488,f39752,...) at vpanic+0x137/frame 0x3ae04438 panic(146f152,0,18d7ef80,fc50a7,18d7ae00,...) at panic+0x14/frame 0x3ae0444c g_io_deliver(261e8cf0,0) at g_io_deliver+0x322/frame 0x3ae04488 g_disk_done(24029678,24029678) at g_disk_done+0xf0/frame 0x3ae044bc cam_iosched_queue_work(18813700,24029678) at cam_iosched_queue_work+0xcb/frame 0x3ae044d8 adastrategy(24029678,0,114,2c4,b9aa800,...) at adastrategy+0x50/frame 0x3ae044f4 g_disk_start(261e8cf0,18d7ef80,18d7ae00,f38877,b9aa800,...) at g_disk_start+0x4de/frame 0x3ae04550 g_io_request(261e8cf0,18d7ef80,33901fec,2b3b598,3ae045f8,...) at g_io_request+0x2aa/frame 0x3ae04578 g_part_start(339019b4,18d88dc0,9ac4900,f38877,b9aa800,...) at g_part_start+0x26f/frame 0x3ae045e8 g_io_request(339019b4,18d88dc0,1d8fb8c,1,41000000,...) at g_io_request+0x2aa/frame 0x3ae04610 g_part_start(2d39fbdc,1,18d8ed00,18d94b80,30ebd600,...) at g_part_start+0x26f/frame 0x3ae04680 g_io_request(2d39fbdc,b8e0400,b8e0400) at g_io_request+0x2aa/frame 0x3ae046a8 g_vfs_strategy(20051828,30ebd600) at g_vfs_strategy+0xe8/frame 0x3ae046c8 softdep_request_cleanup(20165000,2a343758,29a3e000,4) at softdep_request_cleanup+0x2b7/frame 0x3ae04720 ffs_alloc(2aafb4fc,0,0,6a4908,0,1000,2050000,29a3e000,3ae047d8) at ffs_alloc+0x2ad/frame 0x3ae0476c ffs_balloc_ufs2(2a343758,0,0,1000,29a3e000,2050000,3ae04900) at ffs_balloc_ufs2+0xd8b/frame 0x3ae048a0 ffs_write(3ae04a08) at ffs_write+0x216/frame 0x3ae04924 VOP_WRITE_APV(1aabe80,3ae04a08) at VOP_WRITE_APV+0xdf/frame 0x3ae049e0 vn_write(2a29ec30,3ae04bd0,2881e500,1,2af64700) at vn_write+0x305/frame 0x3ae04a3c vn_io_fault_doio(2af64700) at vn_io_fault_doio+0x3a/frame 0x3ae04a84 vn_io_fault1(2af64700) at vn_io_fault1+0x164/frame 0x3ae04b34 vn_io_fault(2a29ec30,3ae04bd0,2881e500,0,2af64700) at vn_io_fault+0x1c4/frame 0x3ae04b70 dofilewrite(2a29ec30,3ae04bd0,ffffffff,ffffffff,0) at dofilewrite+0x71/frame 0x3ae04ba4 sys_write(2af64700,2af64998,2af64700,28409000,2af64700,...) at sys_write+0x78/frame 0x3ae04c08 syscall(3ae04ce8,3b,3b,3b,1000,...) at syscall+0x1b8/frame 0x3ae04cdc Xint0x80_syscall() at 0xffc033f9/frame 0x3ae04cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- Tracing command sshd pid 5105 tid 100292 td 0x28d16c00 sched_switch(28d16c00,104) at sched_switch+0x4e9/frame 0x28328970 mi_switch(104) at mi_switch+0x1c4/frame 0x283289a8 sleepq_switch(2040,28d16c00,0,0,1d73214,...) at sleepq_switch+0xf7/frame 0x283289c8 sleepq_catch_signals(28328a30,f845b8,188265e4,0,13fc2f9,...) at sleepq_catch_signals+0x278/frame 0x283289f0 sleepq_wait_sig(188265e4,0) at sleepq_wait_sig+0xe/frame 0x283289f8 _cv_wait_sig(188265e4,188265d0) at _cv_wait_sig+0x138/frame 0x28328a30 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x28328a4c kern_select(28d16c00,e,2047c270,2047c278,0,0,20) at kern_select+0x8e5/frame 0x28328bd4 sys_select(28d16c00,28d16e98,28d16c00,28cc99cc,28d16c00,...) at sys_select+0x41/frame 0x28328c08 syscall(28328ce8,3b,3b,3b,20482000,...) at syscall+0x1b8/frame 0x28328cdc Xint0x80_syscall() at 0xffc033f9/frame 0x28328cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- Tracing command sshd pid 5103 tid 100300 td 0x28d15380 sched_switch(28d15380,104) at sched_switch+0x4e9/frame 0x283509a0 mi_switch(104) at mi_switch+0x1c4/frame 0x283509d8 sleepq_switch(300,28d15380,0,0,1d714d4,...) at sleepq_switch+0xf7/frame 0x283509f8 sleepq_catch_signals(28350a60,f845b8,9bf6864,0,13fc2f9,...) at sleepq_catch_signals+0x278/frame 0x28350a20 sleepq_wait_sig(9bf6864,0) at sleepq_wait_sig+0xe/frame 0x28350a28 _cv_wait_sig(9bf6864,9bf6850) at _cv_wait_sig+0x138/frame 0x28350a60 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x28350a7c kern_poll(28d15380,ffbfe7a0,1,0,0) at kern_poll+0x4a4/frame 0x28350bd4 sys_poll(28d15380,28d15618,28d15380,27f71d10,28d15380,...) at sys_poll+0x50/frame 0x28350c08 syscall(28350ce8,3b,3b,3b,20482000,...) at syscall+0x1b8/frame 0x28350cdc Xint0x80_syscall() at 0xffc033f9/frame 0x28350cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d98, esp = 0xffc0bfe8, ebp = 0xffbfecbc --- Tracing command sh pid 1825 tid 100213 td 0x25ad3500 sched_switch(25ad3500,104) at sched_switch+0x4e9/frame 0x27829a70 mi_switch(104) at mi_switch+0x1c4/frame 0x27829aa4 sleepq_switch(f00,14e4b25,5c,0,1d720d4,...) at sleepq_switch+0xf7/frame 0x27829ac4 sleepq_catch_signals(27829b34,feff3c,27edc4f8,5c,27edc4f8,...) at sleepq_catch_signals+0x278/frame 0x27829aec sleepq_wait_sig(27edc4f8,5c) at sleepq_wait_sig+0xe/frame 0x27829af4 _sleep(27edc4f8,27edc688,15c,14e4cd5,0,0,0,0,100) at _sleep+0x27c/frame 0x27829b34 pipe_read(1fc5d0b4,27829bd0,22c69200,0,25ad3500) at pipe_read+0x452/frame 0x27829b74 dofileread(1fc5d0b4,27829bd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x27829ba4 sys_read(25ad3500,25ad3798,25ad3500,2029b000,25ad3500,...) at sys_read+0x78/frame 0x27829c08 syscall(27829ce8,3b,3b,3b,2,...) at syscall+0x1b8/frame 0x27829cdc Xint0x80_syscall() at 0xffc033f9/frame 0x27829cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe75c --- Tracing command awk pid 1823 tid 100222 td 0x27ed8880 sched_switch(27ed8880,104) at sched_switch+0x4e9/frame 0x27968a70 mi_switch(104) at mi_switch+0x1c4/frame 0x27968aa4 sleepq_switch(3800,14e4b25,5c,0,1d749d4,...) at sleepq_switch+0xf7/frame 0x27968ac4 sleepq_catch_signals(27968b34,feff3c,27ece000,5c,27ece000,...) at sleepq_catch_signals+0x278/frame 0x27968aec sleepq_wait_sig(27ece000,5c) at sleepq_wait_sig+0xe/frame 0x27968af4 _sleep(27ece000,27ece190,15c,14e4cd5,0,0,0,0,100) at _sleep+0x27c/frame 0x27968b34 pipe_read(1fc33384,27968bd0,22c69200,0,27ed8880) at pipe_read+0x452/frame 0x27968b74 dofileread(1fc33384,27968bd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x27968ba4 sys_read(27ed8880,27ed8b18,27ed8880,27ee0a20,b8565b8,...) at sys_read+0x78/frame 0x27968c08 syscall(27968ce8,3b,3b,3b,20661e34,...) at syscall+0x1b8/frame 0x27968cdc Xint0x80_syscall() at 0xffc033f9/frame 0x27968cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe75c --- Tracing command sh pid 1822 tid 100218 td 0x25ad2e00 sched_switch(25ad2e00,104) at sched_switch+0x4e9/frame 0x2784c9c8 mi_switch(104) at mi_switch+0x1c4/frame 0x2784c9fc sleepq_switch(1680,14e4b25,6c,0,1d72854,...) at sleepq_switch+0xf7/frame 0x2784ca1c sleepq_catch_signals(2784ca8c,feff3c,27ec86dc,6c,200246,...) at sleepq_catch_signals+0x278/frame 0x2784ca44 sleepq_wait_sig(27ec86dc,6c) at sleepq_wait_sig+0xe/frame 0x2784ca4c _sleep(27ec86dc,27ec8784,36c,15016eb,0,0,0,0,100) at _sleep+0x27c/frame 0x2784ca8c kern_wait6(25ad2e00,7,0,0,2784cb14,30,0,0) at kern_wait6+0x3b8/frame 0x2784cae8 sys_wait4(25ad2e00,25ad3098,25ad2e00,27ec86dc,25ad2e00,...) at sys_wait4+0x74/frame 0x2784cc08 syscall(2784cce8,3b,3b,3b,ffbfe46c,...) at syscall+0x1b8/frame 0x2784ccdc Xint0x80_syscall() at 0xffc033f9/frame 0x2784ccdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d98, esp = 0xffc0bfe8, ebp = 0xffbfecbc --- Tracing command sh pid 1801 tid 100219 td 0x22ebd000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1a5e62c,1a5e644,3ae03f14,103d33c,1a5e644,...) at db_trace_self_wrapper+0x2a/frame 0x3ae03ed0 kdb_backtrace(3ae03ff8,13d05e1,1430c59,0,3ae04030,...) at kdb_backtrace+0x2e/frame 0x3ae03f30 kdb_reenter(1430c59,0,3ae04030,3ae0400c,28409000,...) at kdb_reenter+0x29/frame 0x3ae03f38 trap(3ae04004,8,28,28,ffffffff,...) at trap+0xd1/frame 0x3ae03ff8 calltrap() at 0xffc0319f/frame 0x3ae03ff8 --- trap 0xc, eip = 0x13ac85a, esp = 0x3ae04044, ebp = 0x3ae04050 --- db_trace_thread(22ebd000,ffffffff) at db_trace_thread+0x2a/frame 0x3ae04050 _db_stack_trace_all(1028685,0,ffffffff,3ae040dc) at _db_stack_trace_all+0x93/frame 0x3ae040b0 db_command(1ae7672) at db_command+0x26a/frame 0x3ae04164 db_script_exec(bd2daf,0,3ae0420c,3ae041f0) at db_script_exec+0x1a6/frame 0x3ae041c4 db_command(3ae042c8,a06536,0,0,2af64700,...) at db_command+0x26a/frame 0x3ae04278 db_command_loop(0,0,2af64700,3ae04280,3ae042c8,...) at db_command_loop+0x3c/frame 0x3ae04280 db_trap(3,0) at db_trap+0xf6/frame 0x3ae042c8 kdb_trap(3,0,3ae043cc) at kdb_trap+0x109/frame 0x3ae042f4 trap(3ae043cc,8,28,28,100,...) at trap+0x382/frame 0x3ae043c0 calltrap() at 0xffc0319f/frame 0x3ae043c0 --- trap 0x3, eip = 0x1028685, esp = 0x3ae0440c, ebp = 0x3ae04410 --- kdb_enter(14df98a,14df98a) at kdb_enter+0x35/frame 0x3ae04410 vpanic(146f152,3ae04458,3ae04458,3ae04488,f39752,...) at vpanic+0x137/frame 0x3ae04438 panic(146f152,0,18d7ef80,fc50a7,18d7ae00,...) at panic+0x14/frame 0x3ae0444c g_io_deliver(261e8cf0,0) at g_io_deliver+0x322/frame 0x3ae04488 g_disk_done(24029678,24029678) at g_disk_done+0xf0/frame 0x3ae044bc cam_iosched_queue_work(18813700,24029678) at cam_iosched_queue_work+0xcb/frame 0x3ae044d8 adastrategy(24029678,0,114,2c4,b9aa800,...) at adastrategy+0x50/frame 0x3ae044f4 g_disk_start(261e8cf0,18d7ef80,18d7ae00,f38877,b9aa800,...) at g_disk_start+0x4de/frame 0x3ae04550 g_io_request(261e8cf0,18d7ef80,33901fec,2b3b598,3ae045f8,...) at g_io_request+0x2aa/frame 0x3ae04578 g_part_start(339019b4,18d88dc0,9ac4900,f38877,b9aa800,...) at g_part_start+0x26f/frame 0x3ae045e8 g_io_request(339019b4,18d88dc0,1d8fb8c,1,41000000,...) at g_io_request+0x2aa/frame 0x3ae04610 g_part_start(2d39fbdc,1,18d8ed00,18d94b80,30ebd600,...) at g_part_start+0x26f/frame 0x3ae04680 g_io_request(2d39fbdc,b8e0400,b8e0400) at g_io_request+0x2aa/frame 0x3ae046a8 g_vfs_strategy(20051828,30ebd600) at g_vfs_strategy+0xe8/frame 0x3ae046c8 softdep_request_cleanup(20165000,2a343758,29a3e000,4) at softdep_request_cleanup+0x2b7/frame 0x3ae04720 ffs_alloc(2aafb4fc,0,0,6a4908,0,1000,2050000,29a3e000,3ae047d8) at ffs_alloc+0x2ad/frame 0x3ae0476c ffs_balloc_ufs2(2a343758,0,0,1000,29a3e000,2050000,3ae04900) at ffs_balloc_ufs2+0xd8b/frame 0x3ae048a0 ffs_write(3ae04a08) at ffs_write+0x216/frame 0x3ae04924 VOP_WRITE_APV(1aabe80,3ae04a08) at VOP_WRITE_APV+0xdf/frame 0x3ae049e0 vn_write(2a29ec30,3ae04bd0,2881e500,1,2af64700) at vn_write+0x305/frame 0x3ae04a3c vn_io_fault_doio(2af64700) at vn_io_fault_doio+0x3a/frame 0x3ae04a84 vn_io_fault1(2af64700) at vn_io_fault1+0x164/frame 0x3ae04b34 vn_io_fault(2a29ec30,3ae04bd0,2881e500,0,2af64700) at vn_io_fault+0x1c4/frame 0x3ae04b70 dofilewrite(2a29ec30,3ae04bd0,ffffffff,ffffffff,0) at dofilewrite+0x71/frame 0x3ae04ba4 sys_write(2af64700,2af64998,2af64700,28409000,2af64700,...) at sys_write+0x78/frame 0x3ae04c08 syscall(3ae04ce8,3b,3b,3b,1000,...) at syscall+0x1b8/frame 0x3ae04cdc Xint0x80_syscall() at 0xffc033f9/frame 0x3ae04cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- Tracing command top pid 1060 tid 100204 td 0x24424a80 sched_switch(24424a80,104) at sched_switch+0x4e9/frame 0x207ed930 mi_switch(104) at mi_switch+0x1c4/frame 0x207ed968 sleepq_switch(2780,1d9f784,0,0,1d73954,...) at sleepq_switch+0xf7/frame 0x207ed988 sleepq_catch_signals(e4,24424a80,1a743c8,24424a80,207ed9e4,...) at sleepq_catch_signals+0x278/frame 0x207ed9b0 sleepq_timedwait_sig(b9d3aa4,0) at sleepq_timedwait_sig+0x17/frame 0x207ed9e0 _cv_timedwait_sig_sbt(b9d3aa4,b9d3a90,fec97848,9718,20000000,0,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x207eda1c seltdwait(9718,20000000,0) at seltdwait+0x5e/frame 0x207eda4c kern_select(24424a80,2,ffbfea68,0,0,207edbf8,20) at kern_select+0x8e5/frame 0x207edbd4 sys_select(24424a80,24424d18,24424a80,244b8398,22ff5b8,...) at sys_select+0x41/frame 0x207edc08 syscall(207edce8,3b,3b,3b,ffbfeae8,...) at syscall+0x1b8/frame 0x207edcdc Xint0x80_syscall() at 0xffc033f9/frame 0x207edcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- Tracing command sh pid 984 tid 100209 td 0x24424700 sched_switch(24424700,104) at sched_switch+0x4e9/frame 0x278099c8 mi_switch(104) at mi_switch+0x1c4/frame 0x278099fc sleepq_switch(3500,14e4b25,6c,0,1d746d4,...) at sleepq_switch+0xf7/frame 0x27809a1c sleepq_catch_signals(27809a8c,feff3c,244b8054,6c,200246,...) at sleepq_catch_signals+0x278/frame 0x27809a44 sleepq_wait_sig(244b8054,6c) at sleepq_wait_sig+0xe/frame 0x27809a4c _sleep(244b8054,244b80fc,36c,15016eb,0,0,0,0,100) at _sleep+0x27c/frame 0x27809a8c kern_wait6(24424700,7,0,0,27809b14,30,0,0) at kern_wait6+0x3b8/frame 0x27809ae8 sys_wait4(24424700,24424998,24424700,244b8054,24424700,...) at sys_wait4+0x74/frame 0x27809c08 syscall(27809ce8,3b,3b,3b,ffbfe754,...) at syscall+0x1b8/frame 0x27809cdc Xint0x80_syscall() at 0xffc033f9/frame 0x27809cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- Tracing command bash pid 947 tid 100108 td 0x18d3fe00 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1a5e62c,1a5e644,3ae03f14,103d33c,1a5e644,...) at db_trace_self_wrapper+0x2a/frame 0x3ae03ed0 kdb_backtrace(3ae03ff8,13d05e1,1430c59,0,3ae04030,...) at kdb_backtrace+0x2e/frame 0x3ae03f30 kdb_reenter(1430c59,0,3ae04030,3ae0400c,28409000,...) at kdb_reenter+0x29/frame 0x3ae03f38 trap(3ae04004,8,28,28,ffffffff,...) at trap+0xd1/frame 0x3ae03ff8 calltrap() at 0xffc0319f/frame 0x3ae03ff8 --- trap 0xc, eip = 0x13ac85a, esp = 0x3ae04044, ebp = 0x3ae04050 --- db_trace_thread(18d3fe00,ffffffff) at db_trace_thread+0x2a/frame 0x3ae04050 _db_stack_trace_all(1028685,0,ffffffff,3ae040dc) at _db_stack_trace_all+0x93/frame 0x3ae040b0 db_command(1ae7672) at db_command+0x26a/frame 0x3ae04164 db_script_exec(bd2daf,0,3ae0420c,3ae041f0) at db_script_exec+0x1a6/frame 0x3ae041c4 db_command(3ae042c8,a06536,0,0,2af64700,...) at db_command+0x26a/frame 0x3ae04278 db_command_loop(0,0,2af64700,3ae04280,3ae042c8,...) at db_command_loop+0x3c/frame 0x3ae04280 db_trap(3,0) at db_trap+0xf6/frame 0x3ae042c8 kdb_trap(3,0,3ae043cc) at kdb_trap+0x109/frame 0x3ae042f4 trap(3ae043cc,8,28,28,100,...) at trap+0x382/frame 0x3ae043c0 calltrap() at 0xffc0319f/frame 0x3ae043c0 --- trap 0x3, eip = 0x1028685, esp = 0x3ae0440c, ebp = 0x3ae04410 --- kdb_enter(14df98a,14df98a) at kdb_enter+0x35/frame 0x3ae04410 vpanic(146f152,3ae04458,3ae04458,3ae04488,f39752,...) at vpanic+0x137/frame 0x3ae04438 panic(146f152,0,18d7ef80,fc50a7,18d7ae00,...) at panic+0x14/frame 0x3ae0444c g_io_deliver(261e8cf0,0) at g_io_deliver+0x322/frame 0x3ae04488 g_disk_done(24029678,24029678) at g_disk_done+0xf0/frame 0x3ae044bc cam_iosched_queue_work(18813700,24029678) at cam_iosched_queue_work+0xcb/frame 0x3ae044d8 adastrategy(24029678,0,114,2c4,b9aa800,...) at adastrategy+0x50/frame 0x3ae044f4 g_disk_start(261e8cf0,18d7ef80,18d7ae00,f38877,b9aa800,...) at g_disk_start+0x4de/frame 0x3ae04550 g_io_request(261e8cf0,18d7ef80,33901fec,2b3b598,3ae045f8,...) at g_io_request+0x2aa/frame 0x3ae04578 g_part_start(339019b4,18d88dc0,9ac4900,f38877,b9aa800,...) at g_part_start+0x26f/frame 0x3ae045e8 g_io_request(339019b4,18d88dc0,1d8fb8c,1,41000000,...) at g_io_request+0x2aa/frame 0x3ae04610 g_part_start(2d39fbdc,1,18d8ed00,18d94b80,30ebd600,...) at g_part_start+0x26f/frame 0x3ae04680 g_io_request(2d39fbdc,b8e0400,b8e0400) at g_io_request+0x2aa/frame 0x3ae046a8 g_vfs_strategy(20051828,30ebd600) at g_vfs_strategy+0xe8/frame 0x3ae046c8 softdep_request_cleanup(20165000,2a343758,29a3e000,4) at softdep_request_cleanup+0x2b7/frame 0x3ae04720 ffs_alloc(2aafb4fc,0,0,6a4908,0,1000,2050000,29a3e000,3ae047d8) at ffs_alloc+0x2ad/frame 0x3ae0476c ffs_balloc_ufs2(2a343758,0,0,1000,29a3e000,2050000,3ae04900) at ffs_balloc_ufs2+0xd8b/frame 0x3ae048a0 ffs_write(3ae04a08) at ffs_write+0x216/frame 0x3ae04924 VOP_WRITE_APV(1aabe80,3ae04a08) at VOP_WRITE_APV+0xdf/frame 0x3ae049e0 vn_write(2a29ec30,3ae04bd0,2881e500,1,2af64700) at vn_write+0x305/frame 0x3ae04a3c vn_io_fault_doio(2af64700) at vn_io_fault_doio+0x3a/frame 0x3ae04a84 vn_io_fault1(2af64700) at vn_io_fault1+0x164/frame 0x3ae04b34 vn_io_fault(2a29ec30,3ae04bd0,2881e500,0,2af64700) at vn_io_fault+0x1c4/frame 0x3ae04b70 dofilewrite(2a29ec30,3ae04bd0,ffffffff,ffffffff,0) at dofilewrite+0x71/frame 0x3ae04ba4 sys_write(2af64700,2af64998,2af64700,28409000,2af64700,...) at sys_write+0x78/frame 0x3ae04c08 syscall(3ae04ce8,3b,3b,3b,1000,...) at syscall+0x1b8/frame 0x3ae04cdc Xint0x80_syscall() at 0xffc033f9/frame 0x3ae04cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- Tracing command csh pid 945 tid 100198 td 0x22def000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1a5e62c,1a5e644,3ae03f14,103d33c,1a5e644,...) at db_trace_self_wrapper+0x2a/frame 0x3ae03ed0 kdb_backtrace(3ae03ff8,13d05e1,1430c59,0,3ae04030,...) at kdb_backtrace+0x2e/frame 0x3ae03f30 kdb_reenter(1430c59,0,3ae04030,3ae0400c,28409000,...) at kdb_reenter+0x29/frame 0x3ae03f38 trap(3ae04004,8,28,28,ffffffff,...) at trap+0xd1/frame 0x3ae03ff8 calltrap() at 0xffc0319f/frame 0x3ae03ff8 --- trap 0xc, eip = 0x13ac85a, esp = 0x3ae04044, ebp = 0x3ae04050 --- db_trace_thread(22def000,ffffffff) at db_trace_thread+0x2a/frame 0x3ae04050 _db_stack_trace_all(1028685,0,ffffffff,3ae040dc) at _db_stack_trace_all+0x93/frame 0x3ae040b0 db_command(1ae7672) at db_command+0x26a/frame 0x3ae04164 db_script_exec(bd2daf,0,3ae0420c,3ae041f0) at db_script_exec+0x1a6/frame 0x3ae041c4 db_command(3ae042c8,a06536,0,0,2af64700,...) at db_command+0x26a/frame 0x3ae04278 db_command_loop(0,0,2af64700,3ae04280,3ae042c8,...) at db_command_loop+0x3c/frame 0x3ae04280 db_trap(3,0) at db_trap+0xf6/frame 0x3ae042c8 kdb_trap(3,0,3ae043cc) at kdb_trap+0x109/frame 0x3ae042f4 trap(3ae043cc,8,28,28,100,...) at trap+0x382/frame 0x3ae043c0 calltrap() at 0xffc0319f/frame 0x3ae043c0 --- trap 0x3, eip = 0x1028685, esp = 0x3ae0440c, ebp = 0x3ae04410 --- kdb_enter(14df98a,14df98a) at kdb_enter+0x35/frame 0x3ae04410 vpanic(146f152,3ae04458,3ae04458,3ae04488,f39752,...) at vpanic+0x137/frame 0x3ae04438 panic(146f152,0,18d7ef80,fc50a7,18d7ae00,...) at panic+0x14/frame 0x3ae0444c g_io_deliver(261e8cf0,0) at g_io_deliver+0x322/frame 0x3ae04488 g_disk_done(24029678,24029678) at g_disk_done+0xf0/frame 0x3ae044bc cam_iosched_queue_work(18813700,24029678) at cam_iosched_queue_work+0xcb/frame 0x3ae044d8 adastrategy(24029678,0,114,2c4,b9aa800,...) at adastrategy+0x50/frame 0x3ae044f4 g_disk_start(261e8cf0,18d7ef80,18d7ae00,f38877,b9aa800,...) at g_disk_start+0x4de/frame 0x3ae04550 g_io_request(261e8cf0,18d7ef80,33901fec,2b3b598,3ae045f8,...) at g_io_request+0x2aa/frame 0x3ae04578 g_part_start(339019b4,18d88dc0,9ac4900,f38877,b9aa800,...) at g_part_start+0x26f/frame 0x3ae045e8 g_io_request(339019b4,18d88dc0,1d8fb8c,1,41000000,...) at g_io_request+0x2aa/frame 0x3ae04610 g_part_start(2d39fbdc,1,18d8ed00,18d94b80,30ebd600,...) at g_part_start+0x26f/frame 0x3ae04680 g_io_request(2d39fbdc,b8e0400,b8e0400) at g_io_request+0x2aa/frame 0x3ae046a8 g_vfs_strategy(20051828,30ebd600) at g_vfs_strategy+0xe8/frame 0x3ae046c8 softdep_request_cleanup(20165000,2a343758,29a3e000,4) at softdep_request_cleanup+0x2b7/frame 0x3ae04720 ffs_alloc(2aafb4fc,0,0,6a4908,0,1000,2050000,29a3e000,3ae047d8) at ffs_alloc+0x2ad/frame 0x3ae0476c ffs_balloc_ufs2(2a343758,0,0,1000,29a3e000,2050000,3ae04900) at ffs_balloc_ufs2+0xd8b/frame 0x3ae048a0 ffs_write(3ae04a08) at ffs_write+0x216/frame 0x3ae04924 VOP_WRITE_APV(1aabe80,3ae04a08) at VOP_WRITE_APV+0xdf/frame 0x3ae049e0 vn_write(2a29ec30,3ae04bd0,2881e500,1,2af64700) at vn_write+0x305/frame 0x3ae04a3c vn_io_fault_doio(2af64700) at vn_io_fault_doio+0x3a/frame 0x3ae04a84 vn_io_fault1(2af64700) at vn_io_fault1+0x164/frame 0x3ae04b34 vn_io_fault(2a29ec30,3ae04bd0,2881e500,0,2af64700) at vn_io_fault+0x1c4/frame 0x3ae04b70 dofilewrite(2a29ec30,3ae04bd0,ffffffff,ffffffff,0) at dofilewrite+0x71/frame 0x3ae04ba4 sys_write(2af64700,2af64998,2af64700,28409000,2af64700,...) at sys_write+0x78/frame 0x3ae04c08 syscall(3ae04ce8,3b,3b,3b,1000,...) at syscall+0x1b8/frame 0x3ae04cdc Xint0x80_syscall() at 0xffc033f9/frame 0x3ae04cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- Tracing command su pid 944 tid 100124 td 0x202f5500 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1a5e62c,1a5e644,3ae03f14,103d33c,1a5e644,...) at db_trace_self_wrapper+0x2a/frame 0x3ae03ed0 kdb_backtrace(3ae03ff8,13d05e1,1430c59,0,3ae04030,...) at kdb_backtrace+0x2e/frame 0x3ae03f30 kdb_reenter(1430c59,0,3ae04030,3ae0400c,28409000,...) at kdb_reenter+0x29/frame 0x3ae03f38 trap(3ae04004,8,28,28,ffffffff,...) at trap+0xd1/frame 0x3ae03ff8 calltrap() at 0xffc0319f/frame 0x3ae03ff8 --- trap 0xc, eip = 0x13ac85a, esp = 0x3ae04044, ebp = 0x3ae04050 --- db_trace_thread(202f5500,ffffffff) at db_trace_thread+0x2a/frame 0x3ae04050 _db_stack_trace_all(1028685,0,ffffffff,3ae040dc) at _db_stack_trace_all+0x93/frame 0x3ae040b0 db_command(1ae7672) at db_command+0x26a/frame 0x3ae04164 db_script_exec(bd2daf,0,3ae0420c,3ae041f0) at db_script_exec+0x1a6/frame 0x3ae041c4 db_command(3ae042c8,a06536,0,0,2af64700,...) at db_command+0x26a/frame 0x3ae04278 db_command_loop(0,0,2af64700,3ae04280,3ae042c8,...) at db_command_loop+0x3c/frame 0x3ae04280 db_trap(3,0) at db_trap+0xf6/frame 0x3ae042c8 kdb_trap(3,0,3ae043cc) at kdb_trap+0x109/frame 0x3ae042f4 trap(3ae043cc,8,28,28,100,...) at trap+0x382/frame 0x3ae043c0 calltrap() at 0xffc0319f/frame 0x3ae043c0 --- trap 0x3, eip = 0x1028685, esp = 0x3ae0440c, ebp = 0x3ae04410 --- kdb_enter(14df98a,14df98a) at kdb_enter+0x35/frame 0x3ae04410 vpanic(146f152,3ae04458,3ae04458,3ae04488,f39752,...) at vpanic+0x137/frame 0x3ae04438 panic(146f152,0,18d7ef80,fc50a7,18d7ae00,...) at panic+0x14/frame 0x3ae0444c g_io_deliver(261e8cf0,0) at g_io_deliver+0x322/frame 0x3ae04488 g_disk_done(24029678,24029678) at g_disk_done+0xf0/frame 0x3ae044bc cam_iosched_queue_work(18813700,24029678) at cam_iosched_queue_work+0xcb/frame 0x3ae044d8 adastrategy(24029678,0,114,2c4,b9aa800,...) at adastrategy+0x50/frame 0x3ae044f4 g_disk_start(261e8cf0,18d7ef80,18d7ae00,f38877,b9aa800,...) at g_disk_start+0x4de/frame 0x3ae04550 g_io_request(261e8cf0,18d7ef80,33901fec,2b3b598,3ae045f8,...) at g_io_request+0x2aa/frame 0x3ae04578 g_part_start(339019b4,18d88dc0,9ac4900,f38877,b9aa800,...) at g_part_start+0x26f/frame 0x3ae045e8 g_io_request(339019b4,18d88dc0,1d8fb8c,1,41000000,...) at g_io_request+0x2aa/frame 0x3ae04610 g_part_start(2d39fbdc,1,18d8ed00,18d94b80,30ebd600,...) at g_part_start+0x26f/frame 0x3ae04680 g_io_request(2d39fbdc,b8e0400,b8e0400) at g_io_request+0x2aa/frame 0x3ae046a8 g_vfs_strategy(20051828,30ebd600) at g_vfs_strategy+0xe8/frame 0x3ae046c8 softdep_request_cleanup(20165000,2a343758,29a3e000,4) at softdep_request_cleanup+0x2b7/frame 0x3ae04720 ffs_alloc(2aafb4fc,0,0,6a4908,0,1000,2050000,29a3e000,3ae047d8) at ffs_alloc+0x2ad/frame 0x3ae0476c ffs_balloc_ufs2(2a343758,0,0,1000,29a3e000,2050000,3ae04900) at ffs_balloc_ufs2+0xd8b/frame 0x3ae048a0 ffs_write(3ae04a08) at ffs_write+0x216/frame 0x3ae04924 VOP_WRITE_APV(1aabe80,3ae04a08) at VOP_WRITE_APV+0xdf/frame 0x3ae049e0 vn_write(2a29ec30,3ae04bd0,2881e500,1,2af64700) at vn_write+0x305/frame 0x3ae04a3c vn_io_fault_doio(2af64700) at vn_io_fault_doio+0x3a/frame 0x3ae04a84 vn_io_fault1(2af64700) at vn_io_fault1+0x164/frame 0x3ae04b34 vn_io_fault(2a29ec30,3ae04bd0,2881e500,0,2af64700) at vn_io_fault+0x1c4/frame 0x3ae04b70 dofilewrite(2a29ec30,3ae04bd0,ffffffff,ffffffff,0) at dofilewrite+0x71/frame 0x3ae04ba4 sys_write(2af64700,2af64998,2af64700,28409000,2af64700,...) at sys_write+0x78/frame 0x3ae04c08 syscall(3ae04ce8,3b,3b,3b,1000,...) at syscall+0x1b8/frame 0x3ae04cdc Xint0x80_syscall() at 0xffc033f9/frame 0x3ae04cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- Tracing command bash pid 923 tid 100202 td 0x24425180 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1a5e62c,1a5e644,3ae03f14,103d33c,1a5e644,...) at db_trace_self_wrapper+0x2a/frame 0x3ae03ed0 kdb_backtrace(3ae03ff8,13d05e1,1430c59,0,3ae04030,...) at kdb_backtrace+0x2e/frame 0x3ae03f30 kdb_reenter(1430c59,0,3ae04030,3ae0400c,28409000,...) at kdb_reenter+0x29/frame 0x3ae03f38 trap(3ae04004,8,28,28,ffffffff,...) at trap+0xd1/frame 0x3ae03ff8 calltrap() at 0xffc0319f/frame 0x3ae03ff8 --- trap 0xc, eip = 0x13ac85a, esp = 0x3ae04044, ebp = 0x3ae04050 --- db_trace_thread(24425180,ffffffff) at db_trace_thread+0x2a/frame 0x3ae04050 _db_stack_trace_all(1028685,0,ffffffff,3ae040dc) at _db_stack_trace_all+0x93/frame 0x3ae040b0 db_command(1ae7672) at db_command+0x26a/frame 0x3ae04164 db_script_exec(bd2daf,0,3ae0420c,3ae041f0) at db_script_exec+0x1a6/frame 0x3ae041c4 db_command(3ae042c8,a06536,0,0,2af64700,...) at db_command+0x26a/frame 0x3ae04278 db_command_loop(0,0,2af64700,3ae04280,3ae042c8,...) at db_command_loop+0x3c/frame 0x3ae04280 db_trap(3,0) at db_trap+0xf6/frame 0x3ae042c8 kdb_trap(3,0,3ae043cc) at kdb_trap+0x109/frame 0x3ae042f4 trap(3ae043cc,8,28,28,100,...) at trap+0x382/frame 0x3ae043c0 calltrap() at 0xffc0319f/frame 0x3ae043c0 --- trap 0x3, eip = 0x1028685, esp = 0x3ae0440c, ebp = 0x3ae04410 --- kdb_enter(14df98a,14df98a) at kdb_enter+0x35/frame 0x3ae04410 vpanic(146f152,3ae04458,3ae04458,3ae04488,f39752,...) at vpanic+0x137/frame 0x3ae04438 panic(146f152,0,18d7ef80,fc50a7,18d7ae00,...) at panic+0x14/frame 0x3ae0444c g_io_deliver(261e8cf0,0) at g_io_deliver+0x322/frame 0x3ae04488 g_disk_done(24029678,24029678) at g_disk_done+0xf0/frame 0x3ae044bc cam_iosched_queue_work(18813700,24029678) at cam_iosched_queue_work+0xcb/frame 0x3ae044d8 adastrategy(24029678,0,114,2c4,b9aa800,...) at adastrategy+0x50/frame 0x3ae044f4 g_disk_start(261e8cf0,18d7ef80,18d7ae00,f38877,b9aa800,...) at g_disk_start+0x4de/frame 0x3ae04550 g_io_request(261e8cf0,18d7ef80,33901fec,2b3b598,3ae045f8,...) at g_io_request+0x2aa/frame 0x3ae04578 g_part_start(339019b4,18d88dc0,9ac4900,f38877,b9aa800,...) at g_part_start+0x26f/frame 0x3ae045e8 g_io_request(339019b4,18d88dc0,1d8fb8c,1,41000000,...) at g_io_request+0x2aa/frame 0x3ae04610 g_part_start(2d39fbdc,1,18d8ed00,18d94b80,30ebd600,...) at g_part_start+0x26f/frame 0x3ae04680 g_io_request(2d39fbdc,b8e0400,b8e0400) at g_io_request+0x2aa/frame 0x3ae046a8 g_vfs_strategy(20051828,30ebd600) at g_vfs_strategy+0xe8/frame 0x3ae046c8 softdep_request_cleanup(20165000,2a343758,29a3e000,4) at softdep_request_cleanup+0x2b7/frame 0x3ae04720 ffs_alloc(2aafb4fc,0,0,6a4908,0,1000,2050000,29a3e000,3ae047d8) at ffs_alloc+0x2ad/frame 0x3ae0476c ffs_balloc_ufs2(2a343758,0,0,1000,29a3e000,2050000,3ae04900) at ffs_balloc_ufs2+0xd8b/frame 0x3ae048a0 ffs_write(3ae04a08) at ffs_write+0x216/frame 0x3ae04924 VOP_WRITE_APV(1aabe80,3ae04a08) at VOP_WRITE_APV+0xdf/frame 0x3ae049e0 vn_write(2a29ec30,3ae04bd0,2881e500,1,2af64700) at vn_write+0x305/frame 0x3ae04a3c vn_io_fault_doio(2af64700) at vn_io_fault_doio+0x3a/frame 0x3ae04a84 vn_io_fault1(2af64700) at vn_io_fault1+0x164/frame 0x3ae04b34 vn_io_fault(2a29ec30,3ae04bd0,2881e500,0,2af64700) at vn_io_fault+0x1c4/frame 0x3ae04b70 dofilewrite(2a29ec30,3ae04bd0,ffffffff,ffffffff,0) at dofilewrite+0x71/frame 0x3ae04ba4 sys_write(2af64700,2af64998,2af64700,28409000,2af64700,...) at sys_write+0x78/frame 0x3ae04c08 syscall(3ae04ce8,3b,3b,3b,1000,...) at syscall+0x1b8/frame 0x3ae04cdc Xint0x80_syscall() at 0xffc033f9/frame 0x3ae04cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- Tracing command sshd pid 922 tid 100173 td 0x22df0880 sched_switch(22df0880,104) at sched_switch+0x4e9/frame 0x20726970 mi_switch(104) at mi_switch+0x1c4/frame 0x207269a8 sleepq_switch(1300,22df0880,0,0,1d724d4,...) at sleepq_switch+0xf7/frame 0x207269c8 sleepq_catch_signals(20726a30,f845b8,18826824,0,13fc2f9,...) at sleepq_catch_signals+0x278/frame 0x207269f0 sleepq_wait_sig(18826824,0) at sleepq_wait_sig+0xe/frame 0x207269f8 _cv_wait_sig(18826824,18826810) at _cv_wait_sig+0x138/frame 0x20726a30 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x20726a4c kern_select(22df0880,10,2047c270,2047c278,0,0,20) at kern_select+0x8e5/frame 0x20726bd4 sys_select(22df0880,22df0b18,22df0880,20426344,1388b7e,...) at sys_select+0x41/frame 0x20726c08 syscall(20726ce8,3b,3b,3b,20482000,...) at syscall+0x1b8/frame 0x20726cdc Xint0x80_syscall() at 0xffc033f9/frame 0x20726cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- Tracing command sshd pid 920 tid 100179 td 0x22c24e00 sched_switch(22c24e00,104) at sched_switch+0x4e9/frame 0x207449a0 mi_switch(104) at mi_switch+0x1c4/frame 0x207449d8 sleepq_switch(340,22c24e00,0,0,1d71514,...) at sleepq_switch+0xf7/frame 0x207449f8 sleepq_catch_signals(20744a60,f845b8,18826964,0,13fc2f9,...) at sleepq_catch_signals+0x278/frame 0x20744a20 sleepq_wait_sig(18826964,0) at sleepq_wait_sig+0xe/frame 0x20744a28 _cv_wait_sig(18826964,18826950) at _cv_wait_sig+0x138/frame 0x20744a60 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x20744a7c kern_poll(22c24e00,ffbfe7a0,1,0,0) at kern_poll+0x4a4/frame 0x20744bd4 sys_poll(22c24e00,22c25098,22c24e00,22c21d10,22c24e00,...) at sys_poll+0x50/frame 0x20744c08 syscall(20744ce8,3b,3b,3b,20482000,...) at syscall+0x1b8/frame 0x20744cdc Xint0x80_syscall() at 0xffc033f9/frame 0x20744cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d98, esp = 0xffc0bfe8, ebp = 0xffbfecbc --- Tracing command bash pid 909 tid 100199 td 0x24425c00 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1a5e62c,1a5e644,3ae03f14,103d33c,1a5e644,...) at db_trace_self_wrapper+0x2a/frame 0x3ae03ed0 kdb_backtrace(3ae03ff8,13d05e1,1430c59,0,3ae04030,...) at kdb_backtrace+0x2e/frame 0x3ae03f30 kdb_reenter(1430c59,0,3ae04030,3ae0400c,28409000,...) at kdb_reenter+0x29/frame 0x3ae03f38 trap(3ae04004,8,28,28,ffffffff,...) at trap+0xd1/frame 0x3ae03ff8 calltrap() at 0xffc0319f/frame 0x3ae03ff8 --- trap 0xc, eip = 0x13ac85a, esp = 0x3ae04044, ebp = 0x3ae04050 --- db_trace_thread(24425c00,ffffffff) at db_trace_thread+0x2a/frame 0x3ae04050 _db_stack_trace_all(1028685,0,ffffffff,3ae040dc) at _db_stack_trace_all+0x93/frame 0x3ae040b0 db_command(1ae7672) at db_command+0x26a/frame 0x3ae04164 db_script_exec(bd2daf,0,3ae0420c,3ae041f0) at db_script_exec+0x1a6/frame 0x3ae041c4 db_command(3ae042c8,a06536,0,0,2af64700,...) at db_command+0x26a/frame 0x3ae04278 db_command_loop(0,0,2af64700,3ae04280,3ae042c8,...) at db_command_loop+0x3c/frame 0x3ae04280 db_trap(3,0) at db_trap+0xf6/frame 0x3ae042c8 kdb_trap(3,0,3ae043cc) at kdb_trap+0x109/frame 0x3ae042f4 trap(3ae043cc,8,28,28,100,...) at trap+0x382/frame 0x3ae043c0 calltrap() at 0xffc0319f/frame 0x3ae043c0 --- trap 0x3, eip = 0x1028685, esp = 0x3ae0440c, ebp = 0x3ae04410 --- kdb_enter(14df98a,14df98a) at kdb_enter+0x35/frame 0x3ae04410 vpanic(146f152,3ae04458,3ae04458,3ae04488,f39752,...) at vpanic+0x137/frame 0x3ae04438 panic(146f152,0,18d7ef80,fc50a7,18d7ae00,...) at panic+0x14/frame 0x3ae0444c g_io_deliver(261e8cf0,0) at g_io_deliver+0x322/frame 0x3ae04488 g_disk_done(24029678,24029678) at g_disk_done+0xf0/frame 0x3ae044bc cam_iosched_queue_work(18813700,24029678) at cam_iosched_queue_work+0xcb/frame 0x3ae044d8 adastrategy(24029678,0,114,2c4,b9aa800,...) at adastrategy+0x50/frame 0x3ae044f4 g_disk_start(261e8cf0,18d7ef80,18d7ae00,f38877,b9aa800,...) at g_disk_start+0x4de/frame 0x3ae04550 g_io_request(261e8cf0,18d7ef80,33901fec,2b3b598,3ae045f8,...) at g_io_request+0x2aa/frame 0x3ae04578 g_part_start(339019b4,18d88dc0,9ac4900,f38877,b9aa800,...) at g_part_start+0x26f/frame 0x3ae045e8 g_io_request(339019b4,18d88dc0,1d8fb8c,1,41000000,...) at g_io_request+0x2aa/frame 0x3ae04610 g_part_start(2d39fbdc,1,18d8ed00,18d94b80,30ebd600,...) at g_part_start+0x26f/frame 0x3ae04680 g_io_request(2d39fbdc,b8e0400,b8e0400) at g_io_request+0x2aa/frame 0x3ae046a8 g_vfs_strategy(20051828,30ebd600) at g_vfs_strategy+0xe8/frame 0x3ae046c8 softdep_request_cleanup(20165000,2a343758,29a3e000,4) at softdep_request_cleanup+0x2b7/frame 0x3ae04720 ffs_alloc(2aafb4fc,0,0,6a4908,0,1000,2050000,29a3e000,3ae047d8) at ffs_alloc+0x2ad/frame 0x3ae0476c ffs_balloc_ufs2(2a343758,0,0,1000,29a3e000,2050000,3ae04900) at ffs_balloc_ufs2+0xd8b/frame 0x3ae048a0 ffs_write(3ae04a08) at ffs_write+0x216/frame 0x3ae04924 VOP_WRITE_APV(1aabe80,3ae04a08) at VOP_WRITE_APV+0xdf/frame 0x3ae049e0 vn_write(2a29ec30,3ae04bd0,2881e500,1,2af64700) at vn_write+0x305/frame 0x3ae04a3c vn_io_fault_doio(2af64700) at vn_io_fault_doio+0x3a/frame 0x3ae04a84 vn_io_fault1(2af64700) at vn_io_fault1+0x164/frame 0x3ae04b34 vn_io_fault(2a29ec30,3ae04bd0,2881e500,0,2af64700) at vn_io_fault+0x1c4/frame 0x3ae04b70 dofilewrite(2a29ec30,3ae04bd0,ffffffff,ffffffff,0) at dofilewrite+0x71/frame 0x3ae04ba4 sys_write(2af64700,2af64998,2af64700,28409000,2af64700,...) at sys_write+0x78/frame 0x3ae04c08 syscall(3ae04ce8,3b,3b,3b,1000,...) at syscall+0x1b8/frame 0x3ae04cdc Xint0x80_syscall() at 0xffc033f9/frame 0x3ae04cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- Tracing command sshd pid 908 tid 100197 td 0x22def380 sched_switch(22def380,104) at sched_switch+0x4e9/frame 0x207b2970 mi_switch(104) at mi_switch+0x1c4/frame 0x207b29a8 sleepq_switch(3780,22def380,0,0,1d74954,...) at sleepq_switch+0xf7/frame 0x207b29c8 sleepq_catch_signals(207b2a30,f845b8,b9d3ae4,0,13fc2f9,...) at sleepq_catch_signals+0x278/frame 0x207b29f0 sleepq_wait_sig(b9d3ae4,0) at sleepq_wait_sig+0xe/frame 0x207b29f8 _cv_wait_sig(b9d3ae4,b9d3ad0) at _cv_wait_sig+0x138/frame 0x207b2a30 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x207b2a4c kern_select(22def380,e,2047c270,2047c278,0,0,20) at kern_select+0x8e5/frame 0x207b2bd4 sys_select(22def380,22def618,22def380,22e67d10,22ff5b8,...) at sys_select+0x41/frame 0x207b2c08 syscall(207b2ce8,3b,3b,3b,20482000,...) at syscall+0x1b8/frame 0x207b2cdc Xint0x80_syscall() at 0xffc033f9/frame 0x207b2cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe75c --- Tracing command sshd pid 906 tid 100175 td 0x22df0180 sched_switch(22df0180,104) at sched_switch+0x4e9/frame 0x207309a0 mi_switch(104) at mi_switch+0x1c4/frame 0x207309d8 sleepq_switch(2640,22df0180,0,0,1d73814,...) at sleepq_switch+0xf7/frame 0x207309f8 sleepq_catch_signals(20730a60,f845b8,1fc0fd64,0,13fc2f9,...) at sleepq_catch_signals+0x278/frame 0x20730a20 sleepq_wait_sig(1fc0fd64,0) at sleepq_wait_sig+0xe/frame 0x20730a28 _cv_wait_sig(1fc0fd64,1fc0fd50) at _cv_wait_sig+0x138/frame 0x20730a60 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x20730a7c kern_poll(22df0180,ffbfe7a0,1,0,0) at kern_poll+0x4a4/frame 0x20730bd4 sys_poll(22df0180,22df0418,22df0180,22e68a20,22df0180,...) at sys_poll+0x50/frame 0x20730c08 syscall(20730ce8,3b,3b,3b,20482000,...) at syscall+0x1b8/frame 0x20730cdc Xint0x80_syscall() at 0xffc033f9/frame 0x20730cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d98, esp = 0xffc0bfe8, ebp = 0xffbfecbc --- Tracing command getty pid 790 tid 100169 td 0x22c25180 sched_switch(22c25180,104) at sched_switch+0x4e9/frame 0x20712a20 mi_switch(104) at mi_switch+0x1c4/frame 0x20712a54 sleepq_switch(2e00,22c25180,0,0,1d73fd4,...) at sleepq_switch+0xf7/frame 0x20712a74 sleepq_catch_signals(20712adc,f845b8,1833cc74,0,150bfe3,...) at sleepq_catch_signals+0x278/frame 0x20712a9c sleepq_wait_sig(1833cc74,0) at sleepq_wait_sig+0xe/frame 0x20712aa4 _cv_wait_sig(1833cc74,1833cc04) at _cv_wait_sig+0x138/frame 0x20712adc tty_wait(1833cc00,1833cc74) at tty_wait+0x35/frame 0x20712af4 ttydisc_read(1833cc00,20712bd0,0) at ttydisc_read+0x1c6/frame 0x20712b28 ttydev_read(182c7000,20712bd0,0) at ttydev_read+0x3d/frame 0x20712b44 devfs_read_f(1fc50708,20712bd0,22ea5e00,0,22c25180) at devfs_read_f+0xaa/frame 0x20712b74 dofileread(1fc50708,20712bd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x20712ba4 sys_read(22c25180,22c25418,22c25180,22c22054,22c25180,...) at sys_read+0x78/frame 0x20712c08 syscall(20712ce8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x20712cdc Xint0x80_syscall() at 0xffc033f9/frame 0x20712cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe75c --- Tracing command getty pid 789 tid 100196 td 0x2029d000 sched_switch(2029d000,104) at sched_switch+0x4e9/frame 0x207a1a20 mi_switch(104) at mi_switch+0x1c4/frame 0x207a1a54 sleepq_switch(2700,2029d000,0,0,1d738d4,...) at sleepq_switch+0xf7/frame 0x207a1a74 sleepq_catch_signals(207a1adc,f845b8,18c1e874,0,150bfe3,...) at sleepq_catch_signals+0x278/frame 0x207a1a9c sleepq_wait_sig(18c1e874,0) at sleepq_wait_sig+0xe/frame 0x207a1aa4 _cv_wait_sig(18c1e874,18c1e804) at _cv_wait_sig+0x138/frame 0x207a1adc tty_wait(18c1e800,18c1e874) at tty_wait+0x35/frame 0x207a1af4 ttydisc_read(18c1e800,207a1bd0,0) at ttydisc_read+0x1c6/frame 0x207a1b28 ttydev_read(18c1fc00,207a1bd0,0) at ttydev_read+0x3d/frame 0x207a1b44 devfs_read_f(1fc50744,207a1bd0,1891b100,0,2029d000) at devfs_read_f+0xaa/frame 0x207a1b74 dofileread(1fc50744,207a1bd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x207a1ba4 sys_read(2029d000,2029d298,2029d000,2029b9cc,2029d000,...) at sys_read+0x78/frame 0x207a1c08 syscall(207a1ce8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x207a1cdc Xint0x80_syscall() at 0xffc033f9/frame 0x207a1cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- Tracing command getty pid 788 tid 100195 td 0x2029d380 sched_switch(2029d380,104) at sched_switch+0x4e9/frame 0x2079ca20 mi_switch(104) at mi_switch+0x1c4/frame 0x2079ca54 sleepq_switch(2600,2029d380,0,0,1d737d4,...) at sleepq_switch+0xf7/frame 0x2079ca74 sleepq_catch_signals(2079cadc,f845b8,18c1ec74,0,150bfe3,...) at sleepq_catch_signals+0x278/frame 0x2079ca9c sleepq_wait_sig(18c1ec74,0) at sleepq_wait_sig+0xe/frame 0x2079caa4 _cv_wait_sig(18c1ec74,18c1ec04) at _cv_wait_sig+0x138/frame 0x2079cadc tty_wait(18c1ec00,18c1ec74) at tty_wait+0x35/frame 0x2079caf4 ttydisc_read(18c1ec00,2079cbd0,0) at ttydisc_read+0x1c6/frame 0x2079cb28 ttydev_read(18c1fe00,2079cbd0,0) at ttydev_read+0x3d/frame 0x2079cb44 devfs_read_f(1fc33438,2079cbd0,22ea5400,0,2029d380) at devfs_read_f+0xaa/frame 0x2079cb74 dofileread(1fc33438,2079cbd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x2079cba4 sys_read(2029d380,2029d618,2029d380,2029bd10,b85c5b8,...) at sys_read+0x78/frame 0x2079cc08 syscall(2079cce8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x2079ccdc Xint0x80_syscall() at 0xffc033f9/frame 0x2079ccdc --- syscall (560, FreeBSD ELF32, sys_kevent), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe878 --- Tracing command getty pid 787 tid 100194 td 0x2029d700 sched_switch(2029d700,104) at sched_switch+0x4e9/frame 0x20797a20 mi_switch(104) at mi_switch+0x1c4/frame 0x20797a54 sleepq_switch(3500,2029d700,0,0,1d746d4,...) at sleepq_switch+0xf7/frame 0x20797a74 sleepq_catch_signals(20797adc,f845b8,18b7a074,0,150bfe3,...) at sleepq_catch_signals+0x278/frame 0x20797a9c sleepq_wait_sig(18b7a074,0) at sleepq_wait_sig+0xe/frame 0x20797aa4 _cv_wait_sig(18b7a074,18b7a004) at _cv_wait_sig+0x138/frame 0x20797adc tty_wait(18b7a000,18b7a074) at tty_wait+0x35/frame 0x20797af4 ttydisc_read(18b7a000,20797bd0,0) at ttydisc_read+0x1c6/frame 0x20797b28 ttydev_read(18a46000,20797bd0,0) at ttydev_read+0x3d/frame 0x20797b44 devfs_read_f(1fc50960,20797bd0,22ebc400,0,2029d700) at devfs_read_f+0xaa/frame 0x20797b74 dofileread(1fc50960,20797bd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x20797ba4 sys_read(2029d700,2029d998,2029d700,2029c054,b8505b8,...) at sys_read+0x78/frame 0x20797c08 syscall(20797ce8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x20797cdc Xint0x80_syscall() at 0xffc033f9/frame 0x20797cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe75c --- Tracing command getty pid 786 tid 100193 td 0x2029da80 sched_switch(2029da80,104) at sched_switch+0x4e9/frame 0x20792a20 mi_switch(104) at mi_switch+0x1c4/frame 0x20792a54 sleepq_switch(3400,2029da80,0,0,1d745d4,...) at sleepq_switch+0xf7/frame 0x20792a74 sleepq_catch_signals(20792adc,f845b8,18b7a474,0,150bfe3,...) at sleepq_catch_signals+0x278/frame 0x20792a9c sleepq_wait_sig(18b7a474,0) at sleepq_wait_sig+0xe/frame 0x20792aa4 _cv_wait_sig(18b7a474,18b7a404) at _cv_wait_sig+0x138/frame 0x20792adc tty_wait(18b7a400,18b7a474) at tty_wait+0x35/frame 0x20792af4 ttydisc_read(18b7a400,20792bd0,0) at ttydisc_read+0x1c6/frame 0x20792b28 ttydev_read(18a46200,20792bd0,0) at ttydev_read+0x3d/frame 0x20792b44 devfs_read_f(1fc50bf4,20792bd0,22ea5600,0,2029da80) at devfs_read_f+0xaa/frame 0x20792b74 dofileread(1fc50bf4,20792bd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x20792ba4 sys_read(2029da80,2029dd18,2029da80,2029c398,22ff5b8,...) at sys_read+0x78/frame 0x20792c08 syscall(20792ce8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x20792cdc Xint0x80_syscall() at 0xffc033f9/frame 0x20792cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- Tracing command getty pid 785 tid 100192 td 0x2029de00 sched_switch(2029de00,104) at sched_switch+0x4e9/frame 0x2078da20 mi_switch(104) at mi_switch+0x1c4/frame 0x2078da54 sleepq_switch(3700,2029de00,0,0,1d748d4,...) at sleepq_switch+0xf7/frame 0x2078da74 sleepq_catch_signals(2078dadc,f845b8,18b7a874,0,150bfe3,...) at sleepq_catch_signals+0x278/frame 0x2078da9c sleepq_wait_sig(18b7a874,0) at sleepq_wait_sig+0xe/frame 0x2078daa4 _cv_wait_sig(18b7a874,18b7a804) at _cv_wait_sig+0x138/frame 0x2078dadc tty_wait(18b7a800,18b7a874) at tty_wait+0x35/frame 0x2078daf4 ttydisc_read(18b7a800,2078dbd0,0) at ttydisc_read+0x1c6/frame 0x2078db28 ttydev_read(18a46400,2078dbd0,0) at ttydev_read+0x3d/frame 0x2078db44 devfs_read_f(1fc508ac,2078dbd0,22ea5500,0,2029de00) at devfs_read_f+0xaa/frame 0x2078db74 dofileread(1fc508ac,2078dbd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x2078dba4 sys_read(2029de00,2029e098,2029de00,2029c6dc,2029de00,...) at sys_read+0x78/frame 0x2078dc08 syscall(2078dce8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x2078dcdc Xint0x80_syscall() at 0xffc033f9/frame 0x2078dcdc --- syscall (560, FreeBSD ELF32, sys_kevent), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe878 --- Tracing command getty pid 784 tid 100190 td 0x22ebe500 sched_switch(22ebe500,104) at sched_switch+0x4e9/frame 0x2077ba20 mi_switch(104) at mi_switch+0x1c4/frame 0x2077ba54 sleepq_switch(3600,22ebe500,0,0,1d747d4,...) at sleepq_switch+0xf7/frame 0x2077ba74 sleepq_catch_signals(2077badc,f845b8,18b7ac74,0,150bfe3,...) at sleepq_catch_signals+0x278/frame 0x2077ba9c sleepq_wait_sig(18b7ac74,0) at sleepq_wait_sig+0xe/frame 0x2077baa4 _cv_wait_sig(18b7ac74,18b7ac04) at _cv_wait_sig+0x138/frame 0x2077badc tty_wait(18b7ac00,18b7ac74) at tty_wait+0x35/frame 0x2077baf4 ttydisc_read(18b7ac00,2077bbd0,0) at ttydisc_read+0x1c6/frame 0x2077bb28 ttydev_read(18a46600,2077bbd0,0) at ttydev_read+0x3d/frame 0x2077bb44 devfs_read_f(1fc5099c,2077bbd0,22c23700,0,22ebe500) at devfs_read_f+0xaa/frame 0x2077bb74 dofileread(1fc5099c,2077bbd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x2077bba4 sys_read(22ebe500,22ebe798,22ebe500,22ebb398,b85c5b8,...) at sys_read+0x78/frame 0x2077bc08 syscall(2077bce8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x2077bcdc Xint0x80_syscall() at 0xffc033f9/frame 0x2077bcdc --- syscall (560, FreeBSD ELF32, sys_kevent), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe878 --- Tracing command getty pid 783 tid 100120 td 0x2029ec00 sched_switch(2029ec00,104) at sched_switch+0x4e9/frame 0x1fb25a20 mi_switch(104) at mi_switch+0x1c4/frame 0x1fb25a54 sleepq_switch(2d00,2029ec00,0,0,1d73ed4,...) at sleepq_switch+0xf7/frame 0x1fb25a74 sleepq_catch_signals(1fb25adc,f845b8,1833c074,0,150bfe3,...) at sleepq_catch_signals+0x278/frame 0x1fb25a9c sleepq_wait_sig(1833c074,0) at sleepq_wait_sig+0xe/frame 0x1fb25aa4 _cv_wait_sig(1833c074,1833c004) at _cv_wait_sig+0x138/frame 0x1fb25adc tty_wait(1833c000,1833c074) at tty_wait+0x35/frame 0x1fb25af4 ttydisc_read(1833c000,1fb25bd0,0) at ttydisc_read+0x1c6/frame 0x1fb25b28 ttydev_read(18a46800,1fb25bd0,0) at ttydev_read+0x3d/frame 0x1fb25b44 devfs_read_f(1fc4b870,1fb25bd0,22ebc300,0,2029ec00) at devfs_read_f+0xaa/frame 0x1fb25b74 dofileread(1fc4b870,1fb25bd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x1fb25ba4 sys_read(2029ec00,2029ee98,2029ec00,2029ca20,2029ec00,...) at sys_read+0x78/frame 0x1fb25c08 syscall(1fb25ce8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x1fb25cdc Xint0x80_syscall() at 0xffc033f9/frame 0x1fb25cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- Tracing command getty pid 782 tid 100106 td 0x18d40180 sched_switch(18d40180,104) at sched_switch+0x4e9/frame 0x1fa9ba20 mi_switch(104) at mi_switch+0x1c4/frame 0x1fa9ba54 sleepq_switch(2c00,18d40180,0,0,1d73dd4,...) at sleepq_switch+0xf7/frame 0x1fa9ba74 sleepq_catch_signals(1fa9badc,f845b8,1833c474,0,150bfe3,...) at sleepq_catch_signals+0x278/frame 0x1fa9ba9c sleepq_wait_sig(1833c474,0) at sleepq_wait_sig+0xe/frame 0x1fa9baa4 _cv_wait_sig(1833c474,1833c404) at _cv_wait_sig+0x138/frame 0x1fa9badc tty_wait(1833c400,1833c474) at tty_wait+0x35/frame 0x1fa9baf4 ttydisc_read(1833c400,1fa9bbd0,0) at ttydisc_read+0x1c6/frame 0x1fa9bb28 ttydev_read(18a46a00,1fa9bbd0,0) at ttydev_read+0x3d/frame 0x1fa9bb44 devfs_read_f(1fc50780,1fa9bbd0,22ebc600,0,18d40180) at devfs_read_f+0xaa/frame 0x1fa9bb74 dofileread(1fc50780,1fa9bbd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x1fa9bba4 sys_read(18d40180,18d40418,18d40180,1fc2c6dc,18d40180,...) at sys_read+0x78/frame 0x1fa9bc08 syscall(1fa9bce8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x1fa9bcdc Xint0x80_syscall() at 0xffc033f9/frame 0x1fa9bcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe75c --- Tracing command inetd pid 762 tid 100178 td 0x22def700 sched_switch(22def700,104) at sched_switch+0x4e9/frame 0x2073f970 mi_switch(104) at mi_switch+0x1c4/frame 0x2073f9a8 sleepq_switch(1340,22def700,0,0,1d72514,...) at sleepq_switch+0xf7/frame 0x2073f9c8 sleepq_catch_signals(2073fa30,f845b8,18826924,0,13fc2f9,...) at sleepq_catch_signals+0x278/frame 0x2073f9f0 sleepq_wait_sig(18826924,0) at sleepq_wait_sig+0xe/frame 0x2073f9f8 _cv_wait_sig(18826924,18826910) at _cv_wait_sig+0x138/frame 0x2073fa30 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x2073fa4c kern_select(22def700,9,ffbfe5d8,0,0,0,20) at kern_select+0x8e5/frame 0x2073fbd4 sys_select(22def700,22def998,22def700,22e68054,22def700,...) at sys_select+0x41/frame 0x2073fc08 syscall(2073fce8,3b,3b,3b,ffbfe658,...) at syscall+0x1b8/frame 0x2073fcdc Xint0x80_syscall() at 0xffc033f9/frame 0x2073fcdc --- syscall (560, FreeBSD ELF32, sys_kevent), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe878 --- Tracing command nfscbd pid 728 tid 100109 td 0x1880f380 sched_switch(1880f380,104) at sched_switch+0x4e9/frame 0x1fac2898 mi_switch(104) at mi_switch+0x1c4/frame 0x1fac28d0 sleepq_switch(640,1d9fe0c,0,0,1d71814,...) at sleepq_switch+0xf7/frame 0x1fac28f0 sleepq_catch_signals(e4,1880f380,1a743c8,1880f380,1fac294c,...) at sleepq_catch_signals+0x278/frame 0x1fac2918 sleepq_timedwait_sig(22c64950,0) at sleepq_timedwait_sig+0x17/frame 0x1fac2948 _cv_timedwait_sig_sbt(22c64950,183600c0,fffffa38,4,0,0,100) at _cv_timedwait_sig_sbt+0x15a/frame 0x1fac2984 svc_run_internal(2029e180,183600c0,183600d0,1836011c,0,...) at svc_run_internal+0xa1c/frame 0x1fac2a68 svc_run(18360000) at svc_run+0x192/frame 0x1fac2a98 nfscbd_nfsd(1880f380,1fac2b50) at nfscbd_nfsd+0xd0/frame 0x1fac2b28 nfssvc_nfscl(1880f380,1880f618) at nfssvc_nfscl+0x233/frame 0x1fac2bf0 sys_nfssvc(1880f380,1880f618,1880f380,1880b688,1880f380,...) at sys_nfssvc+0x92/frame 0x1fac2c08 syscall(1fac2ce8,3b,3b,3b,ffbfee18,...) at syscall+0x1b8/frame 0x1fac2cdc Xint0x80_syscall() at 0xffc033f9/frame 0x1fac2cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d98, esp = 0xffc0bfe8, ebp = 0xffbfecbc --- Tracing command nfscbd pid 728 tid 100182 td 0x2029e880 sched_switch(2029e880,104) at sched_switch+0x4e9/frame 0x20753b20 mi_switch(104) at mi_switch+0x1c4/frame 0x20753b58 sleepq_switch(2c80,2029e880,0,0,1d73e54,...) at sleepq_switch+0xf7/frame 0x20753b78 sleepq_catch_signals(20753be0,f845b8,9bd62d0,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x20753ba0 sleepq_wait_sig(9bd62d0,0) at sleepq_wait_sig+0xe/frame 0x20753ba8 _cv_wait_sig(9bd62d0,183600c0) at _cv_wait_sig+0x138/frame 0x20753be0 svc_run_internal(20753cd4,fa73e8,183600c0,20753ce8,120ed90,...) at svc_run_internal+0xb45/frame 0x20753cb0 svc_thread_start(183600c0,20753ce8) at svc_thread_start+0xd/frame 0x20753cb8 fork_exit(120ed90,183600c0,20753ce8,3b,3b,...) at fork_exit+0x68/frame 0x20753cd4 fork_trampoline() at 0xffc0340e/frame 0x20753cd4 --- kthread start Tracing command nfscbd pid 728 tid 100183 td 0x2029e500 sched_switch(2029e500,104) at sched_switch+0x4e9/frame 0x20758b20 mi_switch(104) at mi_switch+0x1c4/frame 0x20758b58 sleepq_switch(c80,2029e500,0,0,1d71e54,...) at sleepq_switch+0xf7/frame 0x20758b78 sleepq_catch_signals(20758be0,f845b8,9bd6250,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x20758ba0 sleepq_wait_sig(9bd6250,0) at sleepq_wait_sig+0xe/frame 0x20758ba8 _cv_wait_sig(9bd6250,183600c0) at _cv_wait_sig+0x138/frame 0x20758be0 svc_run_internal(20758cd4,fa73e8,183600c0,20758ce8,120ed90,...) at svc_run_internal+0xb45/frame 0x20758cb0 svc_thread_start(183600c0,20758ce8) at svc_thread_start+0xd/frame 0x20758cb8 fork_exit(120ed90,183600c0,20758ce8,3b,3b,...) at fork_exit+0x68/frame 0x20758cd4 fork_trampoline() at 0xffc0340e/frame 0x20758cd4 --- kthread start Tracing command nfscbd pid 728 tid 100184 td 0x2029e180 sched_switch(2029e180,104) at sched_switch+0x4e9/frame 0x2075db20 mi_switch(104) at mi_switch+0x1c4/frame 0x2075db58 sleepq_switch(2600,2029e180,0,0,1d737d4,...) at sleepq_switch+0xf7/frame 0x2075db78 sleepq_catch_signals(2075dbe0,f845b8,22c648d0,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x2075dba0 sleepq_wait_sig(22c648d0,0) at sleepq_wait_sig+0xe/frame 0x2075dba8 _cv_wait_sig(22c648d0,183600c0) at _cv_wait_sig+0x138/frame 0x2075dbe0 svc_run_internal(2075dcd4,fa73e8,183600c0,2075dce8,120ed90,...) at svc_run_internal+0xb45/frame 0x2075dcb0 svc_thread_start(183600c0,2075dce8) at svc_thread_start+0xd/frame 0x2075dcb8 fork_exit(120ed90,183600c0,2075dce8,3b,3b,...) at fork_exit+0x68/frame 0x2075dcd4 fork_trampoline() at 0xffc0340e/frame 0x2075dcd4 --- kthread start Tracing command nfscbd pid 727 tid 100171 td 0x202f4000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1a5e62c,1a5e644,3ae03f14,103d33c,1a5e644,...) at db_trace_self_wrapper+0x2a/frame 0x3ae03ed0 kdb_backtrace(3ae03ff8,13d05e1,1430c59,0,3ae04030,...) at kdb_backtrace+0x2e/frame 0x3ae03f30 kdb_reenter(1430c59,0,3ae04030,3ae0400c,28409000,...) at kdb_reenter+0x29/frame 0x3ae03f38 trap(3ae04004,8,28,28,ffffffff,...) at trap+0xd1/frame 0x3ae03ff8 calltrap() at 0xffc0319f/frame 0x3ae03ff8 --- trap 0xc, eip = 0x13ac85a, esp = 0x3ae04044, ebp = 0x3ae04050 --- db_trace_thread(202f4000,ffffffff) at db_trace_thread+0x2a/frame 0x3ae04050 _db_stack_trace_all(1028685,0,ffffffff,3ae040dc) at _db_stack_trace_all+0x93/frame 0x3ae040b0 db_command(1ae7672) at db_command+0x26a/frame 0x3ae04164 db_script_exec(bd2daf,0,3ae0420c,3ae041f0) at db_script_exec+0x1a6/frame 0x3ae041c4 db_command(3ae042c8,a06536,0,0,2af64700,...) at db_command+0x26a/frame 0x3ae04278 db_command_loop(0,0,2af64700,3ae04280,3ae042c8,...) at db_command_loop+0x3c/frame 0x3ae04280 db_trap(3,0) at db_trap+0xf6/frame 0x3ae042c8 kdb_trap(3,0,3ae043cc) at kdb_trap+0x109/frame 0x3ae042f4 trap(3ae043cc,8,28,28,100,...) at trap+0x382/frame 0x3ae043c0 calltrap() at 0xffc0319f/frame 0x3ae043c0 --- trap 0x3, eip = 0x1028685, esp = 0x3ae0440c, ebp = 0x3ae04410 --- kdb_enter(14df98a,14df98a) at kdb_enter+0x35/frame 0x3ae04410 vpanic(146f152,3ae04458,3ae04458,3ae04488,f39752,...) at vpanic+0x137/frame 0x3ae04438 panic(146f152,0,18d7ef80,fc50a7,18d7ae00,...) at panic+0x14/frame 0x3ae0444c g_io_deliver(261e8cf0,0) at g_io_deliver+0x322/frame 0x3ae04488 g_disk_done(24029678,24029678) at g_disk_done+0xf0/frame 0x3ae044bc cam_iosched_queue_work(18813700,24029678) at cam_iosched_queue_work+0xcb/frame 0x3ae044d8 adastrategy(24029678,0,114,2c4,b9aa800,...) at adastrategy+0x50/frame 0x3ae044f4 g_disk_start(261e8cf0,18d7ef80,18d7ae00,f38877,b9aa800,...) at g_disk_start+0x4de/frame 0x3ae04550 g_io_request(261e8cf0,18d7ef80,33901fec,2b3b598,3ae045f8,...) at g_io_request+0x2aa/frame 0x3ae04578 g_part_start(339019b4,18d88dc0,9ac4900,f38877,b9aa800,...) at g_part_start+0x26f/frame 0x3ae045e8 g_io_request(339019b4,18d88dc0,1d8fb8c,1,41000000,...) at g_io_request+0x2aa/frame 0x3ae04610 g_part_start(2d39fbdc,1,18d8ed00,18d94b80,30ebd600,...) at g_part_start+0x26f/frame 0x3ae04680 g_io_request(2d39fbdc,b8e0400,b8e0400) at g_io_request+0x2aa/frame 0x3ae046a8 g_vfs_strategy(20051828,30ebd600) at g_vfs_strategy+0xe8/frame 0x3ae046c8 softdep_request_cleanup(20165000,2a343758,29a3e000,4) at softdep_request_cleanup+0x2b7/frame 0x3ae04720 ffs_alloc(2aafb4fc,0,0,6a4908,0,1000,2050000,29a3e000,3ae047d8) at ffs_alloc+0x2ad/frame 0x3ae0476c ffs_balloc_ufs2(2a343758,0,0,1000,29a3e000,2050000,3ae04900) at ffs_balloc_ufs2+0xd8b/frame 0x3ae048a0 ffs_write(3ae04a08) at ffs_write+0x216/frame 0x3ae04924 VOP_WRITE_APV(1aabe80,3ae04a08) at VOP_WRITE_APV+0xdf/frame 0x3ae049e0 vn_write(2a29ec30,3ae04bd0,2881e500,1,2af64700) at vn_write+0x305/frame 0x3ae04a3c vn_io_fault_doio(2af64700) at vn_io_fault_doio+0x3a/frame 0x3ae04a84 vn_io_fault1(2af64700) at vn_io_fault1+0x164/frame 0x3ae04b34 vn_io_fault(2a29ec30,3ae04bd0,2881e500,0,2af64700) at vn_io_fault+0x1c4/frame 0x3ae04b70 dofilewrite(2a29ec30,3ae04bd0,ffffffff,ffffffff,0) at dofilewrite+0x71/frame 0x3ae04ba4 sys_write(2af64700,2af64998,2af64700,28409000,2af64700,...) at sys_write+0x78/frame 0x3ae04c08 syscall(3ae04ce8,3b,3b,3b,1000,...) at syscall+0x1b8/frame 0x3ae04cdc Xint0x80_syscall() at 0xffc033f9/frame 0x3ae04cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- Tracing command cron pid 691 tid 100176 td 0x22defe00 sched_switch(22defe00,104) at sched_switch+0x4e9/frame 0x20735a70 mi_switch(104) at mi_switch+0x1c4/frame 0x20735aa8 sleepq_switch(d00,3,6c,0,1d71ed4,...) at sleepq_switch+0xf7/frame 0x20735ac8 sleepq_catch_signals(14e4b25,183,22defe00,100,1d5f8cc,...) at sleepq_catch_signals+0x278/frame 0x20735af0 sleepq_timedwait_sig(1d5f8cc,6c,0,5587550,1fc884d8,...) at sleepq_timedwait_sig+0x17/frame 0x20735b1c _sleep(1d5f8cc,0,16c,14f7fc3,7e3482bf,974e,bf2120f3,3,200) at _sleep+0x21a/frame 0x20735b5c kern_clock_nanosleep(22defe00,0,0,20735be0,20735be8) at kern_clock_nanosleep+0x1b7/frame 0x20735bc4 sys_nanosleep(22defe00,22df0098,22defe00,22e686dc,22defe00,...) at sys_nanosleep+0x37/frame 0x20735c08 syscall(20735ce8,3b,3b,3b,ffbfece0,...) at syscall+0x1b8/frame 0x20735cdc Xint0x80_syscall() at 0xffc033f9/frame 0x20735cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- Tracing command sendmail pid 687 tid 100126 td 0x2004ee00 sched_switch(2004ee00,104) at sched_switch+0x4e9/frame 0x20613ac0 mi_switch(104) at mi_switch+0x1c4/frame 0x20613af4 sleepq_switch(3e40,14e4b25,74,0,1d75014,...) at sleepq_switch+0xf7/frame 0x20613b14 sleepq_catch_signals(20613b84,feff3c,9b08d74,74,2004ee00,...) at sleepq_catch_signals+0x278/frame 0x20613b3c sleepq_wait_sig(9b08d74,74) at sleepq_wait_sig+0xe/frame 0x20613b44 _sleep(9b08d74,9b08db8,174,144471d,0,0,0,0,100) at _sleep+0x27c/frame 0x20613b84 kern_sigsuspend(2004ee00,0,0,0,0) at kern_sigsuspend+0xca/frame 0x20613bcc sys_sigsuspend(2004ee00,2004f098,2004ee00,9b08d10,2004ee00,...) at sys_sigsuspend+0x3b/frame 0x20613c08 syscall(20613ce8,3b,3b,3b,20a20000,...) at syscall+0x1b8/frame 0x20613cdc Xint0x80_syscall() at 0xffc033f9/frame 0x20613cdc --- syscall (560, FreeBSD ELF32, sys_kevent), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe878 --- Tracing command sendmail pid 684 tid 100127 td 0x2004ea80 sched_switch(2004ea80,104) at sched_switch+0x4e9/frame 0x20618930 mi_switch(104) at mi_switch+0x1c4/frame 0x20618968 sleepq_switch(36c0,1d9ff6c,0,0,1d74894,...) at sleepq_switch+0xf7/frame 0x20618988 sleepq_catch_signals(e4,2004ea80,1a743c8,2004ea80,206189e4,...) at sleepq_catch_signals+0x278/frame 0x206189b0 sleepq_timedwait_sig(1fc0ff24,0) at sleepq_timedwait_sig+0x17/frame 0x206189e0 _cv_timedwait_sig_sbt(1fc0ff24,1fc0ff10,611766c9,971a,50000000,0,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x20618a1c seltdwait(971a,50000000,0) at seltdwait+0x5e/frame 0x20618a4c kern_select(2004ea80,5,ffbfd120,0,0,20618bf8,20) at kern_select+0x8e5/frame 0x20618bd4 sys_select(2004ea80,2004ed18,2004ea80,9b089cc,b85c5b8,...) at sys_select+0x41/frame 0x20618c08 syscall(20618ce8,3b,3b,3b,20a20000,...) at syscall+0x1b8/frame 0x20618cdc Xint0x80_syscall() at 0xffc033f9/frame 0x20618cdc --- syscall (560, FreeBSD ELF32, sys_kevent), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe878 --- Tracing command sshd pid 681 tid 100174 td 0x22df0500 sched_switch(22df0500,104) at sched_switch+0x4e9/frame 0x2072b970 mi_switch(104) at mi_switch+0x1c4/frame 0x2072b9a8 sleepq_switch(2600,22df0500,0,0,1d737d4,...) at sleepq_switch+0xf7/frame 0x2072b9c8 sleepq_catch_signals(2072ba30,f845b8,b9d3ca4,0,13fc2f9,...) at sleepq_catch_signals+0x278/frame 0x2072b9f0 sleepq_wait_sig(b9d3ca4,0) at sleepq_wait_sig+0xe/frame 0x2072b9f8 _cv_wait_sig(b9d3ca4,b9d3c90) at _cv_wait_sig+0x138/frame 0x2072ba30 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x2072ba4c kern_select(22df0500,7,2047c180,0,0,0,20) at kern_select+0x8e5/frame 0x2072bbd4 sys_select(22df0500,22df0798,22df0500,20426000,22df0500,...) at sys_select+0x41/frame 0x2072bc08 syscall(2072bce8,3b,3b,3b,20482000,...) at syscall+0x1b8/frame 0x2072bcdc Xint0x80_syscall() at 0xffc033f9/frame 0x2072bcdc --- syscall (560, FreeBSD ELF32, sys_kevent), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe878 --- Tracing command smartd pid 641 tid 100119 td 0x18d3f380 sched_switch(18d3f380,104) at sched_switch+0x4e9/frame 0x1fb18a70 mi_switch(104) at mi_switch+0x1c4/frame 0x1fb18aa8 sleepq_switch(d80,6c,6c,0,1d71f54,...) at sleepq_switch+0xf7/frame 0x1fb18ac8 sleepq_catch_signals(14e4b25,183,18d3f380,100,1d5f8ce,...) at sleepq_catch_signals+0x278/frame 0x1fb18af0 sleepq_timedwait_sig(1d5f8ce,6c,0,0,b9c7aa8,...) at sleepq_timedwait_sig+0x17/frame 0x1fb18b1c _sleep(1d5f8ce,0,16c,14f7fc3,ddc4308c,9ceb,90000000,6c,200) at _sleep+0x21a/frame 0x1fb18b5c kern_clock_nanosleep(18d3f380,0,0,1fb18be0,1fb18be8) at kern_clock_nanosleep+0x1b7/frame 0x1fb18bc4 sys_nanosleep(18d3f380,18d3f618,18d3f380,1fc2b9cc,18d3f380,...) at sys_nanosleep+0x37/frame 0x1fb18c08 syscall(1fb18ce8,3b,3b,3b,20830000,...) at syscall+0x1b8/frame 0x1fb18cdc Xint0x80_syscall() at 0xffc033f9/frame 0x1fb18cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d98, esp = 0xffc0bfe8, ebp = 0xffbfecbc --- Tracing command powerd pid 617 tid 100112 td 0x18923380 sched_switch(18923380,104) at sched_switch+0x4e9/frame 0x1faf5930 mi_switch(104) at mi_switch+0x1c4/frame 0x1faf5968 sleepq_switch(2200,1da001c,0,0,1d733d4,...) at sleepq_switch+0xf7/frame 0x1faf5988 sleepq_catch_signals(e4,18923380,1a743c8,18923380,1faf59e4,...) at sleepq_catch_signals+0x278/frame 0x1faf59b0 sleepq_timedwait_sig(9bf6ce4,0) at sleepq_timedwait_sig+0x17/frame 0x1faf59e0 _cv_timedwait_sig_sbt(9bf6ce4,9bf6cd0,9b43b98e,9717,4000000,0,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x1faf5a1c seltdwait(9717,4000000,0) at seltdwait+0x5e/frame 0x1faf5a4c kern_select(18923380,0,ffbfed38,0,ffbfed38,1faf5bf8,20) at kern_select+0x8e5/frame 0x1faf5bd4 sys_select(18923380,18923618,18923380,182cb344,b8505b8,...) at sys_select+0x41/frame 0x1faf5c08 syscall(1faf5ce8,3b,3b,3b,ffbfedb8,...) at syscall+0x1b8/frame 0x1faf5cdc Xint0x80_syscall() at 0xffc033f9/frame 0x1faf5cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d98, esp = 0xffc0bfe8, ebp = 0xffbfecbc --- Tracing command ntpd pid 614 tid 100168 td 0x22c25500 sched_switch(22c25500,104) at sched_switch+0x4e9/frame 0x2070d970 mi_switch(104) at mi_switch+0x1c4/frame 0x2070d9a8 sleepq_switch(640,22c25500,0,0,1d71814,...) at sleepq_switch+0xf7/frame 0x2070d9c8 sleepq_catch_signals(2070da30,f845b8,1fc0fde4,0,13fc2f9,...) at sleepq_catch_signals+0x278/frame 0x2070d9f0 sleepq_wait_sig(1fc0fde4,0) at sleepq_wait_sig+0xe/frame 0x2070d9f8 _cv_wait_sig(1fc0fde4,1fc0fdd0) at _cv_wait_sig+0x138/frame 0x2070da30 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x2070da4c kern_select(22c25500,1b,ffbfe248,0,0,0,20) at kern_select+0x8e5/frame 0x2070dbd4 sys_select(22c25500,22c25798,22c25500,22c22398,2070dcdc,...) at sys_select+0x41/frame 0x2070dc08 syscall(2070dce8,3b,3b,3b,209ab000,...) at syscall+0x1b8/frame 0x2070dcdc Xint0x80_syscall() at 0xffc033f9/frame 0x2070dcdc --- syscall (560, FreeBSD ELF32, sys_kevent), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe878 --- Tracing command rpc.lockd pid 578 tid 100134 td 0x202f4a80 sched_switch(202f4a80,104) at sched_switch+0x4e9/frame 0x206538e8 mi_switch(104) at mi_switch+0x1c4/frame 0x20653920 sleepq_switch(6c0,1da00cc,0,0,1d71894,...) at sleepq_switch+0xf7/frame 0x20653940 sleepq_catch_signals(e4,202f4a80,1a743c8,202f4a80,2065399c,...) at sleepq_catch_signals+0x278/frame 0x20653968 sleepq_timedwait_sig(22c64b50,0) at sleepq_timedwait_sig+0x17/frame 0x20653998 _cv_timedwait_sig_sbt(22c64b50,204580c0,fffffa38,4,0,0,100) at _cv_timedwait_sig_sbt+0x15a/frame 0x206539d4 svc_run_internal(4,20458000,20653c08,11f1fd0,188fb3e0,...) at svc_run_internal+0xa1c/frame 0x20653ab4 svc_run(20458000) at svc_run+0x192/frame 0x20653ae4 sys_nlm_syscall(202f4a80,202f4d18,202f4a80,20427054,202f4a80,...) at sys_nlm_syscall+0x620/frame 0x20653c08 syscall(20653ce8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x20653cdc Xint0x80_syscall() at 0xffc033f9/frame 0x20653cdc --- syscall (560, FreeBSD ELF32, sys_kevent), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe878 --- Tracing command rpc.statd pid 575 tid 100167 td 0x202f4700 sched_switch(202f4700,104) at sched_switch+0x4e9/frame 0x206fc930 mi_switch(104) at mi_switch+0x1c4/frame 0x206fc968 sleepq_switch(2340,1da0124,0,0,1d73514,...) at sleepq_switch+0xf7/frame 0x206fc988 sleepq_catch_signals(e4,202f4700,1a743c8,202f4700,206fc9e4,...) at sleepq_catch_signals+0x278/frame 0x206fc9b0 sleepq_timedwait_sig(188269e4,0) at sleepq_timedwait_sig+0x17/frame 0x206fc9e0 _cv_timedwait_sig_sbt(188269e4,188269d0,5f8e2f77,9730,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x206fca1c seltdwait(9730,e0000000,1) at seltdwait+0x5e/frame 0x206fca4c kern_select(202f4700,8,ffbfec98,0,0,206fcbf8,20) at kern_select+0x8e5/frame 0x206fcbd4 sys_select(202f4700,202f4998,202f4700,20426d10,1388b7e,...) at sys_select+0x41/frame 0x206fcc08 syscall(206fcce8,3b,3b,3b,ffbfec98,...) at syscall+0x1b8/frame 0x206fccdc Xint0x80_syscall() at 0xffc033f9/frame 0x206fccdc --- syscall (560, FreeBSD ELF32, sys_kevent), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe878 --- Tracing command nfsd pid 574 tid 100135 td 0x22c25880 sched_switch(22c25880,104) at sched_switch+0x4e9/frame 0x20658418 mi_switch(104) at mi_switch+0x1c4/frame 0x20658450 sleepq_switch(3c80,1da0c24,0,0,1d74e54,...) at sleepq_switch+0xf7/frame 0x20658470 sleepq_catch_signals(e4,22c25880,1a743c8,22c25880,206584cc,...) at sleepq_catch_signals+0x278/frame 0x20658498 sleepq_timedwait_sig(188722d0,0) at sleepq_timedwait_sig+0x17/frame 0x206584c8 _cv_timedwait_sig_sbt(188722d0,1881c0c0,fffffa38,4,0,0,100) at _cv_timedwait_sig_sbt+0x15a/frame 0x20658504 svc_run_internal(22c6b180,1881c0c0,1881c0d0,1881c11c,0,...) at svc_run_internal+0xa1c/frame 0x206585e8 svc_run(1881c000) at svc_run+0x192/frame 0x20658618 nfsrvd_nfsd(22c25880,206587d0) at nfsrvd_nfsd+0x2e8/frame 0x20658748 nfssvc_nfsd(22c25880,22c25b18) at nfssvc_nfsd+0x4cb/frame 0x20658bf0 sys_nfssvc(22c25880,22c25b18,22c25880,22c226dc,22c25880,...) at sys_nfssvc+0x92/frame 0x20658c08 syscall(20658ce8,3b,3b,3b,ffbfeb50,...) at syscall+0x1b8/frame 0x20658cdc Xint0x80_syscall() at 0xffc033f9/frame 0x20658cdc --- syscall (560, FreeBSD ELF32, sys_kevent), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe878 --- Tracing command nfsd pid 574 tid 100136 td 0x22c5fc00 sched_switch(22c5fc00,104) at sched_switch+0x4e9/frame 0x20661b20 mi_switch(104) at mi_switch+0x1c4/frame 0x20661b58 sleepq_switch(1e80,22c5fc00,0,0,1d73054,...) at sleepq_switch+0xf7/frame 0x20661b78 sleepq_catch_signals(20661be0,f845b8,1891aad0,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x20661ba0 sleepq_wait_sig(1891aad0,0) at sleepq_wait_sig+0xe/frame 0x20661ba8 _cv_wait_sig(1891aad0,1881c0c0) at _cv_wait_sig+0x138/frame 0x20661be0 svc_run_internal(20661cd4,fa73e8,1881c0c0,20661ce8,120ed90,...) at svc_run_internal+0xb45/frame 0x20661cb0 svc_thread_start(1881c0c0,20661ce8) at svc_thread_start+0xd/frame 0x20661cb8 fork_exit(120ed90,1881c0c0,20661ce8,3b,3b,...) at fork_exit+0x68/frame 0x20661cd4 fork_trampoline() at 0xffc0340e/frame 0x20661cd4 --- kthread start Tracing command nfsd pid 574 tid 100137 td 0x22c5f880 sched_switch(22c5f880,104) at sched_switch+0x4e9/frame 0x20666b20 mi_switch(104) at mi_switch+0x1c4/frame 0x20666b58 sleepq_switch(2040,22c5f880,0,0,1d73214,...) at sleepq_switch+0xf7/frame 0x20666b78 sleepq_catch_signals(20666be0,f845b8,1835d150,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x20666ba0 sleepq_wait_sig(1835d150,0) at sleepq_wait_sig+0xe/frame 0x20666ba8 _cv_wait_sig(1835d150,1881c0c0) at _cv_wait_sig+0x138/frame 0x20666be0 svc_run_internal(20666cd4,fa73e8,1881c0c0,20666ce8,120ed90,...) at svc_run_internal+0xb45/frame 0x20666cb0 svc_thread_start(1881c0c0,20666ce8) at svc_thread_start+0xd/frame 0x20666cb8 fork_exit(120ed90,1881c0c0,20666ce8,3b,3b,...) at fork_exit+0x68/frame 0x20666cd4 fork_trampoline() at 0xffc0340e/frame 0x20666cd4 --- kthread start Tracing command nfsd pid 574 tid 100138 td 0x22c5f500 sched_switch(22c5f500,104) at sched_switch+0x4e9/frame 0x2066bb20 mi_switch(104) at mi_switch+0x1c4/frame 0x2066bb58 sleepq_switch(3e80,22c5f500,0,0,1d75054,...) at sleepq_switch+0xf7/frame 0x2066bb78 sleepq_catch_signals(2066bbe0,f845b8,1891aa50,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x2066bba0 sleepq_wait_sig(1891aa50,0) at sleepq_wait_sig+0xe/frame 0x2066bba8 _cv_wait_sig(1891aa50,1881c0c0) at _cv_wait_sig+0x138/frame 0x2066bbe0 svc_run_internal(2066bcd4,fa73e8,1881c0c0,2066bce8,120ed90,...) at svc_run_internal+0xb45/frame 0x2066bcb0 svc_thread_start(1881c0c0,2066bce8) at svc_thread_start+0xd/frame 0x2066bcb8 fork_exit(120ed90,1881c0c0,2066bce8,3b,3b,...) at fork_exit+0x68/frame 0x2066bcd4 fork_trampoline() at 0xffc0340e/frame 0x2066bcd4 --- kthread start Tracing command nfsd pid 574 tid 100139 td 0x22c5f180 sched_switch(22c5f180,104) at sched_switch+0x4e9/frame 0x20670b20 mi_switch(104) at mi_switch+0x1c4/frame 0x20670b58 sleepq_switch(1e40,22c5f180,0,0,1d73014,...) at sleepq_switch+0xf7/frame 0x20670b78 sleepq_catch_signals(20670be0,f845b8,1891a9d0,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x20670ba0 sleepq_wait_sig(1891a9d0,0) at sleepq_wait_sig+0xe/frame 0x20670ba8 _cv_wait_sig(1891a9d0,1881c0c0) at _cv_wait_sig+0x138/frame 0x20670be0 svc_run_internal(20670cd4,fa73e8,1881c0c0,20670ce8,120ed90,...) at svc_run_internal+0xb45/frame 0x20670cb0 svc_thread_start(1881c0c0,20670ce8) at svc_thread_start+0xd/frame 0x20670cb8 fork_exit(120ed90,1881c0c0,20670ce8,3b,3b,...) at fork_exit+0x68/frame 0x20670cd4 fork_trampoline() at 0xffc0340e/frame 0x20670cd4 --- kthread start Tracing command nfsd pid 574 tid 100140 td 0x22c5ee00 sched_switch(22c5ee00,104) at sched_switch+0x4e9/frame 0x20675b20 mi_switch(104) at mi_switch+0x1c4/frame 0x20675b58 sleepq_switch(0,22c5ee00,0,0,1d711d4,...) at sleepq_switch+0xf7/frame 0x20675b78 sleepq_catch_signals(20675be0,f845b8,1835d0d0,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x20675ba0 sleepq_wait_sig(1835d0d0,0) at sleepq_wait_sig+0xe/frame 0x20675ba8 _cv_wait_sig(1835d0d0,1881c0c0) at _cv_wait_sig+0x138/frame 0x20675be0 svc_run_internal(20675cd4,fa73e8,1881c0c0,20675ce8,120ed90,...) at svc_run_internal+0xb45/frame 0x20675cb0 svc_thread_start(1881c0c0,20675ce8) at svc_thread_start+0xd/frame 0x20675cb8 fork_exit(120ed90,1881c0c0,20675ce8,3b,3b,...) at fork_exit+0x68/frame 0x20675cd4 fork_trampoline() at 0xffc0340e/frame 0x20675cd4 --- kthread start Tracing command nfsd pid 574 tid 100141 td 0x22c5ea80 sched_switch(22c5ea80,104) at sched_switch+0x4e9/frame 0x2067ab20 mi_switch(104) at mi_switch+0x1c4/frame 0x2067ab58 sleepq_switch(3e40,22c5ea80,0,0,1d75014,...) at sleepq_switch+0xf7/frame 0x2067ab78 sleepq_catch_signals(2067abe0,f845b8,1891a950,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x2067aba0 sleepq_wait_sig(1891a950,0) at sleepq_wait_sig+0xe/frame 0x2067aba8 _cv_wait_sig(1891a950,1881c0c0) at _cv_wait_sig+0x138/frame 0x2067abe0 svc_run_internal(2067acd4,fa73e8,1881c0c0,2067ace8,120ed90,...) at svc_run_internal+0xb45/frame 0x2067acb0 svc_thread_start(1881c0c0,2067ace8) at svc_thread_start+0xd/frame 0x2067acb8 fork_exit(120ed90,1881c0c0,2067ace8,3b,3b,...) at fork_exit+0x68/frame 0x2067acd4 fork_trampoline() at 0xffc0340e/frame 0x2067acd4 --- kthread start Tracing command nfsd pid 574 tid 100142 td 0x22c5e700 sched_switch(22c5e700,104) at sched_switch+0x4e9/frame 0x2067fb20 mi_switch(104) at mi_switch+0x1c4/frame 0x2067fb58 sleepq_switch(1e00,22c5e700,0,0,1d72fd4,...) at sleepq_switch+0xf7/frame 0x2067fb78 sleepq_catch_signals(2067fbe0,f845b8,1891a8d0,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x2067fba0 sleepq_wait_sig(1891a8d0,0) at sleepq_wait_sig+0xe/frame 0x2067fba8 _cv_wait_sig(1891a8d0,1881c0c0) at _cv_wait_sig+0x138/frame 0x2067fbe0 svc_run_internal(2067fcd4,fa73e8,1881c0c0,2067fce8,120ed90,...) at svc_run_internal+0xb45/frame 0x2067fcb0 svc_thread_start(1881c0c0,2067fce8) at svc_thread_start+0xd/frame 0x2067fcb8 fork_exit(120ed90,1881c0c0,2067fce8,3b,3b,...) at fork_exit+0x68/frame 0x2067fcd4 fork_trampoline() at 0xffc0340e/frame 0x2067fcd4 --- kthread start Tracing command nfsd pid 574 tid 100143 td 0x22c5e380 sched_switch(22c5e380,104) at sched_switch+0x4e9/frame 0x20684b20 mi_switch(104) at mi_switch+0x1c4/frame 0x20684b58 sleepq_switch(2000,22c5e380,0,0,1d731d4,...) at sleepq_switch+0xf7/frame 0x20684b78 sleepq_catch_signals(20684be0,f845b8,1835d050,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x20684ba0 sleepq_wait_sig(1835d050,0) at sleepq_wait_sig+0xe/frame 0x20684ba8 _cv_wait_sig(1835d050,1881c0c0) at _cv_wait_sig+0x138/frame 0x20684be0 svc_run_internal(20684cd4,fa73e8,1881c0c0,20684ce8,120ed90,...) at svc_run_internal+0xb45/frame 0x20684cb0 svc_thread_start(1881c0c0,20684ce8) at svc_thread_start+0xd/frame 0x20684cb8 fork_exit(120ed90,1881c0c0,20684ce8,3b,3b,...) at fork_exit+0x68/frame 0x20684cd4 fork_trampoline() at 0xffc0340e/frame 0x20684cd4 --- kthread start Tracing command nfsd pid 574 tid 100144 td 0x22c5e000 sched_switch(22c5e000,104) at sched_switch+0x4e9/frame 0x20689b20 mi_switch(104) at mi_switch+0x1c4/frame 0x20689b58 sleepq_switch(3e00,22c5e000,0,0,1d74fd4,...) at sleepq_switch+0xf7/frame 0x20689b78 sleepq_catch_signals(20689be0,f845b8,1891a850,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x20689ba0 sleepq_wait_sig(1891a850,0) at sleepq_wait_sig+0xe/frame 0x20689ba8 _cv_wait_sig(1891a850,1881c0c0) at _cv_wait_sig+0x138/frame 0x20689be0 svc_run_internal(20689cd4,fa73e8,1881c0c0,20689ce8,120ed90,...) at svc_run_internal+0xb45/frame 0x20689cb0 svc_thread_start(1881c0c0,20689ce8) at svc_thread_start+0xd/frame 0x20689cb8 fork_exit(120ed90,1881c0c0,20689ce8,3b,3b,...) at fork_exit+0x68/frame 0x20689cd4 fork_trampoline() at 0xffc0340e/frame 0x20689cd4 --- kthread start Tracing command nfsd pid 574 tid 100145 td 0x22c63c00 sched_switch(22c63c00,104) at sched_switch+0x4e9/frame 0x2068eb20 mi_switch(104) at mi_switch+0x1c4/frame 0x2068eb58 sleepq_switch(1dc0,22c63c00,0,0,1d72f94,...) at sleepq_switch+0xf7/frame 0x2068eb78 sleepq_catch_signals(2068ebe0,f845b8,1891a7d0,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x2068eba0 sleepq_wait_sig(1891a7d0,0) at sleepq_wait_sig+0xe/frame 0x2068eba8 _cv_wait_sig(1891a7d0,1881c0c0) at _cv_wait_sig+0x138/frame 0x2068ebe0 svc_run_internal(2068ecd4,fa73e8,1881c0c0,2068ece8,120ed90,...) at svc_run_internal+0xb45/frame 0x2068ecb0 svc_thread_start(1881c0c0,2068ece8) at svc_thread_start+0xd/frame 0x2068ecb8 fork_exit(120ed90,1881c0c0,2068ece8,3b,3b,...) at fork_exit+0x68/frame 0x2068ecd4 fork_trampoline() at 0xffc0340e/frame 0x2068ecd4 --- kthread start Tracing command nfsd pid 574 tid 100146 td 0x22c63880 sched_switch(22c63880,104) at sched_switch+0x4e9/frame 0x20693b20 mi_switch(104) at mi_switch+0x1c4/frame 0x20693b58 sleepq_switch(7c0,22c63880,0,0,1d71994,...) at sleepq_switch+0xf7/frame 0x20693b78 sleepq_catch_signals(20693be0,f845b8,22c64f50,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x20693ba0 sleepq_wait_sig(22c64f50,0) at sleepq_wait_sig+0xe/frame 0x20693ba8 _cv_wait_sig(22c64f50,1881c0c0) at _cv_wait_sig+0x138/frame 0x20693be0 svc_run_internal(20693cd4,fa73e8,1881c0c0,20693ce8,120ed90,...) at svc_run_internal+0xb45/frame 0x20693cb0 svc_thread_start(1881c0c0,20693ce8) at svc_thread_start+0xd/frame 0x20693cb8 fork_exit(120ed90,1881c0c0,20693ce8,3b,3b,...) at fork_exit+0x68/frame 0x20693cd4 fork_trampoline() at 0xffc0340e/frame 0x20693cd4 --- kthread start Tracing command nfsd pid 574 tid 100147 td 0x22c63500 sched_switch(22c63500,104) at sched_switch+0x4e9/frame 0x20698b20 mi_switch(104) at mi_switch+0x1c4/frame 0x20698b58 sleepq_switch(3dc0,22c63500,0,0,1d74f94,...) at sleepq_switch+0xf7/frame 0x20698b78 sleepq_catch_signals(20698be0,f845b8,1891a750,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x20698ba0 sleepq_wait_sig(1891a750,0) at sleepq_wait_sig+0xe/frame 0x20698ba8 _cv_wait_sig(1891a750,1881c0c0) at _cv_wait_sig+0x138/frame 0x20698be0 svc_run_internal(20698cd4,fa73e8,1881c0c0,20698ce8,120ed90,...) at svc_run_internal+0xb45/frame 0x20698cb0 svc_thread_start(1881c0c0,20698ce8) at svc_thread_start+0xd/frame 0x20698cb8 fork_exit(120ed90,1881c0c0,20698ce8,3b,3b,...) at fork_exit+0x68/frame 0x20698cd4 fork_trampoline() at 0xffc0340e/frame 0x20698cd4 --- kthread start Tracing command nfsd pid 574 tid 100148 td 0x22c63180 sched_switch(22c63180,104) at sched_switch+0x4e9/frame 0x2069db20 mi_switch(104) at mi_switch+0x1c4/frame 0x2069db58 sleepq_switch(1d80,22c63180,0,0,1d72f54,...) at sleepq_switch+0xf7/frame 0x2069db78 sleepq_catch_signals(2069dbe0,f845b8,1891a6d0,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x2069dba0 sleepq_wait_sig(1891a6d0,0) at sleepq_wait_sig+0xe/frame 0x2069dba8 _cv_wait_sig(1891a6d0,1881c0c0) at _cv_wait_sig+0x138/frame 0x2069dbe0 svc_run_internal(2069dcd4,fa73e8,1881c0c0,2069dce8,120ed90,...) at svc_run_internal+0xb45/frame 0x2069dcb0 svc_thread_start(1881c0c0,2069dce8) at svc_thread_start+0xd/frame 0x2069dcb8 fork_exit(120ed90,1881c0c0,2069dce8,3b,3b,...) at fork_exit+0x68/frame 0x2069dcd4 fork_trampoline() at 0xffc0340e/frame 0x2069dcd4 --- kthread start Tracing command nfsd pid 574 tid 100149 td 0x22c62e00 sched_switch(22c62e00,104) at sched_switch+0x4e9/frame 0x206a2b20 mi_switch(104) at mi_switch+0x1c4/frame 0x206a2b58 sleepq_switch(2780,22c62e00,0,0,1d73954,...) at sleepq_switch+0xf7/frame 0x206a2b78 sleepq_catch_signals(206a2be0,f845b8,22c64ed0,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x206a2ba0 sleepq_wait_sig(22c64ed0,0) at sleepq_wait_sig+0xe/frame 0x206a2ba8 _cv_wait_sig(22c64ed0,1881c0c0) at _cv_wait_sig+0x138/frame 0x206a2be0 svc_run_internal(206a2cd4,fa73e8,1881c0c0,206a2ce8,120ed90,...) at svc_run_internal+0xb45/frame 0x206a2cb0 svc_thread_start(1881c0c0,206a2ce8) at svc_thread_start+0xd/frame 0x206a2cb8 fork_exit(120ed90,1881c0c0,206a2ce8,3b,3b,...) at fork_exit+0x68/frame 0x206a2cd4 fork_trampoline() at 0xffc0340e/frame 0x206a2cd4 --- kthread start Tracing command nfsd pid 574 tid 100150 td 0x22c62a80 sched_switch(22c62a80,104) at sched_switch+0x4e9/frame 0x206a7b20 mi_switch(104) at mi_switch+0x1c4/frame 0x206a7b58 sleepq_switch(3d80,22c62a80,0,0,1d74f54,...) at sleepq_switch+0xf7/frame 0x206a7b78 sleepq_catch_signals(206a7be0,f845b8,1891a650,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x206a7ba0 sleepq_wait_sig(1891a650,0) at sleepq_wait_sig+0xe/frame 0x206a7ba8 _cv_wait_sig(1891a650,1881c0c0) at _cv_wait_sig+0x138/frame 0x206a7be0 svc_run_internal(206a7cd4,fa73e8,1881c0c0,206a7ce8,120ed90,...) at svc_run_internal+0xb45/frame 0x206a7cb0 svc_thread_start(1881c0c0,206a7ce8) at svc_thread_start+0xd/frame 0x206a7cb8 fork_exit(120ed90,1881c0c0,206a7ce8,3b,3b,...) at fork_exit+0x68/frame 0x206a7cd4 fork_trampoline() at 0xffc0340e/frame 0x206a7cd4 --- kthread start Tracing command nfsd pid 574 tid 100151 td 0x22c62700 sched_switch(22c62700,104) at sched_switch+0x4e9/frame 0x206acb20 mi_switch(104) at mi_switch+0x1c4/frame 0x206acb58 sleepq_switch(780,22c62700,0,0,1d71954,...) at sleepq_switch+0xf7/frame 0x206acb78 sleepq_catch_signals(206acbe0,f845b8,22c64e50,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x206acba0 sleepq_wait_sig(22c64e50,0) at sleepq_wait_sig+0xe/frame 0x206acba8 _cv_wait_sig(22c64e50,1881c0c0) at _cv_wait_sig+0x138/frame 0x206acbe0 svc_run_internal(206accd4,fa73e8,1881c0c0,206acce8,120ed90,...) at svc_run_internal+0xb45/frame 0x206accb0 svc_thread_start(1881c0c0,206acce8) at svc_thread_start+0xd/frame 0x206accb8 fork_exit(120ed90,1881c0c0,206acce8,3b,3b,...) at fork_exit+0x68/frame 0x206accd4 fork_trampoline() at 0xffc0340e/frame 0x206accd4 --- kthread start Tracing command nfsd pid 574 tid 100152 td 0x22c62380 sched_switch(22c62380,104) at sched_switch+0x4e9/frame 0x206b1b20 mi_switch(104) at mi_switch+0x1c4/frame 0x206b1b58 sleepq_switch(1d40,22c62380,0,0,1d72f14,...) at sleepq_switch+0xf7/frame 0x206b1b78 sleepq_catch_signals(206b1be0,f845b8,1891a5d0,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x206b1ba0 sleepq_wait_sig(1891a5d0,0) at sleepq_wait_sig+0xe/frame 0x206b1ba8 _cv_wait_sig(1891a5d0,1881c0c0) at _cv_wait_sig+0x138/frame 0x206b1be0 svc_run_internal(206b1cd4,fa73e8,1881c0c0,206b1ce8,120ed90,...) at svc_run_internal+0xb45/frame 0x206b1cb0 svc_thread_start(1881c0c0,206b1ce8) at svc_thread_start+0xd/frame 0x206b1cb8 fork_exit(120ed90,1881c0c0,206b1ce8,3b,3b,...) at fork_exit+0x68/frame 0x206b1cd4 fork_trampoline() at 0xffc0340e/frame 0x206b1cd4 --- kthread start Tracing command nfsd pid 574 tid 100153 td 0x22c62000 sched_switch(22c62000,104) at sched_switch+0x4e9/frame 0x206b6b20 mi_switch(104) at mi_switch+0x1c4/frame 0x206b6b58 sleepq_switch(3d40,22c62000,0,0,1d74f14,...) at sleepq_switch+0xf7/frame 0x206b6b78 sleepq_catch_signals(206b6be0,f845b8,1891a550,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x206b6ba0 sleepq_wait_sig(1891a550,0) at sleepq_wait_sig+0xe/frame 0x206b6ba8 _cv_wait_sig(1891a550,1881c0c0) at _cv_wait_sig+0x138/frame 0x206b6be0 svc_run_internal(206b6cd4,fa73e8,1881c0c0,206b6ce8,120ed90,...) at svc_run_internal+0xb45/frame 0x206b6cb0 svc_thread_start(1881c0c0,206b6ce8) at svc_thread_start+0xd/frame 0x206b6cb8 fork_exit(120ed90,1881c0c0,206b6ce8,3b,3b,...) at fork_exit+0x68/frame 0x206b6cd4 fork_trampoline() at 0xffc0340e/frame 0x206b6cd4 --- kthread start Tracing command nfsd pid 574 tid 100154 td 0x22c67c00 sched_switch(22c67c00,104) at sched_switch+0x4e9/frame 0x206bbb20 mi_switch(104) at mi_switch+0x1c4/frame 0x206bbb58 sleepq_switch(2740,22c67c00,0,0,1d73914,...) at sleepq_switch+0xf7/frame 0x206bbb78 sleepq_catch_signals(206bbbe0,f845b8,22c64dd0,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x206bbba0 sleepq_wait_sig(22c64dd0,0) at sleepq_wait_sig+0xe/frame 0x206bbba8 _cv_wait_sig(22c64dd0,1881c0c0) at _cv_wait_sig+0x138/frame 0x206bbbe0 svc_run_internal(206bbcd4,fa73e8,1881c0c0,206bbce8,120ed90,...) at svc_run_internal+0xb45/frame 0x206bbcb0 svc_thread_start(1881c0c0,206bbce8) at svc_thread_start+0xd/frame 0x206bbcb8 fork_exit(120ed90,1881c0c0,206bbce8,3b,3b,...) at fork_exit+0x68/frame 0x206bbcd4 fork_trampoline() at 0xffc0340e/frame 0x206bbcd4 --- kthread start Tracing command nfsd pid 574 tid 100155 td 0x22c67880 sched_switch(22c67880,104) at sched_switch+0x4e9/frame 0x206c0b20 mi_switch(104) at mi_switch+0x1c4/frame 0x206c0b58 sleepq_switch(1d00,22c67880,0,0,1d72ed4,...) at sleepq_switch+0xf7/frame 0x206c0b78 sleepq_catch_signals(206c0be0,f845b8,1891a4d0,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x206c0ba0 sleepq_wait_sig(1891a4d0,0) at sleepq_wait_sig+0xe/frame 0x206c0ba8 _cv_wait_sig(1891a4d0,1881c0c0) at _cv_wait_sig+0x138/frame 0x206c0be0 svc_run_internal(206c0cd4,fa73e8,1881c0c0,206c0ce8,120ed90,...) at svc_run_internal+0xb45/frame 0x206c0cb0 svc_thread_start(1881c0c0,206c0ce8) at svc_thread_start+0xd/frame 0x206c0cb8 fork_exit(120ed90,1881c0c0,206c0ce8,3b,3b,...) at fork_exit+0x68/frame 0x206c0cd4 fork_trampoline() at 0xffc0340e/frame 0x206c0cd4 --- kthread start Tracing command nfsd pid 574 tid 100156 td 0x22c67500 sched_switch(22c67500,104) at sched_switch+0x4e9/frame 0x206c5b20 mi_switch(104) at mi_switch+0x1c4/frame 0x206c5b58 sleepq_switch(3d00,22c67500,0,0,1d74ed4,...) at sleepq_switch+0xf7/frame 0x206c5b78 sleepq_catch_signals(206c5be0,f845b8,1891a450,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x206c5ba0 sleepq_wait_sig(1891a450,0) at sleepq_wait_sig+0xe/frame 0x206c5ba8 _cv_wait_sig(1891a450,1881c0c0) at _cv_wait_sig+0x138/frame 0x206c5be0 svc_run_internal(206c5cd4,fa73e8,1881c0c0,206c5ce8,120ed90,...) at svc_run_internal+0xb45/frame 0x206c5cb0 svc_thread_start(1881c0c0,206c5ce8) at svc_thread_start+0xd/frame 0x206c5cb8 fork_exit(120ed90,1881c0c0,206c5ce8,3b,3b,...) at fork_exit+0x68/frame 0x206c5cd4 fork_trampoline() at 0xffc0340e/frame 0x206c5cd4 --- kthread start Tracing command nfsd pid 574 tid 100157 td 0x22c67180 sched_switch(22c67180,104) at sched_switch+0x4e9/frame 0x206cab20 mi_switch(104) at mi_switch+0x1c4/frame 0x206cab58 sleepq_switch(740,22c67180,0,0,1d71914,...) at sleepq_switch+0xf7/frame 0x206cab78 sleepq_catch_signals(206cabe0,f845b8,22c64d50,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x206caba0 sleepq_wait_sig(22c64d50,0) at sleepq_wait_sig+0xe/frame 0x206caba8 _cv_wait_sig(22c64d50,1881c0c0) at _cv_wait_sig+0x138/frame 0x206cabe0 svc_run_internal(206cacd4,fa73e8,1881c0c0,206cace8,120ed90,...) at svc_run_internal+0xb45/frame 0x206cacb0 svc_thread_start(1881c0c0,206cace8) at svc_thread_start+0xd/frame 0x206cacb8 fork_exit(120ed90,1881c0c0,206cace8,3b,3b,...) at fork_exit+0x68/frame 0x206cacd4 fork_trampoline() at 0xffc0340e/frame 0x206cacd4 --- kthread start Tracing command nfsd pid 574 tid 100158 td 0x22c66e00 sched_switch(22c66e00,104) at sched_switch+0x4e9/frame 0x206cfb20 mi_switch(104) at mi_switch+0x1c4/frame 0x206cfb58 sleepq_switch(1cc0,22c66e00,0,0,1d72e94,...) at sleepq_switch+0xf7/frame 0x206cfb78 sleepq_catch_signals(206cfbe0,f845b8,1891a3d0,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x206cfba0 sleepq_wait_sig(1891a3d0,0) at sleepq_wait_sig+0xe/frame 0x206cfba8 _cv_wait_sig(1891a3d0,1881c0c0) at _cv_wait_sig+0x138/frame 0x206cfbe0 svc_run_internal(206cfcd4,fa73e8,1881c0c0,206cfce8,120ed90,...) at svc_run_internal+0xb45/frame 0x206cfcb0 svc_thread_start(1881c0c0,206cfce8) at svc_thread_start+0xd/frame 0x206cfcb8 fork_exit(120ed90,1881c0c0,206cfce8,3b,3b,...) at fork_exit+0x68/frame 0x206cfcd4 fork_trampoline() at 0xffc0340e/frame 0x206cfcd4 --- kthread start Tracing command nfsd pid 574 tid 100159 td 0x22c66a80 sched_switch(22c66a80,104) at sched_switch+0x4e9/frame 0x206d4b20 mi_switch(104) at mi_switch+0x1c4/frame 0x206d4b58 sleepq_switch(2700,22c66a80,0,0,1d738d4,...) at sleepq_switch+0xf7/frame 0x206d4b78 sleepq_catch_signals(206d4be0,f845b8,22c64cd0,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x206d4ba0 sleepq_wait_sig(22c64cd0,0) at sleepq_wait_sig+0xe/frame 0x206d4ba8 _cv_wait_sig(22c64cd0,1881c0c0) at _cv_wait_sig+0x138/frame 0x206d4be0 svc_run_internal(206d4cd4,fa73e8,1881c0c0,206d4ce8,120ed90,...) at svc_run_internal+0xb45/frame 0x206d4cb0 svc_thread_start(1881c0c0,206d4ce8) at svc_thread_start+0xd/frame 0x206d4cb8 fork_exit(120ed90,1881c0c0,206d4ce8,3b,3b,...) at fork_exit+0x68/frame 0x206d4cd4 fork_trampoline() at 0xffc0340e/frame 0x206d4cd4 --- kthread start Tracing command nfsd pid 574 tid 100160 td 0x22c66700 sched_switch(22c66700,104) at sched_switch+0x4e9/frame 0x206d9b20 mi_switch(104) at mi_switch+0x1c4/frame 0x206d9b58 sleepq_switch(3cc0,22c66700,0,0,1d74e94,...) at sleepq_switch+0xf7/frame 0x206d9b78 sleepq_catch_signals(206d9be0,f845b8,1891a350,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x206d9ba0 sleepq_wait_sig(1891a350,0) at sleepq_wait_sig+0xe/frame 0x206d9ba8 _cv_wait_sig(1891a350,1881c0c0) at _cv_wait_sig+0x138/frame 0x206d9be0 svc_run_internal(206d9cd4,fa73e8,1881c0c0,206d9ce8,120ed90,...) at svc_run_internal+0xb45/frame 0x206d9cb0 svc_thread_start(1881c0c0,206d9ce8) at svc_thread_start+0xd/frame 0x206d9cb8 fork_exit(120ed90,1881c0c0,206d9ce8,3b,3b,...) at fork_exit+0x68/frame 0x206d9cd4 fork_trampoline() at 0xffc0340e/frame 0x206d9cd4 --- kthread start Tracing command nfsd pid 574 tid 100161 td 0x22c66380 sched_switch(22c66380,104) at sched_switch+0x4e9/frame 0x206deb20 mi_switch(104) at mi_switch+0x1c4/frame 0x206deb58 sleepq_switch(700,22c66380,0,0,1d718d4,...) at sleepq_switch+0xf7/frame 0x206deb78 sleepq_catch_signals(206debe0,f845b8,22c64c50,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x206deba0 sleepq_wait_sig(22c64c50,0) at sleepq_wait_sig+0xe/frame 0x206deba8 _cv_wait_sig(22c64c50,1881c0c0) at _cv_wait_sig+0x138/frame 0x206debe0 svc_run_internal(206decd4,fa73e8,1881c0c0,206dece8,120ed90,...) at svc_run_internal+0xb45/frame 0x206decb0 svc_thread_start(1881c0c0,206dece8) at svc_thread_start+0xd/frame 0x206decb8 fork_exit(120ed90,1881c0c0,206dece8,3b,3b,...) at fork_exit+0x68/frame 0x206decd4 fork_trampoline() at 0xffc0340e/frame 0x206decd4 --- kthread start Tracing command nfsd pid 574 tid 100162 td 0x22c66000 sched_switch(22c66000,104) at sched_switch+0x4e9/frame 0x206e3b20 mi_switch(104) at mi_switch+0x1c4/frame 0x206e3b58 sleepq_switch(1c80,22c66000,0,0,1d72e54,...) at sleepq_switch+0xf7/frame 0x206e3b78 sleepq_catch_signals(206e3be0,f845b8,1891a2d0,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x206e3ba0 sleepq_wait_sig(1891a2d0,0) at sleepq_wait_sig+0xe/frame 0x206e3ba8 _cv_wait_sig(1891a2d0,1881c0c0) at _cv_wait_sig+0x138/frame 0x206e3be0 svc_run_internal(206e3cd4,fa73e8,1881c0c0,206e3ce8,120ed90,...) at svc_run_internal+0xb45/frame 0x206e3cb0 svc_thread_start(1881c0c0,206e3ce8) at svc_thread_start+0xd/frame 0x206e3cb8 fork_exit(120ed90,1881c0c0,206e3ce8,3b,3b,...) at fork_exit+0x68/frame 0x206e3cd4 fork_trampoline() at 0xffc0340e/frame 0x206e3cd4 --- kthread start Tracing command nfsd pid 574 tid 100163 td 0x22c6bc00 sched_switch(22c6bc00,104) at sched_switch+0x4e9/frame 0x206e8b20 mi_switch(104) at mi_switch+0x1c4/frame 0x206e8b58 sleepq_switch(3c80,22c6bc00,0,0,1d74e54,...) at sleepq_switch+0xf7/frame 0x206e8b78 sleepq_catch_signals(206e8be0,f845b8,1891a250,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x206e8ba0 sleepq_wait_sig(1891a250,0) at sleepq_wait_sig+0xe/frame 0x206e8ba8 _cv_wait_sig(1891a250,1881c0c0) at _cv_wait_sig+0x138/frame 0x206e8be0 svc_run_internal(206e8cd4,fa73e8,1881c0c0,206e8ce8,120ed90,...) at svc_run_internal+0xb45/frame 0x206e8cb0 svc_thread_start(1881c0c0,206e8ce8) at svc_thread_start+0xd/frame 0x206e8cb8 fork_exit(120ed90,1881c0c0,206e8ce8,3b,3b,...) at fork_exit+0x68/frame 0x206e8cd4 fork_trampoline() at 0xffc0340e/frame 0x206e8cd4 --- kthread start Tracing command nfsd pid 574 tid 100164 td 0x22c6b880 sched_switch(22c6b880,104) at sched_switch+0x4e9/frame 0x206edb20 mi_switch(104) at mi_switch+0x1c4/frame 0x206edb58 sleepq_switch(26c0,22c6b880,0,0,1d73894,...) at sleepq_switch+0xf7/frame 0x206edb78 sleepq_catch_signals(206edbe0,f845b8,22c64bd0,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x206edba0 sleepq_wait_sig(22c64bd0,0) at sleepq_wait_sig+0xe/frame 0x206edba8 _cv_wait_sig(22c64bd0,1881c0c0) at _cv_wait_sig+0x138/frame 0x206edbe0 svc_run_internal(206edcd4,fa73e8,1881c0c0,206edce8,120ed90,...) at svc_run_internal+0xb45/frame 0x206edcb0 svc_thread_start(1881c0c0,206edce8) at svc_thread_start+0xd/frame 0x206edcb8 fork_exit(120ed90,1881c0c0,206edce8,3b,3b,...) at fork_exit+0x68/frame 0x206edcd4 fork_trampoline() at 0xffc0340e/frame 0x206edcd4 --- kthread start Tracing command nfsd pid 574 tid 100165 td 0x22c6b500 sched_switch(22c6b500,104) at sched_switch+0x4e9/frame 0x206f2b20 mi_switch(104) at mi_switch+0x1c4/frame 0x206f2b58 sleepq_switch(1c40,22c6b500,0,0,1d72e14,...) at sleepq_switch+0xf7/frame 0x206f2b78 sleepq_catch_signals(206f2be0,f845b8,1891a1d0,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x206f2ba0 sleepq_wait_sig(1891a1d0,0) at sleepq_wait_sig+0xe/frame 0x206f2ba8 _cv_wait_sig(1891a1d0,1881c0c0) at _cv_wait_sig+0x138/frame 0x206f2be0 svc_run_internal(206f2cd4,fa73e8,1881c0c0,206f2ce8,120ed90,...) at svc_run_internal+0xb45/frame 0x206f2cb0 svc_thread_start(1881c0c0,206f2ce8) at svc_thread_start+0xd/frame 0x206f2cb8 fork_exit(120ed90,1881c0c0,206f2ce8,3b,3b,...) at fork_exit+0x68/frame 0x206f2cd4 fork_trampoline() at 0xffc0340e/frame 0x206f2cd4 --- kthread start Tracing command nfsd pid 574 tid 100166 td 0x22c6b180 sched_switch(22c6b180,104) at sched_switch+0x4e9/frame 0x206f7b20 mi_switch(104) at mi_switch+0x1c4/frame 0x206f7b58 sleepq_switch(3c40,22c6b180,0,0,1d74e14,...) at sleepq_switch+0xf7/frame 0x206f7b78 sleepq_catch_signals(206f7be0,f845b8,1891a150,0,1467ecb,...) at sleepq_catch_signals+0x278/frame 0x206f7ba0 sleepq_wait_sig(1891a150,0) at sleepq_wait_sig+0xe/frame 0x206f7ba8 _cv_wait_sig(1891a150,1881c0c0) at _cv_wait_sig+0x138/frame 0x206f7be0 svc_run_internal(206f7cd4,fa73e8,1881c0c0,206f7ce8,120ed90,...) at svc_run_internal+0xb45/frame 0x206f7cb0 svc_thread_start(1881c0c0,206f7ce8) at svc_thread_start+0xd/frame 0x206f7cb8 fork_exit(120ed90,1881c0c0,206f7ce8,3b,3b,...) at fork_exit+0x68/frame 0x206f7cd4 fork_trampoline() at 0xffc0340e/frame 0x206f7cd4 --- kthread start Tracing command nfsd pid 571 tid 100125 td 0x2004f180 sched_switch(2004f180,104) at sched_switch+0x4e9/frame 0x2060e970 mi_switch(104) at mi_switch+0x1c4/frame 0x2060e9a8 sleepq_switch(640,2004f180,0,0,1d71814,...) at sleepq_switch+0xf7/frame 0x2060e9c8 sleepq_catch_signals(2060ea30,f845b8,b9d3d24,0,13fc2f9,...) at sleepq_catch_signals+0x278/frame 0x2060e9f0 sleepq_wait_sig(b9d3d24,0) at sleepq_wait_sig+0xe/frame 0x2060e9f8 _cv_wait_sig(b9d3d24,b9d3d10) at _cv_wait_sig+0x138/frame 0x2060ea30 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x2060ea4c kern_select(2004f180,7,ffbfec08,0,0,0,20) at kern_select+0x8e5/frame 0x2060ebd4 sys_select(2004f180,2004f418,2004f180,9b09054,2004f180,...) at sys_select+0x41/frame 0x2060ec08 syscall(2060ece8,3b,3b,3b,ffbfec88,...) at syscall+0x1b8/frame 0x2060ecdc Xint0x80_syscall() at 0xffc033f9/frame 0x2060ecdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe75c --- Tracing command mountd pid 565 tid 100123 td 0x202f5880 sched_switch(202f5880,104) at sched_switch+0x4e9/frame 0x20604930 mi_switch(104) at mi_switch+0x1c4/frame 0x20604968 sleepq_switch(3200,1da0c7c,0,0,1d743d4,...) at sleepq_switch+0xf7/frame 0x20604988 sleepq_catch_signals(e4,202f5880,1a743c8,202f5880,206049e4,...) at sleepq_catch_signals+0x278/frame 0x206049b0 sleepq_timedwait_sig(9bf6ca4,0) at sleepq_timedwait_sig+0x17/frame 0x206049e0 _cv_timedwait_sig_sbt(9bf6ca4,9bf6c90,9a89f922,9717,4000000,0,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x20604a1c seltdwait(9717,4000000,0) at seltdwait+0x5e/frame 0x20604a4c kern_select(202f5880,a,ffbfe878,0,0,20604bf8,20) at kern_select+0x8e5/frame 0x20604bd4 sys_select(202f5880,202f5b18,202f5880,1fc2b000,b8565b8,...) at sys_select+0x41/frame 0x20604c08 syscall(20604ce8,3b,3b,3b,ffbfe8f8,...) at syscall+0x1b8/frame 0x20604cdc Xint0x80_syscall() at 0xffc033f9/frame 0x20604cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- Tracing command nfsuserd pid 559 tid 100133 td 0x22c25c00 sched_switch(22c25c00,104) at sched_switch+0x4e9/frame 0x20646930 mi_switch(104) at mi_switch+0x1c4/frame 0x20646968 sleepq_switch(1380,1da10f4,0,0,1d72554,...) at sleepq_switch+0xf7/frame 0x20646988 sleepq_catch_signals(e4,22c25c00,1a743c8,22c25c00,206469e4,...) at sleepq_catch_signals+0x278/frame 0x206469b0 sleepq_timedwait_sig(18826a24,0) at sleepq_timedwait_sig+0x17/frame 0x206469e0 _cv_timedwait_sig_sbt(18826a24,18826a10,ba6f1ffe,9732,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x20646a1c seltdwait(9732,e0000000,1) at seltdwait+0x5e/frame 0x20646a4c kern_select(22c25c00,4,ffbfdb88,0,0,20646bf8,20) at kern_select+0x8e5/frame 0x20646bd4 sys_select(22c25c00,22c25e98,22c25c00,22c22a20,b8565b8,...) at sys_select+0x41/frame 0x20646c08 syscall(20646ce8,3b,3b,3b,ffbfdb88,...) at syscall+0x1b8/frame 0x20646cdc Xint0x80_syscall() at 0xffc033f9/frame 0x20646cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe75c --- Tracing command nfsuserd pid 557 tid 100132 td 0x2004e000 sched_switch(2004e000,104) at sched_switch+0x4e9/frame 0x20641930 mi_switch(104) at mi_switch+0x1c4/frame 0x20641968 sleepq_switch(2680,1da0e34,0,0,1d73854,...) at sleepq_switch+0xf7/frame 0x20641988 sleepq_catch_signals(e4,2004e000,1a743c8,2004e000,206419e4,...) at sleepq_catch_signals+0x278/frame 0x206419b0 sleepq_timedwait_sig(1fc0fe64,0) at sleepq_timedwait_sig+0x17/frame 0x206419e0 _cv_timedwait_sig_sbt(1fc0fe64,1fc0fe50,9bf82f99,9731,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x20641a1c seltdwait(9731,e0000000,1) at seltdwait+0x5e/frame 0x20641a4c kern_select(2004e000,4,ffbfdb88,0,0,20641bf8,20) at kern_select+0x8e5/frame 0x20641bd4 sys_select(2004e000,2004e298,2004e000,9b08000,2004e000,...) at sys_select+0x41/frame 0x20641c08 syscall(20641ce8,3b,3b,3b,ffbfdb88,...) at syscall+0x1b8/frame 0x20641cdc Xint0x80_syscall() at 0xffc033f9/frame 0x20641cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe75c --- Tracing command nfsuserd pid 556 tid 100131 td 0x2004e380 sched_switch(2004e380,104) at sched_switch+0x4e9/frame 0x2063c930 mi_switch(104) at mi_switch+0x1c4/frame 0x2063c968 sleepq_switch(380,1da1464,0,0,1d71554,...) at sleepq_switch+0xf7/frame 0x2063c988 sleepq_catch_signals(e4,2004e380,1a743c8,2004e380,2063c9e4,...) at sleepq_catch_signals+0x278/frame 0x2063c9b0 sleepq_timedwait_sig(18826a64,0) at sleepq_timedwait_sig+0x17/frame 0x2063c9e0 _cv_timedwait_sig_sbt(18826a64,18826a50,5f8e2f77,9730,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x2063ca1c seltdwait(9730,e0000000,1) at seltdwait+0x5e/frame 0x2063ca4c kern_select(2004e380,4,ffbfdb88,0,0,2063cbf8,20) at kern_select+0x8e5/frame 0x2063cbd4 sys_select(2004e380,2004e618,2004e380,9b08344,2004e380,...) at sys_select+0x41/frame 0x2063cc08 syscall(2063cce8,3b,3b,3b,ffbfdb88,...) at syscall+0x1b8/frame 0x2063ccdc Xint0x80_syscall() at 0xffc033f9/frame 0x2063ccdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe75c --- Tracing command nfsuserd pid 555 tid 100130 td 0x2004e700 sched_switch(2004e700,104) at sched_switch+0x4e9/frame 0x20637930 mi_switch(104) at mi_switch+0x1c4/frame 0x20637968 sleepq_switch(1680,1da11a4,0,0,1d72854,...) at sleepq_switch+0xf7/frame 0x20637988 sleepq_catch_signals(e4,2004e700,1a743c8,2004e700,206379e4,...) at sleepq_catch_signals+0x278/frame 0x206379b0 sleepq_timedwait_sig(1fc0fea4,0) at sleepq_timedwait_sig+0x17/frame 0x206379e0 _cv_timedwait_sig_sbt(1fc0fea4,1fc0fe90,a6abe4e9,971a,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x20637a1c seltdwait(971a,e0000000,1) at seltdwait+0x5e/frame 0x20637a4c kern_select(2004e700,4,ffbfdb88,0,0,20637bf8,20) at kern_select+0x8e5/frame 0x20637bd4 sys_select(2004e700,2004e998,2004e700,9b08688,22ff5b8,...) at sys_select+0x41/frame 0x20637c08 syscall(20637ce8,3b,3b,3b,ffbfdb88,...) at syscall+0x1b8/frame 0x20637cdc Xint0x80_syscall() at 0xffc033f9/frame 0x20637cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe75c --- Tracing command nfsuserd pid 554 tid 100129 td 0x202f4e00 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1a5e62c,1a5e644,3ae03f14,103d33c,1a5e644,...) at db_trace_self_wrapper+0x2a/frame 0x3ae03ed0 kdb_backtrace(3ae03ff8,13d05e1,1430c59,0,3ae04030,...) at kdb_backtrace+0x2e/frame 0x3ae03f30 kdb_reenter(1430c59,0,3ae04030,3ae0400c,28409000,...) at kdb_reenter+0x29/frame 0x3ae03f38 trap(3ae04004,8,28,28,ffffffff,...) at trap+0xd1/frame 0x3ae03ff8 calltrap() at 0xffc0319f/frame 0x3ae03ff8 --- trap 0xc, eip = 0x13ac85a, esp = 0x3ae04044, ebp = 0x3ae04050 --- db_trace_thread(202f4e00,ffffffff) at db_trace_thread+0x2a/frame 0x3ae04050 _db_stack_trace_all(1028685,0,ffffffff,3ae040dc) at _db_stack_trace_all+0x93/frame 0x3ae040b0 db_command(1ae7672) at db_command+0x26a/frame 0x3ae04164 db_script_exec(bd2daf,0,3ae0420c,3ae041f0) at db_script_exec+0x1a6/frame 0x3ae041c4 db_command(3ae042c8,a06536,0,0,2af64700,...) at db_command+0x26a/frame 0x3ae04278 db_command_loop(0,0,2af64700,3ae04280,3ae042c8,...) at db_command_loop+0x3c/frame 0x3ae04280 db_trap(3,0) at db_trap+0xf6/frame 0x3ae042c8 kdb_trap(3,0,3ae043cc) at kdb_trap+0x109/frame 0x3ae042f4 trap(3ae043cc,8,28,28,100,...) at trap+0x382/frame 0x3ae043c0 calltrap() at 0xffc0319f/frame 0x3ae043c0 --- trap 0x3, eip = 0x1028685, esp = 0x3ae0440c, ebp = 0x3ae04410 --- kdb_enter(14df98a,14df98a) at kdb_enter+0x35/frame 0x3ae04410 vpanic(146f152,3ae04458,3ae04458,3ae04488,f39752,...) at vpanic+0x137/frame 0x3ae04438 panic(146f152,0,18d7ef80,fc50a7,18d7ae00,...) at panic+0x14/frame 0x3ae0444c g_io_deliver(261e8cf0,0) at g_io_deliver+0x322/frame 0x3ae04488 g_disk_done(24029678,24029678) at g_disk_done+0xf0/frame 0x3ae044bc cam_iosched_queue_work(18813700,24029678) at cam_iosched_queue_work+0xcb/frame 0x3ae044d8 adastrategy(24029678,0,114,2c4,b9aa800,...) at adastrategy+0x50/frame 0x3ae044f4 g_disk_start(261e8cf0,18d7ef80,18d7ae00,f38877,b9aa800,...) at g_disk_start+0x4de/frame 0x3ae04550 g_io_request(261e8cf0,18d7ef80,33901fec,2b3b598,3ae045f8,...) at g_io_request+0x2aa/frame 0x3ae04578 g_part_start(339019b4,18d88dc0,9ac4900,f38877,b9aa800,...) at g_part_start+0x26f/frame 0x3ae045e8 g_io_request(339019b4,18d88dc0,1d8fb8c,1,41000000,...) at g_io_request+0x2aa/frame 0x3ae04610 g_part_start(2d39fbdc,1,18d8ed00,18d94b80,30ebd600,...) at g_part_start+0x26f/frame 0x3ae04680 g_io_request(2d39fbdc,b8e0400,b8e0400) at g_io_request+0x2aa/frame 0x3ae046a8 g_vfs_strategy(20051828,30ebd600) at g_vfs_strategy+0xe8/frame 0x3ae046c8 softdep_request_cleanup(20165000,2a343758,29a3e000,4) at softdep_request_cleanup+0x2b7/frame 0x3ae04720 ffs_alloc(2aafb4fc,0,0,6a4908,0,1000,2050000,29a3e000,3ae047d8) at ffs_alloc+0x2ad/frame 0x3ae0476c ffs_balloc_ufs2(2a343758,0,0,1000,29a3e000,2050000,3ae04900) at ffs_balloc_ufs2+0xd8b/frame 0x3ae048a0 ffs_write(3ae04a08) at ffs_write+0x216/frame 0x3ae04924 VOP_WRITE_APV(1aabe80,3ae04a08) at VOP_WRITE_APV+0xdf/frame 0x3ae049e0 vn_write(2a29ec30,3ae04bd0,2881e500,1,2af64700) at vn_write+0x305/frame 0x3ae04a3c vn_io_fault_doio(2af64700) at vn_io_fault_doio+0x3a/frame 0x3ae04a84 vn_io_fault1(2af64700) at vn_io_fault1+0x164/frame 0x3ae04b34 vn_io_fault(2a29ec30,3ae04bd0,2881e500,0,2af64700) at vn_io_fault+0x1c4/frame 0x3ae04b70 dofilewrite(2a29ec30,3ae04bd0,ffffffff,ffffffff,0) at dofilewrite+0x71/frame 0x3ae04ba4 sys_write(2af64700,2af64998,2af64700,28409000,2af64700,...) at sys_write+0x78/frame 0x3ae04c08 syscall(3ae04ce8,3b,3b,3b,1000,...) at syscall+0x1b8/frame 0x3ae04cdc Xint0x80_syscall() at 0xffc033f9/frame 0x3ae04cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- Tracing command watchdogd pid 511 tid 100121 td 0x18d3f000 sched_switch(18d3f000,104) at sched_switch+0x4e9/frame 0x1fb2aa70 mi_switch(104) at mi_switch+0x1c4/frame 0x1fb2aaa8 sleepq_switch(d00,3,6c,0,1d71ed4,...) at sleepq_switch+0xf7/frame 0x1fb2aac8 sleepq_catch_signals(14e4b25,183,18d3f000,100,1d5f8cc,...) at sleepq_catch_signals+0x278/frame 0x1fb2aaf0 sleepq_timedwait_sig(1d5f8cc,6c,0,13fc2f9,32e,...) at sleepq_timedwait_sig+0x17/frame 0x1fb2ab1c _sleep(1d5f8cc,0,16c,14f7fc3,bf0b1576,9728,c0000000,3,200) at _sleep+0x21a/frame 0x1fb2ab5c kern_clock_nanosleep(18d3f000,0,0,1fb2abe0,1fb2abe8) at kern_clock_nanosleep+0x1b7/frame 0x1fb2abc4 sys_nanosleep(18d3f000,18d3f298,18d3f000,1fc2b688,22fe040,...) at sys_nanosleep+0x37/frame 0x1fb2ac08 syscall(1fb2ace8,3b,3b,3b,3c,...) at syscall+0x1b8/frame 0x1fb2acdc Xint0x80_syscall() at 0xffc033f9/frame 0x1fb2acdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- Tracing command rpcbind pid 422 tid 100107 td 0x1880f700 sched_switch(1880f700,104) at sched_switch+0x4e9/frame 0x1fab0960 mi_switch(104) at mi_switch+0x1c4/frame 0x1fab0998 sleepq_switch(2380,1da0f94,0,0,1d73554,...) at sleepq_switch+0xf7/frame 0x1fab09b8 sleepq_catch_signals(e4,1880f700,1a743c8,1880f700,1fab0a14,...) at sleepq_catch_signals+0x278/frame 0x1fab09e0 sleepq_timedwait_sig(18826ae4,0) at sleepq_timedwait_sig+0x17/frame 0x1fab0a10 _cv_timedwait_sig_sbt(18826ae4,18826ad0,23f832b3,971e,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x1fab0a4c seltdwait(971e,e0000000,1) at seltdwait+0x5e/frame 0x1fab0a7c kern_poll(1880f700,ffbfcd58,8,1fab0bf0,0) at kern_poll+0x4a4/frame 0x1fab0bd4 sys_poll(1880f700,1880f998,1880f700,1880b9cc,1880f700,...) at sys_poll+0x50/frame 0x1fab0c08 syscall(1fab0ce8,3b,3b,3b,40,...) at syscall+0x1b8/frame 0x1fab0cdc Xint0x80_syscall() at 0xffc033f9/frame 0x1fab0cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe804 --- Tracing command syslogd pid 413 tid 100111 td 0x18d3fa80 sched_switch(18d3fa80,104) at sched_switch+0x4e9/frame 0x1faf0970 mi_switch(104) at mi_switch+0x1c4/frame 0x1faf09a8 sleepq_switch(2640,18d3fa80,0,0,1d73814,...) at sleepq_switch+0xf7/frame 0x1faf09c8 sleepq_catch_signals(1faf0a30,f845b8,b9d3da4,0,13fc2f9,...) at sleepq_catch_signals+0x278/frame 0x1faf09f0 sleepq_wait_sig(b9d3da4,0) at sleepq_wait_sig+0xe/frame 0x1faf09f8 _cv_wait_sig(b9d3da4,b9d3d90) at _cv_wait_sig+0x138/frame 0x1faf0a30 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x1faf0a4c kern_select(18d3fa80,a,20686000,0,0,0,20) at kern_select+0x8e5/frame 0x1faf0bd4 sys_select(18d3fa80,18d3fd18,18d3fa80,1fc2c054,a4c795a,...) at sys_select+0x41/frame 0x1faf0c08 syscall(1faf0ce8,3b,3b,3b,20686000,...) at syscall+0x1b8/frame 0x1faf0cdc Xint0x80_syscall() at 0xffc033f9/frame 0x1faf0cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d98, esp = 0xffc0bfe8, ebp = 0xffbfecbc --- Tracing command devd pid 341 tid 100122 td 0x202f5c00 sched_switch(202f5c00,104) at sched_switch+0x4e9/frame 0x1fb53930 mi_switch(104) at mi_switch+0x1c4/frame 0x1fb53968 sleepq_switch(1640,1da140c,0,0,1d72814,...) at sleepq_switch+0xf7/frame 0x1fb53988 sleepq_catch_signals(e4,202f5c00,1a743c8,202f5c00,1fb539e4,...) at sleepq_catch_signals+0x278/frame 0x1fb539b0 sleepq_timedwait_sig(b9d3d64,0) at sleepq_timedwait_sig+0x17/frame 0x1fb539e0 _cv_timedwait_sig_sbt(b9d3d64,b9d3d50,de1ccbb3,973b,c0000000,3,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x1fb53a1c seltdwait(973b,c0000000,3) at seltdwait+0x5e/frame 0x1fb53a4c kern_select(202f5c00,6,ffbfcd38,0,0,1fb53bf8,20) at kern_select+0x8e5/frame 0x1fb53bd4 sys_select(202f5c00,202f5e98,202f5c00,1fc2b344,b8505b8,...) at sys_select+0x41/frame 0x1fb53c08 syscall(1fb53ce8,3b,3b,3b,ffbfcdb8,...) at syscall+0x1b8/frame 0x1fb53cdc Xint0x80_syscall() at 0xffc033f9/frame 0x1fb53cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d98, esp = 0xffc0bfe8, ebp = 0xffbfecbc --- Tracing command syncer pid 24 tid 100104 td 0x18923700 sched_switch(18923700,104) at sched_switch+0x4e9/frame 0x1fa91b88 mi_switch(104) at mi_switch+0x1c4/frame 0x1fa91bc0 sleepq_switch(1a743c8,18923700,1fa91c0c,fc4aae,1dd07a8,...) at sleepq_switch+0xf7/frame 0x1fa91be0 sleepq_timedwait(1dd07b8,0) at sleepq_timedwait+0x3f/frame 0x1fa91c08 _cv_timedwait_sbt(1dd07b8,1dd0798,fffffed8,0,0,0,100) at _cv_timedwait_sbt+0x157/frame 0x1fa91c44 sched_sync(0,1fa91ce8) at sched_sync+0x676/frame 0x1fa91cb8 fork_exit(10b4cc0,0,1fa91ce8,0,0,...) at fork_exit+0x68/frame 0x1fa91cd4 fork_trampoline() at 0xffc0340e/frame 0x1fa91cd4 --- kthread start Tracing command vnlru pid 23 tid 100103 td 0x18923a80 sched_switch(18923a80,104) at sched_switch+0x4e9/frame 0x1fa8cb70 mi_switch(104) at mi_switch+0x1c4/frame 0x1fa8cba8 sleepq_switch(18923a80,0,182cb9cc,1fa8cc2c,18923a80,...) at sleepq_switch+0xf7/frame 0x1fa8cbc8 sleepq_timedwait(182cb9cc,60,1fa8cc20,fc57b5,1483e64,...) at sleepq_timedwait+0x3f/frame 0x1fa8cbec _sleep(182cb9cc,1a8d380,260,150c53f,fffffed8,0,0,0,100) at _sleep+0x22a/frame 0x1fa8cc2c vnlru_proc(0,1fa8cce8) at vnlru_proc+0xac1/frame 0x1fa8ccb8 fork_exit(10b3d50,0,1fa8cce8,0,0,...) at fork_exit+0x68/frame 0x1fa8ccd4 fork_trampoline() at 0xffc0340e/frame 0x1fa8ccd4 --- kthread start Tracing command bufdaemon pid 22 tid 100099 td 0x18923e00 sched_switch(18923e00,104) at sched_switch+0x4e9/frame 0xb5b1bb8 mi_switch(104) at mi_switch+0x1c4/frame 0xb5b1bf0 sleepq_switch(183,18923e00,0,1dcff04,b5b1c78,...) at sleepq_switch+0xf7/frame 0xb5b1c10 sleepq_timedwait(1dcff04,54,a4c795a,10000,13fc75a,...) at sleepq_timedwait+0x3f/frame 0xb5b1c38 _sleep(1dcff04,1a81040,54,142c597,1999997c,0,0,0,100) at _sleep+0x22a/frame 0xb5b1c78 buf_daemon(0,b5b1ce8) at buf_daemon+0xf5/frame 0xb5b1cb8 fork_exit(108c940,0,b5b1ce8,0,0,...) at fork_exit+0x68/frame 0xb5b1cd4 fork_trampoline() at 0xffc0340e/frame 0xb5b1cd4 --- kthread start Tracing command bufdaemon pid 22 tid 100100 td 0x1880fa80 sched_switch(1880fa80,104) at sched_switch+0x4e9/frame 0xb5d6bb0 mi_switch(104) at mi_switch+0x1c4/frame 0xb5d6be8 sleepq_switch(1880fa80,0,1a82340,b5d6c6c,1880fa80,...) at sleepq_switch+0xf7/frame 0xb5d6c08 sleepq_timedwait(1a82340,5c,b608da0,a4c795a,13fc75a,...) at sleepq_timedwait+0x3f/frame 0xb5d6c2c _sleep(1a82340,1a82280,25c,142fff2,fffffed8,0,0,0,100) at _sleep+0x22a/frame 0xb5d6c6c bufspace_daemon(1a81140,b5d6ce8) at bufspace_daemon+0x2cb/frame 0xb5d6cb8 fork_exit(108cc50,1a81140,b5d6ce8,0,0,...) at fork_exit+0x68/frame 0xb5d6cd4 fork_trampoline() at 0xffc0340e/frame 0xb5d6cd4 --- kthread start Tracing command bufdaemon pid 22 tid 100114 td 0x9b01000 sched_switch(9b01000,104) at sched_switch+0x4e9/frame 0x1faffbc0 mi_switch(104) at mi_switch+0x1c4/frame 0x1faffbf8 sleepq_switch(9b01000,0,9bd8684,1faffc7c,9b01000,...) at sleepq_switch+0xf7/frame 0x1faffc18 sleepq_timedwait(9bd8684,54,1faffc70,fe0530,14680b7,...) at sleepq_timedwait+0x3f/frame 0x1faffc3c _sleep(9bd8684,9bd8600,54,145e7af,7fffff6c,0,0,0,100) at _sleep+0x22a/frame 0x1faffc7c softdep_flush(1fe1a040,1faffce8) at softdep_flush+0x158/frame 0x1faffcb8 fork_exit(124b080,1fe1a040,1faffce8,0,0,...) at fork_exit+0x68/frame 0x1faffcd4 fork_trampoline() at 0xffc0340e/frame 0x1faffcd4 --- kthread start Tracing command bufdaemon pid 22 tid 100115 td 0x2004fc00 sched_switch(2004fc00,104) at sched_switch+0x4e9/frame 0x1fb04bc0 mi_switch(104) at mi_switch+0x1c4/frame 0x1fb04bf8 sleepq_switch(2004fc00,0,9bd8484,1fb04c7c,2004fc00,...) at sleepq_switch+0xf7/frame 0x1fb04c18 sleepq_timedwait(9bd8484,54,1fb04c70,fe0530,14680b7,...) at sleepq_timedwait+0x3f/frame 0x1fb04c3c _sleep(9bd8484,9bd8400,54,145e7af,7fffff6c,0,0,0,100) at _sleep+0x22a/frame 0x1fb04c7c softdep_flush(18cc9580,1fb04ce8) at softdep_flush+0x158/frame 0x1fb04cb8 fork_exit(124b080,18cc9580,1fb04ce8,0,0,...) at fork_exit+0x68/frame 0x1fb04cd4 fork_trampoline() at 0xffc0340e/frame 0x1fb04cd4 --- kthread start Tracing command bufdaemon pid 22 tid 100116 td 0x2004f880 sched_switch(2004f880,104) at sched_switch+0x4e9/frame 0x1fb09bc0 mi_switch(104) at mi_switch+0x1c4/frame 0x1fb09bf8 sleepq_switch(2004f880,0,9bd8284,1fb09c7c,2004f880,...) at sleepq_switch+0xf7/frame 0x1fb09c18 sleepq_timedwait(9bd8284,54,1fb09c70,fe0530,14680b7,...) at sleepq_timedwait+0x3f/frame 0x1fb09c3c _sleep(9bd8284,9bd8200,54,145e7af,7fffff6c,0,0,0,100) at _sleep+0x22a/frame 0x1fb09c7c softdep_flush(18cc8ac0,1fb09ce8) at softdep_flush+0x158/frame 0x1fb09cb8 fork_exit(124b080,18cc8ac0,1fb09ce8,0,0,...) at fork_exit+0x68/frame 0x1fb09cd4 fork_trampoline() at 0xffc0340e/frame 0x1fb09cd4 --- kthread start Tracing command bufdaemon pid 22 tid 100117 td 0x2004f500 sched_switch(2004f500,104) at sched_switch+0x4e9/frame 0x1fb0e938 mi_switch(104) at mi_switch+0x1c4/frame 0x1fb0e970 sleepq_switch(b7684c8,5c) at sleepq_switch+0xf7/frame 0x1fb0e990 _sleep(b7684c8,9a2d5d4,5c,142c52a,0,0,0,0,100) at _sleep+0x2f3/frame 0x1fb0e9d0 bufwait(b7684c8) at bufwait+0x6b/frame 0x1fb0ea0c breadn_flags(20051758,5ee8120,0,5ee8120,0,4000,0,0,0,0,80,0,1fb0eae0) at breadn_flags+0x1b0/frame 0x1fb0ea30 ffs_breadz(1ff30700,20051758,5ee8120,0,5ee8120,0,4000,0,0,0,0,0,0,1fb0eae0) at ffs_breadz+0x39/frame 0x1fb0ea78 indir_trunc(0,ffff87f4,ffffffff,20165000,2c94f980,...) at indir_trunc+0x794/frame 0x1fb0eb44 indir_trunc(0,fffff7f3,ffffffff) at indir_trunc+0x49b/frame 0x1fb0ebe8 handle_workitem_freeblocks(fc50a7,3,2004f500,1ff30700,a,...) at handle_workitem_freeblocks+0x230/frame 0x1fb0ec20 process_worklist_item(200) at process_worklist_item+0x25c/frame 0x1fb0ec78 softdep_process_worklist(2004f500,2004f500,182cbd10,124b080,1fb0ecd4,...) at softdep_process_worklist+0xb3/frame 0x1fb0eca0 softdep_flush(18cc8000,1fb0ece8) at softdep_flush+0xce/frame 0x1fb0ecb8 fork_exit(124b080,18cc8000,1fb0ece8,0,0,...) at fork_exit+0x68/frame 0x1fb0ecd4 fork_trampoline() at 0xffc0340e/frame 0x1fb0ecd4 --- kthread start Tracing command vmdaemon pid 21 tid 100098 td 0x18924180 sched_switch(18924180,104) at sched_switch+0x4e9/frame 0xb5acba8 mi_switch(104) at mi_switch+0x1c4/frame 0xb5acbdc sleepq_switch(1de7270,74) at sleepq_switch+0xf7/frame 0xb5acbfc _sleep(1de7270,1de725c,74,142c590,0,0,0,0,100) at _sleep+0x2f3/frame 0xb5acc3c vm_daemon(0,b5acce8) at vm_daemon+0x7f/frame 0xb5accb8 fork_exit(12b6090,0,b5acce8,0,0,...) at fork_exit+0x68/frame 0xb5accd4 fork_trampoline() at 0xffc0340e/frame 0xb5accd4 --- kthread start Tracing command pagedaemon pid 20 tid 100097 td 0x18924500 sched_switch(18924500,104) at sched_switch+0x4e9/frame 0xb5a7a58 mi_switch(104) at mi_switch+0x1c4/frame 0xb5a7a90 sleepq_switch(18924500,0,1de62bc,b5a7b14,18924500,...) at sleepq_switch+0xf7/frame 0xb5a7ab0 sleepq_timedwait(1de62bc,54,10000,a4c795a,144b10b,...) at sleepq_timedwait+0x3f/frame 0xb5a7ad4 _sleep(1de62bc,1de5fc0,254,142c590,1999997c,0,0,0,100) at _sleep+0x22a/frame 0xb5a7b14 vm_pageout_worker(0,0,18924500,182cc398,18924500,...) at vm_pageout_worker+0x26a/frame 0xb5a7c94 vm_pageout(0,b5a7ce8) at vm_pageout+0x18a/frame 0xb5a7cb8 fork_exit(12aacd0,0,b5a7ce8,0,0,...) at fork_exit+0x68/frame 0xb5a7cd4 fork_trampoline() at 0xffc0340e/frame 0xb5a7cd4 --- kthread start Tracing command pagedaemon pid 20 tid 100101 td 0x18d40c00 sched_switch(18d40c00,104) at sched_switch+0x4e9/frame 0xb5fba80 mi_switch(104) at mi_switch+0x1c4/frame 0xb5fbab4 sleepq_switch(1de62c8,54) at sleepq_switch+0xf7/frame 0xb5fbad4 _sleep(1de62c8,1de5f00,54,1521946,0,0,0,0,100) at _sleep+0x2f3/frame 0xb5fbb14 vm_pageout_laundry_worker(0,b5fbce8) at vm_pageout_laundry_worker+0xc9b/frame 0xb5fbcb8 fork_exit(12abf80,0,b5fbce8,0,0,...) at fork_exit+0x68/frame 0xb5fbcd4 fork_trampoline() at 0xffc0340e/frame 0xb5fbcd4 --- kthread start Tracing command pagedaemon pid 20 tid 100102 td 0x18d40880 sched_switch(18d40880,104) at sched_switch+0x4e9/frame 0x1dfe7bc0 mi_switch(104) at mi_switch+0x1c4/frame 0x1dfe7bf8 sleepq_switch(18d40880,0,1de5b0c,1dfe7c44,10368eb,...) at sleepq_switch+0xf7/frame 0x1dfe7c18 sleepq_timedwait(1284ce0,54,1836cf9,1dfe7c8c,147bedf,...) at sleepq_timedwait+0x3f/frame 0x1dfe7c40 _sleep(1284ce0,1de5b0c,54,148f085,fffffed8,0,0,0,100) at _sleep+0x22a/frame 0x1dfe7c80 uma_reclaim_worker(0,1dfe7ce8) at uma_reclaim_worker+0xe4/frame 0x1dfe7cb8 fork_exit(12851e0,0,1dfe7ce8,0,0,...) at fork_exit+0x68/frame 0x1dfe7cd4 fork_trampoline() at 0xffc0340e/frame 0x1dfe7cd4 --- kthread start Tracing command rand_harvestq pid 19 tid 100091 td 0x1880fe00 sched_switch(1880fe00,104) at sched_switch+0x4e9/frame 0xb589b80 mi_switch(104) at mi_switch+0x1c4/frame 0xb589bb8 sleepq_switch(1880fe00,0,1bad37c,b589c3c,1880fe00,...) at sleepq_switch+0xf7/frame 0xb589bd8 sleepq_timedwait(1bad37c,0,1bb2474,0,b589c38,...) at sleepq_timedwait+0x3f/frame 0xb589bfc _sleep(1bad37c,0,0,142fff2,19999999,0,0,0,4) at _sleep+0x22a/frame 0xb589c3c random_kthread(0,b589ce8) at random_kthread+0x50/frame 0xb589cb8 fork_exit(d46510,0,b589ce8,0,0,...) at fork_exit+0x68/frame 0xb589cd4 fork_trampoline() at 0xffc0340e/frame 0xb589cd4 --- kthread start Tracing command soaiod4 pid 18 tid 100090 td 0x18810180 sched_switch(18810180,104) at sched_switch+0x4e9/frame 0xb57fbc0 mi_switch(104) at mi_switch+0x1c4/frame 0xb57fbf8 sleepq_switch(18810180,0,1dcfa68,b57fc7c,18810180,...) at sleepq_switch+0xf7/frame 0xb57fc18 sleepq_timedwait(1dcfa68,0,283a58c8,436351,14aa42c,...) at sleepq_timedwait+0x3f/frame 0xb57fc3c _sleep(1dcfa68,1dcfa48,0,142fff2,ffffdd50,1d,0,0,100) at _sleep+0x22a/frame 0xb57fc7c soaio_kproc_loop(4,b57fce8) at soaio_kproc_loop+0x148/frame 0xb57fcb8 fork_exit(1054460,4,b57fce8,0,0,...) at fork_exit+0x68/frame 0xb57fcd4 fork_trampoline() at 0xffc0340e/frame 0xb57fcd4 --- kthread start Tracing command soaiod3 pid 17 tid 100089 td 0x18810500 sched_switch(18810500,104) at sched_switch+0x4e9/frame 0xb57abc0 mi_switch(104) at mi_switch+0x1c4/frame 0xb57abf8 sleepq_switch(18810500,0,1dcfa68,b57ac7c,18810500,...) at sleepq_switch+0xf7/frame 0xb57ac18 sleepq_timedwait(1dcfa68,0,0,1436351,14aa42c,...) at sleepq_timedwait+0x3f/frame 0xb57ac3c _sleep(1dcfa68,1dcfa48,0,142fff2,ffffdd50,1d,0,0,100) at _sleep+0x22a/frame 0xb57ac7c soaio_kproc_loop(3,b57ace8) at soaio_kproc_loop+0x148/frame 0xb57acb8 fork_exit(1054460,3,b57ace8,0,0,...) at fork_exit+0x68/frame 0xb57acd4 fork_trampoline() at 0xffc0340e/frame 0xb57acd4 --- kthread start Tracing command soaiod2 pid 16 tid 100088 td 0x18810880 sched_switch(18810880,104) at sched_switch+0x4e9/frame 0xb575bc0 mi_switch(104) at mi_switch+0x1c4/frame 0xb575bf8 sleepq_switch(18810880,0,1dcfa68,b575c7c,18810880,...) at sleepq_switch+0xf7/frame 0xb575c18 sleepq_timedwait(1dcfa68,0,0,1436351,14aa42c,...) at sleepq_timedwait+0x3f/frame 0xb575c3c _sleep(1dcfa68,1dcfa48,0,142fff2,ffffdd50,1d,0,0,100) at _sleep+0x22a/frame 0xb575c7c soaio_kproc_loop(2,b575ce8) at soaio_kproc_loop+0x148/frame 0xb575cb8 fork_exit(1054460,2,b575ce8,0,0,...) at fork_exit+0x68/frame 0xb575cd4 fork_trampoline() at 0xffc0340e/frame 0xb575cd4 --- kthread start Tracing command soaiod1 pid 9 tid 100087 td 0x18810c00 sched_switch(18810c00,104) at sched_switch+0x4e9/frame 0xb570bc0 mi_switch(104) at mi_switch+0x1c4/frame 0xb570bf8 sleepq_switch(18810c00,0,1dcfa68,b570c7c,18810c00,...) at sleepq_switch+0xf7/frame 0xb570c18 sleepq_timedwait(1dcfa68,0,1d4ce48,b570cb8,14aa42c,...) at sleepq_timedwait+0x3f/frame 0xb570c3c _sleep(1dcfa68,1dcfa48,0,142fff2,ffffdd50,1d,0,0,100) at _sleep+0x22a/frame 0xb570c7c soaio_kproc_loop(1,b570ce8) at soaio_kproc_loop+0x148/frame 0xb570cb8 fork_exit(1054460,1,b570ce8,0,0,...) at fork_exit+0x68/frame 0xb570cd4 fork_trampoline() at 0xffc0340e/frame 0xb570cd4 --- kthread start Tracing command fdc0 pid 8 tid 100084 td 0x18314180 sched_switch(18314180,104) at sched_switch+0x4e9/frame 0xb55dba8 mi_switch(104) at mi_switch+0x1c4/frame 0xb55dbe0 sleepq_switch(9bf743c,5c) at sleepq_switch+0xf7/frame 0xb55dc00 _sleep(9bf743c,9bf74f8,5c,142fff2,0,0,0,0,100) at _sleep+0x2f3/frame 0xb55dc40 fdc_thread(9bf7400,b55dce8) at fdc_thread+0x361/frame 0xb55dcb8 fork_exit(1333300,9bf7400,b55dce8,0,0,...) at fork_exit+0x68/frame 0xb55dcd4 fork_trampoline() at 0xffc0340e/frame 0xb55dcd4 --- kthread start Tracing command usb pid 15 tid 100049 td 0x18037e00 sched_switch(18037e00,104) at sched_switch+0x4e9/frame 0xb4a5c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4a5c3c sleepq_switch(18019aac,0) at sleepq_switch+0xf7/frame 0xb4a5c5c _cv_wait(18019aac,18019cd8) at _cv_wait+0x135/frame 0xb4a5c94 usb_process(18019aa4,b4a5ce8) at usb_process+0x9d/frame 0xb4a5cb8 fork_exit(e39020,18019aa4,b4a5ce8,0,0,...) at fork_exit+0x68/frame 0xb4a5cd4 fork_trampoline() at 0xffc0340e/frame 0xb4a5cd4 --- kthread start Tracing command usb pid 15 tid 100050 td 0x18037a80 sched_switch(18037a80,104) at sched_switch+0x4e9/frame 0xb4aac08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4aac3c sleepq_switch(18019adc,0) at sleepq_switch+0xf7/frame 0xb4aac5c _cv_wait(18019adc,18019cd8) at _cv_wait+0x135/frame 0xb4aac94 usb_process(18019ad4,b4aace8) at usb_process+0x9d/frame 0xb4aacb8 fork_exit(e39020,18019ad4,b4aace8,0,0,...) at fork_exit+0x68/frame 0xb4aacd4 fork_trampoline() at 0xffc0340e/frame 0xb4aacd4 --- kthread start Tracing command usb pid 15 tid 100051 td 0x18037700 sched_switch(18037700,104) at sched_switch+0x4e9/frame 0xb4afc08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4afc3c sleepq_switch(18019b0c,0) at sleepq_switch+0xf7/frame 0xb4afc5c _cv_wait(18019b0c,18019cd8) at _cv_wait+0x135/frame 0xb4afc94 usb_process(18019b04,b4afce8) at usb_process+0x9d/frame 0xb4afcb8 fork_exit(e39020,18019b04,b4afce8,0,0,...) at fork_exit+0x68/frame 0xb4afcd4 fork_trampoline() at 0xffc0340e/frame 0xb4afcd4 --- kthread start Tracing command usb pid 15 tid 100052 td 0x18037380 sched_switch(18037380,104) at sched_switch+0x4e9/frame 0xb4b4c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4b4c3c sleepq_switch(18019b3c,0) at sleepq_switch+0xf7/frame 0xb4b4c5c _cv_wait(18019b3c,18019cd8) at _cv_wait+0x135/frame 0xb4b4c94 usb_process(18019b34,b4b4ce8) at usb_process+0x9d/frame 0xb4b4cb8 fork_exit(e39020,18019b34,b4b4ce8,0,0,...) at fork_exit+0x68/frame 0xb4b4cd4 fork_trampoline() at 0xffc0340e/frame 0xb4b4cd4 --- kthread start Tracing command usb pid 15 tid 100053 td 0x18037000 sched_switch(18037000,104) at sched_switch+0x4e9/frame 0xb4b9c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4b9c3c sleepq_switch(18019b6c,0) at sleepq_switch+0xf7/frame 0xb4b9c5c _cv_wait(18019b6c,18019cd8) at _cv_wait+0x135/frame 0xb4b9c94 usb_process(18019b64,b4b9ce8) at usb_process+0x9d/frame 0xb4b9cb8 fork_exit(e39020,18019b64,b4b9ce8,0,0,...) at fork_exit+0x68/frame 0xb4b9cd4 fork_trampoline() at 0xffc0340e/frame 0xb4b9cd4 --- kthread start Tracing command usb pid 15 tid 100055 td 0x182db880 sched_switch(182db880,104) at sched_switch+0x4e9/frame 0xb4c4c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4c4c3c sleepq_switch(182d4aac,0) at sleepq_switch+0xf7/frame 0xb4c4c5c _cv_wait(182d4aac,182d4cd8) at _cv_wait+0x135/frame 0xb4c4c94 usb_process(182d4aa4,b4c4ce8) at usb_process+0x9d/frame 0xb4c4cb8 fork_exit(e39020,182d4aa4,b4c4ce8,0,0,...) at fork_exit+0x68/frame 0xb4c4cd4 fork_trampoline() at 0xffc0340e/frame 0xb4c4cd4 --- kthread start Tracing command usb pid 15 tid 100056 td 0x182db500 sched_switch(182db500,104) at sched_switch+0x4e9/frame 0xb4c9c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4c9c3c sleepq_switch(182d4adc,0) at sleepq_switch+0xf7/frame 0xb4c9c5c _cv_wait(182d4adc,182d4cd8) at _cv_wait+0x135/frame 0xb4c9c94 usb_process(182d4ad4,b4c9ce8) at usb_process+0x9d/frame 0xb4c9cb8 fork_exit(e39020,182d4ad4,b4c9ce8,0,0,...) at fork_exit+0x68/frame 0xb4c9cd4 fork_trampoline() at 0xffc0340e/frame 0xb4c9cd4 --- kthread start Tracing command usb pid 15 tid 100057 td 0x182db180 sched_switch(182db180,104) at sched_switch+0x4e9/frame 0xb4cec08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4cec3c sleepq_switch(182d4b0c,0) at sleepq_switch+0xf7/frame 0xb4cec5c _cv_wait(182d4b0c,182d4cd8) at _cv_wait+0x135/frame 0xb4cec94 usb_process(182d4b04,b4cece8) at usb_process+0x9d/frame 0xb4cecb8 fork_exit(e39020,182d4b04,b4cece8,0,0,...) at fork_exit+0x68/frame 0xb4cecd4 fork_trampoline() at 0xffc0340e/frame 0xb4cecd4 --- kthread start Tracing command usb pid 15 tid 100058 td 0x182dae00 sched_switch(182dae00,104) at sched_switch+0x4e9/frame 0xb4d3c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4d3c3c sleepq_switch(182d4b3c,0) at sleepq_switch+0xf7/frame 0xb4d3c5c _cv_wait(182d4b3c,182d4cd8) at _cv_wait+0x135/frame 0xb4d3c94 usb_process(182d4b34,b4d3ce8) at usb_process+0x9d/frame 0xb4d3cb8 fork_exit(e39020,182d4b34,b4d3ce8,0,0,...) at fork_exit+0x68/frame 0xb4d3cd4 fork_trampoline() at 0xffc0340e/frame 0xb4d3cd4 --- kthread start Tracing command usb pid 15 tid 100059 td 0x182daa80 sched_switch(182daa80,104) at sched_switch+0x4e9/frame 0xb4d8c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4d8c3c sleepq_switch(182d4b6c,0) at sleepq_switch+0xf7/frame 0xb4d8c5c _cv_wait(182d4b6c,182d4cd8) at _cv_wait+0x135/frame 0xb4d8c94 usb_process(182d4b64,b4d8ce8) at usb_process+0x9d/frame 0xb4d8cb8 fork_exit(e39020,182d4b64,b4d8ce8,0,0,...) at fork_exit+0x68/frame 0xb4d8cd4 fork_trampoline() at 0xffc0340e/frame 0xb4d8cd4 --- kthread start Tracing command usb pid 15 tid 100061 td 0x182da380 sched_switch(182da380,104) at sched_switch+0x4e9/frame 0xb4e3c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4e3c3c sleepq_switch(182deaac,0) at sleepq_switch+0xf7/frame 0xb4e3c5c _cv_wait(182deaac,182decd8) at _cv_wait+0x135/frame 0xb4e3c94 usb_process(182deaa4,b4e3ce8) at usb_process+0x9d/frame 0xb4e3cb8 fork_exit(e39020,182deaa4,b4e3ce8,0,0,...) at fork_exit+0x68/frame 0xb4e3cd4 fork_trampoline() at 0xffc0340e/frame 0xb4e3cd4 --- kthread start Tracing command usb pid 15 tid 100062 td 0x182da000 sched_switch(182da000,104) at sched_switch+0x4e9/frame 0xb4e8c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4e8c3c sleepq_switch(182deadc,0) at sleepq_switch+0xf7/frame 0xb4e8c5c _cv_wait(182deadc,182decd8) at _cv_wait+0x135/frame 0xb4e8c94 usb_process(182dead4,b4e8ce8) at usb_process+0x9d/frame 0xb4e8cb8 fork_exit(e39020,182dead4,b4e8ce8,0,0,...) at fork_exit+0x68/frame 0xb4e8cd4 fork_trampoline() at 0xffc0340e/frame 0xb4e8cd4 --- kthread start Tracing command usb pid 15 tid 100063 td 0x182e3c00 sched_switch(182e3c00,104) at sched_switch+0x4e9/frame 0xb4edc08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4edc3c sleepq_switch(182deb0c,0) at sleepq_switch+0xf7/frame 0xb4edc5c _cv_wait(182deb0c,182decd8) at _cv_wait+0x135/frame 0xb4edc94 usb_process(182deb04,b4edce8) at usb_process+0x9d/frame 0xb4edcb8 fork_exit(e39020,182deb04,b4edce8,0,0,...) at fork_exit+0x68/frame 0xb4edcd4 fork_trampoline() at 0xffc0340e/frame 0xb4edcd4 --- kthread start Tracing command usb pid 15 tid 100064 td 0x182e3880 sched_switch(182e3880,104) at sched_switch+0x4e9/frame 0xb4f2c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4f2c3c sleepq_switch(182deb3c,0) at sleepq_switch+0xf7/frame 0xb4f2c5c _cv_wait(182deb3c,182decd8) at _cv_wait+0x135/frame 0xb4f2c94 usb_process(182deb34,b4f2ce8) at usb_process+0x9d/frame 0xb4f2cb8 fork_exit(e39020,182deb34,b4f2ce8,0,0,...) at fork_exit+0x68/frame 0xb4f2cd4 fork_trampoline() at 0xffc0340e/frame 0xb4f2cd4 --- kthread start Tracing command usb pid 15 tid 100065 td 0x182e3500 sched_switch(182e3500,104) at sched_switch+0x4e9/frame 0xb4f7c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4f7c3c sleepq_switch(182deb6c,0) at sleepq_switch+0xf7/frame 0xb4f7c5c _cv_wait(182deb6c,182decd8) at _cv_wait+0x135/frame 0xb4f7c94 usb_process(182deb64,b4f7ce8) at usb_process+0x9d/frame 0xb4f7cb8 fork_exit(e39020,182deb64,b4f7ce8,0,0,...) at fork_exit+0x68/frame 0xb4f7cd4 fork_trampoline() at 0xffc0340e/frame 0xb4f7cd4 --- kthread start Tracing command usb pid 15 tid 100066 td 0x182e3180 sched_switch(182e3180,104) at sched_switch+0x4e9/frame 0xb4fdc08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4fdc3c sleepq_switch(182e6aac,0) at sleepq_switch+0xf7/frame 0xb4fdc5c _cv_wait(182e6aac,182e6cd8) at _cv_wait+0x135/frame 0xb4fdc94 usb_process(182e6aa4,b4fdce8) at usb_process+0x9d/frame 0xb4fdcb8 fork_exit(e39020,182e6aa4,b4fdce8,0,0,...) at fork_exit+0x68/frame 0xb4fdcd4 fork_trampoline() at 0xffc0340e/frame 0xb4fdcd4 --- kthread start Tracing command usb pid 15 tid 100067 td 0x182e2e00 sched_switch(182e2e00,104) at sched_switch+0x4e9/frame 0xb502c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb502c3c sleepq_switch(182e6adc,0) at sleepq_switch+0xf7/frame 0xb502c5c _cv_wait(182e6adc,182e6cd8) at _cv_wait+0x135/frame 0xb502c94 usb_process(182e6ad4,b502ce8) at usb_process+0x9d/frame 0xb502cb8 fork_exit(e39020,182e6ad4,b502ce8,0,0,...) at fork_exit+0x68/frame 0xb502cd4 fork_trampoline() at 0xffc0340e/frame 0xb502cd4 --- kthread start Tracing command usb pid 15 tid 100068 td 0x182e2a80 sched_switch(182e2a80,104) at sched_switch+0x4e9/frame 0xb507c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb507c3c sleepq_switch(182e6b0c,0) at sleepq_switch+0xf7/frame 0xb507c5c _cv_wait(182e6b0c,182e6cd8) at _cv_wait+0x135/frame 0xb507c94 usb_process(182e6b04,b507ce8) at usb_process+0x9d/frame 0xb507cb8 fork_exit(e39020,182e6b04,b507ce8,0,0,...) at fork_exit+0x68/frame 0xb507cd4 fork_trampoline() at 0xffc0340e/frame 0xb507cd4 --- kthread start Tracing command usb pid 15 tid 100069 td 0x182e2700 sched_switch(182e2700,104) at sched_switch+0x4e9/frame 0xb50cc08 mi_switch(104) at mi_switch+0x1c4/frame 0xb50cc3c sleepq_switch(182e6b3c,0) at sleepq_switch+0xf7/frame 0xb50cc5c _cv_wait(182e6b3c,182e6cd8) at _cv_wait+0x135/frame 0xb50cc94 usb_process(182e6b34,b50cce8) at usb_process+0x9d/frame 0xb50ccb8 fork_exit(e39020,182e6b34,b50cce8,0,0,...) at fork_exit+0x68/frame 0xb50ccd4 fork_trampoline() at 0xffc0340e/frame 0xb50ccd4 --- kthread start Tracing command usb pid 15 tid 100070 td 0x182e2380 sched_switch(182e2380,104) at sched_switch+0x4e9/frame 0xb511c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb511c3c sleepq_switch(182e6b6c,0) at sleepq_switch+0xf7/frame 0xb511c5c _cv_wait(182e6b6c,182e6cd8) at _cv_wait+0x135/frame 0xb511c94 usb_process(182e6b64,b511ce8) at usb_process+0x9d/frame 0xb511cb8 fork_exit(e39020,182e6b64,b511ce8,0,0,...) at fork_exit+0x68/frame 0xb511cd4 fork_trampoline() at 0xffc0340e/frame 0xb511cd4 --- kthread start Tracing command usb pid 15 tid 100071 td 0x182e2000 sched_switch(182e2000,104) at sched_switch+0x4e9/frame 0xb517c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb517c3c sleepq_switch(182eeaac,0) at sleepq_switch+0xf7/frame 0xb517c5c _cv_wait(182eeaac,182eecd8) at _cv_wait+0x135/frame 0xb517c94 usb_process(182eeaa4,b517ce8) at usb_process+0x9d/frame 0xb517cb8 fork_exit(e39020,182eeaa4,b517ce8,0,0,...) at fork_exit+0x68/frame 0xb517cd4 fork_trampoline() at 0xffc0340e/frame 0xb517cd4 --- kthread start Tracing command usb pid 15 tid 100072 td 0x182f3c00 sched_switch(182f3c00,104) at sched_switch+0x4e9/frame 0xb51cc08 mi_switch(104) at mi_switch+0x1c4/frame 0xb51cc3c sleepq_switch(182eeadc,0) at sleepq_switch+0xf7/frame 0xb51cc5c _cv_wait(182eeadc,182eecd8) at _cv_wait+0x135/frame 0xb51cc94 usb_process(182eead4,b51cce8) at usb_process+0x9d/frame 0xb51ccb8 fork_exit(e39020,182eead4,b51cce8,0,0,...) at fork_exit+0x68/frame 0xb51ccd4 fork_trampoline() at 0xffc0340e/frame 0xb51ccd4 --- kthread start Tracing command usb pid 15 tid 100073 td 0x182f3880 sched_switch(182f3880,104) at sched_switch+0x4e9/frame 0xb521c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb521c3c sleepq_switch(182eeb0c,0) at sleepq_switch+0xf7/frame 0xb521c5c _cv_wait(182eeb0c,182eecd8) at _cv_wait+0x135/frame 0xb521c94 usb_process(182eeb04,b521ce8) at usb_process+0x9d/frame 0xb521cb8 fork_exit(e39020,182eeb04,b521ce8,0,0,...) at fork_exit+0x68/frame 0xb521cd4 fork_trampoline() at 0xffc0340e/frame 0xb521cd4 --- kthread start Tracing command usb pid 15 tid 100074 td 0x182f3500 sched_switch(182f3500,104) at sched_switch+0x4e9/frame 0xb526c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb526c3c sleepq_switch(182eeb3c,0) at sleepq_switch+0xf7/frame 0xb526c5c _cv_wait(182eeb3c,182eecd8) at _cv_wait+0x135/frame 0xb526c94 usb_process(182eeb34,b526ce8) at usb_process+0x9d/frame 0xb526cb8 fork_exit(e39020,182eeb34,b526ce8,0,0,...) at fork_exit+0x68/frame 0xb526cd4 fork_trampoline() at 0xffc0340e/frame 0xb526cd4 --- kthread start Tracing command usb pid 15 tid 100075 td 0x182f3180 sched_switch(182f3180,104) at sched_switch+0x4e9/frame 0xb52bc08 mi_switch(104) at mi_switch+0x1c4/frame 0xb52bc3c sleepq_switch(182eeb6c,0) at sleepq_switch+0xf7/frame 0xb52bc5c _cv_wait(182eeb6c,182eecd8) at _cv_wait+0x135/frame 0xb52bc94 usb_process(182eeb64,b52bce8) at usb_process+0x9d/frame 0xb52bcb8 fork_exit(e39020,182eeb64,b52bce8,0,0,...) at fork_exit+0x68/frame 0xb52bcd4 fork_trampoline() at 0xffc0340e/frame 0xb52bcd4 --- kthread start Tracing command usb pid 15 tid 100077 td 0x182f2a80 sched_switch(182f2a80,104) at sched_switch+0x4e9/frame 0xb536c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb536c3c sleepq_switch(18014c84,0) at sleepq_switch+0xf7/frame 0xb536c5c _cv_wait(18014c84,18014eb0) at _cv_wait+0x135/frame 0xb536c94 usb_process(18014c7c,b536ce8) at usb_process+0x9d/frame 0xb536cb8 fork_exit(e39020,18014c7c,b536ce8,0,0,...) at fork_exit+0x68/frame 0xb536cd4 fork_trampoline() at 0xffc0340e/frame 0xb536cd4 --- kthread start Tracing command usb pid 15 tid 100078 td 0x182f2700 sched_switch(182f2700,104) at sched_switch+0x4e9/frame 0xb53bc08 mi_switch(104) at mi_switch+0x1c4/frame 0xb53bc3c sleepq_switch(18014cb4,0) at sleepq_switch+0xf7/frame 0xb53bc5c _cv_wait(18014cb4,18014eb0) at _cv_wait+0x135/frame 0xb53bc94 usb_process(18014cac,b53bce8) at usb_process+0x9d/frame 0xb53bcb8 fork_exit(e39020,18014cac,b53bce8,0,0,...) at fork_exit+0x68/frame 0xb53bcd4 fork_trampoline() at 0xffc0340e/frame 0xb53bcd4 --- kthread start Tracing command usb pid 15 tid 100079 td 0x182f2380 sched_switch(182f2380,104) at sched_switch+0x4e9/frame 0xb540c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb540c3c sleepq_switch(18014ce4,0) at sleepq_switch+0xf7/frame 0xb540c5c _cv_wait(18014ce4,18014eb0) at _cv_wait+0x135/frame 0xb540c94 usb_process(18014cdc,b540ce8) at usb_process+0x9d/frame 0xb540cb8 fork_exit(e39020,18014cdc,b540ce8,0,0,...) at fork_exit+0x68/frame 0xb540cd4 fork_trampoline() at 0xffc0340e/frame 0xb540cd4 --- kthread start Tracing command usb pid 15 tid 100080 td 0x182f2000 sched_switch(182f2000,104) at sched_switch+0x4e9/frame 0xb545c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb545c3c sleepq_switch(18014d14,0) at sleepq_switch+0xf7/frame 0xb545c5c _cv_wait(18014d14,18014eb0) at _cv_wait+0x135/frame 0xb545c94 usb_process(18014d0c,b545ce8) at usb_process+0x9d/frame 0xb545cb8 fork_exit(e39020,18014d0c,b545ce8,0,0,...) at fork_exit+0x68/frame 0xb545cd4 fork_trampoline() at 0xffc0340e/frame 0xb545cd4 --- kthread start Tracing command usb pid 15 tid 100081 td 0x18314c00 sched_switch(18314c00,104) at sched_switch+0x4e9/frame 0xb54ac08 mi_switch(104) at mi_switch+0x1c4/frame 0xb54ac3c sleepq_switch(18014d44,0) at sleepq_switch+0xf7/frame 0xb54ac5c _cv_wait(18014d44,18014eb0) at _cv_wait+0x135/frame 0xb54ac94 usb_process(18014d3c,b54ace8) at usb_process+0x9d/frame 0xb54acb8 fork_exit(e39020,18014d3c,b54ace8,0,0,...) at fork_exit+0x68/frame 0xb54acd4 fork_trampoline() at 0xffc0340e/frame 0xb54acd4 --- kthread start Tracing command cam pid 7 tid 100044 td 0x9b01380 sched_switch(9b01380,104) at sched_switch+0x4e9/frame 0x12bffbd8 mi_switch(104) at mi_switch+0x1c4/frame 0x12bffc0c sleepq_switch(1ae46c0,5c) at sleepq_switch+0xf7/frame 0x12bffc2c _sleep(1ae46c0,1ae4680,5c,142fff2,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bffc6c xpt_done_td(1ae4680,12bffce8) at xpt_done_td+0x72/frame 0x12bffcb8 fork_exit(908180,1ae4680,12bffce8,0,0,...) at fork_exit+0x68/frame 0x12bffcd4 fork_trampoline() at 0xffc0340e/frame 0x12bffcd4 --- kthread start Tracing command cam pid 7 tid 100096 td 0x18924880 sched_switch(18924880,104) at sched_switch+0x4e9/frame 0xb5a2be8 mi_switch(104) at mi_switch+0x1c4/frame 0xb5a2c20 sleepq_switch(1ae45ec,5c) at sleepq_switch+0xf7/frame 0xb5a2c40 _sleep(1ae45ec,1ae465c,5c,142fff2,0,0,0,0,100) at _sleep+0x2f3/frame 0xb5a2c80 xpt_scanner_thread(0,b5a2ce8) at xpt_scanner_thread+0x56/frame 0xb5a2cb8 fork_exit(908420,0,b5a2ce8,0,0,...) at fork_exit+0x68/frame 0xb5a2cd4 fork_trampoline() at 0xffc0340e/frame 0xb5a2cd4 --- kthread start Tracing command crypto returns 3 pid 6 tid 100043 td 0x9b01700 sched_switch(9b01700,104) at sched_switch+0x4e9/frame 0x12bfabc0 mi_switch(104) at mi_switch+0x1c4/frame 0x12bfabf4 sleepq_switch(b9f89c4,6c) at sleepq_switch+0xf7/frame 0x12bfac14 _sleep(b9f89c4,b9f89a8,6c,1517198,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bfac54 crypto_ret_proc(b9f89a8,12bface8) at crypto_ret_proc+0x1b9/frame 0x12bfacb8 fork_exit(11fa440,b9f89a8,12bface8,0,0,...) at fork_exit+0x68/frame 0x12bfacd4 fork_trampoline() at 0xffc0340e/frame 0x12bfacd4 --- kthread start Tracing command crypto returns 2 pid 5 tid 100042 td 0x9b01a80 sched_switch(9b01a80,104) at sched_switch+0x4e9/frame 0x12bf5bc0 mi_switch(104) at mi_switch+0x1c4/frame 0x12bf5bf4 sleepq_switch(b9f898c,6c) at sleepq_switch+0xf7/frame 0x12bf5c14 _sleep(b9f898c,b9f8970,6c,1517198,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bf5c54 crypto_ret_proc(b9f8970,12bf5ce8) at crypto_ret_proc+0x1b9/frame 0x12bf5cb8 fork_exit(11fa440,b9f8970,12bf5ce8,0,0,...) at fork_exit+0x68/frame 0x12bf5cd4 fork_trampoline() at 0xffc0340e/frame 0x12bf5cd4 --- kthread start Tracing command crypto returns 1 pid 4 tid 100041 td 0x9b01e00 sched_switch(9b01e00,104) at sched_switch+0x4e9/frame 0x12bf0bc0 mi_switch(104) at mi_switch+0x1c4/frame 0x12bf0bf4 sleepq_switch(b9f8954,6c) at sleepq_switch+0xf7/frame 0x12bf0c14 _sleep(b9f8954,b9f8938,6c,1517198,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bf0c54 crypto_ret_proc(b9f8938,12bf0ce8) at crypto_ret_proc+0x1b9/frame 0x12bf0cb8 fork_exit(11fa440,b9f8938,12bf0ce8,0,0,...) at fork_exit+0x68/frame 0x12bf0cd4 fork_trampoline() at 0xffc0340e/frame 0x12bf0cd4 --- kthread start Tracing command crypto returns 0 pid 3 tid 100040 td 0x9b02180 sched_switch(9b02180,104) at sched_switch+0x4e9/frame 0x12bebbc0 mi_switch(104) at mi_switch+0x1c4/frame 0x12bebbf4 sleepq_switch(b9f891c,6c) at sleepq_switch+0xf7/frame 0x12bebc14 _sleep(b9f891c,b9f8900,6c,1517198,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bebc54 crypto_ret_proc(b9f8900,12bebce8) at crypto_ret_proc+0x1b9/frame 0x12bebcb8 fork_exit(11fa440,b9f8900,12bebce8,0,0,...) at fork_exit+0x68/frame 0x12bebcd4 fork_trampoline() at 0xffc0340e/frame 0x12bebcd4 --- kthread start Tracing command crypto pid 2 tid 100039 td 0x9b02500 sched_switch(9b02500,104) at sched_switch+0x4e9/frame 0x12be6bd8 mi_switch(104) at mi_switch+0x1c4/frame 0x12be6c0c sleepq_switch(1de4a20,6c) at sleepq_switch+0xf7/frame 0x12be6c2c _sleep(1de4a20,1de4a08,6c,14c94a1,0,0,0,0,100) at _sleep+0x2f3/frame 0x12be6c6c crypto_proc(0,12be6ce8) at crypto_proc+0x29f/frame 0x12be6cb8 fork_exit(11f9ee0,0,12be6ce8,0,0,...) at fork_exit+0x68/frame 0x12be6cd4 fork_trampoline() at 0xffc0340e/frame 0x12be6cd4 --- kthread start Tracing command sequencer 00 pid 14 tid 100034 td 0xb9f9700 sched_switch(b9f9700,104) at sched_switch+0x4e9/frame 0x12bcdbe0 mi_switch(104) at mi_switch+0x1c4/frame 0x12bcdc18 sleepq_switch(9ae8e4c,0) at sleepq_switch+0xf7/frame 0x12bcdc38 _cv_wait(9ae8e4c,9ae8e04) at _cv_wait+0x135/frame 0x12bcdc70 seq_eventthread(9ae8e00,12bcdce8) at seq_eventthread+0x7a/frame 0x12bcdcb8 fork_exit(dc6f30,9ae8e00,12bcdce8,0,0,...) at fork_exit+0x68/frame 0x12bcdcd4 fork_trampoline() at 0xffc0340e/frame 0x12bcdcd4 --- kthread start Tracing command geom pid 13 tid 100031 td 0xb9fa180 sched_switch(b9fa180,104) at sched_switch+0x4e9/frame 0x12bbebe8 mi_switch(104) at mi_switch+0x1c4/frame 0x12bbec20 sleepq_switch(1d4c288,5c) at sleepq_switch+0xf7/frame 0x12bbec40 _sleep(1d4c288,1d4c1f4,25c,142fff2,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bbec80 g_run_events(0,12bbece8) at g_run_events+0x4d/frame 0x12bbecb8 fork_exit(f39fe0,0,12bbece8,0,0,...) at fork_exit+0x68/frame 0x12bbecd4 fork_trampoline() at 0xffc0340e/frame 0x12bbecd4 --- kthread start Tracing command geom pid 13 tid 100032 td 0xb9f9e00 sched_switch(b9f9e00,104) at sched_switch+0x4e9/frame 0x12bc3be0 mi_switch(104) at mi_switch+0x1c4/frame 0x12bc3c18 sleepq_switch(1d4c28c,5c) at sleepq_switch+0xf7/frame 0x12bc3c38 _sleep(1d4c28c,1d4c238,25c,142fff2,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bc3c78 g_io_schedule_up(b9f9e00) at g_io_schedule_up+0x36/frame 0x12bc3cac g_up_procbody(0,12bc3ce8) at g_up_procbody+0x5b/frame 0x12bc3cb8 fork_exit(f3a030,0,12bc3ce8,0,0,...) at fork_exit+0x68/frame 0x12bc3cd4 fork_trampoline() at 0xffc0340e/frame 0x12bc3cd4 --- kthread start Tracing command geom pid 13 tid 100033 td 0xb9f9a80 sched_switch(b9f9a80,104) at sched_switch+0x4e9/frame 0x12bc8be0 mi_switch(104) at mi_switch+0x1c4/frame 0x12bc8c14 sleepq_switch(1d4c290,5c) at sleepq_switch+0xf7/frame 0x12bc8c34 _sleep(1d4c290,1d4c218,25c,142fff2,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bc8c74 g_io_schedule_down(b9f9a80) at g_io_schedule_down+0x36/frame 0x12bc8cac g_down_procbody(0,12bc8ce8) at g_down_procbody+0x5b/frame 0x12bc8cb8 fork_exit(f3a090,0,12bc8ce8,0,0,...) at fork_exit+0x68/frame 0x12bc8cd4 fork_trampoline() at 0xffc0340e/frame 0x12bc8cd4 --- kthread start Tracing command intr pid 12 tid 100016 td 0xb9ee700 sched_switch(b9ee700,109) at sched_switch+0x4e9/frame 0x12b63c30 mi_switch(109) at mi_switch+0x1c4/frame 0x12b63c64 ithread_loop(b8de5a0,12b63ce8) at ithread_loop+0x85/frame 0x12b63cb8 fork_exit(fa9ed0,b8de5a0,12b63ce8,0,0,...) at fork_exit+0x68/frame 0x12b63cd4 fork_trampoline() at 0xffc0340e/frame 0x12b63cd4 --- kthread start Tracing command intr pid 12 tid 100019 td 0xb9f3c00 __stop_set_sysinit_set() at 0xffc03404 Tracing command intr pid 12 tid 100022 td 0xb9f3180 sched_switch(b9f3180,109) at sched_switch+0x4e9/frame 0x12b81c30 mi_switch(109) at mi_switch+0x1c4/frame 0x12b81c64 ithread_loop(b8de580,12b81ce8) at ithread_loop+0x85/frame 0x12b81cb8 fork_exit(fa9ed0,b8de580,12b81ce8,0,0,...) at fork_exit+0x68/frame 0x12b81cd4 fork_trampoline() at 0xffc0340e/frame 0x12b81cd4 --- kthread start Tracing command intr pid 12 tid 100024 td 0xb9f2a80 sched_switch(b9f2a80,109) at sched_switch+0x4e9/frame 0x12b8fc30 mi_switch(109) at mi_switch+0x1c4/frame 0x12b8fc64 ithread_loop(b8de570,12b8fce8) at ithread_loop+0x85/frame 0x12b8fcb8 fork_exit(fa9ed0,b8de570,12b8fce8,0,0,...) at fork_exit+0x68/frame 0x12b8fcd4 fork_trampoline() at 0xffc0340e/frame 0x12b8fcd4 --- kthread start Tracing command intr pid 12 tid 100025 td 0xb9f2700 __stop_set_sysinit_set() at 0xffc03404 Tracing command intr pid 12 tid 100026 td 0xb9f2380 __stop_set_sysinit_set() at 0xffc03404 Tracing command intr pid 12 tid 100027 td 0xb9f2000 __stop_set_sysinit_set() at 0xffc03404 Tracing command intr pid 12 tid 100028 td 0xb9fac00 sched_switch(b9fac00,109) at sched_switch+0x4e9/frame 0x12ba3c30 mi_switch(109) at mi_switch+0x1c4/frame 0x12ba3c64 ithread_loop(b8de530,12ba3ce8) at ithread_loop+0x85/frame 0x12ba3cb8 fork_exit(fa9ed0,b8de530,12ba3ce8,0,0,...) at fork_exit+0x68/frame 0x12ba3cd4 fork_trampoline() at 0xffc0340e/frame 0x12ba3cd4 --- kthread start Tracing command intr pid 12 tid 100029 td 0xb9fa880 __stop_set_sysinit_set() at 0xffc03404 Tracing command intr pid 12 tid 100045 td 0x18038c00 sched_switch(18038c00,109) at sched_switch+0x4e9/frame 0xb48ac30 mi_switch(109) at mi_switch+0x1c4/frame 0xb48ac64 ithread_loop(18036fa0,b48ace8) at ithread_loop+0x85/frame 0xb48acb8 fork_exit(fa9ed0,18036fa0,b48ace8,0,0,...) at fork_exit+0x68/frame 0xb48acd4 fork_trampoline() at 0xffc0340e/frame 0xb48acd4 --- kthread start Tracing command intr pid 12 tid 100046 td 0x18038880 sched_switch(18038880,109) at sched_switch+0x4e9/frame 0xb494c30 mi_switch(109) at mi_switch+0x1c4/frame 0xb494c64 ithread_loop(18036f80,b494ce8) at ithread_loop+0x85/frame 0xb494cb8 fork_exit(fa9ed0,18036f80,b494ce8,0,0,...) at fork_exit+0x68/frame 0xb494cd4 fork_trampoline() at 0xffc0340e/frame 0xb494cd4 --- kthread start Tracing command intr pid 12 tid 100047 td 0x18038500 sched_switch(18038500,109) at sched_switch+0x4e9/frame 0xb49ac30 mi_switch(109) at mi_switch+0x1c4/frame 0xb49ac64 ithread_loop(18036f70,b49ace8) at ithread_loop+0x85/frame 0xb49acb8 fork_exit(fa9ed0,18036f70,b49ace8,0,0,...) at fork_exit+0x68/frame 0xb49acd4 fork_trampoline() at 0xffc0340e/frame 0xb49acd4 --- kthread start Tracing command intr pid 12 tid 100048 td 0x18038180 sched_switch(18038180,109) at sched_switch+0x4e9/frame 0xb4a0c30 mi_switch(109) at mi_switch+0x1c4/frame 0xb4a0c64 ithread_loop(18036f60,b4a0ce8) at ithread_loop+0x85/frame 0xb4a0cb8 fork_exit(fa9ed0,18036f60,b4a0ce8,0,0,...) at fork_exit+0x68/frame 0xb4a0cd4 fork_trampoline() at 0xffc0340e/frame 0xb4a0cd4 --- kthread start Tracing command intr pid 12 tid 100054 td 0x182dbc00 sched_switch(182dbc00,109) at sched_switch+0x4e9/frame 0xb4bfc30 mi_switch(109) at mi_switch+0x1c4/frame 0xb4bfc64 ithread_loop(18036f50,b4bfce8) at ithread_loop+0x85/frame 0xb4bfcb8 fork_exit(fa9ed0,18036f50,b4bfce8,0,0,...) at fork_exit+0x68/frame 0xb4bfcd4 fork_trampoline() at 0xffc0340e/frame 0xb4bfcd4 --- kthread start Tracing command intr pid 12 tid 100060 td 0x182da700 sched_switch(182da700,109) at sched_switch+0x4e9/frame 0xb4dec30 mi_switch(109) at mi_switch+0x1c4/frame 0xb4dec64 ithread_loop(18036f40,b4dece8) at ithread_loop+0x85/frame 0xb4decb8 fork_exit(fa9ed0,18036f40,b4dece8,0,0,...) at fork_exit+0x68/frame 0xb4decd4 fork_trampoline() at 0xffc0340e/frame 0xb4decd4 --- kthread start Tracing command intr pid 12 tid 100076 td 0x182f2e00 __stop_set_sysinit_set() at 0xffc03404 Tracing command intr pid 12 tid 100082 td 0x18314880 __stop_set_sysinit_set() at 0xffc03404 Tracing command intr pid 12 tid 100083 td 0x18314500 sched_switch(18314500,109) at sched_switch+0x4e9/frame 0xb558c30 mi_switch(109) at mi_switch+0x1c4/frame 0xb558c64 ithread_loop(18036f10,b558ce8) at ithread_loop+0x85/frame 0xb558cb8 fork_exit(fa9ed0,18036f10,b558ce8,0,0,...) at fork_exit+0x68/frame 0xb558cd4 fork_trampoline() at 0xffc0340e/frame 0xb558cd4 --- kthread start Tracing command intr pid 12 tid 100085 td 0x18313e00 __stop_set_sysinit_set() at 0xffc03404 Tracing command idle pid 11 tid 100003 td 0xb9c3500 (CPU 0) cpustop_handler(12b21bc4,12b21bb8,13d05cb,8,14a9f5c,...) at cpustop_handler+0x22/frame 0x12b21aec ipi_nmi_handler(8,14a9f5c,a40,b9f2a80,b9c1398,...) at ipi_nmi_handler+0x30/frame 0x12b21af8 trap(12b21bc4,8,28,28,9bd8e00,...) at trap+0xbb/frame 0x12b21bb8 calltrap() at 0xffc0319f/frame 0x12b21bb8 --- trap 0x13, eip = 0x1384635, esp = 0x12b21c04, ebp = 0x12b21c04 --- acpi_cpu_c1(9717,66c4262f,9717,c9dcc63a,3f68,...) at acpi_cpu_c1+0x5/frame 0x12b21c04 acpi_cpu_idle(7b243f8,0) at acpi_cpu_idle+0x22f/frame 0x12b21c34 cpu_idle_acpi(7b243f8,0) at cpu_idle_acpi+0x35/frame 0x12b21c48 cpu_idle(0) at cpu_idle+0xa3/frame 0x12b21c64 sched_idletd(0,12b21ce8) at sched_idletd+0x33e/frame 0x12b21cb8 fork_exit(10117d0,0,12b21ce8,0,0,...) at fork_exit+0x68/frame 0x12b21cd4 fork_trampoline() at 0xffc0340e/frame 0x12b21cd4 --- kthread start Tracing command idle pid 11 tid 100004 td 0xb9c3180 (CPU 1) cpustop_handler(12b26bc4,12b26bb8,13d05cb,a4c795a,fc53e3,...) at cpustop_handler+0x22/frame 0x12b26aec ipi_nmi_handler(a4c795a,fc53e3,0,8,b9c1398,...) at ipi_nmi_handler+0x30/frame 0x12b26af8 trap(12b26bc4,8,28,28,9bd8c00,...) at trap+0xbb/frame 0x12b26bb8 calltrap() at 0xffc0319f/frame 0x12b26bb8 --- trap 0x13, eip = 0x1384635, esp = 0x12b26c04, ebp = 0x12b26c04 --- acpi_cpu_c1(9717,66c0ed4e,9717,c9db7ace,3f68,...) at acpi_cpu_c1+0x5/frame 0x12b26c04 acpi_cpu_idle(316fdade,0) at acpi_cpu_idle+0x22f/frame 0x12b26c34 cpu_idle_acpi(316fdade,0) at cpu_idle_acpi+0x35/frame 0x12b26c48 cpu_idle(0) at cpu_idle+0xa3/frame 0x12b26c64 sched_idletd(0,12b26ce8) at sched_idletd+0x33e/frame 0x12b26cb8 fork_exit(10117d0,0,12b26ce8,0,0,...) at fork_exit+0x68/frame 0x12b26cd4 fork_trampoline() at 0xffc0340e/frame 0x12b26cd4 --- kthread start Tracing command idle pid 11 tid 100005 td 0xb9c2e00 (CPU 2) cpustop_handler(12b2bbc4,12b2bbb8,13d05cb,a4c795a,fc53e3,...) at cpustop_handler+0x22/frame 0x12b2baec ipi_nmi_handler(a4c795a,fc53e3,0,8,b9c1398,...) at ipi_nmi_handler+0x30/frame 0x12b2baf8 trap(12b2bbc4,8,28,28,9bd8a00,...) at trap+0xbb/frame 0x12b2bbb8 calltrap() at 0xffc0319f/frame 0x12b2bbb8 --- trap 0x13, eip = 0x1384635, esp = 0x12b2bc04, ebp = 0x12b2bc04 --- acpi_cpu_c1(9717,66c38b85,9717,c9dc83e5,3f68,...) at acpi_cpu_c1+0x5/frame 0x12b2bc04 acpi_cpu_idle(38802e09,0) at acpi_cpu_idle+0x22f/frame 0x12b2bc34 cpu_idle_acpi(38802e09,0) at cpu_idle_acpi+0x35/frame 0x12b2bc48 cpu_idle(0) at cpu_idle+0xa3/frame 0x12b2bc64 sched_idletd(0,12b2bce8) at sched_idletd+0x33e/frame 0x12b2bcb8 fork_exit(10117d0,0,12b2bce8,0,0,...) at fork_exit+0x68/frame 0x12b2bcd4 fork_trampoline() at 0xffc0340e/frame 0x12b2bcd4 --- kthread start Tracing command idle pid 11 tid 100006 td 0xb9c2a80 sched_switch(b9c2a80,608) at sched_switch+0x4e9/frame 0x12b30c20 mi_switch(608) at mi_switch+0x1c4/frame 0x12b30c54 critical_exit_preempt(0) at critical_exit_preempt+0x5c/frame 0x12b30c64 sched_idletd(0,12b30ce8) at sched_idletd+0x33e/frame 0x12b30cb8 fork_exit(10117d0,0,12b30ce8,0,0,...) at fork_exit+0x68/frame 0x12b30cd4 fork_trampoline() at 0xffc0340e/frame 0x12b30cd4 --- kthread start Tracing command init pid 1 tid 100002 td 0xb9c3880 sched_switch(b9c3880,104) at sched_switch+0x4e9/frame 0x12b189c8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b189fc sleepq_switch(3280,14e4b25,6c,0,1d74454,...) at sleepq_switch+0xf7/frame 0x12b18a1c sleepq_catch_signals(12b18a8c,feff3c,b9c16dc,6c,200246,...) at sleepq_catch_signals+0x278/frame 0x12b18a44 sleepq_wait_sig(b9c16dc,6c) at sleepq_wait_sig+0xe/frame 0x12b18a4c _sleep(b9c16dc,b9c1784,36c,15016eb,0,0,0,0,100) at _sleep+0x27c/frame 0x12b18a8c kern_wait6(b9c3880,7,0,0,12b18b14,30,0,0) at kern_wait6+0x3b8/frame 0x12b18ae8 sys_wait4(b9c3880,b9c3b18,b9c3880,b9c16dc,22ff5b8,...) at sys_wait4+0x74/frame 0x12b18c08 syscall(12b18ce8,3b,3b,3b,ffbfe8e8,...) at syscall+0x1b8/frame 0x12b18cdc Xint0x80_syscall() at 0xffc033f9/frame 0x12b18cdc --- syscall (560, FreeBSD ELF32, sys_kevent), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe878 --- Tracing command audit pid 10 tid 100001 td 0xb9c3c00 sched_switch(b9c3c00,104) at sched_switch+0x4e9/frame 0x12b13be0 mi_switch(104) at mi_switch+0x1c4/frame 0x12b13c18 sleepq_switch(1de4d0c,0) at sleepq_switch+0xf7/frame 0x12b13c38 _cv_wait(1de4d0c,1de4c68) at _cv_wait+0x135/frame 0x12b13c70 audit_worker(0,12b13ce8) at audit_worker+0x88/frame 0x12b13cb8 fork_exit(121bfe0,0,12b13ce8,0,0,...) at fork_exit+0x68/frame 0x12b13cd4 fork_trampoline() at 0xffc0340e/frame 0x12b13cd4 --- kthread start Tracing command kernel pid 0 tid 100000 td 0x1d4caa0 sched_switch(1d4caa0,104) at sched_switch+0x4e9/frame 0x22f9c10 mi_switch(104) at mi_switch+0x1c4/frame 0x22f9c48 sleepq_switch(1d4caa0,0,1d4c75c,22f9ccc,1d4caa0,...) at sleepq_switch+0xf7/frame 0x22f9c68 sleepq_timedwait(1d4c75c,54,246,0,40000,...) at sleepq_timedwait+0x3f/frame 0x22f9c8c _sleep(1d4c75c,0,54,150452d,fffff470,9,0,0,100) at _sleep+0x22a/frame 0x22f9ccc swapper(0,0,0,1ac7598) at swapper+0x6d/frame 0x22f9d14 mi_startup() at mi_startup+0x2c6/frame 0x22f9d38 btext() at btext+0x5f Tracing command kernel pid 0 tid 100007 td 0xb9c2700 sched_switch(b9c2700,104) at sched_switch+0x4e9/frame 0x12b36bf8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b36c2c sleepq_switch(b9caa00,0) at sleepq_switch+0xf7/frame 0x12b36c4c msleep_spin_sbt(b9caa00,b9caa40,142fff2,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12b36c84 gtaskqueue_thread_loop(9a3b804,12b36ce8) at gtaskqueue_thread_loop+0xfc/frame 0x12b36cb8 fork_exit(1027520,9a3b804,12b36ce8,0,0,...) at fork_exit+0x68/frame 0x12b36cd4 fork_trampoline() at 0xffc0340e/frame 0x12b36cd4 --- kthread start Tracing command kernel pid 0 tid 100008 td 0xb9c2380 sched_switch(b9c2380,104) at sched_switch+0x4e9/frame 0x12b3bbf8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b3bc2c sleepq_switch(b9ca800,0) at sleepq_switch+0xf7/frame 0x12b3bc4c msleep_spin_sbt(b9ca800,b9ca840,142fff2,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12b3bc84 gtaskqueue_thread_loop(9a3b814,12b3bce8) at gtaskqueue_thread_loop+0xfc/frame 0x12b3bcb8 fork_exit(1027520,9a3b814,12b3bce8,0,0,...) at fork_exit+0x68/frame 0x12b3bcd4 fork_trampoline() at 0xffc0340e/frame 0x12b3bcd4 --- kthread start Tracing command kernel pid 0 tid 100009 td 0xb9c2000 sched_switch(b9c2000,104) at sched_switch+0x4e9/frame 0x12b40bf8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b40c2c sleepq_switch(b9ca600,0) at sleepq_switch+0xf7/frame 0x12b40c4c msleep_spin_sbt(b9ca600,b9ca640,142fff2,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12b40c84 gtaskqueue_thread_loop(9a3b824,12b40ce8) at gtaskqueue_thread_loop+0xfc/frame 0x12b40cb8 fork_exit(1027520,9a3b824,12b40ce8,0,0,...) at fork_exit+0x68/frame 0x12b40cd4 fork_trampoline() at 0xffc0340e/frame 0x12b40cd4 --- kthread start Tracing command kernel pid 0 tid 100010 td 0xb9efc00 sched_switch(b9efc00,104) at sched_switch+0x4e9/frame 0x12b45bf8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b45c2c sleepq_switch(b9ca400,0) at sleepq_switch+0xf7/frame 0x12b45c4c msleep_spin_sbt(b9ca400,b9ca440,142fff2,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12b45c84 gtaskqueue_thread_loop(9a3b834,12b45ce8) at gtaskqueue_thread_loop+0xfc/frame 0x12b45cb8 fork_exit(1027520,9a3b834,12b45ce8,0,0,...) at fork_exit+0x68/frame 0x12b45cd4 fork_trampoline() at 0xffc0340e/frame 0x12b45cd4 --- kthread start Tracing command kernel pid 0 tid 100011 td 0xb9ef880 sched_switch(b9ef880,104) at sched_switch+0x4e9/frame 0x12b4abf8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b4ac2c sleepq_switch(b9ca200,0) at sleepq_switch+0xf7/frame 0x12b4ac4c msleep_spin_sbt(b9ca200,b9ca240,142fff2,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12b4ac84 gtaskqueue_thread_loop(9a3b404,12b4ace8) at gtaskqueue_thread_loop+0xfc/frame 0x12b4acb8 fork_exit(1027520,9a3b404,12b4ace8,0,0,...) at fork_exit+0x68/frame 0x12b4acd4 fork_trampoline() at 0xffc0340e/frame 0x12b4acd4 --- kthread start Tracing command kernel pid 0 tid 100012 td 0xb9ef500 sched_switch(b9ef500,104) at sched_switch+0x4e9/frame 0x12b4fbf8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b4fc2c sleepq_switch(b9ca000,0) at sleepq_switch+0xf7/frame 0x12b4fc4c msleep_spin_sbt(b9ca000,b9ca040,142fff2,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12b4fc84 gtaskqueue_thread_loop(9a3b004,12b4fce8) at gtaskqueue_thread_loop+0xfc/frame 0x12b4fcb8 fork_exit(1027520,9a3b004,12b4fce8,0,0,...) at fork_exit+0x68/frame 0x12b4fcd4 fork_trampoline() at 0xffc0340e/frame 0x12b4fcd4 --- kthread start Tracing command kernel pid 0 tid 100013 td 0xb9ef180 sched_switch(b9ef180,104) at sched_switch+0x4e9/frame 0x12b54bf8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b54c2c sleepq_switch(b9f0d00,0) at sleepq_switch+0xf7/frame 0x12b54c4c msleep_spin_sbt(b9f0d00,b9f0d40,142fff2,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12b54c84 gtaskqueue_thread_loop(9a3b014,12b54ce8) at gtaskqueue_thread_loop+0xfc/frame 0x12b54cb8 fork_exit(1027520,9a3b014,12b54ce8,0,0,...) at fork_exit+0x68/frame 0x12b54cd4 fork_trampoline() at 0xffc0340e/frame 0x12b54cd4 --- kthread start Tracing command kernel pid 0 tid 100014 td 0xb9eee00 sched_switch(b9eee00,104) at sched_switch+0x4e9/frame 0x12b59bf8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b59c2c sleepq_switch(b9f0b00,0) at sleepq_switch+0xf7/frame 0x12b59c4c msleep_spin_sbt(b9f0b00,b9f0b40,142fff2,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12b59c84 gtaskqueue_thread_loop(9a3b024,12b59ce8) at gtaskqueue_thread_loop+0xfc/frame 0x12b59cb8 fork_exit(1027520,9a3b024,12b59ce8,0,0,...) at fork_exit+0x68/frame 0x12b59cd4 fork_trampoline() at 0xffc0340e/frame 0x12b59cd4 --- kthread start Tracing command kernel pid 0 tid 100015 td 0xb9eea80 sched_switch(b9eea80,104) at sched_switch+0x4e9/frame 0x12b5ebf8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b5ec2c sleepq_switch(b9f0900,0) at sleepq_switch+0xf7/frame 0x12b5ec4c msleep_spin_sbt(b9f0900,b9f0940,142fff2,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12b5ec84 gtaskqueue_thread_loop(9a3b034,12b5ece8) at gtaskqueue_thread_loop+0xfc/frame 0x12b5ecb8 fork_exit(1027520,9a3b034,12b5ece8,0,0,...) at fork_exit+0x68/frame 0x12b5ecd4 fork_trampoline() at 0xffc0340e/frame 0x12b5ecd4 --- kthread start Tracing command kernel pid 0 tid 100017 td 0xb9ee380 sched_switch(b9ee380,104) at sched_switch+0x4e9/frame 0x12b68be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b68c20 sleepq_switch(b9f0500,0) at sleepq_switch+0xf7/frame 0x12b68c40 _sleep(b9f0500,b9f0540,0,142fff2,0,0,0,0,100) at _sleep+0x2f3/frame 0x12b68c80 taskqueue_thread_loop(1de35d4,12b68ce8) at taskqueue_thread_loop+0xd5/frame 0x12b68cb8 fork_exit(103c410,1de35d4,12b68ce8,0,0,...) at fork_exit+0x68/frame 0x12b68cd4 fork_trampoline() at 0xffc0340e/frame 0x12b68cd4 --- kthread start Tracing command kernel pid 0 tid 100018 td 0xb9ee000 sched_switch(b9ee000,104) at sched_switch+0x4e9/frame 0x12b6dbe8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b6dc20 sleepq_switch(b9f0300,0) at sleepq_switch+0xf7/frame 0x12b6dc40 _sleep(b9f0300,b9f0340,0,142fff2,0,0,0,0,100) at _sleep+0x2f3/frame 0x12b6dc80 taskqueue_thread_loop(1d759d4,12b6dce8) at taskqueue_thread_loop+0xd5/frame 0x12b6dcb8 fork_exit(103c410,1d759d4,12b6dce8,0,0,...) at fork_exit+0x68/frame 0x12b6dcd4 fork_trampoline() at 0xffc0340e/frame 0x12b6dcd4 --- kthread start Tracing command kernel pid 0 tid 100020 td 0xb9f3880 sched_switch(b9f3880,104) at sched_switch+0x4e9/frame 0x12b77be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b77c20 sleepq_switch(b9f4e00,0) at sleepq_switch+0xf7/frame 0x12b77c40 _sleep(b9f4e00,b9f4e40,0,142fff2,0,0,0,0,100) at _sleep+0x2f3/frame 0x12b77c80 taskqueue_thread_loop(1dcfe98,12b77ce8) at taskqueue_thread_loop+0xd5/frame 0x12b77cb8 fork_exit(103c410,1dcfe98,12b77ce8,0,0,...) at fork_exit+0x68/frame 0x12b77cd4 fork_trampoline() at 0xffc0340e/frame 0x12b77cd4 --- kthread start Tracing command kernel pid 0 tid 100021 td 0xb9f3500 sched_switch(b9f3500,104) at sched_switch+0x4e9/frame 0x12b7cbe8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b7cc20 sleepq_switch(b9f4c00,0) at sleepq_switch+0xf7/frame 0x12b7cc40 _sleep(b9f4c00,b9f4c40,0,142fff2,0,0,0,0,100) at _sleep+0x2f3/frame 0x12b7cc80 taskqueue_thread_loop(1d4d820,12b7cce8) at taskqueue_thread_loop+0xd5/frame 0x12b7ccb8 fork_exit(103c410,1d4d820,12b7cce8,0,0,...) at fork_exit+0x68/frame 0x12b7ccd4 fork_trampoline() at 0xffc0340e/frame 0x12b7ccd4 --- kthread start Tracing command kernel pid 0 tid 100023 td 0xb9f2e00 sched_switch(b9f2e00,104) at sched_switch+0x4e9/frame 0x12b86be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b86c20 sleepq_switch(b9f4800,0) at sleepq_switch+0xf7/frame 0x12b86c40 _sleep(b9f4800,b9f4840,0,142fff2,0,0,0,0,100) at _sleep+0x2f3/frame 0x12b86c80 taskqueue_thread_loop(1dd18ec,12b86ce8) at taskqueue_thread_loop+0xd5/frame 0x12b86cb8 fork_exit(103c410,1dd18ec,12b86ce8,0,0,...) at fork_exit+0x68/frame 0x12b86cd4 fork_trampoline() at 0xffc0340e/frame 0x12b86cd4 --- kthread start Tracing command kernel pid 0 tid 100030 td 0xb9fa500 sched_switch(b9fa500,104) at sched_switch+0x4e9/frame 0x12bb9be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12bb9c20 sleepq_switch(b9f4000,0) at sleepq_switch+0xf7/frame 0x12bb9c40 _sleep(b9f4000,b9f4040,0,142fff2,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bb9c80 taskqueue_thread_loop(1d70cd4,12bb9ce8) at taskqueue_thread_loop+0xd5/frame 0x12bb9cb8 fork_exit(103c410,1d70cd4,12bb9ce8,0,0,...) at fork_exit+0x68/frame 0x12bb9cd4 fork_trampoline() at 0xffc0340e/frame 0x12bb9cd4 --- kthread start Tracing command kernel pid 0 tid 100035 td 0xb9f9380 sched_switch(b9f9380,104) at sched_switch+0x4e9/frame 0x12bd2be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12bd2c20 sleepq_switch(b9ff900,0) at sleepq_switch+0xf7/frame 0x12bd2c40 _sleep(b9ff900,b9ff940,0,142fff2,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bd2c80 taskqueue_thread_loop(1de4a50,12bd2ce8) at taskqueue_thread_loop+0xd5/frame 0x12bd2cb8 fork_exit(103c410,1de4a50,12bd2ce8,0,0,...) at fork_exit+0x68/frame 0x12bd2cd4 fork_trampoline() at 0xffc0340e/frame 0x12bd2cd4 --- kthread start Tracing command kernel pid 0 tid 100036 td 0xb9f9000 sched_switch(b9f9000,104) at sched_switch+0x4e9/frame 0x12bd7be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12bd7c20 sleepq_switch(b9ff900,0) at sleepq_switch+0xf7/frame 0x12bd7c40 _sleep(b9ff900,b9ff940,0,142fff2,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bd7c80 taskqueue_thread_loop(1de4a50,12bd7ce8) at taskqueue_thread_loop+0xd5/frame 0x12bd7cb8 fork_exit(103c410,1de4a50,12bd7ce8,0,0,...) at fork_exit+0x68/frame 0x12bd7cd4 fork_trampoline() at 0xffc0340e/frame 0x12bd7cd4 --- kthread start Tracing command kernel pid 0 tid 100037 td 0x9b02c00 sched_switch(9b02c00,104) at sched_switch+0x4e9/frame 0x12bdcbe8 mi_switch(104) at mi_switch+0x1c4/frame 0x12bdcc20 sleepq_switch(b9ff900,0) at sleepq_switch+0xf7/frame 0x12bdcc40 _sleep(b9ff900,b9ff940,0,142fff2,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bdcc80 taskqueue_thread_loop(1de4a50,12bdcce8) at taskqueue_thread_loop+0xd5/frame 0x12bdccb8 fork_exit(103c410,1de4a50,12bdcce8,0,0,...) at fork_exit+0x68/frame 0x12bdccd4 fork_trampoline() at 0xffc0340e/frame 0x12bdccd4 --- kthread start Tracing command kernel pid 0 tid 100038 td 0x9b02880 sched_switch(9b02880,104) at sched_switch+0x4e9/frame 0x12be1be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12be1c20 sleepq_switch(b9ff900,0) at sleepq_switch+0xf7/frame 0x12be1c40 _sleep(b9ff900,b9ff940,0,142fff2,0,0,0,0,100) at _sleep+0x2f3/frame 0x12be1c80 taskqueue_thread_loop(1de4a50,12be1ce8) at taskqueue_thread_loop+0xd5/frame 0x12be1cb8 fork_exit(103c410,1de4a50,12be1ce8,0,0,...) at fork_exit+0x68/frame 0x12be1cd4 fork_trampoline() at 0xffc0340e/frame 0x12be1cd4 --- kthread start Tracing command kernel pid 0 tid 100086 td 0x18313a80 sched_switch(18313a80,104) at sched_switch+0x4e9/frame 0xb567bf8 mi_switch(104) at mi_switch+0x1c4/frame 0xb567c2c sleepq_switch(18315800,0) at sleepq_switch+0xf7/frame 0xb567c4c msleep_spin_sbt(18315800,18315840,142fff2,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0xb567c84 taskqueue_thread_loop(1def49c,b567ce8) at taskqueue_thread_loop+0xfc/frame 0xb567cb8 fork_exit(103c410,1def49c,b567ce8,0,0,...) at fork_exit+0x68/frame 0xb567cd4 fork_trampoline() at 0xffc0340e/frame 0xb567cd4 --- kthread start Tracing command kernel pid 0 tid 100092 td 0x18313700 sched_switch(18313700,104) at sched_switch+0x4e9/frame 0xb58ebf8 mi_switch(104) at mi_switch+0x1c4/frame 0xb58ec2c sleepq_switch(187ff100,0) at sleepq_switch+0xf7/frame 0xb58ec4c msleep_spin_sbt(187ff100,187ff140,142fff2,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0xb58ec84 taskqueue_thread_loop(1ae8618,b58ece8) at taskqueue_thread_loop+0xfc/frame 0xb58ecb8 fork_exit(103c410,1ae8618,b58ece8,0,0,...) at fork_exit+0x68/frame 0xb58ecd4 fork_trampoline() at 0xffc0340e/frame 0xb58ecd4 --- kthread start Tracing command kernel pid 0 tid 100093 td 0x18313380 sched_switch(18313380,104) at sched_switch+0x4e9/frame 0xb593bf8 mi_switch(104) at mi_switch+0x1c4/frame 0xb593c2c sleepq_switch(187ff100,0) at sleepq_switch+0xf7/frame 0xb593c4c msleep_spin_sbt(187ff100,187ff140,142fff2,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0xb593c84 taskqueue_thread_loop(1ae8618,b593ce8) at taskqueue_thread_loop+0xfc/frame 0xb593cb8 fork_exit(103c410,1ae8618,b593ce8,0,0,...) at fork_exit+0x68/frame 0xb593cd4 fork_trampoline() at 0xffc0340e/frame 0xb593cd4 --- kthread start Tracing command kernel pid 0 tid 100094 td 0x18313000 sched_switch(18313000,104) at sched_switch+0x4e9/frame 0xb598bf8 mi_switch(104) at mi_switch+0x1c4/frame 0xb598c2c sleepq_switch(187ff100,0) at sleepq_switch+0xf7/frame 0xb598c4c msleep_spin_sbt(187ff100,187ff140,142fff2,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0xb598c84 taskqueue_thread_loop(1ae8618,b598ce8) at taskqueue_thread_loop+0xfc/frame 0xb598cb8 fork_exit(103c410,1ae8618,b598ce8,0,0,...) at fork_exit+0x68/frame 0xb598cd4 fork_trampoline() at 0xffc0340e/frame 0xb598cd4 --- kthread start Tracing command kernel pid 0 tid 100095 td 0x18924c00 sched_switch(18924c00,104) at sched_switch+0x4e9/frame 0xb59dbe8 mi_switch(104) at mi_switch+0x1c4/frame 0xb59dc20 sleepq_switch(9b0a500,0) at sleepq_switch+0xf7/frame 0xb59dc40 _sleep(9b0a500,9b0a540,0,142fff2,0,0,0,0,100) at _sleep+0x2f3/frame 0xb59dc80 taskqueue_thread_loop(1ae4670,b59dce8) at taskqueue_thread_loop+0xd5/frame 0xb59dcb8 fork_exit(103c410,1ae4670,b59dce8,0,0,...) at fork_exit+0x68/frame 0xb59dcd4 fork_trampoline() at 0xffc0340e/frame 0xb59dcd4 --- kthread start db:0:pho> dump Physical memory: 3428 MB Dumping 346 MB: 331 315 299 283 267 251 235 219 203 187 171 155 139 123 107 91 75 59 43 27 11 Dump complete db> reset