GENERIC HEAD from 2020-02-12 14:06:02 UTC, r357822M, vmcore.1180 GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb ---<>--- Copyright (c) 1992-2020 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 13.0-CURRENT #0 r357822M: Wed Feb 12 16:09:14 CET 2020 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 FreeBSD clang version 9.0.0 (tags/RELEASE_900/final 372316) (based on LLVM 9.0.0) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. VT(vga): resolution 640x480 CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.01-MHz 686-class CPU) Origin="AuthenticAMD" Id=0x100f23 Family=0x10 Model=0x2 Stepping=3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff SVM: NP,NAsids=64 TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 3522170880 (3359 MB) : Trying to mount root from ufs:/dev/ada0s1a [rw]... Root mount waiting for: CAMWARNING: WITNESS option enabled, expect reduced performance. usbus0WARNING: DIAGNOSTIC option enabled, expect reduced performance. usbus1ugen2.1: at usbus2 ugen3.1: at usbus3 ugen0.1: at usbus0 uhub0ugen4.1: at usbus4 on usbus2 uhub1 on usbus0 uhub2 on usbus3 uhub2: on usbus3 uhub0: on usbus2 uhub3 on usbus4 uhub1: on usbus0 uhub3: on usbus4 Expensive timeout(9) function: 0xa3de50(0x17eb9000) 0.009069309 s ugen1.1: at usbus1 usbus2ugen5.1: at usbus5 uhub4 on usbus1 usbus3uhub4: on usbus1 usbus4(aprobe0:ahcich0:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00 uhub5(aprobe0:ahcich0:0:15:0): CAM status: Command timeout usbus5(aprobe0:ahcich0:0:15:0): Error 5, Retries exhausted on usbus5 uhub5: on usbus5 (aprobe0:ahcich0:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00 (aprobe0:ahcich0:0:15:0): CAM status: Command timeout (aprobe0:ahcich0:0:15:0): Error 5, Retries exhausted ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ATA8-ACS SATA 2.x device ada0: Serial Number 9VMJ99G5 ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 476940MB (976773168 512 byte sectors) uhub2: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub0: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered Root mount waiting for: usbus5 Root mount waiting for: usbus5 Root mount waiting for: usbus5 Root mount waiting for: usbus5 uhub5: 10 ports with 10 removable, self powered mountroot: waiting for device /dev/ada0s1a... Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. warning: total configured swap (4194304 pages) exceeds maximum recommended amount (2009952 pages). warning: increase kern.maxswzone or reduce amount of swap. Starting file system checks: /dev/ada0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1a: clean, 605471 free (2239 frags, 75404 blocks, 0.2% fragmentation) /dev/ada0s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1f: clean, 721575 free (10959 frags, 88827 blocks, 1.1% fragmentation) /dev/label/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/label/tmp: clean, 34786717 free (26261 frags, 4345057 blocks, 0.0% fragmentation) /dev/ada0s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1d: clean, 900215 free (293599 frags, 75827 blocks, 2.9% fragmentation) /dev/ada0s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1e: clean, 5708942 free (163798 frags, 693143 blocks, 0.6% fragmentation) Mounting local filesystems:. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/arduino/lib /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/e2fsprogs /usr/local/lib/gcc7 /usr/local/lib/gcc8 /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/perl5/5.28/mach/CORE /usr/local/lib/pth /usr/local/lib/qt4 /usr/local/lib/qt5 /usr/local/llvm37/lib /usr/local/llvm39/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Setting hostname: x4.osted.lan. Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy: . lo0: link state changed to UP re0: link state changed to DOWN Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=8209b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 media: Ethernet autoselect (none) status: no carrier nd6 options=29 Starting devd. re0: link state changed to UP add host 127.0.0.1: gateway lo0 fib 0: route already in table add net default: gateway 192.168.1.1 add host ::1: gateway lo0 fib 0: route already in table add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Creating and/or trimming log files. Starting syslogd. Starting rpcbind. NFS access cache time=60 Starting watchdogd. No core dumps found. Clearing /tmp (X related). Starting nfsuserd. Starting mountd. Starting nfsd. Starting statd. Starting lockd. Recovering vi editor sessions:. Updating motd:. Mounting late filesystems:. Starting ntpd. Starting powerd. Starting smartd. Feb 12 16:27:58 x4 smartd[626]: Device: /dev/ada0, WARNING: A firmware update for this drive may be available, Feb 12 16:27:58 x4 smartd[626]: see the following Seagate web pages: Feb 12 16:27:58 x4 smartd[626]: http://knowledge.seagate.com/articles/en_US/FAQ/207931en Feb 12 16:27:58 x4 smartd[626]: http://knowledge.seagate.com/articles/en_US/FAQ/213891en Configuring vt: keymap blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting sendmail_submit. Starting sendmail_msp_queue. Starting cron. Local package initialization: backup. Starting nfscbd. Starting default mousedmoused: unable to open /dev/psm0: No such file or directory . Starting inetd. Wed Feb 12 16:28 FreeBSD/i386 (x4.osted.lan) (ttyu0) login: Feb 12 16:31:19 x4 su[913]: pho to root on /dev/pts/0 20200212 16:31:34 all (1/144): marcus5.sh lock order reversal: 1st 0x2597ba44 syncer (syncer) @ kern/vfs_subr.c:2466 2nd 0x2ae53764 ufs (ufs) @ kern/vfs_subr.c:3251 stack backtrace: #0 0x1035ed1 at witness_debugger+0x61 #1 0x1035cca at witness_checkorder+0x90a #2 0xfaaf3d at lockmgr_xlock_hard+0x4d #3 0xfabb7c at __lockmgr_args+0x6bc #4 0x12abdef at ffs_lock+0x6f #5 0x142ce58 at VOP_LOCK1_APV+0x38 #6 0x10a9d2f at _vn_lock+0x4f #7 0x1095a95 at vput_final+0x115 #8 0x1094f65 at vget_finish+0x105 #9 0x1094e35 at vget+0x65 #10 0x12aa668 at ffs_sync+0x4e8 #11 0x109f071 at sync_fsync+0xe1 #12 0x142bd1d at VOP_FSYNC_APV+0x4d #13 0x109cb83 at sched_sync+0x3e3 #14 0xf98ac9 at fork_exit+0x69 #15 0xffc033de at _end+0xfdda5c6e Feb 12 16:41:53 x4 kernel: pid 16375 (fts), jid 0, uid 1004, was killed: out of swap space lock order reversal: 1st 0x23625e94 ufs (ufs) @ kern/vfs_mount.c:1628 2nd 0x25980e94 devfs (devfs) @ ufs/ffs/ffs_vfsops.c:1404 stack backtrace: #0 0x1035ed1 at witness_debugger+0x61 #1 0x1035cca at witness_checkorder+0x90a #2 0xfaaa12 at lockmgr_lock_fast_path+0x132 #3 0x10841a4 at vop_stdlock+0x24 #4 0x142ce58 at VOP_LOCK1_APV+0x38 #5 0x10a9d2f at _vn_lock+0x4f #6 0x12a752b at ffs_flushfiles+0x9b #7 0x128c16b at softdep_flushfiles+0x9b #8 0x12a9bc2 at ffs_unmount+0x72 #9 0x108e048 at dounmount+0x3f8 #10 0x108dbed at kern_unmount+0x25d #11 0x108d983 at sys_unmount+0x13 #12 0x1417376 at syscall+0x306 #13 0xffc033c9 at _end+0xfdda5c59 pts leak: 1/2 20200212 16:48:17 all (2/144): pthread2.sh pts leak: 1/3 20200212 17:07:41 all (3/144): gnop.sh GEOM_NOP: Device md10.nop created. Feb 12 17:10:10 x4 kernel: pid 11191 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:10:20 x4 kernel: pid 11187 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:10:31 x4 kernel: pid 11138 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:10:41 x4 kernel: pid 11174 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:10:51 x4 kernel: pid 11142 (swap), jid 0, uid 1004, was killed: out of swap space lock order reversal: 1st 0x1e22300 cpslk (cpslk) @ i386/i386/copyout.c:121 2nd 0x18d2ce5c bufwait (bufwait) @ vm/vm_pager.c:391 stack backtrace: #0 0x1035ed1 at witness_debugger+0x61 #1 0x1035cca at witness_checkorder+0x90a #2 0xfaaf3d at lockmgr_xlock_hard+0x4d #3 0xfabb7c at __lockmgr_args+0x6bc #4 0x12f0199 at pbuf_ctor+0x79 #5 0x12c5f30 at item_ctor+0x330 #6 0x12c5541 at uma_zalloc_arg+0x171 #7 0x12be4c7 at swap_pager_getpages+0x2e7 #8 0x12f03c4 at vm_pager_get_pages+0x44 #9 0x12cdf02 at vm_fault+0x7b2 #10 0x12cd632 at vm_fault_trap+0x52 #11 0x1416d71 at trap_pfault+0x161 #12 0x1416318 at trap+0x3a8 #13 0xffc0316f at g_nop_debug+0xc7ffdfd3 #14 0x28 at ??+0 GEOM_NOP: Device md10.nop removed. GEOM_NOP: Device md10.nop created. Feb 12 17:14:41 x4 kernel: pid 14531 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:15:58 x4 kernel: pid 14533 (swap), jid 0, uid 1004, was killed: out of swap space GEOM_NOP: Device md10.nop removed. GEOM_NOP: Device md10.nop created. Feb 12 17:16:56 x4 kernel: pid 16351 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:17:06 x4 kernel: pid 16333 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:17:16 x4 kernel: pid 16324 (swap), jid 0, uid 1004, was killed: out of swap space GEOM_NOP: Device md10.nop removed. GEOM_NOP: Device md10.nop created. Feb 12 17:21:55 x4 kernel: pid 19889 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:22:05 x4 kernel: pid 19946 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:22:16 x4 kernel: pid 19903 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:24:15 x4 kernel: pid 22244 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:24:26 x4 kernel: pid 22257 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:24:36 x4 kernel: pid 22253 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:24:48 x4 kernel: pid 22298 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:25:00 x4 kernel: pid 22289 (swap), jid 0, uid 1004, was killed: out of swap space GEOM_NOP: Device md10.nop removed. mount leak: 4/101 20200212 17:25:50 all (4/144): umountf4.sh freework leak: 171/199. newblk leak: 596/633. mount leak: 1/102 20200212 17:28:00 all (5/144): umountf5.sh Feb 12 17:28:09 x4 kernel: pid 50125 (creat), uid 0 inumber 449 on /mnt: out of inodes Feb 12 17:28:15 x4 kernel: pid 50166 (swap), jid 0, uid 0, was killed: out of swap space Feb 12 17:28:26 x4 kernel: pid 50165 (swap), jid 0, uid 0, was killed: out of swap space Feb 12 17:28:45 x4 kernel: pid 50163 (swap), jid 0, uid 0, was killed: out of swap space Feb 12 17:29:24 x4 kernel: pid 50153 (swap), jid 0, uid 0, was killed: out of swap space 20200212 17:29:51 all (6/144): quota4.sh lock order reversal: 1st 0x23cf8448 FFS (FFS Lock) @ ufs/ufs/ufs_quota.c:759 2nd 0x207ba580 struct mount mtx (struct mount mtx) @ ufs/ufs/ufs_quota.c:765 stack backtrace: #0 0x1035ed1 at witness_debugger+0x61 #1 0x1035cca at witness_checkorder+0x90a #2 0xfb57d6 at __mtx_lock_flags+0x76 #3 0x12b6d7e at quotaoff_inchange+0x9e #4 0x12b6e53 at quotaoff+0x93 #5 0x12a75a7 at ffs_flushfiles+0x117 #6 0x128c16b at softdep_flushfiles+0x9b #7 0x12a9bc2 at ffs_unmount+0x72 #8 0x108e048 at dounmount+0x3f8 #9 0x108dbed at kern_unmount+0x25d #10 0x108d983 at sys_unmount+0x13 #11 0x1417376 at syscall+0x306 #12 0xffc033c9 at _end+0xfdda5c59 newblk leak: 62882/63515. mount leak: 1/103 20200212 17:32:25 all (7/144): trim.sh Feb 12 17:32:44 x4 kernel: pid 65044 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:32:54 x4 kernel: pid 65040 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:36:56 x4 kernel: pid 68865 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:37:07 x4 kernel: pid 68887 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:37:17 x4 kernel: pid 68908 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:39:11 x4 kernel: pid 70222 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:39:22 x4 kernel: pid 70229 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:39:27 x4 kernel: pid 70188 (creat), uid 1004 inumber 8193 on /mnt: out of inodes Feb 12 17:39:40 x4 kernel: pid 70112 (openat), uid 1004 inumber 8192 on /mnt: out of inodes Feb 12 17:39:42 x4 kernel: pid 70124 (rw), uid 1004 inumber 8196 on /mnt: out of inodes Feb 12 17:39:50 x4 kernel: pid 70128 (rw), uid 1004 inumber 4098 on /mnt: out of inodes Feb 12 17:40:00 x4 kernel: pid 70215 (creat), uid 1004 inumber 8199 on /mnt: out of inodes Feb 12 17:40:03 x4 kernel: pid 70104 (symlink), uid 1004 inumber 4121 on /mnt: out of inodes Feb 12 17:40:09 x4 kernel: pid 70187 (mkdir), uid 1004 inumber 4057 on /mnt: out of inodes Feb 12 17:41:14 x4 kernel: pid 71390 (link), uid 1004 inumber 4134 on /mnt: out of inodes Feb 12 17:41:24 x4 kernel: pid 71513 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:41:33 x4 kernel: pid 71511 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:41:41 x4 kernel: pid 71398 (creat), uid 1004 inumber 4114 on /mnt: out of inodes Feb 12 17:41:44 x4 kernel: pid 71518 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:41:48 x4 kernel: pid 71486 (mkdir), uid 1004 inumber 13037 on /mnt: out of inodes Feb 12 17:41:51 x4 kernel: pid 71406 (creat), uid 1004 inumber 4138 on /mnt: out of inodes Feb 12 17:41:56 x4 kernel: pid 71396 (creat), uid 1004 inumber 4126 on /mnt: out of inodes Feb 12 17:42:00 x4 kernel: pid 71479 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:42:04 x4 kernel: pid 71392 (creat), uid 1004 inumber 4128 on /mnt: out of inodes Feb 12 17:42:06 x4 kernel: pid 71396 (creat), uid 1004 inumber 4126 on /mnt: out of inodes Feb 12 17:42:08 x4 kernel: pid 71392 (creat), uid 1004 inumber 4128 on /mnt: out of inodes Feb 12 17:42:09 x4 kernel: pid 71383 (link), uid 1004 inumber 4136 on /mnt: out of inodes Feb 12 17:42:11 x4 kernel: pid 71493 (mkdir), uid 1004 inumber 4646 on /mnt: out of inodes Feb 12 17:42:19 x4 kernel: pid 71398 (creat), uid 1004 inumber 4114 on /mnt: out of inodes Feb 12 17:42:31 x4 kernel: pid 71403 (creat), uid 1004 inumber 4145 on /mnt: out of inodes Feb 12 17:42:34 x4 kernel: pid 71406 (creat), uid 1004 inumber 4138 on /mnt: out of inodes Feb 12 17:42:38 x4 kernel: pid 71389 (creat), uid 1004 inumber 4099 on /mnt: out of inodes Feb 12 17:42:41 x4 kernel: pid 71398 (creat), uid 1004 inumber 4114 on /mnt: out of inodes Feb 12 17:42:51 x4 kernel: pid 71392 (creat), uid 1004 inumber 4128 on /mnt: out of inodes Feb 12 17:42:54 x4 kernel: pid 71402 (creat), uid 1004 inumber 4137 on /mnt: out of inodes Feb 12 17:43:03 x4 syslogd: last message repeated 1 times Feb 12 17:43:24 x4 kernel: pid 72908 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:43:35 x4 kernel: pid 72904 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:43:45 x4 kernel: pid 72881 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:45:43 x4 kernel: pid 74146 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:45:54 x4 kernel: pid 74133 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:47:48 x4 kernel: pid 75501 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:48:05 x4 kernel: pid 75487 (swap), jid 0, uid 1004, was killed: out of swap space mount leak: 3/106 20200212 17:51:44 all (8/144): trim4.sh Feb 12 17:54:13 x4 kernel: pid 87079 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:54:24 x4 kernel: pid 87110 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 17:54:35 x4 kernel: pid 87095 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 18:00:38 x4 kernel: pid 99230 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 18:03:34 x4 kernel: pid 2423 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 18:03:45 x4 kernel: pid 2405 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 18:03:55 x4 kernel: pid 2414 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 18:04:06 x4 kernel: pid 2409 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 18:07:16 x4 kernel: pid 7290 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 18:07:27 x4 kernel: pid 7305 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 18:07:37 x4 kernel: pid 7310 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 18:09:09 x4 kernel: pid 8688 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 18:09:21 x4 kernel: pid 8687 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 18:09:41 x4 kernel: pid 8680 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 18:10:58 x4 kernel: pid 8682 (swap), jid 0, uid 1004, was killed: out of swap space mount leak: 3/109 20200212 18:11:09 all (9/144): mountro5.sh Feb 12 18:11:29 x4 kernel: pid 10046 (swap), jid 0, uid 0, was killed: out of swap space Feb 12 18:11:39 x4 kernel: pid 10037 (swap), jid 0, uid 0, was killed: out of swap space Feb 12 18:11:49 x4 kernel: pid 10034 (swap), jid 0, uid 0, was killed: out of swap space mount leak: 1/110 20200212 18:12:47 all (10/144): write.sh freework leak: 699/898. newblk leak: 12045/75560. 20200212 18:38:07 all (11/144): rename2.sh 20200212 18:58:31 all (12/144): buildworld3.sh mount leak: 1/111 20200212 19:18:39 all (13/144): core3.sh Feb 12 19:18:44 x4 kernel: Failed to write core file for process core3 (error 5) Feb 12 19:19:15 x4 syslogd: last message repeated 3086 times Feb 12 19:21:16 x4 syslogd: last message repeated 15688 times Feb 12 19:28:43 x4 syslogd: last message repeated 61399 times mount leak: 748/859 GEOM leak: 342/586 20200212 19:38:46 all (14/144): kinfo3.sh 20200212 19:58:56 all (15/144): quota10.sh 20200212 20:19:08 all (16/144): symlink.sh 20200212 20:39:40 all (17/144): suj.sh 20200212 20:59:57 all (18/144): linger4.sh 20200212 21:20:11 all (19/144): newfs2.sh mount leak: 15/874 20200212 21:40:40 all (20/144): kinfo.sh 20200212 22:00:48 all (21/144): suj18.sh Feb 12 22:03:50 x4 kernel: pid 86911 (rw), uid 1004 inumber 74429 on /mnt: filesystem full Feb 12 22:04:59 x4 kernel: pid 94666 (rw), uid 1004 inumber 74212 on /mnt: filesystem full mount leak: 1/875 20200212 22:21:12 all (22/144): ffs_syncvnode2.sh Feb 12 22:23:24 x4 kernel: pid 65905 (rw), uid 1004 inumber 65574 on /mnt: filesystem full Feb 12 22:24:01 x4 kernel: pid 65913 (mkdir), uid 1004 inumber 1508 on /mnt: filesystem full Feb 12 22:27:44 x4 kernel: pid 82238 (mkdir), uid 1004 inumber 66672 on /mnt: filesystem full Feb 12 22:28:25 x4 kernel: pid 82254 (rw), uid 1004 inumber 196912 on /mnt: filesystem full Feb 12 22:29:06 x4 kernel: pid 82262 (rw), uid 1004 inumber 196924 on /mnt: filesystem full Feb 12 22:29:49 x4 kernel: pid 91377 (mkdir), uid 1004 inumber 66768 on /mnt: filesystem full Feb 12 22:32:32 x4 kernel: pid 98058 (rw), uid 1004 inumber 131218 on /mnt: filesystem full Feb 12 22:40:10 x4 kernel: pid 17057 (rw), uid 1004 inumber 203368 on /mnt: filesystem full mount leak: 1/876 20200212 22:41:41 all (23/144): snap7.sh Feb 12 22:42:11 x4 mountd[550]: can't export /mnt lock order reversal: 1st 0x9abc048 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:302 2nd 0x25f5971c snaplk (snaplk) @ ufs/ffs/ffs_snapshot.c:2327 stack backtrace: #0 0x1035ed1 at witness_debugger+0x61 #1 0x1035cca at witness_checkorder+0x90a #2 0xfaaf3d at lockmgr_xlock_hard+0x4d #3 0xfabb7c at __lockmgr_args+0x6bc #4 0x128aeff at ffs_copyonwrite+0x18f #5 0x12aad83 at ffs_geom_strategy+0x73 #6 0x12bbe87 at ufs_strategy+0x87 #7 0x142d238 at VOP_STRATEGY_APV+0x58 #8 0x10731bf at bufstrategy+0x3f #9 0x107306a at bufwrite+0x1fa #10 0x12acc81 at ffs_syncvnode+0x2b1 #11 0x12abca8 at ffs_fsync+0x18 #12 0x142bd1d at VOP_FSYNC_APV+0x4d #13 0x12b9def at ufs_remove+0xbf #14 0x142be9d at VOP_REMOVE_APV+0x6d #15 0x10a2c65 at kern_funlinkat+0x285 #16 0x10a29cf at sys_unlink+0x1f #17 0x1417376 at syscall+0x306 lock order reversal: 1st 0x25f5971c snaplk (snaplk) @ ufs/ufs/ufs_vnops.c:918 2nd 0x2af811a4 ufs (ufs) @ ufs/ffs/ffs_snapshot.c:1650 stack backtrace: #0 0x1035ed1 at witness_debugger+0x61 #1 0x1035cca at witness_checkorder+0x90a #2 0xfaaf3d at lockmgr_xlock_hard+0x4d #3 0xfabb7c at __lockmgr_args+0x6bc #4 0x128930c at ffs_snapremove+0xdc #5 0x12834e0 at ffs_truncate+0x4a0 #6 0x12b26f5 at ufs_inactive+0x1e5 #7 0x142ca6d at VOP_INACTIVE_APV+0x4d #8 0x109674c at vinactivef+0xfc #9 0x1095c0c at vput_final+0x28c #10 0x1095ce8 at vput+0x88 #11 0x10a2cc1 at kern_funlinkat+0x2e1 #12 0x10a29cf at sys_unlink+0x1f #13 0x1417376 at syscall+0x306 #14 0xffc033c9 at tmpfs_node_pool+0xc7ff4d39 mount leak: 2397/3273 20200212 23:01:50 all (24/144): ffs_syncvnode.sh Feb 12 23:04:01 x4 kernel: pid 45447 (rw), uid 1004 inumber 131096 on /mnt: filesystem full Feb 12 23:20:11 x4 kernel: pid 94097 (rw), uid 1004 inumber 197019 on /mnt: filesystem full mount leak: 1/3274 20200212 23:22:13 all (25/144): stack_guard_page.sh Feb 12 23:22:35 x4 kernel: pid 180 (swap), jid 0, uid 1004, was killed: out of swap space Feb 12 23:22:46 x4 kernel: pid 234 (swap), jid 0, uid 1004, was killed: out of swap space swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio Feb 12 23:22:57 x4 kernel: pid 220 (swap), jid 0, uid 1004, was killed: out of swap space mount leak: 1/3275 20200212 23:24:11 all (26/144): namecache.sh 20200212 23:45:01 all (27/144): dfull.sh Feb 12 23:45:08 x4 kernel: pid 97427 (rw), uid 0 inumber 49163 on /mnt: filesystem full Feb 12 23:46:09 x4 kernel: pid 99673 (rw), uid 0 inumber 10 on /mnt: filesystem full Feb 12 23:47:11 x4 kernel: pid 2012 (rw), uid 0 inumber 743 on /mnt: filesystem full Feb 12 23:48:10 x4 kernel: pid 4657 (rw), uid 0 inumber 22 on /mnt: filesystem full Feb 12 23:49:11 x4 kernel: pid 9807 (rw), uid 0 inumber 49427 on /mnt: filesystem full Feb 12 23:50:19 x4 kernel: pid 11800 (rw), uid 0 inumber 12180 on /mnt: filesystem full Feb 12 23:51:13 x4 kernel: pid 14288 (rw), uid 0 inumber 34 on /mnt: filesystem full Feb 12 23:51:17 x4 kernel: pid 14287 (rw), uid 0 inumber 10 on /mnt: filesystem full Feb 12 23:51:20 x4 kernel: pid 14289 (rw), uid 0 inumber 1399 on /mnt: filesystem full Feb 12 23:51:38 x4 inetd[749]: discard/udp server failing (looping), service terminated Feb 12 23:52:13 x4 kernel: pid 20093 (rw), uid 0 inumber 3974 on /mnt: filesystem full Feb 12 23:53:12 x4 kernel: pid 23007 (rw), uid 0 inumber 49158 on /mnt: filesystem full Feb 12 23:54:14 x4 kernel: pid 25535 (rw), uid 0 inumber 49198 on /mnt: filesystem full Feb 12 23:54:17 x4 kernel: pid 25529 (rw), uid 0 inumber 51378 on /mnt: filesystem full Feb 12 23:55:16 x4 kernel: pid 29212 (rw), uid 0 inumber 12 on /mnt: filesystem full Feb 12 23:55:28 x4 kernel: pid 29211 (rw), uid 0 inumber 16438 on /mnt: filesystem full Feb 12 23:56:15 x4 kernel: pid 34578 (rw), uid 0 inumber 49 on /mnt: filesystem full Feb 12 23:57:15 x4 kernel: pid 37922 (rw), uid 0 inumber 49160 on /mnt: filesystem full Feb 12 23:58:17 x4 kernel: pid 39767 (rw), uid 0 inumber 49186 on /mnt: filesystem full Feb 12 23:58:20 x4 kernel: pid 39756 (rw), uid 0 inumber 49220 on /mnt: filesystem full Feb 12 23:59:17 x4 kernel: pid 43223 (rw), uid 0 inumber 6558 on /mnt: filesystem full Feb 12 23:59:23 x4 kernel: pid 43207 (rw), uid 0 inumber 3841 on /mnt: filesystem full Feb 12 23:59:26 x4 kernel: pid 43224 (rw), uid 0 inumber 9177 on /mnt: filesystem full Feb 13 00:00:17 x4 kernel: pid 48035 (rw), uid 0 inumber 50959 on /mnt: filesystem full Feb 13 00:00:19 x4 kernel: pid 48034 (rw), uid 0 inumber 50957 on /mnt: filesystem full Feb 13 00:00:21 x4 kernel: pid 48177 (rw), uid 0 inumber 49199 on /mnt: filesystem full Feb 13 00:00:40 x4 kernel: pid 48159 (rw), uid 0 inumber 50875 on /mnt: filesystem full Feb 13 00:01:16 x4 kernel: pid 48169 (rw), uid 0 inumber 53825 on /mnt: filesystem full Feb 13 00:01:22 x4 kernel: pid 52678 (rw), uid 0 inumber 893 on /mnt: filesystem full Feb 13 00:01:52 x4 inetd[749]: discard/udp server failing (looping), service terminated Feb 13 00:02:22 x4 kernel: pid 55680 (rw), uid 0 inumber 49290 on /mnt: filesystem full Feb 13 00:02:42 x4 kernel: pid 55673 (rw), uid 0 inumber 21255 on /mnt: filesystem full Feb 13 00:03:22 x4 kernel: pid 59517 (rw), uid 0 inumber 911 on /mnt: filesystem full Feb 13 00:03:23 x4 kernel: pid 59509 (rw), uid 0 inumber 927 on /mnt: filesystem full Feb 13 00:04:24 x4 kernel: pid 62778 (rw), uid 0 inumber 49160 on /mnt: filesystem full mount leak: 1/3276 20200213 00:05:25 all (28/144): symlink3.sh mount leak: 1/3277 20200213 00:25:58 all (29/144): zfs.sh ZFS NOTICE: Prefetch is disabled by default on i386 -- to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Feb 13 00:33:20 x4 kernel: pid 29889 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 00:36:21 x4 kernel: pid 34583 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 00:36:31 x4 kernel: pid 34571 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 00:36:43 x4 kernel: pid 34599 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 00:36:53 x4 kernel: pid 34604 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 00:37:45 x4 kernel: pid 34565 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 00:39:10 x4 kernel: pid 41657 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 00:39:20 x4 kernel: pid 41699 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 00:39:31 x4 kernel: pid 41660 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 00:40:10 x4 kernel: pid 41701 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 00:40:55 x4 kernel: pid 41676 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 00:41:43 x4 kernel: pid 45226 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 00:41:54 x4 kernel: pid 45243 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 00:42:19 x4 kernel: pid 45252 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 00:42:36 x4 kernel: pid 45320 (swap), jid 0, uid 0, was killed: out of swap space Expensive timeout(9) function: 0xe22d30(0x181a3aa0) 0.057812978 s Warning: memory type solaris leaked memory on destroy (4 allocations, 576 bytes leaked). 20200213 00:46:23 all (30/144): snap3.sh mount leak: 18/3295 20200213 01:07:15 all (31/144): sendfile10.sh lock order reversal: 1st 0x20cc6e38 so_snd_sx (so_snd_sx) @ kern/uipc_sockbuf.c:390 2nd 0x2fb6fa44 ufs (ufs) @ kern/kern_sendfile.c:767 stack backtrace: #0 0x1035ed1 at witness_debugger+0x61 #1 0x1035cca at witness_checkorder+0x90a #2 0xfaab6c at lockmgr_slock_hard+0x4c #3 0xfab5e3 at __lockmgr_args+0x123 #4 0x12abdef at ffs_lock+0x6f #5 0x142ce58 at VOP_LOCK1_APV+0x38 #6 0x10a9d2f at _vn_lock+0x4f #7 0xfd19fb at vn_sendfile+0x78b #8 0xfd37d5 at sendfile+0x125 #9 0xfd36a0 at sys_sendfile+0x10 #10 0x1417376 at syscall+0x306 #11 0xffc033c9 at tmpfs_node_pool+0xc7ff4d39 mount leak: 1/3296 20200213 01:09:30 all (32/144): thr2.sh Feb 13 01:11:46 x4 kernel: pid 79424 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 01:11:56 x4 kernel: pid 79423 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 01:12:09 x4 kernel: pid 79421 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 01:13:58 x4 kernel: pid 82752 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 01:14:08 x4 kernel: pid 82783 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 01:14:18 x4 kernel: pid 82773 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 01:16:07 x4 kernel: pid 85015 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 01:18:28 x4 kernel: pid 90899 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 01:18:40 x4 kernel: pid 90890 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 01:22:27 x4 kernel: pid 4323 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 01:22:38 x4 kernel: pid 4339 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 01:24:01 x4 kernel: pid 4319 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 01:24:38 x4 kernel: pid 8399 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 01:24:49 x4 kernel: pid 8372 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 01:24:59 x4 kernel: pid 8404 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 01:26:31 x4 kernel: pid 11726 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 01:26:44 x4 kernel: pid 11761 (swap), jid 0, uid 0, was killed: out of swap space 20200213 01:30:27 all (33/144): burnin.sh mount leak: 1/3297 20200213 01:51:21 all (34/144): snap5.sh mount leak: 1/3298 20200213 02:11:44 all (35/144): shm2.sh Feb 13 02:12:14 x4 kernel: pid 37708 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 02:12:25 x4 kernel: pid 37700 (swap), jid 0, uid 0, was killed: out of swap space 20200213 02:13:55 all (36/144): vmstat2.sh Feb 13 02:14:14 x4 kernel: pid 38936 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 02:14:29 x4 kernel: pid 38953 (swap), jid 0, uid 0, was killed: out of swap space 20200213 02:16:07 all (37/144): alternativeFlushPath.sh 20200213 02:37:20 all (38/144): md2.sh Feb 13 02:40:02 x4 kernel: pid 49878 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 02:41:55 x4 kernel: pid 49872 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 02:42:13 x4 kernel: pid 51080 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 02:46:20 x4 kernel: pid 55600 (swap), jid 0, uid 0, was killed: out of swap space g_vfs_done():md10[WRITE(offset=93454336, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=155648000, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=168001536, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=216399872, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=216432640, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=17039360, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=17072128, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=93454336, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=147324928, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=155648000, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=168001536, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=216399872, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=216432640, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=231112704, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=17039360, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=17072128, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=92700672, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=93454336, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=168001536, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=216432640, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=231112704, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=17039360, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=17072128, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=92700672, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=217251840, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=92700672, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=93913088, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=168493056, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=217251840, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=93913088, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=93913088, length=32768)]error = 28 swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio Feb 13 02:48:31 x4 kernel: pid 56835 (swap), jid 0, uid 0, was killed: out of swap space lock order reversal: 1st 0x20cc1c18 so_snd_sx (so_snd_sx) @ kern/uipc_sockbuf.c:393 2nd 0x18d27e5c bufwait (bufwait) @ vm/vm_pager.c:391 stack backtrace: #0 0x1035ed1 at witness_debugger+0x61 #1 0x1035cca at witness_checkorder+0x90a #2 0xfaaf3d at lockmgr_xlock_hard+0x4d #3 0xfabb7c at __lockmgr_args+0x6bc #4 0x12f0199 at pbuf_ctor+0x79 #5 0x12c5f30 at item_ctor+0x330 #6 0x12c5541 at uma_zalloc_arg+0x171 #7 0x12be4c7 at swap_pager_getpages+0x2e7 #8 0x12f03c4 at vm_pager_get_pages+0x44 #9 0x12cdf02 at vm_fault+0x7b2 #10 0x12cfc12 at vm_fault_quick_hold_pages+0x132 #11 0x13f13b6 at cp_slow0+0x56 #12 0x13f16b7 at copyin+0xa7 #13 0x102f6fe at uiomove_faultflag+0x12e #14 0x102f5c3 at uiomove+0x13 #15 0x10586d6 at m_uiotombuf+0x126 #16 0x10615dc at sosend_generic+0x2dc #17 0x10618c0 at sosend+0x50 g_vfs_done():md10[WRITE(offset=110854144, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=110886912, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=103219200, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=110919680, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=110952448, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=161677312, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=103219200, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=110919680, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=110952448, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=103219200, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=110592000, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=110919680, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=110952448, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=103219200, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=110592000, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=110919680, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=110952448, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=110592000, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=110919680, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=110952448, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=100270080, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=105218048, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=123011072, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=147521536, length=32768)]error = 28 Feb 13 02:48:49 x4 kernel: pid 56822 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 02:48:56 x4 kernel: pid 56807 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 02:50:45 x4 kernel: pid 58040 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 02:51:05 x4 kernel: pid 58056 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 02:51:24 x4 kernel: pid 58021 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 02:52:47 x4 kernel: pid 59263 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 02:53:00 x4 kernel: pid 59269 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 02:53:10 x4 kernel: pid 59234 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 02:53:20 x4 kernel: pid 59265 (swap), jid 0, uid 0, was killed: out of swap space g_vfs_done():md10[WRITE(offset=124944384, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=124977152, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=71696384, length=12288)]error = 28 g_vfs_done():md10[WRITE(offset=124944384, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=124977152, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=71610368, length=4096)]error = 28 g_vfs_done():md10[WRITE(offset=71761920, length=16384)]error = 28 g_vfs_done():md10[WRITE(offset=74526720, length=4096)]error = 28 g_vfs_done():md10[WRITE(offset=103612416, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=119111680, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=125272064, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=103612416, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=119111680, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=125272064, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=125304832, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=125337600, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=125370368, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=125304832, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=125337600, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=125370368, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=125403136, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=125435904, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=125468672, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=125403136, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=109445120, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=109477888, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=83525632, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=105742336, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=105807872, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=83525632, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=105709568, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=111411200, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=71860224, length=16384)]error = 28 g_vfs_done():md10[WRITE(offset=111411200, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=111411200, length=32768)]error = 28 Feb 13 02:54:55 x4 kernel: pid 60442 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 02:55:07 x4 kernel: pid 60466 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 02:55:23 x4 kernel: pid 60458 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 02:56:59 x4 kernel: pid 61674 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 02:57:10 x4 kernel: pid 61661 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 02:57:22 x4 kernel: pid 61712 (swap), jid 0, uid 0, was killed: out of swap space mount leak: 1/3299 20200213 02:58:53 all (39/144): softupdate.sh Feb 13 03:13:12 x4 kernel: pid 33148 (rw), uid 0 inumber 16566 on /mnt: filesystem full Feb 13 03:16:04 x4 kernel: pid 43134 (rw), uid 0 inumber 49554 on /mnt: filesystem full Feb 13 03:16:05 x4 kernel: pid 43129 (rw), uid 0 inumber 51487 on /mnt: filesystem full mount leak: 2/3301 20200213 03:20:50 all (40/144): graid1_10.sh mount leak: 1/3302 20200213 03:23:00 all (41/144): suj13.sh Feb 13 03:25:27 x4 kernel: pid 93724 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:25:40 x4 kernel: pid 93729 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:29:35 x4 kernel: pid 1141 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:29:46 x4 kernel: pid 1138 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:29:57 x4 kernel: pid 1149 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:30:20 x4 kernel: pid 1150 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:31:44 x4 kernel: pid 2323 (creat), uid 1004 inumber 4143 on /mnt: out of inodes Feb 13 03:31:53 x4 kernel: pid 2375 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:32:07 x4 kernel: pid 2332 (creat), uid 1004 inumber 4158 on /mnt: out of inodes Feb 13 03:32:14 x4 kernel: pid 2364 (mkdir), uid 1004 inumber 3750 on /mnt: out of inodes Feb 13 03:32:31 x4 kernel: pid 2401 (openat), uid 1004 inumber 4137 on /mnt: out of inodes Feb 13 03:32:42 x4 kernel: pid 2320 (creat), uid 1004 inumber 4151 on /mnt: out of inodes Feb 13 03:32:47 x4 kernel: pid 2408 (openat), uid 1004 inumber 4127 on /mnt: out of inodes Feb 13 03:35:53 x4 kernel: pid 5265 (creat), uid 1004 inumber 8271 on /mnt: out of inodes Feb 13 03:35:59 x4 kernel: pid 5218 (openat), uid 1004 inumber 8262 on /mnt: out of inodes Feb 13 03:36:08 x4 kernel: pid 5260 (creat), uid 1004 inumber 8281 on /mnt: out of inodes Feb 13 03:36:11 x4 kernel: pid 5265 (creat), uid 1004 inumber 8271 on /mnt: out of inodes Feb 13 03:36:14 x4 kernel: pid 5272 (creat), uid 1004 inumber 8298 on /mnt: out of inodes Feb 13 03:36:16 x4 kernel: pid 5268 (creat), uid 1004 inumber 8263 on /mnt: out of inodes Feb 13 03:36:19 x4 kernel: pid 5259 (creat), uid 1004 inumber 8267 on /mnt: out of inodes Feb 13 03:36:22 x4 kernel: pid 5268 (creat), uid 1004 inumber 8263 on /mnt: out of inodes Feb 13 03:36:25 x4 kernel: pid 5273 (creat), uid 1004 inumber 8273 on /mnt: out of inodes Feb 13 03:36:28 x4 kernel: pid 5271 (creat), uid 1004 inumber 8299 on /mnt: out of inodes Feb 13 03:36:30 x4 kernel: pid 5259 (creat), uid 1004 inumber 8267 on /mnt: out of inodes Feb 13 03:36:33 x4 kernel: pid 5281 (creat), uid 1004 inumber 8292 on /mnt: out of inodes Feb 13 03:36:40 x4 kernel: pid 5271 (creat), uid 1004 inumber 8299 on /mnt: out of inodes Feb 13 03:36:43 x4 kernel: pid 5273 (creat), uid 1004 inumber 8273 on /mnt: out of inodes Feb 13 03:36:46 x4 kernel: pid 5270 (creat), uid 1004 inumber 8295 on /mnt: out of inodes Feb 13 03:36:48 x4 kernel: pid 5284 (creat), uid 1004 inumber 8285 on /mnt: out of inodes Feb 13 03:36:51 x4 kernel: pid 5283 (creat), uid 1004 inumber 8294 on /mnt: out of inodes Feb 13 03:36:56 x4 kernel: pid 5268 (creat), uid 1004 inumber 8263 on /mnt: out of inodes Feb 13 03:36:59 x4 kernel: pid 5228 (link), uid 1004 inumber 8215 on /mnt: out of inodes Feb 13 03:37:04 x4 kernel: pid 5263 (creat), uid 1004 inumber 8276 on /mnt: out of inodes Feb 13 03:37:07 x4 kernel: pid 5258 (creat), uid 1004 inumber 8238 on /mnt: out of inodes Feb 13 03:37:11 x4 kernel: pid 5233 (link), uid 1004 inumber 8248 on /mnt: out of inodes Feb 13 03:37:14 x4 kernel: pid 5268 (creat), uid 1004 inumber 8263 on /mnt: out of inodes Feb 13 03:37:16 x4 kernel: pid 5284 (creat), uid 1004 inumber 8285 on /mnt: out of inodes Feb 13 03:37:18 x4 kernel: pid 5279 (creat), uid 1004 inumber 8289 on /mnt: out of inodes Feb 13 03:37:24 x4 kernel: pid 5263 (creat), uid 1004 inumber 8276 on /mnt: out of inodes Feb 13 03:37:26 x4 kernel: pid 5275 (creat), uid 1004 inumber 8283 on /mnt: out of inodes Feb 13 03:37:29 x4 kernel: pid 5270 (creat), uid 1004 inumber 8295 on /mnt: out of inodes Feb 13 03:37:32 x4 kernel: pid 5260 (creat), uid 1004 inumber 8281 on /mnt: out of inodes Feb 13 03:37:36 x4 kernel: pid 5275 (creat), uid 1004 inumber 8283 on /mnt: out of inodes Feb 13 03:40:02 x4 kernel: pid 15771 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:40:15 x4 kernel: pid 15809 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:40:25 x4 kernel: pid 15812 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:42:15 x4 kernel: pid 17148 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:42:25 x4 kernel: pid 17186 (swap), jid 0, uid 1004, was killed: out of swap space mount leak: 1/3303 20200213 03:44:04 all (42/144): ftruncate3.sh mount leak: 1/3304 GEOM leak: 29/615 20200213 03:46:40 all (43/144): tmpfs13.sh mount leak: 1/3305 20200213 03:48:56 all (44/144): graid3.sh GEOM_RAID3: Device raid3/data launched (3/3). Feb 13 03:49:22 x4 kernel: pid 29212 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:49:34 x4 kernel: pid 29236 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:49:45 x4 kernel: pid 29273 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:49:56 x4 kernel: pid 29233 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:51:30 x4 kernel: pid 30577 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:51:43 x4 kernel: pid 30590 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:51:56 x4 kernel: pid 30595 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:52:17 x4 kernel: pid 30603 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:52:28 x4 kernel: pid 30597 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:55:42 x4 kernel: pid 36644 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:55:54 x4 kernel: pid 36642 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:56:06 x4 kernel: pid 36629 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:56:17 x4 kernel: pid 36649 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:56:27 x4 kernel: pid 36647 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:56:39 x4 kernel: pid 36641 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:58:00 x4 kernel: pid 37968 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:58:13 x4 kernel: pid 37979 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 03:58:24 x4 kernel: pid 37966 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:00:13 x4 kernel: pid 39680 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:00:24 x4 kernel: pid 39676 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:00:36 x4 kernel: pid 39672 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:00:48 x4 kernel: pid 39671 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:01:01 x4 kernel: pid 39665 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:01:11 x4 kernel: pid 39688 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:01:24 x4 kernel: pid 39686 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:02:19 x4 kernel: pid 41012 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:02:33 x4 kernel: pid 41002 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:02:58 x4 kernel: pid 40956 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:02:58 x4 kernel: pid 41010 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:03:40 x4 kernel: pid 40997 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:04:44 x4 kernel: pid 42346 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:04:55 x4 kernel: pid 42330 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:06:46 x4 kernel: pid 43618 (rw), uid 1004 inumber 87957 on /mnt: filesystem full Feb 13 04:08:15 x4 kernel: pid 43600 (rw), uid 1004 inumber 87704 on /mnt: filesystem full Feb 13 04:08:34 x4 kernel: pid 27809 (xz), jid 0, uid 0, was killed: out of swap space Feb 13 04:08:45 x4 kernel: pid 48774 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:08:56 x4 kernel: pid 48773 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:09:08 x4 kernel: pid 48767 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:09:18 x4 kernel: pid 48762 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:09:28 x4 kernel: pid 48759 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:09:39 x4 kernel: pid 48758 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:09:49 x4 kernel: pid 48751 (swap), jid 0, uid 1004, was killed: out of swap space GEOM_RAID3: Device data: provider raid3/data destroyed. GEOM_RAID3: Device data destroyed. 20200213 04:10:58 all (45/144): suj12.sh GEOM_NOP: Device md10.nop created. Feb 13 04:11:31 x4 kernel: pid 51229 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:11:43 x4 kernel: pid 51266 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:11:55 x4 kernel: pid 51264 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:12:04 x4 kernel: pid 51265 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:15:55 x4 kernel: pid 60786 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:16:06 x4 kernel: pid 60783 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:16:31 x4 kernel: pid 60706 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:16:48 x4 kernel: pid 60700 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:18:28 x4 kernel: pid 62112 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:18:38 x4 kernel: pid 62113 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:18:50 x4 kernel: pid 62148 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:22:43 x4 kernel: pid 67107 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:22:52 x4 kernel: pid 67060 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:23:03 x4 kernel: pid 67063 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:24:53 x4 kernel: pid 69421 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:25:06 x4 kernel: pid 69420 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:25:17 x4 kernel: pid 69377 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:25:28 x4 kernel: pid 69395 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:25:44 x4 kernel: pid 69392 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:27:05 x4 kernel: pid 70680 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:27:16 x4 kernel: pid 70678 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:27:26 x4 kernel: pid 70686 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:27:39 x4 kernel: pid 70685 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:28:32 x4 kernel: pid 70684 (swap), jid 0, uid 1004, was killed: out of swap space GEOM_NOP: Device md10.nop removed. mount leak: 1/3306 20200213 04:31:51 all (46/144): nullfs19.sh mount leak: 1/3307 20200213 04:34:01 all (47/144): gbde.sh Feb 13 04:34:31 x4 kernel: pid 73081 (pkg), jid 0, uid 0, was killed: out of swap space Feb 13 04:34:44 x4 kernel: pid 86037 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:35:01 x4 kernel: pid 86090 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:36:39 x4 kernel: pid 88544 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:38:22 x4 kernel: pid 88530 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:39:04 x4 kernel: pid 90987 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:39:14 x4 kernel: pid 90985 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:39:26 x4 kernel: pid 90968 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:39:36 x4 kernel: pid 90953 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:41:46 x4 kernel: pid 94464 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:44:00 x4 kernel: pid 96932 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:44:12 x4 kernel: pid 96931 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:45:46 x4 kernel: pid 96896 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:46:22 x4 kernel: pid 99349 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:46:32 x4 kernel: pid 99347 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:50:44 x4 kernel: pid 13378 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:50:54 x4 kernel: pid 13388 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:51:05 x4 kernel: pid 13383 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:51:17 x4 kernel: pid 13379 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:52:00 x4 kernel: pid 13385 (swap), jid 0, uid 1004, was killed: out of swap space mount leak: 1/3308 20200213 04:55:23 all (48/144): fts.sh Feb 13 04:55:50 x4 kernel: pid 19782 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:55:58 x4 kernel: pid 19764 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:56:09 x4 kernel: pid 19776 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:57:51 x4 kernel: pid 21128 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:58:03 x4 kernel: pid 21110 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 04:59:57 x4 kernel: pid 23483 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 05:00:09 x4 kernel: pid 23550 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 05:00:19 x4 kernel: pid 23566 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 05:00:30 x4 kernel: pid 23554 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 05:06:13 x4 kernel: pid 27500 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 05:06:23 x4 kernel: pid 27515 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 05:06:34 x4 kernel: pid 27514 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 05:06:45 x4 kernel: pid 27508 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 05:09:08 x4 kernel: pid 28984 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 05:09:20 x4 kernel: pid 28980 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 05:09:29 x4 kernel: pid 28974 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 05:10:48 x4 kernel: pid 30347 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 05:10:54 x4 kernel: pid 30357 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 05:11:06 x4 kernel: pid 30356 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 05:12:41 x4 kernel: pid 31709 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 05:12:51 x4 kernel: pid 31712 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 05:13:21 x4 kernel: pid 31738 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 05:15:02 x4 kernel: pid 33124 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 05:15:10 x4 kernel: pid 33109 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 05:15:21 x4 kernel: pid 33119 (swap), jid 0, uid 1004, was killed: out of swap space mount leak: 1/3309 20200213 05:16:49 all (49/144): marcus2.sh Feb 13 05:17:42 x4 kernel: pid 35662 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 05:17:53 x4 kernel: pid 35669 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 05:18:20 x4 kernel: pid 35792 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 05:34:10 x4 kernel: pid 58405 (creat), uid 1004 inumber 196901 on /mnt: out of inodes mount leak: 1/3310 20200213 05:37:56 all (50/144): pthread7.sh 20200213 06:02:07 all (51/144): marcus.sh Feb 13 06:02:40 x4 kernel: pid 75394 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:02:50 x4 kernel: pid 75332 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:03:06 x4 kernel: pid 75308 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:03:13 x4 kernel: pid 75315 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:03:39 x4 kernel: pid 75402 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:03:51 x4 kernel: pid 75382 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:07:04 x4 kernel: pid 78012 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:07:14 x4 kernel: pid 78011 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:07:25 x4 kernel: pid 78008 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:09:04 x4 kernel: pid 79304 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:09:14 x4 kernel: pid 79292 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:11:08 x4 kernel: pid 81654 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:11:19 x4 kernel: pid 81670 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:11:34 x4 kernel: pid 81682 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:13:25 x4 kernel: pid 82970 (rw), uid 1004 inumber 70689 on /mnt: filesystem full Feb 13 06:14:07 x4 kernel: pid 83034 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:14:21 x4 kernel: pid 83074 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:14:29 x4 kernel: pid 83073 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:14:45 x4 kernel: pid 83041 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:15:35 x4 kernel: pid 84790 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:15:46 x4 kernel: pid 84894 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:15:56 x4 kernel: pid 84786 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:17:42 x4 kernel: pid 86970 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:17:52 x4 kernel: pid 86966 (swap), jid 0, uid 1004, was killed: out of swap space swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio Feb 13 06:18:03 x4 kernel: pid 86967 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:19:53 x4 kernel: pid 88280 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:20:04 x4 kernel: pid 88284 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:20:21 x4 kernel: pid 88268 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:22:03 x4 kernel: pid 89585 (swap), jid 0, uid 1004, was killed: out of swap space swap_pager: cannot allocate bio Feb 13 06:22:28 x4 kernel: pid 89563 (swap), jid 0, uid 1004, was killed: out of swap space mount leak: 1/3311 20200213 06:24:05 all (52/144): mkfifo.sh Feb 13 06:26:32 x4 kernel: pid 96875 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:26:42 x4 kernel: pid 96882 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:26:56 x4 kernel: pid 96878 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:27:07 x4 kernel: pid 96867 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:28:47 x4 kernel: pid 99230 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:28:59 x4 kernel: pid 99245 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:31:00 x4 kernel: pid 4119 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:31:10 x4 kernel: pid 4109 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:31:21 x4 kernel: pid 4105 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:33:15 x4 kernel: pid 6553 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:33:26 x4 kernel: pid 6554 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:33:36 x4 kernel: pid 6563 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:33:47 x4 kernel: pid 6565 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:35:11 x4 kernel: pid 7472 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:37:13 x4 kernel: pid 10195 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:37:23 x4 kernel: pid 10221 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:37:35 x4 kernel: pid 10226 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:37:45 x4 kernel: pid 10225 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:39:20 x4 kernel: pid 11781 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:41:27 x4 kernel: pid 14228 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:41:40 x4 kernel: pid 14232 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:41:51 x4 kernel: pid 14222 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:43:41 x4 kernel: pid 16774 (swap), jid 0, uid 1004, was killed: out of swap space swap_pager: cannot allocate bio Feb 13 06:43:51 x4 kernel: pid 16773 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 06:44:03 x4 kernel: pid 16772 (swap), jid 0, uid 1004, was killed: out of swap space mount leak: 1/3312 20200213 06:45:51 all (53/144): zfs3.sh ZFS NOTICE: Prefetch is disabled by default on i386 -- to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Warning: memory type solaris leaked memory on destroy (4 allocations, 576 bytes leaked). 20200213 07:00:58 all (54/144): suj31.sh Feb 13 07:03:24 x4 kernel: pid 93719 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:03:34 x4 kernel: pid 93734 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:03:49 x4 kernel: pid 93735 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:04:00 x4 kernel: pid 93736 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:04:11 x4 kernel: pid 93732 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:08:18 x4 kernel: pid 99948 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:08:28 x4 kernel: pid 99943 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:08:39 x4 kernel: pid 99875 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:08:49 x4 kernel: pid 99893 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:08:59 x4 kernel: pid 99912 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:10:30 x4 kernel: pid 1344 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:10:40 x4 kernel: pid 1335 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:11:16 x4 kernel: pid 1303 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:12:52 x4 kernel: pid 2667 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:13:02 x4 kernel: pid 2680 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:13:12 x4 kernel: pid 2676 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:13:22 x4 kernel: pid 2674 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:13:33 x4 kernel: pid 2701 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:17:14 x4 kernel: pid 11139 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:17:24 x4 kernel: pid 11140 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:17:35 x4 kernel: pid 11137 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:17:46 x4 kernel: pid 11144 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:19:44 x4 kernel: pid 14674 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:19:55 x4 kernel: pid 14675 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:20:05 x4 kernel: pid 14711 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:20:15 x4 kernel: pid 14698 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:20:25 x4 kernel: pid 14688 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:21:44 x4 kernel: pid 15946 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:21:56 x4 kernel: pid 16002 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:22:07 x4 kernel: pid 15995 (swap), jid 0, uid 1004, was killed: out of swap space Feb 13 07:22:19 x4 kernel: pid 15992 (swap), jid 0, uid 1004, was killed: out of swap space mount leak: 1/3313 20200213 07:23:58 all (55/144): lockf4.sh mount leak: 1/3314 20200213 07:45:05 all (56/144): dev3.sh lock order reversal: (sleepable after non-sleepable) 1st 0x9d15604 seqflq (seqflq) @ dev/sound/midi/sequencer.c:757 2nd 0x1bf921c midistat lock (midistat lock) @ dev/sound/midi/midi.c:1491 stack backtrace: #0 0x1035ed1 at witness_debugger+0x61 #1 0x1035cca at witness_checkorder+0x90a #2 0xfdd53d at _sx_xlock+0x4d #3 0xdc556a at midimapper_open+0x1a #4 0xdc7f1d at mseq_open+0xdd #5 0xea6900 at devfs_open+0x100 #6 0x142aa4d at VOP_OPEN_APV+0x4d #7 0x10a9b04 at vn_open_vnode+0x184 #8 0x10a94fe at vn_open_cred+0x39e #9 0x10a9150 at vn_open+0x20 #10 0x10a185c at kern_openat+0x23c #11 0x10a1acf at sys_openat+0x2f #12 0x1417376 at syscall+0x306 #13 0xffc033c9 at __stop_set_sysinit_set+0xc7fdd275 pts leak: 5/8 20200213 08:08:10 all (57/144): procstat.sh Feb 13 08:08:42 x4 kernel: pid 48004 (swap), jid 0, uid 0, was killed: out of swap space Feb 13 08:09:01 x4 kernel: pid 48009 (swap), jid 0, uid 0, was killed: out of swap space mount leak: 1/3315 20200213 08:10:45 all (58/144): vnodes.sh 20200213 08:13:19 all (59/144): syscall4.sh Kernel page fault with the following non-sleepable locks held: exclusive sleep mutex semid (semid) r = 0 (0x18342c3c) locked @ kern/sysv_sem.c:579 exclusive sleep mutex semid (semid) r = 0 (0x18342c28) locked @ kern/sysv_sem.c:579 exclusive sleep mutex semid (semid) r = 0 (0x18342c14) locked @ kern/sysv_sem.c:579 exclusive sleep mutex semid (semid) r = 0 (0x18342c00) locked @ kern/sysv_sem.c:751 exclusive sleep mutex sem (sem) r = 0 (0x1dfc794) locked @ kern/sysv_sem.c:750 stack backtrace: #0 0x1035ed1 at witness_debugger+0x61 #1 0x1036c85 at witness_warn+0x355 #2 0x1416c6a at trap_pfault+0x5a #3 0x1416318 at trap+0x3a8 #4 0xffc0316f at exit_tag+0xc7fd4a7f #5 0x28 at ??+0 Fatal trap 12: page fault while in kernel mode cpuid = 3; apic id = 03 fault virtual address = 0x1833f004 fault code = supervisor read data, page not present instruction pointer = 0x20:0x1046228 stack pointer = 0x28:0x289edb50 frame pointer = 0x28:0x289edb70 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 75723 (ipcrm) trap number = 12 panic: page fault cpuid = 3 time = 1581578014 KDB: stack backtrace: db_trace_self_wrapper(e643cc,1aa5e60,0,289ed94c,bd5380,...) at db_trace_self_wrapper+0x2a/frame 0x289ed920 kdb_backtrace(7,37637830,fce98d9a,289edb10,289edb10,...) at kdb_backtrace+0x2e/frame 0x289ed980 vpanic(1461fb6,289ed9c4,289ed9c4,289eda00,1416c07,...) at vpanic+0x11f/frame 0x289ed9a4 panic(1461fb6,14f09c1,14adb26,0,c,...) at panic+0x14/frame 0x289ed9b8 trap_fatal(2cb86000,1dbc480,2b31380,9890a6c,0,...) at trap_fatal+0x357/frame 0x289eda00 trap_pfault(1833f004,0,0) at trap_pfault+0x72/frame 0x289eda3c trap(289edb10,8,28,28,3ff,...) at trap+0x3a8/frame 0x289edb04 calltrap() at 0xffc0316f/frame 0x289edb04 --- trap 0xc, eip = 0x1046228, esp = 0x289edb50, ebp = 0x289edb70 --- sem_remove(289edbac,0,18342c10,1833d000,0,...) at sem_remove+0x208/frame 0x289edb70 kern_semctl(27069a80,10000,0,0,289edbc8,289edbd4) at kern_semctl+0x892/frame 0x289edba8 sys___semctl(27069a80,27069d0c) at sys___semctl+0xa0/frame 0x289edc08 syscall(289edce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x289edcdc Xint0x80_syscall() at 0xffc033c9/frame 0x289edcdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc015b0, esp = 0xffc0dfe8, ebp = 0xffbfe760 --- KDB: enter: panic [ thread pid 75723 tid 102315 ] Stopped at kdb_enter+0x35: movl $0,kdb_why db> run pho db:0:pho> set $lines 20000 db:0:pho> run pho1 db:1:pho1> bt Tracing pid 75723 tid 102315 td 0x27069a80 kdb_enter(1521cf1,1521cf1) at kdb_enter+0x35/frame 0x289ed978 vpanic(1461fb6,289ed9c4,289ed9c4,289eda00,1416c07,...) at vpanic+0x137/frame 0x289ed9a4 panic(1461fb6,14f09c1,14adb26,0,c,...) at panic+0x14/frame 0x289ed9b8 trap_fatal(2cb86000,1dbc480,2b31380,9890a6c,0,...) at trap_fatal+0x357/frame 0x289eda00 trap_pfault(1833f004,0,0) at trap_pfault+0x72/frame 0x289eda3c trap(289edb10,8,28,28,3ff,...) at trap+0x3a8/frame 0x289edb04 calltrap() at 0xffc0316f/frame 0x289edb04 --- trap 0xc, eip = 0x1046228, esp = 0x289edb50, ebp = 0x289edb70 --- sem_remove(289edbac,0,18342c10,1833d000,0,...) at sem_remove+0x208/frame 0x289edb70 kern_semctl(27069a80,10000,0,0,289edbc8,289edbd4) at kern_semctl+0x892/frame 0x289edba8 sys___semctl(27069a80,27069d0c) at sys___semctl+0xa0/frame 0x289edc08 syscall(289edce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x289edcdc Xint0x80_syscall() at 0xffc033c9/frame 0x289edcdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc015b0, esp = 0xffc0dfe8, ebp = 0xffbfe760 --- db:1:pho1> show allpcpu Current CPU: 3 cpuid = 0 dynamic pcpu = 0x7ecec0 curthread = 0x2ccaa180: pid 75735 tid 101049 critnest 2 "egrep" curpcb = 0x2ce5bd40 fpcurthread = 0x2ccaa180: pid 75735 "egrep" idlethread = 0x9df8e00: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 trampstk = 0xffc07ff0 kesp0 = 0x2ce5bd30 common_tssp = 0xffc01260 tlb gen = 424692445 curvnet = 0 spin locks held: cpuid = 1 dynamic pcpu = 0x8147ec0 curthread = 0x27011380: pid 73010 tid 100236 critnest 0 "swap" curpcb = 0x274d4d40 fpcurthread = 0x27011380: pid 73010 "swap" idlethread = 0x9df8a80: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 trampstk = 0xffc0aff0 kesp0 = 0x274d4d30 common_tssp = 0xffc012c8 tlb gen = 424692447 curvnet = 0 spin locks held: cpuid = 2 dynamic pcpu = 0x814dec0 curthread = 0x9df8700: pid 11 tid 100005 critnest 1 "idle: cpu2" curpcb = 0x1292fd40 fpcurthread = none idlethread = 0x9df8700: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 trampstk = 0xffc0bff0 kesp0 = 0x1292fd30 common_tssp = 0xffc01330 tlb gen = 424692447 curvnet = 0 spin locks held: cpuid = 3 dynamic pcpu = 0x8153ec0 curthread = 0x27069a80: pid 75723 tid 102315 critnest 1 "ipcrm" curpcb = 0x289edd40 fpcurthread = 0x27069a80: pid 75723 "ipcrm" idlethread = 0x9df8380: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 trampstk = 0xffc0dff0 kesp0 = 0x289edd30 common_tssp = 0xffc01398 tlb gen = 424692447 curvnet = 0 spin locks held: db:1:pho1> show alllocks Process 75735 (egrep) thread 0x2ccaa180 (101049) exclusive sleep mutex pmap (pmap) r = 0 (0x28c77f34) locked @ i386/i386/pmap.c:4781 exclusive rw pmap pv global (pmap pv global) r = 0 (0x1e1e2c0) locked @ i386/i386/pmap.c:4780 Process 75723 (ipcrm) thread 0x27069a80 (102315) exclusive sleep mutex semid (semid) r = 0 (0x18342c3c) locked @ kern/sysv_sem.c:579 exclusive sleep mutex semid (semid) r = 0 (0x18342c28) locked @ kern/sysv_sem.c:579 exclusive sleep mutex semid (semid) r = 0 (0x18342c14) locked @ kern/sysv_sem.c:579 exclusive sleep mutex semid (semid) r = 0 (0x18342c00) locked @ kern/sysv_sem.c:751 exclusive sleep mutex sem (sem) r = 0 (0x1dfc794) locked @ kern/sysv_sem.c:750 db:1:pho1> show lockedvnods Locked vnodes db:1:pho1> show allchains chain 1: thread 101734 (pid 75648, xargs) sleeping on 0x27e0f0a4 "wait" chain 2: thread 102375 (pid 75129, sh) sleeping on 0x27e0faa0 "wait" chain 3: thread 100790 (pid 73032, aiod4) sleeping on 0x2858b000 "aiordy" chain 4: thread 100814 (pid 73031, aiod3) sleeping on 0x2858caa0 "aiordy" chain 5: thread 100994 (pid 73030, aiod2) sleeping on 0x2cb87aa0 "aiordy" chain 6: thread 100513 (pid 73029, aiod1) sleeping on 0x27e29000 "aiordy" chain 7: thread 100789 (pid 73009, swap) sleeping on 0x2858b354 "wait" chain 8: thread 100184 (pid 73001, swap) sleeping on 0x1d901fe "nanslp" chain 9: thread 101048 (pid 72995, md10) sleeping on 0x5dce4000 "mdwait" chain 10: thread 100240 (pid 72481, sh) sleeping on 0x271e19fc "wait" chain 11: thread 100817 (pid 72480, tee) sleeping on 0x479ff000 "piperd" chain 12: thread 100242 (pid 72479, sh) sleeping on 0x271e1354 "wait" chain 13: thread 100712 (pid 46686, sleep) sleeping on 0x1d901fd "nanslp" chain 14: thread 100110 (pid 19975, top) sleeping on 0x27123264 "select" chain 15: thread 100653 (pid 95904, bash) sleeping on 0x27f62aa0 "wait" chain 16: thread 101282 (pid 95820, sshd) sleeping on 0x27ff6024 "select" chain 17: thread 100999 (pid 95490, sshd) sleeping on 0x236c0da4 "select" chain 18: thread 100974 (pid 3454, bash) sleeping on 0x28411354 "wait" chain 19: thread 100973 (pid 3453, sshd) sleeping on 0x9df49e4 "select" chain 20: thread 100972 (pid 3451, sshd) sleeping on 0x28dcf424 "select" chain 21: thread 100209 (pid 1494, sh) sleeping on 0x204861a8 "piperd" chain 22: thread 100214 (pid 1493, awk) sleeping on 0x20487000 "piperd" chain 23: thread 100182 (pid 1492, sh) sleeping on 0x181839fc "wait" chain 24: thread 100176 (pid 1471, sh) sleeping on 0x23cd20a4 "wait" chain 25: thread 100180 (pid 952, sh) sleeping on 0x23cd16a8 "wait" chain 26: thread 100130 (pid 915, bash) sleeping on 0x1818474c "wait" chain 27: thread 100203 (pid 913, csh) sleeping on 0x23ccb70c "pause" chain 28: thread 100191 (pid 912, su) sleeping on 0x205d1000 "wait" chain 29: thread 100192 (pid 901, bash) sleeping on 0x18890aa0 "wait" chain 30: thread 100202 (pid 900, sshd) sleeping on 0x183acf24 "select" chain 31: thread 100113 (pid 898, sshd) sleeping on 0x183ad1e4 "select" chain 32: thread 100199 (pid 774, getty) sleeping on 0x18210c74 "ttyin" chain 33: thread 100198 (pid 773, getty) sleeping on 0x18966c74 "ttyin" chain 34: thread 100197 (pid 772, getty) sleeping on 0x18967074 "ttyin" chain 35: thread 100196 (pid 771, getty) sleeping on 0x18967474 "ttyin" chain 36: thread 100195 (pid 770, getty) sleeping on 0x18967874 "ttyin" chain 37: thread 100190 (pid 769, getty) sleeping on 0x18967c74 "ttyin" chain 38: thread 100193 (pid 768, getty) sleeping on 0x18968074 "ttyin" chain 39: thread 100194 (pid 767, getty) sleeping on 0x18968474 "ttyin" chain 40: thread 100105 (pid 766, getty) sleeping on 0x18968874 "ttyin" chain 41: thread 100171 (pid 749, inetd) sleeping on 0x183ad224 "select" chain 42: thread 100178 (pid 715, nfscbd: master) sleeping on 0x9935b50 "rpcsvc" chain 43: thread 100186 (pid 715, nfscbd: service) sleeping on 0x188d5a50 "rpcsvc" chain 44: thread 100187 (pid 715, nfscbd: service) sleeping on 0x188d59d0 "rpcsvc" chain 45: thread 100188 (pid 715, nfscbd: service) sleeping on 0x188d5950 "rpcsvc" chain 46: thread 100132 (pid 714, nfscbd) sleeping on 0x20cc1948 "accept" chain 47: thread 100109 (pid 678, cron) sleeping on 0x1d901fd "nanslp" chain 48: thread 100174 (pid 674, sendmail) sleeping on 0x205d1a60 "pause" chain 49: thread 100134 (pid 671, sendmail) sleeping on 0x9df5c24 "select" chain 50: thread 100124 (pid 668, sshd) sleeping on 0x183ad3a4 "select" chain 51: thread 100123 (pid 628, smartd) sleeping on 0x1d901fd "nanslp" chain 52: thread 100114 (pid 604, powerd) sleeping on 0x183ad7a4 "select" chain 53: thread 100133 (pid 601, ntpd) sleeping on 0x187df264 "select" chain 54: thread 100170 (pid 563, NLM: master) sleeping on 0x188d5c50 "rpcsvc" chain 55: thread 100169 (pid 560, rpc.statd) sleeping on 0x17ee05a4 "select" chain 56: thread 100137 (pid 559, nfsd: master) sleeping on 0x188d6c50 "rpcsvc" chain 57: thread 100138 (pid 559, nfsd: service) sleeping on 0x188d61d0 "rpcsvc" chain 58: thread 100139 (pid 559, nfsd: service) sleeping on 0x188d6150 "rpcsvc" chain 59: thread 100140 (pid 559, nfsd: service) sleeping on 0x188d6b50 "rpcsvc" chain 60: thread 100141 (pid 559, nfsd: service) sleeping on 0x99366d0 "rpcsvc" chain 61: thread 100142 (pid 559, nfsd: service) sleeping on 0x9936650 "rpcsvc" chain 62: thread 100143 (pid 559, nfsd: service) sleeping on 0x99365d0 "rpcsvc" chain 63: thread 100144 (pid 559, nfsd: service) sleeping on 0x188d60d0 "rpcsvc" chain 64: thread 100145 (pid 559, nfsd: service) sleeping on 0x9936550 "rpcsvc" chain 65: thread 100146 (pid 559, nfsd: service) sleeping on 0x99364d0 "rpcsvc" chain 66: thread 100147 (pid 559, nfsd: service) sleeping on 0x9936450 "rpcsvc" chain 67: thread 100148 (pid 559, nfsd: service) sleeping on 0x188d6050 "rpcsvc" chain 68: thread 100149 (pid 559, nfsd: service) sleeping on 0x99363d0 "rpcsvc" chain 69: thread 100150 (pid 559, nfsd: service) sleeping on 0x188d5f50 "rpcsvc" chain 70: thread 100151 (pid 559, nfsd: service) sleeping on 0x9936350 "rpcsvc" chain 71: thread 100152 (pid 559, nfsd: service) sleeping on 0x99362d0 "rpcsvc" chain 72: thread 100153 (pid 559, nfsd: service) sleeping on 0x9936250 "rpcsvc" chain 73: thread 100154 (pid 559, nfsd: service) sleeping on 0x188d5ed0 "rpcsvc" chain 74: thread 100155 (pid 559, nfsd: service) sleeping on 0x99361d0 "rpcsvc" chain 75: thread 100156 (pid 559, nfsd: service) sleeping on 0x9936150 "rpcsvc" chain 76: thread 100157 (pid 559, nfsd: service) sleeping on 0x188d5e50 "rpcsvc" chain 77: thread 100158 (pid 559, nfsd: service) sleeping on 0x99360d0 "rpcsvc" chain 78: thread 100159 (pid 559, nfsd: service) sleeping on 0x9936050 "rpcsvc" chain 79: thread 100160 (pid 559, nfsd: service) sleeping on 0x9935f50 "rpcsvc" chain 80: thread 100161 (pid 559, nfsd: service) sleeping on 0x9935ed0 "rpcsvc" chain 81: thread 100162 (pid 559, nfsd: service) sleeping on 0x188d5dd0 "rpcsvc" chain 82: thread 100163 (pid 559, nfsd: service) sleeping on 0x188d5d50 "rpcsvc" chain 83: thread 100164 (pid 559, nfsd: service) sleeping on 0x9935e50 "rpcsvc" chain 84: thread 100165 (pid 559, nfsd: service) sleeping on 0x188d5cd0 "rpcsvc" chain 85: thread 100166 (pid 559, nfsd: service) sleeping on 0x9935dd0 "rpcsvc" chain 86: thread 100167 (pid 559, nfsd: service) sleeping on 0x9935d50 "rpcsvc" chain 87: thread 100168 (pid 559, nfsd: service) sleeping on 0x9935cd0 "rpcsvc" chain 88: thread 100136 (pid 556, nfsd) sleeping on 0x183ad4a4 "select" chain 89: thread 100111 (pid 550, mountd) sleeping on 0x9df5b64 "select" chain 90: thread 100135 (pid 543, nfsuserd) sleeping on 0x9df5fa4 "select" chain 91: thread 100120 (pid 542, nfsuserd) sleeping on 0x9df5be4 "select" chain 92: thread 100127 (pid 541, nfsuserd) sleeping on 0x187df224 "select" chain 93: thread 100131 (pid 540, nfsuserd) sleeping on 0x187df2a4 "select" chain 94: thread 100121 (pid 539, nfsuserd) sleeping on 0x1888970c "pause" chain 95: thread 100125 (pid 496, watchdogd) sleeping on 0x1d901fe "nanslp" chain 96: thread 100129 (pid 411, rpcbind) sleeping on 0x183ad9e4 "select" chain 97: thread 100128 (pid 402, syslogd) sleeping on 0x183ad864 "select" chain 98: thread 100122 (pid 330, devd) sleeping on 0x187deda4 "select" chain 99: thread 100104 (pid 25, vnlru) sleeping on 0x187b39fc "vlruwt" chain 100: thread 100103 (pid 24, syncer) sleeping on 0x1dfd128 "syncer" chain 101: thread 100099 (pid 23, bufdaemon) sleeping on 0x1dfcaec "qsleep" chain 102: thread 100102 (pid 23, bufspacedaemon-0) sleeping on 0x1aca040 "-" chain 103: thread 100116 (pid 23, /home worker) sleeping on 0x18da4884 "sdflush" chain 104: thread 100117 (pid 23, /tmp worker) sleeping on 0x99d5e84 "sdflush" chain 105: thread 100118 (pid 23, /usr worker) sleeping on 0x1819f084 "sdflush" chain 106: thread 100119 (pid 23, /var worker) sleeping on 0x18da4e84 "sdflush" chain 107: thread 101686 (pid 23, /mnt worker) sleeping on 0x48009c84 "sdflush" chain 108: thread 100098 (pid 22, vmdaemon) sleeping on 0x1e11e30 "psleep" chain 109: thread 100097 (pid 21, dom0) sleeping on 0x1e4ad5c "psleep" chain 110: thread 100100 (pid 21, laundry: dom0) sleeping on 0x1e4ad68 "launds" chain 111: thread 100101 (pid 21, uma) sleeping on 0x12c8a10 "umarcl" chain 112: thread 100094 (pid 20, rand_harvestq) sleeping on 0x1bf407c "-" chain 113: thread 100090 (pid 19, sctp_iterator) sleeping on 0x1e5bd78 "waiting_for_work" chain 114: thread 100089 (pid 18, soaiod4) sleeping on 0x1dfc728 "-" chain 115: thread 100088 (pid 17, soaiod3) sleeping on 0x1dfc728 "-" chain 116: thread 100087 (pid 16, soaiod2) sleeping on 0x1dfc728 "-" chain 117: thread 100086 (pid 9, soaiod1) sleeping on 0x1dfc728 "-" chain 118: thread 100083 (pid 8, fdc0) sleeping on 0x1813023c "-" chain 119: thread 100048 (pid 15, usbus0) sleeping on 0x17ec2aac "-" chain 120: thread 100049 (pid 15, usbus0) sleeping on 0x17ec2adc "-" chain 121: thread 100050 (pid 15, usbus0) sleeping on 0x17ec2b0c "-" chain 122: thread 100051 (pid 15, usbus0) sleeping on 0x17ec2b3c "-" chain 123: thread 100052 (pid 15, usbus0) sleeping on 0x17ec2b6c "-" chain 124: thread 100054 (pid 15, usbus1) sleeping on 0x181a3aac "-" chain 125: thread 100055 (pid 15, usbus1) sleeping on 0x181a3adc "-" chain 126: thread 100056 (pid 15, usbus1) sleeping on 0x181a3b0c "-" chain 127: thread 100057 (pid 15, usbus1) sleeping on 0x181a3b3c "-" chain 128: thread 100058 (pid 15, usbus1) sleeping on 0x181a3b6c "-" chain 129: thread 100060 (pid 15, usbus2) sleeping on 0x181afaac "-" chain 130: thread 100061 (pid 15, usbus2) sleeping on 0x181afadc "-" chain 131: thread 100062 (pid 15, usbus2) sleeping on 0x181afb0c "-" chain 132: thread 100063 (pid 15, usbus2) sleeping on 0x181afb3c "-" chain 133: thread 100064 (pid 15, usbus2) sleeping on 0x181afb6c "-" chain 134: thread 100065 (pid 15, usbus3) sleeping on 0x181b4aac "-" chain 135: thread 100066 (pid 15, usbus3) sleeping on 0x181b4adc "-" chain 136: thread 100067 (pid 15, usbus3) sleeping on 0x181b4b0c "-" chain 137: thread 100068 (pid 15, usbus3) sleeping on 0x181b4b3c "-" chain 138: thread 100069 (pid 15, usbus3) sleeping on 0x181b4b6c "-" chain 139: thread 100070 (pid 15, usbus4) sleeping on 0x181beaac "-" chain 140: thread 100071 (pid 15, usbus4) sleeping on 0x181beadc "-" chain 141: thread 100072 (pid 15, usbus4) sleeping on 0x181beb0c "-" chain 142: thread 100073 (pid 15, usbus4) sleeping on 0x181beb3c "-" chain 143: thread 100074 (pid 15, usbus4) sleeping on 0x181beb6c "-" chain 144: thread 100076 (pid 15, usbus5) sleeping on 0x17ebdc84 "-" chain 145: thread 100077 (pid 15, usbus5) sleeping on 0x17ebdcb4 "-" chain 146: thread 100078 (pid 15, usbus5) sleeping on 0x17ebdce4 "-" chain 147: thread 100079 (pid 15, usbus5) sleeping on 0x17ebdd14 "-" chain 148: thread 100080 (pid 15, usbus5) sleeping on 0x17ebdd44 "-" chain 149: thread 100043 (pid 7, doneq0) sleeping on 0x1b2d540 "-" chain 150: thread 100096 (pid 7, scanner) sleeping on 0x1b2d468 "-" chain 151: thread 100042 (pid 6, crypto returns 3) sleeping on 0x9906dc4 "crypto_ret_wait" chain 152: thread 100041 (pid 5, crypto returns 2) sleeping on 0x9906d8c "crypto_ret_wait" chain 153: thread 100040 (pid 4, crypto returns 1) sleeping on 0x9906d54 "crypto_ret_wait" chain 154: thread 100039 (pid 3, crypto returns 0) sleeping on 0x9906d1c "crypto_ret_wait" chain 155: thread 100038 (pid 2, crypto) sleeping on 0x1e10b28 "crypto_wait" chain 156: thread 100033 (pid 14, sequencer 00) sleeping on 0x9d1564c "seqstate" chain 157: thread 100030 (pid 13, g_event) sleeping on 0x1e4a2f0 "-" chain 158: thread 100031 (pid 13, g_up) sleeping on 0x1e4a2f8 "-" chain 159: thread 100032 (pid 13, g_down) sleeping on 0x1e4a2f4 "-" chain 160: thread 100002 (pid 1, init) sleeping on 0x9df10a4 "wait" chain 161: thread 100001 (pid 10, audit) sleeping on 0x1e5c50c "audit_worker_cv" chain 162: thread 100000 (pid 0, swapper) sleeping on 0x1e1fce0 "swapin" chain 163: thread 100008 (pid 0, thread taskq) sleeping on 0x9d3b600 "-" chain 164: thread 100010 (pid 0, config_0) sleeping on 0x9d36e00 "-" chain 165: thread 100011 (pid 0, kqueue_ctx taskq) sleeping on 0x9d3b100 "-" chain 166: thread 100012 (pid 0, aiod_kick taskq) sleeping on 0x9d3ae00 "-" chain 167: thread 100014 (pid 0, softirq_0) sleeping on 0x9d36d00 "-" chain 168: thread 100015 (pid 0, softirq_1) sleeping on 0x9d36c00 "-" chain 169: thread 100016 (pid 0, softirq_2) sleeping on 0x9d36b00 "-" chain 170: thread 100017 (pid 0, softirq_3) sleeping on 0x9d36a00 "-" chain 171: thread 100018 (pid 0, if_io_tqg_0) sleeping on 0x9d36900 "-" chain 172: thread 100019 (pid 0, if_io_tqg_1) sleeping on 0x9d36800 "-" chain 173: thread 100020 (pid 0, if_io_tqg_2) sleeping on 0x9d36700 "-" chain 174: thread 100021 (pid 0, if_io_tqg_3) sleeping on 0x9d36600 "-" chain 175: thread 100022 (pid 0, if_config_tqg_0) sleeping on 0x9d36500 "-" chain 176: thread 100029 (pid 0, firmware taskq) sleeping on 0x9d39a00 "-" chain 177: thread 100034 (pid 0, crypto_0) sleeping on 0x9d39400 "-" chain 178: thread 100035 (pid 0, crypto_1) sleeping on 0x9d39400 "-" chain 179: thread 100036 (pid 0, crypto_2) sleeping on 0x9d39400 "-" chain 180: thread 100037 (pid 0, crypto_3) sleeping on 0x9d39400 "-" chain 181: thread 100085 (pid 0, mca taskq) sleeping on 0x17ee5400 "-" chain 182: thread 100091 (pid 0, acpi_task_0) sleeping on 0x183a9d00 "-" chain 183: thread 100092 (pid 0, acpi_task_1) sleeping on 0x183a9d00 "-" chain 184: thread 100093 (pid 0, acpi_task_2) sleeping on 0x183a9d00 "-" chain 185: thread 100095 (pid 0, CAM taskq) sleeping on 0x9d38900 "-" db:1:pho1> show mount 0x18dc7000 /dev/ada0s1a on / (ufs) 0x18ca3040 devfs on /dev (devfs) 0x18dc9b00 /dev/ada0s1f on /home (ufs) 0x18dc9040 /dev/label/tmp on /tmp (ufs) 0x18dc8580 /dev/ada0s1d on /usr (ufs) 0x18dc7ac0 /dev/ada0s1e on /var (ufs) 0x207b9ac0 procfs on /proc (procfs) 0x207ba580 /dev/md10a on /mnt (ufs) More info: show mount db:1:pho1> show bufqueues bqempty: 1793 Buf domain 0 freebufs 1864 lofreebufs 311 hifreebufs 466 bufspace 60483584 maxbufspace 94683136 hibufspace 94027776 lobufspace 89326372 bufspacethresh 91677074 numdirtybuffers 21 lodirtybuffers 732 hidirtybuffers 1464 dirtybufthresh 1317 cleanq count 3860 (59602944) dirtyq count 21 (372736) wakeup 0 lim 28 CPU 0, 26, 8, 0, Locked buffers: 0 space 0 Total buffers: 5779 space 60483584 db:0:pho> ps pid ppid pgrp uid state wmesg wchan cmd 75735 75129 1471 0 RE CPU 0 egrep 75723 75648 952 0 R+ CPU 3 ipcrm 75648 72481 952 0 S+ wait 0x27e0f0a4 xargs 75129 1492 1471 0 S wait 0x27e0faa0 sh 73032 0 0 0 DL aiordy 0x2858b000 [aiod4] 73031 0 0 0 DL aiordy 0x2858caa0 [aiod3] 73030 0 0 0 DL aiordy 0x2cb87aa0 [aiod2] 73029 0 0 0 DL aiordy 0x27e29000 [aiod1] 73010 73009 73001 0 R CPU 1 swap 73009 73001 73001 0 S wait 0x2858b354 swap 73001 1 73001 0 Ss nanslp 0x1d901fe swap 72995 0 0 0 DL mdwait 0x5dce4000 [md10] 72481 72479 952 0 S+ wait 0x271e19fc sh 72480 952 952 0 S+ piperd 0x479ff000 tee 72479 952 952 0 S+ wait 0x271e1354 sh 46686 3454 46686 1001 S+ nanslp 0x1d901fd sleep 19975 95904 19975 1001 S+ select 0x27123264 top 95904 95820 95904 1001 SWs+ wait 0x27f62aa0 bash 95820 95490 95490 1001 S select 0x27ff6024 sshd 95490 668 95490 0 Ss select 0x236c0da4 sshd 3454 3453 3454 1001 Ss+ wait 0x28411354 bash 3453 3451 3451 1001 S select 0x9df49e4 sshd 3451 668 3451 0 Ss select 0x28dcf424 sshd 1494 1471 1471 0 S piperd 0x204861a8 sh 1493 1471 1471 0 S piperd 0x20487000 awk 1492 1471 1471 0 S wait 0x181839fc sh 1471 1 1471 0 SWs wait 0x23cd20a4 sh 952 915 952 0 S+ wait 0x23cd16a8 sh 915 913 915 0 SW+ wait 0x1818474c bash 913 912 913 0 SW+ pause 0x23ccb70c csh 912 901 912 1001 SW+ wait 0x205d1000 su 901 900 901 1001 SWs+ wait 0x18890aa0 bash 900 898 898 1001 S select 0x183acf24 sshd 898 668 898 0 Ss select 0x183ad1e4 sshd 774 1 774 0 Ss+ ttyin 0x18210c74 getty 773 1 773 0 Ss+ ttyin 0x18966c74 getty 772 1 772 0 Ss+ ttyin 0x18967074 getty 771 1 771 0 Ss+ ttyin 0x18967474 getty 770 1 770 0 Ss+ ttyin 0x18967874 getty 769 1 769 0 Ss+ ttyin 0x18967c74 getty 768 1 768 0 Ss+ ttyin 0x18968074 getty 767 1 767 0 Ss+ ttyin 0x18968474 getty 766 1 766 0 Ss+ ttyin 0x18968874 getty 749 1 749 0 Ss select 0x183ad224 inetd 715 714 714 0 S (threaded) nfscbd 100178 S rpcsvc 0x9935b50 nfscbd: master 100186 S rpcsvc 0x188d5a50 nfscbd: service 100187 S rpcsvc 0x188d59d0 nfscbd: service 100188 S rpcsvc 0x188d5950 nfscbd: service 714 1 714 0 SWs accept 0x20cc1948 nfscbd 678 1 678 0 Ss nanslp 0x1d901fd cron 674 1 674 25 SWs pause 0x205d1a60 sendmail 671 1 671 0 Ss select 0x9df5c24 sendmail 668 1 668 0 Ss select 0x183ad3a4 sshd 628 1 627 0 SW nanslp 0x1d901fd smartd 604 1 604 0 Ss select 0x183ad7a4 powerd 601 1 601 0 Ss (threaded) ntpd 100133 S select 0x187df264 ntpd 563 1 563 0 Ss rpcsvc 0x188d5c50 NLM: master 560 1 560 0 Ss select 0x17ee05a4 rpc.statd 559 556 556 0 S (threaded) nfsd 100137 S rpcsvc 0x188d6c50 nfsd: master 100138 S rpcsvc 0x188d61d0 nfsd: service 100139 S rpcsvc 0x188d6150 nfsd: service 100140 S rpcsvc 0x188d6b50 nfsd: service 100141 S rpcsvc 0x99366d0 nfsd: service 100142 S rpcsvc 0x9936650 nfsd: service 100143 S rpcsvc 0x99365d0 nfsd: service 100144 S rpcsvc 0x188d60d0 nfsd: service 100145 S rpcsvc 0x9936550 nfsd: service 100146 S rpcsvc 0x99364d0 nfsd: service 100147 S rpcsvc 0x9936450 nfsd: service 100148 S rpcsvc 0x188d6050 nfsd: service 100149 S rpcsvc 0x99363d0 nfsd: service 100150 S rpcsvc 0x188d5f50 nfsd: service 100151 S rpcsvc 0x9936350 nfsd: service 100152 S rpcsvc 0x99362d0 nfsd: service 100153 S rpcsvc 0x9936250 nfsd: service 100154 S rpcsvc 0x188d5ed0 nfsd: service 100155 S rpcsvc 0x99361d0 nfsd: service 100156 S rpcsvc 0x9936150 nfsd: service 100157 S rpcsvc 0x188d5e50 nfsd: service 100158 S rpcsvc 0x99360d0 nfsd: service 100159 S rpcsvc 0x9936050 nfsd: service 100160 S rpcsvc 0x9935f50 nfsd: service 100161 S rpcsvc 0x9935ed0 nfsd: service 100162 S rpcsvc 0x188d5dd0 nfsd: service 100163 S rpcsvc 0x188d5d50 nfsd: service 100164 S rpcsvc 0x9935e50 nfsd: service 100165 S rpcsvc 0x188d5cd0 nfsd: service 100166 S rpcsvc 0x9935dd0 nfsd: service 100167 S rpcsvc 0x9935d50 nfsd: service 100168 S rpcsvc 0x9935cd0 nfsd: service 556 1 556 0 Ss select 0x183ad4a4 nfsd 550 1 550 0 Ss select 0x9df5b64 mountd 543 539 539 0 S select 0x9df5fa4 nfsuserd 542 539 539 0 S select 0x9df5be4 nfsuserd 541 539 539 0 S select 0x187df224 nfsuserd 540 539 539 0 S select 0x187df2a4 nfsuserd 539 1 539 0 SWs pause 0x1888970c nfsuserd 496 1 496 0 Ss nanslp 0x1d901fe watchdogd 411 1 411 0 Ss select 0x183ad9e4 rpcbind 402 1 402 0 Ss select 0x183ad864 syslogd 330 1 330 0 Ss select 0x187deda4 devd 25 0 0 0 DL vlruwt 0x187b39fc [vnlru] 24 0 0 0 DL syncer 0x1dfd128 [syncer] 23 0 0 0 DL (threaded) [bufdaemon] 100099 D qsleep 0x1dfcaec [bufdaemon] 100102 D - 0x1aca040 [bufspacedaemon-0] 100116 D sdflush 0x18da4884 [/home worker] 100117 D sdflush 0x99d5e84 [/tmp worker] 100118 D sdflush 0x1819f084 [/usr worker] 100119 D sdflush 0x18da4e84 [/var worker] 101686 D sdflush 0x48009c84 [/mnt worker] 22 0 0 0 DL psleep 0x1e11e30 [vmdaemon] 21 0 0 0 DL (threaded) [pagedaemon] 100097 D psleep 0x1e4ad5c [dom0] 100100 D launds 0x1e4ad68 [laundry: dom0] 100101 D umarcl 0x12c8a10 [uma] 20 0 0 0 DL - 0x1bf407c [rand_harvestq] 19 0 0 0 DL waiting 0x1e5bd78 [sctp_iterator] 18 0 0 0 DL - 0x1dfc728 [soaiod4] 17 0 0 0 DL - 0x1dfc728 [soaiod3] 16 0 0 0 DL - 0x1dfc728 [soaiod2] 9 0 0 0 DL - 0x1dfc728 [soaiod1] 8 0 0 0 DL - 0x1813023c [fdc0] 15 0 0 0 DL (threaded) [usb] 100048 D - 0x17ec2aac [usbus0] 100049 D - 0x17ec2adc [usbus0] 100050 D - 0x17ec2b0c [usbus0] 100051 D - 0x17ec2b3c [usbus0] 100052 D - 0x17ec2b6c [usbus0] 100054 D - 0x181a3aac [usbus1] 100055 D - 0x181a3adc [usbus1] 100056 D - 0x181a3b0c [usbus1] 100057 D - 0x181a3b3c [usbus1] 100058 D - 0x181a3b6c [usbus1] 100060 D - 0x181afaac [usbus2] 100061 D - 0x181afadc [usbus2] 100062 D - 0x181afb0c [usbus2] 100063 D - 0x181afb3c [usbus2] 100064 D - 0x181afb6c [usbus2] 100065 D - 0x181b4aac [usbus3] 100066 D - 0x181b4adc [usbus3] 100067 D - 0x181b4b0c [usbus3] 100068 D - 0x181b4b3c [usbus3] 100069 D - 0x181b4b6c [usbus3] 100070 D - 0x181beaac [usbus4] 100071 D - 0x181beadc [usbus4] 100072 D - 0x181beb0c [usbus4] 100073 D - 0x181beb3c [usbus4] 100074 D - 0x181beb6c [usbus4] 100076 D - 0x17ebdc84 [usbus5] 100077 D - 0x17ebdcb4 [usbus5] 100078 D - 0x17ebdce4 [usbus5] 100079 D - 0x17ebdd14 [usbus5] 100080 D - 0x17ebdd44 [usbus5] 7 0 0 0 DL (threaded) [cam] 100043 D - 0x1b2d540 [doneq0] 100096 D - 0x1b2d468 [scanner] 6 0 0 0 DL crypto_ 0x9906dc4 [crypto returns 3] 5 0 0 0 DL crypto_ 0x9906d8c [crypto returns 2] 4 0 0 0 DL crypto_ 0x9906d54 [crypto returns 1] 3 0 0 0 DL crypto_ 0x9906d1c [crypto returns 0] 2 0 0 0 DL crypto_ 0x1e10b28 [crypto] 14 0 0 0 DL seqstat 0x9d1564c [sequencer 00] 13 0 0 0 DL (threaded) [geom] 100030 D - 0x1e4a2f0 [g_event] 100031 D - 0x1e4a2f8 [g_up] 100032 D - 0x1e4a2f4 [g_down] 12 0 0 0 WL (threaded) [intr] 100007 I [swi6: Giant taskq] 100009 I [swi5: fast taskq] 100013 I [swi6: task queue] 100023 I [swi1: netisr 0] 100024 I [swi4: clock (0)] 100025 I [swi4: clock (1)] 100026 I [swi4: clock (2)] 100027 I [swi4: clock (3)] 100028 I [swi3: vm] 100044 I [irq24: hdac0] 100045 I [irq25: re0] 100046 I [irq22: ahci0] 100047 I [irq16: hdac1 ohci0] 100053 I [irq17: ohci1 ohci3] 100059 I [irq18: ohci2 ohci4] 100075 I [irq19: ehci0] 100081 I [irq14: ata0] 100082 I [swi0: uart] 100084 I [irq1: atkbd0] 11 0 0 0 RL (threaded) [idle] 100003 CanRun [idle: cpu0] 100004 CanRun [idle: cpu1] 100005 Run CPU 2 [idle: cpu2] 100006 CanRun [idle: cpu3] 1 0 1 0 SLs wait 0x9df10a4 [init] 10 0 0 0 DL audit_w 0x1e5c50c [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D swapin 0x1e1fce0 [swapper] 100008 D - 0x9d3b600 [thread taskq] 100010 D - 0x9d36e00 [config_0] 100011 D - 0x9d3b100 [kqueue_ctx taskq] 100012 D - 0x9d3ae00 [aiod_kick taskq] 100014 D - 0x9d36d00 [softirq_0] 100015 D - 0x9d36c00 [softirq_1] 100016 D - 0x9d36b00 [softirq_2] 100017 D - 0x9d36a00 [softirq_3] 100018 D - 0x9d36900 [if_io_tqg_0] 100019 D - 0x9d36800 [if_io_tqg_1] 100020 D - 0x9d36700 [if_io_tqg_2] 100021 D - 0x9d36600 [if_io_tqg_3] 100022 D - 0x9d36500 [if_config_tqg_0] 100029 D - 0x9d39a00 [firmware taskq] 100034 D - 0x9d39400 [crypto_0] 100035 D - 0x9d39400 [crypto_1] 100036 D - 0x9d39400 [crypto_2] 100037 D - 0x9d39400 [crypto_3] 100085 D - 0x17ee5400 [mca taskq] 100091 D - 0x183a9d00 [acpi_task_0] 100092 D - 0x183a9d00 [acpi_task_1] 100093 D - 0x183a9d00 [acpi_task_2] 100095 D - 0x9d38900 [CAM taskq] db:0:pho> show freepages DOMAIN: 0 FREE LIST 0: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 09 (002048K) | 000000 08 (001024K) | 000000 07 (000512K) | 000000 06 (000256K) | 000000 05 (000128K) | 000000 04 (000064K) | 000000 03 (000032K) | 000000 02 (000016K) | 000000 01 (000008K) | 000000 00 (000004K) | 000000 FREE LIST 1: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 09 (002048K) | 000000 08 (001024K) | 000014 07 (000512K) | 000271 06 (000256K) | 002216 05 (000128K) | 006008 04 (000064K) | 010330 03 (000032K) | 015040 02 (000016K) | 000212 01 (000008K) | 000096 00 (000004K) | 000001 FREE LIST 2: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 09 (002048K) | 000000 08 (001024K) | 000000 07 (000512K) | 000000 06 (000256K) | 000001 05 (000128K) | 000001 04 (000064K) | 000001 03 (000032K) | 000002 02 (000016K) | 000003 01 (000008K) | 000003 00 (000004K) | 000001 db:0:pho> acttrace Tracing command egrep pid 75735 tid 101049 td 0x2ccaa180 (CPU 0) cpustop_handler(200046,2ce5b8b8,2ce5b8ac,141602b,f2d,...) at cpustop_handler+0x24/frame 0x2ce5b7e0 ipi_nmi_handler(f2d,2ce5b818,1,2ccaa180,28def74c,...) at ipi_nmi_handler+0x31/frame 0x2ce5b7f0 trap(2ce5b8b8,8,28,28,5d00,...) at trap+0xbb/frame 0x2ce5b8ac calltrap() at 0xffc0316f/frame 0x2ce5b8ac --- trap 0x13, eip = 0xfffac1, esp = 0x2ce5b8f8, ebp = 0x2ce5b960 --- sched_clock(2ccaa180,1) at sched_clock+0xf1/frame 0x2ce5b960 statclock(1,0) at statclock+0x124/frame 0x2ce5b998 handleevents(0) at handleevents+0x167/frame 0x2ce5b9dc timercb(1e1efcc,0) at timercb+0x19e/frame 0x2ce5ba1c lapic_handle_timer(2ce5ba48,8,28,28,400000,...) at lapic_handle_timer+0x7d/frame 0x2ce5ba3c Xtimerint() at 0xffc048f3/frame 0x2ce5ba3c --- interrupt, eip = 0x13f9e38, esp = 0x2ce5ba88, ebp = 0x2ce5ba8c --- spinlock_exit(1e57e68,0,1487673,6c3) at spinlock_exit+0x38/frame 0x2ce5ba8c smp_targeted_tlb_shootdown(0,9fc5000,9fc6000,1e59f28) at smp_targeted_tlb_shootdown+0x224/frame 0x2ce5bad0 pmap_invalidate_range_int(9fc6000) at pmap_invalidate_range_int+0x108/frame 0x2ce5bb00 pmap_pae_qremove(9fc5000,1) at pmap_pae_qremove+0x3f/frame 0x2ce5bb18 free_pv_chunk(984d280,9fc5d00,330,b6f86467,360fb4c,...) at free_pv_chunk+0x52/frame 0x2ce5bb30 pmap_pae_remove_pages(28c77f30) at pmap_pae_remove_pages+0x9c/frame 0x2ce5bba8 vmspace_exit(2ccaa180) at vmspace_exit+0xa0/frame 0x2ce5bbc4 exit1(2ccaa180,0,0,2ce5bcdc,1417376,...) at exit1+0x510/frame 0x2ce5bbf4 sys_sys_exit(2ccaa180,2ccaa40c) at sys_sys_exit+0x12/frame 0x2ce5bc08 syscall(2ce5bce8,3b,3b,3b,206d7009,...) at syscall+0x306/frame 0x2ce5bcdc Xint0x80_syscall() at 0xffc033c9/frame 0x2ce5bcdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe660 --- Tracing command ipcrm pid 75723 tid 102315 td 0x27069a80 (CPU 3) kdb_enter(1521cf1,1521cf1) at kdb_enter+0x35/frame 0x289ed978 vpanic(1461fb6,289ed9c4,289ed9c4,289eda00,1416c07,...) at vpanic+0x137/frame 0x289ed9a4 panic(1461fb6,14f09c1,14adb26,0,c,...) at panic+0x14/frame 0x289ed9b8 trap_fatal(2cb86000,1dbc480,2b31380,9890a6c,0,...) at trap_fatal+0x357/frame 0x289eda00 trap_pfault(1833f004,0,0) at trap_pfault+0x72/frame 0x289eda3c trap(289edb10,8,28,28,3ff,...) at trap+0x3a8/frame 0x289edb04 calltrap() at 0xffc0316f/frame 0x289edb04 --- trap 0xc, eip = 0x1046228, esp = 0x289edb50, ebp = 0x289edb70 --- sem_remove(289edbac,0,18342c10,1833d000,0,...) at sem_remove+0x208/frame 0x289edb70 kern_semctl(27069a80,10000,0,0,289edbc8,289edbd4) at kern_semctl+0x892/frame 0x289edba8 sys___semctl(27069a80,27069d0c) at sys___semctl+0xa0/frame 0x289edc08 syscall(289edce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x289edcdc Xint0x80_syscall() at 0xffc033c9/frame 0x289edcdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc015b0, esp = 0xffc0dfe8, ebp = 0xffbfe760 --- Tracing command swap pid 73010 tid 100236 td 0x27011380 (CPU 1) cpustop_handler(200046,ffc0afa8,ffc0af9c,141602b,0,...) at cpustop_handler+0x24/frame 0xffc0aed0 ipi_nmi_handler(0,0,0,0,271e274c,...) at ipi_nmi_handler+0x31/frame 0xffc0aee0 trap(ffc0afa8,3b,3b,3b,0,...) at trap+0xbb/frame 0xffc0af9c calltrap() at 0xffc0316f/frame 0xffc0af9c --- trap 0x13, eip = 0x4026f5, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command idle pid 11 tid 100005 td 0x9df8700 (CPU 2) cpustop_handler(46,1292fbb8,1292fbac,141602b,30000,...) at cpustop_handler+0x24/frame 0x1292fae0 ipi_nmi_handler(30000,0,1292fb28,fb67e1,9df0d50,...) at ipi_nmi_handler+0x31/frame 0x1292faf0 trap(1292fbb8,8,28,28,17e6f400,...) at trap+0xbb/frame 0x1292fbac calltrap() at 0xffc0316f/frame 0x1292fbac --- trap 0x13, eip = 0x13c2845, esp = 0x1292fbf8, ebp = 0x1292fbf8 --- acpi_cpu_c1(1444cf2,312,3e505c6b,9c605b0,39a9a94f,...) at acpi_cpu_c1+0x5/frame 0x1292fbf8 acpi_cpu_idle(1f605a50,0) at acpi_cpu_idle+0x22f/frame 0x1292fc2c cpu_idle_acpi(1f605a50,0) at cpu_idle_acpi+0x35/frame 0x1292fc40 cpu_idle(0) at cpu_idle+0xa3/frame 0x1292fc5c sched_idletd(0,1292fce8) at sched_idletd+0x32e/frame 0x1292fcb4 fork_exit(10007f0,0,1292fce8,0,0,...) at fork_exit+0x69/frame 0x1292fcd4 fork_trampoline() at 0xffc033de/frame 0x1292fcd4 --- kthread start db:0:pho> alltrace Tracing command egrep pid 75735 tid 101049 td 0x2ccaa180 (CPU 0) cpustop_handler(200046,2ce5b8b8,2ce5b8ac,141602b,f2d,...) at cpustop_handler+0x24/frame 0x2ce5b7e0 ipi_nmi_handler(f2d,2ce5b818,1,2ccaa180,28def74c,...) at ipi_nmi_handler+0x31/frame 0x2ce5b7f0 trap(2ce5b8b8,8,28,28,5d00,...) at trap+0xbb/frame 0x2ce5b8ac calltrap() at 0xffc0316f/frame 0x2ce5b8ac --- trap 0x13, eip = 0xfffac1, esp = 0x2ce5b8f8, ebp = 0x2ce5b960 --- sched_clock(2ccaa180,1) at sched_clock+0xf1/frame 0x2ce5b960 statclock(1,0) at statclock+0x124/frame 0x2ce5b998 handleevents(0) at handleevents+0x167/frame 0x2ce5b9dc timercb(1e1efcc,0) at timercb+0x19e/frame 0x2ce5ba1c lapic_handle_timer(2ce5ba48,8,28,28,400000,...) at lapic_handle_timer+0x7d/frame 0x2ce5ba3c Xtimerint() at 0xffc048f3/frame 0x2ce5ba3c --- interrupt, eip = 0x13f9e38, esp = 0x2ce5ba88, ebp = 0x2ce5ba8c --- spinlock_exit(1e57e68,0,1487673,6c3) at spinlock_exit+0x38/frame 0x2ce5ba8c smp_targeted_tlb_shootdown(0,9fc5000,9fc6000,1e59f28) at smp_targeted_tlb_shootdown+0x224/frame 0x2ce5bad0 pmap_invalidate_range_int(9fc6000) at pmap_invalidate_range_int+0x108/frame 0x2ce5bb00 pmap_pae_qremove(9fc5000,1) at pmap_pae_qremove+0x3f/frame 0x2ce5bb18 free_pv_chunk(984d280,9fc5d00,330,b6f86467,360fb4c,...) at free_pv_chunk+0x52/frame 0x2ce5bb30 pmap_pae_remove_pages(28c77f30) at pmap_pae_remove_pages+0x9c/frame 0x2ce5bba8 vmspace_exit(2ccaa180) at vmspace_exit+0xa0/frame 0x2ce5bbc4 exit1(2ccaa180,0,0,2ce5bcdc,1417376,...) at exit1+0x510/frame 0x2ce5bbf4 sys_sys_exit(2ccaa180,2ccaa40c) at sys_sys_exit+0x12/frame 0x2ce5bc08 syscall(2ce5bce8,3b,3b,3b,206d7009,...) at syscall+0x306/frame 0x2ce5bcdc Xint0x80_syscall() at 0xffc033c9/frame 0x2ce5bcdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe660 --- Tracing command ipcrm pid 75723 tid 102315 td 0x27069a80 (CPU 3) kdb_enter(1521cf1,1521cf1) at kdb_enter+0x35/frame 0x289ed978 vpanic(1461fb6,289ed9c4,289ed9c4,289eda00,1416c07,...) at vpanic+0x137/frame 0x289ed9a4 panic(1461fb6,14f09c1,14adb26,0,c,...) at panic+0x14/frame 0x289ed9b8 trap_fatal(2cb86000,1dbc480,2b31380,9890a6c,0,...) at trap_fatal+0x357/frame 0x289eda00 trap_pfault(1833f004,0,0) at trap_pfault+0x72/frame 0x289eda3c trap(289edb10,8,28,28,3ff,...) at trap+0x3a8/frame 0x289edb04 calltrap() at 0xffc0316f/frame 0x289edb04 --- trap 0xc, eip = 0x1046228, esp = 0x289edb50, ebp = 0x289edb70 --- sem_remove(289edbac,0,18342c10,1833d000,0,...) at sem_remove+0x208/frame 0x289edb70 kern_semctl(27069a80,10000,0,0,289edbc8,289edbd4) at kern_semctl+0x892/frame 0x289edba8 sys___semctl(27069a80,27069d0c) at sys___semctl+0xa0/frame 0x289edc08 syscall(289edce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x289edcdc Xint0x80_syscall() at 0xffc033c9/frame 0x289edcdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc015b0, esp = 0xffc0dfe8, ebp = 0xffbfe760 --- Tracing command xargs pid 75648 tid 101734 td 0x284b7000 sched_switch(284b7000,104) at sched_switch+0x4f3/frame 0x274cb9b8 mi_switch(104) at mi_switch+0x1c4/frame 0x274cb9f8 sleepq_switch(1500,6c,0,1d9f3d4,27e0f0a4,...) at sleepq_switch+0xf7/frame 0x274cba18 sleepq_catch_signals(274cba8c,fdf10e,27e0f0a4,6c,200246,...) at sleepq_catch_signals+0x368/frame 0x274cba44 sleepq_wait_sig(27e0f0a4,6c) at sleepq_wait_sig+0xe/frame 0x274cba4c _sleep(27e0f0a4,27e0f14c,36c,1542bd4,0,0,0,0,100) at _sleep+0x27e/frame 0x274cba8c kern_wait6(284b7000,7,0,0,274cbb14,30,0,0) at kern_wait6+0x3b8/frame 0x274cbae8 sys_wait4(284b7000,284b728c) at sys_wait4+0x79/frame 0x274cbc08 syscall(274cbce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x274cbcdc Xint0x80_syscall() at 0xffc033c9/frame 0x274cbcdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe660 --- Tracing command sh pid 75129 tid 102375 td 0x2a932880 sched_switch(2a932880,104) at sched_switch+0x4f3/frame 0x27cf69b8 mi_switch(104) at mi_switch+0x1c4/frame 0x27cf69f8 sleepq_switch(1680,6c,0,1d9f554,27e0faa0,...) at sleepq_switch+0xf7/frame 0x27cf6a18 sleepq_catch_signals(27cf6a8c,fdf10e,27e0faa0,6c,200246,...) at sleepq_catch_signals+0x368/frame 0x27cf6a44 sleepq_wait_sig(27e0faa0,6c) at sleepq_wait_sig+0xe/frame 0x27cf6a4c _sleep(27e0faa0,27e0fb48,36c,1542bd4,0,0,0,0,100) at _sleep+0x27e/frame 0x27cf6a8c kern_wait6(2a932880,7,0,0,27cf6b14,30,0,0) at kern_wait6+0x3b8/frame 0x27cf6ae8 sys_wait4(2a932880,2a932b0c) at sys_wait4+0x79/frame 0x27cf6c08 syscall(27cf6ce8,3b,3b,3b,ffbfe3a4,...) at syscall+0x306/frame 0x27cf6cdc Xint0x80_syscall() at 0xffc033c9/frame 0x27cf6cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0xffc01530, esp = 0xffc0bfe8, ebp = 0xffbfe374 --- Tracing command aiod4 pid 73032 tid 100790 td 0x28571a80 sched_switch(28571a80,104) at sched_switch+0x4f3/frame 0x283efba8 mi_switch(104) at mi_switch+0x1c4/frame 0x283efbe8 sleepq_switch(28571a80,0,2858b000,283efc6c,28571a80,...) at sleepq_switch+0xf7/frame 0x283efc08 sleepq_timedwait(2858b000,5c,10000,10000,150a712,...) at sleepq_timedwait+0x3f/frame 0x283efc2c _sleep(2858b000,1dfca60,5c,148dc86,ffffdd50,1d,0,0,100) at _sleep+0x22c/frame 0x283efc6c aio_daemon(4,283efce8) at aio_daemon+0x282/frame 0x283efcb4 fork_exit(10725f0,4,283efce8,0,0,...) at fork_exit+0x69/frame 0x283efcd4 fork_trampoline() at 0xffc033de/frame 0x283efcd4 --- kthread start Tracing command aiod3 pid 73031 tid 100814 td 0x285d0880 sched_switch(285d0880,104) at sched_switch+0x4f3/frame 0x28acaba8 mi_switch(104) at mi_switch+0x1c4/frame 0x28acabe8 sleepq_switch(285d0880,0,2858caa0,28acac6c,285d0880,...) at sleepq_switch+0xf7/frame 0x28acac08 sleepq_timedwait(2858caa0,5c,0,147aaa0,150a712,...) at sleepq_timedwait+0x3f/frame 0x28acac2c _sleep(2858caa0,1dfca60,5c,148dc86,ffffdd50,1d,0,0,100) at _sleep+0x22c/frame 0x28acac6c aio_daemon(3,28acace8) at aio_daemon+0x282/frame 0x28acacb4 fork_exit(10725f0,3,28acace8,0,0,...) at fork_exit+0x69/frame 0x28acacd4 fork_trampoline() at 0xffc033de/frame 0x28acacd4 --- kthread start Tracing command aiod2 pid 73030 tid 100994 td 0x284b8880 sched_switch(284b8880,104) at sched_switch+0x4f3/frame 0x29308ba8 mi_switch(104) at mi_switch+0x1c4/frame 0x29308be8 sleepq_switch(284b8880,0,2cb87aa0,29308c6c,284b8880,...) at sleepq_switch+0xf7/frame 0x29308c08 sleepq_timedwait(2cb87aa0,5c,0,147aaa0,150a712,...) at sleepq_timedwait+0x3f/frame 0x29308c2c _sleep(2cb87aa0,1dfca60,5c,148dc86,ffffdd50,1d,0,0,100) at _sleep+0x22c/frame 0x29308c6c aio_daemon(2,29308ce8) at aio_daemon+0x282/frame 0x29308cb4 fork_exit(10725f0,2,29308ce8,0,0,...) at fork_exit+0x69/frame 0x29308cd4 fork_trampoline() at 0xffc033de/frame 0x29308cd4 --- kthread start Tracing command aiod1 pid 73029 tid 100513 td 0x27e6d700 sched_switch(27e6d700,104) at sched_switch+0x4f3/frame 0x27d67ba8 mi_switch(104) at mi_switch+0x1c4/frame 0x27d67be8 sleepq_switch(27e6d700,0,27e29000,27d67c6c,27e6d700,...) at sleepq_switch+0xf7/frame 0x27d67c08 sleepq_timedwait(27e29000,5c,0,27d67c64,150a712,...) at sleepq_timedwait+0x3f/frame 0x27d67c2c _sleep(27e29000,1dfca60,5c,148dc86,ffffdd50,1d,0,0,100) at _sleep+0x22c/frame 0x27d67c6c aio_daemon(1,27d67ce8) at aio_daemon+0x282/frame 0x27d67cb4 fork_exit(10725f0,1,27d67ce8,0,0,...) at fork_exit+0x69/frame 0x27d67cd4 fork_trampoline() at 0xffc033de/frame 0x27d67cd4 --- kthread start Tracing command swap pid 73010 tid 100236 td 0x27011380 (CPU 1) cpustop_handler(200046,ffc0afa8,ffc0af9c,141602b,0,...) at cpustop_handler+0x24/frame 0xffc0aed0 ipi_nmi_handler(0,0,0,0,271e274c,...) at ipi_nmi_handler+0x31/frame 0xffc0aee0 trap(ffc0afa8,3b,3b,3b,0,...) at trap+0xbb/frame 0xffc0af9c calltrap() at 0xffc0316f/frame 0xffc0af9c --- trap 0x13, eip = 0x4026f5, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command swap pid 73009 tid 100789 td 0x28571e00 sched_switch(28571e00,104) at sched_switch+0x4f3/frame 0x283e69b8 mi_switch(104) at mi_switch+0x1c4/frame 0x283e69f8 sleepq_switch(39c0,6c,0,1da1894,2858b354,...) at sleepq_switch+0xf7/frame 0x283e6a18 sleepq_catch_signals(283e6a8c,fdf10e,2858b354,6c,200246,...) at sleepq_catch_signals+0x368/frame 0x283e6a44 sleepq_wait_sig(2858b354,6c) at sleepq_wait_sig+0xe/frame 0x283e6a4c _sleep(2858b354,2858b3fc,36c,1542bd4,0,0,0,0,100) at _sleep+0x27e/frame 0x283e6a8c kern_wait6(28571e00,0,11d32,0,283e6b14,30,0,0) at kern_wait6+0x3b8/frame 0x283e6ae8 sys_wait4(28571e00,2857208c) at sys_wait4+0x79/frame 0x283e6c08 syscall(283e6ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x283e6cdc Xint0x80_syscall() at 0xffc033c9/frame 0x283e6cdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe660 --- Tracing command swap pid 73001 tid 100184 td 0x23820a80 sched_switch(23820a80,104) at sched_switch+0x4f3/frame 0x23a55a68 mi_switch(104) at mi_switch+0x1c4/frame 0x23a55aa4 sleepq_switch(3fc0,6c,0,1da1e94,18183354,...) at sleepq_switch+0xf7/frame 0x23a55ac4 sleepq_catch_signals(0,1526a9e,23820a80,100,1d901fe,...) at sleepq_catch_signals+0x368/frame 0x23a55af0 sleepq_timedwait_sig(1d901fe,6c,23a55bc0,0,0,...) at sleepq_timedwait_sig+0x17/frame 0x23a55b1c _sleep(1d901fe,0,16c,15397ea,9d2e921a,ddb2,10000000,0,200) at _sleep+0x21c/frame 0x23a55b5c kern_clock_nanosleep(23820a80,0,0,23a55be8,23a55be0) at kern_clock_nanosleep+0x1b4/frame 0x23a55bc4 sys_nanosleep(23820a80,23820d0c) at sys_nanosleep+0x55/frame 0x23a55c08 syscall(23a55ce8,3b,3b,3b,1,...) at syscall+0x306/frame 0x23a55cdc Xint0x80_syscall() at 0xffc033c9/frame 0x23a55cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0xffc01530, esp = 0xffc0bfe8, ebp = 0xffbfe374 --- Tracing command md10 pid 72995 tid 101048 td 0x2ccaa500 sched_switch(2ccaa500,104) at sched_switch+0x4f3/frame 0x2ce56bd0 mi_switch(104) at mi_switch+0x1c4/frame 0x2ce56c0c sleepq_switch(5dce4000,5c) at sleepq_switch+0xf7/frame 0x2ce56c2c _sleep(5dce4000,5dce4028,25c,14e9020,0,0,0,0,100) at _sleep+0x2f7/frame 0x2ce56c6c md_kthread(5dce4000,2ce56ce8) at md_kthread+0xc2/frame 0x2ce56cb4 fork_exit(be0810,5dce4000,2ce56ce8,0,0,...) at fork_exit+0x69/frame 0x2ce56cd4 fork_trampoline() at 0xffc033de/frame 0x2ce56cd4 --- kthread start Tracing command sh pid 72481 tid 100240 td 0x27010500 sched_switch(27010500,104) at sched_switch+0x4f3/frame 0x274f89b8 mi_switch(104) at mi_switch+0x1c4/frame 0x274f89f8 sleepq_switch(3940,6c,0,1da1814,271e19fc,...) at sleepq_switch+0xf7/frame 0x274f8a18 sleepq_catch_signals(274f8a8c,fdf10e,271e19fc,6c,200246,...) at sleepq_catch_signals+0x368/frame 0x274f8a44 sleepq_wait_sig(271e19fc,6c) at sleepq_wait_sig+0xe/frame 0x274f8a4c _sleep(271e19fc,271e1aa4,36c,1542bd4,0,0,0,0,100) at _sleep+0x27e/frame 0x274f8a8c kern_wait6(27010500,7,0,0,274f8b14,30,0,0) at kern_wait6+0x3b8/frame 0x274f8ae8 sys_wait4(27010500,2701078c) at sys_wait4+0x79/frame 0x274f8c08 syscall(274f8ce8,3b,3b,3b,ffbfe5e4,...) at syscall+0x306/frame 0x274f8cdc Xint0x80_syscall() at 0xffc033c9/frame 0x274f8cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc015b0, esp = 0xffc0dfe8, ebp = 0xffbfe760 --- Tracing command tee pid 72480 tid 100817 td 0x285cfe00 sched_switch(285cfe00,104) at sched_switch+0x4f3/frame 0x28ae5a50 mi_switch(104) at mi_switch+0x1c4/frame 0x28ae5a8c sleepq_switch(3c00,5c,0,1da1ad4,2858c0a4,...) at sleepq_switch+0xf7/frame 0x28ae5aac sleepq_catch_signals(28ae5b20,fdf10e,479ff000,5c,479ff1a0,...) at sleepq_catch_signals+0x368/frame 0x28ae5ad8 sleepq_wait_sig(479ff000,5c) at sleepq_wait_sig+0xe/frame 0x28ae5ae0 _sleep(479ff000,479ff190,15c,1526c2d,0,0,0,0,100) at _sleep+0x27e/frame 0x28ae5b20 pipe_read(3216b578,28ae5bd8,6a79c900,0,285cfe00) at pipe_read+0x3bf/frame 0x28ae5b60 dofileread(3216b578,28ae5bd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x28ae5b90 kern_readv(285cfe00,0,28ae5bd8) at kern_readv+0x3b/frame 0x28ae5bbc sys_read(285cfe00,285d008c) at sys_read+0x48/frame 0x28ae5c08 syscall(28ae5ce8,3b,3b,3b,ffbfe7f0,...) at syscall+0x306/frame 0x28ae5cdc Xint0x80_syscall() at 0xffc033c9/frame 0x28ae5cdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f5, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command sh pid 72479 tid 100242 td 0x2700fe00 sched_switch(2700fe00,104) at sched_switch+0x4f3/frame 0x2750a9b8 mi_switch(104) at mi_switch+0x1c4/frame 0x2750a9f8 sleepq_switch(11c0,6c,0,1d9f094,271e1354,...) at sleepq_switch+0xf7/frame 0x2750aa18 sleepq_catch_signals(2750aa8c,fdf10e,271e1354,6c,200246,...) at sleepq_catch_signals+0x368/frame 0x2750aa44 sleepq_wait_sig(271e1354,6c) at sleepq_wait_sig+0xe/frame 0x2750aa4c _sleep(271e1354,271e13fc,36c,1542bd4,0,0,0,0,100) at _sleep+0x27e/frame 0x2750aa8c kern_wait6(2700fe00,7,0,0,2750ab14,30,0,0) at kern_wait6+0x3b8/frame 0x2750aae8 sys_wait4(2700fe00,2701008c) at sys_wait4+0x79/frame 0x2750ac08 syscall(2750ace8,3b,3b,3b,ffbfdd04,...) at syscall+0x306/frame 0x2750acdc Xint0x80_syscall() at 0xffc033c9/frame 0x2750acdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe660 --- Tracing command sleep pid 46686 tid 100712 td 0x28493380 sched_switch(28493380,104) at sched_switch+0x4f3/frame 0x28131a68 mi_switch(104) at mi_switch+0x1c4/frame 0x28131aa4 sleepq_switch(3f00,6c,3,1da1dd4,284566a8,...) at sleepq_switch+0xf7/frame 0x28131ac4 sleepq_catch_signals(0,1526a9e,28493380,100,1d901fd,...) at sleepq_catch_signals+0x368/frame 0x28131af0 sleepq_timedwait_sig(1d901fd,6c,14e4c87,60a,28131b98,...) at sleepq_timedwait_sig+0x17/frame 0x28131b1c _sleep(1d901fd,0,16c,15397ea,27ef87a1,ddbb,c0000000,3,200) at _sleep+0x21c/frame 0x28131b5c kern_clock_nanosleep(28493380,0,0,28131be8,28131be0) at kern_clock_nanosleep+0x1b4/frame 0x28131bc4 sys_nanosleep(28493380,2849360c) at sys_nanosleep+0x55/frame 0x28131c08 syscall(28131ce8,3b,3b,3b,ffbfeae0,...) at syscall+0x306/frame 0x28131cdc Xint0x80_syscall() at 0xffc033c9/frame 0x28131cdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f5, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command top pid 19975 tid 100110 td 0x18891000 sched_switch(18891000,104) at sched_switch+0x4f3/frame 0x202b0930 mi_switch(104) at mi_switch+0x1c4/frame 0x202b096c sleepq_switch(1580,0,1db8228,1d9f454,18889d50,...) at sleepq_switch+0xf7/frame 0x202b098c sleepq_catch_signals(e2,18891000,1abc268,18891000,202b09ec,...) at sleepq_catch_signals+0x368/frame 0x202b09b8 sleepq_timedwait_sig(27123264,0) at sleepq_timedwait_sig+0x17/frame 0x202b09e8 _cv_timedwait_sig_sbt(27123264,27123250,3f7c8d0d,ddb3,20000000,0,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x202b0a24 seltdwait(ddb3,20000000,0) at seltdwait+0x5e/frame 0x202b0a54 kern_select(18891000,2,ffbfeb38,0,0,202b0bf8,20) at kern_select+0x8ed/frame 0x202b0bd4 sys_select(18891000,1889128c) at sys_select+0x41/frame 0x202b0c08 syscall(202b0ce8,3b,3b,3b,ffbfebb8,...) at syscall+0x306/frame 0x202b0cdc Xint0x80_syscall() at 0xffc033c9/frame 0x202b0cdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe660 --- Tracing command bash pid 95904 tid 100653 td 0x27ff3a80 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa5ebc,1aa5ed4,289ed478,102bc8c,1aa5ed4,...) at db_trace_self_wrapper+0x2a/frame 0x289ed430 kdb_backtrace(289ed558,1416041,38,a07e50,38,...) at kdb_backtrace+0x2e/frame 0x289ed490 kdb_reenter(38,a07e50,38,a07e50,2cb86000,...) at kdb_reenter+0x29/frame 0x289ed498 trap(289ed564,8,28,28,ffffffff,...) at trap+0xd1/frame 0x289ed558 calltrap() at 0xffc0316f/frame 0x289ed558 --- trap 0xc, eip = 0x13f347a, esp = 0x289ed5a4, ebp = 0x289ed5b0 --- db_trace_thread(27ff3a80,ffffffff) at db_trace_thread+0x2a/frame 0x289ed5b0 _db_stack_trace_all(1017c05,0,ffffffff,289ed63c) at _db_stack_trace_all+0x93/frame 0x289ed610 db_command(1b2ff32) at db_command+0x26a/frame 0x289ed6c4 db_script_exec(bd452f,0,289ed76c,289ed750) at db_script_exec+0x1a6/frame 0x289ed724 db_command(289ed828,a07d16,0,0,27069a80,...) at db_command+0x26a/frame 0x289ed7d8 db_command_loop(0,0,27069a80,289ed7e0,289ed828,...) at db_command_loop+0x3c/frame 0x289ed7e0 db_trap(3,0) at db_trap+0xf6/frame 0x289ed828 kdb_trap(3,0,289ed934) at kdb_trap+0x119/frame 0x289ed85c trap(289ed934,8,28,28,100,...) at trap+0x38a/frame 0x289ed928 calltrap() at 0xffc0316f/frame 0x289ed928 --- trap 0x3, eip = 0x1017c05, esp = 0x289ed974, ebp = 0x289ed978 --- kdb_enter(1521cf1,1521cf1) at kdb_enter+0x35/frame 0x289ed978 vpanic(1461fb6,289ed9c4,289ed9c4,289eda00,1416c07,...) at vpanic+0x137/frame 0x289ed9a4 panic(1461fb6,14f09c1,14adb26,0,c,...) at panic+0x14/frame 0x289ed9b8 trap_fatal(2cb86000,1dbc480,2b31380,9890a6c,0,...) at trap_fatal+0x357/frame 0x289eda00 trap_pfault(1833f004,0,0) at trap_pfault+0x72/frame 0x289eda3c trap(289edb10,8,28,28,3ff,...) at trap+0x3a8/frame 0x289edb04 calltrap() at 0xffc0316f/frame 0x289edb04 --- trap 0xc, eip = 0x1046228, esp = 0x289edb50, ebp = 0x289edb70 --- sem_remove(289edbac,0,18342c10,1833d000,0,...) at sem_remove+0x208/frame 0x289edb70 kern_semctl(27069a80,10000,0,0,289edbc8,289edbd4) at kern_semctl+0x892/frame 0x289edba8 sys___semctl(27069a80,27069d0c) at sys___semctl+0xa0/frame 0x289edc08 syscall(289edce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x289edcdc Xint0x80_syscall() at 0xffc033c9/frame 0x289edcdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc015b0, esp = 0xffc0dfe8, ebp = 0xffbfe760 --- Tracing command sshd pid 95820 tid 101282 td 0x2d9b9a80 sched_switch(2d9b9a80,104) at sched_switch+0x4f3/frame 0x2e41f970 mi_switch(104) at mi_switch+0x1c4/frame 0x2e41f9ac sleepq_switch(1100,0,2d9b9a80,1d9efd4,2df9d6a8,...) at sleepq_switch+0xf7/frame 0x2e41f9cc sleepq_catch_signals(2e41fa38,f76998,27ff6024,0,1441215,...) at sleepq_catch_signals+0x368/frame 0x2e41f9f8 sleepq_wait_sig(27ff6024,0) at sleepq_wait_sig+0xe/frame 0x2e41fa00 _cv_wait_sig(27ff6024,27ff6010) at _cv_wait_sig+0x138/frame 0x2e41fa38 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x2e41fa54 kern_select(2d9b9a80,e,2047a270,2047a278,0,0,20) at kern_select+0x8ed/frame 0x2e41fbd4 sys_select(2d9b9a80,2d9b9d0c) at sys_select+0x41/frame 0x2e41fc08 syscall(2e41fce8,3b,3b,3b,20480000,...) at syscall+0x306/frame 0x2e41fcdc Xint0x80_syscall() at 0xffc033c9/frame 0x2e41fcdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc015b0, esp = 0xffc0dfe8, ebp = 0xffbfe760 --- Tracing command sshd pid 95490 tid 100999 td 0x284b7a80 sched_switch(284b7a80,104) at sched_switch+0x4f3/frame 0x29325998 mi_switch(104) at mi_switch+0x1c4/frame 0x293259d4 sleepq_switch(2a40,0,284b7a80,1da0914,2cb86d50,...) at sleepq_switch+0xf7/frame 0x293259f4 sleepq_catch_signals(29325a60,f76998,236c0da4,0,1441215,...) at sleepq_catch_signals+0x368/frame 0x29325a20 sleepq_wait_sig(236c0da4,0) at sleepq_wait_sig+0xe/frame 0x29325a28 _cv_wait_sig(236c0da4,236c0d90) at _cv_wait_sig+0x138/frame 0x29325a60 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x29325a7c kern_poll(284b7a80,ffbfe800,1,0,0) at kern_poll+0x476/frame 0x29325bd4 sys_poll(284b7a80,284b7d0c) at sys_poll+0x50/frame 0x29325c08 syscall(29325ce8,3b,3b,3b,20480000,...) at syscall+0x306/frame 0x29325cdc Xint0x80_syscall() at 0xffc033c9/frame 0x29325cdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f5, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command bash pid 3454 tid 100974 td 0x284baa80 sched_switch(284baa80,104) at sched_switch+0x4f3/frame 0x292749b8 mi_switch(104) at mi_switch+0x1c4/frame 0x292749f8 sleepq_switch(11c0,6c,0,1d9f094,28411354,...) at sleepq_switch+0xf7/frame 0x29274a18 sleepq_catch_signals(29274a8c,fdf10e,28411354,6c,200246,...) at sleepq_catch_signals+0x368/frame 0x29274a44 sleepq_wait_sig(28411354,6c) at sleepq_wait_sig+0xe/frame 0x29274a4c _sleep(28411354,284113fc,36c,1542bd4,0,0,0,0,100) at _sleep+0x27e/frame 0x29274a8c kern_wait6(284baa80,7,0,0,29274b14,36,0,0) at kern_wait6+0x3b8/frame 0x29274ae8 sys_wait4(284baa80,284bad0c) at sys_wait4+0x79/frame 0x29274c08 syscall(29274ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x29274cdc Xint0x80_syscall() at 0xffc033c9/frame 0x29274cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0xffc01530, esp = 0xffc0bfe8, ebp = 0xffbfe374 --- Tracing command sshd pid 3453 tid 100973 td 0x28da4500 sched_switch(28da4500,104) at sched_switch+0x4f3/frame 0x29267970 mi_switch(104) at mi_switch+0x1c4/frame 0x292679ac sleepq_switch(2b40,0,28da4500,1da0a14,28def0a4,...) at sleepq_switch+0xf7/frame 0x292679cc sleepq_catch_signals(29267a38,f76998,9df49e4,0,1441215,...) at sleepq_catch_signals+0x368/frame 0x292679f8 sleepq_wait_sig(9df49e4,0) at sleepq_wait_sig+0xe/frame 0x29267a00 _cv_wait_sig(9df49e4,9df49d0) at _cv_wait_sig+0x138/frame 0x29267a38 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x29267a54 kern_select(28da4500,e,2047a270,2047a278,0,0,20) at kern_select+0x8ed/frame 0x29267bd4 sys_select(28da4500,28da478c) at sys_select+0x41/frame 0x29267c08 syscall(29267ce8,3b,3b,3b,20480000,...) at syscall+0x306/frame 0x29267cdc Xint0x80_syscall() at 0xffc033c9/frame 0x29267cdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe660 --- Tracing command sshd pid 3451 tid 100972 td 0x28cb8a80 sched_switch(28cb8a80,104) at sched_switch+0x4f3/frame 0x2925e998 mi_switch(104) at mi_switch+0x1c4/frame 0x2925e9d4 sleepq_switch(3400,0,28cb8a80,1da12d4,28c829fc,...) at sleepq_switch+0xf7/frame 0x2925e9f4 sleepq_catch_signals(2925ea60,f76998,28dcf424,0,1441215,...) at sleepq_catch_signals+0x368/frame 0x2925ea20 sleepq_wait_sig(28dcf424,0) at sleepq_wait_sig+0xe/frame 0x2925ea28 _cv_wait_sig(28dcf424,28dcf410) at _cv_wait_sig+0x138/frame 0x2925ea60 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x2925ea7c kern_poll(28cb8a80,ffbfe800,1,0,0) at kern_poll+0x476/frame 0x2925ebd4 sys_poll(28cb8a80,28cb8d0c) at sys_poll+0x50/frame 0x2925ec08 syscall(2925ece8,3b,3b,3b,20480000,...) at syscall+0x306/frame 0x2925ecdc Xint0x80_syscall() at 0xffc033c9/frame 0x2925ecdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f5, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command sh pid 1494 tid 100209 td 0x2382f380 sched_switch(2382f380,104) at sched_switch+0x4f3/frame 0x23af9a50 mi_switch(104) at mi_switch+0x1c4/frame 0x23af9a8c sleepq_switch(3240,5c,0,1da1114,270359fc,...) at sleepq_switch+0xf7/frame 0x23af9aac sleepq_catch_signals(23af9b20,fdf10e,204861a8,5c,20486348,...) at sleepq_catch_signals+0x368/frame 0x23af9ad8 sleepq_wait_sig(204861a8,5c) at sleepq_wait_sig+0xe/frame 0x23af9ae0 _sleep(204861a8,20486338,15c,1526c2d,0,0,0,0,100) at _sleep+0x27e/frame 0x23af9b20 pipe_read(18df5498,23af9bd8,18186300,0,2382f380) at pipe_read+0x3bf/frame 0x23af9b60 dofileread(18df5498,23af9bd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x23af9b90 kern_readv(2382f380,0,23af9bd8) at kern_readv+0x3b/frame 0x23af9bbc sys_read(2382f380,2382f60c) at sys_read+0x48/frame 0x23af9c08 syscall(23af9ce8,3b,3b,3b,206f6584,...) at syscall+0x306/frame 0x23af9cdc Xint0x80_syscall() at 0xffc033c9/frame 0x23af9cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0xffc01530, esp = 0xffc0bfe8, ebp = 0xffbfe374 --- Tracing command awk pid 1493 tid 100214 td 0x1889fa80 sched_switch(1889fa80,104) at sched_switch+0x4f3/frame 0x27398a50 mi_switch(104) at mi_switch+0x1c4/frame 0x27398a8c sleepq_switch(1c00,5c,0,1d9fad4,1888fd50,...) at sleepq_switch+0xf7/frame 0x27398aac sleepq_catch_signals(27398b20,fdf10e,20487000,5c,204871a0,...) at sleepq_catch_signals+0x368/frame 0x27398ad8 sleepq_wait_sig(20487000,5c) at sleepq_wait_sig+0xe/frame 0x27398ae0 _sleep(20487000,20487190,15c,1526c2d,0,0,0,0,100) at _sleep+0x27e/frame 0x27398b20 pipe_read(270674d0,27398bd8,18186300,0,1889fa80) at pipe_read+0x3bf/frame 0x27398b60 dofileread(270674d0,27398bd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x27398b90 kern_readv(1889fa80,0,27398bd8) at kern_readv+0x3b/frame 0x27398bbc sys_read(1889fa80,1889fd0c) at sys_read+0x48/frame 0x27398c08 syscall(27398ce8,3b,3b,3b,2064f5cc,...) at syscall+0x306/frame 0x27398cdc Xint0x80_syscall() at 0xffc033c9/frame 0x27398cdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe660 --- Tracing command sh pid 1492 tid 100182 td 0x23821180 sched_switch(23821180,104) at sched_switch+0x4f3/frame 0x23a4b9b8 mi_switch(104) at mi_switch+0x1c4/frame 0x23a4b9f8 sleepq_switch(3140,6c,0,1da1014,181839fc,...) at sleepq_switch+0xf7/frame 0x23a4ba18 sleepq_catch_signals(23a4ba8c,fdf10e,181839fc,6c,200246,...) at sleepq_catch_signals+0x368/frame 0x23a4ba44 sleepq_wait_sig(181839fc,6c) at sleepq_wait_sig+0xe/frame 0x23a4ba4c _sleep(181839fc,18183aa4,36c,1542bd4,0,0,0,0,100) at _sleep+0x27e/frame 0x23a4ba8c kern_wait6(23821180,7,0,0,23a4bb14,30,0,0) at kern_wait6+0x3b8/frame 0x23a4bae8 sys_wait4(23821180,2382140c) at sys_wait4+0x79/frame 0x23a4bc08 syscall(23a4bce8,3b,3b,3b,ffbfe524,...) at syscall+0x306/frame 0x23a4bcdc Xint0x80_syscall() at 0xffc033c9/frame 0x23a4bcdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0xffc01530, esp = 0xffc0bfe8, ebp = 0xffbfe374 --- Tracing command sh pid 1471 tid 100176 td 0x236c6000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa5ebc,1aa5ed4,289ed478,102bc8c,1aa5ed4,...) at db_trace_self_wrapper+0x2a/frame 0x289ed430 kdb_backtrace(289ed558,1416041,35,a07e50,35,...) at kdb_backtrace+0x2e/frame 0x289ed490 kdb_reenter(35,a07e50,35,a07e50,2cb86000,...) at kdb_reenter+0x29/frame 0x289ed498 trap(289ed564,8,28,28,ffffffff,...) at trap+0xd1/frame 0x289ed558 calltrap() at 0xffc0316f/frame 0x289ed558 --- trap 0xc, eip = 0x13f347a, esp = 0x289ed5a4, ebp = 0x289ed5b0 --- db_trace_thread(236c6000,ffffffff) at db_trace_thread+0x2a/frame 0x289ed5b0 _db_stack_trace_all(1017c05,0,ffffffff,289ed63c) at _db_stack_trace_all+0x93/frame 0x289ed610 db_command(1b2ff32) at db_command+0x26a/frame 0x289ed6c4 db_script_exec(bd452f,0,289ed76c,289ed750) at db_script_exec+0x1a6/frame 0x289ed724 db_command(289ed828,a07d16,0,0,27069a80,...) at db_command+0x26a/frame 0x289ed7d8 db_command_loop(0,0,27069a80,289ed7e0,289ed828,...) at db_command_loop+0x3c/frame 0x289ed7e0 db_trap(3,0) at db_trap+0xf6/frame 0x289ed828 kdb_trap(3,0,289ed934) at kdb_trap+0x119/frame 0x289ed85c trap(289ed934,8,28,28,100,...) at trap+0x38a/frame 0x289ed928 calltrap() at 0xffc0316f/frame 0x289ed928 --- trap 0x3, eip = 0x1017c05, esp = 0x289ed974, ebp = 0x289ed978 --- kdb_enter(1521cf1,1521cf1) at kdb_enter+0x35/frame 0x289ed978 vpanic(1461fb6,289ed9c4,289ed9c4,289eda00,1416c07,...) at vpanic+0x137/frame 0x289ed9a4 panic(1461fb6,14f09c1,14adb26,0,c,...) at panic+0x14/frame 0x289ed9b8 trap_fatal(2cb86000,1dbc480,2b31380,9890a6c,0,...) at trap_fatal+0x357/frame 0x289eda00 trap_pfault(1833f004,0,0) at trap_pfault+0x72/frame 0x289eda3c trap(289edb10,8,28,28,3ff,...) at trap+0x3a8/frame 0x289edb04 calltrap() at 0xffc0316f/frame 0x289edb04 --- trap 0xc, eip = 0x1046228, esp = 0x289edb50, ebp = 0x289edb70 --- sem_remove(289edbac,0,18342c10,1833d000,0,...) at sem_remove+0x208/frame 0x289edb70 kern_semctl(27069a80,10000,0,0,289edbc8,289edbd4) at kern_semctl+0x892/frame 0x289edba8 sys___semctl(27069a80,27069d0c) at sys___semctl+0xa0/frame 0x289edc08 syscall(289edce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x289edcdc Xint0x80_syscall() at 0xffc033c9/frame 0x289edcdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc015b0, esp = 0xffc0dfe8, ebp = 0xffbfe760 --- Tracing command sh pid 952 tid 100180 td 0x236c5500 sched_switch(236c5500,104) at sched_switch+0x4f3/frame 0x23a3d9b8 mi_switch(104) at mi_switch+0x1c4/frame 0x23a3d9f8 sleepq_switch(2f80,6c,0,1da0e54,23cd16a8,...) at sleepq_switch+0xf7/frame 0x23a3da18 sleepq_catch_signals(23a3da8c,fdf10e,23cd16a8,6c,200246,...) at sleepq_catch_signals+0x368/frame 0x23a3da44 sleepq_wait_sig(23cd16a8,6c) at sleepq_wait_sig+0xe/frame 0x23a3da4c _sleep(23cd16a8,23cd1750,36c,1542bd4,0,0,0,0,100) at _sleep+0x27e/frame 0x23a3da8c kern_wait6(236c5500,7,0,0,23a3db14,30,0,0) at kern_wait6+0x3b8/frame 0x23a3dae8 sys_wait4(236c5500,236c578c) at sys_wait4+0x79/frame 0x23a3dc08 syscall(23a3dce8,3b,3b,3b,ffbfdf24,...) at syscall+0x306/frame 0x23a3dcdc Xint0x80_syscall() at 0xffc033c9/frame 0x23a3dcdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe660 --- Tracing command bash pid 915 tid 100130 td 0x23673180 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa5ebc,1aa5ed4,289ed478,102bc8c,1aa5ed4,...) at db_trace_self_wrapper+0x2a/frame 0x289ed430 kdb_backtrace(289ed558,1416041,36,a07e50,36,...) at kdb_backtrace+0x2e/frame 0x289ed490 kdb_reenter(36,a07e50,36,a07e50,2cb86000,...) at kdb_reenter+0x29/frame 0x289ed498 trap(289ed564,8,28,28,ffffffff,...) at trap+0xd1/frame 0x289ed558 calltrap() at 0xffc0316f/frame 0x289ed558 --- trap 0xc, eip = 0x13f347a, esp = 0x289ed5a4, ebp = 0x289ed5b0 --- db_trace_thread(23673180,ffffffff) at db_trace_thread+0x2a/frame 0x289ed5b0 _db_stack_trace_all(1017c05,0,ffffffff,289ed63c) at _db_stack_trace_all+0x93/frame 0x289ed610 db_command(1b2ff32) at db_command+0x26a/frame 0x289ed6c4 db_script_exec(bd452f,0,289ed76c,289ed750) at db_script_exec+0x1a6/frame 0x289ed724 db_command(289ed828,a07d16,0,0,27069a80,...) at db_command+0x26a/frame 0x289ed7d8 db_command_loop(0,0,27069a80,289ed7e0,289ed828,...) at db_command_loop+0x3c/frame 0x289ed7e0 db_trap(3,0) at db_trap+0xf6/frame 0x289ed828 kdb_trap(3,0,289ed934) at kdb_trap+0x119/frame 0x289ed85c trap(289ed934,8,28,28,100,...) at trap+0x38a/frame 0x289ed928 calltrap() at 0xffc0316f/frame 0x289ed928 --- trap 0x3, eip = 0x1017c05, esp = 0x289ed974, ebp = 0x289ed978 --- kdb_enter(1521cf1,1521cf1) at kdb_enter+0x35/frame 0x289ed978 vpanic(1461fb6,289ed9c4,289ed9c4,289eda00,1416c07,...) at vpanic+0x137/frame 0x289ed9a4 panic(1461fb6,14f09c1,14adb26,0,c,...) at panic+0x14/frame 0x289ed9b8 trap_fatal(2cb86000,1dbc480,2b31380,9890a6c,0,...) at trap_fatal+0x357/frame 0x289eda00 trap_pfault(1833f004,0,0) at trap_pfault+0x72/frame 0x289eda3c trap(289edb10,8,28,28,3ff,...) at trap+0x3a8/frame 0x289edb04 calltrap() at 0xffc0316f/frame 0x289edb04 --- trap 0xc, eip = 0x1046228, esp = 0x289edb50, ebp = 0x289edb70 --- sem_remove(289edbac,0,18342c10,1833d000,0,...) at sem_remove+0x208/frame 0x289edb70 kern_semctl(27069a80,10000,0,0,289edbc8,289edbd4) at kern_semctl+0x892/frame 0x289edba8 sys___semctl(27069a80,27069d0c) at sys___semctl+0xa0/frame 0x289edc08 syscall(289edce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x289edcdc Xint0x80_syscall() at 0xffc033c9/frame 0x289edcdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc015b0, esp = 0xffc0dfe8, ebp = 0xffbfe760 --- Tracing command csh pid 913 tid 100203 td 0x23830180 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa5ebc,1aa5ed4,289ed478,102bc8c,1aa5ed4,...) at db_trace_self_wrapper+0x2a/frame 0x289ed430 kdb_backtrace(289ed558,1416041,35,a07e50,35,...) at kdb_backtrace+0x2e/frame 0x289ed490 kdb_reenter(35,a07e50,35,a07e50,2cb86000,...) at kdb_reenter+0x29/frame 0x289ed498 trap(289ed564,8,28,28,ffffffff,...) at trap+0xd1/frame 0x289ed558 calltrap() at 0xffc0316f/frame 0x289ed558 --- trap 0xc, eip = 0x13f347a, esp = 0x289ed5a4, ebp = 0x289ed5b0 --- db_trace_thread(23830180,ffffffff) at db_trace_thread+0x2a/frame 0x289ed5b0 _db_stack_trace_all(1017c05,0,ffffffff,289ed63c) at _db_stack_trace_all+0x93/frame 0x289ed610 db_command(1b2ff32) at db_command+0x26a/frame 0x289ed6c4 db_script_exec(bd452f,0,289ed76c,289ed750) at db_script_exec+0x1a6/frame 0x289ed724 db_command(289ed828,a07d16,0,0,27069a80,...) at db_command+0x26a/frame 0x289ed7d8 db_command_loop(0,0,27069a80,289ed7e0,289ed828,...) at db_command_loop+0x3c/frame 0x289ed7e0 db_trap(3,0) at db_trap+0xf6/frame 0x289ed828 kdb_trap(3,0,289ed934) at kdb_trap+0x119/frame 0x289ed85c trap(289ed934,8,28,28,100,...) at trap+0x38a/frame 0x289ed928 calltrap() at 0xffc0316f/frame 0x289ed928 --- trap 0x3, eip = 0x1017c05, esp = 0x289ed974, ebp = 0x289ed978 --- kdb_enter(1521cf1,1521cf1) at kdb_enter+0x35/frame 0x289ed978 vpanic(1461fb6,289ed9c4,289ed9c4,289eda00,1416c07,...) at vpanic+0x137/frame 0x289ed9a4 panic(1461fb6,14f09c1,14adb26,0,c,...) at panic+0x14/frame 0x289ed9b8 trap_fatal(2cb86000,1dbc480,2b31380,9890a6c,0,...) at trap_fatal+0x357/frame 0x289eda00 trap_pfault(1833f004,0,0) at trap_pfault+0x72/frame 0x289eda3c trap(289edb10,8,28,28,3ff,...) at trap+0x3a8/frame 0x289edb04 calltrap() at 0xffc0316f/frame 0x289edb04 --- trap 0xc, eip = 0x1046228, esp = 0x289edb50, ebp = 0x289edb70 --- sem_remove(289edbac,0,18342c10,1833d000,0,...) at sem_remove+0x208/frame 0x289edb70 kern_semctl(27069a80,10000,0,0,289edbc8,289edbd4) at kern_semctl+0x892/frame 0x289edba8 sys___semctl(27069a80,27069d0c) at sys___semctl+0xa0/frame 0x289edc08 syscall(289edce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x289edcdc Xint0x80_syscall() at 0xffc033c9/frame 0x289edcdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc015b0, esp = 0xffc0dfe8, ebp = 0xffbfe760 --- Tracing command su pid 912 tid 100191 td 0x188a0c00 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa5ebc,1aa5ed4,289ed478,102bc8c,1aa5ed4,...) at db_trace_self_wrapper+0x2a/frame 0x289ed430 kdb_backtrace(289ed558,1416041,34,a07e50,34,...) at kdb_backtrace+0x2e/frame 0x289ed490 kdb_reenter(34,a07e50,34,a07e50,2cb86000,...) at kdb_reenter+0x29/frame 0x289ed498 trap(289ed564,8,28,28,ffffffff,...) at trap+0xd1/frame 0x289ed558 calltrap() at 0xffc0316f/frame 0x289ed558 --- trap 0xc, eip = 0x13f347a, esp = 0x289ed5a4, ebp = 0x289ed5b0 --- db_trace_thread(188a0c00,ffffffff) at db_trace_thread+0x2a/frame 0x289ed5b0 _db_stack_trace_all(1017c05,0,ffffffff,289ed63c) at _db_stack_trace_all+0x93/frame 0x289ed610 db_command(1b2ff32) at db_command+0x26a/frame 0x289ed6c4 db_script_exec(bd452f,0,289ed76c,289ed750) at db_script_exec+0x1a6/frame 0x289ed724 db_command(289ed828,a07d16,0,0,27069a80,...) at db_command+0x26a/frame 0x289ed7d8 db_command_loop(0,0,27069a80,289ed7e0,289ed828,...) at db_command_loop+0x3c/frame 0x289ed7e0 db_trap(3,0) at db_trap+0xf6/frame 0x289ed828 kdb_trap(3,0,289ed934) at kdb_trap+0x119/frame 0x289ed85c trap(289ed934,8,28,28,100,...) at trap+0x38a/frame 0x289ed928 calltrap() at 0xffc0316f/frame 0x289ed928 --- trap 0x3, eip = 0x1017c05, esp = 0x289ed974, ebp = 0x289ed978 --- kdb_enter(1521cf1,1521cf1) at kdb_enter+0x35/frame 0x289ed978 vpanic(1461fb6,289ed9c4,289ed9c4,289eda00,1416c07,...) at vpanic+0x137/frame 0x289ed9a4 panic(1461fb6,14f09c1,14adb26,0,c,...) at panic+0x14/frame 0x289ed9b8 trap_fatal(2cb86000,1dbc480,2b31380,9890a6c,0,...) at trap_fatal+0x357/frame 0x289eda00 trap_pfault(1833f004,0,0) at trap_pfault+0x72/frame 0x289eda3c trap(289edb10,8,28,28,3ff,...) at trap+0x3a8/frame 0x289edb04 calltrap() at 0xffc0316f/frame 0x289edb04 --- trap 0xc, eip = 0x1046228, esp = 0x289edb50, ebp = 0x289edb70 --- sem_remove(289edbac,0,18342c10,1833d000,0,...) at sem_remove+0x208/frame 0x289edb70 kern_semctl(27069a80,10000,0,0,289edbc8,289edbd4) at kern_semctl+0x892/frame 0x289edba8 sys___semctl(27069a80,27069d0c) at sys___semctl+0xa0/frame 0x289edc08 syscall(289edce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x289edcdc Xint0x80_syscall() at 0xffc033c9/frame 0x289edcdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc015b0, esp = 0xffc0dfe8, ebp = 0xffbfe760 --- Tracing command bash pid 901 tid 100192 td 0x188a0880 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa5ebc,1aa5ed4,289ed478,102bc8c,1aa5ed4,...) at db_trace_self_wrapper+0x2a/frame 0x289ed430 kdb_backtrace(289ed558,1416041,36,a07e50,36,...) at kdb_backtrace+0x2e/frame 0x289ed490 kdb_reenter(36,a07e50,36,a07e50,2cb86000,...) at kdb_reenter+0x29/frame 0x289ed498 trap(289ed564,8,28,28,ffffffff,...) at trap+0xd1/frame 0x289ed558 calltrap() at 0xffc0316f/frame 0x289ed558 --- trap 0xc, eip = 0x13f347a, esp = 0x289ed5a4, ebp = 0x289ed5b0 --- db_trace_thread(188a0880,ffffffff) at db_trace_thread+0x2a/frame 0x289ed5b0 _db_stack_trace_all(1017c05,0,ffffffff,289ed63c) at _db_stack_trace_all+0x93/frame 0x289ed610 db_command(1b2ff32) at db_command+0x26a/frame 0x289ed6c4 db_script_exec(bd452f,0,289ed76c,289ed750) at db_script_exec+0x1a6/frame 0x289ed724 db_command(289ed828,a07d16,0,0,27069a80,...) at db_command+0x26a/frame 0x289ed7d8 db_command_loop(0,0,27069a80,289ed7e0,289ed828,...) at db_command_loop+0x3c/frame 0x289ed7e0 db_trap(3,0) at db_trap+0xf6/frame 0x289ed828 kdb_trap(3,0,289ed934) at kdb_trap+0x119/frame 0x289ed85c trap(289ed934,8,28,28,100,...) at trap+0x38a/frame 0x289ed928 calltrap() at 0xffc0316f/frame 0x289ed928 --- trap 0x3, eip = 0x1017c05, esp = 0x289ed974, ebp = 0x289ed978 --- kdb_enter(1521cf1,1521cf1) at kdb_enter+0x35/frame 0x289ed978 vpanic(1461fb6,289ed9c4,289ed9c4,289eda00,1416c07,...) at vpanic+0x137/frame 0x289ed9a4 panic(1461fb6,14f09c1,14adb26,0,c,...) at panic+0x14/frame 0x289ed9b8 trap_fatal(2cb86000,1dbc480,2b31380,9890a6c,0,...) at trap_fatal+0x357/frame 0x289eda00 trap_pfault(1833f004,0,0) at trap_pfault+0x72/frame 0x289eda3c trap(289edb10,8,28,28,3ff,...) at trap+0x3a8/frame 0x289edb04 calltrap() at 0xffc0316f/frame 0x289edb04 --- trap 0xc, eip = 0x1046228, esp = 0x289edb50, ebp = 0x289edb70 --- sem_remove(289edbac,0,18342c10,1833d000,0,...) at sem_remove+0x208/frame 0x289edb70 kern_semctl(27069a80,10000,0,0,289edbc8,289edbd4) at kern_semctl+0x892/frame 0x289edba8 sys___semctl(27069a80,27069d0c) at sys___semctl+0xa0/frame 0x289edc08 syscall(289edce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x289edcdc Xint0x80_syscall() at 0xffc033c9/frame 0x289edcdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc015b0, esp = 0xffc0dfe8, ebp = 0xffbfe760 --- Tracing command sshd pid 900 tid 100202 td 0x23673880 sched_switch(23673880,104) at sched_switch+0x4f3/frame 0x23adb970 mi_switch(104) at mi_switch+0x1c4/frame 0x23adb9ac sleepq_switch(3ac0,0,23673880,1da1994,23ccb9fc,...) at sleepq_switch+0xf7/frame 0x23adb9cc sleepq_catch_signals(23adba38,f76998,183acf24,0,1441215,...) at sleepq_catch_signals+0x368/frame 0x23adb9f8 sleepq_wait_sig(183acf24,0) at sleepq_wait_sig+0xe/frame 0x23adba00 _cv_wait_sig(183acf24,183acf10) at _cv_wait_sig+0x138/frame 0x23adba38 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x23adba54 kern_select(23673880,e,2047a270,2047a278,0,0,20) at kern_select+0x8ed/frame 0x23adbbd4 sys_select(23673880,23673b0c) at sys_select+0x41/frame 0x23adbc08 syscall(23adbce8,3b,3b,3b,20480000,...) at syscall+0x306/frame 0x23adbcdc Xint0x80_syscall() at 0xffc033c9/frame 0x23adbcdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe660 --- Tracing command sshd pid 898 tid 100113 td 0x991e500 sched_switch(991e500,104) at sched_switch+0x4f3/frame 0x202df998 mi_switch(104) at mi_switch+0x1c4/frame 0x202df9d4 sleepq_switch(d40,0,991e500,1d9ec14,9dee000,...) at sleepq_switch+0xf7/frame 0x202df9f4 sleepq_catch_signals(202dfa60,f76998,183ad1e4,0,1441215,...) at sleepq_catch_signals+0x368/frame 0x202dfa20 sleepq_wait_sig(183ad1e4,0) at sleepq_wait_sig+0xe/frame 0x202dfa28 _cv_wait_sig(183ad1e4,183ad1d0) at _cv_wait_sig+0x138/frame 0x202dfa60 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x202dfa7c kern_poll(991e500,ffbfe800,1,0,0) at kern_poll+0x476/frame 0x202dfbd4 sys_poll(991e500,991e78c) at sys_poll+0x50/frame 0x202dfc08 syscall(202dfce8,3b,3b,3b,20480000,...) at syscall+0x306/frame 0x202dfcdc Xint0x80_syscall() at 0xffc033c9/frame 0x202dfcdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe660 --- Tracing command getty pid 774 tid 100199 td 0x236c4000 sched_switch(236c4000,104) at sched_switch+0x4f3/frame 0x23ab0a00 mi_switch(104) at mi_switch+0x1c4/frame 0x23ab0a3c sleepq_switch(1e00,0,236c4000,1d9fcd4,23ccc0a4,...) at sleepq_switch+0xf7/frame 0x23ab0a5c sleepq_catch_signals(23ab0ac8,f76998,18210c74,0,154d58b,...) at sleepq_catch_signals+0x368/frame 0x23ab0a88 sleepq_wait_sig(18210c74,0) at sleepq_wait_sig+0xe/frame 0x23ab0a90 _cv_wait_sig(18210c74,18210c04) at _cv_wait_sig+0x138/frame 0x23ab0ac8 tty_wait(18210c00,18210c74) at tty_wait+0x35/frame 0x23ab0ae0 ttydisc_read(18210c00,23ab0bd8,0) at ttydisc_read+0x1c6/frame 0x23ab0b14 ttydev_read(9966000,23ab0bd8,0) at ttydev_read+0x3d/frame 0x23ab0b30 devfs_read_f(18df5540,23ab0bd8,9d24100,0,236c4000) at devfs_read_f+0xaa/frame 0x23ab0b60 dofileread(18df5540,23ab0bd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x23ab0b90 kern_readv(236c4000,0,23ab0bd8) at kern_readv+0x3b/frame 0x23ab0bbc sys_read(236c4000,236c428c) at sys_read+0x48/frame 0x23ab0c08 syscall(23ab0ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x23ab0cdc Xint0x80_syscall() at 0xffc033c9/frame 0x23ab0cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0xffc01530, esp = 0xffc0bfe8, ebp = 0xffbfe374 --- Tracing command getty pid 773 tid 100198 td 0x236c4380 sched_switch(236c4380,104) at sched_switch+0x4f3/frame 0x23aaba00 mi_switch(104) at mi_switch+0x1c4/frame 0x23aaba3c sleepq_switch(600,0,236c4380,1d9e4d4,23ccc3f8,...) at sleepq_switch+0xf7/frame 0x23aaba5c sleepq_catch_signals(23aabac8,f76998,18966c74,0,154d58b,...) at sleepq_catch_signals+0x368/frame 0x23aaba88 sleepq_wait_sig(18966c74,0) at sleepq_wait_sig+0xe/frame 0x23aaba90 _cv_wait_sig(18966c74,18966c04) at _cv_wait_sig+0x138/frame 0x23aabac8 tty_wait(18966c00,18966c74) at tty_wait+0x35/frame 0x23aabae0 ttydisc_read(18966c00,23aabbd8,0) at ttydisc_read+0x1c6/frame 0x23aabb14 ttydev_read(181bb600,23aabbd8,0) at ttydev_read+0x3d/frame 0x23aabb30 devfs_read_f(18df55b0,23aabbd8,18186700,0,236c4380) at devfs_read_f+0xaa/frame 0x23aabb60 dofileread(18df55b0,23aabbd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x23aabb90 kern_readv(236c4380,0,23aabbd8) at kern_readv+0x3b/frame 0x23aabbbc sys_read(236c4380,236c460c) at sys_read+0x48/frame 0x23aabc08 syscall(23aabce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x23aabcdc Xint0x80_syscall() at 0xffc033c9/frame 0x23aabcdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f5, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command getty pid 772 tid 100197 td 0x236c4700 sched_switch(236c4700,104) at sched_switch+0x4f3/frame 0x23aa6a00 mi_switch(104) at mi_switch+0x1c4/frame 0x23aa6a3c sleepq_switch(100,0,236c4700,1d9dfd4,23ccc74c,...) at sleepq_switch+0xf7/frame 0x23aa6a5c sleepq_catch_signals(23aa6ac8,f76998,18967074,0,154d58b,...) at sleepq_catch_signals+0x368/frame 0x23aa6a88 sleepq_wait_sig(18967074,0) at sleepq_wait_sig+0xe/frame 0x23aa6a90 _cv_wait_sig(18967074,18967004) at _cv_wait_sig+0x138/frame 0x23aa6ac8 tty_wait(18967000,18967074) at tty_wait+0x35/frame 0x23aa6ae0 ttydisc_read(18967000,23aa6bd8,0) at ttydisc_read+0x1c6/frame 0x23aa6b14 ttydev_read(181bb800,23aa6bd8,0) at ttydev_read+0x3d/frame 0x23aa6b30 devfs_read_f(20431578,23aa6bd8,1888cb00,0,236c4700) at devfs_read_f+0xaa/frame 0x23aa6b60 dofileread(20431578,23aa6bd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x23aa6b90 kern_readv(236c4700,0,23aa6bd8) at kern_readv+0x3b/frame 0x23aa6bbc sys_read(236c4700,236c498c) at sys_read+0x48/frame 0x23aa6c08 syscall(23aa6ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x23aa6cdc Xint0x80_syscall() at 0xffc033c9/frame 0x23aa6cdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe660 --- Tracing command getty pid 771 tid 100196 td 0x236c4a80 sched_switch(236c4a80,104) at sched_switch+0x4f3/frame 0x23aa1a00 mi_switch(104) at mi_switch+0x1c4/frame 0x23aa1a3c sleepq_switch(0,0,236c4a80,1d9ded4,23cccaa0,...) at sleepq_switch+0xf7/frame 0x23aa1a5c sleepq_catch_signals(23aa1ac8,f76998,18967474,0,154d58b,...) at sleepq_catch_signals+0x368/frame 0x23aa1a88 sleepq_wait_sig(18967474,0) at sleepq_wait_sig+0xe/frame 0x23aa1a90 _cv_wait_sig(18967474,18967404) at _cv_wait_sig+0x138/frame 0x23aa1ac8 tty_wait(18967400,18967474) at tty_wait+0x35/frame 0x23aa1ae0 ttydisc_read(18967400,23aa1bd8,0) at ttydisc_read+0x1c6/frame 0x23aa1b14 ttydev_read(181bba00,23aa1bd8,0) at ttydev_read+0x3d/frame 0x23aa1b30 devfs_read_f(2044ece8,23aa1bd8,18186800,0,236c4a80) at devfs_read_f+0xaa/frame 0x23aa1b60 dofileread(2044ece8,23aa1bd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x23aa1b90 kern_readv(236c4a80,0,23aa1bd8) at kern_readv+0x3b/frame 0x23aa1bbc sys_read(236c4a80,236c4d0c) at sys_read+0x48/frame 0x23aa1c08 syscall(23aa1ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x23aa1cdc Xint0x80_syscall() at 0xffc033c9/frame 0x23aa1cdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe660 --- Tracing command getty pid 770 tid 100195 td 0x236c4e00 sched_switch(236c4e00,104) at sched_switch+0x4f3/frame 0x23a9ca00 mi_switch(104) at mi_switch+0x1c4/frame 0x23a9ca3c sleepq_switch(300,0,236c4e00,1d9e1d4,23cd1000,...) at sleepq_switch+0xf7/frame 0x23a9ca5c sleepq_catch_signals(23a9cac8,f76998,18967874,0,154d58b,...) at sleepq_catch_signals+0x368/frame 0x23a9ca88 sleepq_wait_sig(18967874,0) at sleepq_wait_sig+0xe/frame 0x23a9ca90 _cv_wait_sig(18967874,18967804) at _cv_wait_sig+0x138/frame 0x23a9cac8 tty_wait(18967800,18967874) at tty_wait+0x35/frame 0x23a9cae0 ttydisc_read(18967800,23a9cbd8,0) at ttydisc_read+0x1c6/frame 0x23a9cb14 ttydev_read(181bbc00,23a9cbd8,0) at ttydev_read+0x3d/frame 0x23a9cb30 devfs_read_f(2044ebd0,23a9cbd8,9d24200,0,236c4e00) at devfs_read_f+0xaa/frame 0x23a9cb60 dofileread(2044ebd0,23a9cbd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x23a9cb90 kern_readv(236c4e00,0,23a9cbd8) at kern_readv+0x3b/frame 0x23a9cbbc sys_read(236c4e00,236c508c) at sys_read+0x48/frame 0x23a9cc08 syscall(23a9cce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x23a9ccdc Xint0x80_syscall() at 0xffc033c9/frame 0x23a9ccdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0xffc01530, esp = 0xffc0bfe8, ebp = 0xffbfe374 --- Tracing command getty pid 769 tid 100190 td 0x20ad5000 sched_switch(20ad5000,104) at sched_switch+0x4f3/frame 0x23a83a00 mi_switch(104) at mi_switch+0x1c4/frame 0x23a83a3c sleepq_switch(200,0,20ad5000,1d9e0d4,205d1354,...) at sleepq_switch+0xf7/frame 0x23a83a5c sleepq_catch_signals(23a83ac8,f76998,18967c74,0,154d58b,...) at sleepq_catch_signals+0x368/frame 0x23a83a88 sleepq_wait_sig(18967c74,0) at sleepq_wait_sig+0xe/frame 0x23a83a90 _cv_wait_sig(18967c74,18967c04) at _cv_wait_sig+0x138/frame 0x23a83ac8 tty_wait(18967c00,18967c74) at tty_wait+0x35/frame 0x23a83ae0 ttydisc_read(18967c00,23a83bd8,0) at ttydisc_read+0x1c6/frame 0x23a83b14 ttydev_read(181bbe00,23a83bd8,0) at ttydev_read+0x3d/frame 0x23a83b30 devfs_read_f(20429a48,23a83bd8,9d24300,0,20ad5000) at devfs_read_f+0xaa/frame 0x23a83b60 dofileread(20429a48,23a83bd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x23a83b90 kern_readv(20ad5000,0,23a83bd8) at kern_readv+0x3b/frame 0x23a83bbc sys_read(20ad5000,20ad528c) at sys_read+0x48/frame 0x23a83c08 syscall(23a83ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x23a83cdc Xint0x80_syscall() at 0xffc033c9/frame 0x23a83cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0xffc01530, esp = 0xffc0bfe8, ebp = 0xffbfe374 --- Tracing command getty pid 768 tid 100193 td 0x23820700 sched_switch(23820700,104) at sched_switch+0x4f3/frame 0x23a92a00 mi_switch(104) at mi_switch+0x1c4/frame 0x23a92a3c sleepq_switch(3d00,0,23820700,1da1bd4,18183000,...) at sleepq_switch+0xf7/frame 0x23a92a5c sleepq_catch_signals(23a92ac8,f76998,18968074,0,154d58b,...) at sleepq_catch_signals+0x368/frame 0x23a92a88 sleepq_wait_sig(18968074,0) at sleepq_wait_sig+0xe/frame 0x23a92a90 _cv_wait_sig(18968074,18968004) at _cv_wait_sig+0x138/frame 0x23a92ac8 tty_wait(18968000,18968074) at tty_wait+0x35/frame 0x23a92ae0 ttydisc_read(18968000,23a92bd8,0) at ttydisc_read+0x1c6/frame 0x23a92b14 ttydev_read(181bc000,23a92bd8,0) at ttydev_read+0x3d/frame 0x23a92b30 devfs_read_f(18df53b8,23a92bd8,23cd0000,0,23820700) at devfs_read_f+0xaa/frame 0x23a92b60 dofileread(18df53b8,23a92bd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x23a92b90 kern_readv(23820700,0,23a92bd8) at kern_readv+0x3b/frame 0x23a92bbc sys_read(23820700,2382098c) at sys_read+0x48/frame 0x23a92c08 syscall(23a92ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x23a92cdc Xint0x80_syscall() at 0xffc033c9/frame 0x23a92cdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f5, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command getty pid 767 tid 100194 td 0x23820380 sched_switch(23820380,104) at sched_switch+0x4f3/frame 0x23a97a00 mi_switch(104) at mi_switch+0x1c4/frame 0x23a97a3c sleepq_switch(3c00,0,23820380,1da1ad4,9df1aa0,...) at sleepq_switch+0xf7/frame 0x23a97a5c sleepq_catch_signals(23a97ac8,f76998,18968474,0,154d58b,...) at sleepq_catch_signals+0x368/frame 0x23a97a88 sleepq_wait_sig(18968474,0) at sleepq_wait_sig+0xe/frame 0x23a97a90 _cv_wait_sig(18968474,18968404) at _cv_wait_sig+0x138/frame 0x23a97ac8 tty_wait(18968400,18968474) at tty_wait+0x35/frame 0x23a97ae0 ttydisc_read(18968400,23a97bd8,0) at ttydisc_read+0x1c6/frame 0x23a97b14 ttydev_read(181bc200,23a97bd8,0) at ttydev_read+0x3d/frame 0x23a97b30 devfs_read_f(2044ec40,23a97bd8,1888d100,0,23820380) at devfs_read_f+0xaa/frame 0x23a97b60 dofileread(2044ec40,23a97bd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x23a97b90 kern_readv(23820380,0,23a97bd8) at kern_readv+0x3b/frame 0x23a97bbc sys_read(23820380,2382060c) at sys_read+0x48/frame 0x23a97c08 syscall(23a97ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x23a97cdc Xint0x80_syscall() at 0xffc033c9/frame 0x23a97cdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe660 --- Tracing command getty pid 766 tid 100105 td 0x18891a80 sched_switch(18891a80,104) at sched_switch+0x4f3/frame 0x1bdffa00 mi_switch(104) at mi_switch+0x1c4/frame 0x1bdffa3c sleepq_switch(3f00,0,18891a80,1da1dd4,1888a74c,...) at sleepq_switch+0xf7/frame 0x1bdffa5c sleepq_catch_signals(1bdffac8,f76998,18968874,0,154d58b,...) at sleepq_catch_signals+0x368/frame 0x1bdffa88 sleepq_wait_sig(18968874,0) at sleepq_wait_sig+0xe/frame 0x1bdffa90 _cv_wait_sig(18968874,18968804) at _cv_wait_sig+0x138/frame 0x1bdffac8 tty_wait(18968800,18968874) at tty_wait+0x35/frame 0x1bdffae0 ttydisc_read(18968800,1bdffbd8,0) at ttydisc_read+0x1c6/frame 0x1bdffb14 ttydev_read(181bc400,1bdffbd8,0) at ttydev_read+0x3d/frame 0x1bdffb30 devfs_read_f(2044ed20,1bdffbd8,18186a00,0,18891a80) at devfs_read_f+0xaa/frame 0x1bdffb60 dofileread(2044ed20,1bdffbd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x1bdffb90 kern_readv(18891a80,0,1bdffbd8) at kern_readv+0x3b/frame 0x1bdffbbc sys_read(18891a80,18891d0c) at sys_read+0x48/frame 0x1bdffc08 syscall(1bdffce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x1bdffcdc Xint0x80_syscall() at 0xffc033c9/frame 0x1bdffcdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f5, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command inetd pid 749 tid 100171 td 0x23830500 sched_switch(23830500,104) at sched_switch+0x4f3/frame 0x23a10970 mi_switch(104) at mi_switch+0x1c4/frame 0x23a109ac sleepq_switch(3d80,0,23830500,1da1c54,187c03f8,...) at sleepq_switch+0xf7/frame 0x23a109cc sleepq_catch_signals(23a10a38,f76998,183ad224,0,1441215,...) at sleepq_catch_signals+0x368/frame 0x23a109f8 sleepq_wait_sig(183ad224,0) at sleepq_wait_sig+0xe/frame 0x23a10a00 _cv_wait_sig(183ad224,183ad210) at _cv_wait_sig+0x138/frame 0x23a10a38 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x23a10a54 kern_select(23830500,9,ffbfe638,0,0,0,20) at kern_select+0x8ed/frame 0x23a10bd4 sys_select(23830500,2383078c) at sys_select+0x41/frame 0x23a10c08 syscall(23a10ce8,3b,3b,3b,ffbfe6b8,...) at syscall+0x306/frame 0x23a10cdc Xint0x80_syscall() at 0xffc033c9/frame 0x23a10cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0xffc01530, esp = 0xffc0bfe8, ebp = 0xffbfe374 --- Tracing command nfscbd pid 715 tid 100178 td 0x236c5c00 sched_switch(236c5c00,104) at sched_switch+0x4f3/frame 0x23a33888 mi_switch(104) at mi_switch+0x1c4/frame 0x23a338c4 sleepq_switch(2c0,0,1dccac8,1d9e194,23cd1d50,...) at sleepq_switch+0xf7/frame 0x23a338e4 sleepq_catch_signals(e2,236c5c00,1abc268,236c5c00,23a33944,...) at sleepq_catch_signals+0x368/frame 0x23a33910 sleepq_timedwait_sig(9935b50,0) at sleepq_timedwait_sig+0x17/frame 0x23a33940 _cv_timedwait_sig_sbt(9935b50,186520c0,fffffa38,4,0,0,100) at _cv_timedwait_sig_sbt+0x15a/frame 0x23a3397c svc_run_internal(20ad5700,186520c0,186520d0,1865211c,0,...) at svc_run_internal+0x97d/frame 0x23a33a60 svc_run(18652000) at svc_run+0x192/frame 0x23a33a90 nfscbd_nfsd(236c5c00,23a33b48) at nfscbd_nfsd+0xd0/frame 0x23a33b20 nfssvc_nfscl(236c5c00,236c5e8c) at nfssvc_nfscl+0x243/frame 0x23a33bf0 sys_nfssvc(236c5c00,236c5e8c) at sys_nfssvc+0x92/frame 0x23a33c08 syscall(23a33ce8,3b,3b,3b,1,...) at syscall+0x306/frame 0x23a33cdc Xint0x80_syscall() at 0xffc033c9/frame 0x23a33cdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe660 --- Tracing command nfscbd pid 715 tid 100186 td 0x20ad5e00 sched_switch(20ad5e00,104) at sched_switch+0x4f3/frame 0x23a63b18 mi_switch(104) at mi_switch+0x1c4/frame 0x23a63b54 sleepq_switch(280,0,20ad5e00,1d9e154,23cd1d50,...) at sleepq_switch+0xf7/frame 0x23a63b74 sleepq_catch_signals(23a63be0,f76998,188d5a50,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x23a63ba0 sleepq_wait_sig(188d5a50,0) at sleepq_wait_sig+0xe/frame 0x23a63ba8 _cv_wait_sig(188d5a50,186520c0) at _cv_wait_sig+0x138/frame 0x23a63be0 svc_run_internal(23a63cd4,f98ac9,186520c0,23a63ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x23a63cac svc_thread_start(186520c0,23a63ce8) at svc_thread_start+0xd/frame 0x23a63cb4 fork_exit(1253d10,186520c0,23a63ce8,3b,3b,...) at fork_exit+0x69/frame 0x23a63cd4 fork_trampoline() at 0xffc033de/frame 0x23a63cd4 --- kthread start Tracing command nfscbd pid 715 tid 100187 td 0x20ad5a80 sched_switch(20ad5a80,104) at sched_switch+0x4f3/frame 0x23a68b18 mi_switch(104) at mi_switch+0x1c4/frame 0x23a68b54 sleepq_switch(2240,0,20ad5a80,1da0114,23cd1d50,...) at sleepq_switch+0xf7/frame 0x23a68b74 sleepq_catch_signals(23a68be0,f76998,188d59d0,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x23a68ba0 sleepq_wait_sig(188d59d0,0) at sleepq_wait_sig+0xe/frame 0x23a68ba8 _cv_wait_sig(188d59d0,186520c0) at _cv_wait_sig+0x138/frame 0x23a68be0 svc_run_internal(23a68cd4,f98ac9,186520c0,23a68ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x23a68cac svc_thread_start(186520c0,23a68ce8) at svc_thread_start+0xd/frame 0x23a68cb4 fork_exit(1253d10,186520c0,23a68ce8,3b,3b,...) at fork_exit+0x69/frame 0x23a68cd4 fork_trampoline() at 0xffc033de/frame 0x23a68cd4 --- kthread start Tracing command nfscbd pid 715 tid 100188 td 0x20ad5700 sched_switch(20ad5700,104) at sched_switch+0x4f3/frame 0x23a6db18 mi_switch(104) at mi_switch+0x1c4/frame 0x23a6db54 sleepq_switch(240,0,20ad5700,1d9e114,23cd1d50,...) at sleepq_switch+0xf7/frame 0x23a6db74 sleepq_catch_signals(23a6dbe0,f76998,188d5950,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x23a6dba0 sleepq_wait_sig(188d5950,0) at sleepq_wait_sig+0xe/frame 0x23a6dba8 _cv_wait_sig(188d5950,186520c0) at _cv_wait_sig+0x138/frame 0x23a6dbe0 svc_run_internal(23a6dcd4,f98ac9,186520c0,23a6dce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x23a6dcac svc_thread_start(186520c0,23a6dce8) at svc_thread_start+0xd/frame 0x23a6dcb4 fork_exit(1253d10,186520c0,23a6dce8,3b,3b,...) at fork_exit+0x69/frame 0x23a6dcd4 fork_trampoline() at 0xffc033de/frame 0x23a6dcd4 --- kthread start Tracing command nfscbd pid 714 tid 100132 td 0x23672a80 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa5ebc,1aa5ed4,289ed478,102bc8c,1aa5ed4,...) at db_trace_self_wrapper+0x2a/frame 0x289ed430 kdb_backtrace(289ed558,1416041,38,a07e50,38,...) at kdb_backtrace+0x2e/frame 0x289ed490 kdb_reenter(38,a07e50,38,a07e50,2cb86000,...) at kdb_reenter+0x29/frame 0x289ed498 trap(289ed564,8,28,28,ffffffff,...) at trap+0xd1/frame 0x289ed558 calltrap() at 0xffc0316f/frame 0x289ed558 --- trap 0xc, eip = 0x13f347a, esp = 0x289ed5a4, ebp = 0x289ed5b0 --- db_trace_thread(23672a80,ffffffff) at db_trace_thread+0x2a/frame 0x289ed5b0 _db_stack_trace_all(1017c05,0,ffffffff,289ed63c) at _db_stack_trace_all+0x93/frame 0x289ed610 db_command(1b2ff32) at db_command+0x26a/frame 0x289ed6c4 db_script_exec(bd452f,0,289ed76c,289ed750) at db_script_exec+0x1a6/frame 0x289ed724 db_command(289ed828,a07d16,0,0,27069a80,...) at db_command+0x26a/frame 0x289ed7d8 db_command_loop(0,0,27069a80,289ed7e0,289ed828,...) at db_command_loop+0x3c/frame 0x289ed7e0 db_trap(3,0) at db_trap+0xf6/frame 0x289ed828 kdb_trap(3,0,289ed934) at kdb_trap+0x119/frame 0x289ed85c trap(289ed934,8,28,28,100,...) at trap+0x38a/frame 0x289ed928 calltrap() at 0xffc0316f/frame 0x289ed928 --- trap 0x3, eip = 0x1017c05, esp = 0x289ed974, ebp = 0x289ed978 --- kdb_enter(1521cf1,1521cf1) at kdb_enter+0x35/frame 0x289ed978 vpanic(1461fb6,289ed9c4,289ed9c4,289eda00,1416c07,...) at vpanic+0x137/frame 0x289ed9a4 panic(1461fb6,14f09c1,14adb26,0,c,...) at panic+0x14/frame 0x289ed9b8 trap_fatal(2cb86000,1dbc480,2b31380,9890a6c,0,...) at trap_fatal+0x357/frame 0x289eda00 trap_pfault(1833f004,0,0) at trap_pfault+0x72/frame 0x289eda3c trap(289edb10,8,28,28,3ff,...) at trap+0x3a8/frame 0x289edb04 calltrap() at 0xffc0316f/frame 0x289edb04 --- trap 0xc, eip = 0x1046228, esp = 0x289edb50, ebp = 0x289edb70 --- sem_remove(289edbac,0,18342c10,1833d000,0,...) at sem_remove+0x208/frame 0x289edb70 kern_semctl(27069a80,10000,0,0,289edbc8,289edbd4) at kern_semctl+0x892/frame 0x289edba8 sys___semctl(27069a80,27069d0c) at sys___semctl+0xa0/frame 0x289edc08 syscall(289edce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x289edcdc Xint0x80_syscall() at 0xffc033c9/frame 0x289edcdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc015b0, esp = 0xffc0dfe8, ebp = 0xffbfe760 --- Tracing command cron pid 678 tid 100109 td 0x18891380 sched_switch(18891380,104) at sched_switch+0x4f3/frame 0x202aba68 mi_switch(104) at mi_switch+0x1c4/frame 0x202abaa4 sleepq_switch(3f00,6c,3,1da1dd4,1888a0a4,...) at sleepq_switch+0xf7/frame 0x202abac4 sleepq_catch_signals(0,1526a9e,18891380,100,1d901fd,...) at sleepq_catch_signals+0x368/frame 0x202abaf0 sleepq_timedwait_sig(1d901fd,6c,0,0,20428e38,...) at sleepq_timedwait_sig+0x17/frame 0x202abb1c _sleep(1d901fd,0,16c,15397ea,76899827,ddcb,bff7f443,3,200) at _sleep+0x21c/frame 0x202abb5c kern_clock_nanosleep(18891380,0,0,202abbe8,202abbe0) at kern_clock_nanosleep+0x1b4/frame 0x202abbc4 sys_nanosleep(18891380,1889160c) at sys_nanosleep+0x55/frame 0x202abc08 syscall(202abce8,3b,3b,3b,ffbfed40,...) at syscall+0x306/frame 0x202abcdc Xint0x80_syscall() at 0xffc033c9/frame 0x202abcdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f5, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command sendmail pid 674 tid 100174 td 0x20ad6180 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa5ebc,1aa5ed4,289ed478,102bc8c,1aa5ed4,...) at db_trace_self_wrapper+0x2a/frame 0x289ed430 kdb_backtrace(289ed558,1416041,3a,a07e50,3a,...) at kdb_backtrace+0x2e/frame 0x289ed490 kdb_reenter(3a,a07e50,3a,a07e50,2cb86000,...) at kdb_reenter+0x29/frame 0x289ed498 trap(289ed564,8,28,28,ffffffff,...) at trap+0xd1/frame 0x289ed558 calltrap() at 0xffc0316f/frame 0x289ed558 --- trap 0xc, eip = 0x13f347a, esp = 0x289ed5a4, ebp = 0x289ed5b0 --- db_trace_thread(20ad6180,ffffffff) at db_trace_thread+0x2a/frame 0x289ed5b0 _db_stack_trace_all(1017c05,0,ffffffff,289ed63c) at _db_stack_trace_all+0x93/frame 0x289ed610 db_command(1b2ff32) at db_command+0x26a/frame 0x289ed6c4 db_script_exec(bd452f,0,289ed76c,289ed750) at db_script_exec+0x1a6/frame 0x289ed724 db_command(289ed828,a07d16,0,0,27069a80,...) at db_command+0x26a/frame 0x289ed7d8 db_command_loop(0,0,27069a80,289ed7e0,289ed828,...) at db_command_loop+0x3c/frame 0x289ed7e0 db_trap(3,0) at db_trap+0xf6/frame 0x289ed828 kdb_trap(3,0,289ed934) at kdb_trap+0x119/frame 0x289ed85c trap(289ed934,8,28,28,100,...) at trap+0x38a/frame 0x289ed928 calltrap() at 0xffc0316f/frame 0x289ed928 --- trap 0x3, eip = 0x1017c05, esp = 0x289ed974, ebp = 0x289ed978 --- kdb_enter(1521cf1,1521cf1) at kdb_enter+0x35/frame 0x289ed978 vpanic(1461fb6,289ed9c4,289ed9c4,289eda00,1416c07,...) at vpanic+0x137/frame 0x289ed9a4 panic(1461fb6,14f09c1,14adb26,0,c,...) at panic+0x14/frame 0x289ed9b8 trap_fatal(2cb86000,1dbc480,2b31380,9890a6c,0,...) at trap_fatal+0x357/frame 0x289eda00 trap_pfault(1833f004,0,0) at trap_pfault+0x72/frame 0x289eda3c trap(289edb10,8,28,28,3ff,...) at trap+0x3a8/frame 0x289edb04 calltrap() at 0xffc0316f/frame 0x289edb04 --- trap 0xc, eip = 0x1046228, esp = 0x289edb50, ebp = 0x289edb70 --- sem_remove(289edbac,0,18342c10,1833d000,0,...) at sem_remove+0x208/frame 0x289edb70 kern_semctl(27069a80,10000,0,0,289edbc8,289edbd4) at kern_semctl+0x892/frame 0x289edba8 sys___semctl(27069a80,27069d0c) at sys___semctl+0xa0/frame 0x289edc08 syscall(289edce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x289edcdc Xint0x80_syscall() at 0xffc033c9/frame 0x289edcdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc015b0, esp = 0xffc0dfe8, ebp = 0xffbfe760 --- Tracing command sendmail pid 671 tid 100134 td 0x236c6a80 sched_switch(236c6a80,104) at sched_switch+0x4f3/frame 0x20ed4930 mi_switch(104) at mi_switch+0x1c4/frame 0x20ed496c sleepq_switch(1e00,0,1dccb78,1d9fcd4,1818a74c,...) at sleepq_switch+0xf7/frame 0x20ed498c sleepq_catch_signals(e2,236c6a80,1abc268,236c6a80,20ed49ec,...) at sleepq_catch_signals+0x368/frame 0x20ed49b8 sleepq_timedwait_sig(9df5c24,0) at sleepq_timedwait_sig+0x17/frame 0x20ed49e8 _cv_timedwait_sig_sbt(9df5c24,9df5c10,10201460,ddb3,50000000,0,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x20ed4a24 seltdwait(ddb3,50000000,0) at seltdwait+0x5e/frame 0x20ed4a54 kern_select(236c6a80,5,ffbfd170,0,0,20ed4bf8,20) at kern_select+0x8ed/frame 0x20ed4bd4 sys_select(236c6a80,236c6d0c) at sys_select+0x41/frame 0x20ed4c08 syscall(20ed4ce8,3b,3b,3b,20c0c000,...) at syscall+0x306/frame 0x20ed4cdc Xint0x80_syscall() at 0xffc033c9/frame 0x20ed4cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc015b0, esp = 0xffc0dfe8, ebp = 0xffbfe760 --- Tracing command sshd pid 668 tid 100124 td 0x20ad6880 sched_switch(20ad6880,104) at sched_switch+0x4f3/frame 0x203aa970 mi_switch(104) at mi_switch+0x1c4/frame 0x203aa9ac sleepq_switch(1dc0,0,20ad6880,1d9fc94,205d20a4,...) at sleepq_switch+0xf7/frame 0x203aa9cc sleepq_catch_signals(203aaa38,f76998,183ad3a4,0,1441215,...) at sleepq_catch_signals+0x368/frame 0x203aa9f8 sleepq_wait_sig(183ad3a4,0) at sleepq_wait_sig+0xe/frame 0x203aaa00 _cv_wait_sig(183ad3a4,183ad390) at _cv_wait_sig+0x138/frame 0x203aaa38 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x203aaa54 kern_select(20ad6880,7,2047a180,0,0,0,20) at kern_select+0x8ed/frame 0x203aabd4 sys_select(20ad6880,20ad6b0c) at sys_select+0x41/frame 0x203aac08 syscall(203aace8,3b,3b,3b,20480000,...) at syscall+0x306/frame 0x203aacdc Xint0x80_syscall() at 0xffc033c9/frame 0x203aacdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f5, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command smartd pid 628 tid 100123 td 0x181df880 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa5ebc,1aa5ed4,289ed478,102bc8c,1aa5ed4,...) at db_trace_self_wrapper+0x2a/frame 0x289ed430 kdb_backtrace(289ed558,1416041,38,a07e50,38,...) at kdb_backtrace+0x2e/frame 0x289ed490 kdb_reenter(38,a07e50,38,a07e50,2cb86000,...) at kdb_reenter+0x29/frame 0x289ed498 trap(289ed564,8,28,28,ffffffff,...) at trap+0xd1/frame 0x289ed558 calltrap() at 0xffc0316f/frame 0x289ed558 --- trap 0xc, eip = 0x13f347a, esp = 0x289ed5a4, ebp = 0x289ed5b0 --- db_trace_thread(181df880,ffffffff) at db_trace_thread+0x2a/frame 0x289ed5b0 _db_stack_trace_all(1017c05,0,ffffffff,289ed63c) at _db_stack_trace_all+0x93/frame 0x289ed610 db_command(1b2ff32) at db_command+0x26a/frame 0x289ed6c4 db_script_exec(bd452f,0,289ed76c,289ed750) at db_script_exec+0x1a6/frame 0x289ed724 db_command(289ed828,a07d16,0,0,27069a80,...) at db_command+0x26a/frame 0x289ed7d8 db_command_loop(0,0,27069a80,289ed7e0,289ed828,...) at db_command_loop+0x3c/frame 0x289ed7e0 db_trap(3,0) at db_trap+0xf6/frame 0x289ed828 kdb_trap(3,0,289ed934) at kdb_trap+0x119/frame 0x289ed85c trap(289ed934,8,28,28,100,...) at trap+0x38a/frame 0x289ed928 calltrap() at 0xffc0316f/frame 0x289ed928 --- trap 0x3, eip = 0x1017c05, esp = 0x289ed974, ebp = 0x289ed978 --- kdb_enter(1521cf1,1521cf1) at kdb_enter+0x35/frame 0x289ed978 vpanic(1461fb6,289ed9c4,289ed9c4,289eda00,1416c07,...) at vpanic+0x137/frame 0x289ed9a4 panic(1461fb6,14f09c1,14adb26,0,c,...) at panic+0x14/frame 0x289ed9b8 trap_fatal(2cb86000,1dbc480,2b31380,9890a6c,0,...) at trap_fatal+0x357/frame 0x289eda00 trap_pfault(1833f004,0,0) at trap_pfault+0x72/frame 0x289eda3c trap(289edb10,8,28,28,3ff,...) at trap+0x3a8/frame 0x289edb04 calltrap() at 0xffc0316f/frame 0x289edb04 --- trap 0xc, eip = 0x1046228, esp = 0x289edb50, ebp = 0x289edb70 --- sem_remove(289edbac,0,18342c10,1833d000,0,...) at sem_remove+0x208/frame 0x289edb70 kern_semctl(27069a80,10000,0,0,289edbc8,289edbd4) at kern_semctl+0x892/frame 0x289edba8 sys___semctl(27069a80,27069d0c) at sys___semctl+0xa0/frame 0x289edc08 syscall(289edce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x289edcdc Xint0x80_syscall() at 0xffc033c9/frame 0x289edcdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc015b0, esp = 0xffc0dfe8, ebp = 0xffbfe760 --- Tracing command powerd pid 604 tid 100114 td 0x181dc380 sched_switch(181dc380,104) at sched_switch+0x4f3/frame 0x202e4930 mi_switch(104) at mi_switch+0x1c4/frame 0x202e496c sleepq_switch(1cc0,0,1dcccd8,1d9fb94,181896a8,...) at sleepq_switch+0xf7/frame 0x202e498c sleepq_catch_signals(e2,181dc380,1abc268,181dc380,202e49ec,...) at sleepq_catch_signals+0x368/frame 0x202e49b8 sleepq_timedwait_sig(183ad7a4,0) at sleepq_timedwait_sig+0x17/frame 0x202e49e8 _cv_timedwait_sig_sbt(183ad7a4,183ad790,7998e5d7,ddb2,4000000,0,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x202e4a24 seltdwait(ddb2,4000000,0) at seltdwait+0x5e/frame 0x202e4a54 kern_select(181dc380,0,ffbfed88,0,ffbfed88,202e4bf8,20) at kern_select+0x8ed/frame 0x202e4bd4 sys_select(181dc380,181dc60c) at sys_select+0x41/frame 0x202e4c08 syscall(202e4ce8,3b,3b,3b,ffbfee08,...) at syscall+0x306/frame 0x202e4cdc Xint0x80_syscall() at 0xffc033c9/frame 0x202e4cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0xffc01530, esp = 0xffc0bfe8, ebp = 0xffbfe374 --- Tracing command ntpd pid 601 tid 100133 td 0x236c6e00 sched_switch(236c6e00,104) at sched_switch+0x4f3/frame 0x20ecf970 mi_switch(104) at mi_switch+0x1c4/frame 0x20ecf9ac sleepq_switch(2580,0,236c6e00,1da0454,1818aaa0,...) at sleepq_switch+0xf7/frame 0x20ecf9cc sleepq_catch_signals(20ecfa38,f76998,187df264,0,1441215,...) at sleepq_catch_signals+0x368/frame 0x20ecf9f8 sleepq_wait_sig(187df264,0) at sleepq_wait_sig+0xe/frame 0x20ecfa00 _cv_wait_sig(187df264,187df250) at _cv_wait_sig+0x138/frame 0x20ecfa38 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x20ecfa54 kern_select(236c6e00,1b,ffbfe2c0,0,0,0,20) at kern_select+0x8ed/frame 0x20ecfbd4 sys_select(236c6e00,236c708c) at sys_select+0x41/frame 0x20ecfc08 syscall(20ecfce8,3b,3b,3b,20994000,...) at syscall+0x306/frame 0x20ecfcdc Xint0x80_syscall() at 0xffc033c9/frame 0x20ecfcdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe660 --- Tracing command rpc.lockd pid 563 tid 100170 td 0x23830880 sched_switch(23830880,104) at sched_switch+0x4f3/frame 0x20ff48e0 mi_switch(104) at mi_switch+0x1c4/frame 0x20ff491c sleepq_switch(300,0,1dccd88,1d9e1d4,187c074c,...) at sleepq_switch+0xf7/frame 0x20ff493c sleepq_catch_signals(e2,23830880,1abc268,23830880,20ff499c,...) at sleepq_catch_signals+0x368/frame 0x20ff4968 sleepq_timedwait_sig(188d5c50,0) at sleepq_timedwait_sig+0x17/frame 0x20ff4998 _cv_timedwait_sig_sbt(188d5c50,236c80c0,fffffa38,4,0,0,100) at _cv_timedwait_sig_sbt+0x15a/frame 0x20ff49d4 svc_run_internal(4,236c8000,20ff4c08,1238575,9c54980,...) at svc_run_internal+0x97d/frame 0x20ff4ab4 svc_run(236c8000) at svc_run+0x192/frame 0x20ff4ae4 sys_nlm_syscall(23830880,23830b0c) at sys_nlm_syscall+0x625/frame 0x20ff4c08 syscall(20ff4ce8,3b,3b,3b,20738000,...) at syscall+0x306/frame 0x20ff4cdc Xint0x80_syscall() at 0xffc033c9/frame 0x20ff4cdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe660 --- Tracing command rpc.statd pid 560 tid 100169 td 0x236c6700 sched_switch(236c6700,104) at sched_switch+0x4f3/frame 0x20fef930 mi_switch(104) at mi_switch+0x1c4/frame 0x20fef96c sleepq_switch(2840,0,1dccde0,1da0714,1818a3f8,...) at sleepq_switch+0xf7/frame 0x20fef98c sleepq_catch_signals(e2,236c6700,1abc268,236c6700,20fef9ec,...) at sleepq_catch_signals+0x368/frame 0x20fef9b8 sleepq_timedwait_sig(17ee05a4,0) at sleepq_timedwait_sig+0x17/frame 0x20fef9e8 _cv_timedwait_sig_sbt(17ee05a4,17ee0590,74fdea49,ddb6,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x20fefa24 seltdwait(ddb6,e0000000,1) at seltdwait+0x5e/frame 0x20fefa54 kern_select(236c6700,8,ffbfecf8,0,0,20fefbf8,20) at kern_select+0x8ed/frame 0x20fefbd4 sys_select(236c6700,236c698c) at sys_select+0x41/frame 0x20fefc08 syscall(20fefce8,3b,3b,3b,ffbfecf8,...) at syscall+0x306/frame 0x20fefcdc Xint0x80_syscall() at 0xffc033c9/frame 0x20fefcdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc015b0, esp = 0xffc0dfe8, ebp = 0xffbfe760 --- Tracing command nfsd pid 559 tid 100137 td 0x187e1e00 sched_switch(187e1e00,104) at sched_switch+0x4f3/frame 0x20f4b440 mi_switch(104) at mi_switch+0x1c4/frame 0x20f4b47c sleepq_switch(f00,0,1dcd8e0,1d9edd4,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20f4b49c sleepq_catch_signals(e2,187e1e00,1abc268,187e1e00,20f4b4fc,...) at sleepq_catch_signals+0x368/frame 0x20f4b4c8 sleepq_timedwait_sig(188d6c50,0) at sleepq_timedwait_sig+0x17/frame 0x20f4b4f8 _cv_timedwait_sig_sbt(188d6c50,186510c0,fffffa38,4,0,0,100) at _cv_timedwait_sig_sbt+0x15a/frame 0x20f4b534 svc_run_internal(23821880,186510c0,186510d0,1865111c,0,...) at svc_run_internal+0x97d/frame 0x20f4b618 svc_run(18651000) at svc_run+0x192/frame 0x20f4b648 nfsrvd_nfsd(187e1e00,20f4b7d0) at nfsrvd_nfsd+0x2ec/frame 0x20f4b778 nfssvc_nfsd(187e1e00,187e208c) at nfssvc_nfsd+0x4ea/frame 0x20f4bbf0 sys_nfssvc(187e1e00,187e208c) at sys_nfssvc+0x92/frame 0x20f4bc08 syscall(20f4bce8,3b,3b,3b,ffbfeba0,...) at syscall+0x306/frame 0x20f4bcdc Xint0x80_syscall() at 0xffc033c9/frame 0x20f4bcdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0xffc01530, esp = 0xffc0bfe8, ebp = 0xffbfe374 --- Tracing command nfsd pid 559 tid 100138 td 0x23672700 sched_switch(23672700,104) at sched_switch+0x4f3/frame 0x20f54b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20f54b54 sleepq_switch(2c40,0,23672700,1da0b14,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20f54b74 sleepq_catch_signals(20f54be0,f76998,188d61d0,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20f54ba0 sleepq_wait_sig(188d61d0,0) at sleepq_wait_sig+0xe/frame 0x20f54ba8 _cv_wait_sig(188d61d0,186510c0) at _cv_wait_sig+0x138/frame 0x20f54be0 svc_run_internal(20f54cd4,f98ac9,186510c0,20f54ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20f54cac svc_thread_start(186510c0,20f54ce8) at svc_thread_start+0xd/frame 0x20f54cb4 fork_exit(1253d10,186510c0,20f54ce8,3b,3b,...) at fork_exit+0x69/frame 0x20f54cd4 fork_trampoline() at 0xffc033de/frame 0x20f54cd4 --- kthread start Tracing command nfsd pid 559 tid 100139 td 0x23672380 sched_switch(23672380,104) at sched_switch+0x4f3/frame 0x20f59b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20f59b54 sleepq_switch(c40,0,23672380,1d9eb14,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20f59b74 sleepq_catch_signals(20f59be0,f76998,188d6150,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20f59ba0 sleepq_wait_sig(188d6150,0) at sleepq_wait_sig+0xe/frame 0x20f59ba8 _cv_wait_sig(188d6150,186510c0) at _cv_wait_sig+0x138/frame 0x20f59be0 svc_run_internal(20f59cd4,f98ac9,186510c0,20f59ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20f59cac svc_thread_start(186510c0,20f59ce8) at svc_thread_start+0xd/frame 0x20f59cb4 fork_exit(1253d10,186510c0,20f59ce8,3b,3b,...) at fork_exit+0x69/frame 0x20f59cd4 fork_trampoline() at 0xffc033de/frame 0x20f59cd4 --- kthread start Tracing command nfsd pid 559 tid 100140 td 0x23672000 sched_switch(23672000,104) at sched_switch+0x4f3/frame 0x20f5eb18 mi_switch(104) at mi_switch+0x1c4/frame 0x20f5eb54 sleepq_switch(ec0,0,23672000,1d9ed94,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20f5eb74 sleepq_catch_signals(20f5ebe0,f76998,188d6b50,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20f5eba0 sleepq_wait_sig(188d6b50,0) at sleepq_wait_sig+0xe/frame 0x20f5eba8 _cv_wait_sig(188d6b50,186510c0) at _cv_wait_sig+0x138/frame 0x20f5ebe0 svc_run_internal(20f5ecd4,f98ac9,186510c0,20f5ece8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20f5ecac svc_thread_start(186510c0,20f5ece8) at svc_thread_start+0xd/frame 0x20f5ecb4 fork_exit(1253d10,186510c0,20f5ece8,3b,3b,...) at fork_exit+0x69/frame 0x20f5ecd4 fork_trampoline() at 0xffc033de/frame 0x20f5ecd4 --- kthread start Tracing command nfsd pid 559 tid 100141 td 0x23671c00 sched_switch(23671c00,104) at sched_switch+0x4f3/frame 0x20f63b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20f63b54 sleepq_switch(2d80,0,23671c00,1da0c54,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20f63b74 sleepq_catch_signals(20f63be0,f76998,99366d0,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20f63ba0 sleepq_wait_sig(99366d0,0) at sleepq_wait_sig+0xe/frame 0x20f63ba8 _cv_wait_sig(99366d0,186510c0) at _cv_wait_sig+0x138/frame 0x20f63be0 svc_run_internal(20f63cd4,f98ac9,186510c0,20f63ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20f63cac svc_thread_start(186510c0,20f63ce8) at svc_thread_start+0xd/frame 0x20f63cb4 fork_exit(1253d10,186510c0,20f63ce8,3b,3b,...) at fork_exit+0x69/frame 0x20f63cd4 fork_trampoline() at 0xffc033de/frame 0x20f63cd4 --- kthread start Tracing command nfsd pid 559 tid 100142 td 0x23671880 sched_switch(23671880,104) at sched_switch+0x4f3/frame 0x20f68b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20f68b54 sleepq_switch(d80,0,23671880,1d9ec54,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20f68b74 sleepq_catch_signals(20f68be0,f76998,9936650,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20f68ba0 sleepq_wait_sig(9936650,0) at sleepq_wait_sig+0xe/frame 0x20f68ba8 _cv_wait_sig(9936650,186510c0) at _cv_wait_sig+0x138/frame 0x20f68be0 svc_run_internal(20f68cd4,f98ac9,186510c0,20f68ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20f68cac svc_thread_start(186510c0,20f68ce8) at svc_thread_start+0xd/frame 0x20f68cb4 fork_exit(1253d10,186510c0,20f68ce8,3b,3b,...) at fork_exit+0x69/frame 0x20f68cd4 fork_trampoline() at 0xffc033de/frame 0x20f68cd4 --- kthread start Tracing command nfsd pid 559 tid 100143 td 0x23671500 sched_switch(23671500,104) at sched_switch+0x4f3/frame 0x20f6db18 mi_switch(104) at mi_switch+0x1c4/frame 0x20f6db54 sleepq_switch(2d40,0,23671500,1da0c14,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20f6db74 sleepq_catch_signals(20f6dbe0,f76998,99365d0,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20f6dba0 sleepq_wait_sig(99365d0,0) at sleepq_wait_sig+0xe/frame 0x20f6dba8 _cv_wait_sig(99365d0,186510c0) at _cv_wait_sig+0x138/frame 0x20f6dbe0 svc_run_internal(20f6dcd4,f98ac9,186510c0,20f6dce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20f6dcac svc_thread_start(186510c0,20f6dce8) at svc_thread_start+0xd/frame 0x20f6dcb4 fork_exit(1253d10,186510c0,20f6dce8,3b,3b,...) at fork_exit+0x69/frame 0x20f6dcd4 fork_trampoline() at 0xffc033de/frame 0x20f6dcd4 --- kthread start Tracing command nfsd pid 559 tid 100144 td 0x23671180 sched_switch(23671180,104) at sched_switch+0x4f3/frame 0x20f72b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20f72b54 sleepq_switch(2c00,0,23671180,1da0ad4,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20f72b74 sleepq_catch_signals(20f72be0,f76998,188d60d0,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20f72ba0 sleepq_wait_sig(188d60d0,0) at sleepq_wait_sig+0xe/frame 0x20f72ba8 _cv_wait_sig(188d60d0,186510c0) at _cv_wait_sig+0x138/frame 0x20f72be0 svc_run_internal(20f72cd4,f98ac9,186510c0,20f72ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20f72cac svc_thread_start(186510c0,20f72ce8) at svc_thread_start+0xd/frame 0x20f72cb4 fork_exit(1253d10,186510c0,20f72ce8,3b,3b,...) at fork_exit+0x69/frame 0x20f72cd4 fork_trampoline() at 0xffc033de/frame 0x20f72cd4 --- kthread start Tracing command nfsd pid 559 tid 100145 td 0x23670e00 sched_switch(23670e00,104) at sched_switch+0x4f3/frame 0x20f77b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20f77b54 sleepq_switch(d40,0,23670e00,1d9ec14,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20f77b74 sleepq_catch_signals(20f77be0,f76998,9936550,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20f77ba0 sleepq_wait_sig(9936550,0) at sleepq_wait_sig+0xe/frame 0x20f77ba8 _cv_wait_sig(9936550,186510c0) at _cv_wait_sig+0x138/frame 0x20f77be0 svc_run_internal(20f77cd4,f98ac9,186510c0,20f77ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20f77cac svc_thread_start(186510c0,20f77ce8) at svc_thread_start+0xd/frame 0x20f77cb4 fork_exit(1253d10,186510c0,20f77ce8,3b,3b,...) at fork_exit+0x69/frame 0x20f77cd4 fork_trampoline() at 0xffc033de/frame 0x20f77cd4 --- kthread start Tracing command nfsd pid 559 tid 100146 td 0x23670a80 sched_switch(23670a80,104) at sched_switch+0x4f3/frame 0x20f7cb18 mi_switch(104) at mi_switch+0x1c4/frame 0x20f7cb54 sleepq_switch(2d00,0,23670a80,1da0bd4,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20f7cb74 sleepq_catch_signals(20f7cbe0,f76998,99364d0,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20f7cba0 sleepq_wait_sig(99364d0,0) at sleepq_wait_sig+0xe/frame 0x20f7cba8 _cv_wait_sig(99364d0,186510c0) at _cv_wait_sig+0x138/frame 0x20f7cbe0 svc_run_internal(20f7ccd4,f98ac9,186510c0,20f7cce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20f7ccac svc_thread_start(186510c0,20f7cce8) at svc_thread_start+0xd/frame 0x20f7ccb4 fork_exit(1253d10,186510c0,20f7cce8,3b,3b,...) at fork_exit+0x69/frame 0x20f7ccd4 fork_trampoline() at 0xffc033de/frame 0x20f7ccd4 --- kthread start Tracing command nfsd pid 559 tid 100147 td 0x23670700 sched_switch(23670700,104) at sched_switch+0x4f3/frame 0x20f81b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20f81b54 sleepq_switch(d00,0,23670700,1d9ebd4,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20f81b74 sleepq_catch_signals(20f81be0,f76998,9936450,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20f81ba0 sleepq_wait_sig(9936450,0) at sleepq_wait_sig+0xe/frame 0x20f81ba8 _cv_wait_sig(9936450,186510c0) at _cv_wait_sig+0x138/frame 0x20f81be0 svc_run_internal(20f81cd4,f98ac9,186510c0,20f81ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20f81cac svc_thread_start(186510c0,20f81ce8) at svc_thread_start+0xd/frame 0x20f81cb4 fork_exit(1253d10,186510c0,20f81ce8,3b,3b,...) at fork_exit+0x69/frame 0x20f81cd4 fork_trampoline() at 0xffc033de/frame 0x20f81cd4 --- kthread start Tracing command nfsd pid 559 tid 100148 td 0x23670380 sched_switch(23670380,104) at sched_switch+0x4f3/frame 0x20f86b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20f86b54 sleepq_switch(c00,0,23670380,1d9ead4,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20f86b74 sleepq_catch_signals(20f86be0,f76998,188d6050,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20f86ba0 sleepq_wait_sig(188d6050,0) at sleepq_wait_sig+0xe/frame 0x20f86ba8 _cv_wait_sig(188d6050,186510c0) at _cv_wait_sig+0x138/frame 0x20f86be0 svc_run_internal(20f86cd4,f98ac9,186510c0,20f86ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20f86cac svc_thread_start(186510c0,20f86ce8) at svc_thread_start+0xd/frame 0x20f86cb4 fork_exit(1253d10,186510c0,20f86ce8,3b,3b,...) at fork_exit+0x69/frame 0x20f86cd4 fork_trampoline() at 0xffc033de/frame 0x20f86cd4 --- kthread start Tracing command nfsd pid 559 tid 100149 td 0x23670000 sched_switch(23670000,104) at sched_switch+0x4f3/frame 0x20f8bb18 mi_switch(104) at mi_switch+0x1c4/frame 0x20f8bb54 sleepq_switch(2cc0,0,23670000,1da0b94,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20f8bb74 sleepq_catch_signals(20f8bbe0,f76998,99363d0,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20f8bba0 sleepq_wait_sig(99363d0,0) at sleepq_wait_sig+0xe/frame 0x20f8bba8 _cv_wait_sig(99363d0,186510c0) at _cv_wait_sig+0x138/frame 0x20f8bbe0 svc_run_internal(20f8bcd4,f98ac9,186510c0,20f8bce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20f8bcac svc_thread_start(186510c0,20f8bce8) at svc_thread_start+0xd/frame 0x20f8bcb4 fork_exit(1253d10,186510c0,20f8bce8,3b,3b,...) at fork_exit+0x69/frame 0x20f8bcd4 fork_trampoline() at 0xffc033de/frame 0x20f8bcd4 --- kthread start Tracing command nfsd pid 559 tid 100150 td 0x2381e700 sched_switch(2381e700,104) at sched_switch+0x4f3/frame 0x20f90b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20f90b54 sleepq_switch(3c0,0,2381e700,1d9e294,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20f90b74 sleepq_catch_signals(20f90be0,f76998,188d5f50,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20f90ba0 sleepq_wait_sig(188d5f50,0) at sleepq_wait_sig+0xe/frame 0x20f90ba8 _cv_wait_sig(188d5f50,186510c0) at _cv_wait_sig+0x138/frame 0x20f90be0 svc_run_internal(20f90cd4,f98ac9,186510c0,20f90ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20f90cac svc_thread_start(186510c0,20f90ce8) at svc_thread_start+0xd/frame 0x20f90cb4 fork_exit(1253d10,186510c0,20f90ce8,3b,3b,...) at fork_exit+0x69/frame 0x20f90cd4 fork_trampoline() at 0xffc033de/frame 0x20f90cd4 --- kthread start Tracing command nfsd pid 559 tid 100151 td 0x2381e380 sched_switch(2381e380,104) at sched_switch+0x4f3/frame 0x20f95b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20f95b54 sleepq_switch(cc0,0,2381e380,1d9eb94,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20f95b74 sleepq_catch_signals(20f95be0,f76998,9936350,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20f95ba0 sleepq_wait_sig(9936350,0) at sleepq_wait_sig+0xe/frame 0x20f95ba8 _cv_wait_sig(9936350,186510c0) at _cv_wait_sig+0x138/frame 0x20f95be0 svc_run_internal(20f95cd4,f98ac9,186510c0,20f95ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20f95cac svc_thread_start(186510c0,20f95ce8) at svc_thread_start+0xd/frame 0x20f95cb4 fork_exit(1253d10,186510c0,20f95ce8,3b,3b,...) at fork_exit+0x69/frame 0x20f95cd4 fork_trampoline() at 0xffc033de/frame 0x20f95cd4 --- kthread start Tracing command nfsd pid 559 tid 100152 td 0x2381e000 sched_switch(2381e000,104) at sched_switch+0x4f3/frame 0x20f9ab18 mi_switch(104) at mi_switch+0x1c4/frame 0x20f9ab54 sleepq_switch(2c80,0,2381e000,1da0b54,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20f9ab74 sleepq_catch_signals(20f9abe0,f76998,99362d0,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20f9aba0 sleepq_wait_sig(99362d0,0) at sleepq_wait_sig+0xe/frame 0x20f9aba8 _cv_wait_sig(99362d0,186510c0) at _cv_wait_sig+0x138/frame 0x20f9abe0 svc_run_internal(20f9acd4,f98ac9,186510c0,20f9ace8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20f9acac svc_thread_start(186510c0,20f9ace8) at svc_thread_start+0xd/frame 0x20f9acb4 fork_exit(1253d10,186510c0,20f9ace8,3b,3b,...) at fork_exit+0x69/frame 0x20f9acd4 fork_trampoline() at 0xffc033de/frame 0x20f9acd4 --- kthread start Tracing command nfsd pid 559 tid 100153 td 0x2381dc00 sched_switch(2381dc00,104) at sched_switch+0x4f3/frame 0x20f9fb18 mi_switch(104) at mi_switch+0x1c4/frame 0x20f9fb54 sleepq_switch(c80,0,2381dc00,1d9eb54,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20f9fb74 sleepq_catch_signals(20f9fbe0,f76998,9936250,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20f9fba0 sleepq_wait_sig(9936250,0) at sleepq_wait_sig+0xe/frame 0x20f9fba8 _cv_wait_sig(9936250,186510c0) at _cv_wait_sig+0x138/frame 0x20f9fbe0 svc_run_internal(20f9fcd4,f98ac9,186510c0,20f9fce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20f9fcac svc_thread_start(186510c0,20f9fce8) at svc_thread_start+0xd/frame 0x20f9fcb4 fork_exit(1253d10,186510c0,20f9fce8,3b,3b,...) at fork_exit+0x69/frame 0x20f9fcd4 fork_trampoline() at 0xffc033de/frame 0x20f9fcd4 --- kthread start Tracing command nfsd pid 559 tid 100154 td 0x2381d880 sched_switch(2381d880,104) at sched_switch+0x4f3/frame 0x20fa4b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20fa4b54 sleepq_switch(2380,0,2381d880,1da0254,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20fa4b74 sleepq_catch_signals(20fa4be0,f76998,188d5ed0,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20fa4ba0 sleepq_wait_sig(188d5ed0,0) at sleepq_wait_sig+0xe/frame 0x20fa4ba8 _cv_wait_sig(188d5ed0,186510c0) at _cv_wait_sig+0x138/frame 0x20fa4be0 svc_run_internal(20fa4cd4,f98ac9,186510c0,20fa4ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20fa4cac svc_thread_start(186510c0,20fa4ce8) at svc_thread_start+0xd/frame 0x20fa4cb4 fork_exit(1253d10,186510c0,20fa4ce8,3b,3b,...) at fork_exit+0x69/frame 0x20fa4cd4 fork_trampoline() at 0xffc033de/frame 0x20fa4cd4 --- kthread start Tracing command nfsd pid 559 tid 100155 td 0x2381d500 sched_switch(2381d500,104) at sched_switch+0x4f3/frame 0x20fa9b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20fa9b54 sleepq_switch(2c40,0,2381d500,1da0b14,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20fa9b74 sleepq_catch_signals(20fa9be0,f76998,99361d0,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20fa9ba0 sleepq_wait_sig(99361d0,0) at sleepq_wait_sig+0xe/frame 0x20fa9ba8 _cv_wait_sig(99361d0,186510c0) at _cv_wait_sig+0x138/frame 0x20fa9be0 svc_run_internal(20fa9cd4,f98ac9,186510c0,20fa9ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20fa9cac svc_thread_start(186510c0,20fa9ce8) at svc_thread_start+0xd/frame 0x20fa9cb4 fork_exit(1253d10,186510c0,20fa9ce8,3b,3b,...) at fork_exit+0x69/frame 0x20fa9cd4 fork_trampoline() at 0xffc033de/frame 0x20fa9cd4 --- kthread start Tracing command nfsd pid 559 tid 100156 td 0x2381d180 sched_switch(2381d180,104) at sched_switch+0x4f3/frame 0x20faeb18 mi_switch(104) at mi_switch+0x1c4/frame 0x20faeb54 sleepq_switch(c40,0,2381d180,1d9eb14,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20faeb74 sleepq_catch_signals(20faebe0,f76998,9936150,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20faeba0 sleepq_wait_sig(9936150,0) at sleepq_wait_sig+0xe/frame 0x20faeba8 _cv_wait_sig(9936150,186510c0) at _cv_wait_sig+0x138/frame 0x20faebe0 svc_run_internal(20faecd4,f98ac9,186510c0,20faece8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20faecac svc_thread_start(186510c0,20faece8) at svc_thread_start+0xd/frame 0x20faecb4 fork_exit(1253d10,186510c0,20faece8,3b,3b,...) at fork_exit+0x69/frame 0x20faecd4 fork_trampoline() at 0xffc033de/frame 0x20faecd4 --- kthread start Tracing command nfsd pid 559 tid 100157 td 0x2381ce00 sched_switch(2381ce00,104) at sched_switch+0x4f3/frame 0x20fb3b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20fb3b54 sleepq_switch(380,0,2381ce00,1d9e254,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20fb3b74 sleepq_catch_signals(20fb3be0,f76998,188d5e50,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20fb3ba0 sleepq_wait_sig(188d5e50,0) at sleepq_wait_sig+0xe/frame 0x20fb3ba8 _cv_wait_sig(188d5e50,186510c0) at _cv_wait_sig+0x138/frame 0x20fb3be0 svc_run_internal(20fb3cd4,f98ac9,186510c0,20fb3ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20fb3cac svc_thread_start(186510c0,20fb3ce8) at svc_thread_start+0xd/frame 0x20fb3cb4 fork_exit(1253d10,186510c0,20fb3ce8,3b,3b,...) at fork_exit+0x69/frame 0x20fb3cd4 fork_trampoline() at 0xffc033de/frame 0x20fb3cd4 --- kthread start Tracing command nfsd pid 559 tid 100158 td 0x2381ca80 sched_switch(2381ca80,104) at sched_switch+0x4f3/frame 0x20fb8b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20fb8b54 sleepq_switch(2c00,0,2381ca80,1da0ad4,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20fb8b74 sleepq_catch_signals(20fb8be0,f76998,99360d0,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20fb8ba0 sleepq_wait_sig(99360d0,0) at sleepq_wait_sig+0xe/frame 0x20fb8ba8 _cv_wait_sig(99360d0,186510c0) at _cv_wait_sig+0x138/frame 0x20fb8be0 svc_run_internal(20fb8cd4,f98ac9,186510c0,20fb8ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20fb8cac svc_thread_start(186510c0,20fb8ce8) at svc_thread_start+0xd/frame 0x20fb8cb4 fork_exit(1253d10,186510c0,20fb8ce8,3b,3b,...) at fork_exit+0x69/frame 0x20fb8cd4 fork_trampoline() at 0xffc033de/frame 0x20fb8cd4 --- kthread start Tracing command nfsd pid 559 tid 100159 td 0x2381c700 sched_switch(2381c700,104) at sched_switch+0x4f3/frame 0x20fbdb18 mi_switch(104) at mi_switch+0x1c4/frame 0x20fbdb54 sleepq_switch(c00,0,2381c700,1d9ead4,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20fbdb74 sleepq_catch_signals(20fbdbe0,f76998,9936050,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20fbdba0 sleepq_wait_sig(9936050,0) at sleepq_wait_sig+0xe/frame 0x20fbdba8 _cv_wait_sig(9936050,186510c0) at _cv_wait_sig+0x138/frame 0x20fbdbe0 svc_run_internal(20fbdcd4,f98ac9,186510c0,20fbdce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20fbdcac svc_thread_start(186510c0,20fbdce8) at svc_thread_start+0xd/frame 0x20fbdcb4 fork_exit(1253d10,186510c0,20fbdce8,3b,3b,...) at fork_exit+0x69/frame 0x20fbdcd4 fork_trampoline() at 0xffc033de/frame 0x20fbdcd4 --- kthread start Tracing command nfsd pid 559 tid 100160 td 0x2381c380 sched_switch(2381c380,104) at sched_switch+0x4f3/frame 0x20fc2b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20fc2b54 sleepq_switch(3c0,0,2381c380,1d9e294,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20fc2b74 sleepq_catch_signals(20fc2be0,f76998,9935f50,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20fc2ba0 sleepq_wait_sig(9935f50,0) at sleepq_wait_sig+0xe/frame 0x20fc2ba8 _cv_wait_sig(9935f50,186510c0) at _cv_wait_sig+0x138/frame 0x20fc2be0 svc_run_internal(20fc2cd4,f98ac9,186510c0,20fc2ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20fc2cac svc_thread_start(186510c0,20fc2ce8) at svc_thread_start+0xd/frame 0x20fc2cb4 fork_exit(1253d10,186510c0,20fc2ce8,3b,3b,...) at fork_exit+0x69/frame 0x20fc2cd4 fork_trampoline() at 0xffc033de/frame 0x20fc2cd4 --- kthread start Tracing command nfsd pid 559 tid 100161 td 0x2381c000 sched_switch(2381c000,104) at sched_switch+0x4f3/frame 0x20fc7b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20fc7b54 sleepq_switch(2380,0,2381c000,1da0254,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20fc7b74 sleepq_catch_signals(20fc7be0,f76998,9935ed0,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20fc7ba0 sleepq_wait_sig(9935ed0,0) at sleepq_wait_sig+0xe/frame 0x20fc7ba8 _cv_wait_sig(9935ed0,186510c0) at _cv_wait_sig+0x138/frame 0x20fc7be0 svc_run_internal(20fc7cd4,f98ac9,186510c0,20fc7ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20fc7cac svc_thread_start(186510c0,20fc7ce8) at svc_thread_start+0xd/frame 0x20fc7cb4 fork_exit(1253d10,186510c0,20fc7ce8,3b,3b,...) at fork_exit+0x69/frame 0x20fc7cd4 fork_trampoline() at 0xffc033de/frame 0x20fc7cd4 --- kthread start Tracing command nfsd pid 559 tid 100162 td 0x236c7c00 sched_switch(236c7c00,104) at sched_switch+0x4f3/frame 0x20fccb18 mi_switch(104) at mi_switch+0x1c4/frame 0x20fccb54 sleepq_switch(2340,0,236c7c00,1da0214,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20fccb74 sleepq_catch_signals(20fccbe0,f76998,188d5dd0,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20fccba0 sleepq_wait_sig(188d5dd0,0) at sleepq_wait_sig+0xe/frame 0x20fccba8 _cv_wait_sig(188d5dd0,186510c0) at _cv_wait_sig+0x138/frame 0x20fccbe0 svc_run_internal(20fcccd4,f98ac9,186510c0,20fccce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20fcccac svc_thread_start(186510c0,20fccce8) at svc_thread_start+0xd/frame 0x20fcccb4 fork_exit(1253d10,186510c0,20fccce8,3b,3b,...) at fork_exit+0x69/frame 0x20fcccd4 fork_trampoline() at 0xffc033de/frame 0x20fcccd4 --- kthread start Tracing command nfsd pid 559 tid 100163 td 0x236c7880 sched_switch(236c7880,104) at sched_switch+0x4f3/frame 0x20fd1b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20fd1b54 sleepq_switch(340,0,236c7880,1d9e214,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20fd1b74 sleepq_catch_signals(20fd1be0,f76998,188d5d50,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20fd1ba0 sleepq_wait_sig(188d5d50,0) at sleepq_wait_sig+0xe/frame 0x20fd1ba8 _cv_wait_sig(188d5d50,186510c0) at _cv_wait_sig+0x138/frame 0x20fd1be0 svc_run_internal(20fd1cd4,f98ac9,186510c0,20fd1ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20fd1cac svc_thread_start(186510c0,20fd1ce8) at svc_thread_start+0xd/frame 0x20fd1cb4 fork_exit(1253d10,186510c0,20fd1ce8,3b,3b,...) at fork_exit+0x69/frame 0x20fd1cd4 fork_trampoline() at 0xffc033de/frame 0x20fd1cd4 --- kthread start Tracing command nfsd pid 559 tid 100164 td 0x236c7500 sched_switch(236c7500,104) at sched_switch+0x4f3/frame 0x20fd6b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20fd6b54 sleepq_switch(380,0,236c7500,1d9e254,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20fd6b74 sleepq_catch_signals(20fd6be0,f76998,9935e50,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20fd6ba0 sleepq_wait_sig(9935e50,0) at sleepq_wait_sig+0xe/frame 0x20fd6ba8 _cv_wait_sig(9935e50,186510c0) at _cv_wait_sig+0x138/frame 0x20fd6be0 svc_run_internal(20fd6cd4,f98ac9,186510c0,20fd6ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20fd6cac svc_thread_start(186510c0,20fd6ce8) at svc_thread_start+0xd/frame 0x20fd6cb4 fork_exit(1253d10,186510c0,20fd6ce8,3b,3b,...) at fork_exit+0x69/frame 0x20fd6cd4 fork_trampoline() at 0xffc033de/frame 0x20fd6cd4 --- kthread start Tracing command nfsd pid 559 tid 100165 td 0x236c7180 sched_switch(236c7180,104) at sched_switch+0x4f3/frame 0x20fdbb18 mi_switch(104) at mi_switch+0x1c4/frame 0x20fdbb54 sleepq_switch(2300,0,236c7180,1da01d4,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20fdbb74 sleepq_catch_signals(20fdbbe0,f76998,188d5cd0,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20fdbba0 sleepq_wait_sig(188d5cd0,0) at sleepq_wait_sig+0xe/frame 0x20fdbba8 _cv_wait_sig(188d5cd0,186510c0) at _cv_wait_sig+0x138/frame 0x20fdbbe0 svc_run_internal(20fdbcd4,f98ac9,186510c0,20fdbce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20fdbcac svc_thread_start(186510c0,20fdbce8) at svc_thread_start+0xd/frame 0x20fdbcb4 fork_exit(1253d10,186510c0,20fdbce8,3b,3b,...) at fork_exit+0x69/frame 0x20fdbcd4 fork_trampoline() at 0xffc033de/frame 0x20fdbcd4 --- kthread start Tracing command nfsd pid 559 tid 100166 td 0x23822000 sched_switch(23822000,104) at sched_switch+0x4f3/frame 0x20fe0b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20fe0b54 sleepq_switch(2340,0,23822000,1da0214,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20fe0b74 sleepq_catch_signals(20fe0be0,f76998,9935dd0,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20fe0ba0 sleepq_wait_sig(9935dd0,0) at sleepq_wait_sig+0xe/frame 0x20fe0ba8 _cv_wait_sig(9935dd0,186510c0) at _cv_wait_sig+0x138/frame 0x20fe0be0 svc_run_internal(20fe0cd4,f98ac9,186510c0,20fe0ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20fe0cac svc_thread_start(186510c0,20fe0ce8) at svc_thread_start+0xd/frame 0x20fe0cb4 fork_exit(1253d10,186510c0,20fe0ce8,3b,3b,...) at fork_exit+0x69/frame 0x20fe0cd4 fork_trampoline() at 0xffc033de/frame 0x20fe0cd4 --- kthread start Tracing command nfsd pid 559 tid 100167 td 0x23821c00 sched_switch(23821c00,104) at sched_switch+0x4f3/frame 0x20fe5b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20fe5b54 sleepq_switch(340,0,23821c00,1d9e214,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20fe5b74 sleepq_catch_signals(20fe5be0,f76998,9935d50,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20fe5ba0 sleepq_wait_sig(9935d50,0) at sleepq_wait_sig+0xe/frame 0x20fe5ba8 _cv_wait_sig(9935d50,186510c0) at _cv_wait_sig+0x138/frame 0x20fe5be0 svc_run_internal(20fe5cd4,f98ac9,186510c0,20fe5ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20fe5cac svc_thread_start(186510c0,20fe5ce8) at svc_thread_start+0xd/frame 0x20fe5cb4 fork_exit(1253d10,186510c0,20fe5ce8,3b,3b,...) at fork_exit+0x69/frame 0x20fe5cd4 fork_trampoline() at 0xffc033de/frame 0x20fe5cd4 --- kthread start Tracing command nfsd pid 559 tid 100168 td 0x23821880 sched_switch(23821880,104) at sched_switch+0x4f3/frame 0x20feab18 mi_switch(104) at mi_switch+0x1c4/frame 0x20feab54 sleepq_switch(2300,0,23821880,1da01d4,187c0aa0,...) at sleepq_switch+0xf7/frame 0x20feab74 sleepq_catch_signals(20feabe0,f76998,9935cd0,0,14abb67,...) at sleepq_catch_signals+0x368/frame 0x20feaba0 sleepq_wait_sig(9935cd0,0) at sleepq_wait_sig+0xe/frame 0x20feaba8 _cv_wait_sig(9935cd0,186510c0) at _cv_wait_sig+0x138/frame 0x20feabe0 svc_run_internal(20feacd4,f98ac9,186510c0,20feace8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20feacac svc_thread_start(186510c0,20feace8) at svc_thread_start+0xd/frame 0x20feacb4 fork_exit(1253d10,186510c0,20feace8,3b,3b,...) at fork_exit+0x69/frame 0x20feacd4 fork_trampoline() at 0xffc033de/frame 0x20feacd4 --- kthread start Tracing command nfsd pid 556 tid 100136 td 0x187e2180 sched_switch(187e2180,104) at sched_switch+0x4f3/frame 0x20f42970 mi_switch(104) at mi_switch+0x1c4/frame 0x20f429ac sleepq_switch(1c00,0,187e2180,1d9fad4,18889000,...) at sleepq_switch+0xf7/frame 0x20f429cc sleepq_catch_signals(20f42a38,f76998,183ad4a4,0,1441215,...) at sleepq_catch_signals+0x368/frame 0x20f429f8 sleepq_wait_sig(183ad4a4,0) at sleepq_wait_sig+0xe/frame 0x20f42a00 _cv_wait_sig(183ad4a4,183ad490) at _cv_wait_sig+0x138/frame 0x20f42a38 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x20f42a54 kern_select(187e2180,7,ffbfec58,0,0,0,20) at kern_select+0x8ed/frame 0x20f42bd4 sys_select(187e2180,187e240c) at sys_select+0x41/frame 0x20f42c08 syscall(20f42ce8,3b,3b,3b,ffbfecd8,...) at syscall+0x306/frame 0x20f42cdc Xint0x80_syscall() at 0xffc033c9/frame 0x20f42cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc015b0, esp = 0xffc0dfe8, ebp = 0xffbfe760 --- Tracing command mountd pid 550 tid 100111 td 0x991ec00 sched_switch(991ec00,104) at sched_switch+0x4f3/frame 0x202c5970 mi_switch(104) at mi_switch+0x1c4/frame 0x202c59ac sleepq_switch(fc0,0,991ec00,1d9ee94,9dee6a8,...) at sleepq_switch+0xf7/frame 0x202c59cc sleepq_catch_signals(202c5a38,f76998,9df5b64,0,1441215,...) at sleepq_catch_signals+0x368/frame 0x202c59f8 sleepq_wait_sig(9df5b64,0) at sleepq_wait_sig+0xe/frame 0x202c5a00 _cv_wait_sig(9df5b64,9df5b50) at _cv_wait_sig+0x138/frame 0x202c5a38 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x202c5a54 kern_select(991ec00,a,ffbfe8c8,0,0,0,20) at kern_select+0x8ed/frame 0x202c5bd4 sys_select(991ec00,991ee8c) at sys_select+0x41/frame 0x202c5c08 syscall(202c5ce8,3b,3b,3b,ffbfe948,...) at syscall+0x306/frame 0x202c5cdc Xint0x80_syscall() at 0xffc033c9/frame 0x202c5cdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f5, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command nfsuserd pid 543 tid 100135 td 0x187e2500 sched_switch(187e2500,104) at sched_switch+0x4f3/frame 0x20f39930 mi_switch(104) at mi_switch+0x1c4/frame 0x20f3996c sleepq_switch(3ec0,0,1dcdba0,1da1d94,18889354,...) at sleepq_switch+0xf7/frame 0x20f3998c sleepq_catch_signals(e2,187e2500,1abc268,187e2500,20f399ec,...) at sleepq_catch_signals+0x368/frame 0x20f399b8 sleepq_timedwait_sig(9df5fa4,0) at sleepq_timedwait_sig+0x17/frame 0x20f399e8 _cv_timedwait_sig_sbt(9df5fa4,9df5f90,75810b32,ddca,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x20f39a24 seltdwait(ddca,e0000000,1) at seltdwait+0x5e/frame 0x20f39a54 kern_select(187e2500,4,ffbfdbf0,0,0,20f39bf8,20) at kern_select+0x8ed/frame 0x20f39bd4 sys_select(187e2500,187e278c) at sys_select+0x41/frame 0x20f39c08 syscall(20f39ce8,3b,3b,3b,ffbfdbf0,...) at syscall+0x306/frame 0x20f39cdc Xint0x80_syscall() at 0xffc033c9/frame 0x20f39cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc015b0, esp = 0xffc0dfe8, ebp = 0xffbfe760 --- Tracing command nfsuserd pid 542 tid 100120 td 0x187e2c00 sched_switch(187e2c00,104) at sched_switch+0x4f3/frame 0x20332930 mi_switch(104) at mi_switch+0x1c4/frame 0x2033296c sleepq_switch(2fc0,0,1dcda40,1da0e94,188899fc,...) at sleepq_switch+0xf7/frame 0x2033298c sleepq_catch_signals(e2,187e2c00,1abc268,187e2c00,203329ec,...) at sleepq_catch_signals+0x368/frame 0x203329b8 sleepq_timedwait_sig(9df5be4,0) at sleepq_timedwait_sig+0x17/frame 0x203329e8 _cv_timedwait_sig_sbt(9df5be4,9df5bd0,6ee7248,ddca,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x20332a24 seltdwait(ddca,e0000000,1) at seltdwait+0x5e/frame 0x20332a54 kern_select(187e2c00,4,ffbfdbf0,0,0,20332bf8,20) at kern_select+0x8ed/frame 0x20332bd4 sys_select(187e2c00,187e2e8c) at sys_select+0x41/frame 0x20332c08 syscall(20332ce8,3b,3b,3b,ffbfdbf0,...) at syscall+0x306/frame 0x20332cdc Xint0x80_syscall() at 0xffc033c9/frame 0x20332cdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe660 --- Tracing command nfsuserd pid 541 tid 100127 td 0x181dc000 sched_switch(181dc000,104) at sched_switch+0x4f3/frame 0x20e91930 mi_switch(104) at mi_switch+0x1c4/frame 0x20e9196c sleepq_switch(3580,0,1dce120,1da1454,18189354,...) at sleepq_switch+0xf7/frame 0x20e9198c sleepq_catch_signals(e2,181dc000,1abc268,181dc000,20e919ec,...) at sleepq_catch_signals+0x368/frame 0x20e919b8 sleepq_timedwait_sig(187df224,0) at sleepq_timedwait_sig+0x17/frame 0x20e919e8 _cv_timedwait_sig_sbt(187df224,187df210,bb047f81,ddc5,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x20e91a24 seltdwait(ddc5,e0000000,1) at seltdwait+0x5e/frame 0x20e91a54 kern_select(181dc000,4,ffbfdbf0,0,0,20e91bf8,20) at kern_select+0x8ed/frame 0x20e91bd4 sys_select(181dc000,181dc28c) at sys_select+0x41/frame 0x20e91c08 syscall(20e91ce8,3b,3b,3b,ffbfdbf0,...) at syscall+0x306/frame 0x20e91cdc Xint0x80_syscall() at 0xffc033c9/frame 0x20e91cdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe660 --- Tracing command nfsuserd pid 540 tid 100131 td 0x23672e00 sched_switch(23672e00,104) at sched_switch+0x4f3/frame 0x20eb5930 mi_switch(104) at mi_switch+0x1c4/frame 0x20eb596c sleepq_switch(1580,0,1dcde60,1d9f454,181843f8,...) at sleepq_switch+0xf7/frame 0x20eb598c sleepq_catch_signals(e2,23672e00,1abc268,23672e00,20eb59ec,...) at sleepq_catch_signals+0x368/frame 0x20eb59b8 sleepq_timedwait_sig(187df2a4,0) at sleepq_timedwait_sig+0x17/frame 0x20eb59e8 _cv_timedwait_sig_sbt(187df2a4,187df290,5be76ca3,ddc9,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x20eb5a24 seltdwait(ddc9,e0000000,1) at seltdwait+0x5e/frame 0x20eb5a54 kern_select(23672e00,4,ffbfdbf0,0,0,20eb5bf8,20) at kern_select+0x8ed/frame 0x20eb5bd4 sys_select(23672e00,2367308c) at sys_select+0x41/frame 0x20eb5c08 syscall(20eb5ce8,3b,3b,3b,ffbfdbf0,...) at syscall+0x306/frame 0x20eb5cdc Xint0x80_syscall() at 0xffc033c9/frame 0x20eb5cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0xffc01530, esp = 0xffc0bfe8, ebp = 0xffbfe374 --- Tracing command nfsuserd pid 539 tid 100121 td 0x187e2880 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa5ebc,1aa5ed4,289ed478,102bc8c,1aa5ed4,...) at db_trace_self_wrapper+0x2a/frame 0x289ed430 kdb_backtrace(289ed558,1416041,3a,a07e50,3a,...) at kdb_backtrace+0x2e/frame 0x289ed490 kdb_reenter(3a,a07e50,3a,a07e50,2cb86000,...) at kdb_reenter+0x29/frame 0x289ed498 trap(289ed564,8,28,28,ffffffff,...) at trap+0xd1/frame 0x289ed558 calltrap() at 0xffc0316f/frame 0x289ed558 --- trap 0xc, eip = 0x13f347a, esp = 0x289ed5a4, ebp = 0x289ed5b0 --- db_trace_thread(187e2880,ffffffff) at db_trace_thread+0x2a/frame 0x289ed5b0 _db_stack_trace_all(1017c05,0,ffffffff,289ed63c) at _db_stack_trace_all+0x93/frame 0x289ed610 db_command(1b2ff32) at db_command+0x26a/frame 0x289ed6c4 db_script_exec(bd452f,0,289ed76c,289ed750) at db_script_exec+0x1a6/frame 0x289ed724 db_command(289ed828,a07d16,0,0,27069a80,...) at db_command+0x26a/frame 0x289ed7d8 db_command_loop(0,0,27069a80,289ed7e0,289ed828,...) at db_command_loop+0x3c/frame 0x289ed7e0 db_trap(3,0) at db_trap+0xf6/frame 0x289ed828 kdb_trap(3,0,289ed934) at kdb_trap+0x119/frame 0x289ed85c trap(289ed934,8,28,28,100,...) at trap+0x38a/frame 0x289ed928 calltrap() at 0xffc0316f/frame 0x289ed928 --- trap 0x3, eip = 0x1017c05, esp = 0x289ed974, ebp = 0x289ed978 --- kdb_enter(1521cf1,1521cf1) at kdb_enter+0x35/frame 0x289ed978 vpanic(1461fb6,289ed9c4,289ed9c4,289eda00,1416c07,...) at vpanic+0x137/frame 0x289ed9a4 panic(1461fb6,14f09c1,14adb26,0,c,...) at panic+0x14/frame 0x289ed9b8 trap_fatal(2cb86000,1dbc480,2b31380,9890a6c,0,...) at trap_fatal+0x357/frame 0x289eda00 trap_pfault(1833f004,0,0) at trap_pfault+0x72/frame 0x289eda3c trap(289edb10,8,28,28,3ff,...) at trap+0x3a8/frame 0x289edb04 calltrap() at 0xffc0316f/frame 0x289edb04 --- trap 0xc, eip = 0x1046228, esp = 0x289edb50, ebp = 0x289edb70 --- sem_remove(289edbac,0,18342c10,1833d000,0,...) at sem_remove+0x208/frame 0x289edb70 kern_semctl(27069a80,10000,0,0,289edbc8,289edbd4) at kern_semctl+0x892/frame 0x289edba8 sys___semctl(27069a80,27069d0c) at sys___semctl+0xa0/frame 0x289edc08 syscall(289edce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x289edcdc Xint0x80_syscall() at 0xffc033c9/frame 0x289edcdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc015b0, esp = 0xffc0dfe8, ebp = 0xffbfe760 --- Tracing command watchdogd pid 496 tid 100125 td 0x20ad6500 sched_switch(20ad6500,104) at sched_switch+0x4f3/frame 0x203b3a68 mi_switch(104) at mi_switch+0x1c4/frame 0x203b3aa4 sleepq_switch(3fc0,6c,3,1da1e94,205d1d50,...) at sleepq_switch+0xf7/frame 0x203b3ac4 sleepq_catch_signals(0,1526a9e,20ad6500,100,1d901fe,...) at sleepq_catch_signals+0x368/frame 0x203b3af0 sleepq_timedwait_sig(1d901fe,6c,0,1441215,32d,...) at sleepq_timedwait_sig+0x17/frame 0x203b3b1c _sleep(1d901fe,0,16c,15397ea,deb7fe66,ddd3,c0000000,3,200) at _sleep+0x21c/frame 0x203b3b5c kern_clock_nanosleep(20ad6500,0,0,203b3be8,203b3be0) at kern_clock_nanosleep+0x1b4/frame 0x203b3bc4 sys_nanosleep(20ad6500,20ad678c) at sys_nanosleep+0x55/frame 0x203b3c08 syscall(203b3ce8,3b,3b,3b,3c,...) at syscall+0x306/frame 0x203b3cdc Xint0x80_syscall() at 0xffc033c9/frame 0x203b3cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0xffc01530, esp = 0xffc0bfe8, ebp = 0xffbfe374 --- Tracing command rpcbind pid 411 tid 100129 td 0x23673500 sched_switch(23673500,104) at sched_switch+0x4f3/frame 0x20e9b958 mi_switch(104) at mi_switch+0x1c4/frame 0x20e9b994 sleepq_switch(f40,0,1dcdc50,1d9ee14,18184aa0,...) at sleepq_switch+0xf7/frame 0x20e9b9b4 sleepq_catch_signals(e2,23673500,1abc268,23673500,20e9ba14,...) at sleepq_catch_signals+0x368/frame 0x20e9b9e0 sleepq_timedwait_sig(183ad9e4,0) at sleepq_timedwait_sig+0x17/frame 0x20e9ba10 _cv_timedwait_sig_sbt(183ad9e4,183ad9d0,76fbeb78,ddcc,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x20e9ba4c seltdwait(ddcc,e0000000,1) at seltdwait+0x5e/frame 0x20e9ba7c kern_poll(23673500,ffbfcdb8,8,20e9bbf0,0) at kern_poll+0x476/frame 0x20e9bbd4 sys_poll(23673500,2367378c) at sys_poll+0x50/frame 0x20e9bc08 syscall(20e9bce8,3b,3b,3b,ffbfcd38,...) at syscall+0x306/frame 0x20e9bcdc Xint0x80_syscall() at 0xffc033c9/frame 0x20e9bcdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc015b0, esp = 0xffc0dfe8, ebp = 0xffbfe760 --- Tracing command syslogd pid 402 tid 100128 td 0x181b3c00 sched_switch(181b3c00,104) at sched_switch+0x4f3/frame 0x20e96970 mi_switch(104) at mi_switch+0x1c4/frame 0x20e969ac sleepq_switch(2f00,0,181b3c00,1da0dd4,18189000,...) at sleepq_switch+0xf7/frame 0x20e969cc sleepq_catch_signals(20e96a38,f76998,183ad864,0,1441215,...) at sleepq_catch_signals+0x368/frame 0x20e969f8 sleepq_wait_sig(183ad864,0) at sleepq_wait_sig+0xe/frame 0x20e96a00 _cv_wait_sig(183ad864,183ad850) at _cv_wait_sig+0x138/frame 0x20e96a38 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x20e96a54 kern_select(181b3c00,a,20671000,0,0,0,20) at kern_select+0x8ed/frame 0x20e96bd4 sys_select(181b3c00,181b3e8c) at sys_select+0x41/frame 0x20e96c08 syscall(20e96ce8,3b,3b,3b,20671000,...) at syscall+0x306/frame 0x20e96cdc Xint0x80_syscall() at 0xffc033c9/frame 0x20e96cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc015b0, esp = 0xffc0dfe8, ebp = 0xffbfe760 --- Tracing command devd pid 330 tid 100122 td 0x20ad6c00 sched_switch(20ad6c00,104) at sched_switch+0x4f3/frame 0x20348930 mi_switch(104) at mi_switch+0x1c4/frame 0x2034896c sleepq_switch(1240,0,1dcdd58,1d9f114,205d23f8,...) at sleepq_switch+0xf7/frame 0x2034898c sleepq_catch_signals(e2,20ad6c00,1abc268,20ad6c00,203489ec,...) at sleepq_catch_signals+0x368/frame 0x203489b8 sleepq_timedwait_sig(187deda4,0) at sleepq_timedwait_sig+0x17/frame 0x203489e8 _cv_timedwait_sig_sbt(187deda4,187ded90,6e14318f,dde1,c0000000,3,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x20348a24 seltdwait(dde1,c0000000,3) at seltdwait+0x5e/frame 0x20348a54 kern_select(20ad6c00,6,ffbfcd98,0,0,20348bf8,20) at kern_select+0x8ed/frame 0x20348bd4 sys_select(20ad6c00,20ad6e8c) at sys_select+0x41/frame 0x20348c08 syscall(20348ce8,3b,3b,3b,ffbfce18,...) at syscall+0x306/frame 0x20348cdc Xint0x80_syscall() at 0xffc033c9/frame 0x20348cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc015b0, esp = 0xffc0dfe8, ebp = 0xffbfe760 --- Tracing command vnlru pid 25 tid 100104 td 0x187cbe00 sched_switch(187cbe00,104) at sched_switch+0x4f3/frame 0x1bdf6b68 mi_switch(104) at mi_switch+0x1c4/frame 0x1bdf6ba8 sleepq_switch(187cbe00,0,187b39fc,1bdf6c2c,187cbe00,...) at sleepq_switch+0xf7/frame 0x1bdf6bc8 sleepq_timedwait(187b39fc,60,1bdf6c20,fb6445,14c6f8c,...) at sleepq_timedwait+0x3f/frame 0x1bdf6bec _sleep(187b39fc,1ad4740,260,154da66,fffffed8,0,0,0,100) at _sleep+0x22c/frame 0x1bdf6c2c vnlru_proc(0,1bdf6ce8) at vnlru_proc+0xb02/frame 0x1bdf6cb4 fork_exit(109b830,0,1bdf6ce8,0,0,...) at fork_exit+0x69/frame 0x1bdf6cd4 fork_trampoline() at 0xffc033de/frame 0x1bdf6cd4 --- kthread start Tracing command syncer pid 24 tid 100103 td 0x187cc180 sched_switch(187cc180,104) at sched_switch+0x4f3/frame 0x1bdf1b80 mi_switch(104) at mi_switch+0x1c4/frame 0x1bdf1bc0 sleepq_switch(1abc268,187cc180,1bdf1c0c,fb56fe,1dfd118,...) at sleepq_switch+0xf7/frame 0x1bdf1be0 sleepq_timedwait(1dfd128,0) at sleepq_timedwait+0x3f/frame 0x1bdf1c08 _cv_timedwait_sbt(1dfd128,1dfd108,fffffed8,0,0,0,100) at _cv_timedwait_sbt+0x157/frame 0x1bdf1c44 sched_sync(0,1bdf1ce8) at sched_sync+0x68c/frame 0x1bdf1cb4 fork_exit(109c7a0,0,1bdf1ce8,0,0,...) at fork_exit+0x69/frame 0x1bdf1cd4 fork_trampoline() at 0xffc033de/frame 0x1bdf1cd4 --- kthread start Tracing command bufdaemon pid 23 tid 100099 td 0x187cc500 sched_switch(187cc500,104) at sched_switch+0x4f3/frame 0xb7dcbb0 mi_switch(104) at mi_switch+0x1c4/frame 0xb7dcbf0 sleepq_switch(187cc500,0,1dfcaec,b7dcc74,187cc500,...) at sleepq_switch+0xf7/frame 0xb7dcc10 sleepq_timedwait(1dfcaec,54,187b414c,187b415c,1441649,...) at sleepq_timedwait+0x3f/frame 0xb7dcc34 _sleep(1dfcaec,1ac8d40,54,1470c47,1999997c,0,0,0,100) at _sleep+0x22c/frame 0xb7dcc74 buf_daemon(0,b7dcce8) at buf_daemon+0xf5/frame 0xb7dccb4 fork_exit(1079ff0,0,b7dcce8,0,0,...) at fork_exit+0x69/frame 0xb7dccd4 fork_trampoline() at 0xffc033de/frame 0xb7dccd4 --- kthread start Tracing command bufdaemon pid 23 tid 100102 td 0x181dc700 sched_switch(181dc700,104) at sched_switch+0x4f3/frame 0x1d3ffba0 mi_switch(104) at mi_switch+0x1c4/frame 0x1d3ffbe0 sleepq_switch(1526a9e,181dc700,0,1aca040,1d3ffc68,...) at sleepq_switch+0xf7/frame 0x1d3ffc00 sleepq_timedwait(1aca040,5c,187b414c,187b415c,1441649,...) at sleepq_timedwait+0x3f/frame 0x1d3ffc28 _sleep(1aca040,1ac9f80,25c,14745d3,fffffed8,0,0,0,100) at _sleep+0x22c/frame 0x1d3ffc68 bufspace_daemon(1ac8e40,1d3ffce8) at bufspace_daemon+0x2db/frame 0x1d3ffcb4 fork_exit(107a300,1ac8e40,1d3ffce8,0,0,...) at fork_exit+0x69/frame 0x1d3ffcd4 fork_trampoline() at 0xffc033de/frame 0x1d3ffcd4 --- kthread start Tracing command bufdaemon pid 23 tid 100116 td 0x187cb700 sched_switch(187cb700,104) at sched_switch+0x4f3/frame 0x202f6bb0 mi_switch(104) at mi_switch+0x1c4/frame 0x202f6bf0 sleepq_switch(1526a9e,187cb700,0,18da4884,202f6c78,...) at sleepq_switch+0xf7/frame 0x202f6c10 sleepq_timedwait(18da4884,54,202f6c6c,fd0030,14abd53,...) at sleepq_timedwait+0x3f/frame 0x202f6c38 _sleep(18da4884,18da4800,54,14a25ed,7fffff6c,0,0,0,100) at _sleep+0x22c/frame 0x202f6c78 softdep_flush(18dc9b00,202f6ce8) at softdep_flush+0x15a/frame 0x202f6cb4 fork_exit(128f4b0,18dc9b00,202f6ce8,0,0,...) at fork_exit+0x69/frame 0x202f6cd4 fork_trampoline() at 0xffc033de/frame 0x202f6cd4 --- kthread start Tracing command bufdaemon pid 23 tid 100117 td 0x187cb380 sched_switch(187cb380,104) at sched_switch+0x4f3/frame 0x202fbbb0 mi_switch(104) at mi_switch+0x1c4/frame 0x202fbbf0 sleepq_switch(1526a9e,187cb380,0,99d5e84,202fbc78,...) at sleepq_switch+0xf7/frame 0x202fbc10 sleepq_timedwait(99d5e84,54,202fbc6c,fd0030,14abd53,...) at sleepq_timedwait+0x3f/frame 0x202fbc38 _sleep(99d5e84,99d5e00,54,14a25ed,7fffff6c,0,0,0,100) at _sleep+0x22c/frame 0x202fbc78 softdep_flush(18dc9040,202fbce8) at softdep_flush+0x15a/frame 0x202fbcb4 fork_exit(128f4b0,18dc9040,202fbce8,0,0,...) at fork_exit+0x69/frame 0x202fbcd4 fork_trampoline() at 0xffc033de/frame 0x202fbcd4 --- kthread start Tracing command bufdaemon pid 23 tid 100118 td 0x187cb000 sched_switch(187cb000,104) at sched_switch+0x4f3/frame 0x20300bb0 mi_switch(104) at mi_switch+0x1c4/frame 0x20300bf0 sleepq_switch(1526a9e,187cb000,0,1819f084,20300c78,...) at sleepq_switch+0xf7/frame 0x20300c10 sleepq_timedwait(1819f084,54,20300c6c,fd0030,14abd53,...) at sleepq_timedwait+0x3f/frame 0x20300c38 _sleep(1819f084,1819f000,54,14a25ed,7fffff6c,0,0,0,100) at _sleep+0x22c/frame 0x20300c78 softdep_flush(18dc8580,20300ce8) at softdep_flush+0x15a/frame 0x20300cb4 fork_exit(128f4b0,18dc8580,20300ce8,0,0,...) at fork_exit+0x69/frame 0x20300cd4 fork_trampoline() at 0xffc033de/frame 0x20300cd4 --- kthread start Tracing command bufdaemon pid 23 tid 100119 td 0x181dfc00 sched_switch(181dfc00,104) at sched_switch+0x4f3/frame 0x20305bb0 mi_switch(104) at mi_switch+0x1c4/frame 0x20305bf0 sleepq_switch(1526a9e,181dfc00,0,18da4e84,20305c78,...) at sleepq_switch+0xf7/frame 0x20305c10 sleepq_timedwait(18da4e84,54,20305c6c,fd0030,14abd53,...) at sleepq_timedwait+0x3f/frame 0x20305c38 _sleep(18da4e84,18da4e00,54,14a25ed,7fffff6c,0,0,0,100) at _sleep+0x22c/frame 0x20305c78 softdep_flush(18dc7ac0,20305ce8) at softdep_flush+0x15a/frame 0x20305cb4 fork_exit(128f4b0,18dc7ac0,20305ce8,0,0,...) at fork_exit+0x69/frame 0x20305cd4 fork_trampoline() at 0xffc033de/frame 0x20305cd4 --- kthread start Tracing command bufdaemon pid 23 tid 101686 td 0x44bbb500 sched_switch(44bbb500,104) at sched_switch+0x4f3/frame 0x71ceabb0 mi_switch(104) at mi_switch+0x1c4/frame 0x71ceabf0 sleepq_switch(1526a9e,44bbb500,0,48009c84,71ceac78,...) at sleepq_switch+0xf7/frame 0x71ceac10 sleepq_timedwait(48009c84,54,71ceac6c,fd0030,14abd53,...) at sleepq_timedwait+0x3f/frame 0x71ceac38 _sleep(48009c84,48009c00,54,14a25ed,7fffff6c,0,0,0,100) at _sleep+0x22c/frame 0x71ceac78 softdep_flush(207ba580,71ceace8) at softdep_flush+0x15a/frame 0x71ceacb4 fork_exit(128f4b0,207ba580,71ceace8,0,0,...) at fork_exit+0x69/frame 0x71ceacd4 fork_trampoline() at 0xffc033de/frame 0x71ceacd4 --- kthread start Tracing command vmdaemon pid 22 tid 100098 td 0x187cc880 sched_switch(187cc880,104) at sched_switch+0x4f3/frame 0xb7d7ba0 mi_switch(104) at mi_switch+0x1c4/frame 0xb7d7bdc sleepq_switch(1e11e30,74) at sleepq_switch+0xf7/frame 0xb7d7bfc _sleep(1e11e30,1e11e1c,74,1470c40,0,0,0,0,100) at _sleep+0x2f7/frame 0xb7d7c3c vm_daemon(0,b7d7ce8) at vm_daemon+0x7f/frame 0xb7d7cb4 fork_exit(12f8160,0,b7d7ce8,0,0,...) at fork_exit+0x69/frame 0xb7d7cd4 fork_trampoline() at 0xffc033de/frame 0xb7d7cd4 --- kthread start Tracing command pagedaemon pid 21 tid 100097 td 0x187ccc00 sched_switch(187ccc00,104) at sched_switch+0x4f3/frame 0xb7d29d0 mi_switch(104) at mi_switch+0x1c4/frame 0xb7d2a10 sleepq_switch(187ccc00,0,1e4ad5c,b7d2a94,187ccc00,...) at sleepq_switch+0xf7/frame 0xb7d2a30 sleepq_timedwait(1e4ad5c,54,10000,10000,148f4e1,...) at sleepq_timedwait+0x3f/frame 0xb7d2a54 _sleep(1e4ad5c,1e4aa80,254,1470c40,1999997c,0,0,0,100) at _sleep+0x22c/frame 0xb7d2a94 vm_pageout_worker(0,187ccc00,187b474c,187ccc00,187b474c,...) at vm_pageout_worker+0x276/frame 0xb7d2c94 vm_pageout(0,b7d2ce8) at vm_pageout+0x146/frame 0xb7d2cb4 fork_exit(12ed410,0,b7d2ce8,0,0,...) at fork_exit+0x69/frame 0xb7d2cd4 fork_trampoline() at 0xffc033de/frame 0xb7d2cd4 --- kthread start Tracing command pagedaemon pid 21 tid 100100 td 0x181dce00 sched_switch(181dce00,104) at sched_switch+0x4f3/frame 0x97fca78 mi_switch(104) at mi_switch+0x1c4/frame 0x97fcab4 sleepq_switch(1e4ad68,54) at sleepq_switch+0xf7/frame 0x97fcad4 _sleep(1e4ad68,1e4a9c0,54,1562d99,0,0,0,0,100) at _sleep+0x2f7/frame 0x97fcb14 vm_pageout_laundry_worker(0,97fcce8) at vm_pageout_laundry_worker+0xccb/frame 0x97fccb4 fork_exit(12eebb0,0,97fcce8,0,0,...) at fork_exit+0x69/frame 0x97fccd4 fork_trampoline() at 0xffc033de/frame 0x97fccd4 --- kthread start Tracing command pagedaemon pid 21 tid 100101 td 0x181dca80 sched_switch(181dca80,104) at sched_switch+0x4f3/frame 0x1d3fabb8 mi_switch(104) at mi_switch+0x1c4/frame 0x1d3fabf8 sleepq_switch(0,1e1195c,1d3fac40,102542b,181dca80,...) at sleepq_switch+0xf7/frame 0x1d3fac18 sleepq_timedwait(12c8a10,54,1d3fac88,fdf5b7,14bf365,...) at sleepq_timedwait+0x3f/frame 0x1d3fac3c _sleep(12c8a10,1e1195c,54,14d2154,fffffed8,0,0,0,100) at _sleep+0x22c/frame 0x1d3fac7c uma_reclaim_worker(0,1d3face8) at uma_reclaim_worker+0xe4/frame 0x1d3facb4 fork_exit(12c8fc0,0,1d3face8,0,0,...) at fork_exit+0x69/frame 0x1d3facd4 fork_trampoline() at 0xffc033de/frame 0x1d3facd4 --- kthread start Tracing command rand_harvestq pid 20 tid 100094 td 0x18892500 sched_switch(18892500,104) at sched_switch+0x4f3/frame 0xb7bfb78 mi_switch(104) at mi_switch+0x1c4/frame 0xb7bfbb8 sleepq_switch(18892500,0,1bf407c,b7bfc3c,18892500,...) at sleepq_switch+0xf7/frame 0xb7bfbd8 sleepq_timedwait(1bf407c,0,1bf916c,0,b7bfc38,...) at sleepq_timedwait+0x3f/frame 0xb7bfbfc _sleep(1bf407c,0,0,14745d3,19999999,0,0,0,4) at _sleep+0x22c/frame 0xb7bfc3c random_kthread(0,b7bfce8) at random_kthread+0x50/frame 0xb7bfcb4 fork_exit(d46790,0,b7bfce8,0,0,...) at fork_exit+0x69/frame 0xb7bfcd4 fork_trampoline() at 0xffc033de/frame 0xb7bfcd4 --- kthread start Tracing command sctp_iterator pid 19 tid 100090 td 0x1889f380 sched_switch(1889f380,104) at sched_switch+0x4f3/frame 0xb7abbe8 mi_switch(104) at mi_switch+0x1c4/frame 0xb7abc28 sleepq_switch(1e5bd78,0) at sleepq_switch+0xf7/frame 0xb7abc48 _sleep(1e5bd78,1e5bd40,0,1468e12,0,0,0,0,100) at _sleep+0x2f7/frame 0xb7abc88 sctp_iterator_thread(0,b7abce8) at sctp_iterator_thread+0x43/frame 0xb7abcb4 fork_exit(115e5c0,0,b7abce8,0,0,...) at fork_exit+0x69/frame 0xb7abcd4 fork_trampoline() at 0xffc033de/frame 0xb7abcd4 --- kthread start Tracing command soaiod4 pid 18 tid 100089 td 0x187e1000 sched_switch(187e1000,104) at sched_switch+0x4f3/frame 0xb7a1bb0 mi_switch(104) at mi_switch+0x1c4/frame 0xb7a1bf0 sleepq_switch(1526a9e,187e1000,0,1dfc728,b7a1c78,...) at sleepq_switch+0xf7/frame 0xb7a1c10 sleepq_timedwait(1dfc728,0,187e1000,13cc6d0,14ecfea,...) at sleepq_timedwait+0x3f/frame 0xb7a1c38 _sleep(1dfc728,1dfc708,0,14745d3,ffffdd50,1d,0,0,100) at _sleep+0x22c/frame 0xb7a1c78 soaio_kproc_loop(4,b7a1ce8) at soaio_kproc_loop+0x148/frame 0xb7a1cb4 fork_exit(10428f0,4,b7a1ce8,0,0,...) at fork_exit+0x69/frame 0xb7a1cd4 fork_trampoline() at 0xffc033de/frame 0xb7a1cd4 --- kthread start Tracing command soaiod3 pid 17 tid 100088 td 0x187e1380 sched_switch(187e1380,104) at sched_switch+0x4f3/frame 0xb79cbb0 mi_switch(104) at mi_switch+0x1c4/frame 0xb79cbf0 sleepq_switch(1526a9e,187e1380,0,1dfc728,b79cc78,...) at sleepq_switch+0xf7/frame 0xb79cc10 sleepq_timedwait(1dfc728,0,187e1380,13cc6d0,14ecfea,...) at sleepq_timedwait+0x3f/frame 0xb79cc38 _sleep(1dfc728,1dfc708,0,14745d3,ffffdd50,1d,0,0,100) at _sleep+0x22c/frame 0xb79cc78 soaio_kproc_loop(3,b79cce8) at soaio_kproc_loop+0x148/frame 0xb79ccb4 fork_exit(10428f0,3,b79cce8,0,0,...) at fork_exit+0x69/frame 0xb79ccd4 fork_trampoline() at 0xffc033de/frame 0xb79ccd4 --- kthread start Tracing command soaiod2 pid 16 tid 100087 td 0x187e1700 sched_switch(187e1700,104) at sched_switch+0x4f3/frame 0xb797bb0 mi_switch(104) at mi_switch+0x1c4/frame 0xb797bf0 sleepq_switch(1526a9e,187e1700,0,1dfc728,b797c78,...) at sleepq_switch+0xf7/frame 0xb797c10 sleepq_timedwait(1dfc728,0,187e1700,13cc6d0,14ecfea,...) at sleepq_timedwait+0x3f/frame 0xb797c38 _sleep(1dfc728,1dfc708,0,14745d3,ffffdd50,1d,0,0,100) at _sleep+0x22c/frame 0xb797c78 soaio_kproc_loop(2,b797ce8) at soaio_kproc_loop+0x148/frame 0xb797cb4 fork_exit(10428f0,2,b797ce8,0,0,...) at fork_exit+0x69/frame 0xb797cd4 fork_trampoline() at 0xffc033de/frame 0xb797cd4 --- kthread start Tracing command soaiod1 pid 9 tid 100086 td 0x187e1a80 sched_switch(187e1a80,104) at sched_switch+0x4f3/frame 0xb792bb0 mi_switch(104) at mi_switch+0x1c4/frame 0xb792bf0 sleepq_switch(1526a9e,187e1a80,0,1dfc728,b792c78,...) at sleepq_switch+0xf7/frame 0xb792c10 sleepq_timedwait(1dfc728,0,2782e8b8,47aaa0,14ecfea,...) at sleepq_timedwait+0x3f/frame 0xb792c38 _sleep(1dfc728,1dfc708,0,14745d3,ffffdd50,1d,0,0,100) at _sleep+0x22c/frame 0xb792c78 soaio_kproc_loop(1,b792ce8) at soaio_kproc_loop+0x148/frame 0xb792cb4 fork_exit(10428f0,1,b792ce8,0,0,...) at fork_exit+0x69/frame 0xb792cd4 fork_trampoline() at 0xffc033de/frame 0xb792cd4 --- kthread start Tracing command fdc0 pid 8 tid 100083 td 0x181dd880 sched_switch(181dd880,104) at sched_switch+0x4f3/frame 0xb783ba0 mi_switch(104) at mi_switch+0x1c4/frame 0xb783bdc sleepq_switch(1813023c,5c) at sleepq_switch+0xf7/frame 0xb783bfc _sleep(1813023c,181302f8,5c,14745d3,0,0,0,0,100) at _sleep+0x2f7/frame 0xb783c3c fdc_thread(18130200,b783ce8) at fdc_thread+0x351/frame 0xb783cb4 fork_exit(13748b0,18130200,b783ce8,0,0,...) at fork_exit+0x69/frame 0xb783cd4 fork_trampoline() at 0xffc033de/frame 0xb783cd4 --- kthread start Tracing command usb pid 15 tid 100048 td 0x17ed6180 sched_switch(17ed6180,104) at sched_switch+0x4f3/frame 0xb6cbc00 mi_switch(104) at mi_switch+0x1c4/frame 0xb6cbc3c sleepq_switch(17ec2aac,0) at sleepq_switch+0xf7/frame 0xb6cbc5c _cv_wait(17ec2aac,17ec2cd8) at _cv_wait+0x135/frame 0xb6cbc94 usb_process(17ec2aa4,b6cbce8) at usb_process+0x9d/frame 0xb6cbcb4 fork_exit(e3aa70,17ec2aa4,b6cbce8,0,0,...) at fork_exit+0x69/frame 0xb6cbcd4 fork_trampoline() at 0xffc033de/frame 0xb6cbcd4 --- kthread start Tracing command usb pid 15 tid 100049 td 0x17ed5e00 sched_switch(17ed5e00,104) at sched_switch+0x4f3/frame 0xb6d0c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb6d0c3c sleepq_switch(17ec2adc,0) at sleepq_switch+0xf7/frame 0xb6d0c5c _cv_wait(17ec2adc,17ec2cd8) at _cv_wait+0x135/frame 0xb6d0c94 usb_process(17ec2ad4,b6d0ce8) at usb_process+0x9d/frame 0xb6d0cb4 fork_exit(e3aa70,17ec2ad4,b6d0ce8,0,0,...) at fork_exit+0x69/frame 0xb6d0cd4 fork_trampoline() at 0xffc033de/frame 0xb6d0cd4 --- kthread start Tracing command usb pid 15 tid 100050 td 0x17ed5a80 sched_switch(17ed5a80,104) at sched_switch+0x4f3/frame 0xb6d5c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb6d5c3c sleepq_switch(17ec2b0c,0) at sleepq_switch+0xf7/frame 0xb6d5c5c _cv_wait(17ec2b0c,17ec2cd8) at _cv_wait+0x135/frame 0xb6d5c94 usb_process(17ec2b04,b6d5ce8) at usb_process+0x9d/frame 0xb6d5cb4 fork_exit(e3aa70,17ec2b04,b6d5ce8,0,0,...) at fork_exit+0x69/frame 0xb6d5cd4 fork_trampoline() at 0xffc033de/frame 0xb6d5cd4 --- kthread start Tracing command usb pid 15 tid 100051 td 0x17ed5700 sched_switch(17ed5700,104) at sched_switch+0x4f3/frame 0xb6dac00 mi_switch(104) at mi_switch+0x1c4/frame 0xb6dac3c sleepq_switch(17ec2b3c,0) at sleepq_switch+0xf7/frame 0xb6dac5c _cv_wait(17ec2b3c,17ec2cd8) at _cv_wait+0x135/frame 0xb6dac94 usb_process(17ec2b34,b6dace8) at usb_process+0x9d/frame 0xb6dacb4 fork_exit(e3aa70,17ec2b34,b6dace8,0,0,...) at fork_exit+0x69/frame 0xb6dacd4 fork_trampoline() at 0xffc033de/frame 0xb6dacd4 --- kthread start Tracing command usb pid 15 tid 100052 td 0x17ed5380 sched_switch(17ed5380,104) at sched_switch+0x4f3/frame 0xb6dfc00 mi_switch(104) at mi_switch+0x1c4/frame 0xb6dfc3c sleepq_switch(17ec2b6c,0) at sleepq_switch+0xf7/frame 0xb6dfc5c _cv_wait(17ec2b6c,17ec2cd8) at _cv_wait+0x135/frame 0xb6dfc94 usb_process(17ec2b64,b6dfce8) at usb_process+0x9d/frame 0xb6dfcb4 fork_exit(e3aa70,17ec2b64,b6dfce8,0,0,...) at fork_exit+0x69/frame 0xb6dfcd4 fork_trampoline() at 0xffc033de/frame 0xb6dfcd4 --- kthread start Tracing command usb pid 15 tid 100054 td 0x9988c00 sched_switch(9988c00,104) at sched_switch+0x4f3/frame 0xb6eac00 mi_switch(104) at mi_switch+0x1c4/frame 0xb6eac3c sleepq_switch(181a3aac,0) at sleepq_switch+0xf7/frame 0xb6eac5c _cv_wait(181a3aac,181a3cd8) at _cv_wait+0x135/frame 0xb6eac94 usb_process(181a3aa4,b6eace8) at usb_process+0x9d/frame 0xb6eacb4 fork_exit(e3aa70,181a3aa4,b6eace8,0,0,...) at fork_exit+0x69/frame 0xb6eacd4 fork_trampoline() at 0xffc033de/frame 0xb6eacd4 --- kthread start Tracing command usb pid 15 tid 100055 td 0x9988880 sched_switch(9988880,104) at sched_switch+0x4f3/frame 0xb6efc00 mi_switch(104) at mi_switch+0x1c4/frame 0xb6efc3c sleepq_switch(181a3adc,0) at sleepq_switch+0xf7/frame 0xb6efc5c _cv_wait(181a3adc,181a3cd8) at _cv_wait+0x135/frame 0xb6efc94 usb_process(181a3ad4,b6efce8) at usb_process+0x9d/frame 0xb6efcb4 fork_exit(e3aa70,181a3ad4,b6efce8,0,0,...) at fork_exit+0x69/frame 0xb6efcd4 fork_trampoline() at 0xffc033de/frame 0xb6efcd4 --- kthread start Tracing command usb pid 15 tid 100056 td 0x9988500 sched_switch(9988500,104) at sched_switch+0x4f3/frame 0xb6f4c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb6f4c3c sleepq_switch(181a3b0c,0) at sleepq_switch+0xf7/frame 0xb6f4c5c _cv_wait(181a3b0c,181a3cd8) at _cv_wait+0x135/frame 0xb6f4c94 usb_process(181a3b04,b6f4ce8) at usb_process+0x9d/frame 0xb6f4cb4 fork_exit(e3aa70,181a3b04,b6f4ce8,0,0,...) at fork_exit+0x69/frame 0xb6f4cd4 fork_trampoline() at 0xffc033de/frame 0xb6f4cd4 --- kthread start Tracing command usb pid 15 tid 100057 td 0x9988180 sched_switch(9988180,104) at sched_switch+0x4f3/frame 0xb6f9c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb6f9c3c sleepq_switch(181a3b3c,0) at sleepq_switch+0xf7/frame 0xb6f9c5c _cv_wait(181a3b3c,181a3cd8) at _cv_wait+0x135/frame 0xb6f9c94 usb_process(181a3b34,b6f9ce8) at usb_process+0x9d/frame 0xb6f9cb4 fork_exit(e3aa70,181a3b34,b6f9ce8,0,0,...) at fork_exit+0x69/frame 0xb6f9cd4 fork_trampoline() at 0xffc033de/frame 0xb6f9cd4 --- kthread start Tracing command usb pid 15 tid 100058 td 0x9987e00 sched_switch(9987e00,104) at sched_switch+0x4f3/frame 0xb6fec00 mi_switch(104) at mi_switch+0x1c4/frame 0xb6fec3c sleepq_switch(181a3b6c,0) at sleepq_switch+0xf7/frame 0xb6fec5c _cv_wait(181a3b6c,181a3cd8) at _cv_wait+0x135/frame 0xb6fec94 usb_process(181a3b64,b6fece8) at usb_process+0x9d/frame 0xb6fecb4 fork_exit(e3aa70,181a3b64,b6fece8,0,0,...) at fork_exit+0x69/frame 0xb6fecd4 fork_trampoline() at 0xffc033de/frame 0xb6fecd4 --- kthread start Tracing command usb pid 15 tid 100060 td 0x181b3880 sched_switch(181b3880,104) at sched_switch+0x4f3/frame 0xb709c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb709c3c sleepq_switch(181afaac,0) at sleepq_switch+0xf7/frame 0xb709c5c _cv_wait(181afaac,181afcd8) at _cv_wait+0x135/frame 0xb709c94 usb_process(181afaa4,b709ce8) at usb_process+0x9d/frame 0xb709cb4 fork_exit(e3aa70,181afaa4,b709ce8,0,0,...) at fork_exit+0x69/frame 0xb709cd4 fork_trampoline() at 0xffc033de/frame 0xb709cd4 --- kthread start Tracing command usb pid 15 tid 100061 td 0x181b3500 sched_switch(181b3500,104) at sched_switch+0x4f3/frame 0xb70ec00 mi_switch(104) at mi_switch+0x1c4/frame 0xb70ec3c sleepq_switch(181afadc,0) at sleepq_switch+0xf7/frame 0xb70ec5c _cv_wait(181afadc,181afcd8) at _cv_wait+0x135/frame 0xb70ec94 usb_process(181afad4,b70ece8) at usb_process+0x9d/frame 0xb70ecb4 fork_exit(e3aa70,181afad4,b70ece8,0,0,...) at fork_exit+0x69/frame 0xb70ecd4 fork_trampoline() at 0xffc033de/frame 0xb70ecd4 --- kthread start Tracing command usb pid 15 tid 100062 td 0x181b3180 sched_switch(181b3180,104) at sched_switch+0x4f3/frame 0xb713c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb713c3c sleepq_switch(181afb0c,0) at sleepq_switch+0xf7/frame 0xb713c5c _cv_wait(181afb0c,181afcd8) at _cv_wait+0x135/frame 0xb713c94 usb_process(181afb04,b713ce8) at usb_process+0x9d/frame 0xb713cb4 fork_exit(e3aa70,181afb04,b713ce8,0,0,...) at fork_exit+0x69/frame 0xb713cd4 fork_trampoline() at 0xffc033de/frame 0xb713cd4 --- kthread start Tracing command usb pid 15 tid 100063 td 0x181b2e00 sched_switch(181b2e00,104) at sched_switch+0x4f3/frame 0xb718c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb718c3c sleepq_switch(181afb3c,0) at sleepq_switch+0xf7/frame 0xb718c5c _cv_wait(181afb3c,181afcd8) at _cv_wait+0x135/frame 0xb718c94 usb_process(181afb34,b718ce8) at usb_process+0x9d/frame 0xb718cb4 fork_exit(e3aa70,181afb34,b718ce8,0,0,...) at fork_exit+0x69/frame 0xb718cd4 fork_trampoline() at 0xffc033de/frame 0xb718cd4 --- kthread start Tracing command usb pid 15 tid 100064 td 0x181b2a80 sched_switch(181b2a80,104) at sched_switch+0x4f3/frame 0xb71dc00 mi_switch(104) at mi_switch+0x1c4/frame 0xb71dc3c sleepq_switch(181afb6c,0) at sleepq_switch+0xf7/frame 0xb71dc5c _cv_wait(181afb6c,181afcd8) at _cv_wait+0x135/frame 0xb71dc94 usb_process(181afb64,b71dce8) at usb_process+0x9d/frame 0xb71dcb4 fork_exit(e3aa70,181afb64,b71dce8,0,0,...) at fork_exit+0x69/frame 0xb71dcd4 fork_trampoline() at 0xffc033de/frame 0xb71dcd4 --- kthread start Tracing command usb pid 15 tid 100065 td 0x181b2700 sched_switch(181b2700,104) at sched_switch+0x4f3/frame 0xb723c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb723c3c sleepq_switch(181b4aac,0) at sleepq_switch+0xf7/frame 0xb723c5c _cv_wait(181b4aac,181b4cd8) at _cv_wait+0x135/frame 0xb723c94 usb_process(181b4aa4,b723ce8) at usb_process+0x9d/frame 0xb723cb4 fork_exit(e3aa70,181b4aa4,b723ce8,0,0,...) at fork_exit+0x69/frame 0xb723cd4 fork_trampoline() at 0xffc033de/frame 0xb723cd4 --- kthread start Tracing command usb pid 15 tid 100066 td 0x181b2380 sched_switch(181b2380,104) at sched_switch+0x4f3/frame 0xb728c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb728c3c sleepq_switch(181b4adc,0) at sleepq_switch+0xf7/frame 0xb728c5c _cv_wait(181b4adc,181b4cd8) at _cv_wait+0x135/frame 0xb728c94 usb_process(181b4ad4,b728ce8) at usb_process+0x9d/frame 0xb728cb4 fork_exit(e3aa70,181b4ad4,b728ce8,0,0,...) at fork_exit+0x69/frame 0xb728cd4 fork_trampoline() at 0xffc033de/frame 0xb728cd4 --- kthread start Tracing command usb pid 15 tid 100067 td 0x181b2000 sched_switch(181b2000,104) at sched_switch+0x4f3/frame 0xb72dc00 mi_switch(104) at mi_switch+0x1c4/frame 0xb72dc3c sleepq_switch(181b4b0c,0) at sleepq_switch+0xf7/frame 0xb72dc5c _cv_wait(181b4b0c,181b4cd8) at _cv_wait+0x135/frame 0xb72dc94 usb_process(181b4b04,b72dce8) at usb_process+0x9d/frame 0xb72dcb4 fork_exit(e3aa70,181b4b04,b72dce8,0,0,...) at fork_exit+0x69/frame 0xb72dcd4 fork_trampoline() at 0xffc033de/frame 0xb72dcd4 --- kthread start Tracing command usb pid 15 tid 100068 td 0x17ee4c00 sched_switch(17ee4c00,104) at sched_switch+0x4f3/frame 0xb732c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb732c3c sleepq_switch(181b4b3c,0) at sleepq_switch+0xf7/frame 0xb732c5c _cv_wait(181b4b3c,181b4cd8) at _cv_wait+0x135/frame 0xb732c94 usb_process(181b4b34,b732ce8) at usb_process+0x9d/frame 0xb732cb4 fork_exit(e3aa70,181b4b34,b732ce8,0,0,...) at fork_exit+0x69/frame 0xb732cd4 fork_trampoline() at 0xffc033de/frame 0xb732cd4 --- kthread start Tracing command usb pid 15 tid 100069 td 0x17ee4880 sched_switch(17ee4880,104) at sched_switch+0x4f3/frame 0xb737c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb737c3c sleepq_switch(181b4b6c,0) at sleepq_switch+0xf7/frame 0xb737c5c _cv_wait(181b4b6c,181b4cd8) at _cv_wait+0x135/frame 0xb737c94 usb_process(181b4b64,b737ce8) at usb_process+0x9d/frame 0xb737cb4 fork_exit(e3aa70,181b4b64,b737ce8,0,0,...) at fork_exit+0x69/frame 0xb737cd4 fork_trampoline() at 0xffc033de/frame 0xb737cd4 --- kthread start Tracing command usb pid 15 tid 100070 td 0x17ee4500 sched_switch(17ee4500,104) at sched_switch+0x4f3/frame 0xb73dc00 mi_switch(104) at mi_switch+0x1c4/frame 0xb73dc3c sleepq_switch(181beaac,0) at sleepq_switch+0xf7/frame 0xb73dc5c _cv_wait(181beaac,181becd8) at _cv_wait+0x135/frame 0xb73dc94 usb_process(181beaa4,b73dce8) at usb_process+0x9d/frame 0xb73dcb4 fork_exit(e3aa70,181beaa4,b73dce8,0,0,...) at fork_exit+0x69/frame 0xb73dcd4 fork_trampoline() at 0xffc033de/frame 0xb73dcd4 --- kthread start Tracing command usb pid 15 tid 100071 td 0x17ee4180 sched_switch(17ee4180,104) at sched_switch+0x4f3/frame 0xb742c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb742c3c sleepq_switch(181beadc,0) at sleepq_switch+0xf7/frame 0xb742c5c _cv_wait(181beadc,181becd8) at _cv_wait+0x135/frame 0xb742c94 usb_process(181bead4,b742ce8) at usb_process+0x9d/frame 0xb742cb4 fork_exit(e3aa70,181bead4,b742ce8,0,0,...) at fork_exit+0x69/frame 0xb742cd4 fork_trampoline() at 0xffc033de/frame 0xb742cd4 --- kthread start Tracing command usb pid 15 tid 100072 td 0x17ee3e00 sched_switch(17ee3e00,104) at sched_switch+0x4f3/frame 0xb747c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb747c3c sleepq_switch(181beb0c,0) at sleepq_switch+0xf7/frame 0xb747c5c _cv_wait(181beb0c,181becd8) at _cv_wait+0x135/frame 0xb747c94 usb_process(181beb04,b747ce8) at usb_process+0x9d/frame 0xb747cb4 fork_exit(e3aa70,181beb04,b747ce8,0,0,...) at fork_exit+0x69/frame 0xb747cd4 fork_trampoline() at 0xffc033de/frame 0xb747cd4 --- kthread start Tracing command usb pid 15 tid 100073 td 0x17ee3a80 sched_switch(17ee3a80,104) at sched_switch+0x4f3/frame 0xb74cc00 mi_switch(104) at mi_switch+0x1c4/frame 0xb74cc3c sleepq_switch(181beb3c,0) at sleepq_switch+0xf7/frame 0xb74cc5c _cv_wait(181beb3c,181becd8) at _cv_wait+0x135/frame 0xb74cc94 usb_process(181beb34,b74cce8) at usb_process+0x9d/frame 0xb74ccb4 fork_exit(e3aa70,181beb34,b74cce8,0,0,...) at fork_exit+0x69/frame 0xb74ccd4 fork_trampoline() at 0xffc033de/frame 0xb74ccd4 --- kthread start Tracing command usb pid 15 tid 100074 td 0x17ee3700 sched_switch(17ee3700,104) at sched_switch+0x4f3/frame 0xb751c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb751c3c sleepq_switch(181beb6c,0) at sleepq_switch+0xf7/frame 0xb751c5c _cv_wait(181beb6c,181becd8) at _cv_wait+0x135/frame 0xb751c94 usb_process(181beb64,b751ce8) at usb_process+0x9d/frame 0xb751cb4 fork_exit(e3aa70,181beb64,b751ce8,0,0,...) at fork_exit+0x69/frame 0xb751cd4 fork_trampoline() at 0xffc033de/frame 0xb751cd4 --- kthread start Tracing command usb pid 15 tid 100076 td 0x181df180 sched_switch(181df180,104) at sched_switch+0x4f3/frame 0xb75cc00 mi_switch(104) at mi_switch+0x1c4/frame 0xb75cc3c sleepq_switch(17ebdc84,0) at sleepq_switch+0xf7/frame 0xb75cc5c _cv_wait(17ebdc84,17ebdeb0) at _cv_wait+0x135/frame 0xb75cc94 usb_process(17ebdc7c,b75cce8) at usb_process+0x9d/frame 0xb75ccb4 fork_exit(e3aa70,17ebdc7c,b75cce8,0,0,...) at fork_exit+0x69/frame 0xb75ccd4 fork_trampoline() at 0xffc033de/frame 0xb75ccd4 --- kthread start Tracing command usb pid 15 tid 100077 td 0x181dee00 sched_switch(181dee00,104) at sched_switch+0x4f3/frame 0xb761c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb761c3c sleepq_switch(17ebdcb4,0) at sleepq_switch+0xf7/frame 0xb761c5c _cv_wait(17ebdcb4,17ebdeb0) at _cv_wait+0x135/frame 0xb761c94 usb_process(17ebdcac,b761ce8) at usb_process+0x9d/frame 0xb761cb4 fork_exit(e3aa70,17ebdcac,b761ce8,0,0,...) at fork_exit+0x69/frame 0xb761cd4 fork_trampoline() at 0xffc033de/frame 0xb761cd4 --- kthread start Tracing command usb pid 15 tid 100078 td 0x181dea80 sched_switch(181dea80,104) at sched_switch+0x4f3/frame 0xb766c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb766c3c sleepq_switch(17ebdce4,0) at sleepq_switch+0xf7/frame 0xb766c5c _cv_wait(17ebdce4,17ebdeb0) at _cv_wait+0x135/frame 0xb766c94 usb_process(17ebdcdc,b766ce8) at usb_process+0x9d/frame 0xb766cb4 fork_exit(e3aa70,17ebdcdc,b766ce8,0,0,...) at fork_exit+0x69/frame 0xb766cd4 fork_trampoline() at 0xffc033de/frame 0xb766cd4 --- kthread start Tracing command usb pid 15 tid 100079 td 0x181de700 sched_switch(181de700,104) at sched_switch+0x4f3/frame 0xb76bc00 mi_switch(104) at mi_switch+0x1c4/frame 0xb76bc3c sleepq_switch(17ebdd14,0) at sleepq_switch+0xf7/frame 0xb76bc5c _cv_wait(17ebdd14,17ebdeb0) at _cv_wait+0x135/frame 0xb76bc94 usb_process(17ebdd0c,b76bce8) at usb_process+0x9d/frame 0xb76bcb4 fork_exit(e3aa70,17ebdd0c,b76bce8,0,0,...) at fork_exit+0x69/frame 0xb76bcd4 fork_trampoline() at 0xffc033de/frame 0xb76bcd4 --- kthread start Tracing command usb pid 15 tid 100080 td 0x181de380 sched_switch(181de380,104) at sched_switch+0x4f3/frame 0xb770c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb770c3c sleepq_switch(17ebdd44,0) at sleepq_switch+0xf7/frame 0xb770c5c _cv_wait(17ebdd44,17ebdeb0) at _cv_wait+0x135/frame 0xb770c94 usb_process(17ebdd3c,b770ce8) at usb_process+0x9d/frame 0xb770cb4 fork_exit(e3aa70,17ebdd3c,b770ce8,0,0,...) at fork_exit+0x69/frame 0xb770cd4 fork_trampoline() at 0xffc033de/frame 0xb770cd4 --- kthread start Tracing command cam pid 7 tid 100043 td 0x9985380 sched_switch(9985380,104) at sched_switch+0x4f3/frame 0xb684bd0 mi_switch(104) at mi_switch+0x1c4/frame 0xb684c0c sleepq_switch(1b2d540,5c) at sleepq_switch+0xf7/frame 0xb684c2c _sleep(1b2d540,1b2d500,5c,14745d3,0,0,0,0,100) at _sleep+0x2f7/frame 0xb684c6c xpt_done_td(1b2d500,b684ce8) at xpt_done_td+0x72/frame 0xb684cb4 fork_exit(907280,1b2d500,b684ce8,0,0,...) at fork_exit+0x69/frame 0xb684cd4 fork_trampoline() at 0xffc033de/frame 0xb684cd4 --- kthread start Tracing command cam pid 7 tid 100096 td 0x18891e00 sched_switch(18891e00,104) at sched_switch+0x4f3/frame 0xb7c9bc8 mi_switch(104) at mi_switch+0x1c4/frame 0xb7c9c04 sleepq_switch(1b2d468,5c) at sleepq_switch+0xf7/frame 0xb7c9c24 _sleep(1b2d468,1b2d4d8,5c,14745d3,0,0,0,0,100) at _sleep+0x2f7/frame 0xb7c9c64 xpt_scanner_thread(0,b7c9ce8) at xpt_scanner_thread+0x56/frame 0xb7c9cb4 fork_exit(907520,0,b7c9ce8,0,0,...) at fork_exit+0x69/frame 0xb7c9cd4 fork_trampoline() at 0xffc033de/frame 0xb7c9cd4 --- kthread start Tracing command crypto returns 3 pid 6 tid 100042 td 0x9985700 sched_switch(9985700,104) at sched_switch+0x4f3/frame 0xb67fbc8 mi_switch(104) at mi_switch+0x1c4/frame 0xb67fc04 sleepq_switch(9906dc4,6c) at sleepq_switch+0xf7/frame 0xb67fc24 _sleep(9906dc4,9906da8,6c,1558535,0,0,0,0,100) at _sleep+0x2f7/frame 0xb67fc64 crypto_ret_proc(9906da8,b67fce8) at crypto_ret_proc+0x202/frame 0xb67fcb4 fork_exit(1240b00,9906da8,b67fce8,0,0,...) at fork_exit+0x69/frame 0xb67fcd4 fork_trampoline() at 0xffc033de/frame 0xb67fcd4 --- kthread start Tracing command crypto returns 2 pid 5 tid 100041 td 0x9985a80 sched_switch(9985a80,104) at sched_switch+0x4f3/frame 0x129fcbc8 mi_switch(104) at mi_switch+0x1c4/frame 0x129fcc04 sleepq_switch(9906d8c,6c) at sleepq_switch+0xf7/frame 0x129fcc24 _sleep(9906d8c,9906d70,6c,1558535,0,0,0,0,100) at _sleep+0x2f7/frame 0x129fcc64 crypto_ret_proc(9906d70,129fcce8) at crypto_ret_proc+0x202/frame 0x129fccb4 fork_exit(1240b00,9906d70,129fcce8,0,0,...) at fork_exit+0x69/frame 0x129fccd4 fork_trampoline() at 0xffc033de/frame 0x129fccd4 --- kthread start Tracing command crypto returns 1 pid 4 tid 100040 td 0x9985e00 sched_switch(9985e00,104) at sched_switch+0x4f3/frame 0x129f7bc8 mi_switch(104) at mi_switch+0x1c4/frame 0x129f7c04 sleepq_switch(9906d54,6c) at sleepq_switch+0xf7/frame 0x129f7c24 _sleep(9906d54,9906d38,6c,1558535,0,0,0,0,100) at _sleep+0x2f7/frame 0x129f7c64 crypto_ret_proc(9906d38,129f7ce8) at crypto_ret_proc+0x202/frame 0x129f7cb4 fork_exit(1240b00,9906d38,129f7ce8,0,0,...) at fork_exit+0x69/frame 0x129f7cd4 fork_trampoline() at 0xffc033de/frame 0x129f7cd4 --- kthread start Tracing command crypto returns 0 pid 3 tid 100039 td 0x9986180 sched_switch(9986180,104) at sched_switch+0x4f3/frame 0x129f2bc8 mi_switch(104) at mi_switch+0x1c4/frame 0x129f2c04 sleepq_switch(9906d1c,6c) at sleepq_switch+0xf7/frame 0x129f2c24 _sleep(9906d1c,9906d00,6c,1558535,0,0,0,0,100) at _sleep+0x2f7/frame 0x129f2c64 crypto_ret_proc(9906d00,129f2ce8) at crypto_ret_proc+0x202/frame 0x129f2cb4 fork_exit(1240b00,9906d00,129f2ce8,0,0,...) at fork_exit+0x69/frame 0x129f2cd4 fork_trampoline() at 0xffc033de/frame 0x129f2cd4 --- kthread start Tracing command crypto pid 2 tid 100038 td 0x9986500 sched_switch(9986500,104) at sched_switch+0x4f3/frame 0x129edbd8 mi_switch(104) at mi_switch+0x1c4/frame 0x129edc14 sleepq_switch(1e10b28,6c) at sleepq_switch+0xf7/frame 0x129edc34 _sleep(1e10b28,1e10b10,6c,150bae0,0,0,0,0,100) at _sleep+0x2f7/frame 0x129edc74 crypto_proc(0,129edce8) at crypto_proc+0x258/frame 0x129edcb4 fork_exit(1240760,0,129edce8,0,0,...) at fork_exit+0x69/frame 0x129edcd4 fork_trampoline() at 0xffc033de/frame 0x129edcd4 --- kthread start Tracing command sequencer 00 pid 14 tid 100033 td 0x9987700 sched_switch(9987700,104) at sched_switch+0x4f3/frame 0x129d4bd8 mi_switch(104) at mi_switch+0x1c4/frame 0x129d4c14 sleepq_switch(9d1564c,0) at sleepq_switch+0xf7/frame 0x129d4c34 _cv_wait(9d1564c,9d15604) at _cv_wait+0x135/frame 0x129d4c6c seq_eventthread(9d15600,129d4ce8) at seq_eventthread+0x7b/frame 0x129d4cb4 fork_exit(dc7c50,9d15600,129d4ce8,0,0,...) at fork_exit+0x69/frame 0x129d4cd4 fork_trampoline() at 0xffc033de/frame 0x129d4cd4 --- kthread start Tracing command geom pid 13 tid 100030 td 0x991b000 sched_switch(991b000,104) at sched_switch+0x4f3/frame 0x129c5be0 mi_switch(104) at mi_switch+0x1c4/frame 0x129c5c1c sleepq_switch(1e4a2f0,5c) at sleepq_switch+0xf7/frame 0x129c5c3c _sleep(1e4a2f0,1d8a768,25c,14745d3,0,0,0,0,100) at _sleep+0x2f7/frame 0x129c5c7c g_run_events(0,129c5ce8) at g_run_events+0x4d/frame 0x129c5cb4 fork_exit(f2c910,0,129c5ce8,0,0,...) at fork_exit+0x69/frame 0x129c5cd4 fork_trampoline() at 0xffc033de/frame 0x129c5cd4 --- kthread start Tracing command geom pid 13 tid 100031 td 0x9df9c00 sched_switch(9df9c00,104) at sched_switch+0x4f3/frame 0x129cabd8 mi_switch(104) at mi_switch+0x1c4/frame 0x129cac14 sleepq_switch(1e4a2f8,5c) at sleepq_switch+0xf7/frame 0x129cac34 _sleep(1e4a2f8,1d8a7ac,25c,14745d3,0,0,0,0,100) at _sleep+0x2f7/frame 0x129cac74 g_io_schedule_up(9df9c00) at g_io_schedule_up+0x36/frame 0x129caca8 g_up_procbody(0,129cace8) at g_up_procbody+0x5b/frame 0x129cacb4 fork_exit(f2c960,0,129cace8,0,0,...) at fork_exit+0x69/frame 0x129cacd4 fork_trampoline() at 0xffc033de/frame 0x129cacd4 --- kthread start Tracing command geom pid 13 tid 100032 td 0x9df9880 sched_switch(9df9880,104) at sched_switch+0x4f3/frame 0x129cfbd0 mi_switch(104) at mi_switch+0x1c4/frame 0x129cfc10 sleepq_switch(1e4a2f4,5c) at sleepq_switch+0xf7/frame 0x129cfc30 _sleep(1e4a2f4,1d8a78c,25c,14745d3,0,0,0,0,100) at _sleep+0x2f7/frame 0x129cfc70 g_io_schedule_down(9df9880) at g_io_schedule_down+0x36/frame 0x129cfca8 g_down_procbody(0,129cfce8) at g_down_procbody+0x5b/frame 0x129cfcb4 fork_exit(f2c9c0,0,129cfce8,0,0,...) at fork_exit+0x69/frame 0x129cfcd4 fork_trampoline() at 0xffc033de/frame 0x129cfcd4 --- kthread start Tracing command intr pid 12 tid 100007 td 0x9df8000 sched_switch(9df8000,109) at sched_switch+0x4f3/frame 0x1293ac28 mi_switch(109) at mi_switch+0x1c4/frame 0x1293ac64 ithread_loop(988efa0,1293ace8) at ithread_loop+0x85/frame 0x1293acb4 fork_exit(f9b620,988efa0,1293ace8,0,0,...) at fork_exit+0x69/frame 0x1293acd4 fork_trampoline() at 0xffc033de/frame 0x1293acd4 --- kthread start Tracing command intr pid 12 tid 100009 td 0x9df3880 exit_tag() at 0xffc033d4 Tracing command intr pid 12 tid 100013 td 0x9df2a80 sched_switch(9df2a80,109) at sched_switch+0x4f3/frame 0x12958c28 mi_switch(109) at mi_switch+0x1c4/frame 0x12958c64 ithread_loop(988ef80,12958ce8) at ithread_loop+0x85/frame 0x12958cb4 fork_exit(f9b620,988ef80,12958ce8,0,0,...) at fork_exit+0x69/frame 0x12958cd4 fork_trampoline() at 0xffc033de/frame 0x12958cd4 --- kthread start Tracing command intr pid 12 tid 100023 td 0x991c880 sched_switch(991c880,109) at sched_switch+0x4f3/frame 0x12996c28 mi_switch(109) at mi_switch+0x1c4/frame 0x12996c64 ithread_loop(988ef70,12996ce8) at ithread_loop+0x85/frame 0x12996cb4 fork_exit(f9b620,988ef70,12996ce8,0,0,...) at fork_exit+0x69/frame 0x12996cd4 fork_trampoline() at 0xffc033de/frame 0x12996cd4 --- kthread start Tracing command intr pid 12 tid 100024 td 0x991c500 sched_switch(991c500,109) at sched_switch+0x4f3/frame 0x1299bc28 mi_switch(109) at mi_switch+0x1c4/frame 0x1299bc64 ithread_loop(988ef60,1299bce8) at ithread_loop+0x85/frame 0x1299bcb4 fork_exit(f9b620,988ef60,1299bce8,0,0,...) at fork_exit+0x69/frame 0x1299bcd4 fork_trampoline() at 0xffc033de/frame 0x1299bcd4 --- kthread start Tracing command intr pid 12 tid 100025 td 0x991c180 exit_tag() at 0xffc033d4 Tracing command intr pid 12 tid 100026 td 0x991be00 exit_tag() at 0xffc033d4 Tracing command intr pid 12 tid 100027 td 0x991ba80 exit_tag() at 0xffc033d4 Tracing command intr pid 12 tid 100028 td 0x991b700 exit_tag() at 0xffc033d4 Tracing command intr pid 12 tid 100044 td 0x17ee3000 sched_switch(17ee3000,109) at sched_switch+0x4f3/frame 0xb6b4c28 mi_switch(109) at mi_switch+0x1c4/frame 0xb6b4c64 ithread_loop(17ed4030,b6b4ce8) at ithread_loop+0x85/frame 0xb6b4cb4 fork_exit(f9b620,17ed4030,b6b4ce8,0,0,...) at fork_exit+0x69/frame 0xb6b4cd4 fork_trampoline() at 0xffc033de/frame 0xb6b4cd4 --- kthread start Tracing command intr pid 12 tid 100045 td 0x17ed6c00 sched_switch(17ed6c00,109) at sched_switch+0x4f3/frame 0xb6bac28 mi_switch(109) at mi_switch+0x1c4/frame 0xb6bac64 ithread_loop(17ed4020,b6bace8) at ithread_loop+0x85/frame 0xb6bacb4 fork_exit(f9b620,17ed4020,b6bace8,0,0,...) at fork_exit+0x69/frame 0xb6bacd4 fork_trampoline() at 0xffc033de/frame 0xb6bacd4 --- kthread start Tracing command intr pid 12 tid 100046 td 0x17ed6880 sched_switch(17ed6880,109) at sched_switch+0x4f3/frame 0xb6c0c28 mi_switch(109) at mi_switch+0x1c4/frame 0xb6c0c64 ithread_loop(17ed4010,b6c0ce8) at ithread_loop+0x85/frame 0xb6c0cb4 fork_exit(f9b620,17ed4010,b6c0ce8,0,0,...) at fork_exit+0x69/frame 0xb6c0cd4 fork_trampoline() at 0xffc033de/frame 0xb6c0cd4 --- kthread start Tracing command intr pid 12 tid 100047 td 0x17ed6500 sched_switch(17ed6500,109) at sched_switch+0x4f3/frame 0xb6c6c28 mi_switch(109) at mi_switch+0x1c4/frame 0xb6c6c64 ithread_loop(17ed4000,b6c6ce8) at ithread_loop+0x85/frame 0xb6c6cb4 fork_exit(f9b620,17ed4000,b6c6ce8,0,0,...) at fork_exit+0x69/frame 0xb6c6cd4 fork_trampoline() at 0xffc033de/frame 0xb6c6cd4 --- kthread start Tracing command intr pid 12 tid 100053 td 0x17ed5000 sched_switch(17ed5000,109) at sched_switch+0x4f3/frame 0xb6e5c28 mi_switch(109) at mi_switch+0x1c4/frame 0xb6e5c64 ithread_loop(988ffa0,b6e5ce8) at ithread_loop+0x85/frame 0xb6e5cb4 fork_exit(f9b620,988ffa0,b6e5ce8,0,0,...) at fork_exit+0x69/frame 0xb6e5cd4 fork_trampoline() at 0xffc033de/frame 0xb6e5cd4 --- kthread start Tracing command intr pid 12 tid 100059 td 0x9987a80 sched_switch(9987a80,109) at sched_switch+0x4f3/frame 0xb704c28 mi_switch(109) at mi_switch+0x1c4/frame 0xb704c64 ithread_loop(988ff90,b704ce8) at ithread_loop+0x85/frame 0xb704cb4 fork_exit(f9b620,988ff90,b704ce8,0,0,...) at fork_exit+0x69/frame 0xb704cd4 fork_trampoline() at 0xffc033de/frame 0xb704cd4 --- kthread start Tracing command intr pid 12 tid 100075 td 0x17ee3380 exit_tag() at 0xffc033d4 Tracing command intr pid 12 tid 100081 td 0x181de000 exit_tag() at 0xffc033d4 Tracing command intr pid 12 tid 100082 td 0x181ddc00 sched_switch(181ddc00,109) at sched_switch+0x4f3/frame 0xb77ec28 mi_switch(109) at mi_switch+0x1c4/frame 0xb77ec64 ithread_loop(988ff60,b77ece8) at ithread_loop+0x85/frame 0xb77ecb4 fork_exit(f9b620,988ff60,b77ece8,0,0,...) at fork_exit+0x69/frame 0xb77ecd4 fork_trampoline() at 0xffc033de/frame 0xb77ecd4 --- kthread start Tracing command intr pid 12 tid 100084 td 0x181dd500 exit_tag() at 0xffc033d4 Tracing command idle pid 11 tid 100003 td 0x9df8e00 sched_switch(9df8e00,608) at sched_switch+0x4f3/frame 0x12925c10 mi_switch(608) at mi_switch+0x1c4/frame 0x12925c4c critical_exit_preempt(0) at critical_exit_preempt+0x5c/frame 0x12925c5c sched_idletd(0,12925ce8) at sched_idletd+0x32e/frame 0x12925cb4 fork_exit(10007f0,0,12925ce8,0,0,...) at fork_exit+0x69/frame 0x12925cd4 fork_trampoline() at 0xffc033de/frame 0x12925cd4 --- kthread start Tracing command idle pid 11 tid 100004 td 0x9df8a80 sched_switch(9df8a80,608) at sched_switch+0x4f3/frame 0x1292ac10 mi_switch(608) at mi_switch+0x1c4/frame 0x1292ac4c critical_exit_preempt(0) at critical_exit_preempt+0x5c/frame 0x1292ac5c sched_idletd(0,1292ace8) at sched_idletd+0x32e/frame 0x1292acb4 fork_exit(10007f0,0,1292ace8,0,0,...) at fork_exit+0x69/frame 0x1292acd4 fork_trampoline() at 0xffc033de/frame 0x1292acd4 --- kthread start Tracing command idle pid 11 tid 100005 td 0x9df8700 (CPU 2) cpustop_handler(46,1292fbb8,1292fbac,141602b,30000,...) at cpustop_handler+0x24/frame 0x1292fae0 ipi_nmi_handler(30000,0,1292fb28,fb67e1,9df0d50,...) at ipi_nmi_handler+0x31/frame 0x1292faf0 trap(1292fbb8,8,28,28,17e6f400,...) at trap+0xbb/frame 0x1292fbac calltrap() at 0xffc0316f/frame 0x1292fbac --- trap 0x13, eip = 0x13c2845, esp = 0x1292fbf8, ebp = 0x1292fbf8 --- acpi_cpu_c1(1444cf2,312,3e505c6b,9c605b0,39a9a94f,...) at acpi_cpu_c1+0x5/frame 0x1292fbf8 acpi_cpu_idle(1f605a50,0) at acpi_cpu_idle+0x22f/frame 0x1292fc2c cpu_idle_acpi(1f605a50,0) at cpu_idle_acpi+0x35/frame 0x1292fc40 cpu_idle(0) at cpu_idle+0xa3/frame 0x1292fc5c sched_idletd(0,1292fce8) at sched_idletd+0x32e/frame 0x1292fcb4 fork_exit(10007f0,0,1292fce8,0,0,...) at fork_exit+0x69/frame 0x1292fcd4 fork_trampoline() at 0xffc033de/frame 0x1292fcd4 --- kthread start Tracing command idle pid 11 tid 100006 td 0x9df8380 sched_switch(9df8380,608) at sched_switch+0x4f3/frame 0x12934c10 mi_switch(608) at mi_switch+0x1c4/frame 0x12934c4c critical_exit_preempt(0) at critical_exit_preempt+0x5c/frame 0x12934c5c sched_idletd(0,12934ce8) at sched_idletd+0x32e/frame 0x12934cb4 fork_exit(10007f0,0,12934ce8,0,0,...) at fork_exit+0x69/frame 0x12934cd4 fork_trampoline() at 0xffc033de/frame 0x12934cd4 --- kthread start Tracing command init pid 1 tid 100002 td 0x9df9180 sched_switch(9df9180,104) at sched_switch+0x4f3/frame 0x1291c9b8 mi_switch(104) at mi_switch+0x1c4/frame 0x1291c9f8 sleepq_switch(2d00,6c,0,1da0bd4,9df10a4,...) at sleepq_switch+0xf7/frame 0x1291ca18 sleepq_catch_signals(1291ca8c,fdf10e,9df10a4,6c,200246,...) at sleepq_catch_signals+0x368/frame 0x1291ca44 sleepq_wait_sig(9df10a4,6c) at sleepq_wait_sig+0xe/frame 0x1291ca4c _sleep(9df10a4,9df114c,36c,1542bd4,0,0,0,0,100) at _sleep+0x27e/frame 0x1291ca8c kern_wait6(9df9180,7,0,0,1291cb14,30,0,0) at kern_wait6+0x3b8/frame 0x1291cae8 sys_wait4(9df9180,9df940c) at sys_wait4+0x79/frame 0x1291cc08 syscall(1291cce8,3b,3b,3b,ffbfe918,...) at syscall+0x306/frame 0x1291ccdc Xint0x80_syscall() at 0xffc033c9/frame 0x1291ccdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0xffc01530, esp = 0xffc0bfe8, ebp = 0xffbfe374 --- Tracing command audit pid 10 tid 100001 td 0x9df9500 sched_switch(9df9500,104) at sched_switch+0x4f3/frame 0x12917bd8 mi_switch(104) at mi_switch+0x1c4/frame 0x12917c18 sleepq_switch(1e5c50c,0) at sleepq_switch+0xf7/frame 0x12917c38 _cv_wait(1e5c50c,1e5c4b8) at _cv_wait+0x135/frame 0x12917c70 audit_worker(0,12917ce8) at audit_worker+0x88/frame 0x12917cb4 fork_exit(125fd20,0,12917ce8,0,0,...) at fork_exit+0x69/frame 0x12917cd4 fork_trampoline() at 0xffc033de/frame 0x12917cd4 --- kthread start Tracing command kernel pid 0 tid 100000 td 0x1e4a380 sched_switch(1e4a380,104) at sched_switch+0x4f3/frame 0x22f9c08 mi_switch(104) at mi_switch+0x1c4/frame 0x22f9c48 sleepq_switch(1e4a380,0,1e1fce0,22f9ccc,1e4a380,...) at sleepq_switch+0xf7/frame 0x22f9c68 sleepq_timedwait(1e1fce0,54,246,0,40000,...) at sleepq_timedwait+0x3f/frame 0x22f9c8c _sleep(1e1fce0,0,54,1545a52,fffff470,9,0,0,100) at _sleep+0x22c/frame 0x22f9ccc swapper(0,0,0,1b104fc) at swapper+0x6d/frame 0x22f9d14 mi_startup() at mi_startup+0x2c6/frame 0x22f9d38 btext() at btext+0x5f Tracing command kernel pid 0 tid 100008 td 0x9df3c00 sched_switch(9df3c00,104) at sched_switch+0x4f3/frame 0x1293fbe0 mi_switch(104) at mi_switch+0x1c4/frame 0x1293fc1c sleepq_switch(9d3b600,0) at sleepq_switch+0xf7/frame 0x1293fc3c _sleep(9d3b600,9d3b640,0,14745d3,0,0,0,0,100) at _sleep+0x2f7/frame 0x1293fc7c taskqueue_thread_loop(1e1f4cc,1293fce8) at taskqueue_thread_loop+0x9b/frame 0x1293fcb4 fork_exit(102ad60,1e1f4cc,1293fce8,0,0,...) at fork_exit+0x69/frame 0x1293fcd4 fork_trampoline() at 0xffc033de/frame 0x1293fcd4 --- kthread start Tracing command kernel pid 0 tid 100010 td 0x9df3500 sched_switch(9df3500,104) at sched_switch+0x4f3/frame 0x12949be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12949c28 sleepq_switch(9d36e00,0) at sleepq_switch+0xf7/frame 0x12949c48 msleep_spin_sbt(9d36e00,9d36e40,14745d3,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12949c80 gtaskqueue_thread_loop(990dc04,12949ce8) at gtaskqueue_thread_loop+0xd9/frame 0x12949cb4 fork_exit(10167e0,990dc04,12949ce8,0,0,...) at fork_exit+0x69/frame 0x12949cd4 fork_trampoline() at 0xffc033de/frame 0x12949cd4 --- kthread start Tracing command kernel pid 0 tid 100011 td 0x9df3180 sched_switch(9df3180,104) at sched_switch+0x4f3/frame 0x1294ebe0 mi_switch(104) at mi_switch+0x1c4/frame 0x1294ec1c sleepq_switch(9d3b100,0) at sleepq_switch+0xf7/frame 0x1294ec3c _sleep(9d3b100,9d3b140,0,14745d3,0,0,0,0,100) at _sleep+0x2f7/frame 0x1294ec7c taskqueue_thread_loop(1e4aff0,1294ece8) at taskqueue_thread_loop+0x9b/frame 0x1294ecb4 fork_exit(102ad60,1e4aff0,1294ece8,0,0,...) at fork_exit+0x69/frame 0x1294ecd4 fork_trampoline() at 0xffc033de/frame 0x1294ecd4 --- kthread start Tracing command kernel pid 0 tid 100012 td 0x9df2e00 sched_switch(9df2e00,104) at sched_switch+0x4f3/frame 0x12953be0 mi_switch(104) at mi_switch+0x1c4/frame 0x12953c1c sleepq_switch(9d3ae00,0) at sleepq_switch+0xf7/frame 0x12953c3c _sleep(9d3ae00,9d3ae40,0,14745d3,0,0,0,0,100) at _sleep+0x2f7/frame 0x12953c7c taskqueue_thread_loop(1e5b940,12953ce8) at taskqueue_thread_loop+0x9b/frame 0x12953cb4 fork_exit(102ad60,1e5b940,12953ce8,0,0,...) at fork_exit+0x69/frame 0x12953cd4 fork_trampoline() at 0xffc033de/frame 0x12953cd4 --- kthread start Tracing command kernel pid 0 tid 100014 td 0x9df2700 sched_switch(9df2700,104) at sched_switch+0x4f3/frame 0x12969be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12969c28 sleepq_switch(9d36d00,0) at sleepq_switch+0xf7/frame 0x12969c48 msleep_spin_sbt(9d36d00,9d36d40,14745d3,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12969c80 gtaskqueue_thread_loop(990d004,12969ce8) at gtaskqueue_thread_loop+0xd9/frame 0x12969cb4 fork_exit(10167e0,990d004,12969ce8,0,0,...) at fork_exit+0x69/frame 0x12969cd4 fork_trampoline() at 0xffc033de/frame 0x12969cd4 --- kthread start Tracing command kernel pid 0 tid 100015 td 0x9df2380 sched_switch(9df2380,104) at sched_switch+0x4f3/frame 0x1296ebe8 mi_switch(104) at mi_switch+0x1c4/frame 0x1296ec28 sleepq_switch(9d36c00,0) at sleepq_switch+0xf7/frame 0x1296ec48 msleep_spin_sbt(9d36c00,9d36c40,14745d3,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x1296ec80 gtaskqueue_thread_loop(990d014,1296ece8) at gtaskqueue_thread_loop+0xd9/frame 0x1296ecb4 fork_exit(10167e0,990d014,1296ece8,0,0,...) at fork_exit+0x69/frame 0x1296ecd4 fork_trampoline() at 0xffc033de/frame 0x1296ecd4 --- kthread start Tracing command kernel pid 0 tid 100016 td 0x9df2000 sched_switch(9df2000,104) at sched_switch+0x4f3/frame 0x12973be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12973c28 sleepq_switch(9d36b00,0) at sleepq_switch+0xf7/frame 0x12973c48 msleep_spin_sbt(9d36b00,9d36b40,14745d3,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12973c80 gtaskqueue_thread_loop(990d024,12973ce8) at gtaskqueue_thread_loop+0xd9/frame 0x12973cb4 fork_exit(10167e0,990d024,12973ce8,0,0,...) at fork_exit+0x69/frame 0x12973cd4 fork_trampoline() at 0xffc033de/frame 0x12973cd4 --- kthread start Tracing command kernel pid 0 tid 100017 td 0x991de00 sched_switch(991de00,104) at sched_switch+0x4f3/frame 0x12978be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12978c28 sleepq_switch(9d36a00,0) at sleepq_switch+0xf7/frame 0x12978c48 msleep_spin_sbt(9d36a00,9d36a40,14745d3,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12978c80 gtaskqueue_thread_loop(990d034,12978ce8) at gtaskqueue_thread_loop+0xd9/frame 0x12978cb4 fork_exit(10167e0,990d034,12978ce8,0,0,...) at fork_exit+0x69/frame 0x12978cd4 fork_trampoline() at 0xffc033de/frame 0x12978cd4 --- kthread start Tracing command kernel pid 0 tid 100018 td 0x991da80 sched_switch(991da80,104) at sched_switch+0x4f3/frame 0x1297dbe8 mi_switch(104) at mi_switch+0x1c4/frame 0x1297dc28 sleepq_switch(9d36900,0) at sleepq_switch+0xf7/frame 0x1297dc48 msleep_spin_sbt(9d36900,9d36940,14745d3,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x1297dc80 gtaskqueue_thread_loop(990d804,1297dce8) at gtaskqueue_thread_loop+0xd9/frame 0x1297dcb4 fork_exit(10167e0,990d804,1297dce8,0,0,...) at fork_exit+0x69/frame 0x1297dcd4 fork_trampoline() at 0xffc033de/frame 0x1297dcd4 --- kthread start Tracing command kernel pid 0 tid 100019 td 0x991d700 sched_switch(991d700,104) at sched_switch+0x4f3/frame 0x12982be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12982c28 sleepq_switch(9d36800,0) at sleepq_switch+0xf7/frame 0x12982c48 msleep_spin_sbt(9d36800,9d36840,14745d3,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12982c80 gtaskqueue_thread_loop(990d814,12982ce8) at gtaskqueue_thread_loop+0xd9/frame 0x12982cb4 fork_exit(10167e0,990d814,12982ce8,0,0,...) at fork_exit+0x69/frame 0x12982cd4 fork_trampoline() at 0xffc033de/frame 0x12982cd4 --- kthread start Tracing command kernel pid 0 tid 100020 td 0x991d380 sched_switch(991d380,104) at sched_switch+0x4f3/frame 0x12987be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12987c28 sleepq_switch(9d36700,0) at sleepq_switch+0xf7/frame 0x12987c48 msleep_spin_sbt(9d36700,9d36740,14745d3,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12987c80 gtaskqueue_thread_loop(990d824,12987ce8) at gtaskqueue_thread_loop+0xd9/frame 0x12987cb4 fork_exit(10167e0,990d824,12987ce8,0,0,...) at fork_exit+0x69/frame 0x12987cd4 fork_trampoline() at 0xffc033de/frame 0x12987cd4 --- kthread start Tracing command kernel pid 0 tid 100021 td 0x991d000 sched_switch(991d000,104) at sched_switch+0x4f3/frame 0x1298cbe8 mi_switch(104) at mi_switch+0x1c4/frame 0x1298cc28 sleepq_switch(9d36600,0) at sleepq_switch+0xf7/frame 0x1298cc48 msleep_spin_sbt(9d36600,9d36640,14745d3,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x1298cc80 gtaskqueue_thread_loop(990d834,1298cce8) at gtaskqueue_thread_loop+0xd9/frame 0x1298ccb4 fork_exit(10167e0,990d834,1298cce8,0,0,...) at fork_exit+0x69/frame 0x1298ccd4 fork_trampoline() at 0xffc033de/frame 0x1298ccd4 --- kthread start Tracing command kernel pid 0 tid 100022 td 0x991cc00 sched_switch(991cc00,104) at sched_switch+0x4f3/frame 0x12991be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12991c28 sleepq_switch(9d36500,0) at sleepq_switch+0xf7/frame 0x12991c48 msleep_spin_sbt(9d36500,9d36540,14745d3,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12991c80 gtaskqueue_thread_loop(990d404,12991ce8) at gtaskqueue_thread_loop+0xd9/frame 0x12991cb4 fork_exit(10167e0,990d404,12991ce8,0,0,...) at fork_exit+0x69/frame 0x12991cd4 fork_trampoline() at 0xffc033de/frame 0x12991cd4 --- kthread start Tracing command kernel pid 0 tid 100029 td 0x991b380 sched_switch(991b380,104) at sched_switch+0x4f3/frame 0x129c0be0 mi_switch(104) at mi_switch+0x1c4/frame 0x129c0c1c sleepq_switch(9d39a00,0) at sleepq_switch+0xf7/frame 0x129c0c3c _sleep(9d39a00,9d39a40,0,14745d3,0,0,0,0,100) at _sleep+0x2f7/frame 0x129c0c7c taskqueue_thread_loop(1d9d554,129c0ce8) at taskqueue_thread_loop+0x9b/frame 0x129c0cb4 fork_exit(102ad60,1d9d554,129c0ce8,0,0,...) at fork_exit+0x69/frame 0x129c0cd4 fork_trampoline() at 0xffc033de/frame 0x129c0cd4 --- kthread start Tracing command kernel pid 0 tid 100034 td 0x9987380 sched_switch(9987380,104) at sched_switch+0x4f3/frame 0x129d9be0 mi_switch(104) at mi_switch+0x1c4/frame 0x129d9c1c sleepq_switch(9d39400,0) at sleepq_switch+0xf7/frame 0x129d9c3c _sleep(9d39400,9d39440,0,14745d3,0,0,0,0,100) at _sleep+0x2f7/frame 0x129d9c7c taskqueue_thread_loop(1e10bcc,129d9ce8) at taskqueue_thread_loop+0x9b/frame 0x129d9cb4 fork_exit(102ad60,1e10bcc,129d9ce8,0,0,...) at fork_exit+0x69/frame 0x129d9cd4 fork_trampoline() at 0xffc033de/frame 0x129d9cd4 --- kthread start Tracing command kernel pid 0 tid 100035 td 0x9987000 sched_switch(9987000,104) at sched_switch+0x4f3/frame 0x129debe0 mi_switch(104) at mi_switch+0x1c4/frame 0x129dec1c sleepq_switch(9d39400,0) at sleepq_switch+0xf7/frame 0x129dec3c _sleep(9d39400,9d39440,0,14745d3,0,0,0,0,100) at _sleep+0x2f7/frame 0x129dec7c taskqueue_thread_loop(1e10bcc,129dece8) at taskqueue_thread_loop+0x9b/frame 0x129decb4 fork_exit(102ad60,1e10bcc,129dece8,0,0,...) at fork_exit+0x69/frame 0x129decd4 fork_trampoline() at 0xffc033de/frame 0x129decd4 --- kthread start Tracing command kernel pid 0 tid 100036 td 0x9986c00 sched_switch(9986c00,104) at sched_switch+0x4f3/frame 0x129e3be0 mi_switch(104) at mi_switch+0x1c4/frame 0x129e3c1c sleepq_switch(9d39400,0) at sleepq_switch+0xf7/frame 0x129e3c3c _sleep(9d39400,9d39440,0,14745d3,0,0,0,0,100) at _sleep+0x2f7/frame 0x129e3c7c taskqueue_thread_loop(1e10bcc,129e3ce8) at taskqueue_thread_loop+0x9b/frame 0x129e3cb4 fork_exit(102ad60,1e10bcc,129e3ce8,0,0,...) at fork_exit+0x69/frame 0x129e3cd4 fork_trampoline() at 0xffc033de/frame 0x129e3cd4 --- kthread start Tracing command kernel pid 0 tid 100037 td 0x9986880 sched_switch(9986880,104) at sched_switch+0x4f3/frame 0x129e8be0 mi_switch(104) at mi_switch+0x1c4/frame 0x129e8c1c sleepq_switch(9d39400,0) at sleepq_switch+0xf7/frame 0x129e8c3c _sleep(9d39400,9d39440,0,14745d3,0,0,0,0,100) at _sleep+0x2f7/frame 0x129e8c7c taskqueue_thread_loop(1e10bcc,129e8ce8) at taskqueue_thread_loop+0x9b/frame 0x129e8cb4 fork_exit(102ad60,1e10bcc,129e8ce8,0,0,...) at fork_exit+0x69/frame 0x129e8cd4 fork_trampoline() at 0xffc033de/frame 0x129e8cd4 --- kthread start Tracing command kernel pid 0 tid 100085 td 0x181dd180 sched_switch(181dd180,104) at sched_switch+0x4f3/frame 0xb78dbe8 mi_switch(104) at mi_switch+0x1c4/frame 0xb78dc28 sleepq_switch(17ee5400,0) at sleepq_switch+0xf7/frame 0xb78dc48 msleep_spin_sbt(17ee5400,17ee5440,14745d3,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0xb78dc80 taskqueue_thread_loop(1e19ea4,b78dce8) at taskqueue_thread_loop+0xd9/frame 0xb78dcb4 fork_exit(102ad60,1e19ea4,b78dce8,0,0,...) at fork_exit+0x69/frame 0xb78dcd4 fork_trampoline() at 0xffc033de/frame 0xb78dcd4 --- kthread start Tracing command kernel pid 0 tid 100091 td 0x1889f000 sched_switch(1889f000,104) at sched_switch+0x4f3/frame 0xb7b0be8 mi_switch(104) at mi_switch+0x1c4/frame 0xb7b0c28 sleepq_switch(183a9d00,0) at sleepq_switch+0xf7/frame 0xb7b0c48 msleep_spin_sbt(183a9d00,183a9d40,14745d3,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0xb7b0c80 taskqueue_thread_loop(1e200cc,b7b0ce8) at taskqueue_thread_loop+0xd9/frame 0xb7b0cb4 fork_exit(102ad60,1e200cc,b7b0ce8,0,0,...) at fork_exit+0x69/frame 0xb7b0cd4 fork_trampoline() at 0xffc033de/frame 0xb7b0cd4 --- kthread start Tracing command kernel pid 0 tid 100092 td 0x18892c00 sched_switch(18892c00,104) at sched_switch+0x4f3/frame 0xb7b5be8 mi_switch(104) at mi_switch+0x1c4/frame 0xb7b5c28 sleepq_switch(183a9d00,0) at sleepq_switch+0xf7/frame 0xb7b5c48 msleep_spin_sbt(183a9d00,183a9d40,14745d3,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0xb7b5c80 taskqueue_thread_loop(1e200cc,b7b5ce8) at taskqueue_thread_loop+0xd9/frame 0xb7b5cb4 fork_exit(102ad60,1e200cc,b7b5ce8,0,0,...) at fork_exit+0x69/frame 0xb7b5cd4 fork_trampoline() at 0xffc033de/frame 0xb7b5cd4 --- kthread start Tracing command kernel pid 0 tid 100093 td 0x18892880 sched_switch(18892880,104) at sched_switch+0x4f3/frame 0xb7babe8 mi_switch(104) at mi_switch+0x1c4/frame 0xb7bac28 sleepq_switch(183a9d00,0) at sleepq_switch+0xf7/frame 0xb7bac48 msleep_spin_sbt(183a9d00,183a9d40,14745d3,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0xb7bac80 taskqueue_thread_loop(1e200cc,b7bace8) at taskqueue_thread_loop+0xd9/frame 0xb7bacb4 fork_exit(102ad60,1e200cc,b7bace8,0,0,...) at fork_exit+0x69/frame 0xb7bacd4 fork_trampoline() at 0xffc033de/frame 0xb7bacd4 --- kthread start Tracing command kernel pid 0 tid 100095 td 0x18892180 sched_switch(18892180,104) at sched_switch+0x4f3/frame 0xb7c4be0 mi_switch(104) at mi_switch+0x1c4/frame 0xb7c4c1c sleepq_switch(9d38900,0) at sleepq_switch+0xf7/frame 0xb7c4c3c _sleep(9d38900,9d38940,0,14745d3,0,0,0,0,100) at _sleep+0x2f7/frame 0xb7c4c7c taskqueue_thread_loop(1b2d4ec,b7c4ce8) at taskqueue_thread_loop+0x9b/frame 0xb7c4cb4 fork_exit(102ad60,1b2d4ec,b7c4ce8,0,0,...) at fork_exit+0x69/frame 0xb7c4cd4 fork_trampoline() at 0xffc033de/frame 0xb7c4cd4 --- kthread start db:0:pho> dump Physical memory: 3428 MB Dumping 321 MB: 306 290 274 258 242 226 210 194 178 162 146 130 114 98 82 66 50 34 18 2 Dump complete db> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 3 (kgdb) bt #0 0x00fd4949 in doadump (textdump=0x0) at ../../../kern/kern_shutdown.c:392 #1 0x00a05154 in db_dump (dummy=0x1017c05, dummy2=0x0, dummy3=0xffffffff, dummy4=0x289ed63c "") at ../../../ddb/db_command.c:575 #2 0x00a04f1a in db_command (cmd_table=, dopager=) at ../../../ddb/db_command.c:482 #3 0x00a09956 in db_script_exec (scriptname=, warnifnotfound=) at ../../../ddb/db_script.c:304 #4 0x00a04f1a in db_command (cmd_table=, dopager=) at ../../../ddb/db_command.c:482 #5 0x00a04c8c in db_command_loop () at ../../../ddb/db_command.c:535 #6 0x00a07d16 in db_trap (type=0x3, code=0x0) at ../../../ddb/db_main.c:252 #7 0x01018369 in kdb_trap (type=0x3, code=0x0, tf=0x289ed934) at ../../../kern/subr_kdb.c:689 #8 0x014162fa in trap (frame=0x289ed934) at ../../../i386/i386/trap.c:668 #9 0xffc0316f in ?? () #10 0x01017c05 in kdb_enter (why=0x1521cf1 "panic", msg=0x1521cf1 "panic") at include/cpufunc.h:62 #11 0x00fd4af7 in vpanic (fmt=0x1461fb6 "%s", ap=0x289ed9c4 "Á\tO\001&ÛJ\001") at ../../../kern/kern_shutdown.c:899 #12 0x00fd4914 in panic (fmt=0x1461fb6 "%s") at ../../../kern/kern_shutdown.c:836 #13 0x01416c07 in trap_fatal (frame=, eva=) at ../../../i386/i386/trap.c:963 #14 0x01416c82 in trap_pfault (frame=, usermode=, eva=0x1833f004, signo=0x0, ucode=0x0) at include/pcpu_aux.h:54 #15 0x01416318 in trap (frame=0x289edb10) at ../../../i386/i386/trap.c:475 #16 0xffc0316f in ?? () #17 0x01046228 in sem_remove (semidx=0x138, cred=) at ../../../kern/sysv_sem.c:582 #18 0x01045bd2 in kern_semctl (td=0x27069a80, semid=0x10000, semnum=0x0, cmd=0x0, arg=0x289edbc8, rval=0x289edbd4) at ../../../kern/sysv_sem.c:768 #19 0x01045300 in sys___semctl (td=0x27069a80, uap=0x27069d0c) at ../../../kern/sysv_sem.c:668 #20 0x01417376 in syscall (frame=0x289edce8) at subr_syscall.c:151 #21 0xffc033c9 in ?? () #22 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal (kgdb) f 17 #17 0x01046228 in sem_remove (semidx=0x138, cred=) at ../../../kern/sysv_sem.c:582 582 sem[i] = sem[i + semakptr->u.sem_nsems]; (kgdb) l 577 if ((sema[i].u.sem_perm.mode & SEM_ALLOC) && 578 sema[i].u.__sem_base > semakptr->u.__sem_base) 579 mtx_lock_flags(&sema_mtx[i], LOP_DUPOK); 580 } 581 for (i = semakptr->u.__sem_base - sem; i < semtot; i++) 582 sem[i] = sem[i + semakptr->u.sem_nsems]; 583 for (i = 0; i < seminfo.semmni; i++) { 584 if ((sema[i].u.sem_perm.mode & SEM_ALLOC) && 585 sema[i].u.__sem_base > semakptr->u.__sem_base) { 586 sema[i].u.__sem_base -= semakptr->u.sem_nsems; (kgdb) info loc semakptr = i = 0xb8 (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/kern/kern_tc.c =================================================================== --- /usr/src/sys/kern/kern_tc.c (revision 357822) +++ /usr/src/sys/kern/kern_tc.c (working copy) @@ -72,6 +72,7 @@ struct timehands { struct timecounter *th_counter; int64_t th_adjustment; uint64_t th_scale; + u_int th_large_delta; u_int th_offset_count; struct bintime th_offset; struct bintime th_bintime; @@ -87,6 +88,7 @@ static struct timehands ths[16] = { [0] = { .th_counter = &dummy_timecounter, .th_scale = (uint64_t)-1 / 1000000, + .th_large_delta = 1000000, .th_offset = { .sec = 1 }, .th_generation = 1, }, @@ -202,23 +204,75 @@ tc_delta(struct timehands *th) * the comment in for a description of these 12 functions. */ -#ifdef FFCLOCK -void -fbclock_binuptime(struct bintime *bt) +static __inline void +bintime_off(struct bintime *bt, u_int off) { struct timehands *th; - unsigned int gen; + struct bintime *btp; + uint64_t scale, x; + u_int delta, gen, large_delta; do { th = timehands; gen = atomic_load_acq_int(&th->th_generation); - *bt = th->th_offset; - bintime_addx(bt, th->th_scale * tc_delta(th)); + btp = (struct bintime *)((vm_offset_t)th + off); + *bt = *btp; + scale = th->th_scale; + delta = tc_delta(th); + large_delta = th->th_large_delta; atomic_thread_fence_acq(); } while (gen == 0 || gen != th->th_generation); + + if (__predict_false(delta >= large_delta)) { + /* Avoid overflow for scale * delta. */ + x = (scale >> 32) * delta; + bt->sec += x >> 32; + bintime_addx(bt, x << 32); + bintime_addx(bt, (scale & 0xffffffff) * delta); + } else { + bintime_addx(bt, scale * delta); + } } +#define GETTHBINTIME(dst, member) \ +do { \ + _Static_assert(_Generic(((struct timehands *)NULL)->member, \ + struct bintime: 1, default: 0) == 1, \ + "struct timehands member is not of struct bintime type"); \ + bintime_off(dst, __offsetof(struct timehands, member)); \ +} while (0) +static __inline void +getthmember(void *out, size_t out_size, u_int off) +{ + struct timehands *th; + u_int gen; + + do { + th = timehands; + gen = atomic_load_acq_int(&th->th_generation); + memcpy(out, (char *)th + off, out_size); + atomic_thread_fence_acq(); + } while (gen == 0 || gen != th->th_generation); +} +#define GETTHMEMBER(dst, member) \ +do { \ + _Static_assert(_Generic(*dst, \ + __typeof(((struct timehands *)NULL)->member): 1, \ + default: 0) == 1, \ + "*dst and struct timehands member have different types"); \ + getthmember(dst, sizeof(*dst), __offsetof(struct timehands, \ + member)); \ +} while (0) + +#ifdef FFCLOCK void +fbclock_binuptime(struct bintime *bt) +{ + + GETTHBINTIME(bt, th_offset); +} + +void fbclock_nanouptime(struct timespec *tsp) { struct bintime bt; @@ -239,16 +293,8 @@ fbclock_microuptime(struct timeval *tvp) void fbclock_bintime(struct bintime *bt) { - struct timehands *th; - unsigned int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *bt = th->th_bintime; - bintime_addx(bt, th->th_scale * tc_delta(th)); - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHBINTIME(bt, th_bintime); } void @@ -272,100 +318,55 @@ fbclock_microtime(struct timeval *tvp) void fbclock_getbinuptime(struct bintime *bt) { - struct timehands *th; - unsigned int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *bt = th->th_offset; - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(bt, th_offset); } void fbclock_getnanouptime(struct timespec *tsp) { - struct timehands *th; - unsigned int gen; + struct bintime bt; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - bintime2timespec(&th->th_offset, tsp); - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(&bt, th_offset); + bintime2timespec(&bt, tsp); } void fbclock_getmicrouptime(struct timeval *tvp) { - struct timehands *th; - unsigned int gen; + struct bintime bt; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - bintime2timeval(&th->th_offset, tvp); - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(&bt, th_offset); + bintime2timeval(&bt, tvp); } void fbclock_getbintime(struct bintime *bt) { - struct timehands *th; - unsigned int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *bt = th->th_bintime; - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(bt, th_bintime); } void fbclock_getnanotime(struct timespec *tsp) { - struct timehands *th; - unsigned int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *tsp = th->th_nanotime; - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(tsp, th_nanotime); } void fbclock_getmicrotime(struct timeval *tvp) { - struct timehands *th; - unsigned int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *tvp = th->th_microtime; - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(tvp, th_microtime); } #else /* !FFCLOCK */ + void binuptime(struct bintime *bt) { - struct timehands *th; - u_int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *bt = th->th_offset; - bintime_addx(bt, th->th_scale * tc_delta(th)); - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHBINTIME(bt, th_offset); } void @@ -389,16 +390,8 @@ microuptime(struct timeval *tvp) void bintime(struct bintime *bt) { - struct timehands *th; - u_int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *bt = th->th_bintime; - bintime_addx(bt, th->th_scale * tc_delta(th)); - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHBINTIME(bt, th_bintime); } void @@ -422,85 +415,47 @@ microtime(struct timeval *tvp) void getbinuptime(struct bintime *bt) { - struct timehands *th; - u_int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *bt = th->th_offset; - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(bt, th_offset); } void getnanouptime(struct timespec *tsp) { - struct timehands *th; - u_int gen; + struct bintime bt; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - bintime2timespec(&th->th_offset, tsp); - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(&bt, th_offset); + bintime2timespec(&bt, tsp); } void getmicrouptime(struct timeval *tvp) { - struct timehands *th; - u_int gen; + struct bintime bt; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - bintime2timeval(&th->th_offset, tvp); - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(&bt, th_offset); + bintime2timeval(&bt, tvp); } void getbintime(struct bintime *bt) { - struct timehands *th; - u_int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *bt = th->th_bintime; - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(bt, th_bintime); } void getnanotime(struct timespec *tsp) { - struct timehands *th; - u_int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *tsp = th->th_nanotime; - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(tsp, th_nanotime); } void getmicrotime(struct timeval *tvp) { - struct timehands *th; - u_int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *tvp = th->th_microtime; - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(tvp, th_microtime); } #endif /* FFCLOCK */ @@ -516,15 +471,8 @@ getboottime(struct timeval *boottime) void getboottimebin(struct bintime *boottimebin) { - struct timehands *th; - u_int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *boottimebin = th->th_boottime; - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(boottimebin, th_boottime); } #ifdef FFCLOCK @@ -1040,15 +988,8 @@ getmicrotime(struct timeval *tvp) void dtrace_getnanotime(struct timespec *tsp) { - struct timehands *th; - u_int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *tsp = th->th_nanotime; - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(tsp, th_nanotime); } /* @@ -1466,6 +1407,7 @@ tc_windup(struct bintime *new_boottimebin) scale += (th->th_adjustment / 1024) * 2199; scale /= th->th_counter->tc_frequency; th->th_scale = scale * 2; + th->th_large_delta = MIN(((uint64_t)1 << 63) / scale, UINT_MAX); /* * Now that the struct timehands is again consistent, set the new -- Test scenario: misc/syscall4.sh (syscall fuzz)