GENERIC HEAD from 2020-11-03 14:44:33 UTC, r367298M, vmcore.1189 GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb ---<>--- Copyright (c) 1992-2020 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 13.0-CURRENT #1 r367298M: Wed Nov 4 07:34:59 CET 2020 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 FreeBSD clang version 11.0.0 (git@github.com:llvm/llvm-project.git llvmorg-11.0.0-0-g176249bd673) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. VT(vga): resolution 640x480 CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz 686-class CPU) Origin="AuthenticAMD" Id=0x100f23 Family=0x10 Model=0x2 Stepping=3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff SVM: NP,NAsids=64 TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 3522457600 (3359 MB) : Trying to mount root from ufs:/dev/ada0s1a [rw]... Root mount waiting for: CAM usbus0 usbus1WARNING: WITNESS option enabled, expect reduced performance. usbus2WARNING: DIAGNOSTIC option enabled, expect reduced performance. usbus3ugen5.1: at usbus5 uhub3: on usbus4 Expensive timeout(9) function: 0xa3ee70(0x1801b000) 0.009487518 s uhub4ugen1.1: at usbus1 on usbus5 uhub5 on usbus1 usbus4uhub4: on usbus5 (aprobe0:ahcich0:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00 usbus5uhub5: on usbus1 (aprobe0:ahcich0:0:15:0): CAM status: Command timeout (aprobe0:ahcich0:0:15:0): Error 5, Retries exhausted ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ATA8-ACS SATA 2.x device ada0: Serial Number 9VMJ99G5 ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 476940MB (976773168 512 byte sectors) uhub1: 2 ports with 2 removable, self powered uhub0: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered Root mount waiting for: usbus5 Root mount waiting for: usbus5 Root mount waiting for: usbus5 Root mount waiting for: usbus5 uhub4: 10 ports with 10 removable, self powered mountroot: waiting for device /dev/ada0s1a... WARNING: / was not properly dismounted Dual Console: Serial Primary, Video Secondary Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. warning: total configured swap (4194304 pages) exceeds maximum recommended amount (2009952 pages). warning: increase kern.maxswzone or reduce amount of swap. Starting file system checks: /dev/ada0s1a: 8096 files, 405725 used, 607290 free (2090 frags, 75650 blocks, 0.2% fragmentation) /dev/ada0s1f: 18353 files, 236755 used, 776260 free (12060 frags, 95525 blocks, 1.2% fragmentation) /dev/label/tmp: INCORRECT BLOCK COUNT I=1587 (20 should be 0) (CORRECTED) /dev/ada0s1d: 490015 files, 8176247 used, 1977952 free (390072 frags, 198485 blocks, 3.8% fragmentation) ** SU+J Recovering /dev/ada0s1e ** Reading 33554432 byte journal from inode 6. ** Building recovery table. ** Resolving unreferenced inode list. ** Processing journal entries. ** 1 journal records in 512 bytes for 6.25% utilization ** Freed 0 inodes (0 dirs) 0 blocks, and 0 frags. ***** FILE SYSTEM MARKED CLEAN ***** /dev/label/tmp: LINK COUNT DIR I=2 OWNER=root MODE=41777 /dev/label/tmp: SIZE=555008 MTIME=Nov 6 10:39 2020 COUNT 1047 SHOULD BE 1045 (ADJUSTED) /dev/label/tmp: UNREF FILE I=1587 OWNER=root MODE=100755 /dev/label/tmp: SIZE=0 MTIME=Nov 6 10:39 2020 (CLEARED) /dev/label/tmp: UNREF DIR I=11257856 OWNER=root MODE=40777 /dev/label/tmp: SIZE=512 MTIME=Nov 6 10:39 2020 (CLEARED) /dev/label/tmp: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED) /dev/label/tmp: SUMMARY INFORMATION BAD (SALVAGED) /dev/label/tmp: BLK(S) MISSING IN BIT MAPS (SALVAGED) /dev/label/tmp: 725805 files, 79205400 used, 39108581 free (24509 frags, 4885509 blocks, 0.0% fragmentation) Mounting local filesystems:. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/arduino/lib /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/e2fsprogs /usr/local/lib/gcc7 /usr/local/lib/gcc8 /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/perl5/5.28/mach/CORE /usr/local/lib/pth /usr/local/lib/qt4 /usr/local/lib/qt5 /usr/local/llvm37/lib /usr/local/llvm39/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout ldconfig: aout support is deprecated Setting hostname: x4.osted.lan. Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy: . lo0: link state changed to UP re0: link state changed to DOWN Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=8209b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 media: Ethernet autoselect (none) status: no carrier nd6 options=29 Starting devd. re0: link state changed to UP add host 127.0.0.1: gateway lo0 fib 0: route already in table add net default: gateway 192.168.1.254 add host ::1: gateway lo0 fib 0: route already in table add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Starting nfsuserd. Creating and/or trimming log files. Starting nfscbd. Recovering vi editor sessions:. Updating motd:. Clearing /tmp (X related). Starting syslogd. Starting watchdogd. Nov 6 12:06:09 x4 savecore[600]: reboot Local package initialization: backup. Starting rpcbind. NFS access cache time=60 Starting mountd. Starting nfsd. Starting statd. Starting lockd. Mounting late filesystems:. Starting ntpd. Starting powerd. Starting default mousedmoused: unable to open /dev/psm0: No such file or directory . Starting smartd. Nov 6 12:06:52 x4 smartd[736]: Device: /dev/ada0, WARNING: A firmware update for this drive may be available, Nov 6 12:06:52 x4 smartd[736]: see the following Seagate web pages: Nov 6 12:06:52 x4 smartd[736]: http://knowledge.seagate.com/articles/en_US/FAQ/207931en Nov 6 12:06:52 x4 smartd[736]: http://knowledge.seagate.com/articles/en_US/FAQ/213891en Starting inetd. Performing sanity check on sshd configuration. Starting sshd. Starting cron. Configuring vt: keymap blanktime. Starting sendmail_submit. Starting sendmail_msp_queue. Fri Nov 6 12:06 FreeBSD/i386 (x4.osted.lan) (ttyu0) login: Nov 6 12:14:26 x4 su[1007]: pho to root on /dev/pts/0 20201106 12:16:03 all (1/54): aslr.sh witness_lock_list_get: witness exhausted Nov 6 12:17:53 x4 kernel: pid 2526 (swap), jid 0, uid 0, was killed: out of swap space 20201106 12:21:46 all (2/54): truncate9.sh stress2: freework leak: 1/8. stress2: newblk leak: 3172/3178. stress2: indirdep leak: 3/3 20201106 12:22:17 all (3/54): umount2.sh stress2: mount leak: 1/98 20201106 12:27:53 all (4/54): msdos6.sh stress2: mount leak: 1/99 20201106 12:33:34 all (5/54): sem_post.sh Nov 6 12:34:00 x4 kernel: pid 1038 (swap), jid 0, uid 0, was killed: out of swap space Nov 6 12:34:12 x4 kernel: pid 1040 (swap), jid 0, uid 0, was killed: out of swap space 20201106 12:39:38 all (6/54): geli.sh GEOM_ELI: Device md10.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: software Nov 6 12:40:03 x4 kernel: pid 20532 (swap), jid 0, uid 1004, was killed: out of swap space Nov 6 12:40:23 x4 kernel: pid 20522 (swap), jid 0, uid 1004, was killed: out of swap space Nov 6 12:40:33 x4 kernel: pid 20520 (swap), jid 0, uid 1004, was killed: out of swap space Nov 6 12:40:46 x4 kernel: pid 20517 (swap), jid 0, uid 1004, was killed: out of swap space Nov 6 12:41:00 x4 kernel: pid 20506 (swap), jid 0, uid 1004, was killed: out of swap space Nov 6 12:41:21 x4 kernel: pid 20515 (swap), jid 0, uid 1004, was killed: out of swap space Nov 6 12:41:45 x4 kernel: pid 20516 (swap), jid 0, uid 1004, was killed: out of swap space Nov 6 12:44:17 x4 kernel: pid 26029 (swap), jid 0, uid 1004, was killed: out of swap space Nov 6 12:44:30 x4 kernel: pid 26092 (swap), jid 0, uid 1004, was killed: out of swap space Nov 6 12:44:40 x4 kernel: pid 26051 (swap), jid 0, uid 1004, was killed: out of swap space Nov 6 12:44:51 x4 kernel: pid 26039 (swap), jid 0, uid 1004, was killed: out of swap space Nov 6 12:45:07 x4 kernel: pid 26083 (swap), jid 0, uid 1004, was killed: out of swap space GEOM_ELI: md10 has been killed. GEOM_ELI: Device md10.eli destroyed. stress2: newblk leak: 208/214. 20201106 12:46:48 all (7/54): pmc3.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/16/48/0x1ff stress2: mount leak: 2/101 20201106 12:53:10 all (8/54): suj3.sh GEOM_ELI: Device md10.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: software Nov 6 12:54:45 x4 kernel: pid 62122 (rw), uid 1004 inumber 16 on /mnt: filesystem full GEOM_ELI: md10 has been killed. GEOM_ELI: Device md10.eli destroyed. stress2: mount leak: 1/102 20201106 12:59:35 all (9/54): suj29.sh Nov 6 12:59:52 x4 kernel: pid 75289 (swap), jid 0, uid 1004, was killed: out of swap space Nov 6 13:00:04 x4 kernel: pid 75298 (swap), jid 0, uid 1004, was killed: out of swap space Nov 6 13:01:29 x4 kernel: pid 75286 (swap), jid 0, uid 1004, was killed: out of swap space Nov 6 13:01:55 x4 kernel: pid 77498 (swap), jid 0, uid 1004, was killed: out of swap space Nov 6 13:02:06 x4 kernel: pid 77480 (swap), jid 0, uid 1004, was killed: out of swap space Nov 6 13:04:00 x4 kernel: pid 79673 (swap), jid 0, uid 1004, was killed: out of swap space Nov 6 13:04:11 x4 kernel: pid 79703 (swap), jid 0, uid 1004, was killed: out of swap space stress2: mount leak: 1/103 20201106 13:06:03 all (10/54): crossmp11.sh Nov 6 13:06:29 x4 kernel: pid 85422 (rw), uid 1004 inumber 49162 on /mnt10: filesystem full Nov 6 13:06:37 x4 kernel: pid 85417 (rw), uid 1004 inumber 49176 on /mnt10: filesystem full Nov 6 13:06:42 x4 kernel: pid 85409 (rw), uid 1004 inumber 49169 on /mnt10: filesystem full Nov 6 13:06:55 x4 kernel: pid 85415 (rw), uid 1004 inumber 49166 on /mnt10: filesystem full Nov 6 13:06:56 x4 kernel: pid 85383 (rw), uid 1004 inumber 73741 on /mnt11: filesystem full Nov 6 13:07:00 x4 kernel: pid 85393 (rw), uid 1004 inumber 73735 on /mnt13: filesystem full Nov 6 13:07:14 x4 kernel: pid 85361 (mkdir), uid 1004 inumber 9 on /mnt13: filesystem full Nov 6 13:08:28 x4 kernel: pid 86984 (mkdir), uid 1004 inumber 27747 on /mnt13: filesystem full Nov 6 13:09:54 x4 kernel: pid 87562 (rw), uid 1004 inumber 49168 on /mnt12: filesystem full Nov 6 13:10:13 x4 kernel: pid 87489 (mkdir), uid 1004 inumber 74028 on /mnt11: filesystem full Nov 6 13:10:18 x4 kernel: pid 87476 (creat), uid 1004 inumber 73741 on /mnt11: filesystem full Nov 6 13:10:24 x4 kernel: pid 89688 (mkdir), uid 1004 inumber 2165 on /mnt10: filesystem full Nov 6 13:10:26 x4 kernel: pid 95652 (rw), uid 1004 inumber 49158 on /mnt12: filesystem full Nov 6 13:10:48 x4 kernel: pid 95822 (rw), uid 1004 inumber 49160 on /mnt13: filesystem full Nov 6 13:11:01 x4 kernel: pid 96121 (rw), uid 1004 inumber 24584 on /mnt11: filesystem full Nov 6 13:11:05 x4 kernel: pid 95645 (rw), uid 1004 inumber 49155 on /mnt12: filesystem full stress2: mount leak: 1/104 20201106 13:13:00 all (11/54): zfs6.sh 20201106 13:13:04 all (12/54): mkfifo2d.sh FreeBSD/i386 (x4.osted.lan) (ttyu0) login: root KDB: enter: Break to debugger [ thread pid 829 tid 100201 ] Stopped at kdb_alt_break_internal+0x13b: movl $0,kdb_why db> ps pid ppid pgrp uid state wmesg wchan cmd 7141 786 7141 0 Ds devfsmo 0x18805bd4 sshd 7140 808 808 25 D devfsmo 0x18805bd4 sendmail 7139 802 802 0 D devfsmo 0x18805bd4 sendmail 7138 7137 584 0 D ufs 0x22c441b4 ls 7137 584 584 0 S wait 0x25db4344 sh 7136 1377 1356 0 S piperd 0x34eea1a8 sh 7135 1377 1356 0 R sed 7134 1377 1356 0 R CPU 0 sed 7132 1237 1083 0 RL+ umount 1489 0 0 0 DL mdwait 0x3881a000 [md10] 1237 1235 1083 0 S+ wait 0x2999b054 sh 1236 1083 1083 0 S+ piperd 0x3df6e1a8 tee 1235 1083 1083 0 S+ wait 0x2bb7a000 sh 6180 1 6144 1001 S+ select 0x1880b9a4 tail 6144 2309 6144 1001 S+ kqread 0x9bca400 tail 2840 1162 2840 1001 R+ top 2309 2224 2309 1001 SWs+ wait 0x202ce6dc bash 2224 2162 2162 1001 S select 0x9ab08a4 sshd 2162 786 2162 0 Ss select 0x18d5d864 sshd db> c KDB: enter: Break to debugger [ thread pid 2840 tid 100514 ] Stopped at kdb_alt_break_internal+0x13b: movl $0,kdb_why db> run pho db:0:pho> set $lines 20000 db:0:pho> run pho1 db:1:pho1> bt Tracing pid 2840 tid 100514 td 0x284e9700 kdb_alt_break_internal(0) at kdb_alt_break_internal+0x13b/frame 0x2885a608 kdb_alt_break(2,1832be8c) at kdb_alt_break+0x10/frame 0x2885a614 uart_intr_rxready(1832be00,1dba,b85c5e4,880,b90ca00,...) at uart_intr_rxready+0xec/frame 0x2885a630 uart_intr(1832be00) at uart_intr+0x116/frame 0x2885a654 intr_event_handle(b90ca00,2885a6c8) at intr_event_handle+0x104/frame 0x2885a678 intr_execute_handlers(b9dc93c,2885a6c8) at intr_execute_handlers+0xbe/frame 0x2885a6a4 lapic_handle_intr(32,2885a6c8,8,28,28,...) at lapic_handle_intr+0x52/frame 0x2885a6b8 Xapic_isr1() at 0xffc0424a/frame 0x2885a6b8 --- interrupt, eip = 0xfc4e35, esp = 0x2885a708, ebp = 0x2885a708 --- __mtx_unlock_flags(1a92450,0,1489335,4eb) at __mtx_unlock_flags+0xb5/frame 0x2885a708 vnlru_free_locked(0,82000,1fc22000,18cd6080,2885a7ac,...) at vnlru_free_locked+0x279/frame 0x2885a768 vn_alloc_hard(2481f7b4,2885a7b4,1281eb8,74,0,...) at vn_alloc_hard+0xaf/frame 0x2885a780 getnewvnode(1413efe,18cd6080,1ab0f40,2885a7e4) at getnewvnode+0x261/frame 0x2885a7ac ffs_vgetf(18cd6080,5ea8,0,202000,2885a844,0) at ffs_vgetf+0xdf/frame 0x2885a7f4 ffs_vget(18cd6080,5ea8,0,202000,2885a844) at ffs_vget+0x19/frame 0x2885a814 ufs_lookup_ino(1fc19468,2885ab84,2885ab9c,0) at ufs_lookup_ino+0xba8/frame 0x2885a8cc ufs_lookup(2885a908) at ufs_lookup+0x16/frame 0x2885a8e4 VOP_CACHEDLOOKUP_APV(1ab0f40,2885a908) at VOP_CACHEDLOOKUP_APV+0x4d/frame 0x2885a8f8 vfs_cache_lookup(2885a960) at vfs_cache_lookup+0x98/frame 0x2885a92c VOP_LOOKUP_APV(1ab0f40,2885a960) at VOP_LOOKUP_APV+0x4d/frame 0x2885a940 lookup(2885ab48) at lookup+0x580/frame 0x2885a994 namei(2885ab48) at namei+0x6df/frame 0x2885aa20 vn_open_cred(2885ab48,2885ac00,0,0,235c4b00,2ae503c0) at vn_open_cred+0x2f8/frame 0x2885aae0 vn_open(2885ab48,2885ac00,0,2ae503c0) at vn_open+0x20/frame 0x2885ab00 kern_openat(284e9700,ffffff9c,22f04206,0,100001,0) at kern_openat+0x2fa/frame 0x2885abe8 sys_open(284e9700,284e9998,284e9700,284de000,284e9700,...) at sys_open+0x1c/frame 0x2885ac08 syscall(2885ace8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x2885acdc Xint0x80_syscall() at 0xffc033f9/frame 0x2885acdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- db:1:pho1> show allpcpu Current CPU: 3 cpuid = 0 dynamic pcpu = 0x830d00 curthread = 0x2817a700: pid 7135 tid 100390 critnest 0 "sed" curpcb = 0x2830bd40 fpcurthread = 0x2817a700: pid 7135 "sed" idlethread = 0xb9c3500: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 trampstk = 0xffc07ff0 kesp0 = 0x2830bd30 common_tssp = 0xffc01260 tlb gen = 33965883 curvnet = 0 spin locks held: cpuid = 1 dynamic pcpu = 0x9d81d00 curthread = 0x284dc180: pid 7134 tid 104564 critnest 0 "sed" curpcb = 0x325ffd40 fpcurthread = 0x284dc180: pid 7134 "sed" idlethread = 0xb9c3180: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 trampstk = 0xffc0aff0 kesp0 = 0x325ffd30 common_tssp = 0xffc012c8 tlb gen = 33965889 curvnet = 0 spin locks held: cpuid = 2 dynamic pcpu = 0x9d87d00 curthread = 0x235e6000: pid 739 tid 100187 critnest 0 "smartd" curpcb = 0x205f7d40 fpcurthread = 0x235e6000: pid 739 "smartd" idlethread = 0xb9c2e00: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 trampstk = 0xffc0bff0 kesp0 = 0x205f7d30 common_tssp = 0xffc01330 tlb gen = 33965889 curvnet = 0 spin locks held: cpuid = 3 dynamic pcpu = 0x9d8dd00 curthread = 0x284e9700: pid 2840 tid 100514 critnest 1 "top" curpcb = 0x2885ad40 fpcurthread = 0x284e9700: pid 2840 "top" idlethread = 0xb9c2a80: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 trampstk = 0xffc0dff0 kesp0 = 0x2885ad30 common_tssp = 0xffc01398 tlb gen = 33965889 curvnet = 0 spin locks held: db:1:pho1> show alllocks db:1:pho1> show lockedvnods Locked vnodes vnode 0x1fc19a48: type VDIR usecount 48, writecount 0, refcount 4 seqc users 0 mountedhere 0 hold count flags () flags (VV_ROOT|VMP_LAZYLIST) v_object 0x18860d74 ref 0 pages 1 cleanbuf 1 dirtybuf 0 lock type ufs: SHARED (count 1) #0 0xfb96d2 at lockmgr_lock_flags+0x192 #1 0x12676c6 at ffs_lock+0x76 #2 0x13ecc38 at VOP_LOCK1_APV+0x38 #3 0x10c2e89 at _vn_lock+0x49 #4 0x10a01f3 at lookup+0xc3 #5 0x109fb2f at namei+0x6df #6 0x10c25f8 at vn_open_cred+0x2f8 #7 0x10c22f0 at vn_open+0x20 #8 0x10b9a6a at kern_openat+0x2fa #9 0x10b9c7f at sys_openat+0x2f #10 0x13d6bc8 at syscall+0x1b8 #11 0xffc033f9 at exit_tag+0xc0fdc24d nlink=47, effnlink=47, size=1536, extsize 0 generation=3134e177, uid=0, gid=0, flags=0x0 ino 2, on dev ada0s1a vnode 0x1fc19468: type VDIR usecount 1, writecount 0, refcount 4 seqc users 0 mountedhere 0 hold count flags () flags () v_object 0x188608f8 ref 0 pages 1 cleanbuf 1 dirtybuf 0 lock type ufs: SHARED (count 1) #0 0xfb96d2 at lockmgr_lock_flags+0x192 #1 0x12676c6 at ffs_lock+0x76 #2 0x13ecc38 at VOP_LOCK1_APV+0x38 #3 0x10c2e89 at _vn_lock+0x49 #4 0x10a01f3 at lookup+0xc3 #5 0x109fb2f at namei+0x6df #6 0x10c25f8 at vn_open_cred+0x2f8 #7 0x10c22f0 at vn_open+0x20 #8 0x10b9a6a at kern_openat+0x2fa #9 0x10b975c at sys_open+0x1c #10 0x13d6bc8 at syscall+0x1b8 #11 0xffc033f9 at exit_tag+0xc0fdc24d nlink=32, effnlink=32, size=2560, extsize 0 generation=1c6dba9d, uid=0, gid=0, flags=0x0 ino 23552, on dev ada0s1a vnode 0x22c44178: type VDIR usecount 2, writecount 0, refcount 2 seqc users 1 mountedhere 0x3d741000 hold count flags () flags () v_object 0x22c578f8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type ufs: EXCL by thread 0x2bad6e00 (pid 7132, umount, tid 101563) with shared waiters pending #0 0xfb96d2 at lockmgr_lock_flags+0x192 #1 0x12676c6 at ffs_lock+0x76 #2 0x13ecc38 at VOP_LOCK1_APV+0x38 #3 0x10c2e89 at _vn_lock+0x49 #4 0x10a4db0 at dounmount+0x60 #5 0x10a4ced at kern_unmount+0x2dd #6 0x10a4a03 at sys_unmount+0x13 #7 0x13d6bc8 at syscall+0x1b8 #8 0xffc033f9 at exit_tag+0xc0fdc24d nlink=2, effnlink=2, size=512, extsize 0 generation=f82d763a, uid=0, gid=0, flags=0x0 ino 164865, on dev ada0s1a vnode 0x1866abc0: type VDIR usecount 7, writecount 0, refcount 1 seqc users 0 mountedhere 0 hold count flags () flags (VV_ROOT) lock type devfs: SHARED (count 4) #0 0xfbadc1 at lockmgr_slock+0xb1 #1 0x109bda8 at vop_lock+0x48 #2 0x13ecc38 at VOP_LOCK1_APV+0x38 #3 0x10c2e89 at _vn_lock+0x49 #4 0x10b154b at vfs_cache_root+0x12b #5 0x10a087e at lookup+0x74e #6 0x109fb2f at namei+0x6df #7 0x10c25f8 at vn_open_cred+0x2f8 #8 0x10c22f0 at vn_open+0x20 #9 0x10b9a6a at kern_openat+0x2fa #10 0x10b9c7f at sys_openat+0x2f #11 0x13d6bc8 at syscall+0x1b8 #12 0xffc033f9 at exit_tag+0xc0fdc24d vnode 0x202b9468: type VDIR usecount 1, writecount 0, refcount 4 seqc users 0 mountedhere 0 hold count flags () flags () v_object 0x1fc92000 ref 0 pages 3 cleanbuf 1 dirtybuf 0 lock type ufs: SHARED (count 1) #0 0xfb96d2 at lockmgr_lock_flags+0x192 #1 0x12676c6 at ffs_lock+0x76 #2 0x13ecc38 at VOP_LOCK1_APV+0x38 #3 0x10c2e89 at _vn_lock+0x49 #4 0x10a01f3 at lookup+0xc3 #5 0x109fb2f at namei+0x6df #6 0xf9f3c3 at kern_execve+0x223 #7 0xf9eeb1 at sys_execve+0x51 #8 0x13d6bc8 at syscall+0x1b8 #9 0xffc033f9 at exit_tag+0xc0fdc24d nlink=2, effnlink=2, size=9728, extsize 0 generation=e46b503, uid=0, gid=0, flags=0x0 ino 1106944, on dev ada0s1d vnode 0x27a45468: type VDIR usecount 2, writecount 0, refcount 4 seqc users 0 mountedhere 0 hold count flags () flags () v_object 0x2774aae4 ref 0 pages 1 cleanbuf 1 dirtybuf 0 lock type ufs: SHARED (count 2) #0 0xfb96d2 at lockmgr_lock_flags+0x192 #1 0x12676c6 at ffs_lock+0x76 #2 0x13ecc38 at VOP_LOCK1_APV+0x38 #3 0x10c2e89 at _vn_lock+0x49 #4 0x10a01f3 at lookup+0xc3 #5 0x109fb2f at namei+0x6df #6 0x10c25f8 at vn_open_cred+0x2f8 #7 0x10c22f0 at vn_open+0x20 #8 0x10b9a6a at kern_openat+0x2fa #9 0x10b975c at sys_open+0x1c #10 0x13d6bc8 at syscall+0x1b8 #11 0xffc033f9 at exit_tag+0xc0fdc24d nlink=2, effnlink=2, size=512, extsize 0 generation=46e7b1cd, uid=0, gid=0, flags=0x0 ino 447489, on dev ada0s1d vnode 0x23bd1bc0: type VDIR usecount 2, writecount 0, refcount 3 seqc users 0 mountedhere 0 hold count flags () flags () v_object 0x230c7b88 ref 0 pages 1 cleanbuf 1 dirtybuf 0 lock type ufs: SHARED (count 1) #0 0xfb96d2 at lockmgr_lock_flags+0x192 #1 0x12676c6 at ffs_lock+0x76 #2 0x13ecc38 at VOP_LOCK1_APV+0x38 #3 0x10c2e89 at _vn_lock+0x49 #4 0x10a01f3 at lookup+0xc3 #5 0x109fb2f at namei+0x6df #6 0x10bbc47 at kern_statat+0x107 #7 0x10bc4c8 at sys_fstatat+0x28 #8 0x13d6bc8 at syscall+0x1b8 #9 0xffc033f9 at exit_tag+0xc0fdc24d nlink=3, effnlink=3, size=512, extsize 0 generation=52685cbc, uid=0, gid=0, flags=0x0 ino 4380672, on dev ada0s1e vnode 0x206128d0: type VDIR usecount 0, writecount 0, refcount 1 seqc users 0 mountedhere 0 hold count flags () flags () v_object 0x2a62c99c ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type ufs: EXCL by thread 0x235e6000 (pid 739, smartd, tid 100187) #0 0xfb96d2 at lockmgr_lock_flags+0x192 #1 0x12676c6 at ffs_lock+0x76 #2 0x13ecc38 at VOP_LOCK1_APV+0x38 #3 0x10a9df6 at vnlru_free_locked+0x2b6 #4 0x10b407f at vn_alloc_hard+0xaf #5 0x10aa3d1 at getnewvnode+0x261 #6 0xea80d2 at devfs_allocv+0x202 #7 0xea896a at devfs_lookup+0x40a #8 0x13e9c6d at VOP_LOOKUP_APV+0x4d #9 0x10a06b0 at lookup+0x580 #10 0x109fb2f at namei+0x6df #11 0x10c25f8 at vn_open_cred+0x2f8 #12 0x10c22f0 at vn_open+0x20 #13 0x10b9a6a at kern_openat+0x2fa #14 0x10b9c7f at sys_openat+0x2f #15 0x13d6bc8 at syscall+0x1b8 #16 0xffc033f9 at exit_tag+0xc0fdc24d nlink=3, effnlink=0, size=0, extsize 0 generation=643c6f30, uid=1004, gid=0, flags=0x0 ino 84816, on dev md10a vnode 0x32dfd2f0: type VDIR usecount 0, writecount 0, refcount 1 seqc users 0 mountedhere 0 hold count flags () flags () lock type ufs: EXCL by thread 0x2bad6e00 (pid 7132, umount, tid 101563) #0 0xfb96d2 at lockmgr_lock_flags+0x192 #1 0x12676c6 at ffs_lock+0x76 #2 0x13ecc38 at VOP_LOCK1_APV+0x38 #3 0x10a9df6 at vnlru_free_locked+0x2b6 #4 0x10b407f at vn_alloc_hard+0xaf #5 0x10aa3d1 at getnewvnode+0x261 #6 0x126291f at ffs_vgetf+0xdf #7 0x12557d9 at handle_workitem_remove+0x49 #8 0x1259810 at process_worklist_item+0x250 #9 0x1246e43 at softdep_process_worklist+0xb3 #10 0x1246d1d at softdep_flushworklist+0x3d #11 0x1265984 at ffs_sync+0x3a4 #12 0x10c4804 at vfs_write_suspend+0x144 #13 0x10c498f at vfs_write_suspend_umnt+0x2f #14 0x1264f74 at ffs_unmount+0x64 #15 0x10a5153 at dounmount+0x403 #16 0x10a4ced at kern_unmount+0x2dd #17 0x10a4a03 at sys_unmount+0x13 nlink=1, effnlink=0, size=0, extsize 0 generation=11f80a06, uid=1004, gid=0, flags=0x0 ino 245337, on dev md10a db:1:pho1> show allchains chain 1: thread 100993 (pid 7141, sshd) is blocked on sx "devfsmount" XLOCK thread 100187 (pid 739, smartd) is running on CPU 2 chain 2: thread 101843 (pid 7140, sendmail) is blocked on sx "devfsmount" XLOCK thread 100187 (pid 739, smartd) is running on CPU 2 chain 3: thread 101564 (pid 7139, sendmail) is blocked on sx "devfsmount" XLOCK thread 100187 (pid 739, smartd) is running on CPU 2 chain 4: thread 100813 (pid 7138, ls) is blocked on lockmgr ufsEXCL thread 101563 (pid 7132, umount) is on a run queue chain 5: thread 101766 (pid 7137, sh) is sleeping on 0x25db4344 "wait" chain 6: thread 101659 (pid 7136, sh) is sleeping on 0x34eea1a8 "piperd" chain 7: thread 102571 (pid 1489, md10) is sleeping on 0x3881a000 "mdwait" chain 8: thread 100916 (pid 1237, sh) is sleeping on 0x2999b054 "wait" chain 9: thread 101566 (pid 1236, tee) is sleeping on 0x3df6e1a8 "piperd" chain 10: thread 103742 (pid 1235, sh) is sleeping on 0x2bb7a000 "wait" chain 11: thread 101606 (pid 6180, tail) is sleeping on 0x1880b9a4 "select" chain 12: thread 101657 (pid 6144, tail) is sleeping on 0x9bca400 "kqread" chain 13: thread 100123 (pid 2309, bash) is sleeping on 0x202ce6dc "wait" chain 14: thread 100230 (pid 2224, sshd) is sleeping on 0x9ab08a4 "select" chain 15: thread 100229 (pid 2162, sshd) is sleeping on 0x18d5d864 "select" chain 16: thread 100190 (pid 1380, sh) is sleeping on 0x23394350 "piperd" chain 17: thread 100120 (pid 1378, awk) is sleeping on 0x233944f8 "piperd" chain 18: thread 100107 (pid 1377, sh) is sleeping on 0x1834e6dc "wait" chain 19: thread 100217 (pid 1356, sh) is sleeping on 0x1fe2b054 "wait" chain 20: thread 100138 (pid 1162, bash) is sleeping on 0x202cd000 "wait" chain 21: thread 100105 (pid 1158, sshd) is sleeping on 0xb9d3de4 "select" chain 22: thread 100220 (pid 1156, sshd) is sleeping on 0x9ab0924 "select" chain 23: thread 100121 (pid 1083, sh) is sleeping on 0x1fe2b398 "wait" chain 24: thread 100215 (pid 1009, bash) is sleeping on 0x2512b398 "wait" chain 25: thread 100211 (pid 1007, csh) is sleeping on 0x2512b740 "pause" chain 26: thread 100108 (pid 1006, su) is sleeping on 0x1fc526dc "wait" chain 27: thread 100144 (pid 988, bash) is sleeping on 0x22c2b398 "wait" chain 28: thread 100208 (pid 987, sshd) is sleeping on 0x1880bb24 "select" chain 29: thread 100114 (pid 985, sshd) is sleeping on 0x18d5d964 "select" chain 30: thread 100200 (pid 828, getty) is sleeping on 0x18c52c74 "ttyin" chain 31: thread 100199 (pid 827, getty) is sleeping on 0x18bac074 "ttyin" chain 32: thread 100198 (pid 826, getty) is sleeping on 0x18bac474 "ttyin" chain 33: thread 100197 (pid 825, getty) is sleeping on 0x18bac874 "ttyin" chain 34: thread 100196 (pid 824, getty) is sleeping on 0x18bacc74 "ttyin" chain 35: thread 100195 (pid 823, getty) is sleeping on 0x188ed074 "ttyin" chain 36: thread 100194 (pid 822, getty) is sleeping on 0x188ed474 "ttyin" chain 37: thread 100193 (pid 821, getty) is sleeping on 0x188ed874 "ttyin" chain 38: thread 100148 (pid 808, sendmail) is sleeping on 0x22c2ad74 "pause" chain 39: thread 100191 (pid 802, sendmail) is sleeping on 0x18d5da64 "select" chain 40: thread 100184 (pid 786, sshd) is sleeping on 0x9ab09e4 "select" chain 41: thread 100147 (pid 747, inetd) is sleeping on 0x18d5dae4 "select" chain 42: thread 100145 (pid 696, powerd) is sleeping on 0x9ab0a24 "select" chain 43: thread 100186 (pid 693, ntpd) is sleeping on 0x1880bbe4 "select" chain 44: thread 100139 (pid 672, NLM: master) is sleeping on 0x1890b950 "rpcsvc" chain 45: thread 100141 (pid 669, rpc.statd) is sleeping on 0x1880bc24 "select" chain 46: thread 100152 (pid 668, nfsd: master) is sleeping on 0x1890b9d0 "rpcsvc" chain 47: thread 100153 (pid 668, nfsd: service) is sleeping on 0x1890bc50 "rpcsvc" chain 48: thread 100154 (pid 668, nfsd: service) is sleeping on 0x9a2b4d0 "rpcsvc" chain 49: thread 100155 (pid 668, nfsd: service) is sleeping on 0x1890bb50 "rpcsvc" chain 50: thread 100156 (pid 668, nfsd: service) is sleeping on 0x9a2b550 "rpcsvc" chain 51: thread 100157 (pid 668, nfsd: service) is sleeping on 0x1890bd50 "rpcsvc" chain 52: thread 100158 (pid 668, nfsd: service) is sleeping on 0x9a2b5d0 "rpcsvc" chain 53: thread 100159 (pid 668, nfsd: service) is sleeping on 0x9a2b350 "rpcsvc" chain 54: thread 100160 (pid 668, nfsd: service) is sleeping on 0x202becd0 "rpcsvc" chain 55: thread 100161 (pid 668, nfsd: service) is sleeping on 0x9a2b3d0 "rpcsvc" chain 56: thread 100162 (pid 668, nfsd: service) is sleeping on 0x1890be50 "rpcsvc" chain 57: thread 100163 (pid 668, nfsd: service) is sleeping on 0x9a2b750 "rpcsvc" chain 58: thread 100164 (pid 668, nfsd: service) is sleeping on 0x9a2b650 "rpcsvc" chain 59: thread 100165 (pid 668, nfsd: service) is sleeping on 0x9a2b2d0 "rpcsvc" chain 60: thread 100166 (pid 668, nfsd: service) is sleeping on 0x202be9d0 "rpcsvc" chain 61: thread 100167 (pid 668, nfsd: service) is sleeping on 0x9a2b250 "rpcsvc" chain 62: thread 100168 (pid 668, nfsd: service) is sleeping on 0x202be950 "rpcsvc" chain 63: thread 100169 (pid 668, nfsd: service) is sleeping on 0x9a2b1d0 "rpcsvc" chain 64: thread 100170 (pid 668, nfsd: service) is sleeping on 0x9a2b150 "rpcsvc" chain 65: thread 100171 (pid 668, nfsd: service) is sleeping on 0x9a2b0d0 "rpcsvc" chain 66: thread 100172 (pid 668, nfsd: service) is sleeping on 0x202be8d0 "rpcsvc" chain 67: thread 100173 (pid 668, nfsd: service) is sleeping on 0x9a2b050 "rpcsvc" chain 68: thread 100174 (pid 668, nfsd: service) is sleeping on 0x202be850 "rpcsvc" chain 69: thread 100175 (pid 668, nfsd: service) is sleeping on 0x235e5f50 "rpcsvc" chain 70: thread 100176 (pid 668, nfsd: service) is sleeping on 0x202be7d0 "rpcsvc" chain 71: thread 100177 (pid 668, nfsd: service) is sleeping on 0x202be750 "rpcsvc" chain 72: thread 100178 (pid 668, nfsd: service) is sleeping on 0x202be6d0 "rpcsvc" chain 73: thread 100179 (pid 668, nfsd: service) is sleeping on 0x202be650 "rpcsvc" chain 74: thread 100180 (pid 668, nfsd: service) is sleeping on 0x235e5ed0 "rpcsvc" chain 75: thread 100181 (pid 668, nfsd: service) is sleeping on 0x202be5d0 "rpcsvc" chain 76: thread 100182 (pid 668, nfsd: service) is sleeping on 0x202be550 "rpcsvc" chain 77: thread 100183 (pid 668, nfsd: service) is sleeping on 0x235e5e50 "rpcsvc" chain 78: thread 100143 (pid 665, nfsd) is sleeping on 0x1880bd24 "select" chain 79: thread 100151 (pid 658, mountd) is sleeping on 0x1880bc64 "select" chain 80: thread 100129 (pid 634, rpcbind) is sleeping on 0x1880bce4 "select" chain 81: thread 100130 (pid 584, watchdogd) is sleeping on 0x1834d344 "wait" chain 82: thread 100111 (pid 575, syslogd) is sleeping on 0x1880be24 "select" chain 83: thread 100131 (pid 533, nfscbd: master) is sleeping on 0x202bee50 "rpcsvc" chain 84: thread 100132 (pid 533, nfscbd: service) is sleeping on 0x9a2bad0 "rpcsvc" chain 85: thread 100133 (pid 533, nfscbd: service) is sleeping on 0x9a2ba50 "rpcsvc" chain 86: thread 100134 (pid 533, nfscbd: service) is sleeping on 0x9a2b9d0 "rpcsvc" chain 87: thread 100112 (pid 532, nfscbd) is sleeping on 0x202f8c1c "accept" chain 88: thread 100128 (pid 449, nfsuserd) is sleeping on 0x1880bda4 "select" chain 89: thread 100127 (pid 448, nfsuserd) is sleeping on 0x18d5dc64 "select" chain 90: thread 100126 (pid 447, nfsuserd) is sleeping on 0x1880bde4 "select" chain 91: thread 100113 (pid 446, nfsuserd) is sleeping on 0x18d5dca4 "select" chain 92: thread 100124 (pid 445, nfsuserd) is sleeping on 0x202ce3fc "pause" chain 93: thread 100125 (pid 380, devd) is sleeping on 0x9ab0b24 "select" chain 94: thread 100104 (pid 24, syncer) is sleeping on 0x1dd58b8 "syncer" chain 95: thread 100103 (pid 23, vnlru) is sleeping on 0x18911d10 "vlruwt" chain 96: thread 100099 (pid 22, bufdaemon) is sleeping on 0x1dd5004 "qsleep" chain 97: thread 100102 (pid 22, bufspacedaemon-0) is sleeping on 0x1a87400 "-" chain 98: thread 100116 (pid 22, /home worker) is sleeping on 0x1fe39e84 "sdflush" chain 99: thread 100117 (pid 22, /tmp worker) is sleeping on 0x1fe39c84 "sdflush" chain 100: thread 100118 (pid 22, /usr worker) is sleeping on 0x1fe39a84 "sdflush" chain 101: thread 100119 (pid 22, /var worker) is sleeping on 0x1fe39884 "sdflush" chain 102: thread 100098 (pid 21, vmdaemon) is sleeping on 0x1dec370 "psleep" chain 103: thread 100097 (pid 20, dom0) is sleeping on 0x1deb3bc "psleep" chain 104: thread 100100 (pid 20, laundry: dom0) is sleeping on 0x1deb3c8 "launds" chain 105: thread 100101 (pid 20, uma) is sleeping on 0x1283f70 "umarcl" chain 106: thread 100091 (pid 19, rand_harvestq) is sleeping on 0x1bb247c "-" chain 107: thread 100090 (pid 18, soaiod4) is sleeping on 0x1dd4b68 "-" chain 108: thread 100089 (pid 17, soaiod3) is sleeping on 0x1dd4b68 "-" chain 109: thread 100088 (pid 16, soaiod2) is sleeping on 0x1dd4b68 "-" chain 110: thread 100087 (pid 9, soaiod1) is sleeping on 0x1dd4b68 "-" chain 111: thread 100084 (pid 8, fdc0) is sleeping on 0x182b9c3c "-" chain 112: thread 100049 (pid 15, usbus0) is sleeping on 0x18024aac "-" chain 113: thread 100050 (pid 15, usbus0) is sleeping on 0x18024adc "-" chain 114: thread 100051 (pid 15, usbus0) is sleeping on 0x18024b0c "-" chain 115: thread 100052 (pid 15, usbus0) is sleeping on 0x18024b3c "-" chain 116: thread 100053 (pid 15, usbus0) is sleeping on 0x18024b6c "-" chain 117: thread 100055 (pid 15, usbus1) is sleeping on 0x182c5aac "-" chain 118: thread 100056 (pid 15, usbus1) is sleeping on 0x182c5adc "-" chain 119: thread 100057 (pid 15, usbus1) is sleeping on 0x182c5b0c "-" chain 120: thread 100058 (pid 15, usbus1) is sleeping on 0x182c5b3c "-" chain 121: thread 100059 (pid 15, usbus1) is sleeping on 0x182c5b6c "-" chain 122: thread 100061 (pid 15, usbus2) is sleeping on 0x182ccaac "-" chain 123: thread 100062 (pid 15, usbus2) is sleeping on 0x182ccadc "-" chain 124: thread 100063 (pid 15, usbus2) is sleeping on 0x182ccb0c "-" chain 125: thread 100064 (pid 15, usbus2) is sleeping on 0x182ccb3c "-" chain 126: thread 100065 (pid 15, usbus2) is sleeping on 0x182ccb6c "-" chain 127: thread 100066 (pid 15, usbus3) is sleeping on 0x182d6aac "-" chain 128: thread 100067 (pid 15, usbus3) is sleeping on 0x182d6adc "-" chain 129: thread 100068 (pid 15, usbus3) is sleeping on 0x182d6b0c "-" chain 130: thread 100069 (pid 15, usbus3) is sleeping on 0x182d6b3c "-" chain 131: thread 100070 (pid 15, usbus3) is sleeping on 0x182d6b6c "-" chain 132: thread 100071 (pid 15, usbus4) is sleeping on 0x182e4aac "-" chain 133: thread 100072 (pid 15, usbus4) is sleeping on 0x182e4adc "-" chain 134: thread 100073 (pid 15, usbus4) is sleeping on 0x182e4b0c "-" chain 135: thread 100074 (pid 15, usbus4) is sleeping on 0x182e4b3c "-" chain 136: thread 100075 (pid 15, usbus4) is sleeping on 0x182e4b6c "-" chain 137: thread 100077 (pid 15, usbus5) is sleeping on 0x1801fc84 "-" chain 138: thread 100078 (pid 15, usbus5) is sleeping on 0x1801fcb4 "-" chain 139: thread 100079 (pid 15, usbus5) is sleeping on 0x1801fce4 "-" chain 140: thread 100080 (pid 15, usbus5) is sleeping on 0x1801fd14 "-" chain 141: thread 100081 (pid 15, usbus5) is sleeping on 0x1801fd44 "-" chain 142: thread 100044 (pid 7, doneq0) is sleeping on 0x1ae97c0 "-" chain 143: thread 100096 (pid 7, scanner) is sleeping on 0x1ae96ec "-" chain 144: thread 100043 (pid 6, crypto returns 3) is sleeping on 0xb9ff6c4 "crypto_ret_wait" chain 145: thread 100042 (pid 5, crypto returns 2) is sleeping on 0xb9ff68c "crypto_ret_wait" chain 146: thread 100041 (pid 4, crypto returns 1) is sleeping on 0xb9ff654 "crypto_ret_wait" chain 147: thread 100040 (pid 3, crypto returns 0) is sleeping on 0xb9ff61c "crypto_ret_wait" chain 148: thread 100039 (pid 2, crypto) is sleeping on 0x1de9b20 "crypto_wait" chain 149: thread 100034 (pid 14, sequencer 00) is sleeping on 0x9ae804c "seqstate" chain 150: thread 100031 (pid 13, g_event) is sleeping on 0x1d51388 "-" chain 151: thread 100032 (pid 13, g_up) is sleeping on 0x1d5138c "-" chain 152: thread 100033 (pid 13, g_down) is sleeping on 0x1d51390 "-" chain 153: thread 100002 (pid 1, init) is sleeping on 0xb9c16dc "wait" chain 154: thread 100001 (pid 10, audit) is sleeping on 0x1de9e0c "audit_worker_cv" chain 155: thread 100000 (pid 0, swapper) is sleeping on 0x1d5185c "swapin" chain 156: thread 100007 (pid 0, if_io_tqg_0) is sleeping on 0xb9caa00 "-" chain 157: thread 100008 (pid 0, if_io_tqg_1) is sleeping on 0xb9ca800 "-" chain 158: thread 100009 (pid 0, if_io_tqg_2) is sleeping on 0xb9ca600 "-" chain 159: thread 100010 (pid 0, if_io_tqg_3) is sleeping on 0xb9ca400 "-" chain 160: thread 100011 (pid 0, if_config_tqg_0) is sleeping on 0xb9ca200 "-" chain 161: thread 100012 (pid 0, softirq_0) is sleeping on 0xb9ca000 "-" chain 162: thread 100013 (pid 0, softirq_1) is sleeping on 0xb9f0d00 "-" chain 163: thread 100014 (pid 0, softirq_2) is sleeping on 0xb9f0b00 "-" chain 164: thread 100015 (pid 0, softirq_3) is sleeping on 0xb9f0900 "-" chain 165: thread 100017 (pid 0, in6m_free taskq) is sleeping on 0xb9f0500 "-" chain 166: thread 100018 (pid 0, thread taskq) is sleeping on 0xb9f0300 "-" chain 167: thread 100020 (pid 0, aiod_kick taskq) is sleeping on 0xb9f4e00 "-" chain 168: thread 100021 (pid 0, kqueue_ctx taskq) is sleeping on 0xb9f4c00 "-" chain 169: thread 100023 (pid 0, inm_free taskq) is sleeping on 0xb9f4800 "-" chain 170: thread 100030 (pid 0, firmware taskq) is sleeping on 0x9aac800 "-" chain 171: thread 100035 (pid 0, crypto_0) is sleeping on 0x9ad6800 "-" chain 172: thread 100036 (pid 0, crypto_1) is sleeping on 0x9ad6800 "-" chain 173: thread 100037 (pid 0, crypto_2) is sleeping on 0x9ad6800 "-" chain 174: thread 100038 (pid 0, crypto_3) is sleeping on 0x9ad6800 "-" chain 175: thread 100086 (pid 0, mca taskq) is sleeping on 0x18330300 "-" chain 176: thread 100092 (pid 0, acpi_task_0) is sleeping on 0x18909400 "-" chain 177: thread 100093 (pid 0, acpi_task_1) is sleeping on 0x18909400 "-" chain 178: thread 100094 (pid 0, acpi_task_2) is sleeping on 0x18909400 "-" chain 179: thread 100095 (pid 0, CAM taskq) is sleeping on 0x9b25d00 "-" db:1:pho1> show mount 0x18cd6080 /dev/ada0s1a on / (ufs) 0x18cd4040 devfs on /dev (devfs) 0x1fe33040 /dev/ada0s1f on /home (ufs) 0x1fe32580 /dev/label/tmp on /tmp (ufs) 0x1fe31ac0 /dev/ada0s1d on /usr (ufs) 0x1fe31000 /dev/ada0s1e on /var (ufs) 0x1fe35080 procfs on /proc (procfs) 0x3d741000 /dev/md10a on /mnt (ufs) More info: show mount db:1:pho1> show bufqueues bqempty: 439 Buf domain 0 freebufs 3660 lofreebufs 311 hifreebufs 466 bufspace 57131008 maxbufspace 94683136 hibufspace 94027776 lobufspace 89326372 bufspacethresh 91677074 numdirtybuffers 3 lodirtybuffers 732 hidirtybuffers 1464 dirtybufthresh 1317 cleanq count 2064 (56248320) dirtyq count 3 (49152) wakeup 0 lim 28 CPU 18, 18, 11, 5, Locked buffers: 0 space 0 Total buffers: 5779 space 57131008 db:0:pho> ps pid ppid pgrp uid state wmesg wchan cmd 7141 786 7141 0 Ds devfsmo 0x18805bd4 sshd 7140 808 808 25 D devfsmo 0x18805bd4 sendmail 7139 802 802 0 D devfsmo 0x18805bd4 sendmail 7138 7137 584 0 D ufs 0x22c441b4 ls 7137 584 584 0 S wait 0x25db4344 sh 7136 1377 1356 0 S piperd 0x34eea1a8 sh 7135 1377 1356 0 R CPU 0 sed 7134 1377 1356 0 R CPU 1 sed 7132 1237 1083 0 RL+ umount 1489 0 0 0 DL mdwait 0x3881a000 [md10] 1237 1235 1083 0 S+ wait 0x2999b054 sh 1236 1083 1083 0 S+ piperd 0x3df6e1a8 tee 1235 1083 1083 0 S+ wait 0x2bb7a000 sh 6180 1 6144 1001 S+ select 0x1880b9a4 tail 6144 2309 6144 1001 S+ kqread 0x9bca400 tail 2840 1162 2840 1001 R+ CPU 3 top 2309 2224 2309 1001 SWs+ wait 0x202ce6dc bash 2224 2162 2162 1001 S select 0x9ab08a4 sshd 2162 786 2162 0 Ss select 0x18d5d864 sshd 1380 1356 1356 0 S piperd 0x23394350 sh 1378 1356 1356 0 S piperd 0x233944f8 awk 1377 1356 1356 0 S wait 0x1834e6dc sh 1356 1 1356 0 SWs wait 0x1fe2b054 sh 1162 1158 1162 1001 SWs+ wait 0x202cd000 bash 1158 1156 1156 1001 S select 0xb9d3de4 sshd 1156 786 1156 0 Ss select 0x9ab0924 sshd 1083 1009 1083 0 S+ wait 0x1fe2b398 sh 1009 1007 1009 0 SW+ wait 0x2512b398 bash 1007 1006 1007 0 SW+ pause 0x2512b740 csh 1006 988 1006 1001 SW+ wait 0x1fc526dc su 988 987 988 1001 SWs+ wait 0x22c2b398 bash 987 985 985 1001 S select 0x1880bb24 sshd 985 786 985 0 Ss select 0x18d5d964 sshd 829 1 829 0 Rs+ getty 828 1 828 0 Ss+ ttyin 0x18c52c74 getty 827 1 827 0 Ss+ ttyin 0x18bac074 getty 826 1 826 0 Ss+ ttyin 0x18bac474 getty 825 1 825 0 Ss+ ttyin 0x18bac874 getty 824 1 824 0 Ss+ ttyin 0x18bacc74 getty 823 1 823 0 Ss+ ttyin 0x188ed074 getty 822 1 822 0 Ss+ ttyin 0x188ed474 getty 821 1 821 0 Ss+ ttyin 0x188ed874 getty 808 1 808 25 Ss pause 0x22c2ad74 sendmail 802 1 802 0 Ss select 0x18d5da64 sendmail 789 1 789 0 Rs cron 786 1 786 0 Ss select 0x9ab09e4 sshd 747 1 747 0 Ss select 0x18d5dae4 inetd 739 1 738 0 R CPU 2 smartd 696 1 696 0 Ss select 0x9ab0a24 powerd 693 1 693 0 Ss (threaded) ntpd 100186 S select 0x1880bbe4 ntpd 672 1 672 0 Ss rpcsvc 0x1890b950 NLM: master 669 1 669 0 Ss select 0x1880bc24 rpc.statd 668 665 665 0 S (threaded) nfsd 100152 S rpcsvc 0x1890b9d0 nfsd: master 100153 S rpcsvc 0x1890bc50 nfsd: service 100154 S rpcsvc 0x9a2b4d0 nfsd: service 100155 S rpcsvc 0x1890bb50 nfsd: service 100156 S rpcsvc 0x9a2b550 nfsd: service 100157 S rpcsvc 0x1890bd50 nfsd: service 100158 S rpcsvc 0x9a2b5d0 nfsd: service 100159 S rpcsvc 0x9a2b350 nfsd: service 100160 S rpcsvc 0x202becd0 nfsd: service 100161 S rpcsvc 0x9a2b3d0 nfsd: service 100162 S rpcsvc 0x1890be50 nfsd: service 100163 S rpcsvc 0x9a2b750 nfsd: service 100164 S rpcsvc 0x9a2b650 nfsd: service 100165 S rpcsvc 0x9a2b2d0 nfsd: service 100166 S rpcsvc 0x202be9d0 nfsd: service 100167 S rpcsvc 0x9a2b250 nfsd: service 100168 S rpcsvc 0x202be950 nfsd: service 100169 S rpcsvc 0x9a2b1d0 nfsd: service 100170 S rpcsvc 0x9a2b150 nfsd: service 100171 S rpcsvc 0x9a2b0d0 nfsd: service 100172 S rpcsvc 0x202be8d0 nfsd: service 100173 S rpcsvc 0x9a2b050 nfsd: service 100174 S rpcsvc 0x202be850 nfsd: service 100175 S rpcsvc 0x235e5f50 nfsd: service 100176 S rpcsvc 0x202be7d0 nfsd: service 100177 S rpcsvc 0x202be750 nfsd: service 100178 S rpcsvc 0x202be6d0 nfsd: service 100179 S rpcsvc 0x202be650 nfsd: service 100180 S rpcsvc 0x235e5ed0 nfsd: service 100181 S rpcsvc 0x202be5d0 nfsd: service 100182 S rpcsvc 0x202be550 nfsd: service 100183 S rpcsvc 0x235e5e50 nfsd: service 665 1 665 0 Ss select 0x1880bd24 nfsd 658 1 658 0 Ss select 0x1880bc64 mountd 634 1 634 0 Ss select 0x1880bce4 rpcbind 584 1 584 0 Ss wait 0x1834d344 watchdogd 575 1 575 0 Ss select 0x1880be24 syslogd 533 532 532 0 S (threaded) nfscbd 100131 S rpcsvc 0x202bee50 nfscbd: master 100132 S rpcsvc 0x9a2bad0 nfscbd: service 100133 S rpcsvc 0x9a2ba50 nfscbd: service 100134 S rpcsvc 0x9a2b9d0 nfscbd: service 532 1 532 0 SWs accept 0x202f8c1c nfscbd 449 445 445 0 S select 0x1880bda4 nfsuserd 448 445 445 0 S select 0x18d5dc64 nfsuserd 447 445 445 0 S select 0x1880bde4 nfsuserd 446 445 445 0 S select 0x18d5dca4 nfsuserd 445 1 445 0 SWs pause 0x202ce3fc nfsuserd 380 1 380 0 Ss select 0x9ab0b24 devd 24 0 0 0 DL syncer 0x1dd58b8 [syncer] 23 0 0 0 DL vlruwt 0x18911d10 [vnlru] 22 0 0 0 RL (threaded) [bufdaemon] 100099 D qsleep 0x1dd5004 [bufdaemon] 100102 D - 0x1a87400 [bufspacedaemon-0] 100116 D sdflush 0x1fe39e84 [/home worker] 100117 D sdflush 0x1fe39c84 [/tmp worker] 100118 D sdflush 0x1fe39a84 [/usr worker] 100119 D sdflush 0x1fe39884 [/var worker] 103721 RunQ [/mnt worker] 21 0 0 0 DL psleep 0x1dec370 [vmdaemon] 20 0 0 0 DL (threaded) [pagedaemon] 100097 D psleep 0x1deb3bc [dom0] 100100 D launds 0x1deb3c8 [laundry: dom0] 100101 D umarcl 0x1283f70 [uma] 19 0 0 0 DL - 0x1bb247c [rand_harvestq] 18 0 0 0 DL - 0x1dd4b68 [soaiod4] 17 0 0 0 DL - 0x1dd4b68 [soaiod3] 16 0 0 0 DL - 0x1dd4b68 [soaiod2] 9 0 0 0 DL - 0x1dd4b68 [soaiod1] 8 0 0 0 DL - 0x182b9c3c [fdc0] 15 0 0 0 DL (threaded) [usb] 100049 D - 0x18024aac [usbus0] 100050 D - 0x18024adc [usbus0] 100051 D - 0x18024b0c [usbus0] 100052 D - 0x18024b3c [usbus0] 100053 D - 0x18024b6c [usbus0] 100055 D - 0x182c5aac [usbus1] 100056 D - 0x182c5adc [usbus1] 100057 D - 0x182c5b0c [usbus1] 100058 D - 0x182c5b3c [usbus1] 100059 D - 0x182c5b6c [usbus1] 100061 D - 0x182ccaac [usbus2] 100062 D - 0x182ccadc [usbus2] 100063 D - 0x182ccb0c [usbus2] 100064 D - 0x182ccb3c [usbus2] 100065 D - 0x182ccb6c [usbus2] 100066 D - 0x182d6aac [usbus3] 100067 D - 0x182d6adc [usbus3] 100068 D - 0x182d6b0c [usbus3] 100069 D - 0x182d6b3c [usbus3] 100070 D - 0x182d6b6c [usbus3] 100071 D - 0x182e4aac [usbus4] 100072 D - 0x182e4adc [usbus4] 100073 D - 0x182e4b0c [usbus4] 100074 D - 0x182e4b3c [usbus4] 100075 D - 0x182e4b6c [usbus4] 100077 D - 0x1801fc84 [usbus5] 100078 D - 0x1801fcb4 [usbus5] 100079 D - 0x1801fce4 [usbus5] 100080 D - 0x1801fd14 [usbus5] 100081 D - 0x1801fd44 [usbus5] 7 0 0 0 DL (threaded) [cam] 100044 D - 0x1ae97c0 [doneq0] 100096 D - 0x1ae96ec [scanner] 6 0 0 0 DL crypto_ 0xb9ff6c4 [crypto returns 3] 5 0 0 0 DL crypto_ 0xb9ff68c [crypto returns 2] 4 0 0 0 DL crypto_ 0xb9ff654 [crypto returns 1] 3 0 0 0 DL crypto_ 0xb9ff61c [crypto returns 0] 2 0 0 0 DL crypto_ 0x1de9b20 [crypto] 14 0 0 0 DL seqstat 0x9ae804c [sequencer 00] 13 0 0 0 DL (threaded) [geom] 100031 D - 0x1d51388 [g_event] 100032 D - 0x1d5138c [g_up] 100033 D - 0x1d51390 [g_down] 12 0 0 0 WL (threaded) [intr] 100016 I [swi6: Giant taskq] 100019 I [swi5: fast taskq] 100022 I [swi6: task queue] 100024 I [swi4: clock (0)] 100025 I [swi4: clock (1)] 100026 I [swi4: clock (2)] 100027 I [swi4: clock (3)] 100028 I [swi1: netisr 0] 100029 I [swi3: vm] 100045 I [irq24: hdac0] 100046 I [irq25: re0] 100047 I [irq22: ahci0] 100048 I [irq16: hdac1 ohci0] 100054 I [irq17: ohci1 ohci3] 100060 I [irq18: ohci2 ohci4] 100076 I [irq19: ehci0] 100082 I [irq14: ata0] 100083 I [swi0: uart] 100085 I [irq1: atkbd0] 11 0 0 0 RL (threaded) [idle] 100003 CanRun [idle: cpu0] 100004 CanRun [idle: cpu1] 100005 CanRun [idle: cpu2] 100006 CanRun [idle: cpu3] 1 0 1 0 SLs wait 0xb9c16dc [init] 10 0 0 0 DL audit_w 0x1de9e0c [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D swapin 0x1d5185c [swapper] 100007 D - 0xb9caa00 [if_io_tqg_0] 100008 D - 0xb9ca800 [if_io_tqg_1] 100009 D - 0xb9ca600 [if_io_tqg_2] 100010 D - 0xb9ca400 [if_io_tqg_3] 100011 D - 0xb9ca200 [if_config_tqg_0] 100012 D - 0xb9ca000 [softirq_0] 100013 D - 0xb9f0d00 [softirq_1] 100014 D - 0xb9f0b00 [softirq_2] 100015 D - 0xb9f0900 [softirq_3] 100017 D - 0xb9f0500 [in6m_free taskq] 100018 D - 0xb9f0300 [thread taskq] 100020 D - 0xb9f4e00 [aiod_kick taskq] 100021 D - 0xb9f4c00 [kqueue_ctx taskq] 100023 D - 0xb9f4800 [inm_free taskq] 100030 D - 0x9aac800 [firmware taskq] 100035 D - 0x9ad6800 [crypto_0] 100036 D - 0x9ad6800 [crypto_1] 100037 D - 0x9ad6800 [crypto_2] 100038 D - 0x9ad6800 [crypto_3] 100086 D - 0x18330300 [mca taskq] 100092 D - 0x18909400 [acpi_task_0] 100093 D - 0x18909400 [acpi_task_1] 100094 D - 0x18909400 [acpi_task_2] 100095 D - 0x9b25d00 [CAM taskq] db:0:pho> show freepages DOMAIN: 0 FREE LIST 0: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 09 (002048K) | 000000 08 (001024K) | 000000 07 (000512K) | 000000 06 (000256K) | 000000 05 (000128K) | 000000 04 (000064K) | 000000 03 (000032K) | 000000 02 (000016K) | 000000 01 (000008K) | 000000 00 (000004K) | 000000 FREE LIST 1: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 09 (002048K) | 000000 08 (001024K) | 000003 07 (000512K) | 000047 06 (000256K) | 000876 05 (000128K) | 003338 04 (000064K) | 009816 03 (000032K) | 019156 02 (000016K) | 028464 01 (000008K) | 034769 00 (000004K) | 041530 FREE LIST 2: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 09 (002048K) | 000000 08 (001024K) | 000000 07 (000512K) | 000000 06 (000256K) | 000000 05 (000128K) | 000000 04 (000064K) | 000003 03 (000032K) | 000004 02 (000016K) | 000009 01 (000008K) | 000008 00 (000004K) | 000006 db:0:pho> acttrace Tracing command sed pid 7135 tid 100390 td 0x2817a700 (CPU 0) cpustop_handler(2830b650,2830b644,13d59fb,1,22ff5b8,...) at cpustop_handler+0x26/frame 0x2830b57c ipi_nmi_handler(1,22ff5b8,42c003b5,1dba,2813f000,...) at ipi_nmi_handler+0x30/frame 0x2830b588 trap(2830b650,8,28,28,2817a700,...) at trap+0xbb/frame 0x2830b644 calltrap() at 0xffc0319f/frame 0x2830b644 --- trap 0x13, eip = 0xfc4b30, esp = 0x2830b690, ebp = 0x2830b6dc --- __mtx_lock_sleep(1a92450,235e6000,0,1489335,4ee,...) at __mtx_lock_sleep+0x1e0/frame 0x2830b6dc __mtx_lock_flags(1a92450,0,1489335,4ee) at __mtx_lock_flags+0xcc/frame 0x2830b708 vnlru_free_locked(0,82000,1ff58000,1fe31ac0,2830b7ac,...) at vnlru_free_locked+0x9a/frame 0x2830b768 vn_alloc_hard(23b4f658,2830b7b4,1281eb8,74,0,...) at vn_alloc_hard+0xaf/frame 0x2830b780 getnewvnode(1413efe,1fe31ac0,1ab0f40,2830b7e4) at getnewvnode+0x261/frame 0x2830b7ac ffs_vgetf(1fe31ac0,6d495,0,202000,2830b844,0) at ffs_vgetf+0xdf/frame 0x2830b7f4 ffs_vget(1fe31ac0,6d495,0,202000,2830b844) at ffs_vget+0x19/frame 0x2830b814 ufs_lookup_ino(27a45468,2830bb84,2830bb9c,0) at ufs_lookup_ino+0xba8/frame 0x2830b8cc ufs_lookup(2830b908) at ufs_lookup+0x16/frame 0x2830b8e4 VOP_CACHEDLOOKUP_APV(1ab0f40,2830b908) at VOP_CACHEDLOOKUP_APV+0x4d/frame 0x2830b8f8 vfs_cache_lookup(2830b960) at vfs_cache_lookup+0x98/frame 0x2830b92c VOP_LOOKUP_APV(1ab0f40,2830b960) at VOP_LOOKUP_APV+0x4d/frame 0x2830b940 lookup(2830bb48) at lookup+0x580/frame 0x2830b994 namei(2830bb48) at namei+0x6df/frame 0x2830ba20 vn_open_cred(2830bb48,2830bc00,0,0,2512c500,2850de88) at vn_open_cred+0x2f8/frame 0x2830bae0 vn_open(2830bb48,2830bc00,0,2850de88) at vn_open+0x20/frame 0x2830bb00 kern_openat(2817a700,ffffff9c,ffbfe264,0,100001,0) at kern_openat+0x2fa/frame 0x2830bbe8 sys_open(2817a700,2817a998,2817a700,2813f000,2817a700,...) at sys_open+0x1c/frame 0x2830bc08 syscall(2830bce8,3b,3b,3b,2060b3d4,...) at syscall+0x1b8/frame 0x2830bcdc Xint0x80_syscall() at 0xffc033f9/frame 0x2830bcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command sed pid 7134 tid 104564 td 0x284dc180 (CPU 1) cpustop_handler(325ff6e0,325ff6d4,13d59fb,325ff644,fc54d5,...) at cpustop_handler+0x26/frame 0x325ff60c ipi_nmi_handler(325ff644,fc54d5,b8505b8,8,2c211a20,...) at ipi_nmi_handler+0x30/frame 0x325ff618 trap(325ff6e0,8,28,28,325ff740,...) at trap+0xbb/frame 0x325ff6d4 calltrap() at 0xffc0319f/frame 0x325ff6d4 --- trap 0x13, eip = 0x10a9bdd, esp = 0x325ff720, ebp = 0x325ff768 --- vnlru_free_locked(0,82000,1ff58000,1fe31ac0,325ff7ac,...) at vnlru_free_locked+0x9d/frame 0x325ff768 vn_alloc_hard(35532488,325ff7b4,1281eb8,74,0,...) at vn_alloc_hard+0xaf/frame 0x325ff780 getnewvnode(1413efe,1fe31ac0,1ab0f40,325ff7e4) at getnewvnode+0x261/frame 0x325ff7ac ffs_vgetf(1fe31ac0,6d495,0,202000,325ff844,0) at ffs_vgetf+0xdf/frame 0x325ff7f4 ffs_vget(1fe31ac0,6d495,0,202000,325ff844) at ffs_vget+0x19/frame 0x325ff814 ufs_lookup_ino(27a45468,325ffb84,325ffb9c,0) at ufs_lookup_ino+0xba8/frame 0x325ff8cc ufs_lookup(325ff908) at ufs_lookup+0x16/frame 0x325ff8e4 VOP_CACHEDLOOKUP_APV(1ab0f40,325ff908) at VOP_CACHEDLOOKUP_APV+0x4d/frame 0x325ff8f8 vfs_cache_lookup(325ff960) at vfs_cache_lookup+0x98/frame 0x325ff92c VOP_LOOKUP_APV(1ab0f40,325ff960) at VOP_LOOKUP_APV+0x4d/frame 0x325ff940 lookup(325ffb48) at lookup+0x580/frame 0x325ff994 namei(325ffb48) at namei+0x6df/frame 0x325ffa20 vn_open_cred(325ffb48,325ffc00,0,0,2512c500,2ae681e0) at vn_open_cred+0x2f8/frame 0x325ffae0 vn_open(325ffb48,325ffc00,0,2ae681e0) at vn_open+0x20/frame 0x325ffb00 kern_openat(284dc180,ffffff9c,ffbfe294,0,100001,0) at kern_openat+0x2fa/frame 0x325ffbe8 sys_open(284dc180,284dc418,284dc180,2c211a20,284dc180,...) at sys_open+0x1c/frame 0x325ffc08 syscall(325ffce8,3b,3b,3b,2060b3d4,...) at syscall+0x1b8/frame 0x325ffcdc Xint0x80_syscall() at 0xffc033f9/frame 0x325ffcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe1ac --- Tracing command top pid 2840 tid 100514 td 0x284e9700 (CPU 3) kdb_alt_break_internal(0) at kdb_alt_break_internal+0x13b/frame 0x2885a608 kdb_alt_break(2,1832be8c) at kdb_alt_break+0x10/frame 0x2885a614 uart_intr_rxready(1832be00,1dba,b85c5e4,880,b90ca00,...) at uart_intr_rxready+0xec/frame 0x2885a630 uart_intr(1832be00) at uart_intr+0x116/frame 0x2885a654 intr_event_handle(b90ca00,2885a6c8) at intr_event_handle+0x104/frame 0x2885a678 intr_execute_handlers(b9dc93c,2885a6c8) at intr_execute_handlers+0xbe/frame 0x2885a6a4 lapic_handle_intr(32,2885a6c8,8,28,28,...) at lapic_handle_intr+0x52/frame 0x2885a6b8 Xapic_isr1() at 0xffc0424a/frame 0x2885a6b8 --- interrupt, eip = 0xfc4e35, esp = 0x2885a708, ebp = 0x2885a708 --- __mtx_unlock_flags(1a92450,0,1489335,4eb) at __mtx_unlock_flags+0xb5/frame 0x2885a708 vnlru_free_locked(0,82000,1fc22000,18cd6080,2885a7ac,...) at vnlru_free_locked+0x279/frame 0x2885a768 vn_alloc_hard(2481f7b4,2885a7b4,1281eb8,74,0,...) at vn_alloc_hard+0xaf/frame 0x2885a780 getnewvnode(1413efe,18cd6080,1ab0f40,2885a7e4) at getnewvnode+0x261/frame 0x2885a7ac ffs_vgetf(18cd6080,5ea8,0,202000,2885a844,0) at ffs_vgetf+0xdf/frame 0x2885a7f4 ffs_vget(18cd6080,5ea8,0,202000,2885a844) at ffs_vget+0x19/frame 0x2885a814 ufs_lookup_ino(1fc19468,2885ab84,2885ab9c,0) at ufs_lookup_ino+0xba8/frame 0x2885a8cc ufs_lookup(2885a908) at ufs_lookup+0x16/frame 0x2885a8e4 VOP_CACHEDLOOKUP_APV(1ab0f40,2885a908) at VOP_CACHEDLOOKUP_APV+0x4d/frame 0x2885a8f8 vfs_cache_lookup(2885a960) at vfs_cache_lookup+0x98/frame 0x2885a92c VOP_LOOKUP_APV(1ab0f40,2885a960) at VOP_LOOKUP_APV+0x4d/frame 0x2885a940 lookup(2885ab48) at lookup+0x580/frame 0x2885a994 namei(2885ab48) at namei+0x6df/frame 0x2885aa20 vn_open_cred(2885ab48,2885ac00,0,0,235c4b00,2ae503c0) at vn_open_cred+0x2f8/frame 0x2885aae0 vn_open(2885ab48,2885ac00,0,2ae503c0) at vn_open+0x20/frame 0x2885ab00 kern_openat(284e9700,ffffff9c,22f04206,0,100001,0) at kern_openat+0x2fa/frame 0x2885abe8 sys_open(284e9700,284e9998,284e9700,284de000,284e9700,...) at sys_open+0x1c/frame 0x2885ac08 syscall(2885ace8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x2885acdc Xint0x80_syscall() at 0xffc033f9/frame 0x2885acdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command smartd pid 739 tid 100187 td 0x235e6000 (CPU 2) cpustop_handler(205f761c,205f7610,13d59fb,205f75b4,b847000,...) at cpustop_handler+0x26/frame 0x205f7544 ipi_nmi_handler(205f75b4,b847000,205f75ac,0,1fc51344,...) at ipi_nmi_handler+0x30/frame 0x205f7550 trap(205f761c,8,28,28,3d741000,...) at trap+0xbb/frame 0x205f7610 calltrap() at 0xffc0319f/frame 0x205f7610 --- trap 0x13, eip = 0xfc4e9f, esp = 0x205f765c, ebp = 0x205f7668 --- __mtx_assert(3d741010,4,140b7af,2b6) at __mtx_assert+0xf/frame 0x205f7668 vop_stdgetwritemount(205f76b4) at vop_stdgetwritemount+0x27d/frame 0x205f7690 VOP_GETWRITEMOUNT_APV(1ab0f40,205f76b4) at VOP_GETWRITEMOUNT_APV+0x40/frame 0x205f76a4 vn_start_write(206128d0,205f76fc,2) at vn_start_write+0x36/frame 0x205f76cc vnlru_free_locked(0,18855600,18d5eb80,18cd4040,205f776c,...) at vnlru_free_locked+0x31d/frame 0x205f7728 vn_alloc_hard(205f776c,fc4dc7,1d3ff68,8,14d566e,...) at vn_alloc_hard+0xaf/frame 0x205f7740 getnewvnode(1400074,18cd4040,1a65c90,205f7790) at getnewvnode+0x261/frame 0x205f776c devfs_allocv(18d5eb80,18cd4040,80000,205f7b7c) at devfs_allocv+0x202/frame 0x205f77a0 devfs_lookup(205f7958) at devfs_lookup+0x40a/frame 0x205f7924 VOP_LOOKUP_APV(1a65c90,205f7958) at VOP_LOOKUP_APV+0x4d/frame 0x205f7938 lookup(205f7b40) at lookup+0x580/frame 0x205f798c namei(205f7b40) at namei+0x6df/frame 0x205f7a18 vn_open_cred(205f7b40,205f7bf8,0,0,1890cb00,2ae50564) at vn_open_cred+0x2f8/frame 0x205f7ad8 vn_open(205f7b40,205f7bf8,0,2ae50564) at vn_open+0x20/frame 0x205f7af8 kern_openat(235e6000,ffffff9c,204b7ec3,0,3,0) at kern_openat+0x2fa/frame 0x205f7be0 sys_openat(235e6000,235e6298,235e6000,1fc51344,235e6000,...) at sys_openat+0x2f/frame 0x205f7c08 syscall(205f7ce8,3b,3b,3b,20835000,...) at syscall+0x1b8/frame 0x205f7cdc Xint0x80_syscall() at 0xffc033f9/frame 0x205f7cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe808 --- db:0:pho> alltrace Tracing command sshd pid 7141 tid 100993 td 0x29e40880 sched_switch(29e40880,104) at sched_switch+0x4e9/frame 0x2a124680 mi_switch(104) at mi_switch+0x1c4/frame 0x2a1246b8 sleepq_switch(18805bd4,0) at sleepq_switch+0xf7/frame 0x2a1246d8 _sx_xlock_hard(18805bd4,235e6004,0,14d566e,15c) at _sx_xlock_hard+0x41f/frame 0x2a124750 _sx_xlock(18805bd4,0,14d566e,15c) at _sx_xlock+0xa5/frame 0x2a124778 devfs_populate_vp(23aba0e8,2a1247dc,12677f8,1ab1668,2a124800,...) at devfs_populate_vp+0x10d/frame 0x2a1247b0 devfs_lookup(2a124958) at devfs_lookup+0x26/frame 0x2a124924 VOP_LOOKUP_APV(1a65c90,2a124958) at VOP_LOOKUP_APV+0x4d/frame 0x2a124938 lookup(2a124b40) at lookup+0x580/frame 0x2a12498c namei(2a124b40) at namei+0x6df/frame 0x2a124a18 vn_open_cred(2a124b40,2a124bf8,0,0,31b90200,2d3e3a8c) at vn_open_cred+0x2f8/frame 0x2a124ad8 vn_open(2a124b40,2a124bf8,0,2d3e3a8c) at vn_open+0x20/frame 0x2a124af8 kern_openat(29e40880,ffffff9c,204aaad9,0,3,0) at kern_openat+0x2fa/frame 0x2a124be0 sys_openat(29e40880,29e40b18,29e40880,29e32688,29e40880,...) at sys_openat+0x2f/frame 0x2a124c08 syscall(2a124ce8,3b,3b,3b,20482000,...) at syscall+0x1b8/frame 0x2a124cdc Xint0x80_syscall() at 0xffc033f9/frame 0x2a124cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe808 --- Tracing command sendmail pid 7140 tid 101843 td 0x2cc1f180 sched_switch(2cc1f180,104) at sched_switch+0x4e9/frame 0x2cb0d680 mi_switch(104) at mi_switch+0x1c4/frame 0x2cb0d6b8 sleepq_switch(18805bd4,0) at sleepq_switch+0xf7/frame 0x2cb0d6d8 _sx_xlock_hard(18805bd4,235e6004,0,14d566e,15c) at _sx_xlock_hard+0x41f/frame 0x2cb0d750 _sx_xlock(18805bd4,0,14d566e,15c) at _sx_xlock+0xa5/frame 0x2cb0d778 devfs_populate_vp(2cb0d7dc,fc4dc7,1ae975c,8,149fd9d,...) at devfs_populate_vp+0x10d/frame 0x2cb0d7b0 devfs_lookup(2cb0d958) at devfs_lookup+0x26/frame 0x2cb0d924 VOP_LOOKUP_APV(1a65c90,2cb0d958) at VOP_LOOKUP_APV+0x4d/frame 0x2cb0d938 lookup(2cb0db40) at lookup+0x580/frame 0x2cb0d98c namei(2cb0db40) at namei+0x6df/frame 0x2cb0da18 vn_open_cred(2cb0db40,2cb0dbf8,0,0,235cf100,235901a4) at vn_open_cred+0x2f8/frame 0x2cb0dad8 vn_open(2cb0db40,2cb0dbf8,0,235901a4) at vn_open+0x20/frame 0x2cb0daf8 kern_openat(2cc1f180,ffffff9c,414daf,0,1,0) at kern_openat+0x2fa/frame 0x2cb0dbe0 sys_openat(2cc1f180,2cc1f418,2cc1f180,2cc179cc,2cc1f180,...) at sys_openat+0x2f/frame 0x2cb0dc08 syscall(2cb0dce8,3b,3b,3b,20a2c000,...) at syscall+0x1b8/frame 0x2cb0dcdc Xint0x80_syscall() at 0xffc033f9/frame 0x2cb0dcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe808 --- Tracing command sendmail pid 7139 tid 101564 td 0x2bad6a80 sched_switch(2bad6a80,104) at sched_switch+0x4e9/frame 0x2bd27680 mi_switch(104) at mi_switch+0x1c4/frame 0x2bd276b8 sleepq_switch(18805bd4,0) at sleepq_switch+0xf7/frame 0x2bd276d8 _sx_xlock_hard(18805bd4,235e6000,0,14d566e,15c) at _sx_xlock_hard+0x41f/frame 0x2bd27750 _sx_xlock(18805bd4,0,14d566e,15c) at _sx_xlock+0xa5/frame 0x2bd27778 devfs_populate_vp(2bd27828,f457af,23b53cf0,b8e0f40,b8e0f40,...) at devfs_populate_vp+0x10d/frame 0x2bd277b0 devfs_lookup(2bd27958) at devfs_lookup+0x26/frame 0x2bd27924 VOP_LOOKUP_APV(1a65c90,2bd27958) at VOP_LOOKUP_APV+0x4d/frame 0x2bd27938 lookup(2bd27b40) at lookup+0x580/frame 0x2bd2798c namei(2bd27b40) at namei+0x6df/frame 0x2bd27a18 vn_open_cred(2bd27b40,2bd27bf8,0,0,2d3a5d00,2ae50b40) at vn_open_cred+0x2f8/frame 0x2bd27ad8 vn_open(2bd27b40,2bd27bf8,0,2ae50b40) at vn_open+0x20/frame 0x2bd27af8 kern_openat(2bad6a80,ffffff9c,414daf,0,1,0) at kern_openat+0x2fa/frame 0x2bd27be0 sys_openat(2bad6a80,2bad6d18,2bad6a80,2bae06dc,2bad6a80,...) at sys_openat+0x2f/frame 0x2bd27c08 syscall(2bd27ce8,3b,3b,3b,20a2c000,...) at syscall+0x1b8/frame 0x2bd27cdc Xint0x80_syscall() at 0xffc033f9/frame 0x2bd27cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command ls pid 7138 tid 100813 td 0x29817500 sched_switch(29817500,104) at sched_switch+0x4e9/frame 0x296bf730 mi_switch(104) at mi_switch+0x1c4/frame 0x296bf768 sleepq_switch(22c441b4,60) at sleepq_switch+0xf7/frame 0x296bf788 sleeplk(22c44220,1413efe,60,33,1) at sleeplk+0x11d/frame 0x296bf7bc lockmgr_slock_hard(22c44220,1489335,b62,0) at lockmgr_slock_hard+0x394/frame 0x296bf82c lockmgr_lock_flags(22c441b4,206000,22c44220,1489335,b62) at lockmgr_lock_flags+0x1c7/frame 0x296bf850 ffs_lock(296bf8a0) at ffs_lock+0x76/frame 0x296bf87c VOP_LOCK1_APV(1ab0f40,296bf8a0) at VOP_LOCK1_APV+0x38/frame 0x296bf890 _vn_lock(22c44178,202000,1489335,b62) at _vn_lock+0x49/frame 0x296bf8c0 vget_finish(22c44178,202000,2) at vget_finish+0x6a/frame 0x296bf8e4 cache_lookup(1fc19a48,296bfaac,296bfac4,0,0) at cache_lookup+0x3e0/frame 0x296bf91c vfs_cache_lookup(296bf990) at vfs_cache_lookup+0x6e/frame 0x296bf95c VOP_LOOKUP_APV(1ab0f40,296bf990) at VOP_LOOKUP_APV+0x4d/frame 0x296bf970 lookup(296bfa70) at lookup+0x580/frame 0x296bf9c4 namei(296bfa70) at namei+0x6df/frame 0x296bfa4c kern_statat(29817500,200,ffffff9c,206bc4f4,0,296bfb30,0) at kern_statat+0x107/frame 0x296bfb0c sys_fstatat(29817500,29817798,29817500,28d6b344,29817500,...) at sys_fstatat+0x28/frame 0x296bfc08 syscall(296bfce8,3b,3b,3b,206bc424,...) at syscall+0x1b8/frame 0x296bfcdc Xint0x80_syscall() at 0xffc033f9/frame 0x296bfcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe808 --- Tracing command sh pid 7137 tid 101766 td 0x2c38f500 sched_switch(2c38f500,104) at sched_switch+0x4e9/frame 0x2c6549d0 mi_switch(104) at mi_switch+0x1c4/frame 0x2c654a04 sleepq_switch(1c0,14e9ff2,6c,0,1d76494,...) at sleepq_switch+0xf7/frame 0x2c654a24 sleepq_catch_signals(2c654a94,fefb0c,25db4344,6c,fd3fb1,...) at sleepq_catch_signals+0x278/frame 0x2c654a4c sleepq_wait_sig(25db4344,6c) at sleepq_wait_sig+0xe/frame 0x2c654a54 _sleep(25db4344,25db43ec,36c,1506bc5,0,0,0,0,100) at _sleep+0x27c/frame 0x2c654a94 kern_wait6(2c38f500,7,0,0,2c654bf0,30,0,0) at kern_wait6+0x3b8/frame 0x2c654af0 sys_wait4(2c38f500,2c38f798,2c38f500,25db4344,2c38f500,...) at sys_wait4+0x73/frame 0x2c654c08 syscall(2c654ce8,3b,3b,3b,ffbfeb14,...) at syscall+0x1b8/frame 0x2c654cdc Xint0x80_syscall() at 0xffc033f9/frame 0x2c654cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d98, esp = 0xffc0bfe8, ebp = 0xffbfeccc --- Tracing command sh pid 7136 tid 101659 td 0x2bbf3500 sched_switch(2bbf3500,104) at sched_switch+0x4e9/frame 0x2c0aaa70 mi_switch(104) at mi_switch+0x1c4/frame 0x2c0aaaa4 sleepq_switch(240,14e9ff2,5c,0,1d76514,...) at sleepq_switch+0xf7/frame 0x2c0aaac4 sleepq_catch_signals(2c0aab34,fefb0c,34eea1a8,5c,34eea1a8,...) at sleepq_catch_signals+0x278/frame 0x2c0aaaec sleepq_wait_sig(34eea1a8,5c) at sleepq_wait_sig+0xe/frame 0x2c0aaaf4 _sleep(34eea1a8,34eea338,15c,14ea1a2,0,0,0,0,100) at _sleep+0x27c/frame 0x2c0aab34 pipe_read(1fc6cf00,2c0aabd0,2512c500,0,2bbf3500) at pipe_read+0x452/frame 0x2c0aab74 dofileread(1fc6cf00,2c0aabd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x2c0aaba4 sys_read(2bbf3500,2bbf3798,2bbf3500,2bb98344,2bbf3500,...) at sys_read+0x78/frame 0x2c0aac08 syscall(2c0aace8,3b,3b,3b,2,...) at syscall+0x1b8/frame 0x2c0aacdc Xint0x80_syscall() at 0xffc033f9/frame 0x2c0aacdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe808 --- Tracing command sed pid 7135 tid 100390 td 0x2817a700 (CPU 0) cpustop_handler(2830b650,2830b644,13d59fb,1,22ff5b8,...) at cpustop_handler+0x26/frame 0x2830b57c ipi_nmi_handler(1,22ff5b8,42c003b5,1dba,2813f000,...) at ipi_nmi_handler+0x30/frame 0x2830b588 trap(2830b650,8,28,28,2817a700,...) at trap+0xbb/frame 0x2830b644 calltrap() at 0xffc0319f/frame 0x2830b644 --- trap 0x13, eip = 0xfc4b30, esp = 0x2830b690, ebp = 0x2830b6dc --- __mtx_lock_sleep(1a92450,235e6000,0,1489335,4ee,...) at __mtx_lock_sleep+0x1e0/frame 0x2830b6dc __mtx_lock_flags(1a92450,0,1489335,4ee) at __mtx_lock_flags+0xcc/frame 0x2830b708 vnlru_free_locked(0,82000,1ff58000,1fe31ac0,2830b7ac,...) at vnlru_free_locked+0x9a/frame 0x2830b768 vn_alloc_hard(23b4f658,2830b7b4,1281eb8,74,0,...) at vn_alloc_hard+0xaf/frame 0x2830b780 getnewvnode(1413efe,1fe31ac0,1ab0f40,2830b7e4) at getnewvnode+0x261/frame 0x2830b7ac ffs_vgetf(1fe31ac0,6d495,0,202000,2830b844,0) at ffs_vgetf+0xdf/frame 0x2830b7f4 ffs_vget(1fe31ac0,6d495,0,202000,2830b844) at ffs_vget+0x19/frame 0x2830b814 ufs_lookup_ino(27a45468,2830bb84,2830bb9c,0) at ufs_lookup_ino+0xba8/frame 0x2830b8cc ufs_lookup(2830b908) at ufs_lookup+0x16/frame 0x2830b8e4 VOP_CACHEDLOOKUP_APV(1ab0f40,2830b908) at VOP_CACHEDLOOKUP_APV+0x4d/frame 0x2830b8f8 vfs_cache_lookup(2830b960) at vfs_cache_lookup+0x98/frame 0x2830b92c VOP_LOOKUP_APV(1ab0f40,2830b960) at VOP_LOOKUP_APV+0x4d/frame 0x2830b940 lookup(2830bb48) at lookup+0x580/frame 0x2830b994 namei(2830bb48) at namei+0x6df/frame 0x2830ba20 vn_open_cred(2830bb48,2830bc00,0,0,2512c500,2850de88) at vn_open_cred+0x2f8/frame 0x2830bae0 vn_open(2830bb48,2830bc00,0,2850de88) at vn_open+0x20/frame 0x2830bb00 kern_openat(2817a700,ffffff9c,ffbfe264,0,100001,0) at kern_openat+0x2fa/frame 0x2830bbe8 sys_open(2817a700,2817a998,2817a700,2813f000,2817a700,...) at sys_open+0x1c/frame 0x2830bc08 syscall(2830bce8,3b,3b,3b,2060b3d4,...) at syscall+0x1b8/frame 0x2830bcdc Xint0x80_syscall() at 0xffc033f9/frame 0x2830bcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command sed pid 7134 tid 104564 td 0x284dc180 (CPU 1) cpustop_handler(325ff6e0,325ff6d4,13d59fb,325ff644,fc54d5,...) at cpustop_handler+0x26/frame 0x325ff60c ipi_nmi_handler(325ff644,fc54d5,b8505b8,8,2c211a20,...) at ipi_nmi_handler+0x30/frame 0x325ff618 trap(325ff6e0,8,28,28,325ff740,...) at trap+0xbb/frame 0x325ff6d4 calltrap() at 0xffc0319f/frame 0x325ff6d4 --- trap 0x13, eip = 0x10a9bdd, esp = 0x325ff720, ebp = 0x325ff768 --- vnlru_free_locked(0,82000,1ff58000,1fe31ac0,325ff7ac,...) at vnlru_free_locked+0x9d/frame 0x325ff768 vn_alloc_hard(35532488,325ff7b4,1281eb8,74,0,...) at vn_alloc_hard+0xaf/frame 0x325ff780 getnewvnode(1413efe,1fe31ac0,1ab0f40,325ff7e4) at getnewvnode+0x261/frame 0x325ff7ac ffs_vgetf(1fe31ac0,6d495,0,202000,325ff844,0) at ffs_vgetf+0xdf/frame 0x325ff7f4 ffs_vget(1fe31ac0,6d495,0,202000,325ff844) at ffs_vget+0x19/frame 0x325ff814 ufs_lookup_ino(27a45468,325ffb84,325ffb9c,0) at ufs_lookup_ino+0xba8/frame 0x325ff8cc ufs_lookup(325ff908) at ufs_lookup+0x16/frame 0x325ff8e4 VOP_CACHEDLOOKUP_APV(1ab0f40,325ff908) at VOP_CACHEDLOOKUP_APV+0x4d/frame 0x325ff8f8 vfs_cache_lookup(325ff960) at vfs_cache_lookup+0x98/frame 0x325ff92c VOP_LOOKUP_APV(1ab0f40,325ff960) at VOP_LOOKUP_APV+0x4d/frame 0x325ff940 lookup(325ffb48) at lookup+0x580/frame 0x325ff994 namei(325ffb48) at namei+0x6df/frame 0x325ffa20 vn_open_cred(325ffb48,325ffc00,0,0,2512c500,2ae681e0) at vn_open_cred+0x2f8/frame 0x325ffae0 vn_open(325ffb48,325ffc00,0,2ae681e0) at vn_open+0x20/frame 0x325ffb00 kern_openat(284dc180,ffffff9c,ffbfe294,0,100001,0) at kern_openat+0x2fa/frame 0x325ffbe8 sys_open(284dc180,284dc418,284dc180,2c211a20,284dc180,...) at sys_open+0x1c/frame 0x325ffc08 syscall(325ffce8,3b,3b,3b,2060b3d4,...) at syscall+0x1b8/frame 0x325ffcdc Xint0x80_syscall() at 0xffc033f9/frame 0x325ffcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe1ac --- Tracing command umount pid 7132 tid 101563 td 0x2bad6e00 sched_switch(2bad6e00,103) at sched_switch+0x4e9/frame 0x2bd1e6c8 mi_switch(103,1d7ac80) at mi_switch+0x1c4/frame 0x2bd1e700 turnstile_wait(27be8c80,1fc9e000,0) at turnstile_wait+0x2f1/frame 0x2bd1e71c __mtx_lock_sleep(3d741010,1fc9e002,0,140b7af,2b4,...) at __mtx_lock_sleep+0x1b1/frame 0x2bd1e77c __mtx_lock_flags(3d741010,0,140b7af,2b4) at __mtx_lock_flags+0xcc/frame 0x2bd1e7a8 vop_stdgetwritemount(2bd1e7f4) at vop_stdgetwritemount+0x23a/frame 0x2bd1e7d0 VOP_GETWRITEMOUNT_APV(1ab0f40,2bd1e7f4) at VOP_GETWRITEMOUNT_APV+0x40/frame 0x2bd1e7e4 vn_start_write(32dfd2f0,2bd1e83c,2) at vn_start_write+0x36/frame 0x2bd1e80c vnlru_free_locked(0,80200,2d2b1000,3d741000,2bd1e8b0,...) at vnlru_free_locked+0x31d/frame 0x2bd1e868 vn_alloc_hard(74,256ec32c,2bd1e8b8,1281eb8,74,...) at vn_alloc_hard+0xaf/frame 0x2bd1e880 getnewvnode(1413efe,3d741000,1ab0f40,2bd1e8e8) at getnewvnode+0x261/frame 0x2bd1e8b0 ffs_vgetf(3d741000,29a9a,0,80200,2bd1e930,1) at ffs_vgetf+0xdf/frame 0x2bd1e8f8 handle_workitem_remove(fc4dc7,8,2bad6e00,22cc4000,a,...) at handle_workitem_remove+0x49/frame 0x2bd1e960 process_worklist_item(200) at process_worklist_item+0x250/frame 0x2bd1e9b8 softdep_process_worklist(2bd1ea10,10aebd0,3d741010,0,1489335,...) at softdep_process_worklist+0xb3/frame 0x2bd1e9e0 softdep_flushworklist(3d741000,2bd1ea68,2bad6e00) at softdep_flushworklist+0x3d/frame 0x2bd1ea0c ffs_sync(3d741000,4) at ffs_sync+0x3a4/frame 0x2bd1ea7c vfs_write_suspend(3d741000,0) at vfs_write_suspend+0x144/frame 0x2bd1ea9c vfs_write_suspend_umnt(3d741000) at vfs_write_suspend_umnt+0x2f/frame 0x2bd1eabc ffs_unmount(3d741000,8000000) at ffs_unmount+0x64/frame 0x2bd1eb08 dounmount(3d741000,8000000,2bad6e00) at dounmount+0x403/frame 0x2bd1eb3c kern_unmount(2bad6e00,ffbfd7f4,8000000) at kern_unmount+0x2dd/frame 0x2bd1ebf4 sys_unmount(2bad6e00,2bad7098,2bad6e00,2bae0a20,2bad6e00,...) at sys_unmount+0x13/frame 0x2bd1ec08 syscall(2bd1ece8,3b,3b,3b,20632218,...) at syscall+0x1b8/frame 0x2bd1ecdc Xint0x80_syscall() at 0xffc033f9/frame 0x2bd1ecdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe1ac --- Tracing command md10 pid 1489 tid 102571 td 0x2ee6f000 sched_switch(2ee6f000,104) at sched_switch+0x4e9/frame 0x2f0bdbd8 mi_switch(104) at mi_switch+0x1c4/frame 0x2f0bdc10 sleepq_switch(3881a000,5c) at sleepq_switch+0xf7/frame 0x2f0bdc30 _sleep(3881a000,3881a028,25c,14ab88f,0,0,0,0,100) at _sleep+0x2f3/frame 0x2f0bdc70 md_kthread(3881a000,2f0bdce8) at md_kthread+0xc1/frame 0x2f0bdcb8 fork_exit(bdef50,3881a000,2f0bdce8,0,0,...) at fork_exit+0x68/frame 0x2f0bdcd4 fork_trampoline() at 0xffc0340e/frame 0x2f0bdcd4 --- kthread start Tracing command sh pid 1237 tid 100916 td 0x299a4880 sched_switch(299a4880,104) at sched_switch+0x4e9/frame 0x29c749d0 mi_switch(104) at mi_switch+0x1c4/frame 0x29c74a04 sleepq_switch(3900,14e9ff2,6c,0,1d79bd4,...) at sleepq_switch+0xf7/frame 0x29c74a24 sleepq_catch_signals(29c74a94,fefb0c,2999b054,6c,fd3fb1,...) at sleepq_catch_signals+0x278/frame 0x29c74a4c sleepq_wait_sig(2999b054,6c) at sleepq_wait_sig+0xe/frame 0x29c74a54 _sleep(2999b054,2999b0fc,36c,1506bc5,0,0,0,0,100) at _sleep+0x27c/frame 0x29c74a94 kern_wait6(299a4880,7,0,0,29c74bf0,30,0,0) at kern_wait6+0x3b8/frame 0x29c74af0 sys_wait4(299a4880,299a4b18,299a4880,2999b054,299a4880,...) at sys_wait4+0x73/frame 0x29c74c08 syscall(29c74ce8,3b,3b,3b,ffbfe490,...) at syscall+0x1b8/frame 0x29c74cdc Xint0x80_syscall() at 0xffc033f9/frame 0x29c74cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command tee pid 1236 tid 101566 td 0x2bad6380 sched_switch(2bad6380,104) at sched_switch+0x4e9/frame 0x2bd39a70 mi_switch(104) at mi_switch+0x1c4/frame 0x2bd39aa4 sleepq_switch(1240,14e9ff2,5c,0,1d77514,...) at sleepq_switch+0xf7/frame 0x2bd39ac4 sleepq_catch_signals(2bd39b34,fefb0c,3df6e1a8,5c,3df6e1a8,...) at sleepq_catch_signals+0x278/frame 0x2bd39aec sleepq_wait_sig(3df6e1a8,5c) at sleepq_wait_sig+0xe/frame 0x2bd39af4 _sleep(3df6e1a8,3df6e338,15c,14ea1a2,0,0,0,0,100) at _sleep+0x27c/frame 0x2bd39b34 pipe_read(235908e8,2bd39bd0,298a0100,0,2bad6380) at pipe_read+0x452/frame 0x2bd39b74 dofileread(235908e8,2bd39bd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x2bd39ba4 sys_read(2bad6380,2bad6618,2bad6380,2bae0054,3faf3737,...) at sys_read+0x78/frame 0x2bd39c08 syscall(2bd39ce8,3b,3b,3b,2062c046,...) at syscall+0x1b8/frame 0x2bd39cdc Xint0x80_syscall() at 0xffc033f9/frame 0x2bd39cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command sh pid 1235 tid 103742 td 0x2d868380 sched_switch(2d868380,104) at sched_switch+0x4e9/frame 0x324ab9d0 mi_switch(104) at mi_switch+0x1c4/frame 0x324aba04 sleepq_switch(2800,14e9ff2,6c,0,1d78ad4,...) at sleepq_switch+0xf7/frame 0x324aba24 sleepq_catch_signals(324aba94,fefb0c,2bb7a000,6c,fd3fb1,...) at sleepq_catch_signals+0x278/frame 0x324aba4c sleepq_wait_sig(2bb7a000,6c) at sleepq_wait_sig+0xe/frame 0x324aba54 _sleep(2bb7a000,2bb7a0a8,36c,1506bc5,0,0,0,0,100) at _sleep+0x27c/frame 0x324aba94 kern_wait6(2d868380,7,0,0,324abbf0,30,0,0) at kern_wait6+0x3b8/frame 0x324abaf0 sys_wait4(2d868380,2d868618,2d868380,2bb7a000,2d868380,...) at sys_wait4+0x73/frame 0x324abc08 syscall(324abce8,3b,3b,3b,ffbfe590,...) at syscall+0x1b8/frame 0x324abcdc Xint0x80_syscall() at 0xffc033f9/frame 0x324abcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe808 --- Tracing command tail pid 6180 tid 101606 td 0x2bb6d700 sched_switch(2bb6d700,104) at sched_switch+0x4e9/frame 0x2beb5970 mi_switch(104) at mi_switch+0x1c4/frame 0x2beb59a8 sleepq_switch(740,2bb6d700,0,0,1d76a14,...) at sleepq_switch+0xf7/frame 0x2beb59c8 sleepq_catch_signals(2beb5a30,f84428,1880b9a4,0,1880b990,...) at sleepq_catch_signals+0x278/frame 0x2beb59f0 sleepq_wait_sig(1880b9a4,0) at sleepq_wait_sig+0xe/frame 0x2beb59f8 _cv_wait_sig(1880b9a4,1880b990) at _cv_wait_sig+0x138/frame 0x2beb5a30 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x2beb5a4c kern_select(2bb6d700,7,ffbfe7a4,0,0,0,20) at kern_select+0x8e5/frame 0x2beb5bd4 sys_select(2bb6d700,2bb6d998,2bb6d700,2bb7b398,3faf3737,...) at sys_select+0x41/frame 0x2beb5c08 syscall(2beb5ce8,3b,3b,3b,7,...) at syscall+0x1b8/frame 0x2beb5cdc Xint0x80_syscall() at 0xffc033f9/frame 0x2beb5cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d98, esp = 0xffc0bfe8, ebp = 0xffbfeccc --- Tracing command tail pid 6144 tid 101657 td 0x2bbf3c00 sched_switch(2bbf3c00,104) at sched_switch+0x4e9/frame 0x2c0987c8 mi_switch(104) at mi_switch+0x1c4/frame 0x2c098800 sleepq_switch(2900,0,68,0,1d78bd4,...) at sleepq_switch+0xf7/frame 0x2c098820 sleepq_catch_signals(14e9ff2,183,2bbf3c00,100,9bca400,...) at sleepq_catch_signals+0x278/frame 0x2c098848 sleepq_timedwait_sig(9bca400,68,2c0988b4,3faf3737,1ab57f0,...) at sleepq_timedwait_sig+0x17/frame 0x2c098874 _sleep(9bca400,9bca400,168,14b8eb8,c951875f,1dba,10000000,0,200) at _sleep+0x21a/frame 0x2c0988b4 kqueue_kevent(0,1,2c098be0,2c098bb0) at kqueue_kevent+0xa0f/frame 0x2c098b14 kern_kevent_fp(2bbf3c00,2b58a3fc,0,1,2c098be0,2c098bb0) at kern_kevent_fp+0x71/frame 0x2c098b38 kern_kevent(2bbf3c00,8,0,1,2c098be0,2c098bb0) at kern_kevent+0x73/frame 0x2c098b88 kern_kevent_generic(2c098be0,1506b3e) at kern_kevent_generic+0x5c/frame 0x2c098bd0 sys_kevent(2bbf3c00,2bbf3e98,2bbf3c00,2bb989cc,3faf3737,...) at sys_kevent+0x64/frame 0x2c098c08 syscall(2c098ce8,3b,3b,3b,294a1288,...) at syscall+0x1b8/frame 0x2c098cdc Xint0x80_syscall() at 0xffc033f9/frame 0x2c098cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d98, esp = 0xffc0bfe8, ebp = 0xffbfeccc --- Tracing command top pid 2840 tid 100514 td 0x284e9700 (CPU 3) kdb_alt_break_internal(0) at kdb_alt_break_internal+0x13b/frame 0x2885a608 kdb_alt_break(2,1832be8c) at kdb_alt_break+0x10/frame 0x2885a614 uart_intr_rxready(1832be00,1dba,b85c5e4,880,b90ca00,...) at uart_intr_rxready+0xec/frame 0x2885a630 uart_intr(1832be00) at uart_intr+0x116/frame 0x2885a654 intr_event_handle(b90ca00,2885a6c8) at intr_event_handle+0x104/frame 0x2885a678 intr_execute_handlers(b9dc93c,2885a6c8) at intr_execute_handlers+0xbe/frame 0x2885a6a4 lapic_handle_intr(32,2885a6c8,8,28,28,...) at lapic_handle_intr+0x52/frame 0x2885a6b8 Xapic_isr1() at 0xffc0424a/frame 0x2885a6b8 --- interrupt, eip = 0xfc4e35, esp = 0x2885a708, ebp = 0x2885a708 --- __mtx_unlock_flags(1a92450,0,1489335,4eb) at __mtx_unlock_flags+0xb5/frame 0x2885a708 vnlru_free_locked(0,82000,1fc22000,18cd6080,2885a7ac,...) at vnlru_free_locked+0x279/frame 0x2885a768 vn_alloc_hard(2481f7b4,2885a7b4,1281eb8,74,0,...) at vn_alloc_hard+0xaf/frame 0x2885a780 getnewvnode(1413efe,18cd6080,1ab0f40,2885a7e4) at getnewvnode+0x261/frame 0x2885a7ac ffs_vgetf(18cd6080,5ea8,0,202000,2885a844,0) at ffs_vgetf+0xdf/frame 0x2885a7f4 ffs_vget(18cd6080,5ea8,0,202000,2885a844) at ffs_vget+0x19/frame 0x2885a814 ufs_lookup_ino(1fc19468,2885ab84,2885ab9c,0) at ufs_lookup_ino+0xba8/frame 0x2885a8cc ufs_lookup(2885a908) at ufs_lookup+0x16/frame 0x2885a8e4 VOP_CACHEDLOOKUP_APV(1ab0f40,2885a908) at VOP_CACHEDLOOKUP_APV+0x4d/frame 0x2885a8f8 vfs_cache_lookup(2885a960) at vfs_cache_lookup+0x98/frame 0x2885a92c VOP_LOOKUP_APV(1ab0f40,2885a960) at VOP_LOOKUP_APV+0x4d/frame 0x2885a940 lookup(2885ab48) at lookup+0x580/frame 0x2885a994 namei(2885ab48) at namei+0x6df/frame 0x2885aa20 vn_open_cred(2885ab48,2885ac00,0,0,235c4b00,2ae503c0) at vn_open_cred+0x2f8/frame 0x2885aae0 vn_open(2885ab48,2885ac00,0,2ae503c0) at vn_open+0x20/frame 0x2885ab00 kern_openat(284e9700,ffffff9c,22f04206,0,100001,0) at kern_openat+0x2fa/frame 0x2885abe8 sys_open(284e9700,284e9998,284e9700,284de000,284e9700,...) at sys_open+0x1c/frame 0x2885ac08 syscall(2885ace8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x2885acdc Xint0x80_syscall() at 0xffc033f9/frame 0x2885acdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command bash pid 2309 tid 100123 td 0x1fcae700 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(200046,1a63644,2885a10c,103cc0c,1a63644,...) at db_trace_self_wrapper+0x2a/frame 0x2885a0c8 kdb_backtrace(2885a1f0,13d5a2b,143607a,0,2885a228,...) at kdb_backtrace+0x2e/frame 0x2885a128 kdb_reenter(143607a,0,2885a228,2885a204,284de000,...) at kdb_reenter+0x29/frame 0x2885a130 trap(2885a1fc,8,28,28,ffffffff,...) at trap+0xeb/frame 0x2885a1f0 calltrap() at 0xffc0319f/frame 0x2885a1f0 --- trap 0xc, eip = 0x13b215a, esp = 0x2885a23c, ebp = 0x2885a248 --- db_trace_thread(1fcae700,ffffffff) at db_trace_thread+0x2a/frame 0x2885a248 _db_stack_trace_all(102809b,0,ffffffff,2885a2d4) at _db_stack_trace_all+0x93/frame 0x2885a2a8 db_command(1aec772) at db_command+0x26a/frame 0x2885a35c db_script_exec(bd2d8f,0,2885a404,2885a3e8) at db_script_exec+0x1a6/frame 0x2885a3bc db_command(2885a4c0,a06536,bd3b00,c960,284e9700,...) at db_command+0x26a/frame 0x2885a470 db_command_loop(bd3b00,c960,284e9700,2885a478,2885a4c0,...) at db_command_loop+0x3c/frame 0x2885a478 db_trap(3,0) at db_trap+0xf6/frame 0x2885a4c0 kdb_trap(3,0,2885a5c0) at kdb_trap+0x109/frame 0x2885a4ec trap(2885a5c0,8,28,28,16,...) at trap+0x382/frame 0x2885a5b4 calltrap() at 0xffc0319f/frame 0x2885a5b4 --- trap 0x3, eip = 0x102809b, esp = 0x2885a600, ebp = 0x2885a608 --- kdb_alt_break_internal(0) at kdb_alt_break_internal+0x13b/frame 0x2885a608 kdb_alt_break(2,1832be8c) at kdb_alt_break+0x10/frame 0x2885a614 uart_intr_rxready(1832be00,1dba,b85c5e4,880,b90ca00,...) at uart_intr_rxready+0xec/frame 0x2885a630 uart_intr(1832be00) at uart_intr+0x116/frame 0x2885a654 intr_event_handle(b90ca00,2885a6c8) at intr_event_handle+0x104/frame 0x2885a678 intr_execute_handlers(b9dc93c,2885a6c8) at intr_execute_handlers+0xbe/frame 0x2885a6a4 lapic_handle_intr(32,2885a6c8,8,28,28,...) at lapic_handle_intr+0x52/frame 0x2885a6b8 Xapic_isr1() at 0xffc0424a/frame 0x2885a6b8 --- interrupt, eip = 0xfc4e35, esp = 0x2885a708, ebp = 0x2885a708 --- __mtx_unlock_flags(1a92450,0,1489335,4eb) at __mtx_unlock_flags+0xb5/frame 0x2885a708 vnlru_free_locked(0,82000,1fc22000,18cd6080,2885a7ac,...) at vnlru_free_locked+0x279/frame 0x2885a768 vn_alloc_hard(2481f7b4,2885a7b4,1281eb8,74,0,...) at vn_alloc_hard+0xaf/frame 0x2885a780 getnewvnode(1413efe,18cd6080,1ab0f40,2885a7e4) at getnewvnode+0x261/frame 0x2885a7ac ffs_vgetf(18cd6080,5ea8,0,202000,2885a844,0) at ffs_vgetf+0xdf/frame 0x2885a7f4 ffs_vget(18cd6080,5ea8,0,202000,2885a844) at ffs_vget+0x19/frame 0x2885a814 ufs_lookup_ino(1fc19468,2885ab84,2885ab9c,0) at ufs_lookup_ino+0xba8/frame 0x2885a8cc ufs_lookup(2885a908) at ufs_lookup+0x16/frame 0x2885a8e4 VOP_CACHEDLOOKUP_APV(1ab0f40,2885a908) at VOP_CACHEDLOOKUP_APV+0x4d/frame 0x2885a8f8 vfs_cache_lookup(2885a960) at vfs_cache_lookup+0x98/frame 0x2885a92c VOP_LOOKUP_APV(1ab0f40,2885a960) at VOP_LOOKUP_APV+0x4d/frame 0x2885a940 lookup(2885ab48) at lookup+0x580/frame 0x2885a994 namei(2885ab48) at namei+0x6df/frame 0x2885aa20 vn_open_cred(2885ab48,2885ac00,0,0,235c4b00,2ae503c0) at vn_open_cred+0x2f8/frame 0x2885aae0 vn_open(2885ab48,2885ac00,0,2ae503c0) at vn_open+0x20/frame 0x2885ab00 kern_openat(284e9700,ffffff9c,22f04206,0,100001,0) at kern_openat+0x2fa/frame 0x2885abe8 sys_open(284e9700,284e9998,284e9700,284de000,284e9700,...) at sys_open+0x1c/frame 0x2885ac08 syscall(2885ace8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x2885acdc Xint0x80_syscall() at 0xffc033f9/frame 0x2885acdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command sshd pid 2224 tid 100230 td 0x2785b000 sched_switch(2785b000,104) at sched_switch+0x4e9/frame 0x23991970 mi_switch(104) at mi_switch+0x1c4/frame 0x239919a8 sleepq_switch(2b00,2785b000,0,0,1d78dd4,...) at sleepq_switch+0xf7/frame 0x239919c8 sleepq_catch_signals(23991a30,f84428,9ab08a4,0,9ab0890,...) at sleepq_catch_signals+0x278/frame 0x239919f0 sleepq_wait_sig(9ab08a4,0) at sleepq_wait_sig+0xe/frame 0x239919f8 _cv_wait_sig(9ab08a4,9ab0890) at _cv_wait_sig+0x138/frame 0x23991a30 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x23991a4c kern_select(2785b000,e,2047c270,2047c278,0,0,20) at kern_select+0x8e5/frame 0x23991bd4 sys_select(2785b000,2785b298,2785b000,261ccd10,2785b000,...) at sys_select+0x41/frame 0x23991c08 syscall(23991ce8,3b,3b,3b,20482000,...) at syscall+0x1b8/frame 0x23991cdc Xint0x80_syscall() at 0xffc033f9/frame 0x23991cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe1ac --- Tracing command sshd pid 2162 tid 100229 td 0x2785b380 sched_switch(2785b380,104) at sched_switch+0x4e9/frame 0x239889a0 mi_switch(104) at mi_switch+0x1c4/frame 0x239889d8 sleepq_switch(2f00,2785b380,0,0,1d791d4,...) at sleepq_switch+0xf7/frame 0x239889f8 sleepq_catch_signals(23988a60,f84428,18d5d864,0,140171b,...) at sleepq_catch_signals+0x278/frame 0x23988a20 sleepq_wait_sig(18d5d864,0) at sleepq_wait_sig+0xe/frame 0x23988a28 _cv_wait_sig(18d5d864,18d5d850) at _cv_wait_sig+0x138/frame 0x23988a60 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x23988a7c kern_poll(2785b380,ffbfe7a4,1,0,0) at kern_poll+0x4a4/frame 0x23988bd4 sys_poll(2785b380,2785b618,2785b380,261cd054,2785b380,...) at sys_poll+0x50/frame 0x23988c08 syscall(23988ce8,3b,3b,3b,20482000,...) at syscall+0x1b8/frame 0x23988cdc Xint0x80_syscall() at 0xffc033f9/frame 0x23988cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe1ac --- Tracing command sh pid 1380 tid 100190 td 0x236e3c00 sched_switch(236e3c00,104) at sched_switch+0x4e9/frame 0x23809a70 mi_switch(104) at mi_switch+0x1c4/frame 0x23809aa4 sleepq_switch(4c0,14e9ff2,5c,0,1d76794,...) at sleepq_switch+0xf7/frame 0x23809ac4 sleepq_catch_signals(23809b34,fefb0c,23394350,5c,23394350,...) at sleepq_catch_signals+0x278/frame 0x23809aec sleepq_wait_sig(23394350,5c) at sleepq_wait_sig+0xe/frame 0x23809af4 _sleep(23394350,233944e0,15c,14ea1a2,0,0,0,0,100) at _sleep+0x27c/frame 0x23809b34 pipe_read(1fc5b4ec,23809bd0,2512c500,0,236e3c00) at pipe_read+0x452/frame 0x23809b74 dofileread(1fc5b4ec,23809bd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x23809ba4 sys_read(236e3c00,236e3e98,236e3c00,1fc51000,236e3c00,...) at sys_read+0x78/frame 0x23809c08 syscall(23809ce8,3b,3b,3b,2,...) at syscall+0x1b8/frame 0x23809cdc Xint0x80_syscall() at 0xffc033f9/frame 0x23809cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d98, esp = 0xffc0bfe8, ebp = 0xffbfeccc --- Tracing command awk pid 1378 tid 100120 td 0x18d77700 sched_switch(18d77700,104) at sched_switch+0x4e9/frame 0x1fb24a70 mi_switch(104) at mi_switch+0x1c4/frame 0x1fb24aa4 sleepq_switch(2f00,14e9ff2,5c,0,1d791d4,...) at sleepq_switch+0xf7/frame 0x1fb24ac4 sleepq_catch_signals(1fb24b34,fefb0c,233944f8,5c,233944f8,...) at sleepq_catch_signals+0x278/frame 0x1fb24aec sleepq_wait_sig(233944f8,5c) at sleepq_wait_sig+0xe/frame 0x1fb24af4 _sleep(233944f8,23394688,15c,14ea1a2,0,0,0,0,100) at _sleep+0x27c/frame 0x1fb24b34 pipe_read(1fc5b870,1fb24bd0,2512c500,0,18d77700) at pipe_read+0x452/frame 0x1fb24b74 dofileread(1fc5b870,1fb24bd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x1fb24ba4 sys_read(18d77700,18d77998,18d77700,1fe2b6dc,22ff5b8,...) at sys_read+0x78/frame 0x1fb24c08 syscall(1fb24ce8,3b,3b,3b,2065ff2c,...) at syscall+0x1b8/frame 0x1fb24cdc Xint0x80_syscall() at 0xffc033f9/frame 0x1fb24cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d98, esp = 0xffc0bfe8, ebp = 0xffbfeccc --- Tracing command sh pid 1377 tid 100107 td 0x1fc9fc00 sched_switch(1fc9fc00,104) at sched_switch+0x4e9/frame 0x1fab79d0 mi_switch(104) at mi_switch+0x1c4/frame 0x1fab7a04 sleepq_switch(e80,14e9ff2,6c,0,1d77154,...) at sleepq_switch+0xf7/frame 0x1fab7a24 sleepq_catch_signals(1fab7a94,fefb0c,1834e6dc,6c,fd3fb1,...) at sleepq_catch_signals+0x278/frame 0x1fab7a4c sleepq_wait_sig(1834e6dc,6c) at sleepq_wait_sig+0xe/frame 0x1fab7a54 _sleep(1834e6dc,1834e784,36c,1506bc5,0,0,0,0,100) at _sleep+0x27c/frame 0x1fab7a94 kern_wait6(1fc9fc00,7,0,0,1fab7bf0,30,0,0) at kern_wait6+0x3b8/frame 0x1fab7af0 sys_wait4(1fc9fc00,1fc9fe98,1fc9fc00,1834e6dc,1fc9fc00,...) at sys_wait4+0x73/frame 0x1fab7c08 syscall(1fab7ce8,3b,3b,3b,ffbfe548,...) at syscall+0x1b8/frame 0x1fab7cdc Xint0x80_syscall() at 0xffc033f9/frame 0x1fab7cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d98, esp = 0xffc0bfe8, ebp = 0xffbfeccc --- Tracing command sh pid 1356 tid 100217 td 0x2785c880 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(200046,1a63644,2885a10c,103cc0c,1a63644,...) at db_trace_self_wrapper+0x2a/frame 0x2885a0c8 kdb_backtrace(2885a1f0,13d5a2b,143607a,0,2885a228,...) at kdb_backtrace+0x2e/frame 0x2885a128 kdb_reenter(143607a,0,2885a228,2885a204,284de000,...) at kdb_reenter+0x29/frame 0x2885a130 trap(2885a1fc,8,28,28,ffffffff,...) at trap+0xeb/frame 0x2885a1f0 calltrap() at 0xffc0319f/frame 0x2885a1f0 --- trap 0xc, eip = 0x13b215a, esp = 0x2885a23c, ebp = 0x2885a248 --- db_trace_thread(2785c880,ffffffff) at db_trace_thread+0x2a/frame 0x2885a248 _db_stack_trace_all(102809b,0,ffffffff,2885a2d4) at _db_stack_trace_all+0x93/frame 0x2885a2a8 db_command(1aec772) at db_command+0x26a/frame 0x2885a35c db_script_exec(bd2d8f,0,2885a404,2885a3e8) at db_script_exec+0x1a6/frame 0x2885a3bc db_command(2885a4c0,a06536,bd3b00,c960,284e9700,...) at db_command+0x26a/frame 0x2885a470 db_command_loop(bd3b00,c960,284e9700,2885a478,2885a4c0,...) at db_command_loop+0x3c/frame 0x2885a478 db_trap(3,0) at db_trap+0xf6/frame 0x2885a4c0 kdb_trap(3,0,2885a5c0) at kdb_trap+0x109/frame 0x2885a4ec trap(2885a5c0,8,28,28,16,...) at trap+0x382/frame 0x2885a5b4 calltrap() at 0xffc0319f/frame 0x2885a5b4 --- trap 0x3, eip = 0x102809b, esp = 0x2885a600, ebp = 0x2885a608 --- kdb_alt_break_internal(0) at kdb_alt_break_internal+0x13b/frame 0x2885a608 kdb_alt_break(2,1832be8c) at kdb_alt_break+0x10/frame 0x2885a614 uart_intr_rxready(1832be00,1dba,b85c5e4,880,b90ca00,...) at uart_intr_rxready+0xec/frame 0x2885a630 uart_intr(1832be00) at uart_intr+0x116/frame 0x2885a654 intr_event_handle(b90ca00,2885a6c8) at intr_event_handle+0x104/frame 0x2885a678 intr_execute_handlers(b9dc93c,2885a6c8) at intr_execute_handlers+0xbe/frame 0x2885a6a4 lapic_handle_intr(32,2885a6c8,8,28,28,...) at lapic_handle_intr+0x52/frame 0x2885a6b8 Xapic_isr1() at 0xffc0424a/frame 0x2885a6b8 --- interrupt, eip = 0xfc4e35, esp = 0x2885a708, ebp = 0x2885a708 --- __mtx_unlock_flags(1a92450,0,1489335,4eb) at __mtx_unlock_flags+0xb5/frame 0x2885a708 vnlru_free_locked(0,82000,1fc22000,18cd6080,2885a7ac,...) at vnlru_free_locked+0x279/frame 0x2885a768 vn_alloc_hard(2481f7b4,2885a7b4,1281eb8,74,0,...) at vn_alloc_hard+0xaf/frame 0x2885a780 getnewvnode(1413efe,18cd6080,1ab0f40,2885a7e4) at getnewvnode+0x261/frame 0x2885a7ac ffs_vgetf(18cd6080,5ea8,0,202000,2885a844,0) at ffs_vgetf+0xdf/frame 0x2885a7f4 ffs_vget(18cd6080,5ea8,0,202000,2885a844) at ffs_vget+0x19/frame 0x2885a814 ufs_lookup_ino(1fc19468,2885ab84,2885ab9c,0) at ufs_lookup_ino+0xba8/frame 0x2885a8cc ufs_lookup(2885a908) at ufs_lookup+0x16/frame 0x2885a8e4 VOP_CACHEDLOOKUP_APV(1ab0f40,2885a908) at VOP_CACHEDLOOKUP_APV+0x4d/frame 0x2885a8f8 vfs_cache_lookup(2885a960) at vfs_cache_lookup+0x98/frame 0x2885a92c VOP_LOOKUP_APV(1ab0f40,2885a960) at VOP_LOOKUP_APV+0x4d/frame 0x2885a940 lookup(2885ab48) at lookup+0x580/frame 0x2885a994 namei(2885ab48) at namei+0x6df/frame 0x2885aa20 vn_open_cred(2885ab48,2885ac00,0,0,235c4b00,2ae503c0) at vn_open_cred+0x2f8/frame 0x2885aae0 vn_open(2885ab48,2885ac00,0,2ae503c0) at vn_open+0x20/frame 0x2885ab00 kern_openat(284e9700,ffffff9c,22f04206,0,100001,0) at kern_openat+0x2fa/frame 0x2885abe8 sys_open(284e9700,284e9998,284e9700,284de000,284e9700,...) at sys_open+0x1c/frame 0x2885ac08 syscall(2885ace8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x2885acdc Xint0x80_syscall() at 0xffc033f9/frame 0x2885acdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command bash pid 1162 tid 100138 td 0x22bc0e00 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(200046,1a63644,2885a10c,103cc0c,1a63644,...) at db_trace_self_wrapper+0x2a/frame 0x2885a0c8 kdb_backtrace(2885a1f0,13d5a2b,143607a,0,2885a228,...) at kdb_backtrace+0x2e/frame 0x2885a128 kdb_reenter(143607a,0,2885a228,2885a204,284de000,...) at kdb_reenter+0x29/frame 0x2885a130 trap(2885a1fc,8,28,28,ffffffff,...) at trap+0xeb/frame 0x2885a1f0 calltrap() at 0xffc0319f/frame 0x2885a1f0 --- trap 0xc, eip = 0x13b215a, esp = 0x2885a23c, ebp = 0x2885a248 --- db_trace_thread(22bc0e00,ffffffff) at db_trace_thread+0x2a/frame 0x2885a248 _db_stack_trace_all(102809b,0,ffffffff,2885a2d4) at _db_stack_trace_all+0x93/frame 0x2885a2a8 db_command(1aec772) at db_command+0x26a/frame 0x2885a35c db_script_exec(bd2d8f,0,2885a404,2885a3e8) at db_script_exec+0x1a6/frame 0x2885a3bc db_command(2885a4c0,a06536,bd3b00,c960,284e9700,...) at db_command+0x26a/frame 0x2885a470 db_command_loop(bd3b00,c960,284e9700,2885a478,2885a4c0,...) at db_command_loop+0x3c/frame 0x2885a478 db_trap(3,0) at db_trap+0xf6/frame 0x2885a4c0 kdb_trap(3,0,2885a5c0) at kdb_trap+0x109/frame 0x2885a4ec trap(2885a5c0,8,28,28,16,...) at trap+0x382/frame 0x2885a5b4 calltrap() at 0xffc0319f/frame 0x2885a5b4 --- trap 0x3, eip = 0x102809b, esp = 0x2885a600, ebp = 0x2885a608 --- kdb_alt_break_internal(0) at kdb_alt_break_internal+0x13b/frame 0x2885a608 kdb_alt_break(2,1832be8c) at kdb_alt_break+0x10/frame 0x2885a614 uart_intr_rxready(1832be00,1dba,b85c5e4,880,b90ca00,...) at uart_intr_rxready+0xec/frame 0x2885a630 uart_intr(1832be00) at uart_intr+0x116/frame 0x2885a654 intr_event_handle(b90ca00,2885a6c8) at intr_event_handle+0x104/frame 0x2885a678 intr_execute_handlers(b9dc93c,2885a6c8) at intr_execute_handlers+0xbe/frame 0x2885a6a4 lapic_handle_intr(32,2885a6c8,8,28,28,...) at lapic_handle_intr+0x52/frame 0x2885a6b8 Xapic_isr1() at 0xffc0424a/frame 0x2885a6b8 --- interrupt, eip = 0xfc4e35, esp = 0x2885a708, ebp = 0x2885a708 --- __mtx_unlock_flags(1a92450,0,1489335,4eb) at __mtx_unlock_flags+0xb5/frame 0x2885a708 vnlru_free_locked(0,82000,1fc22000,18cd6080,2885a7ac,...) at vnlru_free_locked+0x279/frame 0x2885a768 vn_alloc_hard(2481f7b4,2885a7b4,1281eb8,74,0,...) at vn_alloc_hard+0xaf/frame 0x2885a780 getnewvnode(1413efe,18cd6080,1ab0f40,2885a7e4) at getnewvnode+0x261/frame 0x2885a7ac ffs_vgetf(18cd6080,5ea8,0,202000,2885a844,0) at ffs_vgetf+0xdf/frame 0x2885a7f4 ffs_vget(18cd6080,5ea8,0,202000,2885a844) at ffs_vget+0x19/frame 0x2885a814 ufs_lookup_ino(1fc19468,2885ab84,2885ab9c,0) at ufs_lookup_ino+0xba8/frame 0x2885a8cc ufs_lookup(2885a908) at ufs_lookup+0x16/frame 0x2885a8e4 VOP_CACHEDLOOKUP_APV(1ab0f40,2885a908) at VOP_CACHEDLOOKUP_APV+0x4d/frame 0x2885a8f8 vfs_cache_lookup(2885a960) at vfs_cache_lookup+0x98/frame 0x2885a92c VOP_LOOKUP_APV(1ab0f40,2885a960) at VOP_LOOKUP_APV+0x4d/frame 0x2885a940 lookup(2885ab48) at lookup+0x580/frame 0x2885a994 namei(2885ab48) at namei+0x6df/frame 0x2885aa20 vn_open_cred(2885ab48,2885ac00,0,0,235c4b00,2ae503c0) at vn_open_cred+0x2f8/frame 0x2885aae0 vn_open(2885ab48,2885ac00,0,2ae503c0) at vn_open+0x20/frame 0x2885ab00 kern_openat(284e9700,ffffff9c,22f04206,0,100001,0) at kern_openat+0x2fa/frame 0x2885abe8 sys_open(284e9700,284e9998,284e9700,284de000,284e9700,...) at sys_open+0x1c/frame 0x2885ac08 syscall(2885ace8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x2885acdc Xint0x80_syscall() at 0xffc033f9/frame 0x2885acdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command sshd pid 1158 tid 100105 td 0x18d78500 sched_switch(18d78500,104) at sched_switch+0x4e9/frame 0x1fa95970 mi_switch(104) at mi_switch+0x1c4/frame 0x1fa959a8 sleepq_switch(3640,18d78500,0,0,1d79914,...) at sleepq_switch+0xf7/frame 0x1fa959c8 sleepq_catch_signals(1fa95a30,f84428,b9d3de4,0,b9d3dd0,...) at sleepq_catch_signals+0x278/frame 0x1fa959f0 sleepq_wait_sig(b9d3de4,0) at sleepq_wait_sig+0xe/frame 0x1fa959f8 _cv_wait_sig(b9d3de4,b9d3dd0) at _cv_wait_sig+0x138/frame 0x1fa95a30 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x1fa95a4c kern_select(18d78500,e,2047c270,2047c278,0,0,20) at kern_select+0x8e5/frame 0x1fa95bd4 sys_select(18d78500,18d78798,18d78500,18911688,b8565b8,...) at sys_select+0x41/frame 0x1fa95c08 syscall(1fa95ce8,3b,3b,3b,20482000,...) at syscall+0x1b8/frame 0x1fa95cdc Xint0x80_syscall() at 0xffc033f9/frame 0x1fa95cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe808 --- Tracing command sshd pid 1156 tid 100220 td 0x27a90880 sched_switch(27a90880,104) at sched_switch+0x4e9/frame 0x238c09a0 mi_switch(104) at mi_switch+0x1c4/frame 0x238c09d8 sleepq_switch(b40,27a90880,0,0,1d76e14,...) at sleepq_switch+0xf7/frame 0x238c09f8 sleepq_catch_signals(238c0a60,f84428,9ab0924,0,140171b,...) at sleepq_catch_signals+0x278/frame 0x238c0a20 sleepq_wait_sig(9ab0924,0) at sleepq_wait_sig+0xe/frame 0x238c0a28 _cv_wait_sig(9ab0924,9ab0910) at _cv_wait_sig+0x138/frame 0x238c0a60 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x238c0a7c kern_poll(27a90880,ffbfe7a4,1,0,0) at kern_poll+0x4a4/frame 0x238c0bd4 sys_poll(27a90880,27a90b18,27a90880,1fe2a688,27a90880,...) at sys_poll+0x50/frame 0x238c0c08 syscall(238c0ce8,3b,3b,3b,20482000,...) at syscall+0x1b8/frame 0x238c0cdc Xint0x80_syscall() at 0xffc033f9/frame 0x238c0cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe808 --- Tracing command sh pid 1083 tid 100121 td 0x18d77380 sched_switch(18d77380,104) at sched_switch+0x4e9/frame 0x1fb419d0 mi_switch(104) at mi_switch+0x1c4/frame 0x1fb41a04 sleepq_switch(ac0,14e9ff2,6c,0,1d76d94,...) at sleepq_switch+0xf7/frame 0x1fb41a24 sleepq_catch_signals(1fb41a94,fefb0c,1fe2b398,6c,fd3fb1,...) at sleepq_catch_signals+0x278/frame 0x1fb41a4c sleepq_wait_sig(1fe2b398,6c) at sleepq_wait_sig+0xe/frame 0x1fb41a54 _sleep(1fe2b398,1fe2b440,36c,1506bc5,0,0,0,0,100) at _sleep+0x27c/frame 0x1fb41a94 kern_wait6(18d77380,7,0,0,1fb41bf0,30,0,0) at kern_wait6+0x3b8/frame 0x1fb41af0 sys_wait4(18d77380,18d77618,18d77380,1fe2b398,18d77380,...) at sys_wait4+0x73/frame 0x1fb41c08 syscall(1fb41ce8,3b,3b,3b,ffbfe7b4,...) at syscall+0x1b8/frame 0x1fb41cdc Xint0x80_syscall() at 0xffc033f9/frame 0x1fb41cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe1ac --- Tracing command bash pid 1009 tid 100215 td 0x235c2700 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(200046,1a63644,2885a10c,103cc0c,1a63644,...) at db_trace_self_wrapper+0x2a/frame 0x2885a0c8 kdb_backtrace(2885a1f0,13d5a2b,143607a,0,2885a228,...) at kdb_backtrace+0x2e/frame 0x2885a128 kdb_reenter(143607a,0,2885a228,2885a204,284de000,...) at kdb_reenter+0x29/frame 0x2885a130 trap(2885a1fc,8,28,28,ffffffff,...) at trap+0xeb/frame 0x2885a1f0 calltrap() at 0xffc0319f/frame 0x2885a1f0 --- trap 0xc, eip = 0x13b215a, esp = 0x2885a23c, ebp = 0x2885a248 --- db_trace_thread(235c2700,ffffffff) at db_trace_thread+0x2a/frame 0x2885a248 _db_stack_trace_all(102809b,0,ffffffff,2885a2d4) at _db_stack_trace_all+0x93/frame 0x2885a2a8 db_command(1aec772) at db_command+0x26a/frame 0x2885a35c db_script_exec(bd2d8f,0,2885a404,2885a3e8) at db_script_exec+0x1a6/frame 0x2885a3bc db_command(2885a4c0,a06536,bd3b00,c960,284e9700,...) at db_command+0x26a/frame 0x2885a470 db_command_loop(bd3b00,c960,284e9700,2885a478,2885a4c0,...) at db_command_loop+0x3c/frame 0x2885a478 db_trap(3,0) at db_trap+0xf6/frame 0x2885a4c0 kdb_trap(3,0,2885a5c0) at kdb_trap+0x109/frame 0x2885a4ec trap(2885a5c0,8,28,28,16,...) at trap+0x382/frame 0x2885a5b4 calltrap() at 0xffc0319f/frame 0x2885a5b4 --- trap 0x3, eip = 0x102809b, esp = 0x2885a600, ebp = 0x2885a608 --- kdb_alt_break_internal(0) at kdb_alt_break_internal+0x13b/frame 0x2885a608 kdb_alt_break(2,1832be8c) at kdb_alt_break+0x10/frame 0x2885a614 uart_intr_rxready(1832be00,1dba,b85c5e4,880,b90ca00,...) at uart_intr_rxready+0xec/frame 0x2885a630 uart_intr(1832be00) at uart_intr+0x116/frame 0x2885a654 intr_event_handle(b90ca00,2885a6c8) at intr_event_handle+0x104/frame 0x2885a678 intr_execute_handlers(b9dc93c,2885a6c8) at intr_execute_handlers+0xbe/frame 0x2885a6a4 lapic_handle_intr(32,2885a6c8,8,28,28,...) at lapic_handle_intr+0x52/frame 0x2885a6b8 Xapic_isr1() at 0xffc0424a/frame 0x2885a6b8 --- interrupt, eip = 0xfc4e35, esp = 0x2885a708, ebp = 0x2885a708 --- __mtx_unlock_flags(1a92450,0,1489335,4eb) at __mtx_unlock_flags+0xb5/frame 0x2885a708 vnlru_free_locked(0,82000,1fc22000,18cd6080,2885a7ac,...) at vnlru_free_locked+0x279/frame 0x2885a768 vn_alloc_hard(2481f7b4,2885a7b4,1281eb8,74,0,...) at vn_alloc_hard+0xaf/frame 0x2885a780 getnewvnode(1413efe,18cd6080,1ab0f40,2885a7e4) at getnewvnode+0x261/frame 0x2885a7ac ffs_vgetf(18cd6080,5ea8,0,202000,2885a844,0) at ffs_vgetf+0xdf/frame 0x2885a7f4 ffs_vget(18cd6080,5ea8,0,202000,2885a844) at ffs_vget+0x19/frame 0x2885a814 ufs_lookup_ino(1fc19468,2885ab84,2885ab9c,0) at ufs_lookup_ino+0xba8/frame 0x2885a8cc ufs_lookup(2885a908) at ufs_lookup+0x16/frame 0x2885a8e4 VOP_CACHEDLOOKUP_APV(1ab0f40,2885a908) at VOP_CACHEDLOOKUP_APV+0x4d/frame 0x2885a8f8 vfs_cache_lookup(2885a960) at vfs_cache_lookup+0x98/frame 0x2885a92c VOP_LOOKUP_APV(1ab0f40,2885a960) at VOP_LOOKUP_APV+0x4d/frame 0x2885a940 lookup(2885ab48) at lookup+0x580/frame 0x2885a994 namei(2885ab48) at namei+0x6df/frame 0x2885aa20 vn_open_cred(2885ab48,2885ac00,0,0,235c4b00,2ae503c0) at vn_open_cred+0x2f8/frame 0x2885aae0 vn_open(2885ab48,2885ac00,0,2ae503c0) at vn_open+0x20/frame 0x2885ab00 kern_openat(284e9700,ffffff9c,22f04206,0,100001,0) at kern_openat+0x2fa/frame 0x2885abe8 sys_open(284e9700,284e9998,284e9700,284de000,284e9700,...) at sys_open+0x1c/frame 0x2885ac08 syscall(2885ace8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x2885acdc Xint0x80_syscall() at 0xffc033f9/frame 0x2885acdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command csh pid 1007 tid 100211 td 0x18d77000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(200046,1a63644,2885a10c,103cc0c,1a63644,...) at db_trace_self_wrapper+0x2a/frame 0x2885a0c8 kdb_backtrace(2885a1f0,13d5a2b,143607a,0,2885a228,...) at kdb_backtrace+0x2e/frame 0x2885a128 kdb_reenter(143607a,0,2885a228,2885a204,284de000,...) at kdb_reenter+0x29/frame 0x2885a130 trap(2885a1fc,8,28,28,ffffffff,...) at trap+0xeb/frame 0x2885a1f0 calltrap() at 0xffc0319f/frame 0x2885a1f0 --- trap 0xc, eip = 0x13b215a, esp = 0x2885a23c, ebp = 0x2885a248 --- db_trace_thread(18d77000,ffffffff) at db_trace_thread+0x2a/frame 0x2885a248 _db_stack_trace_all(102809b,0,ffffffff,2885a2d4) at _db_stack_trace_all+0x93/frame 0x2885a2a8 db_command(1aec772) at db_command+0x26a/frame 0x2885a35c db_script_exec(bd2d8f,0,2885a404,2885a3e8) at db_script_exec+0x1a6/frame 0x2885a3bc db_command(2885a4c0,a06536,bd3b00,c960,284e9700,...) at db_command+0x26a/frame 0x2885a470 db_command_loop(bd3b00,c960,284e9700,2885a478,2885a4c0,...) at db_command_loop+0x3c/frame 0x2885a478 db_trap(3,0) at db_trap+0xf6/frame 0x2885a4c0 kdb_trap(3,0,2885a5c0) at kdb_trap+0x109/frame 0x2885a4ec trap(2885a5c0,8,28,28,16,...) at trap+0x382/frame 0x2885a5b4 calltrap() at 0xffc0319f/frame 0x2885a5b4 --- trap 0x3, eip = 0x102809b, esp = 0x2885a600, ebp = 0x2885a608 --- kdb_alt_break_internal(0) at kdb_alt_break_internal+0x13b/frame 0x2885a608 kdb_alt_break(2,1832be8c) at kdb_alt_break+0x10/frame 0x2885a614 uart_intr_rxready(1832be00,1dba,b85c5e4,880,b90ca00,...) at uart_intr_rxready+0xec/frame 0x2885a630 uart_intr(1832be00) at uart_intr+0x116/frame 0x2885a654 intr_event_handle(b90ca00,2885a6c8) at intr_event_handle+0x104/frame 0x2885a678 intr_execute_handlers(b9dc93c,2885a6c8) at intr_execute_handlers+0xbe/frame 0x2885a6a4 lapic_handle_intr(32,2885a6c8,8,28,28,...) at lapic_handle_intr+0x52/frame 0x2885a6b8 Xapic_isr1() at 0xffc0424a/frame 0x2885a6b8 --- interrupt, eip = 0xfc4e35, esp = 0x2885a708, ebp = 0x2885a708 --- __mtx_unlock_flags(1a92450,0,1489335,4eb) at __mtx_unlock_flags+0xb5/frame 0x2885a708 vnlru_free_locked(0,82000,1fc22000,18cd6080,2885a7ac,...) at vnlru_free_locked+0x279/frame 0x2885a768 vn_alloc_hard(2481f7b4,2885a7b4,1281eb8,74,0,...) at vn_alloc_hard+0xaf/frame 0x2885a780 getnewvnode(1413efe,18cd6080,1ab0f40,2885a7e4) at getnewvnode+0x261/frame 0x2885a7ac ffs_vgetf(18cd6080,5ea8,0,202000,2885a844,0) at ffs_vgetf+0xdf/frame 0x2885a7f4 ffs_vget(18cd6080,5ea8,0,202000,2885a844) at ffs_vget+0x19/frame 0x2885a814 ufs_lookup_ino(1fc19468,2885ab84,2885ab9c,0) at ufs_lookup_ino+0xba8/frame 0x2885a8cc ufs_lookup(2885a908) at ufs_lookup+0x16/frame 0x2885a8e4 VOP_CACHEDLOOKUP_APV(1ab0f40,2885a908) at VOP_CACHEDLOOKUP_APV+0x4d/frame 0x2885a8f8 vfs_cache_lookup(2885a960) at vfs_cache_lookup+0x98/frame 0x2885a92c VOP_LOOKUP_APV(1ab0f40,2885a960) at VOP_LOOKUP_APV+0x4d/frame 0x2885a940 lookup(2885ab48) at lookup+0x580/frame 0x2885a994 namei(2885ab48) at namei+0x6df/frame 0x2885aa20 vn_open_cred(2885ab48,2885ac00,0,0,235c4b00,2ae503c0) at vn_open_cred+0x2f8/frame 0x2885aae0 vn_open(2885ab48,2885ac00,0,2ae503c0) at vn_open+0x20/frame 0x2885ab00 kern_openat(284e9700,ffffff9c,22f04206,0,100001,0) at kern_openat+0x2fa/frame 0x2885abe8 sys_open(284e9700,284e9998,284e9700,284de000,284e9700,...) at sys_open+0x1c/frame 0x2885ac08 syscall(2885ace8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x2885acdc Xint0x80_syscall() at 0xffc033f9/frame 0x2885acdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command su pid 1006 tid 100108 td 0x18d4ae00 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(200046,1a63644,2885a10c,103cc0c,1a63644,...) at db_trace_self_wrapper+0x2a/frame 0x2885a0c8 kdb_backtrace(2885a1f0,13d5a2b,143607a,0,2885a228,...) at kdb_backtrace+0x2e/frame 0x2885a128 kdb_reenter(143607a,0,2885a228,2885a204,284de000,...) at kdb_reenter+0x29/frame 0x2885a130 trap(2885a1fc,8,28,28,ffffffff,...) at trap+0xeb/frame 0x2885a1f0 calltrap() at 0xffc0319f/frame 0x2885a1f0 --- trap 0xc, eip = 0x13b215a, esp = 0x2885a23c, ebp = 0x2885a248 --- db_trace_thread(18d4ae00,ffffffff) at db_trace_thread+0x2a/frame 0x2885a248 _db_stack_trace_all(102809b,0,ffffffff,2885a2d4) at _db_stack_trace_all+0x93/frame 0x2885a2a8 db_command(1aec772) at db_command+0x26a/frame 0x2885a35c db_script_exec(bd2d8f,0,2885a404,2885a3e8) at db_script_exec+0x1a6/frame 0x2885a3bc db_command(2885a4c0,a06536,bd3b00,c960,284e9700,...) at db_command+0x26a/frame 0x2885a470 db_command_loop(bd3b00,c960,284e9700,2885a478,2885a4c0,...) at db_command_loop+0x3c/frame 0x2885a478 db_trap(3,0) at db_trap+0xf6/frame 0x2885a4c0 kdb_trap(3,0,2885a5c0) at kdb_trap+0x109/frame 0x2885a4ec trap(2885a5c0,8,28,28,16,...) at trap+0x382/frame 0x2885a5b4 calltrap() at 0xffc0319f/frame 0x2885a5b4 --- trap 0x3, eip = 0x102809b, esp = 0x2885a600, ebp = 0x2885a608 --- kdb_alt_break_internal(0) at kdb_alt_break_internal+0x13b/frame 0x2885a608 kdb_alt_break(2,1832be8c) at kdb_alt_break+0x10/frame 0x2885a614 uart_intr_rxready(1832be00,1dba,b85c5e4,880,b90ca00,...) at uart_intr_rxready+0xec/frame 0x2885a630 uart_intr(1832be00) at uart_intr+0x116/frame 0x2885a654 intr_event_handle(b90ca00,2885a6c8) at intr_event_handle+0x104/frame 0x2885a678 intr_execute_handlers(b9dc93c,2885a6c8) at intr_execute_handlers+0xbe/frame 0x2885a6a4 lapic_handle_intr(32,2885a6c8,8,28,28,...) at lapic_handle_intr+0x52/frame 0x2885a6b8 Xapic_isr1() at 0xffc0424a/frame 0x2885a6b8 --- interrupt, eip = 0xfc4e35, esp = 0x2885a708, ebp = 0x2885a708 --- __mtx_unlock_flags(1a92450,0,1489335,4eb) at __mtx_unlock_flags+0xb5/frame 0x2885a708 vnlru_free_locked(0,82000,1fc22000,18cd6080,2885a7ac,...) at vnlru_free_locked+0x279/frame 0x2885a768 vn_alloc_hard(2481f7b4,2885a7b4,1281eb8,74,0,...) at vn_alloc_hard+0xaf/frame 0x2885a780 getnewvnode(1413efe,18cd6080,1ab0f40,2885a7e4) at getnewvnode+0x261/frame 0x2885a7ac ffs_vgetf(18cd6080,5ea8,0,202000,2885a844,0) at ffs_vgetf+0xdf/frame 0x2885a7f4 ffs_vget(18cd6080,5ea8,0,202000,2885a844) at ffs_vget+0x19/frame 0x2885a814 ufs_lookup_ino(1fc19468,2885ab84,2885ab9c,0) at ufs_lookup_ino+0xba8/frame 0x2885a8cc ufs_lookup(2885a908) at ufs_lookup+0x16/frame 0x2885a8e4 VOP_CACHEDLOOKUP_APV(1ab0f40,2885a908) at VOP_CACHEDLOOKUP_APV+0x4d/frame 0x2885a8f8 vfs_cache_lookup(2885a960) at vfs_cache_lookup+0x98/frame 0x2885a92c VOP_LOOKUP_APV(1ab0f40,2885a960) at VOP_LOOKUP_APV+0x4d/frame 0x2885a940 lookup(2885ab48) at lookup+0x580/frame 0x2885a994 namei(2885ab48) at namei+0x6df/frame 0x2885aa20 vn_open_cred(2885ab48,2885ac00,0,0,235c4b00,2ae503c0) at vn_open_cred+0x2f8/frame 0x2885aae0 vn_open(2885ab48,2885ac00,0,2ae503c0) at vn_open+0x20/frame 0x2885ab00 kern_openat(284e9700,ffffff9c,22f04206,0,100001,0) at kern_openat+0x2fa/frame 0x2885abe8 sys_open(284e9700,284e9998,284e9700,284de000,284e9700,...) at sys_open+0x1c/frame 0x2885ac08 syscall(2885ace8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x2885acdc Xint0x80_syscall() at 0xffc033f9/frame 0x2885acdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command bash pid 988 tid 100144 td 0x22bc0380 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(200046,1a63644,2885a10c,103cc0c,1a63644,...) at db_trace_self_wrapper+0x2a/frame 0x2885a0c8 kdb_backtrace(2885a1f0,13d5a2b,143607a,0,2885a228,...) at kdb_backtrace+0x2e/frame 0x2885a128 kdb_reenter(143607a,0,2885a228,2885a204,284de000,...) at kdb_reenter+0x29/frame 0x2885a130 trap(2885a1fc,8,28,28,ffffffff,...) at trap+0xeb/frame 0x2885a1f0 calltrap() at 0xffc0319f/frame 0x2885a1f0 --- trap 0xc, eip = 0x13b215a, esp = 0x2885a23c, ebp = 0x2885a248 --- db_trace_thread(22bc0380,ffffffff) at db_trace_thread+0x2a/frame 0x2885a248 _db_stack_trace_all(102809b,0,ffffffff,2885a2d4) at _db_stack_trace_all+0x93/frame 0x2885a2a8 db_command(1aec772) at db_command+0x26a/frame 0x2885a35c db_script_exec(bd2d8f,0,2885a404,2885a3e8) at db_script_exec+0x1a6/frame 0x2885a3bc db_command(2885a4c0,a06536,bd3b00,c960,284e9700,...) at db_command+0x26a/frame 0x2885a470 db_command_loop(bd3b00,c960,284e9700,2885a478,2885a4c0,...) at db_command_loop+0x3c/frame 0x2885a478 db_trap(3,0) at db_trap+0xf6/frame 0x2885a4c0 kdb_trap(3,0,2885a5c0) at kdb_trap+0x109/frame 0x2885a4ec trap(2885a5c0,8,28,28,16,...) at trap+0x382/frame 0x2885a5b4 calltrap() at 0xffc0319f/frame 0x2885a5b4 --- trap 0x3, eip = 0x102809b, esp = 0x2885a600, ebp = 0x2885a608 --- kdb_alt_break_internal(0) at kdb_alt_break_internal+0x13b/frame 0x2885a608 kdb_alt_break(2,1832be8c) at kdb_alt_break+0x10/frame 0x2885a614 uart_intr_rxready(1832be00,1dba,b85c5e4,880,b90ca00,...) at uart_intr_rxready+0xec/frame 0x2885a630 uart_intr(1832be00) at uart_intr+0x116/frame 0x2885a654 intr_event_handle(b90ca00,2885a6c8) at intr_event_handle+0x104/frame 0x2885a678 intr_execute_handlers(b9dc93c,2885a6c8) at intr_execute_handlers+0xbe/frame 0x2885a6a4 lapic_handle_intr(32,2885a6c8,8,28,28,...) at lapic_handle_intr+0x52/frame 0x2885a6b8 Xapic_isr1() at 0xffc0424a/frame 0x2885a6b8 --- interrupt, eip = 0xfc4e35, esp = 0x2885a708, ebp = 0x2885a708 --- __mtx_unlock_flags(1a92450,0,1489335,4eb) at __mtx_unlock_flags+0xb5/frame 0x2885a708 vnlru_free_locked(0,82000,1fc22000,18cd6080,2885a7ac,...) at vnlru_free_locked+0x279/frame 0x2885a768 vn_alloc_hard(2481f7b4,2885a7b4,1281eb8,74,0,...) at vn_alloc_hard+0xaf/frame 0x2885a780 getnewvnode(1413efe,18cd6080,1ab0f40,2885a7e4) at getnewvnode+0x261/frame 0x2885a7ac ffs_vgetf(18cd6080,5ea8,0,202000,2885a844,0) at ffs_vgetf+0xdf/frame 0x2885a7f4 ffs_vget(18cd6080,5ea8,0,202000,2885a844) at ffs_vget+0x19/frame 0x2885a814 ufs_lookup_ino(1fc19468,2885ab84,2885ab9c,0) at ufs_lookup_ino+0xba8/frame 0x2885a8cc ufs_lookup(2885a908) at ufs_lookup+0x16/frame 0x2885a8e4 VOP_CACHEDLOOKUP_APV(1ab0f40,2885a908) at VOP_CACHEDLOOKUP_APV+0x4d/frame 0x2885a8f8 vfs_cache_lookup(2885a960) at vfs_cache_lookup+0x98/frame 0x2885a92c VOP_LOOKUP_APV(1ab0f40,2885a960) at VOP_LOOKUP_APV+0x4d/frame 0x2885a940 lookup(2885ab48) at lookup+0x580/frame 0x2885a994 namei(2885ab48) at namei+0x6df/frame 0x2885aa20 vn_open_cred(2885ab48,2885ac00,0,0,235c4b00,2ae503c0) at vn_open_cred+0x2f8/frame 0x2885aae0 vn_open(2885ab48,2885ac00,0,2ae503c0) at vn_open+0x20/frame 0x2885ab00 kern_openat(284e9700,ffffff9c,22f04206,0,100001,0) at kern_openat+0x2fa/frame 0x2885abe8 sys_open(284e9700,284e9998,284e9700,284de000,284e9700,...) at sys_open+0x1c/frame 0x2885ac08 syscall(2885ace8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x2885acdc Xint0x80_syscall() at 0xffc033f9/frame 0x2885acdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command sshd pid 987 tid 100208 td 0x235c3180 sched_switch(235c3180,104) at sched_switch+0x4e9/frame 0x2387b970 mi_switch(104) at mi_switch+0x1c4/frame 0x2387b9a8 sleepq_switch(27c0,235c3180,0,0,1d78a94,...) at sleepq_switch+0xf7/frame 0x2387b9c8 sleepq_catch_signals(2387ba30,f84428,1880bb24,0,1880bb10,...) at sleepq_catch_signals+0x278/frame 0x2387b9f0 sleepq_wait_sig(1880bb24,0) at sleepq_wait_sig+0xe/frame 0x2387b9f8 _cv_wait_sig(1880bb24,1880bb10) at _cv_wait_sig+0x138/frame 0x2387ba30 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x2387ba4c kern_select(235c3180,10,2047c270,2047c278,0,0,20) at kern_select+0x8e5/frame 0x2387bbd4 sys_select(235c3180,235c3418,235c3180,22c2a344,b85c5b8,...) at sys_select+0x41/frame 0x2387bc08 syscall(2387bce8,3b,3b,3b,20482000,...) at syscall+0x1b8/frame 0x2387bcdc Xint0x80_syscall() at 0xffc033f9/frame 0x2387bcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe1ac --- Tracing command sshd pid 985 tid 100114 td 0x1fc9f500 sched_switch(1fc9f500,104) at sched_switch+0x4e9/frame 0x1faee9a0 mi_switch(104) at mi_switch+0x1c4/frame 0x1faee9d8 sleepq_switch(2f40,1fc9f500,0,0,1d79214,...) at sleepq_switch+0xf7/frame 0x1faee9f8 sleepq_catch_signals(1faeea60,f84428,18d5d964,0,140171b,...) at sleepq_catch_signals+0x278/frame 0x1faeea20 sleepq_wait_sig(18d5d964,0) at sleepq_wait_sig+0xe/frame 0x1faeea28 _cv_wait_sig(18d5d964,18d5d950) at _cv_wait_sig+0x138/frame 0x1faeea60 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x1faeea7c kern_poll(1fc9f500,ffbfe7a4,1,0,0) at kern_poll+0x4a4/frame 0x1faeebd4 sys_poll(1fc9f500,1fc9f798,1fc9f500,1834e054,1fc9f500,...) at sys_poll+0x50/frame 0x1faeec08 syscall(1faeece8,3b,3b,3b,20482000,...) at syscall+0x1b8/frame 0x1faeecdc Xint0x80_syscall() at 0xffc033f9/frame 0x1faeecdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d98, esp = 0xffc0bfe8, ebp = 0xffbfeccc --- Tracing command getty pid 829 tid 100201 td 0x23beb500 sched_switch(23beb500,103) at sched_switch+0x4e9/frame 0x2384c5c8 mi_switch(103,1d7ae60) at mi_switch+0x1c4/frame 0x2384c600 turnstile_wait(231b5500,2817a700,0) at turnstile_wait+0x2f1/frame 0x2384c61c __mtx_lock_sleep(1a92450,2817a700,0,1489335,4ee,...) at __mtx_lock_sleep+0x1b1/frame 0x2384c67c __mtx_lock_flags(1a92450,0,1489335,4ee) at __mtx_lock_flags+0xcc/frame 0x2384c6a8 vnlru_free_locked(0,82000,1ff58000,1fe31ac0,2384c74c,...) at vnlru_free_locked+0x9a/frame 0x2384c708 vn_alloc_hard(333a6b54,2384c754,1281eb8,74,0,...) at vn_alloc_hard+0xaf/frame 0x2384c720 getnewvnode(1413efe,1fe31ac0,1ab0f40,2384c784) at getnewvnode+0x261/frame 0x2384c74c ffs_vgetf(1fe31ac0,10e613,0,202000,2384c7e4,0) at ffs_vgetf+0xdf/frame 0x2384c794 ffs_vget(1fe31ac0,10e613,0,202000,2384c7e4) at ffs_vget+0x19/frame 0x2384c7b4 ufs_lookup_ino(202b9468,2384ca80,2384ca98,0) at ufs_lookup_ino+0xba8/frame 0x2384c86c ufs_lookup(2384c8a8) at ufs_lookup+0x16/frame 0x2384c884 VOP_CACHEDLOOKUP_APV(1ab0f40,2384c8a8) at VOP_CACHEDLOOKUP_APV+0x4d/frame 0x2384c898 vfs_cache_lookup(2384c900) at vfs_cache_lookup+0x98/frame 0x2384c8cc VOP_LOOKUP_APV(1ab0f40,2384c900) at VOP_LOOKUP_APV+0x4d/frame 0x2384c8e0 lookup(2384ca44) at lookup+0x580/frame 0x2384c934 namei(2384ca44) at namei+0x6df/frame 0x2384c9bc kern_execve(23beb500,2384cbc8,0,1fc2d9b0) at kern_execve+0x223/frame 0x2384cbb0 sys_execve(23beb500,23beb798,23beb500,23bef6dc,23beb500,...) at sys_execve+0x51/frame 0x2384cc08 syscall(2384cce8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x2384ccdc Xint0x80_syscall() at 0xffc033f9/frame 0x2384ccdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command getty pid 828 tid 100200 td 0x23beb880 sched_switch(23beb880,104) at sched_switch+0x4e9/frame 0x23847a20 mi_switch(104) at mi_switch+0x1c4/frame 0x23847a54 sleepq_switch(1600,23beb880,0,0,1d778d4,...) at sleepq_switch+0xf7/frame 0x23847a74 sleepq_catch_signals(23847adc,f84428,18c52c74,0,15114d4,...) at sleepq_catch_signals+0x278/frame 0x23847a9c sleepq_wait_sig(18c52c74,0) at sleepq_wait_sig+0xe/frame 0x23847aa4 _cv_wait_sig(18c52c74,18c52c04) at _cv_wait_sig+0x138/frame 0x23847adc tty_wait(18c52c00,18c52c74) at tty_wait+0x35/frame 0x23847af4 ttydisc_read(18c52c00,23847bd0,0) at ttydisc_read+0x1c6/frame 0x23847b28 ttydev_read(18c53e00,23847bd0,0) at ttydev_read+0x3d/frame 0x23847b44 devfs_read_f(1fc6c9d8,23847bd0,22bc7700,0,23beb880) at devfs_read_f+0xaa/frame 0x23847b74 dofileread(1fc6c9d8,23847bd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x23847ba4 sys_read(23beb880,23bebb18,23beb880,23befa20,69,...) at sys_read+0x78/frame 0x23847c08 syscall(23847ce8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x23847cdc Xint0x80_syscall() at 0xffc033f9/frame 0x23847cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe1ac --- Tracing command getty pid 827 tid 100199 td 0x23bebc00 sched_switch(23bebc00,104) at sched_switch+0x4e9/frame 0x23842a20 mi_switch(104) at mi_switch+0x1c4/frame 0x23842a54 sleepq_switch(2d00,23bebc00,0,0,1d78fd4,...) at sleepq_switch+0xf7/frame 0x23842a74 sleepq_catch_signals(23842adc,f84428,18bac074,0,15114d4,...) at sleepq_catch_signals+0x278/frame 0x23842a9c sleepq_wait_sig(18bac074,0) at sleepq_wait_sig+0xe/frame 0x23842aa4 _cv_wait_sig(18bac074,18bac004) at _cv_wait_sig+0x138/frame 0x23842adc tty_wait(18bac000,18bac074) at tty_wait+0x35/frame 0x23842af4 ttydisc_read(18bac000,23842bd0,0) at ttydisc_read+0x1c6/frame 0x23842b28 ttydev_read(18aed000,23842bd0,0) at ttydev_read+0x3d/frame 0x23842b44 devfs_read_f(22c29870,23842bd0,202cf300,0,23bebc00) at devfs_read_f+0xaa/frame 0x23842b74 dofileread(22c29870,23842bd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x23842ba4 sys_read(23bebc00,23bebe98,23bebc00,23a5c000,23bebc00,...) at sys_read+0x78/frame 0x23842c08 syscall(23842ce8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x23842cdc Xint0x80_syscall() at 0xffc033f9/frame 0x23842cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe1ac --- Tracing command getty pid 826 tid 100198 td 0x236e2000 sched_switch(236e2000,104) at sched_switch+0x4e9/frame 0x2383da20 mi_switch(104) at mi_switch+0x1c4/frame 0x2383da54 sleepq_switch(2c00,236e2000,0,0,1d78ed4,...) at sleepq_switch+0xf7/frame 0x2383da74 sleepq_catch_signals(2383dadc,f84428,18bac474,0,15114d4,...) at sleepq_catch_signals+0x278/frame 0x2383da9c sleepq_wait_sig(18bac474,0) at sleepq_wait_sig+0xe/frame 0x2383daa4 _cv_wait_sig(18bac474,18bac404) at _cv_wait_sig+0x138/frame 0x2383dadc tty_wait(18bac400,18bac474) at tty_wait+0x35/frame 0x2383daf4 ttydisc_read(18bac400,2383dbd0,0) at ttydisc_read+0x1c6/frame 0x2383db28 ttydev_read(18aed200,2383dbd0,0) at ttydev_read+0x3d/frame 0x2383db44 devfs_read_f(1fc34f3c,2383dbd0,1ff5a400,0,236e2000) at devfs_read_f+0xaa/frame 0x2383db74 dofileread(1fc34f3c,2383dbd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x2383dba4 sys_read(236e2000,236e2298,236e2000,23a5c344,236e2000,...) at sys_read+0x78/frame 0x2383dc08 syscall(2383dce8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x2383dcdc Xint0x80_syscall() at 0xffc033f9/frame 0x2383dcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe808 --- Tracing command getty pid 825 tid 100197 td 0x236e2380 sched_switch(236e2380,104) at sched_switch+0x4e9/frame 0x23838a20 mi_switch(104) at mi_switch+0x1c4/frame 0x23838a54 sleepq_switch(2f00,236e2380,0,0,1d791d4,...) at sleepq_switch+0xf7/frame 0x23838a74 sleepq_catch_signals(23838adc,f84428,18bac874,0,15114d4,...) at sleepq_catch_signals+0x278/frame 0x23838a9c sleepq_wait_sig(18bac874,0) at sleepq_wait_sig+0xe/frame 0x23838aa4 _cv_wait_sig(18bac874,18bac804) at _cv_wait_sig+0x138/frame 0x23838adc tty_wait(18bac800,18bac874) at tty_wait+0x35/frame 0x23838af4 ttydisc_read(18bac800,23838bd0,0) at ttydisc_read+0x1c6/frame 0x23838b28 ttydev_read(18aed400,23838bd0,0) at ttydev_read+0x3d/frame 0x23838b44 devfs_read_f(1fc62c30,23838bd0,22bc7800,0,236e2380) at devfs_read_f+0xaa/frame 0x23838b74 dofileread(1fc62c30,23838bd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x23838ba4 sys_read(236e2380,236e2618,236e2380,23a5c688,236e2380,...) at sys_read+0x78/frame 0x23838c08 syscall(23838ce8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x23838cdc Xint0x80_syscall() at 0xffc033f9/frame 0x23838cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command getty pid 824 tid 100196 td 0x236e2700 sched_switch(236e2700,104) at sched_switch+0x4e9/frame 0x23833a20 mi_switch(104) at mi_switch+0x1c4/frame 0x23833a54 sleepq_switch(2e00,236e2700,0,0,1d790d4,...) at sleepq_switch+0xf7/frame 0x23833a74 sleepq_catch_signals(23833adc,f84428,18bacc74,0,15114d4,...) at sleepq_catch_signals+0x278/frame 0x23833a9c sleepq_wait_sig(18bacc74,0) at sleepq_wait_sig+0xe/frame 0x23833aa4 _cv_wait_sig(18bacc74,18bacc04) at _cv_wait_sig+0x138/frame 0x23833adc tty_wait(18bacc00,18bacc74) at tty_wait+0x35/frame 0x23833af4 ttydisc_read(18bacc00,23833bd0,0) at ttydisc_read+0x1c6/frame 0x23833b28 ttydev_read(18aed600,23833bd0,0) at ttydev_read+0x3d/frame 0x23833b44 devfs_read_f(1fc62bf4,23833bd0,235cf200,0,236e2700) at devfs_read_f+0xaa/frame 0x23833b74 dofileread(1fc62bf4,23833bd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x23833ba4 sys_read(236e2700,236e2998,236e2700,23a5c9cc,236e2700,...) at sys_read+0x78/frame 0x23833c08 syscall(23833ce8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x23833cdc Xint0x80_syscall() at 0xffc033f9/frame 0x23833cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d98, esp = 0xffc0bfe8, ebp = 0xffbfeccc --- Tracing command getty pid 823 tid 100195 td 0x236e2a80 sched_switch(236e2a80,104) at sched_switch+0x4e9/frame 0x2382ea20 mi_switch(104) at mi_switch+0x1c4/frame 0x2382ea54 sleepq_switch(2900,236e2a80,0,0,1d78bd4,...) at sleepq_switch+0xf7/frame 0x2382ea74 sleepq_catch_signals(2382eadc,f84428,188ed074,0,15114d4,...) at sleepq_catch_signals+0x278/frame 0x2382ea9c sleepq_wait_sig(188ed074,0) at sleepq_wait_sig+0xe/frame 0x2382eaa4 _cv_wait_sig(188ed074,188ed004) at _cv_wait_sig+0x138/frame 0x2382eadc tty_wait(188ed000,188ed074) at tty_wait+0x35/frame 0x2382eaf4 ttydisc_read(188ed000,2382ebd0,0) at ttydisc_read+0x1c6/frame 0x2382eb28 ttydev_read(18aed800,2382ebd0,0) at ttydev_read+0x3d/frame 0x2382eb44 devfs_read_f(1fc62b40,2382ebd0,22c2c600,0,236e2a80) at devfs_read_f+0xaa/frame 0x2382eb74 dofileread(1fc62b40,2382ebd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x2382eba4 sys_read(236e2a80,236e2d18,236e2a80,23a5cd10,b85c5b8,...) at sys_read+0x78/frame 0x2382ec08 syscall(2382ece8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x2382ecdc Xint0x80_syscall() at 0xffc033f9/frame 0x2382ecdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe1ac --- Tracing command getty pid 822 tid 100194 td 0x236e2e00 sched_switch(236e2e00,104) at sched_switch+0x4e9/frame 0x23829a20 mi_switch(104) at mi_switch+0x1c4/frame 0x23829a54 sleepq_switch(2800,236e2e00,0,0,1d78ad4,...) at sleepq_switch+0xf7/frame 0x23829a74 sleepq_catch_signals(23829adc,f84428,188ed474,0,15114d4,...) at sleepq_catch_signals+0x278/frame 0x23829a9c sleepq_wait_sig(188ed474,0) at sleepq_wait_sig+0xe/frame 0x23829aa4 _cv_wait_sig(188ed474,188ed404) at _cv_wait_sig+0x138/frame 0x23829adc tty_wait(188ed400,188ed474) at tty_wait+0x35/frame 0x23829af4 ttydisc_read(188ed400,23829bd0,0) at ttydisc_read+0x1c6/frame 0x23829b28 ttydev_read(18aeda00,23829bd0,0) at ttydev_read+0x3d/frame 0x23829b44 devfs_read_f(22c29d5c,23829bd0,1891d000,0,236e2e00) at devfs_read_f+0xaa/frame 0x23829b74 dofileread(22c29d5c,23829bd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x23829ba4 sys_read(236e2e00,236e3098,236e2e00,23a5d054,236e2e00,...) at sys_read+0x78/frame 0x23829c08 syscall(23829ce8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x23829cdc Xint0x80_syscall() at 0xffc033f9/frame 0x23829cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe808 --- Tracing command getty pid 821 tid 100193 td 0x236e3180 sched_switch(236e3180,104) at sched_switch+0x4e9/frame 0x23824a20 mi_switch(104) at mi_switch+0x1c4/frame 0x23824a54 sleepq_switch(2b00,236e3180,0,0,1d78dd4,...) at sleepq_switch+0xf7/frame 0x23824a74 sleepq_catch_signals(23824adc,f84428,188ed874,0,15114d4,...) at sleepq_catch_signals+0x278/frame 0x23824a9c sleepq_wait_sig(188ed874,0) at sleepq_wait_sig+0xe/frame 0x23824aa4 _cv_wait_sig(188ed874,188ed804) at _cv_wait_sig+0x138/frame 0x23824adc tty_wait(188ed800,188ed874) at tty_wait+0x35/frame 0x23824af4 ttydisc_read(188ed800,23824bd0,0) at ttydisc_read+0x1c6/frame 0x23824b28 ttydev_read(18aedc00,23824bd0,0) at ttydev_read+0x3d/frame 0x23824b44 devfs_read_f(1fc6ca8c,23824bd0,22bc7900,0,236e3180) at devfs_read_f+0xaa/frame 0x23824b74 dofileread(1fc6ca8c,23824bd0,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x23824ba4 sys_read(236e3180,236e3418,236e3180,23a5d398,b8505b8,...) at sys_read+0x78/frame 0x23824c08 syscall(23824ce8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x23824cdc Xint0x80_syscall() at 0xffc033f9/frame 0x23824cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command sendmail pid 808 tid 100148 td 0x235c3c00 sched_switch(235c3c00,104) at sched_switch+0x4e9/frame 0x20530ac0 mi_switch(104) at mi_switch+0x1c4/frame 0x20530af4 sleepq_switch(3640,14e9ff2,74,0,1d79914,...) at sleepq_switch+0xf7/frame 0x20530b14 sleepq_catch_signals(20530b84,fefb0c,22c2ad74,74,235c3c00,...) at sleepq_catch_signals+0x278/frame 0x20530b3c sleepq_wait_sig(22c2ad74,74) at sleepq_wait_sig+0xe/frame 0x20530b44 _sleep(22c2ad74,22c2adb8,174,1449b82,0,0,0,0,100) at _sleep+0x27c/frame 0x20530b84 kern_sigsuspend(235c3c00,0,0,0,0) at kern_sigsuspend+0xca/frame 0x20530bcc sys_sigsuspend(235c3c00,235c3e98,235c3c00,22c2ad10,235c3c00,...) at sys_sigsuspend+0x3b/frame 0x20530c08 syscall(20530ce8,3b,3b,3b,20a2c000,...) at syscall+0x1b8/frame 0x20530cdc Xint0x80_syscall() at 0xffc033f9/frame 0x20530cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command sendmail pid 802 tid 100191 td 0x236e3880 sched_switch(236e3880,104) at sched_switch+0x4e9/frame 0x2380e930 mi_switch(104) at mi_switch+0x1c4/frame 0x2380e968 sleepq_switch(2f80,1da4b44,0,0,1d79254,...) at sleepq_switch+0xf7/frame 0x2380e988 sleepq_catch_signals(e4,236e3880,1a79480,236e3880,2380e9e4,...) at sleepq_catch_signals+0x278/frame 0x2380e9b0 sleepq_timedwait_sig(18d5da64,0) at sleepq_timedwait_sig+0x17/frame 0x2380e9e0 _cv_timedwait_sig_sbt(18d5da64,18d5da50,3c01d988,1dba,50000000,0,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x2380ea1c seltdwait(1dba,50000000,0) at seltdwait+0x5e/frame 0x2380ea4c kern_select(236e3880,5,ffbfd0dc,0,0,2380ebf8,20) at kern_select+0x8e5/frame 0x2380ebd4 sys_select(236e3880,236e3b18,236e3880,23a5da20,236e3880,...) at sys_select+0x41/frame 0x2380ec08 syscall(2380ece8,3b,3b,3b,20a2c000,...) at syscall+0x1b8/frame 0x2380ecdc Xint0x80_syscall() at 0xffc033f9/frame 0x2380ecdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command cron pid 789 tid 100140 td 0x1fc9e000 sched_switch(1fc9e000,103) at sched_switch+0x4e9/frame 0x20500658 mi_switch(103,1d7ae60) at mi_switch+0x1c4/frame 0x20500690 turnstile_wait(231b5500,2817a700,0) at turnstile_wait+0x2f1/frame 0x205006ac __mtx_lock_sleep(1a92450,23beb500,0,1489335,4ee,...) at __mtx_lock_sleep+0x1b1/frame 0x2050070c __mtx_lock_flags(1a92450,0,1489335,4ee) at __mtx_lock_flags+0xcc/frame 0x20500738 vnlru_free_locked(0,82000,20181000,1fe31000,205007dc,...) at vnlru_free_locked+0x9a/frame 0x20500798 vn_alloc_hard(3afea740,205007e4,1281eb8,74,0,...) at vn_alloc_hard+0xaf/frame 0x205007b0 getnewvnode(1413efe,1fe31000,1ab0f40,20500814) at getnewvnode+0x261/frame 0x205007dc ffs_vgetf(1fe31000,42d801,0,202000,20500874,0) at ffs_vgetf+0xdf/frame 0x20500824 ffs_vget(1fe31000,42d801,0,202000,20500874) at ffs_vget+0x19/frame 0x20500844 ufs_lookup_ino(23bd1bc0,20500aac,20500ac4,0) at ufs_lookup_ino+0xba8/frame 0x205008fc ufs_lookup(20500938) at ufs_lookup+0x16/frame 0x20500914 VOP_CACHEDLOOKUP_APV(1ab0f40,20500938) at VOP_CACHEDLOOKUP_APV+0x4d/frame 0x20500928 vfs_cache_lookup(20500990) at vfs_cache_lookup+0x98/frame 0x2050095c VOP_LOOKUP_APV(1ab0f40,20500990) at VOP_LOOKUP_APV+0x4d/frame 0x20500970 lookup(20500a70) at lookup+0x580/frame 0x205009c4 namei(20500a70) at namei+0x6df/frame 0x20500a4c kern_statat(1fc9e000,0,ffffff9c,4022de,0,20500b30,0) at kern_statat+0x107/frame 0x20500b0c sys_fstatat(1fc9e000,1fc9e298,1fc9e000,22c2fa20,1fc9e000,...) at sys_fstatat+0x28/frame 0x20500c08 syscall(20500ce8,3b,3b,3b,ffbfeb84,...) at syscall+0x1b8/frame 0x20500cdc Xint0x80_syscall() at 0xffc033f9/frame 0x20500cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe1ac --- Tracing command sshd pid 786 tid 100184 td 0x22c46e00 sched_switch(22c46e00,104) at sched_switch+0x4e9/frame 0x205e8970 mi_switch(104) at mi_switch+0x1c4/frame 0x205e89a8 sleepq_switch(3b40,22c46e00,0,0,1d79e14,...) at sleepq_switch+0xf7/frame 0x205e89c8 sleepq_catch_signals(205e8a30,f84428,9ab09e4,0,9ab09d0,...) at sleepq_catch_signals+0x278/frame 0x205e89f0 sleepq_wait_sig(9ab09e4,0) at sleepq_wait_sig+0xe/frame 0x205e89f8 _cv_wait_sig(9ab09e4,9ab09d0) at _cv_wait_sig+0x138/frame 0x205e8a30 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x205e8a4c kern_select(22c46e00,7,2047c180,0,0,0,20) at kern_select+0x8e5/frame 0x205e8bd4 sys_select(22c46e00,22c47098,22c46e00,22c2e9cc,22c46e00,...) at sys_select+0x41/frame 0x205e8c08 syscall(205e8ce8,3b,3b,3b,20482000,...) at syscall+0x1b8/frame 0x205e8cdc Xint0x80_syscall() at 0xffc033f9/frame 0x205e8cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command inetd pid 747 tid 100147 td 0x22c5d880 sched_switch(22c5d880,104) at sched_switch+0x4e9/frame 0x2052b970 mi_switch(104) at mi_switch+0x1c4/frame 0x2052b9a8 sleepq_switch(f80,22c5d880,0,0,1d77254,...) at sleepq_switch+0xf7/frame 0x2052b9c8 sleepq_catch_signals(2052ba30,f84428,18d5dae4,0,140171b,...) at sleepq_catch_signals+0x278/frame 0x2052b9f0 sleepq_wait_sig(18d5dae4,0) at sleepq_wait_sig+0xe/frame 0x2052b9f8 _cv_wait_sig(18d5dae4,18d5dad0) at _cv_wait_sig+0x138/frame 0x2052ba30 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x2052ba4c kern_select(22c5d880,9,ffbfe5dc,0,0,0,20) at kern_select+0x8e5/frame 0x2052bbd4 sys_select(22c5d880,22c5db18,22c5d880,1fc51d10,22c5d880,...) at sys_select+0x41/frame 0x2052bc08 syscall(2052bce8,3b,3b,3b,ffbfe65c,...) at syscall+0x1b8/frame 0x2052bcdc Xint0x80_syscall() at 0xffc033f9/frame 0x2052bcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command smartd pid 739 tid 100187 td 0x235e6000 (CPU 2) cpustop_handler(205f761c,205f7610,13d59fb,205f75b4,b847000,...) at cpustop_handler+0x26/frame 0x205f7544 ipi_nmi_handler(205f75b4,b847000,205f75ac,0,1fc51344,...) at ipi_nmi_handler+0x30/frame 0x205f7550 trap(205f761c,8,28,28,3d741000,...) at trap+0xbb/frame 0x205f7610 calltrap() at 0xffc0319f/frame 0x205f7610 --- trap 0x13, eip = 0xfc4e9f, esp = 0x205f765c, ebp = 0x205f7668 --- __mtx_assert(3d741010,4,140b7af,2b6) at __mtx_assert+0xf/frame 0x205f7668 vop_stdgetwritemount(205f76b4) at vop_stdgetwritemount+0x27d/frame 0x205f7690 VOP_GETWRITEMOUNT_APV(1ab0f40,205f76b4) at VOP_GETWRITEMOUNT_APV+0x40/frame 0x205f76a4 vn_start_write(206128d0,205f76fc,2) at vn_start_write+0x36/frame 0x205f76cc vnlru_free_locked(0,18855600,18d5eb80,18cd4040,205f776c,...) at vnlru_free_locked+0x31d/frame 0x205f7728 vn_alloc_hard(205f776c,fc4dc7,1d3ff68,8,14d566e,...) at vn_alloc_hard+0xaf/frame 0x205f7740 getnewvnode(1400074,18cd4040,1a65c90,205f7790) at getnewvnode+0x261/frame 0x205f776c devfs_allocv(18d5eb80,18cd4040,80000,205f7b7c) at devfs_allocv+0x202/frame 0x205f77a0 devfs_lookup(205f7958) at devfs_lookup+0x40a/frame 0x205f7924 VOP_LOOKUP_APV(1a65c90,205f7958) at VOP_LOOKUP_APV+0x4d/frame 0x205f7938 lookup(205f7b40) at lookup+0x580/frame 0x205f798c namei(205f7b40) at namei+0x6df/frame 0x205f7a18 vn_open_cred(205f7b40,205f7bf8,0,0,1890cb00,2ae50564) at vn_open_cred+0x2f8/frame 0x205f7ad8 vn_open(205f7b40,205f7bf8,0,2ae50564) at vn_open+0x20/frame 0x205f7af8 kern_openat(235e6000,ffffff9c,204b7ec3,0,3,0) at kern_openat+0x2fa/frame 0x205f7be0 sys_openat(235e6000,235e6298,235e6000,1fc51344,235e6000,...) at sys_openat+0x2f/frame 0x205f7c08 syscall(205f7ce8,3b,3b,3b,20835000,...) at syscall+0x1b8/frame 0x205f7cdc Xint0x80_syscall() at 0xffc033f9/frame 0x205f7cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe808 --- Tracing command powerd pid 696 tid 100145 td 0x22c5dc00 sched_switch(22c5dc00,104) at sched_switch+0x4e9/frame 0x20519930 mi_switch(104) at mi_switch+0x1c4/frame 0x20519968 sleepq_switch(b80,1da4cfc,0,0,1d76e54,...) at sleepq_switch+0xf7/frame 0x20519988 sleepq_catch_signals(e4,22c5dc00,1a79480,22c5dc00,205199e4,...) at sleepq_catch_signals+0x278/frame 0x205199b0 sleepq_timedwait_sig(9ab0a24,0) at sleepq_timedwait_sig+0x17/frame 0x205199e0 _cv_timedwait_sig_sbt(9ab0a24,9ab0a10,7c01d9f0,1dba,4000000,0,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x20519a1c seltdwait(1dba,4000000,0) at seltdwait+0x5e/frame 0x20519a4c kern_select(22c5dc00,0,ffbfece8,0,ffbfece8,20519bf8,20) at kern_select+0x8e5/frame 0x20519bd4 sys_select(22c5dc00,22c5de98,22c5dc00,1fc52054,22c5dc00,...) at sys_select+0x41/frame 0x20519c08 syscall(20519ce8,3b,3b,3b,ffbfed68,...) at syscall+0x1b8/frame 0x20519cdc Xint0x80_syscall() at 0xffc033f9/frame 0x20519cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d98, esp = 0xffc0bfe8, ebp = 0xffbfeccc --- Tracing command ntpd pid 693 tid 100186 td 0x235e6380 sched_switch(235e6380,104) at sched_switch+0x4e9/frame 0x205f2970 mi_switch(104) at mi_switch+0x1c4/frame 0x205f29a8 sleepq_switch(17c0,235e6380,0,0,1d77a94,...) at sleepq_switch+0xf7/frame 0x205f29c8 sleepq_catch_signals(205f2a30,f84428,1880bbe4,0,1880bbd0,...) at sleepq_catch_signals+0x278/frame 0x205f29f0 sleepq_wait_sig(1880bbe4,0) at sleepq_wait_sig+0xe/frame 0x205f29f8 _cv_wait_sig(1880bbe4,1880bbd0) at _cv_wait_sig+0x138/frame 0x205f2a30 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x205f2a4c kern_select(235e6380,1b,ffbfe25c,0,0,0,20) at kern_select+0x8e5/frame 0x205f2bd4 sys_select(235e6380,235e6618,235e6380,1fc51688,205f2cdc,...) at sys_select+0x41/frame 0x205f2c08 syscall(205f2ce8,3b,3b,3b,209ae000,...) at syscall+0x1b8/frame 0x205f2cdc Xint0x80_syscall() at 0xffc033f9/frame 0x205f2cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe1ac --- Tracing command rpc.lockd pid 672 tid 100139 td 0x22bc0a80 sched_switch(22bc0a80,104) at sched_switch+0x4e9/frame 0x204f78e8 mi_switch(104) at mi_switch+0x1c4/frame 0x204f7920 sleepq_switch(3a40,1da4dac,0,0,1d79d14,...) at sleepq_switch+0xf7/frame 0x204f7940 sleepq_catch_signals(e4,22bc0a80,1a79480,22bc0a80,204f799c,...) at sleepq_catch_signals+0x278/frame 0x204f7968 sleepq_timedwait_sig(1890b950,0) at sleepq_timedwait_sig+0x17/frame 0x204f7998 _cv_timedwait_sig_sbt(1890b950,235f10c0,fffffa38,4,0,0,100) at _cv_timedwait_sig_sbt+0x15a/frame 0x204f79d4 svc_run_internal(4,235f1000,204f7c08,11f1470,1fc1fce0,...) at svc_run_internal+0xa1c/frame 0x204f7ab4 svc_run(235f1000) at svc_run+0x192/frame 0x204f7ae4 sys_nlm_syscall(22bc0a80,22bc0d18,22bc0a80,22c2ba20,22bc0a80,...) at sys_nlm_syscall+0x620/frame 0x204f7c08 syscall(204f7ce8,3b,3b,3b,2074b000,...) at syscall+0x1b8/frame 0x204f7cdc Xint0x80_syscall() at 0xffc033f9/frame 0x204f7cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe1ac --- Tracing command rpc.statd pid 669 tid 100141 td 0x22c47c00 sched_switch(22c47c00,104) at sched_switch+0x4e9/frame 0x20505930 mi_switch(104) at mi_switch+0x1c4/frame 0x20505968 sleepq_switch(2600,1da4e04,0,0,1d788d4,...) at sleepq_switch+0xf7/frame 0x20505988 sleepq_catch_signals(e4,22c47c00,1a79480,22c47c00,205059e4,...) at sleepq_catch_signals+0x278/frame 0x205059b0 sleepq_timedwait_sig(1880bc24,0) at sleepq_timedwait_sig+0x17/frame 0x205059e0 _cv_timedwait_sig_sbt(1880bc24,1880bc10,1495353c,1dd5,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x20505a1c seltdwait(1dd5,e0000000,1) at seltdwait+0x5e/frame 0x20505a4c kern_select(22c47c00,8,ffbfeca8,0,0,20505bf8,20) at kern_select+0x8e5/frame 0x20505bd4 sys_select(22c47c00,22c47e98,22c47c00,22c2f6dc,22c47c00,...) at sys_select+0x41/frame 0x20505c08 syscall(20505ce8,3b,3b,3b,ffbfeca8,...) at syscall+0x1b8/frame 0x20505cdc Xint0x80_syscall() at 0xffc033f9/frame 0x20505cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command nfsd pid 668 tid 100152 td 0x22c47180 sched_switch(22c47180,104) at sched_switch+0x4e9/frame 0x20548418 mi_switch(104) at mi_switch+0x1c4/frame 0x20548450 sleepq_switch(1a40,1da548c,0,0,1d77d14,...) at sleepq_switch+0xf7/frame 0x20548470 sleepq_catch_signals(e4,22c47180,1a79480,22c47180,205484cc,...) at sleepq_catch_signals+0x278/frame 0x20548498 sleepq_timedwait_sig(1890b9d0,0) at sleepq_timedwait_sig+0x17/frame 0x205484c8 _cv_timedwait_sig_sbt(1890b9d0,188030c0,fffffa38,4,0,0,100) at _cv_timedwait_sig_sbt+0x15a/frame 0x20548504 svc_run_internal(235e6a80,188030c0,188030d0,1880311c,0,...) at svc_run_internal+0xa1c/frame 0x205485e8 svc_run(18803000) at svc_run+0x192/frame 0x20548618 nfsrvd_nfsd(22c47180,205487d0) at nfsrvd_nfsd+0x2e8/frame 0x20548748 nfssvc_nfsd(22c47180,22c47418) at nfssvc_nfsd+0x4cb/frame 0x20548bf0 sys_nfssvc(22c47180,22c47418,22c47180,22c2ed10,22c47180,...) at sys_nfssvc+0x92/frame 0x20548c08 syscall(20548ce8,3b,3b,3b,ffbfec94,...) at syscall+0x1b8/frame 0x20548cdc Xint0x80_syscall() at 0xffc033f9/frame 0x20548cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command nfsd pid 668 tid 100153 td 0x22c5d500 sched_switch(22c5d500,104) at sched_switch+0x4e9/frame 0x2054db20 mi_switch(104) at mi_switch+0x1c4/frame 0x2054db58 sleepq_switch(3b00,22c5d500,0,0,1d79dd4,...) at sleepq_switch+0xf7/frame 0x2054db78 sleepq_catch_signals(2054dbe0,f84428,1890bc50,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x2054dba0 sleepq_wait_sig(1890bc50,0) at sleepq_wait_sig+0xe/frame 0x2054dba8 _cv_wait_sig(1890bc50,188030c0) at _cv_wait_sig+0x138/frame 0x2054dbe0 svc_run_internal(2054dcd4,fa7168,188030c0,2054dce8,120e230,...) at svc_run_internal+0xb45/frame 0x2054dcb0 svc_thread_start(188030c0,2054dce8) at svc_thread_start+0xd/frame 0x2054dcb8 fork_exit(120e230,188030c0,2054dce8,3b,3b,...) at fork_exit+0x68/frame 0x2054dcd4 fork_trampoline() at 0xffc0340e/frame 0x2054dcd4 --- kthread start Tracing command nfsd pid 668 tid 100154 td 0x22c5d180 sched_switch(22c5d180,104) at sched_switch+0x4e9/frame 0x20552b20 mi_switch(104) at mi_switch+0x1c4/frame 0x20552b58 sleepq_switch(1900,22c5d180,0,0,1d77bd4,...) at sleepq_switch+0xf7/frame 0x20552b78 sleepq_catch_signals(20552be0,f84428,9a2b4d0,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x20552ba0 sleepq_wait_sig(9a2b4d0,0) at sleepq_wait_sig+0xe/frame 0x20552ba8 _cv_wait_sig(9a2b4d0,188030c0) at _cv_wait_sig+0x138/frame 0x20552be0 svc_run_internal(20552cd4,fa7168,188030c0,20552ce8,120e230,...) at svc_run_internal+0xb45/frame 0x20552cb0 svc_thread_start(188030c0,20552ce8) at svc_thread_start+0xd/frame 0x20552cb8 fork_exit(120e230,188030c0,20552ce8,3b,3b,...) at fork_exit+0x68/frame 0x20552cd4 fork_trampoline() at 0xffc0340e/frame 0x20552cd4 --- kthread start Tracing command nfsd pid 668 tid 100155 td 0x22c5ce00 sched_switch(22c5ce00,104) at sched_switch+0x4e9/frame 0x20557b20 mi_switch(104) at mi_switch+0x1c4/frame 0x20557b58 sleepq_switch(3ac0,22c5ce00,0,0,1d79d94,...) at sleepq_switch+0xf7/frame 0x20557b78 sleepq_catch_signals(20557be0,f84428,1890bb50,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x20557ba0 sleepq_wait_sig(1890bb50,0) at sleepq_wait_sig+0xe/frame 0x20557ba8 _cv_wait_sig(1890bb50,188030c0) at _cv_wait_sig+0x138/frame 0x20557be0 svc_run_internal(20557cd4,fa7168,188030c0,20557ce8,120e230,...) at svc_run_internal+0xb45/frame 0x20557cb0 svc_thread_start(188030c0,20557ce8) at svc_thread_start+0xd/frame 0x20557cb8 fork_exit(120e230,188030c0,20557ce8,3b,3b,...) at fork_exit+0x68/frame 0x20557cd4 fork_trampoline() at 0xffc0340e/frame 0x20557cd4 --- kthread start Tracing command nfsd pid 668 tid 100156 td 0x22c5ca80 sched_switch(22c5ca80,104) at sched_switch+0x4e9/frame 0x2055cb20 mi_switch(104) at mi_switch+0x1c4/frame 0x2055cb58 sleepq_switch(3940,22c5ca80,0,0,1d79c14,...) at sleepq_switch+0xf7/frame 0x2055cb78 sleepq_catch_signals(2055cbe0,f84428,9a2b550,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x2055cba0 sleepq_wait_sig(9a2b550,0) at sleepq_wait_sig+0xe/frame 0x2055cba8 _cv_wait_sig(9a2b550,188030c0) at _cv_wait_sig+0x138/frame 0x2055cbe0 svc_run_internal(2055ccd4,fa7168,188030c0,2055cce8,120e230,...) at svc_run_internal+0xb45/frame 0x2055ccb0 svc_thread_start(188030c0,2055cce8) at svc_thread_start+0xd/frame 0x2055ccb8 fork_exit(120e230,188030c0,2055cce8,3b,3b,...) at fork_exit+0x68/frame 0x2055ccd4 fork_trampoline() at 0xffc0340e/frame 0x2055ccd4 --- kthread start Tracing command nfsd pid 668 tid 100157 td 0x22c5c700 sched_switch(22c5c700,104) at sched_switch+0x4e9/frame 0x20561b20 mi_switch(104) at mi_switch+0x1c4/frame 0x20561b58 sleepq_switch(3b40,22c5c700,0,0,1d79e14,...) at sleepq_switch+0xf7/frame 0x20561b78 sleepq_catch_signals(20561be0,f84428,1890bd50,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x20561ba0 sleepq_wait_sig(1890bd50,0) at sleepq_wait_sig+0xe/frame 0x20561ba8 _cv_wait_sig(1890bd50,188030c0) at _cv_wait_sig+0x138/frame 0x20561be0 svc_run_internal(20561cd4,fa7168,188030c0,20561ce8,120e230,...) at svc_run_internal+0xb45/frame 0x20561cb0 svc_thread_start(188030c0,20561ce8) at svc_thread_start+0xd/frame 0x20561cb8 fork_exit(120e230,188030c0,20561ce8,3b,3b,...) at fork_exit+0x68/frame 0x20561cd4 fork_trampoline() at 0xffc0340e/frame 0x20561cd4 --- kthread start Tracing command nfsd pid 668 tid 100158 td 0x22c5c380 sched_switch(22c5c380,104) at sched_switch+0x4e9/frame 0x20566b20 mi_switch(104) at mi_switch+0x1c4/frame 0x20566b58 sleepq_switch(1940,22c5c380,0,0,1d77c14,...) at sleepq_switch+0xf7/frame 0x20566b78 sleepq_catch_signals(20566be0,f84428,9a2b5d0,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x20566ba0 sleepq_wait_sig(9a2b5d0,0) at sleepq_wait_sig+0xe/frame 0x20566ba8 _cv_wait_sig(9a2b5d0,188030c0) at _cv_wait_sig+0x138/frame 0x20566be0 svc_run_internal(20566cd4,fa7168,188030c0,20566ce8,120e230,...) at svc_run_internal+0xb45/frame 0x20566cb0 svc_thread_start(188030c0,20566ce8) at svc_thread_start+0xd/frame 0x20566cb8 fork_exit(120e230,188030c0,20566ce8,3b,3b,...) at fork_exit+0x68/frame 0x20566cd4 fork_trampoline() at 0xffc0340e/frame 0x20566cd4 --- kthread start Tracing command nfsd pid 668 tid 100159 td 0x22c5c000 sched_switch(22c5c000,104) at sched_switch+0x4e9/frame 0x2056bb20 mi_switch(104) at mi_switch+0x1c4/frame 0x2056bb58 sleepq_switch(38c0,22c5c000,0,0,1d79b94,...) at sleepq_switch+0xf7/frame 0x2056bb78 sleepq_catch_signals(2056bbe0,f84428,9a2b350,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x2056bba0 sleepq_wait_sig(9a2b350,0) at sleepq_wait_sig+0xe/frame 0x2056bba8 _cv_wait_sig(9a2b350,188030c0) at _cv_wait_sig+0x138/frame 0x2056bbe0 svc_run_internal(2056bcd4,fa7168,188030c0,2056bce8,120e230,...) at svc_run_internal+0xb45/frame 0x2056bcb0 svc_thread_start(188030c0,2056bce8) at svc_thread_start+0xd/frame 0x2056bcb8 fork_exit(120e230,188030c0,2056bce8,3b,3b,...) at fork_exit+0x68/frame 0x2056bcd4 fork_trampoline() at 0xffc0340e/frame 0x2056bcd4 --- kthread start Tracing command nfsd pid 668 tid 100160 td 0x235dfc00 sched_switch(235dfc00,104) at sched_switch+0x4e9/frame 0x20570b20 mi_switch(104) at mi_switch+0x1c4/frame 0x20570b58 sleepq_switch(f00,235dfc00,0,0,1d771d4,...) at sleepq_switch+0xf7/frame 0x20570b78 sleepq_catch_signals(20570be0,f84428,202becd0,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x20570ba0 sleepq_wait_sig(202becd0,0) at sleepq_wait_sig+0xe/frame 0x20570ba8 _cv_wait_sig(202becd0,188030c0) at _cv_wait_sig+0x138/frame 0x20570be0 svc_run_internal(20570cd4,fa7168,188030c0,20570ce8,120e230,...) at svc_run_internal+0xb45/frame 0x20570cb0 svc_thread_start(188030c0,20570ce8) at svc_thread_start+0xd/frame 0x20570cb8 fork_exit(120e230,188030c0,20570ce8,3b,3b,...) at fork_exit+0x68/frame 0x20570cd4 fork_trampoline() at 0xffc0340e/frame 0x20570cd4 --- kthread start Tracing command nfsd pid 668 tid 100161 td 0x235df880 sched_switch(235df880,104) at sched_switch+0x4e9/frame 0x20575b20 mi_switch(104) at mi_switch+0x1c4/frame 0x20575b58 sleepq_switch(18c0,235df880,0,0,1d77b94,...) at sleepq_switch+0xf7/frame 0x20575b78 sleepq_catch_signals(20575be0,f84428,9a2b3d0,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x20575ba0 sleepq_wait_sig(9a2b3d0,0) at sleepq_wait_sig+0xe/frame 0x20575ba8 _cv_wait_sig(9a2b3d0,188030c0) at _cv_wait_sig+0x138/frame 0x20575be0 svc_run_internal(20575cd4,fa7168,188030c0,20575ce8,120e230,...) at svc_run_internal+0xb45/frame 0x20575cb0 svc_thread_start(188030c0,20575ce8) at svc_thread_start+0xd/frame 0x20575cb8 fork_exit(120e230,188030c0,20575ce8,3b,3b,...) at fork_exit+0x68/frame 0x20575cd4 fork_trampoline() at 0xffc0340e/frame 0x20575cd4 --- kthread start Tracing command nfsd pid 668 tid 100162 td 0x235df500 sched_switch(235df500,104) at sched_switch+0x4e9/frame 0x2057ab20 mi_switch(104) at mi_switch+0x1c4/frame 0x2057ab58 sleepq_switch(3b80,235df500,0,0,1d79e54,...) at sleepq_switch+0xf7/frame 0x2057ab78 sleepq_catch_signals(2057abe0,f84428,1890be50,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x2057aba0 sleepq_wait_sig(1890be50,0) at sleepq_wait_sig+0xe/frame 0x2057aba8 _cv_wait_sig(1890be50,188030c0) at _cv_wait_sig+0x138/frame 0x2057abe0 svc_run_internal(2057acd4,fa7168,188030c0,2057ace8,120e230,...) at svc_run_internal+0xb45/frame 0x2057acb0 svc_thread_start(188030c0,2057ace8) at svc_thread_start+0xd/frame 0x2057acb8 fork_exit(120e230,188030c0,2057ace8,3b,3b,...) at fork_exit+0x68/frame 0x2057acd4 fork_trampoline() at 0xffc0340e/frame 0x2057acd4 --- kthread start Tracing command nfsd pid 668 tid 100163 td 0x235df180 sched_switch(235df180,104) at sched_switch+0x4e9/frame 0x2057fb20 mi_switch(104) at mi_switch+0x1c4/frame 0x2057fb58 sleepq_switch(39c0,235df180,0,0,1d79c94,...) at sleepq_switch+0xf7/frame 0x2057fb78 sleepq_catch_signals(2057fbe0,f84428,9a2b750,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x2057fba0 sleepq_wait_sig(9a2b750,0) at sleepq_wait_sig+0xe/frame 0x2057fba8 _cv_wait_sig(9a2b750,188030c0) at _cv_wait_sig+0x138/frame 0x2057fbe0 svc_run_internal(2057fcd4,fa7168,188030c0,2057fce8,120e230,...) at svc_run_internal+0xb45/frame 0x2057fcb0 svc_thread_start(188030c0,2057fce8) at svc_thread_start+0xd/frame 0x2057fcb8 fork_exit(120e230,188030c0,2057fce8,3b,3b,...) at fork_exit+0x68/frame 0x2057fcd4 fork_trampoline() at 0xffc0340e/frame 0x2057fcd4 --- kthread start Tracing command nfsd pid 668 tid 100164 td 0x235dee00 sched_switch(235dee00,104) at sched_switch+0x4e9/frame 0x20584b20 mi_switch(104) at mi_switch+0x1c4/frame 0x20584b58 sleepq_switch(3980,235dee00,0,0,1d79c54,...) at sleepq_switch+0xf7/frame 0x20584b78 sleepq_catch_signals(20584be0,f84428,9a2b650,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x20584ba0 sleepq_wait_sig(9a2b650,0) at sleepq_wait_sig+0xe/frame 0x20584ba8 _cv_wait_sig(9a2b650,188030c0) at _cv_wait_sig+0x138/frame 0x20584be0 svc_run_internal(20584cd4,fa7168,188030c0,20584ce8,120e230,...) at svc_run_internal+0xb45/frame 0x20584cb0 svc_thread_start(188030c0,20584ce8) at svc_thread_start+0xd/frame 0x20584cb8 fork_exit(120e230,188030c0,20584ce8,3b,3b,...) at fork_exit+0x68/frame 0x20584cd4 fork_trampoline() at 0xffc0340e/frame 0x20584cd4 --- kthread start Tracing command nfsd pid 668 tid 100165 td 0x235dea80 sched_switch(235dea80,104) at sched_switch+0x4e9/frame 0x20589b20 mi_switch(104) at mi_switch+0x1c4/frame 0x20589b58 sleepq_switch(1880,235dea80,0,0,1d77b54,...) at sleepq_switch+0xf7/frame 0x20589b78 sleepq_catch_signals(20589be0,f84428,9a2b2d0,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x20589ba0 sleepq_wait_sig(9a2b2d0,0) at sleepq_wait_sig+0xe/frame 0x20589ba8 _cv_wait_sig(9a2b2d0,188030c0) at _cv_wait_sig+0x138/frame 0x20589be0 svc_run_internal(20589cd4,fa7168,188030c0,20589ce8,120e230,...) at svc_run_internal+0xb45/frame 0x20589cb0 svc_thread_start(188030c0,20589ce8) at svc_thread_start+0xd/frame 0x20589cb8 fork_exit(120e230,188030c0,20589ce8,3b,3b,...) at fork_exit+0x68/frame 0x20589cd4 fork_trampoline() at 0xffc0340e/frame 0x20589cd4 --- kthread start Tracing command nfsd pid 668 tid 100166 td 0x235de700 sched_switch(235de700,104) at sched_switch+0x4e9/frame 0x2058eb20 mi_switch(104) at mi_switch+0x1c4/frame 0x2058eb58 sleepq_switch(e40,235de700,0,0,1d77114,...) at sleepq_switch+0xf7/frame 0x2058eb78 sleepq_catch_signals(2058ebe0,f84428,202be9d0,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x2058eba0 sleepq_wait_sig(202be9d0,0) at sleepq_wait_sig+0xe/frame 0x2058eba8 _cv_wait_sig(202be9d0,188030c0) at _cv_wait_sig+0x138/frame 0x2058ebe0 svc_run_internal(2058ecd4,fa7168,188030c0,2058ece8,120e230,...) at svc_run_internal+0xb45/frame 0x2058ecb0 svc_thread_start(188030c0,2058ece8) at svc_thread_start+0xd/frame 0x2058ecb8 fork_exit(120e230,188030c0,2058ece8,3b,3b,...) at fork_exit+0x68/frame 0x2058ecd4 fork_trampoline() at 0xffc0340e/frame 0x2058ecd4 --- kthread start Tracing command nfsd pid 668 tid 100167 td 0x235de380 sched_switch(235de380,104) at sched_switch+0x4e9/frame 0x20593b20 mi_switch(104) at mi_switch+0x1c4/frame 0x20593b58 sleepq_switch(3880,235de380,0,0,1d79b54,...) at sleepq_switch+0xf7/frame 0x20593b78 sleepq_catch_signals(20593be0,f84428,9a2b250,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x20593ba0 sleepq_wait_sig(9a2b250,0) at sleepq_wait_sig+0xe/frame 0x20593ba8 _cv_wait_sig(9a2b250,188030c0) at _cv_wait_sig+0x138/frame 0x20593be0 svc_run_internal(20593cd4,fa7168,188030c0,20593ce8,120e230,...) at svc_run_internal+0xb45/frame 0x20593cb0 svc_thread_start(188030c0,20593ce8) at svc_thread_start+0xd/frame 0x20593cb8 fork_exit(120e230,188030c0,20593ce8,3b,3b,...) at fork_exit+0x68/frame 0x20593cd4 fork_trampoline() at 0xffc0340e/frame 0x20593cd4 --- kthread start Tracing command nfsd pid 668 tid 100168 td 0x235de000 sched_switch(235de000,104) at sched_switch+0x4e9/frame 0x20598b20 mi_switch(104) at mi_switch+0x1c4/frame 0x20598b58 sleepq_switch(2e40,235de000,0,0,1d79114,...) at sleepq_switch+0xf7/frame 0x20598b78 sleepq_catch_signals(20598be0,f84428,202be950,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x20598ba0 sleepq_wait_sig(202be950,0) at sleepq_wait_sig+0xe/frame 0x20598ba8 _cv_wait_sig(202be950,188030c0) at _cv_wait_sig+0x138/frame 0x20598be0 svc_run_internal(20598cd4,fa7168,188030c0,20598ce8,120e230,...) at svc_run_internal+0xb45/frame 0x20598cb0 svc_thread_start(188030c0,20598ce8) at svc_thread_start+0xd/frame 0x20598cb8 fork_exit(120e230,188030c0,20598ce8,3b,3b,...) at fork_exit+0x68/frame 0x20598cd4 fork_trampoline() at 0xffc0340e/frame 0x20598cd4 --- kthread start Tracing command nfsd pid 668 tid 100169 td 0x235e1c00 sched_switch(235e1c00,104) at sched_switch+0x4e9/frame 0x2059db20 mi_switch(104) at mi_switch+0x1c4/frame 0x2059db58 sleepq_switch(1840,235e1c00,0,0,1d77b14,...) at sleepq_switch+0xf7/frame 0x2059db78 sleepq_catch_signals(2059dbe0,f84428,9a2b1d0,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x2059dba0 sleepq_wait_sig(9a2b1d0,0) at sleepq_wait_sig+0xe/frame 0x2059dba8 _cv_wait_sig(9a2b1d0,188030c0) at _cv_wait_sig+0x138/frame 0x2059dbe0 svc_run_internal(2059dcd4,fa7168,188030c0,2059dce8,120e230,...) at svc_run_internal+0xb45/frame 0x2059dcb0 svc_thread_start(188030c0,2059dce8) at svc_thread_start+0xd/frame 0x2059dcb8 fork_exit(120e230,188030c0,2059dce8,3b,3b,...) at fork_exit+0x68/frame 0x2059dcd4 fork_trampoline() at 0xffc0340e/frame 0x2059dcd4 --- kthread start Tracing command nfsd pid 668 tid 100170 td 0x235e1880 sched_switch(235e1880,104) at sched_switch+0x4e9/frame 0x205a2b20 mi_switch(104) at mi_switch+0x1c4/frame 0x205a2b58 sleepq_switch(3840,235e1880,0,0,1d79b14,...) at sleepq_switch+0xf7/frame 0x205a2b78 sleepq_catch_signals(205a2be0,f84428,9a2b150,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x205a2ba0 sleepq_wait_sig(9a2b150,0) at sleepq_wait_sig+0xe/frame 0x205a2ba8 _cv_wait_sig(9a2b150,188030c0) at _cv_wait_sig+0x138/frame 0x205a2be0 svc_run_internal(205a2cd4,fa7168,188030c0,205a2ce8,120e230,...) at svc_run_internal+0xb45/frame 0x205a2cb0 svc_thread_start(188030c0,205a2ce8) at svc_thread_start+0xd/frame 0x205a2cb8 fork_exit(120e230,188030c0,205a2ce8,3b,3b,...) at fork_exit+0x68/frame 0x205a2cd4 fork_trampoline() at 0xffc0340e/frame 0x205a2cd4 --- kthread start Tracing command nfsd pid 668 tid 100171 td 0x235e1500 sched_switch(235e1500,104) at sched_switch+0x4e9/frame 0x205a7b20 mi_switch(104) at mi_switch+0x1c4/frame 0x205a7b58 sleepq_switch(1800,235e1500,0,0,1d77ad4,...) at sleepq_switch+0xf7/frame 0x205a7b78 sleepq_catch_signals(205a7be0,f84428,9a2b0d0,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x205a7ba0 sleepq_wait_sig(9a2b0d0,0) at sleepq_wait_sig+0xe/frame 0x205a7ba8 _cv_wait_sig(9a2b0d0,188030c0) at _cv_wait_sig+0x138/frame 0x205a7be0 svc_run_internal(205a7cd4,fa7168,188030c0,205a7ce8,120e230,...) at svc_run_internal+0xb45/frame 0x205a7cb0 svc_thread_start(188030c0,205a7ce8) at svc_thread_start+0xd/frame 0x205a7cb8 fork_exit(120e230,188030c0,205a7ce8,3b,3b,...) at fork_exit+0x68/frame 0x205a7cd4 fork_trampoline() at 0xffc0340e/frame 0x205a7cd4 --- kthread start Tracing command nfsd pid 668 tid 100172 td 0x235e1180 sched_switch(235e1180,104) at sched_switch+0x4e9/frame 0x205acb20 mi_switch(104) at mi_switch+0x1c4/frame 0x205acb58 sleepq_switch(e00,235e1180,0,0,1d770d4,...) at sleepq_switch+0xf7/frame 0x205acb78 sleepq_catch_signals(205acbe0,f84428,202be8d0,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x205acba0 sleepq_wait_sig(202be8d0,0) at sleepq_wait_sig+0xe/frame 0x205acba8 _cv_wait_sig(202be8d0,188030c0) at _cv_wait_sig+0x138/frame 0x205acbe0 svc_run_internal(205accd4,fa7168,188030c0,205acce8,120e230,...) at svc_run_internal+0xb45/frame 0x205accb0 svc_thread_start(188030c0,205acce8) at svc_thread_start+0xd/frame 0x205accb8 fork_exit(120e230,188030c0,205acce8,3b,3b,...) at fork_exit+0x68/frame 0x205accd4 fork_trampoline() at 0xffc0340e/frame 0x205accd4 --- kthread start Tracing command nfsd pid 668 tid 100173 td 0x235e0e00 sched_switch(235e0e00,104) at sched_switch+0x4e9/frame 0x205b1b20 mi_switch(104) at mi_switch+0x1c4/frame 0x205b1b58 sleepq_switch(3800,235e0e00,0,0,1d79ad4,...) at sleepq_switch+0xf7/frame 0x205b1b78 sleepq_catch_signals(205b1be0,f84428,9a2b050,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x205b1ba0 sleepq_wait_sig(9a2b050,0) at sleepq_wait_sig+0xe/frame 0x205b1ba8 _cv_wait_sig(9a2b050,188030c0) at _cv_wait_sig+0x138/frame 0x205b1be0 svc_run_internal(205b1cd4,fa7168,188030c0,205b1ce8,120e230,...) at svc_run_internal+0xb45/frame 0x205b1cb0 svc_thread_start(188030c0,205b1ce8) at svc_thread_start+0xd/frame 0x205b1cb8 fork_exit(120e230,188030c0,205b1ce8,3b,3b,...) at fork_exit+0x68/frame 0x205b1cd4 fork_trampoline() at 0xffc0340e/frame 0x205b1cd4 --- kthread start Tracing command nfsd pid 668 tid 100174 td 0x235e0a80 sched_switch(235e0a80,104) at sched_switch+0x4e9/frame 0x205b6b20 mi_switch(104) at mi_switch+0x1c4/frame 0x205b6b58 sleepq_switch(2e00,235e0a80,0,0,1d790d4,...) at sleepq_switch+0xf7/frame 0x205b6b78 sleepq_catch_signals(205b6be0,f84428,202be850,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x205b6ba0 sleepq_wait_sig(202be850,0) at sleepq_wait_sig+0xe/frame 0x205b6ba8 _cv_wait_sig(202be850,188030c0) at _cv_wait_sig+0x138/frame 0x205b6be0 svc_run_internal(205b6cd4,fa7168,188030c0,205b6ce8,120e230,...) at svc_run_internal+0xb45/frame 0x205b6cb0 svc_thread_start(188030c0,205b6ce8) at svc_thread_start+0xd/frame 0x205b6cb8 fork_exit(120e230,188030c0,205b6ce8,3b,3b,...) at fork_exit+0x68/frame 0x205b6cd4 fork_trampoline() at 0xffc0340e/frame 0x205b6cd4 --- kthread start Tracing command nfsd pid 668 tid 100175 td 0x235e0700 sched_switch(235e0700,104) at sched_switch+0x4e9/frame 0x205bbb20 mi_switch(104) at mi_switch+0x1c4/frame 0x205bbb58 sleepq_switch(3c0,235e0700,0,0,1d76694,...) at sleepq_switch+0xf7/frame 0x205bbb78 sleepq_catch_signals(205bbbe0,f84428,235e5f50,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x205bbba0 sleepq_wait_sig(235e5f50,0) at sleepq_wait_sig+0xe/frame 0x205bbba8 _cv_wait_sig(235e5f50,188030c0) at _cv_wait_sig+0x138/frame 0x205bbbe0 svc_run_internal(205bbcd4,fa7168,188030c0,205bbce8,120e230,...) at svc_run_internal+0xb45/frame 0x205bbcb0 svc_thread_start(188030c0,205bbce8) at svc_thread_start+0xd/frame 0x205bbcb8 fork_exit(120e230,188030c0,205bbce8,3b,3b,...) at fork_exit+0x68/frame 0x205bbcd4 fork_trampoline() at 0xffc0340e/frame 0x205bbcd4 --- kthread start Tracing command nfsd pid 668 tid 100176 td 0x235e0380 sched_switch(235e0380,104) at sched_switch+0x4e9/frame 0x205c0b20 mi_switch(104) at mi_switch+0x1c4/frame 0x205c0b58 sleepq_switch(dc0,235e0380,0,0,1d77094,...) at sleepq_switch+0xf7/frame 0x205c0b78 sleepq_catch_signals(205c0be0,f84428,202be7d0,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x205c0ba0 sleepq_wait_sig(202be7d0,0) at sleepq_wait_sig+0xe/frame 0x205c0ba8 _cv_wait_sig(202be7d0,188030c0) at _cv_wait_sig+0x138/frame 0x205c0be0 svc_run_internal(205c0cd4,fa7168,188030c0,205c0ce8,120e230,...) at svc_run_internal+0xb45/frame 0x205c0cb0 svc_thread_start(188030c0,205c0ce8) at svc_thread_start+0xd/frame 0x205c0cb8 fork_exit(120e230,188030c0,205c0ce8,3b,3b,...) at fork_exit+0x68/frame 0x205c0cd4 fork_trampoline() at 0xffc0340e/frame 0x205c0cd4 --- kthread start Tracing command nfsd pid 668 tid 100177 td 0x235e0000 sched_switch(235e0000,104) at sched_switch+0x4e9/frame 0x205c5b20 mi_switch(104) at mi_switch+0x1c4/frame 0x205c5b58 sleepq_switch(2dc0,235e0000,0,0,1d79094,...) at sleepq_switch+0xf7/frame 0x205c5b78 sleepq_catch_signals(205c5be0,f84428,202be750,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x205c5ba0 sleepq_wait_sig(202be750,0) at sleepq_wait_sig+0xe/frame 0x205c5ba8 _cv_wait_sig(202be750,188030c0) at _cv_wait_sig+0x138/frame 0x205c5be0 svc_run_internal(205c5cd4,fa7168,188030c0,205c5ce8,120e230,...) at svc_run_internal+0xb45/frame 0x205c5cb0 svc_thread_start(188030c0,205c5ce8) at svc_thread_start+0xd/frame 0x205c5cb8 fork_exit(120e230,188030c0,205c5ce8,3b,3b,...) at fork_exit+0x68/frame 0x205c5cd4 fork_trampoline() at 0xffc0340e/frame 0x205c5cd4 --- kthread start Tracing command nfsd pid 668 tid 100178 td 0x235e7c00 sched_switch(235e7c00,104) at sched_switch+0x4e9/frame 0x205cab20 mi_switch(104) at mi_switch+0x1c4/frame 0x205cab58 sleepq_switch(d80,235e7c00,0,0,1d77054,...) at sleepq_switch+0xf7/frame 0x205cab78 sleepq_catch_signals(205cabe0,f84428,202be6d0,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x205caba0 sleepq_wait_sig(202be6d0,0) at sleepq_wait_sig+0xe/frame 0x205caba8 _cv_wait_sig(202be6d0,188030c0) at _cv_wait_sig+0x138/frame 0x205cabe0 svc_run_internal(205cacd4,fa7168,188030c0,205cace8,120e230,...) at svc_run_internal+0xb45/frame 0x205cacb0 svc_thread_start(188030c0,205cace8) at svc_thread_start+0xd/frame 0x205cacb8 fork_exit(120e230,188030c0,205cace8,3b,3b,...) at fork_exit+0x68/frame 0x205cacd4 fork_trampoline() at 0xffc0340e/frame 0x205cacd4 --- kthread start Tracing command nfsd pid 668 tid 100179 td 0x235e7880 sched_switch(235e7880,104) at sched_switch+0x4e9/frame 0x205cfb20 mi_switch(104) at mi_switch+0x1c4/frame 0x205cfb58 sleepq_switch(2d80,235e7880,0,0,1d79054,...) at sleepq_switch+0xf7/frame 0x205cfb78 sleepq_catch_signals(205cfbe0,f84428,202be650,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x205cfba0 sleepq_wait_sig(202be650,0) at sleepq_wait_sig+0xe/frame 0x205cfba8 _cv_wait_sig(202be650,188030c0) at _cv_wait_sig+0x138/frame 0x205cfbe0 svc_run_internal(205cfcd4,fa7168,188030c0,205cfce8,120e230,...) at svc_run_internal+0xb45/frame 0x205cfcb0 svc_thread_start(188030c0,205cfce8) at svc_thread_start+0xd/frame 0x205cfcb8 fork_exit(120e230,188030c0,205cfce8,3b,3b,...) at fork_exit+0x68/frame 0x205cfcd4 fork_trampoline() at 0xffc0340e/frame 0x205cfcd4 --- kthread start Tracing command nfsd pid 668 tid 100180 td 0x235e7500 sched_switch(235e7500,104) at sched_switch+0x4e9/frame 0x205d4b20 mi_switch(104) at mi_switch+0x1c4/frame 0x205d4b58 sleepq_switch(2380,235e7500,0,0,1d78654,...) at sleepq_switch+0xf7/frame 0x205d4b78 sleepq_catch_signals(205d4be0,f84428,235e5ed0,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x205d4ba0 sleepq_wait_sig(235e5ed0,0) at sleepq_wait_sig+0xe/frame 0x205d4ba8 _cv_wait_sig(235e5ed0,188030c0) at _cv_wait_sig+0x138/frame 0x205d4be0 svc_run_internal(205d4cd4,fa7168,188030c0,205d4ce8,120e230,...) at svc_run_internal+0xb45/frame 0x205d4cb0 svc_thread_start(188030c0,205d4ce8) at svc_thread_start+0xd/frame 0x205d4cb8 fork_exit(120e230,188030c0,205d4ce8,3b,3b,...) at fork_exit+0x68/frame 0x205d4cd4 fork_trampoline() at 0xffc0340e/frame 0x205d4cd4 --- kthread start Tracing command nfsd pid 668 tid 100181 td 0x235e7180 sched_switch(235e7180,104) at sched_switch+0x4e9/frame 0x205d9b20 mi_switch(104) at mi_switch+0x1c4/frame 0x205d9b58 sleepq_switch(d40,235e7180,0,0,1d77014,...) at sleepq_switch+0xf7/frame 0x205d9b78 sleepq_catch_signals(205d9be0,f84428,202be5d0,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x205d9ba0 sleepq_wait_sig(202be5d0,0) at sleepq_wait_sig+0xe/frame 0x205d9ba8 _cv_wait_sig(202be5d0,188030c0) at _cv_wait_sig+0x138/frame 0x205d9be0 svc_run_internal(205d9cd4,fa7168,188030c0,205d9ce8,120e230,...) at svc_run_internal+0xb45/frame 0x205d9cb0 svc_thread_start(188030c0,205d9ce8) at svc_thread_start+0xd/frame 0x205d9cb8 fork_exit(120e230,188030c0,205d9ce8,3b,3b,...) at fork_exit+0x68/frame 0x205d9cd4 fork_trampoline() at 0xffc0340e/frame 0x205d9cd4 --- kthread start Tracing command nfsd pid 668 tid 100182 td 0x235e6e00 sched_switch(235e6e00,104) at sched_switch+0x4e9/frame 0x205deb20 mi_switch(104) at mi_switch+0x1c4/frame 0x205deb58 sleepq_switch(2d40,235e6e00,0,0,1d79014,...) at sleepq_switch+0xf7/frame 0x205deb78 sleepq_catch_signals(205debe0,f84428,202be550,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x205deba0 sleepq_wait_sig(202be550,0) at sleepq_wait_sig+0xe/frame 0x205deba8 _cv_wait_sig(202be550,188030c0) at _cv_wait_sig+0x138/frame 0x205debe0 svc_run_internal(205decd4,fa7168,188030c0,205dece8,120e230,...) at svc_run_internal+0xb45/frame 0x205decb0 svc_thread_start(188030c0,205dece8) at svc_thread_start+0xd/frame 0x205decb8 fork_exit(120e230,188030c0,205dece8,3b,3b,...) at fork_exit+0x68/frame 0x205decd4 fork_trampoline() at 0xffc0340e/frame 0x205decd4 --- kthread start Tracing command nfsd pid 668 tid 100183 td 0x235e6a80 sched_switch(235e6a80,104) at sched_switch+0x4e9/frame 0x205e3b20 mi_switch(104) at mi_switch+0x1c4/frame 0x205e3b58 sleepq_switch(380,235e6a80,0,0,1d76654,...) at sleepq_switch+0xf7/frame 0x205e3b78 sleepq_catch_signals(205e3be0,f84428,235e5e50,0,188030c0,...) at sleepq_catch_signals+0x278/frame 0x205e3ba0 sleepq_wait_sig(235e5e50,0) at sleepq_wait_sig+0xe/frame 0x205e3ba8 _cv_wait_sig(235e5e50,188030c0) at _cv_wait_sig+0x138/frame 0x205e3be0 svc_run_internal(205e3cd4,fa7168,188030c0,205e3ce8,120e230,...) at svc_run_internal+0xb45/frame 0x205e3cb0 svc_thread_start(188030c0,205e3ce8) at svc_thread_start+0xd/frame 0x205e3cb8 fork_exit(120e230,188030c0,205e3ce8,3b,3b,...) at fork_exit+0x68/frame 0x205e3cd4 fork_trampoline() at 0xffc0340e/frame 0x205e3cd4 --- kthread start Tracing command nfsd pid 665 tid 100143 td 0x22bc0700 sched_switch(22bc0700,104) at sched_switch+0x4e9/frame 0x2050f970 mi_switch(104) at mi_switch+0x1c4/frame 0x2050f9a8 sleepq_switch(2640,22bc0700,0,0,1d78914,...) at sleepq_switch+0xf7/frame 0x2050f9c8 sleepq_catch_signals(2050fa30,f84428,1880bd24,0,1880bd10,...) at sleepq_catch_signals+0x278/frame 0x2050f9f0 sleepq_wait_sig(1880bd24,0) at sleepq_wait_sig+0xe/frame 0x2050f9f8 _cv_wait_sig(1880bd24,1880bd10) at _cv_wait_sig+0x138/frame 0x2050fa30 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x2050fa4c kern_select(22bc0700,7,ffbfeb44,0,0,0,20) at kern_select+0x8e5/frame 0x2050fbd4 sys_select(22bc0700,22bc0998,22bc0700,22c2b6dc,22bc0700,...) at sys_select+0x41/frame 0x2050fc08 syscall(2050fce8,3b,3b,3b,ffbfebc4,...) at syscall+0x1b8/frame 0x2050fcdc Xint0x80_syscall() at 0xffc033f9/frame 0x2050fcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe1ac --- Tracing command mountd pid 658 tid 100151 td 0x22c47500 sched_switch(22c47500,104) at sched_switch+0x4e9/frame 0x20543930 mi_switch(104) at mi_switch+0x1c4/frame 0x20543968 sleepq_switch(3600,1da5b6c,0,0,1d798d4,...) at sleepq_switch+0xf7/frame 0x20543988 sleepq_catch_signals(e4,22c47500,1a79480,22c47500,205439e4,...) at sleepq_catch_signals+0x278/frame 0x205439b0 sleepq_timedwait_sig(1880bc64,0) at sleepq_timedwait_sig+0x17/frame 0x205439e0 _cv_timedwait_sig_sbt(1880bc64,1880bc50,54954b08,1dba,4000000,0,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x20543a1c seltdwait(1dba,4000000,0) at seltdwait+0x5e/frame 0x20543a4c kern_select(22c47500,a,ffbfe824,0,0,20543bf8,20) at kern_select+0x8e5/frame 0x20543bd4 sys_select(22c47500,22c47798,22c47500,22c2f054,3faf3737,...) at sys_select+0x41/frame 0x20543c08 syscall(20543ce8,3b,3b,3b,ffbfe8a4,...) at syscall+0x1b8/frame 0x20543cdc Xint0x80_syscall() at 0xffc033f9/frame 0x20543cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe808 --- Tracing command rpcbind pid 634 tid 100129 td 0x22bc1c00 sched_switch(22bc1c00,104) at sched_switch+0x4e9/frame 0x204b9960 mi_switch(104) at mi_switch+0x1c4/frame 0x204b9998 sleepq_switch(1600,1da5384,0,0,1d778d4,...) at sleepq_switch+0xf7/frame 0x204b99b8 sleepq_catch_signals(e4,22bc1c00,1a79480,22bc1c00,204b9a14,...) at sleepq_catch_signals+0x278/frame 0x204b99e0 sleepq_timedwait_sig(1880bce4,0) at sleepq_timedwait_sig+0x17/frame 0x204b9a10 _cv_timedwait_sig_sbt(1880bce4,1880bcd0,47c8c7d7,1dbb,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x204b9a4c seltdwait(1dbb,e0000000,1) at seltdwait+0x5e/frame 0x204b9a7c kern_poll(22bc1c00,ffbfcbd4,8,204b9bf0,0) at kern_poll+0x4a4/frame 0x204b9bd4 sys_poll(22bc1c00,22bc1e98,22bc1c00,202cdd10,22bc1c00,...) at sys_poll+0x50/frame 0x204b9c08 syscall(204b9ce8,3b,3b,3b,40,...) at syscall+0x1b8/frame 0x204b9cdc Xint0x80_syscall() at 0xffc033f9/frame 0x204b9cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe808 --- Tracing command watchdogd pid 584 tid 100130 td 0x1fc9e700 sched_switch(1fc9e700,104) at sched_switch+0x4e9/frame 0x204be9d0 mi_switch(104) at mi_switch+0x1c4/frame 0x204bea04 sleepq_switch(25c0,14e9ff2,6c,0,1d78894,...) at sleepq_switch+0xf7/frame 0x204bea24 sleepq_catch_signals(204bea94,fefb0c,1834d344,6c,fd3fb1,...) at sleepq_catch_signals+0x278/frame 0x204bea4c sleepq_wait_sig(1834d344,6c) at sleepq_wait_sig+0xe/frame 0x204bea54 _sleep(1834d344,1834d3ec,36c,1506bc5,0,0,0,0,100) at _sleep+0x27c/frame 0x204bea94 kern_wait6(1fc9e700,0,1be1,0,204bebf0,30,0,0) at kern_wait6+0x3b8/frame 0x204beaf0 sys_wait4(1fc9e700,1fc9e998,1fc9e700,1834d344,b8505b8,...) at sys_wait4+0x73/frame 0x204bec08 syscall(204bece8,3b,3b,3b,1be1,...) at syscall+0x1b8/frame 0x204becdc Xint0x80_syscall() at 0xffc033f9/frame 0x204becdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe808 --- Tracing command syslogd pid 575 tid 100111 td 0x18d78180 sched_switch(18d78180,104) at sched_switch+0x4e9/frame 0x1fadf970 mi_switch(104) at mi_switch+0x1c4/frame 0x1fadf9a8 sleepq_switch(2680,18d78180,0,0,1d78954,...) at sleepq_switch+0xf7/frame 0x1fadf9c8 sleepq_catch_signals(1fadfa30,f84428,1880be24,0,1880be10,...) at sleepq_catch_signals+0x278/frame 0x1fadf9f0 sleepq_wait_sig(1880be24,0) at sleepq_wait_sig+0xe/frame 0x1fadf9f8 _cv_wait_sig(1880be24,1880be10) at _cv_wait_sig+0x138/frame 0x1fadfa30 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x1fadfa4c kern_select(18d78180,a,20685000,0,0,0,20) at kern_select+0x8e5/frame 0x1fadfbd4 sys_select(18d78180,18d78418,18d78180,18911344,3faf3737,...) at sys_select+0x41/frame 0x1fadfc08 syscall(1fadfce8,3b,3b,3b,20685000,...) at syscall+0x1b8/frame 0x1fadfcdc Xint0x80_syscall() at 0xffc033f9/frame 0x1fadfcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command nfscbd pid 533 tid 100131 td 0x1fc9e380 sched_switch(1fc9e380,104) at sched_switch+0x4e9/frame 0x204c3898 mi_switch(104) at mi_switch+0x1c4/frame 0x204c38d0 sleepq_switch(2f80,1da5f34,0,0,1d79254,...) at sleepq_switch+0xf7/frame 0x204c38f0 sleepq_catch_signals(e4,1fc9e380,1a79480,1fc9e380,204c394c,...) at sleepq_catch_signals+0x278/frame 0x204c3918 sleepq_timedwait_sig(202bee50,0) at sleepq_timedwait_sig+0x17/frame 0x204c3948 _cv_timedwait_sig_sbt(202bee50,183540c0,fffffa38,4,0,0,100) at _cv_timedwait_sig_sbt+0x15a/frame 0x204c3984 svc_run_internal(18d4a000,183540c0,183540d0,1835411c,0,...) at svc_run_internal+0xa1c/frame 0x204c3a68 svc_run(18354000) at svc_run+0x192/frame 0x204c3a98 nfscbd_nfsd(1fc9e380,204c3b50) at nfscbd_nfsd+0xd0/frame 0x204c3b28 nfssvc_nfscl(1fc9e380,1fc9e618) at nfssvc_nfscl+0x233/frame 0x204c3bf0 sys_nfssvc(1fc9e380,1fc9e618,1fc9e380,1834d000,1fc9e380,...) at sys_nfssvc+0x92/frame 0x204c3c08 syscall(204c3ce8,3b,3b,3b,ffbfee18,...) at syscall+0x1b8/frame 0x204c3cdc Xint0x80_syscall() at 0xffc033f9/frame 0x204c3cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command nfscbd pid 533 tid 100132 td 0x18d4a700 sched_switch(18d4a700,104) at sched_switch+0x4e9/frame 0x204c8b20 mi_switch(104) at mi_switch+0x1c4/frame 0x204c8b58 sleepq_switch(1a80,18d4a700,0,0,1d77d54,...) at sleepq_switch+0xf7/frame 0x204c8b78 sleepq_catch_signals(204c8be0,f84428,9a2bad0,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x204c8ba0 sleepq_wait_sig(9a2bad0,0) at sleepq_wait_sig+0xe/frame 0x204c8ba8 _cv_wait_sig(9a2bad0,183540c0) at _cv_wait_sig+0x138/frame 0x204c8be0 svc_run_internal(204c8cd4,fa7168,183540c0,204c8ce8,120e230,...) at svc_run_internal+0xb45/frame 0x204c8cb0 svc_thread_start(183540c0,204c8ce8) at svc_thread_start+0xd/frame 0x204c8cb8 fork_exit(120e230,183540c0,204c8ce8,3b,3b,...) at fork_exit+0x68/frame 0x204c8cd4 fork_trampoline() at 0xffc0340e/frame 0x204c8cd4 --- kthread start Tracing command nfscbd pid 533 tid 100133 td 0x18d4a380 sched_switch(18d4a380,104) at sched_switch+0x4e9/frame 0x204cdb20 mi_switch(104) at mi_switch+0x1c4/frame 0x204cdb58 sleepq_switch(3a80,18d4a380,0,0,1d79d54,...) at sleepq_switch+0xf7/frame 0x204cdb78 sleepq_catch_signals(204cdbe0,f84428,9a2ba50,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x204cdba0 sleepq_wait_sig(9a2ba50,0) at sleepq_wait_sig+0xe/frame 0x204cdba8 _cv_wait_sig(9a2ba50,183540c0) at _cv_wait_sig+0x138/frame 0x204cdbe0 svc_run_internal(204cdcd4,fa7168,183540c0,204cdce8,120e230,...) at svc_run_internal+0xb45/frame 0x204cdcb0 svc_thread_start(183540c0,204cdce8) at svc_thread_start+0xd/frame 0x204cdcb8 fork_exit(120e230,183540c0,204cdce8,3b,3b,...) at fork_exit+0x68/frame 0x204cdcd4 fork_trampoline() at 0xffc0340e/frame 0x204cdcd4 --- kthread start Tracing command nfscbd pid 533 tid 100134 td 0x18d4a000 sched_switch(18d4a000,104) at sched_switch+0x4e9/frame 0x204d2b20 mi_switch(104) at mi_switch+0x1c4/frame 0x204d2b58 sleepq_switch(1a40,18d4a000,0,0,1d77d14,...) at sleepq_switch+0xf7/frame 0x204d2b78 sleepq_catch_signals(204d2be0,f84428,9a2b9d0,0,146d361,...) at sleepq_catch_signals+0x278/frame 0x204d2ba0 sleepq_wait_sig(9a2b9d0,0) at sleepq_wait_sig+0xe/frame 0x204d2ba8 _cv_wait_sig(9a2b9d0,183540c0) at _cv_wait_sig+0x138/frame 0x204d2be0 svc_run_internal(204d2cd4,fa7168,183540c0,204d2ce8,120e230,...) at svc_run_internal+0xb45/frame 0x204d2cb0 svc_thread_start(183540c0,204d2ce8) at svc_thread_start+0xd/frame 0x204d2cb8 fork_exit(120e230,183540c0,204d2ce8,3b,3b,...) at fork_exit+0x68/frame 0x204d2cd4 fork_trampoline() at 0xffc0340e/frame 0x204d2cd4 --- kthread start Tracing command nfscbd pid 532 tid 100112 td 0x18d77e00 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(200046,1a63644,2885a10c,103cc0c,1a63644,...) at db_trace_self_wrapper+0x2a/frame 0x2885a0c8 kdb_backtrace(2885a1f0,13d5a2b,143607a,0,2885a228,...) at kdb_backtrace+0x2e/frame 0x2885a128 kdb_reenter(143607a,0,2885a228,2885a204,284de000,...) at kdb_reenter+0x29/frame 0x2885a130 trap(2885a1fc,8,28,28,ffffffff,...) at trap+0xeb/frame 0x2885a1f0 calltrap() at 0xffc0319f/frame 0x2885a1f0 --- trap 0xc, eip = 0x13b215a, esp = 0x2885a23c, ebp = 0x2885a248 --- db_trace_thread(18d77e00,ffffffff) at db_trace_thread+0x2a/frame 0x2885a248 _db_stack_trace_all(102809b,0,ffffffff,2885a2d4) at _db_stack_trace_all+0x93/frame 0x2885a2a8 db_command(1aec772) at db_command+0x26a/frame 0x2885a35c db_script_exec(bd2d8f,0,2885a404,2885a3e8) at db_script_exec+0x1a6/frame 0x2885a3bc db_command(2885a4c0,a06536,bd3b00,c960,284e9700,...) at db_command+0x26a/frame 0x2885a470 db_command_loop(bd3b00,c960,284e9700,2885a478,2885a4c0,...) at db_command_loop+0x3c/frame 0x2885a478 db_trap(3,0) at db_trap+0xf6/frame 0x2885a4c0 kdb_trap(3,0,2885a5c0) at kdb_trap+0x109/frame 0x2885a4ec trap(2885a5c0,8,28,28,16,...) at trap+0x382/frame 0x2885a5b4 calltrap() at 0xffc0319f/frame 0x2885a5b4 --- trap 0x3, eip = 0x102809b, esp = 0x2885a600, ebp = 0x2885a608 --- kdb_alt_break_internal(0) at kdb_alt_break_internal+0x13b/frame 0x2885a608 kdb_alt_break(2,1832be8c) at kdb_alt_break+0x10/frame 0x2885a614 uart_intr_rxready(1832be00,1dba,b85c5e4,880,b90ca00,...) at uart_intr_rxready+0xec/frame 0x2885a630 uart_intr(1832be00) at uart_intr+0x116/frame 0x2885a654 intr_event_handle(b90ca00,2885a6c8) at intr_event_handle+0x104/frame 0x2885a678 intr_execute_handlers(b9dc93c,2885a6c8) at intr_execute_handlers+0xbe/frame 0x2885a6a4 lapic_handle_intr(32,2885a6c8,8,28,28,...) at lapic_handle_intr+0x52/frame 0x2885a6b8 Xapic_isr1() at 0xffc0424a/frame 0x2885a6b8 --- interrupt, eip = 0xfc4e35, esp = 0x2885a708, ebp = 0x2885a708 --- __mtx_unlock_flags(1a92450,0,1489335,4eb) at __mtx_unlock_flags+0xb5/frame 0x2885a708 vnlru_free_locked(0,82000,1fc22000,18cd6080,2885a7ac,...) at vnlru_free_locked+0x279/frame 0x2885a768 vn_alloc_hard(2481f7b4,2885a7b4,1281eb8,74,0,...) at vn_alloc_hard+0xaf/frame 0x2885a780 getnewvnode(1413efe,18cd6080,1ab0f40,2885a7e4) at getnewvnode+0x261/frame 0x2885a7ac ffs_vgetf(18cd6080,5ea8,0,202000,2885a844,0) at ffs_vgetf+0xdf/frame 0x2885a7f4 ffs_vget(18cd6080,5ea8,0,202000,2885a844) at ffs_vget+0x19/frame 0x2885a814 ufs_lookup_ino(1fc19468,2885ab84,2885ab9c,0) at ufs_lookup_ino+0xba8/frame 0x2885a8cc ufs_lookup(2885a908) at ufs_lookup+0x16/frame 0x2885a8e4 VOP_CACHEDLOOKUP_APV(1ab0f40,2885a908) at VOP_CACHEDLOOKUP_APV+0x4d/frame 0x2885a8f8 vfs_cache_lookup(2885a960) at vfs_cache_lookup+0x98/frame 0x2885a92c VOP_LOOKUP_APV(1ab0f40,2885a960) at VOP_LOOKUP_APV+0x4d/frame 0x2885a940 lookup(2885ab48) at lookup+0x580/frame 0x2885a994 namei(2885ab48) at namei+0x6df/frame 0x2885aa20 vn_open_cred(2885ab48,2885ac00,0,0,235c4b00,2ae503c0) at vn_open_cred+0x2f8/frame 0x2885aae0 vn_open(2885ab48,2885ac00,0,2ae503c0) at vn_open+0x20/frame 0x2885ab00 kern_openat(284e9700,ffffff9c,22f04206,0,100001,0) at kern_openat+0x2fa/frame 0x2885abe8 sys_open(284e9700,284e9998,284e9700,284de000,284e9700,...) at sys_open+0x1c/frame 0x2885ac08 syscall(2885ace8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x2885acdc Xint0x80_syscall() at 0xffc033f9/frame 0x2885acdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command nfsuserd pid 449 tid 100128 td 0x1fc9ea80 sched_switch(1fc9ea80,104) at sched_switch+0x4e9/frame 0x204b4930 mi_switch(104) at mi_switch+0x1c4/frame 0x204b4968 sleepq_switch(640,1da5f8c,0,0,1d76914,...) at sleepq_switch+0xf7/frame 0x204b4988 sleepq_catch_signals(e4,1fc9ea80,1a79480,1fc9ea80,204b49e4,...) at sleepq_catch_signals+0x278/frame 0x204b49b0 sleepq_timedwait_sig(1880bda4,0) at sleepq_timedwait_sig+0x17/frame 0x204b49e0 _cv_timedwait_sig_sbt(1880bda4,1880bd90,2e2f25fc,1dc3,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x204b4a1c seltdwait(1dc3,e0000000,1) at seltdwait+0x5e/frame 0x204b4a4c kern_select(1fc9ea80,4,ffbfdba0,0,0,204b4bf8,20) at kern_select+0x8e5/frame 0x204b4bd4 sys_select(1fc9ea80,1fc9ed18,1fc9ea80,1834d688,1fc9ea80,...) at sys_select+0x41/frame 0x204b4c08 syscall(204b4ce8,3b,3b,3b,ffbfdba0,...) at syscall+0x1b8/frame 0x204b4cdc Xint0x80_syscall() at 0xffc033f9/frame 0x204b4cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d98, esp = 0xffc0bfe8, ebp = 0xffbfeccc --- Tracing command nfsuserd pid 448 tid 100127 td 0x1fc9ee00 sched_switch(1fc9ee00,104) at sched_switch+0x4e9/frame 0x204af930 mi_switch(104) at mi_switch+0x1c4/frame 0x204af968 sleepq_switch(2e00,1da6144,0,0,1d790d4,...) at sleepq_switch+0xf7/frame 0x204af988 sleepq_catch_signals(e4,1fc9ee00,1a79480,1fc9ee00,204af9e4,...) at sleepq_catch_signals+0x278/frame 0x204af9b0 sleepq_timedwait_sig(18d5dc64,0) at sleepq_timedwait_sig+0x17/frame 0x204af9e0 _cv_timedwait_sig_sbt(18d5dc64,18d5dc50,1518c201,1dc5,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x204afa1c seltdwait(1dc5,e0000000,1) at seltdwait+0x5e/frame 0x204afa4c kern_select(1fc9ee00,4,ffbfdba0,0,0,204afbf8,20) at kern_select+0x8e5/frame 0x204afbd4 sys_select(1fc9ee00,1fc9f098,1fc9ee00,1834d9cc,1fc9ee00,...) at sys_select+0x41/frame 0x204afc08 syscall(204afce8,3b,3b,3b,ffbfdba0,...) at syscall+0x1b8/frame 0x204afcdc Xint0x80_syscall() at 0xffc033f9/frame 0x204afcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe808 --- Tracing command nfsuserd pid 447 tid 100126 td 0x1fc9f180 sched_switch(1fc9f180,104) at sched_switch+0x4e9/frame 0x204aa930 mi_switch(104) at mi_switch+0x1c4/frame 0x204aa968 sleepq_switch(1640,1da64b4,0,0,1d77914,...) at sleepq_switch+0xf7/frame 0x204aa988 sleepq_catch_signals(e4,1fc9f180,1a79480,1fc9f180,204aa9e4,...) at sleepq_catch_signals+0x278/frame 0x204aa9b0 sleepq_timedwait_sig(1880bde4,0) at sleepq_timedwait_sig+0x17/frame 0x204aa9e0 _cv_timedwait_sig_sbt(1880bde4,1880bdd0,6fb881a8,1dc6,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x204aaa1c seltdwait(1dc6,e0000000,1) at seltdwait+0x5e/frame 0x204aaa4c kern_select(1fc9f180,4,ffbfdba0,0,0,204aabf8,20) at kern_select+0x8e5/frame 0x204aabd4 sys_select(1fc9f180,1fc9f418,1fc9f180,1834dd10,1fc9f180,...) at sys_select+0x41/frame 0x204aac08 syscall(204aace8,3b,3b,3b,ffbfdba0,...) at syscall+0x1b8/frame 0x204aacdc Xint0x80_syscall() at 0xffc033f9/frame 0x204aacdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01e18, esp = 0xffc0dfe8, ebp = 0xffbfe1ac --- Tracing command nfsuserd pid 446 tid 100113 td 0x1fc9f880 sched_switch(1fc9f880,104) at sched_switch+0x4e9/frame 0x1fae9930 mi_switch(104) at mi_switch+0x1c4/frame 0x1fae9968 sleepq_switch(1e00,1da61f4,0,0,1d780d4,...) at sleepq_switch+0xf7/frame 0x1fae9988 sleepq_catch_signals(e4,1fc9f880,1a79480,1fc9f880,1fae99e4,...) at sleepq_catch_signals+0x278/frame 0x1fae99b0 sleepq_timedwait_sig(18d5dca4,0) at sleepq_timedwait_sig+0x17/frame 0x1fae99e0 _cv_timedwait_sig_sbt(18d5dca4,18d5dc90,c7c898dc,1dc3,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x1fae9a1c seltdwait(1dc3,e0000000,1) at seltdwait+0x5e/frame 0x1fae9a4c kern_select(1fc9f880,4,ffbfdba0,0,0,1fae9bf8,20) at kern_select+0x8e5/frame 0x1fae9bd4 sys_select(1fc9f880,1fc9fb18,1fc9f880,1834e398,1fc9f880,...) at sys_select+0x41/frame 0x1fae9c08 syscall(1fae9ce8,3b,3b,3b,ffbfdba0,...) at syscall+0x1b8/frame 0x1fae9cdc Xint0x80_syscall() at 0xffc033f9/frame 0x1fae9cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe808 --- Tracing command nfsuserd pid 445 tid 100124 td 0x1fcae380 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(200046,1a63644,2885a10c,103cc0c,1a63644,...) at db_trace_self_wrapper+0x2a/frame 0x2885a0c8 kdb_backtrace(2885a1f0,13d5a2b,143607a,0,2885a228,...) at kdb_backtrace+0x2e/frame 0x2885a128 kdb_reenter(143607a,0,2885a228,2885a204,284de000,...) at kdb_reenter+0x29/frame 0x2885a130 trap(2885a1fc,8,28,28,ffffffff,...) at trap+0xeb/frame 0x2885a1f0 calltrap() at 0xffc0319f/frame 0x2885a1f0 --- trap 0xc, eip = 0x13b215a, esp = 0x2885a23c, ebp = 0x2885a248 --- db_trace_thread(1fcae380,ffffffff) at db_trace_thread+0x2a/frame 0x2885a248 _db_stack_trace_all(102809b,0,ffffffff,2885a2d4) at _db_stack_trace_all+0x93/frame 0x2885a2a8 db_command(1aec772) at db_command+0x26a/frame 0x2885a35c db_script_exec(bd2d8f,0,2885a404,2885a3e8) at db_script_exec+0x1a6/frame 0x2885a3bc db_command(2885a4c0,a06536,bd3b00,c960,284e9700,...) at db_command+0x26a/frame 0x2885a470 db_command_loop(bd3b00,c960,284e9700,2885a478,2885a4c0,...) at db_command_loop+0x3c/frame 0x2885a478 db_trap(3,0) at db_trap+0xf6/frame 0x2885a4c0 kdb_trap(3,0,2885a5c0) at kdb_trap+0x109/frame 0x2885a4ec trap(2885a5c0,8,28,28,16,...) at trap+0x382/frame 0x2885a5b4 calltrap() at 0xffc0319f/frame 0x2885a5b4 --- trap 0x3, eip = 0x102809b, esp = 0x2885a600, ebp = 0x2885a608 --- kdb_alt_break_internal(0) at kdb_alt_break_internal+0x13b/frame 0x2885a608 kdb_alt_break(2,1832be8c) at kdb_alt_break+0x10/frame 0x2885a614 uart_intr_rxready(1832be00,1dba,b85c5e4,880,b90ca00,...) at uart_intr_rxready+0xec/frame 0x2885a630 uart_intr(1832be00) at uart_intr+0x116/frame 0x2885a654 intr_event_handle(b90ca00,2885a6c8) at intr_event_handle+0x104/frame 0x2885a678 intr_execute_handlers(b9dc93c,2885a6c8) at intr_execute_handlers+0xbe/frame 0x2885a6a4 lapic_handle_intr(32,2885a6c8,8,28,28,...) at lapic_handle_intr+0x52/frame 0x2885a6b8 Xapic_isr1() at 0xffc0424a/frame 0x2885a6b8 --- interrupt, eip = 0xfc4e35, esp = 0x2885a708, ebp = 0x2885a708 --- __mtx_unlock_flags(1a92450,0,1489335,4eb) at __mtx_unlock_flags+0xb5/frame 0x2885a708 vnlru_free_locked(0,82000,1fc22000,18cd6080,2885a7ac,...) at vnlru_free_locked+0x279/frame 0x2885a768 vn_alloc_hard(2481f7b4,2885a7b4,1281eb8,74,0,...) at vn_alloc_hard+0xaf/frame 0x2885a780 getnewvnode(1413efe,18cd6080,1ab0f40,2885a7e4) at getnewvnode+0x261/frame 0x2885a7ac ffs_vgetf(18cd6080,5ea8,0,202000,2885a844,0) at ffs_vgetf+0xdf/frame 0x2885a7f4 ffs_vget(18cd6080,5ea8,0,202000,2885a844) at ffs_vget+0x19/frame 0x2885a814 ufs_lookup_ino(1fc19468,2885ab84,2885ab9c,0) at ufs_lookup_ino+0xba8/frame 0x2885a8cc ufs_lookup(2885a908) at ufs_lookup+0x16/frame 0x2885a8e4 VOP_CACHEDLOOKUP_APV(1ab0f40,2885a908) at VOP_CACHEDLOOKUP_APV+0x4d/frame 0x2885a8f8 vfs_cache_lookup(2885a960) at vfs_cache_lookup+0x98/frame 0x2885a92c VOP_LOOKUP_APV(1ab0f40,2885a960) at VOP_LOOKUP_APV+0x4d/frame 0x2885a940 lookup(2885ab48) at lookup+0x580/frame 0x2885a994 namei(2885ab48) at namei+0x6df/frame 0x2885aa20 vn_open_cred(2885ab48,2885ac00,0,0,235c4b00,2ae503c0) at vn_open_cred+0x2f8/frame 0x2885aae0 vn_open(2885ab48,2885ac00,0,2ae503c0) at vn_open+0x20/frame 0x2885ab00 kern_openat(284e9700,ffffff9c,22f04206,0,100001,0) at kern_openat+0x2fa/frame 0x2885abe8 sys_open(284e9700,284e9998,284e9700,284de000,284e9700,...) at sys_open+0x1c/frame 0x2885ac08 syscall(2885ace8,3b,3b,3b,0,...) at syscall+0x1b8/frame 0x2885acdc Xint0x80_syscall() at 0xffc033f9/frame 0x2885acdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d18, esp = 0xffc0afe8, ebp = 0xffbfe808 --- Tracing command devd pid 380 tid 100125 td 0x1fcae000 sched_switch(1fcae000,104) at sched_switch+0x4e9/frame 0x20451930 mi_switch(104) at mi_switch+0x1c4/frame 0x20451968 sleepq_switch(bc0,1da60ec,0,0,1d76e94,...) at sleepq_switch+0xf7/frame 0x20451988 sleepq_catch_signals(e4,1fcae000,1a79480,1fcae000,204519e4,...) at sleepq_catch_signals+0x278/frame 0x204519b0 sleepq_timedwait_sig(9ab0b24,0) at sleepq_timedwait_sig+0x17/frame 0x204519e0 _cv_timedwait_sig_sbt(9ab0b24,9ab0b10,fafc0306,1dd2,c0000000,3,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x20451a1c seltdwait(1dd2,c0000000,3) at seltdwait+0x5e/frame 0x20451a4c kern_select(1fcae000,6,ffbfcd3c,0,0,20451bf8,20) at kern_select+0x8e5/frame 0x20451bd4 sys_select(1fcae000,1fcae298,1fcae000,202ce054,1fcae000,...) at sys_select+0x41/frame 0x20451c08 syscall(20451ce8,3b,3b,3b,ffbfcdbc,...) at syscall+0x1b8/frame 0x20451cdc Xint0x80_syscall() at 0xffc033f9/frame 0x20451cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01c98, esp = 0xffc07fe8, ebp = 0xffbfe808 --- Tracing command syncer pid 24 tid 100104 td 0x18d78880 sched_switch(18d78880,104) at sched_switch+0x4e9/frame 0x1fa8cb98 mi_switch(104) at mi_switch+0x1c4/frame 0x1fa8cbd0 sleepq_switch(18d78880,1fa8cc18,fc47ce,1dd58a8,1dd58b8,...) at sleepq_switch+0xf7/frame 0x1fa8cbf0 sleepq_timedwait(1dd58b8,0) at sleepq_timedwait+0x3f/frame 0x1fa8cc14 _cv_timedwait_sbt(1dd58b8,1dd5898,fffffed8,0,0,0,100) at _cv_timedwait_sbt+0x157/frame 0x1fa8cc50 sched_sync(0,1fa8cce8) at sched_sync+0x662/frame 0x1fa8ccb8 fork_exit(10b43c0,0,1fa8cce8,0,0,...) at fork_exit+0x68/frame 0x1fa8ccd4 fork_trampoline() at 0xffc0340e/frame 0x1fa8ccd4 --- kthread start Tracing command vnlru pid 23 tid 100103 td 0x18d78c00 sched_switch(18d78c00,104) at sched_switch+0x4e9/frame 0x1fa87b70 mi_switch(104) at mi_switch+0x1c4/frame 0x1fa87ba8 sleepq_switch(18d78c00,0,18911d10,1fa87c2c,18d78c00,...) at sleepq_switch+0xf7/frame 0x1fa87bc8 sleepq_timedwait(18911d10,60,1fa87c20,fc54d5,1d76e04,...) at sleepq_timedwait+0x3f/frame 0x1fa87bec _sleep(18911d10,1a92440,260,1511a30,fffffed8,0,0,0,100) at _sleep+0x22a/frame 0x1fa87c2c vnlru_proc(0,1fa87ce8) at vnlru_proc+0xae1/frame 0x1fa87cb8 fork_exit(10b3430,0,1fa87ce8,0,0,...) at fork_exit+0x68/frame 0x1fa87cd4 fork_trampoline() at 0xffc0340e/frame 0x1fa87cd4 --- kthread start Tracing command bufdaemon pid 22 tid 100099 td 0x18913000 sched_switch(18913000,104) at sched_switch+0x4e9/frame 0xb5aebb8 mi_switch(104) at mi_switch+0x1c4/frame 0xb5aebf0 sleepq_switch(183,18913000,0,1dd5004,b5aec78,...) at sleepq_switch+0xf7/frame 0xb5aec10 sleepq_timedwait(1dd5004,54,3faf3737,1a86110,18913000,...) at sleepq_timedwait+0x3f/frame 0xb5aec38 _sleep(1dd5004,1a86100,54,14319b8,1999997c,0,0,0,100) at _sleep+0x22a/frame 0xb5aec78 buf_daemon(0,b5aece8) at buf_daemon+0xf6/frame 0xb5aecb8 fork_exit(108c120,0,b5aece8,0,0,...) at fork_exit+0x68/frame 0xb5aecd4 fork_trampoline() at 0xffc0340e/frame 0xb5aecd4 --- kthread start Tracing command bufdaemon pid 22 tid 100102 td 0x18d4b500 sched_switch(18d4b500,104) at sched_switch+0x4e9/frame 0xb5fdbb0 mi_switch(104) at mi_switch+0x1c4/frame 0xb5fdbe8 sleepq_switch(18d4b500,0,1a87400,b5fdc6c,18d4b500,...) at sleepq_switch+0xf7/frame 0xb5fdc08 sleepq_timedwait(1a87400,5c,0,3faf3737,108c400,...) at sleepq_timedwait+0x3f/frame 0xb5fdc2c _sleep(1a87400,1a87340,25c,1435413,fffffed8,0,0,0,100) at _sleep+0x22a/frame 0xb5fdc6c bufspace_daemon(1a86200,b5fdce8) at bufspace_daemon+0x2cb/frame 0xb5fdcb8 fork_exit(108c400,1a86200,b5fdce8,0,0,...) at fork_exit+0x68/frame 0xb5fdcd4 fork_trampoline() at 0xffc0340e/frame 0xb5fdcd4 --- kthread start Tracing command bufdaemon pid 22 tid 100116 td 0x1fcaf880 sched_switch(1fcaf880,104) at sched_switch+0x4e9/frame 0x1fb00bc0 mi_switch(104) at mi_switch+0x1c4/frame 0x1fb00bf8 sleepq_switch(1fcaf880,0,1fe39e84,1fb00c7c,1fcaf880,...) at sleepq_switch+0xf7/frame 0x1fb00c18 sleepq_timedwait(1fe39e84,54,1fb00c70,fe0120,1fe39e00,...) at sleepq_timedwait+0x3f/frame 0x1fb00c3c _sleep(1fe39e84,1fe39e00,54,1463c25,7fffff6c,0,0,0,100) at _sleep+0x22a/frame 0x1fb00c7c softdep_flush(1fe33040,1fb00ce8) at softdep_flush+0x158/frame 0x1fb00cb8 fork_exit(124a460,1fe33040,1fb00ce8,0,0,...) at fork_exit+0x68/frame 0x1fb00cd4 fork_trampoline() at 0xffc0340e/frame 0x1fb00cd4 --- kthread start Tracing command bufdaemon pid 22 tid 100117 td 0x1fcaf500 sched_switch(1fcaf500,104) at sched_switch+0x4e9/frame 0x1fb05bc0 mi_switch(104) at mi_switch+0x1c4/frame 0x1fb05bf8 sleepq_switch(1fcaf500,0,1fe39c84,1fb05c7c,1fcaf500,...) at sleepq_switch+0xf7/frame 0x1fb05c18 sleepq_timedwait(1fe39c84,54,1fb05c70,fe0120,1fe39c00,...) at sleepq_timedwait+0x3f/frame 0x1fb05c3c _sleep(1fe39c84,1fe39c00,54,1463c25,7fffff6c,0,0,0,100) at _sleep+0x22a/frame 0x1fb05c7c softdep_flush(1fe32580,1fb05ce8) at softdep_flush+0x158/frame 0x1fb05cb8 fork_exit(124a460,1fe32580,1fb05ce8,0,0,...) at fork_exit+0x68/frame 0x1fb05cd4 fork_trampoline() at 0xffc0340e/frame 0x1fb05cd4 --- kthread start Tracing command bufdaemon pid 22 tid 100118 td 0x1fcaf180 sched_switch(1fcaf180,104) at sched_switch+0x4e9/frame 0x1fb0abc0 mi_switch(104) at mi_switch+0x1c4/frame 0x1fb0abf8 sleepq_switch(1fcaf180,0,1fe39a84,1fb0ac7c,1fcaf180,...) at sleepq_switch+0xf7/frame 0x1fb0ac18 sleepq_timedwait(1fe39a84,54,1fb0ac70,fe0120,1fe39a00,...) at sleepq_timedwait+0x3f/frame 0x1fb0ac3c _sleep(1fe39a84,1fe39a00,54,1463c25,7fffff6c,0,0,0,100) at _sleep+0x22a/frame 0x1fb0ac7c softdep_flush(1fe31ac0,1fb0ace8) at softdep_flush+0x158/frame 0x1fb0acb8 fork_exit(124a460,1fe31ac0,1fb0ace8,0,0,...) at fork_exit+0x68/frame 0x1fb0acd4 fork_trampoline() at 0xffc0340e/frame 0x1fb0acd4 --- kthread start Tracing command bufdaemon pid 22 tid 100119 td 0x1fcaee00 sched_switch(1fcaee00,104) at sched_switch+0x4e9/frame 0x1fb0fbc0 mi_switch(104) at mi_switch+0x1c4/frame 0x1fb0fbf8 sleepq_switch(1fcaee00,0,1fe39884,1fb0fc7c,1fcaee00,...) at sleepq_switch+0xf7/frame 0x1fb0fc18 sleepq_timedwait(1fe39884,54,1fb0fc70,fe0120,1fe39800,...) at sleepq_timedwait+0x3f/frame 0x1fb0fc3c _sleep(1fe39884,1fe39800,54,1463c25,7fffff6c,0,0,0,100) at _sleep+0x22a/frame 0x1fb0fc7c softdep_flush(1fe31000,1fb0fce8) at softdep_flush+0x158/frame 0x1fb0fcb8 fork_exit(124a460,1fe31000,1fb0fce8,0,0,...) at fork_exit+0x68/frame 0x1fb0fcd4 fork_trampoline() at 0xffc0340e/frame 0x1fb0fcd4 --- kthread start Tracing command bufdaemon pid 22 tid 103721 td 0x2c256700 sched_switch(2c256700,103) at sched_switch+0x4e9/frame 0x324749e8 mi_switch(103,1d7ae60) at mi_switch+0x1c4/frame 0x32474a20 turnstile_wait(231b5500,2817a700,0) at turnstile_wait+0x2f1/frame 0x32474a3c __mtx_lock_sleep(1a92450,1fc9e000,0,1489335,4ee,...) at __mtx_lock_sleep+0x1b1/frame 0x32474a9c __mtx_lock_flags(1a92450,0,1489335,4ee) at __mtx_lock_flags+0xcc/frame 0x32474ac8 vnlru_free_locked(0,80200,2d2b1000,3d741000,32474b70,...) at vnlru_free_locked+0x9a/frame 0x32474b28 vn_alloc_hard(74,3adf0a6c,32474b78,1281eb8,74,...) at vn_alloc_hard+0xaf/frame 0x32474b40 getnewvnode(1413efe,3d741000,1ab0f40,32474ba8) at getnewvnode+0x261/frame 0x32474b70 ffs_vgetf(3d741000,2c3b6,0,80200,32474bf0,1) at ffs_vgetf+0xdf/frame 0x32474bb8 handle_workitem_remove(fc4dc7,3,2c256700,22cc4000,a,...) at handle_workitem_remove+0x49/frame 0x32474c20 process_worklist_item(200) at process_worklist_item+0x250/frame 0x32474c78 softdep_process_worklist(2c256700,2c256700,18912054,124a460,32474cd4,...) at softdep_process_worklist+0xb3/frame 0x32474ca0 softdep_flush(3d741000,32474ce8) at softdep_flush+0xce/frame 0x32474cb8 fork_exit(124a460,3d741000,32474ce8,0,0,...) at fork_exit+0x68/frame 0x32474cd4 fork_trampoline() at 0xffc0340e/frame 0x32474cd4 --- kthread start Tracing command vmdaemon pid 21 tid 100098 td 0x18913380 sched_switch(18913380,104) at sched_switch+0x4e9/frame 0xb5a9ba8 mi_switch(104) at mi_switch+0x1c4/frame 0xb5a9bdc sleepq_switch(1dec370,74) at sleepq_switch+0xf7/frame 0xb5a9bfc _sleep(1dec370,1dec35c,74,14319b1,0,0,0,0,100) at _sleep+0x2f3/frame 0xb5a9c3c vm_daemon(0,b5a9ce8) at vm_daemon+0x7f/frame 0xb5a9cb8 fork_exit(12bba00,0,b5a9ce8,0,0,...) at fork_exit+0x68/frame 0xb5a9cd4 fork_trampoline() at 0xffc0340e/frame 0xb5a9cd4 --- kthread start Tracing command pagedaemon pid 20 tid 100097 td 0x18913700 sched_switch(18913700,104) at sched_switch+0x4e9/frame 0xb5a4a58 mi_switch(104) at mi_switch+0x1c4/frame 0xb5a4a90 sleepq_switch(18913700,0,1deb3bc,b5a4b14,18913700,...) at sleepq_switch+0xf7/frame 0xb5a4ab0 sleepq_timedwait(1deb3bc,54,144e,3faf3737,0,...) at sleepq_timedwait+0x3f/frame 0xb5a4ad4 _sleep(1deb3bc,1deb0c0,254,14319b1,1999997c,0,0,0,100) at _sleep+0x22a/frame 0xb5a4b14 vm_pageout_worker(0,0,18913700,189126dc,18913700,...) at vm_pageout_worker+0x26a/frame 0xb5a4c94 vm_pageout(0,b5a4ce8) at vm_pageout+0x18a/frame 0xb5a4cb8 fork_exit(12b0640,0,b5a4ce8,0,0,...) at fork_exit+0x68/frame 0xb5a4cd4 fork_trampoline() at 0xffc0340e/frame 0xb5a4cd4 --- kthread start Tracing command pagedaemon pid 20 tid 100100 td 0x18d4bc00 sched_switch(18d4bc00,104) at sched_switch+0x4e9/frame 0xb5f3a80 mi_switch(104) at mi_switch+0x1c4/frame 0xb5f3ab4 sleepq_switch(1deb3c8,54) at sleepq_switch+0xf7/frame 0xb5f3ad4 _sleep(1deb3c8,1deb000,54,1526e3c,0,0,0,0,100) at _sleep+0x2f3/frame 0xb5f3b14 vm_pageout_laundry_worker(0,b5f3ce8) at vm_pageout_laundry_worker+0xca5/frame 0xb5f3cb8 fork_exit(12b18f0,0,b5f3ce8,0,0,...) at fork_exit+0x68/frame 0xb5f3cd4 fork_trampoline() at 0xffc0340e/frame 0xb5f3cd4 --- kthread start Tracing command pagedaemon pid 20 tid 100101 td 0x18d4b880 sched_switch(18d4b880,104) at sched_switch+0x4e9/frame 0xb5f8bc0 mi_switch(104) at mi_switch+0x1c4/frame 0xb5f8bf8 sleepq_switch(18d4b880,0,1deac0c,b5f8c44,10361bb,...) at sleepq_switch+0xf7/frame 0xb5f8c18 sleepq_timedwait(1283f70,54,1393d57,18d4b880,0,...) at sleepq_timedwait+0x3f/frame 0xb5f8c40 _sleep(1283f70,1deac0c,54,149455d,fffffed8,0,0,0,100) at _sleep+0x22a/frame 0xb5f8c80 uma_reclaim_worker(0,b5f8ce8) at uma_reclaim_worker+0xe4/frame 0xb5f8cb8 fork_exit(1284470,0,b5f8ce8,0,0,...) at fork_exit+0x68/frame 0xb5f8cd4 fork_trampoline() at 0xffc0340e/frame 0xb5f8cd4 --- kthread start Tracing command rand_harvestq pid 19 tid 100091 td 0x18914c00 sched_switch(18914c00,104) at sched_switch+0x4e9/frame 0xb585b80 mi_switch(104) at mi_switch+0x1c4/frame 0xb585bb8 sleepq_switch(18914c00,0,1bb247c,b585c3c,18914c00,...) at sleepq_switch+0xf7/frame 0xb585bd8 sleepq_timedwait(1bb247c,0,1bb7574,0,b585c38,...) at sleepq_timedwait+0x3f/frame 0xb585bfc _sleep(1bb247c,0,0,1435413,19999999,0,0,0,4) at _sleep+0x22a/frame 0xb585c3c random_kthread(0,b585ce8) at random_kthread+0x50/frame 0xb585cb8 fork_exit(d46450,0,b585ce8,0,0,...) at fork_exit+0x68/frame 0xb585cd4 fork_trampoline() at 0xffc0340e/frame 0xb585cd4 --- kthread start Tracing command soaiod4 pid 18 tid 100090 td 0xb9f2000 sched_switch(b9f2000,104) at sched_switch+0x4e9/frame 0xb57bbc0 mi_switch(104) at mi_switch+0x1c4/frame 0xb57bbf8 sleepq_switch(b9f2000,0,1dd4b68,b57bc7c,b9f2000,...) at sleepq_switch+0xf7/frame 0xb57bc18 sleepq_timedwait(1dd4b68,0,0,143b77d,14af924,...) at sleepq_timedwait+0x3f/frame 0xb57bc3c _sleep(1dd4b68,1dd4b48,0,1435413,ffffdd50,1d,0,0,100) at _sleep+0x22a/frame 0xb57bc7c soaio_kproc_loop(4,b57bce8) at soaio_kproc_loop+0x148/frame 0xb57bcb8 fork_exit(1053de0,4,b57bce8,0,0,...) at fork_exit+0x68/frame 0xb57bcd4 fork_trampoline() at 0xffc0340e/frame 0xb57bcd4 --- kthread start Tracing command soaiod3 pid 17 tid 100089 td 0xb9f2380 sched_switch(b9f2380,104) at sched_switch+0x4e9/frame 0xb576bc0 mi_switch(104) at mi_switch+0x1c4/frame 0xb576bf8 sleepq_switch(b9f2380,0,1dd4b68,b576c7c,b9f2380,...) at sleepq_switch+0xf7/frame 0xb576c18 sleepq_timedwait(1dd4b68,0,0,143b77d,14af924,...) at sleepq_timedwait+0x3f/frame 0xb576c3c _sleep(1dd4b68,1dd4b48,0,1435413,ffffdd50,1d,0,0,100) at _sleep+0x22a/frame 0xb576c7c soaio_kproc_loop(3,b576ce8) at soaio_kproc_loop+0x148/frame 0xb576cb8 fork_exit(1053de0,3,b576ce8,0,0,...) at fork_exit+0x68/frame 0xb576cd4 fork_trampoline() at 0xffc0340e/frame 0xb576cd4 --- kthread start Tracing command soaiod2 pid 16 tid 100088 td 0xb9f2700 sched_switch(b9f2700,104) at sched_switch+0x4e9/frame 0xb571bc0 mi_switch(104) at mi_switch+0x1c4/frame 0xb571bf8 sleepq_switch(b9f2700,0,1dd4b68,b571c7c,b9f2700,...) at sleepq_switch+0xf7/frame 0xb571c18 sleepq_timedwait(1dd4b68,0,0,143b77d,14af924,...) at sleepq_timedwait+0x3f/frame 0xb571c3c _sleep(1dd4b68,1dd4b48,0,1435413,ffffdd50,1d,0,0,100) at _sleep+0x22a/frame 0xb571c7c soaio_kproc_loop(2,b571ce8) at soaio_kproc_loop+0x148/frame 0xb571cb8 fork_exit(1053de0,2,b571ce8,0,0,...) at fork_exit+0x68/frame 0xb571cd4 fork_trampoline() at 0xffc0340e/frame 0xb571cd4 --- kthread start Tracing command soaiod1 pid 9 tid 100087 td 0xb9f2a80 sched_switch(b9f2a80,104) at sched_switch+0x4e9/frame 0xb56cbc0 mi_switch(104) at mi_switch+0x1c4/frame 0xb56cbf8 sleepq_switch(b9f2a80,0,1dd4b68,b56cc7c,b9f2a80,...) at sleepq_switch+0xf7/frame 0xb56cc18 sleepq_timedwait(1dd4b68,0,0,143b77d,14af924,...) at sleepq_timedwait+0x3f/frame 0xb56cc3c _sleep(1dd4b68,1dd4b48,0,1435413,ffffdd50,1d,0,0,100) at _sleep+0x22a/frame 0xb56cc7c soaio_kproc_loop(1,b56cce8) at soaio_kproc_loop+0x148/frame 0xb56ccb8 fork_exit(1053de0,1,b56cce8,0,0,...) at fork_exit+0x68/frame 0xb56ccd4 fork_trampoline() at 0xffc0340e/frame 0xb56ccd4 --- kthread start Tracing command fdc0 pid 8 tid 100084 td 0x18305700 sched_switch(18305700,104) at sched_switch+0x4e9/frame 0xb559ba8 mi_switch(104) at mi_switch+0x1c4/frame 0xb559be0 sleepq_switch(182b9c3c,5c) at sleepq_switch+0xf7/frame 0xb559c00 _sleep(182b9c3c,182b9cf8,5c,1435413,0,0,0,0,100) at _sleep+0x2f3/frame 0xb559c40 fdc_thread(182b9c00,b559ce8) at fdc_thread+0x361/frame 0xb559cb8 fork_exit(1338c70,182b9c00,b559ce8,0,0,...) at fork_exit+0x68/frame 0xb559cd4 fork_trampoline() at 0xffc0340e/frame 0xb559cd4 --- kthread start Tracing command usb pid 15 tid 100049 td 0x9b22380 sched_switch(9b22380,104) at sched_switch+0x4e9/frame 0xb4a1c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4a1c3c sleepq_switch(18024aac,0) at sleepq_switch+0xf7/frame 0xb4a1c5c _cv_wait(18024aac,18024cd8) at _cv_wait+0x135/frame 0xb4a1c94 usb_process(18024aa4,b4a1ce8) at usb_process+0x9d/frame 0xb4a1cb8 fork_exit(e38f70,18024aa4,b4a1ce8,0,0,...) at fork_exit+0x68/frame 0xb4a1cd4 fork_trampoline() at 0xffc0340e/frame 0xb4a1cd4 --- kthread start Tracing command usb pid 15 tid 100050 td 0x9b22000 sched_switch(9b22000,104) at sched_switch+0x4e9/frame 0xb4a6c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4a6c3c sleepq_switch(18024adc,0) at sleepq_switch+0xf7/frame 0xb4a6c5c _cv_wait(18024adc,18024cd8) at _cv_wait+0x135/frame 0xb4a6c94 usb_process(18024ad4,b4a6ce8) at usb_process+0x9d/frame 0xb4a6cb8 fork_exit(e38f70,18024ad4,b4a6ce8,0,0,...) at fork_exit+0x68/frame 0xb4a6cd4 fork_trampoline() at 0xffc0340e/frame 0xb4a6cd4 --- kthread start Tracing command usb pid 15 tid 100051 td 0x182c0c00 sched_switch(182c0c00,104) at sched_switch+0x4e9/frame 0xb4abc08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4abc3c sleepq_switch(18024b0c,0) at sleepq_switch+0xf7/frame 0xb4abc5c _cv_wait(18024b0c,18024cd8) at _cv_wait+0x135/frame 0xb4abc94 usb_process(18024b04,b4abce8) at usb_process+0x9d/frame 0xb4abcb8 fork_exit(e38f70,18024b04,b4abce8,0,0,...) at fork_exit+0x68/frame 0xb4abcd4 fork_trampoline() at 0xffc0340e/frame 0xb4abcd4 --- kthread start Tracing command usb pid 15 tid 100052 td 0x182c0880 sched_switch(182c0880,104) at sched_switch+0x4e9/frame 0xb4b0c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4b0c3c sleepq_switch(18024b3c,0) at sleepq_switch+0xf7/frame 0xb4b0c5c _cv_wait(18024b3c,18024cd8) at _cv_wait+0x135/frame 0xb4b0c94 usb_process(18024b34,b4b0ce8) at usb_process+0x9d/frame 0xb4b0cb8 fork_exit(e38f70,18024b34,b4b0ce8,0,0,...) at fork_exit+0x68/frame 0xb4b0cd4 fork_trampoline() at 0xffc0340e/frame 0xb4b0cd4 --- kthread start Tracing command usb pid 15 tid 100053 td 0x182c0500 sched_switch(182c0500,104) at sched_switch+0x4e9/frame 0xb4b5c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4b5c3c sleepq_switch(18024b6c,0) at sleepq_switch+0xf7/frame 0xb4b5c5c _cv_wait(18024b6c,18024cd8) at _cv_wait+0x135/frame 0xb4b5c94 usb_process(18024b64,b4b5ce8) at usb_process+0x9d/frame 0xb4b5cb8 fork_exit(e38f70,18024b64,b4b5ce8,0,0,...) at fork_exit+0x68/frame 0xb4b5cd4 fork_trampoline() at 0xffc0340e/frame 0xb4b5cd4 --- kthread start Tracing command usb pid 15 tid 100055 td 0x182bfe00 sched_switch(182bfe00,104) at sched_switch+0x4e9/frame 0xb4c0c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4c0c3c sleepq_switch(182c5aac,0) at sleepq_switch+0xf7/frame 0xb4c0c5c _cv_wait(182c5aac,182c5cd8) at _cv_wait+0x135/frame 0xb4c0c94 usb_process(182c5aa4,b4c0ce8) at usb_process+0x9d/frame 0xb4c0cb8 fork_exit(e38f70,182c5aa4,b4c0ce8,0,0,...) at fork_exit+0x68/frame 0xb4c0cd4 fork_trampoline() at 0xffc0340e/frame 0xb4c0cd4 --- kthread start Tracing command usb pid 15 tid 100056 td 0x182bfa80 sched_switch(182bfa80,104) at sched_switch+0x4e9/frame 0xb4c5c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4c5c3c sleepq_switch(182c5adc,0) at sleepq_switch+0xf7/frame 0xb4c5c5c _cv_wait(182c5adc,182c5cd8) at _cv_wait+0x135/frame 0xb4c5c94 usb_process(182c5ad4,b4c5ce8) at usb_process+0x9d/frame 0xb4c5cb8 fork_exit(e38f70,182c5ad4,b4c5ce8,0,0,...) at fork_exit+0x68/frame 0xb4c5cd4 fork_trampoline() at 0xffc0340e/frame 0xb4c5cd4 --- kthread start Tracing command usb pid 15 tid 100057 td 0x182bf700 sched_switch(182bf700,104) at sched_switch+0x4e9/frame 0xb4cac08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4cac3c sleepq_switch(182c5b0c,0) at sleepq_switch+0xf7/frame 0xb4cac5c _cv_wait(182c5b0c,182c5cd8) at _cv_wait+0x135/frame 0xb4cac94 usb_process(182c5b04,b4cace8) at usb_process+0x9d/frame 0xb4cacb8 fork_exit(e38f70,182c5b04,b4cace8,0,0,...) at fork_exit+0x68/frame 0xb4cacd4 fork_trampoline() at 0xffc0340e/frame 0xb4cacd4 --- kthread start Tracing command usb pid 15 tid 100058 td 0x182bf380 sched_switch(182bf380,104) at sched_switch+0x4e9/frame 0xb4cfc08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4cfc3c sleepq_switch(182c5b3c,0) at sleepq_switch+0xf7/frame 0xb4cfc5c _cv_wait(182c5b3c,182c5cd8) at _cv_wait+0x135/frame 0xb4cfc94 usb_process(182c5b34,b4cfce8) at usb_process+0x9d/frame 0xb4cfcb8 fork_exit(e38f70,182c5b34,b4cfce8,0,0,...) at fork_exit+0x68/frame 0xb4cfcd4 fork_trampoline() at 0xffc0340e/frame 0xb4cfcd4 --- kthread start Tracing command usb pid 15 tid 100059 td 0x182bf000 sched_switch(182bf000,104) at sched_switch+0x4e9/frame 0xb4d4c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4d4c3c sleepq_switch(182c5b6c,0) at sleepq_switch+0xf7/frame 0xb4d4c5c _cv_wait(182c5b6c,182c5cd8) at _cv_wait+0x135/frame 0xb4d4c94 usb_process(182c5b64,b4d4ce8) at usb_process+0x9d/frame 0xb4d4cb8 fork_exit(e38f70,182c5b64,b4d4ce8,0,0,...) at fork_exit+0x68/frame 0xb4d4cd4 fork_trampoline() at 0xffc0340e/frame 0xb4d4cd4 --- kthread start Tracing command usb pid 15 tid 100061 td 0x182d2880 sched_switch(182d2880,104) at sched_switch+0x4e9/frame 0xb4dfc08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4dfc3c sleepq_switch(182ccaac,0) at sleepq_switch+0xf7/frame 0xb4dfc5c _cv_wait(182ccaac,182cccd8) at _cv_wait+0x135/frame 0xb4dfc94 usb_process(182ccaa4,b4dfce8) at usb_process+0x9d/frame 0xb4dfcb8 fork_exit(e38f70,182ccaa4,b4dfce8,0,0,...) at fork_exit+0x68/frame 0xb4dfcd4 fork_trampoline() at 0xffc0340e/frame 0xb4dfcd4 --- kthread start Tracing command usb pid 15 tid 100062 td 0x182d2500 sched_switch(182d2500,104) at sched_switch+0x4e9/frame 0xb4e4c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4e4c3c sleepq_switch(182ccadc,0) at sleepq_switch+0xf7/frame 0xb4e4c5c _cv_wait(182ccadc,182cccd8) at _cv_wait+0x135/frame 0xb4e4c94 usb_process(182ccad4,b4e4ce8) at usb_process+0x9d/frame 0xb4e4cb8 fork_exit(e38f70,182ccad4,b4e4ce8,0,0,...) at fork_exit+0x68/frame 0xb4e4cd4 fork_trampoline() at 0xffc0340e/frame 0xb4e4cd4 --- kthread start Tracing command usb pid 15 tid 100063 td 0x182d2180 sched_switch(182d2180,104) at sched_switch+0x4e9/frame 0xb4e9c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4e9c3c sleepq_switch(182ccb0c,0) at sleepq_switch+0xf7/frame 0xb4e9c5c _cv_wait(182ccb0c,182cccd8) at _cv_wait+0x135/frame 0xb4e9c94 usb_process(182ccb04,b4e9ce8) at usb_process+0x9d/frame 0xb4e9cb8 fork_exit(e38f70,182ccb04,b4e9ce8,0,0,...) at fork_exit+0x68/frame 0xb4e9cd4 fork_trampoline() at 0xffc0340e/frame 0xb4e9cd4 --- kthread start Tracing command usb pid 15 tid 100064 td 0x182d1e00 sched_switch(182d1e00,104) at sched_switch+0x4e9/frame 0xb4eec08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4eec3c sleepq_switch(182ccb3c,0) at sleepq_switch+0xf7/frame 0xb4eec5c _cv_wait(182ccb3c,182cccd8) at _cv_wait+0x135/frame 0xb4eec94 usb_process(182ccb34,b4eece8) at usb_process+0x9d/frame 0xb4eecb8 fork_exit(e38f70,182ccb34,b4eece8,0,0,...) at fork_exit+0x68/frame 0xb4eecd4 fork_trampoline() at 0xffc0340e/frame 0xb4eecd4 --- kthread start Tracing command usb pid 15 tid 100065 td 0x182d1a80 sched_switch(182d1a80,104) at sched_switch+0x4e9/frame 0xb4f3c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4f3c3c sleepq_switch(182ccb6c,0) at sleepq_switch+0xf7/frame 0xb4f3c5c _cv_wait(182ccb6c,182cccd8) at _cv_wait+0x135/frame 0xb4f3c94 usb_process(182ccb64,b4f3ce8) at usb_process+0x9d/frame 0xb4f3cb8 fork_exit(e38f70,182ccb64,b4f3ce8,0,0,...) at fork_exit+0x68/frame 0xb4f3cd4 fork_trampoline() at 0xffc0340e/frame 0xb4f3cd4 --- kthread start Tracing command usb pid 15 tid 100066 td 0x182d1700 sched_switch(182d1700,104) at sched_switch+0x4e9/frame 0xb4f9c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4f9c3c sleepq_switch(182d6aac,0) at sleepq_switch+0xf7/frame 0xb4f9c5c _cv_wait(182d6aac,182d6cd8) at _cv_wait+0x135/frame 0xb4f9c94 usb_process(182d6aa4,b4f9ce8) at usb_process+0x9d/frame 0xb4f9cb8 fork_exit(e38f70,182d6aa4,b4f9ce8,0,0,...) at fork_exit+0x68/frame 0xb4f9cd4 fork_trampoline() at 0xffc0340e/frame 0xb4f9cd4 --- kthread start Tracing command usb pid 15 tid 100067 td 0x182d1380 sched_switch(182d1380,104) at sched_switch+0x4e9/frame 0xb4fec08 mi_switch(104) at mi_switch+0x1c4/frame 0xb4fec3c sleepq_switch(182d6adc,0) at sleepq_switch+0xf7/frame 0xb4fec5c _cv_wait(182d6adc,182d6cd8) at _cv_wait+0x135/frame 0xb4fec94 usb_process(182d6ad4,b4fece8) at usb_process+0x9d/frame 0xb4fecb8 fork_exit(e38f70,182d6ad4,b4fece8,0,0,...) at fork_exit+0x68/frame 0xb4fecd4 fork_trampoline() at 0xffc0340e/frame 0xb4fecd4 --- kthread start Tracing command usb pid 15 tid 100068 td 0x182d1000 sched_switch(182d1000,104) at sched_switch+0x4e9/frame 0xb503c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb503c3c sleepq_switch(182d6b0c,0) at sleepq_switch+0xf7/frame 0xb503c5c _cv_wait(182d6b0c,182d6cd8) at _cv_wait+0x135/frame 0xb503c94 usb_process(182d6b04,b503ce8) at usb_process+0x9d/frame 0xb503cb8 fork_exit(e38f70,182d6b04,b503ce8,0,0,...) at fork_exit+0x68/frame 0xb503cd4 fork_trampoline() at 0xffc0340e/frame 0xb503cd4 --- kthread start Tracing command usb pid 15 tid 100069 td 0x182dec00 sched_switch(182dec00,104) at sched_switch+0x4e9/frame 0xb508c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb508c3c sleepq_switch(182d6b3c,0) at sleepq_switch+0xf7/frame 0xb508c5c _cv_wait(182d6b3c,182d6cd8) at _cv_wait+0x135/frame 0xb508c94 usb_process(182d6b34,b508ce8) at usb_process+0x9d/frame 0xb508cb8 fork_exit(e38f70,182d6b34,b508ce8,0,0,...) at fork_exit+0x68/frame 0xb508cd4 fork_trampoline() at 0xffc0340e/frame 0xb508cd4 --- kthread start Tracing command usb pid 15 tid 100070 td 0x182de880 sched_switch(182de880,104) at sched_switch+0x4e9/frame 0xb50dc08 mi_switch(104) at mi_switch+0x1c4/frame 0xb50dc3c sleepq_switch(182d6b6c,0) at sleepq_switch+0xf7/frame 0xb50dc5c _cv_wait(182d6b6c,182d6cd8) at _cv_wait+0x135/frame 0xb50dc94 usb_process(182d6b64,b50dce8) at usb_process+0x9d/frame 0xb50dcb8 fork_exit(e38f70,182d6b64,b50dce8,0,0,...) at fork_exit+0x68/frame 0xb50dcd4 fork_trampoline() at 0xffc0340e/frame 0xb50dcd4 --- kthread start Tracing command usb pid 15 tid 100071 td 0x182de500 sched_switch(182de500,104) at sched_switch+0x4e9/frame 0xb513c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb513c3c sleepq_switch(182e4aac,0) at sleepq_switch+0xf7/frame 0xb513c5c _cv_wait(182e4aac,182e4cd8) at _cv_wait+0x135/frame 0xb513c94 usb_process(182e4aa4,b513ce8) at usb_process+0x9d/frame 0xb513cb8 fork_exit(e38f70,182e4aa4,b513ce8,0,0,...) at fork_exit+0x68/frame 0xb513cd4 fork_trampoline() at 0xffc0340e/frame 0xb513cd4 --- kthread start Tracing command usb pid 15 tid 100072 td 0x182de180 sched_switch(182de180,104) at sched_switch+0x4e9/frame 0xb518c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb518c3c sleepq_switch(182e4adc,0) at sleepq_switch+0xf7/frame 0xb518c5c _cv_wait(182e4adc,182e4cd8) at _cv_wait+0x135/frame 0xb518c94 usb_process(182e4ad4,b518ce8) at usb_process+0x9d/frame 0xb518cb8 fork_exit(e38f70,182e4ad4,b518ce8,0,0,...) at fork_exit+0x68/frame 0xb518cd4 fork_trampoline() at 0xffc0340e/frame 0xb518cd4 --- kthread start Tracing command usb pid 15 tid 100073 td 0x182dde00 sched_switch(182dde00,104) at sched_switch+0x4e9/frame 0xb51dc08 mi_switch(104) at mi_switch+0x1c4/frame 0xb51dc3c sleepq_switch(182e4b0c,0) at sleepq_switch+0xf7/frame 0xb51dc5c _cv_wait(182e4b0c,182e4cd8) at _cv_wait+0x135/frame 0xb51dc94 usb_process(182e4b04,b51dce8) at usb_process+0x9d/frame 0xb51dcb8 fork_exit(e38f70,182e4b04,b51dce8,0,0,...) at fork_exit+0x68/frame 0xb51dcd4 fork_trampoline() at 0xffc0340e/frame 0xb51dcd4 --- kthread start Tracing command usb pid 15 tid 100074 td 0x182dda80 sched_switch(182dda80,104) at sched_switch+0x4e9/frame 0xb522c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb522c3c sleepq_switch(182e4b3c,0) at sleepq_switch+0xf7/frame 0xb522c5c _cv_wait(182e4b3c,182e4cd8) at _cv_wait+0x135/frame 0xb522c94 usb_process(182e4b34,b522ce8) at usb_process+0x9d/frame 0xb522cb8 fork_exit(e38f70,182e4b34,b522ce8,0,0,...) at fork_exit+0x68/frame 0xb522cd4 fork_trampoline() at 0xffc0340e/frame 0xb522cd4 --- kthread start Tracing command usb pid 15 tid 100075 td 0x182dd700 sched_switch(182dd700,104) at sched_switch+0x4e9/frame 0xb527c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb527c3c sleepq_switch(182e4b6c,0) at sleepq_switch+0xf7/frame 0xb527c5c _cv_wait(182e4b6c,182e4cd8) at _cv_wait+0x135/frame 0xb527c94 usb_process(182e4b64,b527ce8) at usb_process+0x9d/frame 0xb527cb8 fork_exit(e38f70,182e4b64,b527ce8,0,0,...) at fork_exit+0x68/frame 0xb527cd4 fork_trampoline() at 0xffc0340e/frame 0xb527cd4 --- kthread start Tracing command usb pid 15 tid 100077 td 0x182dd000 sched_switch(182dd000,104) at sched_switch+0x4e9/frame 0xb532c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb532c3c sleepq_switch(1801fc84,0) at sleepq_switch+0xf7/frame 0xb532c5c _cv_wait(1801fc84,1801feb0) at _cv_wait+0x135/frame 0xb532c94 usb_process(1801fc7c,b532ce8) at usb_process+0x9d/frame 0xb532cb8 fork_exit(e38f70,1801fc7c,b532ce8,0,0,...) at fork_exit+0x68/frame 0xb532cd4 fork_trampoline() at 0xffc0340e/frame 0xb532cd4 --- kthread start Tracing command usb pid 15 tid 100078 td 0x18306c00 sched_switch(18306c00,104) at sched_switch+0x4e9/frame 0xb537c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb537c3c sleepq_switch(1801fcb4,0) at sleepq_switch+0xf7/frame 0xb537c5c _cv_wait(1801fcb4,1801feb0) at _cv_wait+0x135/frame 0xb537c94 usb_process(1801fcac,b537ce8) at usb_process+0x9d/frame 0xb537cb8 fork_exit(e38f70,1801fcac,b537ce8,0,0,...) at fork_exit+0x68/frame 0xb537cd4 fork_trampoline() at 0xffc0340e/frame 0xb537cd4 --- kthread start Tracing command usb pid 15 tid 100079 td 0x18306880 sched_switch(18306880,104) at sched_switch+0x4e9/frame 0xb53cc08 mi_switch(104) at mi_switch+0x1c4/frame 0xb53cc3c sleepq_switch(1801fce4,0) at sleepq_switch+0xf7/frame 0xb53cc5c _cv_wait(1801fce4,1801feb0) at _cv_wait+0x135/frame 0xb53cc94 usb_process(1801fcdc,b53cce8) at usb_process+0x9d/frame 0xb53ccb8 fork_exit(e38f70,1801fcdc,b53cce8,0,0,...) at fork_exit+0x68/frame 0xb53ccd4 fork_trampoline() at 0xffc0340e/frame 0xb53ccd4 --- kthread start Tracing command usb pid 15 tid 100080 td 0x18306500 sched_switch(18306500,104) at sched_switch+0x4e9/frame 0xb541c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb541c3c sleepq_switch(1801fd14,0) at sleepq_switch+0xf7/frame 0xb541c5c _cv_wait(1801fd14,1801feb0) at _cv_wait+0x135/frame 0xb541c94 usb_process(1801fd0c,b541ce8) at usb_process+0x9d/frame 0xb541cb8 fork_exit(e38f70,1801fd0c,b541ce8,0,0,...) at fork_exit+0x68/frame 0xb541cd4 fork_trampoline() at 0xffc0340e/frame 0xb541cd4 --- kthread start Tracing command usb pid 15 tid 100081 td 0x18306180 sched_switch(18306180,104) at sched_switch+0x4e9/frame 0xb546c08 mi_switch(104) at mi_switch+0x1c4/frame 0xb546c3c sleepq_switch(1801fd44,0) at sleepq_switch+0xf7/frame 0xb546c5c _cv_wait(1801fd44,1801feb0) at _cv_wait+0x135/frame 0xb546c94 usb_process(1801fd3c,b546ce8) at usb_process+0x9d/frame 0xb546cb8 fork_exit(e38f70,1801fd3c,b546ce8,0,0,...) at fork_exit+0x68/frame 0xb546cd4 fork_trampoline() at 0xffc0340e/frame 0xb546cd4 --- kthread start Tracing command cam pid 7 tid 100044 td 0x9b23500 sched_switch(9b23500,104) at sched_switch+0x4e9/frame 0x12bffbd8 mi_switch(104) at mi_switch+0x1c4/frame 0x12bffc0c sleepq_switch(1ae97c0,5c) at sleepq_switch+0xf7/frame 0x12bffc2c _sleep(1ae97c0,1ae9780,5c,1435413,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bffc6c xpt_done_td(1ae9780,12bffce8) at xpt_done_td+0x72/frame 0x12bffcb8 fork_exit(908180,1ae9780,12bffce8,0,0,...) at fork_exit+0x68/frame 0x12bffcd4 fork_trampoline() at 0xffc0340e/frame 0x12bffcd4 --- kthread start Tracing command cam pid 7 tid 100096 td 0x18913a80 sched_switch(18913a80,104) at sched_switch+0x4e9/frame 0xb59ebe8 mi_switch(104) at mi_switch+0x1c4/frame 0xb59ec20 sleepq_switch(1ae96ec,5c) at sleepq_switch+0xf7/frame 0xb59ec40 _sleep(1ae96ec,1ae975c,5c,1435413,0,0,0,0,100) at _sleep+0x2f3/frame 0xb59ec80 xpt_scanner_thread(0,b59ece8) at xpt_scanner_thread+0x56/frame 0xb59ecb8 fork_exit(908420,0,b59ece8,0,0,...) at fork_exit+0x68/frame 0xb59ecd4 fork_trampoline() at 0xffc0340e/frame 0xb59ecd4 --- kthread start Tracing command crypto returns 3 pid 6 tid 100043 td 0x9b23880 sched_switch(9b23880,104) at sched_switch+0x4e9/frame 0x12bfabc0 mi_switch(104) at mi_switch+0x1c4/frame 0x12bfabf4 sleepq_switch(b9ff6c4,6c) at sleepq_switch+0xf7/frame 0x12bfac14 _sleep(b9ff6c4,b9ff6a8,6c,151c689,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bfac54 crypto_ret_proc(b9ff6a8,12bface8) at crypto_ret_proc+0x1b9/frame 0x12bfacb8 fork_exit(11f98e0,b9ff6a8,12bface8,0,0,...) at fork_exit+0x68/frame 0x12bfacd4 fork_trampoline() at 0xffc0340e/frame 0x12bfacd4 --- kthread start Tracing command crypto returns 2 pid 5 tid 100042 td 0x9b23c00 sched_switch(9b23c00,104) at sched_switch+0x4e9/frame 0x12bf5bc0 mi_switch(104) at mi_switch+0x1c4/frame 0x12bf5bf4 sleepq_switch(b9ff68c,6c) at sleepq_switch+0xf7/frame 0x12bf5c14 _sleep(b9ff68c,b9ff670,6c,151c689,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bf5c54 crypto_ret_proc(b9ff670,12bf5ce8) at crypto_ret_proc+0x1b9/frame 0x12bf5cb8 fork_exit(11f98e0,b9ff670,12bf5ce8,0,0,...) at fork_exit+0x68/frame 0x12bf5cd4 fork_trampoline() at 0xffc0340e/frame 0x12bf5cd4 --- kthread start Tracing command crypto returns 1 pid 4 tid 100041 td 0x9adb000 sched_switch(9adb000,104) at sched_switch+0x4e9/frame 0x12bf0bc0 mi_switch(104) at mi_switch+0x1c4/frame 0x12bf0bf4 sleepq_switch(b9ff654,6c) at sleepq_switch+0xf7/frame 0x12bf0c14 _sleep(b9ff654,b9ff638,6c,151c689,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bf0c54 crypto_ret_proc(b9ff638,12bf0ce8) at crypto_ret_proc+0x1b9/frame 0x12bf0cb8 fork_exit(11f98e0,b9ff638,12bf0ce8,0,0,...) at fork_exit+0x68/frame 0x12bf0cd4 fork_trampoline() at 0xffc0340e/frame 0x12bf0cd4 --- kthread start Tracing command crypto returns 0 pid 3 tid 100040 td 0x9adb380 sched_switch(9adb380,104) at sched_switch+0x4e9/frame 0x12bebbc0 mi_switch(104) at mi_switch+0x1c4/frame 0x12bebbf4 sleepq_switch(b9ff61c,6c) at sleepq_switch+0xf7/frame 0x12bebc14 _sleep(b9ff61c,b9ff600,6c,151c689,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bebc54 crypto_ret_proc(b9ff600,12bebce8) at crypto_ret_proc+0x1b9/frame 0x12bebcb8 fork_exit(11f98e0,b9ff600,12bebce8,0,0,...) at fork_exit+0x68/frame 0x12bebcd4 fork_trampoline() at 0xffc0340e/frame 0x12bebcd4 --- kthread start Tracing command crypto pid 2 tid 100039 td 0x9adb700 sched_switch(9adb700,104) at sched_switch+0x4e9/frame 0x12be6bd8 mi_switch(104) at mi_switch+0x1c4/frame 0x12be6c0c sleepq_switch(1de9b20,6c) at sleepq_switch+0xf7/frame 0x12be6c2c _sleep(1de9b20,1de9b08,6c,14ce9af,0,0,0,0,100) at _sleep+0x2f3/frame 0x12be6c6c crypto_proc(0,12be6ce8) at crypto_proc+0x29f/frame 0x12be6cb8 fork_exit(11f9380,0,12be6ce8,0,0,...) at fork_exit+0x68/frame 0x12be6cd4 fork_trampoline() at 0xffc0340e/frame 0x12be6cd4 --- kthread start Tracing command sequencer 00 pid 14 tid 100034 td 0x9adc880 sched_switch(9adc880,104) at sched_switch+0x4e9/frame 0x12bcdbe0 mi_switch(104) at mi_switch+0x1c4/frame 0x12bcdc18 sleepq_switch(9ae804c,0) at sleepq_switch+0xf7/frame 0x12bcdc38 _cv_wait(9ae804c,9ae8004) at _cv_wait+0x135/frame 0x12bcdc70 seq_eventthread(9ae8000,12bcdce8) at seq_eventthread+0x7a/frame 0x12bcdcb8 fork_exit(dc6e80,9ae8000,12bcdce8,0,0,...) at fork_exit+0x68/frame 0x12bcdcd4 fork_trampoline() at 0xffc0340e/frame 0x12bcdcd4 --- kthread start Tracing command geom pid 13 tid 100031 td 0x9aa5380 sched_switch(9aa5380,104) at sched_switch+0x4e9/frame 0x12bbebe8 mi_switch(104) at mi_switch+0x1c4/frame 0x12bbec20 sleepq_switch(1d51388,5c) at sleepq_switch+0xf7/frame 0x12bbec40 _sleep(1d51388,1d512f4,25c,1435413,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bbec80 g_run_events(0,12bbece8) at g_run_events+0x4d/frame 0x12bbecb8 fork_exit(f39ef0,0,12bbece8,0,0,...) at fork_exit+0x68/frame 0x12bbecd4 fork_trampoline() at 0xffc0340e/frame 0x12bbecd4 --- kthread start Tracing command geom pid 13 tid 100032 td 0x9aa5000 sched_switch(9aa5000,104) at sched_switch+0x4e9/frame 0x12bc3be0 mi_switch(104) at mi_switch+0x1c4/frame 0x12bc3c18 sleepq_switch(1d5138c,5c) at sleepq_switch+0xf7/frame 0x12bc3c38 _sleep(1d5138c,1d51338,25c,1435413,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bc3c78 g_io_schedule_up(9aa5000) at g_io_schedule_up+0x36/frame 0x12bc3cac g_up_procbody(0,12bc3ce8) at g_up_procbody+0x5b/frame 0x12bc3cb8 fork_exit(f39f40,0,12bc3ce8,0,0,...) at fork_exit+0x68/frame 0x12bc3cd4 fork_trampoline() at 0xffc0340e/frame 0x12bc3cd4 --- kthread start Tracing command geom pid 13 tid 100033 td 0x9adcc00 sched_switch(9adcc00,104) at sched_switch+0x4e9/frame 0x12bc8be0 mi_switch(104) at mi_switch+0x1c4/frame 0x12bc8c14 sleepq_switch(1d51390,5c) at sleepq_switch+0xf7/frame 0x12bc8c34 _sleep(1d51390,1d51318,25c,1435413,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bc8c74 g_io_schedule_down(9adcc00) at g_io_schedule_down+0x36/frame 0x12bc8cac g_down_procbody(0,12bc8ce8) at g_down_procbody+0x5b/frame 0x12bc8cb8 fork_exit(f39fa0,0,12bc8ce8,0,0,...) at fork_exit+0x68/frame 0x12bc8cd4 fork_trampoline() at 0xffc0340e/frame 0x12bc8cd4 --- kthread start Tracing command intr pid 12 tid 100016 td 0xb9ee700 sched_switch(b9ee700,109) at sched_switch+0x4e9/frame 0x12b63c30 mi_switch(109) at mi_switch+0x1c4/frame 0x12b63c64 ithread_loop(b8de5a0,12b63ce8) at ithread_loop+0x85/frame 0x12b63cb8 fork_exit(fa9c50,b8de5a0,12b63ce8,0,0,...) at fork_exit+0x68/frame 0x12b63cd4 fork_trampoline() at 0xffc0340e/frame 0x12b63cd4 --- kthread start Tracing command intr pid 12 tid 100019 td 0xb9f3c00 exit_tag() at 0xffc03404 Tracing command intr pid 12 tid 100022 td 0xb9f3180 sched_switch(b9f3180,109) at sched_switch+0x4e9/frame 0x12b81c30 mi_switch(109) at mi_switch+0x1c4/frame 0x12b81c64 ithread_loop(b8de580,12b81ce8) at ithread_loop+0x85/frame 0x12b81cb8 fork_exit(fa9c50,b8de580,12b81ce8,0,0,...) at fork_exit+0x68/frame 0x12b81cd4 fork_trampoline() at 0xffc0340e/frame 0x12b81cd4 --- kthread start Tracing command intr pid 12 tid 100024 td 0x9aa6c00 sched_switch(9aa6c00,109) at sched_switch+0x4e9/frame 0x12b8fc30 mi_switch(109) at mi_switch+0x1c4/frame 0x12b8fc64 ithread_loop(9aa4fa0,12b8fce8) at ithread_loop+0x85/frame 0x12b8fcb8 fork_exit(fa9c50,9aa4fa0,12b8fce8,0,0,...) at fork_exit+0x68/frame 0x12b8fcd4 fork_trampoline() at 0xffc0340e/frame 0x12b8fcd4 --- kthread start Tracing command intr pid 12 tid 100025 td 0x9aa6880 exit_tag() at 0xffc03404 Tracing command intr pid 12 tid 100026 td 0x9aa6500 exit_tag() at 0xffc03404 Tracing command intr pid 12 tid 100027 td 0x9aa6180 exit_tag() at 0xffc03404 Tracing command intr pid 12 tid 100028 td 0x9aa5e00 sched_switch(9aa5e00,109) at sched_switch+0x4e9/frame 0x12ba3c30 mi_switch(109) at mi_switch+0x1c4/frame 0x12ba3c64 ithread_loop(9aa4f60,12ba3ce8) at ithread_loop+0x85/frame 0x12ba3cb8 fork_exit(fa9c50,9aa4f60,12ba3ce8,0,0,...) at fork_exit+0x68/frame 0x12ba3cd4 fork_trampoline() at 0xffc0340e/frame 0x12ba3cd4 --- kthread start Tracing command intr pid 12 tid 100029 td 0x9aa5a80 exit_tag() at 0xffc03404 Tracing command intr pid 12 tid 100045 td 0x9b23180 sched_switch(9b23180,109) at sched_switch+0x4e9/frame 0xb48ac30 mi_switch(109) at mi_switch+0x1c4/frame 0xb48ac64 ithread_loop(9aa4eb0,b48ace8) at ithread_loop+0x85/frame 0xb48acb8 fork_exit(fa9c50,9aa4eb0,b48ace8,0,0,...) at fork_exit+0x68/frame 0xb48acd4 fork_trampoline() at 0xffc0340e/frame 0xb48acd4 --- kthread start Tracing command intr pid 12 tid 100046 td 0x9b22e00 sched_switch(9b22e00,109) at sched_switch+0x4e9/frame 0xb490c30 mi_switch(109) at mi_switch+0x1c4/frame 0xb490c64 ithread_loop(9aa4e90,b490ce8) at ithread_loop+0x85/frame 0xb490cb8 fork_exit(fa9c50,9aa4e90,b490ce8,0,0,...) at fork_exit+0x68/frame 0xb490cd4 fork_trampoline() at 0xffc0340e/frame 0xb490cd4 --- kthread start Tracing command intr pid 12 tid 100047 td 0x9b22a80 sched_switch(9b22a80,109) at sched_switch+0x4e9/frame 0xb496c30 mi_switch(109) at mi_switch+0x1c4/frame 0xb496c64 ithread_loop(9aa4e80,b496ce8) at ithread_loop+0x85/frame 0xb496cb8 fork_exit(fa9c50,9aa4e80,b496ce8,0,0,...) at fork_exit+0x68/frame 0xb496cd4 fork_trampoline() at 0xffc0340e/frame 0xb496cd4 --- kthread start Tracing command intr pid 12 tid 100048 td 0x9b22700 sched_switch(9b22700,109) at sched_switch+0x4e9/frame 0xb49cc30 mi_switch(109) at mi_switch+0x1c4/frame 0xb49cc64 ithread_loop(9aa4e70,b49cce8) at ithread_loop+0x85/frame 0xb49ccb8 fork_exit(fa9c50,9aa4e70,b49cce8,0,0,...) at fork_exit+0x68/frame 0xb49ccd4 fork_trampoline() at 0xffc0340e/frame 0xb49ccd4 --- kthread start Tracing command intr pid 12 tid 100054 td 0x182c0180 sched_switch(182c0180,109) at sched_switch+0x4e9/frame 0xb4bbc30 mi_switch(109) at mi_switch+0x1c4/frame 0xb4bbc64 ithread_loop(9aa4e60,b4bbce8) at ithread_loop+0x85/frame 0xb4bbcb8 fork_exit(fa9c50,9aa4e60,b4bbce8,0,0,...) at fork_exit+0x68/frame 0xb4bbcd4 fork_trampoline() at 0xffc0340e/frame 0xb4bbcd4 --- kthread start Tracing command intr pid 12 tid 100060 td 0x182d2c00 sched_switch(182d2c00,109) at sched_switch+0x4e9/frame 0xb4dac30 mi_switch(109) at mi_switch+0x1c4/frame 0xb4dac64 ithread_loop(9aa4e50,b4dace8) at ithread_loop+0x85/frame 0xb4dacb8 fork_exit(fa9c50,9aa4e50,b4dace8,0,0,...) at fork_exit+0x68/frame 0xb4dacd4 fork_trampoline() at 0xffc0340e/frame 0xb4dacd4 --- kthread start Tracing command intr pid 12 tid 100076 td 0x182dd380 exit_tag() at 0xffc03404 Tracing command intr pid 12 tid 100082 td 0x18305e00 exit_tag() at 0xffc03404 Tracing command intr pid 12 tid 100083 td 0x18305a80 sched_switch(18305a80,109) at sched_switch+0x4e9/frame 0xb554c30 mi_switch(109) at mi_switch+0x1c4/frame 0xb554c64 ithread_loop(9aa4e20,b554ce8) at ithread_loop+0x85/frame 0xb554cb8 fork_exit(fa9c50,9aa4e20,b554ce8,0,0,...) at fork_exit+0x68/frame 0xb554cd4 fork_trampoline() at 0xffc0340e/frame 0xb554cd4 --- kthread start Tracing command intr pid 12 tid 100085 td 0x18305380 exit_tag() at 0xffc03404 Tracing command idle pid 11 tid 100003 td 0xb9c3500 sched_switch(b9c3500,608) at sched_switch+0x4e9/frame 0x12b21c20 mi_switch(608) at mi_switch+0x1c4/frame 0x12b21c54 critical_exit_preempt(0) at critical_exit_preempt+0x5c/frame 0x12b21c64 sched_idletd(0,12b21ce8) at sched_idletd+0x33e/frame 0x12b21cb8 fork_exit(1011070,0,12b21ce8,0,0,...) at fork_exit+0x68/frame 0x12b21cd4 fork_trampoline() at 0xffc0340e/frame 0x12b21cd4 --- kthread start Tracing command idle pid 11 tid 100004 td 0xb9c3180 sched_switch(b9c3180,108) at sched_switch+0x4e9/frame 0x12b26c30 mi_switch(108) at mi_switch+0x1c4/frame 0x12b26c64 sched_idletd(0,12b26ce8) at sched_idletd+0x7c/frame 0x12b26cb8 fork_exit(1011070,0,12b26ce8,0,0,...) at fork_exit+0x68/frame 0x12b26cd4 fork_trampoline() at 0xffc0340e/frame 0x12b26cd4 --- kthread start Tracing command idle pid 11 tid 100005 td 0xb9c2e00 sched_switch(b9c2e00,108) at sched_switch+0x4e9/frame 0x12b2bc30 mi_switch(108) at mi_switch+0x1c4/frame 0x12b2bc64 sched_idletd(0,12b2bce8) at sched_idletd+0x7c/frame 0x12b2bcb8 fork_exit(1011070,0,12b2bce8,0,0,...) at fork_exit+0x68/frame 0x12b2bcd4 fork_trampoline() at 0xffc0340e/frame 0x12b2bcd4 --- kthread start Tracing command idle pid 11 tid 100006 td 0xb9c2a80 sched_switch(b9c2a80,108) at sched_switch+0x4e9/frame 0x12b30c30 mi_switch(108) at mi_switch+0x1c4/frame 0x12b30c64 sched_idletd(0,12b30ce8) at sched_idletd+0x7c/frame 0x12b30cb8 fork_exit(1011070,0,12b30ce8,0,0,...) at fork_exit+0x68/frame 0x12b30cd4 fork_trampoline() at 0xffc0340e/frame 0x12b30cd4 --- kthread start Tracing command init pid 1 tid 100002 td 0xb9c3880 sched_switch(b9c3880,104) at sched_switch+0x4e9/frame 0x12b189d0 mi_switch(104) at mi_switch+0x1c4/frame 0x12b18a04 sleepq_switch(3280,14e9ff2,6c,0,1d79554,...) at sleepq_switch+0xf7/frame 0x12b18a24 sleepq_catch_signals(12b18a94,fefb0c,b9c16dc,6c,fd3fb1,...) at sleepq_catch_signals+0x278/frame 0x12b18a4c sleepq_wait_sig(b9c16dc,6c) at sleepq_wait_sig+0xe/frame 0x12b18a54 _sleep(b9c16dc,b9c1784,36c,1506bc5,0,0,0,0,100) at _sleep+0x27c/frame 0x12b18a94 kern_wait6(b9c3880,7,0,0,12b18bf0,30,0,0) at kern_wait6+0x3b8/frame 0x12b18af0 sys_wait4(b9c3880,b9c3b18,b9c3880,b9c16dc,1488da8,...) at sys_wait4+0x73/frame 0x12b18c08 syscall(12b18ce8,3b,3b,3b,ffbfe8f0,...) at syscall+0x1b8/frame 0x12b18cdc Xint0x80_syscall() at 0xffc033f9/frame 0x12b18cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xffc01d98, esp = 0xffc0bfe8, ebp = 0xffbfeccc --- Tracing command audit pid 10 tid 100001 td 0xb9c3c00 sched_switch(b9c3c00,104) at sched_switch+0x4e9/frame 0x12b13be0 mi_switch(104) at mi_switch+0x1c4/frame 0x12b13c18 sleepq_switch(1de9e0c,0) at sleepq_switch+0xf7/frame 0x12b13c38 _cv_wait(1de9e0c,1de9d68) at _cv_wait+0x135/frame 0x12b13c70 audit_worker(0,12b13ce8) at audit_worker+0x88/frame 0x12b13cb8 fork_exit(121b480,0,12b13ce8,0,0,...) at fork_exit+0x68/frame 0x12b13cd4 fork_trampoline() at 0xffc0340e/frame 0x12b13cd4 --- kthread start Tracing command kernel pid 0 tid 100000 td 0x1d51ba0 sched_switch(1d51ba0,104) at sched_switch+0x4e9/frame 0x22f9c10 mi_switch(104) at mi_switch+0x1c4/frame 0x22f9c48 sleepq_switch(1d51ba0,0,1d5185c,22f9ccc,1d51ba0,...) at sleepq_switch+0xf7/frame 0x22f9c68 sleepq_timedwait(1d5185c,54,1393d57,3faf3737,0,...) at sleepq_timedwait+0x3f/frame 0x22f9c8c _sleep(1d5185c,0,54,1509a07,fffff470,9,0,0,100) at _sleep+0x22a/frame 0x22f9ccc swapper(0,0,0,1acc654) at swapper+0x6d/frame 0x22f9d14 mi_startup() at mi_startup+0x2c6/frame 0x22f9d38 btext() at btext+0x5f Tracing command kernel pid 0 tid 100007 td 0xb9c2700 sched_switch(b9c2700,104) at sched_switch+0x4e9/frame 0x12b36bf8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b36c2c sleepq_switch(b9caa00,0) at sleepq_switch+0xf7/frame 0x12b36c4c msleep_spin_sbt(b9caa00,b9caa40,1435413,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12b36c84 gtaskqueue_thread_loop(9a3b804,12b36ce8) at gtaskqueue_thread_loop+0xfc/frame 0x12b36cb8 fork_exit(1026dc0,9a3b804,12b36ce8,0,0,...) at fork_exit+0x68/frame 0x12b36cd4 fork_trampoline() at 0xffc0340e/frame 0x12b36cd4 --- kthread start Tracing command kernel pid 0 tid 100008 td 0xb9c2380 sched_switch(b9c2380,104) at sched_switch+0x4e9/frame 0x12b3bbf8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b3bc2c sleepq_switch(b9ca800,0) at sleepq_switch+0xf7/frame 0x12b3bc4c msleep_spin_sbt(b9ca800,b9ca840,1435413,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12b3bc84 gtaskqueue_thread_loop(9a3b814,12b3bce8) at gtaskqueue_thread_loop+0xfc/frame 0x12b3bcb8 fork_exit(1026dc0,9a3b814,12b3bce8,0,0,...) at fork_exit+0x68/frame 0x12b3bcd4 fork_trampoline() at 0xffc0340e/frame 0x12b3bcd4 --- kthread start Tracing command kernel pid 0 tid 100009 td 0xb9c2000 sched_switch(b9c2000,104) at sched_switch+0x4e9/frame 0x12b40bf8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b40c2c sleepq_switch(b9ca600,0) at sleepq_switch+0xf7/frame 0x12b40c4c msleep_spin_sbt(b9ca600,b9ca640,1435413,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12b40c84 gtaskqueue_thread_loop(9a3b824,12b40ce8) at gtaskqueue_thread_loop+0xfc/frame 0x12b40cb8 fork_exit(1026dc0,9a3b824,12b40ce8,0,0,...) at fork_exit+0x68/frame 0x12b40cd4 fork_trampoline() at 0xffc0340e/frame 0x12b40cd4 --- kthread start Tracing command kernel pid 0 tid 100010 td 0xb9efc00 sched_switch(b9efc00,104) at sched_switch+0x4e9/frame 0x12b45bf8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b45c2c sleepq_switch(b9ca400,0) at sleepq_switch+0xf7/frame 0x12b45c4c msleep_spin_sbt(b9ca400,b9ca440,1435413,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12b45c84 gtaskqueue_thread_loop(9a3b834,12b45ce8) at gtaskqueue_thread_loop+0xfc/frame 0x12b45cb8 fork_exit(1026dc0,9a3b834,12b45ce8,0,0,...) at fork_exit+0x68/frame 0x12b45cd4 fork_trampoline() at 0xffc0340e/frame 0x12b45cd4 --- kthread start Tracing command kernel pid 0 tid 100011 td 0xb9ef880 sched_switch(b9ef880,104) at sched_switch+0x4e9/frame 0x12b4abf8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b4ac2c sleepq_switch(b9ca200,0) at sleepq_switch+0xf7/frame 0x12b4ac4c msleep_spin_sbt(b9ca200,b9ca240,1435413,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12b4ac84 gtaskqueue_thread_loop(9a3b404,12b4ace8) at gtaskqueue_thread_loop+0xfc/frame 0x12b4acb8 fork_exit(1026dc0,9a3b404,12b4ace8,0,0,...) at fork_exit+0x68/frame 0x12b4acd4 fork_trampoline() at 0xffc0340e/frame 0x12b4acd4 --- kthread start Tracing command kernel pid 0 tid 100012 td 0xb9ef500 sched_switch(b9ef500,104) at sched_switch+0x4e9/frame 0x12b4fbf8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b4fc2c sleepq_switch(b9ca000,0) at sleepq_switch+0xf7/frame 0x12b4fc4c msleep_spin_sbt(b9ca000,b9ca040,1435413,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12b4fc84 gtaskqueue_thread_loop(9a3b004,12b4fce8) at gtaskqueue_thread_loop+0xfc/frame 0x12b4fcb8 fork_exit(1026dc0,9a3b004,12b4fce8,0,0,...) at fork_exit+0x68/frame 0x12b4fcd4 fork_trampoline() at 0xffc0340e/frame 0x12b4fcd4 --- kthread start Tracing command kernel pid 0 tid 100013 td 0xb9ef180 sched_switch(b9ef180,104) at sched_switch+0x4e9/frame 0x12b54bf8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b54c2c sleepq_switch(b9f0d00,0) at sleepq_switch+0xf7/frame 0x12b54c4c msleep_spin_sbt(b9f0d00,b9f0d40,1435413,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12b54c84 gtaskqueue_thread_loop(9a3b014,12b54ce8) at gtaskqueue_thread_loop+0xfc/frame 0x12b54cb8 fork_exit(1026dc0,9a3b014,12b54ce8,0,0,...) at fork_exit+0x68/frame 0x12b54cd4 fork_trampoline() at 0xffc0340e/frame 0x12b54cd4 --- kthread start Tracing command kernel pid 0 tid 100014 td 0xb9eee00 sched_switch(b9eee00,104) at sched_switch+0x4e9/frame 0x12b59bf8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b59c2c sleepq_switch(b9f0b00,0) at sleepq_switch+0xf7/frame 0x12b59c4c msleep_spin_sbt(b9f0b00,b9f0b40,1435413,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12b59c84 gtaskqueue_thread_loop(9a3b024,12b59ce8) at gtaskqueue_thread_loop+0xfc/frame 0x12b59cb8 fork_exit(1026dc0,9a3b024,12b59ce8,0,0,...) at fork_exit+0x68/frame 0x12b59cd4 fork_trampoline() at 0xffc0340e/frame 0x12b59cd4 --- kthread start Tracing command kernel pid 0 tid 100015 td 0xb9eea80 sched_switch(b9eea80,104) at sched_switch+0x4e9/frame 0x12b5ebf8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b5ec2c sleepq_switch(b9f0900,0) at sleepq_switch+0xf7/frame 0x12b5ec4c msleep_spin_sbt(b9f0900,b9f0940,1435413,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12b5ec84 gtaskqueue_thread_loop(9a3b034,12b5ece8) at gtaskqueue_thread_loop+0xfc/frame 0x12b5ecb8 fork_exit(1026dc0,9a3b034,12b5ece8,0,0,...) at fork_exit+0x68/frame 0x12b5ecd4 fork_trampoline() at 0xffc0340e/frame 0x12b5ecd4 --- kthread start Tracing command kernel pid 0 tid 100017 td 0xb9ee380 sched_switch(b9ee380,104) at sched_switch+0x4e9/frame 0x12b68be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b68c20 sleepq_switch(b9f0500,0) at sleepq_switch+0xf7/frame 0x12b68c40 _sleep(b9f0500,b9f0540,0,1435413,0,0,0,0,100) at _sleep+0x2f3/frame 0x12b68c80 taskqueue_thread_loop(1de86d4,12b68ce8) at taskqueue_thread_loop+0xd5/frame 0x12b68cb8 fork_exit(103bce0,1de86d4,12b68ce8,0,0,...) at fork_exit+0x68/frame 0x12b68cd4 fork_trampoline() at 0xffc0340e/frame 0x12b68cd4 --- kthread start Tracing command kernel pid 0 tid 100018 td 0xb9ee000 sched_switch(b9ee000,104) at sched_switch+0x4e9/frame 0x12b6dbe8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b6dc20 sleepq_switch(b9f0300,0) at sleepq_switch+0xf7/frame 0x12b6dc40 _sleep(b9f0300,b9f0340,0,1435413,0,0,0,0,100) at _sleep+0x2f3/frame 0x12b6dc80 taskqueue_thread_loop(1d7aad4,12b6dce8) at taskqueue_thread_loop+0xd5/frame 0x12b6dcb8 fork_exit(103bce0,1d7aad4,12b6dce8,0,0,...) at fork_exit+0x68/frame 0x12b6dcd4 fork_trampoline() at 0xffc0340e/frame 0x12b6dcd4 --- kthread start Tracing command kernel pid 0 tid 100020 td 0xb9f3880 sched_switch(b9f3880,104) at sched_switch+0x4e9/frame 0x12b77be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b77c20 sleepq_switch(b9f4e00,0) at sleepq_switch+0xf7/frame 0x12b77c40 _sleep(b9f4e00,b9f4e40,0,1435413,0,0,0,0,100) at _sleep+0x2f3/frame 0x12b77c80 taskqueue_thread_loop(1dd4f98,12b77ce8) at taskqueue_thread_loop+0xd5/frame 0x12b77cb8 fork_exit(103bce0,1dd4f98,12b77ce8,0,0,...) at fork_exit+0x68/frame 0x12b77cd4 fork_trampoline() at 0xffc0340e/frame 0x12b77cd4 --- kthread start Tracing command kernel pid 0 tid 100021 td 0xb9f3500 sched_switch(b9f3500,104) at sched_switch+0x4e9/frame 0x12b7cbe8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b7cc20 sleepq_switch(b9f4c00,0) at sleepq_switch+0xf7/frame 0x12b7cc40 _sleep(b9f4c00,b9f4c40,0,1435413,0,0,0,0,100) at _sleep+0x2f3/frame 0x12b7cc80 taskqueue_thread_loop(1d52920,12b7cce8) at taskqueue_thread_loop+0xd5/frame 0x12b7ccb8 fork_exit(103bce0,1d52920,12b7cce8,0,0,...) at fork_exit+0x68/frame 0x12b7ccd4 fork_trampoline() at 0xffc0340e/frame 0x12b7ccd4 --- kthread start Tracing command kernel pid 0 tid 100023 td 0xb9f2e00 sched_switch(b9f2e00,104) at sched_switch+0x4e9/frame 0x12b86be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12b86c20 sleepq_switch(b9f4800,0) at sleepq_switch+0xf7/frame 0x12b86c40 _sleep(b9f4800,b9f4840,0,1435413,0,0,0,0,100) at _sleep+0x2f3/frame 0x12b86c80 taskqueue_thread_loop(1dd69ec,12b86ce8) at taskqueue_thread_loop+0xd5/frame 0x12b86cb8 fork_exit(103bce0,1dd69ec,12b86ce8,0,0,...) at fork_exit+0x68/frame 0x12b86cd4 fork_trampoline() at 0xffc0340e/frame 0x12b86cd4 --- kthread start Tracing command kernel pid 0 tid 100030 td 0x9aa5700 sched_switch(9aa5700,104) at sched_switch+0x4e9/frame 0x12bb9be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12bb9c20 sleepq_switch(9aac800,0) at sleepq_switch+0xf7/frame 0x12bb9c40 _sleep(9aac800,9aac840,0,1435413,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bb9c80 taskqueue_thread_loop(1d75dd4,12bb9ce8) at taskqueue_thread_loop+0xd5/frame 0x12bb9cb8 fork_exit(103bce0,1d75dd4,12bb9ce8,0,0,...) at fork_exit+0x68/frame 0x12bb9cd4 fork_trampoline() at 0xffc0340e/frame 0x12bb9cd4 --- kthread start Tracing command kernel pid 0 tid 100035 td 0x9adc500 sched_switch(9adc500,104) at sched_switch+0x4e9/frame 0x12bd2be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12bd2c20 sleepq_switch(9ad6800,0) at sleepq_switch+0xf7/frame 0x12bd2c40 _sleep(9ad6800,9ad6840,0,1435413,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bd2c80 taskqueue_thread_loop(1de9b50,12bd2ce8) at taskqueue_thread_loop+0xd5/frame 0x12bd2cb8 fork_exit(103bce0,1de9b50,12bd2ce8,0,0,...) at fork_exit+0x68/frame 0x12bd2cd4 fork_trampoline() at 0xffc0340e/frame 0x12bd2cd4 --- kthread start Tracing command kernel pid 0 tid 100036 td 0x9adc180 sched_switch(9adc180,104) at sched_switch+0x4e9/frame 0x12bd7be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12bd7c20 sleepq_switch(9ad6800,0) at sleepq_switch+0xf7/frame 0x12bd7c40 _sleep(9ad6800,9ad6840,0,1435413,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bd7c80 taskqueue_thread_loop(1de9b50,12bd7ce8) at taskqueue_thread_loop+0xd5/frame 0x12bd7cb8 fork_exit(103bce0,1de9b50,12bd7ce8,0,0,...) at fork_exit+0x68/frame 0x12bd7cd4 fork_trampoline() at 0xffc0340e/frame 0x12bd7cd4 --- kthread start Tracing command kernel pid 0 tid 100037 td 0x9adbe00 sched_switch(9adbe00,104) at sched_switch+0x4e9/frame 0x12bdcbe8 mi_switch(104) at mi_switch+0x1c4/frame 0x12bdcc20 sleepq_switch(9ad6800,0) at sleepq_switch+0xf7/frame 0x12bdcc40 _sleep(9ad6800,9ad6840,0,1435413,0,0,0,0,100) at _sleep+0x2f3/frame 0x12bdcc80 taskqueue_thread_loop(1de9b50,12bdcce8) at taskqueue_thread_loop+0xd5/frame 0x12bdccb8 fork_exit(103bce0,1de9b50,12bdcce8,0,0,...) at fork_exit+0x68/frame 0x12bdccd4 fork_trampoline() at 0xffc0340e/frame 0x12bdccd4 --- kthread start Tracing command kernel pid 0 tid 100038 td 0x9adba80 sched_switch(9adba80,104) at sched_switch+0x4e9/frame 0x12be1be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12be1c20 sleepq_switch(9ad6800,0) at sleepq_switch+0xf7/frame 0x12be1c40 _sleep(9ad6800,9ad6840,0,1435413,0,0,0,0,100) at _sleep+0x2f3/frame 0x12be1c80 taskqueue_thread_loop(1de9b50,12be1ce8) at taskqueue_thread_loop+0xd5/frame 0x12be1cb8 fork_exit(103bce0,1de9b50,12be1ce8,0,0,...) at fork_exit+0x68/frame 0x12be1cd4 fork_trampoline() at 0xffc0340e/frame 0x12be1cd4 --- kthread start Tracing command kernel pid 0 tid 100086 td 0x18305000 sched_switch(18305000,104) at sched_switch+0x4e9/frame 0xb563bf8 mi_switch(104) at mi_switch+0x1c4/frame 0xb563c2c sleepq_switch(18330300,0) at sleepq_switch+0xf7/frame 0xb563c4c msleep_spin_sbt(18330300,18330340,1435413,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0xb563c84 taskqueue_thread_loop(1df459c,b563ce8) at taskqueue_thread_loop+0xfc/frame 0xb563cb8 fork_exit(103bce0,1df459c,b563ce8,0,0,...) at fork_exit+0x68/frame 0xb563cd4 fork_trampoline() at 0xffc0340e/frame 0xb563cd4 --- kthread start Tracing command kernel pid 0 tid 100092 td 0x18914880 sched_switch(18914880,104) at sched_switch+0x4e9/frame 0xb58abf8 mi_switch(104) at mi_switch+0x1c4/frame 0xb58ac2c sleepq_switch(18909400,0) at sleepq_switch+0xf7/frame 0xb58ac4c msleep_spin_sbt(18909400,18909440,1435413,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0xb58ac84 taskqueue_thread_loop(1aed718,b58ace8) at taskqueue_thread_loop+0xfc/frame 0xb58acb8 fork_exit(103bce0,1aed718,b58ace8,0,0,...) at fork_exit+0x68/frame 0xb58acd4 fork_trampoline() at 0xffc0340e/frame 0xb58acd4 --- kthread start Tracing command kernel pid 0 tid 100093 td 0x18914500 sched_switch(18914500,104) at sched_switch+0x4e9/frame 0xb58fbf8 mi_switch(104) at mi_switch+0x1c4/frame 0xb58fc2c sleepq_switch(18909400,0) at sleepq_switch+0xf7/frame 0xb58fc4c msleep_spin_sbt(18909400,18909440,1435413,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0xb58fc84 taskqueue_thread_loop(1aed718,b58fce8) at taskqueue_thread_loop+0xfc/frame 0xb58fcb8 fork_exit(103bce0,1aed718,b58fce8,0,0,...) at fork_exit+0x68/frame 0xb58fcd4 fork_trampoline() at 0xffc0340e/frame 0xb58fcd4 --- kthread start Tracing command kernel pid 0 tid 100094 td 0x18914180 sched_switch(18914180,104) at sched_switch+0x4e9/frame 0xb594bf8 mi_switch(104) at mi_switch+0x1c4/frame 0xb594c2c sleepq_switch(18909400,0) at sleepq_switch+0xf7/frame 0xb594c4c msleep_spin_sbt(18909400,18909440,1435413,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0xb594c84 taskqueue_thread_loop(1aed718,b594ce8) at taskqueue_thread_loop+0xfc/frame 0xb594cb8 fork_exit(103bce0,1aed718,b594ce8,0,0,...) at fork_exit+0x68/frame 0xb594cd4 fork_trampoline() at 0xffc0340e/frame 0xb594cd4 --- kthread start Tracing command kernel pid 0 tid 100095 td 0x18913e00 sched_switch(18913e00,104) at sched_switch+0x4e9/frame 0xb599be8 mi_switch(104) at mi_switch+0x1c4/frame 0xb599c20 sleepq_switch(9b25d00,0) at sleepq_switch+0xf7/frame 0xb599c40 _sleep(9b25d00,9b25d40,0,1435413,0,0,0,0,100) at _sleep+0x2f3/frame 0xb599c80 taskqueue_thread_loop(1ae9770,b599ce8) at taskqueue_thread_loop+0xd5/frame 0xb599cb8 fork_exit(103bce0,1ae9770,b599ce8,0,0,...) at fork_exit+0x68/frame 0xb599cd4 fork_trampoline() at 0xffc0340e/frame 0xb599cd4 --- kthread start db:0:pho> dump Physical memory: 3428 MB Dumping 583 MB: 568 552 536 520 504 488 472 456 440 424 408 392 376 360 344 328 312 296 280 264 248 232 216 200 184 168 152 136 120 104 88 72 56 40 24 8 Dump complete db> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 3 $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/ufs/ffs/ffs_softdep.c =================================================================== --- /usr/src/sys/ufs/ffs/ffs_softdep.c (revision 367298) +++ /usr/src/sys/ufs/ffs/ffs_softdep.c (working copy) @@ -1464,7 +1464,7 @@ worklist_speedup(mp) } static void -softdep_send_speedup(struct ufsmount *ump, size_t shortage, u_int flags) +softdep_send_speedup(struct ufsmount *ump, uint64_t shortage, u_int flags) { struct buf *bp; @@ -1474,7 +1474,7 @@ static void bp = malloc(sizeof(*bp), M_TRIM, M_WAITOK | M_ZERO); bp->b_iocmd = BIO_SPEEDUP; bp->b_ioflags = flags; - bp->b_bcount = shortage; + bp->b_bcount = shortage <= (uint64_t)LONG_MAX ? shortage : LONG_MAX; g_vfs_strategy(ump->um_bo, bp); bufwait(bp); free(bp, M_TRIM); Index: /usr/src/sys/vm/vm_map.c =================================================================== --- /usr/src/sys/vm/vm_map.c (revision 367298) +++ /usr/src/sys/vm/vm_map.c (working copy) @@ -175,29 +175,106 @@ static void vm_map_wire_entry_failure(vm_map_t map start = end; \ } +#ifndef UMA_MD_SMALL_ALLOC + /* + * Allocate a new slab for kernel map entries. The kernel map may be locked or + * unlocked, depending on whether the request is coming from the kernel map or a + * submap. This function allocates a virtual address range directly from the + * kernel map instead of the kmem_* layer to avoid recursion on the kernel map + * lock and also to avoid triggering allocator recursion in the vmem boundary + * tag allocator. + */ +static void * +kmapent_alloc(uma_zone_t zone, vm_size_t bytes, int domain, uint8_t *pflag, + int wait) +{ + vm_offset_t addr; + int error, locked; + + *pflag = UMA_SLAB_PRIV; + + if (!(locked = vm_map_locked(kernel_map))) + vm_map_lock(kernel_map); + addr = vm_map_findspace(kernel_map, vm_map_min(kernel_map), bytes); + if (addr + bytes < addr || addr + bytes > vm_map_max(kernel_map)) + panic("%s: kernel map is exhausted", __func__); + error = vm_map_insert(kernel_map, NULL, 0, addr, addr + bytes, + VM_PROT_RW, VM_PROT_RW, MAP_NOFAULT); + if (error != KERN_SUCCESS) + panic("%s: vm_map_insert() failed: %d", __func__, error); + if (!locked) + vm_map_unlock(kernel_map); + error = kmem_back_domain(domain, kernel_object, addr, bytes, M_NOWAIT | + M_USE_RESERVE | (wait & M_ZERO)); + if (error == KERN_SUCCESS) { + return ((void *)addr); + } else { + if (!locked) + vm_map_lock(kernel_map); + vm_map_delete(kernel_map, addr, bytes); + if (!locked) + vm_map_unlock(kernel_map); + return (NULL); + } +} + +static void +kmapent_free(void *item, vm_size_t size, uint8_t pflag) +{ + vm_offset_t addr; + int error; + + if ((pflag & UMA_SLAB_PRIV) == 0) + /* XXX leaked */ + return; + + addr = (vm_offset_t)item; + kmem_unback(kernel_object, addr, size); + error = vm_map_remove(kernel_map, addr, addr + size); + KASSERT(error == KERN_SUCCESS, + ("%s: vm_map_remove failed: %d", __func__, error)); +} + +/* + * The worst-case upper bound on the number of kernel map entries that may be + * created before the zone must be replenished in _vm_map_unlock(). + */ +#define KMAPENT_RESERVE 1 + +#endif /* !UMD_MD_SMALL_ALLOC */ + +/* * vm_map_startup: * - * Initialize the vm_map module. Must be called before - * any other vm_map routines. + * Initialize the vm_map module. Must be called before any other vm_map + * routines. * - * Map and entry structures are allocated from the general - * purpose memory pool with some exceptions: - * - * - The kernel map and kmem submap are allocated statically. - * - Kernel map entries are allocated out of a static pool. - * - * These restrictions are necessary since malloc() uses the - * maps and requires map entries. + * User map and entry structures are allocated from the general purpose + * memory pool. Kernel maps are statically defined. Kernel map entries + * require special handling to avoid recursion; see the comments above + * kmapent_alloc() and in vm_map_entry_create(). */ - void vm_map_startup(void) { mtx_init(&map_sleep_mtx, "vm map sleep mutex", NULL, MTX_DEF); + + /* + * Disable the use of per-CPU buckets: map entry allocation is + * serialized by the kernel map lock. + */ kmapentzone = uma_zcreate("KMAP ENTRY", sizeof(struct vm_map_entry), NULL, NULL, NULL, NULL, UMA_ALIGN_PTR, - UMA_ZONE_MTXCLASS | UMA_ZONE_VM); + UMA_ZONE_VM | UMA_ZONE_NOBUCKET); +#ifndef UMA_MD_SMALL_ALLOC + /* Reserve an extra map entry for use when replenishing the reserve. */ + uma_zone_reserve(kmapentzone, KMAPENT_RESERVE + 1); + uma_prealloc(kmapentzone, KMAPENT_RESERVE + 1); + uma_zone_set_allocf(kmapentzone, kmapent_alloc); + uma_zone_set_freef(kmapentzone, kmapent_free); +#endif + mapentzone = uma_zcreate("MAP ENTRY", sizeof(struct vm_map_entry), NULL, NULL, NULL, NULL, UMA_ALIGN_PTR, 0); vmspace_zone = uma_zcreate("VMSPACE", sizeof(struct vmspace), NULL, @@ -630,9 +707,15 @@ _vm_map_unlock(vm_map_t map, const char *file, int { VM_MAP_UNLOCK_CONSISTENT(map); - if (map->system_map) + if (map->system_map) { +#ifndef UMA_MD_SMALL_ALLOC + if (map == kernel_map && (map->flags & MAP_REPLENISH) != 0) { + uma_prealloc(kmapentzone, 1); + map->flags &= ~MAP_REPLENISH; + } +#endif mtx_unlock_flags_(&map->system_mtx, 0, file, line); - else { + } else { sx_xunlock_(&map->lock, file, line); vm_map_process_deferred(); } @@ -652,9 +735,11 @@ void _vm_map_unlock_read(vm_map_t map, const char *file, int line) { - if (map->system_map) + if (map->system_map) { + KASSERT((map->flags & MAP_REPLENISH) == 0, + ("%s: MAP_REPLENISH leaked", __func__)); mtx_unlock_flags_(&map->system_mtx, 0, file, line); - else { + } else { sx_sunlock_(&map->lock, file, line); vm_map_process_deferred(); } @@ -726,6 +811,8 @@ _vm_map_lock_downgrade(vm_map_t map, const char *f { if (map->system_map) { + KASSERT((map->flags & MAP_REPLENISH) == 0, + ("%s: MAP_REPLENISH leaked", __func__)); mtx_assert_(&map->system_mtx, MA_OWNED, file, line); } else { VM_MAP_UNLOCK_CONSISTENT(map); @@ -769,10 +856,13 @@ _vm_map_unlock_and_wait(vm_map_t map, int timo, co VM_MAP_UNLOCK_CONSISTENT(map); mtx_lock(&map_sleep_mtx); - if (map->system_map) + if (map->system_map) { + KASSERT((map->flags & MAP_REPLENISH) == 0, + ("%s: MAP_REPLENISH leaked", __func__)); mtx_unlock_flags_(&map->system_mtx, 0, file, line); - else + } else { sx_xunlock_(&map->lock, file, line); + } return (msleep(&map->root, &map_sleep_mtx, PDROP | PVM, "vmmaps", timo)); } @@ -895,12 +985,33 @@ vm_map_entry_create(vm_map_t map) { vm_map_entry_t new_entry; - if (map->system_map) +#ifndef UMA_MD_SMALL_ALLOC + if (map == kernel_map) { + VM_MAP_ASSERT_LOCKED(map); + + /* + * A new slab of kernel map entries cannot be allocated at this + * point because the kernel map has not yet been updated to + * reflect the caller's request. Therefore, we allocate a new + * map entry, dipping into the reserve if necessary, and set a + * flag indicating that the reserve must be replenished before + * the map is unlocked. + */ + new_entry = uma_zalloc(kmapentzone, M_NOWAIT | M_NOVM); + if (new_entry == NULL) { + new_entry = uma_zalloc(kmapentzone, + M_NOWAIT | M_NOVM | M_USE_RESERVE); + kernel_map->flags |= MAP_REPLENISH; + } + } else +#endif + if (map->system_map) { new_entry = uma_zalloc(kmapentzone, M_NOWAIT); - else + } else { new_entry = uma_zalloc(mapentzone, M_WAITOK); - if (new_entry == NULL) - panic("vm_map_entry_create: kernel resources exhausted"); + } + KASSERT(new_entry != NULL, + ("vm_map_entry_create: kernel resources exhausted")); return (new_entry); } @@ -1786,6 +1897,8 @@ vm_map_findspace(vm_map_t map, vm_offset_t start, vm_size_t left_length, max_free_left, max_free_right; vm_offset_t gap_end; + VM_MAP_ASSERT_LOCKED(map); + /* * Request must fit within min/max VM address and must avoid * address wrap. Index: /usr/src/sys/vm/vm_map.h =================================================================== --- /usr/src/sys/vm/vm_map.h (revision 367298) +++ /usr/src/sys/vm/vm_map.h (working copy) @@ -227,6 +227,7 @@ struct vm_map { #define MAP_IS_SUB_MAP 0x04 /* has parent */ #define MAP_ASLR 0x08 /* enabled ASLR */ #define MAP_ASLR_IGNSTART 0x10 +#define MAP_REPLENISH 0x20 #ifdef _KERNEL #if defined(KLD_MODULE) && !defined(KLD_TIED) -- Test scenario: misc/mkfifo2d.sh