GENERIC HEAD from 2011-09-28 03:11:51 UTC, r225825, vmcore.201 KDB: debugger backends: ddb KDB: current backend: ddb 524288K of memory above 4GB ignored Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #0 r225825: Wed Sep 28 10:30:33 CEST 2011 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Family = 10 Model = 2 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 4294967296 (4096 MB) avail memory = 3531759616 (3368 MB) : Trying to mount root from ufs:/dev/ad4s1a [rw]... WARNING: / was not properly dismounted Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Starting ddb. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/ad4s1a: 6151 files, 749523 used, 263492 free (1324 frags, 32771 blocks, 0.1% fragmentation) /dev/ad4s1f: 17340 files, 821970 used, 191045 free (4413 frags, 23329 blocks, 0.4% fragmentation) /dev/label/tmp: 92437 files, 23661424 used, 94652557 free (2173 frags, 11831298 blocks, 0.0% fragmentation) /dev/ad4s1d: 857004 files, 8635586 used, 1518613 free (272693 frags, 155740 blocks, 2.7% fragmentation) /dev/ad4s1e: 819393 files, 22915007 used, 2473510 free (78094 frags, 299427 blocks, 0.3% fragmentation) Mounting local file systems:. Setting hostname: x4.osted.lan. re0: link state changed to DOWN Starting Network: lo0 re0 fwe0 fwip0. lo0: flags=8049 metric 0 mtu 16384 options=3 inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=389b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::221:85ff:fe15:337d%re0 prefixlen 64 tentative scopeid 0x1 nd6 options=29 media: Ethernet autoselect (none) status: no carrier fwe0: flags=8802 metric 0 mtu 1500 options=8 ether 02:dc:10:62:ad:eb nd6 options=29 ch 1 dma -1 fwip0: flags=8802 metric 0 mtu 1500 lladdr 0.dc.10.0.1.62.ad.eb.a.2.ff.fe.0.0.0.0 nd6 options=29 Starting devd. Starting Network: usbus0. Starting Network: usbus1. Starting Network: usbus2. Starting Network: usbus3. Starting Network: usbus4. Starting Network: usbus5. Starting Network: fwe0. fwe0: flags=8802 metric 0 mtu 1500 options=8 ether 02re0: lin:dc:10:62:ad:ebk state changed nd6 options=29to UP ch 1 dma -1 Starting Network: fwip0. fwip0: flags=8802 metric 0 mtu 1500 lladdr 0.dc.10.0.1.62.ad.eb.a.2.ff.fe.0.0.0.0 nd6 options=29 add net default: gateway 192.168.1.1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/kde4/lib /usr/local/lib /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/nss /usr/local/lib/qt4 /usr/local/lib/virtualbox /usr/local/lib/wine a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Creating and/or trimming log files. Starting syslogd. No core dumps found. Additional ABI support: linux. Starting rpcbind. NFS access cache time=60 lock order reversal: 1st 0xe0b1bac0 bufwait (bufwait) @ kern/vfs_bio.c:2658 2nd 0xc7bcfe00 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper(c0f07948,2e687361,38323a63,c7000a34,a7a,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a451fb,c0f0b32b,c7165370,c71693a0,f1801878,...) at kdb_backtrace+0x2a _witness_debugger(c0f0b32b,c7bcfe00,c0f3aba6,c71693a0,c0f3a838,...) at _witness_debugger+0x25 witness_checkorder(c7bcfe00,9,c0f3a82f,11c,0,...) at witness_checkorder+0x839 _sx_xlock(c7bcfe00,0,c0f3a82f,11c,c78fd828,...) at _sx_xlock+0x85 ufsdirhash_acquire(e0b1ba60,c78fd828,f18019f4,e178dde4,f1801948,...) at ufsdirhash_acquire+0x48 ufsdirhash_add(c78fd828,f18019f4,1de4,f1801934,f1801938,...) at ufsdirhash_add+0x13 ufs_direnter(c798ec3c,c7db3828,f18019f4,f1801b84,e0b1c0e0,...) at ufs_direnter+0x789 ufs_mkdir(f1801c14,c0f58d2e,0,0,f1801bac,...) at ufs_mkdir+0xa36 VOP_MKDIR_APV(c1052180,f1801c14,f1801b84,f1801bac,0,...) at VOP_MKDIR_APV+0xc5 kern_mkdirat(c77fab80,ffffff9c,28404020,0,1c0,...) at kern_mkdirat+0x291 kern_mkdir(c77fab80,28404020,0,1c0,f1801d1c,...) at kern_mkdir+0x2e sys_mkdir(c77fab80,f1801cec,c0f525ca,c0f0bea6,202,...) at sys_mkdir+0x29 syscall(f1801d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (136, FreeBSD ELF32, sys_mkdir), eip = 0x28174303, esp = 0xbfbfe8cc, ebp = 0xbfbfed78 --- Clearing /tmp (X related). Starting mountd. NFSv4 is disabled Starting nfsd. Recovering vi editor sessions:lock order reversal: 1st 0xc7e8a1b4 ufs (ufs) @ kern/vfs_lookup.c:530 2nd 0xe0b22c80 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:260 3rd 0xc7f35724 ufs (ufs) @ kern/vfs_subr.c:2134 KDB: stack backtrace: db_trace_self_wrapper(c0f07948,3a632e73,a303632,0,0,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a451fb,c0f0b344,c7165370,c7169338,f1878344,...) at kdb_backtrace+0x2a _witness_debugger(c0f0b344,c7f35724,c0efa584,c7169338,c0f1306a,...) at _witness_debugger+0x25 witness_checkorder(c7f35724,9,c0f13061,856,0,...) at witness_checkorder+0x839 __lockmgr_args(c7f35724,80100,c7f35790,0,0,...) at __lockmgr_args+0x834 ffs_lock(f1878468,c0a5661b,c0f12333,80100,c7f356cc,...) at ffs_lock+0xa1 VOP_LOCK1_APV(c1052180,f1878468,c795f0b0,c1062980,c7f356cc,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c7f356cc,80100,c0f13061,856,4,...) at _vn_lock+0x78 vget(c7f356cc,80100,c795f000,50,0,...) at vget+0xbb vfs_hash_get(c7beb87c,15f687,80000,c795f000,f18785ac,...) at vfs_hash_get+0xed ffs_vgetf(c7beb87c,15f687,80000,f18785ac,1,...) at ffs_vgetf+0x49 softdep_sync_buf(c7e8a15c,e0b22c20,1,106,0,...) at softdep_sync_buf+0xac9 ffs_syncvnode(c7e8a15c,1,c7165370,c11e8270,f187863c,...) at ffs_syncvnode+0x24c ffs_truncate(c7e8a15c,200,0,880,c792e300,...) at ffs_truncate+0x817 ufs_direnter(c7e8a15c,c7f356cc,f1878918,f1878bc0,0,...) at ufs_direnter+0x971 ufs_makeinode(f1878bc0,c1052700,f1878b04,f1878a60,c0d67c55,...) at ufs_makeinode+0x635 ufs_create(f1878b04,c0f58ef4,0,0,f1878b80,...) at ufs_create+0x30 VOP_CREATE_APV(c1052180,f1878b04,f1878bc0,f1878a9c,0,...) at VOP_CREATE_APV+0xc5 vn_open_cred(f1878b80,f1878c48,1b0,0,c792e300,...) at vn_open_cred+0x205 vn_open(f1878b80,f1878c48,1b0,c78ecd20,4e82e13e,...) at vn_open+0x3b kern_openat(c795f000,ffffff9c,28839f60,0,a03,...) at kern_openat+0x1ec kern_open(c795f000,28839f60,0,a02,1b0,...) at kern_open+0x35 sys_open(c795f000,f1878cec,c0f525ca,c0f0bea6,212,...) at sys_open+0x30 syscall(f1878d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (5, FreeBSD ELF32, sys_open), eip = 0x283d6343, esp = 0xbfbfc4fc, ebp = 0xbfbfc588 --- . Updating motd:. Starting ntpd. Starting default moused. Configuring syscons: keymap blanktime. Starting sshd. Starting cron. Local package initialization: backuplock order reversal: 1st 0xc7bdd5c8 ufs (ufs) @ kern/vfs_mount.c:1194 2nd 0xc7974724 devfs (devfs) @ ufs/ffs/ffs_softdep.c:1833 KDB: stack backtrace: db_trace_self_wrapper(c0f07948,38313a63,a3333,c0f3c205,c19b4888,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a451fb,c0f0b32b,c7169338,c7169200,f17f1a10,...) at kdb_backtrace+0x2a _witness_debugger(c0f0b32b,c7974724,c0ef23e7,c7169200,c0f36d39,...) at _witness_debugger+0x25 witness_checkorder(c7974724,9,c0f36d30,729,c7974790,...) at witness_checkorder+0x839 __lockmgr_args(c7974724,80400,c7974790,0,0,...) at __lockmgr_args+0x834 vop_stdlock(f17f1b30,4,c0f01fa7,80400,c79746cc,...) at vop_stdlock+0x65 VOP_LOCK1_APV(c1027b00,f17f1b30,c121b990,c1062980,c79746cc,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c79746cc,80400,c0f36d30,729,c7bec000,...) at _vn_lock+0x78 softdep_flushworklist(c7bec000,f17f1bdc,c77fb8a0,5dc,0,...) at softdep_flushworklist+0x47 ffs_sync(c7bec000,1,f17f1c30,4ef,c7025000,...) at ffs_sync+0x2fd dounmount(c7bec000,8080000,c77fb8a0,474,5b85c5ea,...) at dounmount+0x480 sys_unmount(c77fb8a0,f17f1cec,c,c,206,...) at sys_unmount+0x331 syscall(f17f1d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280de9ab, esp = 0xbfbfe64c, ebp = 0xbfbfe718 --- ** /dev/label/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 92437 files, 23661424 used, 94652557 free (2173 frags, 11831298 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** ** /dev/label/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 92437 files, 23661424 used, 94652557 free (2173 frags, 11831298 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** fsck -y /tmp watchdogd. Starting inetd. Wed Sep 28 11:06:33 CEST 2011 FreeBSD/i386 (x4.osted.lan) (console) login: FreeBSD/i386 (x4.osted.lan) (console) login: root Password: Sep 28 11:26:28 x4 login: ROOT LOGIN (root) ON console Last login: Wed Sep 28 10:56:26 on console FreeBSD 10.0-CURRENT (PHO) #0 r225825: Wed Sep 28 10:30:33 CEST 2011 Welcome to FreeBSD! Before seeking technical support, please use the following resources: o Security advisories and updated errata information for all releases are at http://www.FreeBSD.org/releases/ - always consult the ERRATA section for your release first as it's updated frequently. o The Handbook and FAQ documents are at http://www.FreeBSD.org/ and, along with the mailing lists, can be searched by going to http://www.FreeBSD.org/search/. If the doc distribution has been installed, they're also available formatted in /usr/share/doc. If you still have a question or problem, please take the output of `uname -a', along with any relevant error messages, and email it as a question to the questions@FreeBSD.org mailing list. If you are unfamiliar with FreeBSD's directory layout, please refer to the hier(7) manual page. If you are not familiar with manual pages, type `man man'. Edit /etc/motd to change this login announcement. You have new mail. x4# init 1 x4# Stopping inetd. Waiting for PIDS: 2184. Shutting down local packages:. Stopping cron. Waiting for PIDS: 2109. Stopping sshd. Waiting for PIDS: 2089. Stopping moused. Waiting for PIDS: 2029, 2029. Stopping ntpd. Waiting for PIDS: 1951. Stopping nfsd. Waiting for PIDS: 1846 1852. Stopping mountd. Waiting for PIDS: 1836. Stopping rpcbind. Waiting for PIDS: 1737. Stopping devd. Waiting for PIDS: 1509. Writing entropy file:. Terminated . Sep 28 11:26:34 x4 syslogd: exiting on signal 15 Enter full pathname of shell or RETURN for /bin/sh: # cd ~pho/stress2/wip # sh -x ./s16b.sh + id -u + [ 0 -ne 0 ] + . ../default.cfg + export RUNDIR=/tmp/stressX + export RUNTIME=2m + export VERBOSE=1 + export LOAD=20 + export INCARNATIONS=20 + export BLASTHOST=192.168.1.3 + export runRUNTIME=3d + export runINCARNATIONS=1 + export runLOAD=100 + export swapINCARNATIONS=40 + export swapLOAD=80 + export syscallKILL=1 + export rwLOAD=70 + export mkdirLOAD=80 + export creatLOAD=80 + export symlinkLOAD=20 + export tcpKILL=1 + export shmINCARNATIONS=5 + export 'TESTPROGS= testcases/rw/rw testcases/swap/swap testcases/creat/creat testcases/mkdir/mkdir testcases/thr1/thr1 testcases/udp/udp testcases/tcp/tcp ' + [ -r default.cfg ] + [ -r default.cfg ] + echo ./s16b.sh + grep -q cleanup.sh + echo ./s16b.sh + diskimage=/var/tmp/diskimage + mntpoint=/mnt + testuser=pho + mdstart=5 + part=a + hostname + [ -f ./x4.osted.lan ] + hostname + [ -f ../x4.osted.lan ] + hostname + . ../x4.osted.lan + export BLASTHOST=192.168.1.5 + export mkfifoKILL=1 + mount + grep -w /mnt + grep -q /dev/md + mdconfig -l + grep -q md5 + mdconfig -a -t swap -s 2g -u 5 + bsdlabel -w md5 auto + newfs -U -j md5a + mount /dev/md5a /mnt + chmod 777 /mnt + cp -R /usr/include /mnt/1 + cp -R /usr/include /mnt/2 + cp -R /usr/include /mnt/3 + cp -R /usr/include /mnt/4 + sleep 2 + rm -rf /mnt/1 + [ ! -s /mnt/.snap/s16b ] + rm -f /mnt/.snap/s16b + mksnap_ffs /mnt /mnt/.snap/s16b fsync: giving up on dirty 0xc83f66cc: tag devfs, type VCHR usecount 1, writecount 0, refcount 64 mountedhere 0xc7877e00 flags () v_object 0xc79544c8 ref 0 pages 666 lock type devfs: EXCL by thread 0xc7ec78a0 (pid 2632) #0 0xc09f45be at __lockmgr_args+0xc1e #1 0xc0a94d45 at vop_stdlock+0x65 #2 0xc0d66c85 at VOP_LOCK1_APV+0xb5 #3 0xc0ab5ee8 at _vn_lock+0x78 #4 0xc0c47497 at softdep_flushworklist+0x47 #5 0xc0c4ae3d at ffs_sync+0x2fd #6 0xc0ab5cc2 at vfs_write_suspend+0x122 #7 0xc0c2f11d at ffs_snapshot+0xced #8 0xc0c4d313 at ffs_mount+0x1c13 #9 0xc0a9ea34 at vfs_donmount+0x1254 #10 0xc0a9f104 at sys_nmount+0x84 #11 0xc0d438d4 at syscall+0x2a4 #12 0xc0d2cd31 at Xint0x80_syscall+0x21 dev md5a mksnap_ffs: Cannot create snapshot /mnt/.snap/s16b: Resource temporarily unavailable + [ ! -s /mnt/.snap/s16b ] + rm -f /mnt/.snap/s16b + mksnap_ffs /mnt /mnt/.snap/s16b load: 0.37 cmd: mksnap_ffs 2634 [runnable] 12.22r 0.00u 12.21s 69% 1020k Expensive timeout(9) function: 0xc09fd720(0) 0.200059304 s load: 2.28 cmd: mksnap_ffs 2634 [running] 342.42r 0.00u 342.41s 99% 1020k load: 1.98 cmd: mksnap_ffs 2634 [running] 1045.42r 0.00u 1045.42s 100% 1020k ~KDB: enter: Break to debugger [ thread pid 11 tid 100003 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> ps pid ppid pgrp uid state wmesg wchan cmd 2634 2607 2607 0 R+ CPU 3 mksnap_ffs 2620 0 0 0 DL mdwait 0xc7564000 [md5] 2607 2606 2607 0 S+ wait 0xc795db10 sh 2606 1 2606 0 Ss+ wait 0xc7919b10 sh 18 0 0 0 DL sdflush 0xc121b9e0 [softdepflush] 17 0 0 0 DL vlruwt 0xc77f6588 [vnlru] 16 0 0 0 DL syncer 0xc12152d4 [syncer] 9 0 0 0 DL psleep 0xc1214f68 [bufdaemon] 8 0 0 0 DL pgzero 0xc121d85c [pagezero] 7 0 0 0 DL psleep 0xc121d448 [vmdaemon] 6 0 0 0 DL psleep 0xc121d410 [pagedaemon] 5 0 0 0 DL ccb_scan 0xc1063dd4 [xpt_thrd] 4 0 0 0 DL waiting_ 0xc1216df8 [sctp_iterator] 3 0 0 0 DL - 0xc73c6c3c [fdc0] 2 0 0 0 SL - 0xc7460000 [fw0_probe] 15 0 0 0 DL (threaded) [usb] 100058 D - 0xc7443d34 [usbus5] 100057 D - 0xc7443d04 [usbus5] 100056 D - 0xc7443cd4 [usbus5] db> bt 2634 Tracing pid 2634 tid 100106 td 0xc795f2e0 cpustop_handler(3,13,f1872424,c0d43cd6,f18723b8,...) at cpustop_handler+0x28 ipi_nmi_handler(f18723b8,0,c795f2e0,c6f2dc80,c7c16b10,...) at ipi_nmi_handler+0x47 trap(f1872430) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0aba0a2, esp = 0xf1872470, ebp = 0xf1872478 --- strncmp(c0f39c10,c0ef94c0,3,c71621e0,f18724b4,...) at strncmp+0x32 fixup_filename(c11e7aa8,c0f39c19,c11e7a88,c1037018,c8e2c4d8,...) at fixup_filename+0x2b witness_lock(c8e2c4d8,8,c0f39c10,5c2,0,...) at witness_lock+0x53 _mtx_lock_flags(c8e2c4d8,0,c0f39c10,5c2,0,...) at _mtx_lock_flags+0x113 ffs_sync(c7be0000,1,c0f14838,482,c0f14838,...) at ffs_sync+0x178 vfs_write_suspend(c7be0000,f1872874,0,8000,c715fe80,...) at vfs_write_suspend+0x122 ffs_snapshot(c7be0000,c73ff2c0,0,0,0,...) at ffs_snapshot+0xced ffs_mount(c7be0000,c7793b00,ff,394,c7162450,...) at ffs_mount+0x1c13 vfs_donmount(c795f2e0,11000,c7888980,c7888980,bfbfe498,...) at vfs_donmount+0x1254 sys_nmount(c795f2e0,f1872cec,c0f525ca,c0f0bfa3,213,...) at sys_nmount+0x84 syscall(f1872d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (378, FreeBSD ELF32, sys_nmount), eip = 0x280dc62b, esp = 0xbfbfe40c, ebp = 0xbfbfeb88 --- db> c ~KDB: enter: Break to debugger [ thread pid 11 tid 100003 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> c^h ^hbt 2634 Tracing pid 2634 tid 100106 td 0xc795f2e0 cpustop_handler(2,13,f1872430,c0d43cd6,447baa9b,...) at cpustop_handler+0x28 ipi_nmi_handler(447baa9b,5ec6f80,c795f2e0,c6f2acb4,c7c16b10,...) at ipi_nmi_handler+0x47 trap(f187243c) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0a565f3, esp = 0xf187247c, ebp = 0xf187247c --- fixup_filename(c71621e0,c795f390,c0f39c19,f18724b4,c0a566bc,...) at fixup_filename+0x3 witness_unlock(c8f91a48,8,c0f39c10,5cb,c8f91984,...) at witness_unlock+0x5a _mtx_unlock_flags(c8f91a48,0,c0f39c10,5cb,0,...) at _mtx_unlock_flags+0xc3 ffs_sync(c7be0000,1,c0f14838,482,c0f14838,...) at ffs_sync+0x1dd vfs_write_suspend(c7be0000,f1872874,0,8000,c715fe80,...) at vfs_write_suspend+0x122 ffs_snapshot(c7be0000,c73ff2c0,0,0,0,...) at ffs_snapshot+0xced ffs_mount(c7be0000,c7793b00,ff,394,c7162450,...) at ffs_mount+0x1c13 vfs_donmount(c795f2e0,11000,c7888980,c7888980,bfbfe498,...) at vfs_donmount+0x1254 sys_nmount(c795f2e0,f1872cec,c0f525ca,c0f0bfa3,213,...) at sys_nmount+0x84 syscall(f1872d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (378, FreeBSD ELF32, sys_nmount), eip = 0x280dc62b, esp = 0xbfbfe40c, ebp = 0xbfbfeb88 --- db> c ~KDB: enter: Break to debugger [ thread pid 11 tid 100003 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> c^h ^hbt 2634 Tracing pid 2634 tid 100106 td 0xc795f2e0 cpustop_handler(1,13,f1872424,c0d43cd6,f18723b8,...) at cpustop_handler+0x28 ipi_nmi_handler(f18723b8,0,c795f2e0,c6f27c80,c7c16b10,...) at ipi_nmi_handler+0x47 trap(f1872430) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0aba0a2, esp = 0xf1872470, ebp = 0xf1872478 --- strncmp(c0f39c10,c0ef94c0,3,c71621e0,f18724b4,...) at strncmp+0x32 fixup_filename(c11e7aa8,c0f39c19,c11e7a88,c1037018,c8cabd00,...) at fixup_filename+0x2b witness_lock(c8cabd00,8,c0f39c10,5c2,0,...) at witness_lock+0x53 _mtx_lock_flags(c8cabd00,0,c0f39c10,5c2,0,...) at _mtx_lock_flags+0x113 ffs_sync(c7be0000,1,c0f14838,482,c0f14838,...) at ffs_sync+0x178 vfs_write_suspend(c7be0000,f1872874,0,8000,c715fe80,...) at vfs_write_suspend+0x122 ffs_snapshot(c7be0000,c73ff2c0,0,0,0,...) at ffs_snapshot+0xced ffs_mount(c7be0000,c7793b00,ff,394,c7162450,...) at ffs_mount+0x1c13 vfs_donmount(c795f2e0,11000,c7888980,c7888980,bfbfe498,...) at vfs_donmount+0x1254 sys_nmount(c795f2e0,f1872cec,c0f525ca,c0f0bfa3,213,...) at sys_nmount+0x84 syscall(f1872d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (378, FreeBSD ELF32, sys_nmount), eip = 0x280dc62b, esp = 0xbfbfe40c, ebp = 0xbfbfeb88 --- db> c ~KDB: enter: Break to debugger [ thread pid 11 tid 100003 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> run pho db:0:pho> bt Tracing pid 11 tid 100003 td 0xc71bb8a0 kdb_enter(c0d81bde,c0f078f9,c6f37ba4,c087c98b,c7454700,...) at kdb_enter+0x3a kdb_break(c7454700,1944280,c0d28695,c7454780,c7454758,...) at kdb_break+0x28 uart_intr(c7454700,0,c124ab20,c6f37be8,0,...) at uart_intr+0x11b intr_event_handle(c7188c80,c6f37c10,c6f37c34,0,c73c5800,...) at intr_event_handle+0x76 intr_execute_handlers(c72000d0,c6f37c10,0,c6f37c50,c0d2d095,...) at intr_execute_handlers+0x49 lapic_handle_intr(3c,c6f37c10) at lapic_handle_intr+0x36 Xapic_isr1() at Xapic_isr1+0x35 --- interrupt, eip = 0xc0d28695, esp = 0xc6f37c50, ebp = 0xc6f37c50 --- acpi_cpu_c1(c6f37c6c,c1064050,0,301,afd,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c6f37ca0,c0d368c9,0,ffffffff,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,ffffffff,c10abd40,2,c6f37ce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c6f37cc4,c0f05a2b,3bb,c71bb8a0,...) at cpu_idle+0x89 sched_idletd(0,c6f37d28,c0efe310,3e5,c71b984c,...) at sched_idletd+0x263 fork_exit(c0a323a0,0,c6f37d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f37d60, ebp = 0 --- db:0:bt> show allpcpu Current CPU: 0 cpuid = 0 dynamic pcpu = 0x7c1f80 curthread = 0xc71bb8a0: pid 11 "idle: cpu0" curpcb = 0xc6f37d80 fpcurthread = none idlethread = 0xc71bb8a0: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 dynamic pcpu = 0x5ec3f80 curthread = 0xc7202000: pid 12 "swi4: clock" curpcb = 0xc6f4ad80 fpcurthread = none idlethread = 0xc71bb5c0: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: cpuid = 2 dynamic pcpu = 0x5ec6f80 curthread = 0xc71bb2e0: pid 11 "idle: cpu2" curpcb = 0xc6f3dd80 fpcurthread = none idlethread = 0xc71bb2e0: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 spin locks held: cpuid = 3 dynamic pcpu = 0x5ec9f80 curthread = 0xc795f2e0: pid 2634 "mksnap_ffs" curpcb = 0xf1872d80 fpcurthread = none idlethread = 0xc71bb000: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 spin locks held: db:0:allpcpu> show alllocks Process 2634 (mksnap_ffs) thread 0xc795f2e0 (100106) exclusive sleep mutex vnode interlock (vnode interlock) r = 0 (0xc8f1aba4) locked @ ufs/ffs/ffs_vfsops.c:1474 exclusive sleep mutex struct mount mtx (struct mount mtx) r = 0 (0xc7be0000) locked @ kern/vfs_mount.c:1687 exclusive sleep mutex Giant (Giant) r = 0 (0xc109d7d0) locked @ kern/vfs_mount.c:1069 db:0:alllocks> show lockedvnods Locked vnodes db:0:lockedvnods> show mount 0xc7897b50 /dev/ad4s1a on / (ufs) 0xc7898000 devfs on /dev (devfs) 0xc7be05a8 /dev/ad4s1f on /home (ufs) 0xc7bebb50 /dev/ad4s1d on /usr (ufs) 0xc7beb87c /dev/ad4s1e on /var (ufs) 0xc7beb5a8 procfs on /proc (procfs) 0xc7be02d4 /dev/label/tmp on /tmp (ufs) 0xc7be0000 /dev/md5a on /mnt (ufs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 2634 2607 2607 0 R+ CPU 3 mksnap_ffs 2620 0 0 0 DL mdwait 0xc7564000 [md5] 2607 2606 2607 0 S+ wait 0xc795db10 sh 2606 1 2606 0 Ss+ wait 0xc7919b10 sh 18 0 0 0 DL sdflush 0xc121b9e0 [softdepflush] 17 0 0 0 DL vlruwt 0xc77f6588 [vnlru] 16 0 0 0 DL syncer 0xc12152d4 [syncer] 9 0 0 0 DL psleep 0xc1214f68 [bufdaemon] 8 0 0 0 DL pgzero 0xc121d85c [pagezero] 7 0 0 0 DL psleep 0xc121d448 [vmdaemon] 6 0 0 0 DL psleep 0xc121d410 [pagedaemon] 5 0 0 0 DL ccb_scan 0xc1063dd4 [xpt_thrd] 4 0 0 0 DL waiting_ 0xc1216df8 [sctp_iterator] 3 0 0 0 DL - 0xc73c6c3c [fdc0] 2 0 0 0 SL - 0xc7460000 [fw0_probe] 15 0 0 0 DL (threaded) [usb] 100058 D - 0xc7443d34 [usbus5] 100057 D - 0xc7443d04 [usbus5] 100056 D - 0xc7443cd4 [usbus5] 100055 D - 0xc7443ca4 [usbus5] 100053 D - 0xc7436b5c [usbus4] 100052 D - 0xc7436b2c [usbus4] 100051 D - 0xc7436afc [usbus4] 100050 D - 0xc7436acc [usbus4] 100049 D - 0xc7431b5c [usbus3] 100048 D - 0xc7431b2c [usbus3] 100047 D - 0xc7431afc [usbus3] 100046 D - 0xc7431acc [usbus3] 100045 D - 0xc7427b5c [usbus2] 100044 D - 0xc7427b2c [usbus2] 100043 D - 0xc7427afc [usbus2] 100042 D - 0xc7427acc [usbus2] 100040 D - 0xc7421b5c [usbus1] 100039 D - 0xc7421b2c [usbus1] 100038 D - 0xc7421afc [usbus1] 100037 D - 0xc7421acc [usbus1] 100035 D - 0xc7416b5c [usbus0] 100034 D - 0xc7416b2c [usbus0] 100033 D - 0xc7416afc [usbus0] 100032 D - 0xc7416acc [usbus0] 14 0 0 0 DL - 0xc109dee4 [yarrow] 13 0 0 0 DL (threaded) [geom] 100015 D - 0xc109bb08 [g_down] 100014 D - 0xc109bb04 [g_up] 100013 D - 0xc109bafc [g_event] 12 0 0 0 RL (threaded) [intr] 100066 I [irq12: psm0] 100065 I [irq1: atkbd0] 100063 I [swi0: uart] 100060 I [irq20: fwohci0] 100059 I [irq14: ata0] 100054 I [irq19: ehci0] 100041 I [irq18: ohci2 ohci4] 100036 I [irq17: ohci1 ohci3] 100031 I [irq16: hdac1 ohci0] 100030 I [irq22: ahci0] 100029 I [irq257: re0] 100028 I [irq256: hdac0] 100026 I [swi5: +] 100025 I [swi2: cambio] 100019 I [swi6: task queue] 100018 I [swi6: Giant taskq] 100012 I [swi4: clock] 100011 I [swi4: clock] 100010 I [swi4: clock] 100009 Run CPU 1 [swi4: clock] 100008 I [swi3: vm] 100007 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100006 CanRun [idle: cpu3] 100005 Run CPU 2 [idle: cpu2] 100004 CanRun [idle: cpu1] 100003 Run CPU 0 [idle: cpu0] 1 0 1 0 SLs wait 0xc71b9b10 [init] 10 0 0 0 DL audit_wo 0xc121b100 [audit] 0 0 0 0 DLs (threaded) [kernel] 100077 D - 0xc77b5a80 [mca taskq] 100067 D - 0xc109dee4 [deadlkres] 100061 D - 0xc7468980 [fw0_taskq] 100027 D - 0xc7191580 [thread taskq] 100024 D - 0xc71916c0 [kqueue taskq] 100023 D - 0xc7191700 [acpi_task_2] 100022 D - 0xc7191700 [acpi_task_1] 100021 D - 0xc7191700 [acpi_task_0] 100020 D - 0xc7191740 [ffs_trim taskq] 100016 D - 0xc7191bc0 [firmware taskq] 100000 D sched 0xc109bbe0 [swapper] db:0:ps> allt Tracing command mksnap_ffs pid 2634 tid 100106 td 0xc795f2e0 cpustop_handler(3,13,f1872438,c0d43cd6,c795f2e0,...) at cpustop_handler+0x28 ipi_nmi_handler(c795f2e0,c6f2dcb4,c6f2dc80,f18723f4,c7c16b10,...) at ipi_nmi_handler+0x47 trap(f1872444) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0a56caf, esp = 0xf1872484, ebp = 0xf18724b8 --- witness_unlock(c8f1aba4,8,c0f39c10,5cb,c8f1aae0,...) at witness_unlock+0x4f _mtx_unlock_flags(c8f1aba4,0,c0f39c10,5cb,0,...) at _mtx_unlock_flags+0xc3 ffs_sync(c7be0000,1,c0f14838,482,c0f14838,...) at ffs_sync+0x1dd vfs_write_suspend(c7be0000,f1872874,0,8000,c715fe80,...) at vfs_write_suspend+0x122 ffs_snapshot(c7be0000,c73ff2c0,0,0,0,...) at ffs_snapshot+0xced ffs_mount(c7be0000,c7793b00,ff,394,c7162450,...) at ffs_mount+0x1c13 vfs_donmount(c795f2e0,11000,c7888980,c7888980,bfbfe498,...) at vfs_donmount+0x1254 sys_nmount(c795f2e0,f1872cec,c0f525ca,c0f0bfa3,213,...) at sys_nmount+0x84 syscall(f1872d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (378, FreeBSD ELF32, sys_nmount), eip = 0x280dc62b, esp = 0xbfbfe40c, ebp = 0xbfbfeb88 --- Tracing command md5 pid 2620 tid 100098 td 0xc7964b80 sched_switch(c7964b80,0,104,191,cd352da4,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c7964b80,0,c0f09280,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7564000,5c,c0ea72d1,0,0,...) at sleepq_wait+0x63 _sleep(c7564000,c7564020,25c,c0ea72d1,0,...) at _sleep+0x372 md_kthread(c7564000,f183dd28,c0efe310,3e5,c795ab10,...) at md_kthread+0x11a fork_exit(c0734270,c7564000,f183dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xf183dd60, ebp = 0 --- Tracing command sh pid 2607 tid 100087 td 0xc7965b80 sched_switch(c7965b80,0,104,191,ca089690,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c7965b80,0,c0f09280,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f09280,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c795db10,6c,c0f0bfff,100,0,...) at sleepq_wait_sig+0x17 _sleep(c795db10,c795db98,16c,c0f0bfff,0,...) at _sleep+0x35c kern_wait(c7965b80,ffffffff,f1811c60,2,0,...) at kern_wait+0x626 sys_wait4(c7965b80,f1811cec,c0f525ca,c0f0bfde,212,...) at sys_wait4+0x3b syscall(f1811d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x2816c5cb, esp = 0xbfbfeadc, ebp = 0xbfbfeaf8 --- Tracing command sh pid 2606 tid 100082 td 0xc77fb000 sched_switch(c77fb000,0,104,191,7e738421,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c77fb000,0,c0f09280,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f09280,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7919b10,6c,c0f0bfff,100,0,...) at sleepq_wait_sig+0x17 _sleep(c7919b10,c7919b98,16c,c0f0bfff,0,...) at _sleep+0x35c kern_wait(c77fb000,ffffffff,f17fdc60,2,0,...) at kern_wait+0x626 sys_wait4(c77fb000,f17fdcec,c,c,212,...) at sys_wait4+0x3b syscall(f17fdd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x2816c5cb, esp = 0xbfbfebfc, ebp = 0xbfbfec18 --- Tracing command softdepflush pid 18 tid 100076 td 0xc743d8a0 sched_switch(c743d8a0,0,104,191,5199971e,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,54,...) at mi_switch+0x1eb sleepq_switch(c743d8a0,0,c0f09280,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c121b9e0,54,c0f39385,0,0,...) at sleepq_timedwait+0x6b _sleep(c121b9e0,c121b990,54,c0f39385,3e8,...) at _sleep+0x342 softdep_flush(0,ed40ed28,c0efe310,3e5,c77f62c4,...) at softdep_flush+0x266 fork_exit(c0c47660,0,ed40ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed40ed60, ebp = 0 --- Tracing command vnlru pid 17 tid 100075 td 0xc743db80 sched_switch(c743db80,0,104,191,d258bf03,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,60,...) at mi_switch+0x1eb sleepq_switch(c743db80,0,c0f09280,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c77f6588,60,c0f13fd2,0,0,...) at sleepq_timedwait+0x6b _sleep(c77f6588,c1215294,260,c0f13fd2,3e8,...) at _sleep+0x342 vnlru_proc(0,ed40bd28,c0efe310,3e5,c77f6588,...) at vnlru_proc+0xe7 fork_exit(c0aaa070,0,ed40bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed40bd60, ebp = 0 --- Tracing command syncer pid 16 tid 100074 td 0xc7559000 sched_switch(c7559000,0,104,191,bce41959,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7559000,0,c0f09280,28b,c7559000,...) at sleepq_switch+0x162 sleepq_timedwait(c12152d4,0,ed408c78,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c12152d4,c12152c0,3e8,76a,4e20,...) at _cv_timedwait+0x252 sched_sync(0,ed408d28,c0efe310,3e5,c77f684c,...) at sched_sync+0x595 fork_exit(c0aa8010,0,ed408d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed408d60, ebp = 0 --- Tracing command bufdaemon pid 9 tid 100073 td 0xc75592e0 sched_switch(c75592e0,0,104,191,a7899582,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,54,...) at mi_switch+0x1eb sleepq_switch(c75592e0,0,c0f09280,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c1214f68,54,c0f10f6f,0,0,...) at sleepq_timedwait+0x6b _sleep(c1214f68,c1214f70,54,c0f10f6f,3e8,...) at _sleep+0x342 buf_daemon(0,ed405d28,c0efe310,3e5,c77f6b10,...) at buf_daemon+0x175 fork_exit(c0a8ed80,0,ed405d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed405d60, ebp = 0 --- Tracing command pagezero pid 8 tid 100072 td 0xc75595c0 sched_switch(c75595c0,0,104,191,63cd8afa,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c75595c0,0,c0f09280,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c121d85c,0,c0f3fcec,0,0,...) at sleepq_timedwait+0x6b _sleep(c121d85c,c121c300,0,c0f3fcec,493e0,...) at _sleep+0x342 vm_pagezero(0,ed402d28,c0efe310,3e5,c77f8000,...) at vm_pagezero+0xdc fork_exit(c0c8bdf0,0,ed402d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed402d60, ebp = 0 --- Tracing command vmdaemon pid 7 tid 100071 td 0xc75598a0 sched_switch(c75598a0,0,104,191,5539a589,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,74,...) at mi_switch+0x1eb sleepq_switch(c75598a0,0,c0f09280,268,0,...) at sleepq_switch+0x162 sleepq_wait(c121d448,74,c0f10f6f,0,0,...) at sleepq_wait+0x63 _sleep(c121d448,c121d44c,74,c0f10f6f,0,...) at _sleep+0x372 vm_daemon(0,ed3ffd28,c0efe310,3e5,c71ba2c4,...) at vm_daemon+0x59 fork_exit(c0c85be0,0,ed3ffd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed3ffd60, ebp = 0 --- Tracing command pagedaemon pid 6 tid 100070 td 0xc7559b80 sched_switch(c7559b80,0,104,191,fa626496,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,54,...) at mi_switch+0x1eb sleepq_switch(c7559b80,0,c0f09280,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c121d410,54,c0f10f6f,0,0,...) at sleepq_timedwait+0x6b _sleep(c121d410,c121c300,54,c0f10f6f,1388,...) at _sleep+0x342 vm_pageout(0,ed3fcd28,c0efe310,3e5,c71ba588,...) at vm_pageout+0x2c7 fork_exit(c0c86cc0,0,ed3fcd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed3fcd60, ebp = 0 --- Tracing command xpt_thrd pid 5 tid 100069 td 0xc755a000 sched_switch(c755a000,0,104,191,5b4010fa,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c755a000,0,c0f09280,268,0,...) at sleepq_switch+0x162 sleepq_wait(c1063dd4,5c,c0d6bc65,0,0,...) at sleepq_wait+0x63 _sleep(c1063dd4,c1063e18,5c,c0d6bc65,0,...) at _sleep+0x372 xpt_scanner_thread(0,ed399d28,c0efe310,3e5,c71ba84c,...) at xpt_scanner_thread+0x47 fork_exit(c0492320,0,ed399d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed399d60, ebp = 0 --- Tracing command sctp_iterator pid 4 tid 100068 td 0xc755a2e0 sched_switch(c755a2e0,0,104,191,17435c59,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c755a2e0,0,c0f09280,268,0,...) at sleepq_switch+0x162 sleepq_wait(c1216df8,0,c0f2727f,0,0,...) at sleepq_wait+0x63 _sleep(c1216df8,c1216dc0,0,c0f2727f,0,...) at _sleep+0x372 sctp_iterator_thread(0,ed396d28,c0efe310,3e5,c71bab10,...) at sctp_iterator_thread+0x5c fork_exit(c0b3a920,0,ed396d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed396d60, ebp = 0 --- Tracing command fdc0 pid 3 tid 100064 td 0xc755b000 sched_switch(c755b000,0,104,191,5198b118,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c755b000,0,c0f09280,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c73c6c3c,5c,c0ef6992,0,0,...) at sleepq_timedwait+0x6b _sleep(c73c6c3c,c73c6cf0,5c,c0ef6992,3e8,...) at _sleep+0x342 fdc_thread(c73c6c00,ed38ad28,c0efe310,3e5,c7420000,...) at fdc_thread+0x27d fork_exit(c0d0b0c0,c73c6c00,ed38ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed38ad60, ebp = 0 --- Tracing command fw0_probe pid 2 tid 100062 td 0xc742f5c0 sched_switch(c742f5c0,0,104,191,5b40755f,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c742f5c0,0,c0f09280,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f09280,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7460000,6c,c0ef6992,100,0,...) at sleepq_wait_sig+0x17 _sleep(c7460000,c7464488,16c,c0ef6992,0,...) at _sleep+0x35c fw_bus_probe_thread(c7460000,ed37ad28,c0efe310,3e5,c74202c4,...) at fw_bus_probe_thread+0xa08 fork_exit(c06b6f10,c7460000,ed37ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed37ad60, ebp = 0 --- Tracing command usb pid 15 tid 100058 td 0xc743b2e0 sched_switch(c743b2e0,0,104,191,48a77758,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c743b2e0,0,c0f09280,268,c743b2e0,...) at sleepq_switch+0x162 sleepq_wait(c7443d34,0,ed363cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7443d34,c7443dd4,c0eddb31,6b,c7443d3c,...) at _cv_wait+0x243 usb_process(c7443d2c,ed363d28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7443d2c,ed363d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed363d60, ebp = 0 --- Tracing command usb pid 15 tid 100057 td 0xc743b5c0 sched_switch(c743b5c0,0,104,191,a5f183e6,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c743b5c0,0,c0f09280,268,c743b5c0,...) at sleepq_switch+0x162 sleepq_wait(c7443d04,0,ed360cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7443d04,c7443dd4,c0eddb31,6b,c7443d0c,...) at _cv_wait+0x243 usb_process(c7443cfc,ed360d28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7443cfc,ed360d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed360d60, ebp = 0 --- Tracing command usb pid 15 tid 100056 td 0xc743b8a0 sched_switch(c743b8a0,0,104,191,484180e0,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c743b8a0,0,c0f09280,268,c743b8a0,...) at sleepq_switch+0x162 sleepq_wait(c7443cd4,0,ed35dcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7443cd4,c7443dd4,c0eddb31,6b,c7443cdc,...) at _cv_wait+0x243 usb_process(c7443ccc,ed35dd28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7443ccc,ed35dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed35dd60, ebp = 0 --- Tracing command usb pid 15 tid 100055 td 0xc743bb80 sched_switch(c743bb80,0,104,191,48416f47,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c743bb80,0,c0f09280,268,c743bb80,...) at sleepq_switch+0x162 sleepq_wait(c7443ca4,0,ed35acac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7443ca4,c7443dd4,c0eddb31,6b,c7443cac,...) at _cv_wait+0x243 usb_process(c7443c9c,ed35ad28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7443c9c,ed35ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed35ad60, ebp = 0 --- Tracing command usb pid 15 tid 100053 td 0xc743d2e0 sched_switch(c743d2e0,0,104,191,48415afa,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c743d2e0,0,c0f09280,268,c743d2e0,...) at sleepq_switch+0x162 sleepq_wait(c7436b5c,0,ed351cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7436b5c,c7436bfc,c0eddb31,6b,c7436b64,...) at _cv_wait+0x243 usb_process(c7436b54,ed351d28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7436b54,ed351d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed351d60, ebp = 0 --- Tracing command usb pid 15 tid 100052 td 0xc743d5c0 sched_switch(c743d5c0,0,104,191,a5f07c73,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c743d5c0,0,c0f09280,268,c743d5c0,...) at sleepq_switch+0x162 sleepq_wait(c7436b2c,0,ed34ecac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7436b2c,c7436bfc,c0eddb31,6b,c7436b34,...) at _cv_wait+0x243 usb_process(c7436b24,ed34ed28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7436b24,ed34ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed34ed60, ebp = 0 --- Tracing command usb pid 15 tid 100051 td 0xc742c000 sched_switch(c742c000,0,104,191,47dd5f98,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c742c000,0,c0f09280,268,c742c000,...) at sleepq_switch+0x162 sleepq_wait(c7436afc,0,ed34bcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7436afc,c7436bfc,c0eddb31,6b,c7436b04,...) at _cv_wait+0x243 usb_process(c7436af4,ed34bd28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7436af4,ed34bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed34bd60, ebp = 0 --- Tracing command usb pid 15 tid 100050 td 0xc742c2e0 sched_switch(c742c2e0,0,104,191,47dd4eb3,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c742c2e0,0,c0f09280,268,c742c2e0,...) at sleepq_switch+0x162 sleepq_wait(c7436acc,0,ed348cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7436acc,c7436bfc,c0eddb31,6b,c7436ad4,...) at _cv_wait+0x243 usb_process(c7436ac4,ed348d28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7436ac4,ed348d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed348d60, ebp = 0 --- Tracing command usb pid 15 tid 100049 td 0xc742c5c0 sched_switch(c742c5c0,0,104,191,47dd3ac6,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c742c5c0,0,c0f09280,268,c742c5c0,...) at sleepq_switch+0x162 sleepq_wait(c7431b5c,0,ed345cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7431b5c,c7431bfc,c0eddb31,6b,c7431b64,...) at _cv_wait+0x243 usb_process(c7431b54,ed345d28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7431b54,ed345d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed345d60, ebp = 0 --- Tracing command usb pid 15 tid 100048 td 0xc742c8a0 sched_switch(c742c8a0,0,104,191,a5f03e1a,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c742c8a0,0,c0f09280,268,c742c8a0,...) at sleepq_switch+0x162 sleepq_wait(c7431b2c,0,ed342cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7431b2c,c7431bfc,c0eddb31,6b,c7431b34,...) at _cv_wait+0x243 usb_process(c7431b24,ed342d28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7431b24,ed342d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed342d60, ebp = 0 --- Tracing command usb pid 15 tid 100047 td 0xc742cb80 sched_switch(c742cb80,0,104,191,420346b9,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c742cb80,0,c0f09280,268,c742cb80,...) at sleepq_switch+0x162 sleepq_wait(c7431afc,0,ed33fcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7431afc,c7431bfc,c0eddb31,6b,c7431b04,...) at _cv_wait+0x243 usb_process(c7431af4,ed33fd28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7431af4,ed33fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed33fd60, ebp = 0 --- Tracing command usb pid 15 tid 100046 td 0xc742e000 sched_switch(c742e000,0,104,191,42033179,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c742e000,0,c0f09280,268,c742e000,...) at sleepq_switch+0x162 sleepq_wait(c7431acc,0,ed33ccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7431acc,c7431bfc,c0eddb31,6b,c7431ad4,...) at _cv_wait+0x243 usb_process(c7431ac4,ed33cd28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7431ac4,ed33cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed33cd60, ebp = 0 --- Tracing command usb pid 15 tid 100045 td 0xc742e2e0 sched_switch(c742e2e0,0,104,191,42031bfa,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c742e2e0,0,c0f09280,268,c742e2e0,...) at sleepq_switch+0x162 sleepq_wait(c7427b5c,0,ed339cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7427b5c,c7427bfc,c0eddb31,6b,c7427b64,...) at _cv_wait+0x243 usb_process(c7427b54,ed339d28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7427b54,ed339d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed339d60, ebp = 0 --- Tracing command usb pid 15 tid 100044 td 0xc742e5c0 sched_switch(c742e5c0,0,104,191,a5efcb70,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c742e5c0,0,c0f09280,268,c742e5c0,...) at sleepq_switch+0x162 sleepq_wait(c7427b2c,0,ed336cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7427b2c,c7427bfc,c0eddb31,6b,c7427b34,...) at _cv_wait+0x243 usb_process(c7427b24,ed336d28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7427b24,ed336d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed336d60, ebp = 0 --- Tracing command usb pid 15 tid 100043 td 0xc742e8a0 sched_switch(c742e8a0,0,104,191,419e8762,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c742e8a0,0,c0f09280,268,c742e8a0,...) at sleepq_switch+0x162 sleepq_wait(c7427afc,0,ed333cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7427afc,c7427bfc,c0eddb31,6b,c7427b04,...) at _cv_wait+0x243 usb_process(c7427af4,ed333d28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7427af4,ed333d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed333d60, ebp = 0 --- Tracing command usb pid 15 tid 100042 td 0xc742eb80 sched_switch(c742eb80,0,104,191,419e7471,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c742eb80,0,c0f09280,268,c742eb80,...) at sleepq_switch+0x162 sleepq_wait(c7427acc,0,e0ffdcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7427acc,c7427bfc,c0eddb31,6b,c7427ad4,...) at _cv_wait+0x243 usb_process(c7427ac4,e0ffdd28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7427ac4,e0ffdd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0ffdd60, ebp = 0 --- Tracing command usb pid 15 tid 100040 td 0xc741c000 sched_switch(c741c000,0,104,191,419e6161,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c741c000,0,c0f09280,268,c741c000,...) at sleepq_switch+0x162 sleepq_wait(c7421b5c,0,e0ff6cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7421b5c,c7421bfc,c0eddb31,6b,c7421b64,...) at _cv_wait+0x243 usb_process(c7421b54,e0ff6d28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7421b54,e0ff6d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0ff6d60, ebp = 0 --- Tracing command usb pid 15 tid 100039 td 0xc741c2e0 sched_switch(c741c2e0,0,104,191,a5ef9821,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c741c2e0,0,c0f09280,268,c741c2e0,...) at sleepq_switch+0x162 sleepq_wait(c7421b2c,0,e0ff3cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7421b2c,c7421bfc,c0eddb31,6b,c7421b34,...) at _cv_wait+0x243 usb_process(c7421b24,e0ff3d28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7421b24,e0ff3d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0ff3d60, ebp = 0 --- Tracing command usb pid 15 tid 100038 td 0xc741c5c0 sched_switch(c741c5c0,0,104,191,413a8f5d,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c741c5c0,0,c0f09280,268,c741c5c0,...) at sleepq_switch+0x162 sleepq_wait(c7421afc,0,e0ff0cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7421afc,c7421bfc,c0eddb31,6b,c7421b04,...) at _cv_wait+0x243 usb_process(c7421af4,e0ff0d28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7421af4,e0ff0d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0ff0d60, ebp = 0 --- Tracing command usb pid 15 tid 100037 td 0xc741c8a0 sched_switch(c741c8a0,0,104,191,413a7da1,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c741c8a0,0,c0f09280,268,c741c8a0,...) at sleepq_switch+0x162 sleepq_wait(c7421acc,0,e0fedcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7421acc,c7421bfc,c0eddb31,6b,c7421ad4,...) at _cv_wait+0x243 usb_process(c7421ac4,e0fedd28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7421ac4,e0fedd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0fedd60, ebp = 0 --- Tracing command usb pid 15 tid 100035 td 0xc741e000 sched_switch(c741e000,0,104,191,413a6861,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c741e000,0,c0f09280,268,c741e000,...) at sleepq_switch+0x162 sleepq_wait(c7416b5c,0,e0fe6cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7416b5c,c7416bfc,c0eddb31,6b,c7416b64,...) at _cv_wait+0x243 usb_process(c7416b54,e0fe6d28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7416b54,e0fe6d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0fe6d60, ebp = 0 --- Tracing command usb pid 15 tid 100034 td 0xc741e2e0 sched_switch(c741e2e0,0,104,191,a5ef3c45,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c741e2e0,0,c0f09280,268,c741e2e0,...) at sleepq_switch+0x162 sleepq_wait(c7416b2c,0,e0fe3cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7416b2c,c7416bfc,c0eddb31,6b,c7416b34,...) at _cv_wait+0x243 usb_process(c7416b24,e0fe3d28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7416b24,e0fe3d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0fe3d60, ebp = 0 --- Tracing command usb pid 15 tid 100033 td 0xc741e5c0 sched_switch(c741e5c0,0,104,191,40d66bb5,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c741e5c0,0,c0f09280,268,c741e5c0,...) at sleepq_switch+0x162 sleepq_wait(c7416afc,0,e0fe0cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7416afc,c7416bfc,c0eddb31,6b,c7416b04,...) at _cv_wait+0x243 usb_process(c7416af4,e0fe0d28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7416af4,e0fe0d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0fe0d60, ebp = 0 --- Tracing command usb pid 15 tid 100032 td 0xc741e8a0 sched_switch(c741e8a0,0,104,191,40d65924,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c741e8a0,0,c0f09280,268,c741e8a0,...) at sleepq_switch+0x162 sleepq_wait(c7416acc,0,e0fddcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7416acc,c7416bfc,c0eddb31,6b,c7416ad4,...) at _cv_wait+0x243 usb_process(c7416ac4,e0fddd28,c0efe310,3e5,c7420588,...) at usb_process+0x193 fork_exit(c08b1170,c7416ac4,e0fddd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0fddd60, ebp = 0 --- Tracing command yarrow pid 14 tid 100017 td 0xc72c45c0 sched_switch(c72c45c0,0,104,191,c79def91,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c72c45c0,0,c0f09280,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c109dee4,0,c0ef6992,2,0,...) at sleepq_timedwait+0x6b _sleep(c109dee4,0,0,c0ef6992,64,...) at _sleep+0x342 pause(c0ef6992,64,c0ebd6ce,111,0,...) at pause+0x47 random_kthread(0,c6f62d28,c0efe310,3e5,c71b9000,...) at random_kthread+0x1ef fork_exit(c07cab60,0,c6f62d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f62d60, ebp = 0 --- Tracing command geom pid 13 tid 100015 td 0xc72c4b80 sched_switch(c72c4b80,0,104,191,ce21b3c4,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c72c4b80,0,c0f09280,268,0,...) at sleepq_switch+0x162 sleepq_wait(c109bb08,5c,c0ef6992,0,0,...) at sleepq_wait+0x63 _sleep(c109bb08,c109ba68,25c,c0ef6992,0,...) at _sleep+0x372 g_io_schedule_down(c72c4b80,0,c0ef84d9,6c,c6f5cd14,...) at g_io_schedule_down+0x5b g_down_procbody(0,c6f5cd28,c0efe310,3e5,c71b92c4,...) at g_down_procbody+0x9d fork_exit(c09a1810,0,c6f5cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f5cd60, ebp = 0 --- Tracing command geom pid 13 tid 100014 td 0xc72c5000 sched_switch(c72c5000,0,104,191,ce294078,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c72c5000,0,c0f09280,268,0,...) at sleepq_switch+0x162 sleepq_wait(c109bb04,5c,c0ef6992,0,0,...) at sleepq_wait+0x63 _sleep(c109bb04,c109ba88,25c,c0ef6992,0,...) at _sleep+0x372 g_io_schedule_up(c72c5000,0,c0ef84d9,5f,c6f59d14,...) at g_io_schedule_up+0x134 g_up_procbody(0,c6f59d28,c0efe310,3e5,c71b92c4,...) at g_up_procbody+0x9d fork_exit(c09a18b0,0,c6f59d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f59d60, ebp = 0 --- Tracing command geom pid 13 tid 100013 td 0xc71be2e0 sched_switch(c71be2e0,0,104,191,ab23c952,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c71be2e0,0,c0f09280,268,0,...) at sleepq_switch+0x162 sleepq_wait(c109bafc,5c,c0ef6992,0,0,...) at sleepq_wait+0x63 _sleep(c109bafc,c109ba2c,25c,c0ef6992,0,...) at _sleep+0x372 g_run_events(c10abd40,0,c0ef84d9,79,c6f56d14,...) at g_run_events+0x526 g_event_procbody(0,c6f56d28,c0efe310,3e5,c71b92c4,...) at g_event_procbody+0x90 fork_exit(c09a1950,0,c6f56d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f56d60, ebp = 0 --- Tracing command intr pid 12 tid 100066 td 0xc755a8a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100065 td 0xc755ab80 sched_switch(c755ab80,0,109,191,40d2b491,...) at sched_switch+0x3bc mi_switch(109,0,c0efe633,554,c755ab80,...) at mi_switch+0x1eb ithread_loop(c7201550,ed38dd28,c0efe310,3e5,c71b9588,...) at ithread_loop+0x1fb fork_exit(c09e3050,c7201550,ed38dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed38dd60, ebp = 0 --- Tracing command intr pid 12 tid 100063 td 0xc742f2e0 sched_switch(c742f2e0,0,109,191,fb9407b,...) at sched_switch+0x3bc mi_switch(109,0,c0efe633,554,c742f2e0,...) at mi_switch+0x1eb ithread_loop(c7201560,ed37dd28,c0efe310,3e5,c71b9588,...) at ithread_loop+0x1fb fork_exit(c09e3050,c7201560,ed37dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed37dd60, ebp = 0 --- Tracing command intr pid 12 tid 100060 td 0xc742fb80 sched_switch(c742fb80,0,109,191,368cc84f,...) at sched_switch+0x3bc mi_switch(109,0,c0efe633,554,c742fb80,...) at mi_switch+0x1eb ithread_loop(c72015c0,ed371d28,c0efe310,3e5,c71b9588,...) at ithread_loop+0x1fb fork_exit(c09e3050,c72015c0,ed371d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed371d60, ebp = 0 --- Tracing command intr pid 12 tid 100059 td 0xc743b000 sched_switch(c743b000,0,109,191,53a0f8a7,...) at sched_switch+0x3bc mi_switch(109,0,c0efe633,554,c743b000,...) at mi_switch+0x1eb ithread_loop(c72015d0,ed369d28,c0efe310,3e5,c71b9588,...) at ithread_loop+0x1fb fork_exit(c09e3050,c72015d0,ed369d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed369d60, ebp = 0 --- Tracing command intr pid 12 tid 100054 td 0xc743d000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100041 td 0xc742f000 sched_switch(c742f000,0,109,191,65e26eae,...) at sched_switch+0x3bc mi_switch(109,0,c0efe633,554,c742f000,...) at mi_switch+0x1eb ithread_loop(c7201670,e0ffad28,c0efe310,3e5,c71b9588,...) at ithread_loop+0x1fb fork_exit(c09e3050,c7201670,e0ffad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0ffad60, ebp = 0 --- Tracing command intr pid 12 tid 100036 td 0xc741cb80 sched_switch(c741cb80,0,109,191,60695f90,...) at sched_switch+0x3bc mi_switch(109,0,c0efe633,554,c741cb80,...) at mi_switch+0x1eb ithread_loop(c72016a0,e0fead28,c0efe310,3e5,c71b9588,...) at ithread_loop+0x1fb fork_exit(c09e3050,c72016a0,e0fead28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0fead60, ebp = 0 --- Tracing command intr pid 12 tid 100031 td 0xc741eb80 sched_switch(c741eb80,0,109,191,dfd6335c,...) at sched_switch+0x3bc mi_switch(109,0,c0efe633,554,c741eb80,...) at mi_switch+0x1eb ithread_loop(c72016d0,e0fdad28,c0efe310,3e5,c71b9588,...) at ithread_loop+0x1fb fork_exit(c09e3050,c72016d0,e0fdad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0fdad60, ebp = 0 --- Tracing command intr pid 12 tid 100030 td 0xc72c52e0 sched_switch(c72c52e0,0,109,191,ce283a47,...) at sched_switch+0x3bc mi_switch(109,0,c0efe633,554,c72c52e0,...) at mi_switch+0x1eb ithread_loop(c72016e0,e0e84d28,c0efe310,3e5,c71b9588,...) at ithread_loop+0x1fb fork_exit(c09e3050,c72016e0,e0e84d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0e84d60, ebp = 0 --- Tracing command intr pid 12 tid 100029 td 0xc72c55c0 sched_switch(c72c55c0,0,109,191,40f349b8,...) at sched_switch+0x3bc mi_switch(109,0,c0efe633,554,c72c55c0,...) at mi_switch+0x1eb ithread_loop(c72016f0,e0e80d28,c0efe310,3e5,c71b9588,...) at ithread_loop+0x1fb fork_exit(c09e3050,c72016f0,e0e80d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0e80d60, ebp = 0 --- Tracing command intr pid 12 tid 100028 td 0xc72c58a0 sched_switch(c72c58a0,0,109,191,372f50d9,...) at sched_switch+0x3bc mi_switch(109,0,c0efe633,554,c72c58a0,...) at mi_switch+0x1eb ithread_loop(c7201740,c6fb1d28,c0efe310,3e5,c71b9588,...) at ithread_loop+0x1fb fork_exit(c09e3050,c7201740,c6fb1d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6fb1d60, ebp = 0 --- Tracing command intr pid 12 tid 100026 td 0xc73a2000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100025 td 0xc73a22e0 sched_switch(c73a22e0,0,109,191,ce28aa87,...) at sched_switch+0x3bc mi_switch(109,0,c0efe633,554,c73a22e0,...) at mi_switch+0x1eb ithread_loop(c7201770,c6f7ad28,c0efe310,3e5,c71b9588,...) at ithread_loop+0x1fb fork_exit(c09e3050,c7201770,c6f7ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f7ad60, ebp = 0 --- Tracing command intr pid 12 tid 100019 td 0xc72c4000 sched_switch(c72c4000,0,109,191,ee4574ad,...) at sched_switch+0x3bc mi_switch(109,0,c0efe633,554,c72c4000,...) at mi_switch+0x1eb ithread_loop(c7201780,c6f68d28,c0efe310,3e5,c71b9588,...) at ithread_loop+0x1fb fork_exit(c09e3050,c7201780,c6f68d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f68d60, ebp = 0 --- Tracing command intr pid 12 tid 100018 td 0xc72c42e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100012 td 0xc71be5c0 sched_switch(c71be5c0,0,109,191,fa624801,...) at sched_switch+0x3bc mi_switch(109,0,c0efe633,554,c71be5c0,...) at mi_switch+0x1eb ithread_loop(c72017a0,c6f53d28,c0efe310,3e5,c71b9588,...) at ithread_loop+0x1fb fork_exit(c09e3050,c72017a0,c6f53d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f53d60, ebp = 0 --- Tracing command intr pid 12 tid 100011 td 0xc71be8a0 sched_switch(c71be8a0,0,109,191,c79da0a3,...) at sched_switch+0x3bc mi_switch(109,0,c0efe633,554,c71be8a0,...) at mi_switch+0x1eb ithread_loop(c72017b0,c6f50d28,c0efe310,3e5,c71b9588,...) at ithread_loop+0x1fb fork_exit(c09e3050,c72017b0,c6f50d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f50d60, ebp = 0 --- Tracing command intr pid 12 tid 100010 td 0xc71beb80 sched_switch(c71beb80,0,109,191,51991bcd,...) at sched_switch+0x3bc mi_switch(109,0,c0efe633,554,c71beb80,...) at mi_switch+0x1eb ithread_loop(c72017c0,c6f4dd28,c0efe310,3e5,c71b9588,...) at ithread_loop+0x1fb fork_exit(c09e3050,c72017c0,c6f4dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f4dd60, ebp = 0 --- Tracing command intr pid 12 tid 100009 td 0xc7202000 cpustop_handler(1,13,c6f4ab50,c0d43cd6,f5,...) at cpustop_handler+0x28 ipi_nmi_handler(f5,0,1cbeb80,c09fc656,c71b9588,...) at ipi_nmi_handler+0x47 trap(c6f4ab5c) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc09fc639, esp = 0xc6f4ab9c, ebp = 0xc6f4abb8 --- _mtx_lock_sleep(c109d7d0,c7202000,0,c0f01fa7,93,...) at _mtx_lock_sleep+0x99 _mtx_lock_flags(c109d7d0,0,c0f01fa7,93,c6f4ac90,...) at _mtx_lock_flags+0xf7 lock_mtx(c109d7d0,1,c0f050fe,216,0,...) at lock_mtx+0x29 softclock(c109f120,c6f4acb8,c09fc054,c10ac3c0,c71881b8,...) at softclock+0x1ea intr_event_execute_handlers(c71b9588,c7188180,c0efe633,55e,c7202000,...) at intr_event_execute_handlers+0x125 ithread_loop(c72017d0,c6f4ad28,c0efe310,3e5,c71b9588,...) at ithread_loop+0xa9 fork_exit(c09e3050,c72017d0,c6f4ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f4ad60, ebp = 0 --- Tracing command intr pid 12 tid 100008 td 0xc72022e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100007 td 0xc72025c0 sched_switch(c72025c0,0,109,191,f49066bb,...) at sched_switch+0x3bc mi_switch(109,0,c0efe633,554,c72025c0,...) at mi_switch+0x1eb ithread_loop(c72017f0,c6f44d28,c0efe310,3e5,c71b9588,...) at ithread_loop+0x1fb fork_exit(c09e3050,c72017f0,c6f44d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f44d60, ebp = 0 --- Tracing command idle pid 11 tid 100006 td 0xc71bb000 sched_switch(c71bb000,0,108,18c,a5efda95,...) at sched_switch+0x3bc mi_switch(108,0,c0f05a2b,3b9,c71bb000,...) at mi_switch+0x1eb sched_idletd(0,c6f40d28,c0efe310,3e5,c71b984c,...) at sched_idletd+0x19f fork_exit(c0a323a0,0,c6f40d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f40d60, ebp = 0 --- Tracing command idle pid 11 tid 100005 td 0xc71bb2e0 cpustop_handler(2,13,c6f3dc04,c0d43cd6,f5,...) at cpustop_handler+0x28 ipi_nmi_handler(f5,0,0,c124b178,c71b984c,...) at ipi_nmi_handler+0x47 trap(c6f3dc10) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0d28695, esp = 0xc6f3dc50, ebp = 0xc6f3dc50 --- acpi_cpu_c1(c6f3dc6c,c1064050,2,301,afd,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c6f3dca0,c0d368c9,0,ffffffff,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,ffffffff,c10aca40,2,c6f3dce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c6f3dcc4,c0f05a2b,3bb,c71bb2e0,...) at cpu_idle+0x89 sched_idletd(0,c6f3dd28,c0efe310,3e5,c71b984c,...) at sched_idletd+0x263 fork_exit(c0a323a0,0,c6f3dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f3dd60, ebp = 0 --- Tracing command idle pid 11 tid 100004 td 0xc71bb5c0 sched_switch(c71bb5c0,0,108,18c,fbc39653,...) at sched_switch+0x3bc mi_switch(108,0,c0f05a2b,3b9,c71bb5c0,...) at mi_switch+0x1eb sched_idletd(0,c6f3ad28,c0efe310,3e5,c71b984c,...) at sched_idletd+0x19f fork_exit(c0a323a0,0,c6f3ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f3ad60, ebp = 0 --- Tracing command idle pid 11 tid 100003 td 0xc71bb8a0 kdb_enter(c0d81bde,c0f078f9,c6f37ba4,c087c98b,c7454700,...) at kdb_enter+0x3a kdb_break(c7454700,1944280,c0d28695,c7454780,c7454758,...) at kdb_break+0x28 uart_intr(c7454700,0,c124ab20,c6f37be8,0,...) at uart_intr+0x11b intr_event_handle(c7188c80,c6f37c10,c6f37c34,0,c73c5800,...) at intr_event_handle+0x76 intr_execute_handlers(c72000d0,c6f37c10,0,c6f37c50,c0d2d095,...) at intr_execute_handlers+0x49 lapic_handle_intr(3c,c6f37c10) at lapic_handle_intr+0x36 Xapic_isr1() at Xapic_isr1+0x35 --- interrupt, eip = 0xc0d28695, esp = 0xc6f37c50, ebp = 0xc6f37c50 --- acpi_cpu_c1(c6f37c6c,c1064050,0,301,afd,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c6f37ca0,c0d368c9,0,ffffffff,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,ffffffff,c10abd40,2,c6f37ce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c6f37cc4,c0f05a2b,3bb,c71bb8a0,...) at cpu_idle+0x89 sched_idletd(0,c6f37d28,c0efe310,3e5,c71b984c,...) at sched_idletd+0x263 fork_exit(c0a323a0,0,c6f37d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f37d60, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc71bbb80 sched_switch(c71bbb80,0,104,191,6d749fbf,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c71bbb80,0,c0f09280,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f09280,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c71b9b10,6c,c0f0bfff,100,0,...) at sleepq_wait_sig+0x17 _sleep(c71b9b10,c71b9b98,16c,c0f0bfff,0,...) at _sleep+0x35c kern_wait(c71bbb80,ffffffff,c6f33c60,2,0,...) at kern_wait+0x626 sys_wait4(c71bbb80,c6f33cec,c,c,282,...) at sys_wait4+0x3b syscall(c6f33d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x8060df7, esp = 0xbfbfe81c, ebp = 0xbfbfe838 --- Tracing command audit pid 10 tid 100001 td 0xc71be000 sched_switch(c71be000,0,104,191,40d423ad,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c71be000,0,c0f09280,268,c71be000,...) at sleepq_switch+0x162 sleepq_wait(c121b100,0,c6f30c8c,1,0,...) at sleepq_wait+0x63 _cv_wait(c121b100,c121b0e4,c0f33cb7,194,0,...) at _cv_wait+0x243 audit_worker(0,c6f30d28,c0efe310,3e5,c71ba000,...) at audit_worker+0x84 fork_exit(c0c07010,0,c6f30d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f30d60, ebp = 0 --- Tracing command kernel pid 0 tid 100077 td 0xc77fbb80 sched_switch(c77fbb80,0,104,191,553b7aab,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c77fbb80,0,c0f09280,268,0,...) at sleepq_switch+0x162 sleepq_wait(c77b5a80,0,c0ef6992,0,0,...) at sleepq_wait+0x63 _sleep(c77b5a80,c77b5a98,0,c0ef6992,0,...) at _sleep+0x372 taskqueue_thread_loop(c127d4a0,f17e9d28,c0efe310,3e5,c109bbe0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a4f0c0,c127d4a0,f17e9d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xf17e9d60, ebp = 0 --- Tracing command kernel pid 0 tid 100067 td 0xc755a5c0 sched_switch(c755a5c0,0,104,191,bcfcde09,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c755a5c0,0,c0f09280,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c109dee4,0,c0ef6992,2,0,...) at sleepq_timedwait+0x6b _sleep(c109dee4,0,0,c0ef6992,bb8,...) at _sleep+0x342 pause(c0ef6992,bb8,124,122,c109bc68,...) at pause+0x47 deadlkres(0,ed393d28,c0efe310,3e5,c109bbe0,...) at deadlkres+0x324 fork_exit(c09c3390,0,ed393d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed393d60, ebp = 0 --- Tracing command kernel pid 0 tid 100061 td 0xc742f8a0 sched_switch(c742f8a0,0,104,191,e7cde973,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c742f8a0,0,c0f09280,268,c795f2e0,...) at sleepq_switch+0x162 sleepq_wait(c7468980,0,c0f04781,c0ef6992,0,...) at sleepq_wait+0x63 msleep_spin(c7468980,c7468998,c0ef6992,0,c0f01fa7,...) at msleep_spin+0x21d taskqueue_thread_loop(c746449c,ed377d28,c0efe310,3e5,c109bbe0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a4f0c0,c746449c,ed377d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed377d60, ebp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xc72c5b80 sched_switch(c72c5b80,0,104,191,53c8b643,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c72c5b80,0,c0f09280,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7191580,0,c0ef6992,0,0,...) at sleepq_wait+0x63 _sleep(c7191580,c7191598,0,c0ef6992,0,...) at _sleep+0x372 taskqueue_thread_loop(c10baae8,c6f80d28,c0efe310,3e5,c109bbe0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a4f0c0,c10baae8,c6f80d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f80d60, ebp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xc73a25c0 sched_switch(c73a25c0,0,104,191,5b4746d2,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c73a25c0,0,c0f09280,268,0,...) at sleepq_switch+0x162 sleepq_wait(c71916c0,0,c0ef6992,0,0,...) at sleepq_wait+0x63 _sleep(c71916c0,c71916d8,0,c0ef6992,0,...) at _sleep+0x372 taskqueue_thread_loop(c109c578,c6f77d28,c0efe310,3e5,c109bbe0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a4f0c0,c109c578,c6f77d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f77d60, ebp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xc73a28a0 sched_switch(c73a28a0,0,104,191,5b4734fa,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c73a28a0,0,c0f09280,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7191700,0,c0f04781,c0ef6992,0,...) at sleepq_wait+0x63 msleep_spin(c7191700,c7191718,c0ef6992,0,c0f01fa7,...) at msleep_spin+0x21d taskqueue_thread_loop(c1066cb8,c6f74d28,c0efe310,3e5,c109bbe0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a4f0c0,c1066cb8,c6f74d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f74d60, ebp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xc73a2b80 sched_switch(c73a2b80,0,104,191,5b4724c2,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c73a2b80,0,c0f09280,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7191700,0,c0f04781,c0ef6992,0,...) at sleepq_wait+0x63 msleep_spin(c7191700,c7191718,c0ef6992,0,c0f01fa7,...) at msleep_spin+0x21d taskqueue_thread_loop(c1066cb8,c6f71d28,c0efe310,3e5,c109bbe0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a4f0c0,c1066cb8,c6f71d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f71d60, ebp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xc72028a0 sched_switch(c72028a0,0,104,191,5b4713dc,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c72028a0,0,c0f09280,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7191700,0,c0f04781,c0ef6992,0,...) at sleepq_wait+0x63 msleep_spin(c7191700,c7191718,c0ef6992,0,c0f01fa7,...) at msleep_spin+0x21d taskqueue_thread_loop(c1066cb8,c6f6ed28,c0efe310,3e5,c109bbe0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a4f0c0,c1066cb8,c6f6ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f6ed60, ebp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xc7202b80 sched_switch(c7202b80,0,104,191,5b40a19f,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7202b80,0,c0f09280,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7191740,0,c0ef6992,0,0,...) at sleepq_wait+0x63 _sleep(c7191740,c7191758,0,c0ef6992,0,...) at _sleep+0x372 taskqueue_thread_loop(c121b6e4,c6f6bd28,c0efe310,3e5,c109bbe0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a4f0c0,c121b6e4,c6f6bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f6bd60, ebp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xc72c48a0 sched_switch(c72c48a0,0,104,191,768bfa10,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c72c48a0,0,c0f09280,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7191bc0,0,c0ef6992,0,0,...) at sleepq_wait+0x63 _sleep(c7191bc0,c7191bd8,0,c0ef6992,0,...) at _sleep+0x372 taskqueue_thread_loop(c10b94e0,c6f5fd28,c0efe310,3e5,c109bbe0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a4f0c0,c10b94e0,c6f5fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f5fd60, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc109beb0 sched_switch(c109beb0,0,104,191,fa636bd2,...) at sched_switch+0x3bc mi_switch(104,0,c0f09280,1f3,54,...) at mi_switch+0x1eb sleepq_switch(c109beb0,0,c0f09280,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c109bbe0,54,c0f0633d,0,0,...) at sleepq_timedwait+0x6b _sleep(c109bbe0,0,54,c0f0633d,2710,...) at _sleep+0x342 scheduler(0,181ec00,181ec00,181e000,1825000,...) at scheduler+0x240 mi_startup() at mi_startup+0xac begin() at begin+0x2c db:0:allt> call doadump Physical memory: 3435 MB Physical memory: 3435 MB Dumping 188 MB:Dumping 188 MB: 173 173 157 157 141 141 125 125 109 109 93 93 77 77 61 61 45 45 29 29 13 13 Dump complete Dump complete = 0 db:0:doadump> reset (kgdb) l *ffs_sync+0x178 0xc0c4acb8 is in ffs_sync (../../../ufs/ffs/ffs_vfsops.c:1475). 1470 * for a quick test. Since there might be hundreds of 1471 * thousands of vnodes, we cannot afford even a subroutine 1472 * call unless there's a good chance that we have work to do. 1473 */ 1474 VI_LOCK(vp); 1475 if (vp->v_iflag & VI_DOOMED) { 1476 VI_UNLOCK(vp); 1477 continue; 1478 } 1479 ip = VTOI(vp); (kgdb) l *ffs_sync+0x1dd 0xc0c4ad1d is in ffs_sync (../../../ufs/ffs/ffs_vfsops.c:1483). 1478 } 1479 ip = VTOI(vp); 1480 if (vp->v_type == VNON || ((ip->i_flag & 1481 (IN_ACCESS | IN_CHANGE | IN_MODIFIED | IN_UPDATE)) == 0 && 1482 vp->v_bufobj.bo_dirty.bv_cnt == 0)) { 1483 VI_UNLOCK(vp); 1484 continue; 1485 } 1486 MNT_IUNLOCK(mp); 1487 if ((error = vget(vp, lockreq, td)) != 0) { (kgdb) $ svn diff -x -p /usr/src/sys