GENERIC HEAD from 2014-03-13 03:42:24 UTC, r263107M, vmcore.749 GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb 4718592K of memory above 4GB ignored Copyright (c) 1992-2014 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.0-CURRENT #1 r263107M: Thu Mar 13 18:08:55 CET 2014 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 FreeBSD clang version 3.4 (tags/RELEASE_34/final 197956) 20140216 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz 686-class CPU) Origin="AuthenticAMD" Id=0x100f23 Family=0x10 Model=0x2 Stepping=3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 2080555008 (1984 MB) : Trying to mount root from ufs:/dev/ad4s1a [rw]... Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Starting ddb. Entropy harvesting: interrupts ethernet point_to_point swi. warning: total configured swap (4194304 pages) exceeds maximum recommended amount (4073440 pages). warning: increase kern.maxswzone or reduce amount of swap. Starting file system checks: /dev/ad4s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1a: clean, 440410 free (2130 frags, 54785 blocks, 0.2% fragmentation) /dev/ad4s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1f: clean, 587790 free (8806 frags, 72373 blocks, 0.9% fragmentation) /dev/label/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/label/tmp: clean, 96981374 free (17662 frags, 12120464 blocks, 0.0% fragmentation) /dev/ad4s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1d: clean, 3131564 free (235860 frags, 361963 blocks, 2.3% fragmentation) /dev/ad4s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1e: clean, 7976989 free (136213 frags, 980097 blocks, 0.5% fragmentation) Mounting local file systems:Running bufdaemon at pid 8 Start thread /home worker Start thread /tmp worker Start thread /usr worker Start thread /var worker . Writing entropy file:. Setting hostname: x4.osted.lan. re0: link state changed to DOWN Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 vhid 165 nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=8209b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::221:85ff:fe15:337d%re0 prefixlen 64 scopeid 0x1 vhid 3 nd6 options=29 media: Ethernet autoselect (none) status: no carrier Starting devd. Starting pflogd: add net default: gateway 192.168.1.1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 re0: link state changed to UP ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/kde4/lib /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/gcc46 /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/pth /usr/local/lib/qt4 /usr/local/lib/virtualbox a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Creating and/or trimming log files. Starting syslogd. Starting rpcbind. NFS access cache time=60 Starting watchdogd. No core dumps found. Additional ABI support: linux. Starting casperd. lock order reversal: 1st 0xe1ded2bc bufwait (bufwait) @ kern/vfs_bio.c:3082 2nd 0xc74e0800 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper(c116bb88,4,c1175fe8,409,4000,...) at db_trace_self_wrapper+0x2d/frame 0xf2fbf768 kdb_backtrace(c116f815,c74e0800,c11a2de5,c6dae408,c11a2a2a,...) at kdb_backtrace+0x30/frame 0xf2fbf7cc witness_checkorder(c74e0800,9,c11a2a21,11c,0,...) at witness_checkorder+0xd04/frame 0xf2fbf818 _sx_xlock(c74e0800,0,c11a2a21,11c,c78cd000,...) at _sx_xlock+0x75/frame 0xf2fbf848 ufsdirhash_add(c75a815c,f2fbf940,3578,f2fbf8c8,f2fbf8cc,...) at ufsdirhash_add+0x4a/frame 0xf2fbf878 ufs_direnter(c758a870,c7f40438,f2fbf940,f2fbfb64,e1df1e08,...) at ufs_direnter+0x604/frame 0xf2fbf8f8 ufs_mkdir(f2fbfc00,c13340b3,c119bc16,c758a870,f2fbfbfc,...) at ufs_mkdir+0x8c2/frame 0xf2fbfa98 VOP_MKDIR_APV(c13f9870,f2fbfc00,f2fbfb64,f2fbfb90,28803020,...) at VOP_MKDIR_APV+0x12f/frame 0xf2fbfac8 kern_mkdirat(c7576000,ffffff9c,28803020,0,1c0) at kern_mkdirat+0x22d/frame 0xf2fbfc24 sys_mkdir(c7576000,f2fbfcc8,f2fbfc98,c0b10b66,c145ff90,...) at sys_mkdir+0x31/frame 0xf2fbfc40 syscall(f2fbfd08) at syscall+0x31b/frame 0xf2fbfcfc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf2fbfcfc --- syscall (136, FreeBSD ELF32, sys_mkdir), eip = 0x28179907, esp = 0xbfbfd924, ebp = 0xbfbfddd0 --- Clearing /tmp (X related). Starting nfsuserd. Starting mountd. Starting nfsd. Recovering vi editor sessions:. Updating motd:. Mounting late file systems:. Starting ntpd. Starting powerd. Removing stale Samba tdb files: ....... done Starting nmbd. Starting smbd. Starting winbindd. Starting smartd. /etc/rc: WARNING: failed to start smartd Starting default moused. Configuring syscons: keymap blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting sendmail_submit. Starting sendmail_msp_queue. Starting cron. Local package initialization: backuplock order reversal: 1st 0xc758a8a4 ufs (ufs) @ kern/vfs_subr.c:2101 2nd 0xe1ded2bc bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:262 3rd 0xc7fc4034 ufs (ufs) @ kern/vfs_subr.c:2101 KDB: stack backtrace: db_trace_self_wrapper(c116bb88,4000,0,200,0,...) at db_trace_self_wrapper+0x2d/frame 0xc68b66b0 kdb_backtrace(c116f82e,c7fc4034,c115681e,c6dae3a0,c1178fb6,...) at kdb_backtrace+0x30/frame 0xc68b6714 witness_checkorder(c7fc4034,9,c1178fad,835,c7fc40a0,...) at witness_checkorder+0xd04/frame 0xc68b6760 __lockmgr_args(c7fc4034,80100,c7fc40a0,0,0,...) at __lockmgr_args+0x8cd/frame 0xc68b6840 ffs_lock(c68b68c0,70c,c6d8d1e0,c6d93228,c6d8d1e0,...) at ffs_lock+0x97/frame 0xc68b687c VOP_LOCK1_APV(c13f9870,c68b68c0,e1dcfc74,c8059000,c140eda0,...) at VOP_LOCK1_APV+0x10a/frame 0xc68b68a8 _vn_lock(c7fc4000,80100,c1178fad,835,c11781f5,...) at _vn_lock+0xca/frame 0xc68b68e8 vget(c7fc4000,80100,c8059000,57,0,...) at vget+0x77/frame 0xc68b6920 vfs_hash_get(c79bf8c4,974c00,80000,c8059000,c68b6a20,...) at vfs_hash_get+0xff/frame 0xc68b694c ffs_vgetf(c79bf8c4,974c00,80000,c68b6a20,1,...) at ffs_vgetf+0x44/frame 0xc68b69a8 softdep_sync_buf(c758a870,e1ded264,1,0,0,...) at softdep_sync_buf+0x35b/frame 0xc68b6a38 ffs_syncvnode(c758a870,1,0,61b,0,...) at ffs_syncvnode+0x2dd/frame 0xc68b6a90 ffs_sync(c79bf8c4,1,c68b6b48,518,c13d3564,...) at ffs_sync+0x23c/frame 0xc68b6b10 dounmount(c79bf8c4,8080000,c8059000,494,0,...) at dounmount+0x4b8/frame 0xc68b6b70 sys_unmount(c8059000,c68b6cc8,c132cd02,d5,0,...) at sys_unmount+0x3af/frame 0xc68b6c40 syscall(c68b6d08) at syscall+0x31b/frame 0xc68b6cfc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xc68b6cfc --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280ca0db, esp = 0xbfbfd694, ebp = 0xbfbfd760 --- lock order reversal: 1st 0xc762f19c ufs (ufs) @ kern/vfs_mount.c:1237 2nd 0xc758ae44 devfs (devfs) @ ufs/ffs/ffs_softdep.c:1900 KDB: stack backtrace: db_trace_self_wrapper(c116bb88,8,c1fafc10,c1fafc00,da,...) at db_trace_self_wrapper+0x2d/frame 0xc68b6828 kdb_backtrace(c116f815,c758ae44,c11626bc,c6dae200,c119e640,...) at kdb_backtrace+0x30/frame 0xc68b688c witness_checkorder(c758ae44,9,c119e637,76c,0,...) at witness_checkorder+0xd04/frame 0xc68b68d8 __lockmgr_args(c758ae44,80400,c758aeb0,0,0,0,c119e637,76c) at __lockmgr_args+0x8cd/frame 0xc68b69b4 vop_stdlock(c68b6a28,c78ba600,c68b6a20,c0ae72bd,c78ba600,...) at vop_stdlock+0x53/frame 0xc68b69e4 VOP_LOCK1_APV(c13c34cc,c68b6a28,c68b6a50,c0d4fc73,c140eda0,...) at VOP_LOCK1_APV+0x10a/frame 0xc68b6a10 _vn_lock(c758ae10,80400,c119e637,76c,c0b9b89e,...) at _vn_lock+0xca/frame 0xc68b6a50 softdep_flushworklist(c79bf8c4,c68b6adc,c8059000,62f,0,...) at softdep_flushworklist+0x68/frame 0xc68b6a90 ffs_sync(c79bf8c4,1,c68b6b48,518,c13d3564,...) at ffs_sync+0x303/frame 0xc68b6b10 dounmount(c79bf8c4,8080000,c8059000,494,0,...) at dounmount+0x4b8/frame 0xc68b6b70 sys_unmount(c8059000,c68b6cc8,c132cd02,d5,0,...) at sys_unmount+0x3af/frame 0xc68b6c40 syscall(c68b6d08) at syscall+0x31b/frame 0xc68b6cfc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xc68b6cfc --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280ca0db, esp = 0xbfbfd694, ebp = 0xbfbfd760 --- Stop thread /tmp worker: searchfailed 0, did cleanups 0 ** /dev/label/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes Mar 13 18:16:50 x4 su: pho to root on /dev/pts/0 ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 136204 files, 21332607 used, 96981374 free (17662 frags, 12120464 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** Start thread /tmp worker fsck -y /tmp. Starting inetd. Thu Mar 13 18:18:12 CET 2014 FreeBSD/i386 (x4.osted.lan) (console) login: 20140313 18:18:20 all (1/1): strealer.sh 20140313 18:18:21 all (1/1): strealer.sh 20140313 18:18:21 all (1/1): strealer.sh 20140313 18:18:22 all (1/1): strealer.sh 20140313 18:18:22 all (1/1): strealer.sh 20140313 18:18:23 all (1/1): strealer.sh 20140313 18:18:23 all (1/1): strealer.sh 20140313 18:18:23 all (1/1): strealer.sh 20140313 18:18:24 all (1/1): strealer.sh 20140313 18:18:24 all (1/1): strealer.sh 20140313 18:18:25 all (1/1): strealer.sh 20140313 18:18:33 all (1/1): stealer.sh lock order reversal: 1st 0xc758a46c syncer (syncer) @ kern/vfs_subr.c:1720 2nd 0xc80b246c ufs (ufs) @ kern/vfs_subr.c:2101 KDB: stack backtrace: db_trace_self_wrapper(c116bb88,c6dae470,c16e0bd4,c6816998,c0b3f2c1,...) at db_trace_self_wrapper+0x2d/frame 0xc6816938 kdb_backtrace(c116f815,c80b246c,c115681e,c6dae3a0,c1178fb6,...) at kdb_backtrace+0x30/frame 0xc681699c witness_checkorder(c80b246c,9,c1178fad,835,c80b24d8,...) at witness_checkorder+0xd04/frame 0xc68169e8 __lockmgr_args(c80b246c,80500,c80b24d8,0,0,...) at __lockmgr_args+0x8cd/frame 0xc6816ac8 ffs_lock(c6816b48,c1169beb,c6d93088,c6dae470,c6d93088,...) at ffs_lock+0x97/frame 0xc6816b04 VOP_LOCK1_APV(c13f9870,c6816b48,c170c794,127e,c140eda0,...) at VOP_LOCK1_APV+0x10a/frame 0xc6816b30 _vn_lock(c80b2438,80500,c1178fad,835,c1178fad,...) at _vn_lock+0xca/frame 0xc6816b70 vget(c80b2438,80500,c7380000,128a,c6816bf8,...) at vget+0x77/frame 0xc6816ba8 vfs_msync(c79bf5d8,2,2,e47,c79bf5d8,...) at vfs_msync+0xc0/frame 0xc6816bf8 sync_fsync(c6816ca8,c1333f8e,80400,c1178fad,6b8,...) at sync_fsync+0x131/frame 0xc6816c20 VOP_FSYNC_APV(c13dd504,c6816ca8,c1178fad,6b8,0,...) at VOP_FSYNC_APV+0x12f/frame 0xc6816c50 sched_sync(0,c6816d08,c1160d7c,3c9,0,...) at sched_sync+0x3b1/frame 0xc6816ccc fork_exit(c0ba04f0,0,c6816d08) at fork_exit+0x7f/frame 0xc6816cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xc6816cf4 --- trap 0, eip = 0, esp = 0xc6816d40, ebp = 0 --- warning: total configured swap (4194304 pages) exceeds maximum recommended amount (4073440 pages). warning: increase kern.maxswzone or reduce amount of swap. 20140313 18:23:14 all (1/1): stealer.sh Mar 13 18:40:45 x4 su: pho to root on /dev/pts/1 Stopping inetd. Waiting for PIDS: 961. Shutting down local packages:. Stopping cron. Waiting for PIDS: 863. Stopping sshd. Waiting for PIDS: 852. Stopping moused. Waiting for PIDS: 802. Stopping winbindd. Waiting for PIDS: 781. Stopping smbd. Waiting for PIDS: 779. Stopping nmbd. Waiting for PIDS: 777. Stopping powerd. Waiting for PIDS: 754. Stopping ntpd. Waiting for PIDS: 751. Stopping nfsd. Waiting for PIDS: 692 700. Stopping mountd. Waiting for PIDS: 686. Stopping nfsuserd. Waiting for PIDS: 675 676 677 678 679. Stopping casperd. Waiting for PIDS: 654. Stopping watchdogd. Waiting for PIDS: 631. Stopping rpcbind. Waiting for PIDS: 622. Stopping devd. Waiting for PIDS: 491. Writing entropy file:. . Terminated Mar 13 18:40:52 x4 syslogd: exiting on signal 15 Mar 13 18:41:12 init: some processes would not die; ps axl advised Enter full pathname of shell or RETURN for /bin/sh: # ps auxwwl USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND UID PPID CPU PRI NI MWCHAN root 11 400.0 0.0 0 32 - RL 6:15PM 67:32.84 [idle] 0 0 0 155 0 - root 0 0.0 0.0 0 88 - DLs 6:15PM 0:00.15 [kernel] 0 0 0 -8 0 - root 1 0.0 0.0 9220 832 - ILs 6:15PM 0:00.07 /sbin/init -- 0 0 0 20 0 wait root 2 0.0 0.0 0 16 - DL 6:15PM 0:12.41 [cam] 0 0 0 -16 0 ccb_scan root 3 0.0 0.0 0 8 - DL 6:15PM 0:00.01 [fdc0] 0 0 0 -16 0 - root 4 0.0 0.0 0 8 - DL 6:15PM 0:00.00 [sctp_iterator] 0 0 0 -16 0 waiting_ root 5 0.0 0.0 0 8 - DL 6:15PM 0:46.99 [pagedaemon] 0 0 0 -16 0 psleep root 6 0.0 0.0 0 8 - DL 6:15PM 0:00.21 [vmdaemon] 0 0 0 -16 0 psleep root 7 0.0 0.0 0 8 - DL 6:15PM 0:00.00 [pagezero] 0 0 0 155 0 pgzero root 8 0.0 0.0 0 40 - DL 6:15PM 0:00.61 [bufdaemon] 0 0 0 20 0 sdflush root 9 0.0 0.0 0 8 - DL 6:15PM 0:00.57 [syncer] 0 0 0 16 0 syncer root 10 0.0 0.0 0 8 - DL 6:15PM 0:00.00 [audit] 0 0 0 -16 0 audit_wo root 12 0.0 0.0 0 160 - WL 6:15PM 0:06.83 [intr] 0 0 0 -84 0 - root 13 0.0 0.0 0 24 - DL 6:15PM 0:03.64 [geom] 0 0 0 -8 0 - root 14 0.0 0.0 0 8 - DL 6:15PM 0:00.87 [rand_harvestq] 0 0 0 -16 0 - root 15 0.0 0.0 0 192 - DL 6:15PM 0:00.08 [usb] 0 0 0 -68 0 - root 16 0.0 0.0 0 8 - DL 6:15PM 0:00.01 [vnlru] 0 0 0 -16 0 vlruwt root 2844 0.0 0.0 0 8 - DL 6:23PM 0:12.63 [md5] 0 0 0 -8 0 mdwait root 3600 0.0 0.1 10376 2652 u0 Ss 6:41PM 0:00.01 -sh (sh) 0 1 0 20 0 wait root 3601 0.0 0.1 10092 1844 u0 R+ 6:41PM 0:00.00 ps auxwwl 0 3600 0 20 0 - stress 2882 0.0 0.0 0 8 0- DE 6:23PM 0:00.14 /tmp/stealer 479 1004 1 0 20 0 objtrm # kill -9 2882 # kill -9 2882 # ~KDB: enter: Break to debugger [ thread pid 11 tid 100005 ] Stopped at kdb_break+0x4e: movl $0,kdb_why db> run pho db:0:pho> bt Tracing pid 11 tid 100005 td 0xc6f08930 kdb_break(c7054600,c132e287,369,c6f08930,c6f08930,...) at kdb_break+0x4e/frame 0xeb106ae8 uart_intr(c7054600,a,c1168da1,2,0,...) at uart_intr+0x91/frame 0xeb106b24 intr_event_handle(c6f51e00,eb106ba0,4a9cc0ab,eb106ba0,30,...) at intr_event_handle+0x7a/frame 0xeb106b50 intr_execute_handlers(c6ececd4,eb106ba0,2) at intr_execute_handlers+0x42/frame 0xeb106b70 lapic_handle_intr(30,eb106ba0) at lapic_handle_intr+0x6f/frame 0xeb106b90 Xapic_isr1() at Xapic_isr1+0x35/frame 0xeb106b90 --- interrupt, eip = 0xc0fdda15, esp = 0xeb106be0, ebp = 0xeb106be0 --- acpi_cpu_c1(eb0f3c90,0,c132e287,303,eb0f3c80,...) at acpi_cpu_c1+0x5/frame 0xeb106be0 acpi_cpu_idle(4a9cc0ab,0,4a9cc0ab,eb106c50,c0fe7efa,...) at acpi_cpu_idle+0x142/frame 0xeb106c20 cpu_idle_acpi(4a9cc0ab,0,c1460708,c1460714,c1460704,...) at cpu_idle_acpi+0x3f/frame 0xeb106c34 cpu_idle(0,2,c1169beb,3f4,c1460708,...) at cpu_idle+0x9a/frame 0xeb106c50 sched_idletd(0,eb106d08,c1160d7c,3c9,0,...) at sched_idletd+0x1c8/frame 0xeb106ccc fork_exit(c0b13070,0,eb106d08) at fork_exit+0x7f/frame 0xeb106cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb106cf4 --- trap 0, eip = 0, esp = 0xeb106d40, ebp = 0 --- db:0:bt> show allpcpu Current CPU: 2 cpuid = 0 dynamic pcpu = 0x813800 curthread = 0xc6f09000: pid 11 "idle: cpu0" curpcb = 0xeb100d60 fpcurthread = none idlethread = 0xc6f09000: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 dynamic pcpu = 0x29cde800 curthread = 0xc6f08c40: pid 11 "idle: cpu1" curpcb = 0xeb103d60 fpcurthread = none idlethread = 0xc6f08c40: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: cpuid = 2 dynamic pcpu = 0x29ce1800 curthread = 0xc6f08930: pid 11 "idle: cpu2" curpcb = 0xeb106d60 fpcurthread = none idlethread = 0xc6f08930: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 spin locks held: cpuid = 3 dynamic pcpu = 0x29ce4800 curthread = 0xc6f08620: pid 11 "idle: cpu3" curpcb = 0xeb109d60 fpcurthread = none idlethread = 0xc6f08620: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 spin locks held: db:0:allpcpu> show alllocks db:0:alllocks> show lockedvnods Locked vnodes db:0:lockedvnods> show mount 0xc75202ec /dev/ad4s1a on / (ufs) 0xc75205d8 devfs on /dev (devfs) 0xc78e4000 /dev/ad4s1f on /home (ufs) 0xc79bf5d8 /dev/ad4s1d on /usr (ufs) 0xc79bf2ec /dev/ad4s1e on /var (ufs) 0xc7520000 procfs on /proc (procfs) 0xc751fbb0 /dev/label/tmp on /tmp (ufs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 3600 1 3600 0 Ss+ ttyin 0xc7054470 sh 2882 1 2855 1004 DE+ objtrm 0xc95a8a5c stealer 2844 0 0 0 DL mdwait 0xc9643800 [md5] 16 0 0 0 DL vlruwt 0xc74598c4 [vnlru] 9 0 0 0 DL syncer 0xc170c7dc [syncer] 8 0 0 0 DL (threaded) [bufdaemon] 100169 D sdflush 0xc78ba084 [/tmp worker] 100087 D sdflush 0xc789f284 [/var worker] 100086 D sdflush 0xc78ba484 [/usr worker] 100084 D sdflush 0xc7052284 [/home worker] 100072 D psleep 0xc170c304 [bufdaemon] 7 0 0 0 DL pgzero 0xc17116a0 [pagezero] 6 0 0 0 DL psleep 0xc17113c4 [vmdaemon] 5 0 0 0 DL psleep 0xc1770604 [pagedaemon] 4 0 0 0 DL waiting_ 0xc176d03c [sctp_iterator] 3 0 0 0 DL - 0xc705ee3c [fdc0] 15 0 0 0 DL (threaded) [usb] 100058 D - 0xc7158d34 [usbus5] 100057 D - 0xc7158d04 [usbus5] 100056 D - 0xc7158cd4 [usbus5] 100055 D - 0xc7158ca4 [usbus5] 100053 D - 0xc714eb5c [usbus4] 100052 D - 0xc714eb2c [usbus4] 100051 D - 0xc714eafc [usbus4] 100050 D - 0xc714eacc [usbus4] 100049 D - 0xc7148b5c [usbus3] 100048 D - 0xc7148b2c [usbus3] 100047 D - 0xc7148afc [usbus3] 100046 D - 0xc7148acc [usbus3] 100045 D - 0xc7145b5c [usbus2] 100044 D - 0xc7145b2c [usbus2] 100043 D - 0xc7145afc [usbus2] 100042 D - 0xc7145acc [usbus2] 100040 D - 0xc713cb5c [usbus1] 100039 D - 0xc713cb2c [usbus1] 100038 D - 0xc713cafc [usbus1] 100037 D - 0xc713cacc [usbus1] 100035 D - 0xc7135b5c [usbus0] 100034 D - 0xc7135b2c [usbus0] 100033 D - 0xc7135afc [usbus0] 100032 D - 0xc7135acc [usbus0] 2 0 0 0 DL (threaded) [cam] 100068 D ccb_scan 0xc14126a8 [scanner] 100019 D - 0xc1412800 [doneq0] 14 0 0 0 DL - 0xc142c438 [rand_harvestq] 13 0 0 0 DL (threaded) [geom] 100015 D - 0xc17691e4 [g_down] 100014 D - 0xc17691e0 [g_up] 100013 D - 0xc17691dc [g_event] 12 0 0 0 WL (threaded) [intr] 100063 I [irq12: psm0] 100062 I [irq1: atkbd0] 100060 I [swi0: uart] 100059 I [irq14: ata0] 100054 I [irq19: ehci0] 100041 I [irq18: ohci2 ohci4] 100036 I [irq17: ohci1 ohci3] 100031 I [irq16: hdac1 ohci0] 100030 I [irq22: ahci0] 100029 I [irq257: re0] 100028 I [irq256: hdac0] 100026 I [swi5: fast taskq] 100024 I [swi6: Giant taskq] 100023 I [swi6: task queue] 100012 I [swi3: vm] 100011 I [swi4: clock (3)] 100010 I [swi4: clock (2)] 100009 I [swi4: clock (1)] 100008 I [swi4: clock (0)] 100007 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100006 Run CPU 3 [idle: cpu3] 100005 Run CPU 2 [idle: cpu2] 100004 Run CPU 1 [idle: cpu1] 100003 Run CPU 0 [idle: cpu0] 1 0 1 0 SLs wait 0xc6f012ec [init] 10 0 0 0 DL audit_wo 0xc176ef18 [audit] 0 0 0 0 DLs (threaded) [kernel] 100067 D - 0xc6fcae80 [CAM taskq] 100065 D - 0xc145252f [deadlkres] 100064 D - 0xc6fca380 [mca taskq] 100027 D - 0xc6fcab00 [ffs_trim taskq] 100025 D - 0xc6fcac00 [thread taskq] 100022 D - 0xc6fcad80 [acpi_task_2] 100021 D - 0xc6fcad80 [acpi_task_1] 100020 D - 0xc6fcad80 [acpi_task_0] 100018 D - 0xc6fcaf00 [kqueue taskq] 100016 D - 0xc6fcb000 [firmware taskq] 100000 D swapin 0xc1769264 [swapper] db:0:ps> allt Tracing command sh pid 3600 tid 100157 td 0xc846b310 sched_switch(c846b310,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xc68c099c mi_switch(104,0,c116d801,1f1,0,...) at mi_switch+0x1c3/frame 0xc68c09d0 sleepq_switch(c846b310,0,c116d801,1a7,b0,...) at sleepq_switch+0x1c5/frame 0xc68c0a04 sleepq_catch_signals(c68c0a6c,c0ad3809,c7054414,0,0,...) at sleepq_catch_signals+0x3ce/frame 0xc68c0a48 sleepq_wait_sig(c7054470,0,c68c0aa0,101,0,...) at sleepq_wait_sig+0x14/frame 0xc68c0a6c _cv_wait_sig(c7054470,c7054404,c1172e31,558,1,...) at _cv_wait_sig+0x26e/frame 0xc68c0ab4 tty_wait(c7054400,c7054470,c68c0c10,1,0,...) at tty_wait+0x4b/frame 0xc68c0ad8 ttydisc_read(c7054400,c68c0c10,0,a1,0,...) at ttydisc_read+0x39d/frame 0xc68c0b1c ttydev_read(c6fb8e00,c68c0c10,0,1,0,...) at ttydev_read+0x81/frame 0xc68c0b3c devfs_read_f(c758d348,c68c0c10,c6ed4b80,0,c846b310,...) at devfs_read_f+0x85/frame 0xc68c0b70 dofileread(c758d348,c68c0c10,ffffffff,ffffffff,0,...) at dofileread+0x9e/frame 0xc68c0ba8 kern_readv(c846b310,0,c68c0c10,0,bfbfddab,...) at kern_readv+0x96/frame 0xc68c0bf0 sys_read(c846b310,c68c0cc8,c100085f,eb0f3c90,0,...) at sys_read+0x5c/frame 0xc68c0c40 syscall(c68c0d08) at syscall+0x31b/frame 0xc68c0cfc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xc68c0cfc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x282081d7, esp = 0xbfbfdd5c, ebp = 0xbfbfdd7c --- Tracing command stealer pid 2882 tid 100202 td 0xc8bfc000 sched_switch(c8bfc000,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xc69cb86c mi_switch(104,0,c116d801,1f1,c95a8a5c,...) at mi_switch+0x1c3/frame 0xc69cb8a0 sleepq_switch(c8bfc000,0,c116d801,267,c95a8a5c,...) at sleepq_switch+0x1c5/frame 0xc69cb8d4 sleepq_wait(c95a8a5c,54,c11a66d4,0,0,...) at sleepq_wait+0x48/frame 0xc69cb8f8 _sleep(c95a8a5c,c95a8a5c,54,c11a66d4,0,...) at _sleep+0x391/frame 0xc69cb954 vm_object_pip_wait(c95a8a5c,c11a66d4,c11a65b5,2c1,246,...) at vm_object_pip_wait+0xb5/frame 0xc69cb9a4 vm_object_terminate(c95a8a5c,c11a65b5,201,c69cba50,c0d91803,...) at vm_object_terminate+0x78/frame 0xc69cba18 vm_object_deallocate(c95a8a5c,c7742474,c1fe0000,0,c9554000,...) at vm_object_deallocate+0x8f4/frame 0xc69cbadc vm_map_entry_deallocate(c9554000,0,b80,0,b7d,...) at vm_map_entry_deallocate+0x36/frame 0xc69cbaf8 vm_map_process_deferred(c755ec7c,c11a564f,b80,bfc00000,1000,b80,c11a564f,c755ec30) at vm_map_process_deferred+0x105/frame 0xc69cbb20 _vm_map_unlock(c755ec30,c11a564f,b80,b80,bfc00000,...) at _vm_map_unlock+0x86/frame 0xc69cbb4c vm_map_remove(c755ec30,1000,bfc00000,c8ad5bb0,c755ec30,...) at vm_map_remove+0xe6/frame 0xc69cbb88 vmspace_dofree(c755ec30,1,0,1,c8ad5bb0,...) at vmspace_dofree+0x36/frame 0xc69cbba4 vmspace_exit(c8bfc000,0,c1160af8,13d,2000,...) at vmspace_exit+0x161/frame 0xc69cbbd0 exit1(c8bfc000,0,c69cbcfc,c0ff9aeb,c8bfc000,...) at exit1+0x63e/frame 0xc69cbc30 sys_sys_exit(c8bfc000,c69cbcc8,0,ab8,c69cbccc,...) at sys_sys_exit+0x1d/frame 0xc69cbc40 syscall(c69cbd08) at syscall+0x31b/frame 0xc69cbcfc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xc69cbcfc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0x28124793, esp = 0xbfbfd4e8, ebp = 0xbfbfd4f8 --- Tracing command md5 pid 2844 tid 100300 td 0xc9573310 sched_switch(c9573310,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xc6b27ba0 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xc6b27bd4 sleepq_switch(c9573310,0,c116d801,267,c9643800,...) at sleepq_switch+0x1c5/frame 0xc6b27c08 sleepq_wait(c9643800,5c,c10f910a,0,0,...) at sleepq_wait+0x48/frame 0xc6b27c2c _sleep(c9643800,c9643820,25c,c10f910a,0,...) at _sleep+0x391/frame 0xc6b27c88 md_kthread(c9643800,c6b27d08,c1160d7c,3c9,0,...) at md_kthread+0x118/frame 0xc6b27ccc fork_exit(c0798c30,c9643800,c6b27d08) at fork_exit+0x7f/frame 0xc6b27cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xc6b27cf4 --- trap 0, eip = 0, esp = 0xc6b27d40, ebp = 0 --- Tracing command vnlru pid 16 tid 100074 td 0xc737fc40 sched_switch(c737fc40,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xc6819b78 mi_switch(104,0,c116d801,1f1,101,...) at mi_switch+0x1c3/frame 0xc6819bac sleepq_switch(c737fc40,0,c116d801,28a,c74598c4,...) at sleepq_switch+0x1c5/frame 0xc6819be0 sleepq_timedwait(c74598c4,60,0,0,0,...) at sleepq_timedwait+0x48/frame 0xc6819c04 _sleep(c74598c4,c170c794,260,c117a38d,fffffed8,...) at _sleep+0x35e/frame 0xc6819c60 vnlru_proc(0,c6819d08,c1160d7c,3c9,0,...) at vnlru_proc+0xcf/frame 0xc6819ccc fork_exit(c0ba0d90,0,c6819d08) at fork_exit+0x7f/frame 0xc6819cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xc6819cf4 --- trap 0, eip = 0, esp = 0xc6819d40, ebp = 0 --- Tracing command syncer pid 9 tid 100073 td 0xc7380000 sched_switch(c7380000,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xc6816b78 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xc6816bac sleepq_switch(c7380000,0,c116d801,28a,0,...) at sleepq_switch+0x1c5/frame 0xc6816be0 sleepq_timedwait(c170c7dc,0,c6816c3c,0,0,...) at sleepq_timedwait+0x48/frame 0xc6816c04 _cv_timedwait_sbt(c170c7dc,c170c7bc,fffffed8,0,0,...) at _cv_timedwait_sbt+0x299/frame 0xc6816c50 sched_sync(0,c6816d08,c1160d7c,3c9,0,...) at sched_sync+0x67d/frame 0xc6816ccc fork_exit(c0ba04f0,0,c6816d08) at fork_exit+0x7f/frame 0xc6816cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xc6816cf4 --- trap 0, eip = 0, esp = 0xc6816d40, ebp = 0 --- Tracing command bufdaemon pid 8 tid 100169 td 0xc7658930 sched_switch(c7658930,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xc68e3ba0 mi_switch(104,0,c116d801,1f1,101,...) at mi_switch+0x1c3/frame 0xc68e3bd4 sleepq_switch(c7658930,0,c116d801,28a,c78ba084,...) at sleepq_switch+0x1c5/frame 0xc68e3c08 sleepq_timedwait(c78ba084,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xc68e3c2c _sleep(c78ba084,c78ba000,54,c11a184b,7fffff6c,...) at _sleep+0x35e/frame 0xc68e3c88 softdep_flush(c751fbb0,c68e3d08,c1160d7c,3c9,0,...) at softdep_flush+0x152/frame 0xc68e3ccc fork_exit(c0d53a80,c751fbb0,c68e3d08) at fork_exit+0x7f/frame 0xc68e3cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xc68e3cf4 --- trap 0, eip = 0, esp = 0xc68e3d40, ebp = 0 --- Tracing command bufdaemon pid 8 tid 100087 td 0xc7576930 sched_switch(c7576930,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xf2f40ba0 mi_switch(104,0,c116d801,1f1,101,...) at mi_switch+0x1c3/frame 0xf2f40bd4 sleepq_switch(c7576930,0,c116d801,28a,c789f284,...) at sleepq_switch+0x1c5/frame 0xf2f40c08 sleepq_timedwait(c789f284,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xf2f40c2c _sleep(c789f284,c789f200,54,c11a184b,7fffff6c,...) at _sleep+0x35e/frame 0xf2f40c88 softdep_flush(c79bf2ec,f2f40d08,c1160d7c,3c9,0,...) at softdep_flush+0x152/frame 0xf2f40ccc fork_exit(c0d53a80,c79bf2ec,f2f40d08) at fork_exit+0x7f/frame 0xf2f40cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xf2f40cf4 --- trap 0, eip = 0, esp = 0xf2f40d40, ebp = 0 --- Tracing command bufdaemon pid 8 tid 100086 td 0xc765f930 sched_switch(c765f930,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xf2f3dba0 mi_switch(104,0,c116d801,1f1,101,...) at mi_switch+0x1c3/frame 0xf2f3dbd4 sleepq_switch(c765f930,0,c116d801,28a,c78ba484,...) at sleepq_switch+0x1c5/frame 0xf2f3dc08 sleepq_timedwait(c78ba484,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xf2f3dc2c _sleep(c78ba484,c78ba400,54,c11a184b,7fffff6c,...) at _sleep+0x35e/frame 0xf2f3dc88 softdep_flush(c79bf5d8,f2f3dd08,c1160d7c,3c9,0,...) at softdep_flush+0x152/frame 0xf2f3dccc fork_exit(c0d53a80,c79bf5d8,f2f3dd08) at fork_exit+0x7f/frame 0xf2f3dcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xf2f3dcf4 --- trap 0, eip = 0, esp = 0xf2f3dd40, ebp = 0 --- Tracing command bufdaemon pid 8 tid 100084 td 0xc737f000 sched_switch(c737f000,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xf2f37ba0 mi_switch(104,0,c116d801,1f1,101,...) at mi_switch+0x1c3/frame 0xf2f37bd4 sleepq_switch(c737f000,0,c116d801,28a,c7052284,...) at sleepq_switch+0x1c5/frame 0xf2f37c08 sleepq_timedwait(c7052284,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xf2f37c2c _sleep(c7052284,c7052200,54,c11a184b,7fffff6c,...) at _sleep+0x35e/frame 0xf2f37c88 softdep_flush(c78e4000,f2f37d08,c1160d7c,3c9,0,...) at softdep_flush+0x152/frame 0xf2f37ccc fork_exit(c0d53a80,c78e4000,f2f37d08) at fork_exit+0x7f/frame 0xf2f37cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xf2f37cf4 --- trap 0, eip = 0, esp = 0xf2f37d40, ebp = 0 --- Tracing command bufdaemon pid 8 tid 100072 td 0xc7380310 sched_switch(c7380310,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb3fdbb0 mi_switch(104,0,c116d801,1f1,101,...) at mi_switch+0x1c3/frame 0xeb3fdbe4 sleepq_switch(c7380310,0,c116d801,28a,c170c304,...) at sleepq_switch+0x1c5/frame 0xeb3fdc18 sleepq_timedwait(c170c304,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xeb3fdc3c _sleep(c170c304,c170c280,54,c1176cd2,fffffed8,...) at _sleep+0x35e/frame 0xeb3fdc98 buf_daemon(0,eb3fdd08,c1160d7c,3c9,0,...) at buf_daemon+0x8c/frame 0xeb3fdccc fork_exit(c0b832c0,0,eb3fdd08) at fork_exit+0x7f/frame 0xeb3fdcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb3fdcf4 --- trap 0, eip = 0, esp = 0xeb3fdd40, ebp = 0 --- Tracing command pagezero pid 7 tid 100071 td 0xc7380620 sched_switch(c7380620,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb3fabac mi_switch(104,0,c116d801,1f1,101,...) at mi_switch+0x1c3/frame 0xeb3fabe0 sleepq_switch(c7380620,0,c116d801,28a,c17116a0,...) at sleepq_switch+0x1c5/frame 0xeb3fac14 sleepq_timedwait(c17116a0,0,12b,0,0,...) at sleepq_timedwait+0x48/frame 0xeb3fac38 _sleep(c17116a0,c1770580,0,c11a9150,fffea520,...) at _sleep+0x35e/frame 0xeb3fac94 vm_pagezero(0,eb3fad08,c1160d7c,3c9,0,...) at vm_pagezero+0xc2/frame 0xeb3faccc fork_exit(c0dbaec0,0,eb3fad08) at fork_exit+0x7f/frame 0xeb3facf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb3facf4 --- trap 0, eip = 0, esp = 0xeb3fad40, ebp = 0 --- Tracing command vmdaemon pid 6 tid 100070 td 0xc7380930 sched_switch(c7380930,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb3f7b78 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb3f7bac sleepq_switch(c7380930,0,c116d801,267,c17113c4,...) at sleepq_switch+0x1c5/frame 0xeb3f7be0 sleepq_wait(c17113c4,74,c1176cd2,0,0,...) at sleepq_wait+0x48/frame 0xeb3f7c04 _sleep(c17113c4,c17113ac,74,c1176cd2,0,...) at _sleep+0x391/frame 0xeb3f7c60 vm_daemon(0,eb3f7d08,c1160d7c,3c9,0,...) at vm_daemon+0xac/frame 0xeb3f7ccc fork_exit(c0db2fb0,0,eb3f7d08) at fork_exit+0x7f/frame 0xeb3f7cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb3f7cf4 --- trap 0, eip = 0, esp = 0xeb3f7d40, ebp = 0 --- Tracing command pagedaemon pid 5 tid 100069 td 0xc7380c40 sched_switch(c7380c40,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb3f4b7c mi_switch(104,0,c116d801,1f1,101,...) at mi_switch+0x1c3/frame 0xeb3f4bb0 sleepq_switch(c7380c40,0,c116d801,28a,c1770604,...) at sleepq_switch+0x1c5/frame 0xeb3f4be4 sleepq_timedwait(c1770604,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xeb3f4c08 _sleep(c1770604,c1770580,54,c1176cd2,fffffed8,...) at _sleep+0x35e/frame 0xeb3f4c64 vm_pageout(0,eb3f4d08,c1160d7c,3c9,0,...) at vm_pageout+0x244/frame 0xeb3f4ccc fork_exit(c0db37e0,0,eb3f4d08) at fork_exit+0x7f/frame 0xeb3f4cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb3f4cf4 --- trap 0, eip = 0, esp = 0xeb3f4d40, ebp = 0 --- Tracing command sctp_iterator pid 4 tid 100066 td 0xc7381620 sched_switch(c7381620,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb26bbb8 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb26bbec sleepq_switch(c7381620,0,c116d801,267,c176d03c,...) at sleepq_switch+0x1c5/frame 0xeb26bc20 sleepq_wait(c176d03c,0,c118daf1,0,0,...) at sleepq_wait+0x48/frame 0xeb26bc44 _sleep(c176d03c,c176d004,0,c118daf1,0,...) at _sleep+0x391/frame 0xeb26bca0 sctp_iterator_thread(0,eb26bd08,c1160d7c,3c9,0,...) at sctp_iterator_thread+0x7c/frame 0xeb26bccc fork_exit(c0c443e0,0,eb26bd08) at fork_exit+0x7f/frame 0xeb26bcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb26bcf4 --- trap 0, eip = 0, esp = 0xeb26bd40, ebp = 0 --- Tracing command fdc0 pid 3 tid 100061 td 0xc7150310 sched_switch(c7150310,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb25bb4c mi_switch(104,0,c116d801,1f1,101,...) at mi_switch+0x1c3/frame 0xeb25bb80 sleepq_switch(c7150310,0,c116d801,28a,c705ee3c,...) at sleepq_switch+0x1c5/frame 0xeb25bbb4 sleepq_timedwait(c705ee3c,5c,0,0,0,...) at sleepq_timedwait+0x48/frame 0xeb25bbd8 _sleep(c705ee3c,c705eef0,5c,c1152ed6,fffffed8,...) at _sleep+0x35e/frame 0xeb25bc34 fdc_thread(c705ee00,eb25bd08,c1160d7c,3c9,0,...) at fdc_thread+0x80a/frame 0xeb25bccc fork_exit(c0f719c0,c705ee00,eb25bd08) at fork_exit+0x7f/frame 0xeb25bcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb25bcf4 --- trap 0, eip = 0, esp = 0xeb25bd40, ebp = 0 --- Tracing command usb pid 15 tid 100058 td 0xc7150c40 sched_switch(c7150c40,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb237bd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb237c08 sleepq_switch(c7150c40,0,c116d801,267,c7158d34,...) at sleepq_switch+0x1c5/frame 0xeb237c3c sleepq_wait(c7158d34,0,eb237c90,1,0,...) at sleepq_wait+0x48/frame 0xeb237c60 _cv_wait(c7158d34,c7158e74,c113dab7,77,c7158d3c,...) at _cv_wait+0x263/frame 0xeb237ca4 usb_process(c7158d2c,eb237d08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb237ccc fork_exit(c0974000,c7158d2c,eb237d08) at fork_exit+0x7f/frame 0xeb237cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb237cf4 --- trap 0, eip = 0, esp = 0xeb237d40, ebp = 0 --- Tracing command usb pid 15 tid 100057 td 0xc7151000 sched_switch(c7151000,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb234bd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb234c08 sleepq_switch(c7151000,0,c116d801,267,c7158d04,...) at sleepq_switch+0x1c5/frame 0xeb234c3c sleepq_wait(c7158d04,0,eb234c90,1,0,...) at sleepq_wait+0x48/frame 0xeb234c60 _cv_wait(c7158d04,c7158e74,c113dab7,77,c7158d0c,...) at _cv_wait+0x263/frame 0xeb234ca4 usb_process(c7158cfc,eb234d08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb234ccc fork_exit(c0974000,c7158cfc,eb234d08) at fork_exit+0x7f/frame 0xeb234cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb234cf4 --- trap 0, eip = 0, esp = 0xeb234d40, ebp = 0 --- Tracing command usb pid 15 tid 100056 td 0xc7151310 sched_switch(c7151310,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb231bd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb231c08 sleepq_switch(c7151310,0,c116d801,267,c7158cd4,...) at sleepq_switch+0x1c5/frame 0xeb231c3c sleepq_wait(c7158cd4,0,eb231c90,1,0,...) at sleepq_wait+0x48/frame 0xeb231c60 _cv_wait(c7158cd4,c7158e74,c113dab7,77,c7158cdc,...) at _cv_wait+0x263/frame 0xeb231ca4 usb_process(c7158ccc,eb231d08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb231ccc fork_exit(c0974000,c7158ccc,eb231d08) at fork_exit+0x7f/frame 0xeb231cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb231cf4 --- trap 0, eip = 0, esp = 0xeb231d40, ebp = 0 --- Tracing command usb pid 15 tid 100055 td 0xc7151620 sched_switch(c7151620,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb22ebd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb22ec08 sleepq_switch(c7151620,0,c116d801,267,c7158ca4,...) at sleepq_switch+0x1c5/frame 0xeb22ec3c sleepq_wait(c7158ca4,0,eb22ec90,1,0,...) at sleepq_wait+0x48/frame 0xeb22ec60 _cv_wait(c7158ca4,c7158e74,c113dab7,77,c7158cac,...) at _cv_wait+0x263/frame 0xeb22eca4 usb_process(c7158c9c,eb22ed08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb22eccc fork_exit(c0974000,c7158c9c,eb22ed08) at fork_exit+0x7f/frame 0xeb22ecf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb22ecf4 --- trap 0, eip = 0, esp = 0xeb22ed40, ebp = 0 --- Tracing command usb pid 15 tid 100053 td 0xc7151c40 sched_switch(c7151c40,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb225bd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb225c08 sleepq_switch(c7151c40,0,c116d801,267,c714eb5c,...) at sleepq_switch+0x1c5/frame 0xeb225c3c sleepq_wait(c714eb5c,0,eb225c90,1,0,...) at sleepq_wait+0x48/frame 0xeb225c60 _cv_wait(c714eb5c,c714ec9c,c113dab7,77,c714eb64,...) at _cv_wait+0x263/frame 0xeb225ca4 usb_process(c714eb54,eb225d08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb225ccc fork_exit(c0974000,c714eb54,eb225d08) at fork_exit+0x7f/frame 0xeb225cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb225cf4 --- trap 0, eip = 0, esp = 0xeb225d40, ebp = 0 --- Tracing command usb pid 15 tid 100052 td 0xc70bbc40 sched_switch(c70bbc40,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb222bd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb222c08 sleepq_switch(c70bbc40,0,c116d801,267,c714eb2c,...) at sleepq_switch+0x1c5/frame 0xeb222c3c sleepq_wait(c714eb2c,0,eb222c90,1,0,...) at sleepq_wait+0x48/frame 0xeb222c60 _cv_wait(c714eb2c,c714ec9c,c113dab7,77,c714eb34,...) at _cv_wait+0x263/frame 0xeb222ca4 usb_process(c714eb24,eb222d08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb222ccc fork_exit(c0974000,c714eb24,eb222d08) at fork_exit+0x7f/frame 0xeb222cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb222cf4 --- trap 0, eip = 0, esp = 0xeb222d40, ebp = 0 --- Tracing command usb pid 15 tid 100051 td 0xc7142000 sched_switch(c7142000,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb21fbd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb21fc08 sleepq_switch(c7142000,0,c116d801,267,c714eafc,...) at sleepq_switch+0x1c5/frame 0xeb21fc3c sleepq_wait(c714eafc,0,eb21fc90,1,0,...) at sleepq_wait+0x48/frame 0xeb21fc60 _cv_wait(c714eafc,c714ec9c,c113dab7,77,c714eb04,...) at _cv_wait+0x263/frame 0xeb21fca4 usb_process(c714eaf4,eb21fd08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb21fccc fork_exit(c0974000,c714eaf4,eb21fd08) at fork_exit+0x7f/frame 0xeb21fcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb21fcf4 --- trap 0, eip = 0, esp = 0xeb21fd40, ebp = 0 --- Tracing command usb pid 15 tid 100050 td 0xc7142310 sched_switch(c7142310,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb21cbd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb21cc08 sleepq_switch(c7142310,0,c116d801,267,c714eacc,...) at sleepq_switch+0x1c5/frame 0xeb21cc3c sleepq_wait(c714eacc,0,eb21cc90,1,0,...) at sleepq_wait+0x48/frame 0xeb21cc60 _cv_wait(c714eacc,c714ec9c,c113dab7,77,c714ead4,...) at _cv_wait+0x263/frame 0xeb21cca4 usb_process(c714eac4,eb21cd08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb21cccc fork_exit(c0974000,c714eac4,eb21cd08) at fork_exit+0x7f/frame 0xeb21ccf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb21ccf4 --- trap 0, eip = 0, esp = 0xeb21cd40, ebp = 0 --- Tracing command usb pid 15 tid 100049 td 0xc7142620 sched_switch(c7142620,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb218bd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb218c08 sleepq_switch(c7142620,0,c116d801,267,c7148b5c,...) at sleepq_switch+0x1c5/frame 0xeb218c3c sleepq_wait(c7148b5c,0,eb218c90,1,0,...) at sleepq_wait+0x48/frame 0xeb218c60 _cv_wait(c7148b5c,c7148c9c,c113dab7,77,c7148b64,...) at _cv_wait+0x263/frame 0xeb218ca4 usb_process(c7148b54,eb218d08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb218ccc fork_exit(c0974000,c7148b54,eb218d08) at fork_exit+0x7f/frame 0xeb218cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb218cf4 --- trap 0, eip = 0, esp = 0xeb218d40, ebp = 0 --- Tracing command usb pid 15 tid 100048 td 0xc7142930 sched_switch(c7142930,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb215bd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb215c08 sleepq_switch(c7142930,0,c116d801,267,c7148b2c,...) at sleepq_switch+0x1c5/frame 0xeb215c3c sleepq_wait(c7148b2c,0,eb215c90,1,0,...) at sleepq_wait+0x48/frame 0xeb215c60 _cv_wait(c7148b2c,c7148c9c,c113dab7,77,c7148b34,...) at _cv_wait+0x263/frame 0xeb215ca4 usb_process(c7148b24,eb215d08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb215ccc fork_exit(c0974000,c7148b24,eb215d08) at fork_exit+0x7f/frame 0xeb215cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb215cf4 --- trap 0, eip = 0, esp = 0xeb215d40, ebp = 0 --- Tracing command usb pid 15 tid 100047 td 0xc7142c40 sched_switch(c7142c40,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb212bd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb212c08 sleepq_switch(c7142c40,0,c116d801,267,c7148afc,...) at sleepq_switch+0x1c5/frame 0xeb212c3c sleepq_wait(c7148afc,0,eb212c90,1,0,...) at sleepq_wait+0x48/frame 0xeb212c60 _cv_wait(c7148afc,c7148c9c,c113dab7,77,c7148b04,...) at _cv_wait+0x263/frame 0xeb212ca4 usb_process(c7148af4,eb212d08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb212ccc fork_exit(c0974000,c7148af4,eb212d08) at fork_exit+0x7f/frame 0xeb212cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb212cf4 --- trap 0, eip = 0, esp = 0xeb212d40, ebp = 0 --- Tracing command usb pid 15 tid 100046 td 0xc7143000 sched_switch(c7143000,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb20fbd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb20fc08 sleepq_switch(c7143000,0,c116d801,267,c7148acc,...) at sleepq_switch+0x1c5/frame 0xeb20fc3c sleepq_wait(c7148acc,0,eb20fc90,1,0,...) at sleepq_wait+0x48/frame 0xeb20fc60 _cv_wait(c7148acc,c7148c9c,c113dab7,77,c7148ad4,...) at _cv_wait+0x263/frame 0xeb20fca4 usb_process(c7148ac4,eb20fd08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb20fccc fork_exit(c0974000,c7148ac4,eb20fd08) at fork_exit+0x7f/frame 0xeb20fcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb20fcf4 --- trap 0, eip = 0, esp = 0xeb20fd40, ebp = 0 --- Tracing command usb pid 15 tid 100045 td 0xc7143310 sched_switch(c7143310,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb20bbd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb20bc08 sleepq_switch(c7143310,0,c116d801,267,c7145b5c,...) at sleepq_switch+0x1c5/frame 0xeb20bc3c sleepq_wait(c7145b5c,0,eb20bc90,1,0,...) at sleepq_wait+0x48/frame 0xeb20bc60 _cv_wait(c7145b5c,c7145c9c,c113dab7,77,c7145b64,...) at _cv_wait+0x263/frame 0xeb20bca4 usb_process(c7145b54,eb20bd08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb20bccc fork_exit(c0974000,c7145b54,eb20bd08) at fork_exit+0x7f/frame 0xeb20bcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb20bcf4 --- trap 0, eip = 0, esp = 0xeb20bd40, ebp = 0 --- Tracing command usb pid 15 tid 100044 td 0xc7143620 sched_switch(c7143620,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb208bd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb208c08 sleepq_switch(c7143620,0,c116d801,267,c7145b2c,...) at sleepq_switch+0x1c5/frame 0xeb208c3c sleepq_wait(c7145b2c,0,eb208c90,1,0,...) at sleepq_wait+0x48/frame 0xeb208c60 _cv_wait(c7145b2c,c7145c9c,c113dab7,77,c7145b34,...) at _cv_wait+0x263/frame 0xeb208ca4 usb_process(c7145b24,eb208d08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb208ccc fork_exit(c0974000,c7145b24,eb208d08) at fork_exit+0x7f/frame 0xeb208cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb208cf4 --- trap 0, eip = 0, esp = 0xeb208d40, ebp = 0 --- Tracing command usb pid 15 tid 100043 td 0xc7143930 sched_switch(c7143930,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb205bd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb205c08 sleepq_switch(c7143930,0,c116d801,267,c7145afc,...) at sleepq_switch+0x1c5/frame 0xeb205c3c sleepq_wait(c7145afc,0,eb205c90,1,0,...) at sleepq_wait+0x48/frame 0xeb205c60 _cv_wait(c7145afc,c7145c9c,c113dab7,77,c7145b04,...) at _cv_wait+0x263/frame 0xeb205ca4 usb_process(c7145af4,eb205d08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb205ccc fork_exit(c0974000,c7145af4,eb205d08) at fork_exit+0x7f/frame 0xeb205cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb205cf4 --- trap 0, eip = 0, esp = 0xeb205d40, ebp = 0 --- Tracing command usb pid 15 tid 100042 td 0xc7143c40 sched_switch(c7143c40,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb202bd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb202c08 sleepq_switch(c7143c40,0,c116d801,267,c7145acc,...) at sleepq_switch+0x1c5/frame 0xeb202c3c sleepq_wait(c7145acc,0,eb202c90,1,0,...) at sleepq_wait+0x48/frame 0xeb202c60 _cv_wait(c7145acc,c7145c9c,c113dab7,77,c7145ad4,...) at _cv_wait+0x263/frame 0xeb202ca4 usb_process(c7145ac4,eb202d08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb202ccc fork_exit(c0974000,c7145ac4,eb202d08) at fork_exit+0x7f/frame 0xeb202cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb202cf4 --- trap 0, eip = 0, esp = 0xeb202d40, ebp = 0 --- Tracing command usb pid 15 tid 100040 td 0xc7144310 sched_switch(c7144310,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb1fbbd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb1fbc08 sleepq_switch(c7144310,0,c116d801,267,c713cb5c,...) at sleepq_switch+0x1c5/frame 0xeb1fbc3c sleepq_wait(c713cb5c,0,eb1fbc90,1,0,...) at sleepq_wait+0x48/frame 0xeb1fbc60 _cv_wait(c713cb5c,c713cc9c,c113dab7,77,c713cb64,...) at _cv_wait+0x263/frame 0xeb1fbca4 usb_process(c713cb54,eb1fbd08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb1fbccc fork_exit(c0974000,c713cb54,eb1fbd08) at fork_exit+0x7f/frame 0xeb1fbcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb1fbcf4 --- trap 0, eip = 0, esp = 0xeb1fbd40, ebp = 0 --- Tracing command usb pid 15 tid 100039 td 0xc6fbe310 sched_switch(c6fbe310,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb1f8bd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb1f8c08 sleepq_switch(c6fbe310,0,c116d801,267,c713cb2c,...) at sleepq_switch+0x1c5/frame 0xeb1f8c3c sleepq_wait(c713cb2c,0,eb1f8c90,1,0,...) at sleepq_wait+0x48/frame 0xeb1f8c60 _cv_wait(c713cb2c,c713cc9c,c113dab7,77,c713cb34,...) at _cv_wait+0x263/frame 0xeb1f8ca4 usb_process(c713cb24,eb1f8d08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb1f8ccc fork_exit(c0974000,c713cb24,eb1f8d08) at fork_exit+0x7f/frame 0xeb1f8cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb1f8cf4 --- trap 0, eip = 0, esp = 0xeb1f8d40, ebp = 0 --- Tracing command usb pid 15 tid 100038 td 0xc6fbe620 sched_switch(c6fbe620,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb1f5bd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb1f5c08 sleepq_switch(c6fbe620,0,c116d801,267,c713cafc,...) at sleepq_switch+0x1c5/frame 0xeb1f5c3c sleepq_wait(c713cafc,0,eb1f5c90,1,0,...) at sleepq_wait+0x48/frame 0xeb1f5c60 _cv_wait(c713cafc,c713cc9c,c113dab7,77,c713cb04,...) at _cv_wait+0x263/frame 0xeb1f5ca4 usb_process(c713caf4,eb1f5d08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb1f5ccc fork_exit(c0974000,c713caf4,eb1f5d08) at fork_exit+0x7f/frame 0xeb1f5cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb1f5cf4 --- trap 0, eip = 0, esp = 0xeb1f5d40, ebp = 0 --- Tracing command usb pid 15 tid 100037 td 0xc6fbe930 sched_switch(c6fbe930,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb1f2bd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb1f2c08 sleepq_switch(c6fbe930,0,c116d801,267,c713cacc,...) at sleepq_switch+0x1c5/frame 0xeb1f2c3c sleepq_wait(c713cacc,0,eb1f2c90,1,0,...) at sleepq_wait+0x48/frame 0xeb1f2c60 _cv_wait(c713cacc,c713cc9c,c113dab7,77,c713cad4,...) at _cv_wait+0x263/frame 0xeb1f2ca4 usb_process(c713cac4,eb1f2d08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb1f2ccc fork_exit(c0974000,c713cac4,eb1f2d08) at fork_exit+0x7f/frame 0xeb1f2cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb1f2cf4 --- trap 0, eip = 0, esp = 0xeb1f2d40, ebp = 0 --- Tracing command usb pid 15 tid 100035 td 0xc70ba000 sched_switch(c70ba000,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb1ebbd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb1ebc08 sleepq_switch(c70ba000,0,c116d801,267,c7135b5c,...) at sleepq_switch+0x1c5/frame 0xeb1ebc3c sleepq_wait(c7135b5c,0,eb1ebc90,1,0,...) at sleepq_wait+0x48/frame 0xeb1ebc60 _cv_wait(c7135b5c,c7135c9c,c113dab7,77,c7135b64,...) at _cv_wait+0x263/frame 0xeb1ebca4 usb_process(c7135b54,eb1ebd08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb1ebccc fork_exit(c0974000,c7135b54,eb1ebd08) at fork_exit+0x7f/frame 0xeb1ebcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb1ebcf4 --- trap 0, eip = 0, esp = 0xeb1ebd40, ebp = 0 --- Tracing command usb pid 15 tid 100034 td 0xc70ba310 sched_switch(c70ba310,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb1e8bd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb1e8c08 sleepq_switch(c70ba310,0,c116d801,267,c7135b2c,...) at sleepq_switch+0x1c5/frame 0xeb1e8c3c sleepq_wait(c7135b2c,0,eb1e8c90,1,0,...) at sleepq_wait+0x48/frame 0xeb1e8c60 _cv_wait(c7135b2c,c7135c9c,c113dab7,77,c7135b34,...) at _cv_wait+0x263/frame 0xeb1e8ca4 usb_process(c7135b24,eb1e8d08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb1e8ccc fork_exit(c0974000,c7135b24,eb1e8d08) at fork_exit+0x7f/frame 0xeb1e8cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb1e8cf4 --- trap 0, eip = 0, esp = 0xeb1e8d40, ebp = 0 --- Tracing command usb pid 15 tid 100033 td 0xc70ba620 sched_switch(c70ba620,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb1e5bd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb1e5c08 sleepq_switch(c70ba620,0,c116d801,267,c7135afc,...) at sleepq_switch+0x1c5/frame 0xeb1e5c3c sleepq_wait(c7135afc,0,eb1e5c90,1,0,...) at sleepq_wait+0x48/frame 0xeb1e5c60 _cv_wait(c7135afc,c7135c9c,c113dab7,77,c7135b04,...) at _cv_wait+0x263/frame 0xeb1e5ca4 usb_process(c7135af4,eb1e5d08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb1e5ccc fork_exit(c0974000,c7135af4,eb1e5d08) at fork_exit+0x7f/frame 0xeb1e5cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb1e5cf4 --- trap 0, eip = 0, esp = 0xeb1e5d40, ebp = 0 --- Tracing command usb pid 15 tid 100032 td 0xc70ba930 sched_switch(c70ba930,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb1e2bd4 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb1e2c08 sleepq_switch(c70ba930,0,c116d801,267,c7135acc,...) at sleepq_switch+0x1c5/frame 0xeb1e2c3c sleepq_wait(c7135acc,0,eb1e2c90,1,0,...) at sleepq_wait+0x48/frame 0xeb1e2c60 _cv_wait(c7135acc,c7135c9c,c113dab7,77,c7135ad4,...) at _cv_wait+0x263/frame 0xeb1e2ca4 usb_process(c7135ac4,eb1e2d08,c1160d7c,3c9,0,...) at usb_process+0xe6/frame 0xeb1e2ccc fork_exit(c0974000,c7135ac4,eb1e2d08) at fork_exit+0x7f/frame 0xeb1e2cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb1e2cf4 --- trap 0, eip = 0, esp = 0xeb1e2d40, ebp = 0 --- Tracing command cam pid 2 tid 100068 td 0xc7381000 sched_switch(c7381000,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb271b90 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb271bc4 sleepq_switch(c7381000,0,c116d801,267,c14126a8,...) at sleepq_switch+0x1c5/frame 0xeb271bf8 sleepq_wait(c14126a8,5c,c102843a,0,0,...) at sleepq_wait+0x48/frame 0xeb271c1c _sleep(c14126a8,c14126f8,5c,c102843a,0,...) at _sleep+0x391/frame 0xeb271c78 xpt_scanner_thread(0,eb271d08,c1160d7c,3c9,0,...) at xpt_scanner_thread+0xac/frame 0xeb271ccc fork_exit(c04c24f0,0,eb271d08) at fork_exit+0x7f/frame 0xeb271cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb271cf4 --- trap 0, eip = 0, esp = 0xeb271d40, ebp = 0 --- Tracing command cam pid 2 tid 100019 td 0xc6fbd000 sched_switch(c6fbd000,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xee628b98 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xee628bcc sleepq_switch(c6fbd000,0,c116d801,267,c1412800,...) at sleepq_switch+0x1c5/frame 0xee628c00 sleepq_wait(c1412800,5c,c1152ed6,0,0,...) at sleepq_wait+0x48/frame 0xee628c24 _sleep(c1412800,c1412780,5c,c1152ed6,0,...) at _sleep+0x391/frame 0xee628c80 xpt_done_td(c1412780,ee628d08,c1160d7c,3c9,0,...) at xpt_done_td+0xd2/frame 0xee628ccc fork_exit(c04c2110,c1412780,ee628d08) at fork_exit+0x7f/frame 0xee628cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xee628cf4 --- trap 0, eip = 0, esp = 0xee628d40, ebp = 0 --- Tracing command rand_harvestq pid 14 tid 100017 td 0xc6fbd620 sched_switch(c6fbd620,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xee621bb4 mi_switch(104,0,c116d801,1f1,c146fc98,...) at mi_switch+0x1c3/frame 0xee621be8 sleepq_switch(c6fbd620,0,c116d801,28a,c142c438,...) at sleepq_switch+0x1c5/frame 0xee621c1c sleepq_timedwait(c142c438,0,c1168212,c1152ed6,0,...) at sleepq_timedwait+0x48/frame 0xee621c40 msleep_spin_sbt(c142c438,c175e26c,c1152ed6,19999999,0,...) at msleep_spin_sbt+0x225/frame 0xee621c88 random_kthread(c085bf10,ee621d08,c1160d7c,3c9,0,...) at random_kthread+0x213/frame 0xee621ccc fork_exit(c085aff0,c085bf10,ee621d08) at fork_exit+0x7f/frame 0xee621cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xee621cf4 --- trap 0, eip = 0, esp = 0xee621d40, ebp = 0 --- Tracing command geom pid 13 tid 100015 td 0xc6fbdc40 sched_switch(c6fbdc40,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xee61bb94 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xee61bbc8 sleepq_switch(c6fbdc40,0,c116d801,267,c17691e4,...) at sleepq_switch+0x1c5/frame 0xee61bbfc sleepq_wait(c17691e4,5c,c1152ed6,0,0,...) at sleepq_wait+0x48/frame 0xee61bc20 _sleep(c17691e4,c1450880,25c,c1152ed6,0,...) at _sleep+0x391/frame 0xee61bc7c g_io_schedule_down(c6fbdc40,0,c1154c2e,6d,ee61bcf4,...) at g_io_schedule_down+0x5c/frame 0xee61bcb4 g_down_procbody(0,ee61bd08,c1160d7c,3c9,0,...) at g_down_procbody+0x9d/frame 0xee61bccc fork_exit(c0a42b10,0,ee61bd08) at fork_exit+0x7f/frame 0xee61bcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xee61bcf4 --- trap 0, eip = 0, esp = 0xee61bd40, ebp = 0 --- Tracing command geom pid 13 tid 100014 td 0xc6fbe000 sched_switch(c6fbe000,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xee618b98 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xee618bcc sleepq_switch(c6fbe000,0,c116d801,267,c17691e0,...) at sleepq_switch+0x1c5/frame 0xee618c00 sleepq_wait(c17691e0,5c,c1152ed6,0,0,...) at sleepq_wait+0x48/frame 0xee618c24 _sleep(c17691e0,c14508a0,25c,c1152ed6,0,...) at _sleep+0x391/frame 0xee618c80 g_io_schedule_up(c6fbe000,0,c1154c2e,60,ee618cf4,...) at g_io_schedule_up+0xe6/frame 0xee618cb4 g_up_procbody(0,ee618d08,c1160d7c,3c9,0,...) at g_up_procbody+0x9d/frame 0xee618ccc fork_exit(c0a42a70,0,ee618d08) at fork_exit+0x7f/frame 0xee618cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xee618cf4 --- trap 0, eip = 0, esp = 0xee618d40, ebp = 0 --- Tracing command geom pid 13 tid 100013 td 0xc6f02000 sched_switch(c6f02000,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xee615ba8 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xee615bdc sleepq_switch(c6f02000,0,c116d801,267,c17691dc,...) at sleepq_switch+0x1c5/frame 0xee615c10 sleepq_wait(c17691dc,5c,c1152ed6,0,0,...) at sleepq_wait+0x48/frame 0xee615c34 _sleep(c17691dc,c1450858,25c,c1152ed6,0,...) at _sleep+0x391/frame 0xee615c90 g_run_events(0,ee615d08,c1160d7c,3c9,0,...) at g_run_events+0x62/frame 0xee615ccc fork_exit(c0a429d0,0,ee615d08) at fork_exit+0x7f/frame 0xee615cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xee615cf4 --- trap 0, eip = 0, esp = 0xee615d40, ebp = 0 --- Tracing command intr pid 12 tid 100063 td 0xc7144c40 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100062 td 0xc7150000 sched_switch(c7150000,0,109,1bb,c1779f34,...) at sched_switch+0x4cf/frame 0xeb25ec60 mi_switch(109,0,c1160fed,55a,8,...) at mi_switch+0x1c3/frame 0xeb25ec94 ithread_loop(c6f53670,eb25ed08,c1160d7c,3c9,0,...) at ithread_loop+0x1fb/frame 0xeb25eccc fork_exit(c0ab8b80,c6f53670,eb25ed08) at fork_exit+0x7f/frame 0xeb25ecf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb25ecf4 --- trap 0, eip = 0, esp = 0xeb25ed40, ebp = 0 --- Tracing command intr pid 12 tid 100060 td 0xc7150620 sched_switch(c7150620,0,109,1bb,0,...) at sched_switch+0x4cf/frame 0xeb24ec60 mi_switch(109,0,c1160fed,55a,8,...) at mi_switch+0x1c3/frame 0xeb24ec94 ithread_loop(c6f53680,eb24ed08,c1160d7c,3c9,0,...) at ithread_loop+0x1fb/frame 0xeb24eccc fork_exit(c0ab8b80,c6f53680,eb24ed08) at fork_exit+0x7f/frame 0xeb24ecf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb24ecf4 --- trap 0, eip = 0, esp = 0xeb24ed40, ebp = 0 --- Tracing command intr pid 12 tid 100059 td 0xc7150930 sched_switch(c7150930,0,109,1bb,c1779f34,...) at sched_switch+0x4cf/frame 0xeb23dc60 mi_switch(109,0,c1160fed,55a,8,...) at mi_switch+0x1c3/frame 0xeb23dc94 ithread_loop(c6f53690,eb23dd08,c1160d7c,3c9,0,...) at ithread_loop+0x1fb/frame 0xeb23dccc fork_exit(c0ab8b80,c6f53690,eb23dd08) at fork_exit+0x7f/frame 0xeb23dcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb23dcf4 --- trap 0, eip = 0, esp = 0xeb23dd40, ebp = 0 --- Tracing command intr pid 12 tid 100054 td 0xc7151930 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100041 td 0xc7144000 sched_switch(c7144000,0,109,1bb,c1779f34,...) at sched_switch+0x4cf/frame 0xeb1ffc60 mi_switch(109,0,c1160fed,55a,8,...) at mi_switch+0x1c3/frame 0xeb1ffc94 ithread_loop(c6f536b0,eb1ffd08,c1160d7c,3c9,0,...) at ithread_loop+0x1fb/frame 0xeb1ffccc fork_exit(c0ab8b80,c6f536b0,eb1ffd08) at fork_exit+0x7f/frame 0xeb1ffcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb1ffcf4 --- trap 0, eip = 0, esp = 0xeb1ffd40, ebp = 0 --- Tracing command intr pid 12 tid 100036 td 0xc6fbec40 sched_switch(c6fbec40,0,109,1bb,c1779f34,...) at sched_switch+0x4cf/frame 0xeb1efc60 mi_switch(109,0,c1160fed,55a,8,...) at mi_switch+0x1c3/frame 0xeb1efc94 ithread_loop(c6f536c0,eb1efd08,c1160d7c,3c9,0,...) at ithread_loop+0x1fb/frame 0xeb1efccc fork_exit(c0ab8b80,c6f536c0,eb1efd08) at fork_exit+0x7f/frame 0xeb1efcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb1efcf4 --- trap 0, eip = 0, esp = 0xeb1efd40, ebp = 0 --- Tracing command intr pid 12 tid 100031 td 0xc70bac40 sched_switch(c70bac40,0,109,1bb,c1779f34,...) at sched_switch+0x4cf/frame 0xeb1dfc60 mi_switch(109,0,c1160fed,55a,8,...) at mi_switch+0x1c3/frame 0xeb1dfc94 ithread_loop(c6f536d0,eb1dfd08,c1160d7c,3c9,0,...) at ithread_loop+0x1fb/frame 0xeb1dfccc fork_exit(c0ab8b80,c6f536d0,eb1dfd08) at fork_exit+0x7f/frame 0xeb1dfcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb1dfcf4 --- trap 0, eip = 0, esp = 0xeb1dfd40, ebp = 0 --- Tracing command intr pid 12 tid 100030 td 0xc70bb000 sched_switch(c70bb000,0,109,1bb,c1779f34,...) at sched_switch+0x4cf/frame 0xee77fc60 mi_switch(109,0,c1160fed,55a,8,...) at mi_switch+0x1c3/frame 0xee77fc94 ithread_loop(c6f536e0,ee77fd08,c1160d7c,3c9,0,...) at ithread_loop+0x1fb/frame 0xee77fccc fork_exit(c0ab8b80,c6f536e0,ee77fd08) at fork_exit+0x7f/frame 0xee77fcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xee77fcf4 --- trap 0, eip = 0, esp = 0xee77fd40, ebp = 0 --- Tracing command intr pid 12 tid 100029 td 0xc70bb310 sched_switch(c70bb310,0,109,1bb,c1160fed,...) at sched_switch+0x4cf/frame 0xee77bc60 mi_switch(109,0,c1160fed,55a,8,...) at mi_switch+0x1c3/frame 0xee77bc94 ithread_loop(c6f536f0,ee77bd08,c1160d7c,3c9,0,...) at ithread_loop+0x1fb/frame 0xee77bccc fork_exit(c0ab8b80,c6f536f0,ee77bd08) at fork_exit+0x7f/frame 0xee77bcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xee77bcf4 --- trap 0, eip = 0, esp = 0xee77bd40, ebp = 0 --- Tracing command intr pid 12 tid 100028 td 0xc70bb620 sched_switch(c70bb620,0,109,1bb,c1160fed,...) at sched_switch+0x4cf/frame 0xee672c60 mi_switch(109,0,c1160fed,55a,8,...) at mi_switch+0x1c3/frame 0xee672c94 ithread_loop(c6f53710,ee672d08,c1160d7c,3c9,0,...) at ithread_loop+0x1fb/frame 0xee672ccc fork_exit(c0ab8b80,c6f53710,ee672d08) at fork_exit+0x7f/frame 0xee672cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xee672cf4 --- trap 0, eip = 0, esp = 0xee672d40, ebp = 0 --- Tracing command intr pid 12 tid 100026 td 0xc6f09930 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100024 td 0xc6fbc000 sched_switch(c6fbc000,0,109,1bb,0,...) at sched_switch+0x4cf/frame 0xee637c60 mi_switch(109,0,c1160fed,55a,8,...) at mi_switch+0x1c3/frame 0xee637c94 ithread_loop(c6f53730,ee637d08,c1160d7c,3c9,0,...) at ithread_loop+0x1fb/frame 0xee637ccc fork_exit(c0ab8b80,c6f53730,ee637d08) at fork_exit+0x7f/frame 0xee637cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xee637cf4 --- trap 0, eip = 0, esp = 0xee637d40, ebp = 0 --- Tracing command intr pid 12 tid 100023 td 0xc6fbc310 sched_switch(c6fbc310,0,109,1bb,c1160fed,...) at sched_switch+0x4cf/frame 0xee634c60 mi_switch(109,0,c1160fed,55a,8,...) at mi_switch+0x1c3/frame 0xee634c94 ithread_loop(c6f53740,ee634d08,c1160d7c,3c9,0,...) at ithread_loop+0x1fb/frame 0xee634ccc fork_exit(c0ab8b80,c6f53740,ee634d08) at fork_exit+0x7f/frame 0xee634cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xee634cf4 --- trap 0, eip = 0, esp = 0xee634d40, ebp = 0 --- Tracing command intr pid 12 tid 100012 td 0xc6f02310 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100011 td 0xc6f02620 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100010 td 0xc6f02930 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100009 td 0xc6f02c40 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100008 td 0xc6f08000 sched_switch(c6f08000,0,109,1bb,c1160fed,...) at sched_switch+0x4cf/frame 0xee606c60 mi_switch(109,0,c1160fed,55a,8,...) at mi_switch+0x1c3/frame 0xee606c94 ithread_loop(c6f537b0,ee606d08,c1160d7c,3c9,0,...) at ithread_loop+0x1fb/frame 0xee606ccc fork_exit(c0ab8b80,c6f537b0,ee606d08) at fork_exit+0x7f/frame 0xee606cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xee606cf4 --- trap 0, eip = 0, esp = 0xee606d40, ebp = 0 --- Tracing command intr pid 12 tid 100007 td 0xc6f08310 sched_switch(c6f08310,0,109,1bb,c1160fed,...) at sched_switch+0x4cf/frame 0xee603c60 mi_switch(109,0,c1160fed,55a,8,...) at mi_switch+0x1c3/frame 0xee603c94 ithread_loop(c6f537c0,ee603d08,c1160d7c,3c9,0,...) at ithread_loop+0x1fb/frame 0xee603ccc fork_exit(c0ab8b80,c6f537c0,ee603d08) at fork_exit+0x7f/frame 0xee603cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xee603cf4 --- trap 0, eip = 0, esp = 0xee603d40, ebp = 0 --- Tracing command idle pid 11 tid 100006 td 0xc6f08620 cpustop_handler(3,13,eb109b94,c0ff85da,3,...) at cpustop_handler+0x25/frame 0xeb109a40 ipi_nmi_handler(3,0,0,30540c,eb109a9c,...) at ipi_nmi_handler+0x37/frame 0xeb109a50 trap(eb109ba0) at trap+0x3a/frame 0xeb109b94 calltrap() at calltrap+0x6/frame 0xeb109b94 --- trap 0x13, eip = 0xc0fdda15, esp = 0xeb109be0, ebp = 0xeb109be0 --- acpi_cpu_c1(eb0f6c90,0,c132e287,303,eb0f6c80,...) at acpi_cpu_c1+0x5/frame 0xeb109be0 acpi_cpu_idle(5fbcbdd5,0,5fbcbdd5,eb109c50,c0fe7efa,...) at acpi_cpu_idle+0x142/frame 0xeb109c20 cpu_idle_acpi(5fbcbdd5,0,c1460e08,c1460e14,c1460e04,...) at cpu_idle_acpi+0x3f/frame 0xeb109c34 cpu_idle(0,2,c1169beb,3f4,c1460e08,...) at cpu_idle+0x9a/frame 0xeb109c50 sched_idletd(0,eb109d08,c1160d7c,3c9,0,...) at sched_idletd+0x1c8/frame 0xeb109ccc fork_exit(c0b13070,0,eb109d08) at fork_exit+0x7f/frame 0xeb109cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb109cf4 --- trap 0, eip = 0, esp = 0xeb109d40, ebp = 0 --- Tracing command idle pid 11 tid 100005 td 0xc6f08930 kdb_break(c7054600,c132e287,369,c6f08930,c6f08930,...) at kdb_break+0x4e/frame 0xeb106ae8 uart_intr(c7054600,a,c1168da1,2,0,...) at uart_intr+0x91/frame 0xeb106b24 intr_event_handle(c6f51e00,eb106ba0,4a9cc0ab,eb106ba0,30,...) at intr_event_handle+0x7a/frame 0xeb106b50 intr_execute_handlers(c6ececd4,eb106ba0,2) at intr_execute_handlers+0x42/frame 0xeb106b70 lapic_handle_intr(30,eb106ba0) at lapic_handle_intr+0x6f/frame 0xeb106b90 Xapic_isr1() at Xapic_isr1+0x35/frame 0xeb106b90 --- interrupt, eip = 0xc0fdda15, esp = 0xeb106be0, ebp = 0xeb106be0 --- acpi_cpu_c1(eb0f3c90,0,c132e287,303,eb0f3c80,...) at acpi_cpu_c1+0x5/frame 0xeb106be0 acpi_cpu_idle(4a9cc0ab,0,4a9cc0ab,eb106c50,c0fe7efa,...) at acpi_cpu_idle+0x142/frame 0xeb106c20 cpu_idle_acpi(4a9cc0ab,0,c1460708,c1460714,c1460704,...) at cpu_idle_acpi+0x3f/frame 0xeb106c34 cpu_idle(0,2,c1169beb,3f4,c1460708,...) at cpu_idle+0x9a/frame 0xeb106c50 sched_idletd(0,eb106d08,c1160d7c,3c9,0,...) at sched_idletd+0x1c8/frame 0xeb106ccc fork_exit(c0b13070,0,eb106d08) at fork_exit+0x7f/frame 0xeb106cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb106cf4 --- trap 0, eip = 0, esp = 0xeb106d40, ebp = 0 --- Tracing command idle pid 11 tid 100004 td 0xc6f08c40 cpustop_handler(1,13,eb103b94,c0ff85da,1,...) at cpustop_handler+0x25/frame 0xeb103a40 ipi_nmi_handler(1,0,0,1ff830,eb103a9c,...) at ipi_nmi_handler+0x37/frame 0xeb103a50 trap(eb103ba0) at trap+0x3a/frame 0xeb103b94 calltrap() at calltrap+0x6/frame 0xeb103b94 --- trap 0x13, eip = 0xc0fdda15, esp = 0xeb103be0, ebp = 0xeb103be0 --- acpi_cpu_c1(eb0f0c90,0,c132e287,303,eb0f0c80,...) at acpi_cpu_c1+0x5/frame 0xeb103be0 acpi_cpu_idle(52add45a,0,52add45a,eb103c50,c0fe7efa,...) at acpi_cpu_idle+0x142/frame 0xeb103c20 cpu_idle_acpi(52add45a,0,c1460008,c1460014,c1460004,...) at cpu_idle_acpi+0x3f/frame 0xeb103c34 cpu_idle(0,2,c1169beb,a2c,c1460008,...) at cpu_idle+0x9a/frame 0xeb103c50 sched_idletd(0,eb103d08,c1160d7c,3c9,0,...) at sched_idletd+0x1c8/frame 0xeb103ccc fork_exit(c0b13070,0,eb103d08) at fork_exit+0x7f/frame 0xeb103cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb103cf4 --- trap 0, eip = 0, esp = 0xeb103d40, ebp = 0 --- Tracing command idle pid 11 tid 100003 td 0xc6f09000 cpustop_handler(0,13,eb100b94,c0ff85da,0,...) at cpustop_handler+0x25/frame 0xeb100a40 ipi_nmi_handler(0,0,0,415cd5,eb100a9c,...) at ipi_nmi_handler+0x37/frame 0xeb100a50 trap(eb100ba0) at trap+0x3a/frame 0xeb100b94 calltrap() at calltrap+0x6/frame 0xeb100b94 --- trap 0x13, eip = 0xc0fdda15, esp = 0xeb100be0, ebp = 0xeb100be0 --- acpi_cpu_c1(c1c25c90,0,c132e287,303,c1c25c80,...) at acpi_cpu_c1+0x5/frame 0xeb100be0 acpi_cpu_idle(d0ca4a0,0,d0ca4a0,eb100c50,c0fe7efa,...) at acpi_cpu_idle+0x142/frame 0xeb100c20 cpu_idle_acpi(d0ca4a0,0,c145f908,c145f914,c145f904,...) at cpu_idle_acpi+0x3f/frame 0xeb100c34 cpu_idle(0,2,c1169beb,a2c,c145f908,...) at cpu_idle+0x9a/frame 0xeb100c50 sched_idletd(0,eb100d08,c1160d7c,3c9,0,...) at sched_idletd+0x1c8/frame 0xeb100ccc fork_exit(c0b13070,0,eb100d08) at fork_exit+0x7f/frame 0xeb100cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb100cf4 --- trap 0, eip = 0, esp = 0xeb100d40, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc6f09310 sched_switch(c6f09310,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb0fc98c mi_switch(104,0,c116d801,1f1,0,...) at mi_switch+0x1c3/frame 0xeb0fc9c0 sleepq_switch(c6f09310,0,c116d801,1a7,c145f880,...) at sleepq_switch+0x1c5/frame 0xeb0fc9f4 sleepq_catch_signals(c116d801,15e,c6f01394,17d0,c6f012ec,...) at sleepq_catch_signals+0x3ce/frame 0xeb0fca38 sleepq_wait_sig(c6f012ec,6c,c1171525,100,0,...) at sleepq_wait_sig+0x14/frame 0xeb0fca5c _sleep(c6f012ec,c6f01384,16c,c1171525,0,...) at _sleep+0x372/frame 0xeb0fcab8 kern_wait6(c6f09310,7,0,0,eb0fcb4c,...) at kern_wait6+0x47b/frame 0xeb0fcb20 sys_wait4(c6f09310,eb0fccc8,0,ab8,eb0fcccc,...) at sys_wait4+0x94/frame 0xeb0fcc40 syscall(eb0fcd08) at syscall+0x31b/frame 0xeb0fccfc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xeb0fccfc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x805fa4f, esp = 0xbfbfd87c, ebp = 0xbfbfd890 --- Tracing command audit pid 10 tid 100001 td 0xc6f09620 sched_switch(c6f09620,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb0f9ba8 mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb0f9bdc sleepq_switch(c6f09620,0,c116d801,267,c176ef18,...) at sleepq_switch+0x1c5/frame 0xeb0f9c10 sleepq_wait(c176ef18,0,eb0f9c64,1,0,...) at sleepq_wait+0x48/frame 0xeb0f9c34 _cv_wait(c176ef18,c176eea4,c119a3f2,18d,0,...) at _cv_wait+0x263/frame 0xeb0f9c78 audit_worker(0,eb0f9d08,c1160d7c,3c9,0,...) at audit_worker+0x94/frame 0xeb0f9ccc fork_exit(c0d1d7b0,0,eb0f9d08) at fork_exit+0x7f/frame 0xeb0f9cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb0f9cf4 --- trap 0, eip = 0, esp = 0xeb0f9d40, ebp = 0 --- Tracing command kernel pid 0 tid 100067 td 0xc7381310 sched_switch(c7381310,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb26ebac mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xeb26ebe0 sleepq_switch(c7381310,0,c116d801,267,c6fcae80,...) at sleepq_switch+0x1c5/frame 0xeb26ec14 sleepq_wait(c6fcae80,0,c1152ed6,0,0,...) at sleepq_wait+0x48/frame 0xeb26ec38 _sleep(c6fcae80,c6fcae98,0,c1152ed6,0,...) at _sleep+0x391/frame 0xeb26ec94 taskqueue_thread_loop(c1412720,eb26ed08,c1160d7c,3c9,0,...) at taskqueue_thread_loop+0x129/frame 0xeb26eccc fork_exit(c0b35290,c1412720,eb26ed08) at fork_exit+0x7f/frame 0xeb26ecf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb26ecf4 --- trap 0, eip = 0, esp = 0xeb26ed40, ebp = 0 --- Tracing command kernel pid 0 tid 100065 td 0xc7144620 sched_switch(c7144620,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb267b68 mi_switch(104,0,c116d801,1f1,101,...) at mi_switch+0x1c3/frame 0xeb267b9c sleepq_switch(c7144620,0,c116d801,28a,c145252f,...) at sleepq_switch+0x1c5/frame 0xeb267bd0 sleepq_timedwait(c145252f,0,2,0,0,...) at sleepq_timedwait+0x48/frame 0xeb267bf4 _sleep(c145252f,0,0,c1152ed6,fffffc88,...) at _sleep+0x35e/frame 0xeb267c50 pause_sbt(c1152ed6,fffffc88,2,0,0,...) at pause_sbt+0xe9/frame 0xeb267c8c deadlkres(0,eb267d08,c1160d7c,3c9,0,...) at deadlkres+0x81/frame 0xeb267ccc fork_exit(c0a95e20,0,eb267d08) at fork_exit+0x7f/frame 0xeb267cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb267cf4 --- trap 0, eip = 0, esp = 0xeb267d40, ebp = 0 --- Tracing command kernel pid 0 tid 100064 td 0xc7144930 sched_switch(c7144930,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xeb264bc0 mi_switch(104,0,c116d801,1f1,c146e840,...) at mi_switch+0x1c3/frame 0xeb264bf4 sleepq_switch(c7144930,0,c116d801,267,c6fca380,...) at sleepq_switch+0x1c5/frame 0xeb264c28 sleepq_wait(c6fca380,0,c1168212,c1152ed6,0,...) at sleepq_wait+0x48/frame 0xeb264c4c msleep_spin_sbt(c6fca380,c6fca398,c1152ed6,0,0,...) at msleep_spin_sbt+0x239/frame 0xeb264c94 taskqueue_thread_loop(c175ad80,eb264d08,c1160d7c,3c9,0,...) at taskqueue_thread_loop+0xcc/frame 0xeb264ccc fork_exit(c0b35290,c175ad80,eb264d08) at fork_exit+0x7f/frame 0xeb264cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xeb264cf4 --- trap 0, eip = 0, esp = 0xeb264d40, ebp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xc70bb930 sched_switch(c70bb930,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xee640bac mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xee640be0 sleepq_switch(c70bb930,0,c116d801,267,c6fcab00,...) at sleepq_switch+0x1c5/frame 0xee640c14 sleepq_wait(c6fcab00,0,c1152ed6,0,0,...) at sleepq_wait+0x48/frame 0xee640c38 _sleep(c6fcab00,c6fcab18,0,c1152ed6,0,...) at _sleep+0x391/frame 0xee640c94 taskqueue_thread_loop(c176ef50,ee640d08,c1160d7c,3c9,0,...) at taskqueue_thread_loop+0x129/frame 0xee640ccc fork_exit(c0b35290,c176ef50,ee640d08) at fork_exit+0x7f/frame 0xee640cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xee640cf4 --- trap 0, eip = 0, esp = 0xee640d40, ebp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xc6f09c40 sched_switch(c6f09c40,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xee63abac mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xee63abe0 sleepq_switch(c6f09c40,0,c116d801,267,c6fcac00,...) at sleepq_switch+0x1c5/frame 0xee63ac14 sleepq_wait(c6fcac00,0,c1152ed6,0,0,...) at sleepq_wait+0x48/frame 0xee63ac38 _sleep(c6fcac00,c6fcac18,0,c1152ed6,0,...) at _sleep+0x391/frame 0xee63ac94 taskqueue_thread_loop(c176c8a8,ee63ad08,c1160d7c,3c9,0,...) at taskqueue_thread_loop+0x129/frame 0xee63accc fork_exit(c0b35290,c176c8a8,ee63ad08) at fork_exit+0x7f/frame 0xee63acf4 fork_trampoline() at fork_trampoline+0x8/frame 0xee63acf4 --- trap 0, eip = 0, esp = 0xee63ad40, ebp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xc6fbc620 sched_switch(c6fbc620,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xee631bc0 mi_switch(104,0,c116d801,1f1,c146e930,...) at mi_switch+0x1c3/frame 0xee631bf4 sleepq_switch(c6fbc620,0,c116d801,267,c6fcad80,...) at sleepq_switch+0x1c5/frame 0xee631c28 sleepq_wait(c6fcad80,0,c1168212,c1152ed6,0,...) at sleepq_wait+0x48/frame 0xee631c4c msleep_spin_sbt(c6fcad80,c6fcad98,c1152ed6,0,0,...) at msleep_spin_sbt+0x239/frame 0xee631c94 taskqueue_thread_loop(c175c8c4,ee631d08,c1160d7c,3c9,0,...) at taskqueue_thread_loop+0xcc/frame 0xee631ccc fork_exit(c0b35290,c175c8c4,ee631d08) at fork_exit+0x7f/frame 0xee631cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xee631cf4 --- trap 0, eip = 0, esp = 0xee631d40, ebp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xc6fbc930 sched_switch(c6fbc930,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xee62ebc0 mi_switch(104,0,c116d801,1f1,c146e930,...) at mi_switch+0x1c3/frame 0xee62ebf4 sleepq_switch(c6fbc930,0,c116d801,267,c6fcad80,...) at sleepq_switch+0x1c5/frame 0xee62ec28 sleepq_wait(c6fcad80,0,c1168212,c1152ed6,0,...) at sleepq_wait+0x48/frame 0xee62ec4c msleep_spin_sbt(c6fcad80,c6fcad98,c1152ed6,0,0,...) at msleep_spin_sbt+0x239/frame 0xee62ec94 taskqueue_thread_loop(c175c8c4,ee62ed08,c1160d7c,3c9,0,...) at taskqueue_thread_loop+0xcc/frame 0xee62eccc fork_exit(c0b35290,c175c8c4,ee62ed08) at fork_exit+0x7f/frame 0xee62ecf4 fork_trampoline() at fork_trampoline+0x8/frame 0xee62ecf4 --- trap 0, eip = 0, esp = 0xee62ed40, ebp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xc6fbcc40 sched_switch(c6fbcc40,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xee62bbc0 mi_switch(104,0,c116d801,1f1,c146e930,...) at mi_switch+0x1c3/frame 0xee62bbf4 sleepq_switch(c6fbcc40,0,c116d801,267,c6fcad80,...) at sleepq_switch+0x1c5/frame 0xee62bc28 sleepq_wait(c6fcad80,0,c1168212,c1152ed6,0,...) at sleepq_wait+0x48/frame 0xee62bc4c msleep_spin_sbt(c6fcad80,c6fcad98,c1152ed6,0,0,...) at msleep_spin_sbt+0x239/frame 0xee62bc94 taskqueue_thread_loop(c175c8c4,ee62bd08,c1160d7c,3c9,0,...) at taskqueue_thread_loop+0xcc/frame 0xee62bccc fork_exit(c0b35290,c175c8c4,ee62bd08) at fork_exit+0x7f/frame 0xee62bcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xee62bcf4 --- trap 0, eip = 0, esp = 0xee62bd40, ebp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xc6fbd310 sched_switch(c6fbd310,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xee625bac mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xee625be0 sleepq_switch(c6fbd310,0,c116d801,267,c6fcaf00,...) at sleepq_switch+0x1c5/frame 0xee625c14 sleepq_wait(c6fcaf00,0,c1152ed6,0,0,...) at sleepq_wait+0x48/frame 0xee625c38 _sleep(c6fcaf00,c6fcaf18,0,c1152ed6,0,...) at _sleep+0x391/frame 0xee625c94 taskqueue_thread_loop(c17699f8,ee625d08,c1160d7c,3c9,0,...) at taskqueue_thread_loop+0x129/frame 0xee625ccc fork_exit(c0b35290,c17699f8,ee625d08) at fork_exit+0x7f/frame 0xee625cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xee625cf4 --- trap 0, eip = 0, esp = 0xee625d40, ebp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xc6fbd930 sched_switch(c6fbd930,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xee61ebac mi_switch(104,0,c116d801,1f1,8,...) at mi_switch+0x1c3/frame 0xee61ebe0 sleepq_switch(c6fbd930,0,c116d801,267,c6fcb000,...) at sleepq_switch+0x1c5/frame 0xee61ec14 sleepq_wait(c6fcb000,0,c1152ed6,0,0,...) at sleepq_wait+0x48/frame 0xee61ec38 _sleep(c6fcb000,c6fcb018,0,c1152ed6,0,...) at _sleep+0x391/frame 0xee61ec94 taskqueue_thread_loop(c146d9c0,ee61ed08,c1160d7c,3c9,0,...) at taskqueue_thread_loop+0x129/frame 0xee61eccc fork_exit(c0b35290,c146d9c0,ee61ed08) at fork_exit+0x7f/frame 0xee61ecf4 fork_trampoline() at fork_trampoline+0x8/frame 0xee61ecf4 --- trap 0, eip = 0, esp = 0xee61ed40, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc1769560 sched_switch(c1769560,0,104,1bb,c0ad4e5e,...) at sched_switch+0x4cf/frame 0xc1c20c28 mi_switch(104,0,c116d801,1f1,101,...) at mi_switch+0x1c3/frame 0xc1c20c5c sleepq_switch(c1769560,0,c116d801,28a,c1769264,...) at sleepq_switch+0x1c5/frame 0xc1c20c90 sleepq_timedwait(c1769264,54,9,0,0,...) at sleepq_timedwait+0x48/frame 0xc1c20cb4 _sleep(c1769264,0,54,c11a6226,fffff470,...) at _sleep+0x35e/frame 0xc1c20d10 swapper() at swapper+0x283/frame 0xc1c20d58 begin() at begin+0x2c db:0:allt> call doadump Physical memory: 2023 MB Dumping 139 MB: 124 108 92 76 60 44 28 12 Dump complete = 0 db:0:doadump> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 2 (kgdb) proc 2882 [Switching to thread 81 (Thread 100202)]#0 sched_switch (td=0x0, newtd=, flags=0x104) at ../../../kern/sched_ule.c:1928 1928 cpuid = PCPU_GET(cpuid); (kgdb) bt #0 sched_switch (td=0x0, newtd=, flags=0x104) at ../../../kern/sched_ule.c:1928 #1 0xc0af3853 in mi_switch (flags=) at ../../../kern/kern_synch.c:493 #2 0xc0b31185 in sleepq_switch (wchan=, pri=) at ../../../kern/subr_sleepqueue.c:537 #3 0xc0b30f68 in sleepq_wait (wchan=0xc95a8a5c, pri=0x54) at ../../../kern/subr_sleepqueue.c:616 #4 0xc0af3071 in _sleep (ident=0xc95a8a5c, lock=0x9c, priority=, wmesg=, sbt=0x0, pr=0x0) at ../../../kern/kern_synch.c:255 #5 0xc0da5c25 in vm_object_pip_wait (object=0xc95a8a5c, waitid=0xc11a66d4 "objtrm") at ../../../vm/vm_object.c:396 #6 0xc0da73f8 in vm_object_terminate (object=0xc95a8a5c) at ../../../vm/vm_object.c:715 #7 0xc0da6704 in vm_object_deallocate (object=0xc95a8a5c) at ../../../vm/vm_object.c:652 #8 0xc0da2a26 in vm_map_entry_deallocate (entry=0xc9554000, system_map=0x0) at ../../../vm/vm_map.c:2763 #9 0xc0d98575 in vm_map_process_deferred () at ../../../vm/vm_map.c:475 #10 0xc0d98466 in _vm_map_unlock (map=0xc755ec30, file=0xc11a564f "../../../vm/vm_map.c", line=0xb80) at ../../../vm/vm_map.c:488 #11 0xc0d9f3a6 in vm_map_remove (map=0xc755ec30, start=0x1000, end=0xbfc00000) at ../../../vm/vm_map.c:2944 #12 0xc0d97f66 in vmspace_dofree (vm=0xc755ec30) at ../../../vm/vm_map.c:327 #13 0xc0d98191 in vmspace_exit (td=0xc8bfc000) at ../../../vm/vm_map.c:405 #14 0xc0ab120e in exit1 (td=0xc8bfc000, rv=) at ../../../kern/kern_exit.c:319 #15 0xc0ab0bcd in sys_sys_exit (td=0xc8bfc000, uap=0xc69cbcc8) at ../../../kern/kern_exit.c:119 #16 0xc0ff9aeb in syscall (frame=) at subr_syscall.c:133 #17 0xc0fe1b71 in Xint0x80_syscall () at ../../../i386/i386/exception.s:269 #18 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 5 #5 0xc0da5c25 in vm_object_pip_wait (object=0xc95a8a5c, waitid=0xc11a66d4 "objtrm") at ../../../vm/vm_object.c:396 396 VM_OBJECT_SLEEP(object, object, PVM, waitid, 0); (kgdb) l 391 { 392 393 VM_OBJECT_ASSERT_WLOCKED(object); 394 while (object->paging_in_progress) { 395 object->flags |= OBJ_PIPWNT; 396 VM_OBJECT_SLEEP(object, object, PVM, waitid, 0); 397 } 398 } 399 400 /* (kgdb) p *object $1 = {lock = {lock_object = {lo_name = 0xc11a6592 "vm object", lo_flags = 0x5630000, lo_data = 0x0, lo_witness = 0xc6d8d318}, rw_lock = 0x1}, object_list = {tqe_next = 0xc7e24ea0, tqe_prev = 0xc749cba8}, shadow_head = {lh_first = 0x0}, shadow_list = {le_next = 0xc7f9be04, le_prev = 0xc75b6a78}, memq = {tqh_first = 0xc424f568, tqh_last = 0xc3e9e0c0}, rtree = { rt_root = 0xe0879420, rt_flags = 0x0}, size = 0xc00, generation = 0x1, ref_count = 0x0, shadow_count = 0x0, memattr = 0x6, type = 0x1, flags = 0x3048, pg_color = 0x8800, paging_in_progress = 0x0, resident_page_count = 0xc00, backing_object = 0x0, backing_object_offset = 0x0, pager_object_list = {tqe_next = 0x0, tqe_prev = 0x0}, rvq = {lh_first = 0x0}, cache = {rt_root = 0x0, rt_flags = 0x0}, handle = 0x0, un_pager = {vnp = {vnp_size = 0x0, writemappings = 0x0}, devp = {devp_pglist = {tqh_first = 0x0, tqh_last = 0x0}, ops = 0x0, dev = 0x0}, sgp = {sgp_pglist = {tqh_first = 0x0, tqh_last = 0x0}}, swp = {swp_tmpfs = 0x0, swp_bcount = 0x0}}, cred = 0xc78b7d00, charge = 0xc00000} (kgdb) p object->flags $2 = 0x3048 (kgdb) f 8 #8 0xc0da2a26 in vm_map_entry_deallocate (entry=0xc9554000, system_map=0x0) at ../../../vm/vm_map.c:2763 2763 vm_object_deallocate(entry->object.vm_object); (kgdb) p *entry $3 = {prev = 0xc755ec30, next = 0xc77424c0, left = 0x0, right = 0xc7742474, start = 0x28800000, end = 0x29400000, avail_ssize = 0x0, adj_free = 0x967df000, max_free = 0x967df000, object = { vm_object = 0xc95a8a5c, sub_map = 0xc95a8a5c}, offset = 0x0, eflags = 0x0, protection = 0x3, max_protection = 0x7, inheritance = 0x1, read_ahead = 0x1f, wired_count = 0x0, next_read = 0x589, cred = 0x0, wiring_thread = 0x0} (kgdb) f 11 #11 0xc0d9f3a6 in vm_map_remove (map=0xc755ec30, start=0x1000, end=0xbfc00000) at ../../../vm/vm_map.c:2944 2944 vm_map_unlock(map); (kgdb) l 2939 int result; 2940 2941 vm_map_lock(map); 2942 VM_MAP_RANGE_CHECK(map, start, end); 2943 result = vm_map_delete(map, start, end); 2944 vm_map_unlock(map); 2945 return (result); 2946 } 2947 2948 /* (kgdb) p *map $4 = {header = {prev = 0xc755ec30, next = 0xc755ec30, left = 0x0, right = 0x0, start = 0x1000, end = 0xbfc00000, avail_ssize = 0x0, adj_free = 0x0, max_free = 0x0, object = { vm_object = 0x0, sub_map = 0x0}, offset = 0x0, eflags = 0x0, protection = 0x0, max_protection = 0x0, inheritance = 0x0, read_ahead = 0x0, wired_count = 0x0, next_read = 0x0, cred = 0x0, wiring_thread = 0x0}, lock = {lock_object = {lo_name = 0xc11704e7 "vm map (user)", lo_flags = 0x2330000, lo_data = 0x0, lo_witness = 0xc6d8d2b0}, sx_lock = 0x1}, system_mtx = {lock_object = {lo_name = 0xc11704b9 "vm map (system)", lo_flags = 0x1430000, lo_data = 0x0, lo_witness = 0xc6d8d110}, mtx_lock = 0x4}, nentries = 0x0, size = 0x0, timestamp = 0x5e, needs_wakeup = 0x0, system_map = 0x0, flags = 0x0, root = 0x0, pmap = 0xc755ece4, busy = 0x0} (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/kern/vfs_bio.c =================================================================== --- /usr/src/sys/kern/vfs_bio.c (revision 263107) +++ /usr/src/sys/kern/vfs_bio.c (working copy) @@ -99,7 +99,8 @@ struct buf_ops buf_ops_bio = { struct buf *buf; /* buffer header pool */ caddr_t unmapped_buf; -static struct proc *bufdaemonproc; +/* Used below and for softdep flushing threads in ufs/ffs/ffs_softdep.c */ +struct proc *bufdaemonproc; static int inmem(struct vnode *vp, daddr_t blkno); static void vm_hold_free_pages(struct buf *bp, int newbsize); Index: /usr/src/sys/ufs/ffs/ffs_softdep.c =================================================================== --- /usr/src/sys/ufs/ffs/ffs_softdep.c (revision 263107) +++ /usr/src/sys/ufs/ffs/ffs_softdep.c (working copy) @@ -908,9 +908,9 @@ static void add_to_worklist(struct worklist *, int static void wake_worklist(struct worklist *); static void wait_worklist(struct worklist *, char *); static void remove_from_worklist(struct worklist *); -static void softdep_flush(void); +static void softdep_flush(void *); static void softdep_flushjournal(struct mount *); -static int softdep_speedup(void); +static int softdep_speedup(struct ufsmount *); static void worklist_speedup(struct mount *); static int journal_mount(struct mount *, struct fs *, struct ucred *); static void journal_unmount(struct ufsmount *); @@ -962,18 +962,21 @@ static int softdep_count_dependencies(struct buf * /* * Global lock over all of soft updates. */ -static struct rwlock lk; -RW_SYSINIT(softdep_lock, &lk, "Softdep Lock"); +static struct mtx lk; +MTX_SYSINIT(softdep_lock, &lk, "Global Softdep Lock", MTX_DEF); +#define ACQUIRE_GBLLOCK(lk) mtx_lock(lk) +#define FREE_GBLLOCK(lk) mtx_unlock(lk) +#define GBLLOCK_OWNED(lk) mtx_assert((lk), MA_OWNED) + /* - * Allow per-filesystem soft-updates locking. - * For now all use the same global lock defined above. + * Per-filesystem soft-updates locking. */ -#define LOCK_PTR(ump) ((ump)->um_softdep->sd_fslock) -#define TRY_ACQUIRE_LOCK(ump) rw_try_wlock((ump)->um_softdep->sd_fslock) -#define ACQUIRE_LOCK(ump) rw_wlock((ump)->um_softdep->sd_fslock) -#define FREE_LOCK(ump) rw_wunlock((ump)->um_softdep->sd_fslock) -#define LOCK_OWNED(ump) rw_assert((ump)->um_softdep->sd_fslock, \ +#define LOCK_PTR(ump) (&(ump)->um_softdep->sd_fslock) +#define TRY_ACQUIRE_LOCK(ump) rw_try_wlock(&(ump)->um_softdep->sd_fslock) +#define ACQUIRE_LOCK(ump) rw_wlock(&(ump)->um_softdep->sd_fslock) +#define FREE_LOCK(ump) rw_wunlock(&(ump)->um_softdep->sd_fslock) +#define LOCK_OWNED(ump) rw_assert(&(ump)->um_softdep->sd_fslock, \ RA_WLOCKED) #define BUF_AREC(bp) lockallowrecurse(&(bp)->b_lock) @@ -1178,7 +1181,7 @@ workitem_free(item, type) KASSERT(ump->softdep_curdeps[item->wk_type] > 0, ("workitem_free: %s: softdep_curdeps[%s] going negative", ump->um_fs->fs_fsmnt, TYPENAME(item->wk_type))); - dep_current[item->wk_type]--; + atomic_subtract_long(&dep_current[item->wk_type], 1); ump->softdep_curdeps[item->wk_type] -= 1; free(item, DtoM(type)); } @@ -1196,11 +1199,13 @@ workitem_alloc(item, type, mp) item->wk_state = 0; ump = VFSTOUFS(mp); - ACQUIRE_LOCK(ump); + ACQUIRE_GBLLOCK(&lk); dep_current[type]++; if (dep_current[type] > dep_highuse[type]) dep_highuse[type] = dep_current[type]; dep_total[type]++; + FREE_GBLLOCK(&lk); + ACQUIRE_LOCK(ump); ump->softdep_curdeps[type] += 1; ump->softdep_deps++; ump->softdep_accdeps++; @@ -1224,11 +1229,13 @@ workitem_reassign(item, newtype) KASSERT(dep_current[item->wk_type] > 0, ("workitem_reassign: %s: dep_current[%s] going negative", VFSTOUFS(item->wk_mp)->um_fs->fs_fsmnt, TYPENAME(item->wk_type))); + ACQUIRE_GBLLOCK(&lk); + dep_current[newtype]++; dep_current[item->wk_type]--; - dep_current[newtype]++; if (dep_current[newtype] > dep_highuse[newtype]) dep_highuse[newtype] = dep_current[newtype]; dep_total[newtype]++; + FREE_GBLLOCK(&lk); item->wk_type = newtype; } @@ -1236,13 +1243,10 @@ workitem_reassign(item, newtype) * Workitem queue management */ static int max_softdeps; /* maximum number of structs before slowdown */ -static int maxindirdeps = 50; /* max number of indirdeps before slowdown */ static int tickdelay = 2; /* number of ticks to pause during slowdown */ static int proc_waiting; /* tracks whether we have a timeout posted */ static int *stat_countp; /* statistic to count in proc_waiting timeout */ static struct callout softdep_callout; -static struct mount *req_pending; -#define ALLCLEAN ((struct mount *)-1) static int req_clear_inodedeps; /* syncer process flush some inodedeps */ static int req_clear_remove; /* syncer process flush some freeblks */ static int softdep_flushcache = 0; /* Should we do BIO_FLUSH? */ @@ -1250,7 +1254,7 @@ static int softdep_flushcache = 0; /* Should we do /* * runtime statistics */ -static int stat_softdep_mounts; /* number of softdep mounted filesystems */ +static int stat_flush_threads; /* number of softdep flushing threads */ static int stat_worklist_push; /* number of worklist cleanups */ static int stat_blk_limit_push; /* number of times block limit neared */ static int stat_ino_limit_push; /* number of times inode limit neared */ @@ -1280,10 +1284,8 @@ SYSCTL_INT(_debug_softdep, OID_AUTO, max_softdeps, &max_softdeps, 0, ""); SYSCTL_INT(_debug_softdep, OID_AUTO, tickdelay, CTLFLAG_RW, &tickdelay, 0, ""); -SYSCTL_INT(_debug_softdep, OID_AUTO, maxindirdeps, CTLFLAG_RW, - &maxindirdeps, 0, ""); -SYSCTL_INT(_debug_softdep, OID_AUTO, softdep_mounts, CTLFLAG_RD, - &stat_softdep_mounts, 0, ""); +SYSCTL_INT(_debug_softdep, OID_AUTO, flush_threads, CTLFLAG_RD, + &stat_flush_threads, 0, ""); SYSCTL_INT(_debug_softdep, OID_AUTO, worklist_push, CTLFLAG_RW, &stat_worklist_push, 0,""); SYSCTL_INT(_debug_softdep, OID_AUTO, blk_limit_push, CTLFLAG_RW, @@ -1341,53 +1343,67 @@ SYSCTL_DECL(_vfs_ffs); static int compute_summary_at_mount = 0; SYSCTL_INT(_vfs_ffs, OID_AUTO, compute_summary_at_mount, CTLFLAG_RW, &compute_summary_at_mount, 0, "Recompute summary at mount"); -static struct proc *softdepproc; -static struct kproc_desc softdep_kp = { - "softdepflush", - softdep_flush, - &softdepproc -}; -SYSINIT(sdproc, SI_SUB_KTHREAD_UPDATE, SI_ORDER_ANY, kproc_start, - &softdep_kp); +static int print_threads = 1; +SYSCTL_INT(_debug_softdep, OID_AUTO, print_threads, CTLFLAG_RW, + &print_threads, 0, "Notify flusher thread start/stop"); +/* List of all filesystems mounted with soft updates */ +static TAILQ_HEAD(, mount_softdeps) softdepmounts; + +/* + * This function cleans the worklist for a filesystem. + * Each filesystem running with soft dependencies gets its own + * thread to run in this function. The thread is started up in + * softdep_mount and shutdown in softdep_unmount. They show up + * as part of the kernel "bufdaemon" process whose process + * entry is available in bufdaemonproc. + */ +static int searchfailed; +extern struct proc *bufdaemonproc; static void -softdep_flush(void) +softdep_flush(addr) + void *addr; { - struct mount *nmp; struct mount *mp; + struct thread *td; struct ufsmount *ump; - struct thread *td; - int remaining; - int progress; td = curthread; td->td_pflags |= TDP_NORUNNINGBUF; - + mp = (struct mount *)addr; + ump = VFSTOUFS(mp); + atomic_add_int(&stat_flush_threads, 1); + if (print_threads) { + if (stat_flush_threads == 1) + printf("Running %s at pid %d\n", bufdaemonproc->p_comm, + bufdaemonproc->p_pid); + printf("Start thread %s\n", td->td_name); + } for (;;) { - kproc_suspend_check(softdepproc); - remaining = progress = 0; - mtx_lock(&mountlist_mtx); - for (mp = TAILQ_FIRST(&mountlist); mp != NULL; mp = nmp) { - nmp = TAILQ_NEXT(mp, mnt_list); - if (MOUNTEDSOFTDEP(mp) == 0) - continue; - if (vfs_busy(mp, MBF_NOWAIT | MBF_MNTLSTLOCK)) - continue; - ump = VFSTOUFS(mp); - progress += softdep_process_worklist(mp, 0); - remaining += ump->softdep_on_worklist; - mtx_lock(&mountlist_mtx); - nmp = TAILQ_NEXT(mp, mnt_list); - vfs_unbusy(mp); + while (softdep_process_worklist(mp, 0) > 0 || + (MOUNTEDSUJ(mp) && + VFSTOUFS(mp)->softdep_jblocks->jb_suspended)) + kthread_suspend_check(); + ACQUIRE_LOCK(ump); + if ((ump->softdep_flags & FLUSH_CLEANUP) == 0) + msleep(&ump->softdep_flushtd, LOCK_PTR(ump), PVM, + "sdflush", hz / 2); + ump->softdep_flags &= ~FLUSH_CLEANUP; + /* + * Check to see if we are done and need to exit. + */ + if ((ump->softdep_flags & FLUSH_EXIT) == 0) { + FREE_LOCK(ump); + continue; } - mtx_unlock(&mountlist_mtx); - if (remaining && progress) - continue; - rw_wlock(&lk); - if (req_pending == NULL) - msleep(&req_pending, &lk, PVM, "sdflush", hz); - req_pending = NULL; - rw_wunlock(&lk); + ump->softdep_flags &= ~FLUSH_EXIT; + FREE_LOCK(ump); + wakeup(&ump->softdep_flags); + if (print_threads) + printf("Stop thread %s: searchfailed %d, did cleanups %d\n", td->td_name, searchfailed, ump->um_softdep->sd_cleanups); + atomic_subtract_int(&stat_flush_threads, 1); + kthread_exit(); + panic("kthread_exit failed\n"); } } @@ -1395,19 +1411,70 @@ static void worklist_speedup(mp) struct mount *mp; { - rw_assert(&lk, RA_WLOCKED); - if (req_pending == 0) { - req_pending = mp; - wakeup(&req_pending); + struct ufsmount *ump; + + ump = VFSTOUFS(mp); + LOCK_OWNED(ump); + if ((ump->softdep_flags & (FLUSH_CLEANUP | FLUSH_EXIT)) == 0) { + ump->softdep_flags |= FLUSH_CLEANUP; + if (ump->softdep_flushtd->td_wchan == &ump->softdep_flushtd) + wakeup(&ump->softdep_flushtd); } } static int -softdep_speedup(void) +softdep_speedup(ump) + struct ufsmount *ump; { + struct ufsmount *altump; + struct mount_softdeps *sdp; - worklist_speedup(ALLCLEAN); + LOCK_OWNED(ump); + worklist_speedup(ump->um_mountp); bd_speedup(); + /* + * If we have global shortages, then we need other + * filesystems to help with the cleanup. Here we wakeup a + * flusher thread for a filesystem that is over its fair + * share of resources. + */ + if (req_clear_inodedeps || req_clear_remove) { + ACQUIRE_GBLLOCK(&lk); + TAILQ_FOREACH(sdp, &softdepmounts, sd_next) { + if ((altump = sdp->sd_ump) == ump) + continue; + if (((req_clear_inodedeps && + altump->softdep_curdeps[D_INODEDEP] > + max_softdeps / stat_flush_threads) || + (req_clear_remove && + altump->softdep_curdeps[D_DIRREM] > + (max_softdeps / 2) / stat_flush_threads)) && + TRY_ACQUIRE_LOCK(altump)) + break; + } + if (sdp == NULL) { + searchfailed++; + FREE_GBLLOCK(&lk); + } else { + /* + * Move to the end of the list so we pick a + * different one on out next try. + */ + TAILQ_REMOVE(&softdepmounts, sdp, sd_next); + TAILQ_INSERT_TAIL(&softdepmounts, sdp, sd_next); + FREE_GBLLOCK(&lk); + if ((altump->softdep_flags & + (FLUSH_CLEANUP | FLUSH_EXIT)) == 0) { + altump->softdep_flags |= FLUSH_CLEANUP; + altump->um_softdep->sd_cleanups++; + if (altump->softdep_flushtd->td_wchan == + &altump->softdep_flushtd) { + wakeup(&altump->softdep_flushtd); + } + } + FREE_LOCK(altump); + } + } return (speedup_syncer()); } @@ -2123,9 +2190,14 @@ inodedep_lookup(mp, inum, flags, inodedeppp) if ((flags & DEPALLOC) == 0) return (0); /* - * If we are over our limit, try to improve the situation. + * If the system is over its limit and our filesystem is + * responsible for more than our share of that usage and + * we are not in a rush, request some inodedep cleanup. */ - if (dep_current[D_INODEDEP] > max_softdeps && (flags & NODELAY) == 0) + while (dep_current[D_INODEDEP] > max_softdeps && + (flags & NODELAY) == 0 && + ump->softdep_curdeps[D_INODEDEP] > + max_softdeps / stat_flush_threads) request_cleanup(mp, FLUSH_INODES); FREE_LOCK(ump); inodedep = malloc(sizeof(struct inodedep), @@ -2317,6 +2389,7 @@ void softdep_initialize() { + TAILQ_INIT(&softdepmounts); max_softdeps = desiredvnodes * 4; /* initialise bioops hack */ @@ -2375,7 +2448,9 @@ softdep_mount(devvp, mp, fs, cred) ump = VFSTOUFS(mp); ump->um_softdep = sdp; MNT_IUNLOCK(mp); - LOCK_PTR(ump) = &lk; + rw_init(LOCK_PTR(ump), "Per-Filesystem Softdep Lock"); + TAILQ_INSERT_TAIL(&softdepmounts, sdp, sd_next); + sdp->sd_ump = ump; LIST_INIT(&ump->softdep_workitem_pending); LIST_INIT(&ump->softdep_journal_pending); TAILQ_INIT(&ump->softdep_unlinked); @@ -2406,8 +2481,13 @@ softdep_mount(devvp, mp, fs, cred) softdep_unmount(mp); return (error); } - atomic_add_int(&stat_softdep_mounts, 1); /* + * Start our flushing thread in the bufdaemon process. + */ + kproc_kthread_add(&softdep_flush, mp, &bufdaemonproc, + &ump->softdep_flushtd, 0, 0, "softdepflush", "%s worker", + mp->mnt_stat.f_mntonname); + /* * When doing soft updates, the counters in the * superblock may have gotten out of sync. Recomputation * can take a long time and can be deferred for background @@ -2462,7 +2542,24 @@ softdep_unmount(mp) MNT_IUNLOCK(mp); journal_unmount(ump); } - atomic_subtract_int(&stat_softdep_mounts, 1); + /* + * Shut down our flushing thread. Check for NULL is if + * softdep_mount errors out before the thread has been created. + */ + if (ump->softdep_flushtd != NULL) { + ACQUIRE_LOCK(ump); + ump->softdep_flags |= FLUSH_EXIT; + wakeup(&ump->softdep_flushtd); + msleep(&ump->softdep_flags, LOCK_PTR(ump), PVM | PDROP, + "sdwait", 0); + KASSERT((ump->softdep_flags & FLUSH_EXIT) == 0, + ("Thread shutdown failed")); + } + /* + * Free up our resources. + */ + rw_destroy(LOCK_PTR(ump)); + TAILQ_REMOVE(&softdepmounts, ump->um_softdep, sd_next); hashdestroy(ump->pagedep_hashtbl, M_PAGEDEP, ump->pagedep_hash_size); hashdestroy(ump->inodedep_hashtbl, M_INODEDEP, ump->inodedep_hash_size); hashdestroy(ump->newblk_hashtbl, M_NEWBLK, ump->newblk_hash_size); @@ -2785,7 +2882,7 @@ journal_space(ump, thresh) */ limit = (max_softdeps / 10) * 9; if (dep_current[D_INODEDEP] > limit && - ump->softdep_curdeps[D_INODEDEP] > limit / stat_softdep_mounts) + ump->softdep_curdeps[D_INODEDEP] > limit / stat_flush_threads) return (0); if (thresh) thresh = jblocks->jb_min; @@ -2810,7 +2907,7 @@ journal_suspend(ump) if ((mp->mnt_kern_flag & MNTK_SUSPEND) == 0) { stat_journal_min++; mp->mnt_kern_flag |= MNTK_SUSPEND; - mp->mnt_susp_owner = FIRST_THREAD_IN_PROC(softdepproc); + mp->mnt_susp_owner = ump->softdep_flushtd; } jblocks->jb_suspended = 1; MNT_IUNLOCK(mp); @@ -2885,7 +2982,7 @@ softdep_prealloc(vp, waitok) process_removes(vp); process_truncates(vp); if (journal_space(ump, 0) == 0) { - softdep_speedup(); + softdep_speedup(ump); if (journal_space(ump, 1) == 0) journal_suspend(ump); } @@ -2929,10 +3026,10 @@ softdep_prelink(dvp, vp) } process_removes(dvp); process_truncates(dvp); - softdep_speedup(); + softdep_speedup(ump); process_worklist_item(UFSTOVFS(ump), 2, LK_NOWAIT); if (journal_space(ump, 0) == 0) { - softdep_speedup(); + softdep_speedup(ump); if (journal_space(ump, 1) == 0) journal_suspend(ump); } @@ -3254,7 +3351,7 @@ softdep_process_journal(mp, needwk, flags) if (flags != MNT_WAIT) break; printf("softdep: Out of journal space!\n"); - softdep_speedup(); + softdep_speedup(ump); msleep(jblocks, LOCK_PTR(ump), PRIBIO, "jblocks", hz); } FREE_LOCK(ump); @@ -3944,7 +4041,7 @@ free_freedep(freedep) /* * Allocate a new freework structure that may be a level in an indirect * when parent is not NULL or a top level block when it is. The top level - * freework structures are allocated without the soft updates lock held + * freework structures are allocated without the per-filesystem lock held * and before the freeblks is visible outside of softdep_setup_freeblocks(). */ static struct freework * @@ -4013,7 +4110,7 @@ cancel_jfreeblk(freeblks, blkno) /* * Allocate a new jfreeblk to journal top level block pointer when truncating - * a file. The caller must add this to the worklist when the soft updates + * a file. The caller must add this to the worklist when the per-filesystem * lock is held. */ static struct jfreeblk * @@ -7393,7 +7490,7 @@ softdep_freefile(pvp, ino, mode) clear_unlinked_inodedep(inodedep); /* * Re-acquire inodedep as we've dropped the - * soft updates lock in clear_unlinked_inodedep(). + * per-filesystem lock in clear_unlinked_inodedep(). */ inodedep_lookup(pvp->v_mount, ino, 0, &inodedep); } @@ -7939,10 +8036,8 @@ indir_trunc(freework, dbn, lbn) * If we're goingaway, free the indirdep. Otherwise it will * linger until the write completes. */ - if (goingaway) { + if (goingaway) free_indirdep(indirdep); - ump->softdep_numindirdeps -= 1; - } } FREE_LOCK(ump); /* Initialize pointers depending on block size. */ @@ -8114,7 +8209,7 @@ cancel_allocindir(aip, bp, freeblks, trunc) * Create the mkdir dependencies for . and .. in a new directory. Link them * in to a newdirblk so any subsequent additions are tracked properly. The * caller is responsible for adding the mkdir1 dependency to the journal - * and updating id_mkdiradd. This function returns with the soft updates + * and updating id_mkdiradd. This function returns with the per-filesystem * lock held. */ static struct mkdir * @@ -8932,12 +9027,16 @@ newdirrem(bp, dp, ip, isrmdir, prevdirremp) panic("newdirrem: whiteout"); dvp = ITOV(dp); /* - * If we are over our limit, try to improve the situation. + * If the system is over its limit and our filesystem is + * responsible for more than our share of that usage and + * we are not a snapshot, request some inodedep cleanup. * Limiting the number of dirrem structures will also limit * the number of freefile and freeblks structures. */ ACQUIRE_LOCK(ip->i_ump); - if (!IS_SNAPSHOT(ip) && dep_current[D_DIRREM] > max_softdeps / 2) + while (!IS_SNAPSHOT(ip) && dep_current[D_DIRREM] > max_softdeps / 2 && + ip->i_ump->softdep_curdeps[D_DIRREM] > + (max_softdeps / 2) / stat_flush_threads) (void) request_cleanup(ITOV(dp)->v_mount, FLUSH_BLOCKS); FREE_LOCK(ip->i_ump); dirrem = malloc(sizeof(struct dirrem), @@ -9888,7 +9987,7 @@ initiate_write_filepage(pagedep, bp) * Wait for all journal remove dependencies to hit the disk. * We can not allow any potentially conflicting directory adds * to be visible before removes and rollback is too difficult. - * The soft updates lock may be dropped and re-acquired, however + * The per-filesystem lock may be dropped and re-acquired, however * we hold the buf locked so the dependency can not go away. */ LIST_FOREACH(dirrem, &pagedep->pd_dirremhd, dm_next) @@ -10352,7 +10451,6 @@ cancel_indirdep(indirdep, bp, freeblks) LIST_REMOVE(indirdep, ir_next); } indirdep->ir_state |= GOINGAWAY; - VFSTOUFS(indirdep->ir_list.wk_mp)->softdep_numindirdeps += 1; /* * Pass in bp for blocks still have journal writes * pending so we can cancel them on their own. @@ -10779,7 +10877,7 @@ softdep_disk_write_complete(bp) ACQUIRE_LOCK(ump); while ((wk = LIST_FIRST(&bp->b_dep)) != NULL) { WORKLIST_REMOVE(wk); - dep_write[wk->wk_type]++; + atomic_add_long(&dep_write[wk->wk_type], 1); if (wk == owk) panic("duplicate worklist: %p\n", wk); owk = wk; @@ -11462,7 +11560,7 @@ diradd_inode_written(dap, inodedep) /* * Returns true if the bmsafemap will have rollbacks when written. Must only - * be called with the soft updates lock and the buf lock on the cg held. + * be called with the per-filesystem lock and the buf lock on the cg held. */ static int bmsafemap_backgroundwrite(bmsafemap, bp) @@ -12886,18 +12984,42 @@ softdep_slowdown(vp) if (journal_space(ump, 0) == 0) jlow = 1; } + /* + * If the system is under its limits and our filesystem is + * not responsible for more than our share of the usage and + * we are not low on journal space, then no need to slow down. + */ max_softdeps_hard = max_softdeps * 11 / 10; if (dep_current[D_DIRREM] < max_softdeps_hard / 2 && dep_current[D_INODEDEP] < max_softdeps_hard && - VFSTOUFS(vp->v_mount)->softdep_numindirdeps < maxindirdeps && - dep_current[D_FREEBLKS] < max_softdeps_hard && jlow == 0) { + dep_current[D_INDIRDEP] < max_softdeps_hard / 1000 && + dep_current[D_FREEBLKS] < max_softdeps_hard && jlow == 0 && + ump->softdep_curdeps[D_DIRREM] < + (max_softdeps_hard / 2) / stat_flush_threads && + ump->softdep_curdeps[D_INODEDEP] < + max_softdeps_hard / stat_flush_threads && + ump->softdep_curdeps[D_INDIRDEP] < + (max_softdeps_hard / 1000) / stat_flush_threads && + ump->softdep_curdeps[D_FREEBLKS] < + max_softdeps_hard / stat_flush_threads) { FREE_LOCK(ump); return (0); } - if (VFSTOUFS(vp->v_mount)->softdep_numindirdeps >= maxindirdeps || jlow) - softdep_speedup(); + /* + * If the journal is low or our filesystem is over its limit + * then speedup the cleanup. + */ + if (ump->softdep_curdeps[D_INDIRDEP] < + (max_softdeps_hard / 1000) / stat_flush_threads || jlow) + softdep_speedup(ump); stat_sync_limit_hit += 1; FREE_LOCK(ump); + /* + * We only slow down the rate at which new dependencies are + * generated if we are not using journaling. With journaling, + * the cleanup should always be sufficient to keep things + * under control. + */ if (DOINGSUJ(vp)) return (0); return (1); @@ -12955,13 +13077,12 @@ softdep_request_cleanup(fs, vp, cred, resource) return (0); } /* - * If we are in need of resources, consider pausing for - * tickdelay to give ourselves some breathing room. + * If we are in need of resources, start by cleaning up + * any block removals associated with our inode. */ ACQUIRE_LOCK(ump); process_removes(vp); process_truncates(vp); - request_cleanup(UFSTOVFS(ump), resource); FREE_LOCK(ump); /* * Now clean up at least as many resources as we will need. @@ -13094,7 +13215,7 @@ request_cleanup(mp, resource) * Next, we attempt to speed up the syncer process. If that * is successful, then we allow the process to continue. */ - if (softdep_speedup() && + if (softdep_speedup(ump) && resource != FLUSH_BLOCKS_WAIT && resource != FLUSH_INODES_WAIT) return(0); @@ -13112,15 +13233,19 @@ request_cleanup(mp, resource) case FLUSH_INODES: case FLUSH_INODES_WAIT: + ACQUIRE_GBLLOCK(&lk); stat_ino_limit_push += 1; req_clear_inodedeps += 1; + FREE_GBLLOCK(&lk); stat_countp = &stat_ino_limit_hit; break; case FLUSH_BLOCKS: case FLUSH_BLOCKS_WAIT: + ACQUIRE_GBLLOCK(&lk); stat_blk_limit_push += 1; req_clear_remove += 1; + FREE_GBLLOCK(&lk); stat_countp = &stat_blk_limit_hit; break; @@ -13131,6 +13256,8 @@ request_cleanup(mp, resource) * Hopefully the syncer daemon will catch up and awaken us. * We wait at most tickdelay before proceeding in any case. */ + ACQUIRE_GBLLOCK(&lk); + FREE_LOCK(ump); proc_waiting += 1; if (callout_pending(&softdep_callout) == FALSE) callout_reset(&softdep_callout, tickdelay > 2 ? tickdelay : 2, @@ -13138,6 +13265,8 @@ request_cleanup(mp, resource) msleep((caddr_t)&proc_waiting, &lk, PPAUSE, "softupdate", 0); proc_waiting -= 1; + FREE_GBLLOCK(&lk); + ACQUIRE_LOCK(ump); return (1); } @@ -13151,16 +13280,13 @@ pause_timer(arg) void *arg; { - rw_assert(&lk, RA_WLOCKED); + GBLLOCK_OWNED(&lk); /* * The callout_ API has acquired mtx and will hold it around this * function call. */ - *stat_countp += 1; - wakeup_one(&proc_waiting); - if (proc_waiting > 0) - callout_reset(&softdep_callout, tickdelay > 2 ? tickdelay : 2, - pause_timer, 0); + *stat_countp += proc_waiting; + wakeup(&proc_waiting); } /* @@ -13171,7 +13297,6 @@ check_clear_deps(mp) struct mount *mp; { - rw_assert(&lk, RA_WLOCKED); /* * If we are suspended, it may be because of our using * too many inodedeps, so help clear them out. @@ -13181,16 +13306,22 @@ check_clear_deps(mp) /* * General requests for cleanup of backed up dependencies */ + ACQUIRE_GBLLOCK(&lk); if (req_clear_inodedeps) { req_clear_inodedeps -= 1; + FREE_GBLLOCK(&lk); clear_inodedeps(mp); - wakeup_one(&proc_waiting); + ACQUIRE_GBLLOCK(&lk); + wakeup(&proc_waiting); } if (req_clear_remove) { req_clear_remove -= 1; + FREE_GBLLOCK(&lk); clear_remove(mp); - wakeup_one(&proc_waiting); + ACQUIRE_GBLLOCK(&lk); + wakeup(&proc_waiting); } + FREE_GBLLOCK(&lk); } /* Index: /usr/src/sys/ufs/ffs/softdep.h =================================================================== --- /usr/src/sys/ufs/ffs/softdep.h (revision 263107) +++ /usr/src/sys/ufs/ffs/softdep.h (working copy) @@ -1025,7 +1025,7 @@ TAILQ_HEAD(indir_hashhead, freework); * Allocated at mount and freed at unmount. */ struct mount_softdeps { - struct rwlock *sd_fslock; /* softdep lock */ + struct rwlock sd_fslock; /* softdep lock */ struct workhead sd_workitem_pending; /* softdep work queue */ struct worklist *sd_worklist_tail; /* Tail pointer for above */ struct workhead sd_journal_pending; /* journal work queue */ @@ -1046,15 +1046,24 @@ struct mount_softdeps { u_long sd_bmhashsize; /* bmsafemap hash table size-1*/ struct indir_hashhead *sd_indirhash; /* indir hash table */ u_long sd_indirhashsize; /* indir hash table size-1 */ - long sd_numindirdeps; /* outstanding indirdeps */ int sd_on_journal; /* Items on the journal list */ int sd_on_worklist; /* Items on the worklist */ int sd_deps; /* Total dependency count */ int sd_accdeps; /* accumulated dep count */ int sd_req; /* Wakeup when deps hits 0. */ + int sd_flags; /* comm with flushing thread */ + int sd_cleanups; /* Calls to cleanup */ + struct thread *sd_flushtd; /* thread handling flushing */ + TAILQ_ENTRY(mount_softdeps) sd_next; /* List of softdep filesystem */ + struct ufsmount *sd_ump; /* our ufsmount structure */ u_long sd_curdeps[D_LAST + 1]; /* count of current deps */ }; /* + * Flags for communicating with the syncer thread. + */ +#define FLUSH_EXIT 0x0001 /* time to exit */ +#define FLUSH_CLEANUP 0x0002 /* need to clear out softdep structures */ +/* * Keep the old names from when these were in the ufsmount structure. */ #define softdep_workitem_pending um_softdep->sd_workitem_pending @@ -1077,10 +1086,11 @@ struct mount_softdeps { #define bmsafemap_hash_size um_softdep->sd_bmhashsize #define indir_hashtbl um_softdep->sd_indirhash #define indir_hash_size um_softdep->sd_indirhashsize -#define softdep_numindirdeps um_softdep->sd_numindirdeps #define softdep_on_journal um_softdep->sd_on_journal #define softdep_on_worklist um_softdep->sd_on_worklist #define softdep_deps um_softdep->sd_deps #define softdep_accdeps um_softdep->sd_accdeps #define softdep_req um_softdep->sd_req +#define softdep_flags um_softdep->sd_flags +#define softdep_flushtd um_softdep->sd_flushtd #define softdep_curdeps um_softdep->sd_curdeps -- Test scenario: stealer.sh