GENERIC HEAD from 2018-02-23 11:20:59 UTC, r326118, vmcore.7 GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-CURRENT #1 r329862M: Sat Feb 24 06:39:59 CET 2018 pho@t2.osted.lan:/var/tmp/numa/sys/i386/compile/PHO i386 FreeBSD clang version 6.0.0 (branches/release_60 325330) (based on LLVM 6.0.0) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Entering uma_startup with 12 boot pages configured startup_alloc from "UMA Kegs", 11 boot pages left startup_alloc from "UMA Zones", 10 boot pages left startup_alloc from "UMA Zones", 9 boot pages left startup_alloc from "UMA Hash", 8 boot pages left startup_alloc from "UMA Zones", 7 boot pages left startup_alloc from "vmem btag", 6 boot pages left startup_alloc from "UMA Zones", 5 boot pages left startup_alloc from "MAP", 4 boot pages left startup_alloc from "UMA Kegs", 3 boot pages left startup_alloc from "KMAP ENTRY", 2 boot pages left startup_alloc from "vmem", 1 boot pages left Entering uma_startup2 with 0 boot pages left VT(vga): resolution 640x480 CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (1995.23-MHz 686-class CPU) Origin="GenuineIntel" Id=0x206d7 Family=0x6 Model=0x2d Stepping=7 Features=0xbfebfbff Features2=0x1fbee3ff AMD Features=0x2c100000 AMD Features2=0x1 XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 68719476736 (65536 MB) avail memory = 3049832448 (2908 MB) : Trying to mount root from ufs:/dev/gpt/ssdrootfs [rw]... Expensive timeout(9) function: 0xc0ae1e20(0xc1900d74) 0.006944578 s uhub0: 4 ports with 4 removable, self powered Setting hostuuid: 2bde2bde-f4e2-e111-aab2-001e6756b69b. Setting hostid: 0x0035ff86. Starting ddb. warning: total configured swap (17563648 pages) exceeds maximum recommended amount (1493360 pages). warning: increase kern.maxswzone or reduce amount of swap. Starting file system checks: /dev/gpt/ssdrootfs: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/gpt/ssdrootfs: clean, 46788392 free (178976 frags, 5826177 blocks, 0.3% fragmentation) /dev/da0p3: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/da0p3: clean, 24840902 free (1150 frags, 3104969 blocks, 0.0% fragmentation) panic: queue 3 underflow cpuid = 3 time = 1519453250 KDB: stack backtrace: db_trace_self_wrapper(c165892c,db2c24f0,c12436a0,d8c20800,db2c250c,...) at db_trace_self_wrapper+0x2a/frame 0xdb2c24d8 kdb_backtrace(c16528a7,5a910442,0,db2c25a4,7,...) at kdb_backtrace+0x2d/frame 0xdb2c2540 vpanic(c1665374,db2c25a4,c1665374,db2c25a4,db2c25a4,...) at vpanic+0x133/frame 0xdb2c2574 kassert_panic(c1665374,3,c1664336,6d3,ca1999b8,...) at kassert_panic+0xd9/frame 0xdb2c2598 bq_remove(ca199a14,4,c1664336,570,c0c73766,...) at bq_remove+0x15b/frame 0xdb2c25b8 binsfree(ca1999b8,c1664336,dd3,8,24,...) at binsfree+0xb2/frame 0xdb2c25f0 bdwrite(ca1999b8,4,1,9,0,...) at bdwrite+0x226/frame 0xdb2c2650 ffs_update(d94d05b0,0,d8e6bc00,0,ffffffff,...) at ffs_update+0x443/frame 0xdb2c26d0 ufs_makeinode(db2c2a28,db2c2a3c,c16a45d4,d9489000,db2c2888,...) at ufs_makeinode+0x31f/frame 0xdb2c2838 ufs_create(db2c28b8,c1869977,d9489000,db2c28ac,c16a7e2a,...) at ufs_create+0x38/frame 0xdb2c285c VOP_CREATE_APV(c194d330,db2c28b8,db2c2a3c,db2c28d0,d94f3380,...) at VOP_CREATE_APV+0xf6/frame 0xdb2c2888 vn_open_cred(db2c29e0,db2c2a98,180,0,d8e6bc00,d94a4930) at vn_open_cred+0x25c/frame 0xdb2c2960 vn_open(db2c29e0,db2c2a98,180,d94a4930,28494409,...) at vn_open+0x3d/frame 0xdb2c2988 kern_openat(d8c57a80,4,28494409,0,100206,180) at kern_openat+0x336/frame 0xdb2c2a80 sys_openat(d8c57a80,d8c57d00,c16494cb,4,c185e2ef,...) at sys_openat+0x53/frame 0xdb2c2aa8 syscall(db2c2ba8) at syscall+0x6ee/frame 0xdb2c2b9c Xint0x80_syscall() at Xint0x80_syscall+0x2b/frame 0xdb2c2b9c --- syscall (499, FreeBSD ELF32, sys_openat), eip = 0x281a4f73, esp = 0xbfbfea68, ebp = 0xbfbfea90 --- KDB: enter: panic [ thread pid 59 tid 100097 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> set $lines 0 db> show bufqueues bqempty: 5373 bqdirty: 1 Buf domain 0 freebufs 5440 lofreebufs 711 hifreebufs 1066 bufspace 5402624 maxbufspace 94683136 hibufspace 94027776 lobufspace 89326372 bufspacethresh 91677074 cleanq count 337 wakeup 0 lim 4 CPU 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, db> run pho db:0:pho> bt Tracing pid 59 tid 100097 td 0xd8c57a80 kdb_enter(c1652a37,c1652a37,0,db2c25a4,7,...) at kdb_enter+0x3a/frame 0xdb2c2540 vpanic(c1665374,db2c25a4,c1665374,db2c25a4,db2c25a4,...) at vpanic+0x150/frame 0xdb2c2574 kassert_panic(c1665374,3,c1664336,6d3,ca1999b8,...) at kassert_panic+0xd9/frame 0xdb2c2598 bq_remove(ca199a14,4,c1664336,570,c0c73766,...) at bq_remove+0x15b/frame 0xdb2c25b8 binsfree(ca1999b8,c1664336,dd3,8,24,...) at binsfree+0xb2/frame 0xdb2c25f0 bdwrite(ca1999b8,4,1,9,0,...) at bdwrite+0x226/frame 0xdb2c2650 ffs_update(d94d05b0,0,d8e6bc00,0,ffffffff,...) at ffs_update+0x443/frame 0xdb2c26d0 ufs_makeinode(db2c2a28,db2c2a3c,c16a45d4,d9489000,db2c2888,...) at ufs_makeinode+0x31f/frame 0xdb2c2838 ufs_create(db2c28b8,c1869977,d9489000,db2c28ac,c16a7e2a,...) at ufs_create+0x38/frame 0xdb2c285c VOP_CREATE_APV(c194d330,db2c28b8,db2c2a3c,db2c28d0,d94f3380,...) at VOP_CREATE_APV+0xf6/frame 0xdb2c2888 vn_open_cred(db2c29e0,db2c2a98,180,0,d8e6bc00,d94a4930) at vn_open_cred+0x25c/frame 0xdb2c2960 vn_open(db2c29e0,db2c2a98,180,d94a4930,28494409,...) at vn_open+0x3d/frame 0xdb2c2988 kern_openat(d8c57a80,4,28494409,0,100206,180) at kern_openat+0x336/frame 0xdb2c2a80 sys_openat(d8c57a80,d8c57d00,c16494cb,4,c185e2ef,...) at sys_openat+0x53/frame 0xdb2c2aa8 syscall(db2c2ba8) at syscall+0x6ee/frame 0xdb2c2b9c Xint0x80_syscall() at Xint0x80_syscall+0x2b/frame 0xdb2c2b9c --- syscall (499, FreeBSD ELF32, sys_openat), eip = 0x281a4f73, esp = 0xbfbfea68, ebp = 0xbfbfea90 --- db:0:bt> show allpcpu Current CPU: 3 cpuid = 0 dynamic pcpu = 0x6beb40 curthread = 0xd5944700: pid 11 tid 100003 "idle: cpu0" curpcb = 0xd34d0c00 fpcurthread = none idlethread = 0xd5944700: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 curvnet = 0 spin locks held: cpuid = 1 dynamic pcpu = 0x8a94b40 curthread = 0xd5944380: pid 11 tid 100004 "idle: cpu1" curpcb = 0xd34d5c00 fpcurthread = none idlethread = 0xd5944380: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 curvnet = 0 spin locks held: cpuid = 2 dynamic pcpu = 0x8a95b40 curthread = 0xd5944000: pid 11 tid 100005 "idle: cpu2" curpcb = 0xd34dac00 fpcurthread = none idlethread = 0xd5944000: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 curvnet = 0 spin locks held: cpuid = 3 dynamic pcpu = 0x8a96b40 curthread = 0xd8c57a80: pid 59 tid 100097 "mount" curpcb = 0xdb2c2c00 fpcurthread = none idlethread = 0xd5943a80: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 curvnet = 0 spin locks held: db:0:allpcpu> show alllocks Process 59 (mount) thread 0xd8c57a80 (100097) exclusive sleep mutex bufq clean subqueue lock (bufq clean subqueue lock) r = 0 (0xc1922340) locked @ kern/vfs_bio.c:1366 exclusive lockmgr bufwait (bufwait) r = 0 (0xca199a14) locked @ kern/vfs_bio.c:3736 exclusive lockmgr ufs (ufs) r = 0 (0xd94d05e4) locked @ ufs/ffs/ffs_vfsops.c:1680 exclusive lockmgr ufs (ufs) r = 0 (0xd9489034) locked @ kern/vfs_subr.c:2602 Process 14 (usb) thread 0xd8361700 (100048) exclusive sx USB config SX lock (USB config SX lock) r = 0 (0xd83cd828) locked @ dev/usb/usb_device.c:2884 Process 14 (usb) thread 0xd8923a80 (100061) exclusive sx USB config SX lock (USB config SX lock) r = 0 (0xd83cc828) locked @ dev/usb/usb_device.c:2884 db:0:alllocks> show lockedvnods Locked vnodes vnode 0xd9489000: tag ufs, type VDIR usecount 2, writecount 0, refcount 5 mountedhere 0 flags (VI_ACTIVE) v_object 0xd8db5be0 ref 0 pages 1 cleanbuf 1 dirtybuf 0 lock type ufs: EXCL by thread 0xd8c57a80 (pid 59, mount, tid 100097) #0 0xc0c4bff7 at __lockmgr_args+0x667 #1 0xc0f97c17 at ffs_lock+0xa7 #2 0xc128aaca at VOP_LOCK1_APV+0xda #3 0xc0d62021 at _vn_lock+0x61 #4 0xc0d4df51 at vget+0xc1 #5 0xc0d323db at cache_lookup+0x97b #6 0xc0d36d15 at vfs_cache_lookup+0xb5 #7 0xc12870e6 at VOP_LOOKUP_APV+0xf6 #8 0xc0d410a3 at lookup+0x663 #9 0xc0d40631 at namei+0x5f1 #10 0xc0d616f3 at vn_open_cred+0xa3 #11 0xc0d6163d at vn_open+0x3d #12 0xc0d58da6 at kern_openat+0x336 #13 0xc0d59063 at sys_openat+0x53 #14 0xc1255ade at syscall+0x6ee #15 0xc123ef5b at Xint0x80_syscall+0x2b ino 19100960, on dev gpt/ssdrootfs vnode 0xd94d05b0: tag ufs, type VREG usecount 1, writecount 0, refcount 1 flags (VI_ACTIVE) lock type ufs: EXCL by thread 0xd8c57a80 (pid 59, mount, tid 100097) #0 0xc0c4c88a at __lockmgr_args+0xefa #1 0xc0f929f5 at ffs_vgetf+0x155 #2 0xc0f92897 at ffs_vget+0x37 #3 0xc0f619fd at ffs_valloc+0x67d #4 0xc0fabc55 at ufs_makeinode+0xb5 #5 0xc0fa7c78 at ufs_create+0x38 #6 0xc1287506 at VOP_CREATE_APV+0xf6 #7 0xc0d618ac at vn_open_cred+0x25c #8 0xc0d6163d at vn_open+0x3d #9 0xc0d58da6 at kern_openat+0x336 #10 0xc0d59063 at sys_openat+0x53 #11 0xc1255ade at syscall+0x6ee #12 0xc123ef5b at Xint0x80_syscall+0x2b ino 19100988, on dev gpt/ssdrootfs db:0:lockedvnods> show mount 0xd93f6000 /dev/gpt/ssdrootfs on / (ufs) 0xd93f7000 devfs on /dev (devfs) More info: show mount db:0:mount> show page vm_cnt.v_free_count: 733048 vm_cnt.v_inactive_count: 80 vm_cnt.v_active_count: 198 vm_cnt.v_laundry_count: 0 vm_cnt.v_wire_count: 12767 vm_cnt.v_free_reserved: 1038 vm_cnt.v_free_min: 4770 vm_cnt.v_free_target: 15966 vm_cnt.v_inactive_target: 23949 db:0:page> ps pid ppid pgrp uid state wmesg wchan cmd 59 58 25 0 R+ CPU 3 mount 58 25 25 0 S+ wait 0xd8e55000 sh 25 1 25 0 Ss+ wait 0xd839ba08 sh 24 0 0 0 DL syncer 0xc1bc6f5c [syncer] 23 0 0 0 DL vlruwt 0xd8d2d358 [vnlru] 22 0 0 0 DL (threaded) [bufdaemon] 100083 D psleep 0xc1bc68ec [bufdaemon] 100086 D - 0xc1923280 [bufspacedaemon-0] 100099 D sdflush 0xdcd69884 [/ worker] 21 0 0 0 DL psleep 0xc1bcbe7c [vmdaemon] 20 0 0 0 DL (threaded) [pagedaemon] 100081 D psleep 0xc1bf90b1 [pagedaemon] 100087 D launds 0xc1bf90b8 [laundry: dom0] 100088 D umarcl 0xc0fb7d80 [uma] 19 0 0 0 DL idle 0xd839c000 [enc_daemon0] 18 0 0 0 DL - 0xc1a21a8c [rand_harvestq] 17 0 0 0 DL waiting 0xc1bf6398 [sctp_iterator] 16 0 0 0 DL - 0xc1bc6518 [soaiod4] 15 0 0 0 DL - 0xc1bc6518 [soaiod3] 9 0 0 0 DL - 0xc1bc6518 [soaiod2] 8 0 0 0 DL - 0xc1bc6518 [soaiod1] 14 0 0 0 DL (threaded) [usb] 100045 D - 0xd8235ca4 [usbus0] 100046 D - 0xd8235cd4 [usbus0] 100047 D - 0xd8235d04 [usbus0] 100048 D USBWAIT 0xc1b48729 [usbus0] 100049 D - 0xd8235d64 [usbus0] 100052 D - 0xd891ec50 [usbus1] 100053 D - 0xd891ec80 [usbus1] 100054 D - 0xd891ecb0 [usbus1] 100055 D - 0xd891ece0 [usbus1] 100056 D - 0xd891ed10 [usbus1] 100058 D - 0xd83a6ca4 [usbus2] 100059 D - 0xd83a6cd4 [usbus2] 100060 D - 0xd83a6d04 [usbus2] 100061 D USBWAIT 0xc1b4872a [usbus2] 100062 D - 0xd83a6d64 [usbus2] 7 0 0 0 DL (threaded) [cam] 100041 D - 0xc196ee40 [doneq0] 100079 D - 0xc196ed6c [scanner] 6 0 0 0 DL crypto_ 0xd5aef0c4 [crypto returns 3] 5 0 0 0 DL crypto_ 0xd5aef08c [crypto returns 2] 4 0 0 0 DL crypto_ 0xd5aef054 [crypto returns 1] 3 0 0 0 DL crypto_ 0xd5aef01c [crypto returns 0] 2 0 0 0 DL crypto_ 0xc1bcad34 [crypto] 13 0 0 0 DL (threaded) [geom] 100028 D - 0xc1bf2838 [g_event] 100029 D - 0xc1bf283c [g_up] 100030 D - 0xc1bf2840 [g_down] 12 0 0 0 WL (threaded) [intr] 100008 I [swi5: fast taskq] 100011 I [swi6: task queue] 100012 I [swi6: Giant taskq] 100022 I [swi3: vm] 100023 I [swi4: clock (0)] 100024 I [swi4: clock (1)] 100025 I [swi4: clock (2)] 100026 I [swi4: clock (3)] 100027 I [swi1: netisr 0] 100042 I [irq264: isci0] 100043 I [irq265: isci0] 100044 I [irq22: ehci0] 100050 I [irq266: hdac0] 100051 I [irq273: xhci0] 100057 I [irq20: ehci1] 100063 I [irq274: ahci0] 100064 I [irq275: hdac1] 100065 I [swi0: uart] 100066 I [irq14: ata0] 100067 I [irq15: ata1] 11 0 0 0 RL (threaded) [idle] 100003 Run CPU 0 [idle: cpu0] 100004 Run CPU 1 [idle: cpu1] 100005 Run CPU 2 [idle: cpu2] 100006 CanRun [idle: cpu3] 1 0 1 0 SLs wait 0xd5940a08 [init] 10 0 0 0 DL audit_w 0xc1bf6b50 [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D swapin 0xc1bf2864 [swapper] 100007 D - 0xd593c980 [thread taskq] 100009 D - 0xd593c880 [kqueue_ctx taskq] 100010 D - 0xd593c800 [aiod_kick taskq] 100013 D - 0xd593c680 [if_config_tqg_0] 100014 D - 0xd593c600 [softirq_0] 100015 D - 0xd593c580 [softirq_1] 100016 D - 0xd593c500 [softirq_2] 100017 D - 0xd593c480 [softirq_3] 100018 D - 0xd593c400 [if_io_tqg_0] 100019 D - 0xd593c380 [if_io_tqg_1] 100020 D - 0xd593c300 [if_io_tqg_2] 100021 D - 0xd593c280 [if_io_tqg_3] 100031 D - 0xd593c100 [firmware taskq] 100032 D - 0xd593bd80 [crypto_0] 100033 D - 0xd593bd80 [crypto_1] 100034 D - 0xd593bd80 [crypto_2] 100035 D - 0xd593bd80 [crypto_3] 100068 D - 0xd8c0c980 [mca taskq] 100075 D - 0xd8dc2d80 [acpi_task_0] 100076 D - 0xd8dc2d80 [acpi_task_1] 100077 D - 0xd8dc2d80 [acpi_task_2] 100078 D - 0xd593bc80 [CAM taskq] 100089 D - 0xd8c0b780 [/ trim] db:0:ps> allt Tracing command mount pid 59 tid 100097 td 0xd8c57a80 (CPU 3) kdb_enter(c1652a37,c1652a37,0,db2c25a4,7,...) at kdb_enter+0x3a/frame 0xdb2c2540 vpanic(c1665374,db2c25a4,c1665374,db2c25a4,db2c25a4,...) at vpanic+0x150/frame 0xdb2c2574 kassert_panic(c1665374,3,c1664336,6d3,ca1999b8,...) at kassert_panic+0xd9/frame 0xdb2c2598 bq_remove(ca199a14,4,c1664336,570,c0c73766,...) at bq_remove+0x15b/frame 0xdb2c25b8 binsfree(ca1999b8,c1664336,dd3,8,24,...) at binsfree+0xb2/frame 0xdb2c25f0 bdwrite(ca1999b8,4,1,9,0,...) at bdwrite+0x226/frame 0xdb2c2650 ffs_update(d94d05b0,0,d8e6bc00,0,ffffffff,...) at ffs_update+0x443/frame 0xdb2c26d0 ufs_makeinode(db2c2a28,db2c2a3c,c16a45d4,d9489000,db2c2888,...) at ufs_makeinode+0x31f/frame 0xdb2c2838 ufs_create(db2c28b8,c1869977,d9489000,db2c28ac,c16a7e2a,...) at ufs_create+0x38/frame 0xdb2c285c VOP_CREATE_APV(c194d330,db2c28b8,db2c2a3c,db2c28d0,d94f3380,...) at VOP_CREATE_APV+0xf6/frame 0xdb2c2888 vn_open_cred(db2c29e0,db2c2a98,180,0,d8e6bc00,d94a4930) at vn_open_cred+0x25c/frame 0xdb2c2960 vn_open(db2c29e0,db2c2a98,180,d94a4930,28494409,...) at vn_open+0x3d/frame 0xdb2c2988 kern_openat(d8c57a80,4,28494409,0,100206,180) at kern_openat+0x336/frame 0xdb2c2a80 sys_openat(d8c57a80,d8c57d00,c16494cb,4,c185e2ef,...) at sys_openat+0x53/frame 0xdb2c2aa8 syscall(db2c2ba8) at syscall+0x6ee/frame 0xdb2c2b9c Xint0x80_syscall() at Xint0x80_syscall+0x2b/frame 0xdb2c2b9c --- syscall (499, FreeBSD ELF32, sys_openat), eip = 0x281a4f73, esp = 0xbfbfea68, ebp = 0xbfbfea90 --- Tracing command sh pid 58 tid 100092 td 0xd8e56a80 sched_switch(d8e56a80,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xdb2a4800 mi_switch(104,0,c165a92d,22b,d8e56a80,...) at mi_switch+0x1f3/frame 0xdb2a4840 sleepq_switch(d8e56a80,0,c165a92d,1ca,6c,...) at sleepq_switch+0x110/frame 0xdb2a4874 sleepq_catch_signals(0,c165a92d,178,c1b66114,d8e56a80,...) at sleepq_catch_signals+0x360/frame 0xdb2a48b8 sleepq_wait_sig(d8e55000,6c,c165eca6,100,0,...) at sleepq_wait_sig+0x14/frame 0xdb2a48dc _sleep(d8e55000,d8e550a8,36c,c165eca6,0,...) at _sleep+0x386/frame 0xdb2a4930 kern_wait6(d8e56a80,7,0,0,db2a49b4,...) at kern_wait6+0x567/frame 0xdb2a4988 sys_wait4(d8e56a80,d8e56d00,c16494cb,4,0,...) at sys_wait4+0x87/frame 0xdb2a4aa8 syscall(db2a4ba8) at syscall+0x6ee/frame 0xdb2a4b9c Xint0x80_syscall() at Xint0x80_syscall+0x2b/frame 0xdb2a4b9c --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x281d9ec3, esp = 0xbfbfdcb8, ebp = 0xbfbfdcd0 --- Tracing command sh pid 25 tid 100090 td 0xd8c56000 sched_switch(d8c56000,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xdb298800 mi_switch(104,0,c165a92d,22b,d8c56000,...) at mi_switch+0x1f3/frame 0xdb298840 sleepq_switch(d8c56000,0,c165a92d,1ca,6c,...) at sleepq_switch+0x110/frame 0xdb298874 sleepq_catch_signals(0,c165a92d,178,c1b67994,d8c56000,...) at sleepq_catch_signals+0x360/frame 0xdb2988b8 sleepq_wait_sig(d839ba08,6c,c165eca6,100,0,...) at sleepq_wait_sig+0x14/frame 0xdb2988dc _sleep(d839ba08,d839bab0,36c,c165eca6,0,...) at _sleep+0x386/frame 0xdb298930 kern_wait6(d8c56000,7,0,0,db2989b4,...) at kern_wait6+0x567/frame 0xdb298988 sys_wait4(d8c56000,d8c56280,c16494cb,4,c185e2ef,...) at sys_wait4+0x87/frame 0xdb298aa8 syscall(db298ba8) at syscall+0x6ee/frame 0xdb298b9c Xint0x80_syscall() at Xint0x80_syscall+0x2b/frame 0xdb298b9c --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x281d9ec3, esp = 0xbfbfea60, ebp = 0xbfbfea78 --- Tracing command syncer pid 24 tid 100085 td 0xd5af3380 sched_switch(d5af3380,0,104,188,c1b57800,...) at sched_switch+0x503/frame 0xd35eea08 mi_switch(104,0,c165a92d,22b,d5af3380,...) at mi_switch+0x1f3/frame 0xd35eea44 sleepq_switch(d5af3380,0,c165a92d,2d6,d5af3380,...) at sleepq_switch+0x110/frame 0xd35eea78 sleepq_timedwait(c1bc6f5c,0,d35eeadc,0,0,...) at sleepq_timedwait+0x53/frame 0xd35eeab8 _cv_timedwait_sbt(c1bc6f5c,c1bc6f3c,fffffed8,0,0,...) at _cv_timedwait_sbt+0x24a/frame 0xd35eeb00 sched_sync(0,d35eeba8,c1649a8c,406,0,...) at sched_sync+0x6e0/frame 0xd35eeb6c fork_exit(c0d534c0,0,d35eeba8) at fork_exit+0x7e/frame 0xd35eeb94 fork_trampoline() at fork_trampoline+0x8/frame 0xd35eeb94 --- trap 0, eip = 0, esp = 0xd35eebe0, ebp = 0 --- Tracing command vnlru pid 23 tid 100084 td 0xd5af3700 sched_switch(d5af3700,0,104,188,c1b57140,...) at sched_switch+0x503/frame 0xd35e99f0 mi_switch(104,0,c165a92d,22b,d5af3700,...) at mi_switch+0x1f3/frame 0xd35e9a2c sleepq_switch(d5af3700,0,c165a92d,2d6,3,...) at sleepq_switch+0x110/frame 0xd35e9a60 sleepq_timedwait(d8d2d358,60,0,0,0,...) at sleepq_timedwait+0x53/frame 0xd35e9a9c _sleep(d8d2d358,c1bc6f08,260,c1668b35,fffffed8,...) at _sleep+0x332/frame 0xd35e9af0 vnlru_proc(0,d35e9ba8,c1649a8c,406,0,...) at vnlru_proc+0xbc/frame 0xd35e9b6c fork_exit(c0d52b40,0,d35e9ba8) at fork_exit+0x7e/frame 0xd35e9b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd35e9b94 --- trap 0, eip = 0, esp = 0xd35e9be0, ebp = 0 --- Tracing command bufdaemon pid 22 tid 100083 td 0xd5af3a80 sched_switch(d5af3a80,0,104,188,c1b57800,...) at sched_switch+0x503/frame 0xd35e4a38 mi_switch(104,0,c165a92d,22b,d5af3a80,...) at mi_switch+0x1f3/frame 0xd35e4a74 sleepq_switch(d5af3a80,0,c165a92d,2d6,3,...) at sleepq_switch+0x110/frame 0xd35e4aa8 sleepq_timedwait(c1bc68ec,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0xd35e4ae4 _sleep(c1bc68ec,c1921fc0,54,c16654f4,fffffed8,...) at _sleep+0x332/frame 0xd35e4b38 buf_daemon(0,d35e4ba8,c1649a8c,406,0,...) at buf_daemon+0xec/frame 0xd35e4b6c fork_exit(c0d2ffc0,0,d35e4ba8) at fork_exit+0x7e/frame 0xd35e4b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd35e4b94 --- trap 0, eip = 0, esp = 0xd35e4be0, ebp = 0 --- Tracing command bufdaemon pid 22 tid 100086 td 0xd8c57000 sched_switch(d8c57000,0,104,188,c1b57140,...) at sched_switch+0x503/frame 0xd35f3a20 mi_switch(104,0,c165a92d,22b,d8c57000,...) at mi_switch+0x1f3/frame 0xd35f3a5c sleepq_switch(d8c57000,0,c165a92d,2d6,3,...) at sleepq_switch+0x110/frame 0xd35f3a90 sleepq_timedwait(c1923280,5c,0,0,0,...) at sleepq_timedwait+0x53/frame 0xd35f3acc _sleep(c1923280,c1923200,25c,c1639c9b,fffffed8,...) at _sleep+0x332/frame 0xd35f3b20 bufspace_daemon(c1922140,d35f3ba8,c1649a8c,406,0,...) at bufspace_daemon+0x2d7/frame 0xd35f3b6c fork_exit(c0d30350,c1922140,d35f3ba8) at fork_exit+0x7e/frame 0xd35f3b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd35f3b94 --- trap 0, eip = 0, esp = 0xd35f3be0, ebp = 0 --- Tracing command bufdaemon pid 22 tid 100099 td 0xdcd58a80 sched_switch(dcd58a80,0,104,188,c1b56a80,...) at sched_switch+0x503/frame 0xdb2d1a30 mi_switch(104,0,c165a92d,22b,dcd58a80,...) at mi_switch+0x1f3/frame 0xdb2d1a6c sleepq_switch(dcd58a80,0,c165a92d,2d6,3,...) at sleepq_switch+0x110/frame 0xdb2d1aa0 sleepq_timedwait(dcd69884,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0xdb2d1adc _sleep(dcd69884,dcd69800,54,c16a09b1,7fffff6c,...) at _sleep+0x332/frame 0xdb2d1b30 softdep_flush(d93f6000,db2d1ba8,c1649a8c,406,0,...) at softdep_flush+0x199/frame 0xdb2d1b6c fork_exit(c0f78270,d93f6000,db2d1ba8) at fork_exit+0x7e/frame 0xdb2d1b94 fork_trampoline() at fork_trampoline+0x8/frame 0xdb2d1b94 --- trap 0, eip = 0, esp = 0xdb2d1be0, ebp = 0 --- Tracing command vmdaemon pid 21 tid 100082 td 0xd5af4000 sched_switch(d5af4000,0,104,188,c1b57140,...) at sched_switch+0x503/frame 0xd35dfa10 mi_switch(104,0,c165a92d,22b,d5af4000,...) at mi_switch+0x1f3/frame 0xd35dfa4c sleepq_switch(d5af4000,0,c165a92d,2b3,d5af4000,...) at sleepq_switch+0x110/frame 0xd35dfa80 sleepq_wait(c1bcbe7c,74,c16654f4,0,0,...) at sleepq_wait+0x48/frame 0xd35dfaa4 _sleep(c1bcbe7c,c1bcbe68,74,c16654f4,0,...) at _sleep+0x396/frame 0xd35dfaf8 vm_daemon(0,d35dfba8,c1649a8c,406,0,...) at vm_daemon+0xa6/frame 0xd35dfb6c fork_exit(c0fe5ae0,0,d35dfba8) at fork_exit+0x7e/frame 0xd35dfb94 fork_trampoline() at fork_trampoline+0x8/frame 0xd35dfb94 --- trap 0, eip = 0, esp = 0xd35dfbe0, ebp = 0 --- Tracing command pagedaemon pid 20 tid 100081 td 0xd5af4380 sched_switch(d5af4380,0,104,188,c1b57ec0,...) at sched_switch+0x503/frame 0xd35da970 mi_switch(104,0,c165a92d,22b,d5af4380,...) at mi_switch+0x1f3/frame 0xd35da9ac sleepq_switch(d5af4380,0,c165a92d,2d6,2,...) at sleepq_switch+0x110/frame 0xd35da9e0 sleepq_timedwait(c1bf90b1,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0xd35daa1c _sleep(c1bf90b1,c1bf8f40,254,c16654f4,1999997c,...) at _sleep+0x332/frame 0xd35daa70 vm_pageout_worker(0,0,d8d2f000,0,0,...) at vm_pageout_worker+0x321/frame 0xd35dab3c vm_pageout(0,d35daba8,c1649a8c,406,0,...) at vm_pageout+0x14a/frame 0xd35dab6c fork_exit(c0fdb4b0,0,d35daba8) at fork_exit+0x7e/frame 0xd35dab94 fork_trampoline() at fork_trampoline+0x8/frame 0xd35dab94 --- trap 0, eip = 0, esp = 0xd35dabe0, ebp = 0 --- Tracing command pagedaemon pid 20 tid 100087 td 0xd8c56a80 sched_switch(d8c56a80,0,104,188,c1b57140,...) at sched_switch+0x503/frame 0xdb2889d8 mi_switch(104,0,c165a92d,22b,d8c56a80,...) at mi_switch+0x1f3/frame 0xdb288a14 sleepq_switch(d8c56a80,0,c165a92d,2b3,d8c56a80,...) at sleepq_switch+0x110/frame 0xdb288a48 sleepq_wait(c1bf90b8,54,c16a974e,0,0,...) at sleepq_wait+0x48/frame 0xdb288a6c _sleep(c1bf90b8,c1bf7ec0,54,c16a974e,0,...) at _sleep+0x396/frame 0xdb288ac0 vm_pageout_laundry_worker(0,db288ba8,c1649a8c,406,0,...) at vm_pageout_laundry_worker+0xd18/frame 0xdb288b6c fork_exit(c0fdb660,0,db288ba8) at fork_exit+0x7e/frame 0xdb288b94 fork_trampoline() at fork_trampoline+0x8/frame 0xdb288b94 --- trap 0, eip = 0, esp = 0xdb288be0, ebp = 0 --- Tracing command pagedaemon pid 20 tid 100088 td 0xd8c56700 sched_switch(d8c56700,0,104,188,c1b57800,...) at sched_switch+0x503/frame 0xdb28da30 mi_switch(104,0,c165a92d,22b,d8c56700,...) at mi_switch+0x1f3/frame 0xdb28da6c sleepq_switch(d8c56700,0,c165a92d,2d6,c1b68c44,...) at sleepq_switch+0x110/frame 0xdb28daa0 sleepq_timedwait(c0fb7d80,54,db28db10,0,0,...) at sleepq_timedwait+0x53/frame 0xdb28dae0 _sleep(c0fb7d80,c1bcbb9c,54,c16a594b,fffffed8,...) at _sleep+0x332/frame 0xdb28db34 uma_reclaim_worker(0,db28dba8,c1649a8c,406,0,...) at uma_reclaim_worker+0x147/frame 0xdb28db6c fork_exit(c0fb8410,0,db28dba8) at fork_exit+0x7e/frame 0xdb28db94 fork_trampoline() at fork_trampoline+0x8/frame 0xdb28db94 --- trap 0, eip = 0, esp = 0xdb28dbe0, ebp = 0 --- Tracing command enc_daemon0 pid 19 tid 100080 td 0xd8c57380 sched_switch(d8c57380,0,104,188,c1b57140,...) at sched_switch+0x503/frame 0xd35d5a28 mi_switch(104,0,c165a92d,22b,d8c57380,...) at mi_switch+0x1f3/frame 0xd35d5a64 sleepq_switch(d8c57380,0,c165a92d,2b3,d8c57380,...) at sleepq_switch+0x110/frame 0xd35d5a98 sleepq_wait(d839c000,78,c12c88f0,0,0,...) at sleepq_wait+0x48/frame 0xd35d5abc _sleep(d839c000,d914745c,78,c12c88f0,0,...) at _sleep+0x396/frame 0xd35d5b10 enc_daemon(d8c32000,d35d5ba8,c1649a8c,406,0,...) at enc_daemon+0x306/frame 0xd35d5b6c fork_exit(c05247f0,d8c32000,d35d5ba8) at fork_exit+0x7e/frame 0xd35d5b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd35d5b94 --- trap 0, eip = 0, esp = 0xd35d5be0, ebp = 0 --- Tracing command rand_harvestq pid 18 tid 100074 td 0xd8e68700 sched_switch(d8e68700,0,104,188,c1b57140,...) at sched_switch+0x503/frame 0xd35b7a08 mi_switch(104,0,c165a92d,22b,d8e68700,...) at mi_switch+0x1f3/frame 0xd35b7a44 sleepq_switch(d8e68700,0,c165a92d,2d6,2,...) at sleepq_switch+0x110/frame 0xd35b7a78 sleepq_timedwait(c1a21a8c,0,0,0,0,...) at sleepq_timedwait+0x53/frame 0xd35b7ab4 _sleep(c1a21a8c,0,0,c1639c9b,19999999,...) at _sleep+0x332/frame 0xd35b7b08 random_kthread(0,d35b7ba8,c1649a8c,406,0,...) at random_kthread+0x7c/frame 0xd35b7b6c fork_exit(c0982110,0,d35b7ba8) at fork_exit+0x7e/frame 0xd35b7b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd35b7b94 --- trap 0, eip = 0, esp = 0xd35b7be0, ebp = 0 --- Tracing command sctp_iterator pid 17 tid 100073 td 0xd8e68a80 sched_switch(d8e68a80,0,104,188,c1b57140,...) at sched_switch+0x503/frame 0xd35b2a58 mi_switch(104,0,c165a92d,22b,d8e68a80,...) at mi_switch+0x1f3/frame 0xd35b2a94 sleepq_switch(d8e68a80,0,c165a92d,2b3,d8e68a80,...) at sleepq_switch+0x110/frame 0xd35b2ac8 sleepq_wait(c1bf6398,0,c168334f,0,0,...) at sleepq_wait+0x48/frame 0xd35b2aec _sleep(c1bf6398,c1bf6360,0,c168334f,0,...) at _sleep+0x396/frame 0xd35b2b40 sctp_iterator_thread(0,d35b2ba8,c1649a8c,406,0,...) at sctp_iterator_thread+0x7c/frame 0xd35b2b6c fork_exit(c0e1e730,0,d35b2ba8) at fork_exit+0x7e/frame 0xd35b2b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd35b2b94 --- trap 0, eip = 0, esp = 0xd35b2be0, ebp = 0 --- Tracing command soaiod4 pid 16 tid 100072 td 0xd5af4700 sched_switch(d5af4700,0,104,188,c1b57ec0,...) at sched_switch+0x503/frame 0xd35aca30 mi_switch(104,0,c165a92d,22b,d5af4700,...) at mi_switch+0x1f3/frame 0xd35aca6c sleepq_switch(d5af4700,0,c165a92d,2d6,1d,...) at sleepq_switch+0x110/frame 0xd35acaa0 sleepq_timedwait(c1bc6518,0,1d,0,0,...) at sleepq_timedwait+0x53/frame 0xd35acadc _sleep(c1bc6518,c1bc64f8,0,c1639c9b,ffffdd50,...) at _sleep+0x332/frame 0xd35acb30 soaio_kproc_loop(4,d35acba8,c1649a8c,406,0,...) at soaio_kproc_loop+0x1a7/frame 0xd35acb6c fork_exit(c0cf03c0,4,d35acba8) at fork_exit+0x7e/frame 0xd35acb94 fork_trampoline() at fork_trampoline+0x8/frame 0xd35acb94 --- trap 0, eip = 0, esp = 0xd35acbe0, ebp = 0 --- Tracing command soaiod3 pid 15 tid 100071 td 0xd5af4a80 sched_switch(d5af4a80,0,104,188,c1b57ec0,...) at sched_switch+0x503/frame 0xd35a7a30 mi_switch(104,0,c165a92d,22b,d5af4a80,...) at mi_switch+0x1f3/frame 0xd35a7a6c sleepq_switch(d5af4a80,0,c165a92d,2d6,1d,...) at sleepq_switch+0x110/frame 0xd35a7aa0 sleepq_timedwait(c1bc6518,0,1d,0,0,...) at sleepq_timedwait+0x53/frame 0xd35a7adc _sleep(c1bc6518,c1bc64f8,0,c1639c9b,ffffdd50,...) at _sleep+0x332/frame 0xd35a7b30 soaio_kproc_loop(3,d35a7ba8,c1649a8c,406,0,...) at soaio_kproc_loop+0x1a7/frame 0xd35a7b6c fork_exit(c0cf03c0,3,d35a7ba8) at fork_exit+0x7e/frame 0xd35a7b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd35a7b94 --- trap 0, eip = 0, esp = 0xd35a7be0, ebp = 0 --- Tracing command soaiod2 pid 9 tid 100070 td 0xd5af5000 sched_switch(d5af5000,0,104,188,c1b57ec0,...) at sched_switch+0x503/frame 0xd35a2a30 mi_switch(104,0,c165a92d,22b,d5af5000,...) at mi_switch+0x1f3/frame 0xd35a2a6c sleepq_switch(d5af5000,0,c165a92d,2d6,1d,...) at sleepq_switch+0x110/frame 0xd35a2aa0 sleepq_timedwait(c1bc6518,0,1d,0,0,...) at sleepq_timedwait+0x53/frame 0xd35a2adc _sleep(c1bc6518,c1bc64f8,0,c1639c9b,ffffdd50,...) at _sleep+0x332/frame 0xd35a2b30 soaio_kproc_loop(2,d35a2ba8,c1649a8c,406,0,...) at soaio_kproc_loop+0x1a7/frame 0xd35a2b6c fork_exit(c0cf03c0,2,d35a2ba8) at fork_exit+0x7e/frame 0xd35a2b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd35a2b94 --- trap 0, eip = 0, esp = 0xd35a2be0, ebp = 0 --- Tracing command soaiod1 pid 8 tid 100069 td 0xd5af5380 sched_switch(d5af5380,0,104,188,c1b57ec0,...) at sched_switch+0x503/frame 0xd359da30 mi_switch(104,0,c165a92d,22b,d5af5380,...) at mi_switch+0x1f3/frame 0xd359da6c sleepq_switch(d5af5380,0,c165a92d,2d6,1d,...) at sleepq_switch+0x110/frame 0xd359daa0 sleepq_timedwait(c1bc6518,0,1d,0,0,...) at sleepq_timedwait+0x53/frame 0xd359dadc _sleep(c1bc6518,c1bc64f8,0,c1639c9b,ffffdd50,...) at _sleep+0x332/frame 0xd359db30 soaio_kproc_loop(1,d359dba8,c1649a8c,406,0,...) at soaio_kproc_loop+0x1a7/frame 0xd359db6c fork_exit(c0cf03c0,1,d359dba8) at fork_exit+0x7e/frame 0xd359db94 fork_trampoline() at fork_trampoline+0x8/frame 0xd359db94 --- trap 0, eip = 0, esp = 0xd359dbe0, ebp = 0 --- Tracing command usb pid 14 tid 100045 td 0xd8362380 sched_switch(d8362380,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd77e0a68 mi_switch(104,0,c165a92d,22b,d8362380,...) at mi_switch+0x1f3/frame 0xd77e0aa8 sleepq_switch(d8362380,0,c165a92d,2b3,c1916078,...) at sleepq_switch+0x110/frame 0xd77e0adc sleepq_wait(d8235ca4,0,d77e0b20,1,0,...) at sleepq_wait+0x48/frame 0xd77e0b00 _cv_wait(d8235ca4,d8235ed0,c161b7a6,78,d8235cac,...) at _cv_wait+0x21f/frame 0xd77e0b44 usb_process(d8235c9c,d77e0ba8,c1649a8c,406,0,...) at usb_process+0x140/frame 0xd77e0b6c fork_exit(c0ab2930,d8235c9c,d77e0ba8) at fork_exit+0x7e/frame 0xd77e0b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd77e0b94 --- trap 0, eip = 0, esp = 0xd77e0be0, ebp = 0 --- Tracing command usb pid 14 tid 100046 td 0xd8362000 sched_switch(d8362000,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd77e5a68 mi_switch(104,0,c165a92d,22b,d8362000,...) at mi_switch+0x1f3/frame 0xd77e5aa8 sleepq_switch(d8362000,0,c165a92d,2b3,c1916078,...) at sleepq_switch+0x110/frame 0xd77e5adc sleepq_wait(d8235cd4,0,d77e5b20,1,0,...) at sleepq_wait+0x48/frame 0xd77e5b00 _cv_wait(d8235cd4,d8235ed0,c161b7a6,78,d8235cdc,...) at _cv_wait+0x21f/frame 0xd77e5b44 usb_process(d8235ccc,d77e5ba8,c1649a8c,406,0,...) at usb_process+0x140/frame 0xd77e5b6c fork_exit(c0ab2930,d8235ccc,d77e5ba8) at fork_exit+0x7e/frame 0xd77e5b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd77e5b94 --- trap 0, eip = 0, esp = 0xd77e5be0, ebp = 0 --- Tracing command usb pid 14 tid 100047 td 0xd8361a80 sched_switch(d8361a80,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd77eaa68 mi_switch(104,0,c165a92d,22b,d8361a80,...) at mi_switch+0x1f3/frame 0xd77eaaa8 sleepq_switch(d8361a80,0,c165a92d,2b3,c1916078,...) at sleepq_switch+0x110/frame 0xd77eaadc sleepq_wait(d8235d04,0,d77eab20,1,0,...) at sleepq_wait+0x48/frame 0xd77eab00 _cv_wait(d8235d04,d8235ed0,c161b7a6,78,d8235d0c,...) at _cv_wait+0x21f/frame 0xd77eab44 usb_process(d8235cfc,d77eaba8,c1649a8c,406,0,...) at usb_process+0x140/frame 0xd77eab6c fork_exit(c0ab2930,d8235cfc,d77eaba8) at fork_exit+0x7e/frame 0xd77eab94 fork_trampoline() at fork_trampoline+0x8/frame 0xd77eab94 --- trap 0, eip = 0, esp = 0xd77eabe0, ebp = 0 --- Tracing command usb pid 14 tid 100048 td 0xd8361700 sched_switch(d8361700,0,104,188,c1b57140,...) at sched_switch+0x503/frame 0xd77ef7d8 mi_switch(104,0,c165a92d,22b,d8361700,...) at mi_switch+0x1f3/frame 0xd77ef814 sleepq_switch(d8361700,0,c165a92d,2d6,3,...) at sleepq_switch+0x110/frame 0xd77ef848 sleepq_timedwait(c1b48729,0,0,0,0,...) at sleepq_timedwait+0x53/frame 0xd77ef888 _sleep(c1b48729,0,0,c161c6e8,6978d484,...) at _sleep+0x332/frame 0xd77ef8dc pause_sbt(c161c6e8,6978d484,0,0,0,100) at pause_sbt+0x121/frame 0xd77ef918 usb_pause_mtx(0,19b,2,8,c00c0002,...) at usb_pause_mtx+0xec/frame 0xd77ef940 uhub_attach(d8dc2c00,d8dc2c00,d77efa58,c16577ab,80000003,...) at uhub_attach+0x87c/frame 0xd77efa10 device_attach(d8dc2c00,4,c165702c,b31,c0caf017,...) at device_attach+0x4e4/frame 0xd77efa6c device_probe_and_attach(d8dc2c00,d77efac8,ffffffff,b4b,1,...) at device_probe_and_attach+0x6e/frame 0xd77efa98 usb_probe_and_attach(d83cd800,ff,0,0,0,...) at usb_probe_and_attach+0x329/frame 0xd77efb0c usb_bus_attach(d8235ddc,d8235ed0,c161b7a6,78,d8235d3c,...) at usb_bus_attach+0x1be/frame 0xd77efb44 usb_process(d8235d2c,d77efba8,c1649a8c,406,0,...) at usb_process+0xf4/frame 0xd77efb6c fork_exit(c0ab2930,d8235d2c,d77efba8) at fork_exit+0x7e/frame 0xd77efb94 fork_trampoline() at fork_trampoline+0x8/frame 0xd77efb94 --- trap 0, eip = 0, esp = 0xd77efbe0, ebp = 0 --- Tracing command usb pid 14 tid 100049 td 0xd8361380 sched_switch(d8361380,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd77f4a68 mi_switch(104,0,c165a92d,22b,d8361380,...) at mi_switch+0x1f3/frame 0xd77f4aa8 sleepq_switch(d8361380,0,c165a92d,2b3,c1916078,...) at sleepq_switch+0x110/frame 0xd77f4adc sleepq_wait(d8235d64,0,d77f4b20,1,0,...) at sleepq_wait+0x48/frame 0xd77f4b00 _cv_wait(d8235d64,d8235ed0,c161b7a6,78,d8235d6c,...) at _cv_wait+0x21f/frame 0xd77f4b44 usb_process(d8235d5c,d77f4ba8,c1649a8c,406,0,...) at usb_process+0x140/frame 0xd77f4b6c fork_exit(c0ab2930,d8235d5c,d77f4ba8) at fork_exit+0x7e/frame 0xd77f4b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd77f4b94 --- trap 0, eip = 0, esp = 0xd77f4be0, ebp = 0 --- Tracing command usb pid 14 tid 100052 td 0xd8358700 sched_switch(d8358700,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd3541a68 mi_switch(104,0,c165a92d,22b,d8358700,...) at mi_switch+0x1f3/frame 0xd3541aa8 sleepq_switch(d8358700,0,c165a92d,2b3,c1916078,...) at sleepq_switch+0x110/frame 0xd3541adc sleepq_wait(d891ec50,0,d3541b20,1,0,...) at sleepq_wait+0x48/frame 0xd3541b00 _cv_wait(d891ec50,d891ee7c,c161b7a6,78,d891ec58,...) at _cv_wait+0x21f/frame 0xd3541b44 usb_process(d891ec48,d3541ba8,c1649a8c,406,0,...) at usb_process+0x140/frame 0xd3541b6c fork_exit(c0ab2930,d891ec48,d3541ba8) at fork_exit+0x7e/frame 0xd3541b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd3541b94 --- trap 0, eip = 0, esp = 0xd3541be0, ebp = 0 --- Tracing command usb pid 14 tid 100053 td 0xd8358380 sched_switch(d8358380,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd3546a68 mi_switch(104,0,c165a92d,22b,d8358380,...) at mi_switch+0x1f3/frame 0xd3546aa8 sleepq_switch(d8358380,0,c165a92d,2b3,c1916078,...) at sleepq_switch+0x110/frame 0xd3546adc sleepq_wait(d891ec80,0,d3546b20,1,0,...) at sleepq_wait+0x48/frame 0xd3546b00 _cv_wait(d891ec80,d891ee7c,c161b7a6,78,d891ec88,...) at _cv_wait+0x21f/frame 0xd3546b44 usb_process(d891ec78,d3546ba8,c1649a8c,406,0,...) at usb_process+0x140/frame 0xd3546b6c fork_exit(c0ab2930,d891ec78,d3546ba8) at fork_exit+0x7e/frame 0xd3546b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd3546b94 --- trap 0, eip = 0, esp = 0xd3546be0, ebp = 0 --- Tracing command usb pid 14 tid 100054 td 0xd8358000 sched_switch(d8358000,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd354ba68 mi_switch(104,0,c165a92d,22b,d8358000,...) at mi_switch+0x1f3/frame 0xd354baa8 sleepq_switch(d8358000,0,c165a92d,2b3,c1916078,...) at sleepq_switch+0x110/frame 0xd354badc sleepq_wait(d891ecb0,0,d354bb20,1,0,...) at sleepq_wait+0x48/frame 0xd354bb00 _cv_wait(d891ecb0,d891ee7c,c161b7a6,78,d891ecb8,...) at _cv_wait+0x21f/frame 0xd354bb44 usb_process(d891eca8,d354bba8,c1649a8c,406,0,...) at usb_process+0x140/frame 0xd354bb6c fork_exit(c0ab2930,d891eca8,d354bba8) at fork_exit+0x7e/frame 0xd354bb94 fork_trampoline() at fork_trampoline+0x8/frame 0xd354bb94 --- trap 0, eip = 0, esp = 0xd354bbe0, ebp = 0 --- Tracing command usb pid 14 tid 100055 td 0xd8925380 sched_switch(d8925380,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd3550a68 mi_switch(104,0,c165a92d,22b,d8925380,...) at mi_switch+0x1f3/frame 0xd3550aa8 sleepq_switch(d8925380,0,c165a92d,2b3,c1916078,...) at sleepq_switch+0x110/frame 0xd3550adc sleepq_wait(d891ece0,0,d3550b20,1,0,...) at sleepq_wait+0x48/frame 0xd3550b00 _cv_wait(d891ece0,d891ee7c,c161b7a6,78,d891ece8,...) at _cv_wait+0x21f/frame 0xd3550b44 usb_process(d891ecd8,d3550ba8,c1649a8c,406,0,...) at usb_process+0x140/frame 0xd3550b6c fork_exit(c0ab2930,d891ecd8,d3550ba8) at fork_exit+0x7e/frame 0xd3550b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd3550b94 --- trap 0, eip = 0, esp = 0xd3550be0, ebp = 0 --- Tracing command usb pid 14 tid 100056 td 0xd8925000 sched_switch(d8925000,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd3555a68 mi_switch(104,0,c165a92d,22b,d8925000,...) at mi_switch+0x1f3/frame 0xd3555aa8 sleepq_switch(d8925000,0,c165a92d,2b3,c1916078,...) at sleepq_switch+0x110/frame 0xd3555adc sleepq_wait(d891ed10,0,d3555b20,1,0,...) at sleepq_wait+0x48/frame 0xd3555b00 _cv_wait(d891ed10,d891ee7c,c161b7a6,78,d891ed18,...) at _cv_wait+0x21f/frame 0xd3555b44 usb_process(d891ed08,d3555ba8,c1649a8c,406,0,...) at usb_process+0x140/frame 0xd3555b6c fork_exit(c0ab2930,d891ed08,d3555ba8) at fork_exit+0x7e/frame 0xd3555b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd3555b94 --- trap 0, eip = 0, esp = 0xd3555be0, ebp = 0 --- Tracing command usb pid 14 tid 100058 td 0xd8924700 sched_switch(d8924700,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd3560a68 mi_switch(104,0,c165a92d,22b,d8924700,...) at mi_switch+0x1f3/frame 0xd3560aa8 sleepq_switch(d8924700,0,c165a92d,2b3,c1916078,...) at sleepq_switch+0x110/frame 0xd3560adc sleepq_wait(d83a6ca4,0,d3560b20,1,0,...) at sleepq_wait+0x48/frame 0xd3560b00 _cv_wait(d83a6ca4,d83a6ed0,c161b7a6,78,d83a6cac,...) at _cv_wait+0x21f/frame 0xd3560b44 usb_process(d83a6c9c,d3560ba8,c1649a8c,406,0,...) at usb_process+0x140/frame 0xd3560b6c fork_exit(c0ab2930,d83a6c9c,d3560ba8) at fork_exit+0x7e/frame 0xd3560b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd3560b94 --- trap 0, eip = 0, esp = 0xd3560be0, ebp = 0 --- Tracing command usb pid 14 tid 100059 td 0xd8924380 sched_switch(d8924380,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd3565a68 mi_switch(104,0,c165a92d,22b,d8924380,...) at mi_switch+0x1f3/frame 0xd3565aa8 sleepq_switch(d8924380,0,c165a92d,2b3,c1916078,...) at sleepq_switch+0x110/frame 0xd3565adc sleepq_wait(d83a6cd4,0,d3565b20,1,0,...) at sleepq_wait+0x48/frame 0xd3565b00 _cv_wait(d83a6cd4,d83a6ed0,c161b7a6,78,d83a6cdc,...) at _cv_wait+0x21f/frame 0xd3565b44 usb_process(d83a6ccc,d3565ba8,c1649a8c,406,0,...) at usb_process+0x140/frame 0xd3565b6c fork_exit(c0ab2930,d83a6ccc,d3565ba8) at fork_exit+0x7e/frame 0xd3565b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd3565b94 --- trap 0, eip = 0, esp = 0xd3565be0, ebp = 0 --- Tracing command usb pid 14 tid 100060 td 0xd8924000 sched_switch(d8924000,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd356aa68 mi_switch(104,0,c165a92d,22b,d8924000,...) at mi_switch+0x1f3/frame 0xd356aaa8 sleepq_switch(d8924000,0,c165a92d,2b3,c1916078,...) at sleepq_switch+0x110/frame 0xd356aadc sleepq_wait(d83a6d04,0,d356ab20,1,0,...) at sleepq_wait+0x48/frame 0xd356ab00 _cv_wait(d83a6d04,d83a6ed0,c161b7a6,78,d83a6d0c,...) at _cv_wait+0x21f/frame 0xd356ab44 usb_process(d83a6cfc,d356aba8,c1649a8c,406,0,...) at usb_process+0x140/frame 0xd356ab6c fork_exit(c0ab2930,d83a6cfc,d356aba8) at fork_exit+0x7e/frame 0xd356ab94 fork_trampoline() at fork_trampoline+0x8/frame 0xd356ab94 --- trap 0, eip = 0, esp = 0xd356abe0, ebp = 0 --- Tracing command usb pid 14 tid 100061 td 0xd8923a80 sched_switch(d8923a80,0,104,188,c1b57800,...) at sched_switch+0x503/frame 0xd356f7d8 mi_switch(104,0,c165a92d,22b,d8923a80,...) at mi_switch+0x1f3/frame 0xd356f814 sleepq_switch(d8923a80,0,c165a92d,2d6,3,...) at sleepq_switch+0x110/frame 0xd356f848 sleepq_timedwait(c1b4872a,0,0,0,0,...) at sleepq_timedwait+0x53/frame 0xd356f888 _sleep(c1b4872a,0,0,c161c6e8,6978d484,...) at _sleep+0x332/frame 0xd356f8dc pause_sbt(c161c6e8,6978d484,0,0,0,100) at pause_sbt+0x121/frame 0xd356f918 usb_pause_mtx(0,19b,2,8,c00c0002,...) at usb_pause_mtx+0xec/frame 0xd356f940 uhub_attach(d8c0bd80,d8c0bd80,d356fa58,c16577ab,80000003,...) at uhub_attach+0x87c/frame 0xd356fa10 device_attach(d8c0bd80,4,c165702c,b31,c0caf017,...) at device_attach+0x4e4/frame 0xd356fa6c device_probe_and_attach(d8c0bd80,d356fac8,ffffffff,b4b,1,...) at device_probe_and_attach+0x6e/frame 0xd356fa98 usb_probe_and_attach(d83cc800,ff,0,0,0,...) at usb_probe_and_attach+0x329/frame 0xd356fb0c usb_bus_attach(d83a6ddc,d83a6ed0,c161b7a6,78,d83a6d3c,...) at usb_bus_attach+0x1be/frame 0xd356fb44 usb_process(d83a6d2c,d356fba8,c1649a8c,406,0,...) at usb_process+0xf4/frame 0xd356fb6c fork_exit(c0ab2930,d83a6d2c,d356fba8) at fork_exit+0x7e/frame 0xd356fb94 fork_trampoline() at fork_trampoline+0x8/frame 0xd356fb94 --- trap 0, eip = 0, esp = 0xd356fbe0, ebp = 0 --- Tracing command usb pid 14 tid 100062 td 0xd8923700 sched_switch(d8923700,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd3574a68 mi_switch(104,0,c165a92d,22b,d8923700,...) at mi_switch+0x1f3/frame 0xd3574aa8 sleepq_switch(d8923700,0,c165a92d,2b3,c1916078,...) at sleepq_switch+0x110/frame 0xd3574adc sleepq_wait(d83a6d64,0,d3574b20,1,0,...) at sleepq_wait+0x48/frame 0xd3574b00 _cv_wait(d83a6d64,d83a6ed0,c161b7a6,78,d83a6d6c,...) at _cv_wait+0x21f/frame 0xd3574b44 usb_process(d83a6d5c,d3574ba8,c1649a8c,406,0,...) at usb_process+0x140/frame 0xd3574b6c fork_exit(c0ab2930,d83a6d5c,d3574ba8) at fork_exit+0x7e/frame 0xd3574b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd3574b94 --- trap 0, eip = 0, esp = 0xd3574be0, ebp = 0 --- Tracing command cam pid 7 tid 100041 td 0xd5af5700 sched_switch(d5af5700,0,104,188,c1b56a80,...) at sched_switch+0x503/frame 0xd76aea38 mi_switch(104,0,c165a92d,22b,d5af5700,...) at mi_switch+0x1f3/frame 0xd76aea74 sleepq_switch(d5af5700,0,c165a92d,2b3,d5af5700,...) at sleepq_switch+0x110/frame 0xd76aeaa8 sleepq_wait(c196ee40,5c,c1639c9b,0,0,...) at sleepq_wait+0x48/frame 0xd76aeacc _sleep(c196ee40,c196ee00,5c,c1639c9b,0,...) at _sleep+0x396/frame 0xd76aeb20 xpt_done_td(c196ee00,d76aeba8,c1649a8c,406,0,...) at xpt_done_td+0x9b/frame 0xd76aeb6c fork_exit(c04e2830,c196ee00,d76aeba8) at fork_exit+0x7e/frame 0xd76aeb94 fork_trampoline() at fork_trampoline+0x8/frame 0xd76aeb94 --- trap 0, eip = 0, esp = 0xd76aebe0, ebp = 0 --- Tracing command cam pid 7 tid 100079 td 0xd8e67380 sched_switch(d8e67380,0,104,188,c1b57140,...) at sched_switch+0x503/frame 0xd35d0a30 mi_switch(104,0,c165a92d,22b,d8e67380,...) at mi_switch+0x1f3/frame 0xd35d0a6c sleepq_switch(d8e67380,0,c165a92d,2b3,d8e67380,...) at sleepq_switch+0x110/frame 0xd35d0aa0 sleepq_wait(c196ed6c,5c,c1639c9b,0,0,...) at sleepq_wait+0x48/frame 0xd35d0ac4 _sleep(c196ed6c,c196edc0,5c,c1639c9b,0,...) at _sleep+0x396/frame 0xd35d0b18 xpt_scanner_thread(0,d35d0ba8,c1649a8c,406,0,...) at xpt_scanner_thread+0x97/frame 0xd35d0b6c fork_exit(c04e2bb0,0,d35d0ba8) at fork_exit+0x7e/frame 0xd35d0b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd35d0b94 --- trap 0, eip = 0, esp = 0xd35d0be0, ebp = 0 --- Tracing command crypto returns 3 pid 6 tid 100040 td 0xd5af5a80 sched_switch(d5af5a80,0,104,188,c1b57800,...) at sched_switch+0x503/frame 0xd76a9a30 mi_switch(104,0,c165a92d,22b,d5af5a80,...) at mi_switch+0x1f3/frame 0xd76a9a6c sleepq_switch(d5af5a80,0,c165a92d,2b3,d5af5a80,...) at sleepq_switch+0x110/frame 0xd76a9aa0 sleepq_wait(d5aef0c4,6c,c1697c39,0,0,...) at sleepq_wait+0x48/frame 0xd76a9ac4 _sleep(d5aef0c4,d5aef0a8,6c,c1697c39,0,...) at _sleep+0x396/frame 0xd76a9b18 crypto_ret_proc(d5aef0a8,d76a9ba8,c1649a8c,406,0,...) at crypto_ret_proc+0x241/frame 0xd76a9b6c fork_exit(c0f1fc60,d5aef0a8,d76a9ba8) at fork_exit+0x7e/frame 0xd76a9b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd76a9b94 --- trap 0, eip = 0, esp = 0xd76a9be0, ebp = 0 --- Tracing command crypto returns 2 pid 5 tid 100039 td 0xd59a0700 sched_switch(d59a0700,0,104,188,c1b57800,...) at sched_switch+0x503/frame 0xd76a4a30 mi_switch(104,0,c165a92d,22b,d59a0700,...) at mi_switch+0x1f3/frame 0xd76a4a6c sleepq_switch(d59a0700,0,c165a92d,2b3,d59a0700,...) at sleepq_switch+0x110/frame 0xd76a4aa0 sleepq_wait(d5aef08c,6c,c1697c39,0,0,...) at sleepq_wait+0x48/frame 0xd76a4ac4 _sleep(d5aef08c,d5aef070,6c,c1697c39,0,...) at _sleep+0x396/frame 0xd76a4b18 crypto_ret_proc(d5aef070,d76a4ba8,c1649a8c,406,0,...) at crypto_ret_proc+0x241/frame 0xd76a4b6c fork_exit(c0f1fc60,d5aef070,d76a4ba8) at fork_exit+0x7e/frame 0xd76a4b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd76a4b94 --- trap 0, eip = 0, esp = 0xd76a4be0, ebp = 0 --- Tracing command crypto returns 1 pid 4 tid 100038 td 0xd59a0a80 sched_switch(d59a0a80,0,104,188,c1b57800,...) at sched_switch+0x503/frame 0xd769fa30 mi_switch(104,0,c165a92d,22b,d59a0a80,...) at mi_switch+0x1f3/frame 0xd769fa6c sleepq_switch(d59a0a80,0,c165a92d,2b3,d59a0a80,...) at sleepq_switch+0x110/frame 0xd769faa0 sleepq_wait(d5aef054,6c,c1697c39,0,0,...) at sleepq_wait+0x48/frame 0xd769fac4 _sleep(d5aef054,d5aef038,6c,c1697c39,0,...) at _sleep+0x396/frame 0xd769fb18 crypto_ret_proc(d5aef038,d769fba8,c1649a8c,406,0,...) at crypto_ret_proc+0x241/frame 0xd769fb6c fork_exit(c0f1fc60,d5aef038,d769fba8) at fork_exit+0x7e/frame 0xd769fb94 fork_trampoline() at fork_trampoline+0x8/frame 0xd769fb94 --- trap 0, eip = 0, esp = 0xd769fbe0, ebp = 0 --- Tracing command crypto returns 0 pid 3 tid 100037 td 0xd5a17000 sched_switch(d5a17000,0,104,188,c1b57ec0,...) at sched_switch+0x503/frame 0xd769aa30 mi_switch(104,0,c165a92d,22b,d5a17000,...) at mi_switch+0x1f3/frame 0xd769aa6c sleepq_switch(d5a17000,0,c165a92d,2b3,d5a17000,...) at sleepq_switch+0x110/frame 0xd769aaa0 sleepq_wait(d5aef01c,6c,c1697c39,0,0,...) at sleepq_wait+0x48/frame 0xd769aac4 _sleep(d5aef01c,d5aef000,6c,c1697c39,0,...) at _sleep+0x396/frame 0xd769ab18 crypto_ret_proc(d5aef000,d769aba8,c1649a8c,406,0,...) at crypto_ret_proc+0x241/frame 0xd769ab6c fork_exit(c0f1fc60,d5aef000,d769aba8) at fork_exit+0x7e/frame 0xd769ab94 fork_trampoline() at fork_trampoline+0x8/frame 0xd769ab94 --- trap 0, eip = 0, esp = 0xd769abe0, ebp = 0 --- Tracing command crypto pid 2 tid 100036 td 0xd5a17380 sched_switch(d5a17380,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd7695a40 mi_switch(104,0,c165a92d,22b,d5a17380,...) at mi_switch+0x1f3/frame 0xd7695a80 sleepq_switch(d5a17380,0,c165a92d,2b3,d5a17380,...) at sleepq_switch+0x110/frame 0xd7695ab4 sleepq_wait(c1bcad34,6c,c1697c2d,0,0,...) at sleepq_wait+0x48/frame 0xd7695ad8 _sleep(c1bcad34,c1bcad1c,6c,c1697c2d,0,...) at _sleep+0x396/frame 0xd7695b2c crypto_proc(0,d7695ba8,c1649a8c,406,0,...) at crypto_proc+0x2a7/frame 0xd7695b6c fork_exit(c0f1f860,0,d7695ba8) at fork_exit+0x7e/frame 0xd7695b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd7695b94 --- trap 0, eip = 0, esp = 0xd7695be0, ebp = 0 --- Tracing command geom pid 13 tid 100028 td 0xd5a1e380 sched_switch(d5a1e380,0,104,188,c1b57140,...) at sched_switch+0x503/frame 0xd766da48 mi_switch(104,0,c165a92d,22b,d5a1e380,...) at mi_switch+0x1f3/frame 0xd766da84 sleepq_switch(d5a1e380,0,c165a92d,2b3,d5a1e380,...) at sleepq_switch+0x110/frame 0xd766dab8 sleepq_wait(c1bf2838,5c,c1639c9b,0,0,...) at sleepq_wait+0x48/frame 0xd766dadc _sleep(c1bf2838,c1b46518,25c,c1639c9b,0,...) at _sleep+0x396/frame 0xd766db30 g_run_events(0,d766dba8,c1649a8c,406,0,...) at g_run_events+0x67/frame 0xd766db6c fork_exit(c0bbb530,0,d766dba8) at fork_exit+0x7e/frame 0xd766db94 fork_trampoline() at fork_trampoline+0x8/frame 0xd766db94 --- trap 0, eip = 0, esp = 0xd766dbe0, ebp = 0 --- Tracing command geom pid 13 tid 100029 td 0xd5a1e000 sched_switch(d5a1e000,0,104,188,c1b57140,...) at sched_switch+0x503/frame 0xd7672a38 mi_switch(104,0,c165a92d,22b,d5a1e000,...) at mi_switch+0x1f3/frame 0xd7672a74 sleepq_switch(d5a1e000,0,c165a92d,2b3,d5a1e000,...) at sleepq_switch+0x110/frame 0xd7672aa8 sleepq_wait(c1bf283c,5c,c1639c9b,0,0,...) at sleepq_wait+0x48/frame 0xd7672acc _sleep(c1bf283c,c1b46560,25c,c1639c9b,0,...) at _sleep+0x396/frame 0xd7672b20 g_io_schedule_up(d5a1e000,0,c163bd22,61,d7672b94,...) at g_io_schedule_up+0xcf/frame 0xd7672b54 g_up_procbody(0,d7672ba8,c1649a8c,406,0,...) at g_up_procbody+0x7d/frame 0xd7672b6c fork_exit(c0bbb5b0,0,d7672ba8) at fork_exit+0x7e/frame 0xd7672b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd7672b94 --- trap 0, eip = 0, esp = 0xd7672be0, ebp = 0 --- Tracing command geom pid 13 tid 100030 td 0xd5a19a80 sched_switch(d5a19a80,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd7677a30 mi_switch(104,0,c165a92d,22b,d5a19a80,...) at mi_switch+0x1f3/frame 0xd7677a70 sleepq_switch(d5a19a80,0,c165a92d,2b3,d5a19a80,...) at sleepq_switch+0x110/frame 0xd7677aa4 sleepq_wait(c1bf2840,5c,c1639c9b,0,0,...) at sleepq_wait+0x48/frame 0xd7677ac8 _sleep(c1bf2840,c1b46540,25c,c1639c9b,0,...) at _sleep+0x396/frame 0xd7677b1c g_io_schedule_down(d5a19a80,0,c163bd22,6d,d7677b94,...) at g_io_schedule_down+0xec/frame 0xd7677b54 g_down_procbody(0,d7677ba8,c1649a8c,406,0,...) at g_down_procbody+0x7d/frame 0xd7677b6c fork_exit(c0bbb630,0,d7677ba8) at fork_exit+0x7e/frame 0xd7677b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd7677b94 --- trap 0, eip = 0, esp = 0xd7677be0, ebp = 0 --- Tracing command intr pid 12 tid 100008 td 0xd5943380 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100011 td 0xd5942700 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100012 td 0xd5942380 sched_switch(d5942380,0,109,188,8,...) at sched_switch+0x503/frame 0xd761daf0 mi_switch(109,0,c1649e31,5a1,c1649a8c,...) at mi_switch+0x1f3/frame 0xd761db30 ithread_loop(d5937140,d761dba8,c1649a8c,406,0,...) at ithread_loop+0x200/frame 0xd761db6c fork_exit(c0c39b30,d5937140,d761dba8) at fork_exit+0x7e/frame 0xd761db94 fork_trampoline() at fork_trampoline+0x8/frame 0xd761db94 --- trap 0, eip = 0, esp = 0xd761dbe0, ebp = 0 --- Tracing command intr pid 12 tid 100022 td 0xd599e380 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100023 td 0xd599e000 sched_switch(d599e000,0,109,188,8,...) at sched_switch+0x503/frame 0xd7654af0 mi_switch(109,0,c1649e31,5a1,c1649a8c,...) at mi_switch+0x1f3/frame 0xd7654b30 ithread_loop(d5937120,d7654ba8,c1649a8c,406,0,...) at ithread_loop+0x200/frame 0xd7654b6c fork_exit(c0c39b30,d5937120,d7654ba8) at fork_exit+0x7e/frame 0xd7654b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd7654b94 --- trap 0, eip = 0, esp = 0xd7654be0, ebp = 0 --- Tracing command intr pid 12 tid 100024 td 0xd5945a80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100025 td 0xd5945700 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100026 td 0xd5945380 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100027 td 0xd5a1e700 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100042 td 0xd8363000 sched_switch(d8363000,0,109,188,8,...) at sched_switch+0x503/frame 0xd77d0af0 mi_switch(109,0,c1649e31,5a1,c1649a8c,...) at mi_switch+0x1f3/frame 0xd77d0b30 ithread_loop(d8357050,d77d0ba8,c1649a8c,406,0,...) at ithread_loop+0x200/frame 0xd77d0b6c fork_exit(c0c39b30,d8357050,d77d0ba8) at fork_exit+0x7e/frame 0xd77d0b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd77d0b94 --- trap 0, eip = 0, esp = 0xd77d0be0, ebp = 0 --- Tracing command intr pid 12 tid 100043 td 0xd8362a80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100044 td 0xd8362700 sched_switch(d8362700,0,109,188,8,...) at sched_switch+0x503/frame 0xd77dbaf0 mi_switch(109,0,c1649e31,5a1,c1649a8c,...) at mi_switch+0x1f3/frame 0xd77dbb30 ithread_loop(d8357030,d77dbba8,c1649a8c,406,0,...) at ithread_loop+0x200/frame 0xd77dbb6c fork_exit(c0c39b30,d8357030,d77dbba8) at fork_exit+0x7e/frame 0xd77dbb94 fork_trampoline() at fork_trampoline+0x8/frame 0xd77dbb94 --- trap 0, eip = 0, esp = 0xd77dbbe0, ebp = 0 --- Tracing command intr pid 12 tid 100050 td 0xd8361000 sched_switch(d8361000,0,109,188,8,...) at sched_switch+0x503/frame 0xd77fdaf0 mi_switch(109,0,c1649e31,5a1,c1649a8c,...) at mi_switch+0x1f3/frame 0xd77fdb30 ithread_loop(d8357020,d77fdba8,c1649a8c,406,0,...) at ithread_loop+0x200/frame 0xd77fdb6c fork_exit(c0c39b30,d8357020,d77fdba8) at fork_exit+0x7e/frame 0xd77fdb94 fork_trampoline() at fork_trampoline+0x8/frame 0xd77fdb94 --- trap 0, eip = 0, esp = 0xd77fdbe0, ebp = 0 --- Tracing command intr pid 12 tid 100051 td 0xd8358a80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100057 td 0xd8924a80 sched_switch(d8924a80,0,109,188,8,...) at sched_switch+0x503/frame 0xd355baf0 mi_switch(109,0,c1649e31,5a1,c1649a8c,...) at mi_switch+0x1f3/frame 0xd355bb30 ithread_loop(d5938f80,d355bba8,c1649a8c,406,0,...) at ithread_loop+0x200/frame 0xd355bb6c fork_exit(c0c39b30,d5938f80,d355bba8) at fork_exit+0x7e/frame 0xd355bb94 fork_trampoline() at fork_trampoline+0x8/frame 0xd355bb94 --- trap 0, eip = 0, esp = 0xd355bbe0, ebp = 0 --- Tracing command intr pid 12 tid 100063 td 0xd8923380 sched_switch(d8923380,0,109,188,8,...) at sched_switch+0x503/frame 0xd357aaf0 mi_switch(109,0,c1649e31,5a1,c1649a8c,...) at mi_switch+0x1f3/frame 0xd357ab30 ithread_loop(d5938f70,d357aba8,c1649a8c,406,0,...) at ithread_loop+0x200/frame 0xd357ab6c fork_exit(c0c39b30,d5938f70,d357aba8) at fork_exit+0x7e/frame 0xd357ab94 fork_trampoline() at fork_trampoline+0x8/frame 0xd357ab94 --- trap 0, eip = 0, esp = 0xd357abe0, ebp = 0 --- Tracing command intr pid 12 tid 100064 td 0xd8923000 sched_switch(d8923000,0,109,188,8,...) at sched_switch+0x503/frame 0xd3583af0 mi_switch(109,0,c1649e31,5a1,c1649a8c,...) at mi_switch+0x1f3/frame 0xd3583b30 ithread_loop(d5938f60,d3583ba8,c1649a8c,406,0,...) at ithread_loop+0x200/frame 0xd3583b6c fork_exit(c0c39b30,d5938f60,d3583ba8) at fork_exit+0x7e/frame 0xd3583b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd3583b94 --- trap 0, eip = 0, esp = 0xd3583be0, ebp = 0 --- Tracing command intr pid 12 tid 100065 td 0xd8363a80 sched_switch(d8363a80,0,109,188,8,...) at sched_switch+0x503/frame 0xd3589af0 mi_switch(109,0,c1649e31,5a1,c1649a8c,...) at mi_switch+0x1f3/frame 0xd3589b30 ithread_loop(d5938f50,d3589ba8,c1649a8c,406,0,...) at ithread_loop+0x200/frame 0xd3589b6c fork_exit(c0c39b30,d5938f50,d3589ba8) at fork_exit+0x7e/frame 0xd3589b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd3589b94 --- trap 0, eip = 0, esp = 0xd3589be0, ebp = 0 --- Tracing command intr pid 12 tid 100066 td 0xd8363700 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100067 td 0xd8363380 fork_trampoline() at fork_trampoline Tracing command idle pid 11 tid 100003 td 0xd5944700 (CPU 0) cpustop_handler(0,d34d0ab4,d34d0aa8,c1254363,d34d0a00,...) at cpustop_handler+0x25/frame 0xd34d09a0 ipi_nmi_handler(d34d0a00,c125ca33,c2027c50,8,c185e2ef,...) at ipi_nmi_handler+0x36/frame 0xd34d09b0 trap(d34d0ab4) at trap+0xa3/frame 0xd34d0aa8 calltrap() at calltrap+0x6/frame 0xd34d0aa8 --- trap 0x13, eip = 0xc126d5b5, esp = 0xd34d0af4, ebp = 0xd34d0b08 --- cpu_idle(1,d34d0b40,c1655fe8,a74,2f0f2f04,...) at cpu_idle+0xd5/frame 0xd34d0b08 sched_idletd(0,d34d0ba8,c1649a8c,406,0,...) at sched_idletd+0x3a6/frame 0xd34d0b6c fork_exit(c0ca8940,0,d34d0ba8) at fork_exit+0x7e/frame 0xd34d0b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd34d0b94 --- trap 0, eip = 0, esp = 0xd34d0be0, ebp = 0 --- Tracing command idle pid 11 tid 100004 td 0xd5944380 (CPU 1) cpustop_handler(1,d34d5ab4,d34d5aa8,c1254363,d34d5a00,...) at cpustop_handler+0x25/frame 0xd34d59a0 ipi_nmi_handler(d34d5a00,c125ca33,ca3fdc50,8,c185e2ef,...) at ipi_nmi_handler+0x36/frame 0xd34d59b0 trap(d34d5ab4) at trap+0xa3/frame 0xd34d5aa8 calltrap() at calltrap+0x6/frame 0xd34d5aa8 --- trap 0x13, eip = 0xc126d5b5, esp = 0xd34d5af4, ebp = 0xd34d5b08 --- cpu_idle(1,d34d5b40,c1655fe8,a74,0,...) at cpu_idle+0xd5/frame 0xd34d5b08 sched_idletd(0,d34d5ba8,c1649a8c,406,0,...) at sched_idletd+0x3a6/frame 0xd34d5b6c fork_exit(c0ca8940,0,d34d5ba8) at fork_exit+0x7e/frame 0xd34d5b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd34d5b94 --- trap 0, eip = 0, esp = 0xd34d5be0, ebp = 0 --- Tracing command idle pid 11 tid 100005 td 0xd5944000 (CPU 2) cpustop_handler(2,d34daab4,d34daaa8,c1254363,7ff6df84,...) at cpustop_handler+0x25/frame 0xd34da9a0 ipi_nmi_handler(7ff6df84,0,d34da9e8,d5944000,c1b57810,...) at ipi_nmi_handler+0x36/frame 0xd34da9b0 trap(d34daab4) at trap+0xa3/frame 0xd34daaa8 calltrap() at calltrap+0x6/frame 0xd34daaa8 --- trap 0x13, eip = 0xc126d5b5, esp = 0xd34daaf4, ebp = 0xd34dab08 --- cpu_idle(1,d34dab40,c1655fe8,a74,0,...) at cpu_idle+0xd5/frame 0xd34dab08 sched_idletd(0,d34daba8,c1649a8c,406,0,...) at sched_idletd+0x3a6/frame 0xd34dab6c fork_exit(c0ca8940,0,d34daba8) at fork_exit+0x7e/frame 0xd34dab94 fork_trampoline() at fork_trampoline+0x8/frame 0xd34dab94 --- trap 0, eip = 0, esp = 0xd34dabe0, ebp = 0 --- Tracing command idle pid 11 tid 100006 td 0xd5943a80 sched_switch(d5943a80,0,108,184,8,...) at sched_switch+0x503/frame 0xd34dfac8 mi_switch(108,0,c1655fe8,a72,0,...) at mi_switch+0x1f3/frame 0xd34dfb08 sched_idletd(0,d34dfba8,c1649a8c,406,0,...) at sched_idletd+0xb0/frame 0xd34dfb6c fork_exit(c0ca8940,0,d34dfba8) at fork_exit+0x7e/frame 0xd34dfb94 fork_trampoline() at fork_trampoline+0x8/frame 0xd34dfb94 --- trap 0, eip = 0, esp = 0xd34dfbe0, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xd5944a80 sched_switch(d5944a80,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xc7ffa800 mi_switch(104,0,c165a92d,22b,d5944a80,...) at mi_switch+0x1f3/frame 0xc7ffa840 sleepq_switch(d5944a80,0,c165a92d,1ca,6c,...) at sleepq_switch+0x110/frame 0xc7ffa874 sleepq_catch_signals(0,c165a92d,178,c1b64d94,d5944a80,...) at sleepq_catch_signals+0x360/frame 0xc7ffa8b8 sleepq_wait_sig(d5940a08,6c,c165eca6,100,0,...) at sleepq_wait_sig+0x14/frame 0xc7ffa8dc _sleep(d5940a08,d5940ab0,36c,c165eca6,0,...) at _sleep+0x386/frame 0xc7ffa930 kern_wait6(d5944a80,7,0,0,c7ffa9b4,...) at kern_wait6+0x567/frame 0xc7ffa988 sys_wait4(d5944a80,d5944d00,c16494cb,4,c185e2ef,...) at sys_wait4+0x87/frame 0xc7ffaaa8 syscall(c7ffaba8) at syscall+0x6ee/frame 0xc7ffab9c Xint0x80_syscall() at Xint0x80_syscall+0x2b/frame 0xc7ffab9c --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x806a257, esp = 0xbfbfe864, ebp = 0xbfbfe878 --- Tracing command audit pid 10 tid 100001 td 0xd5945000 sched_switch(d5945000,0,104,188,c1b57140,...) at sched_switch+0x503/frame 0xc7ff5a40 mi_switch(104,0,c165a92d,22b,d5945000,...) at mi_switch+0x1f3/frame 0xc7ff5a7c sleepq_switch(d5945000,0,c165a92d,2b3,c1916078,...) at sleepq_switch+0x110/frame 0xc7ff5ab0 sleepq_wait(c1bf6b50,0,c7ff5af4,1,0,...) at sleepq_wait+0x48/frame 0xc7ff5ad4 _cv_wait(c1bf6b50,c1bf6ad8,c169a39e,1be,8,...) at _cv_wait+0x21f/frame 0xc7ff5b18 audit_worker(0,c7ff5ba8,c1649a8c,406,0,...) at audit_worker+0x94/frame 0xc7ff5b6c fork_exit(c0f41f80,0,c7ff5ba8) at fork_exit+0x7e/frame 0xc7ff5b94 fork_trampoline() at fork_trampoline+0x8/frame 0xc7ff5b94 --- trap 0, eip = 0, esp = 0xc7ff5be0, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc1bf2bc0 sched_switch(c1bf2bc0,0,104,188,c1b57140,...) at sched_switch+0x503/frame 0xc2022ab0 mi_switch(104,0,c165a92d,22b,c1bf2bc0,...) at mi_switch+0x1f3/frame 0xc2022aec sleepq_switch(c1bf2bc0,0,c165a92d,2d6,c,...) at sleepq_switch+0x110/frame 0xc2022b20 sleepq_timedwait(c1bf2864,54,9,0,0,...) at sleepq_timedwait+0x53/frame 0xc2022b5c _sleep(c1bf2864,0,54,c16aafa2,fffff470,...) at _sleep+0x332/frame 0xc2022bb0 swapper() at swapper+0x2c8/frame 0xc2022bf8 begin() at begin+0x2f Tracing command kernel pid 0 tid 100007 td 0xd5943700 sched_switch(d5943700,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd7604a48 mi_switch(104,0,c165a92d,22b,d5943700,...) at mi_switch+0x1f3/frame 0xd7604a88 sleepq_switch(d5943700,0,c165a92d,2b3,d5943700,...) at sleepq_switch+0x110/frame 0xd7604abc sleepq_wait(d593c980,0,c1639c9b,0,0,...) at sleepq_wait+0x48/frame 0xd7604ae0 _sleep(d593c980,d593c99c,0,c1639c9b,0,...) at _sleep+0x396/frame 0xd7604b34 taskqueue_thread_loop(c1bf5edc,d7604ba8,c1649a8c,406,0,...) at taskqueue_thread_loop+0x144/frame 0xd7604b6c fork_exit(c0cd44c0,c1bf5edc,d7604ba8) at fork_exit+0x7e/frame 0xd7604b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd7604b94 --- trap 0, eip = 0, esp = 0xd7604be0, ebp = 0 --- Tracing command kernel pid 0 tid 100009 td 0xd5943000 sched_switch(d5943000,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd760ea48 mi_switch(104,0,c165a92d,22b,d5943000,...) at mi_switch+0x1f3/frame 0xd760ea88 sleepq_switch(d5943000,0,c165a92d,2b3,d5943000,...) at sleepq_switch+0x110/frame 0xd760eabc sleepq_wait(d593c880,0,c1639c9b,0,0,...) at sleepq_wait+0x48/frame 0xd760eae0 _sleep(d593c880,d593c89c,0,c1639c9b,0,...) at _sleep+0x396/frame 0xd760eb34 taskqueue_thread_loop(c1bf3154,d760eba8,c1649a8c,406,0,...) at taskqueue_thread_loop+0x144/frame 0xd760eb6c fork_exit(c0cd44c0,c1bf3154,d760eba8) at fork_exit+0x7e/frame 0xd760eb94 fork_trampoline() at fork_trampoline+0x8/frame 0xd760eb94 --- trap 0, eip = 0, esp = 0xd760ebe0, ebp = 0 --- Tracing command kernel pid 0 tid 100010 td 0xd5942a80 sched_switch(d5942a80,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd7613a48 mi_switch(104,0,c165a92d,22b,d5942a80,...) at mi_switch+0x1f3/frame 0xd7613a88 sleepq_switch(d5942a80,0,c165a92d,2b3,d5942a80,...) at sleepq_switch+0x110/frame 0xd7613abc sleepq_wait(d593c800,0,c1639c9b,0,0,...) at sleepq_wait+0x48/frame 0xd7613ae0 _sleep(d593c800,d593c81c,0,c1639c9b,0,...) at _sleep+0x396/frame 0xd7613b34 taskqueue_thread_loop(c1bf5f38,d7613ba8,c1649a8c,406,0,...) at taskqueue_thread_loop+0x144/frame 0xd7613b6c fork_exit(c0cd44c0,c1bf5f38,d7613ba8) at fork_exit+0x7e/frame 0xd7613b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd7613b94 --- trap 0, eip = 0, esp = 0xd7613be0, ebp = 0 --- Tracing command kernel pid 0 tid 100013 td 0xd5942000 sched_switch(d5942000,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd7622a58 mi_switch(104,0,c165a92d,22b,d5942000,...) at mi_switch+0x1f3/frame 0xd7622a98 sleepq_switch(d5942000,0,c165a92d,2b3,d593c680,...) at sleepq_switch+0x110/frame 0xd7622acc sleepq_wait(d593c680,0,c1653df3,c1639c9b,0,...) at sleepq_wait+0x48/frame 0xd7622af0 msleep_spin_sbt(d593c680,d593c69c,c1639c9b,0,0,...) at msleep_spin_sbt+0x226/frame 0xd7622b34 gtaskqueue_thread_loop(d599c004,d7622ba8,c1649a8c,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xd7622b6c fork_exit(c0cbe330,d599c004,d7622ba8) at fork_exit+0x7e/frame 0xd7622b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd7622b94 --- trap 0, eip = 0, esp = 0xd7622be0, ebp = 0 --- Tracing command kernel pid 0 tid 100014 td 0xd59a0380 sched_switch(d59a0380,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd7627a58 mi_switch(104,0,c165a92d,22b,d59a0380,...) at mi_switch+0x1f3/frame 0xd7627a98 sleepq_switch(d59a0380,0,c165a92d,2b3,d593c600,...) at sleepq_switch+0x110/frame 0xd7627acc sleepq_wait(d593c600,0,c1653df3,c1639c9b,0,...) at sleepq_wait+0x48/frame 0xd7627af0 msleep_spin_sbt(d593c600,d593c61c,c1639c9b,0,0,...) at msleep_spin_sbt+0x226/frame 0xd7627b34 gtaskqueue_thread_loop(d599bc04,d7627ba8,c1649a8c,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xd7627b6c fork_exit(c0cbe330,d599bc04,d7627ba8) at fork_exit+0x7e/frame 0xd7627b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd7627b94 --- trap 0, eip = 0, esp = 0xd7627be0, ebp = 0 --- Tracing command kernel pid 0 tid 100015 td 0xd59a0000 sched_switch(d59a0000,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd762ca58 mi_switch(104,0,c165a92d,22b,d59a0000,...) at mi_switch+0x1f3/frame 0xd762ca98 sleepq_switch(d59a0000,0,c165a92d,2b3,d593c580,...) at sleepq_switch+0x110/frame 0xd762cacc sleepq_wait(d593c580,0,c1653df3,c1639c9b,0,...) at sleepq_wait+0x48/frame 0xd762caf0 msleep_spin_sbt(d593c580,d593c59c,c1639c9b,0,0,...) at msleep_spin_sbt+0x226/frame 0xd762cb34 gtaskqueue_thread_loop(d599bc14,d762cba8,c1649a8c,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xd762cb6c fork_exit(c0cbe330,d599bc14,d762cba8) at fork_exit+0x7e/frame 0xd762cb94 fork_trampoline() at fork_trampoline+0x8/frame 0xd762cb94 --- trap 0, eip = 0, esp = 0xd762cbe0, ebp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xd599fa80 sched_switch(d599fa80,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd7631a58 mi_switch(104,0,c165a92d,22b,d599fa80,...) at mi_switch+0x1f3/frame 0xd7631a98 sleepq_switch(d599fa80,0,c165a92d,2b3,d593c500,...) at sleepq_switch+0x110/frame 0xd7631acc sleepq_wait(d593c500,0,c1653df3,c1639c9b,0,...) at sleepq_wait+0x48/frame 0xd7631af0 msleep_spin_sbt(d593c500,d593c51c,c1639c9b,0,0,...) at msleep_spin_sbt+0x226/frame 0xd7631b34 gtaskqueue_thread_loop(d599bc24,d7631ba8,c1649a8c,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xd7631b6c fork_exit(c0cbe330,d599bc24,d7631ba8) at fork_exit+0x7e/frame 0xd7631b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd7631b94 --- trap 0, eip = 0, esp = 0xd7631be0, ebp = 0 --- Tracing command kernel pid 0 tid 100017 td 0xd599f700 sched_switch(d599f700,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd7636a58 mi_switch(104,0,c165a92d,22b,d599f700,...) at mi_switch+0x1f3/frame 0xd7636a98 sleepq_switch(d599f700,0,c165a92d,2b3,d593c480,...) at sleepq_switch+0x110/frame 0xd7636acc sleepq_wait(d593c480,0,c1653df3,c1639c9b,0,...) at sleepq_wait+0x48/frame 0xd7636af0 msleep_spin_sbt(d593c480,d593c49c,c1639c9b,0,0,...) at msleep_spin_sbt+0x226/frame 0xd7636b34 gtaskqueue_thread_loop(d599bc34,d7636ba8,c1649a8c,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xd7636b6c fork_exit(c0cbe330,d599bc34,d7636ba8) at fork_exit+0x7e/frame 0xd7636b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd7636b94 --- trap 0, eip = 0, esp = 0xd7636be0, ebp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xd599f380 sched_switch(d599f380,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd763ba58 mi_switch(104,0,c165a92d,22b,d599f380,...) at mi_switch+0x1f3/frame 0xd763ba98 sleepq_switch(d599f380,0,c165a92d,2b3,d593c400,...) at sleepq_switch+0x110/frame 0xd763bacc sleepq_wait(d593c400,0,c1653df3,c1639c9b,0,...) at sleepq_wait+0x48/frame 0xd763baf0 msleep_spin_sbt(d593c400,d593c41c,c1639c9b,0,0,...) at msleep_spin_sbt+0x226/frame 0xd763bb34 gtaskqueue_thread_loop(d599b804,d763bba8,c1649a8c,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xd763bb6c fork_exit(c0cbe330,d599b804,d763bba8) at fork_exit+0x7e/frame 0xd763bb94 fork_trampoline() at fork_trampoline+0x8/frame 0xd763bb94 --- trap 0, eip = 0, esp = 0xd763bbe0, ebp = 0 --- Tracing command kernel pid 0 tid 100019 td 0xd599f000 sched_switch(d599f000,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd7640a58 mi_switch(104,0,c165a92d,22b,d599f000,...) at mi_switch+0x1f3/frame 0xd7640a98 sleepq_switch(d599f000,0,c165a92d,2b3,d593c380,...) at sleepq_switch+0x110/frame 0xd7640acc sleepq_wait(d593c380,0,c1653df3,c1639c9b,0,...) at sleepq_wait+0x48/frame 0xd7640af0 msleep_spin_sbt(d593c380,d593c39c,c1639c9b,0,0,...) at msleep_spin_sbt+0x226/frame 0xd7640b34 gtaskqueue_thread_loop(d599b814,d7640ba8,c1649a8c,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xd7640b6c fork_exit(c0cbe330,d599b814,d7640ba8) at fork_exit+0x7e/frame 0xd7640b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd7640b94 --- trap 0, eip = 0, esp = 0xd7640be0, ebp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xd599ea80 sched_switch(d599ea80,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd7645a58 mi_switch(104,0,c165a92d,22b,d599ea80,...) at mi_switch+0x1f3/frame 0xd7645a98 sleepq_switch(d599ea80,0,c165a92d,2b3,d593c300,...) at sleepq_switch+0x110/frame 0xd7645acc sleepq_wait(d593c300,0,c1653df3,c1639c9b,0,...) at sleepq_wait+0x48/frame 0xd7645af0 msleep_spin_sbt(d593c300,d593c31c,c1639c9b,0,0,...) at msleep_spin_sbt+0x226/frame 0xd7645b34 gtaskqueue_thread_loop(d599b824,d7645ba8,c1649a8c,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xd7645b6c fork_exit(c0cbe330,d599b824,d7645ba8) at fork_exit+0x7e/frame 0xd7645b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd7645b94 --- trap 0, eip = 0, esp = 0xd7645be0, ebp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xd599e700 sched_switch(d599e700,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd764aa58 mi_switch(104,0,c165a92d,22b,d599e700,...) at mi_switch+0x1f3/frame 0xd764aa98 sleepq_switch(d599e700,0,c165a92d,2b3,d593c280,...) at sleepq_switch+0x110/frame 0xd764aacc sleepq_wait(d593c280,0,c1653df3,c1639c9b,0,...) at sleepq_wait+0x48/frame 0xd764aaf0 msleep_spin_sbt(d593c280,d593c29c,c1639c9b,0,0,...) at msleep_spin_sbt+0x226/frame 0xd764ab34 gtaskqueue_thread_loop(d599b834,d764aba8,c1649a8c,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xd764ab6c fork_exit(c0cbe330,d599b834,d764aba8) at fork_exit+0x7e/frame 0xd764ab94 fork_trampoline() at fork_trampoline+0x8/frame 0xd764ab94 --- trap 0, eip = 0, esp = 0xd764abe0, ebp = 0 --- Tracing command kernel pid 0 tid 100031 td 0xd5a19700 sched_switch(d5a19700,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd767ca48 mi_switch(104,0,c165a92d,22b,d5a19700,...) at mi_switch+0x1f3/frame 0xd767ca88 sleepq_switch(d5a19700,0,c165a92d,2b3,d5a19700,...) at sleepq_switch+0x110/frame 0xd767cabc sleepq_wait(d593c100,0,c1639c9b,0,0,...) at sleepq_wait+0x48/frame 0xd767cae0 _sleep(d593c100,d593c11c,0,c1639c9b,0,...) at _sleep+0x396/frame 0xd767cb34 taskqueue_thread_loop(c1b643a4,d767cba8,c1649a8c,406,0,...) at taskqueue_thread_loop+0x144/frame 0xd767cb6c fork_exit(c0cd44c0,c1b643a4,d767cba8) at fork_exit+0x7e/frame 0xd767cb94 fork_trampoline() at fork_trampoline+0x8/frame 0xd767cb94 --- trap 0, eip = 0, esp = 0xd767cbe0, ebp = 0 --- Tracing command kernel pid 0 tid 100032 td 0xd5a19380 sched_switch(d5a19380,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd7681a48 mi_switch(104,0,c165a92d,22b,d5a19380,...) at mi_switch+0x1f3/frame 0xd7681a88 sleepq_switch(d5a19380,0,c165a92d,2b3,d5a19380,...) at sleepq_switch+0x110/frame 0xd7681abc sleepq_wait(d593bd80,0,c1639c9b,0,0,...) at sleepq_wait+0x48/frame 0xd7681ae0 _sleep(d593bd80,d593bd9c,0,c1639c9b,0,...) at _sleep+0x396/frame 0xd7681b34 taskqueue_thread_loop(c1bcadd8,d7681ba8,c1649a8c,406,0,...) at taskqueue_thread_loop+0x144/frame 0xd7681b6c fork_exit(c0cd44c0,c1bcadd8,d7681ba8) at fork_exit+0x7e/frame 0xd7681b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd7681b94 --- trap 0, eip = 0, esp = 0xd7681be0, ebp = 0 --- Tracing command kernel pid 0 tid 100033 td 0xd5a19000 sched_switch(d5a19000,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd7686a48 mi_switch(104,0,c165a92d,22b,d5a19000,...) at mi_switch+0x1f3/frame 0xd7686a88 sleepq_switch(d5a19000,0,c165a92d,2b3,d5a19000,...) at sleepq_switch+0x110/frame 0xd7686abc sleepq_wait(d593bd80,0,c1639c9b,0,0,...) at sleepq_wait+0x48/frame 0xd7686ae0 _sleep(d593bd80,d593bd9c,0,c1639c9b,0,...) at _sleep+0x396/frame 0xd7686b34 taskqueue_thread_loop(c1bcadd8,d7686ba8,c1649a8c,406,0,...) at taskqueue_thread_loop+0x144/frame 0xd7686b6c fork_exit(c0cd44c0,c1bcadd8,d7686ba8) at fork_exit+0x7e/frame 0xd7686b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd7686b94 --- trap 0, eip = 0, esp = 0xd7686be0, ebp = 0 --- Tracing command kernel pid 0 tid 100034 td 0xd5a17a80 sched_switch(d5a17a80,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd768ba48 mi_switch(104,0,c165a92d,22b,d5a17a80,...) at mi_switch+0x1f3/frame 0xd768ba88 sleepq_switch(d5a17a80,0,c165a92d,2b3,d5a17a80,...) at sleepq_switch+0x110/frame 0xd768babc sleepq_wait(d593bd80,0,c1639c9b,0,0,...) at sleepq_wait+0x48/frame 0xd768bae0 _sleep(d593bd80,d593bd9c,0,c1639c9b,0,...) at _sleep+0x396/frame 0xd768bb34 taskqueue_thread_loop(c1bcadd8,d768bba8,c1649a8c,406,0,...) at taskqueue_thread_loop+0x144/frame 0xd768bb6c fork_exit(c0cd44c0,c1bcadd8,d768bba8) at fork_exit+0x7e/frame 0xd768bb94 fork_trampoline() at fork_trampoline+0x8/frame 0xd768bb94 --- trap 0, eip = 0, esp = 0xd768bbe0, ebp = 0 --- Tracing command kernel pid 0 tid 100035 td 0xd5a17700 sched_switch(d5a17700,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd7690a48 mi_switch(104,0,c165a92d,22b,d5a17700,...) at mi_switch+0x1f3/frame 0xd7690a88 sleepq_switch(d5a17700,0,c165a92d,2b3,d5a17700,...) at sleepq_switch+0x110/frame 0xd7690abc sleepq_wait(d593bd80,0,c1639c9b,0,0,...) at sleepq_wait+0x48/frame 0xd7690ae0 _sleep(d593bd80,d593bd9c,0,c1639c9b,0,...) at _sleep+0x396/frame 0xd7690b34 taskqueue_thread_loop(c1bcadd8,d7690ba8,c1649a8c,406,0,...) at taskqueue_thread_loop+0x144/frame 0xd7690b6c fork_exit(c0cd44c0,c1bcadd8,d7690ba8) at fork_exit+0x7e/frame 0xd7690b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd7690b94 --- trap 0, eip = 0, esp = 0xd7690be0, ebp = 0 --- Tracing command kernel pid 0 tid 100068 td 0xd8c57700 sched_switch(d8c57700,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd3598a58 mi_switch(104,0,c165a92d,22b,d8c57700,...) at mi_switch+0x1f3/frame 0xd3598a98 sleepq_switch(d8c57700,0,c165a92d,2b3,d8c0c980,...) at sleepq_switch+0x110/frame 0xd3598acc sleepq_wait(d8c0c980,0,c1653df3,c1639c9b,0,...) at sleepq_wait+0x48/frame 0xd3598af0 msleep_spin_sbt(d8c0c980,d8c0c99c,c1639c9b,0,0,...) at msleep_spin_sbt+0x226/frame 0xd3598b34 taskqueue_thread_loop(c1be6d6c,d3598ba8,c1649a8c,406,0,...) at taskqueue_thread_loop+0xf3/frame 0xd3598b6c fork_exit(c0cd44c0,c1be6d6c,d3598ba8) at fork_exit+0x7e/frame 0xd3598b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd3598b94 --- trap 0, eip = 0, esp = 0xd3598be0, ebp = 0 --- Tracing command kernel pid 0 tid 100075 td 0xd8e68380 sched_switch(d8e68380,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd35bca58 mi_switch(104,0,c165a92d,22b,d8e68380,...) at mi_switch+0x1f3/frame 0xd35bca98 sleepq_switch(d8e68380,0,c165a92d,2b3,d8dc2d80,...) at sleepq_switch+0x110/frame 0xd35bcacc sleepq_wait(d8dc2d80,0,c1653df3,c1639c9b,0,...) at sleepq_wait+0x48/frame 0xd35bcaf0 msleep_spin_sbt(d8dc2d80,d8dc2d9c,c1639c9b,0,0,...) at msleep_spin_sbt+0x226/frame 0xd35bcb34 taskqueue_thread_loop(c1be895c,d35bcba8,c1649a8c,406,0,...) at taskqueue_thread_loop+0xf3/frame 0xd35bcb6c fork_exit(c0cd44c0,c1be895c,d35bcba8) at fork_exit+0x7e/frame 0xd35bcb94 fork_trampoline() at fork_trampoline+0x8/frame 0xd35bcb94 --- trap 0, eip = 0, esp = 0xd35bcbe0, ebp = 0 --- Tracing command kernel pid 0 tid 100076 td 0xd8e68000 sched_switch(d8e68000,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd35c1a58 mi_switch(104,0,c165a92d,22b,d8e68000,...) at mi_switch+0x1f3/frame 0xd35c1a98 sleepq_switch(d8e68000,0,c165a92d,2b3,d8dc2d80,...) at sleepq_switch+0x110/frame 0xd35c1acc sleepq_wait(d8dc2d80,0,c1653df3,c1639c9b,0,...) at sleepq_wait+0x48/frame 0xd35c1af0 msleep_spin_sbt(d8dc2d80,d8dc2d9c,c1639c9b,0,0,...) at msleep_spin_sbt+0x226/frame 0xd35c1b34 taskqueue_thread_loop(c1be895c,d35c1ba8,c1649a8c,406,0,...) at taskqueue_thread_loop+0xf3/frame 0xd35c1b6c fork_exit(c0cd44c0,c1be895c,d35c1ba8) at fork_exit+0x7e/frame 0xd35c1b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd35c1b94 --- trap 0, eip = 0, esp = 0xd35c1be0, ebp = 0 --- Tracing command kernel pid 0 tid 100077 td 0xd8e67a80 sched_switch(d8e67a80,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd35c6a58 mi_switch(104,0,c165a92d,22b,d8e67a80,...) at mi_switch+0x1f3/frame 0xd35c6a98 sleepq_switch(d8e67a80,0,c165a92d,2b3,d8dc2d80,...) at sleepq_switch+0x110/frame 0xd35c6acc sleepq_wait(d8dc2d80,0,c1653df3,c1639c9b,0,...) at sleepq_wait+0x48/frame 0xd35c6af0 msleep_spin_sbt(d8dc2d80,d8dc2d9c,c1639c9b,0,0,...) at msleep_spin_sbt+0x226/frame 0xd35c6b34 taskqueue_thread_loop(c1be895c,d35c6ba8,c1649a8c,406,0,...) at taskqueue_thread_loop+0xf3/frame 0xd35c6b6c fork_exit(c0cd44c0,c1be895c,d35c6ba8) at fork_exit+0x7e/frame 0xd35c6b94 fork_trampoline() at fork_trampoline+0x8/frame 0xd35c6b94 --- trap 0, eip = 0, esp = 0xd35c6be0, ebp = 0 --- Tracing command kernel pid 0 tid 100078 td 0xd8e67700 sched_switch(d8e67700,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xd35cba48 mi_switch(104,0,c165a92d,22b,d8e67700,...) at mi_switch+0x1f3/frame 0xd35cba88 sleepq_switch(d8e67700,0,c165a92d,2b3,d8e67700,...) at sleepq_switch+0x110/frame 0xd35cbabc sleepq_wait(d593bc80,0,c1639c9b,0,0,...) at sleepq_wait+0x48/frame 0xd35cbae0 _sleep(d593bc80,d593bc9c,0,c1639c9b,0,...) at _sleep+0x396/frame 0xd35cbb34 taskqueue_thread_loop(c196ede8,d35cbba8,c1649a8c,406,0,...) at taskqueue_thread_loop+0x144/frame 0xd35cbb6c fork_exit(c0cd44c0,c196ede8,d35cbba8) at fork_exit+0x7e/frame 0xd35cbb94 fork_trampoline() at fork_trampoline+0x8/frame 0xd35cbb94 --- trap 0, eip = 0, esp = 0xd35cbbe0, ebp = 0 --- Tracing command kernel pid 0 tid 100089 td 0xd8c56380 sched_switch(d8c56380,0,104,188,c12436a0,...) at sched_switch+0x503/frame 0xdb292a48 mi_switch(104,0,c165a92d,22b,d8c56380,...) at mi_switch+0x1f3/frame 0xdb292a88 sleepq_switch(d8c56380,0,c165a92d,2b3,d8c56380,...) at sleepq_switch+0x110/frame 0xdb292abc sleepq_wait(d8c0b780,0,c1639c9b,0,0,...) at sleepq_wait+0x48/frame 0xdb292ae0 _sleep(d8c0b780,d8c0b79c,0,c1639c9b,0,...) at _sleep+0x396/frame 0xdb292b34 taskqueue_thread_loop(d943919c,db292ba8,c1649a8c,406,0,...) at taskqueue_thread_loop+0x144/frame 0xdb292b6c fork_exit(c0cd44c0,d943919c,db292ba8) at fork_exit+0x7e/frame 0xdb292b94 fork_trampoline() at fork_trampoline+0x8/frame 0xdb292b94 --- trap 0, eip = 0, esp = 0xdb292be0, ebp = 0 --- db:0:allt> dump Physical memory: 2967 MB Dumping 129 MB: 114 98 82 66 50 34 18 2 Dump complete db:0:dump> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 3 (kgdb) bt #0 doadump (textdump=) at pcpu.h:216 #1 0xc05ad24e in db_dump (dummy=0xc0cbf8fa, dummy2=0x0, dummy3=0xffffffff, dummy4=0xdb2c21a8 "") at ../../../ddb/db_command.c:574 #2 0xc05ad027 in db_command (cmd_table=) at ../../../ddb/db_command.c:481 #3 0xc05b1b68 in db_script_exec (scriptname=0xc1be86c4 "dump", warnifnotfound=) at ../../../ddb/db_script.c:304 #4 0xc05ad027 in db_command (cmd_table=) at ../../../ddb/db_command.c:481 #5 0xc05acd80 in db_command_loop () at ../../../ddb/db_command.c:534 #6 0xc05affbd in db_trap (type=0x3, code=0xffff0ff0) at ../../../ddb/db_main.c:250 #7 0xc0cc00ae in kdb_trap (type=0x3, code=0xffff0ff0, tf=0xdb2c24f4) at ../../../kern/subr_kdb.c:697 #8 0xc1254ad7 in trap (frame=0xdb2c24f4) at ../../../i386/i386/trap.c:677 #9 0xc123eea7 in calltrap () at ../../../i386/i386/exception.s:172 #10 0x00000000 in ?? () Current language: auto; currently minimal (kgdb) p *(struct bufqueue *)0xca199a14 $2 = {bq_lock = {lock_object = {lo_name = 0xc166430c "bufwait", lo_flags = 0x6730000, lo_data = 0x0, lo_witness = 0xc81f2758}, mtx_lock = 0xd8c57a80}, bq_queue = { tqh_first = 0xc0d618ac, tqh_last = 0xc0d6163d}, bq_index = 0xa6, bq_subqueue = 0xc0d5, bq_len = 0xc0d59063} (kgdb) p *(struct buf *)0xca1999b8 $3 = {b_bufobj = 0xd93feef8, b_bcount = 0x8000, b_caller1 = 0x0, b_data = 0xca48c000 "ķA\030", b_error = 0x0, b_iocmd = 0x2, b_ioflags = 0x0, b_iooffset = 0x2461278000, b_resid = 0x0, b_iodone = 0, b_ckhashcalc = 0, b_ckhash = 0x0, b_blkno = 0x123093c0, b_offset = 0x2461278000, b_bobufs = {tqe_next = 0x0, tqe_prev = 0xca1b67e0}, b_vflags = 0x0, b_qindex = 0x3, b_domain = 0x0, b_subqueue = 0x4, b_flags = 0xa00210a0, b_xflags = 0x1, b_lock = {lock_object = {lo_name = 0xc166430c "bufwait", lo_flags = 0x6730000, lo_data = 0x0, lo_witness = 0xc81f2758}, lk_lock = 0xd8c57a80, lk_exslpfail = 0x0, lk_timo = 0x0, lk_pri = 0x60, lk_stack = {depth = 0xd, pcs = 0xca199a38}}, b_bufsize = 0x8000, b_runningbufspace = 0x0, b_kvasize = 0x8000, b_dirtyoff = 0x0, b_dirtyend = 0x0, b_kvabase = 0xca48c000 "ķA\030", b_lblkno = 0x123093c0, b_vp = 0xd93fee38, b_rcred = 0x0, b_wcred = 0x0, {b_freelist = {tqe_next = 0xca19981c, tqe_prev = 0xca199c48}, {b_pgiodone = 0xca19981c, b_pgbefore = 0xca199c48, b_pgafter = 0x0}}, b_cluster = {cluster_head = {tqh_first = 0x0, tqh_last = 0x0}, cluster_entry = {tqe_next = 0x0, tqe_prev = 0x0}}, b_pages = 0xca199ac0, b_npages = 0x8, b_dep = {lh_first = 0xd9391400}, b_fsprivate1 = 0x0, b_fsprivate2 = 0x0, b_fsprivate3 = 0x0} (kgdb) $ svnlite diff -x -p /var/tmp/numa Index: /var/tmp/numa/sys/kern/vfs_bio.c =================================================================== --- /var/tmp/numa/sys/kern/vfs_bio.c (revision 329862) +++ /var/tmp/numa/sys/kern/vfs_bio.c (working copy) @@ -1129,6 +1129,7 @@ bufinit(void) * concurrency but less accurate LRU. */ clean_domains = MIN(howmany(maxbufspace, 256*1024*1024), CLEAN_DOMAINS); + clean_domains = 1; /* pho XXX */ for (i = 0 ; i < clean_domains; i++) { struct bufdomain *bd; @@ -1717,7 +1718,7 @@ bd_init(struct bufdomain *bd) domain = bd - bdclean; bd->bd_cleanq = &bd->bd_subq[mp_ncpus]; - bq_init(bd->bd_cleanq, QUEUE_CLEAN, -1, "bufq clean lock"); + bq_init(bd->bd_cleanq, QUEUE_CLEAN, mp_ncpus, "bufq clean lock"); for (i = 0; i <= mp_maxid; i++) bq_init(&bd->bd_subq[i], QUEUE_CLEAN, i, "bufq clean subqueue lock"); Index: /var/tmp/numa/sys/vm/vm_phys.h =================================================================== --- /var/tmp/numa/sys/vm/vm_phys.h (revision 329862) +++ /var/tmp/numa/sys/vm/vm_phys.h (working copy) @@ -51,6 +51,7 @@ struct mem_affinity { #ifdef NUMA extern struct mem_affinity *mem_affinity; extern int *mem_locality; +extern int vm_ndomains; /* pho XXX */ #endif struct vm_freelist { $