HEAD from 216137M, no vmcore KDB: debugger backends: ddb KDB: current backend: ddb 524288K of memory above 4GB ignored Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.0-CURRENT #0 r216137M: Fri Dec 3 01:20:16 CET 2010 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Family = 10 Model = 2 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 4294967296 (4096 MB) avail memory = 3536179200 (3372 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: <060508 APIC1048> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: <060508 RSDT1048> on motherboard acpi0: Power Button (fixed) acpi0: reservation of fee00000, 1000 (3) failed acpi0: reservation of ffb80000, 80000 (3) failed acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, d7f00000 (3) failed Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 vgapci0: port 0xc000-0xc0ff mem 0xf0000000-0xf7ffffff,0xfe9f0000-0xfe9fffff,0xfe800000-0xfe8fffff irq 18 at device 5.0 on pci1 pci1: at device 5.2 (no driver attached) pcib2: at device 7.0 on pci0 pci2: on pcib2 re0: port 0xd800-0xd8ff mem 0xfeaff000-0xfeafffff irq 19 at device 0.0 on pci2 re0: Using 1 MSI messages re0: Chip rev. 0x38000000 re0: MAC rev. 0x00000000 miibus0: on re0 rgephy0: PHY 1 on miibus0 rgephy0: 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re0: Ethernet address: 00:21:85:15:33:7d atapci0: port 0xb000-0xb007,0xa000-0xa003,0x9000-0x9007,0x8000-0x8003,0x7000-0x700f mem 0xfe7ff800-0xfe7ffbff irq 22 at device 18.0 on pci0 atapci0: AHCI v1.10 controller with 4 3Gbps ports, PM supported ata2: on atapci0 ata2: port is not ready (timeout 0ms) tfd = 000001d0 ata2: software reset clear timeout ata3: on atapci0 ata4: on atapci0 ata5: on atapci0 ohci0: mem 0xfe7fe000-0xfe7fefff irq 16 at device 19.0 on pci0 usbus0: on ohci0 ohci1: mem 0xfe7fd000-0xfe7fdfff irq 17 at device 19.1 on pci0 usbus1: on ohci1 ohci2: mem 0xfe7fc000-0xfe7fcfff irq 18 at device 19.2 on pci0 usbus2: on ohci2 ohci3: mem 0xfe7fb000-0xfe7fbfff irq 17 at device 19.3 on pci0 usbus3: on ohci3 ohci4: mem 0xfe7fa000-0xfe7fafff irq 18 at device 19.4 on pci0 usbus4: on ohci4 ehci0: mem 0xfe7ff000-0xfe7ff0ff irq 19 at device 19.5 on pci0 ehci0: AMD SB600/700 quirk applied usbus5: EHCI version 1.0 usbus5: on ehci0 pci0: at device 20.0 (no driver attached) atapci1: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0 ata0: on atapci1 pci0: at device 20.2 (no driver attached) isab0: at device 20.3 on pci0 isa0: on isab0 pcib3: at device 20.4 on pci0 pci3: on pcib3 fwohci0: port 0xe800-0xe87f mem 0xfebff800-0xfebfffff irq 20 at device 6.0 on pci3 fwohci0: OHCI version 1.10 (ROM=1) fwohci0: No. of Isochronous channels is 4. fwohci0: EUI64 00:dc:10:00:01:62:ad:eb fwohci0: Phy 1394a available S400, 2 ports. fwohci0: Link S400, max_rec 2048 bytes. firewire0: on fwohci0 fwe0: on firewire0 if_fwe0: Fake Ethernet address: 02:dc:10:62:ad:eb fwe0: Ethernet address: 02:dc:10:62:ad:eb fwip0: on firewire0 fwip0: Firewire address: 00:dc:10:00:01:62:ad:eb @ 0xfffe00000000, S400, maxrec 2048 sbp0: on firewire0 dcons_crom0: on firewire0 dcons_crom0: bus_addr 0xe271c0 fwohci0: Initiate bus reset fwohci0: fwohci_intr_core: BUS reset fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode acpi_button0: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (115200,n,8,1) fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: model IntelliMouse Explorer, device ID 4 pmtimer0 on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: parallel port not found. acpi_throttle0: on cpu0 acpi_throttle0: CLK_VAL field overlaps THT_EN bit device_attach: acpi_throttle0 attach returned 6 hwpstate0: on cpu0 Timecounters tick every 1.000 msec firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me) firewire0: bus manager 0 usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 12Mbps Full Speed USB v1.0 ugen0.1: at usbus0 uhub0: on usbus0 Expensive timeout(9) function: 0xc08ffc80(0xc7016870) 0.010591848 s ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 480Mbps High Speed USB v2.0 acd0: DVDR at ata0-slave UDMA33 ugen4.1: at usbus4 uhub4: on usbus4 ugen5.1: at usbus5 uhub5: on usbus5 ad4: 476940MB at ata2-master UDMA100 SATA 3Gb/s uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub5: 10 ports with 10 removable, self powered Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0xef3704f0 fault code = supervisor read, page not present instruction pointer = 0x20:0xc08b2eb0 stack pointer = 0x28:0xc6b27bc4 frame pointer = 0x28:0xc6b27be4 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 12 (swi4: clock) [ thread pid 12 tid 100007 ] Stopped at _mtx_lock_spin_flags+0x40: movl 0x10(%esi),%eax db> set $lines 0 db> bt Tracing pid 12 tid 100007 td 0xc6df35a0 _mtx_lock_spin_flags(ef3704e0,0,c0cef540,132,4365517,...) at _mtx_lock_spin_flags+0x40 callout_lock(c70543ec,1,c0cef540,199,c6df3844,...) at callout_lock+0x3c softclock(c0e535e0,c6b27cb8,c08b2474,c0e56f00,c6d79eb8,...) at softclock+0x36c intr_event_execute_handlers(c6dab7f8,c6d79e80,c0ce94e3,533,c6d79ef0,...) at intr_event_execute_handlers+0x125 ithread_loop(c6d83740,c6b27d28,c0ce929d,33b,c6dab7f8,...) at ithread_loop+0xac fork_exit(c089aa10,c6d83740,c6b27d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b27d60, ebp = 0 --- db> show allpcpu Current CPU: 0 cpuid = 0 dynamic pcpu = 0x606700 curthread = 0xc6df35a0: pid 12 "swi4: clock" curpcb = 0xc6b27d80 fpcurthread = none idlethread = 0xc6dad000: tid 100006 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 dynamic pcpu = 0x5ceb700 curthread = none curpcb = 0 fpcurthread = none idlethread = 0xc6dad2d0: tid 100005 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: cpuid = 2 dynamic pcpu = 0x5cee700 curthread = none curpcb = 0 fpcurthread = none idlethread = 0xc6dad5a0: tid 100004 "idle: cpu2" APIC ID = 2 currentldt = 0x50 spin locks held: cpuid = 3 dynamic pcpu = 0x5cf1700 curthread = none curpcb = 0 fpcurthread = none idlethread = 0xc6dad870: tid 100003 "idle: cpu3" APIC ID = 3 currentldt = 0x50 spin locks held: db> show alllocks db> show lockedvnods Locked vnodes db> ps pid ppid pgrp uid state wmesg wchan cmd 8 0 0 0 DL ccb_scan 0xc0e1f6d4 [xpt_thrd] 7 0 0 0 DL waiting_ 0xc0fc1e58 [sctp_iterator] 6 0 0 0 DL - 0xc705103c [fdc0] 5 0 0 0 SL - 0xc7057000 [fw0_probe] 14 0 0 0 DL (threaded) [usb] 100056 D - 0xc7033d34 [usbus5] 100055 D - 0xc7033d04 [usbus5] 100054 D - 0xc7033cd4 [usbus5] 100053 D - 0xc7033ca4 [usbus5] 100051 D - 0xc702cb5c [usbus4] 100050 D - 0xc702cb2c [usbus4] 100049 D - 0xc702cafc [usbus4] 100048 D - 0xc702cacc [usbus4] 100047 D - 0xc7027b5c [usbus3] 100046 D - 0xc7027b2c [usbus3] 100045 D - 0xc7027afc [usbus3] 100044 D - 0xc7027acc [usbus3] 100043 D - 0xc701db5c [usbus2] 100042 D - 0xc701db2c [usbus2] 100041 D - 0xc701dafc [usbus2] 100040 D - 0xc701dacc [usbus2] 100038 D - 0xc7018b5c [usbus1] 100037 D - 0xc7018b2c [usbus1] 100036 D - 0xc7018afc [usbus1] 100035 D - 0xc7018acc [usbus1] 100033 D - 0xc700eb5c [usbus0] 100032 D - 0xc700eb2c [usbus0] 100031 D - 0xc700eafc [usbus0] 100030 D - 0xc700eacc [usbus0] 13 0 0 0 DL - 0xc0e533e4 [yarrow] 4 0 0 0 DL - 0xc0e510a4 [g_down] 3 0 0 0 DL - 0xc0e510a0 [g_up] 2 0 0 0 DL - 0xc0e51098 [g_event] 12 0 0 0 RL (threaded) [intr] 100064 I [irq12: psm0] 100063 I [irq1: atkbd0] 100061 I [swi0: uart] 100058 I [irq20: fwohci0] 100057 I [irq14: ata0] 100052 I [irq19: ehci0] 100039 I [irq18: ohci2 ohci4] 100034 I [irq17: ohci1 ohci3] 100029 I [irq16: ohci0] 100028 I [irq22: atapci0] 100027 I [irq9: acpi0] 100026 I [swi5: +] 100021 I [swi2: cambio] 100020 I [swi6: task queue] 100019 I [swi6: Giant taskq] 100012 I [swi1: netisr 0] 100011 I [swi3: vm] 100010 I [swi4: clock] 100009 I [swi4: clock] 100008 I [swi4: clock] 100007 Run CPU 0 [swi4: clock] 11 0 0 0 RL (threaded) [idle] 100006 CanRun [idle: cpu0] 100005 CanRun [idle: cpu1] 100004 CanRun [idle: cpu2] 100003 CanRun [idle: cpu3] 1 0 0 0 ?L [kernel] 10 0 0 0 DL audit_wo 0xc0fcb600 [audit] 0 0 0 0 DLs (threaded) [kernel] 100065 D - 0xc0e533e4 [deadlkres] 100059 D - 0xc6d6e980 [fw0_taskq] 100025 D - 0xc6d6eb80 [kqueue taskq] 100024 D - 0xc6d6ebc0 [acpi_task_2] 100023 D - 0xc6d6ebc0 [acpi_task_1] 100022 D - 0xc6d6ebc0 [acpi_task_0] 100018 D - 0xc6d6ec80 [thread taskq] 100016 D - 0xc6d6ed00 [firmware taskq] 100000 D conifhk 0xc0deaf74 [swapper] db> allt Tracing command xpt_thrd pid 8 tid 100067 td 0xc716b5a0 sched_switch(c716b5a0,0,104,191,593a1368,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,4c,...) at mi_switch+0x200 sleepq_switch(c716b5a0,0,c0cf29c6,268,0,...) at sleepq_switch+0x15f sleepq_wait(c0e1f6d4,4c,c0c83b30,0,0,...) at sleepq_wait+0x63 _sleep(c0e1f6d4,c0e1f718,4c,c0c83b30,0,...) at _sleep+0x372 xpt_scanner_thread(0,e7523d28,c0ce929d,33b,c6dac7f8,...) at xpt_scanner_thread+0x47 fork_exit(c0486250,0,e7523d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7523d60, ebp = 0 --- Tracing command sctp_iterator pid 7 tid 100066 td 0xc716b870 sched_switch(c716b870,0,104,191,5914480a,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c716b870,0,c0cf29c6,268,0,...) at sleepq_switch+0x15f sleepq_wait(c0fc1e58,0,c0d091c5,0,0,...) at sleepq_wait+0x63 _sleep(c0fc1e58,c0fc1e20,0,c0d091c5,0,...) at _sleep+0x372 sctp_iterator_thread(0,e7520d28,c0ce929d,33b,c6dacaa0,...) at sctp_iterator_thread+0x5c fork_exit(c09ecb90,0,e7520d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7520d60, ebp = 0 --- Tracing command fdc0 pid 6 tid 100062 td 0xc70245a0 sched_switch(c70245a0,0,104,191,e1dcff1,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,4c,...) at mi_switch+0x200 sleepq_switch(c70245a0,0,c0cf29c6,28b,0,...) at sleepq_switch+0x15f sleepq_timedwait(c705103c,4c,c0ce2e1c,0,0,...) at sleepq_timedwait+0x6b _sleep(c705103c,c70510f0,4c,c0ce2e1c,3e8,...) at _sleep+0x342 fdc_thread(c7051000,e7514d28,c0ce929d,33b,c6dacd48,...) at fdc_thread+0x27d fork_exit(c0bcf6d0,c7051000,e7514d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7514d60, ebp = 0 --- Tracing command fw0_probe pid 5 tid 100060 td 0xc7024b40 sched_switch(c7024b40,0,104,191,73a3073d,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,5c,...) at mi_switch+0x200 sleepq_switch(c7024b40,0,c0cf29c6,1a7,5c,...) at sleepq_switch+0x15f sleepq_catch_signals(c0cf29c6,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7057000,5c,c0ce2e1c,100,0,...) at sleepq_wait_sig+0x17 _sleep(c7057000,c705b488,15c,c0ce2e1c,0,...) at _sleep+0x35c fw_bus_probe_thread(c7057000,e7504d28,c0ce929d,33b,c6f79000,...) at fw_bus_probe_thread+0xa08 fork_exit(c067f740,c7057000,e7504d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7504d60, ebp = 0 --- Tracing command usb pid 14 tid 100056 td 0xc704b870 sched_switch(c704b870,0,104,191,61067f0c,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c704b870,0,c0cf29c6,268,c704b870,...) at sleepq_switch+0x15f sleepq_wait(c7033d34,0,e74f1cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7033d34,c7033dd4,c0cdb4c9,6c,c7033d3c,...) at _cv_wait+0x243 usb_process(c7033d2c,e74f1d28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c7033d2c,e74f1d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74f1d60, ebp = 0 --- Tracing command usb pid 14 tid 100055 td 0xc704bb40 sched_switch(c704bb40,0,104,191,348723c4,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c704bb40,0,c0cf29c6,268,c704bb40,...) at sleepq_switch+0x15f sleepq_wait(c7033d04,0,e74eecac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7033d04,c7033dd4,c0cdb4c9,6c,c7033d0c,...) at _cv_wait+0x243 usb_process(c7033cfc,e74eed28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c7033cfc,e74eed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74eed60, ebp = 0 --- Tracing command usb pid 14 tid 100054 td 0xc704d000 sched_switch(c704d000,0,104,191,609e31e4,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c704d000,0,c0cf29c6,268,c704d000,...) at sleepq_switch+0x15f sleepq_wait(c7033cd4,0,e74ebcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7033cd4,c7033dd4,c0cdb4c9,6c,c7033cdc,...) at _cv_wait+0x243 usb_process(c7033ccc,e74ebd28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c7033ccc,e74ebd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74ebd60, ebp = 0 --- Tracing command usb pid 14 tid 100053 td 0xc704d2d0 sched_switch(c704d2d0,0,104,191,609e2124,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c704d2d0,0,c0cf29c6,268,c704d2d0,...) at sleepq_switch+0x15f sleepq_wait(c7033ca4,0,e74e8cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7033ca4,c7033dd4,c0cdb4c9,6c,c7033cac,...) at _cv_wait+0x243 usb_process(c7033c9c,e74e8d28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c7033c9c,e74e8d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74e8d60, ebp = 0 --- Tracing command usb pid 14 tid 100051 td 0xc7021000 sched_switch(c7021000,0,104,191,609e0c01,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c7021000,0,c0cf29c6,268,c7021000,...) at sleepq_switch+0x15f sleepq_wait(c702cb5c,0,e74dfcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c702cb5c,c702cbfc,c0cdb4c9,6c,c702cb64,...) at _cv_wait+0x243 usb_process(c702cb54,e74dfd28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c702cb54,e74dfd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74dfd60, ebp = 0 --- Tracing command usb pid 14 tid 100050 td 0xc70212d0 sched_switch(c70212d0,0,104,191,2b9f10c0,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c70212d0,0,c0cf29c6,268,c70212d0,...) at sleepq_switch+0x15f sleepq_wait(c702cb2c,0,e74dccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c702cb2c,c702cbfc,c0cdb4c9,6c,c702cb34,...) at _cv_wait+0x243 usb_process(c702cb24,e74dcd28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c702cb24,e74dcd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74dcd60, ebp = 0 --- Tracing command usb pid 14 tid 100049 td 0xc70215a0 sched_switch(c70215a0,0,104,191,60395d26,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c70215a0,0,c0cf29c6,268,c70215a0,...) at sleepq_switch+0x15f sleepq_wait(c702cafc,0,e74d9cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c702cafc,c702cbfc,c0cdb4c9,6c,c702cb04,...) at _cv_wait+0x243 usb_process(c702caf4,e74d9d28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c702caf4,e74d9d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74d9d60, ebp = 0 --- Tracing command usb pid 14 tid 100048 td 0xc7021870 sched_switch(c7021870,0,104,191,60394a25,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c7021870,0,c0cf29c6,268,c7021870,...) at sleepq_switch+0x15f sleepq_wait(c702cacc,0,e74d6cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c702cacc,c702cbfc,c0cdb4c9,6c,c702cad4,...) at _cv_wait+0x243 usb_process(c702cac4,e74d6d28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c702cac4,e74d6d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74d6d60, ebp = 0 --- Tracing command usb pid 14 tid 100047 td 0xc7021b40 sched_switch(c7021b40,0,104,191,603935e8,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c7021b40,0,c0cf29c6,268,c7021b40,...) at sleepq_switch+0x15f sleepq_wait(c7027b5c,0,e74d2cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7027b5c,c7027bfc,c0cdb4c9,6c,c7027b64,...) at _cv_wait+0x243 usb_process(c7027b54,e74d2d28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c7027b54,e74d2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74d2d60, ebp = 0 --- Tracing command usb pid 14 tid 100046 td 0xc7023000 sched_switch(c7023000,0,104,191,284bae5d,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c7023000,0,c0cf29c6,268,c7023000,...) at sleepq_switch+0x15f sleepq_wait(c7027b2c,0,e74cfcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7027b2c,c7027bfc,c0cdb4c9,6c,c7027b34,...) at _cv_wait+0x243 usb_process(c7027b24,e74cfd28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c7027b24,e74cfd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74cfd60, ebp = 0 --- Tracing command usb pid 14 tid 100045 td 0xc70232d0 sched_switch(c70232d0,0,104,191,5a6eef10,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c70232d0,0,c0cf29c6,268,c70232d0,...) at sleepq_switch+0x15f sleepq_wait(c7027afc,0,e74cccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7027afc,c7027bfc,c0cdb4c9,6c,c7027b04,...) at _cv_wait+0x243 usb_process(c7027af4,e74ccd28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c7027af4,e74ccd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74ccd60, ebp = 0 --- Tracing command usb pid 14 tid 100044 td 0xc70235a0 sched_switch(c70235a0,0,104,191,5a6ede5d,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c70235a0,0,c0cf29c6,268,c70235a0,...) at sleepq_switch+0x15f sleepq_wait(c7027acc,0,e74c9cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7027acc,c7027bfc,c0cdb4c9,6c,c7027ad4,...) at _cv_wait+0x243 usb_process(c7027ac4,e74c9d28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c7027ac4,e74c9d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74c9d60, ebp = 0 --- Tracing command usb pid 14 tid 100043 td 0xc7023870 sched_switch(c7023870,0,104,191,5a6ec777,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c7023870,0,c0cf29c6,268,c7023870,...) at sleepq_switch+0x15f sleepq_wait(c701db5c,0,e74c5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c701db5c,c701dbfc,c0cdb4c9,6c,c701db64,...) at _cv_wait+0x243 usb_process(c701db54,e74c5d28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c701db54,e74c5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74c5d60, ebp = 0 --- Tracing command usb pid 14 tid 100042 td 0xc7023b40 sched_switch(c7023b40,0,104,191,271d8cb8,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c7023b40,0,c0cf29c6,268,c7023b40,...) at sleepq_switch+0x15f sleepq_wait(c701db2c,0,e74c2cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c701db2c,c701dbfc,c0cdb4c9,6c,c701db34,...) at _cv_wait+0x243 usb_process(c701db24,e74c2d28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c701db24,e74c2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74c2d60, ebp = 0 --- Tracing command usb pid 14 tid 100041 td 0xc7024000 sched_switch(c7024000,0,104,191,5a0a0f53,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c7024000,0,c0cf29c6,268,c7024000,...) at sleepq_switch+0x15f sleepq_wait(c701dafc,0,e74bfcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c701dafc,c701dbfc,c0cdb4c9,6c,c701db04,...) at _cv_wait+0x243 usb_process(c701daf4,e74bfd28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c701daf4,e74bfd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74bfd60, ebp = 0 --- Tracing command usb pid 14 tid 100040 td 0xc7014000 sched_switch(c7014000,0,104,191,5a09fcd2,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c7014000,0,c0cf29c6,268,c7014000,...) at sleepq_switch+0x15f sleepq_wait(c701dacc,0,e74bccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c701dacc,c701dbfc,c0cdb4c9,6c,c701dad4,...) at _cv_wait+0x243 usb_process(c701dac4,e74bcd28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c701dac4,e74bcd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74bcd60, ebp = 0 --- Tracing command usb pid 14 tid 100038 td 0xc70145a0 sched_switch(c70145a0,0,104,191,5a09e923,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c70145a0,0,c0cf29c6,268,c70145a0,...) at sleepq_switch+0x15f sleepq_wait(c7018b5c,0,e74b5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7018b5c,c7018bfc,c0cdb4c9,6c,c7018b64,...) at _cv_wait+0x243 usb_process(c7018b54,e74b5d28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c7018b54,e74b5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74b5d60, ebp = 0 --- Tracing command usb pid 14 tid 100037 td 0xc7014870 sched_switch(c7014870,0,104,191,25d3ee05,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c7014870,0,c0cf29c6,268,c7014870,...) at sleepq_switch+0x15f sleepq_wait(c7018b2c,0,e74b2cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7018b2c,c7018bfc,c0cdb4c9,6c,c7018b34,...) at _cv_wait+0x243 usb_process(c7018b24,e74b2d28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c7018b24,e74b2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74b2d60, ebp = 0 --- Tracing command usb pid 14 tid 100036 td 0xc7014b40 sched_switch(c7014b40,0,104,191,59a49d1d,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c7014b40,0,c0cf29c6,268,c7014b40,...) at sleepq_switch+0x15f sleepq_wait(c7018afc,0,e74afcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7018afc,c7018bfc,c0cdb4c9,6c,c7018b04,...) at _cv_wait+0x243 usb_process(c7018af4,e74afd28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c7018af4,e74afd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74afd60, ebp = 0 --- Tracing command usb pid 14 tid 100035 td 0xc7016000 sched_switch(c7016000,0,104,191,59a48ae0,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c7016000,0,c0cf29c6,268,c7016000,...) at sleepq_switch+0x15f sleepq_wait(c7018acc,0,e74accac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7018acc,c7018bfc,c0cdb4c9,6c,c7018ad4,...) at _cv_wait+0x243 usb_process(c7018ac4,e74acd28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c7018ac4,e74acd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74acd60, ebp = 0 --- Tracing command usb pid 14 tid 100033 td 0xc70165a0 sched_switch(c70165a0,0,104,191,59a475b5,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c70165a0,0,c0cf29c6,268,c70165a0,...) at sleepq_switch+0x15f sleepq_wait(c700eb5c,0,e74a5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c700eb5c,c700ebfc,c0cdb4c9,6c,c700eb64,...) at _cv_wait+0x243 usb_process(c700eb54,e74a5d28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c700eb54,e74a5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74a5d60, ebp = 0 --- Tracing command usb pid 14 tid 100032 td 0xc7016870 sched_switch(c7016870,0,104,191,24010f36,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c7016870,0,c0cf29c6,268,c7016870,...) at sleepq_switch+0x15f sleepq_wait(c700eb2c,0,e74a2cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c700eb2c,c700ebfc,c0cdb4c9,6c,c700eb34,...) at _cv_wait+0x243 usb_process(c700eb24,e74a2d28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c700eb24,e74a2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74a2d60, ebp = 0 --- Tracing command usb pid 14 tid 100031 td 0xc7016b40 sched_switch(c7016b40,0,104,191,593f3d9d,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c7016b40,0,c0cf29c6,268,c7016b40,...) at sleepq_switch+0x15f sleepq_wait(c700eafc,0,e749fcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c700eafc,c700ebfc,c0cdb4c9,6c,c700eb04,...) at _cv_wait+0x243 usb_process(c700eaf4,e749fd28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c700eaf4,e749fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe749fd60, ebp = 0 --- Tracing command usb pid 14 tid 100030 td 0xc6ebb2d0 sched_switch(c6ebb2d0,0,104,191,593f295e,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c6ebb2d0,0,c0cf29c6,268,c6ebb2d0,...) at sleepq_switch+0x15f sleepq_wait(c700eacc,0,e749ccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c700eacc,c700ebfc,c0cdb4c9,6c,c700ead4,...) at _cv_wait+0x243 usb_process(c700eac4,e749cd28,c0ce929d,33b,c6f792a8,...) at usb_process+0x193 fork_exit(c07ec100,c700eac4,e749cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe749cd60, ebp = 0 --- Tracing command yarrow pid 13 tid 100017 td 0xc6eba5a0 sched_switch(c6eba5a0,0,104,191,67b3d01a,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c6eba5a0,0,c0cf29c6,28b,2,...) at sleepq_switch+0x15f sleepq_timedwait(c0e533e4,0,c0ce2e1c,2,0,...) at sleepq_timedwait+0x6b _sleep(c0e533e4,0,0,c0ce2e1c,64,...) at _sleep+0x342 pause(c0ce2e1c,64,c0cceb8f,111,0,...) at pause+0x47 random_kthread(0,c6b45d28,c0ce929d,33b,c6f79550,...) at random_kthread+0x1ef fork_exit(c075ce00,0,c6b45d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b45d60, ebp = 0 --- Tracing command g_down pid 4 tid 100015 td 0xc6ebab40 sched_switch(c6ebab40,0,104,191,8432a8e5,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,4c,...) at mi_switch+0x200 sleepq_switch(c6ebab40,0,c0cf29c6,268,0,...) at sleepq_switch+0x15f sleepq_wait(c0e510a4,4c,c0ce2e1c,0,0,...) at sleepq_wait+0x63 _sleep(c0e510a4,c0e51008,24c,c0ce2e1c,0,...) at _sleep+0x372 g_io_schedule_down(c6ebab40,0,c0ce45a3,74,0,...) at g_io_schedule_down+0x5b g_down_procbody(0,c6b3fd28,c0ce929d,33b,c6dab000,...) at g_down_procbody+0x8d fork_exit(c085bd40,0,c6b3fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b3fd60, ebp = 0 --- Tracing command g_up pid 3 tid 100014 td 0xc6ebb000 sched_switch(c6ebb000,0,104,191,8432ea1c,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,4c,...) at mi_switch+0x200 sleepq_switch(c6ebb000,0,c0cf29c6,268,0,...) at sleepq_switch+0x15f sleepq_wait(c0e510a0,4c,c0ce2e1c,0,0,...) at sleepq_wait+0x63 _sleep(c0e510a0,c0e51028,24c,c0ce2e1c,0,...) at _sleep+0x372 g_io_schedule_up(c6ebb000,0,c0ce45a3,5d,0,...) at g_io_schedule_up+0x134 g_up_procbody(0,c6b3cd28,c0ce929d,33b,c6dab2a8,...) at g_up_procbody+0x8d fork_exit(c085bdd0,0,c6b3cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b3cd60, ebp = 0 --- Tracing command g_event pid 2 tid 100013 td 0xc6dae2d0 sched_switch(c6dae2d0,0,104,191,8437e235,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,4c,...) at mi_switch+0x200 sleepq_switch(c6dae2d0,0,c0cf29c6,268,0,...) at sleepq_switch+0x15f sleepq_wait(c0e51098,4c,c0ce2e1c,0,0,...) at sleepq_wait+0x63 _sleep(c0e51098,c0e50fe8,24c,c0ce2e1c,0,...) at _sleep+0x372 g_run_events(c0e56f00,0,c0ce45a3,8b) at g_run_events+0x526 g_event_procbody(0,c6b39d28,c0ce929d,33b,c6dab550,...) at g_event_procbody+0x8a fork_exit(c085be60,0,c6b39d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b39d60, ebp = 0 --- Tracing command intr pid 12 tid 100064 td 0xc716c000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100063 td 0xc70242d0 sched_switch(c70242d0,0,109,191,593a3a0e,...) at sched_switch+0x3bc mi_switch(109,0,c0ce94e3,52d,c6d789f0,...) at mi_switch+0x200 ithread_loop(c71548b0,e7517d28,c0ce929d,33b,c6dab7f8,...) at ithread_loop+0x206 fork_exit(c089aa10,c71548b0,e7517d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7517d60, ebp = 0 --- Tracing command intr pid 12 tid 100061 td 0xc7024870 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100058 td 0xc704b2d0 sched_switch(c704b2d0,0,109,191,58dfda44,...) at sched_switch+0x3bc mi_switch(109,0,c0ce94e3,52d,c6d78070,...) at mi_switch+0x200 ithread_loop(c7013c90,e74fbd28,c0ce929d,33b,c6dab7f8,...) at ithread_loop+0x206 fork_exit(c089aa10,c7013c90,e74fbd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74fbd60, ebp = 0 --- Tracing command intr pid 12 tid 100057 td 0xc704b5a0 sched_switch(c704b5a0,0,109,191,663207a7,...) at sched_switch+0x3bc mi_switch(109,0,c0ce94e3,52d,c6d78370,...) at mi_switch+0x200 ithread_loop(c7013310,e74f7d28,c0ce929d,33b,c6dab7f8,...) at ithread_loop+0x206 fork_exit(c089aa10,c7013310,e74f7d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74f7d60, ebp = 0 --- Tracing command intr pid 12 tid 100052 td 0xc704d5a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100039 td 0xc70142d0 sched_switch(c70142d0,0,109,191,7e8e29c4,...) at sched_switch+0x3bc mi_switch(109,0,c0ce94e3,52d,c6d78170,...) at mi_switch+0x200 ithread_loop(c70048c0,e74b9d28,c0ce929d,33b,c6dab7f8,...) at ithread_loop+0x206 fork_exit(c089aa10,c70048c0,e74b9d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74b9d60, ebp = 0 --- Tracing command intr pid 12 tid 100034 td 0xc70162d0 sched_switch(c70162d0,0,109,191,78c3f05c,...) at sched_switch+0x3bc mi_switch(109,0,c0ce94e3,52d,c6d781f0,...) at mi_switch+0x200 ithread_loop(c7004ad0,e74a9d28,c0ce929d,33b,c6dab7f8,...) at ithread_loop+0x206 fork_exit(c089aa10,c7004ad0,e74a9d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74a9d60, ebp = 0 --- Tracing command intr pid 12 tid 100029 td 0xc6ebb5a0 sched_switch(c6ebb5a0,0,109,191,76ef89b3,...) at sched_switch+0x3bc mi_switch(109,0,c0ce94e3,52d,c6d78270,...) at mi_switch+0x200 ithread_loop(c6fe9300,e7499d28,c0ce929d,33b,c6dab7f8,...) at ithread_loop+0x206 fork_exit(c089aa10,c6fe9300,e7499d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7499d60, ebp = 0 --- Tracing command intr pid 12 tid 100028 td 0xc6ebb870 sched_switch(c6ebb870,0,109,191,841626ef,...) at sched_switch+0x3bc mi_switch(109,0,c0ce94e3,52d,c6df20f0,...) at mi_switch+0x200 ithread_loop(c70042a0,e738ad28,c0ce929d,33b,c6dab7f8,...) at ithread_loop+0x206 fork_exit(c089aa10,c70042a0,e738ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe738ad60, ebp = 0 --- Tracing command intr pid 12 tid 100027 td 0xc6ebbb40 sched_switch(c6ebbb40,0,109,191,593ae24a,...) at sched_switch+0x3bc mi_switch(109,0,c0ce94e3,52d,c6d785f0,...) at mi_switch+0x200 ithread_loop(c6faf250,c6b8ad28,c0ce929d,33b,c6dab7f8,...) at ithread_loop+0x206 fork_exit(c089aa10,c6faf250,c6b8ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b8ad60, ebp = 0 --- Tracing command intr pid 12 tid 100026 td 0xc6fa6000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100021 td 0xc6df3870 sched_switch(c6df3870,0,109,191,70145fdb,...) at sched_switch+0x3bc mi_switch(109,0,c0ce94e3,52d,c6d79370,...) at mi_switch+0x200 ithread_loop(c6d834a0,c6b51d28,c0ce929d,33b,c6dab7f8,...) at ithread_loop+0x206 fork_exit(c089aa10,c6d834a0,c6b51d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b51d60, ebp = 0 --- Tracing command intr pid 12 tid 100020 td 0xc6df3b40 sched_switch(c6df3b40,0,109,191,6b772030,...) at sched_switch+0x3bc mi_switch(109,0,c0ce94e3,52d,c6d793f0,...) at mi_switch+0x200 ithread_loop(c6d834b0,c6b4ed28,c0ce929d,33b,c6dab7f8,...) at ithread_loop+0x206 fork_exit(c089aa10,c6d834b0,c6b4ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b4ed60, ebp = 0 --- Tracing command intr pid 12 tid 100019 td 0xc6eba000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100012 td 0xc6dae5a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100011 td 0xc6dae870 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100010 td 0xc6daeb40 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100009 td 0xc6df3000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100008 td 0xc6df32d0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100007 td 0xc6df35a0 _mtx_lock_spin_flags(ef3704e0,0,c0cef540,132,4365517,...) at _mtx_lock_spin_flags+0x40 callout_lock(c70543ec,1,c0cef540,199,c6df3844,...) at callout_lock+0x3c softclock(c0e535e0,c6b27cb8,c08b2474,c0e56f00,c6d79eb8,...) at softclock+0x36c intr_event_execute_handlers(c6dab7f8,c6d79e80,c0ce94e3,533,c6d79ef0,...) at intr_event_execute_handlers+0x125 ithread_loop(c6d83740,c6b27d28,c0ce929d,33b,c6dab7f8,...) at ithread_loop+0xac fork_exit(c089aa10,c6d83740,c6b27d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b27d60, ebp = 0 --- Tracing command idle pid 11 tid 100006 td 0xc6dad000 sched_switch(c6dad000,0,608,18c,7011803b,...) at sched_switch+0x3bc mi_switch(608,0,c0cee78d,cd,0,...) at mi_switch+0x200 critical_exit(0,2710,c0e56f00,c6dad000,c6b23ce8,...) at critical_exit+0xa8 cpu_idle(0,0,c0cefd5b,9d9,c6dad000,...) at cpu_idle+0x9e sched_idletd(0,c6b23d28,c0ce929d,33b,c6dabaa0,...) at sched_idletd+0x263 fork_exit(c08e8210,0,c6b23d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b23d60, ebp = 0 --- Tracing command idle pid 11 tid 100005 td 0xc6dad2d0 fork_trampoline() at fork_trampoline Tracing command idle pid 11 tid 100004 td 0xc6dad5a0 fork_trampoline() at fork_trampoline Tracing command idle pid 11 tid 100003 td 0xc6dad870 fork_trampoline() at fork_trampoline Tracing command kernel pid 1 tid 100002 td 0xc6dadb40 fork_trampoline() at fork_trampoline Tracing command audit pid 10 tid 100001 td 0xc6dae000 sched_switch(c6dae000,0,104,191,593c81aa,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c6dae000,0,c0cf29c6,268,c6dae000,...) at sleepq_switch+0x15f sleepq_wait(c0fcb600,0,c6b13c8c,1,0,...) at sleepq_wait+0x63 _cv_wait(c0fcb600,c0fcb5e4,c0d14123,194,0,...) at _cv_wait+0x243 audit_worker(0,c6b13d28,c0ce929d,33b,c6dac000,...) at audit_worker+0x84 fork_exit(c0ad3af0,0,c6b13d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b13d60, ebp = 0 --- Tracing command kernel pid 0 tid 100065 td 0xc716bb40 sched_switch(c716bb40,0,104,191,a2d50d32,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c716bb40,0,c0cf29c6,28b,2,...) at sleepq_switch+0x15f sleepq_timedwait(c0e533e4,0,c0ce2e1c,2,0,...) at sleepq_timedwait+0x6b _sleep(c0e533e4,0,0,c0ce2e1c,bb8,...) at _sleep+0x342 pause(c0ce2e1c,bb8,120,11e,c0e51208,...) at pause+0x47 deadlkres(0,e751dd28,c0ce929d,33b,c0e51180,...) at deadlkres+0x302 fork_exit(c087cca0,0,e751dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe751dd60, ebp = 0 --- Tracing command kernel pid 0 tid 100059 td 0xc704b000 sched_switch(c704b000,0,104,191,58e029a2,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c704b000,0,c0cf29c6,268,c0e51430,...) at sleepq_switch+0x15f sleepq_wait(c6d6e980,0,c0ceec90,c0ce2e1c,0,...) at sleepq_wait+0x63 msleep_spin(c6d6e980,c6d6e998,c0ce2e1c,0,c0cec850,...) at msleep_spin+0x21d taskqueue_thread_loop(c705b49c,e7501d28,c0ce929d,33b,c0e51180,...) at taskqueue_thread_loop+0x8f fork_exit(c0902ee0,c705b49c,e7501d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7501d60, ebp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xc6fa62d0 sched_switch(c6fa62d0,0,104,191,6116d4c2,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c6fa62d0,0,c0cf29c6,268,0,...) at sleepq_switch+0x15f sleepq_wait(c6d6eb80,0,c0ce2e1c,0,0,...) at sleepq_wait+0x63 _sleep(c6d6eb80,c6d6eb98,0,c0ce2e1c,0,...) at _sleep+0x372 taskqueue_thread_loop(c0e51ad8,c6b5dd28,c0ce929d,33b,c0e51180,...) at taskqueue_thread_loop+0xb5 fork_exit(c0902ee0,c0e51ad8,c6b5dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b5dd60, ebp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xc6fa65a0 sched_switch(c6fa65a0,0,104,191,6116bded,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c6fa65a0,0,c0cf29c6,268,0,...) at sleepq_switch+0x15f sleepq_wait(c6d6ebc0,0,c0ceec90,c0ce2e1c,0,...) at sleepq_wait+0x63 msleep_spin(c6d6ebc0,c6d6ebd8,c0ce2e1c,0,c0cec850,...) at msleep_spin+0x21d taskqueue_thread_loop(c0e225a0,c6b5ad28,c0ce929d,33b,c0e51180,...) at taskqueue_thread_loop+0x8f fork_exit(c0902ee0,c0e225a0,c6b5ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b5ad60, ebp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xc6fa6870 sched_switch(c6fa6870,0,104,191,6116aefc,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c6fa6870,0,c0cf29c6,268,0,...) at sleepq_switch+0x15f sleepq_wait(c6d6ebc0,0,c0ceec90,c0ce2e1c,0,...) at sleepq_wait+0x63 msleep_spin(c6d6ebc0,c6d6ebd8,c0ce2e1c,0,c0cec850,...) at msleep_spin+0x21d taskqueue_thread_loop(c0e225a0,c6b57d28,c0ce929d,33b,c0e51180,...) at taskqueue_thread_loop+0x8f fork_exit(c0902ee0,c0e225a0,c6b57d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b57d60, ebp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xc6fa6b40 sched_switch(c6fa6b40,0,104,191,61169d45,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c6fa6b40,0,c0cf29c6,268,0,...) at sleepq_switch+0x15f sleepq_wait(c6d6ebc0,0,c0ceec90,c0ce2e1c,0,...) at sleepq_wait+0x63 msleep_spin(c6d6ebc0,c6d6ebd8,c0ce2e1c,0,c0cec850,...) at msleep_spin+0x21d taskqueue_thread_loop(c0e225a0,c6b54d28,c0ce929d,33b,c0e51180,...) at taskqueue_thread_loop+0x8f fork_exit(c0902ee0,c0e225a0,c6b54d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b54d60, ebp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xc6eba2d0 sched_switch(c6eba2d0,0,104,191,61117b18,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c6eba2d0,0,c0cf29c6,268,0,...) at sleepq_switch+0x15f sleepq_wait(c6d6ec80,0,c0ce2e1c,0,0,...) at sleepq_wait+0x63 _sleep(c6d6ec80,c6d6ec98,0,c0ce2e1c,0,...) at _sleep+0x372 taskqueue_thread_loop(c0e65c28,c6b48d28,c0ce929d,33b,c0e51180,...) at taskqueue_thread_loop+0xb5 fork_exit(c0902ee0,c0e65c28,c6b48d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b48d60, ebp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xc6eba870 sched_switch(c6eba870,0,104,191,610c3369,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,0,...) at mi_switch+0x200 sleepq_switch(c6eba870,0,c0cf29c6,268,0,...) at sleepq_switch+0x15f sleepq_wait(c6d6ed00,0,c0ce2e1c,0,0,...) at sleepq_wait+0x63 _sleep(c6d6ed00,c6d6ed18,0,c0ce2e1c,0,...) at _sleep+0x372 taskqueue_thread_loop(c0e646a0,c6b42d28,c0ce929d,33b,c0e51180,...) at taskqueue_thread_loop+0xb5 fork_exit(c0902ee0,c0e646a0,c6b42d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b42d60, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc0e51430 sched_switch(c0e51430,0,104,191,6b784782,...) at sched_switch+0x3bc mi_switch(104,0,c0cf29c6,1f3,60,...) at mi_switch+0x200 sleepq_switch(c0e51430,0,c0cf29c6,28b,0,...) at sleepq_switch+0x15f sleepq_timedwait(c0deaf74,60,c0cf08d3,0,0,...) at sleepq_timedwait+0x6b _sleep(c0deaf74,c0e63f48,60,c0cf08d3,ea60,...) at _sleep+0x342 boot_run_interrupt_driven_config_hooks(0,141ec00,141ec00,141e000,1425000,...) at boot_run_interrupt_driven_config_hooks+0x7f mi_startup() at mi_startup+0xac begin() at begin+0x2c db> call doadump Cannot dump. Device not defined or unavailable. = 0x30 db> reset (kgdb) l *callout_lock+0x3c 0xc08d54ec is in callout_lock (../../../kern/kern_timeout.c:307). 302 303 for (;;) { 304 cpu = c->c_cpu; 305 cc = CC_CPU(cpu); 306 CC_LOCK(cc); 307 if (cpu == c->c_cpu) 308 break; 309 CC_UNLOCK(cc); 310 } 311 return (cc); (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/kern/kern_timeout.c =================================================================== --- /usr/src/sys/kern/kern_timeout.c (revision 216137) +++ /usr/src/sys/kern/kern_timeout.c (working copy) @@ -105,12 +105,10 @@ struct callout_cpu { struct callout_tailq *cc_callwheel; struct callout_list cc_callfree; struct callout *cc_next; - struct callout *cc_curr; void *cc_cookie; int cc_ticks; int cc_softticks; int cc_cancel; - int cc_waiting; int cc_firsttick; }; @@ -133,19 +131,12 @@ MALLOC_DEFINE(M_CALLOUT, "callout", "Callout datas /** * Locked by cc_lock: - * cc_curr - If a callout is in progress, it is curr_callout. - * If curr_callout is non-NULL, threads waiting in - * callout_drain() will be woken up as soon as the - * relevant callout completes. * cc_cancel - Changing to 1 with both callout_lock and c_lock held * guarantees that the current callout will not run. * The softclock() function sets this to 0 before it * drops callout_lock to acquire c_lock, and it calls * the handler only if curr_cancelled is still 0 after * c_lock is successfully acquired. - * cc_waiting - If a thread is waiting in callout_drain(), then - * callout_wait is nonzero. Set only when - * curr_callout is non-NULL. */ /* @@ -340,6 +331,7 @@ void softclock(void *arg) { struct callout_cpu *cc; + struct callout_cpu *cc_handover; struct callout *c; struct callout_tailq *bucket; int curticks; @@ -412,7 +404,7 @@ softclock(void *arg) c->c_flags = (c->c_flags & ~CALLOUT_PENDING); } - cc->cc_curr = c; + c->c_flags |= CALLOUT_EXECUTING; cc->cc_cancel = 0; CC_UNLOCK(cc); if (c_lock != NULL) { @@ -476,8 +468,13 @@ softclock(void *arg) if ((c_flags & CALLOUT_RETURNUNLOCKED) == 0) class->lc_unlock(c_lock); skip: - CC_LOCK(cc); + /* + * Callout may have migrated, thus execute the + * handover operations with the correct lock. + */ + cc_handover = callout_lock(c); + /* * If the current callout is locally * allocated (from timeout(9)) * then put it on the freelist. @@ -488,24 +485,23 @@ softclock(void *arg) * callout pointer. */ if (c_flags & CALLOUT_LOCAL_ALLOC) { - KASSERT(c->c_flags == - CALLOUT_LOCAL_ALLOC, - ("corrupted callout")); + MPASS(cc == cc_handover); c->c_func = NULL; SLIST_INSERT_HEAD(&cc->cc_callfree, c, c_links.sle); } - cc->cc_curr = NULL; - if (cc->cc_waiting) { + c->c_flags &= ~CALLOUT_EXECUTING; + if (c->c_flags & CALLOUT_DRAINING) { /* * There is someone waiting * for the callout to complete. */ - cc->cc_waiting = 0; - CC_UNLOCK(cc); - wakeup(&cc->cc_waiting); - CC_LOCK(cc); - } + c->c_flags &= ~CALLOUT_DRAINING; + CC_UNLOCK(cc_handover); + wakeup(&c->c_flags); + } else + CC_UNLOCK(cc_handover); + CC_LOCK(cc); steps = 0; c = cc->cc_next; } @@ -619,7 +615,7 @@ callout_reset_on(struct callout *c, int to_ticks, cpu = c->c_cpu; retry: cc = callout_lock(c); - if (cc->cc_curr == c) { + if (c->c_flags & CALLOUT_EXECUTING) { /* * We're being asked to reschedule a callout which is * currently in progress. If there is a lock then we @@ -627,7 +623,7 @@ retry: */ if (c->c_lock != NULL && !cc->cc_cancel) cancelled = cc->cc_cancel = 1; - if (cc->cc_waiting) { + if (c->c_flags & CALLOUT_DRAINING) { /* * Someone has called callout_drain to kill this * callout. Don't reschedule. @@ -735,12 +731,12 @@ again: * If it wasn't on the queue and it isn't the current * callout, then we can't stop it, so just bail. */ - if (cc->cc_curr != c) { + if (!(c->c_flags & CALLOUT_EXECUTING)) { CTR3(KTR_CALLOUT, "failed to stop %p func %p arg %p", c, c->c_func, c->c_arg); CC_UNLOCK(cc); if (sq_locked) - sleepq_release(&cc->cc_waiting); + sleepq_release(&c->c_flags); return (0); } @@ -751,7 +747,7 @@ again: * just wait for the current invocation to * finish. */ - while (cc->cc_curr == c) { + while (c->c_flags & CALLOUT_EXECUTING) { /* * Use direct calls to sleepqueue interface @@ -772,17 +768,16 @@ again: */ if (!sq_locked) { CC_UNLOCK(cc); - sleepq_lock(&cc->cc_waiting); + sleepq_lock(&c->c_flags); sq_locked = 1; goto again; } - cc->cc_waiting = 1; + c->c_flags |= CALLOUT_DRAINING; DROP_GIANT(); CC_UNLOCK(cc); - sleepq_add(&cc->cc_waiting, - &cc->cc_lock.lock_object, "codrain", + sleepq_add(&c->c_flags, NULL, "codrain", SLEEPQ_SLEEP, 0); - sleepq_wait(&cc->cc_waiting, 0); + sleepq_wait(&c->c_flags, 0); sq_locked = 0; /* Reacquire locks previously released. */ @@ -811,7 +806,7 @@ again: return (0); } if (sq_locked) - sleepq_release(&cc->cc_waiting); + sleepq_release(&c->c_flags); c->c_flags &= ~(CALLOUT_ACTIVE | CALLOUT_PENDING); Index: /usr/src/sys/sys/callout.h =================================================================== --- /usr/src/sys/sys/callout.h (revision 216137) +++ /usr/src/sys/sys/callout.h (working copy) @@ -64,6 +64,8 @@ struct callout { #define CALLOUT_MPSAFE 0x0008 /* callout handler is mp safe */ #define CALLOUT_RETURNUNLOCKED 0x0010 /* handler returns with mtx unlocked */ #define CALLOUT_SHAREDLOCK 0x0020 /* callout lock held in shared mode */ +#define CALLOUT_EXECUTING 0x0040 /* callout is currently executing */ +#define CALLOUT_DRAINING 0x0080 /* callout is under draining op */ struct callout_handle { struct callout *callout;