GENERIC stable8 from 2011-03-25 13:09:57 UTC, r220001M, vmcore.1 KDB: current backend: ddb 524288K of memory above 4GB ignored Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.2-STABLE #0 r220001M: Thu Mar 31 10:04:52 CEST 2011 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 WARNING: DIAGNOSTIC option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1799.99-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Family = 10 Model = 2 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 4294967296 (4096 MB) avail memory = 3536519168 (3372 MB) ACPI APIC Table: <060508 APIC1048> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: <060508 RSDT1048> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of fee00000, 1000 (3) failed acpi0: reservation of ffb80000, 80000 (3) failed acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, d7f00000 (3) failed Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 vgapci0: port 0xc000-0xc0ff mem 0xf0000000-0xf7ffffff,0xfe9f0000-0xfe9fffff,0xfe800000-0xfe8fffff irq 18 at device 5.0 on pci1 pci1: at device 5.2 (no driver attached) pcib2: at device 7.0 on pci0 pci2: on pcib2 re0: port 0xd800-0xd8ff mem 0xfeaff000-0xfeafffff irq 19 at device 0.0 on pci2 re0: Using 1 MSI message re0: Chip rev. 0x38000000 re0: MAC rev. 0x00000000 miibus0: on re0 rgephy0: PHY 1 on miibus0 rgephy0: 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re0: Ethernet address: 00:21:85:15:33:7d re0: [ITHREAD] atapci0: port 0xb000-0xb007,0xa000-0xa003,0x9000-0x9007,0x8000-0x8003,0x7000-0x700f mem 0xfe7ff800-0xfe7ffbff irq 22 at device 18.0 on pci0 atapci0: [ITHREAD] atapci0: AHCI v1.10 controller with 4 3Gbps ports, PM supported ata2: on atapci0 ata2: port is not ready (timeout 0ms) tfd = 000001d0 ata2: software reset clear timeout ata2: [ITHREAD] ata3: on atapci0 ata3: [ITHREAD] ata4: on atapci0 ata4: [ITHREAD] ata5: on atapci0 ata5: [ITHREAD] ohci0: mem 0xfe7fe000-0xfe7fefff irq 16 at device 19.0 on pci0 ohci0: [ITHREAD] usbus0: on ohci0 ohci1: mem 0xfe7fd000-0xfe7fdfff irq 17 at device 19.1 on pci0 ohci1: [ITHREAD] usbus1: on ohci1 ohci2: mem 0xfe7fc000-0xfe7fcfff irq 18 at device 19.2 on pci0 ohci2: [ITHREAD] usbus2: on ohci2 ohci3: mem 0xfe7fb000-0xfe7fbfff irq 17 at device 19.3 on pci0 ohci3: [ITHREAD] usbus3: on ohci3 ohci4: mem 0xfe7fa000-0xfe7fafff irq 18 at device 19.4 on pci0 ohci4: [ITHREAD] usbus4: on ohci4 ehci0: mem 0xfe7ff000-0xfe7ff0ff irq 19 at device 19.5 on pci0 ehci0: [ITHREAD] ehci0: AMD SB600/700 quirk applied usbus5: EHCI version 1.0 usbus5: on ehci0 pci0: at device 20.0 (no driver attached) atapci1: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0 ata0: on atapci1 ata0: [ITHREAD] pci0: at device 20.2 (no driver attached) isab0: at device 20.3 on pci0 isa0: on isab0 pcib3: at device 20.4 on pci0 pci3: on pcib3 fwohci0: port 0xe800-0xe87f mem 0xfebff800-0xfebfffff irq 20 at device 6.0 on pci3 fwohci0: [ITHREAD] fwohci0: OHCI version 1.10 (ROM=1) fwohci0: No. of Isochronous channels is 4. fwohci0: EUI64 00:dc:10:00:01:62:ad:eb fwohci0: Phy 1394a available S400, 2 ports. fwohci0: Link S400, max_rec 2048 bytes. firewire0: on fwohci0 dcons_crom0: on firewire0 dcons_crom0: bus_addr 0xe047a0 fwe0: on firewire0 if_fwe0: Fake Ethernet address: 02:dc:10:62:ad:eb fwe0: Ethernet address: 02:dc:10:62:ad:eb fwip0: on firewire0 fwip0: Firewire address: 00:dc:10:00:01:62:ad:eb @ 0xfffe00000000, S400, maxrec 2048 fwohci0: Initiate bus reset fwohci0: fwohci_intr_core: BUS reset fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode acpi_button0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart0: console (115200,n,8,1) fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model IntelliMouse Explorer, device ID 4 pmtimer0 on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: parallel port not found. acpi_throttle0: on cpu0 acpi_throttle0: CLK_VAL field overlaps THT_EN bit device_attach: acpi_throttle0 attach returned 6 hwpstate0: on cpu0 Timecounters tick every 1.000 msec firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me) firewire0: bus manager 0 usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 12Mbps Full Speed USB v1.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 480Mbps High Speed USB v2.0 acd0: DVDR at ata0-slave UDMA33 ugen0.1: at usbus0 uhub0: on usbus0 Expensive timeout(9) function: 0xc08ff3a0(0xc73952e0) 0.015496941 s ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 ugen4.1: at usbus4 uhub4: on usbus4 ugen5.1: at usbus5 uhub5: on usbus5 ad4: 476940MB at ata2-master UDMA100 SATA 3Gb/s SMP: AP CPU #2 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #1 Launched! WARNING: DIAGNOSTIC option enabled, expect reduced performance. uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered Root mount waiting for: usbus5 Root mount waiting for: usbus5 Root mount waiting for: usbus5 Root mount waiting for: usbus5 uhub5: 10 ports with 10 removable, self powered Trying to mount root from ufs:/dev/ad4s2a Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Starting ddb. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/ad4s2a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s2a: clean, 527434 free (1082 frags, 65794 blocks, 0.1% fragmentation) /dev/ad4s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1f: clean, 651538 free (1290 frags, 81281 blocks, 0.1% fragmentation) /dev/ad4s1g: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1g: clean, 91353171 free (3059 frags, 11418764 blocks, 0.0% fragmentation) /dev/ad4s2d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s2d: clean, 8629786 free (41106 frags, 1073585 blocks, 0.4% fragmentation) /dev/ad4s2e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s2e: clean, 25385115 free (507 frags, 3173076 blocks, 0.0% fragmentation) Mounting local file systems:. Setting hostname: x4.osted.lan. re0: link state changed to DOWN Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=3 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 inet6 ::1 prefixlen 128 inet 127.0.0.1 netmask 0xff000000 nd6 options=3 re0: flags=8843 metric 0 mtu 1500 options=389b ether 00:21:85:15:33:7d inet 192.168.1.7 netmask 0xffffff00 broadcast 192.168.1.255 media: Ethernet autoselect (none) status: no carrier add net default: gateway 192.168.1.1 Starting devd. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Creating and/or trimming log files. Starting syslogd. No core dumps found. Additional ABI support: linux. Starting rpcbind. NFS access cache time=60 Clearing /tmp (X related). Starting mountd. Starting nfsd. Updating motd:. Starting ntpd. Configuring syscons: keymap blanktime. Starting sshd. Starting cron. Starting default moused. Starting inetd. Thu Mar 31 10:33:51 CEST 2011 FreeBSD/i386 (x4.osted.lan) (console) login: Mar 31 10:35:15 x4 su: pho to root on /dev/pts/0 SU test #1 panic: __lockmgr_args: recursing on non recursive lockmgr bufwait @ (null):0 cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper(c0ce5b2f,c8000a0d,104,e9cb954c,c08cf06b,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0d0fbd1,0,c0ce195b,e9cb95a0,0,...) at kdb_backtrace+0x2a panic(c0ce195b,c0c6c288,c0ceab5b,0,0,...) at panic+0x117 __lockmgr_args(db3a4080,80900,c8693d0c,0,0,...) at __lockmgr_args+0x65b getdirtybuf(0,104,8fd63316,1c0,0,...) at getdirtybuf+0xa7 softdep_sync_metadata(c8693c18,3,0,0,0,...) at softdep_sync_metadata+0x180 ffs_syncvnode(c8693c18,1,0,0,0,...) at ffs_syncvnode+0x3f6 softdep_request_cleanup(c82b7000,c8693c18,c826be80,0,0,...) at softdep_request_cleanup+0x524 ffs_alloc(c872d740,1f,0,3aaa8,0,...) at ffs_alloc+0x2ff ffs_balloc_ufs2(c8693c18,7c000,0,4000,c826be80,...) at ffs_balloc_ufs2+0x1af3 ffs_write(e9cb9b94,c0d13d6d,c8693c70,0,c0c28172,...) at ffs_write+0x3f5 VOP_WRITE_APV(c0de2680,e9cb9b94,c8693c18,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c7f9f578,e9cb9c28,c826be80,0,c83328a0,...) at vn_write+0x1da dofilewrite(e9cb9c28,ffffffff,ffffffff,0,c7f9f578,...) at dofilewrite+0x97 kern_writev(c83328a0,3,e9cb9c28,e9cb9c48,1,...) at kern_writev+0x58 write(c83328a0,e9cb9cec,e9cb9d28,c08e6fcb,c0e35af4,...) at write+0x4f syscallenter(c83328a0,e9cb9ce4,e9cb9ce4,0,c7f002e0,...) at syscallenter+0x329 syscall(e9cb9d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfea0c, ebp = 0xbfbfeab8 --- KDB: enter: panic [thread pid 3586 tid 100157 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> run pho db:0:pho> bt Tracing pid 3586 tid 100157 td 0xc83328a0 kdb_enter(c0ce3779,c0ce3779,c0ce195b,e9cb95a0,0,...) at kdb_enter+0x3a panic(c0ce195b,c0c6c288,c0ceab5b,0,0,...) at panic+0x134 __lockmgr_args(db3a4080,80900,c8693d0c,0,0,...) at __lockmgr_args+0x65b getdirtybuf(0,104,8fd63316,1c0,0,...) at getdirtybuf+0xa7 softdep_sync_metadata(c8693c18,3,0,0,0,...) at softdep_sync_metadata+0x180 ffs_syncvnode(c8693c18,1,0,0,0,...) at ffs_syncvnode+0x3f6 softdep_request_cleanup(c82b7000,c8693c18,c826be80,0,0,...) at softdep_request_cleanup+0x524 ffs_alloc(c872d740,1f,0,3aaa8,0,...) at ffs_alloc+0x2ff ffs_balloc_ufs2(c8693c18,7c000,0,4000,c826be80,...) at ffs_balloc_ufs2+0x1af3 ffs_write(e9cb9b94,c0d13d6d,c8693c70,0,c0c28172,...) at ffs_write+0x3f5 VOP_WRITE_APV(c0de2680,e9cb9b94,c8693c18,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c7f9f578,e9cb9c28,c826be80,0,c83328a0,...) at vn_write+0x1da dofilewrite(e9cb9c28,ffffffff,ffffffff,0,c7f9f578,...) at dofilewrite+0x97 kern_writev(c83328a0,3,e9cb9c28,e9cb9c48,1,...) at kern_writev+0x58 write(c83328a0,e9cb9cec,e9cb9d28,c08e6fcb,c0e35af4,...) at write+0x4f syscallenter(c83328a0,e9cb9ce4,e9cb9ce4,0,c7f002e0,...) at syscallenter+0x329 syscall(e9cb9d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfea0c, ebp = 0xbfbfeab8 --- db:0:bt> show allpcpu Current CPU: 0 cpuid = 0 dynamic pcpu = 0x629000 curthread = 0xc83328a0: pid 3586 "linger2" curpcb = 0xe9cb9d80 fpcurthread = none idlethread = 0xc7158000: tid 100006 "idle: cpu0" APIC ID = 0 currentldt = 0x50 cpuid = 1 dynamic pcpu = 0x60e8000 curthread = 0xc719d000: pid 12 "swi4: clock" curpcb = 0xc6f17d80 fpcurthread = none idlethread = 0xc71582e0: tid 100005 "idle: cpu1" APIC ID = 1 currentldt = 0x50 cpuid = 2 dynamic pcpu = 0x60eb000 curthread = 0xc71585c0: pid 11 "idle: cpu2" curpcb = 0xc6f07d80 fpcurthread = none idlethread = 0xc71585c0: tid 100004 "idle: cpu2" APIC ID = 2 currentldt = 0x50 cpuid = 3 dynamic pcpu = 0x60ee000 curthread = 0xc71588a0: pid 11 "idle: cpu3" curpcb = 0xc6f04d80 fpcurthread = none idlethread = 0xc71588a0: tid 100003 "idle: cpu3" APIC ID = 3 currentldt = 0x50 db:0:allpcpu> show alllocks No such command db:0:alllocks> show lockedvnods Locked vnodes 0xc7e15560: tag ufs, type VREG usecount 1, writecount 1, refcount 59 mountedhere 0 flags () v_object 0xc86953b8 ref 0 pages 228 lock type ufs: EXCL by thread 0xc7f2b5c0 (pid 3584) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc0b168da at ffs_lock+0x8a #2 0xc0c2a87d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc0966aa6 at vn_write+0x166 #5 0xc0906547 at dofilewrite+0x97 #6 0xc0906838 at kern_writev+0x58 #7 0xc09068bf at write+0x4f #8 0xc0902179 at syscallenter+0x329 #9 0xc0c0a15f at syscall+0x4f #10 0xc0bf06d1 at Xint0x80_syscall+0x21 ino 831, on dev md5a 0xc85eeac0: tag ufs, type VREG usecount 1, writecount 1, refcount 57 mountedhere 0 flags () v_object 0xc875f990 ref 0 pages 220 lock type ufs: EXCL by thread 0xc7b782e0 (pid 3587) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc0b168da at ffs_lock+0x8a #2 0xc0c2a87d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc0966aa6 at vn_write+0x166 #5 0xc0906547 at dofilewrite+0x97 #6 0xc0906838 at kern_writev+0x58 #7 0xc09068bf at write+0x4f #8 0xc0902179 at syscallenter+0x329 #9 0xc0c0a15f at syscall+0x4f #10 0xc0bf06d1 at Xint0x80_syscall+0x21 ino 899, on dev md5a 0xc86b46b8: tag ufs, type VREG usecount 1, writecount 1, refcount 59 mountedhere 0 flags () v_object 0xc8353330 ref 0 pages 228 lock type ufs: EXCL by thread 0xc7f2d5c0 (pid 3581) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc0b168da at ffs_lock+0x8a #2 0xc0c2a87d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc0966aa6 at vn_write+0x166 #5 0xc0906547 at dofilewrite+0x97 #6 0xc0906838 at kern_writev+0x58 #7 0xc09068bf at write+0x4f #8 0xc0902179 at syscallenter+0x329 #9 0xc0c0a15f at syscall+0x4f #10 0xc0bf06d1 at Xint0x80_syscall+0x21 ino 900, on dev md5a 0xc8693158: tag ufs, type VREG usecount 1, writecount 1, refcount 53 mountedhere 0 flags () v_object 0xc875eb28 ref 0 pages 204 lock type ufs: EXCL by thread 0xc87aa2e0 (pid 3582) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc0b168da at ffs_lock+0x8a #2 0xc0c2a87d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc0966aa6 at vn_write+0x166 #5 0xc0906547 at dofilewrite+0x97 #6 0xc0906838 at kern_writev+0x58 #7 0xc09068bf at write+0x4f #8 0xc0902179 at syscallenter+0x329 #9 0xc0c0a15f at syscall+0x4f #10 0xc0bf06d1 at Xint0x80_syscall+0x21 ino 901, on dev md5a 0xc8693810: tag ufs, type VREG usecount 1, writecount 1, refcount 48 mountedhere 0 flags () v_object 0xc8353d48 ref 0 pages 184 lock type ufs: EXCL by thread 0xc87aab80 (pid 3590) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc0b168da at ffs_lock+0x8a #2 0xc0c2a87d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc0966aa6 at vn_write+0x166 #5 0xc0906547 at dofilewrite+0x97 #6 0xc0906838 at kern_writev+0x58 #7 0xc09068bf at write+0x4f #8 0xc0902179 at syscallenter+0x329 #9 0xc0c0a15f at syscall+0x4f #10 0xc0bf06d1 at Xint0x80_syscall+0x21 ino 902, on dev md5a 0xc8693c18: tag ufs, type VREG usecount 1, writecount 1, refcount 34 mountedhere 0 flags () v_object 0xc8761330 ref 0 pages 128 lock type ufs: EXCL by thread 0xc83328a0 (pid 3586) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc0b168da at ffs_lock+0x8a #2 0xc0c2a87d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc0966aa6 at vn_write+0x166 #5 0xc0906547 at dofilewrite+0x97 #6 0xc0906838 at kern_writev+0x58 #7 0xc09068bf at write+0x4f #8 0xc0902179 at syscallenter+0x329 #9 0xc0c0a15f at syscall+0x4f #10 0xc0bf06d1 at Xint0x80_syscall+0x21 ino 903, on dev md5a 0xc86cc408: tag ufs, type VREG usecount 1, writecount 1, refcount 38 mountedhere 0 flags () v_object 0xc89c9b28 ref 0 pages 144 lock type ufs: EXCL by thread 0xc7f002e0 (pid 3583) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc0b168da at ffs_lock+0x8a #2 0xc0c2a87d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc0966aa6 at vn_write+0x166 #5 0xc0906547 at dofilewrite+0x97 #6 0xc0906838 at kern_writev+0x58 #7 0xc09068bf at write+0x4f #8 0xc0902179 at syscallenter+0x329 #9 0xc0c0a15f at syscall+0x4f #10 0xc0bf06d1 at Xint0x80_syscall+0x21 ino 904, on dev md5a 0xc854f2b0: tag ufs, type VREG usecount 1, writecount 1, refcount 31 mountedhere 0 flags () v_object 0xc8761990 ref 0 pages 116 lock type ufs: EXCL by thread 0xc832d5c0 (pid 3585) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc0b168da at ffs_lock+0x8a #2 0xc0c2a87d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc0966aa6 at vn_write+0x166 #5 0xc0906547 at dofilewrite+0x97 #6 0xc0906838 at kern_writev+0x58 #7 0xc09068bf at write+0x4f #8 0xc0902179 at syscallenter+0x329 #9 0xc0c0a15f at syscall+0x4f #10 0xc0bf06d1 at Xint0x80_syscall+0x21 ino 905, on dev md5a 0xc862b560: tag ufs, type VREG usecount 1, writecount 1, refcount 27 mountedhere 0 flags () v_object 0xc84cb770 ref 0 pages 100 lock type ufs: EXCL by thread 0xc8086000 (pid 3588) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc0b168da at ffs_lock+0x8a #2 0xc0c2a87d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc0966aa6 at vn_write+0x166 #5 0xc0906547 at dofilewrite+0x97 #6 0xc0906838 at kern_writev+0x58 #7 0xc09068bf at write+0x4f #8 0xc0902179 at syscallenter+0x329 #9 0xc0c0a15f at syscall+0x4f #10 0xc0bf06d1 at Xint0x80_syscall+0x21 ino 906, on dev md5a 0xc86b3408: tag ufs, type VREG usecount 1, writecount 1, refcount 3 mountedhere 0 flags () v_object 0xc8789220 ref 0 pages 4 lock type ufs: EXCL by thread 0xc8a2e2e0 (pid 3589) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc0b168da at ffs_lock+0x8a #2 0xc0c2a87d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc0966aa6 at vn_write+0x166 #5 0xc0906547 at dofilewrite+0x97 #6 0xc0906838 at kern_writev+0x58 #7 0xc09068bf at write+0x4f #8 0xc0902179 at syscallenter+0x329 #9 0xc0c0a15f at syscall+0x4f #10 0xc0bf06d1 at Xint0x80_syscall+0x21 ino 907, on dev md5a db:0:lockedvnods> show mount 0xc7b4db40 /dev/ad4s2a on / (ufs) 0xc7b4e000 devfs on /dev (devfs) 0xc7b4d870 /dev/ad4s1f on /home (ufs) 0xc7b4d5a0 /dev/ad4s1g on /tmp (ufs) 0xc7b4d2d0 /dev/ad4s2d on /usr (ufs) 0xc7b4d000 /dev/ad4s2e on /var (ufs) 0xc834e5a0 /dev/md5a on /mnt (ufs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 3590 1239 1238 1001 L+ *FFS 0xc808aa80 linger2 3589 1239 1238 1001 L+ *FFS 0xc808aa80 linger2 3588 1239 1238 1001 L+ *FFS 0xc808aa80 linger2 3587 1239 1238 1001 S+ getblk 0xdb184100 linger2 3586 1239 1238 1001 R+ CPU 0 linger2 3585 1239 1238 1001 L+ *FFS 0xc808aa80 linger2 3584 1239 1238 1001 L+ *FFS 0xc808aa80 linger2 3583 1239 1238 1001 S+ getblk 0xdb2eba84 linger2 3582 1239 1238 1001 R+ linger2 3581 1239 1238 1001 S+ getblk 0xdb2eba84 linger2 3580 1974 1969 1001 S nanslp 0xc0e30784 sleep 2049 2048 2049 1001 Ss+ ttyin 0xc80d3470 bash 2048 2036 2036 1001 S select 0xc87487e4 sshd 2036 1067 2036 0 Ss sbwait 0xc808ba60 sshd 1975 1969 1969 1001 S piperd 0xc7f24310 awk 1974 1969 1969 1001 S wait 0xc8781560 sh 1973 1971 1973 1001 Ss kqread 0xc8654c00 tail 1972 1970 1972 1001 Ss+ select 0xc84d3924 top 1971 1964 1964 1001 S select 0xc847dba4 sshd 1970 1963 1963 1001 S select 0xc83891a4 sshd 1969 1968 1969 1001 Ss wait 0xc897e560 sh 1968 1962 1962 1001 S select 0xc857b9e4 sshd 1964 1067 1964 0 Ss sbwait 0xc8298254 sshd 1963 1067 1963 0 Ss sbwait 0xc808b728 sshd 1962 1067 1962 0 Ss sbwait 0xc88ac728 sshd 1239 1238 1238 1001 S+ wait 0xc80832b0 linger2 1238 1237 1238 1001 S+ wait 0xc82c3810 bash 1237 1218 1212 0 S+ wait 0xc8267560 su 1232 0 0 0 SL mdwait 0xc7b5d800 [md5] 1218 1217 1212 0 S+ wait 0xc7f8e810 sh 1217 1212 1212 0 S+ wait 0xc82c4810 time 1212 1205 1212 0 S+ wait 0xc80852b0 sh 1205 1201 1205 0 S+ wait 0xc82c5ac0 bash 1201 1200 1201 0 S+ pause 0xc8084b18 csh 1200 1196 1200 1001 S+ wait 0xc7ab8810 su 1196 1195 1196 1001 Ss+ wait 0xc8083810 bash 1195 1193 1193 1001 S select 0xc7a95b24 sshd 1193 1067 1193 0 Ss sbwait 0xc8081a60 sshd 1190 1 1190 0 Ss+ ttyin 0xc74d8070 getty 1189 1 1189 0 Ss+ ttyin 0xc74d8270 getty 1188 1 1188 0 Ss+ ttyin 0xc73d7a70 getty 1187 1 1187 0 Ss+ ttyin 0xc7367670 getty 1186 1 1186 0 Ss+ ttyin 0xc7367870 getty 1185 1 1185 0 Ss+ ttyin 0xc7367c70 getty 1184 1 1184 0 Ss+ ttyin 0xc73d7670 getty 1183 1 1183 0 Ss+ ttyin 0xc73d7870 getty 1182 1 1182 0 Ss+ ttyin 0xc7367a70 getty 1143 1 1143 0 Ss select 0xc7ee72e4 inetd 1117 1 1117 0 Ss select 0xc7ad7be4 moused 1093 1 1093 0 Ss nanslp 0xc0e30784 cron 1086 1 1086 25 Ss pause 0xc80835b8 sendmail 1082 1 1082 0 Ss select 0xc808c064 sendmail 1067 1 1067 0 Ss select 0xc808c564 sshd 1022 1 1022 0 Rs ntpd 898 897 897 0 S (threaded) nfsd 100116 S rpcsvc 0xc7ee7390 nfsd: service 100115 S rpcsvc 0xc7ee73d0 nfsd: service 100114 S rpcsvc 0xc7ee7410 nfsd: service 100102 S rpcsvc 0xc7b22190 nfsd: master 897 1 897 0 Ss select 0xc7b23364 nfsd 888 1 888 0 Ss select 0xc7b222e4 mountd 791 1 791 0 Ss select 0xc7b23564 rpcbind 765 1 765 0 Ss select 0xc7b226e4 syslogd 617 1 617 0 Ss select 0xc7b231e4 devd 21 0 0 0 SL c_flowcl 0xc0e444f0 [flowcleaner] 20 0 0 0 SL getblk 0xdb184100 [softdepflush] 19 0 0 0 SL syncer 0xc0e442f8 [syncer] 18 0 0 0 SL vlruwt 0xc7ab72b0 [vnlru] 17 0 0 0 SL psleep 0xc0e44028 [bufdaemon] 16 0 0 0 SL pgzero 0xc0e50874 [pagezero] 15 0 0 0 SL psleep 0xc0e5049c [vmdaemon] 9 0 0 0 SL psleep 0xc0e50464 [pagedaemon] 8 0 0 0 SL ccb_scan 0xc0dfccd4 [xpt_thrd] 7 0 0 0 SL waiting_ 0xc0e45d38 [sctp_iterator] 6 0 0 0 SL - 0xc73d743c [fdc0] 5 0 0 0 SL - 0xc73de000 [fw0_probe] 14 0 0 0 SL (threaded) usb 100058 D - 0xc73bfd34 [usbus5] 100057 D - 0xc73bfd04 [usbus5] 100056 D - 0xc73bfcd4 [usbus5] 100055 D - 0xc73bfca4 [usbus5] 100053 D - 0xc73b2b5c [usbus4] 100052 D - 0xc73b2b2c [usbus4] 100051 D - 0xc73b2afc [usbus4] 100050 D - 0xc73b2acc [usbus4] 100049 D - 0xc73aeb5c [usbus3] 100048 D - 0xc73aeb2c [usbus3] 100047 D - 0xc73aeafc [usbus3] 100046 D - 0xc73aeacc [usbus3] 100045 D - 0xc73a1b5c [usbus2] 100044 D - 0xc73a1b2c [usbus2] 100043 D - 0xc73a1afc [usbus2] 100042 D - 0xc73a1acc [usbus2] 100040 D - 0xc739eb5c [usbus1] 100039 D - 0xc739eb2c [usbus1] 100038 D - 0xc739eafc [usbus1] 100037 D - 0xc739eacc [usbus1] 100035 D - 0xc738db5c [usbus0] 100034 D - 0xc738db2c [usbus0] 100033 D - 0xc738dafc [usbus0] 100032 D - 0xc738dacc [usbus0] 13 0 0 0 SL - 0xc0e305e4 [yarrow] 4 0 0 0 SL - 0xc0e2e2e4 [g_down] 3 0 0 0 SL - 0xc0e2e2e0 [g_up] 2 0 0 0 SL - 0xc0e2e2d8 [g_event] 12 0 0 0 RL (threaded) intr 100066 I [irq12: psm0] 100065 I [irq1: atkbd0] 100063 I [swi0: uart] 100060 I [irq20: fwohci0] 100059 I [irq14: ata0] 100054 I [irq19: ehci0] 100041 I [irq18: ohci2 ohci4] 100036 I [irq17: ohci1 ohci3] 100031 I [irq16: ohci0] 100030 I [irq22: atapci0] 100029 I [irq256: re0] 100028 I [irq9: acpi0] 100026 I [swi5: +] 100021 I [swi2: cambio] 100019 I [swi6: task queue] 100018 I [swi6: Giant taskq] 100012 I [swi4: clock] 100011 I [swi4: clock] 100010 I [swi4: clock] 100009 Run CPU 1 [swi4: clock] 100008 I [swi1: netisr 0] 100007 I [swi3: vm] 11 0 0 0 RL (threaded) idle 100006 CanRun [idle: cpu0] 100005 CanRun [idle: cpu1] 100004 Run CPU 2 [idle: cpu2] 100003 Run CPU 3 [idle: cpu3] 1 0 1 0 SLs wait 0xc7156ac0 [init] 10 0 0 0 SL audit_wo 0xc0e4f4c0 [audit] 0 0 0 0 SLs (threaded) kernel 100061 D - 0xc73dd580 [fw0_taskq] 100027 D - 0xc732c9c0 [thread taskq] 100025 D - 0xc732ca80 [ffs_trim taskq] 100024 D - 0xc732cc00 [acpi_task_2] 100023 D - 0xc732cc00 [acpi_task_1] 100022 D - 0xc732cc00 [acpi_task_0] 100020 D - 0xc732cd40 [kqueue taskq] 100016 D - 0xc7137880 [firmware taskq] 100000 D sched 0xc0e2e3c0 [swapper] db:0:ps> allt Tracing command linger2 pid 3590 tid 100190 td 0xc87aab80 sched_switch(c87aab80,0,103,8fd4baa3,1c0,...) at sched_switch+0x2b6 mi_switch(103,0,c0ce6e74,2d6,0,...) at mi_switch+0x11b turnstile_wait(c808aa80,c83328a0,0,20,5,...) at turnstile_wait+0x2dc _mtx_lock_sleep(c7f97148,c87aab80,0,0,0,...) at _mtx_lock_sleep+0x10c ffs_reallocblks(e9d2d9f4,c0d13cf6,c8693810,c839d5a4,e9d2da28,...) at ffs_reallocblks+0xb8e VOP_REALLOCBLKS_APV(c0de2680,e9d2d9f4,0,4000,0,...) at VOP_REALLOCBLKS_APV+0x6b cluster_write(c8693810,db23bcf4,b4000,0,41,...) at cluster_write+0x3a2 ffs_write(e9d2db94,c0d13d6d,c8693868,0,c0c28172,...) at ffs_write+0x637 VOP_WRITE_APV(c0de2680,e9d2db94,c8693810,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c7f873f0,e9d2dc28,c826be80,0,c87aab80,...) at vn_write+0x1da dofilewrite(e9d2dc28,ffffffff,ffffffff,0,c7f873f0,...) at dofilewrite+0x97 kern_writev(c87aab80,3,e9d2dc28,e9d2dc48,1,...) at kern_writev+0x58 write(c87aab80,e9d2dcec,e9d2dd28,c08e6fdc,1,...) at write+0x4f syscallenter(c87aab80,e9d2dce4,e9d2dce4,0,c73bab80,...) at syscallenter+0x329 syscall(e9d2dd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfea0c, ebp = 0xbfbfeab8 --- Tracing command linger2 pid 3589 tid 100223 td 0xc8a2e2e0 sched_switch(c8a2e2e0,0,103,8fd4af83,1c0,...) at sched_switch+0x2b6 mi_switch(103,0,c0ce6e74,2d6,c83328a0,...) at mi_switch+0x11b turnstile_wait(c808aa80,c83328a0,0,c7f97100,4000,...) at turnstile_wait+0x2dc _mtx_lock_sleep(c7f97148,c8a2e2e0,0,0,0,...) at _mtx_lock_sleep+0x10c ffs_balloc_ufs2(c86b3408,4000,0,4000,c826be80,...) at ffs_balloc_ufs2+0x1185 ffs_write(e9dc4b94,c0d13d6d,c86b3460,0,c0c28172,...) at ffs_write+0x3f5 VOP_WRITE_APV(c0de2680,e9dc4b94,c86b3408,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c88367a8,e9dc4c28,c826be80,0,c8a2e2e0,...) at vn_write+0x1da dofilewrite(e9dc4c28,ffffffff,ffffffff,0,c88367a8,...) at dofilewrite+0x97 kern_writev(c8a2e2e0,3,e9dc4c28,e9dc4c48,1,...) at kern_writev+0x58 write(c8a2e2e0,e9dc4cec,e9dc4d28,c08e6fdc,c0e3526c,...) at write+0x4f syscallenter(c8a2e2e0,e9dc4ce4,e9dc4ce4,0,c725fb80,...) at syscallenter+0x329 syscall(e9dc4d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfea0c, ebp = 0xbfbfeab8 --- Tracing command linger2 pid 3588 tid 100169 td 0xc8086000 sched_switch(c8086000,0,103,8fd3f473,1c0,...) at sched_switch+0x2b6 mi_switch(103,0,c0ce6e74,2d6,e9cdd648,...) at mi_switch+0x11b turnstile_wait(c808aa80,c83328a0,0,0,c5a8,...) at turnstile_wait+0x2dc _mtx_lock_sleep(c7f97148,c8086000,0,0,0,...) at _mtx_lock_sleep+0x10c ffs_blkfree_cg(3a4b8,0,4000,38a,c866c040,...) at ffs_blkfree_cg+0x33f ffs_blkfree(c7f97100,c82b7000,c834c000,3a4b8,0,...) at ffs_blkfree+0x43 handle_workitem_freefrag(0,e9cdd770,2,0,0,...) at handle_workitem_freefrag+0x51 process_worklist_item(c0e4fb44,c8086000,0,0,0,...) at process_worklist_item+0x241 softdep_request_cleanup(c82b7000,c862b560,c826be80,0,0,...) at softdep_request_cleanup+0x396 ffs_alloc(c83da000,18,0,3cd98,0,...) at ffs_alloc+0x2ff ffs_balloc_ufs2(c862b560,60000,0,4000,c826be80,...) at ffs_balloc_ufs2+0x1af3 ffs_write(e9cddb94,c0d13d6d,c862b5b8,0,c0c28172,...) at ffs_write+0x3f5 VOP_WRITE_APV(c0de2680,e9cddb94,c862b560,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c7b7cb28,e9cddc28,c826be80,0,c8086000,...) at vn_write+0x1da dofilewrite(e9cddc28,ffffffff,ffffffff,0,c7b7cb28,...) at dofilewrite+0x97 kern_writev(c8086000,3,e9cddc28,e9cddc48,1,...) at kern_writev+0x58 write(c8086000,e9cddcec,e9cddd28,c08e6fdc,c0e3526c,...) at write+0x4f syscallenter(c8086000,e9cddce4,e9cddce4,0,c725fb80,...) at syscallenter+0x329 syscall(e9cddd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfea0c, ebp = 0xbfbfeab8 --- Tracing command linger2 pid 3587 tid 100085 td 0xc7b782e0 sched_switch(c7b782e0,0,104,8fd4c71e,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c7b782e0,db184100,e9b525d0,...) at mi_switch+0x11b sleepq_switch(c7b782e0,0,c0ce693d,260,50,...) at sleepq_switch+0xcc sleepq_wait(db184100,50,c0ceab21,4,0,...) at sleepq_wait+0x39 __lockmgr_args(db184100,81900,c834c0f4,c0ceab21,50,...) at __lockmgr_args+0x893 getblk(c834c000,16f940,0,4000,0,...) at getblk+0x1b1 breadn(c834c000,16f940,0,4000,0,...) at breadn+0x44 bread(c834c000,16f940,0,4000,0,...) at bread+0x4c ffs_alloccg(c8a324fc,4,639a0,0,4000,...) at ffs_alloccg+0x179 ffs_hashalloc(639a0,0,4000,c0af2de0,0,...) at ffs_hashalloc+0x3a ffs_alloc(c8a324fc,36,0,639a0,0,...) at ffs_alloc+0x1bb ffs_balloc_ufs2(c85eeac0,d8000,0,4000,c826be80,...) at ffs_balloc_ufs2+0x1af3 ffs_write(e9b52b94,c0d13d6d,c85eeb18,0,c0c28172,...) at ffs_write+0x3f5 VOP_WRITE_APV(c0de2680,e9b52b94,c85eeac0,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c7f87818,e9b52c28,c826be80,0,c7b782e0,...) at vn_write+0x1da dofilewrite(e9b52c28,ffffffff,ffffffff,0,c7f87818,...) at dofilewrite+0x97 kern_writev(c7b782e0,3,e9b52c28,e9b52c48,1,...) at kern_writev+0x58 write(c7b782e0,e9b52cec,e9b52d28,c08e6fcb,c0e35af4,...) at write+0x4f syscallenter(c7b782e0,e9b52ce4,e9b52ce4,0,c8a2e2e0,...) at syscallenter+0x329 syscall(e9b52d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfea0c, ebp = 0xbfbfeab8 --- Tracing command linger2 pid 3586 tid 100157 td 0xc83328a0 kdb_enter(c0ce3779,c0ce3779,c0ce195b,e9cb95a0,0,...) at kdb_enter+0x3a panic(c0ce195b,c0c6c288,c0ceab5b,0,0,...) at panic+0x134 __lockmgr_args(db3a4080,80900,c8693d0c,0,0,...) at __lockmgr_args+0x65b getdirtybuf(0,104,8fd63316,1c0,0,...) at getdirtybuf+0xa7 softdep_sync_metadata(c8693c18,3,0,0,0,...) at softdep_sync_metadata+0x180 ffs_syncvnode(c8693c18,1,0,0,0,...) at ffs_syncvnode+0x3f6 softdep_request_cleanup(c82b7000,c8693c18,c826be80,0,0,...) at softdep_request_cleanup+0x524 ffs_alloc(c872d740,1f,0,3aaa8,0,...) at ffs_alloc+0x2ff ffs_balloc_ufs2(c8693c18,7c000,0,4000,c826be80,...) at ffs_balloc_ufs2+0x1af3 ffs_write(e9cb9b94,c0d13d6d,c8693c70,0,c0c28172,...) at ffs_write+0x3f5 VOP_WRITE_APV(c0de2680,e9cb9b94,c8693c18,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c7f9f578,e9cb9c28,c826be80,0,c83328a0,...) at vn_write+0x1da dofilewrite(e9cb9c28,ffffffff,ffffffff,0,c7f9f578,...) at dofilewrite+0x97 kern_writev(c83328a0,3,e9cb9c28,e9cb9c48,1,...) at kern_writev+0x58 write(c83328a0,e9cb9cec,e9cb9d28,c08e6fcb,c0e35af4,...) at write+0x4f syscallenter(c83328a0,e9cb9ce4,e9cb9ce4,0,c7f002e0,...) at syscallenter+0x329 syscall(e9cb9d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfea0c, ebp = 0xbfbfeab8 --- Tracing command linger2 pid 3585 tid 100163 td 0xc832d5c0 sched_switch(c832d5c0,0,103,8fd61f93,1c0,...) at sched_switch+0x2b6 mi_switch(103,0,c0ce6e74,2d6,c83328a0,...) at mi_switch+0x11b turnstile_wait(c808aa80,c83328a0,0,0,c7f97100,...) at turnstile_wait+0x2dc _mtx_lock_sleep(c7f97148,c832d5c0,0,0,0,...) at _mtx_lock_sleep+0x10c softdep_request_cleanup(c82b7000,c854f2b0,c826be80,0,0,...) at softdep_request_cleanup+0xb3 ffs_alloc(c86c36cc,1c,0,3cc80,0,...) at ffs_alloc+0x2ff ffs_balloc_ufs2(c854f2b0,70000,0,4000,c826be80,...) at ffs_balloc_ufs2+0x1af3 ffs_write(e9ccbb94,c0d13d6d,c854f308,0,c0c28172,...) at ffs_write+0x3f5 VOP_WRITE_APV(c0de2680,e9ccbb94,c854f2b0,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c7bcf7a8,e9ccbc28,c826be80,0,c832d5c0,...) at vn_write+0x1da dofilewrite(e9ccbc28,ffffffff,ffffffff,0,c7bcf7a8,...) at dofilewrite+0x97 kern_writev(c832d5c0,3,e9ccbc28,e9ccbc48,1,...) at kern_writev+0x58 write(c832d5c0,e9ccbcec,e9ccbd28,c08e6fdc,c0e3456c,...) at write+0x4f syscallenter(c832d5c0,e9ccbce4,e9ccbce4,0,c73bab80,...) at syscallenter+0x329 syscall(e9ccbd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfea0c, ebp = 0xbfbfeab8 --- Tracing command linger2 pid 3584 tid 100110 td 0xc7f2b5c0 sched_switch(c7f2b5c0,0,103,8fd4d8b8,1c0,...) at sched_switch+0x2b6 mi_switch(103,0,c0ce6e74,2d6,0,...) at mi_switch+0x11b turnstile_wait(c808aa80,c83328a0,0,20,c7f97100,...) at turnstile_wait+0x2dc _mtx_lock_sleep(c7f97148,c7f2b5c0,0,0,0,...) at _mtx_lock_sleep+0x10c ffs_clusteralloc(c8a32d98,4,63cf8,0,4,...) at ffs_clusteralloc+0x4f8 ffs_hashalloc(63cf8,0,4,c0af27b0,dbca1000,...) at ffs_hashalloc+0x3a ffs_reallocblks(e9c2c9f4,c0d13cf6,c7e15560,c87f6d60,e9c2ca28,...) at ffs_reallocblks+0xc13 VOP_REALLOCBLKS_APV(c0de2680,e9c2c9f4,0,4000,0,...) at VOP_REALLOCBLKS_APV+0x6b cluster_write(c7e15560,db361854,e0000,0,41,...) at cluster_write+0x3a2 ffs_write(e9c2cb94,c0d13d6d,c7e155b8,0,c0c28172,...) at ffs_write+0x637 VOP_WRITE_APV(c0de2680,e9c2cb94,c7e15560,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c7bcfc08,e9c2cc28,c826be80,0,c7f2b5c0,...) at vn_write+0x1da dofilewrite(e9c2cc28,ffffffff,ffffffff,0,c7bcfc08,...) at dofilewrite+0x97 kern_writev(c7f2b5c0,3,e9c2cc28,e9c2cc48,1,...) at kern_writev+0x58 write(c7f2b5c0,e9c2ccec,e9c2cd28,c08e6fdc,c0e3456c,...) at write+0x4f syscallenter(c7f2b5c0,e9c2cce4,e9c2cce4,0,c73bab80,...) at syscallenter+0x329 syscall(e9c2cd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfea0c, ebp = 0xbfbfeab8 --- Tracing command linger2 pid 3583 tid 100089 td 0xc7f002e0 sched_switch(c7f002e0,0,104,8fd49694,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c7f002e0,db2eba84,e9bb6500,...) at mi_switch+0x11b sleepq_switch(c7f002e0,0,c0ce693d,260,50,...) at sleepq_switch+0xcc sleepq_wait(db2eba84,50,c0ceab21,4,0,...) at sleepq_wait+0x39 __lockmgr_args(db2eba84,81900,c834c0f4,c0ceab21,50,...) at __lockmgr_args+0x893 getblk(c834c000,b7d00,0,4000,0,...) at getblk+0x1b1 breadn(c834c000,b7d00,0,4000,0,...) at breadn+0x44 bread(c834c000,b7d00,0,4000,0,...) at bread+0x4c ffs_blkfree_cg(3a5b8,0,4000,38a,c8371ae0,...) at ffs_blkfree_cg+0x272 ffs_blkfree(c7f97100,c82b7000,c834c000,3a5b8,0,...) at ffs_blkfree+0x43 handle_workitem_freefrag(0,e9bb6770,2,c0e4fb44,e9bb6784,...) at handle_workitem_freefrag+0x51 process_worklist_item(c0e4fb44,c7f002e0,0,0,0,...) at process_worklist_item+0x241 softdep_request_cleanup(c82b7000,c86cc408,c826be80,0,0,...) at softdep_request_cleanup+0x396 ffs_alloc(c875b074,23,0,3a298,0,...) at ffs_alloc+0x2ff ffs_balloc_ufs2(c86cc408,8c000,0,4000,c826be80,...) at ffs_balloc_ufs2+0x1af3 ffs_write(e9bb6b94,c0d13d6d,c86cc460,0,c0c28172,...) at ffs_write+0x3f5 VOP_WRITE_APV(c0de2680,e9bb6b94,c86cc408,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c7f87a48,e9bb6c28,c826be80,0,c7f002e0,...) at vn_write+0x1da dofilewrite(e9bb6c28,ffffffff,ffffffff,0,c7f87a48,...) at dofilewrite+0x97 kern_writev(c7f002e0,3,e9bb6c28,e9bb6c48,1,...) at kern_writev+0x58 write(c7f002e0,e9bb6cec,e9bb6d28,c08e6fdc,c0e3456c,...) at write+0x4f syscallenter(c7f002e0,e9bb6ce4,e9bb6ce4,0,c73bab80,...) at syscallenter+0x329 syscall(e9bb6d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfea0c, ebp = 0xbfbfeab8 --- Tracing command linger2 pid 3582 tid 100193 td 0xc87aa2e0 sched_switch(c87aa2e0,0,104,8fd0db58,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c87aa2e0,0,e9d36714,...) at mi_switch+0x11b sleepq_switch(c87aa2e0,0,c0ce693d,260,68,...) at sleepq_switch+0xcc sleepq_wait(c0e4fb5c,68,c0d00bf7,0,0,...) at sleepq_wait+0x39 _sleep(c0e4fb5c,c0e4fb44,68,c0d00bf7,0,...) at _sleep+0x2a2 request_cleanup(c8693158,1,db2963c8,db296428,c869324c,...) at request_cleanup+0x174 softdep_request_cleanup(c82b7000,c8693158,c826be80,0,0,...) at softdep_request_cleanup+0x14c ffs_alloc(c86c22b8,32,0,63668,0,...) at ffs_alloc+0x2ff ffs_balloc_ufs2(c8693158,c8000,0,4000,c826be80,...) at ffs_balloc_ufs2+0x1af3 ffs_write(e9d36b94,c0d13d6d,c86931b0,0,c0c28172,...) at ffs_write+0x3f5 VOP_WRITE_APV(c0de2680,e9d36b94,c8693158,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c7b7d9a0,e9d36c28,c826be80,0,c87aa2e0,...) at vn_write+0x1da dofilewrite(e9d36c28,ffffffff,ffffffff,0,c7b7d9a0,...) at dofilewrite+0x97 kern_writev(c87aa2e0,3,e9d36c28,e9d36c48,1,...) at kern_writev+0x58 write(c87aa2e0,e9d36cec,e9d36c80,c08e6fcb,c0e35474,...) at write+0x4f syscallenter(c87aa2e0,e9d36ce4,c08e9056,c87aa2e0,c87aab80,...) at syscallenter+0x329 syscall(e9d36d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfea0c, ebp = 0xbfbfeab8 --- Tracing command linger2 pid 3581 tid 100120 td 0xc7f2d5c0 sched_switch(c7f2d5c0,0,104,8fd4e655,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c7f2d5c0,db2eba84,e9c4a500,...) at mi_switch+0x11b sleepq_switch(c7f2d5c0,0,c0ce693d,260,50,...) at sleepq_switch+0xcc sleepq_wait(db2eba84,50,c0ceab21,4,0,...) at sleepq_wait+0x39 __lockmgr_args(db2eba84,81900,c834c0f4,c0ceab21,50,...) at __lockmgr_args+0x893 getblk(c834c000,b7d00,0,4000,0,...) at getblk+0x1b1 breadn(c834c000,b7d00,0,4000,0,...) at breadn+0x44 bread(c834c000,b7d00,0,4000,0,...) at bread+0x4c ffs_blkfree_cg(3a5a8,0,4000,38a,c87a1b00,...) at ffs_blkfree_cg+0x272 ffs_blkfree(c7f97100,c82b7000,c834c000,3a5a8,0,...) at ffs_blkfree+0x43 handle_workitem_freefrag(0,e9c4a770,2,0,0,...) at handle_workitem_freefrag+0x51 process_worklist_item(c0e4fb44,c7f2d5c0,0,0,0,...) at process_worklist_item+0x241 softdep_request_cleanup(c82b7000,c86b46b8,c826be80,0,0,...) at softdep_request_cleanup+0x396 ffs_alloc(c83daae0,38,0,61f08,0,...) at ffs_alloc+0x2ff ffs_balloc_ufs2(c86b46b8,e0000,0,4000,c826be80,...) at ffs_balloc_ufs2+0x1af3 ffs_write(e9c4ab94,c0d13d6d,c86b4710,0,c0c28172,...) at ffs_write+0x3f5 VOP_WRITE_APV(c0de2680,e9c4ab94,c86b46b8,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c7f9fc40,e9c4ac28,c826be80,0,c7f2d5c0,...) at vn_write+0x1da dofilewrite(e9c4ac28,ffffffff,ffffffff,0,c7f9fc40,...) at dofilewrite+0x97 kern_writev(c7f2d5c0,3,e9c4ac28,e9c4ac48,1,...) at kern_writev+0x58 write(c7f2d5c0,e9c4acec,e9c4ad28,c08e6fdc,c0e358ec,...) at write+0x4f syscallenter(c7f2d5c0,e9c4ace4,e9c4ace4,0,c725fb80,...) at syscallenter+0x329 syscall(e9c4ad28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfea0c, ebp = 0xbfbfeab8 --- Tracing command sleep pid 3580 tid 100204 td 0xc89905c0 sched_switch(c89905c0,0,104,cecbe092,1be,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c897cac0,e9d71b88,...) at mi_switch+0x11b sleepq_switch(c89905c0,0,c0ce693d,1a0,c89907c0,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,100,e9d71be0,c08cf478,...) at sleepq_catch_signals+0x52 sleepq_timedwait_sig(c0e30784,5c,c0ce4170,100,0,...) at sleepq_timedwait_sig+0x1c _sleep(c0e30784,0,15c,c0ce4170,2711,...) at _sleep+0x258 kern_nanosleep(c89905c0,e9d71c34,e9d71c3c,a,0,...) at kern_nanosleep+0xc1 nanosleep(c89905c0,e9d71cec,e9d71d28,28106210,c897cac0,...) at nanosleep+0x6f syscallenter(c89905c0,e9d71ce4,e9d71ce4,0,e9d71d28,...) at syscallenter+0x329 syscall(e9d71d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x2815ef57, esp = 0xbfbfecac, ebp = 0xbfbfecf8 --- Tracing command bash pid 2049 tid 100129 td 0xc7f00000 sched_switch(c7f00000,0,104,ea1345b2,107,...) at sched_switch+0x2b6 mi_switch(104,0,0,c7efb000,e9c65aa4,...) at mi_switch+0x11b sleepq_switch(c7f00000,0,c0ce693d,1a0,c7f00000,...) at sleepq_switch+0xcc sleepq_catch_signals(0,0,e9c65aec,c087f20b,c80d3470,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c80d3470,0,c0ce8e7a,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c80d3470,c80d3404,0,c80d3400,e9c65b58,...) at _cv_wait_sig+0x18b tty_wait(c80d3400,c80d3470,e9c65c28,1,0,...) at tty_wait+0x23 ttydisc_read(c80d3400,e9c65c28,0,e9c65ba8,e9c65ba4,...) at ttydisc_read+0x1cc ttydev_read(c871f900,e9c65c28,0,0,1,...) at ttydev_read+0x10a devfs_read_f(c7f9f6c8,e9c65c28,c85d9200,0,c7f00000,...) at devfs_read_f+0x7e dofileread(e9c65c28,ffffffff,ffffffff,0,c7f9f6c8,...) at dofileread+0x96 kern_readv(c7f00000,0,e9c65c28,e9c65c48,1,...) at kern_readv+0x58 read(c7f00000,e9c65cec,e9c65d28,28408400,c7efb000,...) at read+0x4f syscallenter(c7f00000,e9c65ce4,e9c65ce4,0,e9c65d28,...) at syscallenter+0x329 syscall(e9c65d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x2834f5d3, esp = 0xbfbfe01c, ebp = 0xbfbfe048 --- Tracing command sshd pid 2048 tid 100209 td 0xc898e8a0 sched_switch(c898e8a0,0,104,ea14f346,107,...) at sched_switch+0x2b6 mi_switch(104,0,c87487c0,c8782ac0,e9d80a18,...) at mi_switch+0x11b sleepq_switch(c898e8a0,0,c0ce693d,1a0,c898e8a0,...) at sleepq_switch+0xcc sleepq_catch_signals(c87487c0,0,e9d80a60,c087f20b,c87487e4,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c87487e4,0,c0ce81e5,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c87487e4,c87487d0,0,40,c88352a0,...) at _cv_wait_sig+0x18b seltdwait(c88352a0,58,c865fd00,c898e8a0,c89a0001,...) at seltdwait+0xd7 kern_select(c898e8a0,a,286030b4,286030d8,0,0,20,c8782ac0,c8782ac0) at kern_select+0x4ee select(c898e8a0,e9d80cec,e9d80d28,4175e6,e9d80c84,...) at select+0x66 syscallenter(c898e8a0,e9d80ce4,e9d80ce4,0,369e99,...) at syscallenter+0x329 syscall(e9d80d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x284a9553, esp = 0xbfbfe1cc, ebp = 0xbfbfe218 --- Tracing command sshd pid 2036 tid 100180 td 0xc87852e0 sched_switch(c87852e0,0,104,4d374a87,100,...) at sched_switch+0x2b6 mi_switch(104,0,158,c8780ac0,e9d0fa8c,...) at mi_switch+0x11b sleepq_switch(c87852e0,0,c0ce693d,1a0,c87852e0,...) at sleepq_switch+0xcc sleepq_catch_signals(158,0,e9d0fae0,c08cf4ac,c808ba60,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c808ba60,58,c0ce9e4f,100,0,...) at sleepq_wait_sig+0x18 _sleep(c808ba60,c808ba20,158,c0ce9e4f,0,...) at _sleep+0x28c sbwait(c808b9fc,1,e9d0fb38,c0945fa5,c74ec5b8,...) at sbwait+0x52 soreceive_generic(c808b9a8,0,e9d0fc28,0,0,...) at soreceive_generic+0x3b1 soreceive(c808b9a8,0,e9d0fc28,0,0,0) at soreceive+0x38 soo_read(c7f9f690,e9d0fc28,c8881b00,0,c87852e0,...) at soo_read+0x4e dofileread(e9d0fc28,ffffffff,ffffffff,0,c7f9f690,...) at dofileread+0x96 kern_readv(c87852e0,5,e9d0fc28,e9d0fc48,1,...) at kern_readv+0x58 read(c87852e0,e9d0fcec,e9d0fd28,c08e6fdc,c0e3456c,...) at read+0x4f syscallenter(c87852e0,e9d0fce4,e9d0fce4,0,c898e8a0,...) at syscallenter+0x329 syscall(e9d0fd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x284a95d3, esp = 0xbfbfdd9c, ebp = 0xbfbfddd8 --- Tracing command awk pid 1975 tid 100179 td 0xc87855c0 sched_switch(c87855c0,0,104,ceaece66,1be,...) at sched_switch+0x2b6 mi_switch(104,0,14c,c8781000,e9d0cb30,...) at mi_switch+0x11b sleepq_switch(c87855c0,0,c0ce693d,1a0,c87855c0,...) at sleepq_switch+0xcc sleepq_catch_signals(14c,0,e9d0cb84,c08cf4ac,c7f24310,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7f24310,4c,c0ce6fd4,100,0,...) at sleepq_wait_sig+0x18 _sleep(c7f24310,c7f24480,14c,c0ce6fd4,0,...) at _sleep+0x28c pipe_read(c8836230,e9d0cc28,c85b6780,0,c87855c0,...) at pipe_read+0x4ca dofileread(e9d0cc28,ffffffff,ffffffff,0,c8836230,...) at dofileread+0x96 kern_readv(c87855c0,0,e9d0cc28,e9d0cc48,1,...) at kern_readv+0x58 read(c87855c0,e9d0ccec,e9d0cd80,e9d0cc88,c0c0da10,...) at read+0x4f syscallenter(c87855c0,e9d0cce4,c0c0a5db,0,369e99,...) at syscallenter+0x329 syscall(e9d0cd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x281b35d3, esp = 0xbfbfe96c, ebp = 0xbfbfe988 --- Tracing command sh pid 1974 tid 100177 td 0xc8785b80 sched_switch(c8785b80,0,104,ceb8ed40,1be,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c8781560,e9d06b50,...) at mi_switch+0x11b sleepq_switch(c8785b80,0,c0ce693d,1a0,c8785b80,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,e9d06ba4,c08cf4ac,c8781560,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c8781560,5c,c0ce72e1,100,0,...) at sleepq_wait_sig+0x18 _sleep(c8781560,c87815e8,15c,c0ce72e1,0,...) at _sleep+0x28c kern_wait(c8785b80,ffffffff,e9d06c44,2,0,...) at kern_wait+0xfa1 wait4(c8785b80,e9d06cec,e9d06d28,8066634,c8781560,...) at wait4+0x3b syscallenter(c8785b80,e9d06ce4,e9d06ce4,0,e9d06d28,...) at syscallenter+0x329 syscall(e9d06d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2815cd5b, esp = 0xbfbfe95c, ebp = 0xbfbfe978 --- Tracing command tail pid 1973 tid 100079 td 0xc7b7b5c0 sched_switch(c7b7b5c0,0,104,7f6a1f3d,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,158,c7b77560,e9b38a68,...) at mi_switch+0x11b sleepq_switch(c7b7b5c0,0,c0ce693d,1a0,c7b7b7c0,...) at sleepq_switch+0xcc sleepq_catch_signals(158,0,100,e9b38ac0,c08cf478,...) at sleepq_catch_signals+0x52 sleepq_timedwait_sig(c8654c00,58,c0cdff82,100,0,...) at sleepq_timedwait_sig+0x1c _sleep(c8654c00,c8654c00,158,c0cdff82,3e9,...) at _sleep+0x258 kern_kevent(c7b7b5c0,4,0,1,e9b38c28,...) at kern_kevent+0x384 kevent(c7b7b5c0,e9b38cec,e9b38d28,40db57,e9b38c84,...) at kevent+0x19b syscallenter(c7b7b5c0,e9b38ce4,e9b38ce4,0,369e99,...) at syscallenter+0x329 syscall(e9b38d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (363, FreeBSD ELF32, kevent), eip = 0x2816304b, esp = 0xbfbfeb4c, ebp = 0xbfbfec08 --- Tracing command top pid 1972 tid 100198 td 0xc87a92e0 sched_switch(c87a92e0,0,104,50a7f91f,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c84d3900,c897e000,e9d5fa14,...) at mi_switch+0x11b sleepq_switch(c87a92e0,0,c0ce693d,1a0,c87a94e0,...) at sleepq_switch+0xcc sleepq_catch_signals(c84d3900,0,c87a92e0,e9d5fa60,c087ed63,...) at sleepq_catch_signals+0x52 sleepq_timedwait_sig(c84d3924,0,c0ce81e5,101,0,...) at sleepq_timedwait_sig+0x1c _cv_timedwait_sig(c84d3924,c84d3910,3e9,c870cd00,c7b7d038,...) at _cv_timedwait_sig+0x1a3 seltdwait(e9d5fbf8,e9d5fc00,c8478e80,c87a92e0,325b1b34,...) at seltdwait+0xc1 kern_select(c87a92e0,2,bfbfebdc,0,0,e9d5fc40,20,1,0) at kern_select+0x4ee select(c87a92e0,e9d5fcec,e9d5fd28,e9d5fc88,c0c0da10,...) at select+0x66 syscallenter(c87a92e0,e9d5fce4,e9d5fce4,0,369e99,...) at syscallenter+0x329 syscall(e9d5fd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x281e9553, esp = 0xbfbfeb7c, ebp = 0xbfbfecd8 --- Tracing command sshd pid 1971 tid 100197 td 0xc87a95c0 sched_switch(c87a95c0,0,104,d5a7ca90,f8,...) at sched_switch+0x2b6 mi_switch(104,0,c847db80,c897e2b0,e9d5ca18,...) at mi_switch+0x11b sleepq_switch(c87a95c0,0,c0ce693d,1a0,c87a95c0,...) at sleepq_switch+0xcc sleepq_catch_signals(c847db80,0,e9d5ca60,c087f20b,c847dba4,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c847dba4,0,c0ce81e5,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c847dba4,c847db90,0,0,c7ef3460,...) at _cv_wait_sig+0x18b seltdwait(c7ef3460,58,c8886080,c87a95c0,c8900001,...) at seltdwait+0xd7 kern_select(c87a95c0,c,286030b4,286030d8,0,0,20,c897e2b0,c897e2b0) at kern_select+0x4ee select(c87a95c0,e9d5ccec,e9d5cd28,28646000,c897e2b0,...) at select+0x66 syscallenter(c87a95c0,e9d5cce4,e9d5cce4,0,e9d5cd28,...) at syscallenter+0x329 syscall(e9d5cd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x284a9553, esp = 0xbfbfe1cc, ebp = 0xbfbfe218 --- Tracing command sshd pid 1970 tid 100151 td 0xc8333b80 sched_switch(c8333b80,0,104,50ab40c7,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c8389180,c82c5560,e9ca7a18,...) at mi_switch+0x11b sleepq_switch(c8333b80,0,c0ce693d,1a0,c8333b80,...) at sleepq_switch+0xcc sleepq_catch_signals(c8389180,0,e9ca7a60,c087f20b,c83891a4,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c83891a4,0,c0ce81e5,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c83891a4,c8389190,0,40,c8835e00,...) at _cv_wait_sig+0x18b seltdwait(c8835e00,58,c8507a00,c8333b80,c7f30001,...) at seltdwait+0xd7 kern_select(c8333b80,a,286030b4,286030d8,0,0,20,c82c5560,c82c5560) at kern_select+0x4ee select(c8333b80,e9ca7cec,e9ca7d28,e9ca7d28,28,...) at select+0x66 syscallenter(c8333b80,e9ca7ce4,e9ca7ce4,0,e9ca7d28,...) at syscallenter+0x329 syscall(e9ca7d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x284a9553, esp = 0xbfbfe1cc, ebp = 0xbfbfe218 --- Tracing command sh pid 1969 tid 100196 td 0xc87a98a0 sched_switch(c87a98a0,0,104,d9475f6a,f8,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c897e560,e9d59b50,...) at mi_switch+0x11b sleepq_switch(c87a98a0,0,c0ce693d,1a0,c87a98a0,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,e9d59ba4,c08cf4ac,c897e560,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c897e560,5c,c0ce72e1,100,0,...) at sleepq_wait_sig+0x18 _sleep(c897e560,c897e5e8,15c,c0ce72e1,0,...) at _sleep+0x28c kern_wait(c87a98a0,ffffffff,e9d59c44,2,0,...) at kern_wait+0xfa1 wait4(c87a98a0,e9d59cec,e9d59d28,281ff6e8,c897e560,...) at wait4+0x3b syscallenter(c87a98a0,e9d59ce4,c0c0a8be,c87a98a0,e9d59d28,...) at syscallenter+0x329 syscall(e9d59d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2815cd5b, esp = 0xbfbfeb5c, ebp = 0xbfbfeb78 --- Tracing command sshd pid 1968 tid 100188 td 0xc8783b80 sched_switch(c8783b80,0,104,d1d5c59c,f8,...) at sched_switch+0x2b6 mi_switch(104,0,c857b9c0,c87332b0,e9d27a18,...) at mi_switch+0x11b sleepq_switch(c8783b80,0,c0ce693d,1a0,c8783b80,...) at sleepq_switch+0xcc sleepq_catch_signals(c857b9c0,0,e9d27a60,c087f20b,c857b9e4,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c857b9e4,0,c0ce81e5,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c857b9e4,c857b9d0,0,0,c7f871f8,...) at _cv_wait_sig+0x18b seltdwait(c7f871f8,58,c8884300,c8783b80,0,...) at seltdwait+0xd7 kern_select(c8783b80,c,286030b4,286030d8,0,0,20,c87332b0,c87332b0) at kern_select+0x4ee select(c8783b80,e9d27cec,e9d27d28,28643000,c87332b0,...) at select+0x66 syscallenter(c8783b80,e9d27ce4,e9d27ce4,0,e9d27d28,...) at syscallenter+0x329 syscall(e9d27d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x284a9553, esp = 0xbfbfe1cc, ebp = 0xbfbfe218 --- Tracing command sshd pid 1964 tid 100176 td 0xc8787000 sched_switch(c8787000,0,104,d254e7fa,f8,...) at sched_switch+0x2b6 mi_switch(104,0,158,c8781810,e9d03a8c,...) at mi_switch+0x11b sleepq_switch(c8787000,0,c0ce693d,1a0,c8787000,...) at sleepq_switch+0xcc sleepq_catch_signals(158,0,e9d03ae0,c08cf4ac,c8298254,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c8298254,58,c0ce9e4f,100,0,...) at sleepq_wait_sig+0x18 _sleep(c8298254,c8298214,158,c0ce9e4f,0,...) at _sleep+0x28c sbwait(c82981f0,1,0,0,0,...) at sbwait+0x52 soreceive_generic(c829819c,0,e9d03c28,0,0,...) at soreceive_generic+0x3b1 soreceive(c829819c,0,e9d03c28,0,0,0) at soreceive+0x38 soo_read(c7b7c818,e9d03c28,c8551a00,0,c8787000,...) at soo_read+0x4e dofileread(e9d03c28,ffffffff,ffffffff,0,c7b7c818,...) at dofileread+0x96 kern_readv(c8787000,5,e9d03c28,e9d03c48,1,...) at kern_readv+0x58 read(c8787000,e9d03cec,e9d03d28,286004a8,c8781810,...) at read+0x4f syscallenter(c8787000,e9d03ce4,e9d03ce4,0,e9d03d28,...) at syscallenter+0x329 syscall(e9d03d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x284a95d3, esp = 0xbfbfdd9c, ebp = 0xbfbfddd8 --- Tracing command sshd pid 1963 tid 100140 td 0xc82698a0 sched_switch(c82698a0,0,104,d23c7f68,f8,...) at sched_switch+0x2b6 mi_switch(104,0,158,c8266ac0,e9c86a8c,...) at mi_switch+0x11b sleepq_switch(c82698a0,0,c0ce693d,1a0,c82698a0,...) at sleepq_switch+0xcc sleepq_catch_signals(158,0,e9c86ae0,c08cf4ac,c808b728,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c808b728,58,c0ce9e4f,100,0,...) at sleepq_wait_sig+0x18 _sleep(c808b728,c808b6e8,158,c0ce9e4f,0,...) at _sleep+0x28c sbwait(c808b6c4,1,c82698a0,c0945f02,c74ec5b8,...) at sbwait+0x52 soreceive_generic(c808b670,0,e9c86c28,0,0,...) at soreceive_generic+0x3b1 soreceive(c808b670,0,e9c86c28,0,0,0) at soreceive+0x38 soo_read(c7f9fc08,e9c86c28,c84c5800,0,c82698a0,...) at soo_read+0x4e dofileread(e9c86c28,ffffffff,ffffffff,0,c7f9fc08,...) at dofileread+0x96 kern_readv(c82698a0,5,e9c86c28,e9c86c48,1,...) at kern_readv+0x58 read(c82698a0,e9c86cec,e9c86d28,2863a00c,c8266ac0,...) at read+0x4f syscallenter(c82698a0,e9c86ce4,c0c0a8be,c82698a0,e9c86d28,...) at syscallenter+0x329 syscall(e9c86d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x284a95d3, esp = 0xbfbfdd9c, ebp = 0xbfbfddd8 --- Tracing command sshd pid 1962 tid 100118 td 0xc7f2db80 sched_switch(c7f2db80,0,104,d1ba37d2,f8,...) at sched_switch+0x2b6 mi_switch(104,0,158,c8084810,e9c44a8c,...) at mi_switch+0x11b sleepq_switch(c7f2db80,0,c0ce693d,1a0,c7f2db80,...) at sleepq_switch+0xcc sleepq_catch_signals(158,0,e9c44ae0,c08cf4ac,c88ac728,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c88ac728,58,c0ce9e4f,100,0,...) at sleepq_wait_sig+0x18 _sleep(c88ac728,c88ac6e8,158,c0ce9e4f,0,...) at _sleep+0x28c sbwait(c88ac6c4,1,0,c8708202,100,...) at sbwait+0x52 soreceive_generic(c88ac670,0,e9c44c28,0,0,...) at soreceive_generic+0x3b1 soreceive(c88ac670,0,e9c44c28,0,0,0) at soreceive+0x38 soo_read(c7b7de00,e9c44c28,c848a500,0,c7f2db80,...) at soo_read+0x4e dofileread(e9c44c28,ffffffff,ffffffff,0,c7b7de00,...) at dofileread+0x96 kern_readv(c7f2db80,5,e9c44c28,e9c44c48,1,...) at kern_readv+0x58 read(c7f2db80,e9c44cec,e9c44d28,286004a8,c8084810,...) at read+0x4f syscallenter(c7f2db80,e9c44ce4,e9c44ce4,0,e9c44d28,...) at syscallenter+0x329 syscall(e9c44d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x284a95d3, esp = 0xbfbfdd9c, ebp = 0xbfbfddd8 --- Tracing command linger2 pid 1239 tid 100125 td 0xc8087b80 sched_switch(c8087b80,0,104,516a23ce,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c80832b0,e9c59b50,...) at mi_switch+0x11b sleepq_switch(c8087b80,0,c0ce693d,1a0,c8087b80,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,e9c59ba4,c08cf4ac,c80832b0,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c80832b0,5c,c0ce72e1,100,0,...) at sleepq_wait_sig+0x18 _sleep(c80832b0,c8083338,15c,c0ce72e1,0,...) at _sleep+0x28c kern_wait(c8087b80,ffffffff,e9c59c44,0,0,...) at kern_wait+0xfa1 wait4(c8087b80,e9c59cec,e9c59d28,bfbfeac8,c80832b0,...) at wait4+0x3b syscallenter(c8087b80,e9c59ce4,e9c59ce4,0,e9c59d28,...) at syscallenter+0x329 syscall(e9c59d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x280ecd5b, esp = 0xbfbfea9c, ebp = 0xbfbfeab8 --- Tracing command bash pid 1238 tid 100160 td 0xc8332000 sched_switch(c8332000,0,104,b841fd28,52,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c82c3810,e9cc2b50,...) at mi_switch+0x11b sleepq_switch(c8332000,0,c0ce693d,1a0,c8332000,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,e9cc2ba4,c08cf4ac,c82c3810,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c82c3810,5c,c0ce72e1,100,0,...) at sleepq_wait_sig+0x18 _sleep(c82c3810,c82c3898,15c,c0ce72e1,0,...) at _sleep+0x28c kern_wait(c8332000,ffffffff,e9cc2c44,0,0,...) at kern_wait+0xfa1 wait4(c8332000,e9cc2cec,e9cc2d28,283626e8,c82c3810,...) at wait4+0x3b syscallenter(c8332000,e9cc2ce4,c0c0a8be,c8332000,e9cc2d28,...) at syscallenter+0x329 syscall(e9cc2d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x282bfd5b, esp = 0xbfbfe63c, ebp = 0xbfbfe658 --- Tracing command su pid 1237 tid 100137 td 0xc826a2e0 sched_switch(c826a2e0,0,104,b7fbd04c,52,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c8267560,e9c7db50,...) at mi_switch+0x11b sleepq_switch(c826a2e0,0,c0ce693d,1a0,c826a2e0,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,e9c7dba4,c08cf4ac,c8267560,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c8267560,5c,c0ce72e1,100,0,...) at sleepq_wait_sig+0x18 _sleep(c8267560,c82675e8,15c,c0ce72e1,0,...) at _sleep+0x28c kern_wait(c826a2e0,4d6,e9c7dc44,2,0,...) at kern_wait+0xfa1 wait4(c826a2e0,e9c7dcec,e9c7dd28,281c16e8,c8267560,...) at wait4+0x3b syscallenter(c826a2e0,e9c7dce4,c0c0a8be,c826a2e0,e9c7dd28,...) at syscallenter+0x329 syscall(e9c7dd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2811ed5b, esp = 0xbfbfe58c, ebp = 0xbfbfe5a8 --- Tracing command md5 pid 1232 tid 100121 td 0xc7f2d2e0 sched_switch(c7f2d2e0,0,104,8fda3bc9,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c7f2d2e0,0,e9c4dc6c,...) at mi_switch+0x11b sleepq_switch(c7f2d2e0,0,c0ce693d,260,24c,...) at sleepq_switch+0xcc sleepq_wait(c7b5d800,4c,c0cba8e6,0,0,...) at sleepq_wait+0x39 _sleep(c7b5d800,c7b5d820,24c,c0cba8e6,0,...) at _sleep+0x2a2 md_kthread(c7b5d800,e9c4dd28,28080cd0,0,0,...) at md_kthread+0x168 fork_exit(c06d71e0,c7b5d800,e9c4dd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe9c4dd60, ebp = 0 --- Tracing command sh pid 1218 tid 100128 td 0xc80872e0 sched_switch(c80872e0,0,104,b785b070,52,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c7f8e810,e9c62b50,...) at mi_switch+0x11b sleepq_switch(c80872e0,0,c0ce693d,1a0,c80872e0,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,e9c62ba4,c08cf4ac,c7f8e810,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7f8e810,5c,c0ce72e1,100,0,...) at sleepq_wait_sig+0x18 _sleep(c7f8e810,c7f8e898,15c,c0ce72e1,0,...) at _sleep+0x28c kern_wait(c80872e0,ffffffff,e9c62c44,2,0,...) at kern_wait+0xfa1 wait4(c80872e0,e9c62cec,e9c62d28,8066634,c7f8e810,...) at wait4+0x3b syscallenter(c80872e0,e9c62ce4,e9c62ce4,0,e9c62d28,...) at syscallenter+0x329 syscall(e9c62d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2815cd5b, esp = 0xbfbfe87c, ebp = 0xbfbfe898 --- Tracing command time pid 1217 tid 100155 td 0xc8333000 sched_switch(c8333000,0,104,6812fc9c,52,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c82c4810,e9cb3b50,...) at mi_switch+0x11b sleepq_switch(c8333000,0,c0ce693d,1a0,c8333000,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,e9cb3ba4,c08cf4ac,c82c4810,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c82c4810,5c,c0ce72e1,100,0,...) at sleepq_wait_sig+0x18 _sleep(c82c4810,c82c4898,15c,c0ce72e1,0,...) at _sleep+0x28c kern_wait(c8333000,4c2,e9cb3c44,0,e9cb3bfc,...) at kern_wait+0xfa1 wait4(c8333000,e9cb3cec,e9cb3d28,2819055c,c82c4810,...) at wait4+0x3b syscallenter(c8333000,e9cb3ce4,e9cb3ce4,0,e9cb3d28,...) at syscallenter+0x329 syscall(e9cb3d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x280edd5b, esp = 0xbfbfe9ec, ebp = 0xbfbfeab8 --- Tracing command sh pid 1212 tid 100148 td 0xc8268000 sched_switch(c8268000,0,104,6745fd57,52,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c80852b0,e9c9eb50,...) at mi_switch+0x11b sleepq_switch(c8268000,0,c0ce693d,1a0,c8268000,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,e9c9eba4,c08cf4ac,c80852b0,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c80852b0,5c,c0ce72e1,100,0,...) at sleepq_wait_sig+0x18 _sleep(c80852b0,c8085338,15c,c0ce72e1,0,...) at _sleep+0x28c kern_wait(c8268000,ffffffff,e9c9ec44,2,0,...) at kern_wait+0xfa1 wait4(c8268000,e9c9ecec,e9c9ed28,8066634,c80852b0,...) at wait4+0x3b syscallenter(c8268000,e9c9ece4,e9c9ece4,0,e9c9ed28,...) at syscallenter+0x329 syscall(e9c9ed28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2815cd5b, esp = 0xbfbfe7bc, ebp = 0xbfbfe7d8 --- Tracing command bash pid 1205 tid 100149 td 0xc8088b80 sched_switch(c8088b80,0,104,6424c752,52,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c82c5ac0,e9ca1b50,...) at mi_switch+0x11b sleepq_switch(c8088b80,0,c0ce693d,1a0,c8088b80,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,e9ca1ba4,c08cf4ac,c82c5ac0,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c82c5ac0,5c,c0ce72e1,100,0,...) at sleepq_wait_sig+0x18 _sleep(c82c5ac0,c82c5b48,15c,c0ce72e1,0,...) at _sleep+0x28c kern_wait(c8088b80,ffffffff,e9ca1c44,6,0,...) at kern_wait+0xfa1 wait4(c8088b80,e9ca1cec,e9ca1d28,28456360,c82c5ac0,...) at wait4+0x3b syscallenter(c8088b80,e9ca1ce4,e9ca1ce4,0,e9ca1d28,...) at syscallenter+0x329 syscall(e9ca1d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x282bfd5b, esp = 0xbfbfe7fc, ebp = 0xbfbfe818 --- Tracing command csh pid 1201 tid 100117 td 0xc7f91000 sched_switch(c7f91000,0,104,89f52592,45,...) at sched_switch+0x2b6 mi_switch(104,0,168,c8084ac0,e9c41b9c,...) at mi_switch+0x11b sleepq_switch(c7f91000,0,c0ce693d,1a0,c7f91000,...) at sleepq_switch+0xcc sleepq_catch_signals(168,0,e9c41bf0,c08cf4ac,c8084b18,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c8084b18,68,c0c9b7f5,100,0,...) at sleepq_wait_sig+0x18 _sleep(c8084b18,c8084b48,168,c0c9b7f5,0,...) at _sleep+0x28c kern_sigsuspend(c7f91000,0,0,0,0,...) at kern_sigsuspend+0xdd sigsuspend(c7f91000,e9c41cec,e9c41d28,2852c000,c8084ac0,...) at sigsuspend+0x4d syscallenter(c7f91000,e9c41ce4,e9c41ce4,0,e9c41d28,...) at syscallenter+0x329 syscall(e9c41d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x28179c0b, esp = 0xbfbfe61c, ebp = 0xbfbfe688 --- Tracing command su pid 1200 tid 100083 td 0xc7b788a0 sched_switch(c7b788a0,0,104,38bf8623,45,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c7ab8810,e9b49b50,...) at mi_switch+0x11b sleepq_switch(c7b788a0,0,c0ce693d,1a0,c7b788a0,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,e9b49ba4,c08cf4ac,c7ab8810,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7ab8810,5c,c0ce72e1,100,0,...) at sleepq_wait_sig+0x18 _sleep(c7ab8810,c7ab8898,15c,c0ce72e1,0,...) at _sleep+0x28c kern_wait(c7b788a0,4b1,e9b49c44,2,0,...) at kern_wait+0xfa1 wait4(c7b788a0,e9b49cec,e9b49d28,281c16e8,c7ab8810,...) at wait4+0x3b syscallenter(c7b788a0,e9b49ce4,c0c0a8be,c7b788a0,e9b49d28,...) at syscallenter+0x329 syscall(e9b49d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2811ed5b, esp = 0xbfbfe63c, ebp = 0xbfbfe658 --- Tracing command bash pid 1196 tid 100123 td 0xc80882e0 sched_switch(c80882e0,0,104,510d612f,44,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c8083810,e9c53b50,...) at mi_switch+0x11b sleepq_switch(c80882e0,0,c0ce693d,1a0,c80882e0,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,e9c53ba4,c08cf4ac,c8083810,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c8083810,5c,c0ce72e1,100,0,...) at sleepq_wait_sig+0x18 _sleep(c8083810,c8083898,15c,c0ce72e1,0,...) at _sleep+0x28c kern_wait(c80882e0,ffffffff,e9c53c44,6,0,...) at kern_wait+0xfa1 wait4(c80882e0,e9c53cec,e9c53d28,28429004,c8083810,...) at wait4+0x3b syscallenter(c80882e0,e9c53ce4,e9c53ce4,0,e9c53d28,...) at syscallenter+0x329 syscall(e9c53d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x282bfd5b, esp = 0xbfbfe9cc, ebp = 0xbfbfe9e8 --- Tracing command sshd pid 1195 tid 100142 td 0xc82692e0 sched_switch(c82692e0,0,104,b628ee87,52,...) at sched_switch+0x2b6 mi_switch(104,0,c7a95b00,c8266560,e9c8ca18,...) at mi_switch+0x11b sleepq_switch(c82692e0,0,c0ce693d,1a0,c82692e0,...) at sleepq_switch+0xcc sleepq_catch_signals(c7a95b00,0,e9c8ca60,c087f20b,c7a95b24,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7a95b24,0,c0ce81e5,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c7a95b24,c7a95b10,0,40,c7b7c038,...) at _cv_wait_sig+0x18b seltdwait(c7b7c038,58,c7eeab80,c82692e0,c7ee0001,...) at seltdwait+0xd7 kern_select(c82692e0,a,286030b4,286030d8,0,0,20,c8266560,c8266560) at kern_select+0x4ee select(c82692e0,e9c8ccec,e9c8cd28,28662000,c8266560,...) at select+0x66 syscallenter(c82692e0,e9c8cce4,e9c8cce4,0,e9c8cd28,...) at syscallenter+0x329 syscall(e9c8cd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x284a9553, esp = 0xbfbfe1cc, ebp = 0xbfbfe218 --- Tracing command sshd pid 1193 tid 100143 td 0xc8269000 sched_switch(c8269000,0,104,985d1ce1,42,...) at sched_switch+0x2b6 mi_switch(104,0,158,c82662b0,e9c8fa8c,...) at mi_switch+0x11b sleepq_switch(c8269000,0,c0ce693d,1a0,c8269000,...) at sleepq_switch+0xcc sleepq_catch_signals(158,0,e9c8fae0,c08cf4ac,c8081a60,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c8081a60,58,c0ce9e4f,100,0,...) at sleepq_wait_sig+0x18 _sleep(c8081a60,c8081a20,158,c0ce9e4f,0,...) at _sleep+0x28c sbwait(c80819fc,1,e9c8fb38,c0945fa5,c74ec5b8,...) at sbwait+0x52 soreceive_generic(c80819a8,0,e9c8fc28,0,0,...) at soreceive_generic+0x3b1 soreceive(c80819a8,0,e9c8fc28,0,0,0) at soreceive+0x38 soo_read(c7b915e8,e9c8fc28,c7b15c00,0,c8269000,...) at soo_read+0x4e dofileread(e9c8fc28,ffffffff,ffffffff,0,c7b915e8,...) at dofileread+0x96 kern_readv(c8269000,5,e9c8fc28,e9c8fc48,1,...) at kern_readv+0x58 read(c8269000,e9c8fcec,e9c8fd28,2860800c,c82662b0,...) at read+0x4f syscallenter(c8269000,e9c8fce4,e9c8fce4,0,e9c8fd28,...) at syscallenter+0x329 syscall(e9c8fd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x284a95d3, esp = 0xbfbfdd9c, ebp = 0xbfbfddd8 --- Tracing command getty pid 1190 tid 100147 td 0xc82682e0 sched_switch(c82682e0,0,104,a50d01e5,21,...) at sched_switch+0x2b6 mi_switch(104,0,c82682e0,c8085560,e9c9baa4,...) at mi_switch+0x11b sleepq_switch(c82682e0,0,c0ce693d,1a0,c82682e0,...) at sleepq_switch+0xcc sleepq_catch_signals(c82682e0,1,e9c9baec,c087f20b,c74d8070,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c74d8070,0,c0ce8e7a,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c74d8070,c0e2ff30,0,c74d8000,e9c9bb58,...) at _cv_wait_sig+0x18b tty_wait(c74d8000,c74d8070,e9c9bc28,1,0,...) at tty_wait+0x23 ttydisc_read(c74d8000,e9c9bc28,0,e9c9bba8,e9c9bba4,...) at ttydisc_read+0x1cc ttydev_read(c73d2c00,e9c9bc28,0,0,1,...) at ttydev_read+0x10a devfs_read_f(c7ef38f8,e9c9bc28,c7154100,0,c82682e0,...) at devfs_read_f+0x7e dofileread(e9c9bc28,ffffffff,ffffffff,0,c7ef38f8,...) at dofileread+0x96 kern_readv(c82682e0,0,e9c9bc28,e9c9bc48,1,...) at kern_readv+0x58 read(c82682e0,e9c9bcec,e9c9bd28,e9c9bc88,c0c0da10,...) at read+0x4f syscallenter(c82682e0,e9c9bce4,e9c9bce4,0,e9c9bd28,...) at syscallenter+0x329 syscall(e9c9bd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f5d3, esp = 0xbfbfedac, ebp = 0xbfbfedd8 --- Tracing command getty pid 1189 tid 100146 td 0xc82685c0 sched_switch(c82685c0,0,104,a4fa9086,21,...) at sched_switch+0x2b6 mi_switch(104,0,c82685c0,c8085810,e9c98aa4,...) at mi_switch+0x11b sleepq_switch(c82685c0,0,c0ce693d,1a0,c82685c0,...) at sleepq_switch+0xcc sleepq_catch_signals(c82685c0,1,e9c98aec,c087f20b,c74d8270,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c74d8270,0,c0ce8e7a,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c74d8270,c0e2ff30,0,c74d8200,e9c98b58,...) at _cv_wait_sig+0x18b tty_wait(c74d8200,c74d8270,e9c98c28,1,0,...) at tty_wait+0x23 ttydisc_read(c74d8200,e9c98c28,0,e9c98ba8,e9c98ba4,...) at ttydisc_read+0x1cc ttydev_read(c74c8000,e9c98c28,0,0,1,...) at ttydev_read+0x10a devfs_read_f(c7b91620,e9c98c28,c7154100,0,c82685c0,...) at devfs_read_f+0x7e dofileread(e9c98c28,ffffffff,ffffffff,0,c7b91620,...) at dofileread+0x96 kern_readv(c82685c0,0,e9c98c28,e9c98c48,1,...) at kern_readv+0x58 read(c82685c0,e9c98cec,e9c98d28,e9c98c88,c0c0da10,...) at read+0x4f syscallenter(c82685c0,e9c98ce4,e9c98ce4,0,e9c98d28,...) at syscallenter+0x329 syscall(e9c98d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f5d3, esp = 0xbfbfedac, ebp = 0xbfbfedd8 --- Tracing command getty pid 1188 tid 100145 td 0xc82688a0 sched_switch(c82688a0,0,104,a50c940b,21,...) at sched_switch+0x2b6 mi_switch(104,0,c82688a0,c8085ac0,e9c95aa4,...) at mi_switch+0x11b sleepq_switch(c82688a0,0,c0ce693d,1a0,c82688a0,...) at sleepq_switch+0xcc sleepq_catch_signals(c82688a0,1,e9c95aec,c087f20b,c73d7a70,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c73d7a70,0,c0ce8e7a,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c73d7a70,c0e2ff30,0,c73d7a00,e9c95b58,...) at _cv_wait_sig+0x18b tty_wait(c73d7a00,c73d7a70,e9c95c28,1,0,...) at tty_wait+0x23 ttydisc_read(c73d7a00,e9c95c28,0,e9c95ba8,e9c95ba4,...) at ttydisc_read+0x1cc ttydev_read(c74c8100,e9c95c28,0,0,1,...) at ttydev_read+0x10a devfs_read_f(c7bcf818,e9c95c28,c7154100,0,c82688a0,...) at devfs_read_f+0x7e dofileread(e9c95c28,ffffffff,ffffffff,0,c7bcf818,...) at dofileread+0x96 kern_readv(c82688a0,0,e9c95c28,e9c95c48,1,...) at kern_readv+0x58 read(c82688a0,e9c95cec,e9c95d28,e9c95c88,c0c0da10,...) at read+0x4f syscallenter(c82688a0,e9c95ce4,e9c95ce4,0,e9c95d28,...) at syscallenter+0x329 syscall(e9c95d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f5d3, esp = 0xbfbfedac, ebp = 0xbfbfedd8 --- Tracing command getty pid 1187 tid 100144 td 0xc8268b80 sched_switch(c8268b80,0,104,a5086118,21,...) at sched_switch+0x2b6 mi_switch(104,0,c8268b80,c8266000,e9c92aa4,...) at mi_switch+0x11b sleepq_switch(c8268b80,0,c0ce693d,1a0,c8268b80,...) at sleepq_switch+0xcc sleepq_catch_signals(c8268b80,1,e9c92aec,c087f20b,c7367670,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7367670,0,c0ce8e7a,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c7367670,c0e2ff30,0,c7367600,e9c92b58,...) at _cv_wait_sig+0x18b tty_wait(c7367600,c7367670,e9c92c28,1,0,...) at tty_wait+0x23 ttydisc_read(c7367600,e9c92c28,0,e9c92ba8,e9c92ba4,...) at ttydisc_read+0x1cc ttydev_read(c74c8200,e9c92c28,0,0,1,...) at ttydev_read+0x10a devfs_read_f(c7bcf700,e9c92c28,c7154100,0,c8268b80,...) at devfs_read_f+0x7e dofileread(e9c92c28,ffffffff,ffffffff,0,c7bcf700,...) at dofileread+0x96 kern_readv(c8268b80,0,e9c92c28,e9c92c48,1,...) at kern_readv+0x58 read(c8268b80,e9c92cec,e9c92d28,e9c92c88,c0c0da10,...) at read+0x4f syscallenter(c8268b80,e9c92ce4,e9c92ce4,0,e9c92d28,...) at syscallenter+0x329 syscall(e9c92d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f5d3, esp = 0xbfbfedac, ebp = 0xbfbfedd8 --- Tracing command getty pid 1186 tid 100141 td 0xc82695c0 sched_switch(c82695c0,0,104,a510125f,21,...) at sched_switch+0x2b6 mi_switch(104,0,c82695c0,c8266810,e9c89aa4,...) at mi_switch+0x11b sleepq_switch(c82695c0,0,c0ce693d,1a0,c82695c0,...) at sleepq_switch+0xcc sleepq_catch_signals(c82695c0,1,e9c89aec,c087f20b,c7367870,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7367870,0,c0ce8e7a,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c7367870,c0e2ff30,0,c7367800,e9c89b58,...) at _cv_wait_sig+0x18b tty_wait(c7367800,c7367870,e9c89c28,1,0,...) at tty_wait+0x23 ttydisc_read(c7367800,e9c89c28,0,e9c89ba8,e9c89ba4,...) at ttydisc_read+0x1cc ttydev_read(c74c8500,e9c89c28,0,0,1,...) at ttydev_read+0x10a devfs_read_f(c7b91000,e9c89c28,c7154100,0,c82695c0,...) at devfs_read_f+0x7e dofileread(e9c89c28,ffffffff,ffffffff,0,c7b91000,...) at dofileread+0x96 kern_readv(c82695c0,0,e9c89c28,e9c89c48,1,...) at kern_readv+0x58 read(c82695c0,e9c89cec,e9c89d28,e9c89c88,c0c0da10,...) at read+0x4f syscallenter(c82695c0,e9c89ce4,e9c89ce4,0,e9c89d28,...) at syscallenter+0x329 syscall(e9c89d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f5d3, esp = 0xbfbfedac, ebp = 0xbfbfedd8 --- Tracing command getty pid 1185 tid 100139 td 0xc8269b80 sched_switch(c8269b80,0,104,a4feeafd,21,...) at sched_switch+0x2b6 mi_switch(104,0,c8269b80,c8267000,e9c83aa4,...) at mi_switch+0x11b sleepq_switch(c8269b80,0,c0ce693d,1a0,c8269b80,...) at sleepq_switch+0xcc sleepq_catch_signals(c8269b80,1,e9c83aec,c087f20b,c7367c70,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7367c70,0,c0ce8e7a,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c7367c70,c0e2ff30,0,c7367c00,e9c83b58,...) at _cv_wait_sig+0x18b tty_wait(c7367c00,c7367c70,e9c83c28,1,0,...) at tty_wait+0x23 ttydisc_read(c7367c00,e9c83c28,0,e9c83ba8,e9c83ba4,...) at ttydisc_read+0x1cc ttydev_read(c74c8600,e9c83c28,0,0,1,...) at ttydev_read+0x10a devfs_read_f(c7bcf620,e9c83c28,c7154100,0,c8269b80,...) at devfs_read_f+0x7e dofileread(e9c83c28,ffffffff,ffffffff,0,c7bcf620,...) at dofileread+0x96 kern_readv(c8269b80,0,e9c83c28,e9c83c48,1,...) at kern_readv+0x58 read(c8269b80,e9c83cec,e9c83d28,e9c83c88,c0c0da10,...) at read+0x4f syscallenter(c8269b80,e9c83ce4,e9c83ce4,0,e9c83d28,...) at syscallenter+0x329 syscall(e9c83d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f5d3, esp = 0xbfbfedac, ebp = 0xbfbfedd8 --- Tracing command getty pid 1184 tid 100138 td 0xc826a000 sched_switch(c826a000,0,104,a5046a82,21,...) at sched_switch+0x2b6 mi_switch(104,0,c826a000,c82672b0,e9c80aa4,...) at mi_switch+0x11b sleepq_switch(c826a000,0,c0ce693d,1a0,c826a000,...) at sleepq_switch+0xcc sleepq_catch_signals(c826a000,1,e9c80aec,c087f20b,c73d7670,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c73d7670,0,c0ce8e7a,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c73d7670,c0e2ff30,0,c73d7600,e9c80b58,...) at _cv_wait_sig+0x18b tty_wait(c73d7600,c73d7670,e9c80c28,1,0,...) at tty_wait+0x23 ttydisc_read(c73d7600,e9c80c28,0,e9c80ba8,e9c80ba4,...) at ttydisc_read+0x1cc ttydev_read(c74c8700,e9c80c28,0,0,1,...) at ttydev_read+0x10a devfs_read_f(c7b7d230,e9c80c28,c7154100,0,c826a000,...) at devfs_read_f+0x7e dofileread(e9c80c28,ffffffff,ffffffff,0,c7b7d230,...) at dofileread+0x96 kern_readv(c826a000,0,e9c80c28,e9c80c48,1,...) at kern_readv+0x58 read(c826a000,e9c80cec,e9c80d28,e9c80c88,c0c0da10,...) at read+0x4f syscallenter(c826a000,e9c80ce4,e9c80ce4,0,e9c80d28,...) at syscallenter+0x329 syscall(e9c80d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f5d3, esp = 0xbfbfedac, ebp = 0xbfbfedd8 --- Tracing command getty pid 1183 tid 100136 td 0xc826a5c0 sched_switch(c826a5c0,0,104,a4fc8fe2,21,...) at sched_switch+0x2b6 mi_switch(104,0,c826a5c0,c8267810,e9c7aaa4,...) at mi_switch+0x11b sleepq_switch(c826a5c0,0,c0ce693d,1a0,c826a5c0,...) at sleepq_switch+0xcc sleepq_catch_signals(c826a5c0,1,e9c7aaec,c087f20b,c73d7870,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c73d7870,0,c0ce8e7a,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c73d7870,c0e2ff30,0,c73d7800,e9c7ab58,...) at _cv_wait_sig+0x18b tty_wait(c73d7800,c73d7870,e9c7ac28,1,0,...) at tty_wait+0x23 ttydisc_read(c73d7800,e9c7ac28,0,e9c7aba8,e9c7aba4,...) at ttydisc_read+0x1cc ttydev_read(c74c8800,e9c7ac28,0,0,1,...) at ttydev_read+0x10a devfs_read_f(c7b7d620,e9c7ac28,c7154100,0,c826a5c0,...) at devfs_read_f+0x7e dofileread(e9c7ac28,ffffffff,ffffffff,0,c7b7d620,...) at dofileread+0x96 kern_readv(c826a5c0,0,e9c7ac28,e9c7ac48,1,...) at kern_readv+0x58 read(c826a5c0,e9c7acec,e9c7ad28,e9c7ac88,c0c0da10,...) at read+0x4f syscallenter(c826a5c0,e9c7ace4,e9c7ace4,0,e9c7ad28,...) at syscallenter+0x329 syscall(e9c7ad28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f5d3, esp = 0xbfbfedac, ebp = 0xbfbfedd8 --- Tracing command getty pid 1182 tid 100135 td 0xc826a8a0 sched_switch(c826a8a0,0,104,675a37d8,52,...) at sched_switch+0x2b6 mi_switch(104,0,2,c8267ac0,e9c77aa4,...) at mi_switch+0x11b sleepq_switch(c826a8a0,0,c0ce693d,1a0,c826a8a0,...) at sleepq_switch+0xcc sleepq_catch_signals(2,0,e9c77aec,c087f20b,c7367a70,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7367a70,0,c0ce8e7a,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c7367a70,c7367a04,0,c7367a00,e9c77b58,...) at _cv_wait_sig+0x18b tty_wait(c7367a00,c7367a70,e9c77c28,1,0,...) at tty_wait+0x23 ttydisc_read(c7367a00,e9c77c28,0,e9c77ba8,e9c77ba4,...) at ttydisc_read+0x1cc ttydev_read(c7146900,e9c77c28,0,0,1,...) at ttydev_read+0x10a devfs_read_f(c7b7d4d0,e9c77c28,c7154100,0,c826a8a0,...) at devfs_read_f+0x7e dofileread(e9c77c28,ffffffff,ffffffff,0,c7b7d4d0,...) at dofileread+0x96 kern_readv(c826a8a0,0,e9c77c28,e9c77c48,1,...) at kern_readv+0x58 read(c826a8a0,e9c77cec,e9c77d28,e9c77c88,c0c0da10,...) at read+0x4f syscallenter(c826a8a0,e9c77ce4,e9c77ce4,0,e9c77d28,...) at syscallenter+0x329 syscall(e9c77d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f5d3, esp = 0xbfbfedac, ebp = 0xbfbfedd8 --- Tracing command inetd pid 1143 tid 100080 td 0xc7b7b2e0 sched_switch(c7b7b2e0,0,104,98cc55ad,21,...) at sched_switch+0x2b6 mi_switch(104,0,c7ee72c0,c7b772b0,e9b3ca18,...) at mi_switch+0x11b sleepq_switch(c7b7b2e0,0,c0ce693d,1a0,c7b7b2e0,...) at sleepq_switch+0xcc sleepq_catch_signals(c7ee72c0,0,e9b3ca60,c087f20b,c7ee72e4,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7ee72e4,0,c0ce81e5,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c7ee72e4,c7ee72d0,0,0,c7bcf2a0,...) at _cv_wait_sig+0x18b seltdwait(c7bcf2a0,58,c7154100,c7b7b2e0,0,...) at seltdwait+0xd7 kern_select(c7b7b2e0,6,bfbfdfc0,0,0,0,20,c7b772b0,c7b772b0) at kern_select+0x4ee select(c7b7b2e0,e9b3ccec,e9b3cd28,28221000,c7b772b0,...) at select+0x66 syscallenter(c7b7b2e0,e9b3cce4,e9b3cce4,0,e9b3cd28,...) at syscallenter+0x329 syscall(e9b3cd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x281a3553, esp = 0xbfbfdf7c, ebp = 0xbfbfee08 --- Tracing command moused pid 1117 tid 100122 td 0xc80885c0 sched_switch(c80885c0,0,104,93c84758,21,...) at sched_switch+0x2b6 mi_switch(104,0,c7ad7bc0,c8083ac0,e9c50a18,...) at mi_switch+0x11b sleepq_switch(c80885c0,0,c0ce693d,1a0,c80885c0,...) at sleepq_switch+0xcc sleepq_catch_signals(c7ad7bc0,0,e9c50a60,c087f20b,c7ad7be4,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7ad7be4,0,c0ce81e5,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c7ad7be4,c7ad7bd0,c0de8880,c73d2e00,c7bcfd20,...) at _cv_wait_sig+0x18b seltdwait(c7bcfd20,58,c7154100,c80885c0,0,...) at seltdwait+0xd7 kern_select(c80885c0,6,bfbfea24,0,0,0,20,c8083ac0,c8083ac0) at kern_select+0x4ee select(c80885c0,e9c50cec,e9c50d28,281d6668,c8083ac0,...) at select+0x66 syscallenter(c80885c0,e9c50ce4,e9c50ce4,0,e9c50d28,...) at syscallenter+0x329 syscall(e9c50d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x281ac553, esp = 0xbfbfe9bc, ebp = 0xbfbfeb48 --- Tracing command cron pid 1093 tid 100078 td 0xc7b7b8a0 sched_switch(c7b7b8a0,0,104,214112e6,1b8,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c7b77810,e9b34b88,...) at mi_switch+0x11b sleepq_switch(c7b7b8a0,0,c0ce693d,1a0,c7b7baa0,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,100,e9b34be0,c08cf478,...) at sleepq_catch_signals+0x52 sleepq_timedwait_sig(c0e30784,5c,c0ce4170,100,0,...) at sleepq_timedwait_sig+0x1c _sleep(c0e30784,0,15c,c0ce4170,ea61,...) at _sleep+0x258 kern_nanosleep(c7b7b8a0,e9b34c34,e9b34c3c,3c,0,...) at kern_nanosleep+0xc1 nanosleep(c7b7b8a0,e9b34cec,e9b34d28,281c3668,c7b77810,...) at nanosleep+0x6f syscallenter(c7b7b8a0,e9b34ce4,e9b34ce4,0,e9b34d28,...) at syscallenter+0x329 syscall(e9b34d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x2817bf57, esp = 0xbfbfec7c, ebp = 0xbfbfeca8 --- Tracing command sendmail pid 1086 tid 100124 td 0xc8088000 sched_switch(c8088000,0,104,7df3c687,21,...) at sched_switch+0x2b6 mi_switch(104,0,168,c8083560,e9c56b9c,...) at mi_switch+0x11b sleepq_switch(c8088000,0,c0ce693d,1a0,c8088000,...) at sleepq_switch+0xcc sleepq_catch_signals(168,0,e9c56bf0,c08cf4ac,c80835b8,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c80835b8,68,c0c9b7f5,100,0,...) at sleepq_wait_sig+0x18 _sleep(c80835b8,c80835e8,168,c0c9b7f5,0,...) at _sleep+0x28c kern_sigsuspend(c8088000,0,0,0,0,...) at kern_sigsuspend+0xdd sigsuspend(c8088000,e9c56cec,e9c56d28,bfbffff4,c8083560,...) at sigsuspend+0x4d syscallenter(c8088000,e9c56ce4,e9c56ce4,0,e9c56d28,...) at syscallenter+0x329 syscall(e9c56d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x28334c0b, esp = 0xbfbfcf8c, ebp = 0xbfbfcfb8 --- Tracing command sendmail pid 1082 tid 100108 td 0xc7f2bb80 sched_switch(c7f2bb80,0,104,7e71f66e,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c808c040,c7f25ac0,e9c26a14,...) at mi_switch+0x11b sleepq_switch(c7f2bb80,0,c0ce693d,1a0,c7f2bd80,...) at sleepq_switch+0xcc sleepq_catch_signals(c808c040,0,c7f2bb80,e9c26a60,c087ed63,...) at sleepq_catch_signals+0x52 sleepq_timedwait_sig(c808c064,0,c0ce81e5,101,0,...) at sleepq_timedwait_sig+0x1c _cv_timedwait_sig(c808c064,c808c050,1389,c7b15880,c7f2bb80,...) at _cv_timedwait_sig+0x1a3 seltdwait(e9c26bf8,e9c26c00,c7b15880,c7f2bb80,e9c26ab8,...) at seltdwait+0xc1 kern_select(c7f2bb80,5,bfbfc500,0,0,e9c26c40,20,5,0) at kern_select+0x4ee select(c7f2bb80,e9c26cec,e9c26d28,4100cf,e9c26c84,...) at select+0x66 syscallenter(c7f2bb80,e9c26ce4,e9c26ce4,0,369e99,...) at syscallenter+0x329 syscall(e9c26d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x283db553, esp = 0xbfbfc46c, ebp = 0xbfbfcf98 --- Tracing command sshd pid 1067 tid 100095 td 0xc74cc2e0 sched_switch(c74cc2e0,0,104,4acc148e,100,...) at sched_switch+0x2b6 mi_switch(104,0,c808c540,c7f8d810,e9bc9a18,...) at mi_switch+0x11b sleepq_switch(c74cc2e0,0,c0ce693d,1a0,c74cc2e0,...) at sleepq_switch+0xcc sleepq_catch_signals(c808c540,0,e9bc9a60,c087f20b,c808c564,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c808c564,0,c0ce81e5,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c808c564,c808c550,58,c7b53880,c74cc2e0,...) at _cv_wait_sig+0x18b seltdwait(c7ef35e8,58,c7b53880,c74cc2e0,0,...) at seltdwait+0xd7 kern_select(c74cc2e0,5,286090ac,0,0,0,20,c7f8d810,c7f8d810) at kern_select+0x4ee select(c74cc2e0,e9bc9cec,e9bc9d28,2861a070,c7f8d810,...) at select+0x66 syscallenter(c74cc2e0,e9bc9ce4,c0c0a8be,c74cc2e0,e9bc9d28,...) at syscallenter+0x329 syscall(e9bc9d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x284a9553, esp = 0xbfbfe28c, ebp = 0xbfbfee28 --- Tracing command ntpd pid 1022 tid 100127 td 0xc80875c0 sched_switch(c80875c0,0,104,35cde0ec,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c7a96880,c7f8eac0,e9c5fa18,...) at mi_switch+0x11b sleepq_switch(c80875c0,0,c0ce693d,1a0,c80875c0,...) at sleepq_switch+0xcc sleepq_catch_signals(c7a96880,0,e9c5fa60,c087f20b,c7a968a4,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7a968a4,0,c0ce81e5,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c7a968a4,c7a96890,58,c7154100,c80875c0,...) at _cv_wait_sig+0x18b seltdwait(c7f87ab8,58,c7154100,c80875c0,0,...) at seltdwait+0xd7 kern_select(c80875c0,1b,bfbfed18,0,0,0,20,c7f8eac0,c7f8eac0) at kern_select+0x4ee select(c80875c0,e9c5fcec,e9c5fd80,e9c5fc88,c0c0da10,...) at select+0x66 syscallenter(c80875c0,e9c5fce4,c0c0a5db,fffffffe,3,...) at syscallenter+0x329 syscall(e9c5fd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x28355553, esp = 0xbfbfecec, ebp = 0xbfbfedb8 --- Tracing command nfsd pid 898 tid 100116 td 0xc7f912e0 sched_switch(c7f912e0,0,104,6b19518,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c74baa80,c7f272b0,e9c3ebc8,...) at mi_switch+0x11b sleepq_switch(c7f912e0,0,c0ce693d,1a0,c7f914e0,...) at sleepq_switch+0xcc sleepq_catch_signals(c74baa80,0,c7f912e0,e9c3ec14,c087ed63,...) at sleepq_catch_signals+0x52 sleepq_timedwait_sig(c7ee7390,0,c0cfead7,101,0,...) at sleepq_timedwait_sig+0x1c _cv_timedwait_sig(c7ee7390,c74baa80,1388,0,0,...) at _cv_timedwait_sig+0x1a3 svc_run_internal(e9c3ed14,c0899d31,c74baa80,e9c3ed28,0,...) at svc_run_internal+0x30d svc_thread_start(c74baa80,e9c3ed28,0,0,0,...) at svc_thread_start+0x10 fork_exit(c0acb730,c74baa80,e9c3ed28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0x804c72e, eip = 0xc, esp = 0x33, ebp = 0 --- Tracing command nfsd pid 898 tid 100115 td 0xc7f915c0 sched_switch(c7f915c0,0,104,703e8b9,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c74baa80,c7f272b0,e9c3bbc8,...) at mi_switch+0x11b sleepq_switch(c7f915c0,0,c0ce693d,1a0,c7f917c0,...) at sleepq_switch+0xcc sleepq_catch_signals(c74baa80,0,c7f915c0,e9c3bc14,c087ed63,...) at sleepq_catch_signals+0x52 sleepq_timedwait_sig(c7ee73d0,0,c0cfead7,101,0,...) at sleepq_timedwait_sig+0x1c _cv_timedwait_sig(c7ee73d0,c74baa80,1388,0,6669,...) at _cv_timedwait_sig+0x1a3 svc_run_internal(e9c3bd14,c0899d31,c74baa80,e9c3bd28,3d6b6361,...) at svc_run_internal+0x30d svc_thread_start(c74baa80,e9c3bd28,3d6b6361,0,f,...) at svc_thread_start+0x10 fork_exit(c0acb730,c74baa80,e9c3bd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0x804c72e, eip = 0xc, esp = 0x33, ebp = 0 --- Tracing command nfsd pid 898 tid 100114 td 0xc7f918a0 sched_switch(c7f918a0,0,104,6b1d394,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c74baa80,c7f272b0,e9c38bc8,...) at mi_switch+0x11b sleepq_switch(c7f918a0,0,c0ce693d,1a0,c7f91aa0,...) at sleepq_switch+0xcc sleepq_catch_signals(c74baa80,0,c7f918a0,e9c38c14,c087ed63,...) at sleepq_catch_signals+0x52 sleepq_timedwait_sig(c7ee7410,0,c0cfead7,101,0,...) at sleepq_timedwait_sig+0x1c _cv_timedwait_sig(c7ee7410,c74baa80,1388,0,19,...) at _cv_timedwait_sig+0x1a3 svc_run_internal(e9c38d14,c0899d31,c74baa80,e9c38d28,0,...) at svc_run_internal+0x30d svc_thread_start(c74baa80,e9c38d28,0,28323ca4,28323ce0,...) at svc_thread_start+0x10 fork_exit(c0acb730,c74baa80,e9c38d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0x804c72e, eip = 0xc, esp = 0x33, ebp = 0 --- Tracing command nfsd pid 898 tid 100102 td 0xc7f92000 sched_switch(c7f92000,0,104,84d8792,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c74baa80,c7f272b0,e9bdea9c,...) at mi_switch+0x11b sleepq_switch(c7f92000,0,c0ce693d,1a0,c7f92200,...) at sleepq_switch+0xcc sleepq_catch_signals(c74baa80,0,c7f92000,e9bdeae8,c087ed63,...) at sleepq_catch_signals+0x52 sleepq_timedwait_sig(c7b22190,0,c0cfead7,101,0,...) at sleepq_timedwait_sig+0x1c _cv_timedwait_sig(c7b22190,c74baa80,1388,c0e358c0,e9bdeb08,...) at _cv_timedwait_sig+0x1a3 svc_run_internal(c7f92168,14,c0cfec83,c0cfdc08,bfbfe9a7,...) at svc_run_internal+0x30d svc_run(c74baa80,0,1,a,c1ae7650,...) at svc_run+0x7b nfssvc_nfsd(bfbfe890,e9bdec0c,c,c7154100,e9bdec20,...) at nfssvc_nfsd+0xed nfssvc_nfsserver(c7f92000,e9bdecec,c7f272b0,0,c7f92000,...) at nfssvc_nfsserver+0x1ff nfssvc(c7f92000,e9bdecec,e9bded28,bfbffff4,c7f272b0,...) at nfssvc+0x63 syscallenter(c7f92000,e9bdece4,e9bdece4,0,e9bded28,...) at syscallenter+0x329 syscall(e9bded28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x280d980b, esp = 0xbfbfe84c, ebp = 0xbfbfeab8 --- Tracing command nfsd pid 897 tid 100113 td 0xc7f91b80 sched_switch(c7f91b80,0,104,7b6e564,1f,...) at sched_switch+0x2b6 mi_switch(104,0,c7b23340,c8085000,e9c35a18,...) at mi_switch+0x11b sleepq_switch(c7f91b80,0,c0ce693d,1a0,c7f91b80,...) at sleepq_switch+0xcc sleepq_catch_signals(c7b23340,0,e9c35a60,c087f20b,c7b23364,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7b23364,0,c0ce81e5,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c7b23364,c7b23350,58,c7154100,c7f91b80,...) at _cv_wait_sig+0x18b seltdwait(c7ef39d8,58,c7154100,c7f91b80,0,...) at seltdwait+0xd7 kern_select(c7f91b80,5,bfbfecd0,0,0,0,20,c8085000,c8085000) at kern_select+0x4ee select(c7f91b80,e9c35cec,e9c35d28,bfbffff4,c8085000,...) at select+0x66 syscallenter(c7f91b80,e9c35ce4,c0c0a8be,c7f91b80,e9c35d28,...) at syscallenter+0x329 syscall(e9c35d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x2817f553, esp = 0xbfbfeabc, ebp = 0xbfbfee08 --- Tracing command mountd pid 888 tid 100092 td 0xc74ccb80 sched_switch(c74ccb80,0,104,b76c5876,52,...) at sched_switch+0x2b6 mi_switch(104,0,c7b222c0,c7f8e2b0,e9bc0a18,...) at mi_switch+0x11b sleepq_switch(c74ccb80,0,c0ce693d,1a0,c74ccb80,...) at sleepq_switch+0xcc sleepq_catch_signals(c7b222c0,0,e9bc0a60,c087f20b,c7b222e4,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7b222e4,0,c0ce81e5,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c7b222e4,c7b222d0,58,c7154100,c74ccb80,...) at _cv_wait_sig+0x18b seltdwait(c7bcfd90,58,c7154100,c74ccb80,c095466a,...) at seltdwait+0xd7 kern_select(c74ccb80,9,bfbfed6c,0,0,0,20,c7f8e2b0,c7f8e2b0) at kern_select+0x4ee select(c74ccb80,e9bc0cec,e9bc0d28,281aa6fc,c7f8e2b0,...) at select+0x66 syscallenter(c74ccb80,e9bc0ce4,e9bc0ce4,0,e9bc0d28,...) at syscallenter+0x329 syscall(e9bc0d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x28193553, esp = 0xbfbfed3c, ebp = 0xbfbfee08 --- Tracing command rpcbind pid 791 tid 100096 td 0xc7f932e0 sched_switch(c7f932e0,0,104,fa56fa61,1bd,...) at sched_switch+0x2b6 mi_switch(104,0,c7b23540,c7f8d560,e9bcca6c,...) at mi_switch+0x11b sleepq_switch(c7f932e0,0,c0ce693d,1a0,c7f934e0,...) at sleepq_switch+0xcc sleepq_catch_signals(c7b23540,0,c7f932e0,e9bccab8,c087ed63,...) at sleepq_catch_signals+0x52 sleepq_timedwait_sig(c7b23564,0,c0ce81e5,101,0,...) at sleepq_timedwait_sig+0x1c _cv_timedwait_sig(c7b23564,c7b23550,7531,c7154100,c7f932e0,...) at _cv_timedwait_sig+0x1a3 seltdwait(e9bccc2c,e9bccc34,c7154100,c7f932e0,c090d4e0,...) at seltdwait+0xc1 poll(c7f932e0,e9bcccec,e9bccd28,2818f5a0,c7f8d560,...) at poll+0x33f syscallenter(c7f932e0,e9bccce4,e9bccce4,0,e9bccd28,...) at syscallenter+0x329 syscall(e9bccd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (209, FreeBSD ELF32, poll), eip = 0x28137eff, esp = 0xbfbfcbcc, ebp = 0xbfbfed98 --- Tracing command syslogd pid 765 tid 100097 td 0xc7f93000 sched_switch(c7f93000,0,104,bbadb102,1bd,...) at sched_switch+0x2b6 mi_switch(104,0,c7b226c0,c7f8d2b0,e9bcfa18,...) at mi_switch+0x11b sleepq_switch(c7f93000,0,c0ce693d,1a0,c7f93000,...) at sleepq_switch+0xcc sleepq_catch_signals(c7b226c0,0,e9bcfa60,c087f20b,c7b226e4,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7b226e4,0,c0ce81e5,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c7b226e4,c7b226d0,c0dcc620,c730be00,c7f9f428,...) at _cv_wait_sig+0x18b seltdwait(c7f9f428,58,c7154100,c7f93000,0,...) at seltdwait+0xd7 kern_select(c7f93000,9,2822c0a8,0,0,0,20,c7f8d2b0,c7f8d2b0) at kern_select+0x4ee select(c7f93000,e9bcfcec,e9bcfd28,2000,c7f8d2b0,...) at select+0x66 syscallenter(c7f93000,e9bcfce4,e9bcfce4,0,0,...) at syscallenter+0x329 syscall(e9bcfd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x28192553, esp = 0xbfbfe29c, ebp = 0xbfbfee08 --- Tracing command devd pid 617 tid 100082 td 0xc7b78b80 sched_switch(c7b78b80,0,104,4c8132c0,100,...) at sched_switch+0x2b6 mi_switch(104,0,c7b231c0,c7ab8ac0,e9b44a18,...) at mi_switch+0x11b sleepq_switch(c7b78b80,0,c0ce693d,1a0,c7b78b80,...) at sleepq_switch+0xcc sleepq_catch_signals(c7b231c0,0,e9b44a60,c087f20b,c7b231e4,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7b231e4,0,c0ce81e5,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c7b231e4,c7b231d0,58,c7154100,c7b78b80,...) at _cv_wait_sig+0x18b seltdwait(c7ef3e70,58,c7154100,c7b78b80,58,...) at seltdwait+0xd7 kern_select(c7b78b80,5,bfbfe990,0,0,0,20,c7ab8ac0,c7ab8ac0) at kern_select+0x4ee select(c7b78b80,e9b44cec,e9b44d28,28104010,c7ab8ac0,...) at select+0x66 syscallenter(c7b78b80,e9b44ce4,e9b44ce4,0,e9b44d28,...) at syscallenter+0x329 syscall(e9b44d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x8088143, esp = 0xbfbfe95c, ebp = 0xbfbfee38 --- Tracing command flowcleaner pid 21 tid 100076 td 0xc73ba8a0 sched_switch(c73ba8a0,0,104,bb955fb9,1be,...) at sched_switch+0x2b6 mi_switch(104,0,c73ba8a0,0,e7940c74,...) at mi_switch+0x11b sleepq_switch(c73ba8a0,0,c0ce693d,283,20,...) at sleepq_switch+0xcc sleepq_timedwait(c0e444f0,0,c0ced698,1,0,...) at sleepq_timedwait+0x3e _cv_timedwait(c0e444f0,c0e444f8,4e20,622,0,...) at _cv_timedwait+0x1a3 flowtable_cleaner(0,e7940d28,0,0,0,...) at flowtable_cleaner+0x244 fork_exit(c0972750,0,e7940d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7940d60, ebp = 0 --- Tracing command softdepflush pid 20 tid 100075 td 0xc73bab80 sched_switch(c73bab80,0,104,8fd4cfe1,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c73bab80,db184100,e793d9e8,...) at mi_switch+0x11b sleepq_switch(c73bab80,0,c0ce693d,260,50,...) at sleepq_switch+0xcc sleepq_wait(db184100,50,c0ceab21,4,0,...) at sleepq_wait+0x39 __lockmgr_args(db184100,81900,c834c0f4,c0ceab21,50,...) at __lockmgr_args+0x893 getblk(c834c000,16f940,0,4000,0,...) at getblk+0x1b1 breadn(c834c000,16f940,0,4000,0,...) at breadn+0x44 bread(c834c000,16f940,0,4000,0,...) at bread+0x4c ffs_blkfree_cg(5d238,0,4000,389,c0903867,...) at ffs_blkfree_cg+0x272 ffs_blkfree(c7f97100,c82b7000,c834c000,5d238,0,...) at ffs_blkfree+0x43 handle_workitem_freefrag(0,e793dc58,2,0,0,...) at handle_workitem_freefrag+0x51 process_worklist_item(c0e4fb5c,c73bab80,0,0,0,...) at process_worklist_item+0x241 softdep_process_worklist(c834e5a0,0,44,c0d01388,3e8,...) at softdep_process_worklist+0x8d softdep_flush(0,e793dd28,0,0,0,...) at softdep_flush+0x3b7 fork_exit(c0b086a0,0,e793dd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe793dd60, ebp = 0 --- Tracing command syncer pid 19 tid 100074 td 0xc74ca000 sched_switch(c74ca000,0,104,8fd17882,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c74ca000,0,e793ac40,...) at mi_switch+0x11b sleepq_switch(c74ca000,0,c0ce693d,283,3ed,...) at sleepq_switch+0xcc sleepq_timedwait(c0e442f8,0,c0cecafb,1,0,...) at sleepq_timedwait+0x3e _cv_timedwait(c0e442f8,c0e442e4,3e8,764,4e20,...) at _cv_timedwait+0x1a3 sched_sync(0,e793ad28,0,0,0,...) at sched_sync+0x59a fork_exit(c095b110,0,e793ad28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe793ad60, ebp = 0 --- Tracing command vnlru pid 18 tid 100073 td 0xc74ca2e0 sched_switch(c74ca2e0,0,104,b31968b3,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c74ca2e0,0,e7937c2c,...) at mi_switch+0x11b sleepq_switch(c74ca2e0,0,c0ce693d,283,250,...) at sleepq_switch+0xcc sleepq_timedwait(c7ab72b0,50,c0ceccf7,0,0,...) at sleepq_timedwait+0x3e _sleep(c7ab72b0,c0e442b8,250,c0ceccf7,3e8,...) at _sleep+0x272 vnlru_proc(0,e7937d28,0,0,0,...) at vnlru_proc+0x10f fork_exit(c09595b0,0,e7937d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7937d60, ebp = 0 --- Tracing command bufdaemon pid 17 tid 100072 td 0xc74ca5c0 sched_switch(c74ca5c0,0,104,9f3a93b3,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c74ca5c0,0,e7934c78,...) at mi_switch+0x11b sleepq_switch(c74ca5c0,0,c0ce693d,283,44,...) at sleepq_switch+0xcc sleepq_timedwait(c0e44028,44,c0cea977,0,0,...) at sleepq_timedwait+0x3e _sleep(c0e44028,c0e4402c,44,c0cea977,3e8,...) at _sleep+0x272 buf_daemon(0,e7934d28,0,0,0,...) at buf_daemon+0x186 fork_exit(c093edc0,0,e7934d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7934d60, ebp = 0 --- Tracing command pagezero pid 16 tid 100071 td 0xc74ca8a0 sched_switch(c74ca8a0,0,104,d2645b7b,194,...) at sched_switch+0x2b6 mi_switch(104,0,c74ca8a0,0,e7931c80,...) at mi_switch+0x11b sleepq_switch(c74ca8a0,0,c0ce693d,283,0,...) at sleepq_switch+0xcc sleepq_timedwait(c0e50874,0,c0d04e12,0,0,...) at sleepq_timedwait+0x3e _sleep(c0e50874,c0e50430,0,c0d04e12,493e0,...) at _sleep+0x272 vm_pagezero(0,e7931d28,0,0,0,...) at vm_pagezero+0xd2 fork_exit(c0b4b8b0,0,e7931d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7931d60, ebp = 0 --- Tracing command vmdaemon pid 15 tid 100070 td 0xc74cab80 sched_switch(c74cab80,0,104,9897955d,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c74cab80,0,e792ec24,...) at mi_switch+0x11b sleepq_switch(c74cab80,0,c0ce693d,260,68,...) at sleepq_switch+0xcc sleepq_wait(c0e5049c,68,c0cea977,0,0,...) at sleepq_wait+0x39 _sleep(c0e5049c,c0e504a0,68,c0cea977,0,...) at _sleep+0x2a2 vm_daemon(0,e792ed28,0,0,0,...) at vm_daemon+0x80 fork_exit(c0b482d0,0,e792ed28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe792ed60, ebp = 0 --- Tracing command pagedaemon pid 9 tid 100069 td 0xc74cb000 sched_switch(c74cb000,0,104,b5c7c1e7,1be,...) at sched_switch+0x2b6 mi_switch(104,0,c74cb000,0,e792bbb0,...) at mi_switch+0x11b sleepq_switch(c74cb000,0,c0ce693d,283,44,...) at sleepq_switch+0xcc sleepq_timedwait(c0e50464,44,c0cea977,0,0,...) at sleepq_timedwait+0x3e _sleep(c0e50464,c0e50430,44,c0cea977,1388,...) at _sleep+0x272 vm_pageout(0,e792bd28,0,0,0,...) at vm_pageout+0x2e3 fork_exit(c0b46d30,0,e792bd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe792bd60, ebp = 0 --- Tracing command xpt_thrd pid 8 tid 100068 td 0xc74cb2e0 sched_switch(c74cb2e0,0,104,879e0693,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c74cb2e0,0,e7928c70,...) at mi_switch+0x11b sleepq_switch(c74cb2e0,0,c0ce693d,260,4c,...) at sleepq_switch+0xcc sleepq_wait(c0dfccd4,4c,c0c82789,0,0,...) at sleepq_wait+0x39 _sleep(c0dfccd4,c0dfcd18,4c,c0c82789,0,...) at _sleep+0x2a2 xpt_scanner_thread(0,e7928d28,0,0,0,...) at xpt_scanner_thread+0x4f fork_exit(c04874b0,0,e7928d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7928d60, ebp = 0 --- Tracing command sctp_iterator pid 7 tid 100067 td 0xc74cb5c0 sched_switch(c74cb5c0,0,104,7d77bfbe,1e,...) at sched_switch+0x2b6 mi_switch(104,0,c74cb5c0,0,e7925c7c,...) at mi_switch+0x11b sleepq_switch(c74cb5c0,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c0e45d38,0,c0cf6893,0,0,...) at sleepq_wait+0x39 _sleep(c0e45d38,c0e45d00,0,c0cf6893,0,...) at _sleep+0x2a2 sctp_iterator_thread(0,e7925d28,0,0,0,...) at sctp_iterator_thread+0x7c fork_exit(c09eba30,0,e7925d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7925d60, ebp = 0 --- Tracing command fdc0 pid 6 tid 100064 td 0xc74cc000 sched_switch(c74cc000,0,104,412b99c0,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c74cc000,0,e791cbcc,...) at mi_switch+0x11b sleepq_switch(c74cc000,0,c0ce693d,283,4c,...) at sleepq_switch+0xcc sleepq_timedwait(c73d743c,4c,c0cdbf72,0,0,...) at sleepq_timedwait+0x3e _sleep(c73d743c,c73d74f0,4c,c0cdbf72,3e8,...) at _sleep+0x272 fdc_thread(c73d7400,e791cd28,0,0,0,...) at fdc_thread+0x3ed fork_exit(c0bcae50,c73d7400,e791cd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe791cd60, ebp = 0 --- Tracing command fw0_probe pid 5 tid 100062 td 0xc73a85c0 sched_switch(c73a85c0,0,104,a95edfa8,1b,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c7157ac0,e790c78c,...) at mi_switch+0x11b sleepq_switch(c73a85c0,0,c0ce693d,1a0,c73a85c0,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,e790c7e0,c08cf4ac,c73de000,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c73de000,5c,c0cdbf72,100,0,...) at sleepq_wait_sig+0x18 _sleep(c73de000,c73e2488,15c,c0cdbf72,0,...) at _sleep+0x28c fw_bus_probe_thread(c73de000,e790cd28,0,0,0,...) at fw_bus_probe_thread+0xb45 fork_exit(c067ee70,c73de000,e790cd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe790cd60, ebp = 0 --- Tracing command usb pid 14 tid 100058 td 0xc73b82e0 sched_switch(c73b82e0,0,104,87990ea7,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73b82e0,0,e78f9c80,...) at mi_switch+0x11b sleepq_switch(c73b82e0,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73bfd34,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c73bfd34,c73bfdd4,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c73bfd2c,e78f9d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73bfd2c,e78f9d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78f9d60, ebp = 0 --- Tracing command usb pid 14 tid 100057 td 0xc73b85c0 sched_switch(c73b85c0,0,104,8707a7ac,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c73b85c0,0,e78f6c80,...) at mi_switch+0x11b sleepq_switch(c73b85c0,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73bfd04,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c73bfd04,c73bfdd4,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c73bfcfc,e78f6d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73bfcfc,e78f6d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78f6d60, ebp = 0 --- Tracing command usb pid 14 tid 100056 td 0xc73b88a0 sched_switch(c73b88a0,0,104,87335c4c,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73b88a0,0,e78f3c80,...) at mi_switch+0x11b sleepq_switch(c73b88a0,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73bfcd4,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c73bfcd4,c73bfdd4,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c73bfccc,e78f3d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73bfccc,e78f3d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78f3d60, ebp = 0 --- Tracing command usb pid 14 tid 100055 td 0xc73b8b80 sched_switch(c73b8b80,0,104,8733523c,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73b8b80,0,e78f0c80,...) at mi_switch+0x11b sleepq_switch(c73b8b80,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73bfca4,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c73bfca4,c73bfdd4,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c73bfc9c,e78f0d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73bfc9c,e78f0d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78f0d60, ebp = 0 --- Tracing command usb pid 14 tid 100053 td 0xc73ba2e0 sched_switch(c73ba2e0,0,104,8733466e,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73ba2e0,0,e78e7c80,...) at mi_switch+0x11b sleepq_switch(c73ba2e0,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73b2b5c,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c73b2b5c,c73b2bfc,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c73b2b54,e78e7d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73b2b54,e78e7d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78e7d60, ebp = 0 --- Tracing command usb pid 14 tid 100052 td 0xc73ba5c0 sched_switch(c73ba5c0,0,104,7ea91b6d,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c73ba5c0,0,e78e4c80,...) at mi_switch+0x11b sleepq_switch(c73ba5c0,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73b2b2c,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c73b2b2c,c73b2bfc,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c73b2b24,e78e4d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73b2b24,e78e4d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78e4d60, ebp = 0 --- Tracing command usb pid 14 tid 100051 td 0xc73a6000 sched_switch(c73a6000,0,104,86d06928,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73a6000,0,e78e1c80,...) at mi_switch+0x11b sleepq_switch(c73a6000,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73b2afc,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c73b2afc,c73b2bfc,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c73b2af4,e78e1d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73b2af4,e78e1d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78e1d60, ebp = 0 --- Tracing command usb pid 14 tid 100050 td 0xc73a62e0 sched_switch(c73a62e0,0,104,86d05e9e,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73a62e0,0,e78dec80,...) at mi_switch+0x11b sleepq_switch(c73a62e0,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73b2acc,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c73b2acc,c73b2bfc,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c73b2ac4,e78ded28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73b2ac4,e78ded28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78ded60, ebp = 0 --- Tracing command usb pid 14 tid 100049 td 0xc73a65c0 sched_switch(c73a65c0,0,104,86d05417,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73a65c0,0,e78dac80,...) at mi_switch+0x11b sleepq_switch(c73a65c0,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73aeb5c,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c73aeb5c,c73aebfc,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c73aeb54,e78dad28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73aeb54,e78dad28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78dad60, ebp = 0 --- Tracing command usb pid 14 tid 100048 td 0xc73a68a0 sched_switch(c73a68a0,0,104,7e042adc,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c73a68a0,0,e78d7c80,...) at mi_switch+0x11b sleepq_switch(c73a68a0,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73aeb2c,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c73aeb2c,c73aebfc,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c73aeb24,e78d7d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73aeb24,e78d7d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78d7d60, ebp = 0 --- Tracing command usb pid 14 tid 100047 td 0xc73a6b80 sched_switch(c73a6b80,0,104,866d35d3,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73a6b80,0,e78d4c80,...) at mi_switch+0x11b sleepq_switch(c73a6b80,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73aeafc,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c73aeafc,c73aebfc,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c73aeaf4,e78d4d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73aeaf4,e78d4d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78d4d60, ebp = 0 --- Tracing command usb pid 14 tid 100046 td 0xc73a7000 sched_switch(c73a7000,0,104,866d2a8a,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73a7000,0,e78d1c80,...) at mi_switch+0x11b sleepq_switch(c73a7000,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73aeacc,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c73aeacc,c73aebfc,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c73aeac4,e78d1d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73aeac4,e78d1d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78d1d60, ebp = 0 --- Tracing command usb pid 14 tid 100045 td 0xc73a72e0 sched_switch(c73a72e0,0,104,866d1e65,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73a72e0,0,e78cdc80,...) at mi_switch+0x11b sleepq_switch(c73a72e0,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73a1b5c,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c73a1b5c,c73a1bfc,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c73a1b54,e78cdd28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73a1b54,e78cdd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78cdd60, ebp = 0 --- Tracing command usb pid 14 tid 100044 td 0xc73a75c0 sched_switch(c73a75c0,0,104,7db1926f,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c73a75c0,0,e78cac80,...) at mi_switch+0x11b sleepq_switch(c73a75c0,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73a1b2c,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c73a1b2c,c73a1bfc,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c73a1b24,e78cad28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73a1b24,e78cad28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78cad60, ebp = 0 --- Tracing command usb pid 14 tid 100043 td 0xc73a78a0 sched_switch(c73a78a0,0,104,860ac183,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73a78a0,0,e78c7c80,...) at mi_switch+0x11b sleepq_switch(c73a78a0,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73a1afc,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c73a1afc,c73a1bfc,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c73a1af4,e78c7d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73a1af4,e78c7d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78c7d60, ebp = 0 --- Tracing command usb pid 14 tid 100042 td 0xc73a7b80 sched_switch(c73a7b80,0,104,860ab77e,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73a7b80,0,e78c4c80,...) at mi_switch+0x11b sleepq_switch(c73a7b80,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73a1acc,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c73a1acc,c73a1bfc,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c73a1ac4,e78c4d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73a1ac4,e78c4d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78c4d60, ebp = 0 --- Tracing command usb pid 14 tid 100040 td 0xc7393000 sched_switch(c7393000,0,104,860aabbd,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c7393000,0,e78bdc80,...) at mi_switch+0x11b sleepq_switch(c7393000,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c739eb5c,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c739eb5c,c739ebfc,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c739eb54,e78bdd28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c739eb54,e78bdd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78bdd60, ebp = 0 --- Tracing command usb pid 14 tid 100039 td 0xc73932e0 sched_switch(c73932e0,0,104,7d7addb0,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c73932e0,0,e78bac80,...) at mi_switch+0x11b sleepq_switch(c73932e0,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c739eb2c,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c739eb2c,c739ebfc,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c739eb24,e78bad28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c739eb24,e78bad28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78bad60, ebp = 0 --- Tracing command usb pid 14 tid 100038 td 0xc73935c0 sched_switch(c73935c0,0,104,85a78bf2,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73935c0,0,e78b7c80,...) at mi_switch+0x11b sleepq_switch(c73935c0,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c739eafc,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c739eafc,c739ebfc,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c739eaf4,e78b7d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c739eaf4,e78b7d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78b7d60, ebp = 0 --- Tracing command usb pid 14 tid 100037 td 0xc73938a0 sched_switch(c73938a0,0,104,85a7816e,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73938a0,0,e78b4c80,...) at mi_switch+0x11b sleepq_switch(c73938a0,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c739eacc,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c739eacc,c739ebfc,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c739eac4,e78b4d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c739eac4,e78b4d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78b4d60, ebp = 0 --- Tracing command usb pid 14 tid 100035 td 0xc7395000 sched_switch(c7395000,0,104,85a7752c,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c7395000,0,e78adc80,...) at mi_switch+0x11b sleepq_switch(c7395000,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c738db5c,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c738db5c,c738dbfc,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c738db54,e78add28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c738db54,e78add28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78add60, ebp = 0 --- Tracing command usb pid 14 tid 100034 td 0xc73952e0 sched_switch(c73952e0,0,104,7d5f760b,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c73952e0,0,e78aac80,...) at mi_switch+0x11b sleepq_switch(c73952e0,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c738db2c,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c738db2c,c738dbfc,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c738db24,e78aad28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c738db24,e78aad28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78aad60, ebp = 0 --- Tracing command usb pid 14 tid 100033 td 0xc73955c0 sched_switch(c73955c0,0,104,853dc54a,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73955c0,0,e78a7c80,...) at mi_switch+0x11b sleepq_switch(c73955c0,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c738dafc,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c738dafc,c738dbfc,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c738daf4,e78a7d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c738daf4,e78a7d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78a7d60, ebp = 0 --- Tracing command usb pid 14 tid 100032 td 0xc73958a0 sched_switch(c73958a0,0,104,853db8eb,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73958a0,0,e78a4c80,...) at mi_switch+0x11b sleepq_switch(c73958a0,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c738dacc,0,c0cdbf72,1,0,...) at sleepq_wait+0x39 _cv_wait(c738dacc,c738dbfc,c0cd52cb,68,0,...) at _cv_wait+0x18b usb_process(c738dac4,e78a4d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c738dac4,e78a4d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78a4d60, ebp = 0 --- Tracing command yarrow pid 13 tid 100017 td 0xc725f5c0 sched_switch(c725f5c0,0,104,b8ce884b,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c725f5c0,0,c6f2fc48,...) at mi_switch+0x11b sleepq_switch(c725f5c0,0,c0ce693d,283,0,...) at sleepq_switch+0xcc sleepq_timedwait(c0e305e4,0,c0cdbf72,2,0,...) at sleepq_timedwait+0x3e _sleep(c0e305e4,0,0,c0cdbf72,64,...) at _sleep+0x272 pause(c0cdbf72,64,0,0,0,...) at pause+0x30 random_kthread(0,c6f2fd28,0,0,0,...) at random_kthread+0x225 fork_exit(c0758a20,0,c6f2fd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f2fd60, ebp = 0 --- Tracing command g_down pid 4 tid 100015 td 0xc725fb80 sched_switch(c725fb80,0,104,8fd97359,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c725fb80,0,c6f29c30,...) at mi_switch+0x11b sleepq_switch(c725fb80,0,c0ce693d,260,24c,...) at sleepq_switch+0xcc sleepq_wait(c0e2e2e4,4c,c0cdbf72,0,0,...) at sleepq_wait+0x39 _sleep(c0e2e2e4,c0e2e248,24c,c0cdbf72,0,...) at _sleep+0x2a2 g_io_schedule_down(c725fb80,4c,c0cdcb11,72,0,...) at g_io_schedule_down+0x58 g_down_procbody(0,c6f29d28,0,0,0,...) at g_down_procbody+0x68 fork_exit(c085f2a0,0,c6f29d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f29d60, ebp = 0 --- Tracing command g_up pid 3 tid 100014 td 0xc7260000 sched_switch(c7260000,0,104,8fe265d9,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c7260000,0,c6f26c58,...) at mi_switch+0x11b sleepq_switch(c7260000,0,c0ce693d,260,24c,...) at sleepq_switch+0xcc sleepq_wait(c0e2e2e0,4c,c0cdbf72,0,0,...) at sleepq_wait+0x39 _sleep(c0e2e2e0,c0e2e268,24c,c0cdbf72,0,...) at _sleep+0x2a2 g_io_schedule_up(c7260000,4c,c0cdcb11,5b,0,...) at g_io_schedule_up+0xae g_up_procbody(0,c6f26d28,0,0,0,...) at g_up_procbody+0x68 fork_exit(c085f310,0,c6f26d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f26d60, ebp = 0 --- Tracing command g_event pid 2 tid 100013 td 0xc715a2e0 sched_switch(c715a2e0,0,104,b754ab94,52,...) at sched_switch+0x2b6 mi_switch(104,0,c715a2e0,0,c6f23c4c,...) at mi_switch+0x11b sleepq_switch(c715a2e0,0,c0ce693d,260,24c,...) at sleepq_switch+0xcc sleepq_wait(c0e2e2d8,4c,c0cdbf72,0,0,...) at sleepq_wait+0x39 _sleep(c0e2e2d8,c0e2e228,24c,c0cdbf72,0,...) at _sleep+0x2a2 g_run_events(c715a2e0,4c,c0cdcb11,89) at g_run_events+0x49b g_event_procbody(0,c6f23d28,0,0,0,...) at g_event_procbody+0x64 fork_exit(c085f380,0,c6f23d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f23d60, ebp = 0 --- Tracing command intr pid 12 tid 100066 td 0xc74cb8a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100065 td 0xc74cbb80 sched_switch(c74cbb80,0,109,853c10ca,1b,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce044e,529,c7154070,...) at mi_switch+0x11b ithread_loop(c74cd310,e791fd28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c74cd310,e791fd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe791fd60, ebp = 0 --- Tracing command intr pid 12 tid 100063 td 0xc73a82e0 sched_switch(c73a82e0,0,109,675a20f5,52,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce044e,529,c74ba0f0,...) at mi_switch+0x11b ithread_loop(c74c0820,e790fd28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c74c0820,e790fd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe790fd60, ebp = 0 --- Tracing command intr pid 12 tid 100060 td 0xc73a8b80 sched_switch(c73a8b80,0,109,85335468,1b,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce044e,529,c719c0f0,...) at mi_switch+0x11b ithread_loop(c73b7610,e7903d28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c73b7610,e7903d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7903d60, ebp = 0 --- Tracing command intr pid 12 tid 100059 td 0xc73b8000 sched_switch(c73b8000,0,109,91f52606,1b,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce044e,529,c719c3f0,...) at mi_switch+0x11b ithread_loop(c7381a80,e78ffd28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c7381a80,e78ffd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78ffd60, ebp = 0 --- Tracing command intr pid 12 tid 100054 td 0xc73ba000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100041 td 0xc73a8000 sched_switch(c73a8000,0,109,ac3ba78f,1b,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce044e,529,c719c1f0,...) at mi_switch+0x11b ithread_loop(c7381420,e78c1d28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c7381420,e78c1d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78c1d60, ebp = 0 --- Tracing command intr pid 12 tid 100036 td 0xc7393b80 sched_switch(c7393b80,0,109,abb3395d,1b,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce044e,529,c719c270,...) at mi_switch+0x11b ithread_loop(c73817f0,e78b1d28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c73817f0,e78b1d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78b1d60, ebp = 0 --- Tracing command intr pid 12 tid 100031 td 0xc7395b80 sched_switch(c7395b80,0,109,ab320150,1b,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce044e,529,c719c2f0,...) at mi_switch+0x11b ithread_loop(c737c5d0,e78a1d28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c737c5d0,e78a1d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78a1d60, ebp = 0 --- Tracing command intr pid 12 tid 100030 td 0xc72602e0 sched_switch(c72602e0,0,109,8a8219e9,1c0,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce044e,529,c719bef0,...) at mi_switch+0x11b ithread_loop(c736da70,e7792d28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c736da70,e7792d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7792d60, ebp = 0 --- Tracing command intr pid 12 tid 100029 td 0xc72605c0 sched_switch(c72605c0,0,109,5b7848ca,1c0,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce044e,529,c736c870,...) at mi_switch+0x11b ithread_loop(c736dbc0,e778ed28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c736dbc0,e778ed28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe778ed60, ebp = 0 --- Tracing command intr pid 12 tid 100028 td 0xc72608a0 sched_switch(c72608a0,0,109,853caa81,1b,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce044e,529,c7153b70,...) at mi_switch+0x11b ithread_loop(c71554f0,c6f77d28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c71554f0,c6f77d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f77d60, ebp = 0 --- Tracing command intr pid 12 tid 100026 td 0xc7330000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100021 td 0xc719d8a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100019 td 0xc725f000 sched_switch(c725f000,0,109,53f1c0af,1f,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce044e,529,c732f970,...) at mi_switch+0x11b ithread_loop(c7128930,c6f35d28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c7128930,c6f35d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f35d60, ebp = 0 --- Tracing command intr pid 12 tid 100018 td 0xc725f2e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100012 td 0xc715a5c0 sched_switch(c715a5c0,0,109,ae2ddaa8,1c0,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce044e,529,c719bb70,...) at mi_switch+0x11b ithread_loop(c7155040,c6f20d28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c7155040,c6f20d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f20d60, ebp = 0 --- Tracing command intr pid 12 tid 100011 td 0xc715a8a0 sched_switch(c715a8a0,0,109,b3197cde,1c0,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce044e,529,c719bbf0,...) at mi_switch+0x11b ithread_loop(c7155050,c6f1dd28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c7155050,c6f1dd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f1dd60, ebp = 0 --- Tracing command intr pid 12 tid 100010 td 0xc715ab80 sched_switch(c715ab80,0,109,b8ce6599,1c0,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce044e,529,c719bc70,...) at mi_switch+0x11b ithread_loop(c7155060,c6f1ad28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c7155060,c6f1ad28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f1ad60, ebp = 0 --- Tracing command intr pid 12 tid 100009 td 0xc719d000 cpustop_handler(2,c6f17b68,c0c0a216,c73d0900,8f,...) at cpustop_handler+0x34 ipi_nmi_handler(c73d0900,8f,100000,c6f17b14,c7156560,...) at ipi_nmi_handler+0x2f trap(c6f17b74) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc04faba3, esp = 0xc6f17bb4, ebp = 0xc6f17bb4 --- acpi_timer_read(c0d27c00,c0e307c0,c6f17bfc,c08d1f7d,c0d27c00,...) at acpi_timer_read+0x13 acpi_timer_get_timecount_safe(c0d27c00,c08d9621,c719d000,c0e307c0,c0e4fb80,...) at acpi_timer_get_timecount_safe+0xa binuptime(c6f17c5c,1,c0e34bc0,c6f17c24,c08e6f30,...) at binuptime+0x3d softclock(c0e307c0,c719d000,0,109,b8a94952,...) at softclock+0x24a intr_event_execute_handlers(c7156560,c719bc80,c0ce044e,529,c719bcf0,...) at intr_event_execute_handlers+0x13b ithread_loop(c7155070,c6f17d28,0,0,0,...) at ithread_loop+0x6b fork_exit(c089ddb0,c7155070,c6f17d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f17d60, ebp = 0 --- Tracing command intr pid 12 tid 100008 td 0xc719d2e0 sched_switch(c719d2e0,0,109,53f18a71,1f,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce044e,529,c719bd70,...) at mi_switch+0x11b ithread_loop(c7155080,c6f14d28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c7155080,c6f14d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f14d60, ebp = 0 --- Tracing command intr pid 12 tid 100007 td 0xc719d5c0 fork_trampoline() at fork_trampoline Tracing command idle pid 11 tid 100006 td 0xc7158000 sched_switch(c7158000,0,60c,8fe2634d,1c0,...) at sched_switch+0x2b6 mi_switch(60c,0,c0ce493e,815,0,...) at mi_switch+0x11b sched_preempt(c7158000,0,c7260000,c6f0dc98,c0bf0dbf,...) at sched_preempt+0x7c ipi_bitmap_handler(8,28,28,18,c0e34540,...) at ipi_bitmap_handler+0x34 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x2f --- interrupt, eip = 0xc08b6a65, esp = 0xc6f0dc84, ebp = 0xc6f0dc98 --- _thread_lock_flags(c7158000,0,c0ce493e,a06,c7158000,...) at _thread_lock_flags+0xa5 sched_idletd(0,c6f0dd28,0,0,0,...) at sched_idletd+0x262 fork_exit(c08e83d0,0,c6f0dd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f0dd60, ebp = 0 --- Tracing command idle pid 11 tid 100005 td 0xc71582e0 sched_switch(c71582e0,0,60c,bbab424f,1c0,...) at sched_switch+0x2b6 mi_switch(60c,0,c0ce493e,815,1,...) at mi_switch+0x11b sched_preempt(c71582e0,1,0,c6f0ac50,c0bf0dbf,...) at sched_preempt+0x7c ipi_bitmap_handler(8,28,28,c726f218,c726f200,...) at ipi_bitmap_handler+0x34 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x2f --- interrupt, eip = 0xc0be88a5, esp = 0xc6f0ac50, ebp = 0xc6f0ac50 --- acpi_cpu_c1(c6f0ac6c,c0dfcf50,1,ffffffff,c6f0ac98,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(c0e34bc0,0,c6f0ac98,c0bfc68b,0,...) at acpi_cpu_idle+0xe9 cpu_idle_amdc1e(0,c6f0ace0,c08e8601,0,c6f0acbc,...) at cpu_idle_amdc1e+0x56 cpu_idle(0,c6f0acbc,c0ce493e,a06,c71582e0,...) at cpu_idle+0x1b sched_idletd(0,c6f0ad28,0,0,0,...) at sched_idletd+0x231 fork_exit(c08e83d0,0,c6f0ad28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f0ad60, ebp = 0 --- Tracing command idle pid 11 tid 100004 td 0xc71585c0 cpustop_handler(4,c6f07c24,c0c0a216,46,c715a8a0,...) at cpustop_handler+0x34 ipi_nmi_handler(46,c715a8a0,c6f07bcc,c089c601,c7156810,...) at ipi_nmi_handler+0x2f trap(c6f07c30) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc08e8382, esp = 0xc6f07c70, ebp = 0xc6f07c98 --- cpu_search_highest(c0e42dd0,c6f07cbc,c0ce493e,a06,c71585c0,...) at cpu_search_highest+0xe2 sched_idletd(0,c6f07d28,0,0,0,...) at sched_idletd+0xca fork_exit(c08e83d0,0,c6f07d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f07d60, ebp = 0 --- Tracing command idle pid 11 tid 100003 td 0xc71588a0 cpustop_handler(8,c6f04c50,c0c0a216,c71588a0,c6f04bd4,...) at cpustop_handler+0x34 ipi_nmi_handler(c71588a0,c6f04bd4,46,c0e30874,c7156810,...) at ipi_nmi_handler+0x2f trap(c6f04c5c) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc08cf0ba, esp = 0xc6f04c9c, ebp = 0xc6f04ce0 --- mi_switch(0,c6f04d28,0,0,0,...) at mi_switch+0x16a fork_exit(c08e83d0,0,c6f04d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f04d60, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc7158b80 sched_switch(c7158b80,0,104,a2efcff4,21,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c7156ac0,c6f00b50,...) at mi_switch+0x11b sleepq_switch(c7158b80,0,c0ce693d,1a0,c7158b80,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,c6f00ba4,c08cf4ac,c7156ac0,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7156ac0,5c,c0ce72e1,100,0,...) at sleepq_wait_sig+0x18 _sleep(c7156ac0,c7156b48,15c,c0ce72e1,0,...) at _sleep+0x28c kern_wait(c7158b80,ffffffff,c6f00c44,0,0,...) at kern_wait+0xfa1 wait4(c7158b80,c6f00cec,c6f00d28,2811afdc,c7156ac0,...) at wait4+0x3b syscallenter(c7158b80,c6f00ce4,c0c0a8be,c7158b80,c6f00d28,...) at syscallenter+0x329 syscall(c6f00d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x8054d7b, esp = 0xbfbfe8fc, ebp = 0xbfbfe918 --- Tracing command audit pid 10 tid 100001 td 0xc715a000 sched_switch(c715a000,0,104,853d050c,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c715a000,0,c6efdc58,...) at mi_switch+0x11b sleepq_switch(c715a000,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c0e4f4c0,0,c0cfeec3,1,0,...) at sleepq_wait+0x39 _cv_wait(c0e4f4c0,c0e4f4a4,0,0,0,...) at _cv_wait+0x18b audit_worker(0,c6efdd28,0,0,0,...) at audit_worker+0x6f fork_exit(c0add620,0,c6efdd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6efdd60, ebp = 0 --- Tracing command kernel pid 0 tid 100061 td 0xc73a88a0 sched_switch(c73a88a0,0,104,b296f1bc,1c0,...) at sched_switch+0x2b6 mi_switch(104,0,c73a88a0,0,e7909c88,...) at mi_switch+0x11b sleepq_switch(c73a88a0,0,c0ce693d,260,c73dd580,...) at sleepq_switch+0xcc sleepq_wait(c73dd580,0,c0cdbf72,0,0,...) at sleepq_wait+0x39 msleep_spin(c73dd580,c73dd598,c0cdbf72,0,e7909ce0,...) at msleep_spin+0x181 taskqueue_thread_loop(c73e249c,e7909d28,0,0,0,...) at taskqueue_thread_loop+0xe4 fork_exit(c0901190,c73e249c,e7909d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7909d60, ebp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xc7260b80 sched_switch(c7260b80,0,104,87bd3426,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c7260b80,0,c6f4dc78,...) at mi_switch+0x11b sleepq_switch(c7260b80,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c732c9c0,0,c0cdbf72,0,0,...) at sleepq_wait+0x39 _sleep(c732c9c0,c732c9d8,0,c0cdbf72,0,...) at _sleep+0x2a2 taskqueue_thread_loop(c0e42fc8,c6f4dd28,0,0,0,...) at taskqueue_thread_loop+0x10a fork_exit(c0901190,c0e42fc8,c6f4dd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f4dd60, ebp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xc73302e0 sched_switch(c73302e0,0,104,87bccf7a,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73302e0,0,c6f47c78,...) at mi_switch+0x11b sleepq_switch(c73302e0,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c732ca80,0,c0cdbf72,0,0,...) at sleepq_wait+0x39 _sleep(c732ca80,c732ca98,0,c0cdbf72,0,...) at _sleep+0x2a2 taskqueue_thread_loop(c0e4faa4,c6f47d28,0,0,0,...) at taskqueue_thread_loop+0x10a fork_exit(c0901190,c0e4faa4,c6f47d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f47d60, ebp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xc73305c0 sched_switch(c73305c0,0,104,87bcc3b4,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73305c0,0,c6f44c88,...) at mi_switch+0x11b sleepq_switch(c73305c0,0,c0ce693d,260,c732cc00,...) at sleepq_switch+0xcc sleepq_wait(c732cc00,0,c0cdbf72,0,0,...) at sleepq_wait+0x39 msleep_spin(c732cc00,c732cc18,c0cdbf72,0,c6f44ce0,...) at msleep_spin+0x181 taskqueue_thread_loop(c0dffba0,c6f44d28,0,0,0,...) at taskqueue_thread_loop+0xe4 fork_exit(c0901190,c0dffba0,c6f44d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f44d60, ebp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xc73308a0 sched_switch(c73308a0,0,104,87bcb8e1,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73308a0,0,c6f41c88,...) at mi_switch+0x11b sleepq_switch(c73308a0,0,c0ce693d,260,c732cc00,...) at sleepq_switch+0xcc sleepq_wait(c732cc00,0,c0cdbf72,0,0,...) at sleepq_wait+0x39 msleep_spin(c732cc00,c732cc18,c0cdbf72,0,c6f41ce0,...) at msleep_spin+0x181 taskqueue_thread_loop(c0dffba0,c6f41d28,0,0,0,...) at taskqueue_thread_loop+0xe4 fork_exit(c0901190,c0dffba0,c6f41d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f41d60, ebp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xc7330b80 sched_switch(c7330b80,0,104,87bcae73,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c7330b80,0,c6f3ec88,...) at mi_switch+0x11b sleepq_switch(c7330b80,0,c0ce693d,260,c732cc00,...) at sleepq_switch+0xcc sleepq_wait(c732cc00,0,c0cdbf72,0,0,...) at sleepq_wait+0x39 msleep_spin(c732cc00,c732cc18,c0cdbf72,0,c6f3ece0,...) at msleep_spin+0x181 taskqueue_thread_loop(c0dffba0,c6f3ed28,0,0,0,...) at taskqueue_thread_loop+0xe4 fork_exit(c0901190,c0dffba0,c6f3ed28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f3ed60, ebp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xc719db80 sched_switch(c719db80,0,104,87ba7106,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c719db80,0,c6f38c78,...) at mi_switch+0x11b sleepq_switch(c719db80,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c732cd40,0,c0cdbf72,0,0,...) at sleepq_wait+0x39 _sleep(c732cd40,c732cd58,0,c0cdbf72,0,...) at _sleep+0x2a2 taskqueue_thread_loop(c0e2ed04,c6f38d28,0,0,0,...) at taskqueue_thread_loop+0x10a fork_exit(c0901190,c0e2ed04,c6f38d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f38d60, ebp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xc725f8a0 sched_switch(c725f8a0,0,104,66512c77,1d,...) at sched_switch+0x2b6 mi_switch(104,0,c725f8a0,0,c6f2cc78,...) at mi_switch+0x11b sleepq_switch(c725f8a0,0,c0ce693d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c7137880,0,c0cdbf72,0,0,...) at sleepq_wait+0x39 _sleep(c7137880,c7137898,0,c0cdbf72,0,...) at _sleep+0x2a2 taskqueue_thread_loop(c0e41a60,c6f2cd28,0,0,0,...) at taskqueue_thread_loop+0x10a fork_exit(c0901190,c0e41a60,c6f2cd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f2cd60, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc0e2e670 sched_switch(c0e2e670,0,104,ba49fc23,1be,...) at sched_switch+0x2b6 mi_switch(104,0,c0e2e670,0,c1420cd8,...) at mi_switch+0x11b sleepq_switch(c0e2e670,0,c0ce693d,283,44,...) at sleepq_switch+0xcc sleepq_timedwait(c0e2e3c0,44,c0ce4ddb,0,0,...) at sleepq_timedwait+0x3e _sleep(c0e2e3c0,0,44,c0ce4ddb,2710,...) at _sleep+0x272 scheduler(0,141ec00,141ec00,141e000,1425000,...) at scheduler+0x2f0 mi_startup() at mi_startup+0xac begin() at begin+0x2c db:0:allt> call doadump Physical memory: 3439 MB Dumping 255 MB: 240 224 208 192 176 160 144 128 112 96 80 64 48 32 16 Dump complete = 0xf db:0:doadump> reset (kgdb) bt #0 doadump () at pcpu.h:231 #1 0xc04d5809 in db_fncall (dummy1=0xc08f576a, dummy2=0x0, dummy3=0xffffffff, dummy4=0xe9cb9244 "X\222Λι") at ../../../ddb/db_command.c:548 #2 0xc04d5c3f in db_command (last_cmdp=0xc0dfdd7c, cmd_table=0x0, dopager=0x0) at ../../../ddb/db_command.c:445 #3 0xc04d5cf4 in db_command_script (command=0xc0dfec88 "call doadump") at ../../../ddb/db_command.c:516 #4 0xc04d9b80 in db_script_exec (scriptname=0xc0dfe5e0 "doadump", warnifnotfound=Variable "warnifnotfound" is not available. ) at ../../../ddb/db_script.c:302 #5 0xc04d9c11 in db_run_cmd (addr=0x1, have_addr=0x0, count=0xc0ea24c0, modif=0xe9cb937c "") at ../../../ddb/db_script.c:375 #6 0xc04d5c01 in db_command (last_cmdp=0xc0dfdd7c, cmd_table=0x0, dopager=0x1) at ../../../ddb/db_command.c:445 #7 0xc04d5d5a in db_command_loop () at ../../../ddb/db_command.c:498 #8 0xc04d7c5d in db_trap (type=0x3, code=0x0) at ../../../ddb/db_main.c:229 #9 0xc08f559e in kdb_trap (type=0x3, code=0x0, tf=0xe9cb9520) at ../../../kern/subr_kdb.c:546 #10 0xc0c0a7cc in trap (frame=0xe9cb9520) at ../../../i386/i386/trap.c:721 #11 0xc0bf066c in calltrap () at ../../../i386/i386/exception.s:168 #12 0xc08f576a in kdb_enter (why=0xc0ce3779 "panic", msg=0xc0ce3779 "panic") at cpufunc.h:71 #13 0xc08c6624 in panic (fmt=0xc0ce195b "%s: recursing on non recursive lockmgr %s @ %s:%d\n") at ../../../kern/kern_shutdown.c:575 #14 0xc08aee4b in __lockmgr_args (lk=0xdb3a4080, flags=Variable "flags" is not available. ) at ../../../kern/kern_lock.c:695 #15 0xc0b05617 in getdirtybuf (bp=0xdb3a4020, mtx=0xc8693d0c, waitfor=0x1) at lockmgr.h:94 #16 0xc0b0cb10 in softdep_sync_metadata (vp=0xc8693c18) at ../../../ufs/ffs/ffs_softdep.c:5307 #17 0xc0b156e6 in ffs_syncvnode (vp=0xc8693c18, waitfor=0x1) at ../../../ufs/ffs/ffs_vnops.c:333 #18 0xc0b07c24 in softdep_request_cleanup (fs=0xc82b7000, vp=0xc8693c18, cred=0xc826be80) at ../../../ufs/ffs/ffs_softdep.c:5892 #19 0xc0af0f4f in ffs_alloc (ip=0xc872d740, lbn=0x1f, bpref=0x3aaa8, size=0x4000, flags=0x41000000, cred=0xc826be80, bnp=0xe9cb9a08) at ../../../ufs/ffs/ffs_alloc.c:220 #20 0xc0af5d33 in ffs_balloc_ufs2 (vp=0xc8693c18, startoffset=Variable "startoffset" is not available. ) at ../../../ufs/ffs/ffs_balloc.c:856 #21 0xc0b15ba5 in ffs_write (ap=0xe9cb9b94) at ../../../ufs/ffs/ffs_vnops.c:745 #22 0xc0c2b0cc in VOP_WRITE_APV (vop=0xc0de2680, a=0xe9cb9b94) at vnode_if.c:951 #23 0xc0966b1a in vn_write (fp=0xc7f9f578, uio=0xe9cb9c28, active_cred=0xc826be80, flags=0x0, td=0xc83328a0) at vnode_if.h:413 #24 0xc0906547 in dofilewrite (td=0xc83328a0, fd=0x3, fp=0xc7f9f578, auio=0xe9cb9c28, offset=0xffffffffffffffff, flags=0x0) at file.h:239 #25 0xc0906838 in kern_writev (td=0xc83328a0, fd=0x3, auio=0xe9cb9c28) at ../../../kern/sys_generic.c:447 #26 0xc09068bf in write (td=0xc83328a0, uap=0xe9cb9cec) at ../../../kern/sys_generic.c:363 #27 0xc0902179 in syscallenter (td=0xc83328a0, sa=0xe9cb9ce4) at ../../../kern/subr_trap.c:315 #28 0xc0c0a15f in syscall (frame=0xe9cb9d28) at ../../../i386/i386/trap.c:1086 #29 0xc0bf06d1 in Xint0x80_syscall () at ../../../i386/i386/exception.s:266 #30 0x00000033 in ?? () (kgdb) f 16 #16 0xc0b0cb10 in softdep_sync_metadata (vp=0xc8693c18) at ../../../ufs/ffs/ffs_softdep.c:5307 5307 bp = getdirtybuf(bp, BO_MTX(bo), MNT_WAIT); (kgdb) l 5302 * all potential buffers on the dirty list will be visible. 5303 */ 5304 BO_LOCK(bo); 5305 drain_output(vp); 5306 while ((bp = TAILQ_FIRST(&bo->bo_dirty.bv_hd)) != NULL) { 5307 bp = getdirtybuf(bp, BO_MTX(bo), MNT_WAIT); 5308 if (bp) 5309 break; 5310 } 5311 BO_UNLOCK(bo); (kgdb) p *bp $2 = {b_bufobj = 0xc8693d0c, b_bcount = 0x4000, b_caller1 = 0x0, b_data = 0xe2021000 "8w\003", b_error = 0x0, b_iocmd = 0x2, b_ioflags = 0x0, b_iooffset = 0x1d0c000, b_resid = 0x0, b_iodone = 0, b_blkno = 0x3e80, b_offset = 0xfffffffffffd0000, b_bobufs = {tqe_next = 0x0, tqe_prev = 0xc8693d30}, b_left = 0x0, b_right = 0x0, b_vflags = 0x1, b_freelist = {tqe_next = 0xdb2a6074, tqe_prev = 0xdb257e44}, b_qindex = 0x2, b_flags = 0xa00200a0, b_xflags = 0x1, b_lock = {lock_object = {lo_name = 0xc0ceab5b "bufwait", lo_flags = 0x5730000, lo_data = 0x0, lo_witness = 0x0}, lk_lock = 0xc83328a0, lk_timo = 0x0, lk_pri = 0x50, lk_stack = { depth = 0xe, pcs = {0xc08af12b, 0xc09409b1, 0xc0941644, 0xc094176c, 0xc0af5915, 0xc0b15ba5, 0xc0c2b0cc, 0xc0966b1a, 0xc0906547, 0xc0906838, 0xc09068bf, 0xc0902179, 0xc0c0a15f, 0xc0bf06d1, 0x0, 0x0, 0x0, 0x0}}}, b_bufsize = 0x4000, b_runningbufspace = 0x0, b_kvabase = 0xe2021000 "8w\003", b_kvasize = 0x4000, b_lblkno = 0xfffffffffffffff4, b_vp = 0xc8693c18, b_dirtyoff = 0x0, b_dirtyend = 0x0, b_rcred = 0x0, b_wcred = 0x0, b_saveaddr = 0xe2021000, b_pager = {pg_reqpage = 0x0}, b_cluster = {cluster_head = {tqh_first = 0xdb3a7874, tqh_last = 0xdb220830}, cluster_entry = {tqe_next = 0xdb3a7874, tqe_prev = 0xdb220830}}, b_pages = {0xc26b4320, 0xc2250dd8, 0xc2250e20, 0xc30c45b8, 0x0 }, b_npages = 0x4, b_dep = {lh_first = 0xc842ede0}, b_fsprivate1 = 0x0, b_fsprivate2 = 0x0, b_fsprivate3 = 0x0, b_pin_count = 0x0} (kgdb) f 19 #19 0xc0af0f4f in ffs_alloc (ip=0xc872d740, lbn=0x1f, bpref=0x3aaa8, size=0x4000, flags=0x41000000, cred=0xc826be80, bnp=0xe9cb9a08) at ../../../ufs/ffs/ffs_alloc.c:220 220 softdep_request_cleanup(fs, ITOV(ip), cred); (kgdb) l 215 (void) chkdq(ip, -btodb(size), cred, FORCE); 216 UFS_LOCK(ump); 217 #endif 218 if (reclaimed == 0) { 219 reclaimed = 1; 220 softdep_request_cleanup(fs, ITOV(ip), cred); 221 goto retry; 222 } 223 UFS_UNLOCK(ump); 224 if (ppsratecheck(&lastfail, &curfail, 1)) { (kgdb) f 14 #14 0xc08aee4b in __lockmgr_args (lk=0xdb3a4080, flags=Variable "flags" is not available. ) at ../../../kern/kern_lock.c:695 695 panic("%s: recursing on non recursive lockmgr %s @ %s:%d\n", (kgdb) p *lk $3 = {lock_object = {lo_name = 0xc0ceab5b "bufwait", lo_flags = 0x5730000, lo_data = 0x0, lo_witness = 0x0}, lk_lock = 0xc83328a0, lk_timo = 0x0, lk_pri = 0x50, lk_stack = {depth = 0xe, pcs = {0xc08af12b, 0xc09409b1, 0xc0941644, 0xc094176c, 0xc0af5915, 0xc0b15ba5, 0xc0c2b0cc, 0xc0966b1a, 0xc0906547, 0xc0906838, 0xc09068bf, 0xc0902179, 0xc0c0a15f, 0xc0bf06d1, 0x0, 0x0, 0x0, 0x0}}} (kgdb) l *0xc08af12b 0xc08af12b is in __lockmgr_args (../../../kern/kern_lock.c:881). 876 LOCK_LOG_LOCK("XLOCK", &lk->lock_object, 0, 877 lk->lk_recurse, file, line); 878 WITNESS_LOCK(&lk->lock_object, LOP_EXCLUSIVE | 879 LK_TRYWIT(flags), file, line); 880 TD_LOCKS_INC(curthread); 881 STACK_SAVE(lk); 882 } 883 break; 884 case LK_DOWNGRADE: 885 _lockmgr_assert(lk, KA_XLOCKED | KA_NOTRECURSED, file, line); (kgdb) l *0xc09409b1 0xc09409b1 is in getblk (../../../kern/vfs_bio.c:2642). 2637 2638 /* 2639 * If we slept and got the lock we have to restart in case 2640 * the buffer changed identities. 2641 */ 2642 if (error == ENOLCK) 2643 goto loop; 2644 /* We timed out or were interrupted. */ 2645 else if (error) 2646 return (NULL); (kgdb) l *0xc0941644 0xc0941644 is in breadn (../../../kern/vfs_bio.c:824). 819 { 820 struct buf *bp; 821 int rv = 0, readwait = 0; 822 823 CTR3(KTR_BUF, "breadn(%p, %jd, %d)", vp, blkno, size); 824 *bpp = bp = getblk(vp, blkno, size, 0, 0, 0); 825 826 /* if not found in cache, do some I/O */ 827 if ((bp->b_flags & B_CACHE) == 0) { 828 if (!TD_IS_IDLETHREAD(curthread)) (kgdb) l *0xc094176c 0xc094176c is in bread (../../../kern/vfs_bio.c:773). 768 bread(struct vnode * vp, daddr_t blkno, int size, struct ucred * cred, 769 struct buf **bpp) 770 { 771 772 return (breadn(vp, blkno, size, 0, 0, 0, cred, bpp)); 773 } 774 775 /* 776 * Attempt to initiate asynchronous I/O on read-ahead blocks. We must 777 * clear BIO_ERROR and B_INVAL prior to initiating I/O . If B_CACHE is set, (kgdb) l *0xc0af5915 0xc0af5915 is in ffs_balloc_ufs2 (../../../ufs/ffs/ffs_balloc.c:784). 779 } 780 /* 781 * Fetch through the indirect blocks, allocating as necessary. 782 */ 783 for (i = 1;;) { 784 error = bread(vp, 785 indirs[i].in_lbn, (int)fs->fs_bsize, NOCRED, &bp); 786 if (error) { 787 brelse(bp); 788 goto fail; (kgdb) l *0xc0b15ba5 0xc0b15ba5 is in ffs_write (../../../ufs/ffs/ffs_vnops.c:745). 740 if (fs->fs_bsize > xfersize) 741 flags |= BA_CLRBUF; 742 else 743 flags &= ~BA_CLRBUF; 744 /* XXX is uio->uio_offset the right thing here? */ 745 error = UFS_BALLOC(vp, uio->uio_offset, xfersize, 746 ap->a_cred, flags, &bp); 747 if (error != 0) { 748 vnode_pager_setsize(vp, ip->i_size); 749 break; (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/ufs/ffs/ffs_softdep.c =================================================================== --- /usr/src/sys/ufs/ffs/ffs_softdep.c (revision 220001) +++ /usr/src/sys/ufs/ffs/ffs_softdep.c (working copy) @@ -62,6 +62,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include @@ -338,9 +339,10 @@ softdep_releasefile(ip) } int -softdep_request_cleanup(fs, vp) +softdep_request_cleanup(fs, vp, cred) struct fs *fs; struct vnode *vp; + struct ucred *cred; { return (0); @@ -678,6 +680,7 @@ static int stat_indir_blk_ptrs; /* bufs redirtied static int stat_inode_bitmap; /* bufs redirtied as inode bitmap not written */ static int stat_direct_blk_ptrs;/* bufs redirtied as direct ptrs not written */ static int stat_dir_entry; /* bufs redirtied as dir entry cannot write */ +static int stat_high_cleanup_delay; /* Maximum cleanup delay (in ticks) */ SYSCTL_INT(_debug, OID_AUTO, max_softdeps, CTLFLAG_RW, &max_softdeps, 0, ""); SYSCTL_INT(_debug, OID_AUTO, tickdelay, CTLFLAG_RW, &tickdelay, 0, ""); @@ -692,6 +695,7 @@ SYSCTL_INT(_debug, OID_AUTO, indir_blk_ptrs, CTLFL SYSCTL_INT(_debug, OID_AUTO, inode_bitmap, CTLFLAG_RW, &stat_inode_bitmap, 0, ""); SYSCTL_INT(_debug, OID_AUTO, direct_blk_ptrs, CTLFLAG_RW, &stat_direct_blk_ptrs, 0, ""); SYSCTL_INT(_debug, OID_AUTO, dir_entry, CTLFLAG_RW, &stat_dir_entry, 0, ""); +SYSCTL_INT(_debug, OID_AUTO, high_cleanup_delay, CTLFLAG_RW, &stat_high_cleanup_delay, 0, ""); /* SYSCTL_INT(_debug, OID_AUTO, worklist_num, CTLFLAG_RD, &softdep_on_worklist, 0, ""); */ SYSCTL_DECL(_vfs_ffs); @@ -5797,20 +5801,26 @@ softdep_slowdown(vp) * If the worklist yields no free space, encourage the syncer daemon * to help us. In no event will we try for longer than tickdelay seconds. */ + +static void check_block_count(char *, struct fs *, struct ufsmount *, + ufs2_daddr_t); + int -softdep_request_cleanup(fs, vp) +softdep_request_cleanup(fs, vp, cred) struct fs *fs; struct vnode *vp; + struct ucred *cred; { struct ufsmount *ump; + struct mount *mp; + struct vnode *lvp, *mvp; long starttime; ufs2_daddr_t needed; int error; ump = VTOI(vp)->i_ump; + mp = vp->v_mount; mtx_assert(UFS_MTX(ump), MA_OWNED); - needed = fs->fs_cstotal.cs_nbfree + fs->fs_contigsumsize; - starttime = time_second + tickdelay; /* * If we are being called because of a process doing a * copy-on-write, then it is not safe to update the vnode @@ -5823,9 +5833,38 @@ int if (error != 0) return (0); } - while (fs->fs_pendingblocks > 0 && fs->fs_cstotal.cs_nbfree <= needed) { - if (time_second > starttime) - return (0); + /* + * If we are in need of resources, consider pausing for + * tickdelay to give ourselves some breathing room. + */ + UFS_UNLOCK(ump); + ACQUIRE_LOCK(&lk); + request_cleanup(UFSTOVFS(ump), FLUSH_REMOVE_WAIT); + FREE_LOCK(&lk); + UFS_LOCK(ump); + /* + * Now clean up at least as much space as we will need. + * + * The amount of space that we will need is enough blocks to + * allocate a full-sized segment (fs_contigsumsize). The number of + * such segments that will be needed is set by the number of + * simultaneous writers (mnt_writeopcount) plus a bit of slop (2) + * in case some more show up while we are cleaning. + * + * Additionally, if we are unpriviledged we will need to ensure + * that we clean up enough blocks to get to the above threshhold + * of the minimum number of blocks required to be kept free by the + * filesystem (fs_minfree). + */ + needed = (mp->mnt_writeopcount + 2) * fs->fs_contigsumsize; + if (priv_check_cred(cred, PRIV_VFS_BLOCKRESERVE, 0)) + needed += fragstoblks(fs, + roundup((fs->fs_dsize * fs->fs_minfree / 100) - + fs->fs_cstotal.cs_nffree, fs->fs_frag)); + starttime = time_second + tickdelay; +retry: + while (ump->softdep_on_worklist > 0 && + fs->fs_cstotal.cs_nbfree <= needed) { UFS_UNLOCK(ump); ACQUIRE_LOCK(&lk); if (ump->softdep_on_worklist > 0 && @@ -5835,14 +5874,126 @@ int UFS_LOCK(ump); continue; } - request_cleanup(UFSTOVFS(ump), FLUSH_REMOVE_WAIT); FREE_LOCK(&lk); UFS_LOCK(ump); } + /* + * If we still need resources and there are no more worklist + * entries to process to obtain them, we have to start flushing + * the dirty vnodes to force the release of additional requests + * to the worklist that we can then process to reap addition + * resources. We start by flushing our own vnode and if that + * does not produce enough resources we walk the vnodes + * associated with the mount point until we get the needed + * worklist requests that we can reap. + */ + if (fs->fs_cstotal.cs_nbfree <= needed) { + if (TAILQ_FIRST(&vp->v_bufobj.bo_dirty.bv_hd) != 0) + (void) ffs_syncvnode(vp, MNT_WAIT); + MNT_ILOCK(mp); + MNT_VNODE_FOREACH(lvp, mp, mvp) { + if (ump->softdep_on_worklist > 0) { + MNT_IUNLOCK(mp); + check_block_count("Retry", fs, ump, needed); + goto retry; + } + VI_LOCK(lvp); + if (TAILQ_FIRST(&lvp->v_bufobj.bo_dirty.bv_hd) == 0 || + VOP_ISLOCKED(lvp) != 0) { + VI_UNLOCK(lvp); + continue; + } + MNT_IUNLOCK(mp); + if (vget(lvp, LK_EXCLUSIVE | LK_INTERLOCK, curthread)) { + VI_UNLOCK(lvp); + MNT_ILOCK(mp); + continue; + } + if (lvp->v_vflag & VV_NOSYNC) { /* unlinked */ + vput(lvp); + MNT_ILOCK(mp); + continue; + } + (void) ffs_syncvnode(lvp, MNT_WAIT); + vput(lvp); + MNT_ILOCK(mp); + } + MNT_IUNLOCK(mp); + check_block_count("Failed", fs, ump, needed); + } + if (time_second - starttime > stat_high_cleanup_delay) + stat_high_cleanup_delay = time_second - starttime; return (1); } /* + * Check out counts above. + */ +static void +check_block_count(action, fs, ump, needed) + char *action; + struct fs *fs; + struct ufsmount *ump; + ufs2_daddr_t needed; +{ + struct worklist *wk; + static int maxFcount = 20, maxRcount = 20; + int blkcnt, dircnt, fragcnt, inocnt; + ufs2_daddr_t pending; + + if (action[0] == 'F' && maxFcount-- <= 0) + return; + if (action[0] == 'R' && maxRcount-- <= 0) + return; + dircnt = 0; + fragcnt = 0; + blkcnt = 0; + inocnt = 0; + pending = 0; + LIST_FOREACH(wk, &ump->softdep_workitem_pending, wk_list) { + switch (wk->wk_type) { + + case D_DIRREM: + /* removal of a directory entry */ + dircnt += 1; + continue; + + case D_FREEBLKS: + /* releasing blocks and/or fragments from a file */ + pending += WK_FREEBLKS(wk)->fb_chkcnt; + blkcnt += 1; + continue; + + case D_FREEFRAG: + /* releasing a fragment when replaced as a file grows */ + fragcnt += 1; + continue; + + case D_FREEFILE: + /* releasing an inode when its link count drops to 0 */ + inocnt += 1; + continue; + + default: + panic("check_block_count: Unknown type %s", + TYPENAME(wk->wk_type)); + /* NOTREACHED */ + } + } + printf("%s reclaim\n", action); + printf("Seeking %lld blocks\n", needed); + printf("Free inodes %lld, free blocks %lld, free fragments %lld\n", + fs->fs_cstotal.cs_nifree, fs->fs_cstotal.cs_nbfree, + fs->fs_cstotal.cs_nffree); + printf("Pending workitems %d, Allocated inodes %lld\n", + ump->softdep_on_worklist, + (fs->fs_ncg * fs->fs_ipg) - fs->fs_cstotal.cs_nifree); + printf("Found %d files with %lld blocks pending\n", blkcnt, pending); + printf("Other types: dirrem %d, freefrag %d, freefile %d\n", + dircnt, fragcnt, inocnt); +} + +/* * If memory utilization has gotten too high, deliberately slow things * down and speed up the I/O processing. */ Index: /usr/src/sys/ufs/ffs/ffs_alloc.c =================================================================== --- /usr/src/sys/ufs/ffs/ffs_alloc.c (revision 220001) +++ /usr/src/sys/ufs/ffs/ffs_alloc.c (working copy) @@ -215,9 +215,9 @@ nospace: (void) chkdq(ip, -btodb(size), cred, FORCE); UFS_LOCK(ump); #endif - if (fs->fs_pendingblocks > 0 && reclaimed == 0) { + if (reclaimed == 0) { reclaimed = 1; - softdep_request_cleanup(fs, ITOV(ip)); + softdep_request_cleanup(fs, ITOV(ip), cred); goto retry; } UFS_UNLOCK(ump); @@ -430,9 +430,9 @@ nospace: /* * no space available */ - if (fs->fs_pendingblocks > 0 && reclaimed == 0) { + if (reclaimed == 0) { reclaimed = 1; - softdep_request_cleanup(fs, vp); + softdep_request_cleanup(fs, vp, cred); UFS_UNLOCK(ump); if (bp) { brelse(bp); Index: /usr/src/sys/ufs/ffs/ffs_extern.h =================================================================== --- /usr/src/sys/ufs/ffs/ffs_extern.h (revision 220001) +++ /usr/src/sys/ufs/ffs/ffs_extern.h (working copy) @@ -114,7 +114,7 @@ int softdep_flushfiles(struct mount *, int, struct void softdep_update_inodeblock(struct inode *, struct buf *, int); void softdep_load_inodeblock(struct inode *); void softdep_freefile(struct vnode *, ino_t, int); -int softdep_request_cleanup(struct fs *, struct vnode *); +int softdep_request_cleanup(struct fs *, struct vnode *, struct ucred *); void softdep_setup_freeblocks(struct inode *, off_t, int); void softdep_setup_inomapdep(struct buf *, struct inode *, ino_t); void softdep_setup_blkmapdep(struct buf *, struct mount *, ufs2_daddr_t);