GENERIC stable8 from 2011-03-25 13:09:57 UTC, r220001M, vmcore.2 524288K of memory above 4GB ignored Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.2-STABLE #1 r220001M: Thu Mar 31 20:40:53 CEST 2011 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 WARNING: DIAGNOSTIC option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1799.99-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Family = 10 Model = 2 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 4294967296 (4096 MB) avail memory = 3536519168 (3372 MB) ACPI APIC Table: <060508 APIC1048> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: <060508 RSDT1048> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of fee00000, 1000 (3) failed acpi0: reservation of ffb80000, 80000 (3) failed acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, d7f00000 (3) failed Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 vgapci0: port 0xc000-0xc0ff mem 0xf0000000-0xf7ffffff,0xfe9f0000-0xfe9fffff,0xfe800000-0xfe8fffff irq 18 at device 5.0 on pci1 pci1: at device 5.2 (no driver attached) pcib2: at device 7.0 on pci0 pci2: on pcib2 re0: port 0xd800-0xd8ff mem 0xfeaff000-0xfeafffff irq 19 at device 0.0 on pci2 re0: Using 1 MSI message re0: Chip rev. 0x38000000 re0: MAC rev. 0x00000000 miibus0: on re0 rgephy0: PHY 1 on miibus0 rgephy0: 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re0: Ethernet address: 00:21:85:15:33:7d re0: [ITHREAD] atapci0: port 0xb000-0xb007,0xa000-0xa003,0x9000-0x9007,0x8000-0x8003,0x7000-0x700f mem 0xfe7ff800-0xfe7ffbff irq 22 at device 18.0 on pci0 atapci0: [ITHREAD] atapci0: AHCI v1.10 controller with 4 3Gbps ports, PM supported ata2: on atapci0 ata2: port is not ready (timeout 0ms) tfd = 000001d0 ata2: software reset clear timeout ata2: [ITHREAD] ata3: on atapci0 ata3: [ITHREAD] ata4: on atapci0 ata4: [ITHREAD] ata5: on atapci0 ata5: [ITHREAD] ohci0: mem 0xfe7fe000-0xfe7fefff irq 16 at device 19.0 on pci0 ohci0: [ITHREAD] usbus0: on ohci0 ohci1: mem 0xfe7fd000-0xfe7fdfff irq 17 at device 19.1 on pci0 ohci1: [ITHREAD] usbus1: on ohci1 ohci2: mem 0xfe7fc000-0xfe7fcfff irq 18 at device 19.2 on pci0 ohci2: [ITHREAD] usbus2: on ohci2 ohci3: mem 0xfe7fb000-0xfe7fbfff irq 17 at device 19.3 on pci0 ohci3: [ITHREAD] usbus3: on ohci3 ohci4: mem 0xfe7fa000-0xfe7fafff irq 18 at device 19.4 on pci0 ohci4: [ITHREAD] usbus4: on ohci4 ehci0: mem 0xfe7ff000-0xfe7ff0ff irq 19 at device 19.5 on pci0 ehci0: [ITHREAD] ehci0: AMD SB600/700 quirk applied usbus5: EHCI version 1.0 usbus5: on ehci0 pci0: at device 20.0 (no driver attached) atapci1: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0 ata0: on atapci1 ata0: [ITHREAD] pci0: at device 20.2 (no driver attached) isab0: at device 20.3 on pci0 isa0: on isab0 pcib3: at device 20.4 on pci0 pci3: on pcib3 fwohci0: port 0xe800-0xe87f mem 0xfebff800-0xfebfffff irq 20 at device 6.0 on pci3 fwohci0: [ITHREAD] fwohci0: OHCI version 1.10 (ROM=1) fwohci0: No. of Isochronous channels is 4. fwohci0: EUI64 00:dc:10:00:01:62:ad:eb fwohci0: Phy 1394a available S400, 2 ports. fwohci0: Link S400, max_rec 2048 bytes. firewire0: on fwohci0 dcons_crom0: on firewire0 dcons_crom0: bus_addr 0xe04720 fwe0: on firewire0 if_fwe0: Fake Ethernet address: 02:dc:10:62:ad:eb fwe0: Ethernet address: 02:dc:10:62:ad:eb fwip0: on firewire0 fwip0: Firewire address: 00:dc:10:00:01:62:ad:eb @ 0xfffe00000000, S400, maxrec 2048 fwohci0: Initiate bus reset fwohci0: fwohci_intr_core: BUS reset fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode acpi_button0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart0: console (115200,n,8,1) fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model IntelliMouse Explorer, device ID 4 pmtimer0 on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: parallel port not found. acpi_throttle0: on cpu0 acpi_throttle0: CLK_VAL field overlaps THT_EN bit device_attach: acpi_throttle0 attach returned 6 hwpstate0: on cpu0 Timecounters tick every 1.000 msec firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me) firewire0: bus manager 0 usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 12Mbps Full Speed USB v1.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 480Mbps High Speed USB v2.0 acd0: DVDR at ata0-slave UDMA33 ugen0.1: at usbus0 uhub0: on usbus0 Expensive timeout(9) function: 0xc08ff3a0(0xc73952e0) 0.016028014 s ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 ugen4.1: at usbus4 uhub4: on usbus4 ugen5.1: at usbus5 uhub5: on usbus5 ad4: 476940MB at ata2-master UDMA100 SATA 3Gb/s SMP: AP CPU #2 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! WARNING: DIAGNOSTIC option enabled, expect reduced performance. uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered Root mount waiting for: usbus5 Root mount waiting for: usbus5 Root mount waiting for: usbus5 Root mount waiting for: usbus5 uhub5: 10 ports with 10 removable, self powered Trying to mount root from ufs:/dev/ad4s2a Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Starting ddb. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/ad4s2a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s2a: clean, 582016 free (1080 frags, 72617 blocks, 0.1% fragmentation) /dev/ad4s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1f: clean, 651531 free (1283 frags, 81281 blocks, 0.1% fragmentation) /dev/ad4s1g: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1g: clean, 91353172 free (3052 frags, 11418765 blocks, 0.0% fragmentation) /dev/ad4s2d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s2d: clean, 8629786 free (41106 frags, 1073585 blocks, 0.4% fragmentation) /dev/ad4s2e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s2e: clean, 25328230 free (518 frags, 3165964 blocks, 0.0% fragmentation) Mounting local file systems:. Setting hostname: x4.osted.lan. re0: link state changed to DOWN Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=3 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 inet6 ::1 prefixlen 128 inet 127.0.0.1 netmask 0xff000000 nd6 options=3 re0: flags=8843 metric 0 mtu 1500 options=389b ether 00:21:85:15:33:7d inet 192.168.1.7 netmask 0xffffff00 broadcast 192.168.1.255 media: Ethernet autoselect (none) status: no carrier add net default: gateway 192.168.1.1 Starting devd. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Creating and/or trimming log files. Starting syslogd. No core dumps found. Additional ABI support: linuxkldload: can't load linux: No such file or directory /etc/rc: WARNING: Unable to load kernel module linux . Starting rpcbind. NFS access cache time=60 Clearing /tmp (X related). Starting mountd. Starting nfsd. Updating motd:. Starting ntpd. Configuring syscons: keymap blanktime. Starting sshd. Starting cron. Starting default moused. Starting inetd. Thu Mar 31 20:42:54 CEST 2011 FreeBSD/i386 (x4.osted.lan) (console) login: Mar 31 20:43:48 x4 su: pho to root on /dev/pts/0 SU test #1 Sleeping thread (tid 100090, pid 1980) owns a non-sleepable lock sched_switch(c7be52e0,0,104,c0239183,e0,...) at sched_switch+0x2b6 mi_switch(104,0,c7be52e0,0,e9b646c0,...) at mi_switch+0x11b sleepq_switch(c7be52e0,0,c0ce691d,260,4f,...) at sleepq_switch+0xcc sleepq_wait(c8291280,4f,c0cea4e8,0,0,...) at sleepq_wait+0x39 _sleep(c8291280,c829124c,4f,c0cea4e8,0,...) at _sleep+0x2a2 bufobj_wwait(c829124c,3,0,0,0,...) at bufobj_wwait+0x43 ffs_syncvnode(c8291158,1,c7be52e0,0,0,...) at ffs_syncvnode+0x3b1 softdep_request_cleanup(c8277800,c8559c18,c8266c80,0,0,...) at softdep_request_cleanup+0x7de ffs_alloc(c86181d0,23,0,37dd0,0,...) at ffs_alloc+0x2ff ffs_balloc_ufs2(c8559c18,8c000,0,4000,c8266c80,...) at ffs_balloc_ufs2+0x1af3 ffs_write(e9b64b94,c0d13d4d,c8559c70,0,c0c28152,...) at ffs_write+0x3f5 VOP_WRITE_APV(c0de2660,e9b64b94,c8559c18,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c7b7e770,e9b64c28,c8266c80,0,c7be52e0,...) at vn_write+0x1da dofilewrite(e9b64c28,ffffffff,ffffffff,0,c7b7e770,...) at dofilewrite+0x97 kern_writev(c7be52e0,3,e9b64c28,e9b64c48,1,...) at kern_writev+0x58 write(c7be52e0,e9b64cec,e9b64d28,c08e6fdc,c0e3586c,...) at write+0x4f syscallenter(c7be52e0,e9b64ce4,e9b64ce4,0,c73bab80,...) at syscallenter+0x329 syscall(e9b64d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfe9ec, ebp = 0xbfbfea98 --- panic: sleeping thread cpuid = 1 KDB: stack backtrace: db_trace_self_wrapper(c0ce5b0f,38393120,6f202930,20736e77,6f6e2061,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0d0fbb1,1,c0ce6eaf,e9b94718,1,...) at kdb_backtrace+0x2a panic(c0ce6eaf,ffffffff,7bc,cb,c7be4000,...) at panic+0x117 propagate_priority(c7be4000,c81f5a80,c0ce6e54,2d6,e9b94758,...) at propagate_priority+0x10c turnstile_wait(c81f5a80,c7be52e0,0,4d94ccf6,c7e97900,...) at turnstile_wait+0x2c8 _mtx_lock_sleep(c7e97948,c7be4000,0,0,0,...) at _mtx_lock_sleep+0x10c softdep_request_cleanup(c8277800,c8460560,c8266c80,0,0,...) at softdep_request_cleanup+0x4af ffs_alloc(c85402b8,2f,0,35410,0,...) at ffs_alloc+0x2ff ffs_balloc_ufs2(c8460560,bc000,0,4000,c8266c80,...) at ffs_balloc_ufs2+0x1af3 ffs_write(e9b94b94,c0d13d4d,c84605b8,0,c0c28152,...) at ffs_write+0x3f5 VOP_WRITE_APV(c0de2660,e9b94b94,c8460560,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c7b9aab8,e9b94c28,c8266c80,0,c7be4000,...) at vn_write+0x1da dofilewrite(e9b94c28,ffffffff,ffffffff,0,c7b9aab8,...) at dofilewrite+0x97 kern_writev(c7be4000,3,e9b94c28,e9b94c48,1,...) at kern_writev+0x58 write(c7be4000,e9b94cec,e9b94c80,c08e6fdc,c0e34b6c,...) at write+0x4f syscallenter(c7be4000,e9b94ce4,c08e9056,c7be4000,c73bab80,...) at syscallenter+0x329 syscall(e9b94d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfe9ec, ebp = 0xbfbfea98 --- KDB: enter: panic [thread pid 1979 tid 100102 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> run pho db:0:pho> bt Tracing pid 1979 tid 100102 td 0xc7be4000 kdb_enter(c0ce3759,c0ce3759,c0ce6eaf,e9b94718,1,...) at kdb_enter+0x3a panic(c0ce6eaf,ffffffff,7bc,cb,c7be4000,...) at panic+0x134 propagate_priority(c7be4000,c81f5a80,c0ce6e54,2d6,e9b94758,...) at propagate_priority+0x10c turnstile_wait(c81f5a80,c7be52e0,0,4d94ccf6,c7e97900,...) at turnstile_wait+0x2c8 _mtx_lock_sleep(c7e97948,c7be4000,0,0,0,...) at _mtx_lock_sleep+0x10c softdep_request_cleanup(c8277800,c8460560,c8266c80,0,0,...) at softdep_request_cleanup+0x4af ffs_alloc(c85402b8,2f,0,35410,0,...) at ffs_alloc+0x2ff ffs_balloc_ufs2(c8460560,bc000,0,4000,c8266c80,...) at ffs_balloc_ufs2+0x1af3 ffs_write(e9b94b94,c0d13d4d,c84605b8,0,c0c28152,...) at ffs_write+0x3f5 VOP_WRITE_APV(c0de2660,e9b94b94,c8460560,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c7b9aab8,e9b94c28,c8266c80,0,c7be4000,...) at vn_write+0x1da dofilewrite(e9b94c28,ffffffff,ffffffff,0,c7b9aab8,...) at dofilewrite+0x97 kern_writev(c7be4000,3,e9b94c28,e9b94c48,1,...) at kern_writev+0x58 write(c7be4000,e9b94cec,e9b94c80,c08e6fdc,c0e34b6c,...) at write+0x4f syscallenter(c7be4000,e9b94ce4,c08e9056,c7be4000,c73bab80,...) at syscallenter+0x329 syscall(e9b94d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfe9ec, ebp = 0xbfbfea98 --- db:0:bt> show allpcpu Current CPU: 1 cpuid = 0 dynamic pcpu = 0x629080 curthread = 0xc7fba000: pid 765 "syslogd" curpcb = 0xe9c23d80 fpcurthread = none idlethread = 0xc7158000: tid 100006 "idle: cpu0" APIC ID = 0 currentldt = 0x50 cpuid = 1 dynamic pcpu = 0x60e8080 curthread = 0xc7be4000: pid 1979 "linger2" curpcb = 0xe9b94d80 fpcurthread = none idlethread = 0xc71582e0: tid 100005 "idle: cpu1" APIC ID = 1 currentldt = 0x50 cpuid = 2 dynamic pcpu = 0x60eb080 curthread = 0xc7260000: pid 3 "g_up" curpcb = 0xc6f26d80 fpcurthread = none idlethread = 0xc71585c0: tid 100004 "idle: cpu2" APIC ID = 2 currentldt = 0x50 cpuid = 3 dynamic pcpu = 0x60ee080 curthread = 0xc82145c0: pid 1983 "linger2" curpcb = 0xe9cf8d80 fpcurthread = none idlethread = 0xc71588a0: tid 100003 "idle: cpu3" APIC ID = 3 currentldt = 0x50 db:0:allpcpu> show alllocks No such command db:0:alllocks> show lockedvnods Locked vnodes 0xc7fb5000: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xc7f44dd0 ref 0 pages 7 lock type ufs: EXCL by thread 0xc7fba000 (pid 765) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc0b168ba at ffs_lock+0x8a #2 0xc0c2a85d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc0966aa6 at vn_write+0x166 #5 0xc0906547 at dofilewrite+0x97 #6 0xc0906838 at kern_writev+0x58 #7 0xc0906a96 at writev+0x46 #8 0xc0902179 at syscallenter+0x329 #9 0xc0c0a13f at syscall+0x4f #10 0xc0bf06b1 at Xint0x80_syscall+0x21 ino 3556382, on dev ad4s2e 0xc8291158: tag ufs, type VDIR usecount 13, writecount 0, refcount 16 mountedhere 0 flags (VV_ROOT) v_object 0xc82965d8 ref 0 pages 2 lock type ufs: EXCL by thread 0xc7be52e0 (pid 1980) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc0b168ba at ffs_lock+0x8a #2 0xc0c2a85d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc0958b49 at vget+0x89 #5 0xc0b07deb at softdep_request_cleanup+0x6eb #6 0xc0af0f4f at ffs_alloc+0x2ff #7 0xc0af5d33 at ffs_balloc_ufs2+0x1af3 #8 0xc0b15b85 at ffs_write+0x3f5 #9 0xc0c2b0ac at VOP_WRITE_APV+0xcc #10 0xc0966b1a at vn_write+0x1da #11 0xc0906547 at dofilewrite+0x97 #12 0xc0906838 at kern_writev+0x58 #13 0xc09068bf at write+0x4f #14 0xc0902179 at syscallenter+0x329 #15 0xc0c0a13f at syscall+0x4f #16 0xc0bf06b1 at Xint0x80_syscall+0x21 ino 2, on dev md5a 0xc8291000: tag syncer, type VNON usecount 1, writecount 0, refcount 2 mountedhere 0 flags () lock type syncer: EXCL by thread 0xc74ca000 (pid 19) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc09461a5 at vop_stdlock+0x65 #2 0xc0c2a85d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc095afb8 at sync_vnode+0x198 #5 0xc095b3d2 at sched_sync+0x2c2 #6 0xc0899d31 at fork_exit+0x91 #7 0xc0bf06c4 at fork_trampoline+0x8 0xc8705158: tag ufs, type VREG usecount 1, writecount 0, refcount 47 mountedhere 0 flags () v_object 0xc858f198 ref 0 pages 260 lock type ufs: EXCL by thread 0xc74ca000 (pid 19) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc0b168ba at ffs_lock+0x8a #2 0xc0c2a85d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc0958b49 at vget+0x89 #5 0xc0b10529 at ffs_sync+0x309 #6 0xc095a63d at sync_fsync+0x24d #7 0xc0c28eb4 at VOP_FSYNC_APV+0x64 #8 0xc095afe1 at sync_vnode+0x1c1 #9 0xc095b3d2 at sched_sync+0x2c2 #10 0xc0899d31 at fork_exit+0x91 #11 0xc0bf06c4 at fork_trampoline+0x8 ino 807, on dev md5a 0xc8460560: tag ufs, type VREG usecount 1, writecount 1, refcount 50 mountedhere 0 flags () v_object 0xc82b1dd0 ref 0 pages 192 lock type ufs: EXCL by thread 0xc7be4000 (pid 1979) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc0b168ba at ffs_lock+0x8a #2 0xc0c2a85d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc0966aa6 at vn_write+0x166 #5 0xc0906547 at dofilewrite+0x97 #6 0xc0906838 at kern_writev+0x58 #7 0xc09068bf at write+0x4f #8 0xc0902179 at syscallenter+0x329 #9 0xc0c0a13f at syscall+0x4f #10 0xc0bf06b1 at Xint0x80_syscall+0x21 ino 890, on dev md5a 0xc86de158: tag ufs, type VREG usecount 1, writecount 1, refcount 56 mountedhere 0 flags () v_object 0xc8319d48 ref 0 pages 216 lock type ufs: EXCL by thread 0xc8878000 (pid 1976) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc0b168ba at ffs_lock+0x8a #2 0xc0c2a85d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc0966aa6 at vn_write+0x166 #5 0xc0906547 at dofilewrite+0x97 #6 0xc0906838 at kern_writev+0x58 #7 0xc09068bf at write+0x4f #8 0xc0902179 at syscallenter+0x329 #9 0xc0c0a13f at syscall+0x4f #10 0xc0bf06b1 at Xint0x80_syscall+0x21 ino 891, on dev md5a 0xc8559c18: tag ufs, type VREG usecount 1, writecount 1, refcount 38 mountedhere 0 flags () v_object 0xc864d7f8 ref 0 pages 144 lock type ufs: EXCL by thread 0xc7be52e0 (pid 1980) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc0b168ba at ffs_lock+0x8a #2 0xc0c2a85d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc0966aa6 at vn_write+0x166 #5 0xc0906547 at dofilewrite+0x97 #6 0xc0906838 at kern_writev+0x58 #7 0xc09068bf at write+0x4f #8 0xc0902179 at syscallenter+0x329 #9 0xc0c0a13f at syscall+0x4f #10 0xc0bf06b1 at Xint0x80_syscall+0x21 ino 892, on dev md5a 0xc8706000: tag ufs, type VREG usecount 1, writecount 1, refcount 46 mountedhere 0 flags () v_object 0xc8637dd0 ref 0 pages 176 lock type ufs: EXCL by thread 0xc87895c0 (pid 1984) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc0b168ba at ffs_lock+0x8a #2 0xc0c2a85d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc0966aa6 at vn_write+0x166 #5 0xc0906547 at dofilewrite+0x97 #6 0xc0906838 at kern_writev+0x58 #7 0xc09068bf at write+0x4f #8 0xc0902179 at syscallenter+0x329 #9 0xc0c0a13f at syscall+0x4f #10 0xc0bf06b1 at Xint0x80_syscall+0x21 ino 893, on dev md5a 0xc86612b0: tag ufs, type VREG usecount 1, writecount 1, refcount 45 mountedhere 0 flags () v_object 0xc863c990 ref 0 pages 172 lock type ufs: EXCL by thread 0xc8213000 (pid 1978) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc0b168ba at ffs_lock+0x8a #2 0xc0c2a85d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc0966aa6 at vn_write+0x166 #5 0xc0906547 at dofilewrite+0x97 #6 0xc0906838 at kern_writev+0x58 #7 0xc09068bf at write+0x4f #8 0xc0902179 at syscallenter+0x329 #9 0xc0c0a13f at syscall+0x4f #10 0xc0bf06b1 at Xint0x80_syscall+0x21 ino 894, on dev md5a 0xc853cac0: tag ufs, type VREG usecount 1, writecount 1, refcount 42 mountedhere 0 flags () v_object 0xc86d52a8 ref 0 pages 160 lock type ufs: EXCL by thread 0xc82112e0 (pid 1981) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc0b168ba at ffs_lock+0x8a #2 0xc0c2a85d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc0966aa6 at vn_write+0x166 #5 0xc0906547 at dofilewrite+0x97 #6 0xc0906838 at kern_writev+0x58 #7 0xc09068bf at write+0x4f #8 0xc0902179 at syscallenter+0x329 #9 0xc0c0a13f at syscall+0x4f #10 0xc0bf06b1 at Xint0x80_syscall+0x21 ino 895, on dev md5a 0xc86e0810: tag ufs, type VREG usecount 1, writecount 1, refcount 39 mountedhere 0 flags () v_object 0xc82b0550 ref 0 pages 148 lock type ufs: EXCL by thread 0xc7badb80 (pid 1982) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc0b168ba at ffs_lock+0x8a #2 0xc0c2a85d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc0966aa6 at vn_write+0x166 #5 0xc0906547 at dofilewrite+0x97 #6 0xc0906838 at kern_writev+0x58 #7 0xc09068bf at write+0x4f #8 0xc0902179 at syscallenter+0x329 #9 0xc0c0a13f at syscall+0x4f #10 0xc0bf06b1 at Xint0x80_syscall+0x21 ino 901, on dev md5a 0xc8706d70: tag ufs, type VREG usecount 1, writecount 1, refcount 35 mountedhere 0 flags () v_object 0xc86d45d8 ref 0 pages 132 lock type ufs: EXCL by thread 0xc7be58a0 (pid 1977) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc0b168ba at ffs_lock+0x8a #2 0xc0c2a85d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc0966aa6 at vn_write+0x166 #5 0xc0906547 at dofilewrite+0x97 #6 0xc0906838 at kern_writev+0x58 #7 0xc09068bf at write+0x4f #8 0xc0902179 at syscallenter+0x329 #9 0xc0c0a13f at syscall+0x4f #10 0xc0bf06b1 at Xint0x80_syscall+0x21 ino 902, on dev md5a 0xc853bac0: tag ufs, type VREG usecount 1, writecount 1, refcount 40 mountedhere 0 flags () v_object 0xc8638990 ref 0 pages 152 lock type ufs: EXCL by thread 0xc88775c0 (pid 1975) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc0b168ba at ffs_lock+0x8a #2 0xc0c2a85d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc0966aa6 at vn_write+0x166 #5 0xc0906547 at dofilewrite+0x97 #6 0xc0906838 at kern_writev+0x58 #7 0xc09068bf at write+0x4f #8 0xc0902179 at syscallenter+0x329 #9 0xc0c0a13f at syscall+0x4f #10 0xc0bf06b1 at Xint0x80_syscall+0x21 ino 903, on dev md5a 0xc84416b8: tag ufs, type VREG usecount 1, writecount 1, refcount 14 mountedhere 0 flags () v_object 0xc87f2bb0 ref 0 pages 48 lock type ufs: EXCL by thread 0xc82145c0 (pid 1983) #0 0xc08af12b at __lockmgr_args+0x93b #1 0xc0b168ba at ffs_lock+0x8a #2 0xc0c2a85d at VOP_LOCK1_APV+0x4d #3 0xc0965ddd at _vn_lock+0x3d #4 0xc0966aa6 at vn_write+0x166 #5 0xc0906547 at dofilewrite+0x97 #6 0xc0906838 at kern_writev+0x58 #7 0xc09068bf at write+0x4f #8 0xc0902179 at syscallenter+0x329 #9 0xc0c0a13f at syscall+0x4f #10 0xc0bf06b1 at Xint0x80_syscall+0x21 ino 904, on dev md5a db:0:lockedvnods> show mount 0xc7b2ab40 /dev/ad4s2a on / (ufs) 0xc7b2b000 devfs on /dev (devfs) 0xc7e595a0 /dev/ad4s1f on /home (ufs) 0xc7e5b000 /dev/ad4s1g on /tmp (ufs) 0xc7e5ab40 /dev/ad4s2d on /usr (ufs) 0xc7e5a870 /dev/ad4s2e on /var (ufs) 0xc7e5a5a0 /dev/md5a on /mnt (ufs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 1984 1238 1237 1001 R+ linger2 1983 1238 1237 1001 R+ CPU 3 linger2 1982 1238 1237 1001 S+ getblk 0xdb139a10 linger2 1981 1238 1237 1001 S+ getblk 0xdb139a10 linger2 1980 1238 1237 1001 S+ bo_wwait 0xc8291280 linger2 1979 1238 1237 1001 L+ *FFS 0xc81f5a80 linger2 1978 1238 1237 1001 R+ linger2 1977 1238 1237 1001 S+ getblk 0xdb139a10 linger2 1976 1238 1237 1001 L+ *FFS 0xc81f5a80 linger2 1975 1238 1237 1001 R+ linger2 1238 1237 1237 1001 S+ wait 0xc8210810 linger2 1237 1236 1237 1001 S+ wait 0xc820f000 bash 1236 1217 1210 0 S+ wait 0xc820fac0 su 1231 0 0 0 RL [md5] 1217 1216 1210 0 S+ wait 0xc7fb4ac0 sh 1216 1210 1210 0 S+ wait 0xc820eac0 time 1211 1206 1210 0 S+ piperd 0xc7bb6620 tee 1210 1206 1210 0 S+ wait 0xc8074000 sh 1206 1202 1206 0 S+ wait 0xc820f2b0 bash 1202 1201 1202 0 S+ pause 0xc820e868 csh 1201 1197 1201 1001 S+ wait 0xc820e560 su 1197 1196 1197 1001 Ss+ wait 0xc80752b0 bash 1196 1194 1194 1001 S select 0xc8072264 sshd 1194 1070 1194 0 Ss sbwait 0xc7fcf0b8 sshd 1193 1 1193 0 Ss+ ttyin 0xc74d8070 getty 1192 1 1192 0 Ss+ ttyin 0xc74d8270 getty 1191 1 1191 0 Ss+ ttyin 0xc73d7a70 getty 1190 1 1190 0 Ss+ ttyin 0xc7367670 getty 1189 1 1189 0 Ss+ ttyin 0xc7367870 getty 1188 1 1188 0 Ss+ ttyin 0xc7367c70 getty 1187 1 1187 0 Ss+ ttyin 0xc73d7670 getty 1186 1 1186 0 Ss+ ttyin 0xc73d7870 getty 1185 1 1185 0 Ss+ ttyin 0xc7367a70 getty 1146 1 1146 0 Ss select 0xc7be6524 inetd 1120 1 1120 0 Ss select 0xc7be69a4 moused 1096 1 1096 0 Ss nanslp 0xc0e30704 cron 1089 1 1089 25 Ss pause 0xc7fb35b8 sendmail 1085 1 1085 0 Ss select 0xc7ad55a4 sendmail 1070 1 1070 0 Ss select 0xc7ad6be4 sshd 1025 1 1025 0 Ss select 0xc7be6de4 ntpd 899 898 898 0 S (threaded) nfsd 100128 S rpcsvc 0xc7ad5a10 nfsd: service 100127 S rpcsvc 0xc7a94510 nfsd: service 100126 S rpcsvc 0xc7ad6ad0 nfsd: service 100110 S rpcsvc 0xc7ad5790 nfsd: master 898 1 898 0 Ss select 0xc7ad6de4 nfsd 889 1 889 0 Ss select 0xc7b1c9a4 mountd 792 1 792 0 Ss select 0xc7a962e4 rpcbind 765 1 765 0 Rs CPU 0 syslogd 617 1 617 0 Ss select 0xc7b1c224 devd 21 0 0 0 SL c_flowcl 0xc0e44470 [flowcleaner] 20 0 0 0 SL sdflush 0xc0e4fb20 [softdepflush] 19 0 0 0 SL wswbuf0 0xc0de4b00 [syncer] 18 0 0 0 SL vlruwt 0xc7ab72b0 [vnlru] 17 0 0 0 SL psleep 0xc0e43fa8 [bufdaemon] 16 0 0 0 SL pgzero 0xc0e507f4 [pagezero] 15 0 0 0 SL psleep 0xc0e5041c [vmdaemon] 9 0 0 0 SL psleep 0xc0e503e4 [pagedaemon] 8 0 0 0 SL ccb_scan 0xc0dfcc54 [xpt_thrd] 7 0 0 0 SL waiting_ 0xc0e45cb8 [sctp_iterator] 6 0 0 0 SL - 0xc73d743c [fdc0] 5 0 0 0 SL - 0xc73de000 [fw0_probe] 14 0 0 0 SL (threaded) usb 100058 D - 0xc73bfd34 [usbus5] 100057 D - 0xc73bfd04 [usbus5] 100056 D - 0xc73bfcd4 [usbus5] 100055 D - 0xc73bfca4 [usbus5] 100053 D - 0xc73b2b5c [usbus4] 100052 D - 0xc73b2b2c [usbus4] 100051 D - 0xc73b2afc [usbus4] 100050 D - 0xc73b2acc [usbus4] 100049 D - 0xc73aeb5c [usbus3] 100048 D - 0xc73aeb2c [usbus3] 100047 D - 0xc73aeafc [usbus3] 100046 D - 0xc73aeacc [usbus3] 100045 D - 0xc73a1b5c [usbus2] 100044 D - 0xc73a1b2c [usbus2] 100043 D - 0xc73a1afc [usbus2] 100042 D - 0xc73a1acc [usbus2] 100040 D - 0xc739eb5c [usbus1] 100039 D - 0xc739eb2c [usbus1] 100038 D - 0xc739eafc [usbus1] 100037 D - 0xc739eacc [usbus1] 100035 D - 0xc738db5c [usbus0] 100034 D - 0xc738db2c [usbus0] 100033 D - 0xc738dafc [usbus0] 100032 D - 0xc738dacc [usbus0] 13 0 0 0 SL - 0xc0e30564 [yarrow] 4 0 0 0 RL [g_down] 3 0 0 0 RL CPU 2 [g_up] 2 0 0 0 SL - 0xc0e2e258 [g_event] 12 0 0 0 RL (threaded) intr 100066 I [irq12: psm0] 100065 I [irq1: atkbd0] 100063 I [swi0: uart] 100060 I [irq20: fwohci0] 100059 I [irq14: ata0] 100054 I [irq19: ehci0] 100041 I [irq18: ohci2 ohci4] 100036 I [irq17: ohci1 ohci3] 100031 I [irq16: ohci0] 100030 I [irq22: atapci0] 100029 I [irq256: re0] 100028 I [irq9: acpi0] 100026 I [swi5: +] 100021 I [swi2: cambio] 100019 I [swi6: task queue] 100018 I [swi6: Giant taskq] 100012 I [swi4: clock] 100011 I [swi4: clock] 100010 I [swi4: clock] 100009 RunQ [swi4: clock] 100008 I [swi1: netisr 0] 100007 I [swi3: vm] 11 0 0 0 RL (threaded) idle 100006 CanRun [idle: cpu0] 100005 CanRun [idle: cpu1] 100004 CanRun [idle: cpu2] 100003 CanRun [idle: cpu3] 1 0 1 0 SLs wait 0xc7156ac0 [init] 10 0 0 0 SL audit_wo 0xc0e4f440 [audit] 0 0 0 0 SLs (threaded) kernel 100061 D - 0xc73dd580 [fw0_taskq] 100027 D - 0xc732c9c0 [thread taskq] 100025 D - 0xc732ca80 [ffs_trim taskq] 100024 D - 0xc732cc00 [acpi_task_2] 100023 D - 0xc732cc00 [acpi_task_1] 100022 D - 0xc732cc00 [acpi_task_0] 100020 D - 0xc732cd40 [kqueue taskq] 100016 D - 0xc7137880 [firmware taskq] 100000 D sched 0xc0e2e340 [swapper] db:0:ps> allt Tracing command linger2 pid 1984 tid 100188 td 0xc87895c0 sched_switch(c87895c0,0,602,c6229d1d,e0,...) at sched_switch+0x2b6 mi_switch(602,0,c0ce3be3,cd,c0e5ef00,...) at mi_switch+0x11b critical_exit(0,c0c21a39,0) at critical_exit+0x92 lapic_handle_timer(e9d0d704) at lapic_handle_timer+0x17a Xtimerint() at Xtimerint+0x20 --- interrupt, eip = 0xc0c21a39, esp = 0xe9d0d744, ebp = 0xe9d0d760 --- DELAY(1,c0e43ad8,c87895c0,c7e97948,e9d0d7a0,...) at DELAY+0x89 _mtx_lock_spin(c0e43adc,c87895c0,0,0,0,...) at _mtx_lock_spin+0x6c turnstile_trywait(c7e97948,c8213000,0,1e,3,...) at turnstile_trywait+0x6f _mtx_lock_sleep(c7e97948,c87895c0,0,0,0,...) at _mtx_lock_sleep+0x99 ffs_reallocblks(e9d0d9f4,c0d13cd6,c8706000,c82c2d1c,e9d0da28,...) at ffs_reallocblks+0xb8e VOP_REALLOCBLKS_APV(c0de2660,e9d0d9f4,0,4000,0,...) at VOP_REALLOCBLKS_APV+0x6b cluster_write(c8706000,db11e3f0,ac000,0,41,...) at cluster_write+0x3a2 ffs_write(e9d0db94,c0d13d4d,c8706058,0,c0c28152,...) at ffs_write+0x637 VOP_WRITE_APV(c0de2660,e9d0db94,c8706000,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c7fc5118,e9d0dc28,c8266c80,0,c87895c0,...) at vn_write+0x1da dofilewrite(e9d0dc28,ffffffff,ffffffff,0,c7fc5118,...) at dofilewrite+0x97 kern_writev(c87895c0,3,e9d0dc28,e9d0dc48,1,...) at kern_writev+0x58 write(c87895c0,e9d0dcec,e9d0dc80,c08e6fcb,c0e34d74,...) at write+0x4f syscallenter(c87895c0,e9d0dce4,c08e9009,c0e34d74,c87895c0,...) at syscallenter+0x329 syscall(e9d0dd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfe9ec, ebp = 0xbfbfea98 --- Tracing command linger2 pid 1983 tid 100181 td 0xc82145c0 cpustop_handler(8,e9cf86bc,c0c0a1f6,c0e35840,b,...) at cpustop_handler+0x34 ipi_nmi_handler(c0e35840,b,e9cf8644,c0e35840,c8774000,...) at ipi_nmi_handler+0x2f trap(e9cf86c8) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0c21a39, esp = 0xe9cf8708, ebp = 0xe9cf8724 --- DELAY(1,c81f5a80,c82145c0,c7e97948,e9cf8764,...) at DELAY+0x89 _mtx_lock_spin(c81f5a80,c82145c0,0,0,0,...) at _mtx_lock_spin+0x6c turnstile_trywait(c7e97948,c7e5a5a0,0,4d94ccf6,c7e97900,...) at turnstile_trywait+0xde _mtx_lock_sleep(c7e97948,c82145c0,0,0,0,...) at _mtx_lock_sleep+0x99 softdep_request_cleanup(c8277800,c84416b8,c8266c80,0,0,...) at softdep_request_cleanup+0x423 ffs_alloc(c8294414,c,0,8,0,...) at ffs_alloc+0x2ff ffs_balloc_ufs2(c84416b8,30000,0,4000,c8266c80,...) at ffs_balloc_ufs2+0x14c4 ffs_write(e9cf8b94,c0d13d4d,c8441710,0,c0c28152,...) at ffs_write+0x3f5 VOP_WRITE_APV(c0de2660,e9cf8b94,c84416b8,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c7ba4af0,e9cf8c28,c8266c80,0,c82145c0,...) at vn_write+0x1da dofilewrite(e9cf8c28,ffffffff,ffffffff,0,c7ba4af0,...) at dofilewrite+0x97 kern_writev(c82145c0,3,e9cf8c28,e9cf8c48,1,...) at kern_writev+0x58 write(c82145c0,e9cf8cec,e9cf8d28,c08e6fcb,c0e346f4,...) at write+0x4f syscallenter(c82145c0,e9cf8ce4,e9cf8ce4,0,c7be4000,...) at syscallenter+0x329 syscall(e9cf8d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfe9ec, ebp = 0xbfbfea98 --- Tracing command linger2 pid 1982 tid 100083 td 0xc7badb80 sched_switch(c7badb80,0,104,c0222a3f,e0,...) at sched_switch+0x2b6 mi_switch(104,0,c7badb80,db139a10,e9b484f4,...) at mi_switch+0x11b sleepq_switch(c7badb80,0,c0ce691d,260,50,...) at sleepq_switch+0xcc sleepq_wait(db139a10,50,c0ceab01,4,0,...) at sleepq_wait+0x39 __lockmgr_args(db139a10,81900,c82914fc,c0ceab01,50,...) at __lockmgr_args+0x893 getblk(c8291408,b7d00,0,4000,0,...) at getblk+0x1b1 breadn(c8291408,b7d00,0,4000,0,...) at breadn+0x44 bread(c8291408,b7d00,0,4000,0,...) at bread+0x4c ffs_blkfree_cg(35a60,0,4000,385,40,...) at ffs_blkfree_cg+0x272 ffs_blkfree(c7e97900,c8277800,c8291408,35a60,0,...) at ffs_blkfree+0x43 handle_workitem_freefrag(c83a2a40,c0de1948,e9b48778,c0b01def,20,...) at handle_workitem_freefrag+0x51 setup_allocindir_phase2(36368,0,35a60,0,0,...) at setup_allocindir_phase2+0x2ae softdep_setup_allocindir_page(c84e64fc,21,0,db3626d0,15,...) at softdep_setup_allocindir_page+0xe2 ffs_reallocblks(e9b489f4,c0d13cd6,c86e0810,c8371920,e9b48a28,...) at ffs_reallocblks+0xe74 VOP_REALLOCBLKS_APV(c0de2660,e9b489f4,0,4000,0,...) at VOP_REALLOCBLKS_APV+0x6b cluster_write(c86e0810,db0f8d40,90000,0,41,...) at cluster_write+0x3a2 ffs_write(e9b48b94,c0d13d4d,c86e0868,0,c0c28152,...) at ffs_write+0x637 VOP_WRITE_APV(c0de2660,e9b48b94,c86e0810,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c7b9aaf0,e9b48c28,c8266c80,0,c7badb80,...) at vn_write+0x1da dofilewrite(e9b48c28,ffffffff,ffffffff,0,c7b9aaf0,...) at dofilewrite+0x97 kern_writev(c7badb80,3,e9b48c28,e9b48c48,1,...) at kern_writev+0x58 write(c7badb80,e9b48cec,e9b48d28,c08e6fdc,c0e3586c,...) at write+0x4f syscallenter(c7badb80,e9b48ce4,e9b48ce4,0,c725fb80,...) at syscallenter+0x329 syscall(e9b48d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfe9ec, ebp = 0xbfbfea98 --- Tracing command linger2 pid 1981 tid 100165 td 0xc82112e0 sched_switch(c82112e0,0,104,c022702c,e0,...) at sched_switch+0x2b6 mi_switch(104,0,c82112e0,db139a10,e9cc85d0,...) at mi_switch+0x11b sleepq_switch(c82112e0,0,c0ce691d,260,50,...) at sleepq_switch+0xcc sleepq_wait(db139a10,50,c0ceab01,4,0,...) at sleepq_wait+0x39 __lockmgr_args(db139a10,81900,c82914fc,c0ceab01,50,...) at __lockmgr_args+0x893 getblk(c8291408,b7d00,0,4000,0,...) at getblk+0x1b1 breadn(c8291408,b7d00,0,4000,0,...) at breadn+0x44 bread(c8291408,b7d00,0,4000,0,...) at bread+0x4c ffs_alloccg(c841b984,2,37e80,0,4000,...) at ffs_alloccg+0x179 ffs_hashalloc(37e80,0,4000,c0af2de0,0,...) at ffs_hashalloc+0x3a ffs_alloc(c841b984,27,0,37e80,0,...) at ffs_alloc+0x1bb ffs_balloc_ufs2(c853cac0,9c000,0,4000,c8266c80,...) at ffs_balloc_ufs2+0x1af3 ffs_write(e9cc8b94,c0d13d4d,c853cb18,0,c0c28152,...) at ffs_write+0x3f5 VOP_WRITE_APV(c0de2660,e9cc8b94,c853cac0,25f,28,...) at VOP_WRITE_APV+0xcc vn_write(c7b9a0a8,e9cc8c28,c8266c80,0,c82112e0,...) at vn_write+0x1da dofilewrite(e9cc8c28,ffffffff,ffffffff,0,c7b9a0a8,...) at dofilewrite+0x97 kern_writev(c82112e0,3,e9cc8c28,e9cc8c48,1,...) at kern_writev+0x58 write(c82112e0,e9cc8cec,e9cc8c80,c08e6fcb,c0e346f4,...) at write+0x4f syscallenter(c82112e0,e9cc8ce4,c08e9056,c82112e0,c87895c0,...) at syscallenter+0x329 syscall(e9cc8d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfe9ec, ebp = 0xbfbfea98 --- Tracing command linger2 pid 1980 tid 100090 td 0xc7be52e0 sched_switch(c7be52e0,0,104,c0239183,e0,...) at sched_switch+0x2b6 mi_switch(104,0,c7be52e0,0,e9b646c0,...) at mi_switch+0x11b sleepq_switch(c7be52e0,0,c0ce691d,260,4f,...) at sleepq_switch+0xcc sleepq_wait(c8291280,4f,c0cea4e8,0,0,...) at sleepq_wait+0x39 _sleep(c8291280,c829124c,4f,c0cea4e8,0,...) at _sleep+0x2a2 bufobj_wwait(c829124c,3,0,0,0,...) at bufobj_wwait+0x43 ffs_syncvnode(c8291158,1,c7be52e0,0,0,...) at ffs_syncvnode+0x3b1 softdep_request_cleanup(c8277800,c8559c18,c8266c80,0,0,...) at softdep_request_cleanup+0x7de ffs_alloc(c86181d0,23,0,37dd0,0,...) at ffs_alloc+0x2ff ffs_balloc_ufs2(c8559c18,8c000,0,4000,c8266c80,...) at ffs_balloc_ufs2+0x1af3 ffs_write(e9b64b94,c0d13d4d,c8559c70,0,c0c28152,...) at ffs_write+0x3f5 VOP_WRITE_APV(c0de2660,e9b64b94,c8559c18,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c7b7e770,e9b64c28,c8266c80,0,c7be52e0,...) at vn_write+0x1da dofilewrite(e9b64c28,ffffffff,ffffffff,0,c7b7e770,...) at dofilewrite+0x97 kern_writev(c7be52e0,3,e9b64c28,e9b64c48,1,...) at kern_writev+0x58 write(c7be52e0,e9b64cec,e9b64d28,c08e6fdc,c0e3586c,...) at write+0x4f syscallenter(c7be52e0,e9b64ce4,e9b64ce4,0,c73bab80,...) at syscallenter+0x329 syscall(e9b64d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfe9ec, ebp = 0xbfbfea98 --- Tracing command linger2 pid 1979 tid 100102 td 0xc7be4000 kdb_enter(c0ce3759,c0ce3759,c0ce6eaf,e9b94718,1,...) at kdb_enter+0x3a panic(c0ce6eaf,ffffffff,7bc,cb,c7be4000,...) at panic+0x134 propagate_priority(c7be4000,c81f5a80,c0ce6e54,2d6,e9b94758,...) at propagate_priority+0x10c turnstile_wait(c81f5a80,c7be52e0,0,4d94ccf6,c7e97900,...) at turnstile_wait+0x2c8 _mtx_lock_sleep(c7e97948,c7be4000,0,0,0,...) at _mtx_lock_sleep+0x10c softdep_request_cleanup(c8277800,c8460560,c8266c80,0,0,...) at softdep_request_cleanup+0x4af ffs_alloc(c85402b8,2f,0,35410,0,...) at ffs_alloc+0x2ff ffs_balloc_ufs2(c8460560,bc000,0,4000,c8266c80,...) at ffs_balloc_ufs2+0x1af3 ffs_write(e9b94b94,c0d13d4d,c84605b8,0,c0c28152,...) at ffs_write+0x3f5 VOP_WRITE_APV(c0de2660,e9b94b94,c8460560,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c7b9aab8,e9b94c28,c8266c80,0,c7be4000,...) at vn_write+0x1da dofilewrite(e9b94c28,ffffffff,ffffffff,0,c7b9aab8,...) at dofilewrite+0x97 kern_writev(c7be4000,3,e9b94c28,e9b94c48,1,...) at kern_writev+0x58 write(c7be4000,e9b94cec,e9b94c80,c08e6fdc,c0e34b6c,...) at write+0x4f syscallenter(c7be4000,e9b94ce4,c08e9056,c7be4000,c73bab80,...) at syscallenter+0x329 syscall(e9b94d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfe9ec, ebp = 0xbfbfea98 --- Tracing command linger2 pid 1978 tid 100156 td 0xc8213000 sched_switch(c8213000,0,103,c0211fe5,e0,...) at sched_switch+0x2b6 mi_switch(103,0,c0ce6e54,2d6,c8213000,...) at mi_switch+0x11b turnstile_wait(c7394a00,c8878000,0,c7e5a5a0,c862115c,...) at turnstile_wait+0x2dc _mtx_lock_sleep(c0e4fac4,c8213000,0,0,0,...) at _mtx_lock_sleep+0x10c workitem_alloc(40,c0de1908,502,c862115c,2,...) at workitem_alloc+0x58 newallocindir(39a80,0,35ee0,0,0,...) at newallocindir+0x42 softdep_setup_allocindir_page(c862115c,27,0,db26d328,1b,...) at softdep_setup_allocindir_page+0x43 ffs_reallocblks(e9ca59f4,c0d13cd6,c86612b0,c84154e4,e9ca5a28,...) at ffs_reallocblks+0xe74 VOP_REALLOCBLKS_APV(c0de2660,e9ca59f4,0,4000,0,...) at VOP_REALLOCBLKS_APV+0x6b cluster_write(c86612b0,db1b133c,a8000,0,41,...) at cluster_write+0x3a2 ffs_write(e9ca5b94,c0d13d4d,c8661308,0,c0c28152,...) at ffs_write+0x637 VOP_WRITE_APV(c0de2660,e9ca5b94,c86612b0,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c7f438c0,e9ca5c28,c8266c80,0,c8213000,...) at vn_write+0x1da dofilewrite(e9ca5c28,ffffffff,ffffffff,0,c7f438c0,...) at dofilewrite+0x97 kern_writev(c8213000,3,e9ca5c28,e9ca5c48,1,...) at kern_writev+0x58 write(c8213000,e9ca5cec,e9ca5d28,c08e6fcb,c0e346f4,...) at write+0x4f syscallenter(c8213000,e9ca5ce4,e9ca5ce4,0,c87895c0,...) at syscallenter+0x329 syscall(e9ca5d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfe9ec, ebp = 0xbfbfea98 --- Tracing command linger2 pid 1977 tid 100086 td 0xc7be58a0 sched_switch(c7be58a0,0,104,c0229a9a,e0,...) at sched_switch+0x2b6 mi_switch(104,0,c7be58a0,db139a10,e9b545d0,...) at mi_switch+0x11b sleepq_switch(c7be58a0,0,c0ce691d,260,50,...) at sleepq_switch+0xcc sleepq_wait(db139a10,50,c0ceab01,4,0,...) at sleepq_wait+0x39 __lockmgr_args(db139a10,81900,c82914fc,c0ceab01,50,...) at __lockmgr_args+0x893 getblk(c8291408,b7d00,0,4000,0,...) at getblk+0x1b1 breadn(c8291408,b7d00,0,4000,0,...) at breadn+0x44 bread(c8291408,b7d00,0,4000,0,...) at bread+0x4c ffs_alloccg(c83ed488,2,38890,0,4000,...) at ffs_alloccg+0x179 ffs_hashalloc(38890,0,4000,c0af2de0,0,...) at ffs_hashalloc+0x3a ffs_alloc(c83ed488,20,0,38890,0,...) at ffs_alloc+0x1bb ffs_balloc_ufs2(c8706d70,80000,0,4000,c8266c80,...) at ffs_balloc_ufs2+0x1af3 ffs_write(e9b54b94,c0d13d4d,c8706dc8,0,c0c28152,...) at ffs_write+0x3f5 VOP_WRITE_APV(c0de2660,e9b54b94,c8706d70,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c7b9a1f8,e9b54c28,c8266c80,0,c7be58a0,...) at vn_write+0x1da dofilewrite(e9b54c28,ffffffff,ffffffff,0,c7b9a1f8,...) at dofilewrite+0x97 kern_writev(c7be58a0,3,e9b54c28,e9b54c48,1,...) at kern_writev+0x58 write(c7be58a0,e9b54cec,e9b54d28,c08e6fdc,c0e3586c,...) at write+0x4f syscallenter(c7be58a0,e9b54ce4,e9b54ce4,0,c7260000,...) at syscallenter+0x329 syscall(e9b54d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfe9ec, ebp = 0xbfbfea98 --- Tracing command linger2 pid 1976 tid 100202 td 0xc8878000 sched_switch(c8878000,0,103,c0219976,e0,...) at sched_switch+0x2b6 mi_switch(103,0,c0ce6e54,2d6,e9d5163c,...) at mi_switch+0x11b turnstile_wait(c81f5a80,c7be52e0,0,0,a228,...) at turnstile_wait+0x2dc _mtx_lock_sleep(c7e97948,c8878000,0,0,0,...) at _mtx_lock_sleep+0x10c ffs_blkfree_cg(38138,0,4000,37b,c0903d6c,...) at ffs_blkfree_cg+0x33f ffs_blkfree(c7e97900,c8277800,c8291408,38138,0,...) at ffs_blkfree+0x43 handle_workitem_freefrag(c0e4fac4,0,0,0,20,...) at handle_workitem_freefrag+0x51 setup_allocindir_phase2(38248,0,38138,0,0,...) at setup_allocindir_phase2+0x2ae softdep_setup_allocindir_page(c83fd414,32,0,db18dff4,26,...) at softdep_setup_allocindir_page+0xe2 ffs_reallocblks(e9d519f4,c0d13cd6,c86de158,c862ba7c,e9d51a28,...) at ffs_reallocblks+0xe74 VOP_REALLOCBLKS_APV(c0de2660,e9d519f4,0,4000,0,...) at VOP_REALLOCBLKS_APV+0x6b cluster_write(c86de158,db2d8524,d4000,0,41,...) at cluster_write+0x3a2 ffs_write(e9d51b94,c0d13d4d,c86de1b0,0,c0c28152,...) at ffs_write+0x637 VOP_WRITE_APV(c0de2660,e9d51b94,c86de158,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c7b9a5e8,e9d51c28,c8266c80,0,c8878000,...) at vn_write+0x1da dofilewrite(e9d51c28,ffffffff,ffffffff,0,c7b9a5e8,...) at dofilewrite+0x97 kern_writev(c8878000,3,e9d51c28,e9d51c48,1,...) at kern_writev+0x58 write(c8878000,e9d51cec,e9d51d28,c08e6fcb,c0e346f4,...) at write+0x4f syscallenter(c8878000,e9d51ce4,e9d51ce4,0,c7be4000,...) at syscallenter+0x329 syscall(e9d51d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfe9ec, ebp = 0xbfbfea98 --- Tracing command linger2 pid 1975 tid 100205 td 0xc88775c0 sched_switch(c88775c0,0,104,c0142068,e0,...) at sched_switch+0x2b6 mi_switch(104,0,c88775c0,db139a10,e9d5a500,...) at mi_switch+0x11b sleepq_switch(c88775c0,0,c0ce691d,260,50,...) at sleepq_switch+0xcc sleepq_wait(db139a10,50,c0ceab01,4,0,...) at sleepq_wait+0x39 __lockmgr_args(db139a10,81900,c82914fc,c0ceab01,50,...) at __lockmgr_args+0x893 getblk(c8291408,b7d00,0,4000,0,...) at getblk+0x1b1 breadn(c8291408,b7d00,0,4000,0,...) at breadn+0x44 bread(c8291408,b7d00,0,4000,0,...) at bread+0x4c ffs_blkfree_cg(417b0,0,4000,383,c83a43e0,...) at ffs_blkfree_cg+0x272 ffs_blkfree(c7e97900,c8277800,c8291408,417b0,0,...) at ffs_blkfree+0x43 handle_workitem_freefrag(0,e9d5a770,2,9e37,0,...) at handle_workitem_freefrag+0x51 process_worklist_item(9e37,0,8,0,c853bbb4,...) at process_worklist_item+0x241 softdep_request_cleanup(c8277800,c853bac0,c8266c80,0,0,...) at softdep_request_cleanup+0x38c ffs_alloc(c860b15c,25,0,47e10,0,...) at ffs_alloc+0x2ff ffs_balloc_ufs2(c853bac0,94000,0,4000,c8266c80,...) at ffs_balloc_ufs2+0x1af3 ffs_write(e9d5ab94,c0d13d4d,c853bb18,0,c0c28152,...) at ffs_write+0x3f5 VOP_WRITE_APV(c0de2660,e9d5ab94,c853bac0,25f,0,...) at VOP_WRITE_APV+0xcc vn_write(c7b9a5b0,e9d5ac28,c8266c80,0,c88775c0,...) at vn_write+0x1da dofilewrite(e9d5ac28,ffffffff,ffffffff,0,c7b9a5b0,...) at dofilewrite+0x97 kern_writev(c88775c0,3,e9d5ac28,e9d5ac48,1,...) at kern_writev+0x58 write(c88775c0,e9d5acec,e9d5ad28,c08e6fcb,c0e35a74,...) at write+0x4f syscallenter(c88775c0,e9d5ace4,e9d5ace4,0,c88775c0,...) at syscallenter+0x329 syscall(e9d5ad28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817c5b3, esp = 0xbfbfe9ec, ebp = 0xbfbfea98 --- Tracing command linger2 pid 1238 tid 100150 td 0xc7be3000 sched_switch(c7be3000,0,104,69a14d08,e0,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c8210810,e9c93b50,...) at mi_switch+0x11b sleepq_switch(c7be3000,0,c0ce691d,1a0,c7be3000,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,e9c93ba4,c08cf4ac,c8210810,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c8210810,5c,c0ce72c1,100,0,...) at sleepq_wait_sig+0x18 _sleep(c8210810,c8210898,15c,c0ce72c1,0,...) at _sleep+0x28c kern_wait(c7be3000,ffffffff,e9c93c44,0,0,...) at kern_wait+0xfa1 wait4(c7be3000,e9c93cec,e9c93d28,bfbfeaa8,c8210810,...) at wait4+0x3b syscallenter(c7be3000,e9c93ce4,e9c93ce4,0,e9c93d28,...) at syscallenter+0x329 syscall(e9c93d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x280ecd5b, esp = 0xbfbfea7c, ebp = 0xbfbfea98 --- Tracing command bash pid 1237 tid 100158 td 0xc82128a0 sched_switch(c82128a0,0,104,bd6f5bfc,39,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c820f000,e9cabb50,...) at mi_switch+0x11b sleepq_switch(c82128a0,0,c0ce691d,1a0,c82128a0,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,e9cabba4,c08cf4ac,c820f000,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c820f000,5c,c0ce72c1,100,0,...) at sleepq_wait_sig+0x18 _sleep(c820f000,c820f088,15c,c0ce72c1,0,...) at _sleep+0x28c kern_wait(c82128a0,ffffffff,e9cabc44,0,0,...) at kern_wait+0xfa1 wait4(c82128a0,e9cabcec,e9cabd28,283626e8,c820f000,...) at wait4+0x3b syscallenter(c82128a0,e9cabce4,c0c0a89e,c82128a0,e9cabd28,...) at syscallenter+0x329 syscall(e9cabd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x282bfd5b, esp = 0xbfbfe63c, ebp = 0xbfbfe658 --- Tracing command su pid 1236 tid 100154 td 0xc82135c0 sched_switch(c82135c0,0,104,bd29bbb1,39,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c820fac0,e9c9fb50,...) at mi_switch+0x11b sleepq_switch(c82135c0,0,c0ce691d,1a0,c82135c0,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,e9c9fba4,c08cf4ac,c820fac0,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c820fac0,5c,c0ce72c1,100,0,...) at sleepq_wait_sig+0x18 _sleep(c820fac0,c820fb48,15c,c0ce72c1,0,...) at _sleep+0x28c kern_wait(c82135c0,4d5,e9c9fc44,2,0,...) at kern_wait+0xfa1 wait4(c82135c0,e9c9fcec,e9c9fd28,281c16e8,c820fac0,...) at wait4+0x3b syscallenter(c82135c0,e9c9fce4,c0c0a89e,c82135c0,e9c9fd28,...) at syscallenter+0x329 syscall(e9c9fd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2811ed5b, esp = 0xbfbfe58c, ebp = 0xbfbfe5a8 --- Tracing command md5 pid 1231 tid 100151 td 0xc8214000 sched_switch(c8214000,0,104,c02254a8,e0,...) at sched_switch+0x2b6 mi_switch(104,0,c8214000,0,e9c96c6c,...) at mi_switch+0x11b sleepq_switch(c8214000,0,c0ce691d,260,24c,...) at sleepq_switch+0xcc sleepq_wait(c7b4b800,4c,c0cba8c6,0,0,...) at sleepq_wait+0x39 _sleep(c7b4b800,c7b4b820,24c,c0cba8c6,0,...) at _sleep+0x2a2 md_kthread(c7b4b800,e9c96d28,1ed,460,0,...) at md_kthread+0x168 fork_exit(c06d71e0,c7b4b800,e9c96d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe9c96d60, ebp = 0 --- Tracing command sh pid 1217 tid 100144 td 0xc81f7b80 sched_switch(c81f7b80,0,104,bcb5cb60,39,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c7fb4ac0,e9c7fb50,...) at mi_switch+0x11b sleepq_switch(c81f7b80,0,c0ce691d,1a0,c81f7b80,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,e9c7fba4,c08cf4ac,c7fb4ac0,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7fb4ac0,5c,c0ce72c1,100,0,...) at sleepq_wait_sig+0x18 _sleep(c7fb4ac0,c7fb4b48,15c,c0ce72c1,0,...) at _sleep+0x28c kern_wait(c81f7b80,ffffffff,e9c7fc44,2,0,...) at kern_wait+0xfa1 wait4(c81f7b80,e9c7fcec,e9c7fd28,8066634,c7fb4ac0,...) at wait4+0x3b syscallenter(c81f7b80,e9c7fce4,e9c7fce4,0,e9c7fd28,...) at syscallenter+0x329 syscall(e9c7fd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2815cd5b, esp = 0xbfbfe7cc, ebp = 0xbfbfe7e8 --- Tracing command time pid 1216 tid 100159 td 0xc82125c0 sched_switch(c82125c0,0,104,6e293747,39,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c820eac0,e9caeb50,...) at mi_switch+0x11b sleepq_switch(c82125c0,0,c0ce691d,1a0,c82125c0,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,e9caeba4,c08cf4ac,c820eac0,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c820eac0,5c,c0ce72c1,100,0,...) at sleepq_wait_sig+0x18 _sleep(c820eac0,c820eb48,15c,c0ce72c1,0,...) at _sleep+0x28c kern_wait(c82125c0,4c1,e9caec44,0,e9caebfc,...) at kern_wait+0xfa1 wait4(c82125c0,e9caecec,e9caed28,2819055c,c820eac0,...) at wait4+0x3b syscallenter(c82125c0,e9caece4,e9caece4,0,e9caed28,...) at syscallenter+0x329 syscall(e9caed28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x280edd5b, esp = 0xbfbfea0c, ebp = 0xbfbfead8 --- Tracing command tee pid 1211 tid 100142 td 0xc7bf1b80 sched_switch(c7bf1b80,0,104,bb58f552,39,...) at sched_switch+0x2b6 mi_switch(104,0,14c,c80742b0,e9c79b30,...) at mi_switch+0x11b sleepq_switch(c7bf1b80,0,c0ce691d,1a0,c7bf1b80,...) at sleepq_switch+0xcc sleepq_catch_signals(14c,0,e9c79b84,c08cf4ac,c7bb6620,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7bb6620,4c,c0ce6fb4,100,0,...) at sleepq_wait_sig+0x18 _sleep(c7bb6620,c7bb6790,14c,c0ce6fb4,0,...) at _sleep+0x28c pipe_read(c7fc52d8,e9c79c28,c80d1980,0,c7bf1b80,...) at pipe_read+0x4ca dofileread(e9c79c28,ffffffff,ffffffff,0,c7fc52d8,...) at dofileread+0x96 kern_readv(c7bf1b80,0,e9c79c28,e9c79c48,1,...) at kern_readv+0x58 read(c7bf1b80,e9c79cec,e9c79d28,28203000,c80742b0,...) at read+0x4f syscallenter(c7bf1b80,e9c79ce4,e9c79ce4,0,e9c79d28,...) at syscallenter+0x329 syscall(e9c79d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x2817c5d3, esp = 0xbfbfeaac, ebp = 0xbfbfeae8 --- Tracing command sh pid 1210 tid 100143 td 0xc7bf18a0 sched_switch(c7bf18a0,0,104,6d82b83b,39,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c8074000,e9c7cb50,...) at mi_switch+0x11b sleepq_switch(c7bf18a0,0,c0ce691d,1a0,c7bf18a0,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,e9c7cba4,c08cf4ac,c8074000,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c8074000,5c,c0ce72c1,100,0,...) at sleepq_wait_sig+0x18 _sleep(c8074000,c8074088,15c,c0ce72c1,0,...) at _sleep+0x28c kern_wait(c7bf18a0,ffffffff,e9c7cc44,2,0,...) at kern_wait+0xfa1 wait4(c7bf18a0,e9c7ccec,e9c7cd28,8066634,c8074000,...) at wait4+0x3b syscallenter(c7bf18a0,e9c7cce4,e9c7cce4,0,e9c7cd28,...) at syscallenter+0x329 syscall(e9c7cd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2815cd5b, esp = 0xbfbfe7dc, ebp = 0xbfbfe7f8 --- Tracing command bash pid 1206 tid 100157 td 0xc8212b80 sched_switch(c8212b80,0,104,6787b4f4,39,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c820f2b0,e9ca8b50,...) at mi_switch+0x11b sleepq_switch(c8212b80,0,c0ce691d,1a0,c8212b80,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,e9ca8ba4,c08cf4ac,c820f2b0,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c820f2b0,5c,c0ce72c1,100,0,...) at sleepq_wait_sig+0x18 _sleep(c820f2b0,c820f338,15c,c0ce72c1,0,...) at _sleep+0x28c kern_wait(c8212b80,ffffffff,e9ca8c44,6,0,...) at kern_wait+0xfa1 wait4(c8212b80,e9ca8cec,e9ca8d28,28456360,c820f2b0,...) at wait4+0x3b syscallenter(c8212b80,e9ca8ce4,e9ca8ce4,0,e9ca8d28,...) at syscallenter+0x329 syscall(e9ca8d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x282bfd5b, esp = 0xbfbfe6cc, ebp = 0xbfbfe6e8 --- Tracing command csh pid 1202 tid 100160 td 0xc82122e0 sched_switch(c82122e0,0,104,64d0bc83,38,...) at sched_switch+0x2b6 mi_switch(104,0,168,c820e810,e9cb1b9c,...) at mi_switch+0x11b sleepq_switch(c82122e0,0,c0ce691d,1a0,c82122e0,...) at sleepq_switch+0xcc sleepq_catch_signals(168,0,e9cb1bf0,c08cf4ac,c820e868,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c820e868,68,c0c9b7d5,100,0,...) at sleepq_wait_sig+0x18 _sleep(c820e868,c820e898,168,c0c9b7d5,0,...) at _sleep+0x28c kern_sigsuspend(c82122e0,0,0,0,0,...) at kern_sigsuspend+0xdd sigsuspend(c82122e0,e9cb1cec,e9cb1d28,2852c000,c820e810,...) at sigsuspend+0x4d syscallenter(c82122e0,e9cb1ce4,e9cb1ce4,0,e9cb1d28,...) at syscallenter+0x329 syscall(e9cb1d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x28179c0b, esp = 0xbfbfe60c, ebp = 0xbfbfe678 --- Tracing command su pid 1201 tid 100161 td 0xc8212000 sched_switch(c8212000,0,104,e2e3791,38,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c820e560,e9cb4b50,...) at mi_switch+0x11b sleepq_switch(c8212000,0,c0ce691d,1a0,c8212000,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,e9cb4ba4,c08cf4ac,c820e560,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c820e560,5c,c0ce72c1,100,0,...) at sleepq_wait_sig+0x18 _sleep(c820e560,c820e5e8,15c,c0ce72c1,0,...) at _sleep+0x28c kern_wait(c8212000,4b2,e9cb4c44,2,0,...) at kern_wait+0xfa1 wait4(c8212000,e9cb4cec,e9cb4d28,281c16e8,c820e560,...) at wait4+0x3b syscallenter(c8212000,e9cb4ce4,c0c0a89e,c8212000,e9cb4d28,...) at syscallenter+0x329 syscall(e9cb4d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2811ed5b, esp = 0xbfbfe62c, ebp = 0xbfbfe648 --- Tracing command bash pid 1197 tid 100137 td 0xc7fb8b80 sched_switch(c7fb8b80,0,104,15741982,37,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c80752b0,e9c6ab50,...) at mi_switch+0x11b sleepq_switch(c7fb8b80,0,c0ce691d,1a0,c7fb8b80,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,e9c6aba4,c08cf4ac,c80752b0,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c80752b0,5c,c0ce72c1,100,0,...) at sleepq_wait_sig+0x18 _sleep(c80752b0,c8075338,15c,c0ce72c1,0,...) at _sleep+0x28c kern_wait(c7fb8b80,ffffffff,e9c6ac44,6,0,...) at kern_wait+0xfa1 wait4(c7fb8b80,e9c6acec,e9c6ad28,28429fc0,c80752b0,...) at wait4+0x3b syscallenter(c7fb8b80,e9c6ace4,e9c6ace4,0,e9c6ad28,...) at syscallenter+0x329 syscall(e9c6ad28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x282bfd5b, esp = 0xbfbfe9cc, ebp = 0xbfbfe9e8 --- Tracing command sshd pid 1196 tid 100163 td 0xc82118a0 sched_switch(c82118a0,0,104,bb5a7d1c,39,...) at sched_switch+0x2b6 mi_switch(104,0,c8072240,c820e000,e9cbaa18,...) at mi_switch+0x11b sleepq_switch(c82118a0,0,c0ce691d,1a0,c82118a0,...) at sleepq_switch+0xcc sleepq_catch_signals(c8072240,0,e9cbaa60,c087f20b,c8072264,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c8072264,0,c0ce81c5,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c8072264,c8072250,0,40,c7b9ac40,...) at _cv_wait_sig+0x18b seltdwait(c7b9ac40,58,c8267200,c82118a0,c7fe0001,...) at seltdwait+0xd7 kern_select(c82118a0,a,286030b4,286030d8,0,0,20,c820e000,c820e000) at kern_select+0x4ee select(c82118a0,e9cbacec,e9cbad28,28662000,c820e000,...) at select+0x66 syscallenter(c82118a0,e9cbace4,e9cbace4,0,e9cbad28,...) at syscallenter+0x329 syscall(e9cbad28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x284a9553, esp = 0xbfbfe1cc, ebp = 0xbfbfe218 --- Tracing command sshd pid 1194 tid 100120 td 0xc7fba8a0 sched_switch(c7fba8a0,0,104,9980b218,35,...) at sched_switch+0x2b6 mi_switch(104,0,158,c7fb2810,e9c1aa8c,...) at mi_switch+0x11b sleepq_switch(c7fba8a0,0,c0ce691d,1a0,c7fba8a0,...) at sleepq_switch+0xcc sleepq_catch_signals(158,0,e9c1aae0,c08cf4ac,c7fcf0b8,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7fcf0b8,58,c0ce9e2f,100,0,...) at sleepq_wait_sig+0x18 _sleep(c7fcf0b8,c7fcf078,158,c0ce9e2f,0,...) at _sleep+0x28c sbwait(c7fcf054,1,e9c1ab38,c0945fa5,c74ec9c0,...) at sbwait+0x52 soreceive_generic(c7fcf000,0,e9c1ac28,0,0,...) at soreceive_generic+0x3b1 soreceive(c7fcf000,0,e9c1ac28,0,0,0) at soreceive+0x38 soo_read(c7c18b28,e9c1ac28,c7b37d80,0,c7fba8a0,...) at soo_read+0x4e dofileread(e9c1ac28,ffffffff,ffffffff,0,c7c18b28,...) at dofileread+0x96 kern_readv(c7fba8a0,5,e9c1ac28,e9c1ac48,1,...) at kern_readv+0x58 read(c7fba8a0,e9c1acec,e9c1ad28,2860800c,c7fb2810,...) at read+0x4f syscallenter(c7fba8a0,e9c1ace4,e9c1ace4,0,e9c1ad28,...) at syscallenter+0x329 syscall(e9c1ad28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x284a95d3, esp = 0xbfbfdd9c, ebp = 0xbfbfddd8 --- Tracing command getty pid 1193 tid 100108 td 0xc7bed8a0 sched_switch(c7bed8a0,0,104,cd490e0e,21,...) at sched_switch+0x2b6 mi_switch(104,0,c7bed8a0,c7be2810,e9bb6aa4,...) at mi_switch+0x11b sleepq_switch(c7bed8a0,0,c0ce691d,1a0,c7bed8a0,...) at sleepq_switch+0xcc sleepq_catch_signals(c7bed8a0,1,e9bb6aec,c087f20b,c74d8070,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c74d8070,0,c0ce8e5a,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c74d8070,c0e2feb0,0,c74d8000,e9bb6b58,...) at _cv_wait_sig+0x18b tty_wait(c74d8000,c74d8070,e9bb6c28,1,0,...) at tty_wait+0x23 ttydisc_read(c74d8000,e9bb6c28,0,e9bb6ba8,e9bb6ba4,...) at ttydisc_read+0x1cc ttydev_read(c73d2c00,e9bb6c28,0,0,1,...) at ttydev_read+0x10a devfs_read_f(c7fc5230,e9bb6c28,c7154100,0,c7bed8a0,...) at devfs_read_f+0x7e dofileread(e9bb6c28,ffffffff,ffffffff,0,c7fc5230,...) at dofileread+0x96 kern_readv(c7bed8a0,0,e9bb6c28,e9bb6c48,1,...) at kern_readv+0x58 read(c7bed8a0,e9bb6cec,e9bb6d28,e9bb6c88,c0c0d9f0,...) at read+0x4f syscallenter(c7bed8a0,e9bb6ce4,e9bb6ce4,0,e9bb6d28,...) at syscallenter+0x329 syscall(e9bb6d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f5d3, esp = 0xbfbfedac, ebp = 0xbfbfedd8 --- Tracing command getty pid 1192 tid 100129 td 0xc7b7c2e0 sched_switch(c7b7c2e0,0,104,cd37cf20,21,...) at sched_switch+0x2b6 mi_switch(104,0,c7b7c2e0,c7b782b0,e9c35aa4,...) at mi_switch+0x11b sleepq_switch(c7b7c2e0,0,c0ce691d,1a0,c7b7c2e0,...) at sleepq_switch+0xcc sleepq_catch_signals(c7b7c2e0,1,e9c35aec,c087f20b,c74d8270,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c74d8270,0,c0ce8e5a,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c74d8270,c0e2feb0,0,c74d8200,e9c35b58,...) at _cv_wait_sig+0x18b tty_wait(c74d8200,c74d8270,e9c35c28,1,0,...) at tty_wait+0x23 ttydisc_read(c74d8200,e9c35c28,0,e9c35ba8,e9c35ba4,...) at ttydisc_read+0x1cc ttydev_read(c74c8000,e9c35c28,0,0,1,...) at ttydev_read+0x10a devfs_read_f(c7b7fc40,e9c35c28,c7154100,0,c7b7c2e0,...) at devfs_read_f+0x7e dofileread(e9c35c28,ffffffff,ffffffff,0,c7b7fc40,...) at dofileread+0x96 kern_readv(c7b7c2e0,0,e9c35c28,e9c35c48,1,...) at kern_readv+0x58 read(c7b7c2e0,e9c35cec,e9c35d28,e9c35c88,c0c0d9f0,...) at read+0x4f syscallenter(c7b7c2e0,e9c35ce4,e9c35ce4,0,e9c35d28,...) at syscallenter+0x329 syscall(e9c35d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f5d3, esp = 0xbfbfedac, ebp = 0xbfbfedd8 --- Tracing command getty pid 1191 tid 100130 td 0xc7b7c000 sched_switch(c7b7c000,0,104,cd44e9db,21,...) at sched_switch+0x2b6 mi_switch(104,0,c7b7c000,c7b78000,e9c39aa4,...) at mi_switch+0x11b sleepq_switch(c7b7c000,0,c0ce691d,1a0,c7b7c000,...) at sleepq_switch+0xcc sleepq_catch_signals(c7b7c000,1,e9c39aec,c087f20b,c73d7a70,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c73d7a70,0,c0ce8e5a,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c73d7a70,c0e2feb0,0,c73d7a00,e9c39b58,...) at _cv_wait_sig+0x18b tty_wait(c73d7a00,c73d7a70,e9c39c28,1,0,...) at tty_wait+0x23 ttydisc_read(c73d7a00,e9c39c28,0,e9c39ba8,e9c39ba4,...) at ttydisc_read+0x1cc ttydev_read(c74c8100,e9c39c28,0,0,1,...) at ttydev_read+0x10a devfs_read_f(c7ba4700,e9c39c28,c7154100,0,c7b7c000,...) at devfs_read_f+0x7e dofileread(e9c39c28,ffffffff,ffffffff,0,c7ba4700,...) at dofileread+0x96 kern_readv(c7b7c000,0,e9c39c28,e9c39c48,1,...) at kern_readv+0x58 read(c7b7c000,e9c39cec,e9c39d28,e9c39c88,c0c0d9f0,...) at read+0x4f syscallenter(c7b7c000,e9c39ce4,e9c39ce4,0,e9c39d28,...) at syscallenter+0x329 syscall(e9c39d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f5d3, esp = 0xbfbfedac, ebp = 0xbfbfedd8 --- Tracing command getty pid 1190 tid 100087 td 0xc7bf15c0 sched_switch(c7bf15c0,0,104,cd504225,21,...) at sched_switch+0x2b6 mi_switch(104,0,c7bf15c0,c7bec560,e9b58aa4,...) at mi_switch+0x11b sleepq_switch(c7bf15c0,0,c0ce691d,1a0,c7bf15c0,...) at sleepq_switch+0xcc sleepq_catch_signals(c7bf15c0,1,e9b58aec,c087f20b,c7367670,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7367670,0,c0ce8e5a,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c7367670,c0e2feb0,0,c7367600,e9b58b58,...) at _cv_wait_sig+0x18b tty_wait(c7367600,c7367670,e9b58c28,1,0,...) at tty_wait+0x23 ttydisc_read(c7367600,e9b58c28,0,e9b58ba8,e9b58ba4,...) at ttydisc_read+0x1cc ttydev_read(c74c8200,e9b58c28,0,0,1,...) at ttydev_read+0x10a devfs_read_f(c7f43af0,e9b58c28,c7154100,0,c7bf15c0,...) at devfs_read_f+0x7e dofileread(e9b58c28,ffffffff,ffffffff,0,c7f43af0,...) at dofileread+0x96 kern_readv(c7bf15c0,0,e9b58c28,e9b58c48,1,...) at kern_readv+0x58 read(c7bf15c0,e9b58cec,e9b58d28,e9b58c88,c0c0d9f0,...) at read+0x4f syscallenter(c7bf15c0,e9b58ce4,e9b58ce4,0,e9b58d28,...) at syscallenter+0x329 syscall(e9b58d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f5d3, esp = 0xbfbfedac, ebp = 0xbfbfedd8 --- Tracing command getty pid 1189 tid 100105 td 0xc7bf02e0 sched_switch(c7bf02e0,0,104,cd41f00f,21,...) at sched_switch+0x2b6 mi_switch(104,0,c7bf02e0,c7beb2b0,e9ba6aa4,...) at mi_switch+0x11b sleepq_switch(c7bf02e0,0,c0ce691d,1a0,c7bf02e0,...) at sleepq_switch+0xcc sleepq_catch_signals(c7bf02e0,1,e9ba6aec,c087f20b,c7367870,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7367870,0,c0ce8e5a,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c7367870,c0e2feb0,0,c7367800,e9ba6b58,...) at _cv_wait_sig+0x18b tty_wait(c7367800,c7367870,e9ba6c28,1,0,...) at tty_wait+0x23 ttydisc_read(c7367800,e9ba6c28,0,e9ba6ba8,e9ba6ba4,...) at ttydisc_read+0x1cc ttydev_read(c74c8500,e9ba6c28,0,0,1,...) at ttydev_read+0x10a devfs_read_f(c7c18310,e9ba6c28,c7154100,0,c7bf02e0,...) at devfs_read_f+0x7e dofileread(e9ba6c28,ffffffff,ffffffff,0,c7c18310,...) at dofileread+0x96 kern_readv(c7bf02e0,0,e9ba6c28,e9ba6c48,1,...) at kern_readv+0x58 read(c7bf02e0,e9ba6cec,e9ba6d28,e9ba6c88,c0c0d9f0,...) at read+0x4f syscallenter(c7bf02e0,e9ba6ce4,e9ba6ce4,0,e9ba6d28,...) at syscallenter+0x329 syscall(e9ba6d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f5d3, esp = 0xbfbfedac, ebp = 0xbfbfedd8 --- Tracing command getty pid 1188 tid 100100 td 0xc7bf0b80 sched_switch(c7bf0b80,0,104,cd2e800e,21,...) at sched_switch+0x2b6 mi_switch(104,0,c7bf0b80,c7bebac0,e9b8caa4,...) at mi_switch+0x11b sleepq_switch(c7bf0b80,0,c0ce691d,1a0,c7bf0b80,...) at sleepq_switch+0xcc sleepq_catch_signals(c7bf0b80,1,e9b8caec,c087f20b,c7367c70,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7367c70,0,c0ce8e5a,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c7367c70,c0e2feb0,0,c7367c00,e9b8cb58,...) at _cv_wait_sig+0x18b tty_wait(c7367c00,c7367c70,e9b8cc28,1,0,...) at tty_wait+0x23 ttydisc_read(c7367c00,e9b8cc28,0,e9b8cba8,e9b8cba4,...) at ttydisc_read+0x1cc ttydev_read(c74c8600,e9b8cc28,0,0,1,...) at ttydev_read+0x10a devfs_read_f(c7fc53b8,e9b8cc28,c7154100,0,c7bf0b80,...) at devfs_read_f+0x7e dofileread(e9b8cc28,ffffffff,ffffffff,0,c7fc53b8,...) at dofileread+0x96 kern_readv(c7bf0b80,0,e9b8cc28,e9b8cc48,1,...) at kern_readv+0x58 read(c7bf0b80,e9b8ccec,e9b8cd28,e9b8cc88,c0c0d9f0,...) at read+0x4f syscallenter(c7bf0b80,e9b8cce4,e9b8cce4,0,e9b8cd28,...) at syscallenter+0x329 syscall(e9b8cd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f5d3, esp = 0xbfbfedac, ebp = 0xbfbfedd8 --- Tracing command getty pid 1187 tid 100080 td 0xc7bae5c0 sched_switch(c7bae5c0,0,104,cd409b2d,21,...) at sched_switch+0x2b6 mi_switch(104,0,c7bae5c0,c7ba9560,e9b3caa4,...) at mi_switch+0x11b sleepq_switch(c7bae5c0,0,c0ce691d,1a0,c7bae5c0,...) at sleepq_switch+0xcc sleepq_catch_signals(c7bae5c0,1,e9b3caec,c087f20b,c73d7670,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c73d7670,0,c0ce8e5a,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c73d7670,c0e2feb0,0,c73d7600,e9b3cb58,...) at _cv_wait_sig+0x18b tty_wait(c73d7600,c73d7670,e9b3cc28,1,0,...) at tty_wait+0x23 ttydisc_read(c73d7600,e9b3cc28,0,e9b3cba8,e9b3cba4,...) at ttydisc_read+0x1cc ttydev_read(c74c8700,e9b3cc28,0,0,1,...) at ttydev_read+0x10a devfs_read_f(c7c18af0,e9b3cc28,c7154100,0,c7bae5c0,...) at devfs_read_f+0x7e dofileread(e9b3cc28,ffffffff,ffffffff,0,c7c18af0,...) at dofileread+0x96 kern_readv(c7bae5c0,0,e9b3cc28,e9b3cc48,1,...) at kern_readv+0x58 read(c7bae5c0,e9b3ccec,e9b3cd28,e9b3cc88,c0c0d9f0,...) at read+0x4f syscallenter(c7bae5c0,e9b3cce4,e9b3cce4,0,e9b3cd28,...) at syscallenter+0x329 syscall(e9b3cd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f5d3, esp = 0xbfbfedac, ebp = 0xbfbfedd8 --- Tracing command getty pid 1186 tid 100133 td 0xc7b795c0 sched_switch(c7b795c0,0,104,cd2461b2,21,...) at sched_switch+0x2b6 mi_switch(104,0,c7b795c0,c7ab8560,e9c5daa4,...) at mi_switch+0x11b sleepq_switch(c7b795c0,0,c0ce691d,1a0,c7b795c0,...) at sleepq_switch+0xcc sleepq_catch_signals(c7b795c0,1,e9c5daec,c087f20b,c73d7870,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c73d7870,0,c0ce8e5a,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c73d7870,c0e2feb0,0,c73d7800,e9c5db58,...) at _cv_wait_sig+0x18b tty_wait(c73d7800,c73d7870,e9c5dc28,1,0,...) at tty_wait+0x23 ttydisc_read(c73d7800,e9c5dc28,0,e9c5dba8,e9c5dba4,...) at ttydisc_read+0x1cc ttydev_read(c74c8800,e9c5dc28,0,0,1,...) at ttydev_read+0x10a devfs_read_f(c7f43348,e9c5dc28,c7154100,0,c7b795c0,...) at devfs_read_f+0x7e dofileread(e9c5dc28,ffffffff,ffffffff,0,c7f43348,...) at dofileread+0x96 kern_readv(c7b795c0,0,e9c5dc28,e9c5dc48,1,...) at kern_readv+0x58 read(c7b795c0,e9c5dcec,e9c5dd28,e9c5dc88,c0c0d9f0,...) at read+0x4f syscallenter(c7b795c0,e9c5dce4,e9c5dce4,0,e9c5dd28,...) at syscallenter+0x329 syscall(e9c5dd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f5d3, esp = 0xbfbfedac, ebp = 0xbfbfedd8 --- Tracing command getty pid 1185 tid 100091 td 0xc7bad8a0 sched_switch(c7bad8a0,0,104,6d966199,39,...) at sched_switch+0x2b6 mi_switch(104,0,2,c7ba8810,e9b68aa4,...) at mi_switch+0x11b sleepq_switch(c7bad8a0,0,c0ce691d,1a0,c7bad8a0,...) at sleepq_switch+0xcc sleepq_catch_signals(2,0,e9b68aec,c087f20b,c7367a70,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7367a70,0,c0ce8e5a,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c7367a70,c7367a04,0,c7367a00,e9b68b58,...) at _cv_wait_sig+0x18b tty_wait(c7367a00,c7367a70,e9b68c28,1,0,...) at tty_wait+0x23 ttydisc_read(c7367a00,e9b68c28,0,e9b68ba8,e9b68ba4,...) at ttydisc_read+0x1cc ttydev_read(c7146900,e9b68c28,0,0,1,...) at ttydev_read+0x10a devfs_read_f(c7f43118,e9b68c28,c7154100,0,c7bad8a0,...) at devfs_read_f+0x7e dofileread(e9b68c28,ffffffff,ffffffff,0,c7f43118,...) at dofileread+0x96 kern_readv(c7bad8a0,0,e9b68c28,e9b68c48,1,...) at kern_readv+0x58 read(c7bad8a0,e9b68cec,e9b68d28,e9b68c88,c0c0d9f0,...) at read+0x4f syscallenter(c7bad8a0,e9b68ce4,e9b68ce4,0,e9b68d28,...) at syscallenter+0x329 syscall(e9b68d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f5d3, esp = 0xbfbfedac, ebp = 0xbfbfedd8 --- Tracing command inetd pid 1146 tid 100145 td 0xc81f78a0 sched_switch(c81f78a0,0,104,c132bc57,21,...) at sched_switch+0x2b6 mi_switch(104,0,c7be6500,c7fb4810,e9c82a18,...) at mi_switch+0x11b sleepq_switch(c81f78a0,0,c0ce691d,1a0,c81f78a0,...) at sleepq_switch+0xcc sleepq_catch_signals(c7be6500,0,e9c82a60,c087f20b,c7be6524,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7be6524,0,c0ce81c5,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c7be6524,c7be6510,0,0,c7c18ab8,...) at _cv_wait_sig+0x18b seltdwait(c7c18ab8,58,c7154100,c81f78a0,0,...) at seltdwait+0xd7 kern_select(c81f78a0,6,bfbfdfc0,0,0,0,20,c7fb4810,c7fb4810) at kern_select+0x4ee select(c81f78a0,e9c82cec,e9c82d28,28221000,c7fb4810,...) at select+0x66 syscallenter(c81f78a0,e9c82ce4,e9c82ce4,0,e9c82d28,...) at syscallenter+0x329 syscall(e9c82d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x281a3553, esp = 0xbfbfdf7c, ebp = 0xbfbfee08 --- Tracing command moused pid 1120 tid 100141 td 0xc7fb8000 sched_switch(c7fb8000,0,104,bc24760a,21,...) at sched_switch+0x2b6 mi_switch(104,0,c7be6980,c8074560,e9c76a18,...) at mi_switch+0x11b sleepq_switch(c7fb8000,0,c0ce691d,1a0,c7fb8000,...) at sleepq_switch+0xcc sleepq_catch_signals(c7be6980,0,e9c76a60,c087f20b,c7be69a4,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7be69a4,0,c0ce81c5,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c7be69a4,c7be6990,c0de8860,c73d2e00,c7b9a070,...) at _cv_wait_sig+0x18b seltdwait(c7b9a070,58,c7154100,c7fb8000,0,...) at seltdwait+0xd7 kern_select(c7fb8000,6,bfbfea24,0,0,0,20,c8074560,c8074560) at kern_select+0x4ee select(c7fb8000,e9c76cec,e9c76d28,281d6668,c8074560,...) at select+0x66 syscallenter(c7fb8000,e9c76ce4,e9c76ce4,0,e9c76d28,...) at syscallenter+0x329 syscall(e9c76d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x281ac553, esp = 0xbfbfe9bc, ebp = 0xbfbfeb48 --- Tracing command cron pid 1096 tid 100103 td 0xc7bad000 sched_switch(c7bad000,0,104,d78e8165,d3,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c7ba8000,e9b99b88,...) at mi_switch+0x11b sleepq_switch(c7bad000,0,c0ce691d,1a0,c7bad200,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,100,e9b99be0,c08cf478,...) at sleepq_catch_signals+0x52 sleepq_timedwait_sig(c0e30704,5c,c0ce4150,100,0,...) at sleepq_timedwait_sig+0x1c _sleep(c0e30704,0,15c,c0ce4150,ea61,...) at _sleep+0x258 kern_nanosleep(c7bad000,e9b99c34,e9b99c3c,3c,0,...) at kern_nanosleep+0xc1 nanosleep(c7bad000,e9b99cec,e9b99d28,281c3668,c7ba8000,...) at nanosleep+0x6f syscallenter(c7bad000,e9b99ce4,e9b99ce4,0,e9b99d28,...) at syscallenter+0x329 syscall(e9b99d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x2817bf57, esp = 0xbfbfec7c, ebp = 0xbfbfeca8 --- Tracing command sendmail pid 1089 tid 100116 td 0xc7be85c0 sched_switch(c7be85c0,0,104,a7667155,21,...) at sched_switch+0x2b6 mi_switch(104,0,168,c7fb3560,e9c0eb9c,...) at mi_switch+0x11b sleepq_switch(c7be85c0,0,c0ce691d,1a0,c7be85c0,...) at sleepq_switch+0xcc sleepq_catch_signals(168,0,e9c0ebf0,c08cf4ac,c7fb35b8,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7fb35b8,68,c0c9b7d5,100,0,...) at sleepq_wait_sig+0x18 _sleep(c7fb35b8,c7fb35e8,168,c0c9b7d5,0,...) at _sleep+0x28c kern_sigsuspend(c7be85c0,0,0,0,0,...) at kern_sigsuspend+0xdd sigsuspend(c7be85c0,e9c0ecec,e9c0ed28,bfbffff4,c7fb3560,...) at sigsuspend+0x4d syscallenter(c7be85c0,e9c0ece4,e9c0ece4,0,e9c0ed28,...) at syscallenter+0x329 syscall(e9c0ed28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x28334c0b, esp = 0xbfbfcf8c, ebp = 0xbfbfcfb8 --- Tracing command sendmail pid 1085 tid 100121 td 0xc7fba5c0 sched_switch(c7fba5c0,0,104,6115c9c6,e0,...) at sched_switch+0x2b6 mi_switch(104,0,c7ad5580,c7fb2560,e9c1da14,...) at mi_switch+0x11b sleepq_switch(c7fba5c0,0,c0ce691d,1a0,c7fba7c0,...) at sleepq_switch+0xcc sleepq_catch_signals(c7ad5580,0,c7fba5c0,e9c1da60,c087ed63,...) at sleepq_catch_signals+0x52 sleepq_timedwait_sig(c7ad55a4,0,c0ce81c5,101,0,...) at sleepq_timedwait_sig+0x1c _cv_timedwait_sig(c7ad55a4,c7ad5590,1389,c719b700,c7fba5c0,...) at _cv_timedwait_sig+0x1a3 seltdwait(e9c1dbf8,e9c1dc00,c719b700,c7fba5c0,e9c1dab8,...) at seltdwait+0xc1 kern_select(c7fba5c0,5,bfbfc500,0,0,e9c1dc40,20,5,0) at kern_select+0x4ee select(c7fba5c0,e9c1dcec,e9c1dd28,e9c1dc88,c0c0d9f0,...) at select+0x66 syscallenter(c7fba5c0,e9c1dce4,e9c1dce4,0,369e99,...) at syscallenter+0x329 syscall(e9c1dd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x283db553, esp = 0xbfbfc46c, ebp = 0xbfbfcf98 --- Tracing command sshd pid 1070 tid 100097 td 0xc7be48a0 sched_switch(c7be48a0,0,104,979127c5,35,...) at sched_switch+0x2b6 mi_switch(104,0,c7ad6bc0,c7be0810,e9b80a18,...) at mi_switch+0x11b sleepq_switch(c7be48a0,0,c0ce691d,1a0,c7be48a0,...) at sleepq_switch+0xcc sleepq_catch_signals(c7ad6bc0,0,e9b80a60,c087f20b,c7ad6be4,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7ad6be4,0,c0ce81c5,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c7ad6be4,c7ad6bd0,58,c7ad4900,c7be48a0,...) at _cv_wait_sig+0x18b seltdwait(c7fc50a8,58,c7ad4900,c7be48a0,0,...) at seltdwait+0xd7 kern_select(c7be48a0,5,286090ac,0,0,0,20,c7be0810,c7be0810) at kern_select+0x4ee select(c7be48a0,e9b80cec,e9b80d28,2861a070,c7be0810,...) at select+0x66 syscallenter(c7be48a0,e9b80ce4,c0c0a89e,c7be48a0,e9b80d28,...) at syscallenter+0x329 syscall(e9b80d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x284a9553, esp = 0xbfbfe28c, ebp = 0xbfbfee28 --- Tracing command ntpd pid 1025 tid 100113 td 0xc7bed000 sched_switch(c7bed000,0,104,8d4619dd,e0,...) at sched_switch+0x2b6 mi_switch(104,0,c7be6dc0,c7fb4000,e9c05a18,...) at mi_switch+0x11b sleepq_switch(c7bed000,0,c0ce691d,1a0,c7bed000,...) at sleepq_switch+0xcc sleepq_catch_signals(c7be6dc0,0,e9c05a60,c087f20b,c7be6de4,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7be6de4,0,c0ce81c5,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c7be6de4,c7be6dd0,58,c7154100,c7bed000,...) at _cv_wait_sig+0x18b seltdwait(c7ba4070,58,c7154100,c7bed000,0,...) at seltdwait+0xd7 kern_select(c7bed000,1b,bfbfed18,0,0,0,20,c7fb4000,c7fb4000) at kern_select+0x4ee select(c7bed000,e9c05cec,e9c05d80,e9c05c88,c0c0d9f0,...) at select+0x66 syscallenter(c7bed000,e9c05ce4,c0c0a5bb,fffffffe,3,...) at syscallenter+0x329 syscall(e9c05d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x28355553, esp = 0xbfbfecec, ebp = 0xbfbfedb8 --- Tracing command nfsd pid 899 tid 100128 td 0xc7be35c0 sched_switch(c7be35c0,0,104,fbfd4667,df,...) at sched_switch+0x2b6 mi_switch(104,0,c74baa80,c7be22b0,e9c32bc8,...) at mi_switch+0x11b sleepq_switch(c7be35c0,0,c0ce691d,1a0,c7be37c0,...) at sleepq_switch+0xcc sleepq_catch_signals(c74baa80,0,c7be35c0,e9c32c14,c087ed63,...) at sleepq_catch_signals+0x52 sleepq_timedwait_sig(c7ad5a10,0,c0cfeab7,101,0,...) at sleepq_timedwait_sig+0x1c _cv_timedwait_sig(c7ad5a10,c74baa80,1388,0,0,...) at _cv_timedwait_sig+0x1a3 svc_run_internal(e9c32d14,c0899d31,c74baa80,e9c32d28,3d6b6361,...) at svc_run_internal+0x30d svc_thread_start(c74baa80,e9c32d28,3d6b6361,0,f,...) at svc_thread_start+0x10 fork_exit(c0acb730,c74baa80,e9c32d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0x804c72e, eip = 0xc, esp = 0x33, ebp = 0 --- Tracing command nfsd pid 899 tid 100127 td 0xc7be38a0 sched_switch(c7be38a0,0,104,fbfe0f9f,df,...) at sched_switch+0x2b6 mi_switch(104,0,c74baa80,c7be22b0,e9c2fbc8,...) at mi_switch+0x11b sleepq_switch(c7be38a0,0,c0ce691d,1a0,c7be3aa0,...) at sleepq_switch+0xcc sleepq_catch_signals(c74baa80,0,c7be38a0,e9c2fc14,c087ed63,...) at sleepq_catch_signals+0x52 sleepq_timedwait_sig(c7a94510,0,c0cfeab7,101,0,...) at sleepq_timedwait_sig+0x1c _cv_timedwait_sig(c7a94510,c74baa80,1388,725f1282,6f635f63,...) at _cv_timedwait_sig+0x1a3 svc_run_internal(e9c2fd14,c0899d31,c74baa80,e9c2fd28,28328d00,...) at svc_run_internal+0x30d svc_thread_start(c74baa80,e9c2fd28,28328d00,28328cd4,0,...) at svc_thread_start+0x10 fork_exit(c0acb730,c74baa80,e9c2fd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0x804c72e, eip = 0xc, esp = 0x33, ebp = 0 --- Tracing command nfsd pid 899 tid 100126 td 0xc7be3b80 sched_switch(c7be3b80,0,104,fbfdb36b,df,...) at sched_switch+0x2b6 mi_switch(104,0,c74baa80,c7be22b0,e9c2cbc8,...) at mi_switch+0x11b sleepq_switch(c7be3b80,0,c0ce691d,1a0,c7be3d80,...) at sleepq_switch+0xcc sleepq_catch_signals(c74baa80,0,c7be3b80,e9c2cc14,c087ed63,...) at sleepq_catch_signals+0x52 sleepq_timedwait_sig(c7ad6ad0,0,c0cfeab7,101,0,...) at sleepq_timedwait_sig+0x1c _cv_timedwait_sig(c7ad6ad0,c74baa80,1388,0,0,...) at _cv_timedwait_sig+0x1a3 svc_run_internal(e9c2cd14,c0899d31,c74baa80,e9c2cd28,0,...) at svc_run_internal+0x30d svc_thread_start(c74baa80,e9c2cd28,0,0,0,...) at svc_thread_start+0x10 fork_exit(c0acb730,c74baa80,e9c2cd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0x804c72e, eip = 0xc, esp = 0x33, ebp = 0 --- Tracing command nfsd pid 899 tid 100110 td 0xc7bed2e0 sched_switch(c7bed2e0,0,104,fbe1dbf3,df,...) at sched_switch+0x2b6 mi_switch(104,0,c74baa80,c7be22b0,e9bbea9c,...) at mi_switch+0x11b sleepq_switch(c7bed2e0,0,c0ce691d,1a0,c7bed4e0,...) at sleepq_switch+0xcc sleepq_catch_signals(c74baa80,0,c7bed2e0,e9bbeae8,c087ed63,...) at sleepq_catch_signals+0x52 sleepq_timedwait_sig(c7ad5790,0,c0cfeab7,101,0,...) at sleepq_timedwait_sig+0x1c _cv_timedwait_sig(c7ad5790,c74baa80,1388,c0e351c0,e9bbeb08,...) at _cv_timedwait_sig+0x1a3 svc_run_internal(c7bed448,14,c0cfec63,c0cfdbe8,bfbfe9a7,...) at svc_run_internal+0x30d svc_run(c74baa80,0,1,a,c1bafdf8,...) at svc_run+0x7b nfssvc_nfsd(bfbfe890,e9bbec0c,c,c7154100,e9bbec20,...) at nfssvc_nfsd+0xed nfssvc_nfsserver(c7bed2e0,e9bbecec,c7be22b0,0,c7bed2e0,...) at nfssvc_nfsserver+0x1ff nfssvc(c7bed2e0,e9bbecec,e9bbed28,bfbffff4,c7be22b0,...) at nfssvc+0x63 syscallenter(c7bed2e0,e9bbece4,e9bbece4,0,e9bbed28,...) at syscallenter+0x329 syscall(e9bbed28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x280d980b, esp = 0xbfbfe84c, ebp = 0xbfbfeab8 --- Tracing command nfsd pid 898 tid 100107 td 0xc7bedb80 sched_switch(c7bedb80,0,104,3561e986,1f,...) at sched_switch+0x2b6 mi_switch(104,0,c7ad6dc0,c7be2ac0,e9bb2a18,...) at mi_switch+0x11b sleepq_switch(c7bedb80,0,c0ce691d,1a0,c7bedb80,...) at sleepq_switch+0xcc sleepq_catch_signals(c7ad6dc0,0,e9bb2a60,c087f20b,c7ad6de4,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7ad6de4,0,c0ce81c5,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c7ad6de4,c7ad6dd0,58,c7154100,c7bedb80,...) at _cv_wait_sig+0x18b seltdwait(c7c188f8,58,c7154100,c7bedb80,0,...) at seltdwait+0xd7 kern_select(c7bedb80,5,bfbfecd0,0,0,0,20,c7be2ac0,c7be2ac0) at kern_select+0x4ee select(c7bedb80,e9bb2cec,e9bb2d28,bfbffff4,c7be2ac0,...) at select+0x66 syscallenter(c7bedb80,e9bb2ce4,c0c0a89e,c7bedb80,e9bb2d28,...) at syscallenter+0x329 syscall(e9bb2d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x2817f553, esp = 0xbfbfeabc, ebp = 0xbfbfee08 --- Tracing command mountd pid 889 tid 100109 td 0xc7bed5c0 sched_switch(c7bed5c0,0,104,bc9cfcdd,39,...) at sched_switch+0x2b6 mi_switch(104,0,c7b1c980,c7be2560,e9bbaa18,...) at mi_switch+0x11b sleepq_switch(c7bed5c0,0,c0ce691d,1a0,c7bed5c0,...) at sleepq_switch+0xcc sleepq_catch_signals(c7b1c980,0,e9bbaa60,c087f20b,c7b1c9a4,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7b1c9a4,0,c0ce81c5,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c7b1c9a4,c7b1c990,58,c7154100,c7bed5c0,...) at _cv_wait_sig+0x18b seltdwait(c7c18c08,58,c7154100,c7bed5c0,c095466a,...) at seltdwait+0xd7 kern_select(c7bed5c0,9,bfbfed6c,0,0,0,20,c7be2560,c7be2560) at kern_select+0x4ee select(c7bed5c0,e9bbacec,e9bbad28,281aa6fc,c7be2560,...) at select+0x66 syscallenter(c7bed5c0,e9bbace4,e9bbace4,0,e9bbad28,...) at syscallenter+0x329 syscall(e9bbad28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x28193553, esp = 0xbfbfed3c, ebp = 0xbfbfee08 --- Tracing command rpcbind pid 792 tid 100118 td 0xc7fbb000 sched_switch(c7fbb000,0,104,d370db2e,db,...) at sched_switch+0x2b6 mi_switch(104,0,c7a962c0,c7fb3000,e9c14a6c,...) at mi_switch+0x11b sleepq_switch(c7fbb000,0,c0ce691d,1a0,c7fbb200,...) at sleepq_switch+0xcc sleepq_catch_signals(c7a962c0,0,c7fbb000,e9c14ab8,c087ed63,...) at sleepq_catch_signals+0x52 sleepq_timedwait_sig(c7a962e4,0,c0ce81c5,101,0,...) at sleepq_timedwait_sig+0x1c _cv_timedwait_sig(c7a962e4,c7a962d0,7531,c7154100,c7fbb000,...) at _cv_timedwait_sig+0x1a3 seltdwait(e9c14c2c,e9c14c34,c7154100,c7fbb000,c090d4e0,...) at seltdwait+0xc1 poll(c7fbb000,e9c14cec,e9c14d28,2818f5a0,c7fb3000,...) at poll+0x33f syscallenter(c7fbb000,e9c14ce4,e9c14ce4,0,e9c14d28,...) at syscallenter+0x329 syscall(e9c14d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (209, FreeBSD ELF32, poll), eip = 0x28137eff, esp = 0xbfbfcbcc, ebp = 0xbfbfed98 --- Tracing command syslogd pid 765 tid 100123 td 0xc7fba000 cpustop_handler(1,e9c23630,c0c0a1f6,7ffed35c,369e99,...) at cpustop_handler+0x34 ipi_nmi_handler(7ffed35c,369e99,0,0,c7fb2000,...) at ipi_nmi_handler+0x2f trap(e9c2363c) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0c21a39, esp = 0xe9c2367c, ebp = 0xe9c23698 --- DELAY(1,c7fba000,3,e1fbd000,e9c236e0,...) at DELAY+0x89 _mtx_lock_spin(c0e42cf4,c7fba000,0,0,0,...) at _mtx_lock_spin+0x6c smp_tlb_shootdown(e9c23708,c0c0198f,e1fbd000,e1fc1000,0,...) at smp_tlb_shootdown+0x9a smp_invlpg_range(e1fbd000,e1fc1000,0,c0ceab3b,e1fc1000,...) at smp_invlpg_range+0x1c pmap_invalidate_range(c0ea1740,e1fbd000,e1fc1000) at pmap_invalidate_range+0x4f pmap_qremove(e1fbd000,4,0,0,320,...) at pmap_qremove+0x54 vfs_vmio_release(db3a1844,80200,0,0,0,...) at vfs_vmio_release+0x1d2 getnewbuf(2800,4000,0,0,0,...) at getnewbuf+0x3a3 getblk(c7fb5000,2,0,2800,0,...) at getblk+0x41b breadn(c7fb5000,2,0,2800,0,...) at breadn+0x44 bread(c7fb5000,2,0,2800,0,...) at bread+0x4c ffs_balloc_ufs2(c7fb5000,a016,0,5c,c7154100,...) at ffs_balloc_ufs2+0xf15 ffs_write(e9c23bb0,c0d13d4d,c7fb5058,0,c0c28152,...) at ffs_write+0x3f5 VOP_WRITE_APV(c0de2660,e9c23bb0,c7fb5000,25f,c0b2fc11,...) at VOP_WRITE_APV+0xcc vn_write(c7b7fb98,c83eb500,c7154100,0,c7fba000,...) at vn_write+0x1da dofilewrite(c83eb500,ffffffff,ffffffff,0,c7b7fb98,...) at dofilewrite+0x97 kern_writev(c7fba000,b,c83eb500,c83eb500,c7fb2000,...) at kern_writev+0x58 writev(c7fba000,e9c23cec,e9c23d80,e9c23c88,c0c0d9f0,...) at writev+0x46 syscallenter(c7fba000,e9c23ce4,c0c0a5bb,0,0,...) at syscallenter+0x329 syscall(e9c23d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (121, FreeBSD ELF32, writev), eip = 0x28178beb, esp = 0xbfbfd60c, ebp = 0xbfbfdd08 --- Tracing command devd pid 617 tid 100098 td 0xc7be45c0 sched_switch(c7be45c0,0,104,bc880563,39,...) at sched_switch+0x2b6 mi_switch(104,0,c7b1c200,c7be0560,e9b84a18,...) at mi_switch+0x11b sleepq_switch(c7be45c0,0,c0ce691d,1a0,c7be45c0,...) at sleepq_switch+0xcc sleepq_catch_signals(c7b1c200,0,e9b84a60,c087f20b,c7b1c224,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7b1c224,0,c0ce81c5,101,0,...) at sleepq_wait_sig+0x18 _cv_wait_sig(c7b1c224,c7b1c210,58,c7154100,c7be45c0,...) at _cv_wait_sig+0x18b seltdwait(c7b7e700,58,c7154100,c7be45c0,58,...) at seltdwait+0xd7 kern_select(c7be45c0,5,bfbfe990,0,0,0,20,c7be0560,c7be0560) at kern_select+0x4ee select(c7be45c0,e9b84cec,e9b84d28,28104010,c7be0560,...) at select+0x66 syscallenter(c7be45c0,e9b84ce4,e9b84ce4,0,e9b84d28,...) at syscallenter+0x329 syscall(e9b84d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x8088143, esp = 0xbfbfe95c, ebp = 0xbfbfee38 --- Tracing command flowcleaner pid 21 tid 100076 td 0xc73ba8a0 sched_switch(c73ba8a0,0,104,8fd50aec,dc,...) at sched_switch+0x2b6 mi_switch(104,0,c73ba8a0,0,e7940c74,...) at mi_switch+0x11b sleepq_switch(c73ba8a0,0,c0ce691d,283,20,...) at sleepq_switch+0xcc sleepq_timedwait(c0e44470,0,c0ced678,1,0,...) at sleepq_timedwait+0x3e _cv_timedwait(c0e44470,c0e44478,4e20,622,0,...) at _cv_timedwait+0x1a3 flowtable_cleaner(0,e7940d28,0,0,0,...) at flowtable_cleaner+0x244 fork_exit(c0972750,0,e7940d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7940d60, ebp = 0 --- Tracing command softdepflush pid 20 tid 100075 td 0xc73bab80 sched_switch(c73bab80,0,104,c023f3b5,e0,...) at sched_switch+0x2b6 mi_switch(104,0,c73bab80,0,e793dc60,...) at mi_switch+0x11b sleepq_switch(c73bab80,0,c0ce691d,283,44,...) at sleepq_switch+0xcc sleepq_timedwait(c0e4fb20,44,c0d01368,0,0,...) at sleepq_timedwait+0x3e _sleep(c0e4fb20,c0e4fac4,44,c0d01368,3e8,...) at _sleep+0x272 softdep_flush(0,e793dd28,0,0,0,...) at softdep_flush+0x30e fork_exit(c0b08680,0,e793dd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe793dd60, ebp = 0 --- Tracing command syncer pid 19 tid 100074 td 0xc74ca000 sched_switch(c74ca000,0,104,c0764a19,e0,...) at sched_switch+0x2b6 mi_switch(104,0,c74ca000,0,e793a9dc,...) at mi_switch+0x11b sleepq_switch(c74ca000,0,c0ce691d,260,44,...) at sleepq_switch+0xcc sleepq_wait(c0de4b00,44,c0d04ae2,0,0,...) at sleepq_wait+0x39 _sleep(c0de4b00,c0e50440,44,c0d04ae2,0,...) at _sleep+0x2a2 getpbuf(c0de4b00,80300,c870524c,0,0,...) at getpbuf+0x88 cluster_wbuild(c8705158,4000,3a,0,6,...) at cluster_wbuild+0x251 vfs_bio_awrite(db203950,0,0,0,0,...) at vfs_bio_awrite+0x2c3 ffs_syncvnode(c8705158,3,c74ca000,c094df31,e793abe4,...) at ffs_syncvnode+0x32e ffs_sync(c7e5a5a0,3,2,c0dd1e60,e793ac10,...) at ffs_sync+0x380 sync_fsync(e793ac58,c0d13bd4,e793ac58,c8291000,e793ac78,...) at sync_fsync+0x24d VOP_FSYNC_APV(c0dd1e60,e793ac58,c0cec286,6c6,0,...) at VOP_FSYNC_APV+0x64 sync_vnode(c0e44278,c0e44264,3e8,764,4e20,...) at sync_vnode+0x1c1 sched_sync(0,e793ad28,0,0,0,...) at sched_sync+0x2c2 fork_exit(c095b110,0,e793ad28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe793ad60, ebp = 0 --- Tracing command vnlru pid 18 tid 100073 td 0xc74ca2e0 sched_switch(c74ca2e0,0,104,b377f1fc,e0,...) at sched_switch+0x2b6 mi_switch(104,0,c74ca2e0,0,e7937c2c,...) at mi_switch+0x11b sleepq_switch(c74ca2e0,0,c0ce691d,283,250,...) at sleepq_switch+0xcc sleepq_timedwait(c7ab72b0,50,c0ceccd7,0,0,...) at sleepq_timedwait+0x3e _sleep(c7ab72b0,c0e44238,250,c0ceccd7,3e8,...) at _sleep+0x272 vnlru_proc(0,e7937d28,0,0,0,...) at vnlru_proc+0x10f fork_exit(c09595b0,0,e7937d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7937d60, ebp = 0 --- Tracing command bufdaemon pid 17 tid 100072 td 0xc74ca5c0 sched_switch(c74ca5c0,0,104,5eb40df6,e0,...) at sched_switch+0x2b6 mi_switch(104,0,c74ca5c0,0,e7934c78,...) at mi_switch+0x11b sleepq_switch(c74ca5c0,0,c0ce691d,283,44,...) at sleepq_switch+0xcc sleepq_timedwait(c0e43fa8,44,c0cea957,0,0,...) at sleepq_timedwait+0x3e _sleep(c0e43fa8,c0e43fac,44,c0cea957,3e8,...) at _sleep+0x272 buf_daemon(0,e7934d28,0,0,0,...) at buf_daemon+0x186 fork_exit(c093edc0,0,e7934d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7934d60, ebp = 0 --- Tracing command pagezero pid 16 tid 100071 td 0xc74ca8a0 sched_switch(c74ca8a0,0,104,7f2de94d,99,...) at sched_switch+0x2b6 mi_switch(104,0,c74ca8a0,0,e7931c80,...) at mi_switch+0x11b sleepq_switch(c74ca8a0,0,c0ce691d,283,0,...) at sleepq_switch+0xcc sleepq_timedwait(c0e507f4,0,c0d04df2,0,0,...) at sleepq_timedwait+0x3e _sleep(c0e507f4,c0e503b0,0,c0d04df2,493e0,...) at _sleep+0x272 vm_pagezero(0,e7931d28,0,0,0,...) at vm_pagezero+0xd2 fork_exit(c0b4b890,0,e7931d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7931d60, ebp = 0 --- Tracing command vmdaemon pid 15 tid 100070 td 0xc74cab80 sched_switch(c74cab80,0,104,bffb219a,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c74cab80,0,e792ec24,...) at mi_switch+0x11b sleepq_switch(c74cab80,0,c0ce691d,260,68,...) at sleepq_switch+0xcc sleepq_wait(c0e5041c,68,c0cea957,0,0,...) at sleepq_wait+0x39 _sleep(c0e5041c,c0e50420,68,c0cea957,0,...) at _sleep+0x2a2 vm_daemon(0,e792ed28,0,0,0,...) at vm_daemon+0x80 fork_exit(c0b482b0,0,e792ed28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe792ed60, ebp = 0 --- Tracing command pagedaemon pid 9 tid 100069 td 0xc74cb000 sched_switch(c74cb000,0,104,bbca9093,e0,...) at sched_switch+0x2b6 mi_switch(104,0,c74cb000,0,e792bbb0,...) at mi_switch+0x11b sleepq_switch(c74cb000,0,c0ce691d,283,44,...) at sleepq_switch+0xcc sleepq_timedwait(c0e503e4,44,c0cea957,0,0,...) at sleepq_timedwait+0x3e _sleep(c0e503e4,c0e503b0,44,c0cea957,1388,...) at _sleep+0x272 vm_pageout(0,e792bd28,0,0,0,...) at vm_pageout+0x2e3 fork_exit(c0b46d10,0,e792bd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe792bd60, ebp = 0 --- Tracing command xpt_thrd pid 8 tid 100068 td 0xc74cb2e0 sched_switch(c74cb2e0,0,104,b0711176,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c74cb2e0,0,e7928c70,...) at mi_switch+0x11b sleepq_switch(c74cb2e0,0,c0ce691d,260,4c,...) at sleepq_switch+0xcc sleepq_wait(c0dfcc54,4c,c0c82769,0,0,...) at sleepq_wait+0x39 _sleep(c0dfcc54,c0dfcc98,4c,c0c82769,0,...) at _sleep+0x2a2 xpt_scanner_thread(0,e7928d28,0,0,0,...) at xpt_scanner_thread+0x4f fork_exit(c04874b0,0,e7928d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7928d60, ebp = 0 --- Tracing command sctp_iterator pid 7 tid 100067 td 0xc74cb5c0 sched_switch(c74cb5c0,0,104,a8611f28,1e,...) at sched_switch+0x2b6 mi_switch(104,0,c74cb5c0,0,e7925c7c,...) at mi_switch+0x11b sleepq_switch(c74cb5c0,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c0e45cb8,0,c0cf6873,0,0,...) at sleepq_wait+0x39 _sleep(c0e45cb8,c0e45c80,0,c0cf6873,0,...) at _sleep+0x2a2 sctp_iterator_thread(0,e7925d28,0,0,0,...) at sctp_iterator_thread+0x7c fork_exit(c09eba30,0,e7925d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7925d60, ebp = 0 --- Tracing command fdc0 pid 6 tid 100064 td 0xc74cc000 sched_switch(c74cc000,0,104,ab0d7f58,e0,...) at sched_switch+0x2b6 mi_switch(104,0,c74cc000,0,e791cbcc,...) at mi_switch+0x11b sleepq_switch(c74cc000,0,c0ce691d,283,4c,...) at sleepq_switch+0xcc sleepq_timedwait(c73d743c,4c,c0cdbf52,0,0,...) at sleepq_timedwait+0x3e _sleep(c73d743c,c73d74f0,4c,c0cdbf52,3e8,...) at _sleep+0x272 fdc_thread(c73d7400,e791cd28,0,0,0,...) at fdc_thread+0x3ed fork_exit(c0bcae30,c73d7400,e791cd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe791cd60, ebp = 0 --- Tracing command fw0_probe pid 5 tid 100062 td 0xc73a85c0 sched_switch(c73a85c0,0,104,d222b210,1b,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c7157ac0,e790c78c,...) at mi_switch+0x11b sleepq_switch(c73a85c0,0,c0ce691d,1a0,c73a85c0,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,e790c7e0,c08cf4ac,c73de000,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c73de000,5c,c0cdbf52,100,0,...) at sleepq_wait_sig+0x18 _sleep(c73de000,c73e2488,15c,c0cdbf52,0,...) at _sleep+0x28c fw_bus_probe_thread(c73de000,e790cd28,0,0,0,...) at fw_bus_probe_thread+0xb45 fork_exit(c067ee70,c73de000,e790cd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe790cd60, ebp = 0 --- Tracing command usb pid 14 tid 100058 td 0xc73b82e0 sched_switch(c73b82e0,0,104,b06c171f,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73b82e0,0,e78f9c80,...) at mi_switch+0x11b sleepq_switch(c73b82e0,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73bfd34,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c73bfd34,c73bfdd4,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c73bfd2c,e78f9d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73bfd2c,e78f9d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78f9d60, ebp = 0 --- Tracing command usb pid 14 tid 100057 td 0xc73b85c0 sched_switch(c73b85c0,0,104,62f9627,e0,...) at sched_switch+0x2b6 mi_switch(104,0,c73b85c0,0,e78f6c80,...) at mi_switch+0x11b sleepq_switch(c73b85c0,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73bfd04,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c73bfd04,c73bfdd4,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c73bfcfc,e78f6d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73bfcfc,e78f6d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78f6d60, ebp = 0 --- Tracing command usb pid 14 tid 100056 td 0xc73b88a0 sched_switch(c73b88a0,0,104,b0065b4e,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73b88a0,0,e78f3c80,...) at mi_switch+0x11b sleepq_switch(c73b88a0,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73bfcd4,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c73bfcd4,c73bfdd4,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c73bfccc,e78f3d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73bfccc,e78f3d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78f3d60, ebp = 0 --- Tracing command usb pid 14 tid 100055 td 0xc73b8b80 sched_switch(c73b8b80,0,104,b0065172,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73b8b80,0,e78f0c80,...) at mi_switch+0x11b sleepq_switch(c73b8b80,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73bfca4,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c73bfca4,c73bfdd4,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c73bfc9c,e78f0d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73bfc9c,e78f0d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78f0d60, ebp = 0 --- Tracing command usb pid 14 tid 100053 td 0xc73ba2e0 sched_switch(c73ba2e0,0,104,b006443b,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73ba2e0,0,e78e7c80,...) at mi_switch+0x11b sleepq_switch(c73ba2e0,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73b2b5c,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c73b2b5c,c73b2bfc,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c73b2b54,e78e7d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73b2b54,e78e7d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78e7d60, ebp = 0 --- Tracing command usb pid 14 tid 100052 td 0xc73ba5c0 sched_switch(c73ba5c0,0,104,fe3e5b0c,df,...) at sched_switch+0x2b6 mi_switch(104,0,c73ba5c0,0,e78e4c80,...) at mi_switch+0x11b sleepq_switch(c73ba5c0,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73b2b2c,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c73b2b2c,c73b2bfc,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c73b2b24,e78e4d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73b2b24,e78e4d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78e4d60, ebp = 0 --- Tracing command usb pid 14 tid 100051 td 0xc73a6000 sched_switch(c73a6000,0,104,afa35804,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73a6000,0,e78e1c80,...) at mi_switch+0x11b sleepq_switch(c73a6000,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73b2afc,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c73b2afc,c73b2bfc,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c73b2af4,e78e1d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73b2af4,e78e1d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78e1d60, ebp = 0 --- Tracing command usb pid 14 tid 100050 td 0xc73a62e0 sched_switch(c73a62e0,0,104,afa34acd,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73a62e0,0,e78dec80,...) at mi_switch+0x11b sleepq_switch(c73a62e0,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73b2acc,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c73b2acc,c73b2bfc,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c73b2ac4,e78ded28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73b2ac4,e78ded28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78ded60, ebp = 0 --- Tracing command usb pid 14 tid 100049 td 0xc73a65c0 sched_switch(c73a65c0,0,104,afa33ed7,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73a65c0,0,e78dac80,...) at mi_switch+0x11b sleepq_switch(c73a65c0,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73aeb5c,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c73aeb5c,c73aebfc,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c73aeb54,e78dad28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73aeb54,e78dad28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78dad60, ebp = 0 --- Tracing command usb pid 14 tid 100048 td 0xc73a68a0 sched_switch(c73a68a0,0,104,fdec3080,df,...) at sched_switch+0x2b6 mi_switch(104,0,c73a68a0,0,e78d7c80,...) at mi_switch+0x11b sleepq_switch(c73a68a0,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73aeb2c,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c73aeb2c,c73aebfc,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c73aeb24,e78d7d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73aeb24,e78d7d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78d7d60, ebp = 0 --- Tracing command usb pid 14 tid 100047 td 0xc73a6b80 sched_switch(c73a6b80,0,104,af40377a,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73a6b80,0,e78d4c80,...) at mi_switch+0x11b sleepq_switch(c73a6b80,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73aeafc,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c73aeafc,c73aebfc,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c73aeaf4,e78d4d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73aeaf4,e78d4d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78d4d60, ebp = 0 --- Tracing command usb pid 14 tid 100046 td 0xc73a7000 sched_switch(c73a7000,0,104,af402bc2,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73a7000,0,e78d1c80,...) at mi_switch+0x11b sleepq_switch(c73a7000,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73aeacc,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c73aeacc,c73aebfc,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c73aeac4,e78d1d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73aeac4,e78d1d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78d1d60, ebp = 0 --- Tracing command usb pid 14 tid 100045 td 0xc73a72e0 sched_switch(c73a72e0,0,104,af401e4c,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73a72e0,0,e78cdc80,...) at mi_switch+0x11b sleepq_switch(c73a72e0,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73a1b5c,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c73a1b5c,c73a1bfc,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c73a1b54,e78cdd28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73a1b54,e78cdd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78cdd60, ebp = 0 --- Tracing command usb pid 14 tid 100044 td 0xc73a75c0 sched_switch(c73a75c0,0,104,fdb50549,df,...) at sched_switch+0x2b6 mi_switch(104,0,c73a75c0,0,e78cac80,...) at mi_switch+0x11b sleepq_switch(c73a75c0,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73a1b2c,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c73a1b2c,c73a1bfc,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c73a1b24,e78cad28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73a1b24,e78cad28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78cad60, ebp = 0 --- Tracing command usb pid 14 tid 100043 td 0xc73a78a0 sched_switch(c73a78a0,0,104,aedd7b09,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73a78a0,0,e78c7c80,...) at mi_switch+0x11b sleepq_switch(c73a78a0,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73a1afc,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c73a1afc,c73a1bfc,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c73a1af4,e78c7d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73a1af4,e78c7d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78c7d60, ebp = 0 --- Tracing command usb pid 14 tid 100042 td 0xc73a7b80 sched_switch(c73a7b80,0,104,aedd705c,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73a7b80,0,e78c4c80,...) at mi_switch+0x11b sleepq_switch(c73a7b80,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c73a1acc,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c73a1acc,c73a1bfc,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c73a1ac4,e78c4d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c73a1ac4,e78c4d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78c4d60, ebp = 0 --- Tracing command usb pid 14 tid 100040 td 0xc7393000 sched_switch(c7393000,0,104,aedd64af,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c7393000,0,e78bdc80,...) at mi_switch+0x11b sleepq_switch(c7393000,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c739eb5c,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c739eb5c,c739ebfc,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c739eb54,e78bdd28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c739eb54,e78bdd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78bdd60, ebp = 0 --- Tracing command usb pid 14 tid 100039 td 0xc73932e0 sched_switch(c73932e0,0,104,fd62a544,df,...) at sched_switch+0x2b6 mi_switch(104,0,c73932e0,0,e78bac80,...) at mi_switch+0x11b sleepq_switch(c73932e0,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c739eb2c,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c739eb2c,c739ebfc,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c739eb24,e78bad28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c739eb24,e78bad28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78bad60, ebp = 0 --- Tracing command usb pid 14 tid 100038 td 0xc73935c0 sched_switch(c73935c0,0,104,ae7abd6c,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73935c0,0,e78b7c80,...) at mi_switch+0x11b sleepq_switch(c73935c0,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c739eafc,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c739eafc,c739ebfc,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c739eaf4,e78b7d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c739eaf4,e78b7d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78b7d60, ebp = 0 --- Tracing command usb pid 14 tid 100037 td 0xc73938a0 sched_switch(c73938a0,0,104,ae7ab32d,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73938a0,0,e78b4c80,...) at mi_switch+0x11b sleepq_switch(c73938a0,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c739eacc,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c739eacc,c739ebfc,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c739eac4,e78b4d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c739eac4,e78b4d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78b4d60, ebp = 0 --- Tracing command usb pid 14 tid 100035 td 0xc7395000 sched_switch(c7395000,0,104,ae7aa536,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c7395000,0,e78adc80,...) at mi_switch+0x11b sleepq_switch(c7395000,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c738db5c,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c738db5c,c738dbfc,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c738db54,e78add28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c738db54,e78add28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78add60, ebp = 0 --- Tracing command usb pid 14 tid 100034 td 0xc73952e0 sched_switch(c73952e0,0,104,fd1064d7,df,...) at sched_switch+0x2b6 mi_switch(104,0,c73952e0,0,e78aac80,...) at mi_switch+0x11b sleepq_switch(c73952e0,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c738db2c,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c738db2c,c738dbfc,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c738db24,e78aad28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c738db24,e78aad28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78aad60, ebp = 0 --- Tracing command usb pid 14 tid 100033 td 0xc73955c0 sched_switch(c73955c0,0,104,ae105a80,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73955c0,0,e78a7c80,...) at mi_switch+0x11b sleepq_switch(c73955c0,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c738dafc,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c738dafc,c738dbfc,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c738daf4,e78a7d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c738daf4,e78a7d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78a7d60, ebp = 0 --- Tracing command usb pid 14 tid 100032 td 0xc73958a0 sched_switch(c73958a0,0,104,ae104c60,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73958a0,0,e78a4c80,...) at mi_switch+0x11b sleepq_switch(c73958a0,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c738dacc,0,c0cdbf52,1,0,...) at sleepq_wait+0x39 _cv_wait(c738dacc,c738dbfc,c0cd52ab,68,0,...) at _cv_wait+0x18b usb_process(c738dac4,e78a4d28,0,0,0,...) at usb_process+0x176 fork_exit(c07ec250,c738dac4,e78a4d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78a4d60, ebp = 0 --- Tracing command yarrow pid 13 tid 100017 td 0xc725f5c0 sched_switch(c725f5c0,0,104,b84bcc3e,e0,...) at sched_switch+0x2b6 mi_switch(104,0,c725f5c0,0,c6f2fc48,...) at mi_switch+0x11b sleepq_switch(c725f5c0,0,c0ce691d,283,0,...) at sleepq_switch+0xcc sleepq_timedwait(c0e30564,0,c0cdbf52,2,0,...) at sleepq_timedwait+0x3e _sleep(c0e30564,0,0,c0cdbf52,64,...) at _sleep+0x272 pause(c0cdbf52,64,0,0,0,...) at pause+0x30 random_kthread(0,c6f2fd28,0,0,0,...) at random_kthread+0x225 fork_exit(c0758a20,0,c6f2fd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f2fd60, ebp = 0 --- Tracing command g_down pid 4 tid 100015 td 0xc725fb80 sched_switch(c725fb80,0,104,c0258d6f,e0,...) at sched_switch+0x2b6 mi_switch(104,0,c725fb80,0,c6f29c30,...) at mi_switch+0x11b sleepq_switch(c725fb80,0,c0ce691d,260,24c,...) at sleepq_switch+0xcc sleepq_wait(c0e2e264,4c,c0cdbf52,0,0,...) at sleepq_wait+0x39 _sleep(c0e2e264,c0e2e1c8,24c,c0cdbf52,0,...) at _sleep+0x2a2 g_io_schedule_down(c725fb80,4c,c0cdcaf1,72,0,...) at g_io_schedule_down+0x58 g_down_procbody(0,c6f29d28,0,0,0,...) at g_down_procbody+0x68 fork_exit(c085f2a0,0,c6f29d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f29d60, ebp = 0 --- Tracing command g_up pid 3 tid 100014 td 0xc7260000 cpustop_handler(4,c6f26b70,c0c0a1f6,c08b2d5d,2,...) at cpustop_handler+0x34 ipi_nmi_handler(c08b2d5d,2,80,c6f26b08,c7156000,...) at ipi_nmi_handler+0x2f trap(c6f26b7c) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0bffbd2, esp = 0xc6f26bbc, ebp = 0xc6f26be4 --- smp_tlb_shootdown(c6f26c0c,c0c0198f,e42c1000,e42d1000,c7b0ee10,...) at smp_tlb_shootdown+0xd2 smp_invlpg_range(e42c1000,e42d1000,c7b0ee10,14000,e42d1000,...) at smp_invlpg_range+0x1c pmap_invalidate_range(c0ea1740,e42c1000,e42d1000) at pmap_invalidate_range+0x4f pmap_qremove(e42c1000,10,db0e8d5c,c8705904,c867b390,...) at pmap_qremove+0x54 cluster_callback(db0e8d5c,db0e8d5c,c867b390,c6f26c74,c0862dd5,...) at cluster_callback+0x34 bufdone(db0e8d5c,c7aa1d10) at bufdone+0x37 g_vfs_done(c867b390,c867b390,c085e3e7,c0e2e1e8,c7260000,...) at g_vfs_done+0x85 biodone(c867b390,c0e2e1e8,24c,c0cdbf52,0,...) at biodone+0xb9 g_io_schedule_up(c7260000,4c,c0cdcaf1,5b,0,...) at g_io_schedule_up+0x79 g_up_procbody(0,c6f26d28,0,0,0,...) at g_up_procbody+0x68 fork_exit(c085f310,0,c6f26d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f26d60, ebp = 0 --- Tracing command g_event pid 2 tid 100013 td 0xc715a2e0 sched_switch(c715a2e0,0,104,bc8679b8,39,...) at sched_switch+0x2b6 mi_switch(104,0,c715a2e0,0,c6f23c4c,...) at mi_switch+0x11b sleepq_switch(c715a2e0,0,c0ce691d,260,24c,...) at sleepq_switch+0xcc sleepq_wait(c0e2e258,4c,c0cdbf52,0,0,...) at sleepq_wait+0x39 _sleep(c0e2e258,c0e2e1a8,24c,c0cdbf52,0,...) at _sleep+0x2a2 g_run_events(c715a2e0,4c,c0cdcaf1,89) at g_run_events+0x49b g_event_procbody(0,c6f23d28,0,0,0,...) at g_event_procbody+0x64 fork_exit(c085f380,0,c6f23d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f23d60, ebp = 0 --- Tracing command intr pid 12 tid 100066 td 0xc74cb8a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100065 td 0xc74cbb80 sched_switch(c74cbb80,0,109,ae0e8b56,1b,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce042e,529,c7154070,...) at mi_switch+0x11b ithread_loop(c74cd310,e791fd28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c74cd310,e791fd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe791fd60, ebp = 0 --- Tracing command intr pid 12 tid 100063 td 0xc73a82e0 sched_switch(c73a82e0,0,109,6d9648ca,39,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce042e,529,c74ba0f0,...) at mi_switch+0x11b ithread_loop(c74c0820,e790fd28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c74c0820,e790fd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe790fd60, ebp = 0 --- Tracing command intr pid 12 tid 100060 td 0xc73a8b80 sched_switch(c73a8b80,0,109,ae059505,1b,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce042e,529,c719c0f0,...) at mi_switch+0x11b ithread_loop(c73b7610,e7903d28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c73b7610,e7903d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7903d60, ebp = 0 --- Tracing command intr pid 12 tid 100059 td 0xc73b8000 sched_switch(c73b8000,0,109,bacece0e,1b,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce042e,529,c719c3f0,...) at mi_switch+0x11b ithread_loop(c7381a80,e78ffd28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c7381a80,e78ffd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78ffd60, ebp = 0 --- Tracing command intr pid 12 tid 100054 td 0xc73ba000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100041 td 0xc73a8000 sched_switch(c73a8000,0,109,d56f6d46,1b,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce042e,529,c719c1f0,...) at mi_switch+0x11b ithread_loop(c7381420,e78c1d28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c7381420,e78c1d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78c1d60, ebp = 0 --- Tracing command intr pid 12 tid 100036 td 0xc7393b80 sched_switch(c7393b80,0,109,d4bbed81,1b,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce042e,529,c719c270,...) at mi_switch+0x11b ithread_loop(c73817f0,e78b1d28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c73817f0,e78b1d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78b1d60, ebp = 0 --- Tracing command intr pid 12 tid 100031 td 0xc7395b80 sched_switch(c7395b80,0,109,d40901c6,1b,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce042e,529,c719c2f0,...) at mi_switch+0x11b ithread_loop(c737c5d0,e78a1d28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c737c5d0,e78a1d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe78a1d60, ebp = 0 --- Tracing command intr pid 12 tid 100030 td 0xc72602e0 sched_switch(c72602e0,0,109,22704051,d5,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce042e,529,c719bef0,...) at mi_switch+0x11b ithread_loop(c736da70,e7792d28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c736da70,e7792d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7792d60, ebp = 0 --- Tracing command intr pid 12 tid 100029 td 0xc72605c0 sched_switch(c72605c0,0,109,36e9d821,dc,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce042e,529,c736c870,...) at mi_switch+0x11b ithread_loop(c736dbc0,e778ed28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c736dbc0,e778ed28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe778ed60, ebp = 0 --- Tracing command intr pid 12 tid 100028 td 0xc72608a0 sched_switch(c72608a0,0,109,ae0f273f,1b,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce042e,529,c7153b70,...) at mi_switch+0x11b ithread_loop(c71554f0,c6f77d28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c71554f0,c6f77d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f77d60, ebp = 0 --- Tracing command intr pid 12 tid 100026 td 0xc7330000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100021 td 0xc719d8a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100019 td 0xc725f000 sched_switch(c725f000,0,109,7edc0d15,1f,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce042e,529,c732f970,...) at mi_switch+0x11b ithread_loop(c7128930,c6f35d28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c7128930,c6f35d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f35d60, ebp = 0 --- Tracing command intr pid 12 tid 100018 td 0xc725f2e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100012 td 0xc715a5c0 sched_switch(c715a5c0,0,109,a2f5866c,e0,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce042e,529,c719bb70,...) at mi_switch+0x11b ithread_loop(c7155040,c6f20d28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c7155040,c6f20d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f20d60, ebp = 0 --- Tracing command intr pid 12 tid 100011 td 0xc715a8a0 sched_switch(c715a8a0,0,109,bf43a42b,e0,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce042e,529,c719bbf0,...) at mi_switch+0x11b ithread_loop(c7155050,c6f1dd28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c7155050,c6f1dd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f1dd60, ebp = 0 --- Tracing command intr pid 12 tid 100010 td 0xc715ab80 sched_switch(c715ab80,0,109,bbca86a9,e0,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce042e,529,c719bc70,...) at mi_switch+0x11b ithread_loop(c7155060,c6f1ad28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c7155060,c6f1ad28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f1ad60, ebp = 0 --- Tracing command intr pid 12 tid 100009 td 0xc719d000 sched_switch(c719d000,0,109,e27677ce,e0,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce042e,529,c719bcf0,...) at mi_switch+0x11b ithread_loop(c7155070,c6f17d28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c7155070,c6f17d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f17d60, ebp = 0 --- Tracing command intr pid 12 tid 100008 td 0xc719d2e0 sched_switch(c719d2e0,0,109,7edbd4e6,1f,...) at sched_switch+0x2b6 mi_switch(109,0,c0ce042e,529,c719bd70,...) at mi_switch+0x11b ithread_loop(c7155080,c6f14d28,0,0,0,...) at ithread_loop+0x160 fork_exit(c089ddb0,c7155080,c6f14d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f14d60, ebp = 0 --- Tracing command intr pid 12 tid 100007 td 0xc719d5c0 fork_trampoline() at fork_trampoline Tracing command idle pid 11 tid 100006 td 0xc7158000 sched_switch(c7158000,0,60c,bfda2729,e0,...) at sched_switch+0x2b6 mi_switch(60c,0,c0ce491e,815,0,...) at mi_switch+0x11b sched_preempt(c7158000,0,c8878000,c6f0dc98,c0bf0d9f,...) at sched_preempt+0x7c ipi_bitmap_handler(8,28,28,1f,c0e344c0,...) at ipi_bitmap_handler+0x34 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x2f --- interrupt, eip = 0xc08b6a25, esp = 0xc6f0dc84, ebp = 0xc6f0dc98 --- _thread_lock_flags(c7158000,0,c0ce491e,a06,c7158000,...) at _thread_lock_flags+0x65 sched_idletd(0,c6f0dd28,0,0,0,...) at sched_idletd+0x262 fork_exit(c08e83d0,0,c6f0dd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f0dd60, ebp = 0 --- Tracing command idle pid 11 tid 100005 td 0xc71582e0 sched_switch(c71582e0,0,108,c01278bd,e0,...) at sched_switch+0x2b6 mi_switch(108,0,c0ce491e,a06,c71582e0,...) at mi_switch+0x11b sched_idletd(0,c6f0ad28,0,0,0,...) at sched_idletd+0x17b fork_exit(c08e83d0,0,c6f0ad28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f0ad60, ebp = 0 --- Tracing command idle pid 11 tid 100004 td 0xc71585c0 sched_switch(c71585c0,0,108,c0143fd0,e0,...) at sched_switch+0x2b6 mi_switch(108,0,c0ce491e,a06,c71585c0,...) at mi_switch+0x11b sched_idletd(0,c6f07d28,0,0,0,...) at sched_idletd+0x17b fork_exit(c08e83d0,0,c6f07d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f07d60, ebp = 0 --- Tracing command idle pid 11 tid 100003 td 0xc71588a0 sched_switch(c71588a0,0,108,c0223432,e0,...) at sched_switch+0x2b6 mi_switch(108,0,c0ce491e,a06,c71588a0,...) at mi_switch+0x11b sched_idletd(0,c6f04d28,0,0,0,...) at sched_idletd+0x17b fork_exit(c08e83d0,0,c6f04d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f04d60, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc7158b80 sched_switch(c7158b80,0,104,cb39c595,21,...) at sched_switch+0x2b6 mi_switch(104,0,15c,c7156ac0,c6f00b50,...) at mi_switch+0x11b sleepq_switch(c7158b80,0,c0ce691d,1a0,c7158b80,...) at sleepq_switch+0xcc sleepq_catch_signals(15c,0,c6f00ba4,c08cf4ac,c7156ac0,...) at sleepq_catch_signals+0x52 sleepq_wait_sig(c7156ac0,5c,c0ce72c1,100,0,...) at sleepq_wait_sig+0x18 _sleep(c7156ac0,c7156b48,15c,c0ce72c1,0,...) at _sleep+0x28c kern_wait(c7158b80,ffffffff,c6f00c44,0,0,...) at kern_wait+0xfa1 wait4(c7158b80,c6f00cec,c6f00d28,2811afdc,c7156ac0,...) at wait4+0x3b syscallenter(c7158b80,c6f00ce4,c0c0a89e,c7158b80,c6f00d28,...) at syscallenter+0x329 syscall(c6f00d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x8054d7b, esp = 0xbfbfe8fc, ebp = 0xbfbfe918 --- Tracing command audit pid 10 tid 100001 td 0xc715a000 sched_switch(c715a000,0,104,ae0f856a,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c715a000,0,c6efdc58,...) at mi_switch+0x11b sleepq_switch(c715a000,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c0e4f440,0,c0cfeea3,1,0,...) at sleepq_wait+0x39 _cv_wait(c0e4f440,c0e4f424,0,0,0,...) at _cv_wait+0x18b audit_worker(0,c6efdd28,0,0,0,...) at audit_worker+0x6f fork_exit(c0add620,0,c6efdd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6efdd60, ebp = 0 --- Tracing command kernel pid 0 tid 100061 td 0xc73a88a0 sched_switch(c73a88a0,0,104,db98f631,e0,...) at sched_switch+0x2b6 mi_switch(104,0,c73a88a0,0,e7909c88,...) at mi_switch+0x11b sleepq_switch(c73a88a0,0,c0ce691d,260,c73dd580,...) at sleepq_switch+0xcc sleepq_wait(c73dd580,0,c0cdbf52,0,0,...) at sleepq_wait+0x39 msleep_spin(c73dd580,c73dd598,c0cdbf52,0,e7909ce0,...) at msleep_spin+0x181 taskqueue_thread_loop(c73e249c,e7909d28,0,0,0,...) at taskqueue_thread_loop+0xe4 fork_exit(c0901190,c73e249c,e7909d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7909d60, ebp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xc7260b80 sched_switch(c7260b80,0,104,b09038a1,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c7260b80,0,c6f4dc78,...) at mi_switch+0x11b sleepq_switch(c7260b80,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c732c9c0,0,c0cdbf52,0,0,...) at sleepq_wait+0x39 _sleep(c732c9c0,c732c9d8,0,c0cdbf52,0,...) at _sleep+0x2a2 taskqueue_thread_loop(c0e42f48,c6f4dd28,0,0,0,...) at taskqueue_thread_loop+0x10a fork_exit(c0901190,c0e42f48,c6f4dd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f4dd60, ebp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xc73302e0 sched_switch(c73302e0,0,104,b08fd3a6,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73302e0,0,c6f47c78,...) at mi_switch+0x11b sleepq_switch(c73302e0,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c732ca80,0,c0cdbf52,0,0,...) at sleepq_wait+0x39 _sleep(c732ca80,c732ca98,0,c0cdbf52,0,...) at _sleep+0x2a2 taskqueue_thread_loop(c0e4fa24,c6f47d28,0,0,0,...) at taskqueue_thread_loop+0x10a fork_exit(c0901190,c0e4fa24,c6f47d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f47d60, ebp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xc73305c0 sched_switch(c73305c0,0,104,b08fc889,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73305c0,0,c6f44c88,...) at mi_switch+0x11b sleepq_switch(c73305c0,0,c0ce691d,260,c732cc00,...) at sleepq_switch+0xcc sleepq_wait(c732cc00,0,c0cdbf52,0,0,...) at sleepq_wait+0x39 msleep_spin(c732cc00,c732cc18,c0cdbf52,0,c6f44ce0,...) at msleep_spin+0x181 taskqueue_thread_loop(c0dffb20,c6f44d28,0,0,0,...) at taskqueue_thread_loop+0xe4 fork_exit(c0901190,c0dffb20,c6f44d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f44d60, ebp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xc73308a0 sched_switch(c73308a0,0,104,b08fbf25,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c73308a0,0,c6f41c88,...) at mi_switch+0x11b sleepq_switch(c73308a0,0,c0ce691d,260,c732cc00,...) at sleepq_switch+0xcc sleepq_wait(c732cc00,0,c0cdbf52,0,0,...) at sleepq_wait+0x39 msleep_spin(c732cc00,c732cc18,c0cdbf52,0,c6f41ce0,...) at msleep_spin+0x181 taskqueue_thread_loop(c0dffb20,c6f41d28,0,0,0,...) at taskqueue_thread_loop+0xe4 fork_exit(c0901190,c0dffb20,c6f41d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f41d60, ebp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xc7330b80 sched_switch(c7330b80,0,104,b08fb503,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c7330b80,0,c6f3ec88,...) at mi_switch+0x11b sleepq_switch(c7330b80,0,c0ce691d,260,c732cc00,...) at sleepq_switch+0xcc sleepq_wait(c732cc00,0,c0cdbf52,0,0,...) at sleepq_wait+0x39 msleep_spin(c732cc00,c732cc18,c0cdbf52,0,c6f3ece0,...) at msleep_spin+0x181 taskqueue_thread_loop(c0dffb20,c6f3ed28,0,0,0,...) at taskqueue_thread_loop+0xe4 fork_exit(c0901190,c0dffb20,c6f3ed28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f3ed60, ebp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xc719db80 sched_switch(c719db80,0,104,b08d8e0f,1b,...) at sched_switch+0x2b6 mi_switch(104,0,c719db80,0,c6f38c78,...) at mi_switch+0x11b sleepq_switch(c719db80,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c732cd40,0,c0cdbf52,0,0,...) at sleepq_wait+0x39 _sleep(c732cd40,c732cd58,0,c0cdbf52,0,...) at _sleep+0x2a2 taskqueue_thread_loop(c0e2ec84,c6f38d28,0,0,0,...) at taskqueue_thread_loop+0x10a fork_exit(c0901190,c0e2ec84,c6f38d28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f38d60, ebp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xc725f8a0 sched_switch(c725f8a0,0,104,8e9b0929,1d,...) at sched_switch+0x2b6 mi_switch(104,0,c725f8a0,0,c6f2cc78,...) at mi_switch+0x11b sleepq_switch(c725f8a0,0,c0ce691d,260,0,...) at sleepq_switch+0xcc sleepq_wait(c7137880,0,c0cdbf52,0,0,...) at sleepq_wait+0x39 _sleep(c7137880,c7137898,0,c0cdbf52,0,...) at _sleep+0x2a2 taskqueue_thread_loop(c0e419e0,c6f2cd28,0,0,0,...) at taskqueue_thread_loop+0x10a fork_exit(c0901190,c0e419e0,c6f2cd28) at fork_exit+0x91 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f2cd60, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc0e2e5f0 sched_switch(c0e2e5f0,0,104,bf4433dc,e0,...) at sched_switch+0x2b6 mi_switch(104,0,c0e2e5f0,0,c1420cd8,...) at mi_switch+0x11b sleepq_switch(c0e2e5f0,0,c0ce691d,283,44,...) at sleepq_switch+0xcc sleepq_timedwait(c0e2e340,44,c0ce4dbb,0,0,...) at sleepq_timedwait+0x3e _sleep(c0e2e340,0,44,c0ce4dbb,2710,...) at _sleep+0x272 scheduler(0,141ec00,141ec00,141e000,1425000,...) at scheduler+0x2f0 mi_startup() at mi_startup+0xac begin() at begin+0x2c db:0:allt> call doadump Physical memory: 3439 MB Dumping 253 MB: 238 222 206 190 174 158 142 126 110 94 78 62 46 30 14 Dump complete = 0xf db:0:doadump> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 1 (kgdb) bt #0 doadump () at pcpu.h:231 #1 0xc04d5809 in db_fncall (dummy1=0xc08f576a, dummy2=0x0, dummy3=0xffffffff, dummy4=0xe9b943bc "ÐC¹é") at ../../../ddb/db_command.c:548 #2 0xc04d5c3f in db_command (last_cmdp=0xc0dfdcfc, cmd_table=0x0, dopager=0x0) at ../../../ddb/db_command.c:445 #3 0xc04d5cf4 in db_command_script (command=0xc0dfec08 "call doadump") at ../../../ddb/db_command.c:516 #4 0xc04d9b80 in db_script_exec (scriptname=0xc0dfe560 "doadump", warnifnotfound=Variable "warnifnotfound" is not available. ) at ../../../ddb/db_script.c:302 #5 0xc04d9c11 in db_run_cmd (addr=0x1, have_addr=0x0, count=0xc0ea2440, modif=0xe9b944f4 "") at ../../../ddb/db_script.c:375 #6 0xc04d5c01 in db_command (last_cmdp=0xc0dfdcfc, cmd_table=0x0, dopager=0x1) at ../../../ddb/db_command.c:445 #7 0xc04d5d5a in db_command_loop () at ../../../ddb/db_command.c:498 #8 0xc04d7c5d in db_trap (type=0x3, code=0x0) at ../../../ddb/db_main.c:229 #9 0xc08f559e in kdb_trap (type=0x3, code=0x0, tf=0xe9b94698) at ../../../kern/subr_kdb.c:546 #10 0xc0c0a7ac in trap (frame=0xe9b94698) at ../../../i386/i386/trap.c:721 #11 0xc0bf064c in calltrap () at ../../../i386/i386/exception.s:168 #12 0xc08f576a in kdb_enter (why=0xc0ce3759 "panic", msg=0xc0ce3759 "panic") at cpufunc.h:71 #13 0xc08c6624 in panic (fmt=0xc0ce6eaf "sleeping thread") at ../../../kern/kern_shutdown.c:575 #14 0xc090343c in propagate_priority (td=0xc7be52e0) at ../../../kern/subr_turnstile.c:222 #15 0xc0904278 in turnstile_wait (ts=0xc81f5a80, owner=0xc7be52e0, queue=Variable "queue" is not available. ) at ../../../kern/subr_turnstile.c:738 #16 0xc08b657c in _mtx_lock_sleep (m=0xc7e97948, tid=0xc7be4000, opts=0x0, file=0x0, line=0x0) at ../../../kern/kern_mutex.c:447 #17 0xc0b07baf in softdep_request_cleanup (fs=0xc8277800, vp=0xc8460560, cred=0xc8266c80) at ../../../ufs/ffs/ffs_softdep.c:5878 #18 0xc0af0f4f in ffs_alloc (ip=0xc85402b8, lbn=0x2f, bpref=0x35410, size=0x4000, flags=0x41000000, cred=0xc8266c80, bnp=0xe9b94a08) at ../../../ufs/ffs/ffs_alloc.c:220 #19 0xc0af5d33 in ffs_balloc_ufs2 (vp=0xc8460560, startoffset=Variable "startoffset" is not available. ) at ../../../ufs/ffs/ffs_balloc.c:856 #20 0xc0b15b85 in ffs_write (ap=0xe9b94b94) at ../../../ufs/ffs/ffs_vnops.c:745 #21 0xc0c2b0ac in VOP_WRITE_APV (vop=0xc0de2660, a=0xe9b94b94) at vnode_if.c:951 #22 0xc0966b1a in vn_write (fp=0xc7b9aab8, uio=0xe9b94c28, active_cred=0xc8266c80, flags=0x0, td=0xc7be4000) at vnode_if.h:413 #23 0xc0906547 in dofilewrite (td=0xc7be4000, fd=0x3, fp=0xc7b9aab8, auio=0xe9b94c28, offset=0xffffffffffffffff, flags=0x0) at file.h:239 #24 0xc0906838 in kern_writev (td=0xc7be4000, fd=0x3, auio=0xe9b94c28) at ../../../kern/sys_generic.c:447 #25 0xc09068bf in write (td=0xc7be4000, uap=0xe9b94cec) at ../../../kern/sys_generic.c:363 #26 0xc0902179 in syscallenter (td=0xc7be4000, sa=0xe9b94ce4) at ../../../kern/subr_trap.c:315 #27 0xc0c0a13f in syscall (frame=0xe9b94d28) at ../../../i386/i386/trap.c:1086 #28 0xc0bf06b1 in Xint0x80_syscall () at ../../../i386/i386/exception.s:266 #29 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 17 #17 0xc0b07baf in softdep_request_cleanup (fs=0xc8277800, vp=0xc8460560, cred=0xc8266c80) at ../../../ufs/ffs/ffs_softdep.c:5878 5878 UFS_LOCK(ump); (kgdb) l 5873 FREE_LOCK(&lk); 5874 UFS_LOCK(ump); 5875 continue; 5876 } 5877 FREE_LOCK(&lk); 5878 UFS_LOCK(ump); 5879 } 5880 /* 5881 * If we still need resources and there are no more worklist 5882 * entries to process to obtain them, we have to start flushing (kgdb) f 14 #14 0xc090343c in propagate_priority (td=0xc7be52e0) at ../../../kern/subr_turnstile.c:222 222 panic("sleeping thread"); (kgdb) p *td $1 = {td_lock = 0xc0e42254, td_proc = 0xc7be12b0, td_plist = {tqe_next = 0x0, tqe_prev = 0xc7be12b8}, td_runq = {tqe_next = 0x0, tqe_prev = 0xc0e34c44}, td_slpq = {tqe_next = 0x0, tqe_prev = 0xc7bee180}, td_lockq = { tqe_next = 0xc87895c0, tqe_prev = 0xe9ca5610}, td_cpuset = 0xc7143e38, td_sel = 0x0, td_sleepqueue = 0x0, td_turnstile = 0xc8879500, td_umtxq = 0xc7a94800, td_tid = 0x186fa, td_sigqueue = {sq_signals = {__bits = {0x0, 0x0, 0x0, 0x0}}, sq_kill = {__bits = {0x0, 0x0, 0x0, 0x0}}, sq_list = {tqh_first = 0x0, tqh_last = 0xc7be5340}, sq_proc = 0xc7be12b0, sq_flags = 0x1}, td_flags = 0x4, td_inhibitors = 0x2, td_pflags = 0x100000, td_dupfd = 0x0, td_sqqueue = 0x0, td_wchan = 0xc8291280, td_wmesg = 0xc0cea4e8 "bo_wwait", td_lastcpu = 0x1, td_oncpu = 0xff, td_owepreempt = 0x0, td_tsqueue = 0x0, td_locks = 0x9, td_rw_rlocks = 0x0, td_lk_slocks = 0x0, td_blocked = 0x0, td_lockname = 0x0, td_contested = {lh_first = 0xc81f5a80}, td_sleeplocks = 0x0, td_intr_nesting_level = 0x0, td_pinned = 0x0, td_ucred = 0xc8266c80, td_estcpu = 0x0, td_slptick = 0x72c47, td_blktick = 0x0, td_ru = {ru_utime = {tv_sec = 0x0, tv_usec = 0x0}, ru_stime = {tv_sec = 0x0, tv_usec = 0x0}, ru_maxrss = 0x764, ru_ixrss = 0x78, ru_idrss = 0x55c8, ru_isrss = 0xf00, ru_minflt = 0x113, ru_majflt = 0x0, ru_nswap = 0x0, ru_inblock = 0x0, ru_oublock = 0xe9, ru_msgsnd = 0x0, ru_msgrcv = 0x0, ru_nsignals = 0x0, ru_nvcsw = 0x1da9, ru_nivcsw = 0x50a}, td_incruntime = 0x195f54c2, td_runtime = 0x195f54c2, td_pticks = 0x0, td_sticks = 0x1d, td_iticks = 0x0, td_uticks = 0x1, td_intrval = 0x0, td_oldsigmask = {__bits = {0x0, 0x0, 0x0, 0x0}}, td_sigmask = {__bits = {0x0, 0x0, 0x0, 0x0}}, td_generation = 0x22b3, td_sigstk = {ss_sp = 0x0, ss_size = 0x0, ss_flags = 0x4}, td_xsig = 0x0, td_profil_addr = 0x0, td_profil_ticks = 0x0, td_name = "linger2", '\0' , td_fpop = 0x0, td_dbgflags = 0x0, td_dbgksi = {ksi_link = {tqe_next = 0x0, tqe_prev = 0x0}, ksi_info = {si_signo = 0x0, si_errno = 0x0, si_code = 0x0, si_pid = 0x0, si_uid = 0x0, si_status = 0x0, si_addr = 0x0, si_value = {sival_int = 0x0, sival_ptr = 0x0, sigval_int = 0x0, sigval_ptr = 0x0}, _reason = {_fault = {_trapno = 0x0}, _timer = {_timerid = 0x0, _overrun = 0x0}, _mesgq = {_mqd = 0x0}, _poll = {_band = 0x0}, __spare__ = {__spare1__ = 0x0, __spare2__ = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}}}, ksi_flags = 0x0, ksi_sigq = 0x0}, td_ng_outbound = 0x0, td_osd = {osd_nslots = 0x0, osd_slots = 0x0, osd_next = {le_next = 0x0, le_prev = 0x0}}, td_rqindex = 0x1a, td_base_pri = 0xa0, td_priority = 0xa0, td_pri_class = 0x3, td_user_pri = 0xb4, td_base_user_pri = 0xb4, td_pcb = 0xe9b64d80, td_state = TDS_INHIBITED, td_retval = {0x0, 0x28191700}, td_slpcallout = {c_links = {sle = {sle_next = 0x0}, tqe = {tqe_next = 0x0, tqe_prev = 0xc723d6e0}}, c_time = 0x6bcdc, c_arg = 0xc7be52e0, c_func = 0xc08ff3a0 , c_lock = 0x0, c_flags = 0x12, c_cpu = 0x3}, td_frame = 0xe9b64d28, td_kstack_obj = 0xc7bcd908, td_kstack = 0xe9b63000, td_kstack_pages = 0x2, td_unused1 = 0x0, td_unused2 = 0x0, td_unused3 = 0x0, td_critnest = 0x1, td_md = {md_spinlock_count = 0x1, md_saved_flags = 0x246}, td_sched = 0xc7be558c, td_ar = 0x0, td_syscalls = 0x91a, td_lprof = {{lh_first = 0x0}, {lh_first = 0x0}}, td_dtrace = 0x0, td_errno = 0x0, td_vnet = 0x0, td_vnet_lpush = 0x0, td_rux = {rux_runtime = 0x0, rux_uticks = 0x0, rux_sticks = 0x0, rux_iticks = 0x0, rux_uu = 0x0, rux_su = 0x0, rux_tu = 0x0}, td_map_def_user = 0x0, td_dbg_forked = 0x0} (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/ufs/ffs/ffs_softdep.c =================================================================== --- /usr/src/sys/ufs/ffs/ffs_softdep.c (revision 220001) +++ /usr/src/sys/ufs/ffs/ffs_softdep.c (working copy) @@ -62,6 +62,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include @@ -338,9 +339,10 @@ softdep_releasefile(ip) } int -softdep_request_cleanup(fs, vp) +softdep_request_cleanup(fs, vp, cred) struct fs *fs; struct vnode *vp; + struct ucred *cred; { return (0); @@ -678,6 +680,7 @@ static int stat_indir_blk_ptrs; /* bufs redirtied static int stat_inode_bitmap; /* bufs redirtied as inode bitmap not written */ static int stat_direct_blk_ptrs;/* bufs redirtied as direct ptrs not written */ static int stat_dir_entry; /* bufs redirtied as dir entry cannot write */ +static int stat_high_cleanup_delay; /* Maximum cleanup delay (in ticks) */ SYSCTL_INT(_debug, OID_AUTO, max_softdeps, CTLFLAG_RW, &max_softdeps, 0, ""); SYSCTL_INT(_debug, OID_AUTO, tickdelay, CTLFLAG_RW, &tickdelay, 0, ""); @@ -692,6 +695,7 @@ SYSCTL_INT(_debug, OID_AUTO, indir_blk_ptrs, CTLFL SYSCTL_INT(_debug, OID_AUTO, inode_bitmap, CTLFLAG_RW, &stat_inode_bitmap, 0, ""); SYSCTL_INT(_debug, OID_AUTO, direct_blk_ptrs, CTLFLAG_RW, &stat_direct_blk_ptrs, 0, ""); SYSCTL_INT(_debug, OID_AUTO, dir_entry, CTLFLAG_RW, &stat_dir_entry, 0, ""); +SYSCTL_INT(_debug, OID_AUTO, high_cleanup_delay, CTLFLAG_RW, &stat_high_cleanup_delay, 0, ""); /* SYSCTL_INT(_debug, OID_AUTO, worklist_num, CTLFLAG_RD, &softdep_on_worklist, 0, ""); */ SYSCTL_DECL(_vfs_ffs); @@ -5797,20 +5801,26 @@ softdep_slowdown(vp) * If the worklist yields no free space, encourage the syncer daemon * to help us. In no event will we try for longer than tickdelay seconds. */ + +static void check_block_count(char *, struct fs *, struct ufsmount *, + ufs2_daddr_t); + int -softdep_request_cleanup(fs, vp) +softdep_request_cleanup(fs, vp, cred) struct fs *fs; struct vnode *vp; + struct ucred *cred; { struct ufsmount *ump; + struct mount *mp; + struct vnode *lvp, *mvp; long starttime; ufs2_daddr_t needed; int error; ump = VTOI(vp)->i_ump; + mp = vp->v_mount; mtx_assert(UFS_MTX(ump), MA_OWNED); - needed = fs->fs_cstotal.cs_nbfree + fs->fs_contigsumsize; - starttime = time_second + tickdelay; /* * If we are being called because of a process doing a * copy-on-write, then it is not safe to update the vnode @@ -5823,9 +5833,38 @@ int if (error != 0) return (0); } - while (fs->fs_pendingblocks > 0 && fs->fs_cstotal.cs_nbfree <= needed) { - if (time_second > starttime) - return (0); + /* + * If we are in need of resources, consider pausing for + * tickdelay to give ourselves some breathing room. + */ + UFS_UNLOCK(ump); + ACQUIRE_LOCK(&lk); + request_cleanup(UFSTOVFS(ump), FLUSH_REMOVE_WAIT); + FREE_LOCK(&lk); + UFS_LOCK(ump); + /* + * Now clean up at least as much space as we will need. + * + * The amount of space that we will need is enough blocks to + * allocate a full-sized segment (fs_contigsumsize). The number of + * such segments that will be needed is set by the number of + * simultaneous writers (mnt_writeopcount) plus a bit of slop (2) + * in case some more show up while we are cleaning. + * + * Additionally, if we are unpriviledged we will need to ensure + * that we clean up enough blocks to get to the above threshhold + * of the minimum number of blocks required to be kept free by the + * filesystem (fs_minfree). + */ + needed = (mp->mnt_writeopcount + 2) * fs->fs_contigsumsize; + if (priv_check_cred(cred, PRIV_VFS_BLOCKRESERVE, 0)) + needed += fragstoblks(fs, + roundup((fs->fs_dsize * fs->fs_minfree / 100) - + fs->fs_cstotal.cs_nffree, fs->fs_frag)); + starttime = time_second + tickdelay; +retry: + while (ump->softdep_on_worklist > 0 && + fs->fs_cstotal.cs_nbfree <= needed) { UFS_UNLOCK(ump); ACQUIRE_LOCK(&lk); if (ump->softdep_on_worklist > 0 && @@ -5835,14 +5874,124 @@ int UFS_LOCK(ump); continue; } - request_cleanup(UFSTOVFS(ump), FLUSH_REMOVE_WAIT); FREE_LOCK(&lk); UFS_LOCK(ump); } + /* + * If we still need resources and there are no more worklist + * entries to process to obtain them, we have to start flushing + * the dirty vnodes to force the release of additional requests + * to the worklist that we can then process to reap addition + * resources. We start by flushing our own vnode and if that + * does not produce enough resources we walk the vnodes + * associated with the mount point until we get the needed + * worklist requests that we can reap. + */ + if (fs->fs_cstotal.cs_nbfree <= needed) { + MNT_ILOCK(mp); + MNT_VNODE_FOREACH(lvp, mp, mvp) { + if (ump->softdep_on_worklist > 0) { + MNT_IUNLOCK(mp); + check_block_count("Retry", fs, ump, needed); + goto retry; + } + VI_LOCK(lvp); + if (TAILQ_FIRST(&lvp->v_bufobj.bo_dirty.bv_hd) == 0 || + VOP_ISLOCKED(lvp) != 0) { + VI_UNLOCK(lvp); + continue; + } + MNT_IUNLOCK(mp); + if (vget(lvp, LK_EXCLUSIVE | LK_INTERLOCK, curthread)) { + VI_UNLOCK(lvp); + MNT_ILOCK(mp); + continue; + } + if (lvp->v_vflag & VV_NOSYNC) { /* unlinked */ + vput(lvp); + MNT_ILOCK(mp); + continue; + } + (void) ffs_syncvnode(lvp, MNT_WAIT); + vput(lvp); + MNT_ILOCK(mp); + } + MNT_IUNLOCK(mp); + check_block_count("Failed", fs, ump, needed); + } + if (time_second - starttime > stat_high_cleanup_delay) + stat_high_cleanup_delay = time_second - starttime; return (1); } /* + * Check out counts above. + */ +static void +check_block_count(action, fs, ump, needed) + char *action; + struct fs *fs; + struct ufsmount *ump; + ufs2_daddr_t needed; +{ + struct worklist *wk; + static int maxFcount = 20, maxRcount = 20; + int blkcnt, dircnt, fragcnt, inocnt; + ufs2_daddr_t pending; + + if (action[0] == 'F' && maxFcount-- <= 0) + return; + if (action[0] == 'R' && maxRcount-- <= 0) + return; + dircnt = 0; + fragcnt = 0; + blkcnt = 0; + inocnt = 0; + pending = 0; + LIST_FOREACH(wk, &ump->softdep_workitem_pending, wk_list) { + switch (wk->wk_type) { + + case D_DIRREM: + /* removal of a directory entry */ + dircnt += 1; + continue; + + case D_FREEBLKS: + /* releasing blocks and/or fragments from a file */ + pending += WK_FREEBLKS(wk)->fb_chkcnt; + blkcnt += 1; + continue; + + case D_FREEFRAG: + /* releasing a fragment when replaced as a file grows */ + fragcnt += 1; + continue; + + case D_FREEFILE: + /* releasing an inode when its link count drops to 0 */ + inocnt += 1; + continue; + + default: + panic("check_block_count: Unknown type %s", + TYPENAME(wk->wk_type)); + /* NOTREACHED */ + } + } + printf("%s reclaim\n", action); + printf("Seeking %lld blocks\n", needed); + printf("Free inodes %lld, free blocks %lld, free fragments %lld\n", + fs->fs_cstotal.cs_nifree, fs->fs_cstotal.cs_nbfree, + fs->fs_cstotal.cs_nffree); + printf("Pending workitems %d, Allocated inodes %lld\n", + ump->softdep_on_worklist, + (fs->fs_ncg * fs->fs_ipg) - fs->fs_cstotal.cs_nifree); + printf("Found %d files with %lld blocks pending\n", blkcnt, pending); + printf("Other types: dirrem %d, freefrag %d, freefile %d\n", + dircnt, fragcnt, inocnt); +} + +/* * If memory utilization has gotten too high, deliberately slow things * down and speed up the I/O processing. */ Index: /usr/src/sys/ufs/ffs/ffs_alloc.c =================================================================== --- /usr/src/sys/ufs/ffs/ffs_alloc.c (revision 220001) +++ /usr/src/sys/ufs/ffs/ffs_alloc.c (working copy) @@ -215,9 +215,9 @@ nospace: (void) chkdq(ip, -btodb(size), cred, FORCE); UFS_LOCK(ump); #endif - if (fs->fs_pendingblocks > 0 && reclaimed == 0) { + if (reclaimed == 0) { reclaimed = 1; - softdep_request_cleanup(fs, ITOV(ip)); + softdep_request_cleanup(fs, ITOV(ip), cred); goto retry; } UFS_UNLOCK(ump); @@ -430,9 +430,9 @@ nospace: /* * no space available */ - if (fs->fs_pendingblocks > 0 && reclaimed == 0) { + if (reclaimed == 0) { reclaimed = 1; - softdep_request_cleanup(fs, vp); + softdep_request_cleanup(fs, vp, cred); UFS_UNLOCK(ump); if (bp) { brelse(bp); Index: /usr/src/sys/ufs/ffs/ffs_extern.h =================================================================== --- /usr/src/sys/ufs/ffs/ffs_extern.h (revision 220001) +++ /usr/src/sys/ufs/ffs/ffs_extern.h (working copy) @@ -114,7 +114,7 @@ int softdep_flushfiles(struct mount *, int, struct void softdep_update_inodeblock(struct inode *, struct buf *, int); void softdep_load_inodeblock(struct inode *); void softdep_freefile(struct vnode *, ino_t, int); -int softdep_request_cleanup(struct fs *, struct vnode *); +int softdep_request_cleanup(struct fs *, struct vnode *, struct ucred *); void softdep_setup_freeblocks(struct inode *, off_t, int); void softdep_setup_inomapdep(struct buf *, struct inode *, ino_t); void softdep_setup_blkmapdep(struct buf *, struct mount *, ufs2_daddr_t); -- Test scenario: linger2.sh