GENERIC HEAD from 2011-06-29 22:12:43 UTC, r223687M, vmcore.173 KDB: debugger backends: ddb KDB: current backend: ddb 524288K of memory above 4GB ignored Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.0-CURRENT #0 r223687M: Thu Jun 30 08:47:14 CEST 2011 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Family = 10 Model = 2 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 4294967296 (4096 MB) avail memory = 3531751424 (3368 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: <060508 APIC1048> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: <060508 RSDT1048> on motherboard acpi0: Power Button (fixed) acpi0: reservation of fee00000, 1000 (3) failed acpi0: reservation of ffb80000, 80000 (3) failed acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, d7f00000 (3) failed Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 vgapci0: port 0xc000-0xc0ff mem 0xf0000000-0xf7ffffff,0xfe9f0000-0xfe9fffff,0xfe800000-0xfe8fffff irq 18 at device 5.0 on pci1 hdac0: mem 0xfe9e8000-0xfe9ebfff irq 19 at device 5.2 on pci1 pcib2: at device 7.0 on pci0 pci2: on pcib2 re0: port 0xd800-0xd8ff mem 0xfeaff000-0xfeafffff irq 19 at device 0.0 on pci2 re0: Using 1 MSI message re0: Chip rev. 0x38000000 re0: MAC rev. 0x00000000 miibus0: on re0 rgephy0: PHY 1 on miibus0 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re0: Ethernet address: 00:21:85:15:33:7d ahci0: port 0xb000-0xb007,0xa000-0xa003,0x9000-0x9007,0x8000-0x8003,0x7000-0x700f mem 0xfe7ff800-0xfe7ffbff irq 22 at device 18.0 on pci0 ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ohci0: mem 0xfe7fe000-0xfe7fefff irq 16 at device 19.0 on pci0 usbus0: on ohci0 ohci1: mem 0xfe7fd000-0xfe7fdfff irq 17 at device 19.1 on pci0 usbus1: on ohci1 ohci2: mem 0xfe7fc000-0xfe7fcfff irq 18 at device 19.2 on pci0 usbus2: on ohci2 ohci3: mem 0xfe7fb000-0xfe7fbfff irq 17 at device 19.3 on pci0 usbus3: on ohci3 ohci4: mem 0xfe7fa000-0xfe7fafff irq 18 at device 19.4 on pci0 usbus4: on ohci4 ehci0: mem 0xfe7ff000-0xfe7ff0ff irq 19 at device 19.5 on pci0 ehci0: AMD SB600/700 quirk applied usbus5: EHCI version 1.0 usbus5: on ehci0 pci0: at device 20.0 (no driver attached) atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0 ata0: on atapci0 hdac1: mem 0xfe7f4000-0xfe7f7fff irq 16 at device 20.2 on pci0 isab0: at device 20.3 on pci0 isa0: on isab0 pcib3: at device 20.4 on pci0 pci3: on pcib3 fwohci0: port 0xe800-0xe87f mem 0xfebff800-0xfebfffff irq 20 at device 6.0 on pci3 fwohci0: OHCI version 1.10 (ROM=1) fwohci0: No. of Isochronous channels is 4. fwohci0: EUI64 00:dc:10:00:01:62:ad:eb fwohci0: Phy 1394a available S400, 2 ports. fwohci0: Link S400, max_rec 2048 bytes. firewire0: on fwohci0 fwe0: on firewire0 if_fwe0: Fake Ethernet address: 02:dc:10:62:ad:eb fwe0: Ethernet address: 02:dc:10:62:ad:eb fwip0: on firewire0 fwip0: Firewire address: 00:dc:10:00:01:62:ad:eb @ 0xfffe00000000, S400, maxrec 2048 sbp0: on firewire0 dcons_crom0: on firewire0 dcons_crom0: bus_addr 0x1063c80 fwohci0: Initiate bus reset fwohci0: fwohci_intr_core: BUS reset fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode acpi_button0: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (115200,n,8,1) fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: model IntelliMouse Explorer, device ID 4 pmtimer0 on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: parallel port not found. acpi_throttle0: on cpu0 acpi_throttle0: CLK_VAL field overlaps THT_EN bit device_attach: acpi_throttle0 attach returned 6 hwpstate0: on cpu0 Timecounters tick every 1.000 msec firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me) firewire0: bus manager 0 hdac0: HDA Codec #0: ATI RS690/780 HDMI pcm0: at cad 0 nid 1 on hdac0 hdac1: HDA Codec #3: Realtek ALC888 pcm1: at cad 3 nid 1 on hdac1 pcm2: at cad 3 nid 1 on hdac1 pcm3: at cad 3 nid 1 on hdac1 usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 12Mbps Full Speed USB v1.0 ugen0.1: at usbus0 uhub0: on usbus0 Expensive timeout(9) function: 0xc0a437f0(0xc74282e0) 0.010637105 s ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 480Mbps High Speed USB v2.0 ugen4.1: at usbus4 uhub4: on usbus4 ugen5.1: at usbus5 uhub5: on usbus5 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub5: 10 ports with 10 removable, self powered ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ATA-8 SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C) ada0: Previously was known as ad4 cd0 at ata0 bus 0 scbus4 target 1 lun 0 cd0: <_NEC DVD_RW ND-4550A 1.06> Removable CD-ROM SCSI-0 device cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present SMP: AP CPU #2 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! Timecounter "TSC-low" frequency 14062675 Hz quality 800 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Trying to mount root from ufs:/dev/ad4s1a [rw]... WARNING: / was not properly dismounted Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Starting ddb. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/ad4s1a: 3817 files, 809453 used, 203562 free (1730 frags, 25229 blocks, 0.2% fragmentation) /dev/ad4s1f: 17210 files, 821348 used, 191667 free (4435 frags, 23404 blocks, 0.4% fragmentation) /dev/label/tmp: 102557 files, 24047813 used, 94266168 free (3024 frags, 11782893 blocks, 0.0% fragmentation) /dev/ad4s1d: 818587 files, 8294317 used, 1859882 free (285266 frags, 196827 blocks, 2.8% fragmentation) /dev/ad4s1e: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED) /dev/ad4s1e: 818550 files, 22866047 used, 2522470 free (72902 frags, 306196 blocks, 0.3% fragmentation) Mounting local file systems:. Setting hostname: x4.osted.lan. re0: link state changed to DOWN Starting Network: lo0 re0 fwe0 fwip0. lo0: flags=8049 metric 0 mtu 16384 options=3 inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=389b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::221:85ff:fe15:337d%re0 prefixlen 64 tentative scopeid 0x1 nd6 options=29 media: Ethernet autoselect (none) status: no carrier fwe0: flags=8802 metric 0 mtu 1500 options=8 ether 02:dc:10:62:ad:eb nd6 options=29 ch 1 dma -1 fwip0: flags=8802 metric 0 mtu 1500 lladdr 0.dc.10.0.1.62.ad.eb.a.2.ff.fe.0.0.0.0 nd6 options=29 Starting devd. Starting Network: usbus0. Starting Network: usbus1. Starting Network: usbus2. Starting Network: usbus3. Starting Network: usbus4. Starting Network: usbus5. re0: link state changed to UP Starting Network: fwe0. fwe0: flags=8802 metric 0 mtu 1500 options=8 ether 02:dc:10:62:ad:eb nd6 options=29 ch 1 dma -1 Starting Network: fwip0. fwip0: flags=8802 metric 0 mtu 1500 lladdr 0.dc.10.0.1.62.ad.eb.a.2.ff.fe.0.0.0.0 nd6 options=29 add net default: gateway 192.168.1.1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/kde4/lib /usr/local/lib /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/nss /usr/local/lib/qt4 /usr/local/lib/virtualbox /usr/local/lib/wine a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Creating and/or trimming log files. Starting syslogd. savecore: reboot after panic: softdep_deallocate_dependencies: unrecovered I/O error Jun 30 09:29:20 x4 savecore: reboot after panic: softdep_deallocate_dependencies: unrecovered I/O error lock order reversal: 1st 0xe0b15ae0 bufwait (bufwait) @ kern/vfs_bio.c:2658 2nd 0xc7d6ae00 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper(c0ef6376,2e687361,38323a63,c7000a34,a7a,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a3d01b,c0ef9d3f,c7165370,c71693a0,ef6617e4,...) at kdb_backtrace+0x2a _witness_debugger(c0ef9d3f,c7d6ae00,c0f290a2,c71693a0,c0f28d34,...) at _witness_debugger+0x25 witness_checkorder(c7d6ae00,9,c0f28d2b,11c,0,...) at witness_checkorder+0x839 _sx_xlock(c7d6ae00,0,c0f28d2b,11c,c7d71000,...) at _sx_xlock+0x85 ufsdirhash_acquire(e0b15a80,e16a2868,198,e16a2880,ef6618b4,...) at ufsdirhash_acquire+0x48 ufsdirhash_add(c7d71000,ef661914,2880,ef6618a0,ef6618a4,...) at ufsdirhash_add+0x13 ufs_direnter(c7d4ad98,c7d78d98,ef661914,ef661ba4,0,...) at ufs_direnter+0x749 ufs_makeinode(ef661ba4,c1040120,ef661b00,ef661a5c,c0d59225,...) at ufs_makeinode+0x5e5 ufs_create(ef661b00,c0f471f4,0,0,ef661b78,...) at ufs_create+0x30 VOP_CREATE_APV(c103fba0,ef661b00,ef661ba4,ef661a98,0,...) at VOP_CREATE_APV+0xc5 vn_open_cred(ef661b78,ef661c2c,180,0,c719fe80,...) at vn_open_cred+0x205 vn_open(ef661b78,ef661c2c,180,c78aa578,0,...) at vn_open+0x3b kern_openat(c77ec000,ffffff9c,28450000,0,602,...) at kern_openat+0x12f kern_open(c77ec000,28450000,0,601,180,...) at kern_open+0x35 open(c77ec000,ef661cec,ef661d80,ef661c88,0,...) at open+0x30 syscallenter(c77ec000,ef661ce4,c0d35cd6,0,0,...) at syscallenter+0x263 syscall(ef661d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (5, FreeBSD ELF32, open), eip = 0x2818a013, esp = 0xbfbfe2ac, ebp = 0xbfbfedb8 --- savecore: writing core to vmcore.172 Writing crash summary to /var/crash/core.txt.172. Additional ABI support: linux. Starting rpcbind. NFS access cache time=60 Clearing /tmp (X related). Starting mountd. NFSv4 is disabled Starting nfsd. Recovering vi editor sessions:. Updating motd:. Starting ntpd. Starting default moused. Configuring syscons: keymap blanktime. Starting sshd. Starting cron. Local package initialization: backuplock order reversal: 1st 0xc7b8c5c8 ufs (ufs) @ kern/vfs_subr.c:2134 2nd 0xe0af0680 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:261 3rd 0xc7f9b310 ufs (ufs) @ kern/vfs_subr.c:2134 KDB: stack backtrace: db_trace_self_wrapper(c0ef6376,3a632e73,a313632,ef6c8800,c0a4e90b,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a3d01b,c0ef9d58,c7165370,c7169338,ef6c88a4,...) at kdb_backtrace+0x2a _witness_debugger(c0ef9d58,c7f9b310,c0ee9124,c7169338,c0f0195b,...) at _witness_debugger+0x25 witness_checkorder(c7f9b310,9,c0f01952,856,0,...) at witness_checkorder+0x839 __lockmgr_args(c7f9b310,80100,c7f9b37c,0,0,...) at __lockmgr_args+0x814 ffs_lock(ef6c89c8,c0a4e90b,c0f00cb7,80100,c7f9b2b8,...) at ffs_lock+0xa1 VOP_LOCK1_APV(c103fba0,ef6c89c8,c789f0b0,c105abc0,c7f9b2b8,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c7f9b2b8,80100,c0f01952,856,4,...) at _vn_lock+0x78 vget(c7f9b2b8,80100,c789f000,50,0,...) at vget+0xbb vfs_hash_get(c7b9b000,10b0400,80000,c789f000,ef6c8b0c,...) at vfs_hash_get+0xed ffs_vgetf(c7b9b000,10b0400,80000,ef6c8b0c,1,...) at ffs_vgetf+0x49 softdep_sync_buf(c7b8c570,e0af0620,1,107,0,...) at softdep_sync_buf+0x4a3 ffs_syncvnode(c7b8c570,1,c789f000,568,0,...) at ffs_syncvnode+0x238 ffs_sync(c7b9b000,1,ef6c8c14,4ee,80,...) at ffs_sync+0x26f dounmount(c7b9b000,8080000,c789f000,473,5b85c5ea,...) at dounmount+0x447 unmount(c789f000,ef6c8cec,28177075,1,0,...) at unmount+0x310 syscallenter(c789f000,ef6c8ce4,c0d3601d,c10959f0,0,...) at syscallenter+0x263 syscall(ef6c8d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (22, FreeBSD ELF32, unmount), eip = 0x280dd6cb, esp = 0xbfbfe64c, ebp = 0xbfbfe718 --- lock order reversal: 1st 0xc7b8cdf0 ufs (ufs) @ kern/vfs_mount.c:1193 2nd 0xc75971b4 devfs (devfs) @ ufs/ffs/ffs_softdep.c:1832 KDB: stack backtrace: db_trace_self_wrapper(c0ef6376,38313a63,a3233,c0f2a721,c19b4888,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a3d01b,c0ef9d3f,c7169338,c7169200,ef6c8a04,...) at kdb_backtrace+0x2a _witness_debugger(c0ef9d3f,c75971b4,c0ee0f87,c7169200,c0f25331,...) at _witness_debugger+0x25 witness_checkorder(c75971b4,9,c0f25328,728,c7597220,...) at witness_checkorder+0x839 __lockmgr_args(c75971b4,80400,c7597220,0,0,...) at __lockmgr_args+0x814 vop_stdlock(ef6c8b24,4,c0ef0a47,80400,c759715c,...) at vop_stdlock+0x65 VOP_LOCK1_APV(c10156a0,ef6c8b24,c1213b30,c105abc0,c759715c,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c759715c,80400,c0f25328,728,c7b9b000,...) at _vn_lock+0x78 softdep_flushworklist(c7b9b000,ef6c8bd0,c789f000,576,0,...) at softdep_flushworklist+0x47 ffs_sync(c7b9b000,1,ef6c8c14,4ee,80,...) at ffs_sync+0x2fd dounmount(c7b9b000,8080000,c789f000,473,5b85c5ea,...) at dounmount+0x447 unmount(c789f000,ef6c8cec,28177075,1,0,...) at unmount+0x310 syscallenter(c789f000,ef6c8ce4,c0d3601d,c10959f0,0,...) at syscallenter+0x263 syscall(ef6c8d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (22, FreeBSD ELF32, unmount), eip = 0x280dd6cb, esp = 0xbfbfe64c, ebp = 0xbfbfe718 --- ** /dev/label/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes Jun 30 09:30:03 x4 su: pho to root on /dev/pts/0 ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 102557 files, 24047813 used, 94266168 free (3024 frags, 11782893 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** ** /dev/label/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 102557 files, 24047813 used, 94266168 free (3024 frags, 11782893 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** fsck -y /tmp watchdogd. Starting inetd. Thu Jun 30 09:33:38 CEST 2011 FreeBSD/i386 (x4.osted.lan) (console) login: Jun 30 10:24:52 x4 su: BAD SU pho to root on /dev/pts/0 Jun 30 10:24:58 x4 su: pho to root on /dev/pts/1 lock order reversal: 1st 0xe0b0ab60 bufwait (bufwait) @ kern/vfs_bio.c:2658 2nd 0xc82c891c snaplk (snaplk) @ ufs/ffs/ffs_snapshot.c:818 KDB: stack backtrace: db_trace_self_wrapper(c0ef6376,632e746f,3831383a,ef76000a,c0a4e90b,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a3d01b,c0ef9d3f,c7165370,c7169b58,ef76b508,...) at kdb_backtrace+0x2a _witness_debugger(c0ef9d3f,c82c891c,c0f24dcd,c7169b58,c0f24d57,...) at _witness_debugger+0x25 witness_checkorder(c82c891c,9,c0f24d4e,332,cabbe634,...) at witness_checkorder+0x839 __lockmgr_args(c82c891c,80400,cabbe634,0,0,...) at __lockmgr_args+0x814 ffs_lock(ef76b62c,e0b19560,100000,80400,cabbe570,...) at ffs_lock+0xa1 VOP_LOCK1_APV(c103fba0,ef76b62c,e0b19608,c105abc0,cabbe570,...) at VOP_LOCK1_APV+0xb5 _vn_lock(cabbe570,80400,c0f24d4e,332,0,...) at _vn_lock+0x78 ffs_snapshot(c7b9a000,c740a720,c0f281d0,176,ef76b980,...) at ffs_snapshot+0x2956 ffs_mount(c7b9a000,c7ba6400,ff,393,0,...) at ffs_mount+0x15a0 vfs_donmount(c7f4db80,211000,c71b8380,c71b8380,c8347000,...) at vfs_donmount+0x1147 nmount(c7f4db80,ef76bcec,ef76bd28,c0ef858a,0,...) at nmount+0x84 syscallenter(c7f4db80,ef76bce4,ef76bce4,0,c10a3f80,...) at syscallenter+0x263 syscall(ef76bd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (378, FreeBSD ELF32, nmount), eip = 0x280db34b, esp = 0xbfbfe0ec, ebp = 0xbfbfe868 --- lock order reversal: 1st 0xc82c891c snaplk (snaplk) @ ufs/ufs/ufs_vnops.c:957 2nd 0xcabbe5c8 ufs (ufs) @ ufs/ffs/ffs_snapshot.c:1620 KDB: stack backtrace: db_trace_self_wrapper(c0ef6376,a303236,c0000a00,c11df138,c7fa5950,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a3d01b,c0ef9d3f,c7169b58,c7169338,ef7f26ec,...) at kdb_backtrace+0x2a _witness_debugger(c0ef9d3f,cabbe5c8,c0ee9124,c7169338,c0f24d57,...) at _witness_debugger+0x25 witness_checkorder(cabbe5c8,9,c0f24d4e,654,0,...) at witness_checkorder+0x839 __lockmgr_args(cabbe5c8,80000,0,0,0,...) at __lockmgr_args+0x814 ffs_snapremove(cabbe570,c0f24c8c,ef7f2850,c0a4e90b,c0efefb4,...) at ffs_snapremove+0x11f ffs_truncate(cabbe570,0,0,c00,0,...) at ffs_truncate+0x617 ufs_inactive(ef7f2ad4,c0f46704,cabbe634,cabbe634,cabbe570,...) at ufs_inactive+0x1ed VOP_INACTIVE_APV(c103fba0,ef7f2ad4,c0f01952,94e,c105ab80,...) at VOP_INACTIVE_APV+0xc5 vinactive(c103fba0,ef7f2b08,c0f01952,8a5,0,...) at vinactive+0xb1 vputx(ef7f2c20,c0aa421d,cabbe570,ffffffdf,cabbe570,...) at vputx+0x36c vput(cabbe570,ffffffdf,cabbe570,ef7f2bd8,cabbe828,...) at vput+0x10 kern_unlinkat(c7fa58a0,ffffff9c,bfbfea92,0,0,...) at kern_unlinkat+0x23d kern_unlink(c7fa58a0,bfbfea92,0,ef7f2c7c,c0a472d3,...) at kern_unlink+0x2f unlink(c7fa58a0,ef7f2cec,ef7f2d28,c0ef858a,0,...) at unlink+0x22 syscallenter(c7fa58a0,ef7f2ce4,ef7f2ce4,0,0,...) at syscallenter+0x263 syscall(ef7f2d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (10, FreeBSD ELF32, unlink), eip = 0x2817253f, esp = 0xbfbfe7cc, ebp = 0xbfbfe848 --- panic: _mtx_lock_sleep: recursed on non-recursive mutex FFS @ ../../../ufs/ufs/ufs_quota.c:1239 cpuid = 3 KDB: enter: panic [ thread pid 3460 tid 100205 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> run pho db:0:pho> bt Tracing pid 3460 tid 100205 td 0xc9a282e0 kdb_enter(c0ef245a,c0ef245a,c0ef0c8d,ef8284bc,3,...) at kdb_enter+0x3a panic(c0ef0c8d,c0f28656,c0f297e1,4d7,c7884e48,...) at panic+0x134 _mtx_lock_sleep(c7884e48,c9a282e0,0,c0f297e1,4d7,...) at _mtx_lock_sleep+0x4a _mtx_lock_flags(c7884e48,0,c0f297e1,4d7,c9a282e0,...) at _mtx_lock_flags+0xf7 dqget(0,c7bab520,0,c7bab4fc,c7bc5d98,...) at dqget+0x9d getinoquota(c7bab4fc,ef828690,2,ef828684,c0d54965,...) at getinoquota+0x4b ufs_inactive(ef8286d4,c0f46704,c7bc5e5c,c7bc5e5c,c7bc5d98,...) at ufs_inactive+0x228 VOP_INACTIVE_APV(c103fba0,ef8286d4,c0f01952,94e,c105ab80,...) at VOP_INACTIVE_APV+0xc5 vinactive(c103fba0,ef828708,c0f01952,8a5,0,...) at vinactive+0xb1 vputx(ef828774,c0c29d43,c7bc5d98,0,c0f25328,...) at vputx+0x36c vput(c7bc5d98,0,c0f25328,242b,1,...) at vput+0x10 handle_workitem_remove(0,ef82879c,2,6b8,ef8287a0,...) at handle_workitem_remove+0x483 process_worklist_item(c1213b30,0,c0f25328,3129,fffee32e,...) at process_worklist_item+0x2b4 softdep_request_cleanup(c7886800,ca793ae0,c7872a80,4,1,...) at softdep_request_cleanup+0x382 ffs_alloc(ca40f3a0,0,0,6a3d58,0,...) at ffs_alloc+0x3ef ffs_balloc_ufs2(ca793ae0,0,0,800,c7872a80,...) at ffs_balloc_ufs2+0x110c ffs_write(ef828b94,c0f4712a,ca793ae0,0,ca793ae0,...) at ffs_write+0x2ad VOP_WRITE_APV(c103fba0,ef828b94,ca793ae0,264,0,...) at VOP_WRITE_APV+0x136 vn_write(c997e690,ef828c28,c7872a80,0,c9a282e0,...) at vn_write+0x1ca dofilewrite(ef828c28,ffffffff,ffffffff,0,c997e690,...) at dofilewrite+0x95 kern_writev(c9a282e0,4,ef828c28,ef828c48,1,...) at kern_writev+0x58 write(c9a282e0,ef828cec,ef828d80,ef828c88,0,...) at write+0x4f syscallenter(c9a282e0,ef828ce4,c0d35cd6,c0a2bba8,c10a3f80,...) at syscallenter+0x263 syscall(ef828d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x28185f93, esp = 0xbfbfed3c, ebp = 0xbfbfed68 --- db:0:bt> show allpcpu Current CPU: 3 cpuid = 0 dynamic pcpu = 0x7c9d00 curthread = 0xc71bd8a0: pid 11 "idle: cpu0" curpcb = 0xc6f37d80 fpcurthread = none idlethread = 0xc71bd8a0: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 dynamic pcpu = 0x5ecbd00 curthread = 0xc7fa22e0: pid 2603 "md5" curpcb = 0xef7c2d80 fpcurthread = none idlethread = 0xc71bd5c0: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: cpuid = 2 dynamic pcpu = 0x5eced00 curthread = 0xc71bd2e0: pid 11 "idle: cpu2" curpcb = 0xc6f3dd80 fpcurthread = none idlethread = 0xc71bd2e0: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 spin locks held: cpuid = 3 dynamic pcpu = 0x5ed1d00 curthread = 0xc9a282e0: pid 3460 "dd" curpcb = 0xef828d80 fpcurthread = none idlethread = 0xc71bd000: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 spin locks held: db:0:allpcpu> show alllocks Process 3460 (dd) thread 0xc9a282e0 (100205) exclusive lockmgr ufs (ufs) r = 0 (0xc7bc5df0) locked @ kern/vfs_subr.c:2134 exclusive sleep mutex FFS (FFS Lock) r = 0 (0xc7884e48) locked @ ufs/ffs/ffs_softdep.c:12632 exclusive lockmgr ufs (ufs) r = 0 (0xca793b38) locked @ kern/vfs_vnops.c:612 Process 3422 (suj22) thread 0xc7f4d8a0 (100153) exclusive lockmgr ufs (ufs) r = 0 (0xca7a05c8) locked @ kern/vfs_vnops.c:612 Process 2612 (sshd) thread 0xc7cd88a0 (100127) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xc7fa63cc) locked @ kern/uipc_sockbuf.c:148 Process 2610 (sshd) thread 0xc78da8a0 (100088) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xca73b08c) locked @ kern/uipc_sockbuf.c:148 Process 2609 (sshd) thread 0xc7cd98a0 (100122) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xc829d08c) locked @ kern/uipc_sockbuf.c:148 Process 2603 (md5) thread 0xc7fa22e0 (100172) exclusive lockmgr bufwait (bufwait) r = 0 (0xe0d982a0) locked @ kern/vfs_bio.c:1911 exclusive lockmgr ufs (ufs) r = 0 (0xc9a4c46c) locked @ dev/md/md.c:618 Process 2561 (sshd) thread 0xc7fa52e0 (100144) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xc801a08c) locked @ kern/uipc_sockbuf.c:148 Process 2552 (sshd) thread 0xc78dc8a0 (100137) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xc828556c) locked @ kern/uipc_sockbuf.c:148 Process 18 (softdepflush) thread 0xc744a8a0 (100076) exclusive lockmgr bufwait (bufwait) r = 0 (0xe0d35580) locked @ kern/vfs_bio.c:1911 exclusive lockmgr snaplk (snaplk) r = 0 (0xc82c891c) locked @ ufs/ffs/ffs_snapshot.c:1756 Process 9 (bufdaemon) thread 0xc75712e0 (100073) exclusive lockmgr ufs (ufs) r = 0 (0xca40bb38) locked @ kern/vfs_bio.c:2372 db:0:alllocks> show lockedvnods Locked vnodes 0xc7bc5d98: 0xc7bc5d98: tag ufs, type VREG tag ufs, type VREG usecount 0, writecount 0, refcount 1 mountedhere 0 usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VI_DOINGINACT) flags (VI_DOINGINACT) v_object 0xc7bb8088 ref 0 pages 0 v_object 0xc7bb8088 ref 0 pages 0 lock type ufs: EXCL by thread 0xc9a282e0 (pid 3460) lock type ufs: EXCL by thread 0xc9a282e0 (pid 3460) #0 0xc09ec6fe at __lockmgr_args+0xbfe #0 0xc09ec6fe at __lockmgr_args+0xbfe #1 0xc0c42591 at ffs_lock+0xa1 #1 0xc0c42591 at ffs_lock+0xa1 #2 0xc0d58255 at VOP_LOCK1_APV+0xb5 #2 0xc0d58255 at VOP_LOCK1_APV+0xb5 #3 0xc0aac668 at _vn_lock+0x78 #3 0xc0aac668 at _vn_lock+0x78 #4 0xc0a9f79b at vget+0xbb #4 0xc0a9f79b at vget+0xbb #5 0xc0a9074d at vfs_hash_get+0xed #5 0xc0a9074d at vfs_hash_get+0xed #6 0xc0c3ca89 at ffs_vgetf+0x49 #6 0xc0c3ca89 at ffs_vgetf+0x49 #7 0xc0c29920 at handle_workitem_remove+0x60 #7 0xc0c29920 at handle_workitem_remove+0x60 #8 0xc0c291a4 at process_worklist_item+0x2b4 #8 0xc0c291a4 at process_worklist_item+0x2b4 #9 0xc0c392c2 at softdep_request_cleanup+0x382 #9 0xc0c392c2 at softdep_request_cleanup+0x382 #10 0xc0c135cf at ffs_alloc+0x3ef #10 0xc0c135cf at ffs_alloc+0x3ef #11 0xc0c175ec at ffs_balloc_ufs2+0x110c #11 0xc0c175ec at ffs_balloc_ufs2+0x110c #12 0xc0c41b5d at ffs_write+0x2ad #12 0xc0c41b5d at ffs_write+0x2ad #13 0xc0d58d86 at VOP_WRITE_APV+0x136 #13 0xc0d58d86 at VOP_WRITE_APV+0x136 #14 0xc0aada4a at vn_write+0x1ca #14 0xc0aada4a at vn_write+0x1ca #15 0xc0a512a5 at dofilewrite+0x95 #15 0xc0a512a5 at dofilewrite+0x95 #16 0xc0a52a48 at kern_writev+0x58 #16 0xc0a52a48 at kern_writev+0x58 #17 0xc0a52b5f at write+0x4f #17 0xc0a52b5f at write+0x4f ino 1742870, on dev ada0s1e ino 1742870, on dev ada0s1e 0xc9a4c414: 0xc9a4c414: tag ufs, type VREG tag ufs, type VREG usecount 1, writecount 1, refcount 1898 mountedhere 0 usecount 1, writecount 1, refcount 1898 mountedhere 0 flags (VV_MD) flags (VV_MD) v_object 0xc83e5330 ref 0 pages 227452 v_object 0xc83e5330 ref 0 pages 227452 lock type ufs: EXCL by thread 0xc7fa22e0 (pid 2603) lock type ufs: EXCL by thread 0xc7fa22e0 (pid 2603) #0 0xc09ec6fe at __lockmgr_args+0xbfe #0 0xc09ec6fe at __lockmgr_args+0xbfe #1 0xc0c42591 at ffs_lock+0xa1 #1 0xc0c42591 at ffs_lock+0xa1 #2 0xc0d58255 at VOP_LOCK1_APV+0xb5 #2 0xc0d58255 at VOP_LOCK1_APV+0xb5 #3 0xc0aac668 at _vn_lock+0x78 #3 0xc0aac668 at _vn_lock+0x78 #4 0xc0732078 at mdstart_vnode+0x498 #4 0xc0732078 at mdstart_vnode+0x498 #5 0xc0730d10 at md_kthread+0x1c0 #5 0xc0730d10 at md_kthread+0x1c0 #6 0xc09d7878 at fork_exit+0xb8 #6 0xc09d7878 at fork_exit+0xb8 #7 0xc0d1ee84 at fork_trampoline+0x8 #7 0xc0d1ee84 at fork_trampoline+0x8 ino 6194208, on dev ada0s1e ino 6194208, on dev ada0s1e 0xca793ae0: 0xca793ae0: tag ufs, type VREG tag ufs, type VREG usecount 1, writecount 1, refcount 1 mountedhere 0 usecount 1, writecount 1, refcount 1 mountedhere 0 flags () flags () v_object 0xc9a2c6e8 ref 0 pages 0 v_object 0xc9a2c6e8 ref 0 pages 0 lock type ufs: EXCL by thread 0xc9a282e0 (pid 3460) lock type ufs: EXCL by thread 0xc9a282e0 (pid 3460) #0 0xc09ec6fe at __lockmgr_args+0xbfe #0 0xc09ec6fe at __lockmgr_args+0xbfe #1 0xc0c42591 at ffs_lock+0xa1 #1 0xc0c42591 at ffs_lock+0xa1 #2 0xc0d58255 at VOP_LOCK1_APV+0xb5 #2 0xc0d58255 at VOP_LOCK1_APV+0xb5 #3 0xc0aac668 at _vn_lock+0x78 #3 0xc0aac668 at _vn_lock+0x78 #4 0xc0aad9d6 at vn_write+0x156 #4 0xc0aad9d6 at vn_write+0x156 #5 0xc0a512a5 at dofilewrite+0x95 #5 0xc0a512a5 at dofilewrite+0x95 #6 0xc0a52a48 at kern_writev+0x58 #6 0xc0a52a48 at kern_writev+0x58 #7 0xc0a52b5f at write+0x4f #7 0xc0a52b5f at write+0x4f #8 0xc0a472d3 at syscallenter+0x263 #8 0xc0a472d3 at syscallenter+0x263 #9 0xc0d357bf at syscall+0x4f #9 0xc0d357bf at syscall+0x4f #10 0xc0d1ee71 at Xint0x80_syscall+0x21 #10 0xc0d1ee71 at Xint0x80_syscall+0x21 ino 1742861, on dev ada0s1e ino 1742861, on dev ada0s1e 0xcb2dfc3c: 0xcb2dfc3c: tag ufs, type VREG tag ufs, type VREG usecount 1, writecount 0, refcount 10 mountedhere 0 usecount 1, writecount 0, refcount 10 mountedhere 0 flags (VV_SYSTEM) flags (VV_SYSTEM) v_object 0xca786770 ref 0 pages 39032 v_object 0xca786770 ref 0 pages 39032 lock type snaplk: EXCL by thread 0xc744a8a0 (pid 18) lock type snaplk: EXCL by thread 0xc744a8a0 (pid 18) with exclusive waiters pending with exclusive waiters pending #0 0xc09ec6fe at __lockmgr_args+0xbfe #0 0xc09ec6fe at __lockmgr_args+0xbfe #1 0xc0c1ccca at ffs_snapblkfree+0xea #1 0xc0c1ccca at ffs_snapblkfree+0xea #2 0xc0c10bcc at ffs_blkfree+0x5c #2 0xc0c10bcc at ffs_blkfree+0x5c #3 0xc0c27940 at handle_workitem_freefrag+0xf0 #3 0xc0c27940 at handle_workitem_freefrag+0xf0 #4 0xc0c291bd at process_worklist_item+0x2cd #4 0xc0c291bd at process_worklist_item+0x2cd #5 0xc0c39f68 at softdep_process_worklist+0x88 #5 0xc0c39f68 at softdep_process_worklist+0x88 #6 0xc0c3a530 at softdep_flush+0x2a0 #6 0xc0c3a530 at softdep_flush+0x2a0 #7 0xc09d7878 at fork_exit+0xb8 #7 0xc09d7878 at fork_exit+0xb8 #8 0xc0d1ee84 at fork_trampoline+0x8 #8 0xc0d1ee84 at fork_trampoline+0x8 ino 2612, on dev md5a ino 2612, on dev md5a 0xcb2dbd98: 0xcb2dbd98: tag ufs, type VREG tag ufs, type VREG usecount 1, writecount 0, refcount 523 mountedhere 0 usecount 1, writecount 0, refcount 523 mountedhere 0 flags (VV_SYSTEM) flags (VV_SYSTEM) v_object 0xc9a2e660 ref 0 pages 4688 v_object 0xc9a2e660 ref 0 pages 4688 lock type snaplk: EXCL by thread 0xc744a8a0 (pid 18) lock type snaplk: EXCL by thread 0xc744a8a0 (pid 18) with exclusive waiters pending with exclusive waiters pending #0 0xc09ec6fe at __lockmgr_args+0xbfe #0 0xc09ec6fe at __lockmgr_args+0xbfe #1 0xc0c1ccca at ffs_snapblkfree+0xea #1 0xc0c1ccca at ffs_snapblkfree+0xea #2 0xc0c10bcc at ffs_blkfree+0x5c #2 0xc0c10bcc at ffs_blkfree+0x5c #3 0xc0c27940 at handle_workitem_freefrag+0xf0 #3 0xc0c27940 at handle_workitem_freefrag+0xf0 #4 0xc0c291bd at process_worklist_item+0x2cd #4 0xc0c291bd at process_worklist_item+0x2cd #5 0xc0c39f68 at softdep_process_worklist+0x88 #5 0xc0c39f68 at softdep_process_worklist+0x88 #6 0xc0c3a530 at softdep_flush+0x2a0 #6 0xc0c3a530 at softdep_flush+0x2a0 #7 0xc09d7878 at fork_exit+0xb8 #7 0xc09d7878 at fork_exit+0xb8 #8 0xc0d1ee84 at fork_trampoline+0x8 #8 0xc0d1ee84 at fork_trampoline+0x8 ino 2340, on dev md5a ino 2340, on dev md5a 0xca40bae0: 0xca40bae0: tag ufs, type VREG tag ufs, type VREG usecount 0, writecount 0, refcount 2 mountedhere 0 usecount 0, writecount 0, refcount 2 mountedhere 0 flags () flags () v_object 0xcb1b7770 ref 0 pages 7 v_object 0xcb1b7770 ref 0 pages 7 lock type ufs: EXCL by thread 0xc75712e0 (pid 9) lock type ufs: EXCL by thread 0xc75712e0 (pid 9) #0 0xc09ec6fe at __lockmgr_args+0xbfe #0 0xc09ec6fe at __lockmgr_args+0xbfe #1 0xc0c42591 at ffs_lock+0xa1 #1 0xc0c42591 at ffs_lock+0xa1 #2 0xc0d58255 at VOP_LOCK1_APV+0xb5 #2 0xc0d58255 at VOP_LOCK1_APV+0xb5 #3 0xc0aac668 at _vn_lock+0x78 #3 0xc0aac668 at _vn_lock+0x78 #4 0xc0a8632b at flushbufqueues+0x56b #4 0xc0a8632b at flushbufqueues+0x56b #5 0xc0a86629 at buf_do_flush+0x19 #5 0xc0a86629 at buf_do_flush+0x19 #6 0xc0a867fa at buf_daemon+0xea #6 0xc0a867fa at buf_daemon+0xea #7 0xc09d7878 at fork_exit+0xb8 #7 0xc09d7878 at fork_exit+0xb8 #8 0xc0d1ee84 at fork_trampoline+0x8 #8 0xc0d1ee84 at fork_trampoline+0x8 ino 16486, on dev md5a ino 16486, on dev md5a 0xca7a0570: 0xca7a0570: tag ufs, type VREG tag ufs, type VREG usecount 1, writecount 1, refcount 3 mountedhere 0 usecount 1, writecount 1, refcount 3 mountedhere 0 flags () flags () v_object 0xca37b220 ref 0 pages 4 v_object 0xca37b220 ref 0 pages 4 lock type ufs: EXCL by thread 0xc7f4d8a0 (pid 3422) lock type ufs: EXCL by thread 0xc7f4d8a0 (pid 3422) #0 0xc09ec6fe at __lockmgr_args+0xbfe #0 0xc09ec6fe at __lockmgr_args+0xbfe #1 0xc0c42591 at ffs_lock+0xa1 #1 0xc0c42591 at ffs_lock+0xa1 #2 0xc0d58255 at VOP_LOCK1_APV+0xb5 #2 0xc0d58255 at VOP_LOCK1_APV+0xb5 #3 0xc0aac668 at _vn_lock+0x78 #3 0xc0aac668 at _vn_lock+0x78 #4 0xc0aad9d6 at vn_write+0x156 #4 0xc0aad9d6 at vn_write+0x156 #5 0xc0a512a5 at dofilewrite+0x95 #5 0xc0a512a5 at dofilewrite+0x95 #6 0xc0a52a48 at kern_writev+0x58 #6 0xc0a52a48 at kern_writev+0x58 #7 0xc0a52b5f at write+0x4f #7 0xc0a52b5f at write+0x4f #8 0xc0a472d3 at syscallenter+0x263 #8 0xc0a472d3 at syscallenter+0x263 #9 0xc0d357bf at syscall+0x4f #9 0xc0d357bf at syscall+0x4f #10 0xc0d1ee71 at Xint0x80_syscall+0x21 #10 0xc0d1ee71 at Xint0x80_syscall+0x21 ino 50414, on dev md5a ino 50414, on dev md5a db:0:lockedvnods> show mount 0xc7870b50 /dev/ad4s1a on / (ufs) 0xc7871000 devfs on /dev (devfs) 0xc7b925a8 /dev/ad4s1f on /home (ufs) 0xc7b9ab50 /dev/ad4s1d on /usr (ufs) 0xc7b9a87c /dev/ad4s1e on /var (ufs) 0xc7b9a5a8 procfs on /proc (procfs) 0xc7b9a2d4 /dev/label/tmp on /tmp (ufs) 0xc7b9a000 /dev/md5a on /mnt (ufs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 3464 2621 2620 1001 S nanslp 0xc1097304 sleep 3460 3439 3439 2 RL CPU 3 dd 3439 3438 3439 2 Ss wait 0xc77e4b00 sh 3438 2173 2173 0 S piperd 0xc7f967a8 cron 3422 2580 2580 0 D+ snaplk 0xc82c891c suj22 2622 2620 2620 1001 S piperd 0xc78e8ab8 awk 2621 2620 2620 1001 S wait 0xc99c7580 sh 2620 2618 2620 1001 Ss wait 0xc9a62840 sh 2619 2617 2619 1001 Ss kqread 0xc787e080 tail 2618 2612 2612 1001 S select 0xc83c8364 sshd 2617 2610 2610 1001 S select 0xc77c5564 sshd 2616 2615 2616 1001 Ss+ select 0xc83c8424 top 2615 2609 2609 1001 S select 0xc83c8324 sshd 2612 2151 2612 0 Ss sbwait 0xc7fa63f8 sshd 2610 2151 2610 0 Ss sbwait 0xca73b0b8 sshd 2609 2151 2609 0 Ss sbwait 0xc829d0b8 sshd 2603 0 0 0 RL CPU 1 [md5] 2580 2574 2580 0 S+ wait 0xc8345b00 sh 2574 2570 2574 0 S+ wait 0xc7bb7000 bash 2570 2569 2570 0 S+ pause 0xc8345898 csh 2569 2564 2569 1001 S+ wait 0xc7899000 su 2564 2563 2564 1001 Ss+ wait 0xc8347b00 bash 2563 2561 2561 1001 S select 0xc7810c24 sshd 2561 2151 2561 0 Ss sbwait 0xc801a0b8 sshd 2555 2554 2555 1001 Ss+ ttyin 0xc7cae870 bash 2554 2552 2552 1001 S select 0xc7810a64 sshd 2552 2151 2552 0 Ss sbwait 0xc8285598 sshd 2364 1 2364 0 Ss+ ttyin 0xc757e070 getty 2363 1 2363 0 Ss+ ttyin 0xc757e270 getty 2362 1 2362 0 Ss+ ttyin 0xc757e470 getty 2361 1 2361 0 Ss+ ttyin 0xc72e5070 getty 2360 1 2360 0 Ss+ ttyin 0xc72e5270 getty 2359 1 2359 0 Ss+ ttyin 0xc72e5470 getty 2358 1 2358 0 Ss+ ttyin 0xc72e5670 getty 2357 1 2357 0 Ss+ ttyin 0xc72e5870 getty 2356 1 2356 0 Ss+ ttyin 0xc72e5a70 getty 2317 1 2317 0 Ss select 0xc7810b24 inetd 2285 1 2285 0 Ss nanslp 0xc1097304 watchdogd 2173 1 2173 0 Ss nanslp 0xc1097304 cron 2165 1 2165 25 Ss pause 0xc7cd2058 sendmail 2159 1 2159 0 Ss select 0xc78108e4 sendmail 2151 1 2151 0 Ss select 0xc77c54e4 sshd 2091 1 2091 0 Ss select 0xc77c56e4 moused 2013 1 2013 0 Ss select 0xc78813a4 ntpd 1909 1908 1908 0 S (threaded) nfsd 100147 S rpcsvc 0xc784ca10 nfsd: service 100146 S rpcsvc 0xc784ca50 nfsd: service 100145 S rpcsvc 0xc784ca90 nfsd: service 100115 S rpcsvc 0xc784b8d0 nfsd: master 1908 1 1908 0 Ss select 0xc78813e4 nfsd 1898 1 1898 0 Ss select 0xc7810be4 mountd 1799 1 1799 0 Ss select 0xc7881424 rpcbind 1716 1 1716 0 Ss select 0xc7810ea4 syslogd 1516 1 1516 0 Ss select 0xc7810ca4 devd 18 0 0 0 DL snaprdb 0xc7f215f0 [softdepflush] 17 0 0 0 DL vlruwt 0xc77e2580 [vnlru] 16 0 0 0 DL syncer 0xc120d4f4 [syncer] 9 0 0 0 DL snaplk 0xc82c891c [bufdaemon] 8 0 0 0 DL pgzero 0xc12159dc [pagezero] 7 0 0 0 DL psleep 0xc12155c8 [vmdaemon] 6 0 0 0 DL psleep 0xc1215590 [pagedaemon] 5 0 0 0 DL ccb_scan 0xc105c054 [xpt_thrd] 4 0 0 0 DL waiting_ 0xc120f018 [sctp_iterator] 3 0 0 0 DL - 0xc755f43c [fdc0] 2 0 0 0 SL - 0xc7479000 [fw0_probe] 15 0 0 0 DL (threaded) [usb] 100058 D - 0xc7450d34 [usbus5] 100057 D - 0xc7450d04 [usbus5] 100056 D - 0xc7450cd4 [usbus5] 100055 D - 0xc7450ca4 [usbus5] 100053 D - 0xc7441b5c [usbus4] 100052 D - 0xc7441b2c [usbus4] 100051 D - 0xc7441afc [usbus4] 100050 D - 0xc7441acc [usbus4] 100049 D - 0xc743db5c [usbus3] 100048 D - 0xc743db2c [usbus3] 100047 D - 0xc743dafc [usbus3] 100046 D - 0xc743dacc [usbus3] 100045 D - 0xc7434b5c [usbus2] 100044 D - 0xc7434b2c [usbus2] 100043 D - 0xc7434afc [usbus2] 100042 D - 0xc7434acc [usbus2] 100040 D - 0xc742fb5c [usbus1] 100039 D - 0xc742fb2c [usbus1] 100038 D - 0xc742fafc [usbus1] 100037 D - 0xc742facc [usbus1] 100035 D - 0xc741fb5c [usbus0] 100034 D - 0xc741fb2c [usbus0] 100033 D - 0xc741fafc [usbus0] 100032 D - 0xc741facc [usbus0] 14 0 0 0 DL - 0xc1096104 [yarrow] 13 0 0 0 DL (threaded) [geom] 100015 D - 0xc1093d28 [g_down] 100014 D - 0xc1093d24 [g_up] 100013 D - 0xc1093d1c [g_event] 12 0 0 0 WL (threaded) [intr] 100066 I [irq12: psm0] 100065 I [irq1: atkbd0] 100063 I [swi0: uart] 100060 I [irq20: fwohci0] 100059 I [irq14: ata0] 100054 I [irq19: ehci0] 100041 I [irq18: ohci2 ohci4] 100036 I [irq17: ohci1 ohci3] 100031 I [irq16: hdac1 ohci0] 100030 I [irq22: ahci0] 100029 I [irq257: re0] 100028 I [irq256: hdac0] 100027 I [swi5: +] 100026 I [swi2: cambio] 100021 I [swi6: task queue] 100019 I [swi6: Giant taskq] 100012 I [swi3: vm] 100011 I [swi4: clock] 100010 I [swi4: clock] 100009 I [swi4: clock] 100008 I [swi4: clock] 100007 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100006 CanRun [idle: cpu3] 100005 Run CPU 2 [idle: cpu2] 100004 CanRun [idle: cpu1] 100003 Run CPU 0 [idle: cpu0] 1 0 1 0 SLs wait 0xc71bbb00 [init] 10 0 0 0 DL audit_wo 0xc1213300 [audit] 0 0 0 0 DLs (threaded) [kernel] 100077 D - 0xc77b4740 [mca taskq] 100067 D - 0xc1096104 [deadlkres] 100061 D - 0xc74817c0 [fw0_taskq] 100025 D - 0xc71936c0 [acpi_task_2] 100024 D - 0xc71936c0 [acpi_task_1] 100023 D - 0xc71936c0 [acpi_task_0] 100022 D - 0xc7193700 [kqueue taskq] 100020 D - 0xc7193780 [ffs_trim taskq] 100018 D - 0xc7193800 [thread taskq] 100016 D - 0xc7193c00 [firmware taskq] 100000 D sched 0xc1093e00 [swapper] db:0:ps> allt Tracing command sleep pid 3464 tid 100096 td 0xc77ec2e0 sched_switch(c77ec2e0,0,104,191,51312618,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c77ec2e0,0,c0ef7c00,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(2711,c0a437f0,c77ec2e0,1,100,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c1097304,6c,c0ef3997,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c1097304,0,16c,c0ef3997,2711,...) at _sleep+0x328 kern_nanosleep(c77ec2e0,ef65ec34,ef65ec3c,a,0,...) at kern_nanosleep+0xc1 nanosleep(c77ec2e0,ef65ecec,ef65ed28,c0ef858a,0,...) at nanosleep+0x6f syscallenter(c77ec2e0,ef65ece4,ef65ece4,0,0,...) at syscallenter+0x263 syscall(ef65ed28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x281664f7, esp = 0xbfbfec5c, ebp = 0xbfbfeca8 --- Tracing command dd pid 3460 tid 100205 td 0xc9a282e0 kdb_enter(c0ef245a,c0ef245a,c0ef0c8d,ef8284bc,3,...) at kdb_enter+0x3a panic(c0ef0c8d,c0f28656,c0f297e1,4d7,c7884e48,...) at panic+0x134 _mtx_lock_sleep(c7884e48,c9a282e0,0,c0f297e1,4d7,...) at _mtx_lock_sleep+0x4a _mtx_lock_flags(c7884e48,0,c0f297e1,4d7,c9a282e0,...) at _mtx_lock_flags+0xf7 dqget(0,c7bab520,0,c7bab4fc,c7bc5d98,...) at dqget+0x9d getinoquota(c7bab4fc,ef828690,2,ef828684,c0d54965,...) at getinoquota+0x4b ufs_inactive(ef8286d4,c0f46704,c7bc5e5c,c7bc5e5c,c7bc5d98,...) at ufs_inactive+0x228 VOP_INACTIVE_APV(c103fba0,ef8286d4,c0f01952,94e,c105ab80,...) at VOP_INACTIVE_APV+0xc5 vinactive(c103fba0,ef828708,c0f01952,8a5,0,...) at vinactive+0xb1 vputx(ef828774,c0c29d43,c7bc5d98,0,c0f25328,...) at vputx+0x36c vput(c7bc5d98,0,c0f25328,242b,1,...) at vput+0x10 handle_workitem_remove(0,ef82879c,2,6b8,ef8287a0,...) at handle_workitem_remove+0x483 process_worklist_item(c1213b30,0,c0f25328,3129,fffee32e,...) at process_worklist_item+0x2b4 softdep_request_cleanup(c7886800,ca793ae0,c7872a80,4,1,...) at softdep_request_cleanup+0x382 ffs_alloc(ca40f3a0,0,0,6a3d58,0,...) at ffs_alloc+0x3ef ffs_balloc_ufs2(ca793ae0,0,0,800,c7872a80,...) at ffs_balloc_ufs2+0x110c ffs_write(ef828b94,c0f4712a,ca793ae0,0,ca793ae0,...) at ffs_write+0x2ad VOP_WRITE_APV(c103fba0,ef828b94,ca793ae0,264,0,...) at VOP_WRITE_APV+0x136 vn_write(c997e690,ef828c28,c7872a80,0,c9a282e0,...) at vn_write+0x1ca dofilewrite(ef828c28,ffffffff,ffffffff,0,c997e690,...) at dofilewrite+0x95 kern_writev(c9a282e0,4,ef828c28,ef828c48,1,...) at kern_writev+0x58 write(c9a282e0,ef828cec,ef828d80,ef828c88,0,...) at write+0x4f syscallenter(c9a282e0,ef828ce4,c0d35cd6,c0a2bba8,c10a3f80,...) at syscallenter+0x263 syscall(ef828d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x28185f93, esp = 0xbfbfed3c, ebp = 0xbfbfed68 --- Tracing command sh pid 3439 tid 100117 td 0xc789eb80 sched_switch(c789eb80,0,104,191,4ce14f75,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c789eb80,0,c0ef7c00,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c77e4b00,6c,c0efa9de,100,0,...) at sleepq_wait_sig+0x17 _sleep(c77e4b00,c77e4b88,16c,c0efa9de,0,...) at _sleep+0x35c kern_wait(c789eb80,ffffffff,ef6ccc44,2,0,...) at kern_wait+0xc06 wait4(c789eb80,ef6cccec,ef6ccd28,c0ef858a,0,...) at wait4+0x3b syscallenter(c789eb80,ef6ccce4,ef6ccce4,0,0,...) at syscallenter+0x263 syscall(ef6ccd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2816b29b, esp = 0xbfbfeb5c, ebp = 0xbfbfeb78 --- Tracing command cron pid 3438 tid 100192 td 0xcad855c0 sched_switch(cad855c0,0,104,191,283dc7cf,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(cad855c0,0,c0ef7c00,1a7,5c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7f967a8,5c,c0efa40f,100,0,...) at sleepq_wait_sig+0x17 _sleep(c7f967a8,c7f96918,15c,c0efa40f,0,...) at _sleep+0x35c pipe_read(c7d44a10,ef801c28,c7872880,0,cad855c0,...) at pipe_read+0x417 dofileread(ef801c28,ffffffff,ffffffff,0,c7d44a10,...) at dofileread+0x9e kern_readv(cad855c0,5,ef801c28,ef801c48,1,...) at kern_readv+0x58 read(cad855c0,ef801cec,ef801d28,c0ef858a,0,...) at read+0x4f syscallenter(cad855c0,ef801ce4,ef801ce4,0,0,...) at syscallenter+0x263 syscall(ef801d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x281a0fb3, esp = 0xbfbfe5cc, ebp = 0xbfbfe5e8 --- Tracing command suj22 pid 3422 tid 100153 td 0xc7f4d8a0 sched_switch(c7f4d8a0,0,104,191,4f926110,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,60,...) at mi_switch+0x1eb sleepq_switch(c7f4d8a0,0,c0ef7c00,268,60,...) at sleepq_switch+0x162 sleepq_wait(c82c891c,60,c0f24dcd,4,0,...) at sleepq_wait+0x63 __lockmgr_args(c82c891c,80900,c9a5fa48,0,0,...) at __lockmgr_args+0xb1f ffs_copyonwrite(c9a5f984,e0d1e2a0,0,e0d1e2a0,ca7a0570,...) at ffs_copyonwrite+0x2c2 ffs_geom_strategy(c9a5fa7c,e0d1e2a0,ef76e9c0,3,c7f4d8a0,...) at ffs_geom_strategy+0xeb ufs_strategy(ef76e9fc,c0f46eef,c7f4d8a0,e0d1e2a0,4,...) at ufs_strategy+0xd3 VOP_STRATEGY_APV(c103fba0,ef76e9fc,c0efefab,c105ac20,ca7a0570,...) at VOP_STRATEGY_APV+0xe5 bufstrategy(ca7a0668,e0d1e2a0,3a6,1000,0,...) at bufstrategy+0x61 bufwrite(e0d1e2a0,1000,ef76ead0,c0c41d78,e0d1e2a0,...) at bufwrite+0x159 bawrite(e0d1e2a0,1000,ef76ec28,1000,c7872d00,...) at bawrite+0x5c ffs_write(ef76eb94,c0f4712a,ca7a0570,0,ca7a0570,...) at ffs_write+0x4c8 VOP_WRITE_APV(c103fba0,ef76eb94,ca7a0570,264,0,...) at VOP_WRITE_APV+0x136 vn_write(c7cb5690,ef76ec28,c7872d00,0,c7f4d8a0,...) at vn_write+0x1ca dofilewrite(ef76ec28,ffffffff,ffffffff,0,c7cb5690,...) at dofilewrite+0x95 kern_writev(c7f4d8a0,3,ef76ec28,ef76ec48,1,...) at kern_writev+0x58 write(c7f4d8a0,ef76ecec,ef76ed28,c0ef858a,0,...) at write+0x4f syscallenter(c7f4d8a0,ef76ece4,ef76ece4,0,c10a3f80,...) at syscallenter+0x263 syscall(ef76ed28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x28181f93, esp = 0xbfbfe7cc, ebp = 0xbfbfe878 --- Tracing command awk pid 2622 tid 100125 td 0xc7cd9000 sched_switch(c7cd9000,0,104,191,50e0e331,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c7cd9000,0,c0ef7c00,1a7,5c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c78e8ab8,5c,c0efa40f,100,0,...) at sleepq_wait_sig+0x17 _sleep(c78e8ab8,c78e8c28,15c,c0efa40f,0,...) at _sleep+0x35c pipe_read(c7cb58f8,ef6f4c28,c7f08100,0,c7cd9000,...) at pipe_read+0x417 dofileread(ef6f4c28,ffffffff,ffffffff,0,c7cb58f8,...) at dofileread+0x9e kern_readv(c7cd9000,0,ef6f4c28,ef6f4c48,1,...) at kern_readv+0x58 read(c7cd9000,ef6f4cec,ef6f4d80,ef6f4c88,0,...) at read+0x4f syscallenter(c7cd9000,ef6f4ce4,c0d35cd6,0,0,...) at syscallenter+0x263 syscall(ef6f4d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x281b8fb3, esp = 0xbfbfe93c, ebp = 0xbfbfe958 --- Tracing command sh pid 2621 tid 100176 td 0xc7f515c0 sched_switch(c7f515c0,0,104,191,50f4f5ef,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c7f515c0,0,c0ef7c00,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c99c7580,6c,c0efa9de,100,0,...) at sleepq_wait_sig+0x17 _sleep(c99c7580,c99c7608,16c,c0efa9de,0,...) at _sleep+0x35c kern_wait(c7f515c0,ffffffff,ef7cec44,2,0,...) at kern_wait+0xc06 wait4(c7f515c0,ef7cecec,ef7ced28,c0ef858a,0,...) at wait4+0x3b syscallenter(c7f515c0,ef7cece4,ef7cece4,0,0,...) at syscallenter+0x263 syscall(ef7ced28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2816b29b, esp = 0xbfbfe97c, ebp = 0xbfbfe998 --- Tracing command sh pid 2620 tid 100182 td 0xc83d48a0 sched_switch(c83d48a0,0,104,191,4938bc81,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c83d48a0,0,c0ef7c00,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c9a62840,6c,c0efa9de,100,0,...) at sleepq_wait_sig+0x17 _sleep(c9a62840,c9a628c8,16c,c0efa9de,0,...) at _sleep+0x35c kern_wait(c83d48a0,ffffffff,ef7e3c44,2,0,...) at kern_wait+0xc06 wait4(c83d48a0,ef7e3cec,2820fb0c,1,0,...) at wait4+0x3b syscallenter(c83d48a0,ef7e3ce4,c0d3601d,c10959f0,0,...) at syscallenter+0x263 syscall(ef7e3d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2816b29b, esp = 0xbfbfeb2c, ebp = 0xbfbfeb48 --- Tracing command tail pid 2619 tid 100098 td 0xc7573b80 sched_switch(c7573b80,0,104,191,2856db72,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,68,...) at mi_switch+0x1eb sleepq_switch(c7573b80,0,c0ef7c00,1a7,68,...) at sleepq_switch+0x162 sleepq_catch_signals(3e9,c0a437f0,c7573b80,2,100,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c787e080,68,c0eec9b4,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c787e080,c787e080,168,c0eec9b4,3e9,...) at _sleep+0x328 kern_kevent(c7573b80,4,0,1,ef664c28,...) at kern_kevent+0x364 kevent(c7573b80,ef664cec,ef664d28,c0ef858a,0,...) at kevent+0x19e syscallenter(c7573b80,ef664ce4,ef664ce4,0,c10a3f80,...) at syscallenter+0x263 syscall(ef664d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (363, FreeBSD ELF32, kevent), eip = 0x2816a62b, esp = 0xbfbfeb1c, ebp = 0xbfbfebd8 --- Tracing command sshd pid 2618 tid 100121 td 0xc7cd9b80 sched_switch(c7cd9b80,0,104,191,4347e4b5,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7cd9b80,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c09f45ba,c83c8350,0,c0ef0a47,c7cd9b80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c83c8364,0,ef6e4a4c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c83c8364,c83c8350,c0efa251,627,c78ab7a8,...) at _cv_wait_sig+0x243 seltdwait(c78ab7a8,58,c7ba0a00,c7cd9b80,f3b,...) at seltdwait+0xa2 kern_select(c7cd9b80,c,288060b4,288060bc,0,0,20,c7cd3000,c7cd3000) at kern_select+0x504 select(c7cd9b80,ef6e4cec,ef6e4d28,c0ef858a,0,...) at select+0x66 syscallenter(c7cd9b80,ef6e4ce4,ef6e4ce4,0,0,...) at syscallenter+0x263 syscall(ef6e4d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x284c4f33, esp = 0xbfbfe19c, ebp = 0xbfbfe1e8 --- Tracing command sshd pid 2617 tid 100093 td 0xc77ecb80 sched_switch(c77ecb80,0,104,191,b42d1473,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c77ecb80,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c09f45ba,c77c5550,0,c0ef0a47,c77ecb80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c77c5564,0,ef655a4c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c77c5564,c77c5550,c0efa251,627,c7cb5e38,...) at _cv_wait_sig+0x243 seltdwait(c7cb5e38,58,c7ba0880,c77ecb80,246,...) at seltdwait+0xa2 kern_select(c77ecb80,c,288060b4,288060bc,0,0,20,c7bb6b00,c7bb6b00) at kern_select+0x504 select(c77ecb80,ef655cec,ef655d28,c0ef858a,0,...) at select+0x66 syscallenter(c77ecb80,ef655ce4,ef655ce4,0,0,...) at syscallenter+0x263 syscall(ef655d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x284c4f33, esp = 0xbfbfe19c, ebp = 0xbfbfe1e8 --- Tracing command top pid 2616 tid 100181 td 0xc83d4b80 sched_switch(c83d4b80,0,104,191,28323c0c,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c83d4b80,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(ef7e0a14,c09f45ba,c83c8410,0,c83d4b80,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c83c8424,0,ef7e0a4c,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c83c8424,c83c8410,3e9,627,c7fee188,...) at _cv_timedwait_sig+0x252 seltdwait(ef7e0bf8,ef7e0c00,c78cda00,c83d4b80,31204830,...) at seltdwait+0x8a kern_select(c83d4b80,2,bfbfebac,0,0,ef7e0c40,20,1,0) at kern_select+0x504 select(c83d4b80,ef7e0cec,c0a2950c,c10a3fb0,0,...) at select+0x66 syscallenter(c83d4b80,ef7e0ce4,ef7e0cc0,c0a2bba8,c10a3f80,...) at syscallenter+0x263 syscall(ef7e0d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x281eef33, esp = 0xbfbfeb4c, ebp = 0xbfbfeca8 --- Tracing command sshd pid 2615 tid 100087 td 0xc77ed5c0 sched_switch(c77ed5c0,0,104,191,283566bf,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c77ed5c0,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c09f45ba,c83c8310,0,c0ef0a47,c77ed5c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c83c8324,0,ef625a4c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c83c8324,c83c8310,c0efa251,627,c78aae38,...) at _cv_wait_sig+0x243 seltdwait(c78aae38,58,c7f08e80,c77ed5c0,246,...) at seltdwait+0xa2 kern_select(c77ed5c0,a,288060b4,288060bc,0,0,20,c7429b00,c7429b00) at kern_select+0x504 select(c77ed5c0,ef625cec,ef625d28,c0ef858a,0,...) at select+0x66 syscallenter(c77ed5c0,ef625ce4,ef625ce4,0,0,...) at syscallenter+0x263 syscall(ef625d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x284c4f33, esp = 0xbfbfe19c, ebp = 0xbfbfe1e8 --- Tracing command sshd pid 2612 tid 100127 td 0xc7cd88a0 sched_switch(c7cd88a0,0,104,191,431722e0,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,68,...) at mi_switch+0x1eb sleepq_switch(c7cd88a0,0,c0ef7c00,1a7,68,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7fa63f8,68,c0efe0a1,100,0,...) at sleepq_wait_sig+0x17 _sleep(c7fa63f8,c7fa63b8,168,c0efe0a1,0) at _sleep+0x35c sbwait(c7fa6394,4,c0efe179,5ec,c7fa63b8,...) at sbwait+0x76 soreceive_generic(c7fa6340,0,ef6fcc28,0,0,...) at soreceive_generic+0x3f0 soreceive(c7fa6340,0,ef6fcc28,0,0,0) at soreceive+0x38 soo_read(c7fee540,ef6fcc28,c71a0200,0,c7cd88a0,...) at soo_read+0x4e dofileread(ef6fcc28,ffffffff,ffffffff,0,c7fee540,...) at dofileread+0x9e kern_readv(c7cd88a0,5,ef6fcc28,ef6fcc48,1,...) at kern_readv+0x58 read(c7cd88a0,ef6fccec,ef6fcd28,c0ef858a,0,...) at read+0x4f syscallenter(c7cd88a0,ef6fcce4,ef6fcce4,0,0,...) at syscallenter+0x263 syscall(ef6fcd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x284c4fb3, esp = 0xbfbfdd3c, ebp = 0xbfbfdd78 --- Tracing command sshd pid 2610 tid 100088 td 0xc78da8a0 sched_switch(c78da8a0,0,104,191,42f8f78e,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,68,...) at mi_switch+0x1eb sleepq_switch(c78da8a0,0,c0ef7c00,1a7,68,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(ca73b0b8,68,c0efe0a1,100,0,...) at sleepq_wait_sig+0x17 _sleep(ca73b0b8,ca73b078,168,c0efe0a1,0) at _sleep+0x35c sbwait(ca73b054,4,c0efe179,5ec,ca73b078,...) at sbwait+0x76 soreceive_generic(ca73b000,0,ef629c28,0,0,...) at soreceive_generic+0x3f0 soreceive(ca73b000,0,ef629c28,0,0,0) at soreceive+0x38 soo_read(c791ecb0,ef629c28,c7f08c80,0,c78da8a0,...) at soo_read+0x4e dofileread(ef629c28,ffffffff,ffffffff,0,c791ecb0,...) at dofileread+0x9e kern_readv(c78da8a0,5,ef629c28,ef629c48,1,...) at kern_readv+0x58 read(c78da8a0,ef629cec,ef629d28,c0ef858a,0,...) at read+0x4f syscallenter(c78da8a0,ef629ce4,ef629ce4,0,0,...) at syscallenter+0x263 syscall(ef629d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x284c4fb3, esp = 0xbfbfdd3c, ebp = 0xbfbfdd78 --- Tracing command sshd pid 2609 tid 100122 td 0xc7cd98a0 sched_switch(c7cd98a0,0,104,191,40d665ef,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,68,...) at mi_switch+0x1eb sleepq_switch(c7cd98a0,0,c0ef7c00,1a7,68,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c829d0b8,68,c0efe0a1,100,0,...) at sleepq_wait_sig+0x17 _sleep(c829d0b8,c829d078,168,c0efe0a1,0) at _sleep+0x35c sbwait(c829d054,4,c0efe179,5ec,c829d078,...) at sbwait+0x76 soreceive_generic(c829d000,0,ef6e8c28,0,0,...) at soreceive_generic+0x3f0 soreceive(c829d000,0,ef6e8c28,0,0,0) at soreceive+0x38 soo_read(c791e4d0,ef6e8c28,c71a0900,0,c7cd98a0,...) at soo_read+0x4e dofileread(ef6e8c28,ffffffff,ffffffff,0,c791e4d0,...) at dofileread+0x9e kern_readv(c7cd98a0,5,ef6e8c28,ef6e8c48,1,...) at kern_readv+0x58 read(c7cd98a0,ef6e8cec,2888a010,1,0,...) at read+0x4f syscallenter(c7cd98a0,ef6e8ce4,c0d3601d,c10959f0,0,...) at syscallenter+0x263 syscall(ef6e8d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x284c4fb3, esp = 0xbfbfdd3c, ebp = 0xbfbfdd78 --- Tracing command md5 pid 2603 tid 100172 td 0xc7fa22e0 cpustop_handler(2,ef7c2a0c,c0d358d6,f5,0,...) at cpustop_handler+0x49 ipi_nmi_handler(f5,0,1b89a4e,c0a03ca5,c99c82c0,...) at ipi_nmi_handler+0x44 trap(ef7c2a18) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0a03ca5, esp = 0xef7c2a58, ebp = 0xef7c2a7c --- panic(c0f281ee,2d00,c7b9a87c,0,0,...) at panic+0x35 ffs_bufwrite(e0c4a9c0,c9955700,100,4000,0,...) at ffs_bufwrite+0x3e ffs_update(c9a4c414,1,ef7c2c6c,2000,c7872d00,...) at ffs_update+0x28c ffs_write(ef7c2c8c,c0f4712a,c9a4c414,0,c9a4c414,...) at ffs_write+0x66a VOP_WRITE_APV(c103fba0,ef7c2c8c,c0e98364,26a,c0ee8060,...) at VOP_WRITE_APV+0x136 mdstart_vnode(c7853800,c77f5260,c0e98364,2fb,0,...) at mdstart_vnode+0x4da md_kthread(c7853800,ef7c2d28,c0eecdb0,390,c99c82c0,...) at md_kthread+0x1c0 fork_exit(c0730b50,c7853800,ef7c2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef7c2d60, ebp = 0 --- Tracing command sh pid 2580 tid 100158 td 0xc83d72e0 sched_switch(c83d72e0,0,104,191,a4833cac,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c83d72e0,0,c0ef7c00,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c8345b00,6c,c0efa9de,100,0,...) at sleepq_wait_sig+0x17 _sleep(c8345b00,c8345b88,16c,c0efa9de,0,...) at _sleep+0x35c kern_wait(c83d72e0,ffffffff,ef77ec44,2,0,...) at kern_wait+0xc06 wait4(c83d72e0,ef77ecec,ef77ed28,c0ef858a,0,...) at wait4+0x3b syscallenter(c83d72e0,ef77ece4,ef77ece4,0,0,...) at syscallenter+0x263 syscall(ef77ed28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2816b29b, esp = 0xbfbfe73c, ebp = 0xbfbfe758 --- Tracing command bash pid 2574 tid 100092 td 0xc77ed000 sched_switch(c77ed000,0,104,191,72c73682,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c77ed000,0,c0ef7c00,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7bb7000,6c,c0efa9de,100,0,...) at sleepq_wait_sig+0x17 _sleep(c7bb7000,c7bb7088,16c,c0efa9de,0,...) at _sleep+0x35c kern_wait(c77ed000,ffffffff,ef652c44,6,0,...) at kern_wait+0xc06 wait4(c77ed000,ef652cec,ef652d28,c0ef858a,0,...) at wait4+0x3b syscallenter(c77ed000,ef652ce4,ef652ce4,0,0,...) at syscallenter+0x263 syscall(ef652d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x282c429b, esp = 0xbfbfe78c, ebp = 0xbfbfe7a8 --- Tracing command csh pid 2570 tid 100159 td 0xc83d7000 sched_switch(c83d7000,0,104,191,6424822f,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,74,...) at mi_switch+0x1eb sleepq_switch(c83d7000,0,c0ef7c00,1a7,74,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c8345898,74,c0d845ae,100,0,...) at sleepq_wait_sig+0x17 _sleep(c8345898,c83458c8,174,c0d845ae,0,...) at _sleep+0x35c kern_sigsuspend(c83d7000,0,0,0,0,...) at kern_sigsuspend+0xba sigsuspend(c83d7000,ef781cec,ef781d28,c0ef858a,0,...) at sigsuspend+0x4d syscallenter(c83d7000,ef781ce4,ef781ce4,0,0,...) at syscallenter+0x263 syscall(ef781d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2818b5eb, esp = 0xbfbfe5ac, ebp = 0xbfbfe618 --- Tracing command su pid 2569 tid 100116 td 0xc789f000 sched_switch(c789f000,0,104,191,f235d57,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c789f000,0,c0ef7c00,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7899000,6c,c0efa9de,100,0,...) at sleepq_wait_sig+0x17 _sleep(c7899000,c7899088,16c,c0efa9de,0,...) at _sleep+0x35c kern_wait(c789f000,a0a,ef6c8c44,2,0,...) at kern_wait+0xc06 wait4(c789f000,ef6c8cec,281c8b0c,1,0,...) at wait4+0x3b syscallenter(c789f000,ef6c8ce4,c0d3601d,c10959f0,0,...) at syscallenter+0x263 syscall(ef6c8d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2812429b, esp = 0xbfbfe5dc, ebp = 0xbfbfe5f8 --- Tracing command bash pid 2564 tid 100148 td 0xc7f4f8a0 sched_switch(c7f4f8a0,0,104,191,356fab44,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c7f4f8a0,0,c0ef7c00,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c8347b00,6c,c0efa9de,100,0,...) at sleepq_wait_sig+0x17 _sleep(c8347b00,c8347b88,16c,c0efa9de,0,...) at _sleep+0x35c kern_wait(c7f4f8a0,ffffffff,ef75fc44,6,0,...) at kern_wait+0xc06 wait4(c7f4f8a0,ef75fcec,ef75fd28,c0ef858a,0,...) at wait4+0x3b syscallenter(c7f4f8a0,ef75fce4,ef75fce4,0,0,...) at syscallenter+0x263 syscall(ef75fd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x282c429b, esp = 0xbfbfe97c, ebp = 0xbfbfe998 --- Tracing command sshd pid 2563 tid 100094 td 0xc77ec8a0 sched_switch(c77ec8a0,0,104,191,e5a6d31e,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c77ec8a0,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c09f45ba,c7810c10,0,c0ef0a47,c77ec8a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7810c24,0,ef658a4c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c7810c24,c7810c10,c0efa251,627,c997e268,...) at _cv_wait_sig+0x243 seltdwait(c997e268,58,c7872c00,c77ec8a0,246,...) at seltdwait+0xa2 kern_select(c77ec8a0,c,288060b4,288060bc,0,0,20,c7bb6840,c7bb6840) at kern_select+0x504 select(c77ec8a0,ef658cec,ef658d28,c0ef858a,0,...) at select+0x66 syscallenter(c77ec8a0,ef658ce4,ef658ce4,0,0,...) at syscallenter+0x263 syscall(ef658d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x284c4f33, esp = 0xbfbfe19c, ebp = 0xbfbfe1e8 --- Tracing command sshd pid 2561 tid 100144 td 0xc7fa52e0 sched_switch(c7fa52e0,0,104,191,accb5b79,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,68,...) at mi_switch+0x1eb sleepq_switch(c7fa52e0,0,c0ef7c00,1a7,68,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c801a0b8,68,c0efe0a1,100,0,...) at sleepq_wait_sig+0x17 _sleep(c801a0b8,c801a078,168,c0efe0a1,0) at _sleep+0x35c sbwait(c801a054,4,c0efe179,5ec,c801a078,...) at sbwait+0x76 soreceive_generic(c801a000,0,ef731c28,0,0,...) at soreceive_generic+0x3f0 soreceive(c801a000,0,ef731c28,0,0,0) at soreceive+0x38 soo_read(c997e038,ef731c28,c7f08700,0,c7fa52e0,...) at soo_read+0x4e dofileread(ef731c28,ffffffff,ffffffff,0,c997e038,...) at dofileread+0x9e kern_readv(c7fa52e0,5,ef731c28,ef731c48,1,...) at kern_readv+0x58 read(c7fa52e0,ef731cec,ef731d28,c0ef858a,0,...) at read+0x4f syscallenter(c7fa52e0,ef731ce4,ef731ce4,0,0,...) at syscallenter+0x263 syscall(ef731d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x284c4fb3, esp = 0xbfbfdd3c, ebp = 0xbfbfdd78 --- Tracing command bash pid 2555 tid 100130 td 0xc7bee000 sched_switch(c7bee000,0,104,191,99d0013e,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7bee000,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c09f45ba,c7cae804,0,c0ef0a47,c7bee000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7cae870,0,ef707ad8,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c7cae870,c7cae804,c0efc2bc,514,0,...) at _cv_wait_sig+0x243 tty_wait(c7cae800,c7cae870,ef707c28,1,0,...) at tty_wait+0x71 ttydisc_read(c7cae800,ef707c28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c7867500,ef707c28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c7fee0e0,ef707c28,c78cd500,0,c7bee000,...) at devfs_read_f+0x7e dofileread(ef707c28,ffffffff,ffffffff,0,c7fee0e0,...) at dofileread+0x9e kern_readv(c7bee000,0,ef707c28,ef707c48,1,...) at kern_readv+0x58 read(c7bee000,ef707cec,ef707d80,ef707c88,0,...) at read+0x4f syscallenter(c7bee000,ef707ce4,c0d35cd6,0,c10a4600,...) at syscallenter+0x263 syscall(ef707d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28354fb3, esp = 0xbfbfdfcc, ebp = 0xbfbfdff8 --- Tracing command sshd pid 2554 tid 100109 td 0xc7beeb80 sched_switch(c7beeb80,0,104,191,99cfc3de,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7beeb80,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c09f45ba,c7810a50,0,c0ef0a47,c7beeb80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7810a64,0,ef69da4c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c7810a64,c7810a50,c0efa251,627,c7fee888,...) at _cv_wait_sig+0x243 seltdwait(c7fee888,58,c7f5a900,c7beeb80,246,...) at seltdwait+0xa2 kern_select(c7beeb80,c,288060b4,288060bc,0,0,20,c7bb7b00,c7bb7b00) at kern_select+0x504 select(c7beeb80,ef69dcec,ef69dd28,c0ef858a,0,...) at select+0x66 syscallenter(c7beeb80,ef69dce4,ef69dce4,0,0,...) at syscallenter+0x263 syscall(ef69dd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x284c4f33, esp = 0xbfbfe19c, ebp = 0xbfbfe1e8 --- Tracing command sshd pid 2552 tid 100137 td 0xc78dc8a0 sched_switch(c78dc8a0,0,104,191,9268ff22,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,68,...) at mi_switch+0x1eb sleepq_switch(c78dc8a0,0,c0ef7c00,1a7,68,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c8285598,68,c0efe0a1,100,0,...) at sleepq_wait_sig+0x17 _sleep(c8285598,c8285558,168,c0efe0a1,0) at _sleep+0x35c sbwait(c8285534,4,c0efe179,5ec,c8285558,...) at sbwait+0x76 soreceive_generic(c82854e0,0,ef71cc28,0,0,...) at soreceive_generic+0x3f0 soreceive(c82854e0,0,ef71cc28,0,0,0) at soreceive+0x38 soo_read(c7feeb28,ef71cc28,c7872e00,0,c78dc8a0,...) at soo_read+0x4e dofileread(ef71cc28,ffffffff,ffffffff,0,c7feeb28,...) at dofileread+0x9e kern_readv(c78dc8a0,5,ef71cc28,ef71cc48,1,...) at kern_readv+0x58 read(c78dc8a0,ef71ccec,ef71cd28,c0ef858a,0,...) at read+0x4f syscallenter(c78dc8a0,ef71cce4,ef71cce4,0,0,...) at syscallenter+0x263 syscall(ef71cd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x284c4fb3, esp = 0xbfbfdd3c, ebp = 0xbfbfdd78 --- Tracing command getty pid 2364 tid 100167 td 0xc7fa42e0 sched_switch(c7fa42e0,0,104,191,806293fc,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7fa42e0,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,c7fa42e0,c7fa42e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c757e070,0,c0efc91e,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c757e070,c10959f0,c0efc2bc,514,0,...) at _cv_wait_sig+0x243 tty_wait(c757e000,c757e070,ef7b3c28,1,0,...) at tty_wait+0x71 ttydisc_read(c757e000,ef7b3c28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c73d9c00,ef7b3c28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c7fee2d8,ef7b3c28,c719fe80,0,c7fa42e0,...) at devfs_read_f+0x7e dofileread(ef7b3c28,ffffffff,ffffffff,0,c7fee2d8,...) at dofileread+0x9e kern_readv(c7fa42e0,0,ef7b3c28,ef7b3c48,1,...) at kern_readv+0x58 read(c7fa42e0,ef7b3cec,ef7b3d28,c0ef858a,0,...) at read+0x4f syscallenter(c7fa42e0,ef7b3ce4,ef7b3ce4,0,0,...) at syscallenter+0x263 syscall(ef7b3d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196fb3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 2363 tid 100166 td 0xc7fa45c0 sched_switch(c7fa45c0,0,104,191,80996788,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7fa45c0,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,c7fa45c0,c7fa45c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c757e270,0,c0efc91e,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c757e270,c10959f0,c0efc2bc,514,0,...) at _cv_wait_sig+0x243 tty_wait(c757e200,c757e270,ef7b0c28,1,0,...) at tty_wait+0x71 ttydisc_read(c757e200,ef7b0c28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c73d9d00,ef7b0c28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c7cb5d58,ef7b0c28,c719fe80,0,c7fa45c0,...) at devfs_read_f+0x7e dofileread(ef7b0c28,ffffffff,ffffffff,0,c7cb5d58,...) at dofileread+0x9e kern_readv(c7fa45c0,0,ef7b0c28,ef7b0c48,1,...) at kern_readv+0x58 read(c7fa45c0,ef7b0cec,ef7b0d28,c0ef858a,0,...) at read+0x4f syscallenter(c7fa45c0,ef7b0ce4,ef7b0ce4,0,0,...) at syscallenter+0x263 syscall(ef7b0d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196fb3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 2362 tid 100165 td 0xc7fa48a0 sched_switch(c7fa48a0,0,104,191,809a832d,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7fa48a0,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,c7fa48a0,c7fa48a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c757e470,0,c0efc91e,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c757e470,c10959f0,c0efc2bc,514,0,...) at _cv_wait_sig+0x243 tty_wait(c757e400,c757e470,ef7adc28,1,0,...) at tty_wait+0x71 ttydisc_read(c757e400,ef7adc28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c73d9e00,ef7adc28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c7feeb98,ef7adc28,c719fe80,0,c7fa48a0,...) at devfs_read_f+0x7e dofileread(ef7adc28,ffffffff,ffffffff,0,c7feeb98,...) at dofileread+0x9e kern_readv(c7fa48a0,0,ef7adc28,ef7adc48,1,...) at kern_readv+0x58 read(c7fa48a0,ef7adcec,ef7add28,c0ef858a,0,...) at read+0x4f syscallenter(c7fa48a0,ef7adce4,ef7adce4,0,0,...) at syscallenter+0x263 syscall(ef7add28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196fb3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 2361 tid 100164 td 0xc7fa4b80 sched_switch(c7fa4b80,0,104,191,80833541,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7fa4b80,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,c7fa4b80,c7fa4b80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c72e5070,0,c0efc91e,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c72e5070,c10959f0,c0efc2bc,514,0,...) at _cv_wait_sig+0x243 tty_wait(c72e5000,c72e5070,ef7aac28,1,0,...) at tty_wait+0x71 ttydisc_read(c72e5000,ef7aac28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c757d000,ef7aac28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c78c51f8,ef7aac28,c719fe80,0,c7fa4b80,...) at devfs_read_f+0x7e dofileread(ef7aac28,ffffffff,ffffffff,0,c78c51f8,...) at dofileread+0x9e kern_readv(c7fa4b80,0,ef7aac28,ef7aac48,1,...) at kern_readv+0x58 read(c7fa4b80,ef7aacec,ef7aad28,c0ef858a,0,...) at read+0x4f syscallenter(c7fa4b80,ef7aace4,ef7aace4,0,0,...) at syscallenter+0x263 syscall(ef7aad28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196fb3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 2360 tid 100163 td 0xc7fa5000 sched_switch(c7fa5000,0,104,191,80baf8c9,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7fa5000,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,c7fa5000,c7fa5000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c72e5270,0,c0efc91e,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c72e5270,c10959f0,c0efc2bc,514,0,...) at _cv_wait_sig+0x243 tty_wait(c72e5200,c72e5270,ef7a7c28,1,0,...) at tty_wait+0x71 ttydisc_read(c72e5200,ef7a7c28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c757d100,ef7a7c28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c7d44770,ef7a7c28,c719fe80,0,c7fa5000,...) at devfs_read_f+0x7e dofileread(ef7a7c28,ffffffff,ffffffff,0,c7d44770,...) at dofileread+0x9e kern_readv(c7fa5000,0,ef7a7c28,ef7a7c48,1,...) at kern_readv+0x58 read(c7fa5000,ef7a7cec,ef7a7d28,c0ef858a,0,...) at read+0x4f syscallenter(c7fa5000,ef7a7ce4,ef7a7ce4,0,0,...) at syscallenter+0x263 syscall(ef7a7d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196fb3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 2359 tid 100136 td 0xc78dcb80 sched_switch(c78dcb80,0,104,191,802c0d58,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c78dcb80,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,c78dcb80,c78dcb80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c72e5470,0,c0efc91e,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c72e5470,c10959f0,c0efc2bc,514,0,...) at _cv_wait_sig+0x243 tty_wait(c72e5400,c72e5470,ef719c28,1,0,...) at tty_wait+0x71 ttydisc_read(c72e5400,ef719c28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c757d200,ef719c28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c78c5968,ef719c28,c719fe80,0,c78dcb80,...) at devfs_read_f+0x7e dofileread(ef719c28,ffffffff,ffffffff,0,c78c5968,...) at dofileread+0x9e kern_readv(c78dcb80,0,ef719c28,ef719c48,1,...) at kern_readv+0x58 read(c78dcb80,ef719cec,ef719d28,c0ef858a,0,...) at read+0x4f syscallenter(c78dcb80,ef719ce4,ef719ce4,0,0,...) at syscallenter+0x263 syscall(ef719d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196fb3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 2358 tid 100102 td 0xc78d5b80 sched_switch(c78d5b80,0,104,191,80c044bd,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c78d5b80,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,c78d5b80,c78d5b80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c72e5670,0,c0efc91e,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c72e5670,c10959f0,c0efc2bc,514,0,...) at _cv_wait_sig+0x243 tty_wait(c72e5600,c72e5670,ef680c28,1,0,...) at tty_wait+0x71 ttydisc_read(c72e5600,ef680c28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c757d300,ef680c28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c78aaa10,ef680c28,c719fe80,0,c78d5b80,...) at devfs_read_f+0x7e dofileread(ef680c28,ffffffff,ffffffff,0,c78aaa10,...) at dofileread+0x9e kern_readv(c78d5b80,0,ef680c28,ef680c48,1,...) at kern_readv+0x58 read(c78d5b80,ef680cec,ef680d28,c0ef858a,0,...) at read+0x4f syscallenter(c78d5b80,ef680ce4,ef680ce4,0,0,...) at syscallenter+0x263 syscall(ef680d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196fb3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 2357 tid 100160 td 0xc83d5b80 sched_switch(c83d5b80,0,104,191,8043f5ab,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c83d5b80,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,c83d5b80,c83d5b80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c72e5870,0,c0efc91e,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c72e5870,c10959f0,c0efc2bc,514,0,...) at _cv_wait_sig+0x243 tty_wait(c72e5800,c72e5870,ef784c28,1,0,...) at tty_wait+0x71 ttydisc_read(c72e5800,ef784c28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c734c200,ef784c28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c78c5118,ef784c28,c719fe80,0,c83d5b80,...) at devfs_read_f+0x7e dofileread(ef784c28,ffffffff,ffffffff,0,c78c5118,...) at dofileread+0x9e kern_readv(c83d5b80,0,ef784c28,ef784c48,1,...) at kern_readv+0x58 read(c83d5b80,ef784cec,ef784d28,c0ef858a,0,...) at read+0x4f syscallenter(c83d5b80,ef784ce4,ef784ce4,0,0,...) at syscallenter+0x263 syscall(ef784d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196fb3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 2356 tid 100106 td 0xc78d5000 sched_switch(c78d5000,0,104,191,fbe1c94,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c78d5000,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c09f45ba,c72e5a04,0,c0ef0a47,c78d5000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c72e5a70,0,ef691ad8,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c72e5a70,c72e5a04,c0efc2bc,514,0,...) at _cv_wait_sig+0x243 tty_wait(c72e5a00,c72e5a70,ef691c28,1,0,...) at tty_wait+0x71 ttydisc_read(c72e5a00,ef691c28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c72ccd00,ef691c28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c791e690,ef691c28,c719fe80,0,c78d5000,...) at devfs_read_f+0x7e dofileread(ef691c28,ffffffff,ffffffff,0,c791e690,...) at dofileread+0x9e kern_readv(c78d5000,0,ef691c28,ef691c48,1,...) at kern_readv+0x58 read(c78d5000,ef691cec,ef691d28,c0ef858a,0,...) at read+0x4f syscallenter(c78d5000,ef691ce4,ef691ce4,0,0,...) at syscallenter+0x263 syscall(ef691d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196fb3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command inetd pid 2317 tid 100119 td 0xc789e5c0 sched_switch(c789e5c0,0,104,191,6254acec,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c789e5c0,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c09f45ba,c7810b10,0,c0ef0a47,c789e5c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7810b24,0,ef6d4a4c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c7810b24,c7810b10,c0efa251,627,c997e348,...) at _cv_wait_sig+0x243 seltdwait(c997e348,58,c719fe80,c789e5c0,c789e670,...) at seltdwait+0xa2 kern_select(c789e5c0,6,bfbfdfa0,0,0,0,20,c77e4580,c77e4580) at kern_select+0x504 select(c789e5c0,ef6d4cec,ef6d4d28,c0ef858a,0,...) at select+0x66 syscallenter(c789e5c0,ef6d4ce4,ef6d4ce4,0,0,...) at syscallenter+0x263 syscall(ef6d4d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x281a8f33, esp = 0xbfbfdf5c, ebp = 0xbfbfede8 --- Tracing command watchdogd pid 2285 tid 100157 td 0xc83d75c0 sched_switch(c83d75c0,0,104,191,1da0c233,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c83d75c0,0,c0ef7c00,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(ea61,c0a437f0,c83d75c0,0,100,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c1097304,6c,c0ef3997,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c1097304,0,16c,c0ef3997,ea61,...) at _sleep+0x328 kern_nanosleep(c83d75c0,ef77ac34,ef77ac3c,3c,0,...) at kern_nanosleep+0xc1 nanosleep(c83d75c0,ef77acec,ef77ad28,c0ef858a,0,...) at nanosleep+0x6f syscallenter(c83d75c0,ef77ace4,ef77ace4,0,0,...) at syscallenter+0x263 syscall(ef77ad28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x281924f7, esp = 0xbfbfecbc, ebp = 0xbfbfece8 --- Tracing command cron pid 2173 tid 100141 td 0xc7f50b80 sched_switch(c7f50b80,0,104,191,27a0c5ec,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c7f50b80,0,c0ef7c00,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(ea61,c0a437f0,c7f50b80,0,100,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c1097304,6c,c0ef3997,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c1097304,0,16c,c0ef3997,ea61,...) at _sleep+0x328 kern_nanosleep(c7f50b80,ef728c34,ef728c3c,3c,0,...) at kern_nanosleep+0xc1 nanosleep(c7f50b80,ef728cec,ef728d28,c0ef858a,0,...) at nanosleep+0x6f syscallenter(c7f50b80,ef728ce4,ef728ce4,0,0,...) at syscallenter+0x263 syscall(ef728d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x281854f7, esp = 0xbfbfec4c, ebp = 0xbfbfec78 --- Tracing command sendmail pid 2165 tid 100126 td 0xc7cd8b80 sched_switch(c7cd8b80,0,104,191,2d8928e8,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,74,...) at mi_switch+0x1eb sleepq_switch(c7cd8b80,0,c0ef7c00,1a7,74,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7cd2058,74,c0d845ae,100,0,...) at sleepq_wait_sig+0x17 _sleep(c7cd2058,c7cd2088,174,c0d845ae,0,...) at _sleep+0x35c kern_sigsuspend(c7cd8b80,0,0,0,0,...) at kern_sigsuspend+0xba sigsuspend(c7cd8b80,ef6f8cec,ef6f8d28,c0ef858a,0,...) at sigsuspend+0x4d syscallenter(c7cd8b80,ef6f8ce4,ef6f8ce4,0,0,...) at syscallenter+0x263 syscall(ef6f8d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2833c5eb, esp = 0xbfbfcf9c, ebp = 0xbfbfcfc8 --- Tracing command sendmail pid 2159 tid 100143 td 0xc7f505c0 sched_switch(c7f505c0,0,104,191,dfb3eaed,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7f505c0,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(ef72ea14,c09f45ba,c78108d0,0,c7f505c0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c78108e4,0,ef72ea4c,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c78108e4,c78108d0,1389,627,c78aa8f8,...) at _cv_timedwait_sig+0x252 seltdwait(ef72ebf8,ef72ec00,c7f5a100,c7f505c0,ef72eaf8,...) at seltdwait+0x8a kern_select(c7f505c0,5,bfbfc510,0,0,ef72ec40,20,5,0) at kern_select+0x504 select(c7f505c0,ef72ecec,ef72ed28,c0ef858a,0,...) at select+0x66 syscallenter(c7f505c0,ef72ece4,ef72ece4,0,0,...) at syscallenter+0x263 syscall(ef72ed28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x283e3f33, esp = 0xbfbfc47c, ebp = 0xbfbfcfa8 --- Tracing command sshd pid 2151 tid 100142 td 0xc7f508a0 sched_switch(c7f508a0,0,104,191,427edcac,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7f508a0,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c09f45ba,c77c54d0,0,c0ef0a47,c7f508a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c77c54e4,0,ef72ba4c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c77c54e4,c77c54d0,c0efa251,627,c791e8c0,...) at _cv_wait_sig+0x243 seltdwait(c791e8c0,58,c7872600,c7f508a0,ef72bab4,...) at seltdwait+0xa2 kern_select(c7f508a0,5,288cc0b4,0,0,0,20,c7cd3580,c7cd3580) at kern_select+0x504 select(c7f508a0,ef72bcec,ef72bd28,c0ef858a,0,...) at select+0x66 syscallenter(c7f508a0,ef72bce4,ef72bce4,0,0,...) at syscallenter+0x263 syscall(ef72bd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x284c4f33, esp = 0xbfbfe25c, ebp = 0xbfbfedf8 --- Tracing command moused pid 2091 tid 100139 td 0xc78dc2e0 sched_switch(c78dc2e0,0,104,191,1d871306,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c78dc2e0,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c09f45ba,c77c56d0,0,c0ef0a47,c78dc2e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c77c56e4,0,ef722a4c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c77c56e4,c77c56d0,c0efa251,627,c78c5c40,...) at _cv_wait_sig+0x243 seltdwait(c78c5c40,58,c719fe80,c78dc2e0,14,...) at seltdwait+0xa2 kern_select(c78dc2e0,6,bfbfea04,0,0,0,20,c7f3c000,c7f3c000) at kern_select+0x504 select(c78dc2e0,ef722cec,ef722d28,c0ef858a,0,...) at select+0x66 syscallenter(c78dc2e0,ef722ce4,ef722ce4,0,0,...) at syscallenter+0x263 syscall(ef722d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x281b3f33, esp = 0xbfbfe99c, ebp = 0xbfbfeb28 --- Tracing command ntpd pid 2013 tid 100101 td 0xc78da000 sched_switch(c78da000,0,104,191,1a9b9d31,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c78da000,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c09f45ba,c7881390,0,c0ef0a47,c78da000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c78813a4,0,ef67ca4c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c78813a4,c7881390,c0efa251,627,c7d44b60,...) at _cv_wait_sig+0x243 seltdwait(c7d44b60,58,c719fe80,c78da000,0,...) at seltdwait+0xa2 kern_select(c78da000,1c,bfbfece8,0,0,0,20,c7be5580,c7be5580) at kern_select+0x504 select(c78da000,ef67ccec,ef67cd80,ef67cc88,0,...) at select+0x66 syscallenter(c78da000,ef67cce4,c0d35cd6,fffffffe,246,...) at syscallenter+0x263 syscall(ef67cd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x28362f33, esp = 0xbfbfecbc, ebp = 0xbfbfed88 --- Tracing command nfsd pid 1909 tid 100147 td 0xc7f4fb80 sched_switch(c7f4fb80,0,104,191,af5138d8,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7f4fb80,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(ef743be0,c09f45ba,c7467480,0,c7f4fb80,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c784ca10,0,ef743c18,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c784ca10,c7467480,1388,3b1,0,...) at _cv_timedwait_sig+0x252 svc_run_internal(ef743d14,c09d7878,c7467480,ef743d28,c0eecdb0,...) at svc_run_internal+0x356 svc_thread_start(c7467480,ef743d28,c0eecdb0,390,c78992c0,...) at svc_thread_start+0x10 fork_exit(c0beaff0,c7467480,ef743d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0x804c12e, eip = 0xc, esp = 0x33, ebp = 0x1 --- Tracing command nfsd pid 1909 tid 100146 td 0xc7f50000 sched_switch(c7f50000,0,104,191,af517725,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7f50000,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(ef740be0,c09f45ba,c7467480,0,c7f50000,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c784ca50,0,ef740c18,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c784ca50,c7467480,1388,3b1,0,...) at _cv_timedwait_sig+0x252 svc_run_internal(ef740d14,c09d7878,c7467480,ef740d28,c0eecdb0,...) at svc_run_internal+0x356 svc_thread_start(c7467480,ef740d28,c0eecdb0,390,c78992c0,...) at svc_thread_start+0x10 fork_exit(c0beaff0,c7467480,ef740d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0x804c12e, eip = 0xc, esp = 0x33, ebp = 0x1 --- Tracing command nfsd pid 1909 tid 100145 td 0xc7f502e0 sched_switch(c7f502e0,0,104,191,af5189b8,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7f502e0,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(ef73dbe0,c09f45ba,c7467480,0,c7f502e0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c784ca90,0,ef73dc18,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c784ca90,c7467480,1388,3b1,ec,...) at _cv_timedwait_sig+0x252 svc_run_internal(ef73dd14,c09d7878,c7467480,ef73dd28,c0eecdb0,...) at svc_run_internal+0x356 svc_thread_start(c7467480,ef73dd28,c0eecdb0,390,c78992c0,...) at svc_thread_start+0x10 fork_exit(c0beaff0,c7467480,ef73dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0x804c12e, eip = 0xc, esp = 0x33, ebp = 0x1 --- Tracing command nfsd pid 1909 tid 100115 td 0xc789f2e0 sched_switch(c789f2e0,0,104,191,af515b48,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c789f2e0,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(ef6c4494,c09f45ba,c7467480,0,c789f2e0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c784b8d0,0,ef6c44cc,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c784b8d0,c7467480,1388,3b1,ef6c4514,...) at _cv_timedwait_sig+0x252 svc_run_internal(c789f490,14,c0f20b1a,c0ee4a5b,ef6c47f4,...) at svc_run_internal+0x356 svc_run(c7467480,0,c0ee4a36,199,0,...) at svc_run+0x7b nfsrvd_nfsd(c789f2e0,ef6c47f4,c,c1214480,ef6c4720,...) at nfsrvd_nfsd+0x1c3 nfssvc_nfsd(c789f2e0,ef6c4cec,c78992c0,0,ef6c4c7c,...) at nfssvc_nfsd+0x167 nfssvc(c789f2e0,ef6c4cec,ef6c4d28,c0ef858a,0,...) at nfssvc+0xeb syscallenter(c789f2e0,ef6c4ce4,ef6c4ce4,0,0,...) at syscallenter+0x263 syscall(ef6c4d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x280de1cb, esp = 0xbfbfe82c, ebp = 0xbfbfea98 --- Tracing command nfsd pid 1908 tid 100084 td 0xc77ed8a0 sched_switch(c77ed8a0,0,104,191,344df6f4,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c77ed8a0,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c09f45ba,c78813d0,0,c0ef0a47,c77ed8a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c78813e4,0,ef619a4c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c78813e4,c78813d0,c0efa251,627,c78aae00,...) at _cv_wait_sig+0x243 seltdwait(c78aae00,58,c719fe80,c77ed8a0,f3b,...) at seltdwait+0xa2 kern_select(c77ed8a0,7,bfbfecb4,0,0,0,20,c77e2000,c77e2000) at kern_select+0x504 select(c77ed8a0,ef619cec,bfbffff4,1,0,...) at select+0x66 syscallenter(c77ed8a0,ef619ce4,c0d3601d,c10959f0,0,...) at syscallenter+0x263 syscall(ef619d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x28184f33, esp = 0xbfbfea9c, ebp = 0xbfbfede8 --- Tracing command mountd pid 1898 tid 100108 td 0xc7bef000 sched_switch(c7bef000,0,104,191,e95fa53b,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7bef000,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c09f45ba,c7810bd0,0,c0ef0a47,c7bef000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7810be4,0,ef698a4c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c7810be4,c7810bd0,c0efa251,627,c7d44508,...) at _cv_wait_sig+0x243 seltdwait(c7d44508,58,c719fe80,c7bef000,c11e0490,...) at seltdwait+0xa2 kern_select(c7bef000,9,bfbfecf4,0,0,0,20,c7be4000,c7be4000) at kern_select+0x504 select(c7bef000,ef698cec,ef698d28,c0ef858a,0,...) at select+0x66 syscallenter(c7bef000,ef698ce4,ef698ce4,0,0,...) at syscallenter+0x263 syscall(ef698d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x2819af33, esp = 0xbfbfec8c, ebp = 0xbfbfedd8 --- Tracing command rpcbind pid 1799 tid 100100 td 0xc75735c0 sched_switch(c75735c0,0,104,191,77aac00a,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c75735c0,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(ef66aa6c,c09f45ba,c7881410,0,c75735c0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c7881424,0,ef66aaa4,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c7881424,c7881410,7531,627,ef66ab5c,...) at _cv_timedwait_sig+0x252 seltdwait(ef66ac2c,ef66ac34,535,c75735c0,ef66ab2c,...) at seltdwait+0x8a poll(c75735c0,ef66acec,ef66ad28,c0ef858a,0,...) at poll+0x300 syscallenter(c75735c0,ef66ace4,ef66ace4,0,0,...) at syscallenter+0x263 syscall(ef66ad28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (209, FreeBSD ELF32, poll), eip = 0x2814010b, esp = 0xbfbfcbac, ebp = 0xbfbfed78 --- Tracing command syslogd pid 1716 tid 100111 td 0xc7bee5c0 sched_switch(c7bee5c0,0,104,191,2831e323,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7bee5c0,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c09f45ba,c7810e90,0,c0ef0a47,c7bee5c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7810ea4,0,ef6a4a4c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c7810ea4,c7810e90,c0efa251,627,c78abb60,...) at _cv_wait_sig+0x243 seltdwait(c78abb60,58,c719fe80,c7bee5c0,0,...) at seltdwait+0xa2 kern_select(c7bee5c0,9,2847f0ac,0,0,0,20,c7bb7580,c7bb7580) at kern_select+0x504 select(c7bee5c0,ef6a4cec,ef6a4d28,c0ef858a,0,...) at select+0x66 syscallenter(c7bee5c0,ef6a4ce4,ef6a4ce4,0,246,...) at syscallenter+0x263 syscall(ef6a4d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x28199f33, esp = 0xbfbfe29c, ebp = 0xbfbfede8 --- Tracing command devd pid 1516 tid 100083 td 0xc78db2e0 sched_switch(c78db2e0,0,104,191,402a8c20,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c78db2e0,0,c0ef7c00,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c09f45ba,c7810c90,0,c0ef0a47,c78db2e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7810ca4,0,ef615a4c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c7810ca4,c7810c90,c0efa251,627,c78abcb0,...) at _cv_wait_sig+0x243 seltdwait(c78abcb0,58,c719fe80,c78db2e0,c0f2a721,...) at seltdwait+0xa2 kern_select(c78db2e0,5,bfbfe960,0,0,0,20,c78d4000,c78d4000) at kern_select+0x504 select(c78db2e0,ef615cec,ef615d28,c0ef858a,0,...) at select+0x66 syscallenter(c78db2e0,ef615ce4,ef615ce4,0,0,...) at syscallenter+0x263 syscall(ef615d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x808b99b, esp = 0xbfbfe92c, ebp = 0xbfbfee08 --- Tracing command softdepflush pid 18 tid 100076 td 0xc744a8a0 sched_switch(c744a8a0,0,104,191,4f984450,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c744a8a0,0,c0ef7c00,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c7f215f0,5c,c0f24d6e,0,0,...) at sleepq_timedwait+0x6b _sleep(c7f215f0,c717c790,5c,c0f24d6e,64,...) at _sleep+0x342 biowait(c7f215f0,c0f24d6e,caf4f6cc,0,6766,...) at biowait+0x7d readblock(6766,0,0,8000,c71a0000,...) at readblock+0xa2 ffs_snapblkfree(c7b96000,c9a5f984,33b34,0,4000,...) at ffs_snapblkfree+0x66c ffs_blkfree(c7b93700,c7b96000,c9a5f984,33b34,0,...) at ffs_blkfree+0x5c handle_workitem_freefrag(0,ed40ec6c,2,6b8,ed40ec70,...) at handle_workitem_freefrag+0xf0 process_worklist_item(c1213b30,0,c0f25328,5e7,c744a8a0,...) at process_worklist_item+0x2cd softdep_process_worklist(c7b9a000,0,c0f25328,55f,3e8,...) at softdep_process_worklist+0x88 softdep_flush(0,ed40ed28,c0eecdb0,390,c77e22c0,...) at softdep_flush+0x2a0 fork_exit(c0c3a290,0,ed40ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed40ed60, ebp = 0 --- Tracing command vnlru pid 17 tid 100075 td 0xc744ab80 sched_switch(c744ab80,0,104,191,51c04a3a,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,60,...) at mi_switch+0x1eb sleepq_switch(c744ab80,0,c0ef7c00,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c77e2580,60,c0f028c4,0,0,...) at sleepq_timedwait+0x6b _sleep(c77e2580,c120d4b4,260,c0f028c4,3e8,...) at _sleep+0x342 vnlru_proc(0,ed40bd28,c0eecdb0,390,c77e2580,...) at vnlru_proc+0xe7 fork_exit(c0aa1000,0,ed40bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed40bd60, ebp = 0 --- Tracing command syncer pid 16 tid 100074 td 0xc7571000 sched_switch(c7571000,0,104,191,50e8d133,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7571000,0,c0ef7c00,28b,c7571000,...) at sleepq_switch+0x162 sleepq_timedwait(c120d4f4,0,ed408c78,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c120d4f4,c120d4e0,3e8,76a,4e20,...) at _cv_timedwait+0x252 sched_sync(0,ed408d28,c0eecdb0,390,c77e2840,...) at sched_sync+0x595 fork_exit(c0a9efa0,0,ed408d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed408d60, ebp = 0 --- Tracing command bufdaemon pid 9 tid 100073 td 0xc75712e0 sched_switch(c75712e0,0,104,191,4f923750,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,60,...) at mi_switch+0x1eb sleepq_switch(c75712e0,0,c0ef7c00,268,60,...) at sleepq_switch+0x162 sleepq_wait(c82c891c,60,c0f24dcd,4,0,...) at sleepq_wait+0x63 __lockmgr_args(c82c891c,80900,c9a5fa48,0,0,...) at __lockmgr_args+0xb1f ffs_copyonwrite(c9a5f984,e0cabe80,0,e0cabe80,ca40bae0,...) at ffs_copyonwrite+0x2c2 ffs_geom_strategy(c9a5fa7c,e0cabe80,ed405b5c,3,c75712e0,...) at ffs_geom_strategy+0xeb ufs_strategy(ed405b98,c0f46eef,c75712e0,e0cabe80,4,...) at ufs_strategy+0xd3 VOP_STRATEGY_APV(c103fba0,ed405b98,c0efefab,c105ac20,ca40bae0,...) at VOP_STRATEGY_APV+0xe5 bufstrategy(ca40bbd8,e0cabe80,3a6,7000,0,...) at bufstrategy+0x61 bufwrite(e0cabe80,0,c0efefab,6dc,c0f01b9f,...) at bufwrite+0x159 vfs_bio_awrite(e0cabe80,0,c0efefab,945,0,...) at vfs_bio_awrite+0x318 flushbufqueues(d0,c0ef0a47,c77e2b88,391,391,...) at flushbufqueues+0x5b0 buf_do_flush(c77e2b00,0,c0efefab,8ac,3e8,...) at buf_do_flush+0x19 buf_daemon(0,ed405d28,c0eecdb0,390,c77e2b00,...) at buf_daemon+0xea fork_exit(c0a86710,0,ed405d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed405d60, ebp = 0 --- Tracing command pagezero pid 8 tid 100072 td 0xc75715c0 sched_switch(c75715c0,0,104,191,2f7da6b0,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c75715c0,0,c0ef7c00,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c12159dc,0,c0f2e1f0,0,0,...) at sleepq_timedwait+0x6b _sleep(c12159dc,c1214480,0,c0f2e1f0,493e0,...) at _sleep+0x342 vm_pagezero(0,ed402d28,c0eecdb0,390,c77e4000,...) at vm_pagezero+0xdc fork_exit(c0c7e0b0,0,ed402d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed402d60, ebp = 0 --- Tracing command vmdaemon pid 7 tid 100071 td 0xc75718a0 sched_switch(c75718a0,0,104,191,75b1d4a4,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,74,...) at mi_switch+0x1eb sleepq_switch(c75718a0,0,c0ef7c00,268,0,...) at sleepq_switch+0x162 sleepq_wait(c12155c8,74,c0eff8f2,0,0,...) at sleepq_wait+0x63 _sleep(c12155c8,c12155cc,74,c0eff8f2,0,...) at _sleep+0x372 vm_daemon(0,ed3ffd28,c0eecdb0,390,c71bc2c0,...) at vm_daemon+0x59 fork_exit(c0c77c60,0,ed3ffd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed3ffd60, ebp = 0 --- Tracing command pagedaemon pid 6 tid 100070 td 0xc7571b80 sched_switch(c7571b80,0,104,191,8af1dad7,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,54,...) at mi_switch+0x1eb sleepq_switch(c7571b80,0,c0ef7c00,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c1215590,54,c0eff8f2,0,0,...) at sleepq_timedwait+0x6b _sleep(c1215590,c1214480,54,c0eff8f2,1388,...) at _sleep+0x342 vm_pageout(0,ed3fcd28,c0eecdb0,390,c71bc580,...) at vm_pageout+0x2c7 fork_exit(c0c78c90,0,ed3fcd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed3fcd60, ebp = 0 --- Tracing command xpt_thrd pid 5 tid 100069 td 0xc7572000 sched_switch(c7572000,0,104,191,7be60c35,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c7572000,0,c0ef7c00,268,0,...) at sleepq_switch+0x162 sleepq_wait(c105c054,5c,c0d5d232,0,0,...) at sleepq_wait+0x63 _sleep(c105c054,c105c098,5c,c0d5d232,0,...) at _sleep+0x372 xpt_scanner_thread(0,ed399d28,c0eecdb0,390,c71bc840,...) at xpt_scanner_thread+0x47 fork_exit(c0490d00,0,ed399d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed399d60, ebp = 0 --- Tracing command sctp_iterator pid 4 tid 100068 td 0xc75722e0 sched_switch(c75722e0,0,104,191,bb4c3f9b,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c75722e0,0,c0ef7c00,268,0,...) at sleepq_switch+0x162 sleepq_wait(c120f018,0,c0f15b5f,0,0,...) at sleepq_wait+0x63 _sleep(c120f018,c120efe0,0,c0f15b5f,0,...) at _sleep+0x372 sctp_iterator_thread(0,ed396d28,c0eecdb0,390,c71bcb00,...) at sctp_iterator_thread+0x5c fork_exit(c0b30960,0,ed396d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed396d60, ebp = 0 --- Tracing command fdc0 pid 3 tid 100064 td 0xc7573000 sched_switch(c7573000,0,104,191,443a8682,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c7573000,0,c0ef7c00,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c755f43c,5c,c0ee5532,0,0,...) at sleepq_timedwait+0x6b _sleep(c755f43c,c755f4f0,5c,c0ee5532,3e8,...) at _sleep+0x342 fdc_thread(c755f400,ed38ad28,c0eecdb0,390,c7429000,...) at fdc_thread+0x27d fork_exit(c0cfd330,c755f400,ed38ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed38ad60, ebp = 0 --- Tracing command fw0_probe pid 2 tid 100062 td 0xc743a5c0 sched_switch(c743a5c0,0,104,191,7be66cc9,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c743a5c0,0,c0ef7c00,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7479000,6c,c0ee5532,100,0,...) at sleepq_wait_sig+0x17 _sleep(c7479000,c747d488,16c,c0ee5532,0,...) at _sleep+0x35c fw_bus_probe_thread(c7479000,ed37ad28,c0eecdb0,390,c74292c0,...) at fw_bus_probe_thread+0xa08 fork_exit(c06b3ed0,c7479000,ed37ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed37ad60, ebp = 0 --- Tracing command usb pid 15 tid 100058 td 0xc74472e0 sched_switch(c74472e0,0,104,191,694ce4fb,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74472e0,0,c0ef7c00,268,c74472e0,...) at sleepq_switch+0x162 sleepq_wait(c7450d34,0,ed363cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7450d34,c7450dd4,c0ecc8fe,6b,c7450d3c,...) at _cv_wait+0x243 usb_process(c7450d2c,ed363d28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c7450d2c,ed363d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed363d60, ebp = 0 --- Tracing command usb pid 15 tid 100057 td 0xc74475c0 sched_switch(c74475c0,0,104,191,fe7e7b00,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74475c0,0,c0ef7c00,268,c74475c0,...) at sleepq_switch+0x162 sleepq_wait(c7450d04,0,ed360cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7450d04,c7450dd4,c0ecc8fe,6b,c7450d0c,...) at _cv_wait+0x243 usb_process(c7450cfc,ed360d28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c7450cfc,ed360d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed360d60, ebp = 0 --- Tracing command usb pid 15 tid 100056 td 0xc74478a0 sched_switch(c74478a0,0,104,191,68e64cbf,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74478a0,0,c0ef7c00,268,c74478a0,...) at sleepq_switch+0x162 sleepq_wait(c7450cd4,0,ed35dcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7450cd4,c7450dd4,c0ecc8fe,6b,c7450cdc,...) at _cv_wait+0x243 usb_process(c7450ccc,ed35dd28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c7450ccc,ed35dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed35dd60, ebp = 0 --- Tracing command usb pid 15 tid 100055 td 0xc7447b80 sched_switch(c7447b80,0,104,191,68e63b89,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7447b80,0,c0ef7c00,268,c7447b80,...) at sleepq_switch+0x162 sleepq_wait(c7450ca4,0,ed35acac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7450ca4,c7450dd4,c0ecc8fe,6b,c7450cac,...) at _cv_wait+0x243 usb_process(c7450c9c,ed35ad28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c7450c9c,ed35ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed35ad60, ebp = 0 --- Tracing command usb pid 15 tid 100053 td 0xc744a2e0 sched_switch(c744a2e0,0,104,191,68e627d2,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c744a2e0,0,c0ef7c00,268,c744a2e0,...) at sleepq_switch+0x162 sleepq_wait(c7441b5c,0,ed351cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7441b5c,c7441bfc,c0ecc8fe,6b,c7441b64,...) at _cv_wait+0x243 usb_process(c7441b54,ed351d28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c7441b54,ed351d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed351d60, ebp = 0 --- Tracing command usb pid 15 tid 100052 td 0xc744a5c0 sched_switch(c744a5c0,0,104,191,f6041b4e,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c744a5c0,0,c0ef7c00,268,c744a5c0,...) at sleepq_switch+0x162 sleepq_wait(c7441b2c,0,ed34ecac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7441b2c,c7441bfc,c0ecc8fe,6b,c7441b34,...) at _cv_wait+0x243 usb_process(c7441b24,ed34ed28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c7441b24,ed34ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed34ed60, ebp = 0 --- Tracing command usb pid 15 tid 100051 td 0xc7438000 sched_switch(c7438000,0,104,191,6882117d,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7438000,0,c0ef7c00,268,c7438000,...) at sleepq_switch+0x162 sleepq_wait(c7441afc,0,ed34bcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7441afc,c7441bfc,c0ecc8fe,6b,c7441b04,...) at _cv_wait+0x243 usb_process(c7441af4,ed34bd28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c7441af4,ed34bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed34bd60, ebp = 0 --- Tracing command usb pid 15 tid 100050 td 0xc74382e0 sched_switch(c74382e0,0,104,191,688200f0,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74382e0,0,c0ef7c00,268,c74382e0,...) at sleepq_switch+0x162 sleepq_wait(c7441acc,0,ed348cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7441acc,c7441bfc,c0ecc8fe,6b,c7441ad4,...) at _cv_wait+0x243 usb_process(c7441ac4,ed348d28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c7441ac4,ed348d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed348d60, ebp = 0 --- Tracing command usb pid 15 tid 100049 td 0xc74385c0 sched_switch(c74385c0,0,104,191,6881ed76,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74385c0,0,c0ef7c00,268,c74385c0,...) at sleepq_switch+0x162 sleepq_wait(c743db5c,0,ed345cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c743db5c,c743dbfc,c0ecc8fe,6b,c743db64,...) at _cv_wait+0x243 usb_process(c743db54,ed345d28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c743db54,ed345d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed345d60, ebp = 0 --- Tracing command usb pid 15 tid 100048 td 0xc74388a0 sched_switch(c74388a0,0,104,191,f2e7c506,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74388a0,0,c0ef7c00,268,c74388a0,...) at sleepq_switch+0x162 sleepq_wait(c743db2c,0,ed342cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c743db2c,c743dbfc,c0ecc8fe,6b,c743db34,...) at _cv_wait+0x243 usb_process(c743db24,ed342d28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c743db24,ed342d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed342d60, ebp = 0 --- Tracing command usb pid 15 tid 100047 td 0xc7438b80 sched_switch(c7438b80,0,104,191,62a91b82,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7438b80,0,c0ef7c00,268,c7438b80,...) at sleepq_switch+0x162 sleepq_wait(c743dafc,0,ed33fcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c743dafc,c743dbfc,c0ecc8fe,6b,c743db04,...) at _cv_wait+0x243 usb_process(c743daf4,ed33fd28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c743daf4,ed33fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed33fd60, ebp = 0 --- Tracing command usb pid 15 tid 100046 td 0xc7439000 sched_switch(c7439000,0,104,191,62a90acc,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7439000,0,c0ef7c00,268,c7439000,...) at sleepq_switch+0x162 sleepq_wait(c743dacc,0,ed33ccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c743dacc,c743dbfc,c0ecc8fe,6b,c743dad4,...) at _cv_wait+0x243 usb_process(c743dac4,ed33cd28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c743dac4,ed33cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed33cd60, ebp = 0 --- Tracing command usb pid 15 tid 100045 td 0xc74392e0 sched_switch(c74392e0,0,104,191,62a8f7e0,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74392e0,0,c0ef7c00,268,c74392e0,...) at sleepq_switch+0x162 sleepq_wait(c7434b5c,0,ed339cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7434b5c,c7434bfc,c0ecc8fe,6b,c7434b64,...) at _cv_wait+0x243 usb_process(c7434b54,ed339d28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c7434b54,ed339d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed339d60, ebp = 0 --- Tracing command usb pid 15 tid 100044 td 0xc74395c0 sched_switch(c74395c0,0,104,191,f1b950f2,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74395c0,0,c0ef7c00,268,c74395c0,...) at sleepq_switch+0x162 sleepq_wait(c7434b2c,0,ed336cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7434b2c,c7434bfc,c0ecc8fe,6b,c7434b34,...) at _cv_wait+0x243 usb_process(c7434b24,ed336d28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c7434b24,ed336d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed336d60, ebp = 0 --- Tracing command usb pid 15 tid 100043 td 0xc74398a0 sched_switch(c74398a0,0,104,191,62446832,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74398a0,0,c0ef7c00,268,c74398a0,...) at sleepq_switch+0x162 sleepq_wait(c7434afc,0,ed333cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7434afc,c7434bfc,c0ecc8fe,6b,c7434b04,...) at _cv_wait+0x243 usb_process(c7434af4,ed333d28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c7434af4,ed333d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed333d60, ebp = 0 --- Tracing command usb pid 15 tid 100042 td 0xc7439b80 sched_switch(c7439b80,0,104,191,624453c5,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7439b80,0,c0ef7c00,268,c7439b80,...) at sleepq_switch+0x162 sleepq_wait(c7434acc,0,e0ffdcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7434acc,c7434bfc,c0ecc8fe,6b,c7434ad4,...) at _cv_wait+0x243 usb_process(c7434ac4,e0ffdd28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c7434ac4,e0ffdd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0ffdd60, ebp = 0 --- Tracing command usb pid 15 tid 100040 td 0xc7426000 sched_switch(c7426000,0,104,191,62443ea4,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7426000,0,c0ef7c00,268,c7426000,...) at sleepq_switch+0x162 sleepq_wait(c742fb5c,0,e0ff6cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c742fb5c,c742fbfc,c0ecc8fe,6b,c742fb64,...) at _cv_wait+0x243 usb_process(c742fb54,e0ff6d28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c742fb54,e0ff6d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0ff6d60, ebp = 0 --- Tracing command usb pid 15 tid 100039 td 0xc74262e0 sched_switch(c74262e0,0,104,191,f08b9696,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74262e0,0,c0ef7c00,268,c74262e0,...) at sleepq_switch+0x162 sleepq_wait(c742fb2c,0,e0ff3cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c742fb2c,c742fbfc,c0ecc8fe,6b,c742fb34,...) at _cv_wait+0x243 usb_process(c742fb24,e0ff3d28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c742fb24,e0ff3d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0ff3d60, ebp = 0 --- Tracing command usb pid 15 tid 100038 td 0xc74265c0 sched_switch(c74265c0,0,104,191,61e0bc80,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74265c0,0,c0ef7c00,268,c74265c0,...) at sleepq_switch+0x162 sleepq_wait(c742fafc,0,e0ff0cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c742fafc,c742fbfc,c0ecc8fe,6b,c742fb04,...) at _cv_wait+0x243 usb_process(c742faf4,e0ff0d28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c742faf4,e0ff0d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0ff0d60, ebp = 0 --- Tracing command usb pid 15 tid 100037 td 0xc74268a0 sched_switch(c74268a0,0,104,191,61e0ab75,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74268a0,0,c0ef7c00,268,c74268a0,...) at sleepq_switch+0x162 sleepq_wait(c742facc,0,e0fedcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c742facc,c742fbfc,c0ecc8fe,6b,c742fad4,...) at _cv_wait+0x243 usb_process(c742fac4,e0fedd28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c742fac4,e0fedd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0fedd60, ebp = 0 --- Tracing command usb pid 15 tid 100035 td 0xc7428000 sched_switch(c7428000,0,104,191,61e096e0,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7428000,0,c0ef7c00,268,c7428000,...) at sleepq_switch+0x162 sleepq_wait(c741fb5c,0,e0fe6cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c741fb5c,c741fbfc,c0ecc8fe,6b,c741fb64,...) at _cv_wait+0x243 usb_process(c741fb54,e0fe6d28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c741fb54,e0fe6d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0fe6d60, ebp = 0 --- Tracing command usb pid 15 tid 100034 td 0xc74282e0 sched_switch(c74282e0,0,104,191,ee9d27fd,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74282e0,0,c0ef7c00,268,c74282e0,...) at sleepq_switch+0x162 sleepq_wait(c741fb2c,0,e0fe3cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c741fb2c,c741fbfc,c0ecc8fe,6b,c741fb34,...) at _cv_wait+0x243 usb_process(c741fb24,e0fe3d28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c741fb24,e0fe3d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0fe3d60, ebp = 0 --- Tracing command usb pid 15 tid 100033 td 0xc74285c0 sched_switch(c74285c0,0,104,191,617c5427,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74285c0,0,c0ef7c00,268,c74285c0,...) at sleepq_switch+0x162 sleepq_wait(c741fafc,0,e0fe0cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c741fafc,c741fbfc,c0ecc8fe,6b,c741fb04,...) at _cv_wait+0x243 usb_process(c741faf4,e0fe0d28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c741faf4,e0fe0d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0fe0d60, ebp = 0 --- Tracing command usb pid 15 tid 100032 td 0xc74288a0 sched_switch(c74288a0,0,104,191,617c429b,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74288a0,0,c0ef7c00,268,c74288a0,...) at sleepq_switch+0x162 sleepq_wait(c741facc,0,e0fddcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c741facc,c741fbfc,c0ecc8fe,6b,c741fad4,...) at _cv_wait+0x243 usb_process(c741fac4,e0fddd28,c0eecdb0,390,c7429580,...) at usb_process+0x193 fork_exit(c08aa910,c741fac4,e0fddd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0fddd60, ebp = 0 --- Tracing command yarrow pid 14 tid 100017 td 0xc72cf5c0 sched_switch(c72cf5c0,0,104,191,55357e33,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c72cf5c0,0,c0ef7c00,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c1096104,0,c0ee5532,2,0,...) at sleepq_timedwait+0x6b _sleep(c1096104,0,0,c0ee5532,64,...) at _sleep+0x342 pause(c0ee5532,64,c0eac60e,111,0,...) at pause+0x47 random_kthread(0,c6f62d28,c0eecdb0,390,c71bb000,...) at random_kthread+0x1ef fork_exit(c07c4b00,0,c6f62d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f62d60, ebp = 0 --- Tracing command geom pid 13 tid 100015 td 0xc72cfb80 sched_switch(c72cfb80,0,104,191,5532895b,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c72cfb80,0,c0ef7c00,268,0,...) at sleepq_switch+0x162 sleepq_wait(c1093d28,5c,c0ee5532,0,0,...) at sleepq_wait+0x63 _sleep(c1093d28,c1093c88,25c,c0ee5532,0,...) at _sleep+0x372 g_io_schedule_down(c72cfb80,0,c0ee7079,6c,c6f5cd14,...) at g_io_schedule_down+0x5b g_down_procbody(0,c6f5cd28,c0eecdb0,390,c71bb2c0,...) at g_down_procbody+0x9d fork_exit(c0999e60,0,c6f5cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f5cd60, ebp = 0 --- Tracing command geom pid 13 tid 100014 td 0xc72d0000 sched_switch(c72d0000,0,104,191,5559138f,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c72d0000,0,c0ef7c00,268,0,...) at sleepq_switch+0x162 sleepq_wait(c1093d24,5c,c0ee5532,0,0,...) at sleepq_wait+0x63 _sleep(c1093d24,c1093ca8,25c,c0ee5532,0,...) at _sleep+0x372 g_io_schedule_up(c72d0000,0,c0ee7079,5f,c6f59d14,...) at g_io_schedule_up+0x134 g_up_procbody(0,c6f59d28,c0eecdb0,390,c71bb2c0,...) at g_up_procbody+0x9d fork_exit(c0999f00,0,c6f59d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f59d60, ebp = 0 --- Tracing command geom pid 13 tid 100013 td 0xc71c22e0 sched_switch(c71c22e0,0,104,191,e8829f6d,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c71c22e0,0,c0ef7c00,268,0,...) at sleepq_switch+0x162 sleepq_wait(c1093d1c,5c,c0ee5532,0,0,...) at sleepq_wait+0x63 _sleep(c1093d1c,c1093c4c,25c,c0ee5532,0,...) at _sleep+0x372 g_run_events(c10a3f80,0,c0ee7079,79,c6f56d14,...) at g_run_events+0x526 g_event_procbody(0,c6f56d28,c0eecdb0,390,c71bb2c0,...) at g_event_procbody+0x90 fork_exit(c0999fa0,0,c6f56d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f56d60, ebp = 0 --- Tracing command intr pid 12 tid 100066 td 0xc75728a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100065 td 0xc7572b80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100063 td 0xc743a2e0 sched_switch(c743a2e0,0,109,191,fbddf01,...) at sched_switch+0x3bc mi_switch(109,0,c0eed0d3,554,c743a2e0,...) at mi_switch+0x1eb ithread_loop(c71b98b0,ed37dd28,c0eecdb0,390,c71bb580,...) at ithread_loop+0x1fb fork_exit(c09db400,c71b98b0,ed37dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed37dd60, ebp = 0 --- Tracing command intr pid 12 tid 100060 td 0xc743ab80 sched_switch(c743ab80,0,109,191,5731ded6,...) at sched_switch+0x3bc mi_switch(109,0,c0eed0d3,554,c743ab80,...) at mi_switch+0x1eb ithread_loop(c71b9a70,ed371d28,c0eecdb0,390,c71bb580,...) at ithread_loop+0x1fb fork_exit(c09db400,c71b9a70,ed371d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed371d60, ebp = 0 --- Tracing command intr pid 12 tid 100059 td 0xc7447000 sched_switch(c7447000,0,109,191,7db44c5a,...) at sched_switch+0x3bc mi_switch(109,0,c0eed0d3,554,c7447000,...) at mi_switch+0x1eb ithread_loop(c71ba280,ed369d28,c0eecdb0,390,c71bb580,...) at ithread_loop+0x1fb fork_exit(c09db400,c71ba280,ed369d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed369d60, ebp = 0 --- Tracing command intr pid 12 tid 100054 td 0xc744a000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100041 td 0xc743a000 sched_switch(c743a000,0,109,191,8687a09e,...) at sched_switch+0x3bc mi_switch(109,0,c0eed0d3,554,c743a000,...) at mi_switch+0x1eb ithread_loop(c71b98d0,e0ffad28,c0eecdb0,390,c71bb580,...) at ithread_loop+0x1fb fork_exit(c09db400,c71b98d0,e0ffad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0ffad60, ebp = 0 --- Tracing command intr pid 12 tid 100036 td 0xc7426b80 sched_switch(c7426b80,0,109,191,810ed9d1,...) at sched_switch+0x3bc mi_switch(109,0,c0eed0d3,554,c7426b80,...) at mi_switch+0x1eb ithread_loop(c71ba100,e0fead28,c0eecdb0,390,c71bb580,...) at ithread_loop+0x1fb fork_exit(c09db400,c71ba100,e0fead28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0fead60, ebp = 0 --- Tracing command intr pid 12 tid 100031 td 0xc7428b80 sched_switch(c7428b80,0,109,191,577d5ec1,...) at sched_switch+0x3bc mi_switch(109,0,c0eed0d3,554,c7428b80,...) at mi_switch+0x1eb ithread_loop(c71ba130,e0fdad28,c0eecdb0,390,c71bb580,...) at ithread_loop+0x1fb fork_exit(c09db400,c71ba130,e0fdad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0fdad60, ebp = 0 --- Tracing command intr pid 12 tid 100030 td 0xc72d02e0 sched_switch(c72d02e0,0,109,191,55576b5e,...) at sched_switch+0x3bc mi_switch(109,0,c0eed0d3,554,c72d02e0,...) at mi_switch+0x1eb ithread_loop(c71b9a30,e0e84d28,c0eecdb0,390,c71bb580,...) at ithread_loop+0x1fb fork_exit(c09db400,c71b9a30,e0e84d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0e84d60, ebp = 0 --- Tracing command intr pid 12 tid 100029 td 0xc72d05c0 sched_switch(c72d05c0,0,109,191,32eda0fb,...) at sched_switch+0x3bc mi_switch(109,0,c0eed0d3,554,c72d05c0,...) at mi_switch+0x1eb ithread_loop(c71b9ae0,e0e80d28,c0eecdb0,390,c71bb580,...) at ithread_loop+0x1fb fork_exit(c09db400,c71b9ae0,e0e80d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0e80d60, ebp = 0 --- Tracing command intr pid 12 tid 100028 td 0xc72d08a0 sched_switch(c72d08a0,0,109,191,57d420c0,...) at sched_switch+0x3bc mi_switch(109,0,c0eed0d3,554,c72d08a0,...) at mi_switch+0x1eb ithread_loop(c71ba200,c6fb1d28,c0eecdb0,390,c71bb580,...) at ithread_loop+0x1fb fork_exit(c09db400,c71ba200,c6fb1d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6fb1d60, ebp = 0 --- Tracing command intr pid 12 tid 100027 td 0xc72d0b80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100026 td 0xc73ab000 sched_switch(c73ab000,0,109,191,555832c8,...) at sched_switch+0x3bc mi_switch(109,0,c0eed0d3,554,c73ab000,...) at mi_switch+0x1eb ithread_loop(c71b9bb0,c6f7dd28,c0eecdb0,390,c71bb580,...) at ithread_loop+0x1fb fork_exit(c09db400,c71b9bb0,c6f7dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f7dd60, ebp = 0 --- Tracing command intr pid 12 tid 100021 td 0xc72078a0 sched_switch(c72078a0,0,109,191,92143105,...) at sched_switch+0x3bc mi_switch(109,0,c0eed0d3,554,c72078a0,...) at mi_switch+0x1eb ithread_loop(c71b9bc0,c6f6ed28,c0eecdb0,390,c71bb580,...) at ithread_loop+0x1fb fork_exit(c09db400,c71b9bc0,c6f6ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f6ed60, ebp = 0 --- Tracing command intr pid 12 tid 100019 td 0xc72cf000 sched_switch(c72cf000,0,109,191,1c38c5eb,...) at sched_switch+0x3bc mi_switch(109,0,c0eed0d3,554,c72cf000,...) at mi_switch+0x1eb ithread_loop(c71b9bd0,c6f68d28,c0eecdb0,390,c71bb580,...) at ithread_loop+0x1fb fork_exit(c09db400,c71b9bd0,c6f68d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f68d60, ebp = 0 --- Tracing command intr pid 12 tid 100012 td 0xc71c25c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100011 td 0xc71c28a0 sched_switch(c71c28a0,0,109,191,552ed7a7,...) at sched_switch+0x3bc mi_switch(109,0,c0eed0d3,554,c71c28a0,...) at mi_switch+0x1eb ithread_loop(c71b9bf0,c6f50d28,c0eecdb0,390,c71bb580,...) at ithread_loop+0x1fb fork_exit(c09db400,c71b9bf0,c6f50d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f50d60, ebp = 0 --- Tracing command intr pid 12 tid 100010 td 0xc71c2b80 sched_switch(c71c2b80,0,109,191,51c0079d,...) at sched_switch+0x3bc mi_switch(109,0,c0eed0d3,554,c71c2b80,...) at mi_switch+0x1eb ithread_loop(c71b9c00,c6f4dd28,c0eecdb0,390,c71bb580,...) at ithread_loop+0x1fb fork_exit(c09db400,c71b9c00,c6f4dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f4dd60, ebp = 0 --- Tracing command intr pid 12 tid 100009 td 0xc7207000 sched_switch(c7207000,0,109,191,440398de,...) at sched_switch+0x3bc mi_switch(109,0,c0eed0d3,554,c7207000,...) at mi_switch+0x1eb ithread_loop(c71b9c10,c6f4ad28,c0eecdb0,390,c71bb580,...) at ithread_loop+0x1fb fork_exit(c09db400,c71b9c10,c6f4ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f4ad60, ebp = 0 --- Tracing command intr pid 12 tid 100008 td 0xc72072e0 sched_switch(c72072e0,0,109,191,595e45f2,...) at sched_switch+0x3bc mi_switch(109,0,c0eed0d3,554,c72072e0,...) at mi_switch+0x1eb ithread_loop(c71b9c20,c6f47d28,c0eecdb0,390,c71bb580,...) at ithread_loop+0x1fb fork_exit(c09db400,c71b9c20,c6f47d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f47d60, ebp = 0 --- Tracing command intr pid 12 tid 100007 td 0xc72075c0 sched_switch(c72075c0,0,109,191,f3b72bb7,...) at sched_switch+0x3bc mi_switch(109,0,c0eed0d3,554,c72075c0,...) at mi_switch+0x1eb ithread_loop(c71b9c30,c6f44d28,c0eecdb0,390,c71bb580,...) at ithread_loop+0x1fb fork_exit(c09db400,c71b9c30,c6f44d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f44d60, ebp = 0 --- Tracing command idle pid 11 tid 100006 td 0xc71bd000 sched_switch(c71bd000,0,608,18c,525f602f,...) at sched_switch+0x3bc mi_switch(608,0,c0ef2c9f,cd,0,...) at mi_switch+0x1eb critical_exit(0,ffffffff,c10a5300,2,c6f40ce8,...) at critical_exit+0xa8 cpu_idle(0,c6f40cc4,c0ef446b,a05,c71bd000,...) at cpu_idle+0x97 sched_idletd(0,c6f40d28,c0eecdb0,390,c71bb840,...) at sched_idletd+0x263 fork_exit(c0a2a2c0,0,c6f40d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f40d60, ebp = 0 --- Tracing command idle pid 11 tid 100005 td 0xc71bd2e0 cpustop_handler(4,c6f3dc04,c0d358d6,c6f2ac80,c6f3db98,...) at cpustop_handler+0x49 ipi_nmi_handler(c6f2ac80,c6f3db98,0,c1238af8,c71bb840,...) at ipi_nmi_handler+0x44 trap(c6f3dc10) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0d1a7d5, esp = 0xc6f3dc50, ebp = 0xc6f3dc50 --- acpi_cpu_c1(c6f3dc6c,c105c2d0,2,301,1036,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c6f3dca0,c0d289f9,0,ffffffff,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,ffffffff,c10a4c80,2,c6f3dce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c6f3dcc4,c0ef446b,a05,c71bd2e0,...) at cpu_idle+0x89 sched_idletd(0,c6f3dd28,c0eecdb0,390,c71bb840,...) at sched_idletd+0x263 fork_exit(c0a2a2c0,0,c6f3dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f3dd60, ebp = 0 --- Tracing command idle pid 11 tid 100004 td 0xc71bd5c0 sched_switch(c71bd5c0,0,608,18c,55595d41,...) at sched_switch+0x3bc mi_switch(608,0,c0ef2c9f,cd,0,...) at mi_switch+0x1eb critical_exit(0,ffffffff,c10a4600,2,c6f3ace8,...) at critical_exit+0xa8 cpu_idle(0,c6f3acc4,c0ef446b,3bb,c71bd5c0,...) at cpu_idle+0x97 sched_idletd(0,c6f3ad28,c0eecdb0,390,c71bb840,...) at sched_idletd+0x263 fork_exit(c0a2a2c0,0,c6f3ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f3ad60, ebp = 0 --- Tracing command idle pid 11 tid 100003 td 0xc71bd8a0 cpustop_handler(1,c6f37c04,c0d358d6,f5,0,...) at cpustop_handler+0x49 ipi_nmi_handler(f5,0,1eac1de,c0d1a7d5,c71bb840,...) at ipi_nmi_handler+0x44 trap(c6f37c10) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0d1a7d5, esp = 0xc6f37c50, ebp = 0xc6f37c50 --- acpi_cpu_c1(c6f37c6c,c105c2d0,0,301,1036,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c6f37ca0,c0d289f9,0,2710,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,2710,c10a3f80,2,c6f37ce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c6f37cc4,c0ef446b,a05,c71bd8a0,...) at cpu_idle+0x89 sched_idletd(0,c6f37d28,c0eecdb0,390,c71bb840,...) at sched_idletd+0x263 fork_exit(c0a2a2c0,0,c6f37d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f37d60, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc71bdb80 sched_switch(c71bdb80,0,104,191,1c702f53,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c71bdb80,0,c0ef7c00,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0ef7c00,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c71bbb00,6c,c0efa9de,100,0,...) at sleepq_wait_sig+0x17 _sleep(c71bbb00,c71bbb88,16c,c0efa9de,0,...) at _sleep+0x35c kern_wait(c71bdb80,ffffffff,c6f33c44,0,0,...) at kern_wait+0xc06 wait4(c71bdb80,c6f33cec,c6f33d28,c0ef858a,0,...) at wait4+0x3b syscallenter(c71bdb80,c6f33ce4,c6f33ce4,0,0,...) at syscallenter+0x263 syscall(c6f33d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x8060de7, esp = 0xbfbfe8bc, ebp = 0xbfbfe8d8 --- Tracing command audit pid 10 tid 100001 td 0xc71c2000 sched_switch(c71c2000,0,104,191,617a2b02,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c71c2000,0,c0ef7c00,268,c71c2000,...) at sleepq_switch+0x162 sleepq_wait(c1213300,0,c6f30c8c,1,0,...) at sleepq_wait+0x63 _cv_wait(c1213300,c12132e4,c0f22597,194,0,...) at _cv_wait+0x243 audit_worker(0,c6f30d28,c0eecdb0,390,c71bc000,...) at audit_worker+0x84 fork_exit(c0bfb140,0,c6f30d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f30d60, ebp = 0 --- Tracing command kernel pid 0 tid 100077 td 0xc77edb80 sched_switch(c77edb80,0,104,191,3a19c4ab,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c77edb80,0,c0ef7c00,268,0,...) at sleepq_switch+0x162 sleepq_wait(c77b4740,0,c0ee5532,0,0,...) at sleepq_wait+0x63 _sleep(c77b4740,c77b4758,0,c0ee5532,0,...) at _sleep+0x372 taskqueue_thread_loop(c126ae20,ed411d28,c0eecdb0,390,c1093e00,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a46f10,c126ae20,ed411d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed411d60, ebp = 0 --- Tracing command kernel pid 0 tid 100067 td 0xc75725c0 sched_switch(c75725c0,0,104,191,51c204d8,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c75725c0,0,c0ef7c00,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c1096104,0,c0ee5532,2,0,...) at sleepq_timedwait+0x6b _sleep(c1096104,0,0,c0ee5532,bb8,...) at _sleep+0x342 pause(c0ee5532,bb8,124,122,c1093e88,...) at pause+0x47 deadlkres(0,ed393d28,c0eecdb0,390,c1093e00,...) at deadlkres+0x324 fork_exit(c09bb920,0,ed393d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed393d60, ebp = 0 --- Tracing command kernel pid 0 tid 100061 td 0xc743a8a0 sched_switch(c743a8a0,0,104,191,51f6b3d9,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c743a8a0,0,c0ef7c00,268,0,...) at sleepq_switch+0x162 sleepq_wait(c74817c0,0,c0ef31c1,c0ee5532,0,...) at sleepq_wait+0x63 msleep_spin(c74817c0,c74817d8,c0ee5532,0,c0ef0a47,...) at msleep_spin+0x21d taskqueue_thread_loop(c747d49c,ed377d28,c0eecdb0,390,c1093e00,...) at taskqueue_thread_loop+0x8f fork_exit(c0a46f10,c747d49c,ed377d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed377d60, ebp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xc73ab2e0 sched_switch(c73ab2e0,0,104,191,7bed1be1,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c73ab2e0,0,c0ef7c00,268,0,...) at sleepq_switch+0x162 sleepq_wait(c71936c0,0,c0ef31c1,c0ee5532,0,...) at sleepq_wait+0x63 msleep_spin(c71936c0,c71936d8,c0ee5532,0,c0ef0a47,...) at msleep_spin+0x21d taskqueue_thread_loop(c105ef38,c6f7ad28,c0eecdb0,390,c1093e00,...) at taskqueue_thread_loop+0x8f fork_exit(c0a46f10,c105ef38,c6f7ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f7ad60, ebp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xc73ab5c0 sched_switch(c73ab5c0,0,104,191,7bed0d3f,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c73ab5c0,0,c0ef7c00,268,0,...) at sleepq_switch+0x162 sleepq_wait(c71936c0,0,c0ef31c1,c0ee5532,0,...) at sleepq_wait+0x63 msleep_spin(c71936c0,c71936d8,c0ee5532,0,c0ef0a47,...) at msleep_spin+0x21d taskqueue_thread_loop(c105ef38,c6f77d28,c0eecdb0,390,c1093e00,...) at taskqueue_thread_loop+0x8f fork_exit(c0a46f10,c105ef38,c6f77d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f77d60, ebp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xc73ab8a0 sched_switch(c73ab8a0,0,104,191,7becfddd,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c73ab8a0,0,c0ef7c00,268,0,...) at sleepq_switch+0x162 sleepq_wait(c71936c0,0,c0ef31c1,c0ee5532,0,...) at sleepq_wait+0x63 msleep_spin(c71936c0,c71936d8,c0ee5532,0,c0ef0a47,...) at msleep_spin+0x21d taskqueue_thread_loop(c105ef38,c6f74d28,c0eecdb0,390,c1093e00,...) at taskqueue_thread_loop+0x8f fork_exit(c0a46f10,c105ef38,c6f74d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f74d60, ebp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xc73abb80 sched_switch(c73abb80,0,104,191,7be6c5ec,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c73abb80,0,c0ef7c00,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7193700,0,c0ee5532,0,0,...) at sleepq_wait+0x63 _sleep(c7193700,c7193718,0,c0ee5532,0,...) at _sleep+0x372 taskqueue_thread_loop(c1094798,c6f71d28,c0eecdb0,390,c1093e00,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a46f10,c1094798,c6f71d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f71d60, ebp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xc7207b80 sched_switch(c7207b80,0,104,191,7be6b3f5,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7207b80,0,c0ef7c00,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7193780,0,c0ee5532,0,0,...) at sleepq_wait+0x63 _sleep(c7193780,c7193798,0,c0ee5532,0,...) at _sleep+0x372 taskqueue_thread_loop(c12138e4,c6f6bd28,c0eecdb0,390,c1093e00,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a46f10,c12138e4,c6f6bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f6bd60, ebp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xc72cf2e0 sched_switch(c72cf2e0,0,104,191,75c38dcd,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c72cf2e0,0,c0ef7c00,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7193800,0,c0ee5532,0,0,...) at sleepq_wait+0x63 _sleep(c7193800,c7193818,0,c0ee5532,0,...) at _sleep+0x372 taskqueue_thread_loop(c10b2d08,c6f65d28,c0eecdb0,390,c1093e00,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a46f10,c10b2d08,c6f65d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f65d60, ebp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xc72cf8a0 sched_switch(c72cf8a0,0,104,191,96a54cc4,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c72cf8a0,0,c0ef7c00,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7193c00,0,c0ee5532,0,0,...) at sleepq_wait+0x63 _sleep(c7193c00,c7193c18,0,c0ee5532,0,...) at _sleep+0x372 taskqueue_thread_loop(c10b1720,c6f5fd28,c0eecdb0,390,c1093e00,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a46f10,c10b1720,c6f5fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f5fd60, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc10940c0 sched_switch(c10940c0,0,104,191,8d337431,...) at sched_switch+0x3bc mi_switch(104,0,c0ef7c00,1f3,54,...) at mi_switch+0x1eb sleepq_switch(c10940c0,0,c0ef7c00,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c1093e00,54,c0ef4d7d,0,0,...) at sleepq_timedwait+0x6b _sleep(c1093e00,0,54,c0ef4d7d,2710,...) at _sleep+0x342 scheduler(0,181ec00,181ec00,181e000,1825000,...) at scheduler+0x240 mi_startup() at mi_startup+0xac begin() at begin+0x2c db:0:allt> call doadump Physical memory: 3435 MB Physical memory: 3435 MB Dumping 249 MB:Dumping 249 MB: 234 234 218 218 202 202 186 186 170 170 154 154 138 138 122 122 106 106 90 90 74 74 58 58 42 42 26 26 10 10 Dump complete Dump complete = 0 db:0:doadump> reset cpu_reset: Restarting BSP cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 3 cpu_reset_proxy: Stopped CPU 3 (kgdb) bt #0 doadump (textdump=0xef8280c4) at pcpu.h:244 #1 0xc04e38d9 in db_fncall (dummy1=0xc0a3885a, dummy2=0x0, dummy3=0xffffffff, dummy4=0xef828158 "l\201\202ï") at ../../../ddb/db_command.c:572 #2 0xc04e3d0f in db_command (last_cmdp=0xc105d0fc, cmd_table=0x0, dopager=0x0) at ../../../ddb/db_command.c:448 #3 0xc04e3dc4 in db_command_script (command=0xc105e008 "call doadump") at ../../../ddb/db_command.c:519 #4 0xc04e8100 in db_script_exec (scriptname=0xc105d960 "doadump", warnifnotfound=Variable "warnifnotfound" is not available. ) at ../../../ddb/db_script.c:302 #5 0xc04e8191 in db_run_cmd (addr=0x1, have_addr=0x0, count=0xc1235e00, modif=0xef828290 "") at ../../../ddb/db_script.c:375 #6 0xc04e3cd1 in db_command (last_cmdp=0xc105d0fc, cmd_table=0x0, dopager=0x1) at ../../../ddb/db_command.c:448 #7 0xc04e3e2a in db_command_loop () at ../../../ddb/db_command.c:501 #8 0xc04e5e1d in db_trap (type=0x3, code=0x0) at ../../../ddb/db_main.c:229 #9 0xc0a389c5 in kdb_trap (type=0x3, code=0x0, tf=0xef82843c) at ../../../kern/subr_kdb.c:531 #10 0xc0d35f0b in trap (frame=0xef82843c) at ../../../i386/i386/trap.c:719 #11 0xc0d1ee0c in calltrap () at ../../../i386/i386/exception.s:168 #12 0xc0a3885a in kdb_enter (why=0xc0ef245a "panic", msg=0xc0ef245a "panic") at cpufunc.h:71 #13 0xc0a03da4 in panic (fmt=0xc0ef0c8d "_mtx_lock_sleep: recursed on non-recursive mutex %s @ %s:%d\n") at ../../../kern/kern_shutdown.c:587 #14 0xc09f472a in _mtx_lock_sleep (m=0xc7884e48, tid=0xc9a282e0, opts=0x0, file=0xc0f297e1 "../../../ufs/ufs/ufs_quota.c", line=0x4d7) at ../../../kern/kern_mutex.c:341 #15 0xc09f4947 in _mtx_lock_flags (m=0xc7884e48, opts=0x0, file=0xc0f297e1 "../../../ufs/ufs/ufs_quota.c", line=0x4d7) at ../../../kern/kern_mutex.c:203 #16 0xc0c4d8ad in dqget (vp=0xc7bc5d98, id=0x2, ump=0xc7884e00, type=0x0, dqp=0xc7bab520) at ../../../ufs/ufs/ufs_quota.c:1239 #17 0xc0c4ebdb in getinoquota (ip=0xc7bab4fc) at ../../../ufs/ufs/ufs_quota.c:134 #18 0xc0c49478 in ufs_inactive (ap=0xef8286d4) at ../../../ufs/ufs/ufs_inode.c:128 #19 0xc0d55a45 in VOP_INACTIVE_APV (vop=0xc1040120, a=0xef8286d4) at vnode_if.c:1863 #20 0xc0a9f681 in vinactive (vp=0xc7bc5d98, td=0xc9a282e0) at vnode_if.h:807 #21 0xc0aa19bc in vputx (vp=0xc7bc5d98, func=0x2) at ../../../kern/vfs_subr.c:2265 #22 0xc0aa1a30 in vput (vp=0xc7bc5d98) at ../../../kern/vfs_subr.c:2292 #23 0xc0c29d43 in handle_workitem_remove (dirrem=0xca4198c0, flags=0x80200) at ../../../ufs/ffs/ffs_softdep.c:9315 #24 0xc0c291a4 in process_worklist_item (mp=0xc7b9a87c, target=0x8, flags=0x200) at ../../../ufs/ffs/ffs_softdep.c:1726 #25 0xc0c392c2 in softdep_request_cleanup (fs=0xc7886800, vp=0xca793ae0, cred=0xc7872a80, resource=0x4) at ../../../ufs/ffs/ffs_softdep.c:12586 #26 0xc0c135cf in ffs_alloc (ip=0xca40f3a0, lbn=0x0, bpref=0x6a3d58, size=0x800, flags=0x2010000, cred=0xc7872a80, bnp=0xef828a1c) at ../../../ufs/ffs/ffs_alloc.c:221 #27 0xc0c175ec in ffs_balloc_ufs2 (vp=0xca793ae0, startoffset=Variable "startoffset" is not available. ) at ../../../ufs/ffs/ffs_balloc.c:749 #28 0xc0c41b5d in ffs_write (ap=0xef828b94) at ../../../ufs/ffs/ffs_vnops.c:716 #29 0xc0d58d86 in VOP_WRITE_APV (vop=0xc103fba0, a=0xef828b94) at vnode_if.c:951 #30 0xc0aada4a in vn_write (fp=0xc997e690, uio=0xef828c28, active_cred=0xc7872a80, flags=0x0, td=0xc9a282e0) at vnode_if.h:413 #31 0xc0a512a5 in dofilewrite (td=0xc9a282e0, fd=0x4, fp=0xc997e690, auio=0xef828c28, offset=0xffffffffffffffff, flags=0x0) at file.h:239 #32 0xc0a52a48 in kern_writev (td=0xc9a282e0, fd=0x4, auio=0xef828c28) at ../../../kern/sys_generic.c:447 #33 0xc0a52b5f in write (td=0xc9a282e0, uap=0xef828cec) at ../../../kern/sys_generic.c:363 #34 0xc0a472d3 in syscallenter (td=0xc9a282e0, sa=0xef828ce4) at ../../../kern/subr_trap.c:344 #35 0xc0d357bf in syscall (frame=0xef828d28) at ../../../i386/i386/trap.c:1082 #36 0xc0d1ee71 in Xint0x80_syscall () at ../../../i386/i386/exception.s:266 #37 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 16 #16 0xc0c4d8ad in dqget (vp=0xc7bc5d98, id=0x2, ump=0xc7884e00, type=0x0, dqp=0xc7bab520) at ../../../ufs/ufs/ufs_quota.c:1239 1239 UFS_LOCK(ump); (kgdb) l 1234 * creation of 100GB+ quota data files. 1235 */ 1236 if ((int)id < 0) 1237 return (EINVAL); 1238 1239 UFS_LOCK(ump); 1240 dqvp = ump->um_quotas[type]; 1241 if (dqvp == NULLVP || (ump->um_qflags[type] & QTF_CLOSING)) { 1242 *dqp = NODQUOT; 1243 UFS_UNLOCK(ump); (kgdb) info loc buf = "A1ðÀà\202¢É|¨¹Ç$\004\000\000 \206\202ï3E\237À|¨¹Ç\004\000\000\000G\nïÀß\000\000\000|¨¹Ç\000\000\000\000\230]¼ÇX\206\202ï/ŪÀ|¨¹Ç" base = Variable "base" is not available. (kgdb) p *ump $1 = {um_mountp = 0xc7b9a87c, um_dev = 0xc7856600, um_cp = 0xc77c5c80, um_bo = 0xc79217c4, um_devvp = 0xc79216cc, um_fstype = 0x2, um_fs = 0xc7886800, um_extattr = {uepm_lock = {lock_object = { lo_name = 0xc0f292bb "ufs_extattr_sx", lo_flags = 0x2330000, lo_data = 0x0, lo_witness = 0xc71692d0}, sx_lock = 0x1}, uepm_list = {lh_first = 0x0}, uepm_ucred = 0x0, uepm_flags = 0x1}, um_nindir = 0x800, um_bptrtodb = 0x2, um_seqinc = 0x8, um_lock = {lock_object = {lo_name = 0xc0f28656 "FFS", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xc7169268}, mtx_lock = 0xc9a282e0}, um_numindirdeps = 0x0, softdep_workitem_pending = {lh_first = 0xef8287a0}, softdep_worklist_tail = 0xef8287a0, softdep_journal_pending = {lh_first = 0x0}, softdep_journal_tail = 0x0, softdep_jblocks = 0x0, softdep_unlinked = { tqh_first = 0x0, tqh_last = 0xc7884e74}, softdep_dirtycg = {lh_first = 0x0}, softdep_on_journal = 0x0, softdep_on_worklist = 0x0, softdep_deps = 0x8, softdep_accdeps = 0x14977, softdep_req = 0x0, um_quotas = { 0x0, 0x0}, um_cred = {0x0, 0x0}, um_btime = {0x0, 0x0}, um_itime = {0x0, 0x0}, um_qflags = "\000", um_savedmaxfilesize = 0x0, um_candelete = 0x0, um_balloc = 0xc0c164e0 , um_blkatoff = 0xc0c3be00 , um_truncate = 0xc0c1ac10 , um_update = 0xc0c1a8d0 , um_valloc = 0xc0c113e0 , um_vfree = 0xc0c10130 , (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/ufs/ufs/ufs_inode.c =================================================================== --- /usr/src/sys/ufs/ufs/ufs_inode.c (revision 223687) +++ /usr/src/sys/ufs/ufs/ufs_inode.c (working copy) @@ -120,15 +120,14 @@ ufs_inactive(ap) isize = ip->i_size; if (ip->i_ump->um_fstype == UFS2) isize += ip->i_din2->di_extsize; - if (ip->i_effnlink <= 0 && isize && !UFS_RDONLY(ip)) { + if (ip->i_effnlink <= 0 && isize && !UFS_RDONLY(ip)) + error = UFS_TRUNCATE(vp, (off_t)0, IO_EXT | IO_NORMAL, + NOCRED, td); + if (ip->i_nlink <= 0 && ip->i_mode && !UFS_RDONLY(ip)) { #ifdef QUOTA if (!getinoquota(ip)) (void)chkiq(ip, -1, NOCRED, FORCE); #endif - error = UFS_TRUNCATE(vp, (off_t)0, IO_EXT | IO_NORMAL, - NOCRED, td); - } - if (ip->i_nlink <= 0 && ip->i_mode && !UFS_RDONLY(ip)) { #ifdef UFS_EXTATTR ufs_extattr_vnode_inactive(vp, td); #endif Index: /usr/src/sys/ufs/ffs/ffs_softdep.c =================================================================== --- /usr/src/sys/ufs/ffs/ffs_softdep.c (revision 223687) +++ /usr/src/sys/ufs/ffs/ffs_softdep.c (working copy) @@ -12516,27 +12516,30 @@ softdep_request_cleanup(fs, vp, cred, resource) stat_cleanup_blkrequests += 1; else stat_cleanup_inorequests += 1; + /* * If we are being called because of a process doing a - * copy-on-write, then it is not safe to update the vnode - * as we may recurse into the copy-on-write routine. + * copy-on-write, then it is not safe to process any + * worklist items as we will recurse into the copyonwrite + * routine. This will result in an incoherent snapshot. */ - if (!(curthread->td_pflags & TDP_COWINPROGRESS)) { - UFS_UNLOCK(ump); - error = ffs_update(vp, 1); - UFS_LOCK(ump); - if (error != 0) - return (0); - } + if (curthread->td_pflags & TDP_COWINPROGRESS) + return (0); + UFS_UNLOCK(ump); + error = ffs_update(vp, 1); + UFS_LOCK(ump); + if (error != 0) + return (0); /* * If we are in need of resources, consider pausing for * tickdelay to give ourselves some breathing room. */ UFS_UNLOCK(ump); ACQUIRE_LOCK(&lk); + process_removes(vp); + process_truncates(vp); request_cleanup(UFSTOVFS(ump), resource); FREE_LOCK(&lk); - UFS_LOCK(ump); /* * Now clean up at least as many resources as we will need. * @@ -12568,29 +12571,23 @@ softdep_request_cleanup(fs, vp, cred, resource) roundup((fs->fs_dsize * fs->fs_minfree / 100) - fs->fs_cstotal.cs_nffree, fs->fs_frag)); } else { + UFS_LOCK(ump); printf("softdep_request_cleanup: Unknown resource type %d\n", resource); return (0); } starttime = time_second; retry: - while ((resource == FLUSH_BLOCKS_WAIT && ump->softdep_on_worklist > 0 && - fs->fs_cstotal.cs_nbfree <= needed) || - (resource == FLUSH_INODES_WAIT && fs->fs_pendinginodes > 0 && - fs->fs_cstotal.cs_nifree <= needed)) { - UFS_UNLOCK(ump); + if ((resource == FLUSH_BLOCKS_WAIT && ump->softdep_on_worklist > 0 && + fs->fs_cstotal.cs_nbfree <= needed) || + (resource == FLUSH_INODES_WAIT && fs->fs_pendinginodes > 0 && + fs->fs_cstotal.cs_nifree <= needed)) { ACQUIRE_LOCK(&lk); - process_removes(vp); - process_truncates(vp); if (ump->softdep_on_worklist > 0 && - process_worklist_item(UFSTOVFS(ump), 1, LK_NOWAIT) != 0) { + process_worklist_item(UFSTOVFS(ump), + ump->softdep_on_worklist, LK_NOWAIT) != 0) stat_worklist_push += 1; - FREE_LOCK(&lk); - UFS_LOCK(ump); - continue; - } FREE_LOCK(&lk); - UFS_LOCK(ump); } /* * If we still need resources and there are no more worklist @@ -12604,7 +12601,6 @@ retry: fs->fs_cstotal.cs_nbfree <= needed) || (resource == FLUSH_INODES_WAIT && fs->fs_pendinginodes > 0 && fs->fs_cstotal.cs_nifree <= needed)) { - UFS_UNLOCK(ump); MNT_ILOCK(mp); MNT_VNODE_FOREACH(lvp, mp, mvp) { VI_LOCK(lvp); @@ -12642,6 +12638,7 @@ retry: } if (time_second - starttime > stat_cleanup_high_delay) stat_cleanup_high_delay = time_second - starttime; + UFS_LOCK(ump); return (1); } -- Test scenario: suj22.sh