GENERIC HEAD r225868, vmcore.33 KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #0: Thu Sep 29 10:50:17 CEST 2011 test@leopard3.netperf.freebsd.org:/mnt/pho/obj/mnt/pho/src/sys/PHO i386 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: Intel(R) Xeon(R) CPU 3050 @ 2.13GHz (2128.04-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x6f2 Family = 6 Model = f Stepping = 2 Features=0xbfebfbff Features2=0xe3bd AMD Features=0x20100000 AMD Features2=0x1 TSC: P-state invariant, performance statistics real memory = 2147483648 (2048 MB) avail memory = 2080456704 (1984 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard ioapic2 irqs 48-71 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 cpu0: on acpi0 cpu1: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 16 at device 1.0 on pci0 pci1: on pcib1 pcib2: at device 0.0 on pci1 pci2: on pcib2 pcib3: at device 0.2 on pci1 pci3: on pcib3 pcib4: irq 16 at device 3.0 on pci0 pci10: on pcib4 pci10: at device 0.0 (no driver attached) pcib5: irq 17 at device 28.0 on pci0 pcib5: failed to allocate initial I/O port window: 0-0xfff pcib5: failed to allocate initial memory window: 0-0xfffff pcib5: failed to allocate initial prefetch window: 0-0xfffff pci14: on pcib5 pcib6: irq 17 at device 28.4 on pci0 pci18: on pcib6 em0: port 0x5000-0x501f mem 0xe0300000-0xe031ffff irq 16 at device 0.0 on pci18 em0: Using an MSI interrupt em0: Ethernet address: 00:30:48:8e:d0:68 pcib7: irq 16 at device 28.5 on pci0 pci19: on pcib7 em1: port 0x6000-0x601f mem 0xe0400000-0xe041ffff irq 17 at device 0.0 on pci19 em1: Using an MSI interrupt em1: Ethernet address: 00:30:48:8e:d0:69 uhci0: port 0x3000-0x301f irq 23 at device 29.0 on pci0 usbus0: on uhci0 uhci1: port 0x3020-0x303f irq 19 at device 29.1 on pci0 usbus1: on uhci1 uhci2: port 0x3040-0x305f irq 18 at device 29.2 on pci0 usbus2: on uhci2 uhci3: port 0x3060-0x307f irq 16 at device 29.3 on pci0 usbus3: on uhci3 ehci0: mem 0xe0000000-0xe00003ff irq 23 at device 29.7 on pci0 usbus4: EHCI version 1.0 usbus4: on ehci0 pcib8: at device 30.0 on pci0 pci20: on pcib8 vgapci0: port 0x7000-0x70ff mem 0xe8000000-0xefffffff,0xe0500000-0xe050ffff irq 16 at device 4.0 on pci20 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x30a0-0x30af at device 31.1 on pci0 ata0: on atapci0 ahci0: port 0x30e8-0x30ef,0x30dc-0x30df,0x30e0-0x30e7,0x30d8-0x30db,0x30b0-0x30bf mem 0xe0000400-0xe00007ff irq 19 at device 31.2 on pci0 ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 pci0: at device 31.3 (no driver attached) acpi_button0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (115200,n,8,1) uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fd0: <1440-KB 3.5" drive> on fdc0 drive 0 pmtimer0 on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x100> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] ppc0: parallel port not found. est0: on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 828082806000828 device_attach: est0 attach returned 6 p4tcc0: on cpu0 est1: on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 828082806000828 device_attach: est1 attach returned 6 p4tcc1: on cpu1 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 12Mbps Full Speed USB v1.0 ugen0.1: at usbus0 uhub0: on usbus0 Expensive timeout(9) function: 0xc0a4bb00(0xc5b045c0) 0.011049728 s ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 usbus4: 480Mbps High Speed USB v2.0 ugen4.1: at usbus4 uhub4: on usbus4 Expensive timeout(9) function: 0xc0529070(0xc5beac00) 0.020016789 s ada0 at ahcich0 bus 0 scbus1 target 0 lun 0 ada0: ATA-7cd0 at ata0 bus 0 scbus0 target 0 lun 0 cd0: Removable CD-ROM SCSI-0 device cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 238475MB (488397168 512 byte sectors: 16H 63S/T 16383C) ada0: Previously was known as ad4 SMP: AP CPU #1 Launched! Timecounter "TSC-low" frequency 16625342 Hz quality 1000 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. uhub1: 2 ports with 2 removable, self powered uhub0: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered Root mount waiting for: usbus4 Root mount waiting for: usbus4 Root mount waiting for: usbus4 uhub4: 8 ports with 8 removable, self powered Trying to mount root from nfs: []... NFS ROOT: 192.168.5.1:/zoo/leopard3 em0: link state changed to UP Interface em0 IP-Address 192.168.5.109 Broadcast 192.168.5.255 Setting hostuuid: 64a93476-27b1-3004-c299-0030488ed164. Setting hostid: 0x7acc54f1. Starting ddb. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/ada0d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0d: clean, 111543524 free (1556 frags, 13942746 blocks, 0.0% fragmentation) mount_nfs: can't update /var/db/mounttab for 192.168.5.1:/zoo/leopard3 Mounting local file systems:. Setting hostname: leopard3.netperf.freebsd.org. Starting Network: lo0 em0 em1. lo0: flags=8049 metric 0 mtu 16384 options=3 inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8 nd6 options=21 em0: flags=8843 metric 0 mtu 1500 options=219b ether 00:30:48:8e:d0:68 inet 192.168.5.109 netmask 0xffffff00 broadcast 192.168.5.255 inet6 fe80::230:48ff:fe8e:d068%em0 prefixlen 64 tentative scopeid 0x1 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active em1: flags=8802 metric 0 mtu 1500 options=219b ether 00:30:48:8e:d0:69 media: Ethernet autoselect status: no carrier Starting devd. Starting Network: em1. em1: flags=8802 metric 0 mtu 1500 options=219b ether 00:30:48:8e:d0:69 media: Ethernet autoselect status: no carrier Starting Network: usbus0. Starting Network: usbus1. Starting Network: usbus2. Starting Network: usbus3. Starting Network: usbus4. add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 Creating and/or trimming log files. Starting syslogd. No core dumps found. ELF ldconfig path: /lib /usr/lib /usr/lib/compat a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Setting date via ntp. 29 Sep 09:17:20 ntpdate[1388]: step time server 192.168.5.1 offset -0.899292 sec Starting rpcbind. NFS access cache time=60 Clearing /tmp (X related). Starting statd. Updating motd:. Configuring syscons: blanktime. Starting sshd. Starting cron. Local package initialization: watchdogd. Starting inetd. Thu Sep 29 09:17:22 UTC 2011 FreeBSD/i386 (leopard3.netperf.freebsd.org) (ttyu0) login: root Sep 29 09:17:59 leopard3 login: ROOT LOGIN (root) ON ttyu0 FreeBSD 10.0-CURRENT (PHO) #0: Thu Sep 29 10:50:17 CEST 2011 Welcome to FreeBSD! Before seeking technical support, please use the following resources: o Security advisories and updated errata information for all releases are at http://www.FreeBSD.org/releases/ - always consult the ERRATA section for your release first as it's updated frequently. o The Handbook and FAQ documents are at http://www.FreeBSD.org/ and, along with the mailing lists, can be searched by going to http://www.FreeBSD.org/search/. If the doc distribution has been installed, they're also available formatted in /usr/share/doc. If you still have a question or problem, please take the output of `uname -a', along with any relevant error messages, and email it as a question to the questions@FreeBSD.org mailing list. If you are unfamiliar with FreeBSD's directory layout, please refer to the hier(7) manual page. If you are not familiar with manual pages, type `man man'. You may also use sysinstall(8) to re-enter the installation and configuration utility. Edit /etc/motd to change this login announcement. leopard3# ~test/keepalive.sh Thu Sep 29 11:18:05 CEST 2011 11:18AM up 59 secs, 1 user, load averages: 1.25, 0.54, 0.21 Sep 29 11:18:14 leopard3 su: test to root on /dev/pts/0 lock order reversal: 1st 0xdf317680 bufwait (bufwait) @ /mnt/pho/src/sys/kern/vfs_bio.c:2658 2nd 0xc63db600 dirhash (dirhash) @ /mnt/pho/src/sys/ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper(c0f08930,7366752f,7366752f,7366752f,7269645f,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a4532b,c0f0c38b,c5947370,c594b748,df7ea878,...) at kdb_backtrace+0x2a _witness_debugger(c0f0c38b,c63db600,c0f3c1fe,c594b748,c0f3be7f,...) at _witness_debugger+0x25 witness_checkorder(c63db600,9,c0f3be7f,11c,0,...) at witness_checkorder+0x839 _sx_xlock(c63db600,0,c0f3be7f,11c,c63f1bc8,...) at _sx_xlock+0x85 ufsdirhash_acquire(df317620,c63f1bc8,df7ea9f4,e041c028,df7ea948,...) at ufsdirhash_acquire+0x48 ufsdirhash_add(c63f1bc8,df7ea9f4,28,df7ea934,df7ea938,...) at ufsdirhash_add+0x13 ufs_direnter(c646bc3c,c6572c3c,df7ea9f4,df7eab84,df317e40,...) at ufs_direnter+0x789 ufs_mkdir(df7eac14,c0f5a576,0,0,df7eabac,...) at ufs_mkdir+0xa36 VOP_MKDIR_APV(c1054420,df7eac14,df7eab84,df7eabac,0,...) at VOP_MKDIR_APV+0xc5 kern_mkdirat(c63188a0,ffffff9c,804c7a8,0,41ed,...) at kern_mkdirat+0x291 kern_mkdir(c63188a0,804c7a8,0,41ed,df7ead1c,...) at kern_mkdir+0x2e sys_mkdir(c63188a0,df7eacec,c0f53da2,c0f0cf26,206,...) at sys_mkdir+0x29 syscall(df7ead28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (136, FreeBSD ELF32, sys_mkdir), eip = 0x28175b33, esp = 0xbfbfe81c, ebp = 0xbfbfe998 --- lock order reversal: 1st 0xc646bc94 ufs (ufs) @ /mnt/pho/src/sys/kern/vfs_subr.c:2134 2nd 0xdf317680 bufwait (bufwait) @ /mnt/pho/src/sys/ufs/ffs/ffs_vnops.c:260 3rd 0xc6572c94 ufs (ufs) @ /mnt/pho/src/sys/kern/vfs_subr.c:2134 KDB: stack backtrace: db_trace_self_wrapper(c0f08930,75735f73,632e7262,3331323a,6f000a34,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a4532b,c0f0c3a4,c5947370,c594b5a8,df7ea3d8,...) at kdb_backtrace+0x2a _witness_debugger(c0f0c3a4,c6572c94,c0efb32c,c594b5a8,c0f141e9,...) at _witness_debugger+0x25 witness_checkorder(c6572c94,9,c0f141e9,856,0,...) at witness_checkorder+0x839 __lockmgr_args(c6572c94,80100,c6572d00,0,0,...) at __lockmgr_args+0x834 ffs_lock(df7ea4fc,c0a5674b,c0f13492,80100,c6572c3c,...) at ffs_lock+0xa1 VOP_LOCK1_APV(c1054420,df7ea4fc,c6318950,c1064c20,c6572c3c,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c6572c3c,80100,c0f141e9,856,4,...) at _vn_lock+0x78 vget(c6572c3c,80100,c63188a0,50,0,...) at vget+0xbb vfs_hash_get(c5ea0000,39800,80000,c63188a0,df7ea640,...) at vfs_hash_get+0xed ffs_vgetf(c5ea0000,39800,80000,df7ea640,1,...) at ffs_vgetf+0x49 softdep_sync_buf(c646bc3c,df317620,1,106,0,...) at softdep_sync_buf+0x4a3 ffs_syncvnode(c646bc3c,1,bfbfe998,bfbfe998,0,...) at ffs_syncvnode+0x24c ffs_truncate(c646bc3c,400,0,880,c5eab680,...) at ffs_truncate+0x817 ufs_direnter(c646bc3c,c6572c3c,df7ea9f4,df7eab84,df317e40,...) at ufs_direnter+0x971 ufs_mkdir(df7eac14,c0f5a576,0,0,df7eabac,...) at ufs_mkdir+0xa36 VOP_MKDIR_APV(c1054420,df7eac14,df7eab84,df7eabac,0,...) at VOP_MKDIR_APV+0xc5 kern_mkdirat(c63188a0,ffffff9c,804c7a8,0,41ed,...) at kern_mkdirat+0x291 kern_mkdir(c63188a0,804c7a8,0,41ed,df7ead1c,...) at kern_mkdir+0x2e sys_mkdir(c63188a0,df7eacec,c0f53da2,c0f0cf26,206,...) at sys_mkdir+0x29 syscall(df7ead28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (136, FreeBSD ELF32, sys_mkdir), eip = 0x28175b33, esp = 0xbfbfe81c, ebp = 0xbfbfe998 --- lock order reversal: 1st 0xc643631c snaplk (snaplk) @ /mnt/pho/src/sys/ufs/ffs/ffs_snapshot.c:2096 2nd 0xc733bdf0 ufs (ufs) @ /mnt/pho/src/sys/ufs/ffs/ffs_snapshot.c:2097 KDB: stack backtrace: db_trace_self_wrapper(c0f08930,7366662f,616e735f,6f687370,3a632e74,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a4532b,c0f0c38b,c594b7b0,c594b5a8,eddaba3c,...) at kdb_backtrace+0x2a _witness_debugger(c0f0c38b,c733bdf0,c0efb32c,c594b5a8,c0f37d9e,...) at _witness_debugger+0x25 witness_checkorder(c733bdf0,9,c0f37d9e,831,0,...) at witness_checkorder+0x839 __lockmgr_args(c733bdf0,80000,0,0,0,...) at __lockmgr_args+0x834 ffs_snapshot_unmount(c5ea0000,0,1,c63d98a0,c6436a00,...) at ffs_snapshot_unmount+0x151 ffs_flushfiles(c5ea0000,0,c63d98a0,eddabba4,3,...) at ffs_flushfiles+0xce softdep_flushfiles(c5ea0000,0,c63d98a0,0,7931,...) at softdep_flushfiles+0x2e ffs_unmount(c5ea0000,8000000,c0f13877,4ef,c5825000,...) at ffs_unmount+0x1af dounmount(c5ea0000,8000000,c63d98a0,474,5951f7f8,...) at dounmount+0x4a1 sys_unmount(c63d98a0,eddabcec,c,c,206,...) at sys_unmount+0x331 syscall(eddabd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280dcb3f, esp = 0xbfbfe1fc, ebp = 0xbfbfe2c8 --- lock order reversal: 1st 0xc641cc94 newnfs (newnfs) @ /mnt/pho/src/sys/kern/vfs_mount.c:1194 2nd 0xc65851b4 devfs (devfs) @ /mnt/pho/src/sys/ufs/ffs/ffs_vfsops.c:1371 KDB: stack backtrace: db_trace_self_wrapper(c0f08930,7366662f,7366662f,7366765f,2e73706f,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a4532b,c0f0c38b,c594b2d0,c594aff8,eddab9fc,...) at kdb_backtrace+0x2a _witness_debugger(c0f0c38b,c65851b4,c0ef2f6f,c594aff8,c0f3b250,...) at _witness_debugger+0x25 witness_checkorder(c65851b4,9,c0f3b250,55b,c6585220,...) at witness_checkorder+0x839 __lockmgr_args(c65851b4,80400,c6585220,0,0,...) at __lockmgr_args+0x834 vop_stdlock(eddabb1c,eddabb3c,c0c2ce18,80400,c658515c,...) at vop_stdlock+0x65 VOP_LOCK1_APV(c1029da0,eddabb1c,c733bd98,c1064c20,c658515c,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c658515c,80400,c0f3b250,55b,c6436a00,...) at _vn_lock+0x78 ffs_flushfiles(c5ea0000,2,c63d98a0,eddabba4,3,...) at ffs_flushfiles+0x11c softdep_flushfiles(c5ea0000,0,c63d98a0,0,7931,...) at softdep_flushfiles+0x2e ffs_unmount(c5ea0000,8000000,c0f13877,4ef,c5825000,...) at ffs_unmount+0x1af dounmount(c5ea0000,8000000,c63d98a0,474,5951f7f8,...) at dounmount+0x4a1 sys_unmount(c63d98a0,eddabcec,c,c,206,...) at sys_unmount+0x331 syscall(eddabd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280dcb3f, esp = 0xbfbfe1fc, ebp = 0xbfbfe2c8 --- Thu Sep 29 11:28:05 CEST 2011 11:28AM up 11 mins, 2 users, load averages: 0.00, 0.08, 0.11 fsync: giving up on dirty 0xc6b37c3c: tag devfs, type VCHR usecount 1, writecount 0, refcount 157 mountedhere 0xc6f14a00 flags () v_object 0xc6ac8880 ref 0 pages 630 lock type devfs: EXCL by thread 0xc63d65c0 (pid 1896) #0 0xc09f46ee at __lockmgr_args+0xc1e #1 0xc0a94e95 at vop_stdlock+0x65 #2 0xc0d67045 at VOP_LOCK1_APV+0xb5 #3 0xc0ab6038 at _vn_lock+0x78 #4 0xc0c47827 at softdep_flushworklist+0x47 #5 0xc0c4b1cd at ffs_sync+0x2fd #6 0xc0ab5e12 at vfs_write_suspend+0x122 #7 0xc0c2f4ad at ffs_snapshot+0xced #8 0xc0c4d6a3 at ffs_mount+0x1c13 #9 0xc0a9eb84 at vfs_donmount+0x1254 #10 0xc0a9f254 at sys_nmount+0x84 #11 0xc0d43c94 at syscall+0x2a4 #12 0xc0d2d0f1 at Xint0x80_syscall+0x21 dev md5a panic: ffs_update: bad link cnt cpuid = 1 KDB: enter: panic [ thread pid 17 tid 100064 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> run pho db:0:pho> bt Tracing pid 17 tid 100064 td 0xc5e5d000 kdb_enter(c0f0495c,c0f0495c,c0f37c9e,df76bba8,1,...) at kdb_enter+0x3a panic(c0f37c9e,16f960,0,4000,0,...) at panic+0x134 ffs_update(c6b3a000,0,c0f38380,2447,1,...) at ffs_update+0x1e5 handle_workitem_remove(0,df76bc6c,2,6b9,df76bc70,...) at handle_workitem_remove+0x478 process_worklist_item(c121dc70,0,c0f38380,5e8,c5e5d000,...) at process_worklist_item+0x2b4 softdep_process_worklist(c65a45a8,0,c0f38380,554,3e8,...) at softdep_process_worklist+0x88 softdep_flush(0,df76bd28,c0eff150,3e5,c5e57588,...) at softdep_flush+0x2b0 fork_exit(c0c479f0,0,df76bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf76bd60, ebp = 0 --- db:0:bt> show allpcpu Current CPU: 1 cpuid = 0 dynamic pcpu = 0x7bfc80 curthread = 0xc599e8a0: pid 11 "idle: cpu0" curpcb = 0xc5611d80 fpcurthread = none idlethread = 0xc599e8a0: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 dynamic pcpu = 0x45a1c80 curthread = 0xc5e5d000: pid 17 "softdepflush" curpcb = 0xdf76bd80 fpcurthread = none idlethread = 0xc599e5c0: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: db:0:allpcpu> show alllocks Process 1897 (sh) thread 0xc631a2e0 (100087) exclusive sleep mutex VM OBJECT (UMA zone) r = 0 (0xc19ca408) locked @ /mnt/pho/src/sys/vm/uma_core.c:2013 exclusive sx user map (user map) r = 0 (0xc59a5c10) locked @ /mnt/pho/src/sys/vm/vm_map.c:3669 Process 1769 (sshd) thread 0xc63185c0 (100096) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xc6568d8c) locked @ /mnt/pho/src/sys/kern/uipc_sockbuf.c:148 Process 17 (softdepflush) thread 0xc5e5d000 (100064) exclusive lockmgr bufwait (bufwait) r = 0 (0xdf3861c0) locked @ /mnt/pho/src/sys/kern/vfs_bio.c:2658 exclusive lockmgr ufs (ufs) r = 0 (0xc6b3a058) locked @ /mnt/pho/src/sys/kern/vfs_subr.c:2134 db:0:alllocks> show lockedvnods Locked vnodes 0xc6b3a000: 0xc6b3a000: tag ufs, type VDIR tag ufs, type VDIR usecount 1, writecount 0, refcount 1 mountedhere 0 usecount 1, writecount 0, refcount 1 mountedhere 0 flags () flags () v_object 0xc659fc38 ref 0 pages 0 v_object 0xc659fc38 ref 0 pages 0 lock type ufs: EXCL by thread 0xc5e5d000 (pid 17) lock type ufs: EXCL by thread 0xc5e5d000 (pid 17) #0 0xc09f46ee at __lockmgr_args+0xc1e #0 0xc09f46ee at __lockmgr_args+0xc1e #1 0xc0c50d21 at ffs_lock+0xa1 #1 0xc0c50d21 at ffs_lock+0xa1 #2 0xc0d67045 at VOP_LOCK1_APV+0xb5 #2 0xc0d67045 at VOP_LOCK1_APV+0xb5 #3 0xc0ab6038 at _vn_lock+0x78 #3 0xc0ab6038 at _vn_lock+0x78 #4 0xc0aa895b at vget+0xbb #4 0xc0aa895b at vget+0xbb #5 0xc0a991dd at vfs_hash_get+0xed #5 0xc0a991dd at vfs_hash_get+0xed #6 0xc0c4a209 at ffs_vgetf+0x49 #6 0xc0c4a209 at ffs_vgetf+0x49 #7 0xc0c36f90 at handle_workitem_remove+0x60 #7 0xc0c36f90 at handle_workitem_remove+0x60 #8 0xc0c36814 at process_worklist_item+0x2b4 #8 0xc0c36814 at process_worklist_item+0x2b4 #9 0xc0c476c8 at softdep_process_worklist+0x88 #9 0xc0c476c8 at softdep_process_worklist+0x88 #10 0xc0c47ca0 at softdep_flush+0x2b0 #10 0xc0c47ca0 at softdep_flush+0x2b0 #11 0xc09df698 at fork_exit+0xb8 #11 0xc09df698 at fork_exit+0xb8 #12 0xc0d2d104 at fork_trampoline+0x8 #12 0xc0d2d104 at fork_trampoline+0x8 ino 94208, on dev md5a ino 94208, on dev md5a db:0:lockedvnods> show mount 0xc5ea0b50 192.168.5.1:/zoo/leopard3 on / (nfs) 0xc5ea1000 devfs on /dev (devfs) 0xc5ea087c /dev/ada0d on /work (ufs) 0xc5ea05a8 /dev/md0 on /var (ufs) 0xc5ea02d4 /dev/md1 on /tmp (ufs) 0xc65a45a8 /dev/md5a on /mnt (ufs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 1897 1873 1873 0 R+ sh 1884 0 0 0 DL mdwait 0xc6326000 [md5] 1873 1780 1873 0 R+ sh 1843 1765 1765 0 S+ nanslp 0xc10a13e4 sleep 1780 1776 1780 0 S+ wait 0xc6234588 sh 1776 1775 1776 0 S+ pause 0xc62df8a4 csh 1775 1774 1775 1001 S+ wait 0xc62dfb10 su 1774 1773 1774 1001 Ss+ wait 0xc62de000 sh 1773 1769 1769 1001 S select 0xc5e9e224 sshd 1769 1665 1769 0 Ss sbwait 0xc6568db8 sshd 1765 1761 1765 0 S+ wait 0xc5b8bb10 sh 1761 1760 1761 0 S+ pause 0xc63f631c csh 1760 1 1760 0 Ss+ wait 0xc6491588 login 1759 1 1759 0 Ss+ ttyin 0xc5a3fe70 getty 1758 1 1758 0 Ss+ ttyin 0xc5c39070 getty 1757 1 1757 0 Ss+ ttyin 0xc5c39270 getty 1756 1 1756 0 Ss+ ttyin 0xc5c39470 getty 1755 1 1755 0 Ss+ ttyin 0xc5a3e070 getty 1754 1 1754 0 Ss+ ttyin 0xc5a3e270 getty 1753 1 1753 0 Ss+ ttyin 0xc5a3e470 getty 1752 1 1752 0 Ss+ ttyin 0xc5a3e670 getty 1714 1 1714 0 Ss select 0xc5e9e064 inetd 1686 1 1686 0 Ss nanslp 0xc10a13e4 watchdogd 1676 1 1676 0 Ss nanslp 0xc10a13e4 cron 1665 1 1665 0 Ss select 0xc5e9fd24 sshd 1524 1 1524 0 Ss select 0xc642d064 rpc.statd 1451 0 0 0 DL mdwait 0xc6325000 [md1] 1396 1 1396 0 Ss select 0xc5e9e764 rpcbind 1366 1 1366 0 Ss select 0xc5e9eb64 syslogd 1187 1 1187 0 Ss select 0xc5e9f064 devd 100 0 0 0 DL mdwait 0xc5c1d800 [md0] 17 0 0 0 RL CPU 1 [softdepflush] 16 0 0 0 DL vlruwt 0xc5e5784c [vnlru] 9 0 0 0 DL syncer 0xc12175b4 [syncer] 8 0 0 0 DL psleep 0xc1217248 [bufdaemon] 7 0 0 0 DL pgzero 0xc121fb5c [pagezero] 6 0 0 0 DL psleep 0xc121f748 [vmdaemon] 5 0 0 0 DL psleep 0xc121f710 [pagedaemon] 4 0 0 0 DL ccb_scan 0xc10660d4 [xpt_thrd] 3 0 0 0 DL waiting_ 0xc12190d8 [sctp_iterator] 2 0 0 0 DL - 0xc5b25a3c [fdc0] 15 0 0 0 DL (threaded) [usb] 100049 D - 0xc5bd5d34 [usbus4] 100048 D - 0xc5bd5d04 [usbus4] 100047 D - 0xc5bd5cd4 [usbus4] 100046 D - 0xc5bd5ca4 [usbus4] 100045 D - 0xc5bbddac [usbus3] 100044 D - 0xc5bbdd7c [usbus3] 100043 D - 0xc5bbdd4c [usbus3] 100042 D - 0xc5bbdd1c [usbus3] 100040 D - 0xc5babdac [usbus2] 100039 D - 0xc5babd7c [usbus2] 100038 D - 0xc5babd4c [usbus2] 100037 D - 0xc5babd1c [usbus2] 100035 D - 0xc5b94dac [usbus1] 100034 D - 0xc5b94d7c [usbus1] 100033 D - 0xc5b94d4c [usbus1] 100032 D - 0xc5b94d1c [usbus1] 100030 D - 0xc5b7bdac [usbus0] 100029 D - 0xc5b7bd7c [usbus0] 100028 D - 0xc5b7bd4c [usbus0] 100027 D - 0xc5b7bd1c [usbus0] 14 0 0 0 DL - 0xc10a01e4 [yarrow] 13 0 0 0 DL (threaded) [geom] 100011 D - 0xc109de08 [g_down] 100010 D - 0xc109de04 [g_up] 100009 D - 0xc109ddfc [g_event] 12 0 0 0 WL (threaded) [intr] 100054 I [irq1: atkbd0] 100052 I [swi0: uart uart] 100051 I [irq258: ahci0] 100050 I [irq14: ata0] 100041 I [irq16: uhci3] 100036 I [irq18: uhci2] 100031 I [irq19: uhci1] 100026 I [irq23: uhci0 ehci0] 100022 I [swi5: +] 100021 I [swi2: cambio] 100015 I [swi6: task queue] 100014 I [swi6: Giant taskq] 100008 I [swi4: clock] 100007 I [swi4: clock] 100006 I [swi3: vm] 100005 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100004 CanRun [idle: cpu1] 100003 Run CPU 0 [idle: cpu0] 1 0 1 0 SLs wait 0xc599cb10 [init] 10 0 0 0 DL audit_wo 0xc121d3e0 [audit] 0 0 0 0 DLs (threaded) [kernel] 100065 D - 0xc5c49400 [mca taskq] 100055 D - 0xc10a01e4 [deadlkres] 100025 D - 0xc5b5f900 [em1 taskq] 100024 D - 0xc5b42680 [em0 taskq] 100023 D - 0xc5977580 [thread taskq] 100020 D - 0xc59776c0 [kqueue taskq] 100019 D - 0xc5977700 [acpi_task_2] 100018 D - 0xc5977700 [acpi_task_1] 100017 D - 0xc5977700 [acpi_task_0] 100016 D - 0xc5977740 [ffs_trim taskq] 100012 D - 0xc5977bc0 [firmware taskq] 100000 D sched 0xc109dee0 [swapper] db:0:ps> allt Tracing command sh pid 1897 tid 100087 td 0xc631a2e0 sched_switch(c631a2e0,0,207,18c,15e9d5b0,...) at sched_switch+0x3bc mi_switch(207,0,c0f0ac32,d4,c631a2e0,...) at mi_switch+0x1eb ast(df7c9d28) at ast+0x2b6 doreti_ast() at doreti_ast+0x17 Tracing command md5 pid 1884 tid 100100 td 0xc63d8b80 sched_switch(c63d8b80,0,104,191,150b2818,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c63d8b80,0,c0f0a2a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c6326000,5c,c0ea7961,0,0,...) at sleepq_wait+0x63 _sleep(c6326000,c6326020,25c,c0ea7961,0,...) at _sleep+0x372 md_kthread(c6326000,df7fbd28,c0eff150,3e5,c63f6588,...) at md_kthread+0x11a fork_exit(c0734300,c6326000,df7fbd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf7fbd60, ebp = 0 --- Tracing command sh pid 1873 tid 100095 td 0xc63188a0 sched_switch(c63188a0,0,207,18c,15e64880,...) at sched_switch+0x3bc mi_switch(207,0,c0f0ac32,d4,c0d5cf96,...) at mi_switch+0x1eb ast(df7ead28) at ast+0x2b6 doreti_ast() at doreti_ast+0x17 Tracing command sleep pid 1843 tid 100125 td 0xc64972e0 sched_switch(c64972e0,0,104,191,e75d9460,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c64972e0,0,c0f0a2a0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(927c1,c0a4bb00,c64972e0,1,100,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c10a13e4,6c,c0f05edf,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c10a13e4,0,16c,c0f05edf,927c1,...) at _sleep+0x328 kern_nanosleep(c64972e0,edda2c50,edda2c58,258,0,...) at kern_nanosleep+0xc1 sys_nanosleep(c64972e0,edda2cec,c0f53da2,c0f0d8d4,246,...) at sys_nanosleep+0x6f syscall(edda2d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x281660b7, esp = 0xbfbfec2c, ebp = 0xbfbfec78 --- Tracing command sh pid 1780 tid 100069 td 0xc62385c0 sched_switch(c62385c0,0,104,191,4b18ee30,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c62385c0,0,c0f0a2a0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f0a2a0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c6234588,6c,c0f0d07f,100,0,...) at sleepq_wait_sig+0x17 _sleep(c6234588,c6234610,16c,c0f0d07f,0,...) at _sleep+0x35c kern_wait(c62385c0,ffffffff,df77dc60,2,0,...) at kern_wait+0x626 sys_wait4(c62385c0,df77dcec,c0f53da2,c0f0d05e,212,...) at sys_wait4+0x3b syscall(df77dd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28161ffb, esp = 0xbfbfe9ac, ebp = 0xbfbfe9c8 --- Tracing command csh pid 1776 tid 100080 td 0xc5e5db80 sched_switch(c5e5db80,0,104,191,558c6de8,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,74,...) at mi_switch+0x1eb sleepq_switch(c5e5db80,0,c0f0a2a0,1a7,74,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f0a2a0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c62df8a4,74,c0d936ce,100,0,...) at sleepq_wait_sig+0x17 _sleep(c62df8a4,c62df8d4,174,c0d936ce,0,...) at _sleep+0x35c kern_sigsuspend(c5e5db80,0,0,0,0,...) at kern_sigsuspend+0xba sys_sigsuspend(c5e5db80,df7aacec,c0f53da2,c0f0cf6e,202,...) at sys_sigsuspend+0x4d syscall(df7aad28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, sys_write), eip = 0x2817d8fb, esp = 0xbfbfe74c, ebp = 0xbfbfe7b8 --- Tracing command su pid 1775 tid 100078 td 0xc6235000 sched_switch(c6235000,0,104,191,10a7d540,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c6235000,0,c0f0a2a0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f0a2a0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c62dfb10,6c,c0f0d07f,100,0,...) at sleepq_wait_sig+0x17 _sleep(c62dfb10,c62dfb98,16c,c0f0d07f,0,...) at _sleep+0x35c kern_wait(c6235000,6f0,df7a1c60,2,0,...) at kern_wait+0x626 sys_wait4(c6235000,df7a1cec,c,c,212,...) at sys_wait4+0x3b syscall(df7a1d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x2812cffb, esp = 0xbfbfe77c, ebp = 0xbfbfe798 --- Tracing command sh pid 1774 tid 100088 td 0xc631a000 sched_switch(c631a000,0,104,191,5aec0c8,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c631a000,0,c0f0a2a0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f0a2a0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c62de000,6c,c0f0d07f,100,0,...) at sleepq_wait_sig+0x17 _sleep(c62de000,c62de088,16c,c0f0d07f,0,...) at _sleep+0x35c kern_wait(c631a000,ffffffff,df7cdc60,2,0,...) at kern_wait+0x626 sys_wait4(c631a000,df7cdcec,c,c,212,...) at sys_wait4+0x3b syscall(df7cdd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28161ffb, esp = 0xbfbfeaac, ebp = 0xbfbfeac8 --- Tracing command sshd pid 1773 tid 100085 td 0xc631a8a0 sched_switch(c631a8a0,0,104,191,15e15120,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c631a8a0,0,c0f0a2a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c09fc5aa,c5e9e210,0,c0f02e87,c631a8a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5e9e224,0,df7c0a68,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5e9e224,c5e9e210,c0f0c8da,668,c62440e0,...) at _cv_wait_sig+0x243 seltdwait(c62440e0,58,c599ac00,c631a8a0,c0f2d8e2,...) at seltdwait+0xa2 kern_select(c631a8a0,a,288060b8,288060cc,0,0,20,c62de84c,0) at kern_select+0x504 sys_select(c631a8a0,df7c0cec,c0f53da2,c0ed484f,206,...) at sys_select+0x66 syscall(df7c0d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x284b0d43, esp = 0xbfbfe19c, ebp = 0xbfbfe1e8 --- Tracing command sshd pid 1769 tid 100096 td 0xc63185c0 sched_switch(c63185c0,0,104,191,bf47b0e8,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,68,...) at mi_switch+0x1eb sleepq_switch(c63185c0,0,c0f0a2a0,1a7,68,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f0a2a0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c6568db8,68,c0f107b1,100,0,...) at sleepq_wait_sig+0x17 _sleep(c6568db8,c6568d78,168,c0f107b1,0) at _sleep+0x35c sbwait(c6568d54,4,c0f10889,5ee,c6568d78,...) at sbwait+0x76 soreceive_generic(c6568d00,0,df7edc44,0,0,...) at soreceive_generic+0x320 soreceive(c6568d00,0,df7edc44,0,0,0) at soreceive+0x38 soo_read(c62440a8,df7edc44,c5eab700,0,c63185c0,...) at soo_read+0x4e dofileread(df7edc44,ffffffff,ffffffff,0,df7edc20,...) at dofileread+0x9e kern_readv(c63185c0,5,df7edc44,df7edc64,1,...) at kern_readv+0x68 sys_read(c63185c0,df7edcec,c0f53da2,c0ef5741,286,...) at sys_read+0x4f syscall(df7edd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x284b0dc3, esp = 0xbfbfdd6c, ebp = 0xbfbfdda8 --- Tracing command sh pid 1765 tid 100074 td 0xc5e5c2e0 sched_switch(c5e5c2e0,0,104,191,e7273330,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c5e5c2e0,0,c0f0a2a0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f0a2a0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5b8bb10,6c,c0f0d07f,100,0,...) at sleepq_wait_sig+0x17 _sleep(c5b8bb10,c5b8bb98,16c,c0f0d07f,0,...) at _sleep+0x35c kern_wait(c5e5c2e0,ffffffff,df791c60,2,0,...) at kern_wait+0x626 sys_wait4(c5e5c2e0,df791cec,c0f53da2,c0f0d05e,212,...) at sys_wait4+0x3b syscall(df791d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28161ffb, esp = 0xbfbfe9dc, ebp = 0xbfbfe9f8 --- Tracing command csh pid 1761 tid 100101 td 0xc63d88a0 sched_switch(c63d88a0,0,104,191,939fbc28,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,74,...) at mi_switch+0x1eb sleepq_switch(c63d88a0,0,c0f0a2a0,1a7,74,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f0a2a0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c63f631c,74,c0d936ce,100,0,...) at sleepq_wait_sig+0x17 _sleep(c63f631c,c63f634c,174,c0d936ce,0,...) at _sleep+0x35c kern_sigsuspend(c63d88a0,0,0,0,0,...) at kern_sigsuspend+0xba sys_sigsuspend(c63d88a0,df7fecec,c0f53da2,c0f0cf6e,202,...) at sys_sigsuspend+0x4d syscall(df7fed28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, sys_write), eip = 0x2817d8fb, esp = 0xbfbfe7ec, ebp = 0xbfbfe858 --- Tracing command login pid 1760 tid 100120 td 0xc64982e0 sched_switch(c64982e0,0,104,191,8129e20,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c64982e0,0,c0f0a2a0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f0a2a0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c6491588,6c,c0f0d07f,100,0,...) at sleepq_wait_sig+0x17 _sleep(c6491588,c6491610,16c,c0f0d07f,0,...) at _sleep+0x35c kern_wait(c64982e0,6e1,edd7dc60,0,0,...) at kern_wait+0x626 sys_wait4(c64982e0,edd7dcec,c,c,212,...) at sys_wait4+0x3b syscall(edd7dd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28125ffb, esp = 0xbfbfecec, ebp = 0xbfbfed08 --- Tracing command getty pid 1759 tid 100115 td 0xc64992e0 sched_switch(c64992e0,0,104,191,65637c98,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c64992e0,0,c0f0a2a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f0a2a0,160,0,c64992e0,c64992e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5a3fe70,0,c0f0ef90,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5a3fe70,c109fad0,c0f0e950,527,0,...) at _cv_wait_sig+0x243 tty_wait(c5a3fe00,c5a3fe70,edd6ec44,1,0,...) at tty_wait+0x71 ttydisc_read(c5a3fe00,edd6ec44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c5b02400,edd6ec44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c6257038,edd6ec44,c5999e00,0,c64992e0,...) at devfs_read_f+0x7e dofileread(edd6ec44,ffffffff,ffffffff,0,edd6ec20,...) at dofileread+0x9e kern_readv(c64992e0,0,edd6ec44,edd6ec64,1,...) at kern_readv+0x68 sys_read(c64992e0,edd6ecec,c0f53da2,c0f25f88,246,...) at sys_read+0x4f syscall(edd6ed28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1758 tid 100089 td 0xc6319b80 sched_switch(c6319b80,0,104,191,65f90c30,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c6319b80,0,c0f0a2a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f0a2a0,160,0,c6319b80,c6319b80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5c39070,0,c0f0ef90,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5c39070,c109fad0,c0f0e950,527,0,...) at _cv_wait_sig+0x243 tty_wait(c5c39000,c5c39070,df7d7c44,1,0,...) at tty_wait+0x71 ttydisc_read(c5c39000,df7d7c44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c5c16300,df7d7c44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c62575b0,df7d7c44,c5999e00,0,c6319b80,...) at devfs_read_f+0x7e dofileread(df7d7c44,ffffffff,ffffffff,0,df7d7c20,...) at dofileread+0x9e kern_readv(c6319b80,0,df7d7c44,df7d7c64,1,...) at kern_readv+0x68 sys_read(c6319b80,df7d7cec,c0f53da2,c0f25f88,246,...) at sys_read+0x4f syscall(df7d7d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1757 tid 100083 td 0xc5e5d2e0 sched_switch(c5e5d2e0,0,104,191,6582c6e8,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5e5d2e0,0,c0f0a2a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f0a2a0,160,0,c5e5d2e0,c5e5d2e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5c39270,0,c0f0ef90,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5c39270,c109fad0,c0f0e950,527,0,...) at _cv_wait_sig+0x243 tty_wait(c5c39200,c5c39270,df7b7c44,1,0,...) at tty_wait+0x71 ttydisc_read(c5c39200,df7b7c44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c5c16200,df7b7c44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c6245738,df7b7c44,c5999e00,0,c5e5d2e0,...) at devfs_read_f+0x7e dofileread(df7b7c44,ffffffff,ffffffff,0,df7b7c20,...) at dofileread+0x9e kern_readv(c5e5d2e0,0,df7b7c44,df7b7c64,1,...) at kern_readv+0x68 sys_read(c5e5d2e0,df7b7cec,c0f53da2,c0f25f88,246,...) at sys_read+0x4f syscall(df7b7d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1756 tid 100109 td 0xc63d6000 sched_switch(c63d6000,0,104,191,65f80498,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c63d6000,0,c0f0a2a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f0a2a0,160,0,c63d6000,c63d6000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5c39470,0,c0f0ef90,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5c39470,c109fad0,c0f0e950,527,0,...) at _cv_wait_sig+0x243 tty_wait(c5c39400,c5c39470,edd34c44,1,0,...) at tty_wait+0x71 ttydisc_read(c5c39400,edd34c44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c5c16100,edd34c44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c6244188,edd34c44,c5999e00,0,c63d6000,...) at devfs_read_f+0x7e dofileread(edd34c44,ffffffff,ffffffff,0,edd34c20,...) at dofileread+0x9e kern_readv(c63d6000,0,edd34c44,edd34c64,1,...) at kern_readv+0x68 sys_read(c63d6000,edd34cec,c0f53da2,c0f25f88,246,...) at sys_read+0x4f syscall(edd34d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1755 tid 100113 td 0xc63d52e0 sched_switch(c63d52e0,0,104,191,65e4f5c0,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c63d52e0,0,c0f0a2a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f0a2a0,160,0,c63d52e0,c63d52e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5a3e070,0,c0f0ef90,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5a3e070,c109fad0,c0f0e950,527,0,...) at _cv_wait_sig+0x243 tty_wait(c5a3e000,c5a3e070,edd68c44,1,0,...) at tty_wait+0x71 ttydisc_read(c5a3e000,edd68c44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c5c16000,edd68c44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c62446c8,edd68c44,c5999e00,0,c63d52e0,...) at devfs_read_f+0x7e dofileread(edd68c44,ffffffff,ffffffff,0,edd68c20,...) at dofileread+0x9e kern_readv(c63d52e0,0,edd68c44,edd68c64,1,...) at kern_readv+0x68 sys_read(c63d52e0,edd68cec,c0f53da2,c0f25f88,246,...) at sys_read+0x4f syscall(edd68d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1754 tid 100114 td 0xc63d5000 sched_switch(c63d5000,0,104,191,65c3c7f0,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c63d5000,0,c0f0a2a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f0a2a0,160,0,c63d5000,c63d5000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5a3e270,0,c0f0ef90,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5a3e270,c109fad0,c0f0e950,527,0,...) at _cv_wait_sig+0x243 tty_wait(c5a3e200,c5a3e270,edd6bc44,1,0,...) at tty_wait+0x71 ttydisc_read(c5a3e200,edd6bc44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c5b02e00,edd6bc44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c62572a0,edd6bc44,c5999e00,0,c63d5000,...) at devfs_read_f+0x7e dofileread(edd6bc44,ffffffff,ffffffff,0,edd6bc20,...) at dofileread+0x9e kern_readv(c63d5000,0,edd6bc44,edd6bc64,1,...) at kern_readv+0x68 sys_read(c63d5000,edd6bcec,c0f53da2,c0f25f88,246,...) at sys_read+0x4f syscall(edd6bd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1753 tid 100072 td 0xc6235b80 sched_switch(c6235b80,0,104,191,65d63148,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c6235b80,0,c0f0a2a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f0a2a0,160,0,c6235b80,c6235b80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5a3e470,0,c0f0ef90,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5a3e470,c109fad0,c0f0e950,527,0,...) at _cv_wait_sig+0x243 tty_wait(c5a3e400,c5a3e470,df789c44,1,0,...) at tty_wait+0x71 ttydisc_read(c5a3e400,df789c44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c5b02d00,df789c44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c6244c08,df789c44,c5999e00,0,c6235b80,...) at devfs_read_f+0x7e dofileread(df789c44,ffffffff,ffffffff,0,df789c20,...) at dofileread+0x9e kern_readv(c6235b80,0,df789c44,df789c64,1,...) at kern_readv+0x68 sys_read(c6235b80,df789cec,c0f53da2,c0f25f88,246,...) at sys_read+0x4f syscall(df789d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1752 tid 100116 td 0xc6499000 sched_switch(c6499000,0,104,191,65acb778,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c6499000,0,c0f0a2a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f0a2a0,160,0,c6499000,c6499000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5a3e670,0,c0f0ef90,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5a3e670,c109fad0,c0f0e950,527,0,...) at _cv_wait_sig+0x243 tty_wait(c5a3e600,c5a3e670,edd71c44,1,0,...) at tty_wait+0x71 ttydisc_read(c5a3e600,edd71c44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c5b02c00,edd71c44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c6244000,edd71c44,c5999e00,0,c6499000,...) at devfs_read_f+0x7e dofileread(edd71c44,ffffffff,ffffffff,0,edd71c20,...) at dofileread+0x9e kern_readv(c6499000,0,edd71c44,edd71c64,1,...) at kern_readv+0x68 sys_read(c6499000,edd71cec,c0f53da2,c0f25f88,246,...) at sys_read+0x4f syscall(edd71d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command inetd pid 1714 tid 100093 td 0xc6319000 sched_switch(c6319000,0,104,191,4f147d20,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c6319000,0,c0f0a2a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c09fc5aa,c5e9e050,0,c0f02e87,c6319000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5e9e064,0,df7e3a68,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5e9e064,c5e9e050,c0f0c8da,668,c6257b28,...) at _cv_wait_sig+0x243 seltdwait(c6257b28,58,c5999e00,c6319000,c0a567ec,...) at seltdwait+0xa2 kern_select(c6319000,8,bfbfdfa0,0,0,0,20,c63362c4,4) at kern_select+0x504 sys_select(c6319000,df7e3cec,c0f53da2,c0f0d0cb,202,...) at sys_select+0x66 syscall(df7e3d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281a8d43, esp = 0xbfbfdf5c, ebp = 0xbfbfede8 --- Tracing command watchdogd pid 1686 tid 100084 td 0xc631ab80 sched_switch(c631ab80,0,104,191,844f41a8,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c631ab80,0,c0f0a2a0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(ea61,c0a4bb00,c631ab80,1,100,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c10a13e4,6c,c0f05edf,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c10a13e4,0,16c,c0f05edf,ea61,...) at _sleep+0x328 kern_nanosleep(c631ab80,df7bbc50,df7bbc58,3c,0,...) at kern_nanosleep+0xc1 sys_nanosleep(c631ab80,df7bbcec,c0f53da2,c0f25f88,206,...) at sys_nanosleep+0x6f syscall(df7bbd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x281920b7, esp = 0xbfbfecbc, ebp = 0xbfbfece8 --- Tracing command cron pid 1676 tid 100070 td 0xc62382e0 sched_switch(c62382e0,0,104,191,1302400,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c62382e0,0,c0f0a2a0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(ea61,c0a4bb00,c62382e0,1,100,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c10a13e4,6c,c0f05edf,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c10a13e4,0,16c,c0f05edf,ea61,...) at _sleep+0x328 kern_nanosleep(c62382e0,df781c50,df781c58,3c,0,...) at kern_nanosleep+0xc1 sys_nanosleep(c62382e0,df781cec,c0f53da2,c0f0d52c,206,...) at sys_nanosleep+0x6f syscall(df781d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x281850b7, esp = 0xbfbfec4c, ebp = 0xbfbfec78 --- Tracing command sshd pid 1665 tid 100108 td 0xc63d62e0 sched_switch(c63d62e0,0,104,191,b12d6ab8,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c63d62e0,0,c0f0a2a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c09fc5aa,c5e9fd10,0,c0f02e87,c63d62e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5e9fd24,0,edd31a68,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5e9fd24,c5e9fd10,c0f0c8da,668,c6244268,...) at _cv_wait_sig+0x243 seltdwait(c6244268,58,c5eab800,c63d62e0,28811000,...) at seltdwait+0xa2 kern_select(c63d62e0,5,2880c0b0,0,0,0,20,c6337b10,a) at kern_select+0x504 sys_select(c63d62e0,edd31cec,c,c,202,...) at sys_select+0x66 syscall(edd31d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x284b0d43, esp = 0xbfbfe25c, ebp = 0xbfbfedf8 --- Tracing command rpc.statd pid 1524 tid 100104 td 0xc63d8000 sched_switch(c63d8000,0,104,191,869d4068,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c63d8000,0,c0f0a2a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(edd25a30,c09fc5aa,c642d050,0,c63d8000,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c642d064,0,edd25a68,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c642d064,c642d050,7531,668,c62572d8,...) at _cv_timedwait_sig+0x252 seltdwait(edd25c14,edd25c1c,c5999e00,c63d8000,28804000,...) at seltdwait+0x8a kern_select(c63d8000,8,bfbfed14,0,0,edd25c5c,20,1e,0) at kern_select+0x504 sys_select(c63d8000,edd25cec,c0f53da2,c0f0d1bc,202,...) at sys_select+0x66 syscall(edd25d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x2818cd43, esp = 0xbfbfec6c, ebp = 0xbfbfeda8 --- Tracing command md1 pid 1451 tid 100071 td 0xc6238000 sched_switch(c6238000,0,104,191,efe02488,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c6238000,0,c0f0a2a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c6325000,5c,c0ea7961,0,0,...) at sleepq_wait+0x63 _sleep(c6325000,c6325020,25c,c0ea7961,0,...) at _sleep+0x372 md_kthread(c6325000,df785d28,c0eff150,3e5,c6234000,...) at md_kthread+0x11a fork_exit(c0734300,c6325000,df785d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf785d60, ebp = 0 --- Tracing command rpcbind pid 1396 tid 100086 td 0xc631a5c0 sched_switch(c631a5c0,0,104,191,865c83a8,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c631a5c0,0,c0f0a2a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(df7c4a88,c09fc5aa,c5e9e750,0,c631a5c0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c5e9e764,0,df7c4ac0,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c5e9e764,c5e9e750,7531,668,df7c4b78,...) at _cv_timedwait_sig+0x252 seltdwait(df7c4c48,df7c4c50,565,c631a5c0,df7c4b48,...) at seltdwait+0x8a sys_poll(c631a5c0,df7c4cec,c0f53da2,c0f0d1bc,202,...) at sys_poll+0x300 syscall(df7c4d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x2813ff4b, esp = 0xbfbfcbac, ebp = 0xbfbfed78 --- Tracing command syslogd pid 1366 tid 100077 td 0xc62352e0 sched_switch(c62352e0,0,104,191,277cf3d0,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c62352e0,0,c0f0a2a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c09fc5aa,c5e9eb50,0,c0f02e87,c62352e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5e9eb64,0,df79da68,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5e9eb64,c5e9eb50,c0f0c8da,668,c6245e38,...) at _cv_wait_sig+0x243 seltdwait(c6245e38,58,c5999e00,c62352e0,0,...) at seltdwait+0xa2 kern_select(c62352e0,9,284760ac,0,0,0,20,c5e592c4,0) at kern_select+0x504 sys_select(c62352e0,df79dcec,c0f53da2,c0f0d06e,246,...) at sys_select+0x66 syscall(df79dd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x28199d43, esp = 0xbfbfe29c, ebp = 0xbfbfede8 --- Tracing command devd pid 1187 tid 100075 td 0xc62358a0 sched_switch(c62358a0,0,104,191,5acdf968,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c62358a0,0,c0f0a2a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c09fc5aa,c5e9f050,0,c0f02e87,c62358a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5e9f064,0,df795a68,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5e9f064,c5e9f050,c0f0c8da,668,c6245818,...) at _cv_wait_sig+0x243 seltdwait(c6245818,58,c5999e00,c62358a0,c62358a0,...) at seltdwait+0xa2 kern_select(c62358a0,5,bfbfe960,0,0,0,20,c5e5984c,46) at kern_select+0x504 sys_select(c62358a0,df795cec,c0f53da2,c0d71751,206,...) at sys_select+0x66 syscall(df795d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x808ba43, esp = 0xbfbfe92c, ebp = 0xbfbfee08 --- Tracing command md0 pid 100 tid 100105 td 0xc63d6b80 sched_switch(c63d6b80,0,104,191,86eea6c8,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c63d6b80,0,c0f0a2a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5c1d800,5c,c0ea7961,0,0,...) at sleepq_wait+0x63 _sleep(c5c1d800,c5c1d820,25c,c0ea7961,0,...) at _sleep+0x372 md_kthread(c5c1d800,edd28d28,c0eff150,3e5,c63f5588,...) at md_kthread+0x11a fork_exit(c0734300,c5c1d800,edd28d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xedd28d60, ebp = 0 --- Tracing command softdepflush pid 17 tid 100064 td 0xc5e5d000 kdb_enter(c0f0495c,c0f0495c,c0f37c9e,df76bba8,1,...) at kdb_enter+0x3a panic(c0f37c9e,16f960,0,4000,0,...) at panic+0x134 ffs_update(c6b3a000,0,c0f38380,2447,1,...) at ffs_update+0x1e5 handle_workitem_remove(0,df76bc6c,2,6b9,df76bc70,...) at handle_workitem_remove+0x478 process_worklist_item(c121dc70,0,c0f38380,5e8,c5e5d000,...) at process_worklist_item+0x2b4 softdep_process_worklist(c65a45a8,0,c0f38380,554,3e8,...) at softdep_process_worklist+0x88 softdep_flush(0,df76bd28,c0eff150,3e5,c5e57588,...) at softdep_flush+0x2b0 fork_exit(c0c479f0,0,df76bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf76bd60, ebp = 0 --- Tracing command vnlru pid 16 tid 100063 td 0xc5bd12e0 sched_switch(c5bd12e0,0,104,191,d331ae40,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,60,...) at mi_switch+0x1eb sleepq_switch(c5bd12e0,0,c0f0a2a0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c5e5784c,60,c0f15162,0,0,...) at sleepq_timedwait+0x6b _sleep(c5e5784c,c1217574,260,c0f15162,3e8,...) at _sleep+0x342 vnlru_proc(0,df768d28,c0eff150,3e5,c5e5784c,...) at vnlru_proc+0xe7 fork_exit(c0aaa1c0,0,df768d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf768d60, ebp = 0 --- Tracing command syncer pid 9 tid 100062 td 0xc5bd15c0 sched_switch(c5bd15c0,0,104,191,dc539ad0,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5bd15c0,0,c0f0a2a0,28b,c5bd15c0,...) at sleepq_switch+0x162 sleepq_timedwait(c12175b4,0,df765c78,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c12175b4,c12175a0,3e8,76a,4e20,...) at _cv_timedwait+0x252 sched_sync(0,df765d28,c0eff150,3e5,c5e57b10,...) at sched_sync+0x595 fork_exit(c0aa8160,0,df765d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf765d60, ebp = 0 --- Tracing command bufdaemon pid 8 tid 100061 td 0xc5bd18a0 sched_switch(c5bd18a0,0,104,191,773eec8,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,54,...) at mi_switch+0x1eb sleepq_switch(c5bd18a0,0,c0f0a2a0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c1217248,54,c0f120a7,0,0,...) at sleepq_timedwait+0x6b _sleep(c1217248,c1217250,54,c0f120a7,3e8,...) at _sleep+0x342 buf_daemon(0,df762d28,c0eff150,3e5,c5e59000,...) at buf_daemon+0x175 fork_exit(c0a8eed0,0,df762d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf762d60, ebp = 0 --- Tracing command pagezero pid 7 tid 100060 td 0xc5bd1b80 sched_switch(c5bd1b80,0,104,191,74555798,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5bd1b80,0,c0f0a2a0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c121fb5c,0,c0f41414,0,0,...) at sleepq_timedwait+0x6b _sleep(c121fb5c,c121e600,0,c0f41414,493e0,...) at _sleep+0x342 vm_pagezero(0,df75fd28,c0eff150,3e5,c599d2c4,...) at vm_pagezero+0xdc fork_exit(c0c8c1b0,0,df75fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf75fd60, ebp = 0 --- Tracing command vmdaemon pid 6 tid 100059 td 0xc5c12000 sched_switch(c5c12000,0,104,191,e2d01d20,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,74,...) at mi_switch+0x1eb sleepq_switch(c5c12000,0,c0f0a2a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c121f748,74,c0f120a7,0,0,...) at sleepq_wait+0x63 _sleep(c121f748,c121f74c,74,c0f120a7,0,...) at _sleep+0x372 vm_daemon(0,df75cd28,c0eff150,3e5,c599d588,...) at vm_daemon+0x59 fork_exit(c0c85fa0,0,df75cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf75cd60, ebp = 0 --- Tracing command pagedaemon pid 5 tid 100058 td 0xc5c122e0 sched_switch(c5c122e0,0,104,191,d7d44c30,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,54,...) at mi_switch+0x1eb sleepq_switch(c5c122e0,0,c0f0a2a0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c121f710,54,c0f120a7,0,0,...) at sleepq_timedwait+0x6b _sleep(c121f710,c121e600,54,c0f120a7,1388,...) at _sleep+0x342 vm_pageout(0,df759d28,c0eff150,3e5,c599d84c,...) at vm_pageout+0x2c7 fork_exit(c0c87080,0,df759d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf759d60, ebp = 0 --- Tracing command xpt_thrd pid 4 tid 100057 td 0xc5c125c0 sched_switch(c5c125c0,0,104,191,cecb9218,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c5c125c0,0,c0f0a2a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c10660d4,5c,c0d6c03d,0,0,...) at sleepq_wait+0x63 _sleep(c10660d4,c1066118,5c,c0d6c03d,0,...) at _sleep+0x372 xpt_scanner_thread(0,df756d28,c0eff150,3e5,c599db10,...) at xpt_scanner_thread+0x47 fork_exit(c04923b0,0,df756d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf756d60, ebp = 0 --- Tracing command sctp_iterator pid 3 tid 100056 td 0xc5c128a0 sched_switch(c5c128a0,0,104,191,b96358a0,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5c128a0,0,c0f0a2a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c12190d8,0,c0f2867f,0,0,...) at sleepq_wait+0x63 _sleep(c12190d8,c12190a0,0,c0f2867f,0,...) at _sleep+0x372 sctp_iterator_thread(0,df753d28,c0eff150,3e5,c5b8b000,...) at sctp_iterator_thread+0x5c fork_exit(c0b3acd0,0,df753d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf753d60, ebp = 0 --- Tracing command fdc0 pid 2 tid 100053 td 0xc5c142e0 sched_switch(c5c142e0,0,104,191,15c92d10,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c5c142e0,0,c0f0a2a0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c5b25a3c,5c,c0ef767a,0,0,...) at sleepq_timedwait+0x6b _sleep(c5b25a3c,c5b25af0,5c,c0ef767a,3e8,...) at _sleep+0x342 fdc_thread(c5b25a00,df74ad28,c0eff150,3e5,c5b8b2c4,...) at fdc_thread+0x27d fork_exit(c0d0b480,c5b25a00,df74ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf74ad60, ebp = 0 --- Tracing command usb pid 15 tid 100049 td 0xc5bce5c0 sched_switch(c5bce5c0,0,104,191,ae0ecd88,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5bce5c0,0,c0f0a2a0,268,c5bce5c0,...) at sleepq_switch+0x162 sleepq_wait(c5bd5d34,0,c5714cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5bd5d34,c5bd5dd4,c0ede531,6b,c5bd5d3c,...) at _cv_wait+0x243 usb_process(c5bd5d2c,c5714d28,c0eff150,3e5,c5b8b588,...) at usb_process+0x193 fork_exit(c08b1220,c5bd5d2c,c5714d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5714d60, ebp = 0 --- Tracing command usb pid 15 tid 100048 td 0xc5bce8a0 sched_switch(c5bce8a0,0,104,191,efbc62c0,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5bce8a0,0,c0f0a2a0,268,c5bce8a0,...) at sleepq_switch+0x162 sleepq_wait(c5bd5d04,0,c5711cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5bd5d04,c5bd5dd4,c0ede531,6b,c5bd5d0c,...) at _cv_wait+0x243 usb_process(c5bd5cfc,c5711d28,c0eff150,3e5,c5b8b588,...) at usb_process+0x193 fork_exit(c08b1220,c5bd5cfc,c5711d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5711d60, ebp = 0 --- Tracing command usb pid 15 tid 100047 td 0xc5bceb80 sched_switch(c5bceb80,0,104,191,ad90ed38,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5bceb80,0,c0f0a2a0,268,c5bceb80,...) at sleepq_switch+0x162 sleepq_wait(c5bd5cd4,0,c570ecac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5bd5cd4,c5bd5dd4,c0ede531,6b,c5bd5cdc,...) at _cv_wait+0x243 usb_process(c5bd5ccc,c570ed28,c0eff150,3e5,c5b8b588,...) at usb_process+0x193 fork_exit(c08b1220,c5bd5ccc,c570ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc570ed60, ebp = 0 --- Tracing command usb pid 15 tid 100046 td 0xc5bd0000 sched_switch(c5bd0000,0,104,191,ad90d910,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5bd0000,0,c0f0a2a0,268,c5bd0000,...) at sleepq_switch+0x162 sleepq_wait(c5bd5ca4,0,c570bcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5bd5ca4,c5bd5dd4,c0ede531,6b,c5bd5cac,...) at _cv_wait+0x243 usb_process(c5bd5c9c,c570bd28,c0eff150,3e5,c5b8b588,...) at usb_process+0x193 fork_exit(c08b1220,c5bd5c9c,c570bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc570bd60, ebp = 0 --- Tracing command usb pid 15 tid 100045 td 0xc5bd02e0 sched_switch(c5bd02e0,0,104,191,ad90c458,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5bd02e0,0,c0f0a2a0,268,c5bd02e0,...) at sleepq_switch+0x162 sleepq_wait(c5bbddac,0,c5705cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5bbddac,c5bbde4c,c0ede531,6b,c5bbddb4,...) at _cv_wait+0x243 usb_process(c5bbdda4,c5705d28,c0eff150,3e5,c5b8b588,...) at usb_process+0x193 fork_exit(c08b1220,c5bbdda4,c5705d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5705d60, ebp = 0 --- Tracing command usb pid 15 tid 100044 td 0xc5bd05c0 sched_switch(c5bd05c0,0,104,191,56add7f8,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5bd05c0,0,c0f0a2a0,268,c5bd05c0,...) at sleepq_switch+0x162 sleepq_wait(c5bbdd7c,0,c5702cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5bbdd7c,c5bbde4c,c0ede531,6b,c5bbdd84,...) at _cv_wait+0x243 usb_process(c5bbdd74,c5702d28,c0eff150,3e5,c5b8b588,...) at usb_process+0x193 fork_exit(c08b1220,c5bbdd74,c5702d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5702d60, ebp = 0 --- Tracing command usb pid 15 tid 100043 td 0xc5bd08a0 sched_switch(c5bd08a0,0,104,191,a67a8018,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5bd08a0,0,c0f0a2a0,268,c5bd08a0,...) at sleepq_switch+0x162 sleepq_wait(c5bbdd4c,0,c56ffcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5bbdd4c,c5bbde4c,c0ede531,6b,c5bbdd54,...) at _cv_wait+0x243 usb_process(c5bbdd44,c56ffd28,c0eff150,3e5,c5b8b588,...) at usb_process+0x193 fork_exit(c08b1220,c5bbdd44,c56ffd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc56ffd60, ebp = 0 --- Tracing command usb pid 15 tid 100042 td 0xc5bd0b80 sched_switch(c5bd0b80,0,104,191,a67a6d78,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5bd0b80,0,c0f0a2a0,268,c5bd0b80,...) at sleepq_switch+0x162 sleepq_wait(c5bbdd1c,0,c56fccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5bbdd1c,c5bbde4c,c0ede531,6b,c5bbdd24,...) at _cv_wait+0x243 usb_process(c5bbdd14,c56fcd28,c0eff150,3e5,c5b8b588,...) at usb_process+0x193 fork_exit(c08b1220,c5bbdd14,c56fcd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc56fcd60, ebp = 0 --- Tracing command usb pid 15 tid 100040 td 0xc5ba3000 sched_switch(c5ba3000,0,104,191,a67a5510,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5ba3000,0,c0f0a2a0,268,c5ba3000,...) at sleepq_switch+0x162 sleepq_wait(c5babdac,0,c56f5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5babdac,c5babe4c,c0ede531,6b,c5babdb4,...) at _cv_wait+0x243 usb_process(c5babda4,c56f5d28,c0eff150,3e5,c5b8b588,...) at usb_process+0x193 fork_exit(c08b1220,c5babda4,c56f5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc56f5d60, ebp = 0 --- Tracing command usb pid 15 tid 100039 td 0xc5ba32e0 sched_switch(c5ba32e0,0,104,191,56ad1220,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5ba32e0,0,c0f0a2a0,268,c5ba32e0,...) at sleepq_switch+0x162 sleepq_wait(c5babd7c,0,c56f2cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5babd7c,c5babe4c,c0ede531,6b,c5babd84,...) at _cv_wait+0x243 usb_process(c5babd74,c56f2d28,c0eff150,3e5,c5b8b588,...) at usb_process+0x193 fork_exit(c08b1220,c5babd74,c56f2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc56f2d60, ebp = 0 --- Tracing command usb pid 15 tid 100038 td 0xc5ba35c0 sched_switch(c5ba35c0,0,104,191,a60012b8,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5ba35c0,0,c0f0a2a0,268,c5ba35c0,...) at sleepq_switch+0x162 sleepq_wait(c5babd4c,0,c56efcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5babd4c,c5babe4c,c0ede531,6b,c5babd54,...) at _cv_wait+0x243 usb_process(c5babd44,c56efd28,c0eff150,3e5,c5b8b588,...) at usb_process+0x193 fork_exit(c08b1220,c5babd44,c56efd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc56efd60, ebp = 0 --- Tracing command usb pid 15 tid 100037 td 0xc5ba38a0 sched_switch(c5ba38a0,0,104,191,a6000080,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5ba38a0,0,c0f0a2a0,268,c5ba38a0,...) at sleepq_switch+0x162 sleepq_wait(c5babd1c,0,c56eccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5babd1c,c5babe4c,c0ede531,6b,c5babd24,...) at _cv_wait+0x243 usb_process(c5babd14,c56ecd28,c0eff150,3e5,c5b8b588,...) at usb_process+0x193 fork_exit(c08b1220,c5babd14,c56ecd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc56ecd60, ebp = 0 --- Tracing command usb pid 15 tid 100035 td 0xc5ba6000 sched_switch(c5ba6000,0,104,191,a5ffe8c8,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5ba6000,0,c0f0a2a0,268,c5ba6000,...) at sleepq_switch+0x162 sleepq_wait(c5b94dac,0,c56e5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5b94dac,c5b94e4c,c0ede531,6b,c5b94db4,...) at _cv_wait+0x243 usb_process(c5b94da4,c56e5d28,c0eff150,3e5,c5b8b588,...) at usb_process+0x193 fork_exit(c08b1220,c5b94da4,c56e5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc56e5d60, ebp = 0 --- Tracing command usb pid 15 tid 100034 td 0xc5ba62e0 sched_switch(c5ba62e0,0,104,191,56ac6450,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5ba62e0,0,c0f0a2a0,268,c5ba62e0,...) at sleepq_switch+0x162 sleepq_wait(c5b94d7c,0,c56e2cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5b94d7c,c5b94e4c,c0ede531,6b,c5b94d84,...) at _cv_wait+0x243 usb_process(c5b94d74,c56e2d28,c0eff150,3e5,c5b8b588,...) at usb_process+0x193 fork_exit(c08b1220,c5b94d74,c56e2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc56e2d60, ebp = 0 --- Tracing command usb pid 15 tid 100033 td 0xc5ba65c0 sched_switch(c5ba65c0,0,104,191,a58556a8,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5ba65c0,0,c0f0a2a0,268,c5ba65c0,...) at sleepq_switch+0x162 sleepq_wait(c5b94d4c,0,c56dfcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5b94d4c,c5b94e4c,c0ede531,6b,c5b94d54,...) at _cv_wait+0x243 usb_process(c5b94d44,c56dfd28,c0eff150,3e5,c5b8b588,...) at usb_process+0x193 fork_exit(c08b1220,c5b94d44,c56dfd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc56dfd60, ebp = 0 --- Tracing command usb pid 15 tid 100032 td 0xc5ba68a0 sched_switch(c5ba68a0,0,104,191,a5854338,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5ba68a0,0,c0f0a2a0,268,c5ba68a0,...) at sleepq_switch+0x162 sleepq_wait(c5b94d1c,0,c56dccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5b94d1c,c5b94e4c,c0ede531,6b,c5b94d24,...) at _cv_wait+0x243 usb_process(c5b94d14,c56dcd28,c0eff150,3e5,c5b8b588,...) at usb_process+0x193 fork_exit(c08b1220,c5b94d14,c56dcd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc56dcd60, ebp = 0 --- Tracing command usb pid 15 tid 100030 td 0xc5b042e0 sched_switch(c5b042e0,0,104,191,a58528f8,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5b042e0,0,c0f0a2a0,268,c5b042e0,...) at sleepq_switch+0x162 sleepq_wait(c5b7bdac,0,c56d5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5b7bdac,c5b7be4c,c0ede531,6b,c5b7bdb4,...) at _cv_wait+0x243 usb_process(c5b7bda4,c56d5d28,c0eff150,3e5,c5b8b588,...) at usb_process+0x193 fork_exit(c08b1220,c5b7bda4,c56d5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc56d5d60, ebp = 0 --- Tracing command usb pid 15 tid 100029 td 0xc5b045c0 sched_switch(c5b045c0,0,104,191,56acdb30,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5b045c0,0,c0f0a2a0,268,c5b045c0,...) at sleepq_switch+0x162 sleepq_wait(c5b7bd7c,0,c56d2cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5b7bd7c,c5b7be4c,c0ede531,6b,c5b7bd84,...) at _cv_wait+0x243 usb_process(c5b7bd74,c56d2d28,c0eff150,3e5,c5b8b588,...) at usb_process+0x193 fork_exit(c08b1220,c5b7bd74,c56d2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc56d2d60, ebp = 0 --- Tracing command usb pid 15 tid 100028 td 0xc5b048a0 sched_switch(c5b048a0,0,104,191,a509aa00,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5b048a0,0,c0f0a2a0,268,c5b048a0,...) at sleepq_switch+0x162 sleepq_wait(c5b7bd4c,0,c56cfcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5b7bd4c,c5b7be4c,c0ede531,6b,c5b7bd54,...) at _cv_wait+0x243 usb_process(c5b7bd44,c56cfd28,c0eff150,3e5,c5b8b588,...) at usb_process+0x193 fork_exit(c08b1220,c5b7bd44,c56cfd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc56cfd60, ebp = 0 --- Tracing command usb pid 15 tid 100027 td 0xc5b04b80 sched_switch(c5b04b80,0,104,191,a5099768,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5b04b80,0,c0f0a2a0,268,c5b04b80,...) at sleepq_switch+0x162 sleepq_wait(c5b7bd1c,0,c56cccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5b7bd1c,c5b7be4c,c0ede531,6b,c5b7bd24,...) at _cv_wait+0x243 usb_process(c5b7bd14,c56ccd28,c0eff150,3e5,c5b8b588,...) at usb_process+0x193 fork_exit(c08b1220,c5b7bd14,c56ccd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc56ccd60, ebp = 0 --- Tracing command yarrow pid 14 tid 100013 td 0xc59a12e0 sched_switch(c59a12e0,0,104,191,142d9de0,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c59a12e0,0,c0f0a2a0,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c10a01e4,0,c0ef767a,2,0,...) at sleepq_timedwait+0x6b _sleep(c10a01e4,0,0,c0ef767a,64,...) at _sleep+0x342 pause(c0ef767a,64,c0ebde6e,111,0,...) at pause+0x47 random_kthread(0,c5636d28,c0eff150,3e5,c599c000,...) at random_kthread+0x1ef fork_exit(c07cac10,0,c5636d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5636d60, ebp = 0 --- Tracing command geom pid 13 tid 100011 td 0xc59a18a0 sched_switch(c59a18a0,0,104,191,150acef8,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c59a18a0,0,c0f0a2a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c109de08,5c,c0ef767a,0,0,...) at sleepq_wait+0x63 _sleep(c109de08,c109dd68,25c,c0ef767a,0,...) at _sleep+0x372 g_io_schedule_down(c59a18a0,0,c0ef9219,6c,c562cd14,...) at g_io_schedule_down+0x5b g_down_procbody(0,c562cd28,c0eff150,3e5,c599c2c4,...) at g_down_procbody+0x9d fork_exit(c09a1940,0,c562cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc562cd60, ebp = 0 --- Tracing command geom pid 13 tid 100010 td 0xc59a1b80 sched_switch(c59a1b80,0,104,191,150b62c8,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c59a1b80,0,c0f0a2a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c109de04,5c,c0ef767a,0,0,...) at sleepq_wait+0x63 _sleep(c109de04,c109dd88,25c,c0ef767a,0,...) at _sleep+0x372 g_io_schedule_up(c59a1b80,0,c0ef9219,5f,c5629d14,...) at g_io_schedule_up+0x134 g_up_procbody(0,c5629d28,c0eff150,3e5,c599c2c4,...) at g_up_procbody+0x9d fork_exit(c09a19e0,0,c5629d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5629d60, ebp = 0 --- Tracing command geom pid 13 tid 100009 td 0xc59ea000 sched_switch(c59ea000,0,104,191,5acbdac8,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c59ea000,0,c0f0a2a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c109ddfc,5c,c0ef767a,0,0,...) at sleepq_wait+0x63 _sleep(c109ddfc,c109dd2c,25c,c0ef767a,0,...) at _sleep+0x372 g_run_events(c10ae040,0,c0ef9219,79,c5626d14,...) at g_run_events+0x526 g_event_procbody(0,c5626d28,c0eff150,3e5,c599c2c4,...) at g_event_procbody+0x90 fork_exit(c09a1a80,0,c5626d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5626d60, ebp = 0 --- Tracing command intr pid 12 tid 100054 td 0xc5c14000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100052 td 0xc5c145c0 sched_switch(c5c145c0,0,109,191,e8239798,...) at sched_switch+0x3bc mi_switch(109,0,c0eff48b,554,c5c145c0,...) at mi_switch+0x1eb ithread_loop(c59e5600,df73dd28,c0eff150,3e5,c599c588,...) at ithread_loop+0x1fb fork_exit(c09e3180,c59e5600,df73dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf73dd60, ebp = 0 --- Tracing command intr pid 12 tid 100051 td 0xc5bce000 sched_switch(c5bce000,0,109,191,f0086ee0,...) at sched_switch+0x3bc mi_switch(109,0,c0eff48b,554,c5bce000,...) at mi_switch+0x1eb ithread_loop(c59e5610,c573ed28,c0eff150,3e5,c599c588,...) at ithread_loop+0x1fb fork_exit(c09e3180,c59e5610,c573ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc573ed60, ebp = 0 --- Tracing command intr pid 12 tid 100050 td 0xc5bce2e0 sched_switch(c5bce2e0,0,109,191,32578960,...) at sched_switch+0x3bc mi_switch(109,0,c0eff48b,554,c5bce2e0,...) at mi_switch+0x1eb ithread_loop(c59e5620,c573ad28,c0eff150,3e5,c599c588,...) at ithread_loop+0x1fb fork_exit(c09e3180,c59e5620,c573ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc573ad60, ebp = 0 --- Tracing command intr pid 12 tid 100041 td 0xc5bd1000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100036 td 0xc5ba3b80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100031 td 0xc5ba6b80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100026 td 0xc5b09000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100022 td 0xc5b09b80 sched_switch(c5b09b80,0,109,191,e1e56b10,...) at sched_switch+0x3bc mi_switch(109,0,c0eff48b,554,c5b09b80,...) at mi_switch+0x1eb ithread_loop(c59e5780,c5651d28,c0eff150,3e5,c599c588,...) at ithread_loop+0x1fb fork_exit(c09e3180,c59e5780,c5651d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5651d60, ebp = 0 --- Tracing command intr pid 12 tid 100021 td 0xc59ea8a0 sched_switch(c59ea8a0,0,109,191,f0089128,...) at sched_switch+0x3bc mi_switch(109,0,c0eff48b,554,c59ea8a0,...) at mi_switch+0x1eb ithread_loop(c59e5790,c564ed28,c0eff150,3e5,c599c588,...) at ithread_loop+0x1fb fork_exit(c09e3180,c59e5790,c564ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc564ed60, ebp = 0 --- Tracing command intr pid 12 tid 100015 td 0xc5b03b80 sched_switch(c5b03b80,0,109,191,e24debe0,...) at sched_switch+0x3bc mi_switch(109,0,c0eff48b,554,c5b03b80,...) at mi_switch+0x1eb ithread_loop(c59e57a0,c563cd28,c0eff150,3e5,c599c588,...) at ithread_loop+0x1fb fork_exit(c09e3180,c59e57a0,c563cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc563cd60, ebp = 0 --- Tracing command intr pid 12 tid 100014 td 0xc5b04000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100008 td 0xc59ea2e0 sched_switch(c59ea2e0,0,109,191,15c91158,...) at sched_switch+0x3bc mi_switch(109,0,c0eff48b,554,c59ea2e0,...) at mi_switch+0x1eb ithread_loop(c59e57c0,c5623d28,c0eff150,3e5,c599c588,...) at ithread_loop+0x1fb fork_exit(c09e3180,c59e57c0,c5623d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5623d60, ebp = 0 --- Tracing command intr pid 12 tid 100007 td 0xc59ea5c0 sched_switch(c59ea5c0,0,109,191,142f4f80,...) at sched_switch+0x3bc mi_switch(109,0,c0eff48b,554,c59ea5c0,...) at mi_switch+0x1eb ithread_loop(c59e57d0,c5620d28,c0eff150,3e5,c599c588,...) at ithread_loop+0x1fb fork_exit(c09e3180,c59e57d0,c5620d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5620d60, ebp = 0 --- Tracing command intr pid 12 tid 100006 td 0xc599e000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100005 td 0xc599e2e0 sched_switch(c599e2e0,0,109,191,6a8e45a8,...) at sched_switch+0x3bc mi_switch(109,0,c0eff48b,554,c599e2e0,...) at mi_switch+0x1eb ithread_loop(c59e57f0,c561ad28,c0eff150,3e5,c599c588,...) at ithread_loop+0x1fb fork_exit(c09e3180,c59e57f0,c561ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc561ad60, ebp = 0 --- Tracing command idle pid 11 tid 100004 td 0xc599e5c0 sched_switch(c599e5c0,0,108,18c,143399b8,...) at sched_switch+0x3bc mi_switch(108,0,c0f069e3,3b9,c599e5c0,...) at mi_switch+0x1eb sched_idletd(0,c5614d28,c0eff150,3e5,c599c84c,...) at sched_idletd+0x19f fork_exit(c0a324d0,0,c5614d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5614d60, ebp = 0 --- Tracing command idle pid 11 tid 100003 td 0xc599e8a0 cpustop_handler(0,13,c5611b84,c0d44096,c124ce20,...) at cpustop_handler+0x28 ipi_nmi_handler(c124ce20,c5611b2c,c0d5d91e,c0d36be0,c599c84c,...) at ipi_nmi_handler+0x47 trap(c5611b90) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0d5b5b3, esp = 0xc5611bd0, ebp = 0xc5611bd4 --- lapic_eoi(c59e20d0,c0d4c3ad,c5611bfc,c599e8a0,0,...) at lapic_eoi+0x13 intr_event_handle(c596ec80,c5611c40,c599e8a0,c599e8a0,c599e8a0,...) at intr_event_handle+0xf5 intr_execute_handlers(c59e20d0,c5611c40,0,c5611c84,c0d2d455,...) at intr_execute_handlers+0x49 lapic_handle_intr(3b,c5611c40) at lapic_handle_intr+0x36 Xapic_isr1() at Xapic_isr1+0x35 --- interrupt, eip = 0xc0d36bfb, esp = 0xc5611c80, ebp = 0xc5611c84 --- spinlock_exit(c10ae040,4,c0f02e87,109) at spinlock_exit+0x2b _mtx_unlock_spin_flags(c10ae040,0,c0f069e3,a05,c599e8a0,...) at _mtx_unlock_spin_flags+0xf4 sched_idletd(0,c5611d28,c0eff150,3e5,c599c84c,...) at sched_idletd+0x2e4 fork_exit(c0a324d0,0,c5611d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5611d60, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc599eb80 sched_switch(c599eb80,0,104,191,344e08b0,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c599eb80,0,c0f0a2a0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f0a2a0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c599cb10,6c,c0f0d07f,100,0,...) at sleepq_wait_sig+0x17 _sleep(c599cb10,c599cb98,16c,c0f0d07f,0,...) at _sleep+0x35c kern_wait(c599eb80,ffffffff,c560dc60,0,0,...) at kern_wait+0x626 sys_wait4(c599eb80,c560dcec,c0f53da2,c0f0ceb6,286,...) at sys_wait4+0x3b syscall(c560dd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x80551d3, esp = 0xbfbfe8bc, ebp = 0xbfbfe8d8 --- Tracing command audit pid 10 tid 100001 td 0xc59a1000 sched_switch(c59a1000,0,104,191,a5049b80,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c59a1000,0,c0f0a2a0,268,c59a1000,...) at sleepq_switch+0x162 sleepq_wait(c121d3e0,0,c560ac8c,1,0,...) at sleepq_wait+0x63 _cv_wait(c121d3e0,c121d3c4,c0f352af,194,0,...) at _cv_wait+0x243 audit_worker(0,c560ad28,c0eff150,3e5,c599d000,...) at audit_worker+0x84 fork_exit(c0c073a0,0,c560ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc560ad60, ebp = 0 --- Tracing command kernel pid 0 tid 100065 td 0xc5e5cb80 sched_switch(c5e5cb80,0,104,191,e2c8b998,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5e5cb80,0,c0f0a2a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5c49400,0,c0ef767a,0,0,...) at sleepq_wait+0x63 _sleep(c5c49400,c5c49418,0,c0ef767a,0,...) at _sleep+0x372 taskqueue_thread_loop(c127f7a0,df76ed28,c0eff150,3e5,c109dee0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a4f1f0,c127f7a0,df76ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf76ed60, ebp = 0 --- Tracing command kernel pid 0 tid 100055 td 0xc5c12b80 sched_switch(c5c12b80,0,104,191,926234a0,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5c12b80,0,c0f0a2a0,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c10a01e4,0,c0ef767a,2,0,...) at sleepq_timedwait+0x6b _sleep(c10a01e4,0,0,c0ef767a,bb8,...) at _sleep+0x342 pause(c0ef767a,bb8,124,122,c109df68,...) at pause+0x47 deadlkres(0,df750d28,c0eff150,3e5,c109dee0,...) at deadlkres+0x324 fork_exit(c09c34c0,0,df750d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf750d60, ebp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xc5b092e0 sched_switch(c5b092e0,0,104,191,a5033820,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5b092e0,0,c0f0a2a0,268,c109e1b0,...) at sleepq_switch+0x162 sleepq_wait(c5b5f900,0,c0f056d9,c0ef767a,0,...) at sleepq_wait+0x63 msleep_spin(c5b5f900,c5b5f918,c0ef767a,0,c0f02e87,...) at msleep_spin+0x21d taskqueue_thread_loop(c5b68568,c56c5d28,c0eff150,3e5,c109dee0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a4f1f0,c5b68568,c56c5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc56c5d60, ebp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xc5b095c0 sched_switch(c5b095c0,0,104,191,15ea3a28,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5b095c0,0,c0f0a2a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5b42680,0,c0f056d9,c0ef767a,0,...) at sleepq_wait+0x63 msleep_spin(c5b42680,c5b42698,c0ef767a,0,c0f02e87,...) at msleep_spin+0x21d taskqueue_thread_loop(c5b50568,c569ad28,c0eff150,3e5,c109dee0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a4f1f0,c5b50568,c569ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc569ad60, ebp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xc5b098a0 sched_switch(c5b098a0,0,104,191,bf3caba8,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5b098a0,0,c0f0a2a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5977580,0,c0ef767a,0,0,...) at sleepq_wait+0x63 _sleep(c5977580,c5977598,0,c0ef767a,0,...) at _sleep+0x372 taskqueue_thread_loop(c10bcdc8,c5654d28,c0eff150,3e5,c109dee0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a4f1f0,c10bcdc8,c5654d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5654d60, ebp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xc59eab80 sched_switch(c59eab80,0,104,191,cecd0980,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c59eab80,0,c0f0a2a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c59776c0,0,c0ef767a,0,0,...) at sleepq_wait+0x63 _sleep(c59776c0,c59776d8,0,c0ef767a,0,...) at _sleep+0x372 taskqueue_thread_loop(c109e878,c564bd28,c0eff150,3e5,c109dee0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a4f1f0,c109e878,c564bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc564bd60, ebp = 0 --- Tracing command kernel pid 0 tid 100019 td 0xc5b03000 sched_switch(c5b03000,0,104,191,ceccf598,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5b03000,0,c0f0a2a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5977700,0,c0f056d9,c0ef767a,0,...) at sleepq_wait+0x63 msleep_spin(c5977700,c5977718,c0ef767a,0,c0f02e87,...) at msleep_spin+0x21d taskqueue_thread_loop(c1068fb8,c5648d28,c0eff150,3e5,c109dee0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a4f1f0,c1068fb8,c5648d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5648d60, ebp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xc5b032e0 sched_switch(c5b032e0,0,104,191,cecce7f8,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5b032e0,0,c0f0a2a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5977700,0,c0f056d9,c0ef767a,0,...) at sleepq_wait+0x63 msleep_spin(c5977700,c5977718,c0ef767a,0,c0f02e87,...) at msleep_spin+0x21d taskqueue_thread_loop(c1068fb8,c5645d28,c0eff150,3e5,c109dee0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a4f1f0,c1068fb8,c5645d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5645d60, ebp = 0 --- Tracing command kernel pid 0 tid 100017 td 0xc5b035c0 sched_switch(c5b035c0,0,104,191,ceccd7b0,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5b035c0,0,c0f0a2a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5977700,0,c0f056d9,c0ef767a,0,...) at sleepq_wait+0x63 msleep_spin(c5977700,c5977718,c0ef767a,0,c0f02e87,...) at msleep_spin+0x21d taskqueue_thread_loop(c1068fb8,c5642d28,c0eff150,3e5,c109dee0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a4f1f0,c1068fb8,c5642d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5642d60, ebp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xc5b038a0 sched_switch(c5b038a0,0,104,191,cecbe8f8,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5b038a0,0,c0f0a2a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5977740,0,c0ef767a,0,0,...) at sleepq_wait+0x63 _sleep(c5977740,c5977758,0,c0ef767a,0,...) at _sleep+0x372 taskqueue_thread_loop(c121d9c4,c563fd28,c0eff150,3e5,c109dee0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a4f1f0,c121d9c4,c563fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc563fd60, ebp = 0 --- Tracing command kernel pid 0 tid 100012 td 0xc59a15c0 sched_switch(c59a15c0,0,104,191,127075a8,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c59a15c0,0,c0f0a2a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5977bc0,0,c0ef767a,0,0,...) at sleepq_wait+0x63 _sleep(c5977bc0,c5977bd8,0,c0ef767a,0,...) at _sleep+0x372 taskqueue_thread_loop(c10bb7e0,c562fd28,c0eff150,3e5,c109dee0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a4f1f0,c10bb7e0,c562fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc562fd60, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc109e1b0 sched_switch(c109e1b0,0,104,191,541c34b0,...) at sched_switch+0x3bc mi_switch(104,0,c0f0a2a0,1f3,54,...) at mi_switch+0x1eb sleepq_switch(c109e1b0,0,c0f0a2a0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c109dee0,54,c0f072fd,0,0,...) at sleepq_timedwait+0x6b _sleep(c109dee0,0,54,c0f072fd,2710,...) at _sleep+0x342 scheduler(0,181ec00,181ec00,181e000,1825000,...) at scheduler+0x240 mi_startup() at mi_startup+0xac begin() at begin+0x2c db:0:allt> call doadump Physical memory: 2026 MB Physical memory: 2026 MB Dumping 178 MB:Dumping 178 MB: 163 163 147 147 131 131 115 115 99 99 83 83 67 67 51 51 35 35 19 19 3 3 Dump complete Dump complete = 0 db:0:doadump> reset cpu_reset: Restarting BSP cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 1 cpu_reset_proxy: Stopped CPU 1 (kgdb) bt #0 doadump (textdump=-545867860) at pcpu.h:244 #1 0xc04e5339 in db_fncall (dummy1=-1062991398, dummy2=0, dummy3=-1, dummy4=0xdf76b840 "T¸vß") at /mnt/pho/src/sys/ddb/db_command.c:572 #2 0xc04e576f in db_command (last_cmdp=0xc106717c, cmd_table=0x0, dopager=0) at /mnt/pho/src/sys/ddb/db_command.c:448 #3 0xc04e5824 in db_command_script (command=0xc1068088 "call doadump") at /mnt/pho/src/sys/ddb/db_command.c:519 #4 0xc04e9b60 in db_script_exec (scriptname=0xc10679e0 "doadump", warnifnotfound=Variable "warnifnotfound" is not available. ) at /mnt/pho/src/sys/ddb/db_script.c:302 #5 0xc04e9bf1 in db_run_cmd (addr=-1056498400, have_addr=0, count=-1067124151, modif=0xdf76b978 "") at /mnt/pho/src/sys/ddb/db_script.c:375 #6 0xc04e5731 in db_command (last_cmdp=0xc106717c, cmd_table=0x0, dopager=1) at /mnt/pho/src/sys/ddb/db_command.c:448 #7 0xc04e588a in db_command_loop () at /mnt/pho/src/sys/ddb/db_command.c:501 #8 0xc04e787d in db_trap (type=3, code=0) at /mnt/pho/src/sys/ddb/db_main.c:229 #9 0xc0a40bd3 in kdb_trap (type=3, code=0, tf=0xdf76bb28) at /mnt/pho/src/sys/kern/subr_kdb.c:620 #10 0xc0d446cb in trap (frame=0xdf76bb28) at /mnt/pho/src/sys/i386/i386/trap.c:719 #11 0xc0d2d08c in calltrap () at /mnt/pho/src/sys/i386/i386/exception.s:168 #12 0xc0a409da in kdb_enter (why=0xc0f0495c "panic", msg=0xc0f0495c "panic") at cpufunc.h:71 #13 0xc0a0bf64 in panic (fmt=0xc0f37c9e "ffs_update: bad link cnt") at /mnt/pho/src/sys/kern/kern_shutdown.c:599 #14 0xc0c27ef5 in ffs_update (vp=0xc6b3a000, waitfor=0) at /mnt/pho/src/sys/ufs/ffs/ffs_inode.c:104 #15 0xc0c373a8 in handle_workitem_remove (dirrem=0xc7182ac0, flags=524800) at /mnt/pho/src/sys/ufs/ffs/ffs_softdep.c:9342 #16 0xc0c36814 in process_worklist_item (mp=0xc65a45a8, target=10, flags=512) at /mnt/pho/src/sys/ufs/ffs/ffs_softdep.c:1727 #17 0xc0c476c8 in softdep_process_worklist (mp=0xc65a45a8, full=0) at /mnt/pho/src/sys/ufs/ffs/ffs_softdep.c:1516 #18 0xc0c47ca0 in softdep_flush () at /mnt/pho/src/sys/ufs/ffs/ffs_softdep.c:1372 #19 0xc09df698 in fork_exit (callout=0xc0c479f0 , arg=0x0, frame=0xdf76bd28) at /mnt/pho/src/sys/kern/kern_fork.c:1005 #20 0xc0d2d104 in fork_trampoline () at /mnt/pho/src/sys/i386/i386/exception.s:275 (kgdb) f 14 #14 0xc0c27ef5 in ffs_update (vp=0xc6b3a000, waitfor=0) at /mnt/pho/src/sys/ufs/ffs/ffs_inode.c:104 104 panic("ffs_update: bad link cnt"); (kgdb) l 99 return (error); 100 } 101 if (DOINGSOFTDEP(vp)) 102 softdep_update_inodeblock(ip, bp, waitfor); 103 else if (ip->i_effnlink != ip->i_nlink) 104 panic("ffs_update: bad link cnt"); 105 if (ip->i_ump->um_fstype == UFS1) 106 *((struct ufs1_dinode *)bp->b_data + 107 ino_to_fsbo(fs, ip->i_number)) = *ip->i_din1; 108 else (kgdb) p *ip $1 = {i_nextsnap = {tqe_next = 0x0, tqe_prev = 0x0}, i_vnode = 0xc6b3a000, i_ump = 0xc6436800, i_flag = 0, i_dev = 0xc6f14a00, i_number = 94208, i_effnlink = 0, i_fs = 0xc63e8800, i_dquot = {0x0, 0x0}, i_count = 40, i_endoff = 5632, i_diroff = 512, i_offset = 5160, i_un = {dirhash = 0x0, snapblklist = 0x0}, i_ea_area = 0x0, i_ea_len = 0, i_ea_error = 0, i_ea_refs = 0, i_mode = 16877, i_nlink = 47, i_size = 0, i_flags = 0, i_gen = 734564397, i_uid = 0, i_gid = 0, dinode_u = {din1 = 0xc6622700, din2 = 0xc6622700}} (kgdb) p *vp $2 = {v_type = VDIR, v_tag = 0xc0efb32c "ufs", v_op = 0xc1054420, v_data = 0xc6642d24, v_mount = 0xc65a45a8, v_nmntvnodes = {tqe_next = 0xc733dd98, tqe_prev = 0xc66406e0}, v_un = {vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0, vu_fifoinfo = 0x0}, v_hashlist = {le_next = 0x0, le_prev = 0xc5cdedb4}, v_hash = 94208, v_cache_src = {lh_first = 0x0}, v_cache_dst = {tqh_first = 0x0, tqh_last = 0xc6b3a030}, v_cache_dd = 0x0, v_cstart = 0, v_lasta = 0, v_lastw = 0, v_clen = 0, v_lock = {lock_object = {lo_name = 0xc0efb32c "ufs", lo_flags = 91947008, lo_data = 0, lo_witness = 0xc594b5a8}, lk_lock = 3320172544, lk_exslpfail = 0, lk_timo = 51, lk_pri = 96, lk_stack = {depth = 13, pcs = {3231663854, 3234139425, 3235278917, 3232456760, 3232401755, 3232338397, 3234112009, 3234033552, 3234031636, 3234100936, 3234102432, 3231577752, 3235041540, 0, 0, 0, 0, 0}}}, v_interlock = {lock_object = {lo_name = 0xc0f0c585 "vnode interlock", lo_flags = 16973824, lo_data = 0, lo_witness = 0xc59441e0}, mtx_lock = 4}, v_vnlock = 0xc6b3a058, v_holdcnt = 1, v_usecount = 1, v_iflag = 0, v_vflag = 0, v_writecount = 0, v_freelist = {tqe_next = 0xc732ec3c, tqe_prev = 0xc757b660}, v_bufobj = {bo_mtx = {lock_object = {lo_name = 0xc0f15137 "bufobj interlock", lo_flags = 16973824, lo_data = 0, lo_witness = 0xc5949a08}, mtx_lock = 4}, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0xc6b3a10c}, bv_root = 0x0, bv_cnt = 0}, bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xc6b3a11c}, bv_root = 0x0, bv_cnt = 0}, bo_numoutput = 0, bo_flag = 0, bo_ops = 0xc1041500, bo_bsize = 16384, bo_object = 0xc659fc38, bo_synclist = {le_next = 0xc69fd3b0, le_prev = 0xc5bf434c}, bo_private = 0xc6b3a000, __bo_vnode = 0xc6b3a000}, v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0} (kgdb) -- Test scenario: suj25.sh + mdconfig -a -t swap -s 2g -u 5 + bsdlabel -w md5 auto + newfs -U -j md5a + mount /dev/md5a /mnt + chmod 777 /mnt + cp -R /usr/include /mnt/1 + cp -R /usr/include /mnt/2 + cp -R /usr/include /mnt/3 + cp -R /usr/include /mnt/4 + rm -rf /mnt/1 + sleep 2 + [ ! -s /mnt/.snap/suj25 ] + rm -f /mnt/.snap/suj25 + mksnap_ffs /mnt /mnt/.snap/suj25 mksnap_ffs: Cannot create snapshot /mnt/.snap/suj25: Resource temporarily unavailable + [ ! -s /mnt/.snap/suj25 ] + rm -f /mnt/.snap/suj25