KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2012 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #0: Wed Feb 8 11:40:59 CET 2012 test@leopard3.netperf.freebsd.org:/mnt/pho/obj/mnt/pho/src/sys/PHO i386 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: Intel(R) Xeon(R) CPU 3050 @ 2.13GHz (2128.04-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x6f2 Family = 6 Model = f Stepping = 2 Features=0xbfebfbff Features2=0xe3bd AMD Features=0x20100000 AMD Features2=0x1 TSC: P-state invariant, performance statistics real memory = 2147483648 (2048 MB) avail memory = 2080346112 (1983 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 random device not loaded; using insecure entropy ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard ioapic2 irqs 48-71 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 cpu1: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 16 at device 1.0 on pci0 pci1: on pcib1 pcib2: at device 0.0 on pci1 pci2: on pcib2 pcib3: at device 0.2 on pci1 pci3: on pcib3 pcib4: irq 16 at device 3.0 on pci0 pci10: on pcib4 pci10: at device 0.0 (no driver attached) pcib5: irq 17 at device 28.0 on pci0 pcib5: failed to allocate initial I/O port window: 0-0xfff pcib5: failed to allocate initial memory window: 0-0xfffff pcib5: failed to allocate initial prefetch window: 0-0xfffff pci14: on pcib5 pcib6: irq 17 at device 28.4 on pci0 pci18: on pcib6 em0: port 0x5000-0x501f mem 0xe0300000-0xe031ffff irq 16 at device 0.0 on pci18 em0: Using an MSI interrupt em0: Ethernet address: 00:30:48:8e:d0:68 pcib7: irq 16 at device 28.5 on pci0 pci19: on pcib7 em1: port 0x6000-0x601f mem 0xe0400000-0xe041ffff irq 17 at device 0.0 on pci19 em1: Using an MSI interrupt em1: Ethernet address: 00:30:48:8e:d0:69 uhci0: port 0x3000-0x301f irq 23 at device 29.0 on pci0 usbus0: on uhci0 uhci1: port 0x3020-0x303f irq 19 at device 29.1 on pci0 usbus1: on uhci1 uhci2: port 0x3040-0x305f irq 18 at device 29.2 on pci0 usbus2: on uhci2 uhci3: port 0x3060-0x307f irq 16 at device 29.3 on pci0 usbus3: on uhci3 ehci0: mem 0xe0000000-0xe00003ff irq 23 at device 29.7 on pci0 usbus4: EHCI version 1.0 usbus4: on ehci0 pcib8: at device 30.0 on pci0 pci20: on pcib8 vgapci0: port 0x7000-0x70ff mem 0xe8000000-0xefffffff,0xe0500000-0xe050ffff irq 16 at device 4.0 on pci20 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x30a0-0x30af at device 31.1 on pci0 ata0: at channel 0 on atapci0 ahci0: port 0x30e8-0x30ef,0x30dc-0x30df,0x30e0-0x30e7,0x30d8-0x30db,0x30b0-0x30bf mem 0xe0000400-0xe00007ff irq 19 at device 31.2 on pci0 ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 pci0: at device 31.3 (no driver attached) acpi_button0: on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (115200,n,8,1) uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fd0: <1440-KB 3.5" drive> on fdc0 drive 0 pmtimer0 on isa0 orm0: at iomem 0xc0000-0xcafff,0xcb000-0xcbfff pnpid ORM0000 on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x100> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] ppc0: parallel port not found. ctl: CAM Target Layer loaded est0: on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 828082806000828 device_attach: est0 attach returned 6 p4tcc0: on cpu0 est1: on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 828082806000828 device_attach: est1 attach returned 6 p4tcc1: on cpu1 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 12Mbps Full Speed USB v1.0 ugen0.1: at usbus0 uhub0: on usbus0 Expensive timeout(9) function: 0xc0a9b270(0xc63acb80) 0.011044979 s ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 usbus4: 480Mbps High Speed USB v2.0 ugen4.1: at usbus4 uhub4: on usbus4 Expensive timeout(9) function: 0xc0554a10(0xc6359d80) 0.020062605 s uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub4: 8 ports with 8 removable, self powered ada0 at ahcich0 bus 0 scbus1 target 0 lun 0 ada0: ATA-7cd0 at ata0 bus 0 scbus0 target 0 lun 0 cd0: Removable CD-ROM SCSI-0 device cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 238475MB (488397168 512 byte sectors: 16H 63S/T 16383C) ada0: Previously was known as ad4 SMP: AP CPU #1 Launched! Timecounter "TSC-low" frequency 16625325 Hz quality 1000 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Trying to mount root from nfs: []... NFS ROOT: 192.168.5.1:/zoo/leopard3 em0: link state changed to UP Interface IP-Address Broadcast Setting hostuuid: 64a93476-27b1-3004-c299-0030488ed164. Setting hostid: 0x7acc54f1. Starting ddb. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/ada0d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0d: clean, 109580993 free (1505 frags, 13697436 blocks, 0.0% fragmentation) mount_nfs: can't update /var/db/mounttab for 192.168.5.1:/zoo/leopard3 Mounting local file systems:. Setting hostname: leopard3.netperf.freebsd.org. Starting Network: lo0 em0 em1. lo0: flags=8049 metric 0 mtu 16384 options=3 nd6 options=21 em0: flags=8843 metric 0 mtu 1500 options=4219b ether 00:30:48:8e:d0:68 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active em1: flags=8c02 metric 0 mtu 1500 options=4219b ether 00:30:48:8e:d0:69 media: Ethernet autoselect status: no carrier Starting devd. Starting Network: em1. em1: flags=8c02 metric 0 mtu 1500 options=4219b ether 00:30:48:8e:d0:69 media: Ethernet autoselect status: no carrier Starting Network: usbus0. Starting Network: usbus1. Starting Network: usbus2. Starting Network: usbus3. Starting Network: usbus4. add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 Creating and/or trimming log files. Starting syslogd. No core dumps found. ELF ldconfig path: /lib /usr/lib /usr/lib/compat a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Setting date via ntp. 8 Feb 11:22:56 ntpdate[1391]: step time server 192.168.5.1 offset -0.388419 sec Starting rpcbind. NFS access cache time=60 Clearing /tmp (X related). Starting mountd. Starting nfsd. Starting statd. Updating motd:. Configuring syscons: blanktime. Starting sshd. Starting cron. Local package initialization: watchdogd. Starting inetd. Wed Feb 8 11:22:57 UTC 2012 FreeBSD/i386 (leopard3.netperf.freebsd.org) (ttyu0) login: Feb 8 12:27:31 leopard3 su: test to root on /dev/pts/0 lock order reversal: 1st 0xdfd17680 bufwait (bufwait) @ /mnt/pho/src/sys/kern/vfs_bio.c:2658 2nd 0xc643fc00 dirhash (dirhash) @ /mnt/pho/src/sys/ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper(c0f9097b,7366752f,7366752f,7366752f,7269645f,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a94a3b,c0f94480,c1272230,11c,ee85d7dc,...) at kdb_backtrace+0x2a _witness_debugger(c0f94480,c643fc00,c0fc4a52,c6158a88,c0fc46d3,...) at _witness_debugger+0x25 witness_checkorder(c643fc00,9,c0fc46d3,11c,0,...) at witness_checkorder+0x86f _sx_xlock(c643fc00,0,c0fc46d3,11c,c8e8e244,...) at _sx_xlock+0xa0 ufsdirhash_acquire(dfd17620,c8e8e244,ee85d918,e0c10028,ee85d8ac,...) at ufsdirhash_acquire+0x48 ufsdirhash_add(c8e8e244,ee85d918,28,ee85d898,ee85d89c,...) at ufsdirhash_add+0x13 ufs_direnter(c90b3984,c90b315c,ee85d918,ee85dbc0,0,...) at ufs_direnter+0x7a9 ufs_makeinode(ee85dbc0,c10dc260,ee85db04,ee85da60,c0de3005,...) at ufs_makeinode+0x65d ufs_create(ee85db04,c0fe5feb,0,0,ee85db80,...) at ufs_create+0x30 VOP_CREATE_APV(c10dbce0,ee85db04,ee85dbc0,ee85da9c,0,...) at VOP_CREATE_APV+0xc5 vn_open_cred(ee85db80,ee85dc48,0,0,c891f700,...) at vn_open_cred+0x205 vn_open(ee85db80,ee85dc48,0,c8cc1d20,246,...) at vn_open+0x3b kern_openat(c90cd5c0,ffffff9c,2842ca15,0,603,...) at kern_openat+0x1ec kern_open(c90cd5c0,2842ca15,0,602,0,...) at kern_open+0x35 sys_open(c90cd5c0,ee85dcec,c0fdec82,c0f7c2c5,246,...) at sys_open+0x30 syscall(ee85dd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (5, FreeBSD ELF32, sys_open), eip = 0x28187b73, esp = 0xbfbfe18c, ebp = 0xbfbfe858 --- lock order reversal: 1st 0xc90b39dc ufs (ufs) @ /mnt/pho/src/sys/kern/vfs_subr.c:2167 2nd 0xdfd17680 bufwait (bufwait) @ /mnt/pho/src/sys/ufs/ffs/ffs_vnops.c:260 3rd 0xc90b31b4 ufs (ufs) @ /mnt/pho/src/sys/kern/vfs_subr.c:2167 KDB: stack backtrace: db_trace_self_wrapper(c0f9097b,75735f73,632e7262,3631323a,6f000a37,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a94a3b,c0f94499,c1272220,877,ee85d33c,...) at kdb_backtrace+0x2a _witness_debugger(c0f94499,c90b31b4,c0f82d94,c61588e8,c0f9c554,...) at _witness_debugger+0x25 witness_checkorder(c90b31b4,9,c0f9c554,877,0,...) at witness_checkorder+0x86f __lockmgr_args(c90b31b4,80100,c90b3220,0,0,...) at __lockmgr_args+0x834 ffs_lock(ee85d460,0,c0f936a5,80100,c90b315c,...) at ffs_lock+0xa1 VOP_LOCK1_APV(c10dbce0,ee85d460,c61511e0,c10ecd00,c90b315c,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c90b315c,80100,c0f9c554,877,4,...) at _vn_lock+0x78 vget(c90b315c,80100,c90cd5c0,50,0,...) at vget+0xbb vfs_hash_get(c90d35a8,5,80000,c90cd5c0,ee85d5a4,...) at vfs_hash_get+0xed ffs_vgetf(c90d35a8,5,80000,ee85d5a4,1,...) at ffs_vgetf+0x49 softdep_sync_buf(c90b3984,dfd17620,1,106,0,...) at softdep_sync_buf+0xac9 ffs_syncvnode(c90b3984,1,bfbfe18c,bfbfe858,bfbfe858,...) at ffs_syncvnode+0x24c ffs_truncate(c90b3984,400,0,880,c891f700,...) at ffs_truncate+0x827 ufs_direnter(c90b3984,c90b315c,ee85d918,ee85dbc0,0,...) at ufs_direnter+0x9a1 ufs_makeinode(ee85dbc0,c10dc260,ee85db04,ee85da60,c0de3005,...) at ufs_makeinode+0x65d ufs_create(ee85db04,c0fe5feb,0,0,ee85db80,...) at ufs_create+0x30 VOP_CREATE_APV(c10dbce0,ee85db04,ee85dbc0,ee85da9c,0,...) at VOP_CREATE_APV+0xc5 vn_open_cred(ee85db80,ee85dc48,0,0,c891f700,...) at vn_open_cred+0x205 vn_open(ee85db80,ee85dc48,0,c8cc1d20,246,...) at vn_open+0x3b kern_openat(c90cd5c0,ffffff9c,2842ca15,0,603,...) at kern_openat+0x1ec kern_open(c90cd5c0,2842ca15,0,602,0,...) at kern_open+0x35 sys_open(c90cd5c0,ee85dcec,c0fdec82,c0f7c2c5,246,...) at sys_open+0x30 syscall(ee85dd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (5, FreeBSD ELF32, sys_open), eip = 0x28187b73, esp = 0xbfbfe18c, ebp = 0xbfbfe858 --- lock order reversal: 1st 0xdfd22940 bufwait (bufwait) @ /mnt/pho/src/sys/kern/vfs_bio.c:2658 2nd 0xc88c821c snaplk (snaplk) @ /mnt/pho/src/sys/ufs/ffs/ffs_snapshot.c:2293 KDB: stack backtrace: db_trace_self_wrapper(c0f9097b,662f7366,662f7366,735f7366,7370616e,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a94a3b,c0f94480,c1271988,8f5,ee83f754,...) at kdb_backtrace+0x2a _witness_debugger(c0f94480,c88c821c,c0fc05e5,c6158c90,c0fc055e,...) at _witness_debugger+0x25 witness_checkorder(c88c821c,9,c0fc055e,8f5,0,...) at witness_checkorder+0x86f __lockmgr_args(c88c821c,80900,c90b3d00,0,0,...) at __lockmgr_args+0x834 ffs_copyonwrite(c90b3c3c,dfd1f000,0,dfcea4a0,c90b4984,...) at ffs_copyonwrite+0x2c2 ffs_geom_strategy(c90b3d34,dfcea4a0,ee83f8d8,c8f7a000,c90db1b0,...) at ffs_geom_strategy+0xa0 ufs_strategy(ee83f914,c0fe5ce6,c8f7a000,dfcea4a0,4,...) at ufs_strategy+0xd3 VOP_STRATEGY_APV(c10dbce0,ee83f914,c0f99a5a,c10ecd60,c90b4984,...) at VOP_STRATEGY_APV+0xe5 bufstrategy(c90b4a7c,dfcea4a0,3a6,dfd21d80,c35611c8,...) at bufstrategy+0x61 bufwrite(dfcea4a0,7,ee83f9b4,c0ae4727,dfcea4a0,...) at bufwrite+0x159 bawrite(dfcea4a0,dfcea5a8,1c,3af,0,...) at bawrite+0x5c cluster_wbuild(c90b4984,4000,7,0,7,...) at cluster_wbuild+0x837 cluster_write(c90b4984,dfd228e0,20000,0,21,...) at cluster_write+0x203 ffs_write(ee83fbac,c0fe5f21,c90b4984,0,c90b4984,...) at ffs_write+0x52b VOP_WRITE_APV(c10dbce0,ee83fbac,c90b4984,280,0,...) at VOP_WRITE_APV+0x136 vn_write(c8cd29a0,ee83fc44,c891fe00,0,c8f7a000,...) at vn_write+0x2bc dofilewrite(ee83fc44,ffffffff,ffffffff,0,ee83fc20,...) at dofilewrite+0x95 kern_writev(c8f7a000,4,ee83fc44,ee83fc64,1,...) at kern_writev+0x68 sys_write(c8f7a000,ee83fcec,c0fdec82,c0f5a9e7,206,...) at sys_write+0x4f syscall(ee83fd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, sys_write), eip = 0x28195da3, esp = 0xbfbfd84c, ebp = 0xbfbfe908 --- fsync: giving up on dirty 0xc90b3c3c: tag devfs, type VCHR usecount 1, writecount 0, refcount 12 mountedhere 0xc88ce700 flags (VV_COPYONWRITE) v_object 0xc90b53f0 ref 0 pages 34 lock type devfs: EXCL by thread 0xc8dbc000 (pid 1867) #0 0xc0a426ee at __lockmgr_args+0xc1e #1 0xc0ae6475 at vop_stdlock+0x65 #2 0xc0de2035 at VOP_LOCK1_APV+0xb5 #3 0xc0b08528 at _vn_lock+0x78 #4 0xc0c993a7 at softdep_flushworklist+0x47 #5 0xc0c9e36d at ffs_sync+0x32d #6 0xc0b08122 at vfs_write_suspend+0x122 #7 0xc0c8250d at ffs_snapshot+0xced #8 0xc0ca08a7 at ffs_mount+0x1ca7 #9 0xc0af062b at vfs_donmount+0x12eb #10 0xc0af0ccf at sys_nmount+0x7f #11 0xc0dbecd7 at syscall+0x2c7 #12 0xc0da8001 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc90b3c3c: tag devfs, type VCHR usecount 1, writecount 0, refcount 12 mountedhere 0xc88ce700 flags (VV_COPYONWRITE) v_object 0xc90b53f0 ref 0 pages 34 lock type devfs: EXCL by thread 0xc8d8b2e0 (pid 1871) #0 0xc0a426ee at __lockmgr_args+0xc1e #1 0xc0ae6475 at vop_stdlock+0x65 #2 0xc0de2035 at VOP_LOCK1_APV+0xb5 #3 0xc0b08528 at _vn_lock+0x78 #4 0xc0c993a7 at softdep_flushworklist+0x47 #5 0xc0c9e36d at ffs_sync+0x32d #6 0xc0b08122 at vfs_write_suspend+0x122 #7 0xc0c8250d at ffs_snapshot+0xced #8 0xc0ca08a7 at ffs_mount+0x1ca7 #9 0xc0af062b at vfs_donmount+0x12eb #10 0xc0af0ccf at sys_nmount+0x7f #11 0xc0dbecd7 at syscall+0x2c7 #12 0xc0da8001 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc90b3c3c: tag devfs, type VCHR usecount 1, writecount 0, refcount 26 mountedhere 0xc88ce700 flags (VV_COPYONWRITE) v_object 0xc90b53f0 ref 0 pages 90 lock type devfs: EXCL by thread 0xc90cd8a0 (pid 1875) #0 0xc0a426ee at __lockmgr_args+0xc1e #1 0xc0ae6475 at vop_stdlock+0x65 #2 0xc0de2035 at VOP_LOCK1_APV+0xb5 #3 0xc0b08528 at _vn_lock+0x78 #4 0xc0c993a7 at softdep_flushworklist+0x47 #5 0xc0c9e36d at ffs_sync+0x32d #6 0xc0b08122 at vfs_write_suspend+0x122 #7 0xc0c8250d at ffs_snapshot+0xced #8 0xc0ca08a7 at ffs_mount+0x1ca7 #9 0xc0af062b at vfs_donmount+0x12eb #10 0xc0af0ccf at sys_nmount+0x7f #11 0xc0dbecd7 at syscall+0x2c7 #12 0xc0da8001 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc90b3c3c: tag devfs, type VCHR usecount 1, writecount 0, refcount 30 mountedhere 0xc88ce700 flags (VV_COPYONWRITE) v_object 0xc90b53f0 ref 0 pages 102 lock type devfs: EXCL by thread 0xc8f7c2e0 (pid 1889) #0 0xc0a426ee at __lockmgr_args+0xc1e #1 0xc0ae6475 at vop_stdlock+0x65 #2 0xc0de2035 at VOP_LOCK1_APV+0xb5 #3 0xc0b08528 at _vn_lock+0x78 #4 0xc0c993a7 at softdep_flushworklist+0x47 #5 0xc0c9e36d at ffs_sync+0x32d #6 0xc0b08122 at vfs_write_suspend+0x122 #7 0xc0c8250d at ffs_snapshot+0xced #8 0xc0ca08a7 at ffs_mount+0x1ca7 #9 0xc0af062b at vfs_donmount+0x12eb #10 0xc0af0ccf at sys_nmount+0x7f #11 0xc0dbecd7 at syscall+0x2c7 #12 0xc0da8001 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc90b3c3c: tag devfs, type VCHR usecount 1, writecount 0, refcount 28 mountedhere 0xc88ce700 flags (VV_COPYONWRITE) v_object 0xc90b53f0 ref 0 pages 98 lock type devfs: EXCL by thread 0xc8f7c2e0 (pid 1889) #0 0xc0a426ee at __lockmgr_args+0xc1e #1 0xc0ae6475 at vop_stdlock+0x65 #2 0xc0de2035 at VOP_LOCK1_APV+0xb5 #3 0xc0b08528 at _vn_lock+0x78 #4 0xc0c9e437 at ffs_sync+0x3f7 #5 0xc0b08122 at vfs_write_suspend+0x122 #6 0xc0c8250d at ffs_snapshot+0xced #7 0xc0ca08a7 at ffs_mount+0x1ca7 #8 0xc0af062b at vfs_donmount+0x12eb #9 0xc0af0ccf at sys_nmount+0x7f #10 0xc0dbecd7 at syscall+0x2c7 #11 0xc0da8001 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc90b3c3c: tag devfs, type VCHR usecount 1, writecount 0, refcount 34 mountedhere 0xc88ce700 flags (VV_COPYONWRITE) v_object 0xc90b53f0 ref 0 pages 118 lock type devfs: EXCL by thread 0xc90cd8a0 (pid 1893) #0 0xc0a426ee at __lockmgr_args+0xc1e #1 0xc0ae6475 at vop_stdlock+0x65 #2 0xc0de2035 at VOP_LOCK1_APV+0xb5 #3 0xc0b08528 at _vn_lock+0x78 #4 0xc0c993a7 at softdep_flushworklist+0x47 #5 0xc0c9e36d at ffs_sync+0x32d #6 0xc0b08122 at vfs_write_suspend+0x122 #7 0xc0c8250d at ffs_snapshot+0xced #8 0xc0ca08a7 at ffs_mount+0x1ca7 #9 0xc0af062b at vfs_donmount+0x12eb #10 0xc0af0ccf at sys_nmount+0x7f #11 0xc0dbecd7 at syscall+0x2c7 #12 0xc0da8001 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc90b3c3c: tag devfs, type VCHR usecount 1, writecount 0, refcount 28 mountedhere 0xc88ce700 flags (VV_COPYONWRITE) v_object 0xc90b53f0 ref 0 pages 98 lock type devfs: EXCL by thread 0xc90cd000 (pid 1897) #0 0xc0a426ee at __lockmgr_args+0xc1e #1 0xc0ae6475 at vop_stdlock+0x65 #2 0xc0de2035 at VOP_LOCK1_APV+0xb5 #3 0xc0b08528 at _vn_lock+0x78 #4 0xc0c993a7 at softdep_flushworklist+0x47 #5 0xc0c9e36d at ffs_sync+0x32d #6 0xc0b08122 at vfs_write_suspend+0x122 #7 0xc0c8250d at ffs_snapshot+0xced #8 0xc0ca08a7 at ffs_mount+0x1ca7 #9 0xc0af062b at vfs_donmount+0x12eb #10 0xc0af0ccf at sys_nmount+0x7f #11 0xc0dbecd7 at syscall+0x2c7 #12 0xc0da8001 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc90b3c3c: tag devfs, type VCHR usecount 1, writecount 0, refcount 28 mountedhere 0xc88ce700 flags (VV_COPYONWRITE) v_object 0xc90b53f0 ref 0 pages 98 lock type devfs: EXCL by thread 0xc90cd000 (pid 1897) #0 0xc0a426ee at __lockmgr_args+0xc1e #1 0xc0ae6475 at vop_stdlock+0x65 #2 0xc0de2035 at VOP_LOCK1_APV+0xb5 #3 0xc0b08528 at _vn_lock+0x78 #4 0xc0c9e437 at ffs_sync+0x3f7 #5 0xc0b08122 at vfs_write_suspend+0x122 #6 0xc0c8250d at ffs_snapshot+0xced #7 0xc0ca08a7 at ffs_mount+0x1ca7 #8 0xc0af062b at vfs_donmount+0x12eb #9 0xc0af0ccf at sys_nmount+0x7f #10 0xc0dbecd7 at syscall+0x2c7 #11 0xc0da8001 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc90b3c3c: tag devfs, type VCHR usecount 1, writecount 0, refcount 34 mountedhere 0xc88ce700 flags (VV_COPYONWRITE) v_object 0xc90b53f0 ref 0 pages 122 lock type devfs: EXCL by thread 0xc8f7a2e0 (pid 1901) #0 0xc0a426ee at __lockmgr_args+0xc1e #1 0xc0ae6475 at vop_stdlock+0x65 #2 0xc0de2035 at VOP_LOCK1_APV+0xb5 #3 0xc0b08528 at _vn_lock+0x78 #4 0xc0c993a7 at softdep_flushworklist+0x47 #5 0xc0c9e36d at ffs_sync+0x32d #6 0xc0b08122 at vfs_write_suspend+0x122 #7 0xc0c8250d at ffs_snapshot+0xced #8 0xc0ca08a7 at ffs_mount+0x1ca7 #9 0xc0af062b at vfs_donmount+0x12eb #10 0xc0af0ccf at sys_nmount+0x7f #11 0xc0dbecd7 at syscall+0x2c7 #12 0xc0da8001 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc90b3c3c: tag devfs, type VCHR usecount 1, writecount 0, refcount 34 mountedhere 0xc88ce700 flags (VV_COPYONWRITE) v_object 0xc90b53f0 ref 0 pages 122 lock type devfs: EXCL by thread 0xc8f7a2e0 (pid 1901) #0 0xc0a426ee at __lockmgr_args+0xc1e #1 0xc0ae6475 at vop_stdlock+0x65 #2 0xc0de2035 at VOP_LOCK1_APV+0xb5 #3 0xc0b08528 at _vn_lock+0x78 #4 0xc0c9e437 at ffs_sync+0x3f7 #5 0xc0b08122 at vfs_write_suspend+0x122 #6 0xc0c8250d at ffs_snapshot+0xced #7 0xc0ca08a7 at ffs_mount+0x1ca7 #8 0xc0af062b at vfs_donmount+0x12eb #9 0xc0af0ccf at sys_nmount+0x7f #10 0xc0dbecd7 at syscall+0x2c7 #11 0xc0da8001 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc90b3c3c: tag devfs, type VCHR usecount 1, writecount 0, refcount 34 mountedhere 0xc88ce700 flags (VV_COPYONWRITE) v_object 0xc90b53f0 ref 0 pages 122 lock type devfs: EXCL by thread 0xc88ea8a0 (pid 1905) #0 0xc0a426ee at __lockmgr_args+0xc1e #1 0xc0ae6475 at vop_stdlock+0x65 #2 0xc0de2035 at VOP_LOCK1_APV+0xb5 #3 0xc0b08528 at _vn_lock+0x78 #4 0xc0c993a7 at softdep_flushworklist+0x47 #5 0xc0c9e36d at ffs_sync+0x32d #6 0xc0b08122 at vfs_write_suspend+0x122 #7 0xc0c8250d at ffs_snapshot+0xced #8 0xc0ca08a7 at ffs_mount+0x1ca7 #9 0xc0af062b at vfs_donmount+0x12eb #10 0xc0af0ccf at sys_nmount+0x7f #11 0xc0dbecd7 at syscall+0x2c7 #12 0xc0da8001 at Xint0x80_syscall+0x21 dev md5a lock order reversal: 1st 0xc88c821c snaplk (snaplk) @ /mnt/pho/src/sys/ufs/ufs/ufs_vnops.c:965 2nd 0xc9153c94 ufs (ufs) @ /mnt/pho/src/sys/ufs/ffs/ffs_snapshot.c:1623 KDB: stack backtrace: db_trace_self_wrapper(c0f9097b,7366662f,616e735f,6f687370,3a632e74,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a94a3b,c0f94480,c1271298,657,ee7c2704,...) at kdb_backtrace+0x2a _witness_debugger(c0f94480,c9153c94,c0f82d94,c61588e8,c0fc055e,...) at _witness_debugger+0x25 witness_checkorder(c9153c94,9,c0fc055e,657,0,...) at witness_checkorder+0x86f __lockmgr_args(c9153c94,80000,0,0,0,...) at __lockmgr_args+0x834 ffs_snapremove(c9153c3c,c0fc0477,1,1,c6151ee0,...) at ffs_snapremove+0x11f ffs_truncate(c9153c3c,0,0,c00,0,...) at ffs_truncate+0x607 ufs_inactive(ee7c2adc,c0fe55b8,c9153d00,c9153d00,c9153c3c,...) at ufs_inactive+0x1fb VOP_INACTIVE_APV(c10dbce0,ee7c2adc,c0f9c554,96f,c10eccc0,...) at VOP_INACTIVE_APV+0xc5 vinactive(c10dbce0,ee7c2b10,c0f9c554,8c6,0,...) at vinactive+0xb1 vputx(ee7c2c3c,c0aff163,c9153c3c,ffffffdf,c9153c3c,...) at vputx+0x36c vput(c9153c3c,ffffffdf,c9153c3c,ee7c2b94,0,...) at vput+0x10 kern_unlinkat(c8dc08a0,ffffff9c,bfbfeb5e,0,0,...) at kern_unlinkat+0x2a3 kern_unlink(c8dc08a0,bfbfeb5e,0,ee7c2d1c,c0dbecd7,...) at kern_unlink+0x2f sys_unlink(c8dc08a0,ee7c2cec,c0fdec82,ee7c2d80,246,...) at sys_unlink+0x22 syscall(ee7c2d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (10, FreeBSD ELF32, sys_unlink), eip = 0x281720af, esp = 0xbfbfe8bc, ebp = 0xbfbfe938 --- Feb 8 11:32:09 leopard3 kernel: pid 1971 (rw), uid 1001 inumber 23576 on /mnt: filesystem full Feb 8 11:34:08 leopard3 kernel: pid 2084 (rw), uid 1001 inumber 70686 on /mnt: filesystem full Feb 8 11:38:25 leopard3 kernel: pid 2761 (rw), uid 1001 inumber 47115 on /mnt: filesystem full Feb 8 11:42:27 leopard3 kernel: pid 3720 (rw), uid 1001 inumber 70673 on /mnt: filesystem full Feb 8 11:44:28 leopard3 kernel: pid 4212 (rw), uid 1001 inumber 23583 on /mnt: filesystem full Feb 8 11:46:29 leopard3 kernel: pid 4693 (rw), uid 1001 inumber 47107 on /mnt: filesystem full lock order reversal: 1st 0xc8ea0b38 newnfs (newnfs) @ /mnt/pho/src/sys/kern/vfs_mount.c:1249 2nd 0xc8fb8310 syncer (syncer) @ /mnt/pho/src/sys/kern/vfs_subr.c:2279 KDB: stack backtrace: db_trace_self_wrapper(c0f9097b,2f6e7265,5f736676,72627573,323a632e,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a94a3b,c0f94480,c1272108,8e7,ee8e99dc,...) at kdb_backtrace+0x2a _witness_debugger(c0f94480,c8fb8310,c0f9d114,c6158748,c0f9c554,...) at _witness_debugger+0x25 witness_checkorder(c8fb8310,9,c0f9c554,8e7,0,...) at witness_checkorder+0x86f __lockmgr_args(c8fb8310,80100,c8fb837c,0,0,...) at __lockmgr_args+0x834 vop_stdlock(ee8e9afc,c0aa4bfa,c0f94876,80100,c8fb82b8,...) at vop_stdlock+0x65 VOP_LOCK1_APV(c10ca6a0,ee8e9afc,c0a4aa5c,c10ecd00,c8fb82b8,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c8fb82b8,80100,c0f9c554,8e7,c129fc20,...) at _vn_lock+0x78 vputx(ee8e9b80,c0afd19d,c8fb82b8,0,c0f9c554,...) at vputx+0x266 vrele(c8fb82b8,0,c0f9c554,db7,c8ea0ae0,...) at vrele+0x10 vfs_deallocate_syncvnode(c90d35a8,0,c0f9bba7,51a,0,...) at vfs_deallocate_syncvnode+0x6d dounmount(c90d35a8,8000000,ca187000,49c,0,...) at dounmount+0x376 sys_unmount(ca187000,ee8e9cec,c,c,206,...) at sys_unmount+0x421 syscall(ee8e9d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280dcb3f, esp = 0xbfbfe1bc, ebp = 0xbfbfe288 --- lock order reversal: 1st 0xc8ea0b38 newnfs (newnfs) @ /mnt/pho/src/sys/kern/vfs_mount.c:1249 2nd 0xc90b3c94 devfs (devfs) @ /mnt/pho/src/sys/ufs/ffs/ffs_softdep.c:1835 KDB: stack backtrace: db_trace_self_wrapper(c0f9097b,7366662f,7366662f,666f735f,70656474,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a94a3b,c0f94480,c1272108,72b,ee8e99a0,...) at kdb_backtrace+0x2a _witness_debugger(c0f94480,c90b3c94,c0f7aaef,c61582d0,c0fc0b40,...) at _witness_debugger+0x25 witness_checkorder(c90b3c94,9,c0fc0b40,72b,c90b3d00,...) at witness_checkorder+0x86f __lockmgr_args(c90b3c94,80400,c90b3d00,0,0,...) at __lockmgr_args+0x834 vop_stdlock(ee8e9ac0,4,c0f8ac4b,80400,c90b3c3c,...) at vop_stdlock+0x65 VOP_LOCK1_APV(c10b19e0,ee8e9ac0,c12a5ee0,c10ecd00,c90b3c3c,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c90b3c3c,80400,c0fc0b40,72b,c90d35a8,...) at _vn_lock+0x78 softdep_flushworklist(c90d35a8,ee8e9b6c,ca187000,5e9,c0f9c554,...) at softdep_flushworklist+0x47 ffs_sync(c90d35a8,1,c0f9bba7,51a,0,...) at ffs_sync+0x32d dounmount(c90d35a8,8000000,ca187000,49c,0,...) at dounmount+0x3f8 sys_unmount(ca187000,ee8e9cec,c,c,206,...) at sys_unmount+0x421 syscall(ee8e9d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280dcb3f, esp = 0xbfbfe1bc, ebp = 0xbfbfe288 --- panic: dqflush: stray dquot 0xc61fc800 cpuid = 1 KDB: enter: panic [ thread pid 5169 tid 100176 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> run pho db:0:pho> bt Tracing pid 5169 tid 100176 td 0xca187000 kdb_enter(c0f8c870,c0f8c870,c0fc53af,ee8e9a30,1,...) at kdb_enter+0x3a panic(c0fc53af,c61fc800,c0fc51b1,64f,0,...) at panic+0x1ac quotaoff1(ee8e9aa8,c0a4a601,c88c8948,4,c0f8ac4b,...) at quotaoff1+0x320 quotaoff_inchange(c88c8948,0,c0fc51b1,2fe,0,...) at quotaoff_inchange+0x18 quotaoff(ca187000,c90d35a8,0,ca187000,c88c8900,...) at quotaoff+0xc6 ffs_flushfiles(c90d35a8,0,ca187000,ee8e9b34,3,...) at ffs_flushfiles+0x6b softdep_flushfiles(c90d35a8,0,ca187000,0,2123f5e,...) at softdep_flushfiles+0x2e ffs_unmount(c90d35a8,8000000,c0f9bba7,51a,0,...) at ffs_unmount+0x1bf dounmount(c90d35a8,8000000,ca187000,49c,0,...) at dounmount+0x419 sys_unmount(ca187000,ee8e9cec,c,c,206,...) at sys_unmount+0x421 syscall(ee8e9d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280dcb3f, esp = 0xbfbfe1bc, ebp = 0xbfbfe288 --- db:0:bt> show allpcpu Current CPU: 1 cpuid = 0 dynamic pcpu = 0x737b00 curthread = 0xc61b88a0: pid 11 "idle: cpu0" curpcb = 0xc5eb2d80 fpcurthread = none idlethread = 0xc61b88a0: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 dynamic pcpu = 0x4dbab00 curthread = 0xca187000: pid 5169 "umount" curpcb = 0xee8e9d80 fpcurthread = none idlethread = 0xc61b85c0: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: db:0:allpcpu> show alllocks Process 5169 (umount) thread 0xca187000 (100176) exclusive sleep mutex dqhlock (dqhlock) r = 0 (0xc12a5fe8) locked @ /mnt/pho/src/sys/ufs/ufs/ufs_quota.c:1615 exclusive sleep mutex Giant (Giant) r = 0 (0xc1128110) locked @ /mnt/pho/src/sys/kern/vfs_mount.c:1164 exclusive lockmgr newnfs (newnfs) r = 0 (0xc8ea0b38) locked @ /mnt/pho/src/sys/kern/vfs_mount.c:1249 Process 1784 (sshd) thread 0xc8cb52e0 (100073) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xc8f6f70c) locked @ /mnt/pho/src/sys/kern/uipc_sockbuf.c:148 db:0:alllocks> show lockedvnods Locked vnodes 0xc8ea0ae0: tag newnfs, type VDIR usecount 1, writecount 0, refcount 1 mountedhere 0xc90d35a8 flags () v_object 0xc8f7f7e0 ref 0 pages 0 lock type newnfs: EXCL by thread 0xca187000 (pid 5169) #0 0xc0a426ee at __lockmgr_args+0xc1e #1 0xc09ab39d at nfs_lock1+0xcd #2 0xc0de2035 at VOP_LOCK1_APV+0xb5 #3 0xc0b08528 at _vn_lock+0x78 #4 0xc0af0d7c at dounmount+0x8c #5 0xc0af17d1 at sys_unmount+0x421 #6 0xc0dbecd7 at syscall+0x2c7 #7 0xc0da8001 at Xint0x80_syscall+0x21 fileid 19383889 fsid 0x3a00ff01 db:0:lockedvnods> show mount 0xc891db50 192.168.5.1:/zoo/leopard3 on / (nfs) 0xc891e000 devfs on /dev (devfs) 0xc891d87c /dev/ada0d on /work (ufs) 0xc891d5a8 procfs on /proc (procfs) 0xc891d2d4 /dev/md0 on /var (ufs) 0xc891d000 /dev/md1 on /tmp (ufs) 0xc90d35a8 /dev/md5a on /mnt (ufs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 5169 1799 1799 0 R+ CPU 1 umount 1812 0 0 0 DL mdwait 0xc90c9800 [md5] 1799 1796 1799 0 S+ wait 0xc8d64b10 sh 1796 1792 1796 0 S+ wait 0xc8f76000 sh 1792 1791 1792 0 S+ pause 0xc8db78a4 csh 1791 1790 1791 1001 S+ wait 0xc8d6484c su 1790 1789 1790 1001 Ss+ wait 0xc8f75b10 sh 1789 1784 1784 1001 S select 0xc884bb64 sshd 1784 1678 1784 0 Ss sbwait 0xc8f6f738 sshd 1773 1 1773 0 Ss+ ttyin 0xc6262270 getty 1772 1 1772 0 Ss+ ttyin 0xc6263670 getty 1771 1 1771 0 Ss+ ttyin 0xc6263870 getty 1770 1 1770 0 Ss+ ttyin 0xc6263a70 getty 1769 1 1769 0 Ss+ ttyin 0xc6263c70 getty 1768 1 1768 0 Ss+ ttyin 0xc6263e70 getty 1767 1 1767 0 Ss+ ttyin 0xc643f070 getty 1766 1 1766 0 Ss+ ttyin 0xc643f270 getty 1765 1 1765 0 Ss+ ttyin 0xc643f470 getty 1727 1 1727 0 Ss select 0xc88f79a4 inetd 1699 1 1699 0 Ss nanslp 0xc1129a64 watchdogd 1689 1 1689 0 Ss nanslp 0xc1129a64 cron 1678 1 1678 0 Ss select 0xc88f75a4 sshd 1537 1 1537 0 Ss select 0xc645c3a4 rpc.statd 1530 1529 1529 0 S (threaded) nfsd 100116 S rpcsvc 0xc8be4a50 nfsd: service 100115 S rpcsvc 0xc88d0290 nfsd: service 100114 S rpcsvc 0xc88d0250 nfsd: service 100084 S rpcsvc 0xc62d2650 nfsd: master 1529 1 1529 0 Ss select 0xc884bc64 nfsd 1520 1 1520 0 Ss select 0xc645c3e4 mountd 1454 0 0 0 DL mdwait 0xc8e81000 [md1] 1399 1 1399 0 Ss select 0xc645c2e4 rpcbind 1369 1 1369 0 Ss select 0xc884b764 syslogd 1190 1 1190 0 Ss select 0xc88f7664 devd 103 0 0 0 DL mdwait 0xc8e82000 [md0] 18 0 0 0 DL sdflush 0xc12a5f20 [softdepflush] 17 0 0 0 DL vlruwt 0xc88e7588 [vnlru] 16 0 0 0 DL syncer 0xc129fc34 [syncer] 9 0 0 0 DL psleep 0xc129f8c8 [bufdaemon] 8 0 0 0 DL pgzero 0xc12a7ddc [pagezero] 7 0 0 0 DL psleep 0xc12a79c8 [vmdaemon] 6 0 0 0 DL psleep 0xc12a7990 [pagedaemon] 5 0 0 0 DL ccb_scan 0xc10ee254 [xpt_thrd] 4 0 0 0 DL waiting_ 0xc12a1758 [sctp_iterator] 3 0 0 0 DL ctl_work 0xc6462000 [ctl_thrd] 2 0 0 0 DL - 0xc634de3c [fdc0] 15 0 0 0 DL (threaded) [usb] 100051 D - 0xc63edd34 [usbus4] 100050 D - 0xc63edd04 [usbus4] 100049 D - 0xc63edcd4 [usbus4] 100048 D - 0xc63edca4 [usbus4] 100047 D - 0xc63dadac [usbus3] 100046 D - 0xc63dad7c [usbus3] 100045 D - 0xc63dad4c [usbus3] 100044 D - 0xc63dad1c [usbus3] 100042 D - 0xc63c5dac [usbus2] 100041 D - 0xc63c5d7c [usbus2] 100040 D - 0xc63c5d4c [usbus2] 100039 D - 0xc63c5d1c [usbus2] 100037 D - 0xc63b2dac [usbus1] 100036 D - 0xc63b2d7c [usbus1] 100035 D - 0xc63b2d4c [usbus1] 100034 D - 0xc63b2d1c [usbus1] 100032 D - 0xc6398dac [usbus0] 100031 D - 0xc6398d7c [usbus0] 100030 D - 0xc6398d4c [usbus0] 100029 D - 0xc6398d1c [usbus0] 14 0 0 0 DL - 0xc1128864 [yarrow] 13 0 0 0 DL (threaded) [geom] 100011 D - 0xc1126468 [g_down] 100010 D - 0xc1126464 [g_up] 100009 D - 0xc112645c [g_event] 12 0 0 0 WL (threaded) [intr] 100056 I [irq1: atkbd0] 100054 I [swi0: uart uart] 100053 I [irq258: ahci0] 100052 I [irq14: ata0] 100043 I [irq16: uhci3] 100038 I [irq18: uhci2] 100033 I [irq19: uhci1] 100028 I [irq23: uhci0 ehci0] 100023 I [swi6: task queue] 100022 I [swi6: Giant taskq] 100020 I [swi5: +] 100019 I [swi2: cambio] 100008 I [swi3: vm] 100007 I [swi4: clock] 100006 I [swi4: clock] 100005 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100004 CanRun [idle: cpu1] 100003 Run CPU 0 [idle: cpu0] 1 0 1 0 SLs wait 0xc61b6b10 [init] 10 0 0 0 DL audit_wo 0xc12a5660 [audit] 0 0 0 0 DLs (threaded) [kernel] 100068 D - 0xc6248280 [mca taskq] 100058 D - 0xc1128864 [deadlkres] 100027 D - 0xc62485c0 [em1 txq] 100026 D - 0xc6248600 [em1 que] 100025 D - 0xc6248640 [em0 txq] 100024 D - 0xc6248680 [em0 que] 100021 D - 0xc6248740 [thread taskq] 100018 D - 0xc62487c0 [kqueue taskq] 100017 D - 0xc6248800 [ffs_trim taskq] 100016 D - 0xc6248840 [acpi_task_2] 100015 D - 0xc6248840 [acpi_task_1] 100014 D - 0xc6248840 [acpi_task_0] 100012 D - 0xc6248880 [firmware taskq] 100000 D sched 0xc1126540 [swapper] db:0:ps> allt Tracing command umount pid 5169 tid 100176 td 0xca187000 kdb_enter(c0f8c870,c0f8c870,c0fc53af,ee8e9a30,1,...) at kdb_enter+0x3a panic(c0fc53af,c61fc800,c0fc51b1,64f,0,...) at panic+0x1ac quotaoff1(ee8e9aa8,c0a4a601,c88c8948,4,c0f8ac4b,...) at quotaoff1+0x320 quotaoff_inchange(c88c8948,0,c0fc51b1,2fe,0,...) at quotaoff_inchange+0x18 quotaoff(ca187000,c90d35a8,0,ca187000,c88c8900,...) at quotaoff+0xc6 ffs_flushfiles(c90d35a8,0,ca187000,ee8e9b34,3,...) at ffs_flushfiles+0x6b softdep_flushfiles(c90d35a8,0,ca187000,0,2123f5e,...) at softdep_flushfiles+0x2e ffs_unmount(c90d35a8,8000000,c0f9bba7,51a,0,...) at ffs_unmount+0x1bf dounmount(c90d35a8,8000000,ca187000,49c,0,...) at dounmount+0x419 sys_unmount(ca187000,ee8e9cec,c,c,206,...) at sys_unmount+0x421 syscall(ee8e9d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280dcb3f, esp = 0xbfbfe1bc, ebp = 0xbfbfe288 --- Tracing command md5 pid 1812 tid 100136 td 0xc90cf5c0 sched_switch(c90cf5c0,0,104,1a3,93d31d08,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c90cf5c0,0,c0f92360,268,0,...) at sleepq_switch+0x162 sleepq_wait(c90c9800,5c,c0f28caa,0,0,...) at sleepq_wait+0x63 _sleep(c90c9800,c90c9820,25c,c0f28caa,0,...) at _sleep+0x39c md_kthread(c90c9800,ee84ed28,c0f86f10,3db,c90c484c,...) at md_kthread+0x11a fork_exit(c07692a0,c90c9800,ee84ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee84ed60, ebp = 0 --- Tracing command sh pid 1799 tid 100081 td 0xc8cb2000 sched_switch(c8cb2000,0,104,1a3,64143830,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c8cb2000,0,c0f92360,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f92360,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c8d64b10,6c,c0f952df,100,0,...) at sleepq_wait_sig+0x17 _sleep(c8d64b10,c8d64b98,16c,c0f952df,0,...) at _sleep+0x385 kern_wait(c8cb2000,ffffffff,ee76ac60,2,0,...) at kern_wait+0x626 sys_wait4(c8cb2000,ee76acec,c0fdec82,c0f952be,212,...) at sys_wait4+0x3b syscall(ee76ad28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28161ffb, esp = 0xbfbfe8bc, ebp = 0xbfbfe8d8 --- Tracing command sh pid 1796 tid 100122 td 0xc8f7bb80 sched_switch(c8f7bb80,0,104,1a3,c41b0f18,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c8f7bb80,0,c0f92360,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f92360,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c8f76000,6c,c0f952df,100,0,...) at sleepq_wait_sig+0x17 _sleep(c8f76000,c8f76088,16c,c0f952df,0,...) at _sleep+0x385 kern_wait(c8f7bb80,ffffffff,ee824c60,2,0,...) at kern_wait+0x626 sys_wait4(c8f7bb80,ee824cec,c,c,212,...) at sys_wait4+0x3b syscall(ee824d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28161ffb, esp = 0xbfbfe9ac, ebp = 0xbfbfe9c8 --- Tracing command csh pid 1792 tid 100093 td 0xc8d8c5c0 sched_switch(c8d8c5c0,0,104,1a3,cae88d88,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,74,...) at mi_switch+0x1ff sleepq_switch(c8d8c5c0,0,c0f92360,1a7,74,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f92360,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c8db78a4,74,c0e1318e,100,0,...) at sleepq_wait_sig+0x17 _sleep(c8db78a4,c8db78d4,174,c0e1318e,0,...) at _sleep+0x385 kern_sigsuspend(c8d8c5c0,0,0,0,0,...) at kern_sigsuspend+0xba sys_sigsuspend(c8d8c5c0,ee7a3cec,c0fdec82,c0f951ce,202,...) at sys_sigsuspend+0x4d syscall(ee7a3d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, sys_write), eip = 0x2817d8fb, esp = 0xbfbfe74c, ebp = 0xbfbfe7b8 --- Tracing command su pid 1791 tid 100082 td 0xc88ecb80 sched_switch(c88ecb80,0,104,1a3,2fb94b90,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c88ecb80,0,c0f92360,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f92360,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c8d6484c,6c,c0f952df,100,0,...) at sleepq_wait_sig+0x17 _sleep(c8d6484c,c8d648d4,16c,c0f952df,0,...) at _sleep+0x385 kern_wait(c88ecb80,700,ee770c60,2,0,...) at kern_wait+0x626 sys_wait4(c88ecb80,ee770cec,c,c,212,...) at sys_wait4+0x3b syscall(ee770d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x2812cffb, esp = 0xbfbfe77c, ebp = 0xbfbfe798 --- Tracing command sh pid 1790 tid 100123 td 0xc8f7b8a0 sched_switch(c8f7b8a0,0,104,1a3,2a89bf58,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c8f7b8a0,0,c0f92360,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f92360,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c8f75b10,6c,c0f952df,100,0,...) at sleepq_wait_sig+0x17 _sleep(c8f75b10,c8f75b98,16c,c0f952df,0,...) at _sleep+0x385 kern_wait(c8f7b8a0,ffffffff,ee827c60,2,0,...) at kern_wait+0x626 sys_wait4(c8f7b8a0,ee827cec,c,c,212,...) at sys_wait4+0x3b syscall(ee827d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28161ffb, esp = 0xbfbfeaac, ebp = 0xbfbfeac8 --- Tracing command sshd pid 1789 tid 100087 td 0xc8d8d8a0 sched_switch(c8d8d8a0,0,104,1a3,3fbb8c0,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8d8d8a0,0,c0f92360,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a4a68a,c884bb50,0,c0f8ac4b,c8d8d8a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c884bb64,0,ee786a58,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c884bb64,c884bb50,c0f94af8,6a4,c8cc1e00,...) at _cv_wait_sig+0x260 seltdwait(c8cc1e00,58,c6202700,c8d8d8a0,ee786acc,...) at seltdwait+0xa2 kern_select(c8d8d8a0,a,288060b8,288060cc,0,0,20,c8d6384c,0) at kern_select+0x655 sys_select(c8d8d8a0,ee786cec,c0fdec82,c0f5a9e7,206,...) at sys_select+0x66 syscall(ee786d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x284b0d43, esp = 0xbfbfe19c, ebp = 0xbfbfe1e8 --- Tracing command sshd pid 1784 tid 100073 td 0xc8cb52e0 sched_switch(c8cb52e0,0,104,1a3,549446f0,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,68,...) at mi_switch+0x1ff sleepq_switch(c8cb52e0,0,c0f92360,1a7,68,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f92360,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c8f6f738,68,c0f98a98,100,0,...) at sleepq_wait_sig+0x17 _sleep(c8f6f738,c8f6f6f8,168,c0f98a98,0) at _sleep+0x385 sbwait(c8f6f6d4,4,c0f98b90,5ee,c8f6f6f8,...) at sbwait+0x76 soreceive_generic(c8f6f680,0,ee748c44,0,0,...) at soreceive_generic+0x320 soreceive(c8f6f680,0,ee748c44,0,0,0) at soreceive+0x38 soo_read(c8cc13f0,ee748c44,c891fd80,0,c8cb52e0,...) at soo_read+0x4e dofileread(ee748c44,ffffffff,ffffffff,0,ee748c20,...) at dofileread+0x9e kern_readv(c8cb52e0,5,ee748c44,ee748c64,1,...) at kern_readv+0x68 sys_read(c8cb52e0,ee748cec,c0fdec82,c0f7d0c1,286,...) at sys_read+0x4f syscall(ee748d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x284b0dc3, esp = 0xbfbfdd6c, ebp = 0xbfbfdda8 --- Tracing command getty pid 1773 tid 100107 td 0xc8dbdb80 sched_switch(c8dbdb80,0,104,1a3,14f860e0,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8dbdb80,0,c0f92360,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a4a68a,c6262204,0,c0f8ac4b,c8dbdb80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c6262270,0,ee7ceaf0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c6262270,c6262204,c0f96be8,527,0,...) at _cv_wait_sig+0x260 tty_wait(c6262200,c6262270,ee7cec44,1,0,...) at tty_wait+0x71 ttydisc_read(c6262200,ee7cec44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c62b8800,ee7cec44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c8cd2dc8,ee7cec44,c6197e80,0,c8dbdb80,...) at devfs_read_f+0x7e dofileread(ee7cec44,ffffffff,ffffffff,0,ee7cec20,...) at dofileread+0x9e kern_readv(c8dbdb80,0,ee7cec44,ee7cec64,1,...) at kern_readv+0x68 sys_read(c8dbdb80,ee7cecec,c0fdec82,c0df15ce,246,...) at sys_read+0x4f syscall(ee7ced28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1772 tid 100078 td 0xc8cb28a0 sched_switch(c8cb28a0,0,104,1a3,7a9751e8,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8cb28a0,0,c0f92360,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f92360,160,0,c8cb28a0,c8cb28a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c6263670,0,c0f97228,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c6263670,c1128110,c0f96be8,527,0,...) at _cv_wait_sig+0x260 tty_wait(c6263600,c6263670,ee75ec44,1,0,...) at tty_wait+0x71 ttydisc_read(c6263600,ee75ec44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c630c600,ee75ec44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c8cc09d8,ee75ec44,c6197e80,0,c8cb28a0,...) at devfs_read_f+0x7e dofileread(ee75ec44,ffffffff,ffffffff,0,ee75ec20,...) at dofileread+0x9e kern_readv(c8cb28a0,0,ee75ec44,ee75ec64,1,...) at kern_readv+0x68 sys_read(c8cb28a0,ee75ecec,c0fdec82,c0df15ce,246,...) at sys_read+0x4f syscall(ee75ed28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1771 tid 100098 td 0xc8d8b5c0 sched_switch(c8d8b5c0,0,104,1a3,7a8ef7f0,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8d8b5c0,0,c0f92360,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f92360,160,0,c8d8b5c0,c8d8b5c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c6263870,0,c0f97228,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c6263870,c1128110,c0f96be8,527,0,...) at _cv_wait_sig+0x260 tty_wait(c6263800,c6263870,ee7b2c44,1,0,...) at tty_wait+0x71 ttydisc_read(c6263800,ee7b2c44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c630c700,ee7b2c44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c8cd2738,ee7b2c44,c6197e80,0,c8d8b5c0,...) at devfs_read_f+0x7e dofileread(ee7b2c44,ffffffff,ffffffff,0,ee7b2c20,...) at dofileread+0x9e kern_readv(c8d8b5c0,0,ee7b2c44,ee7b2c64,1,...) at kern_readv+0x68 sys_read(c8d8b5c0,ee7b2cec,c0fdec82,c0df15ce,246,...) at sys_read+0x4f syscall(ee7b2d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1770 tid 100095 td 0xc8d8c000 sched_switch(c8d8c000,0,104,1a3,7adeacc0,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8d8c000,0,c0f92360,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f92360,160,0,c8d8c000,c8d8c000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c6263a70,0,c0f97228,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c6263a70,c1128110,c0f96be8,527,0,...) at _cv_wait_sig+0x260 tty_wait(c6263a00,c6263a70,ee7a9c44,1,0,...) at tty_wait+0x71 ttydisc_read(c6263a00,ee7a9c44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c630c800,ee7a9c44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c8cc0578,ee7a9c44,c6197e80,0,c8d8c000,...) at devfs_read_f+0x7e dofileread(ee7a9c44,ffffffff,ffffffff,0,ee7a9c20,...) at dofileread+0x9e kern_readv(c8d8c000,0,ee7a9c44,ee7a9c64,1,...) at kern_readv+0x68 sys_read(c8d8c000,ee7a9cec,c0fdec82,c0df15ce,246,...) at sys_read+0x4f syscall(ee7a9d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1769 tid 100089 td 0xc8d8d2e0 sched_switch(c8d8d2e0,0,104,1a3,7ae4f468,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8d8d2e0,0,c0f92360,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f92360,160,0,c8d8d2e0,c8d8d2e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c6263c70,0,c0f97228,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c6263c70,c1128110,c0f96be8,527,0,...) at _cv_wait_sig+0x260 tty_wait(c6263c00,c6263c70,ee78fc44,1,0,...) at tty_wait+0x71 ttydisc_read(c6263c00,ee78fc44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c630c900,ee78fc44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c8cd2188,ee78fc44,c6197e80,0,c8d8d2e0,...) at devfs_read_f+0x7e dofileread(ee78fc44,ffffffff,ffffffff,0,ee78fc20,...) at dofileread+0x9e kern_readv(c8d8d2e0,0,ee78fc44,ee78fc64,1,...) at kern_readv+0x68 sys_read(c8d8d2e0,ee78fcec,c0fdec82,c0df15ce,246,...) at sys_read+0x4f syscall(ee78fd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1768 tid 100105 td 0xc8dc02e0 sched_switch(c8dc02e0,0,104,1a3,7ad0ebe8,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8dc02e0,0,c0f92360,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f92360,160,0,c8dc02e0,c8dc02e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c6263e70,0,c0f97228,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c6263e70,c1128110,c0f96be8,527,0,...) at _cv_wait_sig+0x260 tty_wait(c6263e00,c6263e70,ee7c8c44,1,0,...) at tty_wait+0x71 ttydisc_read(c6263e00,ee7c8c44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c630ca00,ee7c8c44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c8cd20a8,ee7c8c44,c6197e80,0,c8dc02e0,...) at devfs_read_f+0x7e dofileread(ee7c8c44,ffffffff,ffffffff,0,ee7c8c20,...) at dofileread+0x9e kern_readv(c8dc02e0,0,ee7c8c44,ee7c8c64,1,...) at kern_readv+0x68 sys_read(c8dc02e0,ee7c8cec,c0fdec82,c0df15ce,246,...) at sys_read+0x4f syscall(ee7c8d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1767 tid 100076 td 0xc88eab80 sched_switch(c88eab80,0,104,1a3,7a9d8200,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c88eab80,0,c0f92360,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f92360,160,0,c88eab80,c88eab80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c643f070,0,c0f97228,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c643f070,c1128110,c0f96be8,527,0,...) at _cv_wait_sig+0x260 tty_wait(c643f000,c643f070,ee754c44,1,0,...) at tty_wait+0x71 ttydisc_read(c643f000,ee754c44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c630cb00,ee754c44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c8cc0b28,ee754c44,c6197e80,0,c88eab80,...) at devfs_read_f+0x7e dofileread(ee754c44,ffffffff,ffffffff,0,ee754c20,...) at dofileread+0x9e kern_readv(c88eab80,0,ee754c44,ee754c64,1,...) at kern_readv+0x68 sys_read(c88eab80,ee754cec,c0fdec82,ee754d80,246,...) at sys_read+0x4f syscall(ee754d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1766 tid 100106 td 0xc8dc0000 sched_switch(c8dc0000,0,104,1a3,7ae02630,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8dc0000,0,c0f92360,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f92360,160,0,c8dc0000,c8dc0000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c643f270,0,c0f97228,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c643f270,c1128110,c0f96be8,527,0,...) at _cv_wait_sig+0x260 tty_wait(c643f200,c643f270,ee7cbc44,1,0,...) at tty_wait+0x71 ttydisc_read(c643f200,ee7cbc44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c630cc00,ee7cbc44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c8cd2c08,ee7cbc44,c6197e80,0,c8dc0000,...) at devfs_read_f+0x7e dofileread(ee7cbc44,ffffffff,ffffffff,0,ee7cbc20,...) at dofileread+0x9e kern_readv(c8dc0000,0,ee7cbc44,ee7cbc64,1,...) at kern_readv+0x68 sys_read(c8dc0000,ee7cbcec,c0fdec82,c0df15ce,246,...) at sys_read+0x4f syscall(ee7cbd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1765 tid 100125 td 0xc8f7b2e0 sched_switch(c8f7b2e0,0,104,1a3,7ac04618,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8f7b2e0,0,c0f92360,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f92360,160,0,c8f7b2e0,c8f7b2e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c643f470,0,c0f97228,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c643f470,c1128110,c0f96be8,527,0,...) at _cv_wait_sig+0x260 tty_wait(c643f400,c643f470,ee82dc44,1,0,...) at tty_wait+0x71 ttydisc_read(c643f400,ee82dc44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c630cd00,ee82dc44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c8cd2a10,ee82dc44,c6197e80,0,c8f7b2e0,...) at devfs_read_f+0x7e dofileread(ee82dc44,ffffffff,ffffffff,0,ee82dc20,...) at dofileread+0x9e kern_readv(c8f7b2e0,0,ee82dc44,ee82dc64,1,...) at kern_readv+0x68 sys_read(c8f7b2e0,ee82dcec,c0fdec82,c0df15ce,246,...) at sys_read+0x4f syscall(ee82dd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command inetd pid 1727 tid 100083 td 0xc88ec8a0 sched_switch(c88ec8a0,0,104,1a3,667c5b80,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c88ec8a0,0,c0f92360,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a4a68a,c88f7990,0,c0f8ac4b,c88ec8a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c88f79a4,0,ee774a58,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c88f79a4,c88f7990,c0f94af8,6a4,c8cd2e00,...) at _cv_wait_sig+0x260 seltdwait(c8cd2e00,58,c6197e80,c88ec8a0,c0a4a601,...) at seltdwait+0xa2 kern_select(c88ec8a0,8,bfbfdfa0,0,0,0,20,c8d64588,4) at kern_select+0x655 sys_select(c88ec8a0,ee774cec,c0fdec82,c0f9532b,202,...) at sys_select+0x66 syscall(ee774d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281a8d43, esp = 0xbfbfdf5c, ebp = 0xbfbfede8 --- Tracing command watchdogd pid 1699 tid 100092 td 0xc8d8c8a0 sched_switch(c8d8c8a0,0,104,1a3,cb6363b8,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c8d8c8a0,0,c0f92360,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(ea61,c0a9b270,c8d8c8a0,1,100,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c1129a64,6c,c0f8de2b,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c1129a64,0,16c,c0f8de2b,ea61,...) at _sleep+0x34f kern_nanosleep(c8d8c8a0,ee7a0c50,ee7a0c58,3c,0,...) at kern_nanosleep+0xc1 sys_nanosleep(c8d8c8a0,ee7a0cec,c0fdec82,c0df15ce,206,...) at sys_nanosleep+0x6f syscall(ee7a0d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x281920b7, esp = 0xbfbfecbc, ebp = 0xbfbfece8 --- Tracing command cron pid 1689 tid 100129 td 0xc8f7a5c0 sched_switch(c8f7a5c0,0,104,1a3,55509238,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c8f7a5c0,0,c0f92360,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(ea61,c0a9b270,c8f7a5c0,1,100,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c1129a64,6c,c0f8de2b,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c1129a64,0,16c,c0f8de2b,ea61,...) at _sleep+0x34f kern_nanosleep(c8f7a5c0,ee839c50,ee839c58,3c,0,...) at kern_nanosleep+0xc1 sys_nanosleep(c8f7a5c0,ee839cec,c0fdec82,c0f9578c,206,...) at sys_nanosleep+0x6f syscall(ee839d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x281850b7, esp = 0xbfbfec4c, ebp = 0xbfbfec78 --- Tracing command sshd pid 1678 tid 100117 td 0xc8dbc5c0 sched_switch(c8dbc5c0,0,104,1a3,52ef08e0,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8dbc5c0,0,c0f92360,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a4a68a,c88f7590,0,c0f8ac4b,c8dbc5c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c88f75a4,0,ee815a58,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c88f75a4,c88f7590,c0f94af8,6a4,c8cc1b60,...) at _cv_wait_sig+0x260 seltdwait(c8cc1b60,58,c6202880,c8dbc5c0,c8c4f828,...) at seltdwait+0xa2 kern_select(c8dbc5c0,7,2880c0b0,0,0,0,20,c8f77000,a) at kern_select+0x655 sys_select(c8dbc5c0,ee815cec,c,c,202,...) at sys_select+0x66 syscall(ee815d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x284b0d43, esp = 0xbfbfe25c, ebp = 0xbfbfedf8 --- Tracing command rpc.statd pid 1537 tid 100080 td 0xc8cb22e0 sched_switch(c8cb22e0,0,104,1a3,378ef130,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8cb22e0,0,c0f92360,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(ee766a20,c0a4a68a,c645c390,0,c8cb22e0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c645c3a4,0,ee766a58,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c645c3a4,c645c390,7531,6a4,c8cc00e0,...) at _cv_timedwait_sig+0x270 seltdwait(ee766c10,ee766c18,c6197e80,c8cb22e0,c8ee02b8,...) at seltdwait+0x8a kern_select(c8cb22e0,8,bfbfed14,0,0,ee766c5c,20,1e,0) at kern_select+0x655 sys_select(c8cb22e0,ee766cec,c0fdec82,c0f9541c,202,...) at sys_select+0x66 syscall(ee766d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x2818cd43, esp = 0xbfbfec6c, ebp = 0xbfbfeda8 --- Tracing command nfsd pid 1530 tid 100116 td 0xc88ea000 sched_switch(c88ea000,0,104,1a3,9f2ba7b8,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c88ea000,0,c0f92360,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(ee812be0,c0a4a68a,c8929500,0,c88ea000,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c8be4a50,0,ee812c18,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c8be4a50,c8929500,1388,3b1,0,...) at _cv_timedwait_sig+0x270 svc_run_internal(ee812d14,c0a2d3d8,c8929500,ee812d28,c0f86f10,...) at svc_run_internal+0x356 svc_thread_start(c8929500,ee812d28,c0f86f10,3db,c8d642c4,...) at svc_thread_start+0x10 fork_exit(c0c4a140,c8929500,ee812d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0x804c72e, eip = 0xc, esp = 0x33, ebp = 0 --- Tracing command nfsd pid 1530 tid 100115 td 0xc88ea2e0 sched_switch(c88ea2e0,0,104,1a3,aa593018,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c88ea2e0,0,c0f92360,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(ee80fbe0,c0a4a68a,c8929500,0,c88ea2e0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c88d0290,0,ee80fc18,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c88d0290,c8929500,1388,3b1,0,...) at _cv_timedwait_sig+0x270 svc_run_internal(ee80fd14,c0a2d3d8,c8929500,ee80fd28,c0f86f10,...) at svc_run_internal+0x356 svc_thread_start(c8929500,ee80fd28,c0f86f10,3db,c8d642c4,...) at svc_thread_start+0x10 fork_exit(c0c4a140,c8929500,ee80fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0x804c72e, eip = 0xc, esp = 0x33, ebp = 0 --- Tracing command nfsd pid 1530 tid 100114 td 0xc88ea5c0 sched_switch(c88ea5c0,0,104,1a3,659da960,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c88ea5c0,0,c0f92360,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(ee80cbe0,c0a4a68a,c8929500,0,c88ea5c0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c88d0250,0,ee80cc18,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c88d0250,c8929500,1388,3b1,8051c40,...) at _cv_timedwait_sig+0x270 svc_run_internal(ee80cd14,c0a2d3d8,c8929500,ee80cd28,c0f86f10,...) at svc_run_internal+0x356 svc_thread_start(c8929500,ee80cd28,c0f86f10,3db,c8d642c4,...) at svc_thread_start+0x10 fork_exit(c0c4a140,c8929500,ee80cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0x804c72e, eip = 0xc, esp = 0x33, ebp = 0 --- Tracing command nfsd pid 1530 tid 100084 td 0xc88ec5c0 sched_switch(c88ec5c0,0,104,1a3,e4751eb0,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c88ec5c0,0,c0f92360,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(ee779a40,c0a4a68a,c8929500,0,c88ec5c0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c62d2650,0,ee779a78,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c62d2650,c8929500,1388,3b1,ee779ac0,...) at _cv_timedwait_sig+0x270 svc_run_internal(c88ec770,14,c0fbbfc2,c8f1f9cf,ee779c2c,...) at svc_run_internal+0x356 svc_run(c8929500,0,c8f1fe17,1bf,c0a667b8,...) at svc_run+0x7b nfssvc_nfsd(bfbfe870,ee779c2c,c,c6197e80,ee779c40,...) at nfssvc_nfsd+0x175 nfssvc_nfsserver(c88ec5c0,ee779cec,bfbfe870,c88ec5c0,ee779d1c,...) at nfssvc_nfsserver+0x55 sys_nfssvc(c88ec5c0,ee779cec,c0fdec82,c0f956eb,202,...) at sys_nfssvc+0x89 syscall(ee779d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (155, FreeBSD ELF32, sys_nfssvc), eip = 0x280dd4fb, esp = 0xbfbfe82c, ebp = 0xbfbfea98 --- Tracing command nfsd pid 1529 tid 100101 td 0xc8dc1000 sched_switch(c8dc1000,0,104,1a3,1aa2a4d8,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8dc1000,0,c0f92360,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a4a68a,c884bc50,0,c0f8ac4b,c8dc1000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c884bc64,0,ee7bba58,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c884bc64,c884bc50,c0f94af8,6a4,c8cd22a0,...) at _cv_wait_sig+0x260 seltdwait(c8cd22a0,58,c6197e80,c8dc1000,c61bf658,...) at seltdwait+0xa2 kern_select(c8dc1000,5,bfbfecb0,0,0,0,20,c8db5000,1) at kern_select+0x655 sys_select(c8dc1000,ee7bbcec,c,c,202,...) at sys_select+0x66 syscall(ee7bbd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x28184d43, esp = 0xbfbfea9c, ebp = 0xbfbfede8 --- Tracing command mountd pid 1520 tid 100090 td 0xc8d8d000 sched_switch(c8d8d000,0,104,1a3,d36b56c8,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8d8d000,0,c0f92360,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a4a68a,c645c3d0,0,c0f8ac4b,c8d8d000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c645c3e4,0,ee793a58,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c645c3e4,c645c3d0,c0f94af8,6a4,c8cd2cb0,...) at _cv_wait_sig+0x260 seltdwait(c8cd2cb0,58,c6197e80,c8d8d000,478,...) at seltdwait+0xa2 kern_select(c8d8d000,9,bfbfed4c,0,0,0,20,c8d63000,bfbfed4c) at kern_select+0x655 sys_select(c8d8d000,ee793cec,c0fdec82,c0f7d0c1,206,...) at sys_select+0x66 syscall(ee793d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x2819ad43, esp = 0xbfbfed1c, ebp = 0xbfbfede8 --- Tracing command md1 pid 1454 tid 100094 td 0xc8d8c2e0 sched_switch(c8d8c2e0,0,104,1a3,64868ed0,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c8d8c2e0,0,c0f92360,268,0,...) at sleepq_switch+0x162 sleepq_wait(c8e81000,5c,c0f28caa,0,0,...) at sleepq_wait+0x63 _sleep(c8e81000,c8e81020,25c,c0f28caa,0,...) at _sleep+0x39c md_kthread(c8e81000,ee7a6d28,c0f86f10,3db,c8db7588,...) at md_kthread+0x11a fork_exit(c07692a0,c8e81000,ee7a6d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee7a6d60, ebp = 0 --- Tracing command rpcbind pid 1399 tid 100113 td 0xc8dbc8a0 sched_switch(c8dbc8a0,0,104,1a3,42d7e088,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8dbc8a0,0,c0f92360,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(ee7e0a84,c0a4a68a,c645c2d0,0,c8dbc8a0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c645c2e4,0,ee7e0abc,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c645c2e4,c645c2d0,7531,6a4,ee7e0b74,...) at _cv_timedwait_sig+0x270 seltdwait(ee7e0c44,ee7e0c4c,5a1,c8dbc8a0,c6154ac0,...) at seltdwait+0x8a sys_poll(c8dbc8a0,ee7e0cec,c0fdec82,c0f9541c,202,...) at sys_poll+0x321 syscall(ee7e0d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x2813ff4b, esp = 0xbfbfcbac, ebp = 0xbfbfed78 --- Tracing command syslogd pid 1369 tid 100112 td 0xc8dbcb80 sched_switch(c8dbcb80,0,104,1a3,7f31fbb0,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8dbcb80,0,c0f92360,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a4a68a,c884b750,0,c0f8ac4b,c8dbcb80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c884b764,0,ee7dda58,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c884b764,c884b750,c0f94af8,6a4,c8cd2af0,...) at _cv_wait_sig+0x260 seltdwait(c8cd2af0,58,c6197e80,c8dbcb80,ee7ddaf4,...) at seltdwait+0xa2 kern_select(c8dbcb80,9,284760ac,0,0,0,20,c8db8588,0) at kern_select+0x655 sys_select(c8dbcb80,ee7ddcec,c0fdec82,c0fc1817,246,...) at sys_select+0x66 syscall(ee7ddd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x28199d43, esp = 0xbfbfe29c, ebp = 0xbfbfede8 --- Tracing command devd pid 1190 tid 100085 td 0xc88ec2e0 sched_switch(c88ec2e0,0,104,1a3,d2c06ab8,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c88ec2e0,0,c0f92360,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a4a68a,c88f7650,0,c0f8ac4b,c88ec2e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c88f7664,0,ee77da58,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c88f7664,c88f7650,c0f94af8,6a4,c8cc1770,...) at _cv_wait_sig+0x260 seltdwait(c8cc1770,58,c6197e80,c88ec2e0,c1272c18,...) at seltdwait+0xa2 kern_select(c88ec2e0,5,bfbfe960,0,0,0,20,c8d64000,46) at kern_select+0x655 sys_select(c88ec2e0,ee77dcec,c0fdec82,c0dec87f,206,...) at sys_select+0x66 syscall(ee77dd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x808ba43, esp = 0xbfbfe92c, ebp = 0xbfbfee08 --- Tracing command md0 pid 103 tid 100111 td 0xc8dbd000 sched_switch(c8dbd000,0,104,1a3,7f316b18,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c8dbd000,0,c0f92360,268,0,...) at sleepq_switch+0x162 sleepq_wait(c8e82000,5c,c0f28caa,0,0,...) at sleepq_wait+0x63 _sleep(c8e82000,c8e82020,25c,c0f28caa,0,...) at _sleep+0x39c md_kthread(c8e82000,ee7dad28,c0f86f10,3db,c8db884c,...) at md_kthread+0x11a fork_exit(c07692a0,c8e82000,ee7dad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee7dad60, ebp = 0 --- Tracing command softdepflush pid 18 tid 100067 td 0xc88eb5c0 sched_switch(c88eb5c0,0,104,1a3,69d76d80,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,54,...) at mi_switch+0x1ff sleepq_switch(c88eb5c0,0,c0f92360,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c12a5f20,54,c0fc3088,0,0,...) at sleepq_timedwait+0x6b _sleep(c12a5f20,c12a5ee0,54,c0fc3088,3e8,...) at _sleep+0x36a softdep_flush(0,ec546d28,c0f86f10,3db,c88e72c4,...) at softdep_flush+0x266 fork_exit(c0c99570,0,ec546d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec546d60, ebp = 0 --- Tracing command vnlru pid 17 tid 100066 td 0xc88eb8a0 sched_switch(c88eb8a0,0,104,1a3,5207c200,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,60,...) at mi_switch+0x1ff sleepq_switch(c88eb8a0,0,c0f92360,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c88e7588,60,c0f9d4cd,0,0,...) at sleepq_timedwait+0x6b _sleep(c88e7588,c129fbf4,260,c0f9d4cd,3e8,...) at _sleep+0x36a vnlru_proc(0,ec543d28,c0f86f10,3db,c88e7588,...) at vnlru_proc+0xe7 fork_exit(c0afbfc0,0,ec543d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec543d60, ebp = 0 --- Tracing command syncer pid 16 tid 100065 td 0xc88ebb80 sched_switch(c88ebb80,0,104,1a3,4b6f57a0,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c88ebb80,0,c0f92360,28b,c88ebb80,...) at sleepq_switch+0x162 sleepq_timedwait(c129fc34,0,ec540c78,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c129fc34,c129fc20,3e8,78b,4e20,...) at _cv_timedwait+0x270 sched_sync(0,ec540d28,c0f86f10,3db,c88e784c,...) at sched_sync+0x595 fork_exit(c0afb330,0,ec540d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec540d60, ebp = 0 --- Tracing command bufdaemon pid 9 tid 100064 td 0xc88ec000 sched_switch(c88ec000,0,104,1a3,47a19278,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,54,...) at mi_switch+0x1ff sleepq_switch(c88ec000,0,c0f92360,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c129f8c8,54,c0f9a3a9,0,0,...) at sleepq_timedwait+0x6b _sleep(c129f8c8,c129f8d0,54,c0f9a3a9,3e8,...) at _sleep+0x36a buf_daemon(0,ec53dd28,c0f86f10,3db,c88e7b10,...) at buf_daemon+0x175 fork_exit(c0adfb20,0,ec53dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec53dd60, ebp = 0 --- Tracing command pagezero pid 8 tid 100063 td 0xc63d62e0 sched_switch(c63d62e0,0,104,1a3,876bdc80,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d62e0,0,c0f92360,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c12a7ddc,0,c0fca1a6,0,0,...) at sleepq_timedwait+0x6b _sleep(c12a7ddc,c12a6880,0,c0fca1a6,493e0,...) at _sleep+0x36a vm_pagezero(0,ec53ad28,c0f86f10,3db,c88e9000,...) at vm_pagezero+0xdc fork_exit(c0ce0730,0,ec53ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec53ad60, ebp = 0 --- Tracing command vmdaemon pid 7 tid 100062 td 0xc63d65c0 sched_switch(c63d65c0,0,104,1a3,d76f17b0,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,74,...) at mi_switch+0x1ff sleepq_switch(c63d65c0,0,c0f92360,268,0,...) at sleepq_switch+0x162 sleepq_wait(c12a79c8,74,c0f9a3a9,0,0,...) at sleepq_wait+0x63 _sleep(c12a79c8,c12a79cc,74,c0f9a3a9,0,...) at _sleep+0x39c vm_daemon(0,ec537d28,c0f86f10,3db,c61b72c4,...) at vm_daemon+0x59 fork_exit(c0cda040,0,ec537d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec537d60, ebp = 0 --- Tracing command pagedaemon pid 6 tid 100061 td 0xc63d68a0 sched_switch(c63d68a0,0,104,1a3,d10dbe0,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,54,...) at mi_switch+0x1ff sleepq_switch(c63d68a0,0,c0f92360,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c12a7990,54,c0f9a3a9,0,0,...) at sleepq_timedwait+0x6b _sleep(c12a7990,c12a6880,54,c0f9a3a9,1388,...) at _sleep+0x36a vm_pageout(0,ec534d28,c0f86f10,3db,c61b7588,...) at vm_pageout+0x2c7 fork_exit(c0cdb100,0,ec534d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec534d60, ebp = 0 --- Tracing command xpt_thrd pid 5 tid 100060 td 0xc63d6b80 sched_switch(c63d6b80,0,104,1a3,fdea54c8,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c63d6b80,0,c0f92360,268,0,...) at sleepq_switch+0x162 sleepq_wait(c10ee254,5c,c0de7077,0,0,...) at sleepq_wait+0x63 _sleep(c10ee254,c10ee298,5c,c0de7077,0,...) at _sleep+0x39c xpt_scanner_thread(0,ec531d28,c0f86f10,3db,c61b784c,...) at xpt_scanner_thread+0x47 fork_exit(c049d300,0,ec531d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec531d60, ebp = 0 --- Tracing command sctp_iterator pid 4 tid 100059 td 0xc640d000 sched_switch(c640d000,0,104,1a3,eedd3c68,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c640d000,0,c0f92360,268,0,...) at sleepq_switch+0x162 sleepq_wait(c12a1758,0,c0fb0d1f,0,0,...) at sleepq_wait+0x63 _sleep(c12a1758,c12a1720,0,c0fb0d1f,0,...) at _sleep+0x39c sctp_iterator_thread(0,ec52ed28,c0f86f10,3db,c61b7b10,...) at sctp_iterator_thread+0x5c fork_exit(c0b8d6a0,0,ec52ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec52ed60, ebp = 0 --- Tracing command ctl_thrd pid 3 tid 100057 td 0xc640d5c0 sched_switch(c640d5c0,0,104,1a3,dd326388,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c640d5c0,0,c0f92360,268,0,...) at sleepq_switch+0x162 sleepq_wait(c6462000,5c,c0df1aaa,0,0,...) at sleepq_wait+0x63 _sleep(c6462000,c6462000,5c,c0df1aaa,0,...) at _sleep+0x39c ctl_work_thread(c6462000,ec528d28,c0f86f10,3db,c63a8000,...) at ctl_work_thread+0x22f6 fork_exit(c04c6700,c6462000,ec528d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec528d60, ebp = 0 --- Tracing command fdc0 pid 2 tid 100055 td 0xc640db80 sched_switch(c640db80,0,104,1a3,9399c760,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c640db80,0,c0f92360,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c634de3c,5c,c0f7f0da,0,0,...) at sleepq_timedwait+0x6b _sleep(c634de3c,c634def0,5c,c0f7f0da,3e8,...) at _sleep+0x36a fdc_thread(c634de00,ec522d28,c0f86f10,3db,c63a82c4,...) at fdc_thread+0x27d fork_exit(c0d5f980,c634de00,ec522d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec522d60, ebp = 0 --- Tracing command usb pid 15 tid 100051 td 0xc63d4000 sched_switch(c63d4000,0,104,1a3,dd2ccdc8,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d4000,0,c0f92360,268,c63d4000,...) at sleepq_switch+0x162 sleepq_wait(c63edd34,0,c5fbbcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63edd34,c63ede4c,c0f654d1,74,c63edd3c,...) at _cv_wait+0x260 usb_process(c63edd2c,c5fbbd28,c0f86f10,3db,c63a8588,...) at usb_process+0x1a3 fork_exit(c08fd190,c63edd2c,c5fbbd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fbbd60, ebp = 0 --- Tracing command usb pid 15 tid 100050 td 0xc63d42e0 sched_switch(c63d42e0,0,104,1a3,bc452bc8,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d42e0,0,c0f92360,268,c63d42e0,...) at sleepq_switch+0x162 sleepq_wait(c63edd04,0,c5fb8cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63edd04,c63ede4c,c0f654d1,74,c63edd0c,...) at _cv_wait+0x260 usb_process(c63edcfc,c5fb8d28,c0f86f10,3db,c63a8588,...) at usb_process+0x1a3 fork_exit(c08fd190,c63edcfc,c5fb8d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fb8d60, ebp = 0 --- Tracing command usb pid 15 tid 100049 td 0xc63d45c0 sched_switch(c63d45c0,0,104,1a3,dcaf7968,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d45c0,0,c0f92360,268,c63d45c0,...) at sleepq_switch+0x162 sleepq_wait(c63edcd4,0,c5fb5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63edcd4,c63ede4c,c0f654d1,74,c63edcdc,...) at _cv_wait+0x260 usb_process(c63edccc,c5fb5d28,c0f86f10,3db,c63a8588,...) at usb_process+0x1a3 fork_exit(c08fd190,c63edccc,c5fb5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fb5d60, ebp = 0 --- Tracing command usb pid 15 tid 100048 td 0xc63d48a0 sched_switch(c63d48a0,0,104,1a3,dcaf6440,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d48a0,0,c0f92360,268,c63d48a0,...) at sleepq_switch+0x162 sleepq_wait(c63edca4,0,c5fb2cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63edca4,c63ede4c,c0f654d1,74,c63edcac,...) at _cv_wait+0x260 usb_process(c63edc9c,c5fb2d28,c0f86f10,3db,c63a8588,...) at usb_process+0x1a3 fork_exit(c08fd190,c63edc9c,c5fb2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fb2d60, ebp = 0 --- Tracing command usb pid 15 tid 100047 td 0xc63d4b80 sched_switch(c63d4b80,0,104,1a3,dcaf4ff0,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d4b80,0,c0f92360,268,c63d4b80,...) at sleepq_switch+0x162 sleepq_wait(c63dadac,0,c5faccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63dadac,c63daec4,c0f654d1,74,c63dadb4,...) at _cv_wait+0x260 usb_process(c63dada4,c5facd28,c0f86f10,3db,c63a8588,...) at usb_process+0x1a3 fork_exit(c08fd190,c63dada4,c5facd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5facd60, ebp = 0 --- Tracing command usb pid 15 tid 100046 td 0xc63d5000 sched_switch(c63d5000,0,104,1a3,1def8ec8,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d5000,0,c0f92360,268,c63d5000,...) at sleepq_switch+0x162 sleepq_wait(c63dad7c,0,c5fa9cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63dad7c,c63daec4,c0f654d1,74,c63dad84,...) at _cv_wait+0x260 usb_process(c63dad74,c5fa9d28,c0f86f10,3db,c63a8588,...) at usb_process+0x1a3 fork_exit(c08fd190,c63dad74,c5fa9d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fa9d60, ebp = 0 --- Tracing command usb pid 15 tid 100045 td 0xc63d52e0 sched_switch(c63d52e0,0,104,1a3,d59e9ae0,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d52e0,0,c0f92360,268,c63d52e0,...) at sleepq_switch+0x162 sleepq_wait(c63dad4c,0,c5fa6cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63dad4c,c63daec4,c0f654d1,74,c63dad54,...) at _cv_wait+0x260 usb_process(c63dad44,c5fa6d28,c0f86f10,3db,c63a8588,...) at usb_process+0x1a3 fork_exit(c08fd190,c63dad44,c5fa6d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fa6d60, ebp = 0 --- Tracing command usb pid 15 tid 100044 td 0xc63d55c0 sched_switch(c63d55c0,0,104,1a3,d59e8740,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d55c0,0,c0f92360,268,c63d55c0,...) at sleepq_switch+0x162 sleepq_wait(c63dad1c,0,c5fa3cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63dad1c,c63daec4,c0f654d1,74,c63dad24,...) at _cv_wait+0x260 usb_process(c63dad14,c5fa3d28,c0f86f10,3db,c63a8588,...) at usb_process+0x1a3 fork_exit(c08fd190,c63dad14,c5fa3d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fa3d60, ebp = 0 --- Tracing command usb pid 15 tid 100042 td 0xc63d5b80 sched_switch(c63d5b80,0,104,1a3,d59e6e38,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d5b80,0,c0f92360,268,c63d5b80,...) at sleepq_switch+0x162 sleepq_wait(c63c5dac,0,c5f9ccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63c5dac,c63c5ec4,c0f654d1,74,c63c5db4,...) at _cv_wait+0x260 usb_process(c63c5da4,c5f9cd28,c0f86f10,3db,c63a8588,...) at usb_process+0x1a3 fork_exit(c08fd190,c63c5da4,c5f9cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f9cd60, ebp = 0 --- Tracing command usb pid 15 tid 100041 td 0xc63d6000 sched_switch(c63d6000,0,104,1a3,1d4d25f0,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d6000,0,c0f92360,268,c63d6000,...) at sleepq_switch+0x162 sleepq_wait(c63c5d7c,0,c5f99cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63c5d7c,c63c5ec4,c0f654d1,74,c63c5d84,...) at _cv_wait+0x260 usb_process(c63c5d74,c5f99d28,c0f86f10,3db,c63a8588,...) at usb_process+0x1a3 fork_exit(c08fd190,c63c5d74,c5f99d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f99d60, ebp = 0 --- Tracing command usb pid 15 tid 100040 td 0xc63a9000 sched_switch(c63a9000,0,104,1a3,d5248578,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63a9000,0,c0f92360,268,c63a9000,...) at sleepq_switch+0x162 sleepq_wait(c63c5d4c,0,c5f96cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63c5d4c,c63c5ec4,c0f654d1,74,c63c5d54,...) at _cv_wait+0x260 usb_process(c63c5d44,c5f96d28,c0f86f10,3db,c63a8588,...) at usb_process+0x1a3 fork_exit(c08fd190,c63c5d44,c5f96d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f96d60, ebp = 0 --- Tracing command usb pid 15 tid 100039 td 0xc63a92e0 sched_switch(c63a92e0,0,104,1a3,d52471b8,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63a92e0,0,c0f92360,268,c63a92e0,...) at sleepq_switch+0x162 sleepq_wait(c63c5d1c,0,c5f93cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63c5d1c,c63c5ec4,c0f654d1,74,c63c5d24,...) at _cv_wait+0x260 usb_process(c63c5d14,c5f93d28,c0f86f10,3db,c63a8588,...) at usb_process+0x1a3 fork_exit(c08fd190,c63c5d14,c5f93d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f93d60, ebp = 0 --- Tracing command usb pid 15 tid 100037 td 0xc63a98a0 sched_switch(c63a98a0,0,104,1a3,d5245938,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63a98a0,0,c0f92360,268,c63a98a0,...) at sleepq_switch+0x162 sleepq_wait(c63b2dac,0,c5f8ccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63b2dac,c63b2ec4,c0f654d1,74,c63b2db4,...) at _cv_wait+0x260 usb_process(c63b2da4,c5f8cd28,c0f86f10,3db,c63a8588,...) at usb_process+0x1a3 fork_exit(c08fd190,c63b2da4,c5f8cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f8cd60, ebp = 0 --- Tracing command usb pid 15 tid 100036 td 0xc63a9b80 sched_switch(c63a9b80,0,104,1a3,1caadf28,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63a9b80,0,c0f92360,268,c63a9b80,...) at sleepq_switch+0x162 sleepq_wait(c63b2d7c,0,c5f89cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63b2d7c,c63b2ec4,c0f654d1,74,c63b2d84,...) at _cv_wait+0x260 usb_process(c63b2d74,c5f89d28,c0f86f10,3db,c63a8588,...) at usb_process+0x1a3 fork_exit(c08fd190,c63b2d74,c5f89d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f89d60, ebp = 0 --- Tracing command usb pid 15 tid 100035 td 0xc63ac000 sched_switch(c63ac000,0,104,1a3,d4aa3b08,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63ac000,0,c0f92360,268,c63ac000,...) at sleepq_switch+0x162 sleepq_wait(c63b2d4c,0,c5f86cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63b2d4c,c63b2ec4,c0f654d1,74,c63b2d54,...) at _cv_wait+0x260 usb_process(c63b2d44,c5f86d28,c0f86f10,3db,c63a8588,...) at usb_process+0x1a3 fork_exit(c08fd190,c63b2d44,c5f86d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f86d60, ebp = 0 --- Tracing command usb pid 15 tid 100034 td 0xc63ac2e0 sched_switch(c63ac2e0,0,104,1a3,d4aa26d0,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63ac2e0,0,c0f92360,268,c63ac2e0,...) at sleepq_switch+0x162 sleepq_wait(c63b2d1c,0,c5f83cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63b2d1c,c63b2ec4,c0f654d1,74,c63b2d24,...) at _cv_wait+0x260 usb_process(c63b2d14,c5f83d28,c0f86f10,3db,c63a8588,...) at usb_process+0x1a3 fork_exit(c08fd190,c63b2d14,c5f83d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f83d60, ebp = 0 --- Tracing command usb pid 15 tid 100032 td 0xc63ac8a0 sched_switch(c63ac8a0,0,104,1a3,d4aa0a78,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63ac8a0,0,c0f92360,268,c63ac8a0,...) at sleepq_switch+0x162 sleepq_wait(c6398dac,0,c5f7ccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c6398dac,c6398ec4,c0f654d1,74,c6398db4,...) at _cv_wait+0x260 usb_process(c6398da4,c5f7cd28,c0f86f10,3db,c63a8588,...) at usb_process+0x1a3 fork_exit(c08fd190,c6398da4,c5f7cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f7cd60, ebp = 0 --- Tracing command usb pid 15 tid 100031 td 0xc63acb80 sched_switch(c63acb80,0,104,1a3,1c091580,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63acb80,0,c0f92360,268,c63acb80,...) at sleepq_switch+0x162 sleepq_wait(c6398d7c,0,c5f79cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c6398d7c,c6398ec4,c0f654d1,74,c6398d84,...) at _cv_wait+0x260 usb_process(c6398d74,c5f79d28,c0f86f10,3db,c63a8588,...) at usb_process+0x1a3 fork_exit(c08fd190,c6398d74,c5f79d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f79d60, ebp = 0 --- Tracing command usb pid 15 tid 100030 td 0xc63332e0 sched_switch(c63332e0,0,104,1a3,d42f62d0,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63332e0,0,c0f92360,268,c63332e0,...) at sleepq_switch+0x162 sleepq_wait(c6398d4c,0,c5f76cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c6398d4c,c6398ec4,c0f654d1,74,c6398d54,...) at _cv_wait+0x260 usb_process(c6398d44,c5f76d28,c0f86f10,3db,c63a8588,...) at usb_process+0x1a3 fork_exit(c08fd190,c6398d44,c5f76d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f76d60, ebp = 0 --- Tracing command usb pid 15 tid 100029 td 0xc63335c0 sched_switch(c63335c0,0,104,1a3,d42f4d00,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63335c0,0,c0f92360,268,c63335c0,...) at sleepq_switch+0x162 sleepq_wait(c6398d1c,0,c5f73cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c6398d1c,c6398ec4,c0f654d1,74,c6398d24,...) at _cv_wait+0x260 usb_process(c6398d14,c5f73d28,c0f86f10,3db,c63a8588,...) at usb_process+0x1a3 fork_exit(c08fd190,c6398d14,c5f73d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f73d60, ebp = 0 --- Tracing command yarrow pid 14 tid 100013 td 0xc61bd2e0 sched_switch(c61bd2e0,0,104,1a3,8bbf9220,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c61bd2e0,0,c0f92360,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c1128864,0,c0f7f0da,2,0,...) at sleepq_timedwait+0x6b _sleep(c1128864,0,0,c0f7f0da,64) at _sleep+0x36a pause(c0f7f0da,64,c0f41130,111,0,...) at pause+0xa0 random_kthread(0,c5ed7d28,c0f86f10,3db,c61b6000,...) at random_kthread+0x1ef fork_exit(c0805670,0,c5ed7d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ed7d60, ebp = 0 --- Tracing command geom pid 13 tid 100011 td 0xc61bd8a0 sched_switch(c61bd8a0,0,104,1a3,93d2dfe8,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c61bd8a0,0,c0f92360,268,0,...) at sleepq_switch+0x162 sleepq_wait(c1126468,5c,c0f7f0da,0,0,...) at sleepq_wait+0x63 _sleep(c1126468,c11263c8,25c,c0f7f0da,0,...) at _sleep+0x39c g_io_schedule_down(c61bd8a0,0,c0f80c81,6c,c5ecdd14,...) at g_io_schedule_down+0x5b g_down_procbody(0,c5ecdd28,c0f86f10,3db,c61b62c4,...) at g_down_procbody+0x9d fork_exit(c09ee4b0,0,c5ecdd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ecdd60, ebp = 0 --- Tracing command geom pid 13 tid 100010 td 0xc61bdb80 sched_switch(c61bdb80,0,104,1a3,93d35878,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c61bdb80,0,c0f92360,268,0,...) at sleepq_switch+0x162 sleepq_wait(c1126464,5c,c0f7f0da,0,0,...) at sleepq_wait+0x63 _sleep(c1126464,c11263e8,25c,c0f7f0da,0,...) at _sleep+0x39c g_io_schedule_up(c61bdb80,0,c0f80c81,5f,c5ecad14,...) at g_io_schedule_up+0x134 g_up_procbody(0,c5ecad28,c0f86f10,3db,c61b62c4,...) at g_up_procbody+0x9d fork_exit(c09ee550,0,c5ecad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ecad60, ebp = 0 --- Tracing command geom pid 13 tid 100009 td 0xc6203000 sched_switch(c6203000,0,104,1a3,d2bd8c78,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c6203000,0,c0f92360,268,0,...) at sleepq_switch+0x162 sleepq_wait(c112645c,5c,c0f7f0da,0,0,...) at sleepq_wait+0x63 _sleep(c112645c,c112638c,25c,c0f7f0da,0,...) at _sleep+0x39c g_run_events(c11366c0,0,c0f80c81,79,c5ec7d14,...) at g_run_events+0x54b g_event_procbody(0,c5ec7d28,c0f86f10,3db,c61b62c4,...) at g_event_procbody+0x90 fork_exit(c09ee5f0,0,c5ec7d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ec7d60, ebp = 0 --- Tracing command intr pid 12 tid 100056 td 0xc640d8a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100054 td 0xc6410000 sched_switch(c6410000,0,109,1a3,14f849a8,...) at sched_switch+0x3ec mi_switch(109,0,c0f8724b,554,c6410000,...) at mi_switch+0x1ff ithread_loop(c640a360,ec515d28,c0f86f10,3db,c61b6588,...) at ithread_loop+0x1fb fork_exit(c0a30ef0,c640a360,ec515d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec515d60, ebp = 0 --- Tracing command intr pid 12 tid 100053 td 0xc64102e0 sched_switch(c64102e0,0,109,1a3,648795a0,...) at sched_switch+0x3ec mi_switch(109,0,c0f8724b,554,c64102e0,...) at mi_switch+0x1ff ithread_loop(c6405b60,c5fe5d28,c0f86f10,3db,c61b6588,...) at ithread_loop+0x1fb fork_exit(c0a30ef0,c6405b60,c5fe5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fe5d60, ebp = 0 --- Tracing command intr pid 12 tid 100052 td 0xc64105c0 sched_switch(c64105c0,0,109,1a3,d7806b68,...) at sched_switch+0x3ec mi_switch(109,0,c0f8724b,554,c64105c0,...) at mi_switch+0x1ff ithread_loop(c640a040,c5fe1d28,c0f86f10,3db,c61b6588,...) at ithread_loop+0x1fb fork_exit(c0a30ef0,c640a040,c5fe1d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fe1d60, ebp = 0 --- Tracing command intr pid 12 tid 100043 td 0xc63d58a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100038 td 0xc63a95c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100033 td 0xc63ac5c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100028 td 0xc63338a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100023 td 0xc63368a0 sched_switch(c63368a0,0,109,1a3,5bf7b310,...) at sched_switch+0x3ec mi_switch(109,0,c0f8724b,554,c63368a0,...) at mi_switch+0x1ff ithread_loop(c618a4b0,c5ef5d28,c0f86f10,3db,c61b6588,...) at ithread_loop+0x1fb fork_exit(c0a30ef0,c618a4b0,c5ef5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ef5d60, ebp = 0 --- Tracing command intr pid 12 tid 100022 td 0xc6336b80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100020 td 0xc6203b80 sched_switch(c6203b80,0,109,1a3,5b8f6898,...) at sched_switch+0x3ec mi_switch(109,0,c0f8724b,554,c6203b80,...) at mi_switch+0x1ff ithread_loop(c618a4d0,c5eecd28,c0f86f10,3db,c61b6588,...) at ithread_loop+0x1fb fork_exit(c0a30ef0,c618a4d0,c5eecd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5eecd60, ebp = 0 --- Tracing command intr pid 12 tid 100019 td 0xc6332000 sched_switch(c6332000,0,109,1a3,6487c3c0,...) at sched_switch+0x3ec mi_switch(109,0,c0f8724b,554,c6332000,...) at mi_switch+0x1ff ithread_loop(c618a4e0,c5ee9d28,c0f86f10,3db,c61b6588,...) at ithread_loop+0x1fb fork_exit(c0a30ef0,c618a4e0,c5ee9d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ee9d60, ebp = 0 --- Tracing command intr pid 12 tid 100008 td 0xc62032e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100007 td 0xc62035c0 sched_switch(c62035c0,0,109,1a3,8bbf7880,...) at sched_switch+0x3ec mi_switch(109,0,c0f8724b,554,c62035c0,...) at mi_switch+0x1ff ithread_loop(c618a720,c5ec1d28,c0f86f10,3db,c61b6588,...) at ithread_loop+0x1fb fork_exit(c0a30ef0,c618a720,c5ec1d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ec1d60, ebp = 0 --- Tracing command intr pid 12 tid 100006 td 0xc61b8000 sched_switch(c61b8000,0,109,1a3,9399a628,...) at sched_switch+0x3ec mi_switch(109,0,c0f8724b,554,c61b8000,...) at mi_switch+0x1ff ithread_loop(c618a730,c5ebed28,c0f86f10,3db,c61b6588,...) at ithread_loop+0x1fb fork_exit(c0a30ef0,c618a730,c5ebed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ebed60, ebp = 0 --- Tracing command intr pid 12 tid 100005 td 0xc61b82e0 sched_switch(c61b82e0,0,109,1a3,982a87d8,...) at sched_switch+0x3ec mi_switch(109,0,c0f8724b,554,c61b82e0,...) at mi_switch+0x1ff ithread_loop(c618a740,c5ebbd28,c0f86f10,3db,c61b6588,...) at ithread_loop+0x1fb fork_exit(c0a30ef0,c618a740,c5ebbd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ebbd60, ebp = 0 --- Tracing command idle pid 11 tid 100004 td 0xc61b85c0 sched_switch(c61b85c0,0,108,19e,93d34d68,...) at sched_switch+0x3ec mi_switch(108,0,c0f8e9d2,a09,c61b85c0,...) at mi_switch+0x1ff sched_idletd(0,c5eb5d28,c0f86f10,3db,c61b684c,...) at sched_idletd+0x2bf fork_exit(c0a81b00,0,c5eb5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5eb5d60, ebp = 0 --- Tracing command idle pid 11 tid 100003 td 0xc61b88a0 cpustop_handler(0,13,c5eb2c38,c0dbf0d6,f5,...) at cpustop_handler+0x28 ipi_nmi_handler(f5,0,1c9c810,c0daee8d,c61b684c,...) at ipi_nmi_handler+0x47 trap(c5eb2c44) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0daee8d, esp = 0xc5eb2c84, ebp = 0xc5eb2c88 --- cpu_idle_mwait(1,2710,c11366c0,1,c5eb2ce8,...) at cpu_idle_mwait+0x4d cpu_idle(1,c5eb2cc4,c0f8e9d2,a0b,c61b88a0,...) at cpu_idle+0x30 sched_idletd(0,c5eb2d28,c0f86f10,3db,c61b684c,...) at sched_idletd+0x263 fork_exit(c0a81b00,0,c5eb2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5eb2d60, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc61b8b80 sched_switch(c61b8b80,0,104,1a3,a89d4e20,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c61b8b80,0,c0f92360,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f92360,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c61b6b10,6c,c0f952df,100,0,...) at sleepq_wait_sig+0x17 _sleep(c61b6b10,c61b6b98,16c,c0f952df,0,...) at _sleep+0x385 kern_wait(c61b8b80,ffffffff,c5eaec60,0,0,...) at kern_wait+0x626 sys_wait4(c61b8b80,c5eaecec,c0fdec82,c0f95116,286,...) at sys_wait4+0x3b syscall(c5eaed28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x80551d3, esp = 0xbfbfe8bc, ebp = 0xbfbfe8d8 --- Tracing command audit pid 10 tid 100001 td 0xc61bd000 sched_switch(c61bd000,0,104,1a3,d42b1180,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c61bd000,0,c0f92360,268,c61bd000,...) at sleepq_switch+0x162 sleepq_wait(c12a5660,0,c5eabc8c,1,0,...) at sleepq_wait+0x63 _cv_wait(c12a5660,c12a5644,c0fbda8f,194,0,...) at _cv_wait+0x260 audit_worker(0,c5eabd28,c0f86f10,3db,c61b7000,...) at audit_worker+0x84 fork_exit(c0c5a490,0,c5eabd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5eabd60, ebp = 0 --- Tracing command kernel pid 0 tid 100068 td 0xc88eb2e0 sched_switch(c88eb2e0,0,104,1a3,d7685e48,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c88eb2e0,0,c0f92360,268,0,...) at sleepq_switch+0x162 sleepq_wait(c6248280,0,c0f7f0da,0,0,...) at sleepq_wait+0x63 _sleep(c6248280,c6248298,0,c0f7f0da,0,...) at _sleep+0x39c taskqueue_thread_loop(c1306ee0,ec549d28,c0f86f10,3db,c1126540,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a9e970,c1306ee0,ec549d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec549d60, ebp = 0 --- Tracing command kernel pid 0 tid 100058 td 0xc640d2e0 sched_switch(c640d2e0,0,104,1a3,ca98bee0,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c640d2e0,0,c0f92360,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c1128864,0,c0f7f0da,2,0,...) at sleepq_timedwait+0x6b _sleep(c1128864,0,0,c0f7f0da,bb8) at _sleep+0x36a pause(c0f7f0da,bb8,124,122,c11265c8,...) at pause+0xa0 deadlkres(0,ec52bd28,c0f86f10,3db,c1126540,...) at deadlkres+0x324 fork_exit(c0a106b0,0,ec52bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec52bd60, ebp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xc6333b80 sched_switch(c6333b80,0,104,1a3,d4284b88,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c6333b80,0,c0f92360,268,c6333b80,...) at sleepq_switch+0x162 sleepq_wait(c62485c0,0,c0f8d5f9,c0f7f0da,0,...) at sleepq_wait+0x63 msleep_spin(c62485c0,c62485d8,c0f7f0da,0,c0f8ac4b,...) at msleep_spin+0x259 taskqueue_thread_loop(c635b66c,c5f6cd28,c0f86f10,3db,c1126540,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9e970,c635b66c,c5f6cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f6cd60, ebp = 0 --- Tracing command kernel pid 0 tid 100026 td 0xc6336000 sched_switch(c6336000,0,104,1a3,d4283a78,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c6336000,0,c0f92360,268,c6336000,...) at sleepq_switch+0x162 sleepq_wait(c6248600,0,c0f8d5f9,c0f7f0da,0,...) at sleepq_wait+0x63 msleep_spin(c6248600,c6248618,c0f7f0da,0,c0f8ac4b,...) at msleep_spin+0x259 taskqueue_thread_loop(c6385544,c5f69d28,c0f86f10,3db,c1126540,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9e970,c6385544,c5f69d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f69d60, ebp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xc63362e0 sched_switch(c63362e0,0,104,1a3,d4282a70,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63362e0,0,c0f92360,268,c63362e0,...) at sleepq_switch+0x162 sleepq_wait(c6248640,0,c0f8d5f9,c0f7f0da,0,...) at sleepq_wait+0x63 msleep_spin(c6248640,c6248658,c0f7f0da,0,c0f8ac4b,...) at msleep_spin+0x259 taskqueue_thread_loop(c635ba6c,c5f3ed28,c0f86f10,3db,c1126540,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9e970,c635ba6c,c5f3ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f3ed60, ebp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xc63365c0 sched_switch(c63365c0,0,104,1a3,7f2538c8,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63365c0,0,c0f92360,268,c63365c0,...) at sleepq_switch+0x162 sleepq_wait(c6248680,0,c0f8d5f9,c0f7f0da,0,...) at sleepq_wait+0x63 msleep_spin(c6248680,c6248698,c0f7f0da,0,c0f8ac4b,...) at msleep_spin+0x259 taskqueue_thread_loop(c6371544,c5f3bd28,c0f86f10,3db,c1126540,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9e970,c6371544,c5f3bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f3bd60, ebp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xc62038a0 sched_switch(c62038a0,0,104,1a3,3a13dba8,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c62038a0,0,c0f92360,268,0,...) at sleepq_switch+0x162 sleepq_wait(c6248740,0,c0f7f0da,0,0,...) at sleepq_wait+0x63 _sleep(c6248740,c6248758,0,c0f7f0da,0,...) at _sleep+0x39c taskqueue_thread_loop(c114542c,c5eefd28,c0f86f10,3db,c1126540,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a9e970,c114542c,c5eefd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5eefd60, ebp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xc63322e0 sched_switch(c63322e0,0,104,1a3,fdebb9d0,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63322e0,0,c0f92360,268,0,...) at sleepq_switch+0x162 sleepq_wait(c62487c0,0,c0f7f0da,0,0,...) at sleepq_wait+0x63 _sleep(c62487c0,c62487d8,0,c0f7f0da,0,...) at _sleep+0x39c taskqueue_thread_loop(c1126eb8,c5ee6d28,c0f86f10,3db,c1126540,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a9e970,c1126eb8,c5ee6d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ee6d60, ebp = 0 --- Tracing command kernel pid 0 tid 100017 td 0xc63325c0 sched_switch(c63325c0,0,104,1a3,fdebab48,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63325c0,0,c0f92360,268,0,...) at sleepq_switch+0x162 sleepq_wait(c6248800,0,c0f7f0da,0,0,...) at sleepq_wait+0x63 _sleep(c6248800,c6248818,0,c0f7f0da,0,...) at _sleep+0x39c taskqueue_thread_loop(c12a5c44,c5ee3d28,c0f86f10,3db,c1126540,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a9e970,c12a5c44,c5ee3d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ee3d60, ebp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xc63328a0 sched_switch(c63328a0,0,104,1a3,fdeb9678,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63328a0,0,c0f92360,268,c63328a0,...) at sleepq_switch+0x162 sleepq_wait(c6248840,0,c0f8d5f9,c0f7f0da,0,...) at sleepq_wait+0x63 msleep_spin(c6248840,c6248858,c0f7f0da,0,c0f8ac4b,...) at msleep_spin+0x259 taskqueue_thread_loop(c10f14b8,c5ee0d28,c0f86f10,3db,c1126540,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9e970,c10f14b8,c5ee0d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ee0d60, ebp = 0 --- Tracing command kernel pid 0 tid 100015 td 0xc6332b80 sched_switch(c6332b80,0,104,1a3,fdeb8708,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c6332b80,0,c0f92360,268,c6332b80,...) at sleepq_switch+0x162 sleepq_wait(c6248840,0,c0f8d5f9,c0f7f0da,0,...) at sleepq_wait+0x63 msleep_spin(c6248840,c6248858,c0f7f0da,0,c0f8ac4b,...) at msleep_spin+0x259 taskqueue_thread_loop(c10f14b8,c5eddd28,c0f86f10,3db,c1126540,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9e970,c10f14b8,c5eddd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5eddd60, ebp = 0 --- Tracing command kernel pid 0 tid 100014 td 0xc6333000 sched_switch(c6333000,0,104,1a3,fdeb7838,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c6333000,0,c0f92360,268,c6333000,...) at sleepq_switch+0x162 sleepq_wait(c6248840,0,c0f8d5f9,c0f7f0da,0,...) at sleepq_wait+0x63 msleep_spin(c6248840,c6248858,c0f7f0da,0,c0f8ac4b,...) at msleep_spin+0x259 taskqueue_thread_loop(c10f14b8,c5edad28,c0f86f10,3db,c1126540,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9e970,c10f14b8,c5edad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5edad60, ebp = 0 --- Tracing command kernel pid 0 tid 100012 td 0xc61bd5c0 sched_switch(c61bd5c0,0,104,1a3,674081e0,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c61bd5c0,0,c0f92360,268,0,...) at sleepq_switch+0x162 sleepq_wait(c6248880,0,c0f7f0da,0,0,...) at sleepq_wait+0x63 _sleep(c6248880,c6248898,0,c0f7f0da,0,...) at _sleep+0x39c taskqueue_thread_loop(c1143e40,c5ed0d28,c0f86f10,3db,c1126540,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a9e970,c1143e40,c5ed0d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ed0d60, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc1126810 sched_switch(c1126810,0,104,1a3,f129d6d0,...) at sched_switch+0x3ec mi_switch(104,0,c0f92360,1f3,54,...) at mi_switch+0x1ff sleepq_switch(c1126810,0,c0f92360,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c1126540,54,c0f8f2ec,0,0,...) at sleepq_timedwait+0x6b _sleep(c1126540,0,54,c0f8f2ec,2710,...) at _sleep+0x36a scheduler(0,181ec00,181ec00,181e000,1825000,...) at scheduler+0x240 mi_startup() at mi_startup+0xac begin() at begin+0x2c db:0:allt> call doadump Physical memory: 2026 MB Dumping 219 MB: 204 188 172 156 140 124 108 92 76 60 44 28 12 Dump complete = 0 db:0:doadump> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 1 (kgdb) bt #0 doadump (textdump=-292645332) at pcpu.h:244 #1 0xc0510a19 in db_fncall (dummy1=-1062666086, dummy2=0, dummy3=-1, dummy4=0xee8e96c0 "Ô\226\216î") at /mnt/pho/src/sys/ddb/db_command.c:573 #2 0xc0510e4f in db_command (last_cmdp=0xc10ef67c, cmd_table=0x0, dopager=0) at /mnt/pho/src/sys/ddb/db_command.c:449 #3 0xc0510f04 in db_command_script (command=0xc10f0588 "call doadump") at /mnt/pho/src/sys/ddb/db_command.c:520 #4 0xc05152f0 in db_script_exec (scriptname=0xc10efee0 "doadump", warnifnotfound=Variable "warnifnotfound" is not available. ) at /mnt/pho/src/sys/ddb/db_script.c:302 #5 0xc0515381 in db_run_cmd (addr=-1055940032, have_addr=0, count=-1066928135, modif=0xee8e97f8 "") at /mnt/pho/src/sys/ddb/db_script.c:375 #6 0xc0510e11 in db_command (last_cmdp=0xc10ef67c, cmd_table=0x0, dopager=1) at /mnt/pho/src/sys/ddb/db_command.c:449 #7 0xc0510f6a in db_command_loop () at /mnt/pho/src/sys/ddb/db_command.c:502 #8 0xc0512f5d in db_trap (type=3, code=0) at /mnt/pho/src/sys/ddb/db_main.c:229 #9 0xc0a90556 in kdb_trap (type=3, code=0, tf=0xee8e99ac) at /mnt/pho/src/sys/kern/subr_kdb.c:629 #10 0xc0dbf70b in trap (frame=0xee8e99ac) at /mnt/pho/src/sys/i386/i386/trap.c:722 #11 0xc0da7f9c in calltrap () at /mnt/pho/src/sys/i386/i386/exception.s:168 #12 0xc0a9009a in kdb_enter (why=0xc0f8c870 "panic", msg=0xc0f8c870 "panic") at cpufunc.h:71 #13 0xc0a5af6c in panic (fmt=0xc0fc53af "dqflush: stray dquot %p") at /mnt/pho/src/sys/kern/kern_shutdown.c:633 #14 0xc0cb1500 in quotaoff1 (td=0xca187000, mp=0xc90d35a8, type=0) at /mnt/pho/src/sys/ufs/ufs/ufs_quota.c:1622 #15 0xc0cb1768 in quotaoff_inchange (td=Variable "td" is not available. ) at /mnt/pho/src/sys/ufs/ufs/ufs_quota.c:729 #16 0xc0cb1916 in quotaoff (td=0xca187000, mp=0xc90d35a8, type=0) at /mnt/pho/src/sys/ufs/ufs/ufs_quota.c:768 #17 0xc0c9db5b in ffs_flushfiles (mp=0xc90d35a8, flags=0, td=0xca187000) at /mnt/pho/src/sys/ufs/ffs/ffs_vfsops.c:1349 #18 0xc0c9944e in softdep_flushfiles (oldmnt=0xc90d35a8, flags=0, td=0xca187000) at /mnt/pho/src/sys/ufs/ffs/ffs_softdep.c:1898 #19 0xc0c9e79f in ffs_unmount (mp=0xc90d35a8, mntflags=134217728) at /mnt/pho/src/sys/ufs/ffs/ffs_vfsops.c:1256 #20 0xc0af1109 in dounmount (mp=0xc90d35a8, flags=134217728, td=0xca187000) at /mnt/pho/src/sys/kern/vfs_mount.c:1326 #21 0xc0af17d1 in sys_unmount (td=0xca187000, uap=0xee8e9cec) at /mnt/pho/src/sys/kern/vfs_mount.c:1224 #22 0xc0dbecd7 in syscall (frame=0xee8e9d28) at subr_syscall.c:131 #23 0xc0da8001 in Xint0x80_syscall () at /mnt/pho/src/sys/i386/i386/exception.s:266 #24 0x00000033 in ?? () (kgdb) f 14 #14 0xc0cb1500 in quotaoff1 (td=0xca187000, mp=0xc90d35a8, type=0) at /mnt/pho/src/sys/ufs/ufs/ufs_quota.c:1622 1622 panic("dqflush: stray dquot %p", dq); (kgdb) l 1617 for (dq = LIST_FIRST(dqh); dq; dq = nextdq) { 1618 nextdq = LIST_NEXT(dq, dq_hash); 1619 if (dq->dq_ump->um_quotas[dq->dq_type] != vp) 1620 continue; 1621 if (dq->dq_cnt) 1622 panic("dqflush: stray dquot %p", dq); 1623 LIST_REMOVE(dq, dq_hash); 1624 dq->dq_ump = (struct ufsmount *)0; 1625 } 1626 } (kgdb) p *(struct dquot *)0xc61fc800 $2 = {dq_hash = {le_next = 0x0, le_prev = 0xc880ec68}, dq_freelist = {tqe_next = 0x0, tqe_prev = 0x0}, dq_lock = {lock_object = {lo_name = 0xc0fc5362 "dqlock", lo_flags = 16973824, lo_data = 0, lo_witness = 0xc6158c28}, mtx_lock = 4}, dq_flags = 0, dq_type = 0, dq_cnt = 1208, dq_id = 1001, dq_ump = 0xc88c8900, dq_dqb = {dqb_bhardlimit = 461852, dqb_bsoftlimit = 461752, dqb_curblocks = 1760, dqb_ihardlimit = 35326, dqb_isoftlimit = 35276, dqb_curinodes = 440, dqb_btime = 1329306389, dqb_itime = 1329305285}} (kgdb) f 19 #19 0xc0c9e79f in ffs_unmount (mp=0xc90d35a8, mntflags=134217728) at /mnt/pho/src/sys/ufs/ffs/ffs_vfsops.c:1256 1256 error = softdep_flushfiles(mp, flags, td); (kgdb) p *mp $3 = {mnt_mtx = {lock_object = {lo_name = 0xc0f7e839 "struct mount mtx", lo_flags = 16973824, lo_data = 0, lo_witness = 0xc6151450}, mtx_lock = 4}, mnt_gen = 1, mnt_list = { tqe_next = 0x0, tqe_prev = 0xc891d018}, mnt_op = 0xc10dba80, mnt_vfc = 0xc10db8e0, mnt_vnodecovered = 0xc8ea0ae0, mnt_syncer = 0x0, mnt_ref = 5, mnt_nvnodelist = { tqh_first = 0xc90b315c, tqh_last = 0xc92dd6e0}, mnt_nvnodelistsize = 4, mnt_writeopcount = 1, mnt_kern_flag = 1627390028, mnt_flag = 4294979840, mnt_noasync = 1, mnt_opt = 0xc6176990, mnt_optnew = 0xc6176a00, mnt_maxsymlinklen = 120, mnt_stat = {f_version = 537068824, f_type = 53, f_flags = 4294979840, f_bsize = 2048, f_iosize = 16384, f_blocks = 506483, f_bfree = 437065, f_bavail = 396547, f_files = 141310, f_ffree = 140843, f_syncwrites = 0, f_asyncwrites = 0, f_syncreads = 0, f_asyncreads = 0, f_spare = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, f_namemax = 255, f_owner = 0, f_fsid = {val = {1328700484, 462567061}}, f_charspare = '\0' , f_fstypename = "ufs", '\0' , f_mntfromname = "/dev/md5a", '\0' , f_mntonname = "/mnt", '\0' }, mnt_cred = 0xc6202500, mnt_data = 0xc88c8900, mnt_time = 0, mnt_iosize_max = 131072, mnt_export = 0xc8f78500, mnt_label = 0x0, mnt_hashseed = 3374514232, mnt_lockref = 0, mnt_secondary_writes = 0, mnt_secondary_accwrites = 34750303, mnt_susp_owner = 0x0, mnt_gjprovider = 0x0, mnt_explock = {lock_object = {lo_name = 0xc0f7e84a "explock", lo_flags = 91422720, lo_data = 0, lo_witness = 0xc61571c0}, lk_lock = 1, lk_exslpfail = 0, lk_timo = 0, lk_pri = 96, lk_stack = {depth = 6, pcs = {3231983342, 3232660951, 3232695918, 3232697551, 3235638487, 3235545089, 0 }}}} (kgdb) p flags $4 = 0 (kgdb) -- Test scenario: suj19.sh