KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2012 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #0: Tue Feb 14 09:04:21 CET 2012 test@leopard3.netperf.freebsd.org:/mnt/pho/obj/mnt/pho/src/sys/PHO i386 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: Intel(R) Xeon(R) CPU 3050 @ 2.13GHz (2128.04-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x6f2 Family = 6 Model = f Stepping = 2 Features=0xbfebfbff Features2=0xe3bd AMD Features=0x20100000 AMD Features2=0x1 TSC: P-state invariant, performance statistics real memory = 2147483648 (2048 MB) avail memory = 2080350208 (1983 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 random device not loaded; using insecure entropy ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard ioapic2 irqs 48-71 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 cpu1: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 16 at device 1.0 on pci0 pci1: on pcib1 pcib2: at device 0.0 on pci1 pci2: on pcib2 pcib3: at device 0.2 on pci1 pci3: on pcib3 pcib4: irq 16 at device 3.0 on pci0 pci10: on pcib4 pci10: at device 0.0 (no driver attached) pcib5: irq 17 at device 28.0 on pci0 pcib5: failed to allocate initial I/O port window: 0-0xfff pcib5: failed to allocate initial memory window: 0-0xfffff pcib5: failed to allocate initial prefetch window: 0-0xfffff pci14: on pcib5 pcib6: irq 17 at device 28.4 on pci0 pci18: on pcib6 em0: port 0x5000-0x501f mem 0xe0300000-0xe031ffff irq 16 at device 0.0 on pci18 em0: Using an MSI interrupt em0: Ethernet address: 00:30:48:8e:d0:68 pcib7: irq 16 at device 28.5 on pci0 pci19: on pcib7 em1: port 0x6000-0x601f mem 0xe0400000-0xe041ffff irq 17 at device 0.0 on pci19 em1: Using an MSI interrupt em1: Ethernet address: 00:30:48:8e:d0:69 uhci0: port 0x3000-0x301f irq 23 at device 29.0 on pci0 usbus0: on uhci0 uhci1: port 0x3020-0x303f irq 19 at device 29.1 on pci0 usbus1: on uhci1 uhci2: port 0x3040-0x305f irq 18 at device 29.2 on pci0 usbus2: on uhci2 uhci3: port 0x3060-0x307f irq 16 at device 29.3 on pci0 usbus3: on uhci3 ehci0: mem 0xe0000000-0xe00003ff irq 23 at device 29.7 on pci0 usbus4: EHCI version 1.0 usbus4: on ehci0 pcib8: at device 30.0 on pci0 pci20: on pcib8 vgapci0: port 0x7000-0x70ff mem 0xe8000000-0xefffffff,0xe0500000-0xe050ffff irq 16 at device 4.0 on pci20 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x30a0-0x30af at device 31.1 on pci0 ata0: at channel 0 on atapci0 ahci0: port 0x30e8-0x30ef,0x30dc-0x30df,0x30e0-0x30e7,0x30d8-0x30db,0x30b0-0x30bf mem 0xe0000400-0xe00007ff irq 19 at device 31.2 on pci0 ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 pci0: at device 31.3 (no driver attached) acpi_button0: on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (115200,n,8,1) uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fd0: <1440-KB 3.5" drive> on fdc0 drive 0 pmtimer0 on isa0 orm0: at iomem 0xc0000-0xcafff,0xcb000-0xcbfff pnpid ORM0000 on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x100> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] ppc0: parallel port not found. ctl: CAM Target Layer loaded est0: on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 828082806000828 device_attach: est0 attach returned 6 p4tcc0: on cpu0 est1: on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 828082806000828 device_attach: est1 attach returned 6 p4tcc1: on cpu1 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 12Mbps Full Speed USB v1.0 ugen0.1: at usbus0 uhub0: on usbus0 Expensive timeout(9) function: 0xc0a9c4e0(0xc63a8b80) 0.011057829 s ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 usbus4: 480Mbps High Speed USB v2.0 ugen4.1: at usbus4 uhub4: on usbus4 Expensive timeout(9) function: 0xc0554be0(0xc634c600) 0.020260675 s uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub4: 8 ports with 8 removable, self powered ada0 at ahcich0 bus 0 scbus1 target 0 lun 0 ada0: ATA-7cd0 at ata0 bus 0 scbus0 target 0 lun 0 cd0: Removable CD-ROM SCSI-0 device cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 238475MB (488397168 512 byte sectors: 16H 63S/T 16383C) ada0: Previously was known as ad4 SMP: AP CPU #1 Launched! Timecounter "TSC-low" frequency 16625326 Hz quality 1000 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Trying to mount root from nfs: []... NFS ROOT: 192.168.5.1:/zoo/leopard3 em0: link state changed to UP Interface em0 IP-Address 192.168.5.109 Broadcast 192.168.5.255 Setting hostuuid: 64a93476-27b1-3004-c299-0030488ed164. Setting hostid: 0x7acc54f1. Starting ddb. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/ada0d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0d: clean, 109483507 free (1507 frags, 13685250 blocks, 0.0% fragmentation) mount_nfs: can't update /var/db/mounttab for 192.168.5.1:/zoo/leopard3 Mounting local file systems:. Setting hostname: leopard3.netperf.freebsd.org. Starting Network: lo0 em0 em1. lo0: flags=8049 metric 0 mtu 16384 options=3 inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8 nd6 options=21 em0: flags=8843 metric 0 mtu 1500 options=4219b ether 00:30:48:8e:d0:68 inet 192.168.5.109 netmask 0xffffff00 broadcast 192.168.5.255 inet6 fe80::230:48ff:fe8e:d068%em0 prefixlen 64 tentative scopeid 0x1 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active em1: flags=8c02 metric 0 mtu 1500 options=4219b ether 00:30:48:8e:d0:69 media: Ethernet autoselect status: no carrier Starting devd. Starting Network: em1. em1: flags=8c02 metric 0 mtu 1500 options=4219b ether 00:30:48:8e:d0:69 media: Ethernet autoselect status: no carrier Starting Network: usbus0. Starting Network: usbus1. Starting Network: usbus2. Starting Network: usbus3. Starting Network: usbus4. add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 Creating and/or trimming log files. Starting syslogd. No core dumps found. ELF ldconfig path: /lib /usr/lib /usr/lib/compat a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Setting date via ntp. 14 Feb 08:35:40 ntpdate[1389]: step time server 192.168.5.1 offset -0.355409 sec Starting rpcbind. NFS access cache time=60 Clearing /tmp (X related). Starting mountd. Feb 14 08:35:40 leopard3 mountd[1517]: can't delete exports for /dev: Invalid argument mount option is unknown Starting nfsd. Starting statd. Updating motd:. Configuring syscons: blanktime. Starting sshd. Starting cron. Local package initialization: watchdogd. Starting inetd. Tue Feb 14 08:35:42 UTC 2012 FreeBSD/i386 (leopard3.netperf.freebsd.org) (ttyu0) login: root Feb 14 08:35:44 leopard3 login: ROOT LOGIN (root) ON ttyu0 FreeBSD 10.0-CURRENT (PHO) #0: Tue Feb 14 09:04:21 CET 2012 Welcome to FreeBSD! Before seeking technical support, please use the following resources: o Security advisories and updated errata information for all releases are at http://www.FreeBSD.org/releases/ - always consult the ERRATA section for your release first as it's updated frequently. o The Handbook and FAQ documents are at http://www.FreeBSD.org/ and, along with the mailing lists, can be searched by going to http://www.FreeBSD.org/search/. If the doc distribution has been installed, they're also available formatted in /usr/share/doc. If you still have a question or problem, please take the output of `uname -a', along with any relevant error messages, and email it as a question to the questions@FreeBSD.org mailing list. If you are unfamiliar with FreeBSD's directory layout, please refer to the hier(7) manual page. If you are not familiar with manual pages, type `man man'. You may also use sysinstall(8) to re-enter the installation and configuration utility. Edit /etc/motd to change this login announcement. leopard3# ~test/keepalive.sh Tue Feb 14 09:35:53 CET 2012 9:35AM up 33 secs, 1 user, load averages: 2.12, 0.65, 0.24 Feb 14 09:36:26 leopard3 su: test to root on /dev/pts/0 Feb 14 08:37:39 leopard3 sshd[1799]: error: PAM: authentication error for test from leopard3.netperf.freebsd.org lock order reversal: 1st 0xdfd17340 bufwait (bufwait) @ /mnt/pho/src/sys/kern/vfs_bio.c:2652 2nd 0xc8dc2800 dirhash (dirhash) @ /mnt/pho/src/sys/ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper(c0f927fb,7366752f,7366752f,7366752f,7269645f,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a95cab,c0f96300,c12741b0,11c,ee83a7e8,...) at kdb_backtrace+0x2a _witness_debugger(c0f96300,c8dc2800,c0fc6912,c615ba88,c0fc6593,...) at _witness_debugger+0x25 witness_checkorder(c8dc2800,9,c0fc6593,11c,0,...) at witness_checkorder+0x86f _sx_xlock(c8dc2800,0,c0fc6593,11c,1400,...) at _sx_xlock+0xa0 ufsdirhash_acquire(c8eff15c,ee83a8ac,c0cae50d,c8ee56cc,1400,...) at ufsdirhash_acquire+0x48 ufsdirhash_newblk(c8ee56cc,1400,14,200,c61a8280,...) at ufsdirhash_newblk+0xe ufs_direnter(c90be000,c91436cc,ee83a918,ee83abc0,0,...) at ufs_direnter+0x22d ufs_makeinode(ee83abc0,c10de1c0,ee83ab04,ee83aa60,c0de4bf5,...) at ufs_makeinode+0x65d ufs_create(ee83ab04,c0fe7eab,0,0,ee83ab80,...) at ufs_create+0x30 VOP_CREATE_APV(c10ddc40,ee83ab04,ee83abc0,ee83aa9c,0,...) at VOP_CREATE_APV+0xc5 vn_open_cred(ee83ab80,ee83ac48,1ed,0,c61a8280,...) at vn_open_cred+0x205 vn_open(ee83ab80,ee83ac48,1ed,c8f47428,c8ec1c30,...) at vn_open+0x3b kern_openat(c8ec1b80,ffffff9c,28830140,0,202,...) at kern_openat+0x1ec kern_open(c8ec1b80,28830140,0,201,1ed,...) at kern_open+0x35 sys_open(c8ec1b80,ee83acec,c0fe0b42,c0f97006,286,...) at sys_open+0x30 syscall(ee83ad28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (5, FreeBSD ELF32, sys_open), eip = 0x28344b73, esp = 0xbfbfe34c, ebp = 0xbfbfec58 --- Feb 14 09:38:10 leopard3 su: test to root on /dev/pts/0 Feb 14 08:38:16 leopard3 mountd[1518]: can't delete exports for /dev: Invalid argument mount option is unknown lock order reversal: 1st 0xc9129310 ufs (ufs) @ /mnt/pho/src/sys/kern/vfs_subr.c:2167 2nd 0xdfd1a5a0 bufwait (bufwait) @ /mnt/pho/src/sys/ufs/ffs/ffs_vnops.c:262 3rd 0xc91271b4 ufs (ufs) @ /mnt/pho/src/sys/kern/vfs_subr.c:2167 KDB: stack backtrace: db_trace_self_wrapper(c0f927fb,75735f73,632e7262,3631323a,6f000a37,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a95cab,c0f96319,c1273ee0,877,ee78d3c8,...) at kdb_backtrace+0x2a _witness_debugger(c0f96319,c91271b4,c0f84bd4,c615b8e8,c0f9e3d4,...) at _witness_debugger+0x25 witness_checkorder(c91271b4,9,c0f9e3d4,877,0,...) at witness_checkorder+0x86f __lockmgr_args(c91271b4,80100,c9127220,0,0,...) at __lockmgr_args+0x834 ffs_lock(ee78d4ec,0,c0f95525,80100,c912715c,...) at ffs_lock+0xa1 VOP_LOCK1_APV(c10ddc40,ee78d4ec,c61541e0,c10eec60,c912715c,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c912715c,80100,c0f9e3d4,877,4,...) at _vn_lock+0x78 vget(c912715c,80100,c8da12e0,50,0,...) at vget+0xbb vfs_hash_get(c91775a8,2e000,80000,c8da12e0,ee78d638,...) at vfs_hash_get+0xed ffs_vgetf(c91775a8,2e000,80000,ee78d638,1,...) at ffs_vgetf+0x49 softdep_sync_buf(c91292b8,dfd1a540,1,108,0,...) at softdep_sync_buf+0x4a3 ffs_syncvnode(c91292b8,1,c1273ed0,c8da12e0,c8da12e0,...) at ffs_syncvnode+0x24c ffs_truncate(c91292b8,400,0,880,c916a680,...) at ffs_truncate+0x827 ufs_direnter(c91292b8,c912715c,ee78d9f4,ee78db84,dfd22f60,...) at ufs_direnter+0x9a1 ufs_mkdir(ee78dc14,c0fe7ce5,0,0,ee78dbac,...) at ufs_mkdir+0xa6f VOP_MKDIR_APV(c10ddc40,ee78dc14,ee78db84,ee78dbac,0,...) at VOP_MKDIR_APV+0xc5 kern_mkdirat(c8da12e0,ffffff9c,bfbfeda5,0,1f8,...) at kern_mkdirat+0x291 kern_mkdir(c8da12e0,bfbfeda5,0,1f8,ee78dd1c,...) at kern_mkdir+0x2e sys_mkdir(c8da12e0,ee78dcec,c0fe0b42,c0f97006,257,...) at sys_mkdir+0x29 syscall(ee78dd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (136, FreeBSD ELF32, sys_mkdir), eip = 0x28185b33, esp = 0xbfbfe8cc, ebp = 0xbfbfe968 --- lock order reversal: 1st 0xdfd22fc0 bufwait (bufwait) @ /mnt/pho/src/sys/ufs/ffs/ffs_softdep.c:12329 2nd 0xc8ed8b1c snaplk (snaplk) @ /mnt/pho/src/sys/ufs/ffs/ffs_snapshot.c:2299 KDB: stack backtrace: db_trace_self_wrapper(c0f927fb,662f7366,662f7366,735f7366,7370616e,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a95cab,c0f96300,c1273f00,8fb,ee78d3e4,...) at kdb_backtrace+0x2a _witness_debugger(c0f96300,c8ed8b1c,c0fc249d,c615bc28,c0fc2416,...) at _witness_debugger+0x25 witness_checkorder(c8ed8b1c,9,c0fc2416,8fb,0,...) at witness_checkorder+0x86f __lockmgr_args(c8ed8b1c,80900,c9129634,0,0,...) at __lockmgr_args+0x834 ffs_copyonwrite(c9129570,dfd22f60,0,dfd22f60,c912715c,...) at ffs_copyonwrite+0x2c2 ffs_geom_strategy(c9129668,dfd22f60,ee78d568,f42,ee78d570,...) at ffs_geom_strategy+0xeb ufs_strategy(ee78d5a4,c0fe7ba6,c0f8caeb,dfd22f60,0,...) at ufs_strategy+0xd3 VOP_STRATEGY_APV(c10ddc40,ee78d5a4,c9127254,c10eecc0,c912715c,...) at VOP_STRATEGY_APV+0xe5 bufstrategy(c9127254,dfd22f60,645,dfd22f60,b8800,...) at bufstrategy+0x61 bufwrite(dfd22f60,80900,c9127254,0,0,...) at bufwrite+0x159 softdep_sync_buf(c91292b8,dfd1a540,1,108,0,...) at softdep_sync_buf+0x7cc ffs_syncvnode(c91292b8,1,c1273ed0,c8da12e0,c8da12e0,...) at ffs_syncvnode+0x24c ffs_truncate(c91292b8,400,0,880,c916a680,...) at ffs_truncate+0x827 ufs_direnter(c91292b8,c912715c,ee78d9f4,ee78db84,dfd22f60,...) at ufs_direnter+0x9a1 ufs_mkdir(ee78dc14,c0fe7ce5,0,0,ee78dbac,...) at ufs_mkdir+0xa6f VOP_MKDIR_APV(c10ddc40,ee78dc14,ee78db84,ee78dbac,0,...) at VOP_MKDIR_APV+0xc5 kern_mkdirat(c8da12e0,ffffff9c,bfbfeda5,0,1f8,...) at kern_mkdirat+0x291 kern_mkdir(c8da12e0,bfbfeda5,0,1f8,ee78dd1c,...) at kern_mkdir+0x2e sys_mkdir(c8da12e0,ee78dcec,c0fe0b42,c0f97006,257,...) at sys_mkdir+0x29 syscall(ee78dd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (136, FreeBSD ELF32, sys_mkdir), eip = 0x28185b33, esp = 0xbfbfe8cc, ebp = 0xbfbfe968 --- lock order reversal: 1st 0xc8f15310 newnfs (newnfs) @ /mnt/pho/src/sys/kern/vfs_mount.c:1249 2nd 0xc917bb38 devfs (devfs) @ /mnt/pho/src/sys/ufs/ffs/ffs_vfsops.c:1376 KDB: stack backtrace: db_trace_self_wrapper(c0f927fb,7366662f,7366662f,7366765f,2e73706f,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a95cab,c0f96300,c1273fe8,560,ee7ae9b8,...) at kdb_backtrace+0x2a _witness_debugger(c0f96300,c917bb38,c0f7c90f,c615b2d0,c0fc58d0,...) at _witness_debugger+0x25 witness_checkorder(c917bb38,9,c0fc58d0,560,c917bba4,...) at witness_checkorder+0x86f __lockmgr_args(c917bb38,80400,c917bba4,0,0,...) at __lockmgr_args+0x834 vop_stdlock(ee7aead8,c8d9f950,c0fc6a99,80400,c917bae0,...) at vop_stdlock+0x65 VOP_LOCK1_APV(c10b38c0,ee7aead8,c91272b8,c10eec60,c917bae0,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c917bae0,80400,c0fc58d0,560,c8ed8a00,...) at _vn_lock+0x78 ffs_flushfiles(c91772d4,0,c8d9f8a0,4,c0f8caeb,...) at ffs_flushfiles+0x11c ffs_unmount(c91772d4,8000000,c0f9da27,51a,c6030000,...) at ffs_unmount+0x1d6 dounmount(c91772d4,8000000,c8d9f8a0,49c,0,...) at dounmount+0x419 sys_unmount(c8d9f8a0,ee7aecec,ee7aecb4,c0a84688,206,...) at sys_unmount+0x421 syscall(ee7aed28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280dcb3f, esp = 0xbfbfe21c, ebp = 0xbfbfe2e8 --- lock order reversal: 1st 0xc8ed8b1c snaplk (snaplk) @ /mnt/pho/src/sys/ufs/ufs/ufs_vnops.c:965 2nd 0xc9127df0 ufs (ufs) @ /mnt/pho/src/sys/ufs/ffs/ffs_snapshot.c:1629 KDB: stack backtrace: db_trace_self_wrapper(c0f927fb,7366662f,616e735f,6f687370,3a632e74,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a95cab,c0f96300,c1273fd8,65d,ee7b4704,...) at kdb_backtrace+0x2a _witness_debugger(c0f96300,c9127df0,c0f84bd4,c615b8e8,c0fc2416,...) at _witness_debugger+0x25 witness_checkorder(c9127df0,9,c0fc2416,65d,0,...) at witness_checkorder+0x86f __lockmgr_args(c9127df0,80000,0,0,0,...) at __lockmgr_args+0x834 ffs_snapremove(c9127d98,c0fc2354,c10c0b00,c0f9b8da,dfd1a400,...) at ffs_snapremove+0x11f ffs_truncate(c9127d98,0,0,c00,0,...) at ffs_truncate+0x607 ufs_inactive(ee7b4adc,c0fe7478,c9127e5c,c9127e5c,c9127d98,...) at ufs_inactive+0x1fb VOP_INACTIVE_APV(c10ddc40,ee7b4adc,c0f9e3d4,96f,c10eec20,...) at VOP_INACTIVE_APV+0xc5 vinactive(c10ddc40,ee7b4b10,c0f9e3d4,8c6,0,...) at vinactive+0xb1 vputx(ee7b4c3c,c0b003e3,c9127d98,ffffffdf,c9127d98,...) at vputx+0x36c vput(c9127d98,ffffffdf,c9127d98,ee7b4b94,2c00,...) at vput+0x10 kern_unlinkat(c8d9f2e0,ffffff9c,bfbfebba,0,0,...) at kern_unlinkat+0x2a3 kern_unlink(c8d9f2e0,bfbfebba,0,ee7b4d1c,c0dc08c7,...) at kern_unlink+0x2f sys_unlink(c8d9f2e0,ee7b4cec,c0fe0b42,c0f9701a,246,...) at sys_unlink+0x22 syscall(ee7b4d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (10, FreeBSD ELF32, sys_unlink), eip = 0x281720af, esp = 0xbfbfe91c, ebp = 0xbfbfe998 --- fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 18 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 58 lock type devfs: EXCL by thread 0xc9209000 (pid 2091) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a Feb 14 08:38:18 leopard3 last message repeated 12 times fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 186 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 730 lock type devfs: EXCL by thread 0xc91378a0 (pid 2141) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc8cbfb80 (pid 2155) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc91e98a0 (pid 2359) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc92095c0 (pid 2383) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc91392e0 (pid 2397) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc92088a0 (pid 2421) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc91375c0 (pid 2435) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc90b68a0 (pid 2479) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc9207000 (pid 3023) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc91ad2e0 (pid 3107) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc91acb80 (pid 3161) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc8e318a0 (pid 3275) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc8da08a0 (pid 3329) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc9137000 (pid 3643) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc88f82e0 (pid 3677) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc91ad8a0 (pid 3691) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc8ebd2e0 (pid 4015) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc8cc4000 (pid 4109) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc8da08a0 (pid 4253) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc92098a0 (pid 4277) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc92098a0 (pid 4361) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc91ab2e0 (pid 4395) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc8d9fb80 (pid 4459) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc91e9000 (pid 4595) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc91e9000 (pid 4859) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc920a000 (pid 5123) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9129570: tag devfs, type VCHR usecount 1, writecount 0, refcount 184 mountedhere 0xc916d600 flags () v_object 0xc8ffc870 ref 0 pages 722 lock type devfs: EXCL by thread 0xc91e9b80 (pid 5177) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a : : fsync: giving up on dirty 0xcb487570: tag devfs, type VCHR usecount 1, writecount 0, refcount 136 mountedhere 0xc9979100 flags () v_object 0xcb52a900 ref 0 pages 530 lock type devfs: EXCL by thread 0xc88f85c0 (pid 22458) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xcb487570: tag devfs, type VCHR usecount 1, writecount 0, refcount 136 mountedhere 0xc9979100 flags () v_object 0xcb52a900 ref 0 pages 530 lock type devfs: EXCL by thread 0xc8b378a0 (pid 22532) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xcb487570: tag devfs, type VCHR usecount 1, writecount 0, refcount 136 mountedhere 0xc9979100 flags () v_object 0xcb52a900 ref 0 pages 530 lock type devfs: EXCL by thread 0xc98e0000 (pid 22566) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xcb487570: tag devfs, type VCHR usecount 1, writecount 0, refcount 136 mountedhere 0xc9979100 flags () v_object 0xcb52a900 ref 0 pages 530 lock type devfs: EXCL by thread 0xc8bd55c0 (pid 23050) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xcb487570: tag devfs, type VCHR usecount 1, writecount 0, refcount 136 mountedhere 0xc9979100 flags () v_object 0xcb52a900 ref 0 pages 530 lock type devfs: EXCL by thread 0xc9317000 (pid 23154) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xcb487570: tag devfs, type VCHR usecount 1, writecount 0, refcount 136 mountedhere 0xc9979100 flags () v_object 0xcb52a900 ref 0 pages 530 lock type devfs: EXCL by thread 0xc96e05c0 (pid 23248) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xcb487570: tag devfs, type VCHR usecount 1, writecount 0, refcount 136 mountedhere 0xc9979100 flags () v_object 0xcb52a900 ref 0 pages 530 lock type devfs: EXCL by thread 0xc8d308a0 (pid 23392) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xcb487570: tag devfs, type VCHR usecount 1, writecount 0, refcount 136 mountedhere 0xc9979100 flags () v_object 0xcb52a900 ref 0 pages 530 lock type devfs: EXCL by thread 0xc96fe5c0 (pid 23406) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xcb487570: tag devfs, type VCHR usecount 1, writecount 0, refcount 136 mountedhere 0xc9979100 flags (VV_COPYONWRITE) v_object 0xcb52a900 ref 0 pages 530 lock type devfs: EXCL by thread 0xc8d2e000 (pid 23762) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xcb487570: tag devfs, type VCHR usecount 1, writecount 0, refcount 136 mountedhere 0xc9979100 flags (VV_COPYONWRITE) v_object 0xcb52a900 ref 0 pages 530 lock type devfs: EXCL by thread 0xc8d2e000 (pid 23762) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a Expensive timeout(9) function: 0xc08e4e70(0xc63ebc78) 0.567285116 s Expensive timeout(9) function: 0xc08e4e70(0xc63ebc78) 0.599727066 s Feb 14 23:18:00 leopard3 last message repeated 1314 times Expensive timeout(9) function: 0xc08e4e70(0xc6395cf0) 0.599993347 s Expensive timeout(9) function: 0xc08e4e70(0xc6395cf0) 0.599999602 s Wed Feb 15 00:25:59 CET 2012 12:25AM up 14:51, 2 users, load averages: 2.32, 4.94, 9.39 Wed Feb 15 00:36:00 CET 2012 12:36AM up 15:01, 2 users, load averages: 2.17, 2.63, 5.96 Expensive timeout(9) function: 0xc08e4e70(0xc63aecf0) 0.733330642 s Expensive timeout(9) function: 0xc08e4e70(0xc6395cf0) 0.799985275 s Wed Feb 15 00:46:01 CET 2012 12:46AM up 15:11, 2 users, load averages: 1.95, 2.12, 4.03 Expensive timeout(9) function: 0xc08e4e70(0xc63d7cf0) 0.999987609 s Wed Feb 15 00:56:02 CET 2012 12:56AM up 15:21, 2 users, load averages: 1.71, 1.83, 2.98 Wed Feb 15 01:06:03 CET 2012 1:06AM up 15:31, 2 users, load averages: 2.39, 2.15, 2.57 Wed Feb 15 01:16:04 CET 2012 1:16AM up 15:41, 2 users, load averages: 1.94, 2.00, 2.28 Stopping inetd. Waiting for PIDS: 1725. Shutting down local packages:. Stopping cron. Stopping sshd. Waiting for PIDS: 1676, 1676. Stopping watchdogd. Waiting for PIDS: 169730 second watchdog timeout expired. Shutdown terminated. Wed Feb 15 00:24:34 UTC 2012 Feb 15 00:24:35 Feb 15 00:24:35 leopard3 syslogd: exiting on signal 15 Feb 15 00:24:56 init: some processes would not die; ps axl advised Enter full pathname of shell or RETURN for /bin/sh: # mount 192.168.5.1:/zoo/leopard3 on / (nfs) devfs on /dev (devfs, local, multilabel) /dev/ada0d on /work (ufs, local) procfs on /proc (procfs, local) /dev/md0 on /var (ufs, local) /dev/md1 on /tmp (ufs, local) /dev/md5a on /mnt (ufs, NFS exported, local) /dev/md6 on /mnt2 (ufs, local, read-only) # umount /mnt2 # mount 192.168.5.1:/zoo/leopard3 on / (nfs) devfs on /dev (devfs, local, multilabel) /dev/ada0d on /work (ufs, local) procfs on /proc (procfs, local) /dev/md0 on /var (ufs, local) /dev/md1 on /tmp (ufs, local) /dev/md5a on /mnt (ufs, NFS exported, local) # KDB: enter: Break to debugger [ thread pid 23766 tid 100275 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> run pho db:0:pho> bt Tracing pid 23766 tid 100275 td 0xc9317000 kdb_enter(c0e02aae,c0f927c6,c642d000,eea793b0,c0a916e3,...) at kdb_enter+0x3a kdb_alt_break_internal(eea793e0,c08c927e,2,c642d058,c0aa6177,...) at kdb_alt_break_internal+0x9a kdb_alt_break(2,c642d058,c0aa6177,c642d080,c642d058,...) at kdb_alt_break+0x13 uart_intr(c642d000,3560e94e,1,c6156190,0,...) at uart_intr+0x1ae intr_event_handle(c61f2d80,eea7944c,eea79470,c1272e50,c93170b0,...) at intr_event_handle+0x76 intr_execute_handlers(c61f10d0,eea7944c,0,eea794a4,c0da9f55,...) at intr_execute_handlers+0x49 lapic_handle_intr(3b,eea7944c) at lapic_handle_intr+0x36 Xapic_isr1() at Xapic_isr1+0x35 --- interrupt, eip = 0xc0aa5e95, esp = 0xeea7948c, ebp = 0xeea794a4 --- witness_lock(c9177000,8,c0fc29eb,33d1,0,...) at witness_lock+0xa5 _mtx_lock_flags(c9177000,0,c0fc29eb,33d1,5fd,...) at _mtx_lock_flags+0x11c softdep_check_suspend(c9177000,cb487570,1,5ac8d7,0,...) at softdep_check_suspend+0x120 ffs_sync(c9177000,1,c0f9fbf8,4b2,c0f9fbf8,...) at ffs_sync+0x4bc vfs_write_suspend(c9177000,eea79874,0,4000,c6151e00,...) at vfs_write_suspend+0x122 ffs_snapshot(c9177000,c8fd8560,c0fc58d0,1a7,0,...) at ffs_snapshot+0xcef ffs_mount(c9177000,c8f28600,ff,3b4,c0ae77da,...) at ffs_mount+0x1d17 vfs_donmount(c9317000,11100,0,c8ac7480,c8ac7480,...) at vfs_donmount+0x12eb sys_nmount(c9317000,eea79cec,c0fe0b42,c0f97103,213,...) at sys_nmount+0x7f syscall(eea79d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (378, FreeBSD ELF32, sys_nmount), eip = 0x280da65b, esp = 0xbfbfe23c, ebp = 0xbfbfe9b8 --- db:0:bt> show allpcpu Current CPU: 0 cpuid = 0 dynamic pcpu = 0x735b80 curthread = 0xc9317000: pid 23766 "mksnap_ffs" curpcb = 0xeea79d80 fpcurthread = none idlethread = 0xc61ad8a0: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 dynamic pcpu = 0x4db8b80 curthread = 0xc61ad5c0: pid 11 "idle: cpu1" curpcb = 0xc5eb5d80 fpcurthread = none idlethread = 0xc61ad5c0: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: db:0:allpcpu> show alllocks Process 23766 (mksnap_ffs) thread 0xc9317000 (100275) exclusive sleep mutex Softdep Lock (Softdep Lock) r = 0 (0xc12a7e60) locked @ /mnt/pho/src/sys/ufs/ffs/ffs_softdep.c:13258 exclusive sleep mutex bufobj interlock (bufobj interlock) r = 0 (0xcb487668) locked @ /mnt/pho/src/sys/ufs/ffs/ffs_vfsops.c:1533 exclusive sleep mutex Giant (Giant) r = 0 (0xc112a090) locked @ /mnt/pho/src/sys/kern/vfs_mount.c:1102 db:0:alllocks> show lockedvnods Locked vnodes db:0:lockedvnods> show mount 0xc8922b50 192.168.5.1:/zoo/leopard3 on / (nfs) 0xc8923000 devfs on /dev (devfs) 0xc892287c /dev/ada0d on /work (ufs) 0xc89225a8 procfs on /proc (procfs) 0xc89222d4 /dev/md0 on /var (ufs) 0xc8922000 /dev/md1 on /tmp (ufs) 0xc9177000 /dev/md5a on /mnt (ufs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 24079 1 24079 0 Ss+ ttyin 0xc6178870 sh 23770 1 1697 0 D suspfs 0xc9177048 ls 23767 1 20317 1001 DE+ suspfs 0xc9177048 find 23766 1 1824 0 R+ CPU 0 mksnap_ffs 23754 0 0 0 DL mdwait 0xc8d15000 [md6] 20302 0 0 0 DL mdwait 0xc95ee000 [md5] 1452 0 0 0 DL mdwait 0xc8ef1800 [md1] 101 0 0 0 DL mdwait 0xc8ef3000 [md0] 18 0 0 0 DL sdflush 0xc12a7ea0 [softdepflush] 17 0 0 0 DL vlruwt 0xc88ec588 [vnlru] 16 0 0 0 DL syncer 0xc12a1bb4 [syncer] 9 0 0 0 DL psleep 0xc12a1848 [bufdaemon] 8 0 0 0 DL pgzero 0xc12a9d5c [pagezero] 7 0 0 0 DL psleep 0xc12a9948 [vmdaemon] 6 0 0 0 DL psleep 0xc12a9910 [pagedaemon] 5 0 0 0 DL ccb_scan 0xc10f01d4 [xpt_thrd] 4 0 0 0 DL waiting_ 0xc12a36d8 [sctp_iterator] 3 0 0 0 DL ctl_work 0xc645a000 [ctl_thrd] 2 0 0 0 DL - 0xc634ee3c [fdc0] 15 0 0 0 DL (threaded) [usb] 100051 D - 0xc63ebd34 [usbus4] 100050 D - 0xc63ebd04 [usbus4] 100049 D - 0xc63ebcd4 [usbus4] 100048 D - 0xc63ebca4 [usbus4] 100047 D - 0xc63d7dac [usbus3] 100046 D - 0xc63d7d7c [usbus3] 100045 D - 0xc63d7d4c [usbus3] 100044 D - 0xc63d7d1c [usbus3] 100042 D - 0xc63c0dac [usbus2] 100041 D - 0xc63c0d7c [usbus2] 100040 D - 0xc63c0d4c [usbus2] 100039 D - 0xc63c0d1c [usbus2] 100037 D - 0xc63aedac [usbus1] 100036 D - 0xc63aed7c [usbus1] 100035 D - 0xc63aed4c [usbus1] 100034 D - 0xc63aed1c [usbus1] 100032 D - 0xc6395dac [usbus0] 100031 D - 0xc6395d7c [usbus0] 100030 D - 0xc6395d4c [usbus0] 100029 D - 0xc6395d1c [usbus0] 14 0 0 0 DL - 0xc112a7e4 [yarrow] 13 0 0 0 DL (threaded) [geom] 100011 D - 0xc11283e8 [g_down] 100010 D - 0xc11283e4 [g_up] 100009 D - 0xc11283dc [g_event] 12 0 0 0 LL (threaded) [intr] 100056 I [irq1: atkbd0] 100054 I [swi0: uart uart] 100053 I [irq258: ahci0] 100052 I [irq14: ata0] 100043 I [irq16: uhci3] 100038 I [irq18: uhci2] 100033 I [irq19: uhci1] 100028 I [irq23: uhci0 ehci0] 100023 I [swi6: task queue] 100022 I [swi6: Giant taskq] 100020 I [swi5: +] 100014 I [swi2: cambio] 100008 I [swi3: vm] 100007 I [swi4: clock] 100006 L *Giant 0xc6172100 [swi4: clock] 100005 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100004 Run CPU 1 [idle: cpu1] 100003 CanRun [idle: cpu0] 1 0 1 0 SLs wait 0xc61abb10 [init] 10 0 0 0 DL audit_wo 0xc12a75e0 [audit] 0 0 0 0 DLs (threaded) [kernel] 100068 D - 0xc64258c0 [mca taskq] 100058 D - 0xc112a7e4 [deadlkres] 100027 D - 0xc6301840 [em1 txq] 100026 D - 0xc6301880 [em1 que] 100025 D - 0xc6301940 [em0 txq] 100024 D - 0xc6301980 [em0 que] 100021 D - 0xc6301e80 [thread taskq] 100019 D - 0xc6302040 [acpi_task_2] 100018 D - 0xc6302040 [acpi_task_1] 100017 D - 0xc6302040 [acpi_task_0] 100016 D - 0xc6302080 [kqueue taskq] 100015 D - 0xc63020c0 [ffs_trim taskq] 100012 D - 0xc6191e80 [firmware taskq] 100000 D sched 0xc11284c0 [swapper] 23768 23767 20317 1001 Z+ chmod db:0:ps> allt Tracing command sh pid 24079 tid 100453 td 0xc966b2e0 sched_switch(c966b2e0,0,104,1a3,2e52dab0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c966b2e0,0,c0f941e0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a4b8fa,c6178804,0,c0f8caeb,c966b2e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c6178870,0,eed2aaf0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c6178870,c6178804,c0f98a70,527,0,...) at _cv_wait_sig+0x260 tty_wait(c6178800,c6178870,eed2ac44,1,0,...) at tty_wait+0x71 ttydisc_read(c6178800,eed2ac44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c623ae00,eed2ac44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c8ffd268,eed2ac44,c61a8e00,0,c966b2e0,...) at devfs_read_f+0x7e dofileread(eed2ac44,ffffffff,ffffffff,0,eed2ac20,...) at dofileread+0x9e kern_readv(c966b2e0,0,eed2ac44,eed2ac64,1,...) at kern_readv+0x68 sys_read(c966b2e0,eed2acec,c0fe0b42,c0df31ae,202,...) at sys_read+0x4f syscall(eed2ad28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281f2dc3, esp = 0xbfbfecac, ebp = 0xbfbfecd8 --- Tracing command ls pid 23770 tid 100361 td 0xc96e12e0 sched_switch(c96e12e0,0,104,1a3,8f6d1458,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,77,...) at mi_switch+0x1ff sleepq_switch(c96e12e0,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c9177048,77,c0f9fc83,0,0,...) at sleepq_wait+0x63 _sleep(c9177048,c9177000,77,c0f9fc83,0,...) at _sleep+0x39c vn_start_write(cb577828,eeb8cb44,1,c0de01b5,0,...) at vn_start_write+0xdd vn_close(cb577828,5,c61a8e00,c96e12e0,c8cd0540,...) at vn_close+0x87 vn_closefile(c8cd0540,c96e12e0,1,0,c8cd0540,...) at vn_closefile+0xe4 _fdrop(c8cd0540,c96e12e0,0,eeb8cc18,0,c0aa5e6a,c1273218,c96e1390,c1273218,c10c43a0,c96e12e0,c9d55e2c,4ce,eeb8cc28,c0a648b5,c9d55e2c,8,c0f87dfc,c8cd0540) at _fdrop+0x43 closef(c8cd0540,c96e12e0,4ce,eeb8cc4c,c9d55e2c,...) at closef+0x2b0 kern_close(c96e12e0,7,eeb8cd1c,c0dc08c7,c96e12e0,...) at kern_close+0x139 sys_close(c96e12e0,eeb8ccec,c0fe0b42,c0f96fd3,246,...) at sys_close+0x1a syscall(eeb8cd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (6, FreeBSD ELF32, sys_close), eip = 0x281d7d83, esp = 0xbfbfe5dc, ebp = 0xbfbfe5f8 --- Tracing command find pid 23767 tid 100448 td 0xc98e4b80 sched_switch(c98e4b80,0,104,1a3,62f89548,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,77,...) at mi_switch+0x1ff sleepq_switch(c98e4b80,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c9177048,77,c0f9fc83,0,0,...) at sleepq_wait+0x63 _sleep(c9177048,c9177000,77,c0f9fc83,0,...) at _sleep+0x39c vn_start_write(cb284570,eed1b89c,1,c61541e0,0,...) at vn_start_write+0xdd vn_close(cb284570,1,c916a200,c98e4b80,c6154248,...) at vn_close+0x87 vn_closefile(c9c85150,c98e4b80,c1272b48,0,c9c85150,...) at vn_closefile+0xe4 _fdrop(c9c85150,c98e4b80,0,eed1b970,0,c0aa5e6a,c1272b38,c98e4c30,c1272b38,c10c43a0,c98e4b80,c939772c,79d,eed1b980,c0a648b5,c939772c,8,c0f87dfc,c9c85150) at _fdrop+0x43 closef(c9c85150,c98e4b80,79d,79a,c6153340,...) at closef+0x2b0 fdfree(c98e4b80,0,c0f88ad3,107,c1272b38,...) at fdfree+0x3ea exit1(c98e4b80,f,c0f8e7bb,b40,c0f90852,...) at exit1+0x573 sigexit(c98e4b80,f,c0f8e7bb,acf,c1272b38,...) at sigexit+0xe33 postsig(f,64,c0f94b96,e8,c1308dc0,...) at postsig+0x20b ast(eed1bd28) at ast+0x398 doreti_ast() at doreti_ast+0x17 Tracing command mksnap_ffs pid 23766 tid 100275 td 0xc9317000 kdb_enter(c0e02aae,c0f927c6,c642d000,eea793b0,c0a916e3,...) at kdb_enter+0x3a kdb_alt_break_internal(eea793e0,c08c927e,2,c642d058,c0aa6177,...) at kdb_alt_break_internal+0x9a kdb_alt_break(2,c642d058,c0aa6177,c642d080,c642d058,...) at kdb_alt_break+0x13 uart_intr(c642d000,3560e94e,1,c6156190,0,...) at uart_intr+0x1ae intr_event_handle(c61f2d80,eea7944c,eea79470,c1272e50,c93170b0,...) at intr_event_handle+0x76 intr_execute_handlers(c61f10d0,eea7944c,0,eea794a4,c0da9f55,...) at intr_execute_handlers+0x49 lapic_handle_intr(3b,eea7944c) at lapic_handle_intr+0x36 Xapic_isr1() at Xapic_isr1+0x35 --- interrupt, eip = 0xc0aa5e95, esp = 0xeea7948c, ebp = 0xeea794a4 --- witness_lock(c9177000,8,c0fc29eb,33d1,0,...) at witness_lock+0xa5 _mtx_lock_flags(c9177000,0,c0fc29eb,33d1,5fd,...) at _mtx_lock_flags+0x11c softdep_check_suspend(c9177000,cb487570,1,5ac8d7,0,...) at softdep_check_suspend+0x120 ffs_sync(c9177000,1,c0f9fbf8,4b2,c0f9fbf8,...) at ffs_sync+0x4bc vfs_write_suspend(c9177000,eea79874,0,4000,c6151e00,...) at vfs_write_suspend+0x122 ffs_snapshot(c9177000,c8fd8560,c0fc58d0,1a7,0,...) at ffs_snapshot+0xcef ffs_mount(c9177000,c8f28600,ff,3b4,c0ae77da,...) at ffs_mount+0x1d17 vfs_donmount(c9317000,11100,0,c8ac7480,c8ac7480,...) at vfs_donmount+0x12eb sys_nmount(c9317000,eea79cec,c0fe0b42,c0f97103,213,...) at sys_nmount+0x7f syscall(eea79d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (378, FreeBSD ELF32, sys_nmount), eip = 0x280da65b, esp = 0xbfbfe23c, ebp = 0xbfbfe9b8 --- Tracing command md6 pid 23754 tid 100298 td 0xc8bd52e0 sched_switch(c8bd52e0,0,104,1a3,ce2152e8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c8bd52e0,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c8d15000,5c,c0f2a96a,0,0,...) at sleepq_wait+0x63 _sleep(c8d15000,c8d15020,25c,c0f2a96a,0,...) at _sleep+0x39c md_kthread(c8d15000,eeabed28,c0f88d50,3db,c8b962c4,...) at md_kthread+0x11a fork_exit(c0769790,c8d15000,eeabed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xeeabed60, ebp = 0 --- Tracing command md5 pid 20302 tid 100168 td 0xc91e92e0 sched_switch(c91e92e0,0,104,1a3,8f708780,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c91e92e0,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c95ee000,5c,c0f2a96a,0,0,...) at sleepq_wait+0x63 _sleep(c95ee000,c95ee020,25c,c0f2a96a,0,...) at _sleep+0x39c md_kthread(c95ee000,ee8c4d28,c0f88d50,3db,c91c72c4,...) at md_kthread+0x11a fork_exit(c0769790,c95ee000,ee8c4d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee8c4d60, ebp = 0 --- Tracing command md1 pid 1452 tid 100091 td 0xc8da0b80 sched_switch(c8da0b80,0,104,1a3,ce80fb38,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c8da0b80,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c8ef1800,5c,c0f2a96a,0,0,...) at sleepq_wait+0x63 _sleep(c8ef1800,c8ef1820,25c,c0f2a96a,0,...) at _sleep+0x39c md_kthread(c8ef1800,ee79bd28,c0f88d50,3db,c8e28000,...) at md_kthread+0x11a fork_exit(c0769790,c8ef1800,ee79bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee79bd60, ebp = 0 --- Tracing command md0 pid 101 tid 100104 td 0xc8ebf5c0 sched_switch(c8ebf5c0,0,104,1a3,2e529978,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c8ebf5c0,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c8ef3000,5c,c0f2a96a,0,0,...) at sleepq_wait+0x63 _sleep(c8ef3000,c8ef3020,25c,c0f2a96a,0,...) at _sleep+0x39c md_kthread(c8ef3000,ee7c5d28,c0f88d50,3db,c8eec000,...) at md_kthread+0x11a fork_exit(c0769790,c8ef3000,ee7c5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee7c5d60, ebp = 0 --- Tracing command softdepflush pid 18 tid 100067 td 0xc88f65c0 sched_switch(c88f65c0,0,104,1a3,2e4d0f88,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,54,...) at mi_switch+0x1ff sleepq_switch(c88f65c0,0,c0f941e0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c12a7ea0,54,c0fc4f33,0,0,...) at sleepq_timedwait+0x6b _sleep(c12a7ea0,c12a7e60,54,c0fc4f33,3e8,...) at _sleep+0x36a softdep_flush(0,ec546d28,c0f88d50,3db,c88ec2c4,...) at softdep_flush+0x266 fork_exit(c0c9af90,0,ec546d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec546d60, ebp = 0 --- Tracing command vnlru pid 17 tid 100066 td 0xc88f68a0 sched_switch(c88f68a0,0,104,1a3,2e4eeab0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,60,...) at mi_switch+0x1ff sleepq_switch(c88f68a0,0,c0f941e0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c88ec588,60,c0f9f34d,0,0,...) at sleepq_timedwait+0x6b _sleep(c88ec588,c12a1b74,260,c0f9f34d,3e8,...) at _sleep+0x36a vnlru_proc(0,ec543d28,c0f88d50,3db,c88ec588,...) at vnlru_proc+0xe7 fork_exit(c0afd240,0,ec543d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec543d60, ebp = 0 --- Tracing command syncer pid 16 tid 100065 td 0xc88f6b80 sched_switch(c88f6b80,0,104,1a3,2e51c6f0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c88f6b80,0,c0f941e0,28b,c88f6b80,...) at sleepq_switch+0x162 sleepq_timedwait(c12a1bb4,0,ec540c78,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c12a1bb4,c12a1ba0,3e8,78b,4e20,...) at _cv_timedwait+0x270 sched_sync(0,ec540d28,c0f88d50,3db,c88ec84c,...) at sched_sync+0x595 fork_exit(c0afc5b0,0,ec540d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec540d60, ebp = 0 --- Tracing command bufdaemon pid 9 tid 100064 td 0xc88f8000 sched_switch(c88f8000,0,104,1a3,2e4efc28,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,54,...) at mi_switch+0x1ff sleepq_switch(c88f8000,0,c0f941e0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c12a1848,54,c0f9c229,0,0,...) at sleepq_timedwait+0x6b _sleep(c12a1848,c12a1850,54,c0f9c229,3e8,...) at _sleep+0x36a buf_daemon(0,ec53dd28,c0f88d50,3db,c88ecb10,...) at buf_daemon+0x175 fork_exit(c0ae0e10,0,ec53dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec53dd60, ebp = 0 --- Tracing command pagezero pid 8 tid 100063 td 0xc63d32e0 sched_switch(c63d32e0,0,104,1a3,95d50ed0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d32e0,0,c0f941e0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c12a9d5c,0,c0fcc066,0,0,...) at sleepq_timedwait+0x6b _sleep(c12a9d5c,c12a8800,0,c0fcc066,493e0,...) at _sleep+0x36a vm_pagezero(0,ec53ad28,c0f88d50,3db,c88ed000,...) at vm_pagezero+0xdc fork_exit(c0ce2320,0,ec53ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec53ad60, ebp = 0 --- Tracing command vmdaemon pid 7 tid 100062 td 0xc63d35c0 sched_switch(c63d35c0,0,104,1a3,b0a06858,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,74,...) at mi_switch+0x1ff sleepq_switch(c63d35c0,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c12a9948,74,c0f9c229,0,0,...) at sleepq_wait+0x63 _sleep(c12a9948,c12a994c,74,c0f9c229,0,...) at _sleep+0x39c vm_daemon(0,ec537d28,c0f88d50,3db,c61ac2c4,...) at vm_daemon+0x59 fork_exit(c0cdbc30,0,ec537d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec537d60, ebp = 0 --- Tracing command pagedaemon pid 6 tid 100061 td 0xc63d38a0 sched_switch(c63d38a0,0,104,1a3,2e4f48d8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,54,...) at mi_switch+0x1ff sleepq_switch(c63d38a0,0,c0f941e0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c12a9910,54,c0f9c229,0,0,...) at sleepq_timedwait+0x6b _sleep(c12a9910,c12a8800,54,c0f9c229,1388,...) at _sleep+0x36a vm_pageout(0,ec534d28,c0f88d50,3db,c61ac588,...) at vm_pageout+0x2c7 fork_exit(c0cdccf0,0,ec534d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec534d60, ebp = 0 --- Tracing command xpt_thrd pid 5 tid 100060 td 0xc63d3b80 sched_switch(c63d3b80,0,104,1a3,f8250e00,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c63d3b80,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c10f01d4,5c,c0de8c57,0,0,...) at sleepq_wait+0x63 _sleep(c10f01d4,c10f0218,5c,c0de8c57,0,...) at _sleep+0x39c xpt_scanner_thread(0,ec531d28,c0f88d50,3db,c61ac84c,...) at xpt_scanner_thread+0x47 fork_exit(c049d320,0,ec531d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec531d60, ebp = 0 --- Tracing command sctp_iterator pid 4 tid 100059 td 0xc640e000 sched_switch(c640e000,0,104,1a3,27c95da8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c640e000,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c12a36d8,0,c0fb2bbf,0,0,...) at sleepq_wait+0x63 _sleep(c12a36d8,c12a36a0,0,c0fb2bbf,0,...) at _sleep+0x39c sctp_iterator_thread(0,ec52ed28,c0f88d50,3db,c61acb10,...) at sctp_iterator_thread+0x5c fork_exit(c0b8ea60,0,ec52ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec52ed60, ebp = 0 --- Tracing command ctl_thrd pid 3 tid 100057 td 0xc640e5c0 sched_switch(c640e5c0,0,104,1a3,d76b9bc8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c640e5c0,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c645a000,5c,c0df368a,0,0,...) at sleepq_wait+0x63 _sleep(c645a000,c645a000,5c,c0df368a,0,...) at _sleep+0x39c ctl_work_thread(c645a000,ec528d28,c0f88d50,3db,c63a4000,...) at ctl_work_thread+0x22f6 fork_exit(c04c6720,c645a000,ec528d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec528d60, ebp = 0 --- Tracing command fdc0 pid 2 tid 100055 td 0xc640eb80 sched_switch(c640eb80,0,104,1a3,2e4d7d80,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c640eb80,0,c0f941e0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c634ee3c,5c,c0f80f1a,0,0,...) at sleepq_timedwait+0x6b _sleep(c634ee3c,c634eef0,5c,c0f80f1a,3e8,...) at _sleep+0x36a fdc_thread(c634ee00,ec522d28,c0f88d50,3db,c63a42c4,...) at fdc_thread+0x27d fork_exit(c0d61570,c634ee00,ec522d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec522d60, ebp = 0 --- Tracing command usb pid 15 tid 100051 td 0xc63d0000 sched_switch(c63d0000,0,104,1a3,d7660d20,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d0000,0,c0f941e0,268,c63d0000,...) at sleepq_switch+0x162 sleepq_wait(c63ebd34,0,c5fbbcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63ebd34,c63ebe4c,c0f672f1,74,c63ebd3c,...) at _cv_wait+0x260 usb_process(c63ebd2c,c5fbbd28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63ebd2c,c5fbbd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fbbd60, ebp = 0 --- Tracing command usb pid 15 tid 100050 td 0xc63d02e0 sched_switch(c63d02e0,0,104,1a3,2e4ea198,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d02e0,0,c0f941e0,268,c63d02e0,...) at sleepq_switch+0x162 sleepq_wait(c63ebd04,0,c5fb8cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63ebd04,c63ebe4c,c0f672f1,74,c63ebd0c,...) at _cv_wait+0x260 usb_process(c63ebcfc,c5fb8d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63ebcfc,c5fb8d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fb8d60, ebp = 0 --- Tracing command usb pid 15 tid 100049 td 0xc63d05c0 sched_switch(c63d05c0,0,104,1a3,d6e8b4e8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d05c0,0,c0f941e0,268,c63d05c0,...) at sleepq_switch+0x162 sleepq_wait(c63ebcd4,0,c5fb5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63ebcd4,c63ebe4c,c0f672f1,74,c63ebcdc,...) at _cv_wait+0x260 usb_process(c63ebccc,c5fb5d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63ebccc,c5fb5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fb5d60, ebp = 0 --- Tracing command usb pid 15 tid 100048 td 0xc63d08a0 sched_switch(c63d08a0,0,104,1a3,d6e89d60,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d08a0,0,c0f941e0,268,c63d08a0,...) at sleepq_switch+0x162 sleepq_wait(c63ebca4,0,c5fb2cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63ebca4,c63ebe4c,c0f672f1,74,c63ebcac,...) at _cv_wait+0x260 usb_process(c63ebc9c,c5fb2d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63ebc9c,c5fb2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fb2d60, ebp = 0 --- Tracing command usb pid 15 tid 100047 td 0xc63d0b80 sched_switch(c63d0b80,0,104,1a3,d6e887f8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d0b80,0,c0f941e0,268,c63d0b80,...) at sleepq_switch+0x162 sleepq_wait(c63d7dac,0,c5faccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63d7dac,c63d7ec4,c0f672f1,74,c63d7db4,...) at _cv_wait+0x260 usb_process(c63d7da4,c5facd28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63d7da4,c5facd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5facd60, ebp = 0 --- Tracing command usb pid 15 tid 100046 td 0xc63d2000 sched_switch(c63d2000,0,104,1a3,2e4de870,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d2000,0,c0f941e0,268,c63d2000,...) at sleepq_switch+0x162 sleepq_wait(c63d7d7c,0,c5fa9cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63d7d7c,c63d7ec4,c0f672f1,74,c63d7d84,...) at _cv_wait+0x260 usb_process(c63d7d74,c5fa9d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63d7d74,c5fa9d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fa9d60, ebp = 0 --- Tracing command usb pid 15 tid 100045 td 0xc63d22e0 sched_switch(c63d22e0,0,104,1a3,cfd42da8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d22e0,0,c0f941e0,268,c63d22e0,...) at sleepq_switch+0x162 sleepq_wait(c63d7d4c,0,c5fa6cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63d7d4c,c63d7ec4,c0f672f1,74,c63d7d54,...) at _cv_wait+0x260 usb_process(c63d7d44,c5fa6d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63d7d44,c5fa6d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fa6d60, ebp = 0 --- Tracing command usb pid 15 tid 100044 td 0xc63d25c0 sched_switch(c63d25c0,0,104,1a3,cfd41b00,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d25c0,0,c0f941e0,268,c63d25c0,...) at sleepq_switch+0x162 sleepq_wait(c63d7d1c,0,c5fa3cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63d7d1c,c63d7ec4,c0f672f1,74,c63d7d24,...) at _cv_wait+0x260 usb_process(c63d7d14,c5fa3d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63d7d14,c5fa3d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fa3d60, ebp = 0 --- Tracing command usb pid 15 tid 100042 td 0xc63d2b80 sched_switch(c63d2b80,0,104,1a3,cfd401e8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d2b80,0,c0f941e0,268,c63d2b80,...) at sleepq_switch+0x162 sleepq_wait(c63c0dac,0,c5f9ccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63c0dac,c63c0ec4,c0f672f1,74,c63c0db4,...) at _cv_wait+0x260 usb_process(c63c0da4,c5f9cd28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63c0da4,c5f9cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f9cd60, ebp = 0 --- Tracing command usb pid 15 tid 100041 td 0xc63d3000 sched_switch(c63d3000,0,104,1a3,2e4d7c30,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d3000,0,c0f941e0,268,c63d3000,...) at sleepq_switch+0x162 sleepq_wait(c63c0d7c,0,c5f99cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63c0d7c,c63c0ec4,c0f672f1,74,c63c0d84,...) at _cv_wait+0x260 usb_process(c63c0d74,c5f99d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63c0d74,c5f99d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f99d60, ebp = 0 --- Tracing command usb pid 15 tid 100040 td 0xc63a6000 sched_switch(c63a6000,0,104,1a3,cf5990b8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63a6000,0,c0f941e0,268,c63a6000,...) at sleepq_switch+0x162 sleepq_wait(c63c0d4c,0,c5f96cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63c0d4c,c63c0ec4,c0f672f1,74,c63c0d54,...) at _cv_wait+0x260 usb_process(c63c0d44,c5f96d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63c0d44,c5f96d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f96d60, ebp = 0 --- Tracing command usb pid 15 tid 100039 td 0xc63a62e0 sched_switch(c63a62e0,0,104,1a3,cf597b98,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63a62e0,0,c0f941e0,268,c63a62e0,...) at sleepq_switch+0x162 sleepq_wait(c63c0d1c,0,c5f93cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63c0d1c,c63c0ec4,c0f672f1,74,c63c0d24,...) at _cv_wait+0x260 usb_process(c63c0d14,c5f93d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63c0d14,c5f93d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f93d60, ebp = 0 --- Tracing command usb pid 15 tid 100037 td 0xc63a68a0 sched_switch(c63a68a0,0,104,1a3,cf596390,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63a68a0,0,c0f941e0,268,c63a68a0,...) at sleepq_switch+0x162 sleepq_wait(c63aedac,0,c5f8ccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63aedac,c63aeec4,c0f672f1,74,c63aedb4,...) at _cv_wait+0x260 usb_process(c63aeda4,c5f8cd28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63aeda4,c5f8cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f8cd60, ebp = 0 --- Tracing command usb pid 15 tid 100036 td 0xc63a6b80 sched_switch(c63a6b80,0,104,1a3,2e4cd1e8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63a6b80,0,c0f941e0,268,c63a6b80,...) at sleepq_switch+0x162 sleepq_wait(c63aed7c,0,c5f89cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63aed7c,c63aeec4,c0f672f1,74,c63aed84,...) at _cv_wait+0x260 usb_process(c63aed74,c5f89d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63aed74,c5f89d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f89d60, ebp = 0 --- Tracing command usb pid 15 tid 100035 td 0xc63a8000 sched_switch(c63a8000,0,104,1a3,cedf6c88,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63a8000,0,c0f941e0,268,c63a8000,...) at sleepq_switch+0x162 sleepq_wait(c63aed4c,0,c5f86cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63aed4c,c63aeec4,c0f672f1,74,c63aed54,...) at _cv_wait+0x260 usb_process(c63aed44,c5f86d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63aed44,c5f86d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f86d60, ebp = 0 --- Tracing command usb pid 15 tid 100034 td 0xc63a82e0 sched_switch(c63a82e0,0,104,1a3,cedf57d8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63a82e0,0,c0f941e0,268,c63a82e0,...) at sleepq_switch+0x162 sleepq_wait(c63aed1c,0,c5f83cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63aed1c,c63aeec4,c0f672f1,74,c63aed24,...) at _cv_wait+0x260 usb_process(c63aed14,c5f83d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63aed14,c5f83d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f83d60, ebp = 0 --- Tracing command usb pid 15 tid 100032 td 0xc63a88a0 sched_switch(c63a88a0,0,104,1a3,cedf3f78,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63a88a0,0,c0f941e0,268,c63a88a0,...) at sleepq_switch+0x162 sleepq_wait(c6395dac,0,c5f7ccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c6395dac,c6395ec4,c0f672f1,74,c6395db4,...) at _cv_wait+0x260 usb_process(c6395da4,c5f7cd28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c6395da4,c5f7cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f7cd60, ebp = 0 --- Tracing command usb pid 15 tid 100031 td 0xc63a8b80 sched_switch(c63a8b80,0,104,1a3,2e4cb570,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63a8b80,0,c0f941e0,268,c63a8b80,...) at sleepq_switch+0x162 sleepq_wait(c6395d7c,0,c5f79cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c6395d7c,c6395ec4,c0f672f1,74,c6395d84,...) at _cv_wait+0x260 usb_process(c6395d74,c5f79d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c6395d74,c5f79d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f79d60, ebp = 0 --- Tracing command usb pid 15 tid 100030 td 0xc632b2e0 sched_switch(c632b2e0,0,104,1a3,ce636e90,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c632b2e0,0,c0f941e0,268,c632b2e0,...) at sleepq_switch+0x162 sleepq_wait(c6395d4c,0,c5f76cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c6395d4c,c6395ec4,c0f672f1,74,c6395d54,...) at _cv_wait+0x260 usb_process(c6395d44,c5f76d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c6395d44,c5f76d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f76d60, ebp = 0 --- Tracing command usb pid 15 tid 100029 td 0xc632b5c0 sched_switch(c632b5c0,0,104,1a3,ce635a50,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c632b5c0,0,c0f941e0,268,c632b5c0,...) at sleepq_switch+0x162 sleepq_wait(c6395d1c,0,c5f73cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c6395d1c,c6395ec4,c0f672f1,74,c6395d24,...) at _cv_wait+0x260 usb_process(c6395d14,c5f73d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c6395d14,c5f73d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f73d60, ebp = 0 --- Tracing command yarrow pid 14 tid 100013 td 0xc61ae2e0 sched_switch(c61ae2e0,0,104,1a3,a0749f68,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c61ae2e0,0,c0f941e0,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c112a7e4,0,c0f80f1a,2,0,...) at sleepq_timedwait+0x6b _sleep(c112a7e4,0,0,c0f80f1a,64) at _sleep+0x36a pause(c0f80f1a,64,c0f42f30,111,0,...) at pause+0xa0 random_kthread(0,c5ed7d28,c0f88d50,3db,c61ab000,...) at random_kthread+0x1ef fork_exit(c0805cf0,0,c5ed7d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ed7d60, ebp = 0 --- Tracing command geom pid 13 tid 100011 td 0xc61ae8a0 sched_switch(c61ae8a0,0,104,1a3,2e51f2b8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c61ae8a0,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c11283e8,5c,c0f80f1a,0,0,...) at sleepq_wait+0x63 _sleep(c11283e8,c1128348,25c,c0f80f1a,0,...) at _sleep+0x39c g_io_schedule_down(c61ae8a0,0,c0f82ac1,6c,c5ecdd14,...) at g_io_schedule_down+0x5b g_down_procbody(0,c5ecdd28,c0f88d50,3db,c61ab2c4,...) at g_down_procbody+0x9d fork_exit(c09ef410,0,c5ecdd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ecdd60, ebp = 0 --- Tracing command geom pid 13 tid 100010 td 0xc61aeb80 sched_switch(c61aeb80,0,104,1a3,2e52cc30,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c61aeb80,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c11283e4,5c,c0f80f1a,0,0,...) at sleepq_wait+0x63 _sleep(c11283e4,c1128368,25c,c0f80f1a,0,...) at _sleep+0x39c g_io_schedule_up(c61aeb80,0,c0f82ac1,5f,c5ecad14,...) at g_io_schedule_up+0x134 g_up_procbody(0,c5ecad28,c0f88d50,3db,c61ab2c4,...) at g_up_procbody+0x9d fork_exit(c09ef4b0,0,c5ecad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ecad60, ebp = 0 --- Tracing command geom pid 13 tid 100009 td 0xc61f9000 sched_switch(c61f9000,0,104,1a3,25b85bd0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c61f9000,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c11283dc,5c,c0f80f1a,0,0,...) at sleepq_wait+0x63 _sleep(c11283dc,c112830c,25c,c0f80f1a,0,...) at _sleep+0x39c g_run_events(c1138640,0,c0f82ac1,79,c5ec7d14,...) at g_run_events+0x54b g_event_procbody(0,c5ec7d28,c0f88d50,3db,c61ab2c4,...) at g_event_procbody+0x90 fork_exit(c09ef550,0,c5ec7d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ec7d60, ebp = 0 --- Tracing command intr pid 12 tid 100056 td 0xc640e8a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100054 td 0xc6411000 sched_switch(c6411000,0,109,1a3,2e503268,...) at sched_switch+0x3ec mi_switch(109,0,c0f8908b,554,c6411000,...) at mi_switch+0x1ff ithread_loop(c61f86e0,ec515d28,c0f88d50,3db,c61ab588,...) at ithread_loop+0x1fb fork_exit(c0a31f50,c61f86e0,ec515d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec515d60, ebp = 0 --- Tracing command intr pid 12 tid 100053 td 0xc64112e0 sched_switch(c64112e0,0,109,1a3,828f200,...) at sched_switch+0x3ec mi_switch(109,0,c0f8908b,554,c64112e0,...) at mi_switch+0x1ff ithread_loop(c61f86f0,c5fe5d28,c0f88d50,3db,c61ab588,...) at ithread_loop+0x1fb fork_exit(c0a31f50,c61f86f0,c5fe5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fe5d60, ebp = 0 --- Tracing command intr pid 12 tid 100052 td 0xc64115c0 sched_switch(c64115c0,0,109,1a3,d1b813c8,...) at sched_switch+0x3ec mi_switch(109,0,c0f8908b,554,c64115c0,...) at mi_switch+0x1ff ithread_loop(c61f8700,c5fe1d28,c0f88d50,3db,c61ab588,...) at ithread_loop+0x1fb fork_exit(c0a31f50,c61f8700,c5fe1d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fe1d60, ebp = 0 --- Tracing command intr pid 12 tid 100043 td 0xc63d28a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100038 td 0xc63a65c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100033 td 0xc63a85c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100028 td 0xc632b8a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100023 td 0xc632c8a0 sched_switch(c632c8a0,0,109,1a3,39145320,...) at sched_switch+0x3ec mi_switch(109,0,c0f8908b,554,c632c8a0,...) at mi_switch+0x1ff ithread_loop(c61f8750,c5ef5d28,c0f88d50,3db,c61ab588,...) at ithread_loop+0x1fb fork_exit(c0a31f50,c61f8750,c5ef5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ef5d60, ebp = 0 --- Tracing command intr pid 12 tid 100022 td 0xc632cb80 sched_switch(c632cb80,0,109,1a3,624cd118,...) at sched_switch+0x3ec mi_switch(109,0,c0f8908b,554,c632cb80,...) at mi_switch+0x1ff ithread_loop(c61f8760,c5ef2d28,c0f88d50,3db,c61ab588,...) at ithread_loop+0x1fb fork_exit(c0a31f50,c61f8760,c5ef2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ef2d60, ebp = 0 --- Tracing command intr pid 12 tid 100020 td 0xc61f9b80 sched_switch(c61f9b80,0,109,1a3,38abc378,...) at sched_switch+0x3ec mi_switch(109,0,c0f8908b,554,c61f9b80,...) at mi_switch+0x1ff ithread_loop(c61f8770,c5eecd28,c0f88d50,3db,c61ab588,...) at ithread_loop+0x1fb fork_exit(c0a31f50,c61f8770,c5eecd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5eecd60, ebp = 0 --- Tracing command intr pid 12 tid 100014 td 0xc632b000 sched_switch(c632b000,0,109,1a3,8293448,...) at sched_switch+0x3ec mi_switch(109,0,c0f8908b,554,c632b000,...) at mi_switch+0x1ff ithread_loop(c61f8780,c5edad28,c0f88d50,3db,c61ab588,...) at ithread_loop+0x1fb fork_exit(c0a31f50,c61f8780,c5edad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5edad60, ebp = 0 --- Tracing command intr pid 12 tid 100008 td 0xc61f92e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100007 td 0xc61f95c0 sched_switch(c61f95c0,0,109,1a3,a0748570,...) at sched_switch+0x3ec mi_switch(109,0,c0f8908b,554,c61f95c0,...) at mi_switch+0x1ff ithread_loop(c61f87d0,c5ec1d28,c0f88d50,3db,c61ab588,...) at ithread_loop+0x1fb fork_exit(c0a31f50,c61f87d0,c5ec1d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ec1d60, ebp = 0 --- Tracing command intr pid 12 tid 100006 td 0xc61ad000 sched_switch(c61ad000,0,103,19e,a0748350,...) at sched_switch+0x3ec mi_switch(103,0,c0f94bec,2e2,300,...) at mi_switch+0x1ff turnstile_wait(c6172100,c9317000,0,1bc,c112a090,...) at turnstile_wait+0x4a4 _mtx_lock_sleep(c112a090,c61ad000,0,c0f8caeb,94,...) at _mtx_lock_sleep+0x15c _mtx_lock_flags(c112a090,0,c0f8caeb,94,c5ebec90,...) at _mtx_lock_flags+0xff lock_mtx(c112a090,1,c0f8fe66,216,1c,...) at lock_mtx+0x29 softclock(c112ba20,c5ebecb8,c0a4b49d,c1138640,c61f4cb8,...) at softclock+0x1ea intr_event_execute_handlers(c61ab588,c61f4c80,c0f8908b,55e,c61ad000,...) at intr_event_execute_handlers+0x125 ithread_loop(c61f87e0,c5ebed28,c0f88d50,3db,c61ab588,...) at ithread_loop+0xa9 fork_exit(c0a31f50,c61f87e0,c5ebed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ebed60, ebp = 0 --- Tracing command intr pid 12 tid 100005 td 0xc61ad2e0 sched_switch(c61ad2e0,0,109,1a3,c1985078,...) at sched_switch+0x3ec mi_switch(109,0,c0f8908b,554,c61ad2e0,...) at mi_switch+0x1ff ithread_loop(c61f87f0,c5ebbd28,c0f88d50,3db,c61ab588,...) at ithread_loop+0x1fb fork_exit(c0a31f50,c61f87f0,c5ebbd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ebbd60, ebp = 0 --- Tracing command idle pid 11 tid 100004 td 0xc61ad5c0 cpustop_handler(1,13,c5eb5c04,c0dc0cc6,f5,...) at cpustop_handler+0x28 ipi_nmi_handler(f5,0,170bd4a,c0da5555,c61ab84c,...) at ipi_nmi_handler+0x47 trap(c5eb5c10) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0da5555, esp = 0xc5eb5c50, ebp = 0xc5eb5c50 --- acpi_cpu_c1(c5eb5c6c,c10f07b0,1,301,ded9,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(ffffffff,c5eb5ca0,c0db37e9,0,ffffffff,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,ffffffff,c1138cc0,1,c5eb5ce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c5eb5cc4,c0f90852,3c0,c61ad5c0,...) at cpu_idle+0x89 sched_idletd(0,c5eb5d28,c0f88d50,3db,c61ab84c,...) at sched_idletd+0x263 fork_exit(c0a82d70,0,c5eb5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5eb5d60, ebp = 0 --- Tracing command idle pid 11 tid 100003 td 0xc61ad8a0 sched_switch(c61ad8a0,0,608,19e,98d83b48,...) at sched_switch+0x3ec mi_switch(608,0,c0f8ef5f,cd,ffffffff,...) at mi_switch+0x1ff critical_exit(0,ffffffff,c1138640,1,c5eb2ce8,...) at critical_exit+0xb5 cpu_idle(0,c5eb2cc4,c0f90852,3c0,c61ad8a0,...) at cpu_idle+0x97 sched_idletd(0,c5eb2d28,c0f88d50,3db,c61ab84c,...) at sched_idletd+0x263 fork_exit(c0a82d70,0,c5eb2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5eb2d60, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc61adb80 sched_switch(c61adb80,0,104,1a3,38b7f248,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c61adb80,0,c0f941e0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f941e0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c61abb10,6c,c0f9715f,100,0,...) at sleepq_wait_sig+0x17 _sleep(c61abb10,c61abb98,16c,c0f9715f,0,...) at _sleep+0x385 kern_wait(c61adb80,ffffffff,c5eaec60,2,0,...) at kern_wait+0x626 sys_wait4(c61adb80,c5eaecec,c,c,282,...) at sys_wait4+0x3b syscall(c5eaed28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x80551d3, esp = 0xbfbfe81c, ebp = 0xbfbfe838 --- Tracing command audit pid 10 tid 100001 td 0xc61ae000 sched_switch(c61ae000,0,104,1a3,ce5ee0a0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c61ae000,0,c0f941e0,268,c61ae000,...) at sleepq_switch+0x162 sleepq_wait(c12a75e0,0,c5eabc8c,1,0,...) at sleepq_wait+0x63 _cv_wait(c12a75e0,c12a75c4,c0fbf92f,194,0,...) at _cv_wait+0x260 audit_worker(0,c5eabd28,c0f88d50,3db,c61ac000,...) at audit_worker+0x84 fork_exit(c0c5b830,0,c5eabd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5eabd60, ebp = 0 --- Tracing command kernel pid 0 tid 100068 td 0xc88f62e0 sched_switch(c88f62e0,0,104,1a3,8fa731d0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c88f62e0,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c64258c0,0,c0f80f1a,0,0,...) at sleepq_wait+0x63 _sleep(c64258c0,c64258d8,0,c0f80f1a,0,...) at _sleep+0x39c taskqueue_thread_loop(c1308e60,ec549d28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a9fbe0,c1308e60,ec549d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec549d60, ebp = 0 --- Tracing command kernel pid 0 tid 100058 td 0xc640e2e0 sched_switch(c640e2e0,0,104,1a3,2e4ed530,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c640e2e0,0,c0f941e0,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c112a7e4,0,c0f80f1a,2,0,...) at sleepq_timedwait+0x6b _sleep(c112a7e4,0,0,c0f80f1a,bb8) at _sleep+0x36a pause(c0f80f1a,bb8,124,122,c1128548,...) at pause+0xa0 deadlkres(0,ec52bd28,c0f88d50,3db,c11284c0,...) at deadlkres+0x324 fork_exit(c0a11720,0,ec52bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec52bd60, ebp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xc632bb80 sched_switch(c632bb80,0,104,1a3,ce5cd568,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c632bb80,0,c0f941e0,268,c632bb80,...) at sleepq_switch+0x162 sleepq_wait(c6301840,0,c0f8f499,c0f80f1a,0,...) at sleepq_wait+0x63 msleep_spin(c6301840,c6301858,c0f80f1a,0,c0f8caeb,...) at msleep_spin+0x259 taskqueue_thread_loop(c634876c,c5f6cd28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9fbe0,c634876c,c5f6cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f6cd60, ebp = 0 --- Tracing command kernel pid 0 tid 100026 td 0xc632c000 sched_switch(c632c000,0,104,1a3,ce5cc328,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c632c000,0,c0f941e0,268,c632c000,...) at sleepq_switch+0x162 sleepq_wait(c6301880,0,c0f8f499,c0f80f1a,0,...) at sleepq_wait+0x63 msleep_spin(c6301880,c6301898,c0f80f1a,0,c0f8caeb,...) at msleep_spin+0x259 taskqueue_thread_loop(c6383544,c5f69d28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9fbe0,c6383544,c5f69d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f69d60, ebp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xc632c2e0 sched_switch(c632c2e0,0,104,1a3,ce5cb268,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c632c2e0,0,c0f941e0,268,c632c2e0,...) at sleepq_switch+0x162 sleepq_wait(c6301940,0,c0f8f499,c0f80f1a,0,...) at sleepq_wait+0x63 msleep_spin(c6301940,c6301958,c0f80f1a,0,c0f8caeb,...) at msleep_spin+0x259 taskqueue_thread_loop(c6348c6c,c5f3ed28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9fbe0,c6348c6c,c5f3ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f3ed60, ebp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xc632c5c0 sched_switch(c632c5c0,0,104,1a3,2e562ec8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c632c5c0,0,c0f941e0,268,c632c5c0,...) at sleepq_switch+0x162 sleepq_wait(c6301980,0,c0f8f499,c0f80f1a,0,...) at sleepq_wait+0x63 msleep_spin(c6301980,c6301998,c0f80f1a,0,c0f8caeb,...) at msleep_spin+0x259 taskqueue_thread_loop(c636f544,c5f3bd28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9fbe0,c636f544,c5f3bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f3bd60, ebp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xc61f98a0 sched_switch(c61f98a0,0,104,1a3,4b8cf160,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c61f98a0,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c6301e80,0,c0f80f1a,0,0,...) at sleepq_wait+0x63 _sleep(c6301e80,c6301e98,0,c0f80f1a,0,...) at _sleep+0x39c taskqueue_thread_loop(c11473ac,c5eefd28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a9fbe0,c11473ac,c5eefd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5eefd60, ebp = 0 --- Tracing command kernel pid 0 tid 100019 td 0xc632a000 sched_switch(c632a000,0,104,1a3,f8266b28,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c632a000,0,c0f941e0,268,c632a000,...) at sleepq_switch+0x162 sleepq_wait(c6302040,0,c0f8f499,c0f80f1a,0,...) at sleepq_wait+0x63 msleep_spin(c6302040,c6302058,c0f80f1a,0,c0f8caeb,...) at msleep_spin+0x259 taskqueue_thread_loop(c10f3438,c5ee9d28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9fbe0,c10f3438,c5ee9d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ee9d60, ebp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xc632a2e0 sched_switch(c632a2e0,0,104,1a3,f8265db0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c632a2e0,0,c0f941e0,268,c632a2e0,...) at sleepq_switch+0x162 sleepq_wait(c6302040,0,c0f8f499,c0f80f1a,0,...) at sleepq_wait+0x63 msleep_spin(c6302040,c6302058,c0f80f1a,0,c0f8caeb,...) at msleep_spin+0x259 taskqueue_thread_loop(c10f3438,c5ee6d28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9fbe0,c10f3438,c5ee6d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ee6d60, ebp = 0 --- Tracing command kernel pid 0 tid 100017 td 0xc632a5c0 sched_switch(c632a5c0,0,104,1a3,f8264f30,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c632a5c0,0,c0f941e0,268,c632a5c0,...) at sleepq_switch+0x162 sleepq_wait(c6302040,0,c0f8f499,c0f80f1a,0,...) at sleepq_wait+0x63 msleep_spin(c6302040,c6302058,c0f80f1a,0,c0f8caeb,...) at msleep_spin+0x259 taskqueue_thread_loop(c10f3438,c5ee3d28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9fbe0,c10f3438,c5ee3d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ee3d60, ebp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xc632a8a0 sched_switch(c632a8a0,0,104,1a3,f82570c0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c632a8a0,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c6302080,0,c0f80f1a,0,0,...) at sleepq_wait+0x63 _sleep(c6302080,c6302098,0,c0f80f1a,0,...) at _sleep+0x39c taskqueue_thread_loop(c1128e38,c5ee0d28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a9fbe0,c1128e38,c5ee0d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ee0d60, ebp = 0 --- Tracing command kernel pid 0 tid 100015 td 0xc632ab80 sched_switch(c632ab80,0,104,1a3,f82560e8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c632ab80,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c63020c0,0,c0f80f1a,0,0,...) at sleepq_wait+0x63 _sleep(c63020c0,c63020d8,0,c0f80f1a,0,...) at _sleep+0x39c taskqueue_thread_loop(c12a7bc4,c5eddd28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a9fbe0,c12a7bc4,c5eddd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5eddd60, ebp = 0 --- Tracing command kernel pid 0 tid 100012 td 0xc61ae5c0 sched_switch(c61ae5c0,0,104,1a3,600ebee8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c61ae5c0,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c6191e80,0,c0f80f1a,0,0,...) at sleepq_wait+0x63 _sleep(c6191e80,c6191e98,0,c0f80f1a,0,...) at _sleep+0x39c taskqueue_thread_loop(c1145dc0,c5ed0d28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a9fbe0,c1145dc0,c5ed0d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ed0d60, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc1128790 sched_switch(c1128790,0,104,1a3,af754a00,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,54,...) at mi_switch+0x1ff sleepq_switch(c1128790,0,c0f941e0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c11284c0,54,c0f9116c,0,0,...) at sleepq_timedwait+0x6b _sleep(c11284c0,0,54,c0f9116c,2710,...) at _sleep+0x36a scheduler(0,181ec00,181ec00,181e000,1825000,...) at scheduler+0x240 mi_startup() at mi_startup+0xac begin() at begin+0x2c db:0:allt> call doadump Physical memory: 2026 MB Dumping 175 MB: 160 144 128 112 96 80 64 48 32 16 Dump complete = 0 db:0:doadump> reset (kgdb) proc 23766 [Switching to thread 74 (Thread 100275)]#0 doadump (textdump=-291008564) at pcpu.h:244 244 __asm("movl %%fs:0,%0" : "=r" (td)); (kgdb) bt #0 doadump (textdump=-291008564) at pcpu.h:244 #1 0xc0510a39 in db_fncall (dummy1=-1062661366, dummy2=0, dummy3=-1, dummy4=0xeea79060 "t\220§î") at /mnt/pho/src/sys/ddb/db_command.c:573 #2 0xc0510e6f in db_command (last_cmdp=0xc10f15fc, cmd_table=0x0, dopager=0) at /mnt/pho/src/sys/ddb/db_command.c:449 #3 0xc0510f24 in db_command_script (command=0xc10f2508 "call doadump") at /mnt/pho/src/sys/ddb/db_command.c:520 #4 0xc0515310 in db_script_exec (scriptname=0xc10f1e60 "doadump", warnifnotfound=Variable "warnifnotfound" is not available. ) at /mnt/pho/src/sys/ddb/db_script.c:302 #5 0xc05153a1 in db_run_cmd (addr=-1055931968, have_addr=0, count=-1066926887, modif=0xeea79198 "") at /mnt/pho/src/sys/ddb/db_script.c:375 #6 0xc0510e31 in db_command (last_cmdp=0xc10f15fc, cmd_table=0x0, dopager=1) at /mnt/pho/src/sys/ddb/db_command.c:449 #7 0xc0510f8a in db_command_loop () at /mnt/pho/src/sys/ddb/db_command.c:502 #8 0xc0512f7d in db_trap (type=3, code=0) at /mnt/pho/src/sys/ddb/db_main.c:229 #9 0xc0a917c6 in kdb_trap (type=3, code=0, tf=0xeea7934c) at /mnt/pho/src/sys/kern/subr_kdb.c:629 #10 0xc0dc12fb in trap (frame=0xeea7934c) at /mnt/pho/src/sys/i386/i386/trap.c:722 #11 0xc0da9b8c in calltrap () at /mnt/pho/src/sys/i386/i386/exception.s:168 #12 0xc0a9130a in kdb_enter (why=0xc0e02aae "break", msg=0xc0f927c6 "Break to debugger") at cpufunc.h:71 #13 0xc0a9165a in kdb_alt_break_internal (key=2, state=Variable "state" is not available. ) at /mnt/pho/src/sys/kern/subr_kdb.c:324 #14 0xc0a916e3 in kdb_alt_break (key=2, state=0xc642d058) at /mnt/pho/src/sys/kern/subr_kdb.c:344 #15 0xc08c927e in uart_intr (arg=0xc642d000) at /mnt/pho/src/sys/dev/uart/uart_core.c:172 #16 0xc0a328b6 in intr_event_handle (ie=0xc61f2d80, frame=0xeea7944c) at /mnt/pho/src/sys/kern/kern_intr.c:1430 #17 0xc0daede9 in intr_execute_handlers (isrc=0xc61f10d0, frame=0xeea7944c) at /mnt/pho/src/sys/i386/i386/intr_machdep.c:256 #18 0xc0dd9596 in lapic_handle_intr (vector=59, frame=0xeea7944c) at /mnt/pho/src/sys/x86/x86/local_apic.c:780 #19 0xc0da9f55 in Xapic_isr1 () at apic_vector.s:89 #20 0xc0aa5e95 in witness_lock (lock=0xc9177000, flags=8, file=0xc0fc29eb "/mnt/pho/src/sys/ufs/ffs/ffs_softdep.c", line=13265) at /mnt/pho/src/sys/kern/subr_witness.c:1393 #21 0xc0a4bccc in _mtx_lock_flags (m=0xc9177000, opts=0, file=0xc0fc29eb "/mnt/pho/src/sys/ufs/ffs/ffs_softdep.c", line=13265) at /mnt/pho/src/sys/kern/kern_mutex.c:209 #22 0xc0c8e070 in softdep_check_suspend (mp=0xc9177000, devvp=0xcb487570, softdep_deps=1, softdep_accdeps=5949655, secondary_writes=0, secondary_accwrites=1087097) at /mnt/pho/src/sys/ufs/ffs/ffs_softdep.c:13265 #23 0xc0c9ffbc in ffs_sync (mp=0xc9177000, waitfor=1) at /mnt/pho/src/sys/ufs/ffs/ffs_vfsops.c:1545 #24 0xc0b093b2 in vfs_write_suspend (mp=0xc9177000) at /mnt/pho/src/sys/kern/vfs_vnops.c:1203 #25 0xc0c83e0f in ffs_snapshot (mp=0xc9177000, snapfile=0xc8fd8560 "/mnt/.snap/suj26") at /mnt/pho/src/sys/ufs/ffs/ffs_snapshot.c:417 #26 0xc0ca23d7 in ffs_mount (mp=0xc9177000) at /mnt/pho/src/sys/ufs/ffs/ffs_vfsops.c:481 #27 0xc0af18ab in vfs_donmount (td=0xc9317000, fsflags=69888, fsoptions=0xc8ac7480) at /mnt/pho/src/sys/kern/vfs_mount.c:957 #28 0xc0af1f4f in sys_nmount (td=0xc9317000, uap=0xeea79cec) at /mnt/pho/src/sys/kern/vfs_mount.c:420 #29 0xc0dc08c7 in syscall (frame=0xeea79d28) at subr_syscall.c:135 #30 0xc0da9bf1 in Xint0x80_syscall () at /mnt/pho/src/sys/i386/i386/exception.s:266 #31 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 25 #25 0xc0c83e0f in ffs_snapshot (mp=0xc9177000, snapfile=0xc8fd8560 "/mnt/.snap/suj26") at /mnt/pho/src/sys/ufs/ffs/ffs_snapshot.c:417 417 if ((error = vfs_write_suspend(vp->v_mount)) != 0) { (kgdb) p *mp $1 = {mnt_mtx = {lock_object = {lo_name = 0xc0f80679 "struct mount mtx", lo_flags = 16973824, lo_data = 0, lo_witness = 0xc6154450}, mtx_lock = 3375460352}, mnt_gen = 378, mnt_list = { tqe_next = 0x0, tqe_prev = 0xc8922018}, mnt_op = 0xc10dd9e0, mnt_vfc = 0xc10dd840, mnt_vnodecovered = 0xc8f15ae0, mnt_syncer = 0xca8a9c3c, mnt_ref = 10, mnt_nvnodelist = { tqh_first = 0xcb577828, tqh_last = 0xca8c52cc}, mnt_nvnodelistsize = 7, mnt_writeopcount = 0, mnt_kern_flag = 1744830532, mnt_flag = 4311814400, mnt_noasync = 1, mnt_opt = 0xc6245810, mnt_optnew = 0xc6245680, mnt_maxsymlinklen = 120, mnt_stat = {f_version = 537068824, f_type = 53, f_flags = 4294971648, f_bsize = 2048, f_iosize = 16384, f_blocks = 1013011, f_bfree = 1003432, f_bavail = 922392, f_files = 282622, f_ffree = 282616, f_syncwrites = 0, f_asyncwrites = 0, f_syncreads = 0, f_asyncreads = 0, f_spare = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, f_namemax = 255, f_owner = 0, f_fsid = {val = {1329261325, 1386667308}}, f_charspare = '\0' , f_fstypename = "ufs", '\0' , f_mntfromname = "/dev/md5a", '\0' , f_mntonname = "/mnt", '\0' }, mnt_cred = 0xc9b83400, mnt_data = 0xc8ed9200, mnt_time = 0, mnt_iosize_max = 131072, mnt_export = 0xc9f68200, mnt_label = 0x0, mnt_hashseed = 3250124098, mnt_lockref = 1, mnt_secondary_writes = 0, mnt_secondary_accwrites = 1087097, mnt_susp_owner = 0xc9317000, mnt_gjprovider = 0x0, mnt_explock = {lock_object = {lo_name = 0xc0f8068a "explock", lo_flags = 91422720, lo_data = 0, lo_witness = 0xc615a1c0}, lk_lock = 1, lk_exslpfail = 0, lk_timo = 0, lk_pri = 96, lk_stack = {depth = 6, pcs = {3231988062, 3232665687, 3232700654, 3232702287, 3235645639, 3235552241, 0 }}}} (kgdb) -- Test scenario: suj26.sh