KDB: debugger backends: ddb KDB: current backend: ddb ---<>--- Copyright (c) 1992-2019 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 13.0-CURRENT #1 r344421M: Fri Feb 22 12:22:04 CET 2019 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 FreeBSD clang version 7.0.1 (tags/RELEASE_701/final 349250) (based on LLVM 7.0.1) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Entering uma_startup with 9 boot pages configured startup_alloc from "UMA Kegs", 8 boot pages left startup_alloc from "UMA Zones", 7 boot pages left startup_alloc from "UMA Hash", 6 boot pages left startup_alloc from "vmem btag", 5 boot pages left startup_alloc from "UMA Zones", 4 boot pages left startup_alloc from "MAP", 3 boot pages left startup_alloc from "KMAP ENTRY", 2 boot pages left startup_alloc from "vmem", 1 boot pages left Entering uma_startup2 with 0 boot pages left VT(vga): resolution 640x480 CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz 686-class CPU) Origin="AuthenticAMD" Id=0x100f23 Family=0x10 Model=0x2 Stepping=3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff SVM: NP,NAsids=64 TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 8250626048 (7868 MB) Event timer "LAPIC" quality 100 ACPI APIC Table: <060508 APIC1048> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) random: unblocking device. ioapic0 irqs 0-23 on motherboard Launching APs: 2 1 3 Timecounter "TSC" frequency 1800015951 Hz quality 800 random: entropy device external interface [ath_hal] loaded module_register_init: MOD_LOAD (vesa, 0x16588d0, 0) error 19 kbd1 at kbdmux0 nexus0 vtvga0: on motherboard cryptosoft0: on motherboard acpi0: <060508 RSDT1048> on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 vgapci0: port 0xc000-0xc0ff mem 0xf0000000-0xf7ffffff,0xfe9f0000-0xfe9fffff,0xfe800000-0xfe8fffff irq 18 at device 5.0 on pci1 vgapci0: Boot video device hdac0: mem 0xfe9e8000-0xfe9ebfff irq 19 at device 5.2 on pci1 pcib2: at device 7.0 on pci0 pci2: on pcib2 re0: port 0xd800-0xd8ff mem 0xfeaff000-0xfeafffff irq 19 at device 0.0 on pci2 re0: Using 1 MSI message re0: Chip rev. 0x38000000 re0: MAC rev. 0x00000000 miibus0: on re0 rgephy0: PHY 1 on miibus0 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re0: Using defaults for TSO: 65518/35/2048 re0: Ethernet address: 00:21:85:15:33:7d ahci0: port 0xb000-0xb007,0xa000-0xa003,0x9000-0x9007,0x8000-0x8003,0x7000-0x700f mem 0xfe7ff800-0xfe7ffbff irq 22 at device 18.0 on pci0 ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier supported ahci0: quirks=0x7000 ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ohci0: mem 0xfe7fe000-0xfe7fefff irq 16 at device 19.0 on pci0 usbus0 on ohci0 usbus0: 12Mbps Full Speed USB v1.0 ohci1: mem 0xfe7fd000-0xfe7fdfff irq 17 at device 19.1 on pci0 usbus1 on ohci1 usbus1: 12Mbps Full Speed USB v1.0 ohci2: mem 0xfe7fc000-0xfe7fcfff irq 18 at device 19.2 on pci0 usbus2 on ohci2 usbus2: 12Mbps Full Speed USB v1.0 ohci3: mem 0xfe7fb000-0xfe7fbfff irq 17 at device 19.3 on pci0 usbus3 on ohci3 usbus3: 12Mbps Full Speed USB v1.0 ohci4: mem 0xfe7fa000-0xfe7fafff irq 18 at device 19.4 on pci0 usbus4 on ohci4 usbus4: 12Mbps Full Speed USB v1.0 ehci0: mem 0xfe7ff000-0xfe7ff0ff irq 19 at device 19.5 on pci0 ehci0: AMD SB600/700 quirk applied usbus5: EHCI version 1.0 usbus5 on ehci0 usbus5: 480Mbps High Speed USB v2.0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0 ata0: at channel 0 on atapci0 hdac1: mem 0xfe7f4000-0xfe7f7fff irq 16 at device 20.2 on pci0 isab0: at device 20.3 on pci0 isa0: on isab0 pcib3: at device 20.4 on pci0 pci3: on pcib3 pci3: at device 6.0 (no driver attached) amdtemp0: on hostb4 amdtemp0: Erratum 319: temperature measurement may be inaccurate acpi_button0: on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (115200,n,8,1) fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] ppc0: parallel port not found. hwpstate0: on cpu0 NULL mp in getnewvnode(9), tag crossmp Timecounters tick every 1.000 msec hdacc0: at cad 0 on hdac0 hdaa0: at nid 1 on hdacc0 pcm0: at nid 3 on hdaa0 hdacc1: at cad 3 on hdac1 hdaa1: at nid 1 on hdacc1 pcm1: at nid 20,22,21,23 and 24,26 on hdaa1 pcm2: at nid 27 and 25 on hdaa1 pcm3: at nid 30 on hdaa1 ugen5.1: at usbus5 ugen1.1: at usbus1 uhub0: ugen3.1: at usbus3 on usbus5 ugen0.1: at usbus0 uhub2: on usbus1 uhub3: on usbus0 ugen4.1: at usbus4 uhub1: on usbus3 uhub4: on usbus4 ugen2.1: at usbus2 uhub5: on usbus2 (aprobe0:ahcich0:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00 Expensive timeout(9) function: 0xa58f40(0x22108000) 0.009495899 s (aprobe0:ahcich0:0:15:0): CAM status: Command timeout (aprobe0:ahcich0:0:15:0): Error 5, Retries exhausted (aprobe0:ahcich0:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00 (aprobe0:ahcich0:0:15:0): CAM status: Command timeout (aprobe0:ahcich0:0:15:0): Error 5, Retries exhausted ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ATA8-ACS SATA 2.x device ada0: Serial Number 9VMJ99G5 ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 476940MB (976773168 512 byte sectors) cd0 at ata0 bus 0 scbus4 target 1 lun 0 cd0: <_NEC DVD_RW ND-4550A 1.06> Removable CD-ROM SCSI device cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered Trying to mount root from ufs:/dev/ada0s1a [rw]... WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. uhub4: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. warning: total configured swap (4194304 pages) exceeds maximum recommended amount (2009920 pages). warning: increase kern.maxswzone or reduce amount of swap. Starting file system checks: /dev/ada0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1a: clean, 605227 free (2635 frags, 75324 blocks, 0.3% fragmentation) /dev/ada0s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1f: clean, 785424 free (11200 frags, 96778 blocks, 1.1% fragmentation) /dev/label/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/label/tmp: clean, 37412366 free (38046 frags, 4671790 blocks, 0.0% fragmentation) /dev/ada0s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1d: clean, 833171 free (223187 frags, 76248 blocks, 2.2% fragmentation) /dev/ada0s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1e: clean, 2671090 free (155218 frags, 314484 blocks, 0.6% fragmentation) Mounting local filesystems:. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/arduino/lib /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/e2fsprogs /usr/local/lib/gcc7 /usr/local/lib/gcc8 /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/perl5/5.28/mach/CORE /usr/local/lib/pth /usr/local/lib/qt4 /usr/local/lib/qt5 /usr/local/llvm37/lib /usr/local/llvm39/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Setting hostname: x4.osted.lan. Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy: uhub0: 10 ports with 10 removable, self powered . lo0: link state changed to UP re0: link state changed to DOWN Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=8209b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 media: Ethernet autoselect (none) status: no carrier nd6 options=29 Starting devd. add host 127.0.0.1: gateway lo0 fib 0: route already in table add net default: gateway 192.168.1.1 add host ::1: gateway lo0 fib 0: route already in table add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Creating and/or trimming log files. Starting syslogd. Starting rpcbind. NFS access cache time=60 Starting watchdogd. No core dumps found. lock order reversal: 1st 0x12664cf4 bufwait (bufwait) @ kern/vfs_bio.c:3906 2nd 0x2fc21000 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:289 stack backtrace: #0 0x11624a1 at witness_debugger+0x81 #1 0x11621e9 at witness_checkorder+0xa09 #2 0x10ff054 at _sx_xlock+0x64 #3 0x143df36 at ufsdirhash_add+0x56 #4 0x1440d13 at ufs_direnter+0x483 #5 0x1449bdf at ufs_mkdir+0x83f #6 0x1734565 at VOP_MKDIR_APV+0x85 #7 0x11e03b4 at kern_mkdirat+0x1d4 #8 0x11e01d2 at sys_mkdir+0x32 #9 0x16ff051 at syscall+0x361 #10 0xffc033a7 at amdsmn_devclass+0xfd7dc3a7 Clearing /tmp (X related). Starting nfsuserd. Starting mountd. Starting nfsd. Starting statd. Starting lockd. Recovering vi editor sessions:. Updating motd:. Mounting late filesystems:. Starting ntpd. Starting powerd. Starting smartd. Feb 22 12:26:02 x4 smartd[624]: Device: /dev/ada0, WARNING: A firmware update for this drive may be available, Feb 22 12:26:02 x4 smartd[624]: see the following Seagate web pages: Feb 22 12:26:02 x4 smartd[624]: http://knowledge.seagate.com/articles/en_US/FAQ/207931en Feb 22 12:26:02 x4 smartd[624]: http://knowledge.seagate.com/articles/en_US/FAQ/213891en Configuring vt: keymap blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting sendmail_submit. Starting sendmail_msp_queue. Starting cron. Local package initialization: backup. Starting nfscbd. Starting default mousedmoused: unable to open /dev/psm0: No such file or directory . Starting inetd. Fri Feb 22 12:26 FreeBSD/i386 (x4.osted.lan) (ttyu0) login: Feb 22 12:26:17 x4 su[797]: pho to root on /dev/pts/0 20190222 12:26:37 all : syscall5.sh 03:00:26 syscall 567 (fhreadlink) 03:02:42 syscall 566 (fhlinkat) 03:04:46 syscall 565 (fhlink) 03:06:50 syscall 564 (getfhat) 03:08:59 syscall 563 (getrandom) Fatal double fault: eip = 0xffc030a0 esp = 0xffc07004 ebp = 0xfa6e8edc cpuid = 0; apic id = 00 panic: double fault cpuid = 0 time = 1550887746 KDB: stack backtrace: db_trace_self_wrapper(185c2e5,1dbde3c,0,1864c05,ffc08f60,...) at db_trace_self_wrapper+0x2a/frame 0xffc08f30 kdb_backtrace(1823570,5c70ab42,0,ffc08ff0,e,...) at kdb_backtrace+0x2d/frame 0xffc08f98 vpanic(17f66a0,ffc08ff0,ffc08ff0,ffc08ff8,16febef,...) at vpanic+0x141/frame 0xffc08fd0 panic(17f66a0,0,0,0,0,...) at panic+0x1b/frame 0xffc08fe4 dblfault_handler() at dblfault_handler+0xaf/frame 0xffc08fe4 --- trap 0x17, eip = 0xffc030a0, esp = 0xffc07004, ebp = 0xfa6e8edc --- exit_tag() at 0xffc030a0/frame 0xfa6e8edc KDB: enter: panic [ thread pid 86123 tid 104397 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> run pho db:0:pho> set $lines 20000 db:0:pho> run pho1 db:1:pho1> bt Tracing pid 86123 tid 104397 td 0x35425700 kdb_enter(1827b35,1827b35,0,ffc08ff0,e,...) at kdb_enter+0x3a/frame 0xffc08f98 vpanic(17f66a0,ffc08ff0,ffc08ff0,ffc08ff8,16febef,...) at vpanic+0x15e/frame 0xffc08fd0 panic(17f66a0,0,0,0,0,...) at panic+0x1b/frame 0xffc08fe4 dblfault_handler() at dblfault_handler+0xaf/frame 0xffc08fe4 --- trap 0x17, eip = 0xffc030a0, esp = 0xffc07004, ebp = 0xfa6e8edc --- exit_tag() at 0xffc030a0/frame 0xfa6e8edc db:1:pho1> show allpcpu Current CPU: 0 cpuid = 0 dynamic pcpu = 0x8d6f80 curthread = 0x35425700: pid 86123 tid 104397 "calls" curpcb = 0x397b2d40 fpcurthread = 0x35425700: pid 86123 "calls" idlethread = 0x129d0700: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 trampstk = 0xffc07ff0 kesp0 = 0x397b2d30 common_tssp = 0xffc01260 tlb gen = 63971504 curvnet = 0 spin locks held: cpuid = 1 dynamic pcpu = 0x10a38f80 curthread = 0x35758a80: pid 86123 tid 104517 "calls" curpcb = 0x3a23bd40 fpcurthread = 0x35758a80: pid 86123 "calls" idlethread = 0x129d0380: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 trampstk = 0xffc0aff0 kesp0 = 0x3a23bd30 common_tssp = 0xffc012c8 tlb gen = 63971505 curvnet = 0 spin locks held: cpuid = 2 dynamic pcpu = 0x10a3df80 curthread = 0x34549380: pid 86123 tid 104516 "calls" curpcb = 0x352ead40 fpcurthread = 0x34549380: pid 86123 "calls" idlethread = 0x129d0000: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 trampstk = 0xffc0bff0 kesp0 = 0x352ead30 common_tssp = 0xffc01330 tlb gen = 63971505 curvnet = 0 spin locks held: cpuid = 3 dynamic pcpu = 0x10a42f80 curthread = 0x382db700: pid 86129 tid 104084 "sleep" curpcb = 0x4b7e3d40 fpcurthread = 0x382db700: pid 86129 "sleep" idlethread = 0x129cfa80: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 trampstk = 0xffc0dff0 kesp0 = 0x4b7e3d30 common_tssp = 0xffc01398 tlb gen = 63971501 curvnet = 0 spin locks held: db:1:pho1> show alllocks db:1:pho1> show lockedvnods Locked vnodes vnode 0x33f1d9f4: tag ufs, type VDIR usecount 2, writecount 0, refcount 4 mountedhere 0 flags (VI_ACTIVE) v_object 0x547d9c30 ref 0 pages 1 cleanbuf 1 dirtybuf 0 lock type ufs: SHARED (count 1) #0 0x10c72d7 at lockmgr_slock_hard+0x2f7 #1 0x10c7958 at __lockmgr_args+0x148 #2 0x1438414 at ffs_lock+0x94 #3 0x1735269 at VOP_LOCK1_APV+0x69 #4 0x11e3c31 at _vn_lock+0x61 #5 0x11e08bd at kern_getdirentries+0x16d #6 0x11e0be6 at sys_getdirentries+0x46 #7 0x16ff051 at syscall+0x361 #8 0xffc033a7 at exit_tag+0xc4bd7377 ino 1323059, on dev ada0s1d db:1:pho1> show allchains chain 1: thread 104241 (pid 86123, syscall4) sleeping on 0x87369880 "uwait" chain 2: thread 100453 (pid 86123, test) sleeping on 0x20b39a7 "nanslp" chain 3: thread 100675 (pid 86117, syscall4) sleeping on 0x7ed8ca08 "wait" chain 4: thread 103282 (pid 86105, md10) sleeping on 0x70f8d000 "mdwait" chain 5: thread 104512 (pid 86076, sleep) sleeping on 0x20b39a7 "nanslp" chain 6: thread 102464 (pid 85968, sh) sleeping on 0x3829e6b0 "wait" chain 7: thread 104187 (pid 71930, cat) sleeping on 0x48e30000 "piperd" chain 8: thread 104716 (pid 71927, sh) sleeping on 0x36712a08 "wait" chain 9: thread 102971 (pid 71926, find) sleeping on 0x126ece14 "biord" chain 10: thread 104535 (pid 71916, sh) sleeping on 0x39832a08 "wait" chain 11: thread 100264 (pid 71915, mail) sleeping on 0x5b3029f0 "piperd" chain 12: thread 100335 (pid 71914, sh) sleeping on 0x34198a08 "wait" chain 13: thread 100485 (pid 71891, sh) sleeping on 0x3411b358 "wait" chain 14: thread 102629 (pid 71889, lockf) sleeping on 0x38a66000 "wait" chain 15: thread 102375 (pid 71887, sh) sleeping on 0x36d106b0 "wait" chain 16: thread 104982 (pid 71883, sh) blocked on lockmgr XR2z4SHARED (count 1581883) chain 17: thread 104259 (pid 71581, mail) sleeping on 0x9ca714f8 "piperd" chain 18: thread 104051 (pid 71580, sh) sleeping on 0x32529358 "wait" chain 19: thread 104861 (pid 71563, sh) sleeping on 0x38a65a08 "wait" chain 20: thread 102745 (pid 71562, lockf) sleeping on 0x399fb6b0 "wait" chain 21: thread 103955 (pid 71558, sh) sleeping on 0x38a3d358 "wait" chain 22: thread 103217 (pid 71556, cron) sleeping on 0x9fea49f0 "piperd" chain 23: thread 104179 (pid 70944, sh) sleeping on 0x39075a08 "wait" chain 24: thread 104112 (pid 70943, tee) sleeping on 0x2fc6f6a0 "piperd" chain 25: thread 100770 (pid 70942, sh) sleeping on 0x35fe3000 "wait" chain 26: thread 101206 (pid 62753, aiod4) sleeping on 0x35fe1358 "aiordy" chain 27: thread 104144 (pid 62752, aiod3) sleeping on 0x311a7000 "aiordy" chain 28: thread 103088 (pid 62751, aiod2) sleeping on 0x391ad000 "aiordy" chain 29: thread 105129 (pid 62750, aiod1) sleeping on 0x44a4c6b0 "aiordy" chain 30: thread 100797 (pid 94606, bash) sleeping on 0x375ee424 "select" chain 31: thread 100654 (pid 94605, sshd) sleeping on 0x36c28b24 "select" chain 32: thread 103468 (pid 94603, sshd) sleeping on 0x3a97daa4 "select" chain 33: thread 100194 (pid 934, sh) sleeping on 0x2fc2f000 "piperd" chain 34: thread 100209 (pid 933, awk) sleeping on 0x2fc6ed40 "piperd" chain 35: thread 100200 (pid 931, sh) sleeping on 0x311a76b0 "wait" chain 36: thread 100208 (pid 923, top) sleeping on 0x271ebaa4 "select" chain 37: thread 100206 (pid 922, tail) sleeping on 0x2708dd00 "kqread" chain 38: thread 100207 (pid 918, sh) sleeping on 0x311a66b0 "wait" chain 39: thread 100169 (pid 914, sshd) sleeping on 0x27060b64 "select" chain 40: thread 100205 (pid 911, sshd) sleeping on 0x271ebc64 "select" chain 41: thread 100170 (pid 910, sshd) sleeping on 0x270607e4 "select" chain 42: thread 100199 (pid 898, sshd) sleeping on 0x2677e4a4 "select" chain 43: thread 100203 (pid 897, sshd) sleeping on 0x26ddae24 "select" chain 44: thread 100123 (pid 895, sshd) sleeping on 0x26ddada4 "select" chain 45: thread 100177 (pid 836, sh) sleeping on 0x26dd7a08 "wait" chain 46: thread 100114 (pid 799, bash) blocked on lockmgr SHARED (count 1581883) chain 47: thread 100127 (pid 797, csh) sleeping on 0x26dcd714 "pause" chain 48: thread 100176 (pid 796, su) blocked on lockmgr X<'SHARED (count 1581883) chain 49: thread 100193 (pid 774, bash) blocked on lockmgr X01='='3SHARED (count 1581883) chain 50: thread 100121 (pid 773, sshd) sleeping on 0x270610e4 "select" chain 51: thread 100173 (pid 771, sshd) sleeping on 0x270613a4 "select" chain 52: thread 100192 (pid 769, getty) sleeping on 0x220f6c74 "ttyin" chain 53: thread 100191 (pid 768, getty) sleeping on 0x12990474 "ttyin" chain 54: thread 100190 (pid 767, getty) sleeping on 0x12990874 "ttyin" chain 55: thread 100189 (pid 766, getty) sleeping on 0x12990c74 "ttyin" chain 56: thread 100188 (pid 765, getty) sleeping on 0x12991074 "ttyin" chain 57: thread 100187 (pid 764, getty) sleeping on 0x12991474 "ttyin" chain 58: thread 100178 (pid 763, getty) sleeping on 0x12991874 "ttyin" chain 59: thread 100179 (pid 762, getty) sleeping on 0x2709b474 "ttyin" chain 60: thread 100105 (pid 761, getty) sleeping on 0x12991c74 "ttyin" chain 61: thread 100106 (pid 744, inetd) sleeping on 0x27060b24 "select" chain 62: thread 100174 (pid 710, nfscbd: master) sleeping on 0x2705b350 "rpcsvc" chain 63: thread 100180 (pid 710, nfscbd: service) sleeping on 0x2705b2d0 "rpcsvc" chain 64: thread 100181 (pid 710, nfscbd: service) sleeping on 0x2705b1d0 "rpcsvc" chain 65: thread 100182 (pid 710, nfscbd: service) sleeping on 0x2705b250 "rpcsvc" chain 66: thread 100167 (pid 709, nfscbd) sleeping on 0x303510c8 "accept" chain 67: thread 100172 (pid 676, cron) sleeping on 0x20b39a7 "nanslp" chain 68: thread 100124 (pid 672, sendmail) sleeping on 0x273cd714 "pause" chain 69: thread 100111 (pid 669, sendmail) sleeping on 0x27060c64 "select" chain 70: thread 100171 (pid 666, sshd) sleeping on 0x27060d64 "select" chain 71: thread 100166 (pid 626, smartd) sleeping on 0x20b39a5 "nanslp" chain 72: thread 100108 (pid 602, powerd) sleeping on 0x26ddb064 "select" chain 73: thread 100107 (pid 599, ntpd) sleeping on 0x2677e864 "select" chain 74: thread 100113 (pid 561, NLM: master) sleeping on 0x2fd90cd0 "rpcsvc" chain 75: thread 100119 (pid 558, rpc.statd) sleeping on 0x2677eb24 "select" chain 76: thread 100134 (pid 557, nfsd: master) sleeping on 0x2705b6d0 "rpcsvc" chain 77: thread 100135 (pid 557, nfsd: service) sleeping on 0x129e6750 "rpcsvc" chain 78: thread 100136 (pid 557, nfsd: service) sleeping on 0x129e6450 "rpcsvc" chain 79: thread 100137 (pid 557, nfsd: service) sleeping on 0x129e67d0 "rpcsvc" chain 80: thread 100138 (pid 557, nfsd: service) sleeping on 0x2fd913d0 "rpcsvc" chain 81: thread 100139 (pid 557, nfsd: service) sleeping on 0x129e6850 "rpcsvc" chain 82: thread 100140 (pid 557, nfsd: service) sleeping on 0x2fd91350 "rpcsvc" chain 83: thread 100141 (pid 557, nfsd: service) sleeping on 0x2fd912d0 "rpcsvc" chain 84: thread 100142 (pid 557, nfsd: service) sleeping on 0x129e68d0 "rpcsvc" chain 85: thread 100143 (pid 557, nfsd: service) sleeping on 0x129e6950 "rpcsvc" chain 86: thread 100144 (pid 557, nfsd: service) sleeping on 0x2fd91250 "rpcsvc" chain 87: thread 100145 (pid 557, nfsd: service) sleeping on 0x129e69d0 "rpcsvc" chain 88: thread 100146 (pid 557, nfsd: service) sleeping on 0x2fd911d0 "rpcsvc" chain 89: thread 100147 (pid 557, nfsd: service) sleeping on 0x129e6a50 "rpcsvc" chain 90: thread 100148 (pid 557, nfsd: service) sleeping on 0x2fd91150 "rpcsvc" chain 91: thread 100149 (pid 557, nfsd: service) sleeping on 0x129e6ad0 "rpcsvc" chain 92: thread 100150 (pid 557, nfsd: service) sleeping on 0x2fd910d0 "rpcsvc" chain 93: thread 100151 (pid 557, nfsd: service) sleeping on 0x129e6b50 "rpcsvc" chain 94: thread 100152 (pid 557, nfsd: service) sleeping on 0x129e6bd0 "rpcsvc" chain 95: thread 100153 (pid 557, nfsd: service) sleeping on 0x2fd91050 "rpcsvc" chain 96: thread 100154 (pid 557, nfsd: service) sleeping on 0x129e6c50 "rpcsvc" chain 97: thread 100155 (pid 557, nfsd: service) sleeping on 0x2fd90f50 "rpcsvc" chain 98: thread 100156 (pid 557, nfsd: service) sleeping on 0x129e6cd0 "rpcsvc" chain 99: thread 100157 (pid 557, nfsd: service) sleeping on 0x2fd90ed0 "rpcsvc" chain 100: thread 100158 (pid 557, nfsd: service) sleeping on 0x129e6d50 "rpcsvc" chain 101: thread 100159 (pid 557, nfsd: service) sleeping on 0x2fd90e50 "rpcsvc" chain 102: thread 100160 (pid 557, nfsd: service) sleeping on 0x129e6dd0 "rpcsvc" chain 103: thread 100161 (pid 557, nfsd: service) sleeping on 0x2fd90dd0 "rpcsvc" chain 104: thread 100162 (pid 557, nfsd: service) sleeping on 0x129e6e50 "rpcsvc" chain 105: thread 100163 (pid 557, nfsd: service) sleeping on 0x2fd90d50 "rpcsvc" chain 106: thread 100164 (pid 557, nfsd: service) sleeping on 0x129e6ed0 "rpcsvc" chain 107: thread 100165 (pid 557, nfsd: service) sleeping on 0x129e6f50 "rpcsvc" chain 108: thread 100129 (pid 554, nfsd) sleeping on 0x270610a4 "select" chain 109: thread 100125 (pid 548, mountd) sleeping on 0x271ec1a4 "select" chain 110: thread 100133 (pid 542, nfsuserd) sleeping on 0x27060ee4 "select" chain 111: thread 100132 (pid 540, nfsuserd) sleeping on 0x271ec164 "select" chain 112: thread 100131 (pid 539, nfsuserd) sleeping on 0x27060f24 "select" chain 113: thread 100130 (pid 538, nfsuserd) sleeping on 0x271ec224 "select" chain 114: thread 100112 (pid 537, nfsuserd) sleeping on 0x26298714 "pause" chain 115: thread 100110 (pid 494, watchdogd) sleeping on 0x20b39a6 "nanslp" chain 116: thread 100109 (pid 405, rpcbind) sleeping on 0x271ec324 "select" chain 117: thread 100120 (pid 396, syslogd) sleeping on 0x271ec364 "select" chain 118: thread 100122 (pid 324, devd) sleeping on 0x27061424 "select" chain 119: thread 100102 (pid 25, syncer) sleeping on 0x212e124 "syncer" chain 120: thread 100100 (pid 24, vnlru) blocked on lockmgr X3&*&'3SHARED (count 1581883) chain 121: thread 100099 (pid 23, bufdaemon) sleeping on 0x212dac4 "qsleep" chain 122: thread 100101 (pid 23, bufspacedaemon-0) sleeping on 0x1de1000 "-" chain 123: thread 100115 (pid 23, /home worker) sleeping on 0x2fdb9884 "sdflush" chain 124: thread 100116 (pid 23, /tmp worker) sleeping on 0x2fdb9684 "sdflush" chain 125: thread 100117 (pid 23, /usr worker) sleeping on 0x2fdf9284 "sdflush" chain 126: thread 100118 (pid 23, /var worker) sleeping on 0x2fdf9084 "sdflush" chain 127: thread 100227 (pid 23, /mnt worker) sleeping on 0x7bfefc84 "sdflush" chain 128: thread 100098 (pid 22, vmdaemon) sleeping on 0x2143678 "psleep" chain 129: thread 100097 (pid 21, dom0) sleeping on 0x2177cb8 "psleep" chain 130: thread 100103 (pid 21, laundry: dom0) sleeping on 0x2177cc4 "launds" chain 131: thread 100104 (pid 21, uma) sleeping on 0x1457510 "umarcl" chain 132: thread 100094 (pid 20, rand_harvestq) sleeping on 0x1f18f3c "-" chain 133: thread 100090 (pid 19, sctp_iterator) sleeping on 0x2176d84 "waiting_for_work" chain 134: thread 100089 (pid 18, soaiod4) sleeping on 0x212d6e8 "-" chain 135: thread 100088 (pid 17, soaiod3) sleeping on 0x212d6e8 "-" chain 136: thread 100087 (pid 16, soaiod2) sleeping on 0x212d6e8 "-" chain 137: thread 100086 (pid 9, soaiod1) sleeping on 0x212d6e8 "-" chain 138: thread 100083 (pid 8, fdc0) sleeping on 0x220e723c "-" chain 139: thread 100048 (pid 15, usbus0) sleeping on 0x22109acc "-" chain 140: thread 100049 (pid 15, usbus0) sleeping on 0x22109afc "-" chain 141: thread 100050 (pid 15, usbus0) sleeping on 0x22109b2c "-" chain 142: thread 100051 (pid 15, usbus0) sleeping on 0x22109b5c "-" chain 143: thread 100052 (pid 15, usbus0) sleeping on 0x22109b8c "-" chain 144: thread 100054 (pid 15, usbus1) sleeping on 0x2629eacc "-" chain 145: thread 100055 (pid 15, usbus1) sleeping on 0x2629eafc "-" chain 146: thread 100056 (pid 15, usbus1) sleeping on 0x2629eb2c "-" chain 147: thread 100057 (pid 15, usbus1) sleeping on 0x2629eb5c "-" chain 148: thread 100058 (pid 15, usbus1) sleeping on 0x2629eb8c "-" chain 149: thread 100060 (pid 15, usbus2) sleeping on 0x263daacc "-" chain 150: thread 100061 (pid 15, usbus2) sleeping on 0x263daafc "-" chain 151: thread 100062 (pid 15, usbus2) sleeping on 0x263dab2c "-" chain 152: thread 100063 (pid 15, usbus2) sleeping on 0x263dab5c "-" chain 153: thread 100064 (pid 15, usbus2) sleeping on 0x263dab8c "-" chain 154: thread 100065 (pid 15, usbus3) sleeping on 0x262c0acc "-" chain 155: thread 100066 (pid 15, usbus3) sleeping on 0x262c0afc "-" chain 156: thread 100067 (pid 15, usbus3) sleeping on 0x262c0b2c "-" chain 157: thread 100068 (pid 15, usbus3) sleeping on 0x262c0b5c "-" chain 158: thread 100069 (pid 15, usbus3) sleeping on 0x262c0b8c "-" chain 159: thread 100070 (pid 15, usbus4) sleeping on 0x2630bacc "-" chain 160: thread 100071 (pid 15, usbus4) sleeping on 0x2630bafc "-" chain 161: thread 100072 (pid 15, usbus4) sleeping on 0x2630bb2c "-" chain 162: thread 100073 (pid 15, usbus4) sleeping on 0x2630bb5c "-" chain 163: thread 100074 (pid 15, usbus4) sleeping on 0x2630bb8c "-" chain 164: thread 100076 (pid 15, usbus5) sleeping on 0x2210eca4 "-" chain 165: thread 100077 (pid 15, usbus5) sleeping on 0x2210ecd4 "-" chain 166: thread 100078 (pid 15, usbus5) sleeping on 0x2210ed04 "-" chain 167: thread 100079 (pid 15, usbus5) sleeping on 0x2210ed34 "-" chain 168: thread 100080 (pid 15, usbus5) sleeping on 0x2210ed64 "-" chain 169: thread 100043 (pid 7, doneq0) sleeping on 0x1e52340 "-" chain 170: thread 100096 (pid 7, scanner) sleeping on 0x1e5226c "-" chain 171: thread 100042 (pid 14, sequencer 00) sleeping on 0x125ca84c "seqstate" chain 172: thread 100041 (pid 6, crypto returns 3) sleeping on 0x125be0c4 "crypto_ret_wait" chain 173: thread 100040 (pid 5, crypto returns 2) sleeping on 0x125be08c "crypto_ret_wait" chain 174: thread 100039 (pid 4, crypto returns 1) sleeping on 0x125be054 "crypto_ret_wait" chain 175: thread 100038 (pid 3, crypto returns 0) sleeping on 0x125be01c "crypto_ret_wait" chain 176: thread 100037 (pid 2, crypto) sleeping on 0x2142438 "crypto_wait" chain 177: thread 100029 (pid 13, g_event) sleeping on 0x2166d3c "-" chain 178: thread 100030 (pid 13, g_up) sleeping on 0x2166d44 "-" chain 179: thread 100031 (pid 13, g_down) sleeping on 0x2166d38 "-" chain 180: thread 100002 (pid 1, init) sleeping on 0x129cca08 "wait" chain 181: thread 100001 (pid 10, audit) sleeping on 0x2177564 "audit_worker_cv" chain 182: thread 100000 (pid 0, swapper) sleeping on 0x2166d5c "swapin" chain 183: thread 100007 (pid 0, aiod_kick taskq) sleeping on 0x129d4e00 "-" chain 184: thread 100010 (pid 0, thread taskq) sleeping on 0x129d4c80 "-" chain 185: thread 100012 (pid 0, config_0) sleeping on 0x129e9a00 "-" chain 186: thread 100013 (pid 0, kqueue_ctx taskq) sleeping on 0x129d4b80 "-" chain 187: thread 100014 (pid 0, if_config_tqg_0) sleeping on 0x129e9980 "-" chain 188: thread 100015 (pid 0, softirq_0) sleeping on 0x129e9900 "-" chain 189: thread 100016 (pid 0, softirq_1) sleeping on 0x129e9880 "-" chain 190: thread 100017 (pid 0, softirq_2) sleeping on 0x129e9800 "-" chain 191: thread 100018 (pid 0, softirq_3) sleeping on 0x129e9780 "-" chain 192: thread 100019 (pid 0, if_io_tqg_0) sleeping on 0x129e9700 "-" chain 193: thread 100020 (pid 0, if_io_tqg_1) sleeping on 0x129e9680 "-" chain 194: thread 100021 (pid 0, if_io_tqg_2) sleeping on 0x129e9600 "-" chain 195: thread 100022 (pid 0, if_io_tqg_3) sleeping on 0x129e9580 "-" chain 196: thread 100032 (pid 0, firmware taskq) sleeping on 0x129d3a80 "-" chain 197: thread 100033 (pid 0, crypto_0) sleeping on 0x129d3980 "-" chain 198: thread 100034 (pid 0, crypto_1) sleeping on 0x129d3980 "-" chain 199: thread 100035 (pid 0, crypto_2) sleeping on 0x129d3980 "-" chain 200: thread 100036 (pid 0, crypto_3) sleeping on 0x129d3980 "-" chain 201: thread 100085 (pid 0, mca taskq) blocked on lockmgr (null)EXCL KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(185c2e5,1dbdea0,8,1765c08,248,...) at db_trace_self_wrapper+0x2a/frame 0xffc08898 kdb_backtrace(175c47c,a,ffc08a18,16fdb70,a19f80,...) at kdb_backtrace+0x2d/frame 0xffc08900 kdb_reenter(a19f80,ffc0893c,a1a26d,a,29,...) at kdb_reenter+0x2a/frame 0xffc08910 trap(ffc08a24,8,28,28,22100e40,...) at trap+0xe0/frame 0xffc08a18 calltrap() at 0xffc0315d/frame 0xffc08a18 --- trap 0xc, eip = 0x115a4d6, esp = 0xffc08a64, ebp = 0xffc08a88 --- print_lockchain(1781ff5,c9,1ddb070,0,0,...) at print_lockchain+0x26/frame 0xffc08a88 db_show_allchains(1140bea,0,ffffffff,ffc08ad0,1e53f00,...) at db_show_allchains+0x78/frame 0xffc08aa4 db_command(1e54da7,1,1e54880,1e54da6,a19f80,...) at db_command+0x297/frame 0xffc08b58 db_script_exec(0,0,fffffff7,ffc08bf0,1e53f04,...) at db_script_exec+0x168/frame 0xffc08bc4 db_command(1e54d06,0,1e54920,1e54d05,ffc08ca8,...) at db_command+0x297/frame 0xffc08c78 db_script_exec(2144ea4,0,ffc08d20,ffc08d14,1e53f04,...) at db_script_exec+0x168/frame 0xffc08ce8 db_command(180afc9,ffc08e00,a19e3d,1827b35,0,...) at db_command+0x297/frame 0xffc08d9c db_command_loop(1827b35,0,ffc08dc7,ffc08dc6,1dbdcf0,...) at db_command_loop+0x50/frame 0xffc08da8 db_trap(3,0,0,0,0,...) at db_trap+0x10d/frame 0xffc08e00 kdb_trap(3,0,ffc08f4c,ff208f01,ffc08ea4,...) at kdb_trap+0x144/frame 0xffc08e38 trap(ffc08f4c,8,28,28,100,...) at trap+0x3cc/frame 0xffc08f40 calltrap() at 0xffc0315d/frame 0xffc08f40 --- trap 0x3, eip = 0x1140bea, esp = 0xffc08f8c, ebp = 0xffc08f98 --- kdb_enter(1827b35,1827b35,0,ffc08ff0,e,...) at kdb_enter+0x3a/frame 0xffc08f98 vpanic(17f66a0,ffc08ff0,ffc08ff0,ffc08ff8,16febef,...) at vpanic+0x15e/frame 0xffc08fd0 panic(17f66a0,0,0,0,0,...) at panic+0x1b/frame 0xffc08fe4 dblfault_handler() at dblfault_handler+0xaf/frame 0xffc08fe4 --- trap 0x17, eip = 0xffc030a0, esp = 0xffc07004, ebp = 0xfa6e8edc --- exit_tag() at 0xffc030a0/frame 0xfa6e8edc Script command 'show allchains' returned error db:1:pho1> show mount 0x27398000 /dev/ada0s1a on / (ufs) 0x271f3000 devfs on /dev (devfs) 0x27397000 /dev/ada0s1f on /home (ufs) 0x27396000 /dev/label/tmp on /tmp (ufs) 0x271f1000 /dev/ada0s1d on /usr (ufs) 0x271f0000 /dev/ada0s1e on /var (ufs) 0x301ae000 procfs on /proc (procfs) 0x3d7fe000 /dev/md10a on /mnt (ufs) More info: show mount db:1:pho1> show bufqueues bqempty: 0 Buf domain 0 freebufs 462 lofreebufs 311 hifreebufs 466 bufspace 28018688 maxbufspace 94683136 hibufspace 94027776 lobufspace 89326372 bufspacethresh 91677074 numdirtybuffers 73 lodirtybuffers 732 hidirtybuffers 1464 dirtybufthresh 1317 cleanq count 5222 (26583040) dirtyq count 73 (1214464) wakeup 0 lim 28 CPU 10, 4, 7, 0, Locked buffers: 1 space 2048 Total buffers: 5779 space 28018688 db:1:pho1> show page vm_cnt.v_free_count: 1440355 vm_cnt.v_inactive_count: 202242 vm_cnt.v_active_count: 4734 vm_cnt.v_laundry_count: 8553 Loading /boot/loader.conf.local -- Test scenario: 20190223 03:00:22 all: syscall5.sh 03:00:26 syscall 567 (fhreadlink) 03:02:42 syscall 566 (fhlinkat) 03:04:46 syscall 565 (fhlink) 03:06:50 syscall 564 (getfhat) 03:08:59 syscall 563 (getrandom)