GENERIC HEAD from 2020-08-01 09:40:19 UTC, r363759, vmcore.871 GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb ---<>--- Copyright (c) 1992-2020 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 13.0-CURRENT #0 r363759: Sat Aug 1 12:34:53 CEST 2020 pho@t2.osted.lan:/usr/src/sys/amd64/compile/PHO amd64 FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.0-97-g6f71678ecd2) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. VT(vga): resolution 640x480 CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (1995.23-MHz K8-class CPU) Origin="GenuineIntel" Id=0x206d7 Family=0x6 Model=0x2d Stepping=7 Features=0xbfebfbff Features2=0x1fbee3ff AMD Features=0x2c100800 AMD Features2=0x1 XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 68719476736 (65536 MB) avail memory = 66705383424 (63615 MB) : Trying to mount root from ufs:/dev/da0p2 [rw]... cd0 at ahcich1 bus 0 scbus2 target 0 lun 0 cd0: Removable CD-ROM SCSI device cd0: Serial Number R93E68ED300PN5 cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed Expensive timeout(9) function: 0xffffffff80a2faa0(0xffffffff818e6898) 0.006709949 s ada0 at ahcich0 bus 0 scbus1 target 0 lun 0 ada0: ACS-2 ATA SATA 3.x device ada0: Serial Number CVCV317607P0240CGN ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 228936MB (468862128 512 byte sectors) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. ses0: ada0 in 'Slot 00', SATA Slot: scbus1 target 0 uma_zalloc_debug: zone "kenv" with the following non-sleepable locks held: ses0: cd0 in 'Slot 01', SATA Slot: scbus2 target 0 exclusive sleep mutex CAM device lock (CAM device lock) r = 0 (0xfffff800089184d0) locked @ cam/cam_xpt.c:2337 stack backtrace: Root mount waiting for: CAM#0 0xffffffff80c41081 at witnes usbus0 usbus1 usbus2 ses s_debugger+0x71 #1 0xffffffff80c4201d at witness_warn+0x40d #2 0xffffffff80ea7696 at uma_zalloc_arg+0x46 #3 0xffffffff80b80b7a at getenv_string_buffer+0x3a #4 0xffffffff80b812a7 at getenv_quad+0x17 #5 0xffffffff80b81272 at getenv_int+0x12 #6 0xffffffff803a3f1a at daregister+0x1ea #7 0xffffffff8037045b at cam_periph_alloc+0x57b #8 0xffffffff803a3872 at daasync+0x2c2 #9 0xffffffff8037abaa at xptsetasyncfunc+0x11a #10 0xffffffff8037c955 at xptdevicetraverse+0xa5 #11 0xffffffff8037c6db at xpttargettraverse+0x6b #12 0xffffffff8037c21f at xptbustraverse+0x6f #13 0xffffffff8037aa1e at xpt_register_async+0x1fe #14 0xffffffff803a3559 at dainit+0x19 #15 0xffffffff8036feb3 at periphdriver_init+0x43 #16 0xffffffff8037a7e2 at xpt_finishconfig_task+0x12 #17 0xffffffff80c3322a at taskqueue_run_locked+0xaa da0 at isci0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SPC-3 SCSI device da0: Serial Number 6SL64N800000N339035N da0: 300.000MB/s transfers da0: Command Queueing enabled da0: 572325MB (1172123568 512 byte sectors) uhub1: 4 ports with 4 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub0: 2 ports with 2 removable, self powered Root mount waiting for: usbus0 usbus2 ugen0.2: at usbus0 uhub3 numa-domain 0 on uhub2 uhub3: on usbus0 ugen2.2: at usbus2 uhub4 numa-domain 0 on uhub0 uhub4: on usbus2 Root mount waiting for: usbus0 usbus2 uhub3: 6 ports with 6 removable, self powered uhub4: 8 ports with 8 removable, self powered Root mount waiting for: usbus2 ugen2.3: at usbus2 uhub5 numa-domain 0 on uhub4 uhub5: on usbus2 uhub5: MTT enabled uhub5: 4 ports with 4 removable, self powered Root mount waiti2 ugen2.4: at usbus2 ukbd0 numa-domain 0 on uhub4 ukbd0: on usbus2 kbd2 at ukbd0 mountroot: waiting for device /dev/da0p2... Setting hostuuid: 2bde2bde-f4e2-e111-aab2-001e6756b69b. Setting hostid: 0x0035ff86. Starting file system checks: /dev/da0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/da0p2: clean, 44124288 free (1189920 frags, 5366796 blocks, 1.2% fragmentation) /dev/da0p3: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/da0p3: clean, 23613881 free (6945 frags, 2950867 blocks, 0.0% fragmentation) Mounting local filesystems:. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/e2fsprogs /usr/local/lib/gcc8 /usr/local/lib/gcc9 /usr/local/lib/mozjpeg /usr/local/lib/nss /usr/local/lib/perl5/5.30/mach/CORE /usr/local/lib/qt4 /usr/local/lib/qt5 32-bit compatibility ldconfig path: /usr/lib32 /usr/local/lib32/compat Setting hostname: t2.osted.lan. Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy: . lo0: link state changed to UP Starting Network: lo0 igb0 igb1. lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21 igb0: flags=8863 metric 0 mtu 1500 options=4e527bb ether 00:1e:67:56:b6:9b inet 192.168.1.109 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::21e:67ff:fe56:b69b%igb0 prefixlen 64 scopeid 0x1 media: Ethernet autoselect (1000baseT ) status: active nd6 options=23 igb1: flags=8822 metric 0 mtu 1500 options=4e527bb ether 00:1e:67:56:b6:9c media: Ethernet autoselect status: no carrier nd6 options=29 Starting devd. Autoloading module: ioat.ko ioat0: mem 0xd7fa0000-0xd7fa3fff irq 31 at device 4.0 numa-domain 0 on pci0 ioat0: Capabilities: 2f7 ioat1: mem 0xd7f90000-0xd7f93fff irq 39 at device 4.1 numa-domain 0 on pci0 ioat1: Capabilities: 2f7 ioat2: mem 0xd7f80000-0xd7f83fff irq 31 at device 4.2 numa-domain 0 on pci0 ioat2: Capabilities: f7 ioat3: mem 0xd7f70000-0xd7f73fff irq 39 at device 4.3 numa-domain 0 on pci0 ioat3: Capabilities: f7 ioat4: mem 0xd7f60000-0xd7f63fff irq 31 at device 4.4 numa-domain 0 on pci0 ioat4: Capabilities: f7 ioat5: mem 0xd7f50000-0xd7f53fff irq 39 at device 4.5 numa-domain 0 on pci0 ioat5: Capabilities: f7 ioat6: mem 0xd7f40000-0xd7f43fff irq 31 at device 4.6 numa-domain 0 on pci0 ioat6: Capabilities: f7 ioat7: mem 0xd7f30000-0xd7f33fff irq 39 at device 4.7 numa-domain 0 on pci0 ioat7: Capabilities: f7 ioat8: mem 0xfbf70000-0xfbf73fff irq 55 at device 4.0 numa-domain 1 on pci11 ioat8: Capabilities: 2f7 ioat9: mem 0xfbf60000-0xfbf63fff irq 63 at device 4.1 numa-domain 1 on pci11 ioat9: Capabilities: 2f7 ioat10: mem 0xfbf50000-0xfbf53fff irq 55 at device 4.2 numa-domain 1 on pci11 ioat10: Capabilities: f7 ioat11: mem 0xfbf40000-0xfbf43fff irq 63 at device 4.3 numa-domain 1 on pci11 ioat11: Capabilities: f7 ioat12: mem 0xfbf30000-0xfbf33fff irq 55 at device 4.4 numa-domain 1 on pci11 ioat12: Capabilities: f7 ioat13: mem 0xfbf20000-0xfbf23fff irq 63 at device 4.5 numa-domain 1 on pci11 ioat13: Capabilities: f7 ioat14: mem 0xfbf10000-0xfbf13fff irq 55 at device 4.6 numa-domain 1 on pci11 ioat14: Capabilities: f7 ioat15: mem 0xfbf00000-0xfbf03fff irq 63 at device 4.7 numa-domain 1 on pci11 ioat15: Capabilities: f7 Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Starting Network: igb1. igb1: flags=8822 metric 0 mtu 1500 options=4e527bb ether 00:1e:67:56:b6:9c media: Ethernet autoselect status: no carrier nd6 options=29 Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Configuring vt: keymap. Autoloading module: uhid.ko Autoloading module: ums.ko ums0 numa-domain 0 on uhub4 ums0: on usbus2 ums0: 3 buttons and [Z] coordinates ID=0 Starting ums0 moused. add host 127.0.0.1: gateway lo0 fib 0: route already in table add net default: gateway 192.168.1.254 add host ::1: gateway lo0 fib 0: route already in table add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Starting rtsold. Creating and/or trimming log files. Starting syslogd. Updating CPU Microcode... CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (1995.23-MHz K8-class CPU) Origin="GenuineIntel" Id=0x206d7 Family=0x6 Model=0x2d Stepping=7 Features=0xbfebfbff Features2=0x1fbee3ff AMD Features=0x2c100800 AMD Features2=0x1 Structured Extended Features3=0x9c000400 XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics Done. No core dumps found. Starting rpcbind. NFS access cache time=60 Clearing /tmp (X related). Starting nfsuserd. Starting mountd. Starting nfsd. Starting statd. Starting lockd. Recovering vi editor sessions:. Updating motd:. Mounting late filesystems:. Security policy loaded: MAC/ntpd (mac_ntpd) Starting ntpd. Starting powerd. Starting sendmail_submit. Starting sendmail_msp_queue. Performing sanity check on sshd configuration. Starting sshd. Configuring vt: keymap blanktime. Starting cron. Local package initialization: backup. Starting nfscbd. Starting default mousedmoused: unable to open /dev/psm0: No such file or directory . Starting inetd. Sat Aug 1 12:41 FreeBSD/amd64 (t2.osted.lan) (ttyu1) login: Aug 1 12:43:03 t2 su[2773]: pho to root on /dev/pts/0 20200801 12:43:26 all (1/726): 256m.sh 20200801 12:43:30 all (2/726): fcntl3.sh freework leak: 1/6. 20200801 12:43:38 all (3/726): fexecve.sh 20200801 12:43:42 all (4/726): fsck2.sh 20200801 12:43:46 all (5/726): holdcnt04.sh 20200801 12:43:50 all (6/726): ldt2.sh 20200801 12:43:54 all (7/726): linux.sh 20200801 12:43:58 all (8/726): lookup_shared.sh 20200801 12:44:01 all (9/726): mac_chkexec.sh 20200801 12:44:05 all (10/726): mac.sh 20200801 12:44:09 all (11/726): machipc.sh 20200801 12:44:13 all (12/726): machipc2.sh 20200801 12:44:16 all (13/726): md4.sh 20200801 12:44:20 all (14/726): mmap16.sh 20200801 12:44:24 all (15/726): nfs14.sh 20200801 12:44:28 all (16/726): numa.sh 20200801 12:44:31 all (17/726): oom.sh 20200801 12:44:35 all (18/726): pagefault.sh 20200801 12:44:39 all (19/726): parallelmount2.sh 20200801 12:44:43 all (20/726): pathconf2.sh 20200801 12:44:46 all (21/726): pcatch.sh 20200801 12:44:50 all (22/726): pcatch2.sh 20200801 12:44:54 all (23/726): pkru.sh 20200801 12:44:58 all (24/726): pkru2.sh 20200801 12:45:01 all (25/726): procfs6.sh 20200801 12:45:05 all (26/726): procstat.sh 20200801 12:45:09 all (27/726): quota5.sh 20200801 12:45:12 all (28/726): ruby.sh 20200801 12:45:16 all (29/726): schedfuzz.sh 20200801 12:45:20 all (30/726): segregs.sh 20200801 12:45:24 all (31/726): select3.sh 20200801 12:45:28 all (32/726): shm_super.sh 20200801 12:45:32 all (33/726): sigaltstack.sh 20200801 12:45:36 all (34/726): trim6.sh 20200801 12:45:40 all (35/726): trim7.sh 20200801 12:45:43 all (36/726): trim8.sh 20200801 12:45:47 all (37/726): random.sh 20200801 12:45:52 all (38/726): smrstress.sh 20200801 12:45:57 all (39/726): 1st.sh 20200801 12:46:03 all (40/726): audit.sh 20200801 12:46:09 all (41/726): beneath2.sh lock order reversal: 1st 0xfffff8012acce800 ufs (ufs) @ kern/vfs_mount.c:1696 2nd 0xfffff80216c4ad10 devfs (devfs) @ kern/vfs_subr.c:3332 stack backtrace: #0 0xffffffff80c41081 at witness_debugger+0x71 #1 0xffffffff80ba47c2 at lockmgr_lock_flags+0x172 #2 0xffffffff810c5670 at VOP_LOCK1_APV+0x40 #3 0xffffffff80cc95b4 at _vn_lock+0x54 #4 0xffffffff80cb25a5 at vput_final+0x135 #5 0xffffffff80e89d1f at ffs_unmount+0x34f #6 0xffffffff80ca959d at dounmount+0x43d #7 0xffffffff80ca9109 at kern_unmount+0x319 #8 0xffffffff81014969 at amd64_syscall+0x159 #9 0xffffffff80fe9dee at fast_syscall_common+0xf8 mount leak: 1/50 20200801 12:46:15 all (42/726): beneath3.sh mount leak: 1/51 20200801 12:46:21 all (43/726): buildworld.sh 20200801 12:46:26 all (44/726): buildworld2.sh 20200801 12:46:32 all (45/726): crossmp2.sh 20200801 13:03:13 all (46/726): crossmp6.sh mount leak: 39/90 20200801 13:08:24 all (47/726): datagram2.sh 20200801 13:08:30 all (48/726): datagram3.sh 20200801 13:08:36 all (49/726): datamove.sh freework leak: 35/41. 20200801 13:08:42 all (50/726): datamove3.sh 20200801 13:08:48 all (51/726): devfs2.sh 20200801 13:08:54 all (52/726): devfs5.sh 20200801 13:09:00 all (53/726): dtrace_fault.sh 20200801 13:09:06 all (54/726): dumpfs.sh newblk leak: 282/312. 20200801 13:09:12 all (55/726): dup2.sh 20200801 13:09:18 all (56/726): extattr_set_fd.sh 20200801 13:09:24 all (57/726): extattrctl.sh 20200801 13:09:30 all (58/726): fifo.sh 20200801 13:09:36 all (59/726): getrandom.sh 20200801 13:09:42 all (60/726): isofs2.sh 20200801 13:09:48 all (61/726): jail.sh 20200801 13:09:54 all (62/726): jail2.sh 20200801 13:10:00 all (63/726): jail3.sh 20200801 13:10:06 all (64/726): jail4.sh 20200801 13:10:12 all (65/726): jexec.sh Aug 1 13:12:00 t2 kernel: pid 45655 (swap), jid 0, uid 0, was killed: out of swap space 20200801 13:12:28 all (66/726): kern_umtx_inf_loop.sh 20200801 13:12:35 all (67/726): kevent10.sh 20200801 13:12:41 all (68/726): ldt.sh kernel trap 9 with interrupts disabled kernel trap 9 with interrupts disabled kernel trap 9 with interrupts disabled kernel trap 9 with interrupts disabled kernel trap 9 with interrupts disabled 20200801 13:12:47 all (69/726): lockd.sh 20200801 13:13:56 all (70/726): lockf.sh 20200801 13:19:08 all (71/726): maxmemdom.sh 20200801 13:19:14 all (72/726): maxproc.sh 20200801 13:19:20 all (73/726): md6.sh 20200801 13:19:26 all (74/726): mdconfig.sh 20200801 13:19:32 all (75/726): memguard4.sh 20200801 13:19:38 all (76/726): mkfifo5.sh 20200801 13:19:44 all (77/726): mkfifo7.sh 20200801 13:19:50 all (78/726): mlockall5.sh 20200801 13:19:56 all (79/726): mmap.sh 20200801 13:20:02 all (80/726): mmap12.sh 20200801 13:20:08 all (81/726): mmap20.sh 20200801 13:20:14 all (82/726): mmap24.sh 20200801 13:20:20 all (83/726): mmap30.sh 20200801 13:20:26 all (84/726): mmap33.sh 20200801 13:20:32 all (85/726): mmap35.sh 20200801 13:20:38 all (86/726): mmap36.sh 20200801 13:20:44 all (87/726): mount.sh 20200801 13:20:50 all (88/726): mprotect.sh 20200801 13:20:56 all (89/726): msdos2.sh 20200801 13:21:02 all (90/726): msdos3.sh 20200801 13:21:08 all (91/726): msdos7.sh lock order reversal: 1st 0xfffff8021cdd1800 msdosfs (msdosfs) @ kern/vfs_syscalls.c:3473 2nd 0xfffff8021ccfaa88 devfs (devfs) @ fs/msdosfs/msdosfs_vnops.c:853 stack backtrace: #0 0xffffffff80c41081 at witness_debugger+0x71 #1 0xffffffff80ba47c2 at lockmgr_lock_flags+0x172 #2 0xffffffff810c5670 at VOP_LOCK1_APV+0x40 #3 0xffffffff80cc95b4 at _vn_lock+0x54 #4 0xffffffff80a7f5f9 at msdosfs_fsync+0x49 #5 0xffffffff810c42b9 at VOP_FSYNC_APV+0x59 #6 0xffffffff80cc5088 at kern_fsync+0x178 #7 0xffffffff81014969 at amd64_syscall+0x159 #8 0xffffffff80fe9dee at fast_syscall_common+0xf8 20200801 13:21:14 all (92/726): msdos8.sh 20200801 13:21:20 all (93/726): nanosleep.sh 20200801 13:21:26 all (94/726): nfs_halfpage.sh 20200801 13:22:33 all (95/726): nfs_halfpage2.sh 20200801 13:23:40 all (96/726): nfs12.sh 20200801 13:29:27 all (97/726): nfs15.sh 20200801 13:34:37 all (98/726): nfs15lockd3.sh 20200801 13:34:57 all (99/726): nfs3.sh 20200801 13:35:55 all (100/726): nfs4.sh 20200801 13:37:53 all (101/726): nfs8.sh 20200801 13:38:06 all (102/726): nfsdelegation.sh 20200801 13:38:13 all (103/726): nfsdepth.sh 20200801 13:41:21 all (104/726): nullfs13.sh 20200801 13:41:27 all (105/726): nullfs14.sh 20200801 13:41:33 all (106/726): nullfs15.sh 20200801 13:41:39 all (107/726): nullfs21.sh 20200801 13:41:45 all (108/726): nullfs23.sh 20200801 13:47:35 all (109/726): nullfs4.sh 20200801 13:47:41 all (110/726): nullfs6.sh 20200801 13:47:47 all (111/726): nullfs7.sh 20200801 13:47:53 all (112/726): nullfs9.sh 20200801 13:47:59 all (113/726): open.sh 20200801 13:48:05 all (114/726): ping.sh 20200801 13:48:13 all (115/726): pmc4.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 IAP/4/48/0x3ff IAF/3/48/0x67 20200801 13:48:19 all (116/726): pmc5.sh 20200801 13:48:25 all (117/726): pmc8.sh 20200801 13:48:31 all (118/726): posix_fadvise.sh 20200801 13:48:37 all (119/726): posix_fadvise2.sh 20200801 13:48:43 all (120/726): procfs5.sh 20200801 13:48:49 all (121/726): ptrace.sh 20200801 13:48:55 all (122/726): ptrace3.sh 20200801 13:49:01 all (123/726): pts2.sh 20200801 13:49:07 all (124/726): rdgsbase.sh 20200801 13:49:13 all (125/726): rename13.sh 20200801 13:49:19 all (126/726): rename4.sh 20200801 13:49:25 all (127/726): revoke.sh 20200801 13:49:31 all (128/726): rwlock_ronly.sh 20200801 13:49:37 all (129/726): seekdir.sh 20200801 13:49:43 all (130/726): sendfile4.sh lock order reversal: 1st 0xfffff8001b239dd0 so_snd_sx (so_snd_sx) @ kern/uipc_sockbuf.c:464 2nd 0xfffff8022137e578 ufs (ufs) @ kern/kern_sendfile.c:850 stack backtrace: #0 0xffffffff80c41081 at witness_debugger+0x71 #1 0xffffffff80ba46cc at lockmgr_lock_flags+0x7c #2 0xffffffff80e8c07d at ffs_lock+0x7d #3 0xffffffff810c5670 at VOP_LOCK1_APV+0x40 #4 0xffffffff80cc95b4 at _vn_lock+0x54 #5 0xffffffff80bd0661 at vn_sendfile+0x6f1 #6 0xffffffff80bd22a7 at sendfile+0x127 #7 0xffffffff81014969 at amd64_syscall+0x159 #8 0xffffffff80fe9dee at fast_syscall_common+0xf8 20200801 13:49:49 all (131/726): setuid.sh 20200801 13:49:56 all (132/726): sigreturn.sh 20200801 13:50:02 all (133/726): sigstop.sh 20200801 13:50:08 all (134/726): smrstress2.sh 20200801 13:50:14 all (135/726): snap12.sh lock order reversal: 1st 0xfffff8012acce800 ufs (ufs) @ kern/vfs_mount.c:1696 2nd 0xfffff805595dfb30 snaplk (snaplk) @ ufs/ffs/ffs_snapshot.c:2138 3rd 0xfffff8017fb3e2f0 ufs (ufs) @ ufs/ffs/ffs_snapshot.c:2139 stack backtrace: #0 0xffffffff80c41081 at witness_debugger+0x71 #1 0xffffffff80ba4ecc at lockmgr_xlock_hard+0x6c #2 0xffffffff80ba5c3a at __lockmgr_args+0x63a #3 0xffffffff80e66f8d at ffs_snapshot_unmount+0x13d #4 0xffffffff80e86fe9 at ffs_flushfiles+0x79 #5 0xffffffff80e6880d at softdep_flushfiles+0x9d #6 0xffffffff80e89a30 at ffs_unmount+0x60 #7 0xffffffff80ca959d at dounmount+0x43d #8 0xffffffff80ca9109 at kern_unmount+0x319 #9 0xffffffff81014969 at amd64_syscall+0x159 #10 0xffffffff80fe9dee at fast_syscall_common+0xf8 20200801 13:50:20 all (136/726): sndstat.sh 20200801 13:50:26 all (137/726): socketpair.sh 20200801 13:50:32 all (138/726): socketpair2.sh 20200801 13:50:38 all (139/726): suj2.sh 20200801 13:50:44 all (140/726): suj7.sh Failed to find journal. Use tunefs to create one Failed to start journal: 2 20200801 13:50:50 all (141/726): syzkaller1.sh 20200801 13:50:57 all (142/726): syzkaller2.sh 20200801 13:51:03 all (143/726): syzkaller3.sh 20200801 13:51:09 all (144/726): syzkaller5.sh 20200801 13:51:15 all (145/726): syzkaller6.sh 20200801 13:51:21 all (146/726): temp.sh 20200801 14:01:34 all (147/726): tmpfs15.sh 20200801 14:01:40 all (148/726): trim5.sh 20200801 14:01:46 all (149/726): truncate4.sh 20200801 14:01:52 all (150/726): truncate6.sh 20200801 14:01:58 all (151/726): uma_zalloc_arg.sh 20200801 14:02:04 all (152/726): unlink.sh 20200801 14:02:11 all (153/726): unload.sh 20200801 14:02:17 all (154/726): unload2.sh 20200801 14:02:23 all (155/726): vfork.sh 20200801 14:02:29 all (156/726): vmio.sh 20200801 14:02:35 all (157/726): watchman.sh 20200801 14:02:41 all (158/726): datagram.sh 20200801 14:02:48 all (159/726): execi386.sh 20200801 14:02:54 all (160/726): fsck3.sh 20200801 14:03:01 all (161/726): geomleak.sh 20200801 14:03:07 all (162/726): gnop5.sh GEOM_NOP: Device md10.nop created. GEOM_NOP: Device md10.nop removed. GEOM_NOP: Device md10.nop created. GEOM_NOP: Device md10.nop removed. GEOM_NOP: Device md10.nop created. GEOM_NOP: Device md10.nop removed. GEOM_NOP: Device md10.nop created. GEOM_NOP: Device md10.nop removed. GEOM_NOP: Device md10.nop created. GEOM_NOP: Device md10.nop removed. 20200801 14:03:14 all (163/726): kevent3.sh 20200801 14:03:21 all (164/726): killpg.sh 20200801 14:03:28 all (165/726): md5.sh 20200801 14:03:34 all (166/726): md9.sh 20200801 14:03:41 all (167/726): mdconfig3.sh 20200801 14:03:48 all (168/726): mmap17.sh 20200801 14:03:55 all (169/726): mmap25.sh 20200801 14:04:02 all (170/726): mmap8.sh 20200801 14:04:09 all (171/726): mmap9.sh 20200801 14:04:16 all (172/726): mountro3.sh newblk leak: 1745/2057. 20200801 14:04:23 all (173/726): msetdomain.sh 20200801 14:04:29 all (174/726): nullfs10.sh 20200801 14:04:35 all (175/726): nullfs12.sh 20200801 14:04:42 all (176/726): nullfs16.sh 20200801 14:04:50 all (177/726): openlock.sh 20200801 14:04:57 all (178/726): ptrace11.sh 20200801 14:05:04 all (179/726): ptrace8.sh 20200801 14:05:11 all (180/726): segnp.sh 20200801 14:05:18 all (181/726): sendfile.sh 20200801 14:05:24 all (182/726): sendfile2.sh 20200801 14:05:30 all (183/726): sendfile6.sh 20200801 14:05:37 all (184/726): snap10.sh lock order reversal: 1st 0xfffff80008af3080 bufwait (bufwait) @ vm/vm_pager.c:391 2nd 0xfffff805595dfb30 snaplk (snaplk) @ dev/md/md.c:987 stack backtrace: #0 0xffffffff80c41081 at witness_debugger+0x71 #1 0xffffffff80ba47c2 at lockmgr_lock_flags+0x172 #2 0xffffffff80e8c07d at ffs_lock+0x7d #3 0xffffffff810c5670 at VOP_LOCK1_APV+0x40 #4 0xffffffff80cc95b4 at _vn_lock+0x54 #5 0xffffffff806ecb32 at mdstart_vnode+0x462 #6 0xffffffff806ea7ee at md_kthread+0x1fe #7 0xffffffff80b8fc50 at fork_exit+0x80 #8 0xffffffff80fea51e at fork_trampoline+0xe mount leak: 1/91 20200801 14:05:44 all (185/726): spin.sh mount leak: 1/92 20200801 14:05:51 all (186/726): suj32.sh mount leak: 1/93 20200801 14:05:58 all (187/726): suj8.sh Aug 1 14:06:01 t2 kernel: pid 21215 (dd), uid 0 inumber 4 on /mnt: filesystem full Aug 1 14:06:02 t2 kernel: pid 21543 (dd), uid 0 inumber 5 on /mnt: filesystem full mount leak: 2/95 20200801 14:06:04 all (188/726): tmpfs4.sh mount leak: 1/96 20200801 14:06:11 all (189/726): tmpfs6.sh mount leak: 1/97 20200801 14:06:18 all (190/726): tmpfs7.sh mount leak: 1/98 20200801 14:06:24 all (191/726): trim2.sh mount leak: 1/99 20200801 14:06:31 all (192/726): ufssuspend.sh 20200801 14:06:37 all (193/726): umtx_suspend.sh 20200801 14:06:44 all (194/726): vm_fault_dontneed.sh 20200801 14:06:52 all (195/726): ext2fs3.sh mount leak: 1/100 20200801 14:07:00 all (196/726): fpclone2.sh 20200801 14:07:18 all (197/726): isofs.sh mount leak: 1/101 20200801 14:07:26 all (198/726): mlockall7.sh 20200801 14:07:34 all (199/726): mmap23.sh 20200801 14:07:42 all (200/726): mmap26.sh 20200801 14:07:50 all (201/726): mmap4.sh vnode_pager_putpages: I/O error 28 0xfffff80559dba288: type VREG usecount 0, writecount 0, refcount 3203 seqc users 0 hold count flags () flags (VI_DOINGINACT|VMP_LAZYLIST) v_object 0xfffff80181d82108 ref 0 pages 25608 cleanbuf 3200 dirtybuf 1 lock type ufs: EXCL by thread 0xfffffe012f5f8e00 (pid 36108, mmap4, tid 100232) #0 0xffffffff80ba4825 at lockmgr_lock_flags+0x1d5 #1 0xffffffff80e8c07d at ffs_lock+0x7d #2 0xffffffff810c5670 at VOP_LOCK1_APV+0x40 #3 0xffffffff80cc95b4 at _vn_lock+0x54 #4 0xffffffff80cb25a5 at vput_final+0x135 #5 0xffffffff80eb615d at vm_map_process_deferred+0xad #6 0xffffffff80ebd6eb at vm_map_remove+0xeb #7 0xffffffff80eb5a20 at vmspace_exit+0xd0 #8 0xffffffff80b8a7a2 at exit1+0x532 #9 0xffffffff80b8a26d at sys_sys_exit+0xd #10 0xffffffff81014969 at amd64_syscall+0x159 #11 0xffffffff80fe9dee at fast_syscall_common+0xf8 nlink=1, effnlink=1, size=104857600, extsize 0 generation=90f709a, uid=0, gid=0, flags=0x0 ino 4, on dev md10a vnode_pager_putpages: residual I/O 32768 at 9760 0xfffff80559dba288: type VREG usecount 0, writecount 0, refcount 3203 seqc users 0 hold count flags () flags (VI_DOINGINACT|VMP_LAZYLIST) v_object 0xfffff80181d82108 ref 0 pages 25608 cleanbuf 3200 dirtybuf 1 lock type ufs: EXCL by thread 0xfffffe012f5f8e00 (pid 36108, mmap4, tid 100232) #0 0xffffffff80ba4825 at lockmgr_lock_flags+0x1d5 #1 0xffffffff80e8c07d at ffs_lock+0x7d #2 0xffffffff810c5670 at VOP_LOCK1_APV+0x40 #3 0xffffffff80cc95b4 at _vn_lock+0x54 #4 0xffffffff80cb25a5 at vput_final+0x135 #5 0xffffffff80eb615d at vm_map_process_deferred+0xad #6 0xffffffff80ebd6eb at vm_map_remove+0xeb #7 0xffffffff80eb5a20 at vmspace_exit+0xd0 Aug 1 14:07:54 #8 0xffffffff80b8a7a2 at exit1+0x532 t2 kernel: pid 3#9 0xffffffff80b8a26d at sys_sys_exit+0xd 6108 (mmap4), ui#10 0xffffffff81014969 at amd64_syscall+0x159 d 0 inumber 4 on#11 0xffffffff80fe9dee at fast_syscall_common+0xf8 nlink=1, effnlink=1, size=104857600, extsize 0 generation=90f709a, uid=0, gid=0, flags=0x0 ino 4, on dev md10a /mnt: filesystem full vnode_pager_putpages: zero-length write at 40075264 resid 131072 0xfffff80559dba288: type VREG usecount 1, writecount 0, refcount 3203 seqc users 0 hold count flags () flags (VMP_LAZYLIST) v_object 0xfffff80181d82108 ref 0 pages 25608 cleanbuf 3201 dirtybuf 0 lock type ufs: EXCL by thread 0xfffffe012a530c00 (pid 36116, umount, tid 100950) #0 0xffffffff80ba4825 at lockmgr_lock_flags+0x1d5 #1 0xffffffff80e8c07d at ffs_lock+0x7d #2 0xffffffff810c5670 at VOP_LOCK1_APV+0x40 #3 0xffffffff80cc95b4 at _vn_lock+0x54 #4 0xffffffff80cb17da at vget_finish+0x7a #5 0xffffffff80cb5193 at vfs_periodic_msync_inactive+0x1b3 #6 0xffffffff80ca952c at dounmount+0x3cc #7 0xffffffff80ca9109 at kern_unmount+0x319 #8 0xffffffff81014969 at amd64_syscall+0x159 #9 0xffffffff80fe9dee at fast_syscall_common+0xf8 nlink=1, effnlink=1, size=104857600, extsize 0 generation=90f709a, uid=0, gid=0, flags=0x0 ino 4, on dev md10a Aug 1 14:07:55 t2 kernel: pid 36116 (umount), uid 0 inumber 4 on /mnt: filesystem full mount leak: 1/102 20200801 14:07:58 all (202/726): mountro2.sh mount leak: 1/103 20200801 14:08:06 all (203/726): nullfs3.sh mount leak: 1/104 20200801 14:08:14 all (204/726): pipe_enomem.sh kern.ipc.maxpipekva exceeded; see tuning(7) 20200801 14:08:21 all (205/726): procfs2.sh 20200801 14:08:29 all (206/726): scp.sh 20200801 14:08:55 all (207/726): truncate5.sh mount leak: 1/105 20200801 14:09:02 all (208/726): udp2.sh Aug 1 14:10:53 t2 kernel: pid 46686 (swap), jid 0, uid 1004, was killed: out of swap space Aug 1 14:12:08 t2 kernel: pid 46636 (swap), jid 0, uid 1004, was killed: out of swap space Aug 1 14:14:10 t2 kernel: pid 48988 (swap), jid 0, uid 1004, was killed: out of swap space Aug 1 14:17:59 t2 kernel: pid 50386 (swap), jid 0, uid 1004, was killed: out of swap space Aug 1 14:18:04 t2 kernel: pid 50358 (swap), jid 0, uid 1004, was killed: out of swap space Expensive timeout(9) function: 0xffffffff80becdc0(0xfffff80008e90a40) 0.013535338 s Expensive timeout(9) function: 0xffffffff80c66c10(0) 0.014011599 s Aug 1 14:20:55 t2 kernel: pid 53064 (swap), jid 0, uid 1004, was killed: out of swap space Expensive timeout(9) function: 0xffffffff80c66ce0(0) 0.023176062 s 20200801 14:21:58 all (209/726): unix_socket.sh 20200801 14:22:07 all (210/726): pmc7.sh 20200801 14:22:11 all (211/726): contigmalloc2.sh 20200801 14:22:18 all (212/726): datamove4.sh mount leak: 1/106 20200801 14:22:28 all (213/726): extattr.sh mount leak: 1/107 20200801 14:22:37 all (214/726): extattr3.sh mount leak: 1/108 20200801 14:22:45 all (215/726): fcntl.sh 20200801 14:22:53 all (216/726): mkfifo6.sh mount leak: 1/109 20200801 14:23:02 all (217/726): msdos9.sh mount leak: 1/110 20200801 14:23:11 all (218/726): rename6.sh mount leak: 1/111 20200801 14:23:19 all (219/726): contigmalloc3.sh 20200801 14:23:26 all (220/726): mmap28.sh Aug 1 14:23:31 t2 kernel: Failed to fully fault in a core file segment at VA 0x801000000 with size 0xfb7000 to be written at offset 0xa6b000 for process mmap28 20200801 14:23:33 all (221/726): mountro4.sh mount leak: 3/114 20200801 14:23:42 all (222/726): pthread.sh 20200801 14:23:53 all (223/726): quota9.sh lock order reversal: 1st 0xfffff808ed4f5490 FFS (FFS Lock) @ ufs/ufs/ufs_quota.c:759 2nd 0xfffffe012a159700 struct mount mtx (struct mount mtx) @ ufs/ufs/ufs_quota.c:765 stack backtrace: #0 0xffffffff80c41081 at witness_debugger+0x71 #1 0xffffffff80bb11a4 at __mtx_lock_flags+0x94 #2 0xffffffff80e97bad at quotaoff_inchange+0xbd #3 0xffffffff80e87096 at ffs_flushfiles+0x126 #4 0xffffffff80e6880d at softdep_flushfiles+0x9d #5 0xffffffff80e89a30 at ffs_unmount+0x60 #6 0xffffffff80ca959d at dounmount+0x43d #7 0xffffffff80ca9109 at kern_unmount+0x319 #8 0xffffffff81014969 at amd64_syscall+0x159 #9 0xffffffff80fe9dee at fast_syscall_common+0xf8 mount leak: 1/115 20200801 14:24:02 all (224/726): tmpfs5.sh mount leak: 1/116 20200801 14:24:13 all (225/726): devfd.sh mount leak: 1/117 20200801 14:24:24 all (226/726): kevent8.sh lock order reversal: 1st 0xfffffe012e898978 filedesc structure (filedesc structure) @ kern/kern_descrip.c:1303 2nd 0xfffff8014ae24d10 ufs (ufs) @ kern/vfs_subr.c:6196 stack backtrace: #0 0xffffffff80c41081 at witness_debugger+0x71 #1 0xffffffff80ba47c2 at lockmgr_lock_flags+0x172 #2 0xffffffff80e8c07d at ffs_lock+0x7d #3 0xffffffff810c5670 at VOP_LOCK1_APV+0x40 #4 0xffffffff80cc95b4 at _vn_lock+0x54 #5 0xffffffff80b84f39 at knlist_remove_kq+0x89 #6 0xffffffff80cbd9f4 at filt_vfsdetach+0x24 #7 0xffffffff80b859f1 at knote_fdclose+0x1b1 #8 0xffffffff80b77ab4 at closefp+0x104 #9 0xffffffff81014969 at amd64_syscall+0x159 #10 0xffffffff80fe9dee at fast_syscall_common+0xf8 mount leak: 1/118 20200801 14:24:35 all (227/726): mlockall4.sh mount leak: 1/119 20200801 14:24:46 all (228/726): mmap13.sh 20200801 14:24:57 all (229/726): ptrace2.sh 20200801 14:25:09 all (230/726): ptrace9.sh 20200801 14:25:20 all (231/726): sendfile18.sh GEOM_NOP: Device md10.nop created. GEOM_NOP: Device md10.nop removed. mount leak: 1/120 20200801 14:25:32 all (232/726): snap2.sh mount leak: 1/121 20200801 14:25:43 all (233/726): tmpfs10.sh mount leak: 1/122 20200801 14:25:54 all (234/726): truncate7.sh mount leak: 1/123 20200801 14:26:06 all (235/726): kevent2.sh 20200801 14:26:19 all (236/726): procfs3.sh 20200801 14:26:30 all (237/726): rename11.sh mount leak: 1/124 20200801 14:26:41 all (238/726): symlink2.sh mount leak: 2/126 20200801 14:26:53 all (239/726): proccontrol.sh 20200801 14:27:07 all (240/726): sigfastblock.sh 20200801 14:27:19 all (241/726): sparse.sh mount leak: 1/127 20200801 14:27:35 all (242/726): symlink4.sh mount leak: 1/128 20200801 14:27:48 all (243/726): truncate3.sh mount leak: 1/129 20200801 14:28:01 all (244/726): linger2.sh Aug 1 14:28:09 t2 kernel: pid 10871 (linger2), uid 1004 inumber 888 on /mnt: filesystem full mount leak: 1/130 20200801 14:28:15 all (245/726): procfs.sh 20200801 14:28:30 all (246/726): pshared.sh 20200801 14:28:44 all (247/726): datamove5.sh lock order reversal: 1st 0xfffff80205c82800 nullfs (nullfs) @ /usr/src/sys/fs/nullfs/null_vnops.c:799 2nd 0xfffffe006e7a1c88 bufwait (bufwait) @ ufs/ffs/ffs_softdep.c:7236 stack backtrace: #0 0xffffffff80c41081 at witness_debugger+0x71 #1 0xffffffff80ba4ecc at lockmgr_xlock_hard+0x6c #2 0xffffffff80ba5c3a at __lockmgr_args+0x63a #3 0xffffffff80e73f76 at trunc_dependencies+0x9c6 #4 0xffffffff80e752d3 at softdep_setup_freeblocks+0x8c3 #5 0xffffffff80e60788 at ffs_truncate+0x748 #6 0xffffffff80e930dc at ufs_inactive+0x1ac #7 0xffffffff810c5209 at VOP_INACTIVE_APV+0x59 #8 0xffffffff80cb3467 at vinactivef+0x107 #9 0xffffffff80cb2708 at vput_final+0x298 #10 0xffffffff82339846 at null_reclaim+0xf6 #11 0xffffffff810c54f9 at VOP_RECLAIM_APV+0x59 #12 0xffffffff80cb4357 at vgonel+0x3d7 #13 0xffffffff80cb4a1c at vrecycle+0x5c #14 0xffffffff823396e7 at null_inactive+0x57 #15 0xffffffff810c5209 at VOP_INACTIVE_APV+0x59 #16 0xffffffff80cb3467 at vinactivef+0x107 #17 0xffffffff80cb2708 at vput_final+0x298 mount leak: 1/131 20200801 14:28:59 all (248/726): thr.sh 20200801 14:29:14 all (249/726): nfs15lockd2.sh 20200801 14:34:28 all (250/726): sendfile24.sh GEOM_NOP: Device md10.nop created. g_vfs_done():md10.nop[READ(offset=16977920, length=4096)]error = 5 g_vfs_done():md10.nop[READ(offset=16977920, length=4096)]error = 5 g_vfs_done():md10.nop[READ(offset=16977920, length=4096)]error = 5 g_vfs_done():md10.nop[READ(offset=16977920, length=4096)]error = 5 g_vfs_done():md10.nop[READ(offset=16977920, length=4096)]error = 5 g_vfs_done():md10.nop[READ(offset=16977920, length=4096)]error = 5 g_vfs_done():md10.nop[READ(offset=16977920, length=4096)]error = 5 g_vfs_done():md10.nop[READ(offset=16977920, length=4096)]error = 5 g_vfs_done():md10.nop[READ(offset=16977920, length=4096)]error = 5 g_vfs_done():md10.nop[READ(offset=16977920, length=4096)]error = 5 g_vfs_done():md10.nop[READ(offset=16977920, length=4096)]error = 5 GEOM_NOP: Device md10.nop removed. mount leak: 1/132 20200801 14:34:43 all (251/726): growfs.sh g_access(1029): provider ufsid/5f25616790032d06 has error 6 set g_access(1029): provider ufsid/5f25616790032d06 has error 6 set g_access(1029): provider ufsid/5f25616790032d06 has error 6 set g_access(1029): provider gptid/6273e456-d3f3-11ea-85ea-001e6756b69b has error 6 set g_access(1029): provider gptid/6273e456-d3f3-11ea-85ea-001e6756b69b has error 6 set g_access(1029): provider gptid/6273e456-d3f3-11ea-85ea-001e6756b69b has error 6 set g_dev_taste: make_dev_p() failed (gp->name=gptid/6273e456-d3f3-11ea-85ea-001e6756b69b, error=17) g_dev_taste: make_dev_p() failed (gp->name=gptid/6273e456-d3f3-11ea-85ea-001e6756b69b, error=17) g_dev_taste: make_dev_p() failed (gp->name=ufsid/5f25616790032d06, error=17) g_dev_taste: make_dev_p() failed (gp->name=gptid/6273e456-d3f3-11ea-85ea-001e6756b69b, error=17) g_dev_taste: make_dev_p() failed (gp->name=ufsid/5f25616790032d06, error=17) g_dev_taste: make_dev_p() failed (gp->name=gptid/6273e456-d3f3-11ea-85ea-001e6756b69b, error=17) g_dev_taste: make_dev_p() failed (gp->name=ufsid/5f25616790032d06, error=17) mount leak: 1/133 GEOM leak: 14/191 20200801 14:35:00 all (252/726): linger3.sh mount leak: 1/134 20200801 14:35:16 all (253/726): tmpfs14.sh mount leak: 1/135 20200801 14:35:32 all (254/726): pipe2.sh 20200801 14:35:51 all (255/726): umountf11.sh mount leak: 1/136 20200801 14:36:10 all (256/726): umountf12.sh mount leak: 1/137 20200801 14:36:28 all (257/726): sendfile12.sh mount leak: 1/138 20200801 14:36:48 all (258/726): mountu.sh mount leak: 3/141 20200801 14:37:18 all (259/726): posix_fadvise3.sh 20200801 14:37:41 all (260/726): pdfork.sh 20200801 14:38:05 all (261/726): rename8.sh mount leak: 1/142 20200801 14:38:26 all (262/726): ffs_blkfree.sh newblk leak: 30733/32790. mount leak: 11/153 20200801 14:38:51 all (263/726): mountro6.sh mount leak: 1/154 20200801 14:39:18 all (264/726): multicast2.sh Aug 1 14:39:21 t2 mDNSResponder[84226]: mDNSResponder (Engineering Build) (Nov 5 2019 03:51:04) starting Aug 1 14:39:21 t2 mDNSResponder[84226]: mDNS_AddDNSServer: Lock not held! mDNS_busy (0) mDNS_reentrancy (0) if_delmulti_locked: detaching ifnet instance 0xfffff8053b984000 if_delmulti_locked: detaching ifnet instance 0xfffff8053b984000 if_delmulti_locked: detaching ifnet instance 0xfffff8053b984000 if_delmulti_locked: detaching ifnet instance 0xfffff80005812800 if_delmulti_locked: detaching ifnet instance 0xfffff80005812800 if_delmulti_locked: detaching ifnet instance 0xfffff8053b984000 if_delmulti_locked: detaching ifnet instance 0xfffff80005812800 if_delmulti_locked: detaching ifnet instance 0xfffff80005812800 if_delmulti_locked: detaching ifnet instance 0xfffff8053b984000 if_delmulti_locked: detaching ifnet instance 0xfffff8053b984000 if_delmulti_locked: detaching ifnet instance 0xfffff8053b984000 Aug 1 14:39:29 t2 mDNSResponder[84226]: mDNSResponder (Engineering Build) (Nov 5 2019 03:51:04) stopping if_delmulti_locked: detaching ifnet instance 0xfffff80005812800 if_delmulti_locked: detaching ifnet instance 0xfffff80005812800 20200801 14:39:43 all (265/726): core4.sh Aug 1 14:39:48 t2 kernel: Failed to write core file for process core4 (error 5) mount leak: 1/155 20200801 14:40:09 all (266/726): fsck5.sh Aug 1 14:40:06 t2 syslogd: last message repeated 59 times mount leak: 1/156 20200801 14:40:36 all (267/726): suj33.sh mount leak: 1/157 20200801 14:41:03 all (268/726): nfs15lockd.sh panic: crunuse: users 0 not > 0 on cred 0xfffff80d94fe3000 cpuid = 2 time = 1596285726 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe09a00136a0 vpanic() at vpanic+0x182/frame 0xfffffe09a00136f0 panic() at panic+0x43/frame 0xfffffe09a0013750 crunuse() at crunuse+0xae/frame 0xfffffe09a0013780 crcowfree() at crcowfree+0x9/frame 0xfffffe09a0013790 thread_reap() at thread_reap+0xc2/frame 0xfffffe09a00137b0 proc_reap() at proc_reap+0x49e/frame 0xfffffe09a00137f0 proc_to_reap() at proc_to_reap+0x3ce/frame 0xfffffe09a0013840 kern_wait6() at kern_wait6+0x1bb/frame 0xfffffe09a00138e0 sys_wait4() at sys_wait4+0x7b/frame 0xfffffe09a0013ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe09a0013bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe09a0013bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x8003e71fa, rsp = 0x7fffffffe4e8, rbp = 0x7fffffffe520 --- KDB: enter: panic [ thread pid 95350 tid 103654 ] Stopped at kdb_enter+0x37: movq $0,0x10b8016(%rip) db> run pho db:0:pho> set $lines 20000 db:0:pho> run pho1 db:1:pho1> dump Dumping 3429 out of 65426 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Dump complete db:1:pho1> bt Tracing pid 95350 tid 103654 td 0xfffffe014eca4300 kdb_enter() at kdb_enter+0x37/frame 0xfffffe09a00136a0 vpanic() at vpanic+0x19e/frame 0xfffffe09a00136f0 panic() at panic+0x43/frame 0xfffffe09a0013750 crunuse() at crunuse+0xae/frame 0xfffffe09a0013780 crcowfree() at crcowfree+0x9/frame 0xfffffe09a0013790 thread_reap() at thread_reap+0xc2/frame 0xfffffe09a00137b0 proc_reap() at proc_reap+0x49e/frame 0xfffffe09a00137f0 proc_to_reap() at proc_to_reap+0x3ce/frame 0xfffffe09a0013840 kern_wait6() at kern_wait6+0x1bb/frame 0xfffffe09a00138e0 sys_wait4() at sys_wait4+0x7b/frame 0xfffffe09a0013ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe09a0013bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe09a0013bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x8003e71fa, rsp = 0x7fffffffe4e8, rbp = 0x7fffffffe520 --- db:1:pho1> show allpcpu Current CPU: 2 cpuid = 0 dynamic pcpu = 0x795ec0 curthread = 0xfffffe006a5c0300: pid 11 tid 100003 critnest 1 "idle: cpu0" curpcb = 0xfffffe006a5c0810 fpcurthread = none idlethread = 0xfffffe006a5c0300: tid 100003 "idle: cpu0" self = 0xffffffff82210000 curpmap = 0xffffffff81c8bc60 tssp = 0xffffffff82210384 rsp0 = 0xfffffe0069fe7cc0 kcr3 = 0x8000000002125002 ucr3 = 0xffffffffffffffff scr3 = 0x171c09a27 gs32p = 0xffffffff82210404 ldt = 0xffffffff82210444 tss = 0xffffffff82210434 curvnet = 0 spin locks held: cpuid = 1 dynamic pcpu = 0xfffffe00eeff6ec0 curthread = 0xfffffe006a5bfc00: pid 11 tid 100004 critnest 0 "idle: cpu1" curpcb = 0xfffffe006a5c0110 fpcurthread = none idlethread = 0xfffffe006a5bfc00: tid 100004 "idle: cpu1" self = 0xffffffff82211000 curpmap = 0xffffffff81c8bc60 tssp = 0xffffffff82211384 rsp0 = 0xfffffe0069feccc0 kcr3 = 0x8000000002125002 ucr3 = 0xffffffffffffffff scr3 = 0x632d91960 gs32p = 0xffffffff82211404 ldt = 0xffffffff82211444 tss = 0xffffffff82211434 curvnet = 0 spin locks held: cpuid = 2 dynamic pcpu = 0xfffffe00eeffdec0 curthread = 0xfffffe014eca4300: pid 95350 tid 103654 critnest 1 "nfs15lockd" curpcb = 0xfffffe014eca4810 fpcurthread = 0xfffffe014eca4300: pid 95350 "nfs15lockd" idlethread = 0xfffffe006a5bf500: tid 100005 "idle: cpu2" self = 0xffffffff82212000 curpmap = 0xfffffe013cc074f0 tssp = 0xffffffff82212384 rsp0 = 0xfffffe09a0013cc0 kcr3 = 0x80000006e79f424e ucr3 = 0x800000063661ea4e scr3 = 0x1a7818a47 gs32p = 0xffffffff82212404 ldt = 0xffffffff82212444 tss = 0xffffffff82212434 curvnet = 0 spin locks held: cpuid = 3 dynamic pcpu = 0xfffffe00ef008ec0 curthread = 0xfffffe006a5bee00: pid 11 tid 100006 critnest 1 "idle: cpu3" curpcb = 0xfffffe006a5bf310 fpcurthread = none idlethread = 0xfffffe006a5bee00: tid 100006 "idle: cpu3" self = 0xffffffff82213000 curpmap = 0xffffffff81c8bc60 tssp = 0xffffffff82213384 rsp0 = 0xfffffe0069ff6cc0 kcr3 = 0x8000000002125002 ucr3 = 0xffffffffffffffff scr3 = 0x6af33d97c gs32p = 0xffffffff82213404 ldt = 0xffffffff82213444 tss = 0xffffffff82213434 curvnet = 0 spin locks held: cpuid = 4 dynamic pcpu = 0xfffffe00ef00fec0 curthread = 0xfffffe006a5be700: pid 11 tid 100007 critnest 1 "idle: cpu4" curpcb = 0xfffffe006a5bec10 fpcurthread = none idlethread = 0xfffffe006a5be700: tid 100007 "idle: cpu4" self = 0xffffffff82214000 curpmap = 0xffffffff81c8bc60 tssp = 0xffffffff82214384 rsp0 = 0xfffffe0069ffbcc0 kcr3 = 0x8000000002125002 ucr3 = 0xffffffffffffffff scr3 = 0x2d54ec82c gs32p = 0xffffffff82214404 ldt = 0xffffffff82214444 tss = 0xffffffff82214434 curvnet = 0 spin locks held: cpuid = 5 dynamic pcpu = 0xfffffe00ef01aec0 curthread = 0xfffffe006a5be000: pid 11 tid 100008 critnest 1 "idle: cpu5" curpcb = 0xfffffe006a5be510 fpcurthread = none idlethread = 0xfffffe006a5be000: tid 100008 "idle: cpu5" self = 0xffffffff82215000 curpmap = 0xffffffff81c8bc60 tssp = 0xffffffff82215384 rsp0 = 0xfffffe00e0530cc0 kcr3 = 0x8000000002125002 ucr3 = 0xffffffffffffffff scr3 = 0x6af33da0b gs32p = 0xffffffff82215404 ldt = 0xffffffff82215444 tss = 0xffffffff82215434 curvnet = 0 spin locks held: cpuid = 6 dynamic pcpu = 0xfffffe00ef021ec0 curthread = 0xfffffe006a5c8a00: pid 11 tid 100009 critnest 1 "idle: cpu6" curpcb = 0xfffffe006a5c8f10 fpcurthread = none idlethread = 0xfffffe006a5c8a00: tid 100009 "idle: cpu6" self = 0xffffffff82216000 curpmap = 0xffffffff81c8bc60 tssp = 0xffffffff82216384 rsp0 = 0xfffffe00e0535cc0 kcr3 = 0x8000000002125002 ucr3 = 0xffffffffffffffff scr3 = 0x63661eb20 gs32p = 0xffffffff82216404 ldt = 0xffffffff82216444 tss = 0xffffffff82216434 curvnet = 0 spin locks held: cpuid = 7 dynamic pcpu = 0xfffffe00ef02cec0 curthread = 0xfffffe006a5c8300: pid 11 tid 100010 critnest 1 "idle: cpu7" curpcb = 0xfffffe006a5c8810 fpcurthread = none idlethread = 0xfffffe006a5c8300: tid 100010 "idle: cpu7" self = 0xffffffff82217000 curpmap = 0xffffffff81c8bc60 tssp = 0xffffffff82217384 rsp0 = 0xfffffe00e053acc0 kcr3 = 0x8000000002125002 ucr3 = 0xffffffffffffffff scr3 = 0x1a7abca40 gs32p = 0xffffffff82217404 ldt = 0xffffffff82217444 tss = 0xffffffff82217434 curvnet = 0 spin locks held: cpuid = 8 dynamic pcpu = 0xfffffe00ef033ec0 curthread = 0xfffffe006a5c7c00: pid 11 tid 100011 critnest 0 "idle: cpu8" curpcb = 0xfffffe006a5c8110 fpcurthread = none idlethread = 0xfffffe006a5c7c00: tid 100011 "idle: cpu8" self = 0xffffffff82218000 curpmap = 0xffffffff81c8bc60 tssp = 0xffffffff82218384 rsp0 = 0xfffffe00e053fcc0 kcr3 = 0x8000000002125002 ucr3 = 0xffffffffffffffff scr3 = 0x632c008b2 gs32p = 0xffffffff82218404 ldt = 0xffffffff82218444 tss = 0xffffffff82218434 curvnet = 0 spin locks held: cpuid = 9 dynamic pcpu = 0xfffffe00ef03eec0 curthread = 0xfffffe006a5c7500: pid 11 tid 100012 critnest 1 "idle: cpu9" curpcb = 0xfffffe006a5c7a10 fpcurthread = none idlethread = 0xfffffe006a5c7500: tid 100012 "idle: cpu9" self = 0xffffffff82219000 curpmap = 0xffffffff81c8bc60 tssp = 0xffffffff82219384 rsp0 = 0xfffffe00e0544cc0 kcr3 = 0x8000000002125002 ucr3 = 0xffffffffffffffff scr3 = 0x632c00b2a gs32p = 0xffffffff82219404 ldt = 0xffffffff82219444 tss = 0xffffffff82219434 curvnet = 0 spin locks held: cpuid = 10 dynamic pcpu = 0xfffffe00ef045ec0 curthread = 0xfffffe006a5c6e00: pid 11 tid 100013 critnest 1 "idle: cpu10" curpcb = 0xfffffe006a5c7310 fpcurthread = none idlethread = 0xfffffe006a5c6e00: tid 100013 "idle: cpu10" self = 0xffffffff8221a000 curpmap = 0xffffffff81c8bc60 tssp = 0xffffffff8221a384 rsp0 = 0xfffffe00e0549cc0 kcr3 = 0x8000000002125002 ucr3 = 0xffffffffffffffff scr3 = 0x171c09b11 gs32p = 0xffffffff8221a404 ldt = 0xffffffff8221a444 tss = 0xffffffff8221a434 curvnet = 0 spin locks held: cpuid = 11 dynamic pcpu = 0xfffffe00ef050ec0 curthread = 0xfffffe006a5c6700: pid 11 tid 100014 critnest 1 "idle: cpu11" curpcb = 0xfffffe006a5c6c10 fpcurthread = none idlethread = 0xfffffe006a5c6700: tid 100014 "idle: cpu11" self = 0xffffffff8221b000 curpmap = 0xffffffff81c8bc60 tssp = 0xffffffff8221b384 rsp0 = 0xfffffe00e054ecc0 kcr3 = 0x8000000002125002 ucr3 = 0xffffffffffffffff scr3 = 0x6af33da31 gs32p = 0xffffffff8221b404 ldt = 0xffffffff8221b444 tss = 0xffffffff8221b434 curvnet = 0 spin locks held: cpuid = 12 dynamic pcpu = 0xfffffe00e86a4ec0 curthread = 0xfffffe006a5c6000: pid 11 tid 100015 critnest 1 "idle: cpu12" curpcb = 0xfffffe006a5c6510 fpcurthread = none idlethread = 0xfffffe006a5c6000: tid 100015 "idle: cpu12" self = 0xffffffff8221c000 curpmap = 0xffffffff81c8bc60 tssp = 0xffffffff8221c384 rsp0 = 0xfffffe00e0553cc0 kcr3 = 0x8000000002125002 ucr3 = 0xffffffffffffffff scr3 = 0xb86b38a93 gs32p = 0xffffffff8221c404 ldt = 0xffffffff8221c444 tss = 0xffffffff8221c434 curvnet = 0 spin locks held: cpuid = 13 dynamic pcpu = 0xfffffe00e86abec0 curthread = 0xfffffe006a5c1800: pid 11 tid 100016 critnest 1 "idle: cpu13" curpcb = 0xfffffe006a5c1d10 fpcurthread = none idlethread = 0xfffffe006a5c1800: tid 100016 "idle: cpu13" self = 0xffffffff8221d000 curpmap = 0xffffffff81c8bc60 tssp = 0xffffffff8221d384 rsp0 = 0xfffffe00e0558cc0 kcr3 = 0x8000000002125002 ucr3 = 0xffffffffffffffff scr3 = 0x33bebb8a6 gs32p = 0xffffffff8221d404 ldt = 0xffffffff8221d444 tss = 0xffffffff8221d434 curvnet = 0 spin locks held: cpuid = 14 dynamic pcpu = 0xfffffe00e86b6ec0 curthread = 0xfffffe006a5cc300: pid 11 tid 100017 critnest 1 "idle: cpu14" curpcb = 0xfffffe006a5cc810 fpcurthread = none idlethread = 0xfffffe006a5cc300: tid 100017 "idle: cpu14" self = 0xffffffff8221e000 curpmap = 0xffffffff81c8bc60 tssp = 0xffffffff8221e384 rsp0 = 0xfffffe00e055dcc0 kcr3 = 0x8000000002125002 ucr3 = 0xffffffffffffffff scr3 = 0x3d44c4b65 gs32p = 0xffffffff8221e404 ldt = 0xffffffff8221e444 tss = 0xffffffff8221e434 curvnet = 0 spin locks held: cpuid = 15 dynamic pcpu = 0xfffffe00e86bdec0 curthread = 0xfffffe006a5cbc00: pid 11 tid 100018 critnest 1 "idle: cpu15" curpcb = 0xfffffe006a5cc110 fpcurthread = none idlethread = 0xfffffe006a5cbc00: tid 100018 "idle: cpu15" self = 0xffffffff8221f000 curpmap = 0xffffffff81c8bc60 tssp = 0xffffffff8221f384 rsp0 = 0xfffffe00e0562cc0 kcr3 = 0x8000000002125002 ucr3 = 0xffffffffffffffff scr3 = 0xcf0395c19 gs32p = 0xffffffff8221f404 ldt = 0xffffffff8221f444 tss = 0xffffffff8221f434 curvnet = 0 spin locks held: cpuid = 16 dynamic pcpu = 0xfffffe00e86c8ec0 curthread = 0xfffffe006a5cb500: pid 11 tid 100019 critnest 1 "idle: cpu16" curpcb = 0xfffffe006a5cba10 fpcurthread = none idlethread = 0xfffffe006a5cb500: tid 100019 "idle: cpu16" self = 0xffffffff82220000 curpmap = 0xffffffff81c8bc60 tssp = 0xffffffff82220384 rsp0 = 0xfffffe00e0567cc0 kcr3 = 0x8000000002125002 ucr3 = 0xffffffffffffffff scr3 = 0x21f5bac07 gs32p = 0xffffffff82220404 ldt = 0xffffffff82220444 tss = 0xffffffff82220434 curvnet = 0 spin locks held: cpuid = 17 dynamic pcpu = 0xfffffe00e86cfec0 curthread = 0xfffffe006a5cae00: pid 11 tid 100020 critnest 1 "idle: cpu17" curpcb = 0xfffffe006a5cb310 fpcurthread = none idlethread = 0xfffffe006a5cae00: tid 100020 "idle: cpu17" self = 0xffffffff82221000 curpmap = 0xffffffff81c8bc60 tssp = 0xffffffff82221384 rsp0 = 0xfffffe00e056ccc0 kcr3 = 0x8000000002125002 ucr3 = 0xffffffffffffffff scr3 = 0x6af33dcb8 gs32p = 0xffffffff82221404 ldt = 0xffffffff82221444 tss = 0xffffffff82221434 curvnet = 0 spin locks held: cpuid = 18 dynamic pcpu = 0xfffffe00e86daec0 curthread = 0xfffffe006a5ca700: pid 11 tid 100021 critnest 1 "idle: cpu18" curpcb = 0xfffffe006a5cac10 fpcurthread = none idlethread = 0xfffffe006a5ca700: tid 100021 "idle: cpu18" self = 0xffffffff82222000 curpmap = 0xffffffff81c8bc60 tssp = 0xffffffff82222384 rsp0 = 0xfffffe00e0571cc0 kcr3 = 0x8000000002125002 ucr3 = 0xffffffffffffffff scr3 = 0xcf07019fa gs32p = 0xffffffff82222404 ldt = 0xffffffff82222444 tss = 0xffffffff82222434 curvnet = 0 spin locks held: cpuid = 19 dynamic pcpu = 0xfffffe00e86e1ec0 curthread = 0xfffffe006a5ca000: pid 11 tid 100022 critnest 1 "idle: cpu19" curpcb = 0xfffffe006a5ca510 fpcurthread = none idlethread = 0xfffffe006a5ca000: tid 100022 "idle: cpu19" self = 0xffffffff82223000 curpmap = 0xffffffff81c8bc60 tssp = 0xffffffff82223384 rsp0 = 0xfffffe00e0576cc0 kcr3 = 0x8000000002125002 ucr3 = 0xffffffffffffffff scr3 = 0xae7213c05 gs32p = 0xffffffff82223404 ldt = 0xffffffff82223444 tss = 0xffffffff82223434 curvnet = 0 spin locks held: cpuid = 20 dynamic pcpu = 0xfffffe00e86ecec0 curthread = 0xfffffe006a5c9800: pid 11 tid 100023 critnest 1 "idle: cpu20" curpcb = 0xfffffe006a5c9d10 fpcurthread = none idlethread = 0xfffffe006a5c9800: tid 100023 "idle: cpu20" self = 0xffffffff82224000 curpmap = 0xffffffff81c8bc60 tssp = 0xffffffff82224384 rsp0 = 0xfffffe00e057bcc0 kcr3 = 0x8000000002125002 ucr3 = 0xffffffffffffffff scr3 = 0x380b43896 gs32p = 0xffffffff82224404 ldt = 0xffffffff82224444 tss = 0xffffffff82224434 curvnet = 0 spin locks held: cpuid = 21 dynamic pcpu = 0xfffffe00e86f3ec0 curthread = 0xfffffe006a5c9100: pid 11 tid 100024 critnest 1 "idle: cpu21" curpcb = 0xfffffe006a5c9610 fpcurthread = none idlethread = 0xfffffe006a5c9100: tid 100024 "idle: cpu21" self = 0xffffffff82225000 curpmap = 0xffffffff81c8bc60 tssp = 0xffffffff82225384 rsp0 = 0xfffffe00e0580cc0 kcr3 = 0x8000000002125002 ucr3 = 0xffffffffffffffff scr3 = 0x107117c79 gs32p = 0xffffffff82225404 ldt = 0xffffffff82225444 tss = 0xffffffff82225434 curvnet = 0 spin locks held: cpuid = 22 dynamic pcpu = 0xfffffe00e86feec0 curthread = 0xfffffe006a5cfc00: pid 11 tid 100025 critnest 1 "idle: cpu22" curpcb = 0xfffffe006a5d0110 fpcurthread = none idlethread = 0xfffffe006a5cfc00: tid 100025 "idle: cpu22" self = 0xffffffff82226000 curpmap = 0xffffffff81c8bc60 tssp = 0xffffffff82226384 rsp0 = 0xfffffe00e0585cc0 kcr3 = 0x8000000002125002 ucr3 = 0xffffffffffffffff scr3 = 0x107117baf gs32p = 0xffffffff82226404 ldt = 0xffffffff82226444 tss = 0xffffffff82226434 curvnet = 0 spin locks held: cpuid = 23 dynamic pcpu = 0xfffffe00e8705ec0 curthread = 0xfffffe006a5cf500: pid 11 tid 100026 critnest 1 "idle: cpu23" curpcb = 0xfffffe006a5cfa10 fpcurthread = none idlethread = 0xfffffe006a5cf500: tid 100026 "idle: cpu23" self = 0xffffffff82227000 curpmap = 0xffffffff81c8bc60 tssp = 0xffffffff82227384 rsp0 = 0xfffffe00e058acc0 kcr3 = 0x8000000002125002 ucr3 = 0xffffffffffffffff scr3 = 0x6af33da05 gs32p = 0xffffffff82227404 ldt = 0xffffffff82227444 tss = 0xffffffff82227434 curvnet = 0 spin locks held: db:1:pho1> show alllocks Process 1808 (nfs15lockd) thread 0xfffffe0158283100 (102160) exclusive lockmgr nfs (nfs) r = 0 (0xfffff80dd5bd1068) locked @ kern/kern_sig.c:3708 Process 95350 (nfs15lockd) thread 0xfffffe014eca4300 (103654) exclusive sleep mutex cred (cred) r = 0 (0xfffff80d94fe3000) locked @ kern/kern_prot.c:1885 db:1:pho1> show lockedvnods Locked vnodes vnode 0xfffff80dd5bd1000: type VREG usecount 2, writecount 1, refcount 1 seqc users 1 hold count flags () flags (VV_VMSIZEVNLOCK|VV(0x20)|VMP_LAZYLIST) v_object 0xfffff80d962f7630 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type nfs: EXCL by thread 0xfffffe0158283100 (pid 1808, nfs15lockd, tid 102160) #0 0xffffffff80ba4825 at lockmgr_lock_flags+0x1d5 #1 0xffffffff810c5670 at VOP_LOCK1_APV+0x40 #2 0xffffffff80abc1bc at nfs_lock+0x2c #3 0xffffffff80c9fc7f at vop_sigdefer+0x2f #4 0xffffffff810c5670 at VOP_LOCK1_APV+0x40 #5 0xffffffff80cc95b4 at _vn_lock+0x54 #6 0xffffffff80bd8522 at sigexit+0xcb2 #7 0xffffffff80bd8ddc at postsig+0x2cc #8 0xffffffff80c3622b at ast+0x54b #9 0xffffffff80fe9e9b at fast_syscall_common+0x1a5 fileid 17381380 fsid 0x3a3abdffeb db:1:pho1> show allchains chain 1: thread 103317 (pid 1811, sleep) is sleeping on 0xffffffff81cab9d2 "nanslp" chain 2: thread 102160 (pid 1808, nfs15lockd) is sleeping on 0xfffff80a055fdc00 "nfsreq" chain 3: thread 100709 (pid 1794, sleep) is sleeping on 0xffffffff81cab9d3 "nanslp" chain 4: thread 103580 (pid 95360, newnfs 1) is sleeping on 0xffffffff81c72ac4 "-" chain 5: thread 100726 (pid 95359, newnfs 0) is sleeping on 0xffffffff81c72ac0 "-" chain 6: thread 101253 (pid 95302, sh) is sleeping on 0xfffff8001b118000 "wait" chain 7: thread 101358 (pid 95301, tee) is sleeping on 0xfffff80241aae000 "piperd" chain 8: thread 101089 (pid 95300, sh) is sleeping on 0xfffff80044f3a000 "wait" chain 9: thread 101687 (pid 28670, ftcleanup) is sleeping on 0xffffffff82c8c4f4 "ftcl" chain 10: thread 101227 (pid 20471, sctp_iterator) is sleeping on 0xffffffff823cef98 "waiting_for_work" chain 11: thread 100343 (pid 12480, top) is sleeping on 0xfffff8001b0142c0 "select" chain 12: thread 100706 (pid 12206, bash) is sleeping on 0xfffff8001b5f2520 "wait" chain 13: thread 100310 (pid 12123, sshd) is sleeping on 0xfffff8001bb819c0 "select" chain 14: thread 100336 (pid 12055, sshd) is sleeping on 0xfffff80003bb0440 "select" chain 15: thread 100301 (pid 11186, sh) is sleeping on 0xfffff8021f2df2f8 "piperd" chain 16: thread 100364 (pid 11184, awk) is sleeping on 0xfffff8021f4e9000 "piperd" chain 17: thread 100377 (pid 11183, sh) is sleeping on 0xfffff80008f5a000 "wait" chain 18: thread 100700 (pid 11162, sh) is sleeping on 0xfffff8021f444000 "wait" chain 19: thread 100359 (pid 8590, sh) is sleeping on 0xfffff80846b28000 "wait" chain 20: thread 100319 (pid 2775, bash) is blocked on lockmgr @ EXCL thread 100319 (pid 2775, bash) is blocked on lockmgr @ EXCL thread 100319 (pid 2775, bash) is blocked on lockmgr @ : : EXCL thread 100319 (pid 2775, bash) is blocked on lockmgr @ EXCL thread 100319 (pid 2775, bash) is blocked on lockmgr @ EXCL thread 100319 (pid 2775, bash) is blocked on lockmgr @ EXCL >Power Cycle> HCopyright(c) 2009 - 2012 Intel Corporation.All rights reserved. Version 2.00.1201.Copyright(c) 2010 - 2012 American Megatrends,Inc. Installed BIOS: SE5C600.86B.01.08.0003 Platform ID: W2600CR (kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=0x0) at ../../../kern/kern_shutdown.c:394 #2 0xffffffff8049c38a in db_dump (dummy=, dummy2=, dummy3=, dummy4=) at ../../../ddb/db_command.c:575 #3 0xffffffff8049c14c in db_command (last_cmdp=, cmd_table=, dopager=0x0) at ../../../ddb/db_command.c:482 #4 0xffffffff804a1158 in db_script_exec (scriptname=, warnifnotfound=) at ../../../ddb/db_script.c:304 #5 0xffffffff8049c14c in db_command (last_cmdp=, cmd_table=, dopager=0x0) at ../../../ddb/db_command.c:482 #6 0xffffffff804a1158 in db_script_exec (scriptname=, warnifnotfound=) at ../../../ddb/db_script.c:304 #7 0xffffffff8049c14c in db_command (last_cmdp=, cmd_table=, dopager=0x1) at ../../../ddb/db_command.c:482 #8 0xffffffff8049bebd in db_command_loop () at ../../../ddb/db_command.c:535 #9 0xffffffff8049f168 in db_trap (type=, code=) at ../../../ddb/db_main.c:270 #10 0xffffffff80c1ebf4 in kdb_trap (type=0x3, code=0x0, tf=) at ../../../kern/subr_kdb.c:699 #11 0xffffffff81013b18 in trap (frame=0xfffffe09a00135d0) at ../../../amd64/amd64/trap.c:576 #12 #13 kdb_enter (why=0xffffffff811deaed "panic", msg=) at ../../../kern/subr_kdb.c:486 #14 0xffffffff80bd398e in vpanic (fmt=, ap=) at ../../../kern/kern_shutdown.c:902 #15 0xffffffff80bd3723 in panic (fmt=0xffffffff81c8c848 "\025\025\032\201ÿÿÿÿ") at ../../../kern/kern_shutdown.c:839 #16 0xffffffff80bc1f0e in crunuse (td=0xfffffe0158133100) at ../../../kern/kern_prot.c:1892 #17 0xffffffff80bc1e49 in crcowfree (td=0xffffffff81c8c848 ) at ../../../kern/kern_prot.c:1908 #18 0xffffffff80be92f2 in thread_cow_free (td=0xfffffe0158133100) at ../../../kern/kern_thread.c:488 #19 thread_reap () at ../../../kern/kern_thread.c:404 #20 0xffffffff80b8bf5e in proc_reap (td=, p=0xfffff8003b297000, status=0xfffffe09a0013aac, options=) at ../../../kern/kern_exit.c:940 #21 0xffffffff80b8c53e in proc_to_reap (td=0xfffffe014eca4300, p=0xfffff8003b297000, idtype=, id=, status=0xfffffe09a0013aac, options=0x30, wrusage=0x0, siginfo=0x0, check_only=0x0) at ../../../kern/kern_exit.c:1104 #22 0xffffffff80b8b6cb in kern_wait6 (td=, idtype=2684433808, id=0x80, status=, options=0x30, wrusage=0x0, siginfo=0x0) at ../../../kern/kern_exit.c:1230 #23 0xffffffff80b8b2cb in kern_wait (td=, pid=, status=, options=0x0, rusage=0x0) at ../../../kern/kern_exit.c:1145 #24 sys_wait4 (td=0xfffffe014eca4300, uap=0xfffffe014eca46e8) at ../../../kern/kern_exit.c:771 #25 0xffffffff81014969 in syscallenter (td=0xfffffe014eca4300) at ../../../amd64/amd64/../../kern/subr_syscall.c:162 #26 amd64_syscall (td=0xfffffe014eca4300, traced=0x0) at ../../../amd64/amd64/trap.c:1157 #27 #28 0x00000008003e71fa in ?? () Backtrace stopped: Cannot access memory at address 0x7fffffffe4e8 (kgdb) f 16 #16 0xffffffff80bc1f0e in crunuse (td=0xfffffe0158133100) at ../../../kern/kern_prot.c:1892 1892 KASSERT(cr->cr_ref > 0, ("%s: ref %d not > 0 on cred %p", (kgdb) l 1887 td->td_ucredref = 0; 1888 KASSERT(cr->cr_users > 0, ("%s: users %d not > 0 on cred %p", 1889 __func__, cr->cr_users, cr)); 1890 cr->cr_users--; 1891 if (cr->cr_users == 0) { 1892 KASSERT(cr->cr_ref > 0, ("%s: ref %d not > 0 on cred %p", 1893 __func__, cr->cr_ref, cr)); 1894 crold = cr; 1895 } else { 1896 cr->cr_ref--; (kgdb) info loc cr = 0xfffff80d94fe3000 crold = (kgdb) p *cr $1 = {cr_mtx = {lock_object = {lo_name = 0xffffffff8111e904 "cred", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xfffff8103fd79b00}, mtx_lock = 0xfffffe014eca4300}, cr_ref = 0x2, cr_users = 0x0, cr_uid = 0x0, cr_ruid = 0x0, cr_svuid = 0x0, cr_ngroups = 0x3, cr_rgid = 0x0, cr_svgid = 0x0, cr_uidinfo = 0xfffff8000337a600, cr_ruidinfo = 0xfffff8000337a600, cr_prison = 0xffffffff81903fc0 , cr_loginclass = 0xfffff80846d545c0, cr_flags = 0x0, cr_pspare2 = {0x0, 0x0}, cr_label = 0x0, cr_audit = {ai_auid = 0xffffffff, ai_mask = {am_success = 0x0, am_failure = 0x0}, ai_termid = {at_port = 0x0, at_type = 0x4, at_addr = {0x0, 0x0, 0x0, 0x0}}, ai_asid = 0x0, ai_flags = 0x0}, cr_groups = 0xfffff80d94fe30bc, cr_agroups = 0x10, cr_smallgroups = {0x0, 0x0, 0x5, 0x0 }} (kgdb) p *td $2 = {td_lock = 0xfffffe006a02e080, td_proc = 0xfffff8003b297000, td_plist = {tqe_next = 0xffffffffffffffff, tqe_prev = 0xffffffffffffffff}, td_runq = {tqe_next = 0xffffffffffffffff, tqe_prev = 0xffffffffffffffff}, td_slpq = { tqe_next = 0xfffffe01494d3500, tqe_prev = 0xfffffe0157e53830}, td_lockq = {tqe_next = 0x0, tqe_prev = 0x0}, td_hash = {le_next = 0xffffffffffffffff, le_prev = 0xffffffffffffffff}, td_cpuset = 0xfffff800039fb000, td_domain = { dr_policy = 0xffffffff81c8c9b0 , dr_iter = 0x0}, td_sel = 0x0, td_sleepqueue = 0xfffff8010711f580, td_turnstile = 0xfffff80d6885da80, td_rlqe = 0x0, td_umtxq = 0xfffff80d6887eb00, td_tid = 0x18dc7, td_sigqueue = { sq_signals = {__bits = {0x0, 0x0, 0x0, 0x0}}, sq_kill = {__bits = {0x0, 0x0, 0x0, 0x0}}, sq_ptrace = {__bits = {0x0, 0x0, 0x0, 0x0}}, sq_list = {tqh_first = 0x0, tqh_last = 0xfffffe01581331d8}, sq_proc = 0xfffff8003b297000, sq_flags = 0x1}, td_lend_user_pri = 0xff, td_flags = 0x8804, td_inhibitors = 0x0, td_pflags = 0x100, td_pflags2 = 0x0, td_dupfd = 0x0, td_sqqueue = 0x0, td_wchan = 0x0, td_wmesg = 0x0, td_owepreempt = 0x0, td_tsqueue = 0x0, td_locks = 0x0, td_rw_rlocks = 0x0, td_sx_slocks = 0x0, td_lk_slocks = 0x0, td_stopsched = 0x0, td_blocked = 0x0, td_lockname = 0x0, td_contested = {lh_first = 0x0}, td_sleeplocks = 0xffffffff81d249d0 , td_intr_nesting_level = 0x0, td_pinned = 0x0, td_realucred = 0xfffff80008e68200, td_ucred = 0xfffff80d94fe3000, td_limit = 0xfffff80846589400, td_slptick = 0x0, td_blktick = 0x0, td_swvoltick = 0x8066a431, td_swinvoltick = 0x8066a432, td_cow = 0x0, td_ru = {ru_utime = {tv_sec = 0x0, tv_usec = 0x0}, ru_stime = {tv_sec = 0x0, tv_usec = 0x0}, ru_maxrss = 0x0, ru_ixrss = 0x0, ru_idrss = 0x0, ru_isrss = 0x0, ru_minflt = 0x0, ru_majflt = 0x0, ru_nswap = 0x0, ru_inblock = 0x0, ru_oublock = 0x2, ru_msgsnd = 0x2, ru_msgrcv = 0x0, ru_nsignals = 0x0, ru_nvcsw = 0xd, ru_nivcsw = 0x3}, td_rux = {rux_runtime = 0x917b7, rux_uticks = 0x0, rux_sticks = 0x0, rux_iticks = 0x0, rux_uu = 0x0, rux_su = 0x11c, rux_tu = 0x11c}, td_incruntime = 0x0, td_runtime = 0x917b7, td_pticks = 0x0, td_sticks = 0x0, td_iticks = 0x0, td_uticks = 0x0, td_intrval = 0x0, td_oldsigmask = {__bits = {0x0, 0x0, 0x0, 0x0}}, td_generation = 0xf, td_sigstk = {ss_sp = 0x0, ss_size = 0x0, ss_flags = 0x0}, td_xsig = 0x0, td_profil_addr = 0x0, td_profil_ticks = 0x0, td_name = "nfs15lockd\000\000\000\000\000\000\000\000\000", td_fpop = 0x0, td_dbgflags = 0x400, td_si = {si_signo = 0x0, si_errno = 0x0, si_code = 0x0, si_pid = 0x0, si_uid = 0x0, si_status = 0x0, si_addr = 0x0, si_value = {sival_int = 0x0, sival_ptr = 0x0, sigval_int = 0x0, sigval_ptr = 0x0}, _reason = {_fault = {_trapno = 0x0}, _timer = {_timerid = 0x0, _overrun = 0x0}, _mesgq = {_mqd = 0x0}, _poll = {_band = 0x0}, __spare__ = {__spare1__ = 0x0, __spare2__ = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}}}, td_ng_outbound = 0x0, td_osd = {osd_nslots = 0x0, osd_slots = 0x0, osd_next = {le_next = 0x0, le_prev = 0x0}}, td_map_def_user = 0x0, td_dbg_forked = 0x0, td_vp_reserved = 0x0, td_no_sleeping = 0x0, td_su = 0x0, td_sleeptimo = 0x0, td_rtcgen = 0x0, td_errno = 0x0, td_vslock_sz = 0x0, td_kcov_info = 0x0, td_ucredref = 0x0, td_sigmask = {__bits = {0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff}}, td_rqindex = 0x37, td_base_pri = 0x85, td_priority = 0x85, td_pri_class = 0x3, td_user_pri = 0x85, td_base_user_pri = 0x85, td_pre_epoch_prio = 0x98, td_rb_list = 0x0, td_rbp_list = 0x0, td_rb_inact = 0x0, td_sa = {code = 0x5c, callp = 0xffffffff818fb9e0 , args = {0x4, 0xd, 0x7fffdfdfcf00, 0x18, 0x8002c4ab0, 0x6666666666666667, 0x0, 0x0}, narg = 0x3}, td_sigblock_ptr = 0x800a12a38, td_sigblock_val = 0x0, td_pcb = 0xfffffe0158133610, td_state = TDS_INACTIVE, td_uretoff = {tdu_retval = {0x0, 0x7fffdfdfcf00}, tdu_off = 0x0}, td_cowgen = 0x0, td_slpcallout = {c_links = {le = {le_next = 0xffffffffffffffff, le_prev = 0xffffffffffffffff}, sle = { sle_next = 0xffffffffffffffff}, tqe = {tqe_next = 0xffffffffffffffff, tqe_prev = 0xffffffffffffffff}}, c_time = 0x1c9fb4f1e5b1, c_precision = 0x41893, c_arg = 0xfffffe0158133100, c_func = 0xffffffff80c2dcd0 , c_lock = 0x0, c_flags = 0x2, c_iflags = 0x110, c_cpu = 0x1}, td_frame = 0xfffffe013e392c00, td_kstack = 0xfffffe013e38f000, td_kstack_pages = 0x4, td_critnest = 0x1, td_md = { md_spinlock_count = 0x1, md_saved_flags = 0x246, md_spurflt_addr = 0x0, md_invl_gen = {gen = 0x0, {link = {le_next = 0x1, le_prev = 0x98}, {next = 0x1, saved_pri = 0x98}}}, md_efirt_tmp = 0x0, md_efirt_dis_pf = 0x0, md_pcb = { pcb_r15 = 0xfffffe0158133290, pcb_r14 = 0xfffffe0158133100, pcb_r13 = 0xfffff8003b297000, pcb_r12 = 0xfffff8003b297158, pcb_rbp = 0xfffffe013e3921d0, pcb_rsp = 0xfffffe013e392198, pcb_rbx = 0xfffff8003b297230, pcb_rip = 0xffffffff80be99ca, pcb_fsbase = 0x800240920, pcb_gsbase = 0x0, pcb_kgsbase = 0x0, pcb_cr0 = 0x0, pcb_cr2 = 0x0, pcb_cr3 = 0x0, pcb_cr4 = 0x0, pcb_dr0 = 0x0, pcb_dr1 = 0x0, pcb_dr2 = 0x0, pcb_dr3 = 0x0, pcb_dr6 = 0x0, pcb_dr7 = 0x0, pcb_gdt = {rd_limit = 0x0, rd_base = 0x0}, pcb_idt = {rd_limit = 0x0, rd_base = 0x0}, pcb_ldt = {rd_limit = 0x0, rd_base = 0x0}, pcb_tr = 0x0, pcb_flags = 0x11, pcb_initial_fpucw = 0x37f, pcb_onfault = 0x0, pcb_saved_ucr3 = 0x107117a91, pcb_tssp = 0x0, pcb_efer = 0x0, pcb_star = 0x0, pcb_lstar = 0x0, pcb_cstar = 0x0, pcb_sfmask = 0x0, pcb_save = 0xfffffe013e392cc0, pcb_pad = {0x0, 0x0, 0x0, 0x0, 0x0}}, md_stack_base = 0xfffffe013e392cc0}, td_ar = 0x0, td_lprof = {{lh_first = 0x0}, {lh_first = 0x0}}, td_dtrace = 0xfffff80ace552d00, td_vnet = 0x0, td_vnet_lpush = 0x0, td_intr_frame = 0x0, td_rfppwait_p = 0x0, td_ma = 0x0, td_ma_cnt = 0x0, td_emuldata = 0x0, td_lastcpu = 0xc, td_oncpu = 0xffffffff, td_lkpi_task = 0x0, td_pmcpend = 0x0} (kgdb) $ svnlite diff -x -p /usr/src/sys -- Test scenario: misc/nfs15lockd.sh